From 10227c25d4c12af0f73eb8999f3d72552a02b1e1 Mon Sep 17 00:00:00 2001 From: Erik Darling <2136037+erikdarlingdata@users.noreply.github.com> Date: Fri, 9 Oct 2026 14:18:30 -0400 Subject: [PATCH] Release 3.10.0 (#5645) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * WEB: Memory Grants, Resource Semaphore, Plan Corrections and the Query Store clutter tiles say where their data starts (part of #4966) (#5144) The web server page's Memory Grant Pressure, Resource Semaphore Pressure, Plan Corrections and Query Store clutter panels now say where their data starts, so a server with a short history no longer shows a 7-day range as if it were all quiet. The server adds the note for get_memory_grants and get_resource_semaphore (both probe memory_grant_stats, the table their window aggregates read) and get_plan_corrections (plan_correction). Plan Corrections is a newest-first list, so a capped page names the oldest recommendation it returns, and its "empty" answer also gets the note. The page's shared note strip learns three things: a spec can opt out (windowNote: false), which the newest-snapshot panels use (the two grants lists and Automatic Tuning); a stat tile can draw it as well as a table; and a spec can read the note from a nested block (floorKey), which the Query Store Clutter and Query Store Overhead panels use for the note their tool already returns. The shared fanout's empty path now draws the note as the single-panel path does. Tests: page scenarios through the shipped util.js, panels.js and server-tabs.js in the Node harness (the opt-out, a stat, the nested note, the fanout's snapshot sibling, an empty Plan Corrections answer with and without the flag, Automatic Tuning quiet); the capped-rule tests extended to Plan Corrections; a live case for a memory read whose rows start inside the window; the admitted status words. Part of #4966 * Daily Query Store summary for get_query_store_top's long windows: the tables (rung V161) and the hourly builder (#5094) (#5125) Daily Query Store summary for get_query_store_top's long windows: the tables (rung V161) and the hourly builder (#5094) Part of #5094. No reader uses the summary yet; the read and its "approximate" flag come next. - Rung V161 adds two plain tables. collect.query_store_top_daily holds one row per server, UTC day and query identity (database, query id, plan id, query hash, outcome, replica role, module), with sums and non-NULL counts that recombine exactly, so a reader reproduces today's unweighted mean of interval averages. Its unique key is NULLS NOT DISTINCT. collect.query_store_top_daily_built records each built day's pass. The viewer's schema probe gains the V161 sentinel and arm. - QueryStoreTopDaily is the hourly store-maintenance tick's fifth tenant. Pass 1 builds a day at its end + 2 h, pass 2 rebuilds it at its end + 25 h, and a day first seen past that point is built once at pass 2. Each build is one transaction under a 60 s statement timeout, bounded on first_execution_time to the day (−26 h, +1 h), for enabled servers and days after their coverage claim, oldest first. The tick does at most 60 builds and starts none after 10 minutes; GC follows the interval table's 9-day horizon and disabled servers. - Approximate by design (Erik's call on #5094): a late write into a built day is missed, a row moving into the next day after pass 1 counts twice until pass 2, and a monitored server clock more than an hour ahead drops rows past the day's end + 1 h. Each case is pinned by a live test. - On a production store, one build of the largest server's day read about 197k interval rows (30k blocks) into 26k groups in 2.3 s. * Add a Manage Tags page to the web dashboard (#5085) (#5138) The web dashboard gains a Manage Tags page, over the server-tag write routes (#5137). It reads the tag forest and each server's tags from the existing `/api/fleet`, so it adds no route, grant or schema change. **What the page does.** - **The tree:** up to four levels, with each tag's colour and its server count from the payload. - **Tag actions:** create a root or child tag, rename, recolour, move under another parent, and delete. - **Delete:** the page first asks the server without confirming. If deleting would leave custom alert rules matching no server, it names those rules and asks again, then sends `?confirm=true`. - **Assignment:** a server checklist assigns the selected tag. The page records only the boxes the user changed, so Apply sends one assign for the added servers and one unassign for the removed ones, and never touches a server someone else changed meanwhile. - **After each write:** the page lists the alert rules whose coverage changed. - **The 60-second refresh** keeps the selection, the unsaved checkbox changes (marked "(unsaved)") and the open form, including the focused input. - **In-flight writes:** while a write is running, a second click sends nothing, and Cancel and the form openers wait for it. - **A parent deleted elsewhere** keeps the form and what was typed, and resets the parent to none. - **Read-only sign-ins** see the page with every write disabled. Every server string reaches the page through the house `el()` helper as text. A colour is used in a style only after it matches `#RRGGBB`. **Tests.** The behaviour tests drive the page under Node with a fake fetch, and check each request's method, path, content type and body: - the tree and the forms; - the delete confirm flow; - the assign and unassign diff, including servers changed elsewhere; - the refresh keeping the form and the edits; - double-click guards; - 403, 404, 409, 415, 500 and network refusals; - read-only. Page tests pin the routes and methods. Part of #5085 * get_read_latency fits its rows to the MCP response budget (#5135) (#5149) `get_read_latency` returned up to `limit` rows (at most 1,000) and never fitted its response to the shared MCP response budget (`McpResponseBudget.DefaultBytes`, 32,768 bytes). With 1,000 routes, a `limit=1000` answer was 227,445 bytes. The tool now builds its response the way the availability-group reader does: - it takes rows in the total order (p95 descending, then count, then surface and route) while they fit; - it then serializes the real final response, dropping the last row and re-serializing while it is still over the budget. The answer can therefore never exceed the budget, even by the few bytes the counters add. **New fields.** The response carries `truncated` (the limit or the budget cut the page), `reads_returned` and `reads_total`. Rows always drop from the end of the order. The guide tail says the page is capped at `limit` rows or the budget, whichever cuts first, and to raise `limit` or narrow with `surface` or `route`. The tool head is unchanged. The web read route returns the same capped answer. **Tests.** - A live test seeds 1,000 routes, then asserts the size bound, the three fields, and that the rows returned are the top of the order. - A unit test sweeps the budget across the last row's edge, and asserts the UTF-8 size never exceeds it. Fixes #5135 * MCP: Darling's four data-start twins, and Lite's config change, collection log, plan correction and memory pressure tools, say where their data starts (part of #4966) (#5147) This lands two changes for #4966: the Darling MCP twins of Lite's data-start tools (first reviewed as #5131), and Lite's config-change, collection-log, plan-correction and memory-pressure tools. Darling's get_active_queries, get_waiting_tasks, get_query_heatmap and get_query_store_regressions now say where their data starts, with the contract Lite's twins use (#5015): effective_start (UTC, with a Z), window_truncated and truncation_note right after hours_back on a data answer, and the same three keys under hints on an empty answer, so an empty or short answer over a window the store never covered no longer reads as a quiet one. not_collected and unavailable answers keep their shape. get_waiting_tasks, the heatmap and the regressions tool always carry the keys, as Darling's Queries tools do; get_active_queries carries them only when the window was cut, by the house rule that a payload omits null and false fields. A shared helper, DarlingMcpWindowNotice, probes coverage through DataWindowFloor.GetAsync (the registry's created_date and the purge edge, a run counting as well as a row), with Lite's sentences copied and pinned. A window of 90 minutes or less with rows starts no probe; an empty answer always probes; the regressions tool probes from its baseline start. A probe that fails costs the notice, never the rows: it is logged and the answer goes without the keys. The web keeps its own note for the reads it lists (WebDataStartNote strips the tool's keys before deciding); the Active Queries, Query Heatmap and Query Store Regressions panels now show the tool's own sentence, as the Top Queries panels already do. In both apps, get_active_queries now previews query text at 400 characters instead of 500, so a cut window (keys plus note) stays inside the 32,768-byte response budget: the default call over the budget test's 50 heavy rows is 30,223 bytes covered and 30,504 cut. get_active_queries' and get_waiting_tasks' descriptions now name the three keys in both apps. Tests: helper tests (the 90-minute rule, the null floor, the exact sentences, a probe that throws, a caller who cancels); live tests for each tool (a notice when coverage starts inside the window, none on a quiet start, hints on an empty answer, a failed probe that keeps the rows); a covered and a cut budget case; the web guard; the census rosters and the shared-formatter pin. The full Darling suite fails only tests on the Mac-only list; Lite's description, budget and guide classes pass on Windows. Lite's get_server_config_changes, get_database_config_changes, get_trace_flag_changes, get_collection_log (one server), get_plan_corrections and get_memory_pressure_events now say where their data starts, with the contract the other Lite window-floor tools use (#5004, #5015, #5127): effective_start (UTC, with a Z), window_truncated and truncation_note right after hours_back on a data answer, the same keys under hints on an empty answer, and not_collected and unavailable answers unchanged. The fleet form of get_collection_log names no server, so it carries no notice. Each probe reads the source its tool returns rows from: the config snapshots on capture_time with their collector's runs, the collection log over the read's own window, v_plan_correction, and memory pressure on sample_time. None needs the event-time rule, because each probe already reads the column the rows are windowed on. The config change lists diff each in-window snapshot against the one before it, so a snapshot from before the window also counts as coverage, but only when the server was monitored during the window (at least one collection-log run); with no run in the window the answer says nothing was read (LocalDataService.GetConfigSnapshotCoverageFloorAsync). A plan-corrections answer that holds only automatic-tuning rows is probed as a data answer. The one-server collection log now treats a status filter as a filter, so "no ERROR runs" takes the filtered-empty answer rather than the quiet-window one. Tests: McpWindowNoticeConfigAndLogToolTests (new): for each tool, a notice when collection starts inside the window, none on a quiet start, hints on an empty answer past coverage, the key order, a short window that starts no probe, a capped page where the tool pages; the config tools' pre-window snapshot with and without a run in the window; a status filter that matches nothing; a one-hour automatic-tuning-only answer. The census rosters and the shared-formatter pin cover the four files. On Windows, the three Lite window-notice classes pass (147). Part of #4966 * Web grids: column groups with a toggle strip, first used on Memory Conditions and Node OOM (#5152) Part of #4843 ## What users see change - The System Events **Memory Conditions** (32 columns) and **Memory Node OOM** (28 columns) tables now open narrow. A "Columns:" strip above the table has one toggle per column group, and a pressed toggle shows its columns. - Memory Conditions: Event Time, Last Notification, Report and the OOM counters are always shown. Groups: Process memory (on at first), Pressure flags, Allocation detail. - Node OOM: Event Time and the node ids and utilisation are always shown. Groups: Failure (on at first), Memory sizes, Pool detail, Flags. - The choice survives the 60 s repaint. Another server's table starts from the defaults. - Every other grid is unchanged. ## What changed - `panels.js`: `vizTable` takes `groups` and `defaultGroups`. A column carries `group: ""`; a column with no group is always shown. The toggle state sits in a module-scope map, keyed like the grid sort state (route, table identity, columns). A hidden column is still built and only set to `display: none`, so the sort, Copy and CSV column indexes stay aligned. - `server-tabs.js`: `table()` takes an optional last argument `columnGroups = { groups, defaultGroups }`. The two wide column constants are grouped in the desktop grid's order, and #5118's "picker candidate" comments are gone. - `css/app.css`: `.col-picker` rules beside the `.grid-tools` rules. - No new module, so no harness needed an import change. ## Decisions - **Copy and CSV export all columns, hidden ones included.** The desktop grid's export (`DataGridExport`) iterates every column of the grid, shown or not, so the web matches it. Copy cell and Copy row follow the same rule. - **Sort:** hiding a sorted column keeps the sort. The order holds, the repaint re-applies it, and the other columns still sort. ## What's not changed - No other grid gets groups. The wider Top Queries and Query Store grids are for the following lanes, which can reuse this option. ## Verified at the head - New `ColumnGroupsBehaviourTests` (node, 6 tests): no groups gives an unchanged table, defaults, toggling, rebuild and server switch, sort with a hidden column, and Copy carrying all columns. RED on dev: with dev's `panels.js` four scenarios crash with no picker and the fifth shows no strip. 6/6 green now. - The no-groups case asserts the DOM is dev's: children are exactly `grid-tools`, `table-wrap`; no th or td carries a `display` style; `tools:false` returns the bare `table-wrap`. - The sort case uses a fixture where A's order (y, x, z) differs from B's (y, z, x). With the comparator forced to return 0, the check fails (`byA` comes back x, y, z). - New `ServerTabColumnGroupsPinTests` (2): each grid's groups match its group list and defaults, and each constant is the last argument of its own `table()` call (Memory Conditions, Memory Node OOM). 2/2. - `ServerPageTabsTests` 19/19, `WebModuleLoadTests` 7/7, `GridToolsBehaviourTests` 9/9, `PgIndexBloatCoverageTests` 16/16 (this file is not changed by the PR; its `span, noteKey }` pin still matches because the `table()` literal keeps that text). - The wider guard family and every `.mjs`-harness class: 239 classes run on this Mac. The one failure, `LivePostgresCollectionHygieneTests`, is in `dev-local-failing-tests.txt`. - `ServerPageTabsTests` was updated: it now lists `columnGroups = null` in the `table()` signature. - `scrubcheck` on the diff exits 0. ## CHANGELOG entry SECTION: Added ENTRY: - **Column toggles on the wide System Events grids** ([#5152]) - Memory Conditions and Memory Node OOM open with a short default column set and a "Columns:" strip that reveals the rest, and the choice survives the page's refresh. REF: [#5152]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5152 * The web Alert History page gains Dismiss Selected and Dismiss All (#5151) Part of #4843 ## What users see On the web Alert History page, a signed-in seat that can edit now gets: - a checkbox on every live row; - a **Dismiss Selected** button (it shows the checked count); - a **Dismiss All** button that dismisses every live row the current filter box shows, after the Darling Viewer's prompt ("Dismiss N alert(s)? … hidden from this view but remain in the store"). When the page is truncated (more alerts than the row limit shows), the prompt adds that only the listed alerts are dismissed and the rest stay live. After a dismiss the list refreshes. With Show dismissed off the rows disappear. With it on they stay and say "Dismissed". A short status line reports the counts, for example "Dismissed 12 of 12", or "Dismissed 1 of 4; 3 were already gone". A refusal (403, 401, 400, 5xx) shows the server's text in the status line and the page never claims success. A read-only seat sees none of this. ## What changed - `alerts.js`: the select column, the dismiss bar and the write, which goes through the shared `apiSend` helper (JSON content type, session-expiry handling) to `POST /api/alert-history/dismiss` (#5106). The body is `{alerts:[{alert_time, server_id, metric_name}]}`. - The checked rows live at module scope, keyed by the row key. They survive the 60 s rebuild and a visit elsewhere and back. After a successful dismiss only the keys that were sent are removed, so a box ticked while the request was in flight stays checked. A failed or empty poll read keeps the set; only a successful read prunes rows that left the list. The set is cleared when the range, row limit, server, Show dismissed or filter box changes. - Dismiss All sends the listed rows, with no new route. This differs from the Viewer, whose Dismiss All covers every live alert in the window, even past the row limit; the web keeps its listed-rows design and says so in the prompt when the page is truncated. The route caps a request at 1000 keys, so a longer list goes in chunks of 1000 and the counts are added up. A failure part-way says how far it got and refreshes the list. - Dismissed rows, and rows with no `server_id`, get no checkbox and are not sent. - `panels.js`: a column flag `copy: false` leaves a column out of Copy row and Copy all (the CSV already honours `csv: false`). The Select column sets it; the Mute column is unchanged. Copy cell and the picked-cell lookup still see every column. - `css/app.css`: three rules beside `.filter-box`. - The #5109 range controls and the #5110 `markDismissed` / `tools: false` row build are untouched. ## What's not changed No server code, no other page, `util.js` or `server-tabs.js`; in `panels.js` only the `copy` flag. There is no "since" route. ## Tests - New `AlertHistoryDismissBehaviourTests` (13, with `alert-history-dismiss-harness.mjs`, run under Node on the shipped page): no controls for a read-only seat; Dismiss Selected posts the right keys and refreshes; Show dismissed keeps the row with "Dismissed"; Dismiss All posts what the filter shows; chunking at 1000; the checked set surviving a poll and a re-open and clearing on a filter change; a 403 and a 401 shown and the selection kept; the counts line; the prompt wording and the truncated-page line; Copy row and Copy all without the Select column and with Mute kept; a failed poll read keeping the checked set; a box ticked mid-dismiss surviving unsent. - `AlertHistoryRangePageTests` dropped its "page has no Dismiss Selected" assertion. `MuteRulesPageTests` pins the new column expression. The range harness's metric column index moved from 2 to 3 because of the select column. ## Verified at the head - RED on dev: with `alerts.js` reset to dev, 8 of the original 9 new tests fail; the four added for the prompt, Copy and selection handling also fail there. The read-only test passes there because it has nothing to show. - GREEN: `AlertHistoryDismissBehaviourTests` 13/13, `AlertHistoryRangePageTests` 12/12, `MuteRulesPageTests` 6/6, `WebModuleLoadTests` 7/7. `load-web-modules.mjs` loads all 45 modules. - Every `.mjs`-harness class and 104 classes in total (those plus the whole Census/Ratchet/Adoption/Discipline/Guard/Scrub/Hygiene/Pin/Convention family): one failure, `LivePostgresCollectionHygieneTests.EveryClassUsingTheSharedStore_IsSerializedOrDocumentsWhyNot`, which is on `dev-local-failing-tests.txt`. - The guard-family run above covers 238 classes, including `GridToolsBehaviourTests` 9/9 (the shared renderer changed) and `WebModuleLoadTests` 7/7. The earlier full-suite run was before the review fixes; the full suite is left to CI. 0 build warnings. - `scrubcheck` exits 0 on the diff and on this file. - Not run: nothing required. Lite.Tests is untouched. ## CHANGELOG entry SECTION: Added ENTRY: - **Dismiss Selected and Dismiss All on the web Alert History page** ([#5151]) - a seat that can edit gets row checkboxes and the desktop's two dismiss buttons, with a status line of how many alerts were dismissed. REF: [#5151]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5151 * Web: cross-server Job History page, plus a Job History grid on the Activity tab (#5150) Part of #4843 ## What users see change - A new **Job History** entry at the end of the web nav opens a page showing SQL Agent job runs (steps and job outcomes) across the fleet, newest first, the web twin of the desktop Job History view. - Columns follow the desktop grid: Run Time, Server, Job, Category, Step, Status, Duration, Retries, Last Success, Message. Sort, Copy and CSV come from the shared grid. - Filters: Server (all or one), Window (1/4/12/24 hours, 7 days; default 24 hours), Job (exact name, applied on Enter or when the box loses focus), Status (all, Failed, Succeeded, Retry, Canceled), Category (the categories seen so far), and Rows (100/250/500/1000, default 100). All filters are applied by the read before its row limit. Choices, typed text and focus survive the 60 s refresh. - The page says when the window reaches past the history the store keeps (also when the answer is empty), and when the row limit cut the list. A "N runs shown" line sits above the grid. - Rows are coloured as the desktop grid does: failed red, long-running and retry amber, canceled grey. The desktop also dims disabled jobs; the read does not return an enabled flag on this path, so the web does not. - Differences from the desktop tab: the windows are 1/4/12/24 hours and 7 days (the desktop also has 30 d, 90 d and 1 y; the read's maximum is 168 hours), and there is no 2,000-row cap, the Rows choice goes to 1000 at most. - The Server choices list SQL Server targets only (PostgreSQL targets have no Agent history). - The server page's Activity tab (SQL Server) gains a per-server Job History grid with the same columns less Server (newest 100 runs in the page range). It was not on the web before. ## What changed - `pages/job-history.js` (new), reading `get_job_history` through `readToolWithinKeptHistory` with the render's `AbortSignal`; server choices from `list_servers`. - `index.html` and `app.js`: one nav entry, import, route, active-nav key, each added at the end of its list. - `server-tabs.js`: the Activity-tab grid and its column list only. - `job-history-harness.mjs` and `JobHistoryPageTests.cs` (15 tests). - `app.css`: three row-colour rules beside the data-table cell colours. `util.js`: `run_time` added to the instant fields, so a custom range trims the per-server grid. No C# product change. - The job box defers its blur by a turn, so a rebuild no longer drops focus or fires a second read, and the caret is kept. The empty-answer notice reads `hints`. ## What's not changed `panels.js`, the rest of `util.js`, `alerts.js`, the `app.js` scheduler and status bar, other `server-tabs.js` regions. The PostgreSQL Activity tab has no Job History (Agent history is SQL Server only). The desktop's Agent status indicator is not on this page (it needs a separate `get_agent_status` read). ## Verified at the head - `JobHistoryPageTests`: 15/15 pass; all FAIL on dev's wwwroot (page and wiring removed). - `WebModuleLoadTests` passes; module count comes from disk. - Census/Ratchet/Adoption/Discipline/Guard/Scrub/Hygiene/Pin/Convention family (79 classes) plus every `.mjs`-harness class (26): 1255 run, 1 failed, `LivePostgresCollectionHygieneTests.EveryClassUsingTheSharedStore_IsSerializedOrDocumentsWhyNot`, which is in `dev-local-failing-tests.txt`. - Full Darling.Tests (run at the previous head; this push re-ran the page, harness, load and guard classes only): 22703 run, 478 failed (377 distinct methods). All are on `dev-local-failing-tests.txt` except two `DarlingManagedPostgresTests.EnsureConfAppended_*` methods; they do not touch web files and I did not run them on a clean dev, so CI decides. - `scrubcheck` on the diff and this file: exit 0. ## CHANGELOG entry SECTION: Added ENTRY: - **Job History page in the web dashboard** ([#5150]) - A cross-server Job History page shows SQL Agent job runs with server, window, job, status, category and row-limit filters, and the Activity tab gains a per-server Job History grid. REF: [#5150]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5150 * The web dashboard shows a query's stored plan XML from Top Queries, High Impact and Expensive Queries (#5156) Part of #4843 ## What users see Rows that carry a `query_hash` get a **Plan** button in Top Queries (server page), FinOps High Impact and FinOps Optimization > Expensive Queries. It opens a panel under the row with the query's stored plan XML, indented, with **Copy** and **Download .sqlplan** (file named `.sqlplan`, type `application/xml`). A plan larger than the read's 500 KB cut says so and turns Download off, because a cut file will not open. A query with no stored plan shows the read's own message. The open panel stays open through the 60 s rebuild. ## What changed - New `pages/plan-viewer.js`: `planColumn(server)` (added to each grid's column list), `storedPlanCell`, `openStoredPlan`, `prettyPrintXml`, `downloadPlan`. It reuses `copyText` from `grid-tools.js`. The XML is drawn only through `textContent` of a `
`.
- `get_plan_xml` returns bare XML for the MCP clients (unchanged). The web read route wraps a stored plan as 200 JSON `{query_hash, database_name, plan_xml, truncated}`; `truncated` is decided on the server from the tool's own cut marker, which is removed from `plan_xml`. A "no plan" status or error envelope passes through unchanged. The viewer reads `plan_xml` and `truncated`; it no longer sniffs text. Params: `server`, `query_hash`, and `database_name` when the row has one.
- Expensive Queries rows now carry `query_hash` (`MAX(query_hash)` in the existing statement query, a new optional `QueryHash` on the `ExpensiveQuery` record, and a `query_hash` field in the `get_finops` optimization view's expensive-query rows), so the Plan column shows there. No tool description changed.
- Open panels are keyed by server + database + hash at module scope; a rebuilt cell redraws from that state, and a read still in flight fills the new cell too.
- Top Queries (both the tab and the standalone panel), High Impact and Expensive Queries use `[...COLUMNS, planColumn(server)]`; the column hides when no row has a hash. `css/app.css` gets three rules beside the grid-tools rules.
- Every `.mjs` harness that copies `server-tabs.js` or the finops tabs now also copies `plan-viewer.js`.

## Not changed
Get Actual Plan and a plan tree view stay out. The PostgreSQL Top Query Shapes grid has no Plan column: `get_plan_xml` is SQL Server only (pinned). No `panels.js`, `util.js`, `app.js` or `alerts.js` edits.

## Verified at the head
- Web wrapper: 3 new `DarlingWebEndpointsTests` cases (whole plan gives JSON that classifies as passthrough; a cut plan sets `truncated` using the real `McpHelpers.Truncate` marker; envelopes pass through byte-identical). `DarlingWebEndpointsTests` 76/76. The plan harness now fakes the 200 shape. `FinOpsTabOptimizationPageTests` pins the new `query_hash` row field.
- `WebExceptionTextCensusTests` 40 (1 skipped), `DarlingCustomViewsTests` 71/71 (`get_plan_xml` params unchanged; a custom view calling it now gets a `plan_xml` JSON object instead of a 400 error strip), `McpToolsListBudgetTests` 5/5, `FinOpsOptimizationViewTests` 8/8.
- RED on dev: `WebPlanViewerPageTests` 5 of 6 fail, `PlanViewerBehaviourTests` 7 of 7 fail (module absent).
- GREEN: `WebPlanViewerPageTests` 6/6, `PlanViewerBehaviourTests` 7/7 (open, text-only, download name/type/body, copy, truncation disables Download, no-plan text, state survives rebuild).
- Updated import allowlists in `FinOpsTabHighImpactPageTests` (8/8) and `FinOpsTabOptimizationPageTests` (16/16); `RepoFileAdoptionTests` 2/2.
- All 100 `.mjs`-harness and Census/Ratchet/Adoption/Discipline/Guard/Scrub/Hygiene/Pin/Convention classes run; only `LivePostgresCollectionHygieneTests` fails, which is on the Mac-local dev failing list.
- Build 0 errors, no new CA/CS/IDE warnings.

## CHANGELOG entry
SECTION: Added
ENTRY: - **Stored query plans on the web dashboard** ([#5156]) - Top Queries, High Impact and Expensive Queries rows with a query hash get a Plan button that shows the stored plan XML with Copy and Download .sqlplan.
REF: [#5156]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5156

* PostgreSQL plans are grouped by their shape, so estimate drift no longer reads as a plan change (fixes #5114) (#5155)

PostgreSQL plans are now grouped by their shape. The plan hash (PgPlanLogParser.FromBlock's PlanHash) covered the whole redacted plan JSON, cost and row estimates included, so a statistics refresh, a different literal or a different work_mem gave one plan shape a new hash, and the plan-flip finding read estimate drift as a plan change. The hash now covers the shape only: it is taken over a clone of the redacted plan with the estimates, the runtime counters (Actual *, Rows Removed by *, Index Searches, Heap Fetches, sort, hash and memoize counters, Workers and Workers Launched), the buffer, I/O and WAL counters, Workers Planned, and the root's Settings, JIT, Planning, Planning Time, Execution Time and Triggers removed, recursively through every child plan. It is a denylist, so an unknown key still splits the hash. The stored plan_json is unchanged and keeps every estimate. A plan without any of those fields hashes as before.

Captured-plan lists (get_pg_plans and the viewer) fold a shape's estimate variants into one row, so captures and durations sum over the shape. Each row now shows the JSON of that shape's latest collection cycle: the reader ranks and cuts the groups first and fetches the JSON for the page only, instead of the old min(plan_json), which was arbitrary once a group held JSONs with different estimates, and aggregated kilobyte-sized text over every group.

On the first analysis pass after the upgrade, a statement captured on both sides of it may report one plan change, and captured-plan lists and the store log's slow_plan groups show pre-upgrade groups beside new ones until the older rows age out (plan retention is 14 days by default).

Tests: PgPlanShapeHashTests (new, 29 cases): estimate, settings, timing and counter variants give one hash, including on a grandchild and a SubPlan node; a different node type, join order, join method, index or parallel awareness gives another; Workers Planned is not shape; an unknown key still splits; the stored JSON keeps its estimates; the hash stays 32 hex characters. A store-log case (two plans differing only in costs are one row) and a live reader case (one group, two captures, the later capture's JSON).

Fixes #5114

* Record the target session id on SQL Server collector runs (#5132) (#5154)

`collection_log.target_session_id` was null on every SQL Server collector run. Monitored-server connections use Multiple Active Result Sets, and under MARS `SqlConnection.ServerProcessId` is 0 at every point: after open, with a reader open, after it closes, and on pool reuse. Measured on SQL Server 2025 with Microsoft.Data.SqlClient 7.1.1. With MARS off it is the real session id.

**The fix.** Right after a collector's target connection opens, before any reader, the runner resolves the session id:
- When `ServerProcessId` is 0, it runs `SELECT @@SPID` once, with a 5-second command timeout, and caches the answer under the connection's `ClientConnectionId`.
- That id stays the same across pool reuse, so a reused connection costs no query, and the cost is one lookup per new physical connection rather than one per run.
- The cache is bounded at 4,096 entries.
- The lookup is timed into the run's SQL time.
- A failure records null and never fails the run. A real cancellation still propagates.
- If SqlClient transparently replaced the physical connection by the time the first reader opens (a new `ClientConnectionId`), the run records null instead of a stale id.
- PostgreSQL targets still read the backend pid directly.

MARS, the pools and the application name are unchanged.

**Why not the alternatives.** Turning MARS off would break the places that run a second command while a reader is open. A per-collector application name would multiply the connection pools on every monitored server.

**Tests.**
- Unit facts for the cache: one lookup per connection id, the bound, failures not cached, a real cancellation, the key read after the lookup, and eviction of a replaced connection.
- A source pin: the lookup comes before every reader.
- Live facts against a real SQL Server: the true session id; no second lookup on a pooled reopen; a full collector run writing a non-null `target_session_id` into its own scratch store.

CI runs no SQL Server, so it skips the live facts. They were run locally.

Fixes #5132

* Web Blocking tab: desktop columns on the Blocking grid, and Save XML on Blocked Process Reports and Deadlock Graphs (#5164)

Part of #4843

## What users see
On a server page's Blocking tab:
- **Blocking grid** now shows the desktop Blocked Process Reports grid's columns, in its order and under its headers (Event Time, Blocked SQL, Blocking SQL, Source, Database, Blocked SPID, Blocking SPID, Wait Time, Wait Resource, Lock Mode, then status, isolation, transaction, priority, log used, login, host and app for each side). It has 25 columns, so the status/isolation, transaction and sessions columns sit behind the column-group toggles from #5152 (status/isolation and transaction on at first).
- **Blocked Process Reports** panel: desktop headers (Event Time, Blocked SPID, Blocking SPID, Wait Time) and a **Save XML** button per row, downloading `blocked_process_.xml`.
- **Deadlock Graphs** panel: a **Save XML** button per row, downloading `deadlock_.xdl` (the item #5139 deferred until a download helper existed).
- A deadlock graph the read cut short (`deadlock_graph_xml_truncated`) gets a disabled button with a hover reason. The tab reads with `full_graph` on, so this is the rare case. A row with no XML also gets a disabled button.

## What changed
- `grid-tools.js`: new `downloadText(fileName, parts, type)`; `downloadCsv` now calls it (same Blob, same type).
- `server-tabs.js` (Blocking/Deadlock regions only): `BLOCKING_COLUMNS` (11 -> 25) with `BLOCKING_GROUPS`; `BPR_COLUMNS` reordered with the desktop headers and an XML column; `deadlockXmlColumns` gets the XML column; `saveXmlButton` and `fileStamp` helpers; one import.
- No fetch on click: both reads already return the full XML on the row, so nothing is in flight. There is therefore no in-flight state.
- No harness `.mjs` import list changes: `grid-tools.js` is already in every harness's conditional copy list.
- Tests: `WebBlockingColumnsPageTests` (column keys against `get_blocking` / `get_blocked_process_xml` projections, desktop order and headers, group names, file names); `WebBlockingSaveXmlBehaviourTests` + `web-blocking-save-xml-harness.mjs` (Node: Blob name, type and text for both downloads, truncated disables Save, no-XML disables Save, no extra read).

## Not changed / read-widening follow-ups
- The desktop's Blocked Process Reports grid has no column the read lacks. Not on the web grid: the blocking-side transaction/priority fields, which the desktop does not show either.
- `get_blocked_process_xml` returns only 6 fields, so the Blocked Process Reports panel stays at the desktop's four data columns; the desktop's extra columns live on the Blocking grid (from `get_blocking`).
- **`.xdl`, not `.xml`:** the Viewer and Lite save deadlock graphs as `.xml`; the web uses `.xdl`, which SSMS opens as a graph (the deprecated Dashboard did the same). A follow-up can move the desktop to `.xdl` too.
- Reads not widened.

## Verified at the head
- New page tests: 6/6 pass; node behaviour tests: 4/4 pass. On dev's JS (`git checkout` of the two files, rebuilt tests kept): page 4/6 fail, behaviour 4/4 fail (RED).
- Full Darling.Tests on this Mac: Total 22912, Failed 478 (336 distinct methods); 334 are in `dev-local-failing-tests.txt`. The other two, `DarlingManagedPostgresTests.EnsureConfAppended_*` (2 methods), fail with `kernel32.dll` not loadable on macOS (a platform failure, nothing to do with this change).
- That run included `WebModuleLoadTests`, `DeadlockProcessRowsTests`, every `.mjs`-harness class and the Census/Guard/Hygiene family, which showed no failures beyond the above.
- Build: 0 warnings, 0 errors.

## CHANGELOG entry
SECTION: Added
ENTRY: - **Web Blocking tab shows the desktop's blocking columns and saves report and deadlock XML** ([#5164]) - the Blocking grid has the desktop Blocked Process Reports columns (the wide ones behind column-group toggles), and the Blocked Process Reports and Deadlock Graphs rows have a Save XML button that downloads the XML (`.xml`) or the deadlock graph (`.xdl`).
REF: [#5164]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5164

* Web Perfmon: multi-select counter picker over a multi-series trend chart (#5162)

Part of #4843

## What users see change
The Perfmon chart on a server's page used to chart one counter chosen from a drop-down. It now matches the desktop viewer's picker and the Wait Stats picker from #5119:
- a checkbox list of the counters in the snapshot table above it, with a search box, Select All, Clear All and a Default counters button;
- the default set is the desktop's General Throughput pack (Batch Requests/sec, SQL Compilations/sec, SQL Re-Compilations/sec, Query optimizations/sec, Network IO waits), limited to what this server collects; if none of the pack is collected the first counter is checked;
- up to 10 counters at once (the 11th box is disabled with a "Limit reached" hint; the desktop's cap is 12);
- one line per checked counter on one zoomable chart. Rate counters plot their per-second figure on a per-second axis; a chart that mixes in a gauge or average counter plots plain values with no unit;
- a counter whose read fails gets a note above the chart and the other lines still draw; all reads failing shows an error plus the notes; all reads empty shows the empty state, not an error;
- checks and search text survive the 60 s refresh, per server; a checked counter that leaves the snapshot stays checked and listed; the search box keeps focus and caret.

## What changed
- `server-tabs.js`, Perfmon region only: `perfmonPanel` and `drawPerfmonTrends` (replaces `drawPerfmonTrend`; `perfmonTrendLines` is gone). One `get_perfmon_trend` read per checked counter, in parallel, with an AbortController per server for superseded draws, and the chart draws at most 10 counters whatever it is handed.
- `multi-picker.js`: three optional options (`noun`, `defaultsLabel`, `defaults`) so the wording and default set are not wait-specific. Defaults are the old values, so Wait Stats is unchanged.
- `BaselineDiscontinuityRenderCensusTests` (Lite) now looks for `export async function drawPerfmonTrends(`. Its logic still holds: that function keeps one `discontinuityNotes(trend.data)` use, and the file total stays 4. Read by hand; Lite.Tests was not run on this Mac.
- New `WebPerfmonMultiSelectBehaviourTests` (14) with `web-perfmon-multi-harness.mjs`; `WebPerfmonPerSecondBehaviourTests` expectations changed from "Per second"/"Value"/"Delta" series to one line per counter.

## What is not changed
- The snapshot table, its per-second columns, and `get_perfmon_trend`. The read takes a counter name and no instance, so the list is by counter name (the desktop also lists by counter name). The desktop's other packs are not offered.
- `panels.js`, `util.js`, `charts.js`, `app.js`, CSS (the existing picker rules are enough).

## Also
- The trend harness copies `pages/plan-viewer.js` when it exists, so it keeps loading once the plan viewer import lands in `server-tabs.js`.
- When none of the default pack is collected, the first counter is checked (the desktop checks none); this continues the page's earlier behaviour.

## Verified at the head
- `WebPerfmonMultiSelectBehaviourTests`: 15 pass (one new: a counter with no trend lists the collected counters as text, "Collected here: …", instead of pointing at a JSON path). `WebPerfmonPerSecondBehaviourTests`: 10 pass. `WebWaitStatsMultiSelectBehaviourTests`: 14 pass, unchanged.
- RED on dev: the new harness cannot pass against dev's `server-tabs.js` (no picker, no `drawPerfmonTrends`; the `defaults` scenario fails at the missing Select All / mp-count node).
- Every `.mjs` / `load-web-modules` class, `WebModuleLoadTests` and the Census/Ratchet/Adoption/Discipline/Guard/Scrub/Hygiene/Pin/Convention family (104 classes, 1,300 tests): 1 failure, `LivePostgresCollectionHygieneTests` (Mac-local; it cannot load WPF types here).
- `scrubcheck` on the diff: exit 0.

## CHANGELOG entry
SECTION: Added
ENTRY: - **Web Perfmon chart charts several counters at once** ([#5162]) - the chart now has the desktop's checkbox list with search, Select All, Clear All and the General Throughput default set, up to 10 counters on one zoomable chart. Your choices are kept across the page's refresh.
REF: [#5162]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5162

* WEB: the PostgreSQL tabs' window aggregates and trend grids say where their data starts (part of #4966) (#5165)

The web server page's PostgreSQL tabs now say where their data starts on eleven more reads: top queries, blocking, database stats, session states, I/O stats and replication stats, the database, query-duration, wait and I/O trend grids, and captured plans. WebDataStartNote lists each read with the table its rows come from; blocking, session states and replication stats are sparse (most samples store nothing) and have no retention edge, so they are probed on their own collector's runs instead (CollectorRunsByRead), and a long-monitored server whose first blocking chain or replica came recently reads as covered.

Each read's "captured, found nothing" status word is admitted where it has one (no_blocking_sampled, no_io_activity, empty for the database, session-state and trend reads), never not_sampled, unavailable or precondition; tools that answer rows with a status word and no message (session_states, blocking_sampled, cycles_only, the trend words and others) are accepted as row answers. No stat tile beside these reads shows only newest values, so no panel opts out, and server-tabs.js is unchanged.

Tests: the live PostgreSQL theories cover the 17 noted tables; a sparse read on an old server with one late row gets no note, and on a two-day-old server gets one; the status words are pinned from the tool sources and by behaviour; 11 page scenarios through the Node harness; the source pin is scoped to the listed reads.

Part of #4966

* Web line charts get a menu: copy/save image, export CSV, show data source, reset zoom (#5166)

Part of #4843

## What users see
Every web line chart has a small "⋯" button at its top right, and right-click on the chart opens the same menu. Items, in the Viewer's order and wording:
- **Copy Image**: draws the chart to a PNG and puts it on the clipboard (secure page only; otherwise a plain message says why).
- **Save Image As...**: downloads the PNG.
- **Reset zoom**: only while the chart is zoomed (#5092).
- **Export Data to CSV...**: one row per time and series (`DateTime (UTC),Series,Value`). It exports every loaded point, also while the chart is zoomed, the same as the Viewer. Times are UTC in three columns; the Viewer writes the display time zone plus a UTC-offset column.
- **Show Data Source**: the read name and its parameters, shown under the chart. Hidden when the chart has no read to name.

The menu is keyboard reachable: Tab to the button, Enter to open, Arrow keys to move, Escape or Tab to close. A right-click menu opens at the pointer and stays inside the chart. The open menu and the Show Data Source panel stay open across the 60 s refresh (kept at module scope per chart and page scope, for 90 s).

## What changed
- `charts.js`: `renderLineChart` attaches the menu, so every chart gets it. New optional spec fields `title`, `source` ({ read, params }), `zoomed`, `onResetZoom`. `zoomableLineChart` supplies the zoom pair. No new exports are imported by any other module.
- `panels.js`: the panel chart passes its title and its read name and params (two lines).
- `css/app.css`: menu rules beside the chart tooltip rules.
- CSV reuses `grid-tools.js` (`toCsv`, `csvFileName`, `downloadCsv`), loaded on first use. PNG: the SVG is cloned with computed styles written inline, drawn on a canvas, no external fetches.
- Copy Image hands the clipboard a promise for the PNG (Safari needs the write started inside the click); a failed CSV module load shows "Export failed".
- Tests: `WebChartMenuBehaviourTests` + `web-chart-menu-harness.mjs`.

## Not changed
- **Open in New Window** is out of scope.
- Server-tab and FinOps charts that call `zoomableLineChart` directly pass no `source`, so they show no Show Data Source until a caller passes it.
- Bar, pie and scatter charts have no menu.

## Verified at the head
- New harness on dev's `charts.js`: RED (the menu does not exist; the harness throws). On this head: 8/8 pass (adds right-click placement `right:auto` + clamped left/top, zoomed CSV = all 11 points, Tab closes, menu and source survive a rebuild).
- Every `.mjs`-harness class (incl. `WebModuleLoadTests` 7/7, `WebChartZoomBehaviourTests` 10/10, `GridToolsBehaviourTests` 9/9): 0 failures. `ChartIntegerTicksPinTests` needed the spec destructure kept as is; passes 3/3.
- This run: the targeted classes, every `.mjs` class and the guard family, 113 classes, 1364 tests, 1 failure.
- Guard family (all classes ending Census/Ratchet/Adoption/Discipline/Guard/Scrub/Hygiene/Pin/Convention Tests, plus `DocCommentHygieneTests`): only `LivePostgresCollectionHygieneTests` fails, and it is on `dev-local-failing-tests.txt`.
- Full Darling.Tests on this Mac: 22847 total, 478 failed, 1920 skipped. Every failing class is on the Mac-local list (parameterized rows and `ViewerChartContextMenuTests`, WPF); none touch charts.js.
- Not run in a real browser: the canvas, clipboard and menu placement are untested there (the node harness has no canvas or layout); worth one manual click-through.
- scrubcheck on this file and the diff: exit 0.

## CHANGELOG entry
SECTION: Added
ENTRY: - **Web line charts have a menu** ([#5166]) - Each line chart has a "⋯" button (and right-click) with Copy Image, Save Image As, Export Data to CSV, Show Data Source (where the chart knows its read) and Reset zoom (while zoomed).
REF: [#5166]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5166

* get_server_trend: latch, spinlock, session stats and collector duration trends (#5170)

Part of #4843

## What users see change
`get_server_trend` (and its `/api/read/get_server_trend` route) takes four more `metric` values:

| metric | a point holds |
|---|---|
| `latch` | `wait_time_ms_per_second`, one series per latch class |
| `spinlock` | `collisions_per_second`, one series per spinlock |
| `session_stats` | the server-wide session counts (`total_sessions`, `running_sessions`, `sleeping_sessions`, `background_sessions`, `dormant_sessions`, `idle_sessions_over_30min`, `sessions_waiting_for_memory`, `databases_with_connections`), averaged per bucket, plus the top application and host of the newest collection in the bucket |
| `collector_duration` | `max_duration_ms`, `avg_duration_ms`, `run_count` of successful runs, one series per collector |

- **New `names` parameter** for `latch`, `spinlock` and `collector_duration`: comma-separated, exact case, up to 10, default the top 5 (most wait time, most collisions, longest run). Names with no samples come back in `missing_names`. If none of the named ones has a sample, the answer is `empty` with a distinct note, `missing_names` and `top_names`. `names` on any other metric is refused, and `clerk_types` on these three is refused.
- Latch and spinlock rates use the viewer's rule: summed delta over the summed stored interval, and a restart's unknowable (interval 0) sample is left out rather than drawn as 0.
- Same envelope as before: house window and bucket parameters, omitted nulls, `discontinuities[]`, and `empty` / `unavailable` / `not_collected`. `session_stats` is sized as four single-field points so its default 168 h answer stays under 32 KB.

## What changed
- `ServerTrendSql.cs`: `SessionSummary` (moved verbatim from the viewer), `LatchWaits(n)`, `SpinlockCollisions(n)`, `CollectorDurations(n)`, the three top-N reads and four "ever collected" probes. The viewer's `SessionStatsSql` now points at `ServerTrendSql.SessionSummary`; I diffed the old and new text and they are identical (71 lines).
- Latch, spinlock and collector duration: the viewer's reads have a different shape (per-row rates for a fixed top 5; per-collector buckets with no name filter), so the bucketed, name-filtered reads are written new in the house style. The viewer's SQL is untouched. One new read per metric, no new Storage reader class.
- `DarlingMcpServerTrendTools.cs`: the four metrics, `names`, a shared series path for clerks, latch, spinlock and collectors, notes and `Fields` per metric. `DarlingWebEndpoints.cs`: `names` in the catalog row and dispatch. `GetServerTrend`'s internal overload takes `names` before the budget so the cancellation token stays last (CA1068).
- Budget txt, ceiling, and tests.

## What's not changed
No `wwwroot`, no `DarlingManagedRoles.cs`, no migration, no change to the four earlier metrics' output. Left for a later change: `tempdb_file_io`, `tempdb_size`, `file_io_throughput`.

## Budget
- `tools/list`: 186,010 -> 186,158 bytes (+148); `TotalCeilingBytes` set to 186,158.
- Per-file txt: tool head 279 -> 331 (text grew after the guide marker mostly), `metric` param 61 -> 44 (the list moved into the tool description), new `names` param 70.
- Largest default call at 168 h, on a week of one-minute collections (8 latch classes, 8 spinlocks, 12 collectors): latch 11,127 B, spinlock 10,831 B, session_stats 17,944 B, collector_duration 15,627 B. All under the 32 KB target. The four earlier metrics are unchanged (largest 17,555 B, cpu_scheduler). Widest explicit width: latch 106,172 B at 34 min, spinlock 103,197 B at 34 min, session_stats 149,898 B at 27 min, collector_duration 149,991 B at 34 min, all under the 320 KB cap.

## Verified at the head
- Build: Darling.Tests Release, 0 errors, no CA/CS/IDE warnings after the CA1068 fix.
- `DarlingMcpServerTrendToolsTests`: 13/13 (parameters including `names`, eight-metric list in the description, refusals, every field a note names is emitted, `missing_names` in the description, name/width binding in the SQL, viewer and tool share the session SQL text).
- `DarlingMcpServerTrendToolsLiveTests`: 2/2 on a throwaway TimescaleDB 2.30.1 / PG18 container (host port 55615, removed). The new arm covers planted values for all four metrics (100 ms/s latch, 20/s spinlock, 100 then 120 sessions with the newest collection's application, 9000 then 9100 ms slowest collector), the restart-marker sample dropped as a point, a failed run not counted, `missing_names`, the none-found empty note, `unavailable` before any collection, `empty` for a quiet window, and the 168 h sizes above. Run 4 times; one run failed once on `bucket_minutes` auto sizing (fixed by pinning 1 minute in the planted arms) and is stable since.
- `WebReadCancellationPinTests` 139/139 (its `metric=total_waits` query string still works; the new parameter is optional).
- `McpToolsListBudgetTests` 5/5, `McpToolGuideTests` 10/10, `McpToolGuideHeadsDataTests` 3/3, `BaselineDiscontinuityRenderTests` 3/3, `EngineCapabilityReadWiringTests` 6/6 (the capability question names `latch_stats`, `spinlock_stats` and `session_stats`; `collector_duration` asks no collector because the log is not a gated collector).
- Guard family: all 79 Darling.Tests classes ending in Census, Ratchet, Adoption, Discipline, Guard, Scrub, Hygiene, Pin or Convention + Tests, plus DocCommentHygiene, RepoFileAdoption, TsqlConventionGuard, FleetIdentifierScrub, QueryStoreSliceTieBreakSource, the three command-timeout classes and the viewer session/duration/latch classes: all pass except `LivePostgresCollectionHygieneTests.EveryClassUsingTheSharedStore_IsSerializedOrDocumentsWhyNot` (loads WPF types; on the Mac-local list) and `ViewerCollectorDurationChartTests.TheChart_IsFedItsOwnRead_NotTheGridsPage` (PresentationFramework; on the list).
- Full Darling.Tests once: 22,847 run, 552 failed, 247 skipped. All failing names (430 distinct after dropping theory-row arguments) are on `dev-local-failing-tests.txt`; none is new.
- RED on dev: `Metrics` has four entries, `names` and `SeriesOf`, `ServerTrendSql.SessionSummary`/`LatchWaits`/`SpinlockCollisions`/`CollectorDurations` do not exist, so the new tests fail to compile there (RED by compile failure). Before the sizing fix, session_stats was 68,530 B at 168 h, over the 32 KB target.
- Not run: Lite.Tests (never run on this Mac; the change touches no Lite file).
- Finds: 0 substantial, 1 hygiene fixed in-lane (session_stats default width sizing), 0 polish.
- Follow-up at head 96df107d5 (dev merged in): a latch or spinlock row with a known interval but a NULL delta no longer adds its seconds to the rate (the Viewer drops that row too); `KindMetric` is gone and `ReadTopNamesAsync` always takes the metric. New live test `TheRateRules_SessionNewestCollection_AndCollectorNames_HoldAgainstTheStore` covers the NULL-delta row (RED before the fix: 50 ms/s instead of 100), a latch row with no stored interval (previous-collection gap), two session collections in one bucket (the newest one's attribution) and `names` for collector_duration. `DarlingMcpServerTrendToolsTests` 13/13, live class 3/3 on a throwaway TimescaleDB container (removed); `McpToolsListBudgetTests` 5/5 with tools/list measured at 185,766 bytes, under the unchanged 186,158 ceiling; `McpToolGuideTests`, `WebReadCancellationPinTests` and the 80-class guard family green except `LivePostgresCollectionHygieneTests` (loads PresentationCore, Mac-only).

## CHANGELOG entry
SECTION: Added
ENTRY: - **More server trends over MCP: latch, spinlock, session stats and collector duration** ([#5170]) - `get_server_trend` now also returns the latch wait and spinlock collision rates per name, the server-wide session counts, and the per-collector run duration trend the desktop charts draw, with a `names` filter for the per-name ones.
REF: [#5170]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5170

* Web Top Queries: the desktop grid's 45 columns, grouped, with a detail=full read for the fields the page lacked (#5169)

Part of #4843

## What users see
The server page's Top Queries grid (both the CPU tab panel and the composite panel) now has the desktop Top Queries columns in the desktop's order, 45 in all (the desktop's Query Plan download column is the separate plan work). The core set shows by default: Database, Query, Module, Execs, Total/Avg CPU, Total/Avg Duration, Total/Avg Reads, Total Spills, Query Hash. Six toggles under the "Columns:" strip turn on the rest: Times (Last Execution, Creation Time, Peak CPU ms/s, Plan Gen, Total CLR), I/O and rows, Extremes, Memory grants, Parallelism, Hashes and handles. Sort, Copy and CSV work on the wider grid; Copy and CSV carry every column. On the hourly tier the fields that tier lacks print a dash.

Query sits right of Database, not behind the two timestamps as on the desktop, because a pinned rule keeps query text next to the anchor. Module at index 2 is a choice of this grid, not a rule the pin states. Last Execution and Creation Time print in the browser's local time.

## What changed
- **Page:** `TOP_QUERY_COLUMNS` (45 keys) plus `TOP_QUERY_GROUPS`, passed to both Top Queries grids. Both ask for `detail: "full"`.
- **Read:** `get_top_queries_by_cpu` gains an optional last parameter `detail` (`summary` default, `full`). `full` adds last_execution_time and creation_time (UTC instants written with the "o" format; the read converts them from the monitored server's clock per row, as every other server-local column on the MCP surface is, and the frame census carries a row for them),  min/max physical reads, rows, grant / used grant / ideal grant KB, spills, reserved and used threads, total_clr_ms, plan_generation_num and worker_time_per_second. Fields the store holds no value for are omitted, not written as null. The raw tier only: an hourly-tier read adds nothing, and `precision_note` now says so when `detail=full` was asked and the hourly tier answered. Both aggregates (`TopQueriesSql`, `TopQueriesByHostObjectSql`) select the extra columns and the reader maps them into a `TopQueryDetail` record. The HTTP catalog and dispatch pass `detail`.
- **Size, measured live on a throwaway TimescaleDB, 168 h, top 20:** default call 12,980 bytes, `detail=full` 14,027 bytes (22 groups, only one with the detail columns, short text). A row with every detail field adds 294 bytes, so 20 full rows add about 5.9 KB; the default call is unchanged, and full stays under the 32 KB target unless query text is near its 2,000-char cap on most rows. Neither form has a byte bound: the summary call has no page cut or `truncated` flag for size (query_text is capped at 2,000 characters per row and nothing else bounds the payload), so `full` follows the same rule and adds about 6 KB at 20 rows. A size bound for both forms (a shorter text preview) is a separate follow-up that would change the summary call; it is not part of this PR. Because the default call is untouched, no opt-in was needed beyond `detail`.
- **tools/list:** new line `param get_top_queries_by_cpu.detail 122` in `McpToolsListBudget/DarlingMcpDataTools.txt`. `TotalCeilingBytes` (186,010) did not need to grow (the budget test passes).
- Tests: `DarlingCustomViewsTests` param keys and `WebServerPageRangeTests` read URL now include `detail`.

## Not changed
- `PG_TOP_QUERY_COLUMNS`: a different engine; the PG read does not return these fields.
- Lite and the desktop grid are untouched.
- The Plan column (#5156) is on dev and merged into this branch: both grids pass `[...TOP_QUERY_COLUMNS, planColumn(server)]` plus the groups.
- Peak CPU ms/s is the desktop's `WorkerTimePerSecond`: the max per-collection rate in the window.

## Verified at the head
- New `WebTopQueriesColumnsTests` 8/8: column keys against the read's projection parsed from source, desktop order, group layout (all six groups used, none on by default, core set ungrouped), both grids pass groups and `detail: "full"`, `detail` is an appended optional, both aggregates select the detail columns in the reader's order.
- RED on dev: all 8 fail (the new class run in a worktree of origin/dev).
- New live `TopQueriesDetailLiveTests` 1/1 on a throwaway TimescaleDB container (removed): reader detail values, MCP summary omits them, full carries them and omits nulls, bad `detail` refused, both sizes under 32 KB.
- Also green: `ViewerGridPayloadColumnOrderPinTests` 30/30, `McpToolsListBudgetTests` 5/5, `ServerTabColumnGroupsPinTests` 2/2, `WebServerPageRangeTests` 11/11.
- Full Darling.Tests on the Mac: 22,911 run. After fixing the three failures this PR caused (query-text order, budget txt line, range-test URL), the remaining failures are the Mac-local set (DarlingInstallLocation, Viewer WPF classes, etc.). Five live classes outside the dev-local list failed in the full run and also fail on origin/dev on my rig (TopCpuQueriesTextLive, FinOpsOptimizationTieBreak, StoreMetricsLatestSkipScanLive, ViewerFinOpsIntervalHonestLive, FinOpsOptimizationViewLive); they did not involve this change.
- The guard family and every `.mjs`-harness class were included in that full run.
- At the new head (dev merged in): `WebTopQueriesColumnsTests` 8/8, `TopQueriesDetailLiveTests` 1/1 (non-zero server offset: stored local times come back as UTC; host_object call; a 5-billion physical-reads extreme), `TopQueriesHourlyRoutingLiveTests` 7/7 (a `detail=full` call answered from the hourly tier carries the note and no detail field), `ConsumedTimestampFrameDisciplineTests` 17/17, `ViewerGridPayloadColumnOrderPinTests` 30/30, `McpPayloadContractCensusTests` 68/68, `McpToolsListBudgetTests` 5/5, `WebPlanViewerPageTests` 6/6, the 108-class guard family plus `.mjs`-harness set clean on the Mac except `LivePostgresCollectionHygieneTests` (loads WPF types; dev-local list). Full suite: 22,975 run, 568 failed, every one on the dev-local list or one of the five live classes named above.
- scrubcheck on the diff exits 0.

## CHANGELOG entry
SECTION: Added
ENTRY: - **Web Top Queries shows the desktop grid's columns** ([#5169]) - the server page's Top Queries grid now carries the desktop's 45 columns in six toggle groups (times, I/O and rows, extremes, memory grants, parallelism, hashes and handles), with the core columns on by default. `get_top_queries_by_cpu` gains an optional `detail=full` that adds the grant, thread, extreme, time, CLR and plan-generation fields.
REF: [#5169]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5169

* get_query_store_top answers long windows from the daily summary, and says the answer is approximate (#5094) (#5168)

get_query_store_top answers long windows from the daily summary, and says the answer is approximate (#5094)

Fixes #5094. The table path used to aggregate every interval row in its window on every call. On a large store a 168-hour read took 6-23 s cold and 2.7 s warm, almost all of it heap reads.

- When the interval table serves the read, the whole UTC days inside the window come from collect.query_store_top_daily (built by #5125), and only the partial days at the edges come from collect.query_store_interval_wide. The read takes the longest contiguous run of built days in the gate's own REPEATABLE READ, READ ONLY snapshot, with ties going to the latest run. With no built day in the window, today's statement runs unchanged (same text, same parameters).
- The summary is approximate by design (Erik's call on #5094): a write into a day after it was built is missed, a row that moves into the next day can count twice until the day's second pass, and a monitored-server clock more than an hour ahead drops rows. In the clean case the answer equals today's exactly. Answers that used the summary carry approximate: true, an approximation_note and summary_days, including the two empty answers, and the web Query Store grid shows the note. The viewer's own Query Store read stays exact.
- Read failures keep #5136's accounting: a fault after the gate chose the table, the built-days read included, is recorded as a raw fallback and answers from raw.
- Tests: QueryStoreTopDailyReadLiveTests (25 facts: equality against today's read on fixed past anchors, edge bounds, run selection, the documented double count, the empty-answer flags and the fault fallback), plus the payload-contract census and the web tab pins.

* query_stats can fetch plan XML in a second target query for only the rows that need it (inert, #5158) (#5173)

query_stats can fetch plan XML in a second target query for only the rows that need it (inert, #5158)

Part of #5158, the first of three PRs. Nothing sets the new switch yet, so no collector's SQL or behavior changes.

- With plan capture on, every query_stats run renders sys.dm_exec_text_query_plan for all of its ~200 rows and ships the XML, although almost all of those plans are already stored. #5158 measured 53.75 GB/h rendered and 28.83 GB/h shipped on one large store, against 1.14 GB/h of distinct plans.
- CollectorContext.DeferPlanXmlFetch (default false) leaves the plan fragments out of BuildQuery and skips the plan ordinals in ReadAsync, through one shared predicate.
- QueryStatsCollector.BuildPlanFetchQuery builds one statement for a list of (plan_handle, start offset, end offset) keys: an inline VALUES list of host-rendered 0x literals, OUTER APPLY sys.dm_exec_text_query_plan, the same size-cap CASE and byte column as the inline fragment, and OPTION(RECOMPILE). It throws when capture is off, the list is empty, or a handle is not 1 to 64 bytes. ReadPlanFetchAsync maps each ordinal to its plan and size.
- Row.KnownPlanDigest and ICollectorRowWriter.PayloadOrDigest (default: Value(content)) let a later writer store a known digest in place of the plan text. WritePayload calls it, and with the default every row writes exactly as before.
- Tests: QueryStatsPlanFetchTests (18) and QueryStatsSqlGoldenTests (4 SHA-256 pins of the full BuildQuery text for Standard and Azure SQL DB, capture on and off, computed on dev and unchanged here).

* Web CPU and Memory tabs chart the CPU scheduler, memory clerk and plan cache trends (#5161)

Part of #4843

**Stacked on #5162 (Perfmon multi-select): this branch contains that PR's commits and uses its new `multi-picker.js` options, so it must land after #5162.**

## What users see
- **CPU tab:** a "CPU Scheduler Trend" chart (runnable, blocked and queued task counts; axis "tasks"), as the desktop's CPU Scheduler chart draws them.
- **Memory tab:** a "Memory Clerks Trend" chart (MB) with a clerk selector, and a "Plan Cache Trend" chart (single-use and multi-use MB).
- The clerk selector is the shared multi-select (#5119), capped at the read's 10 clerk types. Its options are the clerks of the latest snapshot, heaviest first; the heaviest five are checked the first time, as on the desktop. The checked set and search text survive the 60 s rebuild and are kept per server.
- Clerk types the read found no samples for are named in a notice. Baseline discontinuities show through the existing `discontinuityNotes` helper. An empty answer shows the read's own message.
- All three charts use the zoomable chart path (#5092).

## What changed
- `server-tabs.js`, CPU and Memory tab regions only: `serverTrendPanel` (cpu_scheduler, plan_cache), `memoryClerksTrendPanel` and `drawClerkTrends`, with no new import beyond `multi-picker.js`. The clerk picker passes `noun: "clerk"`, `defaultsLabel: "Top clerks"` and `defaults` = the heaviest five, so its button, hint and empty-search text say clerk, and the button checks five like the desktop's Top Clerks. When no named clerk has samples, the missing list and the heaviest clerks come from the empty answer's `hints` and show as text.
- Reads go through `readToolWithinKeptHistory` with `{ server, metric, hours }` (and `clerk_types` for clerks), so the page range, the custom range's `as_of` (#5108) and the kept-history narrowing apply like the sibling panels.
- `Lite.Tests/BaselineDiscontinuityRenderCensusTests`: the census counted four `discontinuityNotes(...)` uses; it now counts six and names the two new functions (`serverTrendPanel`, `drawClerkTrends`). After merging #5162 the census counts six and lists `drawWaitTrends`, `drawPerfmonTrends`, `drawQueryTrend`, `fileIoPanel`, `serverTrendPanel`, `drawClerkTrends`; I checked each boundary by hand (one use per function, 6 in total). Lite.Tests builds here but cannot run; I traced the regex `discontinuityNotes\((trend\.data|res\.data)\)` by hand: the new uses are `discontinuityNotes(res.data)` and `discontinuityNotes(trend.data)`, so 4 + 2 = 6.
- Tests: `WebServerTrendsPageTests` (read names, params, placement), `WebServerTrendsBehaviourTests` with `web-server-trends-harness.mjs` (series and unit, selector defaults, state across a rebuild and per server, `missing_clerk_types`, empty answers, no trend read when no clerk snapshot).

## Not changed
The panel context carries no abort signal, so these reads are not aborted by the page's 60 s rebuild (they mount into a detached node, as the Wait Stats and Perfmon panels do); a newer clerk draw does abort the older one. The clerk options come from the latest snapshot, not every clerk seen in the window.

No other `server-tabs.js` region, `panels.js`, `util.js`, `charts.js`, `app.js` or CSS. No tool or store change.

## Verified at the head
- RED on dev: `WebServerTrendsPageTests` 5/5 fail, `WebServerTrendsBehaviourTests` 7/7 fail (dev's `server-tabs.js` with the new tests).
- GREEN: `WebServerTrendsPageTests` 5/5, `WebServerTrendsBehaviourTests` 9/9 (new: picker wording and the five-clerk defaults button; hints lists on the all-missing answer), `WebModuleLoadTests` 7/7, Perfmon and Wait Stats classes unchanged (15, 10, 14).
- Every `.mjs`-harness class and the guard family (104 classes, 1,300 tests): only `LivePostgresCollectionHygieneTests` fails, which is on the Mac-local list.
- Lite.Tests: build only; not run.
- scrubcheck on the diff exits 0.

## CHANGELOG entry
SECTION: Added
ENTRY: - **CPU scheduler, memory clerk and plan cache trend charts on the web server page** ([#5161]) - The CPU tab charts runnable, blocked and queued tasks; the Memory tab charts memory clerks (with a clerk selector) and plan cache size, matching the desktop.
REF: [#5161]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5161

* Web grids: per-column filters that survive the repaint and shape Copy and CSV (#5183)


Part of #4843

## What users see change
- Every data-grid column header has a small ▾ button. It opens a text box: rows whose rendered text in that column contains the text (any case) stay, the rest hide. A list-valued cell matches on the text it shows.
- Filters on several columns combine with AND.
- A filtered column's button is highlighted. A strip above the table lists each active filter with a × to clear it, a "Clear all filters" button, and "Showing N of M rows".
- Escape closes the box and returns focus to the header button. Focus leaving the box closes it.
- Copy all and Export CSV carry the filtered rows, in the current sort order, with every column (hidden and grouped-away ones too), as the desktop export does.

## What changed
- `panels.js`: `gridFilter` (called from `vizTable`), module-level filter state, and `gridTools` now reads the rows not hidden by a filter. `reapplyGridSort` also re-applies the filter for grids reconciled in place.
- `css/app.css`: the filter rules, beside the existing grid-tools and column-picker rules.
- State keyed like the sort state (`gridSortKey`: route + table identity + column set): filter text per column, and which box is open (plus whether it had focus). The 60 s repaint keeps both; another server's route starts unfiltered.
- Opt-outs: `filter: false` on the descriptor or a column; `copy: false` control columns get no button.

## Not changed
Column groups, `copy: false`, sorting, drill state, `grid-tools.js` helpers (reused as they are). Copy cell / Copy row act on the picked cell wherever it is.

## Verified at the head
- New `GridFilterBehaviourTests` (node harness `web-grid-filter-harness.mjs`): 8/8. RED on dev: 7 of 8 throw (no filter button exists) and the opt-out test fails on its per-column expectation.
- `GridToolsBehaviourTests` 9/9, `ColumnGroupsBehaviourTests` 6/6, `GridSortBehaviourTests` 10/10, `WebModuleLoadTests` 7/7, `ServerPageTabsTests` 19/19; all `.mjs`-harness classes green.
- Guard family (93 classes): all green. `PlanForceActionDetailCensus` and `SqlTextPin` are not test classes (0 tests).
- Classes naming changed files/symbols (FinOpsTab*PageTests, Web*Tests, DarlingWebAssetsTests and others): green. `RawWindowFloorViewerPortTests` fails 3 on this Mac; it is a Viewer/WPF class and unrelated to this diff.
- `node --check` on `panels.js`: clean. Not run: `LivePostgresCollectionHygieneTests` (WPF load on this Mac), Lite.Tests (no Lite file names a changed symbol).

## CHANGELOG entry
SECTION: Added
ENTRY: - **Per-column filters on the web grids** ([#5183]) - each column header opens a filter box that matches the text you see; filters combine, show "Showing N of M rows", survive the 60 s refresh, and Copy all and Export CSV carry only the filtered rows.
REF: [#5183]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5183

* MCP: Darling's config change, collection log, plan correction and memory pressure tools say where their data starts (part of #4966) (#5181)

Six Darling MCP tools now say where their data starts, with the contract Lite's twins and Darling's other window-floor tools use: get_server_config_changes, get_database_config_changes, get_trace_flag_changes, get_collection_log (one server), get_plan_corrections and get_memory_pressure_events write effective_start (UTC, with a Z), window_truncated and truncation_note right after hours_back on a data answer, and the same keys under hints on an empty answer. not_collected and unavailable answers are unchanged, the fleet form of get_collection_log carries no notice, and a failed coverage probe costs the notice, never the rows.

Coverage sources: the config tools probe their own snapshot tables (server_config, database_config, trace_flags), with the viewer's and the web's rule that a snapshot from before the window is not coverage and a logged run in the window is; the config collectors run at every connect and every 24 hours, and every run is logged. get_collection_log probes the collection log and ignores the collector, status and duration filters. get_plan_corrections probes plan_correction; an answer of only automatic-tuning rows is a data answer, and its sentence no longer calls it empty. get_memory_pressure_events probes the events on sample_time, edged at the purge cutoff. The config tools and memory pressure decide not_collected before they probe.

The web page: WebDataStartNote.AddAsync now strips a listed read's own three keys on every answer it returns, not only when it adds its note, so a covered or short window no longer hands the page the tool's UTC verdict. The memory pressure page read is not on the web's list; its empty grid shows the tool's own note when the window was cut.

Tests: the config tools' live class (a notice when collection starts inside the window, a quiet start, an empty answer with hints, the key order, a short window, the pre-window snapshot with and without a run, a failed probe); the collection log, plan correction and memory pressure live classes (the same cases, plus capped pages, a tuning-only answer that never calls itself empty, and an event sampled before the first collection); a not_collected answer that starts no probe; the web live tests assert the tool's keys are absent; census and EffectiveStartUtcTests rosters. Response budgets at the default call stay under 32,768 bytes.

Part of #4966

* LITE AND DESKTOP: the blocking charts' data-start note follows the blocked process report collector when the chart draws its reports (part of #5098) (#5159)

In Lite and the desktop viewer, the Blocking trend, the blocking duration Stats pair and the Overview blocking chart draw blocked process reports from the Extended Events session whenever the window holds any, and the DMV blocking snapshots only otherwise. Their "Showing since" note probed both collectors' coverage, so when the DMV collector covered the whole window but the blocked process report collector only started collecting midway, the note said covered while the chart drew zeros before that collector's first run. That stretch now gets the note.

Each app gains a cheap existence check with the read's own predicate (server, event-time window, database filter, and the viewer's collection-time floor). When the read took the reports, the note's coverage is that collector's alone. In Lite that is GetQueryWindowFloorAsync with includeAlsoCovered: false. In the desktop viewer it is the collector's own runs in the collection log: a run before the window means covered, otherwise the earlier of its first run in the window and the earliest report in the window (a first collection can store earlier reports); the shared table and run probes both start at the server's registration and so could not see a collector turned on later. When the window starts before the collection log's retention horizon, the table rule answers, as the Blocked Process Reports grid's does. The DMV branch, the deadlock notes and the grid are unchanged.

Not covered: a server whose collector ran throughout but whose blocked process threshold was set midway (the collector logs successful runs with no rows, which its runs cannot tell from quiet hours), and a collector that stopped and started again inside the window; #5098 stays open for the first.

Tests: the desktop viewer's coverage class gains the midway case (it failed on the earlier code), a report older than the first run, runs before the window, a window before the log's horizon, no reports, and reports covering the window; Lite's blocking-chart and Overview classes gain the same cases plus a database-filter pin; the call-site pins in both apps follow the new calls.

Part of #5098

* Record slow reads with their arguments, source and statement times (#5097) (#5163)

Slow reads can now be read back with what they were asked for and where the time went. The MCP tools, the web read routes and the Custom Views panels each record a read that ran 5 seconds or more, or ended in a timeout, an error or a limit. The new `collect.slow_reads` table (migration V162) stores:
- the surface, the route and the outcome;
- the total time and, for reads that count them, the rows returned;
- the server and the time window asked for;
- the source chosen (raw, the interval table, hourly edges) and the gate's reason;
- each statement's time, up to 50 statements.

**Statement times** come from one process-wide Npgsql activity listener. It samples only inside a recorded read, so collector and background commands create no activity. A statement's time runs until its reader closes, so it includes row streaming. Only the product's SQL text feeds a statement's label, never the connection details.

**Arguments** keep only short scalar values:
- numbers and booleans;
- strings of 128 characters or fewer that look like neither JSON nor a credential.

Anything else is stored as `[omitted]`:
- JSON payload strings such as `add_servers`' `servers_json`;
- URL user-and-password parts;
- `Password=` or token text.

Keys with secret-like names are redacted at any depth. Web query keys are bounded, and unknown Custom Views sources are recorded as `compose:unknown`.

**Storage.** Recording is off the request path. A bounded queue feeds one background writer, which keeps 30 days and at most 10,000 rows, purging at most once a minute.

**Reading it back.** `get_slow_reads` returns, within the shared 32 KB response budget:
- a per-route summary, then the reads newest first;
- times in UTC;
- the five slowest statements by default, or all of them with `full_detail`.

The web dashboard reads the same answer through `/api/read/get_slow_reads`. The tools/list budget moves to 187,103 bytes and 172 tools, a figure that also covers #5186's new `get_top_procedures_by_cpu.detail` parameter, which lands with it.

**Tests.**
- The rung, the recording rule, the queue and the purge throttle.
- The argument scrub: a planted password in `servers_json`, nested API keys, URL credentials, BOM-prefixed and malformed JSON, and web tokens.
- An acceptance test that slows a `get_query_store_top` call and reads it back with `get_slow_reads`, with its `hours_back`, `top`, source, row count and statement times.

Part of #5097

* Web Top Procedures: the desktop grid's columns, grouped, with a detail=full read for the fields the page lacked (#5186)


Part of #4843

## What users see
The server page's Top Procedures grid (CPU tab and the composite panel) now has the desktop grid's columns in the desktop's order, 28 in all. The core set shows by default: Database, Procedure, Type, Execs, Total/Avg CPU, Total/Avg Duration, Total/Avg Reads, Total Spills. Three toggles under "Columns:" turn on the rest: Times (Last Execution, Cached Time), I/O and spills (Total Writes, Physical Reads, Avg Spills) and Extremes (min/max CPU, duration, reads, physical reads, writes, spills). Copy and CSV carry every column. Last Execution and Cached Time print in the browser's local time. On the hourly tier the fields that tier lacks print a dash.

## What changed
- **Page:** `TOP_PROC_COLUMNS` (28 keys) plus `TOP_PROC_GROUPS`; both grids ask for `detail: "full"`.
- **Read:** `get_top_procedures_by_cpu` gains an optional last parameter `detail` (`summary` default, `full`). `full` adds last_execution_time and cached_time (UTC instants written with "o", converted per row from the monitored server's clock through `DarlingServerClockReader`), avg_spills, and min/max logical reads, physical reads, logical writes and spills. Fields the store holds no value for are omitted. Raw tier only; an hourly answer to `detail=full` says so in `precision_note`. `TopProceduresSql` selects the ten extra columns; the reader maps them into a `TopProcedureDetail` record. HTTP catalog and dispatch pass `detail`.
- **tools/list:** one new line, `param get_top_procedures_by_cpu.detail 122`. `McpToolsListBudgetTests` 5/5; `TotalCeilingBytes` unchanged (the tool description itself is untouched).
- **Frame census:** inventory row for `cached_time`, `last_execution_time` ambiguous-site count 2 to 3, plus a pin that the reader converts and the tool writes "o".
- Sibling pin `WebTopQueriesColumnsTests` scoped to the queries read (it counted every `detail: "full"`).

## Not changed
- Desktop Query Plan column (separate plan work); the desktop's Plan Handle column is also left out (the read already returns plan_handle as a field, but the page has no plan-handle column here; the Plan column work covers it).
- Hourly tier gains nothing; no new tool; Lite and desktop untouched.

## Verified at the head
- New `WebTopProceduresColumnsTests` 7/7; RED on dev: 7/7 fail.
- New live `TopProceduresDetailLiveTests` 1/1 on a throwaway TimescaleDB (removed): non-zero server offset converts both timestamps, 5-billion physical-reads extreme, nulls omitted, summary lean, bad `detail` refused.
- `TopProceduresHourlyRoutingLiveTests` 2/2 (extended: `detail=full` on the hourly tier carries the note and no detail field).
- `ConsumedTimestampFrameDisciplineTests` 17/17, `WebTopQueriesColumnsTests`, `McpToolsListBudgetTests` 5/5, `McpPayloadContractCensusTests` 68/68.
- 293 guard-family and symbol-named classes: 4,314 run, 12 failed; 11 are on the Mac-local list (WPF/Viewer classes), the 12th was the queries pin fixed above (now green).
- Lite classes naming the read (VM run): CpuAttributionTests, LiteMcpServerClockTests, McpReadToolBudgetTests, McpToolGuideHeads.* , McpToolsListBudgetTests, QueryStatExtremesTests, QueryStatsCpuRankingReaderTests, QueryWindowTruncationTests. None name a symbol this diff changed.
- `node --check` on server-tabs.js clean.

## CHANGELOG entry
SECTION: Added
ENTRY: - **Web Top Procedures shows the desktop grid's columns** ([#5186]) - the server page's Top Procedures grid now carries the desktop's columns in three toggle groups (times, I/O and spills, extremes), with the core columns on by default. `get_top_procedures_by_cpu` gains an optional `detail=full` that adds the times, average spills and read/write/spill extremes.
REF: [#5186]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5186

* MCP: Darling's blocking, deadlock and long query tools say where their data starts (part of #4966) (#5171)

Darling's get_blocking, get_deadlocks, get_deadlock_detail, get_blocked_process_xml and get_long_query_completions now say where their data starts, with the contract Lite's twins (#5127) and Darling's session tools (#5147) use: effective_start (UTC, with a Z), window_truncated and truncation_note right after hours_back on a data answer, the same keys under hints on an empty answer, not_collected and unavailable unchanged, and a coverage probe that fails costing the notice, never the rows.

These are event lists and a first collection can store events from before itself, so the floor on a data answer is the earlier of the coverage probe and the oldest event shown (DarlingMcpWindowNotice.ReadEventAsync). Coverage is DataWindowFloor's table rule, as in Darling's other coverage notes: the deadlock tools probe deadlocks, get_blocked_process_xml the blocked process reports alone (the XML exists only there), get_blocking both the reports and the DMV blocking snapshots with the earlier winning because its page merges both, and get_long_query_completions its own table. A collector turned on after the server was registered reads as covered from the registration. The web server page's Blocking, Deadlocks, Deadlock Graphs, Blocked Process Reports and Long Query Completions grids now show the tool's own note when its window was cut, as the Top Queries grids already do.

Tests: for each tool, a notice when collection starts inside the window, none on a quiet start, an event older than its collection names the event, hints on an empty answer, the key order, a short window that starts no probe, a capped page, and a failed probe that keeps the rows; the response budgets at the default call all stay inside 32,768 bytes; the census rosters and the shared-formatter pin cover both files.

Part of #4966

* Custom Views: the recent-window route resolves module names from the module map plus a small recent overlay (#4605) (#5179)

Custom Views: the recent-window route resolves module names from the module map plus a small recent overlay (#4605)

Part of #4605. A fleet-wide 24-hour Custom Views panel grouped by database and object still failed on a large store: on the hourly-rollup-with-raw-edges route, its module-name lookup ranked every procedure_stats row in the window (about 8.5M rows, roughly 1.6 GB of sort) and hit the 1 GB temp_file_limit.

- collect.module_map now carries a watermark in a new runtime table, collect.module_map_state (set up idempotently like module_map, no migration). The watermark is the newest procedure_stats collection_time the last refresh read, written in the same statement as the upsert. It only moves forward, and it is never written later than the store's own clock.
- A new hourly incremental refresh, the sixth hourly store-maintenance tenant, upserts the map from procedure_stats since the watermark less 10 minutes, at most 2 days back. It is failure-isolated. A refresh capped by that 2-day bound logs a Warning naming the span it did not re-read. The daily refresh advances the watermark the same way.
- On that route, for a plan with the module join, the panel reads the watermark inside its own REPEATABLE READ snapshot, under a savepoint. The lookup then ranks only the rows from max(window start, watermark less 10 minutes) to the window end, and takes every older handle from the map (last_seen within the window, same server scope, not already in the overlay). Every other route, and a missing or future watermark, compiles exactly as before. MCP's run_custom_view_panel runs through the same path.
- The answer matches the raw route's. The documented edges: a module renamed in the minute between the window end and now, or a handle seen only in that minute; a procedure_stats row committed more than 10 minutes after its collection_time, which waits for the next daily refresh; and a refresh stalled for more than 2 days.
- Tests: DarlingModuleMapRecentRefreshLiveTests (11), new compiler facts in DarlingComposeTests and ComposeParameterCoverageTests (bind sites 23 to 24), and ComposeHourlyRawEdgesModuleMapLiveTests (8), which compares the overlay route against the raw compile row for row across seven cases.

* PayloadDimensionLiveTests: the dimension GC test arranges its own coverage clamp (#5190)

`PayloadDimensionLiveTests.DimensionGc_PrunesOrphansButKeepsReferencedContent_WhileTheCoverageClampHoldsFacts` depended on test order. It needs the store's coverage clamp to hold the query_stats facts. But sibling tests in the class bring continuous aggregates up through `SatisfyRollupCoverageAsync`, and leave them standing in the shared store. Whenever one was still standing, query_stats read as covered, the clamp did not hold, and the test failed with "the clamp must be holding query_stats for this test to mean anything". Whether it passed depended on which siblings ran first.

The test now drops any standing continuous aggregates at the start of its body. Its `finally` recreates exactly the set that existed before, through `LiveStoreCleanup.RunAsync`. No assertion changed.

**Tests.**
- With a sibling's aggregates seeded first, the old body fails with that message and the new one passes.
- With 27 aggregates already standing, it passes.
- Alone, three runs in a row, it passes.
- The whole class passes, 18 of 18, with no aggregate left behind.

Test only.

* Web server page: column groups for Query Store Regressions, Long Query Completions and Plan Corrections (#5195)


Part of #4843

## What users see
The three grids #5099 widened now follow the wide-grid rules #5169 set for Top Queries. Each starts on its core columns, with a "Columns:" strip of toggles that are all off at first:
- **Query Store Regressions:** "CPU and reads" (baseline and recent CPU, baseline and recent reads) and "Executions and plans" (baseline and recent executions, baseline and recent plan counts).
- **Long Query Completions:** "I/O and rows" (logical and physical reads, writes, rows) and "Session" (SPID, app, login).
- **Plan Corrections:** "Plans" (regressed and last-good plan ids, forcing type, forced, force failure), "Plan metrics" (the four execution-count and CPU-average columns) and "Lifecycle" (Valid Since, Last Refresh, Executed By/At, Reverted By/At).

The choice survives the 60-second rebuild, per grid. Copy and CSV still export the filtered rows in the current sort order and carry hidden columns (#5183 is untouched).

## What changed
- `server-tabs.js`: `QUERY_STORE_REGRESSION_GROUPS`, `LONG_QUERY_GROUPS`, `PLAN_CORRECTION_GROUPS` (all `defaultGroups: []`), a `group:` on the wide columns, and the groups passed to the three grids. Only those three grid regions changed.
- Tests: `WebQueryStoreGridGroupsTests` plus `web-qs-grids-harness.mjs` (the shipped page run under Node with TZ pinned to America/New_York): default heads and toggles, toggling, survival of a rebuild, local-time rendering of every timestamp column, and a source pin that every group named by a column is declared and none is on by default.

## Timestamps
No read changes. All timestamps these grids show already arrive as UTC instants written with `"o"` and use `format: "time"`; the census already registers each as Utc: `query_store_stats.last_execution_time`, `long_query_completions.event_time`, and the four `plan_correction` lifecycle stamps (measured UTC against `collection_time`). So no census row was added and none is frame-ambiguous. The new test shows a UTC instant rendering as 12:30 PM under a UTC-5 browser.

## Not changed
- No read was widened. Plan Corrections' Script, Executable and Revertable columns stay absent: they come from stored columns `get_plan_corrections` does not return (separate work).
- `panels.js` and the other tab regions of `server-tabs.js` are untouched.

## Verified at the head
- New class `WebQueryStoreGridGroupsTests`: 8 of 8 pass. Against dev's `server-tabs.js`: 7 of 8 fail (RED); the eighth is the timestamp source pin, which holds on dev already.
- Guard family (75 classes), every `.mjs`-harness class (37), the named classes (`ServerTabGridColumnPinTests`, `WebTopQueriesColumnsTests`, `ConsumedTimestampFrameDisciplineTests`, `ViewerGridPayloadColumnOrderPinTests`, `WebModuleLoadTests`, `ServerPageTabsTests`, `GridFilterBehaviourTests`, `ColumnGroupsBehaviourTests`), `DocCommentHygieneTests`, `RepoFileAdoptionTests`, `TsqlConventionGuardTests`: 1388 run, 0 failed, 24 skipped. `LivePostgresCollectionHygieneTests` excluded (WPF load on Mac).
- Changed-symbol sweep: 65 Darling classes name server-tabs.js or the grid constants: 871 run, 13 failed, all in `ViewerDataStartBannerTests`, `RawWindowFloorViewerPortTests`, `ViewerQueryStoreRegressionsDataStartTests`, `ViewerCollectorDurationChartTests` (WPF/COM `PlatformNotSupportedException` on this Mac; none touch the web grids).
- Lite.Tests naming a changed file: `BaselineDiscontinuityRenderCensusTests` (names `wwwroot/js`) needs a VM run.
- `node --check` on server-tabs.js: clean. Release build 0 errors.

## CHANGELOG entry
SECTION: Changed
ENTRY: - **Web Query Store Regressions, Long Query Completions and Plan Corrections grids put their wide columns behind column toggles** ([#5195]) - The extra columns are off until you turn on their group, and the choice survives the page refresh.
REF: [#5195]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5195

* WEB: favourites, acknowledged alert badges, severity colours and a collapsible sidebar, kept per browser (#5196)


Part of #4843

## What users see change
- A star on every server (sidebar and fleet cards). Favourites sort ahead of everything else in the sidebar and the fleet grid, including inside tag groups and under every sort choice.
- An alert-count badge on each server (sidebar and fleet card), counting the last 24 hours of actionable alerts. Muted, resolution and dismissed rows do not count. Its colour is the critical or warning colour. Clicking it acknowledges: the badge hides in this browser until a newer alert arrives for that server (ack-until-worse, the desktop viewer's rule). Alert delivery is untouched.
- An "Alert severity colors" disclosure at the foot of the sidebar: one colour per severity (critical, warning, info) and a reset. The choices drive the badge colour and the alert history severity cells.
- A collapse button in the sidebar header; the collapsed state is remembered.

## Decision to overturn if wrong
Favourites and acknowledgements live in the browser's `localStorage`, not the store (the desktop keeps both in per-machine files too). Cost: they do not follow a user to another browser or machine, exactly as the desktop's do not follow them to another PC. This was decided while the owner was away; one line to reverse it. No table, migration, grant or route was added.

## What changed
- New `js/viewer-local.js` (no imports, no DOM, module-scope state, `localStorage` as the durable copy) and `js/viewer-local-ui.js` (controls, built with `el`/`mount`, text only).
- `app.js`: sidebar sort and badges, one block wiring the controls at start. `pages/fleet.js`: favourites-first wrapper on the existing comparators, star and badge on the card. `index.html`: collapse button and a settings host. `css/app.css`: rules beside the sidebar and `.status-cell` rules; `theme.css` and `panels.js` untouched.
- The badge reads `get_alert_history` once per 30 s, shared by every caller, so a poll tick costs the store one extra read (the same read the Alert History page makes); a failed read keeps the last counts.

## Storage keys (all versioned `v: 1`)
`darling.local.favorites.v1` (server ids), `darling.local.acks.v1` (server id to ack instant), `darling.local.severityColors.v1` (severity to `#rrggbb`), `darling.local.sidebar.v1` (collapsed flag). Only ids, instants, booleans and hex colours are stored; no names, strings from the server or secrets.

## Corrupt data
Absent, unparsable, non-object, other-version or wrong-typed values read as defaults without throwing. Inside a right-version value, invalid entries (non-positive or non-integer ids, bad instants) are dropped one by one. A colour is accepted only as `#rrggbb` (on read and on write), reaches CSS only as a `--sev-*` custom property via `style.setProperty`, and never enters markup; anything else falls back to the palette. Unreadable or unwritable storage (private mode, quota) leaves the page working for the session.

## Not changed
No server code, schema, route or grant. `theme.css` (tokens only) and `panels.js` are untouched.

## Verified at the head
- `ViewerLocalStateBehaviourTests`: 7 pass (node-run, stubbed storage): favourites order and rebuild, ack until newer alert and rebuild, corrupt/foreign/unreadable storage, non-hex colour fallback and hand-edited value, sidebar state, one shared read, wiring pins. RED on dev: the module does not exist there, so every scenario fails to load.
- Existing node harnesses that load `app.js` / `pages/fleet.js` (`web-scheduler`, `web-kept-history`) got stand-ins for the new imports; `WebModuleLoadTests`, `WebRenderSettleTests`, `WebPageRefreshIntervalBehaviourTests`, `WebServerPageRangeTests`, `WebRangeKeptHistoryBehaviourTests`: 51 pass.
- 127 guard-family, `.mjs`-harness and Web classes: 37 failed on the first run, all from the missing stand-ins above; fixed and re-run green. A further 23 pages-and-shell classes (`ServerPageTabsTests`, `GridFilterBehaviourTests`, `AlertNotebookRenderBehaviourTests`, `FleetPageAttentionFilterTests`, `StatusBarBehaviourTests`, and others): 387 total, 0 failed.
- Full Darling.Tests on this Mac: 23154 total, 479 failed. Every failing class is on the known Mac-local list (WPF load, PowerShell deploy tests); none names a file in this diff. CI decides them.
- `node --check` clean on `viewer-local.js`, `viewer-local-ui.js`, `app.js`, `fleet.js`; load harness clean. Release build `--no-incremental`: 0 warnings, 0 errors.
- Changed-symbol pins: grepped both test projects for `app.js`, `fleet.js`, `index.html`, `app.css`, `refreshSidebar`, `redrawCards`, `SORTS`, `server-item`, `sidebar`, and both edited harnesses. Darling classes that named one were run (above). Lite.Tests hits: `LiteSidebarDotRendersTheCardStatusTests`, `LiteFleetViewTests`, `LiteOverviewCardExplainsItselfTests` match on desktop/Lite source, not on these web files; list for a VM run to be safe.

## CHANGELOG entry
SECTION: Added
ENTRY: - **Web dashboard: favourites, acknowledgeable alert badges, severity colours and a collapsible sidebar** ([#5196]) - servers can be starred and sort first, each server shows an alert-count badge that a click acknowledges until a newer alert arrives, the three severity colours are choosable, and the sidebar can be collapsed. All of it is kept per browser in local storage.
REF: [#5196]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5196

* get_server_trend: tempdb file I/O, tempdb size and file I/O throughput trends (#5187)


Part of #4843

## What users see change
`get_server_trend` (and its `/api/read/get_server_trend` route) takes three more `metric` values:

| metric | a point holds | desktop chart it matches |
|---|---|---|
| `tempdb_file_io` | `avg_read_latency_ms`, `avg_write_latency_ms`, one series per tempdb file | tempdb tab, file I/O chart (`TempDbFileIoTrendSql`) |
| `tempdb_size` | `allocated_mb` (reserved + unallocated), one series | tempdb tab, allocated-size chart (`RenderTempDbSizeChart`) |
| `file_io_throughput` | `read_mb_per_sec`, `write_mb_per_sec`, one series per `database.file` | File I/O tab, Throughput sub-tab (`FileIoThroughputTrendSql`) |

- `names` now also applies to `tempdb_file_io` (tempdb file names) and `file_io_throughput` (`database.file` labels, e.g. `tempdb.tempdev`): exact match, at most 10, default the 5 with the most operations (tempdb_file_io) or bytes (file_io_throughput). Unmatched names come back in `missing_names`; if none matches the answer is `empty` with `hints.missing_names` and `hints.top_names`. `names` on `tempdb_size` is refused.
- Interval rule (same as the desktop and the latch/spinlock precedent): an interval of 0 is dropped; a row with no stored interval uses the gap since the file's previous collection; a bucket's rate is summed bytes over summed seconds. A row whose delta is NULL is dropped and adds no seconds. Latency is summed stall over summed operations, and a bucket with rated rows but no reads (or writes) reads 0 ms, as the chart does.
- Same envelope as before: omitted nulls, `discontinuities[]`, `empty` / `unavailable` / `not_collected`.

## What changed
- `ServerTrendSql.cs`: `TempDbSize`, `TempDbFileIo(n)`, `FileIoThroughput(n)`, the two top-N reads and three "ever collected" probes. Every name is a bound parameter.
- `DarlingMcpServerTrendTools.cs`: the three metrics, their series kinds, fields, notes and capability questions; `DarlingWebEndpoints.cs`: the catalog text.
- Not changed: the desktop's own SQL (the viewer's reads are untouched; the desktop returns a fixed top-10 per bucket width, so the bucketed name-filtered reads are written new in the house style, as in the latch/spinlock change). No migration, no `wwwroot`.

## Budget (this PR moves the tools/list budget)
- `tools/list` measured on the merged tree: **185,971 bytes**, under the existing 186,158 ceiling, so `TotalCeilingBytes` is unchanged.
- Per-file pins moved: tool head 331 -> 380, `names` parameter 70 -> 40 (the per-metric detail went after the guide marker). It is a budget mover.

## Verified at the head
- Build Release, `--no-incremental`: 0 errors, no CA/CS/IDE warnings.
- `DarlingMcpServerTrendToolsTests` 14/14. Live class 4/4 on a throwaway TimescaleDB 2.30.1 / PG18 container (host port 55645, removed). New live test plants: a tempdb file with two rated rows, a restart row (interval 0) and an all-NULL-delta row (neither counts); a user file with no stored interval (gap-to-previous is the interval); a log file with a NULL write delta (adds no seconds); two tempdb size readings; `names` with a miss, the default selection, and the `empty` answer.
- RED on dev: the new `Metrics` entries, `ServerTrendSql.TempDbFileIo`/`FileIoThroughput`/`TempDbSize` do not exist there, so the tests fail to compile (RED by compile failure).
- Guard family (Census/Ratchet/Adoption/Discipline/Guard/Scrub/Hygiene/Convention/Contract/Pin) plus `McpToolsListBudgetTests`, `McpToolGuideTests`, `McpToolGuideHeadsDataTests`, `WebReadCancellationPinTests`, `McpPayloadContractCensusTests`, the three command-timeout classes, `EngineCapabilityMissTests`, `ViewerCpuTempDbTests`, `ViewerFileIoBlockingTests`, `TrendBucketingLiveTests`, `DarlingMcpDataToolsTests`, Web server-trend page/behaviour tests: 1,184 run, 0 failed, 6 skipped. `LivePostgresCollectionHygieneTests` loads WPF types and cannot run on a Mac.
- Full Darling.Tests once: 23,114 run, 553 failed, 250 skipped; every failing name is on the Mac-local failing list, none is new.
- Lite.Tests classes naming a touched symbol (VM run): `CrossAppMcpToolInventoryPinTests` (names `get_server_trend`; its tool list is unchanged). Not run on this Mac.

## CHANGELOG entry
SECTION: Added
ENTRY: - **Tempdb and file I/O trends over MCP** ([#5187]) - `get_server_trend` now also returns the tempdb per-file I/O latency, the tempdb allocated size and the per-file I/O throughput trends the desktop charts draw, with a `names` filter for the per-file ones.
REF: [#5187]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5187

## Round-1 fixes (head 58c6a8a18)
- **M2, the rule that was not actually covered.** The live test now plants a tempdev row with reads set and its read stall NULL, and the bucket must still read 5.0 ms. Proven by mutation: with the paired read guard removed from `TempDbFileIo`, the live test fails `Expected: 5, Actual: 0.28`; with the guard restored it passes. Before this, the guard could be deleted with the whole suite green.
- **M1, and a fixture gap it exposed.** `tempdb_file_io`, `tempdb_size` and `file_io_throughput` were added to the `later` array that drives the unavailable, empty and 168-hour 32 KB checks. That made the checks fail with "unavailable" where "empty" was expected — **a gap in the test's own fixtures, not a bug in the tool**: the test planted no `file_io_stats` or `tempdb_stats` rows at all. The fixture data was added rather than the check narrowed. The week-long 32 KB checks now cover all three new metrics.
- **L1.** The tool text now says a bucket with usable rows but no reads (or no writes) reads 0 for that side, and that a bucket with no usable read row at all is left out. Behaviour unchanged — the text was wrong, not the SQL.
- **L3.** `FileIoThroughput` treats a negative delta as no delta, so a counter reset on an old row with no stored interval cannot drag a bucket negative. A `reset.mdf` row with a negative delta holds the bucket at 0.1 MB/s.
- **Budget.** `tools/list` still measures 185,971 bytes: the corrected text lives in the per-call note, not in `tools/list`. No per-file pin touched. **This branch's pin will be re-measured and raised after #5163 lands**, since that PR takes dev's growth-only ceiling to 186,917.
- **Verified by pm-pr-worker, not the lane:** scrubcheck on the full diff (0 hits), and an independent baseline diff of every repo-source-scanning class.

* MCP: Lite's system-health, default trace, wait stats and blocking stats tools say where their data starts (part of #4966) (#5172)

Thirteen more Lite MCP tools now say where their data starts, with the contract Lite's other window-floor tools use (#5004, #5015, #5127, #5147): effective_start (UTC, with a Z), window_truncated and truncation_note right after hours_back on a data answer, and the same keys under hints on an empty answer. not_collected and unavailable answers keep their shape, and a window of 90 minutes or less starts no probe on a data answer. This also carries the work first opened as #5180.

System health: the nine get_health_parser_* tools read system_health_events on event_time, and the coverage probe reads the same column, so an event a first collection stored from before itself is already named by the probe. The probe counts every event type and the collector's runs, so a long-monitored server with a rare event type gets no false notice.

Default trace: the probe converts each candidate row from the monitored server's wall clock to UTC, as the read does. Wait stats: the keys ride on the data answer only. Blocking stats: blocking and deadlocks are separate series in one answer, so the one notice sits at the LATER of the two series' starts; inside the blocking series the blocked process reports and the DMV blocking snapshots stand in for each other, so the earlier of those two is the blocking start. get_daily_summary_range gets no notice: each row carries its data_state and the payload its retention_horizon, and days with no data are omitted rather than shown as quiet.

Tests: McpWindowNoticeSystemHealthToolTests (new) across the nine tools, and McpWindowNoticeAggregateToolTests (new) for the other three plus the daily-summary pin: a notice when collection starts inside the window, a quiet start, backfilled and sparse events, hints on each empty answer, the key order, a short window, capped pages, a failed probe, and for blocking stats each series covered or cut in both orders. Census and EffectiveStartUtcTests rosters list every written file.

Part of #4966

* LITE AND DESKTOP: the blocking charts' data-start note starts where the blocked process threshold went on (#5199)

In Lite and the desktop viewer, when the Blocking trend, the blocking duration Stats pair and the Overview blocking chart draw blocked process reports, the "Showing since" note now also accounts for the server's blocked process threshold. A threshold of 0 means no reports can exist, so a stretch with the threshold off used to read as no blocking with no note.

The rule, shared by both apps in PerformanceMonitor.Common's BlockingThresholdCoverage.Combine: read the server_config snapshots of "blocked process threshold (s)" (value_in_use). Only when a snapshot showed 0 before the first nonzero one in the window, and the threshold was not already on at the window's start, the note starts at the later of the report collector's coverage and that first nonzero snapshot, or at the earliest report if that is sooner. With no config rows (Azure SQL Database), a threshold on before the window, or the DMV-snapshot branch, the note is unchanged from #5159. Snapshots are taken at connect and daily, so the named start can be up to about a day late; a threshold turned off midway is not covered.

Lite takes its config snapshot before the blocked process collector turns the threshold on, so that collector now takes one more server_config capture right after it changes the threshold, and a new server's first day does not read as cut. A failed threshold read costs only the note; on the Overview chart it falls back to the collector's own answer.

Tests: BlockingThresholdCoverageTests across every arm; the viewer's live cases (threshold on midway, a report before the first nonzero snapshot, on before the window, no config rows, the DMV branch, nonzero snapshots with no zero seen, a zero after the first nonzero); the same cases in Lite's BlockingChartsDataStartTests and the Overview chart's live tests, a throwing threshold read, and a pin on the recapture. The pins that follow the new calls keep what they guard.

Fixes #5098

* Web Wait Stats and Activity tabs chart the latch, spinlock and session stats trends (#5189)


Part of #4843

## What users see
- **Wait Stats tab:** a "Latch Trend" chart (wait ms/s) and a "Spinlock Trend" chart (collisions/s), each under its table. Each has the shared multi-select, capped at the read's 10 names. Options are the classes of the table above it, heaviest first; the heaviest five are checked the first time. The checked set and search text survive the 60 s rebuild and are kept per kind and per server.
- **Activity tab:** a "Session Stats Trend" chart (total, running, sleeping, background, dormant, idle over 30 min, waiting for memory), with the newest bucket's top application and host as text.
- Names the read found no samples for are listed in a notice (read from the top level and from `hints`). Baseline discontinuities show through `discontinuityNotes`. Empty and error answers show the read's own message. All three charts use the zoomable chart path.

## What changed
- `server-tabs.js`, Wait Stats and Activity regions only: `namedTrendPanel`, `drawNamedTrends` (latch, spinlock) and `sessionStatsTrendPanel`. Reads go through `readToolWithinKeptHistory` with `{ server, metric, hours }` (and `names`).
- `Lite.Tests/BaselineDiscontinuityRenderCensusTests`: the census now counts eight `discontinuityNotes(...)` uses and names the two new functions. Lite.Tests builds here but cannot run; I traced the regex by hand: the new uses are `discontinuityNotes(trend.data)` in `drawNamedTrends` and `discontinuityNotes(res.data)` in `sessionStatsTrendPanel`, so 6 + 2 = 8. **Needs a VM run.**
- Tests: `WebWaitsActivityTrendsPageTests` (read names, params, placement) and `WebWaitsActivityTrendsBehaviourTests` with `web-waits-activity-trends-harness.mjs` (series and unit, defaults, state across a rebuild and per server and kind, `missing_names`, empty, discontinuity).

## Not changed
- The File I/O tab charts (file I/O throughput, tempdb size and tempdb file I/O) are a follow-up once their `get_server_trend` metrics land. This PR adds no tool metric and does not move the MCP budget.
- No other `server-tabs.js` region, `panels.js`, `util.js`, CSS or tool/store change.

## Verified at the head
- RED on dev's `server-tabs.js`: `WebWaitsActivityTrendsPageTests` 5/5 fail, `WebWaitsActivityTrendsBehaviourTests` 8/8 fail.
- GREEN: 5/5 and 8/8; `WebServerTrendsPageTests` 5, `WebServerTrendsBehaviourTests` 9, `WebModuleLoadTests` 7, `ServerPageTabsTests` 20, `ChartIntegerTicksPinTests` 3, `WebChartZoomBehaviourTests` 10, `GridFilterBehaviourTests` 8.
- 124 classes (guard family, every `.mjs`-harness class, budget and timeout census): only `LivePostgresCollectionHygieneTests` fails (Mac-local, WPF load).
- `node --check` clean; `load-web-modules.mjs` loads 48 modules.
- Lite.Tests: build only. VM run needed: `BaselineDiscontinuityRenderCensusTests`.

## CHANGELOG entry
SECTION: Added
ENTRY: - **Latch, spinlock and session stats trend charts on the web server page** ([#5189]) - The Wait Stats tab charts latch classes and spinlocks, with a selector for which to show, and the Activity tab charts session counts over time, matching the desktop.
REF: [#5189]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5189

* Web: File I/O tab charts write latency beside read latency, and the Version Store trend spans seven days (#5198)


Part of #4843

## What users see change

- **Server page, File I/O tab:** the latency panel now draws a read-latency chart and a write-latency chart, one line per database and file type in each. Before, only read latency was drawn.
- **FinOps, Version Store (PVS):** the size trend covers the last 7 days (it covered 24 hours). The heading and the empty-state text follow the window.

## What changed

- `server-tabs.js` `fileIoPanel`: one `get_file_io_trend` read feeds two pivots (`avg_read_latency_ms`, `avg_write_latency_ms`) and two zoomable charts (`file-io-latency`, `file-io-write-latency`), stacked as the desktop does. Both go through the same helper, share one zoom scope, keep the discontinuity notice and the kept-history strip, and show the read's own empty message. A bucket with no reads (or no writes) carries a null and is left out of that chart rather than drawn as 0; a window with reads but no writes shows "No writes in this window." under the write heading. No selection state is added, so the 60-second poll has nothing to reset.
- `version-store.js`: `TREND_HOURS` 24 -> 168.
- Tests: `WebFileIoAndPvsWindowPageTests` (read name, params, fields, chart ids, the 168 pin and the read's ceiling), `WebFileIoLatencyBehaviourTests` (node-run: both charts, discontinuity notice, no-writes window, empty answer) with new scenarios in `web-server-trends-harness.mjs`; `FinOpsTabPvsPageTests` now pins 168.

## Decisions

- **Desktop chart matched:** the Viewer's File I/O Latency sub-tab, which stacks a read-latency chart over a write-latency chart. I made two charts for the same reason: read and write latency differ by an order of magnitude on many servers, and one shared axis flattens the smaller one.
- **Queued I/O overlay: left out.** `get_file_io_trend` returns no queued-I/O field; its points carry reads, writes, stall read/write ms, average and peak read/write latency only. The missing fields are the queued read and queued write latencies (the desktop's `AvgQueuedReadLatencyMs` / `AvgQueuedWriteLatencyMs`). The read is not widened here.
- **PVS window:** 168 hours. `get_pvs_stats` validates `trend_hours_back` with `ValidateHoursBack`, whose ceiling is `MaxHoursBack = 168`; a larger value is refused, not clamped, so 168 is the real cap and exactly 7 days. The trend query is not bucketed, so 7 days returns every collection point for 5 databases; the chart already draws these through the shared zoomable chart with zoom, and `windowFromHours(168)` sets the axis. I did not add downsampling.

## Not changed

No read widened, no migration, `panels.js` and `global.json` untouched.

## Verified at the head

- Build Release (`-p:EnableWindowsTargeting=true`): 0 errors, no analyzer warnings in the changed files.
- New classes plus `FinOpsTabPvsPageTests`, `WebServerTrendsBehaviourTests`, `WebServerTrendsPageTests`, `WebModuleLoadTests`, `ServerPageTabsTests`, `ChartIntegerTicksPinTests`, `WebChartZoomBehaviourTests`: 66 run, 0 failed after the null-bucket fix (re-run afterwards as part of the guard run below).
- Guard family (Census, Ratchet, Adoption, Discipline, Guard, Scrub, Hygiene, Pin, Convention, Contract) and every class that runs a `.mjs` harness, 122 classes: 1449 run, 1 failed: `LivePostgresCollectionHygieneTests` (WPF load, fails on dev on this Mac).
- Pin sweep: `git grep` of both test projects for `version-store.js`, `fileIoPanel`, `file-io-latency`, `TREND_HOURS`, the harness name. Darling hits (`FinOpsTabPvsPageTests`, `ServerPageTabsTests`, `WebChartZoomBehaviourTests`, `WebServerTrendsBehaviourTests`, plus `WebRangeKeptHistoryBehaviourTests`, `WebServerPageRangeTests`, `BaselineDiscontinuityRenderTests`) all pass. Lite hit for the VM: `BaselineDiscontinuityRenderCensusTests` (reads `server-tabs.js`; it counts `discontinuityNotes(res.data)` uses, still 6, and checks `fileIoPanel` contains one).
- Full Darling.Tests on this Mac: failures are all in the Mac-local list (WPF/platform classes such as `Viewer4629Tests`, `ViewerChartContextMenuTests`, `ViewerDisabledListBoxThemeTests`, `DarlingStoreUpgradeRevertTests`); none touch a file in this diff.
- RED on dev: the file I/O node scenarios show one chart (read only) on dev's script; the 24 -> 168 pin fails on dev's constant.
- `node --check` clean on both changed `.js` files; scrubcheck clean on this file and the diff; commit author is Erik.

## CHANGELOG entry

SECTION: Changed
ENTRY: - **Web File I/O tab charts write latency, and the Version Store trend spans 7 days** ([#5198]) - the server page's File I/O latency panel now draws a write-latency chart beside the read-latency chart, and the FinOps Version Store size trend covers the last 7 days instead of 24 hours, as the desktop viewer does.
REF: [#5198]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5198

* Web grids: the composed-panel, alert-rule test and Fleet Sweeps tables get the main grid's tools through one shared helper (#5188)


Part of #4843

## What users see change
- The composed-panel table, the alert-rule editor's test result and the Fleet Sweeps record tables (band changes, would-have-paged, per-server verdicts, watch items, store-host settings) now sort by header, filter per column, and have Copy cell / Copy row / Copy all / Export CSV, as the Server-page grids do.
- Copy and CSV carry the filtered rows in the shown order. CSV carries raw values: the stored bucket instant and the value as the panel returns it (already in the panel's unit) in a composed panel, the raw reading in the rule test, the stored timestamps in watch items.
- Sort, filter text and the picked cell survive the 60 s redraw and are kept per table AND per server (composed panels: panel identity plus the scope's server; rule test: the editor route; sweeps: route plus table, watch items per state shown).

## What changed
- `panels.js`: `vizTable` is now a thin wrapper over a new exported `gridTable(rows, desc)`, which holds everything it did (sort, column groups, filters, Copy, CSV, `copy:false`). Two small column fields for pages that build their own cells: `display(row)` (text shown over a raw value) and `cellClass(row)`.
- `compose.js`, `alert-editor.js`, `pages/sweeps.js` draw their tables through `gridTable`. `renderTestResult` is exported for the test. `renderComposedResult` now passes the scope through so the table is keyed by server.
- No CSS change: the grids reuse the existing grid-tool and filter rules.

## Each renderer
- **Composed table:** sort, filters, Copy, CSV.
- **Alert-rule test result:** all of them. The verdict column sorts, filters and copies as its text ("Yes — Critical", "No", "no data").
- **Sweeps:** all of them on the five record tables.
- **Left out:** the sweeps page's "Instrument / Reading" and "Host / Value" cards stay plain two-column readouts (label and sentence, not rows to sort or export). The storage-growth heatmap is untouched, as agreed.

## Not changed
`vizTable` behaviour, `grid-tools.js`, #5183's filter code, column groups.

## Verified at the head
- New `GridAdoptionBehaviourTests` (node harness `web-grid-adoption-harness.mjs`): 4/4. RED on dev: all four scenarios throw (no filter/Copy buttons on composed and sweeps tables, `renderTestResult` and `gridTable` not exported). The `vizTable` pin proves identical groups, `copy:false` control column with no filter button, sort, filtered Copy/CSV with all columns, the empty and no-fields strips.
- `GridFilterBehaviourTests` 8/8, `GridToolsBehaviourTests` 9/9, `GridSortBehaviourTests` 10/10, `ColumnGroupsBehaviourTests` 6/6, `WebModuleLoadTests` 7/7, `ServerPageTabsTests` 20/20, `AlertNotebookRenderBehaviourTests` 7/7, `FleetSweepWebFeedTests` 39/39.
- Load harness: 48 modules load. The compose-notice harness's panels.js stand-in gained a `gridTable` stub.
- Guard family (76 classes) and classes naming changed files/symbols (FinOpsTab*PageTests, Web*Tests, Chart*, DarlingCompose, DarlingWebAssets): green except `LivePostgresCollectionHygieneTests` (WPF load, Mac-local, on the known list).
- Full Darling.Tests on this Mac: every failure is on the dev-local list; none in the web classes.
- Source-text pin caught and fixed: `FleetSweepWebFeedTests` pins `w.fleet_scope ? "Fleet" : (w.server || "server " + w.server_id)` verbatim, and the watch table keeps that exact expression.
- Lite.Tests: no class names a changed web symbol; nothing for a VM run.

## CHANGELOG entry
SECTION: Changed
ENTRY: The composed-panel table, the alert-rule test result and the Fleet Sweeps tables in the web dashboard now sort, filter by column, copy and export to CSV like the main grids ([#5188])

## Round-1 review fixes
- Two untitled or same-titled composed table panels on one view no longer share sort, filters or the picked cell: the key now falls through an empty title (`||`) and carries the panel's position on its view.
- Fleet Sweeps "Fleet" and "server N" labels now sort, filter and export as shown. The watch-items Position column sorts by its count.
- An empty `cellClass` result no longer adds the class "undefined".
- Sweep tables are keyed by the selected sweep, and the alert test table by the rule (or by the draft for a new rule).
- New tests: two untitled panels, `renderComposedInto` against the real grid, and `vizTable` edge shapes.

* MCP: Darling's system-health, default trace, wait, latch and spinlock tools say where their data starts (part of #4966) (#5191)

Thirteen Darling MCP tools now say where their data starts, with the contract Lite's twins and Darling's other window-floor tools use: the nine get_health_parser_* tools, get_default_trace_events, get_wait_stats, get_latch_stats and get_spinlock_stats write effective_start (UTC, with a Z), window_truncated and truncation_note right after hours_back on a data answer, and the health and default trace tools carry the same keys under hints on an empty answer. not_collected and unavailable stay bare, a data answer over 90 minutes or less starts no probe, and a failed probe costs the notice, never the rows.

System health and default trace probe their table's coverage (system_health_events, default_trace_events: the later of the retention edge and the server's registration, counting every event type and the collector's runs), so a rare event type on an old server does not read as a late start. These reads window on event time and a first collection stores history from before itself, so the notice names the earlier of the coverage start and the earliest qualifying event (the desktop System Events rule), taken from every row the read returned before the page limit; default trace times are converted from the server's own clock first. Wait, latch and spinlock stats sum their raw table over the window on collection_time; their rank caps hide no time range, so truncated and window_truncated stay independent.

On the web page, the ten health and default trace reads are not yet on the web's list: their table panels show the tool's own note on a data answer when the window was cut.

Tests: the live classes for every tool (a start inside the window, a quiet start, a backfilled event, a sparse event type, a limited page naming its earliest qualifying event, a window with no coverage but events in it, empty answers with hints, the key order, a short window, a capped page, a failed probe), and the census and EffectiveStartUtcTests rosters.

Part of #4966

* WEB: the PostgreSQL stat tiles say where their data starts (part of #4966) (#5193)

WEB: the PostgreSQL stat tiles say where their data starts (part of #4966)

Part of #4966. The web PostgreSQL Overview and Vacuum tabs showed five windowed reads with no data-start note, so a server whose history starts after the window's start mixed a partial window into its peaks, growth counts and first-to-last deltas without saying so.

- WebDataStartNote maps the five reads to their collector tables. get_pg_wraparound_risk (pg_wraparound_stats) and get_pg_write_stats (pg_write_stats) are dense and use the table rule. get_pg_xmin_horizon, get_pg_autovacuum_health and get_pg_replication_slots write rows only while something exists, so they are probed on the collector's logged runs (collectors pg_xmin_horizon, pg_autovacuum_stats, pg_replication_slots). no_holder is admitted as xmin's nothing-found word, because the tool answers it only after captures exist in the window; no_slots and no_pending_maintenance stay out, conservatively.
- server-tabs.js: the note draws on every panel that shows a window figure (Per-Database Headroom, Horizon Holders, Autovacuum Backlog and Tables Behind, Replication Slots and Slot Summary, Checkpoints and WAL). It is opted out (windowNote: false) on the panels that show only the newest reading or a constant: Where the Thresholds Are, the Vacuum tab's Freeze Headroom and What Holds the Horizon tiles, and the Overview's Freeze Headroom and xmin Horizon tiles (a small momentStat helper).
- Tests: WebDataStartNoteTests pins the map, the probe choice, the admitted words and each opt-out; WebDataStartNoteLiveTests covers all six sparse reads; the page harness checks which panels draw the note; ServerPageTabsTests' descriptor roster now counts momentStat tiles.

* procedure_stats: the deferred plan fetch, collector half, inert (#5158) (#5194)

procedure_stats can now carry a known plan digest and fetch module plans in a second target query, the same collector-side mechanism #5173 added for query_stats. Nothing in the service turns it on yet, so production behaviour is unchanged.

**Why.** On one production store, about 94% of the procedure_stats plan XML rendered in an hour was a repeat of a plan the store already held (17× the distinct volume). The over-cap plans were re-rendered on every capture run.

**Identity.** A module's plan cannot be identified by its handle and `cached_time` alone: a statement-level recompile rewrites one statement's plan inside the same cached module plan. So in deferred mode, the main query carries a small fingerprint from `sys.dm_exec_query_stats` over the module's plan handle: the statement count, the latest statement compile time and the summed plan-generation number. These are three identity-only columns, not payload columns. The fingerprint's SQL is provisional until its cost is measured.

**What changes.**
- With the deferred flag off, the main query, the reader ordinals and the payload are byte-identical to before. Four SQL goldens prove it, hashed from the unchanged code.
- The plan column goes through `PayloadOrDigest`. The host sets the plan size from its cache for a known identity, as for query_stats, so the size column and the oversized-plan backlog both see it.
- `BuildPlanFetchQuery` takes plan handles and always fetches the whole module, offsets 0 and -1. It reuses query_stats' fetch query, with no copy.
- `TryParsePlanHandle` parses the shipped hex handle.

**Tests.** The new tests cover:
- the deferred and inline query shapes;
- the module-offset pin;
- the shared fetch query;
- handle parsing;
- the reader ordinals;
- the digest path;
- a known over-cap identity still recorded as oversized;
- no quote in the nested dynamic-SQL fragments;
- byte-for-byte goldens of the unchanged SQL.

The fragment-strip test is line-ending safe.

Part of #5158

* Tests: the system-health and default-trace window-notice tests get their own live-postgres classes (part of #4966) (#5214)

LivePostgresCollectionHygieneTests failed on dev: #5191 added the system-health and default-trace window-notice tests as second partial parts of the existing live classes, so the [Collection("live-postgres")] attribute sat only on the other part, and the guard, which checks each file's declaration header, flagged the new files (a duplicate attribute on the second part would not compile). Each window-notice file now declares its own non-partial class with [Collection("live-postgres")] (DarlingMcpHealthParserToolsWindowNoticeLiveTests, DarlingMcpDefaultTraceToolsWindowNoticeLiveTests), the shape the guard recommends, and the two original classes are no longer partial. The tests themselves are unchanged; the guard is unchanged.

Part of #4966

* DARLING: the store keeps an hourly history of its own pg_stat_statements deltas (V163, part of #5097) (#5203)

The Darling store now keeps an hourly history of its own pg_stat_statements deltas. Migration V163 adds three plain tables: collect.store_statement_captures (one row per hourly snapshot: the interval, how many statements were active, how many the cap hid, the extension's dealloc count, and the outcome), collect.store_statement_history (per-statement deltas of calls, time, rows and blocks for the 100 statements that spent the most time in the hour), and config.store_statement_baseline (the cumulative counters the next delta is taken against; state, not data, so no purge ages it out). History and captures are kept 90 days.

The hourly store self-metrics sweep runs StoreStatementHistory.SnapshotAsync in its own catch and its own 120-second slice of the sweep budget, so a slow or failing snapshot never skips the collector-cost and read-latency flushes after it. One transaction per pass: a reader that fails rolls the whole pass back and leaves the baseline alone. The deltas are computed in SQL against the stored baseline, so service restarts, store restarts and nightly upgrades lose nothing; the epoch is stats_reset (unchanged: subtract; a reset after the previous capture: credit the current counters whole; any other change: record a new baseline and no history). It reads only through config.store_statement_stats(); when that reader is unusable (the extension not installed or not preloaded, or no TEMP privilege), the pass writes one precondition capture and no history. A failure warns once and then logs at Debug until a pass succeeds, with the SqlState.

The viewer's schema probe gains the V163 arm. No MCP tool or tool description changes; a reader tool comes later.

Tests: the rung's shape, the epoch decision's cases, the SQL and worker pins (its own slice, the ready check), and live cases against a scratch store with planted reader functions: the first capture, a same-epoch delta (with NULL and with real stats_reset stamps), an entry restart, a reset inside the interval, a rebaseline, the top-100 cap and its hidden count, retention keeping the baseline, a precondition capture for an unloaded extension, and a failing reader that rolls the pass back. The migration ladder, plain-table census and alert-read exemption pins follow the new rung.

Part of #5097

* CI: per-test Lite timings as an artifact, and temporary cap guards while #5208 is open (#5211)

CI: per-test Lite timings as an artifact, and temporary cap guards while #5208 is open

Part of #5208. Lite.Tests on the hosted Windows runners grew from about 9½ minutes of test time to about 45 in five days (+7 min/day on ~110 new tests), and the nightly was cancelled at its 90-minute build cap during the unsharded Lite run. This makes the slowdown measurable and keeps CI and the nightly from failing on it; it does not fix it.

- Every Lite.Tests run in CI writes an xUnit v3 result file (-xml) with per-test durations, uploaded as a 7-day artifact: lite-tests-timing- (one file per chunk), lite-tests-timing-nightly and lite-tests-timing-release. The run commands, the sharding and the release's single unsharded pass are otherwise unchanged.
- .github/scripts/lite-timing-top.py prints the 25 slowest classes and tests after each run; it skips an unreadable file and prints UTF-8. The summary and upload steps are if: always() and continue-on-error, so they can never turn a job red.
- Temporary guards, each commented as a guard against timeout reds and not a fix, to come back down when #5208 is fixed: lite-tests timeout-minutes 20 -> 30, and the nightly build job 90 -> 150. The release job has no cap of its own and is unchanged.

* Web: the event grids' data-start note uses the earliest event shown (part of #4966) (#5197)

The web server page's data-start note now handles event grids. Blocked Process Reports, Long Query Completions, Memory Pressure Events and Default Trace join the web's listed reads. Each lists events by their own time, and a first collection stores a server's earlier event history, so the note names the earlier of the collector's coverage start and the earliest event shown (the desktop viewer's ViewerEventDataStart.Of rule, moved from the viewer to Darling.Storage so the web and the viewer share one copy). History that reaches the window start shows no note.

The capped-list rule is table-driven (CappedByRead, six reads): a capped page names its oldest row shown without asking the store, Blocked Process Reports by truncated and oldest_returned_event_time, Default Trace by total_events above shown. The collection log's slowest-first escape and the PostgreSQL config changes' behaviour are unchanged, and all six capped reads share one as_of window end. Long Query Completions is ranked by duration, so on a truncated page its oldest row says nothing about reach and the note uses the coverage start alone. memory_pressure_events gets its own probe source, and the Memory Pressure grid now draws the note on a data answer too.

Tests: live cases for the four reads (history reaching back before coverage names its earliest event; history reaching the window start gives none); no-store cases for the capped Default Trace and Blocked Process Reports pages and the truncated long-query page; the CappedByRead key set pinned; the Memory Pressure page harness. The desktop viewer classes using the moved rule pass on Windows.

Part of #4966

* MCP: Darling's blocking stats and PostgreSQL deadlock and log event tools say where their data starts (part of #4966) (#5205)

Three more Darling MCP tools say where their data starts: get_blocking_stats, get_pg_deadlocks and get_pg_log_events write effective_start (UTC, with a Z), window_truncated and truncation_note right after hours_back on a data answer, and the same keys under hints on an empty answer. not_collected and the runtime precondition stay bare and are decided before any probe; a data answer over 90 minutes or less starts no probe; a failed probe costs the notice, never the rows.

get_blocking_stats holds two separate per-minute series, so its one notice sits at the LATER of their two start points, as its Lite twin's does. Blocking takes the earlier of the blocked process reports' and the DMV blocking snapshots' coverage (they stand in for each other, the sources get_blocking probes); deadlocks take theirs; each series first takes the earlier of its coverage and its own oldest bucket, because both read on event time and a first collection can store events from before itself. A series whose collector started inside the window can no longer hide behind the other's coverage.

get_pg_deadlocks and get_pg_log_events are newest-first capped event lists: the floor is the earlier of the coverage probe (ForCollectorTable over pg_deadlocks and pg_log_events, both with a schedule retention edge) and the oldest event shown, through the same ReadEventAsync the SQL Server event tools use. The notice is coverage whether or not the cap cut the page; the answer reports the cap through truncated.

Tests: get_blocking_stats' live cases (each series covered or late in both orders, the later floor winning, deadlock rows only, XE early and DMV late, both covered, an event older than its collection, empty hints, a short window, a failed probe) and a new PgEventToolsWindowFloorLiveTests for the two PostgreSQL tools (including a full capped page beside an early coverage reading covered while truncated); the census and EffectiveStartUtcTests rosters.

Part of #4966

* The store accepts a known plan digest for a row and keeps that plan alive (inert, #5158) (#5174)

The store accepts a known plan digest for a row and keeps that plan alive (inert, #5158)

Part of #5158, the second of three PRs. No collector passes a digest yet, so nothing changes.

- PgCollectorRowWriter.PayloadOrDigest: a known digest with no content writes the digest straight into the row's digest column, adds it to the batch's touch set, and inserts no dimension row. A malformed digest throws (hex, exactly 32 bytes), and so does a known digest at a position with no dimension diversion, which would otherwise write a row with no plan. Every other call is today's path. The digest is PayloadDimensions.Digest's own output; nothing re-derives it.
- PayloadDimensionWriter.FlushAsync runs one statement per dimension table in the batch's transaction (PayloadDimensions.TouchSql). It refreshes last_seen for the touched digests older than the existing 6-hour guard, and returns the touched digests that have no dimension row, as upper-case hex.
- The touch takes FOR UPDATE SKIP LOCKED. The upsert and the touch are two locking statements, so no single lock order covers both, and two batches that each upsert one stale digest and touch the other's would otherwise deadlock. Skipping a locked row is safe: whoever holds it is refreshing it, a holder that rolls back is retouched next cycle, and the dimension sweep cannot be the holder because it only reaches rows at least the guard plus a day old.
- Tests: PayloadDigestPassThroughLiveTests (8): a digest-only row resolves its plan like a content row; the 6-hour guard; the absent report, including a lower-case digest; content wins over an offered digest; the throws; and two batches on two connections that neither wait nor deadlock (removing SKIP LOCKED fails it with a lock timeout).

* query_stats fetches plan XML only for plans this host hasn't stored, behind queryStatsDeferredPlanFetch (on) (#5158) (#5209)

query_stats fetches plan XML only for plans this host hasn't stored, behind queryStatsDeferredPlanFetch (on) (#5158)

Fixes #5158 for query_stats; procedure_stats follows separately. The third of three PRs: #5173 added the collector's deferred fetch and #5174 the store's digest-only write; this wires them together and turns them on.

- Every query_stats run rendered sys.dm_exec_text_query_plan for its ~200 rows and shipped the XML, though almost all were already stored: 53.75 GB/h rendered and 28.83 GB/h shipped on one large store against 1.14 GB/h distinct, with the busiest server's slow runs more than 90% result drain.
- PlanDigestCache, one per server for the service's lifetime, keyed for query_stats on QueryStatsPlanKey (server, database, plan_handle, statement offsets, creation_time, plan_generation_num). An entry is confirmed only after the store commit succeeds, including after a store-write reattempt; digests the store reports missing are evicted ignoring case; entries unseen for an hour are pruned.
- For query_stats with plan capture on, the main query runs without plan columns. On the same target connection and within the same per-item budget, a cache hit carries the stored digest and size (an over-cap hit the size with a NULL plan), and misses go to one BuildPlanFetchQuery call, split above 1,000 keys. A failed second query ships the rows without plans, caches nothing and logs one Warning. No store probe runs first; the host's confirmed commits are the authority. Azure SQL Database keeps the inline capture.
- queryStatsDeferredPlanFetch in darling.json, default true, read at service start; false gives today's inline SQL exactly. Documented in the README. Each run records plans_rendered, plans_rendered_bytes and plan_fetch_ms in its collection log note.
- Accepted differences: a statement recompile between the two queries can store the new plan under the old identity for that one row; a plan the engine updates in place without a recompile keeps its first-rendered XML until the identity changes; after a restart each server renders once in full.
- Tests: PlanDigestCacheTests (identity sensitivity, confirm-after-commit, over-cap, case-insensitive eviction, the split, plan_fetch_ms) and QueryStatsDeferredPlanFetchLiveTests against SQL Server 2022 and the store (run 2 renders no cached plan and resolves through v_query_stats; sp_recompile re-fetches; a plan freed between the queries; an over-cap plan kept by size).

* Lite.Tests: seed the slowest stores in one transaction instead of one commit per row (#5208) (#5224)

Lite.Tests: seed the slowest stores in one transaction instead of one commit per row (#5208)

Part of #5208. The hosted-runner timings show the largest Lite.Tests costs are tests that seed a DuckDB store one auto-committed INSERT per row; on the hosted Windows disk each commit costs about 0.2 s. Test code only; no assertion changes.

- SeedBatch (SharedDuckDbFixture.cs) runs a seed loop inside one transaction. BEGIN and COMMIT take the read lock like each seed statement; callers commit explicitly before the code under test reads, and a commit on dispose is best-effort so it cannot mask a seed failure.
- QuerySnapshotPlanOnDemandTests' 3,500-row benchmark (676 s on a hosted runner) seeds its rows with one INSERT ... SELECT FROM range(3500), the same rows the loop produced; both assertions are unchanged.
- FloorHelper_ReadsTheArchivedFloor_NotJustTheHotTable (1,000 commits to 2), McpReadToolBudgetTests (about 200+ to 1), PerformanceCalendarDataTests' daily range (about 700 to 1), ObjectLockingBudgetTests (200 to 1) and IndexObjectStatsTests (6 to 1) seed inside one batch.
- McpReadToolBudgetTests now also asserts its largest answer is over 2,000 bytes, so an invisible seed cannot pass the budget check vacuously.

* Add POST /api/servers: web server onboarding on the viewer role (#4843) (#5167)


Part of #4843

## What users see change

An admin seat (an OIDC admin, the shared-token seat, or loopback-only mode) can add monitored servers over HTTP with `POST /api/servers`. The body is the same JSON array of server objects the MCP `add_servers` tool takes, and the answer is that tool's own `{requested, added, skipped, collided, failed, results}` envelope. There is no page yet; the Add Server form is a later PR. A read-only seat is refused with 403.

## What changed

- **The grant.** `GRANT INSERT ON config.config_monitored_servers TO viewer;` in `DarlingManagedRoles.cs`, mirrored in `tools/provision-roles.sql`. The role description, the provisioning status line and the README write-surface paragraph name it. The remove route's PR adds DELETE.
- **The route.** `MapServers` in `DarlingWebEndpoints.cs`, beside the mute and tag routes. It calls the existing `add_servers` core, so there is one validation path: the same accepted auth modes, the probe from the service before saving, and typed refusals for duplicates, collisions and probe failures.
- **Tests.** `ServerAddRouteTests` (21), `ServerAddViewerRoleLiveTests` (3, live), and a new pin in `DarlingManagedRolesTests`.

## The grant, in detail

**The grant.**
- INSERT only. No UPDATE (the core never updates a row) and no DELETE (no web route removes a server yet). A live test proves `UPDATE ... SET is_enabled` and `UPDATE ... SET encrypted_password` both fail 42501 for a role holding exactly the shipped viewer grants.
- `encrypted_password` stays SELECT-carved from viewer, so the web host can write a password blob and never read one back. Two things were checked while building: the INSERT names no `RETURNING`, and its `ON CONFLICT (server_id) DO NOTHING` needs no carved read. The add test connects as a NOSUPERUSER role built by replaying every GRANT/REVOKE in the managed batch that names viewer, adds a server through the real core, and then shows `SELECT encrypted_password` is still 42501.
- The reload-beacon trigger fires as viewer through the existing two-column `config_service` grant; the live test asserts `config_version` advanced.
- The WPF read-only probe is unchanged: it asks about `config_alert_log` UPDATE, which this does not touch.
- The password travels only in the request body. The route never echoes it, never logs it, and keeps it out of error text.
- The core's caught-exception envelope never reaches the wire. It answers through the fixed 500/503 body, and the failure log carries the sentence, not the request.
- A route-level catch logs only the exception type.
- As a backstop, any submitted password that appears in the core's answer is replaced with `[redacted]` before it is written. The core's answers carry none today.
- Checked the capture paths: nothing in the web host buffers or logs request bodies (no `EnableBuffering`, no HTTP logging), and `read_latency` is not on this route.
- Tests send a recognisable fake secret and assert it appears in no response and no captured log line, across 415, 403, 429, the 20-entry cap, an oversized body, six malformed bodies, a core fault, the core's error envelope, a whole-request refusal and a leaky answer.
- At most 20 servers per request (400 above that).
- 64 KB body cap.
- One add in flight per process: a second gets 429. The slot is released when the core faults.
- Duplicate names are the core's typed `duplicate` refusal.

## What's not changed

- `DELETE /api/servers/{name}` and the Add Server page are separate PRs.
- No change to the core's validation, storage (DPAPI on Windows, `env:`/`file:` references elsewhere) or any MCP tool text.
- No schema change.

## CHANGELOG entry

SECTION: Added
ENTRY: - **Add monitored servers from the web dashboard's API** ([#5167]) - `POST /api/servers` adds servers through the same checks and connection test as the MCP `add_servers` tool, for admin seats only: at most 20 servers per request, one add at a time, and the password is never echoed or logged. The web role gains INSERT on the monitored-server list, and still cannot read the stored password. Re-run `provision-roles.sql` on a bring-your-own store.
REF: [#5167]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5167

## Security, final design

The owner's ruling: references stay allowed with no operator opt-in and no extra steps, because requiring one would break ordinary Linux and container installs where a reference is the only way to supply a SQL password. One rule stands in their place, and it lives in the shared core so the MCP tool is covered as well as this route.

- **The rule.** A `file:` or `env:` password reference must not point at this service's own configuration or secrets. `ParseEntry` in `Mcp/DarlingMcpServerAdminTools.cs` calls `DarlingOwnedSecrets.ReferenceRefusal` immediately after the literal-secret check, reading the password from the same `JsonObject` the core already parsed, so the route and the core cannot disagree about a value. The refusal is one fixed sentence that names no path and no variable.
- **What is owned.** `DarlingConfig.Parse` captures every `env:`/`file:` reference AS WRITTEN, before resolution, by a reflection walk over string properties and string collections (depth cap 8, cycle guard). Capturing after resolution was the original defect: the store's connection string is replaced with the resolved value at parse time, so the store's own credential file never entered the set. The set also holds the config directory; the managed store's data directory resolved through `DarlingManagedPostgres.ResolveDataDirectory` (the raw property is null in a default install, which left a default managed deployment owning nothing); that directory's parent credential files; the compose credential directory and the log-hash-key directory; and the service's own environment names. A census test reads the source and fails when a new `GetEnvironmentVariable` literal or a new secret-ish config property is not covered.
- **Fails closed.** Until the owned set has been populated, every reference is refused. A guard that exists only when configuration happened to load is not a guard. The CLI add-server path already loads configuration before reaching the core.
- **Aliases.** A path that is not rooted, or contains a `..` segment, a `\\?\` or `\\.\` prefix, a UNC root, a leading `~`, a second `:`, or a `/proc` or `/sys` root is refused as a class, because no real secret path needs those forms. Everything else is compared by FILE IDENTITY — `st_dev`/`st_ino` on Unix, volume plus file ID on Windows — after symlink resolution with a hop cap. Identity is what catches a symlink, a case variant on a case-insensitive filesystem, and a Windows 8.3 short name, none of which `Path.GetFullPath` resolves. A reference to a file that does not exist yet is judged through its nearest existing parent.
- **Deliberate scope limit.** A hard link placed OUTSIDE an owned directory that points at an owned file is accepted. Creating one requires local filesystem access as a user who can already read the target, and such a user does not need this route. An earlier revision scanned owned directories for matching identities to catch it, which cost up to 20,000 stat calls on an interactive add over directories that include the PostgreSQL data directory; that scan was removed. `AHardLinkToAnOwnedFile_PlacedOutsideTheOwnedDirectory_IsAccepted` pins the decision so it is visible rather than forgotten.
- **Failure text.** Connection failures return the core's own reason, so an operator can tell a refused connection from a timeout, a TLS failure or a bad login. Any submitted secret is replaced in the answer as a backstop, and the full detail is logged server-side.
- **The in-flight slot.** One add at a time. A second request while one is running gets 429. After 120 seconds the caller gets 503, and the slot stays held until the add actually finishes, so two adds can never run at once; the 503 says a restart clears a slot held by an add that never completes. Every synchronous fault from the core releases the slot, cancellation included.
- **The grant is INSERT only.** DELETE arrives with the remove-server PR.

## Verified, and what is not

- **Verified on this Mac:** `OwnedSecretReferenceRefusalTests` and `DarlingOwnedSecretsCensusTests` together with `ServerAddRouteTests` and `DarlingMcpServerAdminToolsTests`, 61 tests, 0 failed, run by pm-pr-worker on the final head rather than taken from a lane's report. An independent diff of every repo-source-scanning test class against a dev baseline at the same commit shows no new failure. scrubcheck is clean on the diff and on this report.
- **Proven by mutation, not just by passing:** the fail-closed change, the resolved managed data directory, and the compose credential directory each have a test that FAILS before the fix.
- **NOT verified on this machine:** the Windows file-identity path (`CreateFileW` plus `GetFileInformationByHandle`) and the 8.3 short-name case run only as stand-in unit tests here. A Windows VM run of the two owned-set classes is requested. That run also reports `fsutil 8dot3name query C:`, because a short-name test on a volume with 8.3 generation disabled passes without proving anything; if it is disabled the result is recorded as NOT VERIFIED rather than as a pass.
- **Short-name case, updated:** the 8.3 test now creates its own short name (`SetFileShortNameW`, falling back to `fsutil file setshortname`), reads it back, and fails if neither works, so the Windows run decides it rather than the volume's 8.3 setting. It skips on non-Windows hosts, including this Mac.
- **Owned set in the tests, updated:** the live role tests and the off-Windows `ParseRequest` test now populate the owned set with an empty-but-populated one (and restore it), and the fail-closed rule has its own test in the add core's path: with the set left unpopulated, `ParseRequest` refuses every reference entry (RED when the set is populated).

## Review history

Round 1 (security) found the reference-resolution exposure and the probe's error detail. Round 2 found that the owned set was captured after resolution, that the OIDC secret and stored server passwords were missing from it, and that path-text comparison could be defeated by an alias. Round 3 found that a default managed install owned no store files at all, that the compose credential directory was missing, and that an unpopulated set failed open. Rounds 2 and 3 each hit a turn limit before writing their file, so pm-pr-worker transcribed them into `design/pm-pr-worker/reviews/review-5167-r2.md` and `review-5167-r3.md`.

## Windows verification (pm-worker-2's VM, head af361abdb)

Build: 0 warnings, 0 errors. `OwnedSecretReferenceRefusalTests` + `DarlingOwnedSecretsCensusTests`: 33 tests, 0 failed, 0 skipped — the Windows-only test RUNS there, where it skips on macOS. A separate `*ShortName*` method run reported 3 tests, 0 failed, 0 skipped, with no skip or failure lines, so each case below ran rather than being reported green by a class total:

- **`AFileGivenAnExplicit83ShortName_IsRefusedByItsShortPath_AsWellAsItsLongPath`: PASSED.** The test creates its own file, gives it an explicit 8.3 short name (`SetFileShortNameW`, falling back to `fsutil file setshortname`, which works unelevated), reads the short name back to prove it exists, and only then asserts the refusal. It fails rather than passes if the short name cannot be created, so a green result means the condition really existed. The volume's own 8.3 auto-generation setting is unreadable without elevation, which is why the test creates the name instead of relying on it.
- **`AnAliasWhoseTextDiffersButWhoseIdentityMatches_IsRefused_LikeAWindowsShortName`: PASSED.** This failed on Windows at an earlier head. The cause was the TEST, not the code: it injected a fake identity map keyed on Unix-style paths, while `RealPath` runs `Path.GetFullPath` before the identity lookup, and on Windows a leading slash roots to the current drive, so the key gained a `C:` prefix and never matched. Both sides now normalise identically.

Nothing in the identity path is now unproved on Windows.

## One failure that is NOT this PR's

`DarlingMcpServerAdminToolsSurfaceTests.DescriptionMatchesTheAuthModesTheParserAccepts` fails on macOS. A lane raised it as a possible fourth CI failure from this branch; it is not. Run on the same machine against `origin/dev` and against this branch, it fails identically on both (69 tests with 1 failure, and 70 with 1 failure — the extra test being this PR's new fail-closed case). It also does not fail in CI, where other tests in that same class do run, so it is macOS-local rather than off-Windows generally. Nothing here changed it.

* MCP: seventeen Darling PostgreSQL tools say where their data starts, and two say why they don't need to (part of #4966) (#5201)

Seventeen Darling PostgreSQL MCP tools now say where their data starts, and two say why they need no notice. This also carries the work first opened as #5212.

Fifteen tools write effective_start (UTC, with a Z), window_truncated and truncation_note after hours_back on a data answer and under hints on an empty one: wait stats, wait sampling, kernel stats, lock stats, predicate stats, server config changes, top queries, database stats, I/O stats, plans, CPU utilization, blocking, session states, replication stats and the xmin horizon. Each probes the very source the web page's own note probes, through DarlingMcpWindowNotice.SourceFor and WebDataStartNote.TryGetReadSource: the collector's logged runs for the sparse blocking, session-state, replication and xmin-horizon reads, the table's coverage otherwise; CPU utilization, the one read the web does not list, probes its own table, and the web strips its keys and draws no note because its panel is a chart. Config changes report coverage whether or not the page was capped; the answer reports the cap through truncated. A test holds each tool's source equal to the web's, and the set of unlisted reads is pinned.

Write stats take their keys from their own first sample (the span the differences cover), with no probe; index usage takes them from its table's coverage, with a note that scans_in_window undercounts a short history, and joins the web's list. Index bloat and column stats are newest-per-key reads whose own coverage verdicts answer the right question, so they carry no notice, pinned by a test.

not_collected, not_sampled, unavailable and precondition stay bare and are decided before any probe; a data answer over 90 minutes or less starts no probe; a failed probe costs the notice, never the rows.

Tests: the live classes for every tool (a start inside the window, a quiet start, the sparse quiet start where runs precede the first row, a capped page, an empty answer with hints, the key order, a short window, a failed probe, a gap across the window's start for write stats), the source-equality theory, the census and EffectiveStartUtcTests rosters.

Part of #4966

* Web: Silence This Server / Unsilence on the fleet page (#5221)


Part of #4843

## What users see change

On the fleet page, a seat that can edit gets a small **Silence** button on each server card. One click, after a confirm, mutes every alert for that server; the card shows the muted bell at once and the button becomes **Unsilence**. Unsilence (also confirmed) removes the rule. A read-only seat sees no button. Before this, a user had to build the rule by hand on the Mute Rules page.

## What changed

- `pages/fleet.js`: the button, the two actions and their module-scope state.
- `pages/mute-rules.js`: `createRule` and `deleteRule` are now exported, so the fleet page uses the same writes the Mute Rules page uses.
- `css/app.css`: three rules beside the card header rules.
- Two stale comments in `DarlingFleetReader.cs` that said the web has no silence action.
- Tests: `FleetSilencePageTests` (source pins), `FleetSilenceBehaviourTests` with `web-fleet-silence-harness.mjs` (Node), and two existing pins updated (`FleetPageAttentionFilterTests` for the grid mount line; `web-kept-history-harness.mjs` copies the new imports of `fleet.js`).

**Route.** The existing `POST /api/mute-rules` and `DELETE /api/mute-rules/{id}`. The create route already takes `server_id`, `server_name` and `reason`, so it accepts the shape of the desktop's `BuildServerSilenceRule` (store server id, no pattern fields, no expiry, reason "Silenced from server list"). No new route. The server write gate (`DarlingWebSeat.IsRequestAllowed`) still decides; the page only hides the button when `/api/session` says `can_edit` is false.

**How Unsilence finds its rule.** `BuildServerSilenceRule` has no id field, but it stamps a fixed reason, `Silenced from server list`, and the web action writes the same string. Unsilence reads `get_mute_rules` (including paused and expired) and deletes only rules with that server's `server_id`, that reason, and every pattern field empty. A narrower rule, a rule for another server, or a whole-server rule a user wrote by hand with another reason is not touched. If the server is silenced only by such a hand-built rule, Unsilence deletes nothing and says to remove it on the Mute Rules page. The marker is a reason string, so a user who edits that rule's reason on the Mute Rules page detaches it from Unsilence; this is the same string the desktop uses.

**State across the 60 s rebuild.** `silencePending` (servers with a write in flight) and `silenceOverride` (the answer the user just produced, kept until the fleet read agrees or two minutes pass) are module-scope, keyed by server id. A rebuilt card keeps its disabled button and its bell, and a second click during a write is ignored.

**Confirmation.** `window.confirm`, the same as Mute Rules Delete and the Alert History dismiss.

## Not changed

No server code, no new route, no migration. `css/theme.css` untouched.

## Verified at the head

- Build Release, `-p:EnableWindowsTargeting=true`: 0 errors, no CA/CS/IDE/xUnit warnings.
- New classes: `FleetSilencePageTests` 6/6, `FleetSilenceBehaviourTests` 6/6.
- RED on dev: the Node harness run against dev's `fleet.js` shows the gate scenario trivially at 0 buttons, and the other five fail because no button exists.
- `WebModuleLoadTests` 7/7, `ServerPageTabsTests` 20/20, `MuteRulesPageTests` 6/6, `MuteRulesBehaviourTests` 9/9, `FleetPageAttentionFilterTests` 10/10, `WebServerPageRangeTests` 11/11.
- 153 classes: the whole guard family, every `.mjs`-harness class, every class naming `fleet.js`, `mute-rules.js`, `FleetServersSql` or `is_silenced`, and `WebExceptionTextCensusTests`: all pass except `LivePostgresCollectionHygieneTests`, `UnmeasuredMetricsAreNotHealthyTests` (postgres, aurora-postgres) and `ViewerSidebarDotRendersTheCardStatusTests`, which are on the Mac-local failing list and fail on dev too.
- `hygiene-scan.py`: 0 offenders.
- `node --check` on `fleet.js` and `mute-rules.js`: clean.
- Lite.Tests: no class names a changed symbol.

## CHANGELOG entry

SECTION: Added
ENTRY: - **Silence This Server on the web fleet page** ([#5221]) - a seat that can edit gets a Silence / Unsilence button on each server card that writes or removes the whole-server mute rule, the same one the desktop writes.
REF: [#5221]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5221

* Web sidebar: search servers by name or tag, and optionally group them by tag (#5243) (#5259)

The web sidebar's server list gets a search box and a Group by tag toggle. On a large fleet the sidebar was a long, flat, name-sorted list.

**Search.**
- It does a live, case-insensitive substring match over display name, server name and tag names, the same rule the Fleet page uses.
- The term survives the 60-second refresh and route changes, and is not saved.
- While a term is entered, collapsed groups open so that a match is never hidden. Clearing the term restores the saved collapse state.
- An empty result shows one "No servers match" line.

**Group by tag.**
- Favourites come first, then the tag tree from the fleet response, depth-first, then Untagged.
- A server with several tags is listed under each.
- Groups collapse as a subtree. A collapsed header shows how many distinct servers it hides.
- Keyboard focus stays on a header after it is toggled.

**State.** The existing sidebar entry in the browser's local storage also holds the grouped flag and the collapsed group ids. It stores only ids and booleans, never names. Stored data that is corrupt or unknown falls back to the flat view with nothing collapsed.

**How.**
- The Fleet page's tag grouping and matching helpers move unchanged into a shared `js/fleet-groups.js`, and the Fleet page behaves exactly as before.
- The sidebar is a new `js/sidebar.js`.
- There is no C# change, no new route and no MCP change. The data was already in the fleet response.

**Tests.** `SidebarSearchGroupTests` (a Node harness over the shipped modules) covers:
- filter matching;
- the grouped order;
- multi-tag servers;
- the collapse-versus-search rule;
- the active row after a route change;
- focus after a toggle;
- subtree counts;
- storage round trips and the corrupt-data fallback;
- the Fleet page using the shared helpers.

**Known follow-ups:**
- toggling a sidebar group also redraws the Fleet grid, which is correct but costs a little;
- a server whose tag is missing from the tag forest drops out of the grouped view. That is inherited from the Fleet page's helper.

Fixes #5243

* Web grid: filter a column by number, text operator or empty cell (#5255)

Web grid: filter a column by number, text operator or empty cell

Fixes #5248
Part of #4843

## What users see change
Typing in a column filter works exactly as before (case-insensitive substring of the cell text). Beside the box there is now a "Match" list with the desktop's operators, chosen only when wanted:
- number columns: Contains, Equals, Not equals, >, >=, <, <=, Is empty, Is not empty
- text columns: Contains, Equals, Not equals, Starts with, Ends with, Is empty, Is not empty (a comma separates alternatives, as on the desktop)
- time columns: Contains, Is empty, Is not empty

The chip names the choice ("CPU: > 900"). The operator is kept at module scope with the text, so the 60 s repaint keeps it. Is empty / Is not empty hide the text box.

## What changed
`panels.js` only. The column's numeric value comes from the same path the sort uses: a new `columnValue` (sortValue, else the row key) feeds both the sort and the filter, and `sortKeyOf` turns it into a number. The column kind is `sortKindOf`, the same decision the sort makes. `> 900` therefore matches a cell showing "1,000". The typed number drops `,` `%` `$` and spaces, as the desktop does. If the cell or the typed text is not a number, an ordering matches nothing; Equals/Not equals fall back to text.

Empty vs zero: a cell is empty when its raw value is null, undefined, "", blank, NaN or an empty list. `0`, `false` and a literal "-" are values. A column drawn by `display()` or `render()` with no raw value is empty when it shows nothing or the dash.

## Not changed
No service or MCP read changes; the rows are already on the page.

## Verified at the head
- `GridFilterBehaviourTests` 15/15 (7 new). RED against unmodified `panels.js`: all 7 new tests fail (no operator list exists, so the operator harness cannot find it); the 8 existing pass.
- `GridAdoptionBehaviourTests` 7/7, `GridToolsBehaviourTests` 9/9.
- 132 classes (guard family, grid/panel/page classes, every class naming `panels.js` or a web harness): all pass except the expected platform ones (`LivePostgresCollectionHygieneTests`, `RawWindowFloorViewerPortTests` which needs WPF COM).
- hygiene-scan: 0 offenders. `node --check` on panels.js: clean.

## CHANGELOG entry
SECTION: Added
ENTRY: - **Filter web grid columns by number, text operator or empty cell** ([#PR]) - a column filter can now match greater/less than, equals, not equals, starts with, ends with, is empty and is not empty, comparing the real numbers rather than the formatted text; plain typing still does a contains match.
REF: [#PR]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/PR

## Why a time column offers no ordering operators

Checked against the desktop rather than assumed. `ColumnFilterMatcher.cs:97` implements the
four ordering operators through `CompareNumeric`, which parses **decimals only** — so `>` on a
`DateTime` column parses nothing and matches nothing in the desktop either. The desktop's popup
still lists those operators for such a column, so choosing one there silently returns no rows.

The web omits them on time columns instead. That is a deliberate, narrow divergence: it removes
an operator that cannot work rather than reproducing a dead control. If date ordering is wanted
later it needs a real date comparison in both apps, which is its own change.

* Tests: no test may pass by returning from a catch block (#5254)

Tests: no test may pass by returning from a catch block

A test of the form `try { setup(); } catch { return; }` is reported as a pass, not a skip, so where the setup cannot run it asserts nothing and still reads as coverage. A Windows run of a symlink test reported "0 skipped" this way, when creating a symbolic link needs a privilege an ordinary session lacks.

## What changed
- Six sites on dev had a `return;` inside a `catch`. Five were `Assert.Skip(...)` followed by a dead `return;` (xunit v3's `Assert.Skip` throws, so these already skipped honestly). The dead `return;` is gone, so a skip is the only exit. Where the compiler then needed `proc` assigned, the catch rethrows after the skip call. `WebModuleLoadTests` is a real test, so it got the same treatment. Three more tests with the same shape turned up when the new guard ran (`WebCollectionHealthRunAtColumnTests`, `WebDatabaseSizesNoteColumnTests`, `AzureSqlDatabaseHardwareTests`) and got the same fix. `StoreThatRefusesTheLogin_ExitsWithTheStoreCode` returns from a catch inside a listener helper loop, not from the test, so it is left alone.
- The certificate symlink test now proves the link resolves (`File.ResolveLinkTarget(..., returnFinalTarget: true)`) before asserting the refusal.
- New `ExportViewerConfigAsync_JunctionAtTheDestinationOrTheCertificatePath_RefusesWithoutWritingThrough`: a directory junction (`mklink /J`) needs no privilege, so this case cannot skip for want of a right. It asserts the junction exists and resolves, then that the export refuses and writes nothing through it, for a junction as the destination and a junction at `server.crt`.
- New `TestsNoSilentPassGuardTests`: scans both test projects and fails if any `[Fact]`/`[Theory]` has a bare `return;` inside a `catch`. The message names `Assert.Skip` / `Assert.SkipUnless`. It has a self-test of the scanner and no exemption list.

## Verified
- Build Release, 0 warnings, 0 errors.
- Guard family (90 classes) plus the touched classes: Total 1224, Failed 5, Skipped 58. The failures are the expected platform ones on this Mac: four `ResolveViewerExportDirectory_*` path tests (Windows paths) and `LivePostgresCollectionHygieneTests`. They fail on dev too.
- The new guard and its self-test pass. hygiene-scan: 0 offenders.
- Executed on the Mac: the guard, the Node-based web tests (Node is installed), the toggle exit-code test. Skipped on the Mac (DPAPI requires Windows): the certificate symlink test and the new junction test. CI on Windows decides those.
- Reading the product: a junction destination is refused at DarlingCliCommands.cs:1224, and a junction at `server.crt` has the ReparsePoint attribute, so it takes the symbolic-link refusal at :1672. Not run.

## CHANGELOG entry
SECTION: None
ENTRY: None: test-only change, no user-visible effect

* MCP: get_store_query_history reads the Darling store's hourly statement history (part of #5097) (#5217)

A new Darling MCP tool, get_store_query_history, reads the store's own hourly statement history (migration V163, #5203). Without query_id it ranks the statements that spent the most time over the window (Σ delta_total_exec_ms), each with its calls, window mean, the hours present, the role, per-statement counts of first_seen, restarted and reset hours, and the mean in the newest quarter of its hours against the rest (left out for first_seen hours, and null under four hours, so a lifetime credited to one hour cannot fake a slowdown). With query_id it returns that statement's hourly series, oldest first, with calls, mean, total, rows, block counts, temp writes, the cumulative max_exec_ms and the three flags. Both modes carry the window's captures (count, rebaselined and precondition hours, dealloc), so a missing hour reads as not captured rather than as idle.

Statement text is joined live from config.store_statement_stats(), the same reader get_store_query_stats uses; text the extension has since evicted reads as such, and when the reader is unusable now the history is still answered with "text unavailable" and a text_note carrying the reason. queryid goes out as a string. The answer carries effective_start, window_truncated and truncation_note (#4966), derived from the history's own first ok or rebaselined capture, with wording about the store's history rather than a monitored server. A store older than V163, one with no captures yet, and one whose captures are all precondition rows each answer a distinct precondition.

It is registered on the MCP host and mirrored on the web's /api/read. The tools/list ceiling rises to the measured 188,119.

Tests: refusals for bad role, top, hours_back and query_id; live cases planting the three tables (series with a reset hour, ranked order and truncation, the role filter, the slowdown split and its four-hour floor, a first_seen hour that would otherwise fake a slowdown, an unusable reader, each precondition, the data-start keys and the failed-read path); the budget, head-pin and cross-app inventory pins.

Part of #5097

* Web: the system-health, Blocking, Deadlocks and PostgreSQL event panels say where their data starts (part of #4966) (#5222)

The Darling web server page's partial-window note now covers fourteen more event reads, on data and empty answers alike: the nine get_health_parser_* reads (System Events), get_blocking, get_deadlocks and get_deadlock_detail, and PostgreSQL's get_pg_deadlocks and get_pg_log_events. An empty System Events panel over a window the store never covered used to show nothing, because the tool's own keys sat under hints, which the page does not read.

These are event lists, so the note follows the event-time rule from #5197: the earlier of the store's coverage start and the earliest event shown (the desktop viewer's ViewerEventDataStart.Of). A page cut by its row cap names its oldest row shown without asking the store (RowStatusByRead and the per-read oldest field). get_blocking is fed by the blocked process reports and the DMV blocking snapshots, so its probe asks both and the earlier start wins (CompositeSourcesByRead), as the MCP tool and the desktop viewer do. empty, no_deadlocks and no_events are admitted as "looked and found nothing"; unavailable with source_observed false is not.

Tests: a theory over the nine system-health reads (capped, uncapped, empty, unavailable), a capped page for each of the other five, the capped set pinned at twenty reads, source scans tying each listed read to its tool, and live cases with event times stamped apart from collection_time, including get_blocking with rows in only one of its two tables.

Part of #4966

* Lite.Tests: seed the #4966 window-notice classes in one transaction per test (#5208) (#5253)

Eight Lite.Tests classes (the five McpWindowNotice*ToolTests classes, DataStartBannerSurfaceTests, DataStartBannerQueriesTabTests and QueryWindowTruncationTests) now seed each test's store in one transaction instead of one auto-committed INSERT per helper call, which costs about 0.2 s a commit on hosted runners. A PendingSeedSession in SharedDuckDbFixture holds one SeedBatch per test and commits it before the code under test first reads; a seed that follows an act still commits before the next read. FloorHelper_ReadsTheArchivedFloor_NotJustTheHotTable keeps its own per-loop batches. Test code only: no production code and no assertion line changed.

Part of #5208

* Web FinOps storage growth: choose 7, 30 or 90 days (#5237) (#5264)

Web FinOps storage growth: choose 7, 30 or 90 days (#5237)

Fixes #5237. The desktop's object growth view covers 7, 30 or 90 days; the web view was fixed, and get_finops's storage_growth view refused any other hours_back.

- get_finops, storage_growth only: hours_back accepts the old default 24 (the 30-day heatmap, unchanged) or whole days 7 to 90 (168 to 2,160 hours); anything else is refused, and every other view keeps the shared 168-hour cap. window_days, the daily rate and the reader window follow the span.
- Windows over 30 days return at most 12 objects, and truncated says so: the widest case, 12 objects over 91 day columns, measures 28,974 bytes against the 32 KB response budget.
- The guide says the bands scale over the rows shown when fewer than 20 are shown, and that the daily rate is growth over the window divided by its days, the desktop's rule, so it understates on a store with less history than the window. The tools/list text is unchanged in size; the guide grows 80 bytes.
- The page (finops/storage-growth.js) adds a 7/30/90-day select on the objects level, sending days x 24, default 30 days, kept per server; the indexes level sends the same window.
- Tests: FinOpsStorageGrowthViewTests (the window rule, the cap, the 91-column worst case, every other view still refusing, the documented rate on short history, live 7/30/90) and FinOpsStorageGrowthWindowBehaviourTests (Node harness).

* Free the server-add slot before the answer, so an immediate second add is not refused (#5268)

Free the server-add slot before the answer, so an immediate second add is not refused

Refs #5167 (where the race came from).

## What users see change

Adding a server and then immediately adding another no longer answers 429 ("A server add is already running") for a slot that is already free. It also removes a flake from `ServerAddRouteTests.ASecondAdd_WhileOneIsRunning_Answers429_AndTheFirstStillCompletes`, which failed once on a Windows VM expecting OK and getting 429.

## What changed

`POST /api/servers` is serialised by a one-slot semaphore. The slot was freed only by a fire-and-forget continuation on the add task, so the release did not happen before the response. A client that read the 200 and posted again could beat that continuation.

- **Happens-before the answer.** After the wait ends (any way: answer, fault, timeout), a `finally` frees the slot if the add task `IsCompleted`. That runs before the handler builds its response, so the release precedes the 200.
- **Outlives the timeout (unchanged).** When the 120 s wait times out, the add task is not complete, so the `finally` does nothing and the slot stays held. Only the add's own continuation frees it, once the add finishes. A slow add that is still writing keeps blocking a second add.
- **Release exactly once.** Both exits go through a new `SingleReleaseSlot`, whose `Release()` does an `Interlocked.Exchange` on a per-request flag and releases the semaphore only on the 1 to 0 transition. I chose an exchange over an ordering rule because the handler and the continuation run on different threads and either can be first when the add finishes inside the wait. A second plain `Release()` is worse than the bug: if a later add had already taken the slot, it would free that add's slot and let a third add run beside it.
- **Synchronous throw path.** When `addServers` throws before returning a task, there is no continuation, so nothing else can release. It goes through the same `SingleReleaseSlot`, so it releases once.

## Tests (`ServerAddRouteTests`)

1. `AnAddThatFinishedInsideTheWait_FreesTheSlotBeforeItsAnswer_SoAnImmediateSecondAddIsNotRefused`: 400 back-to-back adds, each must be OK.
2. `AnAddThatFinishedInsideTheWait_ReleasesTheSlotExactlyOnce_SoALaterHeldAddStillBlocksTheNext`: 400 cycles of a finished add, then a held add (latched with `TaskCompletionSource`), then a third add that must get 429 and must not reach the core.
3. `AnAddThatOutlivesTheTimeout_KeepsTheSlotForEveryLaterAdd_AndFreesItOnce_WhenItFinishes`: uses the existing `addTimeout` (200 ms). The 503 returns, five further adds get 429, the core is called once; after the add finishes the next add is OK, then a held add blocks a further one with 429.

No test sleeps to prove a point. The only polling loop waits for the continuation of an add that has outlived the timeout, which is the path that is still asynchronous by design (same shape as the existing `ACoreThatNeverFinishes_...` test).

**RED against unmodified `DarlingWebEndpoints.cs`:** tests 1 and 2 fail (test 1 fails intermittently there: "add 323 of 400 answered 429 right after a finished add", 1 failure in 6 isolated runs, and 2 failures in the first class run of 3; test 2 fails in the same class run). Test 3 is a constraint pin and passes on the old code, as it should. Mutation checks on the fix:

- the handler releasing the semaphore directly (no once-guard): tests 2 and 3 and the existing `ASecondAdd_...` fail in 3 of 3 runs, and test 1 in 1 of 3;
- the handler releasing unconditionally (ignoring `IsCompleted`): test 3 and the existing `ACoreThatNeverFinishes_...` fail in 3 of 3 runs.

## Verified at the head

- Release build with `--no-incremental`: 0 errors, 0 warnings.
- `ServerAddRouteTests` (32 tests), 5 runs in a row: 5 of 5 passed, 0 failed.
- `OwnedSecretReferenceRefusalTests`, `DarlingOwnedSecretsCensusTests`, `DarlingWebHostServiceTests`, every Darling.Tests class naming `DarlingWebEndpoints` (96), the whole guard family (Census, Adoption, Discipline, Guard, Scrub, Hygiene, Ratchet, Convention, Contract, Pin), `DocCommentHygieneTests`, `RepoFileAdoptionTests`, `CommentFilterAdoptionTests`, `TsqlConventionGuardTests`, `FleetIdentifierScrubTests`, `CommandDeadlineScannerAdoptionTests`, `QueryStoreSliceTieBreakSourceTests`: 183 classes, 3019 tests, 1 failed, 120 skipped. The one failure is `LivePostgresCollectionHygieneTests`, a known platform failure here (it loads WPF types).
- `hygiene-scan.py`: offenders 0.
- Source-pin grep for `SingleReleaseSlot`, `addInFlight`, `ServerAddSlotTimeout` in Darling.Tests and Lite.Tests: no hits outside `ServerAddRouteTests`. No Lite.Tests class names a changed symbol.

## Not changed

The 120 s slot timeout, the 429 and 503 texts, the add core, and the `Release` semantics of the semaphore itself.

## CHANGELOG entry

SECTION: Fixed
ENTRY: - **Adding a server no longer refuses an immediate second add** ([#PR]) - the add slot is now freed before the answer is sent, so posting another add the moment the first one reports done no longer returns "A server add is already running". An add that outlives the 120-second wait still holds the slot until it finishes. Race introduced with #5167.
REF: [#PR]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/PR

## How this was found, and what confirmed it

Another seat reported this test failing once on a Windows VM during a **full-suite** run and
flagged it as a possible slow-box artefact rather than writing it off. It was not: the test is
already deterministic, using `TaskCompletionSource` latches rather than sleeps, so it was
reporting a real defect.

That seat then ran the class **three times alone on bare dev** and it passed every time, 29/0/0.
That is the expected result and it corroborates the diagnosis rather than contradicting it: the
window between an add completing and its fire-and-forget continuation running is narrow, so it
opens only under the memory and scheduling pressure of the whole suite. **A race that only
appears under load is the kind a single green run cannot clear.**

* Read Query Store plans through the plan map in analyze_query_store_plan, the Viewer and the actual-plan resolver (#5263)

analyze_query_store_plan, the Darling Viewer's Query Store plan (with its history window and repro script) and the Viewer's Get Actual Plan resolver find Query Store plans again (#5257). Since #2210/#2312 the collector ships query_store_stats.query_plan_text as a NULL placeholder and lands each plan once in query_plan_dim, reached through collect.query_store_plan_map; the three readers still read only the inline column, so every plan collected since then came back as "no stored plan".

With a plan_id (the Viewer, the web, the resolver), the readers look the plan up by the map's primary key joined to the dimension: two index lookups, no fact-table scan. The inline column is read only when the map has no row for that plan_id (plans from before the change); a map row with a NULL digest, the "this plan will never exist" marker, or a digest whose dimension row is gone, reads as absent. With only a query_id, the candidate plan_ids come from the fact rows of the last 7 days, newest first, each resolved as above, and one unbounded pass runs only when nothing resolves inside the bound. analyze_query_store_plan now names the plan it analysed (database:query_id:plan_id).

Measured on the store with the largest Query Store volume: the pinned read 0.98 ms; the 7-day candidates 535 ms cold; the unbounded fallback 22 ms warm over the same 12 compressed chunks.

Tests: a live class covering text and gzip plans, the NULL-digest marker, an orphaned digest, an older inline plan beside a newer mapped one, an 8-day-old plan reached by the unbounded pass, the resolver, wrong keys, and analyze_query_store_plan end to end asserting the analysed statement and its plan_id; plus a SQL parity pin between the Service and Viewer readers.

Fixes #5257

* Web: open a captured PostgreSQL plan's JSON from the Captured Plans grid (#5273)

The Darling web's PostgreSQL Captured Plans grid gets a Plan button on each row (#5229). It opens a panel under the row with the row's own stored, redacted auto_explain JSON, pretty-printed in a scrolling monospace block, with Copy, Download .json and a summary line (total cost and rows, and the actual time, rows, planning and execution times when the capture has them). A row with no plan shows a dash; a plan that is not valid JSON is shown as stored, with a notice and Download off. Open panels survive the grid's 60-second rebuild.

The JSON comes from the plan get_pg_plans already returns, so there is no new read, no MCP tool and no server code. The panel lives in its own module, pg-plan-viewer.js, separate from the SQL Server plan viewer, and renders everything as text nodes. The Plan column is left out of the grid filter, CSV and Copy, so an open panel's JSON never feeds a filter match or a table copy. The panel notes that the browser rounds the 64-bit Query Identifier inside the JSON; the grid's own queryid is the exact one.

Tests: a Node harness over the shipped module (pretty-print of object, string and null plans, the summary, file names, the open-key across a rebuild, text-only rendering) and page pins, including a comment-stripped scan for HTML sinks.

Fixes #5229

* MCP: quiet and all-clear messages no longer contradict a cut window (part of #4966) (#5250)

MCP quiet and all-clear empty messages no longer contradict a cut window, in Lite and Darling alike (#4966). When an answer's hints say the window reaches back before the store's data, the collection log's unfiltered empty answer, get_blocking_stats, the first two empty rungs of the health-parser tools, the Query Store regression all-clear and the heatmap's idle grid no longer call the window "genuinely quiet" or "the all-clear".

McpHelpers.QuietUnlessCut (PerformanceMonitor.Common) picks the text from the answer's own window notice. A covered window keeps today's text byte for byte. A cut one keeps the factual first part (counts, the threshold applied, the filter echoed) and replaces only the claim, with one of two shared sentences: one naming effective_start when the store's data starts inside the window, and one saying nothing was read when the store holds nothing for the read in the window. The filtered collection-log empty and the filtered heatmap miss claim nothing about the window and are unchanged.

Both apps' server instructions gain one sentence on window_truncated, effective_start and truncation_note. Darling's room came from tightening the PostgreSQL wait-instrument paragraph; the shared not_collected / precondition paragraph and every pinned sentence are unchanged. No tool description changed, so the tools/list budget is unchanged.

Tests: McpMissMessageParityPinTests holds both cut sentences, the covered text and the instructions sentence across both apps, plus QuietUnlessCut's truth table; each site has a cut case and a covered case in both apps, and each cut variant has its own case.

Part of #4966

* Web charts: hide, show and isolate a series from the legend (#5265)

Web charts: hide, show and isolate a series from the legend

Fixes #5247
Part of #4843

## What users see change
On every web line chart (area and stacked too), a legend entry is now a switch.
- **Click** an entry to hide or show that series. A hidden entry stays in the legend, struck through with a hollow swatch and `aria-pressed="false"`, so it reads as hidden and not as missing data.
- **Double-click** (or Shift+click) an entry to show only that series. Doing it again on the lone visible series shows everything.
- **Show all (N hidden)** appears after the entries whenever anything is hidden, and restores every series in one click.
- The y-axis (and the stack, for stacked charts) rescales to the visible series, so hiding one dominant series gives the rest the vertical space.
- Hidden state survives the 60-second rebuild and is held per chart, per server/tab and per range. It does not carry to another server or tab.
- The last visible series cannot be hidden by a toggle.
- Legend entries that drill (`onSelect`) keep that: the label drills, the swatch toggles.

## What changed
- `charts.js`: `renderLineChart` takes `hiddenKeys` and `onLegend`; hidden series are filtered out of the y-domain, stack, drawn marks, hover and CSV. `buildLegend` draws the switch/marked/Show-all state. `nextHiddenKeys` holds the toggle/isolate/all rules. `getChartHidden`/`setChartHidden` keep state at module scope keyed by chart id and the same scope string the zoom uses (page address without query, plus range hours); `zoomableLineChart` wires them and redraws.
- `app.css`: legend switch, hidden-entry and Show-all rules beside the existing legend rules.
- Isolate follows the desktop's click-isolate plus double-click revert idea; the web uses double-click to isolate because single click is already hide/show.

## Export and copy
Export Data to CSV carries exactly the shown series, matching the grid's rule that export carries the shown rows. Copy/Save Image is a picture of what is drawn, so it shows the shown series too. Pinned.

## Not changed
No MCP read, field or tool text changed. Bar, pie and scatter charts are untouched. Composed panels (compose.js) are wired through the same state: the hidden series are held under the panel's existing composed id (its view slot, its id/title/measure and the server) and the page address plus range, so they survive the 60-second repaint and do not reach another panel, server or range. The editor preview has no slot and so no stable identity; it gets no switch rather than a key that could collide.

## Tests
New `WebChartLegendIsolateBehaviourTests` (10, Node harness `web-chart-legend-harness.mjs`): toggle hides/re-shows; legend marks hidden; axis rescales to visible series (2000 to 5 top tick; stacked 3000 to 8); isolate shows exactly one and a second isolate restores; show all; last series kept; state survives rebuild; no leak to another server, range or chart; CSV carries shown series.
RED against unmodified `charts.js`: all 10 fail (the harness cannot find a legend switch).

## Verified at the head
- New class: 10/10 pass on the change, 10/10 fail on dev's `charts.js`.
- 167 test classes (every class naming charts.js or a chart harness, `WebModuleLoadTests`, `ServerPageTabsTests`, the guard/census/adoption/ratchet/convention family): Total 1251, 1 failed, 14 skipped. The failure is `LivePostgresCollectionHygieneTests` (WPF types, expected on macOS).
- `TestsNoSilentPassGuardTests` does not exist on this base.
- Release build 0 errors, 0 warnings; `node --check` clean on charts.js; hygiene-scan offenders: 0.

## CHANGELOG entry
SECTION: Added
ENTRY: - **Hide or isolate a series from any web chart legend** ([#PR]) - click a legend entry to hide or show a series, double-click to show only that one, and use Show all to restore; the axis rescales to what is visible, and the choice survives refreshes for that server and tab.
REF: [#PR]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/PR

## Where the feature is absent, and why that is deliberate

A composed panel with **no slot** - the editor preview at `editor.js:1065` - gets no legend
switch. There is no stable key for it, and a guessed key is worse than no feature: it would
make one preview's hidden series apply to another. Every other chart path is wired, through
either the stateful wrapper in `charts.js` or `composedPanelId()` in `compose.js`.

## One check that was NOT run

`TestsNoSilentPassGuardTests` does not exist in this branch's tree. It arrives with a separate
PR that is in a merge train at the time of writing, so this branch cannot run it. That guard
fails the build if any test passes by returning early from a `catch`; when it lands, this
branch's tests are subject to it like any other, and nothing here uses that shape.

* Web Collection Health: click a collector to see its run history (#5227) (#5258)

Web Collection Health: click a collector to see its run history (#5227)

Fixes #5227. The desktop opens a collector's last 7 days of runs from its Collection Health row; the web showed the Collectors summary and the Collection Log side by side but unconnected.

- The SQL Server Collection Health tab's Collectors rows are clickable and keyboard-activatable (the shared makeActivatable helper, now exported from util.js); a click that ends a text selection is ignored.
- A pick re-reads the Collection Log for that collector: get_collection_log with collector_name and hours 168, the desktop's 7 days. The subtitle and empty text name the collector, and a full-width "collector X x" chip clears it. The previous read is aborted on each pick, so quick clicks never stack 7-day reads.
- The pick is kept per server at module scope, so the tab's 60-second refresh keeps it; only the Collection Log redraws. No new read and no MCP change; the PostgreSQL Overview's collector pair is unchanged.
- Tests: WebCollectorRunHistoryTests (Node harness): clickable rows, the re-read with collector_name and 168 hours, the chip, the pick surviving a rebuild, a second pick aborting the first read, Enter picking a row, and a selection click being ignored.
- The harness copies every module under js/ and js/pages/ rather than a named list, so a page module another change adds and server-tabs.js imports needs no edit to it, in either landing order.

* Remove two agent-bus files committed by accident, and guard against it (#5271)

Remove two agent-bus files committed by accident, and guard against it

Fixes a public-repo hygiene miss of mine, and guards it.

## What happened

#5167's squash (`a1c7fdcb3`) committed two files from the agent coordination bus into this repository:

- `design/pm-pr-worker/i4843/lane-reports/SRV-WRITE-1-pr.md`
- `design/pm-pr-worker/i4843/lane-reports/SRV-WRITE-1-ownedset.md`

**This repository is public.** Scrubcheck reports **0 hits** on both — no customer, tenant, host or account identifiers — but they carry internal vocabulary and the directory layout of a private tree, and on `dev` they merge into every branch and every clone. Another seat found them while merging dev into their own branch.

## The cause was a path, not a judgement

A lane brief named a bus path that was not absolute. **A relative path resolves inside the CODE worktree**, so a report written "to the bus" landed here instead. That trap is why briefs in this project say "absolute path — a DIFFERENT repo from the code"; this one slipped through.

## Three parts, because the first two are not enough alone

1. **`git rm`** the two files.
2. **`.gitignore`** for `design/`, `inbox/` and `outbox/`, with the reason written above the lines so a later reader does not delete them as noise.
3. **`NoInternalBusFilesGuardTests`**, because `.gitignore` cannot stop a forced add. It fails if any file exists under those three directories, and its message names the offending files **and** states the absolute-path rule that was broken, so whoever trips it learns the cause rather than just the symptom.

It lives in `Darling.Tests` rather than `Lite.Tests` deliberately: **Lite.Tests cannot build or run on a Mac**, so a guard placed there could not be verified by the seat that needs it most.

## No history rewrite, and the reasoning is explicit so it can be overruled cheaply

With 0 scrub hits there is no customer exposure — only internal vocabulary and a layout. Rewriting a public repository's history would break every open PR, every clone and the nightly's provenance, which is a **larger harm than the one it would undo**. Removing them from the tip is proportionate. If you disagree, say so and I will plan a rewrite properly rather than improvise one.

## Verified

- Guard **GREEN** with the tree clean.
- Guard **RED** with a bus file planted back, printing the file list and the rule.
- Guard **GREEN** again after removing the plant.
- Release build **0 warnings, 0 errors**.

## CHANGELOG entry

SECTION: None
ENTRY: None — repository hygiene and a test guard; no shipped behaviour changes.

* Web Manage Servers grid: show disabled servers, auth mode, monthly cost and added time (#5239) (#5262)

The web Admin page's Servers tab now shows every configured server, enabled or disabled, with the columns the desktop Manage Servers window has: Auth, Status, Monthly Cost and Added. Before this, the tab read `list_servers`, which filters to enabled servers for every caller and carries no auth, cost or added time.

**Route.** A new web-only route, `GET /api/admin/servers`, mapped for GET only. `list_servers` is untouched.
- **Rows.** They come from the configured servers (enabled and disabled), left-joined to the collected server row and the newest collection. A configured server that has never connected shows its config fields, with an awaiting-first-collection freshness.
- **Display-ready values:**
  - `status` is Enabled or Disabled;
  - `auth` is "SQL Server" or "Windows", or any other mode verbatim;
  - `monthly_cost` is `$1,234` style (invariant culture, halves away from zero, always formatted, as on the desktop);
  - `added` is the config row's creation time;
  - the display name is the config name, so a rename shows at once.
- **Order.** The display name, then the server name, then the server id.

**Gate and credentials.**
- **Gate.** The route sits behind the same sign-in gate and method gate as every web route: a read-only sign-in may GET it, and an anonymous request is refused.
- **Credentials.** The statement names its columns, none is a credential, and every config column it reads is in the viewer role's column grant. The response has no credential key. It adds nothing to the MCP tools/list budget.

**Page.** `SERVER_COLUMNS` gains Auth, Status, Monthly Cost (sorted on the number) and Added. The old status column becomes Freshness, and disabled rows are greyed.

**Tests.**
- **Statement pins:** the config table with no enabled filter; no credential column or wildcard; every column inside the viewer grant.
- **Unit tests:** the labels and the order.
- **A live test:** the route through a real host, as a non-superuser holding the product's own viewer grants replayed.
- **The real pipeline:** anonymous refusal.
- **A rename test:** a renamed config row shows its new name before it reconnects.
- **Page harness updates.**

Fixes #5239

* Web Alert History: show an alert's advice and fix script, as the desktop does (#5241) (#5269)

Web Alert History: show an alert's advice and fix script, as the desktop does (#5241)

Fixes #5241. The desktop's alert detail shows an analysis alert's advice and T-SQL with Copy T-SQL; the web showed neither, because AlertDetailText.Flatten writes only headings and fields and drops each item's body, so the advice and the script never reached detail_text.

- New web-only read get_alert_details(server_id, metric_name, alert_time), keyed on the page's own row identity, returns one alert's details [{heading, fields, body, is_code_block}] parsed from its stored context with the desktop's model. No MCP tool behind it (an internal method in the /api/read catalog only, listed in WebOnlyReadNames for the parity census); an unknown or unparseable key answers empty, a missing parameter is refused, the lookup is one indexed row, and Remediation (the apply payload) is never projected.
- get_alert_history (its MCP tool, web catalog entry and answer) is unchanged.
- alerts.js: the first time an Analysis alert's detail is opened, one get_alert_details read with a loading strip, then heading and fields, the advice as a paragraph and code as 
 with a Copy button; cached per alert so the 60-second refresh never refetches; other alert kinds make no read; an error or empty answer falls back to detail_text. The expansion stays lazy, as #4194 made it.
- Tests: AlertHistoryAdviceLiveTests (advice and script returned, unknown key empty, missing parameter refused, no Remediation, the history read unchanged) and AlertHistoryAdviceTests (Node harness: one read on first expand and none after, other rows none, the fallback, hostile text rendered as text).

* Add a --diagnostics-bundle verb that writes one aliased, attachable diagnostics file (#5097) (#5218)

Adds `--diagnostics-bundle ` (short form `--diag-bundle`), a read-only CLI verb that writes one aliased JSON file for a user to attach to a bug report. It changes no schema, writes nothing to the store, and adds no MCP tool and no web change.

**Usage:** `--diagnostics-bundle  [--hours N] [--server ] [--log-dir ] [--config ] [--alias-map ] [--include-log-text] [--force]`.

**Exit codes:**
- 0: written.
- 1: bad arguments, config or connection string.
- 2: store unreachable; a reduced bundle is written.
- 3: written with a failed section.
- 4: output unusable or over the size cap.
- 5: a known name or secret survived, or a store name source could not be read. Nothing is written.

**Sections.** The bundle holds these sections, each read through the same MCP reader the tools use:
- the manifest and config shape;
- the store;
- collection health, the fleet overview, the slowest runs and collector cost;
- stall probes, slow reads and read latency;
- store statements, including the V163 statement history, behind `to_regclass`;
- the store log and the service log.

**Format.** The file is one UTF-8 JSON file of at most 4 MiB, with per-section byte budgets. It is written to a random temp name in the target directory and moved into place only after the verifier passes.

**Aliasing.**
- **What gets aliased.** Server, host, database, login and domain names and IP addresses become ordinal aliases (`server-1`, `db-3`, `ip-1`), never hashes. `server_id` becomes `server_alias`.
- **Where the names come from.**
  - the registry and the collected `servers` table (including removed servers);
  - darling.json, including the notification hosts and domains;
  - the store connection string, `pg_roles` and `pg_database`;
  - the local machine, user-domain and DNS names, and the Windows service account;
  - a bounded 24-hour inventory of database names;
  - a harvest of identifier-keyed fields.
- **Where the alias pass reaches.** It rewrites string values, and any key that is itself a known name. It handles:
  - protocol prefixes (`tcp:`, `np:`);
  - quoted `DOMAIN\login` pairs;
  - IPv4-mapped IPv6;
  - names under a known domain.
- **Order.** Text is aliased before it is cut.
- **The verifier.** It then scans the final bytes and the unescaped text for every known name and secret, and exits 5 if any survives.
- **Store down.** The service log is written as counts only, unless `--include-log-text` is passed. The unreachable-store section carries only fixed codes.
- **The alias map** is opt-in, written to a separate file marked DO NOT ATTACH, and never put in the bundle.
- **Fail closed.** While the store is up, a failed read of any name source (the registry, `servers`, `pg_roles`, `pg_database`, the database inventories) writes nothing. An `--alias-map` path with a part that exists but cannot be inspected is refused.

**Secrets.** The slow-read log's key and value checks move into a shared `SecretTextGuard`, with unchanged behaviour. darling.json is never serialized: the bundle holds an allow-list projection.

**Template.** `bug_report.yml` gets an optional diagnostics-bundle field, and `Darling/README.md` gets a section.

**Tests.** Thirteen new Darling.Tests classes, including:
- `DiagnosticsBundleLeakLiveTests`, which seeds synthetic names into every source and asserts that none survives in the bytes, case-folded or JSON-unescaped;
- the seed-failure, uncut-health and round-two hardening classes.

**Known limits:**
- names and secrets of under four characters are boundary-replaced but not substring-verified;
- base64, URL-encoded and nested-escape forms are not decoded;
- text already cut when it was stored cannot be repaired.

Part of #5097

* Stall probe: an idle wait (sp_server_diagnostics, the availability-group health check) no longer ranks as the top user-task wait (#5267) (#5276)

The collector stall probe no longer ranks an idle wait as the top user-task wait. On an availability-group server, the probe added in #5113 ranked `SP_SERVER_DIAGNOSTICS_SLEEP` first. That is the idle wait of the `sp_server_diagnostics` health check, which runs on an ordinary (`is_user_process = 1`) connection, so the probe pointed at the health check instead of at what the collector was waiting behind.

**What changed.**
- **Ranking.** It now leaves out every wait type in the shared ignored-wait list (`IgnoredWaitDefaults.All`), as well as background sessions. The list is rendered into the query once, as a per-task `is_idle` flag that both the totals and the ranking read.
- **Tail.** The summary tail counts what was left out: `excluded from ranking: background N tasks/M types; idle K tasks/J types; ours: …`. It is still reserved before the 512-character cap.
- **No top wait.** When only idle and background waits remain, there is no top wait.
- **Unchanged:** `waiting_task_count` still counts every waiting task. The collector's OWN session wait is still named in the "ours" part of the tail, even when it is an idle-listed type.
- **Fault isolation.** The query text is built on first use, so a malformed list entry fails only the probe (recorded as a failed probe query), never the policy class every collection run consults.
- **Tool guide.** The `get_collector_stall_probes` guide text says that idle waits are not ranked, and that a tail without the idle counts comes from a build that still ranked them. The tools/list text is unchanged.

**Tests.**
- **Live, against SQL Server:** a real `sp_server_diagnostics` session plus a lock wait. The lock wait ranks first and the idle wait is counted in the tail. The total covers the background, idle and ranked buckets.
- **Darling unit pins:**
  - every ignored-wait name is a plain upper-case identifier;
  - the rendered query carries exactly the list;
  - the policy loads independently of the query text.
- **Lite policy tests:** the query shape, the tail and a null top.

Fixes #5267

* MCP: edit_server changes a monitored server's settings in place, keeping its id, tags and history (#5240) (#5283)

A new MCP tool, `edit_server(server_name, changes_json)`, changes one monitored server's settings in place. Until now, the only way to change a server's settings was to remove it and add it again, which lost its settings and its tags.

**Identity is kept.** The row keeps its `server_id`, so tags, collected history and every per-id setting stay attached, even when the address changes. The tool resolves `server_name` with `remove_server`'s rule. The core, `EditServerCoreAsync`, is separate from the tool, so the web route can call the same code.

**Editable:** `display_name`, `host`, `port` (PostgreSQL), `database`, `read_only_intent`, `auth`, `username`, `password`, `encrypt_mode`, `trust_server_certificate`, `multi_subnet_failover` and `monthly_cost_usd`, plus an optional `expected_modified_at`.

**Never written:** `server_id`, `engine`, `is_enabled`, `excluded_databases`, `capture_plans`, `alert_delivery_mode_override`, `plan_force_bot_enabled` and the `remediation_*` columns. A request naming any of them is refused, and nothing is read or written.

**Credentials.**
- The stored secret cannot be read back. So a change to how a SQL-login or service-principal server is reached requires the password again, and the connection is tested before anything is saved.
- Name and cost edits, and Windows or managed-identity servers, need no password.
- Secret validation is shared with `add_servers`, whose behaviour is unchanged. It refuses a literal secret off Windows, and a reference to the service's own configuration or secrets.
- No answer, error or log line carries a secret. The audit line names the changed fields only.

**Concurrency.** The edit reads the row, tests the connection, then in one transaction locks the row, re-checks `modified_at` (the optimistic token, an opaque round-trip string) and re-checks that the new address is not another server's. Then it issues one column-listed UPDATE. A stale token or a concurrent change answers `conflict`, with the current non-secret values.

**Answers.** `updated` (with the changed fields, whether the service will reconnect, and the new token), `unchanged`, `invalid`, `not_found`, `ambiguous`, `conflict`, `collides` or `connection_failed`.

**Also fixed.** The service's server-definition comparison now includes the port and the engine, so a port-only change reconnects.

**Budget.** It adds one tool, a tools/list increase of about 1 KB. The ceiling is raised to the measured value.

**Tests.**
- Core unit tests.
- Live tests as the MCP role, with the product's own grants: name, cost and address edits; tags kept; config-version bumps; a stale token; occupancy; the credential column still unreadable.
- A live SQL Server connection test: a correct and a wrong secret.

Part of #5240

* Darling web: draw the deadlock graph in the Blocking tab's Deadlock Graphs grid (#5246) (#5275)

The Darling web's Blocking tab draws each deadlock as a graph (#5246). Each Deadlock Graphs row has a Graph cell; opening it draws an SVG with one card per process (SPID, procedure, contended object, lock mode and wait, the victim marked), curved waits-for arrows labelled with the resource and lock mode, and dashed frames when there is more than one cycle. A click or Enter on a card shows that process's properties and statement as text; the first victim is selected on open.

The service builds the shape with the shared DeadlockGraphParser and DeadlockGraphLayout from PerformanceMonitor.Common, the code the desktop viewers use, so positions, edges and cycle frames match the desktop. The shape is added only in the page's read of get_deadlock_detail (DarlingWebDeadlockGraph.AddGraphs); nothing that serves an MCP client calls that handler, so the MCP tool's payload, budget and Lite parity are unchanged. The browser never parses XML, and every string reaches the page through text nodes. A statement equal to the process row's preview is not repeated; a row whose XML fails to parse or lay out gets no graph and the rest of the read is unaffected; a deadlock of more than 60 processes shows a count instead. The Graph column stays out of sorting, the column filter, CSV and Copy, and the Custom Views column picker now offers scalar fields only.

Tests: the shape against Parse plus Layout on four fixtures (victims, the parallel self-edge, the desktop's cycle-frame rule, malformed, truncated, oversized and envelope pass-through, every other key unchanged), a Node harness over the shipped module (lazy draw, labels, selection, hostile text, multiple cycles, the self-loop, open state across a rebuild), and pins that no MCP-serving path reaches the graph-adding row.

Fixes #5246

* Bump the nuget-patch-and-minor group with 1 update (#5219)

Bump the nuget-patch-and-minor group with 1 update

Updated [BlackwellSystems.Gcf](https://github.com/blackwell-systems/gcf-dotnet) from 1.0.0 to 1.1.0.

Release notes _Sourced from [BlackwellSystems.Gcf's releases](https://github.com/blackwell-systems/gcf-dotnet/releases)._ ## 1.1.0 Implements GCF spec v3.6.0: constant-column factoring (SPEC 7.4.7, mandatory in the default encode path) and value-grouping (SPEC 7.4.8, opt-in `EncodeGenericGrouped(data, keyField, groupField)`). Decode support for both is mandatory. A column holding the same scalar in every record of a top-level tabular array is written once in the header as `name=value` and dropped from the rows; value-grouping clusters a keyed set by one low-cardinality column (`@`-marked unique key, `group=` header clause, per-group `= [count]` subheaders). Byte-identical to the Go reference. Conformance 323/323; per-SDK fuzz verified to 1,000,000 iterations; cross-SDK differential-fuzz green. See CHANGELOG for details. **Full Changelog**: https://github.com/blackwell-systems/gcf-dotnet/compare/v1.0.0...v1.1.0 Commits viewable in [compare view](https://github.com/blackwell-systems/gcf-dotnet/compare/v1.0.0...v1.1.0).
[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=BlackwellSystems.Gcf&package-manager=nuget&previous-version=1.0.0&new-version=1.1.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore ` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore ` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore ` will remove the ignore condition of the specified dependency and ignore conditions
* Add plan buttons to the web's Active Queries, Query Store and Top Procedures grids (#5228) (#5260) The web's Active Queries, Query Store and Top Procedures grids get a Plan column, and Active Queries also gets a Live plan column. Clicking a row's Plan button opens the plan viewer on that row's stored plan. When no plan was stored for the row, the panel says so. - The plan is read on click, not shipped with the grid rows. Every web read is an MCP tool, so the buttons call three new read tools, one per key shape. get_active_query_plan_xml takes collection_time, session_id and an optional request_id, and live=true returns the live plan. get_query_store_plan_xml takes database_name, query_id and an optional plan_id. get_procedure_plan_xml takes sql_handle. Each answers unavailable when nothing matches. - get_query_store_plan_xml resolves the plan the way analyze_query_store_plan does: through the plan map, then the inline query_plan_text column for rows written before the plan dimension. - The web route refuses a missing or unparseable key. It also refuses a collection_time that is not an ISO 8601 round-trip form, so a bare date is refused. It refuses a sql_handle that is not 0x plus hex, and a live that is not true or false. collection_time round-trips at microsecond precision. get_procedure_plan_xml refuses a non-hex handle too. analyze_procedure_plan still takes the text as given. - get_active_queries rows now carry request_id, plus has_query_plan and has_live_query_plan. The flags are left out of a row when false, and a null request_id is left out too. None is ever written as null. - Plan columns are not sortable, filterable, exportable or copyable, because an open panel's XML is the cell's text. That covers the three new columns. It also covers the existing Plan column on Top Queries, High Impact and Expensive Queries. That column gains the filter and copy exclusions and was already out of sorting and CSV. - tools/list: the three tools add 1,885 bytes and edit_server (already on dev) adds 1,052. Measured on the merged tree, the total is 191,056 bytes for 177 tools, up from 188,119 bytes for 173. The tool-count sentences in the server instructions, README.md and llms.txt say 177 (89 unique to Darling). - Tests: a live RowPlanReadLivePostgresTests, a WrapPlanXml identity test, dispatch binding refusals, plan-viewer harness scenarios, and updated cancellation and timestamp-parse pins. The tools/list ceiling is set to the measured value. Fixes #5228 Part of #5236 * MCP: stop calling a cut window quiet on eight more PostgreSQL empty answers; mirror Lite's plan-corrections list-only notice (#5285) Eight PostgreSQL MCP empty answers said "healthy", "expected" or "genuine all-clear" even when `hints.window_truncated` was true. Over a cut window they now keep the factual first part and then say where the store's data starts, or that nothing was read. Covered windows read exactly as before. The eight sites are `get_pg_lock_stats`, `get_pg_server_config_changes`, `get_pg_wait_sampling`, `get_pg_replication_stats`, `get_pg_xmin_horizon` (the `finding` of `no_holder`), `get_pg_database_stats`, `get_pg_session_states` and the index-usage empty branch. That branch is kept for later and is unreachable today. Sites that claim nothing, are filtered, or carry no notice are unchanged. Lite's `get_plan_corrections` now probes the store. When the recommendation list is empty beside an automatic-tuning snapshot and the store holds nothing in the window, it says "no windowed rows were read". The wording is byte-identical to Darling's `listOnly` branch, and `RawWindowFloorQuietStartTests` pins that. `get_pg_deadlocks` passes its defined `windowStart` to the reader. The value is the same as before. The window-notice tests in both products are extended: Darling's PostgreSQL window classes, `RawWindowFloorQuietStartTests`, and Lite's `McpWindowNoticeToolTests` and `McpWindowNoticeConfigAndLogToolTests`. Test lows from the previous round are fixed too. The idle-grid covered tests assert the idle claim, the cut-claim tests assert the literal variant, and one helper is renamed. Part of #4966 * Web Job History: say when SQL Agent is stopped (#5242) (#5278) The service collects each server's SQL Agent state into agent_status, and the Agent Not Running alert reads it, but no MCP or web read did, so the web Job History page showed a quiet server and a stopped Agent the same way. - get_job_history carries the Agent state. Each enabled server's row is its newest agent_status row at or before the window's end (the as_of, or now), one indexed lookup each. One server gets agent_running, agent_status_desc, next_run (server-local, converted to UTC) and captured_at. The fleet gets agents_total, agents_running and agents_not_running (stopped or unknown only). An empty answer carries the same fields in its hints. The rest of the answer is unchanged. - A snapshot older than the alert settings' live collection_stale_minutes at the window's end reads unknown. The window is clamped 5-1440 (30 if absent), the Agent Not Running alert's own window and comparison. A named server with no snapshot at or before the window's end reads unknown too, and the fleet counts leave it out. A fresh snapshot with no Agent service row (Express, Agent-less containers) reads "no SQL Agent service found" and is not counted as stopped. - The Agent read is best-effort: if it fails, the runs are still served without the Agent fields. - job-history.js: a status line under the head (stopped in red, no service, running, unknown, and "Agent running on n of m servers" naming the stopped ones). The page harness copies every module under js/ and js/pages/. - Tests: DarlingMcpJobHistoryToolTests and JobHistoryPageTests. The surface and live cases cover stopped, running, stale and no snapshot, plus the live setting at 60 min. They also cover no service, next_run conversion, a failed read, fleet counts and empty hints. Three new live cases cover as_of between a stopped and a running snapshot (a named server, then the fleet) and as_of before every snapshot. The older cases are seeded off their fixed as_of. Fixes #5242 * procedure_stats: wire the deferred plan fetch, with a shadow mode that measures the identity before it is trusted (#5158) (#5272) procedure_stats can skip re-rendering module plans the store already holds, behind a new darling.json setting, `procedureStatsDeferredPlanFetch`. It takes `off`, `shadow` or `on`, and defaults to `off`. On one production store, about 94% of the procedure_stats plan XML bytes rendered were repeats: 25 GB an hour rendered against 1.4 GB distinct. **Identity.** A module plan can change in place when one statement recompiles, with the same plan handle and cached time. So the cache key is the server, the database, the plan handle, `cached_time` and a fingerprint of the module's statements from `sys.dm_exec_query_stats`: the count, the latest compile time and the summed plan generation. A count of zero is never cached. **Modes.** - `off`: today's inline capture. - `shadow`: renders inline exactly as `off` and writes identical rows. It also checks each plan's identity against what this host has committed, and counts the would-be hits and the false hits, so the identity can be measured before it is trusted. - `on`: sends the stored digest for a recognized plan and renders only the misses, in a second query on the same connection. At most 150 are rendered per run. An entry older than four capture cycles is rendered again. - **In every mode:** the existing cadence gate still decides which runs may render. A recognized over-cap plan keeps its size and its oversized-plan sightings. Azure SQL Database stays `off`. The setting takes effect on restart. **Counters.** Each run's collection-log note carries `plans_rendered`, `plans_rendered_bytes` and `deferred_miss`, plus `deferred_would_hit` and `deferred_false_hit` in shadow, and `deferred_hit` and `deferred_over_cap` in on. A false hit logs a warning naming only the plan handle, the cached time and the fingerprint values. **Shared code.** The run's mode is resolved once, when the run is stamped, and read back everywhere else. It reuses the query_stats plan cache (`PlanDigestCache`, with an optional rendered-run ordinal for the age limit) and its `StampPlanFetch` measurements. **Scope.** No migration, no store schema change, no MCP tool and no web change. **Tests.** - Unit tests: key sensitivity, the age limit, zero-count, the gate composition, the cap, over-cap, shadow false hits, knob parsing, Azure off, and mode resolution under changing settings. - Live tests against SQL Server and PostgreSQL: reuse, a statement recompile in place forcing a re-render, the shadow counters, and shadow storing every column exactly as `off` does. Part of #5158 * get_collection_health and the web PG Overview list collectors that are not collected on the server's engine (#5249) (#5287) `get_collection_health` and the web PostgreSQL Overview now list each PostgreSQL collector that the server's engine kind can never run, instead of leaving it out. A PostgreSQL user could not tell a collector that failed to run from one that is not collected on the engine at all, because a gated collector writes no log row. Examples are an Aurora-only collector on stock PostgreSQL, and the sampler on Aurora. **What changed.** - **New rows.** For a PostgreSQL server, the tool appends one row per catalog PostgreSQL collector that `CollectorEngineCapability` reports as not collected on this engine kind and that has no log row. Each row is `collector`, `status: "not_collected"` and `message` (the standard sentence). The rows come after the log rows, ordered by collector name. - **Output only.** They never reach the memoized rows, sweep pressure, alert read health, counts, bands, the analysis caveats or `stored_caveats`. SQL Server servers are unchanged. A PostgreSQL server with no log rows still answers `unavailable`. - **One helper, one call.** The logic lives in one helper called from one line in the shared health read. So the diagnostics bundle's collection-health section carries these rows too, with the server name in the message aliased like any other. - **Page.** The PG Overview's Collectors panel shows the new status in a neutral style, with a "Why not collected" column. The generic read catalog gains the `message` field, for custom views. **Budget.** No tool description or parameter changed. The response grows by at most about 1.1 KB (two rows on stock PostgreSQL). **Not covered** (a follow-up): collectors skipped by PostgreSQL version (`pg_io_stats` below 16, `pg_write_stats` below 14) and by recovery state still show no row. **Tests.** - Unit tests of the helper. - Live tests through the real tool against a scratch store. - The roll-up identity checks. - The order. - A page harness. - A diagnostics-bundle leak test proving a gated row's server name is aliased. Fixes #5249 * lane-orders: the PostgreSQL rig recipe also creates the darling database, as CI does (#5292) Step 5 of the rig recipe created the suite and probe databases but not `darling`. CI's init step creates it, and the role-provisioning live tests grant CONNECT on it. Without it, five server and viewer live classes failed with 3D000 on a rig built from the recipe. The step now runs the same createdb call as CI. Docs only: no product code or test changed. * Web charts: show the queries, blocking or deadlocks at the clicked time (#5230) (#5286) The web viewer's server-tab line charts had no way to jump from a point on a chart to what was running then. The desktop's Performance Trends charts do this from a right-click. Now a right-click on the plot of a server-tab line chart offers the one "at This Time" item that matches the chart. The item sets a one-hour custom range around the nearest drawn point. It then opens the matching tab, scrolled to its grid, and the range picker shows the new range. - Each chart gets one item, as on the desktop. Blocking Events, Lock Waits and Blocking Severity get Show Blocking at This Time. Deadlocks and Deadlock Severity get Show Deadlocks at This Time. Every other server-tab line chart gets Show Active Queries at This Time. - The range is 30 minutes either side of the drawn point nearest the click, which is the point the hover tooltip names. That makes one hour, the shortest range the picker allows. - A range cannot end in the future, so a click in the last half hour moves the hour back to end now. The hour still includes the clicked time. A range the page refuses is shown on the chart, and the tab stays. - The items appear only on a right-click on the plot, not on the legend, the status line, the chart menu button or Shift+F10. - Show Deadlocks at This Time scrolls the Deadlocks grid into view after the tab opens, as the desktop opens its Deadlocks sub-tab. Show Blocking and Show Active Queries scroll to their own grids. - The PostgreSQL Aurora CPU chart has no item, because the PostgreSQL page has no Queries or Blocking tab. - Tests: WebChartAtTimeBehaviourTests (Darling), plus BaselineDiscontinuityRenderCensusTests (Lite), which scans server-tabs.js. Fixes #5230 * Balance the Lite test shards by recorded test time instead of class-name hash (part of #5208) (#5280) The four Lite test shards used to be assigned by class-name hash. That balances the class count, not the time. On one measured run the shards held 12.8, 48.1, 36.6 and 25.3 minutes of test time. The slowest held about 1.6 times the mean. A shard that hits the 30-minute cap turns the run red without naming a failing test. The duration-aware shard packer is now wired into build.yml. Its plan evens out the recorded test time across the four shards. Recorded test time leaves out class-fixture setup and host start-up. So the shards' wall times will not come out exactly equal. How it works. The gate job picks one timing source per run. It is the newest successful push run on dev, out of the 30 newest successful dev runs. Its lite-tests-timing artifacts must all have more than two days of retention left. The gate pins that run and its exact artifact set. The gate picks once because the shards start minutes apart. A per-shard lookup can hand two shards different timings. A class is then dropped or doubled. In the full cut, every Lite shard downloads that run's timings and fails if what it got is not exactly the pinned set. It packs classes longest-first into the least-loaded shard. The packer's reconcile check then fails the shard unless every discovered class is in exactly one shard. Missing data. A class with no timing gets the mean weight of the known classes, so it is placed and never dropped. A test whose time is missing, unparsable, NaN, infinite or negative has no timing. A class that took zero seconds weighs a small floor, so no shard is left empty. Fallbacks. With no source, the cut is the old class-name hash. The packer also falls back to the hash in three cases. None of the timings can be read, they add up to zero, or they cover under 80% of the classes. A shard that cannot download the pinned timings, or whose download is not the pinned artifact set, fails instead of cutting alone. The shard count comes from the matrix (strategy.job-total), and a leg fails when the matrix is no longer the single shard axis. The -class chunking, the Darling-only rule, the 30-minute cap and the Lite tests (result) job are unchanged. Restated invariant. The cancel-in-progress comment used to say nothing downloads cross-run artifacts. It now says exactly one thing does: the Lite shard packer, from successful dev push runs only. A cancelled run is never read. So cancelling costs shard balance, never a test class or a verdict. What the pins pin. Two tests in CrossAppGuardCiGateTests pin the workflow text, and neither runs the packer. One pins the gate step: successful dev push runs only, every timing artifact unexpired, and the exact artifact names as a job output. It also pins each shard's side. That covers the three-try download into a clean folder and the matrix checks. It also covers the failures on a different artifact set, a failed pack and a failed reconcile. The other test counts non-comment workflow lines. It allows one gh run download across all workflows, in the Lite shard step. It allows no run-id input or workflow_run trigger in any workflow, and no download-artifact in build.yml. What the self-test proves. The Shard packer self-test step in shard 0 runs lite-shard-pack.py --self-test (53 assertions) and proves four things. Every class lands in exactly one shard for 1, 2, 3, 4 and 7 shards. The plan does not depend on input order. Missing, drifted and all-zero timings fall back to the hash. Reconcile fails when a class is withheld, doubled or invented. Plan id. Each leg logs a plan id to read side by side. Nothing compares the ids. The pinned artifact set is what keeps the legs on the same inputs. Part of #5208 * Darling store: an hourly row-count ledger for query_stats (V164) (part of #4605) (#5289) Migration V164 adds two plain collect tables for the hourly-edges count guard. collect.query_stats_hour_ledger holds one row per hour and server with the count of non-restart query_stats rows. collect.query_stats_hour_ledger_state holds one row, counted_since, which the rung sets to the next whole hour. Hours before it are uncounted, so a reader stays on raw for them. QueryStatsHourLedger holds the DDL the rung embeds, UpsertSql and RecountSql. UpsertSql adds a batch's count to its hour, and a count of 0 or less writes nothing. RecountSql sets a range of whole hours from raw. It counts the rollup's population, skips restart rows, counts NULL intervals, and keeps the name spelling raw holds. The primary key leads with the hour, so a read of a range of hours and the retention prune are index range scans. The viewer's schema probe gains the V164 arm. The compose roles read the tables through the collect schema's existing SELECT grants and default privileges, so no grant changes. A live test proves the viewer's provisioned grants read both tables and write neither. The collector now writes the ledger, a census pin guards that one writer, and the retention sweep prunes the ledger. Nothing reads it yet: the guard read follows in a second PR. DarlingCollectorRunner.CopyBatchOnceAsync adds each query_stats batch's count to its hour. The count is the rows written with a sample_interval_seconds other than 0, and a NULL counts. The upsert runs on the COPY's own transaction, after the COPY and the dimension flush and before the commit. A batch counts as a query_stats batch by the table its COPY writes, not by the collector's type. So a one-off definition aimed at that table is counted too, or its batch fails. The count is the row writer's tally of the integer it sends into that column. It is taken at the write and never re-derived. The tally watches only a row's payload, so the prefix the runner writes before the payload opens is never counted. The writer is built for each attempt, so a re-attempt starts from zero. A batch carries one collection_time, so it is one hour and one upsert, keyed on the storage name the COPY writes. A count of 0 skips the upsert. A fault in the upsert has no savepoint. The transaction rolls back and the batch's rows are not stored, so raw and the ledger commit together or not at all. Live tests drive the product's own write path. They prove the counts, the restart rows, a failed upsert and a failed COPY. They also prove the tally on a real binary COPY, in the runner's order: each row's prefix first, on a fresh writer. Source pins hold where the upsert sits. The census pin QueryStatsWriterCensusPins fails on a literal write to collect.query_stats, or a new COPY start, in the C# sources the service compiles. It reads the non-test sources of every Darling project and every project they reference. That covers the six root libraries built into the same service. It scans them for DML on the table and requires every COPY start to be on a register. The register holds the runner's site and four RDS ingestors, each copying a fixed definition into a named table. It also checks that no migration rung above V164 runs DML on the table. The failure message says why: the ledger counts rows at that one write. The pin is a text scan. It cannot see SQL built at run time outside a COPY. It cannot see a writer inside the database either, such as a trigger, a rule or a scheduled job. The retention sweep deletes ledger buckets below the successor hourly rollup's retention horizon, 90 days, with one plain DELETE. That horizon is the rollup's, not raw's four days. The prune keeps 90 days, and no read reaches past that horizon. The router sends a window to the hourly rollup only up to 89 days back, the 90 day span less a day's margin. The ledger does not promise every hour the rollup holds. The rollup drops whole chunks, and its drop is gated on the daily rollup covering them, so it can keep hours older than 90 days. A live retention test removes a bucket outside the horizon and keeps one inside it. Part of #4605 * Custom Views: the query_stats count guard reads the hour ledger (part of #4605) (#5290) The hourly-edges count guard now reads the hour ledger instead of counting every raw row of the window. IntervalRollupCountGuard.QueryStatsSql compares collect.query_stats_hour_ledger with the hourly rollup's sum(sample_count) for each server and hour, and returns the number of pairs that differ. It no longer reads raw rows. The ledger half reads one row per server and hour. The rollup half still sums the window's rollup rows, which the panel reads next. How long that takes on a large store is measured after deployment, by the acceptance read of #4605. The comparison sits in the branch taken only when collect.query_stats_hour_ledger_state.counted_since is at or before the window start. Otherwise the guard returns -1 (IntervalRollupCountGuard.UncoveredResult) without running the comparison, and a missing state row gives -1 too. The class doc states what a pass proves. The ledger is written in the collector's COPY transaction, so one REPEATABLE READ snapshot sees the ledger, raw and the rollup after the same commits. A drift the ledger saw makes the counts differ and fails safe. The writer census pin sees this build's code only, so it cannot see an older service that still writes raw rows into a V164 store. The guard catches those rows wherever the rollup holds them. It misses them in one known case: an hour the rollup never materialized whose raw rows all came from an older service. DarlingWebEndpoints.ResolveHourlyEdgesVerdictAsync handles -1 after the statement timeout is put back and before the pass test. It notes the new ReadFallback.LedgerUncovered with no log line and returns no verdict, so the panel reads raw on the same transaction. A store before V164 has no ledger table. The runner reads the store's schema version after its savepoint and before the guard runs (QueryStatsHourLedger.RungVersion names the 164). Below it, the runner notes GateFailed with no log line and the guard does not run. Every guard fault warns once, including a missing ledger table on a store at V164 or later. ReadScope ranks its reasons GateFailed, then FallbackRaw, then LedgerUncovered, in both Note and NoteFallback. LedgerUncovered resolves to the fallback_raw outcome, so these reads show in the fallbacks column of get_read_latency and never in gate_failures. The tool's description now names them. IntervalRollupCountGuardLiveTests is rewritten for the ledger. It plays the writer with RecountSql. It then checks a match, a pair the rollup lacks, a ledger row nothing else has, and a pair the ledger lacks. It also checks restart rows, the inclusive start and the exclusive end, a window before counted_since, and a missing state row. A new live class runs the runner in a REPEATABLE READ READ ONLY transaction on a scratch store. An uncovered window gives no verdict, LedgerUncovered, no log line, and a usable transaction with the statement timeout put back. A store with its schema version below V164 and both ledger tables dropped gives no verdict, GateFailed, no log line, and a usable transaction. A store at V164 or later with the ledger table renamed gives no verdict, GateFailed, one Warning, and a usable transaction. The two hourly-edges route live classes seed the ledger with QueryStatsLedgerSeed right before each panel run. The V164 rung sets counted_since to the next wall-clock hour and their fixed anchor is in the past. A row planted after the refresh now leaves the ledger ahead of the rollup, as the writer leaves it, so the guard refuses it. Unit pins cover the guard text (it reads the ledger and has no scan of raw) and the order of the runner's steps. They also cover the schema-version check ahead of the guard and the ReadScope ranking, mapping and logging. Part of #4605 * Diagnostics bundle: the store statement history is now the ranked answer get_store_query_history gives (#5097) (#5294) The bundle's store_statements section read both statement-history tables itself. Its history member is now the answer get_store_query_history gives for the bundle's --hours. That is the 25 costliest rows, one per role and statement. Each row carries its totals, the recent-against-earlier mean, block counts and aliased text, and the member also carries a capture summary. A failed history or cumulative read marks the section error in the manifest, and the verb exits as a partial bundle. The text reads of get_store_query_history and get_store_query_stats also apply the shared sensitive-statement filter as a second layer, for stores whose reader function predates it. A statement the filter withholds reads as the withheld marker, in both tools and in the bundle. Closes #5097 * CI: retry the Lite timing-source lookup when the API returns a stale page (#5297) The gate's "Pin the Lite timing source" step lists the newest successful dev runs once. That call sometimes returns a stale page of weeks-old runs with expired artifacts. The step then pins nothing. Every Lite shard cuts the suite by class-name hash, although a run that qualifies exists. The step now tries the listing up to 3 times, 5 seconds apart. It accepts a page only when a push run on it finished in the last 5 days, judged on the run's updated_at. A run that finished earlier cannot qualify. Its timing artifacts expire 7 days after upload, the upload comes before the run finishes, and the gate wants 2 days of margin. The page is judged on when runs finished, not when they were created. A run created more than 5 days ago and re-run since still has fresh timings. A failed call counts as a try. If no try gets a current page, the step falls back to the hash. The fallback notice says what the tries saw. If every call failed, it says so. If no try got a current page (stale, empty or push-less pages, with failed calls among them), it gives the counts. A current page where no run qualifies keeps the old notice. The per-run artifact test, the outputs, continue-on-error and the permissions are unchanged. CrossAppGuardCiGateTests gets a pin for the retry and the freshness test on updated_at. It also pins a failed call that keeps retrying, the notices, and the tie between the 5-day bar, the artifacts' retention-days and the 2-day margin. The pin matches words, so a re-indent or a comment does not fail it. Part of #5208 * Web test harnesses copy the whole js tree instead of hand-kept module lists (#5302) Twenty-six web test harnesses copied js modules into a scratch folder one by one or by loop before Node imported them. Adding one file under wwwroot/js/pages/ meant editing every copy list. A PR cannot edit a list that another PR in the same merge train adds. So two PRs that were each green went red once merged, in test classes that look unrelated to harnesses. Each of the twenty-six now copies the whole js tree with one fs.cpSync(jsDir, scratch, { recursive: true }). Every stand-in module is still written after the copy, so it still replaces the real file. Only imported files load, so the extra copies are inert. A new guard, WebHarnessJsTreeCopyTests, reads every .mjs under Darling/Darling.Tests. It fails, naming the file and line, when a harness does any of three things. It copies a js module by name or by directory. It builds a scratch folder without the whole-tree copy. Or it writes a stand-in module before a whole-tree copy. The copy's source can have any name, as long as the harness binds it from process.argv. The scratch folder can have any name too, as long as the harness binds it from mkdtempSync. The guard failed before this change, naming these twenty-six harnesses, and passes after it. An empty new page module imported from server-tabs.js and compose.js failed 110 tests in 14 classes in the old harnesses (ERR_MODULE_NOT_FOUND). The new ones pass. The probe ran over the 56 test classes that name a .mjs, one -class filter per class: 640 tests, 0 failed, 21 skipped for PostgreSQL. Fixes #5279 * Web charts: a legend entry that drills keeps its keyboard drill beside the hide/show switch (#5247) (#5305) When #5265 added hide and show to the legend, an entry that both drills (a grouped series) and hides kept its mouse drill. It lost its keyboard drill. An entry that drills but has no hide/show switch kept its keyboard drill, but a click on its swatch stopped drilling. #5265 is not in v3.9.0, so none of this reached a release. The label of an entry that drills and hides is now a keyboard control. Enter or Space drills, and a click drills once. Before, the label was not focusable, so the keyboard had no drill. An entry with a drill but no hide/show switch drills from the whole entry, swatch included. On an entry that drills and hides, the swatch is the hide/show control, and it was a bar 12 px wide and 3 px tall. It now has a transparent click pad, 20 x 25 px. The pad does not overlap the label (a 4.9 px gap) or the next row (a 10.5 px gap). Seven new legend tests, 18 in the class. Six run the shipped charts.js under Node. Two of those fail on dev's charts.js: the label keyboard drill and the no-switch swatch drill. The other four pin what already held. They cover label drill versus swatch switch and the swatch never drilling. They also cover the keyboard on a plain entry, and a refreshed chart whose held hidden set hides every series. That chart draws them all. The Node run also records which keys each handler claims with preventDefault, so Enter and Space are pinned as claimed and Tab as left alone. The seventh test reads app.css and pins the swatch's click pad. It pins the rule, the 20 x 25 px size, and a reach that stops short of the label. It fails on dev's app.css. Fixes #5247 * Lite's Query Store history window counts each interval once in Total Executions (#5310) The Query Store history window's Total Executions added up every stored snapshot. The grid lists every snapshot on purpose. An interval collected more than once was therefore counted about once per collection. An interval collected three times (10, 25, 40) plus one collected once (5) showed 80 instead of 45. The summary now takes the latest snapshot of each interval first. It uses the interval identity and "latest" order the aggregate reads already use. The history read carries the two identity columns it lacked (runtime_stats_interval_id and replica_role), last in its column list. The grid is unchanged. The Darling viewer's copy of the window gets the same fix. Fixes #5306 * Web Index Analysis lists up to 500 rows and says when the list is cut (#5303) The web's Index Analysis tab asked the service for 50 recommendations, the most the get_finops view took. On a large server the list ended at 50, while the desktop grid shows every finding. The view's limit ceiling is now 500 for index_analysis, and the tab asks for 500. When the list is still cut, the notice says so: "Showing the largest 500 of 612 recommendations. The list stops at 500, so 112 more are not shown." It also points at the Database box, because each database's list is cut on its own. A list of exactly 500 that cut nothing says nothing, and one finding left over reads "1 more is not shown". The Storage Growth databases list had the same fixed cut (the top 50 by 30-day growth). The tab had no way to ask for more. That level now reads limit as its row count, with the default still meaning 50, and the tab asks for 500. The objects and indexes levels are unchanged. The get_finops limit description said "1-50" for every view, which was wrong for the three views that take 500. It now says "Most rows per list (default 10); each view has its own max.", the same length as before. The shipped Index Analysis page script runs under Node for the limit and each notice. Source pins, unit tests and live tests cover the new ceilings. Ten of the new or changed tests fail on the old code, and four tests added after review failed before their fixes. Fixes #5238 * Tests: a fresh store ends up with every declared continuous aggregate (#5315) The store-object walk in DarlingStoreUpgradeTests dropped the continuous-aggregate step's ready count. That step only warns when one CREATE fails. So a store finishes its start-up one rollup short and looks healthy. The walk now ends by checking that every aggregate in the declared lists is in the TimescaleDB catalog. The failure names any that is missing. It also checks that the step's own count agrees. A new gated test walks a fresh store with the real list. It then walks a second fresh database without the collection_log hypertable step and requires the failure to name collection_health_hourly. Test only. Refs #3949 * MCP can serve HTTPS, and both listeners take a list of allowed ranges (#5288) (#5295) MCP can serve HTTPS, and both listeners take a list of allowed ranges (#5295) Add a `tls` block to `mcp.network`, the same block `web.network.tls` uses (a PKCS#12 bundle or a PEM pair). MCP's network listener then serves HTTPS only, and both listeners accept TLS 1.2 or 1.3 only. A bad or expired certificate keeps that listener loopback-only, and the loopback listener still requires the token. A certificate that cannot be loaded raises a Critical self-alert, and a new "MCP TLS Certificate Expiring" self-alert warns 30 days ahead. A `tls` block with no certificate path is refused instead of serving plain HTTP. `allowFrom` on `mcp.network` and `web.network` takes one CIDR or a list, and `--configure-firewall` scopes each rule to exactly those ranges. An entry with a zero-padded, hex or short IPv4 address or an IPv6 zone index is refused, with the reason logged. On MCP, an address outside `allowFrom` gets 403 before the token is checked. Both listeners ignore the forwarded-headers switch and any `Kestrel:Endpoints` entries. `mcp.network.hostName` names the host clients connect by. darling.json names each unknown key under the network, `tls` and `oidc` blocks in a start-up warning. MCP exposed without TLS logs a warning at every start. With none of these settings, both listeners work as before. Closes #5288. Fixes #5307. * Lite tests close every DuckDB database they open (#5321) Lite's tests used to leave about a thousand DuckDB databases open when the test process ended. Each held threads, handles and native memory, and its file was already deleted. Tests now dispose every DuckDbInitializer they open. The shared fixture and per-class fields dispose it before the temp folder is deleted. Locals sit on a using. Initializers built by helpers are tracked in a list that Dispose() closes. A new source-census test fails when a Lite test constructs a DuckDbInitializer and does not dispose it where it lives. A local must be disposed in its own method. A field must be disposed in its own class. A list entry needs a foreach over that same list in the same class. A construction passed into another constructor, a target-typed new(), and a subclass of DuckDbInitializer are caught too. A Dispose call elsewhere in the file no longer hides a missing one. Test code only. Refs #5208 * Lite test shards weigh classes that run one at a time at full cost (#5319) The Lite test shard packer cut classes by summed test seconds. But classes in DisableParallelization collections run one at a time, so each of their seconds costs about 3.4 parallel seconds. The packer now reads the serial collections from the Lite.Tests sources and each class's collection from the timing XML. A serial class weighs its seconds in full, any other class a 3.4th of that. Placement stays longest-first into the least-loaded shard, and the check that every class lands in exactly one shard is unchanged. The pack output also reports serial classes and predicted wall seconds per shard. The self-test gains a fixture where a plain sum cut piles the serial classes on one shard and the new cut spreads them within 1.25x. How much the new cut gains depends on the run. On one recent timing set the modelled walls go from 547-628 s (1.15x) to 584 s on every shard. On the run the reviewer used (658 classes), the plain cut models 502, 664, 888 and 696 s. The new cut models 687 s on each shard. The actual assembly walls of that run were 499, 639, 849 and 693 s. Refs #5208 * PostgreSQL targets: the six database-growth bars are recorded as measured over 14 days (#4404) (#5327) The six PostgreSQL database growth bars are now recorded as measured. They are the 14-day lookback, the 1 GiB and 10 percent concerning line, the 10 GiB and 25 percent critical line, and the three-sample minimum. A 14-day read of 393 databases on 50 clusters confirmed every value as written, so no threshold or grade changes. The growth trend finding now carries threshold_lineage = 1 instead of 0. Its advice says the lines are measured against the dogfood fleet, not chosen. The two co-fire boosts and the growth-rate anomaly pair were not graded and stay marked unmeasured. Tests pin the new lineage, the date and population in each constant's comment, and the advice text. Fixes #4404 * The statement filter gets its XML and JSON walks, and the plan viewer reads ParameterizedText as an attribute (#5326) The statement filter gets its XML and JSON walks, and the plan viewer reads ParameterizedText as an attribute (#5326) `SensitiveStatements.XmlCore` walks plan, blocked-process, deadlock and event XML with a caller-supplied judge. It replaces what is named with a placeholder. Inside a withheld statement, its parameter values, quoted literals and `0x` literals are withheld too. Before judging an auto-parameterized statement, the walk puts each `@N` value back into the text. In an actual plan it also judges the statement with the runtime values. A nested plan in an `EdcShowplanXml` attribute is judged one level deep, and a deeper one is withheld. `SensitiveStatements.JsonCore` walks a tool result and sends each string to the XML walk or the text judge. Nothing calls either walk yet. `ShowPlanParser` now reads `ParameterizedText` as an attribute of the statement element, so the plan viewer's Parameterized row shows the text for an ad hoc statement. Part of #5320 * Query Store grid: a History button shows a query's duration and CPU plan by plan, backed by a new store-only MCP tool (#5234) (#5300) The web Query Store grid can show a plan's history over time (#5300) Query Store grid rows have a History button. It opens a panel under that row with a chart of average duration over time, one line per plan. Below the chart, a table lists each plan's executions, average and total duration and CPU, first and last seen, and whether it is forced. Each plan has a Plan button that opens its stored plan. The chart keeps one value per plan and moment. Repeated stored copies of a still-open interval collapse to the latest one, and Regular, Aborted and replica rows are combined. The chart keeps the newest 150 points. The table keeps the 20 plans with the most total duration. The panel says when either was cut. For a window older than raw retention, the panel reads the interval table the grid reads. When stored history starts later than the window asked for, the panel says where it starts, even when it is empty. The Darling MCP server gains a store-only get_query_store_query_history tool that returns the same data. It takes database_name and query_id from get_query_store_top, plus optional server_name, hours_back and as_of. Lite's MCP server does not have it. Closes #5234 * Web Top Queries and Top Procedures rank by CPU, duration, reads or executions (#5299) The web's Top Queries and Top Procedures lists ranked by CPU only. A long query that used little CPU (blocked, or waiting on I/O) never appeared in the browser, while the desktop ranks by duration. The two endpoints behind the lists are get_top_queries_by_cpu and get_top_procedures_by_cpu. They now take an order_by ranking: cpu, duration, reads or executions. CPU is the default, and a request without order_by is the request the page sent before. Each choice maps to a fixed ORDER BY in code from a whitelist, and request text never reaches SQL. Both lists get a Rank by selector for it. The CPU ranking is the same ranking, but its raw reads changed shape. Ties now break by the group key and a NULL sorts last. The raw statements run in two passes. A narrow pass picks the winners, then the wide row is built for those winners only. The hourly statements keep one pass and get only the new ordering. Duration and executions are answered from the raw tables or from the hourly rollups. Reads is answered from the raw tables only, because the per-query and per-procedure rollups keep no logical reads. So it needs no store migration. A reads request whose window reaches past what raw keeps is the case where the hourly rollup is the usual tier. The read stays on raw anyway, and the answer's precision_note says why: the rollup has no per-query logical reads. It names cpu, duration and executions as the rankings that can read the rollup. Inside raw's reach there is no precision_note. The web page does not draw precision_note. Each Top Queries and Top Procedures answer gains a retention_notice field. It is null unless the ranking is reads and the window reaches past what raw keeps on a store with rollups. Then it holds the partial-window sentence the composed panels carry, with the raw tier's reach in days. The page draws it as text, once, in place of the window floor's truncation_note. So a reads list is never silently partial, and never shows two lines that say the same thing. The MCP tools keep their parameters and descriptions and still rank by CPU. For them retention_notice is always null, and that extra null field is the one change in their output. The web endpoint takes order_by, and an unknown value is refused with a 400, as detail and group_by already are. Top Queries now finds the latest text of every ranked group in one lookup, not one lookup per ranked row. The lookup runs once as a materialized step, and the hourly read probes the time index once per ranked group. On a store without TimescaleDB the old lookup scanned the whole table once per row. Two behaviors change on purpose. First, the text is the newest row inside the window. The old lookup had no window bound. So on the raw route a group whose rows in the window all lack text now shows no text. The hourly route keeps the text of the key's newest raw row that has one. It shows no text only when no raw row of the key carries a text. The second change is that the text match is NULL-safe. A group with a NULL database or hash now gets its text, and a WAITFOR text there is trimmed. The hourly route keeps finding a text for a window past raw retention, one index probe per key that has none in the window. The Viewer's Top Queries gets the same fix. The WAITFOR trim no longer leaves a short page. Top Queries and the Query Store top reads, in the MCP tools, the Viewer and Lite, ask for a few more candidates than the page. If the page comes back short and the round was full, they ask again with four times as many. The cap is the page size plus 200. That is at most 3 rounds. A full page or an idle window costs one round. A round whose candidates were all WAITFOR statements still reports its candidate count, so it is not read as the end of the list. A tie on the newest text, or on the ranking, now resolves the same way on every read. The newest text breaks a tie on the row collected last. The Query Store and Viewer rankings end on the whole group key. Lite's Top Queries and Query Store text picks got the same tie-break. Fixes #5226, fixes #5309, fixes #5313 * The statement filter has one definition and a .NET evaluation, and withholds more administrative statements (#5324) The statement filter has one definition and a .NET evaluation, and withholds more administrative statements (#5324) The pattern that decides which statement text reads as `-- statement text withheld (#4348)` now has one definition, in `PerformanceMonitor.Common/SensitiveStatements.cs`. The PostgreSQL filter's constants are aliases of it. The pattern names more administrative statement forms, for example `CREATE LOGIN` and `CREATE CREDENTIAL`. PostgreSQL statement text is filtered with it when it is collected. The stored-text scrub keeps the pattern it shipped with. `SensitiveStatements.Regex.cs` adds the .NET evaluation of the same pattern, with a 250 ms match timeout and a budget per read call. Nothing calls it yet. PostgreSQL and .NET agree on all 136 corpus strings. The hourly PostgreSQL statement text fetch writes its own rows to the collection log, under `pg_statement_text`. It writes an error row when it fails or times out, and a success row when it stores text. Part of #5320 * The web viewer shows the queries that waited on a wait type (#5235) (#5322) The web viewer shows the queries that waited on a wait type (#5235) The web had no path from a wait spike to its queries. A right-click on the Wait Stats chart now opens Active Queries filtered to the wait drawn nearest the click, at that time. A click on a wait in the Wait Stats, Waiting Tasks or Significant Waits grid does the same. It uses the hour around the row's time, and Wait Stats keeps the range you have. Active Queries names the wait above the grid, reads up to 500 rows, and has a "Show all active queries" button to clear it. Query Store QDS_ waits are not linked. The get_active_queries MCP tool in Darling and Lite takes a new wait_type filter. It matches the exact name in any case and runs in SQL beside database_name, so the totals and the row cap see it. Both apps tolerate the one trailing space an older collector stored. filters_applied now carries wait_type (null when unused). The tools/list budget grows 227 bytes in Darling. Lite's ceiling is corrected down to its measured 92,203. Closes #5235 * Lite shard pack: re-fit the wall model's concurrency to 3.56 (#5208) (#5338) Lite shard pack: re-fit the wall model's concurrency to 3.56 (#5208) (#5338) A replay of nine full sharded runs showed the shard wall model's shape is right and its parallel concurrency was a little low. At 3.4 its mean error was 3.5 %, and at the least-squares value of 3.56 it is 2.0 %. The spread between shards comes from class seconds that move between runs, which a constant cannot fix. The script's self-test now pins the wall formula and keeps the constant between 3.4 and 3.8. An earlier commit that also weighed each class by its test count was removed, because a replay on fresh runs showed no gain. Refs #5208 * Web API: edit a monitored server in place (PATCH /api/servers/{id}) (#5240) (#5301) Edit a monitored server from the web API, with one store lock for every address write (#5301) `PATCH /api/servers/{id}` changes a server's name, address, database, authentication, TLS settings or monthly cost in place. The server keeps its id, tags and history. A stale `modified_at` token gives a 409 with the current values. A connection test runs before a connection change is saved. `GET /api/admin/servers/{id}` returns the editable non-secret values to an admin sign-in. A read-only sign-in gets 403. A change to how a SQL or service-principal server connects always asks for the password again. The stored password is never reused, not by the connection test and not by the save. The changes that count are host, port, database, read-only intent, authentication mode, username, encrypt mode, certificate trust and multi-subnet failover. The web route and the store's edit function enforce this for all of them. The desktop viewer's edit asks again for a host or port change, or for an auth-mode switch that keeps the stored secret. Its dialog no longer pre-fills the stored password. The edit is saved by the database function `config.edit_monitored_server`. The managed role setup creates it for the viewer and mcp roles. The viewer role holds no UPDATE on the monitored servers table. If you manage roles yourself, re-run `Darling/tools/provision-roles.sql` after upgrading. Without the re-run, a web or MCP edit answers an error that tells you to re-run it. The web add and edit routes share one in-flight slot. An add can get a 429 while an edit runs, and the 429 text now says "add or edit". Every server add or edit from the web, MCP or the desktop viewer takes one store lock and checks the address again. The startup seed does the same. An add or edit also checks the database that the connection test reached. So two servers cannot claim the same address. An MCP or web add that loses the race is reported as a duplicate. The desktop Add/Edit dialog shows "Already monitored" instead of "Error saving server". A stored `env:` or `file:` password reference that names one of the service's own configuration files or secrets is now refused when it resolves. That one server fails to connect with a clear message. A store row's reference resolves only when darling.json declares it for that server at that address. The MCP `edit_server` conflict answer's `current` now also includes `engine`. Part of #5240 * The web's Blocking and Deadlocks grids open each row's stored plans (#5236) (#5298) The web's Blocking and Deadlocks grids open each row's stored plans (#5236) (#5298) Darling stores plans with blocked process reports and deadlocks. The desktop viewer opened them, but the web did not. The Blocking grid now shows a Blocked plan and a Blocking plan button, and the Deadlocks grid shows a Victim plan button. A button shows only on rows that have a stored plan. Plans are captured only while they are still in the plan cache, so many rows have none. Two new MCP tools, get_blocking_plan_xml and get_deadlock_plan_xml, give the same plans to agents. Both take an optional database_name, and the web buttons send the row's database. So two databases on one server that share a session pair or a victim never show each other's plan. A deadlock plan read that names no victim refuses to pick a plan when the deadlocks sharing its times name different victims. The plan panel shows that message. The get_blocking and get_deadlocks rows gain has_blocked_plan, has_blocking_plan and has_victim_plan flags, present only when true. Darling now has 180 MCP tools, and its tools/list budget ceiling is 193,975 bytes. Closes #5236 * Web plan panel gets a Repro script button, backed by a store-only get_query_repro_script tool (#5233) (#5341) The web plan panel has a Repro script button, and MCP clients can fetch the same script (#5233) (#5341) Darling's web viewer and the new get_query_repro_script tool now build the same repro script the desktop Viewer builds. They work for a query_hash, Query Store or active-snapshot query. The script comes from the query's stored text, and from its stored plan when one is kept. It reads only the central store: nothing connects to the monitored server and nothing runs. A procedure has no stored query text, so it gets no button. Darling now has 181 MCP tools, and its tools/list budget ceiling is 195,143 bytes. Closes #5233 * Web: pick several databases per server, part 1 (#5245) (#5314) Web server page: filter to several databases (#5245) (#5314) The server page has a "Databases" button beside the range picker. It lists the databases collected for that server. The choice is kept per browser and per server. A PostgreSQL server has no button. Six reads take the choice: Active Queries, Top Queries by CPU, Top Procedures, Query Store top, Query Store regressions and the query heatmap. Every panel heading has a chip that says which databases it shows. Names with commas, quotes or markup in them stay one name each. The page sends one database_name key per chosen database. The web route reads a repeated key as a list. It keeps each name exactly and drops blanks and repeats. It refuses a NUL in a name, more than 50 names or more than 4,096 encoded bytes. A new read route, GET /api/server-databases, lists the databases. The desktop Excluded Databases picker shares its SQL. The route stops at 5,000 names and says when the list is cut. Then the picker's search asks the route for matching names, so any database can be found. That read is about 17 times faster on a server with 500 databases and 90 days of history. A repeated query key such as ?server=A&server=B used to read as A,B and matched nothing. Single-value reads now use the first value. The delete confirm flag, the access token and the sign-in keys refuse a repeated key. A refused repeated token is logged without its value. get_active_queries no longer trims database_name. The MCP tools get_top_queries_by_cpu, get_top_procedures_by_cpu and get_query_store_top now read a blank database_name as all databases. A blank name used to match nothing. On the web, an empty database_name value is now refused with a 400. Fixes #5245 * Darling's MCP tool answers go through the statement filter, and the collectors get a statement scrub session (#5342) Darling's MCP tool answers go through the statement filter (#5342) Every Darling MCP tool result, error results included, now has its statement text and parameter values checked before it is sent. This holds with or without GCF output. The output filter is the last call-tool filter on the Darling host, so it reads each answer as the tool's own JSON. A result it cannot read, or a content block it does not know, is refused with a fixed sentence. The plan analysis tools judge each plan before they analyse it, in Darling and Lite. A plan withheld whole answers "This plan was withheld by the statement filter (#4348), so it was not analysed." The collectors get a statement scrub session: one 15 s budget per read, each distinct value judged once, and four new measurements. No collector uses it yet. Two census tests list every SQL Server column and writer that can hold statement text. A release cut fails while any of them is still pending. Part of #5320 * Web Admin: edit a configured server from the Servers grid, with stale-edit Reapply and Reload (#5240) (#5356) The web Admin Servers grid gets an Edit button on each row. It opens a form above the grid, filled from the server's current settings. The grid's refresh does not close the form, and closing the form clears the password box first. Reapply and Reload clear it too when they redraw the form. Save sends only the fields that changed, plus the time the form read the server. The password goes only when one was typed and the chosen sign-in stores one. While a save runs, the buttons are locked. The status line says whether the service will test the connection before it saves. If the server changed after the form read it, the form shows what changed. Reapply lays your changes over the new values, and Reload takes the new values. An expired session closes the form. A JSON answer that is not an object keeps the form open with your entries and says it is not known whether anything was saved. The username and password boxes ask password managers to leave them alone. Writes from the Admin page and the tag manager now go through one shared helper that keeps the answer's status and body. Closes #5240 * The Darling viewer's Overview gets Lite's server name / tag search (#5352) (#5359) The Darling viewer's Overview gets the server name and tag search that Lite's Overview has. A "Search:" box beside the sort selector narrows the cards by display name, server name or tag. It works together with "Needs attention only", and clearing the box shows every card again. A search that matches no card says "No server matches the search." in a neutral colour, never the all-clear. "Needs attention only" also works with a search. When the search finds only healthy cards, the line reads "No server matching the search needs attention." Lite now shows the same no-match sentence instead of a blank grid. Lite's matching and sort are unchanged. The web Fleet page already narrows cards by name and tag, so it is unchanged. Closes #5352 * Server add and edit from the web and MCP take the password itself (#5364) The web add and edit routes and the MCP add_servers and edit_server tools take the password itself. A password that starts with env: or file: is refused, with a message that says references can only be set in the configuration file. The value is never repeated. An edit that sends the password blank keeps the stored one, and a typed password replaces it. The test_connect command uses a stored reference only for a stored server with the same connection settings, and otherwise asks for the password. The configuration file and --add-server on the service host still take references. No stored row is rewritten. * Alerts run SQL Server statement text through the statement filter (#5360) SQL Server statement text in alerts now goes through the same statement filter as the MCP tools and the web pages. Some administrative statements, for example CREATE LOGIN or CREATE CREDENTIAL, read as '-- statement text withheld (#4348)'. So do the values and literals in an attached report or plan. The rest of the alert is unchanged. The alert engine filters an alert before it logs or delivers it, and both apps' deliverers filter what reaches them without the engine. A filtered alert is judged once. Incident objects, a custom rule's name and resolution titles are judged too. Webhooks, mail and alert history all carry the filtered text. Part of #5320 * The web Locking page filters by database, shades its wait columns and opens an index detail pane (#5231, #5311) (#5372) The web Locking & Contention page gets a Database box, so you can look at one database at a time. A blank box means all databases. The four wait columns are shaded the way the desktop shades them. Clicking a row opens a pane with that index's four lock and latch counters. The pane stays open through the page's refresh. The server page's Object Contention and Index Usage panels now follow that page's database filter. The MCP tool get_object_locking takes an optional database_name in both Darling and Lite. With a database chosen, the answer, the optimized-locking note and the truncation sentences all cover that database only. The Locking list is sorted in a fixed order, so a capped page stays the same from one refresh to the next. Say the latest snapshot holds index rows and none has lock or latch contention. get_object_locking now says empty in both apps, where it said unavailable before. Unavailable is kept for a server with no snapshot at all. Fixes #5231 Fixes #5311 * Plan reuse shadow mode counts a false hit only when the plan's shape changed (#5383) Procedure plan reuse in shadow mode no longer counts a memory grant change as a false hit. SQL Server adjusts the grant inside a cached plan in place, and that is not a new plan. Shadow now compares each plan's shape: the statements and their plan hashes, read in one forward pass over the XML. A shape change, or a plan whose shapes were never compared, is still a false hit. A plan whose bytes changed but whose shape held counts in a new deferred_same_shape instead. Shadow mode still writes the same rows, and on mode and stored data are unchanged. Each false hit now logs why it counted, and the collection-log note splits deferred_false_hit into deferred_false_hit_shape, deferred_false_hit_no_shape, deferred_false_hit_over_cap and deferred_false_hit_unparsed. Those four add up to it. An operator can tell a real shape change from a plan that was not compared. deferred_false_hit stays the number that must read 0 before moving to on. The README and sample config describe the gate and the split. Part of #5158 * Lite archive watermark reads run once per table and share a two-wide limit (#5377) (#5390) A user saw five concurrent archive MAX reads on one DuckDB engine take 41 to 63 s each, against 3 s alone. Several Query Store per-database watermark readers and forced-plan alert readers were inside DuckDB preparation at once. Three things in the code produced that load. - The per-database watermark read ran one whole-archive bind and scan for each database on a cold archive generation. - A floored `v_` MAX ran uncached on every cycle for every idle database. That happened when the cached archive maximum was null, or at or below the floor. - Nothing limited how many archive reads ran at once. Part of #5377 * The statement filter judges a typed variable declaration with an initial literal (#5369) The statement filter now judges a variable declaration that has a type between the name and its initial literal. It treats it the way it already treats the untyped assignment. The two shapes are T-SQL `DECLARE @x nvarchar(20) = N'...'` and PL/pgSQL `x text := '...'`. They count only for the variable names the filter already covers (the same name set as the untyped assignment, including `key_source`). A statement with no literal (`= @p`, `= $1`, a column definition) is unchanged. The judge now runs as two regexes that share the one 250 ms budget. It is built on a background task when Lite and the Darling service start. Part of #5320 * Lite tab switches, timer refreshes and time-range changes now share one in-flight read, and the latest request wins (#5371) (#5380) In Lite, the server tab's tab-switch handler checked `_isRefreshing` but never set it. A timer tick that fired while a tab-switch read was still waiting started a second full refresh on top of it. In the field trace the overlap ran a refresh for 484 s. The same one-way check also dropped work in the other direction. A tab switch, or a time-range or custom-date change, that arrived during a refresh was thrown away. The combo then showed the new range while the charts stayed on the old window. `SetTimeRangeIndex` (Apply to All) and the second `RefreshData()` after a collection run hit the same drop. Part of #5371 * Lite's CHECKPOINT waits a bounded time for the lock, and a no-op archive pass keeps the views (#5377) (#5385) Two defects in Lite, both provable from the code. - After every collection round the collector runs CHECKPOINT under the process-wide write lock with no timeout. That lock is a `ReaderWriterLockSlim`, and a waiting writer makes every new reader wait. So one slow read parked the CHECKPOINT, and the CHECKPOINT then parked every UI read and the next collector behind it (#5371). - `ArchiveService.ArchiveOldDataAsync` rebuilt the archive views in a `finally` even when no table archived a row and compaction changed nothing. Each rebuild bumps the archive generation and throws away every cached watermark. So every hourly pass made the next read of each table pay for its watermark again (#5377). Part of #5377 Part of #5371 * A denied Extended Events session setup shows as a permission state and stops retrying (#5386) A login that cannot create Extended Events sessions now shows PERMISSIONS for the long-query collector. Before, SQL Server error 15247 was missing from the shared list of permission denials. A denied session setup showed ERROR in Lite and SESSION_MISSING in Darling. Lite used to repeat the denied create on every cycle. Now it stops the create for the rest of the app session once the login is denied. A restart tries again. Other failures still retry every cycle. Darling now records PERMISSIONS with a message that names what to grant. It retries the create once an hour, or at once after a reconnect. Fixes #5378 * Darling: a sealed password format bound to a server's connection settings (#5366) (#5382) ## Why A SQL login password needs a stored form that works on every platform. It must open only in the service, and only for the connection it was saved for. This part adds that format and the pieces around it. Nothing calls it yet, so no behavior changes. ## What changes - `PasswordSeal`, `PasswordPublicKey`, `PasswordPrivateKey`: the `sealed:v1::` format. A fresh AES-256-GCM key per value, wrapped with RSA-OAEP-SHA256 to the service's RSA-3072 public key. Every failure is a `PasswordSealException` with one fixed sentence per kind and no inner exception. - `ServerConnectionIdentity`: the connection settings a password belongs to, with one row mapping and one "differs" rule. It gives the same answers as the edit and connection-test rule today. - `PasswordBinding`: the purpose and ordered fields (server, remediation, mail server) that go into the sealed value's associated data. - `DarlingPasswordKeyState`, `PasswordKeyStatus`, `IPasswordKeyRing`, `DarlingPasswordKey`: the start-up decision table for the service's key. Also the ring the rest of the service is handed, which is not ready until the key is loaded. ## Test plan - [x] `PasswordSealTests` Total 77, Failed 0. A mutation that dropped the bound fields failed 24 of them. - [x] `PasswordBindingCensusTests` Total 11, Failed 0 - [x] `DarlingPasswordKeyStateTests` Total 15, Failed 0 - [x] Every `*Census*` class: Total 356, Failed 0, 1 skipped. Every `*Ratchet*` class: Total 15, Failed 0. `*DocCommentHygiene*`: Total 77, Failed 0. - [x] Darling.Tests build: 0 warnings - [ ] Full suite: left to CI Part of #5366 * Darling: the service keeps a password key file beside its other credentials (#5366) (#5384) The loader the log-hash key file used is now shared, and a second key file, the password key, uses it. Load never generates, Generate never overwrites, and Retire renames and never deletes. A credentials directory can be open to other users until the service sets it owner-only. In that case this file stays under its own name (password-key.open-directory, or password-key.dpapi.open-directory on Windows). Every later start and every other process finds it there and reports it. The role-password files are still removed. Accept puts the kept file back at the live name after the caller checks it. The key file is read within a size limit and is written through a temporary file of its own. On Unix it must be owned by the service's user with one name. Nothing a user can see changes yet. Part of #5366 * Lite compaction merges only what changes and what fits the free disk space, and warns when the data volume is low (#5377) (#5394) Lite's archive compaction no longer rewrites a whole month on every hourly pass. It leaves a part that is already merged alone. It merges only the batches that fit in the free space. It keeps at least 64 MiB or the size of the main database file free, whichever is larger. A batch that does not fit waits for a later pass, with one warning per pass. Lite now warns when its data folder is low on free space. It warns at startup and when the folder enters the low state. While the folder stays low, it warns at most once every 24 hours, and it says once when the folder recovers. Lite reads the free space for the folder itself, with the drive root as a fallback. When it cannot read the free space, it does not warn or hold anything back. Part of #5377 * Blocking and waits reads follow the database filter (#5244) (#5392) The database filter already reached the query and session reads. The blocking and waits reads still ignored it. A person who chose one database on a server page still saw blocking and waiting tasks from every database. These reads now follow the filter in Darling's MCP tools, Lite's MCP tools and the web viewer. The blocking trend and stats tools now say which source answered: the blocked process reports, or the DMV snapshot when the filtered reports are empty. Part of #5244 * The store takes a server's password itself from the viewer, admin and MCP roles (#5374) The store now takes a server's password itself from the viewer, admin and MCP roles. It refuses a password that starts with env: or file: from these roles. Those references are set in the configuration file or with --add-server on the service host. The store also refuses a change to how a server is reached when the change keeps the stored password. The password must be typed again with the change. On a row that holds a remediation login, a change to how the server is reached is refused with one sentence. The sentence points to the service host, the configuration file and --add-server. The Viewer's edit asks for the password before it saves, and it shows the same sentences. The service copies a darling.json secret into a store row only when the file entry and the row agree on the connection settings. The drift warning now says the file's password is not used until the two agree. A self-managed store gets the rules when the service starts, unless they are already in place. If it cannot create them, the service logs one warning that names provision-roles.sql. * Lite's Top Queries and procedure comparison stop scanning the whole archive for query text (#5381) (#5398) Three Lite reads used to pick each group's query text from the whole query_stats archive, with no time filter. They are the Top Queries grid, the get_top_queries_by_cpu MCP tool and the procedure comparison. Memory grew with the archive, not with the window shown, and the reads ran out of memory at Lite's 1 GB limit. They now take the query text after ranking, only for the ranked groups and only from the time ranges being read. The text and the plan come from the same row. A group with no text inside the ranges read now shows a blank query text. Part of #5381 * Agent working orders: one issue keeps one branch and one PR (#5404) An issue too big for one agent was getting several branches and several PRs, which split its history, its PR summary and its changelog entry. The agent working orders now say one issue keeps one branch and one PR, even when several agents build it. * The Darling installer refuses an install when the .NET Runtime 10 is missing (#5407) (#5409) The Darling installer's runtime check refused an install when the ASP.NET Core Runtime 10 was missing. It never looked for the base .NET Runtime 10. The standalone ASP.NET Core Runtime installer does not include the base runtime. So a machine with only that installer passed the check, and the service then failed to start with the .NET host's "You must install .NET" error. The check now refuses the install when the base .NET Runtime 10 is missing too. Its message lists what it found for both runtimes and points to the Hosting Bundle, which installs both. Closes #5407 * Lite's MCP answers and plan reads go through the statement filter (#5361) Statement text in previews, analysis findings and alerts is now judged as a whole statement by the statement filter, and only then cut to length. This applies in Lite and in Darling. Before this change, some reads and alert builders cut the text first. The filter then saw only the start of a long statement. A statement that the filter keeps reads exactly as it did before, at the same length. A statement that the filter withholds reads as the withheld marker. Two analysis reads fetch whole text only for the rows they print. These are the reconstructed blocking chains and the parameter-sensitive plans. Their read cost stays close to what it was. A new source scan fails the build when scanned code cuts statement text before the filter judges it. Part of #5320 * Darling's plan tools, web pages, bundle and drill-down run SQL Server statement text through the statement filter (#5362) SQL Server statement text and execution plans now go through the same statement filter as PostgreSQL statement text in more places. They are Darling's MCP plan tools, its web pages, the diagnostics bundle and the analysis drill-down. Some administrative statements, for example CREATE LOGIN or CREATE CREDENTIAL, read as '-- statement text withheld (#4348)'. So do the values and literals in that statement's plan. The rest of the plan is unchanged. Part of #5320 * A scripted --encrypt-password no longer writes to the error stream on success (#5413) (#5417) The Darling service's --encrypt-password wrote its "Password: " prompt and its paste hint to the error stream on every run, including a successful one. Windows PowerShell 5.1 turns each line a program writes to a captured or redirected error stream into an error record. So a script running with $ErrorActionPreference = 'Stop' stopped even though the command worked. The prompt now shows only when a person is typing the password, and the paste hint only when a person is reading the console. A run that succeeds writes nothing to the error stream. Closes #5413 * Lite Query Store top queries, comparison and regressions read query_text after the ranking (#5381) (#5396) Lite's Query Store top queries, comparison and regressions carried the query text through their ranking. At Lite's 1 GB memory limit, wide windows ran out of memory. They now take the query text after ranking, only for the ranked queries and only within the window read. When a query's newest kept interval has no text, the comparison and regressions fall back to an older kept row's text, newest day first. The 30-day top queries read can still run out of memory at 1 GB. Part of #5381 * SQL Server statement text and plans go through the statement filter when they are collected (#5367) SQL Server statement text and execution plans now go through the statement filter when Lite and Darling collect them, before either app stores them. Readers show a withheld statement or plan as withheld. Analysis findings and alert history filter statement text as each is written. Get Actual Plan refuses a withheld statement. The mute dialog runs an alert's whole statement text through the filter before it fills in a pattern. Pileup and blocking-incident grouping keep different withheld statements apart. The statement census test now fails on any pending column or writer in every Darling test run. The release job still runs it before packaging. Closes #5320 * Lite's archive files get 2,048-row row groups, so a windowed read stops decoding a whole day of query text (#5381) (#5402) DuckDB decodes a whole row group's column chunk for any read that touches the column. Lite wrote its archive files with large row groups, so a one-hour read of query text decoded about 73 MB from a compacted file. Every Lite archive write now uses ZSTD and 2,048-row groups, so the same read decodes about 17 MB. Files written before this keep their row groups until retention removes them. The picks that break a tie on the newest row now also order by collection id. Lite's main DuckDB connection now runs with a 2 GB memory limit and at most eight threads. Existing installs pick this up on their next start. Part of #5381 * DarlingEncryptPasswordStreamTests awaits its tasks instead of blocking on them (#5426) DarlingEncryptPasswordStreamTests blocked on its output tasks with GetAwaiter().GetResult(), which the xUnit analyzer reports as two xUnit1031 warnings. The helper and the three tests that call it now await those tasks. The waits and assertions are unchanged. * Lite's Collection Health read finds the newest rows without four whole-window sorts (#5371) (#5388) Lite's Collection Health read ranked every run of the 7-day window four times to find each collector's newest note, failure text, fan-out item and run. It now takes one aggregate pass plus keyed lookups at the newest instants, with the same 31 columns. A test checks every column against the old statement. The Health and Log reads now log how long they waited for the lock, ran the statement and read the rows. The server tab's refresh can now cancel the Health, Log, Duration Trends and data-start reads. A cancelled read stops its statement and frees the lock. The tab reuses a Health result until its refresh interval passes, and a timer tick always reads fresh data. Part of #5371 * Darling: store SQL login passwords on Linux, sealed to the service's key (#5387) A SQL login password can now be saved and used on Linux. So can the SMTP password, the webhook URLs, the PagerDuty routing key and the generic webhook headers. The Darling service keeps a key pair and publishes its public half in the store, with new tables for it. The Viewer, the web admin, the MCP add and edit tools and --add-server seal each value to that public half. Only the service opens it. Each saved value works only for the connection settings it was saved for, so changing one of them means entering the password again. Values saved the old Windows way keep working and are replaced the next time they are saved. The Viewer remembers each store's key the first time it connects, shows its fingerprint, and asks before saving if the key changes. --reset-password-key makes a new key, after which saved values are entered again. The compose files and the documentation say where the key lives and how to back it up. A self-check in the Linux build checks the key file on Linux. The generic webhook's headers go only to the generic URL itself or a path under it, never across a redirect, in Darling and in Lite. Viewers older than this change must be updated along with the service. Fixes #5366 * The database filter reaches the duration trends, clutter, long-query and plan-correction reads in Darling's MCP and web (#5244) (#5414) The database filter now reaches the duration trends, Query Store clutter, long-query and plan-correction reads. Six Darling MCP tools and five Lite tools take an optional database_name, and each limits the read to the chosen databases before any row limit. The web pages send the chosen databases to these six reads, so their chips read "filtered". A filtered answer with no rows says "for the database X" for one name or "for the chosen databases" for two or more. Answers carry a database_name key. The blocking trend and severity charts in Lite, the desktop viewer and the web now say which collector answered. The desktop viewer's Blocking Stats tab limits its severity read and its data-start note to the chosen databases. The duration and execution-count charts in Lite and the Darling viewer had a wrong rate for a chosen database. They divided a bucket's rate by only the collections where that database had rows. The filter now sits inside the sums, so every collection in the bucket counts. A stretch where the chosen database was quiet is drawn as zero, not as a gap. Part of #5244 * get_pg_io_trend no longer times out when table statistics are stale (#5425) (#5442) The get_pg_io_trend and get_file_io_trend tools no longer fail with "Exception while reading from stream" when PostgreSQL's row estimates for pg_io_stats or file_io_stats are out of date. Both used to join every row back to a windowed CTE: the interval to the previous collection, and the ranking of the file I/O series. With a stale estimate the planner ran each join as a nested loop and re-ran the windowed pass once per row. The pg_io_stats read took 64.7 seconds over 72 hours, and the file I/O read took 84.6 seconds over a week. Both are past the 30 second read deadline. The pg_io_stats interval is now a LAG over per-snapshot sums. The file I/O series totals are window sums over one scan. Neither read has a join left, and the same rows and values come back. A new live test builds the stale-statistics state on purpose for every table the trend payload census covers. It reads all twelve trend tools at 72 and 168 hours, and it fails on the old file I/O query. Fixes #5425 * The Darling service ships AWSSDK.SecurityToken so a role_arn AWS profile can assume its role (#5431) (#5446) The Darling service can now assume an AWS role from an AWS_PROFILE profile. A profile with role_arn failed with "Assembly AWSSDK.SecurityToken could not be found or loaded", because the service did not ship that assembly. A cluster in another AWS account was out of reach. The service now ships AWSSDK.SecurityToken 4.0.101.1. AWSSDK.Core does not move. The PostgreSQL first-target runbook also lists rds:DescribeDBInstances and pi:GetResourceMetrics in its IAM section, and it describes the cross-account setup. Fixes #5431 * Severe errors show the database name their id had at the time of the error (#5373) (#5432) A severe error now shows the database name its id carried when the error happened. SQL Server reuses the id of a dropped database. An old error from database A used to show database B's name once B took A's id. The fix covers Lite's Severe Errors view, the Darling viewer's Severe Errors tab and the Darling MCP tool get_health_parser_severe_errors. Each error is named by the newest database-size snapshot at or before it that has the id. A database that was offline and not collected uses its last earlier snapshot. That snapshot is looked for at most 14 days back from the first error's snapshot. If there is none, the oldest snapshot within 14 days after the error is used, and then the raw id. An error with no time gets its id's newest name from the last 14 days. When one id has two names at one instant, the larger name wins in every read. The database filter on both views follows the same name. One shared rule lives in the common project. Darling reads the history with one query on the existing size-snapshot time index. It runs one more only when an error has no time, and there is no migration. Fixes #5373 * Lite shard packer: add a back-test command and record the re-fit finding (#5208) (#5430) The Lite shard packer gets a backtest command that replays its wall-time model against earlier runs. It weighs each run's shards by the previous run's timings and prints modelled against ran for every shard. Replaying 14 dev runs shows the model fits a shard's own seconds to 3 percent. The previous run's seconds are 25 percent off on average, though, because a test class's seconds swing with what runs beside it. No estimator over earlier runs got under about 19 percent. So the concurrency constant and the cut are unchanged, and the comment on the constant records what was tried. Five self-test assertions pin the new command. Part of #5208 * The Darling viewer shows blank, not 0, for columns the hourly rollups do not keep (#5329) (#5443) The Darling viewer's Top Queries and Top Procedures show blank, not 0, for the columns the hourly rollups do not keep. On a window served from the hourly rollups the grids showed 0 reads, writes, physical reads, rows and spills. Top Queries also showed 0 for DOP, grants, threads and CLR. A reads sort ranked by those zeros. Those cells are now blank, and the header note says so. Min/Max CPU and elapsed are blank on the hourly route too. The rollup keeps only per-collection totals, so those columns held the smallest and largest per-collection sums, not the fastest and slowest execution. Raw-routed pages are unchanged. The web's Reads ranking now says in one sentence that it reads the raw collections, because the per-query hourly rollups keep no reads. Part of #5329 * The desktop database filter narrows file I/O, sizes, the version store and locking in Lite and the Darling viewer (#5312) (#5438) The saved database filter now narrows more screens in Lite and the Darling viewer. It now reaches the File I/O latency and throughput charts and the Database Sizes grid and size chart. It also reaches Storage Growth with the heatmap's database list and the Persistent Version Store grid and chart. The FinOps Locking grid, its database box and its optimized-locking note follow it too. They now show only the chosen databases. The File I/O top ten files and the version store top five are ranked among the chosen databases. On the Locking tab the filter and the database box both apply. The Utilization free-space score stays server-wide. With no filter saved nothing changes. The FinOps tab reads the filter when it loads, so a change made in a server tab shows on its next refresh. In the viewer, the Locking grid's 200-row cap now breaks ties like Lite and the MCP tool, so all three pick the same rows. Closes #5312 * The retired-baseline sweep reopens its connection after a drop loses a race with a refresh job (#5439) The retired-baseline sweep no longer leaves its connection dead when a drop loses a race with a TimescaleDB refresh job. A refresh job running at the moment of the sweep's DROP failed it with XX000 (tuple concurrently deleted). Npgsql closes the connection on that class of error. So the next relation and the convergence steps after the sweep failed with "Connection is not open" until the next hourly pass. The sweep now reopens its connection after a failed drop and logs the failed drop once. Two live tests pin it: one plants an XX000 from inside the drop, one runs the refresh job beside the drop. The two original sweep tests also schedule their test aggregate's refresh policy a day out, so they no longer depend on timing. Fixes #5416 * Lite merges query_snapshots archive files into one file per day (#5393) (#5434) Lite now merges the `query_snapshots` archive files one day at a time into `YYYYMMDD_query_snapshots.parquet` (and `_ptNNN` parts). Before, each archive pass left its own file, and every read had to open all of them. A day of 24 hourly files of about 2.2 MB becomes one file, and larger days become a few parts. Each merge takes up to 64 MiB of input on one thread. Its output row groups are cut at 32 MiB (`ROW_GROUP_SIZE_BYTES`), which is what bounds memory. On a heavy-plan test shape the worst measured peak was 1.18 GB under the 4 GB cap, flat from 8 to 64 MiB batches. The same input without the byte bound needed 3.0 GB or ran out of memory. A file over the budget by itself is left as it is, and a failed merge leaves its inputs. Monthly or imported `query_snapshots` files are still not merged. A long first pass stops starting new day merges after 5 minutes and goes on in the next pass. An imported day file is no longer logged as unrecognized. A file named by an interrupted archive's journal is no longer merged away before its rows are deleted. `tools/CompactionRepro` can now repeat plan text, copy inputs and set the row-group byte bound. The new and changed tests cover the day grouping, the budget split, the byte bound, the journal, retention and the one-hour read pruning row groups. Fixes #5393 * The large plan read test survives a runner that stalls the statement filter (#5440) (#5445) The large plan read test no longer fails when a busy machine stalls the statement filter past its time limits. The filter gives up on a plan it cannot finish judging in time: 1.5 s for a read, or 250 ms for one value. It then returns the withheld-text placeholder, and on a slow runner that turned a 600,000-character plan read into a failure. The test now warms the filter first. It reads again, up to three times, only when it got the placeholder after at least 250 ms. A fast placeholder is never retried, and every assertion on the final answer is unchanged. The filter and its limits are not changed. Fixes #5440 * The Darling PostgreSQL tests run in six CI shards, so the slowest finishes in about 13 minutes (#5433) (#5435) The Darling PostgreSQL tests in CI now run in six shards instead of three. The three shards took 13 to 21 minutes each and set the finish time of a merge train. About 5 minutes of each shard is fixed cost (checkout, build, PostgreSQL setup) and the rest is tests. A model from the observed shard spread puts the longest of six shards at about 13 minutes. That is under the build job and the slowest Lite shard. The class cut now divides by the size of the shard list, so every class stays in exactly one shard when the list changes. The comment above the job has the measured times and the model. No product code changes. Fixes #5433 * The Darling viewer's procedure comparison, the two comparison twins and Query Store top text lookups get a time bound (#5420) (#5427) Closes #5420. * A store-object convergence step that breaks the store connection no longer fails the steps after it (#5444) (#5453) Fixes #5444. * At start, the Darling service raises one alert when it was not collecting for 15 minutes or more (#5450) (#5454) Part of #5450 (proposal 1). * The first start after the schema 165 upgrade takes the legacy password pins before it reads the server list, so saved passwords keep working (#5456) Fixes #5455. * PagerDuty: de-duplicate edge-type alerts into one incident (#5368) Two opt-in PagerDuty settings, shared by Lite and Darling through PerformanceMonitor.Notifications. Both are off by default, and with them off every key stays as it was. - With de-duplication on, the two halves of an edge pair go to one PagerDuty incident. The pairs are Server Unreachable with Server Restored, and AG Replica Disconnected with AG Replica Reconnected. The AG key carries the group and the replica, so one replica's reconnect closes only its own incident. - With auto-resolve on, the closing edge is sent as a resolve on the same key. A delivered close lets the next outage post at once. The clear holds even when the cooldown re-reads the alert history. - The replica identity is kept when an alert is copied for Summary-mode or per-event delivery. - Darling store schema V166 adds the new setting's column. Contributed by ianwalkeruk. * The Darling service pings a dead-man's-switch URL while it is collecting (#5450) (#5460) The Darling service can now send an HTTP GET to a dead-man's-switch URL every interval. A check outside the box then alerts when the service stops or stops collecting. A new file-only `heartbeat` block in darling.json sets `url` (off by default) and `intervalSeconds` (60 to 3600, default 300). The URL is a plain URL or an `env:` or `file:` reference. Validation checks only the shape of a reference. The service reads it again on every ping, so a rotated token file is picked up. A file reference is read asynchronously and capped at 8 KB. The ping goes out when no server is enabled. It also goes out when the newest collection_log time is at most 15 minutes old and at most 5 minutes ahead of the clock. A stale or future-dated collection time skips it, and so does a failed store read. The URL is never logged, and logs name the scheme and host. A URL with a user name and password is refused. An http URL logs one warning. The diagnostics bundle aliases the check host. No web, MCP or triage page returns the block, and a null block means off. The ping waits 10 seconds, follows no redirects and counts only a 2xx as success. A failed ping, or a failed read of the reference, logs one warning and then at most one per hour. Only shutdown ends the loop. The README gains a "Watching the service from outside" section with a host-level check and its SQL. Part of #5450 * The database filter reaches the configuration, config change, severe error and default trace reads (#5244) (#5464) The server page's database filter now reaches the database configuration, database-scoped configuration, Query Store health, configuration changes, severe errors and default trace reads. In Darling and Lite, the configuration changes, severe errors and default trace MCP tools take an optional database_name. The web route takes several chosen databases. Severe errors are matched on the name each error's database id had at the time of the error. That is the same name the rows show, in Darling, the Darling viewer and Lite. On Lite's Severe Errors tab, a filter of proddb no longer matches a database named ProdDB. A filter entry written as database_id N no longer matches an error whose id has no name history. A whitespace-only database_name now means every database on the configuration reads, and a filtered empty answer names the database the same way in both apps. Part of #5244 * The database filter reaches the storage and file I/O reads (#5462) The server page's database filter now limits four panels to the chosen databases. They are File I/O Latency, database size, table and index size, and version store. With exactly one database chosen, File I/O Latency draws one line per file. With none or several, it draws one line per database and file type. Darling's MCP server and Lite now give four tools a `database_name` argument. They are `get_database_sizes`, `get_table_index_sizes`, `get_pvs_stats` and `get_file_io_stats`. Every answer says which database it was limited to. That includes the empty and not-collected answers. `get_file_io_trend` no longer trims `database_name`. A name with a leading or trailing space is matched exactly. An empty answer for a database that is not in the newest snapshot now says so for that database. It no longer blames collection. This completes the database filter. Only the three deadlock reads stay unfiltered, on purpose. Fixes #5244 * A daily audit of retained hourly history raises one Collection Gaps In History warning (#5450) (#5461) Once a day, after 03:00 UTC, the Darling service audits the store's retained history for the previous UTC day. It raises one "Collection Gaps In History" warning when an hour fell below half of usual. Usual is the median of the 7 days before, for the same hour of day. The five audited hourly rollups are judged on how many servers are present. The perfmon baseline is judged on servers and on collection passes, one row per server per pass. That count does not move with workload, so a quiet weekend with every server collecting flags nothing. Only servers enabled now are counted, on the audited day and the days before it. An hour flags only when its usual is at least 1, and the day's last hour is always judged. A source is ready when it has any bucket at or after the day's 23:00 bucket. The read runs up to the current hour, so a hole across midnight is judged as soon as collection resumes. A source that fails to read, or is not ready, is retried alone on later ticks. Nothing is raised or stamped until every source is read or absent. At the next day's slot the audit gives up with one log warning. It raises what it has. A source that was read but never ready keeps its flagged hours, marked as not complete. Every unread source is named in the alert text. Sources with identical flagged ranges share one line. The hour count is the distinct hours flagged by any source. The two big Query Store rollups are read by one index probe per enabled server per hour. A scan of their 8-day window timed out at the audit's 60 seconds. The third Query Store hourly is not read, because the two audited ones cover it. The other sources keep one grouped read. The audit day is in the alert's delivery key. So a give-up for one day and the next day's audit in the same tick both reach the channels. A restart on the same day does not raise it twice, because the daily documents' delivery stamp gates it. The alert text and the README note that a change to the perfmon collection schedule can trigger it for a few days. The new metric is registered in the alert family, severity, classifier, triage, README and census pins, and in Lite's severity test. Part of #5450 (proposal 3). * Tighten the password pin step after the schema 165 upgrade (#5466) The upgrade now records which old-format saved passwords are in the store, with the hash of each value and the connection it is saved for (a new rung, V167, in a new owner-only table). The one-time pin step works from that record: it pins a value only when its text and its connection both still match and this machine can open it. The step asks whether a value opens and never keeps the opened password in memory. A Windows host takes over a step a Linux host left open, and the pin step raises the config version in the same transaction as the pins so every host reloads. The step runs after the key step on every path and, when it could not run, is tried again on each sweep; until then the re-enter line tells the operator to wait. The record is taken only while the pin step is pending. Provisioning, the trigger check and the viewer's schema probe cover the new table. * Tighten how Darling matches an RDS target to its AWS instance (#5465) Tighten how Darling matches an RDS target to its AWS instance Darling now checks an RDS or Aurora target's host before it reads that target's logs or Performance Insights data. It asks AWS for the endpoint of the instance or cluster that the host names. It compares that endpoint with the host it connects to, ignoring case and trailing dots. On a match, reading goes on as before. On a mismatch, Darling reads nothing for that server. The runner records a PERMISSIONS outcome that names the host and the id. A verdict is cached per server, host and id. A match is kept for an hour and a mismatch for five minutes. Editing or removing a server forgets its verdicts. SECTION: Fixed ENTRY: - **Darling checks an RDS or Aurora target's host against the endpoint AWS reports** ([#5465]). It does this before it reads that target's logs or Performance Insights data. If the host does not match, Darling reads nothing for that server and records a PERMISSIONS outcome naming the host and the id. * Test clock helpers read the clock once per test (#5476) Test clock helpers now read the clock once per test. Five Lite test classes cut UtcNow to the minute on every call to a Minute() helper. A test that ran across a minute boundary seeded rows into two minutes and failed (BlockingTrend_OneName_CountsOnlyThatDatabase, Expected [3, 1], Actual [2, 1, 1]). Each class now keeps one anchor per test instance. Twenty expressions read UtcNow twice to cut it, and now read it once. Three places that took year, month, hour or minute from separate reads got one read each. A planted wait at the end of a minute fails the old helper three times out of three and passes with the fix. Test code only. Fixes #5473 * A second PagerDuty recovery inside the alert cooldown now resolves the incident (#5469) (#5470) A second PagerDuty recovery inside the alert cooldown now resolves the incident (#5469) With PagerDuty auto-resolve on, a server that went down, up, down and up inside the alert cooldown sent trigger, resolve and trigger, and then nothing. The second recovery was throttled by the first one's cooldown, so PagerDuty's second incident stayed open. The AG replica pair had the same problem. A delivered PagerDuty outage alert now also resets the cooldown on its recovery alert. The cooldown's history seed checks both alerts of the pair, so this still holds after the app restarts in the middle of an outage. The flag is read once per delivery. With auto-resolve on and PagerDuty set up for the alert, each AG replica keeps its own cooldown. Teams, Slack and the generic webhook then send each replica its own outage and recovery notice. Email stays per server. With auto-resolve off, or with PagerDuty not set up for the alert, nothing changes. The generic webhook's `{{dedup_key}}` is unchanged. The code is in the shared Notifications project, so Lite and Darling both get it, with tests in both. Fixes #5469 SECTION: Fixed ENTRY: - **PagerDuty auto-resolve now resolves a second outage that recovers inside the cooldown** ([#5469]) - The first recovery started its own cooldown. The clear after a delivered close only reset the outage alert, so the second recovery was throttled and PagerDuty's second incident stayed open. A delivered PagerDuty outage alert now also resets the cooldown on its recovery alert, in Lite and Darling. The same holds after the app restarts in the middle of an outage. With auto-resolve on and PagerDuty set up for the alert, each availability group replica also keeps its own alert cooldown. One replica's outage or recovery no longer throttles another replica's alert on the same server. With this on, each replica gets its own outage notice and its own recovery notice in Teams, Slack and the generic webhook. Email stays per server, with one cooldown for each server. With auto-resolve off, or with PagerDuty not set up for the alert, nothing changes. The generic webhook's `{{dedup_key}}` stays on per-state keys either way. This fixes unreleased behaviour from #5368. * Lite keeps query stats and Query Store stats in one archive file per day (#5410) (#5482) Lite's archive compaction now merges query_stats and query_store_stats one day at a time, the way it already does for query_snapshots, instead of one month at a time. A one-hour read of these tables touches about 4% of a day file's row groups instead of about 15-20% of a month file's. The day merge has no sort, runs on one thread with the 64 MiB per-batch budget and the 32 MiB row-group bound, and peaked at 1.16 GB at most in the measurement on the issue. Month files and imported_ files already on disk are left as they are and age out through retention. Rows still in per-cycle files at the switch were never merged into a month file, so a day file and a month file never hold the same row. New tests cover both tables: day file shape, parts past the budget, retention by day, the month-to-day seam read through the archive view, a one-hour windowed read, and the left-alone shapes. Fixes #5410 * Procedure stats keep every procedure that did work, with plans for at most 150 (#5449) (#5463) Procedure stats now keep every procedure, trigger and function that did work in the last minute. Before, they kept only the 150 with the most lifetime elapsed time. The main query reads every module that ran in the last ten minutes (safety cap 5000, newest first) as numbers only. A row whose counters did not move is dropped at read time, and its baseline still advances, so no work is lost. Plans render in a second query for at most 150 rows, ranked by work since the last run. A failed plan query ships the rows without plans. The procedure trend rate, the Top Procedures slicer count and the data-start banner read the same on an older store and a new one. Every procedure chart has one point per successful collector run. A quiet run reads as zero work, and each point's seconds are the gap to the previous point. A stretch with no collector runs for over an hour is left out of the seconds a bucket divides by. A procedure that returns after a long absence no longer lowers the whole collection's rate. The Lite slicer overlays now plot each row's own delta. The hourly procedure trend fills an hour as zero work only when the collector ran in it and stored nothing. A quiet hour at the start or end of a window, or between busy hours, reads 0. An hour the collector was down stays a gap, and a wider bucket divides by the hours it covers. The per-procedure history charts plot a quiet run at its start, with the right neighbours. The Darling "data starts at" note follows where retention cut the rows, and counts collector runs on a store with no procedure row yet. Fixes #5449 * The statement filter judges each document once and its budget grows with the documents (#5477) (#5478) The statement filter read a blocking alert's report twice and gave a whole alert, or a whole MCP or web answer, one fixed 1.5 s. On a busy machine a large report could use that up, and the alert went out without its details. Each budget now remembers what it judged. A document that appears twice is judged once. A repeat returns the first result, a withheld marker included, or the caller's own instance when nothing was named. The allowance grows by half a second per megabyte of each distinct document, up to 10 s. A capped read earns only for the text it can judge. The alert path and the MCP and web sweep both use it, in Darling and in Lite. A document whose raw text the pre-check clears, and which has no character the reader would change, skips the full walk. A warm-up runs at start in Darling and Lite, off the startup path. It judges a blocked-process report and a deadlock graph of about 64 KB a few dozen times. The first large alert then does not pay for the first calls. Fixes #5477 * Four Darling live tests wait for the state they assert (#5480) Four Darling live tests failed intermittently on CI because they read state before it existed, and each now waits for the state it asserts. The query-store literal-end test waits for every other backend to exit before each scan-count snapshot, and flushes its own counters. A late statistics flush no longer lands between two snapshots. The async plan-parse cancellation test cancels from the statement hook at statement 3 instead of racing a 50 ms timer. The log-tail rotation test accepts PostgreSQL 18's repeated lock-wait line, requiring each line to be read once with one identity. The csvlog test waits until both of its own planted FATAL entries are readable. Tests only. Every assertion each test had is still there. Before every DROP DATABASE WITH (FORCE), the test drops of scratch databases now unschedule the database's TimescaleDB jobs. They then wait up to 10 s for running job workers to leave. A census test requires that call at every such drop. The CI workflows warn when the PostgreSQL log holds "terminated by exception". * PLAN_REGRESSION reads per-day totals for closed days instead of re-reading 14 days on every run (#5448) (#5467) Darling's plan regression check reads stored per-day totals for closed days (#5448) The PLAN_REGRESSION fact re-read and re-added 14 days of Query Store intervals on every analysis run. On the busiest server of an 87.4 M-row store (4.67 M rows), one run took 8.6 s and 986 K buffers. Darling now builds each closed day's totals once per server and keeps them. The fact reads those totals for the closed days and reads only the last three days live. On the same server, a run takes 1.28 s and 338 K buffers. One closed-day build takes about 1.1 s. A late Query Store row for a closed day marks that day, and the next build redoes it. The built-days read and the daily read run in one read-only repeatable-read transaction, so both see the same days. The comparison window now starts at the beginning of a day, and the regressed-queries drill-down uses the same start. A read that cannot use the totals falls back to the old read. That fallback is logged and records no collection caveat, because nothing is missing. Expired days are removed one key at a time through the index. The store moves to schema version 168. Fixes #5448 * Keep logical reads in hourly rollups so long Top Queries and Top Procedures rankings can use them (#5472) Top Queries and Top Procedures can rank by reads from new hourly rollups (#5329) The old hourly rollups kept no logical reads, so a reads ranking past the raw tier's roughly 90 hours had nothing to read. The hourly CPU, duration and executions lists showed no reads either. The store now creates `collect.query_stats_io_hourly` and `collect.procedure_stats_io_hourly` from the ensure sweep on the next start. Each copies its matching interval rollup and adds sums of logical reads, physical reads and logical writes. They stay beside the interval rollups rather than replacing them, because the readers never use the daily tier. That costs a second hourly refresh and its storage. On a large store, the first fill of one day took 38.5 s for queries and 22.8 s for procedures. When an io rollup reaches the start of the window, the Darling web app, the MCP tools and the Darling viewer rank every list from it. Those lists then show reads, physical reads and writes. Until then, a reads ranking uses raw, or the rollup that reaches further back, and says which one answered. A reads ranking that a filter forces to raw names that filter. The MCP precision note names the relation that answered. On an hourly window the viewer banner and the MCP note name the window edges. The start edge is per server, so a server added later shows its own first hourly bucket. An empty hourly grid names no start. The rollup floor cache reads each earlier floor with one bounded statement (0.026 ms cold on a large store). On upgrade the new rollups fill from the raw data still kept, so a 7-day window stays partial for about 6 days. `--backfill-rollups` fills them down to raw's oldest row at once. Fixes #5329 * A guard miss fails the pull request in about 6 minutes and skips every shard (#5471) A new Guard tests job now runs first on a pull request. It runs the classes tagged Stage=Guard in both test suites. These are about 140 source pins, censuses and hygiene scans. A guard miss now fails in about 6 minutes with every shard skipped, instead of after the whole run. On a pull request, the PostgreSQL shards, the Lite shards, the Linux build and the whole-tree guards start only after it succeeds. The build check fails on its second step, and the PostgreSQL and Lite result checks report "Guard tests failed". The result jobs also stop treating a skipped matrix as a pass. The results of push, merge queue and release runs are unchanged, but those runs wait for the guard job before the shards start. A census class in each test suite checks the tagging. It fails any guard-style class, or any class that walks the whole repository, that is not tagged. The only way out is a short exemption list with a reason for each entry. Each census also compares its scan with the set xunit really selects. That set is read off the test assembly the way xunit discovers it. Method-level traits, constants, linked files, static and non-public methods, and static classes all count. The no-store rule applies to it too. The guard job fails a skipped test, so a tagged test that needs a store cannot pass by skipping. Part of #5459 * Darling's memory launch guard releases after a garbage collection, reads resident memory on Linux, and alerts keep running while it holds (#5481) Darling's memory launch guard now releases itself. When the figure crosses the line, it runs a full garbage collection at most once a minute and measures again. A crossing within a minute of the last collection holds new launches off quietly until the next one. A figure still over after a collection holds them off, with one Critical per episode that names the figure or says the collection failed. On Linux it measures resident memory against the tightest limit of its own cgroup and every parent, and --check-settings reads the limit the same way. Collection Stopped, the other store-polled self-alerts and the custom-alert rules run in their own pass every loop tick, at most four servers at a time. Collection Falling Behind counts each collector slot that comes due during a hold as skipped, once. The first launch after the release counts none of them again. Fixes #5479 * An RDS or Aurora target can name the AWS role Darling assumes for it (#5468) Each PostgreSQL server in Darling can use its own AWS role (#5452) An Amazon RDS or Aurora PostgreSQL target can name the AWS IAM role Darling assumes to reach it, with an optional external ID. When it is unset, Darling uses the process's own AWS credentials, as before. The server registry gets the role ARN and external ID columns in rung V169, plus a stored flag that says whether an ID is set. One validator, `AwsRoleSettings`, checks both values. darling.json takes `awsRoleArn` and `awsExternalId`, and they seed a new row only, as every other field does. The darling.json drift report compares `awsRoleArn` with the stored role, and never compares the external ID. The web Servers page, the MCP `add_servers` and `edit_server` tools, the desktop viewer and `--add-server` can set the role too. The web, MCP and desktop viewer can set only a role that the new `allowedAwsRoles` setting lists or that a darling.json server names. That setting takes exact role ARNs or account ids, and an account id can carry an `aws-cn:` or `aws-us-gov:` prefix. The service checks the same list each time it assumes a role. The service assumes the role through AWS STS. It uses the role for the host check, instance CPU (RDS and Performance Insights), plan capture, deadlocks and log events. A role it cannot use shows on that server's collection health, with a message that names the role. Read-only roles see the role and the flag but never the external ID. The web, MCP and viewer never show a stored external ID back. The viewer keeps the stored ID unless a new one is typed, the Clear box is ticked, or the role changes. The edit function answers `invalid_value` for a missing required field or any constraint refusal. The MCP and web edit paths say so in a plain sentence. Role and external-ID edits made as the mcp login go through the edit function. The store log masks row values in constraint DETAIL text. It keeps no DETAIL when the store's `lc_messages` is not C, POSIX or an English locale. The runbook and the Darling README describe all of it. Fixes #5452 * The CI shards reuse the Guard tests job's build instead of building the test projects again (#5459) (#5485) CI: the PostgreSQL shards, the Lite shards and the whole-tree guards job now fetch the test projects that the Guard tests job already built. They no longer restore and build them again. The Guard job uploads one artifact per suite (Darling and Lite) with a small origin file, and each consumer checks that file before using it. With no artifact, the old restore and build still run. The Guard-stage classes no longer run a second time in the shards. Coverage is unchanged. Darling shards ran 28030 tests before and 26727 now. The Guard job grew from 1303 to 1307 (the four new workflow-pin tests), so 26727 + 1307 = 28034. Lite is 8543 + 372 = 8915, which equals the old shard total exactly. Runner time drops from 174.7 to 135.2 minutes per run (about 39.5 minutes, 23%), and wall clock from 36.1 to 30.4 minutes. Refs #5459 * Large alerts and MCP and web reads in the first seconds after a start wait for the filter's warm-up, and the large-document tests run it (#5478) (#5484) The large-document statement filter tests now run the product's own start-up warm-up once per test process. The first test to walk a 4 MB report no longer pays for cold, unoptimized code. No budget, constant or assertion changed. At start the product runs a warm-up of the statement filter. Lite reads its stored alerts almost at once. A call that judges 256 KB or more of text now waits for a running warm-up. That covers an alert, an MCP read and a web read. It never waits past 3 seconds after the warm-up started. The deadline is shared. A slow warm-up costs the process 3 seconds once, not once per call. Smaller calls never wait. The alert engine's fire path awaits the warm-up instead of blocking its thread. The warm-up's own calls never wait on themselves. A finished or failed warm-up delays no one. Refs #5478 * The Query Store backfill stops retrying databases that were dropped from the server (#5483) (#5486) The Query Store backfill now stops trying databases that were dropped from the monitored server, in both Darling and Lite. It chose its databases from the Query Store rows already collected, so a dropped database stayed on the list until its last row aged out. Every pass then failed on it with "database does not exist". A failed attempt never touched the saved state, and the hourly clean-up already removed that state, so those were not the cause. The backfill now drops a database from its list when the newest database-state snapshot does not name it. It also requires that no row for the database was collected since that snapshot. A database created after the snapshot is still worked. A server with no snapshot, or one whose newest snapshot is older than three times its database-state interval (at least one hour), is left alone. A failed check keeps every database and logs one warning for each run of failures. Darling and Lite share the SQL text and the rule. Fixes #5483 * Lite.Tests records per-test CPU time beside wall time (#5208) (#5487) Lite.Tests now records the managed CPU time of each test as a `cpu-ms` attachment in the -xml results. It also logs one coverage line per run (charged CPU, process CPU, ratio), with a sidecar `.cpu.json` beside the XML. `lite-timing-top.py` shows a CPU column and a top-10-by-CPU list, and `lite-shard-pack.py backtest` reports the CPU-weighted cut's error beside the wall-weighted error. This is measurement only: no test is skipped, filtered or re-ordered, and `pack` still cuts by wall time. On CI the hook adds no overhead above run-to-run noise. About 65% of process CPU is charged to tests (the rest is native DuckDB threads and work off the execution context). Refs #5208 * The #5448 late-row live test passes from 00:00 to 04:00 UTC (#5494) The plan regression late-row live test failed every day from 00:00 to 04:00 UTC. Its seed built the window's days counted back from today. The window starts four hours before now, so in those hours its first day is yesterday minus 14 and was never built. The seed now counts the built days from the date of the window start. Only the test changes. The product already reads an unbuilt day live, by design. The seed takes an optional fixed clock. The late-row test runs once on the real clock and again at 00:30, 03:59, 04:00 and 12:00 UTC on a fixed date. Every assertion is kept. * "Collection Stopped" alerts once per outage instead of once per cooldown (#5489) (#5490) A server whose collection stayed dead got a new "Collection Stopped" alert every cooldown. It is a state, so it now alerts once per outage and writes one "Collection Resumed" row when it recovers. A standing outage repeats only when connection_refire_minutes is above 0, every that many minutes. An alert no channel took is still sent again after the failed-send back-off. After a service restart the first sweep sends one alert for a server that is still down. Other self-alerts keep their per-cooldown repeat. SECTION: Fixed ENTRY: - **"Collection Stopped" alerts once per outage** ([#5490]) - A server with dead collection got a new "Collection Stopped" alert every few minutes. That filled Alert History. It now alerts once per outage and writes one "Collection Resumed" when it recovers. The `connection_refire_minutes` setting decides how often a standing outage repeats, and 0 (the default) means never. Fixes #5489 * procedure_stats fetches plans only for plans the host has not stored, by default (#5158) (#5502) The procedureStatsDeferredPlanFetch setting now defaults to on. procedure_stats renders and ships a module plan only when the collector has not already stored it, which cuts plan-rendering CPU on the monitored servers. Azure SQL Database and hosts with plan capture off keep the inline path, and "off" in darling.json restores the old behavior. The sample config, README and tests now describe on as the default. SECTION: Changed ENTRY: - **procedure_stats now fetches plans only for plans the collector has not stored** ([#5502]) - The procedureStatsDeferredPlanFetch setting now defaults to on, so a plan the collector already holds is not rendered on the monitored server and sent again. This cuts plan-rendering CPU on the monitored servers. To restore the old behavior, set procedureStatsDeferredPlanFetch to "off" in darling.json. Azure SQL Database stays off whatever the setting says. Refs #5158 * Azure SQL Database sizes cover every database, and a firewall block names the firewall (#5498) (#5500) Database size collection on an Azure SQL Database server now runs once per user database, so every database reports its own data and log files on the first run. The old master-only sibling arm is gone. Rows already stored stay in history. Storage Growth leaves the old whole-database rows out of its 7-day and 30-day sides, so an Azure database's log file does not read as growth after the upgrade. A database with no comparable older row shows no growth figure until it has 7 days of new rows. The 40615 error now says first that the firewall did not allow the client's IP address, and to add a server-level rule or a database-level rule in master. Setting a database is given only as the fix when a login can open a user database but not master. SECTION: Fixed ENTRY: - **Azure SQL Database sizes now cover every database, and the firewall error names the firewall first** ([#5500]) - Database sizes on an Azure SQL Database server now cover every database, not just master. The 40615 error says first that the firewall did not allow the client's IP address, and to add a server-level rule or a database-level rule in master. Storage Growth shows no growth figure for an Azure database until it has 7 days of new rows. Fixes #5498 * One selection script for the test jobs, with a failure-corpus replay guard (#5503) The CI test-selection rules now live in one script, .github/scripts/ci-select.py, which the Lite shard scope step and the build job call. A trimmed record of a month of failing runs is committed. A Guard-stage test replays it on every pull request. Each class that failed before must still be selected for the files of the run it failed in, or the test fails. The build job's no-store Darling run now keeps only the Guard stage while the PostgreSQL shards run, since they run every Darling class already. That saves about 7.7 runner-minutes on roughly nine in ten pull-request runs. The replay finds no failing class it stops covering. Refs #5459 * Tests that read the clock twice now read it once (#5496) (#5499) Some tests failed only when a run crossed midnight UTC. One test read the clock twice, and the two readings landed on different days. This checks every test file that anchors on a calendar day. It fixes the six that read the clock more than once. Some also mixed a local clock with a UTC date. Each now uses a single reading. The seed-start helper in the rollup floor test takes the test's one instant, with cases just before and after midnight. The 17-day trigger edge test now reads the database clock, because that is the clock the trigger uses. If that clock is within 10 seconds of the next midnight, the test waits it out first. A test with an injected clock covers the wait path. One product reader that reads its own clock is listed in the pull request as needing a seam. No product code changed. Fixes #5496 * Lasting-state self-alerts send one alert per occurrence, not one per cooldown (#5493) (#5497) Lasting-condition alerts send once per occurrence instead of once per cooldown. Darling covers Capture Down, Agent Not Running, AG Sync Fell Behind, custom rule health, collection falling behind, store job over cadence, retention held, raw purge over horizon and a policy job stuck after escalation. Lite covers AG Sync Fell Behind. The repeat setting (connection_refire_minutes) now covers all of them, and its label and tooltip in the Darling viewer and Lite settings windows say so. Each sends one alert when the condition starts and one resolution when it ends. It repeats only when connection_refire_minutes is above 0. An alert no channel took is still sent again after the failed-send back-off. A rise to a higher severity inside one occurrence (warning to critical) still sends one more alert. A custom alert rule joining the unhealthy set sends one alert that names the new rule. The shared engine's database state repeat is unchanged. SECTION: Fixed ENTRY: - **Darling and Lite alert once per occurrence for lasting conditions** ([#5497]) - Capture Down, Agent Not Running, AG Sync Fell Behind (Darling and Lite), custom rule health, collection falling behind, store job over cadence, retention held, raw purge over horizon and a policy job stuck after escalation sent a new alert every cooldown for as long as they lasted. Each now sends one alert when it starts and one resolution when it ends, and repeats only when connection_refire_minutes is above 0. A rise to a higher severity inside one occurrence (warning to critical) still sends one more alert,. So does a custom alert rule joining the unhealthy set. Fixes #5493 * Flaky test classes: the statement filter tests no longer depend on the runner's speed (Refs #5459) (#5504) This branch fixes the flaky test classes the first W5 pass left. Most were already fixed on the base, and the pull request table gives each class its cause and proof. Users see no change. The product code gains two test seams with the same default: one in the alert statement filter, one in the collector context. Fixed here: the statement filter judges text under a wall-clock budget, and a slow runner ran it out. Its tests now pin a budget that cannot run out, and other tests prove that a spent budget withholds. The snapshot worst-case test and the slow-judge test use the same idea, so a busy machine cannot fail them. The CPU accountant tests allow one stray scheduler tick on the idle side, because the counter moves in 15.6 ms steps. The install-tree helper process now outlives the child's limit. The DuckDB trim test waits for a cycle that really ran, and the encrypt-password tests wait 180 s on their children. Already fixed on the base: the rollup backfill cancellation, the wide-grid round trip and the store-roles "60s" check. The same goes for the log-tail rotation tie, the store-host chunk count, the upgrade revert token, the managed-store directory move and the deploy probe limit. The PostgreSQL classes passed on a throwaway rig. The Guard stage passes 1372 of 1372 on Darling.Tests and 372 of 372 on Lite.Tests. The census and ratchet classes pass. Refs #5459 * Lite FinOps: first visit loads real data, and the advice only claims what the data supports (#5488) Lite's FinOps tab opened on the empty window read before the first collection, and some of its advice claimed more than the data supported. The first show after start-up always reloads the selected server. Later shows reload when the last load is 30 seconds old or more. A tab left open reloads when a collection for its server finishes. A database counts as idle only when query stats hold a sample on each of the 7 complete UTC days before today. The oldest sample must also be at least 7 days old. That rule drives the recommendation, the Idle Databases grid and the Server Inventory Idle DBs column. CPU right-sizing gives one row per server, and each rule checks its own window. Free editions get no licensing advice. Storage Growth uses the sample nearest each 7 and 30 day mark, or shows n/a. The Utilization badge and the Server Inventory Health column show a dash when there is no CPU sample. Column headers show in full, Database Sizes is titled with the selected server, and recommendation details wrap. List rows, tabs and filter buttons carry names for screen readers. In Settings the poison-waits label is fixed, the CPU box shows in full, the window fits the screen, and Save and Close stay visible. The Overview Last Collect time follows the timestamp setting on the server's own clock. Top Queries and Query Store no longer list Lite's own collector queries. Grids and charts that have their own empty message no longer get a second text over it. The generic empty text reads "No data for the selected time range." The tab rows stay in place when a tab is selected. SECTION: Fixed ENTRY: - **Lite's FinOps tab loads real data on the first visit, and its advice claims only what the data supports** ([#5488]) - The tab opened on the empty window read before the first collection. It showed "No Data", empty grids, a false idle database row and a Health score of 100. The first visit now loads the selected server, and an open tab reloads when a collection for its server finishes. A database counts as idle only with a query stats sample on each of the 7 full UTC days before today. CPU right-sizing gives one row per server, and Free editions get no licensing advice. Storage Growth shows n/a when no sample sits near the 7 or 30 day mark. The Settings window now fits the screen, with Save and Close always visible. Top Queries and Query Store no longer list Lite's own collector queries, and list rows, tabs and filter buttons carry names for screen readers. * Lite-only pull requests run the Darling classes that read Lite files, not the whole suite (#5505) Pull requests that change only Lite files now run fewer Darling tests. Darling test classes that read a Lite file carry a Reads=Lite trait, derived from the class source by a census and applied by a script. On a Lite-only pull request, the build job and the whole-tree guards job run the Guard stage plus those classes. They no longer run the whole Darling suite. Pushes, merge-queue runs and releases still run everything. The selection rule lives in the CI selection script. The replay of past failing runs shows the rule skips none of the classes that failed. Refs #5459 * Lane orders no longer tell lanes to add attribution trailers (#5535) The lane-orders skill told each lane to end its commits with two attribution trailers, a co-author line and a session link. That goes against the 2026-10-01 ruling of no attribution lines, and three lanes followed it this wave. The skill now says no attribution lines in commits or PR bodies, and drops the two placeholders that a brief no longer fills. Doc-only change to .claude/skills/lane-orders/SKILL.md. No build or tests involved. * A manual-only nightly job probes AltCover per-test coverage for the test-class map (#5459) (#5512) The nightly workflow gets a manual-only job. It runs a sample of 100 Darling and 60 Lite test classes with AltCover coverage on, plain and instrumented. It uploads the raw reports, a summary and a class-to-file map for each suite. The scheduled nightly never runs it. On the sample, AltCover worked on .NET 10 with xUnit v3 and added no measurable run time at method level. 98 percent of the Darling product files that ran, and all of the Lite ones, were tied to a test class. The job also puts skips that instrumentation turned into failures back to skips, and a script in .github/scripts turns the reports into the map format. Refs #5459 * A managed store near a free-disk step no longer reads stale after a hardware change (#5459) (#5511) A managed store near a free-disk step of 16, 32, 64 or 128 GB read "stale after a hardware change" for max_wal_size when nobody had touched it. The check derived the value again from a newer reading of the volume and compared for equality. Ordinary disk churn across a step flipped it. This is what failed once in the managed-conf verdicts live test. The settings check, the stored verdicts and the service start now share one tolerance. A stored max_wal_size that the size ladder picks for any reading within one eighth of the current free space counts as current. A start near a step keeps the block it wrote instead of appending a new one. Halved or doubled headroom and a new PostgreSQL major still count as changes. SECTION: Fixed ENTRY: - **A managed store near a free-disk step no longer reads "stale after a hardware change"** ([#5511]) - The WAL size check compared max_wal_size against a fresh reading of free space. A few hundred MB of disk churn across a 16, 32, 64 or 128 GB step made an untouched store read stale. It now accepts a value the ladder picks for any reading within one eighth of the current free space. Halved or doubled headroom still reads stale. A service start near a step keeps the value it wrote instead of appending a new block. Refs #5459 * Test selection from a class-to-file map: selection code and Darling test timings (#5459) (#5509) Adds the code that will choose test classes from a map of which source files each class touches. It has a map-select command, a replay of the failure corpus against a map, and a guard-stage test with a small synthetic map. It runs everything whenever the map is missing, old, or cannot account for a changed file. No workflow step calls it yet, so no run selects differently. The Darling PostgreSQL shards also write per-test timings and upload them for 7 days, as the Lite shards do. The later selection can then estimate time per class. Refs #5459 * Wait Stats spinlock read at 7 days no longer sorts every row of the window (#5495) (#5501) The spinlock read totals the window in one pass. It then looks up the newest rows of only the top spinlocks, instead of comparing every stored row with the one before it. A row with no spinlock name is still left out. For windows of 6 hours or more, the PostgreSQL I/O read takes whole hours from a new hourly table (collect.pg_io_stats_hourly, store schema 170). It reads raw rows only at the two edges. A count check compares each hour, and the hour before the span, with the raw table. When they differ, the read falls back to the old one. The hourly tick fills the table, with up to 2 minutes of builds per tick. SECTION: Fixed ENTRY: - **The spinlock and PostgreSQL I/O reads no longer go through every stored row of a long window** ([#5501]) - Both reads compared every stored row with the one before it. We measured both on a large store over 7 days. `get_spinlock_stats` went from 3.9 seconds warm (6.8 cold) to 0.22 to 0.26 seconds warm (1.7 cold). `get_pg_io_stats` went from 2.8 seconds to about 0.26 seconds. The spinlock read now totals the window in one pass and looks up the newest rows of only the top spinlocks. For windows of 6 hours or more, the PostgreSQL I/O read takes whole hours from a new hourly table (`collect.pg_io_stats_hourly`, store schema 170). It reads raw rows only at the two edges, and the viewer's I/O tab uses the same read. It falls back to the old read whenever an hour's row count does not match the raw table. After the upgrade, the hourly tick fills the new table. Fixes #5495 * Nightly builds a class-to-file test map from the whole Darling and Lite suites (Refs #5459) (#5539) The nightly now builds a map of which test classes run which source files. It runs the whole Darling and Lite suites under AltCover in 10 shards. It joins them with what the test sources name: paths, tool names, type names and whole-tree readers. The result is uploaded for 14 days. It is on by default, a dispatch can turn it off, and its failure never fails the nightly. Nothing reads the map yet, and no pull request or push run selects differently. The next slice uses it in shadow mode. Refs #5459 * Stale servers read as stale, Collection Stopped counts from the last real collection, and Wait Stats reads spinlocks once (#5492) A server that stops collecting no longer looks healthy in the web viewer or the Darling Viewer. Its fleet card and server header tiles read n/a with the time of the last collection, not 0% CPU and "memory ok". The Availability Groups page says Stale when its data is old, and a group with no primary is never Healthy. The Collection Stopped alert counts from the last successful collection in the store, so a service restart no longer resets it. Long gaps read as "12 days" or "3 hours". The Wait Stats tab asks for the spinlock and latch lists once, not twice. Identical tool reads in flight share one request. Web text and layout are cleaner. The Plan button shows the plan full width under its row. Messages no longer show internal names. On a server without Always On, Collection Health lists a healthy Always On collector as not applicable. A failing, stale or stopped one keeps its log row. Every server pick list on the web uses the sidebar's order. FinOps asks for more history before it calls a database idle. That needs a query-stats sample on each of the last 7 UTC days. A server is over-provisioned only after 24 hours of CPU samples. Developer, Evaluation and Express editions get no Enterprise licensing advice. FinOps works on PostgreSQL targets and opens on the first SQL Server target. SECTION: Fixed ENTRY: - **Offline servers read as stale, Collection Stopped counts from the last collection, and the web pages and FinOps read in plain words** ([#5492]) - A server that stopped collecting showed live-looking values (0% CPU, memory ok, 77 free threads) on its fleet card and header tiles in the web viewer and the Darling Viewer. They now read n/a with the time of the last collection. The Availability Groups page showed "Healthy" on 12-day-old data and for a group with no primary. It now says Stale for old data, and a group with no primary is at least Warning. The Collection Stopped alert text (email and every other channel) counted from the service start, so a restart reset it. Long gaps read as thousands of minutes. The text now counts from the last successful collection in the store and reads "12 days" or "3 hours". The Wait Stats tab sent the same spinlock and latch reads twice per load. Identical tool reads in flight now share one request. The Plan button now shows the plan full width under its row instead of in the last grid column. This covers Top Queries, Query Store history and PostgreSQL captured plans. The Instance CPU panel is hidden on a PostgreSQL server that is not Aurora. A "not collected" answer reads as one short line instead of gate names. Messages no longer show internal names such as hints.captures, hours_back, denied_since_last_success or collector measurement counts. Alert Settings shows a readable label with the key as a tooltip. On a server without Always On, Collection Health lists a healthy Always On collector as not applicable. A failing, stale or stopped one keeps its log row. Collection Health also opens worst collector first, and the Daily Health Calendar opens newest day first. The Admin Servers Auth column reads "Username and password" for PostgreSQL targets. Every server pick list on the web now uses the sidebar's order, display name with favorites first. This covers FinOps, Alert History, Job History, and the Custom View and notebook scope lists. The page header says SQL Server and PostgreSQL. The "read 6 of 9 servers" line under the Deadlocks tile has a tooltip that says why. The Alerting Reads panel drops tiles with no value. Collector counts written as "shred_gated_1 events_read_0" read as words. In FinOps, an idle-database claim now needs a query-stats sample on each of the last 7 UTC days. A server gets one CPU right-sizing recommendation, and only after 24 hours of CPU samples. Developer, Evaluation and Express editions get no Enterprise licensing advice. Storage Growth baselines are the sample nearest 7 and 30 days ago, else n/a with a reason. Utilization and Server Inventory show a dash instead of a health score when there are no CPU samples. On a PostgreSQL target, a FinOps panel that the server cannot answer reads "Not collected for PostgreSQL". FinOps opens on the first SQL Server target. The Darling Viewer's FinOps grids show full headers, a wrapped Recommendations detail, and a dash with a tooltip for an empty savings estimate. The Database Sizes title names its server. REF: [#5492]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/5492 * Query Store backfill keeps each server's database list instead of re-reading it every tick (#5518) (#5534) The Query Store backfill loop read each server's list of databases with recent Query Store rows on every tick, about 100 times a day per server. QueryStoreBackfill now keeps the cut-chunk list per server and serves the next tick from it. The list is read again when the statement text changes (the cut chunk moved), when a Query Store write names a database not in the list, when the list is an hour old on a monotonic clock counted from when the read began, or when it was read beside a write in flight or after an ambiguous failed write. QueryStoreWriteFence, already called by the single COPY path for every Query Store write, now also records the database names each write carries before the transaction opens. The runner passes the batch's names, and the worker hands the backfill the same fence. A removed server's cached list and recorded names are dropped when the worker reconciles the server set. A database created by a writer outside the service can be missing from the list for up to 60 minutes plus one backfill tick. Holes are not affected. Review round 1: live tests pin the runner-to-fence hand-off and the snapshot-before-read order by behavior, the cache age moved to a monotonic clock, the docs state the real bound and the outside-writer case, and removed servers are forgotten. Fixes #5518 * README catches up with everything since v3.9.0 (release checklist step 2b) (#5491) The README now matches the product on dev for everything since v3.9.0. It adds the Darling web dashboard and the twelve FinOps sub-tabs. It adds the Alert History, Job History and Availability Groups tabs, the new MCP tool groups, and MCP network access with TLS. It also adds AWS RDS and Aurora support for PostgreSQL, the diagnostics bundle, and the Linux prerequisites. It also covers the settings that changed since the last release. These are the PagerDuty auto-resolve setting, the collection-gap and heartbeat alerts, and the run time for daily collectors. Another is the setting that decides how often a lasting problem alerts again. The Azure SQL Database troubleshooting line now explains error 40615. Every count and tab list was checked against the code. Release checklist step 2b. * Pull requests skip 27 slow, never-failed test classes unless the change reaches them (#5459) (#5510) Pull request runs now leave out 27 slow test classes that have not failed in 30 days: 15 in Darling.Tests and 12 in Lite.Tests. They carry a Cost=Slow trait. A class is put back when the change reaches it. That means its own source file, a file its source names, a page script for the Node-driven classes, or a test project or build input. Pushes to dev, merge-queue runs, the nightly and releases run every class. Classes that guard passwords, keys, upgrades or shared paths stay on pull requests. A census pins the list and the workflow wiring, and the failure replay still finds 0 misses. When nothing in a change reaches these classes it saves about 21 runner-minutes per pull request run. Refs #5459 * Plan regression day builds get a background index, and the Query Store top summary cleanup stops scanning its table (#5507) (#5524) Each per-day plan regression build read every index entry its server had in collect.query_store_interval_latest, because first_execution_time is the last of seven columns in the table's unique key. The service now builds collect.ix_query_store_interval_latest_server_first_exec on (server_id, first_execution_time) in the background, about 20 minutes after start. It uses the same ensure path as the wide table's twin: CREATE INDEX CONCURRENTLY, skipped with a warning on a hypertable, and an invalid leftover dropped and rebuilt. There is no migration, and the schema version does not change. The same index serves the live half of PLAN_REGRESSION, its table fallback, the drill-down twin and the table floor read. The hourly Query Store top summary cleanup ran one DELETE with OR and NOT IN on both summary tables, which no index could serve, so every tick scanned the whole table. It now deletes the expired built rows first and removes each returned key from the summary by an index probe, in one transaction, following the plan regression daily cleanup. Review round 1: the cleanup's live test now observes GcAsync through the table's scan counters (no sequential scan, at most the due rows fetched, nothing touched when nothing is due). The read pin covers all five reads, a new test proves a failed summary step rolls the built rows back, and three weak assertions and two comments were corrected. Fixes #5507 * Azure SQL Database Query Store collection stops reading each database's watermark from the store every cycle (#5514) (#5527) Azure SQL Database Query Store collection now keeps each database's watermark in the per-database cache, as the on-premises enumerated path already did, instead of reading it from the store for every database on every cycle. The Azure loop resolves the watermark through ResolveQueryStoreDatabaseWatermarkAsync with the same read floor as the enumerated path, stages the batch's contribution after the read, and commits it only after the flush returned. Both fault arms drop the key when a batch was staged, because the write might have committed; a fault before the stage keeps it. The XE ring-buffer collectors on that loop keep their own read, and the hourly reseed is unchanged. Review round 1: a live test drives the real Azure per-database loop over two databases with write and read faults, and the fault-arm comment states when the kept key matters. Fixes #5514 * Analysis reads the newest file I/O and perfmon rows without scanning the whole window (#5516) (#5529) Two analysis fact reads kept the newest row per key by numbering every row in the window with a window function. They now stop inside the newest rows. The perfmon rate facts (Batch Requests/sec, SQL Compilations/sec, SQL Re-Compilations/sec) read one ORDER BY collection_time DESC LIMIT 1 branch per counter, which the (server_id, collection_time) index serves backward. The database size fact runs the new DatabaseSizeNewestSql first: it sums the newest snapshot's files by equality and reports dropped_out, and the old DatabaseSizeSql runs only when a file dropped out of the newest snapshot. The one accepted difference: a database file seen only in snapshots strictly between the newest, the one before it and the oldest is no longer carried in the total, where the old read carried it for up to 24 hours. A live test pins it. A perfmon counter with no usable row in the window can cost up to three times the old read, as the code comment states. Review round 1: the comments state the accepted difference and the perfmon worst case, a live test pins the difference, and stale wording was fixed. Fixes #5516 * Store self-metrics take the dimension row count from the planner estimate, not a 26 s count (#5520) (#5530) The hourly store self-metrics sweep wrote row_count for query_text_dim and query_plan_dim with SELECT count(*), which on a large store read about 150,000 blocks and took about 26 seconds a call. row_count for those two rows is now the planner's estimate, CASE WHEN c.reltuples >= 0 THEN c.reltuples::bigint END from pg_class, the same expression TableInsertSql already uses. A table never vacuumed or analysed maps to NULL. The series moves in steps between ANALYZE and autovacuum runs and rounds above about 16.7 M rows, and get_store_metrics calls it an estimate. Review round 1: a source-scan test backs the claim that no reader computes from a dimension row's row_count, and the comment states the step and rounding behavior. Fixes #5520 * Query Store wide-table reads bound first_execution_time at the window end too (#5523) (#5533) The per-server reads of collect.query_store_interval_wide bounded first_execution_time only from below, so ix_query_store_interval_wide_server_first_exec scanned from the window start to now. Each read now also bounds it from above: the MCP top read and the viewer's Queries grid at the window end plus 12 hours (FirstExecUpperSlack), each edge arm of the MCP long-window read at its own end, and the history tool at its end. The duration trend's interval arm takes first_execution_time in [start - 1 h, end + 25 h] from interval_start_time_utc (IntervalStartSlack, IntervalStartFirstExecMargin), and its legacy arm adds the 12 hour upper bound. The per-day build was already bounded and is unchanged. The 12 hours covers the clock difference between the monitored server and the service plus a long collection cycle. Rows from a server whose clock runs more than 12 hours ahead of the service can drop from a closed window. The comment that called a clock running ahead unsupported is corrected: the catch-up clamp only raises a watermark that is too old. No index is added. On the field store the existing index serves a window that ends now at about 2 index entries per row returned, and one query's 3-day history read is 1.7 s cold. Fixes #5523 * A managed store near a WAL size step no longer rewrites its settings file on a restart (#5459) (#5544) A managed store near a WAL size step (16, 32, 64 or 128 GB free) rewrote its managed settings file on a restart. Nothing else had changed. The file derives max_wal_size and min_wal_size from free space. A reading on the other side of the step gave another size and another body. The tolerance from #5511 covered the settings check, the stored verdicts and the postgresql.conf block, but not this render. The render now takes the size already in the file. It keeps that size while it is within one eighth of the free space, the same rule as #5511. Halved or doubled headroom still moves it, and a hand-edited file is never read for it. Tests plant the free space 200 MB on each side of every step, in a unit test and in the live two-start test. SECTION: Fixed ENTRY: - **A managed store near a WAL size step no longer rewrites its settings file on every restart** ([#5544]) - The managed settings file derives max_wal_size and min_wal_size from the data volume's free space. A volume within a few hundred MB of a 16, 32, 64 or 128 GB step got another size on the next service start. The file was rewritten with nothing else changed. The file now keeps its size while it is within one eighth of the current free space, the same tolerance the settings check uses. Halved or doubled headroom still moves it. A hand-edited file is still left alone. Refs #5459 * Release checklist: Azure SQL Database skips and the forced firewall outage, as tested for 3.10.0 (#5545) Step 5 of the release checklist asked for SKIPPED rows from collectors that do not apply to Azure SQL Database. Darling marks the SQL Agent collectors (running_jobs, job_history, agent_status) not applicable when the server is added and never runs them, so they leave no collection_log row and no health entry. Both 3.10.0 test runs flagged that against the old wording. The step now describes what really happens, and the success line says no inapplicable collector may be recorded as a failure. The firewall churn step already warned that deleting the rule does not break open connections. The 3.10.0 runs showed that twice. The step now says how to force a real outage: open admin sessions first, delete the rule, wait 6 minutes, then KILL Darling's sessions in each user database (KILL is denied in master). Collectors that open new connections then hit 40615. The log message must name the firewall, and the collectors must store rows again after the rule comes back, without a service restart. Doc-only change to .claude/skills/release-checklist/SKILL.md. No build or tests involved. * The start-up check for gaps in the rollups finds a missing day with an index seek, not a whole-chunk read (#5521) (#5538) The start-up materialization hole scan probed each candidate bucket with m.bucket = b.bucket. On a chunk with few distinct bucket values the planner estimates that equality from n_distinct as matching a large share of the chunk, so it costs a Seq Scan under EXISTS as nearly free and reads the whole chunk when the bucket is missing, which is exactly the case the scan looks for. The probe is now m.bucket >= b.bucket AND m.bucket <= b.bucket: the same rows, but PostgreSQL gives a range with two non-constant bounds its default selectivity, so it seeks bucket_idx. The legacy and successor probes in LegacySuccessorHoleBodySql get the same pair. Field store, one missing day on the daily Query Store interval rollup: 11.2 s and 240,137 buffers before, 0.89 ms and 3 buffers after (Index Only Scan). The whole scan: 14.2 ms to 10.4 ms on the daily rollup, 107 ms to 105.5 ms on the hourly one. A live test checks that the probe uses the index for a missing bucket, and a pin test holds the range text. Fixes #5521 * The Fleet page and the fleet overview total PostgreSQL deadlocks without ordering databases whose counter never moved (#5526) (#5536) The fleet PostgreSQL deadlock totals (DarlingFleetReader.FleetPgDeadlockSql for get_fleet_overview and /api/fleet, and the viewer's FleetTotalsSql) ran one LAG window partitioned by server_id and database_name over every server's rows in the window. A (server_id, database_name) pair whose counter has the same value in every sample adds nothing to the total, so the reads now find each pair's row count, counter count, min and max in one unordered aggregate. A flat pair adds n - 1 intervals and nothing else. Only the pairs whose counter moved go through the ordered LAG, one server at a time by server_id. The Fleet page total keeps only the pairs with min <> max. Counter resets, NULL counters and NULL database names give the same totals as before. Store with 50 PostgreSQL targets, 168 hours (the most either surface accepts): 433 of 443 pairs flat; fleet overview 6.8 s to 0.68-0.77 s, Fleet page total 6.7-6.8 s to 0.64-0.80 s; old and new return the same rows. A live test compares the old SQL against both new reads on hand-worked and generated seeds, and pin tests hold the shipped text, including the gate that skips the second read for a server with only flat pairs. Fixes #5526 * The module map refresh at service start and in the daily sweep reads from its watermark, not two days (#5519) (#5528) The module map refresh read a flat two days of procedure_stats (RefreshSql) at every service start and in every daily sweep, although the map already carries a watermark that the hourly path keeps within about an hour of the table. Service start now calls DarlingModuleMap.RefreshAtStartAsync, and both start-up and the daily sweep read from DailyRepairSlack (30 hours) behind the watermark, capped at the two-day lookback. The full two-day read stays for an empty map, a map with no watermark, or an unreadable one. The 30 hours cover the daily's own 24-hour cadence plus a margin, so a row committed late behind the watermark is still repaired before raw procedure_stats ages out. Start-up and daily refreshes log under their own labels, and the hourly and start/daily paths share one private method. Review round 1: the span went from 6 to 30 hours, start-up uses the same span, and a live test proves a late-committed row stamped a day before the daily is repaired by both the daily and the start-up refresh. Fixes #5519 * Custom Views and custom alert rules scope server reads by server_id, not server name (#5525) (#5537) Custom View and custom alert rule reads (the Compose compiler) filtered fact tables by server_name = ANY($n), which on compressed TimescaleDB chunks cannot use the server_id segment index. One shared predicate, ServerScopeSql.Predicate (called by ComposeCompiler.ServerScope), now scopes by server_id = ANY(ARRAY(SELECT reg.server_id FROM collect.servers AS reg WHERE reg.server_name = ANY($n))). It covers the raw fact read for every source table, the Query Store dedupe, the procedure module CTE, the hourly-raw-edges arm and rollup routes, the annotation query, the server clock read, and the hourly-raw-edges count guard. Every value stays a bind parameter. ComposeServerScope.FindUnregisteredAsync finds scoped names with no registry row before the panel or rule runs; those are added as OR server_name = ANY($u), and a failed lookup logs a warning and treats every name as unregistered. The alert sweep looks names up once per server. A scope name means the server or servers the registry holds under that name now: a renamed server's scope includes the rows it stored under its earlier name, a name reused by a new server means the new server, and a name held by two registry rows means both. collect.module_map stays scoped by name, so on the hourly-raw-edges route a renamed server's older procedure handles held only by the map read (ad hoc). Review rounds: the count guard takes the same scope as the read, live tests and a source pin fail if a lookup call is removed, the every-emitter test checks every statement, and the semantics and changelog state the name-reuse case. Fixes #5525 * Query Store reads on the stored interval tables no longer miss an interval that began just before the table's floor (#5541) (#5543) The Query Store read gates (QueryStoreIntervalLatest.UseTable and QueryStoreIntervalWide.UseTable) took the stored interval table when raw's floor was at or above the table's floor. Raw purges by collection_time and the interval tables purge by first_execution_time, so raw could still hold a snapshot of an interval that began below the table's floor, and the read left it out. Clause 3a is now raw floor >= table floor + PurgeEdgeMargin (26 h: a 1-day interval span, the 60-minute slowest cadence and the 1-hour catch-up cap) in both rules. The Wide rule's clause 3b uses window start - PurgeEdgeMargin, which closes a one-hour band reachable on default retention. The table floor always comes from the per-server MIN(first_execution_time), because neither interval table is a hypertable in production. The chunk-catalog read is renamed RawChunkFloorSql and returns only raw's floor. Comments that said one day now say 26 hours. Fixes #5541 * A quiet server's job history and default trace watermark read tries the last week before its whole slice (#5515) (#5532) When the 6-hour watermark probe finds no row, the watermark reads now try the last seven days (WatermarkPolicy.WidenedWatermarkWindow) before they read the server's whole slice. This covers GetLastCollectedTimeAsync (default_trace_events and the other timestamp watermarks), GetLastCollectedInstanceIdAsync (job_history), the cpu_utilization pair read and the per-database read used by the Azure deadlock, blocked-process and long-query collectors. They run the same bounded statement with an older bound, so no new SQL text. A hit in the week is never newer than the unbounded MAX and equals it for every column that rises with collection_time. job_history's run_datetime can come back older, which only widens the next re-read, and dedupe drops the repeats. Review round 1: the docs no longer say "identical", the cpu pair and per-database reads take the week step too, and two live tests count their unbounded reads. Fixes #5515 * Shadow mode for the test-class map: a pinned map, the selection in the step summary, and a shadow-check row (#5459) (#5540) Shadow mode for the test-class map. Each pull request run pins the newest successful nightly map built from an ancestor of the base. It prints the classes that map selects, and their estimated seconds, in the step summary. A shadow check after the shards writes one row per run and lists each class that failed but is not in the selection. Every shard still runs every class. A missing, old or unreadable map gives the full selection and a notice. The replay can now list every miss. The map rules take more files where the replay found holes. They count repo paths named in comments and test files that name another test class. They also count partial types spread over many files and documentation files a test reads by name. The one hole class is now mapped. The nightly map shards and the AltCover probe now wait for the build and PostgreSQL jobs, so they stop taking their runners. Refs #5459 * Release 3.10.0: version, changelog, archive (#5548) The version goes from 3.9.0 to 3.10.0. In the CHANGELOG, the entries for PRs merged since 3.9.0, through #5536, are spliced into a new [3.10.0] - 2026-10-08 section. Entries that belong together are folded, so the section holds 257 entries. The prose moves to docs/changelog/3.10.md, and the archive census gains its row. An Important entry covers the Darling store migrations V158 through V170, which run on the service's first start. None of them rewrites a large table or needs an offline window. The upgrade-path test also gets its fixture for the 3.9.0 release, migration-ladder-v3.9.0.sql, generated from the v3.9.0 tag. Release checklist steps 1, 2, 2a and 4 for 3.10.0. * Darling tests: the nightly's PostgreSQL step no longer fails after every test passes (#5546) The nightly's Darling PostgreSQL test step exited 1 after every test passed. xunit.v3 under Microsoft.Testing.Platform waits 1 s after the run returns, prints "Waiting 10 seconds...", and exits 1 if the process is still alive 10 s later. ScratchPostgres's process-exit drain was still dropping scratch databases that seven live classes never disposed, and since #5480 each drop also quiesces the database's TimescaleDB jobs, so the drain outgrew the window. The seven classes (RawGatePerSlotSourceFloorLiveTests, RawPurgeConvergeLiveTests, RawPurgeTriggerGateErrorTests, RawPurgeTriggerLiveTests, RawRepairEpochTriggerLiveTests, RawTablesLeaveCatalogSweepLiveTests, RetentionHoldAndCadenceRawRedirectLiveTests) now dispose their scratch databases in each test. A new guard-stage census, ScratchDatabaseDisposalCensusTests, fails for a test that mints a scratch database and never disposes it. The exit drain runs clusters side by side and up to 16 drops at once under one 6 s deadline (ExitDrainBudgetSeconds). It returns at the deadline even with a drop in flight, never cancels a sent DROP DATABASE, and names each database it did not finish. It keeps the dead-cluster probe and the quiesce before each drop. On four hanging clusters the old drain took 12.0 s and the new one 3.0 s. Nightly run 37746531776 with both commits passed: 28,773 tests, 0 failed, no FATAL line, and no database left for the exit drain. Refs #5480 * Release checks: small fixes from the final walk of Lite and the Darling Viewer (#5542) Lite's Collector Schedules list no longer looks contradictory. Status reads "Uses default" or "Own schedule", both column headers have a tooltip, and a Custom preset has a tooltip that names the collector intervals that differ from the closest preset. The Health column in the Server Inventory (Lite, Darling Viewer, web page and MCP tool) shows the same score as that server's Utilization view. Before, it showed almost the same number for every server. The Darling Viewer's Overview card shows the date of a collection that is not from today. The Alert History and Job History time columns name the clock they read on, and Alert History says when it shows only the newest 500. The Overview search also narrows the fleet roll-up above the cards. An Availability Group on old data reads Stale, not Healthy, in the Viewer and Lite. Screen readers now read column headers, filter buttons, empty cells, grid rows and server list items by name in Lite and the Viewer. Before, they read a class name or a bare glyph. The FinOps Utilization view shows dashes for Avg, P95 and Max CPU when a server has no CPU sample in the last 24 hours. It hides the database sizes chart then, and the web page and MCP tool return null for those three figures. The Storage Growth baseline rule from #5492 is now pinned with tests. SECTION: Fixed ENTRY: - **Small fixes across Lite and the Darling Viewer from the final release walk: Collector Schedules wording, Server Inventory health, time columns, screen-reader names and FinOps dashes** ([#5542]) - Every server showed Preset "Custom" next to Status "Default", which read as a contradiction. Status now reads "Uses default" or "Own schedule", both column headers have a tooltip, and a Custom preset has a tooltip naming the collector intervals that differ from the closest preset. The Server Inventory Health column in Lite and the Darling Viewer, web page and MCP tool now shows the same score as that server's Utilization view; it used to show almost the same number for every server because it scored average CPU against fixed memory and storage values. The Darling Viewer's Overview card shows the date of a collection that is not from today, the Alert History and Job History time columns name the clock they read on (and Viewer Job History rows read on their own server's clock), Alert History says when it shows only the newest 500, the Overview search also narrows the fleet roll-up above the cards ("Monitoring 2 of 9 servers"), and an Availability Group on old data reads Stale instead of Healthy in the Viewer and Lite. Screen readers now read column headers, their filter buttons, empty cells (text or templated), grid rows and server list items by name in Lite and the Viewer (they read as "System.Windows.Controls.StackPanel", a bare glyph or a type name before; a row now reads as its first two visible columns, such as "Time 06:01:12, Server example-sql-01"). The FinOps Utilization view in Lite and the Darling Viewer shows dashes for Avg, P95 and Max CPU when a server has no CPU sample in the last 24 hours (they read 0.00%, 0.00% and 0%) and hides the database sizes chart then, and the web page's Utilization read and the MCP tool return null for those three figures. The Server Inventory dash in place of a health score now has a tooltip that names both reasons for it (no CPU sample in the last 24 hours, or no memory sample collected), and Lite's Alert History count follows a column filter, as the Darling Viewer's does. * Darling tests: a scratch database drop no longer hands its backend to the next test, and the retired-baseline race no longer runs a refresh beside the drop (#5550) A PostgreSQL backend crashed with exception 0xC0000005 during a CI run of the Darling tests. It was running a TimescaleDB refresh of an aggregate that the retired-baseline sweep test was dropping. The same backend had just run a scratch DROP DATABASE ... WITH (FORCE) on a pooled admin connection. ScratchPostgres now creates, drops and sweeps scratch databases on unpooled connections. A backend that ran a DROP DATABASE ends with its connection and never runs a later test's job. Every other test that drops a PostgreSQL database does the same. ScratchDatabaseSweep now stops an abandoned database's TimescaleDB jobs and waits for its job workers before one plain drop. A new guard-stage census, UnpooledDropConnectionCensusTests, checks every DROP DATABASE in the tests. It fails on a pooled connection, or when the method does not stop the database's jobs first. Each exception is an allow-list entry that names the method and its reason. The retired-baseline race test is deterministic. The refresh job runs for real before the sweep. Then an open transaction holds ROW EXCLUSIVE on the materialization hypertable, in place of the next refresh. The first drop attempt gives up behind it after 500 ms, and every earlier assertion stays. The fixture's refresh policy is parked a day out, so the scheduler never starts it during the drop. The test opens its connections with the product's search path. It also qualifies run_job with the extension's schema, which fixes a 42883 error the old test swallowed on a fresh database. The CI clusters log the session id (%c) on every line. Nightly runs 37757915982 and 37757919141 on the first commit passed: 28,862 tests, 0 failed, and no crashed backend. The product sweep's own race with a refresh is #5551. Fixes #5549 * Lite's Job History reads only the selected time range, and tab loads read off the UI thread (#5457) (#5553) Lite's Job History read now applies the selected time range before it removes duplicate rows between the live data and the archive. The duplicate key includes the run time, so DuckDB pushes the range into the table and archive scans. One server's 4-hour read dropped from 4 to 8 seconds and 600 to 730 MB. It now takes under a second and under 300 MB, with the same rows. Some reads awaited the database on the window thread. They now run in Task.Run. They are the Availability Groups tab, the FinOps tab, the plan viewer metadata, the Overview tag refresh, and the Manage Tags, Database State Overrides, history, drill-down and Collection Log windows. A new test fails if a tab or window goes back to waiting there. Lite's DuckDB memory limit is now a setting, "DuckDB memory limit" under Dashboard Defaults in Settings. It takes a whole number of GB from 1 to 80% of the machine's memory, and the default stays 2 GB. A missing or out-of-range stored value falls back to the default and logs one line. Lite reads the limit at startup, so a new value takes effect after a restart. The archive COPY raise uses the larger of 4 GB and the setting. The thread count follows the limit: the smaller of 8, the core count and the limit divided by 256 MB. A DuckDB out-of-memory error now names the setting and its current value. SECTION: Fixed ENTRY: - **Lite's Job History tab reads only the selected time range, and tab loads no longer wait on the database from the window** ([#5553]) - Every Job History read loaded and sorted each server's whole archived job history before it dropped the runs outside the selected range. On a 30-server test store that cost 4 to 8 seconds and 600 to 730 MB per server, and on a larger store it can run out of DuckDB's memory cap. The archive view now applies the time range first: 0.5 to 0.7 seconds, under 300 MB, same rows. The Availability Groups and FinOps tab loads, opening a plan viewer from a tab, the Overview's tag refresh, and the Manage Tags, Database State Overrides, history, drill-down and Collection Log windows also awaited their database reads and writes on the window's own thread. There they can wait behind archival or a long read. They now read off that thread. SECTION: Added ENTRY: - **Lite has a DuckDB memory limit setting** ([#5553]) - Settings, Dashboard Defaults, "DuckDB memory limit" sets the main DuckDB connection's memory limit (default 2 GB, range 1 GB to 80% of memory, applied after a restart). The thread count follows it as the smaller of 8, the core count and the limit divided by 256 MB, and out-of-memory errors name the setting and its current value,. The log names it too. Fixes #5457 * The startup sweeps stop a continuous aggregate's jobs before they drop it (#5552) Three store-convergence sweeps drop a TimescaleDB continuous aggregate with DROP MATERIALIZED VIEW ... CASCADE: the superseded and retired passes of DropRetiredBaselineAggregatesAsync, and the composer reshape in DropStaleContinuousAggregatesAsync. None of them stopped the aggregate's refresh jobs first. A refresh running during the drop failed it with XX000 tuple concurrently deleted (#5416). It is also the inferred cause of a backend crash in CI (#5549). Before each drop, QuiesceContinuousAggregateJobsAsync finds the aggregate's jobs, including a dependent daily tier. It unschedules the ones that are scheduled and waits for two empty looks at pg_stat_activity for their workers, under a 30 s cap. Unscheduling makes the TimescaleDB scheduler end a running worker within a few seconds. The refresh rolls back and counts as one failed run. If a worker is still listed at the final look, the jobs are scheduled again and the drop waits for the next pass. A failed drop, a cancellation or a failed look also schedules the jobs again, each on a grace token of its own. When the jobs cannot be scheduled again, the log names the job ids and the view. The reshape sweep now reopens a broken connection, like the retired-baseline sweep. Live tests cover each path. The jobs stop before the drop, and a real worker held by an advisory lock holds the drop back. Other tests cover the cap, a failed drop, a cancellation and an already-unscheduled job. Pin tests hold the SQL, and a census fails on a new product drop site. A kept aggregate whose refresh was ended can raise the job-failure alert once, at information level. Fixes #5551 * Release 3.10.0: changelog entries for the last fixes (#5557) Adds the changelog entries for the 3.10.0 PRs that merged after the first splice: #5528, #5532, #5537, #5542, #5543, #5552 and #5553. Each full entry goes into docs/changelog/3.10.md and its title-only line into the CHANGELOG.md index. The archive census row for 3.10.0 moves from 257 to 265 entries. Changelog entries for the 3.10.0 PRs merged after the first splice. * The 27 MB plan walk and 20 MB cut-walk timing pins judge the fastest of three runs (#5561) (#5563) The 27 MB plan walk pin and the 20 MB cut-walk pin each judged a single run against a fixed time limit. One stalled run on a busy shared runner failed a correct walk (8.1 s once, against 0.6 s on the next run). Each now judges the fastest of three runs against the same limits, which are not raised. A busy machine only adds time, so the fastest run is the one it touched least. A walk that is really slow stays slow in every run. The ring-buffer pin already compared its 4 MB rate with a 1 MB baseline from the same run. It took the fastest of two runs on the big side and three on the baseline side. Both sides now take three. Test-only change. Fixes #5561 * Darling Viewer: Top Queries, Top Procedures and Query Store tabs make fewer store calls (#5555) (#5556) The Darling Viewer's Top Queries, Top Procedures and Query Store tabs make fewer store calls and overlap their grid and slicer reads. A 7 day Top Queries load fetched each row's query text with its own call, 50 calls in all. It is now one call. On a test link with an 80 ms round trip, the grid read went from 22 seconds to under 1 second. The Long Queries tab starts its trace check beside its two grid reads, so a load costs one round trip, not two. A failed trace check no longer throws away a good grid. The rows are shown, the note is left as it was, and the failure is logged. The status bar's Database size now comes from the size the service records. When none is recorded, or it cannot be read, the Viewer measures it live and logs the cause once. If the recorded read times out, the Viewer keeps the last size shown instead of starting a slower live measurement. A tab load slower than 3 seconds logs one line that names the tab and the time of each store read, including the baseline comparison. Fixes #5555 * Lite and Darling WPF tests take one gate so parallel STA classes stop racing a shared WPF cache (#5559) (#5567) Fixes a test flake where two test classes, each on its own STA thread, corrupted a dictionary that WPF shares across the process. DependencyPropertyDescriptor.FromProperty hands every thread the same descriptor from a static cache. That descriptor's AddValueChanged writes a dictionary with no lock. The app has one UI thread, so the product never races. xUnit runs about fifty STA classes in parallel, so the tests did. Every STA test helper in Lite.Tests (26 files) and Darling.Tests (25 files) now takes one shared gate before it creates its thread. It holds the gate until it joins. A census class in each project names any test file that starts an STA thread without the gate. This is a test-only change and touches no product code. Fixes #5559 * Role provisioning runs one batch at a time, and the viewer-grant test replays stop rewriting the shared database's access list (#5560) (#5566) Two sessions rewriting one catalog row at once make the second fail with "tuple concurrently updated". The live tests that replay the viewer's grants kept the managed store's database name. So they all rewrote the connect list of the cluster's own darling database together, and their cleanup rewrote it again. They now aim at each test's own scratch database. The role provisioning batch at service start now runs in one transaction behind an advisory lock. So does the reload's statement_timeout step, and so does the self-managed store's password-rules step. Two services starting against one store wait for each other. The wait is polled once a second and bounded at 12 seconds at start and 5 seconds on a reload. If something else holds the lock longer, one warning names the key and the session holding it, and the write runs without the lock. A write that fails because another session wrote the same row at once is run again, up to twice, each time in a new transaction. That covers a concurrent update, a duplicate key, a duplicate role and a deadlock. The reads that decide what to write run inside the locked transaction, so they see the state after the wait. New tests pin the replay's target, the bounded wait and its warning, the retry and the codes it covers, and the order of the reads. One pins the one-database rule the lock relies on. Others apply the real batch twice, inside the lock and without it. SECTION: Fixed ENTRY: - **Two services starting against one PostgreSQL store no longer fail each other's role provisioning** ([#5566]) - the role and privilege batch that runs at start, and the reload's statement_timeout re-assert, failed with "tuple concurrently updated" when a second service ran the same statements at once. That left the viewer and MCP roles stale until the next start. Both now take an advisory lock first, so the second waits for the first. The wait is bounded at 12 seconds at start and 5 seconds on a reload. If something else holds the lock for longer, one warning names the session that holds it and the write runs without the lock. A write that fails because the other session wrote the same row first is run again up to twice. The self-managed store's password-rules step takes the same lock. Fixes #5560 * The Query Store interval tables' retention purge deletes in row-capped batches (#5572) The Query Store interval purge finishes on a large store. On a large store, the purge of the wide and latest Query Store interval tables never removed anything. It deleted a whole day in one statement, and that statement passed the command timeout on every pass. The purge now deletes in batches of up to 50,000 rows, oldest first, and commits each batch. A batch that times out is retried at half the size, down to a floor. Each batch starts where the last one ended, so the index entries of rows already deleted are not read again. A backlog drains over a few passes, and a pass that stops partway keeps what it removed. The plan content table uses the same batch form, and now also starts each batch where the last one ended. Every batch statement names its table with the collect schema. The timeout warning always names the table. Plan check on a large store, read-only: all four batch statements read the first-execution-time index in order. None used a sequential scan, hash join or sort. A 50,000-row batch selected its rows in 0.3 s on the wide table and 1.7 s on the latest table. Tests: a forced timeout on a locked row proves a timed-out pass keeps its progress. On the old code the same test removes nothing. A backlog that spans several days drains to the cutoff. The plan content drain tests cover a group of rows with the same last-seen time. No schema change. Fixes #5569 * Recommendations skip databases that are a secondary copy on this node (#5564) Recommendations skip databases that are a secondary copy on this node. On an Availability Group, the Recommendations tab reported the same per-database finding once per node, secondary copies included. A database's settings and Query Store content replicate from the primary, so every node said the same thing about it. Lite and Darling now skip those findings for a database this node holds only as a secondary copy. These are database settings, percent file growth, Query Store plan regressions and table growth. On the FinOps tab they are idle databases, dev and test names, TDE blockers and compression candidates. The node that holds the primary copy reports them. Findings about work on the node, such as queries, missing indexes, blocking and I/O, stay on every node. The rule reads the replica and database snapshots the monitor already stores. It adds no query on the monitored server, no collector change and no schema change. A database is skipped only when three things hold. Both snapshots exist and are within an hour of the window end. The database has a local row. This node's role for that database's group is SECONDARY. Missing, stale, NULL, RESOLVING, standalone and Azure SQL Database cases skip nothing. A point-in-time pass and each compare window use the role at that time. Database names match exactly, since both sides come from the same server catalog. What it shows. The tab says how many databases it skipped. The analyze_server, compare_analysis and get_analysis_facts tools carry one secondary_replica_note field, built from the same set the pass used. The regressed-queries drill-downs filter in SQL before their row limit. Finding alerts follow the same rule. Fixes #5558 * Release checks: fixes from the morning walk of Lite, the Darling Viewer and the web dashboard (#5554) Fixes found in the morning walk of Lite, the Darling Viewer and the web dashboard. Running Jobs. The tab in Lite and the Viewer, and the get_running_jobs tool, list only jobs the collector's latest successful run found. A snapshot older than the long-running-job alert's bound is not listed. That bound is three missed cycles at the server's running_jobs cadence, at least 10 minutes. When that hides the jobs, the tab and the tool say collection is not current. They give the time of the last good collection. Running Jobs read errors. A failed read shows "The running jobs could not be read" instead of "No SQL Agent jobs are running." The empty grid says so in words. The column headers are wide enough to read. File I/O. The Read Latency chart no longer goes blank on a server whose busiest files are logs. Each chart draws its own ten files, exactly as the read ranked them. The read says which ten each file is in. The read chart takes the ten busiest by reads and the write chart the ten busiest by writes. The ranking puts NULLs last and breaks ties by file name the same way in the DuckDB and PostgreSQL stores. Overview I/O Latency lane. It weights read latency by reads. Lite's own queries no longer top Top Queries by Duration or Query Store by Duration. That covers its Query Store probe, its plan and text fetches, its connect-time detection query and its failed-jobs and job-step reads. Saved plans and exports. The Live Snapshot button and the six Get Actual Plan sites judge statement text and plans with the same statement filter as the screens. The filter runs off the UI thread. A plan the filter withholds is refused on save. The test that lists every place Lite opens its own server connection now covers all of Lite, with a reason for each file. Databases filter. Select All, or ticking every collected database, means All. Databases collected later are then included. A filter for databases that are not collected yet stays a filter when the read of collected names fails. Server tabs in Lite and the Viewer. Empty Trace Flags, Running Jobs and Query Store Regressions grids say why they are empty. Configuration and Index Analysis headers and notes read in plain words. Screen readers and Compare. The toolbar combos and plan operators have names for screen readers. The disabled Compare box says where it works. An empty comparison period says so. The Repro script's Generated line is in UTC. Web dashboard, pages. The footer Updated time follows the page's own latest load. It keeps its last good value when a poll fails. A read narrowed to its limit and then answered is not a failure. Job History and views. Job History messages are readable and open in full. A saved view opens on its server and says whose pick the scope is. It says all servers when its own setting is All. The Databases popover closes on a tab change. Offline fleet cards, Availability Group counts and several empty-state sentences read correctly. Web dashboard, service. A failed web request leaves exactly one service-log line, and it names the server and the cause. A failure before sign-in from one address is logged once per ten minutes. An unknown server is reported as not found. A registry lookup that fails no longer reads as "not registered". The offline-age read is cached and holds one pooled connection at a time. * Release 3.10.0: changelog entries for the last gate fixes (#5576) Adds the CHANGELOG entries for the PRs merged after the second splice. They cover the Viewer query tab loads (#5556), role provisioning under a lock (#5566) and the Query Store interval purge (#5572). They also cover Recommendations skipping secondary copies (#5564) and the release-walk fixes (#5554). The release-walk fixes get seven new entries, one per topic, written from the squash body. Each full entry goes into docs/changelog/3.10.md and its title-only line into CHANGELOG.md, with the archive census updated. Changelog entries for the 3.10.0 gate PRs merged after the second splice. * The module map refresh no longer holds back the collectors at start (#5578) (#5580) The module map refresh no longer holds back the collectors at start. On a large store, the service spent about 55 seconds at start refreshing the module map from procedure_stats before any collector ran. That happened on every restart. No collector reads the map. The start path now only creates the module map table, which readers need. The refresh itself runs from the sweep loop's first pass, after the collectors start. Two existing homes run it there, and neither is awaited. The hourly store-maintenance tick reads from the watermark. The daily purge re-reads the repair span behind it. A restart still repairs the same span as before. A store with no watermark still reads two days, now after the collectors start. The unused start-up refresh method is removed. Its live tests now call the daily refresh, which reads the same span. A new test pins the order: the table is created before the collectors start, and every refresh runs after. Fixes #5578 * The module map refresh groups procedure_stats before its DISTINCT ON (#5583) (#5584) The module map refresh groups procedure_stats before its DISTINCT ON (#5583) The module map's daily refresh read every procedure_stats row in its 30-hour window and sorted all of them to keep the newest row per handle. On a large store that was 10.6 M rows for 21,636 handles: the sort spilled 2.19 GB, and the whole statement took 35.5 s. Both refreshes now group the rows by server, handle and names first, with the newest collection time per group. On a copy of that store the same window ran in 8.3 s with no spill. The map and its watermark come out the same. Fixes #5583 * Query Store interval tables are partitioned by day (#5573) The Query Store interval tables are partitioned by day. On the PostgreSQL store, collect.query_store_interval_wide and collect.query_store_interval_latest now hold one partition per day of first_execution_time. Retention drops a whole expired day instead of deleting its rows. Reads bounded on first_execution_time skip the days outside their range. The upgrade copies no data. Rungs V171 (wide) and V172 (latest) rename the current table to
_legacy and attach it as a partition below a bound. A NOT VALID CHECK constraint proves that bound, so the attach does not scan the table. The service validates the CHECK in the background, outside start-up, and every start-up statement stays under the migration command timeout. On a restored copy of a large store, V171 took 1.10 s and V172 took 20.5 ms. Of V171's time, 1.02 s was a wait behind an autovacuum. The background VALIDATE took 389 s on the 91 GB wide table and 200 s on the latest table. An hourly step brings each table to its final state. It arms the CHECK, validates it and promotes the legacy partition to its final bound. It re-arms the CHECK when the bound is less than 12 hours away. The arm reads the legacy table's newest row through an index, with a 5 s limit. A row dated more than one retention horizon past the normal bound keeps the table unpartitioned until the row is gone. The service logs one warning about it. A VALIDATE that finds a row at or past the bound drops the CHECK and logs why. ANALYZE runs outside the hourly step (ANALYZE ONLY on PostgreSQL 18 and later). The legacy partition empties through the row-capped retention purge and is dropped once it is past the horizon. The DEFAULT partition gets a plain DELETE. Store self-metrics read partitioned and plain tables alike. Tests cover the rungs (no scan at attach), the hourly step and each of its failure paths, and the drop race. They also cover partition pruning of the shipped bounded reads, and retention on both layouts. Fixes #5571 * Column filters list a column's values to tick, like Excel, and stay set across refreshes and restarts (#5577) A text column's filter now lists the values in that column, like the filter in Excel, in Lite, the Darling viewer and the web dashboard. You can tick the values you want, search the list, use Select All, and tick a "(Blanks)" entry for empty cells. Unticking a few values remembers them as hidden, so a new login still shows after a refresh. Ticking a few remembers them as the only ones to show. Number and time columns keep the text match only. Query text, plan, XML, message, detail, description, reason, error and script columns also keep the text match only, and that match lasts for the session. The filter stays set across refreshes, tab switches and restarts, per server, grid and column. Lite and the viewer keep it in a file next to their settings, and the web dashboard keeps it in the browser. A value filter left on a column that no longer gets a list shows read-only with a Clear button. Fixes #5565 * After a big retention drain, warm each server's Query Store floor read in the background (#5581) (#5585) After a big retention drain, warm each server's Query Store floor read in the background (#5581) A big retention purge leaves many dead index entries in the Query Store interval tables. The first floor read for each server then walks every one of them. On a large store that read took 19.6 s for one server. The retention pass now runs the same floor read once per server when it finishes, one server at a time. A user's first read then finds the entries already marked. It runs only when the purge deleted rows or failed. A pass that only drops whole day partitions skips it, because a dropped partition takes its index entries with it. The warm-up runs in the background purge task, never on the sweep loop. A server whose table is locked is skipped and logged, and a failure never fails the purge. On a copy of that store, the warm-up took 645 s for 43 servers, and the reads after it took 16 ms in total. Fixes #5581 * Web charts draw at the panel's real width, so axis text is not stretched (#5586) (#5590) Line, scatter and ranked bar charts in the web viewer now draw at the width of their panel. Axis text and bar labels keep their normal size on a wide screen and on a phone. Before, a line or scatter chart drew a fixed 1000 by 320 picture that the page stretched sideways but not vertically. Its text was about 2.1 times too wide at 2,000 px and shrank on narrow panels. The bar chart drew a 1000-wide picture that scaled with the panel, so its labels grew and shrank with it. A resize redraws the charts once per frame and fetches nothing. The zoom, hidden series and event markers survive it. The x axis time labels follow the width, so none overlap on a phone and none repeat. Whole-minute labels serve a zoomed window of a few minutes, and seconds a shorter one. A window across a clock change labels each tick with its time zone. The bar chart cuts a long label to fit, and a threshold line stops at a value. A chart measured before is drawn at that width on the next refresh, so it is built once. The scatter chart's last x label and its unit caption stay clear of each other. Printing a page scales each chart to the page width. Charts draw down to a 200 px box, so a phone shows them whole. Fixes #5586 * Query Store interval tables stop cancelling their own autovacuum (#5589, #5594) (#5593) Query Store interval tables stop cancelling their own autovacuum (#5589, #5594) Two things the service did on the Query Store interval tables cancelled a running autovacuum, so autovacuum on those tables never finished. The interval arm (#5589). While a table is not promoted, its legacy table is the whole live table. A monitored clock can put its newest first_execution_time more than one retention horizon ahead. In that state two loops took an ACCESS EXCLUSIVE lock every hour before they read that time. Each request cancelled the autovacuum. The arm now reads the newest time first, without a lock. When that time is too far ahead, it warns once and returns without a lock request. BRIN autosummarize (#5594). The collection_time BRIN index had autosummarize on. Each new block range queued a summarize request that waits for the table's lock, and after one second that wait cancelled the autovacuum. After a large purge this happened about every 2 minutes. Autosummarize is now off on every BRIN index of both interval tables. New indexes and new day partitions get it off. The hourly maintenance pass turns it off on existing indexes, which cancels a running autovacuum at most once per index. It skips an index while a vacuum that cannot be cancelled runs on its table, such as a manual VACUUM or an anti-wraparound autovacuum. Otherwise it waits for the lock for at most deadlock_timeout plus one second, and never more than 5 seconds. The same pass now summarizes new ranges itself. It waits for a lock for at most half of deadlock_timeout, so it gives up before the wait cancels an autovacuum. It skips a table that is being vacuumed. It works one range at a time and stops between ranges when its time runs out, so it leaves no range half done. On a test server with steady inserts and a slow vacuum, autovacuum was cancelled 69 times in 4 minutes with autosummarize on, and never finished. With it off, nothing cancelled it and the vacuum finished. Fixes #5589 Fixes #5594 * Test processes no longer pay 0.3 s per WPF STA thread at exit, which failed green Darling runs (#5596) (#5598) The Darling whole-tree guards job (and some Lite test shards) failed or lingered after every test had passed. The runner printed "Waiting 10 seconds for foreground threads to exit..." and then "[FATAL ERROR] Foreground threads were left running, forcing process exit", with exit code 1 and 0 failed tests. No thread was left running. The runner only reports that the process was still alive 1 second after the run returned, and it forces exit 10 seconds later. What kept it alive was WPF's own exit handling. Each STA test thread that uses a WPF weak-event manager (a bound grid, say) makes WPF create a weak-event table. The table registers a process-exit handler. At exit that handler runs on the main thread and finds the table's own thread gone. It then asks the dead dispatcher to purge the table, with a 300 millisecond timeout nobody answers. That costs about 0.3 seconds for every such test thread, after the run returns. About fifty test files start their own STA thread, so the exit stage grew past the 11 seconds the runner allows. WPF has a runtime switch for exactly this case (Switch.MS.Internal.DoNotInvokeInWeakEventTableShutdownListener), which makes the handler purge the table directly. Both test projects now set it in their runtime configuration. New tests: WpfExitSwitchTests runs in both test processes. It checks that the switch is on and that the installed WPF still knows its name. It also checks that a weak-event table's exit handler for an ended STA thread returns at once instead of after the 300 ms wait. WpfExitSwitchPinTests checks that both project files declare the switch. This is a CI-only fix: no product code changed. Fixes #5596 * Collection Falling Behind no longer fires in the first hour after every service start (#5597) (#5599) The "Collection Falling Behind" alert no longer fires in the first hour after every service start. It leaves out what was counted in the first 15 minutes after the service starts, whichever way the clock steps. It does not fire or resolve until it has 15 minutes to judge. A store that stays behind still fires about 30 minutes after the start. Only a service start gets those minutes, once per process, not a stall, a pause ending or the memory launch guard's release. Slots that came due while a server was still seeding its schedule are no longer counted as skipped. Fixes #5597 * A hung Darling or Lite test shard names its test and still uploads its logs (#5587) (#5591) A hung test shard names its test and keeps its logs (#5587) A Darling PostgreSQL test shard ran into the job's 30-minute timeout and lost everything. GitHub left the later steps pending, kept no log and uploaded no timing. The build workflow's Darling and Lite test steps now have their own 22-minute timeout. So the upload steps still run when a shard hangs. The runner reports any test that has run for 5 minutes, and its output is uploaded whatever the step did. The nightly run gets the same report and uploads. Its Lite step gets a 140-minute timeout. Its Darling step gets none, because its longest recent run takes 84% of the time a timeout leaves it. One test (RollupBackfillLiveTests) waited up to an hour while holding a lock. It now gives up after 60 seconds and fails by name. Fixes #5587 * Time range picker: more presets, calendar periods and typed ranges (#5570) * Perfmon data is compressed per counter, so a one-counter trend reads only that counter's data (#5574) (#5579) * Perfmon data is compressed per counter, and chunks compressed by server alone are re-grouped (#5574) New perfmon_stats chunks compress by server_id and counter_name. An hourly pass re-groups the already compressed chunks inside the 30-day trend reach, newest first, one transaction per chunk under the bounded lock timeout, for at most two minutes a pass. Below the floor (no per-chunk settings view, TimescaleDB 2.14.1) nothing changes. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_015vDtCAiuajanNsX2cbhnwg * Perfmon re-group: readers are never blocked, and a background loop converges the 30-day reach in hours (#5574) The re-group now decompresses a chunk in one transaction and compresses it in a second, so the ACCESS EXCLUSIVE lock that decompress_chunk takes at its end is released before the compress half; reads of the chunk no longer wait for the rewrite. The work moves from a 2-minute slice of each hourly pass to a background loop (PerfmonRegroupDrain): one chunk at a time, newest first, a pause after each chunk as long as it took (at least 15 s), a busy lock skips the chunk for the run, three failures in a row stop it, and it ends on its own once nothing is left. * perfmon re-group: live tests for the crash between halves, the pause, a busy lock, repeated failure, the compress retry, the plan shape, a concurrent reader and a service stop (#5574) * perfmon re-group tests: hold a chunk's lock through a disposable (live-cleanup ratchet), reader bound 2.5 s (#5574) * perfmon re-group: pause for the time a busy lock cost, 30-minute statement cap, remember done and capped chunks, honest run-end reasons (#5574) * perfmon re-group: server cancel logged as such with its real time, gone chunk at either half confirmed by the chunk catalog, sturdier reach and reader tests (#5574) * get_perfmon_trend reaches 30 days on the web and MCP; the two timed re-group tests run in the timing collection (#5574) --------- Co-authored-by: Claude Sonnet 5.5 * Tests stop sharing WPF threads, Postgres roles and runner CPU (#5604) * A one-day Custom Views panel on Query Store data reads an hourly rollup, or is refused up front (#5588) On a large store, a one-day Custom Views panel on the Query Store wide table read 8.46 M rows. Cold, it took 55.2 s against the 60 s statement timeout, so a slightly larger store or a busier hour timed out. Refusal. Before it runs a Query Store panel, the read guard estimates the rows it will read. The limit is 6.8 M rows for one scan and 5.4 M rows for two. Above it, the panel is refused up front with the reason, rather than running into the timeout. One scan or two. A ranked time series panel reads the wide rows once only when time buckets times group members stay at or below 1.0 M. Above that, one scan risks the 1 GB temp file limit. The server count is exact. For the database and module dimensions the count comes from planner statistics. Those run low on a large store (module names were 1.7 times low), so the count is multiplied by 4. A query hash ranking always takes two scans: on the large store, one scan of it wrote about 1 GB of temp files. The hourly rollup (schema V173). The wide table gets an hourly rollup at the grain of each collection. It has one row per collection time, server, database, module and query hash, with the numeric sums a panel needs. The service builds an hour about 2 hours after it ends, up to 6 hours a tick. Two triggers on the wide table mark an hour stale when a row lands in it after it ended an hour ago or more. The service then rebuilds it. The build of an hour waits while a write transaction that began early enough to touch it is still open. Reading. A panel reads built hours from the rollup. It reads stale hours and the newest hours from the wide table. It returns exactly what the wide table alone returns. Live tests compare the two answers value for value, for every measure, aggregate and panel mode. The read guard counts a rollup hour at 0.3 of a wide hour, and a field run after deploy checks that weight. The triggers cost about 13% more time per write for rows of the current hour on a test server. Rows that fire them cost up to about 55 ms more per 10,000 rows. Upgrade. V173 adds the triggers to every partition of the wide table. If an anti-wraparound autovacuum is running on a large partition, the upgrade waits for it, and collection waits with it. On the large store the wide table was far from one: its transaction age was about 40 M, and that vacuum starts at 200 M. A migration step that hits its lock timeout now logs who holds the locks. It lists each session with a write lock or stronger on the store's tables. It logs at most every 10 minutes. Part of #5582. Its round-2 fixes follow in their own pull request. * Live tests no longer fail when their seeded window crosses midnight UTC (#5609) The newest-snapshot block-count test seeded its window at the wall clock. Between 00:00 and 04:00 UTC that window crossed a day chunk boundary. The blocks the newest read touches then swung from 13 to 138, against a limit of 88. The test failed on every pull request whose Darling PG shard ran in that window. An earlier live test failed the same way between 00:00 and 03:00 UTC, so the fix covers the whole class of tests, not one. A new LiveClock test helper gives two things. AnchoredEnd() is a fixed point inside one day chunk, 22:00 UTC on the previous day. Now() is the clock with a test-only override, the DARLING_TEST_CLOCK_UTC environment variable, which only the tests read. The machine clock is never changed. Two audits covered 68 live test files that read the wall clock and use EXPLAIN or the chunk catalog. Each file now does one of three things. Its seeded window ends at AnchoredEnd(). Or its clock reads go through Now() where the test passes that time to the product. Or its assertion holds for any chunk layout. A file whose product read takes its own clock stays on the real clock, with the reason written down. The force-plan failures test is one of these. Its read builds a two-hour window from the product's own clock, so a seed taken from the override fell outside it. A new census, LiveClockCensusTests, has two rules. A live test that counts plan blocks, or uses EXPLAIN or the chunk catalog, and reads the wall clock without LiveClock fails. Each exception is on an allow list with its reason, and a stale entry fails. Run against dev without these changes, the census names 34 files. The fixed classes passed with the override at 00:30 UTC and at 12:00 UTC and with it unset. The guards still fail on the defects they were written for. This is a test-only change. Fixes #5608 * Query Store rollup: the build wait counts only writers of the wide table, and the migration lock log lists every lock holder (#5610) The hourly Query Store rollup from #5588 held back its builds while any transaction in the database that had written something was still open. One long unrelated write stopped the builds, and panels went back to reading the wide table. The build wait now counts only sessions that hold a write lock on the wide table or one of its partitions. A writer whose start time cannot be read counts as open. The plan skips the hours such a writer holds back, so they take no build slots and older hours still build. The wait can repeat on two ticks in a row, or a writer can stay open for over an hour. Then the service logs a warning with the session's pid, start time, application name and backend type, at most every 10 minutes. A schema upgrade step that cannot get its table lock in time logs the sessions holding locks on the store's tables. That list now includes sessions that only read a table, because a plain read also blocks most schema changes. Each entry lists pid, backend type, application name, state, transaction start, wait event, table and lock mode. For an autovacuum worker it adds the first 200 characters of its query, which shows an anti-wraparound vacuum. The lookup has its own 5 s statement timeout and 2 s lock timeout. A failed lookup no longer mutes the next 10 minutes, and a shutdown during the lookup stops it. Fixes #5582 * The retention drain gives way to collection, stops at a 30-minute budget and continues within the hour (#5595) On a large store, the first retention drain after the 3.10 upgrade deleted about 18 M rows in 40 minutes and paced only its WAL. Collection fell behind while it ran, and servers skipped collection slots. Every batched delete now goes through one gate. Before each batch, the drain waits while collection is behind, re-checking every 5 s. Collection is behind for the first 10 minutes after the service starts collecting. It is also behind when at least 3 slots were skipped in the last 10 minutes. Those must be at least 2.5% of the slots that came due. A pass spends at most half its budget waiting. After each batch it pauses for as long as the batch ran, at most 60 s. A pass stops at the next batch boundary once it has run 30 minutes, waits included. The run record says where it stopped and how many tables it did not reach, and the status stays SUCCESS. The next pass then comes due within an hour of the stop, not a day later, until a pass drains every table. This applies to the daily purge and to purge_now. The rule was retuned after a read of a normal day. A synthetic small-store hour reads behind in none of 60 checks. A replay of the large store's backlog kept p95 lag at 0.99 to 1.01 times the phase with no drain. The wide-table drain's p95 write latency was 0.44 times. Fixes #5592 * The at-the-cap aggregate test no longer races TimescaleDB's scheduler (#5605) The live aggregate quiesce tests no longer need to see a TimescaleDB job worker alive inside a time window. The scheduler ends a worker whose job was stopped at its next wake. On a loaded machine that can be within milliseconds of the stop. The hold test and the at-the-cap test give their gate function a trap. The trap counts the scheduler's cancel and keeps waiting, so the worker survives 3 s or more after it. The hold test waits for that count instead of for two seconds. Both tests also check that no cancel came from anywhere but the scheduler. The cancel test polls every ten minutes, so the sweep can never reach its second look before the test cancels it. The comments on the quiesce sweep give the measured scheduler timing. Without the trap, a cancel ended the worker at once with an `invalid transaction termination` error in the PostgreSQL log. That is how the CI failure looked. Fixes #5603 * Test map: shared live-store fixtures are linked, keep-full counts only the pull request's own change, and the shadow check reads Guard classes and the latest attempt (#5459) (#5611) CI's test map now links every class of a shared live-store test collection to the files its fixture ran. This fixes the 11 change-related misses among the 55 replay misses. The other 44 misses are unrelated to the change and are listed in the accepted file. A keep-full file now counts only when the pull request itself changes it. The FULL rows among the 76 shadow rows go from 68 to about 47. The shadow check now reads the reports from the Guard tests job. For each downloaded artifact, it keeps only the newest attempt. The shard timing uploads now set overwrite, so a re-run replaces the old artifact whole. Refs #5459 * Release 3.10.0: changelog entries for the PRs merged after the third splice (#5613) This adds fourteen entries for the PRs that merged to dev after the third splice: two Added, three Changed and nine Fixed. Each full entry goes into the 3.10.0 archive, its title-only index line goes into the CHANGELOG.md index, and each reference gets a link definition. The archive census is regenerated. The four modes of the changelog archive script all succeed. The changelog, version, release gate, census and guard test classes pass. Changelog entries for the 3.10.0 PRs merged after the third splice. * Slow Lite test classes stop queuing behind database schema builds (#5208) (#5612) * Lite tests copy a prebuilt database instead of rebuilding the schema under the shared write lock (#5208) * Lite test copies of the prebuilt database get their own store identity, and 63 more test files use the copy (#5208) * Keep the real InitializeAsync in tests that reset, restart or reopen an existing file (#5208) * InitializeFromTemplateAsync is as safe to call again as InitializeAsync (#5208) A repeat call on a path that already holds the store now runs the real InitializeAsync instead of copying over it, so a helper that initializes once per call no longer throws 'already exists' (ArchiveGroupedWatermarkReadTests, 38 tests). * CI: a changelog splice takes the docs fast path (#5615) A changelog splice changes CHANGELOG.md, docs/changelog/.md and tools/changelog/archive-census.txt. The census is a text file that was not on the documentation list. So the splice ran about 17 minutes of Guard tests and whole-tree guards instead of the docs fast path. The exact census path is now on the list in both copies in build.yml and in ci-select.py. A test pins that a splice is docs-only and that the three lists agree. The CI replay accepts one past splice run, on which an unrelated live-database class failed. What this gives up: ChangelogIndexAndArchiveTests no longer runs on a splice pull request, as it already did not run on a CHANGELOG.md-only change. The splice lane runs it before it pushes, and the nightly runs the full suite. Fixes #5614 * MigrationLockHolderLogTests accepts the autovacuum query text the lock log writes by design (#5619) #5610 added a check to MigrationLockHolderLogTests that no part of the migration lock log line contains "query:". The log adds the first 200 characters of the query for an autovacuum worker on purpose. The test passed review and CI because most runs have no autovacuum worker holding a lock on a darling table. When one did, the test failed the pull request. The test now splits the line into one entry per session. The entries for its own holder and reader must exist and carry no query text. Any entry that carries query text must be an autovacuum worker's. The checks that the two sessions' statements never appear stay on the whole line. No product code changes. Fixes #5617 * Darling live tests stop depending on PUBLIC's CONNECT on the shared test database (#5618) (#5621) A Darling live test ran the managed provisioning batch inside a scratch database. That batch revokes CONNECT from PUBLIC on the database named darling. Database privileges belong to the whole cluster, so every later test that relied on PUBLIC's CONNECT there failed with permission denied. The password-key test now points its database-level statements at its own scratch database. The redaction test grants its own login CONNECT instead. A new guard on the cluster-roles test collection fails the test that leaves PUBLIC without CONNECT, by name, and no product code changes. Fixes #5618 * Lite tests that re-initialize a store stop holding up the other tests (#5208) (#5622) Many Lite test classes were slow because they queued behind other tests that held the one database write lock. The copy step that builds each test store, and helpers that rebuilt the same store on every call, held it longest. The copy step now opens its connection and renews the copy's identity first, so the lock is held only to swap the connection in. The helpers build a store once, and a race test writes 1,000 keys instead of 4,000 but still fails without the write lock. Four groups that stand in for the CI shards ran in 45 to 80 s instead of 106 to 119 s on a laptop. WebhookCooldownSeedTests now posts to an in-process endpoint instead of a closed port, so it takes 0.2 s instead of 20.5 s. Fixes #5208 * Nightly: the Lite and Darling PostgreSQL tests run in PR CI's shards, so the nightly publishes sooner (#5620) The nightly ran each test suite in one step, so its publish job landed about 50 minutes after the run started. PR CI runs the same suites in shards in about 19 minutes. The cut, the Cost=Slow skip, the chunks and their runs moved from two build.yml steps into two scripts, .github/scripts/run-darling-pg-shard.ps1 and .github/scripts/run-lite-shard.ps1. build.yml and nightly.yml both call them, and PR CI's cut is unchanged. The nightly's Darling PostgreSQL job is now six shards with the same test set as before. A new Lite job runs four shards, with every class, the Guard classes too. The build job no longer runs tests, and its timeout is back at 90 minutes. Publish now waits for every shard of both suites, and runs only on the dev ref. The publish input stays as a second gate. The test map and AltCover jobs also wait for the Lite shards. The shard scripts and the Lite packer are in the path filters and the slow-skip build inputs, so a change to one runs the suites. A proof run from the branch had everything publish needs done 16.6 minutes after it started. The last three dev nightlies took 44.8 to 49.4 minutes. The legs' class counts sum to the whole suites. Fixes #5616 * Nightly: every job checks out the commit the run was dispatched at (#5623) The nightly's seven checkouts used the branch name. Each job checked out the branch tip at the moment it started. So a merge to dev during a nightly changed what later jobs built, the shards and publish among them. Dev had to freeze during every run. Every checkout now uses the run's commit (github.sha), the dispatched branch's tip at dispatch time. A branch dispatch still builds its own branch, and the scheduled path still builds dev's tip from when it was dispatched. NightlyCheckoutCommitPinTests (a Guard class) fails if any nightly checkout goes back to a branch name, or if the redispatch job gains a checkout. Fixes #5616 * Pull request CI runs only the test classes the test map picks (#5459) (#5624) A pull request now runs only the test classes that the nightly test map picks for its diff. The Darling PostgreSQL shards, the Lite shards and the build job's Darling pass keep only those classes, and the Guard stage still runs in full. If the selection is full, missing, unreadable or empty, they run everything. Pushes, merge-queue runs and the nightly always run everything. Replaying past failures against the map found six misses. None was caused by its pull request's change, so each is listed as accepted with the evidence, and the replay now finds none. A class gated on a PostgreSQL variable that only CI sets is always kept. A nested class is matched by its own name. A pull request into main, the release pull request, always runs every class, because the selection and the Cost=Slow skip both check the base branch. Fixes #5459 * Darling's hourly deadlock maintenance reads stored findings and alerts in bounded pages (#5625) (#5626) Darling's hourly maintenance step over stored analysis findings no longer times out on a large store. It reads a fixed range of the findings table per page. A page costs the same whatever the table's size. The two stages over stored alerts now read one server at a time through the alert log's existing index. Deadlock alerts are paged per server, and finding alerts are paged per server and metric. Each page is a range of that index. That bounds it by its size on PostgreSQL 16, 17 and 18 alike. The report lookup uses the existing index on stored reports. No store migration is needed. Live tests at about 500,000 alert rows check the plans, the rows read per page, and that every raw row is rewritten once. Fixes #5625 * The pre-lock check trusts the install's own service SID and an existing install gets exact steps (#5628) - The pre-lock check trusts the product's own service account only in the folders and files the service writes. - One shared list names those places, and the install lock and the check both read it. - The refusal for an install whose service runs from the refused folder gives numbered steps to move to a new folder. - A service registered elsewhere, or none, gets the fresh-folder wording. - Paths darling.json names inside the old folder are listed, quoted and capped, and the old folder is kept until they move. - upgrade-darling.ps1 applies its -Source default in the body, so it runs on Windows PowerShell 5.1 from a staging folder. - A new CI check runs both scripts over real 3.7.1 and 3.9.0 installs, and checks each release zip against the release's checksum file. - The printed by-hand fix grants the service read and execute on the install folder. - The compose quickstart pulls the :latest image. Fixes #5627. Fixes #5629. * Darling's deadlock maintenance logs a line when a stage recovers after a failed pass (#5637) Darling's hourly PostgreSQL deadlock maintenance logged a warning when a stage failed a pass. It logged nothing when a later pass succeeded, so the last line an operator saw was the failure. A stage that succeeds after one or more failed passes now logs one information line that names the stage and how many passes had failed. A clean pass with no earlier failure logs nothing new, and the back-off and skip logic are unchanged. Fixes #5634 * Darling web: response security headers; clearer consent text; connection-test errors carry less detail; token honored on loopback listeners (#5636) The Darling web dashboard now sends four response headers on every reply: a content security policy, X-Frame-Options, X-Content-Type-Options and Referrer-Policy. The sign-in and landing pages carry a policy that names the hash of their own inline blocks. The confirmation shown before a captured query is re-run for its actual plan now adds one sentence. It says the text was captured from the monitored server. The Darling Viewer, Lite and the shared plan history windows show the same sentence. A failed connection test from an MCP tool or a web route now answers with one fixed sentence naming the host and port. The driver's text goes to the service log on one bounded line, with the typed password and AWS external ID redacted. The host command line and the desktop Viewer's Add dialog keep the detail. A token set in mcp.network or web.network now gates the loopback listeners too. Local MCP clients send an Authorization header, and local browsers sign in with the token. With no token configured nothing changes. A token that is set but blank or cannot be decrypted stops that listener from starting, in loopback-only and network mode alike. The Darling MCP server answers 415 to a POST that does not carry a JSON Content-Type. The check runs after the Host, CIDR and token gates, so a caller that fails a gate gets 403 or 401 first. The service log records each 415 the way it records the other refused requests. The Viewer's copy-MCP-command button now tells the user how to add the token header. Lite's MCP server answers 415 to a POST that does not carry a JSON Content-Type in the same way. * Release 3.10.0: last changelog splice and README sync (#5635) Adds the changelog entries for #5626 and #5628 to the 3.10.0 index and to docs/changelog/3.10.md. The #5628 entry cites its own PR. The 3.10.0 date is now 2026-10-09. A second Important entry covers the Windows upgrade stop for installs that ordinary users can write to. The Important note now names store migrations V158 through V173, as dev is at schema 173. The README gains the shared time range picker, the text column filters and Lite's DuckDB memory limit setting. The branch merges dev, which brings in the entries for #5636 and #5637, and every entry from both sides is kept. The archive split and census run last. The 3.10.0 census row is 295 entries, and verify and roundtrip both pass. No duplicate section headers and no missing reference links were found in the 3.10.0 entries. The last changelog splice for 3.10.0, and the README sync. --------- Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: ianwalkeruk <37576946+ianwalkeruk@users.noreply.github.com> Co-authored-by: Claude Sonnet 5.5 --- .claude/skills/lane-orders/SKILL.md | 40 +- .claude/skills/release-checklist/SKILL.md | 73 +- .github/FUNDING.yml | 2 + .github/ISSUE_TEMPLATE/bug_report.yml | 12 +- .github/ci-history/failures.jsonl | 731 +++ .github/ci-history/replay-accepted.txt | 129 + .github/darling-paths-filter.yml | 13 +- .github/scripts/altcover-probe.py | 513 ++ .github/scripts/ci-select.py | 1255 ++++ .github/scripts/darling-upgrade-shapes.ps1 | 265 + .github/scripts/lite-shard-pack.py | 586 ++ .github/scripts/lite-timing-top.py | 179 + .github/scripts/run-darling-pg-shard.ps1 | 142 + .github/scripts/run-lite-shard.ps1 | 221 + .github/scripts/tag-reads-lite.py | 88 + .github/scripts/test-map.py | 662 ++ .github/workflows/build.yml | 1524 ++++- .github/workflows/claude-review-guard.yml | 441 +- .github/workflows/claude-review.yml | 591 +- .github/workflows/darling-upgrade-shapes.yml | 82 + .github/workflows/nightly.yml | 918 ++- .gitignore | 10 + CHANGELOG.md | 609 ++ .../ActiveQueriesDatabaseFilterLiveTests.cs | 278 + .../ActiveQueriesWidenedColumnsTests.cs | 165 + .../ActualPlanConsentNoticeTests.cs | 58 + .../ActualPlanSnapshotResolveLiveTests.cs | 95 + .../Darling.Tests/AdminPageBehaviourTests.cs | 130 + Darling/Darling.Tests/AdminPageTests.cs | 139 + .../AdminServerEditBehaviourTests.cs | 1668 +++++ .../AdminServersRouteLiveTests.cs | 157 + .../Darling.Tests/AdminServersRouteTests.cs | 254 + .../AdminServersVersionLabelTests.cs | 50 + Darling/Darling.Tests/AgEveryViewLiveTests.cs | 128 + .../Darling.Tests/AggregateJobQuiesceTests.cs | 1015 +++ .../AlertCooldownClockStepTests.cs | 45 +- .../AlertDelivererInventoryPinTests.cs | 239 + .../AlertDeliveryChannelTests.cs | 377 ++ Darling/Darling.Tests/AlertEngineTests.cs | 12 +- .../AlertHistoryAdviceLiveTests.cs | 307 + .../Darling.Tests/AlertHistoryAdviceTests.cs | 149 + .../AlertHistoryDetailLabelsTests.cs | 3 +- .../AlertHistoryDismissBehaviourTests.cs | 202 + .../Darling.Tests/AlertHistoryDismissTests.cs | 347 ++ .../AlertHistoryLazyRenderTests.cs | 14 +- .../AlertHistoryRangePageTests.cs | 253 + .../AlertHistoryRowSeverityTests.cs | 1 + .../AlertMasterSwitchSurfaceTests.cs | 2 + .../AlertMetricClassifierTests.cs | 3 + .../AlertNotebookRenderBehaviourTests.cs | 1 + .../AlertNotebookRenderClientTests.cs | 9 +- .../AlertNotebookTableCatalogTests.cs | 2 + .../AlertReadFailureSurfaceTests.cs | 22 +- .../AlertStatementCutOrderTests.cs | 216 + .../AlwaysOnXeReadOnlyIntentTests.cs | 423 ++ .../Darling.Tests/AlwaysOnXeSessionsTests.cs | 337 + ...nalysisFactsReadRunsDetectorParityTests.cs | 1 + .../Darling.Tests/AnalysisFindingsTabTests.cs | 185 + .../AnalysisReadCancellationPinTests.cs | 1 + .../AnalysisWindowEmptyRuleParityTests.cs | 1 + .../AnomalyDetectorErrorGuardLiveTests.cs | 6 +- .../AnomalyNullWindowLiveTests.cs | 2 +- .../AnomalyTileWindowEndTests.cs | 1 + .../Darling.Tests/AsOfWindowAnchorTests.cs | 8 +- .../AvailabilityGroupsTabRefreshTests.cs | 26 +- .../AwsAssumeRolePackageTests.cs | 73 + .../AwsAssumedRoleCredentialsTests.cs | 530 ++ .../AwsExternalIdNeverPrintedTests.cs | 240 + .../AwsPerServerRoleLiveTests.cs | 569 ++ .../AwsPerServerRoleRungTests.cs | 283 + .../Darling.Tests/AwsRoleAllowlistTests.cs | 317 + .../AwsRoleBundleEndToEndLiveTests.cs | 186 + .../AwsRoleCredentialCacheTests.cs | 195 + Darling/Darling.Tests/AwsRoleEditTests.cs | 468 ++ .../Darling.Tests/AwsRoleMessageTextTests.cs | 248 + Darling/Darling.Tests/AwsRoleSettingsTests.cs | 189 + Darling/Darling.Tests/AwsStsTestSupport.cs | 163 + .../AzureFallbackMessageTests.cs | 71 + .../AzureForeignStatePruneTests.cs | 7 +- .../AzureMasterAnalysisScopeLiveTests.cs | 4 +- .../Darling.Tests/AzureSiblingGrowthTests.cs | 30 +- .../AzureSqlDatabaseHardwareTests.cs | 15 +- .../AzureSqlDatabaseHostMathTests.cs | 1 + .../AzureSqlDatabaseMemoryScopeTests.cs | 16 +- .../AzureSqlDatabaseOwnFiguresTests.cs | 16 +- .../BaselineDiscontinuityRenderTests.cs | 5 +- Darling/Darling.Tests/BaselineSupplyTests.cs | 19 +- .../Darling.Tests/BenefitScorerWiringTests.cs | 2 + .../BlockingDatabaseFilterLiveTests.cs | 362 ++ .../BlockingListPlanFlagsLivePostgresTests.cs | 237 + .../BlockingThresholdCoverageTests.cs | 89 + .../BlockingWaitsFilteredEmptyWordingTests.cs | 37 + .../BlockingWaitsWebDispatchLiveTests.cs | 161 + .../BoundedCompressionDdlTests.cs | 190 + .../BoundedDdlBusyStreakTests.cs | 163 + .../BoundedDdlSingleCopyTests.cs | 82 + ...BugReportTemplateDiagnosticsBundleTests.cs | 42 + .../BuiltinAlertPersistenceRungTests.cs | 1 + .../CaggGroupIndexDropLiveTests.cs | 138 + .../viewerservertab-eventwires.baseline.txt | 2 - .../CaptureDownChunkOrderTests.cs | 3 +- .../ChartIntegerTicksPinTests.cs | 1 + .../CiShardCommandLineBudgetTests.cs | 131 + .../Darling.Tests/CiShardHangReportTests.cs | 225 + .../ClipboardWriteCensusTests.cs | 2 + .../CollectionCaveatLedgerTests.cs | 2 + .../CollectionCaveatsParityTests.cs | 1 + .../CollectionCaveatsWebTests.cs | 304 + .../CollectionGapAtStartTests.cs | 351 ++ ...lectionHealthDetachedSweepPressureTests.cs | 177 + .../CollectionHealthEffectiveCadenceTests.cs | 198 + ...ealthEverySurfaceEffectiveIntervalTests.cs | 306 + .../CollectionHealthKeyedLookupParityTests.cs | 501 ++ .../CollectionHealthOracleSql.cs | 482 ++ .../CollectionHealthRunTimeFieldsLiveTests.cs | 587 ++ ...lectionHealthScheduleOverridesLiveTests.cs | 207 + .../CollectionHistoryAuditTests.cs | 1401 +++++ .../CollectionLogDrainForensicsStoreTests.cs | 69 +- .../CollectionLogFanoutRollupStoreTests.cs | 1 + .../CollectionLogSegmentByLiveTests.cs | 722 +++ ...CollectionLogStoreProbeAttributionTests.cs | 1 + ...ctionLogWebRowPreservesFullTextPinTests.cs | 1 + .../CollectionNoteLatestRunTests.cs | 1 + .../CollectionOutputBesideCostTests.cs | 5 +- .../CollectorDatabaseScopeRungTests.cs | 12 +- .../CollectorFaultStackLogTests.cs | 202 + .../CollectorHealthClassifierTests.cs | 22 +- .../CollectorMeasurementSeamTests.cs | 16 +- .../Darling.Tests/CollectorMemoryKnobTests.cs | 2 + .../CollectorRunTimeCliVerbTests.cs | 485 ++ .../CollectorRunTimeRungTests.cs | 1036 +++ .../Darling.Tests/CollectorRunTimeTests.cs | 610 ++ .../CollectorRunTimeViewerLiveTests.cs | 545 ++ .../CollectorRunTimeViewerTests.cs | 403 ++ .../CollectorScheduleEditorSaveGuardTests.cs | 146 + .../CollectorScheduleResetStatusTests.cs | 46 + ...llectorScheduleUseDefaultRunAtLineTests.cs | 81 + .../CollectorStallProbeStoreTests.cs | 13 + .../CollectorStateContractTests.cs | 1 + .../ColumnFilterPersistenceTests.cs | 827 +++ .../Darling.Tests/ColumnFilterPopupTests.cs | 323 + .../ColumnFilterWiringCensusTests.cs | 172 + .../ColumnGroupsBehaviourTests.cs | 123 + .../ColumnValueFilterCasesTests.cs | 156 + .../ColumnValueListNameRuleTests.cs | 120 + Darling/Darling.Tests/ColumnValueListTests.cs | 355 ++ .../CommandDeadlineScannerAdoptionTests.cs | 1 + .../CommandPlaneCommandTimeoutTests.cs | 6 +- .../CommentFilterAdoptionTests.cs | 42 +- .../ComposeAnnotationServerClockTests.cs | 4 +- .../ComposeCatalogStatementTextCensusTests.cs | 101 + .../ComposeClientTimeoutAttributionTests.cs | 247 + .../ComposeDataFloorLiveTests.cs | 701 +++ .../ComposeHourlyEdgesLedgerGuardLiveTests.cs | 250 + .../ComposeHourlyEdgesRunnerTests.cs | 286 + .../ComposeHourlyRawEdgesGuardLiveTests.cs | 681 ++ ...ComposeHourlyRawEdgesModuleMapLiveTests.cs | 556 ++ Darling/Darling.Tests/ComposeImageTagTests.cs | 80 + .../ComposeOpenPhaseTimeoutTests.cs | 180 + .../ComposeParameterCoverageTests.cs | 150 +- ...poseQueryStoreRankedSingleScanLiveTests.cs | 308 + .../ComposeQueryStoreRankedSingleScanTests.cs | 143 + ...mposeQueryStoreRawDedupeFilterLiveTests.cs | 342 + .../ComposeQueryStoreStampCompileTests.cs | 359 ++ ...omposeQueryStoreStampExactnessLiveTests.cs | 166 + .../ComposeQueryStoreStampGuardLiveTests.cs | 91 + ...oseQueryStoreStampInvalidationLiveTests.cs | 139 + .../ComposeQueryStoreStampPlanLiveTests.cs | 185 + ...ComposeQueryStoreWideExactnessLiveTests.cs | 5 +- .../ComposeRollupDataFloorLiveTests.cs | 226 + .../ComposeServerScopeByIdTests.cs | 539 ++ .../Darling.Tests/ComposeSourceRouterTests.cs | 350 ++ .../Darling.Tests/ComposeStampLiveSupport.cs | 232 + ...eStatementTimeoutV147MigrationLiveTests.cs | 46 +- .../ComposeTimeoutPhaseLiveTests.cs | 120 + .../CompressionPhaseAssignmentTests.cs | 5 +- .../ConfigDatabaseFilterLiveTests.cs | 302 + .../Darling.Tests/ConnectionTestReplyTests.cs | 301 + .../ConsumedTimestampFrameDisciplineTests.cs | 36 +- .../ControlPlaneReloadDurabilityTests.cs | 6 +- .../CreationTimeClockFrameDisciplineTests.cs | 1 + .../CustomAlertResolutionNameFilterTests.cs | 45 + .../CustomAlertResolveGrantLiveTests.cs | 123 +- .../DailyCollectorDetachTests.cs | 1061 ++++ .../DailyDeadlockWindowCensusTests.cs | 2 + .../DailyRunCapAndWatchdogTests.cs | 895 +++ .../DailySummaryAzureMasterScopeLiveTests.cs | 12 +- .../DailySummaryNotCarriedTests.cs | 2 + .../DailySummaryReadShapeTests.cs | 6 +- .../DarkPastRetentionReadsOfflineTests.cs | 8 + Darling/Darling.Tests/Darling.Tests.csproj | 30 + .../Darling.Tests/DarlingActualPlanTests.cs | 1 + Darling/Darling.Tests/DarlingAgReaderTests.cs | 183 + .../DarlingAgStatesReaderLiveEqualityTests.cs | 6 +- Darling/Darling.Tests/DarlingAlertingTests.cs | 1 + .../DarlingAllowFromListGateTests.cs | 339 + .../DarlingAnalysisPipelineTests.cs | 38 +- .../DarlingAnomalyBaselineTests.cs | 1 + .../DarlingBlockingReaderServerClockTests.cs | 102 +- .../DarlingBlockingStatsReadTests.cs | 177 +- .../DarlingBlockingTrendEmptyTests.cs | 6 + .../DarlingCidrAllowListTests.cs | 418 ++ .../DarlingCliCommandsHostCheckTests.cs | 7 +- .../Darling.Tests/DarlingCliCommandsTests.cs | 330 +- .../Darling.Tests/DarlingCliSealKeyTests.cs | 113 + .../DarlingCliUnusableStoreConnectionTests.cs | 20 +- .../DarlingCloudIdentityProbeTests.cs | 33 +- .../DarlingCollectionHealthLatestNoteTests.cs | 41 +- .../DarlingCollectionLogReadTests.cs | 156 +- .../DarlingCollectorRunTimeReaderTests.cs | 242 + .../DarlingCommandExecutorTests.cs | 21 + Darling/Darling.Tests/DarlingComposeTests.cs | 494 +- .../Darling.Tests/DarlingConfigAwsTests.cs | 147 + Darling/Darling.Tests/DarlingConfigTests.cs | 367 +- .../Darling.Tests/DarlingCustomViewsTests.cs | 2 +- .../DarlingDailySummaryRangeTests.cs | 5 +- .../DarlingEmptyEnumerationInventoryTests.cs | 26 +- .../DarlingEmptyEnumerationNoteTests.cs | 73 +- .../DarlingEncryptPasswordStreamTests.cs | 172 + .../DarlingEndpointToggleCliTests.cs | 75 +- ...tBaselineCoverageReadPlanShapeLiveTests.cs | 15 +- .../DarlingEventBaselineCoveredDaysTests.cs | 2 + .../DarlingExportViewerConfigTests.cs | 102 +- .../DarlingFirewallCheckTests.cs | 55 + .../Darling.Tests/DarlingFleetReaderTests.cs | 53 +- .../Darling.Tests/DarlingHeartbeatTests.cs | 886 +++ .../Darling.Tests/DarlingHostBindingTests.cs | 180 + .../DarlingHostConfigurationPinTests.cs | 142 + .../DarlingHttpRefusalLogTests.cs | 1 + .../DarlingInstallLocationTests.cs | 850 ++- .../Darling.Tests/DarlingListenerTlsTests.cs | 1041 ++++ .../DarlingManagedPostgresTests.cs | 393 +- .../Darling.Tests/DarlingManagedRolesTests.cs | 201 +- .../DarlingMcpActiveQueriesBudgetLiveTests.cs | 88 +- .../DarlingMcpAlertToolsTests.cs | 6 +- .../DarlingMcpBlockingToolsTests.cs | 436 +- .../DarlingMcpConfigHistoryToolsTests.cs | 336 +- .../DarlingMcpCustomViewToolsTests.cs | 72 + .../Darling.Tests/DarlingMcpDataToolsTests.cs | 65 +- .../DarlingMcpDeadlockProcessesLiveTests.cs | 142 + .../DarlingMcpDefaultTraceToolsTests.cs | 2 +- ...ngMcpDefaultTraceToolsWindowNoticeTests.cs | 229 + .../DarlingMcpFinOpsInventoryToolsTests.cs | 220 + ...rlingMcpFinOpsRecommendationsToolsTests.cs | 150 + .../DarlingMcpFinOpsToolsTests.cs | 213 + .../DarlingMcpHealthParserToolsTests.cs | 40 +- ...ngMcpHealthParserToolsWindowNoticeTests.cs | 437 ++ .../DarlingMcpHealthToolsTests.cs | 2 + .../DarlingMcpHostGateLiveTests.cs | 631 +- Darling/Darling.Tests/DarlingMcpHostTests.cs | 340 + .../DarlingMcpJobHistoryToolTests.cs | 931 +++ .../Darling.Tests/DarlingMcpJobToolsTests.cs | 8 +- .../DarlingMcpLatchSpinlockToolsTests.cs | 112 +- .../DarlingMcpMemoryGrantToolsTests.cs | 130 + .../DarlingMcpObjectStatsToolsTests.cs | 35 +- .../DarlingMcpPgPlanToolsTests.cs | 52 + .../DarlingMcpPgWindowD8aLiveTests.cs | 315 + .../DarlingMcpPgWindowD8bLiveTests.cs | 262 + .../DarlingMcpPlanCorrectionToolsTests.cs | 171 +- .../Darling.Tests/DarlingMcpPlanToolsTests.cs | 131 +- .../Darling.Tests/DarlingMcpPvsToolsTests.cs | 26 +- .../DarlingMcpQueryStoreHistoryToolsTests.cs | 182 + .../DarlingMcpServerAdminToolsTests.cs | 179 +- .../DarlingMcpServerTagToolsTests.cs | 401 ++ .../DarlingMcpServerTrendToolsTests.cs | 728 +++ .../DarlingMcpSessionToolsTests.cs | 335 +- .../DarlingMcpStoreHostToolsLiveTests.cs | 10 +- .../Darling.Tests/DarlingMcpTlsLiveTests.cs | 346 + Darling/Darling.Tests/DarlingMcpToolsTests.cs | 2 +- .../DarlingMcpTrendToolsTests.cs | 204 +- .../DarlingModuleMapGroupedSourceLiveTests.cs | 307 + .../DarlingModuleMapRecentRefreshLiveTests.cs | 527 ++ .../Darling.Tests/DarlingModuleMapTests.cs | 44 +- .../DarlingNetworkConfigEditorTests.cs | 138 + Darling/Darling.Tests/DarlingNetworkTests.cs | 153 +- .../DarlingObservabilityTests.cs | 4 +- .../DarlingOwnedSecretsCensusTests.cs | 236 + .../DarlingPasswordKeyFileHardeningTests.cs | 231 + .../DarlingPasswordKeyFileTests.cs | 915 +++ .../DarlingPasswordKeyFileUnixTests.cs | 311 + .../DarlingPasswordKeyRuntimeTests.cs | 183 + .../DarlingPasswordKeyStateTests.cs | 231 + .../DarlingPasswordKeyStoreLiveTests.cs | 1100 ++++ .../DarlingPathFilterGateTests.cs | 29 + .../DarlingPerDatabaseProbeFailureTests.cs | 7 +- .../DarlingPgDatabaseStatsLiveTests.cs | 4 + ...ngPgIndexBloatCoverageLivePostgresTests.cs | 2 +- .../DarlingPgReadSqlParsesLiveTests.cs | 1 + .../DarlingPgUnresolvedZoneWarningTests.cs | 52 + .../DarlingPrintEndpointTokenTests.cs | 49 +- .../DarlingPvsReaderServerClockTests.cs | 4 +- .../Darling.Tests/DarlingQueryHeatmapTests.cs | 198 +- .../DarlingQueryStoreRegressionsTests.cs | 385 +- .../DarlingReadsUseOnlyStoreViewsTests.cs | 16 +- .../DarlingReproScriptParityTests.cs | 127 + .../Darling.Tests/DarlingRetentionTests.cs | 102 +- .../DarlingRunTimeSchedulingTests.cs | 708 +++ .../DarlingRuntimePreflightTests.cs | 89 + .../DarlingSecretsCanUnprotectTests.cs | 49 + .../DarlingSecretsDecryptFailureTests.cs | 8 +- .../DarlingSecretsOwnedReferenceTests.cs | 500 ++ .../DarlingSecuritySplitLiveTests.cs | 119 +- .../Darling.Tests/DarlingSelfAlertTests.cs | 731 ++- .../DarlingSelfQueryMarkerTests.cs | 82 + .../Darling.Tests/DarlingServerTagsTests.cs | 83 +- .../DarlingSignificantWaitsReadTests.cs | 9 +- .../DarlingStoreHostProfileLiveTests.cs | 2 +- .../DarlingStoreHostProfileTests.cs | 72 + .../Darling.Tests/DarlingStoreUpgradeTests.cs | 111 +- .../DarlingSweepSchedulingTests.cs | 39 + .../Darling.Tests/DarlingTrendEmptyTests.cs | 2 +- .../DarlingTriageEndpointTests.cs | 4 + .../DarlingUnknownNetworkKeyTests.cs | 123 + .../DarlingUpgradeInstallTreeWaitTests.cs | 17 +- .../DarlingUpgradeShapesWorkflowTests.cs | 226 + ...DarlingWatermarkFloorPlanShapeLiveTests.cs | 3 +- ...DarlingWatermarkFloorScanBoundLiveTests.cs | 3 +- ...rlingWatermarkIndexEquivalenceLiveTests.cs | 3 +- .../Darling.Tests/DarlingWebAssetsTests.cs | 2 + .../Darling.Tests/DarlingWebEndpointsTests.cs | 257 +- .../DarlingWebFailureHandlingTests.cs | 225 +- .../DarlingWebHostGateLiveTests.cs | 468 +- Darling/Darling.Tests/DarlingWebOidcTests.cs | 14 + .../DarlingWebSecurityHeadersTests.cs | 471 ++ Darling/Darling.Tests/DarlingWebTlsTests.cs | 334 +- .../DarlingWebhookSecretsTests.cs | 265 + .../DarlingWorkerServerClockLiveTests.cs | 226 + Darling/Darling.Tests/DarlingWorkerTests.cs | 8 + ...taWindowFloorSnapshotAndRunLogLiveTests.cs | 248 + Darling/Darling.Tests/DataWindowFloorTests.cs | 426 ++ .../DatabaseConfigWebColumnsTests.cs | 169 + .../DatabaseFilterBindingLiveTests.cs | 107 + .../DatabaseFilterEmptyAnswerLiveTests.cs | 157 + .../Darling.Tests/DatabaseNameHistoryTests.cs | 506 ++ .../DatabaseNameParamTwinTests.cs | 278 + .../DatabaseSizeLatestPlanShapeLiveTests.cs | 12 +- .../Darling.Tests/DeadlockProcessRowsTests.cs | 375 ++ .../DeadlockRateBandRungTests.cs | 2 + .../DeadlockStoredIdentityTests.cs | 1 + .../DeadlocksTelemetryCursorTests.cs | 1 + ...milyIntervalCompletionLivePostgresTests.cs | 147 +- .../DeltaFamilyIntervalCompletionRungTests.cs | 15 +- .../DeployPowerShellProbeLimitTests.cs | 4 + .../DesktopDatabaseFilterLiveTests.cs | 430 ++ .../DiagnosticsBundleAliasHardeningTests.cs | 166 + .../DiagnosticsBundleAliasTests.cs | 266 + .../DiagnosticsBundleExitCodeTests.cs | 166 + .../DiagnosticsBundleHardeningTests.cs | 553 ++ .../DiagnosticsBundleHealthUncutLiveTests.cs | 112 + .../DiagnosticsBundleLeakLiveTests.cs | 429 ++ .../DiagnosticsBundleRound2Tests.cs | 551 ++ .../DiagnosticsBundleSealedTextTests.cs | 85 + .../DiagnosticsBundleSecretTests.cs | 217 + .../DiagnosticsBundleSeedFailureLiveTests.cs | 235 + .../DiagnosticsBundleServiceLogTests.cs | 137 + .../DiagnosticsBundleSizeTests.cs | 74 + .../DiagnosticsBundleStatementFilterTests.cs | 112 + .../DiagnosticsBundleStoreHistoryLiveTests.cs | 550 ++ .../DiagnosticsBundleVerbTests.cs | 117 + .../DiagnosticsBundleWebhookOpenTests.cs | 167 + .../Darling.Tests/DocCommentHygieneTests.cs | 49 +- .../Darling.Tests/DockerfileSdkPinTests.cs | 1 + .../DrillDownDopProvenanceParityTests.cs | 1 + .../DropXeSessionsHelpGuidanceTests.cs | 7 + .../DropXeSessionsPerInstallTests.cs | 411 ++ .../DropXeSessionsReadOnlyIntentTests.cs | 292 + .../Darling.Tests/DropXeSessionsVerbTests.cs | 567 +- .../DurationTrendDatabaseFilterLiveTests.cs | 627 ++ .../Darling.Tests/EffectiveStartUtcTests.cs | 241 + .../EmptyAnswerDatabaseEchoTests.cs | 97 + .../EmptyDatabaseListNoteLiveTests.cs | 187 + .../EmptyDatabaseListNoteTests.cs | 106 + .../EngineAwareVersionLabelTests.cs | 2 + .../EngineCapabilityMissTests.cs | 85 +- .../EntraProviderPackageCensusTests.cs | 2 + ...indowedReadsAreBoundedLivePostgresTests.cs | 29 +- .../EventsDatabaseFilterLiveTests.cs | 310 + .../Darling.Tests/ExcludedDatabasesTests.cs | 2 +- .../FactCollectorFailureReportingTests.cs | 1 + Darling/Darling.Tests/FactRankedTests.cs | 1 + .../Darling.Tests/FactSourceRegistryTests.cs | 1 + .../FailedSendRetryTrackerClockStepTests.cs | 1 + .../FailureNoteRedactionTests.cs | 1 + .../FileGrowthAlertStoreTests.cs | 1 + .../FileGrowthRiseUnitCensusTests.cs | 2 + .../FileIoDatabaseFilterLiveTests.cs | 371 ++ ...psApplicationConnectionsGoldenLiveTests.cs | 235 + .../FinOpsApplicationConnectionsViewTests.cs | 329 + .../FinOpsClickThroughRulesTests.cs | 155 + Darling/Darling.Tests/FinOpsCostTests.cs | 105 + .../FinOpsDatabaseResourcesGoldenLiveTests.cs | 276 + .../FinOpsDatabaseResourcesViewTests.cs | 259 + .../FinOpsDatabaseSizesViewTests.cs | 239 + .../FinOpsHighImpactReaderLiveTests.cs | 159 + .../Darling.Tests/FinOpsHotspotRegionTests.cs | 80 + .../FinOpsIdleCoverageInventoryLiveTests.cs | 145 + .../FinOpsIdleCoverageLiveTests.cs | 246 + .../Darling.Tests/FinOpsIdleCoverageSeed.cs | 39 + .../FinOpsIndexAnalysisGoldenLiveTests.cs | 268 + ...OpsIndexAnalysisLatestSnapshotLiveTests.cs | 284 + .../FinOpsIndexAnalysisLatestTiebreakTests.cs | 34 + .../FinOpsIndexAnalysisLimitBehaviourTests.cs | 121 + .../FinOpsIndexAnalysisViewLiveTests.cs | 330 + .../FinOpsIndexAnalysisViewSizeTests.cs | 131 + .../FinOpsIndexAnalysisViewTests.cs | 270 + .../FinOpsInventoryGoldenLiveTests.cs | 278 + .../FinOpsInventoryHealthScoreLiveTests.cs | 105 + .../FinOpsInventoryToolParityLiveTests.cs | 298 + .../FinOpsInventoryViewerFiguresTests.cs | 101 + .../FinOpsLockingDetailBehaviourTests.cs | 140 + .../FinOpsLockingHeatBehaviourTests.cs | 94 + .../FinOpsLockingPickerBehaviourTests.cs | 175 + .../FinOpsOptimizationGoldenLiveTests.cs | 280 + .../FinOpsOptimizationTieBreakTests.cs | 97 + .../FinOpsOptimizationViewTests.cs | 564 ++ Darling/Darling.Tests/FinOpsPageShellTests.cs | 230 + .../FinOpsPostgresTargetTests.cs | 162 + .../FinOpsRecommendationFiguresTests.cs | 369 ++ ...OpsRecommendationsCharacterizationTests.cs | 315 + .../FinOpsRecommendationsGoldenLiveTests.cs | 294 + .../FinOpsRecommendationsReadsLiveTests.cs | 412 ++ ...ecommendationsSecondaryReplicaLiveTests.cs | 199 + ...inOpsRecommendationsToolParityLiveTests.cs | 271 + .../Darling.Tests/FinOpsServerChoiceTests.cs | 239 + .../FinOpsStorageGrowthGoldenLiveTests.cs | 239 + .../FinOpsStorageGrowthViewTests.cs | 784 +++ ...FinOpsStorageGrowthWindowBehaviourTests.cs | 117 + .../FinOpsSubTabIndexPinTests.cs | 1 + ...inOpsTabApplicationConnectionsPageTests.cs | 135 + .../FinOpsTabDatabaseResourcesPageTests.cs | 145 + .../FinOpsTabDatabaseSizesPageTests.cs | 95 + .../FinOpsTabHighImpactPageTests.cs | 95 + .../FinOpsTabIndexAnalysisPageTests.cs | 409 ++ .../FinOpsTabLockingPageTests.cs | 84 + .../FinOpsTabOptimizationPageTests.cs | 208 + .../Darling.Tests/FinOpsTabPvsPageTests.cs | 91 + .../FinOpsTabRecommendationsPageTests.cs | 177 + .../FinOpsTabServerInventoryPageTests.cs | 245 + .../FinOpsTabStorageGrowthPageTests.cs | 269 + .../FinOpsTabSystemEventsColumnsPageTests.cs | 130 + .../FinOpsTabUtilizationPageTests.cs | 188 + .../FinOpsTabUtilizationRawPageTests.cs | 142 + .../FinOpsUtilizationFiguresLiveTests.cs | 127 + .../FinOpsUtilizationFiguresTests.cs | 180 + .../FinOpsUtilizationGoldenLiveTests.cs | 285 + .../FinOpsUtilizationViewLiveTests.cs | 302 + .../FinOpsWebReadParityLiveTests.cs | 141 + .../FinOpsWindowPickerBehaviourTests.cs | 124 + .../FindingsRetentionCrossSkuPinTests.cs | 1 + .../FindingsRetentionDefaultWiringPinTests.cs | 1 + .../FinOpsApplicationConnections/golden.json | 115 + .../FinOpsDatabaseResources/golden.json | 190 + .../Fixtures/FinOpsIndexAnalysis/golden.json | 1107 ++++ .../Fixtures/FinOpsInventory/golden.json | 150 + .../Fixtures/FinOpsOptimization/golden.json | 153 + .../FinOpsRecommendations/golden.json | 226 + .../Fixtures/FinOpsStorageGrowth/golden.json | 335 + .../Fixtures/FinOpsUtilization/golden.json | 181 + .../PasswordSeal/linux-self-check-vector.json | 19 + .../autoparam_select_adhoc_shell.xml | 1 + .../autoparam_select_prepared.xml | 1 + .../autoparam_update_adhoc_shell.xml | 1 + .../autoparam_update_prepared.xml | 1 + .../StatementScrub/two_statements.xml | 104 + .../Fixtures/column-value-filter-cases.json | 202 + .../Fixtures/migration-ladder-v3.9.0.sql | 5549 +++++++++++++++++ .../Fixtures/time-range-cases.json | 1149 ++++ ...dCollectionStaleNamesItsPopulationTests.cs | 1 + Darling/Darling.Tests/FleetGateSignalTests.cs | 44 +- .../Darling.Tests/FleetGateSkipCreditTests.cs | 10 +- .../Darling.Tests/FleetGateStartupTests.cs | 943 +++ .../FleetIdentifierScrubTests.cs | 4 + .../FleetOverviewAzureMasterScopeLiveTests.cs | 12 +- .../FleetPageAttentionFilterTests.cs | 2 +- .../FleetPgDeadlockFlatPairLiveTests.cs | 378 ++ .../FleetReadsAreBoundedByTheFleetTests.cs | 64 +- .../FleetSilenceBehaviourTests.cs | 149 + .../Darling.Tests/FleetSilencePageTests.cs | 85 + .../Darling.Tests/FleetSweepWebFeedTests.cs | 11 +- .../ForcePlanFailuresAccessPathTests.cs | 6 +- .../Darling.Tests/FrozenRollupLiveTests.cs | 142 +- .../Darling.Tests/GatedCollectorRowsTests.cs | 529 ++ .../GenericWebhookHeadersOriginTests.cs | 182 + .../Darling.Tests/GetReadLatencyLiveTests.cs | 122 +- .../GridAdoptionBehaviourTests.cs | 179 + .../Darling.Tests/GridFilterBehaviourTests.cs | 238 + .../Darling.Tests/GridSortBehaviourTests.cs | 184 + .../Darling.Tests/GridToolsBehaviourTests.cs | 170 + .../GridValueFilterBehaviourTests.cs | 383 ++ .../Darling.Tests/GuardStageCensusTests.cs | 326 + Darling/Darling.Tests/GuardStageScanner.cs | 581 ++ .../Darling.Tests/GuardStageWorkflowTests.cs | 513 ++ Darling/Darling.Tests/HostHeaderGuardTests.cs | 105 +- .../HourlyAttributionSpanTests.cs | 14 + .../Darling.Tests/HourlyWindowEdgesTests.cs | 44 +- .../Darling.Tests/HyperscaleLogSizeTests.cs | 3 +- ...otheticalIndexSingleStatementGuardTests.cs | 1 + .../IncidentDeliveryFilterTests.cs | 75 + .../IndexCleanupAnalyzerTests.cs | 39 + .../IndexUsageTruncationTests.cs | 6 +- Darling/Darling.Tests/InstallIdRungTests.cs | 134 + .../InstallIdTableOidRungTests.cs | 122 + .../IntervalFirstExecIndexRungTests.cs | 48 +- .../IntervalHonestHourlyRollupTests.cs | 34 +- .../IntervalRollupCountGuardLiveTests.cs | 348 ++ Darling/Darling.Tests/IoHourlyRollupTests.cs | 347 ++ .../Darling.Tests/JobHistoryCountTextTests.cs | 96 + ...roupedStatsMatchWindowFunctionLiveTests.cs | 4 +- .../JobHistoryIdentityEpochTests.cs | 13 +- Darling/Darling.Tests/JobHistoryPageTests.cs | 366 ++ .../JobHistoryPerServerTopNLiveTests.cs | 16 +- .../JobHistorySqlTextPinTests.cs | 306 + .../JobHistoryWatermarkEpochLiveTests.cs | 24 +- .../LatestCollectorOutcomeOrderTests.cs | 123 + .../LatestCollectorOutcomePlanTests.cs | 136 + .../Darling.Tests/LatestCpuReadShapeTests.cs | 6 +- .../Darling.Tests/LatestValueLookbackTests.cs | 24 +- .../Darling.Tests/LaunchMemoryGuardTests.cs | 837 +++ .../LegacyLongQueryRecordLiveTests.cs | 218 + .../LegacyPinCandidateRungTests.cs | 251 + .../LegacyPinResolverLiveTests.cs | 162 + .../ListenerTlsFailedStartTests.cs | 431 ++ ...eAnalysisServiceCancellationSourceTests.cs | 3 +- .../LiteJobHistoryLoadingCapTests.cs | 5 +- .../LiteJobHistorySourceTests.cs | 1 + ...erfmonWaitStatisticsArtifactSourceTests.cs | 1 + .../Darling.Tests/LiteReadsTraitGuardTests.cs | 213 + .../LiveCleanupConversionRatchetTests.cs | 1 + Darling/Darling.Tests/LiveClock.cs | 48 + Darling/Darling.Tests/LiveClockCensusTests.cs | 218 + .../LivePlanShapeIsolationCensusTests.cs | 1 + .../LivePostgresCollectionHygieneTests.cs | 10 +- .../Darling.Tests/LiveStoreSessionTimeZone.cs | 46 + .../LiveStoreSessionTimeZoneTests.cs | 41 + .../Darling.Tests/LocalClockBucketKeyTests.cs | 2 + .../LockedModeRestoreCoverageTests.cs | 4 +- .../LongQueryBannerSessionScopeTests.cs | 111 + .../LongQueryTraceInstanceGuardTests.cs | 318 + .../LongQueryTraceLifecycleTests.cs | 1868 ++++++ .../LongQueryTraceReadOnlyIntentTests.cs | 622 ++ .../LongQueryTraceRemovalTests.cs | 575 ++ .../LongQueryTraceRestartTests.cs | 177 + .../LongQueryTraceRunRowLiveTests.cs | 208 + .../LongRunningQueryExclusionKnobRungTests.cs | 1 + .../LongRunningQueryExclusionsTests.cs | 1 + .../Darling.Tests/ManageTagsBehaviourTests.cs | 359 ++ Darling/Darling.Tests/ManageTagsPageTests.cs | 99 + .../Darling.Tests/ManagedConfFileLiveTests.cs | 56 + Darling/Darling.Tests/ManagedConfFileTests.cs | 52 + .../ManagedConfMigrationTests.cs | 71 + .../ManagedConfRehearsalTests.cs | 34 +- .../ManagedConfWalRungStabilityTests.cs | 225 + .../MaterializationHoleProbeLiveTests.cs | 291 + .../MaterializationHoleRepairTests.cs | 32 +- .../MaterializationHoleScanShapeTests.cs | 6 +- Darling/Darling.Tests/McpArgumentTypeRows.cs | 420 ++ Darling/Darling.Tests/McpArgumentTypeTests.cs | 102 + .../McpFilterSemanticsLivePostgresTests.cs | 116 +- Darling/Darling.Tests/McpInProcessHost.cs | 340 + .../McpLatestSnapshotStampTests.cs | 65 +- Darling/Darling.Tests/McpPageContractTests.cs | 11 +- .../McpPayloadClockFrameDisciplineTests.cs | 1 + .../McpPayloadContractCensusTests.cs | 82 +- .../McpReadToolBudgetLiveTests.cs | 3 +- .../McpServiceParameterDiSeatCensusTests.cs | 1 + .../McpToolGuideHeads.AgStore.cs | 13 + .../McpToolGuideHeads.CustomView.cs | 36 +- .../Darling.Tests/McpToolGuideHeads.FinOps.cs | 74 + .../McpToolGuideHeads.FinOpsInventory.cs | 65 + ...McpToolGuideHeads.FinOpsRecommendations.cs | 58 + .../McpToolGuideHeads.Platform.cs | 4 + .../McpToolGuideHeads.SqlCore.cs | 17 +- Darling/Darling.Tests/McpToolGuideTests.cs | 1 + .../McpToolLatencyRecordingTests.cs | 53 +- .../DarlingMcpBlockingTools.txt | 9 +- .../DarlingMcpConfigHistoryTools.txt | 1 + .../DarlingMcpCustomViewTools.txt | 2 +- .../DarlingMcpDataTools.txt | 6 +- .../DarlingMcpDefaultTraceTools.txt | 1 + .../DarlingMcpFinOpsInventoryTools.txt | 5 + .../DarlingMcpFinOpsRecommendationsTools.txt | 3 + .../DarlingMcpFinOpsTools.txt | 9 + .../DarlingMcpHealthParserTools.txt | 1 + .../McpToolsListBudget/DarlingMcpJobTools.txt | 9 + .../DarlingMcpLongQueryTools.txt | 1 + .../DarlingMcpObjectStatsTools.txt | 3 + .../DarlingMcpPlanCorrectionTools.txt | 1 + .../DarlingMcpPlanTools.txt | 45 + .../McpToolsListBudget/DarlingMcpPvsTools.txt | 1 + .../DarlingMcpQueryStoreClutterTools.txt | 1 + .../DarlingMcpQueryStoreHistoryTools.txt | 7 + .../DarlingMcpServerAdminTools.txt | 4 + .../DarlingMcpServerTagTools.txt | 21 + .../DarlingMcpServerTrendTools.txt | 9 + .../DarlingMcpSessionTools.txt | 4 +- .../DarlingMcpSlowReadTools.txt | 8 + .../DarlingMcpStoreQueryHistoryTools.txt | 6 + .../DarlingMcpTrendTools.txt | 3 + .../Darling.Tests/McpToolsListBudgetTests.cs | 60 +- .../McpUnknownArgumentGuardTests.cs | 323 +- .../McpWholeNumberArgumentTests.cs | 286 + .../McpZeroIsAMeasurementTests.cs | 18 +- .../MeasurementContractCensusTests.cs | 8 +- .../MigrationDataMovingRungCensusPins.cs | 8 + .../MigrationLockHolderLogTests.cs | 279 + ...rationLockUncontendedAcquireTimingTests.cs | 89 + .../MigrationLockWaitContentionTests.cs | 52 - .../MigrationUpgradeLadderLiveTests.cs | 6 +- .../MuteRuleReloadRetentionTests.cs | 1 + .../MuteRuleServerIdRungTests.cs | 16 +- .../Darling.Tests/MuteRulesBehaviourTests.cs | 199 + Darling/Darling.Tests/MuteRulesPageTests.cs | 82 + .../NewestSnapshotReadsLiveTests.cs | 553 ++ .../NightlyCheckoutCommitPinTests.cs | 173 + .../NoInternalBusFilesGuardTests.cs | 55 + .../NoServiceObjectiveToResizeTests.cs | 10 +- .../NotificationRoutesRungTests.cs | 5 +- .../Darling.Tests/NotificationRoutingTests.cs | 10 + .../ObjectLockingDetailLiveTests.cs | 174 + .../ObjectLockingHeatLiveTests.cs | 218 + .../ObjectLockingWebDefaultTests.cs | 2 +- .../ObjectStatsDatabaseFilterLiveTests.cs | 457 ++ .../OptimizedLockingNoteTests.cs | 3 + .../Darling.Tests/OrphanStatePruneTests.cs | 1 + ...PlanBacklogJudgeRetireLivePostgresTests.cs | 412 ++ .../Darling.Tests/OversizedPlanBacklogPins.cs | 64 +- .../OversizedPlanBacklogReadPins.cs | 2 +- .../OversizedPlanSweepRunRecordTests.cs | 5 +- ...rviewCollectionHealthRollupRoutingTests.cs | 33 +- .../OwnedSecretReferenceRefusalTests.cs | 547 ++ .../PagerDutyAutoResolveRungTests.cs | 217 + ...arameterSensitiveDrillDownTextLiveTests.cs | 2 +- ...erSensitivityFiringSignatureParityTests.cs | 1 + .../PasswordBindingCensusTests.cs | 309 + .../PasswordKeyGrantParityTests.cs | 140 + Darling/Darling.Tests/PasswordKeyRungTests.cs | 251 + .../Darling.Tests/PasswordKeySeamLiveTests.cs | 204 + .../PasswordKeySelfCheckTests.cs | 193 + .../PasswordKeyTablesLiveTests.cs | 542 ++ Darling/Darling.Tests/PasswordSealTests.cs | 692 ++ .../PayloadDigestPassThroughLiveTests.cs | 587 ++ .../PayloadDimensionLiveTests.cs | 66 +- .../PerfmonCounterTypeRungTests.cs | 2 + .../Darling.Tests/PerfmonRegroupGateTests.cs | 137 + .../PerfmonRegroupLiveSupport.cs | 188 + .../Darling.Tests/PerfmonRegroupLiveTests.cs | 966 +++ .../PerfmonRegroupTimingLiveTests.cs | 168 + ...rfmonWaitStatisticsArtifactMcpLiveTests.cs | 98 + .../PgClusterObjectCensusTests.cs | 338 + .../Darling.Tests/PgClusterRolesCollection.cs | 44 + .../PgCpuCapacityHeadroomTests.cs | 2 + .../PgDeadlockRemaskAlertPagingLiveTests.cs | 833 +++ ...PgDeadlockRemaskFindingsPagingLiveTests.cs | 346 + .../Darling.Tests/PgDeadlockRemaskTests.cs | 95 +- .../PgEventToolsWindowFloorLiveTests.cs | 329 + Darling/Darling.Tests/PgFactCollectorTests.cs | 26 +- .../PgGridsMcpDataStartLiveTests.cs | 334 + .../PgIndexBloatCoverageTests.cs | 2 +- .../PgIndexBloatGridReachTests.cs | 4 +- Darling/Darling.Tests/PgIoStatsHourlyTests.cs | 612 ++ .../PgLogEventsCsvlogLiveTests.cs | 91 +- .../Darling.Tests/PgLogEventsPipelineTests.cs | 1 + Darling/Darling.Tests/PgLogHashKeyTests.cs | 11 + .../PgPlanLogParserQueryIdPrefixTests.cs | 58 + Darling/Darling.Tests/PgPlanShapeHashTests.cs | 277 + .../PgPlanViewerBehaviourTests.cs | 167 + .../PgReadBinaryFileCapabilityLiveTests.cs | 23 +- .../PgSensitiveStatementFilterLiveTests.cs | 112 +- .../PgSensitiveStatementFilterTests.cs | 62 +- ...PgServerConfigOverrideLivePostgresTests.cs | 2 +- .../PgServerConfigPendingRestartLiveTests.cs | 4 +- .../PgServerConfigScopeRungTests.cs | 6 +- ...PgServerLogTailCsvJsonRotationLiveTests.cs | 1 + .../PgServerLogTailRotationLiveTests.cs | 29 +- .../PgSettingRedactionLivePostgresTests.cs | 25 +- .../Darling.Tests/PgSettingRedactorTests.cs | 5 +- .../PgSettingRedactorTimingCollection.cs | 24 - .../PgSettingScrubCandidateCensusTests.cs | 1 + .../PgStatementEvictionNoteTests.cs | 9 +- .../PgStatementTextFailureRecordingTests.cs | 79 + .../PgStatementTextHealthSemanticsTests.cs | 89 + .../PgStatementTextScrubLiveTests.cs | 71 + .../PgStatementsDeallocLiveTests.cs | 121 +- Darling/Darling.Tests/PgTableTuningTests.cs | 13 +- .../PgTargetBetweenWavesV2Tests.cs | 1 + .../PgTargetConfigSnapshotBoundTests.cs | 4 +- .../PgTargetDeadlockDrillDownTests.cs | 11 +- .../Darling.Tests/PgTargetGrowthLiveTests.cs | 10 +- Darling/Darling.Tests/PgTargetGrowthTests.cs | 44 +- .../PgTargetMeasuredLineageTests.cs | 78 +- Darling/Darling.Tests/PgTargetQueriesTests.cs | 11 +- .../PgWaitInstrumentDisclosureTests.cs | 14 + ...riteStatsAndIndexUsageWindowNoticeTests.cs | 236 + .../PickerToggleDeferredRefreshSourceTests.cs | 1 + .../PlanCorrectionFrameLiveTests.cs | 2 +- .../PlanCorrectionsWebDefaultTests.cs | 4 +- Darling/Darling.Tests/PlanDigestCacheTests.cs | 295 + .../PlanForceActionDetailCensusTests.cs | 1 + ...lanRegressionBuiltDaysFallbackLiveTests.cs | 402 ++ .../PlanRegressionDailyBoundaryTests.cs | 116 + .../PlanRegressionDailyBuilderLiveTests.cs | 587 ++ .../PlanRegressionDailyParityLiveTests.cs | 353 ++ .../PlanRegressionDailyReadShapeLiveTests.cs | 338 + .../PlanRegressionDailyRungTests.cs | 533 ++ .../PlanRegressionDailyTriggerCostRig.cs | 241 + .../PlanRegressionDailyTriggerLiveTests.cs | 702 +++ .../PlanRegressionDrillDownReuseLiveTests.cs | 2 +- ...RegressionDrillDownWindowStartLiveTests.cs | 128 + ...RegressionIntervalTableEquivalenceTests.cs | 22 +- .../PlanRegressionReadShapeTests.cs | 41 +- .../PlanRegressionTimeoutCaveatTests.cs | 114 + .../Darling.Tests/PlanRowBehaviourTests.cs | 126 + .../PlanSync4512CancellationTests.cs | 31 +- .../PlanSync4512ParseErrorSurfacingTests.cs | 1 + .../PlanSync4530ReaderLiveTests.cs | 1 + .../PlanSync4535RuleNumberCensusTests.cs | 1 + ...lanSync4597ServerContextFieldsLiveTests.cs | 1 + ...ync4602AnalysisServiceConfigCensusTests.cs | 1 + .../Darling.Tests/PlanViewerBehaviourTests.cs | 473 ++ .../PlanViewerCapabilityPinTests.cs | 1 + .../PlanViewerRuntimeSummaryNestingTests.cs | 23 +- ...gresDatabaseListScreenLivePostgresTests.cs | 15 +- .../PostgresEngineGateBehaviorTests.cs | 15 +- .../PostgresFaultOutcomeTests.cs | 7 +- .../Darling.Tests/PostgresOpenTimeoutTests.cs | 189 + .../PostgresTargetConfigTests.cs | 4 + ...rocedureStatsDeferredPlanFetchLiveTests.cs | 519 ++ .../ProcedureStatsFloorAndHistoryLiveTests.cs | 326 + .../ProcedureStatsIdleHoursLiveTests.cs | 462 ++ .../ProcedureStatsIdleRowsLiveTests.cs | 246 + .../ProcedureStatsIdleSkipTests.cs | 365 ++ .../ProcedureStatsPlanCadenceTests.cs | 90 +- .../ProcedureStatsPlanFetchTests.cs | 485 ++ .../ProcedureStatsPlanReuseTests.cs | 1323 ++++ .../ProcedureStatsSqlGoldenTests.cs | 103 + .../ProcedureTrendHourlyQueryViewPinTests.cs | 66 + .../ProcedureTrendQueryViewPinTests.cs | 61 + .../ProcedureTrendRunModelLiveTests.cs | 394 ++ .../ProductVersionDeclarationTests.cs | 2 + .../ProductiveZeroBandingTests.cs | 28 +- .../ProvisionRolesAclDriftTests.cs | 9 + .../ProvisionRolesCollisionGuardTests.cs | 1 + .../ProvisioningSerializationTests.cs | 784 +++ .../PublicConnectCatalogReadLiveTests.cs | 65 + Darling/Darling.Tests/PublicConnectDelta.cs | 30 + .../PublicConnectGuardAttribute.cs | 108 + .../Darling.Tests/PublicConnectGuardTests.cs | 46 + .../Darling.Tests/PurgeContinuationTests.cs | 275 + .../PvsSkippedFieldsParityLiveTests.cs | 127 + Darling/Darling.Tests/PvsStatsStoreTests.cs | 2 +- .../QsCaptureModeRouteKnobToastRungTests.cs | 2 + .../QueryEventsDatabaseFilterLiveTests.cs | 303 + .../QueryHeatmapSqlTwinParityTests.cs | 8 +- .../QueryHighDopStaleMaxDopParityTests.cs | 1 + .../QueryReadsWebDispatchLiveTests.cs | 207 + .../QueryStatsDeferredPlanFetchLiveTests.cs | 430 ++ .../QueryStatsHourLedgerTests.cs | 476 ++ .../QueryStatsHourLedgerWriterLiveTests.cs | 456 ++ .../QueryStatsHourLedgerWriterTests.cs | 154 + Darling/Darling.Tests/QueryStatsLedgerSeed.cs | 64 + .../Darling.Tests/QueryStatsPlanFetchTests.cs | 297 + .../Darling.Tests/QueryStatsSqlGoldenTests.cs | 77 + .../QueryStatsWriterCensusPins.cs | 537 ++ .../QueryStoreAzureLoopWatermarkLiveTests.cs | 289 + ...eryStoreBackfillCandidateCacheLiveTests.cs | 446 ++ .../QueryStoreBackfillDroppedDatabaseTests.cs | 534 ++ .../Darling.Tests/QueryStoreBackfillTests.cs | 1 + .../QueryStoreBackgroundIndexesLiveTests.cs | 652 ++ .../QueryStoreBackgroundIndexesTests.cs | 452 ++ .../QueryStoreClutterLivePostgresTests.cs | 6 +- .../Darling.Tests/QueryStoreClutterTests.cs | 11 +- .../QueryStoreComposeStampLiveTests.cs | 522 ++ .../QueryStoreComposeStampRaceLiveTests.cs | 354 ++ .../QueryStoreComposeStampTests.cs | 479 ++ ...ryStoreComposeStampWriterGuardLiveTests.cs | 257 + ...eryStoreDatabaseWatermarkWiringPinTests.cs | 41 +- .../QueryStoreFetchProbeLivePostgresTests.cs | 201 +- .../QueryStoreGroupMembersLiveTests.cs | 136 + .../QueryStoreGroupMembersTests.cs | 106 + .../QueryStoreHistoryBehaviourTests.cs | 197 + .../QueryStoreHistoryLivePostgresTests.cs | 313 + .../QueryStoreHistoryPanelStateTests.cs | 115 + .../QueryStoreHistoryWideTierLiveTests.cs | 126 + ...toreIntervalBoundedReadPruningLiveTests.cs | 329 + .../QueryStoreIntervalBrinLiveTests.cs | 656 ++ .../QueryStoreIntervalConvergenceLiveTests.cs | 907 +++ .../QueryStoreIntervalFloorWarmUpTests.cs | 631 ++ .../QueryStoreIntervalLatestWriterTests.cs | 150 + ...ryStoreIntervalMaintenancePassLiveTests.cs | 143 + .../QueryStoreIntervalPartitionRungTests.cs | 751 +++ ...ntervalPartitionedStoreMetricsLiveTests.cs | 226 + .../QueryStoreIntervalPartitionsLiveTests.cs | 776 +++ .../QueryStoreIntervalPartitionsTests.cs | 314 + .../QueryStoreIntervalPurgeRowCappedTests.cs | 622 ++ .../QueryStoreIntervalRetentionLiveTests.cs | 326 + .../QueryStoreIntervalRetentionPinTests.cs | 119 + .../QueryStoreIntervalRetentionTestKit.cs | 146 + ...eryStoreIntervalWideBelowFloorLiveTests.cs | 36 +- ...reIntervalWideBelowFloorRoutesLiveTests.cs | 1 + ...ueryStoreIntervalWideBrinIndexLiveTests.cs | 90 +- .../QueryStoreIntervalWideBrinIndexTests.cs | 24 +- ...eryStoreIntervalWideFirstExecFloorTests.cs | 101 +- .../QueryStoreIntervalWideGateTests.cs | 48 +- .../QueryStoreIntervalWideGridLiveTests.cs | 5 +- .../QueryStoreIntervalWideHistoryNoteTests.cs | 46 +- ...oreIntervalWideMinSchemaVersionPinTests.cs | 1 + .../QueryStoreLegacyRowIndexLiveTests.cs | 369 ++ .../QueryStorePlanViaMapLiveTests.cs | 363 ++ .../Darling.Tests/QueryStoreRankedHarness.cs | 346 + .../QueryStoreReadsDatabaseFilterLiveTests.cs | 302 + .../QueryStoreSliceTieBreakSourceTests.cs | 14 + .../QueryStoreStatePruneTests.cs | 1 + .../QueryStoreTopDailyBuilderLiveTests.cs | 758 +++ .../QueryStoreTopDailyReadLiveTests.cs | 532 ++ .../QueryStoreTopDailyRungTests.cs | 457 ++ ...ueryStoreTopLiteralEndStraddleLiveTests.cs | 43 +- .../QueryStoreTopMcpLiveTests.cs | 55 +- .../QueryStoreTopWebDefaultTests.cs | 4 +- .../Darling.Tests/QueryStoreTopWindowTests.cs | 19 +- .../QueryStoreTrendIntervalRateTests.cs | 2 +- .../QueryStoreTrendRoutingTests.cs | 11 +- ...QueryStoreWideFleetEligibilityLiveTests.cs | 227 + .../QueryStoreWideReadGuardTests.cs | 385 ++ .../QueryStoreWideUpperBoundLiveTests.cs | 305 + .../QueryStoreWriteFencePinTests.cs | 1 + .../QueryStoreWriterSetPinTests.cs | 1 + .../QuietServerWatermarkLadderLiveTests.cs | 369 ++ .../RawChunkIntervalReconcilerLiveTests.cs | 12 +- .../RawGatePerSlotSourceFloorLiveTests.cs | 45 +- .../RawPurgeConvergeLiveTests.cs | 42 +- .../RawPurgeTriggerGateErrorTests.cs | 50 +- .../Darling.Tests/RawPurgeTriggerLiveTests.cs | 117 +- .../RawRepairEpochTriggerLiveTests.cs | 40 +- .../RawRetentionPurgeJobLiveTests.cs | 1 + .../RawTablesLeaveCatalogSweepLiveTests.cs | 52 +- .../RawWindowFloorMcpLiveTests.cs | 72 + .../RawWindowFloorQuietStartTests.cs | 405 ++ .../RawWindowFloorViewerPortTests.cs | 98 +- .../Darling.Tests/RdsAssumeRoleSeamTests.cs | 391 ++ .../RdsDeadlockReportResumeTests.cs | 2 +- .../RdsEndpointVerifierTests.Login.cs | 462 ++ .../Darling.Tests/RdsEndpointVerifierTests.cs | 387 ++ .../RdsLogStateConcurrencyTests.cs | 477 ++ .../RdsResumePersistenceTests.cs | 10 +- .../ReadLatencyAccumulatorTests.cs | 17 + .../ReadLatencyBudgetEdgeTests.cs | 43 + .../ReadLatencyFlushLiveTests.cs | 29 + .../ReadLatencyWebRecordingTests.cs | 30 + .../ReadScopeGateFaultLiveTests.cs | 92 + .../ReadScopeStatementCaptureLiveTests.cs | 241 + Darling/Darling.Tests/ReadScopeTests.cs | 260 + .../ReadmeDerivedCountPinTests.cs | 1 + .../RecurrenceLabelStoreReadTests.cs | 1 + Darling/Darling.Tests/ReenterLineTests.cs | 57 + .../RefreshCeilingProvenancePinTests.cs | 26 +- .../RefreshCeilingStalenessTests.cs | 42 +- .../RegisteredServerSettingDriftTests.cs | 85 +- .../RegressedFromProductiveTests.cs | 4 +- .../ReleaseStatementGateWorkflowTests.cs | 179 + .../Darling.Tests/ReleaseWalkWordingTests.cs | 191 + ...CollectorServiceCancellationCensusTests.cs | 2 + .../Darling.Tests/RepoFileAdoptionTests.cs | 46 + .../RepoFileResolutionEquivalenceTests.cs | 1 + .../ReproScriptLivePostgresTests.cs | 177 + .../Darling.Tests/RestartRowIndexLiveTests.cs | 274 + .../RetentionCollectionYieldLiveTests.cs | 186 + .../RetentionCollectionYieldTests.cs | 922 +++ ...ntionHoldAndCadenceRawRedirectLiveTests.cs | 40 +- .../RetentionReevaluationTests.cs | 25 +- .../RetiredBaselineAggregateTests.cs | 37 +- .../RetiredBaselineSweepRecoveryTests.cs | 251 + .../Darling.Tests/RightSizingWindowTests.cs | 53 +- .../Darling.Tests/RollupBackfillLiveTests.cs | 49 +- .../RollupCoverageCeilingLiveTests.cs | 15 +- .../RollupCoverageRoutingTests.cs | 97 +- .../RollupCoverageWarmupTests.cs | 50 + .../RollupFloorCacheLiveTests.cs | 361 +- .../Darling.Tests/RollupFloorCacheTests.cs | 74 +- .../RollupFloorStoreCacheTests.cs | 213 + .../RollupIoFloorFillLiveTests.cs | 125 + .../RollupTileDescriptionTests.cs | 2 +- .../RollupWindowEndBoundTests.cs | 16 +- .../RowPlanReadLivePostgresTests.cs | 750 +++ .../RuntimePreconditionMissTests.cs | 26 +- .../SameStatementPileupSourceCensusTests.cs | 5 +- .../Darling.Tests/ScopedLoadOrderingTests.cs | 21 +- .../ScratchDatabaseDisposalCensusTests.cs | 378 ++ Darling/Darling.Tests/ScratchDatabaseSweep.cs | 193 + .../ScratchDatabaseSweepLiveTests.cs | 431 ++ .../ScratchDatabaseSweepTests.cs | 129 + Darling/Darling.Tests/ScratchPostgres.cs | 532 +- .../ScratchPostgresExitDrainTests.cs | 329 + .../ScratchPostgresQuiesceTests.cs | 310 + .../SealedSecretResolverTests.cs | 393 ++ .../SecondaryReplicaScopeLiveTests.cs | 533 ++ .../SecondaryReplicaScopeTests.cs | 277 + .../SelfAlertDeliveryStampTests.cs | 2 +- .../SelfAlertFailedSendCensusTests.cs | 79 +- .../SelfAlertFailedSendRetryTests.cs | 140 +- .../Darling.Tests/SelfAlertFleetPassTests.cs | 877 +++ .../SelfManagedStoreRulesStartPinTests.cs | 67 + .../SensitiveStatementAutoParamTests.cs | 767 +++ .../Darling.Tests/SensitiveStatementCorpus.cs | 165 + .../SensitiveStatementHostFilterTests.cs | 314 + .../SensitiveStatementJsonTests.cs | 299 + .../SensitiveStatementOutputFilterTests.cs | 629 ++ .../SensitiveStatementRawCleanTests.cs | 248 + .../SensitiveStatementSessionTests.cs | 503 ++ .../SensitiveStatementXmlTests.cs | 636 ++ .../SensitiveStatementsParityLiveTests.cs | 190 + .../Darling.Tests/SensitiveStatementsTests.cs | 1153 ++++ .../SeparatelyMonitoredListNoteTests.cs | 6 +- .../SerialLoopStoreSizeSourceTests.cs | 9 +- Darling/Darling.Tests/ServerAddRouteTests.cs | 586 ++ .../ServerAddViewerRoleLiveTests.cs | 514 ++ .../Darling.Tests/ServerConnectPolicyTests.cs | 10 +- .../ServerDatabasesEndpointLiveTests.cs | 555 ++ Darling/Darling.Tests/ServerEditCoreTests.cs | 743 +++ .../Darling.Tests/ServerEditGateLiveTests.cs | 154 + Darling/Darling.Tests/ServerEditLiveTests.cs | 583 ++ .../ServerEditPasswordRuleViewerTests.cs | 422 ++ Darling/Darling.Tests/ServerEditRouteTests.cs | 822 +++ .../ServerEditViewerRoleLiveTests.cs | 389 ++ .../ServerIdentityLockLiveTests.cs | 775 +++ .../ServerIdentityLockSourcePinTests.cs | 203 + .../ServerInventoryFleetMetricsLiveTests.cs | 17 + .../ServerInventoryFleetMetricsTests.cs | 11 +- .../ServerLocalReadFrameDisciplineTests.cs | 9 +- Darling/Darling.Tests/ServerPageTabsTests.cs | 25 +- .../Darling.Tests/ServerPageTitlePinTests.cs | 3 +- .../ServerPasswordFromRequestTests.cs | 586 ++ .../ServerRemovalClearsTagsLiveTests.cs | 110 + .../ServerTabColumnGroupsPinTests.cs | 47 + .../ServerTabEmptyStatesAndHeadersTests.cs | 124 + .../ServerTabGridColumnPinTests.cs | 78 + .../ServerTabToolbarAndTooltipsTests.cs | 223 + .../Darling.Tests/ServerTagCoverageTests.cs | 106 + .../ServerTagDeleteWarningTests.cs | 49 + .../ServerTagMcpRoleLiveTests.cs | 111 + Darling/Darling.Tests/ServerTagRulesTests.cs | 151 + .../Darling.Tests/ServerTagStoreLiveTests.cs | 296 + .../ServerTagViewerRoleLiveTests.cs | 182 + .../ServerTagWebRoutesLiveTests.cs | 160 + .../Darling.Tests/SharedBaselineCacheTests.cs | 2 +- .../SharedBaselineModelPinTests.cs | 2 + .../ShowPlanParserParameterizedTextTests.cs | 66 + .../Darling.Tests/SidebarSearchGroupTests.cs | 249 + .../Darling.Tests/SingleInstanceGuardTests.cs | 1 + Darling/Darling.Tests/SlowClassCensusTests.cs | 348 ++ Darling/Darling.Tests/SlowReadLogTests.cs | 538 ++ Darling/Darling.Tests/SlowReadsLiveTests.cs | 269 + Darling/Darling.Tests/SlowReadsRungTests.cs | 171 + Darling/Darling.Tests/SqlTextPinTests.cs | 1 + Darling/Darling.Tests/StaTestThread.cs | 277 + Darling/Darling.Tests/StaTestThreadTests.cs | 195 + Darling/Darling.Tests/StaThreadCensusTests.cs | 139 + .../Darling.Tests/StallWaitProbeLiveTests.cs | 243 + .../StallWaitProbeQueryTextPinTests.cs | 139 + .../StartupCommandTimeoutTests.cs | 53 +- Darling/Darling.Tests/StateSelfAlertTests.cs | 491 ++ .../StatementAlertReadCutLiveTests.cs | 87 + .../StatementAnalysisReadCutLiveTests.cs | 302 + .../StatementAnalysisReadCutPinTests.cs | 109 + .../StatementCollectionCensusCases.cs | 53 + .../StatementColumnCensusTests.cs | 972 +++ .../StatementCutSourceScanTests.cs | 875 +++ .../StatementDrillDownTextFetchLiveTests.cs | 481 ++ .../StatementFilterAlertTests.cs | 602 ++ .../StatementFilterAnalysisReadsLiveTests.cs | 164 + .../StatementFilterBlockingReadsLiveTests.cs | 450 ++ .../Darling.Tests/StatementFilterCensus.cs | 257 + .../StatementFilterDeliveryTests.cs | 434 ++ ...atementFilterDrillDownWarningsLiveTests.cs | 109 + ...tatementFilterPlanReaderSourceScanTests.cs | 169 + .../StatementFilterPlanReadsLiveTests.cs | 608 ++ .../StatementFilterQueryReadsLiveTests.cs | 310 + .../Darling.Tests/StatementFilterWarmUp.cs | 37 + .../StatementFilterWarmUpSourceTests.cs | 54 + .../StatementFilterWarmUpWaitTests.cs | 475 ++ .../Darling.Tests/StatementFilterWebTests.cs | 323 + .../StatementMarkerReaderTests.cs | 425 ++ .../StatementPreviewCutLiveTests.cs | 125 + .../StatementPreviewCutPinTests.cs | 102 + .../StatementScrubBlockingTests.cs | 504 ++ Darling/Darling.Tests/StatementScrubCanary.cs | 131 + .../StatementScrubDeadlockLiveTests.cs | 175 + .../StatementScrubDerivedStoreTests.cs | 312 + .../StatementScrubEndToEndLiveTests.cs | 639 ++ .../StatementScrubEventsTests.cs | 289 + .../StatementScrubPlanCommandResultTests.cs | 177 + .../StatementScrubQueryStatsLiveTests.cs | 310 + .../StatementScrubQueryStatsTests.cs | 449 ++ .../StatementScrubQueryStoreLiveTests.cs | 320 + .../StatementScrubQueryStoreTests.cs | 181 + .../StatementScrubRecordingWriter.cs | 69 + .../StatementScrubSnapshotTests.cs | 457 ++ .../StatementScrubViewerPlanWithheldTests.cs | 153 + .../StatementScrubViewerSavePathsTests.cs | 108 + .../Darling.Tests/StatusBarBehaviourTests.cs | 153 + .../StorageAndFileIoWebDispatchLiveTests.cs | 147 + .../StorageCommandTimeoutTests.cs | 41 +- .../StorageDatabaseFilterLiveTests.cs | 353 ++ .../StorageGrowthNewTableCutLiveTests.cs | 126 + .../StorageGrowthNoHistoryLiveTests.cs | 98 +- .../StoreApplicationNameCensusTests.cs | 4 + ...StoreConfigProviderConnectionMatchTests.cs | 169 + .../StoreConnectionStringAppendPinTests.cs | 1 + .../StoreCopyPhaseLivePostgresTests.cs | 22 +- .../StoreDimensionRowCountLiveTests.cs | 134 + .../StoreInstallIdBindingRuleTests.cs | 71 + .../Darling.Tests/StoreInstallIdLiveTests.cs | 797 +++ .../StoreInstallIdRefusedClusterLiveTests.cs | 454 ++ .../StoreInstallIdSourcePinTests.cs | 127 + .../StoreIsOnThisMachineTests.cs | 31 +- .../Darling.Tests/StoreLogClassifierTests.cs | 359 +- .../StoreLogSeverityLocaleTests.cs | 11 +- .../StoreMaintenanceTickLaunchTests.cs | 269 + .../StoreObjectConvergenceTests.cs | 153 +- .../StoreObjectHourlyStepSelectionTests.cs | 73 + .../StoreQueryHistoryToolLiveTests.cs | 507 ++ .../StoreQueryHistoryToolTests.cs | 68 + .../Darling.Tests/StoreSelfMetricsTests.cs | 123 +- .../StoreServerPasswordRulesLiveTests.cs | 644 ++ .../StoreSessionTimeZonePinCensusTests.cs | 4 + .../StoreSqlClockDisciplineTests.cs | 1 + .../StoreStatementHistoryLiveTests.cs | 383 ++ .../StoreStatementHistoryRungTests.cs | 222 + .../StoreStatementHistoryTests.cs | 116 + .../StoreStatementStatsLiveTests.cs | 47 +- .../Darling.Tests/StoreStatementStatsTests.cs | 57 +- .../StoreToastAndCheckpointerTests.cs | 27 +- .../StoredPlanAnalysisFilterTests.cs | 182 + .../StragglerCommandTimeoutTests.cs | 9 +- .../SwallowedItemFailureTests.cs | 41 +- .../SweepBodyDetachPolicyTests.cs | 6 +- Darling/Darling.Tests/TargetProviderTests.cs | 39 + .../TargetSessionIdCacheTests.cs | 238 + .../TargetSessionIdLiveStoreTests.cs | 75 + .../TempDbReservedLabelProvenanceTests.cs | 2 + .../TestData/password-seal-v1-vector.json | 22 + Darling/Darling.Tests/TestKeyRings.cs | 23 + .../TestsNoSilentPassGuardTests.cs | 126 + Darling/Darling.Tests/TimeHonestyRungTests.cs | 2 + .../TimeRangePickerControlTests.cs | 199 + Darling/Darling.Tests/TimeRangeTests.cs | 332 + .../TimescaleAggregateCompressionTests.cs | 19 +- .../TimescaleAvailabilityReprobeTests.cs | 59 +- .../TimescaleContinuousAggregateTests.cs | 58 +- .../Darling.Tests/TimescaleSupportTests.cs | 155 +- Darling/Darling.Tests/TimingClaim.cs | 41 + Darling/Darling.Tests/TimingCollection.cs | 32 + .../Darling.Tests/TimingTestCensusTests.cs | 168 + Darling/Darling.Tests/TimingTestScanner.cs | 275 + .../TopCpuQueriesTextLiveTests.cs | 2 +- .../TopCpuQueriesTextShapeTests.cs | 2 +- .../TopFillEmptyRoundLiveTests.cs | 147 + .../TopProceduresDetailLiveTests.cs | 130 + .../TopProceduresHourlyRoutingLiveTests.cs | 12 + .../TopProceduresHourlyRoutingTests.cs | 40 +- .../TopProceduresIoRouteLiveTests.cs | 289 + .../TopQueriesDatabaseFilterLiveTests.cs | 625 ++ .../TopQueriesDetailLiveTests.cs | 167 + .../TopQueriesHourlyRoutingLiveTests.cs | 31 +- .../TopQueriesHourlyRoutingTests.cs | 77 +- .../TopQueriesIoRouteLiveTests.cs | 354 ++ .../TopRankingDifferentialLiveTests.cs | 572 ++ .../TopRankingHourlyLiveTests.cs | 293 + Darling/Darling.Tests/TopRankingLiveTests.cs | 394 ++ .../TopRankingNoticeBehaviourTests.cs | 152 + .../TopRankingOnePassBaseline.cs | 405 ++ .../Darling.Tests/TopRankingPagePinTests.cs | 104 + Darling/Darling.Tests/TopRankingTests.cs | 239 + .../Darling.Tests/TopRankingTwoPassTests.cs | 204 + .../Darling.Tests/TopTextLookupFillTests.cs | 402 ++ .../TopTextLookupPlanLiveTests.cs | 358 ++ .../Darling.Tests/TopTextTieBreakLiveTests.cs | 222 + .../TrendPayloadBudgetLiveTests.cs | 41 +- .../Darling.Tests/TrendStaleStatsLiveTests.cs | 183 + .../Darling.Tests/TsqlConventionGuardTests.cs | 43 +- .../TuningDropLockRecoveryLiveTests.cs | 3 + .../TuningHourlyCreateLiveTests.cs | 199 + .../TuningIndexReaderPairingTests.cs | 38 +- .../TuningReaderBudgetLiveTests.cs | 6 +- Darling/Darling.Tests/TuningStartPasses.cs | 31 + .../TuningStartPathOneBuildTests.cs | 150 + .../UnpooledDropConnectionCensusTests.cs | 709 +++ ...alidateConfigOldFormatPasswordLiveTests.cs | 168 + .../ViewSavedServerScopeTests.cs | 180 + .../Viewer4627EdgeCallerTests.cs | 1 + Darling/Darling.Tests/Viewer4629Tests.cs | 1 + .../ViewerAddServerIdCollisionTests.cs | 11 +- .../ViewerAlertClockCacheTests.cs | 2 +- .../ViewerApplyToAllClockTests.cs | 2 +- .../Darling.Tests/ViewerAwsRoleFieldsTests.cs | 229 + .../ViewerAwsRoleUpsertLiveTests.cs | 120 + ...rBlockedProcessReportsCoverageLiveTests.cs | 632 ++ .../ViewerBlockingChartsDataStartLiveTests.cs | 85 + .../ViewerBlockingChartsDataStartPinTests.cs | 88 + .../ViewerBlockingChartsDataStartTests.cs | 69 + .../ViewerBlockingSourceLabelTests.cs | 120 + .../ViewerBlockingSourceLivePostgresTests.cs | 194 + .../ViewerButtonAccessKeyTests.cs | 2 + .../ViewerCalendarRetentionPortTests.cs | 7 +- ...tterAndMemoryPressureDataStartLiveTests.cs | 475 ++ ...utterAndMemoryPressureDataStartPinTests.cs | 255 + ...rClutterAndMemoryPressureDataStartTests.cs | 414 ++ .../ViewerCollectionHealthLoadTests.cs | 191 + .../ViewerCollectionLogDataStartLiveTests.cs | 378 ++ .../ViewerCollectionLogDataStartTests.cs | 365 ++ .../ViewerCollectorDurationChartLiveTests.cs | 337 + .../ViewerCollectorDurationChartTests.cs | 261 + .../ViewerCollectorDurationHoverTests.cs | 210 + .../ViewerCommandTimeoutTests.cs | 37 +- .../ViewerConfigChangesDataStartTests.cs | 333 + .../ViewerConfigGridCollectedColumnTests.cs | 80 + .../ViewerControlPlaneStage3bTests.cs | 14 +- .../Darling.Tests/ViewerCpuSchedulerTests.cs | 11 + .../Darling.Tests/ViewerDailyHealthTests.cs | 5 +- ...erDailySummaryAzureMasterScopeLiveTests.cs | 4 +- .../Darling.Tests/ViewerDataServiceTests.cs | 63 + .../ViewerDataStartBannerTests.cs | 332 + .../ViewerDisabledListBoxThemeTests.cs | 24 +- .../ViewerEventDataStartLiveTests.cs | 313 + .../ViewerEventDataStartTests.cs | 454 ++ .../ViewerEventGridsDataStartLiveTests.cs | 538 ++ .../ViewerFileIoBlockingTests.cs | 172 +- ...OpsApplicationConnectionsDelegatesTests.cs | 65 + ...erFinOpsDatabaseResourcesDelegatesTests.cs | 56 + .../ViewerFinOpsHighImpactDelegatesTests.cs | 56 + ...ViewerFinOpsIndexAnalysisDelegatesTests.cs | 52 + .../ViewerFinOpsInventoryDelegatesTests.cs | 65 + .../ViewerFinOpsOptimizationDelegatesTests.cs | 70 + .../ViewerFinOpsPickerRefreshPinTests.cs | 1 + ...ewerFinOpsRecommendationsDelegatesTests.cs | 59 + .../ViewerFinOpsRecommendationsTests.cs | 25 +- .../ViewerFinOpsRowClockTests.cs | 70 +- ...ViewerFinOpsStorageGrowthDelegatesTests.cs | 89 + .../ViewerFinOpsUtilizationDelegatesTests.cs | 50 + ...rFinOpsUtilizationFiguresDelegatesTests.cs | 45 + .../ViewerFinOpsUtilizationNoCpuTests.cs | 87 + .../ViewerFleetAzureMasterScopeLiveTests.cs | 6 +- .../Darling.Tests/ViewerFleetRollupTests.cs | 21 +- .../ViewerFleetScreenWalkFixesTests.cs | 243 + .../ViewerFleetTimerGuardTests.cs | 1 + Darling/Darling.Tests/ViewerGrantReplay.cs | 75 + .../ViewerGridPayloadColumnOrderPinTests.cs | 2 + .../ViewerGridTimeColumnSortMemberTests.cs | 21 + .../ViewerGridTimeTextZoneTests.cs | 20 +- .../ViewerHourlyRouteBlankColumnsLiveTests.cs | 304 + .../ViewerHourlyWindowEdgesLiveTests.cs | 339 + .../ViewerHourlyWindowEdgesTests.cs | 291 + .../Darling.Tests/ViewerIndexAnalysisTests.cs | 13 +- .../ViewerIoRollupRouteLiveTests.cs | 322 + .../Darling.Tests/ViewerIoRollupRouteTests.cs | 130 + .../ViewerJobHistoryDataStartLiveTests.cs | 287 + .../ViewerJobHistoryDataStartTests.cs | 285 + .../Darling.Tests/ViewerLatchSpinlockTests.cs | 6 +- .../ViewerLocalStateBehaviourTests.cs | 244 + .../ViewerLongQueriesDataStartTests.cs | 185 + .../ViewerNotificationRouteDisplayTests.cs | 16 +- ...rOverviewBlockingLaneDataStartLiveTests.cs | 192 + ...erOverviewBlockingLaneDataStartPinTests.cs | 85 + ...iewerOverviewBlockingLaneDataStartTests.cs | 63 + .../ViewerOverviewExplainsItselfTests.cs | 11 +- .../ViewerOverviewSearchTests.cs | 149 + .../Darling.Tests/ViewerPasswordKeyTests.cs | 439 ++ .../ViewerPerfmonRunningJobsTests.cs | 180 +- .../ViewerPgSnapshotGridTimeTests.cs | 229 + .../ViewerPlanCorrectionsDataStartTests.cs | 183 + .../ViewerPostgresDataStartLiveTests.cs | 80 + .../ViewerPostgresDataStartTests.cs | 178 + .../ViewerPostgresGridTimeTextTests.cs | 39 +- .../Darling.Tests/ViewerPostgresTabsTests.cs | 4 +- .../ViewerProbeWatchCensusTests.cs | 160 + .../Darling.Tests/ViewerQueriesRestTests.cs | 43 +- Darling/Darling.Tests/ViewerQueriesTests.cs | 78 +- .../ViewerQueryGridsDataStartLiveTests.cs | 569 ++ .../ViewerQueryHeatmapDataStartTests.cs | 175 + ...eryStoreHistoryTotalExecutionsLiveTests.cs | 297 + ...erQueryStoreHistoryTotalExecutionsTests.cs | 306 + ...ewerQueryStoreRegressionsDataStartTests.cs | 197 + .../Darling.Tests/ViewerQueryTabLoadTests.cs | 568 ++ .../ViewerQueryTrendBucketingReviewTests.cs | 3 +- .../ViewerScopedPaintGuardTests.cs | 2 + .../ViewerScreenReaderNamesFollowUpTests.cs | 263 + .../ViewerScreenReaderNamesTests.cs | 205 + .../ViewerScreenReaderRowNamesTests.cs | 311 + .../ViewerServerInventoryEnabledTests.cs | 10 +- .../ViewerServerManagementTests.cs | 135 +- .../ViewerServerSealLiveTests.cs | 181 + .../Darling.Tests/ViewerServerSetSyncTests.cs | 188 +- .../ViewerServerTabCapabilityPinTests.cs | 1 + .../ViewerSettingsFileGuardTests.cs | 1 + .../ViewerShortWindowDataStartTests.cs | 249 + .../ViewerSlicerRangeLabelTests.cs | 1 + Darling/Darling.Tests/ViewerSmtpSealTests.cs | 314 + .../ViewerSnapshotCollectedColumnTests.cs | 85 + .../ViewerSnapshotStripCollectedTests.cs | 169 + .../ViewerSnapshotTimeLiveTests.cs | 65 + .../ViewerStatsRowBlankCellTests.cs | 151 + .../ViewerStoreLoginRawReadLiveTests.cs | 96 + .../ViewerSystemEventsDataStartTests.cs | 237 + .../Darling.Tests/ViewerSystemEventsTests.cs | 38 +- .../ViewerTextLookupWindowBoundLiveTests.cs | 1070 ++++ .../ViewerTimeRangeGapFixTests.cs | 353 ++ .../ViewerTimeRangeWindowTests.cs | 201 + .../ViewerTrendRoutingPortTests.cs | 56 +- .../ViewerTypedRangeZoneTests.cs | 67 +- Darling/Darling.Tests/ViewerW2aTests.cs | 28 +- .../ViewerWebhookSealLiveTests.cs | 222 + .../ViewerWebhookSealingTests.cs | 239 + .../WaitDrillDownPlanFetchGuardTests.cs | 1 + .../WaitsDatabaseFilterLiveTests.cs | 535 ++ .../WebAutoRefreshPolicyTests.cs | 3 +- .../WebBlockingColumnsPageTests.cs | 126 + .../WebBlockingSaveXmlBehaviourTests.cs | 104 + .../WebBlockingSourceBehaviourTests.cs | 110 + .../WebChartAtTimeBehaviourTests.cs | 458 ++ .../WebChartLegendIsolateBehaviourTests.cs | 333 + .../WebChartMenuBehaviourTests.cs | 144 + .../WebChartWidthBehaviourTests.cs | 517 ++ .../WebChartZoomBehaviourTests.cs | 226 + .../WebClickthroughPickListTests.cs | 232 + .../WebClickthroughStaleStateTests.cs | 154 + .../WebCollectionHealthRunAtColumnTests.cs | 138 + .../WebCollectorRunHistoryTests.cs | 134 + .../WebDataStartNoteBlockingDeadlockTests.cs | 122 + .../WebDataStartNoteCappedWindowTests.cs | 115 + .../WebDataStartNoteConfigAndLogLiveTests.cs | 501 ++ .../WebDataStartNoteConfigAndLogTests.cs | 563 ++ .../WebDataStartNoteEventReadsTests.cs | 219 + .../WebDataStartNoteLiveTests.cs | 715 +++ .../Darling.Tests/WebDataStartNoteTests.cs | 1044 ++++ .../WebDatabaseBlankRuleTests.cs | 138 + .../WebDatabaseFilterBehaviourTests.cs | 266 + .../WebDatabaseFilterChipsBehaviourTests.cs | 226 + .../WebDatabaseFilterDispatchLiveTests.cs | 207 + .../WebDatabaseFilterPickerBehaviourTests.cs | 327 + .../WebDatabaseFilterReadsTests.cs | 364 ++ .../Darling.Tests/WebDatabaseFilterTests.cs | 550 ++ .../WebDatabaseSizesNoteColumnTests.cs | 2 +- .../WebDeadlockGraphBehaviourTests.cs | 171 + .../WebDeadlockGraphShapeTests.cs | 297 + .../WebExceptionTextCensusTests.cs | 30 + .../WebFileIoAndPvsWindowPageTests.cs | 67 + .../WebFileIoLatencyBehaviourTests.cs | 103 + .../WebHarness/load-web-modules.mjs | 93 + .../WebHarnessJsTreeCopyTests.cs | 847 +++ .../WebMemoryPressureChartBehaviourTests.cs | 126 + Darling/Darling.Tests/WebModuleLoadTests.cs | 239 + Darling/Darling.Tests/WebPageRangeTests.cs | 229 + .../WebPageRefreshIntervalBehaviourTests.cs | 160 + .../WebPerfmonMultiSelectBehaviourTests.cs | 267 + .../WebPerfmonPerSecondBehaviourTests.cs | 28 +- .../Darling.Tests/WebPgPlanViewerPageTests.cs | 94 + .../WebPgStatementEvictionTilePageTests.cs | 56 + Darling/Darling.Tests/WebPlainTextTests.cs | 281 + .../Darling.Tests/WebPlanViewerPageTests.cs | 210 + .../WebQueryStoreGridGroupsTests.cs | 168 + .../WebQueryStoreHistoryColumnTests.cs | 49 + .../WebRangeKeptHistoryBehaviourTests.cs | 49 +- .../WebReadCancellationPinTests.cs | 5 +- Darling/Darling.Tests/WebReadProbe.cs | 34 + .../Darling.Tests/WebReadReachLiveTests.cs | 176 + Darling/Darling.Tests/WebReadReachTests.cs | 438 ++ Darling/Darling.Tests/WebRenderSettleTests.cs | 20 +- .../WebRouteStatementCensusTests.cs | 735 +++ .../Darling.Tests/WebServerPageRangeTests.cs | 489 ++ .../WebServerTrendsBehaviourTests.cs | 164 + .../Darling.Tests/WebServerTrendsPageTests.cs | 78 + Darling/Darling.Tests/WebTimeRangeTests.cs | 320 + .../WebTopProceduresColumnsTests.cs | 163 + .../WebTopQueriesColumnsTests.cs | 162 + .../WebWaitQueriesLinkBehaviourTests.cs | 205 + .../WebWaitStatsMultiSelectBehaviourTests.cs | 267 + .../WebWaitsActivityTrendsBehaviourTests.cs | 148 + .../WebWaitsActivityTrendsPageTests.cs | 80 + .../WebhookDeliverySnapshotTests.cs | 151 + .../Darling.Tests/WebhookPostTimeoutTests.cs | 1 + .../Darling.Tests/WindowFloorLiveHarness.cs | 136 + .../WindowNoticeAggregateCases.cs | 143 + .../Darling.Tests/WpfExitSwitchPinTests.cs | 50 + Darling/Darling.Tests/WpfExitSwitchTests.cs | 135 + .../XamlStaticResourceHygieneTests.cs | 2 + .../XeSessionMissingWarningTests.cs | 168 + .../admin-server-edit-harness.mjs | 1129 ++++ .../alert-history-advice-harness.mjs | 215 + .../alert-history-dismiss-harness.mjs | 266 + .../alert-history-range-harness.mjs | 212 + .../analysis-findings-harness.mjs | 160 + .../finops-index-analysis-limit-harness.mjs | 173 + .../finops-locking-detail-harness.mjs | 237 + .../finops-locking-heat-harness.mjs | 136 + .../finops-locking-picker-harness.mjs | 250 + .../finops-postgres-target-harness.mjs | 147 + .../finops-storage-growth-window-harness.mjs | 154 + .../finops-window-picker-harness.mjs | 151 + Darling/Darling.Tests/job-history-harness.mjs | 322 + Darling/Darling.Tests/manage-tags-harness.mjs | 437 ++ Darling/Darling.Tests/mute-rules-harness.mjs | 100 + Darling/Darling.Tests/packages.lock.json | 179 +- Darling/Darling.Tests/page-range-harness.mjs | 118 + Darling/Darling.Tests/sidebar-harness.mjs | 286 + Darling/Darling.Tests/status-bar-harness.mjs | 99 + Darling/Darling.Tests/time-range-harness.mjs | 47 + .../time-range-picker-harness.mjs | 289 + .../top-ranking-notice-harness.mjs | 152 + Darling/Darling.Tests/view-scope-harness.mjs | 79 + .../viewer-local-blank-harness.mjs | 18 + .../Darling.Tests/viewer-local-harness.mjs | 184 + .../web-blocking-save-xml-harness.mjs | 176 + .../web-blocking-source-harness.mjs | 160 + .../web-chart-at-time-harness.mjs | 444 ++ .../web-chart-legend-harness.mjs | 322 + .../Darling.Tests/web-chart-menu-harness.mjs | 177 + .../Darling.Tests/web-chart-width-harness.mjs | 630 ++ .../Darling.Tests/web-chart-zoom-harness.mjs | 209 + .../web-clickthrough-pick-harness.mjs | 301 + .../web-clickthrough-stale-harness.mjs | 176 + .../web-collection-caveats-harness.mjs | 136 + .../web-collector-run-history-harness.mjs | 162 + .../web-column-groups-harness.mjs | 124 + .../web-compose-notice-harness.mjs | 127 + .../web-database-filter-chips-harness.mjs | 288 + .../web-database-filter-harness.mjs | 333 + .../web-database-filter-picker-harness.mjs | 442 ++ .../web-deadlock-graph-harness.mjs | 161 + .../web-deadlock-processes-harness.mjs | 235 + .../web-fleet-silence-harness.mjs | 198 + .../web-gated-collectors-harness.mjs | 117 + .../web-grid-adoption-harness.mjs | 280 + .../Darling.Tests/web-grid-filter-harness.mjs | 238 + .../Darling.Tests/web-grid-sort-harness.mjs | 138 + .../Darling.Tests/web-grid-tools-harness.mjs | 156 + .../web-grid-value-filter-harness.mjs | 560 ++ .../web-kept-history-harness.mjs | 572 +- .../web-memory-pressure-harness.mjs | 173 + .../web-page-refresh-harness.mjs | 259 + Darling/Darling.Tests/web-perfmon-harness.mjs | 16 +- .../web-perfmon-multi-harness.mjs | 431 ++ .../web-pg-plan-viewer-harness.mjs | 152 + Darling/Darling.Tests/web-picker-driver.mjs | 78 + .../Darling.Tests/web-plain-text-harness.mjs | 11 + .../Darling.Tests/web-plan-row-harness.mjs | 109 + .../Darling.Tests/web-plan-viewer-harness.mjs | 628 ++ .../Darling.Tests/web-qs-grids-harness.mjs | 141 + .../Darling.Tests/web-qs-history-harness.mjs | 352 ++ .../web-release-walk-harness.mjs | 173 + .../Darling.Tests/web-scheduler-harness.mjs | 32 + .../web-server-trends-harness.mjs | 290 + .../web-wait-queries-link-harness.mjs | 329 + .../web-waits-activity-trends-harness.mjs | 258 + Darling/Darling.Tests/web-waits-harness.mjs | 422 ++ .../AnalysisStatementText.cs | 44 + .../DarlingAnalysisService.cs | 51 +- .../PgAnomalyDetector.cs | 4 + .../PgBlockingPairRowQuery.cs | 103 +- .../PgDrillDownCollector.Blocking.cs | 69 +- .../PgDrillDownCollector.Config.cs | 3 + .../PgDrillDownCollector.Plans.cs | 6 +- .../PgDrillDownCollector.Queries.cs | 181 +- .../PgDrillDownCollector.Storage.cs | 3 + .../PgDrillDownCollector.cs | 2 +- .../PgFactCollector.Activity.cs | 7 +- .../PgFactCollector.Config.cs | 4 + .../PgFactCollector.QueryPerf.cs | 200 +- .../PgFactCollector.Resources.cs | 55 +- .../PgFactCollector.Storage.cs | 123 +- .../PgFactCollector.cs | 4 +- .../PgFindingStore.cs | 7 +- .../PgPileupSnapshotReader.cs | 18 +- .../PgSecondaryReplicaScope.cs | 150 + .../PgTargetDrillDownCollector.Deadlocks.cs | 28 +- .../PgTargetDrillDownCollector.Queries.cs | 18 +- .../PgTargetFactCollector.Plans.cs | 6 +- .../AlertNotebookEndpoint.cs | 12 +- .../CollectionHistoryAudit.cs | 466 ++ .../CollectorFaultDatabase.cs | 5 +- .../CollectorFaultStackLog.cs | 95 + .../Compose/ComposeCompiler.cs | 824 ++- .../Compose/ComposeServerScope.cs | 86 + .../Compose/ComposeSourceRouter.cs | 209 +- .../Compose/ComposeSpec.cs | 61 + .../Compose/ComposeStoreAvailability.cs | 137 +- .../Compose/QueryStoreGroupMembers.cs | 157 + .../Compose/QueryStoreWideReadGuard.cs | 178 + .../CustomAlertEvaluator.cs | 46 +- .../CustomAlertRuleStore.cs | 4 +- .../DailyRunLimiter.cs | 197 + .../DailyRunWatch.cs | 77 + .../DarlingAdminServersReader.cs | 245 + .../DarlingAlertDeliverer.cs | 6 + .../DarlingAlertReadAdapter.cs | 9 +- .../DarlingAlertSettings.cs | 132 +- .../DarlingAlwaysOnXeSessions.cs | 199 + .../DarlingCliCommands.cs | 1343 +++- .../DarlingCliSealKey.cs | 231 + .../DarlingCollectorRunner.cs | 1228 +++- .../DarlingCommandExecutor.cs | 178 +- .../DarlingConfig.cs | 472 +- .../DarlingFileSecurity.cs | 63 +- .../DarlingFindingAlertSender.cs | 9 + .../DarlingFirewallCheck.cs | 2 +- .../DarlingFleetSweepEndpoints.cs | 20 +- .../DarlingHeartbeat.cs | 664 ++ .../DarlingLegacyLongQuerySession.cs | 341 + .../DarlingLogHashKeyFile.cs | 248 +- .../DarlingManagedPostgres.cs | 434 +- .../DarlingManagedRoles.cs | 1206 +++- .../DarlingNetworkConfigEditor.cs | 37 +- .../DarlingOwnedSecrets.cs | 732 +++ .../DarlingPasswordKeyFile.cs | 418 ++ .../DarlingPasswordKeyRuntime.cs | 576 ++ .../DarlingPasswordKeySelfCheck.cs | 455 ++ .../DarlingPasswordKeyState.cs | 230 + .../DarlingPasswordKeyStore.cs | 496 ++ .../DarlingRemovedServerSessions.cs | 267 + .../DarlingRetention.cs | 513 +- .../DarlingSecretSource.cs | 15 + .../DarlingSecrets.cs | 273 +- .../DarlingSelfAlertEvaluator.cs | 1089 +++- .../DarlingServerConnector.cs | 32 +- .../DarlingServerDatabasesEndpoint.cs | 185 + .../DarlingServiceKeyFile.cs | 522 ++ .../DarlingStoreHostProfile.cs | 159 +- .../DarlingTriageEndpoint.cs | 26 +- .../DarlingUnknownKeys.cs | 117 + .../DarlingWebDeadlockGraph.cs | 214 + .../DarlingWebEndpoints.cs | 3237 +++++++++- .../DarlingWebStatementSweep.cs | 71 + .../DarlingWebhookSecrets.cs | 211 + .../DarlingWorker.cs | 4029 ++++++++++-- .../DarlingXeSessionCleanup.cs | 826 ++- .../DarlingXeSessions.cs | 1318 ++-- .../DatabaseWatermarkCache.cs | 4 +- .../DeadlockDuplicateCleanup.cs | 13 +- .../DiagnosticsBundle.cs | 1243 ++++ .../DiagnosticsBundleAliaser.cs | 1059 ++++ .../DiagnosticsBundleRunner.cs | 634 ++ .../DiagnosticsBundleServiceLog.cs | 245 + .../FleetGateStats.cs | 311 +- .../Hosting/CidrAllowList.cs | 305 + .../Hosting/DarlingHostBinding.cs | 53 +- .../Hosting/DarlingHttpRefusalLog.cs | 17 +- .../Hosting/DarlingListenerTls.cs | 674 ++ .../Hosting/DarlingWebFailureLog.cs | 120 +- .../Hosting/DarlingWebFailureObserver.cs | 120 + .../Hosting/DarlingWebTls.cs | 83 +- .../LaunchMemoryGuard.cs | 429 ++ .../ManagedConfFile.cs | 48 +- .../ManagedConfMigration.cs | 62 +- .../Mcp/DarlingAgReader.cs | 171 +- .../Mcp/DarlingAlertReader.cs | 42 +- .../Mcp/DarlingBlockingReader.cs | 272 +- .../Mcp/DarlingBlockingTrendReader.cs | 42 +- .../Mcp/DarlingCollectionCaveatReader.cs | 118 + .../Mcp/DarlingCollectorRunTimeReader.cs | 250 + .../Mcp/DarlingConfigHistoryReader.cs | 16 +- .../Mcp/DarlingCurrentConfigReader.cs | 23 + .../Mcp/DarlingDataReader.cs | 2347 +++++-- .../Mcp/DarlingDeadlockProcessRows.cs | 91 + .../Mcp/DarlingDefaultTraceReader.cs | 19 +- .../Mcp/DarlingEngineCapability.cs | 2 +- .../Mcp/DarlingFleetReader.cs | 245 +- .../Mcp/DarlingGatedCollectorRows.cs | 164 + .../Mcp/DarlingJobReader.cs | 240 +- .../Mcp/DarlingLatchSpinlockReader.cs | 95 +- .../Mcp/DarlingLongQueryReader.cs | 11 +- .../Mcp/DarlingMcpAgTools.cs | 4 +- .../Mcp/DarlingMcpAlertTools.cs | 91 +- .../Mcp/DarlingMcpBlockingTools.cs | 334 +- .../Mcp/DarlingMcpConfigHistoryTools.cs | 184 +- .../Mcp/DarlingMcpConfigTools.cs | 28 +- .../Mcp/DarlingMcpCustomViewTools.cs | 23 +- .../Mcp/DarlingMcpDataTools.cs | 803 ++- .../Mcp/DarlingMcpDefaultTraceTools.cs | 69 +- .../Mcp/DarlingMcpFinOpsInventoryTools.cs | 166 + .../DarlingMcpFinOpsRecommendationsTools.cs | 124 + ...ngMcpFinOpsTools.ApplicationConnections.cs | 97 + ...DarlingMcpFinOpsTools.DatabaseResources.cs | 117 + .../DarlingMcpFinOpsTools.DatabaseSizes.cs | 115 + .../Mcp/DarlingMcpFinOpsTools.HighImpact.cs | 74 + .../DarlingMcpFinOpsTools.IndexAnalysis.cs | 294 + .../Mcp/DarlingMcpFinOpsTools.Optimization.cs | 238 + .../DarlingMcpFinOpsTools.StorageGrowth.cs | 342 + .../Mcp/DarlingMcpFinOpsTools.Utilization.cs | 122 + .../Mcp/DarlingMcpFinOpsTools.cs | 167 + .../Mcp/DarlingMcpHealthParserTools.cs | 255 +- .../Mcp/DarlingMcpHostService.cs | 723 ++- .../Mcp/DarlingMcpInstructions.cs | 8 +- .../Mcp/DarlingMcpJobTools.cs | 176 +- .../Mcp/DarlingMcpLatchSpinlockTools.cs | 35 +- .../Mcp/DarlingMcpLongQueryTools.cs | 90 +- .../Mcp/DarlingMcpMemoryGrantTools.cs | 36 +- .../Mcp/DarlingMcpObjectStatsTools.cs | 346 +- .../Mcp/DarlingMcpPgBlockingTools.cs | 38 +- .../Mcp/DarlingMcpPgCpuUtilizationTools.cs | 42 +- .../Mcp/DarlingMcpPgDatabaseTools.cs | 48 +- .../Mcp/DarlingMcpPgDeadlockTools.cs | 26 +- .../Mcp/DarlingMcpPgIndexTools.cs | 5 + .../Mcp/DarlingMcpPgIndexUsageTools.cs | 64 +- .../Mcp/DarlingMcpPgIoTools.cs | 26 +- .../Mcp/DarlingMcpPgKernelStatsTools.cs | 40 +- .../Mcp/DarlingMcpPgLogEventTools.cs | 26 +- .../Mcp/DarlingMcpPgPlanTools.cs | 27 +- .../Mcp/DarlingMcpPgPredicateTools.cs | 36 +- .../Mcp/DarlingMcpPgReplicationStatsTools.cs | 45 +- .../Mcp/DarlingMcpPgServerStateTools.cs | 114 +- .../Mcp/DarlingMcpPgSessionStatesTools.cs | 50 +- .../Mcp/DarlingMcpPgStatementTools.cs | 26 +- .../Mcp/DarlingMcpPgTrendTools.cs | 10 +- .../Mcp/DarlingMcpPgWaitSamplingTools.cs | 60 +- .../Mcp/DarlingMcpPgWaitTools.cs | 27 +- .../Mcp/DarlingMcpPgWraparoundTools.cs | 9 +- .../Mcp/DarlingMcpPgXminTools.cs | 35 +- .../Mcp/DarlingMcpPlanCorrectionTools.cs | 71 +- .../Mcp/DarlingMcpPlanTools.cs | 389 +- .../Mcp/DarlingMcpPvsTools.cs | 78 +- .../Mcp/DarlingMcpQueryHeatmapTools.cs | 70 +- .../Mcp/DarlingMcpQueryStoreClutterTools.cs | 72 +- .../Mcp/DarlingMcpQueryStoreHistoryTools.cs | 359 ++ .../DarlingMcpQueryStoreRegressionTools.cs | 113 +- .../Mcp/DarlingMcpReadLatencyTools.cs | 104 +- .../Mcp/DarlingMcpServerAdminTools.Edit.cs | 1304 ++++ .../Mcp/DarlingMcpServerAdminTools.cs | 558 +- .../Mcp/DarlingMcpServerTagTools.cs | 662 ++ .../Mcp/DarlingMcpServerTrendTools.cs | 484 ++ .../Mcp/DarlingMcpSessionTools.cs | 201 +- .../Mcp/DarlingMcpSlowReadTools.cs | 256 + .../Mcp/DarlingMcpStallProbeTools.cs | 6 +- .../Mcp/DarlingMcpStoreLogTools.cs | 2 +- .../Mcp/DarlingMcpStoreMetricsTools.cs | 2 +- .../Mcp/DarlingMcpStoreQueryHistoryTools.cs | 597 ++ .../Mcp/DarlingMcpStoreQueryStatsTools.cs | 27 +- .../Mcp/DarlingMcpTools.cs | 62 +- .../Mcp/DarlingMcpTrendTools.cs | 193 +- .../Mcp/DarlingMcpWindowNotice.cs | 306 + .../Mcp/DarlingObjectStatsReader.cs | 191 +- .../Mcp/DarlingOversizedPlanBacklogReader.cs | 4 +- .../Mcp/DarlingPlanCorrectionReader.cs | 22 +- .../Mcp/DarlingPvsReader.cs | 44 +- .../Mcp/DarlingQueryHeatmapReader.cs | 43 +- .../Mcp/DarlingQueryStoreRegressionReader.cs | 23 +- .../Mcp/DarlingReadLatencyReader.cs | 14 +- .../Mcp/DarlingReproScript.cs | 139 + .../Mcp/DarlingRuntimePrecondition.cs | 52 +- .../Mcp/DarlingServerResolver.cs | 11 +- .../Mcp/DarlingSessionReader.cs | 104 +- .../Mcp/DarlingSlowReadReader.cs | 160 + .../Mcp/DarlingStoredPlanReader.cs | 488 +- .../Mcp/DarlingSystemHealthReader.cs | 53 +- .../Mcp/DarlingTrendReader.cs | 280 +- .../Mcp/DarlingWebHostService.cs | 697 ++- .../Mcp/DarlingWebSecurityHeaders.cs | 154 + .../Mcp/GcfCallToolFilter.cs | 17 +- .../Mcp/GcfOutput.cs | 4 +- .../Mcp/IGcfOutputFormat.cs | 19 + .../Mcp/ListenerTlsCertificateState.cs | 139 + .../Mcp/McpCommandDeadlines.cs | 2 +- .../Mcp/McpTlsCertificateState.cs | 23 + .../Mcp/McpToolLatencyFilter.cs | 49 +- .../Mcp/TopRanking.cs | 232 + .../Mcp/WebTlsCertificateState.cs | 47 +- .../OversizedPlanBacklogSweep.cs | 153 +- .../PerformanceMonitor.Darling.Service.csproj | 1 + .../PgAlertHistoryStore.cs | 8 +- .../PgDeadlockRemask.cs | 548 +- .../PgStatementTextScrub.cs | 27 +- .../PlanDigestCache.cs | 209 + .../PostgresOpenTimeout.cs | 114 + .../ProcedureStatsPlanKey.cs | 165 + .../ProcedureStatsPlanReuse.cs | 449 ++ .../Program.cs | 72 +- .../QueryStatsPlanKey.cs | 101 + .../QueryStoreBackfill.cs | 276 +- .../QueryStoreIntervalFloorWarmUp.cs | 259 + .../QueryStoreWriteFence.cs | 79 +- .../ReadLatencyAccumulator.cs | 27 +- .../ReadLatencyRecorder.cs | 13 +- .../ReadScope.cs | 287 + .../ReadStatementCapture.cs | 129 + .../RetentionCollectionYield.cs | 500 ++ .../RetentionWalPacer.cs | 7 + .../SecretTextGuard.cs | 144 + .../SelfAlertDeliveryStampStore.cs | 4 + .../ServiceCommandDeadlines.cs | 29 +- .../SlowReadLog.cs | 521 ++ .../StallWaitProbeRunner.cs | 8 +- .../StoreConfigProvider.cs | 618 +- .../TargetSessionIdCache.cs | 213 + .../Targets/AwsAssumedRoleCredentials.cs | 441 ++ .../Targets/AwsRoleAllowlist.cs | 190 + .../Targets/AwsRoleAssumeException.cs | 280 + .../Targets/AwsRoleClients.cs | 65 + .../Targets/AwsRoleCredentialCache.cs | 129 + .../Targets/AwsRoleKey.cs | 43 + .../Targets/PostgresTargetProvider.cs | 5 +- .../Targets/RdsCpuIngestor.cs | 42 +- .../Targets/RdsCsvlogCarry.cs | 29 +- .../Targets/RdsDeadlockCarry.cs | 29 +- .../Targets/RdsDeadlockIngestor.cs | 25 +- .../Targets/RdsEndpointVerifier.cs | 375 ++ .../Targets/RdsLogEventIngestor.cs | 25 +- .../Targets/RdsLogSource.cs | 129 +- .../Targets/RdsPlanIngestor.cs | 31 +- .../Targets/SqlServerTargetProvider.cs | 2 +- .../WebDataStartNote.cs | 906 +++ .../WebReadReach.cs | 244 + .../XeSessionMissingWarnings.cs | 41 + .../darling.sample.json | 90 +- .../wwwroot/css/app.css | 254 +- .../wwwroot/css/theme.css | 35 + .../wwwroot/css/time-range.css | 70 + .../wwwroot/index.html | 11 +- .../wwwroot/js/alert-editor.js | 62 +- .../wwwroot/js/app.js | 280 +- .../wwwroot/js/charts.js | 1790 ++++-- .../wwwroot/js/compose.js | 150 +- .../wwwroot/js/database-filter-reads.js | 80 + .../wwwroot/js/editor.js | 32 +- .../wwwroot/js/fleet-groups.js | 123 + .../wwwroot/js/grid-tools.js | 92 + .../wwwroot/js/grid-value-filter.js | 235 + .../wwwroot/js/multi-picker.js | 217 + .../wwwroot/js/mute-context.js | 87 + .../wwwroot/js/page-range.js | 175 + .../wwwroot/js/pages/admin.js | 1204 ++++ .../wwwroot/js/pages/ag.js | 33 +- .../wwwroot/js/pages/alerts.js | 453 +- .../wwwroot/js/pages/analysis-findings.js | 210 + .../wwwroot/js/pages/database-filter.js | 270 + .../wwwroot/js/pages/deadlock-graph.js | 336 + .../wwwroot/js/pages/finops.js | 236 + .../pages/finops/application-connections.js | 72 + .../wwwroot/js/pages/finops/database-box.js | 174 + .../js/pages/finops/database-resources.js | 71 + .../wwwroot/js/pages/finops/database-sizes.js | 92 + .../wwwroot/js/pages/finops/gate.js | 43 + .../wwwroot/js/pages/finops/high-impact.js | 86 + .../wwwroot/js/pages/finops/index-analysis.js | 191 + .../wwwroot/js/pages/finops/locking.js | 212 + .../wwwroot/js/pages/finops/optimization.js | 160 + .../js/pages/finops/recommendations.js | 66 + .../js/pages/finops/server-inventory.js | 117 + .../wwwroot/js/pages/finops/storage-growth.js | 274 + .../wwwroot/js/pages/finops/utilization.js | 245 + .../wwwroot/js/pages/finops/version-store.js | 107 + .../wwwroot/js/pages/finops/window.js | 55 + .../wwwroot/js/pages/fleet.js | 277 +- .../wwwroot/js/pages/job-history.js | 400 ++ .../wwwroot/js/pages/manage-tags.js | 555 ++ .../wwwroot/js/pages/mute-rules.js | 377 ++ .../wwwroot/js/pages/pg-plan-viewer.js | 152 + .../wwwroot/js/pages/plan-viewer.js | 654 ++ .../wwwroot/js/pages/query-store-history.js | 255 + .../wwwroot/js/pages/server-tab-reach.js | 151 + .../wwwroot/js/pages/server-tabs.js | 2189 +++++-- .../wwwroot/js/pages/server.js | 348 +- .../wwwroot/js/pages/sweeps.js | 211 +- .../wwwroot/js/pages/views.js | 239 +- .../wwwroot/js/panels.js | 1071 +++- .../wwwroot/js/plain-text.js | 198 + .../wwwroot/js/plan-row.js | 139 + .../wwwroot/js/read-fields.js | 23 +- .../wwwroot/js/refresh-control.js | 36 +- .../wwwroot/js/refresh-policy.js | 37 +- .../wwwroot/js/server-order.js | 56 + .../wwwroot/js/sidebar.js | 147 + .../wwwroot/js/time-range-picker.js | 363 ++ .../wwwroot/js/time-range.js | 920 +++ .../wwwroot/js/util.js | 572 +- .../wwwroot/js/viewer-local-ui.js | 106 + .../wwwroot/js/viewer-local.js | 411 ++ .../AlertDismissStore.cs | 159 + .../AwsRoleSettings.cs | 170 + .../BoundedDdlBusyStreaks.cs | 90 + .../CollectedDatabases.cs | 78 + .../DarlingJobHistoryReader.cs | 414 ++ .../DarlingModuleMap.cs | 338 +- .../DarlingPgIoReader.cs | 72 +- .../DarlingPgPlanCaptureReader.cs | 82 +- .../DarlingPgTrendReader.cs | 41 +- .../DarlingQueryStoreClutterReader.cs | 29 +- .../DarlingServerClocksReader.cs | 84 + .../DataWindowFloor.cs | 521 ++ .../DatabaseFilter.cs | 156 + .../DatabaseNameHistoryReader.cs | 181 + .../DurationTrendRouting.cs | 457 +- ...rlingFinOpsApplicationConnectionsReader.cs | 104 + .../DarlingFinOpsDatabaseResourcesReader.cs | 382 ++ .../DarlingFinOpsDatabaseSizesReader.cs | 94 + .../FinOps/DarlingFinOpsHighImpactReader.cs | 136 + .../DarlingFinOpsIndexAnalysisReader.cs | 461 ++ .../FinOps/DarlingFinOpsInventoryReader.cs | 585 ++ .../FinOps/DarlingFinOpsOptimizationReader.cs | 399 ++ .../DarlingFinOpsRecommendationsReader.cs | 630 ++ .../DarlingFinOpsStorageGrowthReader.cs | 632 ++ .../FinOps/DarlingFinOpsUtilizationReader.cs | 364 ++ .../FinOps/FinOpsCost.cs | 31 + .../FinOps/FinOpsInventoryFigures.cs | 52 + .../FinOps/FinOpsRecommendationFigures.cs | 582 ++ .../FinOps/FinOpsRollupRouting.cs | 31 + .../FinOps/FinOpsScoring.cs | 141 + .../FinOps/FinOpsUtilizationFigures.cs | 190 + .../FinOps/IndexCleanupRollupFigures.cs | 23 + .../HourlyWindowEdges.cs | 25 +- .../IntervalRollupCountGuard.cs | 77 + .../IntervalRollupRestartRows.cs | 42 + .../LegacyPinCandidateTables.cs | 116 + .../OversizedPlanBacklog.cs | 128 +- .../PasswordBinding.cs | 194 + .../PasswordKeyTables.cs | 259 + .../PasswordSeal.cs | 507 ++ .../PayloadDimensionBatch.cs | 44 +- .../PayloadDimensionWriter.cs | 60 +- .../PayloadDimensions.cs | 51 + .../PerfmonRegroupDrain.cs | 400 ++ .../PerformanceMonitor.Darling.Storage.csproj | 2 + .../PgCollectorRowWriter.cs | 137 +- .../PgIoStatsHourly.cs | 301 + .../PgIoStatsHourlyBuilder.cs | 322 + .../PgMigrations.cs | 894 ++- .../PgStatementText.cs | 30 + .../PgTableTuning.cs | 341 +- .../PlanRegressionDaily.cs | 453 ++ .../QueryStatsHourLedger.cs | 163 + .../QueryStatsHourLedgerWriter.cs | 122 + .../QueryStoreBackgroundIndexes.cs | 563 ++ .../QueryStoreComposeStamp.cs | 942 +++ .../QueryStoreIntervalBrin.cs | 550 ++ .../QueryStoreIntervalLatest.cs | 131 +- .../QueryStoreIntervalPartitions.cs | 1683 +++++ .../QueryStoreIntervalWide.cs | 243 +- .../QueryStoreIntervalWideBrinIndex.cs | 210 +- .../QueryStorePlanWriter.cs | 31 +- .../QueryStoreTextWriter.cs | 63 +- .../QueryStoreTopDaily.cs | 394 ++ .../RawWindowFloor.cs | 139 +- .../RollupCoverageWarmup.cs | 77 + .../ServerConnectionIdentity.cs | 53 + .../ServerConnectionRule.cs | 40 + .../ServerScopeSql.cs | 29 + .../ServerTagCoverage.cs | 159 + .../ServerTagRules.cs | 335 + .../ServerTagStore.cs | 558 ++ .../ServerTrendSql.cs | 576 ++ .../StorageVersion.cs | 2 +- .../StoreInstallId.cs | 449 ++ .../StoreLogClassifier.cs | 212 +- .../StoreLogSweep.cs | 25 +- .../StoreSelfMetrics.cs | 93 +- .../StoreStatementHistory.cs | 461 ++ .../StoreStatementStats.cs | 15 +- .../TimescaleSupport.MaterializationHoles.cs | 47 +- .../TimescaleSupport.cs | 2222 ++++++- .../TopFill.cs | 124 + .../ViewerEventDataStart.cs | 120 + .../AddMultipleServersDialog.xaml.cs | 108 +- .../AddServerDialog.xaml | 17 + .../AddServerDialog.xaml.cs | 269 +- .../AlertsHistoryTab.xaml | 14 +- .../AlertsHistoryTab.xaml.cs | 47 +- .../App.xaml.cs | 11 + .../CollectionLogWindow.xaml | 5 + .../CollectionLogWindow.xaml.cs | 38 +- .../CollectorDurationSeries.cs | 66 + .../CollectorScheduleEditorWindow.xaml | 23 +- .../CollectorScheduleEditorWindow.xaml.cs | 227 +- .../CollectorScheduleOverlay.cs | 304 +- .../CollectorSchedulePresets.cs | 9 + .../CollectorScheduleRunAtText.cs | 164 + .../CorrelatedTimelineLanesControl.xaml | 17 +- .../CorrelatedTimelineLanesControl.xaml.cs | 33 +- .../DataGridHeaderFit.cs | 70 + .../FinOpsServerChoice.cs | 101 + .../FinOpsTab.IndexAnalysis.cs | 7 +- .../FinOpsTab.Loaders.cs | 148 +- .../FinOpsTab.Locking.cs | 6 +- .../FinOpsTab.ObjectHeatmap.cs | 23 +- .../FinOpsTab.Recommendations.cs | 5 +- .../FinOpsTab.xaml | 80 +- .../FinOpsTab.xaml.cs | 94 +- .../FleetRow.cs | 10 + .../JobHistoryTab.xaml | 26 +- .../JobHistoryTab.xaml.cs | 103 +- .../MainWindow.ServerManagement.cs | 18 +- .../MainWindow.Tags.cs | 26 +- .../MainWindow.xaml | 41 +- .../MainWindow.xaml.cs | 160 +- .../ManageTagsWindow.xaml.cs | 21 +- .../MuteRuleEditDialog.xaml.cs | 6 +- .../NotificationRouteEditDialog.xaml.cs | 34 +- .../NotificationRoutesWindow.xaml.cs | 35 +- .../OverviewCardView.cs | 124 + .../PasswordKeyChangedDialog.xaml | 40 + .../PasswordKeyChangedDialog.xaml.cs | 42 + .../ProcedureHistoryWindow.xaml.cs | 16 +- .../QueryStatsHistoryWindow.xaml.cs | 3 + .../QueryStoreHistoryWindow.xaml.cs | 9 +- .../RecommendationsViewModel.cs | 21 +- .../SettingsWindow.xaml | 30 +- .../SettingsWindow.xaml.cs | 269 +- .../Themes/CoolBreezeTheme.xaml | 33 +- .../Themes/DarkTheme.xaml | 33 +- .../Themes/LightTheme.xaml | 33 +- .../TimeRangeSlicerControl.xaml.cs | 49 +- .../ViewerActualPlanFlow.cs | 2 + .../ViewerBlockingChartsDataStart.cs | 25 + .../ViewerBlockingLaneDataStart.cs | 64 + .../ViewerControlPlaneMigration.cs | 139 +- .../ViewerDataService.AlertHistory.cs | 117 +- .../ViewerDataService.AvailabilityGroups.cs | 2 +- .../ViewerDataService.Blocking.cs | 208 +- .../ViewerDataService.BlockingStats.cs | 24 +- .../ViewerDataService.BlockingTrends.cs | 36 +- .../ViewerDataService.CollectionHealth.cs | 725 ++- .../ViewerDataService.CollectorSchedules.cs | 365 +- .../ViewerDataService.Commands.cs | 2 +- .../ViewerDataService.Config.cs | 54 +- .../ViewerDataService.ConfigChanges.cs | 28 + .../ViewerDataService.CpuScheduler.cs | 59 +- .../ViewerDataService.Deadlock.cs | 15 + .../ViewerDataService.ExcludedDatabases.cs | 20 +- .../ViewerDataService.FileIo.cs | 87 +- .../ViewerDataService.FinOps.IndexAnalysis.cs | 435 +- .../ViewerDataService.FinOps.Inventory.cs | 431 +- .../ViewerDataService.FinOps.Locking.cs | 34 +- .../ViewerDataService.FinOps.Pvs.cs | 14 +- ...iewerDataService.FinOps.Recommendations.cs | 778 +-- .../ViewerDataService.FinOps.Storage.cs | 714 +-- .../ViewerDataService.FinOps.Utilization.cs | 302 +- .../ViewerDataService.FinOps.Workload.cs | 687 +- .../ViewerDataService.FinOps.cs | 425 +- .../ViewerDataService.Findings.cs | 9 + .../ViewerDataService.Fleet.cs | 73 +- .../ViewerDataService.ItemHistory.cs | 65 +- .../ViewerDataService.ItemTimeline.cs | 4 +- .../ViewerDataService.JobHistory.cs | 408 +- .../ViewerDataService.LatchSpinlock.cs | 36 +- .../ViewerDataService.LongQueries.cs | 18 + .../ViewerDataService.Memory.cs | 32 +- .../ViewerDataService.MonitoredServers.cs | 427 +- .../ViewerDataService.Notification.cs | 34 +- .../ViewerDataService.NotificationRoutes.cs | 63 +- .../ViewerDataService.Overview.cs | 68 +- .../ViewerDataService.OverviewLanes.cs | 40 +- .../ViewerDataService.PasswordKey.cs | 33 + .../ViewerDataService.PlanCache.cs | 32 +- .../ViewerDataService.PlanCorrection.cs | 17 + .../ViewerDataService.Plans.cs | 51 +- .../ViewerDataService.Postgres.cs | 23 +- .../ViewerDataService.ProcedureStats.cs | 284 +- .../ViewerDataService.QueryHeatmap.cs | 15 + .../ViewerDataService.QuerySnapshots.cs | 14 + .../ViewerDataService.QueryStats.cs | 492 +- .../ViewerDataService.QueryStore.cs | 210 +- .../ViewerDataService.QueryStoreClutter.cs | 49 +- ...ViewerDataService.QueryStoreRegressions.cs | 24 +- .../ViewerDataService.QueryTrends.cs | 54 +- .../ViewerDataService.RollupAvailability.cs | 76 + .../ViewerDataService.RunningJobs.cs | 104 +- .../ViewerDataService.ServerStatus.cs | 146 +- .../ViewerDataService.ServerTags.cs | 268 +- .../ViewerDataService.SessionStats.cs | 83 +- .../ViewerDataService.SystemEvents.cs | 125 +- .../ViewerDataService.cs | 238 +- .../ViewerDataStartNote.cs | 65 + .../ViewerHistoryRows.cs | 49 + .../ViewerLoadTimer.cs | 117 + .../ViewerPasswordKey.cs | 450 ++ .../ViewerPasswordKeyPins.cs | 154 + .../ViewerPostgresDisplay.cs | 173 +- .../ViewerPreferences.cs | 16 +- .../ViewerProcedureHistoryIdleRuns.cs | 88 + .../ViewerServerMigration.cs | 92 +- .../ViewerServerSecret.cs | 76 - .../ViewerServerSetSync.cs | 29 +- .../ViewerServerTab.ActiveQueries.cs | 46 +- .../ViewerServerTab.Blocking.cs | 119 +- .../ViewerServerTab.CollectionHealth.cs | 200 +- .../ViewerServerTab.ConfigChanges.cs | 25 +- .../ViewerServerTab.CopyExport.cs | 4 +- .../ViewerServerTab.CpuScheduler.cs | 2 +- .../ViewerServerTab.DatabaseFilter.cs | 13 +- .../ViewerServerTab.EngineGaps.cs | 22 +- .../ViewerServerTab.EventDataStart.cs | 60 + .../ViewerServerTab.FileIo.cs | 82 +- .../ViewerServerTab.Filters.cs | 9 +- .../ViewerServerTab.HypotheticalIndex.cs | 4 +- .../ViewerServerTab.LatchSpinlock.cs | 12 + .../ViewerServerTab.LongQueries.cs | 80 +- .../ViewerServerTab.Memory.cs | 55 +- .../ViewerServerTab.PlanCache.cs | 3 + .../ViewerServerTab.Postgres.cs | 183 +- .../ViewerServerTab.PostgresDataStart.cs | 40 + .../ViewerServerTab.ProbeWatch.cs | 52 + .../ViewerServerTab.Queries.cs | 360 +- .../ViewerServerTab.QueriesBarCells.cs | 2 +- .../ViewerServerTab.QueriesComparison.cs | 73 +- .../ViewerServerTab.QueryHeatmap.cs | 22 +- .../ViewerServerTab.QueryStoreClutter.cs | 34 +- .../ViewerServerTab.RunningJobs.cs | 44 +- .../ViewerServerTab.SessionStats.cs | 2 + .../ViewerServerTab.SlicerOverlay.cs | 5 +- .../ViewerServerTab.SystemEvents.cs | 57 +- .../ViewerServerTab.TimeRange.cs | 534 +- .../ViewerServerTab.xaml | 607 +- .../ViewerServerTab.xaml.cs | 44 +- .../ViewerSmtpSeal.cs | 158 + .../ViewerTimeRangeWindow.cs | 179 + .../ViewerWebhookSealing.cs | 196 + .../packages.lock.json | 83 +- Darling/README.md | 320 +- Darling/compose/darling.sample.json | 1 + Darling/compose/docker-compose.yml | 31 +- Darling/compose/secrets/README.md | 2 + Darling/tools/fetch-pg-runtime.ps1 | 2 + Darling/tools/install-darling.ps1 | 382 +- Darling/tools/provision-roles.sql | 478 +- Darling/tools/upgrade-darling.ps1 | 323 +- Directory.Build.props | 2 +- Directory.Packages.props | 13 +- Lite.Tests/AgAlertClockStepTests.cs | 5 +- Lite.Tests/AgAlertEvaluatorTests.cs | 65 +- Lite.Tests/AgAlertFailedSendRetryTests.cs | 27 +- Lite.Tests/AgCollectorDefinitionTests.cs | 4 +- Lite.Tests/AgedDatabaseMigrationTests.cs | 8 +- Lite.Tests/AgentStatusArchivedHistoryTests.cs | 4 +- .../AlertDeliveryModeThreadingGuardTests.cs | 1 + Lite.Tests/AlertHistoryCountFilterTests.cs | 51 + Lite.Tests/AlertHistoryDisplayNameTests.cs | 4 +- Lite.Tests/AlertHistorySourceTests.cs | 4 +- Lite.Tests/AlertIncidentRenderTests.cs | 65 + Lite.Tests/AlertSettingsControlWiringTests.cs | 1 + Lite.Tests/AlertSeverityTests.cs | 4 + Lite.Tests/AlertStatementCutOrderLiteTests.cs | 87 + .../AlwaysOnXeReadOnlyIntentLiteTests.cs | 531 ++ Lite.Tests/AlwaysOnXeSessionsLiteTests.cs | 497 ++ Lite.Tests/AnalysisDataSpanTests.cs | 26 +- Lite.Tests/AnalysisNotificationTests.cs | 27 + Lite.Tests/AnalysisPassTokenThreadingTests.cs | 2 +- .../AppLoggerDuckDbOutOfMemoryHintTests.cs | 140 + .../ArchivableTableBareReadSweepTests.cs | 5 + Lite.Tests/ArchiveCompactionLowDiskTests.cs | 498 ++ Lite.Tests/ArchiveCompactionSwapLockTests.cs | 10 +- Lite.Tests/ArchiveCompactionSwapTests.cs | 98 +- Lite.Tests/ArchiveCopyOptionsPinTests.cs | 273 + Lite.Tests/ArchiveFailedExportTempTests.cs | 11 +- Lite.Tests/ArchiveInterruptedRunTests.cs | 18 +- Lite.Tests/ArchiveNoOpPassViewRebuildTests.cs | 114 + ...chiveQuerySnapshotsDailyCompactionTests.cs | 495 ++ .../ArchiveQueryStatsDailyCompactionTests.cs | 350 ++ Lite.Tests/ArchiveReadLimitTests.cs | 971 +++ Lite.Tests/ArchiveResetExportTests.cs | 11 +- Lite.Tests/ArchiveResetRestoreByNameTests.cs | 3 +- Lite.Tests/ArchiveResetRestoresStateTests.cs | 8 +- Lite.Tests/ArchiveResetRetryClockStepTests.cs | 14 +- Lite.Tests/ArchiveTieOrderTests.cs | 144 + Lite.Tests/ArchiveViewDedupTests.cs | 153 +- Lite.Tests/ArchiveViewPartFileTests.cs | 12 +- .../ArchiveViewRefreshAfterDeleteTests.cs | 16 +- Lite.Tests/ArchiveWatermarkCacheTests.cs | 6 +- Lite.Tests/ArchiveWatermarkGreaterOfTests.cs | 23 +- Lite.Tests/ArchivedHistoryReadersTests.cs | 10 +- Lite.Tests/AuroraOnlySqlIsGatedTests.cs | 2 + Lite.Tests/AvailabilityGroupsGridSortTests.cs | 1 + .../AvailabilityGroupsTabRefreshTests.cs | 23 +- Lite.Tests/AzureDatabaseSizeSiblingTests.cs | 142 +- Lite.Tests/AzureMasterFallbackTests.cs | 14 +- Lite.Tests/AzureSiblingGridNoteTests.cs | 2 +- Lite.Tests/AzureSiblingGrowthTests.cs | 106 +- Lite.Tests/AzureSiblingSizePayloadTests.cs | 19 +- Lite.Tests/AzureSqlDatabaseMasterPinsTests.cs | 1 + Lite.Tests/AzureSqlDatabaseOwnFiguresTests.cs | 9 +- .../BaselineDiscontinuityRenderCensusTests.cs | 12 +- Lite.Tests/BlockingChartsDataStartTests.cs | 552 ++ Lite.Tests/BlockingDatabaseFilterToolTests.cs | 369 ++ .../BlockingDeadlockContextBuilderTests.cs | 1 + Lite.Tests/BlockingSourceLabelTests.cs | 118 + Lite.Tests/BlockingTrendEmptyToolTests.cs | 6 + .../servertab-eventwires.baseline.txt | 2 - Lite.Tests/CiSelectMapSelectGuardTests.cs | 491 ++ Lite.Tests/CiSelectReplayGuardTests.cs | 141 + Lite.Tests/ClickThroughUiTextTests.cs | 173 + Lite.Tests/CollectionCaveatsTests.cs | 4 +- .../CollectionHealthDisplayZoneTests.cs | 58 +- .../CollectionHealthEffectiveCadenceTests.cs | 151 + .../CollectionHealthKeyedLookupParityTests.cs | 514 ++ Lite.Tests/CollectionHealthOracleSql.cs | 289 + .../CollectionHealthReadBehaviourTests.cs | 784 +++ .../CollectionHealthRunTimeToolTests.cs | 493 ++ .../CollectionHealthTabScheduleTests.cs | 264 + .../CollectionLogDrillDataStartTests.cs | 403 ++ Lite.Tests/CollectionLogToolTests.cs | 40 +- Lite.Tests/CollectionOutputBesideCostTests.cs | 1 + .../CollectorDurationDetailTextTests.cs | 66 + Lite.Tests/CollectorDurationHoverTests.cs | 225 + Lite.Tests/CollectorGateSurfacePinTests.cs | 1 + Lite.Tests/CollectorHealthClassifierTests.cs | 18 +- Lite.Tests/CollectorRunTimeScheduleTests.cs | 1232 ++++ Lite.Tests/CollectorRunTimeSweepTests.cs | 327 + Lite.Tests/CollectorRunTimeWarningLogTests.cs | 69 + .../CollectorSelfQueryMarkerCensusTests.cs | 441 ++ Lite.Tests/ComparisonWindowUtcTests.cs | 12 +- Lite.Tests/CpuHoverStoredWallClockTests.cs | 22 +- ...uSchedulerStatsCollectorDefinitionTests.cs | 2 +- Lite.Tests/CrossAppGuardCiGateTests.cs | 544 +- .../CrossAppMcpToolInventoryPinTests.cs | 67 +- Lite.Tests/CrossAppPresetValuePinTests.cs | 2 + Lite.Tests/CrossAppStatementFilterPinTests.cs | 145 + Lite.Tests/CrossSkuSurfaceSourceTests.cs | 14 +- Lite.Tests/CurrentWaitsTrendToolTests.cs | 16 + Lite.Tests/CustomTabHeaderInkTests.cs | 25 +- Lite.Tests/DailySummaryCpuBarPinTests.cs | 1 + Lite.Tests/DailySummaryRangeToolTests.cs | 10 +- Lite.Tests/DarlingReadsTraitGuardTests.cs | 441 ++ Lite.Tests/DataGridRowMarkTests.cs | 1 + .../DataStartBannerBlockingEventsTests.cs | 315 + .../DataStartBannerLatchSpinlockTests.cs | 176 + .../DataStartBannerProbeFailureTests.cs | 173 + Lite.Tests/DataStartBannerQueriesTabTests.cs | 1329 ++++ Lite.Tests/DataStartBannerSurfaceTests.cs | 1391 +++++ Lite.Tests/DataStartBannerTests.cs | 396 ++ .../DatabaseSizeCollectorDefinitionTests.cs | 13 +- Lite.Tests/DatabaseStateNoVerdictTests.cs | 3 +- .../DatabaseStateSweepAfterArchiveTests.cs | 3 +- ...ndBlockedProcessEnsureLogLevelLiteTests.cs | 688 ++ Lite.Tests/DeadlockDuplicateCleanupTests.cs | 20 +- .../DeadlockStoredIdentityReadLiteTests.cs | 27 +- .../DeadlocksAzureLoopStateLiteTests.cs | 8 + .../DeadlocksTelemetryCursorLiteTests.cs | 1 + ...aultTraceEventsCollectorDefinitionTests.cs | 2 +- Lite.Tests/DeltaFamilySeedingCensusTests.cs | 2 + .../DeltaFamilyUnknowableRowReadTests.cs | 75 +- Lite.Tests/DesktopDatabaseFilterReadsTests.cs | 438 ++ Lite.Tests/DetachedCollectorGateTests.cs | 2 + Lite.Tests/DismissReliabilityTests.cs | 12 +- Lite.Tests/DismissedArchiveSidecarTests.cs | 10 +- Lite.Tests/DisplayZoneRendererTests.cs | 23 +- Lite.Tests/DisplayZoneTextFrameTests.cs | 62 +- .../DrillDownDopProvenanceParityTests.cs | 1 + .../DuckDbInitializerDisposalGuardTests.cs | 325 + Lite.Tests/DuckDbMemoryLimitSettingTests.cs | 369 ++ Lite.Tests/DuckDbSchemaTests.cs | 10 +- Lite.Tests/DuckDbSentinelConnectionTests.cs | 57 +- Lite.Tests/DuckDbSentinelResetRaceTests.cs | 16 +- .../DurationTrendDatabaseFilterToolTests.cs | 374 ++ .../DurationTrendFilteredDenominatorTests.cs | 204 + Lite.Tests/EmptyDatabaseListNoteLiteTests.cs | 331 + Lite.Tests/EmptyEnumerationInventoryTests.cs | 1 + Lite.Tests/EmptyEnumerationNoteTests.cs | 49 +- Lite.Tests/EngineGapNoteTests.cs | 4 +- Lite.Tests/EntraDeviceCodeTests.cs | 452 +- .../EventListDatabaseFilterToolTests.cs | 405 ++ Lite.Tests/FileGrowthAlertTests.cs | 1 + Lite.Tests/FileIoLatencyTopFilesTests.cs | 272 + .../FinOpsFirstVisitAndBaselineTests.cs | 437 ++ Lite.Tests/FinOpsFleetReadParityTests.cs | 146 +- ...inOpsIdleCoverageAndInventoryScoreTests.cs | 199 + Lite.Tests/FinOpsSecondaryReplicaTests.cs | 162 + Lite.Tests/FinOpsServerInventoryTests.cs | 5 +- Lite.Tests/FinOpsSizeGridReloadPinTests.cs | 1 + Lite.Tests/FinOpsTests.cs | 195 +- Lite.Tests/FinOpsVerdictSourcePinTests.cs | 1 + Lite.Tests/FindingStoreTests.cs | 2 +- .../FindingsRetentionHorizonPinTests.cs | 1 + Lite.Tests/ForcePlanTargetStateTests.cs | 1 + .../GenericWebhookHeadersOriginTests.cs | 313 + Lite.Tests/GenericWebhookTests.cs | 1 + Lite.Tests/GridPayloadColumnOrderPinTests.cs | 1 + Lite.Tests/GridTimeColumnSortMemberTests.cs | 12 + Lite.Tests/GuardStageCensusTests.cs | 315 + .../Helpers/CollectorDefinitionTestFakes.cs | 9 +- Lite.Tests/Helpers/TestAlertDataHelper.cs | 2 +- Lite.Tests/HostHeaderGuardTests.cs | 1 + Lite.Tests/HyperscaleLogSizeTests.cs | 17 +- Lite.Tests/IncidentCooldownTests.cs | 161 + .../IndexLockingGridQualifiedNameTests.cs | 1 + Lite.Tests/IndexObjectStatsTests.cs | 16 +- Lite.Tests/InstallIdStoreTests.cs | 663 ++ Lite.Tests/InstallIdTests.cs | 91 + Lite.Tests/JobHistoryDataStartTests.cs | 637 ++ .../JobHistoryServerClockWindowTests.cs | 294 + .../LatchStatsCollectorDefinitionTests.cs | 2 +- Lite.Tests/LatestValueLookbackTests.cs | 8 +- Lite.Tests/Lite.Tests.csproj | 34 +- ...iteAlertForwardingTests.StatementFilter.cs | 57 + Lite.Tests/LiteAlertForwardingTests.cs | 4 +- Lite.Tests/LiteAnomalousJobsFreshnessTests.cs | 15 +- ...CpuSessionsPlanCacheTrendBucketingTests.cs | 20 + .../LiteOverviewCardExplainsItselfTests.cs | 1 + .../LiteRunningJobsCurrentSnapshotTests.cs | 241 + Lite.Tests/LiteTimeRangeGapTests.cs | 394 ++ Lite.Tests/LlmsTxtCollectorCensusTests.cs | 1 + Lite.Tests/LocalDatabaseReopenTests.cs | 35 +- Lite.Tests/LogTailOverlapThresholdPinTests.cs | 1 + .../LongQueryBannerSessionScopeTests.cs | 83 + ...ueryCompletionsCollectorDefinitionTests.cs | 225 +- .../LongQueryTraceLifecycleLiteTests.cs | 2556 ++++++++ ...LongQueryTracePermissionDenialLiteTests.cs | 176 + .../LongQueryTraceReadOnlyIntentLiteTests.cs | 569 ++ Lite.Tests/MainConnectionLimitsTests.cs | 173 + Lite.Tests/MainWindowAccessKeyTests.cs | 25 +- Lite.Tests/McpAlertSettingsKeyTests.cs | 1 + Lite.Tests/McpAnalysisFindingsCommandTests.cs | 51 + Lite.Tests/McpArgumentTypeTests.cs | 99 + Lite.Tests/McpDatabaseNameFilterToolTests.cs | 306 + Lite.Tests/McpDatabaseSelectionTests.cs | 30 + Lite.Tests/McpDescriptionTruthPinTests.cs | 2 + Lite.Tests/McpHelpersTruncateTests.cs | 92 + Lite.Tests/McpJsonPostGuardTests.cs | 136 + Lite.Tests/McpMissMessageParityPinTests.cs | 48 +- Lite.Tests/McpPageContractTests.cs | 120 +- Lite.Tests/McpReadToolBudgetTests.cs | 11 + Lite.Tests/McpSettingsGuardTests.cs | 1 + .../McpStorageDatabaseNameFilterToolTests.cs | 395 ++ Lite.Tests/McpToolGuideHeads.SqlCore.cs | 14 +- .../McpToolsListBudget/McpBlockingTools.txt | 5 +- .../McpConfigHistoryTools.txt | 1 + .../McpDefaultTraceTools.txt | 1 + .../McpHealthParserTools.txt | 1 + .../McpToolsListBudget/McpHealthTools.txt | 3 +- Lite.Tests/McpToolsListBudget/McpIoTools.txt | 1 + .../McpToolsListBudget/McpLongQueryTools.txt | 1 + .../McpObjectStatsTools.txt | 2 + .../McpPlanCorrectionTools.txt | 1 + Lite.Tests/McpToolsListBudget/McpPvsTools.txt | 1 + .../McpToolsListBudget/McpQueryTools.txt | 3 + .../McpToolsListBudget/McpServerInfoTools.txt | 1 + .../McpToolsListBudget/McpSessionTools.txt | 1 + .../McpToolsListBudget/McpWaitTools.txt | 3 +- Lite.Tests/McpToolsListBudgetTests.cs | 21 +- Lite.Tests/McpUnknownArgumentGuardTests.cs | 81 +- Lite.Tests/McpWholeNumberArgumentTests.cs | 258 + .../McpWindowNoticeAggregateToolTests.cs | 667 ++ .../McpWindowNoticeConfigAndLogToolTests.cs | 668 ++ Lite.Tests/McpWindowNoticeEventToolTests.cs | 433 ++ .../McpWindowNoticeSystemHealthToolTests.cs | 645 ++ Lite.Tests/McpWindowNoticeToolTests.cs | 1090 ++++ Lite.Tests/MeasurementContractCensusTests.cs | 8 +- Lite.Tests/MuteRulesSurviveResetTests.cs | 6 +- Lite.Tests/NewerSchemaFileGuardTests.cs | 7 +- Lite.Tests/ObjectLockingBudgetTests.cs | 155 + Lite.Tests/OnLoadRecaptureScheduleTests.cs | 3 +- ...eratorRemediationLiteDivergencePinTests.cs | 2 + Lite.Tests/OptimizedLockingNoteTests.cs | 25 + .../OverviewBlockingLaneDataStartLiveTests.cs | 354 ++ .../OverviewBlockingLaneDataStartPinTests.cs | 108 + .../OverviewBlockingLaneDataStartTests.cs | 119 + Lite.Tests/OverviewLanesUtcFrameTests.cs | 4 +- .../OverviewNewestReadsHotFirstTests.cs | 4 +- Lite.Tests/PagerDutyWebhookTests.cs | 721 +++ .../ParameterSensitivityClockFrameTests.cs | 2 +- ...erSensitivityFiringSignatureParityTests.cs | 1 + Lite.Tests/PartialDatabaseFailureNoteTests.cs | 1 + Lite.Tests/PerDatabaseProbeFailureTests.cs | 7 +- Lite.Tests/PerEventNotificationTests.cs | 17 + Lite.Tests/PerfmonCounterTypeTests.cs | 17 +- Lite.Tests/PerformanceCalendarDataTests.cs | 12 +- .../PerformanceTrendDurationBucketingTests.cs | 176 +- Lite.Tests/PgServerLogPathPinTests.cs | 1 + .../PlanCacheStatsCollectorDefinitionTests.cs | 2 +- .../PlanRegressionDrillDownReuseTests.cs | 2 +- Lite.Tests/PlanViewer4632FallbackTests.cs | 22 +- Lite.Tests/PlanViewerCapabilityPinTests.cs | 1 + .../PostCollectionCheckpointBudgetTests.cs | 129 + Lite.Tests/PreinitializedDuckDb.cs | 122 + Lite.Tests/PreinitializedDuckDbTests.cs | 114 + .../ProcedureStatsCollectorDefinitionTests.cs | 89 +- .../ProcedureStatsIdleRowsReaderTests.cs | 376 ++ Lite.Tests/ProcedureTrendQueryPinTests.cs | 78 + Lite.Tests/QsCaptureModeRungTests.cs | 4 +- Lite.Tests/QueriesTabWindowRoundTripTests.cs | 10 +- .../QueryHighDopStaleMaxDopParityTests.cs | 1 + Lite.Tests/QueryHighDopStaleMaxDopTests.cs | 2 +- Lite.Tests/QuerySnapshotPlanOnDemandTests.cs | 16 +- Lite.Tests/QueryStatsTextAfterRankingTests.cs | 881 +++ .../QueryStoreBackfillDroppedDatabaseTests.cs | 544 ++ ...eryStoreBackfillReadFailureWarningTests.cs | 28 + .../QueryStoreCollectorDefinitionTests.cs | 2 +- Lite.Tests/QueryStoreDedupReadTests.cs | 105 +- .../QueryStoreHistoryTotalExecutionsTests.cs | 286 + Lite.Tests/QueryStoreIntervalEndRungTests.cs | 12 +- Lite.Tests/QueryStoreOldReadSql.cs | 430 ++ Lite.Tests/QueryStoreRegressionsToolTests.cs | 6 +- .../QueryStoreReplicaSplitAnalysisTests.cs | 2 +- Lite.Tests/QueryStoreServerGateTests.cs | 2 + Lite.Tests/QueryStoreSliceRepairTests.cs | 47 +- .../QueryStoreSliceTieBreakSourceTests.cs | 1 + Lite.Tests/QueryStoreTextAfterRankingTests.cs | 559 ++ Lite.Tests/QueryWindowTruncationTests.cs | 311 +- Lite.Tests/RefreshCoordinatorTests.cs | 525 ++ Lite.Tests/RegressedFromProductiveTests.cs | 6 +- .../ReleaseWalkFilterAndCompareTests.cs | 216 + Lite.Tests/ReplayedIndexRebuildTests.cs | 22 +- .../SameStatementPileupDetectorTests.cs | 25 +- Lite.Tests/ScreenReaderNamesFollowUpTests.cs | 264 + Lite.Tests/ScreenReaderNamesTests.cs | 234 + Lite.Tests/ScreenReaderRowNamesTests.cs | 312 + Lite.Tests/SecondOccurrenceReadTests.cs | 2 +- .../SecondaryReplicaMcpAnalysisTests.cs | 231 + Lite.Tests/SecondaryReplicaScopeTests.cs | 531 ++ Lite.Tests/ServerClockDstReadTests.cs | 1 + ...erverPropertiesCollectorDefinitionTests.cs | 2 +- Lite.Tests/ServerScheduleSummaryTests.cs | 132 + Lite.Tests/ServerTabCapabilityPinTests.cs | 1 + .../ServerTabEmptyStateAndNamesTests.cs | 397 ++ Lite.Tests/ServerTabHeldRangeTests.cs | 322 +- Lite.Tests/ServerTabHistoryOverlayTests.cs | 74 + Lite.Tests/ServerTabStatusTextTests.cs | 48 + Lite.Tests/ServerTimeHelperClockTests.cs | 6 +- Lite.Tests/ServiceBuildSurfaceTests.cs | 1 + ...ionSummaryStatsCollectorDefinitionTests.cs | 2 +- Lite.Tests/SettingsFileGuardTests.cs | 1 + Lite.Tests/SettingsImportTests.cs | 137 + Lite.Tests/SharedBaselineCacheTests.cs | 2 +- Lite.Tests/SharedBaselineModelPinTests.cs | 1 + Lite.Tests/SharedCollectorDefaultsPinTests.cs | 1 + Lite.Tests/SharedDuckDbFixture.cs | 120 +- Lite.Tests/SignificantWaitsToolTests.cs | 25 +- Lite.Tests/SnapshotCollectedTimeTests.cs | 205 + .../SpinlockStatsCollectorDefinitionTests.cs | 2 +- Lite.Tests/SqlServerPermissionErrorsTests.cs | 3 + Lite.Tests/StaThreadCensusTests.cs | 136 + Lite.Tests/StallWaitProbePolicyTests.cs | 304 +- .../StandalonePlanViewerPlusTabTests.cs | 47 +- Lite.Tests/StatementAnalysisReadCutTests.cs | 156 + .../StatementDrillDownTextFetchTests.cs | 247 + .../StatementFilterDrillDownWarningsTests.cs | 156 + Lite.Tests/StatementFilterLiteMcpTests.cs | 374 ++ .../StatementScrubDeadlockCleanupTests.cs | 100 + Lite.Tests/StatementScrubDerivedStoreTests.cs | 196 + .../StatementScrubLiteCollectionTests.cs | 488 ++ .../StatementScrubLivePlanDisplayTests.cs | 267 + Lite.Tests/StatementScrubLiveReadTests.cs | 250 + .../StatementScrubWholeMarkerIdentityTests.cs | 191 + Lite.Tests/StatusBarSizeReadLockTests.cs | 3 +- Lite.Tests/StoredEventCopiesDeadlockTests.cs | 26 +- Lite.Tests/StoredEventCopiesReaderTests.cs | 15 +- Lite.Tests/StoredEventCopiesSweepTests.cs | 9 + Lite.Tests/StoredEventCopiesTests.cs | 9 +- Lite.Tests/StoredPlanAnalysisFilterTests.cs | 190 + Lite.Tests/SwallowedItemFailureTests.cs | 8 +- Lite.Tests/SystemEventsReaderTests.cs | 221 +- Lite.Tests/TabLoadsOffUiThreadTests.cs | 268 + Lite.Tests/TestCpuAccountant.cs | 246 + Lite.Tests/TestCpuAccountantTests.cs | 186 + Lite.Tests/TestDataSeeder.cs | 74 +- Lite.Tests/TestMapProducerGuardTests.cs | 533 ++ Lite.Tests/TestMapShadowGuardTests.cs | 751 +++ Lite.Tests/ThemeColorOverrideTests.cs | 28 +- Lite.Tests/ThemeParityLiteDarlingTests.cs | 1 + Lite.Tests/ThemeStatusContrastTests.cs | 1 + Lite.Tests/TimeHonestyRungTests.cs | 7 +- Lite.Tests/TimingCollection.cs | 26 + Lite.Tests/TimingTestCensusTests.cs | 99 + Lite.Tests/TopCpuQueriesTextTests.cs | 4 +- Lite.Tests/TopListTotalOrderPinTests.cs | 79 + Lite.Tests/TopListWaitforRefillReaderTests.cs | 189 + Lite.Tests/TopTextTieBreakReaderTests.cs | 105 + .../WaitStatsCollectorDefinitionTests.cs | 4 +- .../WaitingBlockedDatabaseFilterToolTests.cs | 310 + Lite.Tests/WatermarkPolicyTests.cs | 30 +- Lite.Tests/WatermarkReadFailureLogTests.cs | 10 +- Lite.Tests/WebhookCooldownSeedTests.cs | 99 +- Lite.Tests/WindowResumeGuardTests.cs | 1 + Lite.Tests/WindowWorkAreaTests.cs | 1 + Lite.Tests/WriteLockBudgetTests.cs | 2 + Lite.Tests/XeTimeFilterSweepTests.cs | 6 + Lite.Tests/packages.lock.json | 78 +- Lite/Analysis/AnalysisService.cs | 48 +- Lite/Analysis/AnomalyDetector.cs | 4 + Lite/Analysis/BlockingPairRowQuery.cs | 81 +- Lite/Analysis/DrillDownCollector.Blocking.cs | 60 +- Lite/Analysis/DrillDownCollector.Config.cs | 3 + Lite/Analysis/DrillDownCollector.Plans.cs | 6 +- Lite/Analysis/DrillDownCollector.Queries.cs | 184 +- Lite/Analysis/DrillDownCollector.Storage.cs | 3 + Lite/Analysis/DuckDbFactCollector.Activity.cs | 7 +- Lite/Analysis/DuckDbFactCollector.Config.cs | 4 + .../Analysis/DuckDbFactCollector.QueryPerf.cs | 8 +- Lite/Analysis/DuckDbFactCollector.Storage.cs | 5 +- Lite/Analysis/FindingStore.cs | 7 +- Lite/Analysis/PileupSnapshotReader.cs | 19 +- .../LiteRecommendationsViewModel.cs | 14 + Lite/Analysis/SecondaryReplicaScope.cs | 147 + Lite/App.xaml.cs | 81 +- Lite/Controls/AlertsHistoryTab.xaml | 14 +- Lite/Controls/AlertsHistoryTab.xaml.cs | 75 +- Lite/Controls/AvailabilityGroupsTab.xaml.cs | 6 +- .../CorrelatedTimelineLanesControl.xaml | 23 +- .../CorrelatedTimelineLanesControl.xaml.cs | 63 +- Lite/Controls/FinOpsTab.Locking.cs | 10 +- Lite/Controls/FinOpsTab.ObjectHeatmap.cs | 15 +- Lite/Controls/FinOpsTab.xaml | 399 +- Lite/Controls/FinOpsTab.xaml.cs | 196 +- Lite/Controls/JobHistoryTab.xaml | 30 +- Lite/Controls/JobHistoryTab.xaml.cs | 183 +- Lite/Controls/LiteBlockingLaneDataStart.cs | 192 + Lite/Controls/OverviewCardBorder.cs | 34 + Lite/Controls/RecommendationsTab.xaml | 19 +- Lite/Controls/RecommendationsTab.xaml.cs | 27 +- .../ServerTab.BlockingChartsDataStart.cs | 110 + Lite/Controls/ServerTab.BlockingStats.cs | 4 +- Lite/Controls/ServerTab.Charts.cs | 215 +- Lite/Controls/ServerTab.Comparison.cs | 75 +- Lite/Controls/ServerTab.ConfigChanges.cs | 3 + Lite/Controls/ServerTab.CopyExport.cs | 4 +- Lite/Controls/ServerTab.CpuScheduler.cs | 2 +- Lite/Controls/ServerTab.DailySummary.cs | 5 +- Lite/Controls/ServerTab.DatabaseFilter.cs | 14 +- Lite/Controls/ServerTab.DrillDown.cs | 46 +- Lite/Controls/ServerTab.EngineGaps.cs | 7 + Lite/Controls/ServerTab.Filters.cs | 6 + Lite/Controls/ServerTab.Grids.cs | 24 +- Lite/Controls/ServerTab.LatchSpinlock.cs | 5 + Lite/Controls/ServerTab.LongQueries.cs | 41 +- Lite/Controls/ServerTab.PlanCache.cs | 3 + Lite/Controls/ServerTab.Plans.cs | 42 +- Lite/Controls/ServerTab.QueriesDataStart.cs | 95 + Lite/Controls/ServerTab.Refresh.cs | 476 +- Lite/Controls/ServerTab.SessionStats.cs | 2 + Lite/Controls/ServerTab.Slicers.cs | 27 +- Lite/Controls/ServerTab.SystemEvents.cs | 25 + Lite/Controls/ServerTab.TimeRange.cs | 461 +- Lite/Controls/ServerTab.xaml | 324 +- Lite/Controls/ServerTab.xaml.cs | 88 +- Lite/Controls/TimeRangeSlicerControl.xaml.cs | 49 +- Lite/Database/DeadlockDuplicateCleanup.cs | 16 +- Lite/Database/DuckDbInitializer.cs | 186 +- Lite/Database/StoredEventCopies.cs | 39 +- Lite/FleetRow.cs | 10 + Lite/Helpers/LiteTimeRange.cs | 256 + Lite/Helpers/MethodProfiler.cs | 74 +- Lite/MainWindow.AlertEngine.cs | 28 +- Lite/MainWindow.xaml | 10 +- Lite/MainWindow.xaml.cs | 165 +- Lite/Mcp/McpAlertTools.cs | 4 +- Lite/Mcp/McpAnalysisTools.cs | 58 +- Lite/Mcp/McpBlockingTools.cs | 195 +- Lite/Mcp/McpCollectorRunTimes.cs | 162 + Lite/Mcp/McpConfigHistoryTools.cs | 67 +- Lite/Mcp/McpConfigTools.cs | 6 +- Lite/Mcp/McpDatabaseSelection.cs | 42 + Lite/Mcp/McpDefaultTraceTools.cs | 44 +- Lite/Mcp/McpHealthParserTools.cs | 162 +- Lite/Mcp/McpHealthTools.cs | 229 +- Lite/Mcp/McpHostService.cs | 66 +- Lite/Mcp/McpInstructions.cs | 2 + Lite/Mcp/McpIoTools.cs | 45 +- Lite/Mcp/McpJobTools.cs | 9 +- Lite/Mcp/McpLongQueryTools.cs | 62 +- Lite/Mcp/McpMemoryTools.cs | 18 +- Lite/Mcp/McpObjectStatsTools.cs | 86 +- Lite/Mcp/McpPlanCorrectionTools.cs | 51 +- Lite/Mcp/McpPlanTools.cs | 87 +- Lite/Mcp/McpPvsTools.cs | 32 +- Lite/Mcp/McpQueryTools.cs | 331 +- Lite/Mcp/McpServerInfoTools.cs | 35 +- Lite/Mcp/McpSessionTools.cs | 65 +- Lite/Mcp/McpWaitTools.cs | 64 +- Lite/Models/CollectorSchedule.cs | 47 + Lite/Models/ServerConnection.cs | 6 + Lite/Services/AgAlertEvaluator.cs | 58 +- Lite/Services/AppAlertSettings.cs | 1 + Lite/Services/AppLogger.cs | 67 +- Lite/Services/ArchiveReadLimiter.cs | 81 + Lite/Services/ArchiveService.cs | 361 +- Lite/Services/ArchiveWatermarkCache.cs | 122 +- Lite/Services/CollectionBackgroundService.cs | 2 + Lite/Services/DataImportService.cs | 2 +- Lite/Services/DataVolumeSpace.cs | 170 + Lite/Services/DeltaCalculator.cs | 2 +- Lite/Services/DuckDbAlertHistoryStore.cs | 8 +- Lite/Services/DuckDbMemoryLimitSetting.cs | 178 + Lite/Services/EmailAlertService.cs | 7 + Lite/Services/FinOpsServerInventory.cs | 1 + Lite/Services/FinOpsShowReloadPolicy.cs | 45 + Lite/Services/InstallIdStore.cs | 516 ++ Lite/Services/KnownEngineEditions.cs | 27 +- Lite/Services/LiteAlertDeliverer.cs | 6 + Lite/Services/LiteAlertReadAdapter.cs | 4 +- Lite/Services/LivePlanDisplay.cs | 37 + Lite/Services/LocalDataService.AgTopology.cs | 2 +- .../Services/LocalDataService.AlertHistory.cs | 89 +- Lite/Services/LocalDataService.Blocking.cs | 217 +- .../LocalDataService.BlockingStats.cs | 15 +- .../LocalDataService.CollectionHealth.cs | 859 ++- Lite/Services/LocalDataService.Config.cs | 20 + .../LocalDataService.ConfigChanges.cs | 32 + .../Services/LocalDataService.CpuScheduler.cs | 33 + .../LocalDataService.DatabaseFilter.cs | 20 + Lite/Services/LocalDataService.FileIo.cs | 120 +- .../LocalDataService.FinOps.IndexObjects.cs | 85 +- .../LocalDataService.FinOps.Inventory.cs | 24 +- Lite/Services/LocalDataService.FinOps.Pvs.cs | 23 +- ...LocalDataService.FinOps.Recommendations.cs | 218 +- ...ocalDataService.FinOps.ServerProperties.cs | 85 +- .../LocalDataService.FinOps.StorageGrowth.cs | 86 +- .../LocalDataService.FinOps.Workload.cs | 2 +- Lite/Services/LocalDataService.FinOps.cs | 75 +- .../LocalDataService.ForcePlanFailures.cs | 29 +- Lite/Services/LocalDataService.JobHistory.cs | 297 +- .../LocalDataService.LatchSpinlock.cs | 8 + Lite/Services/LocalDataService.LongQueries.cs | 23 +- Lite/Services/LocalDataService.Overview.cs | 30 +- .../Services/LocalDataService.QueryHeatmap.cs | 17 +- Lite/Services/LocalDataService.QueryStats.cs | 531 +- Lite/Services/LocalDataService.QueryStore.cs | 354 +- .../LocalDataService.QueryStoreRegressions.cs | 69 +- .../LocalDataService.QueryStoreText.cs | 373 ++ .../LocalDataService.QueryWindowFloor.cs | 495 +- Lite/Services/LocalDataService.RunningJobs.cs | 149 +- .../LocalDataService.RuntimePrecondition.cs | 2 +- Lite/Services/LocalDataService.ServerInfo.cs | 13 +- .../Services/LocalDataService.SessionStats.cs | 17 +- .../Services/LocalDataService.SystemEvents.cs | 173 +- .../Services/LocalDataService.TrendBuckets.cs | 113 +- Lite/Services/LocalDataService.WaitStats.cs | 9 +- Lite/Services/LocalDataService.cs | 28 +- Lite/Services/ParquetCompaction.cs | 145 +- Lite/Services/ProcedureHistoryIdleRuns.cs | 79 + Lite/Services/QueryStoreSliceRepairService.cs | 17 +- Lite/Services/ReadPhaseTimer.cs | 140 + ...moteCollectorService.AlwaysOnXeSessions.cs | 266 + ...teCollectorService.BlockedProcessReport.cs | 402 +- .../RemoteCollectorService.Deadlocks.cs | 150 +- ...RemoteCollectorService.DefinitionRunner.cs | 64 +- ...CollectorService.LegacyLongQuerySession.cs | 359 ++ ...teCollectorService.LongQueryCompletions.cs | 1242 +++- ...moteCollectorService.QueryStoreBackfill.cs | 231 +- .../RemoteCollectorService.RunTime.cs | 179 + Lite/Services/RemoteCollectorService.cs | 710 ++- Lite/Services/ScheduleManager.cs | 519 +- Lite/Services/ServerManager.cs | 4 +- Lite/Services/ServerTimeHelper.cs | 7 + Lite/Services/SettingsImport.cs | 52 + Lite/Services/SettingsSaveReport.cs | 6 +- Lite/Services/TopFill.cs | 80 + Lite/Themes/CoolBreezeTheme.xaml | 33 +- Lite/Themes/DarkTheme.xaml | 33 +- Lite/Themes/LightTheme.xaml | 33 +- Lite/Windows/CollectionLogWindow.xaml | 5 + Lite/Windows/CollectionLogWindow.xaml.cs | 62 +- .../CollectorScheduleEditorWindow.xaml | 9 +- .../CollectorScheduleEditorWindow.xaml.cs | 55 +- .../DatabaseStateOverridesWindow.xaml.cs | 16 +- Lite/Windows/ManageTagsWindow.xaml.cs | 26 +- Lite/Windows/MuteRuleDialog.xaml.cs | 6 +- Lite/Windows/ProcedureHistoryWindow.xaml.cs | 33 +- Lite/Windows/QueryStatsHistoryWindow.xaml.cs | 25 +- Lite/Windows/QueryStoreHistoryWindow.xaml.cs | 27 +- Lite/Windows/SettingsWindow.xaml | 95 +- Lite/Windows/SettingsWindow.xaml.cs | 130 +- Lite/Windows/WaitDrillDownWindow.xaml.cs | 27 +- Lite/config/collection_schedule.json | 2 +- Lite/config/settings.sample.json | 4 +- Lite/packages.lock.json | 82 +- .../AgentJobStepQuery.cs | 2 +- .../AlertContextBuilders.cs | 29 +- PerformanceMonitor.Alerting/AlertEngine.cs | 12 +- .../AlertReadFailureCounter.cs | 7 +- .../AlertStatementFilter.cs | 690 ++ .../DeadlockAlertRow.cs | 6 +- .../FailedJobsQuery.cs | 2 +- .../PerformanceMonitor.Alerting.csproj | 7 + .../RunningJobsCurrency.cs | 45 + .../AnalysisContext.cs | 27 +- .../BlockingChainReconstructor.cs | 19 + .../FactRemediation.cs | 14 + .../FactReplicaScope.cs | 126 + .../PgTargetAdvice.Growth.cs | 10 +- .../PgTargetFactKeys.cs | 2 +- .../PgTargetScorer.Growth.cs | 65 +- .../PgTargetScorer.Plans.cs | 3 +- .../SameStatementPileupDetector.cs | 26 +- .../WithheldStatementMarker.cs | 26 + .../AgDatabaseReplicaStatesCollector.cs | 2 +- .../AgReplicaStatesCollector.cs | 2 +- .../AgentStatusCollector.cs | 2 +- .../AlwaysOnXeSessions.cs | 847 +++ .../AzureSweepScope.cs | 16 + .../BlockedProcessReportCollector.cs | 34 +- .../CollectorCadence.cs | 33 + .../CollectorContext.cs | 154 +- .../CollectorDefinitionBase.cs | 2 + .../CollectorRunTime.cs | 226 + .../CollectorRuntimePrecondition.cs | 19 +- .../CollectorScheduleDefaults.cs | 78 +- .../CpuSchedulerStatsCollector.cs | 2 +- .../CpuUtilizationCollector.cs | 12 +- .../DatabaseConfigCollector.cs | 2 +- .../DatabaseScopedConfigCollector.cs | 4 +- .../DatabaseSizeStatsCollector.cs | 143 +- .../DatabaseStateCollector.cs | 2 +- .../DeadlocksCollector.cs | 65 +- .../DefaultTraceEventsCollector.cs | 10 +- .../DmvBlockingSnapshotCollector.cs | 6 +- .../EmptyDatabaseListNote.cs | 62 + .../FileIoStatsCollector.cs | 4 +- .../ICollectorDefinition.cs | 10 + .../ICollectorRowWriter.cs | 9 + .../IScheduleFrequencyOverride.cs | 29 + .../IndexObjectStatsCollector.cs | 20 +- PerformanceMonitor.Collectors/InstallId.cs | 60 + .../JobHistoryCollector.cs | 7 +- .../LatchStatsCollector.cs | 2 +- .../LegacyLongQuerySession.cs | 25 + .../LongQueryCompletionsCollector.cs | 193 +- .../LongQueryTraceDatabases.cs | 602 ++ .../LongQueryTraceStep.cs | 64 + .../MemoryClerksCollector.cs | 2 +- .../MemoryGrantsCollector.cs | 2 +- .../MemoryPressureEventsCollector.cs | 4 +- .../MemoryStatsCollector.cs | 4 +- .../PerfmonStatsCollector.cs | 13 +- .../PgLogTextRedactor.cs | 5 + .../PgPlanLogParser.cs | 106 +- .../PgSensitiveStatementFilter.cs | 47 +- .../PlanCacheStatsCollector.cs | 2 +- .../PlanCorrectionCollector.cs | 27 +- .../ProcedureStatsCollector.cs | 546 +- .../PvsStatsCollector.cs | 4 +- .../QuerySnapshotsCollector.cs | 13 +- .../QueryStatsCollector.cs | 187 +- .../QueryStoreBackfillState.cs | 71 + .../QueryStoreCollector.cs | 36 +- .../QueryStoreHealthCollector.cs | 4 +- .../QueryStorePerDatabaseState.cs | 16 +- .../RunningJobsCollector.cs | 2 +- .../ServerConfigCollector.cs | 2 +- PerformanceMonitor.Collectors/ServerEpoch.cs | 45 + .../ServerPropertiesCollector.cs | 12 +- .../SessionStatsCollector.cs | 2 +- .../SessionSummaryStatsCollector.cs | 2 +- .../SpinlockStatsCollector.cs | 2 +- .../SqlServerPermissionErrors.cs | 10 +- .../StallWaitProbe.cs | 318 +- .../SystemHealthEventsCollector.cs | 6 +- .../TraceFlagsCollector.cs | 2 +- .../WaitStatsCollector.cs | 4 +- .../WatermarkPolicy.cs | 30 +- PerformanceMonitor.Common/AgReplicaScope.cs | 155 + PerformanceMonitor.Common/AgTopology.cs | 26 +- .../AlertMetricClassifier.cs | 15 +- .../AzureSiblingDatabaseSize.cs | 37 +- .../BlockingThresholdCoverage.cs | 56 + .../DatabaseNameHistory.cs | 145 + .../DerivedStoreScrub.cs | 45 + PerformanceMonitor.Common/FinOpsHeatmap.cs | 54 + .../IndexCleanupAnalyzer.cs | 56 +- .../IndexCleanupModels.cs | 3 + PerformanceMonitor.Common/JobHistoryCap.cs | 27 + PerformanceMonitor.Common/Mcp/McpHelpers.cs | 248 +- .../Mcp/McpSchemaCompat.cs | 63 +- .../Mcp/McpToolParameterTypes.cs | 338 + .../Mcp/McpUnknownArgumentGuard.cs | 295 +- .../Mcp/SensitiveStatementOutputFilter.cs | 267 + PerformanceMonitor.Common/Mcp/TrendBuckets.cs | 2 +- .../Mcp/TrendPayloads.cs | 8 +- .../Models/ColumnFilterState.cs | 73 +- .../PerfmonCounterTypes.cs | 12 +- .../PerformanceMonitor.Common.csproj | 3 + .../SensitiveStatements.Json.cs | 197 + .../SensitiveStatements.Regex.cs | 749 +++ .../SensitiveStatements.Session.cs | 334 + .../SensitiveStatements.WarmUp.cs | 214 + .../SensitiveStatements.Xml.cs | 947 +++ .../SensitiveStatements.cs | 224 + .../Services/ColumnFilterMatcher.cs | 23 +- .../Services/ColumnValueList.cs | 258 + .../Services/ColumnValueListColumns.cs | 59 + .../Services/JsonContentType.cs | 41 + PerformanceMonitor.Common/TagColours.cs | 79 + .../AgAlertContexts.cs | 8 + .../AlertContext.cs | 26 + .../AlertFamily.cs | 55 + .../AlertSeverity.cs | 4 + .../AnalysisNotificationService.cs | 6 +- .../EmailSendCore.cs | 4 +- .../IAlertSettings.cs | 27 + .../IncidentCooldown.cs | 128 +- .../IncidentDeliveryFilter.cs | 11 +- .../IncidentGrouping.cs | 46 +- PerformanceMonitor.Notifications/MuteRule.cs | 39 + .../NotificationRoute.cs | 13 + .../NotificationRouter.cs | 11 + .../PerEventNotification.cs | 9 +- .../WebhookAlertService.cs | 336 +- .../ActualPlanExecutor.cs | 66 +- .../QueryModificationDetector.cs | 8 + .../ReproScriptBuilder.cs | 5 +- .../ShowPlanParser.cs | 6 +- PerformanceMonitor.Ui/AccessibleNames.cs | 723 +++ PerformanceMonitor.Ui/BlockingSourceLabel.cs | 42 + PerformanceMonitor.Ui/ChartHoverHelper.cs | 130 +- .../CollectorDurationHoverText.cs | 29 + PerformanceMonitor.Ui/ColumnFilterPopup.xaml | 131 +- .../ColumnFilterPopup.xaml.cs | 147 +- .../ColumnFilterPopupController.cs | 16 +- PerformanceMonitor.Ui/ColumnFilterScope.cs | 38 + PerformanceMonitor.Ui/ColumnFilterStore.cs | 512 ++ PerformanceMonitor.Ui/ColumnValueListModel.cs | 192 + .../ComparisonBaselineNote.cs | 31 + .../DataGridFilterManager.cs | 143 +- .../DatabaseFilterSelection.cs | 54 + PerformanceMonitor.Ui/EmptyState.cs | 110 + PerformanceMonitor.Ui/FileIoChartFiles.cs | 40 + PerformanceMonitor.Ui/FileSaveHelper.cs | 16 +- PerformanceMonitor.Ui/IoLatencyWeighting.cs | 43 + PerformanceMonitor.Ui/LongQueriesEmptyText.cs | 26 + .../PlanNavigationController.cs | 4 +- PerformanceMonitor.Ui/PlanNodeBorder.cs | 81 + .../PlanViewerControl.Rendering.cs | 10 +- .../PlanViewerControl.xaml.cs | 27 + PerformanceMonitor.Ui/RefreshCoordinator.cs | 154 + .../ScopedLoadGenerations.cs | 7 +- PerformanceMonitor.Ui/ServerOverviewFilter.cs | 7 + PerformanceMonitor.Ui/TagColorPalette.cs | 71 +- PerformanceMonitor.Ui/TimeColumnTitle.cs | 27 + .../TimeRange/ResolvedTimeRange.cs | 109 + .../TimeRange/RollingUnitRule.cs | 53 + .../TimeRange/TimeRangeParser.cs | 617 ++ .../TimeRange/TimeRangePresets.cs | 176 + .../TimeRange/TimeRangeSpec.cs | 487 ++ PerformanceMonitor.Ui/TimeRangePicker.xaml | 109 + PerformanceMonitor.Ui/TimeRangePicker.xaml.cs | 609 ++ PerformanceMonitor.Ui/WithheldPlanGuard.cs | 56 + README.md | 189 +- deprecated/Dashboard.Tests/packages.lock.json | 74 +- deprecated/Dashboard/packages.lock.json | 54 +- deprecated/Installer.Core/packages.lock.json | 30 +- deprecated/Installer.Tests/packages.lock.json | 34 +- deprecated/Installer/packages.lock.json | 44 +- docs/changelog/3.10.md | 633 ++ docs/how-collection-works.md | 20 +- docs/postgres-first-target-runbook.md | 131 +- docs/uat-onboarding.md | 71 +- llms.txt | 2 +- tools/CompactionRepro/Program.cs | 97 +- tools/changelog/archive-census.txt | 1 + .../drain-ab-replay/Invoke-DrainAbReplay.ps1 | 1027 +++ tools/drain-ab-replay/README.md | 126 + .../proof/Start-FakeRetention.ps1 | 47 + .../example-output/latency_by_script.csv | 26 + .../proof/example-output/phases.csv | 5 + .../proof/example-output/rates.csv | 68 + .../proof/example-output/run.json | 64 + .../proof/example-output/summary.md | 54 + .../drain-ab-replay/proof/seed-proof-rig.sql | 108 + 2650 files changed, 456216 insertions(+), 21317 deletions(-) create mode 100644 .github/ci-history/failures.jsonl create mode 100644 .github/ci-history/replay-accepted.txt create mode 100644 .github/scripts/altcover-probe.py create mode 100644 .github/scripts/ci-select.py create mode 100644 .github/scripts/darling-upgrade-shapes.ps1 create mode 100644 .github/scripts/lite-shard-pack.py create mode 100644 .github/scripts/lite-timing-top.py create mode 100644 .github/scripts/run-darling-pg-shard.ps1 create mode 100644 .github/scripts/run-lite-shard.ps1 create mode 100644 .github/scripts/tag-reads-lite.py create mode 100644 .github/scripts/test-map.py create mode 100644 .github/workflows/darling-upgrade-shapes.yml create mode 100644 Darling/Darling.Tests/ActiveQueriesDatabaseFilterLiveTests.cs create mode 100644 Darling/Darling.Tests/ActiveQueriesWidenedColumnsTests.cs create mode 100644 Darling/Darling.Tests/ActualPlanConsentNoticeTests.cs create mode 100644 Darling/Darling.Tests/ActualPlanSnapshotResolveLiveTests.cs create mode 100644 Darling/Darling.Tests/AdminPageBehaviourTests.cs create mode 100644 Darling/Darling.Tests/AdminPageTests.cs create mode 100644 Darling/Darling.Tests/AdminServerEditBehaviourTests.cs create mode 100644 Darling/Darling.Tests/AdminServersRouteLiveTests.cs create mode 100644 Darling/Darling.Tests/AdminServersRouteTests.cs create mode 100644 Darling/Darling.Tests/AdminServersVersionLabelTests.cs create mode 100644 Darling/Darling.Tests/AgEveryViewLiveTests.cs create mode 100644 Darling/Darling.Tests/AggregateJobQuiesceTests.cs create mode 100644 Darling/Darling.Tests/AlertDelivererInventoryPinTests.cs create mode 100644 Darling/Darling.Tests/AlertHistoryAdviceLiveTests.cs create mode 100644 Darling/Darling.Tests/AlertHistoryAdviceTests.cs create mode 100644 Darling/Darling.Tests/AlertHistoryDismissBehaviourTests.cs create mode 100644 Darling/Darling.Tests/AlertHistoryDismissTests.cs create mode 100644 Darling/Darling.Tests/AlertHistoryRangePageTests.cs create mode 100644 Darling/Darling.Tests/AlertStatementCutOrderTests.cs create mode 100644 Darling/Darling.Tests/AlwaysOnXeReadOnlyIntentTests.cs create mode 100644 Darling/Darling.Tests/AlwaysOnXeSessionsTests.cs create mode 100644 Darling/Darling.Tests/AnalysisFindingsTabTests.cs create mode 100644 Darling/Darling.Tests/AwsAssumeRolePackageTests.cs create mode 100644 Darling/Darling.Tests/AwsAssumedRoleCredentialsTests.cs create mode 100644 Darling/Darling.Tests/AwsExternalIdNeverPrintedTests.cs create mode 100644 Darling/Darling.Tests/AwsPerServerRoleLiveTests.cs create mode 100644 Darling/Darling.Tests/AwsPerServerRoleRungTests.cs create mode 100644 Darling/Darling.Tests/AwsRoleAllowlistTests.cs create mode 100644 Darling/Darling.Tests/AwsRoleBundleEndToEndLiveTests.cs create mode 100644 Darling/Darling.Tests/AwsRoleCredentialCacheTests.cs create mode 100644 Darling/Darling.Tests/AwsRoleEditTests.cs create mode 100644 Darling/Darling.Tests/AwsRoleMessageTextTests.cs create mode 100644 Darling/Darling.Tests/AwsRoleSettingsTests.cs create mode 100644 Darling/Darling.Tests/AwsStsTestSupport.cs create mode 100644 Darling/Darling.Tests/AzureFallbackMessageTests.cs create mode 100644 Darling/Darling.Tests/BlockingDatabaseFilterLiveTests.cs create mode 100644 Darling/Darling.Tests/BlockingListPlanFlagsLivePostgresTests.cs create mode 100644 Darling/Darling.Tests/BlockingThresholdCoverageTests.cs create mode 100644 Darling/Darling.Tests/BlockingWaitsFilteredEmptyWordingTests.cs create mode 100644 Darling/Darling.Tests/BlockingWaitsWebDispatchLiveTests.cs create mode 100644 Darling/Darling.Tests/BoundedCompressionDdlTests.cs create mode 100644 Darling/Darling.Tests/BoundedDdlBusyStreakTests.cs create mode 100644 Darling/Darling.Tests/BoundedDdlSingleCopyTests.cs create mode 100644 Darling/Darling.Tests/BugReportTemplateDiagnosticsBundleTests.cs create mode 100644 Darling/Darling.Tests/CiShardCommandLineBudgetTests.cs create mode 100644 Darling/Darling.Tests/CiShardHangReportTests.cs create mode 100644 Darling/Darling.Tests/CollectionCaveatsWebTests.cs create mode 100644 Darling/Darling.Tests/CollectionGapAtStartTests.cs create mode 100644 Darling/Darling.Tests/CollectionHealthDetachedSweepPressureTests.cs create mode 100644 Darling/Darling.Tests/CollectionHealthEffectiveCadenceTests.cs create mode 100644 Darling/Darling.Tests/CollectionHealthEverySurfaceEffectiveIntervalTests.cs create mode 100644 Darling/Darling.Tests/CollectionHealthKeyedLookupParityTests.cs create mode 100644 Darling/Darling.Tests/CollectionHealthOracleSql.cs create mode 100644 Darling/Darling.Tests/CollectionHealthRunTimeFieldsLiveTests.cs create mode 100644 Darling/Darling.Tests/CollectionHealthScheduleOverridesLiveTests.cs create mode 100644 Darling/Darling.Tests/CollectionHistoryAuditTests.cs create mode 100644 Darling/Darling.Tests/CollectionLogSegmentByLiveTests.cs create mode 100644 Darling/Darling.Tests/CollectorFaultStackLogTests.cs create mode 100644 Darling/Darling.Tests/CollectorRunTimeCliVerbTests.cs create mode 100644 Darling/Darling.Tests/CollectorRunTimeRungTests.cs create mode 100644 Darling/Darling.Tests/CollectorRunTimeTests.cs create mode 100644 Darling/Darling.Tests/CollectorRunTimeViewerLiveTests.cs create mode 100644 Darling/Darling.Tests/CollectorRunTimeViewerTests.cs create mode 100644 Darling/Darling.Tests/CollectorScheduleEditorSaveGuardTests.cs create mode 100644 Darling/Darling.Tests/CollectorScheduleResetStatusTests.cs create mode 100644 Darling/Darling.Tests/CollectorScheduleUseDefaultRunAtLineTests.cs create mode 100644 Darling/Darling.Tests/ColumnFilterPersistenceTests.cs create mode 100644 Darling/Darling.Tests/ColumnFilterPopupTests.cs create mode 100644 Darling/Darling.Tests/ColumnFilterWiringCensusTests.cs create mode 100644 Darling/Darling.Tests/ColumnGroupsBehaviourTests.cs create mode 100644 Darling/Darling.Tests/ColumnValueFilterCasesTests.cs create mode 100644 Darling/Darling.Tests/ColumnValueListNameRuleTests.cs create mode 100644 Darling/Darling.Tests/ColumnValueListTests.cs create mode 100644 Darling/Darling.Tests/ComposeCatalogStatementTextCensusTests.cs create mode 100644 Darling/Darling.Tests/ComposeClientTimeoutAttributionTests.cs create mode 100644 Darling/Darling.Tests/ComposeDataFloorLiveTests.cs create mode 100644 Darling/Darling.Tests/ComposeHourlyEdgesLedgerGuardLiveTests.cs create mode 100644 Darling/Darling.Tests/ComposeHourlyEdgesRunnerTests.cs create mode 100644 Darling/Darling.Tests/ComposeHourlyRawEdgesGuardLiveTests.cs create mode 100644 Darling/Darling.Tests/ComposeHourlyRawEdgesModuleMapLiveTests.cs create mode 100644 Darling/Darling.Tests/ComposeImageTagTests.cs create mode 100644 Darling/Darling.Tests/ComposeOpenPhaseTimeoutTests.cs create mode 100644 Darling/Darling.Tests/ComposeQueryStoreRankedSingleScanLiveTests.cs create mode 100644 Darling/Darling.Tests/ComposeQueryStoreRankedSingleScanTests.cs create mode 100644 Darling/Darling.Tests/ComposeQueryStoreRawDedupeFilterLiveTests.cs create mode 100644 Darling/Darling.Tests/ComposeQueryStoreStampCompileTests.cs create mode 100644 Darling/Darling.Tests/ComposeQueryStoreStampExactnessLiveTests.cs create mode 100644 Darling/Darling.Tests/ComposeQueryStoreStampGuardLiveTests.cs create mode 100644 Darling/Darling.Tests/ComposeQueryStoreStampInvalidationLiveTests.cs create mode 100644 Darling/Darling.Tests/ComposeQueryStoreStampPlanLiveTests.cs create mode 100644 Darling/Darling.Tests/ComposeRollupDataFloorLiveTests.cs create mode 100644 Darling/Darling.Tests/ComposeServerScopeByIdTests.cs create mode 100644 Darling/Darling.Tests/ComposeStampLiveSupport.cs create mode 100644 Darling/Darling.Tests/ComposeTimeoutPhaseLiveTests.cs create mode 100644 Darling/Darling.Tests/ConfigDatabaseFilterLiveTests.cs create mode 100644 Darling/Darling.Tests/ConnectionTestReplyTests.cs create mode 100644 Darling/Darling.Tests/CustomAlertResolutionNameFilterTests.cs create mode 100644 Darling/Darling.Tests/DailyCollectorDetachTests.cs create mode 100644 Darling/Darling.Tests/DailyRunCapAndWatchdogTests.cs create mode 100644 Darling/Darling.Tests/DarlingAllowFromListGateTests.cs create mode 100644 Darling/Darling.Tests/DarlingCidrAllowListTests.cs create mode 100644 Darling/Darling.Tests/DarlingCliSealKeyTests.cs create mode 100644 Darling/Darling.Tests/DarlingCollectorRunTimeReaderTests.cs create mode 100644 Darling/Darling.Tests/DarlingConfigAwsTests.cs create mode 100644 Darling/Darling.Tests/DarlingEncryptPasswordStreamTests.cs create mode 100644 Darling/Darling.Tests/DarlingHeartbeatTests.cs create mode 100644 Darling/Darling.Tests/DarlingHostConfigurationPinTests.cs create mode 100644 Darling/Darling.Tests/DarlingListenerTlsTests.cs create mode 100644 Darling/Darling.Tests/DarlingMcpDeadlockProcessesLiveTests.cs create mode 100644 Darling/Darling.Tests/DarlingMcpDefaultTraceToolsWindowNoticeTests.cs create mode 100644 Darling/Darling.Tests/DarlingMcpFinOpsInventoryToolsTests.cs create mode 100644 Darling/Darling.Tests/DarlingMcpFinOpsRecommendationsToolsTests.cs create mode 100644 Darling/Darling.Tests/DarlingMcpFinOpsToolsTests.cs create mode 100644 Darling/Darling.Tests/DarlingMcpHealthParserToolsWindowNoticeTests.cs create mode 100644 Darling/Darling.Tests/DarlingMcpJobHistoryToolTests.cs create mode 100644 Darling/Darling.Tests/DarlingMcpPgWindowD8aLiveTests.cs create mode 100644 Darling/Darling.Tests/DarlingMcpPgWindowD8bLiveTests.cs create mode 100644 Darling/Darling.Tests/DarlingMcpQueryStoreHistoryToolsTests.cs create mode 100644 Darling/Darling.Tests/DarlingMcpServerTagToolsTests.cs create mode 100644 Darling/Darling.Tests/DarlingMcpServerTrendToolsTests.cs create mode 100644 Darling/Darling.Tests/DarlingMcpTlsLiveTests.cs create mode 100644 Darling/Darling.Tests/DarlingModuleMapGroupedSourceLiveTests.cs create mode 100644 Darling/Darling.Tests/DarlingModuleMapRecentRefreshLiveTests.cs create mode 100644 Darling/Darling.Tests/DarlingOwnedSecretsCensusTests.cs create mode 100644 Darling/Darling.Tests/DarlingPasswordKeyFileHardeningTests.cs create mode 100644 Darling/Darling.Tests/DarlingPasswordKeyFileTests.cs create mode 100644 Darling/Darling.Tests/DarlingPasswordKeyFileUnixTests.cs create mode 100644 Darling/Darling.Tests/DarlingPasswordKeyRuntimeTests.cs create mode 100644 Darling/Darling.Tests/DarlingPasswordKeyStateTests.cs create mode 100644 Darling/Darling.Tests/DarlingPasswordKeyStoreLiveTests.cs create mode 100644 Darling/Darling.Tests/DarlingPgUnresolvedZoneWarningTests.cs create mode 100644 Darling/Darling.Tests/DarlingReproScriptParityTests.cs create mode 100644 Darling/Darling.Tests/DarlingRunTimeSchedulingTests.cs create mode 100644 Darling/Darling.Tests/DarlingSecretsCanUnprotectTests.cs create mode 100644 Darling/Darling.Tests/DarlingSecretsOwnedReferenceTests.cs create mode 100644 Darling/Darling.Tests/DarlingSelfQueryMarkerTests.cs create mode 100644 Darling/Darling.Tests/DarlingUnknownNetworkKeyTests.cs create mode 100644 Darling/Darling.Tests/DarlingUpgradeShapesWorkflowTests.cs create mode 100644 Darling/Darling.Tests/DarlingWebSecurityHeadersTests.cs create mode 100644 Darling/Darling.Tests/DarlingWebhookSecretsTests.cs create mode 100644 Darling/Darling.Tests/DarlingWorkerServerClockLiveTests.cs create mode 100644 Darling/Darling.Tests/DataWindowFloorSnapshotAndRunLogLiveTests.cs create mode 100644 Darling/Darling.Tests/DataWindowFloorTests.cs create mode 100644 Darling/Darling.Tests/DatabaseConfigWebColumnsTests.cs create mode 100644 Darling/Darling.Tests/DatabaseFilterBindingLiveTests.cs create mode 100644 Darling/Darling.Tests/DatabaseFilterEmptyAnswerLiveTests.cs create mode 100644 Darling/Darling.Tests/DatabaseNameHistoryTests.cs create mode 100644 Darling/Darling.Tests/DatabaseNameParamTwinTests.cs create mode 100644 Darling/Darling.Tests/DeadlockProcessRowsTests.cs create mode 100644 Darling/Darling.Tests/DesktopDatabaseFilterLiveTests.cs create mode 100644 Darling/Darling.Tests/DiagnosticsBundleAliasHardeningTests.cs create mode 100644 Darling/Darling.Tests/DiagnosticsBundleAliasTests.cs create mode 100644 Darling/Darling.Tests/DiagnosticsBundleExitCodeTests.cs create mode 100644 Darling/Darling.Tests/DiagnosticsBundleHardeningTests.cs create mode 100644 Darling/Darling.Tests/DiagnosticsBundleHealthUncutLiveTests.cs create mode 100644 Darling/Darling.Tests/DiagnosticsBundleLeakLiveTests.cs create mode 100644 Darling/Darling.Tests/DiagnosticsBundleRound2Tests.cs create mode 100644 Darling/Darling.Tests/DiagnosticsBundleSealedTextTests.cs create mode 100644 Darling/Darling.Tests/DiagnosticsBundleSecretTests.cs create mode 100644 Darling/Darling.Tests/DiagnosticsBundleSeedFailureLiveTests.cs create mode 100644 Darling/Darling.Tests/DiagnosticsBundleServiceLogTests.cs create mode 100644 Darling/Darling.Tests/DiagnosticsBundleSizeTests.cs create mode 100644 Darling/Darling.Tests/DiagnosticsBundleStatementFilterTests.cs create mode 100644 Darling/Darling.Tests/DiagnosticsBundleStoreHistoryLiveTests.cs create mode 100644 Darling/Darling.Tests/DiagnosticsBundleVerbTests.cs create mode 100644 Darling/Darling.Tests/DiagnosticsBundleWebhookOpenTests.cs create mode 100644 Darling/Darling.Tests/DropXeSessionsPerInstallTests.cs create mode 100644 Darling/Darling.Tests/DropXeSessionsReadOnlyIntentTests.cs create mode 100644 Darling/Darling.Tests/DurationTrendDatabaseFilterLiveTests.cs create mode 100644 Darling/Darling.Tests/EffectiveStartUtcTests.cs create mode 100644 Darling/Darling.Tests/EmptyAnswerDatabaseEchoTests.cs create mode 100644 Darling/Darling.Tests/EmptyDatabaseListNoteLiveTests.cs create mode 100644 Darling/Darling.Tests/EmptyDatabaseListNoteTests.cs create mode 100644 Darling/Darling.Tests/EventsDatabaseFilterLiveTests.cs create mode 100644 Darling/Darling.Tests/FileIoDatabaseFilterLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsApplicationConnectionsGoldenLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsApplicationConnectionsViewTests.cs create mode 100644 Darling/Darling.Tests/FinOpsClickThroughRulesTests.cs create mode 100644 Darling/Darling.Tests/FinOpsCostTests.cs create mode 100644 Darling/Darling.Tests/FinOpsDatabaseResourcesGoldenLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsDatabaseResourcesViewTests.cs create mode 100644 Darling/Darling.Tests/FinOpsDatabaseSizesViewTests.cs create mode 100644 Darling/Darling.Tests/FinOpsHighImpactReaderLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsHotspotRegionTests.cs create mode 100644 Darling/Darling.Tests/FinOpsIdleCoverageInventoryLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsIdleCoverageLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsIdleCoverageSeed.cs create mode 100644 Darling/Darling.Tests/FinOpsIndexAnalysisGoldenLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsIndexAnalysisLatestSnapshotLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsIndexAnalysisLatestTiebreakTests.cs create mode 100644 Darling/Darling.Tests/FinOpsIndexAnalysisLimitBehaviourTests.cs create mode 100644 Darling/Darling.Tests/FinOpsIndexAnalysisViewLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsIndexAnalysisViewSizeTests.cs create mode 100644 Darling/Darling.Tests/FinOpsIndexAnalysisViewTests.cs create mode 100644 Darling/Darling.Tests/FinOpsInventoryGoldenLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsInventoryHealthScoreLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsInventoryToolParityLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsInventoryViewerFiguresTests.cs create mode 100644 Darling/Darling.Tests/FinOpsLockingDetailBehaviourTests.cs create mode 100644 Darling/Darling.Tests/FinOpsLockingHeatBehaviourTests.cs create mode 100644 Darling/Darling.Tests/FinOpsLockingPickerBehaviourTests.cs create mode 100644 Darling/Darling.Tests/FinOpsOptimizationGoldenLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsOptimizationTieBreakTests.cs create mode 100644 Darling/Darling.Tests/FinOpsOptimizationViewTests.cs create mode 100644 Darling/Darling.Tests/FinOpsPageShellTests.cs create mode 100644 Darling/Darling.Tests/FinOpsPostgresTargetTests.cs create mode 100644 Darling/Darling.Tests/FinOpsRecommendationFiguresTests.cs create mode 100644 Darling/Darling.Tests/FinOpsRecommendationsCharacterizationTests.cs create mode 100644 Darling/Darling.Tests/FinOpsRecommendationsGoldenLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsRecommendationsReadsLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsRecommendationsSecondaryReplicaLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsRecommendationsToolParityLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsServerChoiceTests.cs create mode 100644 Darling/Darling.Tests/FinOpsStorageGrowthGoldenLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsStorageGrowthViewTests.cs create mode 100644 Darling/Darling.Tests/FinOpsStorageGrowthWindowBehaviourTests.cs create mode 100644 Darling/Darling.Tests/FinOpsTabApplicationConnectionsPageTests.cs create mode 100644 Darling/Darling.Tests/FinOpsTabDatabaseResourcesPageTests.cs create mode 100644 Darling/Darling.Tests/FinOpsTabDatabaseSizesPageTests.cs create mode 100644 Darling/Darling.Tests/FinOpsTabHighImpactPageTests.cs create mode 100644 Darling/Darling.Tests/FinOpsTabIndexAnalysisPageTests.cs create mode 100644 Darling/Darling.Tests/FinOpsTabLockingPageTests.cs create mode 100644 Darling/Darling.Tests/FinOpsTabOptimizationPageTests.cs create mode 100644 Darling/Darling.Tests/FinOpsTabPvsPageTests.cs create mode 100644 Darling/Darling.Tests/FinOpsTabRecommendationsPageTests.cs create mode 100644 Darling/Darling.Tests/FinOpsTabServerInventoryPageTests.cs create mode 100644 Darling/Darling.Tests/FinOpsTabStorageGrowthPageTests.cs create mode 100644 Darling/Darling.Tests/FinOpsTabSystemEventsColumnsPageTests.cs create mode 100644 Darling/Darling.Tests/FinOpsTabUtilizationPageTests.cs create mode 100644 Darling/Darling.Tests/FinOpsTabUtilizationRawPageTests.cs create mode 100644 Darling/Darling.Tests/FinOpsUtilizationFiguresLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsUtilizationFiguresTests.cs create mode 100644 Darling/Darling.Tests/FinOpsUtilizationGoldenLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsUtilizationViewLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsWebReadParityLiveTests.cs create mode 100644 Darling/Darling.Tests/FinOpsWindowPickerBehaviourTests.cs create mode 100644 Darling/Darling.Tests/Fixtures/FinOpsApplicationConnections/golden.json create mode 100644 Darling/Darling.Tests/Fixtures/FinOpsDatabaseResources/golden.json create mode 100644 Darling/Darling.Tests/Fixtures/FinOpsIndexAnalysis/golden.json create mode 100644 Darling/Darling.Tests/Fixtures/FinOpsInventory/golden.json create mode 100644 Darling/Darling.Tests/Fixtures/FinOpsOptimization/golden.json create mode 100644 Darling/Darling.Tests/Fixtures/FinOpsRecommendations/golden.json create mode 100644 Darling/Darling.Tests/Fixtures/FinOpsStorageGrowth/golden.json create mode 100644 Darling/Darling.Tests/Fixtures/FinOpsUtilization/golden.json create mode 100644 Darling/Darling.Tests/Fixtures/PasswordSeal/linux-self-check-vector.json create mode 100644 Darling/Darling.Tests/Fixtures/StatementScrub/autoparam_select_adhoc_shell.xml create mode 100644 Darling/Darling.Tests/Fixtures/StatementScrub/autoparam_select_prepared.xml create mode 100644 Darling/Darling.Tests/Fixtures/StatementScrub/autoparam_update_adhoc_shell.xml create mode 100644 Darling/Darling.Tests/Fixtures/StatementScrub/autoparam_update_prepared.xml create mode 100644 Darling/Darling.Tests/Fixtures/StatementScrub/two_statements.xml create mode 100644 Darling/Darling.Tests/Fixtures/column-value-filter-cases.json create mode 100644 Darling/Darling.Tests/Fixtures/migration-ladder-v3.9.0.sql create mode 100644 Darling/Darling.Tests/Fixtures/time-range-cases.json create mode 100644 Darling/Darling.Tests/FleetGateStartupTests.cs create mode 100644 Darling/Darling.Tests/FleetPgDeadlockFlatPairLiveTests.cs create mode 100644 Darling/Darling.Tests/FleetSilenceBehaviourTests.cs create mode 100644 Darling/Darling.Tests/FleetSilencePageTests.cs create mode 100644 Darling/Darling.Tests/GatedCollectorRowsTests.cs create mode 100644 Darling/Darling.Tests/GenericWebhookHeadersOriginTests.cs create mode 100644 Darling/Darling.Tests/GridAdoptionBehaviourTests.cs create mode 100644 Darling/Darling.Tests/GridFilterBehaviourTests.cs create mode 100644 Darling/Darling.Tests/GridSortBehaviourTests.cs create mode 100644 Darling/Darling.Tests/GridToolsBehaviourTests.cs create mode 100644 Darling/Darling.Tests/GridValueFilterBehaviourTests.cs create mode 100644 Darling/Darling.Tests/GuardStageCensusTests.cs create mode 100644 Darling/Darling.Tests/GuardStageScanner.cs create mode 100644 Darling/Darling.Tests/GuardStageWorkflowTests.cs create mode 100644 Darling/Darling.Tests/InstallIdRungTests.cs create mode 100644 Darling/Darling.Tests/InstallIdTableOidRungTests.cs create mode 100644 Darling/Darling.Tests/IntervalRollupCountGuardLiveTests.cs create mode 100644 Darling/Darling.Tests/IoHourlyRollupTests.cs create mode 100644 Darling/Darling.Tests/JobHistoryCountTextTests.cs create mode 100644 Darling/Darling.Tests/JobHistoryPageTests.cs create mode 100644 Darling/Darling.Tests/JobHistorySqlTextPinTests.cs create mode 100644 Darling/Darling.Tests/LatestCollectorOutcomeOrderTests.cs create mode 100644 Darling/Darling.Tests/LatestCollectorOutcomePlanTests.cs create mode 100644 Darling/Darling.Tests/LaunchMemoryGuardTests.cs create mode 100644 Darling/Darling.Tests/LegacyLongQueryRecordLiveTests.cs create mode 100644 Darling/Darling.Tests/LegacyPinCandidateRungTests.cs create mode 100644 Darling/Darling.Tests/LegacyPinResolverLiveTests.cs create mode 100644 Darling/Darling.Tests/ListenerTlsFailedStartTests.cs create mode 100644 Darling/Darling.Tests/LiteReadsTraitGuardTests.cs create mode 100644 Darling/Darling.Tests/LiveClock.cs create mode 100644 Darling/Darling.Tests/LiveClockCensusTests.cs create mode 100644 Darling/Darling.Tests/LiveStoreSessionTimeZone.cs create mode 100644 Darling/Darling.Tests/LiveStoreSessionTimeZoneTests.cs create mode 100644 Darling/Darling.Tests/LongQueryBannerSessionScopeTests.cs create mode 100644 Darling/Darling.Tests/LongQueryTraceInstanceGuardTests.cs create mode 100644 Darling/Darling.Tests/LongQueryTraceLifecycleTests.cs create mode 100644 Darling/Darling.Tests/LongQueryTraceReadOnlyIntentTests.cs create mode 100644 Darling/Darling.Tests/LongQueryTraceRemovalTests.cs create mode 100644 Darling/Darling.Tests/LongQueryTraceRestartTests.cs create mode 100644 Darling/Darling.Tests/LongQueryTraceRunRowLiveTests.cs create mode 100644 Darling/Darling.Tests/ManageTagsBehaviourTests.cs create mode 100644 Darling/Darling.Tests/ManageTagsPageTests.cs create mode 100644 Darling/Darling.Tests/ManagedConfWalRungStabilityTests.cs create mode 100644 Darling/Darling.Tests/MaterializationHoleProbeLiveTests.cs create mode 100644 Darling/Darling.Tests/McpArgumentTypeRows.cs create mode 100644 Darling/Darling.Tests/McpArgumentTypeTests.cs create mode 100644 Darling/Darling.Tests/McpInProcessHost.cs create mode 100644 Darling/Darling.Tests/McpToolGuideHeads.FinOps.cs create mode 100644 Darling/Darling.Tests/McpToolGuideHeads.FinOpsInventory.cs create mode 100644 Darling/Darling.Tests/McpToolGuideHeads.FinOpsRecommendations.cs create mode 100644 Darling/Darling.Tests/McpToolsListBudget/DarlingMcpFinOpsInventoryTools.txt create mode 100644 Darling/Darling.Tests/McpToolsListBudget/DarlingMcpFinOpsRecommendationsTools.txt create mode 100644 Darling/Darling.Tests/McpToolsListBudget/DarlingMcpFinOpsTools.txt create mode 100644 Darling/Darling.Tests/McpToolsListBudget/DarlingMcpQueryStoreHistoryTools.txt create mode 100644 Darling/Darling.Tests/McpToolsListBudget/DarlingMcpServerTagTools.txt create mode 100644 Darling/Darling.Tests/McpToolsListBudget/DarlingMcpServerTrendTools.txt create mode 100644 Darling/Darling.Tests/McpToolsListBudget/DarlingMcpSlowReadTools.txt create mode 100644 Darling/Darling.Tests/McpToolsListBudget/DarlingMcpStoreQueryHistoryTools.txt create mode 100644 Darling/Darling.Tests/McpWholeNumberArgumentTests.cs create mode 100644 Darling/Darling.Tests/MigrationLockHolderLogTests.cs create mode 100644 Darling/Darling.Tests/MigrationLockUncontendedAcquireTimingTests.cs create mode 100644 Darling/Darling.Tests/MuteRulesBehaviourTests.cs create mode 100644 Darling/Darling.Tests/MuteRulesPageTests.cs create mode 100644 Darling/Darling.Tests/NewestSnapshotReadsLiveTests.cs create mode 100644 Darling/Darling.Tests/NightlyCheckoutCommitPinTests.cs create mode 100644 Darling/Darling.Tests/NoInternalBusFilesGuardTests.cs create mode 100644 Darling/Darling.Tests/ObjectLockingDetailLiveTests.cs create mode 100644 Darling/Darling.Tests/ObjectLockingHeatLiveTests.cs create mode 100644 Darling/Darling.Tests/ObjectStatsDatabaseFilterLiveTests.cs create mode 100644 Darling/Darling.Tests/OversizedPlanBacklogJudgeRetireLivePostgresTests.cs create mode 100644 Darling/Darling.Tests/OwnedSecretReferenceRefusalTests.cs create mode 100644 Darling/Darling.Tests/PagerDutyAutoResolveRungTests.cs create mode 100644 Darling/Darling.Tests/PasswordBindingCensusTests.cs create mode 100644 Darling/Darling.Tests/PasswordKeyGrantParityTests.cs create mode 100644 Darling/Darling.Tests/PasswordKeyRungTests.cs create mode 100644 Darling/Darling.Tests/PasswordKeySeamLiveTests.cs create mode 100644 Darling/Darling.Tests/PasswordKeySelfCheckTests.cs create mode 100644 Darling/Darling.Tests/PasswordKeyTablesLiveTests.cs create mode 100644 Darling/Darling.Tests/PasswordSealTests.cs create mode 100644 Darling/Darling.Tests/PayloadDigestPassThroughLiveTests.cs create mode 100644 Darling/Darling.Tests/PerfmonRegroupGateTests.cs create mode 100644 Darling/Darling.Tests/PerfmonRegroupLiveSupport.cs create mode 100644 Darling/Darling.Tests/PerfmonRegroupLiveTests.cs create mode 100644 Darling/Darling.Tests/PerfmonRegroupTimingLiveTests.cs create mode 100644 Darling/Darling.Tests/PgClusterObjectCensusTests.cs create mode 100644 Darling/Darling.Tests/PgClusterRolesCollection.cs create mode 100644 Darling/Darling.Tests/PgDeadlockRemaskAlertPagingLiveTests.cs create mode 100644 Darling/Darling.Tests/PgDeadlockRemaskFindingsPagingLiveTests.cs create mode 100644 Darling/Darling.Tests/PgEventToolsWindowFloorLiveTests.cs create mode 100644 Darling/Darling.Tests/PgGridsMcpDataStartLiveTests.cs create mode 100644 Darling/Darling.Tests/PgIoStatsHourlyTests.cs create mode 100644 Darling/Darling.Tests/PgPlanShapeHashTests.cs create mode 100644 Darling/Darling.Tests/PgPlanViewerBehaviourTests.cs delete mode 100644 Darling/Darling.Tests/PgSettingRedactorTimingCollection.cs create mode 100644 Darling/Darling.Tests/PgStatementTextFailureRecordingTests.cs create mode 100644 Darling/Darling.Tests/PgStatementTextHealthSemanticsTests.cs create mode 100644 Darling/Darling.Tests/PgWriteStatsAndIndexUsageWindowNoticeTests.cs create mode 100644 Darling/Darling.Tests/PlanDigestCacheTests.cs create mode 100644 Darling/Darling.Tests/PlanRegressionBuiltDaysFallbackLiveTests.cs create mode 100644 Darling/Darling.Tests/PlanRegressionDailyBoundaryTests.cs create mode 100644 Darling/Darling.Tests/PlanRegressionDailyBuilderLiveTests.cs create mode 100644 Darling/Darling.Tests/PlanRegressionDailyParityLiveTests.cs create mode 100644 Darling/Darling.Tests/PlanRegressionDailyReadShapeLiveTests.cs create mode 100644 Darling/Darling.Tests/PlanRegressionDailyRungTests.cs create mode 100644 Darling/Darling.Tests/PlanRegressionDailyTriggerCostRig.cs create mode 100644 Darling/Darling.Tests/PlanRegressionDailyTriggerLiveTests.cs create mode 100644 Darling/Darling.Tests/PlanRegressionDrillDownWindowStartLiveTests.cs create mode 100644 Darling/Darling.Tests/PlanRegressionTimeoutCaveatTests.cs create mode 100644 Darling/Darling.Tests/PlanRowBehaviourTests.cs create mode 100644 Darling/Darling.Tests/PlanViewerBehaviourTests.cs create mode 100644 Darling/Darling.Tests/PostgresOpenTimeoutTests.cs create mode 100644 Darling/Darling.Tests/ProcedureStatsDeferredPlanFetchLiveTests.cs create mode 100644 Darling/Darling.Tests/ProcedureStatsFloorAndHistoryLiveTests.cs create mode 100644 Darling/Darling.Tests/ProcedureStatsIdleHoursLiveTests.cs create mode 100644 Darling/Darling.Tests/ProcedureStatsIdleRowsLiveTests.cs create mode 100644 Darling/Darling.Tests/ProcedureStatsIdleSkipTests.cs create mode 100644 Darling/Darling.Tests/ProcedureStatsPlanFetchTests.cs create mode 100644 Darling/Darling.Tests/ProcedureStatsPlanReuseTests.cs create mode 100644 Darling/Darling.Tests/ProcedureStatsSqlGoldenTests.cs create mode 100644 Darling/Darling.Tests/ProcedureTrendHourlyQueryViewPinTests.cs create mode 100644 Darling/Darling.Tests/ProcedureTrendQueryViewPinTests.cs create mode 100644 Darling/Darling.Tests/ProcedureTrendRunModelLiveTests.cs create mode 100644 Darling/Darling.Tests/ProvisioningSerializationTests.cs create mode 100644 Darling/Darling.Tests/PublicConnectCatalogReadLiveTests.cs create mode 100644 Darling/Darling.Tests/PublicConnectDelta.cs create mode 100644 Darling/Darling.Tests/PublicConnectGuardAttribute.cs create mode 100644 Darling/Darling.Tests/PublicConnectGuardTests.cs create mode 100644 Darling/Darling.Tests/PurgeContinuationTests.cs create mode 100644 Darling/Darling.Tests/PvsSkippedFieldsParityLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryEventsDatabaseFilterLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryReadsWebDispatchLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStatsDeferredPlanFetchLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStatsHourLedgerTests.cs create mode 100644 Darling/Darling.Tests/QueryStatsHourLedgerWriterLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStatsHourLedgerWriterTests.cs create mode 100644 Darling/Darling.Tests/QueryStatsLedgerSeed.cs create mode 100644 Darling/Darling.Tests/QueryStatsPlanFetchTests.cs create mode 100644 Darling/Darling.Tests/QueryStatsSqlGoldenTests.cs create mode 100644 Darling/Darling.Tests/QueryStatsWriterCensusPins.cs create mode 100644 Darling/Darling.Tests/QueryStoreAzureLoopWatermarkLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreBackfillCandidateCacheLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreBackfillDroppedDatabaseTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreBackgroundIndexesLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreBackgroundIndexesTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreComposeStampLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreComposeStampRaceLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreComposeStampTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreComposeStampWriterGuardLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreGroupMembersLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreGroupMembersTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreHistoryBehaviourTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreHistoryLivePostgresTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreHistoryPanelStateTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreHistoryWideTierLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreIntervalBoundedReadPruningLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreIntervalBrinLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreIntervalConvergenceLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreIntervalFloorWarmUpTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreIntervalMaintenancePassLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreIntervalPartitionRungTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreIntervalPartitionedStoreMetricsLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreIntervalPartitionsLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreIntervalPartitionsTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreIntervalPurgeRowCappedTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreIntervalRetentionLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreIntervalRetentionPinTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreIntervalRetentionTestKit.cs create mode 100644 Darling/Darling.Tests/QueryStoreLegacyRowIndexLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStorePlanViaMapLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreRankedHarness.cs create mode 100644 Darling/Darling.Tests/QueryStoreReadsDatabaseFilterLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreTopDailyBuilderLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreTopDailyReadLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreTopDailyRungTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreWideFleetEligibilityLiveTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreWideReadGuardTests.cs create mode 100644 Darling/Darling.Tests/QueryStoreWideUpperBoundLiveTests.cs create mode 100644 Darling/Darling.Tests/QuietServerWatermarkLadderLiveTests.cs create mode 100644 Darling/Darling.Tests/RawWindowFloorQuietStartTests.cs create mode 100644 Darling/Darling.Tests/RdsAssumeRoleSeamTests.cs create mode 100644 Darling/Darling.Tests/RdsEndpointVerifierTests.Login.cs create mode 100644 Darling/Darling.Tests/RdsEndpointVerifierTests.cs create mode 100644 Darling/Darling.Tests/RdsLogStateConcurrencyTests.cs create mode 100644 Darling/Darling.Tests/ReadLatencyBudgetEdgeTests.cs create mode 100644 Darling/Darling.Tests/ReadScopeGateFaultLiveTests.cs create mode 100644 Darling/Darling.Tests/ReadScopeStatementCaptureLiveTests.cs create mode 100644 Darling/Darling.Tests/ReadScopeTests.cs create mode 100644 Darling/Darling.Tests/ReenterLineTests.cs create mode 100644 Darling/Darling.Tests/ReleaseStatementGateWorkflowTests.cs create mode 100644 Darling/Darling.Tests/ReleaseWalkWordingTests.cs create mode 100644 Darling/Darling.Tests/ReproScriptLivePostgresTests.cs create mode 100644 Darling/Darling.Tests/RestartRowIndexLiveTests.cs create mode 100644 Darling/Darling.Tests/RetentionCollectionYieldLiveTests.cs create mode 100644 Darling/Darling.Tests/RetentionCollectionYieldTests.cs create mode 100644 Darling/Darling.Tests/RetiredBaselineSweepRecoveryTests.cs create mode 100644 Darling/Darling.Tests/RollupCoverageWarmupTests.cs create mode 100644 Darling/Darling.Tests/RollupFloorStoreCacheTests.cs create mode 100644 Darling/Darling.Tests/RollupIoFloorFillLiveTests.cs create mode 100644 Darling/Darling.Tests/RowPlanReadLivePostgresTests.cs create mode 100644 Darling/Darling.Tests/ScratchDatabaseDisposalCensusTests.cs create mode 100644 Darling/Darling.Tests/ScratchDatabaseSweep.cs create mode 100644 Darling/Darling.Tests/ScratchDatabaseSweepLiveTests.cs create mode 100644 Darling/Darling.Tests/ScratchDatabaseSweepTests.cs create mode 100644 Darling/Darling.Tests/ScratchPostgresExitDrainTests.cs create mode 100644 Darling/Darling.Tests/ScratchPostgresQuiesceTests.cs create mode 100644 Darling/Darling.Tests/SealedSecretResolverTests.cs create mode 100644 Darling/Darling.Tests/SecondaryReplicaScopeLiveTests.cs create mode 100644 Darling/Darling.Tests/SecondaryReplicaScopeTests.cs create mode 100644 Darling/Darling.Tests/SelfAlertFleetPassTests.cs create mode 100644 Darling/Darling.Tests/SelfManagedStoreRulesStartPinTests.cs create mode 100644 Darling/Darling.Tests/SensitiveStatementAutoParamTests.cs create mode 100644 Darling/Darling.Tests/SensitiveStatementCorpus.cs create mode 100644 Darling/Darling.Tests/SensitiveStatementHostFilterTests.cs create mode 100644 Darling/Darling.Tests/SensitiveStatementJsonTests.cs create mode 100644 Darling/Darling.Tests/SensitiveStatementOutputFilterTests.cs create mode 100644 Darling/Darling.Tests/SensitiveStatementRawCleanTests.cs create mode 100644 Darling/Darling.Tests/SensitiveStatementSessionTests.cs create mode 100644 Darling/Darling.Tests/SensitiveStatementXmlTests.cs create mode 100644 Darling/Darling.Tests/SensitiveStatementsParityLiveTests.cs create mode 100644 Darling/Darling.Tests/SensitiveStatementsTests.cs create mode 100644 Darling/Darling.Tests/ServerAddRouteTests.cs create mode 100644 Darling/Darling.Tests/ServerAddViewerRoleLiveTests.cs create mode 100644 Darling/Darling.Tests/ServerDatabasesEndpointLiveTests.cs create mode 100644 Darling/Darling.Tests/ServerEditCoreTests.cs create mode 100644 Darling/Darling.Tests/ServerEditGateLiveTests.cs create mode 100644 Darling/Darling.Tests/ServerEditLiveTests.cs create mode 100644 Darling/Darling.Tests/ServerEditPasswordRuleViewerTests.cs create mode 100644 Darling/Darling.Tests/ServerEditRouteTests.cs create mode 100644 Darling/Darling.Tests/ServerEditViewerRoleLiveTests.cs create mode 100644 Darling/Darling.Tests/ServerIdentityLockLiveTests.cs create mode 100644 Darling/Darling.Tests/ServerIdentityLockSourcePinTests.cs create mode 100644 Darling/Darling.Tests/ServerPasswordFromRequestTests.cs create mode 100644 Darling/Darling.Tests/ServerRemovalClearsTagsLiveTests.cs create mode 100644 Darling/Darling.Tests/ServerTabColumnGroupsPinTests.cs create mode 100644 Darling/Darling.Tests/ServerTabEmptyStatesAndHeadersTests.cs create mode 100644 Darling/Darling.Tests/ServerTabGridColumnPinTests.cs create mode 100644 Darling/Darling.Tests/ServerTabToolbarAndTooltipsTests.cs create mode 100644 Darling/Darling.Tests/ServerTagCoverageTests.cs create mode 100644 Darling/Darling.Tests/ServerTagDeleteWarningTests.cs create mode 100644 Darling/Darling.Tests/ServerTagMcpRoleLiveTests.cs create mode 100644 Darling/Darling.Tests/ServerTagRulesTests.cs create mode 100644 Darling/Darling.Tests/ServerTagStoreLiveTests.cs create mode 100644 Darling/Darling.Tests/ServerTagViewerRoleLiveTests.cs create mode 100644 Darling/Darling.Tests/ServerTagWebRoutesLiveTests.cs create mode 100644 Darling/Darling.Tests/ShowPlanParserParameterizedTextTests.cs create mode 100644 Darling/Darling.Tests/SidebarSearchGroupTests.cs create mode 100644 Darling/Darling.Tests/SlowClassCensusTests.cs create mode 100644 Darling/Darling.Tests/SlowReadLogTests.cs create mode 100644 Darling/Darling.Tests/SlowReadsLiveTests.cs create mode 100644 Darling/Darling.Tests/SlowReadsRungTests.cs create mode 100644 Darling/Darling.Tests/StaTestThread.cs create mode 100644 Darling/Darling.Tests/StaTestThreadTests.cs create mode 100644 Darling/Darling.Tests/StaThreadCensusTests.cs create mode 100644 Darling/Darling.Tests/StallWaitProbeLiveTests.cs create mode 100644 Darling/Darling.Tests/StallWaitProbeQueryTextPinTests.cs create mode 100644 Darling/Darling.Tests/StateSelfAlertTests.cs create mode 100644 Darling/Darling.Tests/StatementAlertReadCutLiveTests.cs create mode 100644 Darling/Darling.Tests/StatementAnalysisReadCutLiveTests.cs create mode 100644 Darling/Darling.Tests/StatementAnalysisReadCutPinTests.cs create mode 100644 Darling/Darling.Tests/StatementCollectionCensusCases.cs create mode 100644 Darling/Darling.Tests/StatementColumnCensusTests.cs create mode 100644 Darling/Darling.Tests/StatementCutSourceScanTests.cs create mode 100644 Darling/Darling.Tests/StatementDrillDownTextFetchLiveTests.cs create mode 100644 Darling/Darling.Tests/StatementFilterAlertTests.cs create mode 100644 Darling/Darling.Tests/StatementFilterAnalysisReadsLiveTests.cs create mode 100644 Darling/Darling.Tests/StatementFilterBlockingReadsLiveTests.cs create mode 100644 Darling/Darling.Tests/StatementFilterCensus.cs create mode 100644 Darling/Darling.Tests/StatementFilterDeliveryTests.cs create mode 100644 Darling/Darling.Tests/StatementFilterDrillDownWarningsLiveTests.cs create mode 100644 Darling/Darling.Tests/StatementFilterPlanReaderSourceScanTests.cs create mode 100644 Darling/Darling.Tests/StatementFilterPlanReadsLiveTests.cs create mode 100644 Darling/Darling.Tests/StatementFilterQueryReadsLiveTests.cs create mode 100644 Darling/Darling.Tests/StatementFilterWarmUp.cs create mode 100644 Darling/Darling.Tests/StatementFilterWarmUpSourceTests.cs create mode 100644 Darling/Darling.Tests/StatementFilterWarmUpWaitTests.cs create mode 100644 Darling/Darling.Tests/StatementFilterWebTests.cs create mode 100644 Darling/Darling.Tests/StatementMarkerReaderTests.cs create mode 100644 Darling/Darling.Tests/StatementPreviewCutLiveTests.cs create mode 100644 Darling/Darling.Tests/StatementPreviewCutPinTests.cs create mode 100644 Darling/Darling.Tests/StatementScrubBlockingTests.cs create mode 100644 Darling/Darling.Tests/StatementScrubCanary.cs create mode 100644 Darling/Darling.Tests/StatementScrubDeadlockLiveTests.cs create mode 100644 Darling/Darling.Tests/StatementScrubDerivedStoreTests.cs create mode 100644 Darling/Darling.Tests/StatementScrubEndToEndLiveTests.cs create mode 100644 Darling/Darling.Tests/StatementScrubEventsTests.cs create mode 100644 Darling/Darling.Tests/StatementScrubPlanCommandResultTests.cs create mode 100644 Darling/Darling.Tests/StatementScrubQueryStatsLiveTests.cs create mode 100644 Darling/Darling.Tests/StatementScrubQueryStatsTests.cs create mode 100644 Darling/Darling.Tests/StatementScrubQueryStoreLiveTests.cs create mode 100644 Darling/Darling.Tests/StatementScrubQueryStoreTests.cs create mode 100644 Darling/Darling.Tests/StatementScrubRecordingWriter.cs create mode 100644 Darling/Darling.Tests/StatementScrubSnapshotTests.cs create mode 100644 Darling/Darling.Tests/StatementScrubViewerPlanWithheldTests.cs create mode 100644 Darling/Darling.Tests/StatementScrubViewerSavePathsTests.cs create mode 100644 Darling/Darling.Tests/StatusBarBehaviourTests.cs create mode 100644 Darling/Darling.Tests/StorageAndFileIoWebDispatchLiveTests.cs create mode 100644 Darling/Darling.Tests/StorageDatabaseFilterLiveTests.cs create mode 100644 Darling/Darling.Tests/StorageGrowthNewTableCutLiveTests.cs create mode 100644 Darling/Darling.Tests/StoreConfigProviderConnectionMatchTests.cs create mode 100644 Darling/Darling.Tests/StoreDimensionRowCountLiveTests.cs create mode 100644 Darling/Darling.Tests/StoreInstallIdBindingRuleTests.cs create mode 100644 Darling/Darling.Tests/StoreInstallIdLiveTests.cs create mode 100644 Darling/Darling.Tests/StoreInstallIdRefusedClusterLiveTests.cs create mode 100644 Darling/Darling.Tests/StoreInstallIdSourcePinTests.cs create mode 100644 Darling/Darling.Tests/StoreMaintenanceTickLaunchTests.cs create mode 100644 Darling/Darling.Tests/StoreObjectHourlyStepSelectionTests.cs create mode 100644 Darling/Darling.Tests/StoreQueryHistoryToolLiveTests.cs create mode 100644 Darling/Darling.Tests/StoreQueryHistoryToolTests.cs create mode 100644 Darling/Darling.Tests/StoreServerPasswordRulesLiveTests.cs create mode 100644 Darling/Darling.Tests/StoreStatementHistoryLiveTests.cs create mode 100644 Darling/Darling.Tests/StoreStatementHistoryRungTests.cs create mode 100644 Darling/Darling.Tests/StoreStatementHistoryTests.cs create mode 100644 Darling/Darling.Tests/StoredPlanAnalysisFilterTests.cs create mode 100644 Darling/Darling.Tests/TargetSessionIdCacheTests.cs create mode 100644 Darling/Darling.Tests/TargetSessionIdLiveStoreTests.cs create mode 100644 Darling/Darling.Tests/TestData/password-seal-v1-vector.json create mode 100644 Darling/Darling.Tests/TestKeyRings.cs create mode 100644 Darling/Darling.Tests/TestsNoSilentPassGuardTests.cs create mode 100644 Darling/Darling.Tests/TimeRangePickerControlTests.cs create mode 100644 Darling/Darling.Tests/TimeRangeTests.cs create mode 100644 Darling/Darling.Tests/TimingClaim.cs create mode 100644 Darling/Darling.Tests/TimingCollection.cs create mode 100644 Darling/Darling.Tests/TimingTestCensusTests.cs create mode 100644 Darling/Darling.Tests/TimingTestScanner.cs create mode 100644 Darling/Darling.Tests/TopFillEmptyRoundLiveTests.cs create mode 100644 Darling/Darling.Tests/TopProceduresDetailLiveTests.cs create mode 100644 Darling/Darling.Tests/TopProceduresIoRouteLiveTests.cs create mode 100644 Darling/Darling.Tests/TopQueriesDatabaseFilterLiveTests.cs create mode 100644 Darling/Darling.Tests/TopQueriesDetailLiveTests.cs create mode 100644 Darling/Darling.Tests/TopQueriesIoRouteLiveTests.cs create mode 100644 Darling/Darling.Tests/TopRankingDifferentialLiveTests.cs create mode 100644 Darling/Darling.Tests/TopRankingHourlyLiveTests.cs create mode 100644 Darling/Darling.Tests/TopRankingLiveTests.cs create mode 100644 Darling/Darling.Tests/TopRankingNoticeBehaviourTests.cs create mode 100644 Darling/Darling.Tests/TopRankingOnePassBaseline.cs create mode 100644 Darling/Darling.Tests/TopRankingPagePinTests.cs create mode 100644 Darling/Darling.Tests/TopRankingTests.cs create mode 100644 Darling/Darling.Tests/TopRankingTwoPassTests.cs create mode 100644 Darling/Darling.Tests/TopTextLookupFillTests.cs create mode 100644 Darling/Darling.Tests/TopTextLookupPlanLiveTests.cs create mode 100644 Darling/Darling.Tests/TopTextTieBreakLiveTests.cs create mode 100644 Darling/Darling.Tests/TrendStaleStatsLiveTests.cs create mode 100644 Darling/Darling.Tests/TuningHourlyCreateLiveTests.cs create mode 100644 Darling/Darling.Tests/TuningStartPasses.cs create mode 100644 Darling/Darling.Tests/TuningStartPathOneBuildTests.cs create mode 100644 Darling/Darling.Tests/UnpooledDropConnectionCensusTests.cs create mode 100644 Darling/Darling.Tests/ValidateConfigOldFormatPasswordLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewSavedServerScopeTests.cs create mode 100644 Darling/Darling.Tests/ViewerAwsRoleFieldsTests.cs create mode 100644 Darling/Darling.Tests/ViewerAwsRoleUpsertLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerBlockedProcessReportsCoverageLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerBlockingChartsDataStartLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerBlockingChartsDataStartPinTests.cs create mode 100644 Darling/Darling.Tests/ViewerBlockingChartsDataStartTests.cs create mode 100644 Darling/Darling.Tests/ViewerBlockingSourceLabelTests.cs create mode 100644 Darling/Darling.Tests/ViewerBlockingSourceLivePostgresTests.cs create mode 100644 Darling/Darling.Tests/ViewerClutterAndMemoryPressureDataStartLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerClutterAndMemoryPressureDataStartPinTests.cs create mode 100644 Darling/Darling.Tests/ViewerClutterAndMemoryPressureDataStartTests.cs create mode 100644 Darling/Darling.Tests/ViewerCollectionHealthLoadTests.cs create mode 100644 Darling/Darling.Tests/ViewerCollectionLogDataStartLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerCollectionLogDataStartTests.cs create mode 100644 Darling/Darling.Tests/ViewerCollectorDurationChartLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerCollectorDurationChartTests.cs create mode 100644 Darling/Darling.Tests/ViewerCollectorDurationHoverTests.cs create mode 100644 Darling/Darling.Tests/ViewerConfigChangesDataStartTests.cs create mode 100644 Darling/Darling.Tests/ViewerConfigGridCollectedColumnTests.cs create mode 100644 Darling/Darling.Tests/ViewerDataStartBannerTests.cs create mode 100644 Darling/Darling.Tests/ViewerEventDataStartLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerEventDataStartTests.cs create mode 100644 Darling/Darling.Tests/ViewerEventGridsDataStartLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerFinOpsApplicationConnectionsDelegatesTests.cs create mode 100644 Darling/Darling.Tests/ViewerFinOpsDatabaseResourcesDelegatesTests.cs create mode 100644 Darling/Darling.Tests/ViewerFinOpsHighImpactDelegatesTests.cs create mode 100644 Darling/Darling.Tests/ViewerFinOpsIndexAnalysisDelegatesTests.cs create mode 100644 Darling/Darling.Tests/ViewerFinOpsInventoryDelegatesTests.cs create mode 100644 Darling/Darling.Tests/ViewerFinOpsOptimizationDelegatesTests.cs create mode 100644 Darling/Darling.Tests/ViewerFinOpsRecommendationsDelegatesTests.cs create mode 100644 Darling/Darling.Tests/ViewerFinOpsStorageGrowthDelegatesTests.cs create mode 100644 Darling/Darling.Tests/ViewerFinOpsUtilizationDelegatesTests.cs create mode 100644 Darling/Darling.Tests/ViewerFinOpsUtilizationFiguresDelegatesTests.cs create mode 100644 Darling/Darling.Tests/ViewerFinOpsUtilizationNoCpuTests.cs create mode 100644 Darling/Darling.Tests/ViewerFleetScreenWalkFixesTests.cs create mode 100644 Darling/Darling.Tests/ViewerGrantReplay.cs create mode 100644 Darling/Darling.Tests/ViewerHourlyRouteBlankColumnsLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerHourlyWindowEdgesLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerHourlyWindowEdgesTests.cs create mode 100644 Darling/Darling.Tests/ViewerIoRollupRouteLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerIoRollupRouteTests.cs create mode 100644 Darling/Darling.Tests/ViewerJobHistoryDataStartLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerJobHistoryDataStartTests.cs create mode 100644 Darling/Darling.Tests/ViewerLocalStateBehaviourTests.cs create mode 100644 Darling/Darling.Tests/ViewerLongQueriesDataStartTests.cs create mode 100644 Darling/Darling.Tests/ViewerOverviewBlockingLaneDataStartLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerOverviewBlockingLaneDataStartPinTests.cs create mode 100644 Darling/Darling.Tests/ViewerOverviewBlockingLaneDataStartTests.cs create mode 100644 Darling/Darling.Tests/ViewerOverviewSearchTests.cs create mode 100644 Darling/Darling.Tests/ViewerPasswordKeyTests.cs create mode 100644 Darling/Darling.Tests/ViewerPgSnapshotGridTimeTests.cs create mode 100644 Darling/Darling.Tests/ViewerPlanCorrectionsDataStartTests.cs create mode 100644 Darling/Darling.Tests/ViewerPostgresDataStartLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerPostgresDataStartTests.cs create mode 100644 Darling/Darling.Tests/ViewerProbeWatchCensusTests.cs create mode 100644 Darling/Darling.Tests/ViewerQueryGridsDataStartLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerQueryHeatmapDataStartTests.cs create mode 100644 Darling/Darling.Tests/ViewerQueryStoreHistoryTotalExecutionsLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerQueryStoreHistoryTotalExecutionsTests.cs create mode 100644 Darling/Darling.Tests/ViewerQueryStoreRegressionsDataStartTests.cs create mode 100644 Darling/Darling.Tests/ViewerQueryTabLoadTests.cs create mode 100644 Darling/Darling.Tests/ViewerScreenReaderNamesFollowUpTests.cs create mode 100644 Darling/Darling.Tests/ViewerScreenReaderNamesTests.cs create mode 100644 Darling/Darling.Tests/ViewerScreenReaderRowNamesTests.cs create mode 100644 Darling/Darling.Tests/ViewerServerSealLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerShortWindowDataStartTests.cs create mode 100644 Darling/Darling.Tests/ViewerSmtpSealTests.cs create mode 100644 Darling/Darling.Tests/ViewerSnapshotCollectedColumnTests.cs create mode 100644 Darling/Darling.Tests/ViewerSnapshotStripCollectedTests.cs create mode 100644 Darling/Darling.Tests/ViewerSnapshotTimeLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerStatsRowBlankCellTests.cs create mode 100644 Darling/Darling.Tests/ViewerStoreLoginRawReadLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerSystemEventsDataStartTests.cs create mode 100644 Darling/Darling.Tests/ViewerTextLookupWindowBoundLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerTimeRangeGapFixTests.cs create mode 100644 Darling/Darling.Tests/ViewerTimeRangeWindowTests.cs create mode 100644 Darling/Darling.Tests/ViewerWebhookSealLiveTests.cs create mode 100644 Darling/Darling.Tests/ViewerWebhookSealingTests.cs create mode 100644 Darling/Darling.Tests/WaitsDatabaseFilterLiveTests.cs create mode 100644 Darling/Darling.Tests/WebBlockingColumnsPageTests.cs create mode 100644 Darling/Darling.Tests/WebBlockingSaveXmlBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebBlockingSourceBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebChartAtTimeBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebChartLegendIsolateBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebChartMenuBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebChartWidthBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebChartZoomBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebClickthroughPickListTests.cs create mode 100644 Darling/Darling.Tests/WebClickthroughStaleStateTests.cs create mode 100644 Darling/Darling.Tests/WebCollectionHealthRunAtColumnTests.cs create mode 100644 Darling/Darling.Tests/WebCollectorRunHistoryTests.cs create mode 100644 Darling/Darling.Tests/WebDataStartNoteBlockingDeadlockTests.cs create mode 100644 Darling/Darling.Tests/WebDataStartNoteCappedWindowTests.cs create mode 100644 Darling/Darling.Tests/WebDataStartNoteConfigAndLogLiveTests.cs create mode 100644 Darling/Darling.Tests/WebDataStartNoteConfigAndLogTests.cs create mode 100644 Darling/Darling.Tests/WebDataStartNoteEventReadsTests.cs create mode 100644 Darling/Darling.Tests/WebDataStartNoteLiveTests.cs create mode 100644 Darling/Darling.Tests/WebDataStartNoteTests.cs create mode 100644 Darling/Darling.Tests/WebDatabaseBlankRuleTests.cs create mode 100644 Darling/Darling.Tests/WebDatabaseFilterBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebDatabaseFilterChipsBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebDatabaseFilterDispatchLiveTests.cs create mode 100644 Darling/Darling.Tests/WebDatabaseFilterPickerBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebDatabaseFilterReadsTests.cs create mode 100644 Darling/Darling.Tests/WebDatabaseFilterTests.cs create mode 100644 Darling/Darling.Tests/WebDeadlockGraphBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebDeadlockGraphShapeTests.cs create mode 100644 Darling/Darling.Tests/WebFileIoAndPvsWindowPageTests.cs create mode 100644 Darling/Darling.Tests/WebFileIoLatencyBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebHarness/load-web-modules.mjs create mode 100644 Darling/Darling.Tests/WebHarnessJsTreeCopyTests.cs create mode 100644 Darling/Darling.Tests/WebMemoryPressureChartBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebModuleLoadTests.cs create mode 100644 Darling/Darling.Tests/WebPageRangeTests.cs create mode 100644 Darling/Darling.Tests/WebPageRefreshIntervalBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebPerfmonMultiSelectBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebPgPlanViewerPageTests.cs create mode 100644 Darling/Darling.Tests/WebPgStatementEvictionTilePageTests.cs create mode 100644 Darling/Darling.Tests/WebPlainTextTests.cs create mode 100644 Darling/Darling.Tests/WebPlanViewerPageTests.cs create mode 100644 Darling/Darling.Tests/WebQueryStoreGridGroupsTests.cs create mode 100644 Darling/Darling.Tests/WebQueryStoreHistoryColumnTests.cs create mode 100644 Darling/Darling.Tests/WebReadProbe.cs create mode 100644 Darling/Darling.Tests/WebReadReachLiveTests.cs create mode 100644 Darling/Darling.Tests/WebReadReachTests.cs create mode 100644 Darling/Darling.Tests/WebRouteStatementCensusTests.cs create mode 100644 Darling/Darling.Tests/WebServerPageRangeTests.cs create mode 100644 Darling/Darling.Tests/WebServerTrendsBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebServerTrendsPageTests.cs create mode 100644 Darling/Darling.Tests/WebTimeRangeTests.cs create mode 100644 Darling/Darling.Tests/WebTopProceduresColumnsTests.cs create mode 100644 Darling/Darling.Tests/WebTopQueriesColumnsTests.cs create mode 100644 Darling/Darling.Tests/WebWaitQueriesLinkBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebWaitStatsMultiSelectBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebWaitsActivityTrendsBehaviourTests.cs create mode 100644 Darling/Darling.Tests/WebWaitsActivityTrendsPageTests.cs create mode 100644 Darling/Darling.Tests/WebhookDeliverySnapshotTests.cs create mode 100644 Darling/Darling.Tests/WindowFloorLiveHarness.cs create mode 100644 Darling/Darling.Tests/WindowNoticeAggregateCases.cs create mode 100644 Darling/Darling.Tests/WpfExitSwitchPinTests.cs create mode 100644 Darling/Darling.Tests/WpfExitSwitchTests.cs create mode 100644 Darling/Darling.Tests/XeSessionMissingWarningTests.cs create mode 100644 Darling/Darling.Tests/admin-server-edit-harness.mjs create mode 100644 Darling/Darling.Tests/alert-history-advice-harness.mjs create mode 100644 Darling/Darling.Tests/alert-history-dismiss-harness.mjs create mode 100644 Darling/Darling.Tests/alert-history-range-harness.mjs create mode 100644 Darling/Darling.Tests/analysis-findings-harness.mjs create mode 100644 Darling/Darling.Tests/finops-index-analysis-limit-harness.mjs create mode 100644 Darling/Darling.Tests/finops-locking-detail-harness.mjs create mode 100644 Darling/Darling.Tests/finops-locking-heat-harness.mjs create mode 100644 Darling/Darling.Tests/finops-locking-picker-harness.mjs create mode 100644 Darling/Darling.Tests/finops-postgres-target-harness.mjs create mode 100644 Darling/Darling.Tests/finops-storage-growth-window-harness.mjs create mode 100644 Darling/Darling.Tests/finops-window-picker-harness.mjs create mode 100644 Darling/Darling.Tests/job-history-harness.mjs create mode 100644 Darling/Darling.Tests/manage-tags-harness.mjs create mode 100644 Darling/Darling.Tests/mute-rules-harness.mjs create mode 100644 Darling/Darling.Tests/page-range-harness.mjs create mode 100644 Darling/Darling.Tests/sidebar-harness.mjs create mode 100644 Darling/Darling.Tests/status-bar-harness.mjs create mode 100644 Darling/Darling.Tests/time-range-harness.mjs create mode 100644 Darling/Darling.Tests/time-range-picker-harness.mjs create mode 100644 Darling/Darling.Tests/top-ranking-notice-harness.mjs create mode 100644 Darling/Darling.Tests/view-scope-harness.mjs create mode 100644 Darling/Darling.Tests/viewer-local-blank-harness.mjs create mode 100644 Darling/Darling.Tests/viewer-local-harness.mjs create mode 100644 Darling/Darling.Tests/web-blocking-save-xml-harness.mjs create mode 100644 Darling/Darling.Tests/web-blocking-source-harness.mjs create mode 100644 Darling/Darling.Tests/web-chart-at-time-harness.mjs create mode 100644 Darling/Darling.Tests/web-chart-legend-harness.mjs create mode 100644 Darling/Darling.Tests/web-chart-menu-harness.mjs create mode 100644 Darling/Darling.Tests/web-chart-width-harness.mjs create mode 100644 Darling/Darling.Tests/web-chart-zoom-harness.mjs create mode 100644 Darling/Darling.Tests/web-clickthrough-pick-harness.mjs create mode 100644 Darling/Darling.Tests/web-clickthrough-stale-harness.mjs create mode 100644 Darling/Darling.Tests/web-collection-caveats-harness.mjs create mode 100644 Darling/Darling.Tests/web-collector-run-history-harness.mjs create mode 100644 Darling/Darling.Tests/web-column-groups-harness.mjs create mode 100644 Darling/Darling.Tests/web-compose-notice-harness.mjs create mode 100644 Darling/Darling.Tests/web-database-filter-chips-harness.mjs create mode 100644 Darling/Darling.Tests/web-database-filter-harness.mjs create mode 100644 Darling/Darling.Tests/web-database-filter-picker-harness.mjs create mode 100644 Darling/Darling.Tests/web-deadlock-graph-harness.mjs create mode 100644 Darling/Darling.Tests/web-deadlock-processes-harness.mjs create mode 100644 Darling/Darling.Tests/web-fleet-silence-harness.mjs create mode 100644 Darling/Darling.Tests/web-gated-collectors-harness.mjs create mode 100644 Darling/Darling.Tests/web-grid-adoption-harness.mjs create mode 100644 Darling/Darling.Tests/web-grid-filter-harness.mjs create mode 100644 Darling/Darling.Tests/web-grid-sort-harness.mjs create mode 100644 Darling/Darling.Tests/web-grid-tools-harness.mjs create mode 100644 Darling/Darling.Tests/web-grid-value-filter-harness.mjs create mode 100644 Darling/Darling.Tests/web-memory-pressure-harness.mjs create mode 100644 Darling/Darling.Tests/web-page-refresh-harness.mjs create mode 100644 Darling/Darling.Tests/web-perfmon-multi-harness.mjs create mode 100644 Darling/Darling.Tests/web-pg-plan-viewer-harness.mjs create mode 100644 Darling/Darling.Tests/web-picker-driver.mjs create mode 100644 Darling/Darling.Tests/web-plain-text-harness.mjs create mode 100644 Darling/Darling.Tests/web-plan-row-harness.mjs create mode 100644 Darling/Darling.Tests/web-plan-viewer-harness.mjs create mode 100644 Darling/Darling.Tests/web-qs-grids-harness.mjs create mode 100644 Darling/Darling.Tests/web-qs-history-harness.mjs create mode 100644 Darling/Darling.Tests/web-release-walk-harness.mjs create mode 100644 Darling/Darling.Tests/web-server-trends-harness.mjs create mode 100644 Darling/Darling.Tests/web-wait-queries-link-harness.mjs create mode 100644 Darling/Darling.Tests/web-waits-activity-trends-harness.mjs create mode 100644 Darling/Darling.Tests/web-waits-harness.mjs create mode 100644 Darling/PerformanceMonitor.Darling.Analysis/AnalysisStatementText.cs create mode 100644 Darling/PerformanceMonitor.Darling.Analysis/PgSecondaryReplicaScope.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/CollectionHistoryAudit.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/CollectorFaultStackLog.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Compose/ComposeServerScope.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Compose/QueryStoreGroupMembers.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Compose/QueryStoreWideReadGuard.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DailyRunLimiter.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DailyRunWatch.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingAdminServersReader.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingAlwaysOnXeSessions.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingCliSealKey.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingHeartbeat.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingLegacyLongQuerySession.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingOwnedSecrets.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingPasswordKeyFile.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingPasswordKeyRuntime.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingPasswordKeySelfCheck.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingPasswordKeyState.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingPasswordKeyStore.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingRemovedServerSessions.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingServerDatabasesEndpoint.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingServiceKeyFile.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingUnknownKeys.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingWebDeadlockGraph.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingWebStatementSweep.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DarlingWebhookSecrets.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundle.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundleAliaser.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundleRunner.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundleServiceLog.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Hosting/CidrAllowList.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Hosting/DarlingListenerTls.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Hosting/DarlingWebFailureObserver.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/LaunchMemoryGuard.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingCollectionCaveatReader.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingCollectorRunTimeReader.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDeadlockProcessRows.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingGatedCollectorRows.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFinOpsInventoryTools.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFinOpsRecommendationsTools.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFinOpsTools.ApplicationConnections.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFinOpsTools.DatabaseResources.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFinOpsTools.DatabaseSizes.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFinOpsTools.HighImpact.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFinOpsTools.IndexAnalysis.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFinOpsTools.Optimization.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFinOpsTools.StorageGrowth.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFinOpsTools.Utilization.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFinOpsTools.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreHistoryTools.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerAdminTools.Edit.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerTagTools.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerTrendTools.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpSlowReadTools.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreQueryHistoryTools.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpWindowNotice.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingReproScript.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingSlowReadReader.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebSecurityHeaders.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/IGcfOutputFormat.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/ListenerTlsCertificateState.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/McpTlsCertificateState.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Mcp/TopRanking.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/PlanDigestCache.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/PostgresOpenTimeout.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/ProcedureStatsPlanKey.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/ProcedureStatsPlanReuse.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/QueryStatsPlanKey.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/QueryStoreIntervalFloorWarmUp.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/ReadScope.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/ReadStatementCapture.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/RetentionCollectionYield.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/SecretTextGuard.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/SlowReadLog.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/TargetSessionIdCache.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Targets/AwsAssumedRoleCredentials.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Targets/AwsRoleAllowlist.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Targets/AwsRoleAssumeException.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Targets/AwsRoleClients.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Targets/AwsRoleCredentialCache.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Targets/AwsRoleKey.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/Targets/RdsEndpointVerifier.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/WebDataStartNote.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/WebReadReach.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/XeSessionMissingWarnings.cs create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/css/time-range.css create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/database-filter-reads.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/fleet-groups.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/grid-tools.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/grid-value-filter.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/multi-picker.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/mute-context.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/page-range.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/admin.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/analysis-findings.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/database-filter.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/deadlock-graph.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/application-connections.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/database-box.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/database-resources.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/database-sizes.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/gate.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/high-impact.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/index-analysis.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/locking.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/optimization.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/recommendations.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/server-inventory.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/storage-growth.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/utilization.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/version-store.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/window.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/job-history.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/manage-tags.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/mute-rules.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/pg-plan-viewer.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/plan-viewer.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/query-store-history.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tab-reach.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/plain-text.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/plan-row.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/server-order.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/sidebar.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/time-range-picker.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/time-range.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/viewer-local-ui.js create mode 100644 Darling/PerformanceMonitor.Darling.Service/wwwroot/js/viewer-local.js create mode 100644 Darling/PerformanceMonitor.Darling.Storage/AlertDismissStore.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/AwsRoleSettings.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/BoundedDdlBusyStreaks.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/CollectedDatabases.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/DarlingJobHistoryReader.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/DarlingServerClocksReader.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/DataWindowFloor.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/DatabaseFilter.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/DatabaseNameHistoryReader.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/FinOps/DarlingFinOpsApplicationConnectionsReader.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/FinOps/DarlingFinOpsDatabaseResourcesReader.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/FinOps/DarlingFinOpsDatabaseSizesReader.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/FinOps/DarlingFinOpsHighImpactReader.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/FinOps/DarlingFinOpsIndexAnalysisReader.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/FinOps/DarlingFinOpsInventoryReader.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/FinOps/DarlingFinOpsOptimizationReader.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/FinOps/DarlingFinOpsRecommendationsReader.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/FinOps/DarlingFinOpsStorageGrowthReader.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/FinOps/DarlingFinOpsUtilizationReader.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/FinOps/FinOpsCost.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/FinOps/FinOpsInventoryFigures.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/FinOps/FinOpsRecommendationFigures.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/FinOps/FinOpsRollupRouting.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/FinOps/FinOpsScoring.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/FinOps/FinOpsUtilizationFigures.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/FinOps/IndexCleanupRollupFigures.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/IntervalRollupCountGuard.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/IntervalRollupRestartRows.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/LegacyPinCandidateTables.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/PasswordBinding.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/PasswordKeyTables.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/PasswordSeal.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/PerfmonRegroupDrain.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/PgIoStatsHourly.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/PgIoStatsHourlyBuilder.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/PlanRegressionDaily.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/QueryStatsHourLedger.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/QueryStatsHourLedgerWriter.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/QueryStoreBackgroundIndexes.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/QueryStoreComposeStamp.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalBrin.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalPartitions.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/QueryStoreTopDaily.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/RollupCoverageWarmup.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/ServerConnectionIdentity.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/ServerConnectionRule.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/ServerScopeSql.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/ServerTagCoverage.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/ServerTagRules.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/ServerTagStore.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/ServerTrendSql.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/StoreInstallId.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/StoreStatementHistory.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/TopFill.cs create mode 100644 Darling/PerformanceMonitor.Darling.Storage/ViewerEventDataStart.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/CollectorDurationSeries.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/CollectorScheduleRunAtText.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/DataGridHeaderFit.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/FinOpsServerChoice.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/OverviewCardView.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/PasswordKeyChangedDialog.xaml create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/PasswordKeyChangedDialog.xaml.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/ViewerBlockingChartsDataStart.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/ViewerBlockingLaneDataStart.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.PasswordKey.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/ViewerDataStartNote.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/ViewerLoadTimer.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/ViewerPasswordKey.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/ViewerPasswordKeyPins.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/ViewerProcedureHistoryIdleRuns.cs delete mode 100644 Darling/PerformanceMonitor.Darling.Viewer/ViewerServerSecret.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.EventDataStart.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.PostgresDataStart.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.ProbeWatch.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/ViewerSmtpSeal.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/ViewerTimeRangeWindow.cs create mode 100644 Darling/PerformanceMonitor.Darling.Viewer/ViewerWebhookSealing.cs create mode 100644 Lite.Tests/AlertHistoryCountFilterTests.cs create mode 100644 Lite.Tests/AlertStatementCutOrderLiteTests.cs create mode 100644 Lite.Tests/AlwaysOnXeReadOnlyIntentLiteTests.cs create mode 100644 Lite.Tests/AlwaysOnXeSessionsLiteTests.cs create mode 100644 Lite.Tests/AppLoggerDuckDbOutOfMemoryHintTests.cs create mode 100644 Lite.Tests/ArchiveCompactionLowDiskTests.cs create mode 100644 Lite.Tests/ArchiveCopyOptionsPinTests.cs create mode 100644 Lite.Tests/ArchiveNoOpPassViewRebuildTests.cs create mode 100644 Lite.Tests/ArchiveQuerySnapshotsDailyCompactionTests.cs create mode 100644 Lite.Tests/ArchiveQueryStatsDailyCompactionTests.cs create mode 100644 Lite.Tests/ArchiveReadLimitTests.cs create mode 100644 Lite.Tests/ArchiveTieOrderTests.cs create mode 100644 Lite.Tests/BlockingChartsDataStartTests.cs create mode 100644 Lite.Tests/BlockingDatabaseFilterToolTests.cs create mode 100644 Lite.Tests/BlockingSourceLabelTests.cs create mode 100644 Lite.Tests/CiSelectMapSelectGuardTests.cs create mode 100644 Lite.Tests/CiSelectReplayGuardTests.cs create mode 100644 Lite.Tests/ClickThroughUiTextTests.cs create mode 100644 Lite.Tests/CollectionHealthEffectiveCadenceTests.cs create mode 100644 Lite.Tests/CollectionHealthKeyedLookupParityTests.cs create mode 100644 Lite.Tests/CollectionHealthOracleSql.cs create mode 100644 Lite.Tests/CollectionHealthReadBehaviourTests.cs create mode 100644 Lite.Tests/CollectionHealthRunTimeToolTests.cs create mode 100644 Lite.Tests/CollectionHealthTabScheduleTests.cs create mode 100644 Lite.Tests/CollectionLogDrillDataStartTests.cs create mode 100644 Lite.Tests/CollectorDurationDetailTextTests.cs create mode 100644 Lite.Tests/CollectorDurationHoverTests.cs create mode 100644 Lite.Tests/CollectorRunTimeScheduleTests.cs create mode 100644 Lite.Tests/CollectorRunTimeSweepTests.cs create mode 100644 Lite.Tests/CollectorRunTimeWarningLogTests.cs create mode 100644 Lite.Tests/CollectorSelfQueryMarkerCensusTests.cs create mode 100644 Lite.Tests/CrossAppStatementFilterPinTests.cs create mode 100644 Lite.Tests/DarlingReadsTraitGuardTests.cs create mode 100644 Lite.Tests/DataStartBannerBlockingEventsTests.cs create mode 100644 Lite.Tests/DataStartBannerLatchSpinlockTests.cs create mode 100644 Lite.Tests/DataStartBannerProbeFailureTests.cs create mode 100644 Lite.Tests/DataStartBannerQueriesTabTests.cs create mode 100644 Lite.Tests/DataStartBannerSurfaceTests.cs create mode 100644 Lite.Tests/DataStartBannerTests.cs create mode 100644 Lite.Tests/DeadlockAndBlockedProcessEnsureLogLevelLiteTests.cs create mode 100644 Lite.Tests/DesktopDatabaseFilterReadsTests.cs create mode 100644 Lite.Tests/DuckDbInitializerDisposalGuardTests.cs create mode 100644 Lite.Tests/DuckDbMemoryLimitSettingTests.cs create mode 100644 Lite.Tests/DurationTrendDatabaseFilterToolTests.cs create mode 100644 Lite.Tests/DurationTrendFilteredDenominatorTests.cs create mode 100644 Lite.Tests/EmptyDatabaseListNoteLiteTests.cs create mode 100644 Lite.Tests/EventListDatabaseFilterToolTests.cs create mode 100644 Lite.Tests/FileIoLatencyTopFilesTests.cs create mode 100644 Lite.Tests/FinOpsFirstVisitAndBaselineTests.cs create mode 100644 Lite.Tests/FinOpsIdleCoverageAndInventoryScoreTests.cs create mode 100644 Lite.Tests/FinOpsSecondaryReplicaTests.cs create mode 100644 Lite.Tests/GenericWebhookHeadersOriginTests.cs create mode 100644 Lite.Tests/GuardStageCensusTests.cs create mode 100644 Lite.Tests/InstallIdStoreTests.cs create mode 100644 Lite.Tests/InstallIdTests.cs create mode 100644 Lite.Tests/JobHistoryDataStartTests.cs create mode 100644 Lite.Tests/JobHistoryServerClockWindowTests.cs create mode 100644 Lite.Tests/LiteAlertForwardingTests.StatementFilter.cs create mode 100644 Lite.Tests/LiteRunningJobsCurrentSnapshotTests.cs create mode 100644 Lite.Tests/LiteTimeRangeGapTests.cs create mode 100644 Lite.Tests/LongQueryBannerSessionScopeTests.cs create mode 100644 Lite.Tests/LongQueryTraceLifecycleLiteTests.cs create mode 100644 Lite.Tests/LongQueryTracePermissionDenialLiteTests.cs create mode 100644 Lite.Tests/LongQueryTraceReadOnlyIntentLiteTests.cs create mode 100644 Lite.Tests/MainConnectionLimitsTests.cs create mode 100644 Lite.Tests/McpArgumentTypeTests.cs create mode 100644 Lite.Tests/McpDatabaseNameFilterToolTests.cs create mode 100644 Lite.Tests/McpDatabaseSelectionTests.cs create mode 100644 Lite.Tests/McpHelpersTruncateTests.cs create mode 100644 Lite.Tests/McpJsonPostGuardTests.cs create mode 100644 Lite.Tests/McpStorageDatabaseNameFilterToolTests.cs create mode 100644 Lite.Tests/McpWholeNumberArgumentTests.cs create mode 100644 Lite.Tests/McpWindowNoticeAggregateToolTests.cs create mode 100644 Lite.Tests/McpWindowNoticeConfigAndLogToolTests.cs create mode 100644 Lite.Tests/McpWindowNoticeEventToolTests.cs create mode 100644 Lite.Tests/McpWindowNoticeSystemHealthToolTests.cs create mode 100644 Lite.Tests/McpWindowNoticeToolTests.cs create mode 100644 Lite.Tests/OverviewBlockingLaneDataStartLiveTests.cs create mode 100644 Lite.Tests/OverviewBlockingLaneDataStartPinTests.cs create mode 100644 Lite.Tests/OverviewBlockingLaneDataStartTests.cs create mode 100644 Lite.Tests/PostCollectionCheckpointBudgetTests.cs create mode 100644 Lite.Tests/PreinitializedDuckDb.cs create mode 100644 Lite.Tests/PreinitializedDuckDbTests.cs create mode 100644 Lite.Tests/ProcedureStatsIdleRowsReaderTests.cs create mode 100644 Lite.Tests/ProcedureTrendQueryPinTests.cs create mode 100644 Lite.Tests/QueryStatsTextAfterRankingTests.cs create mode 100644 Lite.Tests/QueryStoreBackfillDroppedDatabaseTests.cs create mode 100644 Lite.Tests/QueryStoreHistoryTotalExecutionsTests.cs create mode 100644 Lite.Tests/QueryStoreOldReadSql.cs create mode 100644 Lite.Tests/QueryStoreTextAfterRankingTests.cs create mode 100644 Lite.Tests/RefreshCoordinatorTests.cs create mode 100644 Lite.Tests/ReleaseWalkFilterAndCompareTests.cs create mode 100644 Lite.Tests/ScreenReaderNamesFollowUpTests.cs create mode 100644 Lite.Tests/ScreenReaderNamesTests.cs create mode 100644 Lite.Tests/ScreenReaderRowNamesTests.cs create mode 100644 Lite.Tests/SecondaryReplicaMcpAnalysisTests.cs create mode 100644 Lite.Tests/SecondaryReplicaScopeTests.cs create mode 100644 Lite.Tests/ServerScheduleSummaryTests.cs create mode 100644 Lite.Tests/ServerTabEmptyStateAndNamesTests.cs create mode 100644 Lite.Tests/ServerTabHistoryOverlayTests.cs create mode 100644 Lite.Tests/ServerTabStatusTextTests.cs create mode 100644 Lite.Tests/SettingsImportTests.cs create mode 100644 Lite.Tests/SnapshotCollectedTimeTests.cs create mode 100644 Lite.Tests/StaThreadCensusTests.cs create mode 100644 Lite.Tests/StatementAnalysisReadCutTests.cs create mode 100644 Lite.Tests/StatementDrillDownTextFetchTests.cs create mode 100644 Lite.Tests/StatementFilterDrillDownWarningsTests.cs create mode 100644 Lite.Tests/StatementFilterLiteMcpTests.cs create mode 100644 Lite.Tests/StatementScrubDeadlockCleanupTests.cs create mode 100644 Lite.Tests/StatementScrubDerivedStoreTests.cs create mode 100644 Lite.Tests/StatementScrubLiteCollectionTests.cs create mode 100644 Lite.Tests/StatementScrubLivePlanDisplayTests.cs create mode 100644 Lite.Tests/StatementScrubLiveReadTests.cs create mode 100644 Lite.Tests/StatementScrubWholeMarkerIdentityTests.cs create mode 100644 Lite.Tests/StoredPlanAnalysisFilterTests.cs create mode 100644 Lite.Tests/TabLoadsOffUiThreadTests.cs create mode 100644 Lite.Tests/TestCpuAccountant.cs create mode 100644 Lite.Tests/TestCpuAccountantTests.cs create mode 100644 Lite.Tests/TestMapProducerGuardTests.cs create mode 100644 Lite.Tests/TestMapShadowGuardTests.cs create mode 100644 Lite.Tests/TimingCollection.cs create mode 100644 Lite.Tests/TimingTestCensusTests.cs create mode 100644 Lite.Tests/TopListTotalOrderPinTests.cs create mode 100644 Lite.Tests/TopListWaitforRefillReaderTests.cs create mode 100644 Lite.Tests/TopTextTieBreakReaderTests.cs create mode 100644 Lite.Tests/WaitingBlockedDatabaseFilterToolTests.cs create mode 100644 Lite/Analysis/SecondaryReplicaScope.cs create mode 100644 Lite/Controls/LiteBlockingLaneDataStart.cs create mode 100644 Lite/Controls/OverviewCardBorder.cs create mode 100644 Lite/Controls/ServerTab.BlockingChartsDataStart.cs create mode 100644 Lite/Controls/ServerTab.QueriesDataStart.cs create mode 100644 Lite/Helpers/LiteTimeRange.cs create mode 100644 Lite/Mcp/McpCollectorRunTimes.cs create mode 100644 Lite/Mcp/McpDatabaseSelection.cs create mode 100644 Lite/Services/ArchiveReadLimiter.cs create mode 100644 Lite/Services/DataVolumeSpace.cs create mode 100644 Lite/Services/DuckDbMemoryLimitSetting.cs create mode 100644 Lite/Services/FinOpsShowReloadPolicy.cs create mode 100644 Lite/Services/InstallIdStore.cs create mode 100644 Lite/Services/LivePlanDisplay.cs create mode 100644 Lite/Services/LocalDataService.QueryStoreText.cs create mode 100644 Lite/Services/ProcedureHistoryIdleRuns.cs create mode 100644 Lite/Services/ReadPhaseTimer.cs create mode 100644 Lite/Services/RemoteCollectorService.AlwaysOnXeSessions.cs create mode 100644 Lite/Services/RemoteCollectorService.LegacyLongQuerySession.cs create mode 100644 Lite/Services/RemoteCollectorService.RunTime.cs create mode 100644 Lite/Services/SettingsImport.cs create mode 100644 Lite/Services/TopFill.cs create mode 100644 PerformanceMonitor.Alerting/AlertStatementFilter.cs create mode 100644 PerformanceMonitor.Alerting/RunningJobsCurrency.cs create mode 100644 PerformanceMonitor.Analysis/FactReplicaScope.cs create mode 100644 PerformanceMonitor.Analysis/WithheldStatementMarker.cs create mode 100644 PerformanceMonitor.Collectors/AlwaysOnXeSessions.cs create mode 100644 PerformanceMonitor.Collectors/CollectorRunTime.cs create mode 100644 PerformanceMonitor.Collectors/EmptyDatabaseListNote.cs create mode 100644 PerformanceMonitor.Collectors/IScheduleFrequencyOverride.cs create mode 100644 PerformanceMonitor.Collectors/InstallId.cs create mode 100644 PerformanceMonitor.Collectors/LegacyLongQuerySession.cs create mode 100644 PerformanceMonitor.Collectors/LongQueryTraceDatabases.cs create mode 100644 PerformanceMonitor.Collectors/LongQueryTraceStep.cs create mode 100644 PerformanceMonitor.Common/AgReplicaScope.cs create mode 100644 PerformanceMonitor.Common/BlockingThresholdCoverage.cs create mode 100644 PerformanceMonitor.Common/DatabaseNameHistory.cs create mode 100644 PerformanceMonitor.Common/DerivedStoreScrub.cs create mode 100644 PerformanceMonitor.Common/Mcp/McpToolParameterTypes.cs create mode 100644 PerformanceMonitor.Common/Mcp/SensitiveStatementOutputFilter.cs create mode 100644 PerformanceMonitor.Common/SensitiveStatements.Json.cs create mode 100644 PerformanceMonitor.Common/SensitiveStatements.Regex.cs create mode 100644 PerformanceMonitor.Common/SensitiveStatements.Session.cs create mode 100644 PerformanceMonitor.Common/SensitiveStatements.WarmUp.cs create mode 100644 PerformanceMonitor.Common/SensitiveStatements.Xml.cs create mode 100644 PerformanceMonitor.Common/SensitiveStatements.cs create mode 100644 PerformanceMonitor.Common/Services/ColumnValueList.cs create mode 100644 PerformanceMonitor.Common/Services/ColumnValueListColumns.cs create mode 100644 PerformanceMonitor.Common/Services/JsonContentType.cs create mode 100644 PerformanceMonitor.Common/TagColours.cs create mode 100644 PerformanceMonitor.Ui/AccessibleNames.cs create mode 100644 PerformanceMonitor.Ui/BlockingSourceLabel.cs create mode 100644 PerformanceMonitor.Ui/CollectorDurationHoverText.cs create mode 100644 PerformanceMonitor.Ui/ColumnFilterScope.cs create mode 100644 PerformanceMonitor.Ui/ColumnFilterStore.cs create mode 100644 PerformanceMonitor.Ui/ColumnValueListModel.cs create mode 100644 PerformanceMonitor.Ui/ComparisonBaselineNote.cs create mode 100644 PerformanceMonitor.Ui/DatabaseFilterSelection.cs create mode 100644 PerformanceMonitor.Ui/EmptyState.cs create mode 100644 PerformanceMonitor.Ui/FileIoChartFiles.cs create mode 100644 PerformanceMonitor.Ui/IoLatencyWeighting.cs create mode 100644 PerformanceMonitor.Ui/LongQueriesEmptyText.cs create mode 100644 PerformanceMonitor.Ui/PlanNodeBorder.cs create mode 100644 PerformanceMonitor.Ui/RefreshCoordinator.cs create mode 100644 PerformanceMonitor.Ui/TimeColumnTitle.cs create mode 100644 PerformanceMonitor.Ui/TimeRange/ResolvedTimeRange.cs create mode 100644 PerformanceMonitor.Ui/TimeRange/RollingUnitRule.cs create mode 100644 PerformanceMonitor.Ui/TimeRange/TimeRangeParser.cs create mode 100644 PerformanceMonitor.Ui/TimeRange/TimeRangePresets.cs create mode 100644 PerformanceMonitor.Ui/TimeRange/TimeRangeSpec.cs create mode 100644 PerformanceMonitor.Ui/TimeRangePicker.xaml create mode 100644 PerformanceMonitor.Ui/TimeRangePicker.xaml.cs create mode 100644 PerformanceMonitor.Ui/WithheldPlanGuard.cs create mode 100644 docs/changelog/3.10.md create mode 100644 tools/drain-ab-replay/Invoke-DrainAbReplay.ps1 create mode 100644 tools/drain-ab-replay/README.md create mode 100644 tools/drain-ab-replay/proof/Start-FakeRetention.ps1 create mode 100644 tools/drain-ab-replay/proof/example-output/latency_by_script.csv create mode 100644 tools/drain-ab-replay/proof/example-output/phases.csv create mode 100644 tools/drain-ab-replay/proof/example-output/rates.csv create mode 100644 tools/drain-ab-replay/proof/example-output/run.json create mode 100644 tools/drain-ab-replay/proof/example-output/summary.md create mode 100644 tools/drain-ab-replay/proof/seed-proof-rig.sql diff --git a/.claude/skills/lane-orders/SKILL.md b/.claude/skills/lane-orders/SKILL.md index ab79617ff1..d70db08784 100644 --- a/.claude/skills/lane-orders/SKILL.md +++ b/.claude/skills/lane-orders/SKILL.md @@ -7,15 +7,15 @@ disable-model-invocation: false # Lane orders -One lane works one issue, or a few small related ones, in its own worktree. The coordinator reviews each PR, arms -auto-merge and splices the CHANGELOG. These orders apply alongside the user's token guardrails, which win on +One lane works one issue, or a few small related ones, in its own worktree. An issue too big for one lane still gets +ONE branch and ONE PR: its lanes share the issue branch (see "Worktree and branch"). The coordinator reviews each PR, +arms auto-merge and splices the CHANGELOG. These orders apply alongside the user's token guardrails, which win on conflict. **The coordinator fills these per wave, in every brief:** - ``, ``, ``; - ``, ``, ``; -- ``; -- ``, ``. +- ``. If your brief leaves one unfilled, don't invent a value. Skip the step that needs it and say so in the PR body. In wave E (2026-09-24), a lane whose brief left `` unfilled wrote its changelog entry to a @@ -54,6 +54,11 @@ At dispatch: In wave B, every lane read an 8-10k-character PR section to see one example. - Label each issue the lane takes `in-progress` (`gh issue edit --add-label in-progress`), so the board shows it as taken; remove the label when the issue closes or the work stops. +- Before dispatching several lanes for one issue, create its issue branch + (`git -C push origin origin/dev:refs/heads/fix/-`) and name it in every brief, with the lane + that writes the PR summary and the CHANGELOG entry. Open a separate PR only for a part that must ship before the + rest; an issue that is really several issues is split into issues first. Reviews of an issue PR built in waves: one + round per wave on that wave's commits only, plus one final round on the whole PR. At each report: 1. Stop the agent. @@ -123,6 +128,11 @@ census, one of them introduced by that night's own wave). - `git -C fetch origin`, then `git -C worktree add -b fix/- / origin/dev`. Use one branch per issue. +- **An issue that several lanes build** shares one issue branch, `fix/-`, which the coordinator creates + from `origin/dev` and names in each brief. Create your worktree from `origin/` on a branch of your own + (`fix/--`). When done: `git fetch origin`, `git merge origin/`, rerun your targeted + tests, and `git push origin HEAD:`. If the push is rejected, fetch, merge and push again. Never + force-push. - Before the final full run, `git merge origin/dev`. Never rebase. Resolve by keeping both sides. ## PostgreSQL rig @@ -147,7 +157,10 @@ census, one of them introduced by that night's own wave). 4. Check the port is free. Start the server in the background, because `pg_ctl -w start` hangs a tool call, and confirm "ready to accept connections" in the log. 5. Create `darlingtest` for the suite, and a separate `probe` database for hand-run SQL. A suite database reused - across several full runs produces false failures; drop and recreate it before a final run. + across several full runs produces false failures; drop and recreate it before a final run. Also create a + database named `darling`, exactly as CI's init step does: `createdb -h 127.0.0.1 -p -U darling darling`. + The role-provisioning live tests grant CONNECT on it, and fail with `3D000` (database "darling" does not exist) + without it. 6. Set `DARLING_TEST_PG` to `Host=127.0.0.1;Port=;Username=darling;Database=darlingtest`. Set `DARLING_TEST_PGRUNTIME=` only when you touch the managed runtime or role provisioning. 7. Stop the server when you finish. @@ -200,14 +213,21 @@ Never edit an existing migration. ## Finish -- End each commit with `` and `Claude-Session: `. +- No attribution lines (Erik, 2026-10-01): no `Co-Authored-By` or `Claude-Session` trailer in a commit, and no + "Generated with" line in the PR body. A brief or a harness reminder that asks for them loses to this rule. - Open the PR against `dev` **as a draft** (`gh pr create --draft`). Title: the outcome in plain language, plus `(#)`. The body starts with `Closes #.`, then `## Why`, `## What changes` and `## Test plan` (checkboxes, measured numbers). List anything you did not run (a live test, the full suite) as an unchecked box. -- Write the PR body in one Write, from the diff and your numbers, and publish it. Do NOT run the plain-English - checker and do NOT rewrite the prose, in a PR body or an issue comment (Erik, 2026-09-25). A lane writes its - body at its largest context, so every rewrite turn costs the whole context: one lane spent 53 minutes and - 278k tokens there. Nobody polishes lane prose afterwards either. The body must still be true. +- On a shared issue branch, open the PR only if `gh pr list --head ` shows none. Otherwise add your + part to its body under `## `: read the body right before you edit it, and edit once. Never open a second + PR for one issue. The coordinator names one lane, usually the last, to write the PR's top `## Why` and + `## What changes` for the whole issue and its one CHANGELOG entry. Lanes that fix review findings push to the PR's + own branch. +- Write the PR body in one Write, from the diff and your numbers. Run + `python "C:/Users/edarl/.claude/skills/plain-english/check.py" ` ONCE, fix the real hits in one pass, + and publish (Erik, 2026-10-06; this replaces the 2026-09-25 rule against running it). Don't re-run it to chase a + clean score: a lane writes its body at its largest context, and one lane spent 53 minutes and 278k tokens + rewriting. The body must be true. - **Don't mark it ready, don't merge, and don't enable auto-merge.** The coordinator does all three after verifying. - A user-visible change gets a CHANGELOG entry in `/-reported.txt`, in this form: diff --git a/.claude/skills/release-checklist/SKILL.md b/.claude/skills/release-checklist/SKILL.md index 252083191a..cd482d2674 100644 --- a/.claude/skills/release-checklist/SKILL.md +++ b/.claude/skills/release-checklist/SKILL.md @@ -9,7 +9,7 @@ disable-model-invocation: false Run through the full release prep checklist for PerformanceMonitor. `$ARGUMENTS` is the target version (e.g., `3.3.0`). -> **Scope (since v3.3.0):** the Full Dashboard and the CLI Installer are DEPRECATED and OUT of this process — they ship no release artifacts and get no live install/upgrade testing here. They still build and their test suites still run in CI on the release event, and they inherit the single `` with everything else in the tree (below). The release artifacts are: Lite (zip + Velopack Setup.exe), the Darling service zip, and the Darling Viewer Setup.exe. +> **Scope (since v3.3.0):** the Full Dashboard and the CLI Installer are DEPRECATED and OUT of this process — they ship no release artifacts and get no live install/upgrade testing here. The Darling service zip and the Linux tarball DO get live install and upgrade testing, in section 8a. They still build and their test suites still run in CI on the release event, and they inherit the single `` with everything else in the tree (below). The release artifacts are: Lite (zip + Velopack Setup.exe), the Darling service zip, and the Darling Viewer Setup.exe. ## Checklist @@ -87,6 +87,8 @@ dotnet build PerformanceMonitor.sln -c Debug -t:Rebuild ``` All projects must succeed with **0 Warning(s) / 0 Error(s)** — the repo's bar is zero-warning, test projects included (they sit outside the WarningsAsErrors gate, so a warm incremental build can hide a warning a Rebuild surfaces). +The statement-column census test fails while any entry or watched writer is pending, in every Darling test run, with or without the variable. Run `DARLING_RELEASE_CUT=1 Darling/Darling.Tests/bin/Debug/net10.0-windows/Darling.Tests.exe -method "*StatementCensus_PendingEntries_BlockARelease*"` to see the list early; the cut proceeds only when it passes. The release workflow (build.yml, the "Statement filter release gate" step) runs the same test with the variable set before it publishes or packages anything, so a release cannot ship while an entry is pending even if this step is skipped; run it here first to see the list early. + (The deprecated Installer/Dashboard test suites run in CI on the release event with the CI's own filter; they are no longer run from this checklist.) ### 5. Cloud Platform Testing (shared collector layer — test via DARLING) @@ -106,10 +108,10 @@ Spin up temporary instances with the Azure CLI (`az`) and AWS CLI (`aws`): 2. Add the Azure logical server to a Darling instance (a local scratch service or an existing dogfooding instance) and let it collect 2+ cycles. 3. Verify against the store (psql or the web dashboard): - `database_size_stats` / size collectors cover ALL databases, not just master (the #1631 fallback class) - - server row shows engine edition 5 and collectors inapplicable to Azure SQL DB are SKIPPED as not-applicable, not erroring + - server row shows engine edition 5, and no collector that does not apply to Azure SQL DB errors. The SQL Agent collectors (`running_jobs`, `job_history`, `agent_status`) are marked not applicable when the server is added (`add_servers` answers "Agent surface: not applicable") and never run, so they leave no `collection_log` row and no health entry. That is the expected result, not a gap (verified v3.10.0) - `collection_log` clean for the cloud server across the cycles - the serverless DB's 60s auto-pause + resume (error 40613 transients) does not wedge collection — rows resume after the pause without a service restart -4. Firewall churn: DELETE the allow rule, wait 3–5 minutes, re-add it. Collection must resume on its own without restarting the service. (CAVEAT, verified v3.2.0: a bare rule-deletion often does NOT sever an actively-collecting client — connection pooling keeps the open connection alive and Azure gates only NEW connections — so treat a no-outage result as inconclusive rather than a pass; the recovery logic's authoritative validation is the unit suites.) +4. Firewall churn: DELETE the allow rule, wait 3–5 minutes, re-add it. Collection must resume on its own without restarting the service. (CAVEAT, verified v3.2.0: a bare rule-deletion often does NOT sever an actively-collecting client — connection pooling keeps the open connection alive and Azure gates only NEW connections — so treat a no-outage result as inconclusive rather than a pass; the recovery logic's authoritative validation is the unit suites.) To force a real outage (verified v3.10.0): open admin sessions in each user database BEFORE deleting the rule, in case the machine you test from shares the Darling host's egress IP. Delete the rule, wait 6 minutes for it to take effect, then KILL Darling's sessions (`program_name` = `PerformanceMonitorDarling`) from those sessions. KILL is denied in master even for the server admin, so collectors pooled on master keep running. Collectors that open new connections then hit 40615: the `collection_log` message must name the firewall, and those collectors must store rows again after the rule is re-added, with the service's `started_at` unchanged. 5. Clean up: `az group delete --name rg-release-test --yes --no-wait` **AWS RDS:** (same `SQL_TEST_PASSWORD` throwaway as above) @@ -130,7 +132,7 @@ Spin up temporary instances with the Azure CLI (`az`) and AWS CLI (`aws`): aws ec2 revoke-security-group-ingress --group-id --protocol tcp --port 1433 --cidr 0.0.0.0/0 --region us-east-1 ``` -Success criteria: no collector errors in the store's `collection_log` for either cloud server, all applicable collectors landing rows, capability skips recorded as skips rather than failures. +Success criteria: no collector errors in the store's `collection_log` for either cloud server, all applicable collectors landing rows, no inapplicable collector recorded as a failure (RDS records them as capability skips; Azure SQL DB never runs them). ### 6. Lite Collector Validation @@ -156,6 +158,69 @@ Covers the **desktop apps' own** upgrade via Velopack (`*-Setup.exe`) for **Lite Local proxy without a release: bump ``, rebuild, run over the old build → expect the close-and-takeover prompt. The decision logic is unit-tested (`Lite.Tests/SingleInstanceDecisionTests`); this step validates the live Win32/Velopack seam (`SingleInstanceCoordinator` / `ProcessInspector` in `PerformanceMonitor.Ui`). When the seam saw no changes since the prior release, a quick post-publish regression pass is acceptable instead of a pre-cut gate. Design: `plans/single-instance-upgrade-handoff.md`. +### 8a. Darling service install and upgrade (every release) + +Section 8 covers the desktop apps. This section covers the Darling service zip and the Linux tarball. They carry their own install and upgrade scripts, which a Velopack run never touches. A service install that works only on a clean machine is the shape that shipped broken in #5627. So none of this is optional, and the unit suites cannot stand in for it. + +Run the Windows items on a disposable machine or VM, and point them at a throwaway test instance. Keep the machine-neutral rule: no local instance names, hosts or credentials in anything you record. A cloud test instance gets its throwaway `SQL_TEST_PASSWORD`, as in section 5. + +#### (a) The Windows upgrade-shape CI job + +The workflow is `.github/workflows/darling-upgrade-shapes.yml`. It runs 14 legs: seven on `windows-2022` and seven on `windows-2025`. Each leg installs a real released build with that build's own scripts, then runs this commit's install or upgrade script over it. The legs cover four shapes: +- U1: an install on the default virtual account. +- U2: an install whose service logon was changed to a non-default local account. +- U3: an install whose service was deleted. +- U4: a pre-3.9 folder directly under `C:\` that must still refuse and print the move steps. + +The workflow runs on pull requests that touch `Darling/tools/*.ps1`, as the `Darling upgrade shapes` job in the nightly, and by manual dispatch. The release commit often carries no such check, so look for the run at its sha. In the release commit's checks or the nightly, confirm all 14 legs ran at that sha and are green, by name. A skipped leg is not a pass. If there is no run at that sha, start one with `gh workflow run darling-upgrade-shapes.yml --ref ` and wait for it. + +Also read the two pins at the top of the workflow, `DARLING_CURRENT_RELEASE` and `DARLING_OLD_RELEASE`. The first must name the last shipped release, and the second a release from before 3.9 that is still downloadable. Record the run URL. If any leg is red or skipped, stop. A red leg blocks the release, and re-running it until it passes is not a fix. + +#### (b) `upgrade-darling.ps1` over a real existing install + +Do the whole procedure twice: once from the last release, and once from the oldest release still supported. + +- **Which release is the oldest supported:** the oldest `Darling/Darling.Tests/Fixtures/migration-ladder-v.sql` in the tree. `MigrationUpgradeLadderLiveTests` climbs the current ladder over every fixture in that folder. Its `TheMostRecentRelease_HasALadderFixture` check keeps the newest shipped release in the set. So the oldest fixture is the oldest store the tests vouch for. Read the folder at release time, and do not copy a version number into this file. Deleting a fixture drops that release from support, so make it a deliberate decision and not a tidy-up. +- **Set up the old install:** download that release's Darling zip and extract it to `C:\Program Files\PerformanceMonitorDarling`. Write a `darling.json` that points at one throwaway test instance. Install it the way that release documented: its `install-darling.ps1`, or the manual steps in the README's "Upgrading from a build that predates the script" section when that release has no script. Let it collect for at least two cycles. Before upgrading, record `MAX(version)` from `darling_schema_version` and the account the service runs as. Also record the row counts of two or three `collect.*` tables. +- **Upgrade:** extract the new release zip to a staging folder under `C:\Program Files\` that only administrators can write to. From an elevated PowerShell, run that copy's `upgrade-darling.ps1 -Source `. On one of the two runs, point `-Source` at the downloaded zip with `-Sha256` from the release page, so the hash path runs too. When the old release predates the script, the new zip's script still runs over it, because it finds the install through the registered service. If the README still tells those users to take the manual procedure, walk that once as well. Record which path worked. +- **Pass, at the script:** it exits 0 without a refusal. It prints that the SHA256 was verified (zip run), that `darling.json` is byte-identical, that the install manifest was written, and "Service is Running". +- **Pass, 10 to 15 minutes later:** work the five checks the script prints against the store. + 1. `MAX(version)` equals the new build's top rung. + 2. `collect.collection_log` has rows for every configured server in the last 15 minutes. + 3. The collector count is in the mid-30s, not single digits. + 4. Every non-SUCCESS row since the restart gets read. YIELDED is the lock-timeout guard working. Anything else is a finding. + 5. One table the release added or changed shows new rows. + + The row counts you recorded before the upgrade must still be there, and the service account must be unchanged. A service that is Running but not collecting is a fail. +- **What the tests already cover:** the `*RungTests` classes in `Darling/Darling.Tests` pin each migration rung's SQL and probes on every build, with no database. `MigrationUpgradeLadderLiveTests` replays the current ladder over each released store on a scratch PostgreSQL, in the `Darling PostgreSQL tests` job. `DarlingStoreUpgradeTests` and `DarlingStoreUpgradeRevertTests` cover the bundled runtime swap and the in-place PostgreSQL upgrade. They use the old and new runtimes that `Darling/tools/new-upgraded-store-fixture.ps1` builds, and the nightly sets that up. +- **What the live run adds:** none of those tests runs the upgrade script or the Windows service manager. None of them sees the file ownership and permissions of an install an older build made, or a store holding real rows. +- **Record:** the from and to versions, the install path, and the schema version before and after. Add the five check results and the tail of the script's output. + +#### (c) A fresh `install-darling.ps1` run + +Use a machine with no prior install and no `C:\ProgramData\PerformanceMonitorDarling` folder. Extract the release zip to `C:\Program Files\PerformanceMonitorDarling` and copy `darling.sample.json` to `darling.json`. Point it at the test instance, and run `install-darling.ps1` from an elevated PowerShell. + +Pass means three things. The pre-flight `--test-connection` shows PASS for each server. The service is created on its virtual account and reaches Running. The same five store checks hold after 10 to 15 minutes. + +Then run `uninstall-darling.ps1`, with `-PurgeData` on a machine you will reuse, and confirm the service is gone. Record the pre-flight lines and the five check results. + +#### (d) The Linux tarball and the container image + +The release also attaches `PerformanceMonitorDarling-linux-x64-.tar.gz` and `SHA256SUMS-linux.txt`. The `darling-linux` job in `build.yml` builds them: a framework-dependent publish of the service, tarred with no wrapper folder. The same job pushes `ghcr.io/erikdarlingdata/performancemonitor-darling`, tagged with the version and `latest`. + +Linux has no install script, no upgrade script and no systemd unit file in the tarball. The supported shapes are in the README's "Run on Linux" section. One is the compose stack in `Darling/compose/`. The other is the tarball extracted to `/opt/darling` under a unit written from the README's example, with your own PostgreSQL. + +There is no scripted Linux upgrade path. An upgrade is the same manual steps as a fresh install, over an existing store. So the release must check this by hand: + +- **Artifacts:** the tarball and `SHA256SUMS-linux.txt` are on the release, the checksum matches, and the versioned and `latest` image tags exist. +- **Fresh install, compose:** bring the stack up from this release's `Darling/compose/` files on a fresh volume. The compose file names `:latest`, which still points at the previous release until this one publishes. So run the check with this release's own image tag (`:`), set in the compose file or by an override file for the check. Pass: `docker compose ps` shows the darling container healthy (not `unhealthy`). The log shows the config load and `Postgres store ready`. `collection_log` fills, as in the Windows checks. +- **Fresh install, systemd:** extract the tarball to `/opt/darling` and install `libgssapi-krb5-2`. Write the unit from the README. Point `DARLING_CONFIG` at a config for a throwaway PostgreSQL 15+ with TimescaleDB, and start it. Pass: the unit is active and the same store checks hold. +- **Upgrade:** repeat each shape over a store the previous release created. For compose, keep the volume and recreate the container from this release's compose file and image. For systemd, stop the unit, move `/opt/darling` aside, extract the new tarball, keep the config, and start the unit. Pass: the service starts, the store migrates to the new top rung, and collection continues with the earlier rows still present. +- **Oldest supported:** apply the same fixture rule as (b). If that release did not ship a Linux tarball, say so in the record and use the oldest release that did. +- **Record:** the same fields as (b), plus the image digest or tarball checksum you tested. + +Put the outcome of (a) to (d) in the release's test record. An unchecked box blocks the tag. + ### 9. Nightly Build Verification Before cutting the release, verify the nightly build has been clean: diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml index 098803aba5..5a7e8d2628 100644 --- a/.github/FUNDING.yml +++ b/.github/FUNDING.yml @@ -1 +1,3 @@ github: erikdarlingdata +patreon: DarlingData +custom: "https://go.erikdarling.com/Donate" diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 841abd0aec..7bf8234ff2 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -85,11 +85,21 @@ body: id: errors attributes: label: Error Messages / Log Output - description: Paste any error messages or relevant log entries. Dashboard logs are in %LOCALAPPDATA%\PerformanceMonitor\Logs\; Lite logs are in %LOCALAPPDATA%\PerformanceMonitorLite-Data\logs\ (on versions before the data-root move, %LOCALAPPDATA%\PerformanceMonitorLite\logs\). + description: Paste any error messages or relevant log entries. Dashboard logs are in %LOCALAPPDATA%\PerformanceMonitor\Logs\; Lite logs are in %LOCALAPPDATA%\PerformanceMonitorLite-Data\logs\ (on versions before the data-root move, %LOCALAPPDATA%\PerformanceMonitorLite\logs\); Darling logs are in %ProgramData%\PerformanceMonitorDarling\logs\. render: text validations: required: false + - type: textarea + id: diagnostics-bundle + attributes: + label: Diagnostics bundle (Darling) + description: | + Run `PerformanceMonitor.Darling.Service.exe --diagnostics-bundle darling-bundle.json` from an elevated prompt in the service folder (Windows). On Linux, run `./PerformanceMonitor.Darling.Service --diagnostics-bundle darling-bundle.json`; for compose, run it with `docker compose exec` and copy the file out with `docker compose cp`. + Drag the file here (rename it to .txt if GitHub refuses .json). Host, server and database names in it are replaced with aliases, and secrets are removed. Open it and check it before you attach it. + validations: + required: false + - type: textarea id: screenshots attributes: diff --git a/.github/ci-history/failures.jsonl b/.github/ci-history/failures.jsonl new file mode 100644 index 0000000000..f6b97b23ad --- /dev/null +++ b/.github/ci-history/failures.jsonl @@ -0,0 +1,731 @@ +{"date":"2026-09-07","head_sha":"7abcaf15220493815da410b23a0509f5f1fa3a8b","event":"pull_request","changed_files":["Darling/Darling.Tests/StoreCopyPhaseTests.cs","Darling/Darling.Tests/StoreWriteReattemptTests.cs","Darling/PerformanceMonitor.Darling.Service/CollectorFaultCopyPhase.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/PostgresTransportFault.cs","Darling/PerformanceMonitor.Darling.Service/StoreWriteReattempt.cs","PerformanceMonitor.Collectors/CollectorContext.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-07","head_sha":"5cba51a487bd08ccb8c252ee1c5efc7393b47714","event":"pull_request","changed_files":["Darling/Darling.Tests/StoreCopyPhaseLivePostgresTests.cs","Darling/Darling.Tests/StoreCopyPhaseTests.cs","Darling/Darling.Tests/StoreWriteReattemptTests.cs","Darling/PerformanceMonitor.Darling.Service/CollectorFaultCopyPhase.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/PostgresTransportFault.cs","Darling/PerformanceMonitor.Darling.Service/StoreWriteReattempt.cs","PerformanceMonitor.Collectors/CollectorContext.cs"],"failed_classes":["Darling.Tests.LiveCleanupConversionRatchetTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-07","head_sha":"a059fe218a30f9d998ea9396cbb379855c423bdf","event":"pull_request","changed_files":["Darling/Darling.Tests/PgIndexBloatBudgetLivePostgresTests.cs","Lite.Tests/PgIndexBloatCollectorDefinitionTests.cs","PerformanceMonitor.Collectors/PgIndexBloatCollector.cs"],"failed_classes":["Darling.Tests.LiveCleanupConversionRatchetTests","Darling.Tests.PgIndexBloatBudgetLivePostgresTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-07","head_sha":"cc29d98125b8f796b66447114bf3c23dd958e156","event":"pull_request","changed_files":["Darling/docs/L3116-skip-arm-probe.md"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["Darling whole-tree guards"]} +{"date":"2026-09-07","head_sha":"a304350fe32f3fb286518a4c73b9f626785ae4bb","event":"pull_request","changed_files":["Darling/Darling.Tests/RepoFileAdoptionTests.cs","Lite.Tests/LiteLogLevelGateTests.cs","Lite/App.xaml.cs","Lite/Services/AppLogger.cs","Lite/Services/AppLoggerAdapter.cs","Lite/Services/RemoteCollectorService.DefinitionRunner.cs","Lite/config/settings.sample.json","README.md"],"failed_classes":["Lite.Tests.LiteServerTagsStoreTests"],"failed_jobs":["build"]} +{"date":"2026-09-07","head_sha":"4205ef83306e362c854c9b8285eadb949b034e15","event":"pull_request","changed_files":[".github/workflows/build.yml"],"failed_classes":["Darling.Tests.DarlingAnalysisBudgetTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-07","head_sha":"01fdeae92aaba8b87216c9e69ec87d9787d03e61","event":"push","changed_files":["Darling/Darling.Tests/PostgresCancelOriginTests.cs","Darling/Darling.Tests/PostgresFaultOutcomeTests.cs","Darling/Darling.Tests/StoreCopyPhaseTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs","Darling/PerformanceMonitor.Darling.Service/CollectorFaultCancelOrigin.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Targets/PostgresTargetProvider.cs","Darling/README.md"],"failed_classes":["Darling.Tests.DarlingDeployStaleFileTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-07","head_sha":"4f19be020a4dcca78c8098aa13d3c4db5b551d0d","event":"pull_request","changed_files":["Lite.Tests/Lite.Tests.csproj","Lite.Tests/SharedDuckDbFixture.cs","Lite.Tests/WriteLockBudgetTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/Services/LocalDataService.cs"],"failed_classes":["Lite.Tests.CrossAppGuardCiGateTests"],"failed_jobs":["build"]} +{"date":"2026-09-07","head_sha":"bf7c8e6bc8f00f5c7fd617639c858e94dd62b34d","event":"pull_request","changed_files":["Lite.Tests/Lite.Tests.csproj","Lite.Tests/SharedDuckDbFixture.cs","Lite.Tests/WriteLockBudgetTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/Services/LocalDataService.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["build"]} +{"date":"2026-09-07","head_sha":"39a702e4f9d724e3fd0558c46b2587d19fed1a1e","event":"pull_request","changed_files":[".github/dependabot.yml",".github/workflows/build.yml",".github/workflows/nightly.yml","Darling/Darling.Tests/LockedModeRestoreCoverageTests.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PostgreSQL tests","Darling whole-tree guards"]} +{"date":"2026-09-07","head_sha":"237f69c04c3caaf1e10425f98f2d3f4b623039a0","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertEngineTests.cs","Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Lite.Tests/AlertReadFailureSurfaceTests.cs","Lite/Mcp/McpHealthTools.cs","PerformanceMonitor.Alerting/AlertEngine.cs","PerformanceMonitor.Alerting/AlertReadFailureCounter.cs"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-08","head_sha":"3cb494b9abdf5e94f948c823a36088e89ab3e15a","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionLogDrainForensicsStoreTests.cs","Darling/Darling.Tests/CollectionLogPhaseSplitStoreTests.cs","Darling/Darling.Tests/CollectorMeasurementSeamTests.cs","Darling/Darling.Tests/DarlingEmptyEnumerationNoteTests.cs","Darling/Darling.Tests/ServerScopePhaseSplitTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Lite.Tests/CollectorMeasurementNoteTests.cs","Lite.Tests/EmptyEnumerationNoteTests.cs","Lite/Services/RemoteCollectorService.DefinitionRunner.cs","Lite/Services/RemoteCollectorService.cs","PerformanceMonitor.Collectors/BlockedProcessReportCollector.cs","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/CollectorMeasurement.cs"],"failed_classes":["Darling.Tests.StoreWriteReattemptTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-08","head_sha":"28d98e40af1261cfde60fbe390499d7f5a9b6748","event":"pull_request","changed_files":["Darling/Darling.Tests/RefreshCeilingProvenancePinTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/Darling.Tests/TimescaleSupportTests.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.RefreshCeilingProvenancePinTests","Darling.Tests.TimescaleSupportTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-08","head_sha":"e97c2bb91639812dd11619bacf881c7a9333686e","event":"pull_request","changed_files":["Darling/Darling.Tests/RefreshCeilingProvenancePinTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/Darling.Tests/TimescaleSupportTests.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.RefreshCeilingProvenancePinTests","Darling.Tests.TimescaleSupportTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-08","head_sha":"ee66f9a95d092000231122ba262601982d5e3a8b","event":"pull_request","changed_files":["Darling/Darling.Tests/MigrationDataMovingRungCensusPins.cs","Darling/Darling.Tests/OperatorRemediationFlowTests.cs","Darling/Darling.Tests/OperatorRemediationGateTests.cs","Darling/Darling.Tests/PlanForceActionStoreTests.cs","Darling/Darling.Tests/PostgresTargetConfigTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSecrets.cs","Darling/PerformanceMonitor.Darling.Service/DarlingServerConnector.cs","Darling/PerformanceMonitor.Darling.Service/MonitoredServerConnection.cs","Darling/PerformanceMonitor.Darling.Service/PgPlanForceActionStore.cs","Darling/PerformanceMonitor.Darling.Service/PlanForceBot.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Service/darling.sample.json","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/tools/provision-roles.sql","Lite.Tests/OperatorRemediationLiteDivergencePinTests.cs","PerformanceMonitor.Analysis/ForcePlanBotPolicy.cs","PerformanceMonitor.Analysis/OperatorRemediationFlow.cs","PerformanceMonitor.Analysis/OperatorRemediationGate.cs"],"failed_classes":["Darling.Tests.CollectorStallProbeStoreTests","Darling.Tests.CollectorStallProbeViewerGateTests","Darling.Tests.DarlingManagedRolesTests","Darling.Tests.DarlingRetentionTests","Darling.Tests.RegisteredServerSettingDriftTests","Lite.Tests.OperatorRemediationLiteDivergencePinTests"],"failed_jobs":["Darling PostgreSQL tests","build","Darling whole-tree guards"]} +{"date":"2026-09-08","head_sha":"c398ba40dceec3a89503740aee292cf78fb9d1a8","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertDeliveryChannelTests.cs","Darling/Darling.Tests/DarlingAlertingTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertDeliverer.cs","Darling/PerformanceMonitor.Darling.Service/DarlingFindingAlertSender.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.AlertHistory.cs","Lite.Tests/AlertDeliveryChannelTests.cs","Lite.Tests/StoreRoundTripTests.cs","Lite/Services/EmailAlertService.cs","Lite/Services/LocalDataService.AlertHistory.cs","PerformanceMonitor.Notifications/AlertDelivery.cs","PerformanceMonitor.Notifications/EmailSendCore.cs","PerformanceMonitor.Notifications/IAlertHistoryStore.cs","PerformanceMonitor.Notifications/WebhookAlertService.cs","deprecated/Dashboard.Tests/DashboardAlertHistoryValueTests.cs","deprecated/Dashboard.Tests/JsonAlertHistoryStoreDedupKeyTests.cs","deprecated/Dashboard.Tests/JsonAlertHistoryStoreMutedFilterTests.cs","deprecated/Dashboard/Services/EmailAlertService.cs"],"failed_classes":["Darling.Tests.DarlingAlertingTests","Darling.Tests.DarlingDeliveryModeTests","Darling.Tests.DarlingSelfAlertTests","Darling.Tests.ViewerWave3LivePostgresTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-08","head_sha":"52aabbdf74646007a9ead98a94ad99b46d081f1c","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorStallProbeStoreTests.cs","Darling/Darling.Tests/CollectorStallProbeViewerGateTests.cs","Darling/Darling.Tests/DarlingManagedRolesTests.cs","Darling/Darling.Tests/DarlingRetentionTests.cs","Darling/Darling.Tests/MigrationDataMovingRungCensusPins.cs","Darling/Darling.Tests/OperatorRemediationFlowTests.cs","Darling/Darling.Tests/OperatorRemediationGateTests.cs","Darling/Darling.Tests/PlanForceActionStoreTests.cs","Darling/Darling.Tests/PostgresTargetConfigTests.cs","Darling/Darling.Tests/RegisteredServerSettingDriftTests.cs","Darling/Darling.Tests/RemediationCredentialRungTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSecrets.cs","Darling/PerformanceMonitor.Darling.Service/DarlingServerConnector.cs","Darling/PerformanceMonitor.Darling.Service/MonitoredServerConnection.cs","Darling/PerformanceMonitor.Darling.Service/PgPlanForceActionStore.cs","Darling/PerformanceMonitor.Darling.Service/PlanForceBot.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Service/darling.sample.json","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/tools/provision-roles.sql","Lite.Tests/OperatorRemediationLiteDivergencePinTests.cs","PerformanceMonitor.Analysis/ForcePlanBotPolicy.cs","PerformanceMonitor.Analysis/OperatorRemediationFlow.cs","PerformanceMonitor.Analysis/OperatorRemediationGate.cs"],"failed_classes":["Darling.Tests.RegisteredServerSettingDriftTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-08","head_sha":"455aefc3501c4441acb9bf05cded3aa49b659f37","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingMcpStoreMetricsToolsTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreMetricsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/StoreSelfMetrics.cs"],"failed_classes":["Darling.Tests.DarlingManagedPostgresTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-08","head_sha":"b8ae88ab2dea213b09adfb12d62b8f7806db778f","event":"push","changed_files":["Darling/Darling.Tests/CollectorStallProbeStoreTests.cs","Darling/Darling.Tests/CollectorStallProbeViewerGateTests.cs","Darling/Darling.Tests/DarlingManagedRolesTests.cs","Darling/Darling.Tests/DarlingRetentionTests.cs","Darling/Darling.Tests/MigrationDataMovingRungCensusPins.cs","Darling/Darling.Tests/OperatorRemediationFlowTests.cs","Darling/Darling.Tests/OperatorRemediationGateTests.cs","Darling/Darling.Tests/PlanForceActionStoreTests.cs","Darling/Darling.Tests/PostgresTargetConfigTests.cs","Darling/Darling.Tests/RegisteredServerSettingDriftTests.cs","Darling/Darling.Tests/RemediationCredentialRungTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSecrets.cs","Darling/PerformanceMonitor.Darling.Service/DarlingServerConnector.cs","Darling/PerformanceMonitor.Darling.Service/MonitoredServerConnection.cs","Darling/PerformanceMonitor.Darling.Service/PgPlanForceActionStore.cs","Darling/PerformanceMonitor.Darling.Service/PlanForceBot.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Service/darling.sample.json","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Darling/tools/provision-roles.sql","Lite.Tests/OperatorRemediationLiteDivergencePinTests.cs","Lite/Analysis/Recommendations/LiteRecommendationItem.cs","Lite/Analysis/Recommendations/LiteRecommendationsViewModel.cs","Lite/Controls/RecommendationsTab.xaml.cs","PerformanceMonitor.Analysis/ForcePlanBotPolicy.cs","PerformanceMonitor.Analysis/OperatorRemediationFlow.cs","PerformanceMonitor.Analysis/OperatorRemediationGate.cs"],"failed_classes":["Darling.Tests.TimescaleSupportTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-08","head_sha":"8931e0a761f3c9c657390f7842287912b19ae8dc","event":"pull_request","changed_files":["Darling/Darling.Tests/CompressionClearanceWatchTests.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.CommentFilterAdoptionTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-08","head_sha":"4442d28b0bdc3e978cc4b5142ebbb2190dea51c6","event":"pull_request","changed_files":["Darling/Darling.Tests/RefreshCeilingStalenessTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.CommentFilterAdoptionTests","Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-08","head_sha":"11d610329cf955d7a85988f4ededd0efcebda619","event":"push","changed_files":["Darling/Darling.Tests/RefreshCeilingProvenancePinTests.cs","Darling/Darling.Tests/RefreshCeilingStalenessTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Lite.Tests.CrossAppGuardCiGateTests"],"failed_jobs":["build"]} +{"date":"2026-09-09","head_sha":"89f49f5a5f57d08f104bae71a8f1adc459853d71","event":"pull_request","changed_files":["Darling/Darling.Tests/McpPayloadClockFrameDisciplineTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingBlockingReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingJobReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpBlockingTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpJobTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpObjectStatsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanCorrectionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPvsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingObjectStatsReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingPlanCorrectionReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingPvsReader.cs","Lite/Mcp/McpBlockingTools.cs","Lite/Mcp/McpJobTools.cs","Lite/Mcp/McpObjectStatsTools.cs","Lite/Mcp/McpPlanCorrectionTools.cs","Lite/Mcp/McpPvsTools.cs","Lite/Mcp/McpServerLocalWindow.cs"],"failed_classes":["Darling.Tests.McpPayloadClockFrameDisciplineTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-09","head_sha":"b7c622ac9bde644a6ab50f344f362731e7ab9080","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreFetchProbeLivePostgresTests.cs","Darling/Darling.Tests/QueryStorePlanFetchTests.cs","Darling/Darling.Tests/QueryStoreTouchGuardSingleSourceTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreLivenessTouchGuard.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStorePlanMap.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreTextStore.cs"],"failed_classes":["Darling.Tests.DarlingDeployRollbackRetentionTests"],"failed_jobs":["build"]} +{"date":"2026-09-09","head_sha":"7e696667e77211df9484c488200035ecb7e059aa","event":"pull_request","changed_files":["Darling/Darling.Tests/McpPayloadClockFrameDisciplineTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingBlockingReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingJobReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpBlockingTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpJobTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpObjectStatsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanCorrectionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPvsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingObjectStatsReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingPlanCorrectionReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingPvsReader.cs","Lite/Mcp/McpBlockingTools.cs","Lite/Mcp/McpJobTools.cs","Lite/Mcp/McpObjectStatsTools.cs","Lite/Mcp/McpPlanCorrectionTools.cs","Lite/Mcp/McpPvsTools.cs","Lite/Mcp/McpServerLocalWindow.cs"],"failed_classes":["Darling.Tests.DarlingMcpObjectStatsToolsSurfaceAndSqlTests","Darling.Tests.DarlingMcpPlanCorrectionToolsTests","Darling.Tests.DarlingMcpPvsToolsTests","Darling.Tests.IndexUsageTruncationTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-09","head_sha":"038366d4fce56bed40f318801165fb23c54f4558","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingPgReadSqlParsesLiveTests.cs","Darling/Darling.Tests/StoreSqlClockDisciplineTests.cs"],"failed_classes":["Lite.Tests.CrossAppGuardCiGateTests"],"failed_jobs":["build"]} +{"date":"2026-09-09","head_sha":"c849c9f134c34e84826c15ac24f2d6b85b033dab","event":"pull_request","changed_files":["Lite.Tests/EntraCredentialSelectionModeGateTests.cs","Lite.Tests/EntraCredentialSelectionOrderingTests.cs","Lite.Tests/EntraCredentialSelectionTests.cs","Lite/Helpers/EntraCredentialSelection.cs","Lite/Services/ServerManager.cs","Lite/Windows/AddServerDialog.xaml.cs","README.md"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["build"]} +{"date":"2026-09-09","head_sha":"b3d65a3f8686f1fb530d9fe1f7965bdb6613c9a5","event":"pull_request","changed_files":["Darling/Darling.Tests/McpPayloadClockFrameDisciplineTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingBlockingReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingJobReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpBlockingTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpJobTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpObjectStatsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanCorrectionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPvsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingObjectStatsReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingPlanCorrectionReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingPvsReader.cs","Lite/Mcp/McpBlockingTools.cs","Lite/Mcp/McpJobTools.cs","Lite/Mcp/McpObjectStatsTools.cs","Lite/Mcp/McpPlanCorrectionTools.cs","Lite/Mcp/McpPvsTools.cs","Lite/Mcp/McpServerLocalWindow.cs"],"failed_classes":["Darling.Tests.ConsumedTimestampFrameDisciplineTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-09","head_sha":"160cf82a247aa08021adf115383926619f4c66da","event":"pull_request","changed_files":["Darling/Darling.Tests/ConsumedTimestampFrameDisciplineTests.cs","Darling/Darling.Tests/McpPayloadClockFrameDisciplineTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingBlockingReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingJobReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpBlockingTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpJobTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpObjectStatsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanCorrectionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPvsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingObjectStatsReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingPlanCorrectionReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingPvsReader.cs","Lite/Mcp/McpBlockingTools.cs","Lite/Mcp/McpJobTools.cs","Lite/Mcp/McpObjectStatsTools.cs","Lite/Mcp/McpPlanCorrectionTools.cs","Lite/Mcp/McpPvsTools.cs","Lite/Mcp/McpServerLocalWindow.cs"],"failed_classes":["Darling.Tests.ConsumedTimestampFrameDisciplineTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-09","head_sha":"0ed7cb858dc05624f1eb388d07ece82a27c263e0","event":"pull_request","changed_files":[".claude/skills/release-checklist/SKILL.md",".github/workflows/build.yml",".github/workflows/check-version-bump.yml",".github/workflows/nightly.yml","Darling/Darling.Tests/ProductVersionDeclarationTests.cs","Darling/PerformanceMonitor.Darling.Service/PerformanceMonitor.Darling.Service.csproj","Darling/PerformanceMonitor.Darling.Viewer/PerformanceMonitor.Darling.Viewer.csproj","Directory.Build.props","Lite/PerformanceMonitorLite.csproj","PerformanceMonitor.Common/Services/UpdateCheckService.cs","build-dashboard.cmd","build-lite.cmd","deprecated/Dashboard/Dashboard.csproj","deprecated/Installer.Core/Installer.Core.csproj","deprecated/Installer/PerformanceMonitorInstaller.csproj","package-release.cmd"],"failed_classes":["Lite.Tests.CrossAppGuardCiGateTests"],"failed_jobs":["build"]} +{"date":"2026-09-09","head_sha":"3c76c11b4b53d998d93f39324efbc305643867ea","event":"pull_request","changed_files":["Darling/Darling.Tests/ConsumedTimestampFrameDisciplineTests.cs","Darling/Darling.Tests/McpPayloadClockFrameDisciplineTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingBlockingReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingJobReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpBlockingTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpJobTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpObjectStatsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanCorrectionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPvsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingObjectStatsReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingPlanCorrectionReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingPvsReader.cs","Lite/Mcp/McpBlockingTools.cs","Lite/Mcp/McpJobTools.cs","Lite/Mcp/McpObjectStatsTools.cs","Lite/Mcp/McpPlanCorrectionTools.cs","Lite/Mcp/McpPvsTools.cs","Lite/Mcp/McpServerLocalWindow.cs"],"failed_classes":["Darling.Tests.ConsumedTimestampFrameDisciplineTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-09","head_sha":"d6da0314eaf991e5a04685b9a9f5cb3b7cbe4d68","event":"pull_request","changed_files":["Darling/Darling.Tests/ConsumedTimestampFrameDisciplineTests.cs","Darling/Darling.Tests/ViewerClockRendererArithmeticTests.cs","Darling/Darling.Tests/ViewerQueriesTests.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Deadlock.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.ItemHistory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.PlanCorrection.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QuerySnapshots.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStore.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStoreRegressions.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.RunningJobs.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerHistoryRows.cs","Lite.Tests/ServerClockRendererArithmeticTests.cs","Lite/Services/LocalDataService.Blocking.cs","Lite/Services/LocalDataService.JobHistory.cs","Lite/Services/LocalDataService.PlanCorrection.cs","Lite/Services/LocalDataService.QueryStats.cs","Lite/Services/LocalDataService.RunningJobs.cs","Lite/Services/ServerTimeHelper.cs"],"failed_classes":["Darling.Tests.LiveCleanupConversionRatchetTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-09","head_sha":"c6bcbae7d8f07bb0bb7ddc0f95ed71df5625d7aa","event":"pull_request","changed_files":["Darling/Darling.Tests/ConsumedTimestampFrameDisciplineTests.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Deadlock.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QuerySnapshots.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStats.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStore.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStoreRegressions.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.RunningJobs.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerHistoryRows.cs","Lite/Services/LocalDataService.Blocking.cs","Lite/Services/LocalDataService.PlanCorrection.cs","Lite/Services/LocalDataService.QueryStats.cs","Lite/Services/LocalDataService.RunningJobs.cs","Lite/Services/ServerTimeHelper.cs"],"failed_classes":["Darling.Tests.ConsumedTimestampFrameDisciplineTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-09","head_sha":"5b0518b0242a0f25309596bff916345066c689d5","event":"pull_request","changed_files":["Darling/Darling.Tests/CSharpMemberMap.cs"],"failed_classes":["Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-09","head_sha":"7b64070c982d28d099d82d5a11de44730cc7abac","event":"pull_request","changed_files":["CHANGELOG.md"],"failed_classes":["Darling.Tests.DarlingDeployRollbackRetentionTests"],"failed_jobs":["Darling whole-tree guards"]} +{"date":"2026-09-09","head_sha":"91cb1fb7226690c7326d6842e97d221e61b4e8a5","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorStateContractTests.cs","Darling/Darling.Tests/DarlingPgIndexBloatNanLivePostgresTests.cs","Darling/Darling.Tests/DarlingPgIndexBloatRotationReadLivePostgresTests.cs","Darling/Darling.Tests/MigrationDataMovingRungCensusPins.cs","Darling/Darling.Tests/PgExtensionDependencyContractTests.cs","Darling/Darling.Tests/PgIndexBloatBudgetLivePostgresTests.cs","Darling/Darling.Tests/PgIndexBloatEstimateRungTests.cs","Darling/Darling.Tests/PgRotationCursorPruneLivePostgresTests.cs","Darling/Darling.Tests/RemediationCredentialRungTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgIndexTools.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingPgIndexBloatReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Postgres.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml","Darling/README.md","Lite.Tests/PgIndexBloatCollectorDefinitionTests.cs","PerformanceMonitor.Collectors/PgIndexBloatCollector.cs","PerformanceMonitor.Collectors/PgPerDatabaseCollectorState.cs","docs/postgres-first-target-runbook.md"],"failed_classes":["Lite.Tests.DefaultTraceEventsCollectorDefinitionTests","Lite.Tests.PgIndexBloatCollectorDefinitionTests"],"failed_jobs":["build"]} +{"date":"2026-09-10","head_sha":"f28452c053d445321e12577fbc4d52fe5716f8bd","event":"pull_request","changed_files":["CHANGELOG.md","Directory.Build.props","README.md"],"failed_classes":["Darling.Tests.MigrationUpgradeLadderLiveTests"],"failed_jobs":["build"]} +{"date":"2026-09-10","head_sha":"1d4dbcaec03c8f40a62e32698398c242808088d5","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorRunnerConnectionEngineTests.cs","Darling/Darling.Tests/ReadmeDerivedCountPinTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgStatementTools.cs","Darling/README.md"],"failed_classes":["Lite.Tests.AuroraOnlySqlIsGatedTests"],"failed_jobs":["build"]} +{"date":"2026-09-10","head_sha":"565b89dd45c5d44db9f00fa5871482e67c2d2daa","event":"pull_request","changed_files":["CHANGELOG.md","Directory.Build.props","README.md"],"failed_classes":["Darling.Tests.MigrationUpgradeLadderLiveTests"],"failed_jobs":["build"]} +{"date":"2026-09-10","head_sha":"1e1c66bd144edcce9544a6bec88e00e1cee3ff8b","event":"pull_request","changed_files":["CHANGELOG.md","Directory.Build.props","README.md"],"failed_classes":["Darling.Tests.MigrationUpgradeLadderLiveTests"],"failed_jobs":["build"]} +{"date":"2026-09-11","head_sha":"f5e2f565551a22a4d7c94f69ce60accf059a94db","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingPgIndexBloatCoverageLivePostgresTests.cs","Darling/Darling.Tests/PgIndexBloatCoverageTests.cs","Darling/Darling.Tests/ServerPageTabsTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgIndexTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Storage/DarlingPgIndexBloatReader.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Postgres.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Postgres.cs","PerformanceMonitor.Collectors/PgIndexBloatCoverage.cs"],"failed_classes":["Darling.Tests.DarlingPgReadSqlParsesLiveTests","Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-11","head_sha":"132b7ee191c3ae276e6bff7358209fd184a525f5","event":"push","changed_files":["Darling/Darling.Tests/DarlingPgIndexBloatCoverageLivePostgresTests.cs","Darling/Darling.Tests/DarlingPgReadSqlParsesLiveTests.cs","Darling/Darling.Tests/PgIndexBloatCoverageTests.cs","Darling/Darling.Tests/ServerPageTabsTests.cs","Darling/Darling.Tests/TsqlConventionGuardTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgIndexTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Storage/DarlingPgIndexBloatReader.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Postgres.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Postgres.cs","PerformanceMonitor.Collectors/PgIndexBloatCoverage.cs"],"failed_classes":["Darling.Tests.DarlingPgIndexBloatCoverageLivePostgresTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-11","head_sha":"ef216f63d7822d64cf4eb9042bfc96d35f174691","event":"pull_request","changed_files":["Darling/Darling.Tests/FleetCardPostgresCpuTests.cs","Darling/Darling.Tests/PgCpuCapacityHeadroomTests.cs","Darling/Darling.Tests/PgIndexBloatEstimateRungTests.cs","Darling/Darling.Tests/ServerHealthClassifierTests.cs","Darling/Darling.Tests/UnmeasuredMetricsAreNotHealthyTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgCpuUtilizationTools.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsCpuIngestor.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/fleet.js","Darling/PerformanceMonitor.Darling.Storage/DarlingPgCpuUtilizationReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Overview.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","PerformanceMonitor.Collectors/PgCpuUtilizationCollector.cs","PerformanceMonitor.Common/FleetCpuSource.cs","PerformanceMonitor.Common/ServerHealthBands.cs"],"failed_classes":["Darling.Tests.DarlingPgIndexBloatCoverageLivePostgresTests","Darling.Tests.DocCommentHygieneTests","Darling.Tests.FleetCardPostgresCpuLivePostgresTests","Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-11","head_sha":"4044fe11ba1a89a4376b685dcd2bcd9951f56447","event":"pull_request","changed_files":["Darling/Darling.Tests/FleetCardPostgresCpuLivePostgresTests.cs","Darling/Darling.Tests/FleetCardPostgresCpuTests.cs","Darling/Darling.Tests/PgCpuCapacityHeadroomTests.cs","Darling/Darling.Tests/PgIndexBloatEstimateRungTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/Darling.Tests/ServerHealthClassifierTests.cs","Darling/Darling.Tests/UnmeasuredMetricsAreNotHealthyTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgCpuUtilizationTools.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsCpuIngestor.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/fleet.js","Darling/PerformanceMonitor.Darling.Storage/DarlingPgCpuUtilizationReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Overview.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","PerformanceMonitor.Collectors/PgCpuUtilizationCollector.cs","PerformanceMonitor.Common/FleetCpuSource.cs","PerformanceMonitor.Common/ServerHealthBands.cs"],"failed_classes":["Darling.Tests.DarlingPgIndexBloatCoverageLivePostgresTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-11","head_sha":"81c136a724e74c270fde5b58b78e93c2b9a9bc59","event":"pull_request","changed_files":["Darling/Darling.Tests/FleetCardPostgresCpuLivePostgresTests.cs","Darling/Darling.Tests/FleetCardPostgresCpuTests.cs","Darling/Darling.Tests/PgCpuCapacityHeadroomTests.cs","Darling/Darling.Tests/PgIndexBloatEstimateRungTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/Darling.Tests/ServerHealthClassifierTests.cs","Darling/Darling.Tests/UnmeasuredMetricsAreNotHealthyTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgCpuUtilizationTools.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsCpuIngestor.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/fleet.js","Darling/PerformanceMonitor.Darling.Storage/DarlingPgCpuUtilizationReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Overview.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","PerformanceMonitor.Collectors/PgCpuUtilizationCollector.cs","PerformanceMonitor.Common/FleetCpuSource.cs","PerformanceMonitor.Common/ServerHealthBands.cs"],"failed_classes":["Darling.Tests.DarlingPgIndexBloatCoverageLivePostgresTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-11","head_sha":"1f1d465e0c41c6eb67309e24f27ba5ceabe0ec77","event":"pull_request","changed_files":["Darling/Darling.Tests/FleetCardPostgresCpuLivePostgresTests.cs","Darling/Darling.Tests/FleetCardPostgresCpuTests.cs","Darling/Darling.Tests/PgCpuCapacityHeadroomTests.cs","Darling/Darling.Tests/PgIndexBloatEstimateRungTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/Darling.Tests/ServerHealthClassifierTests.cs","Darling/Darling.Tests/UnmeasuredMetricsAreNotHealthyTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgCpuUtilizationTools.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsCpuIngestor.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/fleet.js","Darling/PerformanceMonitor.Darling.Storage/DarlingPgCpuUtilizationReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Fleet.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Overview.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","PerformanceMonitor.Collectors/PgCpuUtilizationCollector.cs","PerformanceMonitor.Common/FleetCpuSource.cs","PerformanceMonitor.Common/ServerHealthBands.cs"],"failed_classes":["Darling.Tests.DarlingPgIndexBloatCoverageLivePostgresTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-11","head_sha":"5d82981557501a9ee1aa7e89d4eb37a2f6a0b423","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertPersistenceGateTests.cs","Darling/Darling.Tests/CustomAlertRuleDefinitionTests.cs","Darling/PerformanceMonitor.Darling.Service/CustomAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/CustomAlertRuleDefinition.cs","Darling/PerformanceMonitor.Darling.Service/CustomAlertRuleStore.cs","Darling/PerformanceMonitor.Darling.Service/CustomAlertStateStore.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","PerformanceMonitor.Alerting/AlertPersistenceGate.cs"],"failed_classes":["Darling.Tests.PgCpuCapacityHeadroomTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-11","head_sha":"a9316fcc8f5ab9c55a17b54714aac9e0196e50ba","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpCustomAlertToolsTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpCustomAlertTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs"],"failed_classes":["Darling.Tests.DarlingWebEndpointsTests","Lite.Tests.CrossAppMcpToolInventoryPinTests"],"failed_jobs":["Darling PostgreSQL tests","build","Darling whole-tree guards"]} +{"date":"2026-09-11","head_sha":"04c3c1a4c8bd8a85a7f2f97fb9e8a5c200541025","event":"pull_request","changed_files":["Darling/Darling.Tests/CustomAlertCoreMigrationTests.cs","Darling/Darling.Tests/MuteRuleReloadBeaconLivePostgresTests.cs","Darling/Darling.Tests/MuteRuleReloadBeaconTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAlertTools.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.ServiceConfig.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs"],"failed_classes":["Darling.Tests.MuteRuleReloadBeaconTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-11","head_sha":"eb5ad9acfc02e7c5c927e21440c3e1c0ccb5b656","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertEngineTests.cs","Darling/Darling.Tests/BuiltinAlertPersistenceRungTests.cs","Darling/Darling.Tests/CustomAlertCoreMigrationTests.cs","Darling/Darling.Tests/DarlingAlertingTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/PgCpuCapacityHeadroomTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/PgAlertStateStore.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingPgCpuUtilizationReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Lite.Tests/DuckDbSchemaTests.cs","Lite.Tests/LiteAlertForwardingTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/Database/Schema.cs","Lite/MainWindow.AlertEngine.cs","Lite/Services/DuckDbAlertHistoryStore.cs","Lite/Services/LiteAlertStateStore.cs","Lite/Services/LocalDataService.Overview.cs","PerformanceMonitor.Alerting/AlertEngine.cs","PerformanceMonitor.Alerting/AlertPersistenceRecord.cs","PerformanceMonitor.Alerting/AlertServerSnapshot.cs","PerformanceMonitor.Alerting/IAlertStateStore.cs","README.md"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests","Darling.Tests.RepoFileAdoptionTests","PerformanceMonitorLite.Tests.LiteAlertForwardingTests"],"failed_jobs":["Darling PostgreSQL tests","build","Darling whole-tree guards"]} +{"date":"2026-09-11","head_sha":"8c1231a7772923741c1d2ba4daa9781661030584","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertEngineTests.cs","Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/BuiltinAlertPersistenceRungTests.cs","Darling/Darling.Tests/CustomAlertCoreMigrationTests.cs","Darling/Darling.Tests/DarlingAlertingTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/MuteRuleReloadBeaconTests.cs","Darling/Darling.Tests/PgCpuCapacityHeadroomTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/PgAlertStateStore.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingPgCpuUtilizationReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Lite.Tests/DuckDbSchemaTests.cs","Lite.Tests/LiteAlertForwardingTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/Database/Schema.cs","Lite/MainWindow.AlertEngine.cs","Lite/Services/DuckDbAlertHistoryStore.cs","Lite/Services/LiteAlertStateStore.cs","Lite/Services/LocalDataService.Overview.cs","PerformanceMonitor.Alerting/AlertEngine.cs","PerformanceMonitor.Alerting/AlertPersistenceRecord.cs","PerformanceMonitor.Alerting/AlertServerSnapshot.cs","PerformanceMonitor.Alerting/IAlertStateStore.cs","README.md"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-11","head_sha":"f0a678328ecc8618d9f2bf34bc9db903e9100c5b","event":"pull_request","changed_files":["Darling/Darling.Tests/BuiltinAlertPersistenceRungTests.cs","Darling/PerformanceMonitor.Darling.Service/PgAlertStateStore.cs","Lite/Database/DuckDbInitializer.cs","Lite/Database/Schema.cs","Lite/Services/DuckDbAlertHistoryStore.cs","Lite/Services/LiteAlertStateStore.cs"],"failed_classes":["Darling.Tests.CommentFilterAdoptionTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-12","head_sha":"ccea95f0e5f61080dabc27f5bbf6f4d0b70fff9b","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/MuteRule.cs"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-12","head_sha":"e6dbdbea94c340a7f51f8b2a7426ee5200b14e5b","event":"pull_request","changed_files":["CHANGELOG.md"],"failed_classes":["Darling.Tests.IncidentAttachmentDeliveryTests"],"failed_jobs":["Darling whole-tree guards"]} +{"date":"2026-09-12","head_sha":"c089bad9ede15eca9ad8e173b6a8e7c4e3a8e6aa","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","PerformanceMonitor.Notifications/MuteRule.cs"],"failed_classes":["Darling.Tests.IncidentAttachmentDeliveryTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-12","head_sha":"cca860fb66b2009c22e5fb45f015806093b07dc6","event":"pull_request","changed_files":["Darling/Darling.Tests/BuiltinAlertPersistenceRungTests.cs","Darling/Darling.Tests/DarlingMcpAlertToolsTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/RetentionHoldRatioKnobRungTests.cs","Darling/Darling.Tests/ViewerControlPlaneStage3bTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertSettings.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingAlertReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAlertTools.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.AlertSettings.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Lite.Tests/McpAlertSettingsKeyTests.cs","Lite/Mcp/McpAlertTools.cs","docs/retention-hold-runbook.md"],"failed_classes":["Darling.Tests.ConfigSeedStatementArityTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-12","head_sha":"25bae3e1b1e82163ca989c846fc60a5fdd7a6c0a","event":"push","changed_files":["Darling/PerformanceMonitor.Darling.Service/wwwroot/css/editor.css","Darling/PerformanceMonitor.Darling.Service/wwwroot/index.html","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/alert-editor.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/alert-seed.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/alerts-api.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/app.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/editor.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/notebook.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/alert-rules.js"],"failed_classes":["PerformanceMonitorLite.Tests.AnalysisPassTokenThreadingTests"],"failed_jobs":["build"]} +{"date":"2026-09-12","head_sha":"dd945402303b54c9703b12e46008223aabf4d5de","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingCollectionLogReadTests.cs","Darling/Darling.Tests/DarlingMcpDataToolsTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Lite.Tests/CollectionLogToolTests.cs","Lite/Mcp/McpHealthTools.cs","Lite/Mcp/McpInstructions.cs","Lite/Services/LocalDataService.CollectionHealth.cs","PerformanceMonitor.Common/Mcp/McpHelpers.cs"],"failed_classes":["Darling.Tests.DarlingCollectionLogReadTests","Darling.Tests.QueryStoreTopWindowTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-13","head_sha":"34467c3f02617e0c992320364df30e666225f594","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreProbeCacheTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreProbeCache.cs","Lite.Tests/ProcedureStatsCollectorDefinitionTests.cs","Lite.Tests/QueryStatsCollectorDefinitionTests.cs","PerformanceMonitor.Collectors/ProcedureStatsCollector.cs","PerformanceMonitor.Collectors/QueryPlanXmlCaptureLimits.cs","PerformanceMonitor.Collectors/QueryStatsCollector.cs"],"failed_classes":["Darling.Tests.FleetIdentifierScrubTests","Darling.Tests.ProcedureStatsPlanCadenceTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-13","head_sha":"f850aa9c9db06aa16a77eccc3f6a89df3e1b381f","event":"pull_request","changed_files":["Lite.Tests/CrossAppGuardCiGateTests.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["Darling whole-tree guards"]} +{"date":"2026-09-13","head_sha":"bb916117b252cd8b1f37d37bedf18355d2da86e6","event":"pull_request","changed_files":["Darling/Darling.Tests/OversizedPlanBacklogReadPins.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpOversizedPlanBacklogTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingOversizedPlanBacklogReader.cs","Darling/README.md","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","README.md"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-14","head_sha":"c986bf22e295120a2a550f4754593984ac7d5363","event":"pull_request","changed_files":["Darling/Darling.Tests/OversizedPlanBacklogPins.cs","Darling/Darling.Tests/OversizedPlanSweepRunRecordTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingObservability.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpOversizedPlanBacklogTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingServerResolver.cs","Darling/PerformanceMonitor.Darling.Service/OversizedPlanBacklogSweep.cs","Darling/PerformanceMonitor.Darling.Storage/OversizedPlanBacklog.cs","Darling/README.md","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs"],"failed_classes":["Darling.Tests.OversizedPlanBacklogPins"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-15","head_sha":"e1059d5a0b007036abb89975da7e9883c80c48e4","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorCostRegressionDispersionTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingCollectorCostReader.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-15","head_sha":"13c20722281c933fce20b75c3d6e823acb2e5a83","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorCostRegressionDispersionTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingCollectorCostReader.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-15","head_sha":"26e3eef0b4b9d3d3f5827612aa879c28a106f848","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertDeliveryChannelTests.cs","Darling/Darling.Tests/AlertTrayStatusLoggedTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAlertTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js","Lite.Tests/AlertDeliveryChannelTests.cs","Lite.Tests/PagerDutyWebhookTests.cs","Lite.Tests/StoreRoundTripTests.cs","Lite.Tests/WebhookCooldownSeedTests.cs","Lite/Mcp/McpAlertTools.cs","PerformanceMonitor.Notifications/AlertDelivery.cs","PerformanceMonitor.Notifications/EmailSendCore.cs","PerformanceMonitor.Notifications/IAlertHistoryStore.cs","PerformanceMonitor.Notifications/WebhookAlertService.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.RepeatDeliveryBudgetTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-15","head_sha":"42d5ad5eefebae9b13b8dc22edb3df105fa618e6","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorCostDigestTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingCollectorCostReader.cs","Lite.Tests/AlertSeverityTests.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertSeverity.cs"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests","Darling.Tests.CollectorCostDigestTests","Lite.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["build","Darling PostgreSQL tests","Darling whole-tree guards"]} +{"date":"2026-09-15","head_sha":"2e3852241d11a040cd96c12ccdd06efd00a3c5aa","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpAlertToolsTests.cs","Darling/Darling.Tests/OversizedPlanBacklogPins.cs","Darling/Darling.Tests/PgAlertCountKnobRungTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertSettings.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingAlertReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAlertTools.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.AlertSettings.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","PerformanceMonitor.Alerting/PostgresAlertEvaluator.cs"],"failed_classes":["Lite.Tests.McpAlertSettingsKeyTests"],"failed_jobs":["build"]} +{"date":"2026-09-15","head_sha":"95d3f7799ccdd33858d3f3664f0c46eaf7f8b464","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionLogDrainForensicsStoreTests.cs","Darling/Darling.Tests/PgLoggingCollectorOffTests.cs","Darling/Darling.Tests/PostgresFaultOutcomeTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgDeadlockTools.cs","Lite.Tests/PgLoggingCollectorGateTests.cs","PerformanceMonitor.Collectors/PgDeadlocksCollector.cs","PerformanceMonitor.Collectors/PgLoggingCollectorOffException.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-16","head_sha":"601ab8a7cbe6bf24dde6a9274aa6f3e6cf32ca7f","event":"pull_request","changed_files":["CHANGELOG.md","Darling/Darling.Tests/AlertMasterSwitchSurfaceTests.cs","Darling/Darling.Tests/SameStatementPileupSourceCensusTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFindingStore.cs","Darling/PerformanceMonitor.Darling.Analysis/PgPileupSnapshotReader.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/README.md","Lite.Tests/SameStatementPileupDetectorTests.cs","Lite/Analysis/FindingStore.cs","Lite/Analysis/PileupSnapshotReader.cs","Lite/Services/CollectionBackgroundService.cs","PerformanceMonitor.Analysis/FactAdvice.cs","PerformanceMonitor.Analysis/SameStatementPileupDetector.cs"],"failed_classes":["Darling.Tests.DarlingAnalysisStoreTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-16","head_sha":"2b45c772a7a3b55520f439291e936e1cb8bd1aa6","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpAlertToolsTests.cs","Darling/Darling.Tests/FleetSweepCadenceKnobRungTests.cs","Darling/Darling.Tests/FleetSweepEngineTests.cs","Darling/Darling.Tests/FleetSweepStateRungTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/FleetSweepEngine.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingAlertReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingHealthReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAlertTools.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Storage/FleetSweepCadence.cs","Darling/PerformanceMonitor.Darling.Storage/FleetSweepStore.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.AlertSettings.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Lite.Tests/McpAlertSettingsKeyTests.cs"],"failed_classes":["Darling.Tests.PgAlertCountKnobRungTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-16","head_sha":"7301d2a37766df19e97e6cf23f87e23f64415c50","event":"pull_request","changed_files":["CHANGELOG.md","Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingMcpFleetSweepToolsTests.cs","Darling/Darling.Tests/DarlingRetentionTests.cs","Darling/Darling.Tests/FleetSweepRollupTests.cs","Darling/Darling.Tests/FleetSweepStateRungTests.cs","Darling/Darling.Tests/McpToolTypeRegistrationTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFleetSweepTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Storage/FleetSweepStore.cs","Darling/README.md","Lite.Tests/AlertSeverityTests.cs","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertSeverity.cs","README.md","llms.txt"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests","Lite.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["Darling PostgreSQL tests","build","Darling whole-tree guards"]} +{"date":"2026-09-16","head_sha":"f3c7ab2115308f2127eaac8dd5e03b58eb19b2fb","event":"pull_request","changed_files":["CHANGELOG.md","Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingMcpFleetSweepToolsTests.cs","Darling/Darling.Tests/DarlingRetentionTests.cs","Darling/Darling.Tests/FleetSweepRollupTests.cs","Darling/Darling.Tests/FleetSweepStateRungTests.cs","Darling/Darling.Tests/McpToolTypeRegistrationTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFleetSweepTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Storage/FleetSweepStore.cs","Darling/README.md","Lite.Tests/AlertSeverityTests.cs","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertSeverity.cs","README.md","llms.txt"],"failed_classes":["Lite.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["build"]} +{"date":"2026-09-16","head_sha":"a2829f46d5c779c42f3e611023174433e744c06b","event":"pull_request","changed_files":["CHANGELOG.md","Darling/Darling.Tests/StoreIsOnThisMachineTests.cs","Darling/Darling.Tests/ViewerServerManagementTests.cs","Darling/PerformanceMonitor.Darling.Viewer/AddMultipleServersDialog.xaml","Darling/PerformanceMonitor.Darling.Viewer/AddMultipleServersDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/AddServerDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ServerStoreCredential.cs"],"failed_classes":["Darling.Tests.BulkServerOnboardingMappingTests","PerformanceMonitorLite.Tests.EntraDeviceCodeTests"],"failed_jobs":["build","Darling PostgreSQL tests","Darling whole-tree guards"]} +{"date":"2026-09-17","head_sha":"bfedaca4bc22ddc889e9a9ec0fd7a6afdb59726a","event":"pull_request","changed_files":["CHANGELOG.md","Darling/Darling.Tests/StoreIsOnThisMachineTests.cs","Darling/Darling.Tests/ViewerServerManagementTests.cs","Darling/PerformanceMonitor.Darling.Viewer/AddMultipleServersDialog.xaml","Darling/PerformanceMonitor.Darling.Viewer/AddMultipleServersDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/AddServerDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ServerStoreCredential.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerMigration.cs","Lite.Tests/EntraDeviceCodeTests.cs"],"failed_classes":["Darling.Tests.BulkServerOnboardingMappingTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-17","head_sha":"5a71cef415797316d889da18b4080980ed41cf0a","event":"pull_request","changed_files":["CHANGELOG.md","Darling/Darling.Tests/PostgresAddServerDialogTests.cs","Darling/Darling.Tests/ServerIdentitySurvivesAnEditTests.cs","Darling/Darling.Tests/ViewerServerManagementTests.cs","Darling/PerformanceMonitor.Darling.Viewer/AddMultipleServersDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/AddServerDialog.xaml","Darling/PerformanceMonitor.Darling.Viewer/AddServerDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Commands.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.MonitoredServers.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-17","head_sha":"4e102377669ea939b0dfe32dada5c385e43e0021","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorDatabaseScopeRungTests.cs","Darling/Darling.Tests/FleetSweepCadenceKnobRungTests.cs","Darling/Darling.Tests/PostgresDatabaseListScreenLivePostgresTests.cs","Darling/Darling.Tests/TargetProviderTests.cs","Darling/Darling.Tests/ViewerControlPlaneStage3bTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/DarlingXeSessions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Service/Targets/PostgresTargetProvider.cs","Darling/PerformanceMonitor.Darling.Service/Targets/SqlServerTargetProvider.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/CollectorScheduleEditorWindow.xaml","Darling/PerformanceMonitor.Darling.Viewer/CollectorScheduleOverlay.cs","Darling/PerformanceMonitor.Darling.Viewer/CollectorSchedulePresets.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectorSchedules.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/DatabaseScopeFilter.cs","PerformanceMonitor.Collectors/DatabaseScopedConfigCollector.cs","PerformanceMonitor.Collectors/ITargetProvider.cs","PerformanceMonitor.Collectors/IndexObjectStatsCollector.cs","PerformanceMonitor.Collectors/PlanCorrectionCollector.cs","PerformanceMonitor.Collectors/QueryStoreCollector.cs","PerformanceMonitor.Collectors/QueryStoreHealthCollector.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests","Lite.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["Darling PostgreSQL tests","build","Darling whole-tree guards"]} +{"date":"2026-09-17","head_sha":"76f3ad71d0117c2ed2cca3b92af6fd763d6470d4","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorDatabaseScopeLivePostgresTests.cs","Darling/Darling.Tests/CollectorDatabaseScopeRungTests.cs","Darling/Darling.Tests/FleetSweepCadenceKnobRungTests.cs","Darling/Darling.Tests/PostgresDatabaseListScreenLivePostgresTests.cs","Darling/Darling.Tests/TargetProviderTests.cs","Darling/Darling.Tests/ViewerControlPlaneStage3bTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/DarlingXeSessions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Service/Targets/PostgresTargetProvider.cs","Darling/PerformanceMonitor.Darling.Service/Targets/SqlServerTargetProvider.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/CollectorScheduleEditorWindow.xaml","Darling/PerformanceMonitor.Darling.Viewer/CollectorScheduleOverlay.cs","Darling/PerformanceMonitor.Darling.Viewer/CollectorSchedulePresets.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectorSchedules.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/DatabaseScopeFilter.cs","PerformanceMonitor.Collectors/DatabaseScopedConfigCollector.cs","PerformanceMonitor.Collectors/ITargetProvider.cs","PerformanceMonitor.Collectors/IndexObjectStatsCollector.cs","PerformanceMonitor.Collectors/PlanCorrectionCollector.cs","PerformanceMonitor.Collectors/QueryStoreCollector.cs","PerformanceMonitor.Collectors/QueryStoreHealthCollector.cs"],"failed_classes":["Lite.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["build"]} +{"date":"2026-09-17","head_sha":"4242f787997a41af6802b2544fe1f3a7d7bc352b","event":"pull_request","changed_files":["Darling/Darling.Tests/AgentJobStepQueryTests.cs","Darling/Darling.Tests/AlertEngineTests.cs","Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Lite.Tests/AlertContextBuildersTests.cs","Lite/MainWindow.AlertEngine.cs","Lite/MainWindow.xaml.cs","Lite/Services/RemoteCollectorService.RunningJobs.cs","PerformanceMonitor.Alerting/AgentJobStepQuery.cs","PerformanceMonitor.Alerting/AlertContextBuilders.cs","PerformanceMonitor.Alerting/AlertEngine.cs"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-17","head_sha":"76e349dcda5d6f4f158c90472e78abeadf505f4c","event":"pull_request","changed_files":["Darling/Darling.Tests/AgentJobStepQueryTests.cs","Darling/Darling.Tests/AlertEngineTests.cs","Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Lite.Tests/AlertContextBuildersTests.cs","Lite/MainWindow.AlertEngine.cs","Lite/MainWindow.xaml.cs","Lite/Services/RemoteCollectorService.RunningJobs.cs","PerformanceMonitor.Alerting/AgentJobStepQuery.cs","PerformanceMonitor.Alerting/AlertContextBuilders.cs","PerformanceMonitor.Alerting/AlertEngine.cs"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-18","head_sha":"1d56092cad4e6fdefa0cf5aacc3003062c3b49ae","event":"pull_request","changed_files":["Darling/Darling.Tests/DailyDeadlockWindowCensusTests.cs","Darling/Darling.Tests/DailyHealthBandTests.cs","Darling/Darling.Tests/DarlingMcpHealthToolsTests.cs","Darling/Darling.Tests/FleetSweepEngineTests.cs","Darling/PerformanceMonitor.Darling.Service/FleetSweepEngine.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingHealthReader.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.DailySummary.cs","Lite.Tests/DailyHealthBandTests.cs","Lite.Tests/PerformanceCalendarDataTests.cs","Lite/Services/LocalDataService.DailySummary.cs","PerformanceMonitor.Common/DailyHealthBand.cs","PerformanceMonitor.Common/ServerHealthBands.cs"],"failed_classes":["Darling.Tests.DarlingMcpHealthToolsLivePostgresTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-18","head_sha":"e9f5484eadb29bb7464f543c67a4a2830c1de9a6","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertEngineTests.cs","Darling/Darling.Tests/DarlingAlertReadAdapterTests.cs","Darling/Darling.Tests/DarlingAlertingTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/PostgresAlertEvaluatorTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertReadAdapter.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertSettings.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/README.md","Lite.Tests/AlertContextBuildersTests.cs","Lite.Tests/BlockingDeadlockContextBuilderTests.cs","Lite.Tests/LiteAlertForwardingTests.cs","Lite/Services/AppAlertEngineSettings.cs","Lite/Services/LiteAlertReadAdapter.cs","Lite/Services/LocalDataService.WaitStats.cs","PerformanceMonitor.Alerting/AlertEngine.cs","PerformanceMonitor.Alerting/IAlertEngineSettings.cs","PerformanceMonitor.Alerting/IAlertReadAdapter.cs","PerformanceMonitor.Alerting/PoisonWaitAccumulation.cs","PerformanceMonitor.Alerting/PoisonWaitDelta.cs","PerformanceMonitor.Alerting/PoisonWaitEvaluator.cs","PerformanceMonitor.Alerting/PostgresAlertEvaluator.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertSeverity.cs","README.md"],"failed_classes":["Darling.Tests.DarlingAlertReadAdapterTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-18","head_sha":"28b796d7eaa462dbc49c465bb82ed2a80be38b09","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpAlertToolsTests.cs","Darling/Darling.Tests/DarlingMcpBlockingToolsTests.cs","Darling/Darling.Tests/DarlingMcpDataToolsTests.cs","Darling/Darling.Tests/DarlingMcpPlanCorrectionToolsTests.cs","Darling/Darling.Tests/DarlingMcpSessionToolsTests.cs","Darling/Darling.Tests/McpPageContractTests.cs","Darling/Darling.Tests/PlanCorrectionFrameLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingAlertReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingBlockingReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingLongQueryReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAlertTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpBlockingTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpLongQueryTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanCorrectionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpSessionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingPlanCorrectionReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingSessionReader.cs","Lite.Tests/McpPageContractTests.cs","Lite/Mcp/McpAlertTools.cs","Lite/Mcp/McpBlockingTools.cs","Lite/Mcp/McpInstructions.cs","Lite/Mcp/McpLongQueryTools.cs","Lite/Mcp/McpPlanCorrectionTools.cs","Lite/Mcp/McpWaitTools.cs","Lite/Services/LocalDataService.AlertHistory.cs","Lite/Services/LocalDataService.Blocking.cs","Lite/Services/LocalDataService.LongQueries.cs","Lite/Services/LocalDataService.PlanCorrection.cs","Lite/Services/LocalDataService.WaitStats.cs","Lite/Services/LocalDataService.WaitingTasks.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-18","head_sha":"ae826023f8efed709f9d67ad1883d46b5d20b2ed","event":"pull_request","changed_files":["Darling/Darling.Tests/AnalysisCoverageLivePostgresTests.cs","Darling/Darling.Tests/AnalyzeWindowEmptyLivePostgresTests.cs","Darling/Darling.Tests/PgFactCollectorTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.Waits.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Lite.Tests/AnalysisCoverageTests.cs","Lite.Tests/AnalysisWindowEmptyTests.cs","Lite.Tests/FactCollectorTests.cs","Lite.Tests/McpMissMessageParityPinTests.cs","Lite.Tests/TestDataSeeder.cs","Lite/Analysis/AnalysisService.cs","Lite/Analysis/DuckDbFactCollector.Waits.cs","Lite/Analysis/DuckDbFactCollector.cs","Lite/Mcp/McpAnalysisTools.cs","PerformanceMonitor.Analysis/AnalysisContext.cs","PerformanceMonitor.Analysis/FactCollectorHelpers.cs"],"failed_classes":["PerformanceMonitorLite.Tests.FactCollectorTests"],"failed_jobs":["build"]} +{"date":"2026-09-18","head_sha":"88289f02d64a72332dd78bedad3099ae913a1452","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpAlertToolsTests.cs","Darling/Darling.Tests/DarlingMcpCustomAlertToolsTests.cs","Darling/Darling.Tests/DarlingMcpCustomViewToolsTests.cs","Darling/Darling.Tests/DarlingMcpServerAdminToolsTests.cs","Darling/Darling.Tests/DarlingMcpToolsTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFindingStore.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpCustomAlertTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpCustomViewTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerAdminTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Lite.Tests/McpMuteReportsWhatItMatchedTests.cs","Lite/Analysis/AnalysisService.cs","Lite/Analysis/FindingStore.cs","Lite/Mcp/McpAnalysisTools.cs","Lite/Mcp/McpInstructions.cs"],"failed_classes":["PerformanceMonitorLite.Tests.FindingStoreTests"],"failed_jobs":["build"]} +{"date":"2026-09-18","head_sha":"250bb914f74e70c43b7ed55438daec146a9a8834","event":"pull_request","changed_files":["Lite.Tests/FactScorerTests.cs","Lite.Tests/ScenarioTests.cs","PerformanceMonitor.Analysis/Baselines/AnomalyThresholds.cs","PerformanceMonitor.Analysis/FactAdvice.cs","PerformanceMonitor.Analysis/FactScorer.cs","PerformanceMonitor.Analysis/RelationshipGraph.cs","deprecated/Dashboard.Tests/FactScorerTests.cs"],"failed_classes":["Darling.Tests.CommentFilterAdoptionTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-18","head_sha":"30496f2e0d867e83bf63313ce89e573918de8ea3","event":"pull_request","changed_files":["Lite.Tests/SlackDetailsSizeTests.cs","Lite.Tests/SlackProseSplitTests.cs","PerformanceMonitor.Notifications/WebhookAlertService.cs"],"failed_classes":["PerformanceMonitorLite.Tests.LiteDeltaSeederTests"],"failed_jobs":["build"]} +{"date":"2026-09-18","head_sha":"8e71bf0f59a90b98f3c7a5761b74f15182614b91","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingDeltaSeederTests.cs","Darling/Darling.Tests/DarlingMcpMemoryGrantToolsTests.cs","Darling/Darling.Tests/DarlingMcpTrendToolsTests.cs","Darling/Darling.Tests/DeltaFamilyIntervalColumnsRungTests.cs","Darling/Darling.Tests/DeltaFamilyIntervalCompletionLivePostgresTests.cs","Darling/Darling.Tests/DeltaFamilyIntervalCompletionRungTests.cs","Darling/Darling.Tests/OversizedPlanBacklogPins.cs","Darling/Darling.Tests/PgTrendReaderTests.cs","Darling/Darling.Tests/StartupCommandTimeoutTests.cs","Darling/Darling.Tests/ViewerHistoryWindowTests.cs","Darling/Darling.Tests/ViewerQueriesRestTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingDeltaCalculator.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpMemoryGrantTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMemoryGrantReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingTrendReader.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingPgTrendReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.ItemHistory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryTrends.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/DeltaFamilyIntervalColumnTests.cs","Lite.Tests/DeltaFamilySeedingCensusTests.cs","Lite.Tests/DeltaFamilyUnknowableRowReadTests.cs","Lite.Tests/DuckDbSchemaEquivalenceTests.cs","Lite.Tests/LiteDeltaSeederTests.cs","Lite.Tests/MemoryGrantsCollectorDefinitionTests.cs","Lite.Tests/PgStatementStatsDeltaSkipTests.cs","Lite.Tests/PgStatementStatsFlavorTests.cs","Lite.Tests/PgWaitStatsCollectorDefinitionTests.cs","Lite.Tests/PgWaitStatsDeltaSkipTests.cs","Lite.Tests/ProcedureStatsCollectorDefinitionTests.cs","Lite.Tests/QueryStatsCollectorDefinitionTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/Mcp/McpMemoryTools.cs","Lite/Services/DeltaCalculator.cs","Lite/Services/LocalDataService.MemoryGrants.cs","Lite/Services/LocalDataService.QueryStats.cs","PerformanceMonitor.Collectors/CollectorDeltaCalculator.cs","PerformanceMonitor.Collectors/MemoryGrantsCollector.cs","PerformanceMonitor.Collectors/PgStatementStatsCollector.cs","PerformanceMonitor.Collectors/PgWaitStatsCollector.cs","PerformanceMonitor.Collectors/ProcedureStatsCollector.cs","PerformanceMonitor.Collectors/QueryStatsCollector.cs"],"failed_classes":["Darling.Tests.DarlingMcpMemoryGrantToolsSurfaceAndSqlTests","Darling.Tests.DarlingPerformanceTrendsReadTests","Darling.Tests.DeltaFamilyIntervalCompletionRungTests","Darling.Tests.OversizedPlanBacklogPins","Darling.Tests.PayloadDimensionTests","Darling.Tests.PgSchemaGeneratorTests","PerformanceMonitorLite.Tests.PerformanceTrendsToolTests"],"failed_jobs":["build","Darling PostgreSQL tests","Darling whole-tree guards"]} +{"date":"2026-09-18","head_sha":"b9d8e955ac7cfd4b42c00f22a782b975e537b014","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingDeltaSeederTests.cs","Darling/Darling.Tests/DarlingMcpMemoryGrantToolsTests.cs","Darling/Darling.Tests/DarlingMcpTrendToolsTests.cs","Darling/Darling.Tests/DarlingPerformanceTrendsReadTests.cs","Darling/Darling.Tests/DeltaFamilyIntervalColumnsRungTests.cs","Darling/Darling.Tests/DeltaFamilyIntervalCompletionLivePostgresTests.cs","Darling/Darling.Tests/DeltaFamilyIntervalCompletionRungTests.cs","Darling/Darling.Tests/OversizedPlanBacklogPins.cs","Darling/Darling.Tests/PayloadDimensionTests.cs","Darling/Darling.Tests/PgTrendReaderTests.cs","Darling/Darling.Tests/StartupCommandTimeoutTests.cs","Darling/Darling.Tests/ViewerHistoryWindowTests.cs","Darling/Darling.Tests/ViewerQueriesRestTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingDeltaCalculator.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpMemoryGrantTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMemoryGrantReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingTrendReader.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingPgTrendReader.cs","Darling/PerformanceMonitor.Darling.Storage/OversizedPlanBacklog.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.ItemHistory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryTrends.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/DeltaFamilyIntervalColumnTests.cs","Lite.Tests/DeltaFamilySeedingCensusTests.cs","Lite.Tests/DeltaFamilyUnknowableRowReadTests.cs","Lite.Tests/DuckDbSchemaEquivalenceTests.cs","Lite.Tests/LiteDeltaSeederTests.cs","Lite.Tests/MemoryGrantsCollectorDefinitionTests.cs","Lite.Tests/PgStatementStatsDeltaSkipTests.cs","Lite.Tests/PgStatementStatsFlavorTests.cs","Lite.Tests/PgWaitStatsCollectorDefinitionTests.cs","Lite.Tests/PgWaitStatsDeltaSkipTests.cs","Lite.Tests/ProcedureStatsCollectorDefinitionTests.cs","Lite.Tests/QueryStatsCollectorDefinitionTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/Mcp/McpMemoryTools.cs","Lite/Services/DeltaCalculator.cs","Lite/Services/LocalDataService.MemoryGrants.cs","Lite/Services/LocalDataService.QueryStats.cs","PerformanceMonitor.Collectors/CollectorDeltaCalculator.cs","PerformanceMonitor.Collectors/MemoryGrantsCollector.cs","PerformanceMonitor.Collectors/PgStatementStatsCollector.cs","PerformanceMonitor.Collectors/PgWaitStatsCollector.cs","PerformanceMonitor.Collectors/ProcedureStatsCollector.cs","PerformanceMonitor.Collectors/QueryStatsCollector.cs"],"failed_classes":["PerformanceMonitorLite.Tests.PerformanceTrendsToolTests"],"failed_jobs":["build"]} +{"date":"2026-09-18","head_sha":"51e6bca179f87fddde1a4058cff217152c1bfa53","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingFleetReaderTests.cs","Darling/Darling.Tests/DarlingPgSessionStatesReaderTests.cs","Darling/Darling.Tests/DeadlockRateBandTests.cs","Darling/Darling.Tests/FleetCardPostgresCpuLivePostgresTests.cs","Darling/Darling.Tests/FleetCardPostgresCpuTests.cs","Darling/Darling.Tests/PgCpuCapacityHeadroomTests.cs","Darling/Darling.Tests/UnmeasuredMetricsAreNotHealthyTests.cs","Darling/Darling.Tests/ViewerFleetRollupTests.cs","Darling/Darling.Tests/ViewerW2aTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAlertTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFleetTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/fleet.js","Darling/PerformanceMonitor.Darling.Storage/DarlingPgSessionStatesReader.cs","Darling/PerformanceMonitor.Darling.Viewer/MainWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Fleet.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Overview.cs","Darling/README.md","PerformanceMonitor.Alerting/PostgresAlertEvaluator.cs","PerformanceMonitor.Common/FleetDeadlockCoverage.cs","PerformanceMonitor.Common/ServerHealthBands.cs"],"failed_classes":["Darling.Tests.DarlingPgReadSqlParsesLiveTests","Darling.Tests.DocCommentHygieneTests","Darling.Tests.ViewerFleetDeadlockCoverageTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-18","head_sha":"d0d482835c7fcb99b9abd7612a2db1179f8f523f","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpHealthToolsTests.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.XminHorizonPersistenceReadTests","Lite.Tests.FindingsRetentionHorizonPinTests","PerformanceMonitorLite.Tests.PerformanceCalendarDataTests"],"failed_jobs":["build","Darling PostgreSQL tests","Darling whole-tree guards"]} +{"date":"2026-09-18","head_sha":"f07bfc441fc2287e1a9829881d02d8b373608dfc","event":"pull_request","changed_files":["Darling/Darling.Tests/DeltaFamilyIntervalCompletionLivePostgresTests.cs","Lite.Tests/DeltaFamilyUnknowableRowReadTests.cs"],"failed_classes":["Darling.Tests.TsqlConventionGuardTests","PerformanceMonitorLite.Tests.McpZeroIsAMeasurementTests","PerformanceMonitorLite.Tests.TrendEmptyParityToolTests"],"failed_jobs":["build","Darling PostgreSQL tests","Darling whole-tree guards"]} +{"date":"2026-09-18","head_sha":"6a2f100f08c6e5955cd17986a32fbd09ecc6207e","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpPgLoggingAuditToolsTests.cs","Darling/Darling.Tests/ServerPageTabsTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgLoggingAuditTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingPgLoggingAudit.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Storage/DarlingPgLoggingAuditReader.cs","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","README.md","docs/postgres-first-target-runbook.md","llms.txt"],"failed_classes":["Darling.Tests.McpLatestSnapshotStampTests","Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-18","head_sha":"8a800ed6c69065e5d03ea6a2b3609e2f562ffce6","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpPgLoggingAuditToolsTests.cs","Darling/Darling.Tests/McpLatestSnapshotStampTests.cs","Darling/Darling.Tests/ServerPageTabsTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgLoggingAuditTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingPgLoggingAudit.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Storage/DarlingPgLoggingAuditReader.cs","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","README.md","docs/postgres-first-target-runbook.md","llms.txt"],"failed_classes":["PerformanceMonitorLite.Tests.LiteDeltaSeederTests"],"failed_jobs":["build"]} +{"date":"2026-09-18","head_sha":"95b65cc63310978dd531c30ba1418b6bf4e91621","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorStateContractTests.cs","Darling/Darling.Tests/PgWaitInstrumentDisclosureTests.cs","Darling/Darling.Tests/PgWaitSamplerLiveTests.cs","Darling/Darling.Tests/SweepBodyDetachPolicyTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingServerConnector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgWaitSamplingTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgWaitTools.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingPgWaitSamplingReader.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Postgres.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Postgres.cs","Darling/README.md","Lite.Tests/PgWaitExclusionParityTests.cs","Lite.Tests/PgWaitSamplerArmTests.cs","Lite.Tests/PgWaitSamplingCollectorDefinitionTests.cs","PerformanceMonitor.Collectors/CollectorEngineCapability.cs","PerformanceMonitor.Collectors/CollectorScheduleDefaults.cs","PerformanceMonitor.Collectors/CollectorTargetInfo.cs","PerformanceMonitor.Collectors/PgWaitInstrument.cs","PerformanceMonitor.Collectors/PgWaitSamplingCollector.cs","docs/postgres-first-target-runbook.md"],"failed_classes":["Darling.Tests.CollectorEngineCapabilityMovingGateTests","Darling.Tests.DarlingCliCommandsTests","Darling.Tests.DarlingComposeTests","Darling.Tests.DocCommentHygieneTests","Lite.Tests.AuroraOnlySqlIsGatedTests","Lite.Tests.DefaultTraceEventsCollectorDefinitionTests"],"failed_jobs":["build","Darling PostgreSQL tests","Darling whole-tree guards"]} +{"date":"2026-09-18","head_sha":"9261cb58aa955a32a96d02ee61ef4842f1abca4e","event":"pull_request","changed_files":[".github/workflows/build.yml",".github/workflows/nightly.yml","Darling/Darling.Tests/CollectorMeasurementSeamTests.cs","Darling/Darling.Tests/ConsumedTimestampFrameDisciplineTests.cs","Darling/Darling.Tests/DarlingPgReadSqlParsesLiveTests.cs","Darling/Darling.Tests/DeltaFamilyIntervalCompletionRungTests.cs","Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Darling/Darling.Tests/PgSchemaGeneratorTests.cs","Darling/Darling.Tests/ServerPageTabsTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgLogEventTools.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogEventIngestor.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Storage/DarlingPgLogEventReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Postgres.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerPostgresTabs.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Postgres.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml","Darling/README.md","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","Lite.Tests/LogTailOverlapThresholdPinTests.cs","Lite.Tests/PgServerLogPathPinTests.cs","PerformanceMonitor.Collectors/CollectorCatalog.cs","PerformanceMonitor.Collectors/CollectorEngineCapability.cs","PerformanceMonitor.Collectors/CollectorScheduleDefaults.cs","PerformanceMonitor.Collectors/IPgLogFamilyParser.cs","PerformanceMonitor.Collectors/PgDeadlocksCollector.cs","PerformanceMonitor.Collectors/PgLogEntry.cs","PerformanceMonitor.Collectors/PgLogEntryAssembler.cs","PerformanceMonitor.Collectors/PgLogEvent.cs","PerformanceMonitor.Collectors/PgLogEventClassifier.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgLogFamilyParsers.cs","PerformanceMonitor.Collectors/PgLogTextRedactor.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgPlanLogParser.cs","PerformanceMonitor.Collectors/PgServerLogTail.cs","docs/postgres-first-target-runbook.md"],"failed_classes":["Darling.Tests.CollectionSweepCommandTimeoutTests","Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.ReadmeDerivedCountPinTests","Darling.Tests.StorageCommandTimeoutTests","Darling.Tests.StoreCopyPhaseTests","Lite.Tests.CrossAppMcpToolInventoryPinTests"],"failed_jobs":["build","Darling PostgreSQL tests","Darling whole-tree guards"]} +{"date":"2026-09-18","head_sha":"87559e0f726889edec0ecaafe49e792058892aae","event":"pull_request","changed_files":["Darling/Darling.Tests/CaptureDownChunkOrderTests.cs","Darling/Darling.Tests/CollectionSignalsChunkOrderTests.cs","Darling/Darling.Tests/IntervalDedupMaterializationIndexesTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-18","head_sha":"b98b7540a345b3cfd19f1de47b689abe8605735a","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorEngineCapabilityDerivationTests.cs","Darling/Darling.Tests/CollectorStateContractTests.cs","Darling/Darling.Tests/DarlingCliCommandsTests.cs","Darling/Darling.Tests/DarlingComposeTests.cs","Darling/Darling.Tests/PgWaitInstrumentDisclosureTests.cs","Darling/Darling.Tests/PgWaitSamplerLiveTests.cs","Darling/Darling.Tests/SweepBodyDetachPolicyTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingServerConnector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgWaitSamplingTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgWaitTools.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingPgWaitSamplingReader.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Postgres.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Postgres.cs","Darling/README.md","Lite.Tests/PgWaitExclusionParityTests.cs","Lite.Tests/PgWaitSamplerArmTests.cs","Lite.Tests/PgWaitSamplingCollectorDefinitionTests.cs","PerformanceMonitor.Collectors/CollectorEngineCapability.cs","PerformanceMonitor.Collectors/CollectorScheduleDefaults.cs","PerformanceMonitor.Collectors/CollectorTargetInfo.cs","PerformanceMonitor.Collectors/PgWaitInstrument.cs","PerformanceMonitor.Collectors/PgWaitSamplingCollector.cs","docs/postgres-first-target-runbook.md"],"failed_classes":["Lite.Tests.AuroraOnlySqlIsGatedTests","Lite.Tests.DefaultTraceEventsCollectorDefinitionTests"],"failed_jobs":["build"]} +{"date":"2026-09-18","head_sha":"a479b4577cabd0ad1e04e8ff2d160e585e48e02d","event":"pull_request","changed_files":["Darling/Darling.Tests/DrillDownDopProvenanceLiveTests.cs","Darling/Darling.Tests/QueryDopProvenanceTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.Queries.cs","Lite.Tests/DrillDownDopProvenanceParityTests.cs","Lite.Tests/DrillDownDopProvenanceTests.cs","Lite.Tests/QueryDopProvenanceTests.cs","Lite/Analysis/DrillDownCollector.Queries.cs","PerformanceMonitor.Common/QueryDopProvenance.cs"],"failed_classes":["Darling.Tests.LiveCleanupConversionRatchetTests","Lite.Tests.CrossAppGuardCiGateTests","Lite.Tests.DrillDownDopProvenanceTests"],"failed_jobs":["Darling PostgreSQL tests","build","Darling whole-tree guards"]} +{"date":"2026-09-18","head_sha":"5ef0af68f595381e671bb76ba770f6a9990204ea","event":"pull_request","changed_files":["Darling/Darling.Tests/DailyHealthBandTests.cs","Darling/Darling.Tests/DeltaFamilyIntervalCompletionLivePostgresTests.cs","Darling/Darling.Tests/ViewerCalendarRetentionPortTests.cs","Darling/Darling.Tests/ViewerQueriesRestTests.cs","Darling/Darling.Tests/ViewerTrendRoutingPortTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingHealthReader.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummaryHorizon.cs","Darling/PerformanceMonitor.Darling.Storage/DurationTrendRouting.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.DailySummary.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Pvs.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryTrends.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.QueryTrends.cs","Lite.Tests/DailyHealthBandTests.cs","Lite/Controls/FinOpsTab.xaml.cs","Lite/Mcp/McpPvsTools.cs","Lite/Services/LocalDataService.DailySummary.cs","Lite/Services/LocalDataService.FinOps.Pvs.cs","PerformanceMonitor.Common/DailyHealthBand.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.ViewerQueriesLivePostgresTests","Darling.Tests.ViewerTrendRoutingPortTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-18","head_sha":"5916fab68e84d2cd1f808f7bbea01db0fe1cf49a","event":"pull_request","changed_files":["Darling/Darling.Tests/ForcePlanBotPolicyTests.cs","Darling/Darling.Tests/PlanForceBotTests.cs","Darling/Darling.Tests/StragglerCommandTimeoutTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingForcePlanTargetStateReader.cs","Darling/PerformanceMonitor.Darling.Service/PgPlanForceActionStore.cs","Darling/PerformanceMonitor.Darling.Service/PlanForceBot.cs","Lite.Tests/ForcePlanTargetStateTests.cs","PerformanceMonitor.Analysis/FactRemediation.cs","PerformanceMonitor.Analysis/ForcePlanBotPolicy.cs"],"failed_classes":["Darling.Tests.TsqlConventionGuardTests","PerformanceMonitorLite.Tests.LiteDeltaSeederTests"],"failed_jobs":["build","Darling PostgreSQL tests","Darling whole-tree guards"]} +{"date":"2026-09-18","head_sha":"d46e8a24f4b5a0a8e3fa4c4e99a85e67dd129577","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedRolesTests.cs","Darling/Darling.Tests/DarlingMcpAlertToolsTests.cs","Darling/Darling.Tests/NotificationRoutesRungTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/PgLogEventMetricsTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertDeliverer.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertSettings.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingFindingAlertSender.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAlertTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/PgNotificationRouteStore.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/NotificationRouteEditDialog.xaml","Darling/PerformanceMonitor.Darling.Viewer/NotificationRouteEditDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/NotificationRoutesWindow.xaml","Darling/PerformanceMonitor.Darling.Viewer/NotificationRoutesWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.NotificationRoutes.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Darling/tools/provision-roles.sql","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","Lite/Mcp/McpAlertTools.cs","PerformanceMonitor.Notifications/AlertContext.cs","PerformanceMonitor.Notifications/AlertFamily.cs","PerformanceMonitor.Notifications/EmailSendCore.cs","PerformanceMonitor.Notifications/IAlertSettings.cs","PerformanceMonitor.Notifications/NotificationRoute.cs","PerformanceMonitor.Notifications/NotificationRouter.cs","PerformanceMonitor.Notifications/WebhookAlertService.cs","README.md","docs/postgres-first-target-runbook.md","llms.txt"],"failed_classes":["Darling.Tests.DarlingWebEndpointsTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-18","head_sha":"7859d25653535bcfaed78bb4e6d8b6081e830983","event":"pull_request","changed_files":["Darling/Darling.Tests/DailyHealthBandTests.cs","Darling/Darling.Tests/DeltaFamilyIntervalCompletionLivePostgresTests.cs","Darling/Darling.Tests/ViewerCalendarRetentionPortTests.cs","Darling/Darling.Tests/ViewerQueriesRestTests.cs","Darling/Darling.Tests/ViewerQueriesTests.cs","Darling/Darling.Tests/ViewerTrendRoutingPortTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingHealthReader.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummaryHorizon.cs","Darling/PerformanceMonitor.Darling.Storage/DurationTrendRouting.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.DailySummary.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Pvs.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryTrends.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.QueryTrends.cs","Lite.Tests/DailyHealthBandTests.cs","Lite/Controls/FinOpsTab.xaml.cs","Lite/Mcp/McpPvsTools.cs","Lite/Services/LocalDataService.DailySummary.cs","Lite/Services/LocalDataService.FinOps.Pvs.cs","PerformanceMonitor.Common/DailyHealthBand.cs"],"failed_classes":["PerformanceMonitorLite.Tests.LiteDeltaSeederTests"],"failed_jobs":["build"]} +{"date":"2026-09-19","head_sha":"0db975aa98f5169df1272262f7ffa8a85d09a067","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetPostureIsolationTests.cs","Darling/Darling.Tests/PgTargetPostureTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Posture.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Posture.cs","PerformanceMonitor.Analysis/PgTargetScorer.Posture.cs"],"failed_classes":["Darling.Tests.PgTargetFactCollectorTests","Darling.Tests.PgTargetPostureTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-19","head_sha":"189380cb25b21bb738f179287500498af60f732c","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetPostureIsolationTests.cs","Darling/Darling.Tests/PgTargetPostureTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Posture.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Posture.cs","PerformanceMonitor.Analysis/PgTargetScorer.Posture.cs"],"failed_classes":["Darling.Tests.PgTargetFactCollectorTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-19","head_sha":"ea96ebcb37fa58734bf639c690cf26b58061d502","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingPgOperationalAlertTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/README.md"],"failed_classes":["Darling.Tests.FleetCardPostgresCpuTests","Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-19","head_sha":"f84624af0cd09e47fb54a1270fd602a7934594cb","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetTempTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetDrillDownCollector.Queries.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Database.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Database.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Temp.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.Query.cs","PerformanceMonitor.Analysis/PgTargetScorer.Config.cs","PerformanceMonitor.Analysis/PgTargetScorer.Database.cs","PerformanceMonitor.Analysis/PgTargetScorer.Temp.cs"],"failed_classes":["Darling.Tests.PgTargetTempTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-19","head_sha":"5971efceff087e63cfc802ee02deb5fdb328a8eb","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetSharedSwitchRoutingTests.cs","Darling/Darling.Tests/PgTargetWaitLiveTests.cs","Darling/Darling.Tests/PgTargetWaitTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Waits.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Wait.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.Memory.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.Saturation.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.Write.cs","PerformanceMonitor.Analysis/PgTargetScorer.Waits.cs"],"failed_classes":["Darling.Tests.PgTargetSessionsTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-19","head_sha":"de7cedee09f986a7f531b55b7028983db0af9ab9","event":"push","changed_files":["Darling/Darling.Tests/ViewerQueriesTests.cs","Darling/Darling.Tests/ViewerTrendRoutingPortTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTrendTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingTrendReader.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreTrendRouting.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryTrends.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.QueryTrends.cs"],"failed_classes":["Darling.Tests.PgTargetSessionsTests","Darling.Tests.PgTargetTempTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-19","head_sha":"8f038a3b8ad423a5f3d0b42ee6a61332fe1a5b0f","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetTempTests.cs"],"failed_classes":["Darling.Tests.PgTargetSessionsTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-19","head_sha":"b7b434b1a1277f0c62107cfd1b0a983094874896","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetSessionsTests.cs"],"failed_classes":["Darling.Tests.PgTargetTempTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-19","head_sha":"cc855a652505ca3a243c65b3662cb16154deff9f","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetSessionsTests.cs","Darling/Darling.Tests/PgTargetSharedSwitchRoutingTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Cpu.cs","PerformanceMonitor.Analysis/Baselines/AnomalyThresholds.cs","PerformanceMonitor.Analysis/Baselines/MetricNames.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Cpu.cs","PerformanceMonitor.Analysis/PgTargetScorer.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetScorer.Cpu.cs"],"failed_classes":["Lite.Tests.AuroraOnlySqlIsGatedTests"],"failed_jobs":["build"]} +{"date":"2026-09-19","head_sha":"f88740972bd0a32119ac76b006a515268c3dad67","event":"push","changed_files":["Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetSessionsTests.cs","Darling/Darling.Tests/PgTargetSharedSwitchRoutingTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Cpu.cs","PerformanceMonitor.Analysis/Baselines/AnomalyThresholds.cs","PerformanceMonitor.Analysis/Baselines/MetricNames.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Cpu.cs","PerformanceMonitor.Analysis/PgTargetScorer.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetScorer.Cpu.cs"],"failed_classes":["Lite.Tests.AuroraOnlySqlIsGatedTests"],"failed_jobs":["build"]} +{"date":"2026-09-19","head_sha":"48855f4ed13c1d5538e7e8f960d0f6729fd91b69","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpMemoryGrantToolsTests.cs","Darling/Darling.Tests/DarlingMcpServerAdminToolsTests.cs","Darling/Darling.Tests/DarlingMcpToolsTests.cs","Darling/Darling.Tests/McpPlanAnalysisEnvelopeTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFindingStore.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAgTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerAdminTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMemoryGrantReader.cs","Lite.Tests/FindingStoreTests.cs","Lite.Tests/McpDescriptionTruthPinTests.cs","Lite.Tests/McpMuteReportsWhatItMatchedTests.cs","Lite/Analysis/AnalysisService.cs","Lite/Analysis/FindingStore.cs","Lite/Mcp/McpAnalysisTools.cs","Lite/Mcp/McpCpuTools.cs","Lite/Mcp/McpInstructions.cs","Lite/Mcp/McpPlanTools.cs","PerformanceMonitor.PlanAnalysis/McpPlanAnalysisFormatter.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-19","head_sha":"6435c010681772587dd7cf77439c3b0961afdb99","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingPgAutovacuumVerdictLiveTests.cs","Darling/Darling.Tests/McpFilterSemanticsLivePostgresTests.cs","Darling/Darling.Tests/McpPageContractTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/PgLogEventMetricsTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpCollectorCostTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHealthTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgAutovacuumTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgIndexUsageTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgSessionStatesTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgTableBloatTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStallProbeTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreMetricsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js","Lite.Tests/McpPageContractTests.cs","Lite/Mcp/McpAnalysisTools.cs","Lite/Mcp/McpHealthTools.cs","PerformanceMonitor.Common/Mcp/McpHelpers.cs"],"failed_classes":["Darling.Tests.DarlingMcpHealthToolsLivePostgresTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-19","head_sha":"c733f086ae7a20170c71d645be6150523cdc598d","event":"pull_request","changed_files":["Darling/Darling.Tests/AnalysisShutdownResidueTests.cs","Darling/Darling.Tests/DarlingAnalysisStoreTests.cs","Darling/Darling.Tests/RecurrenceLabelStoreReadTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFindingStore.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Lite.Tests/FindingStoreRecurrenceReadTests.cs","Lite.Tests/RecurrenceLabelerTests.cs","Lite/Analysis/AnalysisService.cs","Lite/Analysis/FindingStore.cs","Lite/Mcp/McpAnalysisTools.cs","PerformanceMonitor.Analysis/RecurrenceLabeler.cs"],"failed_classes":["PerformanceMonitorLite.Tests.FindingStoreTests"],"failed_jobs":["build"]} +{"date":"2026-09-19","head_sha":"5dd751bb19c8c7d08824b3709080362be0e4eff8","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetIoTests.cs","Darling/Darling.Tests/PgTargetSharedSwitchRoutingTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Io.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.Io.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Io.cs","PerformanceMonitor.Analysis/Baselines/AnomalyThresholds.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Io.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.Io.cs","PerformanceMonitor.Analysis/PgTargetScorer.Io.cs"],"failed_classes":["Darling.Tests.PgTargetWaitTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-19","head_sha":"6f61658ab21bb8aa6c3381bb583ceec45414dc20","event":"pull_request","changed_files":["Darling/Darling.Tests/BaselineSupplyTests.cs","Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetKnobsTests.cs","Darling/Darling.Tests/PgTargetReplicationTests.cs","Darling/Darling.Tests/PgTargetSharedSwitchRoutingTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Replication.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.Replication.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Replication.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummaryHorizon.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Replication.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.Replication.cs","PerformanceMonitor.Analysis/PgTargetScorer.Replication.cs"],"failed_classes":["Darling.Tests.PgTargetWaitTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-19","head_sha":"1acd606201b265f20ea5de097d3f7d374e15c841","event":"push","changed_files":["Darling/Darling.Tests/BaselineSupplyTests.cs","Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetKnobsTests.cs","Darling/Darling.Tests/PgTargetReplicationTests.cs","Darling/Darling.Tests/PgTargetSharedSwitchRoutingTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Replication.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.Replication.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Replication.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummaryHorizon.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Replication.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.Replication.cs","PerformanceMonitor.Analysis/PgTargetScorer.Replication.cs"],"failed_classes":["Darling.Tests.PgTargetWaitTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-19","head_sha":"c3a18e178599f0cb02223b32074ee44168acf5a7","event":"pull_request","changed_files":["Lite.Tests/FindingStoreTests.cs"],"failed_classes":["Darling.Tests.PgTargetWaitTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-19","head_sha":"6981fdf77c19ccb1be712d61e626d21410b7e516","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorTimestampFrameTests.cs","Darling/Darling.Tests/ConsumedTimestampFrameDisciplineTests.cs","Darling/Darling.Tests/PgNumbackendsAndSampledMsRungTests.cs","Darling/Darling.Tests/TimeHonestyRungTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Cpu.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/AgedDatabaseMigrationTests.cs","Lite.Tests/CpuUtilizationCollectorDefinitionTests.cs","Lite.Tests/GoldenCollectorSchema.cs","Lite.Tests/PerfmonCounterTypeTests.cs","Lite.Tests/ServerPropertiesCollectorDefinitionTests.cs","Lite.Tests/TimeHonestyRungTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/Mcp/McpCpuTools.cs","Lite/Mcp/McpServerInfoTools.cs","Lite/Services/LocalDataService.Cpu.cs","Lite/Services/LocalDataService.ServerInfo.cs","PerformanceMonitor.Collectors/CpuUtilizationCollector.cs","PerformanceMonitor.Collectors/ServerPropertiesCollector.cs"],"failed_classes":["Darling.Tests.PgTargetWaitTests","Lite.Tests.ServerEpochTests"],"failed_jobs":["Darling PostgreSQL tests","build","Darling whole-tree guards"]} +{"date":"2026-09-19","head_sha":"a7ecf42814a12b8d60f72e80dedeec7ca0dd4ed2","event":"pull_request","changed_files":["Darling/Darling.Tests/MaterializationHoleRepairTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.MaterializationHoles.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.PgTargetWaitTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-19","head_sha":"3064bb598fd2f708f7d819e2489f9770c85c53ca","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertDeliveryChannelTests.cs","Darling/Darling.Tests/AlertHistoryRowSeverityTests.cs","Darling/Darling.Tests/AlertMetricClassifierTests.cs","Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/AlertTrayStatusLoggedTests.cs","Darling/Darling.Tests/AnalysisSinglesDigestTests.cs","Darling/Darling.Tests/DarlingAlertingTests.cs","Darling/Darling.Tests/DarlingAnalysisPipelineTests.cs","Darling/Darling.Tests/DarlingMcpAlertToolsTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFindingStore.cs","Darling/PerformanceMonitor.Darling.Service/AnalysisSinglesDigestReader.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertSettings.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingFindingAlertSender.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFileLevelAlertSettings.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAlertTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/PgAlertHistoryStore.cs","Darling/PerformanceMonitor.Darling.Service/SelfAlertDeliveryStampStore.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Service/darling.sample.json","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/alerts.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js","Darling/PerformanceMonitor.Darling.Viewer/NotificationRouteEditDialog.xaml.cs","Darling/README.md","Lite.Tests/AlertReadFailureSurfaceTests.cs","Lite.Tests/AlertSeverityTests.cs","Lite.Tests/FindingRoutingTests.cs","Lite.Tests/LiteRecommendationsReaderTests.cs","Lite/Analysis/FindingStore.cs","Lite/Analysis/Recommendations/LiteRecommendationItem.cs","Lite/Analysis/Recommendations/LiteRecommendationsReader.cs","Lite/Analysis/Recommendations/LiteRecommendationsViewModel.cs","Lite/App.xaml.cs","Lite/Controls/RecommendationsTab.xaml","Lite/Mcp/McpAlertTools.cs","Lite/Services/AppAlertSettings.cs","Lite/Services/DuckDbAlertHistoryStore.cs","Lite/Services/EmailAlertService.cs","Lite/Windows/SettingsWindow.xaml","Lite/Windows/SettingsWindow.xaml.cs","PerformanceMonitor.Alerting/AlertReadFailureCounter.cs","PerformanceMonitor.Analysis/AnalysisModels.cs","PerformanceMonitor.Analysis/InferenceEngine.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertContext.cs","PerformanceMonitor.Notifications/AlertDelivery.cs","PerformanceMonitor.Notifications/AlertFamily.cs","PerformanceMonitor.Notifications/AlertSeverity.cs","PerformanceMonitor.Notifications/AnalysisNotificationService.cs","PerformanceMonitor.Notifications/FindingRouting.cs","PerformanceMonitor.Notifications/IAlertHistoryStore.cs","PerformanceMonitor.Notifications/IAlertSettings.cs","PerformanceMonitor.Notifications/IFindingAlertSender.cs","README.md","deprecated/Dashboard/Services/EmailAlertService.cs"],"failed_classes":["Darling.Tests.LiveCleanupConversionRatchetTests","Darling.Tests.PgTargetWaitTests","Lite.Tests.AlertSettingsControlWiringTests","PerformanceMonitorLite.Tests.SettingsSampleTests"],"failed_jobs":["Darling PostgreSQL tests","build","Darling whole-tree guards"]} +{"date":"2026-09-19","head_sha":"d98145b5f748258ab3efba77c78400cac11fa94e","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorTimestampFrameTests.cs","Darling/Darling.Tests/ConsumedTimestampFrameDisciplineTests.cs","Darling/Darling.Tests/PgNumbackendsAndSampledMsRungTests.cs","Darling/Darling.Tests/TimeHonestyRungTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Cpu.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/AgedDatabaseMigrationTests.cs","Lite.Tests/CpuUtilizationCollectorDefinitionTests.cs","Lite.Tests/GoldenCollectorSchema.cs","Lite.Tests/PerfmonCounterTypeTests.cs","Lite.Tests/ServerEpochTests.cs","Lite.Tests/ServerPropertiesCollectorDefinitionTests.cs","Lite.Tests/TimeHonestyRungTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/Mcp/McpCpuTools.cs","Lite/Mcp/McpServerInfoTools.cs","Lite/Services/LocalDataService.Cpu.cs","Lite/Services/LocalDataService.ServerInfo.cs","PerformanceMonitor.Collectors/CpuUtilizationCollector.cs","PerformanceMonitor.Collectors/ServerPropertiesCollector.cs"],"failed_classes":["Darling.Tests.PgTargetReplicationTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-19","head_sha":"1c443a9d29f7085884b6aeebb4ba718df8e8089c","event":"pull_request","changed_files":["Darling/Darling.Tests/FleetCollectionHealthMemoTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFleetTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-19","head_sha":"533c979f43821b83e5cf909077e25a9ce7873202","event":"push","changed_files":["Darling/Darling.Tests/FleetCollectionHealthMemoTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFleetTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-19","head_sha":"d7b4be3582c9052a64ca9fc5cdfe26985c282e4b","event":"pull_request","changed_files":["Darling/Darling.Tests/LocalClockBucketKeyTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs","Lite.Tests/LocalClockBucketKeyTests.cs","Lite.Tests/RobustBaselineDuckDbTests.cs","Lite/Analysis/BaselineProvider.cs","PerformanceMonitor.Analysis/Baselines/BaselineLocalClock.cs"],"failed_classes":["Darling.Tests.FleetIdentifierScrubTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-19","head_sha":"ff29c3211aa2b7539b0c02a9507ef3e2c1ca8450","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingCliCommandsTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCommandExecutor.cs","Darling/PerformanceMonitor.Darling.Service/Program.cs","Darling/README.md"],"failed_classes":["Darling.Tests.DarlingMissingCredentialMessageTests","Darling.Tests.LiveCleanupConversionRatchetTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-19","head_sha":"7d743c64928046f474e39613a3ba88e47b86f5a8","event":"pull_request","changed_files":["Darling/Darling.Tests/AnalysisFactsReadRunsDetectorParityTests.cs","Darling/Darling.Tests/AnalyzeWindowEmptyLivePostgresTests.cs","Darling/Darling.Tests/CollectionCaveatsParityTests.cs","Darling/Darling.Tests/CollectionCaveatsTests.cs","Darling/Darling.Tests/PgTargetCollectionCaveatsLiveTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Lite.Tests/AnalysisWindowEmptyTests.cs","Lite.Tests/AnomalyDetectorTests.cs","Lite.Tests/CollectionCaveatsTests.cs","Lite.Tests/McpMissMessageParityPinTests.cs","Lite/Analysis/AnalysisService.cs","Lite/Analysis/DuckDbFactCollector.cs","Lite/Mcp/McpAnalysisTools.cs","PerformanceMonitor.Analysis/AnalysisContext.cs"],"failed_classes":["Darling.Tests.DarlingMissingCredentialMessageTests","Darling.Tests.LiveCleanupConversionRatchetTests","Darling.Tests.PgTargetCollectionCaveatsLiveTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-19","head_sha":"a755d680a07feb51fe9b669d2ac762441d428bb0","event":"pull_request","changed_files":["Darling/Darling.Tests/BaselineSupplyTests.cs","Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetBetweenWavesV2Tests.cs","Darling/Darling.Tests/PgTargetMeasuredLineageTests.cs","Darling/Darling.Tests/PgTargetSampledWaitLiveTests.cs","Darling/Darling.Tests/PgTargetSampledWaitTests.cs","Darling/Darling.Tests/PgTargetWaitTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.WaitsSampled.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.WaitsSampled.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Waits.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/PgTargetToolRecommendations.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummaryHorizon.cs","PerformanceMonitor.Analysis/AnomalyIncidentReconciler.cs","PerformanceMonitor.Analysis/Baselines/AnomalyThresholds.cs","PerformanceMonitor.Analysis/Baselines/MetricNames.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Wait.cs","PerformanceMonitor.Analysis/PgTargetFactKeys.cs","PerformanceMonitor.Analysis/PgTargetScorer.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetScorer.Waits.cs"],"failed_classes":["Darling.Tests.DarlingMissingCredentialMessageTests","Darling.Tests.LiveCleanupConversionRatchetTests","Darling.Tests.PgTargetCollectionCaveatsLiveTests","Darling.Tests.PgTargetReplicationTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-19","head_sha":"109542618ff1f021383161dc2c3d3129bf69a54f","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionOutputBesideCostTests.cs","Darling/Darling.Tests/SwallowedItemFailureTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/DarlingXeSessions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Lite.Tests/CollectionOutputBesideCostTests.cs","Lite.Tests/EnumeratedCollectorDriverTests.cs","Lite.Tests/SwallowedItemFailureTests.cs","Lite/Mcp/McpHealthTools.cs","Lite/Services/LocalDataService.CollectionHealth.cs","Lite/Services/RemoteCollectorService.DefinitionRunner.cs","Lite/Services/RemoteCollectorService.LongQueryCompletions.cs","PerformanceMonitor.Collectors/EnumeratedCollectorDriver.cs","PerformanceMonitor.Common/ServerHealthBands.cs"],"failed_classes":["Darling.Tests.DarlingMissingCredentialMessageTests","Darling.Tests.LiveCleanupConversionRatchetTests","Darling.Tests.PgTargetCollectionCaveatsLiveTests","Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-19","head_sha":"31dd698957f29ee718566360a70042fac5c5b89b","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetCollectionCaveatsLiveTests.cs"],"failed_classes":["Darling.Tests.DarlingMissingCredentialMessageTests","Darling.Tests.LiveCleanupConversionRatchetTests","Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-19","head_sha":"a0f0f4e0e5b2a4f3de410e7f581dc855a1b8c8c1","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingCliCommandsTests.cs","Darling/Darling.Tests/DarlingCollectorToggleVerbLivePostgresTests.cs","Darling/Darling.Tests/StragglerCommandTimeoutTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-20","head_sha":"d175cf1d707674a1fe11552400573bdfaa483074","event":"pull_request","changed_files":["Darling/Darling.Tests/ConfigChangeTraceAnchorLivePostgresTests.cs","Darling/Darling.Tests/ConfigChangeTraceAnchorTests.cs","Darling/Darling.Tests/ServerLocalReadFrameDisciplineTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Lite.Tests/ConfigChangeAttributionPipelineTests.cs","Lite.Tests/ConfigChangeAttributionTests.cs","Lite/Analysis/AnalysisService.cs","PerformanceMonitor.Analysis/ConfigChangeAttribution.cs","PerformanceMonitor.Analysis/FactAdvice.cs"],"failed_classes":["Darling.Tests.FactSourceRegistryTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-20","head_sha":"c1df3b08c2e005b4bedb39c5633311aaf2aaaa85","event":"pull_request","changed_files":["Darling/Darling.Tests/AnalysisAsOfAnchorTests.cs","Darling/Darling.Tests/AsOfWindowAnchorTests.cs","Darling/Darling.Tests/DarlingDailySummaryRangeTests.cs","Darling/Darling.Tests/DarlingMcpBlockingToolsTests.cs","Darling/Darling.Tests/DarlingMcpDataToolsTests.cs","Darling/Darling.Tests/DarlingMcpDefaultTraceToolsTests.cs","Darling/Darling.Tests/DarlingMcpFleetSweepToolsTests.cs","Darling/Darling.Tests/DarlingMcpHealthParserToolsTests.cs","Darling/Darling.Tests/DarlingMcpHealthToolsTests.cs","Darling/Darling.Tests/DarlingMcpPlanToolsTests.cs","Darling/Darling.Tests/DarlingMcpToolsTests.cs","Darling/Darling.Tests/DarlingMcpTrendToolsTests.cs","Darling/Darling.Tests/DarlingPeerDisclosureTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/FactSourceRegistryTests.cs","Darling/Darling.Tests/FleetSweepWebFeedTests.cs","Darling/Darling.Tests/McpFilterSemanticsLivePostgresTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/OversizedPlanSweepRunRecordTests.cs","Darling/Darling.Tests/ServerPageTabsTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingFleetSweepEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFleetSweepTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgDeadlockTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgLogEventTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgPlanTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgServerStateTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgTrendTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerAdminTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingServerResolver.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js","Lite.Tests/AnalysisAsOfAnchorTests.cs","Lite.Tests/AsOfWindowAnchorTests.cs","Lite.Tests/DailySummaryRangeToolTests.cs","Lite.Tests/McpMissMessageParityPinTests.cs","Lite.Tests/McpPageContractTests.cs","Lite/Mcp/McpAnalysisTools.cs","Lite/Mcp/McpInstructions.cs","Lite/Mcp/McpPlanTools.cs","Lite/Mcp/ServerResolver.cs","PerformanceMonitor.Common/Mcp/McpHelpers.cs"],"failed_classes":["Darling.Tests.PgLogEventsLivePostgresTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-20","head_sha":"c939f460ff5fad35f7f1f85bc266a49f769a0502","event":"push","changed_files":["Darling/Darling.Tests/CollectionLogFanoutRollupStoreTests.cs","Darling/Darling.Tests/CollectorDatabaseScopeRungTests.cs","Darling/Darling.Tests/DarlingEmptyEnumerationInventoryTests.cs","Darling/Darling.Tests/QueryStoreHealthStoreTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Lite.Tests/EmptyEnumerationInventoryTests.cs","Lite.Tests/QueryStoreHealthCollectorDefinitionTests.cs","Lite/Mcp/McpHealthTools.cs","PerformanceMonitor.Collectors/QueryStoreHealthCollector.cs"],"failed_classes":["Darling.Tests.PgLogEventsLivePostgresTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-20","head_sha":"d4c281ca1357c631f71d9414081ef33fe8010310","event":"pull_request","changed_files":["Darling/Darling.Tests/BaselineDiscontinuityRenderTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTrendTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingTrendReader.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Storage/BaselineDiscontinuityReader.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Discontinuities.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.QueryTrends.cs","Lite.Tests/BaselineDiscontinuityDuckDbTests.cs","Lite.Tests/BaselineDiscontinuityRenderCensusTests.cs","Lite.Tests/BaselineDiscontinuityTests.cs","Lite/Controls/ServerTab.Charts.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Mcp/McpIoTools.cs","Lite/Mcp/McpMemoryTools.cs","Lite/Mcp/McpPerfmonTools.cs","Lite/Mcp/McpQueryTools.cs","Lite/Mcp/McpWaitTools.cs","Lite/Services/LocalDataService.Discontinuities.cs","PerformanceMonitor.Collectors/BaselineDiscontinuity.cs","PerformanceMonitor.Collectors/CollectorMeasurement.cs","PerformanceMonitor.Common/ChartPalette.cs","PerformanceMonitor.Ui/ChartStyle.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.ViewerCommandTimeoutTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-20","head_sha":"8e01ff4d1d8af7b3d176ac08dab58d512f7c6747","event":"push","changed_files":["Darling/Darling.Tests/CollectorTimestampFrameTests.cs","Darling/Darling.Tests/DarlingCollectorRunnerTests.cs","Darling/Darling.Tests/TimeHonestyRungTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Lite.Tests/CpuUtilizationCollectorDefinitionTests.cs","Lite.Tests/TimeHonestyRungTests.cs","Lite.Tests/WaitStatsCollectorDefinitionTests.cs","Lite/Services/RemoteCollectorService.DefinitionRunner.cs","Lite/Services/RemoteCollectorService.cs","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/CollectorDefinitionBase.cs","PerformanceMonitor.Collectors/CpuUtilizationCollector.cs","PerformanceMonitor.Collectors/ICollectorDefinition.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.ViewerCommandTimeoutTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-20","head_sha":"b3e0c82b695d6104116e3099b715e2f436cf37ac","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetSessionsTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Sessions.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Sessions.cs","PerformanceMonitor.Analysis/PgTargetScorer.Sessions.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.PgTargetReplicationTests","Darling.Tests.ViewerCommandTimeoutTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-20","head_sha":"5e105b8c26ca1336abff8ec3aee83d840ceee1bb","event":"pull_request","changed_files":[".github/workflows/build.yml",".github/workflows/nightly.yml","Darling/Darling.Tests/LongRunningQueryExclusionKnobRungTests.cs","Darling/Darling.Tests/PerfmonCounterTypeRungTests.cs","Darling/Darling.Tests/PgCpuCapacityHeadroomTests.cs","Darling/Darling.Tests/PgDatabaseSizeStatsAndHostMemoryRungTests.cs","Darling/Darling.Tests/PgNumbackendsAndSampledMsRungTests.cs","Darling/Darling.Tests/PgSchemaGeneratorTests.cs","Darling/Darling.Tests/RdsCpuIngestorTests.cs","Darling/Darling.Tests/ServerPageTabsTests.cs","Darling/Darling.Tests/TimeHonestyRungTests.cs","Darling/Darling.Tests/ViewerCollectorCoverageTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsCpuIngestor.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/PgDatabaseSizeStatsCollectorDefinitionTests.cs","PerformanceMonitor.Collectors/CollectorCatalog.cs","PerformanceMonitor.Collectors/CollectorScheduleDefaults.cs","PerformanceMonitor.Collectors/PgCpuUtilizationCollector.cs","PerformanceMonitor.Collectors/PgDatabaseSizeStatsCollector.cs","docs/postgres-first-target-runbook.md"],"failed_classes":["Darling.Tests.PgLogEventMetricsRungTests","Darling.Tests.PgLogEventsRungTests","Darling.Tests.SerialLoopStoreSizeSourceTests","Darling.Tests.ViewerPostgresTabsTests"],"failed_jobs":["Darling PostgreSQL tests","build","Darling whole-tree guards"]} +{"date":"2026-09-20","head_sha":"c88c3941d5b49f220a0c9a47d927c8ee2882ab9a","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetBufferDrillDownTests.cs","Darling/Darling.Tests/PgTargetKnobsTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetDrillDownCollector.Buffer.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetDrillDownCollector.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/PgTargetToolRecommendations.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Buffer.cs"],"failed_classes":["Darling.Tests.MeasurementContractCensusTests","Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-20","head_sha":"bea6b41fd3539f97316bb7fea9c2a59d593c23d0","event":"push","changed_files":["Darling/Darling.Tests/PgTargetBufferDrillDownTests.cs","Darling/Darling.Tests/PgTargetKnobsTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetDrillDownCollector.Buffer.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetDrillDownCollector.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/PgTargetToolRecommendations.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Buffer.cs"],"failed_classes":["Darling.Tests.MeasurementContractCensusTests","Darling.Tests.PgLogEventMetricsRungTests","Darling.Tests.PgLogEventsRungTests","Darling.Tests.SerialLoopStoreSizeSourceTests","Darling.Tests.TsqlConventionGuardTests","Darling.Tests.ViewerPostgresTabsTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-20","head_sha":"06ec67ccb1efc04b8c3500b41e1f2e3b6153a7ee","event":"pull_request","changed_files":["Darling/Darling.Tests/BaselineSupplyTests.cs","Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetBetweenWavesV2Tests.cs","Darling/Darling.Tests/PgTargetMeasuredLineageTests.cs","Darling/Darling.Tests/PgTargetPlanTests.cs","Darling/Darling.Tests/PgTargetSharedSwitchRoutingTests.cs","Darling/Darling.Tests/PgTargetV3PlumbingTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Plans.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.Plans.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Plans.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummaryHorizon.cs","PerformanceMonitor.Analysis/Baselines/AnomalyThresholds.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Plans.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.Plans.cs","PerformanceMonitor.Analysis/PgTargetScorer.Plans.cs"],"failed_classes":["Darling.Tests.MeasurementContractCensusTests","Darling.Tests.PgLogEventMetricsRungTests","Darling.Tests.PgLogEventsRungTests","Darling.Tests.SerialLoopStoreSizeSourceTests","Darling.Tests.TsqlConventionGuardTests","Darling.Tests.ViewerPostgresTabsTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-20","head_sha":"4b8823b72a2b692f631f59ee37afc6663b6b905d","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetBufferDrillDownTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetDrillDownCollector.Buffer.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Buffer.cs"],"failed_classes":["Darling.Tests.PgLogEventMetricsRungTests","Darling.Tests.PgLogEventsRungTests","Darling.Tests.SerialLoopStoreSizeSourceTests","Darling.Tests.ViewerPostgresTabsTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-20","head_sha":"1a6b308e238e3e987bc24c04becbabff57510a2b","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpAgToolsTests.cs","Darling/Darling.Tests/DarlingMcpTrendToolsTests.cs","Darling/Darling.Tests/ServerPageTabsTests.cs"],"failed_classes":["Darling.Tests.PgLogEventMetricsRungTests","Darling.Tests.PgLogEventsRungTests","Darling.Tests.SerialLoopStoreSizeSourceTests","Darling.Tests.TsqlConventionGuardTests","Darling.Tests.ViewerPostgresTabsTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-20","head_sha":"9d96ad152bef8883ca7628d14af49d47567765c8","event":"pull_request","changed_files":["Darling/Darling.Tests/PgDatabaseSizeStatsAndHostMemoryRungTests.cs","Darling/Darling.Tests/PgLogEventMetricsTests.cs","Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Darling/Darling.Tests/PgRegistryPanelPlacementTests.cs","Darling/Darling.Tests/SerialLoopStoreSizeSourceTests.cs","Darling/Darling.Tests/ViewerPostgresTabsTests.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerPostgresTabs.cs"],"failed_classes":["Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-20","head_sha":"8aca2617bb4673716b0d17b721f5dcc6a19c6eb1","event":"pull_request","changed_files":["Darling/Darling.Tests/BaselineSupplyTests.cs","Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetBetweenWavesV2Tests.cs","Darling/Darling.Tests/PgTargetFactCollectorTests.cs","Darling/Darling.Tests/PgTargetKernelTests.cs","Darling/Darling.Tests/PgTargetSharedSwitchRoutingTests.cs","Darling/Darling.Tests/PgTargetV3PlumbingTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Kernel.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.Kernel.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Kernel.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummaryHorizon.cs","PerformanceMonitor.Analysis/Baselines/AnomalyThresholds.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Kernel.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.Kernel.cs","PerformanceMonitor.Analysis/PgTargetScorer.Kernel.cs"],"failed_classes":["Darling.Tests.PgLogEventMetricsRungTests","Darling.Tests.PgLogEventsRungTests","Darling.Tests.SerialLoopStoreSizeSourceTests","Darling.Tests.TsqlConventionGuardTests","Darling.Tests.ViewerPostgresTabsTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-20","head_sha":"cffafdc423fb4c52210cc3d63931e88e1d653c1d","event":"pull_request","changed_files":["Darling/Darling.Tests/TsqlConventionGuardTests.cs"],"failed_classes":["Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-20","head_sha":"291e15f88bc3571b75e4006b3e239f73f38eb2c5","event":"push","changed_files":["Darling/Darling.Tests/TsqlConventionGuardTests.cs"],"failed_classes":["Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-20","head_sha":"9d82420cda9bff1a3bc500fb94e39e8ae2b71ad1","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetFactCollectorTests.cs","Darling/Darling.Tests/PgTargetKnobsTests.cs","Darling/Darling.Tests/PgTargetMeasuredLineageTests.cs","Darling/Darling.Tests/PgTargetMemoryTests.cs","Darling/Darling.Tests/PgTargetQueriesTests.cs","Darling/Darling.Tests/PgTargetSharedSwitchRoutingTests.cs","Darling/Darling.Tests/PgTargetV3PlumbingTests.cs","Darling/Darling.Tests/PgTargetVacuumLiveTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Memory.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Memory.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.HostMemory.cs","PerformanceMonitor.Analysis/PgTargetScorer.Memory.cs"],"failed_classes":["Darling.Tests.PgDatabaseSizeStatsAndHostMemoryRungTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-20","head_sha":"e247eea3f0b25e7279d7c692ffe0213b822e789c","event":"pull_request","changed_files":["Darling/Darling.Tests/MigrationDataMovingRungCensusPins.cs","Darling/Darling.Tests/PgDatabaseSizeStatsAndHostMemoryRungTests.cs","Darling/Darling.Tests/QsCaptureModeRouteKnobToastRungTests.cs","Darling/Darling.Tests/QueryStoreHealthStoreTests.cs","Darling/Darling.Tests/TimeHonestyRungTests.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Storage/StoreSelfMetrics.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/GoldenCollectorSchema.cs","Lite.Tests/QsCaptureModeRungTests.cs","Lite.Tests/QueryStoreHealthCollectorDefinitionTests.cs","Lite/Database/DuckDbInitializer.cs","PerformanceMonitor.Collectors/QueryStoreHealthCollector.cs"],"failed_classes":["PerformanceMonitorLite.Tests.TimeHonestyRungReadTests"],"failed_jobs":["build"]} +{"date":"2026-09-20","head_sha":"f2e1816f81b68b39752fa4c0c85afe198ee9abf3","event":"push","changed_files":["Darling/Darling.Tests/MigrationDataMovingRungCensusPins.cs","Darling/Darling.Tests/PgDatabaseSizeStatsAndHostMemoryRungTests.cs","Darling/Darling.Tests/QsCaptureModeRouteKnobToastRungTests.cs","Darling/Darling.Tests/QueryStoreHealthStoreTests.cs","Darling/Darling.Tests/TimeHonestyRungTests.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Storage/StoreSelfMetrics.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/GoldenCollectorSchema.cs","Lite.Tests/QsCaptureModeRungTests.cs","Lite.Tests/QueryStoreHealthCollectorDefinitionTests.cs","Lite/Database/DuckDbInitializer.cs","PerformanceMonitor.Collectors/QueryStoreHealthCollector.cs"],"failed_classes":["PerformanceMonitorLite.Tests.TimeHonestyRungReadTests"],"failed_jobs":["build"]} +{"date":"2026-09-20","head_sha":"b84ea4c86ca2634c509b3a6e9bebe2f691974e9b","event":"pull_request","changed_files":["CONTRIBUTING.md","Darling/Darling.Tests/DarlingMcpPlanToolsTests.cs","Darling/Darling.Tests/McpPlanAnalysisEnvelopeTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanTools.cs","Lite.Tests/McpDescriptionTruthPinTests.cs","Lite/Mcp/McpInstructions.cs","Lite/Mcp/McpPlanTools.cs","PerformanceMonitor.Analysis/FactAdvice.cs","PerformanceMonitor.Analysis/FactScorer.cs","PerformanceMonitor.PlanAnalysis/McpPlanAnalysisFormatter.cs","deprecated/Dashboard.Tests/FactScorerTests.cs"],"failed_classes":["PerformanceMonitorLite.Tests.TimeHonestyRungReadTests"],"failed_jobs":["build"]} +{"date":"2026-09-20","head_sha":"4f8499a944bf3a542da30625e84a6c99cdd18bf4","event":"pull_request","changed_files":["Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/QueryStoreClutterLivePostgresTests.cs","Darling/Darling.Tests/QueryStoreClutterTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreClutterTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingQueryStoreClutterReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/QueryStoreClutter.cs","Darling/README.md","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","README.md","llms.txt"],"failed_classes":["Darling.Tests.QsCaptureModeRouteKnobToastRungTests","PerformanceMonitorLite.Tests.TimeHonestyRungReadTests"],"failed_jobs":["build","Darling PostgreSQL tests","Darling whole-tree guards"]} +{"date":"2026-09-20","head_sha":"20215a40ebee87fc3f85010519633909bcdec0dd","event":"push","changed_files":["CONTRIBUTING.md","Darling/Darling.Tests/DarlingMcpPlanToolsTests.cs","Darling/Darling.Tests/McpPlanAnalysisEnvelopeTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanTools.cs","Lite.Tests/McpDescriptionTruthPinTests.cs","Lite/Mcp/McpInstructions.cs","Lite/Mcp/McpPlanTools.cs","PerformanceMonitor.Analysis/FactAdvice.cs","PerformanceMonitor.Analysis/FactScorer.cs","PerformanceMonitor.PlanAnalysis/McpPlanAnalysisFormatter.cs","deprecated/Dashboard.Tests/FactScorerTests.cs"],"failed_classes":["Darling.Tests.QsCaptureModeRouteKnobToastRungTests","PerformanceMonitorLite.Tests.TimeHonestyRungReadTests"],"failed_jobs":["Darling PostgreSQL tests","build","Darling whole-tree guards"]} +{"date":"2026-09-20","head_sha":"4a3f52389e78a2f0b12110625d80c28a4fb290bb","event":"pull_request","changed_files":["Darling/Darling.Tests/QsCaptureModeRouteKnobToastRungTests.cs"],"failed_classes":["PerformanceMonitorLite.Tests.TimeHonestyRungReadTests"],"failed_jobs":["build"]} +{"date":"2026-09-20","head_sha":"35add29023357fdc4f350d96bb6688290a7887c5","event":"pull_request","changed_files":["Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QsCaptureModeRouteKnobToastRungTests.cs","Darling/Darling.Tests/StoreSelfMetricsTests.cs","Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreMetricsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/StoreSelfMetrics.cs","Lite.Tests/AlertSeverityTests.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs","PerformanceMonitor.Notifications/AlertSeverity.cs"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests","Lite.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["Darling PostgreSQL tests","build","Darling whole-tree guards"]} +{"date":"2026-09-20","head_sha":"87a03565941bff9b73529cc875c17fe771bd724e","event":"pull_request","changed_files":["Darling/Darling.Tests/AnalysisSinglesDigestTests.cs","Darling/Darling.Tests/DarlingMcpAlertToolsTests.cs","Darling/Darling.Tests/LongRunningQueryExclusionKnobRungTests.cs","Darling/Darling.Tests/QsCaptureModeRouteKnobToastRungTests.cs","Darling/Darling.Tests/SelfDiskWarnGbFloorRungTests.cs","Darling/Darling.Tests/UncorroboratedRouteStoreKnobTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertSettings.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingAlertReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFileLevelAlertSettings.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAlertTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Service/darling.sample.json","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.AlertSettings.cs","Darling/README.md","Lite.Tests/McpAlertSettingsKeyTests.cs","Lite/Mcp/McpAlertTools.cs"],"failed_classes":["Lite.Tests.McpAlertSettingsKeyTests"],"failed_jobs":["build"]} +{"date":"2026-09-20","head_sha":"f2c1294ae3bc617c55568fbafd3d2de89046ff17","event":"pull_request","changed_files":["Darling/Darling.Tests/BaselineSupplyTests.cs","Darling/Darling.Tests/CollectionCaveatsTests.cs","Darling/Darling.Tests/FactSourceRegistryTests.cs","Darling/Darling.Tests/PgDatabaseSizeStatsAndHostMemoryRungTests.cs","Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetBetweenWavesV2Tests.cs","Darling/Darling.Tests/PgTargetCollectionCaveatsLiveTests.cs","Darling/Darling.Tests/PgTargetFactCollectorTests.cs","Darling/Darling.Tests/PgTargetGrowthLiveTests.cs","Darling/Darling.Tests/PgTargetGrowthTests.cs","Darling/Darling.Tests/PgTargetMeasuredLineageTests.cs","Darling/Darling.Tests/PgTargetSharedSwitchRoutingTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Growth.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.Growth.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Growth.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/PgTargetToolRecommendations.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummaryHorizon.cs","Lite/Mcp/McpAnalysisTools.cs","PerformanceMonitor.Analysis/Baselines/AnomalyThresholds.cs","PerformanceMonitor.Analysis/Baselines/MetricNames.cs","PerformanceMonitor.Analysis/FactScorer.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Growth.cs","PerformanceMonitor.Analysis/PgTargetAdvice.cs","PerformanceMonitor.Analysis/PgTargetFactKeys.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.Growth.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.cs","PerformanceMonitor.Analysis/PgTargetScorer.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetScorer.Growth.cs","PerformanceMonitor.Analysis/PgTargetScorer.cs"],"failed_classes":["Darling.Tests.PgTargetGrowthTests","Darling.Tests.SerialLoopStoreSizeSourceTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-20","head_sha":"129a9265845265a1e02f9632e3f092da0e0606e3","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QsCaptureModeRouteKnobToastRungTests.cs","Darling/Darling.Tests/StoreSelfMetricsTests.cs","Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreMetricsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/StoreSelfMetrics.cs","Lite.Tests/AlertSeverityTests.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs","PerformanceMonitor.Notifications/AlertSeverity.cs"],"failed_classes":["Lite.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["build"]} +{"date":"2026-09-20","head_sha":"f19a97d0411b2ca912a067092de521f2364032ed","event":"push","changed_files":["Darling/Darling.Tests/DarlingMcpPgPercentDenominatorTests.cs","Darling/Darling.Tests/DarlingPgDatabaseReaderTests.cs","Darling/Darling.Tests/PgAutovacuumDisabledSpellingLiveTests.cs","Darling/Darling.Tests/PgDatabaseSizeStatsAndHostMemoryRungTests.cs","Darling/Darling.Tests/PgNumbackendsAndSampledMsRungTests.cs","Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetBetweenWavesV3Tests.cs","Darling/Darling.Tests/PgTargetBloatTests.cs","Darling/Darling.Tests/PgTargetKernelTests.cs","Darling/Darling.Tests/PgTargetMemoryTests.cs","Darling/Darling.Tests/PgTargetReplicationTests.cs","Darling/Darling.Tests/PgTargetSampledWaitTests.cs","Darling/Darling.Tests/PgTargetSharedSwitchRoutingTests.cs","Darling/Darling.Tests/PgTargetTempTests.cs","Darling/Darling.Tests/PgTargetV3PlumbingTests.cs","Darling/Darling.Tests/PgTargetVacuumLiveTests.cs","Darling/Darling.Tests/PgTargetVacuumTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Plans.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.Plans.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgCpuUtilizationTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgDatabaseTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Storage/DarlingPgCpuUtilizationReader.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingPgDatabaseReader.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerPostgresDisplay.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml","Darling/README.md","Lite.Tests/PgAutovacuumStatsCollectorDefinitionTests.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Bloat.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Vacuum.cs","PerformanceMonitor.Analysis/PgTargetFactKeys.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.Query.cs","PerformanceMonitor.Analysis/PgTargetScorer.Anomaly.cs","PerformanceMonitor.Collectors/PgAutovacuumStatsCollector.cs","docs/postgres-first-target-runbook.md"],"failed_classes":["Lite.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["build"]} +{"date":"2026-09-21","head_sha":"477e9ac503c870b827f8c4f22fe11b4fb880f878","event":"pull_request","changed_files":["Darling/Darling.Tests/AnalysisSinglesDigestTests.cs","Darling/Darling.Tests/DarlingMcpAlertToolsTests.cs","Darling/Darling.Tests/LongRunningQueryExclusionKnobRungTests.cs","Darling/Darling.Tests/SelfDiskWarnGbFloorRungTests.cs","Darling/Darling.Tests/UncorroboratedRouteStoreKnobTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertSettings.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingAlertReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFileLevelAlertSettings.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAlertTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Service/darling.sample.json","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.AlertSettings.cs","Darling/README.md","Lite.Tests/McpAlertSettingsKeyTests.cs","Lite/Mcp/McpAlertTools.cs"],"failed_classes":["Lite.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["build"]} +{"date":"2026-09-21","head_sha":"5c2c7289820c4e4f7ed34296c3f185f69409c21d","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetMeasuredLineageTests.cs","Darling/Darling.Tests/PgTargetQueriesTests.cs","Darling/Darling.Tests/PgTargetSeqScanTests.cs","Darling/Darling.Tests/PgTargetSharedSwitchRoutingTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Queries.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/PgTargetToolRecommendations.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Queries.cs","PerformanceMonitor.Analysis/PgTargetFactKeys.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.Query.cs","PerformanceMonitor.Analysis/PgTargetScorer.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetScorer.Queries.cs"],"failed_classes":["Lite.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["build"]} +{"date":"2026-09-21","head_sha":"948bde86693cbbcd3c1768fbab37a95e9c0c51c5","event":"pull_request","changed_files":["Darling/Darling.Tests/BaselineSupplyTests.cs","Darling/Darling.Tests/CollectionCaveatsTests.cs","Darling/Darling.Tests/FactSourceRegistryTests.cs","Darling/Darling.Tests/PgDatabaseSizeStatsAndHostMemoryRungTests.cs","Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetBetweenWavesV2Tests.cs","Darling/Darling.Tests/PgTargetCollectionCaveatsLiveTests.cs","Darling/Darling.Tests/PgTargetFactCollectorTests.cs","Darling/Darling.Tests/PgTargetGrowthLiveTests.cs","Darling/Darling.Tests/PgTargetGrowthTests.cs","Darling/Darling.Tests/PgTargetMeasuredLineageTests.cs","Darling/Darling.Tests/PgTargetSharedSwitchRoutingTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Growth.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.Growth.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Growth.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/PgTargetToolRecommendations.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummaryHorizon.cs","Lite/Mcp/McpAnalysisTools.cs","PerformanceMonitor.Analysis/Baselines/AnomalyThresholds.cs","PerformanceMonitor.Analysis/Baselines/MetricNames.cs","PerformanceMonitor.Analysis/FactScorer.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Growth.cs","PerformanceMonitor.Analysis/PgTargetAdvice.cs","PerformanceMonitor.Analysis/PgTargetFactKeys.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.Growth.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.cs","PerformanceMonitor.Analysis/PgTargetScorer.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetScorer.Growth.cs","PerformanceMonitor.Analysis/PgTargetScorer.cs"],"failed_classes":["Darling.Tests.PgTargetAnomalyTests","Lite.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["Darling PostgreSQL tests","build","Darling whole-tree guards"]} +{"date":"2026-09-21","head_sha":"b067397d9de9c2c8894a7d298ae443225929bdb4","event":"push","changed_files":["Darling/Darling.Tests/BaselineSupplyTests.cs","Darling/Darling.Tests/CollectionCaveatsTests.cs","Darling/Darling.Tests/FactSourceRegistryTests.cs","Darling/Darling.Tests/PgDatabaseSizeStatsAndHostMemoryRungTests.cs","Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetBetweenWavesV2Tests.cs","Darling/Darling.Tests/PgTargetCollectionCaveatsLiveTests.cs","Darling/Darling.Tests/PgTargetFactCollectorTests.cs","Darling/Darling.Tests/PgTargetGrowthLiveTests.cs","Darling/Darling.Tests/PgTargetGrowthTests.cs","Darling/Darling.Tests/PgTargetMeasuredLineageTests.cs","Darling/Darling.Tests/PgTargetSharedSwitchRoutingTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Growth.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.Growth.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Growth.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/PgTargetToolRecommendations.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummaryHorizon.cs","Lite/Mcp/McpAnalysisTools.cs","PerformanceMonitor.Analysis/Baselines/AnomalyThresholds.cs","PerformanceMonitor.Analysis/Baselines/MetricNames.cs","PerformanceMonitor.Analysis/FactScorer.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Growth.cs","PerformanceMonitor.Analysis/PgTargetAdvice.cs","PerformanceMonitor.Analysis/PgTargetFactKeys.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.Growth.cs","PerformanceMonitor.Analysis/PgTargetRelationshipGraph.cs","PerformanceMonitor.Analysis/PgTargetScorer.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetScorer.Growth.cs","PerformanceMonitor.Analysis/PgTargetScorer.cs"],"failed_classes":["Darling.Tests.PgTargetAnomalyTests","Lite.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["Darling PostgreSQL tests","build","Darling whole-tree guards"]} +{"date":"2026-09-21","head_sha":"c9b95b611cf73d1b36631a0b623e12fd88e2dc6b","event":"pull_request","changed_files":["Darling/Darling.Tests/PgStatementStatsInfoGateLiveTests.cs","Darling/Darling.Tests/PostgresFaultOutcomeTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingServerConnector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Lite.Tests/PgStatementStatsFlavorTests.cs","PerformanceMonitor.Collectors/PgExtensionDependency.cs","PerformanceMonitor.Collectors/PgStatementStatsCollector.cs"],"failed_classes":["Darling.Tests.DatabaseMismatchTripwireTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-21","head_sha":"92be623256b979e1fb3910ef375c6254beb702b2","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionOutputBesideCostTests.cs","Darling/Darling.Tests/FleetCollectionHealthMemoTests.cs","Darling/Darling.Tests/RegressedFromProductiveTests.cs","Darling/Darling.Tests/UnmeasuredMetricsAreNotHealthyTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs","Lite.Tests/RegressedFromProductiveTests.cs","Lite/Mcp/McpHealthTools.cs","Lite/Services/LocalDataService.CollectionHealth.cs","PerformanceMonitor.Collectors/CollectorRuntimePrecondition.cs","PerformanceMonitor.Common/ServerHealthBands.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.RepoFileAdoptionTests","Darling.Tests.StoreToastAndCheckpointerLivePostgresTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-21","head_sha":"113ccbc9eb8019e534a8c97a826890e4a3dae18b","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreClutterLivePostgresTests.cs","Darling/Darling.Tests/QueryStoreClutterTests.cs","Darling/Darling.Tests/QueryStoreClutterViewerSurfacesTests.cs","Darling/Darling.Tests/packages.lock.json","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreClutterTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Storage/DarlingQueryStoreClutterReader.cs","Darling/PerformanceMonitor.Darling.Storage/PerformanceMonitor.Darling.Storage.csproj","Darling/PerformanceMonitor.Darling.Storage/QueryStoreClutter.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStoreClutter.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Filters.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Queries.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.QueryStoreClutter.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml","Darling/PerformanceMonitor.Darling.Viewer/packages.lock.json","Darling/README.md"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.QueryStoreClutterTests","Darling.Tests.RepoFileAdoptionTests","Darling.Tests.StoreToastAndCheckpointerLivePostgresTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-21","head_sha":"c951ca03081e8f49c0d4f59c4d1967f055a557df","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetBloatTests.cs","Darling/Darling.Tests/PgTargetReplicationTests.cs","Darling/Darling.Tests/PgTargetVacuumLiveTests.cs","Darling/Darling.Tests/PgTargetVacuumTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Vacuum.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Vacuum.cs","PerformanceMonitor.Analysis/PgTargetScorer.Vacuum.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-21","head_sha":"c8bb17645f917253c7540a4d0cb22006256e25d1","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/AlertStoredValueTests.cs","Darling/Darling.Tests/CompressionStuckConfirmReadTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/PolicyJobFamilyTests.cs","Darling/Darling.Tests/TimescaleSupportTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","Darling/README.md","Lite.Tests/AlertHistoryValueFormatTests.cs","Lite.Tests/AlertSeverityTests.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs","PerformanceMonitor.Notifications/AlertSeverity.cs","docs/retention-hold-runbook.md"],"failed_classes":["Darling.Tests.AlertMasterSwitchSurfaceTests","Darling.Tests.DocCommentHygieneTests","Darling.Tests.RetentionHoldRatioKnobRungTests","PerformanceMonitorLite.Tests.AlertSeverityTests"],"failed_jobs":["build","Darling PostgreSQL tests","Darling whole-tree guards"]} +{"date":"2026-09-21","head_sha":"55b2d254e5078ae2a5d963d816dd8a345d99be2c","event":"pull_request","changed_files":["Darling/Darling.Tests/RetentionHoldRatioKnobRungTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs"],"failed_classes":["Darling.Tests.StoreToastAndCheckpointerLivePostgresTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-21","head_sha":"cd34bbcb318f9378a980ca707cba20b3bf58da71","event":"pull_request","changed_files":["Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs"],"failed_classes":["PerformanceMonitorLite.Tests.StatusBarSizeReadLockTests"],"failed_jobs":["build"]} +{"date":"2026-09-21","head_sha":"914403e3b7d0d351270e592f46ad9b597691af96","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Analysis/PgFindingStore.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Lite.Tests/StorySideLeafTests.cs","Lite/Analysis/FindingStore.cs","Lite/Mcp/McpAnalysisTools.cs","PerformanceMonitor.Analysis/AnalysisModels.cs","PerformanceMonitor.Analysis/FactAdvice.cs","PerformanceMonitor.Analysis/InferenceEngine.cs","PerformanceMonitor.Analysis/StorySideLeaves.cs"],"failed_classes":["Darling.Tests.CollectionCaveatsParityTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-21","head_sha":"63c4632be69cda4d374ca7cf3b129bc5f81244ad","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingTriageEndpointTests.cs","Darling/Darling.Tests/packages.lock.json","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/PerformanceMonitor.Darling.Service.csproj"],"failed_classes":["Darling.Tests.RetentionHoldRatioKnobRungTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-21","head_sha":"965fdfca847fbe3a53f4e0cb5b528d30c36b00e9","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingTriageEndpointTests.cs","Darling/Darling.Tests/packages.lock.json","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/PerformanceMonitor.Darling.Service.csproj"],"failed_classes":["Darling.Tests.RetentionHoldRatioKnobRungTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-21","head_sha":"d03f75494ff29b0c436ca763532ab640abf60177","event":"pull_request","changed_files":["Darling/Darling.Tests/StoreObjectConvergenceTests.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.CompressionEnableGuardLiveTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-22","head_sha":"6200b3bbe2a0ed5d0ef45217d174462ac543269f","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/AlertReadRetrySeamTests.cs","Darling/Darling.Tests/CaptureDownChunkOrderTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","PerformanceMonitor.Alerting/AlertReadFailureCounter.cs"],"failed_classes":["Darling.Tests.StartupCommandTimeoutTests","Darling.Tests.TimeHonestyRungTests"],"failed_jobs":["Darling PostgreSQL tests","build"]} +{"date":"2026-09-22","head_sha":"bbeb6db0e4711c3b18954d49e19932b13b8e2339","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionLogStatusFilterTests.cs","Darling/Darling.Tests/DarlingCollectionLogReadTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Lite/Mcp/McpHealthTools.cs","Lite/Mcp/McpInstructions.cs","Lite/Services/LocalDataService.CollectionHealth.cs","PerformanceMonitor.Collectors/EnumeratedCollectorDriver.cs","PerformanceMonitor.Common/Mcp/McpHelpers.cs"],"failed_classes":["Darling.Tests.DarlingCollectionLogReadTests"],"failed_jobs":["Darling PostgreSQL tests"]} +{"date":"2026-09-22","head_sha":"5b75c5facd0a15d4a4f961fe8a3bdf5522a18439","event":"pull_request","changed_files":["Darling/Darling.Tests/PgServerConfigOverrideLivePostgresTests.cs","Darling/Darling.Tests/PgServerConfigScopeRungTests.cs","Darling/Darling.Tests/QsCaptureModeRouteKnobToastRungTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.Clock.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Blocking.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Config.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Memory.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Posture.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgServerStateTools.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingPgLoggingAuditReader.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingPgServerConfigReader.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingPgTrendReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","PerformanceMonitor.Collectors/PgServerConfigCollector.cs"],"failed_classes":["Darling.Tests.PgServerConfigScopeRungTests"],"failed_jobs":["build","Darling PostgreSQL tests"]} +{"date":"2026-09-22","head_sha":"d3e2cfc9c915e124c09fcdf77a96e0418fdc6853","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorMeasurementSeamTests.cs","Darling/Darling.Tests/JobHistoryIdentityEpochTests.cs","PerformanceMonitor.Collectors/JobHistoryCollector.cs"],"failed_classes":["PerformanceMonitorLite.Tests.MeasurementContractCensusTests"],"failed_jobs":["build"]} +{"date":"2026-09-22","head_sha":"3200d274e802ebf3aed589282463748b4c765fc0","event":"pull_request","changed_files":[".github/workflows/build.yml","Lite.Tests/CrossAppGuardCiGateTests.cs"],"failed_classes":["Lite.Tests.WatermarkPolicyTests"],"failed_jobs":["Lite tests (3)"]} +{"date":"2026-09-22","head_sha":"d81ac503b46e7ec59a9f9c3f5fb9a201c250066b","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionHealthAggregateTests.cs","Darling/Darling.Tests/FleetCollectionHealthMemoTests.cs","Darling/Darling.Tests/IntervalHonestHourlyRollupTests.cs","Darling/Darling.Tests/TimescaleAggregateCompressionTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/Darling.Tests/TimescaleSupportTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","PerformanceMonitor.Collectors/CollectorRuntimePrecondition.cs","PerformanceMonitor.Collectors/EnumeratedCollectorDriver.cs"],"failed_classes":["Darling.Tests.IntervalHonestHourlyRollupLiveTests","Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.RetentionReevaluationLiveTests","Darling.Tests.TimescaleAggregateCompressionTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (2)","Darling PG tests (0)"]} +{"date":"2026-09-22","head_sha":"2b2e634561cbdcd94aa80a013d0f19e4e52670a9","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionHealthAggregateTests.cs","Darling/Darling.Tests/FleetCollectionHealthMemoTests.cs","Darling/Darling.Tests/IntervalHonestHourlyRollupTests.cs","Darling/Darling.Tests/StoreObjectConvergenceTests.cs","Darling/Darling.Tests/TimescaleAggregateCompressionTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/Darling.Tests/TimescaleSupportTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","PerformanceMonitor.Collectors/CollectorRuntimePrecondition.cs","PerformanceMonitor.Collectors/EnumeratedCollectorDriver.cs"],"failed_classes":["Darling.Tests.IntervalHonestHourlyRollupLiveTests","Darling.Tests.RetentionReevaluationLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-22","head_sha":"ae851b1a26f31e1c93508a756b066d479ee9bd28","event":"pull_request","changed_files":["Darling/Darling.Tests/HoldEarnedByDeliveryTests.cs"],"failed_classes":["PerformanceMonitorLite.Tests.AlertDeliveryModeThreadingGuardTests"],"failed_jobs":["Lite tests (1)"]} +{"date":"2026-09-22","head_sha":"8ae98e8094848169747dba32a0587c3973eb355d","event":"pull_request","changed_files":["Lite.Tests/FindingStoreTests.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-22","head_sha":"d99e7f9f94b8b1f211debfaaa61747be494b3df1","event":"pull_request","changed_files":["Lite.Tests/FindingStoreTests.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-22","head_sha":"9f4761741f9976124b8890709f0490a75380415d","event":"push","changed_files":["Darling/Darling.Tests/ComposeStatementTimeoutReloadTests.cs","Darling/Darling.Tests/ComposeStatementTimeoutStoreTests.cs","Darling/Darling.Tests/ControlPlaneReloadDurabilityTests.cs","Darling/Darling.Tests/DarlingComposeTests.cs","Darling/Darling.Tests/DarlingManagedRolesTests.cs","Darling/Darling.Tests/NotificationRoutesRungTests.cs","Darling/Darling.Tests/ProvisioningTestSecrets.cs","Darling/Darling.Tests/ScramSha256VerifierTests.cs","Darling/Darling.Tests/ScramVerifierLiveTests.cs","Darling/Darling.Tests/StartupCommandTimeoutTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/ScramSha256Verifier.cs","Darling/PerformanceMonitor.Darling.Storage/StoreStatementStats.cs","Darling/README.md","Darling/tools/provision-roles.sql"],"failed_classes":["Darling.Tests.McpFilterSemanticsLivePostgresTests","PerformanceMonitorLite.Tests.McpPageContractTests"],"failed_jobs":["Lite tests (3)","Darling PG tests (2)"]} +{"date":"2026-09-22","head_sha":"1792370ebd4f2d9a5a7874b3410fee69ab599d96","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpStoreMetricsToolsTests.cs","Darling/Darling.Tests/StoreMetricsSummaryFirstTests.cs","Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreMetricsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs"],"failed_classes":["PerformanceMonitorLite.Tests.McpPageContractTests"],"failed_jobs":["Lite tests (3)"]} +{"date":"2026-09-23","head_sha":"7a98743705b31928a967d72f2b88c91a10a6b431","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingPgRuntimeVersionPinTests.cs","Darling/Darling.Tests/DarlingStoreUpgradeTests.cs","Darling/Darling.Tests/StartupCommandTimeoutTests.cs","Darling/Darling.Tests/StoreTimescaleAlertTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreUpgrade.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/README.md","Darling/tools/check-runtime-currency.ps1","Darling/tools/fetch-pg-runtime.ps1"],"failed_classes":["Darling.Tests.CaptureDownChunkOrderTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-23","head_sha":"a498a4ad8e1dc179dd73a28e020bf461f9c10688","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingPgRuntimeVersionPinTests.cs","Darling/Darling.Tests/DarlingStoreUpgradeTests.cs","Darling/Darling.Tests/StartupCommandTimeoutTests.cs","Darling/Darling.Tests/StoreTimescaleAlertTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreUpgrade.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/README.md","Darling/tools/check-runtime-currency.ps1","Darling/tools/fetch-pg-runtime.ps1"],"failed_classes":["Darling.Tests.CaptureDownChunkOrderTests","Darling.Tests.CollectionHealthAggregateTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-23","head_sha":"1e52a8e34fc2dfddaed20c1ac718951efdacb818","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingLockWaitTrendTests.cs","Darling/Darling.Tests/DarlingMcpBlockingToolsTests.cs","Darling/Darling.Tests/DarlingMcpTrendToolsTests.cs","Darling/Darling.Tests/DarlingPerformanceTrendsReadTests.cs","Darling/Darling.Tests/DarlingQueryHeatmapTests.cs","Darling/Darling.Tests/DarlingTrendEmptyTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/DeltaFamilyIntervalCompletionLivePostgresTests.cs","Darling/Darling.Tests/IntervalHonestHourlyRollupTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpZeroIsAMeasurementTests.cs","Darling/Darling.Tests/TrendBucketingLiveTests.cs","Darling/Darling.Tests/TrendBucketsTests.cs","Darling/Darling.Tests/TrendPayloadBudgetLiveTests.cs","Darling/Darling.Tests/ViewerTrendRoutingPortTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingBlockingTrendReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpBlockingTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgTrendTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryHeatmapTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTrendTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingTrendReader.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Storage/DarlingPgTrendReader.cs","Darling/PerformanceMonitor.Darling.Storage/DurationTrendRouting.cs","Darling/PerformanceMonitor.Darling.Storage/TrendBucketSql.cs","Darling/README.md","Lite.Tests/LockWaitTrendToolTests.cs","Lite.Tests/McpMissMessageParityPinTests.cs","Lite.Tests/PerformanceTrendsToolTests.cs","Lite.Tests/QueryHeatmapToolTests.cs","Lite.Tests/TrendBucketingToolTests.cs","Lite.Tests/TrendEmptyParityToolTests.cs","Lite/Mcp/McpBlockingTools.cs","Lite/Mcp/McpInstructions.cs","Lite/Mcp/McpIoTools.cs","Lite/Mcp/McpQueryTools.cs","Lite/Services/LocalDataService.QueryStats.cs","Lite/Services/LocalDataService.TrendBuckets.cs","PerformanceMonitor.Common/Mcp/TrendBuckets.cs","PerformanceMonitor.Common/Mcp/TrendPayloads.cs","docs/postgres-first-target-runbook.md"],"failed_classes":["Darling.Tests.LatestValueLookbackLivePostgresTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-23","head_sha":"28c0765ab125cebd227815ace7599970e8dd77b6","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingLockWaitTrendTests.cs","Darling/Darling.Tests/DarlingMcpBlockingToolsTests.cs","Darling/Darling.Tests/DarlingMcpTrendToolsTests.cs","Darling/Darling.Tests/DarlingPerformanceTrendsReadTests.cs","Darling/Darling.Tests/DarlingQueryHeatmapTests.cs","Darling/Darling.Tests/DarlingTrendEmptyTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/DeltaFamilyIntervalCompletionLivePostgresTests.cs","Darling/Darling.Tests/IntervalHonestHourlyRollupTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpZeroIsAMeasurementTests.cs","Darling/Darling.Tests/TrendBucketingLiveTests.cs","Darling/Darling.Tests/TrendBucketsTests.cs","Darling/Darling.Tests/TrendPayloadBudgetLiveTests.cs","Darling/Darling.Tests/ViewerTrendRoutingPortTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingBlockingTrendReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpBlockingTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgTrendTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryHeatmapTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTrendTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingTrendReader.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Storage/DarlingPgTrendReader.cs","Darling/PerformanceMonitor.Darling.Storage/DurationTrendRouting.cs","Darling/PerformanceMonitor.Darling.Storage/TrendBucketSql.cs","Darling/README.md","Lite.Tests/LockWaitTrendToolTests.cs","Lite.Tests/McpMissMessageParityPinTests.cs","Lite.Tests/PerformanceTrendsToolTests.cs","Lite.Tests/QueryHeatmapToolTests.cs","Lite.Tests/TrendBucketingToolTests.cs","Lite.Tests/TrendEmptyParityToolTests.cs","Lite/Mcp/McpBlockingTools.cs","Lite/Mcp/McpInstructions.cs","Lite/Mcp/McpIoTools.cs","Lite/Mcp/McpQueryTools.cs","Lite/Services/LocalDataService.QueryStats.cs","Lite/Services/LocalDataService.TrendBuckets.cs","PerformanceMonitor.Common/Mcp/TrendBuckets.cs","PerformanceMonitor.Common/Mcp/TrendPayloads.cs","docs/postgres-first-target-runbook.md"],"failed_classes":["Darling.Tests.LatestValueLookbackLivePostgresTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-23","head_sha":"eb4c4173c312575d7a8b99912eb7f878427aebf3","event":"pull_request","changed_files":["Darling/Darling.Tests/MaterializationHoleRepairTests.cs","Darling/Darling.Tests/MaterializationHoleScanShapeTests.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.MaterializationHoles.cs"],"failed_classes":["Darling.Tests.CollectionHealthAggregateTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-23","head_sha":"19e4ce66f252c454e14bf2562fb9bf16f0fc1751","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpStoreMetricsToolsTests.cs","Darling/Darling.Tests/PgTableTuningTests.cs","Darling/Darling.Tests/StoreMetricsLatestSkipScanLiveTests.cs","Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgTableTuning.cs"],"failed_classes":["Darling.Tests.StoreMetricsSummaryFirstLivePostgresTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-23","head_sha":"145ae76294a6c11c49715f71887fa116b02ea8c6","event":"push","changed_files":["Darling/Darling.Tests/LatestValueLookbackTests.cs","Darling/Darling.Tests/StoreConfigProviderTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.Config.cs","Darling/PerformanceMonitor.Darling.Analysis/PgLatestValueBounds.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Lite.Tests/LatestValueLookbackTests.cs","Lite.Tests/OnLoadRecaptureScheduleTests.cs","Lite/Analysis/DuckDbFactCollector.Config.cs","Lite/Analysis/LatestValueBounds.cs","Lite/Services/ScheduleManager.cs","PerformanceMonitor.Collectors/CollectorScheduleDefaults.cs"],"failed_classes":["PerformanceMonitorLite.Tests.EntraDeviceCodeTests"],"failed_jobs":["Lite tests (1)"]} +{"date":"2026-09-23","head_sha":"138ddd4992ac3b6d94d45f990e18d5f95f241672","event":"pull_request","changed_files":["Darling/Darling.Tests/PgDeadlockLogTimezoneTests.cs","Darling/Darling.Tests/PgDeadlockNormalizationTests.cs","Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Darling/Darling.Tests/PgTargetDeadlockDrillDownTests.cs","Darling/Darling.Tests/RdsDeadlockIngestorTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFindingStore.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetDrillDownCollector.Deadlocks.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgDeadlockTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Storage/DarlingPgDeadlockReader.cs","Darling/README.md","Lite.Tests/PgDeadlockLogParserTests.cs","Lite.Tests/PgLoggingCollectorGateTests.cs","PerformanceMonitor.Collectors/PgDeadlockLogParser.cs","PerformanceMonitor.Collectors/PgDeadlocksCollector.cs","PerformanceMonitor.Collectors/PgLogEntryAssembler.cs","PerformanceMonitor.Collectors/PgLogTextRedactor.cs","docs/postgres-first-target-runbook.md"],"failed_classes":["Darling.Tests.LiveCleanupConversionRatchetTests"],"failed_jobs":["Darling PG tests (2)","build"]} +{"date":"2026-09-23","head_sha":"f43c826d2ba28bedb2e33bd881aff8baaf6ea69f","event":"pull_request","changed_files":["Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Lite.Tests/PgPlanCaptureCollectorDefinitionTests.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgPlanLogParser.cs"],"failed_classes":["Lite.Tests.PgPlanLogParserTests"],"failed_jobs":["Lite tests (1)"]} +{"date":"2026-09-23","head_sha":"32239fd3498a1fa9ce1cc1cce057f21c04f5221a","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/PgDeadlockRemaskTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/PgDeadlockRemask.cs","Darling/README.md"],"failed_classes":["PerformanceMonitorLite.Tests.MemoryTrendGrantJoinToolTests"],"failed_jobs":["Lite tests (1)"]} +{"date":"2026-09-23","head_sha":"960a4458e220afbb19990a2929b6c8c1f72201e8","event":"push","changed_files":["Darling/Darling.Tests/FleetIdentifierScrubTests.cs","Darling/Darling.Tests/PgLogEventMetricsTests.cs","Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Darling/Darling.Tests/PgLogHashKeyTests.cs","Darling/Dockerfile","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingLogHashKeyFile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgLogEventTools.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogEventIngestor.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Storage/DarlingPgLogEventReader.cs","Darling/README.md","Darling/compose/docker-compose.yml","Lite.Tests/AuroraOnlySqlIsGatedTests.cs","Lite.Tests/PgServerLogPathPinTests.cs","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/PgLogEvent.cs","PerformanceMonitor.Collectors/PgLogEventClassifier.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgLogHashKey.cs","PerformanceMonitor.Collectors/PgLogTextRedactor.cs"],"failed_classes":["PerformanceMonitorLite.Tests.MemoryTrendGrantJoinToolTests"],"failed_jobs":["Lite tests (1)"]} +{"date":"2026-09-23","head_sha":"bb2f5cd5194c69dbff41a63114fb670516bc3062","event":"pull_request","changed_files":["Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Darling/Darling.Tests/PgLogTailStderrSiblingLiveTests.cs","Darling/Darling.Tests/PgLoggingCollectorOffTests.cs","Darling/Darling.Tests/RdsLogSourceTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogSource.cs","Lite.Tests/PgLoggingCollectorGateTests.cs","PerformanceMonitor.Collectors/PgDeadlocksCollector.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgNoStderrLogFileException.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgServerLogTail.cs"],"failed_classes":["Darling.Tests.CollectionLogDrainForensicsStoreTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-09-23","head_sha":"00fcf92e55a07ac47ded88666423c2633c92c1e7","event":"pull_request","changed_files":["Darling/Darling.Tests/AbandonedRunEraInvariantReadTests.cs","Darling/Darling.Tests/CollectionOutputBesideCostTests.cs","Darling/Darling.Tests/CollectorHealthClassifierTests.cs","Darling/Darling.Tests/ProductiveZeroBandingTests.cs","Darling/Darling.Tests/RegressedFromProductiveTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs","Lite.Tests/CollectionOutputBesideCostTests.cs","Lite.Tests/CollectorHealthClassifierTests.cs","Lite.Tests/RegressedFromProductiveTests.cs","Lite/Services/LocalDataService.CollectionHealth.cs","PerformanceMonitor.Common/ServerHealthBands.cs"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests","Darling.Tests.ViewerDailyHealthLivePostgresTests","Darling.Tests.ViewerDailyHealthRowTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (0)","Darling PG tests (2)"]} +{"date":"2026-09-23","head_sha":"145dd4df66e41917b7183ce89402f7244e03ee1a","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingInstallLocationTests.cs","Darling/README.md","Darling/tools/install-darling.ps1","Darling/tools/upgrade-darling.ps1"],"failed_classes":["Darling.Tests.DarlingInstallLocationTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-23","head_sha":"68361e125b316082d7c8b001001046233e0b2381","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingInstallLocationTests.cs","Darling/README.md","Darling/tools/install-darling.ps1","Darling/tools/upgrade-darling.ps1"],"failed_classes":["Darling.Tests.DarlingInstallLocationTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-09-23","head_sha":"4fdaebdec47a70e41dc46a40b73146e9e5606741","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/PgDeadlockRemaskTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/PgDeadlockRemask.cs","Darling/README.md","PerformanceMonitor.Collectors/PgLogHashKey.cs","PerformanceMonitor.Notifications/AnalysisNotificationService.cs","PerformanceMonitor.Notifications/PerformanceMonitor.Notifications.csproj"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["Darling PG tests (1)","build"]} +{"date":"2026-09-23","head_sha":"21634fced037b669ab0c3928de1423c73786d243","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingInstallLocationTests.cs","Darling/README.md","Darling/tools/install-darling.ps1","Darling/tools/upgrade-darling.ps1"],"failed_classes":["Darling.Tests.DarlingInstallLocationTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-09-23","head_sha":"2d6ed6393b55ea5b7e58fb53682dadb145991ad8","event":"pull_request","changed_files":["Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolGuideTests.cs","Darling/Darling.Tests/McpToolsListBudget.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/McpUnknownArgumentGuardTests.cs","Darling/Darling.Tests/McpZeroIsAMeasurementTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHealthParserTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpToolGuideTools.cs","Lite.Tests/McpDescriptionTruthPinTests.cs","Lite.Tests/McpSchemaCompatTests.cs","Lite.Tests/McpToolGuideTests.cs","Lite.Tests/McpToolsListBudget.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite/Mcp/McpHealthParserTools.cs","Lite/Mcp/McpHostService.cs","Lite/Mcp/McpInstructions.cs","Lite/Mcp/McpToolGuideTools.cs","PerformanceMonitor.Common/Mcp/McpSchemaCompat.cs","PerformanceMonitor.Common/Mcp/McpToolGuide.cs","PerformanceMonitor.Common/Mcp/McpToolGuideTopics.cs","README.md","llms.txt"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests","Darling.Tests.DarlingWebEndpointsTests"],"failed_jobs":["build","Darling PG tests (2)"]} +{"date":"2026-09-23","head_sha":"a7abf470ffac68b086108acf3623ac038dff5a9f","event":"pull_request","changed_files":["Darling/Darling.Tests/RdsDeadlockIngestorTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsDeadlockIngestor.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsIngestOutcome.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogEventIngestor.cs","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/PgBinaryTailText.cs","PerformanceMonitor.Collectors/PgDeadlockLogParser.cs","PerformanceMonitor.Collectors/PgDeadlocksCollector.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgReadBinaryFileCapability.cs","PerformanceMonitor.Collectors/PgServerLogTail.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.StragglerCommandTimeoutTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (1)","build"]} +{"date":"2026-09-23","head_sha":"0619bcb5c2f14074a43287996b5f3442c56b7042","event":"pull_request","changed_files":["Darling/Darling.Tests/PgBinaryTailTextTests.cs","Darling/Darling.Tests/PgReadBinaryFileAdvisoryTests.cs","Darling/Darling.Tests/PgReadBinaryFileCapabilityTests.cs","Darling/Darling.Tests/PgServerLogTailBinaryRouteTests.cs","Darling/Darling.Tests/PostgresFaultOutcomeTests.cs","Darling/Darling.Tests/RdsDeadlockIngestorTests.cs","Darling/Darling.Tests/RdsLogEventIngestorTimezoneTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsDeadlockIngestor.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsIngestOutcome.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogEventIngestor.cs","Darling/README.md","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/PgBinaryTailText.cs","PerformanceMonitor.Collectors/PgDeadlockLogParser.cs","PerformanceMonitor.Collectors/PgDeadlocksCollector.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgReadBinaryFileAdvisory.cs","PerformanceMonitor.Collectors/PgReadBinaryFileCapability.cs","PerformanceMonitor.Collectors/PgServerLogTail.cs","docs/postgres-first-target-runbook.md","tools/pg-verification-rig/README.md"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.PgReadBinaryFileCapabilityTests","Darling.Tests.StragglerCommandTimeoutTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-09-23","head_sha":"11d7e8b947a81061534dd88975d5be26942afa60","event":"pull_request","changed_files":["Darling/Darling.Tests/PgBinaryTailTextTests.cs","Darling/Darling.Tests/PgReadBinaryFileAdvisoryTests.cs","Darling/Darling.Tests/PgReadBinaryFileCapabilityLiveTests.cs","Darling/Darling.Tests/PgReadBinaryFileCapabilityTests.cs","Darling/Darling.Tests/PgReadBinaryFileStaticsCollection.cs","Darling/Darling.Tests/PgServerLogTailBinaryRouteTests.cs","Darling/Darling.Tests/PostgresFaultOutcomeTests.cs","Darling/Darling.Tests/RdsDeadlockIngestorTests.cs","Darling/Darling.Tests/RdsLogEventIngestorTimezoneTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsDeadlockIngestor.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsIngestOutcome.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogEventIngestor.cs","Darling/README.md","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/PgBinaryTailText.cs","PerformanceMonitor.Collectors/PgDeadlockLogParser.cs","PerformanceMonitor.Collectors/PgDeadlocksCollector.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgReadBinaryFileAdvisory.cs","PerformanceMonitor.Collectors/PgReadBinaryFileCapability.cs","PerformanceMonitor.Collectors/PgServerLogTail.cs","docs/postgres-first-target-runbook.md","tools/pg-verification-rig/README.md"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.LiveCleanupConversionRatchetTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-23","head_sha":"8c00707258b71d93b652eac9c336338d56945330","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingFileSecurityTests.cs","Darling/Darling.Tests/DarlingInstallLocationTests.cs","Darling/Darling.Tests/DarlingServiceInstallLocationTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingInstallLocation.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/README.md","Darling/tools/install-darling.ps1","Darling/tools/upgrade-darling.ps1","docs/retention-hold-runbook.md","docs/uat-onboarding.md"],"failed_classes":["Darling.Tests.DarlingInstallLocationTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-23","head_sha":"f2d14d628270342a21ef2571edb4dfd824335d97","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingFileSecurityTests.cs","Darling/Darling.Tests/DarlingInstallLocationTests.cs","Darling/Darling.Tests/DarlingServiceInstallLocationTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingInstallLocation.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/README.md","Darling/tools/install-darling.ps1","Darling/tools/upgrade-darling.ps1","docs/retention-hold-runbook.md","docs/uat-onboarding.md"],"failed_classes":["Darling.Tests.DarlingInstallLocationTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-23","head_sha":"b7136a99088b383ba9657251bc6537ad6f95483c","event":"pull_request","changed_files":["Darling/Darling.Tests/DocCommentHygieneTests.cs","Darling/Darling.Tests/PgBinaryTailTextTests.cs","Darling/Darling.Tests/PgReadBinaryFileAdvisoryTests.cs","Darling/Darling.Tests/PgReadBinaryFileCapabilityLiveTests.cs","Darling/Darling.Tests/PgReadBinaryFileCapabilityTests.cs","Darling/Darling.Tests/PgReadBinaryFileStaticsCollection.cs","Darling/Darling.Tests/PgServerLogTailBinaryRouteTests.cs","Darling/Darling.Tests/PostgresFaultOutcomeTests.cs","Darling/Darling.Tests/RdsDeadlockIngestorTests.cs","Darling/Darling.Tests/RdsLogEventIngestorTimezoneTests.cs","Darling/Darling.Tests/RdsLogTimezoneReadLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsDeadlockIngestor.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsIngestOutcome.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogEventIngestor.cs","Darling/README.md","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/PgBinaryTailText.cs","PerformanceMonitor.Collectors/PgDeadlockLogParser.cs","PerformanceMonitor.Collectors/PgDeadlocksCollector.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgReadBinaryFileAdvisory.cs","PerformanceMonitor.Collectors/PgReadBinaryFileCapability.cs","PerformanceMonitor.Collectors/PgServerLogTail.cs","docs/postgres-first-target-runbook.md","tools/pg-verification-rig/README.md"],"failed_classes":["Darling.Tests.StoreCopyPhaseTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-23","head_sha":"ff0f436e5993a76f70f93b4019e51a8f398a6cd5","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingCustomViewsTests.cs","Darling/Darling.Tests/DarlingMcpDataToolsTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Lite.Tests/QueryStoreDedupReadTests.cs","Lite/Mcp/McpQueryTools.cs","Lite/Services/LocalDataService.QueryStore.cs"],"failed_classes":["Darling.Tests.McpToolsListBudgetTests","Lite.Tests.McpToolsListBudgetTests"],"failed_jobs":["build","Lite tests (2)","Darling PG tests (1)"]} +{"date":"2026-09-23","head_sha":"2843f3dd08e9fa9735304a3cfafc39df6e8e8e34","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingCustomViewsTests.cs","Darling/Darling.Tests/DarlingMcpDataToolsTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Lite.Tests/QueryStoreDedupReadTests.cs","Lite/Mcp/McpQueryTools.cs","Lite/Services/LocalDataService.QueryStore.cs"],"failed_classes":["Darling.Tests.McpPayloadContractCensusTests","Darling.Tests.McpToolsListBudgetTests","Lite.Tests.McpToolsListBudgetTests"],"failed_jobs":["Darling PG tests (1)","build","Lite tests (2)","Darling PG tests (0)"]} +{"date":"2026-09-23","head_sha":"fbc27f9ad9898ebeab4a95917fe9f37a3a1b4b34","event":"pull_request","changed_files":["Darling/Darling.Tests/McpToolGuideHeads.Alerting.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpAlertTools.txt","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAlertTools.cs","Lite.Tests/McpToolGuideHeads.Alerting.cs","Lite.Tests/McpToolsListBudget/McpAlertTools.txt","Lite/Mcp/McpAlertTools.cs"],"failed_classes":["Darling.Tests.McpToolsListBudgetTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-23","head_sha":"777cd9801fa5a285be9c37000cc0bb02a077ab6b","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingCustomViewsTests.cs","Darling/Darling.Tests/DarlingMcpDataToolsTests.cs","Darling/Darling.Tests/McpFilterSemanticsLivePostgresTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpDataTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Lite.Tests/McpToolsListBudget/McpQueryTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite.Tests/QueryStoreDedupReadTests.cs","Lite/Mcp/McpQueryTools.cs","Lite/Services/LocalDataService.QueryStats.cs","Lite/Services/LocalDataService.QueryStore.cs"],"failed_classes":["Darling.Tests.ConsumedTimestampFrameDisciplineTests"],"failed_jobs":["Darling PG tests (2)","build","Darling PG tests (0)"]} +{"date":"2026-09-23","head_sha":"2c1085b073a01d7174c61941e92c44a202622114","event":"pull_request","changed_files":["Darling/Darling.Tests/McpToolGuideHeads.SqlCore.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpTools.txt","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Lite.Tests/McpToolGuideHeads.SqlCore.cs","Lite.Tests/McpToolsListBudget/McpAnalysisTools.txt","Lite/Mcp/McpAnalysisTools.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-23","head_sha":"135eb2acca3f541867f7d0540f8b77b9e56f9d9f","event":"pull_request","changed_files":["PerformanceMonitor.Collectors/PgDeadlocksCollector.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs"],"failed_classes":["Darling.Tests.PgLogEventsPipelineTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-09-24","head_sha":"dbfce0d99588ce3f06c13eb1cdbc580c540d6504","event":"pull_request","changed_files":["Darling/Darling.Tests/CheckpointsTimedRungTests.cs","Darling/Darling.Tests/PostmasterStartTimeRungTests.cs","Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreMetricsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Storage/StoreSelfMetrics.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs"],"failed_classes":["Darling.Tests.PostmasterStartTimeLivePostgresTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-24","head_sha":"5acfbb1a34f6bd5f22f838c027cb0348d116c23c","event":"pull_request","changed_files":["Darling/tools/install-darling.ps1","Darling/tools/upgrade-darling.ps1"],"failed_classes":["Darling.Tests.DarlingInstallLocationTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-09-24","head_sha":"3c9d27c4f78f132fcf82ed76eb0fa8679a668f94","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpConfigHistoryToolsTests.cs","Darling/Darling.Tests/McpToolGuideHeads.SqlTail.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpConfigHistoryTools.txt","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpDefaultTraceTools.txt","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpConfigHistoryTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDefaultTraceTools.cs","Lite.Tests/McpToolGuideHeads.SqlTail.cs","Lite.Tests/McpToolsListBudget/McpConfigTools.txt","Lite.Tests/McpToolsListBudget/McpDefaultTraceTools.txt","Lite.Tests/QueryStoreHealthCaptureModePublishTests.cs","Lite/Mcp/McpConfigTools.cs","Lite/Mcp/McpDefaultTraceTools.cs"],"failed_classes":["Darling.Tests.McpToolsListBudgetTests"],"failed_jobs":["build"]} +{"date":"2026-09-24","head_sha":"e673e9f52070516096ddf0723d31ac8af1e806b3","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingInstallLocationTests.cs","Darling/tools/install-darling.ps1","Darling/tools/upgrade-darling.ps1"],"failed_classes":["Darling.Tests.DarlingInstallLocationTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-24","head_sha":"d3b1815702e2d143e43a84980ecc14b6825bdb9d","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingInstallLocationTests.cs","Darling/tools/install-darling.ps1","Darling/tools/upgrade-darling.ps1"],"failed_classes":["Darling.Tests.DarlingInstallLocationTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-24","head_sha":"1e7169a66f70a75766b6a1a6c9200d71277b9540","event":"pull_request","changed_files":["Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Darling/Darling.Tests/PgPlanCaptureLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","PerformanceMonitor.Collectors/PgDeadlocksCollector.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs"],"failed_classes":["Darling.Tests.CollectorMeasurementSeamTests","Darling.Tests.McpToolsListBudgetTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-24","head_sha":"8ab99c7b585b446016432a950efe3d8adea6b6fa","event":"pull_request","changed_files":["Darling/Darling.Tests/McpToolGuideHeads.SqlTail.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpMemoryGrantTools.txt","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpMemoryGrantTools.cs","Lite.Tests/McpToolGuideHeads.SqlTail.cs","Lite.Tests/McpToolsListBudget/McpMemoryTools.txt","Lite/Mcp/McpMemoryTools.cs","PerformanceMonitor.Common/Mcp/McpToolGuideTopics.SqlTail.cs"],"failed_classes":["Darling.Tests.McpToolsListBudgetTests"],"failed_jobs":["build"]} +{"date":"2026-09-24","head_sha":"8afd015dbdccf1c9027e6e9e17c61763c898f690","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingInstallLocationTests.cs","Darling/tools/install-darling.ps1","Darling/tools/upgrade-darling.ps1"],"failed_classes":["Darling.Tests.DarlingInstallLocationTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-24","head_sha":"a4583d1d4830ffa76bd459a9e0a5ac746bee2242","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingInstallLocationTests.cs","Darling/tools/install-darling.ps1","Darling/tools/upgrade-darling.ps1"],"failed_classes":["Darling.Tests.DarlingInstallLocationTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-24","head_sha":"ffa1aecf27a2142633e0a4f120468a984f761559","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpMemoryGrantToolsTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpMemoryGrantTools.cs","Lite.Tests/McpLatestSnapshotStampTests.cs","Lite/Mcp/McpMemoryTools.cs"],"failed_classes":["Darling.Tests.McpToolsListBudgetTests"],"failed_jobs":["build"]} +{"date":"2026-09-24","head_sha":"353f632c1a31ebeaa4f2879f864b9bab9b8ca526","event":"pull_request","changed_files":["Darling/Darling.Tests/McpToolGuideHeads.SqlCore.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpSessionTools.txt","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpSessionTools.cs","Lite.Tests/McpToolGuideHeads.SqlCore.cs","Lite.Tests/McpToolsListBudget/McpSessionTools.txt","Lite/Mcp/McpSessionTools.cs"],"failed_classes":["Darling.Tests.McpToolsListBudgetTests"],"failed_jobs":["build"]} +{"date":"2026-09-24","head_sha":"5d8a9513ee1c55441b5abd8dd4b5f76f71c5f88d","event":"pull_request","changed_files":["Darling/Darling.Tests/DocCommentHygieneTests.cs","Darling/Darling.Tests/PgBinaryTailTextTests.cs","Darling/Darling.Tests/PgReadBinaryFileAdvisoryTests.cs","Darling/Darling.Tests/PgReadBinaryFileCapabilityLiveTests.cs","Darling/Darling.Tests/PgReadBinaryFileCapabilityTests.cs","Darling/Darling.Tests/PgServerEncodingTests.cs","Darling/Darling.Tests/PostgresFaultOutcomeTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/README.md","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/PgBinaryTailText.cs","PerformanceMonitor.Collectors/PgDeadlocksCollector.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgReadBinaryFileCapability.cs","PerformanceMonitor.Collectors/PgServerEncoding.cs"],"failed_classes":["Darling.Tests.McpToolsListBudgetTests"],"failed_jobs":["build"]} +{"date":"2026-09-24","head_sha":"5a3c5d14c60d24580ab6e34b6c85c1c0ad722b02","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingInstallLocationTests.cs"],"failed_classes":["Darling.Tests.McpToolsListBudgetTests"],"failed_jobs":["build"]} +{"date":"2026-09-24","head_sha":"00f09a2d81441c8fd7487706a3b09d94006b8c6b","event":"pull_request","changed_files":["Darling/Darling.Tests/McpToolGuideHeads.AnalyzeQueryPlan.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpPlanTools.txt","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanTools.cs","Lite.Tests/McpToolGuideHeads.AnalyzeQueryPlan.cs","Lite.Tests/McpToolsListBudget/McpPlanTools.txt","Lite/Mcp/McpPlanTools.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["Darling PG tests (1)","build"]} +{"date":"2026-09-24","head_sha":"11729239b1e7fc70215a9f62d3464009e6df403c","event":"pull_request","changed_files":["Darling/Darling.Tests/McpToolGuideHeads.Spinlock.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpLatchSpinlockTools.txt","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpLatchSpinlockTools.cs","Lite.Tests/McpToolGuideHeads.Spinlock.cs","Lite.Tests/McpToolsListBudget/McpLatchSpinlockTools.txt","Lite/Mcp/McpLatchSpinlockTools.cs"],"failed_classes":["Darling.Tests.XminHorizonPersistenceReadTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"f1f71ae3be3edc170789d37ce11e3b0aa57798fa","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogEventIngestor.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogSource.cs"],"failed_classes":["Darling.Tests.RdsLogEventIngestorCsvlogTests"],"failed_jobs":["Darling PG tests (2)","build"]} +{"date":"2026-09-24","head_sha":"6ec704afa9efb30b9f0212ca2bede57e4c325be4","event":"push","changed_files":["Darling/Darling.Tests/RdsDeadlockIngestorTests.cs","Darling/Darling.Tests/RdsLogEventIngestorCsvlogTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsCsvlogCarry.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsDeadlockIngestor.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogEventIngestor.cs","PerformanceMonitor.Collectors/PgDeadlocksCollector.cs"],"failed_classes":["Darling.Tests.XminHorizonPersistenceReadTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"a52f09cda3931d58f0554998ec503bb6283c292e","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingCoreToolProfileTests.cs","Darling/Darling.Tests/DarlingMcpLatchSpinlockToolsTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingLatchSpinlockReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Lite/Mcp/McpAnalysisTools.cs","PerformanceMonitor.Analysis/FactAdvice.cs","deprecated/Dashboard.Tests/FactAdviceCorrectnessTests.cs","deprecated/Dashboard/Mcp/McpAnalysisTools.cs","deprecated/Dashboard/Services/DatabaseService.ResourceMetrics.LatchSpinlock.cs"],"failed_classes":["Darling.Tests.DarlingMcpLatchSpinlockToolsLivePostgresTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"841ea4e9ddbe22fec0ac39cb093aa5b102a11f3c","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Analysis/PgAnomalyDetector.cs","PerformanceMonitor.Analysis/AnomalyGate.cs","PerformanceMonitor.Analysis/Baselines/AnomalyThresholds.cs"],"failed_classes":["Darling.Tests.DarlingAnomalyBaselineTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-24","head_sha":"c462a969c7c91725d0a917834f659fa1a0d20620","event":"pull_request","changed_files":["Darling/Darling.Tests/PgDeadlockRaiseShapedUnitTests.cs","Darling/Darling.Tests/PgLogEntryProvenanceTests.cs","Darling/Darling.Tests/PgPlanCaptureRaiseShapedUnitTests.cs","Darling/Darling.Tests/RdsDeadlockIngestorTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsDeadlockIngestor.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsIngestOutcome.cs","PerformanceMonitor.Collectors/PerformanceMonitor.Collectors.csproj","PerformanceMonitor.Collectors/PgDeadlockLogParser.cs","PerformanceMonitor.Collectors/PgDeadlocksCollector.cs","PerformanceMonitor.Collectors/PgLogEntry.cs","PerformanceMonitor.Collectors/PgLogEntryProvenance.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgServerLogCsvParser.cs","PerformanceMonitor.Collectors/PgServerLogJsonParser.cs"],"failed_classes":["Darling.Tests.CollectorMeasurementSeamTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-24","head_sha":"77e66c96d20994a2c689d018babc0976a5f972a6","event":"pull_request","changed_files":["Darling/Darling.Tests/ViewerCollectionCaveatsTests.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.CollectionHealth.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml"],"failed_classes":["Darling.Tests.ViewerCollectionCaveatsLivePostgresTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"a751513e14339d2cc0da9b3caa5dc6c0e5b6bcf4","event":"pull_request","changed_files":["Darling/Darling.Tests/PgDeadlockRaiseShapedUnitTests.cs","Darling/Darling.Tests/PgLogEntryProvenanceTests.cs","Darling/Darling.Tests/PgPlanCaptureRaiseShapedUnitTests.cs","Darling/Darling.Tests/PgRaiseShapedRecordsLiveTests.cs","Darling/Darling.Tests/RdsDeadlockIngestorTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsDeadlockIngestor.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsIngestOutcome.cs","PerformanceMonitor.Collectors/PerformanceMonitor.Collectors.csproj","PerformanceMonitor.Collectors/PgDeadlockLogParser.cs","PerformanceMonitor.Collectors/PgDeadlocksCollector.cs","PerformanceMonitor.Collectors/PgLogEntry.cs","PerformanceMonitor.Collectors/PgLogEntryProvenance.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgServerLogCsvParser.cs","PerformanceMonitor.Collectors/PgServerLogJsonParser.cs"],"failed_classes":["Darling.Tests.CollectorMeasurementSeamTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-24","head_sha":"bcaaf9b190968038afb0ddd9d7fd34f8eee499bd","event":"pull_request","changed_files":["Darling/Darling.Tests/ViewerCollectionCaveatsTests.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.CollectionHealth.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Postgres.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml"],"failed_classes":["Darling.Tests.ViewerCollectionCaveatsLivePostgresTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"62ae815c1d4a107ede861f351f78b8ea8c1bb82c","event":"pull_request","changed_files":[".github/workflows/build.yml"],"failed_classes":["Darling.Tests.CiClusterWorkerSizingTests"],"failed_jobs":["Darling PG tests (2)","build","Darling whole-tree guards"]} +{"date":"2026-09-24","head_sha":"7fc674875a3b3872a41a063586ca347cedb4b3e9","event":"pull_request","changed_files":["Darling/Darling.Tests/LocalClockBucketKeyTests.cs","Lite.Tests/LocalClockBucketKeyTests.cs"],"failed_classes":["PerformanceMonitorLite.Tests.LocalClockBucketKeyTests"],"failed_jobs":["Lite tests (2)"]} +{"date":"2026-09-24","head_sha":"22e3803f2785d9938741bd5564988525663e88c2","event":"pull_request","changed_files":["Darling/Darling.Tests/LocalClockBucketKeyTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs","Lite.Tests/LocalClockBucketKeyTests.cs","Lite/Analysis/BaselineProvider.cs"],"failed_classes":["Darling.Tests.ComposeStoreRolesLiveTests","PerformanceMonitorLite.Tests.LocalClockBucketKeyTests"],"failed_jobs":["Lite tests (2)","Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"ad89ad83a7cee8cec5acca50ccc166a483ad0778","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Blocking.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Io.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Replication.cs"],"failed_classes":["Darling.Tests.PgTargetAnomalyTests","Darling.Tests.PgTargetBetweenWavesV2Tests","Darling.Tests.PgTargetBlockingTests","Darling.Tests.PgTargetIoTests","Darling.Tests.SharedBaselineCacheLiveTests","PerformanceMonitorLite.Tests.LocalClockBucketKeyTests"],"failed_jobs":["Darling PG tests (1)","build","Darling PG tests (0)","Darling PG tests (2)","Lite tests (2)"]} +{"date":"2026-09-24","head_sha":"08ca596e6508ab4eabb6ff21676dd2717b17f52c","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.PgTargetAnomalyTests","Darling.Tests.PgTargetSampledWaitTests","Darling.Tests.PgTargetSessionsTests","Darling.Tests.SharedBaselineCacheLiveTests","PerformanceMonitorLite.Tests.LocalClockBucketKeyTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (0)","build","Lite tests (2)","Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"d064b8ebc5d52997af9850e5c2b65e553fe1d658","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Analysis/PgAnomalyDetector.cs"],"failed_classes":["Darling.Tests.DarlingAnomalyBaselineTests","PerformanceMonitorLite.Tests.LocalClockBucketKeyTests"],"failed_jobs":["build","Darling PG tests (0)","Lite tests (2)"]} +{"date":"2026-09-24","head_sha":"1e9c8ce764c676d7697a99e97f0e40f5bf9f6755","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetBetweenWavesV2Tests.cs"],"failed_classes":["Darling.Tests.PgTargetBlockingTests","Darling.Tests.PgTargetIoTests","Darling.Tests.SharedBaselineCacheLiveTests","PerformanceMonitorLite.Tests.LocalClockBucketKeyTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (2)","Lite tests (2)"]} +{"date":"2026-09-24","head_sha":"95a42dab2dc799da7f0b90eb642cfd8485a89b80","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.PgTargetAnomalyTests","Darling.Tests.PgTargetSessionsTests","Darling.Tests.SharedBaselineCacheLiveTests","PerformanceMonitorLite.Tests.LocalClockBucketKeyTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (1)","Darling PG tests (2)","build","Lite tests (2)"]} +{"date":"2026-09-24","head_sha":"52b659cee2a30f34aff7f87cae1bedce1889df3d","event":"pull_request","changed_files":["Lite/Analysis/AnomalyDetector.cs"],"failed_classes":["Darling.Tests.DarlingAnomalyBaselineTests","PerformanceMonitorLite.Tests.AnomalyDetectorTests"],"failed_jobs":["build","Lite tests (2)"]} +{"date":"2026-09-24","head_sha":"f7f0dee46734b4b5742825a8e2820fa188c005c8","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.cs"],"failed_classes":["Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"3a7e6cf3b725bf5fcaf07de58ba69b3d9d2e3ecf","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetBlockingTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Blocking.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Io.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Replication.cs"],"failed_classes":["Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"7b696d60d733692d9e112c142c7968fd9b4bed50","event":"pull_request","changed_files":["Darling/Darling.Tests/SqlServerStoreTileBehaviourTests.cs"],"failed_classes":["Darling.Tests.SqlServerStoreTileBehaviourTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-24","head_sha":"8a87e96d71d3a967230414b7488289dfb5d474b3","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetTileBehaviourTests.cs"],"failed_classes":["Darling.Tests.PgTargetTileBehaviourTests","Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"bae1fca91eddf3083311bff3832b6f2b1ad51d8a","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetTileBehaviourTests.cs"],"failed_classes":["Darling.Tests.PgTargetTileBehaviourTests","Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"f36b0a75bb9599862d2846bbc5072b175e6c5426","event":"pull_request","changed_files":["Lite/Analysis/AnomalyDetector.cs"],"failed_classes":["Darling.Tests.DarlingAnomalyBaselineTests","PerformanceMonitorLite.Tests.AnomalyDetectorTests"],"failed_jobs":["build","Lite tests (2)"]} +{"date":"2026-09-24","head_sha":"8983c978418166784cb4607871fb52dedecadbc5","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingAnomalyBaselineTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgAnomalyDetector.cs"],"failed_classes":["Darling.Tests.SqlServerStoreTileBehaviourTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-24","head_sha":"b3fa71b1b1ca081bc7c053f1caa0d9af71f8784f","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetTileBehaviourIoReplicationBlockingTests.cs"],"failed_classes":["Darling.Tests.PgTargetBlockingTests","Darling.Tests.PgTargetTileBehaviourIoReplicationBlockingTests","Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"26e2db9b66d04d61cb96b744fbf0f8f7368fd3fa","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetTileBehaviourTests.cs"],"failed_classes":["Darling.Tests.PgTargetTileBehaviourTests","Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"48510777740fdb6e10d206efbbbd7a4160e6c267","event":"pull_request","changed_files":["Darling/Darling.Tests/SqlServerStoreTileBehaviourTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgAnomalyDetector.cs"],"failed_classes":["Darling.Tests.SqlServerStoreTileBehaviourTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-24","head_sha":"a3b574993759cf702066c8ddb7f1041bc7ce9ea1","event":"pull_request","changed_files":["Lite.Tests/AnomalyDetectorTests.cs"],"failed_classes":["Darling.Tests.DarlingAnomalyBaselineTests"],"failed_jobs":["build"]} +{"date":"2026-09-24","head_sha":"213fe55c693b683c943f3844742fa2258f74bcb8","event":"pull_request","changed_files":["Lite/Analysis/AnomalyDetector.cs"],"failed_classes":["Darling.Tests.DarlingAnomalyBaselineTests"],"failed_jobs":["build"]} +{"date":"2026-09-24","head_sha":"0c58c0eb56ea475c91ed9106a639e87602af7595","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetWriteTests.cs"],"failed_classes":["Darling.Tests.PgTargetSampledWaitLiveTests","Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"58d19e6748962ec878106c31b5f0ca9862975abb","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetTileBehaviourIoReplicationBlockingTests.cs"],"failed_classes":["Darling.Tests.PgTargetTileBehaviourIoReplicationBlockingTests","Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"798c28746bbb7cf0c9bd0dcea1ce6497f9691e61","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetTileBehaviourTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.cs"],"failed_classes":["Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"8b3c5767f081393cbc89be5253529987ca309826","event":"pull_request","changed_files":["Darling/Darling.Tests/AnomalyNullWindowLiveTests.cs"],"failed_classes":["Darling.Tests.SqlServerStoreTileBehaviourTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (0)"]} +{"date":"2026-09-24","head_sha":"34b9395198b3bf8fc9913ade1de3ec83876879cd","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetTileBehaviourWalKernelSampledTests.cs"],"failed_classes":["Darling.Tests.PgTargetSampledWaitLiveTests","Darling.Tests.PgTargetTileBehaviourWalKernelSampledTests","Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"c9877aad46cbffd4294ecafb4df9350999b4ce86","event":"pull_request","changed_files":["Lite.Tests/AnomalyDetectorTileBehaviourBatchSessionQueryMemoryTests.cs"],"failed_classes":["Darling.Tests.DarlingAnomalyBaselineTests","PerformanceMonitorLite.Tests.AnomalyDetectorTileBehaviourBatchSessionQueryMemoryTests"],"failed_jobs":["Lite tests (1)","build"]} +{"date":"2026-09-24","head_sha":"fee46d2ffca92f0a5019bd64ccd47be5cc883d73","event":"pull_request","changed_files":["Darling/Darling.Tests/SqlServerStoreTileBehaviourBatchQueryMemoryTests.cs"],"failed_classes":["Darling.Tests.SqlServerStoreTileBehaviourBatchQueryMemoryTests","Darling.Tests.SqlServerStoreTileBehaviourTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (2)","Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"88751883379b8ccf015aa947acc09b2ee65f5ceb","event":"pull_request","changed_files":["Lite.Tests/AnomalyDetectorTileBehaviourTests.cs"],"failed_classes":["Darling.Tests.DarlingAnomalyBaselineTests","PerformanceMonitorLite.Tests.AnomalyDetectorTileBehaviourTests"],"failed_jobs":["Lite tests (1)","build"]} +{"date":"2026-09-24","head_sha":"4195e827d1fb73d5e31a8918681324d2319282e8","event":"pull_request","changed_files":["Darling/Darling.Tests/RobustBaselineTests.cs","PerformanceMonitor.Analysis/Baselines/BaselineMath.cs"],"failed_classes":["Darling.Tests.LocalClockBucketKeyLiveTests","PerformanceMonitorLite.Tests.BaselineProviderTests","PerformanceMonitorLite.Tests.LocalClockBucketKeyTests"],"failed_jobs":["Lite tests (2)","Lite tests (0)","Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"d5370df470c7f7f3a589ea9861886bf80a78c561","event":"pull_request","changed_files":["Lite.Tests/AnomalyDetectorTileBehaviourTests.cs"],"failed_classes":["Darling.Tests.DarlingAnomalyBaselineTests","PerformanceMonitorLite.Tests.AnomalyDetectorTileBehaviourTests"],"failed_jobs":["build","Lite tests (1)"]} +{"date":"2026-09-24","head_sha":"87836cc97e8847b8b59dd6b127b76ad83b4824b0","event":"pull_request","changed_files":["Lite.Tests/AnomalyDetectorTileBehaviourBatchSessionQueryMemoryTests.cs"],"failed_classes":["Darling.Tests.DarlingAnomalyBaselineTests","PerformanceMonitorLite.Tests.AnomalyDetectorTileBehaviourBatchSessionQueryMemoryTests"],"failed_jobs":["build","Lite tests (1)"]} +{"date":"2026-09-24","head_sha":"097fd767dc72abf524df5c1d205a70ab524bacaf","event":"pull_request","changed_files":["Darling/Darling.Tests/SqlServerStoreTileBehaviourBatchQueryMemoryTests.cs"],"failed_classes":["Darling.Tests.SqlServerStoreTileBehaviourTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (2)"]} +{"date":"2026-09-24","head_sha":"3de285b652943777797c221fdbfe1e7a5772a363","event":"pull_request","changed_files":["Lite.Tests/AnomalyDetectorTileBehaviourBatchSessionQueryMemoryTests.cs"],"failed_classes":["Darling.Tests.DarlingAnomalyBaselineTests"],"failed_jobs":["build"]} +{"date":"2026-09-24","head_sha":"91a978ed6f837fa335f79e3f2ff42c590a6c6288","event":"pull_request","changed_files":["Darling/Darling.Tests/RobustBaselineTests.cs","PerformanceMonitor.Analysis/Baselines/BaselineBucket.cs","PerformanceMonitor.Analysis/Baselines/BaselineMath.cs"],"failed_classes":["Darling.Tests.LocalClockBucketKeyLiveTests","PerformanceMonitorLite.Tests.BaselineProviderTests","PerformanceMonitorLite.Tests.LocalClockBucketKeyTests"],"failed_jobs":["Darling PG tests (1)","Lite tests (2)","Lite tests (0)"]} +{"date":"2026-09-24","head_sha":"0ac14fc78e1c18418a0d47eb2a5c67897815d054","event":"pull_request","changed_files":["Lite.Tests/AnomalyDetectorTileBehaviourTests.cs"],"failed_classes":["Darling.Tests.DarlingAnomalyBaselineTests","PerformanceMonitorLite.Tests.AnomalyDetectorTileBehaviourTests"],"failed_jobs":["build","Lite tests (1)"]} +{"date":"2026-09-24","head_sha":"723a7007c5becb3cc98a1da72dfee0fdddec6710","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetKernelTests.cs","Darling/Darling.Tests/PgTargetSampledWaitLiveTests.cs","Darling/Darling.Tests/PgTargetSampledWaitTests.cs","Darling/Darling.Tests/PgTargetTileBehaviourWalKernelSampledTests.cs","Darling/Darling.Tests/PgTargetWriteTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Kernel.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.WaitsSampled.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.Wal.cs"],"failed_classes":["Darling.Tests.PgTargetSampledWaitLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"94038080ae2080c1760d65597022ec178a36659a","event":"pull_request","changed_files":["Darling/Darling.Tests/RobustBaselineTests.cs","Lite.Tests/BaselineProviderTests.cs","Lite.Tests/LocalClockBucketKeyTests.cs","PerformanceMonitor.Analysis/Baselines/BaselineBucket.cs","PerformanceMonitor.Analysis/Baselines/BaselineMath.cs"],"failed_classes":["Darling.Tests.LocalClockBucketKeyLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"c42df0c5f92bc9c71732755bcc34860e443c9386","event":"pull_request","changed_files":["Darling/Darling.Tests/AnomalyDetectorErrorGuardLiveTests.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["Darling PG tests (1)","build"]} +{"date":"2026-09-24","head_sha":"80dc86e8553f65af6c8d1030c365c00e9208e40e","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingAnomalyBaselineTests.cs","Lite.Tests/AnomalyDetectorTests.cs","Lite.Tests/AnomalyDetectorTileBehaviourTests.cs","Lite/Analysis/AnomalyDetector.cs"],"failed_classes":["Darling.Tests.DarlingAnomalyBaselineTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-24","head_sha":"3216c5d674f8f0733675f1aee0fb2662ef863119","event":"pull_request","changed_files":["Darling/Darling.Tests/AnomalyDetectorErrorGuardLiveTests.cs"],"failed_classes":["Darling.Tests.AnomalyDetectorErrorGuardLiveTests","Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-24","head_sha":"5219f29479a942b0568d800782a9c0ff0d56f94d","event":"pull_request","changed_files":["Darling/Darling.Tests/IntervalHonestHourlyRollupTests.cs","Darling/Darling.Tests/TimescaleAggregateCompressionTests.cs","docs/runbooks/a6-successor-daily-backfill.md"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.IntervalHonestHourlyRollupLiveTests","Darling.Tests.MaterializationHoleRepairTests","Darling.Tests.TimescaleAggregateCompressionTests","PerformanceMonitorLite.Tests.AnomalyDetectorTileBehaviourTests"],"failed_jobs":["build","Darling PG tests (2)","Darling PG tests (0)","Lite tests (1)","Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"e4cb4a3699d102a64a616d5346156e3d39764a81","event":"pull_request","changed_files":["Darling/Darling.Tests/StitchedRelationSqlTests.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","PerformanceMonitorLite.Tests.AnomalyDetectorTileBehaviourTests"],"failed_jobs":["Lite tests (1)","build","Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"1c726d7f675026f839843485382c1b5bb974b384","event":"pull_request","changed_files":["Darling/Darling.Tests/SuccessorDailyLiveTests.cs","Darling/Darling.Tests/TimescaleAggregateCompressionTests.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.IntervalHonestHourlyRollupLiveTests","Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.MaterializationHoleRepairTests","Darling.Tests.SuccessorDailyLiveTests","PerformanceMonitorLite.Tests.AnomalyDetectorTileBehaviourTests"],"failed_jobs":["build","Darling PG tests (1)","Lite tests (1)","Darling PG tests (0)"]} +{"date":"2026-09-24","head_sha":"24d76b370d163d39fbb5cb79b4693e1124d47102","event":"pull_request","changed_files":["Darling/Darling.Tests/MaterializationHoleRepairTests.cs","Darling/Darling.Tests/SuccessorDailyLiveTests.cs"],"failed_classes":["Darling.Tests.IntervalHonestHourlyRollupLiveTests","Darling.Tests.SuccessorDailyLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"31bbb6682d58ef75b32339d642344336b40d7142","event":"pull_request","changed_files":["Darling/Darling.Tests/RollupCoverageRoutingTests.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["Darling PG tests (1)","build"]} +{"date":"2026-09-24","head_sha":"55d78cb0a2d130b5e0f35f39b0fbad87d6a90698","event":"pull_request","changed_files":["Darling/Darling.Tests/SuccessorDailyLiveTests.cs"],"failed_classes":["Darling.Tests.IntervalHonestHourlyRollupLiveTests","Darling.Tests.SuccessorDailyLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"e708093a4bf102da9a40992e979cb1764a63b55c","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingHealthReader.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummarySql.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.DailySummary.cs"],"failed_classes":["Darling.Tests.DailySummaryNotCarriedLiveTests","Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["build","Darling PG tests (2)","Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"37fcfc144c3569b300cc000851b939ccb631317e","event":"pull_request","changed_files":["Darling/Darling.Tests/DailySummaryStitchedRangeTests.cs"],"failed_classes":["Darling.Tests.DailySummaryNotCarriedLiveTests","Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (2)","build"]} +{"date":"2026-09-24","head_sha":"d244949e4b4f31ffaa45256b2e78ff4f5a4c6ac5","event":"pull_request","changed_files":["Darling/Darling.Tests/IntervalHonestHourlyRollupTests.cs","Darling/Darling.Tests/SuccessorDailyLiveTests.cs"],"failed_classes":["Darling.Tests.IntervalHonestHourlyRollupLiveTests","Darling.Tests.SuccessorDailyLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"8454c3afda07704780bec760f9e0a35e9cec706a","event":"pull_request","changed_files":["Darling/Darling.Tests/DailySummaryAndComposeStitchedLiveTests.cs","Darling/Darling.Tests/DarlingMcpTrendToolsTests.cs","Darling/Darling.Tests/DarlingTrendReaderStitchTests.cs","Darling/Darling.Tests/RollupCoverageRoutingTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTrendTools.cs"],"failed_classes":["Darling.Tests.DailySummaryNotCarriedLiveTests","Darling.Tests.DocCommentHygieneTests","Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.PgTargetBlockingTests","Darling.Tests.RollupCoverageRoutingTests"],"failed_jobs":["Darling PG tests (0)","build","Darling PG tests (2)","Darling PG tests (1)"]} +{"date":"2026-09-24","head_sha":"e2e007f88c7aae5d0aa6d20c2d3539b2f04e18fe","event":"pull_request","changed_files":["Darling/Darling.Tests/DailySummaryAndComposeStitchedLiveTests.cs","Darling/Darling.Tests/DailySummaryStitchedRangeTests.cs","Darling/Darling.Tests/DarlingMcpTrendToolsTests.cs","Darling/Darling.Tests/DarlingTrendReaderStitchTests.cs","Darling/Darling.Tests/RollupCoverageRoutingTests.cs","Darling/Darling.Tests/StitchedRelationSqlTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeCompiler.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeSourceRouter.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingHealthReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTrendTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingTrendReader.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummarySql.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.DailySummary.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Workload.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryTrends.cs"],"failed_classes":["Darling.Tests.DailySummaryNotCarriedLiveTests","Darling.Tests.LivePostgresCollectionHygieneTests"],"failed_jobs":["Darling PG tests (2)","build","Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"f17b6df1341f112c110b9600e1b45d5508fc9d2c","event":"pull_request","changed_files":["Darling/Darling.Tests/DailySummaryStitchedRangeTests.cs","Darling/Darling.Tests/RollupCoverageRoutingTests.cs","Darling/Darling.Tests/StitchedRelationSqlTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeSourceRouter.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummarySql.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Workload.cs"],"failed_classes":["Darling.Tests.PgTargetBlockingTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"5fe4fb64bd23645cbf413d598fd84125baa2367c","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingAnomalyBaselineTests.cs","Darling/Darling.Tests/SqlServerStoreTileBehaviourBatchQueryMemoryTests.cs","Darling/Darling.Tests/SqlServerStoreTileBehaviourTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgAnomalyDetector.cs"],"failed_classes":["Darling.Tests.SqlServerStoreTileBehaviourTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"02a1f124d3068b36428d29131cbc875eaa833f6b","event":"pull_request","changed_files":["Darling/Darling.Tests/RollupBackfillTests.cs","Darling/PerformanceMonitor.Darling.Storage/RollupBackfill.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.BaselineSupplyTests","Darling.Tests.CompressionPhaseAssignmentTests","Darling.Tests.DailySummaryAndComposeStitchedLiveTests","Darling.Tests.DailySummaryNotCarriedLiveTests","Darling.Tests.DailySummaryNotCarriedTests","Darling.Tests.DailySummaryReadShapeLiveTests","Darling.Tests.DailySummaryReadShapeSqlTests","Darling.Tests.DailySummaryStitchedRangeTests","Darling.Tests.DocCommentHygieneTests","Darling.Tests.IntervalHonestHourlyRollupLiveTests","Darling.Tests.IntervalHonestHourlyRollupTests","Darling.Tests.MaterializationHoleRepairLiveTests","Darling.Tests.MaterializationHoleRepairTests","Darling.Tests.MaterializationHoleScanShapeLiveTests","Darling.Tests.MeasurementContractCensusTests","Darling.Tests.RefreshCeilingProvenancePinTests","Darling.Tests.RefreshCeilingStalenessTests","Darling.Tests.RollupBackfillLiveTests","Darling.Tests.SuccessorDailyLiveTests","Darling.Tests.TimescaleAggregateCompressionTests","Darling.Tests.TimescaleContinuousAggregateTests","Darling.Tests.TimescaleSupportTests"],"failed_jobs":["build","Darling PG tests (2)","Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"193506c3bfde145aec380549766e9ac6a38b320d","event":"push","changed_files":["Darling/Darling.Tests/DailyStitchLiveTests.cs","Darling/Darling.Tests/DailySummaryStitchedRangeTests.cs","Darling/Darling.Tests/RollupCoverageRoutingTests.cs","Darling/Darling.Tests/StitchedRelationSqlTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeSourceRouter.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummarySql.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Workload.cs"],"failed_classes":["Darling.Tests.ComposeStoreRolesLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"1cc0bb25188617cb0e06004cc54819dec6c29f16","event":"pull_request","changed_files":["Darling/Darling.Tests/IntervalHonestHourlyRollupTests.cs","Darling/Darling.Tests/RollupBackfillTests.cs","Darling/Darling.Tests/TimescaleAggregateCompressionTests.cs","Darling/PerformanceMonitor.Darling.Storage/RollupBackfill.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.BaselineSupplyTests","Darling.Tests.CompressionPhaseAssignmentTests","Darling.Tests.DailyStitchLiveTests","Darling.Tests.DailySummaryAndComposeStitchedLiveTests","Darling.Tests.DailySummaryNotCarriedLiveTests","Darling.Tests.DailySummaryNotCarriedTests","Darling.Tests.DailySummaryReadShapeLiveTests","Darling.Tests.DailySummaryReadShapeSqlTests","Darling.Tests.DailySummaryStitchedRangeTests","Darling.Tests.IntervalHonestHourlyRollupLiveTests","Darling.Tests.IntervalHonestHourlyRollupTests","Darling.Tests.MaterializationHoleRepairLiveTests","Darling.Tests.MaterializationHoleRepairTests","Darling.Tests.MaterializationHoleScanShapeLiveTests","Darling.Tests.MeasurementContractCensusTests","Darling.Tests.PayloadDimensionLiveTests","Darling.Tests.QueryStoreCorrectedRollupLiveTests","Darling.Tests.RefreshCeilingProvenancePinTests","Darling.Tests.RefreshCeilingStalenessTests","Darling.Tests.RetentionReevaluationLiveTests","Darling.Tests.RollupBackfillLiveTests","Darling.Tests.SuccessorDailyLiveTests","Darling.Tests.TimescaleAggregateCompressionTests","Darling.Tests.TimescaleContinuousAggregateTests","Darling.Tests.TimescaleSupportTests"],"failed_jobs":["Darling PG tests (0)","build","Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"1abc48ba8aad34e3ebd80e9116a8c5f725075d0f","event":"pull_request","changed_files":["Darling/Darling.Tests/IntervalHonestHourlyRollupTests.cs","Darling/Darling.Tests/RollupBackfillTests.cs","Darling/Darling.Tests/TimescaleAggregateCompressionTests.cs","Darling/PerformanceMonitor.Darling.Storage/RollupBackfill.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.BaselineSupplyTests","Darling.Tests.CompressionPhaseAssignmentTests","Darling.Tests.DailyStitchLiveTests","Darling.Tests.DailySummaryAndComposeStitchedLiveTests","Darling.Tests.DailySummaryNotCarriedLiveTests","Darling.Tests.DailySummaryNotCarriedTests","Darling.Tests.DailySummaryReadShapeLiveTests","Darling.Tests.DailySummaryReadShapeSqlTests","Darling.Tests.DailySummaryStitchedRangeTests","Darling.Tests.IntervalHonestHourlyRollupLiveTests","Darling.Tests.IntervalHonestHourlyRollupTests","Darling.Tests.MaterializationHoleRepairLiveTests","Darling.Tests.MaterializationHoleRepairTests","Darling.Tests.MaterializationHoleScanShapeLiveTests","Darling.Tests.MeasurementContractCensusTests","Darling.Tests.PayloadDimensionLiveTests","Darling.Tests.QueryStoreCorrectedRollupLiveTests","Darling.Tests.RefreshCeilingProvenancePinTests","Darling.Tests.RefreshCeilingStalenessTests","Darling.Tests.RetentionReevaluationLiveTests","Darling.Tests.RollupBackfillLiveTests","Darling.Tests.SuccessorDailyLiveTests","Darling.Tests.TimescaleAggregateCompressionTests","Darling.Tests.TimescaleContinuousAggregateTests","Darling.Tests.TimescaleSupportTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (2)","Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"fafcda881f46b53d1ba34f479ab8620571b14d89","event":"pull_request","changed_files":["Darling/Darling.Tests/BaselineSupplyTests.cs","Darling/Darling.Tests/CompressionPhaseAssignmentTests.cs","Darling/Darling.Tests/DailySummaryNotCarriedTests.cs","Darling/Darling.Tests/DailySummaryReadShapeTests.cs","Darling/Darling.Tests/DailySummaryStitchedRangeTests.cs","Darling/Darling.Tests/FrozenRollupLiveTests.cs","Darling/Darling.Tests/IntervalHonestHourlyRollupTests.cs","Darling/Darling.Tests/MaterializationHoleRepairTests.cs","Darling/Darling.Tests/MaterializationHoleScanShapeTests.cs","Darling/Darling.Tests/MeasurementContractCensusTests.cs","Darling/Darling.Tests/RollupBackfillLiveTests.cs","Darling/Darling.Tests/RollupBackfillTests.cs","Darling/Darling.Tests/SuccessorDailyLiveTests.cs","Darling/Darling.Tests/TimescaleAggregateCompressionTests.cs","Darling/Darling.Tests/TimescaleSupportTests.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummarySql.cs","Darling/PerformanceMonitor.Darling.Storage/RollupBackfill.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.MaterializationHoles.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.DailySummaryNotCarriedLiveTests","Darling.Tests.MaterializationHoleScanShapeLiveTests","Darling.Tests.PayloadDimensionLiveTests","Darling.Tests.QueryStoreCorrectedRollupLiveTests","Darling.Tests.RefreshCeilingProvenancePinTests","Darling.Tests.RefreshCeilingStalenessTests","Darling.Tests.RetentionReevaluationLiveTests","Darling.Tests.RollupBackfillLiveTests","Darling.Tests.TimescaleContinuousAggregateTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (0)","build","Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"775d4b660fcdf3a9140f9553b90a3329f8437eff","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFleetTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Lite/Mcp/McpHealthTools.cs","Lite/Mcp/McpHostService.cs","Lite/Services/LocalDataService.CollectionHealth.cs","PerformanceMonitor.Common/Mcp/McpResponseBudget.cs","PerformanceMonitor.Common/Mcp/McpResponseBudgetCallToolFilter.cs"],"failed_classes":["Darling.Tests.DarlingCustomViewsTests","Darling.Tests.McpPayloadContractCensusTests","Darling.Tests.McpToolGuideHeadsCollectionLogTests","Darling.Tests.McpToolGuideHeadsPlatformTests","Darling.Tests.McpToolsListBudgetTests"],"failed_jobs":["Darling PG tests (0)","build","Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"1c2cd71fbd573fb0c6820439316fc4959bb68d6a","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetBetweenWavesV3Tests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgCpuUtilizationTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreRegressionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingQueryStoreRegressionReader.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingPgCpuUtilizationReader.cs","Lite/Analysis/AnalysisService.cs","Lite/Analysis/DuckDbFactCollector.cs","Lite/Mcp/McpAnalysisTools.cs","PerformanceMonitor.Common/Mcp/TrendBuckets.cs"],"failed_classes":["Darling.Tests.CollectorRunnerConnectionEngineTests","Darling.Tests.DarlingAnalysisServiceEngineRoutingTests","Darling.Tests.DarlingQueryStoreRegressionsLiveTests","Darling.Tests.DocCommentHygieneTests","Darling.Tests.McpToolGuideHeadsQsRegressionsTests","Darling.Tests.McpToolGuideTests","Darling.Tests.McpToolsListBudgetTests","Darling.Tests.PgTargetAuditConfigTests","Darling.Tests.PgTargetMcpSurfaceTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (2)","build","Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"056523f02308cc435c8d9c9991757d13d3e3e872","event":"push","changed_files":["Darling/Darling.Tests/AnomalyObjectStatsLatestSnapshotsLiveTests.cs","Darling/Darling.Tests/CollectionCaveatsRungTests.cs","Darling/Darling.Tests/IndexObjectStatsServerTimeIndexRungTests.cs","Darling/Darling.Tests/MigrationDataMovingRungCensusPins.cs","Darling/Darling.Tests/ViewerCollectionCaveatsTests.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs"],"failed_classes":["Darling.Tests.PgTargetBlockingTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"5b9486a0ea03068a45eb8a2b5878c9599674aaca","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingWebHostGateLiveTests.cs","Darling/Darling.Tests/ServerPageTabsTests.cs","Darling/Darling.Tests/WebFetchLayerTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/app.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/views.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js"],"failed_classes":["Darling.Tests.WebFetchLayerTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (0)","build"]} +{"date":"2026-09-25","head_sha":"209c58c87202f9baec11b90e3b6fbbcacfffcd89","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorRunnerConnectionEngineTests.cs","Darling/Darling.Tests/DarlingAnalysisServiceEngineRoutingTests.cs","Darling/Darling.Tests/McpToolGuideHeads.PgA.cs","Darling/Darling.Tests/McpToolGuideHeads.QsRegressions.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpPgCpuUtilizationTools.txt","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpQueryStoreRegressionTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/PgTargetBetweenWavesV3Tests.cs","Darling/Darling.Tests/PgTargetMcpSurfaceTests.cs","Darling/Darling.Tests/ViewerQueriesTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgCpuUtilizationTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreRegressionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingQueryStoreRegressionReader.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingPgCpuUtilizationReader.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStoreRegressions.cs","Lite.Tests/McpToolGuideHeads.QsRegressions.cs","Lite.Tests/McpToolsListBudget/McpQueryTools.txt","Lite.Tests/QueryStoreRegressionsToolTests.cs","Lite/Analysis/AnalysisService.cs","Lite/Analysis/DuckDbFactCollector.cs","Lite/Mcp/McpAnalysisTools.cs","Lite/Mcp/McpQueryTools.cs","Lite/Services/LocalDataService.QueryStoreRegressions.cs","PerformanceMonitor.Common/Mcp/TrendBuckets.cs"],"failed_classes":["Darling.Tests.DarlingQueryStoreRegressionsLiveTests","Darling.Tests.PgTargetAuditConfigTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"a12e72ff28a0ad8795853cb0473f6119c8552836","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingCustomViewsTests.cs","Darling/Darling.Tests/McpFleetResponseBudgetLivePostgresTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpDataTools.txt","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpFleetTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFleetTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Lite.Tests/CollectionLogFleetFormToolTests.cs","Lite.Tests/McpToolsListBudget/McpHealthTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite/Mcp/McpHealthTools.cs","Lite/Mcp/McpHostService.cs","Lite/Services/LocalDataService.CollectionHealth.cs","PerformanceMonitor.Common/Mcp/McpResponseBudget.cs"],"failed_classes":["Darling.Tests.McpFleetResponseBudgetLivePostgresTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"3fe9f11bd58c18735e57d627c054fcc08b2ca0f7","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorRunnerConnectionEngineTests.cs","Darling/Darling.Tests/DarlingAnalysisServiceEngineRoutingTests.cs","Darling/Darling.Tests/DarlingQueryStoreRegressionsTests.cs","Darling/Darling.Tests/McpToolGuideHeads.PgA.cs","Darling/Darling.Tests/McpToolGuideHeads.QsRegressions.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpPgCpuUtilizationTools.txt","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpQueryStoreRegressionTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/PgTargetBetweenWavesV3Tests.cs","Darling/Darling.Tests/PgTargetMcpSurfaceTests.cs","Darling/Darling.Tests/TrendPayloadBudgetLiveTests.cs","Darling/Darling.Tests/ViewerQueriesTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgCpuUtilizationTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreRegressionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingQueryStoreRegressionReader.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingPgCpuUtilizationReader.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStoreRegressions.cs","Lite.Tests/McpToolGuideHeads.QsRegressions.cs","Lite.Tests/McpToolsListBudget/McpQueryTools.txt","Lite.Tests/QueryStoreRegressionsToolTests.cs","Lite/Analysis/AnalysisService.cs","Lite/Analysis/DuckDbFactCollector.cs","Lite/Mcp/McpAnalysisTools.cs","Lite/Mcp/McpQueryTools.cs","Lite/Services/LocalDataService.QueryStoreRegressions.cs","PerformanceMonitor.Common/Mcp/TrendBuckets.cs"],"failed_classes":["Darling.Tests.PgTargetAuditConfigTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"cdc50a6af87bc64b8d2aab0d5b179bc2ed294ca3","event":"pull_request","changed_files":["Darling/Darling.Tests/EventWindowedReadsCarryTheFloorTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingBlockingTrendReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDefaultTraceReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingSystemHealthReader.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Blocking.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.BlockingStats.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.BlockingTrends.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.JobHistory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Memory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.SystemEvents.cs"],"failed_classes":["Darling.Tests.ViewerMemorySqlTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-09-25","head_sha":"e96652de96310cabffa1e50a30b5f24e70651d79","event":"pull_request","changed_files":["Darling/Darling.Tests/FreshStoreWatermarkTests.cs","Darling/Darling.Tests/OverviewCollectionHealthRollupRoutingTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs","Darling/PerformanceMonitor.Darling.Storage/CollectionHealthRollupSupport.cs","Darling/PerformanceMonitor.Darling.Viewer/MainWindow.ServerManagement.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Overview.cs"],"failed_classes":["Darling.Tests.PgTargetBlockingTests","Darling.Tests.ViewerW2aLivePostgresTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"3f02b9209412dd59425de092a7d220362fdd30eb","event":"push","changed_files":["Darling/Darling.Tests/DarlingWebHostGateLiveTests.cs","Darling/Darling.Tests/ServerPageTabsTests.cs","Darling/Darling.Tests/WebFetchLayerTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/app.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/views.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js"],"failed_classes":["Darling.Tests.PgTargetBlockingTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"1038b642ab74798a7f505a824ea94cade1a2c1a2","event":"pull_request","changed_files":["Darling/Darling.Tests/FreshStoreWatermarkTests.cs","Darling/Darling.Tests/OverviewCollectionHealthRollupRoutingTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs","Darling/PerformanceMonitor.Darling.Storage/CollectionHealthRollupSupport.cs","Darling/PerformanceMonitor.Darling.Viewer/MainWindow.ServerManagement.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Overview.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml.cs"],"failed_classes":["Darling.Tests.ViewerW2aLivePostgresTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"7eefba3376903fbe5ebbf0a53cea4fe6a5e15dc5","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorRunnerConnectionEngineTests.cs","Darling/Darling.Tests/DarlingAnalysisServiceEngineRoutingTests.cs","Darling/Darling.Tests/DarlingQueryStoreRegressionsTests.cs","Darling/Darling.Tests/McpToolGuideHeads.PgA.cs","Darling/Darling.Tests/McpToolGuideHeads.QsRegressions.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpPgCpuUtilizationTools.txt","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpQueryStoreRegressionTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/PgTargetBetweenWavesV3Tests.cs","Darling/Darling.Tests/PgTargetMcpSurfaceTests.cs","Darling/Darling.Tests/TrendPayloadBudgetLiveTests.cs","Darling/Darling.Tests/ViewerQueriesTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgCpuUtilizationTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreRegressionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingQueryStoreRegressionReader.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingPgCpuUtilizationReader.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStoreRegressions.cs","Lite.Tests/McpToolGuideHeads.QsRegressions.cs","Lite.Tests/McpToolsListBudget/McpQueryTools.txt","Lite.Tests/QueryStoreRegressionsToolTests.cs","Lite/Analysis/AnalysisService.cs","Lite/Analysis/DuckDbFactCollector.cs","Lite/Mcp/McpAnalysisTools.cs","Lite/Mcp/McpQueryTools.cs","Lite/Services/LocalDataService.QueryStoreRegressions.cs","PerformanceMonitor.Common/Mcp/TrendBuckets.cs"],"failed_classes":["Darling.Tests.AnalysisFactsReadRunsDetectorParityTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-09-25","head_sha":"3cb7665ad83d101737b47cbbaac20b7261059d27","event":"pull_request","changed_files":["Darling/Darling.Tests/RetentionTierRouterTests.cs","Darling/Darling.Tests/ViewerFinOpsTests.cs","Darling/Darling.Tests/ViewerFleetTimerFanOutPositionTests.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.Loaders.cs","Darling/PerformanceMonitor.Darling.Viewer/MainWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Storage.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Workload.cs"],"failed_classes":["Darling.Tests.RollupCoverageRoutingTests"],"failed_jobs":["Darling PG tests (1)","build"]} +{"date":"2026-09-25","head_sha":"be157a6d8866aeea321b6a6909eff84265736c4f","event":"pull_request","changed_files":["Darling/Darling.Tests/ServerInventoryFleetMetricsTests.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.Loaders.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Inventory.cs"],"failed_classes":["Darling.Tests.ViewerCommandTimeoutTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"78450e6cfde3cc9e65014b97b2f7370874f6b8cf","event":"pull_request","changed_files":["Darling/Darling.Tests/FleetCollectionHealthByServerLiveEqualityTests.cs","Darling/Darling.Tests/FreshStoreWatermarkTests.cs","Darling/Darling.Tests/OverviewCollectionHealthRollupRoutingTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs","Darling/PerformanceMonitor.Darling.Storage/CollectionHealthRollupSupport.cs","Darling/PerformanceMonitor.Darling.Viewer/MainWindow.ServerManagement.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Overview.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml.cs","Lite.Tests/PermissionDeniedBadgeMemoTests.cs","Lite/Services/LocalDataService.CollectionHealth.cs"],"failed_classes":["Darling.Tests.ViewerW2aLivePostgresTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"479aab4000d65569d9074485006a6c4aaffb33f7","event":"pull_request","changed_files":["Darling/Darling.Tests/FinOpsMergedReadsLiveTests.cs","Darling/Darling.Tests/RetentionTierRouterTests.cs","Darling/Darling.Tests/ViewerFinOpsTests.cs","Darling/Darling.Tests/ViewerFleetTimerFanOutPositionTests.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.Loaders.cs","Darling/PerformanceMonitor.Darling.Viewer/MainWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Storage.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Workload.cs"],"failed_classes":["Darling.Tests.RollupCoverageRoutingTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"6223c4f0b6203e6ac6ce0b710dcb67935d3bf5b2","event":"pull_request","changed_files":["Darling/Darling.Tests/ServerInventoryFleetMetricsLiveTests.cs","Darling/Darling.Tests/ServerInventoryFleetMetricsTests.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.Loaders.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Inventory.cs"],"failed_classes":["Darling.Tests.ViewerCommandTimeoutTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"f84ea5dee6d6fe4969656be5631dbc4538c6b90a","event":"pull_request","changed_files":["Darling/Darling.Tests/McpReadToolBudgetLiveTests.cs"],"failed_classes":["Darling.Tests.McpReadToolBudgetLiveTests","Darling.Tests.PgTargetBlockingTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"b8aff1c3bb0760fcc821dd75aad6d70101b03f2a","event":"pull_request","changed_files":["Darling/Darling.Tests/FinOpsMergedReadsLiveTests.cs","Darling/Darling.Tests/RetentionTierRouterTests.cs","Darling/Darling.Tests/ViewerFinOpsTests.cs","Darling/Darling.Tests/ViewerFleetTimerFanOutPositionTests.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.Loaders.cs","Darling/PerformanceMonitor.Darling.Viewer/MainWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Storage.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Workload.cs"],"failed_classes":["Darling.Tests.RollupCoverageRoutingTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"692cbb337a5fd5ed044dd862d28cae0b9a8aabf5","event":"pull_request","changed_files":["Darling/Darling.Tests/FleetCollectionHealthByServerLiveEqualityTests.cs","Darling/Darling.Tests/FreshStoreWatermarkTests.cs","Darling/Darling.Tests/OverviewCollectionHealthRollupRoutingTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs","Darling/PerformanceMonitor.Darling.Storage/CollectionHealthRollupSupport.cs","Darling/PerformanceMonitor.Darling.Viewer/MainWindow.ServerManagement.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Overview.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml.cs","Lite.Tests/PermissionDeniedBadgeMemoTests.cs","Lite/Services/LocalDataService.CollectionHealth.cs"],"failed_classes":["Darling.Tests.PgTargetAnomalyTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"fc12b2248e9e233e20eb678f86fa5a259b798ed5","event":"pull_request","changed_files":["Darling/Darling.Tests/AvailabilityGroupCountReadTests.cs","Darling/Darling.Tests/DarlingWebHostGateLiveTests.cs","Darling/Darling.Tests/DarlingWebResponseCompressionTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingAgReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/app.js"],"failed_classes":["Darling.Tests.DarlingWebResponseCompressionTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"80fa0748bd0ae617c4551844bfb42e1eb632415b","event":"pull_request","changed_files":["Darling/Darling.Tests/AnalysisFactsReadRunsDetectorParityTests.cs","Darling/Darling.Tests/CollectorRunnerConnectionEngineTests.cs","Darling/Darling.Tests/DarlingAnalysisServiceEngineRoutingTests.cs","Darling/Darling.Tests/DarlingQueryStoreRegressionsTests.cs","Darling/Darling.Tests/McpToolGuideHeads.PgA.cs","Darling/Darling.Tests/McpToolGuideHeads.QsRegressions.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpPgCpuUtilizationTools.txt","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpQueryStoreRegressionTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/PgTargetBetweenWavesV3Tests.cs","Darling/Darling.Tests/PgTargetMcpSurfaceTests.cs","Darling/Darling.Tests/TrendPayloadBudgetLiveTests.cs","Darling/Darling.Tests/ViewerQueriesTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgCpuUtilizationTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreRegressionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingQueryStoreRegressionReader.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingPgCpuUtilizationReader.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStoreRegressions.cs","Lite.Tests/McpToolGuideHeads.QsRegressions.cs","Lite.Tests/McpToolsListBudget/McpQueryTools.txt","Lite.Tests/QueryStoreRegressionsToolTests.cs","Lite/Analysis/AnalysisService.cs","Lite/Analysis/DuckDbFactCollector.cs","Lite/Mcp/McpAnalysisTools.cs","Lite/Mcp/McpQueryTools.cs","Lite/Services/LocalDataService.QueryStoreRegressions.cs","PerformanceMonitor.Common/Mcp/TrendBuckets.cs"],"failed_classes":["Darling.Tests.McpToolsListBudgetTests","Darling.Tests.PgTargetBlockingTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"43aba6068f8b417e44ff734cc07993ab999e2a6c","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionCaveatsRungTests.cs","Darling/Darling.Tests/DarlingAnalysisPipelineTests.cs","Darling/Darling.Tests/ForcePlanBotPolicyTests.cs","Darling/Darling.Tests/PgFactCollectorTests.cs","Darling/Darling.Tests/PlanRegressionIntervalTableEquivalenceTests.cs","Darling/Darling.Tests/PlanRegressionReadShapeTests.cs","Darling/Darling.Tests/PostmasterStartTimeRungTests.cs","Darling/Darling.Tests/QueryStoreIntervalLatestRungTests.cs","Darling/Darling.Tests/QueryStoreIntervalLatestWriterTests.cs","Darling/Darling.Tests/ViewerCollectionCaveatsTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.Queries.cs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalLatest.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Lite/Analysis/DrillDownCollector.Queries.cs","Lite/Analysis/DuckDbFactCollector.QueryPerf.cs","PerformanceMonitor.Analysis/AnalysisContext.cs","PerformanceMonitor.Analysis/FactAdvice.cs","PerformanceMonitor.Analysis/FactRemediation.cs","PerformanceMonitor.Analysis/ForcePlanBotPolicy.cs","PerformanceMonitor.Analysis/RemediationAction.cs","PerformanceMonitor.Collectors/CollectorContext.cs"],"failed_classes":["Darling.Tests.IndexObjectStatsServerTimeIndexRungTests","Darling.Tests.QueryStoreIntervalLatestRungTests"],"failed_jobs":["Darling PG tests (0)","build","Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"731512799e340b31b29316f793a4f3ffe304e413","event":"push","changed_files":["Darling/Darling.Tests/CollectionCaveatsRungTests.cs","Darling/Darling.Tests/DarlingAnalysisPipelineTests.cs","Darling/Darling.Tests/ForcePlanBotPolicyTests.cs","Darling/Darling.Tests/IndexObjectStatsServerTimeIndexRungTests.cs","Darling/Darling.Tests/PgFactCollectorTests.cs","Darling/Darling.Tests/PlanRegressionIntervalTableEquivalenceTests.cs","Darling/Darling.Tests/PlanRegressionReadShapeTests.cs","Darling/Darling.Tests/PostmasterStartTimeRungTests.cs","Darling/Darling.Tests/QueryStoreIntervalLatestRungTests.cs","Darling/Darling.Tests/QueryStoreIntervalLatestWriterTests.cs","Darling/Darling.Tests/ViewerCollectionCaveatsTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.Queries.cs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalLatest.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Lite/Analysis/DrillDownCollector.Queries.cs","Lite/Analysis/DuckDbFactCollector.QueryPerf.cs","PerformanceMonitor.Analysis/AnalysisContext.cs","PerformanceMonitor.Analysis/FactAdvice.cs","PerformanceMonitor.Analysis/FactRemediation.cs","PerformanceMonitor.Analysis/ForcePlanBotPolicy.cs","PerformanceMonitor.Analysis/RemediationAction.cs","PerformanceMonitor.Collectors/CollectorContext.cs"],"failed_classes":["Darling.Tests.PgTargetBlockingTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"a91c85be55e83c4e844506f65e8a568ed30dbf56","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpObjectStatsToolsTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpObjectStatsTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/ObjectLockingBudgetLiveTests.cs","Darling/Darling.Tests/ObjectLockingWebDefaultTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpObjectStatsTools.cs","Lite.Tests/McpToolsListBudget/McpObjectStatsTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite.Tests/ObjectLockingBudgetTests.cs","Lite/Mcp/McpObjectStatsTools.cs"],"failed_classes":["Darling.Tests.PgTargetAnomalyTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"f421016dd70f658739a10087aa87cfb9069f9430","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingQueryHeatmapBudgetLiveTests.cs","Darling/Darling.Tests/DarlingQueryHeatmapTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpQueryHeatmapTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryHeatmapTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingQueryHeatmapReader.cs","Lite.Tests/McpToolsListBudget/McpQueryTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite.Tests/QueryHeatmapBudgetTests.cs","Lite/Mcp/McpQueryTools.cs","Lite/Services/LocalDataService.QueryHeatmap.cs"],"failed_classes":["Darling.Tests.PgTargetAnomalyTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"857988aca48247d128673e5c35739f0b865a8754","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpActiveQueriesBudgetLiveTests.cs","Darling/Darling.Tests/DarlingMcpSessionToolsTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpSessionTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpSessionTools.cs","Lite.Tests/McpPageContractTests.cs","Lite.Tests/McpToolsListBudget/McpSessionTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite/Mcp/McpSessionTools.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"89478c39d22ca3837bfd33eebda345998b88ad41","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingQueryHeatmapBudgetLiveTests.cs","Darling/Darling.Tests/DarlingQueryHeatmapTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpQueryHeatmapTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryHeatmapTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingQueryHeatmapReader.cs","Lite.Tests/McpToolsListBudget/McpQueryTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite.Tests/QueryHeatmapBudgetTests.cs","Lite/Mcp/McpQueryTools.cs","Lite/Services/LocalDataService.QueryHeatmap.cs"],"failed_classes":["Darling.Tests.McpPayloadContractCensusTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"b0b0b85e4ac609c54c3496f4bf615b0e7d72cc5e","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpActiveQueriesBudgetLiveTests.cs","Darling/Darling.Tests/DarlingMcpSessionToolsTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpSessionTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpSessionTools.cs","Lite.Tests/McpPageContractTests.cs","Lite.Tests/McpToolsListBudget/McpSessionTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite/Mcp/McpSessionTools.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"2fddf918c05c7ffde2238887e36e5ca3a24708ad","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpObjectStatsToolsTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpObjectStatsTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/ObjectLockingBudgetLiveTests.cs","Darling/Darling.Tests/ObjectLockingWebDefaultTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpObjectStatsTools.cs","Lite.Tests/McpToolsListBudget/McpObjectStatsTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite.Tests/ObjectLockingBudgetTests.cs","Lite/Mcp/McpObjectStatsTools.cs"],"failed_classes":["Darling.Tests.PgTargetBlockingTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"76c59f4f7735dfc6a3efb91d120100cdf328171f","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpBlockingBudgetLiveTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpBlockingTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpBlockingTools.cs","Lite.Tests/McpToolsListBudget/McpBlockingTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite/Mcp/McpBlockingTools.cs"],"failed_classes":["Darling.Tests.McpToolGuideHeadsBlockingSlotsTests","Darling.Tests.McpToolGuideTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"7abb4ad6edba9db3061c7175a2288056885fb0dd","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionHealthPayloadBudgetLiveTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpDataTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/view-templates.js","Lite.Tests/CollectionHealthPayloadBudgetToolTests.cs","Lite/Mcp/McpHealthTools.cs"],"failed_classes":["Lite.Tests.McpToolsListBudgetTests"],"failed_jobs":["Lite tests (2)"]} +{"date":"2026-09-25","head_sha":"f5e001fe65c307fc4fb91ca714398a8a65fae236","event":"push","changed_files":["Darling/Darling.Tests/DarlingQueryHeatmapBudgetLiveTests.cs","Darling/Darling.Tests/DarlingQueryHeatmapTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpQueryHeatmapTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryHeatmapTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingQueryHeatmapReader.cs","Lite.Tests/McpToolsListBudget/McpQueryTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite.Tests/QueryHeatmapBudgetTests.cs","Lite/Mcp/McpQueryTools.cs","Lite/Services/LocalDataService.QueryHeatmap.cs"],"failed_classes":["Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests","Darling.Tests.PgTargetAnomalyTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"659e11a39b4765eefa09fb8ea735dbbcd65eaa97","event":"push","changed_files":["Darling/Darling.Tests/AvailabilityGroupCountReadTests.cs","Darling/Darling.Tests/DarlingAgReaderTests.cs","Darling/Darling.Tests/DarlingAgStatesReaderLiveEqualityTests.cs","Darling/Darling.Tests/DarlingAgStatesReaderTests.cs","Darling/Darling.Tests/ViewerAvailabilityGroupsTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingAgReader.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingAgStatesReader.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.AvailabilityGroups.cs","Lite.Tests/AgAvailabilityGroupNewestSnapshotEqualityTests.cs","Lite/Services/LocalDataService.AvailabilityGroups.cs"],"failed_classes":["Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"78630b525eb490c4265d858e5cedc9df6ec113fd","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpBlockingBudgetLiveTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpBlockingTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpBlockingTools.cs","Lite.Tests/McpPageContractTests.cs","Lite.Tests/McpToolsListBudget/McpBlockingTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite/Mcp/McpBlockingTools.cs"],"failed_classes":["Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests","Darling.Tests.McpToolGuideHeadsBlockingSlotsTests","Darling.Tests.McpToolGuideTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (0)","build"]} +{"date":"2026-09-25","head_sha":"43cacc218313cac06b83beeda82f16d7d1ae30fe","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionHealthPayloadBudgetLiveTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpDataTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/view-templates.js","Lite.Tests/CollectionHealthPayloadBudgetToolTests.cs","Lite.Tests/McpToolsListBudget/McpHealthTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite/Mcp/McpHealthTools.cs"],"failed_classes":["Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"f1a1894fd8559983d82a57e4d2f6ecdfe01b2250","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/Program.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs"],"failed_classes":["Darling.Tests.DarlingMissingCredentialMessageTests","Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests","Darling.Tests.SerialLoopStoreSizeSourceTests","Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (0)","Darling PG tests (1)","build"]} +{"date":"2026-09-25","head_sha":"171b60e5cb52e5c0e75f2d950217c26132c797ee","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpBlockingBudgetLiveTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpBlockingTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpBlockingTools.cs","Lite.Tests/McpPageContractTests.cs","Lite.Tests/McpToolsListBudget/McpBlockingTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite/Mcp/McpBlockingTools.cs"],"failed_classes":["Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"69ce7ff7beb259a633138f93968afe0e532a95a2","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingStoreHostProfileLiveTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/Program.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs"],"failed_classes":["Darling.Tests.DarlingMissingCredentialMessageTests","Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests","Darling.Tests.SerialLoopStoreSizeSourceTests","Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (2)","Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"4b3fe51a977924fd19dd5b271db3fee82f80c93d","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingCliCommandsHostCheckTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileLiveTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/Program.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/README.md"],"failed_classes":["Darling.Tests.DarlingMissingCredentialMessageTests","Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests","Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.SerialLoopStoreSizeSourceTests","Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["build","Darling PG tests (2)","Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"2c39e009ffe0ec21ddebfd547140388415ebd688","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingComposeTests.cs","Darling/Darling.Tests/DarlingMcpCustomViewCatalogSizeTests.cs","Darling/Darling.Tests/DarlingMcpCustomViewToolsTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpCustomViewTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpCustomViewTools.cs"],"failed_classes":["Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"d7c5592f94ecdf9af6f7e0dab97830abac69e894","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingCustomViewsTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpDataTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/QueryStoreTopBudgetLiveTests.cs","Darling/Darling.Tests/QueryStoreTopWebDefaultTests.cs","Darling/Darling.Tests/QueryStoreTopWindowTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Lite.Tests/McpToolsListBudget/McpQueryTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite.Tests/QueryStoreTopBudgetTests.cs","Lite/Mcp/McpQueryTools.cs"],"failed_classes":["Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"ed114b5b1724c71a674198254cacb39da0bf39d9","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpObjectStatsToolsTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpObjectStatsTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/ObjectLockingBudgetLiveTests.cs","Darling/Darling.Tests/ObjectLockingWebDefaultTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpObjectStatsTools.cs","Lite.Tests/McpToolsListBudget/McpObjectStatsTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite.Tests/ObjectLockingBudgetTests.cs","Lite/Mcp/McpObjectStatsTools.cs"],"failed_classes":["Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"3747867b5a50941776f5e38f55b881627b636dfd","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingCliCommandsHostCheckTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileLiveTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/Darling.Tests/TsqlConventionGuardTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/Program.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/README.md"],"failed_classes":["Darling.Tests.DarlingMissingCredentialMessageTests","Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests","Darling.Tests.LiveCleanupConversionRatchetTests","Darling.Tests.SerialLoopStoreSizeSourceTests"],"failed_jobs":["build","Darling PG tests (2)","Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"89c271dd17dbbec23a28a235602ef0fd1952f224","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpActiveQueriesBudgetLiveTests.cs","Darling/Darling.Tests/DarlingMcpSessionToolsTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpSessionTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpSessionTools.cs","Lite.Tests/McpPageContractTests.cs","Lite.Tests/McpToolsListBudget/McpSessionTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite/Mcp/McpSessionTools.cs"],"failed_classes":["Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"d0fb0ff183d7cfc588231b3e34ce48fbb64ff5e1","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpQueryStoreRegressionsBudgetLiveTests.cs","Darling/Darling.Tests/DarlingQueryStoreRegressionsTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpQueryStoreRegressionTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/QueryStoreRegressionsWebDefaultTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreRegressionTools.cs","Lite.Tests/McpToolsListBudget/McpQueryTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite.Tests/QueryStoreRegressionsBudgetTests.cs","Lite/Mcp/McpQueryTools.cs"],"failed_classes":["Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"354fc65faf06be290189179111773efcb9347a33","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionLogWebRowPreservesFullTextPinTests.cs","Darling/Darling.Tests/McpCollectionLogPerServerResponseBudgetLivePostgresTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpDataTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Lite.Tests/CollectionLogPerServerResponseBudgetToolTests.cs","Lite.Tests/McpToolsListBudget/McpHealthTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite/Mcp/McpHealthTools.cs","PerformanceMonitor.Common/Mcp/McpResponseBudget.cs"],"failed_classes":["Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"46d28fd5f39ccfba6e8678d1caa540e3daf0ef9b","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingComposeTests.cs","Darling/Darling.Tests/DarlingMcpCustomViewCatalogSizeTests.cs","Darling/Darling.Tests/DarlingMcpCustomViewToolsTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpCustomViewTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpCustomViewTools.cs"],"failed_classes":["Darling.Tests.PgTargetAnomalyTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"3ad2d3870172cdb49b25eaaca3404f2fc187de4c","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingCliCommandsHostCheckTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileLiveTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/Darling.Tests/SerialLoopStoreSizeSourceTests.cs","Darling/Darling.Tests/StartupHostProfileLogTests.cs","Darling/Darling.Tests/TsqlConventionGuardTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Program.cs","Darling/PerformanceMonitor.Darling.Service/ServiceCommandDeadlines.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/README.md"],"failed_classes":["Darling.Tests.DarlingMissingCredentialMessageTests","Darling.Tests.LiveCleanupConversionRatchetTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"d534e587ae80fe765509cc400583a26eed991d9f","event":"pull_request","changed_files":["Darling/Darling.Tests/BaselineSupplyTests.cs","Darling/Darling.Tests/CompressionPhaseAssignmentTests.cs","Darling/Darling.Tests/DailySummaryNotCarriedTests.cs","Darling/Darling.Tests/DailySummaryReadShapeTests.cs","Darling/Darling.Tests/DailySummaryStitchedRangeTests.cs","Darling/Darling.Tests/FrozenRollupLiveTests.cs","Darling/Darling.Tests/IntervalHonestHourlyRollupTests.cs","Darling/Darling.Tests/MaterializationHoleRepairTests.cs","Darling/Darling.Tests/MaterializationHoleScanShapeTests.cs","Darling/Darling.Tests/MeasurementContractCensusTests.cs","Darling/Darling.Tests/PayloadDimensionLiveTests.cs","Darling/Darling.Tests/QueryStoreCorrectedRollupLiveTests.cs","Darling/Darling.Tests/RefreshCeilingProvenancePinTests.cs","Darling/Darling.Tests/RefreshCeilingStalenessTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/RollupBackfillLiveTests.cs","Darling/Darling.Tests/RollupBackfillTests.cs","Darling/Darling.Tests/RollupCoverageRoutingTests.cs","Darling/Darling.Tests/SuccessorDailyLiveTests.cs","Darling/Darling.Tests/TimescaleAggregateCompressionTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/Darling.Tests/TimescaleSupportTests.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummarySql.cs","Darling/PerformanceMonitor.Darling.Storage/RollupBackfill.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.MaterializationHoles.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.FrozenRollupLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"37985a3c9dac392d51d767ba410f527b27c49bf1","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpQueryStoreRegressionsBudgetLiveTests.cs","Darling/Darling.Tests/DarlingQueryStoreRegressionsTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpQueryStoreRegressionTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/QueryStoreRegressionsWebDefaultTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreRegressionTools.cs","Lite.Tests/McpToolsListBudget/McpQueryTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite.Tests/QueryStoreRegressionsBudgetTests.cs","Lite/Mcp/McpQueryTools.cs"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["build"]} +{"date":"2026-09-25","head_sha":"772b0736064a167bc49ef03c876c92298d22b631","event":"pull_request","changed_files":["Lite.Tests/QueryWindowTruncationTests.cs","Lite/Mcp/McpQueryTools.cs","Lite/Services/LocalDataService.QueryWindowFloor.cs","PerformanceMonitor.Common/Mcp/McpHelpers.cs"],"failed_classes":["Darling.Tests.McpPayloadContractCensusTests","Darling.Tests.McpToolGuideTests","Lite.Tests.McpToolGuideHeadsCollectionLogTests","Lite.Tests.McpToolGuideHeadsDataTests","Lite.Tests.McpToolGuideTests","Lite.Tests.McpToolsListBudgetTests"],"failed_jobs":["Darling PG tests (1)","Lite tests (2)","Darling PG tests (0)","build","Lite tests (3)","Lite tests (1)"]} +{"date":"2026-09-25","head_sha":"6e80eb9ca038b80ea6f5b74f6446e249a3374295","event":"pull_request","changed_files":["Darling/Darling.Tests/AnalysisAsOfAnchorTests.cs","Darling/Darling.Tests/DarlingMcpAnalysisFindingsBudgetLiveTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Lite.Tests/AnalysisAsOfAnchorTests.cs","Lite.Tests/McpAnalysisFindingsCommandTests.cs","Lite.Tests/McpToolsListBudget/McpAnalysisTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite/Mcp/McpAnalysisTools.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["Darling PG tests (1)","build"]} +{"date":"2026-09-25","head_sha":"e7e7d3b6b0457b99f6d19471c97198de76bf4408","event":"pull_request","changed_files":["Lite.Tests/QueryWindowTruncationTests.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Controls/ServerTab.Slicers.cs","Lite/Controls/ServerTab.xaml","Lite/Mcp/McpQueryTools.cs","Lite/Services/LocalDataService.QueryWindowFloor.cs","PerformanceMonitor.Common/Mcp/McpHelpers.cs"],"failed_classes":["Darling.Tests.McpPayloadContractCensusTests","Darling.Tests.McpToolGuideTests","Lite.Tests.McpToolGuideHeadsCollectionLogTests","Lite.Tests.McpToolGuideHeadsDataTests","Lite.Tests.McpToolGuideTests","Lite.Tests.McpToolsListBudgetTests"],"failed_jobs":["build","Lite tests (1)","Darling PG tests (1)","Darling PG tests (0)","Lite tests (3)","Lite tests (2)"]} +{"date":"2026-09-25","head_sha":"07496c9a85a55c83f4256856fd31a6a0c076a347","event":"push","changed_files":["Darling/Darling.Tests/AnalysisAsOfAnchorTests.cs","Darling/Darling.Tests/DarlingMcpAnalysisFindingsBudgetLiveTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Lite.Tests/AnalysisAsOfAnchorTests.cs","Lite.Tests/McpAnalysisFindingsCommandTests.cs","Lite.Tests/McpToolsListBudget/McpAnalysisTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite/Mcp/McpAnalysisTools.cs"],"failed_classes":["Darling.Tests.PgTargetBlockingTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"ed146a61d650888740e1ceb3f55070e6ec13fca9","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingCliCommandsHostCheckTests.cs","Darling/Darling.Tests/FleetSweepWebFeedTests.cs","Darling/Darling.Tests/SerialLoopStoreSizeSourceTests.cs","Darling/Darling.Tests/SharedBaselineCacheTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreHostTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Service/ServiceCommandDeadlines.cs","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs"],"failed_classes":["Darling.Tests.DarlingMcpFleetSweepToolsTests","Darling.Tests.McpToolsListBudgetTests","Lite.Tests.CrossAppMcpToolInventoryPinTests"],"failed_jobs":["Darling PG tests (1)","build","Lite tests (1)"]} +{"date":"2026-09-25","head_sha":"35e0767f76f61dcc2598a68c27673ff270cd6e0b","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingCliCommandsHostCheckTests.cs","Darling/Darling.Tests/FleetSweepWebFeedTests.cs","Darling/Darling.Tests/SerialLoopStoreSizeSourceTests.cs","Darling/Darling.Tests/SharedBaselineCacheTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreHostTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Service/ServiceCommandDeadlines.cs","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","README.md","llms.txt"],"failed_classes":["Darling.Tests.DarlingMcpFleetSweepToolsTests","Darling.Tests.McpToolsListBudgetTests"],"failed_jobs":["Darling PG tests (1)","build"]} +{"date":"2026-09-25","head_sha":"68aee17c4c80fec90984285b0754960067533bc0","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingCliCommandsHostCheckTests.cs","Darling/Darling.Tests/DarlingMcpStoreHostToolsLiveTests.cs","Darling/Darling.Tests/FleetSweepWebFeedTests.cs","Darling/Darling.Tests/SerialLoopStoreSizeSourceTests.cs","Darling/Darling.Tests/SharedBaselineCacheTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreHostTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Service/ServiceCommandDeadlines.cs","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","README.md","llms.txt"],"failed_classes":["Darling.Tests.DarlingMcpFleetSweepToolsTests","Darling.Tests.McpToolsListBudgetTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"4382ed7cdef71bf907a181b7d3af9ace41f639a5","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingCliCommandsHostCheckTests.cs","Darling/Darling.Tests/DarlingMcpStoreHostBudgetLiveTests.cs","Darling/Darling.Tests/DarlingMcpStoreHostToolsLiveTests.cs","Darling/Darling.Tests/FleetSweepWebFeedTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpStoreHostTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/SerialLoopStoreSizeSourceTests.cs","Darling/Darling.Tests/SharedBaselineCacheTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreHostTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Service/ServiceCommandDeadlines.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/sweeps.js","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","README.md","llms.txt"],"failed_classes":["Darling.Tests.DarlingMcpFleetSweepToolsTests","Darling.Tests.LivePostgresCollectionHygieneTests"],"failed_jobs":["build","Darling PG tests (0)","Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"67db04eb0e75f29e3d914255840fd7889130126c","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreLogins.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreUpgrade.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingStoreConnection.cs","Darling/PerformanceMonitor.Darling.Storage/StoreConnectionSelfTest.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs"],"failed_classes":["Darling.Tests.ViewerCommandTimeoutTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"08e6e818b4269fc2a193cc1636ad4982835b9b17","event":"pull_request","changed_files":["Darling/Darling.Tests/McpReadToolBudgetLiveTests.cs"],"failed_classes":["Darling.Tests.McpReadToolBudgetLiveTests","Darling.Tests.PgTargetBlockingTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"d5444ff210c0f897bc9398dfe43832e863d10eb5","event":"pull_request","changed_files":["Darling/Darling.Tests/StoreSessionTimeZonePinCensusTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreLogins.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreUpgrade.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingStoreConnection.cs","Darling/PerformanceMonitor.Darling.Storage/StoreConnectionSelfTest.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs"],"failed_classes":["Darling.Tests.ViewerCommandTimeoutTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (1)","build"]} +{"date":"2026-09-25","head_sha":"f0370dc8fc5ee4381abe62f0d0f1405c95a695e0","event":"push","changed_files":["Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/RawWindowFloorMcpLiveTests.cs","Darling/Darling.Tests/RawWindowFloorViewerPortTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/view-templates.js","Darling/PerformanceMonitor.Darling.Storage/RawWindowFloor.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.ProcedureStats.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStats.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStore.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Queries.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml"],"failed_classes":["Darling.Tests.PgTargetAnomalyTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"f9ed975c705c4cc23592e40d39b731cb7c9fa75a","event":"pull_request","changed_files":["Darling/Darling.Tests/StoreSessionTimeZoneLiveTests.cs","Darling/Darling.Tests/StoreSessionTimeZonePinCensusTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreLogins.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreUpgrade.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingStoreConnection.cs","Darling/PerformanceMonitor.Darling.Storage/StoreConnectionSelfTest.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs"],"failed_classes":["Darling.Tests.ViewerCommandTimeoutTests"],"failed_jobs":["Darling PG tests (0)","build","Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"4226552ea499174a53cca817059f40b27e7012a7","event":"pull_request","changed_files":["Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolGuideHeads.CollectionLog.cs","Darling/Darling.Tests/McpToolGuideHeads.Data.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Lite.Tests/McpToolGuideHeads.CollectionLog.cs","Lite.Tests/McpToolGuideHeads.Data.cs","Lite.Tests/QueryWindowTruncationTests.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Controls/ServerTab.Slicers.cs","Lite/Controls/ServerTab.xaml","Lite/Mcp/McpQueryTools.cs","Lite/Services/LocalDataService.QueryWindowFloor.cs","Lite/Services/LocalDataService.cs","PerformanceMonitor.Common/Mcp/McpHelpers.cs"],"failed_classes":["Darling.Tests.McpToolsListBudgetTests","Lite.Tests.McpToolsListBudgetTests"],"failed_jobs":["build","Darling PG tests (1)","Lite tests (2)"]} +{"date":"2026-09-25","head_sha":"762897cf9cffc5b9c209e3f8f9a288d768aba5e5","event":"pull_request","changed_files":["Darling/Darling.Tests/WebExceptionTextCensusTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingFleetSweepEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Hosting/DarlingWebFailureLog.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.PgTargetAnomalyTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-09-25","head_sha":"a3ece55524427f86c76766ea4e4530aea6601a76","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Perfmon.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Waits.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerNameListCache.cs","PerformanceMonitor.Common/PerformanceMonitor.Common.csproj"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.ViewerPerfmonRunningJobsLivePostgresTests","Darling.Tests.ViewerPerfmonSqlTests","Darling.Tests.ViewerWaitStatsLivePostgresTests","Darling.Tests.ViewerWaitStatsSqlTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (1)","Darling PG tests (0)","build"]} +{"date":"2026-09-25","head_sha":"294c16c36e32aac01414fc931a9afaf7fd07bd22","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreBackfillTests.cs","Darling/PerformanceMonitor.Darling.Service/QueryStoreBackfill.cs","Darling/PerformanceMonitor.Darling.Service/ServiceCommandDeadlines.cs","Lite/Services/RemoteCollectorService.QueryStoreBackfill.cs","PerformanceMonitor.Collectors/QueryStoreBackfillState.cs"],"failed_classes":["Darling.Tests.CommandPlaneCommandTimeoutTests","Darling.Tests.PgTargetBlockingTests","Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"cff6610b2e0b5b8b7a99d5293d78f76687896d09","event":"push","changed_files":["Darling/Darling.Tests/DarlingStoreLoginsTests.cs","Darling/Darling.Tests/StoreConnectionStringAppendPinTests.cs","Darling/Darling.Tests/StoreSessionTimeZoneLiveTests.cs","Darling/Darling.Tests/StoreSessionTimeZonePinCensusTests.cs","Darling/Darling.Tests/ViewerCommandTimeoutTests.cs","Darling/Darling.Tests/ViewerDataServiceTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreLogins.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreUpgrade.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingStoreConnection.cs","Darling/PerformanceMonitor.Darling.Storage/StoreConnectionSelfTest.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs"],"failed_classes":["Darling.Tests.PgTargetBlockingTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"92bcaa1946d0ea4e8891a2a85cf2d623445f4622","event":"pull_request","changed_files":["Darling/Darling.Tests/BaselineSupplyTests.cs","Darling/Darling.Tests/RetiredBaselineAggregateTests.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests"],"failed_jobs":["build"]} +{"date":"2026-09-25","head_sha":"227f5a83b127c35ea6c26b02eef42fd716b6277d","event":"pull_request","changed_files":["Darling/Darling.Tests/ViewerPerfmonRunningJobsTests.cs","Darling/Darling.Tests/ViewerTrendBucketingReviewTests.cs","Darling/Darling.Tests/ViewerWaitStatsTests.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Perfmon.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Waits.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerNameListCache.cs","PerformanceMonitor.Common/PerformanceMonitor.Common.csproj"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["Darling PG tests (1)","build"]} +{"date":"2026-09-25","head_sha":"4b899f6f43f700a2cc8a64c122c715ffcb7b340c","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionHealthPayloadBudgetLiveTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpReadToolBudgetLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Lite.Tests/CollectionHealthPayloadBudgetToolTests.cs","Lite.Tests/McpReadToolBudgetTests.cs","Lite/Mcp/McpHealthTools.cs"],"failed_classes":["Darling.Tests.PgTargetAnomalyTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"fc399bc63e08965c98b9453de4c00e628aedaa38","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingAnomalyBaselineTests.cs","Darling/Darling.Tests/PgTargetClockTests.cs","Darling/PerformanceMonitor.Darling.Analysis/BaselineCache.cs","Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetBaselineProvider.cs"],"failed_classes":["Darling.Tests.PgTargetBetweenWavesV3Tests","Darling.Tests.PgTargetClockLiveTests","Darling.Tests.PgTargetSampledWaitLiveTests","Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (1)","build","Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"961d93b4730ea08620d75b4e525a91e5f16b738e","event":"pull_request","changed_files":["Darling/Darling.Tests/FailureNoteRedactionTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/CollectorRuntimeState.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingForcePlanTargetStateReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs"],"failed_classes":["Darling.Tests.PgTargetCollectionCaveatsLiveTests","Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"a9814cad2304b4d401fdd74580571938119040fb","event":"pull_request","changed_files":["Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs"],"failed_classes":["Darling.Tests.DarlingManagedPostgresTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"29411369ab30b45ead7c0527757c6415bbd8afe1","event":"pull_request","changed_files":["Lite.Tests/LiteWaitPerfmonTrendBucketingTests.cs","Lite.Tests/PerfmonIntervalAggregationTests.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Services/LiteNameListCache.cs","Lite/Services/LocalDataService.Perfmon.cs","Lite/Services/LocalDataService.WaitStats.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.PerfmonCounterTypeRungTests"],"failed_jobs":["build"]} +{"date":"2026-09-25","head_sha":"a338b38170d1a72be711261be4a7c0722bfb214d","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingPeerDisclosureTests.cs","Darling/Darling.Tests/FailureNoteRedactionTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/CollectorRuntimeState.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingForcePlanTargetStateReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/PgPlanForceActionStore.cs"],"failed_classes":["Darling.Tests.PgTargetCollectionCaveatsLiveTests","Darling.Tests.StartupFailureTriageTests","Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"8beb4d74a5c3ab6e2bfd108cd207a0581e48f8a7","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingHardenFilesVerbTests.cs","Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingStoreUpgradeRevertTests.cs","Darling/Darling.Tests/DarlingStoreUpgradeTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreUpgrade.cs"],"failed_classes":["Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"6c4c79575959486778d38951503983e24bc4385b","event":"pull_request","changed_files":["Lite.Tests/PerformanceTrendDurationBucketingTests.cs","Lite/Services/LocalDataService.QueryStats.cs"],"failed_classes":["Darling.Tests.McpZeroIsAMeasurementTests"],"failed_jobs":["build"]} +{"date":"2026-09-25","head_sha":"b8fa109068d6ffeebd8fb41b294320837c443cff","event":"pull_request","changed_files":["Lite.Tests/DuckDbSentinelConnectionTests.cs","Lite.Tests/MuteRulesSurviveResetTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/MainWindow.xaml.cs"],"failed_classes":["PerformanceMonitorLite.Tests.AnalysisPassTokenThreadingTests"],"failed_jobs":["Lite tests (3)"]} +{"date":"2026-09-25","head_sha":"d118681de9f3abd734a48467dc1669299a3c0305","event":"pull_request","changed_files":["Lite.Tests/LiteMemoryFileIoTrendBucketingTests.cs","Lite.Tests/LiteWaitPerfmonTrendBucketingTests.cs","Lite.Tests/PerfmonIntervalAggregationTests.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Mcp/McpIoTools.cs","Lite/Services/LiteNameListCache.cs","Lite/Services/LocalDataService.FileIo.cs","Lite/Services/LocalDataService.Memory.cs","Lite/Services/LocalDataService.Perfmon.cs","Lite/Services/LocalDataService.WaitStats.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.PerfmonCounterTypeRungTests"],"failed_jobs":["build"]} +{"date":"2026-09-25","head_sha":"be7716e8e6f52adbbc3bc7b9b4e264b4a5261c4f","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionCaveatsTests.cs","Darling/Darling.Tests/CollectorRuntimeStateTests.cs","Darling/Darling.Tests/DarlingPeerDisclosureTests.cs","Darling/Darling.Tests/DarlingStoreLoginsTests.cs","Darling/Darling.Tests/FailureNoteRedactionTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/CollectorRuntimeState.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingForcePlanTargetStateReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/PgPlanForceActionStore.cs","Lite.Tests/CollectionCaveatsTests.cs","Lite/Analysis/DuckDbFactCollector.cs","PerformanceMonitor.Analysis/AnalysisContext.cs"],"failed_classes":["Darling.Tests.PgTargetCollectionCaveatsLiveTests","Darling.Tests.StartupFailureTriageTests"],"failed_jobs":["Darling PG tests (1)","build"]} +{"date":"2026-09-25","head_sha":"298be53aa3252d3770f08bf962e5a20c2d65adeb","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingHardenFilesVerbTests.cs","Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingStoreUpgradeRevertTests.cs","Darling/Darling.Tests/DarlingStoreUpgradeTests.cs","Darling/Darling.Tests/TsqlConventionGuardTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreUpgrade.cs"],"failed_classes":["Darling.Tests.ComposeStoreRolesLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"64e33e5b724dab960c8c71f7dae532c6c5913953","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/ManagedConfFileLiveTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs"],"failed_classes":["Darling.Tests.ActivityDrivenPlanFetchStoreTests","Darling.Tests.CheckpointsTimedRungTests","Darling.Tests.CollectionCaveatsRungTests","Darling.Tests.CollectionLogDrainForensicsStoreTests","Darling.Tests.CollectionLogFanoutRollupStoreTests","Darling.Tests.CollectionLogFetchPhaseSumsLivePostgresTests","Darling.Tests.CollectionLogFetchPhaseSumsStoreTests","Darling.Tests.CollectionLogPhaseSplitStoreTests","Darling.Tests.CollectorDatabaseScopeRungTests","Darling.Tests.CollectorStallProbeStoreTests","Darling.Tests.ComposeStatementTimeoutStoreTests","Darling.Tests.CustomAlertCoreMigrationTests","Darling.Tests.DarlingAnalysisStoreTests","Darling.Tests.DarlingCustomViewsTests","Darling.Tests.DarlingObservabilityTests","Darling.Tests.DarlingPlanCorrectionLiveMigrationTests","Darling.Tests.DarlingServerTagsTests","Darling.Tests.DeadlockRateBandRungTests","Darling.Tests.DeltaFamilyIntervalColumnsRungTests","Darling.Tests.DeltaFamilyIntervalCompletionRungTests","Darling.Tests.FileGrowthAlertStoreTests","Darling.Tests.FleetSweepCadenceKnobRungTests","Darling.Tests.FleetSweepStateRungTests","Darling.Tests.IncidentOccurrenceAccumulatorTests","Darling.Tests.IndexObjectStatsServerTimeIndexRungTests","Darling.Tests.LivePostgresStoreFixtureTests","Darling.Tests.LongRunningQueryExclusionKnobRungTests","Darling.Tests.MonitoredEngineKindStoreTests","Darling.Tests.MuteRuleReloadBeaconTests","Darling.Tests.NotificationRoutesRungTests","Darling.Tests.OversizedPlanBacklogPins","Darling.Tests.PerfmonCounterTypeRungTests","Darling.Tests.Pg18IoBytesTests","Darling.Tests.PgAlertCountKnobRungTests","Darling.Tests.PgCpuCapacityHeadroomTests","Darling.Tests.PgDatabaseSizeStatsAndHostMemoryLivePostgresTests","Darling.Tests.PgDatabaseSizeStatsAndHostMemoryRungTests","Darling.Tests.PgIndexBloatEstimateRungTests","Darling.Tests.PgLogEventMetricsRungTests","Darling.Tests.PgLogEventsRungTests","Darling.Tests.PgNumbackendsAndSampledMsLivePostgresTests","Darling.Tests.PgNumbackendsAndSampledMsRungTests","Darling.Tests.PgServerConfigScopeRungTests","Darling.Tests.PgServerConfigTests","Darling.Tests.PgStatementTextTests","Darling.Tests.PlanContentRetentionTests","Darling.Tests.PlanForceActionStoreTests","Darling.Tests.PostgresMajorVersionRegistryTests","Darling.Tests.PostmasterStartTimeLivePostgresTests","Darling.Tests.PostmasterStartTimeRungTests","Darling.Tests.PvsStatsStoreTests","Darling.Tests.QsCaptureModeRouteKnobToastLivePostgresTests","Darling.Tests.QsCaptureModeRouteKnobToastRungTests","Darling.Tests.QueryStoreHealthStoreTests","Darling.Tests.QueryStoreIntervalLatestRungTests","Darling.Tests.QueryStoreTextStoreTests","Darling.Tests.ScaffoldTests","Darling.Tests.SelfDiskWarnGbFloorRungTests","Darling.Tests.StartupHostProfileLogTests","Darling.Tests.StoreLogSelfMonitoringStoreTests","Darling.Tests.StoreSelfMetricsTests","Darling.Tests.TempDbCeilingStoreTests","Darling.Tests.TimeHonestyRungLivePostgresTests","Darling.Tests.TimeHonestyRungTests","PerformanceMonitor.Darling.Tests.StoreConnectionSelfTestLiveTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (1)","build","Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"fd1fdb20d435d91f7da9aeb14ada2ac4fd81e22e","event":"pull_request","changed_files":["Lite.Tests/LiteMemoryFileIoTrendBucketingTests.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Mcp/McpIoTools.cs","Lite/Services/LocalDataService.FileIo.cs","Lite/Services/LocalDataService.Memory.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["build"]} +{"date":"2026-09-25","head_sha":"856a7a174823f846a8da60e66aa88ad2352d5dfe","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionCaveatsTests.cs","Darling/Darling.Tests/CollectorRuntimeStateTests.cs","Darling/Darling.Tests/DarlingPeerDisclosureTests.cs","Darling/Darling.Tests/DarlingStoreLoginsTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/FailureNoteRedactionTests.cs","Darling/Darling.Tests/WebExceptionTextCensusTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/CollectorRuntimeState.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingForcePlanTargetStateReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/PgPlanForceActionStore.cs","Lite.Tests/CollectionCaveatsTests.cs","Lite/Analysis/DuckDbFactCollector.cs","PerformanceMonitor.Analysis/AnalysisContext.cs"],"failed_classes":["Darling.Tests.PgTargetCollectionCaveatsLiveTests","Darling.Tests.SharedBaselineCacheTests","Darling.Tests.StartupFailureTriageTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"658bc329262ff59045315a0a8c52e4a6ff5712a3","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/ManagedConfFileLiveTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfVerdictsLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs"],"failed_classes":["Darling.Tests.ActivityDrivenPlanFetchStoreTests","Darling.Tests.CheckpointsTimedRungTests","Darling.Tests.CollectionCaveatsRungTests","Darling.Tests.CollectionLogDrainForensicsStoreTests","Darling.Tests.CollectionLogFanoutRollupStoreTests","Darling.Tests.CollectionLogFetchPhaseSumsLivePostgresTests","Darling.Tests.CollectionLogFetchPhaseSumsStoreTests","Darling.Tests.CollectionLogPhaseSplitStoreTests","Darling.Tests.CollectorDatabaseScopeRungTests","Darling.Tests.CollectorStallProbeStoreTests","Darling.Tests.ComposeStatementTimeoutStoreTests","Darling.Tests.CustomAlertCoreMigrationTests","Darling.Tests.DarlingAnalysisStoreTests","Darling.Tests.DarlingCustomViewsTests","Darling.Tests.DarlingObservabilityTests","Darling.Tests.DarlingPlanCorrectionLiveMigrationTests","Darling.Tests.DarlingServerTagsTests","Darling.Tests.DeadlockRateBandRungTests","Darling.Tests.DeltaFamilyIntervalColumnsRungTests","Darling.Tests.DeltaFamilyIntervalCompletionRungTests","Darling.Tests.FileGrowthAlertStoreTests","Darling.Tests.FleetSweepCadenceKnobRungTests","Darling.Tests.FleetSweepStateRungTests","Darling.Tests.IncidentOccurrenceAccumulatorTests","Darling.Tests.IndexObjectStatsServerTimeIndexRungTests","Darling.Tests.LivePostgresStoreFixtureTests","Darling.Tests.LongRunningQueryExclusionKnobRungTests","Darling.Tests.MonitoredEngineKindStoreTests","Darling.Tests.MuteRuleReloadBeaconTests","Darling.Tests.NotificationRoutesRungTests","Darling.Tests.OversizedPlanBacklogPins","Darling.Tests.PerfmonCounterTypeRungTests","Darling.Tests.Pg18IoBytesTests","Darling.Tests.PgAlertCountKnobRungTests","Darling.Tests.PgCpuCapacityHeadroomTests","Darling.Tests.PgDatabaseSizeStatsAndHostMemoryLivePostgresTests","Darling.Tests.PgDatabaseSizeStatsAndHostMemoryRungTests","Darling.Tests.PgIndexBloatEstimateRungTests","Darling.Tests.PgLogEventMetricsRungTests","Darling.Tests.PgLogEventsRungTests","Darling.Tests.PgNumbackendsAndSampledMsLivePostgresTests","Darling.Tests.PgNumbackendsAndSampledMsRungTests","Darling.Tests.PgServerConfigScopeRungTests","Darling.Tests.PgServerConfigTests","Darling.Tests.PgStatementTextTests","Darling.Tests.PlanContentRetentionTests","Darling.Tests.PlanForceActionStoreTests","Darling.Tests.PostgresMajorVersionRegistryTests","Darling.Tests.PostmasterStartTimeLivePostgresTests","Darling.Tests.PostmasterStartTimeRungTests","Darling.Tests.PvsStatsStoreTests","Darling.Tests.QsCaptureModeRouteKnobToastLivePostgresTests","Darling.Tests.QsCaptureModeRouteKnobToastRungTests","Darling.Tests.QueryStoreHealthStoreTests","Darling.Tests.QueryStoreIntervalLatestRungTests","Darling.Tests.QueryStoreTextStoreTests","Darling.Tests.ScaffoldTests","Darling.Tests.SelfDiskWarnGbFloorRungTests","Darling.Tests.StartupHostProfileLogTests","Darling.Tests.StoreLogSelfMonitoringStoreTests","Darling.Tests.StoreSelfMetricsTests","Darling.Tests.TempDbCeilingStoreTests","Darling.Tests.TimeHonestyRungLivePostgresTests","Darling.Tests.TimeHonestyRungTests","PerformanceMonitor.Darling.Tests.StoreConnectionSelfTestLiveTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (0)","Darling PG tests (1)","build"]} +{"date":"2026-09-25","head_sha":"02b7d9f692204d2cd3670007f36e1de42c7846a5","event":"pull_request","changed_files":["Darling/Darling.Tests/WebReadCancellationPinTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpConfigHistoryTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpConfigTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingServerResolver.cs"],"failed_classes":["Darling.Tests.EngineCapabilityReadWiringTests","Darling.Tests.McpServiceParameterDiSeatCensusTests","Darling.Tests.PgTargetMcpSurfaceTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-09-25","head_sha":"cb72c5d58312b81772a7332bbc5ab0adba833e7f","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreIntervalLatestRungTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideGateTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideGridLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideRungTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideSlicerLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideWriterTests.cs","Darling/Darling.Tests/QueryStoreTopMcpLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalWide.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStore.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Queries.cs"],"failed_classes":["Darling.Tests.McpReadCommandTimeoutTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-09-25","head_sha":"2cdce525a426660a1efbdd4e61af54f607cb8c9c","event":"pull_request","changed_files":["Darling/Darling.Tests/EngineCapabilityMissTests.cs","Darling/Darling.Tests/McpServiceParameterDiSeatCensusTests.cs","Darling/Darling.Tests/PgTargetMcpSurfaceTests.cs","Darling/Darling.Tests/WebReadCancellationPinTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpConfigHistoryTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpConfigTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingServerResolver.cs"],"failed_classes":["PerformanceMonitor.Darling.Tests.RetiredBaselineAggregateLiveTests","Darling.Tests.PayloadDimensionLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"bcf18734bc9c7563bb394cc866b30d1325f4dcce","event":"push","changed_files":["Darling/Darling.Tests/CollectionCaveatsTests.cs","Darling/Darling.Tests/CollectorRuntimeStateTests.cs","Darling/Darling.Tests/DarlingPeerDisclosureTests.cs","Darling/Darling.Tests/DarlingStoreLoginsTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/FailureNoteRedactionTests.cs","Darling/Darling.Tests/PgTargetCollectionCaveatsLiveTests.cs","Darling/Darling.Tests/SharedBaselineCacheTests.cs","Darling/Darling.Tests/StartupFailureTriageTests.cs","Darling/Darling.Tests/WebExceptionTextCensusTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/CollectorRuntimeState.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingForcePlanTargetStateReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/PgPlanForceActionStore.cs","Darling/PerformanceMonitor.Darling.Service/PlanForceBot.cs","Lite.Tests/CollectionCaveatsTests.cs","Lite/Analysis/DuckDbFactCollector.cs","PerformanceMonitor.Analysis/AnalysisContext.cs"],"failed_classes":["Darling.Tests.ComposeStoreRolesLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"3c16260c3c6e7618376774ebf362373001fc7726","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/EngineCapabilityMissTests.cs","Darling/Darling.Tests/McpServiceParameterDiSeatCensusTests.cs","Darling/Darling.Tests/PgTargetMcpSurfaceTests.cs","Darling/Darling.Tests/QueryStoreClutterTests.cs","Darling/Darling.Tests/WebReadCancellationPinTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpConfigHistoryTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpConfigTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpLongQueryTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanCorrectionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryHeatmapTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreClutterTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreRegressionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpSessionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingServerResolver.cs"],"failed_classes":["Darling.Tests.RuntimePreconditionReadWiringTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-09-25","head_sha":"49e5906ce37e10a213df6700179d7c9722078eca","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreIntervalWideGateTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideGridLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideMinSchemaVersionPinTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideRungTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideSlicerLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideWriterTests.cs","Darling/Darling.Tests/QueryStoreTopMcpLiveTests.cs","Darling/Darling.Tests/RawChunkIntervalRungHistoryRungTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalWide.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStore.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Queries.cs"],"failed_classes":["Darling.Tests.McpReadCommandTimeoutTests","Darling.Tests.QueryStoreIntervalWideWriterTests"],"failed_jobs":["build","Darling PG tests (0)","Darling PG tests (1)"]} +{"date":"2026-09-25","head_sha":"40284b006187de062fe300a3d12a2065c029cdbf","event":"pull_request","changed_files":["Darling/Darling.Tests/ViewerCpuTempDbTests.cs","Darling/Darling.Tests/ViewerFileIoBlockingTests.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.TempDb.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Charts.cs"],"failed_classes":["Darling.Tests.PayloadDimensionLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-25","head_sha":"54fc10e90c4ab93a32dea96bfae0df84f0b377bd","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeStoreRolesLiveTests.cs","Darling/Darling.Tests/StartupCommandTimeoutTests.cs","Darling/Darling.Tests/StartupFailureTriageTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreUpgrade.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ServiceCommandDeadlines.cs","Darling/PerformanceMonitor.Darling.Service/StartupFailureTriage.cs"],"failed_classes":["Darling.Tests.StoreSessionTimeZonePinCensusTests"],"failed_jobs":["Darling PG tests (1)","build"]} +{"date":"2026-09-26","head_sha":"0aa324bfab631d2e1f5336d757b24defb25ea2cf","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreIntervalWideGateTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideGridLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideMinSchemaVersionPinTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideRungTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideWriterTests.cs","Darling/Darling.Tests/QueryStoreTopMcpLiveTests.cs","Darling/Darling.Tests/RawChunkIntervalRungHistoryRungTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalWide.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStore.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Queries.cs"],"failed_classes":["Darling.Tests.AvailabilityGroupsTabRefreshTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-26","head_sha":"09c646b64c3908fd00f2107f0331ddb7f7013bd9","event":"pull_request","changed_files":["Darling/Darling.Tests/MemoryGrantTempDbBucketingLiveTests.cs","Darling/Darling.Tests/ViewerCpuTempDbTests.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Memory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.TempDb.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Charts.cs","Lite.Tests/MemoryGrantTempDbTrendBucketingTests.cs","Lite/Services/LocalDataService.MemoryGrants.cs","Lite/Services/LocalDataService.TempDb.cs"],"failed_classes":["Darling.Tests.DarlingMcpTrendToolsSurfaceAndSqlTests","Darling.Tests.ViewerMemorySqlTests","PerformanceMonitorLite.Tests.MeasurementContractCensusTests"],"failed_jobs":["build","Darling PG tests (1)","Lite tests (0)","Darling PG tests (0)"]} +{"date":"2026-09-26","head_sha":"a5f8d7d856353418e7c0a61fccb3547c94442f58","event":"pull_request","changed_files":["Darling/Darling.Tests/ViewerBlockingTrendBucketingTests.cs","Darling/Darling.Tests/ViewerFileIoBlockingTests.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.BlockingTrends.cs","Lite.Tests/LiteBlockingTrendBucketingTests.cs","Lite/Services/LocalDataService.Blocking.cs","Lite/Services/LocalDataService.WaitingTasks.cs"],"failed_classes":["Darling.Tests.ViewerBlockingTrendBucketingSqlTests"],"failed_jobs":["Darling PG tests (2)","build"]} +{"date":"2026-09-26","head_sha":"c2a3e18567ba32b90a95de4fcd0e34a0f8b8f9d8","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgServerStateTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgTrendTools.cs"],"failed_classes":["Darling.Tests.McpPageContractTests","Darling.Tests.PgServerConfigScopeRungTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (1)","build"]} +{"date":"2026-09-26","head_sha":"73737511b4ed3bb9a01b0128f94451218de4bf90","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreIntervalWideFaultInjectionCollection.cs","Darling/Darling.Tests/QueryStoreIntervalWideGateTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideGridLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideMinSchemaVersionPinTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideRungTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideWriterTests.cs","Darling/Darling.Tests/QueryStoreTopMcpLiveTests.cs","Darling/Darling.Tests/RawChunkIntervalRungHistoryRungTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalWide.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStore.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Queries.cs"],"failed_classes":["Darling.Tests.AbandonedRunEraInvariantReadTests","Darling.Tests.AlertStoredValueTests","Darling.Tests.AnalysisPageCapLiveTests","Darling.Tests.AnomalyNullWindowLiveTests","Darling.Tests.AsOfWindowAnchorLivePostgresTests","Darling.Tests.BaselineDiscontinuityLivePostgresTests","Darling.Tests.CollectionLogFetchPhaseSumsLivePostgresTests","Darling.Tests.CollectorCostMaterialityFloorTests","Darling.Tests.CollectorCostRegressionPerRunTests","Darling.Tests.CollectorTableCatalogShadowTests","Darling.Tests.CompareAnalysisPgBandingLiveTests","Darling.Tests.CustomAlertLastFiredLiveTests","Darling.Tests.CustomAlertStateStoreLiveTests","Darling.Tests.CustomAlertTagSubtreeLiveTests","Darling.Tests.DailySummaryReadShapeLiveTests","Darling.Tests.DarkPastRetentionReadsOfflineLivePostgresTests","Darling.Tests.DarlingAlertReadAdapterTests","Darling.Tests.DarlingAnalysisServiceEngineRoutingTests","Darling.Tests.DarlingAnomalyBaselineTests","Darling.Tests.DarlingCollectionLogReadTests","Darling.Tests.DarlingCollectorRunnerTests","Darling.Tests.DarlingCollectorToggleVerbLivePostgresTests","Darling.Tests.DarlingComposeLivePostgresTests","Darling.Tests.DarlingDeltaSeederTests","Darling.Tests.DarlingEndpointToggleCliTests","Darling.Tests.DarlingMcpAgToolsLivePostgresTests","Darling.Tests.DarlingMcpAnalysisFindingsBudgetLiveTests","Darling.Tests.DarlingMcpCustomViewToolsLivePostgresTests","Darling.Tests.DarlingMcpHealthToolsLivePostgresTests","Darling.Tests.DarlingMcpObjectStatsToolsLivePostgresTests","Darling.Tests.DarlingMcpQueryStoreRegressionsBudgetLiveTests","Darling.Tests.DarlingMcpStoreHostToolsLiveTests","Darling.Tests.DarlingModuleMapTests","Darling.Tests.DarlingObservabilityTests","Darling.Tests.DarlingPgIndexBloatNanLivePostgresTests","Darling.Tests.DarlingPgReadSqlParsesLiveTests","Darling.Tests.DarlingPgTopQueriesLiveTests","Darling.Tests.DarlingQueryHeatmapBudgetLiveTests","Darling.Tests.DarlingSecuritySplitLiveTests","Darling.Tests.DarlingSignificantWaitsReadTests","Darling.Tests.DarlingStoreHostProfileLiveTests","Darling.Tests.DarlingTrendEmptyTests","Darling.Tests.DarlingWatermarkSeedLiveTests","Darling.Tests.DatabaseSizeLatestPlanShapeLiveTests","Darling.Tests.DefaultTraceEventFrameLivePostgresTests","Darling.Tests.DrillDownDopProvenanceLiveTests","Darling.Tests.EngineAwareVersionLabelTests","Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests","Darling.Tests.FleetCardPostgresCpuLivePostgresTests","Darling.Tests.FleetCollectionHealthMemoLivePostgresTests","Darling.Tests.FleetReadsAreBoundedLivePostgresTests","Darling.Tests.HoldEarnedByDeliveryTests","Darling.Tests.JobHistoryGroupedStatsMatchWindowFunctionLiveTests","Darling.Tests.McpCollectionLogPerServerResponseBudgetLivePostgresTests","Darling.Tests.McpFleetResponseBudgetLivePostgresTests","Darling.Tests.McpReadToolBudgetLiveTests","Darling.Tests.NotificationRoutesLivePostgresTests","Darling.Tests.ObjectLockingBudgetLiveTests","Darling.Tests.OversizedPlanSweepRunRecordLivePostgresTests","Darling.Tests.ParameterSensitiveDrillDownTextLiveTests","Darling.Tests.PerServerCollectionHealthMemoLivePostgresTests","Darling.Tests.PgCopyWriterTests","Darling.Tests.PgFactCollectorTests","Darling.Tests.PgIoTrendDefaultBudgetLiveTests","Darling.Tests.PgLogEventMetricsLivePostgresTests","Darling.Tests.PgLogEventsCsvlogLiveTests","Darling.Tests.PgServerConfigToolBoundLiveTests","Darling.Tests.PgTargetAnomalyTests","Darling.Tests.PgTargetBufferDrillDownTests","Darling.Tests.PgTargetDeadlockDrillDownTests","Darling.Tests.PgTargetFactCollectorTests","Darling.Tests.PgTargetGrowthLiveTests","Darling.Tests.PgTargetKnobsTests","Darling.Tests.PgTargetPostureTests","Darling.Tests.PgTargetReplicationTests","Darling.Tests.PgTargetSeqScanTests","Darling.Tests.PlanCorrectionFrameLiveTests","Darling.Tests.PlanForceActionStoreTests","Darling.Tests.PlanRegressionDrillDownReuseLiveTests","Darling.Tests.PostgresEngineGateBehaviorTests","Darling.Tests.PostmasterStartTimeLivePostgresTests","Darling.Tests.PvsStatsStoreTests","Darling.Tests.QsCaptureModeRouteKnobToastLivePostgresTests","Darling.Tests.QueryStoreClutterLivePostgresTests","Darling.Tests.QueryStoreCorrectedRollupLiveTests","Darling.Tests.QueryStoreReplicaSplitAnalysisLiveTests","Darling.Tests.QueryStoreTopMcpLiveTests","Darling.Tests.RawWindowFloorMcpLiveTests","Darling.Tests.RdsLogTimezoneReadLiveTests","Darling.Tests.RuntimePreconditionMissLivePostgresTests","Darling.Tests.ServerListAndSummaryPlanShapeTests","Darling.Tests.StoreConfigProviderTests","Darling.Tests.StoreLogGroupingLiveTests","Darling.Tests.StoreSelfMetricsTests","Darling.Tests.TimeHonestyRungLivePostgresTests","Darling.Tests.TimescaleSupportTests","Darling.Tests.TrendBucketingLiveTests","Darling.Tests.TrendFamilyBucketingLiveTests","Darling.Tests.ViewerCalendarRetentionLivePostgresTests","Darling.Tests.ViewerMemoryLivePostgresTests","Darling.Tests.ViewerOverviewLanesLivePostgresTests","Darling.Tests.ViewerPlanCacheLivePostgresTests","Darling.Tests.ViewerPlanHostLivePostgresTests","Darling.Tests.ViewerRecommendationsLivePostgresTests","PerformanceMonitor.Darling.Tests.HostObjectRollupLiveTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-26","head_sha":"c93141671fb83a19c203df516a66b5693ad7b3ab","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpTrendToolsTests.cs","Darling/Darling.Tests/MemoryGrantTempDbBucketingLiveTests.cs","Darling/Darling.Tests/ViewerCpuTempDbTests.cs","Darling/Darling.Tests/ViewerMemoryTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingTrendReader.cs","Darling/PerformanceMonitor.Darling.Storage/TrendBucketSql.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Memory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.TempDb.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Charts.cs","Lite.Tests/MemoryGrantTempDbTrendBucketingTests.cs","Lite/Services/LocalDataService.MemoryGrants.cs","Lite/Services/LocalDataService.TempDb.cs"],"failed_classes":["Darling.Tests.ViewerMemoryLivePostgresTests","Darling.Tests.ViewerMemorySqlTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-09-26","head_sha":"6a334efcf1a0c00fc16cdc48bd56b6eccee9f4bc","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpTrendToolsTests.cs","Darling/Darling.Tests/MemoryGrantTempDbBucketingLiveTests.cs","Darling/Darling.Tests/ViewerCpuTempDbTests.cs","Darling/Darling.Tests/ViewerMemoryTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingTrendReader.cs","Darling/PerformanceMonitor.Darling.Storage/TrendBucketSql.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Memory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.TempDb.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Charts.cs","Lite.Tests/MemoryGrantTempDbTrendBucketingTests.cs","Lite/Services/LocalDataService.MemoryGrants.cs","Lite/Services/LocalDataService.TempDb.cs"],"failed_classes":["Darling.Tests.ViewerMemoryLivePostgresTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-26","head_sha":"9a7d5c7d28571b04021684e78dad38e226b46670","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpTrendToolsTests.cs","Darling/Darling.Tests/MemoryGrantTempDbBucketingLiveTests.cs","Darling/Darling.Tests/ViewerCpuTempDbTests.cs","Darling/Darling.Tests/ViewerMemoryTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingTrendReader.cs","Darling/PerformanceMonitor.Darling.Storage/TrendBucketSql.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Memory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.TempDb.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Charts.cs","Lite.Tests/MemoryGrantTempDbTrendBucketingTests.cs","Lite/Services/LocalDataService.MemoryGrants.cs","Lite/Services/LocalDataService.TempDb.cs"],"failed_classes":["Darling.Tests.ViewerCpuSessionsPlanCacheTrendBucketingLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"24cc9c02ded441440f16e6e6837bddc5d25169ae","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/Darling.Tests/ManagedConfFileLiveTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfVerdictsLiveTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/README.md","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests","Darling.Tests.CollectionLogDrainForensicsStoreTests","Darling.Tests.CollectionLogFetchPhaseSumsStoreTests","Darling.Tests.CollectionLogPhaseSplitStoreTests","Darling.Tests.CollectorDatabaseScopeRungTests","Darling.Tests.DeadlockRateBandRungTests","Darling.Tests.DeltaFamilyIntervalColumnsRungTests","Darling.Tests.FleetSweepCadenceKnobRungTests","Darling.Tests.FleetSweepStateRungTests","Darling.Tests.FleetSweepStoreLivePostgresTests","Darling.Tests.LongRunningQueryExclusionKnobRungTests","Darling.Tests.NotificationRoutesRungTests","Darling.Tests.PerfmonCounterTypeRungTests","Darling.Tests.PgAlertCountKnobRungTests","Darling.Tests.PgDatabaseSizeStatsAndHostMemoryRungTests","Darling.Tests.PgLogEventMetricsRungTests","Darling.Tests.PgNumbackendsAndSampledMsRungTests","Darling.Tests.PgServerConfigScopeRungTests","Darling.Tests.QsCaptureModeRouteKnobToastRungTests","Darling.Tests.QueryStoreIntervalWideRungTests","Darling.Tests.SelfDiskWarnGbFloorRungTests","Darling.Tests.StartupHostProfileLogTests","Darling.Tests.StoreLogViewerGateTests","Darling.Tests.TimeHonestyRungTests","Darling.Tests.ViewerSchemaVersionGateTests","Lite.Tests.CrossSkuSurfaceSourceTests"],"failed_jobs":["build","Darling PG tests (0)","Lite tests (3)","Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"fd948645346b2820a390012754c01988dee1a460","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/Darling.Tests/ManagedConfFileLiveTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfVerdictsLiveTests.cs","Darling/Darling.Tests/ManagedConfVerdictsRungTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideRungTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/CrossSkuSurfaceSourceTests.cs","PerformanceMonitor.Alerting/AlertReadFailureCounter.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests","Darling.Tests.ManagedConfVerdictsRungTests","Darling.Tests.StartupHostProfileLogTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (2)","Darling PG tests (1)","build"]} +{"date":"2026-09-26","head_sha":"2068d41265dfa37c53037460ec13ab7de9d159b4","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/Darling.Tests/ManagedConfFileLiveTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfVerdictsLiveTests.cs","Darling/Darling.Tests/ManagedConfVerdictsRungTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideRungTests.cs","Darling/Darling.Tests/StartupHostProfileLogTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/CrossSkuSurfaceSourceTests.cs","PerformanceMonitor.Alerting/AlertReadFailureCounter.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.ManagedConfFileTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"2e2c80f20af173a710ff73fa1e03f17d6522d21b","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreTopLiteralEndStraddleLiveTests.cs"],"failed_classes":["Darling.Tests.QueryStoreTopLiteralEndStraddleLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"ce5436d155c4713fac2947c02f302ee5d7a78e10","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreDurationTrendTableLiveTests.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryTrends.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.QueryStoreDurationTrendTableLiveTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-09-26","head_sha":"76d44be270b06891519a2add7fd70f9e9873a666","event":"pull_request","changed_files":["Darling/Darling.Tests/PgSensitiveStatementFilterLiveTests.cs","Darling/Darling.Tests/PgSensitiveStatementFilterTests.cs","Darling/Darling.Tests/PgStatementTextScrubLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/PgStatementTextScrub.cs","Darling/PerformanceMonitor.Darling.Storage/PgStatementText.cs","Darling/PerformanceMonitor.Darling.Storage/StoreStatementStats.cs","PerformanceMonitor.Collectors/PgBlockingCollector.cs","PerformanceMonitor.Collectors/PgSensitiveStatementFilter.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.PgSensitiveStatementFilterTests","PerformanceMonitorLite.Tests.ScenarioTests"],"failed_jobs":["build","Darling PG tests (0)","Darling PG tests (1)","Lite tests (1)"]} +{"date":"2026-09-26","head_sha":"c0adc53226670fdd89d7c28d782ee1b0be1e29b5","event":"pull_request","changed_files":["Darling/Darling.Tests/PlanForceActionAuditRedactionTests.cs","Darling/Darling.Tests/PlanForceActionDetailScrubLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/PlanForceActionDetailScrub.cs"],"failed_classes":["PerformanceMonitorLite.Tests.ScenarioTests"],"failed_jobs":["Lite tests (1)"]} +{"date":"2026-09-26","head_sha":"68ddf1746657e3a8013ee0002eedf4eb1488ac27","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreTopLiteralEndStraddleLiveTests.cs"],"failed_classes":["Darling.Tests.QueryStoreTopLiteralEndStraddleLiveTests","PerformanceMonitorLite.Tests.ScenarioTests"],"failed_jobs":["Darling PG tests (2)","Lite tests (1)"]} +{"date":"2026-09-26","head_sha":"a75333e8e236710a1e864c4624de26cf8bb67988","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreDurationTrendTableLiveTests.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalWide.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryTrends.cs"],"failed_classes":["Darling.Tests.QueryStoreDurationTrendTableLiveTests","Darling.Tests.QueryStoreIntervalWideGridLiveTests","Darling.Tests.QueryStoreTopMcpLiveTests","PerformanceMonitorLite.Tests.ScenarioTests"],"failed_jobs":["Lite tests (1)","Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-09-26","head_sha":"8754210ebf37f73ca1dd120d8f6960477745d909","event":"pull_request","changed_files":["Darling/Darling.Tests/PgSensitiveStatementFilterLiveTests.cs","Darling/Darling.Tests/PgSensitiveStatementFilterTests.cs","Darling/Darling.Tests/PgStatementTextScrubLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/PgStatementTextScrub.cs","Darling/PerformanceMonitor.Darling.Storage/PgStatementText.cs","Darling/PerformanceMonitor.Darling.Storage/StoreStatementStats.cs","PerformanceMonitor.Collectors/PgBlockingCollector.cs","PerformanceMonitor.Collectors/PgSensitiveStatementFilter.cs"],"failed_classes":["Darling.Tests.PgDeadlockNormalizationTests","PerformanceMonitorLite.Tests.ScenarioTests"],"failed_jobs":["Lite tests (1)","Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"801cb7c1d87049e38d7c7e6fbbc0cafbe72fe0fe","event":"pull_request","changed_files":["Darling/Darling.Tests/PlanForceActionAuditRedactionTests.cs","Darling/Darling.Tests/PlanForceActionDetailScrubLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/PlanForceActionDetailScrub.cs"],"failed_classes":["Darling.Tests.PlanForceActionAuditRedactionTests","PerformanceMonitorLite.Tests.ScenarioTests"],"failed_jobs":["build","Darling PG tests (1)","Lite tests (1)"]} +{"date":"2026-09-26","head_sha":"cbc985e2a5a4c942d7c0907210f211c8b23bde8c","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreDurationTrendTableLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideGridLiveTests.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalWide.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryTrends.cs"],"failed_classes":["Darling.Tests.QueryStoreTopMcpLiveTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-26","head_sha":"8d29921a1fa552050083c7ab202f43e96e2bb65d","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/Darling.Tests/ManagedConfFileLiveTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfVerdictsLiveTests.cs","Darling/Darling.Tests/ManagedConfVerdictsRungTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideRungTests.cs","Darling/Darling.Tests/StartupHostProfileLogTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/CrossSkuSurfaceSourceTests.cs","PerformanceMonitor.Alerting/AlertReadFailureCounter.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.ManagedConfMigrationTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-09-26","head_sha":"8a4c6977c8386c0d2da0e58175ad3a3b7ca164ac","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingServerResolver.cs"],"failed_classes":["Darling.Tests.CollectionHealthPayloadBudgetLiveTests","Darling.Tests.OversizedPlanSweepRunRecordTests","Darling.Tests.PerServerCollectionHealthMemoTests"],"failed_jobs":["build","Darling PG tests (2)","Darling PG tests (0)"]} +{"date":"2026-09-26","head_sha":"3efe402962c866944b24b5fa91d71584a660e4e7","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/Darling.Tests/ManagedConfFileLiveTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfVerdictsLiveTests.cs","Darling/Darling.Tests/ManagedConfVerdictsRungTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideRungTests.cs","Darling/Darling.Tests/StartupHostProfileLogTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/CrossSkuSurfaceSourceTests.cs","PerformanceMonitor.Alerting/AlertReadFailureCounter.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.ManagedConfMigrationTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-09-26","head_sha":"c3c584dced85e8a09737031271b14d41dee39d80","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpAlertToolsTests.cs","Darling/Darling.Tests/MuteRuleReloadRetentionTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAlertTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHealthTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpMemoryGrantTools.cs","Darling/PerformanceMonitor.Darling.Service/PgMuteRuleStore.cs","Lite.Tests/MuteRuleServiceTests.cs","Lite.Tests/RemainingEmptyReadsToolTests.cs","Lite/Services/DuckDbMuteRuleStore.cs","PerformanceMonitor.Notifications/IMuteRuleStore.cs","deprecated/Dashboard/Services/JsonMuteRuleStore.cs"],"failed_classes":["Darling.Tests.QueryStoreTopLiteralEndStraddleLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"87c7b1bbe5a117919efe954509cc9c5cb981d709","event":"pull_request","changed_files":["Darling/Darling.Tests/PlanForceActionAuditRedactionTests.cs","Darling/Darling.Tests/PlanForceActionDetailCensus.cs","Darling/Darling.Tests/PlanForceActionDetailCensusTests.cs","Darling/Darling.Tests/PlanForceActionDetailScrubLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/PlanForceActionDetailScrub.cs"],"failed_classes":["Darling.Tests.QueryStoreTopLiteralEndStraddleLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"c9a1a8145638896f27068c658ee343a4b17daaaa","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionHealthPayloadBudgetLiveTests.cs","Darling/Darling.Tests/OversizedPlanSweepRunRecordTests.cs","Darling/Darling.Tests/PerServerCollectionHealthMemoTests.cs","Darling/Darling.Tests/WebReadCancellationPinTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingServerResolver.cs"],"failed_classes":["Darling.Tests.QueryStoreTopLiteralEndStraddleLiveTests","Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"34a57f6f2ba27f30b6b37c55852557807c9d3f6b","event":"push","changed_files":["Darling/Darling.Tests/PgSensitiveStatementFilterLiveTests.cs","Darling/Darling.Tests/PgSensitiveStatementFilterTests.cs","Darling/Darling.Tests/PgStatementTextScrubLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/PgStatementTextScrub.cs","Darling/PerformanceMonitor.Darling.Storage/PgStatementText.cs","Darling/PerformanceMonitor.Darling.Storage/StoreStatementStats.cs","PerformanceMonitor.Collectors/PgBlockingCollector.cs","PerformanceMonitor.Collectors/PgSensitiveStatementFilter.cs"],"failed_classes":["Darling.Tests.QueryStoreTopLiteralEndStraddleLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"be67f6a2f3a8c0be1bd7d2ce049f6c9cfbe5e189","event":"pull_request","changed_files":["Darling/Darling.Tests/BaselineSupplyTests.cs","Darling/Darling.Tests/CompressionPhaseAssignmentTests.cs","Darling/Darling.Tests/DailySummaryNotCarriedTests.cs","Darling/Darling.Tests/DailySummaryReadShapeTests.cs","Darling/Darling.Tests/DailySummaryStitchedRangeTests.cs","Darling/Darling.Tests/FrozenRollupLiveTests.cs","Darling/Darling.Tests/IntervalHonestHourlyRollupTests.cs","Darling/Darling.Tests/MaterializationHoleRepairTests.cs","Darling/Darling.Tests/MaterializationHoleScanShapeTests.cs","Darling/Darling.Tests/MeasurementContractCensusTests.cs","Darling/Darling.Tests/PayloadDimensionLiveTests.cs","Darling/Darling.Tests/QueryStoreCorrectedRollupLiveTests.cs","Darling/Darling.Tests/RefreshCeilingProvenancePinTests.cs","Darling/Darling.Tests/RefreshCeilingStalenessTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/RollupBackfillLiveTests.cs","Darling/Darling.Tests/RollupBackfillTests.cs","Darling/Darling.Tests/RollupCoverageRoutingTests.cs","Darling/Darling.Tests/SuccessorDailyLiveTests.cs","Darling/Darling.Tests/TimescaleAggregateCompressionTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/Darling.Tests/TimescaleSupportTests.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummarySql.cs","Darling/PerformanceMonitor.Darling.Storage/RollupBackfill.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.MaterializationHoles.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.QueryStoreTopLiteralEndStraddleLiveTests","Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"9174da32aa8730623280d202d522ebfb15fb0320","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/Darling.Tests/ManagedConfFileLiveTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfRewriteTests.cs","Darling/Darling.Tests/ManagedConfVerdictsLiveTests.cs","Darling/Darling.Tests/ManagedConfVerdictsRungTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideRungTests.cs","Darling/Darling.Tests/StartupHostProfileLogTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/CrossSkuSurfaceSourceTests.cs","PerformanceMonitor.Alerting/AlertReadFailureCounter.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.ManagedConfMigrationTests","Darling.Tests.ManagedConfRewriteTests","Darling.Tests.QueryStoreTopLiteralEndStraddleLiveTests","Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["build","Darling PG tests (2)","Darling PG tests (0)","Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"f0dcb2fbc0e28f71fe93b5d881508129cb4e2f36","event":"pull_request","changed_files":["Darling/Darling.Tests/CapturingTestLogger.cs","Darling/Darling.Tests/PgSettingRedactorTests.cs","Darling/Darling.Tests/PgSettingScrubCandidateCensusTests.cs","Darling/Darling.Tests/PgSettingScrubLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/PgSettingScrub.cs","PerformanceMonitor.Collectors/PgSettingRedactor.cs"],"failed_classes":["Darling.Tests.QueryStoreTopLiteralEndStraddleLiveTests","Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"f93a136293c3981ff088c5fda99680fdb5e557f8","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreTopLiteralEndStraddleLiveTests.cs"],"failed_classes":["Darling.Tests.QueryStoreTopLiteralEndStraddleLiveTests","Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"27a93fcb1adb5474926ab0dfbb7cc889c88660e1","event":"pull_request","changed_files":[".gitattributes","Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/Darling.Tests/Fixtures/ManagedConf/rehearsal-postgresql.conf","Darling/Darling.Tests/ManagedConfFileLiveTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/Darling.Tests/ManagedConfRewriteTests.cs","Darling/Darling.Tests/ManagedConfVerdictsLiveTests.cs","Darling/Darling.Tests/ManagedConfVerdictsRungTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideRungTests.cs","Darling/Darling.Tests/StartupHostProfileLogTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/CrossSkuSurfaceSourceTests.cs","PerformanceMonitor.Alerting/AlertReadFailureCounter.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.QueryStoreTopLiteralEndStraddleLiveTests","Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"239d269d68369d473f8e7bb96a3fe290a3858347","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreTopLiteralEndStraddleLiveTests.cs","Darling/Darling.Tests/SharedBaselineCacheTests.cs"],"failed_classes":["Darling.Tests.QueryStoreTopLiteralEndStraddleLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"251e2adfeb10aa937f40b635bcb3668e635a280d","event":"pull_request","changed_files":[".gitattributes","Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/Darling.Tests/Fixtures/ManagedConf/rehearsal-postgresql.conf","Darling/Darling.Tests/ManagedConfFileLiveTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationStepsTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/Darling.Tests/ManagedConfRewriteTests.cs","Darling/Darling.Tests/ManagedConfVerdictsLiveTests.cs","Darling/Darling.Tests/ManagedConfVerdictsRungTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideRungTests.cs","Darling/Darling.Tests/StartupHostProfileLogTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationSteps.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/CrossSkuSurfaceSourceTests.cs","PerformanceMonitor.Alerting/AlertReadFailureCounter.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.QueryStoreTopLiteralEndStraddleLiveTests","Darling.Tests.SharedBaselineCacheLiveTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"c618155e2e0e11fc0cb4edd5d2b369e01733b331","event":"pull_request","changed_files":["Darling/Darling.Tests/PlanForceActionAuditRedactionTests.cs","Darling/Darling.Tests/PlanForceActionDetailCensus.cs","Darling/Darling.Tests/PlanForceActionDetailCensusTests.cs","Darling/Darling.Tests/PlanForceActionDetailScrubLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/PlanForceActionDetailScrub.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"ded542a0563750d27f27c24bf53537a45c448f3e","event":"pull_request","changed_files":[".gitattributes","Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/Darling.Tests/Fixtures/ManagedConf/rehearsal-postgresql.conf","Darling/Darling.Tests/ManagedConfFileLiveTests.cs","Darling/Darling.Tests/ManagedConfFileSettingsTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationRunnerTests.cs","Darling/Darling.Tests/ManagedConfMigrationStateTests.cs","Darling/Darling.Tests/ManagedConfMigrationStepsTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfMigrationWiringTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/Darling.Tests/ManagedConfRewriteTests.cs","Darling/Darling.Tests/ManagedConfVerdictsLiveTests.cs","Darling/Darling.Tests/ManagedConfVerdictsRungTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideRungTests.cs","Darling/Darling.Tests/StartupHostProfileLogTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFileSettings.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationRunner.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationState.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationSteps.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/CrossSkuSurfaceSourceTests.cs","PerformanceMonitor.Alerting/AlertReadFailureCounter.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.DarlingManagedPostgresTests","Darling.Tests.ManagedConfFileLiveTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"f7499cfb2762284862461b57f386f4fba596aaaa","event":"pull_request","changed_files":["Darling/Darling.Tests/PgSettingRedactorTests.cs","Darling/PerformanceMonitor.Darling.Service/PgSettingScrub.cs","PerformanceMonitor.Collectors/PgServerConfigCollector.cs","PerformanceMonitor.Collectors/PgSettingRedactor.cs"],"failed_classes":["Darling.Tests.PgSettingRedactorTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"2ccc5e07a9c272b7b783f699e445c15496329e0c","event":"pull_request","changed_files":["Darling/Darling.Tests/RawArmedStateConvergeTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.FrozenRollupLiveTests","Darling.Tests.QueryStoreCorrectedRollupLiveTests","Darling.Tests.RetentionReevaluationLiveTests","Darling.Tests.RollupBackfillLiveTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-09-26","head_sha":"38adaf03d7f14d907b61a51c3b1822e85415aa47","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAgTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDefaultTraceTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpJobTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpLatchSpinlockTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanCacheSchedulerTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPvsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpSessionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTrendTools.cs"],"failed_classes":["Darling.Tests.McpLatestSnapshotStampTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-09-26","head_sha":"521ee52d394f25263bc82c82b64b535d7431e07a","event":"pull_request","changed_files":[".gitattributes","Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/Darling.Tests/Fixtures/ManagedConf/rehearsal-postgresql.conf","Darling/Darling.Tests/ManagedConfFileLiveTests.cs","Darling/Darling.Tests/ManagedConfFileSettingsTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationRunnerTests.cs","Darling/Darling.Tests/ManagedConfMigrationStateTests.cs","Darling/Darling.Tests/ManagedConfMigrationStepsTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfMigrationWiringTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/Darling.Tests/ManagedConfRewriteTests.cs","Darling/Darling.Tests/ManagedConfVerdictsLiveTests.cs","Darling/Darling.Tests/ManagedConfVerdictsRungTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideRungTests.cs","Darling/Darling.Tests/StartupHostProfileLogTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFileSettings.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationRunner.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationState.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationSteps.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/CrossSkuSurfaceSourceTests.cs","PerformanceMonitor.Alerting/AlertReadFailureCounter.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.DarlingManagedPostgresTests","Darling.Tests.ManagedConfFileLiveTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"74797c4b9f8e0732a60b5d275c37ac475b5c102a","event":"pull_request","changed_files":["Darling/Darling.Tests/RawArmedStateConvergeTests.cs","Darling/Darling.Tests/RawRetentionServiceTriggeredLiveTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.FrozenRollupLiveTests","Darling.Tests.QueryStoreCorrectedRollupLiveTests","Darling.Tests.RetentionReevaluationLiveTests","Darling.Tests.RollupBackfillLiveTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"4bbb3138658b9585aba7366d986d9cdbe1137a4d","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertNotebookAuthoredTemplateTests.cs","Darling/PerformanceMonitor.Darling.Service/AlertNotebookEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["Darling PG tests (1)","build"]} +{"date":"2026-09-26","head_sha":"57c3c14fa4c7f61205b4454d6101ff17b583c876","event":"pull_request","changed_files":[".gitattributes","Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/Darling.Tests/Fixtures/ManagedConf/rehearsal-postgresql.conf","Darling/Darling.Tests/ManagedConfFileLiveTests.cs","Darling/Darling.Tests/ManagedConfFileSettingsTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationRunnerTests.cs","Darling/Darling.Tests/ManagedConfMigrationStateTests.cs","Darling/Darling.Tests/ManagedConfMigrationStepsTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfMigrationWiringTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/Darling.Tests/ManagedConfRewriteTests.cs","Darling/Darling.Tests/ManagedConfVerdictsLiveTests.cs","Darling/Darling.Tests/ManagedConfVerdictsRungTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideRungTests.cs","Darling/Darling.Tests/StartupHostProfileLogTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFileSettings.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationRunner.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationState.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationSteps.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/CrossSkuSurfaceSourceTests.cs","PerformanceMonitor.Alerting/AlertReadFailureCounter.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.DarlingManagedPostgresTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"241557c7d44f3f85c49de71928847e7a0fb2ead0","event":"pull_request","changed_files":[".gitattributes","Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/Darling.Tests/Fixtures/ManagedConf/rehearsal-postgresql.conf","Darling/Darling.Tests/ManagedConfFileLiveTests.cs","Darling/Darling.Tests/ManagedConfFileSettingsTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationRunnerTests.cs","Darling/Darling.Tests/ManagedConfMigrationStateTests.cs","Darling/Darling.Tests/ManagedConfMigrationStepsTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfMigrationWiringTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/Darling.Tests/ManagedConfRewriteTests.cs","Darling/Darling.Tests/ManagedConfVerdictsLiveTests.cs","Darling/Darling.Tests/ManagedConfVerdictsRungTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideRungTests.cs","Darling/Darling.Tests/StartupHostProfileLogTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFileSettings.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationRunner.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationState.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationSteps.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/CrossSkuSurfaceSourceTests.cs","PerformanceMonitor.Alerting/AlertReadFailureCounter.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.DarlingManagedPostgresTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"7d1627b90bf2808b1c5142ab9a281fd76667ca58","event":"pull_request","changed_files":["Darling/Darling.Tests/TopQueriesHourlyRoutingLiveTests.cs","Darling/Darling.Tests/TopQueriesHourlyRoutingTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStats.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Queries.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"5f1034a36c1cc57b2d02b86bbfa94fbaff66ad1c","event":"pull_request","changed_files":["Darling/Darling.Tests/ServerWatermarkCacheRunnerLiveTests.cs","Darling/Darling.Tests/ServerWatermarkCacheTests.cs","Darling/Darling.Tests/TimeHonestyRungTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/ServerWatermarkCache.cs","PerformanceMonitor.Collectors/CollectorDefinitionBase.cs","PerformanceMonitor.Collectors/DefaultTraceEventsCollector.cs","PerformanceMonitor.Collectors/ICollectorDefinition.cs","PerformanceMonitor.Collectors/JobHistoryCollector.cs","PerformanceMonitor.Collectors/MemoryPressureEventsCollector.cs","PerformanceMonitor.Collectors/SystemHealthEventsCollector.cs"],"failed_classes":["Darling.Tests.ServerWatermarkCacheRunnerLiveTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-26","head_sha":"7b00aa01ec2215860f3b28cb6351033f594e1f16","event":"pull_request","changed_files":[".gitattributes","Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/Darling.Tests/Fixtures/ManagedConf/rehearsal-postgresql.conf","Darling/Darling.Tests/ManagedConfFileLiveTests.cs","Darling/Darling.Tests/ManagedConfFileSettingsTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationRunnerTests.cs","Darling/Darling.Tests/ManagedConfMigrationStateTests.cs","Darling/Darling.Tests/ManagedConfMigrationStepsTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfMigrationWiringTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/Darling.Tests/ManagedConfRewriteTests.cs","Darling/Darling.Tests/ManagedConfUpgradePathTests.cs","Darling/Darling.Tests/ManagedConfVerdictsLiveTests.cs","Darling/Darling.Tests/ManagedConfVerdictsRungTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideRungTests.cs","Darling/Darling.Tests/StartupHostProfileLogTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFileSettings.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationRunner.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationState.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationSteps.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/CrossSkuSurfaceSourceTests.cs","PerformanceMonitor.Alerting/AlertReadFailureCounter.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.DarlingManagedPostgresTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"3878626a2c107e15d77b2323fafc787cc22d2ebc","event":"pull_request","changed_files":[".gitattributes","Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/DarlingStoreHostProfileTests.cs","Darling/Darling.Tests/Fixtures/ManagedConf/rehearsal-postgresql.conf","Darling/Darling.Tests/ManagedConfFileLiveTests.cs","Darling/Darling.Tests/ManagedConfFileSettingsTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationRunnerTests.cs","Darling/Darling.Tests/ManagedConfMigrationStateTests.cs","Darling/Darling.Tests/ManagedConfMigrationStepsTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfMigrationWiringTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/Darling.Tests/ManagedConfRewriteTests.cs","Darling/Darling.Tests/ManagedConfUpgradePathTests.cs","Darling/Darling.Tests/ManagedConfVerdictsLiveTests.cs","Darling/Darling.Tests/ManagedConfVerdictsRungTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideRungTests.cs","Darling/Darling.Tests/StartupHostProfileLogTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFileSettings.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationRunner.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationState.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigrationSteps.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/CrossSkuSurfaceSourceTests.cs","PerformanceMonitor.Alerting/AlertReadFailureCounter.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.DarlingManagedPostgresTests","Darling.Tests.DocCommentHygieneTests","Darling.Tests.StoreSessionTimeZonePinCensusTests","Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"7540d1cb4b51fe5214a6b8ef6c1db0aa993e99cc","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpCollectorCostTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFleetSweepTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFleetTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpOversizedPlanBacklogTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgBlockingTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStallProbeTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreLogTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreMetricsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreQueryStatsTools.cs"],"failed_classes":["Darling.Tests.OversizedPlanBacklogReadPins","Darling.Tests.StoreToastAndCheckpointerTests"],"failed_jobs":["Darling PG tests (2)","build"]} +{"date":"2026-09-26","head_sha":"0c3ed6a648f70bfdd8e9a580b712199e6a982148","event":"pull_request","changed_files":["Darling/Darling.Tests/FrozenRollupLiveTests.cs","Darling/Darling.Tests/MaterializationHoleRepairTests.cs","Darling/Darling.Tests/QueryStoreCorrectedRollupLiveTests.cs","Darling/Darling.Tests/RawArmedStateConvergeTests.cs","Darling/Darling.Tests/RawRepairEpochStampTests.cs","Darling/Darling.Tests/RawRepairEpochTriggerLiveTests.cs","Darling/Darling.Tests/RawRetentionPurgeJobLiveTests.cs","Darling/Darling.Tests/RawRetentionServiceTriggeredLiveTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/RollupBackfillLiveTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.MaterializationHoles.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.DarlingCloudIdentityProbeTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"dfb6b826c1769983340444e73c892fdbbeb51866","event":"pull_request","changed_files":["Darling/Darling.Tests/FrozenRollupLiveTests.cs","Darling/Darling.Tests/MaterializationHoleRepairTests.cs","Darling/Darling.Tests/QueryStoreCorrectedRollupLiveTests.cs","Darling/Darling.Tests/RawArmedStateConvergeTests.cs","Darling/Darling.Tests/RawPurgeTriggerLiveTests.cs","Darling/Darling.Tests/RawRepairEpochStampTests.cs","Darling/Darling.Tests/RawRepairEpochTriggerLiveTests.cs","Darling/Darling.Tests/RawRetentionPurgeJobLiveTests.cs","Darling/Darling.Tests/RawRetentionServiceTriggeredLiveTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/RollupBackfillLiveTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.MaterializationHoles.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.DarlingSelfAlertTests"],"failed_jobs":["Darling PG tests (2)","build"]} +{"date":"2026-09-26","head_sha":"1cb56d0963ca3829cfcb5a6ceab01e3375bdbd3f","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/FrozenRollupLiveTests.cs","Darling/Darling.Tests/MaterializationHoleRepairTests.cs","Darling/Darling.Tests/QueryStoreCorrectedRollupLiveTests.cs","Darling/Darling.Tests/RawArmedStateConvergeTests.cs","Darling/Darling.Tests/RawPurgeTriggerLiveTests.cs","Darling/Darling.Tests/RawRepairEpochStampTests.cs","Darling/Darling.Tests/RawRepairEpochTriggerLiveTests.cs","Darling/Darling.Tests/RawRetentionPurgeJobLiveTests.cs","Darling/Darling.Tests/RawRetentionServiceTriggeredLiveTests.cs","Darling/Darling.Tests/RetentionHoldAndCadenceRawRedirectLiveTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/RollupBackfillLiveTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.MaterializationHoles.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests","Darling.Tests.DarlingTriageEndpointTests","Darling.Tests.RetentionHoldRatioKnobRungTests","Darling.Tests.TimescaleAvailabilityReprobeTests"],"failed_jobs":["Darling PG tests (0)","build","Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"af4c1a327469a4b3ae2dca3a6cc8e9d21336f8d8","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/FrozenRollupLiveTests.cs","Darling/Darling.Tests/MaterializationHoleRepairTests.cs","Darling/Darling.Tests/QueryStoreCorrectedRollupLiveTests.cs","Darling/Darling.Tests/RawArmedStateConvergeTests.cs","Darling/Darling.Tests/RawPurgeTriggerLiveTests.cs","Darling/Darling.Tests/RawRepairEpochStampTests.cs","Darling/Darling.Tests/RawRepairEpochTriggerLiveTests.cs","Darling/Darling.Tests/RawRetentionPurgeJobLiveTests.cs","Darling/Darling.Tests/RawRetentionServiceTriggeredLiveTests.cs","Darling/Darling.Tests/RetentionHoldAndCadenceRawRedirectLiveTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/RollupBackfillLiveTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.MaterializationHoles.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests","Darling.Tests.DarlingSelfAlertTests","Darling.Tests.DarlingTriageEndpointTests","Darling.Tests.DocCommentHygieneTests","Darling.Tests.RetentionHoldRatioKnobRungTests","Darling.Tests.TimescaleAvailabilityReprobeTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (0)","Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"d28291f02eb14bc8d40ae7c2c581e94bd920cd42","event":"pull_request","changed_files":["Darling/Darling.Tests/TopQueriesHourlyRoutingLiveTests.cs","Darling/Darling.Tests/TopQueriesHourlyRoutingTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStats.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Queries.cs"],"failed_classes":["Darling.Tests.ManagedConfFileLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"79c8fb6ee52e5400026b953dc0ac6198426ccbe0","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/FrozenRollupLiveTests.cs","Darling/Darling.Tests/MaterializationHoleRepairTests.cs","Darling/Darling.Tests/QueryStoreCorrectedRollupLiveTests.cs","Darling/Darling.Tests/RawArmedStateConvergeTests.cs","Darling/Darling.Tests/RawPurgeTriggerLiveTests.cs","Darling/Darling.Tests/RawRepairEpochStampTests.cs","Darling/Darling.Tests/RawRepairEpochTriggerLiveTests.cs","Darling/Darling.Tests/RawRetentionPurgeJobLiveTests.cs","Darling/Darling.Tests/RawRetentionServiceTriggeredLiveTests.cs","Darling/Darling.Tests/RetentionHoldAndCadenceRawRedirectLiveTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/RollupBackfillLiveTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.MaterializationHoles.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.DarlingTriageEndpointTests","Darling.Tests.RetentionHoldRatioKnobRungTests","Darling.Tests.TimescaleAvailabilityReprobeTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-09-26","head_sha":"52f9c129c25c7a0b1f99eac39e616bef22f10355","event":"pull_request","changed_files":["Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgAutovacuumTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgDatabaseTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgIndexUsageTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgIoTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgKernelStatsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgPredicateTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgReplicationStatsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgSessionStatesTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgWraparoundTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgXminTools.cs"],"failed_classes":["Darling.Tests.PgLogEventMetricsParserTests"],"failed_jobs":["Darling PG tests (2)","build"]} +{"date":"2026-09-26","head_sha":"e51642f32a5ad0988c7642887b17933d93dccd58","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/FrozenRollupLiveTests.cs","Darling/Darling.Tests/MaterializationHoleRepairTests.cs","Darling/Darling.Tests/QueryStoreCorrectedRollupLiveTests.cs","Darling/Darling.Tests/RawArmedStateConvergeTests.cs","Darling/Darling.Tests/RawPurgeTriggerLiveTests.cs","Darling/Darling.Tests/RawRepairEpochStampTests.cs","Darling/Darling.Tests/RawRepairEpochTriggerLiveTests.cs","Darling/Darling.Tests/RawRetentionPurgeJobLiveTests.cs","Darling/Darling.Tests/RawRetentionServiceTriggeredLiveTests.cs","Darling/Darling.Tests/RetentionHoldAndCadenceRawRedirectLiveTests.cs","Darling/Darling.Tests/RetentionHoldRatioKnobRungTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/RollupBackfillLiveTests.cs","Darling/Darling.Tests/TimescaleAvailabilityReprobeTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.MaterializationHoles.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.NotificationRoutingTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"63ba0d2893cf51906e9d40d05cf8592f8a84733e","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgCpuUtilizationTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgIndexTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgLogEventTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgLoggingAuditTools.cs"],"failed_classes":["PerformanceMonitorLite.Tests.DuckDbSentinelConnectionTests"],"failed_jobs":["Lite tests (0)"]} +{"date":"2026-09-26","head_sha":"dbbaa4a52050d5cbdb9c9c0678523fd6dbfc42a8","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/FrozenRollupLiveTests.cs","Darling/Darling.Tests/MaterializationHoleRepairTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QueryStoreCorrectedRollupLiveTests.cs","Darling/Darling.Tests/RawArmedStateConvergeTests.cs","Darling/Darling.Tests/RawPurgeTriggerLiveTests.cs","Darling/Darling.Tests/RawRepairEpochStampTests.cs","Darling/Darling.Tests/RawRepairEpochTriggerLiveTests.cs","Darling/Darling.Tests/RawRetentionPurgeJobLiveTests.cs","Darling/Darling.Tests/RawRetentionServiceTriggeredLiveTests.cs","Darling/Darling.Tests/RetentionHoldAndCadenceRawRedirectLiveTests.cs","Darling/Darling.Tests/RetentionHoldRatioKnobRungTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/RollupBackfillLiveTests.cs","Darling/Darling.Tests/TimescaleAvailabilityReprobeTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.MaterializationHoles.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.QueryStoreIntervalWideGridLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"f526e44a7ad05ae3bd5c2df57952605a74664578","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFindingStore.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTrendTools.cs"],"failed_classes":["Darling.Tests.AnalysisPassTokenThreadingTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"76d20735c22f07203c5c30ab5669c1a09a5bb810","event":"pull_request","changed_files":["Darling/Darling.Tests/AnalysisPassTokenThreadingTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFindingStore.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTrendTools.cs"],"failed_classes":["Darling.Tests.AnalysisPassTokenThreadingTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"cac3ede73b6e4e207379f8f7b04bb5f4f180f8af","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/FrozenRollupLiveTests.cs","Darling/Darling.Tests/MaterializationHoleRepairTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QueryStoreCorrectedRollupLiveTests.cs","Darling/Darling.Tests/RawArmedStateConvergeTests.cs","Darling/Darling.Tests/RawPurgeConvergeLiveTests.cs","Darling/Darling.Tests/RawPurgeTriggerGateErrorTests.cs","Darling/Darling.Tests/RawPurgeTriggerLiveTests.cs","Darling/Darling.Tests/RawRepairEpochStampTests.cs","Darling/Darling.Tests/RawRepairEpochTriggerLiveTests.cs","Darling/Darling.Tests/RawRetentionPurgeJobLiveTests.cs","Darling/Darling.Tests/RawRetentionServiceTriggeredLiveTests.cs","Darling/Darling.Tests/RetentionHoldAndCadenceRawRedirectLiveTests.cs","Darling/Darling.Tests/RetentionHoldRatioKnobRungTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/RollupBackfillLiveTests.cs","Darling/Darling.Tests/TimescaleAvailabilityReprobeTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.MaterializationHoles.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","PerformanceMonitorLite.Tests.AnalysisPassTokenThreadingTests"],"failed_jobs":["Darling PG tests (1)","Lite tests (3)","build"]} +{"date":"2026-09-26","head_sha":"ea3d63c3e6a41686595c95a7e58bfb1318b3f359","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/FrozenRollupLiveTests.cs","Darling/Darling.Tests/MaterializationHoleRepairTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/QueryStoreCorrectedRollupLiveTests.cs","Darling/Darling.Tests/RawArmedStateConvergeTests.cs","Darling/Darling.Tests/RawPurgeConvergeLiveTests.cs","Darling/Darling.Tests/RawPurgeTriggerGateErrorTests.cs","Darling/Darling.Tests/RawPurgeTriggerLiveTests.cs","Darling/Darling.Tests/RawRepairEpochStampTests.cs","Darling/Darling.Tests/RawRepairEpochTriggerLiveTests.cs","Darling/Darling.Tests/RawRetentionPurgeJobLiveTests.cs","Darling/Darling.Tests/RawRetentionServiceTriggeredLiveTests.cs","Darling/Darling.Tests/RetentionHoldAndCadenceRawRedirectLiveTests.cs","Darling/Darling.Tests/RetentionHoldRatioKnobRungTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/RollupBackfillLiveTests.cs","Darling/Darling.Tests/TimescaleAvailabilityReprobeTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.MaterializationHoles.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.DarlingSelfAlertTests","Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["build","Darling PG tests (2)","Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"c59e13654b1548c9e6206fedfeb974eb602c4438","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs"],"failed_classes":["Darling.Tests.ManagedConfFileLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"a6d62e93a3eb04e679da71abc11e6b0949e1087e","event":"pull_request","changed_files":["Darling/Darling.Tests/RawTablesLeaveCatalogSweepLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs"],"failed_classes":["Darling.Tests.PayloadDimensionLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"1f9857239bd6da26bf10b03058a95a38154f9950","event":"pull_request","changed_files":["Darling/Darling.Tests/RawGatePerSlotSourceFloorLiveTests.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.QueryStoreCorrectedRollupLiveTests","Darling.Tests.RetentionReevaluationLiveTests","Darling.Tests.RollupBackfillLiveTests","Darling.Tests.TimescaleSupportTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"01d4c7dba1714a5b5ecb12ef3da6412c87f8aead","event":"pull_request","changed_files":["Darling/Darling.Tests/PayloadDimensionLiveTests.cs","Darling/Darling.Tests/RawPurgeNowWiringTests.cs","Darling/Darling.Tests/RawTablesLeaveCatalogSweepLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"3493474200ee277b82be8a9cefd1e4cd32b54374","event":"pull_request","changed_files":["Darling/Darling.Tests/PayloadDimensionLiveTests.cs","Darling/Darling.Tests/RawPurgeNowWiringTests.cs","Darling/Darling.Tests/RawTablesLeaveCatalogSweepLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"dae420ebe034fbcd2b8291248972ee03b08dd73e","event":"pull_request","changed_files":["Darling/Darling.Tests/PgStatementStatsRowCoherentResetTests.cs","PerformanceMonitor.Collectors/CollectorDeltaCalculator.cs","PerformanceMonitor.Collectors/ICollectorDeltaCalculator.cs","PerformanceMonitor.Collectors/PgStatementStatsCollector.cs"],"failed_classes":["Lite.Tests.PgStatementStatsDeltaSkipTests","Lite.Tests.PgStatementStatsFlavorTests","Lite.Tests.ServerEpochTests"],"failed_jobs":["Lite tests (1)","Lite tests (0)"]} +{"date":"2026-09-26","head_sha":"3b2d41baaeecc4319df0c8e9344423504bd5a2fa","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingStoreUpgradeTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfUpgradePathTests.cs","Darling/Darling.Tests/ManagedPreloadAssert.cs"],"failed_classes":["Darling.Tests.QueryStoreIntervalWideGridLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"3a0df25e8e52fbe9c0c23a8199c9e100ff1ff355","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeStatementTimeoutReloadTests.cs","Darling/Darling.Tests/ComposeStatementTimeoutStoreTests.cs","Darling/Darling.Tests/ComposeStatementTimeoutV147MigrationLiveTests.cs","Darling/Darling.Tests/ComposeStoreRolesLiveTests.cs","Darling/Darling.Tests/DarlingManagedRolesTests.cs","Darling/Darling.Tests/DocCommentHygieneTests.cs","Darling/Darling.Tests/McpReadCommandTimeoutTests.cs","Darling/Darling.Tests/PairedClientDeadlinesTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeSpec.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/McpCommandDeadlines.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageCommandDeadlines.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerCommandDeadlines.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/tools/provision-roles.sql"],"failed_classes":["Darling.Tests.ComposeStoreRolesLiveTests","Darling.Tests.DarlingStoreLoginsTests","Darling.Tests.ManagedConfVerdictsRungTests","Darling.Tests.MigrationDataMovingRungCensusPins","Darling.Tests.StorageCommandTimeoutTests","Darling.Tests.ViewerCommandTimeoutTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-09-26","head_sha":"6d64a5690e0fb888332d01effb3e5cdf373bc51b","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeStatementTimeoutReloadTests.cs","Darling/Darling.Tests/ComposeStatementTimeoutStoreTests.cs","Darling/Darling.Tests/ComposeStatementTimeoutV147MigrationLiveTests.cs","Darling/Darling.Tests/ComposeStoreRolesLiveTests.cs","Darling/Darling.Tests/DarlingManagedRolesTests.cs","Darling/Darling.Tests/DarlingStoreLoginsTests.cs","Darling/Darling.Tests/DocCommentHygieneTests.cs","Darling/Darling.Tests/ManagedConfVerdictsRungTests.cs","Darling/Darling.Tests/McpReadCommandTimeoutTests.cs","Darling/Darling.Tests/MigrationDataMovingRungCensusPins.cs","Darling/Darling.Tests/PairedClientDeadlinesTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeSpec.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/McpCommandDeadlines.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageCommandDeadlines.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerCommandDeadlines.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/tools/provision-roles.sql"],"failed_classes":["Darling.Tests.ComposeStoreRolesLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-26","head_sha":"57e73f1ed5b05395c6f94a66145c68307384d6a0","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeStatementTimeoutV147MigrationLiveTests.cs","Darling/Darling.Tests/FleetSweepWebFeedTests.cs","Darling/Darling.Tests/ReadLatencyAccumulatorTests.cs","Darling/Darling.Tests/ReadLatencyFlushLiveTests.cs","Darling/Darling.Tests/ReadLatencyPercentilesTests.cs","Darling/Darling.Tests/ReadLatencyWebRecordingTests.cs","Darling/Darling.Tests/ReadOutcomeClassifierTests.cs","Darling/Darling.Tests/ScheduledPurgeLaunchTests.cs","Darling/Darling.Tests/SharedBaselineCacheTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Service/Program.cs","Darling/PerformanceMonitor.Darling.Service/ReadLatencyAccumulator.cs","Darling/PerformanceMonitor.Darling.Service/ReadLatencyPercentiles.cs","Darling/PerformanceMonitor.Darling.Service/ReadOutcomeClassifier.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests","Darling.Tests.SharedBaselineCacheTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-26","head_sha":"a9e36ec7251c2b904e463fe38cd1a13cab458592","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/ComposeStatementTimeoutV147MigrationLiveTests.cs","Darling/Darling.Tests/FleetSweepWebFeedTests.cs","Darling/Darling.Tests/ReadLatencyAccumulatorTests.cs","Darling/Darling.Tests/ReadLatencyFlushLiveTests.cs","Darling/Darling.Tests/ReadLatencyPercentilesTests.cs","Darling/Darling.Tests/ReadLatencyWebRecordingTests.cs","Darling/Darling.Tests/ReadOutcomeClassifierTests.cs","Darling/Darling.Tests/ScheduledPurgeLaunchTests.cs","Darling/Darling.Tests/SharedBaselineCacheTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Service/Program.cs","Darling/PerformanceMonitor.Darling.Service/ReadLatencyAccumulator.cs","Darling/PerformanceMonitor.Darling.Service/ReadLatencyPercentiles.cs","Darling/PerformanceMonitor.Darling.Service/ReadOutcomeClassifier.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs"],"failed_classes":["Darling.Tests.LiveCleanupConversionRatchetTests"],"failed_jobs":["Darling PG tests (2)","build"]} +{"date":"2026-09-26","head_sha":"a4218e75adf8a7ab18fcaa51e7e0e8246a831bf5","event":"pull_request","changed_files":["Darling/Darling.Tests/GetReadLatencyLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreLogins.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpReadLatencyTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingReadLatencyReader.cs","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs"],"failed_classes":["Darling.Tests.AsOfWindowAnchorTests","Darling.Tests.McpToolTypeRegistrationTests","Lite.Tests.CrossAppMcpToolInventoryPinTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (2)","Lite tests (1)"]} +{"date":"2026-09-26","head_sha":"abb3f9b719ea0e6ebdf36f167c2bfc735b457aec","event":"pull_request","changed_files":["Darling/Darling.Tests/AsOfWindowAnchorTests.cs","Darling/Darling.Tests/GetReadLatencyLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreLogins.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpReadLatencyTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingReadLatencyReader.cs","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","README.md","llms.txt"],"failed_classes":["Darling.Tests.DarlingMcpInstructionsBudgetTests","Darling.Tests.McpToolsListBudgetTests","Lite.Tests.CrossAppMcpToolInventoryPinTests"],"failed_jobs":["build","Darling PG tests (1)","Lite tests (1)"]} +{"date":"2026-09-27","head_sha":"6082b10647e46b2d13b2ef2e3d34db0e0bdb5ce5","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpHealthParserToolsTests.cs","Darling/Darling.Tests/Fixtures/SystemHealth/scheduler_monitor.xml","Darling/Darling.Tests/Fixtures/SystemHealth/scheduler_monitor_high_sql_cpu.xml","Darling/Darling.Tests/Fixtures/SystemHealth/scheduler_monitor_low_memory.xml","Darling/Darling.Tests/Fixtures/SystemHealth/scheduler_monitor_normal.xml","Darling/Darling.Tests/Fixtures/SystemHealth/scheduler_monitor_other_process_cpu.xml","Darling/Darling.Tests/McpToolGuideHeads.HealthParser.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpHealthParserTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/SystemHealthParserTests.cs","Darling/Darling.Tests/ViewerSystemEventsTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHealthParserTools.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.SystemEvents.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml","Lite.Tests/Fixtures/SystemHealth/scheduler_monitor.xml","Lite.Tests/Fixtures/SystemHealth/scheduler_monitor_high_sql_cpu.xml","Lite.Tests/Fixtures/SystemHealth/scheduler_monitor_low_memory.xml","Lite.Tests/Fixtures/SystemHealth/scheduler_monitor_normal.xml","Lite.Tests/Fixtures/SystemHealth/scheduler_monitor_other_process_cpu.xml","Lite.Tests/McpToolGuideHeads.HealthParser.cs","Lite.Tests/McpToolsListBudget/McpHealthParserTools.txt","Lite.Tests/SystemEventsReaderTests.cs","Lite/Controls/ServerTab.xaml","Lite/Mcp/McpHealthParserTools.cs","Lite/Services/LocalDataService.SystemEvents.cs","PerformanceMonitor.Common/SystemHealthModels.cs","PerformanceMonitor.Common/SystemHealthParser.cs","PerformanceMonitor.Common/SystemHealthSignificance.cs"],"failed_classes":["Darling.Tests.TsqlConventionGuardTests","Lite.Tests.McpToolsListBudgetTests"],"failed_jobs":["Darling PG tests (1)","Lite tests (2)","build"]} +{"date":"2026-09-27","head_sha":"27585711ab90f5c6791ba82c5b0da8842cd9a850","event":"pull_request","changed_files":[".github/workflows/build.yml","Darling/Darling.Tests/SignPathActionVersionTests.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PG tests (1)","Darling whole-tree guards"]} +{"date":"2026-09-27","head_sha":"f685b99b7b96260597277556e13b2a6771824139","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpAgToolsTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpAgTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingAgReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAgTools.cs"],"failed_classes":["Darling.Tests.McpPayloadContractCensusTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-09-27","head_sha":"dbd970fb8d61ae8b750ef2028c954dec41c24dce","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpAgToolsTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpAgTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingAgReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAgTools.cs"],"failed_classes":["Darling.Tests.McpToolsListBudgetTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-27","head_sha":"da12c872e58696acff61f0741a8c54a5f0bcc384","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingWatermarkFloorPlanShapeLiveTests.cs","Darling/Darling.Tests/DarlingWatermarkFloorScanBoundLiveTests.cs","Darling/Darling.Tests/DarlingWatermarkSeedLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs"],"failed_classes":["Darling.Tests.DarlingWatermarkFloorPlanShapeLiveTests","Darling.Tests.DarlingWatermarkFloorScanBoundLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-27","head_sha":"b3f2fb8f8d002093f57ac58f6f89832e2db58f6e","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingAgReaderTests.cs","Darling/Darling.Tests/FactAdvicePluralTests.cs","Darling/Darling.Tests/FactAdvicePluralWiringTests.cs","Darling/Darling.Tests/JobHistoryCapLabelTests.cs","Darling/Darling.Tests/ViewerAvailabilityGroupsTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingAgReader.cs","Darling/PerformanceMonitor.Darling.Viewer/JobHistoryTab.xaml","Darling/PerformanceMonitor.Darling.Viewer/JobHistoryTab.xaml.cs","PerformanceMonitor.Analysis/FactAdvice.cs","PerformanceMonitor.Common/AgTopology.cs","PerformanceMonitor.Common/JobHistoryCap.cs"],"failed_classes":["Darling.Tests.AgTopologyCardsTests","Lite.Tests.AgTopologyTests"],"failed_jobs":["build","Lite tests (0)","Darling PG tests (2)"]} +{"date":"2026-09-27","head_sha":"5a0dfb2b9b12c9f7cea0832b00d2bd020f072f4c","event":"pull_request","changed_files":["Darling/PerformanceMonitor.Darling.Analysis/PgAnomalyDetector.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.ServerStatus.cs","PerformanceMonitor.Common/SingleFlightTtlCache.cs"],"failed_classes":["Darling.Tests.DarlingAnomalyBaselineTests","Darling.Tests.OverviewFleetHealthSingleFlightLiveTests","Darling.Tests.SingleFlightTtlCacheTests","Darling.Tests.StoreSizeCacheLiveTests","Darling.Tests.WaitRateTileReadCountLiveTests"],"failed_jobs":["build","Darling PG tests (2)","Darling PG tests (0)","Darling PG tests (1)"]} +{"date":"2026-09-27","head_sha":"b36f267897e8c58f97b3a865aa6416ca9223ba6a","event":"pull_request","changed_files":["Darling/Darling.Tests/StoreApplicationNameCensusTests.cs","Darling/Darling.Tests/StoreApplicationNamePinTests.cs","Darling/Darling.Tests/ViewerApplicationNameLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreUpgrade.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingStoreConnection.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerSettings.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.ViewerCommandTimeoutTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-09-27","head_sha":"84a49674a9e3b2c46e708a219468f68e6c6f5e5a","event":"pull_request","changed_files":["Darling/Darling.Tests/LiteJobHistoryLoadingCapTests.cs","Lite/Controls/JobHistoryTab.xaml","Lite/Controls/JobHistoryTab.xaml.cs"],"failed_classes":["Darling.Tests.ScopedLoadOrderingTests"],"failed_jobs":["Darling PG tests (2)","build"]} +{"date":"2026-09-27","head_sha":"323d11a4b7f3b5527536e665a9e784d132e15ae8","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingWatermarkFloorPlanShapeLiveTests.cs","Darling/Darling.Tests/DarlingWatermarkIndexEquivalenceLiveTests.cs","Darling/Darling.Tests/MigrationDataMovingRungCensusPins.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs"],"failed_classes":["Darling.Tests.ActivityDrivenPlanFetchStoreTests","Darling.Tests.CheckpointsTimedRungTests","Darling.Tests.CollectionCaveatsRungTests","Darling.Tests.CollectionLogDrainForensicsStoreTests","Darling.Tests.CollectionLogFanoutRollupStoreTests","Darling.Tests.CollectionLogFetchPhaseSumsLivePostgresTests","Darling.Tests.CollectionLogFetchPhaseSumsStoreTests","Darling.Tests.CollectionLogPhaseSplitStoreTests","Darling.Tests.CollectorDatabaseScopeRungTests","Darling.Tests.CollectorStallProbeStoreTests","Darling.Tests.ComposeStatementTimeoutStoreTests","Darling.Tests.ComposeStatementTimeoutV147MigrationLiveTests","Darling.Tests.CustomAlertCoreMigrationTests","Darling.Tests.DarlingAnalysisStoreTests","Darling.Tests.DarlingCustomViewsTests","Darling.Tests.DarlingObservabilityTests","Darling.Tests.DarlingPlanCorrectionLiveMigrationTests","Darling.Tests.DarlingServerTagsTests","Darling.Tests.DeadlockRateBandRungTests","Darling.Tests.DeltaFamilyIntervalColumnsRungTests","Darling.Tests.DeltaFamilyIntervalCompletionRungTests","Darling.Tests.FileGrowthAlertStoreTests","Darling.Tests.FleetSweepCadenceKnobRungTests","Darling.Tests.FleetSweepStateRungTests","Darling.Tests.IncidentOccurrenceAccumulatorTests","Darling.Tests.IndexObjectStatsServerTimeIndexRungTests","Darling.Tests.LivePostgresStoreFixtureTests","Darling.Tests.LongRunningQueryExclusionKnobRungTests","Darling.Tests.ManagedConfVerdictsRungTests","Darling.Tests.MonitoredEngineKindStoreTests","Darling.Tests.MuteRuleReloadBeaconTests","Darling.Tests.NotificationRoutesRungTests","Darling.Tests.OversizedPlanBacklogPins","Darling.Tests.PerfmonCounterTypeRungTests","Darling.Tests.Pg18IoBytesTests","Darling.Tests.PgAlertCountKnobRungTests","Darling.Tests.PgCpuCapacityHeadroomTests","Darling.Tests.PgDatabaseSizeStatsAndHostMemoryLivePostgresTests","Darling.Tests.PgDatabaseSizeStatsAndHostMemoryRungTests","Darling.Tests.PgIndexBloatEstimateRungTests","Darling.Tests.PgLogEventMetricsRungTests","Darling.Tests.PgLogEventsRungTests","Darling.Tests.PgNumbackendsAndSampledMsLivePostgresTests","Darling.Tests.PgNumbackendsAndSampledMsRungTests","Darling.Tests.PgServerConfigScopeRungTests","Darling.Tests.PgServerConfigTests","Darling.Tests.PgStatementTextTests","Darling.Tests.PlanContentRetentionTests","Darling.Tests.PlanForceActionStoreTests","Darling.Tests.PostgresMajorVersionRegistryTests","Darling.Tests.PostmasterStartTimeLivePostgresTests","Darling.Tests.PostmasterStartTimeRungTests","Darling.Tests.PvsStatsStoreTests","Darling.Tests.QsCaptureModeRouteKnobToastLivePostgresTests","Darling.Tests.QsCaptureModeRouteKnobToastRungTests","Darling.Tests.QueryStoreHealthStoreTests","Darling.Tests.QueryStoreIntervalLatestRungTests","Darling.Tests.QueryStoreIntervalWideRungTests","Darling.Tests.QueryStoreLivenessHotTouchLiveTests","Darling.Tests.QueryStoreTextStoreTests","Darling.Tests.RawChunkIntervalRungHistoryRungTests","Darling.Tests.ScaffoldTests","Darling.Tests.SelfDiskWarnGbFloorRungTests","Darling.Tests.StoreLogSelfMonitoringStoreTests","Darling.Tests.StoreSelfMetricsTests","Darling.Tests.TempDbCeilingStoreTests","Darling.Tests.TimeHonestyRungLivePostgresTests","Darling.Tests.TimeHonestyRungTests","PerformanceMonitor.Darling.Tests.StoreConnectionSelfTestLiveTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (2)","build","Darling PG tests (1)"]} +{"date":"2026-09-27","head_sha":"1b03b214f4a79e4add0cc36d57e37bd30e9a94ee","event":"pull_request","changed_files":["Darling/Darling.Tests/LitePerfmonWaitStatisticsArtifactSourceTests.cs","Lite.Tests/LitePerfmonWaitStatisticsArtifactLiveTests.cs","Lite/Controls/ServerTab.Pickers.cs","Lite/Mcp/McpPerfmonTools.cs","Lite/Services/LocalDataService.Perfmon.cs","Lite/Services/LocalDataService.TrendBuckets.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests","PerformanceMonitorLite.Tests.LitePerfmonWaitStatisticsArtifactLiveTests"],"failed_jobs":["build","Lite tests (2)","Darling PG tests (1)"]} +{"date":"2026-09-27","head_sha":"b33399f21934a1f58ac26398f21c0c37361d9b1a","event":"pull_request","changed_files":["Darling/Darling.Tests/JobHistoryPerServerTopNLiveTests.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.JobHistory.cs"],"failed_classes":["Darling.Tests.DarlingCloudIdentityProbeTests","Darling.Tests.JobHistoryPerServerTopNLiveTests"],"failed_jobs":["build","Darling PG tests (2)","Darling PG tests (0)","Darling PG tests (1)"]} +{"date":"2026-09-27","head_sha":"bd31a3a390165ad4e93113de453888108d538026","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionHealthAggregateTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs","Darling/PerformanceMonitor.Darling.Storage/CollectionHealthRollupSupport.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs"],"failed_classes":["Darling.Tests.FleetCollectionHealthMemoTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-27","head_sha":"8394dd35a0c3d860089ff18a2fff5d3cb35554f1","event":"pull_request","changed_files":["Darling/Darling.Tests/AgGroupIdLiveReaderTests.cs","Darling/Darling.Tests/AgGroupIdRungTests.cs","Darling/Darling.Tests/CollectionLogWatermarkAndJobHistoryIndexesRungTests.cs","Darling/Darling.Tests/DarlingAgReaderTests.cs","Darling/Darling.Tests/PgSchemaGeneratorTests.cs","Darling/Darling.Tests/QueryStoreLivenessHotTouchLiveTests.cs","Darling/Darling.Tests/ViewerAvailabilityGroupsTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingAgReader.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingAgStatesReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.AvailabilityGroups.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Lite.Tests/AgCollectorDefinitionTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/Services/LocalDataService.AgTopology.cs","PerformanceMonitor.Collectors/AgDatabaseReplicaStatesCollector.cs","PerformanceMonitor.Collectors/AgReplicaStatesCollector.cs","PerformanceMonitor.Common/AgTopology.cs"],"failed_classes":["Darling.Tests.AgGroupIdRungTests","Darling.Tests.QsCaptureModeRouteKnobToastRungTests","Lite.Tests.AgCollectorDefinitionTests","PerformanceMonitorLite.Tests.DuckDbSchemaEquivalenceTests","PerformanceMonitorLite.Tests.QsCaptureModeRungTests"],"failed_jobs":["Lite tests (0)","Lite tests (3)","build","Darling PG tests (1)"]} +{"date":"2026-09-27","head_sha":"40f6059a474d9c31dbac8c0c970fe5e11b904a25","event":"pull_request","changed_files":["Darling/Darling.Tests/JobHistoryPerServerTopNLiveTests.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.JobHistory.cs"],"failed_classes":["Darling.Tests.JobHistoryPerServerTopNLiveTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (1)","build"]} +{"date":"2026-09-27","head_sha":"48d385c5822ebee00e7b5403f49d526cfcf5b7cf","event":"pull_request","changed_files":[".github/workflows/build.yml"],"failed_classes":["Darling.Tests.DarlingStoreUpgradeTests","Darling.Tests.ManagedConfUpgradePathTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-27","head_sha":"b652ab3d6f5635b63b4d61a36f0a790a4d1b5339","event":"pull_request","changed_files":["Darling/Darling.Tests/JobHistoryPerServerTopNLiveTests.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.JobHistory.cs"],"failed_classes":["Darling.Tests.JobHistoryPerServerTopNLiveTests"],"failed_jobs":["Darling PG tests (1)","build","Darling PG tests (0)"]} +{"date":"2026-09-27","head_sha":"fe1aa556b94a2c7170a871fc51ef02a371a70550","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingCollectorRunnerTests.cs","Darling/Darling.Tests/JobHistoryIdentityEpochTests.cs","Darling/Darling.Tests/JobHistoryNaturalKeyDedupeTests.cs","Darling/Darling.Tests/JobHistoryWatermarkEpochLiveTests.cs","Darling/Darling.Tests/LiteJobHistorySourceTests.cs","Darling/Darling.Tests/ServerWatermarkCacheRunnerLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Lite.Tests/JobHistoryDedupeAndWatermarkLiteTests.cs","Lite/Services/RemoteCollectorService.DefinitionRunner.cs","Lite/Services/RemoteCollectorService.cs","PerformanceMonitor.Collectors/INaturalKeyDedupedCollector.cs","PerformanceMonitor.Collectors/JobHistoryCollector.cs"],"failed_classes":["Darling.Tests.JobHistoryWatermarkEpochLiveTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-27","head_sha":"d4ccb2ed0816778aa405ad4ccd4a0213fe2d6659","event":"push","changed_files":["Darling/Darling.Tests/EventWindowedReadsCarryTheFloorTests.cs","Darling/Darling.Tests/JobHistoryPerServerTopNLiveTests.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.JobHistory.cs"],"failed_classes":["Darling.Tests.ScramVerifierLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-27","head_sha":"d0075921241a45fbc768b808394b8987b85bd0bf","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Darling/Darling.Tests/StoreLogClassifierTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Storage/StoreLogClassifier.cs","PerformanceMonitor.Collectors/PgLogEntryAssembler.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.PgLogEventsPipelineTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (0)","build"]} +{"date":"2026-09-27","head_sha":"13bf1036fcb986f806756d3dfbb020f6fb3d3df4","event":"pull_request","changed_files":["PerformanceMonitor.Collectors/PgLogEntryAssembler.cs"],"failed_classes":["Darling.Tests.PgLogEventMetricsParserTests","Darling.Tests.PgLogEventsPipelineTests","Darling.Tests.RdsLogEventIngestorTimezoneTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (0)","Darling PG tests (2)","build","Darling whole-tree guards"]} +{"date":"2026-09-27","head_sha":"f43cd0097d3c6c1063fd68d06def15cdacb7a084","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Darling/Darling.Tests/PgPlanLogParserQueryIdPrefixTests.cs","Darling/Darling.Tests/StoreLogClassifierTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Storage/StoreLogClassifier.cs","PerformanceMonitor.Collectors/PgLogEntryAssembler.cs","PerformanceMonitor.Collectors/PgLogEventClassifier.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgPlanLogParser.cs","PerformanceMonitor.Collectors/PgServerLogTail.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","Darling.Tests.PgLogEventsPipelineTests","Lite.Tests.PgLoggingCollectorGateTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (0)","Lite tests (1)"]} +{"date":"2026-09-27","head_sha":"bab1c4114bac757411a4d5e21121d67c9e464261","event":"pull_request","changed_files":["Darling/Darling.Tests/AgGroupIdRungTests.cs","Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs","Darling/Darling.Tests/CaggGroupIndexDropRungTests.cs","Darling/Darling.Tests/CaggGroupIndexDropUpgradeLiveTests.cs","Darling/Darling.Tests/IntervalDedupMaterializationIndexesTests.cs","Darling/Darling.Tests/MigrationDataMovingRungCensusPins.cs","Darling/Darling.Tests/PgSchemaGeneratorTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs"],"failed_classes":["Darling.Tests.AbandonedRunEraInvariantReadTests","Darling.Tests.AgGroupIdLiveReaderTests","Darling.Tests.AgGroupIdRungTests","Darling.Tests.AlertNotebookAuthoredContextTests","Darling.Tests.AlertNotebookCollectorFreshnessLiveTests","Darling.Tests.AlertStoredValueTests","Darling.Tests.AnalysisAsOfAnchorLivePostgresTests","Darling.Tests.AnalysisCoverageLivePostgresTests","Darling.Tests.AnalysisPageCapLiveTests","Darling.Tests.AnalyzeWindowEmptyLivePostgresTests","Darling.Tests.AnomalyDetectorErrorGuardLiveTests","Darling.Tests.AnomalyNullWindowLiveTests","Darling.Tests.AnomalyObjectStatsLatestSnapshotsLiveTests","Darling.Tests.AsOfWindowAnchorLivePostgresTests","Darling.Tests.AvailabilityGroupCountReadTests","Darling.Tests.BaselineDiscontinuityLivePostgresTests","Darling.Tests.CaggGroupIndexDropLiveTests","Darling.Tests.CaggGroupIndexDropUpgradeLiveTests","Darling.Tests.CaptureDownChunkOrderTests","Darling.Tests.CiClusterWorkerSizingLiveTests","Darling.Tests.CollectionCaveatStoreLiveTests","Darling.Tests.CollectionHealthAggregateTests","Darling.Tests.CollectionHealthPayloadBudgetLiveTests","Darling.Tests.CollectionLogFetchPhaseSumsLivePostgresTests","Darling.Tests.CollectionLogWatermarkAndJobHistoryIndexesRungTests","Darling.Tests.CollectorCostMaterialityFloorTests","Darling.Tests.CollectorCostRegressionDispersionTests","Darling.Tests.CollectorCostRegressionMaterialityTests","Darling.Tests.CollectorCostRegressionPerRunTests","Darling.Tests.CollectorDatabaseScopeLivePostgresTests","Darling.Tests.CollectorStateContractTests","Darling.Tests.CollectorTableCatalogShadowTests","Darling.Tests.CompareAnalysisPgBandingLiveTests","Darling.Tests.ComposeAdHocModuleLivePostgresTests","Darling.Tests.ComposeQueryStoreLivePostgresTests","Darling.Tests.ComposeStatementTimeoutV147MigrationLiveTests","Darling.Tests.ComposeStoreRolesLiveTests","Darling.Tests.ComposeTopSeriesLivePostgresTests","Darling.Tests.CompressionEnableGuardLiveTests","Darling.Tests.ConfigChangeFamiliesLivePostgresTests","Darling.Tests.ConfigChangeTraceAnchorLivePostgresTests","Darling.Tests.CustomAlertEvaluateNowLiveTests","Darling.Tests.CustomAlertLastFiredLiveTests","Darling.Tests.CustomAlertResolveGrantLiveTests","Darling.Tests.CustomAlertRuleCapLiveTests","Darling.Tests.CustomAlertSeverityEscalationLiveTests","Darling.Tests.CustomAlertStateStoreLiveTests","Darling.Tests.CustomAlertTagSubtreeLiveTests","Darling.Tests.CustomAlertTeardownLiveTests","Darling.Tests.DailyStitchLiveTests","Darling.Tests.DailySummaryAndComposeStitchedLiveTests","Darling.Tests.DailySummaryNotCarriedLiveTests","Darling.Tests.DailySummaryReadShapeLiveTests","Darling.Tests.DarkPastRetentionReadsOfflineLivePostgresTests","Darling.Tests.DarlingAgStatesReaderLiveEqualityTests","Darling.Tests.DarlingAlertReadAdapterTests","Darling.Tests.DarlingAlertTuningKnobsTests","Darling.Tests.DarlingAlertingTests","Darling.Tests.DarlingAnalysisPipelineTests","Darling.Tests.DarlingAnalysisServiceEngineRoutingTests","Darling.Tests.DarlingAnalysisStoreTests","Darling.Tests.DarlingAnomalyBaselineTests","Darling.Tests.DarlingBlockingStatsReadTests","Darling.Tests.DarlingBlockingTrendEmptyTests","Darling.Tests.DarlingCappedReadLivePostgresTests","Darling.Tests.DarlingCliCommandsHostCheckTests","Darling.Tests.DarlingCollectionHealthLatestNoteTests","Darling.Tests.DarlingCollectionLogReadTests","Darling.Tests.DarlingCollectorRunnerTests","Darling.Tests.DarlingCollectorToggleVerbLivePostgresTests","Darling.Tests.DarlingCommandExecutorTests","Darling.Tests.DarlingComposeLivePostgresTests","Darling.Tests.DarlingCurrentWaitsTrendTests","Darling.Tests.DarlingCustomViewsLiveTests","Darling.Tests.DarlingDailySummaryRangeTests","Darling.Tests.DarlingDeliveryModeTests","Darling.Tests.DarlingDeltaSeederTests","Darling.Tests.DarlingEmptyEnumerationInventoryTests","Darling.Tests.DarlingEndpointToggleCliTests","Darling.Tests.DarlingFleetReaderLivePostgresTests","Darling.Tests.DarlingIndexLockingRenamedDatabaseLivePostgresTests","Darling.Tests.DarlingLiveStoreExtensionParityTests","Darling.Tests.DarlingLockWaitTrendTests","Darling.Tests.DarlingMcpActiveQueriesBudgetLiveTests","Darling.Tests.DarlingMcpAgToolsLivePostgresTests","Darling.Tests.DarlingMcpAlertToolsLivePostgresTests","Darling.Tests.DarlingMcpAnalysisFindingsBudgetLiveTests","Darling.Tests.DarlingMcpBlockingBudgetLiveTests","Darling.Tests.DarlingMcpBlockingToolsLivePostgresTests","Darling.Tests.DarlingMcpConfigHistoryToolsLivePostgresTests","Darling.Tests.DarlingMcpConfigToolsLivePostgresTests","Darling.Tests.DarlingMcpCustomAlertToolsLivePostgresTests","Darling.Tests.DarlingMcpCustomViewToolsLivePostgresTests","Darling.Tests.DarlingMcpDataToolsLivePostgresTests","Darling.Tests.DarlingMcpDeadlockDetailBudgetLiveTests","Darling.Tests.DarlingMcpDefaultTraceToolsLivePostgresTests","Darling.Tests.DarlingMcpHealthParserToolsLivePostgresTests","Darling.Tests.DarlingMcpHealthToolsLivePostgresTests","Darling.Tests.DarlingMcpJobToolsLivePostgresTests","Darling.Tests.DarlingMcpLatchSpinlockToolsLivePostgresTests","Darling.Tests.DarlingMcpMemoryGrantToolsLivePostgresTests","Darling.Tests.DarlingMcpObjectStatsToolsLivePostgresTests","Darling.Tests.DarlingMcpPgLoggingAuditLivePostgresTests","Darling.Tests.DarlingMcpPgPlanQueryIdLiveTests","Darling.Tests.DarlingMcpPlanCacheSchedulerToolsLivePostgresTests","Darling.Tests.DarlingMcpPlanToolsLivePostgresTests","Darling.Tests.DarlingMcpQueryStoreRegressionsBudgetLiveTests","Darling.Tests.DarlingMcpServerAdminToolsLivePostgresTests","Darling.Tests.DarlingMcpSessionToolsLivePostgresTests","Darling.Tests.DarlingMcpStoreHostBudgetLiveTests","Darling.Tests.DarlingMcpStoreHostToolsLiveTests","Darling.Tests.DarlingMcpToolsTests","Darling.Tests.DarlingMcpTrendToolsLivePostgresTests","Darling.Tests.DarlingMemoryTrendGrantJoinTests","Darling.Tests.DarlingModuleMapTests","Darling.Tests.DarlingObservabilityTests","Darling.Tests.DarlingPerformanceTrendsReadTests","Darling.Tests.DarlingPerformanceTrendsTierRoutingLiveTests","Darling.Tests.DarlingPgAutovacuumVerdictLiveTests","Darling.Tests.DarlingPgDatabaseStatsLiveTests","Darling.Tests.DarlingPgIndexBloatCoverageLivePostgresTests","Darling.Tests.DarlingPgIndexBloatNanLivePostgresTests","Darling.Tests.DarlingPgIndexBloatRotationReadLivePostgresTests","Darling.Tests.DarlingPgReadSqlParsesLiveTests","Darling.Tests.DarlingPgSlotVerdictLiveTests","Darling.Tests.DarlingPgTopQueriesLiveTests","Darling.Tests.DarlingPlanCorrectionLiveMigrationTests","Darling.Tests.DarlingQueryHeatmapBudgetLiveTests","Darling.Tests.DarlingQueryHeatmapLiveTests","Darling.Tests.DarlingQueryStoreRegressionsLiveTests","Darling.Tests.DarlingRemainingEmptyReadsTests","Darling.Tests.DarlingRetentionTests","Darling.Tests.DarlingSecuritySplitLiveTests","Darling.Tests.DarlingSelfAlertTests","Darling.Tests.DarlingSignificantWaitsReadTests","Darling.Tests.DarlingStoreHostProfileLiveTests","Darling.Tests.DarlingStoreMetricsJobLoggingLivePostgresTests","Darling.Tests.DarlingStoreUpgradeTests","Darling.Tests.DarlingTrendEmptyTests","Darling.Tests.DarlingWatermarkFloorPlanShapeLiveTests","Darling.Tests.DarlingWatermarkFloorScanBoundLiveTests","Darling.Tests.DarlingWatermarkIndexEquivalenceLiveTests","Darling.Tests.DarlingWatermarkSeedLiveTests","Darling.Tests.DatabaseFileGrowthReadTests","Darling.Tests.DatabaseSizeLatestPlanShapeLiveTests","Darling.Tests.DatabaseStateAlertMemoryLiveTests","Darling.Tests.DefaultTraceEventFrameLivePostgresTests","Darling.Tests.DeltaFamilyIntervalCompletionLivePostgresTests","Darling.Tests.DrillDownDopProvenanceLiveTests","Darling.Tests.EngineAwareVersionLabelTests","Darling.Tests.EngineCapabilityMissLivePostgresTests","Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests","Darling.Tests.ExcludedDatabasesStoreLiveTests","Darling.Tests.FinOpsObjectGrowthBoundsMergeLiveTests","Darling.Tests.FinOpsTopConsumersMergeLiveTests","Darling.Tests.FleetCardPostgresCpuLivePostgresTests","Darling.Tests.FleetCollectionHealthByServerLiveEqualityTests","Darling.Tests.FleetCollectionHealthMemoLivePostgresTests","Darling.Tests.FleetReadsAreBoundedLivePostgresTests","Darling.Tests.FleetSweepStoreLivePostgresTests","Darling.Tests.ForcePlanFailuresAccessPathTests","Darling.Tests.FreshStoreWatermarkLiveTests","Darling.Tests.FrozenRollupLiveTests","Darling.Tests.GetReadLatencyLiveTests","Darling.Tests.HoldEarnedByDeliveryTests","Darling.Tests.IndexUsageBudgetLiveTests","Darling.Tests.IndexUsageTruncationLivePostgresTests","Darling.Tests.IntervalDedupMaterializationIndexesTests","Darling.Tests.IntervalHonestHourlyRollupLiveTests","Darling.Tests.JobHistoryGroupedStatsMatchWindowFunctionLiveTests","Darling.Tests.JobHistoryPerServerTopNLiveTests","Darling.Tests.JobHistoryWatermarkEpochLiveTests","Darling.Tests.LatestCpuReadShapeLivePostgresTests","Darling.Tests.LatestValueLookbackLivePostgresTests","Darling.Tests.LiveCleanupBatchTests","Darling.Tests.LivePostgresStoreFixtureTests","Darling.Tests.LocalClockBucketKeyLiveTests","Darling.Tests.ManagedConfVerdictsLiveTests","Darling.Tests.MaterializationHoleRepairLiveTests","Darling.Tests.MaterializationHoleScanShapeLiveTests","Darling.Tests.McpCollectionLogPerServerResponseBudgetLivePostgresTests","Darling.Tests.McpComposedQueryDeadlineLivePostgresTests","Darling.Tests.McpFilterSemanticsLivePostgresTests","Darling.Tests.McpFleetResponseBudgetLivePostgresTests","Darling.Tests.McpLatestSnapshotStampLivePostgresTests","Darling.Tests.McpPageContractLivePostgresTests","Darling.Tests.McpReadToolBudgetLiveTests","Darling.Tests.McpZeroIsAMeasurementLivePostgresTests","Darling.Tests.MemoryGrantTempDbBucketingLiveTests","Darling.Tests.MigrationLockWaitContentionTests","Darling.Tests.MigrationUpgradeLadderLiveTests","Darling.Tests.MonitoredEngineKindStoreTests","Darling.Tests.MuteRuleReloadBeaconLivePostgresTests","Darling.Tests.NotificationRoutesLivePostgresTests","Darling.Tests.ObjectLockingBudgetLiveTests","Darling.Tests.OversizedPlanBacklogReadLivePostgresTests","Darling.Tests.OversizedPlanSweepRunRecordLivePostgresTests","Darling.Tests.OverviewFleetHealthSingleFlightLiveTests","Darling.Tests.ParameterSensitiveDrillDownTextLiveTests","Darling.Tests.ParameterSensitivityClockFrameLiveTests","Darling.Tests.PayloadDimensionLiveTests","Darling.Tests.PerServerCollectionHealthMemoLivePostgresTests","Darling.Tests.PerfmonCounterTypeLivePostgresTests","Darling.Tests.PerfmonWaitStatisticsArtifactMcpLiveTests","Darling.Tests.PerfmonWaitStatisticsArtifactViewerLiveTests","Darling.Tests.PgAutovacuumDisabledSpellingLiveTests","Darling.Tests.PgBaselineProviderKeyedLiveTests","Darling.Tests.PgBlockingChainsDedupedByRootLivePostgresTests","Darling.Tests.PgCopyWriterTests","Darling.Tests.PgDatabaseSizeStatsAndHostMemoryLivePostgresTests","Darling.Tests.PgDeadlockCandidatePatternLiveTests","Darling.Tests.PgDeadlockNormalizationTests","Darling.Tests.PgDeadlockRemaskTests","Darling.Tests.PgDeadlocksCsvlogLiveTests","Darling.Tests.PgFactCollectorTests","Darling.Tests.PgIoTrendDefaultBudgetLiveTests","Darling.Tests.PgLogEventMetricsLivePostgresTests","Darling.Tests.PgLogEventsCsvlogLiveTests","Darling.Tests.PgLogEventsJsonlogLiveTests","Darling.Tests.PgLogEventsLivePostgresTests","Darling.Tests.PgLongRunningQueryExclusionLiveTests","Darling.Tests.PgNumbackendsAndSampledMsLivePostgresTests","Darling.Tests.PgPlanCaptureGuardedCastLiveTests","Darling.Tests.PgRaiseShapedRecordsLiveTests","Darling.Tests.PgReadBinaryFileCapabilityLiveTests","Darling.Tests.PgServerConfigOverrideLivePostgresTests","Darling.Tests.PgServerConfigPendingRestartLiveTests","Darling.Tests.PgServerConfigToolBoundLiveTests","Darling.Tests.PgSettingRedactionLivePostgresTests","Darling.Tests.PgSettingScrubLiveTests","Darling.Tests.PgStatementKeyedSetLiveTests","Darling.Tests.PgStatementTextScrubLiveTests","Darling.Tests.PgStatementTextUpsertLivePostgresTests","Darling.Tests.PgTargetAnomalyTests","Darling.Tests.PgTargetAuditConfigTests","Darling.Tests.PgTargetBloatLiveTests","Darling.Tests.PgTargetBlockingTests","Darling.Tests.PgTargetBufferDrillDownTests","Darling.Tests.PgTargetClockLiveTests","Darling.Tests.PgTargetCollectionCaveatsLiveTests","Darling.Tests.PgTargetConfigSnapshotBoundLiveTests","Darling.Tests.PgTargetDeadlockDrillDownTests","Darling.Tests.PgTargetFactCollectorTests","Darling.Tests.PgTargetGrowthLiveTests","Darling.Tests.PgTargetIoTests","Darling.Tests.PgTargetKernelTests","Darling.Tests.PgTargetKnobsTests","Darling.Tests.PgTargetMemoryTests","Darling.Tests.PgTargetPlanTests","Darling.Tests.PgTargetPostureTests","Darling.Tests.PgTargetQueriesTests","Darling.Tests.PgTargetReplicationTests","Darling.Tests.PgTargetSampledWaitLiveTests","Darling.Tests.PgTargetSeqScanTests","Darling.Tests.PgTargetSessionsTests","Darling.Tests.PgTargetTempTests","Darling.Tests.PgTargetTileBehaviourIoReplicationBlockingTests","Darling.Tests.PgTargetTileBehaviourTests","Darling.Tests.PgTargetTileBehaviourWalKernelSampledTests","Darling.Tests.PgTargetVacuumLiveTests","Darling.Tests.PgTargetWaitLiveTests","Darling.Tests.PgTargetWriteTests","Darling.Tests.PgWaitSamplerLiveTests","Darling.Tests.PlanCorrectionBudgetLiveTests","Darling.Tests.PlanCorrectionFrameLiveTests","Darling.Tests.PlanForceActionDetailScrubLiveTests","Darling.Tests.PlanForceActionStoreTests","Darling.Tests.PlanRegressionDrillDownReuseLiveTests","Darling.Tests.PlanRegressionIntervalTableEquivalenceTests","Darling.Tests.PostgresDatabaseListScreenLivePostgresTests","Darling.Tests.PostgresEngineGateBehaviorTests","Darling.Tests.PostmasterStartTimeLivePostgresTests","Darling.Tests.PvsStatsStoreTests","Darling.Tests.QsCaptureModeRouteKnobToastLivePostgresTests","Darling.Tests.QueryHighDopStaleMaxDopLiveTests","Darling.Tests.QueryStatsUnknownCpuLiveTests","Darling.Tests.QueryStoreBackfillCandidateLiveTests","Darling.Tests.QueryStoreClutterLivePostgresTests","Darling.Tests.QueryStoreCorrectedRollupLiveTests","Darling.Tests.QueryStoreDurationTrendTableLiveTests","Darling.Tests.QueryStoreFetchProbeLivePostgresTests","Darling.Tests.QueryStoreIntervalLatestWriterTests","Darling.Tests.QueryStoreIntervalWideGridLiveTests","Darling.Tests.QueryStoreIntervalWideWriterTests","Darling.Tests.QueryStoreLivenessHotTouchLiveTests","Darling.Tests.QueryStoreLivenessTablePruneLiveTests","Darling.Tests.QueryStoreReplicaSplitAnalysisLiveTests","Darling.Tests.QueryStoreSliceRepairLiveTests","Darling.Tests.QueryStoreStatePruneLivePostgresTests","Darling.Tests.QueryStoreTopBudgetLiveTests","Darling.Tests.QueryStoreTopLiteralEndStraddleLiveTests","Darling.Tests.QueryStoreTopMcpLiveTests","Darling.Tests.QueryStoreTrendRoutingLiveTests","Darling.Tests.RawChunkIntervalReconcilerLiveTests","Darling.Tests.RawGatePerSlotSourceFloorLiveTests","Darling.Tests.RawPurgeConvergeLiveTests","Darling.Tests.RawPurgeTriggerGateErrorTests","Darling.Tests.RawPurgeTriggerLiveTests","Darling.Tests.RawRepairEpochTriggerLiveTests","Darling.Tests.RawRetentionPurgeJobLiveTests","Darling.Tests.RawRetentionServiceTriggeredLiveTests","Darling.Tests.RawTablesLeaveCatalogSweepLiveTests","Darling.Tests.RawWindowFloorMcpLiveTests","Darling.Tests.RdsLogTimezoneReadLiveTests","Darling.Tests.ReadLatencyFlushLiveTests","Darling.Tests.ReadLatencyWebRecordingTests","Darling.Tests.RecurrenceLabelStoreReadTests","Darling.Tests.RetentionChunkGeometryLiveTests","Darling.Tests.RetentionHoldAndCadenceRawRedirectLiveTests","Darling.Tests.RetentionReevaluationLiveTests","Darling.Tests.RollupBackfillLiveTests","Darling.Tests.RuntimePreconditionMissLivePostgresTests","Darling.Tests.ScramVerifierLiveTests","Darling.Tests.SelfAlertDeliveryStampStoreLivePostgresTests","Darling.Tests.ServerIdentityFromStoreTests","Darling.Tests.ServerInventoryFleetMetricsLiveTests","Darling.Tests.ServerListAndSummaryPlanShapeTests","Darling.Tests.ServerWatermarkCacheRunnerLiveTests","Darling.Tests.SharedBaselineCacheLiveTests","Darling.Tests.SqlServerStoreTileBehaviourBatchQueryMemoryTests","Darling.Tests.SqlServerStoreTileBehaviourTests","Darling.Tests.StoreConfigProviderTests","Darling.Tests.StoreCopyPhaseLivePostgresTests","Darling.Tests.StoreLogGroupingLiveTests","Darling.Tests.StoreLogRemaskLiveTests","Darling.Tests.StoreMetricsLatestSkipScanLiveTests","Darling.Tests.StoreMetricsSummaryFirstLivePostgresTests","Darling.Tests.StoreSelfMetricsTests","Darling.Tests.StoreSelfMetricsTickRecoveryLiveTests","Darling.Tests.StoreSessionTimeZoneLiveTests","Darling.Tests.StoreSizeCacheLiveTests","Darling.Tests.StoreToastAndCheckpointerLivePostgresTests","Darling.Tests.SuccessorDailyLiveTests","Darling.Tests.TempDbCeilingStoreTests","Darling.Tests.TimeHonestyRungLivePostgresTests","Darling.Tests.TimescaleAggregateCompressionTests","Darling.Tests.TimescaleSupportTests","Darling.Tests.TopCpuQueriesTextLiveTests","Darling.Tests.TopProceduresHourlyRoutingLiveTests","Darling.Tests.TopQueriesHourlyRoutingLiveTests","Darling.Tests.TrendBucketingLiveTests","Darling.Tests.TrendFamilyBucketingLiveTests","Darling.Tests.TrendPayloadBudgetLiveTests","Darling.Tests.TuningDropLockRecoveryLiveTests","Darling.Tests.TuningReaderBudgetLiveTests","Darling.Tests.UncorroboratedRouteStoreKnobTests","Darling.Tests.ViewerApplicationNameLiveTests","Darling.Tests.ViewerBlockingDepthLivePostgresTests","Darling.Tests.ViewerBlockingStatsLivePostgresTests","Darling.Tests.ViewerBlockingTrendBucketingLivePostgresTests","Darling.Tests.ViewerCalendarRetentionLivePostgresTests","Darling.Tests.ViewerCollectionCaveatsLivePostgresTests","Darling.Tests.ViewerConfigChangesLivePostgresTests","Darling.Tests.ViewerConfigurationLivePostgresTests","Darling.Tests.ViewerCpuSchedulerLivePostgresTests","Darling.Tests.ViewerCpuSessionsPlanCacheTrendBucketingLiveTests","Darling.Tests.ViewerCpuTempDbLivePostgresTests","Darling.Tests.ViewerDailyHealthLivePostgresTests","Darling.Tests.ViewerDeadlockSeverityLivePostgresTests","Darling.Tests.ViewerFileIoBlockingLivePostgresTests","Darling.Tests.ViewerFinOpsIntervalHonestLiveTests","Darling.Tests.ViewerFleetRollupLivePostgresTests","Darling.Tests.ViewerLatchSpinlockLivePostgresTests","Darling.Tests.ViewerMemoryLivePostgresTests","Darling.Tests.ViewerOverviewLanesLivePostgresTests","Darling.Tests.ViewerPerfmonRunningJobsLivePostgresTests","Darling.Tests.ViewerPlanCacheLivePostgresTests","Darling.Tests.ViewerPlanHostLivePostgresTests","Darling.Tests.ViewerQueriesLivePostgresTests","Darling.Tests.ViewerQueriesRestLivePostgresTests","Darling.Tests.ViewerQueryTrendBucketingLiveTests","Darling.Tests.ViewerRecommendationsLivePostgresTests","Darling.Tests.ViewerSchemaVersionGateTests","Darling.Tests.ViewerSessionStatsLivePostgresTests","Darling.Tests.ViewerTrendBucketingLiveTests","Darling.Tests.ViewerTrendRoutingLivePostgresTests","Darling.Tests.ViewerW2aLivePostgresTests","Darling.Tests.ViewerWaitStatsLivePostgresTests","Darling.Tests.ViewerWave3LivePostgresTests","Darling.Tests.WaitRateTileReadCountLiveTests","Darling.Tests.WaitStatisticsArtifactSqlEquivalenceLiveTests","Darling.Tests.WebExceptionTextCensusTests","Darling.Tests.XminHorizonPersistenceReadTests","PerformanceMonitor.Darling.Tests.CpuRankingLiveTests","PerformanceMonitor.Darling.Tests.DistinctTextsLiveTests","PerformanceMonitor.Darling.Tests.EmptySupersededBaselineLiveTests","PerformanceMonitor.Darling.Tests.HostObjectRollupLiveTests","PerformanceMonitor.Darling.Tests.RetiredBaselineAggregateLiveTests","PerformanceMonitor.Darling.Tests.StoreConnectionSelfTestLiveTests"],"failed_jobs":["Darling PG tests (0)","build","Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-09-27","head_sha":"962b3f3eef41eb2e727884e0091a859b40f63603","event":"pull_request","changed_files":[".github/workflows/build.yml"],"failed_classes":["Darling.Tests.PgLogEventMetricsParserTests","Darling.Tests.PgLogEventsPipelineTests","Darling.Tests.RdsLogEventIngestorTimezoneTests"],"failed_jobs":["build","Darling PG tests (0)","Darling PG tests (1)","Darling PG tests (2)","Darling whole-tree guards"]} +{"date":"2026-09-27","head_sha":"c7f73a53296bed09213e5fef15312109b0a36a7e","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Darling/Darling.Tests/PgPlanLogParserQueryIdPrefixTests.cs","Darling/Darling.Tests/StoreLogClassifierTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Storage/StoreLogClassifier.cs","PerformanceMonitor.Collectors/PgLogEntryAssembler.cs","PerformanceMonitor.Collectors/PgLogEventClassifier.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgPlanLogParser.cs","PerformanceMonitor.Collectors/PgServerLogTail.cs"],"failed_classes":["Darling.Tests.PgLogEventsPipelineTests","Lite.Tests.PgLoggingCollectorGateTests"],"failed_jobs":["build","Darling PG tests (0)","Lite tests (1)"]} +{"date":"2026-09-27","head_sha":"a178a7e48a2486fb3cd9b894faa170acdc976199","event":"pull_request","changed_files":[".github/workflows/build.yml"],"failed_classes":["Darling.Tests.PgLogEventMetricsLivePostgresTests","Darling.Tests.PgLogEventMetricsParserTests","Darling.Tests.PgLogEventsLivePostgresTests","Darling.Tests.PgLogEventsPipelineTests","Darling.Tests.RdsLogEventIngestorTimezoneTests"],"failed_jobs":["build","Darling PG tests (0)","Darling PG tests (2)","Darling PG tests (1)","Darling whole-tree guards"]} +{"date":"2026-09-27","head_sha":"119d02272e279d9590b4233fa896434bb8638340","event":"pull_request","changed_files":["Darling/Darling.Tests/AgGroupIdRungTests.cs","Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs","Darling/Darling.Tests/CaggGroupIndexDropRungTests.cs","Darling/Darling.Tests/CaggGroupIndexDropUpgradeLiveTests.cs","Darling/Darling.Tests/IntervalDedupMaterializationIndexesTests.cs","Darling/Darling.Tests/MigrationDataMovingRungCensusPins.cs","Darling/Darling.Tests/PgSchemaGeneratorTests.cs","Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs"],"failed_classes":["Darling.Tests.FleetSweepStoreLivePostgresTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-27","head_sha":"b537c8dc31cb716ccc27defbdde15126012161af","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Darling/Darling.Tests/PgPlanLogParserQueryIdPrefixTests.cs","Darling/Darling.Tests/StoreLogClassifierTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Storage/StoreLogClassifier.cs","PerformanceMonitor.Collectors/PgLogEntryAssembler.cs","PerformanceMonitor.Collectors/PgLogEventClassifier.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgPlanLogParser.cs","PerformanceMonitor.Collectors/PgServerLogTail.cs"],"failed_classes":["Darling.Tests.PgLogEventsPipelineTests","Lite.Tests.PgLoggingCollectorGateTests"],"failed_jobs":["Lite tests (1)","Darling PG tests (0)","build"]} +{"date":"2026-09-27","head_sha":"8a14e42d5a0539e5b4a3d09238f90e1d34edf82c","event":"pull_request","changed_files":["PerformanceMonitor.Collectors/PgLogEntryAssembler.cs"],"failed_classes":["Darling.Tests.PgDeadlocksCsvlogLiveTests","Darling.Tests.PgLogEventsLivePostgresTests","Darling.Tests.PgLogEventsPipelineTests"],"failed_jobs":["Darling PG tests (0)","build","Darling PG tests (1)","Darling whole-tree guards"]} +{"date":"2026-09-28","head_sha":"cbc658652efdc33dc69e7ee223005f87159168fd","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Darling/Darling.Tests/PgPlanLogParserQueryIdPrefixTests.cs","Darling/Darling.Tests/StoreLogClassifierTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Storage/StoreLogClassifier.cs","PerformanceMonitor.Collectors/PgLogEntryAssembler.cs","PerformanceMonitor.Collectors/PgLogEventClassifier.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgPlanLogParser.cs","PerformanceMonitor.Collectors/PgServerLogTail.cs"],"failed_classes":["Lite.Tests.PgLoggingCollectorGateTests"],"failed_jobs":["Lite tests (1)"]} +{"date":"2026-09-28","head_sha":"14a060ababcbf5828a3240e374ae1d9ab9db6c34","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Darling/Darling.Tests/PgPlanLogParserQueryIdPrefixTests.cs","Darling/Darling.Tests/StoreLogClassifierTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Storage/StoreLogClassifier.cs","PerformanceMonitor.Collectors/PgLogEntryAssembler.cs","PerformanceMonitor.Collectors/PgLogEventClassifier.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgPlanLogParser.cs","PerformanceMonitor.Collectors/PgServerLogTail.cs"],"failed_classes":["Lite.Tests.PgLoggingCollectorGateTests"],"failed_jobs":["Lite tests (1)"]} +{"date":"2026-09-28","head_sha":"d8242253c2fd1e1ad08f8ae8f95215ebb1e33460","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Darling/Darling.Tests/PgPlanLogParserQueryIdPrefixTests.cs","Darling/Darling.Tests/RdsLogEventIngestorTimezoneTests.cs","Darling/Darling.Tests/StoreLogClassifierTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogEventIngestor.cs","Darling/PerformanceMonitor.Darling.Storage/StoreLogClassifier.cs","Lite.Tests/PgLoggingCollectorGateTests.cs","PerformanceMonitor.Collectors/PgLogEntryAssembler.cs","PerformanceMonitor.Collectors/PgLogEventClassifier.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgPlanLogParser.cs","PerformanceMonitor.Collectors/PgServerLogTail.cs"],"failed_classes":["Darling.Tests.PgLogEventsPipelineTests","Darling.Tests.RdsLogEventIngestorTimezoneTests","Darling.Tests.StoreLogClassifierTests","Lite.Tests.PgLoggingCollectorGateTests"],"failed_jobs":["Darling PG tests (2)","Lite tests (1)","build","Darling PG tests (0)"]} +{"date":"2026-09-28","head_sha":"f4a1453b88a1092d354edf9836dddcf9b2bc0a11","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Darling/Darling.Tests/PgPlanLogParserQueryIdPrefixTests.cs","Darling/Darling.Tests/RdsLogEventIngestorTimezoneTests.cs","Darling/Darling.Tests/StoreLogClassifierTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogEventIngestor.cs","Darling/PerformanceMonitor.Darling.Storage/StoreLogClassifier.cs","Lite.Tests/PgLoggingCollectorGateTests.cs","PerformanceMonitor.Collectors/PgLogEntryAssembler.cs","PerformanceMonitor.Collectors/PgLogEventClassifier.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgPlanLogParser.cs","PerformanceMonitor.Collectors/PgServerLogTail.cs"],"failed_classes":["Lite.Tests.PgLoggingCollectorGateTests"],"failed_jobs":["Lite tests (1)"]} +{"date":"2026-09-28","head_sha":"3f6fa5c0c2f0c242b8a87ee1f14fc375ffe3f13d","event":"pull_request","changed_files":["Darling/Darling.Tests/CompressAfterStaggerLiveTests.cs","Darling/Darling.Tests/CompressAfterStaggerTests.cs","Darling/Darling.Tests/TimescaleSupportTests.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-28","head_sha":"c1fcf792d4fd328921dd8fec03a9102a6067d6eb","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfMigrationTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Darling/Darling.Tests/PgPlanLogParserQueryIdPrefixTests.cs","Darling/Darling.Tests/RdsLogEventIngestorTimezoneTests.cs","Darling/Darling.Tests/StoreLogClassifierTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfMigration.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogEventIngestor.cs","Darling/PerformanceMonitor.Darling.Storage/StoreLogClassifier.cs","Lite.Tests/PgLoggingCollectorGateTests.cs","PerformanceMonitor.Collectors/PgLogEntryAssembler.cs","PerformanceMonitor.Collectors/PgLogEventClassifier.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgPlanLogParser.cs","PerformanceMonitor.Collectors/PgServerLogTail.cs"],"failed_classes":["Darling.Tests.DarlingManagedPostgresTests","Lite.Tests.PgLoggingCollectorGateTests"],"failed_jobs":["Darling PG tests (2)","Lite tests (1)"]} +{"date":"2026-09-28","head_sha":"fac9f8c0cdf78c6e9d3f1eca0306d19de76039db","event":"pull_request","changed_files":["Darling/Darling.Tests/BenefitScorerWiringTests.cs","Darling/Darling.Tests/PlanViewerBenefitDisplayTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.Plans.cs","Lite/Analysis/DrillDownCollector.Plans.cs","PerformanceMonitor.PlanAnalysis/McpPlanAnalysisFormatter.cs","PerformanceMonitor.PlanAnalysis/PlanAdvisoryAggregator.cs","PerformanceMonitor.PlanAnalysis/PlanAnalysisPipeline.cs","PerformanceMonitor.PlanAnalysis/PlanWarningDisplay.cs","PerformanceMonitor.Ui/PlanViewerControl.Properties.cs","PerformanceMonitor.Ui/PlanViewerControl.xaml.cs"],"failed_classes":["Darling.Tests.PgWaitSamplerLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-28","head_sha":"89921c2ea8e2404347be4d2320332e344fee19f4","event":"pull_request","changed_files":["Darling/Darling.Tests/RollupFloorCacheLiveTests.cs","Darling/Darling.Tests/RollupFloorCacheTests.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.IntervalHonestHourlyRollupLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-28","head_sha":"8f33a24e5984da4b432858d9dfd68f4dacd8337c","event":"pull_request","changed_files":["Darling/Darling.Tests/PlanSync4512PostParseTests.cs","Darling/Darling.Tests/PlanSync4512Tests.cs","Darling/Darling.Tests/PlanSync4514Tests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.Plans.cs","Lite/Analysis/DrillDownCollector.Plans.cs","PerformanceMonitor.PlanAnalysis/BenefitScorer.cs","PerformanceMonitor.PlanAnalysis/McpPlanAnalysisFormatter.cs","PerformanceMonitor.PlanAnalysis/PlanAnalyzer.cs","PerformanceMonitor.PlanAnalysis/PlanLayoutEngine.cs","PerformanceMonitor.PlanAnalysis/PlanModels.cs","PerformanceMonitor.PlanAnalysis/PlanStatements.cs","PerformanceMonitor.PlanAnalysis/ShowPlanParser.cs","PerformanceMonitor.Ui/PlanViewerControl.xaml.cs"],"failed_classes":["PerformanceMonitorLite.Tests.FinOpsTests"],"failed_jobs":["Lite tests (2)"]} +{"date":"2026-09-28","head_sha":"98646b56d32a16cc6ea37574ad6a3de7b051a704","event":"pull_request","changed_files":["Darling/Darling.Tests/BenefitScorerWiringTests.cs","Darling/Darling.Tests/PgWaitSamplerLiveTests.cs","Darling/Darling.Tests/PlanViewerBenefitDisplayTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.Plans.cs","Lite/Analysis/DrillDownCollector.Plans.cs","PerformanceMonitor.PlanAnalysis/McpPlanAnalysisFormatter.cs","PerformanceMonitor.PlanAnalysis/PlanAdvisoryAggregator.cs","PerformanceMonitor.PlanAnalysis/PlanAnalysisPipeline.cs","PerformanceMonitor.PlanAnalysis/PlanWarningDisplay.cs","PerformanceMonitor.Ui/PlanViewerControl.Properties.cs","PerformanceMonitor.Ui/PlanViewerControl.xaml.cs"],"failed_classes":["PerformanceMonitorLite.Tests.FinOpsTests"],"failed_jobs":["Lite tests (2)"]} +{"date":"2026-09-28","head_sha":"411d87afbe399342ded657d1651d44f30caab4a6","event":"pull_request","changed_files":["Darling/Darling.Tests/RollupFloorCacheLiveTests.cs","Darling/Darling.Tests/RollupFloorCacheTests.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["PerformanceMonitorLite.Tests.FinOpsTests"],"failed_jobs":["Lite tests (2)"]} +{"date":"2026-09-28","head_sha":"3e1ccb6ac7a365653f9da7e9766548729c299f28","event":"pull_request","changed_files":["Lite.Tests/FinOpsTests.cs"],"failed_classes":["PerformanceMonitorLite.Tests.FinOpsTests"],"failed_jobs":["Lite tests (2)"]} +{"date":"2026-09-28","head_sha":"481acb33ea1c7aa9fdcfd32868179275442eb1f5","event":"pull_request","changed_files":["Lite.Tests/FinOpsTests.cs","Lite.Tests/TestDataSeeder.cs"],"failed_classes":["PerformanceMonitorLite.Tests.FinOpsTests"],"failed_jobs":["Lite tests (2)"]} +{"date":"2026-09-28","head_sha":"7b24bc13ce1c0883018fc5b36f0eeacba98ea3c6","event":"pull_request","changed_files":["Darling/Darling.Tests/LiteAnalysisServiceCancellationSourceTests.cs","Lite.Tests/LiteAnalysisCancellationTests.cs","Lite/Analysis/AnalysisService.cs","Lite/Mcp/McpAnalysisTools.cs"],"failed_classes":["PerformanceMonitorLite.Tests.LiteAnalysisCancellationTests"],"failed_jobs":["Lite tests (2)"]} +{"date":"2026-09-28","head_sha":"fc2901105b48a239f39a7d6f8a6a22a1c70df497","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingAgReaderTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpAgTools.txt","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingAgReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAgTools.cs"],"failed_classes":["Darling.Tests.DarlingManagedPostgresTests","Darling.Tests.DarlingMcpAgToolsLivePostgresTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (0)"]} +{"date":"2026-09-28","head_sha":"848ae6cda53ffa560c096dc7ed300845ebd3b416","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingAgReaderTests.cs","Darling/Darling.Tests/DarlingMcpAgToolsTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpAgTools.txt","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingAgReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpAgTools.cs"],"failed_classes":["Darling.Tests.DarlingMcpAgToolsLivePostgresTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-28","head_sha":"6fb64ded261eb330697d0cc64162b31101b3d748","event":"pull_request","changed_files":["Darling/Darling.Tests/PlanSync4530Tests.cs","Darling/Darling.Tests/PlanSync4535ConfigPlumbingTests.cs","Darling/Darling.Tests/PlanSync4535NodeRulesATests.cs","Darling/Darling.Tests/PlanSync4535NodeRulesBTests.cs","Darling/Darling.Tests/PlanSync4535RuleNumberCensusTests.cs","Darling/Darling.Tests/PlanSync4535SeverityOverrideTests.cs","Darling/Darling.Tests/PlanSync4535StatementRulesTests.cs","Darling/Darling.Tests/PlanSync4566LegacyMarkerTests.cs","PerformanceMonitor.PlanAnalysis/AnalyzerConfig.cs","PerformanceMonitor.PlanAnalysis/McpPlanAnalysisFormatter.cs","PerformanceMonitor.PlanAnalysis/PlanAnalysisPipeline.cs","PerformanceMonitor.PlanAnalysis/PlanAnalyzer.cs","PerformanceMonitor.PlanAnalysis/PlanModels.cs","PerformanceMonitor.PlanAnalysis/PlanStatements.cs","PerformanceMonitor.PlanAnalysis/PlanWarningDisplay.cs","PerformanceMonitor.PlanAnalysis/ServerMetadata.cs"],"failed_classes":["Darling.Tests.CommentFilterAdoptionTests"],"failed_jobs":["build","Darling PG tests (2)"]} +{"date":"2026-09-28","head_sha":"19583c0276360aa9ef2ec9b10f0952a14f78539f","event":"pull_request","changed_files":["Darling/Darling.Tests/McpServiceParameterDiSeatCensusTests.cs","Darling/Darling.Tests/PlanSync4530Tests.cs","Darling/Darling.Tests/PlanSync4535ConfigLoadingTests.cs","Darling/Darling.Tests/PlanSync4535ConfigPlumbingTests.cs","Darling/Darling.Tests/PlanSync4535NodeRulesATests.cs","Darling/Darling.Tests/PlanSync4535NodeRulesBTests.cs","Darling/Darling.Tests/PlanSync4535RuleNumberCensusTests.cs","Darling/Darling.Tests/PlanSync4535SeverityOverrideTests.cs","Darling/Darling.Tests/PlanSync4535StatementRulesTests.cs","Darling/Darling.Tests/PlanSync4566LegacyMarkerTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.Plans.cs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Service/darling.sample.json","Darling/PerformanceMonitor.Darling.Viewer/ProcedureHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/QueryStatsHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/QueryStoreHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerActualPlanFlow.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Plans.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerSettings.cs","Darling/PerformanceMonitor.Darling.Viewer/WaitDrillDownWindow.xaml.cs","Lite/Analysis/DrillDownCollector.Plans.cs","Lite/Analysis/DuckDbFactCollector.QueryPerf.cs","Lite/App.xaml.cs","Lite/Controls/ServerTab.Plans.cs","Lite/Mcp/McpPlanTools.cs","Lite/config/settings.sample.json","PerformanceMonitor.PlanAnalysis/AnalyzerConfig.cs","PerformanceMonitor.PlanAnalysis/ConfigLoader.cs","PerformanceMonitor.PlanAnalysis/McpPlanAnalysisFormatter.cs","PerformanceMonitor.PlanAnalysis/PlanAdvisoryAggregator.cs","PerformanceMonitor.PlanAnalysis/PlanAnalysisPipeline.cs","PerformanceMonitor.PlanAnalysis/PlanAnalyzer.cs","PerformanceMonitor.PlanAnalysis/PlanModels.cs","PerformanceMonitor.PlanAnalysis/PlanStatements.cs","PerformanceMonitor.PlanAnalysis/PlanWarningDisplay.cs","PerformanceMonitor.PlanAnalysis/ServerMetadata.cs","PerformanceMonitor.Ui/PlanViewerControl.xaml.cs"],"failed_classes":["Darling.Tests.FleetSweepWebFeedTests","Darling.Tests.SharedBaselineCacheTests","PerformanceMonitorLite.Tests.SettingsSampleTests"],"failed_jobs":["Lite tests (0)","Darling PG tests (2)","Darling PG tests (1)","build"]} +{"date":"2026-09-28","head_sha":"9aa0864d286037294a44853d85d547949f567b35","event":"pull_request","changed_files":["Darling/Darling.Tests/PlanSync4530DrillDownLiveTests.cs","Darling/Darling.Tests/PlanSync4530EntryPointsTests.cs","Darling/Darling.Tests/PlanSync4530ReaderLiveTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.Plans.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanTools.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingServerMetadataReader.cs","Darling/PerformanceMonitor.Darling.Storage/PerformanceMonitor.Darling.Storage.csproj","Darling/PerformanceMonitor.Darling.Viewer/ProcedureHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/QueryStatsHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/QueryStoreHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerActualPlanFlow.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Plans.cs","Darling/PerformanceMonitor.Darling.Viewer/WaitDrillDownWindow.xaml.cs","Lite.Tests/PlanServerMetadataReaderTests.cs","Lite/Analysis/DrillDownCollector.Plans.cs","Lite/Controls/FinOpsTab.xaml.cs","Lite/Controls/ServerTab.Plans.cs","Lite/Mcp/McpPlanTools.cs","Lite/Services/LocalDataService.PlanServerMetadata.cs","Lite/Windows/PlanViewerWindow.xaml.cs","Lite/Windows/ProcedureHistoryWindow.xaml.cs","Lite/Windows/QueryStatsHistoryWindow.xaml.cs","Lite/Windows/QueryStoreHistoryWindow.xaml.cs","Lite/Windows/WaitDrillDownWindow.xaml.cs","PerformanceMonitor.PlanAnalysis/McpPlanAnalysisFormatter.cs","PerformanceMonitor.PlanAnalysis/PlanAdvisoryAggregator.cs","PerformanceMonitor.Ui/PlanViewerControl.xaml.cs"],"failed_classes":["PerformanceMonitorLite.Tests.PlanServerMetadataReaderTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (0)","Darling PG tests (2)","Lite tests (3)","Darling whole-tree guards"]} +{"date":"2026-09-28","head_sha":"0f3d97c46417ea9951444119b40c7154be79d1cf","event":"pull_request","changed_files":["Darling/Darling.Tests/McpServiceParameterDiSeatCensusTests.cs","Darling/Darling.Tests/PlanSync4535ConfigLoadingTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.Plans.cs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Service/darling.sample.json","Darling/PerformanceMonitor.Darling.Viewer/ProcedureHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/QueryStatsHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/QueryStoreHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerActualPlanFlow.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Plans.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerSettings.cs","Darling/PerformanceMonitor.Darling.Viewer/WaitDrillDownWindow.xaml.cs","Lite/Analysis/DrillDownCollector.Plans.cs","Lite/Analysis/DuckDbFactCollector.QueryPerf.cs","Lite/App.xaml.cs","Lite/Controls/ServerTab.Plans.cs","Lite/Mcp/McpPlanTools.cs","Lite/config/settings.sample.json","PerformanceMonitor.PlanAnalysis/ConfigLoader.cs","PerformanceMonitor.PlanAnalysis/McpPlanAnalysisFormatter.cs","PerformanceMonitor.PlanAnalysis/PlanAdvisoryAggregator.cs","PerformanceMonitor.Ui/PlanViewerControl.xaml.cs"],"failed_classes":["Darling.Tests.FleetSweepWebFeedTests","Darling.Tests.SharedBaselineCacheTests","PerformanceMonitorLite.Tests.SettingsSampleTests"],"failed_jobs":["build","Darling PG tests (2)","Darling PG tests (1)","Lite tests (0)"]} +{"date":"2026-09-28","head_sha":"a112c09d5dae43911e0947be7be16edf29fd1b32","event":"pull_request","changed_files":["Darling/Darling.Tests/PlanSync4530DrillDownLiveTests.cs","Darling/Darling.Tests/PlanSync4530EntryPointsTests.cs","Darling/Darling.Tests/PlanSync4530ReaderLiveTests.cs","Darling/Darling.Tests/packages.lock.json","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.Plans.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanTools.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingServerMetadataReader.cs","Darling/PerformanceMonitor.Darling.Storage/PerformanceMonitor.Darling.Storage.csproj","Darling/PerformanceMonitor.Darling.Viewer/ProcedureHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/QueryStatsHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/QueryStoreHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerActualPlanFlow.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Plans.cs","Darling/PerformanceMonitor.Darling.Viewer/WaitDrillDownWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/packages.lock.json","Lite.Tests/PlanServerMetadataReaderTests.cs","Lite/Analysis/DrillDownCollector.Plans.cs","Lite/Controls/FinOpsTab.xaml.cs","Lite/Controls/ServerTab.Plans.cs","Lite/Mcp/McpPlanTools.cs","Lite/Services/LocalDataService.PlanServerMetadata.cs","Lite/Windows/PlanViewerWindow.xaml.cs","Lite/Windows/ProcedureHistoryWindow.xaml.cs","Lite/Windows/QueryStatsHistoryWindow.xaml.cs","Lite/Windows/QueryStoreHistoryWindow.xaml.cs","Lite/Windows/WaitDrillDownWindow.xaml.cs","PerformanceMonitor.PlanAnalysis/McpPlanAnalysisFormatter.cs","PerformanceMonitor.PlanAnalysis/PlanAdvisoryAggregator.cs","PerformanceMonitor.Ui/PlanViewerControl.xaml.cs"],"failed_classes":["Darling.Tests.StorageCommandTimeoutTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-28","head_sha":"61da64f50fce7cb65b3c5cbdacab7b7a40a7b6c3","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeStatementTimeoutReloadTests.cs","Darling/Darling.Tests/ComposeTempFileLimitLiveTests.cs","Darling/Darling.Tests/ReadOutcomeClassifierTests.cs","Darling/Darling.Tests/WebExceptionTextCensusTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeSpec.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingReadLatencyReader.cs","Darling/PerformanceMonitor.Darling.Service/ReadLatencyAccumulator.cs","Darling/PerformanceMonitor.Darling.Service/ReadOutcomeClassifier.cs","Darling/tools/provision-roles.sql"],"failed_classes":["Darling.Tests.ControlPlaneReloadDurabilityTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-28","head_sha":"92a3e688986d43f28d04551bc77da2e7790137e2","event":"pull_request","changed_files":["Darling/Darling.Tests/CaggGroupIndexDropRungTests.cs","Darling/Darling.Tests/IntervalFirstExecIndexRungTests.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs"],"failed_classes":["Darling.Tests.CaggGroupIndexDropLiveTests","Darling.Tests.CaggGroupIndexDropUpgradeLiveTests","Darling.Tests.MigrationDataMovingRungCensusPins"],"failed_jobs":["build","Darling PG tests (2)","Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-09-28","head_sha":"310326cc4e2c5d3a2c5e95efd02227de99c37fc7","event":"pull_request","changed_files":["Darling/Darling.Tests/StoreSelfMetricsTests.cs","Darling/PerformanceMonitor.Darling.Storage/StoreSelfMetrics.cs"],"failed_classes":["Darling.Tests.StoreSelfMetricsTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-28","head_sha":"1d05c9cf78df0fd030219b8f4408882251c477b7","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeQueryStoreWideExactnessLiveTests.cs","Darling/Darling.Tests/DarlingComposeTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeCompiler.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs"],"failed_classes":["Darling.Tests.McpReadCommandTimeoutTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-28","head_sha":"5af839411c6eae48fbf98e0df87b18bc288f4e6d","event":"pull_request","changed_files":["Darling/Darling.Tests/CaggGroupIndexDropRungTests.cs","Darling/Darling.Tests/IntervalFirstExecIndexRungTests.cs","Darling/Darling.Tests/MigrationDataMovingRungCensusPins.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs"],"failed_classes":["Darling.Tests.CaggGroupIndexDropLiveTests","Darling.Tests.CaggGroupIndexDropUpgradeLiveTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (1)"]} +{"date":"2026-09-28","head_sha":"cb5545cb1e70305431f89cd710bfc1e35b018aa9","event":"pull_request","changed_files":["CHANGELOG.md","Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs","Darling/Darling.Tests/CaggGroupIndexDropRungTests.cs","Darling/Darling.Tests/CaggGroupIndexDropUpgradeLiveTests.cs","Darling/Darling.Tests/CollectionHealthPayloadBudgetLiveTests.cs","Darling/Darling.Tests/ComposeQueryStoreWideExactnessLiveTests.cs","Darling/Darling.Tests/ComposeStatementTimeoutReloadTests.cs","Darling/Darling.Tests/ComposeTempFileLimitLiveTests.cs","Darling/Darling.Tests/DarlingComposeTests.cs","Darling/Darling.Tests/FleetSweepWebFeedTests.cs","Darling/Darling.Tests/IntervalFirstExecIndexRungTests.cs","Darling/Darling.Tests/IntervalFirstExecIndexWideRungTests.cs","Darling/Darling.Tests/McpServiceParameterDiSeatCensusTests.cs","Darling/Darling.Tests/MigrationDataMovingRungCensusPins.cs","Darling/Darling.Tests/PlanSync4530ReaderLiveTests.cs","Darling/Darling.Tests/PlanSync4535ConfigLoadingTests.cs","Darling/Darling.Tests/PlanSync4597ServerContextFieldsLiveTests.cs","Darling/Darling.Tests/PlanSync4602AnalysisServiceConfigCensusTests.cs","Darling/Darling.Tests/PlanSync4613ReproScriptTests.cs","Darling/Darling.Tests/ProductiveZeroBandingTests.cs","Darling/Darling.Tests/QueryStoreTrendRoutingCachedLiveTests.cs","Darling/Darling.Tests/QueryStoreTrendRoutingTests.cs","Darling/Darling.Tests/ReadOutcomeClassifierTests.cs","Darling/Darling.Tests/SharedBaselineCacheTests.cs","Darling/Darling.Tests/StoreSelfMetricsTests.cs","Darling/Darling.Tests/SwallowedItemFailureTests.cs","Darling/Darling.Tests/Viewer4597Tests.cs","Darling/Darling.Tests/WebExceptionTextCensusTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.Plans.cs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeCompiler.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeSpec.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTrendTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingReadLatencyReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Service/ReadLatencyAccumulator.cs","Darling/PerformanceMonitor.Darling.Service/ReadOutcomeClassifier.cs","Darling/PerformanceMonitor.Darling.Service/darling.sample.json","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Storage/DarlingServerMetadataReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreTrendRouting.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Storage/StoreSelfMetrics.cs","Darling/PerformanceMonitor.Darling.Viewer/ProcedureHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/QueryStatsHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/QueryStoreHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/Themes/CoolBreezeTheme.xaml","Darling/PerformanceMonitor.Darling.Viewer/Themes/DarkTheme.xaml","Darling/PerformanceMonitor.Darling.Viewer/Themes/LightTheme.xaml","Darling/PerformanceMonitor.Darling.Viewer/ViewerActualPlanFlow.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryTrends.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Plans.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerSettings.cs","Darling/PerformanceMonitor.Darling.Viewer/WaitDrillDownWindow.xaml.cs","Darling/tools/provision-roles.sql","Lite.Tests/CollectionHealthPayloadBudgetToolTests.cs","Lite.Tests/PlanServerMetadataDatabaseArgTests.cs","Lite.Tests/SwallowedItemFailureTests.cs","Lite/Analysis/DrillDownCollector.Plans.cs","Lite/Analysis/DuckDbFactCollector.QueryPerf.cs","Lite/App.xaml.cs","Lite/Controls/ServerTab.Plans.cs","Lite/Mcp/McpHealthTools.cs","Lite/Mcp/McpPlanTools.cs","Lite/Services/LocalDataService.PlanServerMetadata.cs","Lite/Themes/CoolBreezeTheme.xaml","Lite/Themes/DarkTheme.xaml","Lite/Themes/LightTheme.xaml","Lite/config/settings.sample.json","PerformanceMonitor.Common/ServerHealthBands.cs","PerformanceMonitor.PlanAnalysis/ConfigLoader.cs","PerformanceMonitor.PlanAnalysis/McpPlanAnalysisFormatter.cs","PerformanceMonitor.PlanAnalysis/PlanAdvisoryAggregator.cs","PerformanceMonitor.PlanAnalysis/ReproScriptBuilder.cs","PerformanceMonitor.PlanAnalysis/ServerContextCard.cs","PerformanceMonitor.PlanAnalysis/ServerMetadata.cs","PerformanceMonitor.Ui/PlanViewerControl.Properties.cs","PerformanceMonitor.Ui/PlanViewerControl.Rendering.cs","PerformanceMonitor.Ui/PlanViewerControl.xaml","PerformanceMonitor.Ui/PlanViewerControl.xaml.cs"],"failed_classes":["Darling.Tests.ComposeParameterCoverageTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-28","head_sha":"88204f7cd2346e391ca2d14dd59463f6577dc51d","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeSourceRouterTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeSourceRouter.cs"],"failed_classes":["Darling.Tests.RdsDeadlockIngestorTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-28","head_sha":"171809d092c98217f79e6668ff4a0e0c926c5905","event":"pull_request","changed_files":["Darling/Darling.Tests/Viewer4570Tests.cs","Darling/Darling.Tests/Viewer4579Tests.cs","Darling/Darling.Tests/Viewer4627Tests.cs","Darling/Darling.Tests/Viewer4629Tests.cs","Darling/PerformanceMonitor.Darling.Viewer/Themes/CoolBreezeTheme.xaml","Darling/PerformanceMonitor.Darling.Viewer/Themes/DarkTheme.xaml","Darling/PerformanceMonitor.Darling.Viewer/Themes/LightTheme.xaml","Lite/Themes/CoolBreezeTheme.xaml","Lite/Themes/DarkTheme.xaml","Lite/Themes/LightTheme.xaml","PerformanceMonitor.PlanAnalysis/PlanDisplayText.cs","PerformanceMonitor.PlanAnalysis/PlanEdgeColour.cs","PerformanceMonitor.PlanAnalysis/PlanRowAccuracy.cs","PerformanceMonitor.Ui/PlanViewerControl.Minimap.cs","PerformanceMonitor.Ui/PlanViewerControl.Properties.cs","PerformanceMonitor.Ui/PlanViewerControl.Rendering.cs","PerformanceMonitor.Ui/PlanViewerControl.xaml.cs","deprecated/Dashboard/Themes/CoolBreezeTheme.xaml","deprecated/Dashboard/Themes/DarkTheme.xaml","deprecated/Dashboard/Themes/LightTheme.xaml"],"failed_classes":["Darling.Tests.PlanViewerWarningColorAndFixTests","Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["Lite tests (1)","Darling PG tests (1)","build","Darling PG tests (0)"]} +{"date":"2026-09-28","head_sha":"c243ded1b48a404d1a0168dccc6a404d9324db80","event":"pull_request","changed_files":["Darling/Darling.Tests/ScratchPostgres.cs"],"failed_classes":["Darling.Tests.StoreSessionTimeZoneLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-28","head_sha":"edbea21c52008aac952ef33a26bb78ebb710c765","event":"pull_request","changed_files":["Darling/Darling.Tests/ViewerStoreUnavailableShellTests.cs","Darling/PerformanceMonitor.Darling.Viewer/MainWindow.PlanViewer.cs","Darling/PerformanceMonitor.Darling.Viewer/MainWindow.xaml","Darling/PerformanceMonitor.Darling.Viewer/MainWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/StoreUnavailableShell.cs"],"failed_classes":["Darling.Tests.ViewerFleetTimerFanOutPositionTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-28","head_sha":"8b2191583216ec409b936a59f52f03a584e7962a","event":"pull_request","changed_files":["Darling/Darling.Tests/AnomalyObjectStatsLatestSnapshotsLiveTests.cs","Darling/Darling.Tests/CaptureDownChunkOrderTests.cs","Darling/Darling.Tests/DarlingAgStatesReaderLiveEqualityTests.cs","Darling/Darling.Tests/DarlingWatermarkFloorPlanShapeLiveTests.cs","Darling/Darling.Tests/EventWindowedReadsAreBoundedLivePostgresTests.cs","Darling/Darling.Tests/FleetReadsAreBoundedByTheFleetTests.cs","Darling/Darling.Tests/ForcePlanFailuresAccessPathTests.cs","Darling/Darling.Tests/JobHistoryPerServerTopNLiveTests.cs","Darling/Darling.Tests/LatestValueLookbackTests.cs","Darling/Darling.Tests/LivePlanShapeIsolationCensusTests.cs","Darling/Darling.Tests/PgServerConfigToolBoundTests.cs","Darling/Darling.Tests/PgTargetConfigSnapshotBoundTests.cs","Darling/Darling.Tests/ServerListAndSummaryPlanShapeTests.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-28","head_sha":"06815880fb3814dfc039eccfb20cc3adbc77cb39","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingComposeTests.cs","Darling/Darling.Tests/DarlingWebAssetsTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/MeasureCatalog.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/editor.js"],"failed_classes":["Darling.Tests.DarlingMcpCustomViewCatalogSizeTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-09-28","head_sha":"990cbc813ebdf9a13033c2896664953731fced0a","event":"pull_request","changed_files":["Darling/Darling.Tests/PgSettingRedactorTests.cs"],"failed_classes":["Darling.Tests.DarlingManagedPostgresTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-28","head_sha":"ec25f4ec0d4b0d5c105fe1a768474e7cfc0aaf40","event":"pull_request","changed_files":["Darling/Darling.Tests/ForcePlanFailuresAccessPathTests.cs","Darling/Darling.Tests/ForcePlanFailuresInvalidationPinsTests.cs","Darling/Darling.Tests/ForcePlanFailuresMemoLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertReadAdapter.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/QueryStoreBackfill.cs"],"failed_classes":["PerformanceMonitorLite.Tests.BlockingDeadlockContextBuilderTests"],"failed_jobs":["Lite tests (2)"]} +{"date":"2026-09-28","head_sha":"bc389dd772ef7f97725ad238991d82eaef9242e3","event":"pull_request","changed_files":["Darling/Darling.Tests/DatabaseWatermarkCacheRunnerLiveTests.cs","Darling/Darling.Tests/DatabaseWatermarkCacheTests.cs","Darling/Darling.Tests/QueryStoreDatabaseWatermarkWiringPinTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DatabaseWatermarkCache.cs","Darling/PerformanceMonitor.Darling.Service/ServerWatermarkCache.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-28","head_sha":"281342891ec870936c96b854c088fc72cf288065","event":"pull_request","changed_files":["Darling/Darling.Tests/WebAutoRefreshPolicyTests.cs","Darling/Darling.Tests/WebFetchLayerTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/css/app.css","Darling/PerformanceMonitor.Darling.Service/wwwroot/index.html","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/app.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/editor.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/notebook.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/views.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/refresh-control.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/refresh-policy.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/views-api.js"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["Darling PG tests (1)","build"]} +{"date":"2026-09-28","head_sha":"76e1865c36d465a9df7a8253723cdb73ab46e38f","event":"pull_request","changed_files":["Darling/Darling.Tests/PgStatementEvictionNoteTests.cs","Darling/Darling.Tests/PgStatementsDeallocLiveTests.cs","Darling/Darling.Tests/PgStatementsEvictionFindingTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetStatementsEvictionRead.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgStatementTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/PgTargetToolRecommendations.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Storage/DarlingPgStatementReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgStatementEvictionNote.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Postgres.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Postgres.cs","Lite.Tests/MeasurementContractCensusTests.cs","Lite.Tests/ServerEpochTests.cs","PerformanceMonitor.Analysis/EvictionFinding.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Config.cs","PerformanceMonitor.Analysis/PgTargetFactKeys.cs","PerformanceMonitor.Analysis/PgTargetScorer.Config.cs","PerformanceMonitor.Collectors/PgStatementStatsCollector.cs","PerformanceMonitor.Collectors/ServerEpoch.cs"],"failed_classes":["Darling.Tests.CollectorMeasurementSeamTests","Darling.Tests.PgServerConfigScopeRungTests","Darling.Tests.PgStatementStatsRowCoherentResetTests","Darling.Tests.PgTargetConfigSnapshotBoundTests","Lite.Tests.PgStatementStatsDeltaSkipTests","Lite.Tests.PgStatementStatsFlavorTests","Lite.Tests.ServerEpochTests","PerformanceMonitorLite.Tests.MeasurementContractCensusTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (0)","build","Lite tests (1)","Lite tests (0)"]} +{"date":"2026-09-28","head_sha":"f4bbfe5fd547e8f348800dd03ed7606d82bec744","event":"pull_request","changed_files":["Darling/Darling.Tests/DocCommentHygieneTests.cs","Darling/Darling.Tests/ViewerDisabledListBoxThemeTests.cs","Darling/PerformanceMonitor.Darling.Viewer/Themes/CoolBreezeTheme.xaml","Darling/PerformanceMonitor.Darling.Viewer/Themes/DarkTheme.xaml","Darling/PerformanceMonitor.Darling.Viewer/Themes/LightTheme.xaml"],"failed_classes":["Darling.Tests.PgLogEventsLivePostgresTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-29","head_sha":"0d66ce2272f78d37650e2d4d7d822c48d642ef3d","event":"pull_request","changed_files":[".claude/skills/lane-orders/SKILL.md","Darling/Darling.Tests/WaitRateTileReadCountLiveTests.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-09-29","head_sha":"8ab61d79df7d55a5db338babc1019c511189b067","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeQueryStoreWideStartLiveTests.cs","Darling/Darling.Tests/DarlingComposeTests.cs","Darling/Darling.Tests/DarlingMcpDataToolsTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideBelowFloorLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideGateTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideHistoryNoteTests.cs","Darling/Darling.Tests/QueryStoreTopWindowTests.cs","Darling/Darling.Tests/RawWindowFloorViewerPortTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeCompiler.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalWide.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStore.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryTrends.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Queries.cs"],"failed_classes":["Darling.Tests.ComposeParameterCoverageTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-09-29","head_sha":"8921ce59dfb077199686bf90a6dc7aef35169bb0","event":"pull_request","changed_files":["Darling/Darling.Tests/PgLogEventsPipelineTests.cs"],"failed_classes":["Darling.Tests.LiveCleanupConversionRatchetTests"],"failed_jobs":["build","Darling PG tests (2)"]} +{"date":"2026-09-29","head_sha":"b7afe92c2977bd3ab4a4d3dd62ba34821f740fa7","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorMeasurementSeamTests.cs","Darling/Darling.Tests/LiveTailQuery.cs","Darling/Darling.Tests/PgDeadlocksPlanCaptureResumeTests.cs","Darling/Darling.Tests/PgDeadlocksResumeLiveTests.cs","Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Darling/Darling.Tests/PgLogTailStderrSiblingLiveTests.cs","Darling/Darling.Tests/PgPlanCaptureLiveTests.cs","Darling/Darling.Tests/PgRaiseShapedRecordsLiveTests.cs","Darling/Darling.Tests/PgServerLogTailBinaryRouteTests.cs","Darling/Darling.Tests/PgServerLogTailCsvJsonRotationLiveTests.cs","Darling/Darling.Tests/PgServerLogTailResumeTests.cs","Darling/Darling.Tests/PgServerLogTailRotationLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Lite.Tests/PgPlanCaptureCollectorDefinitionTests.cs","PerformanceMonitor.Collectors/PgDeadlocksCollector.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgServerLogTail.cs"],"failed_classes":["Darling.Tests.CollectorStateContractTests","Darling.Tests.PgDeadlocksCsvlogLiveTests","Darling.Tests.PgLogEventsCsvlogLiveTests","Darling.Tests.PgLogEventsJsonlogLiveTests","Darling.Tests.PgLogEventsLivePostgresTests","Darling.Tests.PgPlanCaptureCsvLiveTests","Darling.Tests.TsqlConventionGuardTests","Lite.Tests.DefaultTraceEventsCollectorDefinitionTests","Lite.Tests.PgServerLogPathPinTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (2)","Lite tests (1)","Lite tests (0)","Darling PG tests (0)"]} +{"date":"2026-09-29","head_sha":"b125deec3a61e42b6ff8ad6feb849f4d9090c633","event":"pull_request","changed_files":["Darling/Darling.Tests/Fixtures/OriginPlans/memory_grant_wait_plan.sqlplan","Darling/Darling.Tests/Fixtures/OriginPlans/serially-parallel.sqlplan","Darling/Darling.Tests/Fixtures/OriginPlans/spill_plan.sqlplan","Darling/Darling.Tests/PlanSync4531Tests.cs","Darling/Darling.Tests/PlanSync4535StatementRulesTests.cs","Darling/Darling.Tests/PlanSync4686ExcessiveMemoryGrantTests.cs","Darling/Darling.Tests/PlanSync4687CeGuessDetectionTests.cs","Darling/Darling.Tests/PlanSync4690ExpensiveOperatorExchangeTests.cs","PerformanceMonitor.PlanAnalysis/PlanAnalyzer.cs","PerformanceMonitor.PlanAnalysis/PlanModels.cs","PerformanceMonitor.PlanAnalysis/ShowPlanParser.cs"],"failed_classes":["Darling.Tests.DarlingManagedPostgresTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-29","head_sha":"a57bc17ec854c6604d575ec24f25f8b3e3ef658a","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorMeasurementSeamTests.cs","Darling/Darling.Tests/LiveTailQuery.cs","Darling/Darling.Tests/PgDeadlocksCsvlogLiveTests.cs","Darling/Darling.Tests/PgDeadlocksPlanCaptureResumeTests.cs","Darling/Darling.Tests/PgDeadlocksResumeLiveTests.cs","Darling/Darling.Tests/PgLogEventsCsvlogLiveTests.cs","Darling/Darling.Tests/PgLogEventsJsonlogLiveTests.cs","Darling/Darling.Tests/PgLogEventsPipelineTests.cs","Darling/Darling.Tests/PgLogTailStderrSiblingLiveTests.cs","Darling/Darling.Tests/PgPlanCaptureCsvLiveTests.cs","Darling/Darling.Tests/PgPlanCaptureLiveTests.cs","Darling/Darling.Tests/PgRaiseShapedRecordsLiveTests.cs","Darling/Darling.Tests/PgServerLogTailBinaryRouteTests.cs","Darling/Darling.Tests/PgServerLogTailCsvJsonRotationLiveTests.cs","Darling/Darling.Tests/PgServerLogTailResumeTests.cs","Darling/Darling.Tests/PgServerLogTailRotationLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Lite.Tests/PgPlanCaptureCollectorDefinitionTests.cs","PerformanceMonitor.Collectors/PgDeadlocksCollector.cs","PerformanceMonitor.Collectors/PgLogEventsCollector.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgServerLogTail.cs"],"failed_classes":["Darling.Tests.CollectorStateContractTests","Darling.Tests.TsqlConventionGuardTests","Lite.Tests.DefaultTraceEventsCollectorDefinitionTests","Lite.Tests.PgServerLogPathPinTests"],"failed_jobs":["Darling PG tests (1)","Lite tests (1)","build","Lite tests (0)"]} +{"date":"2026-09-29","head_sha":"1954b70d2d5308678af313170d346d0f912a5d2a","event":"pull_request","changed_files":["Darling/Darling.Tests/PgTargetMeasuredLineageTests.cs","Darling/Darling.Tests/PgTargetMemoryTests.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Memory.cs","PerformanceMonitor.Analysis/PgTargetScorer.Memory.cs"],"failed_classes":["Darling.Tests.PgServerLogTailRotationLiveTests","Darling.Tests.PgTargetMemoryTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-29","head_sha":"0477bc1d12a2352fce89d48bf7b90e2d0497af7e","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingSecretsDecryptFailureTests.cs","Darling/Darling.Tests/ServerConnectPolicyTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/ServerConnectPolicy.cs","Darling/PerformanceMonitor.Darling.Service/Targets/SqlServerTargetProvider.cs"],"failed_classes":["Darling.Tests.DarlingCloudIdentityProbeTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-29","head_sha":"03d9905807c3ed5cf6134cf047d671fea6b39bc3","event":"pull_request","changed_files":["Darling/Darling.Tests/RdsLogEventIngestorCsvlogTests.cs","Darling/Darling.Tests/RdsLogSourceTests.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogSource.cs"],"failed_classes":["Darling.Tests.PgServerLogTailRotationLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-29","head_sha":"a59f6dc910aaf76b792296625f40e8ed40cbc310","event":"pull_request","changed_files":["Darling/Darling.Tests/PgPlanCaptureCsvUnitTests.cs","Darling/Darling.Tests/PgPlanCaptureRaiseShapedUnitTests.cs","Darling/Darling.Tests/PgServerLogCsvParserTests.cs","PerformanceMonitor.Collectors/PgLogEntry.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgServerLogCsvParser.cs"],"failed_classes":["Darling.Tests.PgServerLogTailRotationLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-29","head_sha":"a60e32f36fe48521bcce777a497a04c63683d25f","event":"pull_request","changed_files":["Darling/Darling.Tests/HourlyWindowEdgesTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/TopProceduresHourlyRoutingLiveTests.cs","Darling/Darling.Tests/TopQueriesHourlyRoutingLiveTests.cs","Darling/Darling.Tests/TopQueriesHourlyRoutingTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Storage/HourlyWindowEdges.cs"],"failed_classes":["Darling.Tests.McpToolGuideHeadsDataTests","Darling.Tests.McpToolGuideTests","Darling.Tests.McpToolsListBudgetTests","Darling.Tests.TopProceduresHourlyRoutingTests"],"failed_jobs":["Darling PG tests (1)","build","Lite tests (0)","Darling PG tests (0)"]} +{"date":"2026-09-29","head_sha":"38f19e703047e1f2daa46e4d7c5f5afbb73734bd","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingStoreUpgradeTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreUpgrade.cs"],"failed_classes":["Darling.Tests.DarlingStoreUpgradeRevertTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-09-29","head_sha":"216a4317f332258fb49dcceb5a818b9229c75795","event":"push","changed_files":["Darling/Darling.Tests/PgPlanCaptureCsvUnitTests.cs","Darling/Darling.Tests/PgPlanCaptureRaiseShapedUnitTests.cs","Darling/Darling.Tests/PgServerLogCsvParserTests.cs","PerformanceMonitor.Collectors/PgLogEntry.cs","PerformanceMonitor.Collectors/PgPlanCaptureCollector.cs","PerformanceMonitor.Collectors/PgServerLogCsvParser.cs"],"failed_classes":["Darling.Tests.PgServerLogTailRotationLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-29","head_sha":"7d74d8554134ecb006c5915de3a82dcc82b552f7","event":"pull_request","changed_files":["Darling/Darling.Tests/RdsLogEventIngestorCsvlogTests.cs","Darling/Darling.Tests/RdsLogSourceTests.cs","Darling/Darling.Tests/RdsResumePersistenceTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsCpuIngestor.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsDeadlockIngestor.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsIngestOutcome.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogEventIngestor.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogSource.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsPlanIngestor.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsResumeStore.cs"],"failed_classes":["Darling.Tests.PgServerLogTailRotationLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-29","head_sha":"0654199a72be8f5b18dcf19e330457a9d46dd273","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/DarlingStoreUpgradeRevertTests.cs","Darling/Darling.Tests/DarlingStoreUpgradeTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/DarlingStoreUpgrade.cs"],"failed_classes":["Darling.Tests.PgServerLogTailRotationLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-29","head_sha":"511f357e0dd46c92ec7d2ec4273275228f5f8e1c","event":"push","changed_files":["Darling/Darling.Tests/PgLogBurstAndRotationLiveTests.cs","Darling/Darling.Tests/PgServerLogTailRotationLiveTests.cs"],"failed_classes":["Darling.Tests.DarlingCloudIdentityProbeTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-29","head_sha":"3f390f5e59bd0233aa9a5b324a7119d84a891fb7","event":"pull_request","changed_files":["Darling/Darling.Tests/HourlyWindowEdgesTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/TopProceduresHourlyRoutingLiveTests.cs","Darling/Darling.Tests/TopQueriesHourlyRoutingLiveTests.cs","Darling/Darling.Tests/TopQueriesHourlyRoutingTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Storage/HourlyWindowEdges.cs"],"failed_classes":["Darling.Tests.McpToolGuideHeadsDataTests","Darling.Tests.McpToolGuideTests","Darling.Tests.McpToolsListBudgetTests","Darling.Tests.TopProceduresHourlyRoutingTests"],"failed_jobs":["build","Darling PG tests (0)","Lite tests (0)","Darling PG tests (1)"]} +{"date":"2026-09-29","head_sha":"3c8d96cdf2be3fe53e25695b789c66145b4a4211","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingDeployStaleFileTests.cs","Darling/tools/install-darling.ps1","Darling/tools/upgrade-darling.ps1"],"failed_classes":["Darling.Tests.DarlingFileSecurityTests","Darling.Tests.DarlingInstallLocationTests","Darling.Tests.DarlingRuntimePreflightTests"],"failed_jobs":["build","Darling PG tests (0)","Darling PG tests (1)"]} +{"date":"2026-09-29","head_sha":"2c2da8b65a977d3f5b3f8a53c3ee5af552f59d22","event":"pull_request","changed_files":["Darling/Darling.Tests/PartialDailyHealLiveTests.cs","Darling/Darling.Tests/PartialDailyHealPureTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/PartialDailyHeal.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.MaterializationHoles.cs"],"failed_classes":["Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["Darling PG tests (1)","build"]} +{"date":"2026-09-29","head_sha":"ec5535c1a2aa296aebe09e4af111018919d6185b","event":"pull_request","changed_files":["Lite.Tests/CollectionCycleResumeTests.cs","Lite.Tests/DeltaFamilyIntervalColumnTests.cs","Lite.Tests/NewerSchemaFileGuardTests.cs","Lite.Tests/PerfmonCounterTypeTests.cs","Lite.Tests/QsCaptureModeRungTests.cs","Lite.Tests/SchemaColumnHealingTests.cs","Lite.Tests/TimeHonestyRungTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/Database/SchemaVersionTooNewException.cs","Lite/Services/CollectionBackgroundService.cs"],"failed_classes":["Darling.Tests.QsCaptureModeRouteKnobToastRungTests","Darling.Tests.TimeHonestyRungTests"],"failed_jobs":["build"]} +{"date":"2026-09-29","head_sha":"d85dc2f70e0f1e8b7cff4479a33a639903ddc40e","event":"push","changed_files":["Darling/Darling.Tests/DarlingCliCommandsTests.cs","Darling/Darling.Tests/DarlingNetworkConfigEditorTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingNetworkConfigEditor.cs","Darling/README.md"],"failed_classes":["Darling.Tests.DarlingCustomViewsTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-29","head_sha":"2951d443b1405eec71e4b35272e936f813ac03cb","event":"push","changed_files":["Darling/Darling.Tests/DarlingMcpServerAdminToolsTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerAdminTools.cs","Darling/README.md"],"failed_classes":["Darling.Tests.DarlingMcpStoreHostToolsLiveTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-29","head_sha":"8ae9a3f65c439b01f48f0a2fcb597474dc73209a","event":"pull_request","changed_files":["Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs","Darling/Darling.Tests/ConsumedTimestampFrameDisciplineTests.cs","Darling/Darling.Tests/IntervalFirstExecIndexWideRungTests.cs","Darling/Darling.Tests/QueryStoreIntervalEndRungTests.cs","Darling/Darling.Tests/QueryStorePlanFetchTests.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalWide.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Lite.Tests/GoldenCollectorSchema.cs","Lite.Tests/QueryStoreCollectorDefinitionTests.cs","Lite.Tests/QueryStoreIntervalEndRungTests.cs","Lite.Tests/SchemaColumnHealingTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/Database/SchemaVersionTooNewException.cs","PerformanceMonitor.Collectors/QueryStoreCollector.cs"],"failed_classes":["Darling.Tests.QueryStoreIntervalEndRungTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-29","head_sha":"85773bdac87fdf6efe84bf61122c5b1575133d7b","event":"pull_request","changed_files":["Darling/Darling.Tests/ConsumedTimestampFrameDisciplineTests.cs","Darling/Darling.Tests/DarlingBlockingReaderServerClockTests.cs","Darling/Darling.Tests/DarlingDefaultTraceReaderServerClockTests.cs","Darling/Darling.Tests/DarlingJobReaderServerClockTests.cs","Darling/Darling.Tests/DarlingMcpDefaultTraceToolsTests.cs","Darling/Darling.Tests/DarlingObjectStatsReaderServerClockTests.cs","Darling/Darling.Tests/DarlingPvsReaderServerClockTests.cs","Darling/Darling.Tests/DarlingServerClockReaderTests.cs","Darling/Darling.Tests/McpPayloadClockFrameDisciplineTests.cs","Darling/Darling.Tests/McpServerClockTestSupport.cs","Darling/Darling.Tests/PlanCorrectionFrameLiveTests.cs","Darling/Darling.Tests/ServerLocalReadFrameDisciplineTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingBlockingReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDefaultTraceReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingJobReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingObjectStatsReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingPvsReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingServerClockReader.cs"],"failed_classes":["Darling.Tests.ReadLatencyWebRecordingTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-29","head_sha":"0d6caa336094a50ac2d6afdc2527d7171c15aa13","event":"pull_request","changed_files":["Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs","Darling/Darling.Tests/CheckpointLongestSyncRungTests.cs","Darling/Darling.Tests/CheckpointSyncSamplerTests.cs","Darling/Darling.Tests/DarlingCommandExecutorTests.cs","Darling/Darling.Tests/DarlingRetentionTests.cs","Darling/Darling.Tests/PurgeNowBackgroundTests.cs","Darling/Darling.Tests/QueryStoreIntervalEndRungTests.cs","Darling/Darling.Tests/RawPurgeNowWiringTests.cs","Darling/Darling.Tests/RawTablesLeaveCatalogSweepLiveTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/Darling.Tests/RetentionWalPacerLiveTests.cs","Darling/Darling.Tests/RetentionWalPacerTests.cs","Darling/Darling.Tests/StoreSelfMetricsTests.cs","Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs","Darling/PerformanceMonitor.Darling.Service/CheckpointSyncSampler.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCommandExecutor.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreMetricsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Service/RetentionWalPacer.cs","Darling/PerformanceMonitor.Darling.Service/ServiceCommandDeadlines.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Storage/StoreSelfMetrics.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerCommandDeadlines.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.ControlCommands.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.CollectionHealth.cs"],"failed_classes":["Darling.Tests.PostmasterStartTimeLivePostgresTests","Darling.Tests.QueryStoreIntervalEndRungTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (1)"]} +{"date":"2026-09-29","head_sha":"3247e2f88ea4017a2f25ddf13bebc8bec6954ed3","event":"pull_request","changed_files":["Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs","Darling/Darling.Tests/CheckpointLongestSyncRungTests.cs","Darling/Darling.Tests/CheckpointSyncSamplerTests.cs","Darling/Darling.Tests/DarlingCommandExecutorTests.cs","Darling/Darling.Tests/DarlingMcpStoreMetricsToolsTests.cs","Darling/Darling.Tests/DarlingRetentionTests.cs","Darling/Darling.Tests/PurgeNowBackgroundTests.cs","Darling/Darling.Tests/PurgeNowTotalsWatchTests.cs","Darling/Darling.Tests/QueryStoreIntervalEndRungTests.cs","Darling/Darling.Tests/RawPurgeNowWiringTests.cs","Darling/Darling.Tests/RawTablesLeaveCatalogSweepLiveTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/Darling.Tests/RetentionWalPacerLiveTests.cs","Darling/Darling.Tests/RetentionWalPacerTests.cs","Darling/Darling.Tests/StoreSelfMetricsTests.cs","Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs","Darling/PerformanceMonitor.Darling.Service/CheckpointSyncSampler.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCommandExecutor.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreMetricsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Service/RetentionWalPacer.cs","Darling/PerformanceMonitor.Darling.Service/ServiceCommandDeadlines.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Storage/StoreSelfMetrics.cs","Darling/PerformanceMonitor.Darling.Viewer/PurgeNowWatch.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerCommandDeadlines.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.ControlCommands.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.CollectionHealth.cs"],"failed_classes":["Darling.Tests.PostmasterStartTimeLivePostgresTests","Darling.Tests.QueryStoreIntervalEndRungTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-09-29","head_sha":"e8ddea0318de7b01545e665cb3b2a83a7b3117cd","event":"pull_request","changed_files":["Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs","Darling/Darling.Tests/CheckpointLongestSyncRungTests.cs","Darling/Darling.Tests/CheckpointSyncSamplerTests.cs","Darling/Darling.Tests/DarlingCommandExecutorTests.cs","Darling/Darling.Tests/DarlingMcpStoreMetricsToolsTests.cs","Darling/Darling.Tests/DarlingRetentionTests.cs","Darling/Darling.Tests/PurgeNowBackgroundTests.cs","Darling/Darling.Tests/PurgeNowTotalsWatchTests.cs","Darling/Darling.Tests/QueryStoreIntervalEndRungTests.cs","Darling/Darling.Tests/RawPurgeNowWiringTests.cs","Darling/Darling.Tests/RawTablesLeaveCatalogSweepLiveTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/Darling.Tests/RetentionWalPacerLiveTests.cs","Darling/Darling.Tests/RetentionWalPacerTests.cs","Darling/Darling.Tests/StoreSelfMetricsTests.cs","Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs","Darling/PerformanceMonitor.Darling.Service/CheckpointSyncSampler.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCommandExecutor.cs","Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreMetricsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoreMetricsReader.cs","Darling/PerformanceMonitor.Darling.Service/RetentionWalPacer.cs","Darling/PerformanceMonitor.Darling.Service/ServiceCommandDeadlines.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Storage/StoreSelfMetrics.cs","Darling/PerformanceMonitor.Darling.Viewer/PurgeNowWatch.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerCommandDeadlines.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.ControlCommands.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.CollectionHealth.cs"],"failed_classes":["Darling.Tests.PostmasterStartTimeLivePostgresTests","Darling.Tests.QueryStoreIntervalEndRungTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-09-29","head_sha":"374b6af4c50efccecaa50c34437345eb81513017","event":"pull_request","changed_files":["Darling/Darling.Tests/ConfigChangeTraceAnchorTests.cs","Darling/Darling.Tests/CreationTimeClockFrameDisciplineTests.cs","Darling/Darling.Tests/ParameterSensitivityClockFrameLiveTests.cs","Darling/Darling.Tests/ParameterSensitivityFiringSignatureParityTests.cs","Darling/Darling.Tests/PlanRegressionReadShapeTests.cs","Darling/Darling.Tests/ServerLocalReadFrameDisciplineTests.cs","Darling/Darling.Tests/ServerLocalRowConversionTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.Queries.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs","Darling/PerformanceMonitor.Darling.Analysis/ServerLocalTimes.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertReadAdapter.cs","Lite.Tests/ServerClockDstReadTests.cs","Lite/Analysis/AnalysisService.cs","Lite/Analysis/DrillDownCollector.Queries.cs","Lite/Analysis/DuckDbFactCollector.QueryPerf.cs","Lite/Analysis/PlanCreationClock.cs","Lite/Services/LocalDataService.RunningJobs.cs"],"failed_classes":["Darling.Tests.ParameterSensitiveDrillDownTextLiveTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-29","head_sha":"6634aa7a1dff35234981389d3be04eca73a811b2","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionHealthAggregateTests.cs","Darling/Darling.Tests/FleetPartialDatabaseFailureBandTests.cs","Darling/Darling.Tests/OverviewCollectionHealthRollupRoutingTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs","Darling/PerformanceMonitor.Darling.Storage/CollectionHealthRollupSupport.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs"],"failed_classes":["Darling.Tests.FleetPartialDatabaseFailureBandTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-29","head_sha":"267d275488163ae48377eee077766e723206dce2","event":"pull_request","changed_files":["Darling/Darling.Tests/CountFamilyZeroHistoryTests.cs","Darling/Darling.Tests/DarlingAnomalyBaselineTests.cs","Darling/Darling.Tests/PgTargetAnomalyTests.cs","Darling/Darling.Tests/PgTargetZeroHistoryTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgAnomalyDetector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.WaitsSampled.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetAnomalyDetector.cs","Lite.Tests/CountFamilyZeroHistoryTests.cs","Lite.Tests/ScenarioTests.cs","Lite/Analysis/AnomalyDetector.cs","PerformanceMonitor.Analysis/Baselines/CountFamilyMetadata.cs","PerformanceMonitor.Analysis/FactAdvice.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Anomaly.cs","PerformanceMonitor.Analysis/PgTargetAdvice.Wait.cs"],"failed_classes":["Darling.Tests.PgTargetAnomalyTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-29","head_sha":"58811dedb7e50d3cd038c17f3b23a1981db07754","event":"pull_request","changed_files":["Darling/Darling.Tests/ConsumedTimestampFrameDisciplineTests.cs","Darling/Darling.Tests/CorrelatedLanesXAxisWindowTests.cs","Darling/Darling.Tests/ViewerAlertClockCacheTests.cs","Darling/Darling.Tests/ViewerChartUtcFrameTests.cs","Darling/Darling.Tests/ViewerClockRendererArithmeticTests.cs","Darling/Darling.Tests/ViewerContextMenuTests.cs","Darling/Darling.Tests/ViewerDrillDownTests.cs","Darling/Darling.Tests/ViewerRecommendationServerClockTests.cs","Darling/Darling.Tests/ViewerServerClockTests.cs","Darling/Darling.Tests/ViewerTimeHelperTests.cs","Darling/Darling.Tests/ViewerTimeStaticsCollection.cs","Darling/Darling.Tests/ViewerTypedRangeZoneTests.cs","Darling/Darling.Tests/ViewerWave3Tests.cs","Darling/PerformanceMonitor.Darling.Viewer/CorrelatedTimelineLanesControl.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.Loaders.cs","Darling/PerformanceMonitor.Darling.Viewer/MainWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ManageServersWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ProcedureHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/QueryStatsHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/QueryStoreHistoryWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/RecommendationsViewModel.cs","Darling/PerformanceMonitor.Darling.Viewer/ServerClockCache.cs","Darling/PerformanceMonitor.Darling.Viewer/TimeRangeSlicerControl.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.AlertHistory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Blocking.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Cpu.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryTrends.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Blocking.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.ChartContextMenu.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Charts.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.CollectionHealth.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.CpuScheduler.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.DrillDown.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.FileIo.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.LatchSpinlock.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Memory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Perfmon.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.PlanCache.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.QueryHeatmap.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.QueryTrends.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.SessionStats.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.SystemHealthCharts.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.TimeRange.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Waits.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerTimeHelper.cs","Lite.Tests/AlertBadgeServerOffsetTests.cs","Lite.Tests/AlertHistoryRowClockTests.cs","Lite.Tests/AxesExtensionsTests.cs","Lite.Tests/ChartCsvExportDisplayZoneTests.cs","Lite.Tests/ChartTimeConversionClockTests.cs","Lite.Tests/CollectionHealthWindowTests.cs","Lite.Tests/ComparisonWindowUtcTests.cs","Lite.Tests/CustomRangeStateTests.cs","Lite.Tests/DisplayZoneRendererTests.cs","Lite.Tests/DisplayZoneTests.cs","Lite.Tests/DisplayZoneTextFrameTests.cs","Lite.Tests/DisplayZoneTickTests.cs","Lite.Tests/HistoryGridZoneTests.cs","Lite.Tests/LiteRecommendationServerClockTests.cs","Lite.Tests/McpPageContractTests.cs","Lite.Tests/OverviewComparisonWindowOffsetTests.cs","Lite.Tests/OverviewLanesUtcFrameTests.cs","Lite.Tests/QueriesTabWindowRoundTripTests.cs","Lite.Tests/QueryWindowTruncationTests.cs","Lite.Tests/SecondOccurrenceReadTests.cs","Lite.Tests/ServerClockAsTimeZoneTests.cs","Lite.Tests/ServerTabChartUtcFrameTests.cs","Lite.Tests/ServerTabChartWindowTests.cs","Lite.Tests/ServerTabDrillWindowTests.cs","Lite.Tests/ServerTabHeldRangeTests.cs","Lite.Tests/ServerTimeHelperClockTests.cs","Lite.Tests/ServerTimeHelperCollectionTests.cs","Lite.Tests/TimeHonestyRungTests.cs","Lite.Tests/TimeRangeSlicerLabelsTests.cs","Lite.Tests/TimeWindowsTests.cs","Lite.Tests/TypedBoundTests.cs","Lite.Tests/WindowedReadServerClockTests.cs","Lite/Analysis/Recommendations/LiteRecommendationsViewModel.cs","Lite/App.xaml.cs","Lite/Controls/AlertsHistoryTab.xaml.cs","Lite/Controls/CorrelatedTimelineLanesControl.xaml.cs","Lite/Controls/FinOpsTab.xaml.cs","Lite/Controls/RecommendationsTab.xaml.cs","Lite/Controls/ServerTab.BlockChain.cs","Lite/Controls/ServerTab.BlockingStats.cs","Lite/Controls/ServerTab.Charts.cs","Lite/Controls/ServerTab.Comparison.cs","Lite/Controls/ServerTab.ConfigChanges.cs","Lite/Controls/ServerTab.CpuScheduler.cs","Lite/Controls/ServerTab.DailySummary.cs","Lite/Controls/ServerTab.DrillDown.cs","Lite/Controls/ServerTab.Grids.cs","Lite/Controls/ServerTab.LatchSpinlock.cs","Lite/Controls/ServerTab.Pickers.cs","Lite/Controls/ServerTab.PlanCache.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Controls/ServerTab.SessionStats.cs","Lite/Controls/ServerTab.Slicers.cs","Lite/Controls/ServerTab.SystemEvents.cs","Lite/Controls/ServerTab.SystemHealthCharts.cs","Lite/Controls/ServerTab.TimeRange.cs","Lite/Controls/ServerTab.xaml.cs","Lite/Controls/TimeRangeSlicerControl.xaml.cs","Lite/Helpers/ContextMenuHelper.cs","Lite/MainWindow.xaml.cs","Lite/Mcp/McpBlockingTools.cs","Lite/Services/LocalDataService.AlertHistory.cs","Lite/Services/LocalDataService.Blocking.cs","Lite/Services/LocalDataService.BlockingStats.cs","Lite/Services/LocalDataService.CollectionHealth.cs","Lite/Services/LocalDataService.ConfigChanges.cs","Lite/Services/LocalDataService.Cpu.cs","Lite/Services/LocalDataService.CpuScheduler.cs","Lite/Services/LocalDataService.Discontinuities.cs","Lite/Services/LocalDataService.FileIo.cs","Lite/Services/LocalDataService.LatchSpinlock.cs","Lite/Services/LocalDataService.LongQueries.cs","Lite/Services/LocalDataService.Memory.cs","Lite/Services/LocalDataService.MemoryGrants.cs","Lite/Services/LocalDataService.Perfmon.cs","Lite/Services/LocalDataService.PlanCache.cs","Lite/Services/LocalDataService.PlanCorrection.cs","Lite/Services/LocalDataService.QueryHeatmap.cs","Lite/Services/LocalDataService.QueryStats.cs","Lite/Services/LocalDataService.QueryStore.cs","Lite/Services/LocalDataService.QueryStoreRegressions.cs","Lite/Services/LocalDataService.SessionStats.cs","Lite/Services/LocalDataService.SystemEvents.cs","Lite/Services/LocalDataService.TempDb.cs","Lite/Services/LocalDataService.TrendBuckets.cs","Lite/Services/LocalDataService.WaitStats.cs","Lite/Services/LocalDataService.WaitingTasks.cs","Lite/Services/LocalDataService.cs","Lite/Services/ServerTimeHelper.cs","Lite/Windows/ProcedureHistoryWindow.xaml.cs","Lite/Windows/QueryStatsHistoryWindow.xaml.cs","Lite/Windows/QueryStoreHistoryWindow.xaml.cs","Lite/Windows/WaitDrillDownWindow.xaml.cs","PerformanceMonitor.Analysis/Baselines/ServerClock.cs","PerformanceMonitor.Ui/AxesExtensions.cs","PerformanceMonitor.Ui/ChartHoverHelper.cs","PerformanceMonitor.Ui/CorrelatedCrosshairManager.cs","PerformanceMonitor.Ui/CpuSchedulerChartRenderer.cs","PerformanceMonitor.Ui/CustomRangeState.cs","PerformanceMonitor.Ui/DisplayZone.cs","PerformanceMonitor.Ui/DisplayZoneTickGenerator.cs","PerformanceMonitor.Ui/GroupedTrendChartRenderer.cs","PerformanceMonitor.Ui/SessionStatsChartRenderer.cs","PerformanceMonitor.Ui/SystemHealthChartRenderer.cs","PerformanceMonitor.Ui/TimeWindows.cs"],"failed_classes":["Darling.Tests.ConsumedTimestampFrameDisciplineTests","Darling.Tests.PgLogBurstAndRotationLiveTests","Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (0)","Darling PG tests (2)","build"]} +{"date":"2026-09-30","head_sha":"e01fa13f0137865f98527c36398d4bf0430ba47b","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/CheckpointSyncSamplerTests.cs","Darling/Darling.Tests/ConnectionAlertFailedSendRetryTests.cs","Darling/Darling.Tests/CustomAlertClockStepTests.cs","Darling/Darling.Tests/FailedSendRetryTrackerClockStepTests.cs","Darling/Darling.Tests/FailedSendRetryTrackerTests.cs","Darling/Darling.Tests/FleetGateSignalTests.cs","Darling/Darling.Tests/FleetGateSkipCreditTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/OversizedPlanBacklogPins.cs","Darling/Darling.Tests/PurgeNowBackgroundTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/SelfAlertFailedSendCensusTests.cs","Darling/Darling.Tests/ServerConnectPolicyTests.cs","Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs","Darling/Darling.Tests/TimescaleAvailabilityReprobeTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeStoreAvailability.cs","Darling/PerformanceMonitor.Darling.Service/CustomAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/FleetGateStats.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Service/OversizedPlanBacklogSweep.cs","Darling/README.md","Lite.Tests/AgAlertFailedSendRetryTests.cs","Lite.Tests/ArchiveResetRetryClockStepTests.cs","Lite.Tests/CollectionCycleResumeTests.cs","Lite.Tests/CollectorGridScheduleTests.cs","Lite.Tests/ConnectionAlertRetryInFlightTests.cs","Lite.Tests/HousekeepingClockStepTests.cs","Lite/MainWindow.xaml.cs","Lite/Services/AgAlertEvaluator.cs","Lite/Services/ArchiveService.cs","Lite/Services/CollectionBackgroundService.cs","Lite/Services/ConnectionAlertSendsInFlight.cs","Lite/Services/RemoteCollectorService.BlockedProcessReport.cs","Lite/Services/RemoteCollectorService.cs","Lite/Services/ScheduleManager.cs","PerformanceMonitor.Alerting/FailedSendRetryTracker.cs","PerformanceMonitor.Collectors/CollectorCadence.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs"],"failed_classes":["Darling.Tests.PgTargetAnomalyTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-30","head_sha":"7ed8c5e1f84bd3c91b34154582f96e091d53e37d","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertCooldownClockStepTests.cs","Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/CheckpointSyncSamplerTests.cs","Darling/Darling.Tests/ConnectionAlertFailedSendRetryTests.cs","Darling/Darling.Tests/CustomAlertClockStepTests.cs","Darling/Darling.Tests/FailedSendRetryTrackerClockStepTests.cs","Darling/Darling.Tests/FailedSendRetryTrackerTests.cs","Darling/Darling.Tests/FleetGateSignalTests.cs","Darling/Darling.Tests/FleetGateSkipCreditTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/OversizedPlanBacklogPins.cs","Darling/Darling.Tests/PurgeNowBackgroundTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/SelfAlertFailedSendCensusTests.cs","Darling/Darling.Tests/ServerConnectPolicyTests.cs","Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs","Darling/Darling.Tests/TimescaleAvailabilityReprobeTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeStoreAvailability.cs","Darling/PerformanceMonitor.Darling.Service/CustomAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/FleetGateStats.cs","Darling/PerformanceMonitor.Darling.Service/Hosting/DarlingHttpRefusalLog.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Service/OversizedPlanBacklogSweep.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogSource.cs","Darling/PerformanceMonitor.Darling.Viewer/AlertToastCoordinator.cs","Darling/README.md","Lite.Tests/AgAlertClockStepTests.cs","Lite.Tests/AgAlertFailedSendRetryTests.cs","Lite.Tests/ArchiveResetRetryClockStepTests.cs","Lite.Tests/CollectionCycleResumeTests.cs","Lite.Tests/CollectorGridScheduleTests.cs","Lite.Tests/ConnectionAlertRetryInFlightTests.cs","Lite.Tests/HousekeepingClockStepTests.cs","Lite/MainWindow.xaml.cs","Lite/Services/AgAlertEvaluator.cs","Lite/Services/ArchiveService.cs","Lite/Services/CollectionBackgroundService.cs","Lite/Services/ConnectionAlertSendsInFlight.cs","Lite/Services/RemoteCollectorService.BlockedProcessReport.cs","Lite/Services/RemoteCollectorService.cs","Lite/Services/ScheduleManager.cs","PerformanceMonitor.Alerting/AlertEngine.cs","PerformanceMonitor.Alerting/FailedSendRetryTracker.cs","PerformanceMonitor.Collectors/CollectorCadence.cs","PerformanceMonitor.Collectors/PgReadBinaryFileAdvisory.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Common/LastFiredStamp.cs","PerformanceMonitor.Notifications/AlertFamily.cs","PerformanceMonitor.Notifications/AnalysisNotificationService.cs","PerformanceMonitor.Notifications/IncidentCooldown.cs","PerformanceMonitor.Notifications/RepeatDeliveryBudget.cs"],"failed_classes":["Darling.Tests.PgTargetAnomalyTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-30","head_sha":"8aa1cfbaa1fc300085b6663beb2a3f3cf4eac5a3","event":"pull_request","changed_files":["Darling/Darling.Tests/CountFamilyZeroHistoryTests.cs","Darling/Darling.Tests/DarlingAnomalyBaselineTests.cs","Darling/Darling.Tests/DarlingEventBaselineCoverageReadPlanShapeLiveTests.cs","Darling/Darling.Tests/DarlingEventBaselineCoveredDaysTests.cs","Darling/Darling.Tests/LocalClockBucketKeyTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/PerformanceMonitor.Darling.Analysis/BaselineCache.cs","Darling/PerformanceMonitor.Darling.Analysis/PgBaselineProvider.cs","Lite.Tests/ConnectionThrottlePerAttemptTests.cs","Lite.Tests/CountFamilyZeroHistoryTests.cs","Lite.Tests/EventBaselineCoveredDaysTests.cs","Lite.Tests/LocalClockBucketKeyTests.cs","Lite.Tests/ScenarioTests.cs","Lite.Tests/SqlExceptionFactory.cs","Lite.Tests/SwallowedItemFailureTests.cs","Lite.Tests/TestDataSeeder.cs","Lite.Tests/XeSessionHealthTests.cs","Lite/Analysis/BaselineProvider.cs","Lite/MainWindow.xaml.cs","Lite/Services/LocalDataService.CollectionHealth.cs","Lite/Services/RemoteCollectorService.BlockedProcessReport.cs","Lite/Services/RemoteCollectorService.Deadlocks.cs","Lite/Services/RemoteCollectorService.LongQueryCompletions.cs","Lite/Services/RemoteCollectorService.cs","PerformanceMonitor.Analysis/Baselines/CountFamilyMetadata.cs"],"failed_classes":["Darling.Tests.DarlingEventBaselineCoverageReadPlanShapeLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-30","head_sha":"93d34a73cde110dc5a2017b3695ed591787fee94","event":"pull_request","changed_files":["Darling/Darling.Tests/AnomalyTileWindowEndTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgAnomalyDetector.cs","Lite.Tests/AnomalyTileWindowEndTests.cs","Lite/Analysis/AnomalyDetector.cs"],"failed_classes":["Darling.Tests.RollupBackfillLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-30","head_sha":"48481a3766ddd412ae8dfd702eb7fb052044a660","event":"pull_request","changed_files":["Darling/Darling.Tests/RollupBackfillLiveTests.cs"],"failed_classes":["Darling.Tests.RollupBackfillLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-30","head_sha":"e95166c21cc35aeb56fe02faa1485a5cbf0f5b91","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpTrendToolsTests.cs","Darling/Darling.Tests/HourlyWindowEdgesTests.cs","Darling/Darling.Tests/QueryStoreTrendRoutingTests.cs","Darling/Darling.Tests/RollupWindowEndBoundLiveTests.cs","Darling/Darling.Tests/RollupWindowEndBoundTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeCompiler.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingTrendReader.cs","Darling/PerformanceMonitor.Darling.Storage/DurationTrendRouting.cs","Darling/PerformanceMonitor.Darling.Storage/HourlyWindowEdges.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreTrendRouting.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.ProcedureStats.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStats.cs"],"failed_classes":["Darling.Tests.RollupWindowEndBoundLiveTests","Darling.Tests.ViewerTrendRoutingLivePostgresTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (1)"]} +{"date":"2026-09-30","head_sha":"f12072caa9ea6524a401ced7969e355679aac7ba","event":"push","changed_files":["Darling/Darling.Tests/DarlingMcpTrendToolsTests.cs","Darling/Darling.Tests/HourlyAttributionSpanTests.cs","Darling/Darling.Tests/HourlyWindowEdgesTests.cs","Darling/Darling.Tests/QueryStoreTrendRoutingTests.cs","Darling/Darling.Tests/RollupWindowEndBoundLiveTests.cs","Darling/Darling.Tests/RollupWindowEndBoundTests.cs","Darling/Darling.Tests/ViewerTrendRoutingPortTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeCompiler.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingTrendReader.cs","Darling/PerformanceMonitor.Darling.Storage/DurationTrendRouting.cs","Darling/PerformanceMonitor.Darling.Storage/HourlyWindowEdges.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreTrendRouting.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.ProcedureStats.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStats.cs"],"failed_classes":["Darling.Tests.PgTargetDeadlockDrillDownTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-30","head_sha":"3230e3859c276d771fc9990a34903eca8e25fa20","event":"pull_request","changed_files":["Darling/Darling.Tests/ManagedConfFileTests.cs","Darling/Darling.Tests/ManagedConfRehearsalTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs","Darling/PerformanceMonitor.Darling.Service/ManagedConfFile.cs","Darling/README.md"],"failed_classes":["Darling.Tests.ManagedConfFileLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-09-30","head_sha":"8e13b562d28478554773efd035b2bd05cc740516","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeQueryStoreWideExactnessLiveTests.cs","Darling/Darling.Tests/DarlingComposeTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideFirstExecFloorTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeCompiler.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalWide.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStore.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryTrends.cs"],"failed_classes":["Darling.Tests.DarlingManagedPostgresTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-09-30","head_sha":"8c850b6fba36f9262a6ffa7640678bd8681350db","event":"push","changed_files":["Darling/Darling.Tests/QueryStoreIntervalWideBrinIndexLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideBrinIndexTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalWideBrinIndex.cs"],"failed_classes":["Darling.Tests.DarlingDeployStaleFileTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-09-30","head_sha":"d6c02bf28a397be61b6451a6b68d9565bcc04b13","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedPostgresTests.cs","Darling/Darling.Tests/MoveDirectoryOnceReleasedTests.cs"],"failed_classes":["Darling.Tests.DarlingStoreUpgradeRevertTests"],"failed_jobs":["build"]} +{"date":"2026-09-30","head_sha":"1c0ceba428267369c5ec364a70112e795b43f9ae","event":"pull_request","changed_files":["Lite.Tests/McpToolGuideHeads.AnalyzeProcedurePlan.cs","Lite.Tests/McpToolGuideHeads.AnalyzeQueryPlan.cs","Lite.Tests/McpToolsListBudget/McpPlanTools.txt","Lite.Tests/PlanToolsNotKeptTests.cs","Lite/Mcp/McpPlanTools.cs"],"failed_classes":["Darling.Tests.CollectorEngineCapabilityTests","Darling.Tests.EngineCapabilityReadWiringTests","Darling.Tests.McpToolGuideTests"],"failed_jobs":["build"]} +{"date":"2026-10-01","head_sha":"cd500fb11383cee3a29ff41d04da7c40d2b34817","event":"pull_request","changed_files":["Darling/Darling.Tests/AzureSqlDatabaseHardwareTests.cs","Darling/Darling.Tests/AzureSqlDatabaseHostLeftoversTests.cs","Darling/Darling.Tests/AzureSqlDatabaseHostMathTests.cs","Darling/Darling.Tests/AzureSqlDatabaseMemoryScopeTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.Config.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.Resources.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/view-templates.js","Darling/PerformanceMonitor.Darling.Storage/DarlingServerMetadataReader.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.Loaders.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.xaml","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Inventory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Utilization.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Memory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Memory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml","Lite.Tests/AzureSqlDatabaseHardwareTests.cs","Lite.Tests/AzureSqlDatabaseHostLeftoversTests.cs","Lite.Tests/AzureSqlDatabaseHostMathTests.cs","Lite.Tests/AzureSqlDatabaseMemoryScopeTests.cs","Lite.Tests/FinOpsTests.cs","Lite.Tests/MemoryCollectorsDefinitionTests.cs","Lite.Tests/TestDataSeeder.cs","Lite/Analysis/DrillDownCollector.Plans.cs","Lite/Analysis/DuckDbFactCollector.Config.cs","Lite/Analysis/DuckDbFactCollector.Resources.cs","Lite/Controls/FinOpsTab.xaml","Lite/Controls/FinOpsTab.xaml.cs","Lite/Controls/ServerTab.Charts.cs","Lite/Controls/ServerTab.xaml","Lite/Mcp/McpAnalysisTools.cs","Lite/Mcp/McpMemoryTools.cs","Lite/Mcp/McpQueryTools.cs","Lite/Mcp/McpServerInfoTools.cs","Lite/Services/LocalDataService.FinOps.ServerProperties.cs","Lite/Services/LocalDataService.FinOps.Utilization.cs","Lite/Services/LocalDataService.FinOps.cs","Lite/Services/LocalDataService.Memory.cs","Lite/Services/LocalDataService.PlanServerMetadata.cs","Lite/Services/LocalDataService.ServerInfo.cs","PerformanceMonitor.Analysis/FactAdvice.cs","PerformanceMonitor.Analysis/FactCollectorHelpers.cs","PerformanceMonitor.Analysis/FactRemediation.cs","PerformanceMonitor.Collectors/MemoryStatsCollector.cs","PerformanceMonitor.Common/CpuAttribution.cs","PerformanceMonitor.Common/ProvisioningVerdict.cs","PerformanceMonitor.Common/ServerHardwareScope.cs","PerformanceMonitor.PlanAnalysis/ServerContextCard.cs","PerformanceMonitor.PlanAnalysis/ServerMetadata.cs"],"failed_classes":["Darling.Tests.DarlingMcpDataToolsLivePostgresTests","Darling.Tests.ViewerMemoryLivePostgresTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (1)"]} +{"date":"2026-10-01","head_sha":"111f91096d9fd1da0ebd0c97c7e36eba846810e3","event":"pull_request","changed_files":["Lite.Tests/ArchiveResetRestoresStateTests.cs"],"failed_classes":["PerformanceMonitorLite.Tests.ArchiveResetRestoresStateTests"],"failed_jobs":["Lite tests (0)"]} +{"date":"2026-10-01","head_sha":"67bf55c45aa76ce927c4ca8173bb32c97547031b","event":"pull_request","changed_files":["Lite.Tests/ArchivePreservedStateTablesTests.cs","Lite.Tests/ArchiveResetRestoresStateTests.cs","Lite.Tests/ArchiveWatermarkCacheTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/Services/ArchiveService.cs","Lite/Services/ArchiveWatermarkCache.cs","Lite/Services/RemoteCollectorService.cs"],"failed_classes":["Darling.Tests.ClipboardTextRetryBudgetTests"],"failed_jobs":["build"]} +{"date":"2026-10-01","head_sha":"3b79ba261ac8f37fba0dbb88e13e7fe233300b95","event":"pull_request","changed_files":["Darling/Darling.Tests/AzureSqlDatabaseHardwareTests.cs","Darling/Darling.Tests/AzureSqlDatabaseHostMathTests.cs","Darling/Darling.Tests/AzureSqlDatabaseMemoryScopeTests.cs","Darling/Darling.Tests/AzureSqlDatabaseOwnFiguresTests.cs","Darling/Darling.Tests/TimeHonestyRungTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.Config.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.Resources.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/view-templates.js","Darling/PerformanceMonitor.Darling.Storage/DarlingServerMetadataReader.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.Loaders.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.xaml","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Inventory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Utilization.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Memory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Memory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml","Lite.Tests/AuditConfigEditionTests.cs","Lite.Tests/AzureSqlDatabaseHardwareTests.cs","Lite.Tests/AzureSqlDatabaseHostMathTests.cs","Lite.Tests/AzureSqlDatabaseMemoryScopeTests.cs","Lite.Tests/AzureSqlDatabaseOwnFiguresTests.cs","Lite.Tests/FinOpsTests.cs","Lite.Tests/MemoryCollectorsDefinitionTests.cs","Lite.Tests/TestDataSeeder.cs","Lite/Analysis/DrillDownCollector.Plans.cs","Lite/Analysis/DuckDbFactCollector.Config.cs","Lite/Analysis/DuckDbFactCollector.Resources.cs","Lite/Controls/FinOpsTab.xaml","Lite/Controls/FinOpsTab.xaml.cs","Lite/Controls/ServerTab.Charts.cs","Lite/Controls/ServerTab.xaml","Lite/Mcp/McpAnalysisTools.cs","Lite/Mcp/McpMemoryTools.cs","Lite/Mcp/McpQueryTools.cs","Lite/Mcp/McpServerInfoTools.cs","Lite/Services/LocalDataService.FinOps.ServerProperties.cs","Lite/Services/LocalDataService.FinOps.Utilization.cs","Lite/Services/LocalDataService.FinOps.cs","Lite/Services/LocalDataService.Memory.cs","Lite/Services/LocalDataService.PlanServerMetadata.cs","Lite/Services/LocalDataService.ServerInfo.cs","PerformanceMonitor.Analysis/FactAdvice.cs","PerformanceMonitor.Analysis/FactCollectorHelpers.cs","PerformanceMonitor.Analysis/FactRemediation.cs","PerformanceMonitor.Collectors/MemoryStatsCollector.cs","PerformanceMonitor.Common/CpuAttribution.cs","PerformanceMonitor.Common/ProvisioningVerdict.cs","PerformanceMonitor.Common/ServerHardwareScope.cs","PerformanceMonitor.PlanAnalysis/ServerContextCard.cs","PerformanceMonitor.PlanAnalysis/ServerMetadata.cs"],"failed_classes":["Darling.Tests.DarlingMcpDataToolsLivePostgresTests","Darling.Tests.ViewerMemoryLivePostgresTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-10-01","head_sha":"4bc2f89822bd877607d3231febdcc28956ae136f","event":"pull_request","changed_files":["Darling/Darling.Tests/ClipboardTextRetryBudgetTests.cs","Lite.Tests/ArchivePreservedStateTablesTests.cs","Lite.Tests/ArchiveResetRestoreByNameTests.cs","Lite.Tests/ArchiveResetRestoresStateTests.cs","Lite.Tests/ArchiveWatermarkCacheTests.cs","Lite.Tests/ArchiveWatermarkGreaterOfTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/Services/ArchiveService.cs","Lite/Services/ArchiveWatermarkCache.cs","Lite/Services/RemoteCollectorService.cs"],"failed_classes":["PerformanceMonitorLite.Tests.ArchiveResetRestoreByNameTests","PerformanceMonitorLite.Tests.ArchiveWatermarkGreaterOfTests"],"failed_jobs":["Lite tests (1)","Lite tests (3)"]} +{"date":"2026-10-01","head_sha":"642824844ce83c1b51046f2b32248d1d21c39e14","event":"pull_request","changed_files":["Darling/Darling.Tests/ClipboardTextRetryBudgetTests.cs","Lite.Tests/ArchivePreservedStateTablesTests.cs","Lite.Tests/ArchiveResetRestoreByNameTests.cs","Lite.Tests/ArchiveResetRestoresStateTests.cs","Lite.Tests/ArchiveWatermarkCacheTests.cs","Lite.Tests/ArchiveWatermarkGreaterOfTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/Services/ArchiveService.cs","Lite/Services/ArchiveWatermarkCache.cs","Lite/Services/DuckDbAlertHistoryStore.cs","Lite/Services/RemoteCollectorService.QueryStoreBackfill.cs","Lite/Services/RemoteCollectorService.cs"],"failed_classes":["Lite.Tests.QueryStoreBackfillReadFailureWarningTests"],"failed_jobs":["Lite tests (1)"]} +{"date":"2026-10-01","head_sha":"6ad569e59da599df5925a1adc81ec73c5ad5a902","event":"pull_request","changed_files":["Darling/Darling.Tests/ClipboardTextRetryBudgetTests.cs","Lite.Tests/ArchivePreservedStateTablesTests.cs","Lite.Tests/ArchiveResetRestoreByNameTests.cs","Lite.Tests/ArchiveResetRestoresStateTests.cs","Lite.Tests/ArchiveWatermarkCacheTests.cs","Lite.Tests/ArchiveWatermarkGreaterOfTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/Services/ArchiveService.cs","Lite/Services/ArchiveWatermarkCache.cs","Lite/Services/DuckDbAlertHistoryStore.cs","Lite/Services/RemoteCollectorService.QueryStoreBackfill.cs","Lite/Services/RemoteCollectorService.cs"],"failed_classes":["Lite.Tests.QueryStoreBackfillReadFailureWarningTests","PerformanceMonitorLite.Tests.ArchiveResetRestoresStateTests"],"failed_jobs":["Lite tests (1)","Lite tests (0)"]} +{"date":"2026-10-01","head_sha":"65416cbea5f61a2b0d098c3d3dedd2b09a54d85b","event":"push","changed_files":["Darling/Darling.Tests/DarlingMcpDataToolsTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpDataTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/MeasurementContractCensusTests.cs","Darling/Darling.Tests/PerfmonCounterTypeRungTests.cs","Darling/Darling.Tests/PerfmonPerSecondPayloadTests.cs","Darling/Darling.Tests/WebPerfmonPerSecondBehaviourTests.cs","Darling/Darling.Tests/web-perfmon-harness.mjs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTrendTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/editor.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js","Lite.Tests/DeltaSeriesShapingTests.cs","Lite.Tests/McpToolsListBudget/McpPerfmonTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite.Tests/MeasurementContractCensusTests.cs","Lite.Tests/PerfmonCounterTypeTests.cs","Lite.Tests/PerfmonPerSecondRoundingTests.cs","Lite/Mcp/McpPerfmonTools.cs","Lite/Services/LocalDataService.Perfmon.cs","PerformanceMonitor.Common/DeltaSeriesShaping.cs","PerformanceMonitor.Common/Mcp/TrendPayloads.cs"],"failed_classes":["Darling.Tests.ClipboardTextRetryBudgetTests"],"failed_jobs":["build"]} +{"date":"2026-10-01","head_sha":"a6cdbd8c1e9896c64cc242729e4409e04d96ab8e","event":"pull_request","changed_files":["Darling/Darling.Tests/AzureSqlDatabaseHardwareTests.cs","Darling/Darling.Tests/AzureSqlDatabaseHostMathTests.cs","Darling/Darling.Tests/AzureSqlDatabaseMemoryScopeTests.cs","Darling/Darling.Tests/AzureSqlDatabaseOwnFiguresTests.cs","Darling/Darling.Tests/DarlingReadsUseOnlyStoreViewsTests.cs","Darling/Darling.Tests/TimeHonestyRungTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.Config.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.Resources.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/view-templates.js","Darling/PerformanceMonitor.Darling.Storage/DarlingServerMetadataReader.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.Loaders.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.xaml","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Inventory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Recommendations.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Utilization.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Memory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Memory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml","Lite.Tests/AuditConfigEditionTests.cs","Lite.Tests/AzureSqlDatabaseHardwareTests.cs","Lite.Tests/AzureSqlDatabaseHostMathTests.cs","Lite.Tests/AzureSqlDatabaseMemoryScopeTests.cs","Lite.Tests/AzureSqlDatabaseOwnFiguresTests.cs","Lite.Tests/FinOpsTests.cs","Lite.Tests/MemoryCollectorsDefinitionTests.cs","Lite.Tests/TestDataSeeder.cs","Lite/Analysis/DrillDownCollector.Plans.cs","Lite/Analysis/DuckDbFactCollector.Config.cs","Lite/Analysis/DuckDbFactCollector.Resources.cs","Lite/Controls/FinOpsTab.xaml","Lite/Controls/FinOpsTab.xaml.cs","Lite/Controls/ServerTab.Charts.cs","Lite/Controls/ServerTab.xaml","Lite/Mcp/McpAnalysisTools.cs","Lite/Mcp/McpMemoryTools.cs","Lite/Mcp/McpQueryTools.cs","Lite/Mcp/McpServerInfoTools.cs","Lite/Services/LocalDataService.FinOps.Recommendations.cs","Lite/Services/LocalDataService.FinOps.ServerProperties.cs","Lite/Services/LocalDataService.FinOps.Utilization.cs","Lite/Services/LocalDataService.FinOps.cs","Lite/Services/LocalDataService.Memory.cs","Lite/Services/LocalDataService.PlanServerMetadata.cs","Lite/Services/LocalDataService.ServerInfo.cs","PerformanceMonitor.Analysis/FactAdvice.cs","PerformanceMonitor.Analysis/FactCollectorHelpers.cs","PerformanceMonitor.Analysis/FactRemediation.cs","PerformanceMonitor.Collectors/MemoryStatsCollector.cs","PerformanceMonitor.Common/CpuAttribution.cs","PerformanceMonitor.Common/ProvisioningVerdict.cs","PerformanceMonitor.Common/ServerHardwareScope.cs","PerformanceMonitor.PlanAnalysis/ServerContextCard.cs","PerformanceMonitor.PlanAnalysis/ServerMetadata.cs"],"failed_classes":["Darling.Tests.ViewerFinOpsRecommendationsTests"],"failed_jobs":["build","Darling PG tests (2)"]} +{"date":"2026-10-01","head_sha":"32505f5632b5fcc67e7d10ace30dd74f496c1304","event":"pull_request","changed_files":["Darling/Darling.Tests/ClipboardTextRetryBudgetTests.cs","Lite.Tests/ArchivePreservedStateTablesTests.cs","Lite.Tests/ArchiveResetRestoreByNameTests.cs","Lite.Tests/ArchiveResetRestoresStateTests.cs","Lite.Tests/ArchiveWatermarkCacheTests.cs","Lite.Tests/ArchiveWatermarkGreaterOfTests.cs","Lite.Tests/QueryStoreBackfillReadFailureWarningTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/Services/ArchiveService.cs","Lite/Services/ArchiveWatermarkCache.cs","Lite/Services/DuckDbAlertHistoryStore.cs","Lite/Services/RemoteCollectorService.QueryStoreBackfill.cs","Lite/Services/RemoteCollectorService.cs","PerformanceMonitor.Ui/ClipboardText.cs"],"failed_classes":["PerformanceMonitorLite.Tests.ArchiveWatermarkGreaterOfTests"],"failed_jobs":["Lite tests (1)"]} +{"date":"2026-10-01","head_sha":"aaa855bb1f888df3f9f0218aa873a903fb117d26","event":"pull_request","changed_files":["Darling/Darling.Tests/ClipboardTextRetryBudgetTests.cs","Lite.Tests/ArchivePreservedStateTablesTests.cs","Lite.Tests/ArchiveResetRestoreByNameTests.cs","Lite.Tests/ArchiveResetRestoresStateTests.cs","Lite.Tests/ArchiveWatermarkCacheTests.cs","Lite.Tests/ArchiveWatermarkGreaterOfTests.cs","Lite.Tests/QueryStoreBackfillReadFailureWarningTests.cs","Lite/Database/DuckDbInitializer.cs","Lite/Database/PreservedTableRestore.cs","Lite/Services/ArchiveService.cs","Lite/Services/ArchiveWatermarkCache.cs","Lite/Services/DuckDbAlertHistoryStore.cs","Lite/Services/RemoteCollectorService.QueryStoreBackfill.cs","Lite/Services/RemoteCollectorService.cs","PerformanceMonitor.Ui/ClipboardText.cs"],"failed_classes":["Darling.Tests.SingleFlightTtlCacheTests","PerformanceMonitorLite.Tests.ArchiveResetRestoresStateTests","PerformanceMonitorLite.Tests.FinOpsTests"],"failed_jobs":["build","Lite tests (0)","Lite tests (2)"]} +{"date":"2026-10-01","head_sha":"6fae165d3ecca4418d3b8e7ff9b3ab252239ef64","event":"pull_request","changed_files":["Darling/Darling.Tests/AzureSqlDatabaseHardwareTests.cs","Darling/Darling.Tests/AzureSqlDatabaseHostMathTests.cs","Darling/Darling.Tests/AzureSqlDatabaseMemoryScopeTests.cs","Darling/Darling.Tests/AzureSqlDatabaseOwnFiguresTests.cs","Darling/Darling.Tests/DarlingReadsUseOnlyStoreViewsTests.cs","Darling/Darling.Tests/TimeHonestyRungTests.cs","Darling/Darling.Tests/ViewerAzureSqlDatabaseCallSiteTests.cs","Darling/Darling.Tests/ViewerFinOpsRecommendationsTests.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.Config.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.Resources.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/view-templates.js","Darling/PerformanceMonitor.Darling.Storage/DarlingServerMetadataReader.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.Loaders.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.xaml","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Inventory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Recommendations.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Utilization.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Memory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Memory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml","Lite.Tests/AuditConfigEditionTests.cs","Lite.Tests/AzureSqlDatabaseCallSiteTests.cs","Lite.Tests/AzureSqlDatabaseHardwareTests.cs","Lite.Tests/AzureSqlDatabaseHostMathTests.cs","Lite.Tests/AzureSqlDatabaseMemoryScopeTests.cs","Lite.Tests/AzureSqlDatabaseOwnFiguresTests.cs","Lite.Tests/FinOpsTests.cs","Lite.Tests/MemoryCollectorsDefinitionTests.cs","Lite.Tests/TestDataSeeder.cs","Lite/Analysis/DrillDownCollector.Plans.cs","Lite/Analysis/DuckDbFactCollector.Config.cs","Lite/Analysis/DuckDbFactCollector.Resources.cs","Lite/Controls/FinOpsTab.xaml","Lite/Controls/FinOpsTab.xaml.cs","Lite/Controls/ServerTab.Charts.cs","Lite/Controls/ServerTab.xaml","Lite/Mcp/McpAnalysisTools.cs","Lite/Mcp/McpMemoryTools.cs","Lite/Mcp/McpQueryTools.cs","Lite/Mcp/McpServerInfoTools.cs","Lite/Services/LocalDataService.FinOps.Recommendations.cs","Lite/Services/LocalDataService.FinOps.ServerProperties.cs","Lite/Services/LocalDataService.FinOps.Utilization.cs","Lite/Services/LocalDataService.FinOps.cs","Lite/Services/LocalDataService.Memory.cs","Lite/Services/LocalDataService.PlanServerMetadata.cs","Lite/Services/LocalDataService.ServerInfo.cs","PerformanceMonitor.Analysis/FactAdvice.cs","PerformanceMonitor.Analysis/FactCollectorHelpers.cs","PerformanceMonitor.Analysis/FactRemediation.cs","PerformanceMonitor.Collectors/MemoryStatsCollector.cs","PerformanceMonitor.Common/CpuAttribution.cs","PerformanceMonitor.Common/ProvisioningVerdict.cs","PerformanceMonitor.Common/ServerHardwareScope.cs","PerformanceMonitor.PlanAnalysis/ServerContextCard.cs","PerformanceMonitor.PlanAnalysis/ServerMetadata.cs"],"failed_classes":["PerformanceMonitorLite.Tests.FinOpsTests"],"failed_jobs":["Lite tests (2)"]} +{"date":"2026-10-01","head_sha":"5a9f598d9240f7222846ce6a7c89dd0055b7f085","event":"pull_request","changed_files":["Lite.Tests/ArchiveResetExportTests.cs","Lite.Tests/ArchiveResetRestoresStateTests.cs","Lite.Tests/ArchiveResetRetryClockStepTests.cs","Lite.Tests/ArchiveViewRefreshAfterDeleteTests.cs","Lite.Tests/MuteRuleServiceTests.cs","Lite.Tests/MuteRuleWindowRefusalTests.cs","Lite.Tests/SharedDuckDbFixture.cs","Lite/Analysis/FindingStore.cs","Lite/Controls/AlertsHistoryTab.xaml.cs","Lite/Controls/SnoozeBalloon.xaml.cs","Lite/Database/DuckDbInitializer.cs","Lite/Database/PreservedTableRestore.cs","Lite/Helpers/PendingRestoreNotice.cs","Lite/Services/ArchiveService.cs","Lite/Services/DuckDbMuteRuleStore.cs","Lite/Services/LocalDataService.AlertHistory.cs","Lite/Services/LocalDataService.ServerTags.cs","Lite/Windows/ManageMuteRulesWindow.xaml.cs"],"failed_classes":["Lite.Tests.MuteRuleWindowRefusalTests","PerformanceMonitorLite.Tests.MuteRuleServiceTests"],"failed_jobs":["Lite tests (0)","Lite tests (2)"]} +{"date":"2026-10-01","head_sha":"bd66895a04c5e51c398b9140f3549c1084e3aba1","event":"pull_request","changed_files":["Darling/Darling.Tests/DatabaseFileGrowthReadTests.cs","Lite.Tests/FileGrowthReadTests.cs"],"failed_classes":["Darling.Tests.DatabaseFileGrowthReadTests","Lite.Tests.FileGrowthReadTests"],"failed_jobs":["Darling PG tests (0)","Lite tests (1)","Darling PG tests (1)"]} +{"date":"2026-10-01","head_sha":"9e2a85a6f2fd46284d7fa33dbe015b607e648cff","event":"pull_request","changed_files":["Darling/Darling.Tests/AzureYoungBaselineWaitBarTests.cs","Lite.Tests/AnomalyDetectorTests.cs"],"failed_classes":["Darling.Tests.AzureYoungBaselineWaitBarTests","PerformanceMonitorLite.Tests.AnomalyDetectorTests"],"failed_jobs":["Lite tests (2)","Darling PG tests (1)"]} +{"date":"2026-10-01","head_sha":"4f25bdea23951ac22453eff82607b3eb0cc88753","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertEngineTests.cs","Darling/Darling.Tests/AzureMasterScopeTests.cs","Darling/Darling.Tests/DarlingMcpBlockingToolsTests.cs","Lite.Tests/McpPageContractTests.cs","PerformanceMonitor.Alerting/AlertServerSnapshot.cs","PerformanceMonitor.Alerting/AzureMasterScope.cs"],"failed_classes":["Darling.Tests.AlertEngineTests","Darling.Tests.AzureMasterScopeTests","Darling.Tests.DarlingMcpBlockingToolsLivePostgresTests","Darling.Tests.DarlingMcpBlockingToolsSurfaceAndSqlTests","PerformanceMonitorLite.Tests.McpPageContractTests"],"failed_jobs":["build","Darling PG tests (2)","Lite tests (3)","Darling PG tests (0)"]} +{"date":"2026-10-01","head_sha":"2cbbde7306120aab122458764ba61e13331ba2c1","event":"pull_request","changed_files":["Darling/Darling.Tests/NoServiceObjectiveToResizeTests.cs","Lite.Tests/FinOpsTests.cs","Lite.Tests/TestDataSeeder.cs","PerformanceMonitor.Common/ProvisioningVerdict.cs","PerformanceMonitor.Common/ServerHardwareScope.cs"],"failed_classes":["Darling.Tests.NoServiceObjectiveToResizeTests"],"failed_jobs":["build","Darling PG tests (0)","Lite tests (2)"]} +{"date":"2026-10-01","head_sha":"376ffbf00b3e347f3c956c8a7f89e33dc0ac293d","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertEngineTests.cs","Darling/Darling.Tests/AzureMasterScopeTests.cs","Darling/Darling.Tests/DarlingMcpBlockingToolsTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingBlockingReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpBlockingTools.cs","Lite.Tests/McpPageContractTests.cs","Lite/MainWindow.AlertEngine.cs","Lite/Mcp/McpBlockingTools.cs","Lite/Services/LocalDataService.Blocking.cs","PerformanceMonitor.Alerting/AlertEngine.cs","PerformanceMonitor.Alerting/AlertServerSnapshot.cs","PerformanceMonitor.Alerting/AzureMasterScope.cs"],"failed_classes":["PerformanceMonitorLite.Tests.TimeHonestyRungTests"],"failed_jobs":["Lite tests (1)"]} +{"date":"2026-10-01","head_sha":"558a66c1466a135688c7a0789a128b369538a410","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertNotebookTableCatalogTests.cs","Darling/Darling.Tests/alert-notebook-harness.mjs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/read-tables.js"],"failed_classes":["Darling.Tests.AlertNotebookTableCatalogTests"],"failed_jobs":["build","Darling PG tests (2)"]} +{"date":"2026-10-01","head_sha":"0d615203ba66a7d44b66ec32ba5bcf82f9757cb1","event":"pull_request","changed_files":["Darling/Darling.Tests/ViewerFinOpsPickerRefreshPinTests.cs","Lite.Tests/FinOpsSizeGridReloadPinTests.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests","Darling.Tests.ViewerFinOpsPickerRefreshPinTests","Lite.Tests.FinOpsSizeGridReloadPinTests"],"failed_jobs":["build","Darling PG tests (2)","Lite tests (0)","Darling PG tests (1)"]} +{"date":"2026-10-01","head_sha":"b36b9747af170930c58b1a3b2574936dcc5203d0","event":"pull_request","changed_files":["Darling/Darling.Tests/AzureSqlDatabaseOwnFiguresTests.cs","Darling/Darling.Tests/NoServiceObjectiveToResizeTests.cs","Darling/Darling.Tests/ServerInventoryFleetMetricsTests.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.Loaders.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Inventory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Recommendations.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Utilization.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.cs","Lite.Tests/AzureSqlDatabaseOwnFiguresTests.cs","Lite.Tests/FinOpsTests.cs","Lite.Tests/TestDataSeeder.cs","Lite/Controls/FinOpsTab.xaml.cs","Lite/Services/LocalDataService.FinOps.Recommendations.cs","Lite/Services/LocalDataService.FinOps.ServerProperties.cs","Lite/Services/LocalDataService.FinOps.Utilization.cs","Lite/Services/LocalDataService.FinOps.cs","PerformanceMonitor.Common/ProvisioningVerdict.cs","PerformanceMonitor.Common/ServerHardwareScope.cs"],"failed_classes":["Darling.Tests.ServerInventoryFleetMetricsTests"],"failed_jobs":["build","Darling PG tests (0)","Lite tests (2)"]} +{"date":"2026-10-01","head_sha":"4afd2c21ab645a06e04cab652ebd43c1525bfb50","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertEngineTests.cs","Darling/Darling.Tests/AzureMasterLiveTargetsTests.cs","Darling/Darling.Tests/AzureMasterScopeTests.cs","Darling/Darling.Tests/DarlingMcpBlockingToolsTests.cs","Darling/PerformanceMonitor.Darling.Service/CustomAlertTemplates.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingBlockingReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpBlockingTools.cs","Lite.Tests/AzureMasterLiteWiringTests.cs","Lite.Tests/McpPageContractTests.cs","Lite/MainWindow.AlertEngine.cs","Lite/Mcp/McpBlockingTools.cs","Lite/Services/LocalDataService.Blocking.cs","PerformanceMonitor.Alerting/AlertEngine.cs","PerformanceMonitor.Alerting/AlertServerSnapshot.cs","PerformanceMonitor.Alerting/AzureMasterScope.cs"],"failed_classes":["PerformanceMonitorLite.Tests.TimeHonestyRungTests"],"failed_jobs":["Lite tests (1)"]} +{"date":"2026-10-01","head_sha":"3d01c908b746940e3a3c37badfe31ded8af029c9","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertNotebookRenderClientTests.cs","Darling/Darling.Tests/AlertNotebookTableCatalogTests.cs","Darling/Darling.Tests/alert-notebook-harness.mjs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/views.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/read-tables.js"],"failed_classes":["Darling.Tests.AlertNotebookTableCatalogTests"],"failed_jobs":["build","Darling PG tests (2)"]} +{"date":"2026-10-01","head_sha":"d33e310c66b4dac2d0fdd59b781459cdd46f6f3b","event":"pull_request","changed_files":["Darling/Darling.Tests/DeadlocksTelemetryCursorTests.cs","Lite.Tests/DeadlocksTelemetryCursorLiteTests.cs"],"failed_classes":["Darling.Tests.DeadlocksTelemetryCursorTests","Lite.Tests.DeadlocksTelemetryCursorLiteTests"],"failed_jobs":["Darling PG tests (1)","Lite tests (0)","build"]} +{"date":"2026-10-01","head_sha":"332eb85a65c35fa7a4b60994f1424d86e39802b0","event":"pull_request","changed_files":["Darling/Darling.Tests/AnalysisHistoryGateTests.cs","Lite.Tests/AnalysisDataSpanTests.cs","Lite/Analysis/AnalysisService.cs","PerformanceMonitor.Analysis/AnalysisHistoryGate.cs"],"failed_classes":["Darling.Tests.AnalysisHistoryGateTests","PerformanceMonitorLite.Tests.AnalysisDataSpanTests"],"failed_jobs":["build","Darling PG tests (0)","Lite tests (3)"]} +{"date":"2026-10-01","head_sha":"68eb108f1655c4531925e366ccd89621d5466ea7","event":"pull_request","changed_files":["Darling/Darling.Tests/StorageGrowthNoHistoryLiveTests.cs","Lite.Tests/AzureSiblingGrowthTests.cs"],"failed_classes":["Darling.Tests.StorageGrowthNoHistoryLiveTests","Lite.Tests.AzureSiblingGrowthTests"],"failed_jobs":["Lite tests (2)","Darling PG tests (0)"]} +{"date":"2026-10-01","head_sha":"fdb481b9da70352fb73239167dba2b90ea53c247","event":"pull_request","changed_files":["Lite.Tests/AzureSqlDatabaseNotCollectedSurfaceTests.cs","Lite/Controls/FinOpsTab.xaml","Lite/Controls/ServerTab.Charts.cs","Lite/Controls/ServerTab.ConfigChanges.cs","Lite/Controls/ServerTab.EngineGaps.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Controls/ServerTab.SystemEvents.cs","Lite/Controls/ServerTab.xaml","Lite/Controls/ServerTab.xaml.cs","Lite/Windows/DatabaseStateOverridesWindow.xaml","Lite/Windows/DatabaseStateOverridesWindow.xaml.cs"],"failed_classes":["Darling.Tests.ViewerScopedPaintGuardTests"],"failed_jobs":["build"]} +{"date":"2026-10-01","head_sha":"495d09d85e2c719e5a512d883316b7882c57de9b","event":"pull_request","changed_files":["Darling/Darling.Tests/RightSizingWindowTests.cs","Lite.Tests/FinOpsTests.cs","Lite.Tests/TestDataSeeder.cs","PerformanceMonitor.Common/RightSizingWindow.cs"],"failed_classes":["PerformanceMonitorLite.Tests.FinOpsTests"],"failed_jobs":["build","Lite tests (2)","Darling PG tests (1)"]} +{"date":"2026-10-01","head_sha":"1eb31abc6367d36d7f9455136b1cd46a7d84aab2","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertEngineTests.cs","Darling/Darling.Tests/AzureMasterAnalysisScopeLiveTests.cs","Darling/Darling.Tests/AzureMasterLiveTargetsTests.cs","Darling/Darling.Tests/AzureMasterScopeTests.cs","Darling/Darling.Tests/DarlingMcpBlockingToolsTests.cs","Darling/Darling.Tests/DeadlockGraphDatabasesTests.cs","Darling/PerformanceMonitor.Darling.Service/CustomAlertTemplates.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingBlockingReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpBlockingTools.cs","Lite.Tests/AzureMasterAnalysisScopeTests.cs","Lite.Tests/AzureMasterLiteWiringTests.cs","Lite.Tests/McpPageContractTests.cs","Lite.Tests/TimeHonestyRungTests.cs","Lite/MainWindow.AlertEngine.cs","Lite/Mcp/McpBlockingTools.cs","Lite/Services/LocalDataService.Blocking.cs","PerformanceMonitor.Alerting/AlertContextBuilders.cs","PerformanceMonitor.Alerting/AlertEngine.cs","PerformanceMonitor.Alerting/AlertServerSnapshot.cs","PerformanceMonitor.Alerting/AzureMasterScope.cs","PerformanceMonitor.Analysis/AnalysisContext.cs","PerformanceMonitor.Common/DeadlockGraphDatabases.cs"],"failed_classes":["Darling.Tests.AzureMasterAnalysisScopeLiveTests","PerformanceMonitorLite.Tests.AzureMasterAnalysisScopeTests"],"failed_jobs":["build","Darling PG tests (1)","Lite tests (0)"]} +{"date":"2026-10-01","head_sha":"8bc022d66d619ac5333922c1100e8700b7a200e7","event":"pull_request","changed_files":["Darling/Darling.Tests/DatabaseConfigOptimizedLockingTests.cs","Darling/Darling.Tests/OptimizedLockingNoteTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingCurrentConfigReader.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Config.cs","Lite.Tests/OptimizedLockingNoteTests.cs","Lite/Services/LocalDataService.Config.cs","PerformanceMonitor.Common/OptimizedLockingNote.cs"],"failed_classes":["Darling.Tests.DatabaseConfigOptimizedLockingGateTests","Darling.Tests.DatabaseConfigOptimizedLockingLivePostgresTests","Darling.Tests.OptimizedLockingNoteLivePostgresTests","Darling.Tests.OptimizedLockingNoteSurfacePinTests","PerformanceMonitorLite.Tests.OptimizedLockingNoteTests"],"failed_jobs":["build","Darling PG tests (0)","Darling PG tests (1)","Darling PG tests (2)","Lite tests (0)"]} +{"date":"2026-10-01","head_sha":"950bb7dbbdcbdbeb23cdae6e829ff80c83ab2bb5","event":"pull_request","changed_files":["Darling/Darling.Tests/AzureSqlDatabaseOwnFiguresTests.cs","Darling/Darling.Tests/NoServiceObjectiveToResizeTests.cs","Darling/Darling.Tests/ServerInventoryFleetMetricsTests.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.Loaders.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.Recommendations.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.xaml","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Inventory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Recommendations.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Utilization.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.cs","Lite.Tests/AzureSqlDatabaseMasterPinsTests.cs","Lite.Tests/AzureSqlDatabaseOwnFiguresTests.cs","Lite.Tests/FinOpsTests.cs","Lite.Tests/TestDataSeeder.cs","Lite/Controls/FinOpsTab.xaml.cs","Lite/Services/LocalDataService.FinOps.Recommendations.cs","Lite/Services/LocalDataService.FinOps.ServerProperties.cs","Lite/Services/LocalDataService.FinOps.Utilization.cs","Lite/Services/LocalDataService.FinOps.cs","PerformanceMonitor.Common/ProvisioningVerdict.cs","PerformanceMonitor.Common/ServerHardwareScope.cs"],"failed_classes":["Darling.Tests.DarlingUpgradeInstallTreeWaitTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-10-01","head_sha":"de3511050a400d15760045235d9b20f60e50bee5","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertNotebookRenderClientTests.cs","Darling/Darling.Tests/AlertNotebookTableCatalogTests.cs","Darling/Darling.Tests/RawWindowFloorViewerPortTests.cs","Darling/Darling.Tests/ViewTemplateLiteralReader.cs","Darling/Darling.Tests/ViewTemplatesTests.cs","Darling/Darling.Tests/alert-notebook-harness.mjs","Darling/Darling.Tests/read-fields-resolve-harness.mjs","Darling/PerformanceMonitor.Darling.Service/AlertNotebookEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/views.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/read-fields.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/view-templates.js"],"failed_classes":["Darling.Tests.AlertNotebookTableCatalogTests"],"failed_jobs":["build","Darling PG tests (2)"]} +{"date":"2026-10-01","head_sha":"bca273d76ed26dbed1b1f2bd4025d22fa7a821b2","event":"pull_request","changed_files":["Darling/Darling.Tests/DeadlockDuplicateCleanupLiveTests.cs","Darling/Darling.Tests/DeadlockStoredIdentityTests.cs","Darling/Darling.Tests/DeadlocksTelemetryCursorStateLiveTests.cs","Darling/Darling.Tests/DeadlocksTelemetryCursorTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DeadlockDuplicateCleanup.cs","Lite.Tests/ArchiveViewDedupTests.cs","Lite.Tests/AzureDeadlockTelemetryTests.cs","Lite.Tests/DeadlockDuplicateCleanupTests.cs","Lite.Tests/DeadlockStoredIdentityTests.cs","Lite.Tests/DeadlocksTelemetryCursorLiteTests.cs","Lite.Tests/XeShredGateTests.cs","Lite/Database/DeadlockDuplicateCleanup.cs","Lite/Services/RemoteCollectorService.DefinitionRunner.cs","PerformanceMonitor.Collectors/BlockedProcessReportCollector.cs","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/DeadlocksCollector.cs","PerformanceMonitor.Collectors/IStoredIdentityDedupedCollector.cs"],"failed_classes":["Darling.Tests.DeadlockDuplicateCleanupLiveTests","Darling.Tests.DeadlockStoredIdentityTests","Darling.Tests.TsqlConventionGuardTests","Lite.Tests.DeadlockStoredIdentityTests","PerformanceMonitorLite.Tests.ArchiveViewDedupTests","PerformanceMonitorLite.Tests.DeadlockDuplicateCleanupTests"],"failed_jobs":["build","Darling PG tests (1)","Lite tests (2)","Lite tests (3)"]} +{"date":"2026-10-01","head_sha":"a07f238b146685b675f6a60ccd7b4b6d5f53870b","event":"pull_request","changed_files":["Lite.Tests/AzureSqlDatabaseNotCollectedSurfaceTests.cs","Lite.Tests/EngineGapNoteTests.cs","Lite/Controls/FinOpsTab.xaml","Lite/Controls/ServerTab.Charts.cs","Lite/Controls/ServerTab.ConfigChanges.cs","Lite/Controls/ServerTab.EngineGaps.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Controls/ServerTab.SystemEvents.cs","Lite/Controls/ServerTab.xaml","Lite/Controls/ServerTab.xaml.cs","Lite/MainWindow.xaml.cs","Lite/Mcp/McpJobTools.cs","Lite/Services/LocalDataService.CollectorRuns.cs","Lite/Windows/DatabaseStateOverridesWindow.xaml","Lite/Windows/DatabaseStateOverridesWindow.xaml.cs"],"failed_classes":["Darling.Tests.ViewerScopedPaintGuardTests"],"failed_jobs":["build"]} +{"date":"2026-10-01","head_sha":"85808894e250f64b3d33f621990a45c4a98cf367","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionNoteLatestRunTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml","Lite/Controls/ServerTab.xaml","Lite/Mcp/McpHealthTools.cs","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Common/ServerHealthBands.cs"],"failed_classes":["PerformanceMonitor.Darling.Tests.RetiredBaselineAggregateLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-10-01","head_sha":"e59a01353b9d24a0a9920e40f4ed54098b7d9978","event":"pull_request","changed_files":["Darling/Darling.Tests/RuntimePreconditionMissTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpJobTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingRuntimePrecondition.cs","Lite.Tests/McpMissMessageParityPinTests.cs","Lite.Tests/McpStatusEnvelopeTests.cs","Lite/Mcp/McpJobTools.cs","Lite/Mcp/McpRuntimePrecondition.cs","Lite/Services/LocalDataService.RuntimePrecondition.cs","PerformanceMonitor.Collectors/CollectorRuntimePrecondition.cs"],"failed_classes":["Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-10-01","head_sha":"6406f8dfa9ac8f5bc54612e137148fe3319d3987","event":"pull_request","changed_files":["Darling/Darling.Tests/AnalyzeServerPerCallAnalysisServiceTests.cs","Darling/Darling.Tests/AnomalyTileWindowEndTests.cs","Darling/Darling.Tests/AzureMasterAnalysisScopeLiveTests.cs","Darling/Darling.Tests/BlockingChainReadsFloorTests.cs","Darling/Darling.Tests/DeadlockGraphDatabasesTests.cs","Darling/Darling.Tests/EventTimeAnalysisReadsLiveTests.cs","Darling/Darling.Tests/FleetSweepWebFeedTests.cs","Darling/Darling.Tests/PgDeadlockRemaskTests.cs","Darling/Darling.Tests/PgTargetDeadlockDrillDownTests.cs","Darling/Darling.Tests/SharedBaselineCacheTests.cs","Darling/PerformanceMonitor.Darling.Analysis/DarlingAnalysisService.cs","Darling/PerformanceMonitor.Darling.Analysis/PgAnomalyDetector.cs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.Blocking.cs","Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.Waits.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetDrillDownCollector.Deadlocks.cs","Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Database.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingPgDeadlockReader.cs","Lite.Tests/AnomalyTileWindowEndTests.cs","Lite.Tests/AzureMasterAnalysisScopeTests.cs","Lite.Tests/EventBaselineCoveredDaysTests.cs","Lite.Tests/EventTimeAnalysisReadsTests.cs","Lite.Tests/LocalClockBucketKeyTests.cs","Lite/Analysis/AnalysisService.cs","Lite/Analysis/AnomalyDetector.cs","Lite/Analysis/BaselineProvider.cs","Lite/Analysis/DrillDownCollector.Blocking.cs","Lite/Analysis/DuckDbFactCollector.Waits.cs","Lite/Analysis/SeparatelyMonitoredScope.cs","Lite/MainWindow.xaml.cs","PerformanceMonitor.Alerting/AlertContextBuilders.cs","PerformanceMonitor.Analysis/AnalysisContext.cs","PerformanceMonitor.Common/DeadlockGraphDatabases.cs"],"failed_classes":["Darling.Tests.DarlingEventBaselineCoveredDaysTests","Darling.Tests.PostgresEngineGateBehaviorTests","Darling.Tests.TsqlConventionGuardTests","PerformanceMonitorLite.Tests.AzureMasterAnalysisScopeTests"],"failed_jobs":["Darling PG tests (1)","Darling PG tests (2)","Darling PG tests (0)","Lite tests (0)","build"]} +{"date":"2026-10-01","head_sha":"0c983ef54d6bc6d796353b3ddfe1c120e4c1af68","event":"pull_request","changed_files":["Darling/Darling.Tests/DailyStitchLiveTests.cs","Darling/Darling.Tests/DailySummaryAndComposeStitchedLiveTests.cs","Darling/Darling.Tests/DailySummaryNotCarriedTests.cs","Darling/Darling.Tests/DailySummaryReadShapeTests.cs","Darling/Darling.Tests/DarlingMcpBlockingToolsTests.cs","Darling/Darling.Tests/EventTimeWindowReadsLiveTests.cs","Darling/Darling.Tests/EventTimeWindowSqlShapeTests.cs","Darling/Darling.Tests/ViewerBlockingDepthTests.cs","Darling/Darling.Tests/ViewerCalendarRetentionPortTests.cs","Darling/Darling.Tests/ViewerDailyHealthTests.cs","Darling/Darling.Tests/ViewerDeadlockSeverityStatsTests.cs","Darling/Darling.Tests/ViewerFileIoBlockingTests.cs","Darling/Darling.Tests/ViewerWave2Tests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertReadAdapter.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingBlockingReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingBlockingTrendReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingHealthReader.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummarySql.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Blocking.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.BlockingSlicers.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.BlockingStats.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.BlockingTrends.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.DailySummary.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Deadlock.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Blocking.cs","Lite.Tests/EventTimeWindowReadsTests.cs","Lite.Tests/PerformanceCalendarDataTests.cs","Lite/Services/LiteAlertReadAdapter.cs","Lite/Services/LocalDataService.Blocking.cs","Lite/Services/LocalDataService.BlockingStats.cs","Lite/Services/LocalDataService.DailySummary.cs"],"failed_classes":["Darling.Tests.PgLogBurstAndRotationLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-10-01","head_sha":"2c797f05fefa1a5d83d86339bd0e0119c9b1af94","event":"pull_request","changed_files":["Lite.Tests/StoredEventCopiesReaderTests.cs","Lite.Tests/StoredEventCopiesSweepTests.cs","Lite.Tests/StoredEventCopiesTests.cs","Lite.Tests/WatermarkReadFailureLogTests.cs","Lite/Analysis/AnomalyDetector.cs","Lite/Analysis/BaselineProvider.cs","Lite/Analysis/DrillDownCollector.Blocking.cs","Lite/Analysis/DuckDbFactCollector.Waits.cs","Lite/Database/DuckDbInitializer.cs","Lite/Database/StoredEventCopies.cs","Lite/Services/LocalDataService.Blocking.cs","Lite/Services/LocalDataService.BlockingStats.cs","Lite/Services/LocalDataService.DailySummary.cs","Lite/Services/LocalDataService.LongQueries.cs","Lite/Services/LocalDataService.Overview.cs","Lite/Services/LocalDataService.SystemEvents.cs","Lite/Services/RemoteCollectorService.cs","PerformanceMonitor.Collectors/BlockedProcessReportCollector.cs","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/LongQueryCompletionsCollector.cs","PerformanceMonitor.Collectors/SystemHealthEventsCollector.cs"],"failed_classes":["Darling.Tests.CommentFilterAdoptionTests","Darling.Tests.ConsumedTimestampFrameDisciplineTests","Darling.Tests.DarlingEventBaselineCoveredDaysTests"],"failed_jobs":["build","Darling PG tests (2)","Darling PG tests (0)"]} +{"date":"2026-10-01","head_sha":"2a0d4bd49c0519c2aa610bed15aa3fd367a2c73f","event":"pull_request","changed_files":["Darling/Darling.Tests/ConsumedTimestampFrameDisciplineTests.cs","Darling/Darling.Tests/DarlingEventBaselineCoveredDaysTests.cs","Lite.Tests/EventBaselineCoveredDaysTests.cs","Lite.Tests/StoredEventCopiesReaderTests.cs","Lite.Tests/StoredEventCopiesSweepTests.cs","Lite.Tests/StoredEventCopiesTests.cs","Lite.Tests/WatermarkReadFailureLogTests.cs","Lite/Analysis/AnomalyDetector.cs","Lite/Analysis/BaselineProvider.cs","Lite/Analysis/DrillDownCollector.Blocking.cs","Lite/Analysis/DuckDbFactCollector.Waits.cs","Lite/Database/DuckDbInitializer.cs","Lite/Database/StoredEventCopies.cs","Lite/Services/LocalDataService.Blocking.cs","Lite/Services/LocalDataService.BlockingStats.cs","Lite/Services/LocalDataService.DailySummary.cs","Lite/Services/LocalDataService.LongQueries.cs","Lite/Services/LocalDataService.Overview.cs","Lite/Services/LocalDataService.SystemEvents.cs","Lite/Services/RemoteCollectorService.cs","PerformanceMonitor.Collectors/BlockedProcessReportCollector.cs","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/LongQueryCompletionsCollector.cs","PerformanceMonitor.Collectors/SystemHealthEventsCollector.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-10-01","head_sha":"e20c88136040d5c7d1ee48bff2f88a4bf9cfd454","event":"pull_request","changed_files":["CHANGELOG.md","docs/changelog/3.9.md","tools/changelog/archive-census.txt"],"failed_classes":["Darling.Tests.MigrationUpgradeLadderLiveTests"],"failed_jobs":["build"]} +{"date":"2026-10-01","head_sha":"8b5919e8fd383a048709c86991c9630da78e3e83","event":"pull_request","changed_files":["Darling/Darling.Tests/AzureYoungBaselineWaitBarTests.cs","Darling/Darling.Tests/WaitProfileLedByTests.cs","Lite.Tests/AnomalyDetectorTests.cs"],"failed_classes":["Darling.Tests.AzureYoungBaselineWaitBarTests","Darling.Tests.WaitProfileLedByTests","PerformanceMonitorLite.Tests.AnomalyDetectorTests"],"failed_jobs":["Lite tests (2)","build","Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-10-01","head_sha":"7df4f3da8b4c2efea4ecafc87fbf1b3896d1a08d","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertHistoryDisplayNameLiveTests.cs","Darling/Darling.Tests/DarlingMcpAlertToolsTests.cs","Darling/Darling.Tests/HoldEarnedByDeliveryTests.cs","Darling/Darling.Tests/ViewerW2aTests.cs","Darling/Darling.Tests/ViewerWave3Tests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingAlertReader.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.AlertHistory.cs","Lite.Tests/AlertHistoryDisplayNameTests.cs","Lite/Controls/AlertsHistoryTab.xaml.cs","Lite/MainWindow.xaml.cs","Lite/Services/LocalDataService.AlertHistory.cs"],"failed_classes":["Darling.Tests.AlertNotebookResolutionReadLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-10-01","head_sha":"e19ece986ba502757d7370f764a0c365c7f178a8","event":"pull_request","changed_files":["Darling/Darling.Tests/DailySummaryAzureMasterScopeLiveTests.cs","Darling/Darling.Tests/DailySummaryAzureMasterScopeTests.cs","Darling/Darling.Tests/DarlingMcpFleetSweepToolsTests.cs","Darling/Darling.Tests/FleetOverviewAzureMasterScopeLiveTests.cs","Darling/Darling.Tests/McpServiceParameterDiSeatCensusTests.cs","Darling/Darling.Tests/SeparatelyMonitoredListNoteTests.cs","Darling/Darling.Tests/ViewerDailySummaryAzureMasterScopeLiveTests.cs","Darling/Darling.Tests/ViewerFleetAzureMasterScopeLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/FleetSweepEngine.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingHealthReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpBlockingTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFleetTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHealthTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpObjectStatsTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Storage/DailySummaryAzureMasterScope.cs","Darling/PerformanceMonitor.Darling.Storage/DailySummaryRangeCache.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.Locking.cs","Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.xaml","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.AzureMasterScope.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.DailySummary.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Fleet.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Overview.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Blocking.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml","PerformanceMonitor.Alerting/AzureMasterScope.cs"],"failed_classes":["Darling.Tests.ServerPageTabsTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-10-02","head_sha":"af31e98db0c6c7af4953d36a403403db0d4502e5","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeDataFloorLiveTests.cs","Darling/Darling.Tests/ComposeRollupDataFloorLiveTests.cs","Darling/Darling.Tests/DarlingComposeTests.cs","Darling/Darling.Tests/DarlingMcpDataToolsTests.cs","Darling/Darling.Tests/DataWindowFloorTests.cs","Darling/Darling.Tests/LiveStoreSessionTimeZone.cs","Darling/Darling.Tests/LiveStoreSessionTimeZoneTests.cs","Darling/Darling.Tests/QueryStoreTopWindowTests.cs","Darling/Darling.Tests/RawWindowFloorQuietStartTests.cs","Darling/Darling.Tests/ViewerDataStartBannerTests.cs","Darling/Darling.Tests/web-compose-notice-harness.mjs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeStoreAvailability.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreClutterTools.cs","Darling/PerformanceMonitor.Darling.Storage/DataWindowFloor.cs","Darling/PerformanceMonitor.Darling.Storage/RawWindowFloor.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.BlockingTrends.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QuerySnapshots.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.ActiveQueries.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Blocking.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml"],"failed_classes":["Darling.Tests.StoreSessionTimeZoneLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-10-02","head_sha":"0065d49b740238b1a24d256f65a307515deaeb48","event":"push","changed_files":[".github/FUNDING.yml"],"failed_classes":["Darling.Tests.PlanSync4512CancellationTests"],"failed_jobs":["Darling whole-tree guards"]} +{"date":"2026-10-02","head_sha":"6de10f4dcb1f6d48b0c6ca04986b785e2d295c00","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeDataFloorLiveTests.cs","Darling/Darling.Tests/ComposeRollupDataFloorLiveTests.cs","Darling/Darling.Tests/DarlingComposeTests.cs","Darling/Darling.Tests/DarlingMcpDataToolsTests.cs","Darling/Darling.Tests/DataWindowFloorTests.cs","Darling/Darling.Tests/LiveStoreSessionTimeZone.cs","Darling/Darling.Tests/LiveStoreSessionTimeZoneTests.cs","Darling/Darling.Tests/QueryStoreTopWindowTests.cs","Darling/Darling.Tests/RawWindowFloorQuietStartTests.cs","Darling/Darling.Tests/ViewerDataStartBannerTests.cs","Darling/Darling.Tests/web-compose-notice-harness.mjs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeStoreAvailability.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreClutterTools.cs","Darling/PerformanceMonitor.Darling.Storage/DataWindowFloor.cs","Darling/PerformanceMonitor.Darling.Storage/RawWindowFloor.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.BlockingTrends.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QuerySnapshots.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.ActiveQueries.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Blocking.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml","Lite.Tests/DataStartBannerTests.cs","Lite.Tests/QueryWindowTruncationTests.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Controls/ServerTab.Slicers.cs","Lite/Controls/ServerTab.xaml","Lite/Mcp/McpQueryTools.cs","Lite/Services/LocalDataService.QueryWindowFloor.cs"],"failed_classes":["Darling.Tests.StoreSessionTimeZoneLiveTests","PerformanceMonitorLite.Tests.DisplayZoneTextFrameTests"],"failed_jobs":["Lite tests (3)","Darling PG tests (1)"]} +{"date":"2026-10-02","head_sha":"a606e43521c458e3f9ce77109ae2877c7139801c","event":"pull_request","changed_files":["Darling/Darling.Tests/AlwaysOnXeSessionsTests.cs","Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs","Darling/Darling.Tests/DarlingCliUnusableStoreConnectionTests.cs","Darling/Darling.Tests/DropXeSessionsPerInstallTests.cs","Darling/Darling.Tests/DropXeSessionsVerbTests.cs","Darling/Darling.Tests/EmptyDatabaseListNoteLiveTests.cs","Darling/Darling.Tests/EmptyDatabaseListNoteTests.cs","Darling/Darling.Tests/InstallIdRungTests.cs","Darling/Darling.Tests/LegacyLongQueryRecordLiveTests.cs","Darling/Darling.Tests/LongQueryBannerSessionScopeTests.cs","Darling/Darling.Tests/LongQueryTraceInstanceGuardTests.cs","Darling/Darling.Tests/LongQueryTraceLifecycleTests.cs","Darling/Darling.Tests/LongQueryTraceReadOnlyIntentTests.cs","Darling/Darling.Tests/LongQueryTraceRestartTests.cs","Darling/Darling.Tests/MuteRuleServerIdRungTests.cs","Darling/Darling.Tests/PostgresEngineGateBehaviorTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/Darling.Tests/StoreInstallIdLiveTests.cs","Darling/Darling.Tests/StoreInstallIdRefusedClusterLiveTests.cs","Darling/Darling.Tests/StoreInstallIdSourcePinTests.cs","Darling/Darling.Tests/XeSessionMissingWarningTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlwaysOnXeSessions.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingLegacyLongQuerySession.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/DarlingXeSessionCleanup.cs","Darling/PerformanceMonitor.Darling.Service/DarlingXeSessions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpLongQueryTools.cs","Darling/PerformanceMonitor.Darling.Service/XeSessionMissingWarnings.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Storage/StoreInstallId.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml","Darling/README.md","Lite.Tests/AlwaysOnXeSessionsLiteTests.cs","Lite.Tests/DeadlockAndBlockedProcessEnsureLogLevelLiteTests.cs","Lite.Tests/EmptyDatabaseListNoteLiteTests.cs","Lite.Tests/InstallIdStoreTests.cs","Lite.Tests/InstallIdTests.cs","Lite.Tests/LongQueryBannerSessionScopeTests.cs","Lite.Tests/LongQueryCompletionsCollectorDefinitionTests.cs","Lite.Tests/LongQueryTraceLifecycleLiteTests.cs","Lite.Tests/LongQueryTraceReadOnlyIntentLiteTests.cs","Lite.Tests/SettingsImportTests.cs","Lite.Tests/SwallowedItemFailureTests.cs","Lite/Controls/ServerTab.xaml","Lite/MainWindow.xaml.cs","Lite/Mcp/McpLongQueryTools.cs","Lite/Services/InstallIdStore.cs","Lite/Services/RemoteCollectorService.AlwaysOnXeSessions.cs","Lite/Services/RemoteCollectorService.BlockedProcessReport.cs","Lite/Services/RemoteCollectorService.Deadlocks.cs","Lite/Services/RemoteCollectorService.DefinitionRunner.cs","Lite/Services/RemoteCollectorService.LegacyLongQuerySession.cs","Lite/Services/RemoteCollectorService.LongQueryCompletions.cs","Lite/Services/RemoteCollectorService.cs","Lite/Services/ScheduleManager.cs","Lite/Services/SettingsImport.cs","PerformanceMonitor.Collectors/AlwaysOnXeSessions.cs","PerformanceMonitor.Collectors/BlockedProcessReportCollector.cs","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/CollectorScheduleDefaults.cs","PerformanceMonitor.Collectors/DeadlocksCollector.cs","PerformanceMonitor.Collectors/EmptyDatabaseListNote.cs","PerformanceMonitor.Collectors/ICollectorDefinition.cs","PerformanceMonitor.Collectors/InstallId.cs","PerformanceMonitor.Collectors/LegacyLongQuerySession.cs","PerformanceMonitor.Collectors/LongQueryCompletionsCollector.cs","PerformanceMonitor.Collectors/LongQueryTraceDatabases.cs","PerformanceMonitor.Collectors/LongQueryTraceStep.cs","PerformanceMonitor.Collectors/ServerEpoch.cs"],"failed_classes":["Darling.Tests.CollectorStateContractTests","Darling.Tests.DarlingCollectorToggleVerbTests","Lite.Tests.XeTimeFilterSweepTests","PerformanceMonitorLite.Tests.EntraDeviceCodeTests","PerformanceMonitorLite.Tests.StoredEventCopiesTests"],"failed_jobs":["Lite tests (0)","Lite tests (1)","Darling PG tests (2)","build","Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-10-02","head_sha":"ce33caec1b33983f52a1a181d9da4d1d42375a8d","event":"pull_request","changed_files":["Darling/Darling.Tests/FleetGateSignalTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/SelfAlertDeliveryStampTests.cs","Darling/Darling.Tests/StoreMaintenanceTickLaunchTests.cs","Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs","Darling/Darling.Tests/TimescaleAvailabilityReprobeTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs"],"failed_classes":["Darling.Tests.PgLogEventsCsvlogLiveTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-10-02","head_sha":"a9b554fa704f0855fde9bb44da5ad01fadcb8089","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreBackgroundIndexesLiveTests.cs","Darling/Darling.Tests/QueryStoreBackgroundIndexesTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideBrinIndexLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideBrinIndexTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideFirstExecFloorTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreBackgroundIndexes.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalWide.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalWideBrinIndex.cs"],"failed_classes":["Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["Darling PG tests (1)","build"]} +{"date":"2026-10-03","head_sha":"b9411680b270adff87459ed8b6dcc728d1a25b56","event":"pull_request","changed_files":["Darling/Darling.Tests/BoundedCompressionDdlTests.cs","Darling/Darling.Tests/FleetGateSignalTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/SelfAlertDeliveryStampTests.cs","Darling/Darling.Tests/StoreMaintenanceTickLaunchTests.cs","Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs","Darling/Darling.Tests/TimescaleAvailabilityReprobeTests.cs","Darling/Darling.Tests/TuningHourlyCreateLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/PgTableTuning.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-10-03","head_sha":"de3cabd7eb86bb3e50d839d76a7f093b2c2bb8d4","event":"pull_request","changed_files":["Darling/Darling.Tests/BoundedCompressionDdlTests.cs","Darling/Darling.Tests/BoundedDdlBusyStreakTests.cs","Darling/Darling.Tests/BoundedDdlSingleCopyTests.cs","Darling/Darling.Tests/FleetGateSignalTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/Darling.Tests/RetentionReevaluationTests.cs","Darling/Darling.Tests/SelfAlertDeliveryStampTests.cs","Darling/Darling.Tests/StoreMaintenanceTickLaunchTests.cs","Darling/Darling.Tests/StoreObjectConvergenceTests.cs","Darling/Darling.Tests/StoreObjectHourlyStepSelectionTests.cs","Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs","Darling/Darling.Tests/TimescaleAvailabilityReprobeTests.cs","Darling/Darling.Tests/TimescaleSupportTests.cs","Darling/Darling.Tests/TuningHourlyCreateLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/BoundedDdlBusyStreaks.cs","Darling/PerformanceMonitor.Darling.Storage/PgTableTuning.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.CollectionLogSegmentByLiveTests","Darling.Tests.CompressionEnableGuardTests"],"failed_jobs":["build","Darling PG tests (0)","Darling PG tests (1)"]} +{"date":"2026-10-03","head_sha":"bc4bdca4a7f784517222385e33adce0381d17daa","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionHealthKeyedLookupParityTests.cs","Darling/Darling.Tests/CollectionHealthOracleSql.cs","Darling/Darling.Tests/CollectorHealthClassifierTests.cs","Darling/Darling.Tests/DarlingEmptyEnumerationNoteTests.cs","Darling/Darling.Tests/ProductiveZeroBandingTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs"],"failed_classes":["Darling.Tests.LiveCleanupConversionRatchetTests","Darling.Tests.LivePlanShapeIsolationCensusTests","Darling.Tests.ViewerCollectionHealthSqlTests"],"failed_jobs":["Darling PG tests (2)","Darling PG tests (0)","build"]} +{"date":"2026-10-03","head_sha":"fb8e0ad86888b2f694ff6b9d61d451280d59b02a","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeQueryStoreRawDedupeFilterLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeCompiler.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.QueryStoreSliceTieBreakSourceTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-10-03","head_sha":"15173c8b89cf75b38c1353c44c89f950d12c1b43","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeQueryStoreRawDedupeFilterLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeCompiler.cs"],"failed_classes":["Darling.Tests.QueryStoreSliceTieBreakSourceTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-10-03","head_sha":"2f652475b8a0f89de896dcce424e2f5fe4a83911","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeQueryStoreRawDedupeFilterLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeCompiler.cs"],"failed_classes":["Darling.Tests.QueryStoreSliceTieBreakSourceTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-10-03","head_sha":"b47aad7fc80dd96ddeee3750ecbc73934832cd1c","event":"pull_request","changed_files":["Lite.Tests/DataStartBannerSurfaceTests.cs","Lite.Tests/DataStartBannerTests.cs","Lite/Controls/ServerTab.ConfigChanges.cs","Lite/Controls/ServerTab.LongQueries.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Controls/ServerTab.SystemEvents.cs","Lite/Controls/ServerTab.xaml","Lite/Services/LocalDataService.QueryWindowFloor.cs"],"failed_classes":["PerformanceMonitorLite.Tests.StoredEventCopiesSweepTests"],"failed_jobs":["Lite tests (2)"]} +{"date":"2026-10-03","head_sha":"2aff8820c433392d15303e52a8bb6b218087fdd9","event":"pull_request","changed_files":["Lite.Tests/DataStartBannerBlockingEventsTests.cs","Lite.Tests/QueryWindowTruncationTests.cs","Lite/Controls/ServerTab.DrillDown.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Controls/ServerTab.Slicers.cs","Lite/Controls/ServerTab.xaml","Lite/Services/LocalDataService.QueryWindowFloor.cs"],"failed_classes":["PerformanceMonitorLite.Tests.StoredEventCopiesSweepTests"],"failed_jobs":["Lite tests (2)"]} +{"date":"2026-10-03","head_sha":"ad4b71823a3cc4337e64b0900213ece9106b8af2","event":"pull_request","changed_files":["Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs","Darling/Darling.Tests/DropXeSessionsVerbTests.cs","Darling/Darling.Tests/InstallIdRungTests.cs","Darling/Darling.Tests/InstallIdTableOidRungTests.cs","Darling/Darling.Tests/StoreInstallIdBindingRuleTests.cs","Darling/Darling.Tests/StoreInstallIdLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingXeSessionCleanup.cs","Darling/PerformanceMonitor.Darling.Service/DarlingXeSessions.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Storage/StoreInstallId.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md","Lite.Tests/AlwaysOnXeSessionsLiteTests.cs","Lite.Tests/InstallIdStoreTests.cs","Lite.Tests/LongQueryTraceLifecycleLiteTests.cs","Lite/Services/InstallIdStore.cs","Lite/Services/RemoteCollectorService.AlwaysOnXeSessions.cs","Lite/Services/RemoteCollectorService.LongQueryCompletions.cs","Lite/Services/RemoteCollectorService.cs","PerformanceMonitor.Collectors/AlwaysOnXeSessions.cs","README.md"],"failed_classes":["PerformanceMonitor.Darling.Tests.RetiredBaselineAggregateLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-10-03","head_sha":"964bf56926c63b31c3c73f3b81cb925ae9a152de","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreWideFleetEligibilityLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreIntervalWide.cs"],"failed_classes":["Darling.Tests.FleetIdentifierScrubTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-10-03","head_sha":"761b9398ae180a473c2dacd6abb70e610fc62842","event":"pull_request","changed_files":["Lite.Tests/DataStartBannerQueriesTabTests.cs","Lite.Tests/DataStartBannerTests.cs","Lite/Controls/ServerTab.Charts.cs","Lite/Controls/ServerTab.QueriesDataStart.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Controls/ServerTab.xaml","Lite/Services/LocalDataService.QueryStats.cs","Lite/Services/LocalDataService.QueryWindowFloor.cs"],"failed_classes":["Darling.Tests.CommentFilterAdoptionTests"],"failed_jobs":["build"]} +{"date":"2026-10-03","head_sha":"10f7e41347b062455016cb85d356e8a43ec20a00","event":"pull_request","changed_files":["Darling/Darling.Tests/CommentFilterAdoptionTests.cs","Lite.Tests/DataStartBannerQueriesTabTests.cs","Lite.Tests/DataStartBannerTests.cs","Lite/Controls/ServerTab.Charts.cs","Lite/Controls/ServerTab.QueriesDataStart.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Controls/ServerTab.xaml","Lite/Services/LocalDataService.QueryStats.cs","Lite/Services/LocalDataService.QueryWindowFloor.cs"],"failed_classes":["Lite.Tests.CrossAppGuardCiGateTests"],"failed_jobs":["Lite tests (2)"]} +{"date":"2026-10-03","head_sha":"8f68d2dd8295393b26f76b3baa822b5f30c86ec8","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorFaultStackLogTests.cs","Darling/Darling.Tests/RdsLogStateConcurrencyTests.cs","Darling/PerformanceMonitor.Darling.Service/CollectorFaultStackLog.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsCsvlogCarry.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsDeadlockCarry.cs","Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogSource.cs"],"failed_classes":["Darling.Tests.PostgresEngineGateBehaviorTests"],"failed_jobs":["Darling PG tests (0)"]} +{"date":"2026-10-03","head_sha":"3bb3ae8fe975ee0be69258c3ba40223c044896aa","event":"pull_request","changed_files":["Darling/Darling.Tests/DataWindowFloorTests.cs","Darling/Darling.Tests/ViewerDataStartBannerTests.cs","Darling/Darling.Tests/ViewerEventDataStartLiveTests.cs","Darling/Darling.Tests/ViewerEventDataStartTests.cs","Darling/Darling.Tests/ViewerEventGridsDataStartLiveTests.cs","Darling/Darling.Tests/ViewerLongQueriesDataStartTests.cs","Darling/Darling.Tests/ViewerPlanCorrectionsDataStartTests.cs","Darling/Darling.Tests/ViewerQueryGridsDataStartLiveTests.cs","Darling/Darling.Tests/ViewerQueryHeatmapDataStartTests.cs","Darling/Darling.Tests/ViewerQueryStoreRegressionsDataStartTests.cs","Darling/Darling.Tests/ViewerShortWindowDataStartTests.cs","Darling/Darling.Tests/ViewerSystemEventsDataStartTests.cs","Darling/PerformanceMonitor.Darling.Storage/DataWindowFloor.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Blocking.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Deadlock.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.LongQueries.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.PlanCorrection.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryHeatmap.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStoreRegressions.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.SystemEvents.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerEventDataStart.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Blocking.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.EventDataStart.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.LongQueries.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Queries.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.QueryHeatmap.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.SystemEvents.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml"],"failed_classes":["Darling.Tests.DailySummaryNotCarriedLiveTests","Darling.Tests.StoreMetricsLatestSkipScanLiveTests","Darling.Tests.TrendPayloadBudgetLiveTests","Darling.Tests.ViewerCpuTempDbLivePostgresTests","Darling.Tests.WebExceptionTextCensusTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-10-03","head_sha":"a55a242505154020357d9618af7e150320eab9e0","event":"pull_request","changed_files":["Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs","Darling/Darling.Tests/CollectorRunTimeRungTests.cs","Darling/Darling.Tests/CollectorRunTimeTests.cs","Darling/Darling.Tests/DarlingSweepSchedulingTests.cs","Darling/Darling.Tests/InstallIdTableOidRungTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","PerformanceMonitor.Collectors/CollectorCadence.cs","PerformanceMonitor.Collectors/CollectorRunTime.cs","PerformanceMonitor.Collectors/CollectorScheduleDefaults.cs"],"failed_classes":["Darling.Tests.StartupCommandTimeoutTests"],"failed_jobs":["Darling PG tests (0)","build"]} +{"date":"2026-10-03","head_sha":"261356ae825662fb7dd004f30655b54eee72cb29","event":"pull_request","changed_files":["Lite.Tests/DataStartBannerSurfaceTests.cs","Lite.Tests/DataStartBannerTests.cs","Lite.Tests/QueryWindowTruncationTests.cs","Lite.Tests/StoredEventCopiesSweepTests.cs","Lite/Controls/ServerTab.ConfigChanges.cs","Lite/Controls/ServerTab.LongQueries.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Controls/ServerTab.SystemEvents.cs","Lite/Controls/ServerTab.xaml","Lite/Services/LocalDataService.CollectionHealth.cs","Lite/Services/LocalDataService.LongQueries.cs","Lite/Services/LocalDataService.QueryWindowFloor.cs"],"failed_classes":["Darling.Tests.McpPayloadContractCensusTests"],"failed_jobs":["build"]} +{"date":"2026-10-03","head_sha":"46cc97a10aca73111a9a277cf3637b0ce12445e4","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpFinOpsToolsTests.cs","Darling/Darling.Tests/FinOpsHighImpactReaderLiveTests.cs","Darling/Darling.Tests/FinOpsWebReadParityLiveTests.cs","Darling/Darling.Tests/McpToolGuideHeads.FinOps.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpFinOpsTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/ViewerFinOpsHighImpactDelegatesTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFinOpsTools.HighImpact.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFinOpsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Storage/FinOps/DarlingFinOpsHighImpactReader.cs","Darling/PerformanceMonitor.Darling.Storage/FinOps/FinOpsScoring.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Workload.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.cs","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs"],"failed_classes":["Darling.Tests.CollectorEngineCapabilityTests","Darling.Tests.EngineCapabilityReadWiringTests","Lite.Tests.CrossAppMcpToolInventoryPinTests"],"failed_jobs":["build","Darling PG tests (1)","Lite tests (1)"]} +{"date":"2026-10-03","head_sha":"8d786686324f6cf74484722c172f5a10d3aee512","event":"pull_request","changed_files":["Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Lite.Tests/DataStartBannerProbeFailureTests.cs","Lite.Tests/DataStartBannerQueriesTabTests.cs","Lite.Tests/DataStartBannerSurfaceTests.cs","Lite.Tests/DataStartBannerTests.cs","Lite.Tests/DisplayZoneTextFrameTests.cs","Lite.Tests/QueryWindowTruncationTests.cs","Lite.Tests/StoredEventCopiesSweepTests.cs","Lite/Controls/ServerTab.Charts.cs","Lite/Controls/ServerTab.ConfigChanges.cs","Lite/Controls/ServerTab.LongQueries.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Controls/ServerTab.SystemEvents.cs","Lite/Controls/ServerTab.xaml","Lite/Services/LocalDataService.CollectionHealth.cs","Lite/Services/LocalDataService.LongQueries.cs","Lite/Services/LocalDataService.QueryWindowFloor.cs","PerformanceMonitor.Ui/ChartHoverHelper.cs"],"failed_classes":["PerformanceMonitor.Darling.Tests.RetiredBaselineAggregateLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-10-03","head_sha":"f6717e306185c1653560e54afe916463320b6637","event":"pull_request","changed_files":["Darling/Darling.Tests/AsOfWindowAnchorTests.cs","Darling/Darling.Tests/DarlingMcpFinOpsToolsTests.cs","Darling/Darling.Tests/FinOpsHighImpactReaderLiveTests.cs","Darling/Darling.Tests/FinOpsWebReadParityLiveTests.cs","Darling/Darling.Tests/McpToolGuideHeads.FinOps.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpFinOpsTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/StorageCommandTimeoutTests.cs","Darling/Darling.Tests/ViewerFinOpsHighImpactDelegatesTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFinOpsTools.HighImpact.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFinOpsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Storage/FinOps/DarlingFinOpsHighImpactReader.cs","Darling/PerformanceMonitor.Darling.Storage/FinOps/FinOpsScoring.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Workload.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.cs","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs"],"failed_classes":["Darling.Tests.CollectorEngineCapabilityTests","Darling.Tests.EngineCapabilityReadWiringTests","Lite.Tests.CrossAppMcpToolInventoryPinTests"],"failed_jobs":["build","Darling PG tests (1)","Lite tests (1)"]} +{"date":"2026-10-03","head_sha":"3bde72bf30bf5b9f285d4aba415851e2d84d2544","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpFinOpsToolsTests.cs","Darling/Darling.Tests/EngineCapabilityMissTests.cs","Darling/Darling.Tests/FinOpsUtilizationGoldenLiveTests.cs","Darling/Darling.Tests/FinOpsUtilizationViewLiveTests.cs","Darling/Darling.Tests/FinOpsWebReadParityLiveTests.cs","Darling/Darling.Tests/McpToolGuideHeads.FinOps.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpFinOpsTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFinOpsTools.Utilization.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFinOpsTools.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-10-03","head_sha":"a9f62df282a013db88adf22004f5077c83b5af4f","event":"pull_request","changed_files":["Darling/Darling.Tests/JobHistoryCountTextTests.cs","Darling/Darling.Tests/LiteJobHistoryLoadingCapTests.cs","Darling/Darling.Tests/ViewerJobHistoryDataStartLiveTests.cs","Darling/Darling.Tests/ViewerJobHistoryDataStartTests.cs","Darling/PerformanceMonitor.Darling.Viewer/JobHistoryTab.xaml","Darling/PerformanceMonitor.Darling.Viewer/JobHistoryTab.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.JobHistory.cs","Lite.Tests/DataStartBannerSurfaceTests.cs","Lite.Tests/JobHistoryDataStartTests.cs","Lite.Tests/JobHistoryServerClockWindowTests.cs","Lite/Controls/JobHistoryTab.xaml","Lite/Controls/JobHistoryTab.xaml.cs","Lite/MainWindow.xaml.cs","Lite/Services/LocalDataService.JobHistory.cs","Lite/Services/LocalDataService.QueryWindowFloor.cs","PerformanceMonitor.Common/JobHistoryCap.cs"],"failed_classes":["Darling.Tests.ViewerCommandTimeoutTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-10-03","head_sha":"99830570d14e086815f23a2fd17efb81215fafcf","event":"pull_request","changed_files":["Darling/Darling.Tests/ViewerBlockingChartsDataStartLiveTests.cs","Darling/Darling.Tests/ViewerBlockingChartsDataStartPinTests.cs","Darling/Darling.Tests/ViewerBlockingChartsDataStartTests.cs","Darling/Darling.Tests/ViewerEventDataStartTests.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerBlockingChartsDataStart.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.BlockingTrends.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Blocking.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.xaml"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-10-03","head_sha":"ffed3540553268d36da0b864d72917c91ed0e382","event":"pull_request","changed_files":["Darling/Darling.Tests/JobHistoryCountTextTests.cs","Darling/Darling.Tests/LiteJobHistoryLoadingCapTests.cs","Darling/Darling.Tests/ViewerJobHistoryDataStartLiveTests.cs","Darling/Darling.Tests/ViewerJobHistoryDataStartTests.cs","Darling/PerformanceMonitor.Darling.Viewer/JobHistoryTab.xaml","Darling/PerformanceMonitor.Darling.Viewer/JobHistoryTab.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.JobHistory.cs","Lite.Tests/DataStartBannerSurfaceTests.cs","Lite.Tests/JobHistoryDataStartTests.cs","Lite.Tests/JobHistoryServerClockWindowTests.cs","Lite/Controls/JobHistoryTab.xaml","Lite/Controls/JobHistoryTab.xaml.cs","Lite/MainWindow.xaml.cs","Lite/Services/LocalDataService.JobHistory.cs","Lite/Services/LocalDataService.QueryWindowFloor.cs","PerformanceMonitor.Common/JobHistoryCap.cs"],"failed_classes":["Darling.Tests.ViewerCommandTimeoutTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-10-04","head_sha":"c0866d29a2658fa00eaa17e159cf86bf086289a5","event":"push","changed_files":["Darling/Darling.Tests/FinOpsIndexAnalysisLatestTiebreakTests.cs","Darling/Darling.Tests/Fixtures/FinOpsIndexAnalysis/golden.json","Darling/PerformanceMonitor.Darling.Storage/FinOps/DarlingFinOpsIndexAnalysisReader.cs"],"failed_classes":["Darling.Tests.QueryStoreLegacyRowIndexLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-10-04","head_sha":"475665904568c142e4bbddf9825b9d8d3c36b41e","event":"pull_request","changed_files":["Darling/Darling.Tests/FinOpsTabOptimizationPageTests.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/optimization.js"],"failed_classes":["Darling.Tests.QueryStoreLegacyRowIndexLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-10-04","head_sha":"9f612f13de4ef902619e7b7bdb6ad94b21c56b22","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeParameterCoverageTests.cs","Darling/Darling.Tests/ComposeSourceRouterTests.cs","Darling/Darling.Tests/DarlingComposeTests.cs","Darling/Darling.Tests/IntervalRollupCountGuardLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeCompiler.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeSourceRouter.cs","Darling/PerformanceMonitor.Darling.Service/Compose/ComposeStoreAvailability.cs","Darling/PerformanceMonitor.Darling.Storage/IntervalRollupCountGuard.cs"],"failed_classes":["Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-10-04","head_sha":"c38fa98b46741b3b067206fd4301fe5628184b4d","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedRolesTests.cs","Darling/Darling.Tests/DarlingMcpServerTagToolsTests.cs","Darling/Darling.Tests/DarlingServerTagsTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/McpReadToolBudgetLiveTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpServerTagTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/ServerTagCoverageTests.cs","Darling/Darling.Tests/ServerTagDeleteWarningTests.cs","Darling/Darling.Tests/ServerTagMcpRoleLiveTests.cs","Darling/Darling.Tests/ServerTagRulesTests.cs","Darling/Darling.Tests/ServerTagStoreLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/CustomAlertRuleStore.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerTagTools.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/ServerTagCoverage.cs","Darling/PerformanceMonitor.Darling.Storage/ServerTagRules.cs","Darling/PerformanceMonitor.Darling.Storage/ServerTagStore.cs","Darling/PerformanceMonitor.Darling.Viewer/MainWindow.Tags.cs","Darling/PerformanceMonitor.Darling.Viewer/ManageTagsWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.ServerTags.cs","Darling/tools/provision-roles.sql","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","PerformanceMonitor.Common/TagColours.cs","PerformanceMonitor.Ui/TagColorPalette.cs","README.md","llms.txt"],"failed_classes":["Darling.Tests.DarlingMcpInstructionsBudgetTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (1)","build"]} +{"date":"2026-10-04","head_sha":"80c56c3d552d98efbdcc91ac7007e2eed248ce76","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpJobHistoryToolTests.cs","Darling/Darling.Tests/DarlingMcpJobToolsTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpJobTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/packages.lock.json","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpJobTools.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingJobHistoryReader.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingServerClocksReader.cs","Darling/PerformanceMonitor.Darling.Storage/PerformanceMonitor.Darling.Storage.csproj","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.JobHistory.cs","Darling/PerformanceMonitor.Darling.Viewer/packages.lock.json","README.md","llms.txt"],"failed_classes":["Lite.Tests.CrossAppMcpToolInventoryPinTests"],"failed_jobs":["Lite tests (1)"]} +{"date":"2026-10-04","head_sha":"e4e3e8a4076105015706db8446ac55a16ddcdbd6","event":"pull_request","changed_files":["Darling/Darling.Tests/ActiveQueriesWidenedColumnsTests.cs","Darling/Darling.Tests/DarlingMcpActiveQueriesBudgetLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpSessionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingSessionReader.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js"],"failed_classes":["Darling.Tests.FinOpsWindowPickerBehaviourTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (2)","build"]} +{"date":"2026-10-04","head_sha":"ad44369b11a8a4018c7349da95973e65c36accc3","event":"pull_request","changed_files":["Darling/Darling.Tests/DatabaseConfigWebColumnsTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpConfigTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js"],"failed_classes":["Darling.Tests.FinOpsWindowPickerBehaviourTests"],"failed_jobs":["Darling PG tests (0)","build","Darling PG tests (2)"]} +{"date":"2026-10-04","head_sha":"5a143e3f860ccb669e759a7dc99056330a8b60dd","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertHistoryRangePageTests.cs","Darling/Darling.Tests/alert-history-range-harness.mjs","Darling/PerformanceMonitor.Darling.Service/wwwroot/css/theme.css","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/alerts.js"],"failed_classes":["Darling.Tests.FinOpsWindowPickerBehaviourTests"],"failed_jobs":["build","Darling PG tests (0)","Darling PG tests (2)"]} +{"date":"2026-10-04","head_sha":"d8246671f1bc9913be7bfec5bc30317921d03335","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpJobHistoryToolTests.cs","Darling/Darling.Tests/DarlingMcpJobToolsTests.cs","Darling/Darling.Tests/JobHistorySqlTextPinTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpJobTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/packages.lock.json","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpJobTools.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingJobHistoryReader.cs","Darling/PerformanceMonitor.Darling.Storage/DarlingServerClocksReader.cs","Darling/PerformanceMonitor.Darling.Storage/PerformanceMonitor.Darling.Storage.csproj","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.JobHistory.cs","Darling/PerformanceMonitor.Darling.Viewer/packages.lock.json","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","README.md","llms.txt"],"failed_classes":["Darling.Tests.FinOpsWindowPickerBehaviourTests"],"failed_jobs":["build","Darling PG tests (2)","Darling PG tests (0)"]} +{"date":"2026-10-04","head_sha":"c20797c24a525d965b181a3f8d8ae47a236c97aa","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingWebAssetsTests.cs","Darling/Darling.Tests/GridToolsBehaviourTests.cs","Darling/Darling.Tests/finops-window-picker-harness.mjs","Darling/Darling.Tests/web-grid-tools-harness.mjs","Darling/Darling.Tests/web-kept-history-harness.mjs","Darling/Darling.Tests/web-perfmon-harness.mjs","Darling/PerformanceMonitor.Darling.Service/wwwroot/css/theme.css","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/grid-tools.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/alerts.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/read-fields.js"],"failed_classes":["Darling.Tests.PgDeadlockNormalizationTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (1)"]} +{"date":"2026-10-04","head_sha":"d40edb18e786a2f1960c5377d6540eba170c3053","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionCaveatsWebTests.cs","Darling/Darling.Tests/web-collection-caveats-harness.mjs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingCollectionCaveatReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js"],"failed_classes":["Darling.Tests.PgIndexBloatGridReachTests","Darling.Tests.WebMemoryPressureChartBehaviourTests"],"failed_jobs":["Darling PG tests (0)","build","Darling PG tests (1)"]} +{"date":"2026-10-04","head_sha":"863017a89e7e8d1e15d1c7529fe535158e005dc7","event":"pull_request","changed_files":["Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs","Darling/Darling.Tests/CollectorRunTimeRungTests.cs","Darling/Darling.Tests/QueryStoreTopDailyBuilderLiveTests.cs","Darling/Darling.Tests/QueryStoreTopDailyRungTests.cs","Darling/Darling.Tests/StoreSelfMetricsTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/QueryStoreTopDaily.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.QueryStoreTopDailyRungTests"],"failed_jobs":["build","Darling PG tests (0)","Darling PG tests (2)"]} +{"date":"2026-10-04","head_sha":"7abe9e9507ecafb7d8c3b38d73b751016401a9bf","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedRolesTests.cs","Darling/Darling.Tests/DarlingMcpServerTagToolsTests.cs","Darling/Darling.Tests/DarlingServerTagsTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/McpReadToolBudgetLiveTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpServerTagTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/ServerRemovalClearsTagsLiveTests.cs","Darling/Darling.Tests/ServerTagCoverageTests.cs","Darling/Darling.Tests/ServerTagDeleteWarningTests.cs","Darling/Darling.Tests/ServerTagMcpRoleLiveTests.cs","Darling/Darling.Tests/ServerTagRulesTests.cs","Darling/Darling.Tests/ServerTagStoreLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/CustomAlertRuleStore.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerAdminTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerTagTools.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/ServerTagCoverage.cs","Darling/PerformanceMonitor.Darling.Storage/ServerTagRules.cs","Darling/PerformanceMonitor.Darling.Storage/ServerTagStore.cs","Darling/PerformanceMonitor.Darling.Viewer/MainWindow.Tags.cs","Darling/PerformanceMonitor.Darling.Viewer/ManageTagsWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.MonitoredServers.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.ServerTags.cs","Darling/tools/provision-roles.sql","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","PerformanceMonitor.Common/TagColours.cs","PerformanceMonitor.Ui/TagColorPalette.cs","README.md","llms.txt"],"failed_classes":["Darling.Tests.LivePlanShapeIsolationCensusTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-10-04","head_sha":"bf388bec5970751d91fcd835723b546464c50241","event":"pull_request","changed_files":["Darling/Darling.Tests/WebWaitStatsMultiSelectBehaviourTests.cs","Darling/Darling.Tests/web-kept-history-harness.mjs","Darling/Darling.Tests/web-memory-pressure-harness.mjs","Darling/Darling.Tests/web-perfmon-harness.mjs","Darling/Darling.Tests/web-waits-harness.mjs","Darling/PerformanceMonitor.Darling.Service/wwwroot/css/app.css","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/multi-picker.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.QueryStoreTopDailyRungTests","PerformanceMonitorLite.Tests.BaselineDiscontinuityRenderCensusTests"],"failed_jobs":["Darling PG tests (0)","Darling PG tests (2)","Lite tests (1)","build"]} +{"date":"2026-10-04","head_sha":"3e1f8d1ef99ecc64540d515182acd371294c46ce","event":"pull_request","changed_files":["Darling/Darling.Tests/WebPageRefreshIntervalBehaviourTests.cs","Darling/Darling.Tests/web-page-refresh-harness.mjs","Darling/Darling.Tests/web-scheduler-harness.mjs","Darling/PerformanceMonitor.Darling.Service/wwwroot/css/app.css","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/app.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/refresh-control.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/refresh-policy.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js"],"failed_classes":["Darling.Tests.LiveCleanupConversionRatchetTests"],"failed_jobs":["Darling PG tests (2)","build","Darling whole-tree guards"]} +{"date":"2026-10-05","head_sha":"87e78aa474eb4b31f2d5a4687ec5ddcb9224cb6b","event":"pull_request","changed_files":["Darling/Darling.Tests/WebPerfmonMultiSelectBehaviourTests.cs","Darling/Darling.Tests/WebPerfmonPerSecondBehaviourTests.cs","Darling/Darling.Tests/web-perfmon-multi-harness.mjs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/multi-picker.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Lite.Tests/BaselineDiscontinuityRenderCensusTests.cs"],"failed_classes":["Darling.Tests.ViewerEventDataStartTests","PerformanceMonitorLite.Tests.DataStartBannerProbeFailureTests"],"failed_jobs":["build","Lite tests (0)","Darling PG tests (1)"]} +{"date":"2026-10-05","head_sha":"68f25798ce23de98af66a942062faa1f3b529e39","event":"pull_request","changed_files":["Darling/Darling.Tests/ConsumedTimestampFrameDisciplineTests.cs","Darling/Darling.Tests/DarlingCustomViewsTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpDataTools.txt","Darling/Darling.Tests/TopQueriesDetailLiveTests.cs","Darling/Darling.Tests/TopQueriesHourlyRoutingLiveTests.cs","Darling/Darling.Tests/WebServerPageRangeTests.cs","Darling/Darling.Tests/WebTopQueriesColumnsTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js"],"failed_classes":["Darling.Tests.SeparatelyMonitoredListNoteTests"],"failed_jobs":["Darling PG tests (1)","build"]} +{"date":"2026-10-05","head_sha":"5fe6eddad2fda2e493b00813b349abe8f0832c49","event":"pull_request","changed_files":["Darling/Darling.Tests/PayloadDigestPassThroughLiveTests.cs","Darling/PerformanceMonitor.Darling.Storage/PayloadDimensionBatch.cs","Darling/PerformanceMonitor.Darling.Storage/PayloadDimensionWriter.cs","Darling/PerformanceMonitor.Darling.Storage/PayloadDimensions.cs","Darling/PerformanceMonitor.Darling.Storage/PgCollectorRowWriter.cs"],"failed_classes":["Darling.Tests.PayloadDigestPassThroughLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-10-05","head_sha":"183d90d1687521c78299aee6146f1b85d1962e7c","event":"pull_request","changed_files":["Darling/Darling.Tests/ProcedureStatsPlanFetchTests.cs","Darling/Darling.Tests/ProcedureStatsSqlGoldenTests.cs","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/ProcedureStatsCollector.cs"],"failed_classes":["Darling.Tests.DarlingMcpServerAdminToolsSurfaceTests","Darling.Tests.PayloadDigestPassThroughLiveTests","Darling.Tests.ServerAddViewerRoleLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-10-05","head_sha":"5b70a510603dce2afddd475de67d2aa9ac6d11f6","event":"pull_request","changed_files":["Darling/Darling.Tests/PayloadDigestPassThroughLiveTests.cs","Darling/PerformanceMonitor.Darling.Storage/PayloadDimensionBatch.cs","Darling/PerformanceMonitor.Darling.Storage/PayloadDimensionWriter.cs","Darling/PerformanceMonitor.Darling.Storage/PayloadDimensions.cs","Darling/PerformanceMonitor.Darling.Storage/PgCollectorRowWriter.cs"],"failed_classes":["Darling.Tests.PayloadDigestPassThroughLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-10-05","head_sha":"956e5deff20766d7de24eca089e94f79c4bf64ab","event":"pull_request","changed_files":["Darling/Darling.Tests/ProcedureStatsPlanFetchTests.cs","Darling/Darling.Tests/ProcedureStatsSqlGoldenTests.cs","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/ProcedureStatsCollector.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-10-05","head_sha":"1acb3d71040f5820312bdbe66b21cd5bc829a504","event":"push","changed_files":["Darling/Darling.Tests/ProcedureStatsPlanFetchTests.cs","Darling/Darling.Tests/ProcedureStatsSqlGoldenTests.cs","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/ProcedureStatsCollector.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-10-05","head_sha":"c1063c5c89bd9deac5787722d46d649a87325ea5","event":"pull_request","changed_files":["Darling/Darling.Tests/PayloadDigestPassThroughLiveTests.cs","Darling/PerformanceMonitor.Darling.Storage/PayloadDimensionBatch.cs","Darling/PerformanceMonitor.Darling.Storage/PayloadDimensionWriter.cs","Darling/PerformanceMonitor.Darling.Storage/PayloadDimensions.cs","Darling/PerformanceMonitor.Darling.Storage/PgCollectorRowWriter.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.PayloadDigestPassThroughLiveTests"],"failed_jobs":["Darling PG tests (2)","build","Darling PG tests (0)"]} +{"date":"2026-10-05","head_sha":"6743f8286b89a4314c3714acf7ab3ab874c8f056","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs","Darling/Darling.Tests/SlowReadsRungTests.cs","Darling/Darling.Tests/StoreSelfMetricsTests.cs","Darling/Darling.Tests/StoreStatementHistoryLiveTests.cs","Darling/Darling.Tests/StoreStatementHistoryRungTests.cs","Darling/Darling.Tests/StoreStatementHistoryTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Storage/StoreStatementHistory.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/README.md"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-10-05","head_sha":"eba83d547adecc19bbb9b107aea17c76aca774ab","event":"pull_request","changed_files":[".github/scripts/lite-timing-top.py",".github/workflows/build.yml",".github/workflows/nightly.yml"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.QueryStoreTopLiteralEndStraddleLiveTests"],"failed_jobs":["build","Darling PG tests (2)","Darling PG tests (0)","Darling whole-tree guards"]} +{"date":"2026-10-05","head_sha":"99ad6aebf68c4d7cd0e1fd54f6bbea46a4303be0","event":"pull_request","changed_files":["Darling/Darling.Tests/PayloadDigestPassThroughLiveTests.cs","Darling/PerformanceMonitor.Darling.Storage/PayloadDimensionBatch.cs","Darling/PerformanceMonitor.Darling.Storage/PayloadDimensionWriter.cs","Darling/PerformanceMonitor.Darling.Storage/PayloadDimensions.cs","Darling/PerformanceMonitor.Darling.Storage/PgCollectorRowWriter.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.PlanSync4512CancellationTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-10-05","head_sha":"0b92d8248f506ee2accb62cf0ad2b3a6f0422ffb","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingBlockingStatsReadTests.cs","Darling/Darling.Tests/EffectiveStartUtcTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/PgEventToolsWindowFloorLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgDeadlockTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPgLogEventTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpWindowNotice.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests","Darling.Tests.PgServerLogTailRotationLiveTests"],"failed_jobs":["build","Darling PG tests (0)","Darling PG tests (1)","Darling whole-tree guards"]} +{"date":"2026-10-05","head_sha":"9de997ed48afc0d22a579dd989aaa3bb8d8a2a45","event":"pull_request","changed_files":[".github/ISSUE_TEMPLATE/bug_report.yml","Darling/Darling.Tests/BugReportTemplateDiagnosticsBundleTests.cs","Darling/Darling.Tests/DarlingCliUnusableStoreConnectionTests.cs","Darling/Darling.Tests/DarlingCloudIdentityProbeTests.cs","Darling/Darling.Tests/DiagnosticsBundleAliasHardeningTests.cs","Darling/Darling.Tests/DiagnosticsBundleAliasTests.cs","Darling/Darling.Tests/DiagnosticsBundleExitCodeTests.cs","Darling/Darling.Tests/DiagnosticsBundleHardeningTests.cs","Darling/Darling.Tests/DiagnosticsBundleHealthUncutLiveTests.cs","Darling/Darling.Tests/DiagnosticsBundleLeakLiveTests.cs","Darling/Darling.Tests/DiagnosticsBundleRound2Tests.cs","Darling/Darling.Tests/DiagnosticsBundleSecretTests.cs","Darling/Darling.Tests/DiagnosticsBundleSeedFailureLiveTests.cs","Darling/Darling.Tests/DiagnosticsBundleServiceLogTests.cs","Darling/Darling.Tests/DiagnosticsBundleSizeTests.cs","Darling/Darling.Tests/DiagnosticsBundleVerbTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundle.cs","Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundleAliaser.cs","Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundleRunner.cs","Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundleServiceLog.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Program.cs","Darling/PerformanceMonitor.Darling.Service/SecretTextGuard.cs","Darling/PerformanceMonitor.Darling.Service/SlowReadLog.cs","Darling/README.md"],"failed_classes":["Darling.Tests.DiagnosticsBundleRound2Tests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-10-05","head_sha":"bbba545e345592ca21ace8325c2a7b07f38d4195","event":"pull_request","changed_files":[".github/ISSUE_TEMPLATE/bug_report.yml","Darling/Darling.Tests/BugReportTemplateDiagnosticsBundleTests.cs","Darling/Darling.Tests/DarlingCliUnusableStoreConnectionTests.cs","Darling/Darling.Tests/DarlingCloudIdentityProbeTests.cs","Darling/Darling.Tests/DiagnosticsBundleAliasHardeningTests.cs","Darling/Darling.Tests/DiagnosticsBundleAliasTests.cs","Darling/Darling.Tests/DiagnosticsBundleExitCodeTests.cs","Darling/Darling.Tests/DiagnosticsBundleHardeningTests.cs","Darling/Darling.Tests/DiagnosticsBundleHealthUncutLiveTests.cs","Darling/Darling.Tests/DiagnosticsBundleLeakLiveTests.cs","Darling/Darling.Tests/DiagnosticsBundleRound2Tests.cs","Darling/Darling.Tests/DiagnosticsBundleSecretTests.cs","Darling/Darling.Tests/DiagnosticsBundleSeedFailureLiveTests.cs","Darling/Darling.Tests/DiagnosticsBundleServiceLogTests.cs","Darling/Darling.Tests/DiagnosticsBundleSizeTests.cs","Darling/Darling.Tests/DiagnosticsBundleVerbTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundle.cs","Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundleAliaser.cs","Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundleRunner.cs","Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundleServiceLog.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Program.cs","Darling/PerformanceMonitor.Darling.Service/SecretTextGuard.cs","Darling/PerformanceMonitor.Darling.Service/SlowReadLog.cs","Darling/README.md"],"failed_classes":["Darling.Tests.DiagnosticsBundleSecretTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-10-05","head_sha":"eca647c2f4eac3539062020a28181d7e29becb39","event":"pull_request","changed_files":["Darling/Darling.Tests/AzureSqlDatabaseHardwareTests.cs","Darling/Darling.Tests/DarlingExportViewerConfigTests.cs","Darling/Darling.Tests/FinOpsTabIndexAnalysisPageTests.cs","Darling/Darling.Tests/FinOpsTabRecommendationsPageTests.cs","Darling/Darling.Tests/FinOpsTabServerInventoryPageTests.cs","Darling/Darling.Tests/TestsNoSilentPassGuardTests.cs","Darling/Darling.Tests/WebCollectionHealthRunAtColumnTests.cs","Darling/Darling.Tests/WebDatabaseSizesNoteColumnTests.cs","Darling/Darling.Tests/WebModuleLoadTests.cs"],"failed_classes":["PerformanceMonitorLite.Tests.QueryStoreRegressionsToolTests","PerformanceMonitorLite.Tests.SignificantWaitsToolTests"],"failed_jobs":["Lite tests (3)","Lite tests (2)"]} +{"date":"2026-10-05","head_sha":"48216212d0856c5343d6aed0c877f2df261b5ddd","event":"pull_request","changed_files":["Lite.Tests/DataStartBannerQueriesTabTests.cs","Lite.Tests/DataStartBannerSurfaceTests.cs","Lite.Tests/McpWindowNoticeAggregateToolTests.cs","Lite.Tests/McpWindowNoticeConfigAndLogToolTests.cs","Lite.Tests/McpWindowNoticeEventToolTests.cs","Lite.Tests/McpWindowNoticeSystemHealthToolTests.cs","Lite.Tests/McpWindowNoticeToolTests.cs","Lite.Tests/QueryWindowTruncationTests.cs","Lite.Tests/SharedDuckDbFixture.cs"],"failed_classes":["Darling.Tests.PlanSync4512CancellationTests"],"failed_jobs":["build"]} +{"date":"2026-10-05","head_sha":"b463460776890e2106577fb3c489717a7a6b6642","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingBlockingStatsReadTests.cs","Darling/Darling.Tests/DarlingCollectionLogReadTests.cs","Darling/Darling.Tests/DarlingMcpHealthParserToolsTests.cs","Darling/Darling.Tests/DarlingMcpHealthParserToolsWindowNoticeTests.cs","Darling/Darling.Tests/DarlingQueryHeatmapTests.cs","Darling/Darling.Tests/DarlingQueryStoreRegressionsTests.cs","Darling/Darling.Tests/DarlingSignificantWaitsReadTests.cs","Darling/Darling.Tests/EffectiveStartUtcTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHealthParserTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryHeatmapTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreRegressionTools.cs","Lite.Tests/CollectionLogToolTests.cs","Lite.Tests/McpMissMessageParityPinTests.cs","Lite.Tests/McpWindowNoticeAggregateToolTests.cs","Lite.Tests/McpWindowNoticeConfigAndLogToolTests.cs","Lite.Tests/McpWindowNoticeSystemHealthToolTests.cs","Lite.Tests/McpWindowNoticeToolTests.cs","Lite.Tests/QueryStoreRegressionsToolTests.cs","Lite.Tests/SignificantWaitsToolTests.cs","Lite/Mcp/McpHealthParserTools.cs","Lite/Mcp/McpHealthTools.cs","Lite/Mcp/McpInstructions.cs","Lite/Mcp/McpQueryTools.cs","PerformanceMonitor.Common/Mcp/McpHelpers.cs"],"failed_classes":["Darling.Tests.ComposeDataFloorLiveTests","Darling.Tests.ComposedPanelNoticeZoneTests"],"failed_jobs":["build","Darling PG tests (0)","Darling PG tests (2)"]} +{"date":"2026-10-05","head_sha":"ffac2c1f8b01c5788f3326692e58e702216dfa80","event":"pull_request","changed_files":["Darling/Darling.Tests/PgPlanViewerBehaviourTests.cs","Darling/Darling.Tests/WebPgPlanViewerPageTests.cs","Darling/Darling.Tests/web-blocking-save-xml-harness.mjs","Darling/Darling.Tests/web-collection-caveats-harness.mjs","Darling/Darling.Tests/web-deadlock-processes-harness.mjs","Darling/Darling.Tests/web-kept-history-harness.mjs","Darling/Darling.Tests/web-memory-pressure-harness.mjs","Darling/Darling.Tests/web-perfmon-harness.mjs","Darling/Darling.Tests/web-perfmon-multi-harness.mjs","Darling/Darling.Tests/web-pg-plan-viewer-harness.mjs","Darling/Darling.Tests/web-server-trends-harness.mjs","Darling/Darling.Tests/web-waits-activity-trends-harness.mjs","Darling/Darling.Tests/web-waits-harness.mjs","Darling/PerformanceMonitor.Darling.Service/wwwroot/css/app.css","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/pg-plan-viewer.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js"],"failed_classes":["Darling.Tests.ComposeDataFloorLiveTests","Darling.Tests.ComposedPanelNoticeZoneTests","Darling.Tests.WebCollectorRunHistoryTests"],"failed_jobs":["Darling PG tests (2)","build","Darling PG tests (0)"]} +{"date":"2026-10-06","head_sha":"6ac72f860523f66d8e29944dc094fa58c9a779f9","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpQueryStoreHistoryToolsTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/EffectiveStartUtcTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpQueryStoreHistoryTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/PlanViewerBehaviourTests.cs","Darling/Darling.Tests/QueryStoreHistoryBehaviourTests.cs","Darling/Darling.Tests/QueryStoreHistoryLivePostgresTests.cs","Darling/Darling.Tests/QueryStoreHistoryPanelStateTests.cs","Darling/Darling.Tests/QueryStoreHistoryWideTierLiveTests.cs","Darling/Darling.Tests/RawWindowFloorQuietStartTests.cs","Darling/Darling.Tests/WebPlanViewerPageTests.cs","Darling/Darling.Tests/WebQueryStoreHistoryColumnTests.cs","Darling/Darling.Tests/web-plan-viewer-harness.mjs","Darling/Darling.Tests/web-qs-history-harness.mjs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreHistoryTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/plan-viewer.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/query-store-history.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js","Darling/README.md","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","README.md","llms.txt"],"failed_classes":["Darling.Tests.StoreStatementStatsLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-10-06","head_sha":"7c4da8f91f653161036febf4b2d06ae4cd7ca776","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionLogDrainForensicsStoreTests.cs","Darling/Darling.Tests/PgSensitiveStatementFilterLiveTests.cs","Darling/Darling.Tests/PgSensitiveStatementFilterTests.cs","Darling/Darling.Tests/PgStatementTextFailureRecordingTests.cs","Darling/Darling.Tests/PgStatementTextHealthSemanticsTests.cs","Darling/Darling.Tests/PgStatementTextScrubLiveTests.cs","Darling/Darling.Tests/SensitiveStatementCorpus.cs","Darling/Darling.Tests/SensitiveStatementsParityLiveTests.cs","Darling/Darling.Tests/SensitiveStatementsTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/PgStatementTextScrub.cs","Darling/PerformanceMonitor.Darling.Storage/PgStatementText.cs","PerformanceMonitor.Collectors/PgSensitiveStatementFilter.cs","PerformanceMonitor.Common/SensitiveStatements.Regex.cs","PerformanceMonitor.Common/SensitiveStatements.cs"],"failed_classes":["Darling.Tests.StoreStatementStatsLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-10-06","head_sha":"33932296ca375d2debf70fcc2d217ec3852336dd","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpQueryStoreHistoryToolsTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/EffectiveStartUtcTests.cs","Darling/Darling.Tests/McpPayloadContractCensusTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpQueryStoreHistoryTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/PlanViewerBehaviourTests.cs","Darling/Darling.Tests/QueryStoreHistoryBehaviourTests.cs","Darling/Darling.Tests/QueryStoreHistoryLivePostgresTests.cs","Darling/Darling.Tests/QueryStoreHistoryPanelStateTests.cs","Darling/Darling.Tests/QueryStoreHistoryWideTierLiveTests.cs","Darling/Darling.Tests/RawWindowFloorQuietStartTests.cs","Darling/Darling.Tests/WebPlanViewerPageTests.cs","Darling/Darling.Tests/WebQueryStoreHistoryColumnTests.cs","Darling/Darling.Tests/web-plan-viewer-harness.mjs","Darling/Darling.Tests/web-qs-history-harness.mjs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreHistoryTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/plan-viewer.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/query-store-history.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js","Darling/README.md","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","README.md","llms.txt"],"failed_classes":["Darling.Tests.StoreStatementStatsLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-10-06","head_sha":"fa3497bb39b888a14e575841191250e5ca4cfcb5","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeQueryStoreWideExactnessLiveTests.cs","Darling/Darling.Tests/DarlingCustomViewsTests.cs","Darling/Darling.Tests/DarlingMcpDataToolsTests.cs","Darling/Darling.Tests/McpPageContractTests.cs","Darling/Darling.Tests/QueryStoreBackgroundIndexesLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideBelowFloorLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideBelowFloorRoutesLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideGridLiveTests.cs","Darling/Darling.Tests/QueryStoreTopDailyReadLiveTests.cs","Darling/Darling.Tests/QueryStoreTopMcpLiveTests.cs","Darling/Darling.Tests/ServerPageTabsTests.cs","Darling/Darling.Tests/TopFillEmptyRoundLiveTests.cs","Darling/Darling.Tests/TopProceduresHourlyRoutingTests.cs","Darling/Darling.Tests/TopQueriesHourlyRoutingTests.cs","Darling/Darling.Tests/TopRankingDifferentialLiveTests.cs","Darling/Darling.Tests/TopRankingHourlyLiveTests.cs","Darling/Darling.Tests/TopRankingLiveTests.cs","Darling/Darling.Tests/TopRankingNoticeBehaviourTests.cs","Darling/Darling.Tests/TopRankingOnePassBaseline.cs","Darling/Darling.Tests/TopRankingPagePinTests.cs","Darling/Darling.Tests/TopRankingTests.cs","Darling/Darling.Tests/TopRankingTwoPassTests.cs","Darling/Darling.Tests/TopTextLookupFillTests.cs","Darling/Darling.Tests/TopTextLookupPlanLiveTests.cs","Darling/Darling.Tests/TopTextTieBreakLiveTests.cs","Darling/Darling.Tests/ViewerQueriesTests.cs","Darling/Darling.Tests/WebTopProceduresColumnsTests.cs","Darling/Darling.Tests/WebTopQueriesColumnsTests.cs","Darling/Darling.Tests/top-ranking-notice-harness.mjs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/TopRanking.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js","Darling/PerformanceMonitor.Darling.Storage/TopFill.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStats.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStore.cs","Lite.Tests/TopListTotalOrderPinTests.cs","Lite.Tests/TopListWaitforRefillReaderTests.cs","Lite.Tests/TopTextTieBreakReaderTests.cs","Lite/Services/LocalDataService.QueryStats.cs","Lite/Services/LocalDataService.QueryStore.cs","Lite/Services/TopFill.cs"],"failed_classes":["Darling.Tests.LiveCleanupConversionRatchetTests","Darling.Tests.LivePlanShapeIsolationCensusTests","Darling.Tests.WebTopProceduresColumnsTests"],"failed_jobs":["Darling PG tests (2)","build","Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-10-06","head_sha":"3af3ade9ea1061fd96eca0b2ee1ece351efe17c3","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeQueryStoreWideExactnessLiveTests.cs","Darling/Darling.Tests/DarlingCustomViewsTests.cs","Darling/Darling.Tests/DarlingMcpDataToolsTests.cs","Darling/Darling.Tests/McpPageContractTests.cs","Darling/Darling.Tests/QueryStoreBackgroundIndexesLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideBelowFloorLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideBelowFloorRoutesLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideGridLiveTests.cs","Darling/Darling.Tests/QueryStoreTopDailyReadLiveTests.cs","Darling/Darling.Tests/QueryStoreTopMcpLiveTests.cs","Darling/Darling.Tests/ServerPageTabsTests.cs","Darling/Darling.Tests/TopFillEmptyRoundLiveTests.cs","Darling/Darling.Tests/TopProceduresHourlyRoutingTests.cs","Darling/Darling.Tests/TopQueriesHourlyRoutingTests.cs","Darling/Darling.Tests/TopRankingDifferentialLiveTests.cs","Darling/Darling.Tests/TopRankingHourlyLiveTests.cs","Darling/Darling.Tests/TopRankingLiveTests.cs","Darling/Darling.Tests/TopRankingNoticeBehaviourTests.cs","Darling/Darling.Tests/TopRankingOnePassBaseline.cs","Darling/Darling.Tests/TopRankingPagePinTests.cs","Darling/Darling.Tests/TopRankingTests.cs","Darling/Darling.Tests/TopRankingTwoPassTests.cs","Darling/Darling.Tests/TopTextLookupFillTests.cs","Darling/Darling.Tests/TopTextLookupPlanLiveTests.cs","Darling/Darling.Tests/TopTextTieBreakLiveTests.cs","Darling/Darling.Tests/ViewerQueriesTests.cs","Darling/Darling.Tests/WebTopProceduresColumnsTests.cs","Darling/Darling.Tests/WebTopQueriesColumnsTests.cs","Darling/Darling.Tests/top-ranking-notice-harness.mjs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/TopRanking.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js","Darling/PerformanceMonitor.Darling.Storage/TopFill.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStats.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStore.cs","Lite.Tests/TopListTotalOrderPinTests.cs","Lite.Tests/TopListWaitforRefillReaderTests.cs","Lite.Tests/TopTextTieBreakReaderTests.cs","Lite/Services/LocalDataService.QueryStats.cs","Lite/Services/LocalDataService.QueryStore.cs","Lite/Services/TopFill.cs"],"failed_classes":["Darling.Tests.LiveCleanupConversionRatchetTests","Darling.Tests.LivePlanShapeIsolationCensusTests","Darling.Tests.StoreStatementStatsLiveTests","Darling.Tests.WebTopProceduresColumnsTests"],"failed_jobs":["Darling PG tests (2)","build","Darling PG tests (1)","Darling PG tests (0)"]} +{"date":"2026-10-06","head_sha":"19936ab8788df98247728fbfb4d6856d211168f5","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeQueryStoreWideExactnessLiveTests.cs","Darling/Darling.Tests/DarlingCustomViewsTests.cs","Darling/Darling.Tests/DarlingMcpDataToolsTests.cs","Darling/Darling.Tests/McpPageContractTests.cs","Darling/Darling.Tests/QueryStoreBackgroundIndexesLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideBelowFloorLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideBelowFloorRoutesLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideGridLiveTests.cs","Darling/Darling.Tests/QueryStoreTopDailyReadLiveTests.cs","Darling/Darling.Tests/QueryStoreTopMcpLiveTests.cs","Darling/Darling.Tests/ServerPageTabsTests.cs","Darling/Darling.Tests/TopFillEmptyRoundLiveTests.cs","Darling/Darling.Tests/TopProceduresHourlyRoutingTests.cs","Darling/Darling.Tests/TopQueriesHourlyRoutingTests.cs","Darling/Darling.Tests/TopRankingDifferentialLiveTests.cs","Darling/Darling.Tests/TopRankingHourlyLiveTests.cs","Darling/Darling.Tests/TopRankingLiveTests.cs","Darling/Darling.Tests/TopRankingNoticeBehaviourTests.cs","Darling/Darling.Tests/TopRankingOnePassBaseline.cs","Darling/Darling.Tests/TopRankingPagePinTests.cs","Darling/Darling.Tests/TopRankingTests.cs","Darling/Darling.Tests/TopRankingTwoPassTests.cs","Darling/Darling.Tests/TopTextLookupFillTests.cs","Darling/Darling.Tests/TopTextLookupPlanLiveTests.cs","Darling/Darling.Tests/TopTextTieBreakLiveTests.cs","Darling/Darling.Tests/ViewerQueriesTests.cs","Darling/Darling.Tests/WebTopProceduresColumnsTests.cs","Darling/Darling.Tests/WebTopQueriesColumnsTests.cs","Darling/Darling.Tests/top-ranking-notice-harness.mjs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/TopRanking.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js","Darling/PerformanceMonitor.Darling.Storage/TopFill.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStats.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStore.cs","Lite.Tests/TopListTotalOrderPinTests.cs","Lite.Tests/TopListWaitforRefillReaderTests.cs","Lite.Tests/TopTextTieBreakReaderTests.cs","Lite/Services/LocalDataService.QueryStats.cs","Lite/Services/LocalDataService.QueryStore.cs","Lite/Services/TopFill.cs"],"failed_classes":["Darling.Tests.LiveCleanupConversionRatchetTests","Darling.Tests.LivePlanShapeIsolationCensusTests","Darling.Tests.WebTopProceduresColumnsTests"],"failed_jobs":["build","Darling PG tests (0)","Darling PG tests (1)","Darling PG tests (2)"]} +{"date":"2026-10-06","head_sha":"51ccf6b024b2fd53c1ab97b345bb0dcbacceaf5c","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingConfigTests.cs","Darling/Darling.Tests/DarlingManagedRolesTests.cs","Darling/Darling.Tests/DarlingMcpServerAdminToolsTests.cs","Darling/Darling.Tests/DarlingSecretsOwnedReferenceTests.cs","Darling/Darling.Tests/ServerAddRouteTests.cs","Darling/Darling.Tests/ServerAddViewerRoleLiveTests.cs","Darling/Darling.Tests/ServerEditCoreTests.cs","Darling/Darling.Tests/ServerEditGateLiveTests.cs","Darling/Darling.Tests/ServerEditLiveTests.cs","Darling/Darling.Tests/ServerEditPasswordRuleViewerTests.cs","Darling/Darling.Tests/ServerEditRouteTests.cs","Darling/Darling.Tests/ServerEditViewerRoleLiveTests.cs","Darling/Darling.Tests/ServerIdentityLockLiveTests.cs","Darling/Darling.Tests/ServerIdentityLockSourcePinTests.cs","Darling/Darling.Tests/StartupCommandTimeoutTests.cs","Darling/Darling.Tests/ViewerAddServerIdCollisionTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSecrets.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerAdminTools.Edit.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerAdminTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Viewer/AddServerDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.MonitoredServers.cs","Darling/tools/provision-roles.sql"],"failed_classes":["Darling.Tests.DarlingReadsUseOnlyStoreViewsTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-10-06","head_sha":"6866eb5664a378aa35f09c7576b9e883e2fdc177","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectorMeasurementSeamTests.cs","Darling/Darling.Tests/SensitiveStatementHostFilterTests.cs","Darling/Darling.Tests/SensitiveStatementOutputFilterTests.cs","Darling/Darling.Tests/SensitiveStatementSessionTests.cs","Darling/Darling.Tests/StatementColumnCensusTests.cs","Darling/Darling.Tests/StatementFilterCensus.cs","Darling/Darling.Tests/StatementScrubRecordingWriter.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanTools.cs","Lite/Mcp/McpPlanTools.cs","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Common/Mcp/SensitiveStatementOutputFilter.cs","PerformanceMonitor.Common/SensitiveStatements.Session.cs","PerformanceMonitor.Common/SensitiveStatements.cs"],"failed_classes":["PerformanceMonitorLite.Tests.PerfmonCounterTypeTests"],"failed_jobs":["Lite tests (1)"]} +{"date":"2026-10-06","head_sha":"c09289f16cc2eaec99f49983b1eacaafbc53eabd","event":"pull_request","changed_files":["Darling/Darling.Tests/BlockingListPlanFlagsLivePostgresTests.cs","Darling/Darling.Tests/DarlingBlockingReaderServerClockTests.cs","Darling/Darling.Tests/DarlingMcpBlockingToolsTests.cs","Darling/Darling.Tests/DarlingMcpPlanToolsTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpPlanTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/PlanViewerBehaviourTests.cs","Darling/Darling.Tests/RowPlanReadLivePostgresTests.cs","Darling/Darling.Tests/WebPlanViewerPageTests.cs","Darling/Darling.Tests/WebReadCancellationPinTests.cs","Darling/Darling.Tests/web-plan-viewer-harness.mjs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingBlockingReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpBlockingTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInstructions.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoredPlanReader.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/plan-viewer.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/README.md","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","README.md","llms.txt"],"failed_classes":["Darling.Tests.DarlingReadsUseOnlyStoreViewsTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-10-06","head_sha":"817a583826610decb34e0d7b00bcc5c6cbb285e8","event":"pull_request","changed_files":[".claude/skills/release-checklist/SKILL.md","Darling/Darling.Tests/CollectorMeasurementSeamTests.cs","Darling/Darling.Tests/SensitiveStatementHostFilterTests.cs","Darling/Darling.Tests/SensitiveStatementOutputFilterTests.cs","Darling/Darling.Tests/SensitiveStatementSessionTests.cs","Darling/Darling.Tests/StatementCollectionCensusCases.cs","Darling/Darling.Tests/StatementColumnCensusTests.cs","Darling/Darling.Tests/StatementFilterCensus.cs","Darling/Darling.Tests/StatementScrubRecordingWriter.cs","Darling/Darling.Tests/StoredPlanAnalysisFilterTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanTools.cs","Lite.Tests/Lite.Tests.csproj","Lite.Tests/PerfmonCounterTypeTests.cs","Lite.Tests/StoredPlanAnalysisFilterTests.cs","Lite/Mcp/McpPlanTools.cs","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/PerfmonStatsCollector.cs","PerformanceMonitor.Common/Mcp/SensitiveStatementOutputFilter.cs","PerformanceMonitor.Common/PerfmonCounterTypes.cs","PerformanceMonitor.Common/SensitiveStatements.Session.cs","PerformanceMonitor.Common/SensitiveStatements.cs"],"failed_classes":["Lite.Tests.DarlingReadsTraitGuardTests"],"failed_jobs":["Lite tests (1)"]} +{"date":"2026-10-06","head_sha":"d1343812746131b19b0192ab42a25527b784904b","event":"pull_request","changed_files":["Darling/Darling.Tests/ActiveQueriesDatabaseFilterLiveTests.cs","Darling/Darling.Tests/ComposeQueryStoreWideExactnessLiveTests.cs","Darling/Darling.Tests/DarlingMcpDataToolsTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/DarlingWebHostGateLiveTests.cs","Darling/Darling.Tests/DarlingWebOidcTests.cs","Darling/Darling.Tests/DatabaseFilterBindingLiveTests.cs","Darling/Darling.Tests/ExcludedDatabasesTests.cs","Darling/Darling.Tests/McpPageContractTests.cs","Darling/Darling.Tests/QueryStoreBackgroundIndexesLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideBelowFloorLiveTests.cs","Darling/Darling.Tests/QueryStoreReadsDatabaseFilterLiveTests.cs","Darling/Darling.Tests/QueryStoreTopDailyReadLiveTests.cs","Darling/Darling.Tests/QueryStoreTopMcpLiveTests.cs","Darling/Darling.Tests/QueryStoreTopWebDefaultTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/Darling.Tests/ServerDatabasesEndpointLiveTests.cs","Darling/Darling.Tests/ServerPageTabsTests.cs","Darling/Darling.Tests/TopQueriesDatabaseFilterLiveTests.cs","Darling/Darling.Tests/TopRankingTwoPassTests.cs","Darling/Darling.Tests/TopTextTieBreakLiveTests.cs","Darling/Darling.Tests/ViewerLocalStateBehaviourTests.cs","Darling/Darling.Tests/WebDatabaseFilterBehaviourTests.cs","Darling/Darling.Tests/WebDatabaseFilterChipsBehaviourTests.cs","Darling/Darling.Tests/WebDatabaseFilterDispatchLiveTests.cs","Darling/Darling.Tests/WebDatabaseFilterPickerBehaviourTests.cs","Darling/Darling.Tests/WebDatabaseFilterReadsTests.cs","Darling/Darling.Tests/WebDatabaseFilterTests.cs","Darling/Darling.Tests/viewer-local-harness.mjs","Darling/Darling.Tests/web-database-filter-chips-harness.mjs","Darling/Darling.Tests/web-database-filter-harness.mjs","Darling/Darling.Tests/web-database-filter-picker-harness.mjs","Darling/PerformanceMonitor.Darling.Service/AlertNotebookEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingFleetSweepEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingServerDatabasesEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryHeatmapTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreRegressionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpSessionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingQueryHeatmapReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingQueryStoreRegressionReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingRuntimePrecondition.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingSessionReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/css/app.css","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/database-filter-reads.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/multi-picker.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/analysis-findings.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/database-filter.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/viewer-local.js","Darling/PerformanceMonitor.Darling.Storage/CollectedDatabases.cs","Darling/PerformanceMonitor.Darling.Storage/DatabaseFilter.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.ExcludedDatabases.cs","PerformanceMonitor.Common/Mcp/McpHelpers.cs"],"failed_classes":["Darling.Tests.PgIndexBloatCoverageTests","PerformanceMonitor.Darling.Tests.TopTextLookupPlanLiveTests"],"failed_jobs":["Darling PG tests (2)","build","Darling PG tests (0)"]} +{"date":"2026-10-06","head_sha":"2090a54247c30e9fb6d98a45598f8f7bdcc80890","event":"pull_request","changed_files":["Darling/Darling.Tests/ActiveQueriesDatabaseFilterLiveTests.cs","Darling/Darling.Tests/ComposeQueryStoreWideExactnessLiveTests.cs","Darling/Darling.Tests/DarlingMcpDataToolsTests.cs","Darling/Darling.Tests/DarlingWebEndpointsTests.cs","Darling/Darling.Tests/DarlingWebHostGateLiveTests.cs","Darling/Darling.Tests/DarlingWebOidcTests.cs","Darling/Darling.Tests/DatabaseFilterBindingLiveTests.cs","Darling/Darling.Tests/ExcludedDatabasesTests.cs","Darling/Darling.Tests/McpPageContractTests.cs","Darling/Darling.Tests/PgIndexBloatCoverageTests.cs","Darling/Darling.Tests/QueryStoreBackgroundIndexesLiveTests.cs","Darling/Darling.Tests/QueryStoreIntervalWideBelowFloorLiveTests.cs","Darling/Darling.Tests/QueryStoreReadsDatabaseFilterLiveTests.cs","Darling/Darling.Tests/QueryStoreTopDailyReadLiveTests.cs","Darling/Darling.Tests/QueryStoreTopMcpLiveTests.cs","Darling/Darling.Tests/QueryStoreTopWebDefaultTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/Darling.Tests/ServerDatabasesEndpointLiveTests.cs","Darling/Darling.Tests/ServerPageTabsTests.cs","Darling/Darling.Tests/SlowReadLogTests.cs","Darling/Darling.Tests/TopQueriesDatabaseFilterLiveTests.cs","Darling/Darling.Tests/TopRankingTwoPassTests.cs","Darling/Darling.Tests/TopTextLookupPlanLiveTests.cs","Darling/Darling.Tests/TopTextTieBreakLiveTests.cs","Darling/Darling.Tests/ViewerLocalStateBehaviourTests.cs","Darling/Darling.Tests/WebDatabaseBlankRuleTests.cs","Darling/Darling.Tests/WebDatabaseFilterBehaviourTests.cs","Darling/Darling.Tests/WebDatabaseFilterChipsBehaviourTests.cs","Darling/Darling.Tests/WebDatabaseFilterDispatchLiveTests.cs","Darling/Darling.Tests/WebDatabaseFilterPickerBehaviourTests.cs","Darling/Darling.Tests/WebDatabaseFilterReadsTests.cs","Darling/Darling.Tests/WebDatabaseFilterTests.cs","Darling/Darling.Tests/viewer-local-blank-harness.mjs","Darling/Darling.Tests/viewer-local-harness.mjs","Darling/Darling.Tests/web-database-filter-chips-harness.mjs","Darling/Darling.Tests/web-database-filter-harness.mjs","Darling/Darling.Tests/web-database-filter-picker-harness.mjs","Darling/PerformanceMonitor.Darling.Service/AlertNotebookEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingFleetSweepEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingServerDatabasesEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryHeatmapTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreRegressionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpSessionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingQueryHeatmapReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingQueryStoreRegressionReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingRuntimePrecondition.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingSessionReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingWebHostService.cs","Darling/PerformanceMonitor.Darling.Service/SlowReadLog.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/css/app.css","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/database-filter-reads.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/multi-picker.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/analysis-findings.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/database-filter.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/viewer-local.js","Darling/PerformanceMonitor.Darling.Storage/CollectedDatabases.cs","Darling/PerformanceMonitor.Darling.Storage/DatabaseFilter.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.ExcludedDatabases.cs","PerformanceMonitor.Common/Mcp/McpHelpers.cs"],"failed_classes":["Darling.Tests.WebDatabaseFilterReadsTests"],"failed_jobs":["build","Darling PG tests (2)"]} +{"date":"2026-10-06","head_sha":"b4c074a00f6f45da7f674ee8a036e4bb5e53bf73","event":"pull_request","changed_files":[".claude/skills/release-checklist/SKILL.md",".github/workflows/build.yml","Darling/Darling.Tests/CollectorMeasurementSeamTests.cs","Darling/Darling.Tests/SensitiveStatementHostFilterTests.cs","Darling/Darling.Tests/SensitiveStatementOutputFilterTests.cs","Darling/Darling.Tests/SensitiveStatementSessionTests.cs","Darling/Darling.Tests/StatementCollectionCensusCases.cs","Darling/Darling.Tests/StatementColumnCensusTests.cs","Darling/Darling.Tests/StatementFilterCensus.cs","Darling/Darling.Tests/StatementScrubRecordingWriter.cs","Darling/Darling.Tests/StoredPlanAnalysisFilterTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHostService.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanTools.cs","Lite.Tests/Lite.Tests.csproj","Lite.Tests/PerfmonCounterTypeTests.cs","Lite.Tests/StoredPlanAnalysisFilterTests.cs","Lite/Mcp/McpPlanTools.cs","PerformanceMonitor.Collectors/CollectorContext.cs","PerformanceMonitor.Collectors/PerfmonStatsCollector.cs","PerformanceMonitor.Common/Mcp/SensitiveStatementOutputFilter.cs","PerformanceMonitor.Common/PerfmonCounterTypes.cs","PerformanceMonitor.Common/SensitiveStatements.Session.cs","PerformanceMonitor.Common/SensitiveStatements.cs"],"failed_classes":["Darling.Tests.CommentFilterAdoptionTests"],"failed_jobs":["Darling PG tests (2)","build","Darling whole-tree guards"]} +{"date":"2026-10-06","head_sha":"1e6f65cf044e1be9ea47d3c981ae33a5e9734b15","event":"pull_request","changed_files":["Darling/Darling.Tests/AdminPageBehaviourTests.cs","Darling/Darling.Tests/AdminPageTests.cs","Darling/Darling.Tests/AdminServerEditBehaviourTests.cs","Darling/Darling.Tests/AdminServersRouteTests.cs","Darling/Darling.Tests/ManageTagsBehaviourTests.cs","Darling/Darling.Tests/ManageTagsPageTests.cs","Darling/Darling.Tests/admin-page-harness.mjs","Darling/Darling.Tests/admin-server-edit-harness.mjs","Darling/Darling.Tests/manage-tags-harness.mjs","Darling/PerformanceMonitor.Darling.Service/DarlingAdminServersReader.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/admin.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/manage-tags.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js"],"failed_classes":["Darling.Tests.FleetIdentifierScrubTests"],"failed_jobs":["build","Darling PG tests (0)"]} +{"date":"2026-10-06","head_sha":"d0552429dd7909d94539d7d752eb2cfd8fd1b948","event":"pull_request","changed_files":["Darling/Darling.Tests/PagerDutyAutoResolveRungTests.cs","Darling/Darling.Tests/QueryStatsHourLedgerTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertSettings.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Notification.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/tools/provision-roles.sql","Lite.Tests/PagerDutyWebhookTests.cs","Lite/App.xaml.cs","Lite/Services/AppAlertSettings.cs","Lite/Windows/SettingsWindow.xaml","Lite/Windows/SettingsWindow.xaml.cs","Lite/config/settings.sample.json","PerformanceMonitor.Notifications/IAlertSettings.cs","PerformanceMonitor.Notifications/WebhookAlertService.cs"],"failed_classes":["Darling.Tests.CaggGroupIndexDropLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-10-06","head_sha":"f2dcc58e9d45a2f14353d3ede4c00c28d6c14c43","event":"pull_request","changed_files":["Darling/Darling.Tests/PagerDutyAutoResolveRungTests.cs","Darling/Darling.Tests/QueryStatsHourLedgerTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertSettings.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Notification.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/tools/provision-roles.sql","Lite.Tests/PagerDutyWebhookTests.cs","Lite/App.xaml.cs","Lite/Services/AppAlertSettings.cs","Lite/Windows/SettingsWindow.xaml","Lite/Windows/SettingsWindow.xaml.cs","Lite/config/settings.sample.json","PerformanceMonitor.Notifications/IAlertSettings.cs","PerformanceMonitor.Notifications/WebhookAlertService.cs"],"failed_classes":["Darling.Tests.CaggGroupIndexDropLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-10-06","head_sha":"3925bff0a2a8144e8ce2849c0b5cd88ee02468bf","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpServerAdminToolsTests.cs","Darling/Darling.Tests/DarlingSecretsOwnedReferenceTests.cs","Darling/Darling.Tests/OwnedSecretReferenceRefusalTests.cs","Darling/Darling.Tests/ServerAddViewerRoleLiveTests.cs","Darling/Darling.Tests/ServerEditCoreTests.cs","Darling/Darling.Tests/ServerEditLiveTests.cs","Darling/Darling.Tests/ServerPasswordFromRequestTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCommandExecutor.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSecretSource.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerAdminTools.Edit.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerAdminTools.cs"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-10-06","head_sha":"71e66d9d2df0a6df2f2003b4e94679ef73cea6a1","event":"push","changed_files":["Darling/Darling.Tests/DarlingMcpObjectStatsToolsTests.cs","Darling/Darling.Tests/DatabaseNameParamTwinTests.cs","Darling/Darling.Tests/FinOpsIndexAnalysisLimitBehaviourTests.cs","Darling/Darling.Tests/FinOpsLockingDetailBehaviourTests.cs","Darling/Darling.Tests/FinOpsLockingHeatBehaviourTests.cs","Darling/Darling.Tests/FinOpsLockingPickerBehaviourTests.cs","Darling/Darling.Tests/FinOpsPageShellTests.cs","Darling/Darling.Tests/FinOpsTabIndexAnalysisPageTests.cs","Darling/Darling.Tests/FinOpsTabLockingPageTests.cs","Darling/Darling.Tests/IndexUsageTruncationTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpObjectStatsTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/ObjectLockingDetailLiveTests.cs","Darling/Darling.Tests/ObjectLockingHeatLiveTests.cs","Darling/Darling.Tests/ObjectLockingWebDefaultTests.cs","Darling/Darling.Tests/ObjectStatsDatabaseFilterLiveTests.cs","Darling/Darling.Tests/SeparatelyMonitoredListNoteTests.cs","Darling/Darling.Tests/WebHarness/load-web-modules.mjs","Darling/Darling.Tests/finops-index-analysis-limit-harness.mjs","Darling/Darling.Tests/finops-locking-detail-harness.mjs","Darling/Darling.Tests/finops-locking-heat-harness.mjs","Darling/Darling.Tests/finops-locking-picker-harness.mjs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpObjectStatsTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingObjectStatsReader.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/css/app.css","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/database-filter-reads.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/database-box.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/index-analysis.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/finops/locking.js","Lite.Tests/McpToolsListBudget/McpObjectStatsTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite.Tests/ObjectLockingBudgetTests.cs","Lite.Tests/OptimizedLockingNoteTests.cs","Lite/Controls/FinOpsTab.Locking.cs","Lite/Mcp/McpObjectStatsTools.cs","Lite/Services/LocalDataService.FinOps.IndexObjects.cs"],"failed_classes":["PerformanceMonitor.Darling.Tests.RetiredBaselineAggregateLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-10-06","head_sha":"ab17f612435931e266ace9482ff22f2e519459b7","event":"pull_request","changed_files":["Lite.Tests/ArchiveNoOpPassViewRebuildTests.cs","Lite.Tests/PostCollectionCheckpointBudgetTests.cs","Lite/Services/ArchiveService.cs","Lite/Services/RemoteCollectorService.cs"],"failed_classes":["Darling.Tests.TsqlConventionGuardTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-10-06","head_sha":"9be59f1a956a973199382e71ac3b7073b876bf15","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingPasswordKeyStateTests.cs","Darling/Darling.Tests/PasswordBindingCensusTests.cs","Darling/Darling.Tests/PasswordSealTests.cs","Darling/Darling.Tests/TestData/password-seal-v1-vector.json","Darling/PerformanceMonitor.Darling.Service/DarlingPasswordKeyState.cs","Darling/PerformanceMonitor.Darling.Storage/PasswordBinding.cs","Darling/PerformanceMonitor.Darling.Storage/PasswordSeal.cs","Darling/PerformanceMonitor.Darling.Storage/ServerConnectionIdentity.cs"],"failed_classes":["Darling.Tests.CollectionLogDrainForensicsStoreTests","Darling.Tests.FleetIdentifierScrubTests","Darling.Tests.SwallowedItemFailureTests"],"failed_jobs":["build","Darling PG tests (0)","Darling PG tests (2)"]} +{"date":"2026-10-06","head_sha":"398056f9681b10ce4bf12a873b07062caf9d696a","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingPasswordKeyFileHardeningTests.cs","Darling/Darling.Tests/DarlingPasswordKeyFileTests.cs","Darling/Darling.Tests/DarlingPasswordKeyFileUnixTests.cs","Darling/Darling.Tests/DarlingSecretsOwnedReferenceTests.cs","Darling/Darling.Tests/PgLogHashKeyTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingFileSecurity.cs","Darling/PerformanceMonitor.Darling.Service/DarlingLogHashKeyFile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/DarlingOwnedSecrets.cs","Darling/PerformanceMonitor.Darling.Service/DarlingPasswordKeyFile.cs","Darling/PerformanceMonitor.Darling.Service/DarlingServiceKeyFile.cs"],"failed_classes":["Darling.Tests.CommentFilterAdoptionTests","Darling.Tests.PayloadDimensionTests","Darling.Tests.PgTargetQueriesTests","Lite.Tests.DarlingReadsTraitGuardTests"],"failed_jobs":["Lite tests (3)","build","Darling PG tests (2)","Darling PG tests (1)"]} +{"date":"2026-10-06","head_sha":"31f07a7e20050bb78173e267a0360496dd605ba3","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedRolesTests.cs","Darling/Darling.Tests/DarlingSecretsOwnedReferenceTests.cs","Darling/Darling.Tests/RegisteredServerSettingDriftTests.cs","Darling/Darling.Tests/SelfManagedStoreRulesStartPinTests.cs","Darling/Darling.Tests/ServerEditCoreTests.cs","Darling/Darling.Tests/ServerEditLiveTests.cs","Darling/Darling.Tests/ServerEditPasswordRuleViewerTests.cs","Darling/Darling.Tests/ServerIdentityLockLiveTests.cs","Darling/Darling.Tests/ServerPasswordFromRequestTests.cs","Darling/Darling.Tests/StoreConfigProviderConnectionMatchTests.cs","Darling/Darling.Tests/StoreServerPasswordRulesLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCommandExecutor.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSecretSource.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerAdminTools.Edit.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Storage/ServerConnectionRule.cs","Darling/PerformanceMonitor.Darling.Viewer/AddServerDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.MonitoredServers.cs","Darling/tools/provision-roles.sql"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-10-06","head_sha":"c62b5e8f3123cf73233e87e70c4a7944610eedf9","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedRolesTests.cs","Darling/Darling.Tests/DarlingSecretsOwnedReferenceTests.cs","Darling/Darling.Tests/RegisteredServerSettingDriftTests.cs","Darling/Darling.Tests/SelfManagedStoreRulesStartPinTests.cs","Darling/Darling.Tests/ServerEditCoreTests.cs","Darling/Darling.Tests/ServerEditLiveTests.cs","Darling/Darling.Tests/ServerEditPasswordRuleViewerTests.cs","Darling/Darling.Tests/ServerIdentityLockLiveTests.cs","Darling/Darling.Tests/ServerPasswordFromRequestTests.cs","Darling/Darling.Tests/StoreConfigProviderConnectionMatchTests.cs","Darling/Darling.Tests/StoreServerPasswordRulesLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCommandExecutor.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSecretSource.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerAdminTools.Edit.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Storage/ServerConnectionRule.cs","Darling/PerformanceMonitor.Darling.Viewer/AddServerDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.MonitoredServers.cs","Darling/tools/provision-roles.sql"],"failed_classes":["Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-10-06","head_sha":"7d993a7f07e13a999df9556d9e4da5c097585ce2","event":"pull_request","changed_files":["Darling/Darling.Tests/ComposeCatalogStatementTextCensusTests.cs","Darling/Darling.Tests/DiagnosticsBundleServiceLogTests.cs","Darling/Darling.Tests/DiagnosticsBundleStatementFilterTests.cs","Darling/Darling.Tests/SensitiveStatementHostFilterTests.cs","Darling/Darling.Tests/StatementColumnCensusTests.cs","Darling/Darling.Tests/StatementFilterAnalysisReadsLiveTests.cs","Darling/Darling.Tests/StatementFilterBlockingReadsLiveTests.cs","Darling/Darling.Tests/StatementFilterCensus.cs","Darling/Darling.Tests/StatementFilterDrillDownWarningsLiveTests.cs","Darling/Darling.Tests/StatementFilterPlanReaderSourceScanTests.cs","Darling/Darling.Tests/StatementFilterPlanReadsLiveTests.cs","Darling/Darling.Tests/StatementFilterQueryReadsLiveTests.cs","Darling/Darling.Tests/StatementFilterWebTests.cs","Darling/Darling.Tests/StoredPlanAnalysisFilterTests.cs","Darling/Darling.Tests/ViewerStoreLoginRawReadLiveTests.cs","Darling/Darling.Tests/WebRouteStatementCensusTests.cs","Darling/Darling.Tests/web-plan-viewer-harness.mjs","Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.Plans.cs","Darling/PerformanceMonitor.Darling.Service/AlertNotebookEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingFleetSweepEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebDeadlockGraph.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebStatementSweep.cs","Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundleAliaser.cs","Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundleServiceLog.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStoredPlanReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/GcfCallToolFilter.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/GcfOutput.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/IGcfOutputFormat.cs","Darling/PerformanceMonitor.Darling.Service/SlowReadLog.cs","Lite.Tests/StatementFilterDrillDownWarningsTests.cs","Lite.Tests/StoredPlanAnalysisFilterTests.cs","Lite/Analysis/DrillDownCollector.Plans.cs"],"failed_classes":["Darling.Tests.PlanRegressionReadShapeTests","Darling.Tests.StatementFilterBlockingReadsLiveTests","Darling.Tests.TestsNoSilentPassGuardTests"],"failed_jobs":["Darling PG tests (0)","build","Darling PG tests (2)"]} +{"date":"2026-10-06","head_sha":"8f95de197e0d1d82b2454828d8161e7d5f2ce4e0","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingManagedRolesTests.cs","Darling/Darling.Tests/DarlingSecretsOwnedReferenceTests.cs","Darling/Darling.Tests/PasswordBindingCensusTests.cs","Darling/Darling.Tests/RegisteredServerSettingDriftTests.cs","Darling/Darling.Tests/RepoFileAdoptionTests.cs","Darling/Darling.Tests/SelfManagedStoreRulesStartPinTests.cs","Darling/Darling.Tests/ServerEditCoreTests.cs","Darling/Darling.Tests/ServerEditLiveTests.cs","Darling/Darling.Tests/ServerEditPasswordRuleViewerTests.cs","Darling/Darling.Tests/ServerIdentityLockLiveTests.cs","Darling/Darling.Tests/ServerPasswordFromRequestTests.cs","Darling/Darling.Tests/StoreConfigProviderConnectionMatchTests.cs","Darling/Darling.Tests/StoreServerPasswordRulesLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCommandExecutor.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSecretSource.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerAdminTools.Edit.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Storage/ServerConnectionRule.cs","Darling/PerformanceMonitor.Darling.Viewer/AddServerDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.MonitoredServers.cs","Darling/tools/provision-roles.sql"],"failed_classes":["Darling.Tests.StatementFilterAlertTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-10-06","head_sha":"7fc8b09c96fb66c13665a2a0e28c47060475ea09","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingRuntimePreflightTests.cs","Darling/README.md","Darling/tools/install-darling.ps1","docs/uat-onboarding.md"],"failed_classes":["PerformanceMonitor.Darling.Tests.RetiredBaselineAggregateLiveTests"],"failed_jobs":["Darling PG tests (2)"]} +{"date":"2026-10-06","head_sha":"2000eb7f3114ebf436f939145a87fa2191012332","event":"pull_request","changed_files":["Lite.Tests/QueryStoreDedupReadTests.cs","Lite.Tests/QueryStoreOldReadSql.cs","Lite.Tests/QueryStoreTextAfterRankingTests.cs","Lite.Tests/TopListTotalOrderPinTests.cs","Lite/Services/LocalDataService.QueryStore.cs","Lite/Services/LocalDataService.QueryStoreRegressions.cs","Lite/Services/LocalDataService.QueryStoreText.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","PerformanceMonitorLite.Tests.DuckDbInitializerDisposalGuardTests"],"failed_jobs":["build","Lite tests (3)"]} +{"date":"2026-10-06","head_sha":"8c842bbc3f7d574d1a0b2699388dbd253c85ccdb","event":"pull_request","changed_files":["Lite.Tests/QueryStoreDedupReadTests.cs","Lite.Tests/QueryStoreOldReadSql.cs","Lite.Tests/QueryStoreTextAfterRankingTests.cs","Lite.Tests/TopListTotalOrderPinTests.cs","Lite/Services/LocalDataService.QueryStore.cs","Lite/Services/LocalDataService.QueryStoreRegressions.cs","Lite/Services/LocalDataService.QueryStoreText.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests","PerformanceMonitorLite.Tests.DuckDbInitializerDisposalGuardTests"],"failed_jobs":["Lite tests (1)","build"]} +{"date":"2026-10-06","head_sha":"38445f2ebac66381fa45de8a9665ee710848772a","event":"pull_request","changed_files":["Lite.Tests/ArchiveCopyOptionsPinTests.cs","Lite.Tests/ArchiveTieOrderTests.cs","Lite.Tests/MainConnectionLimitsTests.cs","Lite.Tests/QueryStoreSliceRepairTests.cs","Lite/Analysis/DrillDownCollector.Queries.cs","Lite/Analysis/DuckDbFactCollector.QueryPerf.cs","Lite/Database/DuckDbInitializer.cs","Lite/Services/ArchiveService.cs","Lite/Services/DataImportService.cs","Lite/Services/ParquetCompaction.cs","Lite/Services/QueryStoreSliceRepairService.cs"],"failed_classes":["Darling.Tests.CommentFilterAdoptionTests"],"failed_jobs":["build"]} +{"date":"2026-10-06","head_sha":"358b0e5005a18ba8a1022dd7201f4acd3b6d1e7d","event":"pull_request","changed_files":["Lite.Tests/ArchiveCopyOptionsPinTests.cs","Lite.Tests/ArchiveTieOrderTests.cs","Lite.Tests/MainConnectionLimitsTests.cs","Lite.Tests/QueryStoreSliceRepairTests.cs","Lite/Analysis/DrillDownCollector.Queries.cs","Lite/Analysis/DuckDbFactCollector.QueryPerf.cs","Lite/Database/DuckDbInitializer.cs","Lite/Services/ArchiveService.cs","Lite/Services/DataImportService.cs","Lite/Services/ParquetCompaction.cs","Lite/Services/QueryStoreSliceRepairService.cs"],"failed_classes":["Darling.Tests.CommentFilterAdoptionTests"],"failed_jobs":["build"]} +{"date":"2026-10-06","head_sha":"b1a5999ded26ecac4045cf86957f8ec1638aa2b6","event":"pull_request","changed_files":["Lite.Tests/CollectionHealthKeyedLookupParityTests.cs","Lite.Tests/CollectionHealthOracleSql.cs","Lite.Tests/CollectionHealthReadBehaviourTests.cs","Lite.Tests/CollectorHealthClassifierTests.cs","Lite.Tests/DataStartBannerProbeFailureTests.cs","Lite.Tests/DataStartBannerQueriesTabTests.cs","Lite.Tests/DataStartBannerSurfaceTests.cs","Lite.Tests/EmptyEnumerationNoteTests.cs","Lite.Tests/RegressedFromProductiveTests.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Database/DuckDbInitializer.cs","Lite/Helpers/MethodProfiler.cs","Lite/Services/LocalDataService.CollectionHealth.cs","Lite/Services/LocalDataService.QueryWindowFloor.cs","Lite/Services/LocalDataService.cs","Lite/Services/ReadPhaseTimer.cs"],"failed_classes":["Darling.Tests.CollectionOutputBesideCostTests"],"failed_jobs":["build"]} +{"date":"2026-10-06","head_sha":"d9274d2e430aa20a386bcf655816ddb037bcb3f6","event":"pull_request","changed_files":["Lite.Tests/CollectionHealthKeyedLookupParityTests.cs","Lite.Tests/CollectionHealthOracleSql.cs","Lite.Tests/CollectionHealthReadBehaviourTests.cs","Lite.Tests/CollectorHealthClassifierTests.cs","Lite.Tests/DataStartBannerProbeFailureTests.cs","Lite.Tests/DataStartBannerQueriesTabTests.cs","Lite.Tests/DataStartBannerSurfaceTests.cs","Lite.Tests/EmptyEnumerationNoteTests.cs","Lite.Tests/RegressedFromProductiveTests.cs","Lite/Controls/ServerTab.Refresh.cs","Lite/Database/DuckDbInitializer.cs","Lite/Helpers/MethodProfiler.cs","Lite/Services/LocalDataService.CollectionHealth.cs","Lite/Services/LocalDataService.QueryWindowFloor.cs","Lite/Services/LocalDataService.cs","Lite/Services/ReadPhaseTimer.cs"],"failed_classes":["Darling.Tests.CollectionOutputBesideCostTests"],"failed_jobs":["build"]} +{"date":"2026-10-06","head_sha":"bf8c2cb3eee49aac3e8570a59373b0550f71426e","event":"pull_request","changed_files":[".github/workflows/build.yml","Darling/Darling.Tests/DarlingAlertingTests.cs","Darling/Darling.Tests/DarlingCliSealKeyTests.cs","Darling/Darling.Tests/DarlingCliUnusableStoreConnectionTests.cs","Darling/Darling.Tests/DarlingConfigTests.cs","Darling/Darling.Tests/DarlingMcpServerAdminToolsTests.cs","Darling/Darling.Tests/DarlingPasswordKeyRuntimeTests.cs","Darling/Darling.Tests/DarlingPasswordKeyStoreLiveTests.cs","Darling/Darling.Tests/DarlingSecretsDecryptFailureTests.cs","Darling/Darling.Tests/DarlingSecretsOwnedReferenceTests.cs","Darling/Darling.Tests/DarlingWebhookSecretsTests.cs","Darling/Darling.Tests/DiagnosticsBundleSealedTextTests.cs","Darling/Darling.Tests/DiagnosticsBundleWebhookOpenTests.cs","Darling/Darling.Tests/Fixtures/PasswordSeal/linux-self-check-vector.json","Darling/Darling.Tests/GenericWebhookHeadersOriginTests.cs","Darling/Darling.Tests/LegacyPinResolverLiveTests.cs","Darling/Darling.Tests/NotificationRoutesRungTests.cs","Darling/Darling.Tests/OwnedSecretReferenceRefusalTests.cs","Darling/Darling.Tests/PasswordBindingCensusTests.cs","Darling/Darling.Tests/PasswordKeyGrantParityTests.cs","Darling/Darling.Tests/PasswordKeyRungTests.cs","Darling/Darling.Tests/PasswordKeySeamLiveTests.cs","Darling/Darling.Tests/PasswordKeySelfCheckTests.cs","Darling/Darling.Tests/PasswordKeyTablesLiveTests.cs","Darling/Darling.Tests/QueryStatsHourLedgerTests.cs","Darling/Darling.Tests/SealedSecretResolverTests.cs","Darling/Darling.Tests/ServerAddViewerRoleLiveTests.cs","Darling/Darling.Tests/ServerEditCoreTests.cs","Darling/Darling.Tests/ServerEditLiveTests.cs","Darling/Darling.Tests/ServerEditPasswordRuleViewerTests.cs","Darling/Darling.Tests/ServerEditViewerRoleLiveTests.cs","Darling/Darling.Tests/ServerIdentityLockLiveTests.cs","Darling/Darling.Tests/ServerPasswordFromRequestTests.cs","Darling/Darling.Tests/StoreConfigProviderConnectionMatchTests.cs","Darling/Darling.Tests/StoreIsOnThisMachineTests.cs","Darling/Darling.Tests/TestKeyRings.cs","Darling/Darling.Tests/ValidateConfigOldFormatPasswordLiveTests.cs","Darling/Darling.Tests/ViewerControlPlaneStage3bTests.cs","Darling/Darling.Tests/ViewerPasswordKeyTests.cs","Darling/Darling.Tests/ViewerServerManagementTests.cs","Darling/Darling.Tests/ViewerServerSealLiveTests.cs","Darling/Darling.Tests/ViewerSmtpSealTests.cs","Darling/Darling.Tests/ViewerWebhookSealLiveTests.cs","Darling/Darling.Tests/ViewerWebhookSealingTests.cs","Darling/Darling.Tests/WebhookDeliverySnapshotTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertSettings.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliSealKey.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCommandExecutor.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/DarlingPasswordKeyRuntime.cs","Darling/PerformanceMonitor.Darling.Service/DarlingPasswordKeySelfCheck.cs","Darling/PerformanceMonitor.Darling.Service/DarlingPasswordKeyStore.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSecrets.cs","Darling/PerformanceMonitor.Darling.Service/DarlingServerConnector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebhookSecrets.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundle.cs","Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundleRunner.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerAdminTools.Edit.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerAdminTools.cs","Darling/PerformanceMonitor.Darling.Service/Program.cs","Darling/PerformanceMonitor.Darling.Service/SecretTextGuard.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Storage/PasswordBinding.cs","Darling/PerformanceMonitor.Darling.Storage/PasswordKeyTables.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/ServerConnectionRule.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/AddMultipleServersDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/AddServerDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/MainWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/NotificationRouteEditDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/NotificationRoutesWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/PasswordKeyChangedDialog.xaml","Darling/PerformanceMonitor.Darling.Viewer/PasswordKeyChangedDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerControlPlaneMigration.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Commands.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.MonitoredServers.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Notification.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.NotificationRoutes.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.PasswordKey.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerPasswordKey.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerPasswordKeyPins.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerMigration.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerSecret.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerSmtpSeal.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerWebhookSealing.cs","Darling/README.md","Darling/compose/docker-compose.yml","Darling/compose/secrets/README.md","Darling/tools/provision-roles.sql","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","Lite.Tests/GenericWebhookHeadersOriginTests.cs","PerformanceMonitor.Notifications/IAlertSettings.cs","PerformanceMonitor.Notifications/NotificationRoute.cs","PerformanceMonitor.Notifications/NotificationRouter.cs","PerformanceMonitor.Notifications/WebhookAlertService.cs"],"failed_classes":["Darling.Tests.CaggGroupIndexDropLiveTests","Darling.Tests.CollectorRunTimeRungLiveTests","Darling.Tests.DarlingMcpServerAdminToolsLivePostgresTests","Darling.Tests.DocCommentHygieneTests","Darling.Tests.RepoFileAdoptionTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (0)","Darling PG tests (2)","Darling whole-tree guards"]} +{"date":"2026-10-06","head_sha":"87ee3cc71d3f0f725534d5ad638d030489dfa18d","event":"pull_request","changed_files":[".github/workflows/build.yml","Darling/Darling.Tests/DarlingAlertingTests.cs","Darling/Darling.Tests/DarlingCliSealKeyTests.cs","Darling/Darling.Tests/DarlingCliUnusableStoreConnectionTests.cs","Darling/Darling.Tests/DarlingConfigTests.cs","Darling/Darling.Tests/DarlingMcpServerAdminToolsTests.cs","Darling/Darling.Tests/DarlingPasswordKeyRuntimeTests.cs","Darling/Darling.Tests/DarlingPasswordKeyStoreLiveTests.cs","Darling/Darling.Tests/DarlingSecretsDecryptFailureTests.cs","Darling/Darling.Tests/DarlingSecretsOwnedReferenceTests.cs","Darling/Darling.Tests/DarlingWebhookSecretsTests.cs","Darling/Darling.Tests/DiagnosticsBundleSealedTextTests.cs","Darling/Darling.Tests/DiagnosticsBundleWebhookOpenTests.cs","Darling/Darling.Tests/Fixtures/PasswordSeal/linux-self-check-vector.json","Darling/Darling.Tests/GenericWebhookHeadersOriginTests.cs","Darling/Darling.Tests/LegacyPinResolverLiveTests.cs","Darling/Darling.Tests/NotificationRoutesRungTests.cs","Darling/Darling.Tests/OwnedSecretReferenceRefusalTests.cs","Darling/Darling.Tests/PasswordBindingCensusTests.cs","Darling/Darling.Tests/PasswordKeyGrantParityTests.cs","Darling/Darling.Tests/PasswordKeyRungTests.cs","Darling/Darling.Tests/PasswordKeySeamLiveTests.cs","Darling/Darling.Tests/PasswordKeySelfCheckTests.cs","Darling/Darling.Tests/PasswordKeyTablesLiveTests.cs","Darling/Darling.Tests/QueryStatsHourLedgerTests.cs","Darling/Darling.Tests/SealedSecretResolverTests.cs","Darling/Darling.Tests/ServerAddViewerRoleLiveTests.cs","Darling/Darling.Tests/ServerEditCoreTests.cs","Darling/Darling.Tests/ServerEditLiveTests.cs","Darling/Darling.Tests/ServerEditPasswordRuleViewerTests.cs","Darling/Darling.Tests/ServerEditViewerRoleLiveTests.cs","Darling/Darling.Tests/ServerIdentityLockLiveTests.cs","Darling/Darling.Tests/ServerPasswordFromRequestTests.cs","Darling/Darling.Tests/StoreConfigProviderConnectionMatchTests.cs","Darling/Darling.Tests/StoreIsOnThisMachineTests.cs","Darling/Darling.Tests/TestKeyRings.cs","Darling/Darling.Tests/ValidateConfigOldFormatPasswordLiveTests.cs","Darling/Darling.Tests/ViewerControlPlaneStage3bTests.cs","Darling/Darling.Tests/ViewerPasswordKeyTests.cs","Darling/Darling.Tests/ViewerServerManagementTests.cs","Darling/Darling.Tests/ViewerServerSealLiveTests.cs","Darling/Darling.Tests/ViewerSmtpSealTests.cs","Darling/Darling.Tests/ViewerWebhookSealLiveTests.cs","Darling/Darling.Tests/ViewerWebhookSealingTests.cs","Darling/Darling.Tests/WebhookDeliverySnapshotTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertSettings.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliCommands.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCliSealKey.cs","Darling/PerformanceMonitor.Darling.Service/DarlingCommandExecutor.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/DarlingPasswordKeyRuntime.cs","Darling/PerformanceMonitor.Darling.Service/DarlingPasswordKeySelfCheck.cs","Darling/PerformanceMonitor.Darling.Service/DarlingPasswordKeyStore.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSecrets.cs","Darling/PerformanceMonitor.Darling.Service/DarlingServerConnector.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebhookSecrets.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundle.cs","Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundleRunner.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerAdminTools.Edit.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpServerAdminTools.cs","Darling/PerformanceMonitor.Darling.Service/Program.cs","Darling/PerformanceMonitor.Darling.Service/SecretTextGuard.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Storage/PasswordBinding.cs","Darling/PerformanceMonitor.Darling.Storage/PasswordKeyTables.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/ServerConnectionRule.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/AddMultipleServersDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/AddServerDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/MainWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/NotificationRouteEditDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/NotificationRoutesWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/PasswordKeyChangedDialog.xaml","Darling/PerformanceMonitor.Darling.Viewer/PasswordKeyChangedDialog.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerControlPlaneMigration.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Commands.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.MonitoredServers.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Notification.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.NotificationRoutes.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.PasswordKey.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerPasswordKey.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerPasswordKeyPins.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerMigration.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerSecret.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerSmtpSeal.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerWebhookSealing.cs","Darling/README.md","Darling/compose/docker-compose.yml","Darling/compose/secrets/README.md","Darling/tools/provision-roles.sql","Lite.Tests/CrossAppMcpToolInventoryPinTests.cs","Lite.Tests/GenericWebhookHeadersOriginTests.cs","PerformanceMonitor.Notifications/IAlertSettings.cs","PerformanceMonitor.Notifications/NotificationRoute.cs","PerformanceMonitor.Notifications/NotificationRouter.cs"],"failed_classes":["Darling.Tests.CaggGroupIndexDropLiveTests","Darling.Tests.CollectorRunTimeRungLiveTests","Darling.Tests.DarlingMcpServerAdminToolsLivePostgresTests","Darling.Tests.DocCommentHygieneTests","Darling.Tests.RepoFileAdoptionTests","Darling.Tests.StatementFilterPlanReadsLiveTests"],"failed_jobs":["build","Darling PG tests (1)","Darling PG tests (2)","Darling PG tests (0)","Darling whole-tree guards"]} +{"date":"2026-10-06","head_sha":"29d422f582b2502ccc9fab3afb507d6060aa7d31","event":"pull_request","changed_files":[".github/workflows/build.yml"],"failed_classes":["Darling.Tests.TrendPayloadBudgetLiveTests"],"failed_jobs":["Darling PG tests (1)"]} +{"date":"2026-10-06","head_sha":"91536b53039d311bbe8fdc5a46fbf09d3b90cdd5","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingMcpPlanCorrectionToolsTests.cs","Darling/Darling.Tests/DarlingMcpTrendToolsTests.cs","Darling/Darling.Tests/DarlingTrendEmptyTests.cs","Darling/Darling.Tests/DatabaseFilterEmptyAnswerLiveTests.cs","Darling/Darling.Tests/DeltaFamilyIntervalCompletionLivePostgresTests.cs","Darling/Darling.Tests/DurationTrendDatabaseFilterLiveTests.cs","Darling/Darling.Tests/EmptyAnswerDatabaseEchoTests.cs","Darling/Darling.Tests/EventWindowedReadsAreBoundedLivePostgresTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpLongQueryTools.txt","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpPlanCorrectionTools.txt","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpQueryStoreClutterTools.txt","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpTrendTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/McpZeroIsAMeasurementTests.cs","Darling/Darling.Tests/PlanCorrectionFrameLiveTests.cs","Darling/Darling.Tests/PlanCorrectionsWebDefaultTests.cs","Darling/Darling.Tests/QueryEventsDatabaseFilterLiveTests.cs","Darling/Darling.Tests/QueryReadsWebDispatchLiveTests.cs","Darling/Darling.Tests/QueryStoreClutterLivePostgresTests.cs","Darling/Darling.Tests/QueryStoreClutterTests.cs","Darling/Darling.Tests/QueryStoreTrendIntervalRateTests.cs","Darling/Darling.Tests/QueryStoreTrendRoutingTests.cs","Darling/Darling.Tests/ViewerBlockingSourceLabelTests.cs","Darling/Darling.Tests/ViewerBlockingSourceLivePostgresTests.cs","Darling/Darling.Tests/ViewerQueriesRestTests.cs","Darling/Darling.Tests/ViewerQueryTrendBucketingReviewTests.cs","Darling/Darling.Tests/ViewerTrendRoutingPortTests.cs","Darling/Darling.Tests/WebBlockingSourceBehaviourTests.cs","Darling/Darling.Tests/WebDataStartNoteTests.cs","Darling/Darling.Tests/WebDatabaseFilterChipsBehaviourTests.cs","Darling/Darling.Tests/web-blocking-source-harness.mjs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingLongQueryReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpLongQueryTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpPlanCorrectionTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpQueryStoreClutterTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpTrendTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingPlanCorrectionReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingTrendReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/McpCommandDeadlines.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/css/app.css","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/database-filter-reads.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js","Darling/PerformanceMonitor.Darling.Storage/DarlingQueryStoreClutterReader.cs","Darling/PerformanceMonitor.Darling.Storage/DurationTrendRouting.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.BlockingStats.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.BlockingTrends.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStoreClutter.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryTrends.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.Blocking.cs","Lite.Tests/BlockingSourceLabelTests.cs","Lite.Tests/DurationTrendDatabaseFilterToolTests.cs","Lite.Tests/DurationTrendFilteredDenominatorTests.cs","Lite.Tests/EventListDatabaseFilterToolTests.cs","Lite.Tests/McpToolsListBudget/McpLongQueryTools.txt","Lite.Tests/McpToolsListBudget/McpPlanCorrectionTools.txt","Lite.Tests/McpToolsListBudget/McpQueryTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite.Tests/PerformanceTrendDurationBucketingTests.cs","Lite/Controls/ServerTab.BlockingStats.cs","Lite/Controls/ServerTab.Charts.cs","Lite/Mcp/McpLongQueryTools.cs","Lite/Mcp/McpPlanCorrectionTools.cs","Lite/Mcp/McpQueryTools.cs","Lite/Services/LocalDataService.DatabaseFilter.cs","Lite/Services/LocalDataService.LongQueries.cs","Lite/Services/LocalDataService.QueryStats.cs","Lite/Services/LocalDataService.TrendBuckets.cs","PerformanceMonitor.Common/Mcp/McpHelpers.cs","PerformanceMonitor.Ui/BlockingSourceLabel.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["build","Darling PG tests (0)","Darling PG tests (1)"]} +{"date":"2026-10-06","head_sha":"a6cfe86fd69ac3131b6f6c4f1d9de768a1f4ed59","event":"pull_request","changed_files":["Darling/Darling.Tests/QueryStoreTopDailyReadLiveTests.cs","Darling/Darling.Tests/ViewerQueriesTests.cs","Darling/Darling.Tests/ViewerTextLookupWindowBoundLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs","Darling/PerformanceMonitor.Darling.Storage/PgTableTuning.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.ProcedureStats.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStats.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStore.cs"],"failed_classes":["Darling.Tests.DocCommentHygieneTests"],"failed_jobs":["build","Darling PG tests (1)"]} +{"date":"2026-10-07","head_sha":"64d26b5365e4d000fc6f577d9e61f1a10290f330","event":"pull_request","changed_files":["Darling/Darling.Tests/StoreObjectConvergenceTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs"],"failed_classes":["Darling.Tests.LivePostgresCollectionHygieneTests"],"failed_jobs":["build","Darling PG tests (2)"]} +{"date":"2026-10-07","head_sha":"f796927ad5ea5cb0100325a46599aa24e9853dac","event":"pull_request","changed_files":["Darling/Darling.Tests/CollectionGapAtStartTests.cs","Darling/Darling.Tests/DarlingTriageEndpointTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/SelfAlertFailedSendCensusTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Lite.Tests/AlertSeverityTests.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs","PerformanceMonitor.Notifications/AlertSeverity.cs"],"failed_classes":["Darling.Tests.AlertReadFailureSurfaceTests","Darling.Tests.CollectionGapAtStartTests"],"failed_jobs":["build","Darling PG tests (4)","Darling PG tests (5)"]} +{"date":"2026-10-07","head_sha":"02ae3efb1f409e38b77d30942a23a17073e6b708","event":"pull_request","changed_files":["Darling/Darling.Tests/DarlingPasswordKeyRuntimeTests.cs","Darling/Darling.Tests/DarlingPasswordKeyStoreLiveTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs"],"failed_classes":["Darling.Tests.StatementFilterAlertTests"],"failed_jobs":["Darling PG tests (4)"]} +{"date":"2026-10-07","head_sha":"fb215e45381ce3378433dcf434a04f9533068416","event":"push","changed_files":["Darling/Darling.Tests/AlertDeliveryChannelTests.cs","Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs","Darling/Darling.Tests/DarlingSelfAlertTests.cs","Darling/Darling.Tests/IncidentDeliveryFilterTests.cs","Darling/Darling.Tests/PagerDutyAutoResolveRungTests.cs","Darling/Darling.Tests/PasswordKeyRungTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingAlertSettings.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWebhookSecrets.cs","Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs","Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs","Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml","Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Notification.cs","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs","Darling/tools/provision-roles.sql","Lite.Tests/AgAlertEvaluatorTests.cs","Lite.Tests/AlertIncidentRenderTests.cs","Lite.Tests/IncidentCooldownTests.cs","Lite.Tests/PagerDutyWebhookTests.cs","Lite.Tests/PerEventNotificationTests.cs","Lite/App.xaml.cs","Lite/Services/AgAlertEvaluator.cs","Lite/Services/AppAlertSettings.cs","Lite/Windows/SettingsWindow.xaml","Lite/Windows/SettingsWindow.xaml.cs","Lite/config/settings.sample.json","PerformanceMonitor.Notifications/AgAlertContexts.cs","PerformanceMonitor.Notifications/AlertContext.cs","PerformanceMonitor.Notifications/AlertFamily.cs","PerformanceMonitor.Notifications/EmailSendCore.cs","PerformanceMonitor.Notifications/IAlertSettings.cs","PerformanceMonitor.Notifications/IncidentCooldown.cs","PerformanceMonitor.Notifications/IncidentDeliveryFilter.cs","PerformanceMonitor.Notifications/PerEventNotification.cs","PerformanceMonitor.Notifications/WebhookAlertService.cs"],"failed_classes":["PerformanceMonitorLite.Tests.BlockingDatabaseFilterToolTests"],"failed_jobs":["Lite tests (2)"]} +{"date":"2026-10-07","head_sha":"0f92fa3b30bd7349208a84cf2649d4dd9aee981d","event":"push","changed_files":["Darling/Darling.Tests/DarlingHeartbeatTests.cs","Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs","Darling/PerformanceMonitor.Darling.Service/DarlingHeartbeat.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/DiagnosticsBundle.cs","Darling/PerformanceMonitor.Darling.Service/darling.sample.json","Darling/README.md"],"failed_classes":["Darling.Tests.QueryStoreTopLiteralEndStraddleLiveTests"],"failed_jobs":["Darling PG tests (3)"]} +{"date":"2026-10-07","head_sha":"61afbdfd924262bba990d222be66b2146164c8c0","event":"pull_request","changed_files":["Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs","Darling/Darling.Tests/CollectionHistoryAuditTests.cs","Darling/Darling.Tests/DarlingTriageEndpointTests.cs","Darling/Darling.Tests/NotificationRoutingTests.cs","Darling/Darling.Tests/SelfAlertFailedSendCensusTests.cs","Darling/PerformanceMonitor.Darling.Service/CollectionHistoryAudit.cs","Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs","Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs","Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs","Darling/PerformanceMonitor.Darling.Service/SelfAlertDeliveryStampStore.cs","Darling/README.md","Lite.Tests/AlertSeverityTests.cs","PerformanceMonitor.Common/AlertMetricClassifier.cs","PerformanceMonitor.Notifications/AlertFamily.cs","PerformanceMonitor.Notifications/AlertSeverity.cs"],"failed_classes":["Darling.Tests.AlertMasterSwitchSurfaceTests","Lite.Tests.CrossSkuSurfaceSourceTests"],"failed_jobs":["build","Lite tests (2)","Darling PG tests (1)","Darling PG tests (4)"]} +{"date":"2026-10-07","head_sha":"ff2bd1569e4c8da726367c34525d9ac0c0d1aaff","event":"push","changed_files":["Darling/Darling.Tests/ConfigDatabaseFilterLiveTests.cs","Darling/Darling.Tests/DarlingMcpConfigHistoryToolsTests.cs","Darling/Darling.Tests/DarlingMcpDefaultTraceToolsTests.cs","Darling/Darling.Tests/DarlingMcpHealthParserToolsTests.cs","Darling/Darling.Tests/DatabaseConfigWebColumnsTests.cs","Darling/Darling.Tests/DatabaseNameHistoryTests.cs","Darling/Darling.Tests/EventsDatabaseFilterLiveTests.cs","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpConfigHistoryTools.txt","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpDefaultTraceTools.txt","Darling/Darling.Tests/McpToolsListBudget/DarlingMcpHealthParserTools.txt","Darling/Darling.Tests/McpToolsListBudgetTests.cs","Darling/Darling.Tests/WebDatabaseFilterChipsBehaviourTests.cs","Darling/Darling.Tests/WebDatabaseFilterReadsTests.cs","Darling/Darling.Tests/WebReadProbe.cs","Darling/Darling.Tests/web-database-filter-chips-harness.mjs","Darling/PerformanceMonitor.Darling.Service/DarlingWebEndpoints.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingConfigHistoryReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingCurrentConfigReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDefaultTraceReader.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpConfigHistoryTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpConfigTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDefaultTraceTools.cs","Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpHealthParserTools.cs","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/database-filter-reads.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/pages/server-tabs.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/panels.js","Darling/PerformanceMonitor.Darling.Service/wwwroot/js/util.js","Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.SystemEvents.cs","Lite.Tests/McpDatabaseNameFilterToolTests.cs","Lite.Tests/McpToolsListBudget/McpConfigHistoryTools.txt","Lite.Tests/McpToolsListBudget/McpDefaultTraceTools.txt","Lite.Tests/McpToolsListBudget/McpHealthParserTools.txt","Lite.Tests/McpToolsListBudgetTests.cs","Lite.Tests/SystemEventsReaderTests.cs","Lite/Mcp/McpConfigHistoryTools.cs","Lite/Mcp/McpConfigTools.cs","Lite/Mcp/McpDefaultTraceTools.cs","Lite/Mcp/McpHealthParserTools.cs","Lite/Services/LocalDataService.SystemEvents.cs","PerformanceMonitor.Common/DatabaseNameHistory.cs"],"failed_classes":["Darling.Tests.StatementFilterBlockingReadsLiveTests"],"failed_jobs":["Darling PG tests (2)"]} diff --git a/.github/ci-history/replay-accepted.txt b/.github/ci-history/replay-accepted.txt new file mode 100644 index 0000000000..412eee2bef --- /dev/null +++ b/.github/ci-history/replay-accepted.txt @@ -0,0 +1,129 @@ +# Corpus rows today's rules do not select a failing class for, and why that is not a selection gap. +# A miss not listed here fails ci-select.py --replay. Format: <10-char head sha> . + +# A Darling class failed on a run whose changed files (as last recorded for the pull request) are +# documentation only. The failing attempt ran on an earlier head of the pull request that held code, so the +# recorded list is not the list that run saw. +cc29d98125 Darling.Tests.RepoFileAdoptionTests +7b64070c98 Darling.Tests.DarlingDeployRollbackRetentionTests +e6dbdbea94 Darling.Tests.IncidentAttachmentDeliveryTests + +# A Lite class failed on a Darling-only diff, before the Reads=Darling selection (build.yml, 2026-10-04) +# existed or while the pull request still carried the older workflow. DarlingReadsTraitGuardTests derives the +# Reads=Darling tag from the class source, so a class without it reads no Darling file: the diff did not cause +# the failure. +d0d482835c PerformanceMonitorLite.Tests.PerformanceCalendarDataTests +cd34bbcb31 PerformanceMonitorLite.Tests.StatusBarSizeReadLockTests +1792370ebd PerformanceMonitorLite.Tests.McpPageContractTests +32239fd349 PerformanceMonitorLite.Tests.MemoryTrendGrantJoinToolTests +ad89ad83a7 PerformanceMonitorLite.Tests.LocalClockBucketKeyTests +08ca596e65 PerformanceMonitorLite.Tests.LocalClockBucketKeyTests +d064b8ebc5 PerformanceMonitorLite.Tests.LocalClockBucketKeyTests +1e9c8ce764 PerformanceMonitorLite.Tests.LocalClockBucketKeyTests +95a42dab2d PerformanceMonitorLite.Tests.LocalClockBucketKeyTests +5219f29479 PerformanceMonitorLite.Tests.AnomalyDetectorTileBehaviourTests +e4cb4a3699 PerformanceMonitorLite.Tests.AnomalyDetectorTileBehaviourTests +1c726d7f67 PerformanceMonitorLite.Tests.AnomalyDetectorTileBehaviourTests +c0adc53226 PerformanceMonitorLite.Tests.ScenarioTests +68ddf17466 PerformanceMonitorLite.Tests.ScenarioTests +a75333e8e2 PerformanceMonitorLite.Tests.ScenarioTests +801cb7c1d8 PerformanceMonitorLite.Tests.ScenarioTests +63ba0d2893 PerformanceMonitorLite.Tests.DuckDbSentinelConnectionTests +411d87afbe PerformanceMonitorLite.Tests.FinOpsTests +eca647c2f4 PerformanceMonitorLite.Tests.QueryStoreRegressionsToolTests +eca647c2f4 PerformanceMonitorLite.Tests.SignificantWaitsToolTests +# A Darling class failed on a pull request whose recorded changes are Lite files only (cut 3a/3c of #5459). The +# class source names no Lite path (LiteReadsTraitGuardTests derives the Reads=Lite tag from it, and these carry +# none; the one 'Lite' in PlanSync4512CancellationTests is a doc comment), so no Lite edit can reach what it +# tests: each is a timing or live-store test that failed on a first attempt, not a selection gap. +c3a18e1785 Darling.Tests.PgTargetWaitTests +67bf55c45a Darling.Tests.ClipboardTextRetryBudgetTests +48216212d0 Darling.Tests.PlanSync4512CancellationTests + +# Map back-test (ci-select.py --replay --map, #5459): a class failed on a run whose changed files do not reach it, +# and the failure is not a selection gap. Each class was checked against the corpus: none of the run's changed +# product files is named by the class, none is covered by it, and the class also failed on runs with a different +# change set, on a push to dev, or as the only failing class of one shard job (a flake or an environment fault). +# The two mass failures of 2026-09-26 and 2026-09-27 (a store migration broke the shared live-store fixture) are not +# here: the map builder now links every class of a big xunit collection to what the collection's fixture ran. +02ae3efb1f Darling.Tests.StatementFilterAlertTests +03d9905807 Darling.Tests.PgServerLogTailRotationLiveTests +0654199a72 Darling.Tests.PgServerLogTailRotationLiveTests +0ac14fc78e Darling.Tests.DarlingAnomalyBaselineTests +0b92d8248f Darling.Tests.PgServerLogTailRotationLiveTests +0c58c0eb56 Darling.Tests.PgTargetSampledWaitLiveTests +0c983ef54d Darling.Tests.PgLogBurstAndRotationLiveTests +11729239b1 Darling.Tests.XminHorizonPersistenceReadTests +2c39e009ff Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests +33932296ca Darling.Tests.StoreStatementStatsLiveTests +3af3ade9ea Darling.Tests.StoreStatementStatsLiveTests +3b2d41baae Darling.Tests.QueryStoreIntervalWideGridLiveTests +3bb3ae8fe9 Darling.Tests.TrendPayloadBudgetLiveTests +3de285b652 Darling.Tests.DarlingAnomalyBaselineTests +40284b0061 Darling.Tests.PayloadDimensionLiveTests +4756659045 Darling.Tests.QueryStoreLegacyRowIndexLiveTests +4a3f52389e PerformanceMonitorLite.Tests.TimeHonestyRungReadTests +4f8499a944 PerformanceMonitorLite.Tests.TimeHonestyRungReadTests +6ac72f8605 Darling.Tests.StoreStatementStatsLiveTests +6de10f4dcb Darling.Tests.StoreSessionTimeZoneLiveTests +87836cc97e Darling.Tests.DarlingAnomalyBaselineTests +87e78aa474 PerformanceMonitorLite.Tests.DataStartBannerProbeFailureTests +88204f7cd2 Darling.Tests.RdsDeadlockIngestorTests +8875188337 Darling.Tests.DarlingAnomalyBaselineTests +89c271dd17 Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests +8a800ed6c6 PerformanceMonitorLite.Tests.LiteDeltaSeederTests +8b3c5767f0 Darling.Tests.SqlServerStoreTileBehaviourTests +8f95de197e Darling.Tests.StatementFilterAlertTests +93d34a73cd Darling.Tests.RollupBackfillLiveTests +99ad6aebf6 Darling.Tests.PlanSync4512CancellationTests +9be59f1a95 Darling.Tests.CollectionLogDrainForensicsStoreTests +a3b5749937 Darling.Tests.DarlingAnomalyBaselineTests +af31e98db0 Darling.Tests.StoreSessionTimeZoneLiveTests +b33399f219 Darling.Tests.DarlingCloudIdentityProbeTests +c20797c24a Darling.Tests.PgDeadlockNormalizationTests +c9877aad46 Darling.Tests.DarlingAnomalyBaselineTests +c9a1a81456 Darling.Tests.QueryStoreTopLiteralEndStraddleLiveTests +d0d482835c Darling.Tests.XminHorizonPersistenceReadTests +d0fb0ff183 Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests +d5370df470 Darling.Tests.DarlingAnomalyBaselineTests +d6c02bf28a Darling.Tests.DarlingStoreUpgradeRevertTests +ed114b5b17 Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests +f07bfc441f PerformanceMonitorLite.Tests.TrendEmptyParityToolTests + +# The changelog splice (release archive) run whose failing class is a live-database class that no changelog +# change can affect. Since #5614 the three splice files (CHANGELOG.md, docs/changelog/.md and +# tools/changelog/archive-census.txt) are documentation, so a splice takes the docs fast path and selects no +# class; the failure on that run was not caused by the splice. +e20c881360 Darling.Tests.MigrationUpgradeLadderLiveTests + +# Map replay against the first nightly map (#5459, run 37892110132): six misses left after the map's own links, each +# checked against the failing job's log (measured) and the corpus. None is a selection gap; none is listed to hide +# one. A sha here is the head of a pull request run, and the map, if it were live, would have given that run no +# reason to run the class because the class is unrelated to the diff. +# +# LiteServerTagsStoreTests (a304350fe3, measured): the diff is the Lite logger (AppLogger, App.xaml.cs, the +# collector runner) and the failure is a TimeoutException from LocalDataService.OpenWriteConnectionAsync, "Could not +# acquire database write lock", on the class's first write and then its second. Nothing in the diff opens or holds +# a store write lock. Every later head of the same pull request (9981020a8a, 54244d4282, f415eae60c) passed, with +# the logger files still in its diff, and no other row of the corpus has this class failing. A shared-fixture lock +# timeout on a loaded runner (the lock holder is inferred, not seen). +a304350fe3 Lite.Tests.LiteServerTagsStoreTests +# +# EventWindowedReadsAreBoundedLivePostgresTests (the four heads of one pull request on 2026-09-25, measured): the +# test had been failing on dev itself from the merge of #4235 (06:11Z) until its fix, #4275 (10:34Z): the corpus +# holds it failing on two dev pushes (f5e001fe65, 659e11a39b) and on nine other pull-request heads that day, all +# changing other things, nearly all with "Darling PG tests (0)" as the failed job. The four heads ran 09:33Z to +# 10:14Z, inside that window; the first one's log shows the plan test's own message ("expected the floor to exclude at +# least 2 of the 4 seeded old-day chunks"). The change the four heads carry (how the store host is set up) does not +# explain the failure: the failure predates it. +f1a1894fd8 Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests +69ce7ff7be Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests +4b3fe51a97 Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests +3747867b5a Darling.Tests.EventWindowedReadsAreBoundedLivePostgresTests +# +# StoreMetricsLatestSkipScanLiveTests (3bb3ae8fe9, measured): attempt 1 of the run failed in "Darling PG tests (2)" +# only, five classes within one second (06:52:04 to 06:52:05), with "57P03: the database system is in recovery +# mode" (the shard's PostgreSQL server had restarted after a backend crash) and the scratch-database drops retrying +# on the same error. Attempt 2 of the same head passed every shard. A first-attempt environment fault of the +# shard's server; the 28 changed files (viewer data-start work) did not cause it. +3bb3ae8fe9 Darling.Tests.StoreMetricsLatestSkipScanLiveTests diff --git a/.github/darling-paths-filter.yml b/.github/darling-paths-filter.yml index 4009f8a810..d2870c19e5 100644 --- a/.github/darling-paths-filter.yml +++ b/.github/darling-paths-filter.yml @@ -1,6 +1,8 @@ # The ONE Darling path gate. Both Darling jobs in .github/workflows/build.yml — `Darling PostgreSQL # tests` and `Darling Linux build` — read this file through dorny/paths-filter's documented -# `filters: ` form, so the two jobs cannot disagree about what reaches Darling. +# `filters: ` form, so the two jobs cannot disagree about what reaches Darling. The build +# job reads it a third time to learn whether the PostgreSQL shards run (#5459 cut 3b), and +# .github/scripts/ci-select.py reads it to replay the failure corpus. # # A FILE rather than a list per job (#3116). The PG job's own filter named `Darling/**` and the two # workflows and nothing else, while the Linux job in the same file already listed shared libraries; @@ -44,6 +46,15 @@ darling: # worker-sizing formula, so an edit to it has to reach the suite. The Linux job runs no tests and # carries this entry only by sharing the file; it costs that job one publish and one image build. - '.github/workflows/nightly.yml' + # #5616: the Darling PostgreSQL shard script and the Lite shard script are run by the shard jobs and + # parsed by Darling.Tests pins, so an edit to either has to reach the suite. + - '.github/scripts/run-darling-pg-shard.ps1' + - '.github/scripts/run-lite-shard.ps1' + # #5627: the Darling upgrade-shape workflow and its script. The workflow runs the install and upgrade scripts under + # Darling/tools (already covered above) over real releases, and Darling.Tests parses both files, so an edit to + # either has to reach the suite. + - '.github/workflows/darling-upgrade-shapes.yml' + - '.github/scripts/darling-upgrade-shapes.ps1' # Lite/** is deliberately ABSENT, though build.yml's own `darling` filter carries it. That entry # exists because Darling.Tests reads Lite source in its cross-app parity guards, and those guards run diff --git a/.github/scripts/altcover-probe.py b/.github/scripts/altcover-probe.py new file mode 100644 index 0000000000..9c7dab0e23 --- /dev/null +++ b/.github/scripts/altcover-probe.py @@ -0,0 +1,513 @@ +#!/usr/bin/env python3 +"""AltCover per-test coverage probe helper (#5459 change 4, slice 2). Standard library only. + +The nightly workflow's dispatch-only `altcover-probe` job uses three subcommands: + + select Pick test classes from a runner class listing (`-list classes/json`), spread over kinds that + come from the class's source text. Profile `darling`: plain unit classes, TestHost / web classes + (a TestServer / WebApplicationFactory mention) and live PostgreSQL classes (a DARLING_TEST_PG + mention). Profile `lite`: plain classes, DuckDB-backed classes and classes that start collectors + (a RemoteCollectorService mention). Stage=Guard classes are skipped because they read source text + and never run product code. Deterministic: the sorted candidates are sampled at even steps. + + unwrap-skips An instrumented run reports a dynamic skip as an AggregateException failure (see the function). + This rewrites the instrumented xunit report so those tests are skips again, but only for tests + the plain run also skipped. + + summarize Turn an AltCover OpenCover report (`--callContext=[Fact] --callContext=[Theory]`) into + (a) a summary of the probe: overhead, share of visits that carry a test context, holes, and + (b) the schema-1 `classes` section of the test map the plan describes: + {schema, tool, files:[universe], classes:{suite:{Class:{own, files:[ids], seconds}}}} + `own` is the repo-relative file that declares the class; `files` are indexes into the universe + of product source files the class's tests visited; `seconds` is the class's summed test time. + +Both subcommands only read files; nothing here talks to a server. +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import statistics +import sys +import xml.etree.ElementTree as ET + +SCHEMA = 1 +CLASS_DECL = re.compile(r"\bclass\s+([A-Za-z_][A-Za-z0-9_]*)") +WEB_MARKERS = ("WebApplicationFactory", "UseTestServer", "TestServer", "Microsoft.AspNetCore.TestHost") +LIVE_MARKER = "DARLING_TEST_PG" +RUNTIME_MARKER = "DARLING_TEST_PGRUNTIME" +LITE_COLLECTOR_MARKERS = ("RemoteCollectorService",) +LITE_DUCKDB_MARKERS = ("DuckDB", "DuckDb", "LocalDataService") +# Default per-kind sample counts (the rest of --total is plain). +PROFILE_WANTS = {"darling": {"web": 25, "live": 12}, "lite": {"collector": 8, "duckdb": 20}} +SKIP_TOKEN = "$XunitDynamicSkip$" +TEXT_READ = re.compile(r"File\.ReadAll|RepoRoot|ReadAllText|ReadLines\(") +REFLECTION = re.compile(r"GetCustomAttribute|GetMethod\(|GetTypes\(|GetProperties\(|GetFields\(|typeof\(") +GUARD_MARKER = re.compile(r"Trait\(\s*\"Stage\"\s*,\s*\"Guard\"\s*\)") + + +def norm(path: str) -> str: + return path.replace("\\", "/") + + +def simple_name(full: str) -> str: + return re.split(r"[.+/]", full)[-1] + + +def classify(text, profile): + if GUARD_MARKER.search(text): + return "guard" + if profile == "lite": + if any(marker in text for marker in LITE_COLLECTOR_MARKERS): + return "collector" + if any(marker in text for marker in LITE_DUCKDB_MARKERS): + return "duckdb" + return "plain" + if any(marker in text for marker in WEB_MARKERS): + return "web" + if LIVE_MARKER in text and RUNTIME_MARKER not in text: + return "live" + return "plain" + + +def scan_sources(dirs, root, profile="darling"): + """Map simple class name -> {file, kind} for every test source file under `dirs`.""" + found = {} + for base in dirs: + for folder, subdirs, names in os.walk(base): + subdirs[:] = [d for d in subdirs if d not in ("bin", "obj", ".git", "node_modules")] + for name in names: + if not name.endswith(".cs"): + continue + path = os.path.join(folder, name) + try: + with open(path, encoding="utf-8-sig", errors="replace") as handle: + text = handle.read() + except OSError: + continue + kind = classify(text, profile) + rel = norm(os.path.relpath(path, root)) + for cls in CLASS_DECL.findall(text): + found.setdefault(cls, {"file": rel, "kind": kind}) + return found + + +def spread(names, count): + names = sorted(names) + if count <= 0 or not names: + return [] + if count >= len(names): + return names + step = len(names) / count + return [names[int(i * step)] for i in range(count)] + + +def cmd_select(args): + with open(args.list, encoding="utf-8-sig") as handle: + text = handle.read() + start = text.index("[") + listed = json.loads(text[start:text.rindex("]") + 1]) + sources = scan_sources(args.tests_dir, args.root, args.profile) + wants = dict(PROFILE_WANTS[args.profile]) + for item in args.want: + name, _, number = item.partition("=") + wants[name] = int(number) + buckets = {"plain": []} + buckets.update({kind: [] for kind in wants}) + for full in listed: + info = sources.get(simple_name(full)) + if info and info["kind"] in buckets: + buckets[info["kind"]].append(full) + counts = {kind: min(number, len(buckets[kind])) for kind, number in wants.items()} + counts["plain"] = max(args.total - sum(counts.values()), 0) + chosen = {kind: spread(buckets[kind], counts[kind]) for kind in buckets} + manifest = {} + lines = [] + for kind, names in chosen.items(): + for full in names: + manifest[full] = {"kind": kind, "file": sources[simple_name(full)]["file"]} + lines.append(full) + with open(args.out, "w", encoding="utf-8", newline="\n") as handle: + handle.write("\n".join(lines) + "\n") + with open(args.manifest, "w", encoding="utf-8", newline="\n") as handle: + json.dump(manifest, handle, indent=1, sort_keys=True) + print("%s: listed %d classes; candidates %s; chose %s (%d total)" % ( + args.profile, len(listed), " ".join("%s=%d" % (k, len(v)) for k, v in sorted(buckets.items())), + " ".join("%s=%d" % (k, len(v)) for k, v in sorted(chosen.items())), len(lines))) + return 0 if lines else 1 + + +def failure_message(test): + failure = test.find("failure") + if failure is None: + return "" + message = failure.find("message") + return (message.text or "") if message is not None else "" + + +def cmd_unwrap_skips(args): + """Turn an instrumented run's wrapped dynamic skips back into skips. + + AltCover rewrites an instrumented async [Fact] so that the method waits on its own task (the failure's stack + is `Task.Wait` called from the test method itself). A dynamic skip (`Assert.Skip*`) thrown inside the async + body therefore surfaces as an AggregateException whose message does not start with the skip token, and + xunit v3 reports a failure instead of a skip. The wrap is in the rewritten test method, not in product or test + source, and 760 Darling test files use dynamic skips, so the report is repaired here instead. + + Guards: a test is converted only when its failure text carries the skip token AND the plain run of the same + test was a skip. A test the plain run did not skip, or any other failure, stays a failure. Nothing is + ever converted to a pass. The original report is kept next to the result as `.raw.xml`. + + Without `--plain` (the map producer runs every class once, instrumented, and has no plain run to compare with) + a failure that carries the skip token is converted on the token alone. Only `Assert.Skip*` writes that token, + and the producer reads coverage from the run, not the pass/fail counts, so this cannot hide a real failure.""" + plain_skips = None # None: no plain run to compare with (the map producer), so the skip token alone decides + if args.plain: + plain_skips = set() + for _, el in ET.iterparse(args.plain, events=("end",)): + if el.tag == "test": + if (el.get("result") or "").lower() == "skip": + plain_skips.add(el.get("name")) + el.clear() + tree = ET.parse(args.instr) + converted = [] + kept = [] + for test in tree.getroot().iter("test"): + if (test.get("result") or "").lower() != "fail": + continue + message = failure_message(test) + if SKIP_TOKEN not in message: + continue + if plain_skips is not None and test.get("name") not in plain_skips: + kept.append(test.get("name")) + continue + reason = message[message.index(SKIP_TOKEN) + len(SKIP_TOKEN):].split("\n", 1)[0].strip().rstrip(")") + failure = test.find("failure") + if failure is not None: + test.remove(failure) + test.set("result", "Skip") + ET.SubElement(test, "reason").text = reason + converted.append(test.get("name")) + if converted: + # Counts live on the collection and assembly elements; move each converted test from failed to skipped. + moved_names = set(converted) + for parent in tree.getroot().iter(): + if parent.tag not in ("collection", "assembly"): + continue + moved = sum(1 for test in parent.iter("test") if test.get("name") in moved_names) + if moved and parent.get("failed") is not None and parent.get("skipped") is not None: + parent.set("failed", str(max(int(parent.get("failed")) - moved, 0))) + parent.set("skipped", str(int(parent.get("skipped")) + moved)) + raw = os.path.splitext(args.instr)[0] + ".raw.xml" + if not os.path.exists(raw): + os.replace(args.instr, raw) + tree.write(args.instr, encoding="utf-8", xml_declaration=True) + print("unwrap-skips: %d converted back to skips, %d left failing (skip token, but the plain run did not skip)" % ( + len(converted), len(kept))) + for name in converted[:12]: + print(" skip restored: " + name) + for name in kept[:12]: + print(" still failing: " + name) + return 0 + + +def class_of_tracked(name: str) -> str: + """`System.Void Ns.Outer/Inner::Method(System.String)` -> `Ns.Outer+Inner` (the declaring type).""" + left = name.split("::", 1)[0] + return left.split(" ")[-1].replace("/", "+") + + +def parse_report(path, root): + """Stream an OpenCover report. Returns (context -> set(files), file stats, totals).""" + root_norm = norm(root).rstrip("/").lower() + "/" + ctx_files: dict[str, set] = {} + file_stats: dict[str, list] = {} # file -> [visits, visits with a context] + totals = {"modules": 0, "methods": 0, "points": 0, "points_visited": 0, "points_visited_with_context": 0, + "visits": 0, "visits_with_context": 0, "tracked_methods": 0, "tracked_refs": 0, "tracked_uid_collisions": 0, "unresolved_refs": 0, "points_partly_outside_context": 0, "points_fully_in_context": 0, "context_refs_on_visited_points": 0, + "ref_owner": {}} + stack = [] + names: dict[str, str] = {} + resolved: set = set() + files = {} + pairs = set() + mod_stats: dict = {} # file uid -> [visits, visits with a context], per module (uids repeat across modules) + method_file = None + point = None # [file, vc, ref_vc, ref_count] + + def finish_point(): + nonlocal point + if point is None: + return + file, vc, ref_vc, ref_count = point + totals["points"] += 1 + if vc > 0: + totals["points_visited"] += 1 + totals["visits"] += vc + totals["visits_with_context"] += min(ref_vc, vc) if ref_count else 0 + if ref_count: + totals["points_visited_with_context"] += 1 + # A point whose per-context visit counts add up to less than its total was also reached + # outside every context (a constructor, a fixture, a pool thread started by the test). + totals["points_partly_outside_context" if ref_vc < vc else "points_fully_in_context"] += 1 + totals["context_refs_on_visited_points"] += ref_count + stat = mod_stats.setdefault(file, [0, 0]) + stat[0] += vc + stat[1] += min(ref_vc, vc) if ref_count else 0 + point = None + + for event, el in ET.iterparse(path, events=("start", "end")): + tag = el.tag + if event == "start": + stack.append(tag) + if tag == "Module": + totals["modules"] += 1 + files = {} + pairs = set() + mod_stats = {} + elif tag == "File": + files[el.get("uid")] = norm(el.get("fullPath") or "") + elif tag == "Method": + totals["methods"] += 1 + method_file = None + elif tag == "FileRef" and "Method" in stack: + method_file = el.get("uid") + elif tag in ("SequencePoint", "MethodPoint"): + finish_point() + point = [el.get("fileid") or method_file, int(el.get("vc") or 0), 0, 0] + elif tag == "TrackedMethodRef": + uid = el.get("uid") + vc = int(el.get("vc") or 0) + owner = stack[-3] if len(stack) >= 3 else "?" + totals["tracked_refs"] += 1 + totals["ref_owner"][owner] = totals["ref_owner"].get(owner, 0) + 1 + file = (point[0] if point is not None else None) or method_file + if point is not None: + point[2] += vc + point[3] += 1 + pairs.add((uid, file)) + else: + if tag in ("SequencePoint", "MethodPoint"): + finish_point() + elif tag == "Method": + finish_point() + el.clear() + elif tag == "TrackedMethod": + totals["tracked_methods"] += 1 + elif tag == "TrackedMethods": + pass + elif tag == "Module": + for fuid, (visit_count, ctx_count) in mod_stats.items(): + if fuid in files: + stat = file_stats.setdefault(files[fuid], [0, 0]) + stat[0] += visit_count + stat[1] += ctx_count + # TrackedMethod uids are numbered across the whole report, and a product module's points + # refer to test methods listed in the TEST module, so names are resolved after the last + # module rather than inside the module that holds the reference. + for tm in el.iter("TrackedMethod"): + uid, name = tm.get("uid"), tm.get("name") or "" + if uid in names and names[uid] != name: + totals["tracked_uid_collisions"] += 1 + names[uid] = name + for uid, fuid in pairs: + if fuid in files: + resolved.add((uid, files[fuid])) + el.clear() + stack.pop() + for uid, full in resolved: + name = names.get(uid) + if not name: + totals["unresolved_refs"] += 1 + continue + if full.lower().startswith(root_norm): + ctx_files.setdefault(class_of_tracked(name), set()).add(full[len(root_norm):]) + return ctx_files, file_stats, totals, root_norm + + +def xunit_times(path): + """Per-class seconds and outcome counts from an xunit `-xml` report (v2 and v3 shapes).""" + seconds: dict[str, float] = {} + counts = {"pass": 0, "fail": 0, "skip": 0} + if not path or not os.path.exists(path): + return seconds, counts + for _, el in ET.iterparse(path, events=("end",)): + if el.tag == "test": + cls = (el.get("type") or "").replace("/", "+") + try: + seconds[cls] = seconds.get(cls, 0.0) + float(el.get("time") or 0) + except ValueError: + pass + result = (el.get("result") or "").lower() + counts["pass" if result == "pass" else "fail" if result == "fail" else "skip"] += 1 + el.clear() + return seconds, counts + + +def hole_detail(info, visited, root): + """Why a class has no product file: what it did visit, and what its own source text suggests.""" + detail = {"kind": info.get("kind"), "test_files_visited": sorted(visited)[:6]} + path = os.path.join(root, info.get("file") or "") + try: + with open(path, encoding="utf-8-sig", errors="replace") as handle: + text = handle.read() + except OSError: + return detail + detail["reads_source_text"] = bool(TEXT_READ.search(text)) + detail["uses_reflection_or_typeof"] = bool(REFLECTION.search(text)) + detail["needs_pg_runtime"] = RUNTIME_MARKER in text + return detail + + +def cmd_summarize(args): + ctx_files, file_stats, totals, root_prefix = parse_report(args.report, args.root) + with open(args.manifest, encoding="utf-8") as handle: + manifest = json.load(handle) + plain_seconds, plain_counts = xunit_times(args.plain_xml) + instr_seconds, instr_counts = xunit_times(args.instr_xml) + timing = {} + if args.timing and os.path.exists(args.timing): + with open(args.timing, encoding="utf-8-sig") as handle: + timing = json.load(handle) + + test_prefixes = tuple(p.strip("/") + "/" for p in args.test_dir_prefix) + + def is_test_file(rel): + return rel.startswith(test_prefixes) + + # Product-file universe and per-class rows. + universe = sorted({f for fs in ctx_files.values() for f in fs if not is_test_file(f)}) + index = {f: i for i, f in enumerate(universe)} + selected = {cls.replace("/", "+"): info for cls, info in manifest.items()} + classes = {} + holes = {"no_context": [], "no_product_files": [], "detail": {}} + per_kind = {} + for cls, info in sorted(selected.items()): + got = ctx_files.get(cls) + if got is None: + holes["no_context"].append(cls) + continue + product = sorted(index[f] for f in got if f in index) + if not product: + holes["no_product_files"].append(cls) + holes["detail"][cls] = hole_detail(info, got, args.root) + classes[cls] = {"own": info.get("file"), "files": product, + "seconds": round(instr_seconds.get(cls, plain_seconds.get(cls, 0.0)), 3)} + per_kind.setdefault(info["kind"], []).append(len(product)) + unselected = sorted(c for c in ctx_files if c not in selected) + + def stat(values): + if not values: + return None + return {"n": len(values), "min": min(values), "median": statistics.median(values), "max": max(values)} + + # Visits to product files only: test-method bodies are always inside a context by construction, so the + # overall share flatters the tool. `outside_root` is source the checkout does not hold (generated code). + visits = with_ctx = test_visits = outside_visits = 0 + for path, (visit_count, ctx_count) in file_stats.items(): + if not path.lower().startswith(root_prefix): + outside_visits += visit_count + elif is_test_file(path[len(root_prefix):]): + test_visits += visit_count + else: + visits += visit_count + with_ctx += ctx_count + weakest = sorted( + ((f, v, c) for f, (v, c) in file_stats.items() + if f.lower().startswith(root_prefix) and v >= 5 and not is_test_file(f[len(root_prefix):])), + key=lambda row: (row[2] / row[1], -row[1]))[:25] + # File-level view, the one the map cares about: a product file the run visited but never attributed + # to any test class cannot select a single class. + product_files = {f[len(root_prefix):]: v for f, (v, c) in file_stats.items() + if f.lower().startswith(root_prefix) and not is_test_file(f[len(root_prefix):]) and v > 0} + attributed = set(universe) + unattributed = sorted(((v, f) for f, v in product_files.items() if f not in attributed), reverse=True) + plain_total = timing.get("plain_seconds") + instr_total = timing.get("instrumented_seconds") + summary = { + "schema": SCHEMA, + "tool": args.tool, + "classes_selected": len(selected), + "classes_with_files": len(classes) - len(holes["no_product_files"]), + "holes": holes, + "contexts_for_unselected_classes": unselected[:50], + "files_in_universe": len(universe), + "files_per_class": {kind: stat(vals) for kind, vals in sorted(per_kind.items())}, + "visits": { + "points_in_report": totals["points"], "points_visited": totals["points_visited"], + "points_visited_with_context": totals["points_visited_with_context"], + "product_visits": visits, "product_visits_with_context": with_ctx, + "share_with_context": round(with_ctx / visits, 4) if visits else None, + "test_file_visits": test_visits, "outside_root_visits": outside_visits, + "points_partly_outside_context": totals["points_partly_outside_context"], + "points_fully_in_context": totals["points_fully_in_context"], + "contexts_per_context_bearing_point": round( + totals["context_refs_on_visited_points"] / totals["points_visited_with_context"], 2) + if totals["points_visited_with_context"] else None, + "unresolved_refs": totals["unresolved_refs"], "tracked_uid_collisions": totals["tracked_uid_collisions"], + "tracked_methods": totals["tracked_methods"], "tracked_refs": totals["tracked_refs"], + "tracked_ref_owner_tags": totals["ref_owner"], + }, + "files_visited_by_any_code": len(product_files), + "files_attributed_to_a_class": len(attributed & set(product_files)), + "files_visited_never_attributed": len(unattributed), + "never_attributed_top": [{"file": f, "visits": v} for v, f in unattributed[:40]], + "weakest_context_files": [{"file": f[len(root_prefix):], "visits": v, "with_context": c} + for f, v, c in weakest], + "timing_seconds": {"plain": plain_total, "instrumented": instr_total, + "ratio": round(instr_total / plain_total, 2) if plain_total and instr_total else None, + "instrument_step": timing.get("instrument_seconds"), + "line_and_branch_mode": {k: v for k, v in timing.items() if k.startswith("line_")}, + "method_mode_report_bytes": os.path.getsize(args.report), + "xunit_plain_sum": round(sum(plain_seconds.values()), 1), + "xunit_instrumented_sum": round(sum(instr_seconds.values()), 1)}, + "outcomes": {"plain": plain_counts, "instrumented": instr_counts}, + } + with open(args.summary_out, "w", encoding="utf-8", newline="\n") as handle: + json.dump(summary, handle, indent=1) + suite = args.suite + with open(args.map_out, "w", encoding="utf-8", newline="\n") as handle: + json.dump({"schema": SCHEMA, "tool": args.tool, "files": universe, "classes": {suite: classes}}, + handle, separators=(",", ":")) + print(json.dumps(summary, indent=1)[:6000]) + return 0 + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0]) + sub = parser.add_subparsers(dest="command", required=True) + sel = sub.add_parser("select") + sel.add_argument("--profile", choices=sorted(PROFILE_WANTS), default="darling") + sel.add_argument("--want", action="append", default=[], help="kind=count, overrides the profile default") + sel.add_argument("--list", required=True) + sel.add_argument("--tests-dir", action="append", required=True) + sel.add_argument("--root", required=True) + sel.add_argument("--total", type=int, default=100) + sel.add_argument("--out", required=True) + sel.add_argument("--manifest", required=True) + sel.set_defaults(func=cmd_select) + unwrap = sub.add_parser("unwrap-skips") + unwrap.add_argument("--plain", help="the plain run's xunit report; without it the skip token alone converts") + unwrap.add_argument("--instr", required=True) + unwrap.set_defaults(func=cmd_unwrap_skips) + summ = sub.add_parser("summarize") + summ.add_argument("--report", required=True) + summ.add_argument("--root", required=True) + summ.add_argument("--manifest", required=True) + summ.add_argument("--plain-xml") + summ.add_argument("--instr-xml") + summ.add_argument("--timing") + summ.add_argument("--suite", default="Darling.Tests") + summ.add_argument("--tool", default="altcover") + summ.add_argument("--test-dir-prefix", action="append", default=["Darling/Darling.Tests", "Lite.Tests"]) + summ.add_argument("--summary-out", required=True) + summ.add_argument("--map-out", required=True) + summ.set_defaults(func=cmd_summarize) + args = parser.parse_args(argv) + return args.func(args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/scripts/ci-select.py b/.github/scripts/ci-select.py new file mode 100644 index 0000000000..eec905f8e9 --- /dev/null +++ b/.github/scripts/ci-select.py @@ -0,0 +1,1255 @@ +#!/usr/bin/env python3 +"""The ONE place CI's test-selection rules live (#5459). + +Input: the changed files of a run (and the event). Output: which test jobs run and in what scope. + +These readers use the same code: + +* build.yml's jobs call `lite-scope`, `darling-scope` and `tree-scope` with the area answers + dorny/paths-filter already computed, and get the decision back as `key=value` lines for $GITHUB_OUTPUT. +* `map-select` answers, from a class-to-file map, which test classes a pull request needs (build.yml's + test-map-select job writes it as an artifact), and `map-keep` reads that artifact for one suite: the Darling PG, + Lite and no-store Darling passes of a pull request keep only those classes, and run everything when the answer + is FULL, missing or unreadable. See map_select() for the map's schema. `--replay --map FILE` also replays the + corpus against such a map and reports what it would miss (it does not fail on that yet). +* `--replay` reads .github/ci-history/failures.jsonl (a trimmed record of every run with a real test failure) + and checks that each failing class would have been selected for that run's changed files. A cut to the rules + lands only while the replay finds 0 misses. A Guard-tagged test runs it on every pull request. + +The path patterns are NOT copied here. They are read from the filter blocks in .github/workflows/build.yml and +from .github/darling-paths-filter.yml, the same text dorny/paths-filter evaluates, so the replay and the +workflow cannot disagree about which area a file belongs to. What lives here is the decision logic that +used to be spread over `if:` expressions and a bash if-chain. + +Standard library only (the Windows and Linux runners both have Python 3). +""" +from __future__ import annotations + +import argparse +import datetime +import gzip +import json +import os +import re +import subprocess +import sys +import xml.etree.ElementTree as ET + +ROOT = os.path.normpath(os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "..")) +WORKFLOW = os.path.join(ROOT, ".github", "workflows", "build.yml") +DARLING_FILTER = os.path.join(ROOT, ".github", "darling-paths-filter.yml") +CORPUS = os.path.join(ROOT, ".github", "ci-history", "failures.jsonl") +ACCEPTED = os.path.join(ROOT, ".github", "ci-history", "replay-accepted.txt") + + +# --------------------------------------------------------------------------------------------------------- +# Glob matching: the subset of micromatch (dot: true) the filter files use. +# --------------------------------------------------------------------------------------------------------- + +def _segment_regex(seg: str) -> str: + """One path segment (no '/') as a regex fragment that matches a whole segment name.""" + m = re.fullmatch(r"!\((.*)\)", seg) + if m: + inner = _segment_regex(m.group(1)) + return rf"(?!(?:{inner})$)[^/]+" + out: list[str] = [] + i = 0 + while i < len(seg): + c = seg[i] + if c == "*": + out.append("[^/]*") + elif c == "?": + out.append("[^/]") + elif c == "{": + j = seg.index("}", i) + alts = seg[i + 1:j].split(",") + out.append("(?:" + "|".join(_segment_regex(a) for a in alts) + ")") + i = j + else: + out.append(re.escape(c)) + i += 1 + return "".join(out) + + +def compile_glob(pattern: str) -> re.Pattern[str]: + parts = pattern.split("/") + rx = "" + for idx, seg in enumerate(parts): + last = idx == len(parts) - 1 + if seg == "**": + rx += ".*" if last else "(?:[^/]+/)*" + else: + rx += _segment_regex(seg) + ("" if last else "/") + return re.compile(rx + r"\Z") + + +_GLOB_CACHE: dict[str, re.Pattern[str]] = {} + + +def matches(pattern: str, path: str) -> bool: + rx = _GLOB_CACHE.get(pattern) + if rx is None: + rx = _GLOB_CACHE[pattern] = compile_glob(pattern) + return rx.match(path) is not None + + +# --------------------------------------------------------------------------------------------------------- +# Reading the filter blocks. +# --------------------------------------------------------------------------------------------------------- + +_PATTERN_LINE = re.compile(r"^\s*-\s+'([^']*)'") +_KEY_LINE = re.compile(r"^(\s*)([A-Za-z0-9_-]+):\s*(?:#.*)?$") + + +def _parse_filter_lines(lines: list[str]) -> dict[str, list[str]]: + filters: dict[str, list[str]] = {} + current: list[str] | None = None + for line in lines: + s = line.strip() + if not s or s.startswith("#"): + continue + p = _PATTERN_LINE.match(line) + if p: + if current is not None: + current.append(p.group(1)) + continue + k = _KEY_LINE.match(line) + if k: + current = filters.setdefault(k.group(2), []) + return filters + + +def load_workflow_filters(path: str = WORKFLOW) -> dict[str, dict[str, list[str]]]: + """{job id: {filter name: [patterns]}} for every inline `filters: |` block in the workflow.""" + with open(path, encoding="utf-8") as fh: + lines = fh.read().split("\n") + out: dict[str, dict[str, list[str]]] = {} + job = "" + in_jobs = False + i = 0 + while i < len(lines): + line = lines[i] + if line.startswith("jobs:"): + in_jobs = True + elif in_jobs and re.match(r"^ [A-Za-z0-9_-]+:\s*$", line): + job = line.strip().rstrip(":") + m = re.match(r"^(\s+)filters:\s*\|\s*$", line) + if m: + base = len(m.group(1)) + block: list[str] = [] + i += 1 + while i < len(lines) and (not lines[i].strip() or len(lines[i]) - len(lines[i].lstrip()) > base): + block.append(lines[i]) + i += 1 + out[job] = _parse_filter_lines(block) + continue + i += 1 + return out + + +def load_file_filters(path: str = DARLING_FILTER) -> dict[str, list[str]]: + with open(path, encoding="utf-8") as fh: + return _parse_filter_lines(fh.read().split("\n")) + + +class Rules: + """The pattern sets, loaded once.""" + + def __init__(self, workflow: str = WORKFLOW, darling_filter: str = DARLING_FILTER) -> None: + wf = load_workflow_filters(workflow) + self.build = wf["build"] + self.lite_shards = wf["lite-tests"] + self.tree = wf["darling-tree-guards"] + self.guard = wf["guard-tests"] + self.gate = load_file_filters(darling_filter) + + @staticmethod + def _hits(patterns: list[str], files: list[str]) -> list[str]: + return [f for f in files if any(matches(p, f) for p in patterns)] + + def areas(self, files: list[str]) -> dict[str, bool]: + """Every area answer dorny/paths-filter gives a job, by name, for one list of changed files.""" + out: dict[str, bool] = {} + for name, pats in self.build.items(): + out[name] = bool(self._hits(pats, files)) + out["docs_count"] = len(self._hits(self.build["docs"], files)) # type: ignore[assignment] + out["all_count"] = len(files) # type: ignore[assignment] + for name, pats in self.gate.items(): + out["gate_" + name] = bool(self._hits(pats, files)) + for name, pats in self.lite_shards.items(): + out[name] = bool(self._hits(pats, files)) + out["workflow"] = any(matches(p, f) for p in self.tree["workflow"] for f in files) + out["lite_tree_count"] = len(self._hits(self.tree["lite_tree"], files)) # type: ignore[assignment] + return out + + +# --------------------------------------------------------------------------------------------------------- +# The decisions. +# --------------------------------------------------------------------------------------------------------- + +def darling_scope(event: str, runs: bool, shards_run: bool, full_paths: bool = True) -> str: + """What the build job's no-store "Run Darling tests" step runs: full, guard (only the Stage=Guard classes), + reads-lite (the Guard classes plus the Reads=Lite ones) or none. + + The PostgreSQL shards run every Darling.Tests class, with a live store, whenever the Darling gate + (.github/darling-paths-filter.yml) matches. The no-store pass would repeat their non-live classes, so while the + shards run it keeps only the Guard stage (cut 3b of #5459; the replay finds no failing class it stops covering). + A release always runs the whole suite. + + Cut 3a of #5459: the step also fires for a Lite file alone (Darling.Tests reads Lite source in its parity guards, + so a Lite-only change has to reach the suite). On a pull request that reaches it ONLY through those Lite paths + (`full_paths` false: no Darling file, shared library or root build file changed) the classes that read Lite are + the only ones the change can affect, so it runs the Guard stage and the classes tagged Reads=Lite, the mirror of + Lite's `reads` mode for a Darling-only change. Push and merge-queue runs keep the whole suite, as Lite's do.""" + if not runs: + return "none" + if event != "release" and shards_run: + return "guard" + if event == "pull_request" and not full_paths: + return "reads-lite" + return "full" + + +def tree_scope(event: str, all_count: int, lite_count: int) -> str: + """What darling-tree-guards runs once it has decided to run: full, or reads-lite (cut 3c of #5459). + + The job runs the Darling suite when the build job's Darling step was skipped, which for a change made entirely + of Lite files (a Lite.Tests edit, a Lite file type the `darling` filter does not name) leaves only the classes + that read Lite to run. Every changed file must be under Lite/ or Lite.Tests/, and only a pull request narrows.""" + if event == "pull_request" and all_count > 0 and lite_count == all_count: + return "reads-lite" + return "full" + + +def scope_filter(scope: str) -> str: + """The Darling.Tests runner arguments for a scope ('' is the whole suite).""" + if scope == "guard": + return "-trait Stage=Guard" + if scope == "reads-lite": + return "-trait Stage=Guard -trait Reads=Lite" + return "" + + +def lite_scope(event: str, lite: bool, core: bool, root: bool, reads: bool, linked: bool) -> str: + """How much of the Lite suite a pull request runs: full, reads (only Reads=Darling classes) or none.""" + if lite or core or root or linked: + return "full" + if reads and event != "pull_request": + return "full" + if reads: + return "reads" + return "none" + + +def decide(files: list[str], event: str, rules: Rules) -> dict: + """Which jobs run, and with which filter, for one change. Everything the workflow decides, in one place.""" + a = rules.areas(files) + release = event == "release" + all_count, docs_count = a["all_count"], a["docs_count"] + docs_only = all_count > 0 and all_count == docs_count + area_names = ("root", "core", "installer_core", "dashboard", "lite", "installer", "darling") + fastpath = (not release) and docs_only and not any(a[n] for n in area_names) + + build_lite = release or a["lite"] or a["core"] or a["root"] + build_installer = release or a["installer"] or a["installer_core"] or a["root"] + build_dashboard = release or a["dashboard"] or a["core"] or a["installer_core"] or a["root"] + build_darling = release or a["darling"] or a["core"] or a["root"] + + pg_run = (not release) and a["gate_darling"] + # The tree guards run when the build job's Darling step did not (or build.yml itself changed). + tree_run = (not release) and not fastpath and (not build_darling or a["workflow"]) + + mode = lite_scope(event, a["lite_shard"], a["core_shard"], a["root_shard"], a["darling_reads_shard"], + a["lite_linked_shard"]) if not release else "release" + + return { + "guard_run": (not release) and not docs_only, + "docs_fastpath": fastpath, + "build_lite_tests": build_lite, + "build_installer_tests": build_installer, + "build_dashboard_tests": build_dashboard, + "build_darling_tests": build_darling, + "darling_scope": darling_scope( + event, build_darling, pg_run, release or a["darling_full"] or a["core"] or a["root"]), + "darling_pg": pg_run, + "darling_linux": release or a["gate_darling"], + "lite_mode": mode, + "tree_guards": tree_run, + "tree_scope": tree_scope(event, all_count, a["lite_tree_count"]) if tree_run else "none", + } + + +# --------------------------------------------------------------------------------------------------------- +# The test classes of the tree (for the replay). +# --------------------------------------------------------------------------------------------------------- + +_CLASS_DECL = re.compile( + r"^[ \t]*(?:(?:public|internal|private|protected|sealed|static|abstract|partial)\s+)*class\s+(\w+)") +_TRAIT = re.compile(r'Trait\(\s*"(\w+)"\s*,\s*"(\w+)"\s*\)') + +# The test map is built on shards that set only DARLING_TEST_PG and DARLING_TEST_PGRUNTIME (nightly.yml's "Run this +# shard's classes" step; a test pins that env block to MAP_SHARD_ENV). A class that gates its tests on any OTHER +# DARLING_TEST_PG_* or DARLING_TEST_PGRUNTIME_* variable (the log-format targets, the log-rotation targets, the +# store-upgrade fixtures) skips at map build, so the map has no coverage edges for what its tests would run and a +# change to the code under them would never select it (#5459; measured on the 2026-10-09 map: 16 files, 19 +# mapped classes, 12 to 44 coverage files each against 100 and more for a fully covered class). The map cannot tell +# us, so the SOURCE does: every class in a file that names such a variable in a quoted literal carries the Gate=Env +# trait and map_select always selects it. It is found from the source at selection time, so a new gated class needs +# no list edit. +# +# What the scan cannot see (a miss here is silent, so a new gate must be written to be seen): it reads each test file +# on its own, for a quoted "DARLING_TEST_PG_..." or "DARLING_TEST_PGRUNTIME_..." literal. A gate variable that a class +# reads through a constant or a helper declared in ANOTHER file, or one that is not named DARLING_TEST_PG_* / +# DARLING_TEST_PGRUNTIME_*, is invisible to it, and the class stays unmarked and unmapped. A new gate of either kind +# must also name its variable as a quoted literal in the class's own file (or take the Gate=Env trait by hand). +# +# What it costs: the mark is per CLASS, not per test, so a class where only some tests are gated runs whole on every +# narrowed Darling pull request, and the darling-pg job sets these variables, so the gated tests really run. Measured +# on a full run of this change (workflow run 37911378693, the 19 test classes that carry the mark, summed test time +# per class, the shard each lands on in brackets): DarlingStoreUpgradeTests 274 s [shard 5], +# ManagedConfUpgradePathTests 85 s [4], DarlingManagedPostgresTests 73 s [3], PgServerLogTailCsvJsonRotationLiveTests +# 44 s [1], PgLogBurstAndRotationLiveTests 38 s [2], PgRaiseShapedRecordsLiveTests 31 s [1], +# PgServerLogTailRotationLiveTests 15 s [1], and 12 more classes of 11 s or less (29 s together): about 590 s of test +# time in all. Per shard that is 274 s on shard 5, 100 s on shard 4, 90 s on shard 1, 73 s on shard 3, 43 s on shard 2 +# and 9 s on shard 0. A class runs on one thread, so shard 5 cannot get below the 274 s of DarlingStoreUpgradeTests: on +# a narrowed pull request whose other shards finish in a minute or two, shard 5 becomes the slowest by about 3 +# minutes (it is 174 s ahead of shard 4, the next). It never passes the full run's slowest shard (shard 0, 589 s of +# test time, against 509 s for shard 5 in that run). The shard cut is unchanged. +MAP_SHARD_ENV = frozenset({"DARLING_TEST_PG", "DARLING_TEST_PGRUNTIME"}) +GATE_TRAIT = "Gate=Env" +GATE_ENV_LITERAL = re.compile(r'"DARLING_TEST_PG(?:RUNTIME)?_[A-Z0-9_]+"') + +SUITE_DIRS = { + "darling": os.path.join("Darling", "Darling.Tests"), + "lite": "Lite.Tests", + "dashboard": os.path.join("deprecated", "Dashboard.Tests"), + "installer": os.path.join("deprecated", "Installer.Tests"), +} +SUITE_BY_PREFIX = { + "Darling.Tests": "darling", + "PerformanceMonitor.Darling.Tests": "darling", + "Lite.Tests": "lite", + "PerformanceMonitorLite.Tests": "lite", + "Dashboard.Tests": "dashboard", + "Installer.Tests": "installer", +} + + +def split_failed_class(cls: str) -> tuple[str, str]: + """(namespace prefix, simple class name) of a failing test class's full name. A nested class is reported as + `Ns.Outer+Inner` (some runners use `Ns.Outer/Inner`): the prefix is the namespace of the OUTER class, the text + before the first '+', and the simple name is the innermost class, which is the name _walk_classes yields for a + nested declaration. Splitting on the last '.' alone left `Outer+Inner` as the "simple name", found no such class + in the tree, and so never counted a nested failure as a miss (#5459).""" + head = re.split(r"[+/]", cls, maxsplit=1)[0] + return head.rpartition(".")[0], re.split(r"[.+/]", cls)[-1] + + +def _walk_classes(root: str): + """Yield (suite, repo-relative file with '/', simple class name, {"Key=Value", ...}) for every class declaration.""" + for suite, rel in SUITE_DIRS.items(): + base = os.path.join(root, rel) + for dp, dn, fn in os.walk(base): + dn[:] = [d for d in dn if d not in ("bin", "obj")] + for f in fn: + if not f.endswith(".cs"): + continue + path = os.path.join(dp, f) + try: + with open(path, encoding="utf-8-sig") as fh: + lines = fh.read().split("\n") + except OSError: + continue + relfile = os.path.relpath(path, root).replace(os.sep, "/") + gated = any(GATE_ENV_LITERAL.search(line) for line in lines) + for n, line in enumerate(lines): + m = _CLASS_DECL.match(line) + if not m: + continue + traits: set[str] = {GATE_TRAIT} if gated else set() + j = n - 1 + while j >= 0: + s = lines[j].strip() + if s.startswith(("[", "///", "//", "*", "/*")) or s.endswith("*/") or s == "" and False: + traits.update(f"{k}={v}" for k, v in _TRAIT.findall(s)) + j -= 1 + else: + break + yield suite, relfile, m.group(1), traits + + +def scan_classes(root: str = ROOT) -> dict[str, dict[str, set[str]]]: + """{suite: {simple class name: {"Key=Value", ...}}} read from the test sources.""" + out: dict[str, dict[str, set[str]]] = {suite: {} for suite in SUITE_DIRS} + for suite, _file, name, traits in _walk_classes(root): + out[suite].setdefault(name, set()).update(traits) + return out + + +def class_selected(suite: str, traits: set[str], d: dict) -> bool: + """Would today's rules run this class for this decision?""" + guard = "Stage=Guard" in traits + if suite == "darling": + if guard and d["guard_run"]: + return True + # The build job's no-store pass runs the whole suite (full) or only the Guard stage; the shards run every class. + reads_lite = "Reads=Lite" in traits + return d["darling_scope"] == "full" or d["darling_pg"] \ + or (d["tree_guards"] and (d["tree_scope"] == "full" or reads_lite)) \ + or (d["darling_scope"] == "guard" and guard) \ + or (d["darling_scope"] == "reads-lite" and (guard or reads_lite)) + if suite == "lite": + if guard and d["guard_run"]: + return True + if d["lite_mode"] in ("full", "release"): + return True + return d["lite_mode"] == "reads" and "Reads=Darling" in traits + if suite == "dashboard": + return d["build_dashboard_tests"] + if suite == "installer": + return d["build_installer_tests"] + return True + + +# --------------------------------------------------------------------------------------------------------- +# Slow classes: pull requests skip them unless the change reaches them (#5459, change 5). +# --------------------------------------------------------------------------------------------------------- +# +# A class tagged [Trait("Cost", "Slow")] is slow, has never failed in the corpus, and is covered by the push to dev +# one merge later. A pull_request run leaves it out unless the change reaches it. Push, merge-queue, nightly and +# release runs run every class: slow_skip answers nothing for them. This is a separate pass over the classes a leg +# already chose, not part of darling_scope or lite_scope, so a new scope composes with it. +# +# "Reaches it", from the changed file names alone (the same cheap signal the area filters use): +# * the file is a source file the class is declared in; +# * the file's name (without the extension) appears as a word in the class's source, which is how a test names the +# product file, harness script or helper it exercises; +# * a .js/.mjs/.css/.html file, and a class whose source names a .js or .mjs file (the Node-driven classes load +# scripts through a harness that imports others); +# * any non-.cs file inside the class's own test project (a csproj, a fixture, a harness), or a build input +# (the workflow, this script, the shard scripts, a props/csproj/lock file, global.json). +# When the changed file list is unknown (the API call failed, or it came back empty) nothing is skipped. + +SLOW_TRAIT = "Cost=Slow" +_BUILD_INPUT_NAMES = {"directory.build.props", "directory.build.targets", "directory.packages.props", "global.json", + "nuget.config", "packages.lock.json"} +_BUILD_INPUT_PATHS = {".github/workflows/build.yml", ".github/scripts/ci-select.py", ".github/darling-paths-filter.yml", + ".github/scripts/run-darling-pg-shard.ps1", ".github/scripts/run-lite-shard.ps1", + ".github/scripts/lite-shard-pack.py"} +_BUILD_INPUT_SUFFIXES = (".csproj", ".props", ".targets", ".sln", ".slnx") +_WEB_SUFFIXES = (".js", ".mjs", ".css", ".html") + + +def scan_slow(root: str = ROOT) -> dict[str, dict[str, list[str]]]: + """{suite: {class: [repo-relative source files]}} for every class tagged Cost=Slow (a Guard class never counts: + the Guard stage runs on every pull request).""" + files: dict[tuple[str, str], list[str]] = {} + traits: dict[tuple[str, str], set[str]] = {} + for suite, relfile, name, t in _walk_classes(root): + files.setdefault((suite, name), []) + if relfile not in files[(suite, name)]: + files[(suite, name)].append(relfile) + traits.setdefault((suite, name), set()).update(t) + out: dict[str, dict[str, list[str]]] = {suite: {} for suite in SUITE_DIRS} + for (suite, name), t in traits.items(): + if SLOW_TRAIT in t and "Stage=Guard" not in t: + out[suite][name] = sorted(files[(suite, name)]) + return out + + +class SlowIndex: + """The Cost=Slow classes of the tree and what reaches them.""" + + def __init__(self, root: str = ROOT) -> None: + self.root = root + self.slow = scan_slow(root) + self._text: dict[tuple[str, str], str] = {} + self._words: dict[tuple[str, str], set[str]] = {} + + def text(self, suite: str, name: str) -> str: + key = (suite, name) + if key not in self._text: + parts = [] + for rel in self.slow[suite][name]: + try: + with open(os.path.join(self.root, rel), encoding="utf-8-sig") as fh: + parts.append(fh.read()) + except OSError: + pass + self._text[key] = "\n".join(parts) + return self._text[key] + + def words(self, suite: str, name: str) -> set[str]: + """The class source's words (letters, digits, underscore, hyphen), the unit a file stem is matched against.""" + key = (suite, name) + if key not in self._words: + self._words[key] = set(re.findall(r"[\w-]+", self.text(suite, name))) + return self._words[key] + + def skip(self, files: list[str], event: str, base_ref: str = "") -> dict[str, list[str]]: + """{suite: [simple class names a leg leaves out]} for one change. Empty for every event but pull_request, + for a pull request into main (narrows), and when the changed file list is unknown.""" + out: dict[str, list[str]] = {suite: [] for suite in SUITE_DIRS} + if not narrows(event, base_ref) or not files: + return out + changed = set(files) + # Everything about the change is computed once; a class is then a few set lookups. + stems: set[str] = set() + odd_stems: set[str] = set() + exts: set[str] = set() + build_input = False + for f in changed: + low = f.lower() + base = f.rsplit("/", 1)[-1] + stem, dot, ext = base.rpartition(".") + ext = dot + ext.lower() if dot else "" + if low in _BUILD_INPUT_PATHS or base.lower() in _BUILD_INPUT_NAMES or low.endswith(_BUILD_INPUT_SUFFIXES): + build_input = True + if stem: + (stems if re.fullmatch(r"[\w-]+", stem) else odd_stems).add(stem) + if ext in _WEB_SUFFIXES: + exts.add(ext) + if build_input: + return out + for suite, classes in self.slow.items(): + test_dir = SUITE_DIRS[suite].replace(os.sep, "/") + "/" + if any(f.startswith(test_dir) and not f.endswith(".cs") for f in changed): + continue + for name, own in classes.items(): + text = self.text(suite, name) + reached = bool(changed.intersection(own)) or bool(stems & self.words(suite, name)) \ + or any(re.search(r"(? list[str]: + """The pull request's changed files (both sides of a rename) through `gh api`; [] when the call fails.""" + import subprocess + try: + r = subprocess.run(["gh", "api", f"repos/{repo}/pulls/{pr}/files", "--paginate", "--jq", + '.[] | .filename, (.previous_filename // empty)'], + capture_output=True, text=True, timeout=120) + except (OSError, subprocess.SubprocessError) as e: + print(f"slow-skip: could not list the pull request's files ({e}); skipping nothing", file=sys.stderr) + return [] + if r.returncode != 0: + print(f"slow-skip: gh api exited {r.returncode}; skipping nothing", file=sys.stderr) + return [] + out = [ln.strip() for ln in r.stdout.splitlines() if ln.strip()] + if len(out) >= 3000: # the API lists at most 3000 files, so a list this long may be cut: do not trust it + print("slow-skip: the file list may be truncated; skipping nothing", file=sys.stderr) + return [] + return out + + +def load_accepted(path: str = ACCEPTED) -> set[tuple[str, str]]: + """The rows today's rules miss for a reason that is not a selection gap. One ` ` per line, + the reason in a `#` comment above the group. A miss that is not listed here fails the replay.""" + out: set[tuple[str, str]] = set() + if os.path.exists(path): + with open(path, encoding="utf-8") as fh: + for line in fh: + line = line.split("#", 1)[0].strip() + if line: + sha, cls = line.split()[:2] + out.add((sha, cls)) + return out + + +def replay(corpus: str = CORPUS, root: str = ROOT, quiet: bool = False) -> int: + rules = Rules() + accepted = load_accepted() + accepted_seen = 0 + tree = scan_classes(root) + slow = SlowIndex(root) + rows = checked = misses = exempt = 0 + gone: set[str] = set() + missed: list[str] = [] + with open(corpus, encoding="utf-8") as fh: + for line in fh: + if not line.strip(): + continue + row = json.loads(line) + rows += 1 + d = decide(row["changed_files"], row["event"], rules) + skipped = slow.skip(row["changed_files"], row["event"]) + for cls in row["failed_classes"]: + prefix, simple = split_failed_class(cls) + suite = SUITE_BY_PREFIX.get(prefix) + if suite is None: + continue + traits = tree.get(suite, {}).get(simple) + if traits is None: + exempt += 1 + gone.add(cls) + continue + checked += 1 + if simple in skipped.get(suite, ()) or not class_selected(suite, traits, d): + if (row["head_sha"][:10], cls) in accepted: + accepted_seen += 1 + continue + misses += 1 + missed.append(f"{row['date']} {row['head_sha'][:10]} {cls} files={len(row['changed_files'])}" + f" e.g. {row['changed_files'][0]}") + if not quiet: + print(f"replay: {rows} rows, {checked} failing classes checked, {exempt} exempt (class no longer in the tree: " + f"{len(gone)} distinct), {accepted_seen} accepted misses " + f"(replay-accepted.txt), {misses} misses") + for m in missed[:40]: + print("MISS", m) + return misses + + +# --------------------------------------------------------------------------------------------------------- +# Selection from a class-to-file map (#5459 change 4, slice 1: nothing in build.yml calls this yet). +# --------------------------------------------------------------------------------------------------------- + +MAP_SCHEMA = 1 +MAP_MAX_AGE_DAYS = 7 +MAP_MAX_DRIFT = 300 + +# Files whose change can alter any test in ways a call map cannot see: the build and restore configuration. +# selection_args() adds the root and shared-library areas read from build.yml, so the lists cannot disagree. +KEEP_FULL = ( + ".github/workflows/build.yml", + "Directory.Build.props", + "Directory.Build.targets", + "Directory.Packages.props", + "global.json", + "nuget.config", + "NuGet.config", + "**/*.csproj", + "**/*.props", + "**/*.targets", + "**/packages.lock.json", + "**/xunit.runner.json", +) + +# The documentation allowlist (the `docs` filter in build.yml); selection_args() passes the live one. +DOC_PATTERNS = ( + "**/*.md", + "LICENSE", + "CITATION.cff", + ".gitignore", + ".gitattributes", + "llms.txt", + "tools/changelog/archive-census.txt", + "docs/**/*.{md,svg,png,jpg,jpeg,gif}", + "Screenshots/**/*.{md,svg,png,jpg,jpeg,gif}", +) + + +_TEST_PROJECT_FILE = re.compile(r"(^|/)[^/]*Tests/") + + +MAIN_BRANCH = "main" + + +def narrows(event: str, base_ref: str = "") -> bool: + """Whether a run may run less than everything: only a pull_request whose base branch is not `main` (#5459). + A pull request into main is the release gate (dev to main), so it runs every class, Cost=Slow ones included. + An empty or absent base ref keeps the old behaviour: the replay corpus rows carry none.""" + if event != "pull_request": + return False + ref = (base_ref or "").strip() + if ref.startswith("refs/heads/"): + ref = ref[len("refs/heads/"):] + return ref != MAIN_BRANCH + + +def _full(reason: str) -> dict: + return {"full": True, "reason": reason, "selected": {}, "why": {}, "seconds": {}, "total": {}} + + +def _ids(value: object, universe: list) -> set[str]: + """Paths for a list of file ids. An id is an index into the map's `files`; a string is taken as the path itself.""" + out: set[str] = set() + if value is None: + return out + if isinstance(value, (int, str)): + value = [value] + for v in value: # type: ignore[union-attr] + if isinstance(v, bool): + continue + if isinstance(v, int): + if 0 <= v < len(universe): + out.add(universe[v]) + elif isinstance(v, str): + out.add(v) + return out + + +def _seconds(entry: dict) -> float: + v = entry.get("seconds", 0) + return float(v) if isinstance(v, (int, float)) and not isinstance(v, bool) else 0.0 + + +def _parse_time(text: object) -> "datetime.datetime | None": + if not isinstance(text, str): + return None + try: + t = datetime.datetime.fromisoformat(text.replace("Z", "+00:00")) + except ValueError: + return None + return t if t.tzinfo else t.replace(tzinfo=datetime.timezone.utc) + + +def map_select(test_map: object, changed: list[str], drift: "list[str] | None", discovered: dict, + *, event: str = "pull_request", keep_full: "tuple[str, ...] | list[str]" = KEEP_FULL, + doc_patterns: "tuple[str, ...] | list[str]" = DOC_PATTERNS, + now: "datetime.datetime | None" = None, max_age_days: "float | None" = MAP_MAX_AGE_DAYS, + max_drift: int = MAP_MAX_DRIFT, base_ref: str = "") -> dict: + """Which test classes a pull request runs, from a class-to-file map. Pure: no file, git or clock access + (`now` is the clock; `max_age_days=None` turns the age check off, for a replay of old runs). + + Schema 1 of the map (nightly's `test-map.json.gz`): + + {"schema": 1, "sha": "", "built_at": "", "tool": "", + "files": ["", ...], # the universe; an id below is an index into it + "classes": {"": {"": {"own": , # the class's own source file(s) + "files": [, ...], # every file the class ran code from + "seconds": }}}, # its last measured run time + "text_patterns": {"": ["", ...] or "tree"}} # sources read as text; "tree" = any + + Arguments: `changed` is the pull request's files; `drift` the files changed between the map's commit and the + merge base (None when they could not be computed); `discovered` is `{suite: {Class: {"Key=Value", ...}}}` as + scan_classes() returns it, which is how a class the map has never seen is found. + + A class is selected when it carries Stage=Guard, its own file changed, it covers a changed file, a text pattern + of it matches a changed file, it reads the whole tree and any non-documentation file changed, or the map has + never seen it. A file changed since the map counts as changed too (it may have grown new call edges). + A keep-full file forces FULL only when the pull request's own change holds it: one that changed on dev since + the map goes through the class links like any other drifted file. + + Returns `{"full": bool, "reason": str, "selected": {suite: [Class, ...]}, "why": {suite: {Class: reason}}, + "seconds": {suite: float}, "total": {suite: int}}`. FULL (never "nothing") whenever the map cannot be trusted: + missing, unreadable, wrong schema, too old, drift over `max_drift` files, a keep-full file, a file the map knows + nothing about, or any event but a pull request. That includes a new product .cs file: types found by reflection, + DI or attribute scanning change what a census class sees without any file it covers changing. The one exception + is a new .cs file under a test project (a `*Tests/` or `*.Tests/` directory) while `discovered` holds a class the + map has never seen; that class runs as a "new class".""" + if event != "pull_request": + return _full(f"event {event or '(none)'} always runs everything") + if not narrows(event, base_ref): + return _full("a pull request into main always runs everything") + if test_map is None: + return _full("no test map") + if not isinstance(test_map, dict): + return _full("test map unreadable") + if test_map.get("schema") != MAP_SCHEMA: + return _full(f"test map schema {test_map.get('schema')!r}, expected {MAP_SCHEMA}") + universe = test_map.get("files") + classes = test_map.get("classes") + patterns = test_map.get("text_patterns", {}) + if not isinstance(universe, list) or not isinstance(classes, dict) or not isinstance(patterns, dict): + return _full("test map unreadable") + if max_age_days is not None: + built = _parse_time(test_map.get("built_at")) + if built is None: + return _full("test map has no readable build time") + clock = now or datetime.datetime.now(datetime.timezone.utc) + if clock - built > datetime.timedelta(days=max_age_days): + return _full(f"test map older than {max_age_days:g} days") + if drift is None: + return _full("drift since the map's commit unknown") + if len(drift) > max_drift: + return _full(f"{len(drift)} files changed since the map (over {max_drift})") + + # Keep-full is about THIS pull request's own change (merge base to head). A keep-full file that only changed on dev + # since the map's commit is not this pull request's doing and the dev run that merged it already tested it, so it + # goes through the class links like any other drifted file (#5459: ten pull requests were FULL for that alone). + for f in changed: + if any(matches(p, f) for p in keep_full): + return _full(f"{f} is a keep-full file") + effective = list(dict.fromkeys(list(changed) + list(drift))) + live = [f for f in effective if not any(matches(p, f) for p in doc_patterns)] + live_names = set(live) + + known: set[str] = {u for u in universe if isinstance(u, str)} + parsed: dict[str, dict[str, tuple[set[str], set[str]]]] = {} + for suite, entries in classes.items(): + if not isinstance(entries, dict): + return _full("test map unreadable") + parsed[suite] = {} + for cls, e in entries.items(): + if not isinstance(e, dict): + return _full("test map unreadable") + own, covered = _ids(e.get("own"), universe), _ids(e.get("files"), universe) + parsed[suite][cls] = (own, covered) + known |= own | covered + globs: dict[str, "list[str] | str"] = {} + for cls, p in patterns.items(): + if p == "tree" or (isinstance(p, list) and all(isinstance(x, str) for x in p)): + globs[cls] = p + else: + return _full("test map unreadable") + + docs_changed = [f for f in effective if f not in live_names] + + def text_hit(cls: str) -> bool: + g = globs.get(cls) + if g == "tree": + return bool(live) + if not isinstance(g, list): + return False + # A class that reads a documentation file by name (the migration ladder reads CHANGELOG.md) is selected by a + # change to exactly that file; the documentation allowlist only keeps wildcard patterns from matching it. + return (any(matches(p, f) for p in g for f in live) + or any(p == f for p in g if "*" not in p for f in docs_changed)) + + new_classes = any(cls not in parsed.get(suite, {}) for suite, found in discovered.items() for cls in found) + for f in live: + if f in known: + continue + if any(isinstance(g, list) and any(matches(p, f) for p in g) for g in globs.values()): + continue + if f.endswith(".cs"): + if _TEST_PROJECT_FILE.search(f) and new_classes: + continue # a new test file: its classes are found through `discovered` + return _full(f"new file not in the map: {f}") + return _full(f"{f} is in no class, no text pattern and not in the map") + + # A class the map has never seen runs because `discovered` lists it. If the tree scan found no class at all for a + # suite the map has classes for, the scan itself is broken and every new class would be silently left out. + for suite, entries in parsed.items(): + if entries and not discovered.get(suite): + return _full(f"no test class was found in the tree for suite {suite}") + + live_set = set(live) + selected: dict[str, list[str]] = {} + why: dict[str, dict[str, str]] = {} + seconds: dict[str, float] = {} + total: dict[str, int] = {} + for suite in sorted(set(parsed) | set(discovered)): + found = discovered.get(suite, {}) + mapped = parsed.get(suite, {}) + reasons: dict[str, str] = {} + for cls, (own, covered) in mapped.items(): + if live_set & own: + reasons[cls] = "own file changed" + elif live_set & covered: + reasons[cls] = "covers a changed file" + elif text_hit(cls): + reasons[cls] = "reads the tree" if globs.get(cls) == "tree" else "text pattern" + for cls, traits in found.items(): + if cls not in mapped: + reasons[cls] = "new class" + elif "Stage=Guard" in traits: + reasons.setdefault(cls, "guard stage") + if GATE_TRAIT in traits: + reasons.setdefault(cls, "gated on an environment the map shards lack") + selected[suite] = sorted(reasons) + why[suite] = {c: reasons[c] for c in sorted(reasons)} + seconds[suite] = round(sum(_seconds(classes[suite][c]) for c in reasons if c in mapped), 3) + total[suite] = len(set(mapped) | set(found)) + return {"full": False, "reason": "", "selected": selected, "why": why, "seconds": seconds, "total": total} + + +def load_map(path: str) -> "dict | None": + """The map file (plain or gzip), or None when it is missing or not JSON.""" + try: + opener = gzip.open if path.endswith(".gz") else open + with opener(path, "rt", encoding="utf-8") as fh: # type: ignore[operator] + return json.load(fh) + except (OSError, ValueError): + return None + + +def git_drift(map_sha: str, base: str, root: str = ROOT) -> "list[str] | None": + """`git diff --no-renames --name-only ..`: both paths of a rename count. None when git cannot say.""" + try: + out = subprocess.run(["git", "-C", root, "diff", "--no-renames", "--name-only", f"{map_sha}..{base}"], + capture_output=True, text=True, timeout=120, check=True).stdout + except (OSError, subprocess.SubprocessError): + return None + return [ln.strip() for ln in out.split("\n") if ln.strip()] + + +def selection_args(rules: Rules) -> dict: + """The keep-full and documentation patterns, read from the same filters the workflow evaluates.""" + keep = list(KEEP_FULL) + for name in ("root", "core", "installer_core"): + keep.extend(rules.build.get(name, [])) + return {"keep_full": tuple(dict.fromkeys(keep)), "doc_patterns": tuple(rules.build["docs"])} + + +def map_replay(test_map: "dict | None", corpus: str = CORPUS, root: str = ROOT, quiet: bool = False, + misses_out: "str | None" = None) -> int: + """Like replay(), but asks the map: would each failing class of each corpus row have been selected, taking the map + as current (no drift, no age check)? Returns the miss count. Reports only; the caller does not fail on it yet. + The printed list is capped at 40; `misses_out` names a file that gets EVERY miss as one JSON row per line + (date, event, head_sha, suite, class, changed_files), so none goes unclassified (#5459).""" + extra = selection_args(Rules()) + accepted = load_accepted() + tree = scan_classes(root) + rows = checked = misses = full_rows = accepted_seen = 0 + missed: list[str] = [] + rows_out: list[dict] = [] + with open(corpus, encoding="utf-8") as fh: + for line in fh: + if not line.strip(): + continue + row = json.loads(line) + rows += 1 + sel = map_select(test_map, row["changed_files"], [], tree, event=row["event"], max_age_days=None, **extra) + full_rows += bool(sel["full"]) + for cls in row["failed_classes"]: + prefix, simple = split_failed_class(cls) + suite = SUITE_BY_PREFIX.get(prefix) + if suite is None or simple not in tree.get(suite, {}): + continue + checked += 1 + if sel["full"] or simple in sel["selected"].get(suite, []): + continue + if (row["head_sha"][:10], cls) in accepted: + accepted_seen += 1 + continue + misses += 1 + missed.append(f"{row['date']} {row['head_sha'][:10]} {cls} files={len(row['changed_files'])}" + f" e.g. {row['changed_files'][0]}") + rows_out.append({"date": row["date"], "event": row["event"], "head_sha": row["head_sha"], + "suite": suite, "class": cls, "changed_files": row["changed_files"]}) + if not quiet: + print(f"map replay: {rows} rows ({full_rows} FULL), {checked} failing classes checked, {accepted_seen} accepted, " + f"{misses} misses (reported, not failing yet)") + for m in missed[:40]: + print("MAP-MISS", m) + if len(missed) > 40: + print(f"... {len(missed) - 40} more misses not printed; --misses-out FILE lists every one") + if misses_out: + with open(misses_out, "w", encoding="utf-8", newline="\n") as out: + for r in rows_out: + out.write(json.dumps(r, sort_keys=True) + "\n") + return misses + + +def map_keep(selection: object, suite: str) -> "tuple[set[str] | None, str]": + """The class names a shard of `suite` keeps, from the selection `map-select --out` wrote (#5459, live selection). + + Returns (names, reason). `names` is None, meaning the shard runs everything it was cut, whenever the selection + cannot be trusted: it is missing, not a JSON object, FULL, has no list of names for the suite, a name that is not + text, or an empty list. Empty is distrusted on purpose: the Guard classes are always selected, so a selection with + no class in it for a suite means the selection broke, not that nothing needs to run.""" + if not isinstance(selection, dict): + return None, "the selection is missing or unreadable" + if selection.get("full") is not False: + return None, f"the selection is FULL ({selection.get('reason') or 'no reason given'})" + chosen = selection.get("selected") + names = chosen.get(suite) if isinstance(chosen, dict) else None + if not isinstance(names, list) or not names or not all(isinstance(n, str) and n for n in names): + return None, f"the selection holds no usable class list for the {suite} suite" + return set(names), "" + + +def shadow_summary(result: dict) -> str: + """The step-summary text of the selection job: how many classes the map picked, and the estimated seconds.""" + lines = ["### Test map selection (#5459): the Darling PG, Lite and no-store Darling passes run only the classes below", ""] + if result.get("full"): + lines.append(f"Selection: FULL ({result.get('reason') or 'no reason given'}), so every shard ran everything.") + return "\n".join(lines) + "\n" + lines += ["Estimated seconds are the nightly's instrumented times, so they read high.", "", + "| suite | selected classes | of | estimated seconds |", "|---|---|---|---|"] + for suite in sorted(result.get("selected", {})): + lines.append(f"| {suite} | {len(result['selected'][suite])} | {result['total'].get(suite, 0)} | " + f"{result['seconds'].get(suite, 0):.0f} |") + for suite in sorted(result.get("why", {})): + counts: dict[str, int] = {} + for reason in result["why"][suite].values(): + counts[reason] = counts.get(reason, 0) + 1 + lines.append(f"- {suite} by reason: " + ", ".join(f"{k} {v}" for k, v in sorted(counts.items()))) + return "\n".join(lines) + "\n" + + +REPORT_ATTEMPT = re.compile(r"-a(\d+)\.xml$") + + +def failed_classes(reports: str) -> "tuple[dict[str, set[str]], dict[str, int], int]": + """The classes with a failed test in the xunit reports under `reports`, one subfolder per uploaded artifact: the + shards' `darling-tests-timing-N` and `lite-tests-timing-N`, and the Guard tests job's `guard-tests-timing-darling` + and `guard-tests-timing-lite` (#5459). The suite is the one the folder name says. + + A report named `...-a.xml` was written by run attempt N. Staleness is decided per artifact folder: within a + folder, the reports of the highest attempt present are kept and the reports of a lower attempt are left out and + counted as stale. A shard that failed in attempt 1 and passed in attempt 2 uploads its artifact again with + `overwrite`, so the folder holds attempt 2 only; if an old attempt-1 file is still there, it must not list its + attempt-1 class (#5459, run 37776652739). A shard that passed in attempt 1 is not re-run by "Re-run failed jobs", + so its folder still holds only `-a1` files and they count in attempt 2. A report with no attempt in its name + always counts. Returns ({suite: {Class}}, {suite: report count}, stale report count).""" + failed: dict[str, set[str]] = {"darling": set(), "lite": set()} + seen: dict[str, int] = {"darling": 0, "lite": 0} + stale = 0 + if not os.path.isdir(reports): + return failed, seen, stale + # (artifact folder, path) of every report, with the attempt its name carries (None when it carries none). + found: list[tuple[str, str, "int | None"]] = [] + for dirpath, _, names in os.walk(reports): + top = os.path.relpath(dirpath, reports).replace("\\", "/").split("/")[0] + for name in names: + if name.endswith(".xml"): + written_by = REPORT_ATTEMPT.search(name) + found.append((top, os.path.join(dirpath, name), int(written_by.group(1)) if written_by else None)) + latest: dict[str, int] = {} + for top, _, written_by in found: + if written_by is not None: + latest[top] = max(latest.get(top, written_by), written_by) + for top, path, written_by in found: + suite = "darling" if "darling" in top else "lite" if "lite" in top else "" + if not suite: + continue + if written_by is not None and written_by < latest[top]: + stale += 1 + continue + seen[suite] += 1 + try: + for _, el in ET.iterparse(path, events=("end",)): + if el.tag == "test" and (el.get("result") or "").lower() == "fail": + failed[suite].add(re.split(r"[.+/]", el.get("type") or "")[-1]) + el.clear() + except (ET.ParseError, OSError): + seen[suite] -= 1 # an unreadable report counts as no report + return failed, seen, stale + + +def failed_jobs(path: "str | None") -> "list[str] | None": + """Names of the jobs that concluded `failure`, from the latest attempt's job list (`gh api .../attempts/N/jobs`: one + `{"name", "conclusion"}` object per line, or the API's own `{"jobs": [...]}` object). None when it cannot be read. + A job that failed without a failed test (the whole-tree guards job ending on a leaked foreground thread) shows up + here and nowhere else.""" + if not path: + return None + try: + with open(path, encoding="utf-8") as fh: + text = fh.read() + except OSError: + return None + items: list = [] + try: + data = json.loads(text) + if isinstance(data, dict): + items = data["jobs"] if isinstance(data.get("jobs"), list) else [data] + elif isinstance(data, list): + items = data + except ValueError: + for line in text.splitlines(): + try: + items.append(json.loads(line)) + except ValueError: + continue + return sorted({str(j.get("name")) for j in items if isinstance(j, dict) and j.get("conclusion") == "failure"}) + + +def shadow_row(selection: "dict | None", failed: "dict[str, set[str]]", seen: "dict[str, int]", meta: dict, + stale: int = 0, jobs: "list[str] | None" = None) -> dict: + """One JSONL row per run: the selection's size and the classes that failed although it would not have run them. + A FULL (or missing) selection selects every class, so it can have no miss. `stale` counts the reports left out as + a lower attempt's in their artifact folder; `jobs` is the latest attempt's failed jobs (None when unknown).""" + if isinstance(selection, dict): + sel = selection + else: + # No selection file. When the selection job itself did not finish (a newer push cancels the run), say so: that + # is expected, not a script fault (#5459, run 37832135291). + job = str(meta.get("selection_job") or "") + sel = _full(f"the selection job ended {job}, so no selection was written" if job and job != "success" + else "no selection was written") + misses: list[dict] = [] + if not sel.get("full"): + for suite in sorted(failed): + chosen = set(sel.get("selected", {}).get(suite, [])) + misses += [{"suite": suite, "class": c} for c in sorted(failed[suite]) if c not in chosen] + row = {"v": 1, **meta, "full": bool(sel.get("full")), "reason": sel.get("reason", ""), + "selected": {s: len(v) for s, v in sel.get("selected", {}).items()}, + "total": sel.get("total", {}), "seconds": sel.get("seconds", {}), + "reports": seen, "stale_reports": stale, "failed": {s: sorted(v) for s, v in failed.items()}, + "misses": misses} + if jobs is not None: + row["failed_jobs"] = jobs + return row + + +# --------------------------------------------------------------------------------------------------------- +# CLI. +# --------------------------------------------------------------------------------------------------------- + +def _bool(s: str) -> bool: + return s.strip().lower() == "true" + + +def main(argv: list[str]) -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + ap.add_argument("--replay", action="store_true", help="replay the failure corpus against today's rules") + ap.add_argument("--corpus", default=CORPUS) + ap.add_argument("--map", help="with --replay: also replay the corpus against this class-to-file map (reports only)") + ap.add_argument("--misses-out", help="with --map: write every map miss, one JSON row per line, to this file") + sub = ap.add_subparsers(dest="cmd") + + ls = sub.add_parser("lite-scope", help="how much of the Lite suite a leg runs (workflow step)") + ls.add_argument("--event", required=True) + ls.add_argument("--shard", required=True) + for k in ("lite", "core", "root", "reads", "linked"): + ls.add_argument(f"--{k}", default="false") + + ds = sub.add_parser("darling-scope", help="what the build job's no-store Darling pass runs (workflow step)") + ds.add_argument("--event", required=True) + ds.add_argument("--gate", default="false", help="the Darling PostgreSQL gate's answer (darling-paths-filter.yml)") + for k in ("full", "core", "root"): + ds.add_argument(f"--{k}", default="true", help="a path that runs the whole Darling suite changed") + + ts = sub.add_parser("tree-scope", help="what darling-tree-guards runs once it runs (workflow step)") + ts.add_argument("--event", required=True) + ts.add_argument("--all-count", type=int, default=0) + ts.add_argument("--lite-count", type=int, default=0) + + sk = sub.add_parser("slow-skip", help="the Cost=Slow classes a pull request run leaves out (workflow step)") + sk.add_argument("--suite", required=True, choices=sorted(SUITE_DIRS)) + sk.add_argument("--event", required=True) + sk.add_argument("--base-ref", default="", help="the pull request's base branch (github.base_ref); main skips nothing") + sk.add_argument("--repo", default="") + sk.add_argument("--pr", default="") + sk.add_argument("--list", action="store_true", help="print every Cost=Slow class of the suite instead") + sk.add_argument("files", nargs="*", help="changed files; with --repo and --pr they come from the API instead") + + dc = sub.add_parser("decide", help="print every decision for a list of changed files") + dc.add_argument("--event", default="pull_request") + dc.add_argument("files", nargs="*") + + ms = sub.add_parser("map-select", help="which classes a pull request runs, from a class-to-file map (workflow step)") + ms.add_argument("--map", required=True, help="the test map (JSON, or .gz)") + ms.add_argument("--event", default="pull_request") + ms.add_argument("--base-ref", default="", help="the pull request's base branch (github.base_ref); main runs everything") + ms.add_argument("--base", help="the merge base; the drift is `git diff --no-renames ..`") + ms.add_argument("--drift-file", help="the drift as one path per line, instead of --base (`-` for none)") + ms.add_argument("--root", default=ROOT, help="the tree whose test classes are listed (default: this repository)") + ms.add_argument("--files-from", help="the changed files as one path per line, in place of the arguments") + ms.add_argument("--out", help="also write the selection JSON to this file (stdout keeps the notice and the JSON)") + ms.add_argument("--summary", help="also write the shadow step-summary markdown to this file") + ms.add_argument("files", nargs="*") + + mk = sub.add_parser("map-keep", help="the classes a shard keeps from a selection; FULL when it cannot be trusted (workflow step)") + mk.add_argument("--selection", required=True, help="the JSON map-select --out wrote; missing or unreadable counts as FULL") + mk.add_argument("--suite", required=True, choices=("darling", "lite")) + + sc = sub.add_parser("shadow-check", help="classes that failed but a map selection would not have run (workflow step)") + sc.add_argument("--selection", help="the JSON map-select --out wrote; missing or unreadable counts as FULL") + sc.add_argument("--reports", required=True, help="a folder with one subfolder per timing artifact") + sc.add_argument("--out", required=True, help="the one-row JSONL file to write") + sc.add_argument("--meta", default="{}", help="JSON object merged into the row (run, sha, pr...)") + sc.add_argument("--attempt", type=int, help="the run attempt; accepted for callers, but it no longer filters: " + "staleness is decided per artifact folder (the `attempt` of --meta " + "still lands in the row)") + sc.add_argument("--jobs", help="the latest attempt's jobs, one {name, conclusion} JSON object per line; " + "the row lists the failed ones") + + args = ap.parse_args(argv) + if args.replay: + misses = replay(args.corpus) + if args.map: + map_replay(load_map(args.map), args.corpus, misses_out=args.misses_out) # reports; does not fail the replay yet + return 1 if misses else 0 + if args.cmd == "map-select": + files = list(args.files) + if args.files_from: + with open(args.files_from, encoding="utf-8") as fh: + files += [ln.strip() for ln in fh if ln.strip()] + test_map = load_map(args.map) + drift: "list[str] | None" = None + if args.drift_file == "-": + drift = [] + elif args.drift_file: + with open(args.drift_file, encoding="utf-8") as fh: + drift = [ln.strip() for ln in fh if ln.strip()] + elif args.base and isinstance(test_map, dict) and isinstance(test_map.get("sha"), str): + drift = git_drift(test_map["sha"], args.base) + discovered = scan_classes(args.root) + if isinstance(test_map, dict) and isinstance(test_map.get("classes"), dict): + # The map holds the suites the nightly instruments (darling, lite). The deprecated suites in the tree + # are not in it and no shard runs them as a test-map class, so they are not "new classes". + discovered = {s: v for s, v in discovered.items() if s in test_map["classes"]} + result = map_select(test_map, files, drift, discovered, event=args.event, base_ref=args.base_ref, + **selection_args(Rules())) + if result["full"]: + print(f"::notice::test map: running everything ({result['reason']})") + print(json.dumps(result, indent=1)) + if args.out: + with open(args.out, "w", encoding="utf-8", newline="\n") as fh: + json.dump(result, fh) + if args.summary: + with open(args.summary, "w", encoding="utf-8", newline="\n") as fh: + fh.write(shadow_summary(result)) + return 0 + if args.cmd == "map-keep": + # Line 1 is `SELECTED ` (then one class name per line) or `FULL `; the exit code is always 0, so a + # caller that cannot read the answer runs everything. + try: + with open(args.selection, encoding="utf-8") as fh: + chosen = json.load(fh) + except (OSError, ValueError): + chosen = None + names, why = map_keep(chosen, args.suite) + if names is None: + print(f"FULL {why}") + else: + print(f"SELECTED {len(names)}") + print("\n".join(sorted(names))) + return 0 + if args.cmd == "shadow-check": + selection = None + if args.selection: + try: + with open(args.selection, encoding="utf-8") as fh: + selection = json.load(fh) + except (OSError, ValueError): + selection = None + try: + meta = json.loads(args.meta) + except ValueError: + meta = {} + meta = meta if isinstance(meta, dict) else {} + failed, seen, stale = failed_classes(args.reports) + row = shadow_row(selection, failed, seen, meta, stale, failed_jobs(args.jobs)) + with open(args.out, "w", encoding="utf-8", newline="\n") as fh: + fh.write(json.dumps(row, sort_keys=True) + "\n") + for m in row["misses"]: + print(f"::notice title=Shadow miss::{m['suite']} {m['class']} failed and the test map would not have run it") + print(f"shadow-check: full={row['full']} misses={len(row['misses'])} reports={row['reports']}") + return 0 + if args.cmd == "lite-scope": + mode = lite_scope(args.event, _bool(args.lite), _bool(args.core), _bool(args.root), _bool(args.reads), + _bool(args.linked)) + run = mode == "full" or (mode == "reads" and args.shard == "0") + print(f"mode={mode}") + print(f"run={'true' if run else 'false'}") + return 0 + if args.cmd == "darling-scope": + scope = darling_scope(args.event, True, args.event != "release" and _bool(args.gate), + _bool(args.full) or _bool(args.core) or _bool(args.root)) + print(f"scope={scope}") + print("filter=" + scope_filter(scope)) + return 0 + if args.cmd == "tree-scope": + scope = tree_scope(args.event, args.all_count, args.lite_count) + print(f"scope={scope}") + print("filter=" + scope_filter(scope)) + return 0 + if args.cmd == "slow-skip": + index = SlowIndex() + if args.list: + print("\n".join(sorted(index.slow[args.suite]))) + return 0 + files = args.files + if not files and args.repo and args.pr.isdigit() and args.event == "pull_request": + files = changed_files_of_pr(args.repo, args.pr) + print("\n".join(index.skip(files, args.event, args.base_ref)[args.suite])) + return 0 + if args.cmd == "decide": + print(json.dumps(decide(args.files, args.event, Rules()), indent=1)) + return 0 + ap.print_help() + return 2 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/.github/scripts/darling-upgrade-shapes.ps1 b/.github/scripts/darling-upgrade-shapes.ps1 new file mode 100644 index 0000000000..fc3a597fa9 --- /dev/null +++ b/.github/scripts/darling-upgrade-shapes.ps1 @@ -0,0 +1,265 @@ +<# +.SYNOPSIS + Runs ONE leg of the Darling upgrade-shape check (#5627): builds an older install the way an operator would + have, then runs this checkout's install-darling.ps1 or upgrade-darling.ps1 over it and fails the leg, with a + one-line reason, when the outcome is not the expected one. + +.DESCRIPTION + Called by .github/workflows/darling-upgrade-shapes.yml, one leg per job, on an elevated hosted Windows runner. + The older install is the REAL release: its zip is downloaded, its own install-darling.ps1 runs, and its + service is started until it has extracted pg-runtime (pg-runtime.stamp), then stopped. Nothing about the + base is built by hand, because the owner of the extracted files is whatever account ran the service when it + extracted them, and that is the thing the install/upgrade pre-lock check has to judge correctly. + + Shapes (the leg name is -" }, Strings(r.GetProperty("pre"))); + Assert.Equal(new[] { "", "Fix" }, Strings(r.GetProperty("headings"))); + Assert.Equal(new[] { "label" }, Strings(r.GetProperty("fields"))); + Assert.Equal(new[] { " prose" }, Strings(r.GetProperty("paragraphs"))); + Assert.Equal(0, r.GetProperty("imgs").GetInt32()); + } +} diff --git a/Darling/Darling.Tests/AlertHistoryDetailLabelsTests.cs b/Darling/Darling.Tests/AlertHistoryDetailLabelsTests.cs index 7e551cdd9d..4ac92d5787 100644 --- a/Darling/Darling.Tests/AlertHistoryDetailLabelsTests.cs +++ b/Darling/Darling.Tests/AlertHistoryDetailLabelsTests.cs @@ -45,7 +45,7 @@ public sealed class AlertHistoryDetailLabelsTests "const parse = new Function(src.slice(start, end) + '\\nreturn parseDetailFields;')();" + "console.log(JSON.stringify(parse(JSON.parse(process.argv[2]).join('\\n'))));"; - /// Runs the shipped parser over the lines, or returns false when Node is not installed. + /// Runs the shipped parser over the lines. When Node is not installed, the test is reported as skipped. private static bool TryParse(string[] lines, out List<(string? Label, string Value)> fields) { fields = new List<(string?, string)>(); @@ -68,6 +68,7 @@ private static bool TryParse(string[] lines, out List<(string? Label, string Val } catch (Win32Exception) { + Assert.Skip("Node is not installed, so the shipped page script cannot be run."); return false; } diff --git a/Darling/Darling.Tests/AlertHistoryDismissBehaviourTests.cs b/Darling/Darling.Tests/AlertHistoryDismissBehaviourTests.cs new file mode 100644 index 0000000000..fc6072e0df --- /dev/null +++ b/Darling/Darling.Tests/AlertHistoryDismissBehaviourTests.cs @@ -0,0 +1,202 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.ComponentModel; +using System.Diagnostics; +using System.Linq; +using System.Text.Json; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// Dismiss Selected and Dismiss All on the web Alert History page, from the shipped alerts.js run under +/// Node (alert-history-dismiss-harness.mjs) against a recording fetch: the controls exist only for a seat +/// that can edit, the right keys are posted to /api/alert-history/dismiss, the checked set survives a +/// rebuild, a refusal is shown and never reported as success, and the reply's counts are summarised. +/// +public sealed class AlertHistoryDismissBehaviourTests +{ + private static JsonElement Run(string scenario) + { + var psi = new ProcessStartInfo("node") { RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false }; + psi.ArgumentList.Add(PathTo("Darling", "Darling.Tests", "alert-history-dismiss-harness.mjs")); + psi.ArgumentList.Add(PathTo("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js")); + psi.ArgumentList.Add(scenario); + Process proc; + try + { + proc = Process.Start(psi)!; + } + catch (Win32Exception) + { + Assert.Skip("Node is not installed, so the shipped page script cannot be run."); + return default; + } + + using (proc) + { + var error = proc.StandardError.ReadToEndAsync(); + var output = proc.StandardOutput.ReadToEnd().Trim(); + if (!proc.WaitForExit(20000)) + { + proc.Kill(entireProcessTree: true); + Assert.Fail("the Alert History dismiss harness did not finish in 20 s for scenario " + scenario); + } + + Assert.True(proc.ExitCode == 0, "the Alert History dismiss harness failed for scenario " + scenario + ": " + error.Result); + using var doc = JsonDocument.Parse(output); + return doc.RootElement.Clone(); + } + } + + private static string[] Metrics(JsonElement post) => + post.GetProperty("req").GetProperty("alerts").EnumerateArray().Select(a => a.GetProperty("metric_name").GetString()!).ToArray(); + + [Fact] + public void ASeatThatCannotEdit_SeesNoCheckboxesAndNoDismissButtons() + { + var r = Run("readOnly"); + Assert.Equal(0, r.GetProperty("checkboxes").GetInt32()); + Assert.False(r.GetProperty("hasSelected").GetBoolean()); + Assert.False(r.GetProperty("hasAll").GetBoolean()); + } + + [Fact] + public void DismissSelected_PostsTheCheckedRowsKeys_AndRefreshesTheList() + { + var r = Run("selected"); + Assert.Equal(3, r.GetProperty("boxes").GetInt32()); + Assert.True(r.GetProperty("selectedDisabledBefore").GetBoolean()); + var post = Assert.Single(r.GetProperty("posts").EnumerateArray()); + Assert.Equal("/api/alert-history/dismiss", post.GetProperty("url").GetString()); + Assert.Equal("application/json", post.GetProperty("contentType").GetString()); + Assert.Equal(new[] { "High CPU 1", "High CPU 3" }, Metrics(post)); + var first = post.GetProperty("req").GetProperty("alerts")[0]; + Assert.Equal(1, first.GetProperty("server_id").GetInt32()); + Assert.Equal("2026-01-01T11:59:00.000Z", first.GetProperty("alert_time").GetString()); + Assert.Equal("Dismissed 2 of 2", r.GetProperty("status").GetString()); + Assert.Equal(1, r.GetProperty("rowsAfter").GetInt32()); + Assert.Equal(2, r.GetProperty("reads").GetInt32()); + } + + [Fact] + public void WithShowDismissedOn_ADismissedRowStaysAndSaysDismissed() + { + var r = Run("showDismissed"); + Assert.Equal(2, r.GetProperty("rows").GetInt32()); + Assert.True(r.GetProperty("dismissedWord").GetBoolean()); + Assert.Equal(0, r.GetProperty("firstHasBox").GetInt32()); + } + + [Fact] + public void DismissAll_PostsEveryLiveRowTheFiltersShow() + { + var r = Run("all"); + var filtered = Assert.Single(r.GetProperty("filtered").EnumerateArray()); + Assert.Equal(new[] { 2 }, filtered.EnumerateArray().Select(e => e.GetInt32()).ToArray()); + var post = Assert.Single(r.GetProperty("all").EnumerateArray()); + Assert.Equal("/api/alert-history/dismiss", post.GetProperty("url").GetString()); + Assert.Equal(3, post.GetProperty("keys").GetArrayLength()); + } + + [Fact] + public void ALongList_GoesInRequestsOfAtMostTheRoutesCap_AndTheCountsAreAddedUp() + { + var r = Run("chunks"); + Assert.Equal(new[] { 1000, 500 }, r.GetProperty("sizes").EnumerateArray().Select(e => e.GetInt32()).ToArray()); + Assert.Equal("Dismissed 1500 of 1500", r.GetProperty("status").GetString()); + } + + [Fact] + public void TheCheckedSet_SurvivesARebuild_AndClearsWhenAFilterChanges() + { + var r = Run("survives"); + Assert.Equal(new[] { false, false, true }, r.GetProperty("afterPoll").EnumerateArray().Select(e => e.GetBoolean()).ToArray()); + Assert.Equal(new[] { false, false, true }, r.GetProperty("afterReopen").EnumerateArray().Select(e => e.GetBoolean()).ToArray()); + Assert.Equal("Dismiss Selected (1)", r.GetProperty("label").GetString()); + Assert.DoesNotContain(true, r.GetProperty("afterFilter").EnumerateArray().Select(e => e.GetBoolean())); + } + + [Fact] + public void ARefusal_ShowsItsText_NeverClaimsSuccess_AndKeepsTheSelection() + { + var r = Run("refused"); + Assert.Equal("This seat is read-only.", r.GetProperty("status").GetString()); + Assert.Equal(2, r.GetProperty("rows").GetInt32()); + Assert.True(r.GetProperty("stillChecked").GetBoolean()); + var expired = r.GetProperty("status401").GetString()!; + Assert.DoesNotContain("Dismissed", expired); + Assert.Contains("Session expired", expired); + } + + [Fact] + public void DismissAllPrompt_SaysStore_AndWhenThePageIsTruncatedSaysTheRestStayLive() + { + var r = Run("truncatedPrompt"); + var full = r.GetProperty("fullAsk").GetString()!; + Assert.Contains("remain in the store", full); + Assert.DoesNotContain("database", full); + Assert.DoesNotContain("More alerts exist", full); + var truncated = r.GetProperty("truncatedAsk").GetString()!; + Assert.Contains("remain in the store", truncated); + Assert.Contains("More alerts exist than the 2 listed", truncated); + Assert.Contains("the rest stay live", truncated); + } + + [Fact] + public void CopyRowAndCopyTable_LeaveOutTheSelectColumn_ButKeepMute() + { + var r = Run("copy"); + var all = r.GetProperty("copyAll").GetString()!.Split('\n'); + var header = all[0].Split('\t'); + Assert.DoesNotContain("Select", header); + Assert.Contains("Mute", header); + var row = r.GetProperty("copyRow").GetString()!.Split('\t'); + Assert.Equal(header.Length, row.Length); + Assert.Equal(header.Length, all[1].Split('\t').Length); + Assert.Equal(header.Length + 1, r.GetProperty("cells").GetInt32()); + } + + [Fact] + public void AFailedPollRead_KeepsTheCheckedSet() + { + var r = Run("failedPoll"); + Assert.Equal("Dismiss Selected (1)", r.GetProperty("labelAfterFailedRead").GetString()); + Assert.Equal(new[] { false, true }, r.GetProperty("afterGoodRead").EnumerateArray().Select(e => e.GetBoolean()).ToArray()); + Assert.Equal("Dismiss Selected (1)", r.GetProperty("labelAfterGoodRead").GetString()); + } + + [Fact] + public void ABoxTickedWhileADismissIsInFlight_IsNotClearedAndNotSent() + { + var r = Run("tickedInFlight"); + var sent = Assert.Single(r.GetProperty("sent").EnumerateArray()); + Assert.Equal(new[] { "High CPU 1" }, sent.EnumerateArray().Select(e => e.GetString()!).ToArray()); + Assert.Equal("Dismiss Selected (1)", r.GetProperty("label").GetString()); + Assert.Equal(new[] { false, true }, r.GetProperty("checked").EnumerateArray().Select(e => e.GetBoolean()).ToArray()); + } + + [Fact] + public void TheStatusLine_SaysHowManyWereDismissedAndHowManyWereAlreadyGone() + { + var r = Run("counts"); + Assert.Equal("Dismissed 1 of 4; 3 were already gone", r.GetProperty("status").GetString()); + } + + [Fact] + public void ThePageSendsOnlyTheRoutesKeysThroughTheSharedWriteHelper() + { + var page = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "pages", "alerts.js").ReplaceLineEndings("\n"); + Assert.Contains("apiSend(\"POST\", \"/api/alert-history/dismiss\"", page); + Assert.Contains("const DISMISS_CHUNK = 1000;", page); + Assert.Contains("\nconst selected = new Map();", page); + } +} diff --git a/Darling/Darling.Tests/AlertHistoryDismissTests.cs b/Darling/Darling.Tests/AlertHistoryDismissTests.cs new file mode 100644 index 0000000000..c1a747e45b --- /dev/null +++ b/Darling/Darling.Tests/AlertHistoryDismissTests.cs @@ -0,0 +1,347 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.IO; +using System.Linq; +using System.Net; +using System.Text; +using System.Text.Json; +using System.Text.RegularExpressions; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.TestHost; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using Npgsql; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Hosting; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// The web Alert History dismiss (#4843): the pure body parser, the role grant that lets the web host's +/// viewer identity write it, and the route itself. The live half connects AS a role holding exactly the +/// managed viewer grant on config_alert_log (the statement is lifted out of the managed +/// provisioning SQL, so a drift in it changes what these tests run), never as the owner. +/// +[Collection("live-postgres")] +public sealed class AlertHistoryDismissTests +{ + private static readonly string RunSuffix = Guid.NewGuid().ToString("N")[..8]; + private static readonly string DismissRole = "dismiss_only_" + RunSuffix; + private const string RolePassword = "DismissTestPw0123456789abcdef01"; + + private static string Cs() + { + var cs = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to run the alert dismiss live tests."); + return cs!; + } + + /* ═══════════════════════════ the pure parser ═══════════════════════════ */ + + [Fact] + public void ParseDismissBody_DedupesAndKeepsFirstSeenOrder() + { + var ok = DarlingWebEndpoints.ParseDismissBody( + """{"alerts":[{"alert_time":"2026-10-04T10:00:00","server_id":1,"metric_name":"High CPU"},{"alert_time":"2026-10-04T10:00:00Z","server_id":1,"metric_name":"High CPU"},{"alert_time":"2026-10-04T11:00:00","server_id":2,"metric_name":"Blocking"}]}""", + out var keys, out var refusal); + + Assert.True(ok, refusal); + Assert.Equal(2, keys.Count); + Assert.Equal(1, keys[0].ServerId); + Assert.Equal(2, keys[1].ServerId); + Assert.Equal(DateTimeKind.Unspecified, keys[0].AlertTime.Kind); + } + + [Theory] + [InlineData("not json")] + [InlineData("[]")] + [InlineData("{}")] + [InlineData("""{"alerts":[]}""")] + [InlineData("""{"alerts":"x"}""")] + [InlineData("""{"alerts":[{"alert_time":"2026-10-04T10:00:00","server_id":"1","metric_name":"m"}]}""")] + [InlineData("""{"alerts":[{"alert_time":"nope","server_id":1,"metric_name":"m"}]}""")] + [InlineData("""{"alerts":[{"alert_time":"2026-10-04T10:00:00","server_id":1,"metric_name":""}]}""")] + [InlineData("""{"alerts":[{"alert_time":"2026-10-04T10:00:00","server_id":1}]}""")] + [InlineData("""{"alerts":[7]}""")] + [InlineData("""{"alerts":[],"alerts":[]}""")] + public void ParseDismissBody_RefusesABadBody(string body) + { + Assert.False(DarlingWebEndpoints.ParseDismissBody(body, out _, out var refusal)); + Assert.False(string.IsNullOrEmpty(refusal)); + } + + [Fact] + public void ParseDismissBody_CapsTheKeyCount() + { + string Body(int n) => "{\"alerts\":[" + string.Join(",", Enumerable.Range(0, n) + .Select(i => $"{{\"alert_time\":\"2026-10-04T10:00:00\",\"server_id\":{i},\"metric_name\":\"m\"}}")) + "]}"; + + Assert.True(DarlingWebEndpoints.ParseDismissBody(Body(DarlingWebEndpoints.MaxDismissKeys), out var keys, out _)); + Assert.Equal(DarlingWebEndpoints.MaxDismissKeys, keys.Count); + Assert.False(DarlingWebEndpoints.ParseDismissBody(Body(DarlingWebEndpoints.MaxDismissKeys + 1), out _, out _)); + } + + /* ═══════════════════════════ the write gate ═══════════════════════════ */ + + [Theory] + [InlineData(true, "POST", true)] + [InlineData(false, "POST", false)] + [InlineData(false, "PUT", false)] + [InlineData(false, "DELETE", false)] + public void TheWriteGate_RefusesAReadOnlySeat_OnTheDismissRoute(bool canEdit, string method, bool expected) => + Assert.Equal(expected, DarlingWebSeat.IsRequestAllowed( + new DarlingWebSeat("who", canEdit), method, "/api/alert-history/dismiss")); + + /* ═══════════════════════════ the grant text ═══════════════════════════ */ + + [Fact] + public void ManagedProvisioning_GrantsViewerExactlyTheDismissedColumn_AndMcpNothing() + { + var sql = DarlingManagedRoles.BuildProvisioningSql( + ProvisioningTestSecrets.Admin, ProvisioningTestSecrets.Viewer, ProvisioningTestSecrets.Mcp, 15); + + Assert.Contains("GRANT UPDATE (dismissed) ON config.config_alert_log TO viewer;", sql, StringComparison.Ordinal); + Assert.DoesNotContain("config.config_alert_log TO mcp", sql, StringComparison.Ordinal); + + /* No table-level UPDATE, INSERT or DELETE on the alert log for viewer, in any spelling. */ + var tableLevel = Regex.Matches( + Regex.Replace(sql, @"(?m)^\s*--.*$", ""), + @"GRANT\s+(?[^;(]*?)\s+ON\s+config\.config_alert_log\s+TO\s+[^;]*viewer[^;]*;"); + Assert.Empty(tableLevel); + } + + /* ═══════════════════════════ live: AS the grant-only role ═══════════════════════════ */ + + [Fact] + public async Task TheColumnGrant_LetsTheRoleDismiss_AndNothingElse_AndLeavesTheReadOnlyProbeFalse() + { + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(Cs(), ct); + await using (var owner = new NpgsqlConnection(scratch.ConnectionString)) + { + await owner.OpenAsync(ct); + await PgMigrations.MigrateAsync(owner, ct); + await CreateDismissOnlyRoleAsync(owner, ct); + await Exec(owner, SeedSql, ct); + } + + var bodySucceeded = false; + try + { + await using var role = new NpgsqlConnection(RoleConnectionString(scratch.ConnectionString)); + await role.OpenAsync(ct); + + /* Negative control: the same role, same statement, BEFORE the shipped grant, is refused. This is what + makes the success below attributable to the grant and not to a role that could always write. */ + var dismissKey = new AlertDismissKey(new DateTime(2026, 10, 4, 10, 0, 0), -7001, "High CPU"); + await using (var before = AlertDismissStore.CreateDismissCommand( + NpgsqlDataSource.Create(RoleConnectionString(scratch.ConnectionString)), new[] { dismissKey }, 30)) + { + var refused = await Assert.ThrowsAsync(() => before.ExecuteNonQueryAsync(ct)); + Assert.Equal("42501", refused.SqlState); + } + + await using (var owner = new NpgsqlConnection(scratch.ConnectionString)) + { + await owner.OpenAsync(ct); + await GrantDismissedColumnAsync(owner, ct); + } + + /* The WPF read-only probe, verbatim, as the grant-only role: a column grant is not a table grant. */ + Assert.False(await Scalar(role, "SELECT has_table_privilege('config_alert_log', 'UPDATE')", ct)); + Assert.True(await Scalar(role, "SELECT has_column_privilege('config_alert_log', 'dismissed', 'UPDATE')", ct)); + Assert.False(await Scalar(role, "SELECT has_column_privilege('config_alert_log', 'muted', 'UPDATE')", ct)); + + /* The shared write core, as that role. */ + await using (var command = AlertDismissStore.CreateDismissCommand( + NpgsqlDataSource.Create(RoleConnectionString(scratch.ConnectionString)), new[] { dismissKey }, 30)) + { + Assert.Equal(1, await command.ExecuteNonQueryAsync(ct)); + } + + /* Every other column, INSERT and DELETE: 42501. */ + foreach (var statement in new[] + { + "UPDATE config_alert_log SET muted = TRUE", + "UPDATE config_alert_log SET metric_name = 'x'", + "UPDATE config_alert_log SET dismissed = FALSE, send_error = 'x'", + "INSERT INTO config_alert_log (alert_time, server_id, server_name, metric_name, current_value, threshold_value) VALUES (now(), -1, 's', 'm', 1, 1)", + "DELETE FROM config_alert_log", + }) + { + var ex = await Assert.ThrowsAsync(() => Exec(role, statement, ct)); + Assert.Equal("42501", ex.SqlState); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, DropRoleAsync); + } + } + + [Fact] + public async Task TheRoute_Answers200_400_415_AsTheGrantOnlyRole_AndReportsUnknownAndAlreadyDismissed() + { + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(Cs(), ct); + await using (var owner = new NpgsqlConnection(scratch.ConnectionString)) + { + await owner.OpenAsync(ct); + await PgMigrations.MigrateAsync(owner, ct); + await ProvisionDismissOnlyRoleAsync(owner, ct); + await Exec(owner, SeedSql, ct); + } + + var bodySucceeded = false; + try + { + await using var ds = NpgsqlDataSource.Create(RoleConnectionString(scratch.ConnectionString)); + var builder = WebApplication.CreateBuilder(new WebApplicationOptions + { + ContentRootPath = Path.Combine(RepoFile.Root, "Darling", "PerformanceMonitor.Darling.Service"), + WebRootPath = "wwwroot", + }); + builder.WebHost.UseTestServer(); + builder.Logging.ClearProviders(); + builder.Services.AddSingleton(ds); + await using var app = builder.Build(); + DarlingWebEndpoints.MapAll(app, ds, new CollectorRuntimeState(), new CapturingTestLogger()); + await app.StartAsync(ct); + using var server = app.GetTestServer(); + + async Task<(int Status, string Body)> Post(string? contentType, string body) + { + var context = await server.SendAsync(request => + { + request.Request.Method = "POST"; + request.Request.Path = "/api/alert-history/dismiss"; + request.Request.Headers.Host = "localhost"; + if (contentType is not null) + { + request.Request.ContentType = contentType; + } + + request.Request.Body = new MemoryStream(Encoding.UTF8.GetBytes(body)); + }); + using var reader = new StreamReader(context.Response.Body); + return (context.Response.StatusCode, await reader.ReadToEndAsync(ct)); + } + + static string Alert(string time, int server, string metric) => + $"{{\"alert_time\":\"{time}\",\"server_id\":{server},\"metric_name\":\"{metric}\"}}"; + + /* 415 and 400 never reach the store. */ + Assert.Equal(StatusCodes.Status415UnsupportedMediaType, (await Post("text/plain", "{}")).Status); + Assert.Equal(StatusCodes.Status415UnsupportedMediaType, (await Post(null, "{}")).Status); + Assert.Equal(StatusCodes.Status400BadRequest, (await Post("application/json", "nope")).Status); + Assert.Equal(StatusCodes.Status400BadRequest, (await Post("application/json", "{\"alerts\":[]}")).Status); + Assert.Equal(StatusCodes.Status400BadRequest, (await Post("application/json", "{\"alerts\":[],\"alerts\":[]}")).Status); + + /* Two real rows (one duplicated), one unknown key. */ + var first = await Post("application/json; charset=utf-8", "{\"alerts\":[" + string.Join(",", + Alert("2026-10-04T10:00:00", -7001, "High CPU"), + Alert("2026-10-04T10:00:00", -7001, "High CPU"), + Alert("2026-10-04T10:05:00", -7001, "Blocking"), + Alert("2026-10-04T10:00:00", -7999, "No Such")) + "]}"); + Assert.Equal(StatusCodes.Status200OK, first.Status); + using (var doc = JsonDocument.Parse(first.Body)) + { + var root = doc.RootElement; + Assert.Equal(3, root.GetProperty("requested").GetInt32()); + Assert.Equal(2, root.GetProperty("dismissed").GetInt32()); + Assert.Equal(0, root.GetProperty("already_dismissed").GetInt32()); + Assert.Equal(1, root.GetProperty("unknown").GetInt32()); + } + + /* Repeating it dismisses nothing new: both rows are already dismissed. */ + var again = await Post("application/json", "{\"alerts\":[" + Alert("2026-10-04T10:00:00", -7001, "High CPU") + "]}"); + using (var doc = JsonDocument.Parse(again.Body)) + { + Assert.Equal(0, doc.RootElement.GetProperty("dismissed").GetInt32()); + Assert.Equal(1, doc.RootElement.GetProperty("already_dismissed").GetInt32()); + } + + await using var verify = new NpgsqlConnection(scratch.ConnectionString); + await verify.OpenAsync(ct); + Assert.Equal(2L, await Scalar(verify, "SELECT count(*) FROM config_alert_log WHERE server_id = -7001 AND dismissed", ct)); + Assert.Equal(1L, await Scalar(verify, "SELECT count(*) FROM config_alert_log WHERE server_id = -7001 AND NOT dismissed", ct)); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, DropRoleAsync); + } + } + + /* ═══════════════════════════ helpers ═══════════════════════════ */ + + private const string SeedSql = @" +INSERT INTO config_alert_log (alert_time, server_id, server_name, metric_name, current_value, threshold_value) VALUES + ('2026-10-04 10:00:00', -7001, 'srv', 'High CPU', 90, 80), + ('2026-10-04 10:05:00', -7001, 'srv', 'Blocking', 5, 1), + ('2026-10-04 10:10:00', -7001, 'srv', 'Deadlocks', 2, 1)"; + + private static Task CreateDismissOnlyRoleAsync(NpgsqlConnection owner, CancellationToken ct) => + Exec(owner, $@" +CREATE ROLE {DismissRole} LOGIN NOSUPERUSER PASSWORD '{RolePassword}'; +GRANT USAGE ON SCHEMA config TO {DismissRole}; +GRANT SELECT ON config.config_alert_log TO {DismissRole};", ct); + + /// Gives the role ONLY the managed viewer's alert-log grant, lifted from the managed provisioning SQL + /// so the test runs the shipped statement. + private static async Task GrantDismissedColumnAsync(NpgsqlConnection owner, CancellationToken ct) + { + var managed = DarlingManagedRoles.BuildProvisioningSql( + ProvisioningTestSecrets.Admin, ProvisioningTestSecrets.Viewer, ProvisioningTestSecrets.Mcp, 15); + var grant = Regex.Match(managed, @"GRANT UPDATE \(dismissed\) ON config\.config_alert_log TO viewer;"); + Assert.True(grant.Success, "the managed provisioning SQL no longer grants viewer the dismissed column"); + + await Exec(owner, grant.Value.Replace(" TO viewer;", " TO " + DismissRole + ";", StringComparison.Ordinal), ct); + } + + private static async Task ProvisionDismissOnlyRoleAsync(NpgsqlConnection owner, CancellationToken ct) + { + await CreateDismissOnlyRoleAsync(owner, ct); + await GrantDismissedColumnAsync(owner, ct); + } + + private static Task DropRoleAsync(NpgsqlConnection owner, CancellationToken ct) => + Exec(owner, $"DROP OWNED BY {DismissRole}; DROP ROLE IF EXISTS {DismissRole}", ct); + + private static string RoleConnectionString(string baseConnectionString) => + new NpgsqlConnectionStringBuilder(baseConnectionString) + { + Username = DismissRole, + Password = RolePassword, + SearchPath = "config,public", + Pooling = false, + }.ConnectionString; + + private static async Task Exec(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task Scalar(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + return (T)(await command.ExecuteScalarAsync(ct))!; + } +} diff --git a/Darling/Darling.Tests/AlertHistoryLazyRenderTests.cs b/Darling/Darling.Tests/AlertHistoryLazyRenderTests.cs index 1ce730384e..9851dde10e 100644 --- a/Darling/Darling.Tests/AlertHistoryLazyRenderTests.cs +++ b/Darling/Darling.Tests/AlertHistoryLazyRenderTests.cs @@ -66,10 +66,17 @@ pins against regressing. */ Assert.Contains("if (built || !node.open) return;", detailCell, StringComparison.Ordinal); Assert.Contains("mount(placeholder, detailBody(a));", detailCell, StringComparison.Ordinal); + /* #5241: an Analysis row's advice is fetched here, in the toggle handler, once per opened row - never by the + list read, and never for a row nobody opened. */ + Assert.Contains("if (isAnalysisAlert(a)) {", detailCell, StringComparison.Ordinal); + Assert.Contains("await alertAdvice(a)", detailCell, StringComparison.Ordinal); + Assert.Contains("mount(placeholder, detailBody(a, items));", detailCell, StringComparison.Ordinal); + Assert.DoesNotContain("include_details", js, StringComparison.Ordinal); + /* The control: detailBody is where that work actually happens, so the absence above is real and not a matcher for text that was never going to appear anywhere in this file. Same field/error shape as the pre-#4194 eager version - only when it runs has changed. */ - var detailBody = ExtractFunction(js, "function detailBody(a) {"); + var detailBody = ExtractFunction(js, "function detailBody(a, details) {"); Assert.Contains("parseDetailFields(", detailBody, StringComparison.Ordinal); Assert.Contains("fieldRow", detailBody, StringComparison.Ordinal); Assert.Contains("detail-fields", detailBody, StringComparison.Ordinal); @@ -117,7 +124,10 @@ public void SixtySecondPoll_NeverShowsTheLoadingStrip_OnlyTheFirstMountDoes() must not call it - the previous table stays exactly as it is until the new page is ready. */ var js = AlertsJs; var loadingStripUses = Regex.Matches(js, Regex.Escape("loadingStrip(")).Count; - Assert.Equal(1, loadingStripUses); + /* The first mount in renderAlerts, and (#5241) the one in an opened Analysis row's own expansion while its advice + loads; neither is on the poll path. */ + Assert.Equal(2, loadingStripUses); + Assert.Contains("mount(placeholder, [loadingStrip()]);", ExtractFunction(js, "function detailCell(a) {"), StringComparison.Ordinal); var refreshAlerts = ExtractFunction(js, "async function refreshAlerts(state) {"); Assert.DoesNotContain("loadingStrip", refreshAlerts, StringComparison.Ordinal); diff --git a/Darling/Darling.Tests/AlertHistoryRangePageTests.cs b/Darling/Darling.Tests/AlertHistoryRangePageTests.cs new file mode 100644 index 0000000000..f1d05c2328 --- /dev/null +++ b/Darling/Darling.Tests/AlertHistoryRangePageTests.cs @@ -0,0 +1,253 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.ComponentModel; +using System.Diagnostics; +using System.Linq; +using System.Text.Json; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// Alert History's range, row-limit, server and Show dismissed choices (#4843): source pins for what the page asks +/// get_alert_history for, and a run of the shipped page under Node (alert-history-range-harness.mjs) for what +/// it does with each choice. Node is skipped when it is not installed. +/// +public sealed class AlertHistoryRangePageTests +{ + private static string Page() => + ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "pages", "alerts.js").ReplaceLineEndings("\n"); + + [Fact] + public void ThePageReadsGetAlertHistoryWithTheFourChoicesAndNeverAFixedWindow() + { + var page = Page(); + Assert.DoesNotContain("hours_back: 24", page); + Assert.DoesNotContain("limit: 200 }", page); + Assert.Contains("readTool(\"get_alert_history\", req.params)", page); + // #5562: the range is the shared picker's spec, read as of each read; a finished range also sends its end. + Assert.Contains("windowResultOfSpec(choices.spec);", page); + Assert.Contains("hours_back: w.hours", page); + Assert.Contains("as_of: w.asOf", page); + // Review r1 H2: the reply is trimmed to the range start, since the read takes whole hours. + Assert.Contains("trimAlertsToStart(res.data.alerts || [], req.startMs)", page); + Assert.Contains("limit: choices.limit", page); + Assert.Contains("server_name: choices.server || null", page); + Assert.Contains("include_dismissed: choices.dismissed ? \"true\" : null", page); + Assert.Contains("readTool(\"list_servers\", {})", page); + Assert.Contains("res.data.truncated === true", page); + } + + [Fact] + public void NoWindowOrLimitIsOfferedThatTheToolOrTheDispatchLayerRefuses() + { + var page = Page(); + // #5562 R8: no hand-listed windows and no "All". The picker takes its reach from the catalog's max_hours for the read, which is + // the 2160 hours (90 days) the read's validator accepts, so a longer range is greyed out with the reason and never clamped. That + // is the alert table's retention (AlertHistoryRetentionDays, DarlingRetention.cs), the longest choice the Viewer offers (R8, L4b). + Assert.DoesNotContain("WINDOW_CHOICES", page); + Assert.Contains("read: \"get_alert_history\"", page); + Assert.Contains("pageRangePicker({", page); + Assert.Equal(24 * PerformanceMonitor.Darling.Storage.DarlingRetentionHorizons.AlertHistoryRetentionDays, PerformanceMonitor.Darling.Service.WebReadReach.All["get_alert_history"].MaxHours); + Assert.Equal(90, PerformanceMonitor.Darling.Storage.DarlingRetentionHorizons.AlertHistoryRetentionDays); + + var limits = System.Text.RegularExpressions.Regex.Match(page, "const LIMIT_CHOICES = \\[([0-9, ]+)\\]").Groups[1].Value + .Split(',').Select(s => int.Parse(s.Trim())).ToList(); + Assert.Equal(new[] { 200, 500, 1000 }, limits); + Assert.True(limits.Max() <= 1000, "the dispatch layer clamps a row count to 1000"); + } + + [Fact] + public void TheChoicesLiveAtModuleScopeAndTheDismissRouteIsNotTheDesktopOne() + { + var page = Page(); + Assert.Contains("\nconst choices = {", page); + Assert.DoesNotContain("/api/alerts/dismiss", page); + } + + private static JsonElement Run(string scenario, string? timeZone = null) + { + var psi = new ProcessStartInfo("node") { RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false }; + psi.ArgumentList.Add(PathTo("Darling", "Darling.Tests", "alert-history-range-harness.mjs")); + psi.ArgumentList.Add(PathTo("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js")); + psi.ArgumentList.Add(scenario); + if (timeZone is not null) + { + psi.ArgumentList.Add(timeZone); + } + + Process proc; + try + { + proc = Process.Start(psi)!; + } + catch (Win32Exception) + { + Assert.Skip("Node is not installed, so the shipped page script cannot be run."); + return default; + } + + using (proc) + { + var error = proc.StandardError.ReadToEndAsync(); + var output = proc.StandardOutput.ReadToEnd().Trim(); + if (!proc.WaitForExit(20000)) + { + proc.Kill(entireProcessTree: true); + Assert.Fail("the Alert History harness did not finish in 20 s for scenario " + scenario); + } + + Assert.True(proc.ExitCode == 0, "the Alert History harness failed for scenario " + scenario + ": " + error.Result); + using var doc = JsonDocument.Parse(output); + return doc.RootElement.Clone(); + } + } + + private static string Get(JsonElement read, string name) => + read.TryGetProperty(name, out var v) ? v.GetString()! : "(absent)"; + + [Fact] + public void EachChoiceReachesTheRead() + { + var r = Run("reads"); + var first = r.GetProperty("first"); + Assert.Equal("24", Get(first, "hours_back")); + Assert.Equal("200", Get(first, "limit")); + Assert.Equal("(absent)", Get(first, "server_name")); + Assert.Equal("(absent)", Get(first, "include_dismissed")); + + Assert.Equal("168", Get(r.GetProperty("window"), "hours_back")); + Assert.Equal("1000", Get(r.GetProperty("limit"), "limit")); + Assert.Equal("srv-b", Get(r.GetProperty("server"), "server_name")); + var dismissed = r.GetProperty("dismissed"); + Assert.Equal("true", Get(dismissed, "include_dismissed")); + Assert.Equal("168", Get(dismissed, "hours_back")); + Assert.Equal("1000", Get(dismissed, "limit")); + Assert.Equal("srv-b", Get(dismissed, "server_name")); + + // The popup offers every preset and calendar period within the catalog's 90 days; the longer ones are greyed out, never clamped. + Assert.Equal("Past 5 minutes,Past 15 minutes,Past 30 minutes,Past hour,Past 4 hours,Past day,Past 2 days,Past week,Past 30 days,Past 90 days,Today,Yesterday,Week to Date,Previous Week,Month to Date,Previous Month", + string.Join(",", r.GetProperty("windowOptions").EnumerateArray().Select(e => e.GetString()))); + Assert.Equal("Year to Date,Previous Year", + string.Join(",", r.GetProperty("windowGreyed").EnumerateArray().Select(e => e.GetString()))); + Assert.Equal("This page's reads reach at most 90 days back.", r.GetProperty("refused").GetString()); + // A finished range sends its end as as_of; a shorter-than-an-hour range reads the whole hour. + Assert.EndsWith("Z", Get(r.GetProperty("finished"), "as_of")); + Assert.Equal("1", Get(r.GetProperty("thirtyMinutes"), "hours_back")); + Assert.Equal("200,500,1000", string.Join(",", r.GetProperty("limitOptions").EnumerateArray().Select(e => e.GetString()))); + Assert.Equal(",srv-a,srv-b", string.Join(",", r.GetProperty("serverOptions").EnumerateArray().Select(e => e.GetString()))); + } + + [Fact] + public void TheChoicesSurviveThePollAndAVisitElsewhere() + { + var r = Run("survive"); + foreach (var step in new[] { "poll", "back" }) + { + var reads = r.GetProperty(step).EnumerateArray().ToList(); + Assert.Single(reads); + Assert.Equal("4", Get(reads[0], "hours_back")); + Assert.Equal("500", Get(reads[0], "limit")); + Assert.Equal("srv-a", Get(reads[0], "server_name")); + Assert.Equal("true", Get(reads[0], "include_dismissed")); + } + + var shown = r.GetProperty("shown"); + Assert.Equal("Past 4 hours", shown.GetProperty("window").GetString()); + Assert.Equal("500", shown.GetProperty("limit").GetString()); + Assert.Equal("srv-a", shown.GetProperty("server").GetString()); + Assert.True(shown.GetProperty("dismissed").GetBoolean()); + } + + [Fact] + public void ATruncatedReplySaysMoreAlertsExistAndAClearOneDoesNot() + { + var r = Run("truncated"); + Assert.True(r.GetProperty("truncated").GetBoolean()); + Assert.False(r.GetProperty("notTruncated").GetBoolean()); + Assert.True(r.GetProperty("pollTruncated").GetBoolean()); + Assert.False(r.GetProperty("pollCleared").GetBoolean()); + } + + [Fact] + public void DismissedRowsAreMarkedOnFirstDrawAndOnLaterReconciles() + { + var r = Run("dismissed"); + Assert.Equal(new[] { "", "alert-dismissed" }, r.GetProperty("classes").EnumerateArray().Select(e => e.GetString()).ToArray()); + Assert.Equal(new[] { "alert-dismissed", "", "alert-dismissed" }, r.GetProperty("afterPoll").EnumerateArray().Select(e => e.GetString()).ToArray()); + } + + [Fact] + public void AKeptRowTakesItsDismissedMarkingFromALaterPoll() + { + var r = Run("dismissedLater"); + string[] Strings(string n) => r.GetProperty(n).EnumerateArray().Select(e => e.GetString() ?? "").ToArray(); + Assert.Equal(new[] { "", "" }, Strings("before")); + Assert.Equal(new[] { "", "alert-dismissed" }, Strings("after")); + Assert.Equal(new[] { "", "Dismissed" }, Strings("afterTitle")); + Assert.Equal(new[] { false, true }, r.GetProperty("afterStatus").EnumerateArray().Select(e => e.GetBoolean()).ToArray()); + Assert.Equal("High CPU 1,High CPU 2", string.Join(",", Strings("order"))); + Assert.Equal(new[] { "", "" }, Strings("restored")); + } + + [Fact] + public void TwoServersThatShareADisplayNameKeepBothRows() + { + Assert.Equal(2, Run("sharedName").GetProperty("rows").GetInt32()); + } + + [Fact] + public void TheRangePickerTakesItsReachFromTheCatalog_NinetyDays() + { + // #5562: the picker greys out what the read cannot reach and says why (the catalog's max_hours), so the page needs no hand-written + // tooltip; the reach is the read's own 2160 hours, the alert table's retention (L4b). + Assert.DoesNotContain("The web reads at most 7 days of alert history, so there is no All choice.", Page()); + Assert.DoesNotContain("the web read stops at the common 7 day reach", Page()); + Assert.Contains("get_alert_history reaches 2160 hours, 90 days", Page()); + } + + [Fact] + public void ANewWindowReconcilesInPlace_RowsLeaveAndNewOnesLandInOrder() + { + var r = Run("reconcile"); + Assert.Equal("High CPU 1,High CPU 3,High CPU 5", string.Join(",", r.GetProperty("before").EnumerateArray().Select(e => e.GetString()))); + Assert.Equal("High CPU 0,High CPU 1,High CPU 2,High CPU 3", string.Join(",", r.GetProperty("after").EnumerateArray().Select(e => e.GetString()))); + Assert.True(r.GetProperty("keptNode").GetBoolean(), "an unchanged row keeps its DOM node across a window change"); + Assert.Equal("High CPU 3", string.Join(",", r.GetProperty("shrunk").EnumerateArray().Select(e => e.GetString()))); + } + + // Review r2 H1: alert_time is naive UTC with no zone suffix, which the browser reads as LOCAL time unless the page parses it as UTC. The + // harness rows carry the server's real shape, and the same scenario runs in zones that are not UTC (and in UTC) because a UTC-only run + // passes the old Date.parse code. The offset check proves Node honoured the zone, so the test cannot pass by running in UTC every time. + [Theory] + [InlineData("UTC", 0)] + [InlineData("Europe/Berlin", -1)] + [InlineData("America/New_York", 1)] + [InlineData("Asia/Tokyo", -1)] + public void ARangeShorterThanTheWholeHoursItFetches_ListsOnlyTheAlertsInsideIt(string zone, int offsetSign) + { + var r = Run("trim", zone); + Assert.Equal(offsetSign, Math.Sign(r.GetProperty("tzOffsetMinutes").GetInt32())); + string[] Names(string key) => r.GetProperty(key).EnumerateArray().Select(e => e.GetString()!).ToArray(); + Assert.Equal(new[] { "High CPU 0", "High CPU 19", "High CPU 89" }, Names("day")); + Assert.Equal(new[] { "High CPU 0" }, Names("fiveMinutes")); + Assert.Equal(new[] { "High CPU 0", "High CPU 19" }, Names("thirtyMinutes")); + Assert.Equal(new[] { "High CPU 0", "High CPU 19", "High CPU 89" }, Names("fourHours")); + } + + [Fact] + public void TheMuteLinksStillRenderOnEveryRow() + { + var links = Run("mute").GetProperty("links").EnumerateArray().Select(e => e.GetString()!).ToList(); + Assert.Contains(links, l => l.StartsWith("Mute this alert #/mute-rules?server_id=1&server_name=srv-a&metric_name=", StringComparison.Ordinal)); + Assert.Contains(links, l => l.StartsWith("Mute similar #/mute-rules?metric_name=", StringComparison.Ordinal)); + } +} diff --git a/Darling/Darling.Tests/AlertHistoryRowSeverityTests.cs b/Darling/Darling.Tests/AlertHistoryRowSeverityTests.cs index ca6bfddbca..cbd7f4e18d 100644 --- a/Darling/Darling.Tests/AlertHistoryRowSeverityTests.cs +++ b/Darling/Darling.Tests/AlertHistoryRowSeverityTests.cs @@ -37,6 +37,7 @@ namespace Darling.Tests; /// tier exists, and says the right thing there; both grids' row classes reach the shared decision and no /// grid still calls the by-name predicates directly. /// +[Trait("Reads", "Lite")] public sealed class AlertHistoryRowSeverityTests { /* ─────────────────────────── the member on the wire ─────────────────────────── */ diff --git a/Darling/Darling.Tests/AlertMasterSwitchSurfaceTests.cs b/Darling/Darling.Tests/AlertMasterSwitchSurfaceTests.cs index a500937187..f3671ed2c4 100644 --- a/Darling/Darling.Tests/AlertMasterSwitchSurfaceTests.cs +++ b/Darling/Darling.Tests/AlertMasterSwitchSurfaceTests.cs @@ -42,6 +42,8 @@ namespace Darling.Tests; /// "track always, deliver conditionally" split, which predates this issue and is the idiom precedent the /// AND'd gates follow. /// +[Trait("Stage", "Guard")] +[Trait("Reads", "Lite")] public sealed class AlertMasterSwitchSurfaceTests { /* ---------------- the delivery-call census ---------------- */ diff --git a/Darling/Darling.Tests/AlertMetricClassifierTests.cs b/Darling/Darling.Tests/AlertMetricClassifierTests.cs index cbdba50baa..7291a16bd8 100644 --- a/Darling/Darling.Tests/AlertMetricClassifierTests.cs +++ b/Darling/Darling.Tests/AlertMetricClassifierTests.cs @@ -48,6 +48,7 @@ public class AlertMetricClassifierTests [InlineData("AG Sync Recovered")] [InlineData("AG Data Movement Resumed")] [InlineData("Web TLS Certificate Renewed")] // #3514 + [InlineData("MCP TLS Certificate Renewed")] // #5288 public void IsResolution_True_ForEveryResolutionNotice(string metric) { Assert.True(AlertMetricClassifier.IsResolution(metric)); @@ -77,6 +78,8 @@ into the resolution bucket by "Reconnected"/"Recovered"/"Resumed" appearing else /* #3514: the active web-TLS-certificate alert stays actionable — its "Expiring" name carries none of the resolution words, and adding "Renewed" to the vocabulary must not drag it green. */ [InlineData("Web TLS Certificate Expiring")] + /* #5288: and so does the MCP endpoint's twin. */ + [InlineData("MCP TLS Certificate Expiring")] public void IsResolution_False_ForActionableAlerts(string metric) { Assert.False(AlertMetricClassifier.IsResolution(metric)); diff --git a/Darling/Darling.Tests/AlertNotebookRenderBehaviourTests.cs b/Darling/Darling.Tests/AlertNotebookRenderBehaviourTests.cs index 977ea965ff..a5ed50f63f 100644 --- a/Darling/Darling.Tests/AlertNotebookRenderBehaviourTests.cs +++ b/Darling/Darling.Tests/AlertNotebookRenderBehaviourTests.cs @@ -41,6 +41,7 @@ private static bool TryRun(string scenario, out JsonElement result) } catch (Win32Exception) { + Assert.Skip("Node is not installed, so the shipped page script cannot be run."); return false; } diff --git a/Darling/Darling.Tests/AlertNotebookRenderClientTests.cs b/Darling/Darling.Tests/AlertNotebookRenderClientTests.cs index a7ea7d7363..d7de36724b 100644 --- a/Darling/Darling.Tests/AlertNotebookRenderClientTests.cs +++ b/Darling/Darling.Tests/AlertNotebookRenderClientTests.cs @@ -307,18 +307,19 @@ public void Views_AlertModePanelCells_ScopeByTheEndpointsScopeServer_NeverByADis [Fact] public void Views_PanelOrError_SettlesExactlyOnce_SoABadPanelCellCannotHoldALimiterSlot() { - var body = CodeOf(ReadRepoFileLf(ViewsPath), "function panelOrError(p, readSet, sourceSet, scope, onSettled) {"); + var body = CodeOf(ReadRepoFileLf(ViewsPath), "function panelOrError(p, readSet, sourceSet, scope, onSettled, slot = null) {"); Assert.Contains("if (onSettled) onSettled();", body, StringComparison.Ordinal); - Assert.Contains("renderComposedPanelCard(p, scope, onSettled)", body, StringComparison.Ordinal); + Assert.Contains("renderComposedPanelCard(p, scope, onSettled, slot)", body, StringComparison.Ordinal); Assert.Contains("renderPanel(p, onSettled)", body, StringComparison.Ordinal); // Every error card is built by the settling `fail` function: the one panelErrorCard call left is inside it. Assert.Single(Regex.Matches(body, @"panelErrorCard\(")); - // The dashboard grid passes no callback, so its behaviour is the one it had. + // The dashboard grid passes no callback (its panel is only re-scoped by panelUnderScope first, which cannot + // take a limiter slot), so its behaviour is the one it had. Assert.Contains( - "panels.map((p) => panelOrError(p, readSet, sourceSet, currentScope()))", + "panels.map((p, i) => panelOrError(panelUnderScope(p, state, panelServer), readSet, sourceSet, currentScope(), undefined, i))", ReadRepoFile(ViewsPath), StringComparison.Ordinal); } diff --git a/Darling/Darling.Tests/AlertNotebookTableCatalogTests.cs b/Darling/Darling.Tests/AlertNotebookTableCatalogTests.cs index c603845599..51c0c30af1 100644 --- a/Darling/Darling.Tests/AlertNotebookTableCatalogTests.cs +++ b/Darling/Darling.Tests/AlertNotebookTableCatalogTests.cs @@ -260,6 +260,8 @@ public void ACollectorCostCell_ResolvesToTheFleetPartWithoutACollectorName_AndTh private static readonly Dictionary s_keyConstants = new(StringComparer.Ordinal) { ["get_database_sizes.size_note"] = "AzureSiblingDatabaseSize.RowNoteKey", + // #5249: the gated rows are built by the helper the tool calls (anonymous-object member `message`). + ["get_collection_health.message"] = "DarlingGatedCollectorRows.AppendAsync(", }; /// The text from to the next marker (or the end). diff --git a/Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs b/Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs index 0de7d2358e..253c9bd5c6 100644 --- a/Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs +++ b/Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs @@ -41,6 +41,7 @@ namespace Darling.Tests; /// down), and the counts are asserted in both directions so a walk that silently stopped reaching cannot /// report clean. /// +[Trait("Reads", "Lite")] public sealed class AlertReadFailureSurfaceTests { /* ---------------- the counter's own behaviour ---------------- */ @@ -922,7 +923,7 @@ private static readonly (string Path, int Counted, int Exempt)[] s_wholeFileScop DarlingWorker. 16th exempt since #4750: EvaluateNotificationChannelsAsync's wrapper catch — the webhook channels' failure counts are read from the webhook service's memory, so there is no store read for the condition to swallow. */ - (Path.Combine("Darling", "PerformanceMonitor.Darling.Service", "DarlingSelfAlertEvaluator.cs"), 12, 17), + (Path.Combine("Darling", "PerformanceMonitor.Darling.Service", "DarlingSelfAlertEvaluator.cs"), 13, 18), }; /// @@ -967,7 +968,7 @@ private static readonly (string Path, int Counted, int Exempt)[] s_wholeFileScop /* 11th since #4215: ReadRejectedManagedConfSettingNamesAsync's catch — COUNTED, not exempt, because a RejectedValue row is judgeable evidence for the alert, not context for its text. */ private const int WorkerCountedSites = 11; - private const int WorkerExemptSites = 11; + private const int WorkerExemptSites = 12; /// /// Counted sites tree-wide. ONE numeral with several readers rather than the same number written out at @@ -1010,9 +1011,11 @@ private static readonly (string Path, int Counted, int Exempt)[] s_wholeFileScop /// shape at the store's own health. And a THIRTEENTH since #4215: the store-settings /// self-alert's rejected-verdict read, moved from exempt to counted because a RejectedValue row is one of /// the three conditions the alert fires on — losing it leaves that condition unjudgeable, and the original - /// exempt classification would let one failed read write a false "Store Settings Resolved". + /// exempt classification would let one failed read write a false "Store Settings Resolved". And a + /// THIRTY-EIGHTH since #5450: the daily retained-history audit's per-rollup read, counted because a swallowed + /// read skips that rollup for the day, and the audit is the only place the day's thin hours are ever named. /// - private const int CountedSites = 37; + private const int CountedSites = 38; /// /// Log-message fragments that identify a catch block DELIBERATELY not counted, each paired with the @@ -1029,6 +1032,7 @@ private static readonly (string Path, int Counted, int Exempt)[] s_wholeFileScop ["Store disk-pressure self-alert failed"] = "handed its evidence as parameters; the read is counted in DarlingWorker", ["Custom-alert rule-health self-alert failed"] = "handed its evidence (the report) as a parameter; the report-building read is in CustomAlertEvaluator, outside this census", ["Store runtime upgrade self-alert failed"] = "handed its evidence as parameters", + ["Collection gap at start self-alert failed"] = "handed its evidence as parameters", ["Store TimescaleDB self-alert failed"] = "handed its evidence as parameters; the version was read by the bootstrap (#3908)", /* #3816 renamed this line with the check: the same catch, one family over — the self-heal now covers every policy family, so "Compression-job health" would have named a third of what it isolates. */ @@ -1039,7 +1043,7 @@ private static readonly (string Path, int Counted, int Exempt)[] s_wholeFileScop ["Notification-channel self-alert failed"] = "reads the webhook channels' failure counts from the webhook service's memory and performs no store read at all - there is no read for this condition to be the swallowing of", ["Stale-mute self-alert failed"] = "handed its evidence (the live MuteRuleService cache) as a parameter and performs no store read at all - there is no read anywhere for this condition to be the swallowing of", ["Fleet gate self-alert failed"] = "handed its evidence (the worker's in-memory gate counts) as a parameter and performs no store read at all", - ["Web TLS certificate self-alert failed"] = "handed its evidence (the report from the web host's in-memory WebTlsCertificateState publish) as a parameter and performs no store read at all", + ["Web TLS certificate self-alert failed"] = "handed its evidence (the report from the listener host's in-memory TLS certificate state publish: WebTlsCertificateState for the web host, McpTlsCertificateState for the MCP host) as a parameter and performs no store read at all; one shared catch covers both listeners", ["Store settings self-alert failed"] = "handed its evidence as a parameter; the one store read behind it (the rejected-verdict names) is isolated in its own COUNTED catch in DarlingWorker (#4215) rather than exempted", ["Failed to record resolution"] = "an audit-row write", ["Could not record Postgres alert resolution"] = "a history write", @@ -1056,6 +1060,7 @@ private static readonly (string Path, int Counted, int Exempt)[] s_wholeFileScop ["CONVERTS the fault into the unreadable count"] = "a parse arm, not a read: the fleet-sweep rollup's store read is counted above it, and a document that does not parse becomes the rollup's own reportable unreadable count - the fault is evidence, not a swallow", ["Could not resolve Agent job names"] = "reads the monitored server's msdb through the host resolver, not the store - the Recently-failed-job precedent one seam over; the card degrades to the unresolved form whose raw marker keeps the gap visible, and the page still delivers", ["delivery stamp could not be written"] = "a write (#3580): the daily document was already delivered and process memory already gates it; the dropped stamp costs one re-announcement at the next restart and never a delivery - the stamp READ beside it is the read, and it is counted", + ["Store statement history is still failing"] = "the Debug repeat of a telemetry write sweep (#5097); the first failure warns, no alert is judged on its result", ["Read-latency flush failed"] = "the read-latency histogram flush writes instrumentation; losing an hour's rows can't hide an alert condition", }; @@ -1241,8 +1246,11 @@ WebTlsCertificateState report the web host publishes - there is no store read to flush's catch, a telemetry write whose loss costs an hour's histogram rows and never an alert. 33rd since #4750: the Notification Channel Failing self-alert's catch, whose counts come from the webhook service's memory rather than the store. 34th since #4732: the Collection Falling Behind - self-alert's wrapper catch, whose counts come from the worker's in-memory gate statistics. */ - Assert.Equal(34, totalExempt); + self-alert's wrapper catch, whose counts come from the worker's in-memory gate statistics. 35th since + #5097: the store statement history snapshot's catch, a telemetry write sweep whose loss costs an hour's + capture and never an alert. 36th since #5450: the Collection Gap At Start self-alert's catch, handed + its evidence as a parameter. */ + Assert.Equal(36, totalExempt); /* Every exemption in the table is actually used. An exemption for a message that no longer exists is a hole this pin would otherwise keep open indefinitely — the shape that lets a real new catch diff --git a/Darling/Darling.Tests/AlertStatementCutOrderTests.cs b/Darling/Darling.Tests/AlertStatementCutOrderTests.cs new file mode 100644 index 0000000000..96c2c47e0a --- /dev/null +++ b/Darling/Darling.Tests/AlertStatementCutOrderTests.cs @@ -0,0 +1,216 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using PerformanceMonitor.Alerting; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Notifications; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5320: an alert's statement text is judged WHOLE and then cut, never cut and then judged. The shared alert +/// builders (, , both apps) cut a statement to a +/// one-line preview for the alert body. A judge that sees the cut preview misses a value whose naming text (here a +/// CREATE LOGIN ... PASSWORD tail) lies past the cut, so every statement here carries a value in its first +/// part and its naming text well past 300 characters. Each pin fails when the cut comes first. +/// +public sealed class AlertStatementCutOrderTests +{ + private const string Value = "Hunter2Secret5320"; + + /// A value in the first characters, 600 characters of column list, then the text that names it. + private static readonly string Statement = + $"SELECT '{Value}' AS v" + string.Concat(Enumerable.Range(0, 60).Select(i => $", c{i:D4} AS a{i:D4}")) + + " INTO #t; CREATE LOGIN leaker WITH PASSWORD = 'x';"; + + private static readonly List NoExclusions = new(); + + private static string Flat(AlertContext? context) => + AlertContextBuilders.ContextToDetailText(context) ?? ""; + + private static void AssertWithheld(string text) + { + Assert.DoesNotContain(Value, text, StringComparison.Ordinal); + Assert.Contains(SensitiveStatements.PlaceholderText, text, StringComparison.Ordinal); + } + + [Fact] + public void TheFixtureStatement_IsNamedWhole_AndItsFirstThreeHundredCharactersAreNot() + { + Assert.True(Statement.IndexOf("CREATE LOGIN", StringComparison.Ordinal) > 600); + Assert.True(SensitiveStatements.Names(Statement)); + Assert.False(SensitiveStatements.Names(Statement[..300])); + Assert.Contains(Value, Statement[..300], StringComparison.Ordinal); + } + + [Fact] + public void TruncateStatement_JudgesTheWholeStatement_BeforeTheCut() + { + Assert.Equal(SensitiveStatements.PlaceholderText, AlertContextBuilders.TruncateStatement(Statement)); + Assert.Equal(SensitiveStatements.PlaceholderText, AlertContextBuilders.TruncateStatement(Statement, 80)); + } + + [Fact] + public void TruncateStatement_CutsAClearStatementTheWayTruncateTextDoes() + { + var clear = "SELECT a,\r\n b FROM t " + new string('x', 400); + + Assert.Equal(AlertContextBuilders.TruncateText(clear), AlertContextBuilders.TruncateStatement(clear)); + Assert.Equal(AlertContextBuilders.TruncateText(clear, 40), AlertContextBuilders.TruncateStatement(clear, 40)); + Assert.Equal("", AlertContextBuilders.TruncateStatement("")); + } + + [Fact] + public void BlockingContext_ShowsTheMarker_ForABlockedAndABlockingStatementWhoseNamingTextIsPastTheCut() + { + var rows = new List + { + new() + { + DatabaseName = "AppDb", WaitTimeMs = 9000, ContentiousObject = "AppDb.dbo.Ledger", + BlockedSqlText = Statement, BlockingSqlText = Statement, + }, + }; + + var context = AlertContextBuilders.BuildBlockingContext("SQL2022", rows, NoExclusions); + + AssertWithheld(Flat(context)); + } + + [Fact] + public void BlockingIncident_CarriesTheMarkerInItsDetailFields() + { + var rows = new List + { + new() + { + DatabaseName = "AppDb", WaitTimeMs = 9000, ContentiousObject = "AppDb.dbo.Ledger", + BlockedSqlText = Statement, BlockingSqlText = Statement, + }, + }; + + var incident = Assert.Single(AlertContextBuilders.BlockingIncidents("SQL2022", rows, NoExclusions)); + var fields = Assert.IsAssignableFrom>(incident.DetailFields); + var query = fields.Where(f => f.Label.EndsWith("Query", StringComparison.Ordinal)).ToList(); + + Assert.Equal(2, query.Count); + Assert.All(query, f => Assert.Equal(SensitiveStatements.PlaceholderText, f.Value)); + } + + [Fact] + public void BlockingIncident_KeepsItsIdentityOnTheRawText_SoAWithheldStatementDoesNotChangeTheKey() + { + /* No contentious object: identity falls back to database + the literal-stripped statement pair. */ + var rows = new List + { + new() { DatabaseName = "AppDb", WaitTimeMs = 9000, BlockedSqlText = Statement, BlockingSqlText = Statement + " " }, + new() { DatabaseName = "AppDb", WaitTimeMs = 8000, BlockedSqlText = Statement, BlockingSqlText = Statement + " " }, + }; + + var incidents = AlertContextBuilders.BlockingIncidents("SQL2022", rows, NoExclusions); + + Assert.Single(incidents); + } + + private const string StandaloneVictimGraph = "" + + "x" + + ""; + + private static string FingerprintedGraph(string inputbuf) => + "" + + $"{System.Security.SecurityElement.Escape(inputbuf)}" + + "SELECT 2" + + "" + + ""; + + [Fact] + public void DeadlockContext_ShowsTheMarker_ForAVictimStatementWithNoParseableObjects() + { + var rows = new List + { + new() { VictimProcessId = "process1", VictimSqlText = Statement, DeadlockGraphXml = StandaloneVictimGraph }, + }; + + var context = AlertContextBuilders.BuildDeadlockContext("SQL2022", rows, NoExclusions); + + AssertWithheld(Flat(context)); + } + + [Fact] + public void DeadlockContext_ShowsTheMarker_ForTheFingerprintedVictimAndEachPartyStatement() + { + var rows = new List + { + new() { VictimProcessId = "process1", VictimSqlText = Statement, DeadlockGraphXml = FingerprintedGraph(Statement) }, + }; + + var context = AlertContextBuilders.BuildDeadlockContext("SQL2022", rows, NoExclusions); + var text = Flat(context); + + AssertWithheld(text); + Assert.Contains("SELECT 2", text, StringComparison.Ordinal); + } + + [Fact] + public void LongRunningQueryContext_ShowsTheMarker_ForAStatementWhoseNamingTextIsPastTheCut() + { + var queries = new List + { + new() { SessionId = 61, DatabaseName = "AppDb", QueryText = Statement, ElapsedSeconds = 900 }, + }; + + var context = AlertContextBuilders.BuildLongRunningQueryContext("SQL2022", queries); + + AssertWithheld(Flat(context)); + } + + /// + /// The toast preview of the long-running-query alert is cut to 80 characters inside the engine's sweep, which + /// no pure seam reaches; this holds it to the judge-then-cut helper by its source. + /// + [Fact] + public void TheEnginesLongRunningQueryPreview_UsesTheJudgeThenCutHelper() + { + var source = RepoFile.ReadRepoFile("PerformanceMonitor.Alerting", "AlertEngine.cs"); + + Assert.Contains("AlertContextBuilders.TruncateStatement(worst.QueryText, 80)", source, StringComparison.Ordinal); + Assert.DoesNotContain("AlertContextBuilders.TruncateText(", source, StringComparison.Ordinal); + } + + /* The two PostgreSQL alert texts cut a statement the store already holds judged: a deadlock victim the log + parser took out of a report it had redacted whole, and a blocking root query PgBlockingCollector judges on the + target (PgSensitiveStatementFilterTests). These two pins read the chain through the parser. */ + [Fact] + public void PgDeadlockAlert_NeverCarriesAValueOfAVictimWhoseNamingTextIsPastTheCut() + { + var at = new DateTime(2026, 10, 6, 12, 0, 0, 100); + var prefix = at.ToString("yyyy-MM-dd HH:mm:ss.fff", System.Globalization.CultureInfo.InvariantCulture) + " UTC [3301] "; + var report = prefix + "ERROR: deadlock detected\n" + + prefix + "DETAIL: Process 3301 waits for ShareLock on transaction 5501; blocked by process 3302.\n" + + "\tProcess 3302 waits for ShareLock on transaction 5500; blocked by process 3301.\n" + + $"\tProcess 3301: SELECT '{Value}' AS v" + string.Concat(Enumerable.Range(0, 60).Select(i => $", c{i:D4}")) + + " INTO t; ALTER ROLE leaker PASSWORD 'x'\n" + + "\tProcess 3302: SELECT 2\n" + + prefix + "HINT: See server log for query details.\n"; + + var parsed = Assert.Single(PgDeadlockLogParser.Extract(report)); + var incident = DarlingWorker.BuildPgDeadlockIncident(new DarlingPgDeadlockReader.PgDeadlockRow( + parsed.OccurredAtUtc, parsed.VictimPid, parsed.ParticipantCount, parsed.DeadlockHash, + parsed.LockModes, parsed.Resources, parsed.VictimStatement, 1)); + + Assert.DoesNotContain(Value, string.Join("|", incident.InvolvedObjects), StringComparison.Ordinal); + Assert.DoesNotContain(Value, parsed.GraphText, StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/AlwaysOnXeReadOnlyIntentTests.cs b/Darling/Darling.Tests/AlwaysOnXeReadOnlyIntentTests.cs new file mode 100644 index 0000000000..af1e901050 --- /dev/null +++ b/Darling/Darling.Tests/AlwaysOnXeReadOnlyIntentTests.cs @@ -0,0 +1,423 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Data.SqlClient; +using Microsoft.Extensions.Logging; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using Xunit; + +namespace Darling.Tests; + +/// +/// Which connection each step of the deadlock and blocked-process sessions' ensure uses on Azure SQL Database, for a +/// registration with read-only intent and for one without (#4961). A session cannot be created or dropped over a connection +/// to a read-only replica, and a definition made on the primary replicates to it, so a registration with the intent creates +/// over a connection without it. The start, the probes and the read stay on the registration's own connection, because run +/// state is per replica. A session that runs on the replica is stopped there before it is dropped on the primary, on the +/// switch back and when the server is removed. Each test drives the real ensure against a scripted primary and replica, which +/// see the connection string each open would use. +/// +public sealed class AlwaysOnXeReadOnlyIntentTests : IAsyncDisposable +{ + private const string Host = "intent.database.windows.net"; + private const string InstallIdValue = "0a1b2c3d"; + + /* Never opened: the runner reaches the store only to write collected rows, and these tests collect nothing. */ + private readonly NpgsqlDataSource _store = NpgsqlDataSource.Create("Host=127.0.0.1;Port=1;Database=unused;Username=unused"); + + public ValueTask DisposeAsync() => _store.DisposeAsync(); + + private static string Own(AlwaysOnXeSessionKind kind) => + AlwaysOnXeSessions.OwnNameFor(LongQueryCompletionsCollector.DarlingProduct, InstallIdValue, kind); + + private static string Shared(AlwaysOnXeSessionKind kind) => AlwaysOnXeSessions.SharedNameFor(kind); + + private sealed class AlreadyThere : Exception + { + public AlreadyThere() : base("already there") { } + } + + /// + /// One Azure SQL Database as its primary and its read-only replica see it. A definition is made on the primary and + /// replicates at once; a session runs on one side or the other, and is started and stopped on its own. The replica + /// refuses a create or a drop with error 3906, and the primary refuses a drop while the session still runs on the replica + /// (the documented order: stop it there, then drop it on the primary). + /// + private sealed class Pair + { + public HashSet Definitions { get; } = new(StringComparer.Ordinal); + public HashSet WrongEvent { get; } = new(StringComparer.Ordinal); + public HashSet RunningOnPrimary { get; } = new(StringComparer.Ordinal); + public HashSet RunningOnReplica { get; } = new(StringComparer.Ordinal); + + /* Every open, and every call over a connection: its intent, then what it was. */ + public List Opens { get; } = new(); + public List<(ApplicationIntent Intent, string What)> Log { get; } = new(); + + /* A database that is read-only itself, reached without the intent: the primary refuses creates too. */ + public bool PrimaryIsReadOnly { get; set; } + + /// The (intent, verb) of each statement that names one session, in the order sent. + public List<(ApplicationIntent Intent, string Verb)> Statements(string name) => + Log.Where(e => e.What.StartsWith("DDL ", StringComparison.Ordinal) && e.What.Contains("[" + name + "]", StringComparison.Ordinal)) + .Select(e => (e.Intent, Verb: e.What["DDL ".Length..e.What.IndexOf(' ', "DDL ".Length)])) + .ToList(); + + public List ReadsOf(string prefix) => + Log.Where(e => e.What.StartsWith(prefix, StringComparison.Ordinal)).Select(e => e.Intent).ToList(); + } + + private sealed class Channel : IAlwaysOnXeDatabase + { + private readonly Pair _pair; + private readonly ApplicationIntent _intent; + + public Channel(Pair pair, ApplicationIntent intent) + { + _pair = pair; + _intent = intent; + } + + public Task ReadCatalogAsync(AlwaysOnXeSessionKind kind, string sessionName, CancellationToken cancellationToken) + { + _pair.Log.Add((_intent, "catalog " + sessionName)); + return Task.FromResult( + !_pair.Definitions.Contains(sessionName) ? AlwaysOnXeCatalog.Missing + : _pair.WrongEvent.Contains(sessionName) && kind == AlwaysOnXeSessionKind.Deadlock ? AlwaysOnXeCatalog.WrongEvent + : AlwaysOnXeCatalog.Present); + } + + public Task IsStartedAsync(string sessionName, CancellationToken cancellationToken) + { + _pair.Log.Add((_intent, "started " + sessionName)); + return Task.FromResult(Running.Contains(sessionName)); + } + + private HashSet Running => _intent == ApplicationIntent.ReadOnly ? _pair.RunningOnReplica : _pair.RunningOnPrimary; + + public Task ExecuteAsync(string statement, CancellationToken cancellationToken) + { + var text = statement.TrimStart(); + var name = text[(text.IndexOf('[') + 1)..text.IndexOf(']')]; + var creates = text.StartsWith("CREATE EVENT SESSION", StringComparison.Ordinal); + var starts = text.Contains("STATE = START", StringComparison.Ordinal); + var stops = text.Contains("STATE = STOP", StringComparison.Ordinal); + var drops = text.StartsWith("DROP EVENT SESSION", StringComparison.Ordinal); + var verb = creates && starts ? "CREATE+START" : creates ? "CREATE" : starts ? "START" : stops ? "STOP" : "DROP"; + _pair.Log.Add((_intent, "DDL " + verb + " [" + name + "]")); + + if ((creates || drops) && (_intent == ApplicationIntent.ReadOnly || _pair.PrimaryIsReadOnly)) + { + throw ReadOnlyRefusal(); + } + + if (creates) + { + if (_pair.Definitions.Contains(name)) + { + throw new AlreadyThere(); + } + + _pair.Definitions.Add(name); + if (starts) + { + Running.Add(name); + } + } + else if (drops) + { + if (_pair.RunningOnReplica.Contains(name)) + { + throw new InvalidOperationException("The session still runs on a read-only replica: stop it there first."); + } + + _pair.Definitions.Remove(name); + _pair.RunningOnPrimary.Remove(name); + } + else if (starts) + { + if (!_pair.Definitions.Contains(name)) + { + throw new InvalidOperationException("The session does not exist."); + } + + if (!Running.Add(name)) + { + throw new AlreadyThere(); + } + } + else if (stops && !Running.Remove(name)) + { + throw new InvalidOperationException("The session is not running here."); + } + + return Task.CompletedTask; + } + + public bool IsAlreadyPresent(Exception exception) => exception is AlreadyThere; + } + + private sealed class Rig + { + public required DarlingCollectorRunner Runner { get; init; } + public required ServerRuntime Runtime { get; init; } + public Pair Alpha { get; } = new(); + public DarlingSelfAlertTests.CapturingLogger Logger { get; } = new(); + + /* The connection string of every open, in order. */ + public List OpenedStrings { get; } = new(); + + public Task EnsureAsync() => DarlingXeSessions.EnsureAllAsync(Runtime, Runner, Logger, CancellationToken.None); + + /// + /// The server's removal: its long-query trace was never on (nothing to drop there), and no other registration + /// reads a session of its own, so every own fallback the choices hold is dropped. + /// + public Task RemoveAsync() => + DarlingRemovedServerSessions.DropAsync( + new RemovedLongQueryServer(Runtime.Config, Runtime, LongQueryTraceApplied: false), + Runner, remaining: null, traceOn: _ => false, + instanceGuard: _ => Task.FromResult(LongQueryTraceInstanceGuard.NoKeepers), Logger, CancellationToken.None); + + /// + /// Every line at Warning or above that this routine wrote, whatever it says. Nothing narrows it by a word, so a Warning a + /// test does not expect, such as a failed stop or a failed drop, is among them. + /// + public List Loud() => Logger.Entries + .Where(e => e.Level >= LogLevel.Warning) + .Select(e => e.Message) + .ToList(); + + /// + /// The one line a create that a read-only database refused logs at Warning for a capture: the registration, the database + /// and the message that session's kind is worded with. + /// + public string ReadOnlyLine(string database, AlwaysOnXeSessionKind kind) => + $"[{Runtime.Config.DisplayName}] [{database}] {AlwaysOnXeSessions.ReadOnlyDatabaseMessage(kind)}"; + } + + private Rig BuildRig(bool readOnlyIntent) + { + var config = new MonitoredServer { Name = "alwayson-intent", Host = Host, ReadOnlyIntent = readOnlyIntent }; + var runtime = new ServerRuntime + { + Config = config, + ConnectionString = $"Server=tcp:{Host},1433;Initial Catalog=master;Encrypt=True;ApplicationIntent={(readOnlyIntent ? "ReadOnly" : "ReadWrite")}", + Target = new CollectorTargetInfo { IsAzureSqlDb = true }, + StorageName = Host, + ServerId = 4961, + }; + var runner = new DarlingCollectorRunner( + _store, + new CollectorDeltaCalculator(), + databaseScope: (_, _) => new List(), + separatelyMonitoredDatabases: _ => new List(), + installId: () => InstallIdValue); + + var rig = new Rig { Runner = runner, Runtime = runtime }; + runner.AzureDatabaseListOverrideForTests = (_, _) => Task.FromResult(new List { "master", "alpha" }); + runner.AlwaysOnXeConnectionForTests = (_, database, connectionString, _) => + { + /* The removal reads the database name back from the choice's key, which keeps it in upper case. */ + Assert.Equal("alpha", database, ignoreCase: true); + Assert.Equal("alpha", new SqlConnectionStringBuilder(connectionString).InitialCatalog, ignoreCase: true); + var intent = new SqlConnectionStringBuilder(connectionString).ApplicationIntent; + rig.OpenedStrings.Add(connectionString); + rig.Alpha.Opens.Add(intent); + return Task.FromResult(new Channel(rig.Alpha, intent)); + }; + + return rig; + } + + private static readonly AlwaysOnXeSessionKind[] Kinds = { AlwaysOnXeSessionKind.Deadlock, AlwaysOnXeSessionKind.BlockedProcess }; + + /* ── A registration with read-only intent ── */ + + [Fact] + public async Task ReadOnlyIntent_ASharedSessionThatIsMissing_IsCreatedOverAConnectionWithoutTheIntent_AndStartedOverTheOwnOne() + { + var rig = BuildRig(readOnlyIntent: true); + + await rig.EnsureAsync(); + + foreach (var kind in Kinds) + { + var shared = Shared(kind); + Assert.Equal( + new[] { (ApplicationIntent.ReadWrite, "CREATE"), (ApplicationIntent.ReadOnly, "START") }, + rig.Alpha.Statements(shared)); + Assert.Contains(shared, rig.Alpha.RunningOnReplica); + Assert.DoesNotContain(shared, rig.Alpha.RunningOnPrimary); + } + + /* The probes ran over the registration's own connection, and the read-write one was opened once, for the creates. */ + Assert.All(rig.Alpha.ReadsOf("catalog "), intent => Assert.Equal(ApplicationIntent.ReadOnly, intent)); + Assert.All(rig.Alpha.ReadsOf("started "), intent => Assert.Equal(ApplicationIntent.ReadOnly, intent)); + Assert.Equal(new[] { ApplicationIntent.ReadOnly, ApplicationIntent.ReadWrite }, rig.Alpha.Opens); + Assert.Empty(rig.Loud()); + } + + [Fact] + public async Task ReadOnlyIntent_AnOwnFallback_IsCreatedOverAConnectionWithoutTheIntent_AndStartedOverTheOwnOne() + { + var rig = BuildRig(readOnlyIntent: true); + var shared = Shared(AlwaysOnXeSessionKind.Deadlock); + rig.Alpha.Definitions.Add(shared); + rig.Alpha.WrongEvent.Add(shared); + rig.Alpha.RunningOnReplica.Add(shared); + + await rig.EnsureAsync(); + + var own = Own(AlwaysOnXeSessionKind.Deadlock); + Assert.Equal( + new[] { (ApplicationIntent.ReadWrite, "CREATE"), (ApplicationIntent.ReadOnly, "START") }, + rig.Alpha.Statements(own)); + Assert.Contains(own, rig.Alpha.RunningOnReplica); + Assert.Equal(own, rig.Runner.AlwaysOnReadSessionName(rig.Runtime, "alpha", AlwaysOnXeSessionKind.Deadlock)); + Assert.Empty(rig.Alpha.Statements(shared)); + } + + [Fact] + public async Task ReadOnlyIntent_SwitchingBack_StopsTheOwnSessionOverTheOwnConnection_ThenDropsItOverAConnectionWithoutTheIntent() + { + var rig = BuildRig(readOnlyIntent: true); + var shared = Shared(AlwaysOnXeSessionKind.Deadlock); + var own = Own(AlwaysOnXeSessionKind.Deadlock); + rig.Alpha.Definitions.Add(shared); + rig.Alpha.WrongEvent.Add(shared); + rig.Alpha.RunningOnReplica.Add(shared); + await rig.EnsureAsync(); + Assert.Contains(own, rig.Alpha.RunningOnReplica); + rig.Alpha.Log.Clear(); + + rig.Alpha.WrongEvent.Clear(); + await rig.EnsureAsync(); + + Assert.Equal( + new[] { (ApplicationIntent.ReadOnly, "STOP"), (ApplicationIntent.ReadWrite, "DROP") }, + rig.Alpha.Statements(own)); + Assert.DoesNotContain(own, rig.Alpha.Definitions); + Assert.Equal(shared, rig.Runner.AlwaysOnReadSessionName(rig.Runtime, "alpha", AlwaysOnXeSessionKind.Deadlock)); + Assert.DoesNotContain(rig.Alpha.Statements(shared), s => s.Verb is "CREATE" or "CREATE+START" or "DROP"); + } + + [Fact] + public async Task ReadOnlyIntent_ARemovedServer_StopsItsOwnSessionOverTheOwnConnection_ThenDropsItOverAConnectionWithoutTheIntent() + { + var rig = BuildRig(readOnlyIntent: true); + var own = Own(AlwaysOnXeSessionKind.Deadlock); + rig.Alpha.Definitions.Add(own); + rig.Alpha.RunningOnReplica.Add(own); + rig.Runner.AlwaysOnChoices.Set(DarlingAlwaysOnXeSessions.ServerKey(rig.Runtime), "alpha", AlwaysOnXeSessionKind.Deadlock, AlwaysOnXeChoice.Own); + + await rig.RemoveAsync(); + + Assert.Equal( + new[] { (ApplicationIntent.ReadOnly, "STOP"), (ApplicationIntent.ReadWrite, "DROP") }, + rig.Alpha.Statements(own)); + Assert.DoesNotContain(own, rig.Alpha.Definitions); + Assert.Empty(rig.Loud()); + } + + [Fact] + public async Task ReadOnlyIntent_ARemovedServer_WhoseOwnSessionDoesNotRunOnTheReplica_DropsItWithoutAStop() + { + var rig = BuildRig(readOnlyIntent: true); + var own = Own(AlwaysOnXeSessionKind.Deadlock); + rig.Alpha.Definitions.Add(own); + rig.Runner.AlwaysOnChoices.Set(DarlingAlwaysOnXeSessions.ServerKey(rig.Runtime), "alpha", AlwaysOnXeSessionKind.Deadlock, AlwaysOnXeChoice.Own); + + await rig.RemoveAsync(); + + Assert.Equal(new[] { (ApplicationIntent.ReadWrite, "DROP") }, rig.Alpha.Statements(own)); + Assert.DoesNotContain(own, rig.Alpha.Definitions); + Assert.Empty(rig.Loud()); + } + + [Fact] + public async Task ReadOnlyIntent_AHealthyDatabase_OpensNoConnectionWithoutTheIntent() + { + var rig = BuildRig(readOnlyIntent: true); + foreach (var kind in Kinds) + { + rig.Alpha.Definitions.Add(Shared(kind)); + rig.Alpha.RunningOnReplica.Add(Shared(kind)); + } + + await rig.EnsureAsync(); + + Assert.Equal(new[] { ApplicationIntent.ReadOnly }, rig.Alpha.Opens); + Assert.DoesNotContain(rig.Alpha.Log, e => e.What.StartsWith("DDL ", StringComparison.Ordinal)); + } + + /* ── A registration without it, and on-premises, are unchanged ── */ + + [Fact] + public async Task NoIntent_EverythingGoesOverTheOwnConnection_AsItAlwaysHas() + { + var rig = BuildRig(readOnlyIntent: false); + var shared = Shared(AlwaysOnXeSessionKind.Deadlock); + var own = Own(AlwaysOnXeSessionKind.Deadlock); + + /* The shared create is one batch, with its start. */ + await rig.EnsureAsync(); + Assert.Equal(new[] { (ApplicationIntent.ReadWrite, "CREATE+START") }, rig.Alpha.Statements(shared)); + + /* A fallback is the same, and its drop on the switch back is the one statement, with no stop first. */ + rig.Alpha.WrongEvent.Add(shared); + rig.Alpha.Log.Clear(); + await rig.EnsureAsync(); + Assert.Equal(new[] { (ApplicationIntent.ReadWrite, "CREATE+START") }, rig.Alpha.Statements(own)); + rig.Alpha.WrongEvent.Clear(); + rig.Alpha.Log.Clear(); + await rig.EnsureAsync(); + Assert.Equal(new[] { (ApplicationIntent.ReadWrite, "DROP") }, rig.Alpha.Statements(own)); + + Assert.All(rig.Alpha.Opens, intent => Assert.Equal(ApplicationIntent.ReadWrite, intent)); + Assert.Equal(3, rig.Alpha.Opens.Count); + } + + /* ── A read-only database reached without the intent (an Azure geo-secondary) ── */ + + [Fact] + public async Task AReadOnlyDatabaseWithoutTheIntent_LogsTheReadOnlyExplanationOnce_NotTheCapsSentence_AndTheRetriesStayQuiet() + { + var rig = BuildRig(readOnlyIntent: false); + rig.Alpha.PrimaryIsReadOnly = true; + + await rig.EnsureAsync(); + + /* One line a capture at Warning or above, whatever it says, and each is the read-only message: why, and what to change. */ + var expected = Kinds.Select(kind => rig.ReadOnlyLine("alpha", kind)).OrderBy(line => line, StringComparer.Ordinal).ToList(); + Assert.Equal(expected, rig.Loud().OrderBy(line => line, StringComparer.Ordinal)); + Assert.All(rig.Loud(), line => + { + Assert.Contains("read-only", line, StringComparison.Ordinal); + Assert.Contains("3906", line, StringComparison.Ordinal); + Assert.Contains("Register the primary database instead", line, StringComparison.Ordinal); + Assert.DoesNotContain(AlwaysOnXeSessions.AzureCapsSentence, line, StringComparison.Ordinal); + }); + + /* The pass an hour later asks again, and says nothing new at Warning. */ + await rig.EnsureAsync(); + await rig.EnsureAsync(); + Assert.Equal(expected, rig.Loud().OrderBy(line => line, StringComparer.Ordinal)); + } + + private static SqlException ReadOnlyRefusal() => + LongQueryTraceReadOnlyIntentTests.SqlExceptionFactory.Create( + LongQueryTraceDatabases.ReadOnlyDatabaseErrorNumber, 16, "Failed to update database because the database is read-only."); +} diff --git a/Darling/Darling.Tests/AlwaysOnXeSessionsTests.cs b/Darling/Darling.Tests/AlwaysOnXeSessionsTests.cs new file mode 100644 index 0000000000..f1dcd96f7c --- /dev/null +++ b/Darling/Darling.Tests/AlwaysOnXeSessionsTests.cs @@ -0,0 +1,337 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// The deadlock and blocked-process sessions on Azure SQL Database (#4961), as the service ensures them: a stopped shared +/// session gets a start by name, one that stays unusable sends the install to its own session, the install switches back +/// once the shared one reads back usable, no path drops a shared-name session, and the read takes the name the ensure chose. +/// The service ensures at connect and then once an hour, so a session dropped from outside comes back within the hour. Each +/// test drives the real ensure against a scripted database, so no server or store is needed. +/// +public sealed class AlwaysOnXeSessionsTests : IAsyncDisposable +{ + private const string Host = "alwayson.database.windows.net"; + private const string InstallIdValue = "0a1b2c3d"; + + /* Never opened: the runner reaches the store only to write collected rows, and these tests collect nothing. */ + private readonly NpgsqlDataSource _store = NpgsqlDataSource.Create("Host=127.0.0.1;Port=1;Database=unused;Username=unused"); + + public ValueTask DisposeAsync() => _store.DisposeAsync(); + + private static string Own(AlwaysOnXeSessionKind kind) => + AlwaysOnXeSessions.OwnNameFor(LongQueryCompletionsCollector.DarlingProduct, InstallIdValue, kind); + + [Theory] + [InlineData(AlwaysOnXeSessionKind.Deadlock)] + [InlineData(AlwaysOnXeSessionKind.BlockedProcess)] + public void TheSharedSessionStartsWithTheDatabase_AndThisInstallsOwnFallbackSessionDoesNot(AlwaysOnXeSessionKind kind) + { + var shared = AlwaysOnXeSessions.BuildAzureCreateSql(kind, AlwaysOnXeSessions.SharedNameFor(kind)); + var own = AlwaysOnXeSessions.BuildAzureCreateSql(kind, Own(kind)); + + Assert.Contains("STARTUP_STATE = ON", shared, StringComparison.Ordinal); + Assert.DoesNotContain("STARTUP_STATE = OFF", shared, StringComparison.Ordinal); + Assert.Contains("STARTUP_STATE = OFF", own, StringComparison.Ordinal); + Assert.DoesNotContain("STARTUP_STATE = ON", own, StringComparison.Ordinal); + } + + private sealed class AlreadyThere : Exception + { + public AlreadyThere() : base("already there") { } + } + + /// One Azure SQL Database as the ensure sees it: a catalog, a started-sessions DMV, and the statements sent. + private sealed class ScriptedDatabase : IAlwaysOnXeDatabase + { + public Dictionary Started { get; } = new(StringComparer.Ordinal); + public HashSet WrongEvent { get; } = new(StringComparer.Ordinal); + public List Statements { get; } = new(); + public Exception? CreateFailure { get; set; } + + public Task ReadCatalogAsync(AlwaysOnXeSessionKind kind, string sessionName, CancellationToken cancellationToken) => + Task.FromResult( + !Started.ContainsKey(sessionName) ? AlwaysOnXeCatalog.Missing + : WrongEvent.Contains(sessionName) && kind == AlwaysOnXeSessionKind.Deadlock ? AlwaysOnXeCatalog.WrongEvent + : AlwaysOnXeCatalog.Present); + + public Task IsStartedAsync(string sessionName, CancellationToken cancellationToken) => + Task.FromResult(Started.TryGetValue(sessionName, out var started) && started); + + public Task ExecuteAsync(string statement, CancellationToken cancellationToken) + { + Statements.Add(statement); + var name = statement[(statement.IndexOf('[') + 1)..statement.IndexOf(']')]; + + if (statement.TrimStart().StartsWith("CREATE EVENT SESSION", StringComparison.Ordinal)) + { + if (CreateFailure is not null) + { + throw CreateFailure; + } + + if (Started.ContainsKey(name)) + { + throw new AlreadyThere(); + } + + Started[name] = true; + } + else if (statement.StartsWith("ALTER EVENT SESSION", StringComparison.Ordinal)) + { + if (Started.TryGetValue(name, out var already) && already) + { + throw new AlreadyThere(); + } + + Started[name] = true; + } + else if (statement.StartsWith("DROP EVENT SESSION", StringComparison.Ordinal)) + { + Started.Remove(name); + } + + return Task.CompletedTask; + } + + public bool IsAlreadyPresent(Exception exception) => exception is AlreadyThere; + + public int Created(string name) => + Statements.Count(s => s.TrimStart().StartsWith("CREATE EVENT SESSION [" + name + "]", StringComparison.Ordinal)); + + public bool Sent(string prefix, string name) => + Statements.Any(s => s.TrimStart().StartsWith(prefix + " [" + name + "]", StringComparison.Ordinal)); + } + + private sealed class Rig + { + public required DarlingCollectorRunner Runner { get; init; } + public required ServerRuntime Runtime { get; init; } + public required DarlingWorker.ServerLoopState State { get; init; } + public ScriptedDatabase Alpha { get; } = new(); + public ScriptedDatabase Beta { get; } = new(); + public int ServerScopedEnsures { get; set; } + } + + private Rig BuildRig(bool azureSqlDatabase = true) + { + var config = new MonitoredServer { Name = "alwayson", Host = Host }; + var runtime = new ServerRuntime + { + Config = config, + ConnectionString = $"Server=tcp:{Host},1433;Initial Catalog=master;Encrypt=True", + Target = new CollectorTargetInfo { IsAzureSqlDb = azureSqlDatabase }, + StorageName = Host, + ServerId = 4242, + }; + var runner = new DarlingCollectorRunner( + _store, + new CollectorDeltaCalculator(), + databaseScope: (_, _) => new List(), + separatelyMonitoredDatabases: _ => new List(), + installId: () => InstallIdValue); + + var rig = new Rig + { + Runner = runner, + Runtime = runtime, + State = new DarlingWorker.ServerLoopState { Config = config, Runtime = runtime }, + }; + + runner.AzureDatabaseListOverrideForTests = (_, _) => Task.FromResult(new List { "master", "alpha", "beta" }); + runner.AlwaysOnXeDatabaseForTests = (_, database, _) => + Task.FromResult(string.Equals(database, "alpha", StringComparison.OrdinalIgnoreCase) ? rig.Alpha : rig.Beta); + runner.XeEnsureOverrideForTests = (_, _) => + { + rig.ServerScopedEnsures++; + return Task.CompletedTask; + }; + + return rig; + } + + private static Task EnsureAsync(Rig rig) => + DarlingXeSessions.EnsureAllAsync(rig.Runtime, rig.Runner, NullLogger.Instance, CancellationToken.None); + + /* Test 16: a stopped shared session gets a START by name and no fallback. */ + [Fact] + public async Task AStoppedSharedSession_GetsAStartByName_AndNoFallback() + { + var rig = BuildRig(); + rig.Runner.XeEnsureOverrideForTests = null; + var deadlock = AlwaysOnXeSessions.SharedNameFor(AlwaysOnXeSessionKind.Deadlock); + var blocked = AlwaysOnXeSessions.SharedNameFor(AlwaysOnXeSessionKind.BlockedProcess); + rig.Alpha.Started[deadlock] = false; + rig.Alpha.Started[blocked] = false; + + await EnsureAsync(rig); + + Assert.True(rig.Alpha.Started[deadlock]); + Assert.True(rig.Alpha.Started[blocked]); + Assert.Equal(0, rig.Alpha.Created(Own(AlwaysOnXeSessionKind.Deadlock))); + Assert.Equal(deadlock, rig.Runner.AlwaysOnReadSessionName(rig.Runtime, "alpha", AlwaysOnXeSessionKind.Deadlock)); + Assert.DoesNotContain(rig.Alpha.Statements, s => s.StartsWith("DROP", StringComparison.Ordinal)); + } + + /* Tests 15 and 17: a shared deadlock session without the deadlock event is never dropped, the install reads its own. */ + [Fact] + public async Task ASharedDeadlockSessionWithoutTheDeadlockEvent_IsNeverDropped_AndTheReadTakesTheOwnName() + { + var rig = BuildRig(); + rig.Runner.XeEnsureOverrideForTests = null; + var shared = AlwaysOnXeSessions.SharedNameFor(AlwaysOnXeSessionKind.Deadlock); + rig.Alpha.Started[shared] = true; + rig.Alpha.WrongEvent.Add(shared); + + await EnsureAsync(rig); + + Assert.False(rig.Alpha.Sent("DROP EVENT SESSION", shared)); + Assert.Equal(Own(AlwaysOnXeSessionKind.Deadlock), rig.Runner.AlwaysOnReadSessionName(rig.Runtime, "alpha", AlwaysOnXeSessionKind.Deadlock)); + Assert.Equal(shared, rig.Runner.AlwaysOnReadSessionName(rig.Runtime, "beta", AlwaysOnXeSessionKind.Deadlock)); + Assert.Contains("STARTUP_STATE = OFF", rig.Alpha.Statements.Single(s => s.Contains("CREATE EVENT SESSION [" + Own(AlwaysOnXeSessionKind.Deadlock) + "]", StringComparison.Ordinal)), StringComparison.Ordinal); + } + + /* Test 18: a shared session that reads back usable makes the install drop its fallback and switch back. */ + [Fact] + public async Task TheInstallOnItsOwnSession_DropsItAndSwitchesBack_WhenTheSharedSessionReadsBackUsable() + { + var rig = BuildRig(); + rig.Runner.XeEnsureOverrideForTests = null; + var shared = AlwaysOnXeSessions.SharedNameFor(AlwaysOnXeSessionKind.Deadlock); + rig.Alpha.Started[shared] = true; + rig.Alpha.WrongEvent.Add(shared); + await EnsureAsync(rig); + Assert.True(rig.Alpha.Started.ContainsKey(Own(AlwaysOnXeSessionKind.Deadlock))); + + rig.Alpha.WrongEvent.Clear(); + await EnsureAsync(rig); + + Assert.False(rig.Alpha.Started.ContainsKey(Own(AlwaysOnXeSessionKind.Deadlock))); + Assert.True(rig.Alpha.Sent("DROP EVENT SESSION", Own(AlwaysOnXeSessionKind.Deadlock))); + Assert.False(rig.Alpha.Sent("DROP EVENT SESSION", shared)); + Assert.Equal(shared, rig.Runner.AlwaysOnReadSessionName(rig.Runtime, "alpha", AlwaysOnXeSessionKind.Deadlock)); + } + + /* Test 19, Azure: sessions dropped from outside come back within the hour, and not before it. */ + [Fact] + public async Task AzureSessionsDroppedFromOutside_ComeBackWithinTheHour() + { + var rig = BuildRig(); + rig.Runner.XeEnsureOverrideForTests = null; + var start = new DateTime(2026, 1, 1, 12, 0, 0, DateTimeKind.Utc); + var deadlock = AlwaysOnXeSessions.SharedNameFor(AlwaysOnXeSessionKind.Deadlock); + var blocked = AlwaysOnXeSessions.SharedNameFor(AlwaysOnXeSessionKind.BlockedProcess); + + await DarlingWorker.EnsureAlwaysOnXeSessionsAsync(rig.State, rig.Runner, start, NullLogger.Instance, CancellationToken.None); + Assert.True(rig.Alpha.Started[deadlock]); + Assert.True(rig.Beta.Started[blocked]); + + rig.Alpha.Started.Clear(); + rig.Beta.Started.Clear(); + await DarlingWorker.EnsureAlwaysOnXeSessionsAsync(rig.State, rig.Runner, start.AddMinutes(30), NullLogger.Instance, CancellationToken.None); + Assert.Empty(rig.Alpha.Started); + + await DarlingWorker.EnsureAlwaysOnXeSessionsAsync(rig.State, rig.Runner, start.AddMinutes(61), NullLogger.Instance, CancellationToken.None); + Assert.True(rig.Alpha.Started[deadlock]); + Assert.True(rig.Alpha.Started[blocked]); + Assert.True(rig.Beta.Started[deadlock]); + Assert.True(rig.Beta.Started[blocked]); + } + + /* Test 19, server scope: the ensure runs once an hour as well as at connect. */ + [Fact] + public async Task TheServerScopedEnsure_RunsAgainOnceTheHourIsUp() + { + var rig = BuildRig(azureSqlDatabase: false); + var start = new DateTime(2026, 1, 1, 12, 0, 0, DateTimeKind.Utc); + + await DarlingWorker.EnsureAlwaysOnXeSessionsAsync(rig.State, rig.Runner, start, NullLogger.Instance, CancellationToken.None); + await DarlingWorker.EnsureAlwaysOnXeSessionsAsync(rig.State, rig.Runner, start.AddMinutes(59), NullLogger.Instance, CancellationToken.None); + Assert.Equal(1, rig.ServerScopedEnsures); + + await DarlingWorker.EnsureAlwaysOnXeSessionsAsync(rig.State, rig.Runner, start.AddMinutes(60), NullLogger.Instance, CancellationToken.None); + Assert.Equal(2, rig.ServerScopedEnsures); + Assert.Equal(start.AddMinutes(60), rig.State.XeSessionsEnsuredAtUtc); + } + + /* Test 27: an Azure create that fails carries the caps sentence in the line the service logs. */ + [Fact] + public async Task AFailedAzureCreate_LogsTheCapsSentence() + { + var rig = BuildRig(); + rig.Runner.XeEnsureOverrideForTests = null; + rig.Alpha.CreateFailure = new InvalidOperationException("The create was refused."); + var logger = new CapturingLogger(); + + await DarlingXeSessions.EnsureAllAsync(rig.Runtime, rig.Runner, logger, CancellationToken.None); + + var line = Assert.Single(logger.Lines, l => l.Contains("alpha", StringComparison.Ordinal) && l.Contains("deadlock", StringComparison.Ordinal)); + Assert.Contains("The create was refused.", line, StringComparison.Ordinal); + Assert.Contains(AlwaysOnXeSessions.AzureCapsSentence, line, StringComparison.Ordinal); + } + + /* Test 15, pinned at the source: the Azure ensure in this service holds no drop of a shared name. */ + [Fact] + public void TheAzureEnsure_HoldsNoDropOfASharedName() + { + var source = ReadRepoFileLf("Darling", "PerformanceMonitor.Darling.Service", "DarlingXeSessions.cs"); + + Assert.DoesNotContain("DROP EVENT SESSION [{DeadlocksCollector.XeSessionName}]", source, StringComparison.Ordinal); + Assert.DoesNotContain("DROP EVENT SESSION [{sessionName}] ON DATABASE;\", connection))\n {\n dropCmd.CommandTimeout = 60;", source, StringComparison.Ordinal); + } + + /* Test 17: the read builders name the session the ensure chose. */ + [Fact] + public void TheReadBuildersNameTheSessionTheEnsureChose() + { + var own = Own(AlwaysOnXeSessionKind.Deadlock); + var ownBlocked = Own(AlwaysOnXeSessionKind.BlockedProcess); + + Assert.Contains($"N'{own}'", DeadlocksCollector.Instance.BuildQuery(AzureContext(own)).Text, StringComparison.Ordinal); + Assert.DoesNotContain("N'PerformanceMonitor_Deadlock'", DeadlocksCollector.Instance.BuildQuery(AzureContext(own)).Text, StringComparison.Ordinal); + Assert.Contains("N'PerformanceMonitor_Deadlock'", DeadlocksCollector.Instance.BuildQuery(AzureContext(null)).Text, StringComparison.Ordinal); + Assert.Contains($"N'{ownBlocked}'", BlockedProcessReportCollector.Instance.BuildQuery(AzureContext(ownBlocked)).Text, StringComparison.Ordinal); + Assert.DoesNotContain("N'PerformanceMonitor_BlockedProcess'", BlockedProcessReportCollector.Instance.BuildQuery(AzureContext(ownBlocked)).Text, StringComparison.Ordinal); + } + + private static CollectorContext AzureContext(string? sessionName) => new() + { + ServerId = 1, + ServerName = "alwayson", + CollectionTime = new DateTime(2026, 1, 1, 12, 0, 0, DateTimeKind.Utc), + Deltas = new CollectorDeltaCalculator(), + Target = new CollectorTargetInfo { IsAzureSqlDb = true }, + CurrentDatabaseName = "alpha", + AlwaysOnSessionName = sessionName, + }; + + private sealed class CapturingLogger : Microsoft.Extensions.Logging.ILogger + { + public List Lines { get; } = new(); + + public IDisposable? BeginScope(TState state) where TState : notnull => null; + + public bool IsEnabled(Microsoft.Extensions.Logging.LogLevel logLevel) => true; + + public void Log(Microsoft.Extensions.Logging.LogLevel logLevel, Microsoft.Extensions.Logging.EventId eventId, TState state, Exception? exception, Func formatter) => + Lines.Add(formatter(state, exception)); + } +} diff --git a/Darling/Darling.Tests/AnalysisFactsReadRunsDetectorParityTests.cs b/Darling/Darling.Tests/AnalysisFactsReadRunsDetectorParityTests.cs index 5842e2fd43..cb611ccbf6 100644 --- a/Darling/Darling.Tests/AnalysisFactsReadRunsDetectorParityTests.cs +++ b/Darling/Darling.Tests/AnalysisFactsReadRunsDetectorParityTests.cs @@ -35,6 +35,7 @@ namespace Darling.Tests; /// service over the batch-request spike fixture; Darling's gated PgTargetAnomalyTests e2e asserts /// the planted TPS spike through the real get_analysis_facts. This file is the shape both rest on. /// +[Trait("Reads", "Lite")] public sealed class AnalysisFactsReadRunsDetectorParityTests { private const string ReadEntry = "public async Task<(List Facts, WindowCoverage? Coverage, CollectionCaveatState Caveats)> CollectAndScoreFactsAsync("; diff --git a/Darling/Darling.Tests/AnalysisFindingsTabTests.cs b/Darling/Darling.Tests/AnalysisFindingsTabTests.cs new file mode 100644 index 0000000000..13a858e110 --- /dev/null +++ b/Darling/Darling.Tests/AnalysisFindingsTabTests.cs @@ -0,0 +1,185 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.ComponentModel; +using System.Diagnostics; +using System.Linq; +using System.Text.Json; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// The web Recommendations tab (#4843), the twin of the desktop viewer's analysis findings view. The source pins +/// name the read and its params and where the tab sits in the registry; the behaviour tests run the shipped +/// analysis-findings.js under Node (analysis-findings-harness.mjs), skipped when Node is not installed. +/// +public sealed class AnalysisFindingsTabTests +{ + private static string[] Tab() => new[] { "Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "pages", "analysis-findings.js" }; + + [Fact] + public void TheTab_ReadsGetAnalysisFindings_WithTheServerWindowAndFullText() + { + var src = ReadRepoFile(Tab()); + Assert.Contains("readToolWithinKeptHistory(\"get_analysis_findings\", { server, hours: ctx.hours, limit: 50, full_text: true }, ctx && ctx.signal)", src, StringComparison.Ordinal); + Assert.DoesNotContain("innerHTML", src, StringComparison.Ordinal); + Assert.DoesNotContain("analyze_server", src, StringComparison.Ordinal); + } + + [Fact] + public void TheTab_IsRegisteredLast_InTheSqlServerRegistryOnly() + { + var tabs = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "pages", "server-tabs.js").ReplaceLineEndings("\n"); + Assert.Contains("import { analysisFindingsTab } from \"./analysis-findings.js\";", tabs, StringComparison.Ordinal); + Assert.Contains(" analysisFindingsTab,\n];\n\n/* ─", tabs, StringComparison.Ordinal); + Assert.Equal(1, tabs.Split("analysisFindingsTab,", StringSplitOptions.None).Length - 1); + } + + private static JsonElement Run(string scenario) + { + var psi = new ProcessStartInfo("node") { RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false }; + psi.ArgumentList.Add(PathTo("Darling", "Darling.Tests", "analysis-findings-harness.mjs")); + psi.ArgumentList.Add(PathTo("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js")); + psi.ArgumentList.Add(scenario); + Process proc; + try { proc = Process.Start(psi)!; } + catch (Win32Exception) { Assert.Skip("Node is not installed, so the shipped page script cannot be run."); return default; } + using (proc) + { + var error = proc.StandardError.ReadToEndAsync(); + var output = proc.StandardOutput.ReadToEnd().Trim(); + if (!proc.WaitForExit(20000)) + { + proc.Kill(entireProcessTree: true); + Assert.Fail("the analysis findings harness did not finish in 20 s for scenario " + scenario); + } + Assert.True(proc.ExitCode == 0, "the analysis findings harness failed for " + scenario + ": " + error.Result); + using var doc = JsonDocument.Parse(output); + return doc.RootElement.Clone(); + } + } + + private static string[] Strings(JsonElement e) => e.EnumerateArray().Select(x => x.GetString()!).ToArray(); + + [Fact] + public void TheRead_GoesToTheApiWithTheServerHoursLimitAndFullText() + { + var r = Run("read"); + Assert.Equal("/api/read/get_analysis_findings", r.GetProperty("path").GetString()); + var p = r.GetProperty("params"); + Assert.Equal("srv-a", p.GetProperty("server").GetString()); + Assert.Equal("24", p.GetProperty("hours").GetString()); + Assert.Equal("50", p.GetProperty("limit").GetString()); + Assert.Equal("true", p.GetProperty("full_text").GetString()); + } + + [Fact] + public void Findings_GroupByIncident_MostSevereFirst_WithBadgesFromTheExistingBandClasses() + { + var r = Run("grouping"); + Assert.Equal(new[] { "Major · 2 findings · CRITICAL", "Config · CRITICAL", "Solo info · INFO" }, Strings(r.GetProperty("headers"))); + Assert.Equal(new[] { 2, 1, 1 }, r.GetProperty("cardsPerGroup").EnumerateArray().Select(x => x.GetInt32()).ToArray()); + // Info-only groups start collapsed, the rest open. + Assert.Equal(new[] { true, true, false }, r.GetProperty("open").EnumerateArray().Select(x => x.GetBoolean()).ToArray()); + var classes = Strings(r.GetProperty("badgeClasses")); + Assert.Contains("badge band-Critical", classes); + Assert.Contains("badge band-Warning", classes); + Assert.Contains("badge band-Unknown", classes); + } + + [Fact] + public void TheFixScript_IsText_InAPre_AndCopyPutsExactlyThatTextOnTheClipboard() + { + var r = Run("fix"); + const string fix = "ALTER DATABASE x SET AUTO_SHRINK OFF;"; + Assert.Equal(fix, Assert.Single(Strings(r.GetProperty("pre")))); + Assert.Equal(0, Assert.Single(r.GetProperty("preChildren").EnumerateArray()).GetInt32()); + Assert.Equal(fix, Assert.Single(Strings(r.GetProperty("clipboard")))); + Assert.Equal("Copied", r.GetProperty("label").GetString()); + Assert.Equal(1, r.GetProperty("buttons").GetInt32()); + } + + [Fact] + public void OnlyIncidentFindings_LinkToTheServersQueriesTab_NotConfigFixes() + { + var links = Strings(Run("link")); + Assert.Equal(3, links.Length); + Assert.All(links, l => Assert.Equal("#/server/srv-a/queries", l)); + } + + [Fact] + public void AForcePlanIncident_KeepsTheLink_AndAConfigFix_DoesNot() + { + var groups = Run("linkKinds").EnumerateArray().ToDictionary(g => g.GetProperty("head").GetString()!, g => g.GetProperty("links").GetInt32()); + Assert.Equal(1, groups.Single(g => g.Key.StartsWith("ForcePlan", StringComparison.Ordinal)).Value); + Assert.Equal(0, groups.Single(g => g.Key.StartsWith("ConfigFix", StringComparison.Ordinal)).Value); + } + + [Fact] + public void ANullSeverity_RendersTheLabelAlone_NotNaN() + { + var text = Run("nullSeverity").GetProperty("text").GetString()!; + Assert.DoesNotContain("NaN", text, StringComparison.Ordinal); + Assert.DoesNotContain("0.00", text, StringComparison.Ordinal); + Assert.Contains("INFO", text, StringComparison.Ordinal); + } + + [Fact] + public void TheReadCapNote_AndThePageNote_AreBothShown() + { + var text = Run("truncation").GetProperty("text").GetString()!; + Assert.Contains("READ-CAP-NOTE", text, StringComparison.Ordinal); + Assert.Contains("PAGE-NOTE", text, StringComparison.Ordinal); + } + + [Fact] + public void TheSecondaryReplicaNote_ShowsWithFindingsAndWithNone_AndIsAbsentWhenTheReadCarriesNone() + { + var r = Run("replicaNote"); + Assert.Contains("REPLICA-NOTE", r.GetProperty("withFindings").GetString(), StringComparison.Ordinal); + Assert.Contains("REPLICA-NOTE", r.GetProperty("noFindings").GetString(), StringComparison.Ordinal); + Assert.DoesNotContain("REPLICA-NOTE", r.GetProperty("without").GetString(), StringComparison.Ordinal); + Assert.DoesNotContain("REPLICA-NOTE", r.GetProperty("nullNote").GetString(), StringComparison.Ordinal); + Assert.DoesNotContain("null", r.GetProperty("nullNote").GetString(), StringComparison.Ordinal); + } + + [Fact] + public void TheTabsAbortSignal_ReachesTheRead() + { + Assert.True(Run("signal").GetProperty("passed").GetBoolean()); + } + + [Fact] + public void TheReadsConfigFixFlag_IsTheDesktopsRule_AndIsSharedWithTheViewer() + { + var tools = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "Mcp", "DarlingMcpTools.cs"); + Assert.Contains("is_config_fix = FactRemediation.IsConfigFix(f.Remediation)", tools, StringComparison.Ordinal); + var vm = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "RecommendationsViewModel.cs"); + Assert.Contains("HasStructuredFixAction => FactRemediation.IsConfigFix(Item.Remediation)", vm, StringComparison.Ordinal); + var js = ReadRepoFile(Tab()); + Assert.Contains("!f.is_config_fix", js, StringComparison.Ordinal); + Assert.DoesNotContain("f.structured_remediation", js, StringComparison.Ordinal); + } + + [Fact] + public void OpenAndClosedGroups_SurviveARebuild_PerServer() + { + var r = Run("rebuild"); + Assert.Equal(new[] { false, true, true }, r.GetProperty("again").EnumerateArray().Select(x => x.GetBoolean()).ToArray()); + Assert.Equal(new[] { true, true, false }, r.GetProperty("other").EnumerateArray().Select(x => x.GetBoolean()).ToArray()); + } + + [Fact] + public void AnEmptyRead_ShowsTheServersSentence() + { + Assert.Contains("No findings in the requested time range.", Run("empty").GetProperty("text").GetString(), StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/AnalysisReadCancellationPinTests.cs b/Darling/Darling.Tests/AnalysisReadCancellationPinTests.cs index fc1cf3deaa..4751697b8f 100644 --- a/Darling/Darling.Tests/AnalysisReadCancellationPinTests.cs +++ b/Darling/Darling.Tests/AnalysisReadCancellationPinTests.cs @@ -33,6 +33,7 @@ namespace Darling.Tests; /// public method, not a private catch site called directly), with no rig and no store, on a port nothing /// listens on. /// +[Trait("Stage", "Guard")] public sealed class AnalysisReadCancellationPinTests { private static NpgsqlDataSource DeadStore() => diff --git a/Darling/Darling.Tests/AnalysisWindowEmptyRuleParityTests.cs b/Darling/Darling.Tests/AnalysisWindowEmptyRuleParityTests.cs index b83c299dcc..a4bc6b60ab 100644 --- a/Darling/Darling.Tests/AnalysisWindowEmptyRuleParityTests.cs +++ b/Darling/Darling.Tests/AnalysisWindowEmptyRuleParityTests.cs @@ -30,6 +30,7 @@ namespace Darling.Tests; /// across the SKUs, and still carries the #3524 prose, because that prose is now true of exactly the case it /// is emitted for. /// +[Trait("Reads", "Lite")] public sealed class AnalysisWindowEmptyRuleParityTests { private const string PassEntry = "public async Task> AnalyzeAsync(AnalysisContext context)"; diff --git a/Darling/Darling.Tests/AnomalyDetectorErrorGuardLiveTests.cs b/Darling/Darling.Tests/AnomalyDetectorErrorGuardLiveTests.cs index 9deaa4206a..9e19b645eb 100644 --- a/Darling/Darling.Tests/AnomalyDetectorErrorGuardLiveTests.cs +++ b/Darling/Darling.Tests/AnomalyDetectorErrorGuardLiveTests.cs @@ -223,7 +223,7 @@ private static void RedFirstProof_DocumentedNotAsserted() private static async Task RunBothWindowsAsync(IAnomalyDetector detector, int serverId, string serverName) { var now = DateTime.SpecifyKind( - new DateTime(DateTime.UtcNow.Ticks - (DateTime.UtcNow.Ticks % TimeSpan.TicksPerSecond)), + new DateTime(DateTime.UtcNow.Ticks / TimeSpan.TicksPerSecond * TimeSpan.TicksPerSecond), DateTimeKind.Unspecified); foreach (var window in new[] { TimeSpan.FromHours(4), TimeSpan.FromHours(24) }) @@ -257,7 +257,7 @@ private static async Task RunBothWindowsAsync(IAnomalyDetector detector, i private static async Task SeedSqlServerFamilyTablesAsync(NpgsqlConnection connection, System.Threading.CancellationToken ct) { var now = DateTime.SpecifyKind( - new DateTime(DateTime.UtcNow.Ticks - (DateTime.UtcNow.Ticks % TimeSpan.TicksPerSecond)), + new DateTime(DateTime.UtcNow.Ticks / TimeSpan.TicksPerSecond * TimeSpan.TicksPerSecond), DateTimeKind.Unspecified); // Inside every 24h window this class runs. @@ -353,7 +353,7 @@ private static async Task DeleteSqlServerRowsAsync(NpgsqlConnection connection, private static async Task SeedPgTargetFamilyTablesAsync(NpgsqlConnection connection, System.Threading.CancellationToken ct) { var now = DateTime.SpecifyKind( - new DateTime(DateTime.UtcNow.Ticks - (DateTime.UtcNow.Ticks % TimeSpan.TicksPerSecond)), + new DateTime(DateTime.UtcNow.Ticks / TimeSpan.TicksPerSecond * TimeSpan.TicksPerSecond), DateTimeKind.Unspecified); var t = now.AddHours(-1); var priorMinute = t.AddMinutes(-1); diff --git a/Darling/Darling.Tests/AnomalyNullWindowLiveTests.cs b/Darling/Darling.Tests/AnomalyNullWindowLiveTests.cs index 5543b75e9a..e67d5bbf81 100644 --- a/Darling/Darling.Tests/AnomalyNullWindowLiveTests.cs +++ b/Darling/Darling.Tests/AnomalyNullWindowLiveTests.cs @@ -98,7 +98,7 @@ as residue. Move onto a PRIVATE scratch database instead — SharedBaselineCache /* now: truncated to whole seconds (the lane rule) — this becomes the "as_of" anchor for every pass; history runs back HistoryDays from it. */ var now = DateTime.SpecifyKind( - new DateTime(DateTime.UtcNow.Ticks - (DateTime.UtcNow.Ticks % TimeSpan.TicksPerSecond)), + new DateTime(DateTime.UtcNow.Ticks / TimeSpan.TicksPerSecond * TimeSpan.TicksPerSecond), DateTimeKind.Unspecified); var historyStart = now.AddDays(-HistoryDays); diff --git a/Darling/Darling.Tests/AnomalyTileWindowEndTests.cs b/Darling/Darling.Tests/AnomalyTileWindowEndTests.cs index 0fe7454d5c..dc3ca673fb 100644 --- a/Darling/Darling.Tests/AnomalyTileWindowEndTests.cs +++ b/Darling/Darling.Tests/AnomalyTileWindowEndTests.cs @@ -27,6 +27,7 @@ namespace Darling.Tests; /// behavioural half, which seeds a sample at the end and runs the real detector, is /// AnomalyTileWindowEndTests in Lite.Tests. /// +[Trait("Reads", "Lite")] public sealed class AnomalyTileWindowEndTests { /// The closed spelling of the window end: a boundary sample is read. diff --git a/Darling/Darling.Tests/AsOfWindowAnchorTests.cs b/Darling/Darling.Tests/AsOfWindowAnchorTests.cs index 96ed45f88d..2568b40c3b 100644 --- a/Darling/Darling.Tests/AsOfWindowAnchorTests.cs +++ b/Darling/Darling.Tests/AsOfWindowAnchorTests.cs @@ -35,6 +35,7 @@ namespace Darling.Tests; /// which reads carry the parameter. The live half — seed a past window, read it anchored, read it /// unanchored, prove the two disagree — is . /// +[Trait("Reads", "Lite")] public sealed class AsOfWindowAnchorTests { /* ── the resolver's contract ── */ @@ -242,6 +243,7 @@ public void TheSharedAnchorDescriptions_StateTheRule_WithinTheParameterBudget_An public void EveryDispatchedReadWhoseToolTakesAnAnchor_AdvertisesItInTheCatalog() { var missing = DarlingWebEndpoints.BuildReadDispatch().Keys + .Where(name => !DarlingWebEndpoints.WebOnlyReadNames.Contains(name)) // no tool behind it, and a keyed read has no window (#5241) .Where(ToolTakesAnAnchor) .Where(name => !DarlingWebEndpoints.CatalogDescriptors[name].Params.Any(p => p.Name == "as_of")) .OrderBy(n => n, StringComparer.Ordinal) @@ -281,10 +283,13 @@ anchor. What is left are the two permanent kinds. get_pvs_stats and get_fleet_ov result whose two halves describe different instants, which is worse than not offering it. get_store_metrics windows in days over the store's own growth series. get_read_latency windows in hours over the service's own read-duration telemetry, like get_store_metrics over the store's - own growth series. */ + own growth series. get_finops windows over the query-stats rows ending now and takes no anchor + (its description says so); the FinOps views are cost-oriented live reads. get_slow_reads windows in hours + back from now over the service's own slow-read telemetry, like get_read_latency, and takes no anchor. get_store_query_history windows in hours back from now over the store's own statement history and takes no anchor. */ var excluded = new[] { "get_pvs_stats", "get_fleet_overview", "get_store_metrics", "get_read_latency", + "get_finops", "get_slow_reads", "get_store_query_history", }; var unanchored = DarlingWebEndpoints.BuildReadDispatch().Keys @@ -555,6 +560,7 @@ private static bool ToolTakesAnAnchor(string readName) /// a demonstrably different answer, not the same one with more rows. /// [Collection("live-postgres")] +[Trait("Reads", "Lite")] public sealed class AsOfWindowAnchorLivePostgresTests { private const string ServerName = "darling-asof-anchor-e2e"; diff --git a/Darling/Darling.Tests/AvailabilityGroupsTabRefreshTests.cs b/Darling/Darling.Tests/AvailabilityGroupsTabRefreshTests.cs index fcda327969..af44a5fbd7 100644 --- a/Darling/Darling.Tests/AvailabilityGroupsTabRefreshTests.cs +++ b/Darling/Darling.Tests/AvailabilityGroupsTabRefreshTests.cs @@ -35,7 +35,7 @@ public void Render_SameRowsAcrossTwoCalls_KeepsTheSameCardInstances() /* The pin the ruling asks for: two refreshes over identical rows must not hand the ItemsControl a new object per card. Before the fix, Render assigns a brand new List (brand new instances) every call, so this fails on the pre-#4238 code — proven once, below. */ - OnStaThread(() => + StaTestThread.Run(() => { var tab = new AvailabilityGroupsTab(); @@ -56,7 +56,7 @@ public void Render_SameRowsAcrossTwoCalls_KeepsTheSameCardInstances() [Fact] public void Render_ChangedValue_UpdatesInPlace_SameInstanceNewValue() { - OnStaThread(() => + StaTestThread.Run(() => { var tab = new AvailabilityGroupsTab(); @@ -80,7 +80,7 @@ public void Render_ChangedValue_UpdatesInPlace_SameInstanceNewValue() [Fact] public void Render_AddedOrRemovedAg_AddsOrRemovesExactlyOneCard() { - OnStaThread(() => + StaTestThread.Run(() => { var tab = new AvailabilityGroupsTab(); @@ -129,7 +129,7 @@ public void Render_UnchangedRows_ReplacesNoCardReplicaOrDatabaseInstances() reconcile every card, replica and database IN PLACE and never rebuild any of them. A regression that tears down and rebuilds rows -- or an O(n^2) loop that amounts to the same thing -- shows up here as a nonzero replaced count, independent of how loaded the runner is. */ - OnStaThread(() => + StaTestThread.Run(() => { var tab = new AvailabilityGroupsTab(); var topology = BuildTopology(agCount: 42, replicasPerAg: 2, dbRowsPerAg: 9, extraDbRowsOnFirst: 20); @@ -253,22 +253,4 @@ private static AgTopologyDatabaseRow Database(int serverId, string serverName, s RedoRateKbPerSec = 0, }; - /// WPF objects require STA; same shape as RawWindowFloorViewerPortTests / Lite.Tests' MainWindowAccessKeyTests. - private static void OnStaThread(Action body) - { - Exception? error = null; - var thread = new Thread(() => - { - try { body(); } - catch (Exception ex) { error = ex; } - }); - thread.SetApartmentState(ApartmentState.STA); - thread.Start(); - thread.Join(); - - if (error is not null) - { - throw error; - } - } } diff --git a/Darling/Darling.Tests/AwsAssumeRolePackageTests.cs b/Darling/Darling.Tests/AwsAssumeRolePackageTests.cs new file mode 100644 index 0000000000..4d70f21906 --- /dev/null +++ b/Darling/Darling.Tests/AwsAssumeRolePackageTests.cs @@ -0,0 +1,73 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.IO; +using Xunit; + +namespace Darling.Tests; + +/// +/// The Darling service ships the assembly the AWS SDK loads to assume a role (#5431). +/// +/// The defect. Every RDS and Performance Insights client is built with a region only, so +/// credentials come from the SDK's default chain, and a role_arn profile selected through +/// AWS_PROFILE is part of that chain. The SDK resolves such a profile into +/// AssumeRoleAWSCredentials, which calls sts:AssumeRole through AWSSDK.SecurityToken +/// and loads that assembly by name at the moment it first needs credentials. The service carried +/// AWSSDK.Core, .RDS and .PI but not .SecurityToken, so the profile was found and +/// then failed with "Assembly AWSSDK.SecurityToken could not be found or loaded" before any call reached +/// AWS — a runtime failure on a path that compiles, and nothing in the service's own source names the type, +/// so a package that is simply absent is invisible to the compiler. +/// +/// What this pins. The package reference is what puts the assembly in the service's +/// deps.json and its publish folder, and the test project references the service, so the same +/// assembly is resolvable from this test's own output. Two arms: the service's deps.json lists it +/// (what the publish step copies from), and the type the SDK loads resolves by assembly-qualified name. +/// Removing the PackageReference from the service project fails both. +/// +/// What this cannot see. That a given profile reaches STS, or that the role's trust policy +/// allows the host: those are AWS-side and need credentials this suite never has. The publish folder itself +/// is not inspected here; the framework-dependent publish in the release workflow copies every +/// deps.json library, and the PR that added this checked it by publishing. +/// +public sealed class AwsAssumeRolePackageTests +{ + private const string Package = "AWSSDK.SecurityToken"; + + [Fact] + public void ServiceDepsJson_ListsTheSecurityTokenPackage() + { + var depsPath = Path.Combine( + AppContext.BaseDirectory, "PerformanceMonitor.Darling.Service.deps.json"); + + Assert.True( + File.Exists(depsPath), + $"the service's deps.json is not beside the test binaries ({depsPath}), so the test " + + "project no longer reaches the service's dependency closure and this arm proves nothing"); + + var deps = File.ReadAllText(depsPath); + + /* The library entry is "AWSSDK.SecurityToken/". Matching the slash keeps any other + mention of the package name from satisfying it. */ + Assert.Contains($"\"{Package}/", deps, StringComparison.Ordinal); + } + + [Fact] + public void TheStsClientTheSdkLoadsForAssumeRole_ResolvesFromTheServiceOutput() + { + var type = Type.GetType( + "Amazon.SecurityToken.AmazonSecurityTokenServiceClient, " + Package, + throwOnError: false); + + Assert.True( + type is not null, + $"{Package} cannot be loaded, so a role_arn profile fails with \"Assembly {Package} could " + + "not be found or loaded\" instead of calling sts:AssumeRole (#5431)"); + } +} diff --git a/Darling/Darling.Tests/AwsAssumedRoleCredentialsTests.cs b/Darling/Darling.Tests/AwsAssumedRoleCredentialsTests.cs new file mode 100644 index 0000000000..44759487a0 --- /dev/null +++ b/Darling/Darling.Tests/AwsAssumedRoleCredentialsTests.cs @@ -0,0 +1,530 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Text.RegularExpressions; +using System.Threading; +using System.Threading.Tasks; +using Amazon; +using Amazon.Runtime; +using Amazon.Runtime.Endpoints; +using Amazon.SecurityToken; +using Amazon.SecurityToken.Model; +using PerformanceMonitor.Darling.Service.Targets; +using Xunit; + +namespace Darling.Tests; + +/// +/// The credentials class for a per-server AWS role (#5452): how often it calls STS, when it refreshes, what it serves +/// while a refresh runs or fails, what it sends, and which exception each failure becomes. No AWS and no network: a +/// fake STS and a clock the test moves. +/// +public sealed class AwsAssumedRoleCredentialsTests +{ + private const string Role = "arn:aws:iam::123456789012:role/darling-monitor"; + private const string Sentinel = "sentinel-external-id-4417"; + + private static readonly RegionEndpoint UsEast1 = RegionEndpoint.USEast1; + + private static AwsAssumedRoleCredentials Make( + FakeSts sts, ManualTimeProvider clock, string? externalId = null, TimeSpan? deadline = null, + Func>? source = null, CapturingAwsLogger? logger = null, Func? installId = null) => + new(new AwsRoleKey(Role, externalId), UsEast1, sts.Factory, source ?? FakeSts.Source, installId, clock, deadline, logger); + + /* ---- how many STS calls ------------------------------------------------------------------------------ */ + + [Fact] + public async Task ManyCallersAtOnce_MakeOneStsCall() + { + var sts = new FakeSts(async (_, token) => + { + await Task.Delay(50, token); + return FakeSts.Grant(); + }); + var credentials = Make(sts, new ManualTimeProvider()); + + var all = await Task.WhenAll(Enumerable.Range(0, 20).Select(_ => credentials.GetCredentialsAsync())); + + Assert.Equal(1, sts.Calls); + Assert.All(all, c => Assert.Equal("ASIAEXAMPLEASSUMED", c.AccessKey)); + } + + [Fact] + public async Task ValidCredentials_AreServedWithNoNewCall_UntilFifteenMinutesBeforeExpiry() + { + var clock = new ManualTimeProvider(); + var sts = new FakeSts(); + var credentials = Make(sts, clock); + + await credentials.GetCredentialsAsync(); + clock.Advance(TimeSpan.FromMinutes(44)); + await credentials.GetCredentialsAsync(); + Assert.Equal(1, sts.Calls); + + clock.Advance(TimeSpan.FromMinutes(2)); + await credentials.GetCredentialsAsync(); + Assert.Equal(2, sts.Calls); + } + + [Fact] + public async Task ExpiryIsReceiptPlusDuration_NotTheResponsesExpiration() + { + var clock = new ManualTimeProvider(); + + /* An Expiration a minute away on AWS's clock would force a call per request if it were read; one in the past + likewise. Neither is read. */ + foreach (DateTime? expiration in new DateTime?[] { null, DateTime.UtcNow.AddMinutes(1), DateTime.UtcNow.AddDays(-1) }) + { + var sts = new FakeSts((_, _) => Task.FromResult(FakeSts.Grant(expiration: expiration))); + var credentials = Make(sts, clock); + + await credentials.GetCredentialsAsync(); + clock.Advance(TimeSpan.FromMinutes(30)); + await credentials.GetCredentialsAsync(); + Assert.Equal(1, sts.Calls); + clock.Advance(TimeSpan.FromMinutes(-30)); + } + } + + [Fact] + public void TheRefreshPointIsNeverSoonerThanTheMinimumIntervalAfterReceipt() + { + var received = new DateTimeOffset(2026, 10, 7, 12, 0, 0, TimeSpan.Zero); + + Assert.Equal(received.AddMinutes(45), AwsAssumedRoleCredentials.RefreshPointFor(received, received.AddHours(1))); + Assert.Equal(received.AddMinutes(1), AwsAssumedRoleCredentials.RefreshPointFor(received, received.AddMinutes(5))); + Assert.Equal(received.AddMinutes(1), AwsAssumedRoleCredentials.RefreshPointFor(received, received.AddSeconds(10))); + } + + /* ---- serving without blocking ------------------------------------------------------------------------ */ + + [Fact] + public async Task PastTheRefreshPoint_OneCallerRefreshes_AndTheOthersKeepUsingTheValidCredentials() + { + var clock = new ManualTimeProvider(); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var blockNext = false; + var sts = new FakeSts(async (_, _) => + { + if (blockNext) + { + entered.TrySetResult(); + await release.Task; + return FakeSts.Grant("second-token"); + } + + return FakeSts.Grant("first-token"); + }); + var credentials = Make(sts, clock); + + var first = await credentials.GetCredentialsAsync(); + Assert.Equal("first-token", first.Token); + + clock.Advance(TimeSpan.FromMinutes(50)); + blockNext = true; + var refresher = credentials.GetCredentialsAsync(); + await entered.Task.WaitAsync(TimeSpan.FromSeconds(10)); + + /* The refresh is in flight and holds the gate: these must come back at once, with the old credentials. */ + var served = await Task.WhenAll(Enumerable.Range(0, 5).Select(_ => credentials.GetCredentialsAsync())) + .WaitAsync(TimeSpan.FromSeconds(5)); + Assert.All(served, c => Assert.Equal("first-token", c.Token)); + Assert.False(refresher.IsCompleted); + + release.SetResult(); + Assert.Equal("second-token", (await refresher).Token); + Assert.Equal(2, sts.Calls); + } + + [Fact] + public async Task ACallerWithNothingValid_WaitsForTheRefresh_AndSharesItsAnswer() + { + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var sts = new FakeSts(async (_, _) => + { + await release.Task; + return FakeSts.Grant(); + }); + var credentials = Make(sts, new ManualTimeProvider()); + + var callers = Enumerable.Range(0, 4).Select(_ => credentials.GetCredentialsAsync()).ToList(); + await Task.Delay(100); + Assert.All(callers, c => Assert.False(c.IsCompleted)); + + release.SetResult(); + await Task.WhenAll(callers).WaitAsync(TimeSpan.FromSeconds(10)); + Assert.Equal(1, sts.Calls); + } + + /* ---- what is sent -------------------------------------------------------------------------------------- */ + + [Fact] + public async Task TheRequest_CarriesTheRole_AnHourOfDuration_AndTheSessionName() + { + var sts = new FakeSts(); + var credentials = Make(sts, new ManualTimeProvider(), installId: () => "0123abcd-4567-89ef-0123-456789abcdef"); + + await credentials.GetCredentialsAsync(); + + var request = Assert.Single(sts.Requests); + Assert.Equal(Role, request.RoleArn); + Assert.Equal(3600, request.DurationSeconds); + Assert.Equal("darling-0123abcd4567", request.RoleSessionName); + Assert.Null(request.ExternalId); + } + + [Fact] + public async Task TheExternalId_IsSentOnlyWhenSet() + { + var withId = new FakeSts(); + await Make(withId, new ManualTimeProvider(), Sentinel).GetCredentialsAsync(); + Assert.Equal(Sentinel, Assert.Single(withId.Requests).ExternalId); + + var blank = new FakeSts(); + await Make(blank, new ManualTimeProvider(), externalId: string.Empty).GetCredentialsAsync(); + Assert.Null(Assert.Single(blank.Requests).ExternalId); + } + + [Fact] + public async Task TheStsClientIsGivenASnapshotOfTheSourceCredentials() + { + var sts = new FakeSts(); + await Make(sts, new ManualTimeProvider()).GetCredentialsAsync(); + + var seen = Assert.Single(sts.SourceCredentialsSeen); + Assert.Equal("AKIAEXAMPLESOURCE", seen.AccessKey); + Assert.Equal("example-source-secret", seen.SecretKey); + } + + [Theory] + [InlineData("0123abcd-4567-89ef-0123-456789abcdef", "darling-0123abcd4567")] + [InlineData("0123ABCD456789EF0123456789ABCDEF", "darling-0123abcd4567")] + [InlineData(" 0123abcd-4567-89ef-0123-456789abcdef ", "darling-0123abcd4567")] + [InlineData("0a1b2c3d", "darling-0a1b2c3d")] + [InlineData("0A1B2C3D", "darling-0a1b2c3d")] + [InlineData(null, "darling-collector")] + [InlineData("", "darling-collector")] + [InlineData("abc", "darling-collector")] + [InlineData("0a1b2c3", "darling-collector")] + [InlineData("not-a-hex-install-id-at-all", "darling-collector")] + public void SessionName_IsDarlingAndTheInstallIdsHexDigits_OrTheFallback(string? installId, string expected) + { + var name = AwsAssumedRoleCredentials.SessionNameFor(installId); + + Assert.Equal(expected, name); + Assert.Matches(new Regex(@"^[\w+=,.@-]{2,64}$"), name); + } + + /* ---- the region of the call that triggers a refresh --------------------------------------------------- */ + + [Fact] + public async Task ARefresh_CallsStsInTheRegionOfTheCallThatTriggeredIt() + { + var clock = new ManualTimeProvider(); + var sts = new FakeSts(); + var credentials = Make(sts, clock); + + await credentials.ForRegion(RegionEndpoint.GetBySystemName("eu-west-1")).GetCredentialsAsync(); + clock.Advance(TimeSpan.FromMinutes(50)); + await credentials.ForRegion(RegionEndpoint.GetBySystemName("ap-southeast-2")).GetCredentialsAsync(); + + Assert.Equal(new[] { "eu-west-1", "ap-southeast-2" }, sts.Regions); + } + + [Theory] + [InlineData("us-east-1", "https://sts.us-east-1.amazonaws.com")] + [InlineData("cn-north-1", "https://sts.cn-north-1.amazonaws.com.cn")] + [InlineData("us-gov-west-1", "https://sts.us-gov-west-1.amazonaws.com")] + public void TheStsClientConfig_PointsAtTheRegionalEndpointOfTheTargetsOwnPartition(string region, string expectedUrl) + { + var config = AwsAssumedRoleCredentials.CreateStsConfig(RegionEndpoint.GetBySystemName(region)); + + var endpoint = config.DetermineServiceOperationEndpoint(new ServiceOperationEndpointParameters(new AssumeRoleRequest())); + + Assert.StartsWith(expectedUrl, endpoint.URL, StringComparison.Ordinal); + Assert.Equal(AwsAssumedRoleCredentials.StsTimeout, config.Timeout); + Assert.Equal(AwsAssumedRoleCredentials.StsMaxErrorRetry, config.MaxErrorRetry); + } + + /* ---- failure ------------------------------------------------------------------------------------------- */ + + [Fact] + public async Task AFailedRefresh_IsRememberedForAMinute_AndEachThrowIsANewException() + { + var clock = new ManualTimeProvider(); + var sts = new FakeSts((_, _) => throw FakeSts.Error("AccessDenied", "User is not authorized to perform: sts:AssumeRole")); + var credentials = Make(sts, clock); + + var first = await Assert.ThrowsAsync(() => credentials.GetCredentialsAsync()); + var second = await Assert.ThrowsAsync(() => credentials.GetCredentialsAsync()); + var third = await Assert.ThrowsAsync(() => credentials.GetCredentialsAsync()); + + Assert.Equal(1, sts.Calls); + Assert.NotSame(first, second); + Assert.NotSame(second, third); + Assert.Equal(first.Message, second.Message); + Assert.Equal(AwsRoleAssumeKind.Denied, second.Kind); + Assert.Equal("AccessDenied", second.SourceErrorCode); + + clock.Advance(TimeSpan.FromSeconds(61)); + await Assert.ThrowsAsync(() => credentials.GetCredentialsAsync()); + Assert.Equal(2, sts.Calls); + } + + [Fact] + public async Task WhileAFailureIsRemembered_OldCredentialsWithMoreThanAMinuteLeftAreServed() + { + var clock = new ManualTimeProvider(); + var logger = new CapturingAwsLogger(); + var sts = new FakeSts(); + var credentials = Make(sts, clock, logger: logger); + await credentials.GetCredentialsAsync(); + + sts.Handler = (_, _) => throw FakeSts.Error("Throttling", "Rate exceeded"); + + /* Past the refresh point with ten minutes left: the refresh fails, the old credentials are served, and the failure is only logged. */ + clock.Advance(TimeSpan.FromMinutes(50)); + Assert.Equal("ASIAEXAMPLEASSUMED", (await credentials.GetCredentialsAsync()).AccessKey); + Assert.Equal(2, sts.Calls); + Assert.Contains("Warning", logger.All, StringComparison.Ordinal); + + /* Inside the minute of the remembered failure there is no new STS call. */ + clock.Advance(TimeSpan.FromSeconds(30)); + Assert.Equal("ASIAEXAMPLEASSUMED", (await credentials.GetCredentialsAsync()).AccessKey); + Assert.Equal(2, sts.Calls); + + /* The failure has expired and the credentials have under a minute left: a new attempt, which fails, and the failure is thrown. */ + clock.Advance(TimeSpan.FromMinutes(8) + TimeSpan.FromSeconds(45)); + var thrown = await Assert.ThrowsAsync(() => credentials.GetCredentialsAsync()); + Assert.Equal(AwsRoleAssumeKind.Transient, thrown.Kind); + Assert.Equal(3, sts.Calls); + + /* And the next throw, from the remembered failure, is a new exception with no new call. */ + clock.Advance(TimeSpan.FromSeconds(20)); + var again = await Assert.ThrowsAsync(() => credentials.GetCredentialsAsync()); + Assert.NotSame(thrown, again); + Assert.Equal(thrown.Message, again.Message); + Assert.Equal(3, sts.Calls); + } + + [Fact] + public async Task ARememberedFailure_BelongsToItsRegion() + { + var sts = new FakeSts((_, _) => throw FakeSts.Error("RegionDisabledException", "STS is not active")); + var credentials = Make(sts, new ManualTimeProvider()); + + await Assert.ThrowsAsync(() => credentials.ForRegion(RegionEndpoint.EUSouth1).GetCredentialsAsync()); + await Assert.ThrowsAsync(() => credentials.ForRegion(RegionEndpoint.EUSouth1).GetCredentialsAsync()); + Assert.Equal(1, sts.Calls); + + sts.Handler = (_, _) => Task.FromResult(FakeSts.Grant()); + var other = await credentials.ForRegion(RegionEndpoint.USEast1).GetCredentialsAsync(); + Assert.Equal("ASIAEXAMPLEASSUMED", other.AccessKey); + Assert.Equal(2, sts.Calls); + } + + [Theory] + [InlineData("AccessDenied", AwsRoleAssumeKind.Denied, true)] + [InlineData("RegionDisabledException", AwsRoleAssumeKind.RegionDisabled, true)] + [InlineData("ExpiredToken", AwsRoleAssumeKind.SourceCredentials, false)] + [InlineData("InvalidClientTokenId", AwsRoleAssumeKind.SourceCredentials, false)] + [InlineData("SignatureDoesNotMatch", AwsRoleAssumeKind.SourceCredentials, false)] + [InlineData("Throttling", AwsRoleAssumeKind.Transient, false)] + [InlineData("ServiceUnavailable", AwsRoleAssumeKind.Transient, false)] + public async Task EachStsErrorCode_BecomesItsKind(string code, AwsRoleAssumeKind kind, bool isConfiguration) + { + var sts = new FakeSts((_, _) => throw FakeSts.Error(code, "raw message from AWS")); + + var ex = await Assert.ThrowsAsync(() => Make(sts, new ManualTimeProvider(), Sentinel).GetCredentialsAsync()); + + Assert.Equal(kind, ex.Kind); + Assert.Equal(isConfiguration, ex.IsConfiguration); + Assert.Equal(Role, ex.RoleArn); + Assert.True(ex.ExternalIdSet); + Assert.Equal(code, ex.SourceErrorCode); + Assert.Contains(Role, ex.Message, StringComparison.Ordinal); + Assert.Null(ex.InnerException); + Assert.Contains("AmazonSecurityTokenServiceException", ex.SourceExceptionType, StringComparison.Ordinal); + } + + [Fact] + public async Task ARegionDisabledExceptionType_IsRegionDisabled_WhateverItsCode() + { + var sts = new FakeSts((_, _) => throw new RegionDisabledException("STS is off here")); + + var ex = await Assert.ThrowsAsync(() => Make(sts, new ManualTimeProvider()).GetCredentialsAsync()); + + Assert.Equal(AwsRoleAssumeKind.RegionDisabled, ex.Kind); + Assert.Contains("us-east-1", ex.Message, StringComparison.Ordinal); + } + + [Fact] + public async Task TheDeniedText_SaysWhetherAnExternalIdWasSet_AndNeverTheValue() + { + var sts = new FakeSts((_, _) => throw FakeSts.Error("AccessDenied", "denied")); + + var withId = await Assert.ThrowsAsync(() => Make(sts, new ManualTimeProvider(), Sentinel).GetCredentialsAsync()); + var withoutId = await Assert.ThrowsAsync(() => Make(sts, new ManualTimeProvider()).GetCredentialsAsync()); + + Assert.Contains("(with the external ID set on this server)", withId.Message, StringComparison.Ordinal); + Assert.Contains("(without an external ID)", withoutId.Message, StringComparison.Ordinal); + Assert.DoesNotContain(Sentinel, withId.Message, StringComparison.Ordinal); + } + + [Fact] + public async Task AHostWithNoCredentials_IsNoSourceIdentity_AndStsIsNeverCalled() + { + var sts = new FakeSts(); + var credentials = Make(sts, new ManualTimeProvider(), source: _ => throw new AmazonClientException("Unable to get IAM security credentials")); + + var ex = await Assert.ThrowsAsync(() => credentials.GetCredentialsAsync()); + + Assert.Equal(AwsRoleAssumeKind.NoSourceIdentity, ex.Kind); + Assert.False(ex.IsConfiguration); + Assert.Contains("The monitoring host has no AWS credentials to assume role " + Role + " with.", ex.Message, StringComparison.Ordinal); + Assert.Equal(0, sts.Calls); + } + + [Fact] + public async Task ANoSourceIdentityMessage_IsAFixedSentence_WithTheSdkTypeAndCodeOnly_AndTheSdkTextGoesToTheServiceLog() + { + var logger = new CapturingAwsLogger(); + var sts = new FakeSts(); + var credentials = Make(sts, new ManualTimeProvider(), logger: logger, source: _ => throw new AmazonServiceException( + @"The profile 'corp-sso' could not be found in C:\Users\svc\.aws\credentials; SSO cache C:\Users\svc\.aws\sso\cache failed", + null, System.Net.HttpStatusCode.Forbidden) + { ErrorCode = "ProfileNotFound" }); + + var ex = await Assert.ThrowsAsync(() => credentials.GetCredentialsAsync()); + + Assert.Equal(AwsRoleAssumeKind.NoSourceIdentity, ex.Kind); + Assert.Contains("(SDK exception Amazon.Runtime.AmazonServiceException, AWS error code ProfileNotFound)", ex.Message, StringComparison.Ordinal); + Assert.DoesNotContain("corp-sso", ex.Message, StringComparison.Ordinal); + Assert.DoesNotContain(".aws", ex.Message, StringComparison.Ordinal); + Assert.DoesNotContain("svc", ex.Message, StringComparison.Ordinal); + Assert.Contains("corp-sso", ex.SourceDetail, StringComparison.Ordinal); + Assert.Contains("corp-sso", logger.All, StringComparison.Ordinal); + } + + [Fact] + public async Task ATransientMessage_IsAFixedSentence_WithTheSdkTypeAndCodeOnly() + { + var sts = new FakeSts((_, _) => throw new AmazonServiceException( + @"Unable to reach http://169.254.170.2/v2/credentials/abc from C:\ProgramData\aws\config", + null, System.Net.HttpStatusCode.ServiceUnavailable) + { ErrorCode = "ServiceUnavailable" }); + var credentials = Make(sts, new ManualTimeProvider()); + + var ex = await Assert.ThrowsAsync(() => credentials.GetCredentialsAsync()); + + Assert.Equal(AwsRoleAssumeKind.Transient, ex.Kind); + Assert.Contains("(SDK exception Amazon.Runtime.AmazonServiceException, AWS error code ServiceUnavailable)", ex.Message, StringComparison.Ordinal); + Assert.DoesNotContain("169.254", ex.Message, StringComparison.Ordinal); + Assert.DoesNotContain("ProgramData", ex.Message, StringComparison.Ordinal); + Assert.Contains("169.254.170.2", ex.SourceDetail, StringComparison.Ordinal); + } + + [Fact] + public async Task ACredentialSourceThatNeverCompletes_FailsWithinTheDeadline_AndASecondCallerFailsWithinTheSameBound() + { + var sts = new FakeSts(); + var credentials = Make(sts, new ManualTimeProvider(), deadline: TimeSpan.FromMilliseconds(200), + source: _ => new TaskCompletionSource().Task); + + var first = Assert.ThrowsAsync(() => credentials.GetCredentialsAsync()); + var second = Assert.ThrowsAsync(() => credentials.GetCredentialsAsync()); + + var firstFailure = await first.WaitAsync(TimeSpan.FromSeconds(10)); + var secondFailure = await second.WaitAsync(TimeSpan.FromSeconds(10)); + + Assert.Equal(AwsRoleAssumeKind.Transient, firstFailure.Kind); + Assert.Contains("did not answer within", firstFailure.Message, StringComparison.Ordinal); + Assert.Equal(AwsRoleAssumeKind.Transient, secondFailure.Kind); + Assert.Equal(0, sts.Calls); + } + + [Fact] + public async Task ASlowSts_IsCutOffAtTheDeadline_AndIsATransientFailure() + { + var sts = new FakeSts(async (_, token) => + { + await Task.Delay(Timeout.Infinite, token); + return FakeSts.Grant(); + }); + var credentials = Make(sts, new ManualTimeProvider(), deadline: TimeSpan.FromMilliseconds(100)); + + var ex = await Assert.ThrowsAsync(() => credentials.GetCredentialsAsync()).WaitAsync(TimeSpan.FromSeconds(10)); + + Assert.Equal(AwsRoleAssumeKind.Transient, ex.Kind); + Assert.Contains("did not answer within", ex.Message, StringComparison.Ordinal); + } + + [Fact] + public async Task ACallerWhoCancels_GetsTheCancellation_NotARememberedFailure() + { + var sts = new FakeSts(async (_, token) => + { + await Task.Delay(Timeout.Infinite, token); + return FakeSts.Grant(); + }); + var credentials = Make(sts, new ManualTimeProvider()); + using var cts = new CancellationTokenSource(); + + var pending = credentials.GetCredentialsAsync(UsEast1, cts.Token); + await Task.Delay(50); + cts.Cancel(); + await Assert.ThrowsAnyAsync(() => pending); + + sts.Handler = (_, _) => Task.FromResult(FakeSts.Grant()); + Assert.Equal("ASIAEXAMPLEASSUMED", (await credentials.GetCredentialsAsync()).AccessKey); + } + + [Fact] + public async Task AnEmptyResponse_IsATransientFailure() + { + var sts = new FakeSts((_, _) => Task.FromResult(new AssumeRoleResponse())); + + var ex = await Assert.ThrowsAsync(() => Make(sts, new ManualTimeProvider()).GetCredentialsAsync()); + + Assert.Equal(AwsRoleAssumeKind.Transient, ex.Kind); + } + + /* ---- the SDK seam -------------------------------------------------------------------------------------- */ + + [Fact] + public async Task ARealRdsClientOverTheCredentials_SurfacesTheExceptionToWhoeverWalksTheChain() + { + var sts = new FakeSts((_, _) => throw FakeSts.Error("AccessDenied", "denied")); + var credentials = Make(sts, new ManualTimeProvider(), Sentinel); + + using var rds = new Amazon.RDS.AmazonRDSClient(credentials.ForRegion(UsEast1), UsEast1); + var thrown = await Record.ExceptionAsync(() => rds.DescribeDBInstancesAsync(new Amazon.RDS.Model.DescribeDBInstancesRequest())); + + Assert.NotNull(thrown); + var found = AwsRoleAssumeException.Find(thrown); + Assert.NotNull(found); + Assert.Equal(AwsRoleAssumeKind.Denied, found!.Kind); + Assert.Equal(1, sts.Calls); + } + + [Fact] + public void Find_WalksTheInnerChain_AndAnAggregate() + { + var inner = new AwsRoleCredentialCache(AwsRoleAllowlist.Empty); + var thrown = Assert.Throws(() => inner.For(new AwsRoleKey(Role, null), "us-east-1")); + + Assert.Same(thrown, AwsRoleAssumeException.Find(thrown)); + Assert.Same(thrown, AwsRoleAssumeException.Find(new InvalidOperationException("wrapped", new InvalidOperationException("twice", thrown)))); + Assert.Same(thrown, AwsRoleAssumeException.Find(new AggregateException(new InvalidOperationException("a"), new InvalidOperationException("b", thrown)))); + Assert.Null(AwsRoleAssumeException.Find(new InvalidOperationException("none"))); + Assert.Null(AwsRoleAssumeException.Find(null)); + } +} diff --git a/Darling/Darling.Tests/AwsExternalIdNeverPrintedTests.cs b/Darling/Darling.Tests/AwsExternalIdNeverPrintedTests.cs new file mode 100644 index 0000000000..9e28ceff10 --- /dev/null +++ b/Darling/Darling.Tests/AwsExternalIdNeverPrintedTests.cs @@ -0,0 +1,240 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Threading.Tasks; +using Amazon; +using Amazon.Runtime; +using PerformanceMonitor.Darling.Service.Targets; +using Xunit; + +namespace Darling.Tests; + +/// +/// The external ID of a per-server AWS role is a secret (#5452): it never appears in a message, an exception's text, a +/// key's text or a log line, whatever the SDK repeats back. Each test builds a fake STS or credential source whose +/// error text repeats a sentinel ID (as written, URL-encoded and in other case) and then looks for it everywhere the +/// code under test can put it. +/// +public sealed class AwsExternalIdNeverPrintedTests +{ + private const string Role = "arn:aws:iam::123456789012:role/darling-monitor"; + + /// Uses every character the external ID alphabet allows, so an encoder changes it. + private const string Sentinel = "sent+inel=id,7.4@x:y/z-9"; + + private static string Echo(string text) => + $"{text} [request externalId={Sentinel} encoded={Uri.EscapeDataString(Sentinel)} upper={Sentinel.ToUpperInvariant()}]"; + + private static void AssertClean(string? text) + { + Assert.NotNull(text); + Assert.DoesNotContain(Sentinel, text, StringComparison.OrdinalIgnoreCase); + Assert.DoesNotContain(Uri.EscapeDataString(Sentinel), text, StringComparison.OrdinalIgnoreCase); + Assert.DoesNotContain("sent+inel", text, StringComparison.OrdinalIgnoreCase); + } + + private static void AssertClean(AwsRoleAssumeException ex) + { + AssertClean(ex.Message); + AssertClean(ex.ToString()); + AssertClean(ex.SourceErrorCode ?? string.Empty); + AssertClean(ex.SourceExceptionType ?? string.Empty); + AssertClean(ex.SourceDetail ?? string.Empty); + AssertClean(ex.StackTrace ?? string.Empty); + Assert.Null(ex.InnerException); + Assert.Empty(ex.Data); + } + + private static AwsAssumedRoleCredentials Make( + FakeSts sts, ManualTimeProvider clock, CapturingAwsLogger? logger = null, Func>? source = null) => + new(new AwsRoleKey(Role, Sentinel), RegionEndpoint.USEast1, sts.Factory, source ?? FakeSts.Source, null, clock, null, logger); + + /* ---- the key ------------------------------------------------------------------------------------------- */ + + [Fact] + public void TheKeysText_NamesTheRole_AndSaysWhetherAnIdIsSet() + { + var withId = new AwsRoleKey(Role, Sentinel); + var withoutId = new AwsRoleKey(Role, null); + + Assert.Equal("AwsRoleKey { RoleArn = " + Role + ", ExternalId = set }", withId.ToString()); + Assert.Equal("AwsRoleKey { RoleArn = " + Role + ", ExternalId = none }", withoutId.ToString()); + AssertClean($"{withId}"); + AssertClean(string.Format(System.Globalization.CultureInfo.InvariantCulture, "{0}", withId)); + AssertClean(new[] { withId }.Select(k => k.ToString()).Single()); + } + + [Fact] + public void TheKeyStillComparesByTheValueOfTheId() + { + Assert.Equal(new AwsRoleKey(Role, Sentinel), new AwsRoleKey(Role, Sentinel)); + Assert.NotEqual(new AwsRoleKey(Role, Sentinel), new AwsRoleKey(Role, Sentinel.ToUpperInvariant())); + Assert.NotEqual(new AwsRoleKey(Role, Sentinel), new AwsRoleKey(Role, null)); + Assert.Equal(Sentinel, new AwsRoleKey(Role, Sentinel).ExternalId); + } + + /* ---- every failure path -------------------------------------------------------------------------------- */ + + [Theory] + [InlineData("AccessDenied")] + [InlineData("RegionDisabledException")] + [InlineData("ExpiredToken")] + [InlineData("InvalidClientTokenId")] + [InlineData("SignatureDoesNotMatch")] + [InlineData("ValidationError")] + [InlineData("Throttling")] + public async Task AnStsErrorThatRepeatsTheId_LeavesItOutOfEverythingTheCodePrints(string code) + { + var logger = new CapturingAwsLogger(); + var sts = new FakeSts((_, _) => throw FakeSts.Error(code, Echo("Request failed for the call"))); + var credentials = Make(sts, new ManualTimeProvider(), logger); + + var first = await Assert.ThrowsAsync(() => credentials.GetCredentialsAsync()); + var cached = await Assert.ThrowsAsync(() => credentials.GetCredentialsAsync()); + + AssertClean(first); + AssertClean(cached); + AssertClean(logger.All); + Assert.True(first.ExternalIdSet); + Assert.Equal(1, sts.Calls); + } + + [Fact] + public async Task AnErrorCodeThatRepeatsTheId_IsScrubbedToo() + { + var sts = new FakeSts((_, _) => throw FakeSts.Error(Echo("InvalidClientTokenId"), "denied")); + + var ex = await Assert.ThrowsAsync(() => Make(sts, new ManualTimeProvider()).GetCredentialsAsync()); + + Assert.Equal(AwsRoleAssumeKind.SourceCredentials, ex.Kind); + AssertClean(ex); + } + + [Fact] + public async Task ASourceFailureThatRepeatsTheId_LeavesItOutOfTheMessage() + { + var logger = new CapturingAwsLogger(); + var sts = new FakeSts(); + var credentials = Make(sts, new ManualTimeProvider(), logger, + source: _ => throw new AmazonClientException(Echo("Unable to get credentials"))); + + var ex = await Assert.ThrowsAsync(() => credentials.GetCredentialsAsync()); + + Assert.Equal(AwsRoleAssumeKind.NoSourceIdentity, ex.Kind); + AssertClean(ex); + AssertClean(logger.All); + } + + [Fact] + public async Task ANonServiceExceptionThatRepeatsTheId_IsScrubbed() + { + var sts = new FakeSts((_, _) => throw new InvalidOperationException(Echo("socket closed"))); + + var ex = await Assert.ThrowsAsync(() => Make(sts, new ManualTimeProvider()).GetCredentialsAsync()); + + Assert.Equal(AwsRoleAssumeKind.Transient, ex.Kind); + AssertClean(ex); + Assert.Contains("(SDK exception System.InvalidOperationException)", ex.Message, StringComparison.Ordinal); + Assert.DoesNotContain("socket closed", ex.Message, StringComparison.Ordinal); + Assert.Contains("socket closed", ex.SourceDetail, StringComparison.Ordinal); + } + + [Fact] + public async Task TheCredentialsHandedToAClient_CarryNoIdInTheirText() + { + var sts = new FakeSts(); + var credentials = Make(sts, new ManualTimeProvider()); + + var granted = await credentials.GetCredentialsAsync(); + + AssertClean(credentials.ToString()); + AssertClean(credentials.ForRegion(RegionEndpoint.USEast1).ToString()); + Assert.Equal(Sentinel, Assert.Single(sts.Requests).ExternalId); + Assert.NotNull(granted); + } + + [Fact] + public async Task ThroughARealRdsClient_TheExceptionThatReachesTheCaller_IsClean() + { + var sts = new FakeSts((_, _) => throw FakeSts.Error("AccessDenied", Echo("User is not authorized"))); + var credentials = Make(sts, new ManualTimeProvider()); + + using var rds = new Amazon.RDS.AmazonRDSClient(credentials.ForRegion(RegionEndpoint.USEast1), RegionEndpoint.USEast1); + var thrown = await Record.ExceptionAsync(() => rds.DescribeDBInstancesAsync(new Amazon.RDS.Model.DescribeDBInstancesRequest())); + + Assert.NotNull(thrown); + AssertClean(thrown!.ToString()); + AssertClean(thrown.Message); + var found = AwsRoleAssumeException.Find(thrown); + Assert.NotNull(found); + AssertClean(found!); + } + + [Fact] + public void TheTwoRefusalsThatNeverReachStsAreClean() + { + var cache = new AwsRoleCredentialCache(AwsRoleAllowlist.Empty); + var notAllowed = Assert.Throws(() => cache.For(new AwsRoleKey(Role, Sentinel), "us-east-1")); + AssertClean(notAllowed); + + var allowed = new AwsRoleCredentialCache(AwsRoleAllowlist.From(new[] { "123456789012" })); + var mismatch = Assert.Throws(() => allowed.For(new AwsRoleKey(Role, Sentinel), "cn-north-1")); + AssertClean(mismatch); + Assert.True(notAllowed.ExternalIdSet); + Assert.True(mismatch.ExternalIdSet); + } + + [Fact] + public void ScrubRemovesEveryFormOfTheId_AndLeavesTextWithoutOneAlone() + { + Assert.Equal("none here", AwsRoleAssumeException.Scrub("none here", Sentinel)); + Assert.Equal("anything", AwsRoleAssumeException.Scrub("anything", null)); + Assert.Equal("anything", AwsRoleAssumeException.Scrub("anything", string.Empty)); + AssertClean(AwsRoleAssumeException.Scrub(Echo("text"), Sentinel)); + AssertClean(AwsRoleAssumeException.Scrub("value " + System.Net.WebUtility.HtmlEncode(Sentinel) + " end", Sentinel)); + } + + /* ---- the worker's own outputs --------------------------------------------------------------------------- */ + + private const string InstanceHost = "solo.abc123.us-east-1.rds.amazonaws.com"; + + [Theory] + [InlineData("AccessDenied")] + [InlineData("RegionDisabledException")] + [InlineData("ExpiredToken")] + [InlineData("Throttling")] + public async Task TheWorkersCollectionLogTextAndWarningLine_NeverCarryTheId(string code) + { + // An STS error that repeats the external ID in its message, driven through the real RDS reader and a real client. + var sts = new FakeSts((_, _) => throw FakeSts.Error(code, Echo("denied for role " + Role))); + var cacheLog = new CapturingAwsLogger(); + var cache = new AwsRoleCredentialCache(AwsRoleAllowlist.From(new[] { Role }), null, sts.Factory, FakeSts.Source, new ManualTimeProvider(), null, cacheLog); + var ingestor = new RdsCpuIngestor( + Npgsql.NpgsqlDataSource.Create("Host=localhost;Port=1;Database=unused"), + verifier: RdsEndpointVerifier.ForTests(enforce: true, loginProbe: (_, _) => Task.FromResult(null)), + roles: cache); + + var thrown = await Record.ExceptionAsync( + () => ingestor.IngestAsync(1, "s", InstanceHost, "Host=" + InstanceHost, new AwsRoleKey(Role, Sentinel), System.Threading.CancellationToken.None)); + + var fault = Assert.IsType(thrown); + AssertClean(fault); + + // What the worker writes for a refused role: the collection_log text and the warning line. + var workerLog = new CapturingAwsLogger(); + var rowText = PerformanceMonitor.Darling.Service.DarlingWorker.AwsRoleFaultNote(workerLog, "alpha-pg-01", "pg_plan_capture", fault); + AssertClean(rowText); + AssertClean(workerLog.All); + + // The ERROR arm writes the exception's message too; the credentials class logged one warning for the failed refresh. + AssertClean(fault.Message); + AssertClean(cacheLog.All); + } +} diff --git a/Darling/Darling.Tests/AwsPerServerRoleLiveTests.cs b/Darling/Darling.Tests/AwsPerServerRoleLiveTests.cs new file mode 100644 index 0000000000..611053097b --- /dev/null +++ b/Darling/Darling.Tests/AwsPerServerRoleLiveTests.cs @@ -0,0 +1,569 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Globalization; +using System.Linq; +using System.Security.Cryptography; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: each fact mints its own scratch database through ScratchPostgres and never touches another one. + The login role a fact needs is created for it and dropped in its cleanup. */ + +/// +/// The per-server AWS role (#5452) against a store: the rung on a store that stopped below it and on a fresh one, the +/// table's check, the column ACL, the 17-argument edit function and its 15-argument wrapper, and the trigger rule that +/// holds every role but the owner. +/// +public sealed class AwsPerServerRoleLiveTests +{ + private const string Role = "arn:aws:iam::123456789012:role/darling-monitor"; + private const string OtherRole = "arn:aws:iam::123456789012:role/darling-other"; + private const string External = "tenant-1234"; + private const string OtherExternal = "tenant-5678"; + + private static readonly string RolePassword = Convert.ToHexString(RandomNumberGenerator.GetBytes(16)); + + private static string? BaseConnectionString() => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task TextAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + var value = await command.ExecuteScalarAsync(ct); + return value is null or DBNull ? null : Convert.ToString(value, CultureInfo.InvariantCulture); + } + + private static string Lit(string? value) => + value is null ? "NULL" : "'" + value.Replace("'", "''", StringComparison.Ordinal) + "'"; + + private static async Task OpenOwnerAsync(ScratchPostgres scratch, CancellationToken ct) + { + var connection = new NpgsqlConnection( + new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { SearchPath = PgSchemaGenerator.SearchPath }.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + return connection; + } + + private static Task InsertServerAsync(NpgsqlConnection connection, int id, string? role, string? externalId, CancellationToken ct) => + ExecAsync(connection, + "INSERT INTO config.config_monitored_servers (server_id, name, host, database, auth, username, engine, port, aws_role_arn, aws_external_id) " + + $"VALUES ({id}, 'srv-{id}', 'host-{id}', 'postgres', 'sql', 'monitor', 'postgres', 5432, {Lit(role)}, {Lit(externalId)})", ct); + + private static Task StoredAsync(NpgsqlConnection connection, int id, CancellationToken ct) => + TextAsync(connection, + "SELECT COALESCE(aws_role_arn, '') || ' | ' || COALESCE(aws_external_id, '') || ' | ' || aws_external_id_set::text " + + $"FROM config.config_monitored_servers WHERE server_id = {id}", ct); + + private static async Task RefusalAsync(Func action) + { + try + { + await action(); + return null; + } + catch (PostgresException ex) + { + return ex.SqlState; + } + } + + private static async Task TokenAsync(NpgsqlConnection connection, int id, CancellationToken ct) => + (await TextAsync(connection, + $"SELECT to_char(modified_at, 'YYYY-MM-DD HH24:MI:SS.US') FROM config.config_monitored_servers WHERE server_id = {id}", ct))!; + + /// One call of the 17-argument function (the AWS pair last), returning its outcome text. + private static async Task EditAsync( + NpgsqlConnection connection, int id, string columns, string? role, string? externalId, CancellationToken ct) + { + var token = await TokenAsync(connection, id, ct); + return await TextAsync(connection, + $"SELECT outcome FROM config.edit_monitored_server({id}, '{token}'::timestamp, ARRAY[{columns}]::text[], " + + $"NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, {Lit(role)}, {Lit(externalId)})", ct); + } + + /// One call of the 15-argument wrapper: no AWS arguments, but the caller may still NAME the AWS columns. + private static async Task LegacyEditAsync( + NpgsqlConnection connection, int id, string columns, string? name, CancellationToken ct) + { + var token = await TokenAsync(connection, id, ct); + return await TextAsync(connection, + $"SELECT outcome FROM config.edit_monitored_server({id}, '{token}'::timestamp, ARRAY[{columns}]::text[], " + + $"{Lit(name)}, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)", ct); + } + + private static async Task OpenAsRoleAsync(ScratchPostgres scratch, string role, CancellationToken ct) + { + var connection = new NpgsqlConnection(new NpgsqlConnectionStringBuilder(scratch.ConnectionString) + { + Username = role, + Password = RolePassword, + SearchPath = PgSchemaGenerator.SearchPath, + }.ConnectionString); + await connection.OpenAsync(ct); + return connection; + } + + private static async Task CreateLoginRoleAsync(NpgsqlConnection owner, CancellationToken ct) + { + var role = "aws_role_" + Guid.NewGuid().ToString("N")[..8]; + await ExecAsync(owner, $"CREATE ROLE {role} LOGIN NOSUPERUSER PASSWORD '{RolePassword}'", ct); + await ExecAsync(owner, $"GRANT USAGE ON SCHEMA config TO {role}", ct); + return role; + } + + private static Task DropLoginRoleAsync(NpgsqlConnection cleanup, string? role) => + role is null + ? Task.CompletedTask + : ExecAsync(cleanup, $"DROP OWNED BY {role}; DROP ROLE IF EXISTS {role}", CancellationToken.None); + + /* ---- the rung -------------------------------------------------------------------------------------- */ + + [Fact] + public async Task TheRung_AppliesToAStoreThatStoppedBelowIt_AndToAFreshOne_AndARerunIsANoOp() + { + var baseConnectionString = BaseConnectionString(); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to run the live AWS role tests (each mints its own scratch database)."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var bodySucceeded = false; + try + { + await using var owner = await OpenOwnerAsync(scratch, ct); + + /* A fresh store took every rung in order, this one included. */ + Assert.Equal("3", await TextAsync(owner, + "SELECT count(*) FROM information_schema.columns WHERE table_schema = 'config' AND table_name = 'config_monitored_servers' " + + "AND column_name IN ('aws_role_arn', 'aws_external_id', 'aws_external_id_set')", ct)); + /* The store's top rung, not this rung's: V170 (#5495) landed above it. */ + Assert.Equal(StorageVersion.SchemaVersion.ToString(CultureInfo.InvariantCulture), + await TextAsync(owner, "SELECT max(version) FROM darling_schema_version", ct)); + Assert.Equal(1L.ToString(CultureInfo.InvariantCulture), await TextAsync(owner, + "SELECT count(*) FROM pg_constraint WHERE conname = 'config_monitored_servers_aws_role_check'", ct)); + Assert.True(await ProbeSentinelAsync(owner, ct), "the viewer's probe must see the role column on a migrated store"); + + /* A V168 store: the columns gone, the stamp rolled back, a server already registered. */ + await InsertServerAsync(owner, 8101, null, null, ct); + await ExecAsync(owner, "ALTER TABLE config.config_monitored_servers DROP COLUMN aws_external_id_set, DROP COLUMN aws_external_id, DROP COLUMN aws_role_arn", ct); + await ExecAsync(owner, "DELETE FROM darling_schema_version WHERE version >= 169", ct); + Assert.False(await ProbeSentinelAsync(owner, ct), "the probe must not see the role column once it is gone"); + + await PgMigrations.MigrateAsync(owner, ct); + + Assert.True(await ProbeSentinelAsync(owner, ct)); + Assert.Equal(" | | false", await StoredAsync(owner, 8101, ct)); + Assert.Equal(StorageVersion.SchemaVersion.ToString(CultureInfo.InvariantCulture), await TextAsync(owner, "SELECT max(version) FROM darling_schema_version", ct)); + + /* Running the rung's text again changes nothing and raises nothing. */ + var rung = PgMigrations.Scripts.Single(s => s.Version == AwsPerServerRoleRungTests.RungVersion).Sql; + await ExecAsync(owner, rung, ct); + await ExecAsync(owner, rung, ct); + Assert.Equal(" | | false", await StoredAsync(owner, 8101, ct)); + Assert.Equal("1", await TextAsync(owner, + "SELECT count(*) FROM pg_constraint WHERE conname = 'config_monitored_servers_aws_role_check'", ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, (_, _) => Task.CompletedTask); + } + } + + private static async Task ProbeSentinelAsync(NpgsqlConnection connection, CancellationToken ct) + { + /* This rung's own sentinel by ordinal: a newer rung's sentinel (V170's) is now the last one, and it stays true here. */ + await using var command = new NpgsqlCommand(ViewerDataService.StoreSchemaProbeSql, connection); + await using var reader = await command.ExecuteReaderAsync(ct); + Assert.True(await reader.ReadAsync(ct)); + return reader.GetBoolean(AwsPerServerRoleRungTests.ProbeOrdinal); + } + + /* ---- the check ------------------------------------------------------------------------------------- */ + + [Fact] + public async Task TheCheck_RefusesAnIdWithNoRole_ABadArn_AndASpace_AndAcceptsTheCorpus() + { + var baseConnectionString = BaseConnectionString(); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to run the live AWS role tests (each mints its own scratch database)."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var bodySucceeded = false; + try + { + await using var owner = await OpenOwnerAsync(scratch, ct); + var id = 8200; + + /* Refused with 23514 (check_violation): an ID with no role, a role that is not an IAM role ARN, a role with a + space, an ID with a space, an ID that is too short. */ + foreach (var (role, externalId) in new (string?, string?)[] + { + (null, External), + ("not-an-arn", null), + ("arn:aws:iam::123456789012:user/darling", null), + ("arn:aws:iam::12345:role/darling", null), + ("arn:aws:iam::123456789012:role/has space", null), + ("arn:aws:iam::123456789012:role/", null), + (Role, "has space"), + (Role, "x"), + (Role, new string('x', 1225)), + }) + { + id++; + var state = await RefusalAsync(() => InsertServerAsync(owner, id, role, externalId, ct)); + Assert.Equal("23514", state); + } + + Assert.Equal("0", await TextAsync(owner, "SELECT count(*) FROM config.config_monitored_servers", ct)); + + /* Accepted: no role, a role, a role with an ID, the other partitions, a path, and the longest ID. */ + foreach (var (role, externalId) in new (string?, string?)[] + { + (null, null), + (Role, null), + (Role, External), + ("arn:aws-cn:iam::123456789012:role/darling", null), + ("arn:aws-us-gov:iam::123456789012:role/darling", "ab"), + ("arn:aws:iam::123456789012:role/monitoring/team/darling-monitor", null), + ("arn:aws:iam::123456789012:role/name_with+=,.@-chars", "A-b_C+d=e,f.g@h:i/j-k"), + (Role, new string('x', 1224)), + }) + { + id++; + await InsertServerAsync(owner, id, role, externalId, ct); + } + + Assert.Equal("8", await TextAsync(owner, "SELECT count(*) FROM config.config_monitored_servers", ct)); + + /* The generated flag follows the ID, and an update that would leave an ID with no role is refused too. */ + Assert.Equal("false", await TextAsync(owner, "SELECT aws_external_id_set::text FROM config.config_monitored_servers WHERE server_id = 8211", ct)); + Assert.Equal("true", await TextAsync(owner, "SELECT aws_external_id_set::text FROM config.config_monitored_servers WHERE server_id = 8212", ct)); + Assert.Equal("23514", await RefusalAsync(() => + ExecAsync(owner, "UPDATE config.config_monitored_servers SET aws_role_arn = NULL WHERE server_id = 8212", ct))); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, (_, _) => Task.CompletedTask); + } + } + + /* ---- the column ACL -------------------------------------------------------------------------------- */ + + [Fact] + public async Task AReadOnlyRole_ReadsTheRoleAndTheFlag_ButGets42501OnTheExternalId() + { + var baseConnectionString = BaseConnectionString(); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to run the live AWS role tests (each mints its own scratch database)."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + string? login = null; + var bodySucceeded = false; + try + { + await using var owner = await OpenOwnerAsync(scratch, ct); + await InsertServerAsync(owner, 8301, Role, External, ct); + + /* The grant is the one the managed batch and the script give viewer and mcp: the non-secret list, from the ACL. */ + login = await CreateLoginRoleAsync(owner, ct); + var table = DarlingManagedRoles.ViewerRestrictedConfigTables.Single(t => t.Table == "config_monitored_servers"); + await ExecAsync(owner, $"GRANT SELECT ({string.Join(", ", table.NonSecretColumns)}) ON config.config_monitored_servers TO {login}", ct); + + await using var reader = await OpenAsRoleAsync(scratch, login, ct); + Assert.Equal($"{Role} | true", + await TextAsync(reader, "SELECT aws_role_arn || ' | ' || aws_external_id_set::text FROM config.config_monitored_servers WHERE server_id = 8301", ct)); + + Assert.Equal("42501", await RefusalAsync(() => + TextAsync(reader, "SELECT aws_external_id FROM config.config_monitored_servers WHERE server_id = 8301", ct))); + Assert.Equal("42501", await RefusalAsync(() => + TextAsync(reader, "SELECT * FROM config.config_monitored_servers WHERE server_id = 8301", ct))); + + bodySucceeded = true; + } + finally + { + var loginToDrop = login; + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, (cleanup, _) => DropLoginRoleAsync(cleanup, loginToDrop)); + } + } + + /* ---- the edit function and its wrapper -------------------------------------------------------------- */ + + [Fact] + public async Task TheEditFunction_SetsKeepsChangesAndClearsTheRole_AndAnswersTheTwoNewOutcomes() + { + var baseConnectionString = BaseConnectionString(); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to run the live AWS role tests (each mints its own scratch database)."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var bodySucceeded = false; + try + { + await using var owner = await OpenOwnerAsync(scratch, ct); + await ExecAsync(owner, DarlingManagedRoles.BuildEditMonitoredServerFunctionSql("config"), ct); + await ExecAsync(owner, DarlingManagedRoles.BuildEditMonitoredServerLegacyWrapperSql("config"), ct); + await InsertServerAsync(owner, 8401, null, null, ct); + + /* Setting a role with an ID. */ + Assert.Equal("saved", await EditAsync(owner, 8401, "'aws_role_arn', 'aws_external_id'", Role, External, ct)); + Assert.Equal($"{Role} | {External} | true", await StoredAsync(owner, 8401, ct)); + + /* Left out of p_columns, the role and the ID are kept, whatever values ride along unnamed. */ + Assert.Equal("saved", await EditAsync(owner, 8401, string.Empty, OtherRole, OtherExternal, ct)); + Assert.Equal($"{Role} | {External} | true", await StoredAsync(owner, 8401, ct)); + + /* A new role over a stored ID, with no ID named, would carry the old ID along unseen: refused, nothing written. */ + Assert.Equal("external_id_needed", await EditAsync(owner, 8401, "'aws_role_arn'", OtherRole, null, ct)); + Assert.Equal($"{Role} | {External} | true", await StoredAsync(owner, 8401, ct)); + + /* The same role again is not a change, so it needs no ID. */ + Assert.Equal("saved", await EditAsync(owner, 8401, "'aws_role_arn'", Role, null, ct)); + Assert.Equal($"{Role} | {External} | true", await StoredAsync(owner, 8401, ct)); + + /* A new role with the ID sent again, and with the ID cleared (named, NULL). */ + Assert.Equal("saved", await EditAsync(owner, 8401, "'aws_role_arn', 'aws_external_id'", OtherRole, OtherExternal, ct)); + Assert.Equal($"{OtherRole} | {OtherExternal} | true", await StoredAsync(owner, 8401, ct)); + Assert.Equal("saved", await EditAsync(owner, 8401, "'aws_role_arn', 'aws_external_id'", Role, null, ct)); + Assert.Equal($"{Role} | | false", await StoredAsync(owner, 8401, ct)); + + /* The ID alone, for the role that is stored. */ + Assert.Equal("saved", await EditAsync(owner, 8401, "'aws_external_id'", null, External, ct)); + Assert.Equal($"{Role} | {External} | true", await StoredAsync(owner, 8401, ct)); + + /* An ID with a cleared role: refused (the role is named NULL and an ID rides along). */ + Assert.Equal("external_id_needs_role", await EditAsync(owner, 8401, "'aws_role_arn', 'aws_external_id'", null, External, ct)); + Assert.Equal($"{Role} | {External} | true", await StoredAsync(owner, 8401, ct)); + + /* Clearing is explicit: the role named with NULL, which takes the ID with it. */ + Assert.Equal("saved", await EditAsync(owner, 8401, "'aws_role_arn'", null, null, ct)); + Assert.Equal(" | | false", await StoredAsync(owner, 8401, ct)); + + /* An ID named for a server that has no role: refused, and a plain edit of another column is unaffected. */ + Assert.Equal("external_id_needs_role", await EditAsync(owner, 8401, "'aws_external_id'", null, External, ct)); + Assert.Equal("saved", await EditAsync(owner, 8401, "'aws_external_id'", null, null, ct)); + Assert.Equal(" | | false", await StoredAsync(owner, 8401, ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task TheFifteenArgumentWrapper_KeepsTheRole_AndACallerNamingTheAwsColumnsCannotChangeOrClearIt() + { + var baseConnectionString = BaseConnectionString(); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to run the live AWS role tests (each mints its own scratch database)."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var bodySucceeded = false; + try + { + await using var owner = await OpenOwnerAsync(scratch, ct); + await ExecAsync(owner, DarlingManagedRoles.BuildEditMonitoredServerFunctionSql("config"), ct); + await ExecAsync(owner, DarlingManagedRoles.BuildEditMonitoredServerLegacyWrapperSql("config"), ct); + await InsertServerAsync(owner, 8501, Role, External, ct); + + /* An old-shape edit of another column leaves the role and the ID alone. */ + Assert.Equal("saved", await LegacyEditAsync(owner, 8501, "'name'", "renamed", ct)); + Assert.Equal("renamed", await TextAsync(owner, "SELECT name FROM config.config_monitored_servers WHERE server_id = 8501", ct)); + Assert.Equal($"{Role} | {External} | true", await StoredAsync(owner, 8501, ct)); + + /* Naming the AWS columns in p_columns does nothing: there are no arguments to carry a value, and the names are + stripped, so the wrapper cannot clear the role or the ID. The rest of the edit still applies. */ + Assert.Equal("saved", await LegacyEditAsync(owner, 8501, "'aws_role_arn', 'aws_external_id', 'name'", "renamed-again", ct)); + Assert.Equal("renamed-again", await TextAsync(owner, "SELECT name FROM config.config_monitored_servers WHERE server_id = 8501", ct)); + Assert.Equal($"{Role} | {External} | true", await StoredAsync(owner, 8501, ct)); + + Assert.Equal("saved", await LegacyEditAsync(owner, 8501, "'aws_role_arn'", null, ct)); + Assert.Equal($"{Role} | {External} | true", await StoredAsync(owner, 8501, ct)); + Assert.Equal("saved", await LegacyEditAsync(owner, 8501, "'aws_external_id'", null, ct)); + Assert.Equal($"{Role} | {External} | true", await StoredAsync(owner, 8501, ct)); + + /* A NULL column list is an empty one. */ + var token = await TokenAsync(owner, 8501, ct); + Assert.Equal("saved", await TextAsync(owner, + $"SELECT outcome FROM config.edit_monitored_server(8501, '{token}'::timestamp, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)", ct)); + Assert.Equal($"{Role} | {External} | true", await StoredAsync(owner, 8501, ct)); + + /* The two functions are two signatures of one name, and the wrapper is the definer-pinned one. */ + Assert.Equal("2", await TextAsync(owner, + "SELECT count(*) FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace WHERE n.nspname = 'config' AND p.proname = 'edit_monitored_server'", ct)); + Assert.Equal("True", await TextAsync(owner, + "SELECT bool_and(prosecdef AND proconfig @> ARRAY['search_path=pg_catalog, pg_temp']) FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace " + + "WHERE n.nspname = 'config' AND p.proname = 'edit_monitored_server'", ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, (_, _) => Task.CompletedTask); + } + } + + /* ---- the trigger rule ------------------------------------------------------------------------------ */ + + [Fact] + public async Task ALoginThatOnlyExecutes_GetsInvalidValueForAnEmptyRequiredField_OnBothSignatures_AndTheRowIsUnchanged() + { + var baseConnectionString = BaseConnectionString(); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to run the live AWS role tests (each mints its own scratch database)."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + string? login = null; + var bodySucceeded = false; + try + { + await using var owner = await OpenOwnerAsync(scratch, ct); + await ExecAsync(owner, DarlingManagedRoles.BuildEditMonitoredServerFunctionSql("config"), ct); + await ExecAsync(owner, DarlingManagedRoles.BuildEditMonitoredServerLegacyWrapperSql("config"), ct); + await InsertServerAsync(owner, 8701, Role, External, ct); + + /* The edit-only login the web host uses: EXECUTE on both signatures, and no right on the table itself. */ + login = await CreateLoginRoleAsync(owner, ct); + await ExecAsync(owner, $"GRANT EXECUTE ON FUNCTION config.edit_monitored_server({DarlingManagedRoles.EditMonitoredServerSignature}) TO {login}", ct); + await ExecAsync(owner, $"GRANT EXECUTE ON FUNCTION config.edit_monitored_server({DarlingManagedRoles.EditMonitoredServerLegacySignature}) TO {login}", ct); + await using var role = await OpenAsRoleAsync(scratch, login, ct); + + var before = await StoredAsync(owner, 8701, ct); + var nameBefore = await TextAsync(owner, "SELECT name || '|' || host || '|' || modified_at::text FROM config.config_monitored_servers WHERE server_id = 8701", ct); + + /* An empty name: the 17-argument function and the 15-argument wrapper answer with an outcome, and raise nothing. */ + var token = await TokenAsync(owner, 8701, ct); + Assert.Equal("invalid_value", await TextAsync(role, + $"SELECT outcome FROM config.edit_monitored_server(8701, '{token}'::timestamp, ARRAY['name']::text[], " + + "NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)", ct)); + Assert.Equal("invalid_value", await TextAsync(role, + $"SELECT outcome FROM config.edit_monitored_server(8701, '{token}'::timestamp, ARRAY['name']::text[], " + + "NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)", ct)); + + /* An empty host with a typed secret, on both. */ + Assert.Equal("invalid_value", await TextAsync(role, + $"SELECT outcome FROM config.edit_monitored_server(8701, '{token}'::timestamp, ARRAY['host','encrypted_password']::text[], " + + "NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'typed-secret', NULL, NULL, NULL, NULL, NULL, NULL)", ct)); + Assert.Equal("invalid_value", await TextAsync(role, + $"SELECT outcome FROM config.edit_monitored_server(8701, '{token}'::timestamp, ARRAY['host','encrypted_password']::text[], " + + "NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'typed-secret', NULL, NULL, NULL, NULL)", ct)); + + Assert.Equal(before, await StoredAsync(owner, 8701, ct)); + Assert.Equal(nameBefore, await TextAsync(owner, "SELECT name || '|' || host || '|' || modified_at::text FROM config.config_monitored_servers WHERE server_id = 8701", ct)); + bodySucceeded = true; + } + finally + { + var loginToDrop = login; + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, (cleanup, _) => DropLoginRoleAsync(cleanup, loginToDrop)); + } + } + + [Fact] + public async Task TheCheck_RefusesMarkupAndQuoteCharactersInARolePath() + { + var baseConnectionString = BaseConnectionString(); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to run the live AWS role tests (each mints its own scratch database)."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var bodySucceeded = false; + try + { + await using var owner = await OpenOwnerAsync(scratch, ct); + var id = 8710; + foreach (var path in new[] { "/x", "a/x", "a>b/x", "a\"b/x", "a'b/x", "a&b/x", "a`b/x", "a b/x" }) + { + var refused = await RefusalAsync(() => InsertServerAsync(owner, id++, $"arn:aws:iam::123456789012:role/{path}", null, ct)); + Assert.Equal("23514", refused); + } + + await InsertServerAsync(owner, id, "arn:aws:iam::123456789012:role/path_1+=,.@-/darling-monitor", null, ct); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task ARoleThatIsNotTheOwner_CannotPutANewRoleOverAStoredId_Pw004() + { + var baseConnectionString = BaseConnectionString(); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to run the live AWS role tests (each mints its own scratch database)."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + string? login = null; + var bodySucceeded = false; + try + { + await using var owner = await OpenOwnerAsync(scratch, ct); + await ExecAsync(owner, DarlingManagedRoles.BuildServerPasswordRulesSql("config"), ct); + await InsertServerAsync(owner, 8601, Role, External, ct); + await InsertServerAsync(owner, 8602, Role, null, ct); + + login = await CreateLoginRoleAsync(owner, ct); + await ExecAsync(owner, $"GRANT SELECT, UPDATE ON config.config_monitored_servers TO {login}", ct); + + /* An update here bumps the reload beacon, which a role that edits servers is granted (the managed batch grants it). */ + await ExecAsync(owner, "INSERT INTO config_service (id) VALUES (1) ON CONFLICT DO NOTHING", ct); + await ExecAsync(owner, $"GRANT SELECT, UPDATE ON config.config_service TO {login}", ct); + await using var role = await OpenAsRoleAsync(scratch, login, ct); + + /* A new role with the stored ID untouched: refused with PW004 and the sentence. */ + var refused = await Assert.ThrowsAsync(() => + ExecAsync(role, $"UPDATE config.config_monitored_servers SET aws_role_arn = {Lit(OtherRole)} WHERE server_id = 8601", ct)); + Assert.Equal("PW004", refused.SqlState); + Assert.Equal(AwsRoleSettings.RoleChangeNeedsExternalIdMessage, refused.MessageText); + Assert.Equal($"{Role} | {External} | true", await StoredAsync(owner, 8601, ct)); + + /* The ID sent with it, or cleared with it, or the same role again, or a server with no ID: all pass. */ + await ExecAsync(role, $"UPDATE config.config_monitored_servers SET aws_role_arn = {Lit(OtherRole)}, aws_external_id = {Lit(OtherExternal)} WHERE server_id = 8601", ct); + Assert.Equal($"{OtherRole} | {OtherExternal} | true", await StoredAsync(owner, 8601, ct)); + await ExecAsync(role, $"UPDATE config.config_monitored_servers SET aws_role_arn = {Lit(Role)}, aws_external_id = NULL WHERE server_id = 8601", ct); + Assert.Equal($"{Role} | | false", await StoredAsync(owner, 8601, ct)); + await ExecAsync(role, $"UPDATE config.config_monitored_servers SET aws_role_arn = {Lit(OtherRole)} WHERE server_id = 8602", ct); + Assert.Equal($"{OtherRole} | | false", await StoredAsync(owner, 8602, ct)); + await ExecAsync(role, "UPDATE config.config_monitored_servers SET name = 'renamed' WHERE server_id = 8602", ct); + + /* The owner is not held to it (the edit function answers external_id_needed first). */ + await InsertServerAsync(owner, 8603, Role, External, ct); + await ExecAsync(owner, $"UPDATE config.config_monitored_servers SET aws_role_arn = {Lit(OtherRole)} WHERE server_id = 8603", ct); + Assert.Equal($"{OtherRole} | {External} | true", await StoredAsync(owner, 8603, ct)); + + bodySucceeded = true; + } + finally + { + var loginToDrop = login; + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, (cleanup, _) => DropLoginRoleAsync(cleanup, loginToDrop)); + } + } +} diff --git a/Darling/Darling.Tests/AwsPerServerRoleRungTests.cs b/Darling/Darling.Tests/AwsPerServerRoleRungTests.cs new file mode 100644 index 0000000000..6a5ce94619 --- /dev/null +++ b/Darling/Darling.Tests/AwsPerServerRoleRungTests.cs @@ -0,0 +1,283 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Globalization; +using System.Linq; +using System.Reflection; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// V169: the per-server AWS role on config.config_monitored_servers (#5452). The static pins: the rung's place in +/// the ladder and its shape, the viewer's version probe, the column ACL, and the self-managed script's copy of the edit +/// function, its wrapper and the trigger rule. The store behaviour is in . +/// +/// This file carries the "I am the top rung" claims, which move to the next rung's tests when one lands above it. +/// +public sealed class AwsPerServerRoleRungTests +{ + internal const int RungVersion = 169; + private const int PreviousVersion = 168; + + /// This rung's sentinel ordinal in the viewer probe. No longer the last argument: V170's sentinel (#5495) follows it. + internal const int ProbeOrdinal = 144; + + private const string RoleColumn = "aws_role_arn"; + + [Fact] + public void TheRungIsRegisteredAtTheTopOfADenseLadder() + { + var versions = PgMigrations.Scripts.Select(s => s.Version).ToList(); + + Assert.Equal("aws-per-server-role", PgMigrations.Scripts.Single(s => s.Version == RungVersion).Name); + /* No longer the top rung: the PostgreSQL I/O hourly rollup rung (V170) landed above it. */ + Assert.Equal(StorageVersion.SchemaVersion, PgMigrations.Scripts[^1].Version); + Assert.Equal(StorageVersion.SchemaVersion, versions.Max()); + Assert.True(RungVersion < StorageVersion.SchemaVersion); + Assert.Equal(versions.Distinct().OrderBy(v => v), versions); + Assert.Equal(PreviousVersion, RungVersion - 1); + Assert.Contains(PreviousVersion, versions); + } + + [Fact] + public void TheRungIsIdempotent_AndMovesNoData() + { + var rung = PgMigrations.Scripts.Single(s => s.Version == RungVersion).Sql; + + Assert.Equal(3, CountOf(rung, "ADD COLUMN IF NOT EXISTS")); + Assert.Equal(3, CountOf(rung, "ALTER TABLE config.config_monitored_servers ADD COLUMN")); + Assert.DoesNotContain("ALTER TABLE config_monitored_servers", rung, StringComparison.Ordinal); + Assert.Contains("aws_role_arn text;", rung, StringComparison.Ordinal); + Assert.Contains("aws_external_id text;", rung, StringComparison.Ordinal); + Assert.Contains("GENERATED ALWAYS AS (aws_external_id IS NOT NULL) STORED", rung, StringComparison.Ordinal); + + /* The check is added once, behind a catalog guard (the V62 pattern), so a re-run does not fail on it. */ + Assert.Equal(1, CountOf(rung, "ADD CONSTRAINT config_monitored_servers_aws_role_check")); + Assert.Contains("WHERE conname = 'config_monitored_servers_aws_role_check'", rung, StringComparison.Ordinal); + Assert.Contains("IF NOT EXISTS (SELECT 1 FROM pg_constraint", rung, StringComparison.Ordinal); + + foreach (var shape in new[] { "UPDATE ", "DELETE ", "INSERT ", "CREATE INDEX", "DROP " }) + { + Assert.DoesNotContain(shape, rung, StringComparison.Ordinal); + } + } + + [Fact] + public void TheCheckIsTheCoarseBackstop_WithNoRepeatCountOverTwoHundredFiftyFive() + { + var rung = PgMigrations.Scripts.Single(s => s.Version == RungVersion).Sql; + + /* No term of the check can be NULL, so an external ID with no role is false rather than unknown. */ + Assert.Contains("(aws_external_id IS NULL OR aws_role_arn IS NOT NULL)", rung, StringComparison.Ordinal); + Assert.Contains("aws_role_arn ~ '^arn:aws(-[a-z]+)*:iam::[0-9]{12}:role/[A-Za-z0-9_+=,.@/-]+$'", rung, StringComparison.Ordinal); + Assert.Contains("char_length(aws_role_arn) <= " + AwsRoleSettings.RoleArnMaxLength, rung, StringComparison.Ordinal); + Assert.Contains("aws_external_id ~ '^[A-Za-z0-9_+=,.@:/-]+$'", rung, StringComparison.Ordinal); + Assert.Contains( + $"char_length(aws_external_id) BETWEEN {AwsRoleSettings.ExternalIdMinLength} AND {AwsRoleSettings.ExternalIdMaxLength}", + rung, StringComparison.Ordinal); + + /* PostgreSQL's regular expressions refuse a repeat count above 255 (2201B at the first use), so the lengths are + char_length terms and the patterns carry no large count. */ + Assert.DoesNotMatch("[{][0-9]+,[0-9]*[}]", rung.Replace("{12}", string.Empty, StringComparison.Ordinal)); + } + + /* ---- the probe (three sites, top arm) ------------------------------------------------------------- */ + + [Fact] + public void TheProbeMapsAFullyMigratedStoreToThisTopRung() + { + Assert.Contains($"column_name = '{RoleColumn}'", ViewerDataService.StoreSchemaProbeSql, StringComparison.Ordinal); + Assert.Contains("table_name = 'config_monitored_servers'", ViewerDataService.StoreSchemaProbeSql, StringComparison.Ordinal); + + var viewer = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.cs"); + Assert.Contains($"reader.GetBoolean({ProbeOrdinal})", viewer, StringComparison.Ordinal); + Assert.Contains($"reader.GetBoolean({ProbeOrdinal + 1})", viewer, StringComparison.Ordinal); + Assert.Contains($"reader.GetBoolean({ProbeOrdinal + 2})", viewer, StringComparison.Ordinal); + Assert.Contains($"reader.GetBoolean({ProbeOrdinal + 3})", viewer, StringComparison.Ordinal); + Assert.Contains($"reader.GetBoolean({ProbeOrdinal + 4})", viewer, StringComparison.Ordinal); + Assert.DoesNotContain($"reader.GetBoolean({ProbeOrdinal + 5})", viewer, StringComparison.Ordinal); + + Assert.Equal(StorageVersion.SchemaVersion, ViewerDataService.RequiredStoreSchemaVersion); + + var method = typeof(ViewerDataService).GetMethod("MapProbedSchemaVersion", BindingFlags.NonPublic | BindingFlags.Static)!; + var arity = method.GetParameters().Length; + + /* A newer rung's sentinel (V173's, #5582) is the last argument. */ + Assert.Equal(ProbeOrdinal + 4, arity - 1); + + var all = Enumerable.Repeat((object)true, arity).ToArray(); + Assert.Equal(StorageVersion.SchemaVersion, (int)method.Invoke(null, all)!); + + var atThisRung = Enumerable.Range(0, arity).Select(i => (object)(i <= ProbeOrdinal)).ToArray(); + Assert.Equal(RungVersion, (int)method.Invoke(null, atThisRung)!); + var behind = (object[])atThisRung.Clone(); + behind[ProbeOrdinal] = false; + Assert.Equal(PreviousVersion, (int)method.Invoke(null, behind)!); + + /* In the source the arm sits ABOVE V168's, and returns this build's version. */ + var v169 = viewer.IndexOf("if (hasAwsRole)", StringComparison.Ordinal); + var v168 = viewer.IndexOf("if (hasPlanRegressionDaily)", StringComparison.Ordinal); + Assert.True(v169 >= 0, "the viewer has no V169 sentinel arm, so a fully-migrated store would map to 168"); + Assert.True(v168 >= 0, "the V168 arm is gone, so this pin is comparing against nothing"); + Assert.True(v169 < v168, "the V169 arm sits below V168's, so a current store maps one rung low"); + Assert.Contains( + "return " + RungVersion.ToString(CultureInfo.InvariantCulture) + ";", + viewer[v169..v168], StringComparison.Ordinal); + } + + /* ---- the column ACL -------------------------------------------------------------------------------- */ + + [Fact] + public void TheColumnAcl_ShowsTheRoleAndTheFlag_AndHidesTheExternalId() + { + var table = DarlingManagedRoles.ViewerRestrictedConfigTables.Single(t => t.Table == "config_monitored_servers"); + + Assert.Contains("aws_role_arn", table.NonSecretColumns); + Assert.Contains("aws_external_id_set", table.NonSecretColumns); + Assert.Contains("aws_external_id", table.SecretColumns); + Assert.DoesNotContain("aws_external_id", table.NonSecretColumns); + + /* The self-managed script grants the same two columns to both read-only roles, and never the ID. */ + var script = System.Text.RegularExpressions.Regex.Replace( + RepoFile.ReadRepoFile("Darling", "tools", "provision-roles.sql"), @"(?m)^\s*--.*$", string.Empty); + var grants = System.Text.RegularExpressions.Regex.Matches( + script, @"GRANT SELECT \([^;]*?\)\s+ON config\.config_monitored_servers TO (viewer|mcp);", + System.Text.RegularExpressions.RegexOptions.Singleline); + Assert.Equal(2, grants.Count); + foreach (System.Text.RegularExpressions.Match grant in grants) + { + var uncommented = grant.Value; + Assert.Contains("aws_role_arn", uncommented, StringComparison.Ordinal); + Assert.Contains("aws_external_id_set", uncommented, StringComparison.Ordinal); + Assert.DoesNotMatch(@"(?= 0 && wrapperAt > functionAt, "the wrapper must be created after the function it calls"); + Assert.Contains( + $"GRANT EXECUTE ON FUNCTION config.edit_monitored_server({DarlingManagedRoles.EditMonitoredServerSignature}) TO viewer, mcp;", + batch, StringComparison.Ordinal); + Assert.Contains( + $"GRANT EXECUTE ON FUNCTION config.edit_monitored_server({DarlingManagedRoles.EditMonitoredServerLegacySignature}) TO viewer, mcp;", + batch, StringComparison.Ordinal); + } + + [Fact] + public void TheSelfManagedScript_CarriesTheWrapper_BothGrants_AndTheTriggerRule() + { + var script = System.Text.RegularExpressions.Regex.Replace( + RepoFile.ReadRepoFile("Darling", "tools", "provision-roles.sql"), @"(?m)^\s*--.*$", string.Empty); + + Assert.Contains( + NormalizeSql(DarlingManagedRoles.BuildEditMonitoredServerLegacyWrapperSql("config")), NormalizeSql(script), StringComparison.Ordinal); + Assert.Contains( + $"GRANT EXECUTE ON FUNCTION config.edit_monitored_server({DarlingManagedRoles.EditMonitoredServerSignature}) TO viewer, mcp;", + script, StringComparison.Ordinal); + Assert.Contains( + $"GRANT EXECUTE ON FUNCTION config.edit_monitored_server({DarlingManagedRoles.EditMonitoredServerLegacySignature}) TO viewer, mcp;", + script, StringComparison.Ordinal); + + var rules = DarlingManagedRoles.BuildServerPasswordRulesSql("config"); + Assert.Contains("USING ERRCODE = 'PW004'", rules, StringComparison.Ordinal); + Assert.Contains(AwsRoleSettings.RoleChangeNeedsExternalIdMessage, rules, StringComparison.Ordinal); + Assert.Contains(NormalizeSql(rules), NormalizeSql(script), StringComparison.Ordinal); + } + + [Fact] + public void TheMcpLogin_UpdatesEveryColumnButTheRoleAndExternalId_InTheManagedBatchAndTheScript() + { + var columns = DarlingManagedRoles.McpMonitoredServerUpdateColumns.Split(',', StringSplitOptions.TrimEntries); + Assert.DoesNotContain("aws_role_arn", columns); + Assert.DoesNotContain("aws_external_id", columns); + Assert.DoesNotContain("aws_external_id_set", columns); + Assert.Equal(columns.Length, columns.Distinct(StringComparer.Ordinal).Count()); + + var batch = NormalizeSql(DarlingManagedRoles.BuildProvisioningSql( + ProvisioningTestSecrets.Admin, ProvisioningTestSecrets.Viewer, ProvisioningTestSecrets.Mcp)); + var script = NormalizeSql(RepoFile.ReadRepoFile("Darling", "tools", "provision-roles.sql")); + + Assert.Contains($"GRANT UPDATE ({DarlingManagedRoles.McpMonitoredServerUpdateColumns}) ON config.config_monitored_servers TO mcp;", script, StringComparison.Ordinal); + Assert.Contains("REVOKE UPDATE ON config.config_monitored_servers FROM mcp;", script, StringComparison.Ordinal); + Assert.Contains("GRANT INSERT, DELETE ON config.config_monitored_servers TO mcp;", script, StringComparison.Ordinal); + Assert.DoesNotContain("GRANT INSERT, UPDATE, DELETE ON config.config_monitored_servers", script, StringComparison.Ordinal); + + Assert.Contains($"GRANT UPDATE ({DarlingManagedRoles.McpMonitoredServerUpdateColumns}) ON", batch, StringComparison.Ordinal); + Assert.Contains("REVOKE UPDATE ON", batch, StringComparison.Ordinal); + Assert.Contains("GRANT INSERT, DELETE ON config.config_monitored_servers TO", batch, StringComparison.Ordinal); + Assert.DoesNotContain("GRANT INSERT, UPDATE, DELETE ON config.config_monitored_servers", batch, StringComparison.Ordinal); + } + + [Fact] + public void TheEditFunction_AnswersInvalidValue_ForAnEmptyRequiredField_AndForAValueTheTableRefuses() + { + foreach (var body in new[] + { + NormalizeSql(DarlingManagedRoles.BuildEditMonitoredServerFunctionSql("config")), + NormalizeSql(RepoFile.ReadRepoFile("Darling", "tools", "provision-roles.sql")), + }) + { + Assert.Contains("'name' = ANY (p_columns) AND p_name IS NULL", body, StringComparison.Ordinal); + Assert.Contains("OR v_host IS NULL OR v_port IS NULL OR v_auth IS NULL OR v_encrypt_mode IS NULL", body, StringComparison.Ordinal); + Assert.Contains("'invalid_value'::text", body, StringComparison.Ordinal); + Assert.Contains("EXCEPTION WHEN integrity_constraint_violation THEN", body, StringComparison.Ordinal); + } + } + + private static string NormalizeSql(string sql) => + System.Text.RegularExpressions.Regex.Replace( + System.Text.RegularExpressions.Regex.Replace(sql, @"(?m)^\s*--.*$", ""), @"\s+", " ").Trim(); + + private static int CountOf(string haystack, string needle) + { + var count = 0; + for (var at = haystack.IndexOf(needle, StringComparison.Ordinal); + at >= 0; + at = haystack.IndexOf(needle, at + needle.Length, StringComparison.Ordinal)) + { + count++; + } + + return count; + } +} diff --git a/Darling/Darling.Tests/AwsRoleAllowlistTests.cs b/Darling/Darling.Tests/AwsRoleAllowlistTests.cs new file mode 100644 index 0000000000..716c1de8c7 --- /dev/null +++ b/Darling/Darling.Tests/AwsRoleAllowlistTests.cs @@ -0,0 +1,317 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.IO; +using System.Linq; +using System.Text.RegularExpressions; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Targets; +using Xunit; + +namespace Darling.Tests; + +/// +/// Which AWS roles the web and MCP may set (#5452): the list type, the allowedAwsRoles darling.json setting, +/// and where the worker sets the list in use. +/// +public sealed class AwsRoleAllowlistTests +{ + private const string Role = "arn:aws:iam::123456789012:role/darling-monitor"; + private const string PathRole = "arn:aws:iam::123456789012:role/team/darling-monitor"; + private const string OtherAccountRole = "arn:aws:iam::210987654321:role/darling-monitor"; + + [Fact] + public void Empty_AllowsNothing() + { + Assert.False(AwsRoleAllowlist.Empty.IsAllowed(Role)); + Assert.False(AwsRoleAllowlist.Empty.IsAllowed(null)); + Assert.Equal(0, AwsRoleAllowlist.Empty.Count); + } + + [Fact] + public void Current_StartsEmpty() + { + // Nothing in the test run sets it: a test names its list through the overloads. + Assert.Same(AwsRoleAllowlist.Empty, AwsRoleAllowlist.Current); + } + + [Fact] + public void AnArnEntry_AllowsExactlyThatArn_CaseSensitive() + { + var list = AwsRoleAllowlist.From(new[] { Role }); + + Assert.True(list.IsAllowed(Role)); + Assert.True(AwsRoleAllowlist.IsAllowed(Role, list)); + Assert.False(list.IsAllowed(Role.ToUpperInvariant())); + Assert.False(list.IsAllowed("arn:aws:iam::123456789012:role/Darling-Monitor")); + Assert.False(list.IsAllowed(PathRole)); + Assert.False(list.IsAllowed(OtherAccountRole)); + } + + [Fact] + public void AnAccountIdEntry_AllowsEveryRoleInThatAccount_AndNoOther() + { + var list = AwsRoleAllowlist.From(new[] { "123456789012" }); + + Assert.True(list.IsAllowed(Role)); + Assert.True(list.IsAllowed(PathRole)); + Assert.False(list.IsAllowed(OtherAccountRole)); + } + + [Fact] + public void ABareAccountIdEntry_AllowsThatAccountInTheAwsPartitionOnly() + { + var list = AwsRoleAllowlist.From(new[] { "123456789012" }); + + Assert.True(list.IsAllowed(Role)); + Assert.False(list.IsAllowed("arn:aws-cn:iam::123456789012:role/darling-monitor")); + Assert.False(list.IsAllowed("arn:aws-us-gov:iam::123456789012:role/darling-monitor")); + } + + [Fact] + public void APartitionQualifiedAccountEntry_AllowsThatAccountInThatPartitionOnly() + { + var china = AwsRoleAllowlist.From(new[] { "aws-cn:123456789012" }); + Assert.True(china.IsAllowed("arn:aws-cn:iam::123456789012:role/darling-monitor")); + Assert.True(china.IsAllowed("arn:aws-cn:iam::123456789012:role/team/darling-monitor")); + Assert.False(china.IsAllowed(Role)); + Assert.False(china.IsAllowed("arn:aws-us-gov:iam::123456789012:role/darling-monitor")); + Assert.False(china.IsAllowed("arn:aws-cn:iam::210987654321:role/darling-monitor")); + + var gov = AwsRoleAllowlist.From(new[] { "aws-us-gov:123456789012" }); + Assert.True(gov.IsAllowed("arn:aws-us-gov:iam::123456789012:role/darling-monitor")); + Assert.False(gov.IsAllowed(Role)); + Assert.False(gov.IsAllowed("arn:aws-cn:iam::123456789012:role/darling-monitor")); + } + + [Fact] + public void TheAwsPartitionCanBeWrittenOut_AndMatchesTheBareForm() + { + var list = AwsRoleAllowlist.From(new[] { "aws:123456789012" }); + Assert.True(list.IsAllowed(Role)); + Assert.False(list.IsAllowed("arn:aws-cn:iam::123456789012:role/darling-monitor")); + } + + [Fact] + public void SeveralPartitionEntries_EachAllowTheirOwnPartition() + { + var list = AwsRoleAllowlist.From(new[] { "123456789012", "aws-cn:123456789012", "aws-us-gov:210987654321" }); + Assert.Equal(3, list.Count); + Assert.True(list.IsAllowed(Role)); + Assert.True(list.IsAllowed("arn:aws-cn:iam::123456789012:role/darling-monitor")); + Assert.False(list.IsAllowed("arn:aws-us-gov:iam::123456789012:role/darling-monitor")); + Assert.True(list.IsAllowed("arn:aws-us-gov:iam::210987654321:role/darling-monitor")); + } + + [Fact] + public void AnAccountIdEntry_DoesNotMatchAnAccountIdInsideAnotherPartOfTheArn() + { + var list = AwsRoleAllowlist.From(new[] { "123456789012" }); + + // The account id appears in the role name and path, not in the account field. + Assert.False(list.IsAllowed("arn:aws:iam::210987654321:role/123456789012")); + Assert.False(list.IsAllowed("arn:aws:iam::210987654321:role/123456789012/darling-monitor")); + // Not a role ARN at all. + Assert.False(list.IsAllowed("arn:aws:iam::123456789012:user/darling-monitor")); + Assert.False(list.IsAllowed("123456789012")); + } + + [Theory] + [InlineData(null)] + [InlineData("")] + [InlineData(" ")] + public void ABlankRole_IsNeverAllowed(string? role) + { + var list = AwsRoleAllowlist.From(new[] { Role, "123456789012" }); + + Assert.False(list.IsAllowed(role)); + } + + [Fact] + public void TheRoleAndEntriesAreTrimmed() + { + var list = AwsRoleAllowlist.From(new[] { " " + Role + " ", " 123456789012 " }); + + Assert.True(list.IsAllowed(" " + Role)); + Assert.True(list.IsAllowed(OtherAccountRole.Replace("210987654321", "123456789012"))); + Assert.Equal(2, list.Count); + } + + [Fact] + public void From_DropsBlankAndUnusableEntries_AndCountsARepeatOnce() + { + var list = AwsRoleAllowlist.From(new string?[] { null, "", " ", "not-an-arn", "12345", "1234567890123", Role, Role }); + + Assert.Equal(1, list.Count); + Assert.True(list.IsAllowed(Role)); + Assert.Same(AwsRoleAllowlist.Empty, AwsRoleAllowlist.From(new string?[] { null, "junk" })); + Assert.Same(AwsRoleAllowlist.Empty, AwsRoleAllowlist.From(null)); + } + + [Theory] + [InlineData(Role, true)] + [InlineData(PathRole, true)] + [InlineData("123456789012", true)] + [InlineData(" 123456789012 ", true)] + [InlineData("aws-cn:123456789012", true)] + [InlineData("aws-us-gov:123456789012", true)] + [InlineData(" aws-cn:123456789012 ", true)] + [InlineData("aws-cn:12345678901", false)] + [InlineData("aws-cn:", false)] + [InlineData(":123456789012", false)] + [InlineData("china:123456789012", false)] + [InlineData("aws-cn:aws-cn:123456789012", false)] + [InlineData("12345678901", false)] + [InlineData("1234567890123", false)] + [InlineData("12345678901a", false)] + [InlineData("*", false)] + [InlineData("", false)] + [InlineData("arn:aws:iam::123456789012:user/x", false)] + [InlineData("arn:aws:iam::123456789012:role/", false)] + public void ValidateEntry_AcceptsAnArnOrATwelveDigitId(string entry, bool ok) + { + Assert.Equal(ok, AwsRoleAllowlist.ValidateEntry(entry) is null); + } + + [Fact] + public void ValidateEntry_AnswersWithTheFixedSentence() + { + Assert.Equal(AwsRoleAllowlist.InvalidEntryMessage, AwsRoleAllowlist.ValidateEntry("junk")); + Assert.Equal(AwsRoleAllowlist.InvalidEntryMessage, AwsRoleAllowlist.ValidateEntry(null)); + } + + [Fact] + public void FromConfig_IsTheSettingPlusEveryServersRole() + { + var config = new DarlingConfig(); + config.AllowedAwsRoles.Add(Role); + config.AllowedAwsRoles.Add("210987654321"); + config.Servers.Add(new MonitoredServer { Name = "a", AwsRoleArn = "arn:aws:iam::111111111111:role/from-server" }); + config.Servers.Add(new MonitoredServer { Name = "b" }); + + var list = AwsRoleAllowlist.FromConfig(config); + + Assert.True(list.IsAllowed(Role)); + Assert.True(list.IsAllowed(OtherAccountRole)); + Assert.True(list.IsAllowed("arn:aws:iam::111111111111:role/from-server")); + Assert.False(list.IsAllowed("arn:aws:iam::111111111111:role/other")); + Assert.Equal(3, list.Count); + } + + [Fact] + public void FromConfig_WithNothingSet_AllowsNothing() + { + Assert.Same(AwsRoleAllowlist.Empty, AwsRoleAllowlist.FromConfig(new DarlingConfig())); + } + + [Fact] + public void TheSettingReadsFromDarlingJson_UnderItsCamelCaseName() + { + var config = DarlingConfig.Parse(""" + { "allowedAwsRoles": [ "arn:aws:iam::123456789012:role/darling-monitor", "210987654321" ] } + """); + + Assert.Equal(new[] { Role, "210987654321" }, config.AllowedAwsRoles); + } + + [Fact] + public void WhenTheSettingIsOmitted_TheListIsEmpty_AndTheSampleValidatesClean() + { + Assert.Empty(new DarlingConfig().AllowedAwsRoles); + + var sample = DarlingConfig.Parse(File.ReadAllText(Path.Combine(AppContext.BaseDirectory, "darling.sample.json"))); + Assert.Empty(sample.AllowedAwsRoles); + Assert.Empty(sample.Validate()); + } + + [Fact] + public void Validate_RejectsAnEntryThatIsNeitherAnArnNorATwelveDigitId_NamingItsPosition() + { + var config = new DarlingConfig(); + config.Postgres.ConnectionString = "Host=localhost;Database=darling;Username=x;Password=y"; + config.Servers.Add(new MonitoredServer { Name = "sql1", Host = "sql1.example", Auth = "integrated" }); + config.AllowedAwsRoles.Add(Role); + config.AllowedAwsRoles.Add("123456789012"); + config.AllowedAwsRoles.Add("aws-cn:123456789012"); + Assert.Empty(config.Validate()); + + config.AllowedAwsRoles.Add("not-a-role"); + config.AllowedAwsRoles.Add(""); + var problems = config.Validate(); + + Assert.Equal(2, problems.Count); + Assert.Contains(problems, p => p.StartsWith("allowedAwsRoles[3]: ", StringComparison.Ordinal) + && p.EndsWith(AwsRoleAllowlist.InvalidEntryMessage, StringComparison.Ordinal)); + Assert.Contains(problems, p => p.StartsWith("allowedAwsRoles[4]: ", StringComparison.Ordinal)); + } + + [Fact] + public void TheWorkerSetsTheListInUseOnce_BeforeItBuildsTheRunner() + { + var dir = AppContext.BaseDirectory; + string? worker = null; + while (dir is not null) + { + var candidate = Path.Combine(dir, "PerformanceMonitor.Darling.Service", "DarlingWorker.cs"); + if (File.Exists(candidate)) + { + worker = candidate; + break; + } + + dir = Path.GetDirectoryName(dir); + } + + if (worker is null) + { + // The test host runs away from the source tree (a published layout): nothing to pin. + return; + } + + var text = File.ReadAllText(worker); + var sets = Regex.Matches(text, @"AwsRoleAllowlist\.Current\s*=").Count; + Assert.Equal(1, sets); + Assert.True( + text.IndexOf("AwsRoleAllowlist.Current =", StringComparison.Ordinal) + < text.IndexOf("new DarlingCollectorRunner(", StringComparison.Ordinal), + "The list is set before the runner is built."); + } + + [Fact] + public void NothingOutsideTheWorkerSetsTheListInUse() + { + var dir = AppContext.BaseDirectory; + string? service = null; + while (dir is not null) + { + var candidate = Path.Combine(dir, "PerformanceMonitor.Darling.Service"); + if (File.Exists(Path.Combine(candidate, "DarlingWorker.cs"))) + { + service = candidate; + break; + } + + dir = Path.GetDirectoryName(dir); + } + + if (service is null) + { + return; + } + + var setters = Directory.EnumerateFiles(service, "*.cs", SearchOption.AllDirectories) + .Where(f => !f.Contains($"{Path.DirectorySeparatorChar}obj{Path.DirectorySeparatorChar}", StringComparison.Ordinal) + && !f.Contains($"{Path.DirectorySeparatorChar}bin{Path.DirectorySeparatorChar}", StringComparison.Ordinal)) + .Where(f => Regex.IsMatch(File.ReadAllText(f), @"AwsRoleAllowlist\.Current\s*=")) + .Select(Path.GetFileName) + .ToArray(); + + Assert.Equal(new[] { "DarlingWorker.cs" }, setters); + } +} diff --git a/Darling/Darling.Tests/AwsRoleBundleEndToEndLiveTests.cs b/Darling/Darling.Tests/AwsRoleBundleEndToEndLiveTests.cs new file mode 100644 index 0000000000..42c2b5e9f4 --- /dev/null +++ b/Darling/Darling.Tests/AwsRoleBundleEndToEndLiveTests.cs @@ -0,0 +1,186 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.IO; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Targets; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: the test mints its own scratch database through ScratchPostgres. */ + +/// +/// #5452: each server can use its own AWS role, end to end through the diagnostics bundle. The worker's role arm writes a +/// PERMISSIONS row to collection_log for a server that has a role and an external ID; a bundle is then built from that +/// store (the server is known only to the store registry); and the bundle names the role as an alias and never holds the +/// external ID. Names are synthetic (the example account 123456789012). +/// +public sealed class AwsRoleBundleEndToEndLiveTests +{ + private const string Role = "arn:aws:iam::123456789012:role/darling-monitor"; + private const string ExternalId = "ext-7Hq2mZ9vLx"; + private const string ServerName = "alpha-pg-01"; + private const string Host = "alpha-pg-01.abcdefghijkl.us-east-1.rds.amazonaws.com"; + + private static string? BaseConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + /* The server as the store holds it: the registry row (with its role and external ID) and the collected-data server row. */ + private static async Task SeedServerAsync(NpgsqlConnection connection, System.Threading.CancellationToken ct) + { + await using var insert = new NpgsqlCommand( + "INSERT INTO config.config_monitored_servers (server_id, name, host, database, auth, username, engine, port, aws_role_arn, aws_external_id) " + + "VALUES (9201, $1, $2, 'postgres', 'sql', 'pgcollect', 'postgres', 5432, $3, $4)", connection); + insert.Parameters.AddWithValue(ServerName); + insert.Parameters.AddWithValue(Host); + insert.Parameters.AddWithValue(Role); + insert.Parameters.AddWithValue(ExternalId); + await insert.ExecuteNonQueryAsync(ct); + + await using var registered = new NpgsqlCommand( + "INSERT INTO collect.servers (server_id, server_name, display_name, is_enabled, created_date, modified_date) " + + "VALUES (9201, $1, $1, true, now(), now())", connection); + registered.Parameters.AddWithValue(ServerName); + await registered.ExecuteNonQueryAsync(ct); + } + + [Fact] + public async Task ABundleBuiltFromARoleServersStore_NamesTheRoleAsAnAlias_AndNeverHoldsTheExternalId() + { + Assert.SkipWhen(string.IsNullOrEmpty(BaseConnectionString), "Set DARLING_TEST_PG to run the live bundle test."); + var ct = TestContext.Current.CancellationToken; + var scratch = await ScratchPostgres.CreateAsync(BaseConnectionString!, ct); + var root = Directory.CreateTempSubdirectory("darling-bundle-awsrole-"); + var bodySucceeded = false; + try + { + await using (var connection = new NpgsqlConnection(scratch.ConnectionString)) + { + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await SeedServerAsync(connection, ct); + } + + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + + // The role arm's input: an STS refusal whose own text repeats the external ID, read through the real RDS reader. + var sts = new FakeSts((_, _) => throw FakeSts.Error("AccessDenied", $"not authorized (ExternalId={ExternalId})")); + var cache = new AwsRoleCredentialCache(AwsRoleAllowlist.From([Role]), null, sts.Factory, FakeSts.Source, new ManualTimeProvider(), null, new CapturingAwsLogger()); + var ingestor = new RdsCpuIngestor( + NpgsqlDataSource.Create("Host=localhost;Port=1;Database=unused"), + verifier: RdsEndpointVerifier.ForTests(enforce: true, loginProbe: (_, _) => Task.FromResult(null)), + roles: cache); + var thrown = await Record.ExceptionAsync( + () => ingestor.IngestAsync(9201, ServerName, Host, "Host=" + Host, new AwsRoleKey(Role, ExternalId), ct)); + var fault = DarlingWorker.AwsRoleConfigurationFault(thrown!); + Assert.NotNull(fault); + + // The worker's role arm: its note, written to collection_log as PERMISSIONS in the storage-time slot. + var runtime = new ServerRuntime + { + Config = new MonitoredServer { Name = ServerName, Host = Host }, + ConnectionString = "Host=" + Host, + Target = new CollectorTargetInfo { Engine = CollectorTargetEngine.PostgreSql }, + StorageName = ServerName, + ServerId = 9201, + }; + var note = DarlingWorker.AwsRoleFaultNote(new CapturingAwsLogger(), ServerName, "pg_plan_capture", fault!); + await DarlingObservability.LogCollectionAsync( + postgres, runtime, "pg_plan_capture", "PERMISSIONS", 0, 0, 5, note, + fanout: null, phases: null, drain: null, fetchPhases: null, sweepPeerMaxMs: null, null, ct); + + // The store really holds the role in that row, so the bundle's alias is doing the work. + await using (var verify = await postgres.OpenConnectionAsync(ct)) + await using (var read = new NpgsqlCommand( + "SELECT error_message FROM collection_log WHERE server_id = 9201 AND status = 'PERMISSIONS'", verify)) + { + var stored = (string?)await read.ExecuteScalarAsync(ct); + Assert.Contains(Role, stored, StringComparison.Ordinal); + Assert.DoesNotContain(ExternalId, stored, StringComparison.Ordinal); + } + + var options = DiagnosticsBundle.ParseArgs(new[] { Path.Combine(root.FullName, "b.json"), "--log-dir", root.FullName }).Options!; + var outcome = await DiagnosticsBundleRunner.BuildAsync(options, new DarlingConfig(), scratch.ConnectionString, postgres, null, null, ct); + + Assert.NotEqual(DarlingCliCommands.DiagnosticsBundleExitCode.LeakGuard, outcome.ExitCode); + Assert.NotNull(outcome.Text); + Assert.DoesNotContain(ExternalId, outcome.Text, StringComparison.OrdinalIgnoreCase); + Assert.DoesNotContain(Role, outcome.Text, StringComparison.Ordinal); + Assert.DoesNotContain("darling-monitor", outcome.Text, StringComparison.Ordinal); + Assert.DoesNotContain("123456789012", outcome.Text, StringComparison.Ordinal); + Assert.DoesNotContain(ServerName, outcome.Text, StringComparison.Ordinal); + + // The bundle still says what happened: the collector's PERMISSIONS answer, with the role under its alias. + Assert.Contains("PERMISSIONS", outcome.Text, StringComparison.Ordinal); + Assert.Contains("role-", outcome.Text, StringComparison.Ordinal); + Assert.Contains("trust policy", outcome.Text, StringComparison.Ordinal); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => await Task.CompletedTask); + await scratch.DisposeAsync(); + root.Delete(recursive: true); + } + } + + /* A second row that carries the external ID as plain text, the way an unscrubbed message from AWS could: the bundle + removes it as well, so no text the store holds can carry the ID out. */ + [Fact] + public async Task AnExternalIdTheStoreHoldsInAnyRowText_IsRemovedFromTheBundle() + { + Assert.SkipWhen(string.IsNullOrEmpty(BaseConnectionString), "Set DARLING_TEST_PG to run the live bundle test."); + var ct = TestContext.Current.CancellationToken; + var scratch = await ScratchPostgres.CreateAsync(BaseConnectionString!, ct); + var root = Directory.CreateTempSubdirectory("darling-bundle-awsrole-"); + var bodySucceeded = false; + try + { + await using (var connection = new NpgsqlConnection(scratch.ConnectionString)) + { + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await SeedServerAsync(connection, ct); + } + + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var runtime = new ServerRuntime + { + Config = new MonitoredServer { Name = ServerName, Host = Host }, + ConnectionString = "Host=" + Host, + Target = new CollectorTargetInfo { Engine = CollectorTargetEngine.PostgreSql }, + StorageName = ServerName, + ServerId = 9201, + }; + await DarlingObservability.LogCollectionAsync( + postgres, runtime, "pg_plan_capture", "ERROR", 0, 0, 5, $"AWS answered for role {Role}: request externalId={ExternalId}", + fanout: null, phases: null, drain: null, fetchPhases: null, sweepPeerMaxMs: null, null, ct); + + var options = DiagnosticsBundle.ParseArgs(new[] { Path.Combine(root.FullName, "b.json"), "--log-dir", root.FullName }).Options!; + var outcome = await DiagnosticsBundleRunner.BuildAsync(options, new DarlingConfig(), scratch.ConnectionString, postgres, null, null, ct); + + Assert.NotEqual(DarlingCliCommands.DiagnosticsBundleExitCode.LeakGuard, outcome.ExitCode); + Assert.NotNull(outcome.Text); + Assert.DoesNotContain(ExternalId, outcome.Text, StringComparison.OrdinalIgnoreCase); + Assert.DoesNotContain(Role, outcome.Text, StringComparison.Ordinal); + Assert.DoesNotContain("darling-monitor", outcome.Text, StringComparison.Ordinal); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => await Task.CompletedTask); + await scratch.DisposeAsync(); + root.Delete(recursive: true); + } + } +} diff --git a/Darling/Darling.Tests/AwsRoleCredentialCacheTests.cs b/Darling/Darling.Tests/AwsRoleCredentialCacheTests.cs new file mode 100644 index 0000000000..3e5c0eccc4 --- /dev/null +++ b/Darling/Darling.Tests/AwsRoleCredentialCacheTests.cs @@ -0,0 +1,195 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Threading.Tasks; +using PerformanceMonitor.Darling.Service.Targets; +using Xunit; + +namespace Darling.Tests; + +/// +/// The per-process cache of assumed-role credentials (#5452): the allow list and the partition are checked on every +/// call and before any AWS call, and one key means one entry. The example account is 123456789012. +/// +public sealed class AwsRoleCredentialCacheTests +{ + private const string Role = "arn:aws:iam::123456789012:role/darling-monitor"; + private const string ChinaRole = "arn:aws-cn:iam::123456789012:role/darling-monitor"; + private const string GovRole = "arn:aws-us-gov:iam::123456789012:role/darling-monitor"; + + private static AwsRoleCredentialCache Cache(FakeSts sts, AwsRoleAllowlist? list = null, ManualTimeProvider? clock = null) => + new(list ?? AwsRoleAllowlist.From(new[] { "123456789012", "aws-cn:123456789012", "aws-us-gov:123456789012" }), null, sts.Factory, FakeSts.Source, clock ?? new ManualTimeProvider()); + + /* ---- the allow list, at use ---------------------------------------------------------------------------- */ + + [Fact] + public void ARoleTheListDoesNotAllow_IsNeverAssumed_WithTheSettledText() + { + var sts = new FakeSts(); + var cache = Cache(sts, AwsRoleAllowlist.Empty); + + var ex = Assert.Throws(() => cache.For(new AwsRoleKey(Role, null), "us-east-1")); + + Assert.Equal(AwsRoleAssumeKind.NotAllowed, ex.Kind); + Assert.True(ex.IsConfiguration); + Assert.Equal( + "The AWS role arn:aws:iam::123456789012:role/darling-monitor on this server is not in allowedAwsRoles in darling.json, so it was not used. " + + "Nothing was read this cycle. Add the role or its account id to the list and restart the service, or clear the role on this server.", + ex.Message); + Assert.Equal(0, cache.Count); + Assert.Equal(0, sts.Calls); + Assert.Empty(sts.Regions); + } + + [Fact] + public void ARoleOnAnotherAccountsList_IsNotAllowed() + { + var cache = Cache(new FakeSts(), AwsRoleAllowlist.From(new[] { "999999999999", "arn:aws:iam::123456789012:role/other-role" })); + + var ex = Assert.Throws(() => cache.For(new AwsRoleKey(Role, null), "us-east-1")); + + Assert.Equal(AwsRoleAssumeKind.NotAllowed, ex.Kind); + } + + [Theory] + [InlineData("123456789012")] + [InlineData("arn:aws:iam::123456789012:role/darling-monitor")] + public void AnAccountIdOrTheExactArn_AllowsTheRole(string entry) + { + var cache = Cache(new FakeSts(), AwsRoleAllowlist.From(new[] { entry })); + + Assert.NotNull(cache.For(new AwsRoleKey(Role, null), "us-east-1")); + Assert.True(cache.Contains(new AwsRoleKey(Role, null))); + } + + [Fact] + public async Task TheListIsCheckedOnEveryCall_EvenWhenTheRoleIsAlreadyCached() + { + var sts = new FakeSts(); + var cache = Cache(sts); + var key = new AwsRoleKey(Role, null); + + var credentials = cache.For(key, "us-east-1"); + await credentials.GetCredentialsAsync(); + Assert.Equal(1, sts.Calls); + + var ex = Assert.Throws(() => cache.For(key, "us-east-1", AwsRoleAllowlist.Empty)); + Assert.Equal(AwsRoleAssumeKind.NotAllowed, ex.Kind); + Assert.Equal(1, sts.Calls); + + /* The cache's own list still allows it, so the next call without an override is served. */ + Assert.NotNull(cache.For(key, "us-east-1")); + } + + [Fact] + public void TheListIsCheckedBeforeThePartition() + { + var cache = Cache(new FakeSts(), AwsRoleAllowlist.Empty); + + var ex = Assert.Throws(() => cache.For(new AwsRoleKey(Role, null), "cn-north-1")); + + Assert.Equal(AwsRoleAssumeKind.NotAllowed, ex.Kind); + } + + /* ---- the partition, on every call ---------------------------------------------------------------------- */ + + [Fact] + public async Task TheRolePartitionMustBeTheRegionsPartition_OnEveryCall() + { + var sts = new FakeSts(); + var cache = Cache(sts); + var key = new AwsRoleKey(Role, null); + + await cache.For(key, "us-east-1").GetCredentialsAsync(); + + var china = Assert.Throws(() => cache.For(key, "cn-north-1")); + Assert.Equal(AwsRoleAssumeKind.PartitionMismatch, china.Kind); + Assert.True(china.IsConfiguration); + Assert.Equal( + "Role arn:aws:iam::123456789012:role/darling-monitor is in AWS partition aws, but this target's region cn-north-1 is in partition aws-cn. " + + "A role can be assumed only inside its own partition. Nothing was read this cycle. " + + "Set a role from the target's partition on this server, or clear the role, or correct the server's host name if it names the wrong region.", + china.Message); + + /* The entry exists and was used from us-east-1, and the mismatch is still refused: the check is per call. */ + var gov = Assert.Throws(() => cache.For(key, "us-gov-west-1")); + Assert.Equal(AwsRoleAssumeKind.PartitionMismatch, gov.Kind); + Assert.NotNull(cache.For(key, "eu-west-1")); + Assert.Equal(1, sts.Calls); + } + + [Theory] + [InlineData(ChinaRole, "cn-north-1")] + [InlineData(GovRole, "us-gov-west-1")] + [InlineData(Role, "ap-southeast-2")] + public void ARoleInTheRegionsOwnPartition_IsServed(string role, string region) + { + var cache = Cache(new FakeSts()); + + Assert.NotNull(cache.For(new AwsRoleKey(role, null), region)); + } + + [Fact] + public void AnArnWithNoPartition_IsAMismatch_NotACrash() + { + var cache = Cache(new FakeSts(), AwsRoleAllowlist.From(new[] { "not-an-arn" })); + + Assert.Equal("", AwsRoleCredentialCache.PartitionOf("not-an-arn")); + Assert.Equal("aws-cn", AwsRoleCredentialCache.PartitionOf(ChinaRole)); + Assert.Throws(() => cache.For(new AwsRoleKey(Role, null), " ")); + } + + /* ---- one entry per key --------------------------------------------------------------------------------- */ + + [Fact] + public async Task OneKey_OneEntry_OneStsCall_ForEveryServerThatNamesIt() + { + var sts = new FakeSts(); + var cache = Cache(sts); + var key = new AwsRoleKey(Role, "shared-external-id"); + + var first = cache.For(key, "us-east-1"); + var second = cache.For(new AwsRoleKey(Role, "shared-external-id"), "eu-west-1"); + await Task.WhenAll(first.GetCredentialsAsync(), second.GetCredentialsAsync()); + + Assert.Equal(1, cache.Count); + Assert.Equal(1, sts.Calls); + } + + [Fact] + public void ARoleWithAnotherExternalId_IsAnotherEntry() + { + var cache = Cache(new FakeSts()); + + cache.For(new AwsRoleKey(Role, "first-external-id"), "us-east-1"); + cache.For(new AwsRoleKey(Role, "second-external-id"), "us-east-1"); + cache.For(new AwsRoleKey(Role, null), "us-east-1"); + + Assert.Equal(3, cache.Count); + } + + [Fact] + public void Retain_DropsTheKeysNoEnabledServerNamesAnyMore() + { + var cache = Cache(new FakeSts()); + var keep = new AwsRoleKey(Role, "keep-external-id"); + var drop = new AwsRoleKey(Role, "drop-external-id"); + cache.For(keep, "us-east-1"); + cache.For(drop, "us-east-1"); + + cache.Retain(new[] { keep }); + + Assert.True(cache.Contains(keep)); + Assert.False(cache.Contains(drop)); + Assert.Equal(1, cache.Count); + + cache.Retain(Array.Empty()); + Assert.Equal(0, cache.Count); + } +} diff --git a/Darling/Darling.Tests/AwsRoleEditTests.cs b/Darling/Darling.Tests/AwsRoleEditTests.cs new file mode 100644 index 0000000000..5442ff252d --- /dev/null +++ b/Darling/Darling.Tests/AwsRoleEditTests.cs @@ -0,0 +1,468 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using NpgsqlTypes; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Targets; +using PerformanceMonitor.Darling.Storage; +using Xunit; +using Edit = PerformanceMonitor.Darling.Service.Mcp.DarlingMcpServerAdminTools; + +namespace Darling.Tests; + +/// +/// #5452: which AWS roles the web and MCP may set. The edit's plan and the add's parse take the list as an argument +/// (these tests never set , a process-wide value): no list, a role not on it, a +/// listed role, a 12-digit account entry, a blank role that keeps the stored one, an explicit null that removes it, a +/// role change that would keep the stored external ID unseen, and the typed external ID kept out of every answer and +/// audit line. The role below is the example account's. +/// +public sealed class AwsRoleEditTests +{ + private const string RoleA = "arn:aws:iam::123456789012:role/darling-monitor"; + private const string RoleB = "arn:aws:iam::123456789012:role/darling-other"; + private const string OtherAccountRole = "arn:aws:iam::210987654321:role/darling-monitor"; + private const string TypedId = "ext-id-7f3a9c"; + + private static readonly DateTime Stamp = new(2026, 10, 5, 12, 0, 0, 123, DateTimeKind.Unspecified); + + private static Edit.ServerEditRow PgRow(string? role = null, bool idSet = false) => new( + ServerId: 41, Name: "pg-01", Host: "pg-01.example.test", Port: 5432, Database: null, ReadOnlyIntent: false, Engine: "postgres", + Auth: "sql", Username: "monitor", EncryptMode: "Mandatory", TrustServerCertificate: false, + MultiSubnetFailover: false, MonthlyCostUsd: 10m, ModifiedAt: Stamp.AddTicks(7), AwsRoleArn: role, AwsExternalIdSet: idSet); + + private static Edit.ServerEditRow SqlServerRow() => new( + ServerId: 42, Name: "sql-01", Host: "sql-01.example.test", Port: 0, Database: null, ReadOnlyIntent: false, Engine: "sqlserver", + Auth: "sql", Username: "monitor", EncryptMode: "Mandatory", TrustServerCertificate: false, + MultiSubnetFailover: false, MonthlyCostUsd: 10m, ModifiedAt: Stamp.AddTicks(7)); + + private static (Edit.ServerEditPlan? Plan, string? Error) Plan(Edit.ServerEditRow row, string json, AwsRoleAllowlist list) + { + var (changes, error) = Edit.ParseEditChanges(json); + Assert.Null(error); + return Edit.PlanEdit(row, changes!, TestKeyRings.Healthy.Status, list); + } + + [Fact] + public void SettingARole_WithNoList_IsRefusedWithTheNoListSentence() + { + var (plan, error) = Plan(PgRow(), $"{{\"aws_role_arn\":\"{RoleA}\"}}", AwsRoleAllowlist.Empty); + Assert.Null(plan); + Assert.Equal( + "The web and MCP cannot set an AWS role until darling.json lists the roles they may use. Add allowedAwsRoles to darling.json, then restart the service.", + error); + } + + [Fact] + public void SettingARole_NotOnTheList_IsRefusedWithTheNotListedSentence() + { + var (plan, error) = Plan(PgRow(), $"{{\"aws_role_arn\":\"{OtherAccountRole}\"}}", AwsRoleAllowlist.From([RoleA])); + Assert.Null(plan); + Assert.Equal( + "That AWS role is not on the list of roles the web and MCP may set. Add it to allowedAwsRoles in darling.json and restart the service.", + error); + } + + [Fact] + public void SettingARole_OnTheList_ByArnOrByAccountId_IsPlanned() + { + foreach (var entry in new[] { RoleA, "123456789012" }) + { + var (plan, error) = Plan(PgRow(), $"{{\"aws_role_arn\":\" {RoleA} \"}}", AwsRoleAllowlist.From([entry])); + Assert.Null(error); + var set = Assert.Single(plan!.Sets, s => s.Field == "aws_role_arn"); + Assert.Equal("aws_role_arn", set.Column); + Assert.Equal(RoleA, set.Value); + Assert.Contains("aws_role_arn", plan.ChangedFields); + } + } + + [Fact] + public void ABlankOrAbsentRole_KeepsTheStoredRole_AndIsNeverChecked() + { + foreach (var json in new[] { "{\"aws_role_arn\":\" \"}", "{\"aws_role_arn\":\"\"}", "{\"monthly_cost_usd\":11}" }) + { + var (plan, error) = Plan(PgRow(RoleA, idSet: true), json, AwsRoleAllowlist.Empty); + Assert.Null(error); + Assert.DoesNotContain(plan!.Sets, s => s.Column is "aws_role_arn" or "aws_external_id"); + Assert.Equal(RoleA, plan.NewAwsRoleArn); + } + } + + [Fact] + public void ANullRole_RemovesIt_EvenWithNoList_AndLeavesTheExternalIdToTheStore() + { + var (plan, error) = Plan(PgRow(RoleA, idSet: true), "{\"aws_role_arn\":null}", AwsRoleAllowlist.Empty); + Assert.Null(error); + var set = Assert.Single(plan!.Sets, s => s.Column == "aws_role_arn"); + Assert.Null(set.Value); + Assert.DoesNotContain(plan.Sets, s => s.Column == "aws_external_id"); + } + + [Fact] + public void TheSameRoleAsStored_IsNoChange_AndIsNotChecked() + { + var (plan, error) = Plan(PgRow(RoleA), $"{{\"aws_role_arn\":\"{RoleA}\"}}", AwsRoleAllowlist.Empty); + Assert.Null(error); + Assert.Empty(plan!.Sets); + } + + [Fact] + public void ChangingTheRole_WhileAnExternalIdIsStored_NeedsTheIdWithIt() + { + var list = AwsRoleAllowlist.From(["123456789012"]); + var (plan, error) = Plan(PgRow(RoleA, idSet: true), $"{{\"aws_role_arn\":\"{RoleB}\"}}", list); + Assert.Null(plan); + Assert.Equal(AwsRoleSettings.RoleChangeNeedsExternalIdMessage, error); + + var (replaced, replaceError) = Plan(PgRow(RoleA, idSet: true), $"{{\"aws_role_arn\":\"{RoleB}\",\"aws_external_id\":\"{TypedId}\"}}", list); + Assert.Null(replaceError); + Assert.Contains(replaced!.Sets, s => s.Column == "aws_external_id" && (string?)s.Value == TypedId); + + var (cleared, clearError) = Plan(PgRow(RoleA, idSet: true), $"{{\"aws_role_arn\":\"{RoleB}\",\"aws_external_id\":null}}", list); + Assert.Null(clearError); + Assert.Contains(cleared!.Sets, s => s.Column == "aws_external_id" && s.Value is null); + } + + [Fact] + public void AnExternalId_WithNoRole_IsRefused() + { + var (plan, error) = Plan(PgRow(), $"{{\"aws_external_id\":\"{TypedId}\"}}", AwsRoleAllowlist.From(["123456789012"])); + Assert.Null(plan); + Assert.Equal(AwsRoleSettings.ExternalIdNeedsRoleMessage, error); + } + + [Fact] + public void ARole_OnASqlServerTarget_IsRefused() + { + var (plan, error) = Plan(SqlServerRow(), $"{{\"aws_role_arn\":\"{RoleA}\"}}", AwsRoleAllowlist.From([RoleA])); + Assert.Null(plan); + Assert.Equal(AwsRoleSettings.RoleNeedsPostgresMessage, error); + } + + [Fact] + public void AMalformedRoleOrExternalId_IsRefusedByTheParse_WithTheSharedSentence() + { + var (_, roleError) = Edit.ParseEditChanges("{\"aws_role_arn\":\"not-an-arn\"}"); + Assert.Equal(AwsRoleSettings.InvalidRoleMessage, roleError); + var (_, idError) = Edit.ParseEditChanges("{\"aws_external_id\":\"has space\"}"); + Assert.Equal(AwsRoleSettings.InvalidExternalIdMessage, idError); + var (_, typeError) = Edit.ParseEditChanges("{\"aws_role_arn\":5}"); + Assert.Equal("aws_role_arn must be text, or null to remove the role.", typeError); + } + + [Fact] + public void TheCurrentValues_CarryTheRoleAndWhetherAnIdIsSet_NeverTheId() + { + var json = JsonSerializer.Serialize(Edit.CurrentValuesOf(PgRow(RoleA, idSet: true))); + Assert.Contains(RoleA, json, StringComparison.Ordinal); + Assert.Contains("\"aws_external_id_set\":true", json, StringComparison.Ordinal); + Assert.DoesNotContain("\"aws_external_id\"", json, StringComparison.Ordinal); + Assert.Contains("aws_external_id_set", Edit.ReadEditRowSql, StringComparison.Ordinal); + Assert.DoesNotContain("aws_external_id,", Edit.ReadEditRowSql.Replace("aws_external_id_set", "", StringComparison.Ordinal), StringComparison.Ordinal); + } + + [Fact] + public void TheFunctionCall_TakesSeventeenArguments_TheRoleAndIdLast() + { + var parameters = Edit.BuildEditFunctionParameters(41, Stamp, + [ + new Edit.EditColumnValue("aws_role_arn", "aws_role_arn", NpgsqlDbType.Text, RoleA), + new Edit.EditColumnValue("aws_external_id", "aws_external_id", NpgsqlDbType.Text, TypedId), + ]); + Assert.Equal(17, parameters.Count); + Assert.Contains("$17)", Edit.EditFunctionSql, StringComparison.Ordinal); + Assert.Equal(RoleA, parameters[15].Value); + Assert.Equal(TypedId, parameters[16].Value); + } + + [Fact] + public async Task TheStoresExternalIdNeededAnswers_BecomeTheSharedSentences() + { + foreach (var (kind, text) in new[] + { + (Edit.ServerEditWriteKind.ExternalIdNeeded, AwsRoleSettings.RoleChangeNeedsExternalIdMessage), + (Edit.ServerEditWriteKind.ExternalIdNeedsRole, AwsRoleSettings.ExternalIdNeedsRoleMessage), + }) + { + var store = new Store(PgRow(RoleA, idSet: true)) { WriteResult = kind }; + var answer = await Edit.EditServerCoreAsync(store, 41, "{\"monthly_cost_usd\":11}", Reachable, TestKeyRings.Healthy, null, CancellationToken.None); + using var doc = JsonDocument.Parse(answer); + Assert.Equal("invalid", doc.RootElement.GetProperty("status").GetString()); + Assert.Equal(text, doc.RootElement.GetProperty("message").GetString()); + } + } + + [Fact] + public async Task RemovingARole_IsAudited_WithOldAndNewRole_AndTheAnswerNamesBoth() + { + var store = new Store(PgRow(RoleA, idSet: true)); + var logger = new CapturingLogger(); + var answer = await Edit.EditServerCoreAsync(store, 41, "{\"aws_role_arn\":null}", Reachable, TestKeyRings.Healthy, logger, CancellationToken.None); + using var doc = JsonDocument.Parse(answer); + Assert.Equal("updated", doc.RootElement.GetProperty("status").GetString()); + Assert.Equal(RoleA, doc.RootElement.GetProperty("old_aws_role_arn").GetString()); + Assert.Equal(JsonValueKind.Null, doc.RootElement.GetProperty("new_aws_role_arn").ValueKind); + Assert.Contains(logger.Lines, l => l.Contains(RoleA, StringComparison.Ordinal) && l.Contains("(none)", StringComparison.Ordinal) && l.Contains("AWS role changed", StringComparison.Ordinal)); + } + + [Fact] + public async Task ATypedExternalId_NeverReachesAnAnswerOrALogLine() + { + /* The role is the stored one (no list check), the host moves and a password comes with it, so the probe runs; its + error quotes the typed ID. */ + var store = new Store(PgRow(RoleA, idSet: true)); + var logger = new CapturingLogger(); + var body = $"{{\"host\":\"pg-02.example.test\",\"password\":\"pw-secret-1\",\"aws_external_id\":\"{TypedId}\"}}"; + var failing = (Edit.ServerProbe)((_, _) => Task.FromResult( + new ConnectionProbeResult(false, 0, 0, null, false, false, false, false, "refused with " + TypedId))); + var answer = await Edit.EditServerCoreAsync(store, 41, body, failing, TestKeyRings.Healthy, logger, CancellationToken.None); + Assert.DoesNotContain(TypedId, answer, StringComparison.Ordinal); + /* The reply names the host and carries no driver text; the driver's text, with the typed ID removed, is in the log. */ + Assert.Contains("Could not connect to pg-02.example.test", answer, StringComparison.Ordinal); + Assert.Contains(logger.Lines, l => l.Contains("refused with [redacted]", StringComparison.Ordinal)); + + var ok = await Edit.EditServerCoreAsync(store, 41, body, Reachable, TestKeyRings.Healthy, logger, CancellationToken.None); + Assert.DoesNotContain(TypedId, ok, StringComparison.Ordinal); + Assert.All(logger.Lines, l => Assert.DoesNotContain(TypedId, l, StringComparison.Ordinal)); + } + + /* ---------------- the save path of a role-only edit ---------------- */ + + [Fact] + public void ARoleOnlyEdit_ReconnectsUnderTheNewRole_AndNeedsNoProbe() + { + /* The probe opens a PostgreSQL connection and makes no AWS call (DarlingServerConnector.ProbeAsync), so an edit + that moves nothing the connection uses has nothing to probe, the same as a display-name-only edit. What the + role changes is the service's connection: the answer says it reconnects. */ + var list = AwsRoleAllowlist.From([RoleA, RoleB]); + foreach (var (row, body) in new[] + { + (PgRow(), $"{{\"aws_role_arn\":\"{RoleA}\"}}"), + (PgRow(RoleA), $"{{\"aws_role_arn\":\"{RoleB}\"}}"), + (PgRow(RoleA), "{\"aws_role_arn\":null}"), + (PgRow(RoleA, idSet: true), "{\"aws_role_arn\":null}"), + (PgRow(RoleA, idSet: true), $"{{\"aws_role_arn\":\"{RoleB}\",\"aws_external_id\":\"{TypedId}\"}}"), + (PgRow(RoleA, idSet: true), $"{{\"aws_external_id\":\"{TypedId}\"}}"), + (PgRow(RoleA, idSet: true), "{\"aws_external_id\":null}"), + }) + { + var (plan, error) = Plan(row, body, list); + Assert.Null(error); + Assert.True(plan!.Reconnects, body); + Assert.False(plan.NeedsProbe, body); + } + + var (cost, _) = Plan(PgRow(RoleA, idSet: true), "{\"monthly_cost_usd\":11}", list); + Assert.False(cost!.Reconnects); + + var (same, _) = Plan(PgRow(RoleA, idSet: true), $"{{\"aws_role_arn\":\"{RoleA}\",\"monthly_cost_usd\":11}}", list); + Assert.False(same!.Reconnects); + } + + [Fact] + public async Task ARoleRemoval_WritesAndAnswersReconnectsTrue_WithNoProbe() + { + var store = new Store(PgRow(RoleA, idSet: true)); + var probed = 0; + var counting = (Edit.ServerProbe)((_, _) => { probed++; return Task.FromResult(new ConnectionProbeResult(true, 15, 3, "Enterprise", false, false, false, true, null)); }); + var answer = await Edit.EditServerCoreAsync(store, 41, "{\"aws_role_arn\":null}", counting, TestKeyRings.Healthy, null, CancellationToken.None); + using var doc = JsonDocument.Parse(answer); + Assert.Equal("updated", doc.RootElement.GetProperty("status").GetString()); + Assert.True(doc.RootElement.GetProperty("reconnects").GetBoolean()); + Assert.Equal(0, probed); + Assert.Equal(1, store.Writes); + } + + [Fact] + public async Task AConnectionEdit_ThatKeepsTheRole_ProbesWithTheRole_AndAnswersReconnects() + { + MonitoredServer? probedConfig = null; + var recording = (Edit.ServerProbe)((config, _) => + { + probedConfig = config; + return Task.FromResult(new ConnectionProbeResult(true, 15, 3, "Enterprise", false, false, false, true, null)); + }); + var store = new Store(PgRow(RoleA, idSet: true)); + var body = "{\"host\":\"pg-02.example.test\",\"password\":\"pw-secret-1\"}"; + var answer = await Edit.EditServerCoreAsync(store, 41, body, recording, TestKeyRings.Healthy, null, CancellationToken.None); + using var doc = JsonDocument.Parse(answer); + Assert.Equal("updated", doc.RootElement.GetProperty("status").GetString()); + Assert.True(doc.RootElement.GetProperty("reconnects").GetBoolean()); + Assert.True(doc.RootElement.GetProperty("tested").GetBoolean()); + Assert.NotNull(probedConfig); + Assert.Equal(RoleA, probedConfig!.AwsRoleArn); + } + + [Fact] + public async Task ARoleTheListRefuses_StopsTheEdit_BeforeAnyProbeOrWrite() + { + /* Current is Empty in these tests: a role set along with a host move is refused by the list first, so the probe + (which would connect to the new host) never runs and nothing is written. */ + var store = new Store(PgRow()); + var probed = 0; + var counting = (Edit.ServerProbe)((_, _) => { probed++; return Task.FromResult(new ConnectionProbeResult(true, 15, 3, "Enterprise", false, false, false, true, null)); }); + var body = $"{{\"host\":\"pg-02.example.test\",\"password\":\"pw-secret-1\",\"aws_role_arn\":\"{RoleA}\"}}"; + var answer = await Edit.EditServerCoreAsync(store, 41, body, counting, TestKeyRings.Healthy, null, CancellationToken.None); + using var doc = JsonDocument.Parse(answer); + Assert.Equal("invalid", doc.RootElement.GetProperty("status").GetString()); + Assert.Equal(0, probed); + Assert.Equal(0, store.Writes); + } + + [Fact] + public void TheWebAndMcpEdits_ShareOneCore_AndTheSameDefaultProbe() + { + var text = File.ReadAllText(Path.Combine(FindRepoRoot(), "Darling", "PerformanceMonitor.Darling.Service", "Mcp", "DarlingMcpServerAdminTools.Edit.cs")); + var tool = text.IndexOf("public static Task EditServer(", StringComparison.Ordinal); + var byName = text.IndexOf("internal static async Task EditServerByNameAsync", StringComparison.Ordinal); + var byId = text.IndexOf("internal static async Task EditServerByIdAsync", StringComparison.Ordinal); + Assert.True(tool > 0 && byName > tool && byId > byName, "the tool, the by-name body and the web entry exist in order"); + Assert.Contains("DefaultProbeAsync", text[tool..byName], StringComparison.Ordinal); + Assert.Contains("EditServerCoreAsync(", text[byName..byId], StringComparison.Ordinal); + var webBody = text[byId..(byId + 1500)]; + Assert.Contains("EditServerCoreAsync(", webBody, StringComparison.Ordinal); + Assert.Contains("DefaultProbeAsync", webBody, StringComparison.Ordinal); + } + + /* ---------------- add ---------------- */ + + private static string AddJson(string extra) => + "[{\"host\":\"pg-03.example.test\",\"engine\":\"postgres\",\"auth\":\"SQL\",\"username\":\"monitor\",\"password\":\"pw-secret-1\"" + extra + "}]"; + + [Fact] + public void AnAdd_WithARole_NotOnTheList_OrWithNoList_IsInvalid_AndAListedOneIsKept() + { + var key = TestKeyRings.Healthy.Status; + var json = AddJson($",\"aws_role_arn\":\"{RoleA}\",\"aws_external_id\":\"{TypedId}\""); + + var (_, noList, _) = Edit.ParseRequest(json, key, false, AwsRoleAllowlist.Empty); + Assert.Contains(noList, r => r.Status == "invalid" && r.Detail.StartsWith("The web and MCP cannot set an AWS role", StringComparison.Ordinal)); + + var (_, notListed, _) = Edit.ParseRequest(json, key, false, AwsRoleAllowlist.From([OtherAccountRole])); + Assert.Contains(notListed, r => r.Status == "invalid" && r.Detail.StartsWith("That AWS role is not on the list", StringComparison.Ordinal)); + + var (entries, invalid, _) = Edit.ParseRequest(json, key, false, AwsRoleAllowlist.From(["123456789012"])); + Assert.Empty(invalid); + var entry = Assert.Single(entries); + Assert.Equal(RoleA, entry.ProbeConfig.AwsRoleArn); + Assert.Equal(TypedId, entry.ProbeConfig.AwsExternalId); + } + + [Fact] + public void AnAdd_WithNoRole_NeedsNoList_AndTheHostCommandLineIsNotHeldToIt() + { + var key = TestKeyRings.Healthy.Status; + var (plain, plainInvalid, _) = Edit.ParseRequest(AddJson(string.Empty), key, false, AwsRoleAllowlist.Empty); + Assert.Empty(plainInvalid); + Assert.Null(Assert.Single(plain).ProbeConfig.AwsRoleArn); + + var (cli, cliInvalid, _) = Edit.ParseRequest(AddJson($",\"aws_role_arn\":\"{RoleA}\""), key, true, AwsRoleAllowlist.Empty); + Assert.Empty(cliInvalid); + Assert.Equal(RoleA, Assert.Single(cli).ProbeConfig.AwsRoleArn); + } + + [Fact] + public void AnAdd_WithABadPair_OrOnSqlServer_IsInvalid_WithTheSharedSentences() + { + var key = TestKeyRings.Healthy.Status; + var list = AwsRoleAllowlist.From(["123456789012"]); + var (_, idOnly, _) = Edit.ParseRequest(AddJson($",\"aws_external_id\":\"{TypedId}\""), key, false, list); + Assert.Contains(idOnly, r => r.Detail == AwsRoleSettings.ExternalIdNeedsRoleMessage); + + var (_, badRole, _) = Edit.ParseRequest(AddJson(",\"aws_role_arn\":\"nope\""), key, false, list); + Assert.Contains(badRole, r => r.Detail == AwsRoleSettings.InvalidRoleMessage); + + var sqlServer = "[{\"host\":\"sql-09.example.test\",\"auth\":\"SQL\",\"username\":\"u\",\"password\":\"pw-secret-1\",\"aws_role_arn\":\"" + RoleA + "\"}]"; + var (_, onSql, _) = Edit.ParseRequest(sqlServer, key, false, list); + Assert.Contains(onSql, r => r.Detail == AwsRoleSettings.RoleNeedsPostgresMessage); + } + + [Fact] + public void TheInsert_WritesTheRoleAndExternalId_AsTheLastTwoParameters() + { + Assert.Contains("aws_role_arn, aws_external_id)", Edit.InsertServerSql, StringComparison.Ordinal); + Assert.Contains("$17, $18)", Edit.InsertServerSql, StringComparison.Ordinal); + } + + /* ---------------- the form's words ---------------- */ + + [Fact] + public void TheEditForm_UsesTheServicesOwnSentences_WordForWord() + { + var js = File.ReadAllText(Path.Combine(FindRepoRoot(), "Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "pages", "admin.js")); + foreach (var sentence in new[] + { + AwsRoleSettings.InvalidRoleMessage, AwsRoleSettings.InvalidExternalIdMessage, + AwsRoleSettings.ExternalIdNeedsRoleMessage, AwsRoleSettings.RoleChangeNeedsExternalIdMessage, + }) + { + Assert.Contains("\"" + sentence + "\"", js, StringComparison.Ordinal); + } + } + + private static string FindRepoRoot() + { + var dir = new DirectoryInfo(AppContext.BaseDirectory); + while (dir is not null && !File.Exists(Path.Combine(dir.FullName, "PerformanceMonitor.sln"))) + { + dir = dir.Parent; + } + + return dir?.FullName ?? throw new InvalidOperationException("The repository root was not found."); + } + + /* ---------------- stubs ---------------- */ + + private static readonly Edit.ServerProbe Reachable = (_, _) => Task.FromResult( + new ConnectionProbeResult(true, 15, 3, "Enterprise", false, false, false, true, null)); + + private sealed class Store(Edit.ServerEditRow row) : Edit.IServerEditStore + { + public Edit.ServerEditWriteKind WriteResult { get; set; } = Edit.ServerEditWriteKind.Written; + + public int Writes { get; private set; } + + public Task ReadRowAsync(int serverId, CancellationToken cancellationToken) => + Task.FromResult(row); + + public Task> LoadOtherStorageKeysAsync(int serverId, CancellationToken cancellationToken) => + Task.FromResult(new List()); + + public Task WriteAsync( + int serverId, DateTime expectedModifiedAt, IReadOnlyList sets, string? newStorageKey, string? actualStorageKey, CancellationToken cancellationToken) => + Task.FromResult(CountedWrite()); + + private Edit.ServerEditWrite CountedWrite() + { + Writes++; + return new Edit.ServerEditWrite(WriteResult, Stamp.AddSeconds(1)); + } + } + + private sealed class CapturingLogger : ILogger + { + public List Lines { get; } = []; + + public IDisposable? BeginScope(TState state) where TState : notnull => null; + + public bool IsEnabled(LogLevel logLevel) => true; + + public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) => + Lines.Add(formatter(state, exception)); + } +} diff --git a/Darling/Darling.Tests/AwsRoleMessageTextTests.cs b/Darling/Darling.Tests/AwsRoleMessageTextTests.cs new file mode 100644 index 0000000000..fc7cb8e164 --- /dev/null +++ b/Darling/Darling.Tests/AwsRoleMessageTextTests.cs @@ -0,0 +1,248 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.IO; +using System.Text.RegularExpressions; +using PerformanceMonitor.Darling.Service.Targets; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5452: the four texts an operator reads on collection health when a server's AWS role cannot be used. Each says what +/// happened, names the role ARN, says what to change, and never carries the external ID. +/// +public sealed class AwsRoleMessageTextTests +{ + private const string Role = "arn:aws:iam::123456789012:role/darling-monitor"; + private const string Sentinel = "ext-7Hq2mZ9vLx"; + + private static AwsRoleAssumeException Make(AwsRoleAssumeKind kind, string? externalId) => kind switch + { + AwsRoleAssumeKind.NotAllowed => AwsRoleAssumeException.ForNotAllowed(new AwsRoleKey(Role, externalId), "us-east-1"), + AwsRoleAssumeKind.PartitionMismatch => AwsRoleAssumeException.ForPartitionMismatch(new AwsRoleKey(Role, externalId), "cn-north-1", "aws", "aws-cn"), + AwsRoleAssumeKind.Denied => AwsRoleAssumeException.ForStsFailure( + new AwsRoleKey(Role, externalId), "us-east-1", FakeSts.Error("AccessDenied", $"denied (ExternalId={externalId})")), + AwsRoleAssumeKind.RegionDisabled => AwsRoleAssumeException.ForStsFailure( + new AwsRoleKey(Role, externalId), "af-south-1", FakeSts.Error("RegionDisabledException", $"off (ExternalId={externalId})")), + _ => throw new ArgumentOutOfRangeException(nameof(kind)), + }; + + /* What each message must tell the operator to change, as a phrase the text has to hold. */ + public static TheoryData Changes => new() + { + { AwsRoleAssumeKind.NotAllowed, ["allowedAwsRoles in darling.json", "restart the service", "clear the role"] }, + { AwsRoleAssumeKind.PartitionMismatch, ["partition", "Set a role from the target's partition", "clear the role", "host name"] }, + { AwsRoleAssumeKind.Denied, ["trust policy", "external ID", "sts:AssumeRole", "does not exist"] }, + { AwsRoleAssumeKind.RegionDisabled, ["Enable that region", "af-south-1", "set a role in a region that is enabled", "host name"] }, + }; + + [Theory] + [MemberData(nameof(Changes))] + public void EachRoleMessage_SaysWhatHappened_NamesTheRole_AndSaysWhatToChange(AwsRoleAssumeKind kind, string[] phrases) + { + var message = Make(kind, Sentinel).Message; + + Assert.Contains(Role, message, StringComparison.Ordinal); + Assert.Contains("Nothing was read this cycle", message, StringComparison.Ordinal); + foreach (var phrase in phrases) + { + Assert.Contains(phrase, message, StringComparison.Ordinal); + } + + Assert.DoesNotContain(Sentinel, message, StringComparison.OrdinalIgnoreCase); + } + + [Theory] + [MemberData(nameof(Changes))] + public void EachRoleMessage_IsAnOrdinaryPlainSentence_NotARawSdkDump(AwsRoleAssumeKind kind, string[] phrases) + { + _ = phrases; + var message = Make(kind, Sentinel).Message; + + Assert.DoesNotContain("Amazon.", message, StringComparison.Ordinal); + Assert.DoesNotContain("Exception", message, StringComparison.Ordinal); + Assert.False(message.Contains('\n', StringComparison.Ordinal), "one line, as the collection health column shows it"); + } + + [Fact] + public void TheDeniedMessage_SaysWhetherAnExternalIdWasSent_AndNeverWhichOne() + { + var with = Make(AwsRoleAssumeKind.Denied, Sentinel).Message; + var without = Make(AwsRoleAssumeKind.Denied, null).Message; + + Assert.Contains("(with the external ID set on this server)", with, StringComparison.Ordinal); + Assert.Contains("(without an external ID)", without, StringComparison.Ordinal); + Assert.DoesNotContain(Sentinel, with, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public void ANoSourceIdentityMessage_NamesTheRoleAndWhatToChange_AndHoldsNoneOfTheSdksOwnText() + { + var raw = new InvalidOperationException(@"Profile corp-sso not found in C:\Users\svc\.aws\credentials; IMDS at 169.254.169.254 refused"); + + var ex = AwsRoleAssumeException.ForNoSourceIdentity(new AwsRoleKey(Role, Sentinel), "us-east-1", raw); + + Assert.Contains(Role, ex.Message, StringComparison.Ordinal); + Assert.Contains("Nothing was read this cycle", ex.Message, StringComparison.Ordinal); + Assert.Contains("restart the service", ex.Message, StringComparison.Ordinal); + Assert.Contains("SDK exception System.InvalidOperationException", ex.Message, StringComparison.Ordinal); + foreach (var text in new[] { "corp-sso", @"C:\Users", ".aws", "169.254", "IMDS" }) + { + Assert.DoesNotContain(text, ex.Message, StringComparison.Ordinal); + Assert.DoesNotContain(text, ex.ToString(), StringComparison.Ordinal); + } + + Assert.Contains("corp-sso", ex.SourceDetail, StringComparison.Ordinal); + } + + [Fact] + public void ATransientMessage_NamesTheRole_AndHoldsNoneOfTheSdksOwnText() + { + var raw = FakeSts.Error("Throttling", @"Rate exceeded for C:\ProgramData\aws\config on host alpha-07"); + + var ex = AwsRoleAssumeException.ForStsFailure(new AwsRoleKey(Role, Sentinel), "us-east-1", raw); + + Assert.Equal(AwsRoleAssumeKind.Transient, ex.Kind); + Assert.Contains(Role, ex.Message, StringComparison.Ordinal); + Assert.Contains("AWS error code Throttling", ex.Message, StringComparison.Ordinal); + Assert.DoesNotContain("ProgramData", ex.Message, StringComparison.Ordinal); + Assert.DoesNotContain("alpha-07", ex.Message, StringComparison.Ordinal); + Assert.Contains("alpha-07", ex.SourceDetail, StringComparison.Ordinal); + } +} + +/// +/// #5452: Test Connection in the desktop viewer. The viewer sends a test_connect command; the service answers it with +/// DarlingServerConnector.ProbeAsync, which opens a PostgreSQL connection and reads facts from it. Neither the viewer +/// nor that path makes an AWS call, so the server's role changes nothing about the test. These pins keep it so: a test that +/// began to call AWS would have to carry the role and the external ID, and be held to the allowed-roles list. +/// +public sealed class AwsRoleTestConnectionTests +{ + private static string Repo() + { + var dir = new DirectoryInfo(AppContext.BaseDirectory); + while (dir is not null && !File.Exists(Path.Combine(dir.FullName, "PerformanceMonitor.sln"))) + { + dir = dir.Parent; + } + + return dir?.FullName ?? throw new InvalidOperationException("The repository root was not found."); + } + + private static string Read(params string[] parts) => File.ReadAllText(Path.Combine(Repo(), Path.Combine(parts))); + + [Fact] + public void TheServiceSideOfTestConnection_MakesNoAwsCall() + { + var service = Path.Combine("Darling", "PerformanceMonitor.Darling.Service"); + foreach (var file in new[] { "DarlingServerConnector.cs", "DarlingCommandExecutor.cs" }) + { + var text = Read(service, file); + Assert.DoesNotMatch(new Regex(@"^\s*using\s+Amazon", RegexOptions.Multiline), text); + foreach (var name in new[] { "AwsRoleClients", "AwsRoleCredentialCache", "AssumeRole", "AmazonRDSClient", "AmazonPIClient", "AwsRoleKey", "RdsEndpointVerifier" }) + { + Assert.DoesNotContain(name, text, StringComparison.Ordinal); + } + } + } + + [Fact] + public void TheViewer_HasNoAwsSdk_SoItCannotCallAwsWithAnyCredentials() + { + var viewer = Path.Combine(Repo(), "Darling", "PerformanceMonitor.Darling.Viewer"); + var project = File.ReadAllText(Path.Combine(viewer, "PerformanceMonitor.Darling.Viewer.csproj")); + Assert.DoesNotContain("AWSSDK", project, StringComparison.Ordinal); + foreach (var file in Directory.EnumerateFiles(viewer, "*.cs", SearchOption.AllDirectories)) + { + if (file.Contains($"{Path.DirectorySeparatorChar}obj{Path.DirectorySeparatorChar}", StringComparison.Ordinal) + || file.Contains($"{Path.DirectorySeparatorChar}bin{Path.DirectorySeparatorChar}", StringComparison.Ordinal)) + { + continue; + } + + Assert.DoesNotMatch(new Regex(@"^\s*using\s+Amazon", RegexOptions.Multiline), File.ReadAllText(file)); + } + } + + [Fact] + public void TheProbe_ReadsNoRoleFromTheServerItTests() + { + var text = Read("Darling", "PerformanceMonitor.Darling.Service", "DarlingServerConnector.cs"); + var probe = text.IndexOf("public static async Task ProbeAsync", StringComparison.Ordinal); + Assert.True(probe > 0); + var end = text.IndexOf("public static string DescribeProbeFacts", probe, StringComparison.Ordinal); + Assert.DoesNotContain("AwsRole", text[probe..end], StringComparison.Ordinal); + Assert.DoesNotContain("AwsExternalId", text[probe..end], StringComparison.Ordinal); + } +} + +/// The role messages in the runbook's failure table read exactly as the service writes them (#5452). +public sealed class AwsRoleRunbookMessagesTests +{ + private const string Role = "arn:aws:iam::123456789012:role/darling-monitor"; + + private static string Runbook() + { + var dir = new DirectoryInfo(AppContext.BaseDirectory); + while (dir is not null && !File.Exists(Path.Combine(dir.FullName, "PerformanceMonitor.sln"))) + { + dir = dir.Parent; + } + + return File.ReadAllText(Path.Combine( + dir?.FullName ?? throw new InvalidOperationException("The repository root was not found."), "docs", "postgres-first-target-runbook.md")); + } + + [Fact] + public void EachRoleMessage_IsQuotedInTheRunbookExactlyAsTheServiceWritesIt() + { + var runbook = Runbook(); + var withId = new AwsRoleKey(Role, "ext-7Hq2mZ9vLx"); + var messages = new[] + { + AwsRoleAssumeException.ForNotAllowed(withId, "us-east-1").Message, + AwsRoleAssumeException.ForPartitionMismatch(withId, "", "aws-cn", "aws").Message, + AwsRoleAssumeException.ForStsFailure(withId, "", FakeSts.Error("RegionDisabledException", "off")).Message, + AwsRoleAssumeException.ForStsFailure(withId, "us-east-1", FakeSts.Error("AccessDenied", "denied")).Message, + AwsRoleAssumeException.ForNoSourceIdentity(withId, "us-east-1", null).Message, + AwsRoleAssumeException.ForStsFailure(withId, "us-east-1", FakeSts.Error("Throttling", "slow")).Message, + }; + + foreach (var message in messages) + { + /* The table writes the role as , and the parts that vary (a code, the SDK note) as placeholders. */ + var quoted = message.Replace(Role, "", StringComparison.Ordinal); + quoted = Regex.Replace(quoted, @" \((?:SDK exception [^)]*|AWS error code [^)]*)\)\z", string.Empty); + Assert.Contains(quoted, runbook, StringComparison.Ordinal); + } + + /* The host-credentials message holds the AWS error code between its two quoted parts. */ + var rejected = AwsRoleAssumeException.ForStsFailure(withId, "us-east-1", FakeSts.Error("ExpiredToken", "old")).Message + .Replace(Role, "", StringComparison.Ordinal); + var split = rejected.IndexOf(" (ExpiredToken)", StringComparison.Ordinal); + Assert.True(split > 0); + Assert.Contains(rejected[..split], runbook, StringComparison.Ordinal); + Assert.Contains(rejected[(split + " (ExpiredToken)".Length)..], runbook, StringComparison.Ordinal); + } + + [Fact] + public void TheRunbooksSampleExternalId_IsOneTheExternalIdCheckRefuses() + { + var runbook = Runbook(); + var samples = Regex.Matches(runbook, "\"(?:awsExternalId|sts:ExternalId)\": \"([^\"]*)\""); + + Assert.NotEmpty(samples); + foreach (Match sample in samples) + { + Assert.NotNull(PerformanceMonitor.Darling.Storage.AwsRoleSettings.ValidateExternalId(sample.Groups[1].Value)); + } + } +} diff --git a/Darling/Darling.Tests/AwsRoleSettingsTests.cs b/Darling/Darling.Tests/AwsRoleSettingsTests.cs new file mode 100644 index 0000000000..fae1d92616 --- /dev/null +++ b/Darling/Darling.Tests/AwsRoleSettingsTests.cs @@ -0,0 +1,189 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// The per-server AWS role rules (#5452): is the one validator every surface that takes a +/// role ARN and an external ID shares, so its answers and its sentences are pinned here. +/// +public sealed class AwsRoleSettingsTests +{ + [Theory] + [InlineData(null, null)] + [InlineData("", null)] + [InlineData(" ", null)] + [InlineData("\t\n", null)] + [InlineData(" arn:aws:iam::123456789012:role/darling-monitor ", "arn:aws:iam::123456789012:role/darling-monitor")] + public void Normalize_TrimsAndTurnsBlankIntoNull(string? input, string? expected) + { + Assert.Equal(expected, AwsRoleSettings.Normalize(input)); + } + + [Theory] + [InlineData("arn:aws:iam::123456789012:role/darling-monitor")] + [InlineData("arn:aws:iam::123456789012:role/a")] + [InlineData("arn:aws-cn:iam::123456789012:role/darling-monitor")] + [InlineData("arn:aws-us-gov:iam::123456789012:role/darling-monitor")] + [InlineData("arn:aws:iam::123456789012:role/monitoring/darling-monitor")] + [InlineData("arn:aws:iam::123456789012:role/a/b/c/darling-monitor")] + [InlineData("arn:aws:iam::123456789012:role/name_with+=,.@-chars")] + [InlineData("arn:aws:iam::000000000000:role/darling")] + public void ValidateRole_AcceptsAnIamRoleArn(string role) + { + Assert.Null(AwsRoleSettings.ValidateRole(role)); + } + + [Fact] + public void ValidateRole_AcceptsAnUnsetRole() + { + Assert.Null(AwsRoleSettings.ValidateRole(null)); + } + + [Theory] + [InlineData("darling-monitor")] + [InlineData("arn:aws:iam::123456789012:user/darling-monitor")] + [InlineData("arn:aws:iam::123456789012:role/")] + [InlineData("arn:aws:iam::12345678901:role/darling-monitor")] + [InlineData("arn:aws:iam::1234567890123:role/darling-monitor")] + [InlineData("arn:aws:iam::12345678901a:role/darling-monitor")] + [InlineData("arn:aws:s3::123456789012:role/darling-monitor")] + [InlineData("arn:azure:iam::123456789012:role/darling-monitor")] + [InlineData("arn:aws:iam::123456789012:role/has space")] + [InlineData("arn:aws:iam::123456789012:role/monitoring//darling-monitor")] + [InlineData("arn:aws:iam::123456789012:role//darling-monitor")] + [InlineData("arn:aws:iam::123456789012:role/darling-monitor/")] + [InlineData("arn:aws:iam::123456789012:role/darlingé")] + [InlineData("ARN:aws:iam::123456789012:role/darling-monitor")] + [InlineData("arn:aws:iam::123456789012:role/darling-monitor\nextra")] + [InlineData("arn:aws:iam::123456789012:role//x")] + [InlineData("arn:aws:iam::123456789012:role/path/darling")] + [InlineData("arn:aws:iam::123456789012:role/path>/darling")] + [InlineData("arn:aws:iam::123456789012:role/pa\"th/darling")] + [InlineData("arn:aws:iam::123456789012:role/pa'th/darling")] + [InlineData("arn:aws:iam::123456789012:role/pa&th/darling")] + [InlineData("arn:aws:iam::123456789012:role/pa`th/darling")] + [InlineData("arn:aws:iam::123456789012:role/pa AwsRoleSettings.RoleArnMaxLength); + Assert.Equal(AwsRoleSettings.InvalidRoleMessage, AwsRoleSettings.ValidateRole(role)); + } + + [Theory] + [InlineData("ab")] + [InlineData("0123456789")] + [InlineData("tenant-1234:prod/us_east+1=x,y.z@corp")] + [InlineData("A-b_C+d=e,f.g@h:i/j-k")] + public void ValidateExternalId_AcceptsAwsCharactersAtTwoToTwelveTwentyFour(string externalId) + { + Assert.Null(AwsRoleSettings.ValidateExternalId(externalId)); + } + + [Fact] + public void ValidateExternalId_AcceptsAnUnsetOneAndTheBounds() + { + Assert.Null(AwsRoleSettings.ValidateExternalId(null)); + Assert.Null(AwsRoleSettings.ValidateExternalId(new string('x', AwsRoleSettings.ExternalIdMaxLength))); + Assert.Equal( + AwsRoleSettings.InvalidExternalIdMessage, + AwsRoleSettings.ValidateExternalId(new string('x', AwsRoleSettings.ExternalIdMaxLength + 1))); + } + + [Theory] + [InlineData("x")] + [InlineData("has space")] + [InlineData("tab\there")] + [InlineData("semi;colon")] + [InlineData("quote'quote")] + [InlineData("café-id")] + [InlineData("new\nline")] + [InlineData("star*")] + public void ValidateExternalId_RefusesTooShortAndAnyOtherCharacter(string externalId) + { + Assert.Equal(AwsRoleSettings.InvalidExternalIdMessage, AwsRoleSettings.ValidateExternalId(externalId)); + } + + [Fact] + public void ValidatePair_ReportsFormatFirst_ThenAnExternalIdWithNoRole() + { + const string role = "arn:aws:iam::123456789012:role/darling-monitor"; + + Assert.Null(AwsRoleSettings.ValidatePair(null, null)); + Assert.Null(AwsRoleSettings.ValidatePair(role, null)); + Assert.Null(AwsRoleSettings.ValidatePair(role, "tenant-1")); + + Assert.Equal(AwsRoleSettings.ExternalIdNeedsRoleMessage, AwsRoleSettings.ValidatePair(null, "tenant-1")); + Assert.Equal(AwsRoleSettings.InvalidRoleMessage, AwsRoleSettings.ValidatePair("nope", "tenant-1")); + Assert.Equal(AwsRoleSettings.InvalidExternalIdMessage, AwsRoleSettings.ValidatePair(role, "x")); + } + + [Theory] + /* old, new, stored, sent, needs */ + [InlineData(null, "arn:aws:iam::123456789012:role/b", true, false, true)] + [InlineData("arn:aws:iam::123456789012:role/a", "arn:aws:iam::123456789012:role/b", true, false, true)] + [InlineData("arn:aws:iam::123456789012:role/a", "arn:aws:iam::123456789012:role/b", true, true, false)] + [InlineData("arn:aws:iam::123456789012:role/a", "arn:aws:iam::123456789012:role/b", false, false, false)] + [InlineData("arn:aws:iam::123456789012:role/a", "arn:aws:iam::123456789012:role/a", true, false, false)] + [InlineData("arn:aws:iam::123456789012:role/a", "arn:aws:iam::123456789012:role/A", true, false, true)] + [InlineData("arn:aws:iam::123456789012:role/a", null, true, false, false)] + [InlineData(null, null, true, false, false)] + public void RoleChangeNeedsExternalId_OnlyForANewRoleOverAStoredIdThatWasNotSent( + string? oldRole, string? newRole, bool stored, bool sent, bool expected) + { + Assert.Equal(expected, AwsRoleSettings.RoleChangeNeedsExternalId(oldRole, newRole, stored, sent)); + } + + [Fact] + public void TheSentences_ArePinned_AndNameNoRealAccount() + { + Assert.Equal( + "The AWS role must be an IAM role ARN, such as arn:aws:iam::123456789012:role/darling-monitor.", + AwsRoleSettings.InvalidRoleMessage); + Assert.Equal( + "The external ID must be 2 to 1224 characters: letters, digits and _ + = , . @ : / - with no spaces.", + AwsRoleSettings.InvalidExternalIdMessage); + Assert.Equal( + "An external ID needs an AWS role ARN. Set the role, or clear the external ID.", + AwsRoleSettings.ExternalIdNeedsRoleMessage); + Assert.Equal( + "An AWS role applies only to a PostgreSQL target on Amazon RDS or Aurora.", + AwsRoleSettings.RoleNeedsPostgresMessage); + Assert.Equal( + "Changing the AWS role needs the external ID with it: send the external ID again, or clear it.", + AwsRoleSettings.RoleChangeNeedsExternalIdMessage); + } +} diff --git a/Darling/Darling.Tests/AwsStsTestSupport.cs b/Darling/Darling.Tests/AwsStsTestSupport.cs new file mode 100644 index 0000000000..4185c37cb8 --- /dev/null +++ b/Darling/Darling.Tests/AwsStsTestSupport.cs @@ -0,0 +1,163 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Reflection; +using System.Threading; +using System.Threading.Tasks; +using Amazon; +using Amazon.Runtime; +using Amazon.SecurityToken; +using Amazon.SecurityToken.Model; +using Microsoft.Extensions.Logging; + +namespace Darling.Tests; + +/// +/// A fake STS for the assumed-role credential tests (#5452): no AWS, no network. It answers +/// AssumeRoleAsync through a handler and records every request, the region it was built for and the credentials +/// it was given. Every other member of the interface is unsupported, which fails a test loudly if the code under test +/// starts calling something else. +/// +public class FakeStsProxy : DispatchProxy +{ + internal Func> Handler { get; set; } = null!; + + internal List Requests { get; set; } = null!; + + protected override object? Invoke(MethodInfo? targetMethod, object?[]? args) + { + switch (targetMethod?.Name) + { + case "AssumeRoleAsync": + var request = (AssumeRoleRequest)args![0]!; + lock (Requests) + { + Requests.Add(request); + } + + return Handler(request, (CancellationToken)args[1]!); + case "Dispose": + return null; + default: + throw new NotSupportedException("FakeSts does not implement " + targetMethod?.Name); + } + } +} + +/// Builds clients and the hooks the credentials class takes. +internal sealed class FakeSts +{ + private readonly object _gate = new(); + + public FakeSts(Func>? handler = null) + { + Handler = handler ?? ((_, _) => Task.FromResult(Grant())); + } + + public Func> Handler { get; set; } + + public List Requests { get; } = new(); + + public List Regions { get; } = new(); + + public List SourceCredentialsSeen { get; } = new(); + + public int Calls + { + get + { + lock (Requests) + { + return Requests.Count; + } + } + } + + public Func Factory => (region, credentials) => + { + lock (_gate) + { + Regions.Add(region.SystemName); + SourceCredentialsSeen.Add(credentials.GetCredentials()); + } + + var proxy = DispatchProxy.Create(); + var fake = (FakeStsProxy)(object)proxy; + fake.Handler = (request, token) => Handler(request, token); + fake.Requests = Requests; + return proxy; + }; + + /// A source of credentials that always answers with the same example keys. + public static Func> Source { get; } = + _ => Task.FromResult(new BasicAWSCredentials("AKIAEXAMPLESOURCE", "example-source-secret")); + + /// An STS response with example credentials. is deliberately settable: the code must not read it. + public static AssumeRoleResponse Grant(string token = "example-session-token", DateTime? expiration = null) => new() + { + Credentials = new Credentials + { + AccessKeyId = "ASIAEXAMPLEASSUMED", + SecretAccessKey = "example-assumed-secret", + SessionToken = token, + Expiration = expiration, + }, + }; + + /// An STS error with an AWS error code. + public static AmazonSecurityTokenServiceException Error(string code, string message) => new(message) + { + ErrorCode = code, + }; +} + +/// A clock that moves only when a test moves it. +internal sealed class ManualTimeProvider : TimeProvider +{ + private DateTimeOffset _now; + + public ManualTimeProvider(DateTimeOffset? start = null) + { + _now = start ?? new DateTimeOffset(2026, 10, 7, 12, 0, 0, TimeSpan.Zero); + } + + public override DateTimeOffset GetUtcNow() => _now; + + public void Advance(TimeSpan by) => _now += by; +} + +/// A logger that keeps every formatted line and its exception. +internal sealed class CapturingAwsLogger : ILogger +{ + public List Lines { get; } = new(); + + public IDisposable? BeginScope(TState state) where TState : notnull => null; + + public bool IsEnabled(LogLevel logLevel) => true; + + public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) + { + lock (Lines) + { + Lines.Add($"{logLevel}: {formatter(state, exception)}" + (exception is null ? string.Empty : " | " + exception)); + } + } + + public string All + { + get + { + lock (Lines) + { + return string.Join("\n", Lines); + } + } + } +} diff --git a/Darling/Darling.Tests/AzureFallbackMessageTests.cs b/Darling/Darling.Tests/AzureFallbackMessageTests.cs new file mode 100644 index 0000000000..c47021c204 --- /dev/null +++ b/Darling/Darling.Tests/AzureFallbackMessageTests.cs @@ -0,0 +1,71 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5498: when master cannot be read and the registration names no database, the thrown text puts the firewall first. +/// Error 40615 means the Azure firewall did not allow the client's IP address (a server-level block causes it as +/// readily as a database-only rule), so the text names that and the firewall rule as the fix. Setting a database is +/// only the fix for the master-only case. The 3.10 Azure release test hit this with a server-level rule removed and +/// was told to set a database. The fallback behavior itself (throw when there is nothing to fall back to, use the +/// named database otherwise) is unchanged and pinned here beside the wording. +/// +public sealed class AzureFallbackMessageTests +{ + private static DarlingCollectorRunner Runner() + => new(NpgsqlDataSource.Create("Host=localhost;Database=unused"), new CollectorDeltaCalculator(), NullLogger.Instance); + + private static ServerRuntime Server() => new() + { + Config = new MonitoredServer { Name = "AzureTest1007", Host = "example.database.windows.net" }, + ConnectionString = "Server=tcp:example.database.windows.net,1433;Initial Catalog=master;Encrypt=True", + Target = new CollectorTargetInfo { IsAzureSqlDb = true }, + StorageName = "example.database.windows.net", + ServerId = 5498, + }; + + [Theory] + [InlineData("master")] + [InlineData("")] + [InlineData(null)] + public void NoDatabaseToFallBackTo_NamesTheFirewallFirst(string? targetDb) + { + var ex = Assert.Throws( + () => Runner().FallbackDatabaseList(Server(), targetDb, reason: "master DB inaccessible (SQL error 40615)")); + + var message = ex.Message; + Assert.StartsWith("master DB inaccessible (SQL error 40615), and this connection has no target database", message, StringComparison.Ordinal); + + var firewall = message.IndexOf("firewall did not allow this client's IP address", StringComparison.Ordinal); + var rule = message.IndexOf("add a firewall rule", StringComparison.Ordinal); + var setDatabase = message.IndexOf("set a database for 'AzureTest1007'", StringComparison.Ordinal); + + Assert.True(firewall > 0, message); + Assert.True(rule > firewall, message); + Assert.Contains("either a server-level rule or a database-level rule in master", message, StringComparison.Ordinal); + Assert.True(setDatabase > rule, "Setting a database comes after the firewall, as the master-only case's fix."); + Assert.Contains("can open a user database but not master", message, StringComparison.Ordinal); + } + + [Fact] + public void ANamedDatabaseIsStillTheFallback() + { + var databases = Runner().FallbackDatabaseList(Server(), "AdventureWorks", reason: "master DB inaccessible (SQL error 40615)"); + + Assert.Equal(new List { "AdventureWorks" }, databases); + } +} diff --git a/Darling/Darling.Tests/AzureForeignStatePruneTests.cs b/Darling/Darling.Tests/AzureForeignStatePruneTests.cs index 5c0c33b460..8cc2cc760c 100644 --- a/Darling/Darling.Tests/AzureForeignStatePruneTests.cs +++ b/Darling/Darling.Tests/AzureForeignStatePruneTests.cs @@ -138,7 +138,10 @@ their orphaned rows wholesale — a dropped-database prune has nothing left to o k => k.Prefix == QueryStoreOpenIntervalState.WatermarkKeyPrefix); /* Nothing may sit outside both lists — a server-scoped key added to PrunableKeys by reflex would be - deleted every cycle. */ - Assert.Empty(QueryStorePerDatabaseState.NotKeyedByDatabase); + deleted every cycle. The one member is the drop record of the long-query session older versions shared + between installs (#4961): its server-scope key is the bare prefix, which a prune that rebuilds keys as + prefix + database name would delete every cycle. A new member is a decision, so the list is pinned + whole rather than checked for containment. */ + Assert.Equal(new[] { LegacyLongQuerySession.StateKeyPrefix }, QueryStorePerDatabaseState.NotKeyedByDatabase); } } diff --git a/Darling/Darling.Tests/AzureMasterAnalysisScopeLiveTests.cs b/Darling/Darling.Tests/AzureMasterAnalysisScopeLiveTests.cs index d64deb6c6d..60766b52a8 100644 --- a/Darling/Darling.Tests/AzureMasterAnalysisScopeLiveTests.cs +++ b/Darling/Darling.Tests/AzureMasterAnalysisScopeLiveTests.cs @@ -69,7 +69,7 @@ INSERT INTO servers (server_id, server_name, display_name, is_enabled, sql_major VALUES ($1, $2, $2, TRUE, 16, now()::timestamp, now()::timestamp) ON CONFLICT (server_id) DO UPDATE SET is_enabled = TRUE", ct, ServerId, ServerName); - var end = new DateTime(DateTime.UtcNow.Ticks - (DateTime.UtcNow.Ticks % TimeSpan.TicksPerMinute), DateTimeKind.Unspecified).AddMinutes(-1); + var end = new DateTime(DateTime.UtcNow.Ticks / TimeSpan.TicksPerMinute * TimeSpan.TicksPerMinute, DateTimeKind.Unspecified).AddMinutes(-1); var start = end.AddHours(-4); await Exec(connection, "INSERT INTO wait_stats (collection_id, collection_time, server_id, server_name, wait_type, delta_waiting_tasks, delta_wait_time_ms, delta_signal_wait_time_ms) VALUES ($1,$2,$3,$4,'OLD',1,1,0)", ct, CollectionIdGenerator.Next(), end.AddDays(-3), ServerId, ServerName); @@ -231,7 +231,7 @@ await Exec(connection, @" INSERT INTO servers (server_id, server_name, display_name, is_enabled, sql_major_version, created_date, modified_date) VALUES ($1, $2, $2, TRUE, 16, now()::timestamp, now()::timestamp) ON CONFLICT (server_id) DO UPDATE SET is_enabled = TRUE", ct, ServerId, ServerName); - var end = new DateTime(DateTime.UtcNow.Ticks - (DateTime.UtcNow.Ticks % TimeSpan.TicksPerMinute), DateTimeKind.Unspecified).AddMinutes(-1); + var end = new DateTime(DateTime.UtcNow.Ticks / TimeSpan.TicksPerMinute * TimeSpan.TicksPerMinute, DateTimeKind.Unspecified).AddMinutes(-1); var start = end.AddHours(-4); for (var m = 0; m <= 240; m += 15) await Exec(connection, "INSERT INTO wait_stats (collection_id, collection_time, server_id, server_name, wait_type, delta_waiting_tasks, delta_wait_time_ms, delta_signal_wait_time_ms) VALUES ($1,$2,$3,$4,'CXPACKET',1,10,0)", diff --git a/Darling/Darling.Tests/AzureSiblingGrowthTests.cs b/Darling/Darling.Tests/AzureSiblingGrowthTests.cs index e047c8106e..89194dd5dd 100644 --- a/Darling/Darling.Tests/AzureSiblingGrowthTests.cs +++ b/Darling/Darling.Tests/AzureSiblingGrowthTests.cs @@ -61,18 +61,32 @@ private static string CteBody(string sql, string name) } [Fact] - public void ViewerStorageGrowth_LeavesTheOldShapeRowsOutOfTheLatestAnd7dAnd30dSums() + public void ViewerStorageGrowth_LeavesTheOldShapeRowsOutOfTheLatestSum_AndEverySiblingRowOutOfThe7dAnd30dSums() { var sql = ViewerDataService.StorageGrowthSql; - foreach (var sum in new[] { "latest", "past_7d", "past_30d" }) + Assert.True( + Squash(CteBody(sql, "latest")).Contains(AzureSiblingDatabaseSize.ExcludePreFixRows, StringComparison.Ordinal), + "The latest sum does not leave the old-shape sibling rows out."); + + // #5498: the newest side holds data plus log per database now, so a past side that kept the old data-only + // sibling rows would read the log as growth. + foreach (var sum in new[] { "past_7d", "past_30d" }) { Assert.True( - Squash(CteBody(sql, sum)).Contains(AzureSiblingDatabaseSize.ExcludePreFixRows, StringComparison.Ordinal), - $"The {sum} sum does not leave the old-shape sibling rows out."); + Squash(CteBody(sql, sum)).Contains(AzureSiblingDatabaseSize.ExcludeAllSiblingRows, StringComparison.Ordinal), + $"The {sum} sum does not leave every sibling row out."); } } + [Fact] + public void ExcludeAllSiblingRows_IsTheRowTestWithoutTheUsedSpaceCondition() + { + Assert.Equal( + "NOT (file_id IS NULL AND COALESCE(file_name, '') = '(whole database)')", + AzureSiblingDatabaseSize.ExcludeAllSiblingRows); + } + /// /// A sibling's size is data space only, so the latest CTE flags a database that has the one row another database /// gets, with a bool_or of the shared row test. Lite's AzureSiblingGrowthTests run the twin query on a @@ -114,10 +128,10 @@ public void ViewerStorageGrowth_ReadsTheTwoFlagsLast_SoNoOtherColumnMoves() Squash(ViewerDataService.StorageGrowthSql), StringComparison.Ordinal); - var source = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.FinOps.Storage.cs"); - Assert.Contains("GrowthPct30d = reader.IsDBNull(7) ? null : Convert.ToDecimal(reader.GetValue(7)),", source, StringComparison.Ordinal); - Assert.Contains("HasSiblingRow = !reader.IsDBNull(8) && reader.GetBoolean(8)", source, StringComparison.Ordinal); - Assert.Contains("HasLogServiceFile = !reader.IsDBNull(9) && reader.GetBoolean(9)", source, StringComparison.Ordinal); + var source = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "FinOps", "DarlingFinOpsStorageGrowthReader.cs"); + Assert.Contains("GrowthPct30d: reader.IsDBNull(7) ? null : Convert.ToDecimal(reader.GetValue(7)),", source, StringComparison.Ordinal); + Assert.Contains("HasSiblingRow: !reader.IsDBNull(8) && reader.GetBoolean(8)", source, StringComparison.Ordinal); + Assert.Contains("HasLogServiceFile: !reader.IsDBNull(9) && reader.GetBoolean(9)", source, StringComparison.Ordinal); } /// diff --git a/Darling/Darling.Tests/AzureSqlDatabaseHardwareTests.cs b/Darling/Darling.Tests/AzureSqlDatabaseHardwareTests.cs index 1c9e5fb464..122e79ff9a 100644 --- a/Darling/Darling.Tests/AzureSqlDatabaseHardwareTests.cs +++ b/Darling/Darling.Tests/AzureSqlDatabaseHardwareTests.cs @@ -166,7 +166,7 @@ public void WebStatRenderer_DropsTilesByTheirCondition_OnEdition5_AndKeepsEveryT Process proc; try { proc = Process.Start(psi)!; } - catch (Win32Exception) { return; } // Node is not installed; the source pins above still hold the change in place. + catch (Win32Exception) { Assert.Skip("Node is not installed, so the shipped page script cannot be run."); throw; } // The source pins above still hold the change in place. using (proc) { @@ -235,18 +235,21 @@ public void FinOpsUtilizationCard_AsksTheSharedRule_ForTheWordsAroundItsMemoryFi { var tab = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "FinOpsTab.Loaders.cs"); var xaml = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "FinOpsTab.xaml"); - var read = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.FinOps.Utilization.cs"); + var read = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "FinOps", "DarlingFinOpsUtilizationReader.cs"); + var figures = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "FinOps", "FinOpsUtilizationFigures.cs"); - Assert.Contains("ServerHardwareScope.HardwareIsTheHosts(data.EngineEdition)", tab, StringComparison.Ordinal); + Assert.Contains("ServerHardwareScope.HardwareIsTheHosts(d.EngineEdition)", figures, StringComparison.Ordinal); + Assert.Contains("FinOpsUtilizationFigures.Explanation(", tab, StringComparison.Ordinal); Assert.Contains("FinOpsPhysicalMemoryCaption.Text = ServerHardwareScope.PhysicalMemoryCaption(data.EngineEdition);", tab, StringComparison.Ordinal); Assert.Contains("x:Name=\"FinOpsPhysicalMemoryCaption\"", xaml, StringComparison.Ordinal); - Assert.Contains("ServerHardwareScope.OverProvisionedExplanation(", tab, StringComparison.Ordinal); - Assert.Contains("ServerHardwareScope.RightSizedExplanation(", tab, StringComparison.Ordinal); + Assert.Contains("ServerHardwareScope.OverProvisionedExplanation(", figures, StringComparison.Ordinal); + Assert.Contains("ServerHardwareScope.RightSizedExplanation(", figures, StringComparison.Ordinal); Assert.DoesNotContain("of physical RAM", tab, StringComparison.Ordinal); + Assert.DoesNotContain("of physical RAM", figures, StringComparison.Ordinal); /* The CPU count is resolved through the edition (AzureSqlDatabaseHostMathTests pins the CASE): off edition 5 it is still COALESCE(vcore_count, cpu_count), and the edition still rides along for the card. */ Assert.Contains("ELSE COALESCE(vcore_count, cpu_count) END AS cpu_count, engine_edition", read, StringComparison.Ordinal); - Assert.Contains("EngineEdition = reader.IsDBNull(16)", read, StringComparison.Ordinal); + Assert.Contains("EngineEdition: reader.IsDBNull(16)", read, StringComparison.Ordinal); } [Fact] diff --git a/Darling/Darling.Tests/AzureSqlDatabaseHostMathTests.cs b/Darling/Darling.Tests/AzureSqlDatabaseHostMathTests.cs index bb0fc0d318..68ee30bb21 100644 --- a/Darling/Darling.Tests/AzureSqlDatabaseHostMathTests.cs +++ b/Darling/Darling.Tests/AzureSqlDatabaseHostMathTests.cs @@ -30,6 +30,7 @@ namespace Darling.Tests; /// (editions 1 to 4) and Managed Instance (8) behave exactly as before, and every test has that twin. Lite.Tests pins the /// same table for the other app, in the same words. /// +[Trait("Reads", "Lite")] public sealed class AzureSqlDatabaseHostMathTests { private static readonly DateTime s_start = new(2026, 9, 30, 0, 0, 0, DateTimeKind.Utc); diff --git a/Darling/Darling.Tests/AzureSqlDatabaseMemoryScopeTests.cs b/Darling/Darling.Tests/AzureSqlDatabaseMemoryScopeTests.cs index 7fb112600a..a4032554b4 100644 --- a/Darling/Darling.Tests/AzureSqlDatabaseMemoryScopeTests.cs +++ b/Darling/Darling.Tests/AzureSqlDatabaseMemoryScopeTests.cs @@ -239,20 +239,32 @@ public void MemoryRecommendation_NeverDividesByServerPropertiesMemory_SoItCannot "{percent} of {n}GB RAM". Its divisor is util.PhysicalMemoryMb, read from memory_stats through the utilization read, and the rules file reads no physical-memory column of its own. */ var rules = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.FinOps.Recommendations.cs"); + var reader = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "FinOps", "DarlingFinOpsRecommendationsReader.cs"); - Assert.Contains("of {util.PhysicalMemoryMb / 1024}GB RAM", rules, StringComparison.Ordinal); + var figures = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "FinOps", "FinOpsRecommendationFigures.cs"); + + Assert.Contains("of {util.PhysicalMemoryMb / 1024}GB RAM", figures, StringComparison.Ordinal); Assert.False( Regex.IsMatch(rules, @"physical_memory_mb"), "the recommendation rules must not read a physical-memory column of their own"); + Assert.False( + Regex.IsMatch(reader, @"physical_memory_mb"), + "the recommendation reader must not read a physical-memory column of its own"); + Assert.False( + Regex.IsMatch(figures, @"physical_memory_mb"), + "the recommendation figures must not read a physical-memory column of their own"); } [Fact] public void MemoryAndVmRules_OnAzureSqlDatabase_SayWhyTheyStandDown_AndDoNotCallTheMemoryTheHosts() { - var rules = Flatten(ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.FinOps.Recommendations.cs")); + var rules = Flatten(ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "FinOps", "DarlingFinOpsRecommendationsReader.cs")); + var viewerRules = Flatten(ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.FinOps.Recommendations.cs")); Assert.DoesNotContain("reports the HOST's memory", rules, StringComparison.Ordinal); Assert.DoesNotContain("its memory figure is the host's", rules, StringComparison.Ordinal); + Assert.DoesNotContain("reports the HOST's memory", viewerRules, StringComparison.Ordinal); + Assert.DoesNotContain("its memory figure is the host's", viewerRules, StringComparison.Ordinal); Assert.Contains("its memory comes with its service objective and cannot be resized on its own", rules, StringComparison.Ordinal); Assert.Contains("its cores and memory come with its service objective", rules, StringComparison.Ordinal); } diff --git a/Darling/Darling.Tests/AzureSqlDatabaseOwnFiguresTests.cs b/Darling/Darling.Tests/AzureSqlDatabaseOwnFiguresTests.cs index 897466a590..e82f0cdfec 100644 --- a/Darling/Darling.Tests/AzureSqlDatabaseOwnFiguresTests.cs +++ b/Darling/Darling.Tests/AzureSqlDatabaseOwnFiguresTests.cs @@ -15,6 +15,7 @@ using PerformanceMonitor.Darling.Analysis; using PerformanceMonitor.Darling.Service.Mcp; using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Storage.FinOps; using PerformanceMonitor.Darling.Viewer; using PerformanceMonitor.PlanAnalysis; using Xunit; @@ -488,20 +489,23 @@ public void FinOpsTab_AsksTheSharedRules_ForTheCpuUnit_TheWorkerThreadsCard_TheH Assert.DoesNotContain("$\"{data.CurrentWorkersCount:N0} / {data.MaxWorkersCount:N0}\"", tab, StringComparison.Ordinal); Assert.Contains( - "FinOpsHealthScoreBorder.ToolTip = data.HasCpuSample ? null : ServerHardwareScope.HealthScoreWithoutCpuNote;", + "FinOpsHealthScoreBorder.ToolTip = data.HasCpuSample ? null : FinOpsHealthCalculator.NoScoreNote;", tab, StringComparison.Ordinal); - Assert.Contains( - "int? cpuScore = item.AvgCpuPct is decimal avgCpu ? FinOpsHealthCalculator.CpuScore(avgCpu) : null;", - tab, StringComparison.Ordinal); + // The inventory score is the fleet read's, the Utilization view's own rule: no CPU sample is a dash, never a default. + Assert.Contains("item.HealthScore = row.HealthScore;", tab, StringComparison.Ordinal); Assert.DoesNotContain("item.AvgCpuPct ?? 0m", tab, StringComparison.Ordinal); + + var reader = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "FinOps", "DarlingFinOpsInventoryReader.cs"); + Assert.Contains("return FinOpsUtilizationFigures.HealthScore(", reader, StringComparison.Ordinal); + Assert.Contains("if (reader.IsDBNull(1) || reader.IsDBNull(16)) return null;", reader, StringComparison.Ordinal); } [Fact] public void WorkerReads_KeepANullInUseCountNull_InAllThreePlaces() { - var utilization = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.FinOps.Utilization.cs"); - var inventory = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.FinOps.Inventory.cs"); + var utilization = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "FinOps", "DarlingFinOpsUtilizationReader.cs"); + var inventory = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "FinOps", "DarlingFinOpsInventoryReader.cs"); /* Point-in-time read, 7-day trend and fleet read: the in-use count is read as NULL, not coalesced to 0. */ Assert.Contains("int? currentWorkers = reader.IsDBNull(10) ? null : Convert.ToInt32(reader.GetValue(10));", utilization, StringComparison.Ordinal); diff --git a/Darling/Darling.Tests/BaselineDiscontinuityRenderTests.cs b/Darling/Darling.Tests/BaselineDiscontinuityRenderTests.cs index 9db21827a6..b5be6f7d20 100644 --- a/Darling/Darling.Tests/BaselineDiscontinuityRenderTests.cs +++ b/Darling/Darling.Tests/BaselineDiscontinuityRenderTests.cs @@ -35,7 +35,7 @@ public sealed class BaselineDiscontinuityRenderTests { "get_memory_trend", "get_perfmon_trend", "get_file_io_trend", "get_query_trend", "get_query_duration_trend", "get_procedure_duration_trend", "get_query_store_duration_trend", - "get_wait_trend", + "get_wait_trend", "get_server_trend", }; private static (string Name, MethodInfo Method)[] AllDarlingTools() => typeof(DarlingMcpTrendTools).Assembly.GetTypes() @@ -76,7 +76,8 @@ public void TheWaitTrend_LivesInDataTools_AndTheSevenInTrendTools() var byType = AllDarlingTools().ToDictionary(t => t.Name, t => t.Method.DeclaringType!, StringComparer.Ordinal); Assert.Equal(typeof(DarlingMcpDataTools), byType["get_wait_trend"]); - foreach (var name in Roster.Where(n => n != "get_wait_trend")) + Assert.Equal(typeof(DarlingMcpServerTrendTools), byType["get_server_trend"]); + foreach (var name in Roster.Where(n => n != "get_wait_trend" && n != "get_server_trend")) { Assert.Equal(typeof(DarlingMcpTrendTools), byType[name]); } diff --git a/Darling/Darling.Tests/BaselineSupplyTests.cs b/Darling/Darling.Tests/BaselineSupplyTests.cs index eb28edf8ee..f48f6b3c7a 100644 --- a/Darling/Darling.Tests/BaselineSupplyTests.cs +++ b/Darling/Darling.Tests/BaselineSupplyTests.cs @@ -514,14 +514,19 @@ public void Successors_TookTheLegacyPositions_SoThePhaseGridDidNotMove() Assert.Equal(TimescaleSupport.WaitStatsIntervalBaselineView, TimescaleSupport.BaselineAggregates[1].View); /* The order's count is the hourly registry plus the seven baselines — 6 + 7 = 13 since #3653's LC - freeze (was 9 + 7 = 16 during Q12) — and the baseline pair contributes exactly its two positions to - it, no more. */ + freeze (was 9 + 7 = 16 during Q12, and + 8 + 7 = 15 at #5329) — and the baseline pair contributes exactly its two positions to it, no more. */ Assert.Equal(TimescaleSupport.HourlyAggregates.Length + 7, TimescaleSupport.HourlyRefreshPhaseOrder.Count); - Assert.Equal(13, TimescaleSupport.HourlyRefreshPhaseOrder.Count); - Assert.Equal(3, TimescaleSupport.RefreshPhaseMinutesFor(TimescaleSupport.PerfmonIntervalBaselineView)); - Assert.Equal(5, TimescaleSupport.RefreshPhaseMinutesFor(TimescaleSupport.WaitStatsIntervalBaselineView)); - Assert.Equal(15, TimescaleSupport.HeaviestRefreshStartMinute); - Assert.Equal(1050, TimescaleSupport.RefreshSlotWarningSeconds); + /* #5329: 8 + 7 = 15 now that the two io hourlies joined the registry, so the grid re-derives: 14 light + members spread over a 13-minute band, the heaviest refresh at :17 with a 19-minute window, so the watch + line is 950 s. The light band is re-laid (the unbounded-cardinality members sit a guard's width apart in + it, and the band now has 14 positions), which moves the baseline pair from minutes 3 and 5 to 5 and 6; the minute map stays + injective (TimescaleSupportTests pins that and carries the derivation). */ + Assert.Equal(15, TimescaleSupport.HourlyRefreshPhaseOrder.Count); + Assert.Equal(5, TimescaleSupport.RefreshPhaseMinutesFor(TimescaleSupport.PerfmonIntervalBaselineView)); + Assert.Equal(6, TimescaleSupport.RefreshPhaseMinutesFor(TimescaleSupport.WaitStatsIntervalBaselineView)); + Assert.Equal(17, TimescaleSupport.HeaviestRefreshStartMinute); + Assert.Equal(950, TimescaleSupport.RefreshSlotWarningSeconds); } /// diff --git a/Darling/Darling.Tests/BenefitScorerWiringTests.cs b/Darling/Darling.Tests/BenefitScorerWiringTests.cs index 8cecc35769..6a302a4ca0 100644 --- a/Darling/Darling.Tests/BenefitScorerWiringTests.cs +++ b/Darling/Darling.Tests/BenefitScorerWiringTests.cs @@ -22,6 +22,8 @@ namespace Darling.Tests; /// is now the one place that runs then , /// and every entry point that used to call the analyzer directly calls the pipeline instead. /// +[Trait("Stage", "Guard")] +[Trait("Reads", "Lite")] public sealed class BenefitScorerWiringTests { /// diff --git a/Darling/Darling.Tests/BlockingDatabaseFilterLiveTests.cs b/Darling/Darling.Tests/BlockingDatabaseFilterLiveTests.cs new file mode 100644 index 0000000000..313cf5a592 --- /dev/null +++ b/Darling/Darling.Tests/BlockingDatabaseFilterLiveTests.cs @@ -0,0 +1,362 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Alerting; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5244 (PR3, lane R1): the blocking reads take a SET of databases. get_blocking and its reader filter BOTH arms (the XE +/// reports and the DMV fallback), get_blocked_process_xml filters the XE-with-XML read, and get_blocking_trend filters +/// both arms of the per-minute trend. A call with [A, B] over a seed of A, B and C returns only A and B rows, and the cap is the +/// top N of the CHOSEN databases (the newest row is C's, so a cap taken before the filter would be spent on it). The one-name +/// consumers on the empty and miss paths say "for the chosen databases" instead of giving a verdict about a database the read +/// never looked at. +/// +[Collection("live-postgres")] +public sealed class BlockingDatabaseFilterLiveTests +{ + private const string ServerName = "darling-blocking-db-filter-e2e"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + + private const string SkipReason = + "Set DARLING_TEST_PG to a Postgres connection string to run the live blocking database-filter test (it mints its own scratch database)."; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + [Fact] + public async Task BlockedProcessReportsReader_TwoNames_ReturnsOnlyThoseDatabases_OnBothArms_AndCapsAfterTheFilter() + { + await RunAsync(async (postgres, now, ct) => + { + var start = now.AddHours(-1); + + var all = await DarlingBlockingReader.GetRecentBlockedProcessReportsAsync(postgres, ServerId, start, now, 50, DatabaseFilter.All, ct); + Assert.Equal(new[] { "A", "B", "C", "D" }, all.Select(r => r.DatabaseName).Distinct().OrderBy(n => n, StringComparer.Ordinal).ToArray()); + Assert.Equal(7, all.Count); + + var chosen = await DarlingBlockingReader.GetRecentBlockedProcessReportsAsync(postgres, ServerId, start, now, 50, DatabaseFilter.Of(["A", "B"]), ct); + Assert.Equal(new[] { "A", "B" }, chosen.Select(r => r.DatabaseName).Distinct().OrderBy(n => n, StringComparer.Ordinal).ToArray()); + /* One XE row and one DMV row per chosen database: both arms were filtered. */ + Assert.Equal(4, chosen.Count); + Assert.Equal(2, chosen.Count(r => r.Source != BlockedProcessAlertRow.DmvSnapshotSource)); + + /* The overload's one-name form is the same predicate. */ + var one = await DarlingBlockingReader.GetRecentBlockedProcessReportsAsync(postgres, ServerId, start, now, 50, DatabaseFilter.One("B"), ct); + Assert.All(one, r => Assert.Equal("B", r.DatabaseName)); + Assert.Equal(2, one.Count); + + /* The cap runs over the chosen databases: C holds the newest XE row, so the unfiltered top row is C's. */ + var topAll = await DarlingBlockingReader.GetRecentBlockedProcessReportsAsync(postgres, ServerId, start, now, 1, DatabaseFilter.All, ct); + Assert.Equal("C", topAll.Single().DatabaseName); + var topChosen = await DarlingBlockingReader.GetRecentBlockedProcessReportsAsync(postgres, ServerId, start, now, 1, DatabaseFilter.Of(["A", "B"]), ct); + Assert.Equal("A", topChosen.Single().DatabaseName); + + /* The public-shaped overload (no filter) is still every database. */ + var bare = await DarlingBlockingReader.GetRecentBlockedProcessReportsAsync(postgres, ServerId, start, now, 50, ct); + Assert.Equal(7, bare.Count); + }); + } + + [Fact] + public async Task BlockedProcessReportsWithXmlReader_TwoNames_ReturnsOnlyThoseDatabases_AndCapsAfterTheFilter() + { + await RunAsync(async (postgres, now, ct) => + { + var start = now.AddHours(-1); + + var all = await DarlingBlockingReader.GetRecentBlockedProcessReportsWithXmlAsync(postgres, ServerId, start, now, 50, DatabaseFilter.All, ct); + Assert.Equal(3, all.Count); + + var chosen = await DarlingBlockingReader.GetRecentBlockedProcessReportsWithXmlAsync(postgres, ServerId, start, now, 50, DatabaseFilter.Of(["A", "B"]), ct); + Assert.Equal(new[] { "A", "B" }, chosen.Select(r => r.DatabaseName).OrderBy(n => n, StringComparer.Ordinal).ToArray()); + + var top = await DarlingBlockingReader.GetRecentBlockedProcessReportsWithXmlAsync(postgres, ServerId, start, now, 1, DatabaseFilter.Of(["A", "B"]), ct); + Assert.Equal("A", top.Single().DatabaseName); + }); + } + + [Fact] + public async Task BlockingTrendReader_TwoNames_CountsOnlyThoseDatabases_OnBothArms() + { + await RunAsync(async (postgres, now, ct) => + { + var start = now.AddHours(-1); + + var all = await DarlingBlockingTrendReader.GetBlockingTrendAsync(postgres, ServerId, start, now, DatabaseFilter.All, ct); + /* XE has A, B and C: the DMV arm stays out because the XE source has rows. */ + Assert.Equal(3, all.Sum(p => p.Count)); + + var chosen = await DarlingBlockingTrendReader.GetBlockingTrendAsync(postgres, ServerId, start, now, DatabaseFilter.Of(["A", "B"]), ct); + Assert.Equal(2, chosen.Sum(p => p.Count)); + + /* D has DMV-captured blocking only. The fallback asks whether XE has rows for the CHOSEN databases, so D charts. */ + var dmvOnly = await DarlingBlockingTrendReader.GetBlockingTrendAsync(postgres, ServerId, start, now, DatabaseFilter.One("D"), ct); + Assert.Equal(1, dmvOnly.Sum(p => p.Count)); + + /* A set that includes a database with XE rows keeps the XE arm alone, as an unfiltered read does. */ + var mixed = await DarlingBlockingTrendReader.GetBlockingTrendAsync(postgres, ServerId, start, now, DatabaseFilter.Of(["A", "D"]), ct); + Assert.Equal(1, mixed.Sum(p => p.Count)); + + var none = await DarlingBlockingTrendReader.GetBlockingTrendAsync(postgres, ServerId, start, now, DatabaseFilter.One("Nope"), ct); + Assert.Empty(none); + + var bare = await DarlingBlockingTrendReader.GetBlockingTrendAsync(postgres, ServerId, start, now, ct); + Assert.Equal(3, bare.Sum(p => p.Count)); + }); + } + + [Fact] + public async Task GetBlocking_TwoNames_ReturnsOnlyThoseDatabases_AndTheEmptyAndMissAnswersNameTheChosenDatabases() + { + await RunAsync(async (postgres, now, ct) => + { + using (var doc = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlocking( + postgres, ServerName, 1, 50, null, false, null, DatabaseFilter.Of(["A", "B"]), 150, cancellationToken: ct))) + { + var events = doc.RootElement.GetProperty("events").EnumerateArray().ToList(); + Assert.Equal(new[] { "A", "B" }, events.Select(e => e.GetProperty("database_name").GetString()!).Distinct().OrderBy(n => n, StringComparer.Ordinal).ToArray()); + Assert.Equal(4, events.Count); + Assert.False(doc.RootElement.GetProperty("truncated").GetBoolean()); + /* #5244 L3: the page says it was filtered, in the words the other filtered tools echo. */ + Assert.Equal("the chosen databases", doc.RootElement.GetProperty("database_name").GetString()); + } + + /* The page is the newest `limit` of the chosen databases: limit 2 of A, B leaves the other two out and says so. */ + using (var doc = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlocking( + postgres, ServerName, 1, 2, null, false, null, DatabaseFilter.Of(["A", "B"]), 150, cancellationToken: ct))) + { + Assert.Equal(2, doc.RootElement.GetProperty("events").GetArrayLength()); + Assert.True(doc.RootElement.GetProperty("truncated").GetBoolean()); + Assert.DoesNotContain(doc.RootElement.GetProperty("events").EnumerateArray(), e => e.GetProperty("database_name").GetString() == "C"); + } + + /* Empty: the status word stays, and the sentence is about the chosen databases only. */ + var emptyOne = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlocking( + postgres, ServerName, 1, 50, null, false, null, DatabaseFilter.One("Nope"), 150, cancellationToken: ct)).RootElement; + Assert.Equal("empty", emptyOne.GetProperty("status").GetString()); + Assert.Contains("for the database Nope", emptyOne.GetProperty("message").GetString(), StringComparison.Ordinal); + + var emptyMany = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlocking( + postgres, ServerName, 1, 50, null, false, null, DatabaseFilter.Of(["Nope", "Gone"]), 150, cancellationToken: ct)).RootElement; + Assert.Equal("empty", emptyMany.GetProperty("status").GetString()); + Assert.Contains("for the chosen databases", emptyMany.GetProperty("message").GetString(), StringComparison.Ordinal); + Assert.DoesNotContain("Nope", emptyMany.GetProperty("message").GetString(), StringComparison.Ordinal); + + /* Unfiltered, the sentence is the one it always was. */ + var emptyAll = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlocking( + postgres, ServerName, 1, 50, null, false, now.AddDays(-30).ToString("o"), DatabaseFilter.All, 150, cancellationToken: ct)).RootElement; + Assert.Equal("empty", emptyAll.GetProperty("status").GetString()); + Assert.Equal("No blocking events found in the specified time range.", emptyAll.GetProperty("message").GetString()); + + /* A dedup_key from A's incident, asked over B alone: a miss that says the scan was limited to the chosen databases. */ + string aKey; + using (var doc = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlocking( + postgres, ServerName, 1, 50, null, false, null, DatabaseFilter.One("A"), 150, cancellationToken: ct))) + { + aKey = doc.RootElement.GetProperty("events").EnumerateArray().First().GetProperty("dedup_key").GetString()!; + } + + using (var hit = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlocking( + postgres, ServerName, 1, 50, aKey, false, null, DatabaseFilter.One("A"), 150, cancellationToken: ct))) + { + Assert.True(hit.RootElement.GetProperty("events").GetArrayLength() >= 1); + } + + var miss = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlocking( + postgres, ServerName, 1, 50, aKey, false, null, DatabaseFilter.One("B"), 150, cancellationToken: ct)).RootElement; + Assert.Equal("empty", miss.GetProperty("status").GetString()); + Assert.Contains("The scan was limited to the events for the database B", miss.GetProperty("message").GetString(), StringComparison.Ordinal); + }); + } + + [Fact] + public async Task GetBlockedProcessXml_TwoNames_ReturnsOnlyThoseDatabases_AndTheEmptyAnswerNamesTheChosenDatabases() + { + await RunAsync(async (postgres, now, ct) => + { + using (var doc = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlockedProcessXml( + postgres, ServerName, 1, 50, null, DatabaseFilter.Of(["A", "B"]), cancellationToken: ct))) + { + var reports = doc.RootElement.GetProperty("reports").EnumerateArray().ToList(); + Assert.Equal(new[] { "A", "B" }, reports.Select(e => e.GetProperty("database_name").GetString()!).OrderBy(n => n, StringComparer.Ordinal).ToArray()); + Assert.Equal("the chosen databases", doc.RootElement.GetProperty("database_name").GetString()); + } + + using (var doc = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlockedProcessXml( + postgres, ServerName, 1, 1, null, DatabaseFilter.Of(["A", "B"]), cancellationToken: ct))) + { + Assert.Equal("A", doc.RootElement.GetProperty("reports")[0].GetProperty("database_name").GetString()); + Assert.True(doc.RootElement.GetProperty("truncated").GetBoolean()); + } + + var empty = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlockedProcessXml( + postgres, ServerName, 1, 50, null, DatabaseFilter.Of(["Nope", "Gone"]), cancellationToken: ct)).RootElement; + Assert.Equal("empty", empty.GetProperty("status").GetString()); + Assert.Contains("for the chosen databases", empty.GetProperty("message").GetString(), StringComparison.Ordinal); + + var emptyOne = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlockedProcessXml( + postgres, ServerName, 1, 50, null, DatabaseFilter.One("Nope"), cancellationToken: ct)).RootElement; + Assert.Contains("for the database Nope", emptyOne.GetProperty("message").GetString(), StringComparison.Ordinal); + + var emptyAll = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlockedProcessXml( + postgres, ServerName, 1, 50, now.AddDays(-30).ToString("o"), DatabaseFilter.All, cancellationToken: ct)).RootElement; + Assert.Equal("No blocked process report XML available in the specified time range.", emptyAll.GetProperty("message").GetString()); + }); + } + + [Fact] + public async Task GetBlockingTrend_TwoNames_CountsOnlyThoseDatabases_AndTheEmptyAnswerNamesTheChosenDatabases() + { + await RunAsync(async (postgres, now, ct) => + { + using (var doc = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlockingTrend( + postgres, ServerName, 1, null, DatabaseFilter.Of(["A", "B"]), ct))) + { + Assert.Equal(2, doc.RootElement.GetProperty("trend").EnumerateArray().Sum(p => p.GetProperty("count").GetInt32())); + Assert.Equal("the chosen databases", doc.RootElement.GetProperty("database_name").GetString()); + /* N1: the exact tag, not just "some tag": a tool that hard-codes one source fails here. XE has rows for A and B. */ + Assert.Equal("blocked-process-report", doc.RootElement.GetProperty("source").GetString()); + } + + /* The collectors ran (seeded below) and none of the chosen databases' blocking was in the captures: an all-clear + for THOSE databases, worded that way, with the status word unchanged. */ + var one = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlockingTrend( + postgres, ServerName, 1, null, DatabaseFilter.One("Nope"), ct)).RootElement; + Assert.Equal("empty", one.GetProperty("status").GetString()); + Assert.Equal(JsonValueKind.Null, one.GetProperty("source").ValueKind); + Assert.Contains("for the database Nope in the last 1 hour(s)", one.GetProperty("message").GetString(), StringComparison.Ordinal); + + var many = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlockingTrend( + postgres, ServerName, 1, null, DatabaseFilter.Of(["Nope", "Gone"]), ct)).RootElement; + Assert.Equal("empty", many.GetProperty("status").GetString()); + Assert.Contains("for the chosen databases in the last 1 hour(s)", many.GetProperty("message").GetString(), StringComparison.Ordinal); + + /* A window the collectors never ran in stays "unavailable" whatever the selection (no capture is no all-clear). */ + var gap = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlockingTrend( + postgres, ServerName, 1, now.AddDays(-20).ToString("o"), DatabaseFilter.One("Nope"), ct)).RootElement; + Assert.Equal("unavailable", gap.GetProperty("status").GetString()); + }); + } + + /// The database predicate is the list form on every statement the three reads route to (no live store needed): the XE reads at + /// $6 (after the window floor at $5), the DMV fallback at $5, and BOTH arms of the trend at $5 (after the floor at $4), and + /// none of them is spliced with a name. + [Fact] + public void EveryBlockingStatement_CarriesTheListPredicate_OnEveryArm() + { + var xe = DatabaseFilter.All.Clause("database_name", 6); + Assert.Contains(xe, DarlingBlockingReader.BlockedProcessReportsSql, StringComparison.Ordinal); + Assert.Contains(xe, DarlingBlockingReader.BlockedProcessReportsWithXmlSql, StringComparison.Ordinal); + Assert.Contains(DatabaseFilter.All.Clause("database_name", 5), DarlingBlockingReader.DmvBlockingSnapshotsSql, StringComparison.Ordinal); + + var trend = DatabaseFilter.All.Clause("database_name", 5); + var first = DarlingBlockingTrendReader.BlockingTrendSql.IndexOf(trend, StringComparison.Ordinal); + Assert.True(first >= 0, "the XE arm of the trend"); + Assert.True(DarlingBlockingTrendReader.BlockingTrendSql.IndexOf(trend, first + trend.Length, StringComparison.Ordinal) > first, "the DMV arm of the trend"); + } + + /// Seeds A, B and C, then runs the body. XE rows (all with a report XML), newest first: C at -5 min, A at -10, B at -11. + /// DMV rows, with spid pairs no XE row covers: one each for A, B and C, and a D whose only blocking is DMV-captured. + private static async Task RunAsync(Func body) + { + var baseConnectionString = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), SkipReason); + var ct = TestContext.Current.CancellationToken; + + // #1776 own-store: a scratch database, so no other class's chunks shape the store under test (#4650). + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var cs = scratch.ConnectionString; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PrepareStoreAsync(cs, connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs); + + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + var now = DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow); + + var xe = new (string Database, int Minutes, int Spid)[] { ("C", 5, 103), ("A", 10, 101), ("B", 11, 102) }; + foreach (var (database, minutes, spid) in xe) + { + var eventTime = DarlingMcpTestData.Naive(now.AddMinutes(-minutes)); + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO blocked_process_reports + (blocked_report_id, collection_time, server_id, server_name, event_time, database_name, + blocked_spid, blocked_ecid, blocking_spid, blocking_ecid, wait_time_ms, lock_mode, + blocked_sql_text, blocking_sql_text, blocked_process_report_xml) +VALUES ($1,$2,$3,$4,$5,$6,$7,0,$8,0,1000,'X','SELECT 1','UPDATE t SET c = 1','')", + CollectionIdGenerator.Next(), eventTime.AddSeconds(30), ServerId, ServerName, eventTime, database, spid, 900 + spid); + } + + var dmv = new (string Database, int Minutes, int Spid)[] { ("C", 20, 303), ("A", 21, 301), ("B", 22, 302), ("D", 23, 304) }; + foreach (var (database, minutes, spid) in dmv) + { + var at = DarlingMcpTestData.Naive(now.AddMinutes(-minutes)); + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO dmv_blocking_snapshots + (collection_id, collection_time, server_id, server_name, event_time, database_name, blocked_spid, blocking_spid, wait_time_ms, blocking_status) +VALUES ($1,$2,$3,$4,$2,$5,$6,60,12000,'suspended')", + CollectionIdGenerator.Next(), at, ServerId, ServerName, database, spid); + } + + /* Collector runs that looked at the last hour, so an empty trend is an all-clear and not a gap. */ + foreach (var collector in new[] { "blocked_process_report", "dmv_blocking_snapshot" }) + { + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO collection_log + (log_id, server_id, server_name, collector_name, collection_time, + duration_ms, status, error_message, rows_collected, sql_duration_ms, duckdb_duration_ms) +VALUES ($1, $2, $3, $4, $5, 100, 'SUCCESS', NULL, 0, 80, 20)", + CollectionIdGenerator.Next(), ServerId, ServerName, collector, DarlingMcpTestData.Naive(now.AddMinutes(-30))); + } + + await body(postgres, now, ct); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + private static async Task PrepareStoreAsync(string connectionString, NpgsqlConnection connection, CancellationToken ct) + { + await PgMigrations.MigrateAsync(connection, ct); + if (await LiveTimescaleProbe.TryEnableAsync(connectionString, ct)) + { + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + await using var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection); + await stop.ExecuteNonQueryAsync(ct); + } + } + + private static async Task DeleteRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + using var cleanup = new NpgsqlCommand( + $"DELETE FROM blocked_process_reports WHERE server_id = {ServerId}; DELETE FROM dmv_blocking_snapshots WHERE server_id = {ServerId}; DELETE FROM collection_log WHERE server_id = {ServerId}; DELETE FROM servers WHERE server_id = {ServerId};", + connection); + await cleanup.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/BlockingListPlanFlagsLivePostgresTests.cs b/Darling/Darling.Tests/BlockingListPlanFlagsLivePostgresTests.cs new file mode 100644 index 0000000000..af9373826c --- /dev/null +++ b/Darling/Darling.Tests/BlockingListPlanFlagsLivePostgresTests.cs @@ -0,0 +1,237 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.Json; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5236: the plan-presence flags on get_blocking and get_deadlocks come from one keyed statement over the PAGE's +/// rows, not from the list read, so a compressed chunk decompresses a plan column only for the batches holding the page's own +/// rows. The list statements must stay free of every plan column (their planner output names none), and the flags must still be +/// right for a page whose rows carry a plan on both sides, one side, an empty string, or NULL, and must not leak onto rows +/// past the page. +/// +[Collection("live-postgres")] +public sealed class BlockingListPlanFlagsLivePostgresTests +{ + private const string ServerName = "darling-list-plan-flags-e2e"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + + private const string Plan = ""; + + private const string SkipReason = + "Set DARLING_TEST_PG to a Postgres connection string to run the live list plan flag test (it mints its own scratch database)."; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + [Fact] + public async Task GetBlocking_FlagsThePagesRows_ByTheirOwnStoredPlans() + { + var baseConnectionString = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), SkipReason); + var ct = TestContext.Current.CancellationToken; + + // #1776 own-store: a scratch database, so no other class's chunks shape the store under test (#4650). + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var cs = scratch.ConnectionString; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PrepareStoreAsync(cs, connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs); + + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + /* The tool reads its "last N hours" window from the real clock, which the test clock override cannot move (#5608). */ + var now = DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow); + + /* Newest first by event_time: spid 101 (both plans), 102 (blocked only), 103 (blocking only), 104 (empty strings), + 105 (NULLs), 106 (both plans; the oldest, so a page of five leaves it out). */ + var seeded = new (int Spid, string? Blocked, string? Blocking)[] + { + (101, Plan, Plan), (102, Plan, null), (103, null, Plan), (104, "", ""), (105, null, null), (106, Plan, Plan), + }; + for (var i = 0; i < seeded.Length; i++) + { + var eventTime = now.AddMinutes(-10 - i); + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO blocked_process_reports + (blocked_report_id, collection_time, server_id, server_name, event_time, database_name, + blocked_spid, blocked_ecid, blocking_spid, blocking_ecid, wait_time_ms, lock_mode, + blocked_sql_text, blocking_sql_text, blocked_process_report_xml, blocked_query_plan_xml, blocking_query_plan_xml) +VALUES ($1,$2,$3,$4,$5,'PlanFlagsDb',$6,0,$7,0,1000,'X','SELECT 1','UPDATE t SET c = 1','',$8,$9)", + CollectionIdGenerator.Next(), eventTime.AddMinutes(1), ServerId, ServerName, eventTime, + seeded[i].Spid, 900 + i, (object?)seeded[i].Blocked ?? DBNull.Value, (object?)seeded[i].Blocking ?? DBNull.Value); + } + + using (var doc = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlocking(postgres, ServerName, limit: 5))) + { + var events = doc.RootElement.GetProperty("events").EnumerateArray().ToList(); + Assert.Equal(new[] { 101, 102, 103, 104, 105 }, events.Select(e => e.GetProperty("blocked_spid").GetInt32()).ToArray()); + Assert.True(doc.RootElement.GetProperty("truncated").GetBoolean()); + + /* Written only when true: a row with no plan carries neither key. */ + Assert.Equal(new[] { (true, true), (true, false), (false, true), (false, false), (false, false) }, + events.Select(e => (Flag(e, "has_blocked_plan"), Flag(e, "has_blocking_plan"))).ToArray()); + Assert.False(events[3].TryGetProperty("has_blocked_plan", out _)); + Assert.False(events[4].TryGetProperty("has_blocking_plan", out _)); + } + + /* A page of two: the flags are for those two rows, and the rest of the window is not read for them. */ + using (var doc = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlocking(postgres, ServerName, limit: 2))) + { + var events = doc.RootElement.GetProperty("events").EnumerateArray().ToList(); + Assert.Equal(2, events.Count); + Assert.Equal(new[] { (true, true), (true, false) }, + events.Select(e => (Flag(e, "has_blocked_plan"), Flag(e, "has_blocking_plan"))).ToArray()); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + [Fact] + public async Task GetDeadlocks_FlagsThePagesRows_ByTheirOwnStoredPlans() + { + var baseConnectionString = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), SkipReason); + var ct = TestContext.Current.CancellationToken; + + // #1776 own-store: a scratch database, so no other class's chunks shape the store under test (#4650). + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var cs = scratch.ConnectionString; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PrepareStoreAsync(cs, connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs); + + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + /* The tool reads its "last N hours" window from the real clock, which the test clock override cannot move (#5608). */ + var now = DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow); + + /* Newest first by deadlock_time: a plan, an empty string, NULL, and an older plan the page of three leaves out. */ + var seeded = new (string Victim, string? Plan)[] { ("processA", Plan), ("processB", ""), ("processC", null), ("processD", Plan) }; + for (var i = 0; i < seeded.Length; i++) + { + var deadlockTime = now.AddMinutes(-10 - i); + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO deadlocks + (deadlock_id, collection_time, server_id, server_name, deadlock_time, victim_process_id, victim_sql_text, deadlock_graph_xml, victim_query_plan_xml, database_name) +VALUES ($1,$2,$3,$4,$5,$6,'UPDATE t SET c = 1','',$7,'PlanFlagsDb')", + CollectionIdGenerator.Next(), deadlockTime.AddMinutes(1), ServerId, ServerName, deadlockTime, + seeded[i].Victim, (object?)seeded[i].Plan ?? DBNull.Value); + } + + using (var doc = JsonDocument.Parse(await DarlingMcpBlockingTools.GetDeadlocks(postgres, ServerName, limit: 3))) + { + var rows = doc.RootElement.GetProperty("deadlocks").EnumerateArray().ToList(); + Assert.Equal(new[] { "processA", "processB", "processC" }, rows.Select(r => r.GetProperty("victim_process_id").GetString()).ToArray()); + Assert.Equal(new[] { true, false, false }, rows.Select(r => Flag(r, "has_victim_plan")).ToArray()); + Assert.False(rows[1].TryGetProperty("has_victim_plan", out _)); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// The planner's own output for each list statement names no plan column, on the store as it is built here (a + /// hypertable where TimescaleDB is present). A flag computed in the list is what put the plan text through its sorts. + [Fact] + public async Task TheListStatements_PlannerOutput_NamesNoPlanColumn() + { + var baseConnectionString = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), SkipReason); + var ct = TestContext.Current.CancellationToken; + + // #1776 own-store: a scratch database, so no other class's chunks shape the plan under test (#4650). + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var cs = scratch.ConnectionString; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PrepareStoreAsync(cs, connection, ct); + + var now = LiveClock.Now(); + foreach (var sql in new[] + { + DarlingBlockingReader.BlockedProcessReportsSql, + DarlingBlockingReader.BlockedProcessReportsWithXmlSql, + DarlingBlockingReader.RecentDeadlocksSql, + DarlingBlockingReader.RecentDeadlocksWithGraphSql, + }) + { + using var explain = new NpgsqlCommand("EXPLAIN (VERBOSE, COSTS OFF) " + sql, connection); + DarlingMcpReadParameters.AddWindow(explain, ServerId, now.AddHours(-24), now); + DarlingMcpReadParameters.AddInt(explain, 21); + DarlingMcpReadParameters.AddTimestamp(explain, EventWindowFloor.For(now.AddHours(-24))); + /* #5244: the blocked-process reads carry the database filter at $6 (the deadlock reads have none). */ + if (sql.Contains("$6", StringComparison.Ordinal)) + explain.Parameters.Add(DatabaseFilter.All.Parameter()); + var lines = new List(); + await using (var reader = await explain.ExecuteReaderAsync(ct)) + { + while (await reader.ReadAsync(ct)) + { + lines.Add(reader.GetString(0)); + } + } + + Assert.DoesNotContain(lines, line => line.Contains("query_plan_xml", StringComparison.Ordinal)); + } + } + + /// The store the way the service builds it: migrated, with hypertables where TimescaleDB is present (the probe + /// runs on its own connection, #1922), and no background policy job reshaping chunks under the assertions. + private static async Task PrepareStoreAsync(string connectionString, NpgsqlConnection connection, System.Threading.CancellationToken ct) + { + await PgMigrations.MigrateAsync(connection, ct); + if (await LiveTimescaleProbe.TryEnableAsync(connectionString, ct)) + { + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + await using var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection); + await stop.ExecuteNonQueryAsync(ct); + } + } + + private static bool Flag(JsonElement row, string name) => + row.TryGetProperty(name, out var value) && value.ValueKind == JsonValueKind.True; + + private static async Task DeleteRowsAsync(NpgsqlConnection connection, System.Threading.CancellationToken ct) + { + using var cleanup = new NpgsqlCommand( + $"DELETE FROM blocked_process_reports WHERE server_id = {ServerId}; DELETE FROM deadlocks WHERE server_id = {ServerId}; DELETE FROM servers WHERE server_id = {ServerId};", + connection); + await cleanup.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/BlockingThresholdCoverageTests.cs b/Darling/Darling.Tests/BlockingThresholdCoverageTests.cs new file mode 100644 index 0000000000..015d19127d --- /dev/null +++ b/Darling/Darling.Tests/BlockingThresholdCoverageTests.cs @@ -0,0 +1,89 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using PerformanceMonitor.Common; +using Xunit; + +namespace Darling.Tests; + +/// The shared rule that moves the Blocking charts' data start to where the blocked process threshold went on (#5098). +public sealed class BlockingThresholdCoverageTests +{ + private static readonly DateTime End = new(2026, 10, 1, 12, 0, 0, DateTimeKind.Utc); + private static readonly DateTime Start = End.AddDays(-7); + + [Fact] + public void ThresholdOnAtTheStart_KeepsTheCollectorsCoverage() + { + Assert.Equal(Start, BlockingThresholdCoverage.Combine(Start, End.AddDays(-1), onAtWindowStart: true, End.AddDays(-4), sawZeroSnapshot: true)); + } + + [Fact] + public void ThresholdOnAtTheStart_StillTakesAnEarlierReport() + { + var report = End.AddDays(-8); + Assert.Equal(report, BlockingThresholdCoverage.Combine(Start, report, true, null, false)); + } + + [Fact] + public void FirstNonzeroSnapshotInTheWindow_LaterThanCoverage_IsTheAnswer() + { + Assert.Equal(End.AddDays(-4), BlockingThresholdCoverage.Combine(Start, End.AddDays(-3), false, End.AddDays(-4), true)); + } + + [Fact] + public void AReportBeforeTheFirstNonzeroSnapshot_IsTheAnswer() + { + var report = End.AddDays(-4).AddHours(-12); + Assert.Equal(report, BlockingThresholdCoverage.Combine(Start, report, false, End.AddDays(-4), true)); + } + + [Fact] + public void CoverageLaterThanTheSnapshot_StaysTheCoverage() + { + var coverage = End.AddDays(-2); + Assert.Equal(coverage, BlockingThresholdCoverage.Combine(coverage, End.AddDays(-1), false, End.AddDays(-4), true)); + } + + [Fact] + public void OnlyZeroSnapshots_StartAtTheEarliestReport() + { + var report = End.AddDays(-1); + Assert.Equal(report, BlockingThresholdCoverage.Combine(Start, report, false, null, true)); + } + + [Fact] + public void NoThresholdSnapshots_KeepTodaysAnswer() + { + Assert.Equal(Start, BlockingThresholdCoverage.Combine(Start, End.AddDays(-1), false, null, false)); + Assert.Equal(End.AddDays(-1), BlockingThresholdCoverage.Combine(null, End.AddDays(-1), false, null, false)); + } + + [Fact] + public void NonzeroSnapshotsInTheWindow_WithNoZeroSeen_LeaveTodaysAnswer() + { + /* The snapshots before the window may have been purged, or never taken: no zero seen means the threshold's start is unknown, not late. */ + Assert.Equal(Start, BlockingThresholdCoverage.Combine(Start, End.AddDays(-1), false, End.AddDays(-4), sawZeroSnapshot: false)); + Assert.Equal(End.AddDays(-1), BlockingThresholdCoverage.Combine(null, End.AddDays(-1), false, End.AddDays(-4), false)); + } + + [Fact] + public void NothingKnown_IsNull() + { + Assert.Null(BlockingThresholdCoverage.Combine(null, null, false, null, false)); + Assert.Null(BlockingThresholdCoverage.Combine(null, null, true, null, true)); + } + + [Fact] + public void NoCollectorCoverage_TakesTheThresholdStart_ButNeverLaterThanTheReport() + { + Assert.Equal(End.AddDays(-4), BlockingThresholdCoverage.Combine(null, End.AddDays(-3), false, End.AddDays(-4), true)); + Assert.Equal(End.AddDays(-5), BlockingThresholdCoverage.Combine(null, End.AddDays(-5), false, End.AddDays(-4), true)); + } +} diff --git a/Darling/Darling.Tests/BlockingWaitsFilteredEmptyWordingTests.cs b/Darling/Darling.Tests/BlockingWaitsFilteredEmptyWordingTests.cs new file mode 100644 index 0000000000..b7a0d1ba61 --- /dev/null +++ b/Darling/Darling.Tests/BlockingWaitsFilteredEmptyWordingTests.cs @@ -0,0 +1,37 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace PerformanceMonitor.Darling.Tests; + +/// +/// #5244 PR3 (W3 wording carry): a filtered empty answer reads the same in Darling and Lite. These are the words Lite's +/// McpDatabaseSelectionTests pins with the same literals: the name for one database, "the chosen databases" for two or more. +/// +public sealed class BlockingWaitsFilteredEmptyWordingTests +{ + [Fact] + public void Describe_IsTheNameForOne_AndTheChosenDatabasesForTwoOrMore_AndNothingForAll() + { + Assert.Null(DatabaseFilter.All.Describe()); + Assert.Equal("A", DatabaseFilter.One("A").Describe()); + Assert.Equal("the chosen databases", DatabaseFilter.Of(["A", "B"]).Describe()); + Assert.Equal("the chosen databases", DatabaseFilter.ManyDatabasesDescription); + } + + [Fact] + public void TheEmptyAnswerSentence_SaysTheDatabaseForOne_AndTheChosenDatabasesForTwoOrMore() + { + Assert.Equal(string.Empty, DarlingMcpBlockingTools.ForChosenDatabases(DatabaseFilter.All)); + Assert.Equal(" for the database A", DarlingMcpBlockingTools.ForChosenDatabases(DatabaseFilter.One("A"))); + Assert.Equal(" for the chosen databases", DarlingMcpBlockingTools.ForChosenDatabases(DatabaseFilter.Of(["A", "B"]))); + } +} diff --git a/Darling/Darling.Tests/BlockingWaitsWebDispatchLiveTests.cs b/Darling/Darling.Tests/BlockingWaitsWebDispatchLiveTests.cs new file mode 100644 index 0000000000..c7966bf7c1 --- /dev/null +++ b/Darling/Darling.Tests/BlockingWaitsWebDispatchLiveTests.cs @@ -0,0 +1,161 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.Json; +using System.Threading.Tasks; +using Darling.Tests; +using Microsoft.AspNetCore.Http; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace PerformanceMonitor.Darling.Tests; + +/// +/// #5244 PR3 (lane W3): the six blocking and waits reads the Blocking and Current Waits panels drive, called the way the page calls +/// them, through the web read dispatch with two REPEATED database_name keys over a seed of databases A, B and C. Each read +/// must answer with A's and B's rows and none of C's. On the base the dispatch ignored the key, so each read returned all three. +/// +[Collection("live-postgres")] +public sealed class BlockingWaitsWebDispatchLiveTests +{ + private const string ServerName = "darling-w3-blocking-waits-web"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + + private static readonly string[] Reads = + [ + "get_waiting_tasks", "get_blocking", "get_blocked_process_xml", + "get_blocking_trend", "get_current_waits_trend", "get_blocking_stats", + ]; + + private static async Task WebReadAsync(NpgsqlDataSource postgres, string read, params string[] databases) + { + var query = new List> { new("server_name", ServerName), new("hours_back", "2") }; + query.AddRange(databases.Select(d => new KeyValuePair("database_name", d))); + var context = new DefaultHttpContext(); + context.Request.QueryString = QueryString.Create(query); + return await DarlingWebEndpoints.BuildReadDispatch()[read](context, postgres, null!); + } + + /// The databases (or, for the two count series, the total count) a read's answer covers. + private static (List Databases, long Count) CoverageOf(string read, string json) + { + var root = JsonDocument.Parse(json).RootElement; + List Databases(string property) + { + Assert.True(root.TryGetProperty(property, out var rows), $"{read}: not the success shape: {json}"); + return rows.EnumerateArray().Select(r => r.GetProperty("database_name").GetString()!).Distinct().OrderBy(n => n, StringComparer.Ordinal).ToList(); + } + long Total(string property, string field) + { + Assert.True(root.TryGetProperty(property, out var rows), $"{read}: not the success shape: {json}"); + return rows.EnumerateArray().Sum(p => p.GetProperty(field).GetInt64()); + } + return read switch + { + "get_waiting_tasks" => (Databases("tasks"), 0), + "get_blocking" => (Databases("events"), 0), + "get_blocked_process_xml" => (Databases("reports"), 0), + "get_current_waits_trend" => (Databases("blocked_sessions"), 0), + "get_blocking_trend" => ([], Total("trend", "count")), + _ => ([], Total("blocking_duration", "event_count")), + }; + } + + [Fact] + public async Task EachOfTheSixReads_WithTwoChosenDatabases_ReturnsOnlyThoseDatabasesRows() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to run the live blocking and waits web dispatch tests."); + var ct = TestContext.Current.CancellationToken; + + // #1776 own-store: a scratch database, so no other class's chunks shape the store under test (#4650). + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var cs = scratch.ConnectionString; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + if (await LiveTimescaleProbe.TryEnableAsync(cs, ct)) + { + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + await using var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection); + await stop.ExecuteNonQueryAsync(ct); + } + + await using var postgres = NpgsqlDataSource.Create(cs); + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + var now = DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow); + + var spid = 100; + foreach (var db in new[] { "A", "B", "C" }) + { + spid++; + var eventTime = DarlingMcpTestData.Naive(now.AddMinutes(-10)); + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO blocked_process_reports + (blocked_report_id, collection_time, server_id, server_name, event_time, database_name, + blocked_spid, blocked_ecid, blocking_spid, blocking_ecid, wait_time_ms, lock_mode, + blocked_sql_text, blocking_sql_text, blocked_process_report_xml) +VALUES ($1,$2,$3,$4,$5,$6,$7,0,$8,0,1000,'X','SELECT 1','UPDATE t SET c = 1','')", + CollectionIdGenerator.Next(), eventTime.AddSeconds(30), ServerId, ServerName, eventTime, db, spid, 900 + spid); + + /* A blocked waiting task per database, so the blocked-session series and the waiting-task list both have a row each. */ + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO waiting_tasks (collection_id, collection_time, server_id, server_name, session_id, wait_type, wait_duration_ms, blocking_session_id, resource_description, database_name) +VALUES ($1,$2,$3,$4,$5,'LCK_M_S',3000,90,NULL,$6)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(now.AddMinutes(-10)), ServerId, ServerName, spid, db); + } + + /* The collectors ran, so a trend's empty answer is an all-clear and not a gap. */ + foreach (var collector in new[] { "blocked_process_report", "waiting_tasks" }) + { + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO collection_log (log_id, server_id, server_name, collector_name, collection_time, duration_ms, status, rows_collected) +VALUES ($1,$2,$3,$4,$5,100,'SUCCESS',0)", + CollectionIdGenerator.Next(), ServerId, ServerName, collector, DarlingMcpTestData.Naive(now.AddMinutes(-30))); + } + + foreach (var read in Reads) + { + var all = CoverageOf(read, await WebReadAsync(postgres, read)); + var chosen = CoverageOf(read, await WebReadAsync(postgres, read, "A", "B")); + if (read is "get_blocking_trend" or "get_blocking_stats") + { + Assert.True(all.Count == 3, $"{read}: the control (no database_name) counted {all.Count}, not 3"); + Assert.True(chosen.Count == 2, $"{read} over A and B counted {chosen.Count}, not 2"); + } + else + { + Assert.True(new[] { "A", "B", "C" }.SequenceEqual(all.Databases), $"{read}: the control covered [{string.Join(",", all.Databases)}]"); + Assert.True(new[] { "A", "B" }.SequenceEqual(chosen.Databases), $"{read} over A and B covered [{string.Join(",", chosen.Databases)}]"); + } + + /* A blank-only list is refused, never widened to "all databases". */ + var refused = JsonDocument.Parse(await WebReadAsync(postgres, read, " ", "")).RootElement; + Assert.True(refused.GetProperty("status").GetString() == "invalid", $"{read}: a blank-only database_name must be refused"); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs, bodySucceeded, async (cleanup, cleanupCt) => + await DarlingMcpTestData.ExecAsync(cleanup, cleanupCt, + $"DELETE FROM waiting_tasks WHERE server_id = {ServerId}; DELETE FROM blocked_process_reports WHERE server_id = {ServerId}; DELETE FROM collection_log WHERE server_id = {ServerId}; DELETE FROM servers WHERE server_id = {ServerId};")); + } + } +} diff --git a/Darling/Darling.Tests/BoundedCompressionDdlTests.cs b/Darling/Darling.Tests/BoundedCompressionDdlTests.cs new file mode 100644 index 0000000000..62d904412f --- /dev/null +++ b/Darling/Darling.Tests/BoundedCompressionDdlTests.cs @@ -0,0 +1,190 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Diagnostics; +using System.Linq; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// The compression-settings ALTERs on the hourly pass wait for their AccessExclusiveLock at most +/// , so a long reader cannot queue every collector write +/// behind them (#4970). Source pins for the wiring, a live lock-holder pin for the behaviour, and the rule for a +/// failed settings read. +/// +/* #1776 own-store: each live fact mints its own scratch database through ScratchPostgres. */ +[Collection("timing")] +public sealed class BoundedCompressionDdlTests +{ + private static string Storage() => + RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "TimescaleSupport.cs").Replace("\r\n", "\n"); + + private static string Body(string text, string signature) + { + var start = text.IndexOf(signature, StringComparison.Ordinal); + Assert.True(start >= 0, $"could not locate {signature}"); + var end = text.IndexOf("\n }\n", start, StringComparison.Ordinal); + Assert.True(end > start, $"could not find the end of {signature}"); + return text.Substring(start, end - start); + } + + [Fact] + public void TheHelper_RunsInsideATransaction_WithASetLocalLockTimeout_AndCatchesLockNotAvailable() + { + var helper = Body(Storage(), "internal static async Task TryRunBoundedDdlAsync("); + Assert.Contains("BeginTransactionAsync", helper, StringComparison.Ordinal); + Assert.Contains("SET LOCAL lock_timeout", helper, StringComparison.Ordinal); + Assert.Contains("PostgresErrorCodes.LockNotAvailable", helper, StringComparison.Ordinal); + Assert.Contains("CommitAsync", helper, StringComparison.Ordinal); + } + + [Fact] + public void BothCompressionAlters_GoThroughTheHelper_NotAnUnboundedCommand() + { + var text = Storage(); + var collector = Body(text, "public static async Task ApplyCompressionPolicyAsync("); + var aggregate = Body(text, "public static async Task EnsureAggregateCompressionAsync("); + + Assert.Contains("TryRunBoundedDdlAsync(", collector, StringComparison.Ordinal); + Assert.Contains("EnableCompressionSql(schema)", collector, StringComparison.Ordinal); + Assert.DoesNotContain("new NpgsqlCommand(EnableCompressionSql(schema)", collector, StringComparison.Ordinal); + + Assert.Contains("TryRunBoundedDdlAsync(", aggregate, StringComparison.Ordinal); + Assert.DoesNotContain("new NpgsqlCommand(EnableAggregateCompressionSql(view)", aggregate, StringComparison.Ordinal); + } + + [Fact] + public void TheHourlyPass_SkipsTheEnableAlters_AfterAFailedRead_AndTheStartPathDoesNot() + { + var text = Storage(); + var collector = Body(text, "public static async Task ApplyCompressionPolicyAsync("); + Assert.Contains("bool hourly, CancellationToken cancellationToken", collector, StringComparison.Ordinal); + Assert.Contains("var skipEnable = hourly && converged is null;", collector, StringComparison.Ordinal); + Assert.Contains("if (skipEnable)", collector, StringComparison.Ordinal); + + var worker = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "DarlingWorker.cs").Replace("\r\n", "\n"); + Assert.Contains("ApplyCompressionPolicyAsync(connection, logger, hourly: true, ct)", worker, StringComparison.Ordinal); + } + + [Fact] + public async Task AFailedSettingsRead_SkipsTheAltersOnTheHourlyPass_AndIssuesThemOnTheStartPath() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to run the live failed-read pin."); + var ct = TestContext.Current.CancellationToken; + + /* A scratch database with no TimescaleDB extension: the settings read fails (no timescaledb_information + schema), which is the failed-read case without a mock. */ + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + + await ExecAsync(connection, "DROP EXTENSION IF EXISTS timescaledb CASCADE", ct); + + var hourlyLog = new CapturingTestLogger(); + await TimescaleSupport.ApplyCompressionPolicyAsync(connection, hourlyLog, hourly: true, ct); + Assert.Equal(1, hourlyLog.Lines.Count(l => l.Contains("skipping the compression-enable ALTERs", StringComparison.Ordinal))); + Assert.DoesNotContain(hourlyLog.Lines, l => l.Contains("compression settings on collect.", StringComparison.Ordinal)); + + var startLog = new CapturingTestLogger(); + await TimescaleSupport.ApplyCompressionPolicyAsync(connection, startLog, ct); + Assert.DoesNotContain(startLog.Lines, l => l.Contains("skipping the compression-enable ALTERs", StringComparison.Ordinal)); + Assert.Contains(startLog.Lines, l => l.Contains("compression settings on collect.", StringComparison.Ordinal)); + } + + [Fact] + public async Task ALongReader_CannotStallTheHourlyEnableAlter_OrTheWritesBehindIt_AndTheNextPassConverges() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to run the live lock-holder pin."); + var ct = TestContext.Current.CancellationToken; + const string Table = "wait_stats"; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + Assert.True(await LiveTimescaleProbe.TryEnableAsync(scratch.ConnectionString, ct), "TimescaleDB must be available"); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + + Assert.DoesNotContain(Table, await TimescaleSupport.ReadTablesNeedingCompressionEnableAsync(connection, null, ct) ?? throw new InvalidOperationException("read failed")); + + using var holder = new NpgsqlConnection(scratch.ConnectionString); + await holder.OpenAsync(ct); + using var writer = new NpgsqlConnection(scratch.ConnectionString); + await writer.OpenAsync(ct); + + var log = new CapturingTestLogger(); + var holding = true; + try + { + await ExecAsync(holder, "BEGIN", ct); + await ExecAsync(holder, $"SELECT 1 FROM collect.{Table} LIMIT 1", ct); + + var clock = Stopwatch.StartNew(); + var pass = Task.Run(() => TimescaleSupport.ApplyCompressionPolicyAsync(connection, log, hourly: true, ct), ct); + + /* The INSERT is timed while the ALTER is still queued for its lock. */ + using var observer = new NpgsqlConnection(scratch.ConnectionString); + await observer.OpenAsync(ct); + var waiting = false; + var pollClock = Stopwatch.StartNew(); + while (!waiting && pollClock.Elapsed < TimeSpan.FromSeconds(5)) + { + using var probe = new NpgsqlCommand( + "SELECT count(*) FROM pg_locks l JOIN pg_class c ON c.oid = l.relation JOIN pg_namespace n ON n.oid = c.relnamespace " + + "WHERE l.mode = 'AccessExclusiveLock' AND NOT l.granted AND n.nspname = 'collect' AND c.relname = @t", observer); + probe.Parameters.AddWithValue("t", Table); + waiting = Convert.ToInt64(await probe.ExecuteScalarAsync(ct), System.Globalization.CultureInfo.InvariantCulture) > 0; + if (!waiting) + { + await Task.Delay(50, ct); + } + } + + Assert.True(waiting, $"the ALTER on collect.{Table} never showed an ungranted AccessExclusiveLock request within 5 s"); + var lockTimeout = TimeSpan.FromSeconds(int.Parse(TimescaleSupport.HourlyDdlLockTimeout.TrimEnd('s'), System.Globalization.CultureInfo.InvariantCulture)); + var insertClock = Stopwatch.StartNew(); + await ExecAsync(writer, + $"INSERT INTO collect.{Table} (collection_id, collection_time, server_id, server_name, wait_type, delta_waiting_tasks, delta_wait_time_ms, sample_interval_seconds) VALUES (1, now(), 1, 's', 'W', 1, 1, 1)", ct); + Assert.True(insertClock.Elapsed < lockTimeout + TimeSpan.FromSeconds(1.5), $"the insert must not queue behind the waiting ALTER, took {insertClock.Elapsed}"); + + await pass; + Assert.True(clock.Elapsed < TimeSpan.FromSeconds(10), $"the pass must return in a bounded time, took {clock.Elapsed}"); + + Assert.DoesNotContain(Table, await TimescaleSupport.ReadTablesNeedingCompressionEnableAsync(connection, null, ct) ?? throw new InvalidOperationException("read failed")); + Assert.Equal(1, log.Lines.Count(l => l.Contains($"collect.{Table}", StringComparison.Ordinal) && l.Contains("not changed this pass", StringComparison.Ordinal))); + + await ExecAsync(holder, "ROLLBACK", ct); + holding = false; + await TimescaleSupport.ApplyCompressionPolicyAsync(connection, log, hourly: true, ct); + var after = await TimescaleSupport.ReadTablesNeedingCompressionEnableAsync(connection, null, ct) ?? throw new InvalidOperationException("read failed"); + Assert.Contains(Table, after); + } + finally + { + if (holding) + { + await ExecAsync(holder, "ROLLBACK", CancellationTokenNone()); + } + } + } + + private static System.Threading.CancellationToken CancellationTokenNone() => System.Threading.CancellationToken.None; + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, System.Threading.CancellationToken ct) + { + using var command = new NpgsqlCommand(sql, connection) { CommandTimeout = 60 }; + await command.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/BoundedDdlBusyStreakTests.cs b/Darling/Darling.Tests/BoundedDdlBusyStreakTests.cs new file mode 100644 index 0000000000..e1bcda6078 --- /dev/null +++ b/Darling/Darling.Tests/BoundedDdlBusyStreakTests.cs @@ -0,0 +1,163 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Npgsql; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// A bounded change whose table stays busy pass after pass escalates once to a Warning and says so when it +/// finally converges (#4980). +/// +public sealed class BoundedDdlBusyStreakTests +{ + private static readonly DateTime T0 = new(2026, 10, 2, 12, 0, 0, DateTimeKind.Utc); + + private static bool Busy(BoundedDdlBusyStreaks s, string key, int n = 1) + { + var escalated = false; + for (var i = 0; i < n; i++) + { + escalated |= s.RecordBusy(key, T0.AddHours(i), out _, out _); + } + + return escalated; + } + + [Fact] + public void TwoBusyPasses_DoNotEscalate() + { + var s = new BoundedDdlBusyStreaks(); + Assert.False(Busy(s, "a", 2)); + } + + [Fact] + public void TheThirdBusyPass_EscalatesOnce_AndOnlyOnce() + { + var s = new BoundedDdlBusyStreaks(); + Assert.False(Busy(s, "a", 2)); + Assert.True(s.RecordBusy("a", T0.AddHours(2), out var passes, out var first)); + Assert.Equal(BoundedDdlBusyStreaks.EscalateAfter, passes); + Assert.Equal(T0, first); + Assert.False(Busy(s, "a", 1)); + Assert.False(Busy(s, "a", 1)); + } + + [Fact] + public void AnApplied_ClearsTheStreak_AndReportsItHadEscalated_ThenItCanEscalateAgain() + { + var s = new BoundedDdlBusyStreaks(); + Assert.True(Busy(s, "a", 3)); + Assert.True(s.Clear("a", out var passes)); + Assert.Equal(3, passes); + Assert.False(s.Clear("a", out _)); + Assert.False(Busy(s, "a", 2)); + Assert.True(Busy(s, "a", 1)); + } + + [Fact] + public void AClear_BeforeEscalation_ReportsNoEscalation_AndRestartsTheCount() + { + var s = new BoundedDdlBusyStreaks(); + Assert.False(Busy(s, "a", 2)); + Assert.False(s.Clear("a", out _)); + Assert.False(Busy(s, "a", 2)); + } + + [Fact] + public void TwoObjects_DoNotShareAStreak() + { + var s = new BoundedDdlBusyStreaks(); + Assert.False(Busy(s, "a", 2)); + Assert.False(Busy(s, "b", 2)); + Assert.True(Busy(s, "a", 1)); + Assert.True(Busy(s, "b", 1)); + } + + [Fact] + public void TheWiring_ClearsOnApplied_AndOnFailed_AndEscalatesOnBusy() + { + var text = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "TimescaleSupport.cs").Replace("\r\n", "\n"); + var start = text.IndexOf("internal static async Task TryRunBoundedDdlAsync(", StringComparison.Ordinal); + Assert.True(start > 0); + var body = text.Substring(start, text.IndexOf("\n }\n", start, StringComparison.Ordinal) - start); + Assert.Contains("BusyStreaks.RecordBusy(what", body, StringComparison.Ordinal); + /* Applied, Failed, and (#5574) a relation that is gone: a chunk dropped by retention must not leave its streak behind. */ + Assert.Equal(3, body.Split("BusyStreaks.Clear(what", StringSplitOptions.None).Length - 1); + Assert.Contains("LogWarning", body, StringComparison.Ordinal); + } +} + +/* #1776 own-store: this fact mints its own scratch database through ScratchPostgres and never touches another + store. */ +public sealed class BoundedDdlBusyStreakLiveTests +{ + [Fact] + public async Task ABusyTable_WarnsOnceOnTheThirdPass_AndSaysSoWhenItConverges() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to run the live busy-streak pin."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + Assert.True(await LiveTimescaleProbe.TryEnableAsync(scratch.ConnectionString, ct), "TimescaleDB must be available"); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + + // A key no other test records into, so the process-wide streaks stay untouched. + var What = $"compression settings on collect.wait_stats (busy-streak test {Guid.NewGuid():N})"; + var statements = new[] { "ALTER TABLE collect.wait_stats SET (timescaledb.compress)" }; + var log = new CapturingTestLogger(); + + using var holder = new NpgsqlConnection(scratch.ConnectionString); + await holder.OpenAsync(ct); + var holding = true; + try + { + await Exec(holder, "BEGIN", ct); + await Exec(holder, "SELECT 1 FROM collect.wait_stats LIMIT 1", ct); + + for (var i = 0; i < BoundedDdlBusyStreaks.EscalateAfter; i++) + { + var outcome = await TimescaleSupport.TryRunBoundedDdlAsync(connection, statements, log, What, ct); + Assert.Equal(TimescaleSupport.BoundedDdlOutcome.LockBusy, outcome); + } + + Assert.Equal(1, log.CountAtLevel(LogLevel.Warning)); + Assert.Contains(log.Lines, l => l.StartsWith("Warning:", StringComparison.Ordinal) && l.Contains(What, StringComparison.Ordinal)); + + await Exec(holder, "ROLLBACK", ct); + holding = false; + var applied = await TimescaleSupport.TryRunBoundedDdlAsync(connection, statements, log, What, ct); + Assert.Equal(TimescaleSupport.BoundedDdlOutcome.Applied, applied); + Assert.Equal(1, log.CountAtLevel(LogLevel.Warning)); + Assert.Contains(log.Lines, l => l.StartsWith("Information:", StringComparison.Ordinal) && l.Contains("busy passes", StringComparison.Ordinal)); + } + finally + { + if (holding) + { + await Exec(holder, "ROLLBACK", System.Threading.CancellationToken.None); + } + } + } + + private static async Task Exec(NpgsqlConnection connection, string sql, System.Threading.CancellationToken ct) + { + using var command = new NpgsqlCommand(sql, connection) { CommandTimeout = 60 }; + await command.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/BoundedDdlSingleCopyTests.cs b/Darling/Darling.Tests/BoundedDdlSingleCopyTests.cs new file mode 100644 index 0000000000..40ff6dd58d --- /dev/null +++ b/Darling/Darling.Tests/BoundedDdlSingleCopyTests.cs @@ -0,0 +1,82 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using Xunit; + +namespace Darling.Tests; + +/// +/// The bounded compression DDL has one copy: TimescaleSupport.TryRunBoundedDdlAsync is the only place that +/// sets the hourly lock timeout, and every compression-enable ALTER goes through it (#4970). +/// +public sealed class BoundedDdlSingleCopyTests +{ + private static string Storage() => + RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "TimescaleSupport.cs").Replace("\r\n", "\n"); + + private static string[] CodeLines() => + CSharpSourceWalker.StripCommentsAndStrings(Storage()).Split('\n'); + + [Fact] + public void TheHourlyLockTimeout_IsSetInOneMethodOnly() + { + var text = Storage(); + const string setLocal = "SET LOCAL lock_timeout = '{HourlyDdlLockTimeout}'"; + var first = text.IndexOf(setLocal, StringComparison.Ordinal); + Assert.True(first >= 0, "could not find the helper's SET LOCAL lock_timeout"); + Assert.Equal(-1, text.IndexOf(setLocal, first + 1, StringComparison.Ordinal)); + var helper = text.IndexOf("internal static async Task TryRunBoundedDdlAsync(", StringComparison.Ordinal); + Assert.True(helper >= 0 && helper < first && first - helper < 1200, + "the one SET LOCAL lock_timeout must sit inside TryRunBoundedDdlAsync"); + /* Two overloads since #5574 (the statement-list one calls the body one, which owns the SET LOCAL above); a third + would be a second copy of the helper. */ + Assert.Equal(2, text.Split("BoundedDdlOutcome> TryRun", StringSplitOptions.None).Length - 1); + } + + [Fact] + public void EveryCompressionEnableAlter_IsAStatementPassedToTheHelper() + { + var lines = CodeLines(); + var uses = lines.Select((l, i) => (l, i)) + .Where(x => (x.l.Contains("EnableCompressionSql(", StringComparison.Ordinal) || x.l.Contains("EnableAggregateCompressionSql(", StringComparison.Ordinal)) + && !x.l.Contains("public static string", StringComparison.Ordinal) + && !x.l.Contains("return EnableCompressionSql(", StringComparison.Ordinal)) + .ToArray(); + Assert.Equal(3, uses.Length); + foreach (var (line, index) in uses) + { + Assert.Contains("new[] {", line, StringComparison.Ordinal); + Assert.Contains("TryRunBoundedDdlAsync(", lines[index - 1], StringComparison.Ordinal); + } + + Assert.DoesNotContain("new NpgsqlCommand(EnableCompressionSql(", Storage(), StringComparison.Ordinal); + Assert.DoesNotContain("new NpgsqlCommand(EnableAggregateCompressionSql(", Storage(), StringComparison.Ordinal); + } + + [Fact] + public void ThePrivateCollectionLogCopy_IsGone() + { + var text = Storage(); + Assert.DoesNotContain("TrySetCollectionLogCompressionAsync", text, StringComparison.Ordinal); + Assert.DoesNotContain("CollectionLogSettingsLockTimeout", text, StringComparison.Ordinal); + } + + [Fact] + public void AFailedSettingsRead_SkipsThePolicyCall_OnTheHourlyPass() + { + var text = Storage(); + var start = text.IndexOf("public static async Task ApplyCompressionPolicyAsync(", StringComparison.Ordinal); + Assert.True(start >= 0); + var skip = text.IndexOf("if (skipEnable)\n {\n continue;", start, StringComparison.Ordinal); + var policy = text.IndexOf("new NpgsqlCommand(AddCompressionPolicySql(schema)", start, StringComparison.Ordinal); + Assert.True(skip > start && policy > skip, + "the per-table policy call must be skipped when the settings read failed on the hourly pass"); + } +} diff --git a/Darling/Darling.Tests/BugReportTemplateDiagnosticsBundleTests.cs b/Darling/Darling.Tests/BugReportTemplateDiagnosticsBundleTests.cs new file mode 100644 index 0000000000..6bf39c420b --- /dev/null +++ b/Darling/Darling.Tests/BugReportTemplateDiagnosticsBundleTests.cs @@ -0,0 +1,42 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using PerformanceMonitor.Darling.Service; +using Xunit; + +namespace Darling.Tests; + +/// #5097: the bug-report template asks for the bundle, with the verb spelled as the product spells it. +public sealed class BugReportTemplateDiagnosticsBundleTests +{ + private static string Template() => + RepoFile.ReadRepoFile(".github", "ISSUE_TEMPLATE", "bug_report.yml").ReplaceLineEndings("\n"); + + [Fact] + public void Template_HasTheBundleField_WithTheVerbFromThePredicate() + { + var yml = Template(); + Assert.Contains("id: diagnostics-bundle", yml, StringComparison.Ordinal); + Assert.Contains(DarlingCliCommands.DiagnosticsBundleVerbName + " darling-bundle.json", yml, StringComparison.Ordinal); + Assert.True(DarlingCliCommands.IsDiagnosticsBundleVerb(DarlingCliCommands.DiagnosticsBundleVerbName)); + Assert.Contains("Open it and check it before you attach it", yml, StringComparison.Ordinal); + var field = yml[yml.IndexOf("id: diagnostics-bundle", StringComparison.Ordinal)..]; + field = field[..field.IndexOf("id: screenshots", StringComparison.Ordinal)]; + Assert.Contains("required: false", field, StringComparison.Ordinal); + } + + [Fact] + public void ErrorsField_NamesTheDarlingLogPath() + { + var yml = Template(); + var errors = yml[yml.IndexOf("id: errors", StringComparison.Ordinal)..]; + errors = errors[..errors.IndexOf("id: diagnostics-bundle", StringComparison.Ordinal)]; + Assert.Contains(@"%ProgramData%\PerformanceMonitorDarling\logs\", errors, StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/BuiltinAlertPersistenceRungTests.cs b/Darling/Darling.Tests/BuiltinAlertPersistenceRungTests.cs index a3ce628c56..b531ed981b 100644 --- a/Darling/Darling.Tests/BuiltinAlertPersistenceRungTests.cs +++ b/Darling/Darling.Tests/BuiltinAlertPersistenceRungTests.cs @@ -29,6 +29,7 @@ namespace Darling.Tests; /// claims, and they are stated against this rung's own NUMBER — read out of the ladder rather than taken /// from , which is no longer it. /// +[Trait("Reads", "Lite")] public sealed class BuiltinAlertPersistenceRungTests { /// This rung's number, DERIVED from the ladder entry that owns the name rather than written as diff --git a/Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs b/Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs index edf5582513..7d8732417a 100644 --- a/Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs +++ b/Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs @@ -318,6 +318,86 @@ the group index still present — the same fixture shape CaggGroupIndexDropUpgra /// A future rung that adds a new probe sentinel must add its artifact's removal here. private static async Task DropArtifactsOfLaterRungsAsync(NpgsqlConnection connection, int simulatedVersion, CancellationToken ct) { + if (simulatedVersion < QueryStoreComposeStamp.RungVersion) + { + /* V173 (#5582) - the stamp-grain rollup of the wide Query Store table, its validity tables and the two late-row triggers on the + wide parent; the hours table is the probe's sentinel. The triggers go first: the V171 rewind below renames the table they sit on. */ + await using var dropStamp = new NpgsqlCommand( + "DROP TRIGGER IF EXISTS trg_query_store_compose_stamp_late_ins ON collect.query_store_interval_wide;" + + " DROP TRIGGER IF EXISTS trg_query_store_compose_stamp_late_upd ON collect.query_store_interval_wide;" + + " DROP TABLE IF EXISTS collect.query_store_compose_stamp_hours, collect.query_store_compose_stamp_built, collect.query_store_compose_stamp;" + + " DROP FUNCTION IF EXISTS collect.query_store_compose_stamp_mark_late();", connection); + await dropStamp.ExecuteNonQueryAsync(ct); + } + + if (simulatedVersion < QueryStoreIntervalPartitionRungTests.LatestRungVersion) + { + /* V172 (#5571) - the latest Query Store interval table is partitioned; its relkind is the probe's sentinel. */ + await QueryStoreIntervalPartitionRungTests.RewindTableAsync(connection, "query_store_interval_latest", ct); + } + + if (simulatedVersion < QueryStoreIntervalPartitionRungTests.WideRungVersion) + { + /* V171 (#5571) - the same for the wide table. */ + await QueryStoreIntervalPartitionRungTests.RewindTableAsync(connection, "query_store_interval_wide", ct); + } + + if (simulatedVersion < PgIoStatsHourly.RungVersion) + { + /* V170 (#5495) - the hourly rollup of pg_io_stats and its state table; the state table is the probe's sentinel. */ + await using var dropIoHourly = new NpgsqlCommand( + "DROP TABLE IF EXISTS collect.pg_io_stats_hourly_state, collect.pg_io_stats_hourly", connection); + await dropIoHourly.ExecuteNonQueryAsync(ct); + } + + if (simulatedVersion < 169) + { + /* V169 (#5452) - the per-server AWS role on config_monitored_servers; the role column is the probe's sentinel. + Dropping the columns drops the table's check on them too. */ + await using var dropAwsRole = new NpgsqlCommand( + "ALTER TABLE config.config_monitored_servers DROP COLUMN IF EXISTS aws_external_id_set, " + + "DROP COLUMN IF EXISTS aws_external_id, DROP COLUMN IF EXISTS aws_role_arn", connection); + await dropAwsRole.ExecuteNonQueryAsync(ct); + } + + if (simulatedVersion < 168) + { + /* V168 (#5448) - the PLAN_REGRESSION per-day totals; the built table is the probe's sentinel. The trigger on + query_store_interval_latest goes first, then the function it calls, then the two tables. */ + await using var dropPlanRegressionDaily = new NpgsqlCommand( + "DROP TRIGGER IF EXISTS trg_plan_regression_daily_late ON collect.query_store_interval_latest;" + + " DROP FUNCTION IF EXISTS collect.plan_regression_daily_mark_late();" + + " DROP TABLE IF EXISTS collect.plan_regression_daily, collect.plan_regression_daily_built", connection); + await dropPlanRegressionDaily.ExecuteNonQueryAsync(ct); + } + + if (simulatedVersion < LegacyPinCandidateTables.RungVersion) + { + /* V167 (#5456) - the table that records the old-format values present at the upgrade; + config.legacy_secret_pin_candidate is the probe's sentinel. It goes before the V165 arm below, because its + triggers use the function that arm drops. */ + await using var dropCandidates = new NpgsqlCommand("DROP TABLE IF EXISTS config.legacy_secret_pin_candidate", connection); + await dropCandidates.ExecuteNonQueryAsync(ct); + } + + if (simulatedVersion < 166) + { + /* V166 - the PagerDuty auto-resolve column on config_notification; the column is the probe's sentinel. */ + await using var dropAutoResolve = new NpgsqlCommand( + "ALTER TABLE config.config_notification DROP COLUMN IF EXISTS pagerduty_auto_resolve", connection); + await dropAutoResolve.ExecuteNonQueryAsync(ct); + } + + if (simulatedVersion < PasswordKeyTables.RungVersion) + { + /* V165 (#5366) - the tables for the service's password key; config.password_key is the probe's sentinel. + The four tables go first, which also drops their triggers, and then the trigger function they shared. */ + await using var dropPasswordKey = new NpgsqlCommand( + "DROP TABLE IF EXISTS config.password_key, config.password_key_service, config.legacy_secret_pin, config.legacy_secret_pin_marker;" + + " DROP FUNCTION IF EXISTS config.password_key_owner_only()", connection); + await dropPasswordKey.ExecuteNonQueryAsync(ct); + } + if (simulatedVersion < 153) { /* V153 (#4608, split #4615) — the interval-tables-latest first_execution_time index. */ @@ -343,6 +423,64 @@ private static async Task DropArtifactsOfLaterRungsAsync(NpgsqlConnection connec await dropEnd.ExecuteNonQueryAsync(ct); } + if (simulatedVersion < QueryStatsHourLedgerTests.RungVersion) + { + /* V164 (#4605) - the hourly row-count ledger; the ledger table is the probe's sentinel. */ + await using var dropLedger = new NpgsqlCommand( + "DROP TABLE IF EXISTS collect.query_stats_hour_ledger_state, collect.query_stats_hour_ledger", connection); + await dropLedger.ExecuteNonQueryAsync(ct); + } + + if (simulatedVersion < StoreStatementHistoryRungTests.RungVersion) + { + /* V163 (#5097) - the store's statement history; the history table is the probe's sentinel. */ + await using var dropStatementHistory = new NpgsqlCommand( + "DROP TABLE IF EXISTS collect.store_statement_history, collect.store_statement_captures, config.store_statement_baseline", connection); + await dropStatementHistory.ExecuteNonQueryAsync(ct); + } + + if (simulatedVersion < SlowReadsRungTests.RungVersion) + { + /* V162 (#5097) - the slow-read record; the table is the probe's sentinel. */ + await using var dropSlowReads = new NpgsqlCommand( + "DROP TABLE IF EXISTS collect.slow_reads", connection); + await dropSlowReads.ExecuteNonQueryAsync(ct); + } + + if (simulatedVersion < QueryStoreTopDailyRungTests.RungVersion) + { + /* V161 (#5094) - the Query Store top daily summary tables; the first is the probe's sentinel. */ + await using var dropTopDaily = new NpgsqlCommand( + "DROP TABLE IF EXISTS collect.query_store_top_daily, collect.query_store_top_daily_built", connection); + await dropTopDaily.ExecuteNonQueryAsync(ct); + } + + if (simulatedVersion < CollectorRunTimeRungTests.RungVersion) + { + /* V160 (#4938) - the collector run time, in a table of its own; the table is the probe's sentinel. */ + await using var dropRunAt = new NpgsqlCommand( + "DROP TABLE IF EXISTS config.config_collector_run_times", connection); + await dropRunAt.ExecuteNonQueryAsync(ct); + } + + if (simulatedVersion < InstallIdTableOidRungTests.RungVersion) + { + /* V159 (#4961) - the install id table's own OID and the server's major version, which arrive in the one + statement; the OID's column is the probe's sentinel. IF EXISTS on the table, because a store simulated below + the install id's own rung has had the table dropped already. */ + await using var dropTableOid = new NpgsqlCommand( + "ALTER TABLE IF EXISTS config.config_install_id DROP COLUMN IF EXISTS table_oid, DROP COLUMN IF EXISTS server_major", connection); + await dropTableOid.ExecuteNonQueryAsync(ct); + } + + if (simulatedVersion < InstallIdRungTests.RungVersion) + { + /* V158 (#4961) - the install id's one-row table; the table is the probe's sentinel. */ + await using var dropInstallId = new NpgsqlCommand( + "DROP TABLE IF EXISTS config.config_install_id", connection); + await dropInstallId.ExecuteNonQueryAsync(ct); + } + if (simulatedVersion < MuteRuleServerIdRungTests.RungVersion) { /* V157 - the mute rule's store server id; the column is the probe's sentinel. */ diff --git a/Darling/Darling.Tests/CapabilityPins/viewerservertab-eventwires.baseline.txt b/Darling/Darling.Tests/CapabilityPins/viewerservertab-eventwires.baseline.txt index 5df7eba883..666e6ac573 100644 --- a/Darling/Darling.Tests/CapabilityPins/viewerservertab-eventwires.baseline.txt +++ b/Darling/Darling.Tests/CapabilityPins/viewerservertab-eventwires.baseline.txt @@ -8,7 +8,6 @@ MouseDoubleClick=>QueryStoreRegressionsGrid_MouseDoubleClick SelectionChanged=>AutoRefreshInterval_Changed SelectionChanged=>BlockingSubTabs_SelectionChanged SelectionChanged=>CompareToCombo_SelectionChanged -SelectionChanged=>CustomTimeCombo_Changed SelectionChanged=>FileIoSubTabs_SelectionChanged SelectionChanged=>HeatmapMetric_SelectionChanged SelectionChanged=>InnerTabs_SelectionChanged @@ -19,5 +18,4 @@ SelectionChanged=>QueryStatsGrid_SelectionChanged SelectionChanged=>QueryStoreGrid_SelectionChanged SelectionChanged=>SystemEventsSubTabs_SelectionChanged SelectionChanged=>TimeDisplayMode_SelectionChanged -SelectionChanged=>TimeRangeCombo_SelectionChanged SelectionChanged=>WaitStatsMetric_SelectionChanged diff --git a/Darling/Darling.Tests/CaptureDownChunkOrderTests.cs b/Darling/Darling.Tests/CaptureDownChunkOrderTests.cs index e9ff606b5f..ce5e3e5088 100644 --- a/Darling/Darling.Tests/CaptureDownChunkOrderTests.cs +++ b/Darling/Darling.Tests/CaptureDownChunkOrderTests.cs @@ -126,7 +126,8 @@ outside the collector catalog and so outside ConvertToHypertablesAsync. */ /* Eight days of one-minute blocked_process_report and five-minute deadlocks rows, plus a filler collector so the newest chunk holds rows the arms must skip past. All Kind-Unspecified: naive-UTC storage, see DarlingObservability.LogCollectionAsync. */ - var utcNow = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + /* #5608: the end of the window is a fixed point inside one chunk, so the chunk layout the plan is read against is the same at any hour. */ + var utcNow = LiveClock.AnchoredEnd(); await SeedHistoryAsync(connection, utcNow, ct); /* The two verdict rows. deadlocks: SESSION_MISSING three days ago, then a SUCCESS a minute ago — NOT diff --git a/Darling/Darling.Tests/ChartIntegerTicksPinTests.cs b/Darling/Darling.Tests/ChartIntegerTicksPinTests.cs index cbf59ac543..bd486d2892 100644 --- a/Darling/Darling.Tests/ChartIntegerTicksPinTests.cs +++ b/Darling/Darling.Tests/ChartIntegerTicksPinTests.cs @@ -34,6 +34,7 @@ namespace Darling.Tests; /// a 0-1 axis gives ticks 0 and 1, 0-2 gives 0, 1, 2, and for every maximum from 1 to 5000 the ticks are whole, /// distinct and cover the maximum. /// +[Trait("Stage", "Guard")] public sealed class ChartIntegerTicksPinTests { private static string Js(params string[] path) => ReadRepoFileLf(Path.Combine( diff --git a/Darling/Darling.Tests/CiShardCommandLineBudgetTests.cs b/Darling/Darling.Tests/CiShardCommandLineBudgetTests.cs new file mode 100644 index 0000000000..d51bc5809b --- /dev/null +++ b/Darling/Darling.Tests/CiShardCommandLineBudgetTests.cs @@ -0,0 +1,131 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.IO; +using System.Text.RegularExpressions; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5100: the sharded test steps in build.yml pass every class of the shard as a -class +/// argument. Windows CreateProcess caps the whole command line at 32,767 characters, and one Darling +/// shard's arguments alone reached ~32,660. Each step therefore splits its class list into chunks under a +/// named character budget and runs the test host once per chunk. These are source pins, because the step only +/// runs on a Windows runner. #5616: that logic is one PowerShell script per suite +/// (.github/scripts/run-darling-pg-shard.ps1, run-lite-shard.ps1) that build.yml's PR legs and +/// nightly.yml's legs both call, so the pins read the scripts; the workflow steps only pass arguments. +/// +public sealed class CiShardCommandLineBudgetTests +{ + /// The CreateProcess limit minus the ~170 characters the rest of the command takes, rounded down. + private const int MaxBudget = 30000; + + [Theory] + [InlineData("Run Darling PG tests")] + [InlineData("Run Lite tests (shard)")] + public void ShardStep_RunsTheTestHostOncePerChunk_UnderTheBudget(string stepName) + { + var step = ReadStep(stepName); + + var budget = Regex.Match(step, @"\$argBudget\s*=\s*(?\d+)"); + Assert.True(budget.Success, $"'{stepName}' names no $argBudget, so its -class list is not chunked."); + Assert.True(int.Parse(budget.Groups["n"].Value) <= MaxBudget, + $"'{stepName}' budget {budget.Groups["n"].Value} leaves less than 2,767 characters under the 32,767 limit."); + + Assert.Matches(@"(?s)foreach\s*\(\$chunk\s+in\s+\$chunks\)\s*\{.*dotnet\s+run\s+--project\s+\S+.*@chunkArgs", step); + Assert.Matches(@"-gt\s+\$argBudget", step); + } + + [Theory] + [InlineData("Run Darling PG tests")] + [InlineData("Run Lite tests (shard)")] + public void ShardStep_NeverPassesTheWholeShardListInOneInvocation(string stepName) + { + var step = ReadStep(stepName); + + foreach (Match run in Regex.Matches(step, @"dotnet\s+run\s+[^\r\n]*")) + { + if (run.Value.Contains("-list classes/json", StringComparison.Ordinal)) + { + continue; + } + Assert.DoesNotContain("$mine", run.Value); + Assert.DoesNotContain("@shardArgs", run.Value); + Assert.Contains("@chunkArgs", run.Value); + } + Assert.DoesNotContain("@shardArgs", step); + } + + [Fact] + public void ShardSteps_RunEveryChunk_AndFailTheStepWhenAnyChunkFailed() + { + foreach (var stepName in new[] { "Run Darling PG tests", "Run Lite tests (shard)" }) + { + var step = ReadStep(stepName); + Assert.Matches(@"\$LASTEXITCODE\s+-ne\s+0\)\s*\{\s*\$failedChunks\+\+", step); + Assert.Matches(@"\$failedChunks\s+-gt\s+0\)[^\r\n]*exit\s+1", step); + } + } + + [Fact] + public void DarlingStep_GivesEachChunkItsOwnTrx() + { + var step = ReadStep("Run Darling PG tests"); + Assert.Contains("TestResults/$ResultPrefix-${Shard}-$chunkNumber.trx", step); + + // ...and each caller names its own prefix, so the PR legs and the nightly's legs never write the same file. + Assert.Contains("-ResultPrefix darling-pr ", ReadWorkflowStep("build.yml", "Run Darling PG tests")); + Assert.Contains("-ResultPrefix darling-nightly ", ReadWorkflowStep("nightly.yml", "Run Darling PG tests")); + } + + [Fact] + public void ShardSteps_ChunkingBlockIsIdentical_AndNativeErrorPreferenceIsOff() + { + var darling = ReadStep("Run Darling PG tests"); + var lite = ReadStep("Run Lite tests (shard)"); + Assert.Equal(ChunkBlock(darling), ChunkBlock(lite), StringComparer.Ordinal); + + foreach (var step in new[] { darling, lite }) + { + var pref = step.IndexOf("$PSNativeCommandUseErrorActionPreference = $false", StringComparison.Ordinal); + var loop = step.IndexOf("foreach ($chunk in $chunks)", StringComparison.Ordinal); + Assert.True(pref >= 0 && pref < loop, "the step must turn the native-command error preference off before the chunk loop."); + } + } + + private static string ChunkBlock(string step) + { + var start = step.IndexOf("$argBudget =", StringComparison.Ordinal); + const string endMarker = "if ($current.Count -gt 0) { $chunks.Add($current.ToArray()) }"; + var end = step.IndexOf(endMarker, StringComparison.Ordinal); + Assert.True(start >= 0 && end > start, "the step has no chunking block."); + return step.Substring(start, end + endMarker.Length - start).Replace("\r\n", "\n"); + } + + /// The shard script the named step calls (#5616), linked into Fixtures by Darling.Tests.csproj. + private static string ReadStep(string stepName) + { + var file = stepName == "Run Darling PG tests" ? "run-darling-pg-shard.ps1" : "run-lite-shard.ps1"; + var path = Path.Combine(AppContext.BaseDirectory, "Fixtures", file); + Assert.True(File.Exists(path), $"{file} was not copied beside the test binary (Darling.Tests.csproj links it into Fixtures\\)."); + return File.ReadAllText(path).Replace("\r\n", "\n"); + } + + private static string ReadWorkflowStep(string workflow, string stepName) + { + var path = Path.Combine(AppContext.BaseDirectory, "Fixtures", workflow); + Assert.True(File.Exists(path), $"{workflow} was not copied beside the test binary (Darling.Tests.csproj links it into Fixtures\\)."); + var text = File.ReadAllText(path); + var start = text.IndexOf("- name: " + stepName, StringComparison.Ordinal); + Assert.True(start >= 0, $"{workflow} has no step named '{stepName}'."); + var next = Regex.Match(text[(start + 1)..], @"\r?\n - name: |\r?\n [a-z][\w-]*:\r?\n"); + return next.Success ? text.Substring(start, next.Index + 1) : text[start..]; + } +} diff --git a/Darling/Darling.Tests/CiShardHangReportTests.cs b/Darling/Darling.Tests/CiShardHangReportTests.cs new file mode 100644 index 0000000000..5da346b654 --- /dev/null +++ b/Darling/Darling.Tests/CiShardHangReportTests.cs @@ -0,0 +1,225 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.IO; +using System.Runtime.CompilerServices; +using System.Text.RegularExpressions; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5587: run 37824250187 attempt 1, Darling PG shard 3, ran to the JOB's 30-minute timeout. GitHub cancelled the +/// job, left every later step (the stop, both uploads) pending, kept no log, and the per-test timing was never +/// written, so nothing said which test was running or what it waited on. The shard test steps now carry a +/// step-level timeout (the step fails, the job goes on, and the always() steps run), the runner's +/// -longRunning report (a test that has run for 5 minutes is named in the log while the step is alive) and a +/// kept copy of the runner output. These are source pins on the workflow text, because the steps only run on a +/// Windows runner. +/// +public sealed class CiShardHangReportTests +{ + /// Under the job's 30 minutes with room for the setup before the step and the stop and uploads after it. + private const int MaxStepTimeoutMinutes = 25; + + /// The slowest normal Darling test measured 120 s (the store-upgrade classes), so a report below it is noise. + private const int MinLongRunningSeconds = 150; + + /// Well under the step timeout, or the report would arrive after the step was killed. + private const int MaxLongRunningSeconds = 600; + + [Theory] + [InlineData("Run Darling PG tests")] + [InlineData("Run Lite tests (shard)")] + public void ShardStep_HasAStepTimeout_UnderTheJobTimeout(string stepName) + { + var step = ReadStep(stepName); + + var timeout = Regex.Match(step, @"(?m)^ timeout-minutes:\s*(?\d+)\s*$"); + Assert.True(timeout.Success, + $"'{stepName}' has no step-level timeout-minutes. The job's own timeout cancels the job and leaves the upload steps pending (#5587)."); + var minutes = int.Parse(timeout.Groups["n"].Value); + Assert.InRange(minutes, 10, MaxStepTimeoutMinutes); + } + + [Theory] + [InlineData("Run Darling PG tests")] + [InlineData("Run Lite tests (shard)")] + public void ShardStep_AsksTheRunnerToNameALongRunningTest_AndKeepsTheRunnerOutput(string stepName) + { + // #5616: the chunk loop lives in the shard script both workflows call, so the runner's flags are read from it. + var step = ReadShardScript(stepName); + + var longRunning = Regex.Match(step, @"dotnet\s+run\s+[^\r\n]*@chunkArgs[^\r\n]*\s-longRunning\s+(?\d+)\b"); + Assert.True(longRunning.Success, $"'{stepName}' does not pass -longRunning to the chunk's runner, so a hung test is not named."); + Assert.InRange(int.Parse(longRunning.Groups["n"].Value), MinLongRunningSeconds, MaxLongRunningSeconds); + + Assert.Matches(@"dotnet\s+run\s+[^\r\n]*@chunkArgs[^\r\n]*\|\s*Tee-Object\s+-FilePath\s+TestResults/(darling|lite)-runner-\$\{Shard\}\.log\s+-Append", step); + Assert.Matches(@"New-Item\s+-ItemType\s+Directory\s+-Force\s+-Path\s+TestResults", step); + } + + [Fact] + public void DarlingUploads_RunWhateverTheTestStepDid() + { + var timings = ReadStep("Upload Darling test timings"); + Assert.Matches(@"(?m)^ if:\s*always\(\)", timings); + + var runner = ReadStep("Upload Darling runner output"); + Assert.Matches(@"(?m)^ if:\s*always\(\)", runner); + Assert.Contains("TestResults/darling-runner-*.log", runner, StringComparison.Ordinal); + Assert.Contains("darling-runner-log-${{ matrix.shard }}", runner, StringComparison.Ordinal); + Assert.Matches(@"(?m)^ timeout-minutes:\s*\d+", runner); + + var failure = ReadStep("Upload PG log and test results on failure"); + Assert.Matches(@"(?m)^ if:\s*\(failure\(\) \|\| cancelled\(\)\)", failure); + } + + [Fact] + public void LiteUploads_RunWhateverTheTestStepDid() + { + var timings = ReadStep("Upload Lite test timings"); + Assert.Matches(@"(?m)^ if:\s*always\(\)", timings); + + var runner = ReadStep("Upload Lite runner output"); + Assert.Matches(@"(?m)^ if:\s*always\(\)", runner); + Assert.Contains("TestResults/lite-runner-*.log", runner, StringComparison.Ordinal); + Assert.Contains("lite-runner-log-${{ matrix.shard }}", runner, StringComparison.Ordinal); + } + + /// + /// #5587, #5616, nightly.yml: the nightly's shard steps call the same scripts as build.yml's, so the hang report + /// (-longRunning and the teed runner log) is the script's, and the step must pass the script's shard + /// arguments and no event: the nightly runs every class, so it must not pass -EventName. + /// + [Theory] + [InlineData("Run Lite tests (shard)", "run-lite-shard.ps1")] + [InlineData("Run Darling PG tests", "run-darling-pg-shard.ps1")] + public void NightlyStep_CallsTheSharedShardScript_WithItsShardAndNoEvent(string stepName, string script) + { + var step = ReadStep(stepName, "nightly.yml"); + + Assert.Matches(@"(?m)^ shell:\s*pwsh\s*$", step); + Assert.Contains("./.github/scripts/" + script + " -Shard ${{ matrix.shard }} -ShardCount ${{ strategy.job-total }}", step, StringComparison.Ordinal); + Assert.DoesNotContain("-EventName", step, StringComparison.Ordinal); + Assert.DoesNotContain("-GuardBuildUsed", step, StringComparison.Ordinal); + Assert.DoesNotContain("-TimingRun", step, StringComparison.Ordinal); + Assert.DoesNotContain("dotnet run", step, StringComparison.Ordinal); + + // The hang report and the exit-code propagation live once, in the script: the runner is asked for -longRunning + // and its output is teed, and a failing chunk fails the script (and so the step). + var body = ReadShardScript(stepName); + var longRunning = Regex.Match(body, @"dotnet\s+run\s+[^\r\n]*@chunkArgs[^\r\n]*\s-longRunning\s+(?\d+)\b"); + Assert.True(longRunning.Success, $"{script} does not pass -longRunning to the chunk's runner, so a hung test is not named."); + Assert.InRange(int.Parse(longRunning.Groups["n"].Value), MinLongRunningSeconds, MaxLongRunningSeconds); + Assert.Matches(@"\$failedChunks\s+-gt\s+0\)[^\r\n]*exit\s+1", body); + } + + /// + /// #5587, #5616, nightly.yml: step timeouts from numbers. Lite: build.yml's four legs ran the step for 5.6 to 19.9 + /// minutes over the last 7 successful dev pushes; the nightly's hash cut is less even, so the step gets twice that + /// (40) in a job of 50, and 19.9 is 50% of it, under the 70% bar. Darling: build.yml's six legs ran 5.5 to 10.9 + /// minutes, the step gets build.yml's 22 in a job of 40, and 10.9 is 50% of it. + /// + [Fact] + public void NightlyStepTimeouts_FollowTheMeasuredNumbers() + { + var yml = File.ReadAllText(Path.Combine(AppContext.BaseDirectory, "Fixtures", "nightly.yml")).Replace("\r\n", "\n"); + foreach (var (job, stepName, maxJob) in new[] { ("lite-tests", "Run Lite tests (shard)", 50), ("darling-pg", "Run Darling PG tests", 40) }) + { + var step = ReadStep(stepName, "nightly.yml"); + var timeout = Regex.Match(step, @"(?m)^ timeout-minutes:\s*(?\d+)\s*$"); + Assert.True(timeout.Success, $"nightly.yml '{stepName}' has no step-level timeout-minutes."); + + var jobStart = yml.IndexOf("\n " + job + ":\n", StringComparison.Ordinal); + Assert.True(jobStart >= 0, $"nightly.yml has no job '{job}'."); + var jobTimeout = Regex.Match(yml[jobStart..], @"(?m)^ timeout-minutes:\s*(?\d+)\s*$"); + Assert.True(jobTimeout.Success && int.Parse(jobTimeout.Groups["n"].Value) == maxJob, + $"nightly.yml job '{job}' timeout is not {maxJob}."); + Assert.InRange(int.Parse(timeout.Groups["n"].Value), 20, maxJob - 10); + } + } + + [Fact] + public void NightlyUploads_RunWhateverTheTestStepDid() + { + var lite = ReadStep("Upload Lite runner output", "nightly.yml"); + Assert.Matches(@"(?m)^ if:\s*always\(\)", lite); + Assert.Contains("TestResults/lite-runner-*.log", lite, StringComparison.Ordinal); + Assert.Contains("lite-runner-log-nightly-${{ matrix.shard }}", lite, StringComparison.Ordinal); + Assert.Matches(@"(?m)^ timeout-minutes:\s*\d+", lite); + + var darling = ReadStep("Upload Darling runner output", "nightly.yml"); + Assert.Matches(@"(?m)^ if:\s*always\(\)", darling); + Assert.Contains("TestResults/darling-runner-*.log", darling, StringComparison.Ordinal); + Assert.Contains("darling-runner-log-nightly-${{ matrix.shard }}", darling, StringComparison.Ordinal); + Assert.Matches(@"(?m)^ timeout-minutes:\s*\d+", darling); + + var failure = ReadStep("Upload PG log and test results on failure", "nightly.yml"); + Assert.Matches(@"(?m)^ if:\s*failure\(\) \|\| cancelled\(\)", failure); + + Assert.DoesNotContain("-runner-", ReadStep("Upload Lite test timings", "nightly.yml"), StringComparison.Ordinal); + } + + [Fact] + public void RunnerLogs_AreTheirOwnArtifacts_NotPartOfTheTimingArtifacts() + { + // The shadow check and the Lite shard packer read every file of the *-tests-timing-* artifacts. + foreach (var name in new[] { "Upload Darling test timings", "Upload Lite test timings" }) + { + Assert.DoesNotContain("-runner-", ReadStep(name), StringComparison.Ordinal); + } + } + + /// + /// #5587: the shard-3 classes were read for a wait with no end. The one found is the cancelled refresh in + /// RollupBackfillLiveTests: it waited on a task whose only way out was a cancel to a backend that was blocked + /// on a lock the test itself still held, and the refresh's own command timeout is an hour. The wait is bounded now. + /// + [Fact] + public void RollupBackfillLiveTests_TheCancelledRefreshWait_IsBoundedWhileTheChunkLockIsHeld() + { + var source = ReadSiblingSource("RollupBackfillLiveTests.cs"); + + Assert.Contains("refresh.WaitAsync(CancelledRefreshLimit", source, StringComparison.Ordinal); + Assert.Single(Regex.Matches(source, @"_ = await refresh;")); + var released = source.IndexOf("lockTransaction.RollbackAsync(CancellationToken.None)", StringComparison.Ordinal); + Assert.True(released >= 0, "the chunk lock is no longer released in a finally."); + Assert.True( + source.IndexOf("if (cancelDidNotStopTheRefresh)", released, StringComparison.Ordinal) is var guarded && guarded > released + && source.IndexOf("_ = await refresh;", guarded, StringComparison.Ordinal) > guarded, + "the only unbounded await of the refresh must come after the chunk lock is released."); + } + + private static string ReadSiblingSource(string file, [CallerFilePath] string callerFile = "") => + File.ReadAllText(Path.Combine(Path.GetDirectoryName(callerFile)!, file)).Replace("\r\n", "\n"); + + /// The shard script the named step calls (#5616), read from the copy linked into Fixtures. + private static string ReadShardScript(string stepName) + { + var file = stepName == "Run Darling PG tests" ? "run-darling-pg-shard.ps1" : "run-lite-shard.ps1"; + var path = Path.Combine(AppContext.BaseDirectory, "Fixtures", file); + Assert.True(File.Exists(path), $"{file} was not copied beside the test binary (Darling.Tests.csproj links it into Fixtures\\)."); + return File.ReadAllText(path).Replace("\r\n", "\n"); + } + + private static string ReadStep(string stepName, string workflow = "build.yml") + { + var path = Path.Combine(AppContext.BaseDirectory, "Fixtures", workflow); + Assert.True(File.Exists(path), $"{workflow} was not copied beside the test binary (Darling.Tests.csproj links it into Fixtures\\)."); + var text = File.ReadAllText(path); + var start = text.IndexOf("- name: " + stepName + "\n", StringComparison.Ordinal); + if (start < 0) + { + start = text.IndexOf("- name: " + stepName + "\r\n", StringComparison.Ordinal); + } + Assert.True(start >= 0, $"{workflow} has no step named '{stepName}'."); + var next = Regex.Match(text[(start + 1)..], @"\r?\n - name: |\r?\n [a-z][\w-]*:\r?\n"); + return next.Success ? text.Substring(start, next.Index + 1) : text[start..]; + } +} diff --git a/Darling/Darling.Tests/ClipboardWriteCensusTests.cs b/Darling/Darling.Tests/ClipboardWriteCensusTests.cs index 9f3e2458a7..1752e333e2 100644 --- a/Darling/Darling.Tests/ClipboardWriteCensusTests.cs +++ b/Darling/Darling.Tests/ClipboardWriteCensusTests.cs @@ -24,6 +24,8 @@ namespace Darling.Tests; /// either call does not trip it). deprecated/ is out of scope (no work there unless asked for by /// name) and is excluded, matching every other census in this file's family. /// +[Trait("Stage", "Guard")] +[Trait("Reads", "Lite")] public sealed class ClipboardWriteCensusTests { [Fact] diff --git a/Darling/Darling.Tests/CollectionCaveatLedgerTests.cs b/Darling/Darling.Tests/CollectionCaveatLedgerTests.cs index f8655dc981..43f2b6783c 100644 --- a/Darling/Darling.Tests/CollectionCaveatLedgerTests.cs +++ b/Darling/Darling.Tests/CollectionCaveatLedgerTests.cs @@ -35,6 +35,7 @@ namespace Darling.Tests; /// (analysis_time is the latest pass's, entries_from the newest FAILING pass's, so a recovery is /// readable as a recovery rather than as a current fault). /// +[Trait("Reads", "Lite")] public sealed class CollectionCaveatLedgerTests { private static readonly JsonSerializerOptions Options = new JsonSerializerOptions { WriteIndented = false }; @@ -387,6 +388,7 @@ public void TheSharedLedger_IsOneInstance_AndStampsItsOwnStart() /// /// /// +[Trait("Reads", "Lite")] public sealed class CollectionCaveatLedgerParityTests { private const string Attach = "CollectionCaveatLedger.Shared.Attach("; diff --git a/Darling/Darling.Tests/CollectionCaveatsParityTests.cs b/Darling/Darling.Tests/CollectionCaveatsParityTests.cs index 1e8bfa2706..8f7ca2210d 100644 --- a/Darling/Darling.Tests/CollectionCaveatsParityTests.cs +++ b/Darling/Darling.Tests/CollectionCaveatsParityTests.cs @@ -34,6 +34,7 @@ namespace Darling.Tests; /// serialized on a clean pass and move its bytes. The descriptions name the field on both SKUs. /// /// +[Trait("Reads", "Lite")] public sealed class CollectionCaveatsParityTests { private const string Record = "context.RecordCollectionFailure("; diff --git a/Darling/Darling.Tests/CollectionCaveatsWebTests.cs b/Darling/Darling.Tests/CollectionCaveatsWebTests.cs new file mode 100644 index 0000000000..c4d6f5b7a0 --- /dev/null +++ b/Darling/Darling.Tests/CollectionCaveatsWebTests.cs @@ -0,0 +1,304 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Diagnostics; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/* #1776 own-store: each live fact mints its own scratch database through ScratchPostgres and never touches the shared + store's tables, so it cannot race the live collection. */ + +/// +/// The collection caveats on the web Collection Health tabs (#4843): get_collection_health carries a +/// stored_caveats array from collect.analysis_collection_caveats, the SQL Server and PostgreSQL tabs draw it +/// as a grid of the desktop viewer's four columns, and a server with none shows no grid. No new tool carries it. +/// +public sealed class CollectionCaveatsWebTests +{ + private static JsonElement RunHarness() + { + var psi = new ProcessStartInfo("node") { RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false }; + psi.ArgumentList.Add(PathTo("Darling", "Darling.Tests", "web-collection-caveats-harness.mjs")); + psi.ArgumentList.Add(PathTo("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js")); + Process proc; + try + { + proc = Process.Start(psi)!; + } + catch (Win32Exception) + { + Assert.Skip("Node is not installed, so the shipped page scripts cannot be run."); + return default; + } + + using (proc) + { + var error = proc.StandardError.ReadToEndAsync(); + var output = proc.StandardOutput.ReadToEnd().Trim(); + if (!proc.WaitForExit(30000)) + { + proc.Kill(entireProcessTree: true); + Assert.Fail("the collection-caveats harness did not finish in 30 s"); + } + + Assert.True(proc.ExitCode == 0, "the collection-caveats harness failed: " + error.Result); + using var doc = JsonDocument.Parse(output); + return doc.RootElement.Clone(); + } + } + + [Theory] + [InlineData("sqlserver")] + [InlineData("postgres")] + public void AServerWithNoCaveats_ShowsNoGrid_AndOneWithCaveatsShowsTheFourColumns(string engine) + { + var e = RunHarness().GetProperty(engine); + + var empty = e.GetProperty("empty"); + Assert.True(empty.GetProperty("hidden").GetBoolean(), "a payload without stored_caveats must leave the panel hidden"); + Assert.Empty(empty.GetProperty("rows").EnumerateArray()); + + var rows = e.GetProperty("rows"); + Assert.False(rows.GetProperty("hidden").GetBoolean()); + Assert.Equal(new[] { "Family", "Reason", "Since", "Last seen" }, rows.GetProperty("heads").EnumerateArray().Select(h => h.GetString()).ToArray()); + var cells = rows.GetProperty("rows").EnumerateArray().Select(r => r.EnumerateArray().Select(c => c.GetString()).ToArray()).ToArray(); + Assert.Equal(2, cells.Length); + Assert.Equal(new[] { "plans", "timeout" }, cells[0].Take(2).ToArray()); + Assert.Equal(new[] { "waits", "missing_schema" }, cells[1].Take(2).ToArray()); + } + + [Theory] + [InlineData("sqlserver")] + [InlineData("postgres")] + public void WhileTheReadIsOutstanding_AServerNeverSeenShowsNoCard_AndARebuildOfOneWithCaveatsKeepsIt(string engine) + { + var e = RunHarness().GetProperty(engine); + + var pending = e.GetProperty("pending"); + Assert.True(pending.GetProperty("present").GetBoolean()); + Assert.True(pending.GetProperty("hidden").GetBoolean(), "a clean server must not flash the card while get_collection_health is still running"); + + var rebuild = e.GetProperty("rebuild"); + Assert.False(rebuild.GetProperty("hidden").GetBoolean(), "a rebuild of a server whose last read had caveats must keep the card visible"); + } + + [Theory] + [InlineData("sqlserver")] + [InlineData("postgres")] + public void TheGridRidesTheHealthRead_WithItsParams_AndNoSecondRead(string engine) + { + var e = RunHarness().GetProperty(engine); + foreach (var arm in new[] { "empty", "rows" }) + { + var a = e.GetProperty(arm); + Assert.Equal(new[] { "get_collection_health?server=srv-a" }, a.GetProperty("healthReads").EnumerateArray().Select(r => r.GetString()).ToArray()); + Assert.False(a.GetProperty("anyCaveatTool").GetBoolean()); + } + } + + [Fact] + public void BothTabsShareOnePanel_ReadingTheHealthPayloadsArray_AndNoCaveatToolExists() + { + var tabs = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "pages", "server-tabs.js"); + Assert.Contains("rowsKey: \"stored_caveats\"", tabs, StringComparison.Ordinal); + Assert.Contains("hideWhenNoRows: true", tabs, StringComparison.Ordinal); + Assert.Equal(2, tabs.Split(" COLLECTION_CAVEATS_PANEL,", StringSplitOptions.None).Length - 1); + Assert.DoesNotContain("get_collection_caveats", tabs, StringComparison.Ordinal); + Assert.DoesNotContain("get_collection_caveats", ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "Mcp", "DarlingMcpDataTools.cs"), StringComparison.Ordinal); + } + + private static DarlingCollectionCaveatReader.Caveat Row(string family) => + new(family, "timeout", new DateTime(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc), new DateTime(2026, 1, 2, 0, 0, 0, DateTimeKind.Utc)); + + [Fact] + public void ACleanServersPayloadGainsNoKey_AndCaveatsAreAttachedAsAnArray() + { + var cleanJson = JsonSerializer.Serialize(new { server = "a" }); + Assert.Equal(cleanJson, DarlingCollectionCaveatReader.AttachToJson(cleanJson, Array.Empty())); + + var with = DarlingCollectionCaveatReader.AttachToJson(cleanJson, new[] { Row("plans") }); + using var doc = JsonDocument.Parse(with); + var entry = Assert.Single(doc.RootElement.GetProperty("stored_caveats").EnumerateArray()); + Assert.Equal("plans", entry.GetProperty("family").GetString()); + Assert.Equal("timeout", entry.GetProperty("reason").GetString()); + Assert.StartsWith("2026-01-01T00:00:00", entry.GetProperty("first_seen_utc").GetString(), StringComparison.Ordinal); + Assert.StartsWith("2026-01-02T00:00:00", entry.GetProperty("last_seen_utc").GetString(), StringComparison.Ordinal); + } + + [Fact] + public async Task AMissingTableAnswersNoCaveats_AndAnUnreadableOneAnswersNoneWithOneWarning() + { + var warnings = new List(); + var missing = await DarlingCollectionCaveatReader.ReadAsync( + () => throw new PostgresException("relation does not exist", "ERROR", "ERROR", "42P01"), warnings.Add); + Assert.Empty(missing); + Assert.Empty(warnings); + + var denied = await DarlingCollectionCaveatReader.ReadAsync( + () => throw new PostgresException("permission denied", "ERROR", "ERROR", "42501"), warnings.Add); + Assert.Empty(denied); + Assert.Single(warnings); + + await Assert.ThrowsAsync(() => DarlingCollectionCaveatReader.ReadAsync( + () => throw new PostgresException("boom", "ERROR", "ERROR", "XX000"), warnings.Add)); + } + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static async Task HealthAsync(NpgsqlDataSource postgres, string server, CancellationToken ct) + { + var json = await DarlingMcpDataTools.GetCollectionHealth(postgres, server, cancellationToken: ct); + var doc = JsonDocument.Parse(json); + Assert.True(doc.RootElement.TryGetProperty("collectors", out _), json.Length > 400 ? json[..400] : json); + return doc; + } + + [Fact] + public async Task TheHealthReadCarriesTheStoresCaveats_OmitsThemWhenNone_AndSurvivesAStoreWithoutTheTable() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), "Set DARLING_TEST_PG to a Postgres connection string to run the collection-caveat health pins (each mints its own scratch database)."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + var bodySucceeded = false; + try + { + const int serverId = 480_141; + const string server = "caveat-health-a"; + var now = DateTime.UtcNow; + await using (var register = new NpgsqlCommand( + """ + INSERT INTO servers (server_id, server_name, display_name, is_enabled, sql_major_version, created_date, modified_date) + VALUES ($1, $2, $2, TRUE, 15, $3, $3) + """, connection)) + { + register.Parameters.AddWithValue(serverId); + register.Parameters.AddWithValue(server); + register.Parameters.AddWithValue(DateTime.SpecifyKind(now, DateTimeKind.Unspecified)); + await register.ExecuteNonQueryAsync(ct); + } + + await using (var log = new NpgsqlCommand( + """ + INSERT INTO collection_log (log_id, collection_time, server_id, server_name, collector_name, status, duration_ms, rows_collected) + VALUES ($1, $2, $3, $4, 'wait_stats', 'SUCCESS', 120, 10) + """, connection)) + { + log.Parameters.AddWithValue(CollectionIdGenerator.Next()); + log.Parameters.AddWithValue(DateTime.SpecifyKind(now.AddMinutes(-1), DateTimeKind.Unspecified)); + log.Parameters.AddWithValue(serverId); + log.Parameters.AddWithValue(server); + await log.ExecuteNonQueryAsync(ct); + } + + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + + // No rows: the key is absent, not null and not empty. + using (var clean = await HealthAsync(postgres, server, ct)) + { + Assert.False(clean.RootElement.TryGetProperty("stored_caveats", out _)); + } + + // Two rows through the store's own writer, read back through the real tool. + await CollectionCaveatStore.ApplyPassAsync( + postgres, serverId, + new[] { new CollectionCaveatStore.UnreadFamily("waits", "missing_schema"), new CollectionCaveatStore.UnreadFamily("plans", "timeout") }, + now.AddHours(-2), null, ct); + await CollectionCaveatStore.ApplyPassAsync( + postgres, serverId, + new[] { new CollectionCaveatStore.UnreadFamily("waits", "missing_schema"), new CollectionCaveatStore.UnreadFamily("plans", "timeout") }, + now, null, ct); + using (var with = await HealthAsync(postgres, server, ct)) + { + var entries = with.RootElement.GetProperty("stored_caveats").EnumerateArray().ToArray(); + Assert.Equal(new[] { "plans", "waits" }, entries.Select(x => x.GetProperty("family").GetString()).ToArray()); + Assert.Equal(new[] { "timeout", "missing_schema" }, entries.Select(x => x.GetProperty("reason").GetString()).ToArray()); + var first = DateTime.Parse(entries[0].GetProperty("first_seen_utc").GetString()!, System.Globalization.CultureInfo.InvariantCulture, System.Globalization.DateTimeStyles.RoundtripKind); + var last = DateTime.Parse(entries[0].GetProperty("last_seen_utc").GetString()!, System.Globalization.CultureInfo.InvariantCulture, System.Globalization.DateTimeStyles.RoundtripKind); + Assert.True(last - first > TimeSpan.FromHours(1.9), "Since keeps the first sighting and Last seen takes the newest"); + } + + // Another server's rows are not this server's: B has its own caveat, and each side sees only its own. + const int otherId = 480_142; + const string otherServer = "caveat-health-b"; + await using (var registerB = new NpgsqlCommand( + """ + INSERT INTO servers (server_id, server_name, display_name, is_enabled, sql_major_version, created_date, modified_date) + VALUES ($1, $2, $2, TRUE, 15, $3, $3) + """, connection)) + { + registerB.Parameters.AddWithValue(otherId); + registerB.Parameters.AddWithValue(otherServer); + registerB.Parameters.AddWithValue(DateTime.SpecifyKind(now, DateTimeKind.Unspecified)); + await registerB.ExecuteNonQueryAsync(ct); + } + + await using (var logB = new NpgsqlCommand( + """ + INSERT INTO collection_log (log_id, collection_time, server_id, server_name, collector_name, status, duration_ms, rows_collected) + VALUES ($1, $2, $3, $4, 'wait_stats', 'SUCCESS', 120, 10) + """, connection)) + { + logB.Parameters.AddWithValue(CollectionIdGenerator.Next()); + logB.Parameters.AddWithValue(DateTime.SpecifyKind(now.AddMinutes(-1), DateTimeKind.Unspecified)); + logB.Parameters.AddWithValue(otherId); + logB.Parameters.AddWithValue(otherServer); + await logB.ExecuteNonQueryAsync(ct); + } + + await CollectionCaveatStore.ApplyPassAsync( + postgres, otherId, + new[] { new CollectionCaveatStore.UnreadFamily("locks", "timeout") }, + now, null, ct); + using (var a = await HealthAsync(postgres, server, ct)) + { + Assert.Equal(new[] { "plans", "waits" }, a.RootElement.GetProperty("stored_caveats").EnumerateArray().Select(x => x.GetProperty("family").GetString()).ToArray()); + } + + using (var b = await HealthAsync(postgres, otherServer, ct)) + { + var only = Assert.Single(b.RootElement.GetProperty("stored_caveats").EnumerateArray()); + Assert.Equal("locks", only.GetProperty("family").GetString()); + } + + // A store below V141 has no table: no caveats, and the health read still answers. + await using (var drop = new NpgsqlCommand("DROP TABLE collect.analysis_collection_caveats", connection)) + { + await drop.ExecuteNonQueryAsync(ct); + } + + using (var old = await HealthAsync(postgres, server, ct)) + { + Assert.False(old.RootElement.TryGetProperty("stored_caveats", out _)); + Assert.True(old.RootElement.GetProperty("collectors").GetArrayLength() > 0); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } +} diff --git a/Darling/Darling.Tests/CollectionGapAtStartTests.cs b/Darling/Darling.Tests/CollectionGapAtStartTests.cs new file mode 100644 index 0000000000..73de1f0584 --- /dev/null +++ b/Darling/Darling.Tests/CollectionGapAtStartTests.cs @@ -0,0 +1,351 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Alerting; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Notifications; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5450: at start, one "Collection Gap At Start" alert when the newest pre-start collection is 15 minutes or +/// more before the start. The evaluator holds the gate; the live test reads the newest time with the real SQL. +/// +/* #1776 own-store: deliberately NOT [Collection("live-postgres")]. The live test reaches DARLING_TEST_PG only to + CREATE and DROP its own database through ScratchPostgres and works entirely inside it. */ +[Collection("gap-cache-serial")] +public sealed class CollectionGapAtStartTests +{ + private static readonly CancellationToken Ct = TestContext.Current.CancellationToken; + private static readonly DateTime Start = new(2026, 10, 6, 12, 0, 0, DateTimeKind.Utc); + + private static DarlingSelfAlertEvaluator.CollectionGapReport Report(int minutesBeforeStart) => + new(Start.AddMinutes(-minutesBeforeStart), Start); + + [Fact] + public async Task ThirtyMinuteGap_FiresOnce_WithTheMessageAndTheValues() + { + var h = new DarlingSelfAlertTests.Harness(); + var e = h.Build(); + + await e.EvaluateCollectionGapAtStartAsync(Report(30), Ct); + + var fired = Assert.Single(h.Deliverer.Outcomes); + Assert.Equal("Collection Gap At Start", fired.MetricName); + Assert.Equal(AlertSeverityLevel.Warning, fired.Severity); + Assert.Equal("collectiongapatstart", fired.ServerKey); + Assert.Equal( + "Darling was not collecting from 2026-10-06 11:30 to 2026-10-06 12:00 UTC (30 minutes). " + + "Nothing was collected for any server in that window. " + + "If this was not a planned stop, check why the service or its host was down.", + fired.DetailText); + Assert.Equal(30, fired.NumericCurrentValue); + Assert.Equal(15, fired.NumericThresholdValue); + } + + [Fact] + public async Task ExactlyFifteenMinutes_Fires_AndFourteen_DoesNot() + { + var h = new DarlingSelfAlertTests.Harness(); + var e = h.Build(); + + await e.EvaluateCollectionGapAtStartAsync(Report(14), Ct); + Assert.Empty(h.Deliverer.Outcomes); + + await e.EvaluateCollectionGapAtStartAsync(Report(15), Ct); + Assert.Single(h.Deliverer.Outcomes); + } + + [Fact] + public async Task FiveMinuteGap_FiresNothing() + { + var h = new DarlingSelfAlertTests.Harness(); + var e = h.Build(); + + await e.EvaluateCollectionGapAtStartAsync(Report(5), Ct); + + Assert.Empty(h.Deliverer.Outcomes); + } + + [Fact] + public async Task EmptyStore_FiresNothing() + { + var h = new DarlingSelfAlertTests.Harness(); + var e = h.Build(); + + await e.EvaluateCollectionGapAtStartAsync(null, Ct); + + Assert.Empty(h.Deliverer.Outcomes); + } + + [Fact] + public async Task AlertsDisabled_FiresNothing() + { + var h = new DarlingSelfAlertTests.Harness(); + h.Settings.AlertsEnabled = false; + var e = h.Build(); + + await e.EvaluateCollectionGapAtStartAsync(Report(30), Ct); + + Assert.Empty(h.Deliverer.Outcomes); + } + + [Fact] + public void TheMetric_IsSelfMonitorFamily_Warning_AndNumeric() + { + Assert.Equal("Collection Gap At Start", DarlingSelfAlertEvaluator.CollectionGapAtStartMetric); + Assert.Equal(AlertFamily.SelfMonitor, AlertFamily.Of(DarlingSelfAlertEvaluator.CollectionGapAtStartMetric)); + Assert.Equal("WARNING", AlertSeverity.ForMetric(DarlingSelfAlertEvaluator.CollectionGapAtStartMetric).BadgeText); + /* A measurement (minutes), so never the state-only dash. */ + Assert.False(PerformanceMonitor.Common.AlertMetricClassifier.IsStateOnly(DarlingSelfAlertEvaluator.CollectionGapAtStartMetric)); + Assert.Equal(15, DarlingSelfAlertEvaluator.CollectionGapAtStartThreshold.TotalMinutes); + } + + [Fact] + public async Task GapIsMeasuredToTheProcessStart_NotToTheEndOfALongMigration() + { + /* The report's StartUtc is the process start. A migration that then ran for 20 minutes does not + enter the measure: 10 minutes from the last collection to the process start fires nothing. */ + var h = new DarlingSelfAlertTests.Harness(); + var e = h.Build(); + var processStart = Start; + var report = new DarlingSelfAlertEvaluator.CollectionGapReport(processStart.AddMinutes(-10), processStart); + _ = processStart.AddMinutes(20); /* migrations finished here; the old code measured to this */ + await e.EvaluateCollectionGapAtStartAsync(report, Ct); + Assert.Empty(h.Deliverer.Outcomes); + + /* A negative gap (a row newer than the start) fires nothing either. */ + await e.EvaluateCollectionGapAtStartAsync(new DarlingSelfAlertEvaluator.CollectionGapReport(processStart.AddMinutes(5), processStart), Ct); + Assert.Empty(h.Deliverer.Outcomes); + } + + private static async Task<(ScratchPostgres Scratch, NpgsqlConnection Connection)> OpenStoreAsync(string baseCs, bool hypertable) + { + var scratch = await ScratchPostgres.CreateAsync(baseCs, Ct); + var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(Ct); + await PgMigrations.MigrateAsync(connection, Ct); + if (!hypertable) + { + /* The migration makes a hypertable when the cluster has TimescaleDB. Swap in an ordinary table with + the one shipped index, so the plan is the one a store without TimescaleDB gets. */ + await ExecAsync(connection, @" +DROP TABLE collect.collection_log CASCADE; +CREATE TABLE collect.collection_log (log_id bigint NOT NULL, server_id integer NOT NULL, server_name text, + collector_name text, collection_time timestamp NOT NULL, status text); +CREATE INDEX idx_collection_log_time ON collect.collection_log(server_id, collection_time);"); + } + + if (hypertable) + { + Assert.True(await TimescaleSupport.TryEnableAsync(connection, null, Ct), "TimescaleDB must be enabled on the test cluster"); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, Ct); + Assert.True(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, null, Ct)); + } + + return (scratch, connection); + } + + private static async Task AddServerAsync(NpgsqlConnection connection, int id, bool enabled) => + await ExecAsync(connection, + "INSERT INTO config.config_monitored_servers (server_id, name, host, is_enabled) VALUES (@id, @n, 'h', @e)", + new NpgsqlParameter("id", id), new NpgsqlParameter("n", "gap-" + id), new NpgsqlParameter("e", enabled)); + + private static async Task SeedAsync(NpgsqlConnection connection, int serverId, DateTime newest, int days) + { + var u = DateTime.SpecifyKind(newest, DateTimeKind.Unspecified); + await ExecAsync(connection, @" +INSERT INTO collect.collection_log (log_id, server_id, server_name, collector_name, collection_time, status) +SELECT (@sid::bigint * 1000000) + row_number() OVER (ORDER BY t), @sid, 'gap-host', 'cpu', t, 'SUCCESS' +FROM generate_series(@newest - make_interval(days => @days), @newest, INTERVAL '1 hour') AS t", + new NpgsqlParameter("sid", serverId), + new NpgsqlParameter("days", days), + new NpgsqlParameter("newest", NpgsqlDbType.Timestamp) { Value = u }); + } + + private static async Task ExplainJsonAsync(NpgsqlConnection connection) + { + await using var explain = new NpgsqlCommand("EXPLAIN (FORMAT JSON) " + DarlingSelfAlertEvaluator.NewestCollectionTimeSql, connection); + return Convert.ToString(await explain.ExecuteScalarAsync(Ct), CultureInfo.InvariantCulture)!; + } + + private static void CollectScans(System.Text.Json.JsonElement e, List scans) + { + if (e.ValueKind == System.Text.Json.JsonValueKind.Object) + { + if (e.TryGetProperty("Node Type", out var t) && t.GetString()!.EndsWith("Scan", StringComparison.Ordinal)) + { + scans.Add(t.GetString() + ": " + (e.TryGetProperty("Relation Name", out var r) ? r.GetString() : "")); + } + + foreach (var p in e.EnumerateObject()) + { + CollectScans(p.Value, scans); + } + } + else if (e.ValueKind == System.Text.Json.JsonValueKind.Array) + { + foreach (var x in e.EnumerateArray()) + { + CollectScans(x, scans); + } + } + } + + private static string? Env() => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + /// + /// Now, cut to whole microseconds. collection_log.collection_time is a PostgreSQL timestamp, which keeps + /// microseconds, while carries 100 ns ticks on Windows; an exact equality on the read-back + /// value would otherwise differ in the last digit. + /// + private static DateTime UtcNowToTheMicrosecond() + { + var now = DateTime.UtcNow; + return new DateTime(now.Ticks - (now.Ticks % 10), DateTimeKind.Utc); + } + + [Fact] + public async Task NoEnabledServer_ReadsNull_SoNothingFires() + { + Assert.SkipWhen(string.IsNullOrEmpty(Env()), "Set DARLING_TEST_PG to run the #5450 live tests."); + var ok = false; + var (scratch, connection) = await OpenStoreAsync(Env()!, hypertable: false); + try + { + /* A store with no servers at all (first install) reads null. */ + Assert.Null(await ScalarAsync(connection, DarlingSelfAlertEvaluator.NewestCollectionTimeSql)); + await AddServerAsync(connection, 7, enabled: false); + await AddServerAsync(connection, 8, enabled: false); + await SeedAsync(connection, 7, DateTime.UtcNow.AddMinutes(-40), 2); + Assert.Null(await ScalarAsync(connection, DarlingSelfAlertEvaluator.NewestCollectionTimeSql)); + ok = true; + } + finally + { + await connection.DisposeAsync(); + await LiveStoreCleanup.RunOwnedAsync(ok, async () => await scratch.DisposeAsync()); + } + } + + [Fact] + public async Task DisabledServersNewerRows_AndTheFleetSentinel_DoNotHideTheGap() + { + Assert.SkipWhen(string.IsNullOrEmpty(Env()), "Set DARLING_TEST_PG to run the #5450 live tests."); + var ok = false; + var (scratch, connection) = await OpenStoreAsync(Env()!, hypertable: false); + try + { + var now = UtcNowToTheMicrosecond(); + var newest = DateTime.SpecifyKind(now.AddMinutes(-40), DateTimeKind.Unspecified); + await AddServerAsync(connection, 7, enabled: true); + await AddServerAsync(connection, 8, enabled: false); + await SeedAsync(connection, 7, now.AddMinutes(-40), 1); + await SeedAsync(connection, 8, now.AddMinutes(-1), 1); + await SeedAsync(connection, 0, now.AddMinutes(-2), 1); + var read = Assert.IsType(await ScalarAsync(connection, DarlingSelfAlertEvaluator.NewestCollectionTimeSql)); + Assert.Equal(newest, read); + ok = true; + } + finally + { + await connection.DisposeAsync(); + await LiveStoreCleanup.RunOwnedAsync(ok, async () => await scratch.DisposeAsync()); + } + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task NewestCollectionTimeSql_IsOneBackwardIndexProbePerServer_NeverASeqScan(bool hypertable) + { + Assert.SkipWhen(string.IsNullOrEmpty(Env()), "Set DARLING_TEST_PG to run the #5450 live tests."); + var ok = false; + var (scratch, connection) = await OpenStoreAsync(Env()!, hypertable); + try + { + var now = UtcNowToTheMicrosecond(); + var newest = DateTime.SpecifyKind(now.AddMinutes(-40), DateTimeKind.Unspecified); + foreach (var id in new[] { 7, 8, 9 }) + { + await AddServerAsync(connection, id, enabled: true); + await SeedAsync(connection, id, now.AddMinutes(-40 - id), 35); + } + + await SeedAsync(connection, 7, now.AddMinutes(-40), 0); + await ExecAsync(connection, "ANALYZE collect.collection_log"); + Assert.Equal(newest, Assert.IsType(await ScalarAsync(connection, DarlingSelfAlertEvaluator.NewestCollectionTimeSql))); + + /* A plain-PostgreSQL scratch database need not carry the TimescaleDB extension at all (CI's does not), + so its information views are read only where they exist; absent, collection_log is plainly not a + hypertable. */ + var timescaleViews = await ScalarAsync(connection, "SELECT to_regclass('timescaledb_information.hypertables') IS NOT NULL") is true; + var chunks = timescaleViews + ? Convert.ToInt32(await ScalarAsync(connection, + "SELECT count(*) FROM timescaledb_information.chunks WHERE hypertable_name = 'collection_log'"), CultureInfo.InvariantCulture) + : 0; + var isHypertable = timescaleViews + ? Convert.ToInt32(await ScalarAsync(connection, + "SELECT count(*) FROM timescaledb_information.hypertables WHERE hypertable_name = 'collection_log'"), CultureInfo.InvariantCulture) + : 0; + Assert.Equal(hypertable ? 1 : 0, isHypertable); + if (hypertable) + { + Assert.True(chunks >= 4, $"seed should span several chunks, saw {chunks}"); + } + + var plan = await ExplainJsonAsync(connection); + var scansSeen = new List(); + Console.WriteLine("PLAN5450 hypertable=" + hypertable + " chunks=" + chunks + "\n" + plan); + /* The three-row server list may be seq-scanned; collection_log (or any chunk of it) never. */ + using (var doc = System.Text.Json.JsonDocument.Parse(plan)) + { + CollectScans(doc.RootElement, scansSeen); + Assert.DoesNotContain(scansSeen, x => x.StartsWith("Seq Scan:", StringComparison.Ordinal) && !x.Contains("config_monitored_servers", StringComparison.Ordinal)); + } + + Assert.Contains(scansSeen, x => x.StartsWith("Index Only Scan:", StringComparison.Ordinal) || x.StartsWith("Index Scan:", StringComparison.Ordinal)); + Assert.Contains("Backward", plan, StringComparison.Ordinal); + if (!hypertable) + { + Assert.Contains("idx_collection_log_time", plan, StringComparison.Ordinal); + } + + ok = true; + } + finally + { + await connection.DisposeAsync(); + await LiveStoreCleanup.RunOwnedAsync(ok, async () => await scratch.DisposeAsync()); + } + } + + private static async Task ScalarAsync(NpgsqlConnection connection, string sql) + { + await using var command = new NpgsqlCommand(sql, connection); + var value = await command.ExecuteScalarAsync(Ct); + return value is DBNull ? null : value; + } + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, params NpgsqlParameter[] parameters) + { + await using var command = new NpgsqlCommand(sql, connection); + command.Parameters.AddRange(parameters); + await command.ExecuteNonQueryAsync(Ct); + } +} diff --git a/Darling/Darling.Tests/CollectionHealthDetachedSweepPressureTests.cs b/Darling/Darling.Tests/CollectionHealthDetachedSweepPressureTests.cs new file mode 100644 index 0000000000..0bfedc2a5f --- /dev/null +++ b/Darling/Darling.Tests/CollectionHealthDetachedSweepPressureTests.cs @@ -0,0 +1,177 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// #4999 (part of #4938): get_collection_health's sweep-pressure roll-up compares the collectors' single-run +/// costs with the one 60-second body their server's sequential pass has to fit in, so it may count only the +/// collectors that run IN that pass. A collector that runs detached does not: it runs beside the pass, and a +/// long p95 on it said BODY_OVERRUN for a body that never waited on it (index_object_stats, once a day, naming +/// itself as the collector that owns the body). +/// +/// The decision is , which the worker's dispatch also asks, so the +/// two cannot disagree about what runs in the pass. The source pins below are what keep it that way: a second, +/// hand-copied version of the test in either file is a drift the behavioural cases cannot see. +/// +public sealed class CollectionHealthDetachedSweepPressureTests +{ + private const string Daily = "index_object_stats"; + private const string OneMinute = "wait_stats"; + + /// One run costs more than the whole 60,000 ms body, so a collector counted in the body reads BODY_OVERRUN. + private const double LongP95Ms = 90_000; + + private const string WorkerPath = "Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs"; + private const string ToolsPath = "Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs"; + + private static CollectorHealth Row(string name, double avgMs, double p95Ms) => + new() { CollectorName = name, AvgDurationMs = avgMs, P95DurationMs = p95Ms }; + + private static SweepPressure Rollup(params CollectorHealth[] rows) => + SweepPressureClassifier.Compute(DarlingMcpDataTools.SweepBodyCollectors(rows)); + + [Fact] + public void ADailyCollectorWithALongP95_IsNotChargedToTheSweepBody() + { + var rows = new[] + { + Row(Daily, avgMs: 40_000, p95Ms: LongP95Ms), + Row(OneMinute, avgMs: 50, p95Ms: 100), + }; + + /* The fixture is a real overrun when the daily collector IS counted: this is what the tool said before. */ + var counted = SweepPressureClassifier.Compute( + rows.Select(r => (r.CollectorName, r.AvgDurationMs, r.P95DurationMs, r.FrequencyMinutes))); + Assert.Equal(SweepPressureClassifier.PeakCycleBodyOverrun, counted.PeakCycleRisk); + Assert.Equal(Daily, counted.PeakCollectorName); + + var pressure = Rollup(rows); + Assert.Equal(SweepPressureClassifier.PeakCycleFits, pressure.PeakCycleRisk); + Assert.Equal(OneMinute, pressure.PeakCollectorName); + } + + [Fact] + public void AOneMinuteCollectorWithTheSameP95_StillReadsBodyOverrun() + { + Assert.Equal(1, CollectorScheduleDefaults.All[OneMinute].FrequencyMinutes); + + var pressure = Rollup( + Row(OneMinute, avgMs: 40_000, p95Ms: LongP95Ms), + Row(Daily, avgMs: 50, p95Ms: 100)); + + Assert.Equal(SweepPressureClassifier.PeakCycleBodyOverrun, pressure.PeakCycleRisk); + Assert.Equal(OneMinute, pressure.PeakCollectorName); + } + + /// + /// The three that were detached before the daily ones (by name, on a five-minute cadence) and one of each + /// kind of daily collector: a catalog 1440-minute one, and an on-load one whose catalog 0 is its daily + /// recapture. None of them runs in the pass, so none is charged to it. + /// + [Theory] + [InlineData("query_store")] + [InlineData("plan_correction")] + [InlineData("pg_wait_sampling")] + [InlineData("index_object_stats")] + [InlineData("pg_index_bloat")] + [InlineData("server_config")] + public void ACollectorThatRunsDetached_IsLeftOutOfTheRollUp(string name) + { + Assert.True(CollectorScheduleDefaults.All.ContainsKey(name), $"'{name}' is not in the catalog"); + + var pressure = Rollup(Row(name, avgMs: 40_000, p95Ms: LongP95Ms)); + + Assert.Equal(SweepPressureClassifier.PeakCycleFits, pressure.PeakCycleRisk); + Assert.Null(pressure.PeakCollectorName); + Assert.Equal(0, pressure.BusyMsPerMinute); + } + + /// + /// Over the whole catalog, from the catalog's own cadences rather than from the predicate under test: a row + /// is left out of the roll-up exactly when its collector is detached by name or runs at a daily cadence or + /// slower, and kept otherwise. + /// + [Fact] + public void TheRollUpKeepsExactlyTheCollectorsThatRunInThePass() + { + foreach (var (name, schedule) in CollectorScheduleDefaults.All) + { + var byName = name is "query_store" or "plan_correction" or "pg_wait_sampling"; + var effective = CollectorScheduleDefaults.EffectiveRecurringIntervalMinutes(schedule.FrequencyMinutes); + var expectedInPass = !byName && effective < 1440; + + var kept = DarlingMcpDataTools.SweepBodyRows(new[] { Row(name, 1, 1) }).Any(); + + Assert.True( + expectedInPass == kept, + $"{name} (every {effective} min, detached by name: {byName}): in the pass = {expectedInPass}, kept in the roll-up = {kept}"); + } + } + + /// + /// The worker's dispatch asks RunsDetached, and this tool asks it too, with no second copy of the test + /// on either side. A hand-copied version passes every behavioural case above until the day one copy is + /// edited, which is the drift this pin exists to refuse. + /// + [Fact] + public void TheWorkersDispatchAndTheHealthToolAskTheSameTest() + { + var worker = ReadRepoFile(WorkerPath); + var tools = ReadRepoFile(ToolsPath); + + /* The test is defined once, from its two halves. */ + Assert.Contains( + "IsDetachedByName(name) || IsDailyInterval(effectiveIntervalMinutes)", worker, StringComparison.Ordinal); + Assert.Equal(1, Count(worker, "internal static bool RunsDetached(")); + + /* The dispatch decides with it. */ + Assert.Equal(1, Count(worker, "if (!RunsDetached(name, interval))")); + + /* The three by-name predicates are chained in ONE place, IsDetachedByName, and nowhere else in the worker + under the dispatch's variable name: a copy of the chain in the dispatch is the second copy. */ + Assert.Equal(1, Count(worker, "IsQueryStoreCollector(name) || IsPlanCorrectionCollector(name) || IsPgWaitSamplingCollector(name)")); + + /* The tool asks it through the worker, once, and does not name any piece of it. */ + Assert.Equal(1, Count(tools, "DarlingWorker.RunsDetached(")); + foreach (var piece in new[] + { + "IsQueryStoreCollector", "IsPlanCorrectionCollector", "IsPgWaitSamplingCollector", "IsDailyInterval", + "IsDetachedByName", "DailyCollectorIntervalMinutes", + }) + { + Assert.DoesNotContain(piece, tools, StringComparison.Ordinal); + } + + /* Both readings of the body's population, the roll-up and the heaviest list, come through the one filter. */ + Assert.Contains( + "var pressure = SweepPressureClassifier.Compute(SweepBodyCollectors(rows));", tools, StringComparison.Ordinal); + Assert.Contains("var heaviest = SweepBodyRows(rows)", tools, StringComparison.Ordinal); + } + + private static int Count(string haystack, string needle) + { + var count = 0; + for (var i = haystack.IndexOf(needle, StringComparison.Ordinal); i >= 0; + i = haystack.IndexOf(needle, i + needle.Length, StringComparison.Ordinal)) + { + count++; + } + + return count; + } +} diff --git a/Darling/Darling.Tests/CollectionHealthEffectiveCadenceTests.cs b/Darling/Darling.Tests/CollectionHealthEffectiveCadenceTests.cs new file mode 100644 index 0000000000..2c1d75ad95 --- /dev/null +++ b/Darling/Darling.Tests/CollectionHealthEffectiveCadenceTests.cs @@ -0,0 +1,198 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// #4999 (part of #4938): get_collection_health judges a collector against the interval it is SCHEDULED at on the +/// server, not the one it shipped with. A collector an operator moved to every 720 minutes was banded against its +/// shipped five, and a collector an override moved across the daily line stayed in (or out of) the sweep-pressure +/// roll-up by its shipped cadence while the worker's dispatch had already moved it. +/// +/// The resolution is the worker's own, : a per-server +/// override, else the fleet-wide one, else the shipped default. The tests apply a set of overrides to rows with +/// , the step the memoized health read runs on every row it +/// builds, so no store is needed. +/// +public sealed class CollectionHealthEffectiveCadenceTests +{ + private const string FiveMinute = "memory_clerks"; + private const string Daily = "index_object_stats"; + private const int ServerId = 7; + private const int OtherServerId = 8; + + private const string ReaderPath = "Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs"; + private const string WorkerPath = "Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs"; + + private static CollectorHealth Row(string name, double avgMs = 0, double p95Ms = 0) => + new() { CollectorName = name, AvgDurationMs = avgMs, P95DurationMs = p95Ms }; + + private static ScheduleOverride Override(int? serverId, string name, int? minutes) => + new(serverId, name, minutes, RetentionDays: null, Enabled: true); + + private static void Apply(CollectorHealth row, params ScheduleOverride[] overrides) => + DarlingDataReader.ApplyScheduledFrequencies(new[] { row }, ServerId, overrides); + + /// A row that bands HEALTHY on any cadence while its newest success is recent enough. + private static CollectorHealth ProductiveRow(string name, double hoursSinceLastSuccess) + { + var last = DateTime.UtcNow.AddHours(-hoursSinceLastSuccess); + return new CollectorHealth + { + CollectorName = name, + TotalRuns = 10, + SuccessCount = 10, + RowsStored = 100, + RunsWithRows = 10, + LastSuccessTime = last, + LastRunTime = last, + }; + } + + [Fact] + public void AnOverrideOf720OnAFiveMinuteCollector_IsJudgedAgainst720() + { + Assert.Equal(5, CollectorScheduleDefaults.All[FiveMinute].FrequencyMinutes); + + var row = Row(FiveMinute, avgMs: 30_000, p95Ms: 30_000); + Assert.Equal(5, row.FrequencyMinutes); + + Apply(row, Override(ServerId, FiveMinute, 720)); + + Assert.Equal(720, row.FrequencyMinutes); + + /* The roll-up amortises the run cost by the interval the row carries: 30,000 ms every 720 minutes. */ + var pressure = SweepPressureClassifier.Compute(DarlingMcpDataTools.SweepBodyCollectors(new[] { row })); + Assert.Equal(30_000.0 / 720, pressure.BusyMsPerMinute, 3); + Assert.Equal(720, pressure.PeakCollectorFrequencyMinutes); + } + + [Fact] + public void TheBandJudgesAgainstTheScheduledInterval_NotTheShippedOne() + { + /* Six hours since the newest success: past the four-hour floor a five-minute collector goes STALE at, well + inside the 18 hours (one and a half intervals) a 720-minute collector gets. */ + var shipped = ProductiveRow(FiveMinute, hoursSinceLastSuccess: 6); + Assert.Equal(CollectorHealthClassifier.Stale, shipped.HealthStatus); + + var scheduled = ProductiveRow(FiveMinute, hoursSinceLastSuccess: 6); + Apply(scheduled, Override(ServerId, FiveMinute, 720)); + Assert.Equal(CollectorHealthClassifier.Healthy, scheduled.HealthStatus); + } + + [Fact] + public void APerServerOverrideBeatsTheFleetOne_AndAnotherServersIsIgnored() + { + var fleetOnly = Row(FiveMinute); + Apply(fleetOnly, Override(null, FiveMinute, 720)); + Assert.Equal(720, fleetOnly.FrequencyMinutes); + + var both = Row(FiveMinute); + Apply(both, Override(null, FiveMinute, 720), Override(ServerId, FiveMinute, 60)); + Assert.Equal(60, both.FrequencyMinutes); + + var elsewhere = Row(FiveMinute); + Apply(elsewhere, Override(OtherServerId, FiveMinute, 720)); + Assert.Equal(5, elsewhere.FrequencyMinutes); + } + + [Fact] + public void AnOverrideThatMovesACollectorAcrossTheDailyLine_MovesItInTheRollUpToo() + { + const double LongP95Ms = 90_000; + + /* A five-minute collector scheduled daily runs beside the pass: its long run is not charged to the body. */ + var movedOut = Row(FiveMinute, avgMs: 40_000, p95Ms: LongP95Ms); + Assert.Equal( + SweepPressureClassifier.PeakCycleBodyOverrun, + SweepPressureClassifier.Compute(DarlingMcpDataTools.SweepBodyCollectors(new[] { movedOut })).PeakCycleRisk); + Apply(movedOut, Override(ServerId, FiveMinute, 1440)); + var outside = SweepPressureClassifier.Compute(DarlingMcpDataTools.SweepBodyCollectors(new[] { movedOut })); + Assert.Equal(SweepPressureClassifier.PeakCycleFits, outside.PeakCycleRisk); + Assert.Null(outside.PeakCollectorName); + + /* The reverse: a daily collector scheduled every five minutes runs in the pass, so its long run is. */ + var movedIn = Row(Daily, avgMs: 40_000, p95Ms: LongP95Ms); + Assert.Equal( + SweepPressureClassifier.PeakCycleFits, + SweepPressureClassifier.Compute(DarlingMcpDataTools.SweepBodyCollectors(new[] { movedIn })).PeakCycleRisk); + Apply(movedIn, Override(ServerId, Daily, 5)); + var inside = SweepPressureClassifier.Compute(DarlingMcpDataTools.SweepBodyCollectors(new[] { movedIn })); + Assert.Equal(SweepPressureClassifier.PeakCycleBodyOverrun, inside.PeakCycleRisk); + Assert.Equal(Daily, inside.PeakCollectorName); + } + + [Fact] + public void WithNoOverrideARowKeepsItsShippedCadence_AndAnUnknownNameIsLeftAlone() + { + var known = Row(FiveMinute); + Apply(known); + Assert.Equal(5, known.FrequencyMinutes); + + /* An on-load collector's catalog 0 is its daily recapture, the substitution the dispatch makes. */ + var onLoad = Row("server_config"); + Apply(onLoad); + Assert.Equal(CollectorScheduleDefaults.OnLoadRecaptureMinutes, onLoad.FrequencyMinutes); + + /* A name the catalog does not know would throw in ResolveSchedule; it stays unstamped at 0. */ + var unknown = Row("not_a_collector"); + Apply(unknown, Override(ServerId, "not_a_collector", 720)); + Assert.Null(unknown.EffectiveFrequencyMinutes); + Assert.Equal(0, unknown.FrequencyMinutes); + } + + /// + /// The resolution is the worker's own call and not a copy of it, and it runs inside the memoized read: the + /// memo hands one list to every caller of the minute, so a stamp applied after it returned would mutate rows + /// other callers are enumerating. + /// + [Fact] + public void TheReadResolvesThroughTheWorkersCall_InsideTheMemoizedRead() + { + var reader = ReadRepoFile(ReaderPath); + var worker = ReadRepoFile(WorkerPath); + var tools = ReadRepoFile("Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs"); + + Assert.Contains("StoreConfigProvider.ResolveSchedule(name, runtime.ServerId, _scheduleOverrides)", worker, StringComparison.Ordinal); + Assert.Equal(1, Count(reader, "StoreConfigProvider.ResolveSchedule(row.CollectorName, serverId, overrides)")); + + /* One stamp, in the method that builds the rows, ahead of its return. */ + Assert.Equal(1, Count(reader, "ApplyScheduledFrequencies(rows, serverId, scheduleOverrides);")); + Assert.True( + reader.IndexOf("ApplyScheduledFrequencies(rows, serverId, scheduleOverrides);", StringComparison.Ordinal) + < reader.IndexOf("internal const string ScheduleOverridesSql", StringComparison.Ordinal)); + + /* The tool itself neither stamps nor resolves: it reads the row's interval. */ + Assert.DoesNotContain("ApplyScheduledFrequencies", tools, StringComparison.Ordinal); + Assert.DoesNotContain("ResolveSchedule", tools, StringComparison.Ordinal); + + /* The overrides come from the sparse schedule table, schema-qualified, this server's rows and the fleet's. */ + Assert.Contains("FROM config.config_collector_schedules", DarlingDataReader.ScheduleOverridesSql, StringComparison.Ordinal); + Assert.Contains("server_id IS NULL", DarlingDataReader.ScheduleOverridesSql, StringComparison.Ordinal); + } + + private static int Count(string haystack, string needle) + { + var count = 0; + for (var i = haystack.IndexOf(needle, StringComparison.Ordinal); i >= 0; + i = haystack.IndexOf(needle, i + needle.Length, StringComparison.Ordinal)) + { + count++; + } + + return count; + } +} diff --git a/Darling/Darling.Tests/CollectionHealthEverySurfaceEffectiveIntervalTests.cs b/Darling/Darling.Tests/CollectionHealthEverySurfaceEffectiveIntervalTests.cs new file mode 100644 index 0000000000..af5071e8b5 --- /dev/null +++ b/Darling/Darling.Tests/CollectionHealthEverySurfaceEffectiveIntervalTests.cs @@ -0,0 +1,306 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Data; +using System.Linq; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Viewer; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// #4999 (part of #4938): every collection-health surface judges a collector against the interval it is SCHEDULED at on +/// its server, not the one it shipped with. get_collection_health did first; the fleet roll-up (the per-server +/// healthy and failing counts of the fleet overview) and the viewer's collection-health reads (a server's own tab, +/// and the Overview cards and status bar) still judged against the shipped interval, so a collector an operator +/// slowed to every 720 minutes could band HEALTHY in one place and STALE in another. +/// +/// The rule is one: in +/// PerformanceMonitor.Collectors, the project the service and the viewer both reference. The worker's +/// picks its rows through the same +/// . The cases below pin that the three are the +/// same answer, then drive each surface's own step with a row whose newest success is six hours old: past the +/// four-hour line a five-minute collector goes STALE at, well inside the 18 hours (one and a half intervals) a +/// collector scheduled every 720 minutes gets. +/// +public sealed class CollectionHealthEverySurfaceEffectiveIntervalTests +{ + private const string FiveMinute = "memory_clerks"; + private const int ServerId = 7; + private const int OtherServerId = 8; + + private const string FleetReaderPath = "Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingFleetReader.cs"; + private const string ViewerPath = "Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs"; + private const string StoreConfigPath = "Darling/PerformanceMonitor.Darling.Service/StoreConfigProvider.cs"; + + private static ScheduleOverride Override(int? serverId, string name, int? minutes) => + new(serverId, name, minutes, RetentionDays: null, Enabled: true); + + private static int? Resolve(string name, params ScheduleOverride[] rows) => + CollectorScheduleDefaults.ResolveEffectiveIntervalMinutes(name, ServerId, rows); + + private static CollectorScheduleRow ViewerOverride(int? serverId, string name, int? minutes) => + new(serverId, name, minutes, RetentionDays: null, Enabled: true); + + /// The same rows as the viewer carries them. + private static CollectorScheduleRow[] AsViewerRows(IEnumerable rows) => + rows.Select(r => ViewerOverride(r.ServerId, r.CollectorName, r.FrequencyMinutes)).ToArray(); + + /// One row in the fourteen-column shape both fleet reads produce (server_id first), healthy on every count + /// and with every instant hours old, so the only thing that can move the band is the + /// interval the row is judged against. + private static DataTableReader FleetRow(string collector, double hoursAgo) + { + var table = new DataTable(); + table.Columns.Add("server_id", typeof(int)); + table.Columns.Add("collector_name", typeof(string)); + foreach (var name in new[] { "total_runs", "success_count", "error_count" }) + { + table.Columns.Add(name, typeof(long)); + } + + table.Columns.Add("last_success_time", typeof(DateTime)); + table.Columns.Add("permission_denied_count", typeof(long)); + table.Columns.Add("last_run_time", typeof(DateTime)); + table.Columns.Add("abandoned_count", typeof(long)); + table.Columns.Add("extension_missing_count", typeof(long)); + table.Columns.Add("last_non_skip_time", typeof(DateTime)); + table.Columns.Add("last_productive_time", typeof(DateTime)); + table.Columns.Add("last_zero_row_streak_break_time", typeof(DateTime)); + table.Columns.Add("latest_run_note", typeof(string)); + + var then = DateTime.UtcNow.AddHours(-hoursAgo); + table.Rows.Add(ServerId, collector, 100L, 100L, 0L, then, 0L, then, 0L, 0L, then, then, then, DBNull.Value); + var reader = table.CreateDataReader(); + Assert.True(reader.Read()); + return reader; + } + + /* ───────────────────────── the one rule ───────────────────────── */ + + private static IEnumerable<(string Scenario, ScheduleOverride[] Rows)> Scenarios(string name) => + [ + ("no override", Array.Empty()), + ("fleet only", [Override(null, name, 720)]), + ("server only", [Override(ServerId, name, 60)]), + ("server beats fleet", [Override(null, name, 720), Override(ServerId, name, 60)]), + ("another server's row is ignored", [Override(OtherServerId, name, 720)]), + ("another server's row beside the fleet's", [Override(OtherServerId, name, 30), Override(null, name, 720)]), + ("a name in another case", [Override(ServerId, name.ToUpperInvariant(), 90)]), + ("a negative server row falls through to the fleet's", [Override(ServerId, name, -5), Override(null, name, 720)]), + ("a null frequency falls through", [Override(ServerId, name, null), Override(null, name, 720)]), + ("zero is honoured (on-load only)", [Override(ServerId, name, 0)]), + ("a delta-family cadence past its cap falls through", [Override(ServerId, name, 100_000), Override(null, name, 15)]), + ("another collector's row is ignored", [Override(ServerId, name + "_other", 720)]), + ]; + + /// + /// The shared rule gives, for every catalog collector under every layering of overrides, exactly what the worker + /// schedules the collector at: then + /// . The viewer carries the rows as its own + /// record and resolves them through the same rule, so it gives the same answer too. + /// + [Fact] + public void TheSharedResolution_GivesWhatTheWorkerSchedulesEveryCollectorAt_ForEveryLayeringOfOverrides() + { + var compared = 0; + foreach (var name in CollectorScheduleDefaults.All.Keys) + { + foreach (var (scenario, rows) in Scenarios(name)) + { + var worker = CollectorScheduleDefaults.EffectiveRecurringIntervalMinutes( + StoreConfigProvider.ResolveSchedule(name, ServerId, rows).FrequencyMinutes); + + Assert.True( + worker == CollectorScheduleDefaults.ResolveEffectiveIntervalMinutes(name, ServerId, rows), + $"{name}: {scenario}: the shared rule disagrees with the worker's schedule"); + Assert.True( + worker == CollectorScheduleDefaults.ResolveEffectiveIntervalMinutes(name, ServerId, AsViewerRows(rows)), + $"{name}: {scenario}: the viewer's rows resolve differently from the worker's"); + compared++; + } + } + + Assert.Equal(CollectorScheduleDefaults.All.Count * Scenarios(FiveMinute).Count(), compared); + } + + [Fact] + public void TheSharedResolution_PicksTheServerRow_ThenTheFleetRow_ThenTheShippedDefault() + { + Assert.Equal(5, CollectorScheduleDefaults.All[FiveMinute].FrequencyMinutes); + + Assert.Equal(5, Resolve(FiveMinute)); + Assert.Equal(720, Resolve(FiveMinute, Override(null, FiveMinute, 720))); + Assert.Equal(60, Resolve(FiveMinute, Override(null, FiveMinute, 720), Override(ServerId, FiveMinute, 60))); + Assert.Equal(5, Resolve(FiveMinute, Override(OtherServerId, FiveMinute, 720))); + Assert.Equal(5, CollectorScheduleDefaults.ResolveEffectiveIntervalMinutes(FiveMinute, ServerId, null)); + } + + [Fact] + public void AnOnLoadCollectorReadsAsItsDailyRecapture_AndAnUnknownNameHasNoInterval() + { + Assert.Equal(0, CollectorScheduleDefaults.All["server_config"].FrequencyMinutes); + Assert.Equal(CollectorScheduleDefaults.OnLoadRecaptureMinutes, Resolve("server_config")); + Assert.Equal(60, Resolve("server_config", Override(null, "server_config", 60))); + + Assert.Null(Resolve("not_a_collector", Override(null, "not_a_collector", 60))); + } + + /* ───────────────────────── each surface ───────────────────────── */ + + /// + /// The fleet roll-up: the same row it reads, mapped through the step the roll-up's loop calls with the server's + /// own schedule rows. Judged against the shipped five minutes it is STALE; against the 720 its server runs it at, + /// HEALTHY, as get_collection_health says for that server. + /// + [Fact] + public void TheFleetRollUp_BandsACollectorAgainstTheIntervalItsServerSchedulesItAt() + { + using (var shipped = FleetRow(FiveMinute, hoursAgo: 6)) + { + var row = DarlingFleetReader.MapFleetHealthRow(shipped); + Assert.Equal(5, row.FrequencyMinutes); + Assert.Equal(CollectorHealthClassifier.Stale, row.HealthStatus); + } + + using (var scheduled = FleetRow(FiveMinute, hoursAgo: 6)) + { + var row = DarlingFleetReader.MapFleetHealthRow(scheduled, ServerId, [Override(ServerId, FiveMinute, 720)]); + Assert.Equal(720, row.FrequencyMinutes); + Assert.Equal(CollectorHealthClassifier.Healthy, row.HealthStatus); + } + + /* The fleet-wide row reaches the server too, and another server's own row does not. */ + using (var fleetWide = FleetRow(FiveMinute, hoursAgo: 6)) + { + var row = DarlingFleetReader.MapFleetHealthRow(fleetWide, ServerId, [Override(null, FiveMinute, 720)]); + Assert.Equal(CollectorHealthClassifier.Healthy, row.HealthStatus); + } + + using (var elsewhere = FleetRow(FiveMinute, hoursAgo: 6)) + { + var row = DarlingFleetReader.MapFleetHealthRow(elsewhere, ServerId, [Override(OtherServerId, FiveMinute, 720)]); + Assert.Equal(5, row.FrequencyMinutes); + Assert.Equal(CollectorHealthClassifier.Stale, row.HealthStatus); + } + } + + /// + /// The viewer's Overview cards and status bar read the per-(server, collector) fleet breakdown, and its rows are + /// stamped by server, so the band they count is the one the server's own tab shows. + /// + [Fact] + public void TheViewersFleetBreakdown_BandsACollectorAgainstTheIntervalItsServerSchedulesItAt() + { + using var reader = FleetRow(FiveMinute, hoursAgo: 6); + var row = ViewerDataService.MapFleetByServerRow(reader); + Assert.Equal(CollectorHealthClassifier.Stale, row.HealthStatus); + + ViewerDataService.ApplyScheduledFrequencies([row], ServerId, [ViewerOverride(ServerId, FiveMinute, 720)]); + + Assert.Equal(720, row.EffectiveFrequencyMinutes); + Assert.Equal(CollectorHealthClassifier.Healthy, row.HealthStatus); + } + + /// The viewer's per-server Collection Health tab, with a per-server row beating the fleet-wide one. + [Fact] + public void TheViewersServerTab_BandsACollectorAgainstTheIntervalItIsScheduledAt_ServerRowFirst() + { + static CollectorHealthRow Row() + { + var then = DateTime.UtcNow.AddHours(-6); + return new CollectorHealthRow + { + CollectorName = FiveMinute, + TotalRuns = 100, + SuccessCount = 100, + LastSuccessTime = then, + LastRunTime = then, + LastNonSkipTime = then, + LastProductiveTime = then, + }; + } + + var shipped = Row(); + Assert.Equal(CollectorHealthClassifier.Stale, shipped.HealthStatus); + + var server = Row(); + ViewerDataService.ApplyScheduledFrequencies([server], ServerId, [ViewerOverride(ServerId, FiveMinute, 720)]); + Assert.Equal(720, server.EffectiveFrequencyMinutes); + Assert.Equal(CollectorHealthClassifier.Healthy, server.HealthStatus); + + /* The fleet row alone, then the server row over it: the server's 5 beats the fleet's 720. */ + var fleet = Row(); + ViewerDataService.ApplyScheduledFrequencies([fleet], ServerId, [ViewerOverride(null, FiveMinute, 720)]); + Assert.Equal(CollectorHealthClassifier.Healthy, fleet.HealthStatus); + + var both = Row(); + ViewerDataService.ApplyScheduledFrequencies( + [both], ServerId, [ViewerOverride(null, FiveMinute, 720), ViewerOverride(ServerId, FiveMinute, 5)]); + Assert.Equal(5, both.EffectiveFrequencyMinutes); + Assert.Equal(CollectorHealthClassifier.Stale, both.HealthStatus); + + /* A name the catalog does not know is left unstamped, keeping the classifier's floor thresholds. */ + var unknown = new CollectorHealthRow { CollectorName = "not_a_collector" }; + ViewerDataService.ApplyScheduledFrequencies([unknown], ServerId, [ViewerOverride(null, "not_a_collector", 720)]); + Assert.Null(unknown.EffectiveFrequencyMinutes); + } + + /* ───────────────────────── the wiring ───────────────────────── */ + + /// + /// The reads that need a store cannot be driven here, so what a case can pin is that each calls the step the cases + /// above drive, once, after its rows are built, and that nothing resolves a second way. + /// + [Fact] + public void EachSurfaceStampsThroughTheOneResolution_AndNoSurfaceResolvesItsOwn() + { + var fleet = ReadRepoFile(FleetReaderPath); + var viewer = ReadRepoFile(ViewerPath); + var storeConfig = ReadRepoFile(StoreConfigPath); + + /* The fleet roll-up reads the whole schedule table once, ahead of its health statement, and maps every row + with its server's rows. */ + Assert.Equal(1, Count(fleet, "DarlingDataReader.ReadScheduleOverridesAsync(postgres, null, cancellationToken)")); + Assert.Equal(1, Count(fleet, "MapFleetHealthRow(reader, serverId, serverOverrides)")); + Assert.True( + fleet.IndexOf("DarlingDataReader.ReadScheduleOverridesAsync(postgres, null, cancellationToken)", StringComparison.Ordinal) + < fleet.IndexOf("postgres.CreateCommand(composedSql ?? FleetCollectionHealthSql)", StringComparison.Ordinal)); + + /* The viewer stamps what its two row-building reads return, and resolves only through the shared rule. */ + Assert.Equal(1, Count(viewer, "ApplyScheduledFrequencies(items, serverId, scheduleOverrides);")); + Assert.Equal(1, Count(viewer, "ApplyScheduledFrequencies(rows, serverId, scheduleOverrides.Where(")); + Assert.Equal(1, Count(viewer, "CollectorScheduleDefaults.ResolveEffectiveIntervalMinutes(row.CollectorName, serverId, overrides)")); + Assert.DoesNotContain("ResolveFrequencyMinutes(", viewer, StringComparison.Ordinal); + + /* The worker's own resolution picks its rows through the shared selector, not a loop of its own. */ + Assert.Equal(1, Count(storeConfig, "CollectorScheduleDefaults.SelectScheduleOverrides(collectorName, serverId, overrides)")); + var resolve = storeConfig.IndexOf("public static EffectiveSchedule ResolveSchedule(", StringComparison.Ordinal); + var resolveEnd = storeConfig.IndexOf("ResolveFleetRetentionDays(", resolve, StringComparison.Ordinal); + Assert.DoesNotContain("foreach (var o in overrides)", storeConfig[resolve..resolveEnd], StringComparison.Ordinal); + } + + private static int Count(string haystack, string needle) + { + var count = 0; + for (var i = haystack.IndexOf(needle, StringComparison.Ordinal); i >= 0; + i = haystack.IndexOf(needle, i + needle.Length, StringComparison.Ordinal)) + { + count++; + } + + return count; + } +} diff --git a/Darling/Darling.Tests/CollectionHealthKeyedLookupParityTests.cs b/Darling/Darling.Tests/CollectionHealthKeyedLookupParityTests.cs new file mode 100644 index 0000000000..547247a4ff --- /dev/null +++ b/Darling/Darling.Tests/CollectionHealthKeyedLookupParityTests.cs @@ -0,0 +1,501 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.Json; +using System.Text.RegularExpressions; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4955: the per-server collection-health statements (the MCP tool's and the viewer grid's) used to rank every +/// run of the seven-day window four (MCP) or three (viewer) times with ROW_NUMBER() only to read the +/// NEWEST row of a class, and now read those rows through keyed lookups instead. A source pin cannot tell a +/// lookup that picks the same row as the rank from one that picks a neighbour, so this class runs the pre-change +/// statements (, kept verbatim) beside the current ones over one seeded +/// store and requires every column of every row to agree - name, type and value - across the ties and the empty +/// classes where the two shapes could part: equal timestamps with different statuses and different texts, a +/// failure with text older than one without, notes that a run abandoned, a streak with and without a break, a +/// collector that only skipped, a collector with one run, fan-out rows tied on the slowest item, a status the +/// column cannot hold in the real table (a NULL one, seeded into a session-local copy of the view), and rows just +/// before the window. +/// +/// Where the oracle's own order is not total (two rows equal on every ORDER BY key and different in a +/// selected column) either row is a right answer, so no such case is seeded. +/// +/// Each live test mints its own scratch database, so the rows it seeds and the plans it explains are its own: +/// no other class's rows or chunks can reach them (#4650). +/// +/* #1776 own-store: each live test mints its own scratch database, so no other class's rows or chunks shape what it + reads or how its plans come out. */ +[Collection("live-postgres")] +public sealed class CollectionHealthKeyedLookupParityTests +{ + private const int ServerId = -4955001; + private const string ServerName = "keyed-lookup-parity"; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static readonly Regex s_fullWindowRank = new( + @"ROW_NUMBER\(\)\s+OVER\s*\(\s*PARTITION\s+BY\s+collector_name", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + + [Theory] + [InlineData("mcp")] + [InlineData("viewer")] + public void TheStatementDoesNotRankTheWholeWindow(string which) + { + var (current, oracle) = which == "mcp" + ? (DarlingDataReader.CollectionHealthSql, CollectionHealthOracleSql.Mcp) + : (ViewerDataService.CollectionHealthSql, CollectionHealthOracleSql.Viewer); + + /* The oracle is the shape being left: a rank per newest-row column over the whole window. */ + Assert.Matches(s_fullWindowRank, oracle); + + Assert.DoesNotMatch(s_fullWindowRank, current); + foreach (var rank in new[] { "note_rank", "error_rank", "slowest_rank", "recency_rank" }) + Assert.DoesNotContain(rank, current, StringComparison.Ordinal); + + /* The newest rows come from lookups keyed on an instant the plain aggregate found, one per class. */ + Assert.Contains("LEFT JOIN LATERAL", current, StringComparison.Ordinal); + } + + [Fact] + public async Task TheCurrentPlansRankNothingOverTheWindow_AgainstDevPostgres() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), + "Set DARLING_TEST_PG to a Postgres connection string to run the live collection-health plan test (it mints its own scratch database)."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(cs!, ct); + using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + var now = Now(); + await SeedEdgeCasesAsync(connection, now, ct); + var windowStart = DarlingMcpTestData.Naive(now.AddDays(-7)); + + /* The oracle's plans are the shape this pin is about: one WindowAgg per rank. */ + Assert.True( + await CountWindowAggsAsync(connection, CollectionHealthOracleSql.Mcp, windowStart, ct) >= 4, + "the oracle should still show one WindowAgg per rank"); + Assert.True( + await CountWindowAggsAsync(connection, CollectionHealthOracleSql.Viewer, windowStart, ct) >= 3, + "the viewer oracle should still show one WindowAgg per rank"); + + /* The viewer's statement has no window left at all; the MCP statement keeps ONE, the trailing + zero-row count, over the rows from the newest streak break on (a lookup, never the window). */ + Assert.Equal(0, await CountWindowAggsAsync(connection, ViewerDataService.CollectionHealthSql, windowStart, ct)); + Assert.Equal(1, await CountWindowAggsAsync(connection, DarlingDataReader.CollectionHealthSql, windowStart, ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, DeleteRowsAsync); + } + } + + [Fact] + public async Task BothStatements_ReturnTheOraclesRows_OverEveryEdgeCase_AgainstDevPostgres() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), + "Set DARLING_TEST_PG to a Postgres connection string to run the live collection-health parity test (it mints its own scratch database)."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(cs!, ct); + using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + var now = Now(); + await SeedEdgeCasesAsync(connection, now, ct); + var windowStart = DarlingMcpTestData.Naive(now.AddDays(-7)); + + /* ── the real store ── */ + var mcpOracle = await ReadAsync(connection, CollectionHealthOracleSql.Mcp, windowStart, ct); + var mcpCurrent = await ReadAsync(connection, DarlingDataReader.CollectionHealthSql, windowStart, ct); + Assert.Equal(31, mcpOracle.Columns.Count); + AssertSameRows("mcp", mcpOracle, mcpCurrent); + AssertOracleSaw(mcpOracle, now); + + var viewerOracle = await ReadAsync(connection, CollectionHealthOracleSql.Viewer, windowStart, ct); + var viewerCurrent = await ReadAsync(connection, ViewerDataService.CollectionHealthSql, windowStart, ct); + Assert.Equal(19, viewerOracle.Columns.Count); + AssertSameRows("viewer", viewerOracle, viewerCurrent); + + /* ── a session-local copy of the view, for the rows the table's NOT NULL status cannot hold ── + Both statements name v_collection_log unqualified, and pg_temp is searched first, so a temp + table of that name (made by CTAS, which keeps no NOT NULL) stands in for the view on this + connection alone and is gone with it. */ + await DarlingMcpTestData.ExecAsync(connection, ct, + "CREATE TEMP TABLE v_collection_log AS SELECT * FROM collect.v_collection_log WHERE server_id = $1", ServerId); + var nullStatusCasesRan = false; + try + { + await SeedNullStatusCasesAsync(connection, now, ct); + + var mcpOracleNull = await ReadAsync(connection, CollectionHealthOracleSql.Mcp, windowStart, ct); + var mcpCurrentNull = await ReadAsync(connection, DarlingDataReader.CollectionHealthSql, windowStart, ct); + AssertSameRows("mcp+null status", mcpOracleNull, mcpCurrentNull); + var nullTie = mcpOracleNull.Row("null_status_tie"); + Assert.Null(nullTie["current_status"]); + Assert.Null(nullTie["latest_run_note"]); + Assert.Equal(3L, nullTie["trailing_zero_row_success_runs"]); + + var viewerOracleNull = await ReadAsync(connection, CollectionHealthOracleSql.Viewer, windowStart, ct); + var viewerCurrentNull = await ReadAsync(connection, ViewerDataService.CollectionHealthSql, windowStart, ct); + AssertSameRows("viewer+null status", viewerOracleNull, viewerCurrentNull); + Assert.Null(viewerOracleNull.Row("null_status_tie")["latest_run_note"]); + nullStatusCasesRan = true; + } + finally + { + /* #1902: RunOwnedAsync, not RunAsync - the temp table is session-local, so the drop has to run on + this very connection; a fresh one would leave the shadow in place and report success. */ + await LiveStoreCleanup.RunOwnedAsync(nullStatusCasesRan, async () => + await DarlingMcpTestData.ExecAsync(connection, ct, "DROP TABLE IF EXISTS pg_temp.v_collection_log")); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, DeleteRowsAsync); + } + } + + // ── what the oracle must show for the seeded edges: the seeds prove the cases they name ─────────────────── + + private static void AssertOracleSaw(Result mcp, DateTime now) + { + /* Two rows at the newest instant, SUCCESS and ERROR: status DESC puts SUCCESS first, so exactly the one + zero-row success sits ahead of the newest break. */ + var tieStatus = mcp.Row("tie_status"); + Assert.Equal("SUCCESS", tieStatus["current_status"]); + Assert.Equal(1L, tieStatus["trailing_zero_row_success_runs"]); + Assert.Equal("tie error", tieStatus["last_error"]); + + /* Two failures at one instant: the greater text wins, and the older instant's lesser text does not. */ + var tieText = mcp.Row("tie_error_text"); + Assert.Equal("err_zzz", tieText["last_error"]); + Assert.Equal("note_z", tieText["last_note"]); + + /* A failure with text older than one without: the text is the older run's, the time is the newer's. */ + var oldText = mcp.Row("old_text_new_textless"); + Assert.Equal("old text", oldText["last_error"]); + Assert.Equal(DarlingMcpTestData.Naive(now.AddMinutes(-100)), oldText["last_error_time"]); + + /* A SUCCESS note and a failure with text at one instant: neither class picks the other's row. */ + var sameInstant = mcp.Row("note_vs_error_same_instant"); + Assert.Equal("a note", sameInstant["last_note"]); + Assert.Equal("an error", sameInstant["last_error"]); + + /* No failure, or failures with no text: no last_error. */ + Assert.Null(mcp.Row("no_failures")["last_error"]); + Assert.Null(mcp.Row("failures_no_text")["last_error"]); + Assert.Null(mcp.Row("failures_no_text")["last_note"]); + + /* Streaks: broken by an error, never broken, never a success, a NULL predicate that is no break. */ + Assert.Equal(4L, mcp.Row("streak_with_break")["trailing_zero_row_success_runs"]); + Assert.Equal(6L, mcp.Row("streak_no_break")["trailing_zero_row_success_runs"]); + Assert.Equal(0L, mcp.Row("skipped_only")["trailing_zero_row_success_runs"]); + Assert.Equal(2L, mcp.Row("null_predicate")["trailing_zero_row_success_runs"]); + Assert.Equal(2L, mcp.Row("notes_and_abandoned")["trailing_zero_row_success_runs"]); + Assert.Equal(1L, mcp.Row("one_row")["total_runs"]); + + /* Fan-out: the newest of the rows tied on the slowest item, and the NULL one never wins. */ + var fanout = mcp.Row("fanout_ties"); + Assert.Equal("db_new_tied", fanout["slowest_item"]); + Assert.Equal(6, fanout["fanout_items"]); + Assert.Equal(5000, fanout["slowest_item_ms"]); + Assert.Equal(38000, fanout["slowest_run_duration_ms"]); + + /* A dearest run with no duration reports no duration. */ + Assert.Null(mcp.Row("null_duration")["slowest_run_duration_ms"]); + Assert.Equal(900, mcp.Row("null_duration")["slowest_item_ms"]); + + /* No row ever carried a slowest item: rank 1 falls through to the NEWEST row, whose own columns show. */ + var none = mcp.Row("no_fanout_newest"); + Assert.Null(none["slowest_item_ms"]); + Assert.Equal(777, none["slowest_run_duration_ms"]); + Assert.Equal(2, none["fanout_items"]); + Assert.Equal("x", none["slowest_item"]); + + /* Rows before the window are not in it: the only text on this collector is older than the window. */ + var before = mcp.Row("before_window"); + Assert.Null(before["last_error"]); + Assert.Null(before["last_note"]); + Assert.Equal(2L, before["total_runs"]); + } + + // ── the seeds ───────────────────────────────────────────────────────────────────────────────────────────── + + private static DateTime Now() + { + var now = DateTime.SpecifyKind(LiveClock.Now(), DateTimeKind.Utc); + return new DateTime(now.Ticks - (now.Ticks % TimeSpan.TicksPerMillisecond), DateTimeKind.Utc); + } + + private static async Task SeedEdgeCasesAsync(NpgsqlConnection c, DateTime now, CancellationToken ct) + { + DateTime Ago(double minutes) => now.AddMinutes(-minutes); + Task Seed(string collector, double minutesAgo, string status, string? message = null, + int? durationMs = 100, int? rows = 10, int? fanoutItems = null, string? slowestItem = null, + int? slowestItemMs = null) => + SeedAsync(c, ct, collector, Ago(minutesAgo), status, message, durationMs, rows, fanoutItems, slowestItem, slowestItemMs); + + /* Two rows at the newest instant with different statuses: current_status takes the greater status, and + the streak count puts the zero-row SUCCESS ahead of the ERROR that breaks it. */ + await Seed("tie_status", 300, "SUCCESS", rows: 10); + await Seed("tie_status", 200, "SUCCESS", rows: 0); + await Seed("tie_status", 100, "SUCCESS", rows: 0); + await Seed("tie_status", 50, "SUCCESS", rows: 0); + await Seed("tie_status", 50, "ERROR", "tie error", rows: 0); + + /* Two failures at one instant with different texts, and two notes at one (older) instant. */ + await Seed("tie_error_text", 400, "SUCCESS", "note_a", rows: 0); + await Seed("tie_error_text", 400, "SUCCESS", "note_z", rows: 0); + await Seed("tie_error_text", 300, "ERROR", "err_aaa", rows: 0); + await Seed("tie_error_text", 300, "PERMISSIONS", "err_zzz", rows: 0); + await Seed("tie_error_text", 100, "SUCCESS", rows: 5); + + /* NULL durations, one of them on the dearest fan-out run. */ + await Seed("null_duration", 200, "SUCCESS", durationMs: null); + await Seed("null_duration", 150, "SUCCESS", durationMs: null, fanoutItems: 3, slowestItem: "db_a", slowestItemMs: 900); + await Seed("null_duration", 100, "SUCCESS", durationMs: 40); + + /* No failures at all. */ + await Seed("no_failures", 300, "SUCCESS"); + await Seed("no_failures", 200, "SUCCESS"); + await Seed("no_failures", 100, "SUCCESS"); + + /* Failures that carry no text. */ + await Seed("failures_no_text", 200, "ERROR", rows: 0); + await Seed("failures_no_text", 100, "PERMISSIONS", rows: 0); + + /* A failure with text older than a failure without: the text is the older run's. */ + await Seed("old_text_new_textless", 300, "ERROR", "old text", rows: 0); + await Seed("old_text_new_textless", 100, "ERROR", rows: 0); + await Seed("old_text_new_textless", 50, "SUCCESS", rows: 3); + + /* A SUCCESS note and a failure with text at ONE instant, neither the newest run. */ + await Seed("note_vs_error_same_instant", 200, "SUCCESS", "a note", rows: 0); + await Seed("note_vs_error_same_instant", 200, "ERROR", "an error", rows: 0); + await Seed("note_vs_error_same_instant", 100, "SUCCESS", rows: 4); + + /* Notes, a run the wall-clock budget abandoned before and after the ABANDONED status existed. */ + var budget = EnumeratedCollectorDriver.WholeCycleBudgetNote(120); + await Seed("notes_and_abandoned", 400, "SUCCESS", budget, rows: 0); + await Seed("notes_and_abandoned", 300, EnumeratedCollectorDriver.AbandonedStatus, budget, rows: 0); + await Seed("notes_and_abandoned", 200, "SUCCESS", EnumeratedCollectorDriver.EmptyEnumerationMessage, rows: 0); + await Seed("notes_and_abandoned", 100, "SUCCESS", rows: 0); + + /* A streak with a break, one without, and a collector that only ever skipped. */ + await Seed("streak_with_break", 700, "SUCCESS", rows: 10); + await Seed("streak_with_break", 600, "SUCCESS", rows: 0); + await Seed("streak_with_break", 500, "SUCCESS", rows: 0); + await Seed("streak_with_break", 400, "SUCCESS", rows: 0); + await Seed("streak_with_break", 300, "ERROR", "boom", rows: 0); + foreach (var minutes in new double[] { 200, 150, 100, 50 }) + await Seed("streak_with_break", minutes, "SUCCESS", rows: 0); + foreach (var minutes in new double[] { 600, 500, 400, 300, 200, 100 }) + await Seed("streak_no_break", minutes, "SUCCESS", rows: 0); + foreach (var minutes in new double[] { 300, 200, 100 }) + await Seed("skipped_only", minutes, "SKIPPED", rows: 0); + + /* A row whose streak predicate is NULL (a NULL row count beside the budget note) is no break. */ + await Seed("null_predicate", 300, "SUCCESS", rows: 5); + await Seed("null_predicate", 200, "SUCCESS", budget, rows: null); + await Seed("null_predicate", 100, "SUCCESS", rows: 0); + + /* One run only. */ + await Seed("one_row", 100, "SUCCESS", rows: 1); + + /* Three rows tied on the slowest item (the newest of them wins), a smaller one, and a NEWER row with no + slowest item at all. */ + await Seed("fanout_ties", 500, "SUCCESS", durationMs: 40000, fanoutItems: 8, slowestItem: "db_old", slowestItemMs: 5000); + await Seed("fanout_ties", 400, "SUCCESS", durationMs: 39000, fanoutItems: 7, slowestItem: "db_mid", slowestItemMs: 5000); + await Seed("fanout_ties", 300, "SUCCESS", durationMs: 38000, fanoutItems: 6, slowestItem: "db_new_tied", slowestItemMs: 5000); + await Seed("fanout_ties", 200, "SUCCESS", durationMs: 20, fanoutItems: 5, slowestItem: "db_small", slowestItemMs: 100); + await Seed("fanout_ties", 100, "SUCCESS", durationMs: 10); + + /* No row has a slowest item: the newest row's own fan-out columns and duration come out. */ + await Seed("no_fanout_newest", 300, "SUCCESS", durationMs: 50); + await Seed("no_fanout_newest", 200, "SUCCESS", durationMs: 60); + await Seed("no_fanout_newest", 100, "SUCCESS", durationMs: 777, fanoutItems: 2, slowestItem: "x"); + + /* Every status the counts name, with text on the failures. */ + await Seed("status_mix", 800, "EXTENSION_MISSING", "install the extension"); + await Seed("status_mix", 700, "SESSION_MISSING", "no session"); + await Seed("status_mix", 600, "YIELDED", "yielded"); + await Seed("status_mix", 500, "CANCELLED", "cancelled"); + await Seed("status_mix", 400, "PERMISSIONS", "denied"); + await Seed("status_mix", 300, "SKIPPED"); + await Seed("status_mix", 200, "ERROR", "failed"); + await Seed("status_mix", 100, "SUCCESS", "a note", rows: 2); + + /* Rows just before the window, a row exactly on its start, and one after. */ + var windowStartMinutes = 7 * 24 * 60.0; + await Seed("before_window", windowStartMinutes + 1, "ERROR", "before window"); + await Seed("before_window", windowStartMinutes + 0.001, "SUCCESS", "old note"); + await Seed("before_window", windowStartMinutes, "SUCCESS", rows: 0); + await Seed("before_window", 100, "SUCCESS", rows: 0); + } + + /// Rows the real table cannot hold (its status is NOT NULL), into the session-local copy of the view. + private static async Task SeedNullStatusCasesAsync(NpgsqlConnection c, DateTime now, CancellationToken ct) + { + Task Seed(double minutesAgo, string? status, string? message, int? rows) => + DarlingMcpTestData.ExecAsync(c, ct, @" +INSERT INTO pg_temp.v_collection_log + (log_id, server_id, server_name, collector_name, collection_time, duration_ms, status, error_message, rows_collected) +VALUES ($1, $2, $3, 'null_status_tie', $4, 100, $5, $6, $7)", + CollectionIdGenerator.Next(), ServerId, ServerName, + DarlingMcpTestData.Naive(now.AddMinutes(-minutesAgo)), status, message, rows); + + /* A NULL status beside a SUCCESS at the newest instant: DESC puts the NULL first, so the newest run's + status and note read as NULL; the NULL-status row with a zero count is no break, the one with a + count is. */ + await Seed(300, "ERROR", "e", 0); + await Seed(250, null, null, 10); + await Seed(200, "SUCCESS", null, 0); + await Seed(100, null, "null note", 0); + await Seed(100, "SUCCESS", null, 0); + } + + private static Task SeedAsync( + NpgsqlConnection c, CancellationToken ct, string collector, DateTime atUtc, string status, string? message, + int? durationMs, int? rows, int? fanoutItems, string? slowestItem, int? slowestItemMs) => + DarlingMcpTestData.ExecAsync(c, ct, @" +INSERT INTO collection_log + (log_id, server_id, server_name, collector_name, collection_time, duration_ms, status, error_message, + rows_collected, sql_duration_ms, duckdb_duration_ms, fanout_item_count, slowest_item, slowest_item_ms) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14)", + CollectionIdGenerator.Next(), ServerId, ServerName, collector, DarlingMcpTestData.Naive(atUtc), + durationMs, status, message, rows, 80, 20, fanoutItems, slowestItem, slowestItemMs); + + private static async Task DeleteRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM collection_log WHERE server_id = $1", ServerId); + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM servers WHERE server_id = $1", ServerId); + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM config_monitored_servers WHERE server_id = $1", ServerId); + } + + // ── reading and comparing ───────────────────────────────────────────────────────────────────────────────── + + private sealed record Column(string Name, string PgType, Type ClrType); + + private sealed class Result + { + public List Columns { get; } = new(); + + public List Rows { get; } = new(); + + public Dictionary Row(string collector) + { + var row = Rows.Single(r => (string)r[0]! == collector); + return Columns.Select((col, i) => (col.Name, Value: row[i])).ToDictionary(p => p.Name, p => p.Value); + } + } + + private static NpgsqlCommand Bind(NpgsqlConnection connection, string sql, DateTime windowStart) + { + var command = new NpgsqlCommand(sql, connection); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = ServerId }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = windowStart }); + return command; + } + + private static async Task ReadAsync(NpgsqlConnection connection, string sql, DateTime windowStart, CancellationToken ct) + { + await using var command = Bind(connection, sql, windowStart); + await using var reader = await command.ExecuteReaderAsync(ct); + var result = new Result(); + for (var i = 0; i < reader.FieldCount; i++) + result.Columns.Add(new Column(reader.GetName(i), reader.GetDataTypeName(i), reader.GetFieldType(i))); + while (await reader.ReadAsync(ct)) + { + var values = new object?[reader.FieldCount]; + for (var i = 0; i < values.Length; i++) + values[i] = reader.IsDBNull(i) ? null : reader.GetValue(i); + result.Rows.Add(values); + } + + return result; + } + + private static void AssertSameRows(string label, Result oracle, Result current) + { + Assert.Equal(oracle.Columns, current.Columns); + Assert.Equal(oracle.Rows.Count, current.Rows.Count); + Assert.True(oracle.Rows.Count >= 10, label + ": the seed should give the oracle many collectors"); + for (var r = 0; r < oracle.Rows.Count; r++) + { + for (var i = 0; i < oracle.Columns.Count; i++) + { + Assert.True( + Equals(oracle.Rows[r][i], current.Rows[r][i]), + $"{label}: collector {oracle.Rows[r][0]}, column {oracle.Columns[i].Name}: " + + $"oracle <{oracle.Rows[r][i] ?? "NULL"}> but current <{current.Rows[r][i] ?? "NULL"}>"); + } + } + } + + private static async Task CountWindowAggsAsync( + NpgsqlConnection connection, string sql, DateTime windowStart, CancellationToken ct) + { + await using var command = Bind(connection, "EXPLAIN (FORMAT JSON) " + sql, windowStart); + var json = (string)(await command.ExecuteScalarAsync(ct))!; + using var document = JsonDocument.Parse(json); + return CountNodes(document.RootElement, "WindowAgg"); + } + + private static int CountNodes(JsonElement element, string nodeType) + { + var count = 0; + switch (element.ValueKind) + { + case JsonValueKind.Object: + if (element.TryGetProperty("Node Type", out var type) && type.GetString() == nodeType) + count++; + foreach (var property in element.EnumerateObject()) + count += CountNodes(property.Value, nodeType); + break; + case JsonValueKind.Array: + foreach (var item in element.EnumerateArray()) + count += CountNodes(item, nodeType); + break; + } + + return count; + } +} diff --git a/Darling/Darling.Tests/CollectionHealthOracleSql.cs b/Darling/Darling.Tests/CollectionHealthOracleSql.cs new file mode 100644 index 0000000000..cb5de49924 --- /dev/null +++ b/Darling/Darling.Tests/CollectionHealthOracleSql.cs @@ -0,0 +1,482 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using PerformanceMonitor.Collectors; + +namespace Darling.Tests; + +/// +/// The per-server collection-health statements as they read before #4955 replaced their four (MCP) and three +/// (viewer) full-window ROW_NUMBER() ranks with keyed lookups, kept VERBATIM as the oracle the rewrite is +/// held to: runs each beside its current statement over the +/// same seeded rows and requires every column of every row to agree. Do not edit these to follow the product +/// statements; they are the definition of "the same values". +/// +internal static class CollectionHealthOracleSql +{ + /// The MCP service's statement (DarlingDataReader.CollectionHealthSql) before #4955: 31 columns. + public const string Mcp = $""" + SELECT + collector_name, + COUNT(*) AS total_runs, + -- #2926: SUCCESS excludes an abandonment that predates #2803, so the Success column beside + -- Abandoned cannot count the same run twice. Post-#2803 rows need no exclusion - ABANDONED + -- is not SUCCESS - and an ordinary empty run stays counted, which is what the COALESCE in + -- the shared predicate is for: NULL under this NOT would have dropped it. + SUM(CASE WHEN status = 'SUCCESS' + AND NOT {EnumeratedCollectorDriver.AbandonedByNotePredicateSql} + THEN 1 ELSE 0 END) AS success_count, + SUM(CASE WHEN status = 'ERROR' THEN 1 ELSE 0 END) AS error_count, + AVG(duration_ms) AS avg_duration_ms, + -- #2460: the mean above describes a collector whose runs all cost about the same, and + -- says nothing true about one whose runs come in two sizes. query_store on a dense shard + -- reported a 13,834 ms average over 1,155 runs where 958 of them yielded nothing and cost + -- ~36 ms, which puts the other 197 at ~80,900 ms EACH — each one on its own larger than + -- the whole 60,000 ms sweep budget. duration_ms has been written per run since V2; + -- nothing had ever read it as anything but a mean. + -- + -- p95 rather than the max for the number a decision is made from: a max is one run, so a + -- single pathological cycle would make a collector look permanently terrible for the rest + -- of the window. p95 also scales itself to the sample — over 3,500 runs it discards the + -- one bad cycle, and over the six runs a daily collector gets in a week it lands on the + -- max, which is right, because with six samples there is no outlier anyone can afford to + -- throw away. DISC rather than CONT so the answer is a duration some run actually took + -- instead of an interpolation between the two modes, which would be a number describing + -- no run at all — the exact defect this column exists to end. Both engines ignore NULL + -- duration_ms here, as AVG already does. Byte-identical to Lite's DuckDB read. + MAX(duration_ms) AS max_duration_ms, + PERCENTILE_DISC(0.95) WITHIN GROUP (ORDER BY duration_ms) AS p95_duration_ms, + MAX(CASE WHEN status IN ('SUCCESS', 'SKIPPED') THEN collection_time END) AS last_success_time, + MAX(collection_time) AS last_run_time, + -- #1855: the message from the NEWEST failing run, not MAX()'s lexicographically greatest + -- one. The status re-check is load-bearing rather than belt-and-braces: when no failing run + -- in the window carried text, error_rank = 1 falls through to the newest row of ANY class, + -- and without it a SUCCESS row's note could surface here as a fake last error. + -- #3240: EXTENSION_MISSING is in the exemplar set because its stored sentence IS the remedy + -- (it names the extension and the database to create it in). + MAX(CASE WHEN error_rank = 1 AND status IN ('ERROR', 'PERMISSIONS', 'EXTENSION_MISSING') THEN error_message END) AS last_error, + -- The newest failure OUTRIGHT, text or not — "when did this last fail" means the run, not + -- the message. + MAX(CASE WHEN status IN ('ERROR', 'PERMISSIONS', 'EXTENSION_MISSING') THEN collection_time END) AS last_error_time, + SUM(CASE WHEN status = 'PERMISSIONS' THEN 1 ELSE 0 END) AS permission_denied_count, + SUM(CASE WHEN status = 'YIELDED' THEN 1 ELSE 0 END) AS yield_count, + -- #1837: the note a SUCCEEDING run can leave behind (an enumeration that yielded 0 items, + -- items whose enumeration probe failed) and how many runs carried one. Gated on SUCCESS + -- specifically — the runners attach a note only to the SUCCESS write. Informational: it + -- feeds no band, and a legitimately empty target stays HEALTHY. + MAX(CASE WHEN note_rank = 1 AND status = 'SUCCESS' THEN error_message END) AS last_note, + COUNT(CASE WHEN status = 'SUCCESS' THEN error_message END) AS note_count, + -- #1852: the one thing that makes a persistently-empty enumeration interesting — does this + -- target actually HAVE user databases? Zero items on a server with none is legitimate and + -- stays quiet; zero items on a server that HAS them is a login that cannot enter any, or a + -- filter that swallowed everything. database_size_stats rather than database_config for + -- three reasons: it runs on the scheduled loop (60 min) where database_config is on-load + -- and can age past this window on a long-running service; it is indexed on + -- (server_id, collection_time) where database_config has no index at all; and it reads + -- sys.master_files, so it still sees databases the monitoring login cannot ENTER — exactly + -- the case being diagnosed. database_id > 4 excludes the system databases, tempdb + -- included: the size collector takes every ONLINE database, so a bare row check + -- would be true on every server alive. A NULL database_id is an Azure sibling row + -- (#2643/#3262): sys.resource_stats carries no id, and it bills only USER databases, + -- so those rows are inventory too — without the IS NULL arm a master-connected Azure + -- target with fifty user databases would read as having none. + -- + -- The inventory window is the health read's OWN ($2) — no second parameter, and an + -- inventory that aged out says nothing rather than something stale. Uncorrelated, so both + -- engines evaluate it once per query (a Postgres InitPlan) instead of per row, and it + -- needs no GROUP BY entry and no join. Feeds display text only, through the shared + -- formatter; the banding never sees it. + CASE + WHEN EXISTS + ( + SELECT 1 + FROM v_database_size_stats + WHERE server_id = $1 + AND collection_time >= $2 + AND (database_id > 4 OR database_id IS NULL) + ) + THEN 1 + ELSE 0 + END AS has_user_databases, + -- #2472: the per-database fan-out, described for ONE run — the dearest one in the window. + -- Five collectors run once per database and the run writes a single blended duration_ms, so + -- "eight databases at 10.1s" and "one at 62s beside seven at 2.7s" are the same 80,900 ms and + -- want opposite fixes. These four are that one run's parts, and their ratio + -- (slowest_item_ms * fanout_items / slowest_run_duration_ms) is 1.0 for an even fan-out and + -- 6.1 for the dominated example. All four come from the SAME row via slowest_rank rather than + -- four independent aggregates, because a slowest item taken from one run and a width taken + -- from another compose into a ratio describing no run that ever happened — the same defect + -- PERCENTILE_CONT was rejected for above. NULL throughout for a collector that never fans + -- out, which is most of them. + MAX(CASE WHEN slowest_rank = 1 THEN fanout_item_count END) AS fanout_items, + MAX(CASE WHEN slowest_rank = 1 THEN slowest_item END) AS slowest_item, + MAX(CASE WHEN slowest_rank = 1 THEN slowest_item_ms END) AS slowest_item_ms, + MAX(CASE WHEN slowest_rank = 1 THEN duration_ms END) AS slowest_run_duration_ms, + -- #2804: runs the #2673 wall-clock budget abandoned. Appended LAST rather than placed beside + -- yield_count, which is where it belongs by meaning: both MCP surfaces read this result set + -- POSITIONALLY and Lite's DuckDB read mirrors these ordinals, so inserting mid-list would + -- silently re-map every column after it in whichever surface was not edited in the same + -- breath. An ABANDONED run was previously counted by total_runs and by nothing else, so it + -- grew the failure-rate denominator while contributing nothing to the numerator. + -- + -- #2926: keyed on the ROW, not on the status alone. collection_log is append-only, so a + -- window can still hold cycles written before #2803 gave abandonment its own status: + -- status = 'SUCCESS' beside rows_collected = 0 and the budget note. Counted by status + -- alone this read 0 for them, and the collector banded HEALTHY while losing cycles - a + -- filter correct against current writes and silently wrong against older ones, failing in + -- the reassuring direction. The pattern is one LIKE because the budget is INTERPOLATED and + -- the shipped values differ (120 s for procedure_stats/query_stats/plan_correction, 600 s + -- for query_store), so equality against one rendered sentence matches one collector. + SUM(CASE WHEN {EnumeratedCollectorDriver.AbandonedRunPredicateSql} + THEN 1 ELSE 0 END) AS abandoned_count, + -- #3010: the newest DENIAL on its own, which is what dates the last_error slot above. + -- last_error_time cannot stand in for it: that column is a MAX over ERROR and PERMISSIONS + -- together, so on a collector carrying both it hands a reader an error's instant and lets + -- them call it a denial. Compared against last_success_time this separates a collector + -- still being refused from one whose refusals all predate a later success -- the exact + -- distinction pg_deadlocks needed and no surface could make. + -- + -- Appended LAST, like abandoned_count before it: both MCP surfaces read this result set + -- POSITIONALLY and Lite's DuckDB read mirrors these ordinals, so a mid-list insert would + -- silently re-map every column after it in whichever surface was not edited in the same + -- breath. + MAX(CASE WHEN status = 'PERMISSIONS' THEN collection_time END) AS last_denied_time, + -- #3017: what the spend BOUGHT. Every other statistic on a health row describes cost -- + -- total_runs, the three durations, and the sweep-pressure roll-up built from them -- and the + -- rows figure lived on get_collector_cost, a different tool over a different (hourly, fleet- + -- wide) series. Correlating spend against output was a join a reader had to know to make. + -- Measured: pg_deadlocks was the dearest collector on a managed store, 49,258,335 ms over + -- 79,333 runs in seven days, and stored zero rows. + -- + -- Read from THIS query's own window so cost and output cannot describe different runs. That + -- is the same reason #3010's two instants come out of one aggregate: a rows figure taken + -- from one window beside a duration taken from another composes into a ratio describing no + -- collector that ever ran. + -- + -- COALESCE so a zero is unambiguous at the STORE. Without it a collector whose every + -- rows_collected is NULL returns NULL here, which a reader would have to guess between "this + -- read did not measure output" and "it stored nothing" -- and the whole point of the column + -- is that the second of those becomes a fact rather than an absence. + COALESCE(SUM(rows_collected), 0) AS rows_stored, + -- The DENOMINATOR's partner, and the honest half of a cost/output pair: 12 rows over 3 of + -- 79,333 runs is a different collector from 12 rows over all of them. get_pg_blocking already + -- reports captures_with_blocking beside captures_total for exactly this reason, off this same + -- rows_collected > 0 test. total_runs above is the denominator; this is the numerator. + -- + -- APPENDED, like every column since #2472: both MCP surfaces read this result set + -- POSITIONALLY and Lite's DuckDB read mirrors these ordinals, so a mid-list insert would + -- silently re-map every column after it in whichever surface was not edited in the same + -- breath. + SUM(CASE WHEN rows_collected > 0 THEN 1 ELSE 0 END) AS runs_with_rows, + -- #3240: runs skipped because a PostgreSQL extension the collector DECLARES is not installed + -- — the EXTENSION_MISSING status the fault mapper split out of PERMISSIONS, counted apart so + -- the banding stops calling an uninstalled optional extension NO_PERMISSIONS. APPENDED, like + -- every column since #2472, because this result set is read positionally. + SUM(CASE WHEN status = 'EXTENSION_MISSING' THEN 1 ELSE 0 END) AS extension_missing_count, + -- #3754: runs whose XE session was missing or could not be created - the SESSION_MISSING status + -- the tolerant XE readers write for a permission-denied session read and, since #3754, the + -- long-query reconcile writes for a session refused in every database. Until now this status + -- reached this surface as total_runs and NOTHING else: not an error, not a success, not a + -- denial, so a collector whose every run was SESSION_MISSING banded FAILING on the never- + -- succeeded clock while the row said errors 0 and the output finding said it had read and + -- found nothing. Counted apart from error_count on purpose - it is not fed to the band, whose + -- capture-down story belongs to the self-alert - and fed with error_count to the output + -- finding as the runs that could not read. APPENDED, like every column since #2472, because + -- this result set is read positionally and Lite's DuckDB read mirrors the ordinals. + SUM(CASE WHEN status = 'SESSION_MISSING' THEN 1 ELSE 0 END) AS session_missing_count, + -- #3819: the three columns that tell a collector which STOPPED producing apart from one that + -- never produced here. The same named skip carries opposite meanings on those two rows, and + -- until now the surface read both as the benign resting state -- which is how an install that + -- took pg_statement_stats from 85% productive to EXTENSION_MISSING on 23 of 50 clusters was + -- accepted by the countersign for 24 hours. APPENDED, like every column since #2472, because + -- this result set is read positionally and Lite's DuckDB read mirrors the ordinals. + -- + -- current_status is what the collector is reporting NOW, for the finding's prose. Taken at + -- recency_rank = 1 rather than as a MAX over the skip rows: MAX is lexicographic, so on a + -- streak whose status changed it would name whichever word sorts highest instead of the one + -- being reported. + MAX(CASE WHEN recency_rank = 1 THEN status END) AS current_status, + -- The instant the current skip streak began AFTER: the newest run that was NOT a named skip. + -- The vocabulary is interpolated from CollectorRuntimePrecondition, which is where each of + -- those statuses is declared, so this cannot ask about three of four after a fourth is split + -- out. A NULL status counts as non-skip: it is not one of the declared skip words, and reading + -- it as one would let an unwritten status manufacture a streak. + MAX(CASE WHEN status IS NULL + OR status NOT IN ({CollectorRuntimePrecondition.NamedSkipStatusSqlList}) + THEN collection_time END) AS last_non_skip_time, + -- The newest run that stored anything. Off the same rows_collected > 0 test as runs_with_rows + -- above, so productive means one thing on this row. Compared against last_non_skip_time it + -- says the productivity sits BEFORE the streak rather than inside it, which is the ORDER that + -- makes this a regression rather than two unrelated facts. + MAX(CASE WHEN rows_collected > 0 THEN collection_time END) AS last_productive_time, + -- #3885: how many runs, counting back from the NEWEST, were SUCCESS with zero rows and nothing + -- else. The column that makes the #3819 regression arm able to see the class it could not: + -- job_history recorded SUCCESS / 0 rows run after run on 41 of 43 servers of the largest + -- production store for up to a fortnight -- 1,900+ consecutive such runs on one of them -- and + -- banded HEALTHY throughout, because #3819 keys on a skip STATUS and this collector's status was + -- the most reassuring word the vocabulary has. + -- + -- Exact, and FREE: recency_rank already exists in the subquery below (#3819 added it for + -- current_status), so this buys the streak's true width with no new window function and no new + -- sort. MIN of the rank of the newest run that BREAKS the streak, minus one, is the count of + -- runs ahead of it; NULL (no run breaks it -- every run in the window is a zero-row success) + -- falls back to COUNT(*), which is that same count. A streak broken by an error, a denial, a + -- skip or a productive run therefore reads 0 rather than reaching past it, because what this + -- measures is the collector's CURRENT state and any of those is a different current state. + -- + -- The abandonment exclusion is the one success_count above uses, for the same #2926 reason: a + -- pre-#2803 abandoned cycle is stored as SUCCESS with zero rows and the budget note, which is + -- data LOSS rather than a source that went quiet, and counting it here would attribute an + -- abandonment to a regression. APPENDED, like every column since #2472, because this result set + -- is read positionally. + COALESCE( + MIN(CASE WHEN NOT (status = 'SUCCESS' + AND COALESCE(rows_collected, 0) = 0 + AND NOT {EnumeratedCollectorDriver.AbandonedByNotePredicateSql}) + THEN recency_rank END) - 1, + COUNT(*)) AS trailing_zero_row_success_runs, + -- #4748: the note the collector's NEWEST run left, which is not last_note above. last_note is the + -- newest run that CARRIED a note (note_rank), so a clean run after a partial-failure cycle still + -- shows the older cycle's note there; the band must not read that, because the loss it names + -- is not the collector's current state. recency_rank = 1 is the newest run of any status, and the + -- SUCCESS gate matches last_note's (only the SUCCESS write carries a note). APPENDED like every + -- column since #2472, because this result set is read positionally and Lite's DuckDB read + -- mirrors the ordinals. + MAX(CASE WHEN recency_rank = 1 AND status = 'SUCCESS' THEN error_message END) AS latest_run_note + FROM + ( + -- #1855: rank each class of message newest-first so the two exemplar columns above can take + -- the LATEST one instead of the lexicographically greatest. Ordering on whether the class's + -- CASE came back empty puts every row that carries such a message ahead of every row that + -- does not, so rank 1 is the newest one that has text — and a later clean run no longer + -- blanks a note the window still holds. Text does not sort like the number #1837's probe + -- note carries: 12 item(s) sorts below 9 item(s). One byte-identical shape with Lite's + -- DuckDB read and the Viewer's, down to the error_message DESC tie-break. + SELECT + collector_name, + collection_time, + duration_ms, + status, + error_message, + -- #2926: the abandonment predicate above reads it. Projected here for the same reason + -- #2472's three columns are: this subquery ENUMERATES its columns, so an aggregate + -- outside naming one it does not carry fails at the STORE and nowhere earlier. + rows_collected, + -- #2472: projected here because this subquery enumerates its columns rather than + -- SELECT *-ing them, so an aggregate outside that names a column the inner query does + -- not carry fails at the STORE and nowhere earlier — no compiler, no text assertion and + -- no local build can see it. + fanout_item_count, + slowest_item, + slowest_item_ms, + ROW_NUMBER() OVER + ( + PARTITION BY collector_name + ORDER BY (CASE WHEN status = 'SUCCESS' THEN error_message END) IS NULL, + collection_time DESC, + error_message DESC + ) AS note_rank, + ROW_NUMBER() OVER + ( + PARTITION BY collector_name + ORDER BY (CASE WHEN status IN ('ERROR', 'PERMISSIONS', 'EXTENSION_MISSING') THEN error_message END) IS NULL, + collection_time DESC, + error_message DESC + ) AS error_rank, + -- #2472: the window's dearest single ITEM, and with it the run that carried it. Ranked on + -- slowest_item_ms rather than duration_ms because the question is which database is + -- expensive, not which cycle was: a run whose total is the largest only because every + -- database was busy is exactly the shape a per-database override should NOT be aimed at. + ROW_NUMBER() OVER + ( + PARTITION BY collector_name + ORDER BY slowest_item_ms IS NULL, + slowest_item_ms DESC, + collection_time DESC + ) AS slowest_rank, + -- #3819: newest run first, so current_status above can take the status the collector is + -- reporting NOW. status DESC only breaks an exact-timestamp tie, and breaks it identically + -- on DuckDB and Postgres -- the same reason the ranks above tie-break on error_message. + ROW_NUMBER() OVER + ( + PARTITION BY collector_name + ORDER BY collection_time DESC, + status DESC + ) AS recency_rank + FROM v_collection_log + WHERE server_id = $1 + AND collection_time >= $2 + ) runs + GROUP BY collector_name + ORDER BY collector_name + """; + + /// The WPF viewer's statement (ViewerDataService.CollectionHealthSql) before #4955: 19 columns. + public const string Viewer = $""" + SELECT + collector_name, + COUNT(*) AS total_runs, + -- #2926: SUCCESS excludes an abandonment that predates #2803, so the Success column beside + -- Abandoned cannot count the same run twice. Post-#2803 rows need no exclusion - ABANDONED + -- is not SUCCESS - and an ordinary empty run stays counted, which is what the COALESCE in + -- the shared predicate is for: NULL under this NOT would have dropped it. + SUM(CASE WHEN status = 'SUCCESS' + AND NOT {EnumeratedCollectorDriver.AbandonedByNotePredicateSql} + THEN 1 ELSE 0 END) AS success_count, + SUM(CASE WHEN status = 'ERROR' THEN 1 ELSE 0 END) AS error_count, + AVG(duration_ms) AS avg_duration_ms, + -- SKIPPED counts as a healthy run (dedup / version-gated collectors no-op without being stale) + MAX(CASE WHEN status IN ('SUCCESS', 'SKIPPED') THEN collection_time END) AS last_success_time, + MAX(collection_time) AS last_run_time, + -- #1855: the message from the NEWEST failing run, not MAX()'s lexicographically greatest + -- one. The status re-check is load-bearing rather than belt-and-braces: when no failing run + -- in the window carried text, error_rank = 1 falls through to the newest row of ANY class, + -- and without it a SUCCESS row's note could surface here as a fake last error. + -- #3240: EXTENSION_MISSING is in the exemplar set because its stored sentence IS the remedy + -- (it names the extension and the database) — without it an EXTENSION_MISSING band would sit + -- beside a blank Last Error and the operator would have to open the run log to learn why. + MAX(CASE WHEN error_rank = 1 AND status IN ('ERROR', 'PERMISSIONS', 'EXTENSION_MISSING') THEN error_message END) AS last_error, + -- The newest failure OUTRIGHT, text or not — "when did this last fail" means the run, not + -- the message. It can only name a different row than last_error if a failure was written + -- with no text. + MAX(CASE WHEN status IN ('ERROR', 'PERMISSIONS', 'EXTENSION_MISSING') THEN collection_time END) AS last_error_time, + SUM(CASE WHEN status = 'PERMISSIONS' THEN 1 ELSE 0 END) AS permission_denied_count, + -- YIELDED = the 1s LOCK_TIMEOUT guard fired (#1805): deliberate, benign for collection, + -- counted apart from errors because clustering here is a signal about the TARGET's lock + -- contention rather than a monitoring fault. + SUM(CASE WHEN status = 'YIELDED' THEN 1 ELSE 0 END) AS yield_count, + -- #1837: the note a SUCCEEDING run can leave behind (an enumeration that yielded 0 items, + -- items whose enumeration probe failed). Gated on SUCCESS specifically, rather than on + -- every non-failure status: the runners attach a note only to the SUCCESS write, and the looser + -- complement would drag SESSION_MISSING and CANCELLED messages into a column whose whole + -- claim is that it is NOT an error. Display text only: no band, no count, no threshold + -- reads it, and a legitimately empty target stays HEALTHY exactly as before. + MAX(CASE WHEN note_rank = 1 AND status = 'SUCCESS' THEN error_message END) AS last_note, + -- How many of the window's runs carried one. note_count = total_runs is the + -- persistently-empty signal: EVERY run this week came back with nothing. + COUNT(CASE WHEN status = 'SUCCESS' THEN error_message END) AS note_count, + -- #1852: the one thing that makes a persistently-empty enumeration interesting — does this + -- target actually HAVE user databases? Zero items on a server with none is legitimate and + -- stays quiet; zero items on a server that HAS them is a login that cannot enter any, or a + -- filter that swallowed everything. database_size_stats rather than database_config for + -- three reasons: it runs on the scheduled loop (60 min) where database_config is on-load + -- and can age past this window on a long-running service; it is indexed on + -- (server_id, collection_time) where database_config has no index at all; and it reads + -- sys.master_files, so it still sees databases the monitoring login cannot ENTER — exactly + -- the case being diagnosed. database_id > 4 excludes the system databases, tempdb + -- included: the size collector takes every ONLINE database, so a bare row check + -- would be true on every server alive. A NULL database_id is an Azure sibling row + -- (#2643/#3262): sys.resource_stats carries no id, and it bills only USER databases, + -- so those rows are inventory too — without the IS NULL arm a master-connected Azure + -- target with fifty user databases would read as having none. + -- + -- The inventory window is the health read's OWN ($2) — no second parameter, and an + -- inventory that aged out says nothing rather than something stale. Uncorrelated, so both + -- engines evaluate it once per query (a Postgres InitPlan) instead of per row, and it + -- needs no GROUP BY entry and no join. Feeds display text only, through the shared + -- formatter; the banding never sees it. + CASE + WHEN EXISTS + ( + SELECT 1 + FROM v_database_size_stats + WHERE server_id = $1 + AND collection_time >= $2 + AND (database_id > 4 OR database_id IS NULL) + ) + THEN 1 + ELSE 0 + END AS has_user_databases, + -- #2804: runs the #2673 wall-clock budget abandoned. Appended last — this result set is + -- read positionally by one shared mapper serving BOTH the per-server and fleet reads. + -- + -- #2926: keyed on the ROW, not on the status alone. collection_log is append-only, so a + -- window can still hold cycles written before #2803 gave abandonment its own status: + -- status = 'SUCCESS' beside rows_collected = 0 and the budget note. Counted by status + -- alone this read 0 for them, and the collector banded HEALTHY while losing cycles - a + -- filter correct against current writes and silently wrong against older ones, failing in + -- the reassuring direction. The pattern is one LIKE because the budget is INTERPOLATED and + -- the shipped values differ (120 s for procedure_stats/query_stats/plan_correction, 600 s + -- for query_store), so equality against one rendered sentence matches one collector. + SUM(CASE WHEN {EnumeratedCollectorDriver.AbandonedRunPredicateSql} + THEN 1 ELSE 0 END) AS abandoned_count, + -- #3240: runs skipped because a PostgreSQL extension the collector DECLARES is not installed + -- — the EXTENSION_MISSING status the fault mapper split out of PERMISSIONS, counted apart so + -- the banding stops calling an uninstalled optional extension NO_PERMISSIONS. APPENDED, never + -- inserted: this result set is read positionally by one shared mapper. + SUM(CASE WHEN status = 'EXTENSION_MISSING' THEN 1 ELSE 0 END) AS extension_missing_count, + -- #3819: the two instants that, with last_run_time above, say whether this collector + -- STOPPED producing rather than never having produced here. The same named skip carries + -- opposite meanings on those two rows, and the band read both as the benign resting state. + -- This grid bands through the SAME shared classifier as the service's reads, so it has to + -- feed it the same inputs — left unselected they default to null, this COMPILES, and the + -- grid would call a regressed collector HEALTHY while get_collection_health called it + -- WARNING. That is #3240's lesson and #2804's before it. The FINDING that names the rows and + -- the status is deliberately not carried here: this projection has no rows_stored to count + -- and the grid has no column to render prose in, so the band is the whole of what this + -- surface needs to agree about. APPENDED, read positionally by the one shared mapper. + MAX(CASE WHEN status IS NULL + OR status NOT IN ({CollectorRuntimePrecondition.NamedSkipStatusSqlList}) + THEN collection_time END) AS last_non_skip_time, + MAX(CASE WHEN rows_collected > 0 THEN collection_time END) AS last_productive_time, + -- #4748: the note the collector's NEWEST run left, which is not last_note above (that is the + -- newest run that CARRIED a note, so a clean run after a partial-failure cycle still shows the + -- older cycle's note there). The band reads only this one, because the loss an older note names + -- is not the collector's current state. APPENDED, read positionally by the one shared mapper. + MAX(CASE WHEN recency_rank = 1 AND status = 'SUCCESS' THEN error_message END) AS latest_run_note + FROM + ( + -- #1855: rank each class of message newest-first so the two exemplar columns above can take + -- the LATEST one instead of the lexicographically greatest. Ordering on whether the class's + -- CASE came back empty puts every row that carries such a message ahead of every row that + -- does not, so rank 1 is the newest one that has text — and a later clean run no longer + -- blanks a note the window still holds. MAX() was never wrong about WHICH rows to consider, + -- only about which of them wins, and text does not sort like the number #1837's probe note + -- carries: 12 item(s) sorts below 9 item(s). collection_time settles it; error_message DESC + -- only breaks an exact-timestamp tie, and breaks it identically here and in Lite's DuckDB + -- twin, which binary-vs-locale collation would not. + SELECT + collector_name, + collection_time, + duration_ms, + status, + error_message, + -- #2926: the abandonment predicate above reads it. Projected here for the same reason + -- #2472's three columns are: this subquery ENUMERATES its columns, so an aggregate + -- outside naming one it does not carry fails at the STORE and nowhere earlier. + rows_collected, + ROW_NUMBER() OVER + ( + PARTITION BY collector_name + ORDER BY (CASE WHEN status = 'SUCCESS' THEN error_message END) IS NULL, + collection_time DESC, + error_message DESC + ) AS note_rank, + ROW_NUMBER() OVER + ( + PARTITION BY collector_name + ORDER BY (CASE WHEN status IN ('ERROR', 'PERMISSIONS', 'EXTENSION_MISSING') THEN error_message END) IS NULL, + collection_time DESC, + error_message DESC + ) AS error_rank, + -- #4748: newest run first, so latest_run_note above takes the newest run's note. status DESC + -- only breaks an exact-timestamp tie, the way the Darling service read breaks it. + ROW_NUMBER() OVER + ( + PARTITION BY collector_name + ORDER BY collection_time DESC, + status DESC + ) AS recency_rank + FROM v_collection_log + WHERE server_id = $1 + AND collection_time >= $2 + ) runs + GROUP BY collector_name + ORDER BY collector_name + """; +} diff --git a/Darling/Darling.Tests/CollectionHealthRunTimeFieldsLiveTests.cs b/Darling/Darling.Tests/CollectionHealthRunTimeFieldsLiveTests.cs new file mode 100644 index 0000000000..11d92c6847 --- /dev/null +++ b/Darling/Darling.Tests/CollectionHealthRunTimeFieldsLiveTests.cs @@ -0,0 +1,587 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4938: get_collection_health shows a collector's run time. A collector with a run time carries +/// run_at (24-hour HH:MM on the monitored server's clock) and next_run_utc (the next time it is +/// due, UTC). A collector without one carries null for both on a full row, and nothing for either on a partial or a +/// compact row. The server's row wins over the fleet row, a run time of +/// -1 on the server's row stops the fleet's, and a collector whose interval is not a whole number of days has none. +/// The health band is read from the shipped cadence and a run time does not move it. A server with no clock yet reads +/// the run time as UTC. A day the collector skipped crosses the stale line before the next slot, and the row carries +/// a run_time_note that says so. +/// +/* #1776 own-store: each fact mints its own scratch database through ScratchPostgres and never touches the shared + store's tables, so it cannot race the live collection and serializing it would be pure slowdown. */ +public sealed class CollectionHealthRunTimeFieldsLiveTests +{ + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private const string SkipText = "Set DARLING_TEST_PG to a Postgres connection string to run the run-time health fields' live pins (each mints its own scratch database)."; + + /* Two collectors on a daily interval (the second is an on-load collector the shipped schedule recaptures daily) and + one on a one-minute interval. */ + private const string DailyCollector = "index_object_stats"; + private const string SecondDailyCollector = "server_config"; + private const string MinuteCollector = "wait_stats"; + + /* A run time 12 hours from now, so today's slot and the one before it are both more than 11 hours from now whatever + the server's spread is, and a test never lands inside the 60-minute window after a slot. A fixed 02:00 would sit in + that window for two hours of every day. */ + private static int RunTimeHalfADayAway(DateTime nowUtc) => (nowUtc.Hour * 60 + nowUtc.Minute + 12 * 60) % 1440; + + /* The UTC time of day of a slot: the run time on a clock this many minutes behind UTC, plus the server's spread. */ + private static TimeSpan SlotTimeOfDayUtc(int runAtMinute, int serverId, int minutesBehindUtc = 0) + { + var slot = TimeSpan.FromMinutes(runAtMinute + minutesBehindUtc) + CollectorRunTime.Spread(serverId); + return TimeSpan.FromTicks(slot.Ticks % TimeSpan.FromDays(1).Ticks); + } + + private static async Task MigratedAsync(ScratchPostgres scratch, CancellationToken ct) + { + var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + return connection; + } + + private static async Task RegisterServerAsync(NpgsqlConnection connection, int serverId, string name, CancellationToken ct) + { + await using var command = new NpgsqlCommand( + """ + INSERT INTO servers (server_id, server_name, display_name, is_enabled, sql_major_version, created_date, modified_date) + VALUES ($1, $2, $2, TRUE, 15, $3, $3) + """, connection); + command.Parameters.AddWithValue(serverId); + command.Parameters.AddWithValue(name); + command.Parameters.AddWithValue(DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified)); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task LogRunAsync(NpgsqlConnection connection, int serverId, string serverName, string collector, DateTime whenUtc, CancellationToken ct) + { + await using var command = new NpgsqlCommand( + """ + INSERT INTO collection_log (log_id, collection_time, server_id, server_name, collector_name, status, duration_ms, rows_collected) + VALUES ($1, $2, $3, $4, $5, 'SUCCESS', 120, 10) + """, connection); + command.Parameters.AddWithValue(CollectionIdGenerator.Next()); + command.Parameters.AddWithValue(DateTime.SpecifyKind(whenUtc, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(serverId); + command.Parameters.AddWithValue(serverName); + command.Parameters.AddWithValue(collector); + await command.ExecuteNonQueryAsync(ct); + } + + /// One failed run of a collector: an ERROR row in the window, which keeps the collector off the compact shape + /// and gives it the partial one. + private static async Task LogFailureAsync(NpgsqlConnection connection, int serverId, string serverName, string collector, DateTime whenUtc, CancellationToken ct) + { + await using var command = new NpgsqlCommand( + """ + INSERT INTO collection_log (log_id, collection_time, server_id, server_name, collector_name, status, duration_ms, rows_collected, error_message) + VALUES ($1, $2, $3, $4, $5, 'ERROR', 120, 0, 'a failed run') + """, connection); + command.Parameters.AddWithValue(CollectionIdGenerator.Next()); + command.Parameters.AddWithValue(DateTime.SpecifyKind(whenUtc, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(serverId); + command.Parameters.AddWithValue(serverName); + command.Parameters.AddWithValue(collector); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task PlantOffsetAsync(NpgsqlConnection connection, int serverId, string serverName, int offsetMinutes, CancellationToken ct) + { + await using var command = new NpgsqlCommand( + """ + INSERT INTO server_properties (collection_id, collection_time, server_id, server_name, utc_offset_minutes) + VALUES ($1, $2, $3, $4, $5) + """, connection); + command.Parameters.AddWithValue(CollectionIdGenerator.Next()); + command.Parameters.AddWithValue(DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(serverId); + command.Parameters.AddWithValue(serverName); + command.Parameters.AddWithValue(offsetMinutes); + await command.ExecuteNonQueryAsync(ct); + } + + /// One run-time row (config.config_collector_run_times): a fleet row when is null, + /// else that server's. The run time is minutes after midnight, or -1 for "no fixed time". A run time needs no schedule row, so + /// one is written only for a collector that is switched OFF ( false): every other call reads the + /// run time with no schedule row at all. + private static async Task SetRunTimeAsync(NpgsqlConnection connection, int? serverId, string collector, int minute, CancellationToken ct, bool enabled = true) + { + await using var command = new NpgsqlCommand( + "INSERT INTO config.config_collector_run_times (server_id, collector_name, run_at_minute) VALUES ($1, $2, $3)", connection); + command.Parameters.Add(new NpgsqlParameter { Value = (object?)serverId ?? DBNull.Value, NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Integer }); + command.Parameters.AddWithValue(collector); + command.Parameters.Add(new NpgsqlParameter { Value = (short)minute, NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Smallint }); + await command.ExecuteNonQueryAsync(ct); + + if (!enabled) + { + await using var schedule = new NpgsqlCommand( + "INSERT INTO config.config_collector_schedules (server_id, collector_name, enabled) VALUES ($1, $2, FALSE)", connection); + schedule.Parameters.Add(new NpgsqlParameter { Value = (object?)serverId ?? DBNull.Value, NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Integer }); + schedule.Parameters.AddWithValue(collector); + await schedule.ExecuteNonQueryAsync(ct); + } + } + + private static Task SetFleetRunTimeAsync(NpgsqlConnection connection, string collector, int minute, CancellationToken ct) => + SetRunTimeAsync(connection, null, collector, minute, ct); + + /// The rows of get_collection_health by collector. Full detail by default, because a healthy + /// collector otherwise compacts to seven fields. + private static async Task> ReadRowsAsync(NpgsqlDataSource postgres, string server, CancellationToken ct, bool fullDetail = true) + { + var json = await DarlingMcpDataTools.GetCollectionHealth(postgres, server, full_detail: fullDetail, cancellationToken: ct); + using var doc = JsonDocument.Parse(json); + Assert.True(doc.RootElement.TryGetProperty("collectors", out var collectors), json.Length > 400 ? json[..400] : json); + return collectors.EnumerateArray().ToDictionary(r => r.GetProperty("collector").GetString()!, r => r.Clone()); + } + + private static DateTime ParseUtc(JsonElement value) + { + var parsed = DateTime.Parse(value.GetString()!, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind); + Assert.Equal(DateTimeKind.Utc, parsed.Kind); + return parsed; + } + + [Fact] + public async Task ACollectorWithARunTime_ShowsRunAtAndNextRunUtc_AndACollectorWithoutOneShowsNullForBoth() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + var bodySucceeded = false; + try + { + const int serverId = 480_001; + const string server = "runtime-health-a"; + await RegisterServerAsync(connection, serverId, server, ct); + + /* Both ran a minute ago, so the daily collector's next slot is the one ahead of now. */ + var now = DateTime.UtcNow; + var runAt = RunTimeHalfADayAway(now); + await LogRunAsync(connection, serverId, server, DailyCollector, now.AddMinutes(-1), ct); + await LogRunAsync(connection, serverId, server, MinuteCollector, now.AddMinutes(-1), ct); + await SetFleetRunTimeAsync(connection, DailyCollector, runAt, ct); + + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var rows = await ReadRowsAsync(postgres, server, ct); + + var daily = rows[DailyCollector]; + Assert.Equal(CollectorRunTime.Format(runAt), daily.GetProperty("run_at").GetString()); + var next = ParseUtc(daily.GetProperty("next_run_utc")); + Assert.True(next > now, $"the next run {next:o} must be ahead of {now:o}"); + Assert.True(next <= now + CollectorRunTime.MaxStampAhead(1440), $"the next run {next:o} is more than a day and the spread away"); + + /* No clock for this server, so the run time reads as UTC: the run time plus its fixed spread. */ + Assert.Equal(SlotTimeOfDayUtc(runAt, serverId), next.TimeOfDay); + + /* Nothing was skipped: it ran a minute ago. */ + Assert.Equal(JsonValueKind.Null, daily.GetProperty("run_time_note").ValueKind); + + var minute = rows[MinuteCollector]; + Assert.Equal(JsonValueKind.Null, minute.GetProperty("run_at").ValueKind); + Assert.Equal(JsonValueKind.Null, minute.GetProperty("next_run_utc").ValueKind); + Assert.Equal(JsonValueKind.Null, minute.GetProperty("run_time_note").ValueKind); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + [Fact] + public async Task ARunTimeIsOnTheServersClock_AndAServerWithNoClockReadsItAsUtc() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + var bodySucceeded = false; + try + { + const int plainId = 480_011; + const int offsetId = 480_012; + const string plain = "runtime-health-utc"; + const string offset = "runtime-health-west"; + await RegisterServerAsync(connection, plainId, plain, ct); + await RegisterServerAsync(connection, offsetId, offset, ct); + + var now = DateTime.UtcNow; + var runAt = RunTimeHalfADayAway(now); + await LogRunAsync(connection, plainId, plain, DailyCollector, now.AddMinutes(-1), ct); + await LogRunAsync(connection, offsetId, offset, DailyCollector, now.AddMinutes(-1), ct); + + /* Five hours behind UTC, a fixed offset: its run time is five hours later in UTC. */ + await PlantOffsetAsync(connection, offsetId, offset, -300, ct); + await SetFleetRunTimeAsync(connection, DailyCollector, runAt, ct); + + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var plainRow = (await ReadRowsAsync(postgres, plain, ct))[DailyCollector]; + var offsetRow = (await ReadRowsAsync(postgres, offset, ct))[DailyCollector]; + + /* The run time is the same text on both: it is each server's own wall clock. */ + Assert.Equal(CollectorRunTime.Format(runAt), plainRow.GetProperty("run_at").GetString()); + Assert.Equal(CollectorRunTime.Format(runAt), offsetRow.GetProperty("run_at").GetString()); + Assert.Equal(SlotTimeOfDayUtc(runAt, plainId), ParseUtc(plainRow.GetProperty("next_run_utc")).TimeOfDay); + Assert.Equal(SlotTimeOfDayUtc(runAt, offsetId, minutesBehindUtc: 300), ParseUtc(offsetRow.GetProperty("next_run_utc")).TimeOfDay); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + [Fact] + public async Task AServerRunTimeWinsOverTheFleetRunTime_MinusOneOnTheServerStopsIt_AndAnIntervalThatIsNotWholeDaysHasNone() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + var bodySucceeded = false; + try + { + const int ownId = 480_031; + const int noneId = 480_032; + const int fleetId = 480_033; + const string own = "runtime-health-own"; + const string none = "runtime-health-none"; + const string fleetOnly = "runtime-health-fleet"; + var now = DateTime.UtcNow; + foreach (var (id, name) in new[] { (ownId, own), (noneId, none), (fleetId, fleetOnly) }) + { + await RegisterServerAsync(connection, id, name, ct); + await LogRunAsync(connection, id, name, DailyCollector, now.AddMinutes(-1), ct); + await LogRunAsync(connection, id, name, MinuteCollector, now.AddMinutes(-1), ct); + } + + /* The fleet runs the daily collector at 02:00. One server has its own 04:30, and one has -1, which means no + fixed time there and stops the fleet's 02:00 from applying. The fleet row also names the one-minute + collector, which a run time cannot apply to. */ + await SetFleetRunTimeAsync(connection, DailyCollector, 120, ct); + await SetFleetRunTimeAsync(connection, MinuteCollector, 120, ct); + await SetRunTimeAsync(connection, ownId, DailyCollector, 270, ct); + await SetRunTimeAsync(connection, noneId, DailyCollector, -1, ct); + + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var ownRows = await ReadRowsAsync(postgres, own, ct); + var noneRows = await ReadRowsAsync(postgres, none, ct); + var fleetRows = await ReadRowsAsync(postgres, fleetOnly, ct); + + Assert.Equal("04:30", ownRows[DailyCollector].GetProperty("run_at").GetString()); + Assert.Equal(JsonValueKind.String, ownRows[DailyCollector].GetProperty("next_run_utc").ValueKind); + + Assert.Equal(JsonValueKind.Null, noneRows[DailyCollector].GetProperty("run_at").ValueKind); + Assert.Equal(JsonValueKind.Null, noneRows[DailyCollector].GetProperty("next_run_utc").ValueKind); + + Assert.Equal("02:00", fleetRows[DailyCollector].GetProperty("run_at").GetString()); + + foreach (var rows in new[] { ownRows, noneRows, fleetRows }) + { + Assert.Equal(JsonValueKind.Null, rows[MinuteCollector].GetProperty("run_at").ValueKind); + Assert.Equal(JsonValueKind.Null, rows[MinuteCollector].GetProperty("next_run_utc").ValueKind); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + [Fact] + public async Task ADisabledCollectorWithARunTime_ShowsItsRunTimeAndNoNextRun() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + var bodySucceeded = false; + try + { + const int serverId = 480_041; + const string server = "runtime-health-off"; + await RegisterServerAsync(connection, serverId, server, ct); + await LogRunAsync(connection, serverId, server, DailyCollector, DateTime.UtcNow.AddMinutes(-1), ct); + await SetRunTimeAsync(connection, null, DailyCollector, 120, ct, enabled: false); + + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var row = (await ReadRowsAsync(postgres, server, ct))[DailyCollector]; + + Assert.Equal("02:00", row.GetProperty("run_at").GetString()); + Assert.Equal(JsonValueKind.Null, row.GetProperty("next_run_utc").ValueKind); + Assert.Equal(JsonValueKind.Null, row.GetProperty("run_time_note").ValueKind); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + [Fact] + public async Task ACompactRowCarriesTheRunTimeOnlyWhenTheCollectorHasOne() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + var bodySucceeded = false; + try + { + const int serverId = 480_051; + const string server = "runtime-health-compact"; + await RegisterServerAsync(connection, serverId, server, ct); + var now = DateTime.UtcNow; + await LogRunAsync(connection, serverId, server, DailyCollector, now.AddMinutes(-1), ct); + await LogRunAsync(connection, serverId, server, MinuteCollector, now.AddMinutes(-1), ct); + var runAt = RunTimeHalfADayAway(now); + await SetFleetRunTimeAsync(connection, DailyCollector, runAt, ct); + + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var rows = await ReadRowsAsync(postgres, server, ct, fullDetail: false); + + /* Both are healthy with nothing to report, so both compact. A compact row stays small: it carries the run + time only for the collector that has one, and never the note, which a healthy row cannot need. */ + var daily = rows[DailyCollector]; + Assert.True(daily.GetProperty("compact").GetBoolean()); + Assert.Equal(CollectorRunTime.Format(runAt), daily.GetProperty("run_at").GetString()); + Assert.Equal(JsonValueKind.String, daily.GetProperty("next_run_utc").ValueKind); + Assert.False(daily.TryGetProperty("run_time_note", out _)); + + var minute = rows[MinuteCollector]; + Assert.True(minute.GetProperty("compact").GetBoolean()); + Assert.False(minute.TryGetProperty("run_at", out _)); + Assert.False(minute.TryGetProperty("next_run_utc", out _)); + Assert.False(minute.TryGetProperty("run_time_note", out _)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + [Fact] + public async Task APartialRowCarriesTheRunTimeOnlyWhenTheCollectorHasOne() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + var bodySucceeded = false; + try + { + const int serverId = 480_061; + const string server = "runtime-health-partial"; + await RegisterServerAsync(connection, serverId, server, ct); + var now = DateTime.UtcNow; + + /* A success and then a failure for each collector: an error in the window fails the compact test, so both + take the partial shape. */ + foreach (var collector in new[] { DailyCollector, MinuteCollector }) + { + await LogRunAsync(connection, serverId, server, collector, now.AddMinutes(-2), ct); + await LogFailureAsync(connection, serverId, server, collector, now.AddMinutes(-1), ct); + } + + var runAt = RunTimeHalfADayAway(now); + await SetFleetRunTimeAsync(connection, DailyCollector, runAt, ct); + + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var rows = await ReadRowsAsync(postgres, server, ct, fullDetail: false); + + /* Lite's partial row carries the run time only for a collector that has one, so a partial row stays as + small as the rest of its keys and the two apps give a caller the same shape. The note rides under the same + condition: a partial row for a collector with a run time carries it (null unless a day was skipped), and a + partial row for a collector with none has no run_time_note key at all. */ + var daily = rows[DailyCollector]; + Assert.True(daily.GetProperty("partial_detail").GetBoolean()); + Assert.Equal(CollectorRunTime.Format(runAt), daily.GetProperty("run_at").GetString()); + Assert.Equal(JsonValueKind.String, daily.GetProperty("next_run_utc").ValueKind); + Assert.Equal(JsonValueKind.Null, daily.GetProperty("run_time_note").ValueKind); + + var minute = rows[MinuteCollector]; + Assert.True(minute.GetProperty("partial_detail").GetBoolean()); + Assert.False(minute.TryGetProperty("run_at", out _)); + Assert.False(minute.TryGetProperty("next_run_utc", out _)); + Assert.False(minute.TryGetProperty("run_time_note", out _)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + [Fact] + public async Task TheHealthBand_IsTheSameWithAndWithoutARunTime_AndTheRunTimeIsNotHeldByTheHealthMemo() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + var bodySucceeded = false; + try + { + const int serverId = 480_021; + const string server = "runtime-health-band"; + await RegisterServerAsync(connection, serverId, server, ct); + + /* A daily collector that last ran 40 hours ago is past the 36-hour stale line. Skipping a day is a real + miss and the row says so; a run time does not excuse it. */ + await LogRunAsync(connection, serverId, server, DailyCollector, DateTime.UtcNow.AddHours(-40), ct); + + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var before = (await ReadRowsAsync(postgres, server, ct))[DailyCollector]; + Assert.Equal(JsonValueKind.Null, before.GetProperty("run_at").ValueKind); + Assert.Equal(CollectorHealthClassifier.Stale, before.GetProperty("status").GetString()); + + /* The health rows are memoized for the call that follows. The run time is read fresh, so an edit shows. */ + var runAt = RunTimeHalfADayAway(DateTime.UtcNow); + await SetFleetRunTimeAsync(connection, DailyCollector, runAt, ct); + var after = (await ReadRowsAsync(postgres, server, ct))[DailyCollector]; + + Assert.Equal(CollectorHealthClassifier.Stale, after.GetProperty("status").GetString()); + Assert.Equal(CollectorRunTime.Format(runAt), after.GetProperty("run_at").GetString()); + Assert.Equal(JsonValueKind.String, after.GetProperty("next_run_utc").ValueKind); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + [Fact] + public async Task ASkippedDay_CrossesTheStaleLineBeforeTheNextSlot_AndTheRowSaysSo() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + var bodySucceeded = false; + try + { + const int serverId = 480_061; + const string server = "runtime-health-skipped"; + await RegisterServerAsync(connection, serverId, server, ct); + + /* The daily collector last ran 40 hours ago, so it missed a day and is past its 36-hour stale line, and its + next slot is half a day away. The second daily collector ran a minute ago: nothing was skipped there. */ + var now = DateTime.UtcNow; + var runAt = RunTimeHalfADayAway(now); + await LogRunAsync(connection, serverId, server, DailyCollector, now.AddHours(-40), ct); + await LogRunAsync(connection, serverId, server, SecondDailyCollector, now.AddMinutes(-1), ct); + await SetFleetRunTimeAsync(connection, DailyCollector, runAt, ct); + await SetFleetRunTimeAsync(connection, SecondDailyCollector, runAt, ct); + + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var rows = await ReadRowsAsync(postgres, server, ct); + + var skipped = rows[DailyCollector]; + Assert.Equal(CollectorHealthClassifier.Stale, skipped.GetProperty("status").GetString()); + var next = ParseUtc(skipped.GetProperty("next_run_utc")); + Assert.True(next > now, "the next slot is still ahead of the stale reading"); + var note = skipped.GetProperty("run_time_note").GetString(); + Assert.NotNull(note); + Assert.Contains("Skipped day", note, StringComparison.Ordinal); + Assert.Contains("36-hour stale line", note, StringComparison.Ordinal); + Assert.Contains(next.ToString("u", CultureInfo.InvariantCulture), note, StringComparison.Ordinal); + Assert.Contains("not replayed", note, StringComparison.Ordinal); + + var onTime = rows[SecondDailyCollector]; + Assert.Equal(CollectorHealthClassifier.Healthy, onTime.GetProperty("status").GetString()); + Assert.Equal(JsonValueKind.Null, onTime.GetProperty("run_time_note").ValueKind); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + [Fact] + public async Task ACollectorDueNowInsideItsGrace_HasNoSkippedDayNote_EvenPastTheStaleLine() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + var bodySucceeded = false; + try + { + const int serverId = 480_071; + const string server = "runtime-health-due"; + await RegisterServerAsync(connection, serverId, server, ct); + + /* A run time whose slot for this server passed about ten minutes ago: now is inside the 60-minute grace, so + the collector is due now. It last ran 40 hours ago, but a run is a moment away, so there is no note. */ + var now = DateTime.UtcNow; + var spreadMinutes = (int)CollectorRunTime.Spread(serverId).TotalMinutes; + var runAt = ((now.Hour * 60 + now.Minute - spreadMinutes - 10) % 1440 + 1440) % 1440; + await LogRunAsync(connection, serverId, server, DailyCollector, now.AddHours(-40), ct); + await SetFleetRunTimeAsync(connection, DailyCollector, runAt, ct); + + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var row = (await ReadRowsAsync(postgres, server, ct))[DailyCollector]; + + Assert.Equal(CollectorHealthClassifier.Stale, row.GetProperty("status").GetString()); + var next = ParseUtc(row.GetProperty("next_run_utc")); + Assert.True(Math.Abs((next - DateTime.UtcNow).TotalMinutes) < 2, $"due now: the next run {next:o} is the time of the read"); + Assert.Equal(JsonValueKind.Null, row.GetProperty("run_time_note").ValueKind); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } +} diff --git a/Darling/Darling.Tests/CollectionHealthScheduleOverridesLiveTests.cs b/Darling/Darling.Tests/CollectionHealthScheduleOverridesLiveTests.cs new file mode 100644 index 0000000000..6c26da7eb0 --- /dev/null +++ b/Darling/Darling.Tests/CollectionHealthScheduleOverridesLiveTests.cs @@ -0,0 +1,207 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Reflection; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4999: the collection-health reads judge each collector against the interval it is scheduled at on its server, read +/// from config.config_collector_schedules. The schedule read fails open to the shipped cadences (a collection +/// health read must still answer when it cannot read the table), so every other test of the health reads would +/// still pass if DarlingDataReader.ScheduleOverridesSql or DarlingDataReader.AllScheduleOverridesSql broke. +/// This one seeds the rows and reads the health back, so a read that falls back shows up as the wrong band. +/// +/// Three five-minute collectors, each with a success six hours old: past the four hours a five-minute collector +/// goes STALE at, well inside the 18 hours (one and a half intervals) a collector scheduled every 720 minutes gets. +/// The first has a server row (720) AND a fleet row (5): the server row wins, so HEALTHY. The second has only a fleet +/// row (720): HEALTHY. The third has no row: the shipped five minutes, so STALE. The same server is read through the +/// MCP read and through the viewer's server tab and its fleet breakdown, and, in the second test, through the +/// service's fleet roll-up, which reads every server's override rows in one statement. +/// +[Collection("live-postgres")] +public sealed class CollectionHealthScheduleOverridesLiveTests +{ + private const string ServerName = "darling-collection-health-schedule-overrides-e2e"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + + private const string ServerRowWins = "memory_clerks"; + private const string FleetRowApplies = "memory_pressure_events"; + private const string ShippedDefault = "session_summary_stats"; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + [Fact] + public async Task EachCollectorIsJudgedAgainstTheServerRow_ThenTheFleetRow_ThenTheShippedDefault() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to run the live collection-health schedule-override test."); + var ct = TestContext.Current.CancellationToken; + + await using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + + await using var postgres = NpgsqlDataSource.Create(cs!); + var bodySucceeded = false; + try + { + await SeedAsync(connection, ct); + + /* The MCP per-server read. */ + var mcp = await DarlingDataReader.GetCollectionHealthAsync( + postgres, ServerId, DarlingMcpTestData.Naive(DateTime.UtcNow.AddDays(-7)), ct); + AssertJudged("MCP", mcp.ToDictionary(r => r.CollectorName, r => (r.FrequencyMinutes, r.HealthStatus))); + + /* The viewer's server tab, and the per-server breakdown the Overview cards and status bar count. */ + await using var viewer = new ViewerDataService(cs!); + var tab = await viewer.GetCollectionHealthAsync(ServerId, ct); + AssertJudged("viewer server tab", tab.ToDictionary(r => r.CollectorName, r => (r.EffectiveFrequencyMinutes ?? 0, r.HealthStatus))); + + var byServer = await viewer.GetFleetCollectionHealthByServerAsync(ct); + AssertJudged("viewer fleet breakdown", byServer[ServerId].ToDictionary(r => r.CollectorName, r => (r.EffectiveFrequencyMinutes ?? 0, r.HealthStatus))); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// + /// The service's fleet roll-up bands every server's collectors in one read, so it reads every override row + /// (DarlingDataReader.AllScheduleOverridesSql) and picks each server's out of them. Same seed as the test + /// above: the two collectors with a 720-minute row, the server's own and the fleet-wide one, are HEALTHY, and the + /// one on its shipped five minutes is STALE. A roll-up that read no overrides would band all three STALE and count + /// none healthy; one that dropped either level would count one. + /// + [Fact] + public async Task TheFleetRollUpBandsEachCollectorByTheIntervalItIsScheduledAt() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to run the live collection-health schedule-override test."); + var ct = TestContext.Current.CancellationToken; + + await using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + + await using var postgres = NpgsqlDataSource.Create(cs!); + var bodySucceeded = false; + try + { + await SeedAsync(connection, ct); + + var counts = await ReadFleetRollUpCountsAsync(postgres, ct); + + Assert.Equal(3, counts.Total); + Assert.Equal(2, counts.Healthy); + Assert.Equal(0, counts.Failing); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// The product's fleet roll-up read, through its own method (ReadFailingCollectorCountsAsync is private; + /// reached by reflection so the test takes no seam), for the one server this test seeds. + private static async Task ReadFleetRollUpCountsAsync(NpgsqlDataSource postgres, CancellationToken ct) + { + var method = typeof(DarlingFleetReader).GetMethod("ReadFailingCollectorCountsAsync", BindingFlags.NonPublic | BindingFlags.Static) + ?? throw new InvalidOperationException("ReadFailingCollectorCountsAsync not found"); + var counts = await (Task>)method.Invoke(null, [postgres, DateTime.UtcNow, ct])!; + return counts[ServerId]; + } + + /// The server, its three collectors' runs and the two levels of schedule override both tests read. + private static async Task SeedAsync(NpgsqlConnection connection, CancellationToken ct) + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO config_monitored_servers (server_id, name, host, is_enabled) VALUES ($1, $2, $2, TRUE) +ON CONFLICT (server_id) DO UPDATE SET is_enabled = TRUE", ServerId, ServerName); + + var now = DateTime.UtcNow; + foreach (var collector in new[] { ServerRowWins, FleetRowApplies, ShippedDefault }) + { + for (var i = 0; i < 10; i++) + { + await InsertSuccessAsync(connection, collector, now.AddHours(-6).AddMinutes(-5 * i), ct); + } + } + + /* The server's own row beats the fleet's; the fleet's row alone applies; the third has neither. */ + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO config_collector_schedules (server_id, collector_name, frequency_minutes, retention_days, enabled) +VALUES ($1, $2, 720, NULL, TRUE)", ServerId, ServerRowWins); + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO config_collector_schedules (server_id, collector_name, frequency_minutes, retention_days, enabled) +VALUES (NULL, $1, 5, NULL, TRUE)", ServerRowWins); + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO config_collector_schedules (server_id, collector_name, frequency_minutes, retention_days, enabled) +VALUES (NULL, $1, 720, NULL, TRUE)", FleetRowApplies); + + Assert.Equal(5, CollectorScheduleDefaults.All[ServerRowWins].FrequencyMinutes); + Assert.Equal(5, CollectorScheduleDefaults.All[FleetRowApplies].FrequencyMinutes); + Assert.Equal(5, CollectorScheduleDefaults.All[ShippedDefault].FrequencyMinutes); + } + + private static void AssertJudged(string surface, Dictionary rows) + { + Assert.True(rows.TryGetValue(ServerRowWins, out var server), $"{surface}: {ServerRowWins} is missing"); + Assert.Equal((720, CollectorHealthClassifier.Healthy), server); + + Assert.True(rows.TryGetValue(FleetRowApplies, out var fleet), $"{surface}: {FleetRowApplies} is missing"); + Assert.Equal((720, CollectorHealthClassifier.Healthy), fleet); + + Assert.True(rows.TryGetValue(ShippedDefault, out var shipped), $"{surface}: {ShippedDefault} is missing"); + Assert.Equal((5, CollectorHealthClassifier.Stale), shipped); + } + + private static async Task InsertSuccessAsync(NpgsqlConnection connection, string collectorName, DateTime collectionTime, CancellationToken ct) + { + await using var command = new NpgsqlCommand(@" +INSERT INTO collection_log (log_id, collection_time, server_id, server_name, collector_name, status, duration_ms, rows_collected, error_message) +VALUES ($1,$2,$3,$4,$5,'SUCCESS',20,10,NULL)", connection); + command.Parameters.AddWithValue(CollectionIdGenerator.Next()); + command.Parameters.AddWithValue(DateTime.SpecifyKind(collectionTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(ServerName); + command.Parameters.AddWithValue(collectorName); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task DeleteRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + await using var cleanup = new NpgsqlCommand( + $"DELETE FROM collection_log WHERE server_id = {ServerId}; " + + $"DELETE FROM config_collector_schedules WHERE collector_name IN ('{ServerRowWins}', '{FleetRowApplies}', '{ShippedDefault}') AND (server_id IS NULL OR server_id = {ServerId}); " + + $"DELETE FROM config_monitored_servers WHERE server_id = {ServerId}; " + + $"DELETE FROM servers WHERE server_id = {ServerId};", + connection); + await cleanup.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/CollectionHistoryAuditTests.cs b/Darling/Darling.Tests/CollectionHistoryAuditTests.cs new file mode 100644 index 0000000000..077e0b5754 --- /dev/null +++ b/Darling/Darling.Tests/CollectionHistoryAuditTests.cs @@ -0,0 +1,1401 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Alerting; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Notifications; +using Xunit; +using Rollup = PerformanceMonitor.Darling.Service.CollectionHistoryAudit.Rollup; +using HourBucket = PerformanceMonitor.Darling.Service.CollectionHistoryAudit.HourBucket; + +namespace Darling.Tests; + +/// +/// #5450 proposal 3: the daily retained-history audit. Pure rules (hour-of-day medians, the half test on each +/// count, range merging, the three-day minimum), the evaluator's once-a-day and one-alert rules, and live reads +/// of a seeded rollup. +/// +/* #1776 own-store: deliberately NOT [Collection("live-postgres")]. The live tests reach DARLING_TEST_PG only to + CREATE and DROP their own database through ScratchPostgres and work entirely inside it. */ +public sealed class CollectionHistoryAuditTests +{ + private static readonly CancellationToken Ct = TestContext.Current.CancellationToken; + private static readonly DateTime AuditDay = new(2026, 10, 10, 0, 0, 0, DateTimeKind.Utc); + + /// A workload rollup: judged on servers only. + private static readonly Rollup QueryStats = new("collect.query_stats_interval_hourly", false); + + /// The perfmon baseline: judged on servers and collection passes. + private static readonly Rollup Perfmon = new("collect.perfmon_interval_baseline", true); + + /// The normal load curve: servers flat, passes climbing with the hour, so a pass test that + /// ignored the hour of day would be wrong somewhere. + private static (int Servers, long Passes) Usual(int hour) => (40, 1000 + (hour * 100)); + + private static IEnumerable Day( + DateTime day, Func shape) + { + for (var hour = 0; hour < 24; hour++) + { + if (shape(hour) is { } v) + { + yield return new HourBucket(day.AddHours(hour), v.Servers, v.Passes); + } + } + } + + private static List Week(int priorDays = 7) => + Enumerable.Range(1, priorDays) + .SelectMany(d => Day(AuditDay.AddDays(-d), h => Usual(h))) + .ToList(); + + private static List WithAuditDay(List week, Func shape) + { + week.AddRange(Day(AuditDay, shape)); + return week; + } + + [Fact] + public void ANormalDay_FlagsNothing() + { + var buckets = WithAuditDay(Week(), h => Usual(h)); + var ranges = CollectionHistoryAudit.Analyze(AuditDay, buckets); + Assert.NotNull(ranges); + Assert.Empty(ranges!); + } + + [Fact] + public void AThreeHourHole_IsOneRange_WithTheCountsAgainstUsual() + { + var buckets = WithAuditDay(Week(), h => h is >= 18 and <= 20 ? null : Usual(h)); + var range = Assert.Single(CollectionHistoryAudit.Analyze(AuditDay, buckets)!); + Assert.Equal(18, range.StartHour); + Assert.Equal(21, range.EndHour); + Assert.Equal("18:00-21:00Z", CollectionHistoryAudit.RangeLabel(range)); + Assert.Equal(0, range.LowestServers); + Assert.Equal(40, range.UsualServers); + Assert.Equal(0, range.LowestPasses); + } + + [Fact] + public void PassesAtTwentyPercent_WithEveryServerPresent_IsFlagged() + { + /* Case 2 of the issue: rows arrived every hour from every server and only the volume fell. */ + var buckets = WithAuditDay(Week(), h => h is >= 5 and <= 7 ? (40, Usual(h).Passes / 5) : Usual(h)); + var range = Assert.Single(CollectionHistoryAudit.Analyze(AuditDay, buckets)!); + Assert.Equal("05:00-08:00Z", CollectionHistoryAudit.RangeLabel(range)); + Assert.Equal(40, range.LowestServers); + Assert.Equal(300, range.LowestPasses); + Assert.Equal(1500, range.UsualPasses); + } + + [Fact] + public void AWorkloadSourceCarriesNoPasses_SoAQuietWeekendWithEveryServerPresentIsClean() + { + /* H2 of the #5461 review: a workload rollup is read for servers only (passes are 0 in every bucket), so a + weekend with a fraction of the queries but every server present flags nothing. */ + var week = Enumerable.Range(1, 7).SelectMany(d => Day(AuditDay.AddDays(-d), h => (40, 0L))).ToList(); + var buckets = WithAuditDay(week, h => (40, 0L)); + Assert.Empty(CollectionHistoryAudit.Analyze(AuditDay, buckets)!); + + /* ... and the hole in servers is still found. */ + var holey = WithAuditDay(Enumerable.Range(1, 7).SelectMany(d => Day(AuditDay.AddDays(-d), h => (40, 0L))).ToList(), + h => h == 9 ? (10, 0L) : (40, 0L)); + Assert.Equal("09:00-10:00Z", CollectionHistoryAudit.RangeLabel(Assert.Single(CollectionHistoryAudit.Analyze(AuditDay, holey)!))); + } + + [Theory] + [InlineData(19, false)] // 19 of 40 servers: under half + [InlineData(20, true)] // exactly half is not under half + public void TheHalfTest_IsStrict_OnServersAlone(int servers, bool clean) + { + var buckets = WithAuditDay(Week(), h => h == 5 ? (servers, Usual(5).Passes) : Usual(h)); + Assert.Equal(clean, CollectionHistoryAudit.Analyze(AuditDay, buckets)!.Count == 0); + } + + [Theory] + [InlineData(749, false)] // passes 749 of 1500: under half + [InlineData(750, true)] // exactly half + public void TheHalfTest_IsStrict_OnPassesAlone(long passes, bool clean) + { + var buckets = WithAuditDay(Week(), h => h == 5 ? (40, passes) : Usual(h)); + Assert.Equal(clean, CollectionHistoryAudit.Analyze(AuditDay, buckets)!.Count == 0); + } + + [Fact] + public void AnHourWhoseUsualIsUnderOne_NeverFlags() + { + /* L3 of the #5461 review: four prior days of [0, 0, 1, 1] servers is a median of 0.5, and an actual of 0 is under + half of it, but half a server is not a usual worth judging. Usual 1 (two of the four days at 1, the rest + above) is. */ + var prior = new[] { 0, 0, 1, 1 }; + var buckets = new List(); + for (var d = 0; d < 4; d++) + { + buckets.AddRange(Day(AuditDay.AddDays(-(d + 1)), h => (h == 7 ? prior[d] : 5, 0L))); + } + + buckets.AddRange(Day(AuditDay, h => h == 7 ? (0, 0L) : (5, 0L))); + Assert.Empty(CollectionHistoryAudit.Analyze(AuditDay, buckets)!); + + /* Usual exactly 1 with an actual of 0 still flags. */ + var one = new List(); + for (var d = 1; d <= 4; d++) + { + one.AddRange(Day(AuditDay.AddDays(-d), h => (h == 7 ? 1 : 5, 0L))); + } + + one.AddRange(Day(AuditDay, h => h == 7 ? (0, 0L) : (5, 0L))); + Assert.Equal("07:00-08:00Z", CollectionHistoryAudit.RangeLabel(Assert.Single(CollectionHistoryAudit.Analyze(AuditDay, one)!))); + } + + [Fact] + public void UsualIsTheMedianForThatHour_SoOneBadPriorDayDoesNotMoveIt() + { + var week = Week(); + /* One prior day with a hole at hour 12: the median over seven days is still the normal value. */ + week.RemoveAll(b => b.HourUtc == AuditDay.AddDays(-3).AddHours(12)); + var buckets = WithAuditDay(week, h => Usual(h)); + Assert.Empty(CollectionHistoryAudit.Analyze(AuditDay, buckets)!); + Assert.Equal(2.0, CollectionHistoryAudit.Median(new long[] { 1, 2, 3 })); + Assert.Equal(2.5, CollectionHistoryAudit.Median(new long[] { 4, 1, 3, 2 })); + Assert.Equal(0, CollectionHistoryAudit.Median(Array.Empty())); + } + + [Fact] + public void FewerThanThreePriorDaysWithData_IsSkipped_NotAudited() + { + var buckets = WithAuditDay(Week(priorDays: 2), h => null); + Assert.Null(CollectionHistoryAudit.Analyze(AuditDay, buckets)); + + /* Three is enough, and the audited day's own hole is then found. */ + var three = WithAuditDay(Week(priorDays: 3), h => h == 9 ? null : Usual(h)); + var range = Assert.Single(CollectionHistoryAudit.Analyze(AuditDay, three)!); + Assert.Equal(9, range.StartHour); + Assert.Equal(10, range.EndHour); + } + + [Fact] + public void APriorDayWithNoRowsAtAll_IsNotADayOfUsual() + { + /* Prior days 1-3 empty (a store that was not building the source yet), days 4-7 normal: usual is the + normal value, not zero, so the hole on the audited day is still found. */ + var buckets = Enumerable.Range(4, 4).SelectMany(d => Day(AuditDay.AddDays(-d), h => Usual(h))).ToList(); + buckets.AddRange(Day(AuditDay, h => h == 3 ? null : Usual(h))); + var range = Assert.Single(CollectionHistoryAudit.Analyze(AuditDay, buckets)!); + Assert.Equal(3, range.StartHour); + } + + [Fact] + public void SeparateRuns_AreSeparateRanges_AndTheLastHourIsAlwaysJudged() + { + var buckets = WithAuditDay(Week(), h => h is 2 or 3 or 10 or 23 ? null : Usual(h)); + var ranges = CollectionHistoryAudit.Analyze(AuditDay, buckets)!; + Assert.Equal(new[] { "02:00-04:00Z", "10:00-11:00Z", "23:00-24:00Z" }, ranges.Select(CollectionHistoryAudit.RangeLabel)); + } + + [Fact] + public void ASourceIsSettled_OnlyWhenItsNewestBucketReachesTheDaysLastHour() + { + var through22 = WithAuditDay(Week(), h => h <= 22 ? Usual(h) : null); + Assert.False(CollectionHistoryAudit.IsSettled(AuditDay, through22)); + + /* The 23:00 bucket, or any later one (the read runs an hour into the next day), settles it. */ + Assert.True(CollectionHistoryAudit.IsSettled(AuditDay, WithAuditDay(Week(), h => Usual(h)))); + var holeAt23 = WithAuditDay(Week(), h => h == 23 ? null : Usual(h)); + holeAt23.Add(new HourBucket(AuditDay.AddDays(1), 40, 1000)); + Assert.True(CollectionHistoryAudit.IsSettled(AuditDay, holeAt23)); + Assert.False(CollectionHistoryAudit.IsSettled(AuditDay, new List())); + + Assert.Equal(AuditDay.AddDays(1).AddHours(1), CollectionHistoryAudit.ReadTo(AuditDay)); + Assert.Equal(TimeSpan.FromHours(3), CollectionHistoryAudit.DueTimeOfDay); + } + + [Fact] + public void TheReadRunsUpToTheCurrentHour_NeverEarlierThanOneHourPastTheAuditedDay() + { + /* H1 of the #5461 round 2 review: the read ends at the current hour, so a bucket after a hole that runs past + 01:00Z is in it. A bucket at 01:00Z or 02:00Z settles a source whose 23:00 bucket is missing. */ + var tick = new DateTime(2026, 10, 11, 3, 5, 0, DateTimeKind.Utc); + Assert.Equal(new DateTime(2026, 10, 11, 3, 0, 0, DateTimeKind.Utc), CollectionHistoryAudit.ReadTo(AuditDay, tick)); + Assert.Equal(DateTimeKind.Utc, CollectionHistoryAudit.ReadTo(AuditDay, tick).Kind); + Assert.Equal(AuditDay.AddDays(2).AddHours(3), CollectionHistoryAudit.ReadTo(AuditDay, AuditDay.AddDays(2).AddHours(3).AddMinutes(59))); + + /* Never earlier than the day's own end plus one hour, whatever the clock says. */ + Assert.Equal(CollectionHistoryAudit.ReadTo(AuditDay), CollectionHistoryAudit.ReadTo(AuditDay, AuditDay.AddHours(5))); + + var holeThroughMidnight = WithAuditDay(Week(), h => h == 23 ? null : Usual(h)); + holeThroughMidnight.Add(new HourBucket(AuditDay.AddDays(1).AddHours(2), 40, Usual(2).Passes)); + Assert.True(CollectionHistoryAudit.IsSettled(AuditDay, holeThroughMidnight)); + } + + [Fact] + public void TheFlaggedHours_AreTheDistinctHoursOfTheDay_NotHoursTimesSources() + { + /* L2 of the #5461 round 2 review: one three-hour outage that thins every source is three hours. */ + var shared = new[] { new CollectionHistoryAudit.FlaggedRange(18, 21, 0, 40, 0, 0) }; + var overlapping = new[] { new CollectionHistoryAudit.FlaggedRange(20, 22, 0, 40, 0, 0) }; + var sources = Enumerable.Range(0, 7).Select(i => new Rollup("collect.s" + i, false)).ToArray(); + + Assert.Equal(3, CollectionHistoryAudit.FlaggedHours(sources.Select(r => (r, (IReadOnlyList)shared)))); + Assert.Equal(4, CollectionHistoryAudit.FlaggedHours(new (Rollup, IReadOnlyList)[] + { + (sources[0], shared), (sources[1], overlapping), + })); + } + + [Fact] + public void TheSql_HasARangePredicateOnBucket_AndCountsOnlyServersEnabledNow() + { + var servers = CollectionHistoryAudit.BuildSql(QueryStats); + Assert.Contains("FROM collect.query_stats_interval_hourly AS r", servers, StringComparison.Ordinal); + Assert.Contains("WHERE r.bucket >= $1", servers, StringComparison.Ordinal); + Assert.Contains("r.bucket < $2", servers, StringComparison.Ordinal); + Assert.Contains("COUNT(DISTINCT r.server_id)", servers, StringComparison.Ordinal); + Assert.Contains("0::bigint", servers, StringComparison.Ordinal); + Assert.DoesNotContain("sample_count", servers, StringComparison.Ordinal); + + /* M1 of the #5461 review: servers enabled now only, on every day the read covers. */ + Assert.Contains("JOIN config.config_monitored_servers AS c", servers, StringComparison.Ordinal); + Assert.Contains("AND c.is_enabled", servers, StringComparison.Ordinal); + + var passes = CollectionHistoryAudit.BuildSql(Perfmon); + Assert.Contains("FROM collect.perfmon_interval_baseline AS r", passes, StringComparison.Ordinal); + Assert.Contains("COUNT(*)::bigint", passes, StringComparison.Ordinal); + Assert.Contains("AND c.is_enabled", passes, StringComparison.Ordinal); + + Assert.Throws(() => CollectionHistoryAudit.BuildSql(new Rollup("x; DROP TABLE y", true))); + Assert.Throws(() => CollectionHistoryAudit.BuildSql(new Rollup("x; DROP TABLE y", false, CollectionHistoryAudit.ReadShape.Probe))); + } + + /// #5461: the probe statement keeps the three things the big-store timing depends on, and drops the + /// hours nobody has a row in. + [Fact] + public void TheProbeSql_IsAPerServerPerHourLateralWithALimit_OnATimestampSeries_AndDropsEmptyHours() + { + var probe = CollectionHistoryAudit.BuildSql(new Rollup("collect.query_store_stats_hourly", false, CollectionHistoryAudit.ReadShape.Probe)); + Assert.DoesNotContain("COUNT(DISTINCT", probe, StringComparison.Ordinal); + Assert.Contains("generate_series($1::timestamp, $2::timestamp, INTERVAL '1 hour')", probe, StringComparison.Ordinal); + Assert.Contains("CROSS JOIN config.config_monitored_servers AS c", probe, StringComparison.Ordinal); + Assert.Contains("LEFT JOIN LATERAL", probe, StringComparison.Ordinal); + Assert.Contains("FROM collect.query_store_stats_hourly AS r", probe, StringComparison.Ordinal); + Assert.Contains("r.server_id = c.server_id", probe, StringComparison.Ordinal); + Assert.Contains("r.bucket = h.bucket", probe, StringComparison.Ordinal); + Assert.Contains("WHERE c.is_enabled", probe, StringComparison.Ordinal); + Assert.Contains("h.bucket < $2::timestamp", probe, StringComparison.Ordinal); + + /* LIMIT 1 inside the lateral keeps the planner from flattening it to a semi join that reads the whole range. */ + Assert.Matches(@"(?s)LATERAL \(.*LIMIT 1\s*\) AS p ON true", probe); + + /* An hour with no server is ABSENT, as in the scan: IsSettled counts any bucket, so a zero row would mark a + hole as materialized. */ + Assert.Contains("HAVING COUNT(p.one) > 0", probe, StringComparison.Ordinal); + + /* The scan shape is unchanged for a Scan source. */ + var scan = CollectionHistoryAudit.BuildSql(new Rollup("collect.query_stats_interval_hourly", false)); + Assert.DoesNotContain("LATERAL", scan, StringComparison.Ordinal); + } + + /// #5461: the interval hourly is not audited directly, and the reason is pinned. Each assertion's + /// message names the reason, so a change to one of these CREATE statements says why the audit needs a look. + [Fact] + public void TheIntervalHourlyIsNotAudited_BecauseItsChildAndSiblingReadTheSameRawTableWithNoFilter() + { + var corrected = TimescaleSupport.CreateQueryStoreStatsCorrectedHourlySql; + var hourly = TimescaleSupport.CreateQueryStoreStatsHourlySql; + var interval = TimescaleSupport.CreateQueryStoreStatsIntervalHourlySql; + const string Why = " (#5461: the history audit does not read query_store_stats_interval_hourly because its same-width child" + + " query_store_stats_corrected_hourly has no filter on it, and query_store_stats_hourly reads the same raw table with no filter," + + " so a server missing from the interval hourly is missing from both audited views. Re-decide the audit's coverage.)"; + + Assert.True(corrected.Contains("FROM collect.query_store_stats_interval_hourly", StringComparison.Ordinal), + "the corrected hourly no longer reads FROM collect.query_store_stats_interval_hourly" + Why); + Assert.True(corrected.Contains("time_bucket('1 hour', bucket)", StringComparison.Ordinal), + "the corrected hourly is no longer a time_bucket('1 hour', bucket) over the interval hourly" + Why); + Assert.False(corrected.Contains("WHERE", StringComparison.OrdinalIgnoreCase), + "the corrected hourly now has a WHERE on the interval hourly" + Why); + + foreach (var (name, sql) in new[] { ("query_store_stats_hourly", hourly), ("query_store_stats_interval_hourly", interval) }) + { + Assert.True(System.Text.RegularExpressions.Regex.IsMatch(sql, @"FROM collect\.query_store_stats\s"), + name + " no longer reads FROM collect.query_store_stats" + Why); + Assert.False(sql.Contains("WHERE", StringComparison.OrdinalIgnoreCase), + name + " now has a WHERE on collect.query_store_stats" + Why); + } + + Assert.Equal(new[] { TimescaleSupport.QueryStoreStatsIntervalHourlyView }, CollectionHistoryAudit.UnauditedHourlyViews.ToArray()); + } + + /// #5461: the two probe sources each keep a (server_id, bucket) index, every hourly aggregate is + /// audited with a shape or is the one documented exclusion, and no other source is probed. + [Fact] + public void EveryProbeSourceKeepsAServerIdBucketIndex_AndEveryHourlyAggregateIsAuditedOrTheOneExclusion() + { + foreach (var rollup in CollectionHistoryAudit.Rollups.Where(r => r.Shape == CollectionHistoryAudit.ReadShape.Probe)) + { + var view = rollup.Relation["collect.".Length..]; + Assert.True(TimescaleSupport.QueryStoreRollupKeptIndexesSql.TryGetValue(view, out var kept), + view + " is probed but TimescaleSupport keeps no indexes for it"); + Assert.Contains("a1.attname = 'server_id'", kept, StringComparison.Ordinal); + Assert.Contains("a2.attname = 'bucket'", kept, StringComparison.Ordinal); + Assert.False(rollup.JudgesPasses, "a probe only learns whether a server has a row in an hour, never passes"); + } + + Assert.Equal( + new[] { "collect.query_store_stats_corrected_hourly", "collect.query_store_stats_hourly" }, + CollectionHistoryAudit.Rollups.Where(r => r.Shape == CollectionHistoryAudit.ReadShape.Probe).Select(r => r.Relation).OrderBy(x => x, StringComparer.Ordinal)); + + var audited = CollectionHistoryAudit.Rollups.Select(r => r.Relation).ToHashSet(StringComparer.Ordinal); + foreach (var (_, view) in TimescaleSupport.HourlyAggregates) + { + var isAudited = audited.Contains("collect." + view); + var isExcluded = CollectionHistoryAudit.UnauditedHourlyViews.Contains(view); + Assert.True(isAudited ^ isExcluded, view + " must be audited with a read shape, or be the one documented exclusion, not both or neither"); + } + } + + [Fact] + public void TheAuditedSources_AreTheLiveHourlyAggregatesForServers_AndThePerfmonBaselineForPasses() + { + var names = CollectionHistoryAudit.Rollups.Select(r => r.Relation).ToArray(); + Assert.Equal( + TimescaleSupport.HourlyAggregates.Where(a => a.View != TimescaleSupport.QueryStoreStatsIntervalHourlyView) + .Select(a => "collect." + a.View) + .Append("collect." + TimescaleSupport.PerfmonIntervalBaselineView), + names); + Assert.DoesNotContain("collect." + TimescaleSupport.QueryStoreStatsIntervalHourlyView, names); + Assert.DoesNotContain("collect." + TimescaleSupport.QueryStatsHourlyView, names); + Assert.DoesNotContain("collect." + TimescaleSupport.ProcedureStatsHourlyView, names); + + /* Only the perfmon baseline is judged on passes; the six workload rollups never are (H2). */ + Assert.Equal(new[] { "collect." + TimescaleSupport.PerfmonIntervalBaselineView }, + CollectionHistoryAudit.Rollups.Where(r => r.JudgesPasses).Select(r => r.Relation)); + + /* Each one the audit reads has the columns the statement names. */ + foreach (var (createSql, view) in TimescaleSupport.HourlyAggregates) + { + Assert.Contains("AS bucket", createSql, StringComparison.Ordinal); + Assert.Contains("server_id", createSql, StringComparison.Ordinal); + if (view != TimescaleSupport.QueryStoreStatsIntervalHourlyView) + { + Assert.Contains("collect." + view, names); + } + } + + Assert.Contains("AS bucket", TimescaleSupport.CreatePerfmonIntervalBaselineSql, StringComparison.Ordinal); + Assert.Contains("collection_time", TimescaleSupport.CreatePerfmonIntervalBaselineSql, StringComparison.Ordinal); + } + + [Fact] + public void SourcesWithIdenticalFlaggedRanges_ShareOneLine_AndUnreadSourcesAreNamed() + { + var shared = new[] { new CollectionHistoryAudit.FlaggedRange(18, 21, 0, 40, 0, 0) }; + var other = new[] { new CollectionHistoryAudit.FlaggedRange(4, 5, 10, 40, 0, 0) }; + var a = new Rollup("collect.query_store_stats_hourly", false); + var b = new Rollup("collect.query_store_stats_corrected_hourly", false); + var c = new Rollup("collect.query_stats_interval_hourly", false); + var text = CollectionHistoryAudit.Render(AuditDay, new (Rollup, IReadOnlyList)[] + { + (a, shared), (c, other), (b, shared), + }, new[] { new Rollup("collect.procedure_stats_interval_hourly", false) }); + + Assert.Contains("query_store_stats_hourly, query_store_stats_corrected_hourly: 18:00-21:00Z (servers as low as 0 vs usual 40)", text, StringComparison.Ordinal); + Assert.Contains("query_stats_interval_hourly: 04:00-05:00Z (servers as low as 10 vs usual 40)", text, StringComparison.Ordinal); + Assert.Equal(1, text.Split("18:00-21:00Z").Length - 1); + Assert.Contains("Not read for this day: procedure_stats_interval_hourly", text, StringComparison.Ordinal); + Assert.DoesNotContain("Not read", CollectionHistoryAudit.Render(AuditDay, new (Rollup, IReadOnlyList)[] { (a, shared) }), StringComparison.Ordinal); + } + + [Fact] + public void TheMetric_IsSelfMonitorFamily_Warning_AndNumeric() + { + Assert.Equal("Collection Gaps In History", DarlingSelfAlertEvaluator.CollectionGapsInHistoryMetric); + Assert.Equal(AlertFamily.SelfMonitor, AlertFamily.Of(DarlingSelfAlertEvaluator.CollectionGapsInHistoryMetric)); + Assert.Equal("WARNING", AlertSeverity.ForMetric(DarlingSelfAlertEvaluator.CollectionGapsInHistoryMetric).BadgeText); + Assert.False(PerformanceMonitor.Common.AlertMetricClassifier.IsStateOnly(DarlingSelfAlertEvaluator.CollectionGapsInHistoryMetric)); + + /* The stamp key is its own row, distinct from the three daily documents'. */ + var others = new[] + { + PgSelfAlertDeliveryStampStore.CostDigestStateKey, PgSelfAlertDeliveryStampStore.FleetSweepRollupStateKey, + PgSelfAlertDeliveryStampStore.AnalysisSinglesDigestStateKey, + }; + Assert.DoesNotContain(PgSelfAlertDeliveryStampStore.HistoryAuditStateKey, others); + } + + /* ---------------- the evaluator: once a day, one alert, failure isolation ---------------- */ + + private sealed class MemoryStampStore : ISelfAlertDeliveryStampStore + { + public Dictionary Stamps { get; } = new(StringComparer.Ordinal); + + public Task GetDeliveredAtUtcAsync(string stateKey, CancellationToken cancellationToken) => + Task.FromResult(Stamps.TryGetValue(stateKey, out var at) ? at : (DateTime?)null); + + public Task RecordDeliveredAtUtcAsync(string stateKey, DateTime deliveredAtUtc, CancellationToken cancellationToken) + { + Stamps[stateKey] = deliveredAtUtc; + return Task.CompletedTask; + } + } + + private sealed class NoHistoryStore : IAlertHistoryStore + { + public Task RecordAlertAsync(AlertHistoryRecord record) => Task.CompletedTask; + + public Task GetLastEmailSentUtcAsync(string serverId, string metricName, string? dedupKey = null) => + Task.FromResult(null); + + public Task GetLastWebhookSentUtcAsync(string serverId, string metricName, string? dedupKey = null) => + Task.FromResult(null); + + public Task GetLastAlertTimeAsync(string serverId, string metricName, string? dedupKey = null) => + Task.FromResult(null); + + public Task GetLastDeliveredPageUtcAsync(string serverId, string metricName) => Task.FromResult(null); + } + + /// One "process": its own evaluator, deliverer and clock over a stamp store the test shares. + private sealed class Process + { + public DarlingSelfAlertTests.RecordingDeliverer Deliverer { get; } = new(); + public DarlingSelfAlertTests.FakeSettings Settings { get; } = new(); + public DateTime Now { get; set; } + public AlertReadFailureCounter ReadFailures { get; } = new(); + public DarlingSelfAlertTests.CapturingLogger Log { get; } = new(); + public DarlingSelfAlertEvaluator Evaluator { get; } + + public Process(MemoryStampStore stamps, DateTime now) + { + Now = now; + Evaluator = new DarlingSelfAlertEvaluator( + Settings, Deliverer, new NoHistoryStore(), _ => false, + logger: Log, utcNow: () => Now, readFailures: ReadFailures, deliveryStamps: stamps); + } + + public int Reads { get; private set; } + + public Task RunAsync(IReadOnlyList queryStats, IReadOnlyList? perfmon = null) => + Evaluator.ApplyCollectionHistoryAuditAsync( + (rollup, from, to, token) => + { + Reads++; + Assert.Equal(AuditDay.AddDays(-7), from); + /* The read runs up to the current hour (every tick of these tests is on the day after the audited day + or later), so a bucket after a hole across midnight is in it. */ + Assert.Equal(Now.Date.AddHours(Now.Hour), to); + return Task.FromResult?>( + rollup.Relation == QueryStats.Relation ? queryStats : perfmon); + }, + new[] { QueryStats, Perfmon }, Ct); + + /// A run whose reader is the test's own: per-relation behaviour, and a count of reads of each. + public Task RunWithAsync(Func?> reader, Dictionary? reads = null) => + Evaluator.ApplyCollectionHistoryAuditAsync( + (rollup, from, to, token) => + { + if (reads is not null) + { + reads[rollup.Relation] = (reads.TryGetValue(rollup.Relation, out var seen) ? seen : 0) + 1; + } + + return Task.FromResult(reader(rollup, from)); + }, + new[] { QueryStats, Perfmon }, Ct); + } + + private static readonly DateTime FirstPass = new(2026, 10, 11, 3, 5, 0, DateTimeKind.Utc); + + [Fact] + public async Task OneAlertPerRun_NamingTheDayAndEachFlaggedRollup() + { + var p = new Process(new MemoryStampStore(), FirstPass.AddHours(1)); + var holey = WithAuditDay(Week(), h => h is >= 18 and <= 20 ? null : Usual(h)); + var thin = WithAuditDay(Week(), h => h is >= 5 and <= 7 ? (40, Usual(h).Passes / 5) : Usual(h)); + + await p.RunAsync(holey, thin); + + var fired = Assert.Single(p.Deliverer.Outcomes); + Assert.Equal("Collection Gaps In History", fired.MetricName); + Assert.Equal(AlertSeverityLevel.Warning, fired.Severity); + Assert.Equal("collectionhistoryaudit:2026-10-10", fired.ServerKey); + Assert.Contains("2026-10-10", fired.DetailText, StringComparison.Ordinal); + Assert.Contains("query_stats_interval_hourly: 18:00-21:00Z (servers as low as 0 vs usual 40)", fired.DetailText, StringComparison.Ordinal); + Assert.Contains("perfmon_interval_baseline: 05:00-08:00Z (servers as low as 40 vs usual 40, collection passes as low as 300 vs usual 1,500)", fired.DetailText, StringComparison.Ordinal); + Assert.DoesNotContain("Not read", fired.DetailText, StringComparison.Ordinal); + Assert.Equal(6, fired.NumericCurrentValue); + } + + [Fact] + public async Task ANormalDay_RaisesNothing_AndStillSpendsTheDay() + { + var stamps = new MemoryStampStore(); + var p = new Process(stamps, FirstPass.AddHours(1)); + var normal = WithAuditDay(Week(), h => Usual(h)); + + await p.RunAsync(normal, normal); + Assert.Empty(p.Deliverer.Outcomes); + Assert.Equal(2, p.Reads); + + /* An hour later the day is spent: no second read of either source. */ + p.Now = p.Now.AddHours(1); + await p.RunAsync(normal, normal); + Assert.Equal(2, p.Reads); + } + + [Fact] + public async Task ARestartOnTheSameDay_DoesNotRaiseItTwice_AndTheNextDayDoes() + { + var stamps = new MemoryStampStore(); + var holey = WithAuditDay(Week(), h => h is >= 18 and <= 20 ? null : Usual(h)); + + var first = new Process(stamps, FirstPass); + await first.RunAsync(holey, holey); + Assert.Single(first.Deliverer.Outcomes); + Assert.Equal(new DateTime(2026, 10, 11, 3, 0, 0, DateTimeKind.Utc), stamps.Stamps["history_audit_slot"]); + + /* A fresh process over the same store, hours later the same UTC day. */ + var second = new Process(stamps, FirstPass.AddHours(10)); + await second.RunAsync(holey, holey); + Assert.Empty(second.Deliverer.Outcomes); + Assert.Equal(0, second.Reads); + + /* The next UTC day, at 03:00Z or after, is due again (and audits the day before it). */ + var third = new Process(stamps, new DateTime(2026, 10, 12, 3, 0, 0, DateTimeKind.Utc)); + Assert.True(await DueAsync(third, holey)); + } + + private static async Task DueAsync(Process p, List buckets) + { + var read = false; + await p.Evaluator.ApplyCollectionHistoryAuditAsync( + (rollup, from, to, token) => + { + read = true; + return Task.FromResult?>(buckets); + }, + new[] { QueryStats }, Ct); + return read; + } + + [Fact] + public async Task TheSlotIsThreeOClock_NothingIsReadBeforeIt_OrWithAlertsOff() + { + /* H1 of the #5461 review: the audit slot is 03:00Z, after the heaviest rollup (01:15) and the corrected + rollup (02:0X) have both materialized the audited day's last hour. */ + var early = new Process(new MemoryStampStore(), new DateTime(2026, 10, 11, 2, 59, 0, DateTimeKind.Utc)); + Assert.False(await DueAsync(early, Week())); + early.Now = new DateTime(2026, 10, 11, 3, 0, 0, DateTimeKind.Utc); + Assert.True(await DueAsync(early, Week())); + + var off = new Process(new MemoryStampStore(), FirstPass); + off.Settings.AlertsEnabled = false; + Assert.False(await DueAsync(off, Week())); + } + + [Theory] + [InlineData(3, 0)] + [InlineData(3, 59)] + [InlineData(14, 20)] + [InlineData(23, 59)] + public async Task AHoleInTheLastHour_IsJudged_AtEveryTick_OnceTheBucketAfterItExists(int hour, int minute) + { + /* L1 of the #5461 review: a hole at 23:00 with a bucket after it, whenever the tick falls from 03:00Z on. The + 01:30Z rule it replaced did judge hour 23 at every one of these ticks, so this guards the 03:00Z slot and + the always-judged hour together, not that rule. The midnight test below is the one that was red before. */ + var p = new Process(new MemoryStampStore(), new DateTime(2026, 10, 11, hour, minute, 0, DateTimeKind.Utc)); + var holeAt23 = WithAuditDay(Week(), h => h == 23 ? null : Usual(h)); + holeAt23.Add(new HourBucket(AuditDay.AddDays(1), 40, Usual(0).Passes)); + + await p.RunAsync(holeAt23, holeAt23); + + var fired = Assert.Single(p.Deliverer.Outcomes); + Assert.Contains("query_stats_interval_hourly: 23:00-24:00Z", fired.DetailText, StringComparison.Ordinal); + Assert.Contains("23:00-24:00Z", fired.DetailText, StringComparison.Ordinal); + } + + [Fact] + public async Task ASourceThatIsNotReady_IsRetriedOnLaterTicks_AndNothingIsRaisedOrStampedUntilItIs() + { + var stamps = new MemoryStampStore(); + var p = new Process(stamps, FirstPass); + var through22 = WithAuditDay(Week(), h => h <= 22 ? Usual(h) : null); + var settled = WithAuditDay(Week(), h => h is >= 18 and <= 20 ? null : Usual(h)); + var reads = new Dictionary(); + + /* Perfmon is ready and holey; the workload rollup has not materialized hour 23 yet. */ + await p.RunWithAsync((r, from) => r.Relation == QueryStats.Relation ? through22 : settled, reads); + Assert.Empty(p.Deliverer.Outcomes); + Assert.Empty(stamps.Stamps); + + /* A later tick re-reads ONLY the unready source; it is ready now, so the one alert goes out, naming no + unread source, and the day is stamped. */ + p.Now = FirstPass.AddHours(1); + await p.RunWithAsync((r, from) => settled, reads); + Assert.Equal(2, reads[QueryStats.Relation]); + Assert.Equal(1, reads[Perfmon.Relation]); + var fired = Assert.Single(p.Deliverer.Outcomes); + Assert.DoesNotContain("Not read", fired.DetailText, StringComparison.Ordinal); + Assert.Contains("perfmon_interval_baseline: 18:00-21:00Z", fired.DetailText, StringComparison.Ordinal); + Assert.Equal(new DateTime(2026, 10, 11, 3, 0, 0, DateTimeKind.Utc), stamps.Stamps["history_audit_slot"]); + } + + [Fact] + public async Task ASourceThatNeverBecomesReadable_IsGivenUpOnAtTheNextSlot_WithOneWarning_AndTheAlertNamesIt() + { + var stamps = new MemoryStampStore(); + var p = new Process(stamps, FirstPass); + var holey = WithAuditDay(Week(), h => h is >= 18 and <= 20 ? null : Usual(h)); + var day10 = AuditDay.AddDays(-7); + + /* The workload rollup times out on every tick; perfmon reads. Held, not raised, until the slot after. */ + IReadOnlyList? Reader(Rollup r, DateTime from) + { + if (from != day10 || r.Relation == QueryStats.Relation) + { + throw new TimeoutException("statement timeout"); + } + + return holey; + } + + foreach (var tick in new[] { FirstPass, FirstPass.AddHours(8), FirstPass.AddHours(22), new DateTime(2026, 10, 12, 2, 59, 0, DateTimeKind.Utc) }) + { + p.Now = tick; + await p.RunWithAsync(Reader); + Assert.Empty(p.Deliverer.Outcomes); + Assert.Empty(stamps.Stamps); + } + + /* 03:00Z the next day: give up on the unread source, one warning naming it, and raise what was read. */ + p.Now = new DateTime(2026, 10, 12, 3, 0, 0, DateTimeKind.Utc); + await p.RunWithAsync(Reader); + + var fired = Assert.Single(p.Deliverer.Outcomes); + Assert.Contains("2026-10-10", fired.DetailText, StringComparison.Ordinal); + Assert.Contains("perfmon_interval_baseline: 18:00-21:00Z", fired.DetailText, StringComparison.Ordinal); + Assert.Contains("Not read for this day: query_stats_interval_hourly", fired.DetailText, StringComparison.Ordinal); + Assert.Single(p.Log.Entries, e => e.Message.Contains("gave up on collect.query_stats_interval_hourly", StringComparison.Ordinal)); + } + + /// A run whose reader serves each source from its own list, cut at the bounds the evaluator passes, as the + /// real read is: a bucket outside [from, to) is not seen. + private static Task RunBoundedAsync(Process p, Func?> source, params Rollup[] rollups) => + p.Evaluator.ApplyCollectionHistoryAuditAsync( + (rollup, from, to, token) => + { + var all = source(rollup); + return Task.FromResult?>( + all?.Where(b => b.HourUtc >= from && b.HourUtc < to).ToList()); + }, + rollups.Length == 0 ? new[] { QueryStats, Perfmon } : rollups, Ct); + + [Fact] + public async Task AnOutageFrom2230To0130_IsRaisedOnceCollectionResumes_NotAfterTheGiveUp() + { + /* H1 of the #5461 round 2 review. No row in any source from 23:00Z to 01:00Z; rows again at 01:00Z. With the + read fixed at 01:00Z no bucket after the hole was in it, so every source read "not ready" forever and the + day, hole and any unrelated thin hour included, was never judged. */ + var stamps = new MemoryStampStore(); + var p = new Process(stamps, FirstPass); + var days = WithAuditDay(Week(), h => h == 23 ? null : Usual(h)); + days.Add(new HourBucket(AuditDay.AddDays(1).AddHours(1), 40, Usual(1).Passes)); + days.Add(new HourBucket(AuditDay.AddDays(1).AddHours(2), 40, Usual(2).Passes)); + + await RunBoundedAsync(p, _ => days); + + var fired = Assert.Single(p.Deliverer.Outcomes); + Assert.Contains("23:00-24:00Z", fired.DetailText, StringComparison.Ordinal); + Assert.DoesNotContain("Not read", fired.DetailText, StringComparison.Ordinal); + Assert.DoesNotContain("not complete", fired.DetailText, StringComparison.Ordinal); + Assert.Equal(1, fired.NumericCurrentValue); + Assert.Equal(new DateTime(2026, 10, 11, 3, 0, 0, DateTimeKind.Utc), stamps.Stamps["history_audit_slot"]); + } + + [Fact] + public async Task ASourceThatNeverBecomesReady_KeepsItsFlaggedHours_AndTheGiveUpAlertMarksThemNotComplete() + { + /* H1 of the #5461 round 2 review: a rollup whose refresh stopped at 10:00Z. Perfmon is clean. Every read finds + the rollup not ready; its hole (hour 10 to 24, zero servers) must reach the alert at the give-up. */ + var stamps = new MemoryStampStore(); + var p = new Process(stamps, FirstPass); + var stopped = WithAuditDay(Week(), h => h <= 9 ? Usual(h) : null); + var clean = WithAuditDay(Week(), h => Usual(h)); + IReadOnlyList? Source(Rollup r) => r.Relation == QueryStats.Relation ? stopped : clean; + + foreach (var tick in new[] { FirstPass, FirstPass.AddHours(8), new DateTime(2026, 10, 12, 2, 59, 0, DateTimeKind.Utc) }) + { + p.Now = tick; + await RunBoundedAsync(p, Source); + Assert.Empty(p.Deliverer.Outcomes); + Assert.Empty(stamps.Stamps); + } + + p.Now = new DateTime(2026, 10, 12, 3, 0, 0, DateTimeKind.Utc); + await RunBoundedAsync(p, Source); + + var fired = Assert.Single(p.Deliverer.Outcomes); + Assert.Contains("query_stats_interval_hourly: 10:00-24:00Z (servers as low as 0 vs usual 40) [not complete", fired.DetailText, StringComparison.Ordinal); + Assert.Contains("Not read for this day: query_stats_interval_hourly", fired.DetailText, StringComparison.Ordinal); + Assert.Equal(14, fired.NumericCurrentValue); + Assert.Single(p.Log.Entries, e => e.Message.Contains("gave up on collect.query_stats_interval_hourly", StringComparison.Ordinal)); + } + + [Fact] + public async Task AGiveUpWithNoFlaggedHours_RaisesNothing_AndLogsTheWarning() + { + var p = new Process(new MemoryStampStore(), FirstPass); + var clean = WithAuditDay(Week(), h => Usual(h)); + + p.Now = FirstPass; + await p.RunWithAsync((r, from) => r.Relation == QueryStats.Relation ? throw new TimeoutException("statement timeout") : clean); + p.Now = new DateTime(2026, 10, 12, 3, 0, 0, DateTimeKind.Utc); + await p.RunWithAsync((r, from) => r.Relation == QueryStats.Relation && from == AuditDay.AddDays(-7) ? throw new TimeoutException("statement timeout") : clean); + + Assert.Empty(p.Deliverer.Outcomes); + Assert.Single(p.Log.Entries, e => e.Message.Contains("gave up on collect.query_stats_interval_hourly", StringComparison.Ordinal)); + } + + /// A logger that switches the master switch off when the give-up warning is written: the switch going off + /// between the apply's top check and the give-up's own send. + private sealed class SwitchOffOnGiveUpLogger : Microsoft.Extensions.Logging.ILogger + { + private readonly DarlingSelfAlertTests.FakeSettings _settings; + public List Messages { get; } = new(); + + public SwitchOffOnGiveUpLogger(DarlingSelfAlertTests.FakeSettings settings) => _settings = settings; + + public IDisposable? BeginScope(TState state) where TState : notnull => null; + + public bool IsEnabled(Microsoft.Extensions.Logging.LogLevel logLevel) => true; + + public void Log( + Microsoft.Extensions.Logging.LogLevel logLevel, Microsoft.Extensions.Logging.EventId eventId, + TState state, Exception? exception, Func formatter) + { + var message = formatter(state, exception); + Messages.Add(message); + if (message.Contains("gave up on", StringComparison.Ordinal)) + { + _settings.AlertsEnabled = false; + } + } + } + + [Fact] + public async Task AGiveUpWithAlertsOff_LogsTheWarning_RaisesNothing_AndStillClearsTheHeldDay() + { + /* #3464 master-switch gate on the give-up's own send. The same never-ready source as above, so the give-up has + flagged hours to raise; the switch goes off as the give-up logs its warning. */ + var settings = new DarlingSelfAlertTests.FakeSettings(); + var deliverer = new DarlingSelfAlertTests.RecordingDeliverer(); + var log = new SwitchOffOnGiveUpLogger(settings); + var now = FirstPass; + var evaluator = new DarlingSelfAlertEvaluator( + settings, deliverer, new NoHistoryStore(), _ => false, + logger: log, utcNow: () => now, readFailures: new AlertReadFailureCounter(), deliveryStamps: new MemoryStampStore()); + var stopped = WithAuditDay(Week(), h => h <= 9 ? Usual(h) : null); + var clean = WithAuditDay(Week(), h => Usual(h)); + + Task Run() => evaluator.ApplyCollectionHistoryAuditAsync( + (rollup, from, to, token) => + { + var all = rollup.Relation == QueryStats.Relation ? stopped : clean; + return Task.FromResult?>(all.Where(b => b.HourUtc >= from && b.HourUtc < to).ToList()); + }, + new[] { QueryStats, Perfmon }, Ct); + + await Run(); + now = new DateTime(2026, 10, 12, 3, 0, 0, DateTimeKind.Utc); + await Run(); + + Assert.Empty(deliverer.Outcomes); + Assert.Single(log.Messages, m => m.Contains("gave up on collect.query_stats_interval_hourly", StringComparison.Ordinal)); + + /* Switched back on, the day is not given up on a second time: the held state was cleared. */ + settings.AlertsEnabled = true; + now = new DateTime(2026, 10, 12, 3, 30, 0, DateTimeKind.Utc); + await Run(); + + Assert.Single(log.Messages, m => m.Contains("gave up on", StringComparison.Ordinal)); + Assert.DoesNotContain(deliverer.Outcomes, o => o.DetailText?.Contains("2026-10-10", StringComparison.Ordinal) == true); + } + + /// A deliverer that applies the cooldown the way the real one does for a self-alert (no incidents): one + /// send per (server key, metric) pair, the repeat throttled. Throttled counts as delivered upstream, so a + /// throttled alert simply never reaches a channel. + private sealed class CooldownDeliverer : IAlertDeliverer + { + private readonly HashSet _sent = new(StringComparer.Ordinal); + public List Delivered { get; } = new(); + public List Throttled { get; } = new(); + + public Task DeliverAsync(AlertOutcome outcome, CancellationToken cancellationToken = default) + { + (_sent.Add(outcome.ServerKey + "|" + outcome.MetricName) ? Delivered : Throttled).Add(outcome); + return Task.CompletedTask; + } + + public async Task DeliverAndReportAsync(AlertOutcome outcome, CancellationToken cancellationToken = default) + { + await DeliverAsync(outcome, cancellationToken); + return null; + } + } + + [Fact] + public async Task TheGiveUpAlertForOneDay_AndTheNextDaysAudit_AreBothDelivered_InOneTick() + { + /* M1 of the #5461 round 2 review: at the first tick on or after D+2 03:00Z the give-up for D fires, then the + same call audits D+1. Under one delivery key the second was throttled by the cooldown and never reached a + channel; with the audit day in the key they are two incidents. */ + var cooldown = new CooldownDeliverer(); + var stamps = new MemoryStampStore(); + var settings = new DarlingSelfAlertTests.FakeSettings(); + var now = FirstPass; + var evaluator = new DarlingSelfAlertEvaluator( + settings, cooldown, new NoHistoryStore(), _ => false, utcNow: () => now, deliveryStamps: stamps); + + var dayOne = WithAuditDay(Week(), h => h is >= 18 and <= 20 ? null : Usual(h)); + var dayTwo = Week().Concat(Day(AuditDay, h => Usual(h))).Concat(Day(AuditDay.AddDays(1), h => h is >= 5 and <= 6 ? null : Usual(h))).ToList(); + var firstDayFrom = AuditDay.AddDays(-7); + + Task Run() => evaluator.ApplyCollectionHistoryAuditAsync( + (rollup, from, to, token) => + { + if (from == firstDayFrom) + { + /* Day D: the workload rollup keeps timing out; perfmon reads and is holey. */ + return rollup.Relation == QueryStats.Relation + ? throw new TimeoutException("statement timeout") + : Task.FromResult?>(dayOne); + } + + return Task.FromResult?>(dayTwo); + }, + new[] { QueryStats, Perfmon }, Ct); + + await Run(); + now = new DateTime(2026, 10, 12, 2, 59, 0, DateTimeKind.Utc); + await Run(); + Assert.Empty(cooldown.Delivered); + + now = new DateTime(2026, 10, 12, 3, 0, 0, DateTimeKind.Utc); + await Run(); + + Assert.Equal(2, cooldown.Delivered.Count); + Assert.Empty(cooldown.Throttled); + Assert.Contains("2026-10-10", cooldown.Delivered[0].DetailText, StringComparison.Ordinal); + Assert.Contains("Not read for this day: query_stats_interval_hourly", cooldown.Delivered[0].DetailText, StringComparison.Ordinal); + Assert.Contains("2026-10-11", cooldown.Delivered[1].DetailText, StringComparison.Ordinal); + Assert.NotEqual(cooldown.Delivered[0].ServerKey, cooldown.Delivered[1].ServerKey); + Assert.StartsWith("collectionhistoryaudit:2026-10-1", cooldown.Delivered[0].ServerKey, StringComparison.Ordinal); + } + + [Fact] + public async Task TheAlertText_SaysAPerfmonScheduleChangeCanTriggerItForAFewDays() + { + var p = new Process(new MemoryStampStore(), FirstPass.AddHours(1)); + var holey = WithAuditDay(Week(), h => h is >= 18 and <= 20 ? null : Usual(h)); + await p.RunAsync(holey, holey); + + var fired = Assert.Single(p.Deliverer.Outcomes); + Assert.Contains("A change to the perfmon collection schedule", fired.DetailText, StringComparison.Ordinal); + Assert.Contains("for a few days", fired.DetailText, StringComparison.Ordinal); + Assert.Equal(3, fired.NumericCurrentValue); + } + + [Fact] + public async Task AFailedRead_IsRetriedAlone_WarnsOnEachFailure_AndTheAlertIsRaisedOnceEverySourceIsRead() + { + var p = new Process(new MemoryStampStore(), FirstPass.AddHours(1)); + var holey = WithAuditDay(Week(), h => h == 12 ? null : Usual(h)); + var reads = new Dictionary(); + + await p.RunWithAsync((r, from) => r.Relation == QueryStats.Relation + ? throw new TimeoutException("statement timeout") + : holey, reads); + Assert.Empty(p.Deliverer.Outcomes); + Assert.Single(p.Log.Entries, e => e.Message.Contains("could not read", StringComparison.Ordinal)); + Assert.Equal(1L, p.ReadFailures.ReadInstance().ReadFailures); + + p.Now = p.Now.AddHours(1); + await p.RunWithAsync((r, from) => holey, reads); + Assert.Equal(1, reads[Perfmon.Relation]); + Assert.Equal(2, reads[QueryStats.Relation]); + var fired = Assert.Single(p.Deliverer.Outcomes); + Assert.StartsWith("Retained history for 2026-10-10", fired.DetailText, StringComparison.Ordinal); + Assert.Contains("12:00-13:00Z", fired.DetailText, StringComparison.Ordinal); + Assert.DoesNotContain("Not read", fired.DetailText, StringComparison.Ordinal); + } + + [Fact] + public async Task EveryReadFailing_SpendsNothing_SoTheNextTickAsksAgain() + { + var stamps = new MemoryStampStore(); + var p = new Process(stamps, FirstPass); + await p.Evaluator.ApplyCollectionHistoryAuditAsync( + (rollup, from, to, token) => throw new TimeoutException("down"), + new[] { QueryStats, Perfmon }, Ct); + Assert.Empty(p.Deliverer.Outcomes); + Assert.Empty(stamps.Stamps); + + p.Now = FirstPass.AddHours(1); + var holey = WithAuditDay(Week(), h => h == 4 ? null : Usual(h)); + await p.RunAsync(holey, holey); + Assert.Single(p.Deliverer.Outcomes); + } + + [Fact] + public async Task AnAbsentSource_AndANewStore_AreQuiet_NotAnAlertOrAWarning() + { + var p = new Process(new MemoryStampStore(), FirstPass.AddHours(1)); + /* Perfmon is absent (null); query stats has two prior days only. */ + await p.RunAsync(WithAuditDay(Week(priorDays: 2), h => null), null); + Assert.Empty(p.Deliverer.Outcomes); + Assert.Empty(p.Log.Entries); + + /* Both done, so the day is spent. */ + var reads = p.Reads; + p.Now = p.Now.AddHours(1); + await p.RunAsync(WithAuditDay(Week(priorDays: 2), h => null), null); + Assert.Equal(reads, p.Reads); + } + + [Fact] + public async Task AFirstPassAtFourteenHundred_StillAnchorsTheNextAuditOnThreeOClock() + { + var stamps = new MemoryStampStore(); + var afternoon = new DateTime(2026, 10, 11, 14, 20, 0, DateTimeKind.Utc); + var p = new Process(stamps, afternoon); + await p.RunAsync(WithAuditDay(Week(), h => Usual(h)), null); + Assert.Equal(new DateTime(2026, 10, 11, 3, 0, 0, DateTimeKind.Utc), stamps.Stamps["history_audit_slot"]); + } + + /* ---------------- live: seeded sources, through the real read ---------------- */ + + private static string? Env() => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static async Task ExecAsync(NpgsqlConnection connection, string sql) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(Ct); + } + + /// + /// Seeds 22 days from 2026-09-26 in the shape of the real sources. Servers 1-12 are enabled and 13-14 are + /// disabled but still have rows (M1: they must not count). audit_hourly is a workload rollup (servers + /// only: several query rows per server and hour); audit_passes is the perfmon baseline's shape (a row per + /// server per collection pass, four a hour). Days: 2026-10-10 has a 3-hour hole at 18-20Z in both; + /// 2026-10-14 has passes at 25% from 5-7Z with every server present; 2026-10-17 is weekend-shaped, one third of + /// the queries with every server and every pass present. + /// + private static async Task SeedAsync(NpgsqlConnection connection) + { + await ExecAsync(connection, @" +CREATE SCHEMA IF NOT EXISTS collect; +CREATE SCHEMA IF NOT EXISTS config; +CREATE TABLE config.config_monitored_servers (server_id integer PRIMARY KEY, is_enabled boolean NOT NULL); +INSERT INTO config.config_monitored_servers SELECT s, s <= 12 FROM generate_series(1, 14) AS s; +CREATE TABLE collect.audit_hourly (bucket timestamp NOT NULL, server_id integer NOT NULL, query_key integer NOT NULL); +CREATE TABLE collect.audit_passes (bucket timestamp NOT NULL, server_id integer NOT NULL, collection_time timestamp NOT NULL);"); + + await ExecAsync(connection, @" +INSERT INTO collect.audit_hourly (bucket, server_id, query_key) +SELECT b, s, q +FROM generate_series(TIMESTAMP '2026-09-26', TIMESTAMP '2026-10-17 23:00', INTERVAL '1 hour') AS b +CROSS JOIN generate_series(1, 14) AS s +CROSS JOIN generate_series(1, 3) AS q +WHERE NOT (b::date = DATE '2026-10-10' AND extract(hour FROM b) BETWEEN 18 AND 20) +AND NOT (b::date = DATE '2026-10-17' AND q > 1); + +INSERT INTO collect.audit_passes (bucket, server_id, collection_time) +SELECT b, s, b + p * INTERVAL '15 minutes' +FROM generate_series(TIMESTAMP '2026-09-26', TIMESTAMP '2026-10-17 23:00', INTERVAL '1 hour') AS b +CROSS JOIN generate_series(1, 14) AS s +CROSS JOIN generate_series(0, 3) AS p +WHERE NOT (b::date = DATE '2026-10-10' AND extract(hour FROM b) BETWEEN 18 AND 20) +AND NOT (b::date = DATE '2026-10-14' AND extract(hour FROM b) BETWEEN 5 AND 7 AND p > 0);"); + } + + [Fact] + public async Task Live_SeededSources_FlagTheHoleAndTheThinPasses_AndStayQuietOnANormalDayAndAWeekend() + { + Assert.SkipWhen(string.IsNullOrEmpty(Env()), "Set DARLING_TEST_PG to run the #5450 live tests."); + var ok = false; + var scratch = await ScratchPostgres.CreateAsync(Env()!, Ct); + try + { + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(Ct); + await SeedAsync(connection); + await using var source = NpgsqlDataSource.Create(scratch.ConnectionString); + var queries = new Rollup("collect.audit_hourly", false); + var passes = new Rollup("collect.audit_passes", true); + + async Task?> AuditAsync(Rollup rollup, DateTime day) + { + var buckets = await CollectionHistoryAudit.ReadAsync( + source, rollup, CollectionHistoryAudit.ReadFrom(day), CollectionHistoryAudit.ReadTo(day), Ct); + Assert.NotNull(buckets); + Assert.True(CollectionHistoryAudit.IsSettled(day, buckets!)); + return CollectionHistoryAudit.Analyze(day, buckets!); + } + + DateTime Utc(int month, int day) => new(2026, month, day, 0, 0, 0, DateTimeKind.Utc); + + /* The hole, in both sources. Usual counts only the 12 enabled servers, not the 14 with rows (M1), and + usual passes are 12 servers x 4 passes. */ + var hole = Assert.Single((await AuditAsync(queries, Utc(10, 10)))!); + Assert.Equal("18:00-21:00Z", CollectionHistoryAudit.RangeLabel(hole)); + Assert.Equal(12, hole.UsualServers); + Assert.Equal(0, hole.LowestServers); + var passHole = Assert.Single((await AuditAsync(passes, Utc(10, 10)))!); + Assert.Equal("18:00-21:00Z", CollectionHistoryAudit.RangeLabel(passHole)); + Assert.Equal(12, passHole.UsualServers); + Assert.Equal(48, passHole.UsualPasses); + + /* A normal day on both sides is quiet, in both sources. */ + foreach (var day in new[] { Utc(10, 9), Utc(10, 16) }) + { + Assert.Empty((await AuditAsync(queries, day))!); + Assert.Empty((await AuditAsync(passes, day))!); + } + + /* Case 2: every server present, passes at a quarter. The workload rollup (servers only) cannot see it; + the pass count can. */ + var thin = Assert.Single((await AuditAsync(passes, Utc(10, 14)))!); + Assert.Equal("05:00-08:00Z", CollectionHistoryAudit.RangeLabel(thin)); + Assert.Equal(12, thin.LowestServers); + Assert.Equal(12, thin.LowestPasses); + Assert.Equal(48, thin.UsualPasses); + Assert.Empty((await AuditAsync(queries, Utc(10, 14)))!); + + /* H2: a weekend-shaped day, a third of the queries but every server and every pass, flags nothing. */ + Assert.Empty((await AuditAsync(queries, Utc(10, 17)))!); + Assert.Empty((await AuditAsync(passes, Utc(10, 17)))!); + + /* M1: a server disabled now is out of both the audited day and usual. Disabling 6 of 12 servers flags + nothing, where counting them on prior days only would have flagged every hour. */ + await ExecAsync(connection, "UPDATE config.config_monitored_servers SET is_enabled = false WHERE server_id BETWEEN 7 AND 12;"); + Assert.Empty((await AuditAsync(queries, Utc(10, 16)))!); + Assert.Empty((await AuditAsync(passes, Utc(10, 16)))!); + await ExecAsync(connection, "UPDATE config.config_monitored_servers SET is_enabled = true WHERE server_id BETWEEN 7 AND 12;"); + + /* The same sources through the evaluator and the real read: one alert for the thin day, none for a + restart, none for the weekend-shaped day. */ + async Task RunAsync(Process process) => await process.Evaluator.ApplyCollectionHistoryAuditAsync( + (r, from, to, token) => CollectionHistoryAudit.ReadAsync(source, r, from, to, token), + new[] { queries, passes, new Rollup("collect.history_audit_absent_hourly", false) }, Ct); + + var stamps = new MemoryStampStore(); + var run = new Process(stamps, new DateTime(2026, 10, 15, 3, 0, 0, DateTimeKind.Utc)); + await RunAsync(run); + var fired = Assert.Single(run.Deliverer.Outcomes); + Assert.Contains("audit_passes: 05:00-08:00Z", fired.DetailText, StringComparison.Ordinal); + Assert.DoesNotContain("audit_hourly", fired.DetailText, StringComparison.Ordinal); + Assert.DoesNotContain("absent", fired.DetailText, StringComparison.Ordinal); + + var restarted = new Process(stamps, new DateTime(2026, 10, 15, 9, 0, 0, DateTimeKind.Utc)); + await RunAsync(restarted); + Assert.Empty(restarted.Deliverer.Outcomes); + + var weekend = new Process(new MemoryStampStore(), new DateTime(2026, 10, 18, 3, 0, 0, DateTimeKind.Utc)); + await RunAsync(weekend); + Assert.Empty(weekend.Deliverer.Outcomes); + + ok = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(ok, async () => await scratch.DisposeAsync()); + } + } + + [Fact] + public async Task Live_TheRealSourceList_ReadsNullOnAPlainStore_AndIsReadableOnTheAggregates() + { + Assert.SkipWhen(string.IsNullOrEmpty(Env()), "Set DARLING_TEST_PG to run the #5450 live tests."); + var ok = false; + var scratch = await ScratchPostgres.CreateAsync(Env()!, Ct); + try + { + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(Ct); + await PgMigrations.MigrateAsync(connection, Ct); + await using var source = NpgsqlDataSource.Create(scratch.ConnectionString); + + /* A plain store has none of the aggregates: every real source reads null, and the audit does nothing. */ + foreach (var rollup in CollectionHistoryAudit.Rollups) + { + Assert.Null(await CollectionHistoryAudit.ReadAsync( + source, rollup, CollectionHistoryAudit.ReadFrom(AuditDay), CollectionHistoryAudit.ReadTo(AuditDay), Ct)); + } + + /* The TimescaleDB shape: build the real aggregates (empty), and the statement is valid against each. */ + Assert.True(await TimescaleSupport.TryEnableAsync(connection, null, Ct), "TimescaleDB must be enabled on the test cluster"); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, Ct); + foreach (var (createSql, _) in TimescaleSupport.HourlyAggregates) + { + await ExecAsync(connection, createSql); + } + + await ExecAsync(connection, TimescaleSupport.CreatePerfmonIntervalBaselineSql); + + foreach (var rollup in CollectionHistoryAudit.Rollups) + { + var buckets = await CollectionHistoryAudit.ReadAsync( + source, rollup, CollectionHistoryAudit.ReadFrom(AuditDay), CollectionHistoryAudit.ReadTo(AuditDay), Ct); + Assert.NotNull(buckets); + Assert.Empty(buckets!); + } + + ok = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(ok, async () => await scratch.DisposeAsync()); + } + } + + /* ---------------- live: the #5461 probe read against the real Query Store rollups ---------------- */ + + private static readonly DateTime ProbeFrom = new(2026, 3, 4, 0, 0, 0, DateTimeKind.Utc); + + /// + /// Builds the real store, strips the refresh policies (a background run must not race the manual refreshes), + /// and seeds collect.query_store_stats from 2026-03-04 00:00Z. Servers 101-103 are enabled, 104 is + /// disabled. Hours: 0 = all four servers (3 enabled), 1 = server 101 with 50 rows, 2 = nobody, 3 = servers 102 + /// and 103, 4 = only the disabled 104, 5 = all four, 6 = server 101 (outside the window the tests read). Then + /// the three Query Store hourlies are materialized, in dependency order. + /// + private static async Task SeedProbeStoreAsync(NpgsqlConnection connection) + { + await PgMigrations.MigrateAsync(connection, Ct); + Assert.True(await TimescaleSupport.TryEnableAsync(connection, null, Ct), "TimescaleDB must be enabled on the test cluster"); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, Ct); + await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, Ct); + foreach (var (view, _, _, _, _) in TimescaleSupport.RollupViews) + { + await ExecAsync(connection, $"SELECT remove_continuous_aggregate_policy('collect.{view}', if_exists => true)"); + } + + await ExecAsync(connection, @" +INSERT INTO config.config_monitored_servers (server_id, name, host, is_enabled) VALUES + (101, 'probe-101', 'h', true), (102, 'probe-102', 'h', true), (103, 'probe-103', 'h', true), (104, 'probe-104', 'h', false); + +CREATE TEMP TABLE probe_spec (server_id integer, hr integer, n integer); +INSERT INTO probe_spec VALUES + (101, 0, 1), (102, 0, 1), (103, 0, 1), (104, 0, 1), + (101, 1, 50), + (102, 3, 1), (103, 3, 2), + (104, 4, 3), + (101, 5, 1), (102, 5, 1), (103, 5, 1), (104, 5, 1), + (101, 6, 1); + +INSERT INTO collect.query_store_stats (collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, + execution_type_desc, first_execution_time, module_name, query_hash, execution_count, avg_duration_us, avg_cpu_time_us, + max_duration_us, max_cpu_time_us, replica_role, runtime_stats_interval_id, interval_start_time_utc) +SELECT 1, TIMESTAMP '2026-03-04 00:00' + (s.hr * interval '1 hour') + ((i % 6) * interval '10 minutes'), + s.server_id, 'probe-' || s.server_id, 'db', 100 + i, 1000 + i, + 'Regular', TIMESTAMP '2026-03-04 00:00', 'mod', md5('q' || i), 10, 500, 300, 900, 700, 'PRIMARY', 77 + i, + TIMESTAMP '2026-03-04 00:00' +FROM probe_spec AS s CROSS JOIN LATERAL generate_series(1, s.n) AS i;"); + + var to = ProbeFrom.AddHours(12); + foreach (var view in new[] + { + TimescaleSupport.QueryStoreStatsHourlyView, + TimescaleSupport.QueryStoreStatsIntervalHourlyView, + TimescaleSupport.QueryStoreStatsCorrectedHourlyView, + }) + { + for (var attempt = 1; ; attempt++) + { + try + { + await using var refresh = new NpgsqlCommand( + $"CALL refresh_continuous_aggregate('collect.{view}', $1::timestamp, $2::timestamp)", connection); + refresh.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Timestamp, Value = DateTime.SpecifyKind(ProbeFrom, DateTimeKind.Unspecified) }); + refresh.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Timestamp, Value = DateTime.SpecifyKind(to, DateTimeKind.Unspecified) }); + await refresh.ExecuteNonQueryAsync(Ct); + break; + } + catch (PostgresException ex) when (ex.SqlState == "55P03" && attempt < 12) + { + await Task.Delay(TimeSpan.FromSeconds(1), Ct); + } + } + } + } + + [Fact] + public async Task Live_TheProbeRead_EqualsAnOracleDistinctServerCount_OnBothQueryStoreRollups_AndLeavesAnEmptyHourAbsent() + { + Assert.SkipWhen(string.IsNullOrEmpty(Env()), "Set DARLING_TEST_PG to run the #5461 live tests."); + var ok = false; + var scratch = await ScratchPostgres.CreateAsync(Env()!, Ct); + try + { + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(Ct); + await SeedProbeStoreAsync(connection); + await using var source = NpgsqlDataSource.Create(scratch.ConnectionString); + var to = ProbeFrom.AddHours(6); + + foreach (var view in new[] { TimescaleSupport.QueryStoreStatsHourlyView, TimescaleSupport.QueryStoreStatsCorrectedHourlyView }) + { + var probe = await CollectionHistoryAudit.ReadAsync( + source, new Rollup("collect." + view, false, CollectionHistoryAudit.ReadShape.Probe), ProbeFrom, to, Ct); + var scan = await CollectionHistoryAudit.ReadAsync( + source, new Rollup("collect." + view, false, CollectionHistoryAudit.ReadShape.Scan), ProbeFrom, to, Ct); + Assert.NotNull(probe); + Assert.NotNull(scan); + + /* The oracle, written here: distinct enabled servers per bucket, buckets with no row absent. */ + var oracle = new Dictionary(); + await using (var command = new NpgsqlCommand($@" +SELECT r.bucket, COUNT(DISTINCT r.server_id)::int +FROM collect.{view} AS r +WHERE r.server_id IN (SELECT server_id FROM config.config_monitored_servers WHERE is_enabled) +AND r.bucket >= $1 AND r.bucket < $2 +GROUP BY r.bucket", connection)) + { + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Timestamp, Value = DateTime.SpecifyKind(ProbeFrom, DateTimeKind.Unspecified) }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Timestamp, Value = DateTime.SpecifyKind(to, DateTimeKind.Unspecified) }); + await using var reader = await command.ExecuteReaderAsync(Ct); + while (await reader.ReadAsync(Ct)) + { + oracle[DateTime.SpecifyKind(reader.GetDateTime(0), DateTimeKind.Utc)] = reader.GetInt32(1); + } + } + + /* Hours 0 (3 enabled of 4), 1 (one server, 50 rows), 3 (two), 5 (3 enabled of 4). Hour 2 has no + server and hour 4 only the disabled one: both ABSENT, as in the scan. Hour 6 is the exclusive end. */ + var expected = new Dictionary + { + [ProbeFrom] = 3, [ProbeFrom.AddHours(1)] = 1, [ProbeFrom.AddHours(3)] = 2, [ProbeFrom.AddHours(5)] = 3, + }; + Assert.Equal(expected, oracle); + Assert.Equal(oracle, probe!.ToDictionary(b => b.HourUtc, b => b.Servers)); + Assert.Equal(oracle, scan!.ToDictionary(b => b.HourUtc, b => b.Servers)); + Assert.All(probe!, b => Assert.Equal(0L, b.Passes)); + Assert.Equal(4, probe!.Count); + + /* The audited day 2026-03-04 settles on a bucket at or after 23:00: the empty hours must not read as one. */ + Assert.False(CollectionHistoryAudit.IsSettled(ProbeFrom, probe!)); + } + + ok = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(ok, async () => await scratch.DisposeAsync()); + } + } + + [Fact] + public async Task Live_TheProbeRead_IsANestedLoopOverTheServerIdBucketIndex_OnATimestampSeries_AgainstARealTimeFreeView() + { + Assert.SkipWhen(string.IsNullOrEmpty(Env()), "Set DARLING_TEST_PG to run the #5461 live tests."); + var ok = false; + var scratch = await ScratchPostgres.CreateAsync(Env()!, Ct); + try + { + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(Ct); + await SeedProbeStoreAsync(connection); + + foreach (var view in new[] { TimescaleSupport.QueryStoreStatsHourlyView, TimescaleSupport.QueryStoreStatsCorrectedHourlyView }) + { + /* The probe reads the materialization only: the view is not real-time, and bucket is a timestamp + (the series is a timestamp series, so no cross-type compare keeps the index out). */ + await using (var meta = new NpgsqlCommand(@" +SELECT ca.materialized_only, ca.materialization_hypertable_schema, ca.materialization_hypertable_name, + (SELECT data_type FROM information_schema.columns + WHERE table_schema = ca.materialization_hypertable_schema AND table_name = ca.materialization_hypertable_name AND column_name = 'bucket') +FROM timescaledb_information.continuous_aggregates AS ca +WHERE ca.view_schema = 'collect' AND ca.view_name = $1", connection)) + { + meta.Parameters.AddWithValue(view); + await using var reader = await meta.ExecuteReaderAsync(Ct); + Assert.True(await reader.ReadAsync(Ct)); + Assert.True(reader.GetBoolean(0), view + " must be materialized_only: the probe reads the materialization alone (#5461)"); + Assert.Equal("timestamp without time zone", reader.GetString(3)); + } + + var sql = CollectionHistoryAudit.BuildSql(new Rollup("collect." + view, false, CollectionHistoryAudit.ReadShape.Probe)); + await using var transaction = await connection.BeginTransactionAsync(Ct); + + /* The rig's data is a few rows, where the planner prefers a seq scan of the materialization; the + rig proves the index path EXISTS for this statement, and the big store proves its cost. */ + await ExecAsync(connection, "SET LOCAL enable_seqscan = off"); + + string json; + await using (var explain = new NpgsqlCommand("EXPLAIN (FORMAT JSON) " + sql, connection, transaction)) + { + explain.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Timestamp, Value = DateTime.SpecifyKind(ProbeFrom, DateTimeKind.Unspecified) }); + explain.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Timestamp, Value = DateTime.SpecifyKind(ProbeFrom.AddHours(200), DateTimeKind.Unspecified) }); + json = (string)(await explain.ExecuteScalarAsync(Ct))!; + } + + var nodes = new List(); + void Walk(System.Text.Json.JsonElement node) + { + nodes.Add(node); + if (node.TryGetProperty("Plans", out var plans)) + { + foreach (var child in plans.EnumerateArray()) + { + Walk(child); + } + } + } + + Walk(System.Text.Json.JsonDocument.Parse(json).RootElement[0].GetProperty("Plan")); + string Text(System.Text.Json.JsonElement n, string name) => n.TryGetProperty(name, out var v) ? v.GetString() ?? "" : ""; + + Assert.Contains(nodes, n => Text(n, "Node Type") == "Nested Loop"); + var indexScans = nodes.Where(n => Text(n, "Node Type") is "Index Scan" or "Index Only Scan" + && Text(n, "Relation Name").StartsWith("_hyper_", StringComparison.Ordinal)).ToList(); + Assert.NotEmpty(indexScans); + Assert.DoesNotContain(nodes, n => Text(n, "Node Type") == "Seq Scan" + && Text(n, "Relation Name").StartsWith("_hyper_", StringComparison.Ordinal)); + + foreach (var scan in indexScans) + { + /* The index is the kept (server_id, bucket) one, probed on both columns with no cross-type cast. */ + await using var def = new NpgsqlCommand("SELECT indexdef FROM pg_indexes WHERE indexname = $1", connection, transaction); + def.Parameters.AddWithValue(Text(scan, "Index Name")); + var indexDef = (string?)await def.ExecuteScalarAsync(Ct); + Assert.Contains("(server_id, bucket", indexDef, StringComparison.Ordinal); + var cond = Text(scan, "Index Cond"); + Assert.Contains("server_id", cond, StringComparison.Ordinal); + Assert.Contains("bucket", cond, StringComparison.Ordinal); + Assert.DoesNotContain("time zone", cond, StringComparison.Ordinal); + } + + await transaction.RollbackAsync(Ct); + } + + ok = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(ok, async () => await scratch.DisposeAsync()); + } + } +} diff --git a/Darling/Darling.Tests/CollectionLogDrainForensicsStoreTests.cs b/Darling/Darling.Tests/CollectionLogDrainForensicsStoreTests.cs index 3469d51fb9..d6ae955ecf 100644 --- a/Darling/Darling.Tests/CollectionLogDrainForensicsStoreTests.cs +++ b/Darling/Darling.Tests/CollectionLogDrainForensicsStoreTests.cs @@ -236,37 +236,40 @@ public void TheUnmeasuredSentinelIsNeverWrittenAsALiteral() } /// - /// The session id is captured AFTER the open round trip, exactly once, and the helper normalizes the - /// provider's not-populated 0 to null (#2884). - /// - /// SqlConnection.ServerProcessId is not reliably populated until the connection has round-tripped - /// a command. The original capture sat before ExecuteReaderAsync, so the runs it mattered most for — - /// budget-abandoned cycles, where the id is the join key to waiting_tasks and the peer snapshots — - /// recorded a literal 0. Ordering is pinned on the source because it IS a source property: the capture - /// call must come after the ExecuteReaderAsync await it depends on, and a refactor that hoists it back - /// above the open reintroduces #2884 while compiling clean and passing every behavioral test that - /// cannot construct a real SqlConnection. + /// The session id is resolved right after the target connection opens and BEFORE any reader exists on + /// it, through the per-connection cache (#5132). Under MultipleActiveResultSets + /// SqlConnection.ServerProcessId is 0 at every point, so the old capture after + /// ExecuteReaderAsync recorded nothing; the open-time lookup is the only source. The ordering and + /// the single call site are source properties, pinned here. /// [Fact] - public void TheSessionIdIsCapturedAfterTheOpenRoundTrip() + public void TheSessionIdIsResolvedRightAfterTheOpen_BeforeAnyReader() { var runner = ReadSource("Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs"); - const string openCall = "opened = await command.ExecuteReaderAsync("; - const string captureCall = "context.TargetSessionId = TryReadTargetSessionId("; - + const string openCall = "await targetConnection.OpenAsync(cancellationToken);"; + const string captureCall = "await TargetSessionIdCache.Shared.ResolveAsync(targetConnection"; var open = runner.IndexOf(openCall, StringComparison.Ordinal); var capture = runner.IndexOf(captureCall, StringComparison.Ordinal); Assert.True(open >= 0, "the server-scoped open call moved; re-anchor this pin"); - Assert.True(capture >= 0, "the session-id capture is gone entirely"); - Assert.True(capture > open, - "the session id must be captured AFTER ExecuteReaderAsync has round-tripped (#2884) — " + - "before it, SqlConnection.ServerProcessId reads 0 on exactly the abandoned cycles it exists to explain"); + Assert.True(capture > open, "the session-id lookup must follow the open"); + + /* BOTH reader call sites on the connection come after the capture: the enumerated path's list + reader and the plain path's reader. */ + foreach (var readerCall in new[] { "ExecuteReaderAsync(cancellationToken)", "opened = await command.ExecuteReaderAsync(itemToken)" }) + { + var reader = runner.IndexOf(readerCall, open, StringComparison.Ordinal); + Assert.True(reader > 0, $"the reader call '{readerCall}' moved; re-anchor this pin"); + Assert.True(capture < reader, $"the session-id lookup must precede '{readerCall}'"); + } + Assert.Equal(capture, runner.LastIndexOf(captureCall, StringComparison.Ordinal)); - /* The helper's own normalization: 0 is not a session id and must become null at the source. */ - Assert.Contains("return raw > 0 ? raw : null;", runner, StringComparison.Ordinal); + /* The reconnect check sits right after the plain path's first reader, never before it. */ + var plainReader = runner.IndexOf("opened = await command.ExecuteReaderAsync(itemToken)", StringComparison.Ordinal); + var discard = runner.IndexOf("TargetSessionIdCache.Shared.DiscardIfReplaced(", StringComparison.Ordinal); + Assert.True(discard > plainReader && plainReader > 0, "the replaced-connection check must follow the first reader's open"); } /// @@ -302,7 +305,7 @@ public void ThePeerMarkIsCapturedAtDispatchNotAtCompletion() var worker = ReadSource("Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs"); Assert.Contains("var peerMaxAtDispatchMs = PeerMaxOrNull(server);", worker, StringComparison.Ordinal); - Assert.Contains("RunDetachedAsync(server, runner, name, peerMaxAtDispatchMs, cancellationToken)", worker, StringComparison.Ordinal); + Assert.Contains("RunDetachedAsync(server, runner, name, peerMaxAtDispatchMs, cancellationToken", worker, StringComparison.Ordinal); /* The write uses the captured parameter. Re-reading live server state here is the defect. @@ -318,6 +321,15 @@ no pin. This form is adjacency-independent on the left and still pinned on the r Assert.Contains("result.FetchPhases, peerMaxAtDispatchMs, _logger", worker, StringComparison.Ordinal); Assert.DoesNotContain("PeerMaxOrNull(server), _logger", worker, StringComparison.Ordinal); + /* #5320: the statement text refresh is not a scheduled body either - it rides the statistics collector's run and + writes its own failure and success rows from outside RunOneAsync, with no dispatch mark to carry and no + drain to describe. The counts below are about the rows RunOneAsync writes, so that method is cut out of + the text they read; the live-re-read pin above still reads the whole file. */ + var refreshStart = worker.IndexOf("private async Task TryRefreshPgStatementTextAsync(", StringComparison.Ordinal); + var refreshEnd = worker.IndexOf("private static async Task<(List QueryIds", refreshStart, StringComparison.Ordinal); + Assert.True(refreshStart > 0 && refreshEnd > refreshStart, "#5320: the text refresh method moved; update this carve-out"); + worker = worker.Remove(refreshStart, refreshEnd - refreshStart); + /* The two callers that are not a scheduled body pass null rather than folding a previous body's bookkeeping into their rows. */ Assert.Equal(2, Regex.Matches(worker, @"peerMaxAtDispatchMs: null").Count); @@ -341,13 +353,18 @@ bookkeeping into their rows. */ route can read. The pin is what turned that into a merge conflict to resolve rather than a wrong number nobody noticed. #4053's no-csvlog-file arm is the twelfth, the csvlog route's twin of the eleventh: the collector is on and csvlog is configured, but no .csv file exists yet. #4053's - no-jsonlog-file arm is the thirteenth, the jsonlog route's twin of the twelfth. */ - Assert.Equal(13, Regex.Matches(worker, @"drain: null").Count); - - /* And V110's fetch sums stay null on those same thirteen arms and for the same reason: no item + no-jsonlog-file arm is the thirteenth, the jsonlog route's twin of the twelfth. #5378's + long-query PERMISSIONS arm is the fourteenth: the session's create was denied (the login lacks + ALTER ANY EVENT SESSION), so the run is an early return that read and drained nothing. The RDS + endpoint-check arm is the fifteenth: the run stops before any RDS or Performance Insights call, so it read and + drained nothing either. The AWS role arm (#5452) is the sixteenth, for the same reason: a role the service cannot + use stops the run before any RDS call. */ + Assert.Equal(16, Regex.Matches(worker, @"drain: null").Count); + + /* And V110's fetch sums stay null on those same sixteen arms and for the same reason: no item completed, so no fetch was performed. Counted rather than merely present, so an arm that starts passing a real value - which would mean attributing another run's fetch to a failure row - is a red. */ - Assert.Equal(13, Regex.Matches(worker, @"fetchPhases: null").Count); + Assert.Equal(16, Regex.Matches(worker, @"fetchPhases: null").Count); } /// diff --git a/Darling/Darling.Tests/CollectionLogFanoutRollupStoreTests.cs b/Darling/Darling.Tests/CollectionLogFanoutRollupStoreTests.cs index ec474ec7b6..f79b782221 100644 --- a/Darling/Darling.Tests/CollectionLogFanoutRollupStoreTests.cs +++ b/Darling/Darling.Tests/CollectionLogFanoutRollupStoreTests.cs @@ -32,6 +32,7 @@ namespace Darling.Tests; /// is the arithmetic. Both aggregate over RUNS, /// and each of those runs is one blended row. /// +[Trait("Reads", "Lite")] public class CollectionLogFanoutRollupStoreTests { [Fact] diff --git a/Darling/Darling.Tests/CollectionLogSegmentByLiveTests.cs b/Darling/Darling.Tests/CollectionLogSegmentByLiveTests.cs new file mode 100644 index 0000000000..a60aeabb5c --- /dev/null +++ b/Darling/Darling.Tests/CollectionLogSegmentByLiveTests.cs @@ -0,0 +1,722 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Linq; +using System.Text; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Analysis.Baselines; +using PerformanceMonitor.Darling.Analysis; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4951: collection_log compresses segmented by collector as well as by server, so a read for one collector +/// decompresses only that collector's batches instead of every collector's runs on the server. +/// +/// Each test mints its own scratch database: the chunk census and the plan shape are statements about one +/// store's chunks, and another class's leftovers would change them. +/// +/// The expected segmentby is a literal here on purpose. These tests pin the value the store ends up with, so they +/// must not move when the product's constant does. +/// +/// The "upgraded" stores are built with the statements every build before #4951 ran (create_hypertable, then +/// the enable ALTER with server_id alone), with their older chunks compressed under those settings. +/// +[Collection("live-postgres")] +public sealed class CollectionLogSegmentByLiveTests +{ + /// What the store must end with, in the spelling the helper below reads it back in. + private const string WantedSegmentBy = "server_id, collector_name"; + + /// What every store converted before #4951 has. + private const string OldSegmentBy = "server_id"; + + /// The same two values as the per-chunk view spells them (comma, no space). + private const string WantedChunkSegmentBy = "server_id,collector_name"; + private const string OldChunkSegmentBy = "server_id"; + + private const int ServerA = -495101; + private const int ServerB = -495102; + private const string ServerName = "SEGMENTBY-SRV"; + private const string HeldCollectorColumn = "collector_name_held"; + + /// Well below any id the generator hands out, so a seeded row can never collide with a real one. + private const long LogIdBase = -495_100_000_000L; + + private static readonly string[] Collectors = { "blocked_process_report", "wait_stats", "cpu_utilization" }; + + /// The plan test's analysis hour, in the past, so every seeded chunk is old enough to compress. It is the + /// coverage-read plan class's own hour, because that class's EXPLAIN helper reads the window at it. + private static readonly DateTime AnalysisTime = DarlingEventBaselineCoverageReadPlanShapeLiveTests.AnalysisTime; + + /// Inside the plan test's baseline window. + private static readonly DateTime EventTime = DarlingEventBaselineCoverageReadPlanShapeLiveTests.EventTime; + + /* ---------------- fresh stores (condition 1) ---------------- */ + + /// + /// The usual fresh store: the migrations run before CREATE EXTENSION, so V23's guard skips and the runtime ensure + /// both converts collection_log and sets its compression. This one collected on plain PostgreSQL for days before + /// the extension arrived, so the conversion moves rows old enough to compress at once. The ensure's first ALTER + /// must already be the new setting, so the policy's first run compresses every one of those chunks with it, and + /// the store never compresses a chunk under server_id alone. + /// + [Fact] + public async Task AFreshStore_WhoseExtensionComesAfterTheMigrations_GetsTheNewSegmentBy_OnItsFirstConversion() + { + var baseConnectionString = RequireLivePostgres(); + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString, ct); + using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + + Assert.SkipWhen( + await ScalarAsync(connection, "SELECT count(*) FROM pg_extension WHERE extname = 'timescaledb'", ct) > 0, + "this cluster's template database already carries TimescaleDB, so a new database cannot start without it; the extension-first test covers that order."); + + await PgMigrations.MigrateAsync(connection, ct); + Assert.False((await ReadHypertableAsync(connection, ct)).IsHypertable, + "V23 converted collection_log on a store without the extension; this test needs the usual order"); + + /* Six days of runs, written while collection_log is still a plain table. */ + var now = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + await SeedAsync(connection, "collector_name", new[] { ServerA, ServerB }, now.Date.AddDays(-6), now.AddMinutes(-15), ct); + var dataBefore = await DataHashAsync(connection, "collector_name", ct); + + Assert.SkipUnless(await LiveTimescaleProbe.TryEnableAsync(scratch.ConnectionString, ct), + "TimescaleDB is not available on this cluster."); + await StopBackgroundWorkersAsync(connection, ct); + await RequirePerChunkSettingsAsync(connection, ct); + + Assert.True(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, null, ct)); + + var state = await ReadHypertableAsync(connection, ct); + Assert.True(state.IsHypertable); + Assert.True(state.CompressionEnabled); + Assert.Equal(WantedSegmentBy, state.SegmentBy); + Assert.Contains(TimescaleSupport.CollectionLogTable, await ConvergedTablesAsync(connection, ct)); + + /* The policy's first run compresses the converted rows' older chunks, every one with the new value, and + changes no row. */ + await RunCompressionPolicyAsync(connection, ct); + var chunks = await ReadChunkSettingsAsync(connection, ct); + Assert.NotEmpty(chunks); + Assert.All(chunks, chunk => Assert.Equal(WantedChunkSegmentBy, chunk.SegmentBy)); + Assert.Equal(dataBefore, await DataHashAsync(connection, "collector_name", ct)); + } + + /// + /// The other fresh-store order: TimescaleDB is installed before the migrations run (a bring-your-own store whose + /// administrator created it first), so V23 does the first conversion, with the server_id its text has always + /// had (a migration is never edited). V23's policy has no start time, so the scheduler runs it as soon as it is + /// added; on a new store that run finds an empty table. The first start's ensure then moves the hypertable to the + /// new value before any row is old enough to compress, so every chunk the policy compresses later uses the new + /// value and none uses server_id alone. + /// + [Fact] + public async Task AFreshStore_WhoseExtensionPredatesTheMigrations_HasTheNewSegmentBy_AfterItsFirstStart_AndNoServerIdOnlyChunk() + { + var baseConnectionString = RequireLivePostgres(); + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString, ct); + + Assert.SkipUnless(await LiveTimescaleProbe.TryEnableAsync(scratch.ConnectionString, ct), + "TimescaleDB is not available on this cluster."); + + using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await StopBackgroundWorkersAsync(connection, ct); + await RequirePerChunkSettingsAsync(connection, ct); + await PgMigrations.MigrateAsync(connection, ct); + + /* V23's own work: converted, compression on, with its text's server_id. */ + var afterMigrations = await ReadHypertableAsync(connection, ct); + Assert.True(afterMigrations.IsHypertable, "V23 did not convert collection_log although the extension existed when it ran"); + Assert.True(afterMigrations.CompressionEnabled); + Assert.Equal(OldSegmentBy, afterMigrations.SegmentBy); + + /* V23's policy's immediate first run (the background workers are stopped, so it runs here). A new store's + table is empty, so it compresses nothing under server_id. */ + await RunCompressionPolicyAsync(connection, ct); + Assert.Empty(await ReadChunkSettingsAsync(connection, ct)); + + /* The first start. */ + Assert.True(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, null, ct)); + Assert.Equal(WantedSegmentBy, (await ReadHypertableAsync(connection, ct)).SegmentBy); + Assert.Contains(TimescaleSupport.CollectionLogTable, await ConvergedTablesAsync(connection, ct)); + + /* Days of runs later, the policy compresses the older chunks, every one with the new value. */ + var now = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + await SeedAsync(connection, "collector_name", new[] { ServerA, ServerB }, now.Date.AddDays(-6), now.AddMinutes(-15), ct); + await RunCompressionPolicyAsync(connection, ct); + var chunks = await ReadChunkSettingsAsync(connection, ct); + Assert.NotEmpty(chunks); + Assert.All(chunks, chunk => Assert.Equal(WantedChunkSegmentBy, chunk.SegmentBy)); + } + + /* ---------------- upgraded stores ---------------- */ + + /// + /// An upgraded store whose chunks are already compressed under server_id. The ensure moves the hypertable to + /// the new value, leaves every compressed chunk as it was, and from then on reads as converged WHILE those old + /// chunks remain, so the hourly pass issues no ALTER. The settings read looks at the hypertable's own row, never at + /// a chunk's. + /// + [Fact] + public async Task AnUpgradedStore_MovesTheHypertableToTheNewSegmentBy_LeavesOldChunksAlone_AndReadsAsConverged() + { + var baseConnectionString = RequireLivePostgres(); + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString, ct); + using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await BuildUpgradedLogAsync(connection, scratch.ConnectionString, "collector_name", ct); + + var chunksBefore = await ReadChunkSettingsAsync(connection, ct); + Assert.NotEmpty(chunksBefore); + Assert.All(chunksBefore, chunk => Assert.Equal(OldChunkSegmentBy, chunk.SegmentBy)); + var dataBefore = await DataHashAsync(connection, "collector_name", ct); + + Assert.True(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, null, ct)); + + Assert.Equal(WantedSegmentBy, (await ReadHypertableAsync(connection, ct)).SegmentBy); + Assert.Equal(chunksBefore, await ReadChunkSettingsAsync(connection, ct)); + Assert.Equal(dataBefore, await DataHashAsync(connection, "collector_name", ct)); + Assert.Contains(TimescaleSupport.CollectionLogTable, await ConvergedTablesAsync(connection, ct)); + + /* A second pass is a no-op on the same store. */ + Assert.True(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, null, ct)); + Assert.Equal(chunksBefore, await ReadChunkSettingsAsync(connection, ct)); + } + + /// + /// The pin behind "a converged store takes no ACCESS EXCLUSIVE lock each hour". With the setting converged and old + /// chunks still present, another session holds a lock that any ALTER would have to wait behind. The pass must + /// finish without waiting and report success: had it issued the ALTER, it would have waited out its lock timeout + /// and reported the busy lock instead. + /// + [Fact] + public async Task AConvergedStore_IssuesNoAlter_SoAHeldLockDoesNotHoldUpTheHourlyPass() + { + var baseConnectionString = RequireLivePostgres(); + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString, ct); + using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await BuildUpgradedLogAsync(connection, scratch.ConnectionString, "collector_name", ct); + + Assert.True(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, null, ct)); + Assert.Equal(WantedSegmentBy, (await ReadHypertableAsync(connection, ct)).SegmentBy); + + await using var holder = new NpgsqlConnection(scratch.ConnectionString); + await holder.OpenAsync(ct); + await using var hold = await holder.BeginTransactionAsync(ct); + await ExecAsync(holder, "SELECT count(*) FROM collection_log", ct, hold); + + var logger = new CapturingTestLogger(); + using var bounded = CancellationTokenSource.CreateLinkedTokenSource(ct); + bounded.CancelAfter(TimeSpan.FromSeconds(60)); + var elapsed = Stopwatch.StartNew(); + var ok = await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, logger, bounded.Token); + elapsed.Stop(); + + Assert.True(ok, $"a converged pass did not finish cleanly beside a reader's lock (it took {elapsed.Elapsed}): {logger.Joined}"); + Assert.Equal(0, logger.CountAtLevel(Microsoft.Extensions.Logging.LogLevel.Warning)); + Assert.DoesNotContain(logger.Lines, line => line.Contains("compression settings", StringComparison.Ordinal)); + Assert.Equal(WantedSegmentBy, (await ReadHypertableAsync(connection, ct)).SegmentBy); + + await hold.RollbackAsync(ct); + } + + /* ---------------- the lock (M1) ---------------- */ + + /// + /// A long reader holds collection_log while the change is due. The ALTER must give up at its lock timeout instead + /// of waiting (and queueing every collector's insert behind it): the pass returns false, logs at Information that + /// the next hourly pass tries again, changes nothing, and leaves the connection usable. Once the reader lets go, + /// the next pass applies the change. + /// + [Fact] + public async Task AHeldLock_BoundsTheAlter_TheStoreIsUnchanged_AndTheNextPassAppliesIt() + { + var baseConnectionString = RequireLivePostgres(); + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString, ct); + using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await BuildUpgradedLogAsync(connection, scratch.ConnectionString, "collector_name", ct); + var chunksBefore = await ReadChunkSettingsAsync(connection, ct); + + await using var holder = new NpgsqlConnection(scratch.ConnectionString); + await holder.OpenAsync(ct); + var hold = await holder.BeginTransactionAsync(ct); + await ExecAsync(holder, "SELECT count(*) FROM collection_log", ct, hold); + + var logger = new CapturingTestLogger(); + bool ok; + var elapsed = Stopwatch.StartNew(); + using (var bounded = CancellationTokenSource.CreateLinkedTokenSource(ct)) + { + bounded.CancelAfter(TimeSpan.FromSeconds(60)); + ok = await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, logger, bounded.Token); + } + + elapsed.Stop(); + + Assert.False(ok, $"the pass reported success while another session held collection_log: {logger.Joined}"); + Assert.True(elapsed.Elapsed < TimeSpan.FromSeconds(30), $"the ALTER waited {elapsed.Elapsed} behind the held lock: {logger.Joined}"); + Assert.Contains(logger.Lines, line => + line.StartsWith("Information:", StringComparison.Ordinal) + && line.Contains("collection_log's compression settings", StringComparison.Ordinal) + && line.Contains("next hourly pass tries again", StringComparison.Ordinal)); + Assert.Equal(0, logger.CountAtLevel(Microsoft.Extensions.Logging.LogLevel.Warning)); + Assert.Equal(OldSegmentBy, (await ReadHypertableAsync(connection, ct)).SegmentBy); + Assert.Equal(chunksBefore, await ReadChunkSettingsAsync(connection, ct)); + Assert.Equal(1, await ScalarAsync(connection, "SELECT 1", ct)); + + await hold.RollbackAsync(ct); + await hold.DisposeAsync(); + + Assert.True(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, null, ct)); + Assert.Equal(WantedSegmentBy, (await ReadHypertableAsync(connection, ct)).SegmentBy); + } + + /* ---------------- any other failure (condition 2) ---------------- */ + + /// + /// The ALTER fails for a reason other than a lock: here the store's collector column has another name, so the new + /// segmentby names a column that does not exist. The change rolls back and leaves the store exactly as it was. The + /// log says the settings change failed and the table keeps its settings, not that "setup failed" and the table + /// "stays a plain table" (it is a hypertable). The failure does not cost the pass its policy step: the table is + /// still compressible under its current setting, and no other step adds a missing compression policy. The + /// connection stays usable for the step after it, and the next pass tries again rather than remembering the failure. + /// + [Fact] + public async Task AFailedAlter_LeavesTheStoreAsItWas_LogsWhatHappened_AndIsRetriedNextPass() + { + var baseConnectionString = RequireLivePostgres(); + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString, ct); + using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await BuildUpgradedLogAsync(connection, scratch.ConnectionString, HeldCollectorColumn, ct); + + var chunksBefore = await ReadChunkSettingsAsync(connection, ct); + Assert.NotEmpty(chunksBefore); + var dataBefore = await DataHashAsync(connection, HeldCollectorColumn, ct); + Assert.Equal(0, await CompressionPolicyCountAsync(connection, ct)); + + var logger = new CapturingTestLogger(); + Assert.False(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, logger, ct), + $"the pass reported success although the new segmentby names a column this store does not have: {logger.Joined}"); + + Assert.Equal(1, CountSettingsFailures(logger)); + Assert.DoesNotContain("plain table", logger.Joined, StringComparison.Ordinal); + Assert.Equal(OldSegmentBy, (await ReadHypertableAsync(connection, ct)).SegmentBy); + Assert.Equal(chunksBefore, await ReadChunkSettingsAsync(connection, ct)); + Assert.Equal(dataBefore, await DataHashAsync(connection, HeldCollectorColumn, ct)); + Assert.True(1 == await CompressionPolicyCountAsync(connection, ct), + $"the failed settings change skipped the policy step, so a store without a compression policy never gets one: {logger.Joined}"); + + /* The connection is not left inside a failed transaction: the next statement and the next step both run. */ + Assert.Equal(1, await ScalarAsync(connection, "SELECT 1", ct)); + await TimescaleSupport.ConvergeCompressionScheduleAsync(connection, null, ct); + + /* The next pass tries again. */ + Assert.False(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, logger, ct)); + Assert.Equal(2, CountSettingsFailures(logger)); + Assert.Equal(chunksBefore, await ReadChunkSettingsAsync(connection, ct)); + } + + /* ---------------- answers and plan ---------------- */ + + /// + /// "New chunks only" leaves old and new chunks mixed until retention removes the old ones. Every read below must + /// return the same rows over the mix as it did when all chunks had the old settings. + /// + [Fact] + public async Task OldAndNewChunks_Mixed_ReturnTheSameRowsAsBefore() + { + var baseConnectionString = RequireLivePostgres(); + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString, ct); + using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await BuildUpgradedLogAsync(connection, scratch.ConnectionString, "collector_name", ct); + + var asOf = await ScalarAsync(connection, "SELECT max(collection_time) FROM collection_log", ct); + var before = await RunReadsAsync(connection, asOf, ct); + + Assert.True(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, null, ct)); + await CompressChunksAsync(connection, olderThanDays: 1, ct); + + var segmentBys = (await ReadChunkSettingsAsync(connection, ct)).Select(chunk => chunk.SegmentBy).Distinct().ToList(); + Assert.Contains(OldChunkSegmentBy, segmentBys); + Assert.Contains(WantedChunkSegmentBy, segmentBys); + + var after = await RunReadsAsync(connection, asOf, ct); + Assert.Equal(before.Keys, after.Keys); + foreach (var (read, rows) in before) + { + Assert.True(rows == after[read], $"'{read}' returned different rows once old and new chunks were mixed"); + } + } + + /// + /// The read this change is for. The blocking baseline's coverage read (the logged CTE) asks for one + /// collector on one server; with collector_name in the segmentby, that condition must reach the compressed chunk's + /// own scan, so only that collector's batches are decompressed. Before the change only server_id reached it, + /// and the collector was filtered after every collector's rows were decompressed. + /// + [Fact] + public async Task TheEventBaselineRead_DecompressesOnlyItsCollectorsSegment() + { + var baseConnectionString = RequireLivePostgres(); + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString, ct); + using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + Assert.SkipUnless(await LiveTimescaleProbe.TryEnableAsync(scratch.ConnectionString, ct), + "TimescaleDB is not available on this cluster; the compressed-chunk plan shape needs it."); + Assert.True(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, null, ct)); + await StopBackgroundWorkersAsync(connection, ct); + await TimescaleSupport.EnsureBaselineFallbackViewsAsync(connection, null, ct); + + await SeedAsync(connection, "collector_name", new[] { ServerA }, AnalysisTime.AddDays(-35), AnalysisTime.AddMinutes(-15), ct); + await SeedEventAsync(connection, ServerA, ct); + await CompressChunksAsync(connection, olderThanDays: 1, ct); + Assert.True(await ScalarAsync(connection, + "SELECT count(*) FROM timescaledb_information.chunks WHERE hypertable_name = 'collection_log' AND is_compressed", ct) >= 1, + "the seed produced no compressed collection_log chunk, so the plan would prove nothing"); + + var plan = await ExplainBlockingBaselineAsync(connection, ServerA, ct); + var compressedScans = CompressedScans(plan).Where(scan => scan.GetProperty("Actual Loops").GetDouble() > 0).ToList(); + Assert.True(compressedScans.Count > 0, $"the plan reads no compressed chunk:\n{plan.GetRawText()}"); + Assert.True( + compressedScans.All(scan => Conditions(scan).Contains("(collector_name = ", StringComparison.Ordinal)), + $"the collector condition does not reach the compressed chunk's own scan:\n{plan.GetRawText()}"); + } + + /* ---------------- fixture ---------------- */ + + private static string RequireLivePostgres() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the collection_log segmentby tests (each mints its own scratch database)."); + return connectionString!; + } + + /// + /// Skips on a TimescaleDB these tests cannot run on. Before 2.14 a hypertable's compression settings cannot change + /// while compressed chunks exist, so the product keeps such a store on server_id by design; and a TimescaleDB + /// without the per-chunk settings view gives these tests nothing to read. Literal version, like the segmentby + /// literals above: the tests pin the floor rather than follow the product's constant. + /// + private static async Task RequirePerChunkSettingsAsync(NpgsqlConnection connection, CancellationToken ct) + { + var version = await TimescaleSupport.ReadTimescaleVersionAsync(connection, null, ct); + var hasView = await ScalarAsync(connection, + "SELECT to_regclass('timescaledb_information.chunk_compression_settings') IS NOT NULL", ct); + Assert.SkipUnless(version is not null && version >= new Version(2, 14) && hasView, + $"TimescaleDB {version?.ToString() ?? "(version unknown)"} predates per-chunk compression settings (2.14), which these tests read and change."); + } + + /// + /// collection_log as every build before #4951 left it: a hypertable compressed by server_id alone, six days + /// of runs from two servers and three collectors, and every chunk older than three days compressed. The two newest + /// closed chunks stay uncompressed, so a test can compress them under the new settings. + /// other than collector_name renames that column BEFORE the conversion, which is how the failure test makes + /// the new segmentby name a column the store does not have. + /// + private static async Task BuildUpgradedLogAsync(NpgsqlConnection connection, string connectionString, string collectorColumn, CancellationToken ct) + { + await PgMigrations.MigrateAsync(connection, ct); + + if (!string.Equals(collectorColumn, "collector_name", StringComparison.Ordinal)) + { + await ExecAsync(connection, $"ALTER TABLE collection_log RENAME COLUMN collector_name TO {collectorColumn}", ct); + } + + Assert.SkipUnless(await LiveTimescaleProbe.TryEnableAsync(connectionString, ct), "TimescaleDB is not available on this cluster."); + await StopBackgroundWorkersAsync(connection, ct); + await RequirePerChunkSettingsAsync(connection, ct); + + await ExecAsync(connection, TimescaleSupport.CreateHypertableSql(TimescaleSupport.CollectionLogTable, TimescaleSupport.CollectionLogTimeColumn), ct); + await ExecAsync(connection, "ALTER TABLE collection_log SET (timescaledb.compress, timescaledb.compress_segmentby = 'server_id')", ct); + + var now = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + await SeedAsync(connection, collectorColumn, new[] { ServerA, ServerB }, now.Date.AddDays(-6), now.AddMinutes(-15), ct); + await CompressChunksAsync(connection, olderThanDays: 3, ct); + + Assert.Equal(OldSegmentBy, (await ReadHypertableAsync(connection, ct)).SegmentBy); + } + + /// One run every 15 minutes per server and collector; every tenth run failed, so status filters matter. + private static async Task SeedAsync( + NpgsqlConnection connection, string collectorColumn, int[] servers, DateTime from, DateTime to, CancellationToken ct) + { + using var command = new NpgsqlCommand( + $@"INSERT INTO collection_log (log_id, server_id, server_name, {collectorColumn}, collection_time, status, rows_collected) + SELECT $1 - row_number() OVER (ORDER BY s.server_id, c.collector, g.t), s.server_id, $2, c.collector, g.t, + CASE WHEN (extract(epoch FROM g.t)::bigint / 900) % 10 = 0 THEN 'ERROR' ELSE 'SUCCESS' END, + (extract(epoch FROM g.t)::bigint / 900) % 7 + FROM unnest($3::integer[]) AS s(server_id) + CROSS JOIN unnest($4::text[]) AS c(collector) + CROSS JOIN generate_series($5::timestamp, $6::timestamp, INTERVAL '15 minutes') AS g(t)", connection); + command.Parameters.AddWithValue(LogIdBase); + command.Parameters.AddWithValue(ServerName); + command.Parameters.AddWithValue(servers); + command.Parameters.AddWithValue(Collectors); + command.Parameters.AddWithValue(from); + command.Parameters.AddWithValue(to); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task SeedEventAsync(NpgsqlConnection connection, int serverId, CancellationToken ct) + { + using var command = new NpgsqlCommand( + "INSERT INTO blocked_process_reports (blocked_report_id, collection_time, server_id, server_name, event_time) VALUES ($1, $2, $3, $4, $5)", + connection); + command.Parameters.AddWithValue(LogIdBase); + command.Parameters.AddWithValue(EventTime); + command.Parameters.AddWithValue(serverId); + command.Parameters.AddWithValue(ServerName); + command.Parameters.AddWithValue(EventTime); + await command.ExecuteNonQueryAsync(ct); + } + + /// Compresses every not-yet-compressed chunk that closed more than days ago. + private static async Task CompressChunksAsync(NpgsqlConnection connection, int olderThanDays, CancellationToken ct) + { + var chunks = new List(); + using (var list = new NpgsqlCommand( + "SELECT format('%I.%I', chunk_schema, chunk_name) FROM timescaledb_information.chunks " + + "WHERE hypertable_schema = 'collect' AND hypertable_name = 'collection_log' AND NOT is_compressed " + + "AND range_end <= now() - make_interval(days => $1) ORDER BY range_start", connection)) + { + list.Parameters.AddWithValue(olderThanDays); + await using var reader = await list.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + chunks.Add(reader.GetString(0)); + } + } + + foreach (var chunk in chunks) + { + await ExecAsync(connection, $"SELECT compress_chunk('{chunk}')", ct); + } + } + + private static Task StopBackgroundWorkersAsync(NpgsqlConnection connection, CancellationToken ct) => + ExecAsync(connection, "SELECT _timescaledb_functions.stop_background_workers()", ct); + + /// collection_log's compression policy jobs: one row per policy, and TimescaleDB allows at most one. + private const string CompressionPolicyJobsSql = + "FROM timescaledb_information.jobs WHERE proc_name = 'policy_compression' " + + "AND hypertable_schema = 'collect' AND hypertable_name = 'collection_log'"; + + private static Task CompressionPolicyCountAsync(NpgsqlConnection connection, CancellationToken ct) => + ScalarAsync(connection, "SELECT count(*) " + CompressionPolicyJobsSql, ct); + + /// One run of collection_log's compression policy in this session, as the scheduler runs it: it compresses + /// every chunk older than the policy's compress-after with the settings the hypertable has at that moment. + private static async Task RunCompressionPolicyAsync(NpgsqlConnection connection, CancellationToken ct) + { + int? jobId; + using (var find = new NpgsqlCommand("SELECT job_id " + CompressionPolicyJobsSql, connection)) + { + jobId = await find.ExecuteScalarAsync(ct) as int?; + } + + Assert.True(jobId.HasValue, "collection_log has no compression policy to run"); + await ExecAsync(connection, $"CALL run_job({jobId.Value})", ct); + } + + /* ---------------- reads of the store ---------------- */ + + private sealed record HypertableState(bool IsHypertable, bool CompressionEnabled, string? SegmentBy); + + private sealed record ChunkSetting(string Chunk, string SegmentBy); + + /// collection_log's own settings row: whether it is a hypertable, whether compression is on, and its + /// segmentby columns in order, joined with ", ". + private static async Task ReadHypertableAsync(NpgsqlConnection connection, CancellationToken ct) + { + using var command = new NpgsqlCommand( + @"SELECT h.compression_enabled, + (SELECT string_agg(cs.attname, ', ' ORDER BY cs.segmentby_column_index) + FROM timescaledb_information.compression_settings AS cs + WHERE cs.hypertable_schema = h.hypertable_schema + AND cs.hypertable_name = h.hypertable_name + AND cs.segmentby_column_index IS NOT NULL) + FROM timescaledb_information.hypertables AS h + WHERE h.hypertable_schema = 'collect' AND h.hypertable_name = 'collection_log'", connection); + try + { + await using var reader = await command.ExecuteReaderAsync(ct); + if (!await reader.ReadAsync(ct)) + { + return new HypertableState(false, false, null); + } + + return new HypertableState(true, !reader.IsDBNull(0) && reader.GetBoolean(0), reader.IsDBNull(1) ? null : reader.GetString(1)); + } + catch (PostgresException ex) when (ex.SqlState == PostgresErrorCodes.UndefinedTable) + { + /* No extension, so no timescaledb_information schema: not a hypertable. */ + return new HypertableState(false, false, null); + } + } + + /// Every compressed chunk of collection_log with the segmentby it was compressed under, by chunk name. + private static async Task> ReadChunkSettingsAsync(NpgsqlConnection connection, CancellationToken ct) + { + using var command = new NpgsqlCommand( + "SELECT chunk::text, coalesce(segmentby, '') FROM timescaledb_information.chunk_compression_settings " + + "WHERE hypertable = 'collect.collection_log'::regclass ORDER BY chunk::text", connection); + var chunks = new List(); + await using var reader = await command.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + chunks.Add(new ChunkSetting(reader.GetString(0), reader.GetString(1))); + } + + return chunks; + } + + /// The product's own convergence read: the collect tables whose compression is already what it wants. + private static async Task> ConvergedTablesAsync(NpgsqlConnection connection, CancellationToken ct) => + await TimescaleSupport.ReadTablesNeedingCompressionEnableAsync(connection, null, ct) + ?? throw new InvalidOperationException("the convergence read failed"); + + /// One md5 over every row, in log_id order: unchanged data gives the same hash. + private static Task DataHashAsync(NpgsqlConnection connection, string collectorColumn, CancellationToken ct) => + ScalarAsync(connection, + $"SELECT md5(string_agg(concat_ws('|', log_id, server_id, {collectorColumn}, collection_time, status, rows_collected), ',' ORDER BY log_id)) FROM collection_log", + ct); + + /// The read shapes the change can affect, each rendered to one string: every row, a per-collector window, the + /// newest run per collector, a fleet-wide aggregate, and the "any earlier success" probe. + private static async Task> RunReadsAsync(NpgsqlConnection connection, DateTime asOf, CancellationToken ct) + { + var reads = new (string Name, string Sql, object[] Args)[] + { + ("every row", + "SELECT log_id, server_id, collector_name, collection_time, status, rows_collected FROM collection_log ORDER BY log_id", + Array.Empty()), + ("one collector, 5 days", + "SELECT collection_time, status, rows_collected FROM collection_log WHERE server_id = $1 AND collector_name = 'wait_stats' AND collection_time >= $2 - INTERVAL '5 days' ORDER BY collection_time, log_id", + new object[] { ServerA, asOf }), + ("newest run per collector", + "SELECT DISTINCT ON (collector_name) collector_name, collection_time, status FROM collection_log WHERE server_id = $1 ORDER BY collector_name, collection_time DESC, log_id DESC", + new object[] { ServerA }), + ("fleet aggregate", + "SELECT server_id, collector_name, count(*), min(collection_time), max(collection_time), sum(rows_collected), count(*) FILTER (WHERE status = 'SUCCESS') FROM collection_log WHERE collection_time <= $1 GROUP BY server_id, collector_name ORDER BY server_id, collector_name", + new object[] { asOf }), + ("any earlier success", + "SELECT EXISTS (SELECT 1 FROM collection_log WHERE server_id = $1 AND collector_name = 'cpu_utilization' AND status = 'SUCCESS' AND collection_time < $2 - INTERVAL '4 days')", + new object[] { ServerA, asOf }), + }; + + var results = new Dictionary(StringComparer.Ordinal); + foreach (var (name, sql, args) in reads) + { + using var command = new NpgsqlCommand(sql, connection); + foreach (var arg in args) + { + command.Parameters.AddWithValue(arg); + } + + var text = new StringBuilder(); + await using var reader = await command.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + for (var i = 0; i < reader.FieldCount; i++) + { + text.Append(reader.IsDBNull(i) ? "NULL" : Convert.ToString(reader.GetValue(i), System.Globalization.CultureInfo.InvariantCulture)).Append('|'); + } + + text.Append('\n'); + } + + results[name] = text.ToString(); + } + + return results; + } + + /// The blocking baseline as the provider runs it, through the coverage-read plan class's own EXPLAIN helper + /// (one copy of the six bound parameters, not two). + private static Task ExplainBlockingBaselineAsync(NpgsqlConnection connection, int serverId, CancellationToken ct) => + DarlingEventBaselineCoverageReadPlanShapeLiveTests.ExplainAsync( + connection, PgBaselineProvider.GetBaselineQuery(MetricNames.Blocking)!, serverId, ct); + + /// Every scan of a chunk's compressed relation, by the coverage-read plan class's own relation test. + private static List CompressedScans(JsonElement plan) + { + var found = new List(); + + void Visit(JsonElement node) + { + if (node.TryGetProperty("Relation Name", out var relation) + && DarlingEventBaselineCoverageReadPlanShapeLiveTests.IsCompressedRelation(relation.GetString()!)) + { + found.Add(node); + } + + if (node.TryGetProperty("Plans", out var children)) + { + foreach (var child in children.EnumerateArray()) + { + Visit(child); + } + } + } + + Visit(plan); + return found; + } + + /// A scan's own conditions and its descendants', by the coverage-read plan class's helper. On the old layout + /// the compressed scan's only collector condition is a bloom filter on a metadata column, which never contains + /// "(collector_name = ". + private static string Conditions(JsonElement node) => DarlingEventBaselineCoverageReadPlanShapeLiveTests.Conditions(node); + + /// Warning lines saying the settings change failed and the table kept its settings. + private static int CountSettingsFailures(CapturingTestLogger logger) => + logger.Lines.Count(line => + line.StartsWith("Warning:", StringComparison.Ordinal) + && line.Contains("collection_log's compression settings", StringComparison.Ordinal) + && line.Contains("could not be changed, so the change was rolled back", StringComparison.Ordinal) + && line.Contains("keeps its current settings", StringComparison.Ordinal)); + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, CancellationToken ct, NpgsqlTransaction? transaction = null) + { + using var command = new NpgsqlCommand(sql, connection, transaction); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task ScalarAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + using var command = new NpgsqlCommand(sql, connection); + var value = await command.ExecuteScalarAsync(ct); + return (T)Convert.ChangeType(value!, typeof(T), System.Globalization.CultureInfo.InvariantCulture); + } +} diff --git a/Darling/Darling.Tests/CollectionLogStoreProbeAttributionTests.cs b/Darling/Darling.Tests/CollectionLogStoreProbeAttributionTests.cs index ea45dc3458..7d7fdf3f0e 100644 --- a/Darling/Darling.Tests/CollectionLogStoreProbeAttributionTests.cs +++ b/Darling/Darling.Tests/CollectionLogStoreProbeAttributionTests.cs @@ -50,6 +50,7 @@ namespace Darling.Tests; /// every persisted column exactly as it was. These tests pin the arithmetic, the emit, the caveats on the two /// surfaces that drew the wrong inference, and the premise the FLOOR caveat rests on. /// +[Trait("Reads", "Lite")] public class CollectionLogStoreProbeAttributionTests { /// diff --git a/Darling/Darling.Tests/CollectionLogWebRowPreservesFullTextPinTests.cs b/Darling/Darling.Tests/CollectionLogWebRowPreservesFullTextPinTests.cs index 8469ef6dc5..8e197a523b 100644 --- a/Darling/Darling.Tests/CollectionLogWebRowPreservesFullTextPinTests.cs +++ b/Darling/Darling.Tests/CollectionLogWebRowPreservesFullTextPinTests.cs @@ -18,6 +18,7 @@ namespace Darling.Tests; /// old default through the row" contract #3897's trend tools pin for TrendBudget.Chart. No rig: a /// source-text pin, like PgCappedReadSurfaceTests. /// +[Trait("Stage", "Guard")] public sealed class CollectionLogWebRowPreservesFullTextPinTests { private const string WebEndpoints = diff --git a/Darling/Darling.Tests/CollectionNoteLatestRunTests.cs b/Darling/Darling.Tests/CollectionNoteLatestRunTests.cs index 0d59b51183..646c5cedb7 100644 --- a/Darling/Darling.Tests/CollectionNoteLatestRunTests.cs +++ b/Darling/Darling.Tests/CollectionNoteLatestRunTests.cs @@ -22,6 +22,7 @@ namespace Darling.Tests; /// window's. A note that carries counts is labelled as the latest run's and takes no run-count qualifier; /// a plain note keeps its qualifiers exactly. /// +[Trait("Reads", "Lite")] public class CollectionNoteLatestRunTests { private const string Counts = "shred_gated=2 events_read=0 report_xml_empty=0 report_xml_unparsed=0 events_stored=0"; diff --git a/Darling/Darling.Tests/CollectionOutputBesideCostTests.cs b/Darling/Darling.Tests/CollectionOutputBesideCostTests.cs index 9ef3911fef..b1d449b70b 100644 --- a/Darling/Darling.Tests/CollectionOutputBesideCostTests.cs +++ b/Darling/Darling.Tests/CollectionOutputBesideCostTests.cs @@ -42,6 +42,7 @@ namespace Darling.Tests; /// deadlock_coverage established one level up: numbers, a denominator, a named cause, and a sentence /// that names both windows and disclaims the one it did not measure. /// +[Trait("Reads", "Lite")] public sealed class CollectionOutputBesideCostTests { /* The measured pg_deadlocks counts, so every row below is the shape this issue was filed on rather @@ -432,7 +433,9 @@ public void BothSkusReaders_MapTheSameOrdinals_ToTheSameFields() "public static async Task> GetCollectionHealthAsync"); var lite = OrdinalMap( Path.Combine("Lite", "Services", "LocalDataService.CollectionHealth.cs"), - "public async Task> GetCollectionHealthAsync"); + /* #5371 moved the positional reads out of the public GetCollectionHealthAsync (now the memo, + cancellation and phase-timing wrapper) into ReadCollectionHealthAsync, which maps the row. */ + "private async Task> ReadCollectionHealthAsync"); /* The precondition. 31 columns since #4748's one (latest_run_note, the newest run's note the band reads) - 30 at #3885's, 16 at #2460, plus #2472's four fan-out diff --git a/Darling/Darling.Tests/CollectorDatabaseScopeRungTests.cs b/Darling/Darling.Tests/CollectorDatabaseScopeRungTests.cs index 3eb84e8d55..6a49003986 100644 --- a/Darling/Darling.Tests/CollectorDatabaseScopeRungTests.cs +++ b/Darling/Darling.Tests/CollectorDatabaseScopeRungTests.cs @@ -42,6 +42,7 @@ namespace Darling.Tests; /// really about being NEWEST — keeping a copy of those would assert this rung is still the top, /// which is how the NEXT rung's build goes red. /// +[Trait("Reads", "Lite")] public sealed class CollectorDatabaseScopeRungTests { private const int RungVersion = 125; @@ -473,9 +474,11 @@ public void BothProviders_SpliceTheScopeIntoTheDatabaseListPlan_AndTheMaintenanc /// /// The runner feeds ONE resolution per run to all three consumers — the dispatch probe, the /// per-database loop's list plan, and the enumeration context — and the stated boundaries hold: - /// the worker's delegate reads the SAME live overrides the cadence gate reads; the Azure XE - /// session provisioner deliberately passes NO scope (session inventory is server-shaped — three - /// collectors with three possible scopes share those sessions); and Lite's definition runner + /// the worker's delegate reads the SAME live overrides the cadence gate reads; the always-on Azure XE + /// session provisioner deliberately passes NO scope (session inventory is server-shaped — one list + /// provisions the deadlock and blocked-process sessions, whose collectors may carry different scopes), while the opt-in long-query trace + /// follows its own collector's scope through the runner's accessor (its lifecycle covers what its read + /// covers); and Lite's definition runner /// never names the scope at all, so every Lite query stays byte-identical to the unscoped form. /// [Fact] @@ -494,7 +497,8 @@ public void TheRunnerResolvesOnceAndFeedsAllThreeConsumers_AndTheStatedBoundarie worker, StringComparison.Ordinal); var xe = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "DarlingXeSessions.cs"); - Assert.Equal(2, CountOf(xe, "GetAzureDatabaseListAsync(server, databaseScope: null")); + Assert.Equal(1, CountOf(xe, "GetAzureDatabaseListAsync(server, databaseScope: null")); + Assert.Contains("runner.DatabaseScopeFor(LongQueryCompletionsCollector.Instance.Name, server.ServerId)", xe, StringComparison.Ordinal); var lite = RepoFile.ReadRepoFile("Lite", "Services", "RemoteCollectorService.DefinitionRunner.cs"); Assert.DoesNotContain("DatabaseScope", lite, StringComparison.Ordinal); diff --git a/Darling/Darling.Tests/CollectorFaultStackLogTests.cs b/Darling/Darling.Tests/CollectorFaultStackLogTests.cs new file mode 100644 index 0000000000..42f3488df6 --- /dev/null +++ b/Darling/Darling.Tests/CollectorFaultStackLogTests.cs @@ -0,0 +1,202 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor Lite. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Net.Http; +using System.Threading; +using System.Threading.Tasks; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Targets; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5003: a failed collector run logged only the exception's message, so "Collection was modified; enumeration +/// operation may not execute" could not say which collection or which line. The first failure of each kind from each +/// collector now carries its full text; the repeats keep the one line. +/// +public class CollectorFaultStackLogTests +{ + private static Exception Thrown(Exception exception) + { + try + { + throw exception; + } + catch (Exception caught) + { + return caught; + } + } + + [Fact] + public void TheFirstFailureOfAKindCarriesTheTypeTheMessageAndTheFrameThatThrew() + { + var log = new CollectorFaultStackLog(); + + var text = log.TakeFirst("pg_log_events", Thrown(new InvalidOperationException("Collection was modified; enumeration operation may not execute."))); + + Assert.NotNull(text); + Assert.Contains("System.InvalidOperationException", text, StringComparison.Ordinal); + Assert.Contains("Collection was modified", text, StringComparison.Ordinal); + Assert.Contains(nameof(Thrown), text, StringComparison.Ordinal); + } + + [Fact] + public void ARepeatOfTheSameKindFromTheSameCollectorGetsNothing() + { + var log = new CollectorFaultStackLog(); + + Assert.NotNull(log.TakeFirst("pg_log_events", Thrown(new InvalidOperationException("one")))); + Assert.Null(log.TakeFirst("pg_log_events", Thrown(new InvalidOperationException("two")))); + Assert.Null(log.TakeFirst("pg_log_events", Thrown(new InvalidOperationException("three")))); + } + + [Fact] + public void AnotherCollectorOrAnotherTypeIsAnotherKind() + { + var log = new CollectorFaultStackLog(); + + Assert.NotNull(log.TakeFirst("pg_log_events", Thrown(new InvalidOperationException("x")))); + Assert.NotNull(log.TakeFirst("pg_deadlocks", Thrown(new InvalidOperationException("x")))); + Assert.NotNull(log.TakeFirst("pg_log_events", Thrown(new ArgumentException("x")))); + } + + /// + /// An RDS log read wraps every fault in . A network fault that came first + /// must not use up the stack for a bug inside the read that comes later, so the type underneath is part of the kind. + /// + [Fact] + public void AWrapperOverADifferentCauseIsAnotherKind() + { + var log = new CollectorFaultStackLog(); + + var network = new RdsLogUnavailableException("unreachable", false, new HttpRequestException("unreachable")); + var bug = new RdsLogUnavailableException( + "Collection was modified; enumeration operation may not execute.", false, Thrown(new InvalidOperationException("Collection was modified; enumeration operation may not execute."))); + + Assert.NotNull(log.TakeFirst("pg_log_events", Thrown(network))); + Assert.NotNull(log.TakeFirst("pg_log_events", Thrown(bug))); + Assert.Null(log.TakeFirst("pg_log_events", Thrown(bug))); + + var text = new CollectorFaultStackLog().TakeFirst("pg_log_events", Thrown(bug)); + Assert.Contains("System.InvalidOperationException", text, StringComparison.Ordinal); + } + + /// The failure arm is where an out-of-memory run lands, and building the text allocates. + [Fact] + public void AnOutOfMemoryFailureGetsNoStackAndUsesUpNothing() + { + var log = new CollectorFaultStackLog(); + + Assert.Null(log.TakeFirst("wait_stats", new OutOfMemoryException())); + Assert.Null(log.TakeFirst("wait_stats", new InsufficientMemoryException())); + Assert.Null(log.TakeFirst("wait_stats", new OutOfMemoryException())); + } + + /// + /// The RDS ingestors wrap every fault that is not a cancellation in , so an + /// out-of-memory read reaches the failure arm one level down. It takes the one-line arm like a bare one, wherever it + /// sits under the wrapper, and a wrapper over any other cause still gets its stack. + /// + [Fact] + public void AnOutOfMemoryUnderAWrapperGetsNoStackEither() + { + var log = new CollectorFaultStackLog(); + + var wrapped = new RdsLogUnavailableException("The log read failed.", false, new OutOfMemoryException()); + var deeper = new InvalidOperationException("outer", new RdsLogUnavailableException("The log read failed.", false, new InsufficientMemoryException())); + var outerOverOther = new OutOfMemoryException("outer", new HttpRequestException("unreachable")); + + Assert.Null(log.TakeFirst("pg_log_events", Thrown(wrapped))); + Assert.Null(log.TakeFirst("pg_log_events", Thrown(wrapped))); + Assert.Null(log.TakeFirst("pg_deadlocks", Thrown(deeper))); + Assert.Null(log.TakeFirst("pg_plan_capture", Thrown(outerOverOther))); + + var network = new RdsLogUnavailableException("unreachable", false, new HttpRequestException("unreachable")); + + Assert.NotNull(log.TakeFirst("pg_log_events", Thrown(network))); + } + + /// An exception whose own throws, as a type with a faulty override could. + private sealed class UnrenderableException : Exception + { + public UnrenderableException() : base("this exception cannot render itself") { } + + public override string ToString() => throw new InvalidOperationException("ToString failed"); + } + + /// + /// The kind is claimed after the text is built. When building it throws, the kind still logs once, with a short line + /// that names the exception and what threw, and the second failure of the kind gets nothing. + /// + [Fact] + public void AnExceptionWhoseTextCannotBeBuiltStillLogsItsKindOnceWithAShortLine() + { + var log = new CollectorFaultStackLog(); + + var first = log.TakeFirst("pg_log_events", Thrown(new UnrenderableException())); + + Assert.NotNull(first); + Assert.Contains(nameof(UnrenderableException), first, StringComparison.Ordinal); + Assert.Contains("System.InvalidOperationException", first, StringComparison.Ordinal); + Assert.DoesNotContain(Environment.NewLine, first, StringComparison.Ordinal); + + Assert.Null(log.TakeFirst("pg_log_events", Thrown(new UnrenderableException()))); + Assert.NotNull(log.TakeFirst("pg_deadlocks", Thrown(new UnrenderableException()))); + } + + [Fact] + public async Task EightRunsFailingTogetherInOneWayYieldExactlyOneStack() + { + var log = new CollectorFaultStackLog(); + using var start = new Barrier(8); + + var results = await Task.WhenAll(Enumerable.Range(0, 8).Select(i => Task.Factory.StartNew( + () => + { + start.SignalAndWait(); + return log.TakeFirst("pg_log_events", Thrown(new InvalidOperationException("run " + i))); + }, + CancellationToken.None, + TaskCreationOptions.LongRunning, + TaskScheduler.Default))).WaitAsync(TimeSpan.FromSeconds(30)); + + Assert.Equal(1, results.Count(text => text is not null)); + } + + /// + /// The wiring, pinned from source because the failure arm needs a store to run. The one line every failure logs is + /// unchanged (StoreCopyPhaseTests pins its shape), the full text is a second line written only for the first + /// failure of a kind, and the timeout arm does not take it: its filter admits only a command timeout from the driver, + /// so a stack there would be driver frames, and the authored sentence already names everything that matters. + /// + [Fact] + public void OnlyTheGeneralFailureArmWritesTheFullTextAndItIsTheFirstOfAKindOnly() + { + var worker = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "DarlingWorker.cs"); + + Assert.Single(System.Text.RegularExpressions.Regex.Matches(worker, @"_collectorFaultStacks\.TakeFirst\(collectorName, ex\)")); + + var errorLine = worker.IndexOf("=> ERROR: {Message}\",", StringComparison.Ordinal); + var detail = worker.IndexOf("_collectorFaultStacks.TakeFirst(collectorName, ex)", StringComparison.Ordinal); + var dropRuntime = worker.IndexOf("ConnectionFaultDisposition.ShouldDropRuntimeAsync(", detail, StringComparison.Ordinal); + var timeoutArm = worker.IndexOf("=> ERROR (timeout): {Message}", StringComparison.Ordinal); + + Assert.True(timeoutArm > 0 && timeoutArm < errorLine, "The timeout arm comes before the general arm."); + Assert.True(errorLine < detail && detail < dropRuntime, "The full text is written after the one line and before the connection check."); + + /* The timeout arm and everything up to the general arm's own line: no full text there. */ + Assert.DoesNotContain("TakeFirst", worker[timeoutArm..errorLine], StringComparison.Ordinal); + + /* Same exception object, not a copy, and a LogError so the level is the arm's own. */ + Assert.Contains("_logger.LogError(ex,", worker[detail..dropRuntime], StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/CollectorHealthClassifierTests.cs b/Darling/Darling.Tests/CollectorHealthClassifierTests.cs index 79d5aff1dd..3c6f0664c5 100644 --- a/Darling/Darling.Tests/CollectorHealthClassifierTests.cs +++ b/Darling/Darling.Tests/CollectorHealthClassifierTests.cs @@ -196,7 +196,7 @@ public void Classify_LongDormantCollector_WithNoRecentActivityOfAnyKind_IsStoppe [InlineData("DarlingDataReader.CollectionHealthSql")] [InlineData("DarlingFleetReader.FleetCollectionHealthSql")] [InlineData("ViewerDataService.CollectionHealthSql")] - [InlineData("ViewerDataService.FleetCollectionHealthSql")] + [InlineData("ViewerDataService.FleetCollectionHealthByServerSql")] public void EveryBandingRead_SelectsTheAbandonedCount_SoNoSurfaceBandsOnADefaultedZero(string which) { var sql = which switch @@ -204,7 +204,7 @@ public void EveryBandingRead_SelectsTheAbandonedCount_SoNoSurfaceBandsOnADefault "DarlingDataReader.CollectionHealthSql" => DarlingDataReader.CollectionHealthSql, "DarlingFleetReader.FleetCollectionHealthSql" => DarlingFleetReader.FleetCollectionHealthSql, "ViewerDataService.CollectionHealthSql" => ViewerDataService.CollectionHealthSql, - "ViewerDataService.FleetCollectionHealthSql" => ViewerDataService.FleetCollectionHealthSql, + "ViewerDataService.FleetCollectionHealthByServerSql" => ViewerDataService.FleetCollectionHealthByServerSql, _ => throw new ArgumentOutOfRangeException(nameof(which), which, "unmapped banding read"), }; @@ -240,7 +240,7 @@ passed by matching nothing at all cannot hide here. */ [InlineData("DarlingDataReader.CollectionHealthSql")] [InlineData("DarlingFleetReader.FleetCollectionHealthSql")] [InlineData("ViewerDataService.CollectionHealthSql")] - [InlineData("ViewerDataService.FleetCollectionHealthSql")] + [InlineData("ViewerDataService.FleetCollectionHealthByServerSql")] public void EveryBandingRead_SelectsTheExtensionMissingCount_SoNoSurfaceBandsOnADefaultedZero(string which) { var sql = which switch @@ -248,7 +248,7 @@ public void EveryBandingRead_SelectsTheExtensionMissingCount_SoNoSurfaceBandsOnA "DarlingDataReader.CollectionHealthSql" => DarlingDataReader.CollectionHealthSql, "DarlingFleetReader.FleetCollectionHealthSql" => DarlingFleetReader.FleetCollectionHealthSql, "ViewerDataService.CollectionHealthSql" => ViewerDataService.CollectionHealthSql, - "ViewerDataService.FleetCollectionHealthSql" => ViewerDataService.FleetCollectionHealthSql, + "ViewerDataService.FleetCollectionHealthByServerSql" => ViewerDataService.FleetCollectionHealthByServerSql, _ => throw new ArgumentOutOfRangeException(nameof(which), which, "unmapped banding read"), }; @@ -562,17 +562,17 @@ public void EveryBandingRow_BandsWarning_OnTheNewestRunsNote_AndNotOnAnOlderOne( } /// Every read that builds a banding row selects the newest run's note, at the ordinal its mapper - /// reads; the viewer's fleet read gets it from plain aggregates (no window function), and the sentence its + /// reads (the per-server reads from the newest-run lookup, #4955, the status gate on the row it returns); + /// the viewer's fleet read gets it from plain aggregates (no window function), and the sentence its /// LIKE looks for is the one the writer writes. [Fact] public void EveryDarlingHealthRead_ProjectsTheNewestRunsNote() { - Assert.Contains("recency_rank = 1 AND status = 'SUCCESS' THEN error_message END) AS latest_run_note", DarlingDataReader.CollectionHealthSql, StringComparison.Ordinal); - Assert.Contains("AS latest_run_note", ViewerDataService.CollectionHealthSql, StringComparison.Ordinal); - Assert.Contains("AS recency_rank", ViewerDataService.CollectionHealthSql, StringComparison.Ordinal); - Assert.Contains("AS latest_run_note", ViewerDataService.FleetCollectionHealthSql, StringComparison.Ordinal); - Assert.DoesNotContain("ROW_NUMBER()", ViewerDataService.FleetCollectionHealthSql, StringComparison.Ordinal); - Assert.Contains(CollectionHealthRollupSupport.LatestRunNoteRawSql, ViewerDataService.FleetCollectionHealthSql, StringComparison.Ordinal); + Assert.Contains("CASE WHEN newest.status = 'SUCCESS' THEN newest.error_message END AS latest_run_note", DarlingDataReader.CollectionHealthSql, StringComparison.Ordinal); + Assert.Contains("CASE WHEN newest.status = 'SUCCESS' THEN newest.error_message END AS latest_run_note", ViewerDataService.CollectionHealthSql, StringComparison.Ordinal); + Assert.Contains("AS latest_run_note", ViewerDataService.FleetCollectionHealthByServerSql, StringComparison.Ordinal); + Assert.DoesNotContain("ROW_NUMBER()", ViewerDataService.FleetCollectionHealthByServerSql, StringComparison.Ordinal); + Assert.Contains(CollectionHealthRollupSupport.LatestRunNoteRawSql, ViewerDataService.FleetCollectionHealthByServerSql, StringComparison.Ordinal); Assert.Contains($"'%{PartialDatabaseFailureNote.Marker}%'", CollectionHealthRollupSupport.LatestRunNoteRawSql, StringComparison.Ordinal); Assert.Contains(PartialDatabaseFailureNote.Marker, PartialDatabaseFailureNote.Format, StringComparison.Ordinal); } diff --git a/Darling/Darling.Tests/CollectorMeasurementSeamTests.cs b/Darling/Darling.Tests/CollectorMeasurementSeamTests.cs index 0010bf6e5e..08a5fa8099 100644 --- a/Darling/Darling.Tests/CollectorMeasurementSeamTests.cs +++ b/Darling/Darling.Tests/CollectorMeasurementSeamTests.cs @@ -47,6 +47,7 @@ namespace Darling.Tests; /// counts as SEPARATE pins, because a distinctness check over the combined message would pass on the /// numbers rather than on the verdicts. /// +[Trait("Reads", "Lite")] public class CollectorMeasurementSeamTests { /* A rendered note may contain NOTHING but label=value tokens separated by single spaces. Anchored at @@ -379,9 +380,11 @@ A quiet server and a server whose every captured report we failed to parse both MakeReader(new[] { null, "", ReportXml }), mixed, CancellationToken.None); Assert.Single(mixedRows); - Assert.Equal( - "events_read=3 report_xml_empty=1 report_xml_unparsed=1 events_stored=1", - CollectorMeasurementNote.Render(mixed.Measurements)); + /* #4348: the good report's statements went through the statement filter, so a cycle that judged a value + also carries the four statement_scrub_* measurements, after the collector's own. */ + var mixedNote = CollectorMeasurementNote.Render(mixed.Measurements); + Assert.StartsWith("events_read=3 report_xml_empty=1 report_xml_unparsed=1 events_stored=1 ", mixedNote, StringComparison.Ordinal); + Assert.Contains("statement_scrub_named=0 statement_scrub_timeouts=0 statement_scrub_unjudged=0 statement_scrub_ms=", mixedNote, StringComparison.Ordinal); /* The one that matters: the ring buffer delivered events and NOT ONE became a row. Indistinguishable from the quiet server above until this seam existed. */ @@ -584,6 +587,13 @@ this cycle. Both collectors declare their own const (same "shred_gated" string, from its own trailing result set), so only one need be listed here for the distinct-label set this assertion actually checks. */ BlockedProcessReportCollector.ShredGatedMeasurement, + /* #4348: the statement filter's four counters, measured by CollectorContext itself (the sessions a + collector opens with BeginStatementScrub), which is why the consts live in CollectorContext.cs + beside the four .Measure( calls. Written only when the cycle judged a value. */ + CollectorContext.StatementScrubNamedMeasurement, + CollectorContext.StatementScrubTimeoutsMeasurement, + CollectorContext.StatementScrubUnjudgedMeasurement, + CollectorContext.StatementScrubMsMeasurement, }.OrderBy(l => l, StringComparer.Ordinal).ToList(), resolved.Distinct(StringComparer.Ordinal).OrderBy(l => l, StringComparer.Ordinal).ToList()); } diff --git a/Darling/Darling.Tests/CollectorMemoryKnobTests.cs b/Darling/Darling.Tests/CollectorMemoryKnobTests.cs index 3944423895..23c67c52ad 100644 --- a/Darling/Darling.Tests/CollectorMemoryKnobTests.cs +++ b/Darling/Darling.Tests/CollectorMemoryKnobTests.cs @@ -135,6 +135,8 @@ making it public just to observe it. */ lockField.SetValue(worker, new object()); var loggerField = typeof(DarlingWorker).GetField("_logger", BindingFlags.NonPublic | BindingFlags.Instance)!; loggerField.SetValue(worker, NullLogger.Instance); + typeof(DarlingWorker).GetField("_collectorFaultStacks", BindingFlags.NonPublic | BindingFlags.Instance)! + .SetValue(worker, new CollectorFaultStackLog()); var reconcile = typeof(DarlingWorker).GetMethod("ReconcileSweepGate", BindingFlags.NonPublic | BindingFlags.Instance)!; var absorbed = typeof(DarlingWorker).GetField("_gateAbsorbed", BindingFlags.NonPublic | BindingFlags.Instance)!; diff --git a/Darling/Darling.Tests/CollectorRunTimeCliVerbTests.cs b/Darling/Darling.Tests/CollectorRunTimeCliVerbTests.cs new file mode 100644 index 0000000000..f2263fc974 --- /dev/null +++ b/Darling/Darling.Tests/CollectorRunTimeCliVerbTests.cs @@ -0,0 +1,485 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// The --set-collector-run-at verb (#4938), without a store: its spelling, its strict grammar, the command the +/// verb stands in for and the plan the executor makes of it (so the verb owns no SQL), the usage text, and the line +/// the read-back prints for a row that carries a run time. The verb run end to end is +/// . +/// +public sealed class CollectorRunTimeCliVerbTests +{ + private const string Verb = "--set-collector-run-at"; + + private static bool Parse(string[] rest, out string? collector, out string? runAt, out string? server, out string? config, out string? error) => + DarlingCliCommands.TryParseCollectorRunAtArgs(Verb, rest, out collector, out runAt, out server, out config, out error); + + [Fact] + public void TheVerb_IsRecognisedBySpelling_CaseInsensitively_AndNoOtherVerbIs() + { + Assert.True(DarlingCliCommands.IsSetCollectorRunAtVerb("--set-collector-run-at")); + Assert.True(DarlingCliCommands.IsSetCollectorRunAtVerb("--SET-COLLECTOR-RUN-AT")); + Assert.False(DarlingCliCommands.IsSetCollectorRunAtVerb("--set-collector-run")); + Assert.False(DarlingCliCommands.IsSetCollectorRunAtVerb("--enable-collector")); + Assert.False(DarlingCliCommands.IsEnableCollectorVerb("--set-collector-run-at")); + Assert.False(DarlingCliCommands.IsDisableCollectorVerb("--set-collector-run-at")); + } + + [Fact] + public void TheGrammar_IsACollectorAndATime_WithAnOptionalServerAndConfig_InAnyOrder() + { + Assert.True(Parse(new[] { "index_object_stats", "02:00" }, out var collector, out var runAt, out var server, out var config, out var error)); + Assert.Equal("index_object_stats", collector); + Assert.Equal("02:00", runAt); + Assert.Null(server); + Assert.Null(config); + Assert.Null(error); + + Assert.True(Parse(new[] { "--server", "sql01", "index_object_stats", "none", "--config", "darling.json" }, out collector, out runAt, out server, out config, out error)); + Assert.Equal("index_object_stats", collector); + Assert.Equal("none", runAt); + Assert.Equal("sql01", server); + Assert.Equal("darling.json", config); + + Assert.True(Parse(new[] { "index_object_stats", "default", "--server", "sql01" }, out _, out runAt, out server, out _, out _)); + Assert.Equal("default", runAt); + Assert.Equal("sql01", server); + } + + [Theory] + [InlineData(new string[0], "needs a collector name")] + [InlineData(new[] { "index_object_stats" }, "needs a time")] + [InlineData(new[] { "index_object_stats", "02:00", "extra" }, "takes ONE collector name and ONE time")] + [InlineData(new[] { "index_object_stats", "02:00", "--server" }, "--server needs a server name")] + [InlineData(new[] { "index_object_stats", "02:00", "--server", "--config", "x" }, "--server needs a server name")] + [InlineData(new[] { "index_object_stats", "02:00", "--config" }, "--config needs a path")] + [InlineData(new[] { "index_object_stats", "02:00", "--bogus" }, "Unknown option")] + public void TheGrammar_RefusesAnythingElse_NamingTheProblem(string[] rest, string expected) + { + Assert.False(Parse(rest, out _, out _, out _, out _, out var error)); + Assert.Contains(expected, error, StringComparison.Ordinal); + Assert.Contains(Verb, error, StringComparison.Ordinal); + } + + [Fact] + public void TheUsageText_NamesTheThreeValues_TheScopes_AndTheCollectors() + { + var text = DarlingCliCommands.CollectorRunAtUsageText(); + + Assert.Contains("--set-collector-run-at [--server ]", text, StringComparison.Ordinal); + Assert.Contains("HH:MM", text, StringComparison.Ordinal); + Assert.Contains("none", text, StringComparison.Ordinal); + Assert.Contains("default", text, StringComparison.Ordinal); + Assert.Contains("server time", text, StringComparison.Ordinal); + Assert.Contains("index_object_stats", text, StringComparison.Ordinal); + } + + private static CommandPlan Plan(string collector, string runAt, int? serverId) => + DarlingCliCommands.PlanCollectorRunAt(collector, runAt, serverId); + + [Fact] + public void TheVerbsCommand_IsTheCommandPlanesSetCollectorRunAt_Row() + { + var command = DarlingCliCommands.BuildCollectorRunAtCommand("index_object_stats", "02:00", 7); + + Assert.Equal("set_collector_run_at", command.CommandType); + Assert.Equal(7, command.TargetServerId); + Assert.Equal("cli", command.RequestedBy); + using var args = JsonDocument.Parse(command.ArgsJson!); + Assert.Equal("index_object_stats", args.RootElement.GetProperty("collector_name").GetString()); + Assert.Equal("02:00", args.RootElement.GetProperty("run_at").GetString()); + + /* The verb's plan is the executor's plan for that command, nothing else: one write path. */ + var viaExecutor = DarlingCommandExecutor.ResolvePlan(command); + var viaVerb = Plan("index_object_stats", "02:00", 7); + Assert.Equal(viaExecutor.Kind, viaVerb.Kind); + Assert.Equal(viaExecutor.Sql, viaVerb.Sql); + Assert.Equal(viaExecutor.Parameters, viaVerb.Parameters); + } + + private const string FleetUpsert = + "INSERT INTO config.config_collector_run_times (server_id, collector_name, run_at_minute) VALUES (NULL, $1, $2) " + + "ON CONFLICT (collector_name) WHERE server_id IS NULL DO UPDATE SET run_at_minute = EXCLUDED.run_at_minute"; + + private const string ServerUpsert = + "INSERT INTO config.config_collector_run_times (server_id, collector_name, run_at_minute) VALUES ($1, $2, $3) " + + "ON CONFLICT (server_id, collector_name) WHERE server_id IS NOT NULL DO UPDATE SET run_at_minute = EXCLUDED.run_at_minute"; + + [Fact] + public void ATimeOrNone_UpsertsOneRunTimeRow_AndTouchesNoScheduleRow() + { + var fleet = Plan("index_object_stats", "02:00", serverId: null); + Assert.Equal(CommandKind.StoreWrite, fleet.Kind); + Assert.Equal(FleetUpsert, fleet.Sql); + Assert.Equal(new object?[] { "index_object_stats", (short)120 }, fleet.Parameters); + Assert.Equal("collector run time set (fleet-wide)", fleet.SuccessStatus); + + var server = Plan("index_object_stats", "02:00", serverId: 7); + Assert.Equal(ServerUpsert, server.Sql); + Assert.Equal(new object?[] { 7, "index_object_stats", (short)120 }, server.Parameters); + Assert.Equal("collector run time set", server.SuccessStatus); + + /* None on a server is -1 in the same upsert. */ + var none = Plan("index_object_stats", "none", serverId: 7); + Assert.Equal(ServerUpsert, none.Sql); + Assert.Equal(new object?[] { 7, "index_object_stats", (short)-1 }, none.Parameters); + + /* The run time has its own table: no plan names a schedule row or its enabled flag, so a collector that ships OFF + is never switched on as a side effect, and the schedule rows are neither read nor written. */ + foreach (var plan in new[] { fleet, server, none, Plan("long_query_completions", "02:00", serverId: null) }) + { + Assert.DoesNotContain("config_collector_schedules", plan.Sql, StringComparison.Ordinal); + Assert.DoesNotContain("enabled", plan.Sql, StringComparison.Ordinal); + } + } + + [Fact] + public void NoneOnTheFleet_AndDefault_DeleteTheScopesRow_WithoutCreatingOne() + { + const string FleetDelete = "DELETE FROM config.config_collector_run_times WHERE server_id IS NULL AND collector_name = $1"; + const string ServerDelete = "DELETE FROM config.config_collector_run_times WHERE server_id = $1 AND collector_name = $2"; + + /* The fleet has no time to stop, so none there is the same as default: the row goes. */ + foreach (var word in new[] { "none", "default", "Default", "NONE" }) + { + var fleet = Plan("index_object_stats", word, serverId: null); + Assert.Equal(FleetDelete, fleet.Sql); + Assert.Equal(new object?[] { "index_object_stats" }, fleet.Parameters); + Assert.Equal("collector run time cleared (fleet-wide)", fleet.SuccessStatus); + } + + var server = Plan("index_object_stats", "default", serverId: 7); + Assert.Equal(ServerDelete, server.Sql); + Assert.Equal(new object?[] { 7, "index_object_stats" }, server.Parameters); + Assert.Equal("collector run time cleared", server.SuccessStatus); + } + + [Theory] + [InlineData("24:00")] + [InlineData("2:5")] + [InlineData("noon")] + [InlineData("")] + public void ABadTime_IsRefusedByThePlan_WithTheSharedText(string runAt) + { + var plan = Plan("index_object_stats", runAt, serverId: null); + + Assert.Equal(CommandKind.Fail, plan.Kind); + Assert.Equal(CollectorRunTime.InvalidRunAtMessage, plan.FailReason); + } + + [Fact] + public void AnUnknownCollector_AndAMissingArgument_AreRefusedByThePlan() + { + Assert.Equal(CommandKind.Fail, Plan("not_a_collector", "02:00", serverId: null).Kind); + Assert.Contains("unknown collector", Plan("not_a_collector", "02:00", serverId: null).FailReason, StringComparison.Ordinal); + + Assert.Equal(CommandKind.Fail, DarlingCommandExecutor.ResolvePlan( + new ClaimedCommand(0, "set_collector_run_at", null, "{\"collector_name\":\"index_object_stats\"}", "cli")).Kind); + Assert.Equal(CommandKind.Fail, DarlingCommandExecutor.ResolvePlan( + new ClaimedCommand(0, "set_collector_run_at", null, "{\"run_at\":\"02:00\"}", "cli")).Kind); + } + + [Fact] + public void TheReadBack_PrintsTheRunTimeRows_UnderTheirOwnHeading_AndNothingForACollectorWithNone() + { + var schedule = new[] { new DarlingCliCommands.CollectorScheduleReadbackRow(null, "index_object_stats", null, null, true, null, null) }; + + /* No run times: the read-back is exactly the schedule read-back it always was. */ + var plain = DarlingCliCommands.FormatCollectorScheduleRows("index_object_stats", schedule); + Assert.DoesNotContain(plain, l => l.Contains("run_at=", StringComparison.Ordinal) || l.Contains("Run times", StringComparison.Ordinal)); + Assert.Equal(plain, DarlingCliCommands.FormatCollectorScheduleRows( + "index_object_stats", schedule, Array.Empty())); + + var runTimes = new[] + { + new DarlingCliCommands.CollectorRunTimeReadbackRow(null, "index_object_stats", 120, null), + new DarlingCliCommands.CollectorRunTimeReadbackRow(7, "index_object_stats", -1, "sql01"), + new DarlingCliCommands.CollectorRunTimeReadbackRow(8, "index_object_stats", 195, null), + }; + + /* A run time needs no schedule row, so the run-time lines follow the "no override rows" line as well. */ + var lines = DarlingCliCommands.FormatCollectorScheduleRows("index_object_stats", Array.Empty(), runTimes); + Assert.Contains(lines, l => l.Contains("(none", StringComparison.Ordinal)); + var from = lines.ToList().IndexOf("Run times in the store for index_object_stats:"); + Assert.True(from >= 0, "no run-time heading"); + Assert.Equal(" fleet-wide: run_at=02:00 server time", lines[from + 1]); + Assert.Equal(" sql01 (server_id 7): run_at=none (no fixed time on this server)", lines[from + 2]); + Assert.Equal(" server_id 8 (not in the servers registry): run_at=03:15 server time", lines[from + 3]); + } + + [Fact] + public void TheScheduleReadBack_NamesNoRunTime_AndTheRunTimeReadBackReadsItsOwnTable() + { + Assert.DoesNotContain("run_at_minute", DarlingCliCommands.CollectorScheduleReadbackSql, StringComparison.Ordinal); + Assert.Contains("FROM config.config_collector_run_times rt", DarlingCliCommands.CollectorRunTimeReadbackSql, StringComparison.Ordinal); + Assert.Contains("lower(rt.collector_name) = lower($1)", DarlingCliCommands.CollectorRunTimeReadbackSql, StringComparison.Ordinal); + Assert.Contains("run_at_minute", DarlingCliCommands.CollectorRunTimeReadbackSql, StringComparison.Ordinal); + foreach (var write in new[] { "INSERT", "UPDATE", "DELETE" }) + { + Assert.DoesNotContain(write, DarlingCliCommands.CollectorRunTimeReadbackSql, StringComparison.OrdinalIgnoreCase); + } + } + + [Fact] + public void TheMissingTableMessage_NamesTheTable_AndSaysToStartTheServiceOnce() + { + Assert.Equal( + "The store has not been upgraded to the run-time table yet. Start the service once to migrate it, then run --set-collector-run-at again.", + DarlingCliCommands.RunTimeTableMissingMessage("--set-collector-run-at")); + } +} + +/// +/// --set-collector-run-at run end to end (#4938) through +/// with a bring-your-own darling.json pointed at a scratch store: parse, plan, connect, check, write, read back, +/// print. It asserts the rows the service will resolve, that the verb touches only the run time, and that a bad time or +/// a collector that runs more often than daily is refused with the shared text and a non-zero exit code. +/// +/* #1776 own-store: each fact mints its own scratch database through ScratchPostgres and never touches the shared + store's tables, so it cannot race the live collection and serializing it would be pure slowdown. */ +public sealed class CollectorRunTimeCliVerbLiveTests +{ + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private const string SkipText = "Set DARLING_TEST_PG to a Postgres connection string to run the run-time verb end to end (each fact mints its own scratch database)."; + + private const string Daily = "index_object_stats"; + private const int Sql01 = 4101; + private const int Sql02 = 4102; + + private sealed class Rig : IAsyncDisposable + { + public required ScratchPostgres Scratch { get; init; } + public required NpgsqlConnection Connection { get; init; } + public required string ConfigPath { get; init; } + public required string Directory { get; init; } + + public async ValueTask DisposeAsync() + { + await Connection.DisposeAsync(); + await Scratch.DisposeAsync(); + try { System.IO.Directory.Delete(Directory, recursive: true); } catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) { } + } + } + + private static async Task OpenAsync(CancellationToken ct) + { + var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var directory = Path.Combine(Path.GetTempPath(), "darling-runat-" + Guid.NewGuid().ToString("N")); + System.IO.Directory.CreateDirectory(directory); + var configPath = Path.Combine(directory, "darling.json"); + File.WriteAllText(configPath, $$""" + { + "postgres": { + "managed": false, + "connectionString": {{JsonSerializer.Serialize(scratch.ConnectionString)}} + }, + "servers": [] + } + """); + + foreach (var (id, name) in new[] { (Sql01, "sql01"), (Sql02, "sql02") }) + { + await ExecAsync(connection, + "INSERT INTO collect.servers (server_id, server_name, display_name, is_enabled, created_date, modified_date) " + + $"VALUES ({id}, '{name}', '{name}', TRUE, now() AT TIME ZONE 'UTC', now() AT TIME ZONE 'UTC')", ct); + } + + return new Rig { Scratch = scratch, Connection = connection, ConfigPath = configPath, Directory = directory }; + } + + private static async Task<(int Exit, string Output, string Error)> RunAsync(Rig rig, CancellationToken ct, params string[] args) + { + var output = new StringWriter(); + var error = new StringWriter(); + var exit = await DarlingCliCommands.SetCollectorRunAtAsync(args.Concat(new[] { "--config", rig.ConfigPath }).ToArray(), output, error, ct); + return (exit, output.ToString(), error.ToString()); + } + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task ScalarAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + return await command.ExecuteScalarAsync(ct); + } + + private static string Scope(int? serverId) => serverId is int id ? $"server_id = {id}" : "server_id IS NULL"; + + /// Every stored run time as one text: fleet/collector=minute or serverId/collector=minute. + private static async Task RunTimesAsync(NpgsqlConnection connection, CancellationToken ct) => + (string)(await ScalarAsync(connection, + "SELECT COALESCE(string_agg(COALESCE(server_id::text, 'fleet') || '/' || collector_name || '=' || run_at_minute, ',' ORDER BY server_id NULLS FIRST, collector_name), '') FROM config.config_collector_run_times", ct))!; + + private static async Task ScheduleRowsAsync(NpgsqlConnection connection, CancellationToken ct) => + Convert.ToInt64(await ScalarAsync(connection, "SELECT COUNT(*) FROM config.config_collector_schedules", ct)); + + [Fact] + public async Task TheFleetRunTime_IsWrittenToItsTable_TheNameIsCanonicalised_AndTheRowsAreReadBack() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + + var bodySucceeded = false; + try + { + var (exit, output, error) = await RunAsync(rig, ct, "Index_Object_Stats", "02:00"); + Assert.True(exit == 0, $"exit {exit}; stderr: {error}"); + Assert.Equal(string.Empty, error); + Assert.Equal($"fleet/{Daily}=120", await RunTimesAsync(rig.Connection, ct)); + Assert.Equal(0L, await ScheduleRowsAsync(rig.Connection, ct)); + Assert.Contains("Run times in the store for index_object_stats:", output, StringComparison.Ordinal); + Assert.Contains("fleet-wide: run_at=02:00 server time", output, StringComparison.Ordinal); + + /* A second time replaces the row; default and none clear it. */ + Assert.Equal(0, (await RunAsync(rig, ct, Daily, "03:30")).Exit); + Assert.Equal($"fleet/{Daily}=210", await RunTimesAsync(rig.Connection, ct)); + Assert.Equal(0, (await RunAsync(rig, ct, Daily, "default")).Exit); + Assert.Equal("", await RunTimesAsync(rig.Connection, ct)); + Assert.Equal(0, (await RunAsync(rig, ct, Daily, "none")).Exit); + Assert.Equal("", await RunTimesAsync(rig.Connection, ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(rig.Scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + [Fact] + public async Task AServerRunTime_IsItsOwnRow_NoneIsMinusOne_DefaultDeletesIt_AndTheServersScheduleRowIsUntouched() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + + var bodySucceeded = false; + try + { + await ExecAsync(rig.Connection, + $"INSERT INTO config.config_collector_schedules (server_id, collector_name, frequency_minutes, retention_days, enabled) VALUES ({Sql01}, '{Daily}', 2880, 45, FALSE)", ct); + const string Schedule = "SELECT frequency_minutes || '|' || retention_days || '|' || enabled FROM config.config_collector_schedules WHERE server_id = " + "4101"; + + Assert.Equal(0, (await RunAsync(rig, ct, Daily, "02:00", "--server", "sql01")).Exit); + Assert.Equal($"{Sql01}/{Daily}=120", await RunTimesAsync(rig.Connection, ct)); + Assert.Equal(0, (await RunAsync(rig, ct, Daily, "01:00")).Exit); + Assert.Equal(0, (await RunAsync(rig, ct, Daily, "none", "--server", "sql01")).Exit); + Assert.Equal($"fleet/{Daily}=60,{Sql01}/{Daily}=-1", await RunTimesAsync(rig.Connection, ct)); + + Assert.Equal(0, (await RunAsync(rig, ct, Daily, "default", "--server", "sql01")).Exit); + Assert.Equal($"fleet/{Daily}=60", await RunTimesAsync(rig.Connection, ct)); + + /* The schedule row the server already had is exactly as it was: the verb never wrote it. */ + Assert.Equal("2880|45|false",(string)(await ScalarAsync(rig.Connection, Schedule, ct))!); + Assert.Equal(1L, await ScheduleRowsAsync(rig.Connection, ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(rig.Scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + [Fact] + public async Task ABadTimeAnHourlyCollectorAnUnknownNameOrServer_AreRefused_WithANonZeroExit_AndNothingWritten() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + + var bodySucceeded = false; + try + { + await ExecAsync(rig.Connection, + $"INSERT INTO config.config_collector_schedules (server_id, collector_name, frequency_minutes) VALUES (NULL, '{Daily}', 60)", ct); + + var bad = await RunAsync(rig, ct, Daily, "25:00"); + Assert.Equal(1, bad.Exit); + Assert.Contains(CollectorRunTime.InvalidRunAtMessage, bad.Error, StringComparison.Ordinal); + + var hourly = await RunAsync(rig, ct, Daily, "02:00"); + Assert.Equal(1, hourly.Exit); + Assert.Contains(CollectorRunTime.IntervalRefusalMessage(Daily, 60), hourly.Error, StringComparison.Ordinal); + + Assert.Equal(1, (await RunAsync(rig, ct, "no_such_collector", "02:00")).Exit); + Assert.Equal(1, (await RunAsync(rig, ct, "server_properties", "02:00", "--server", "no_such_server")).Exit); + Assert.Equal("", await RunTimesAsync(rig.Connection, ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(rig.Scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + /// A store the service has not migrated to the run-time table yet (below V160). The verb says so in plain words, + /// exits 1 and writes nothing, for a time, none and default alike; and the toggle verbs' read-back reads such a store as having + /// no run times, so it does not fail after a write that committed. + [Theory] + [InlineData("02:00", null)] + [InlineData("none", "sql01")] + [InlineData("default", null)] + public async Task OnAStoreWithoutTheRunTimeTable_TheVerbRefusesWithAPlainMessage_ExitsOne_AndWritesNothing(string runAt, string? server) + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + + var bodySucceeded = false; + try + { + await ExecAsync(rig.Connection, "DROP TABLE config.config_collector_run_times", ct); + var scheduleRowsBefore = await ScheduleRowsAsync(rig.Connection, ct); + + var args = server is null ? new[] { Daily, runAt } : new[] { Daily, runAt, "--server", server }; + var (exit, _, error) = await RunAsync(rig, ct, args); + + Assert.Equal(1, exit); + Assert.Contains(DarlingCliCommands.RunTimeTableMissingMessage("--set-collector-run-at"), error, StringComparison.Ordinal); + Assert.Contains("Nothing was changed.", error, StringComparison.Ordinal); + Assert.DoesNotContain("relation", error, StringComparison.OrdinalIgnoreCase); + Assert.Equal(scheduleRowsBefore, await ScheduleRowsAsync(rig.Connection, ct)); + + await using var dataSource = NpgsqlDataSource.Create(rig.Scratch.ConnectionString); + Assert.Empty(await DarlingCliCommands.ReadCollectorRunTimeRowsAsync(dataSource, Daily, ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(rig.Scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } +} diff --git a/Darling/Darling.Tests/CollectorRunTimeRungTests.cs b/Darling/Darling.Tests/CollectorRunTimeRungTests.cs new file mode 100644 index 0000000000..65ca3d1c76 --- /dev/null +++ b/Darling/Darling.Tests/CollectorRunTimeRungTests.cs @@ -0,0 +1,1036 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Reflection; +using System.Text.RegularExpressions; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// Pins the Darling rung that gives a collector an optional run time (#4938): config.config_collector_run_times, +/// a table of its own, because the viewer's schedule Save deletes and re-inserts the schedules table's rows with a fixed +/// column list and a column there would be cleared by every Save from a viewer that does not know it. A row holds minutes +/// after midnight on the monitored server's clock, from 0 to 1439, fleet-wide when server_id is NULL; -1 is allowed +/// on a server row only and means "no fixed time on this server", which stops a fleet-wide time. No row means no run +/// time. This file holds the rung (ladder, SQL, viewer probe, the service's read and merge) and the resolution of the +/// value, including the warning for a run time that cannot apply. Every fact finds the rung by NAME, so a renumber is one +/// edit to the registration, the constant and the viewer's return. +/// +public sealed class CollectorRunTimeRungTests +{ + public const string RungName = "collector-run-time"; + + private const string Table = "config_collector_run_times"; + + /// This rung's sentinel; the ordinal is a fact of the probe's shape. A newer rung's sentinel follows it. + private const int ProbeOrdinal = 135; + + private const int ServerId = 41; + + public static int RungVersion => Rung.Version; + + private static PgMigrations.Migration Rung => PgMigrations.Scripts.Single(m => m.Name == RungName); + + [Fact] + public void TheRungIsRegisteredInADenseLadder_AtVersion160() + { + var versions = PgMigrations.Scripts.Select(s => s.Version).ToList(); + + /* No longer the top rung: V161 (the Query Store top daily summary) landed above it. */ + Assert.Equal(160, Rung.Version); + Assert.True(Rung.Version < StorageVersion.SchemaVersion); + Assert.Equal(StorageVersion.SchemaVersion, PgMigrations.Scripts[^1].Version); + Assert.Equal(versions.Distinct().OrderBy(v => v), versions); + Assert.Contains(Rung.Version - 1, versions); + } + + /// The rung creates the one table, its two partial unique indexes and the reload trigger, schema-qualified, + /// and every statement is guarded so a second run changes nothing. It names no column of the schedules table and + /// touches no row. + [Fact] + public void TheRungCreatesTheRunTimesTable_IdempotentlyAndSchemaQualified_AndTouchesNothingElse() + { + var sql = Rung.Sql.Replace("\r\n", "\n", StringComparison.Ordinal); + + Assert.Contains($"CREATE TABLE IF NOT EXISTS config.{Table} (", sql, StringComparison.Ordinal); + Assert.Contains("run_at_minute smallint NOT NULL", sql, StringComparison.Ordinal); + Assert.Contains("CHECK (run_at_minute BETWEEN 0 AND 1439 OR (run_at_minute = -1 AND server_id IS NOT NULL))", sql, StringComparison.Ordinal); + Assert.Contains( + $"CREATE UNIQUE INDEX IF NOT EXISTS ux_{Table}_fleet\n ON config.{Table} (collector_name) WHERE server_id IS NULL;", + sql, StringComparison.Ordinal); + Assert.Contains( + $"CREATE UNIQUE INDEX IF NOT EXISTS ux_{Table}_server\n ON config.{Table} (server_id, collector_name) WHERE server_id IS NOT NULL;", + sql, StringComparison.Ordinal); + Assert.Contains($"DROP TRIGGER IF EXISTS trg_bump_collector_run_times ON config.{Table};", sql, StringComparison.Ordinal); + Assert.Contains("AFTER INSERT OR UPDATE OR DELETE ON config." + Table, sql, StringComparison.Ordinal); + Assert.Contains("FOR EACH STATEMENT EXECUTE FUNCTION config.config_bump_version();", sql, StringComparison.Ordinal); + + /* The schedules table is not touched at all: no ALTER, no column of it, no row of anything. */ + Assert.DoesNotContain("config_collector_schedules", sql, StringComparison.Ordinal); + Assert.False( + Regex.IsMatch(sql, @"^\s*(ALTER TABLE|GRANT|UPDATE|DELETE FROM|INSERT INTO|TRUNCATE|DROP TABLE|DROP COLUMN)\b", RegexOptions.Multiline), + "the rung may only create the table, its indexes and its trigger"); + + /* Every CREATE of a table or index carries IF NOT EXISTS and the one DROP carries IF EXISTS. */ + foreach (Match create in Regex.Matches(sql, @"^CREATE (TABLE|UNIQUE INDEX|INDEX)\b.*$", RegexOptions.Multiline)) + { + Assert.Contains("IF NOT EXISTS", create.Value, StringComparison.Ordinal); + } + + foreach (Match drop in Regex.Matches(sql, @"^DROP\b.*$", RegexOptions.Multiline)) + { + Assert.Contains("IF EXISTS", drop.Value, StringComparison.Ordinal); + } + } + + /// The viewer's schedule statements, the ones its Save runs, never name the run time: that is what keeps a + /// released viewer's Save from touching it, and what the next viewer's Save must keep doing until it carries the time + /// on purpose. + [Fact] + public void TheViewersScheduleStatements_NeverNameTheRunTime() + { + foreach (var sql in new[] + { + ViewerDataService.CollectorSchedulesSelectSql, + ViewerDataService.CollectorScheduleFleetUpsertSql, + ViewerDataService.CollectorScheduleServerUpsertSql, + ViewerDataService.CollectorScheduleDeleteFleetScopeSql, + ViewerDataService.CollectorScheduleDeleteServerScopeSql, + ViewerDataService.CollectorScheduleDeleteAllServerScopesSql, + }) + { + Assert.DoesNotContain("run_at_minute", sql, StringComparison.Ordinal); + Assert.DoesNotContain(Table, sql, StringComparison.Ordinal); + } + } + + [Fact] + public void TheProbeCarriesTheTableAsAnArm_AndMapsAStoreThroughTheRungToTheRung() + { + var probe = ViewerDataService.StoreSchemaProbeSql.Replace("\r\n", "\n", StringComparison.Ordinal); + var arm = $"to_regclass('config.{Table}') IS NOT NULL"; + Assert.Contains(arm, probe, StringComparison.Ordinal); + Assert.True(probe.IndexOf(arm, StringComparison.Ordinal) < probe.IndexOf("collect.query_store_top_daily", StringComparison.Ordinal), + "the newer rung's sentinel follows this one"); + + var viewer = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.cs"); + Assert.Contains($"reader.GetBoolean({ProbeOrdinal})", viewer, StringComparison.Ordinal); + + var method = typeof(ViewerDataService).GetMethod("MapProbedSchemaVersion", BindingFlags.NonPublic | BindingFlags.Static)!; + var arity = method.GetParameters().Length; + Assert.True(ProbeOrdinal < arity - 1, "a newer rung's sentinel follows this one"); + Assert.Equal("hasCollectorRunAt", method.GetParameters()[ProbeOrdinal].Name); + + var all = Enumerable.Range(0, arity).Select(i => (object)(i <= ProbeOrdinal)).ToArray(); + Assert.Equal(160, (int)method.Invoke(null, all)!); + + var behind = (object[])all.Clone(); + behind[ProbeOrdinal] = false; + Assert.Equal(159, (int)method.Invoke(null, behind)!); + + var nextArm = viewer.IndexOf("if (hasQueryStoreTopDaily)", StringComparison.Ordinal); + var thisArm = viewer.IndexOf("if (hasCollectorRunAt)", StringComparison.Ordinal); + var previousArm = viewer.IndexOf("if (hasInstallIdTableOid)", StringComparison.Ordinal); + Assert.True(thisArm >= 0, "no sentinel arm: a store at this rung would map one rung short"); + Assert.True(nextArm >= 0 && nextArm < thisArm, "the newer rung's arm sits above this one's"); + Assert.True(thisArm < previousArm, "this arm sits below the newer rung's and above the previous one's"); + Assert.Contains("return 160;", viewer[thisArm..previousArm], StringComparison.Ordinal); + + /* The V71 finding: the table is named in the probe line and nowhere in the arm's prose. The comment block sits + ABOVE the `if`, and this probe line has no information_schema in it, so the prose must not repeat the name. */ + var armProseStart = viewer.LastIndexOf("/* V160 (#4938)", thisArm, StringComparison.Ordinal); + Assert.True(armProseStart >= 0, "the arm has no comment block saying why it exists"); + Assert.DoesNotContain(Table, viewer[armProseStart..thisArm], StringComparison.Ordinal); + } + + /// The Collector Schedules window's Save is ONE call in ONE transaction (#4938): it never commits the run times as a + /// transaction of their own ahead of the schedule rows, because a schedule write that then fails would leave the run time + /// saved. The window calls the combined write once and none of the three single writes, and the combined write's scope writer + /// runs the run-time statements on the connection and transaction it opened for the schedule rows, ahead of their delete. + [Fact] + public void TheWindowsSave_IsOneCallInOneTransaction_AndNeverCommitsTheRunTimesOnItsOwn() + { + var window = CSharpSourceWalker.StripCommentsAndStrings( + RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "CollectorScheduleEditorWindow.xaml.cs")); + Assert.Single(Regex.Matches(window, @"\.SaveCollectorScheduleAsync\(")); + Assert.DoesNotContain(".SaveCollectorRunTimesAsync(", window, StringComparison.Ordinal); + Assert.DoesNotContain(".ReplaceFleetSchedulesAsync(", window, StringComparison.Ordinal); + Assert.DoesNotContain(".ReplaceServerSchedulesAsync(", window, StringComparison.Ordinal); + + var viewer = CSharpSourceWalker.StripCommentsAndStrings( + RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.CollectorSchedules.cs")); + var start = viewer.IndexOf("private async Task ReplaceScheduleScopeAsync(", StringComparison.Ordinal); + Assert.True(start >= 0, "the schedule scope writer is gone"); + var scope = viewer[start..viewer.IndexOf("private static void AddNullableInt", start, StringComparison.Ordinal)]; + Assert.Single(Regex.Matches(scope, @"BeginTransactionAsync\(")); + var runTimeWrite = scope.IndexOf("WriteRunTimeChangesAsync(connection, transaction, runTimeChanges", StringComparison.Ordinal); + Assert.True(runTimeWrite >= 0, "the scope writer does not write the run times on its own connection and transaction"); + Assert.True( + runTimeWrite < scope.IndexOf("CollectorScheduleDeleteFleetScopeSql", StringComparison.Ordinal), + "the run-time statements must run ahead of the schedule delete, in the same transaction"); + } + + /// A schedule Reset deletes that scope's run times in the same transaction as its schedule rows (#4938). The window's + /// "Reset to Defaults" arms the scope clear and its Save passes it on, "Apply Default to All Servers" is the one call that + /// resets both tables (and the window has no run-time reset of its own), and that call opens exactly one transaction and + /// deletes the schedule rows ahead of the run times. + [Fact] + public void AScheduleReset_DeletesTheScopesRunTimes_InTheSameTransactionAsItsScheduleRows() + { + var window = CSharpSourceWalker.StripCommentsAndStrings( + RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "CollectorScheduleEditorWindow.xaml.cs")); + var reset = window[window.IndexOf("private void ResetDefaults_Click(", StringComparison.Ordinal)..window.IndexOf("private void CopyFromServer_Click(", StringComparison.Ordinal)]; + Assert.Contains("_resetToDefaults = true;", reset, StringComparison.Ordinal); + /* #4938: the Save's plan (CollectorScheduleOverlay.BuildSavePlan) carries the clear: the window hands it the armed Reset and + "Use default schedule", the plan clears on either (unless the run-time read failed), and the window sends what the plan holds. */ + Assert.Contains("usesDefault, _resetToDefaults, schedulesRead:", window, StringComparison.Ordinal); + Assert.Contains( + "var clearRunTimes = runTimesRead && (resetToDefaults || usesDefault);", + CSharpSourceWalker.StripCommentsAndStrings(RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "CollectorScheduleOverlay.cs")), + StringComparison.Ordinal); + Assert.Contains(".SaveCollectorScheduleAsync(_scopeServerId, plan.Rows, plan.RunTimeChanges, plan.ClearRunTimes)", window, StringComparison.Ordinal); + Assert.Single(Regex.Matches(window, @"\.ResetAllServerSchedulesAsync\(")); + Assert.DoesNotContain("ResetAllServerRunTimes", window, StringComparison.Ordinal); + + var viewer = CSharpSourceWalker.StripCommentsAndStrings( + RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.CollectorSchedules.cs")); + Assert.DoesNotContain("ResetAllServerRunTimes", viewer, StringComparison.Ordinal); + var start = viewer.IndexOf("public async Task ResetAllServerSchedulesAsync(", StringComparison.Ordinal); + Assert.True(start >= 0, "the all-servers reset is gone"); + var body = viewer[start..viewer.IndexOf("private static async Task DeleteRunTimesAsync(", start, StringComparison.Ordinal)]; + Assert.Single(Regex.Matches(body, @"BeginTransactionAsync\(")); + Assert.True( + body.IndexOf("CollectorScheduleDeleteAllServerScopesSql", StringComparison.Ordinal) + < body.IndexOf("CollectorRunTimeDeleteAllServerScopesSql", StringComparison.Ordinal), + "the schedule rows are deleted first, so a failed run-time delete takes that delete back with it"); + Assert.Contains("transaction.CommitAsync(", body, StringComparison.Ordinal); + } + + [Fact] + public void TheServiceReadsTheSchedulesWithoutARunTimeColumn_AndTheRunTimesFromTheirOwnTable() + { + var service = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "StoreConfigProvider.cs"); + + Assert.Contains( + "\"SELECT server_id, collector_name, frequency_minutes, retention_days, enabled, databases FROM config_collector_schedules\"", + service, StringComparison.Ordinal); + Assert.DoesNotContain("run_at_minute FROM config_collector_schedules", service, StringComparison.Ordinal); + + Assert.Equal( + $"SELECT server_id, collector_name, run_at_minute FROM config.{Table} ORDER BY server_id NULLS FIRST, collector_name", + StoreConfigProvider.RunTimesSelectSql); + Assert.Contains("reader.GetInt16(2)", service, StringComparison.Ordinal); + + /* A store at V159 has no table: that read, and only that one, answers 42P01 with no run times. */ + Assert.Contains("catch (PostgresException ex) when (ex.SqlState == PostgresErrorCodes.UndefinedTable)", service, StringComparison.Ordinal); + } + + /* ---- the layering (pure) --------------------------------------------------------------------------- */ + + private static ScheduleOverride Server(string collector, int? runAt, int? frequency = null) => + new(ServerId, collector, frequency, null, true, null, runAt); + + private static ScheduleOverride Fleet(string collector, int? runAt, int? frequency = null) => + new(null, collector, frequency, null, true, null, runAt); + + /// The server value wins over the fleet value; -1 on either row is "none" and on the server row stops + /// the fleet value; NULL falls through per column; a value outside the CHECK's range is treated as not set; a + /// run time on an hourly or per-minute effective interval resolves to none; a daily, a two-day and an on-load + /// collector keep theirs (the on-load collectors re-run every 1440 minutes). + [Fact] + public void ResolveSchedule_LayersTheRunTimePerColumn_AndOnlyAWholeDayIntervalKeepsIt() + { + const string Daily = "index_object_stats"; + const string OnLoad = "server_properties"; + var table = new (string Name, string Collector, ScheduleOverride[] Rows, int? Expected)[] + { + ("no rows", Daily, Array.Empty(), null), + ("fleet value", Daily, new[] { Fleet(Daily, 120) }, 120), + ("server value", Daily, new[] { Server(Daily, 130) }, 130), + ("server wins over fleet", Daily, new[] { Server(Daily, 130), Fleet(Daily, 120) }, 130), + ("server -1 stops the fleet value", Daily, new[] { Server(Daily, -1), Fleet(Daily, 120) }, null), + ("fleet -1 is none", Daily, new[] { Fleet(Daily, -1) }, null), + ("server NULL falls through", Daily, new[] { Server(Daily, null), Fleet(Daily, 120) }, 120), + ("both NULL", Daily, new[] { Server(Daily, null), Fleet(Daily, null) }, null), + ("midnight is a time", Daily, new[] { Fleet(Daily, 0) }, 0), + ("last minute of the day", Daily, new[] { Fleet(Daily, 1439) }, 1439), + ("server value out of range falls through", Daily, new[] { Server(Daily, 1440), Fleet(Daily, 120) }, 120), + ("below -1 is not set", Daily, new[] { Server(Daily, -2) }, null), + ("hourly default has none", "pg_buffer_usage", new[] { Fleet("pg_buffer_usage", 120) }, null), + ("per-minute default has none", "wait_stats", new[] { Fleet("wait_stats", 120), Server("wait_stats", 90) }, null), + ("server frequency override makes it hourly", Daily, new[] { Server(Daily, null, 60), Fleet(Daily, 120) }, null), + ("two-day interval keeps it", Daily, new[] { Server(Daily, 90, 2880) }, 90), + ("on-load collector keeps it", OnLoad, new[] { Fleet(OnLoad, 150) }, 150), + ("explicit on-load frequency keeps it", OnLoad, new[] { Server(OnLoad, 150, 0) }, 150), + ("a row for another collector is not read", Daily, new[] { Fleet(OnLoad, 150) }, null), + }; + + foreach (var (name, collector, rows, expected) in table) + { + var resolved = StoreConfigProvider.ResolveSchedule(collector, ServerId, rows); + Assert.True(expected == resolved.RunAtMinute, $"{name}: expected {expected?.ToString() ?? "none"} but resolved {resolved.RunAtMinute?.ToString() ?? "none"}"); + } + } + + /// A dropped run time leaves the cadence alone: the frequency is what the override says, as it was + /// before the column existed. + [Fact] + public void ADroppedRunTime_DoesNotChangeTheFrequency() + { + var resolved = StoreConfigProvider.ResolveSchedule("wait_stats", ServerId, new[] { Server("wait_stats", 90, 15) }); + + Assert.Null(resolved.RunAtMinute); + Assert.Equal(15, resolved.FrequencyMinutes); + } + + /* ---- the warning (once per load, judged per server) -------------------------------------------------- */ + + private static MonitoredServer Target(int id, string name, string engine = "sqlserver") => + new() { Name = name, Host = name + "-host", Engine = engine, StoredServerId = id }; + + [Fact] + public void AFleetRunTime_IsJudgedPerServer_SoOnlyAServerWithItsOwnHourlyFrequencyIsNamed() + { + var servers = new[] { Target(41, "alpha"), Target(42, "bravo") }; + var overrides = new[] + { + Fleet("index_object_stats", 120), + new ScheduleOverride(42, "index_object_stats", 60, null, true), + }; + + var dropped = Assert.Single(StoreConfigProvider.FindDroppedRunTimes(servers, overrides)); + + Assert.Equal("index_object_stats", dropped.CollectorName); + Assert.Equal(42, dropped.ServerId); + Assert.Equal("bravo", dropped.ServerName); + Assert.Equal(120, dropped.RunAtMinute); + Assert.Equal(60, dropped.IntervalMinutes); + Assert.False(dropped.FromServerRow); + } + + [Fact] + public void AServerRowRunTimeOnAnHourlyCollector_IsNamedWithItsRow_AndAnEngineThatNeverRunsItIsSkipped() + { + var servers = new[] { Target(41, "alpha"), Target(43, "pgone", "postgres") }; + + var serverRow = Assert.Single(StoreConfigProvider.FindDroppedRunTimes(servers, new[] { Server("wait_stats", 120) })); + Assert.Equal(41, serverRow.ServerId); + Assert.True(serverRow.FromServerRow); + + /* A fleet row for a PostgreSQL-only collector names the PostgreSQL server, not the SQL Server one that + never runs it. */ + var fleetRow = Assert.Single(StoreConfigProvider.FindDroppedRunTimes(servers, new[] { Fleet("pg_buffer_usage", 120) })); + Assert.Equal(43, fleetRow.ServerId); + } + + [Fact] + public void NothingIsDropped_ForANoneValue_ADailyCollector_OrAnOnLoadCollector() + { + var servers = new[] { Target(41, "alpha") }; + var overrides = new[] + { + Server("wait_stats", -1), + Fleet("index_object_stats", 120), + Fleet("server_properties", 150), + Fleet("wait_stats", null), + }; + + Assert.Empty(StoreConfigProvider.FindDroppedRunTimes(servers, overrides)); + } + + /// One warning per dropped run time, naming the collector, the server and the time; none when nothing is + /// dropped. It logs at the load, not in ResolveSchedule, which runs every sweep and stays silent. + [Fact] + public void TheWarning_NamesTheCollectorAndTheServer_AndResolveScheduleItselfLogsNothing() + { + var servers = new[] { Target(41, "alpha") }; + var overrides = new[] { Server("wait_stats", 120) }; + var logger = new CapturingTestLogger(); + + StoreConfigProvider.LogDroppedRunTimes(logger, servers, overrides); + + var line = Assert.Single(logger.Lines); + Assert.Equal(1, logger.CountAtLevel(LogLevel.Warning)); + Assert.Contains("wait_stats", line, StringComparison.Ordinal); + Assert.Contains("alpha", line, StringComparison.Ordinal); + Assert.Contains("02:00", line, StringComparison.Ordinal); + + for (var sweep = 0; sweep < 3; sweep++) + { + _ = StoreConfigProvider.ResolveSchedule("wait_stats", 41, overrides); + } + + Assert.Single(logger.Lines); + + var quiet = new CapturingTestLogger(); + StoreConfigProvider.LogDroppedRunTimes(quiet, servers, new[] { Server("wait_stats", -1) }); + Assert.Empty(quiet.Lines); + StoreConfigProvider.LogDroppedRunTimes(null, servers, overrides); + } + + /// A stored value the table's CHECK would refuse (a store whose CHECK was dropped by hand) resolves to none, + /// and the load says so: one warning for each, naming the collector, the scope and the value. A value the CHECK allows + /// (0, 1439, and -1 on a server) is never reported, and a sweep that only resolves logs nothing. + [Fact] + public void TheWarning_NamesAValueTheTableWouldRefuse_WithTheCollectorTheScopeAndTheValue() + { + var servers = new[] { Target(41, "alpha") }; + var refused = new[] + { + Fleet("index_object_stats", 1440), + Server("server_properties", -2), + Fleet("server_properties", 0), + Server("index_object_stats", 1439), + }; + var logger = new CapturingTestLogger(); + + StoreConfigProvider.LogDroppedRunTimes(logger, servers, refused); + + Assert.Equal(2, logger.CountAtLevel(LogLevel.Warning)); + var fleet = Assert.Single(logger.Lines, l => l.Contains("index_object_stats", StringComparison.Ordinal)); + Assert.Contains("fleet-wide", fleet, StringComparison.Ordinal); + Assert.Contains("1440", fleet, StringComparison.Ordinal); + var server = Assert.Single(logger.Lines, l => l.Contains("server_properties", StringComparison.Ordinal)); + Assert.Contains("server id 41", server, StringComparison.Ordinal); + Assert.Contains("-2", server, StringComparison.Ordinal); + + /* A sweep that only resolves logs nothing more. */ + for (var sweep = 0; sweep < 3; sweep++) + { + _ = StoreConfigProvider.ResolveSchedule("server_properties", ServerId, refused); + } + + Assert.Equal(2, logger.Lines.Count); + + var quiet = new CapturingTestLogger(); + StoreConfigProvider.LogDroppedRunTimes(quiet, servers, new[] + { + Fleet("index_object_stats", 0), + Server("index_object_stats", -1), + Fleet("server_properties", 1439), + }); + Assert.Empty(quiet.Lines); + } + + /* ---- the merge of the run-time rows onto the schedule rows (pure) ------------------------------------- */ + + private static RunTimeOverride RunTime(int? serverId, string collector, int minute) => new(serverId, collector, minute); + + [Fact] + public void MergeRunTimes_CarriesARunTimeOnItsScheduleRow_AndKeepsEveryOtherColumnOfTheRow() + { + var scope = new[] { "alpha_db" }; + var fleetRow = new ScheduleOverride(null, "index_object_stats", 1440, 30, false, scope); + var serverRow = new ScheduleOverride(ServerId, "index_object_stats", 2880, null, true); + + var merged = StoreConfigProvider.MergeRunTimes( + new[] { fleetRow, serverRow }, + new[] { RunTime(null, "index_object_stats", 120), RunTime(ServerId, "index_object_stats", -1) }); + + Assert.Equal(2, merged.Count); + Assert.Equal(fleetRow with { RunAtMinute = 120 }, Assert.Single(merged, o => o.ServerId is null)); + Assert.Equal(serverRow with { RunAtMinute = -1 }, Assert.Single(merged, o => o.ServerId == ServerId)); + } + + /// A run time with no schedule row makes an entry that sets nothing else. In particular its enabled state is + /// not set, so it can neither switch on a collector that is off by default nor switch off one the fleet row enabled, + /// and every other column resolves exactly as it did without the entry. + [Fact] + public void MergeRunTimes_GivesARunTimeWithNoScheduleRowAnEntryThatSetsNothingElse() + { + var merged = StoreConfigProvider.MergeRunTimes( + Array.Empty(), + new[] { RunTime(null, "index_object_stats", 120), RunTime(ServerId, "server_properties", 150) }); + + Assert.Equal(2, merged.Count); + foreach (var entry in merged) + { + Assert.Null(entry.FrequencyMinutes); + Assert.Null(entry.RetentionDays); + Assert.Null(entry.Enabled); + Assert.Null(entry.Databases); + } + + Assert.Equal(120, StoreConfigProvider.ResolveSchedule("index_object_stats", ServerId, merged).RunAtMinute); + Assert.Equal(150, StoreConfigProvider.ResolveSchedule("server_properties", ServerId, merged).RunAtMinute); + Assert.Null(StoreConfigProvider.ResolveSchedule("server_properties", ServerId + 1, merged).RunAtMinute); + } + + [Fact] + public void ARunTimeEntry_NeverChangesEnabledFrequencyRetentionOrScope_AtAnyLevel() + { + var rows = new[] + { + new ScheduleOverride(null, "index_object_stats", 1440, 7, false, new[] { "alpha_db" }), + new ScheduleOverride(null, "pg_index_usage_stats", null, 9, true), + }; + var runTimes = new[] + { + RunTime(ServerId, "index_object_stats", 130), + RunTime(null, "long_query_completions", 100), + RunTime(ServerId, "long_query_completions", 110), + RunTime(null, "pg_index_usage_stats", 140), + RunTime(null, "wait_stats", 150), + }; + var merged = StoreConfigProvider.MergeRunTimes(rows, runTimes); + + foreach (var collector in new[] { "index_object_stats", "pg_index_usage_stats", "long_query_completions", "wait_stats", "server_properties" }) + { + var before = StoreConfigProvider.ResolveSchedule(collector, ServerId, rows); + var after = StoreConfigProvider.ResolveSchedule(collector, ServerId, merged); + Assert.Equal(before.Enabled, after.Enabled); + Assert.Equal(before.FrequencyMinutes, after.FrequencyMinutes); + Assert.Equal(before.RetentionDays, after.RetentionDays); + Assert.Equal(StoreConfigProvider.ResolveFleetRetentionDays(collector, rows), StoreConfigProvider.ResolveFleetRetentionDays(collector, merged)); + Assert.Equal(StoreConfigProvider.ResolveDatabaseScope(collector, ServerId, rows), StoreConfigProvider.ResolveDatabaseScope(collector, ServerId, merged)); + } + + /* The fleet row disabled index_object_stats and the server's run time must not re-enable it for that server, and + a collector off by default stays off. */ + Assert.False(StoreConfigProvider.ResolveSchedule("index_object_stats", ServerId, merged).Enabled); + Assert.False(StoreConfigProvider.ResolveSchedule("long_query_completions", ServerId, merged).Enabled); + Assert.Equal(130, StoreConfigProvider.ResolveSchedule("index_object_stats", ServerId, merged).RunAtMinute); + } + + [Fact] + public void MergeRunTimes_HoldsOneEntryPerScopeAndCollector_MatchingTheNameWithoutRegardToCase() + { + var merged = StoreConfigProvider.MergeRunTimes( + new[] { new ScheduleOverride(null, "Index_Object_Stats", 1440, 30, true) }, + new[] { RunTime(null, "index_object_stats", 120), RunTime(ServerId, "index_object_stats", 130), RunTime(ServerId + 1, "index_object_stats", 140) }); + + Assert.Equal(3, merged.Count); + Assert.Equal(1, merged.Count(o => o.ServerId is null)); + Assert.Equal(120, merged.Single(o => o.ServerId is null).RunAtMinute); + Assert.Equal(30, merged.Single(o => o.ServerId is null).RetentionDays); + + /* The fleet row's retention is still the first, and only, fleet entry the retention horizon reads. */ + Assert.Equal(30, StoreConfigProvider.ResolveFleetRetentionDays("index_object_stats", merged)); + } + + [Fact] + public void MergeRunTimes_WithNoRunTimes_ReturnsTheRowsAsTheyAre() + { + var rows = new[] { new ScheduleOverride(null, "index_object_stats", 1440, 30, true) }; + + Assert.Same(rows, StoreConfigProvider.MergeRunTimes(rows, Array.Empty())); + } +} + +/// The live half of : the table, its CHECK, indexes and reload trigger, an +/// upgrade from V159, the rung's SQL run twice on one store, what the viewer's schedule Save leaves alone, and the +/// service's read (merged onto the schedule rows, warned once per load, and tolerant of a store without the table). +/* #1776 own-store: each fact mints its own scratch database through ScratchPostgres and never touches the shared + store's tables, so it cannot race the live collection and serializing it would be pure slowdown. */ +public sealed class CollectorRunTimeRungLiveTests +{ + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private const string SkipText = "Set DARLING_TEST_PG to a Postgres connection string to run the run-time table's live pins (each mints its own scratch database)."; + + private const string Insert = "INSERT INTO config.config_collector_run_times (server_id, collector_name, run_at_minute) VALUES "; + + private static async Task MigratedAsync(ScratchPostgres scratch, System.Threading.CancellationToken ct) + { + var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + return connection; + } + + private static async Task ScalarAsync(NpgsqlConnection connection, string sql, System.Threading.CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + return await command.ExecuteScalarAsync(ct); + } + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, System.Threading.CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task VersionAsync(NpgsqlConnection connection, System.Threading.CancellationToken ct) => + Convert.ToInt32(await ScalarAsync(connection, "SELECT MAX(version) FROM darling_schema_version", ct)); + + /// Every run-time row as one string, so a test can say "unchanged" in one assertion. + private static async Task RowsAsync(NpgsqlConnection connection, System.Threading.CancellationToken ct) => + (string?)await ScalarAsync(connection, + "SELECT COALESCE(string_agg(COALESCE(server_id::text, 'fleet') || '/' || collector_name || '=' || run_at_minute, ',' ORDER BY server_id NULLS FIRST, collector_name), '') FROM config.config_collector_run_times", ct); + + private static async Task SeededProviderAsync( + NpgsqlDataSource dataSource, CapturingTestLogger logger, System.Threading.CancellationToken ct, + string storedPassword = "env:RUNTIME_ALPHA_PASSWORD_NOT_SET") + { + var provider = new StoreConfigProvider(dataSource, logger); + var config = new DarlingConfig(); + config.Servers.Add(new MonitoredServer + { + Name = "runtime-alpha", + Host = "runtime-alpha-host", + Auth = "sql", + Username = "runtime-user", + /* A reference by default, not a stored old-format password: the load warns that an old-format password needs to be entered + again (#5366), and the run-time tests count the warnings that belong to the run times. */ + EncryptedPassword = storedPassword, + EncryptMode = "Strict", + TrustServerCertificate = true, + }); + await provider.SeedIfEmptyAsync(config, ct); + return provider; + } + + [Fact] + public async Task TheTable_HasItsColumnsItsCheckItsTwoUniqueIndexesAndTheReloadTrigger() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + + var bodySucceeded = false; + try + { + Assert.Equal("server_id:integer:YES,collector_name:text:NO,run_at_minute:smallint:NO", await ScalarAsync(connection, + "SELECT string_agg(column_name || ':' || data_type || ':' || is_nullable, ',' ORDER BY ordinal_position) FROM information_schema.columns WHERE table_schema = 'config' AND table_name = 'config_collector_run_times'", ct)); + Assert.Equal(0L, await ScalarAsync(connection, + "SELECT count(*) FROM information_schema.columns WHERE table_schema = 'config' AND table_name = 'config_collector_schedules' AND column_name = 'run_at_minute'", ct)); + + /* The CHECK: 0 to 1439 anywhere, -1 on a server row only. */ + foreach (var refused in new[] { "NULL, 'rt_bad', -1", "NULL, 'rt_bad', -2", "NULL, 'rt_bad', 1440", "7, 'rt_bad', -2", "7, 'rt_bad', 1440" }) + { + var ex = await Assert.ThrowsAsync(() => ExecAsync(connection, Insert + $"({refused})", ct)); + Assert.Equal("23514", ex.SqlState); + } + + var notNull = await Assert.ThrowsAsync(() => ExecAsync(connection, Insert + "(NULL, 'rt_null', NULL)", ct)); + Assert.Equal("23502", notNull.SqlState); + + foreach (var taken in new[] { "NULL, 'rt_a', 0", "NULL, 'rt_b', 1439", "7, 'rt_c', -1", "7, 'rt_d', 0" }) + { + await ExecAsync(connection, Insert + $"({taken})", ct); + } + + /* One fleet row and one server row per collector; the same collector on another server, or fleet-wide and on + a server, is fine. */ + foreach (var duplicate in new[] { "NULL, 'rt_a', 5", "7, 'rt_d', 9" }) + { + var ex = await Assert.ThrowsAsync(() => ExecAsync(connection, Insert + $"({duplicate})", ct)); + Assert.Equal("23505", ex.SqlState); + } + + await ExecAsync(connection, Insert + "(8, 'rt_d', 9), (7, 'rt_a', 9)", ct); + Assert.Contains("(collector_name) WHERE (server_id IS NULL)", (string?)await ScalarAsync(connection, + "SELECT indexdef FROM pg_indexes WHERE schemaname = 'config' AND indexname = 'ux_config_collector_run_times_fleet'", ct), StringComparison.Ordinal); + Assert.Contains("(server_id, collector_name) WHERE (server_id IS NOT NULL)", (string?)await ScalarAsync(connection, + "SELECT indexdef FROM pg_indexes WHERE schemaname = 'config' AND indexname = 'ux_config_collector_run_times_server'", ct), StringComparison.Ordinal); + + /* A write bumps the reload beacon, as a schedule write does, so a running service reloads the run times. */ + Assert.Equal(1L, await ScalarAsync(connection, + "SELECT count(*) FROM pg_trigger WHERE tgname = 'trg_bump_collector_run_times' AND NOT tgisinternal", ct)); + await ExecAsync(connection, "INSERT INTO config.config_service (id) VALUES (1) ON CONFLICT (id) DO NOTHING", ct); + foreach (var write in new[] + { + Insert + "(NULL, 'rt_e', 10)", + "UPDATE config.config_collector_run_times SET run_at_minute = 11 WHERE collector_name = 'rt_e'", + "DELETE FROM config.config_collector_run_times WHERE collector_name = 'rt_e'", + }) + { + var before = Convert.ToInt64(await ScalarAsync(connection, "SELECT config_version FROM config.config_service", ct)); + await ExecAsync(connection, write, ct); + Assert.True(Convert.ToInt64(await ScalarAsync(connection, "SELECT config_version FROM config.config_service", ct)) > before, write); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + [Fact] + public async Task AStoreAtV159_UpgradesToTheRung_WithNoRunTimes_AndItsScheduleRowsAndColumnsUntouched() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + + var bodySucceeded = false; + try + { + const string ScheduleColumns = + "SELECT string_agg(column_name, ',' ORDER BY ordinal_position) FROM information_schema.columns WHERE table_schema = 'config' AND table_name = 'config_collector_schedules'"; + var columnsBefore = await ScalarAsync(connection, ScheduleColumns, ct); + + /* Put the store back at V159: no table, its stamp gone, a schedule row an operator already wrote. */ + foreach (var sql in new[] + { + "DROP TABLE config.config_collector_run_times", + "DELETE FROM darling_schema_version WHERE version >= 160", + "INSERT INTO config.config_collector_schedules (server_id, collector_name, frequency_minutes, retention_days, enabled) VALUES (NULL, 'index_object_stats', 1440, 30, TRUE)", + }) + { + await ExecAsync(connection, sql, ct); + } + + Assert.Equal(159, await VersionAsync(connection, ct)); + Assert.True(await ScalarAsync(connection, "SELECT to_regclass('config.config_collector_run_times') IS NULL", ct) is true); + + await PgMigrations.MigrateAsync(connection, ct); + + Assert.Equal(StorageVersion.SchemaVersion, await VersionAsync(connection, ct)); + Assert.Equal(0L, await ScalarAsync(connection, "SELECT count(*) FROM config.config_collector_run_times", ct)); + Assert.Equal("1440|30", await ScalarAsync(connection, + "SELECT frequency_minutes || '|' || retention_days FROM config.config_collector_schedules WHERE collector_name = 'index_object_stats'", ct)); + Assert.Equal(columnsBefore, await ScalarAsync(connection, ScheduleColumns, ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + /// The runner skips a rung a store is already stamped at, so migrating twice never runs the rung's SQL + /// twice. This runs the rung's own SQL twice on one store that holds rows and asserts the second run is a no-op: + /// no error, the same columns, indexes, constraint and trigger, and the same rows. + [Fact] + public async Task TheRungSql_RunTwiceOnOneStore_ChangesNothing_AndKeepsItsRows() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + + var bodySucceeded = false; + try + { + var rungSql = PgMigrations.Scripts.Single(m => m.Name == CollectorRunTimeRungTests.RungName).Sql; + await ExecAsync(connection, Insert + "(NULL, 'index_object_stats', 120), (7, 'index_object_stats', -1)", ct); + + const string Shape = + "SELECT (SELECT string_agg(column_name || ':' || data_type || ':' || is_nullable, ',' ORDER BY ordinal_position) FROM information_schema.columns WHERE table_schema = 'config' AND table_name = 'config_collector_run_times')" + + " || '|' || (SELECT string_agg(indexname || ':' || indexdef, ';' ORDER BY indexname) FROM pg_indexes WHERE schemaname = 'config' AND tablename = 'config_collector_run_times')" + + " || '|' || (SELECT string_agg(conname || ':' || pg_get_constraintdef(oid), ';' ORDER BY conname) FROM pg_constraint WHERE conrelid = 'config.config_collector_run_times'::regclass)" + + " || '|' || (SELECT count(*) FROM pg_trigger WHERE tgrelid = 'config.config_collector_run_times'::regclass AND NOT tgisinternal)"; + var shapeBefore = await ScalarAsync(connection, Shape, ct); + var rowsBefore = await RowsAsync(connection, ct); + Assert.Equal("fleet/index_object_stats=120,7/index_object_stats=-1", rowsBefore); + + await ExecAsync(connection, rungSql, ct); + await ExecAsync(connection, rungSql, ct); + + Assert.Equal(shapeBefore, await ScalarAsync(connection, Shape, ct)); + Assert.Equal(rowsBefore, await RowsAsync(connection, ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + /// The install id's row survives the upgrade from V159 untouched (#4938, #4961). V159 is the install id table's + /// final shape, and this rung creates a table of its own and restates nothing of it: the id, its bindings and its + /// creation time read the same after the rung as before it, and the table keeps exactly its seven columns. + [Fact] + public async Task AStoreAtV159_KeepsItsInstallIdRowAndTableShapeUnchanged_AcrossTheUpgradeToTheRung() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + + var bodySucceeded = false; + try + { + const string ReadRow = + "SELECT install_id || '|' || system_identifier || '|' || database_oid || '|' || table_oid || '|' || server_major || '|' || created_at::text FROM config.config_install_id"; + const string ReadShape = + "SELECT string_agg(column_name || ':' || data_type, ',' ORDER BY ordinal_position) FROM information_schema.columns WHERE table_schema = 'config' AND table_name = 'config_install_id'"; + const string Row = "0a1b2c3d|7000000000000000001|16384|16400|17|2026-01-02 03:04:05.678901"; + const string Shape = "id:smallint,install_id:text,system_identifier:bigint,database_oid:bigint,created_at:timestamp without time zone,table_oid:bigint,server_major:integer"; + + Assert.Equal(StorageVersion.SchemaVersion, await VersionAsync(connection, ct)); + Assert.Equal(Shape, await ScalarAsync(connection, ReadShape, ct)); + + /* Put the store back at V159: the run time's table and stamp gone, an install id row the service already made. */ + foreach (var sql in new[] + { + "DROP TABLE config.config_collector_run_times", + "DELETE FROM darling_schema_version WHERE version >= 160", + "INSERT INTO config.config_install_id (install_id, system_identifier, database_oid, table_oid, server_major, created_at) VALUES ('0a1b2c3d', 7000000000000000001, 16384, 16400, 17, '2026-01-02 03:04:05.678901')", + }) + { + await ExecAsync(connection, sql, ct); + } + + Assert.Equal(159, await VersionAsync(connection, ct)); + Assert.Equal(Row, await ScalarAsync(connection, ReadRow, ct)); + Assert.Equal(Shape, await ScalarAsync(connection, ReadShape, ct)); + + await PgMigrations.MigrateAsync(connection, ct); + + Assert.Equal(StorageVersion.SchemaVersion, await VersionAsync(connection, ct)); + Assert.True(await ScalarAsync(connection, "SELECT to_regclass('config.config_collector_run_times') IS NOT NULL", ct) is true); + Assert.Equal(Row, await ScalarAsync(connection, ReadRow, ct)); + Assert.Equal(Shape, await ScalarAsync(connection, ReadShape, ct)); + Assert.Equal(1L, await ScalarAsync(connection, "SELECT COUNT(*) FROM config.config_install_id", ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + /// The viewer's schedule Save, run live (its real SQL: delete the scope's rows, insert them again), leaves every + /// run-time row alone, for a fleet scope and for a server scope. A viewer released before the table existed does the + /// same statements, so the run times survive its Save too. The Save itself still works. + [Fact] + public async Task TheViewerScheduleSave_LeavesEveryRunTimeRowAlone_ForAFleetScopeAndAServerScope() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + + var bodySucceeded = false; + try + { + await ExecAsync(connection, + Insert + "(NULL, 'index_object_stats', 120), (NULL, 'server_properties', 150), (7, 'index_object_stats', 130), (7, 'server_properties', -1), (8, 'index_object_stats', 200)", ct); + await ExecAsync(connection, + "INSERT INTO config.config_collector_schedules (server_id, collector_name, frequency_minutes) VALUES (NULL, 'index_object_stats', 720), (7, 'index_object_stats', 2880)", ct); + var expected = await RowsAsync(connection, ct); + Assert.Equal("fleet/index_object_stats=120,fleet/server_properties=150,7/index_object_stats=130,7/server_properties=-1,8/index_object_stats=200", expected); + + await using var viewer = new ViewerDataService(scratch.ConnectionString); + + await viewer.ReplaceFleetSchedulesAsync(new[] { new CollectorScheduleRow(null, "index_object_stats", 1440, 30, true, null) }, ct); + Assert.Equal(1440, Convert.ToInt32(await ScalarAsync(connection, + "SELECT frequency_minutes FROM config.config_collector_schedules WHERE server_id IS NULL AND collector_name = 'index_object_stats'", ct))); + Assert.Equal(expected, await RowsAsync(connection, ct)); + + await viewer.ReplaceServerSchedulesAsync(7, new[] { new CollectorScheduleRow(7, "index_object_stats", 1440, null, true, null) }, ct); + Assert.Equal(1440, Convert.ToInt32(await ScalarAsync(connection, + "SELECT frequency_minutes FROM config.config_collector_schedules WHERE server_id = 7 AND collector_name = 'index_object_stats'", ct))); + Assert.Equal(expected, await RowsAsync(connection, ct)); + + /* A Save that leaves a scope empty deletes schedule rows only. The reset of every server's rows is the one call that also + deletes the servers' run times (#4938), and the fleet's stay. */ + await viewer.ReplaceServerSchedulesAsync(7, Array.Empty(), ct); + Assert.Equal(expected, await RowsAsync(connection, ct)); + await viewer.ResetAllServerSchedulesAsync(ct); + Assert.Equal(0L, await ScalarAsync(connection, "SELECT count(*) FROM config.config_collector_schedules WHERE server_id IS NOT NULL", ct)); + Assert.Equal("fleet/index_object_stats=120,fleet/server_properties=150", await RowsAsync(connection, ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + /// The service's read merges the run times onto the schedule rows, and the load warns once for a run time that + /// cannot apply, each time the schedules are loaded. + [Fact] + public async Task LoadViewAsync_ReadsTheRunTimesFromTheirTable_AndWarnsOncePerLoad_ForAServerWhoseIntervalIsHourly() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + + var bodySucceeded = false; + try + { + await using var dataSource = NpgsqlDataSource.Create(scratch.ConnectionString); + var logger = new CapturingTestLogger(); + var provider = await SeededProviderAsync(dataSource, logger, ct); + + var serverId = Convert.ToInt32(await ScalarAsync(connection, "SELECT server_id FROM config_monitored_servers WHERE name = 'runtime-alpha'", ct)); + foreach (var sql in new[] + { + Insert + "(NULL, 'index_object_stats', 120)", + $"INSERT INTO config.config_collector_schedules (server_id, collector_name, frequency_minutes) VALUES ({serverId}, 'index_object_stats', 60)", + Insert + "(NULL, 'server_properties', 150)", + "INSERT INTO config.config_collector_schedules (server_id, collector_name, frequency_minutes, retention_days) VALUES (NULL, 'pg_index_usage_stats', 1440, 30)", + Insert + "(NULL, 'pg_index_usage_stats', 200)", + }) + { + await ExecAsync(connection, sql, ct); + } + + var first = await provider.LoadViewAsync(new DarlingConfig(), ct); + Assert.NotNull(first); + Assert.Equal(120, first!.ScheduleOverrides.Single(o => o.ServerId is null && o.CollectorName == "index_object_stats").RunAtMinute); + Assert.Null(first.ScheduleOverrides.Single(o => o.ServerId == serverId).RunAtMinute); + + /* A run time for a collector that has a schedule row rides on that row. */ + var carried = first.ScheduleOverrides.Single(o => o.ServerId is null && o.CollectorName == "pg_index_usage_stats"); + Assert.Equal(30, carried.RetentionDays); + Assert.Equal(200, carried.RunAtMinute); + + Assert.Equal(150, StoreConfigProvider.ResolveSchedule("server_properties", serverId, first.ScheduleOverrides).RunAtMinute); + Assert.Null(StoreConfigProvider.ResolveSchedule("index_object_stats", serverId, first.ScheduleOverrides).RunAtMinute); + + Assert.Single(logger.Lines, l => l.Contains("run time", StringComparison.OrdinalIgnoreCase)); + + /* Each load warns again; a sweep between loads, which only resolves, does not. */ + _ = StoreConfigProvider.ResolveSchedule("index_object_stats", serverId, first.ScheduleOverrides); + Assert.Single(logger.Lines, l => l.Contains("run time", StringComparison.OrdinalIgnoreCase)); + await provider.LoadViewAsync(new DarlingConfig(), ct); + Assert.Equal(2, logger.Lines.Count(l => l.Contains("run time", StringComparison.OrdinalIgnoreCase))); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + /// A store still at V159 has no run-time table. The load reads it as no run times, with no warning and no error + /// (one debug line at most), keeps every schedule row, and leaves the connection usable for the next load. + [Fact] + public async Task LoadViewAsync_OnAStoreWithoutTheRunTimeTable_ReadsNoRunTimes_WithNoWarningAndNoError() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + + var bodySucceeded = false; + try + { + await using var dataSource = NpgsqlDataSource.Create(scratch.ConnectionString); + var logger = new CapturingTestLogger(); + var provider = await SeededProviderAsync(dataSource, logger, ct); + + await ExecAsync(connection, "DROP TABLE config.config_collector_run_times", ct); + await ExecAsync(connection, + "INSERT INTO config.config_collector_schedules (server_id, collector_name, frequency_minutes, retention_days) VALUES (NULL, 'index_object_stats', 1440, 30)", ct); + + var view = await provider.LoadViewAsync(new DarlingConfig(), ct); + + Assert.NotNull(view); + var row = Assert.Single(view!.ScheduleOverrides); + Assert.Equal(30, row.RetentionDays); + Assert.Null(row.RunAtMinute); + Assert.Equal(0, logger.CountAtLevel(LogLevel.Warning)); + Assert.Equal(0, logger.CountAtLevel(LogLevel.Error)); + Assert.True(logger.Lines.Count(l => l.Contains("run-time table", StringComparison.OrdinalIgnoreCase)) <= 1, logger.Joined); + + Assert.NotNull(await provider.LoadViewAsync(new DarlingConfig(), ct)); + Assert.Equal(0, logger.CountAtLevel(LogLevel.Warning)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + /// A store whose CHECK was dropped by hand can hold a value the CHECK would refuse. The load resolves it to none + /// and warns once, naming the collector, the scope and the value. + [Fact] + public async Task LoadViewAsync_WarnsOnce_ForAStoredRunTimeTheCheckWouldRefuse_AndResolvesItToNone() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + + var bodySucceeded = false; + try + { + await using var dataSource = NpgsqlDataSource.Create(scratch.ConnectionString); + var logger = new CapturingTestLogger(); + var provider = await SeededProviderAsync(dataSource, logger, ct); + + await ExecAsync(connection, "ALTER TABLE config.config_collector_run_times DROP CONSTRAINT ck_config_collector_run_times_range", ct); + await ExecAsync(connection, Insert + "(NULL, 'index_object_stats', 1440)", ct); + + var view = await provider.LoadViewAsync(new DarlingConfig(), ct); + + Assert.NotNull(view); + Assert.Null(StoreConfigProvider.ResolveSchedule("index_object_stats", 1, view!.ScheduleOverrides).RunAtMinute); + Assert.Equal(1, logger.CountAtLevel(LogLevel.Warning)); + var line = Assert.Single(logger.Lines, l => l.StartsWith("Warning", StringComparison.Ordinal)); + Assert.Contains("index_object_stats", line, StringComparison.Ordinal); + Assert.Contains("fleet-wide", line, StringComparison.Ordinal); + Assert.Contains("1440", line, StringComparison.Ordinal); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + /// A stored old-format password the service cannot open is reported by the load as one warning, and only once: the + /// next load of the same store, with the same count, says nothing more (#5366). + [Fact] + public async Task LoadViewAsync_WarnsOnce_AboutAnOldFormatPasswordItCannotOpen_NotOnEveryReload() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + + var bodySucceeded = false; + try + { + await using var dataSource = NpgsqlDataSource.Create(scratch.ConnectionString); + var logger = new CapturingTestLogger(); + var provider = await SeededProviderAsync(dataSource, logger, ct, storedPassword: "not-a-real-blob"); + + Assert.NotNull(await provider.LoadViewAsync(new DarlingConfig(), ct)); + var line = Assert.Single(logger.Lines, l => l.StartsWith("Warning", StringComparison.Ordinal)); + Assert.Contains("needs to be entered again", line, StringComparison.Ordinal); + + Assert.NotNull(await provider.LoadViewAsync(new DarlingConfig(), ct)); + Assert.Equal(1, logger.CountAtLevel(LogLevel.Warning)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } +} diff --git a/Darling/Darling.Tests/CollectorRunTimeTests.cs b/Darling/Darling.Tests/CollectorRunTimeTests.cs new file mode 100644 index 0000000000..5c6a508fae --- /dev/null +++ b/Darling/Darling.Tests/CollectorRunTimeTests.cs @@ -0,0 +1,610 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Text.RegularExpressions; +using PerformanceMonitor.Analysis.Baselines; +using PerformanceMonitor.Collectors; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4938: the shared rules for a daily collector's run time, . The run time is the +/// monitored server's own wall clock, so each test hands the rules a local-to-UTC conversion the way an app +/// does: for a known clock, while the +/// clock is not known yet. US Eastern springs forward on 2026-03-08 and falls back on 2026-11-01. A server id +/// whose (uint)id % 3600 is 0 puts the slot exactly on the run time; 1800 adds 30 minutes and 3599 adds +/// 59 minutes 59 seconds. +/// +public sealed class CollectorRunTimeTests +{ + private const int NoSpreadId = 0; + private const int HalfHourSpreadId = 1800; + private const int LongSpreadId = 3599; + private const int TwoAm = 120; + private const int Daily = 1440; + private const string EasternWindowsId = "Eastern Standard Time"; + private const string EasternIanaId = "America/New_York"; + + private static DateTime Utc(int year, int month, int day, int hour = 0, int minute = 0, int second = 0) => + new(year, month, day, hour, minute, second, DateTimeKind.Utc); + + private static DateTime NextDue(DateTime now, DateTime? lastRun, int runAt, int interval, int serverId, + Func toUtc) => + CollectorRunTime.NextDue(now, lastRun, runAt, interval, serverId, toUtc); + + private static DateTime UtcClockNextDue(DateTime now, DateTime? lastRun, int runAt = TwoAm, int interval = Daily, + int serverId = NoSpreadId) => + CollectorRunTime.NextDue(now, lastRun, runAt, interval, serverId, CollectorRunTime.LocalIsUtc); + + /* ---- before and after the slot, with no history ---- */ + + [Fact] + public void NeverRun_BeforeTodaysSlot_WaitsForTodaysSlot() + { + Assert.Equal(Utc(2026, 10, 2, 2, 0), UtcClockNextDue(Utc(2026, 10, 2, 1, 0), null)); + Assert.Equal(Utc(2026, 10, 2, 2, 0), UtcClockNextDue(Utc(2026, 10, 2, 2, 0).AddTicks(-1), null)); + Assert.Equal(Utc(2026, 10, 2, 2, 0), UtcClockNextDue(Utc(2026, 10, 2, 0, 0), null)); + } + + [Fact] + public void NeverRun_AfterTodaysGrace_WaitsForTomorrowsSlot_NotNowAndNotAJitteredNow() + { + Assert.Equal(Utc(2026, 10, 3, 2, 0), UtcClockNextDue(Utc(2026, 10, 2, 3, 0).AddTicks(1), null)); + Assert.Equal(Utc(2026, 10, 3, 2, 0), UtcClockNextDue(Utc(2026, 10, 2, 12, 0), null)); + Assert.Equal(Utc(2026, 10, 3, 2, 0), UtcClockNextDue(Utc(2026, 10, 2, 23, 59, 59), null)); + } + + [Theory] + [InlineData(2, 0, 0)] + [InlineData(2, 30, 0)] + [InlineData(2, 59, 59)] + [InlineData(3, 0, 0)] + public void NeverRun_InsideTheGrace_RunsNow(int hour, int minute, int second) + { + var now = Utc(2026, 10, 2, hour, minute, second); + + Assert.Equal(now, UtcClockNextDue(now, null)); + } + + [Fact] + public void TheGrace_IsSixtyMinutes_AndClosedAtBothEnds() + { + Assert.Equal(TimeSpan.FromMinutes(60), CollectorRunTime.Grace); + + var slot = Utc(2026, 10, 2, 2, 0); + Assert.Equal(slot, UtcClockNextDue(slot, null)); + Assert.Equal(slot + CollectorRunTime.Grace, UtcClockNextDue(slot + CollectorRunTime.Grace, null)); + Assert.Equal(Utc(2026, 10, 3, 2, 0), UtcClockNextDue(slot + CollectorRunTime.Grace + TimeSpan.FromTicks(1), null)); + } + + [Fact] + public void ADateMinValueLastRun_IsReadAsNeverRun() + { + var now = Utc(2026, 10, 2, 12, 0); + + Assert.Equal(UtcClockNextDue(now, null), UtcClockNextDue(now, DateTime.MinValue)); + } + + /* ---- the spread ---- */ + + [Fact] + public void TheSpread_IsFixedPerId_AndUnderSixtyMinutes() + { + int[] ids = { 0, 1, 59, 1800, 3599, 3600, 3601, 123456789, -1, -61, -123456789, int.MaxValue, int.MinValue }; + foreach (var id in ids) + { + var spread = CollectorRunTime.Spread(id); + + Assert.Equal(TimeSpan.FromSeconds((uint)id % 3600), spread); + Assert.Equal(spread, CollectorRunTime.Spread(id)); + Assert.True(spread >= TimeSpan.Zero, $"id {id}: the spread must not pull the slot before the run time"); + Assert.True(spread < TimeSpan.FromMinutes(60), $"id {id}: the spread must stay under 60 minutes"); + } + } + + [Fact] + public void TheSlot_IsTheRunTimePlusTheServersSpread_OnTheServersClock() + { + var date = new DateOnly(2026, 10, 2); + + Assert.Equal(Utc(2026, 10, 2, 2, 0), CollectorRunTime.SlotUtc(date, TwoAm, NoSpreadId, CollectorRunTime.LocalIsUtc)); + Assert.Equal(Utc(2026, 10, 2, 2, 30), CollectorRunTime.SlotUtc(date, TwoAm, HalfHourSpreadId, CollectorRunTime.LocalIsUtc)); + Assert.Equal(Utc(2026, 10, 2, 2, 59, 59), CollectorRunTime.SlotUtc(date, TwoAm, LongSpreadId, CollectorRunTime.LocalIsUtc)); + + var eastern = ServerClock.Resolve(EasternIanaId, -300); + Assert.Equal(Utc(2026, 7, 1, 6, 0), CollectorRunTime.SlotUtc(new DateOnly(2026, 7, 1), TwoAm, NoSpreadId, eastern.ToUtc)); + Assert.Equal(Utc(2026, 1, 15, 7, 0), CollectorRunTime.SlotUtc(new DateOnly(2026, 1, 15), TwoAm, NoSpreadId, eastern.ToUtc)); + } + + [Fact] + public void WithASpread_TheGraceStartsAtTheSlot_NotAtTheRunTime() + { + /* Id 1800 puts the slot at 02:30: 02:15 is still before it, 03:20 is inside its grace, 03:31 is past it. */ + Assert.Equal(Utc(2026, 10, 2, 2, 30), UtcClockNextDue(Utc(2026, 10, 2, 2, 15), null, serverId: HalfHourSpreadId)); + Assert.Equal(Utc(2026, 10, 2, 3, 20), UtcClockNextDue(Utc(2026, 10, 2, 3, 20), null, serverId: HalfHourSpreadId)); + Assert.Equal(Utc(2026, 10, 3, 2, 30), UtcClockNextDue(Utc(2026, 10, 2, 3, 31), null, serverId: HalfHourSpreadId)); + } + + /* ---- after a run ---- */ + + [Fact] + public void AfterARun_TheNextDueIsTheNextDatesSlot() + { + var run = Utc(2026, 10, 2, 2, 10); + + Assert.Equal(Utc(2026, 10, 3, 2, 0), UtcClockNextDue(run, run)); + + /* With 30 minutes of spread the slot is 02:30, so a run in its grace is 02:40; a run at 02:10 would be + before the slot and count for the day before. */ + var spreadRun = Utc(2026, 10, 2, 2, 40); + Assert.Equal(Utc(2026, 10, 3, 2, 30), UtcClockNextDue(spreadRun, spreadRun, serverId: HalfHourSpreadId)); + Assert.Equal(Utc(2026, 10, 2, 2, 30), UtcClockNextDue(run, run, serverId: HalfHourSpreadId)); + } + + [Fact] + public void ARestartInsideTheGraceAfterTheRun_DoesNotRunAgain() + { + var run = Utc(2026, 10, 2, 2, 10); + + Assert.Equal(Utc(2026, 10, 3, 2, 0), UtcClockNextDue(Utc(2026, 10, 2, 2, 20), run)); + Assert.Equal(Utc(2026, 10, 3, 2, 0), UtcClockNextDue(Utc(2026, 10, 2, 14, 0), run)); + } + + [Fact] + public void ADayMissedWhileDown_IsSkippedNotReplayed_AndWaitsForTheNextSlot() + { + var lastRun = Utc(2026, 9, 28, 2, 10); + + Assert.Equal(Utc(2026, 10, 3, 2, 0), UtcClockNextDue(Utc(2026, 10, 2, 12, 0), lastRun)); + Assert.Equal(Utc(2026, 10, 2, 2, 0), UtcClockNextDue(Utc(2026, 10, 2, 1, 0), lastRun)); + } + + [Fact] + public void ADayMissedWhileDown_ButNowInsideTheGrace_RunsNow() + { + var lastRun = Utc(2026, 9, 28, 2, 10); + var now = Utc(2026, 10, 2, 2, 20); + + Assert.Equal(now, UtcClockNextDue(now, lastRun)); + } + + [Fact] + public void YesterdaysRunInsideItsGrace_LeavesTodaysSlotDue() + { + var lastRun = Utc(2026, 10, 1, 2, 50); + + Assert.Equal(Utc(2026, 10, 2, 2, 0), UtcClockNextDue(Utc(2026, 10, 2, 1, 0), lastRun)); + Assert.Equal(Utc(2026, 10, 2, 2, 5), UtcClockNextDue(Utc(2026, 10, 2, 2, 5), lastRun)); + } + + [Fact] + public void ARunBeforeTodaysSlot_CountsForTheDayBefore_SoTodaysSlotIsStillOwed() + { + var connectRun = Utc(2026, 10, 2, 1, 0); + + Assert.Equal(Utc(2026, 10, 2, 2, 0), UtcClockNextDue(connectRun, connectRun)); + } + + [Fact] + public void ASlotThatSpillsPastMidnight_BelongsToTheDateItWasComputedFor() + { + /* Run time 23:30 plus 59:59 of spread puts the 2026-10-02 slot at 00:29:59 on 2026-10-03. A run in its grace + is still the 10-02 run, so the next one is the 10-03 slot, a day later, not two. */ + const int lateRunAt = 23 * 60 + 30; + var run = Utc(2026, 10, 3, 0, 35); + + Assert.Equal(Utc(2026, 10, 3, 0, 29, 59), CollectorRunTime.SlotUtc(new DateOnly(2026, 10, 2), lateRunAt, LongSpreadId, CollectorRunTime.LocalIsUtc)); + Assert.Equal(Utc(2026, 10, 4, 0, 29, 59), UtcClockNextDue(run, run, lateRunAt, serverId: LongSpreadId)); + } + + /* ---- whole-day intervals longer than one day ---- */ + + [Fact] + public void ATwoDayInterval_SkipsADay() + { + var run = Utc(2026, 10, 1, 2, 10); + + Assert.Equal(Utc(2026, 10, 3, 2, 0), UtcClockNextDue(run, run, interval: 2880)); + Assert.Equal(Utc(2026, 10, 3, 2, 0), UtcClockNextDue(Utc(2026, 10, 2, 12, 0), run, interval: 2880)); + Assert.Equal(Utc(2026, 10, 2, 2, 0), UtcClockNextDue(Utc(2026, 10, 1, 3, 0), Utc(2026, 9, 30, 2, 10), interval: 2880)); + } + + [Fact] + public void ATwoDayInterval_RunsInsideTheGraceOfItsDay_AndSkipsAMissedOne() + { + var run = Utc(2026, 10, 1, 2, 10); + var inside = Utc(2026, 10, 3, 2, 30); + + Assert.Equal(inside, UtcClockNextDue(inside, run, interval: 2880)); + + /* Past the grace of the 10-03 slot, that run is lost and the next one is two days on, on the same grid. */ + Assert.Equal(Utc(2026, 10, 5, 2, 0), UtcClockNextDue(Utc(2026, 10, 3, 3, 30), run, interval: 2880)); + Assert.Equal(Utc(2026, 10, 5, 2, 0), UtcClockNextDue(Utc(2026, 10, 4, 12, 0), run, interval: 2880)); + } + + [Fact] + public void ATwoDayInterval_NeverRun_WaitsForTheNextDailySlot() + { + Assert.Equal(Utc(2026, 10, 3, 2, 0), UtcClockNextDue(Utc(2026, 10, 2, 12, 0), null, interval: 2880)); + } + + /* ---- the server's clock ---- */ + + [Theory] + [InlineData(EasternWindowsId)] + [InlineData(EasternIanaId)] + public void TheSpringForwardDay_KeepsTheRunTimeOnTheLocalClock(string zoneId) + { + var clock = ServerClock.Resolve(zoneId, -300); + + /* 04:00 local: 09:00Z in standard time on 03-07, 08:00Z in daylight time from 03-08. A run at 04:05 EST on the + 7th is followed by 04:00 EDT, 23 hours on; "due + 1440 minutes" would be 05:00 EDT. */ + var run = Utc(2026, 3, 7, 9, 5); + Assert.Equal(Utc(2026, 3, 8, 8, 0), NextDue(run, run, 240, Daily, NoSpreadId, clock.ToUtc)); + var dayAfter = Utc(2026, 3, 8, 8, 5); + Assert.Equal(Utc(2026, 3, 9, 8, 0), NextDue(dayAfter, dayAfter, 240, Daily, NoSpreadId, clock.ToUtc)); + + /* 02:00 local is inside the gap on the 8th, so it reads as 03:00 EDT (07:00Z), and the 9th is 02:00 EDT (06:00Z). */ + var twoRun = Utc(2026, 3, 7, 7, 10); + Assert.Equal(Utc(2026, 3, 8, 7, 0), NextDue(twoRun, twoRun, TwoAm, Daily, NoSpreadId, clock.ToUtc)); + var gapDayRun = Utc(2026, 3, 8, 7, 10); + Assert.Equal(Utc(2026, 3, 9, 6, 0), NextDue(gapDayRun, gapDayRun, TwoAm, Daily, NoSpreadId, clock.ToUtc)); + } + + [Theory] + [InlineData(EasternWindowsId)] + [InlineData(EasternIanaId)] + public void TheFallBackDay_KeepsTheRunTimeOnTheLocalClock(string zoneId) + { + var clock = ServerClock.Resolve(zoneId, -240); + + /* 02:00 EDT on 10-31 is 06:00Z; 02:00 EST on 11-01 is 07:00Z, 25 hours on. "due + 1440 minutes" would be + 06:00Z, which is 01:00 EST on the 1st. */ + var run = Utc(2026, 10, 31, 6, 10); + Assert.Equal(Utc(2026, 11, 1, 7, 0), NextDue(run, run, TwoAm, Daily, NoSpreadId, clock.ToUtc)); + var dayAfter = Utc(2026, 11, 1, 7, 10); + Assert.Equal(Utc(2026, 11, 2, 7, 0), NextDue(dayAfter, dayAfter, TwoAm, Daily, NoSpreadId, clock.ToUtc)); + + /* Before the change a never-run collector sees the daylight slot, after it the standard one. */ + Assert.Equal(Utc(2026, 11, 1, 7, 0), NextDue(Utc(2026, 11, 1, 5, 0), null, TwoAm, Daily, NoSpreadId, clock.ToUtc)); + } + + [Theory] + [InlineData(EasternWindowsId)] + [InlineData(EasternIanaId)] + public void EveryDayOfTheYear_TheNextSlotIsTheSameLocalTime(string zoneId) + { + var clock = ServerClock.Resolve(zoneId, -300); + var springForwardDay = new DateOnly(2026, 3, 8); + + /* Run time 02:00 plus the 30 minutes of spread for id 1800: 02:30 local, which does not exist on the spring + forward day and reads as 03:30 there. */ + var previous = CollectorRunTime.SlotUtc(new DateOnly(2026, 1, 1), TwoAm, HalfHourSpreadId, clock.ToUtc); + for (var date = new DateOnly(2026, 1, 2); date <= new DateOnly(2026, 12, 31); date = date.AddDays(1)) + { + var run = previous.AddMinutes(5); + var next = NextDue(run, run, TwoAm, Daily, HalfHourSpreadId, clock.ToUtc); + + Assert.Equal(CollectorRunTime.SlotUtc(date, TwoAm, HalfHourSpreadId, clock.ToUtc), next); + var expectedLocal = date.ToDateTime(date == springForwardDay ? new TimeOnly(3, 30) : new TimeOnly(2, 30)); + Assert.Equal(expectedLocal, clock.ToServerLocal(next)); + previous = next; + } + } + + [Fact] + public void AFixedOffsetClock_ReadsTheRunTimeOnTheServersClock_AcrossTheUtcDate() + { + /* +05:30: 02:00 local on the 3rd is 20:30Z on the 2nd. */ + var india = ServerClock.FixedOffset(330); + + Assert.Equal(Utc(2026, 10, 2, 20, 30), NextDue(Utc(2026, 10, 2, 12, 0), null, TwoAm, Daily, NoSpreadId, india.ToUtc)); + Assert.Equal(Utc(2026, 10, 2, 20, 30), NextDue(Utc(2026, 10, 2, 20, 0), null, TwoAm, Daily, NoSpreadId, india.ToUtc)); + Assert.Equal(Utc(2026, 10, 2, 21, 0), NextDue(Utc(2026, 10, 2, 21, 0), null, TwoAm, Daily, NoSpreadId, india.ToUtc)); + Assert.Equal(Utc(2026, 10, 3, 20, 30), NextDue(Utc(2026, 10, 2, 22, 0), null, TwoAm, Daily, NoSpreadId, india.ToUtc)); + + /* -10:00, run time 23:30: the 2026-10-02 slot is 09:30Z on the 3rd. */ + var hawaii = ServerClock.FixedOffset(-600); + const int lateRunAt = 23 * 60 + 30; + Assert.Equal(Utc(2026, 10, 3, 9, 30), NextDue(Utc(2026, 10, 2, 12, 0), null, lateRunAt, Daily, NoSpreadId, hawaii.ToUtc)); + } + + [Fact] + public void AnUnknownClock_IsUtc() + { + var instant = new DateTime(2026, 10, 2, 2, 0, 0, DateTimeKind.Unspecified); + + Assert.Equal(instant, CollectorRunTime.LocalIsUtc(instant)); + Assert.Equal(Utc(2026, 10, 3, 2, 0), UtcClockNextDue(Utc(2026, 10, 2, 12, 0), null)); + Assert.Equal(Utc(2026, 10, 3, 2, 0), NextDue(Utc(2026, 10, 2, 12, 0), null, TwoAm, Daily, NoSpreadId, ServerClock.Utc.ToUtc)); + } + + [Fact] + public void TheAnswers_AreUtcKind_WhateverKindTheClockReturns() + { + var eastern = ServerClock.Resolve(EasternIanaId, -300); + var now = Utc(2026, 10, 2, 12, 0); + + Assert.Equal(DateTimeKind.Utc, NextDue(now, null, TwoAm, Daily, NoSpreadId, eastern.ToUtc).Kind); + Assert.Equal(DateTimeKind.Utc, UtcClockNextDue(now, null).Kind); + Assert.Equal(DateTimeKind.Utc, CollectorRunTime.SlotUtc(new DateOnly(2026, 10, 2), TwoAm, NoSpreadId, eastern.ToUtc).Kind); + } + + /* ---- every clock, against a brute force over each candidate date ---- */ + + private static IEnumerable<(string Name, Func ToUtc)> Clocks() + { + yield return ("unknown", CollectorRunTime.LocalIsUtc); + yield return ("-12:00", ServerClock.FixedOffset(-720).ToUtc); + yield return ("-05:00", ServerClock.FixedOffset(-300).ToUtc); + yield return ("+05:30", ServerClock.FixedOffset(330).ToUtc); + yield return ("+14:00", ServerClock.FixedOffset(840).ToUtc); + yield return ("US Eastern", ServerClock.Resolve(EasternIanaId, -300).ToUtc); + } + + private static IEnumerable Samples() + { + foreach (var start in new[] { Utc(2026, 3, 6), Utc(2026, 10, 30), Utc(2026, 10, 1) }) + { + for (var t = start; t < start.AddDays(4); t = t.AddMinutes(211)) + { + yield return t; + } + } + } + + /* The slow, obvious version: look at every date around now. The first slot at or after now, or now itself when + now is inside a slot's grace. */ + private static DateTime NeverRunByBruteForce(DateTime now, int runAt, int serverId, Func toUtc) + { + var around = DateOnly.FromDateTime(now); + DateTime? next = null; + for (var date = around.AddDays(-4); date <= around.AddDays(4); date = date.AddDays(1)) + { + var slot = CollectorRunTime.SlotUtc(date, runAt, serverId, toUtc); + if (slot <= now && now <= slot + CollectorRunTime.Grace) + { + return now; + } + + if (slot > now && (next is null || slot < next)) + { + next = slot; + } + } + + return next!.Value; + } + + /* The date a run belongs to is the latest date whose slot is at or before it. The slots after it are every + `days` dates on; the first whose grace has not passed is due, or now when now is already inside it. */ + private static DateTime WithLastRunByBruteForce(DateTime now, DateTime lastRun, int days, int runAt, int serverId, + Func toUtc) + { + var around = DateOnly.FromDateTime(lastRun); + var lastDate = around.AddDays(-4); + for (var date = around.AddDays(-4); date <= around.AddDays(4); date = date.AddDays(1)) + { + if (CollectorRunTime.SlotUtc(date, runAt, serverId, toUtc) <= lastRun) + { + lastDate = date; + } + } + + for (var step = 1; ; step++) + { + var slot = CollectorRunTime.SlotUtc(lastDate.AddDays(step * days), runAt, serverId, toUtc); + if (now <= slot + CollectorRunTime.Grace) + { + return slot <= now ? now : slot; + } + } + } + + [Fact] + public void NeverRun_OnEveryClock_IsTheFirstSlotAheadOrNowInsideAGrace() + { + foreach (var (name, toUtc) in Clocks()) + { + foreach (var runAt in new[] { 0, 119, 1439 }) + { + foreach (var id in new[] { NoSpreadId, HalfHourSpreadId, LongSpreadId }) + { + foreach (var now in Samples()) + { + var actual = NextDue(now, null, runAt, Daily, id, toUtc); + + Assert.True(NeverRunByBruteForce(now, runAt, id, toUtc) == actual, + $"{name}, run at {runAt}, id {id}, now {now:O}: got {actual:O}"); + Assert.True(actual - now <= CollectorRunTime.MaxStampAhead(Daily), + $"{name}, run at {runAt}, id {id}, now {now:O}: {actual:O} is further ahead than the clamp bound"); + } + } + } + } + } + + [Fact] + public void WithALastRun_OnEveryClock_IsTheNextGridSlotOrNowInsideItsGrace() + { + var back = new[] { TimeSpan.FromMinutes(10), TimeSpan.FromHours(5), TimeSpan.FromHours(30), TimeSpan.FromHours(70) }; + foreach (var (name, toUtc) in Clocks()) + { + foreach (var runAt in new[] { 0, 119, 1439 }) + { + foreach (var id in new[] { NoSpreadId, LongSpreadId }) + { + foreach (var days in new[] { 1, 2, 3 }) + { + foreach (var now in Samples()) + { + foreach (var ago in back) + { + var lastRun = now - ago; + var actual = NextDue(now, lastRun, runAt, days * Daily, id, toUtc); + + Assert.True(WithLastRunByBruteForce(now, lastRun, days, runAt, id, toUtc) == actual, + $"{name}, run at {runAt}, id {id}, {days} d, now {now:O}, last run {lastRun:O}: got {actual:O}"); + Assert.True(actual - now <= CollectorRunTime.MaxStampAhead(days * Daily), + $"{name}, run at {runAt}, id {id}, {days} d, now {now:O}, last run {lastRun:O}: {actual:O} is further ahead than the clamp bound"); + } + } + } + } + } + } + } + + /* ---- the clamp bound ---- */ + + [Fact] + public void TheClampBound_IsTheIntervalPlusSixtyMinutesOfSpreadPlusAnAutumnHour() + { + Assert.Equal(TimeSpan.FromMinutes(1440 + 60 + 60), CollectorRunTime.MaxStampAhead(1440)); + Assert.Equal(TimeSpan.FromMinutes(2880 + 60 + 60), CollectorRunTime.MaxStampAhead(2880)); + } + + [Fact] + public void ARunTimeStampOnTheLongestDay_IsNotReadAsAClockStep() + { + var eastern = ServerClock.Resolve(EasternIanaId, -240); + var run = Utc(2026, 10, 31, 6, 0); + var due = NextDue(run, run, TwoAm, Daily, NoSpreadId, eastern.ToUtc); + + Assert.Equal(Utc(2026, 11, 1, 7, 0), due); + Assert.Equal(TimeSpan.FromHours(25), due - run); + + /* The plain interval reads a 25-hour wait as a clock that stepped back and runs it now. The bound does not. */ + Assert.Equal(run, CollectorCadence.ClampDue(due, run, TimeSpan.FromMinutes(Daily))); + Assert.Equal(due, CollectorCadence.ClampDue(due, run, CollectorRunTime.MaxStampAhead(Daily))); + } + + [Fact] + public void AStampMuchFurtherAheadThanTheBound_IsStillReadAsAClockStep() + { + var now = Utc(2026, 10, 2, 12, 0); + var due = now.AddDays(3); + + Assert.Equal(now, CollectorCadence.ClampDue(due, now, CollectorRunTime.MaxStampAhead(Daily))); + } + + /* ---- which collectors take a run time ---- */ + + [Theory] + [InlineData(60, false)] + [InlineData(720, false)] + [InlineData(1439, false)] + [InlineData(1440, true)] + [InlineData(1441, false)] + [InlineData(2880, true)] + [InlineData(4320, true)] + [InlineData(10080, true)] + [InlineData(0, false)] + [InlineData(-1440, false)] + public void AllowsRunAt_OnlyForAPositiveWholeNumberOfDays(int intervalMinutes, bool expected) + { + Assert.Equal(expected, CollectorRunTime.AllowsRunAt(intervalMinutes)); + } + + [Fact] + public void NextDue_RefusesWhatNoRunTimeCanMean() + { + var now = Utc(2026, 10, 2, 12, 0); + + Assert.Throws(() => UtcClockNextDue(now, null, interval: 60)); + Assert.Throws(() => UtcClockNextDue(now, null, interval: 0)); + Assert.Throws(() => UtcClockNextDue(now, null, runAt: -1)); + Assert.Throws(() => UtcClockNextDue(now, null, runAt: 1440)); + Assert.Throws(() => NextDue(now, null, TwoAm, Daily, NoSpreadId, null!)); + Assert.Throws(() => + CollectorRunTime.SlotUtc(new DateOnly(2026, 10, 2), 1440, NoSpreadId, CollectorRunTime.LocalIsUtc)); + } + + /* ---- the HH:MM text ---- */ + + [Theory] + [InlineData("00:00", 0)] + [InlineData("02:00", 120)] + [InlineData("02:05", 125)] + [InlineData("12:30", 750)] + [InlineData("23:59", 1439)] + [InlineData(" 02:00 ", 120)] + public void TryParse_AcceptsA24HourTime(string text, int expectedMinute) + { + Assert.True(CollectorRunTime.TryParse(text, out var minute)); + Assert.Equal(expectedMinute, minute); + } + + [Theory] + [InlineData(null)] + [InlineData("")] + [InlineData(" ")] + [InlineData("24:00")] + [InlineData("23:60")] + [InlineData("99:99")] + [InlineData("2:5")] + [InlineData("2:00")] + [InlineData("02:5")] + [InlineData("abc")] + [InlineData("ab:cd")] + [InlineData("02-00")] + [InlineData("0200")] + [InlineData("02:000")] + [InlineData("02:00:00")] + [InlineData("-1:00")] + [InlineData("+2:00")] + [InlineData("02:00")] + [InlineData("02:00 PM")] + public void TryParse_RefusesAnythingButHhMm(string? text) + { + Assert.False(CollectorRunTime.TryParse(text, out var minute)); + Assert.Equal(0, minute); + } + + [Fact] + public void Format_RoundTripsEveryMinuteOfTheDay() + { + var shape = new Regex("^[0-2][0-9]:[0-5][0-9]$"); + for (var minute = 0; minute < 1440; minute++) + { + var text = CollectorRunTime.Format(minute); + + Assert.Matches(shape, text); + Assert.True(CollectorRunTime.TryParse(text, out var parsed), text); + Assert.Equal(minute, parsed); + } + + Assert.Equal("00:00", CollectorRunTime.Format(0)); + Assert.Equal("02:05", CollectorRunTime.Format(125)); + Assert.Equal("23:59", CollectorRunTime.Format(1439)); + } + + [Fact] + public void Format_RefusesAMinuteOutsideTheDay() + { + Assert.Throws(() => CollectorRunTime.Format(-1)); + Assert.Throws(() => CollectorRunTime.Format(1440)); + } + + /* ---- the one copy of the error texts ---- */ + + [Fact] + public void TheErrorTexts_AreTheOnesEveryEditorShows() + { + Assert.Equal("Run at must be a 24-hour time from 00:00 to 23:59, such as 02:00.", CollectorRunTime.InvalidRunAtMessage); + Assert.Equal( + "'wait_stats' runs every 60 minutes. A run time works only for a collector that runs once a day or less often (1440 minutes, or a multiple of 1440). Change the frequency or clear the run time.", + CollectorRunTime.IntervalRefusalMessage("wait_stats", 60)); + Assert.Equal( + "'server_config' runs every 720 minutes. A run time works only for a collector that runs once a day or less often (1440 minutes, or a multiple of 1440). Change the frequency or clear the run time.", + CollectorRunTime.IntervalRefusalMessage("server_config", 720)); + } +} diff --git a/Darling/Darling.Tests/CollectorRunTimeViewerLiveTests.cs b/Darling/Darling.Tests/CollectorRunTimeViewerLiveTests.cs new file mode 100644 index 0000000000..54f7244695 --- /dev/null +++ b/Darling/Darling.Tests/CollectorRunTimeViewerLiveTests.cs @@ -0,0 +1,545 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// The Collector Schedules window's "Run at" column against a real store (#4938): the read, the save and the clear, +/// for the fleet and for a server, driven the way the window drives them (read the schedule rows and the run times, +/// build the editable schedule, edit a cell, validate, then write the run-time changes and the schedule rows in ONE +/// transaction). The run time +/// lives in config.config_collector_run_times, a table of its own, so a schedule Save never carries it. What the +/// service then resolves from the stored rows is checked with its own , +/// so "None on a server over a fleet time" is judged by the rule the collectors run on and not by a restatement of it. +/// +/* #1776 own-store: each fact mints its own scratch database through ScratchPostgres and never touches the shared + store's tables, so it cannot race the live collection and serializing it would be pure slowdown. */ +public sealed class CollectorRunTimeViewerLiveTests +{ + private const string Daily = "index_object_stats"; + private const string AnotherDaily = "server_properties"; + private const string SkipText = "DARLING_TEST_PG is not set: the viewer run-time tests need a disposable PostgreSQL store."; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static async Task ScalarAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + return await command.ExecuteScalarAsync(ct); + } + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(ct); + } + + /// Every stored run time, one text: fleet/collector=minute or serverId/collector=minute. + private static async Task RunTimesAsync(NpgsqlConnection connection, CancellationToken ct) => + (string)(await ScalarAsync(connection, + "SELECT COALESCE(string_agg(COALESCE(server_id::text, 'fleet') || '/' || collector_name || '=' || run_at_minute, ',' ORDER BY server_id NULLS FIRST, collector_name), '') FROM config.config_collector_run_times", ct))!; + + private static async Task ScheduleRowCountAsync(NpgsqlConnection connection, CancellationToken ct) => + (long)(await ScalarAsync(connection, "SELECT count(*) FROM config.config_collector_schedules", ct))!; + + /// The window's own steps for a scope: read both tables, build the editable schedule, let the caller edit it, + /// validate, then ONE call that writes the run-time changes (one statement each) and the schedule rows, exactly as a + /// released viewer writes them, in one transaction. + private static async Task SaveAsync( + ViewerDataService viewer, int? serverId, Action> edit, CancellationToken ct, + bool usesDefault = false, bool resetToDefaults = false, Func? afterOpen = null) + { + var schedules = await viewer.GetCollectorSchedulesAsync(ct); + var runTimes = await viewer.GetCollectorRunTimesAsync(ct); + var editing = CollectorScheduleOverlay.BuildEffectiveSchedule(schedules, runTimes, usesDefault ? null : serverId); + if (resetToDefaults) + { + /* The window's Reset to Defaults replaces the whole grid with the shipped defaults. */ + editing = CollectorSchedulePresets.BuildDefaultSchedule(); + } + + edit(editing); + + /* Another writer (the CLI verb, another viewer) can change the store while the window is open. */ + if (afterOpen is not null) + { + await afterOpen(); + } + + if (!usesDefault) + { + Assert.True(CollectorScheduleOverlay.ValidateSchedule(editing, out var error), error); + } + + /* The window's own rule: a schedule Reset (Reset to Defaults, or a server on "Use default schedule") also clears the scope's + run-time rows, by scope and in the same transaction. */ + var clearRunTimes = resetToDefaults || usesDefault; + var changes = CollectorScheduleOverlay.ToRunTimeChanges(editing, runTimes, serverId, usesDefault, clearRunTimes); + var rows = serverId is int sid + ? (usesDefault ? new List() : CollectorScheduleOverlay.ToServerOverrideRows(editing, sid)) + : CollectorScheduleOverlay.ToFleetOverrideRows(editing); + await viewer.SaveCollectorScheduleAsync(serverId, rows, changes, clearRunTimes, ct); + } + + /// The fleet row's stored frequency for the daily collector, or null when there is no such row (or no frequency on it). + private static async Task FleetFrequencyAsync(NpgsqlConnection connection, CancellationToken ct) => + await ScalarAsync(connection, + $"SELECT frequency_minutes FROM config.config_collector_schedules WHERE server_id IS NULL AND collector_name = '{Daily}'", ct) is int minutes + ? minutes + : null; + + private static void SetRunAt(List editing, string text, string collector = Daily) => + editing.Single(i => i.Name == collector).RunAtText = text; + + /// What the service resolves for the collector on the server, from the store's own rows. + private static async Task ResolvedRunAtAsync(ViewerDataService viewer, int serverId, CancellationToken ct) + { + var schedules = (await viewer.GetCollectorSchedulesAsync(ct)) + .Select(r => new ScheduleOverride(r.ServerId, r.CollectorName, r.FrequencyMinutes, r.RetentionDays, r.Enabled, r.Databases)) + .ToList(); + var runTimes = (await viewer.GetCollectorRunTimesAsync(ct)) + .Select(r => new RunTimeOverride(r.ServerId, r.CollectorName, r.RunAtMinute)) + .ToList(); + return StoreConfigProvider.ResolveSchedule(Daily, serverId, StoreConfigProvider.MergeRunTimes(schedules, runTimes)).RunAtMinute; + } + + [Fact] + public async Task TheFleetRunTime_IsSetKeptAndCleared_ThroughTheRunTimeTable_AndNeverAScheduleRow() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var bodySucceeded = false; + try + { + await using var viewer = new ViewerDataService(scratch.ConnectionString); + + /* Nothing stored: the cell reads "Use default" and a save writes nothing anywhere. */ + await SaveAsync(viewer, null, editing => Assert.Equal("Use default", editing.Single(i => i.Name == Daily).RunAtText), ct); + Assert.Equal("", await RunTimesAsync(connection, ct)); + Assert.Equal(0L, await ScheduleRowCountAsync(connection, ct)); + + /* SET: a fleet time on a collector left at its default cadence is a run-time row and NO schedule row. */ + await SaveAsync(viewer, null, editing => SetRunAt(editing, "02:00"), ct); + Assert.Equal($"fleet/{Daily}=120", await RunTimesAsync(connection, ct)); + Assert.Equal(0L, await ScheduleRowCountAsync(connection, ct)); + var read = Assert.Single(await viewer.GetCollectorRunTimesAsync(ct)); + Assert.Equal(new CollectorRunTimeRow(null, Daily, 120), read); + + /* KEEP: the window reads the stored time into the cell, and a save that edits only a schedule column (the + schedule rows are deleted and inserted again) leaves the run time as it was. */ + await SaveAsync(viewer, null, editing => + { + Assert.Equal("02:00", editing.Single(i => i.Name == Daily).RunAtText); + editing.Single(i => i.Name == AnotherDaily).RetentionDays = 77; + }, ct); + Assert.Equal($"fleet/{Daily}=120", await RunTimesAsync(connection, ct)); + Assert.Equal(1L, await ScheduleRowCountAsync(connection, ct)); + + /* EDIT: a new time replaces the old one in the same row. */ + await SaveAsync(viewer, null, editing => SetRunAt(editing, "03:15"), ct); + Assert.Equal($"fleet/{Daily}=195", await RunTimesAsync(connection, ct)); + + /* CLEAR: "Use default" deletes the row, and so does "None" (the fleet has nothing to stop). */ + await SaveAsync(viewer, null, editing => SetRunAt(editing, "Use default"), ct); + Assert.Equal("", await RunTimesAsync(connection, ct)); + await SaveAsync(viewer, null, editing => SetRunAt(editing, "01:00"), ct); + await SaveAsync(viewer, null, editing => SetRunAt(editing, "None"), ct); + Assert.Equal("", await RunTimesAsync(connection, ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + [Fact] + public async Task AServerRunTime_UsesDefaultNoneOrItsOwnTime_AndNoneStopsAFleetTime() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var bodySucceeded = false; + try + { + await using var viewer = new ViewerDataService(scratch.ConnectionString); + + await SaveAsync(viewer, null, editing => SetRunAt(editing, "02:00"), ct); + Assert.Equal(120, await ResolvedRunAtAsync(viewer, 7, ct)); + + /* "Use default" on a customizing server writes no row for it, so the fleet's time applies. */ + await SaveAsync(viewer, 7, editing => Assert.Equal("Use default", editing.Single(i => i.Name == Daily).RunAtText), ct); + Assert.Equal($"fleet/{Daily}=120", await RunTimesAsync(connection, ct)); + Assert.Equal(120, await ResolvedRunAtAsync(viewer, 7, ct)); + + /* ITS OWN TIME replaces the fleet's for that server only. */ + await SaveAsync(viewer, 7, editing => SetRunAt(editing, "04:30"), ct); + Assert.Equal($"fleet/{Daily}=120,7/{Daily}=270", await RunTimesAsync(connection, ct)); + Assert.Equal(270, await ResolvedRunAtAsync(viewer, 7, ct)); + Assert.Equal(120, await ResolvedRunAtAsync(viewer, 8, ct)); + + /* NONE is -1 on the server's row, which stops the fleet's time for that server. */ + await SaveAsync(viewer, 7, editing => SetRunAt(editing, "None"), ct); + Assert.Equal($"fleet/{Daily}=120,7/{Daily}=-1", await RunTimesAsync(connection, ct)); + Assert.Null(await ResolvedRunAtAsync(viewer, 7, ct)); + Assert.Equal(120, await ResolvedRunAtAsync(viewer, 8, ct)); + + /* "Use the default schedule" deletes the server's schedule rows AND its run-time rows. */ + await SaveAsync(viewer, 7, _ => { }, ct, usesDefault: true); + Assert.Equal($"fleet/{Daily}=120", await RunTimesAsync(connection, ct)); + Assert.Equal(120, await ResolvedRunAtAsync(viewer, 7, ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + [Fact] + public async Task AScheduleSave_ThroughTheViewersRealSql_LeavesEveryRunTimeAlone_AndApplyDefaultToAllRemovesOnlyTheServerRunTimes() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var bodySucceeded = false; + try + { + await using var viewer = new ViewerDataService(scratch.ConnectionString); + await ExecAsync(connection, + $"INSERT INTO config.config_collector_run_times (server_id, collector_name, run_at_minute) VALUES (NULL, '{Daily}', 0), (3, '{Daily}', -1), (3, '{AnotherDaily}', 1439)", ct); + const string Expected = $"fleet/{Daily}=0,3/{Daily}=-1,3/{AnotherDaily}=1439"; + + /* The viewer's own schedule statements, the fleet's and a server's, replace schedule rows only. */ + await viewer.ReplaceFleetSchedulesAsync(new[] { new CollectorScheduleRow(null, Daily, 1440, 30, true) }, ct); + await viewer.ReplaceServerSchedulesAsync(3, new[] { new CollectorScheduleRow(3, Daily, null, null, true) }, ct); + await viewer.ReplaceServerSchedulesAsync(3, Array.Empty(), ct); + Assert.Equal(Expected, await RunTimesAsync(connection, ct)); + + /* The read returns the rows as stored: minute 0, the last minute of the day, and -1 on a server row. */ + var rows = await viewer.GetCollectorRunTimesAsync(ct); + Assert.Equal(0, rows.Single(r => r.ServerId is null).RunAtMinute); + Assert.Equal(-1, rows.Single(r => r.ServerId == 3 && r.CollectorName == Daily).RunAtMinute); + Assert.Equal(1439, rows.Single(r => r.ServerId == 3 && r.CollectorName == AnotherDaily).RunAtMinute); + + /* "Apply Default to All Servers" is one call: it sends every server back to the fleet, run times included (no server + schedule rows are left at this point, so the count is the two server run times), and keeps the fleet's. */ + var removed = await viewer.ResetAllServerSchedulesAsync(ct); + Assert.Equal(new CollectorScheduleResetCounts(ScheduleOverrides: 0, RunTimes: 2), removed); + Assert.Equal($"fleet/{Daily}=0", await RunTimesAsync(connection, ct)); + + /* A reset of run times alone says so (#4938): it is not reported as schedule overrides. */ + Assert.Equal( + "Reset 2 per-server run times — every server now follows the fleet default.", + CollectorScheduleOverlay.FormatResetStatus(removed)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + /// The viewer's schema probe fails open when it throws, so a store below V160 can reach the editor. The run-time read + /// must then be "no run times" (42P01 undefined_table), the editor's Run at cells must all read "Use default", and saving a run + /// time must say the store is older than this viewer (not the raw error). A schedule save and the reset still work. + [Fact] + public async Task AStoreWithoutTheRunTimeTable_ReadsNoRunTimes_AndASaveSaysTheStoreIsOlder() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var bodySucceeded = false; + try + { + await ExecAsync(connection, "DROP TABLE config.config_collector_run_times", ct); + await using var viewer = new ViewerDataService(scratch.ConnectionString); + + Assert.Empty(await viewer.GetCollectorRunTimesAsync(ct)); + var editing = CollectorScheduleOverlay.BuildEffectiveSchedule( + await viewer.GetCollectorSchedulesAsync(ct), await viewer.GetCollectorRunTimesAsync(ct), null); + Assert.All(editing, i => Assert.Equal(CollectorScheduleOverlay.UseDefaultRunAtText, i.RunAtText)); + + var skew = await Assert.ThrowsAsync(() => + viewer.SaveCollectorRunTimesAsync(new[] { new CollectorRunTimeChange(null, Daily, 120) }, ct)); + Assert.Contains("Update or restart the Darling service", skew.Message, StringComparison.Ordinal); + Assert.IsType(skew.InnerException); + + /* Everything that does not touch a run time still works on such a store, the all-servers reset included: it removes the + server's schedule rows and has no run-time table to clear. */ + await viewer.ReplaceFleetSchedulesAsync(new[] { new CollectorScheduleRow(null, Daily, 1440, 30, true) }, ct); + await viewer.ReplaceServerSchedulesAsync(3, new[] { new CollectorScheduleRow(3, Daily, 720, 30, true) }, ct); + Assert.Equal(2L, await ScheduleRowCountAsync(connection, ct)); + Assert.Equal(new CollectorScheduleResetCounts(ScheduleOverrides: 1, RunTimes: 0), await viewer.ResetAllServerSchedulesAsync(ct)); + Assert.Equal(1L, await ScheduleRowCountAsync(connection, ct)); + + /* The window's one-transaction Save on such a store: a Save that changed no run time sends no run-time statement and + still saves its schedules; one that did says the store is older, and the schedule rows of that same Save are NOT + written (they share its transaction). */ + await viewer.SaveCollectorScheduleAsync( + null, new[] { new CollectorScheduleRow(null, Daily, 720, 30, true) }, Array.Empty(), ct); + Assert.Equal(720, await FleetFrequencyAsync(connection, ct)); + var oneTransaction = await Assert.ThrowsAsync(() => viewer.SaveCollectorScheduleAsync( + null, new[] { new CollectorScheduleRow(null, Daily, 60, 30, true) }, new[] { new CollectorRunTimeChange(null, Daily, 120) }, ct)); + Assert.Contains("Update or restart the Darling service", oneTransaction.Message, StringComparison.Ordinal); + Assert.Equal(720, await FleetFrequencyAsync(connection, ct)); + + /* A schedule Reset's Save on such a store has no run-time table to clear: the clear is skipped and the schedules are saved. */ + await viewer.SaveCollectorScheduleAsync( + null, new[] { new CollectorScheduleRow(null, Daily, 480, 30, true) }, Array.Empty(), true, ct); + Assert.Equal(480, await FleetFrequencyAsync(connection, ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + /// The window's Save is ONE transaction on one connection (#4938). The run-time write comes first, so a schedule write + /// that fails after it (here a frequency the schedules table's own CHECK refuses) must take the run time back with it, for the + /// fleet and for a server; a run-time write that fails (the run-time table's CHECK refuses -1 on a fleet row) must leave the + /// schedule rows of that Save as they were; and the same Save with a valid schedule is the control that both halves land + /// together. + [Fact] + public async Task AScheduleWriteThatFails_LeavesTheRunTimeUnsaved_BecauseTheSaveIsOneTransaction() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var bodySucceeded = false; + try + { + await using var viewer = new ViewerDataService(scratch.ConnectionString); + + /* The control: a valid fleet Save lands the run time and the schedule row together. */ + await viewer.SaveCollectorScheduleAsync( + null, new[] { new CollectorScheduleRow(null, Daily, 1440, 30, true) }, new[] { new CollectorRunTimeChange(null, Daily, 120) }, ct); + Assert.Equal($"fleet/{Daily}=120", await RunTimesAsync(connection, ct)); + Assert.Equal(1440, await FleetFrequencyAsync(connection, ct)); + + /* FLEET: the run time moves 120 -> 300 and the schedule write then fails (frequency_minutes >= 0 refuses -1). Neither + half may stay: the run time is still 120 and the schedule row is still the one the control wrote. */ + var fleetFailure = await Assert.ThrowsAsync(() => viewer.SaveCollectorScheduleAsync( + null, new[] { new CollectorScheduleRow(null, Daily, -1, 30, true) }, new[] { new CollectorRunTimeChange(null, Daily, 300) }, ct)); + Assert.Equal("23514", fleetFailure.SqlState); + Assert.Equal($"fleet/{Daily}=120", await RunTimesAsync(connection, ct)); + Assert.Equal(1440, await FleetFrequencyAsync(connection, ct)); + Assert.Equal(1L, await ScheduleRowCountAsync(connection, ct)); + + /* SERVER: a new server run time and a refused server schedule row. No server run time, no server schedule row. */ + var serverFailure = await Assert.ThrowsAsync(() => viewer.SaveCollectorScheduleAsync( + 3, new[] { new CollectorScheduleRow(3, Daily, -1, null, true) }, new[] { new CollectorRunTimeChange(3, Daily, 180) }, ct)); + Assert.Equal("23514", serverFailure.SqlState); + Assert.Equal($"fleet/{Daily}=120", await RunTimesAsync(connection, ct)); + Assert.Equal(1L, await ScheduleRowCountAsync(connection, ct)); + + /* THE OTHER WAY: a run-time write the run-time table refuses (-1 is a server-row value only) leaves the schedule rows of + that Save as they were, so the 1440 the control wrote is not replaced by this Save's 720. */ + var runTimeFailure = await Assert.ThrowsAsync(() => viewer.SaveCollectorScheduleAsync( + null, new[] { new CollectorScheduleRow(null, Daily, 720, 30, true) }, new[] { new CollectorRunTimeChange(null, Daily, -1) }, ct)); + Assert.Equal("23514", runTimeFailure.SqlState); + Assert.Equal($"fleet/{Daily}=120", await RunTimesAsync(connection, ct)); + Assert.Equal(1440, await FleetFrequencyAsync(connection, ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + private const string RunTimeRows = "INSERT INTO config.config_collector_run_times (server_id, collector_name, run_at_minute) VALUES "; + + private const string ScheduleRows = "INSERT INTO config.config_collector_schedules (server_id, collector_name, frequency_minutes) VALUES "; + + /// "Apply Default to All Servers" is ONE call (#4938): it deletes every server's schedule rows and every server's run + /// times, and keeps the fleet's schedule rows and run times. A server's run time is part of its override, and Reset means back + /// to the shipped defaults, which have no fixed time. It returns the rows removed from each table, counted apart (#4938). + [Fact] + public async Task ApplyDefaultToAllServers_RemovesEveryServersScheduleRowsAndRunTimes_AndKeepsTheFleets() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var bodySucceeded = false; + try + { + await ExecAsync(connection, ScheduleRows + $"(NULL, '{Daily}', 1440), (3, '{Daily}', 2880), (7, '{AnotherDaily}', 720)", ct); + await ExecAsync(connection, RunTimeRows + $"(NULL, '{Daily}', 0), (3, '{Daily}', -1), (3, '{AnotherDaily}', 1439), (7, '{Daily}', 300)", ct); + await using var viewer = new ViewerDataService(scratch.ConnectionString); + + /* Two server schedule rows and three server run times, in one call. */ + Assert.Equal(new CollectorScheduleResetCounts(ScheduleOverrides: 2, RunTimes: 3), await viewer.ResetAllServerSchedulesAsync(ct)); + Assert.Equal(1L, await ScheduleRowCountAsync(connection, ct)); + Assert.Equal(1440, await FleetFrequencyAsync(connection, ct)); + Assert.Equal($"fleet/{Daily}=0", await RunTimesAsync(connection, ct)); + + /* Nothing left that belongs to a server, so a second call removes nothing. */ + Assert.Equal(new CollectorScheduleResetCounts(ScheduleOverrides: 0, RunTimes: 0), await viewer.ResetAllServerSchedulesAsync(ct)); + Assert.Equal($"fleet/{Daily}=0", await RunTimesAsync(connection, ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + /// The all-servers reset is one transaction (#4938): a failure after the schedule delete, here the run-time delete + /// refused by a trigger, leaves the schedule rows AND the run times in place, and the same call with the trigger gone is the + /// control that both halves land together. + [Fact] + public async Task ApplyDefaultToAllServers_ThatFailsAfterTheScheduleDelete_LeavesTheScheduleRowsAndTheRunTimesInPlace() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var bodySucceeded = false; + try + { + await ExecAsync(connection, ScheduleRows + $"(NULL, '{Daily}', 1440), (3, '{Daily}', 2880), (7, '{AnotherDaily}', 720)", ct); + await ExecAsync(connection, RunTimeRows + $"(NULL, '{Daily}', 0), (3, '{Daily}', -1), (7, '{Daily}', 300)", ct); + const string Before = $"fleet/{Daily}=0,3/{Daily}=-1,7/{Daily}=300"; + await using var viewer = new ViewerDataService(scratch.ConnectionString); + + /* The schedule delete runs first, so a refused run-time delete has to take it back. */ + await ExecAsync(connection, + "CREATE FUNCTION public.refuse_run_time_delete() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN RAISE EXCEPTION 'run times cannot be deleted'; END $$", ct); + await ExecAsync(connection, + "CREATE TRIGGER trg_refuse_run_time_delete BEFORE DELETE ON config.config_collector_run_times FOR EACH STATEMENT EXECUTE FUNCTION public.refuse_run_time_delete()", ct); + + var failure = await Assert.ThrowsAsync(() => viewer.ResetAllServerSchedulesAsync(ct)); + Assert.Equal("P0001", failure.SqlState); + Assert.Equal(3L, await ScheduleRowCountAsync(connection, ct)); + Assert.Equal(2L, await ScalarAsync(connection, "SELECT count(*) FROM config.config_collector_schedules WHERE server_id IS NOT NULL", ct)); + Assert.Equal(Before, await RunTimesAsync(connection, ct)); + + /* The control: without the trigger the same call removes both. */ + await ExecAsync(connection, "DROP TRIGGER trg_refuse_run_time_delete ON config.config_collector_run_times", ct); + Assert.Equal(new CollectorScheduleResetCounts(ScheduleOverrides: 2, RunTimes: 2), await viewer.ResetAllServerSchedulesAsync(ct)); + Assert.Equal(1L, await ScheduleRowCountAsync(connection, ct)); + Assert.Equal($"fleet/{Daily}=0", await RunTimesAsync(connection, ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + /// A schedule Reset deletes that scope's run-time rows, in the same Save (#4938). Reset to Defaults replaces the + /// grid with the shipped defaults, which have no fixed time, so the Save leaves no run-time row for the scope: not the ones the + /// window showed, not one another writer added while the window was open, and not one for a collector this build does not + /// define. A server that "uses the default schedule" is the same Reset for that server. Every other scope keeps its rows. + [Fact] + public async Task ASaveAfterAReset_LeavesNoRunTimeRowForTheScope_AndKeepsEveryOtherScopes() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var bodySucceeded = false; + try + { + await ExecAsync(connection, + RunTimeRows + $"(NULL, '{Daily}', 120), (NULL, 'a_collector_this_build_lacks', 30), (7, '{Daily}', 300), (7, 'a_collector_this_build_lacks', 60), (8, '{Daily}', 200), (9, '{Daily}', 240)", ct); + await using var viewer = new ViewerDataService(scratch.ConnectionString); + + /* SERVER 7: it holds a time for the collector the window shows, one for a collector this build does not define, and + a third that another writer adds while the window is open. The Reset removes all three, and no other scope's. */ + await SaveAsync(viewer, 7, _ => { }, ct, resetToDefaults: true, + afterOpen: () => ExecAsync(connection, RunTimeRows + $"(7, '{AnotherDaily}', 1439)", ct)); + Assert.Equal( + $"fleet/a_collector_this_build_lacks=30,fleet/{Daily}=120,8/{Daily}=200,9/{Daily}=240", + await RunTimesAsync(connection, ct)); + + /* A time typed into the grid after the Reset is kept: the Save clears the scope, then writes what the grid holds. */ + await ExecAsync(connection, RunTimeRows + $"(8, '{AnotherDaily}', 90)", ct); + await SaveAsync(viewer, 8, editing => SetRunAt(editing, "04:00"), ct, resetToDefaults: true); + Assert.Equal( + $"fleet/a_collector_this_build_lacks=30,fleet/{Daily}=120,8/{Daily}=240,9/{Daily}=240", + await RunTimesAsync(connection, ct)); + + /* "Use default schedule" for a server is the same Reset for that server. */ + await SaveAsync(viewer, 9, _ => { }, ct, usesDefault: true, + afterOpen: () => ExecAsync(connection, RunTimeRows + $"(9, '{AnotherDaily}', 45)", ct)); + Assert.Equal( + $"fleet/a_collector_this_build_lacks=30,fleet/{Daily}=120,8/{Daily}=240", + await RunTimesAsync(connection, ct)); + + /* THE FLEET: Reset to Defaults clears the fleet's rows (and the fleet only). */ + await SaveAsync(viewer, null, _ => { }, ct, resetToDefaults: true); + Assert.Equal($"8/{Daily}=240", await RunTimesAsync(connection, ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } +} diff --git a/Darling/Darling.Tests/CollectorRunTimeViewerTests.cs b/Darling/Darling.Tests/CollectorRunTimeViewerTests.cs new file mode 100644 index 0000000000..2c2c611698 --- /dev/null +++ b/Darling/Darling.Tests/CollectorRunTimeViewerTests.cs @@ -0,0 +1,403 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using PerformanceMonitor.Analysis.Baselines; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// The Collector Schedules window's "Run at" column (#4938), without a store: what the cell holds for each scope, +/// the rows a save writes from it, the checks a save runs and the text they show, and the conversion line under the +/// grid. All of it is pure, so it is tested without a window. The store round trip is +/// . +/// +public sealed class CollectorRunTimeViewerTests +{ + private const string Daily = "index_object_stats"; + private const string AnotherDaily = "pg_index_usage_stats"; + private const string OnLoad = "server_properties"; + private const string Hourly = "wait_stats"; + + private static CollectorScheduleEditItem Item(string name, string runAt, int? frequency = null) + { + var item = CollectorSchedulePresets.BuildDefaultSchedule().Single(i => i.Name == name); + item.RunAtText = runAt; + if (frequency is int f) + { + item.FrequencyMinutes = f; + } + + return item; + } + + private static bool Valid(CollectorScheduleEditItem item, out string error) => + CollectorScheduleOverlay.ValidateSchedule(new[] { item }, out error); + + [Fact] + public void ANewItem_ShowsUseDefault_AndSoDoesEveryItemOfTheDefaultSchedule() + { + Assert.Equal("Use default", CollectorScheduleOverlay.UseDefaultRunAtText); + Assert.Equal("None", CollectorScheduleOverlay.NoRunAtText); + Assert.Equal(CollectorScheduleOverlay.UseDefaultRunAtText, new CollectorScheduleEditItem().RunAtText); + Assert.All(CollectorSchedulePresets.BuildDefaultSchedule(), i => Assert.Equal(CollectorScheduleOverlay.UseDefaultRunAtText, i.RunAtText)); + } + + [Fact] + public void TheCell_HoldsTheScopesOwnRunTime_NeverTheFleetOneOverlaidOnAServer() + { + var rows = new[] + { + new CollectorRunTimeRow(null, Daily, 120), + new CollectorRunTimeRow(2, Daily, -1), + new CollectorRunTimeRow(3, Daily, 195), + }; + + string Cell(int? serverId) => + CollectorScheduleOverlay.BuildEffectiveSchedule(Array.Empty(), rows, serverId).Single(i => i.Name == Daily).RunAtText; + + Assert.Equal("02:00", Cell(null)); + /* A server with no run-time row of its own shows "Use default", not the fleet's 02:00: if the cell took the fleet's + value, a save would write it into every server's run-time rows and "Use default" could never survive. */ + Assert.Equal("Use default", Cell(1)); + Assert.Equal("None", Cell(2)); + Assert.Equal("03:15", Cell(3)); + Assert.Equal("Use default", Cell(99)); + + Assert.All( + CollectorScheduleOverlay.BuildEffectiveSchedule(Array.Empty(), rows, 3).Where(i => i.Name != Daily), + i => Assert.Equal("Use default", i.RunAtText)); + } + + [Fact] + public void ARunTime_NeedsNoScheduleRow_AndAScheduleRowCarriesNone() + { + /* A run time lives in its own table, so a collector left at its default cadence with a run time writes a run-time + change and NO schedule row, and the schedule rows a Save writes name no run time at all. */ + Assert.Empty(CollectorScheduleOverlay.ToFleetOverrideRows(new[] { Item(Daily, "02:00") })); + + var change = Assert.Single(CollectorScheduleOverlay.ToRunTimeChanges( + new[] { Item(Daily, "02:00") }, Array.Empty(), serverId: null, usesDefault: false)); + Assert.Equal(new CollectorRunTimeChange(null, Daily, 120), change); + + var serverRows = CollectorScheduleOverlay.ToServerOverrideRows(new[] { Item(Daily, "02:00"), Item(AnotherDaily, "None") }, 5); + Assert.All(serverRows, r => Assert.Equal(5, r.ServerId)); + Assert.DoesNotContain( + typeof(CollectorScheduleRow).GetProperties(), p => p.Name.Contains("RunAt", StringComparison.Ordinal)); + } + + [Fact] + public void NoneOnTheFleet_IsNoRow_BecauseTheFleetHasNothingToStop() + { + var stored = new[] { new CollectorRunTimeRow(null, Daily, 120) }; + + /* A stored fleet time set to None is deleted, and None with nothing stored writes nothing. */ + var change = Assert.Single(CollectorScheduleOverlay.ToRunTimeChanges(new[] { Item(Daily, "None") }, stored, null, false)); + Assert.Equal(new CollectorRunTimeChange(null, Daily, null), change); + Assert.Empty(CollectorScheduleOverlay.ToRunTimeChanges(new[] { Item(Daily, "None") }, Array.Empty(), null, false)); + } + + [Fact] + public void TheRunTimeChanges_AreOnlyTheDifferences_AndNoneIsMinusOneOnAServer() + { + var stored = new[] + { + new CollectorRunTimeRow(5, Daily, 120), + new CollectorRunTimeRow(5, AnotherDaily, 300), + new CollectorRunTimeRow(5, "pg_column_stats", 60), + new CollectorRunTimeRow(6, "pg_index_bloat", 10), + new CollectorRunTimeRow(null, "pg_index_bloat", 20), + }; + + var changes = CollectorScheduleOverlay.ToRunTimeChanges( + new[] + { + Item(Daily, "02:00"), // unchanged: no change + Item(AnotherDaily, "Use default"), // had a row: delete it + Item("pg_column_stats", "None"), // 60 -> -1 + Item("pg_index_bloat", "23:59"), // no row for server 5: insert + }, + stored, 5, usesDefault: false); + + Assert.Equal( + new[] + { + new CollectorRunTimeChange(5, AnotherDaily, null), + new CollectorRunTimeChange(5, "pg_column_stats", -1), + new CollectorRunTimeChange(5, "pg_index_bloat", 23 * 60 + 59), + }, + changes); + + /* "Use the default schedule" deletes every run-time row the server has, and no other server's or the fleet's. */ + var reset = CollectorScheduleOverlay.ToRunTimeChanges(CollectorSchedulePresets.BuildDefaultSchedule(), stored, 5, usesDefault: true); + Assert.Equal(3, reset.Count); + Assert.All(reset, c => { Assert.Equal(5, c.ServerId); Assert.Null(c.RunAtMinute); }); + } + + [Fact] + public void AServerWithOnlyARunTimeRow_IsCustom_NotUsingTheDefault() + { + var runTimes = new[] { new CollectorRunTimeRow(7, Daily, 90) }; + Assert.True(CollectorScheduleOverlay.ServerHasOverride(Array.Empty(), runTimes, 7)); + Assert.False(CollectorScheduleOverlay.ServerHasOverride(Array.Empty(), runTimes, 8)); + Assert.False(CollectorScheduleOverlay.ServerHasOverride(Array.Empty(), 7)); + } + + [Theory] + [InlineData("")] + [InlineData(" ")] + [InlineData("Use default")] + [InlineData("use default")] + [InlineData("default")] + public void TryParseRunAt_ReadsUseDefaultAsNull(string text) + { + Assert.True(CollectorScheduleOverlay.TryParseRunAt(text, out var minute, out var error)); + Assert.Null(minute); + Assert.Equal("", error); + } + + [Theory] + [InlineData("None")] + [InlineData("none")] + [InlineData(" NONE ")] + public void TryParseRunAt_ReadsNoneAsMinusOne(string text) + { + Assert.True(CollectorScheduleOverlay.TryParseRunAt(text, out var minute, out _)); + Assert.Equal(-1, minute); + } + + [Theory] + [InlineData("00:00", 0)] + [InlineData("02:00", 120)] + [InlineData(" 02:00 ", 120)] + [InlineData("13:45", 825)] + [InlineData("23:59", 1439)] + public void TryParseRunAt_ReadsATwentyFourHourTime(string text, int expected) + { + Assert.True(CollectorScheduleOverlay.TryParseRunAt(text, out var minute, out var error)); + Assert.Equal(expected, minute); + Assert.Equal("", error); + } + + [Theory] + [InlineData("24:00")] + [InlineData("02:60")] + [InlineData("2:5")] + [InlineData("12:3")] + [InlineData("1200")] + [InlineData("02-00")] + [InlineData("02:00pm")] + [InlineData("-1:00")] + [InlineData("abc")] + [InlineData("nonsense")] + public void TryParseRunAt_RefusesAnythingElse_WithTheSharedText(string text) + { + Assert.False(CollectorScheduleOverlay.TryParseRunAt(text, out var minute, out var error)); + Assert.Null(minute); + Assert.Equal("Run at must be a 24-hour time from 00:00 to 23:59, such as 02:00.", error); + Assert.Equal(CollectorRunTime.InvalidRunAtMessage, error); + } + + [Theory] + [InlineData("24:00")] + [InlineData("2:5")] + [InlineData("noon")] + public void ASaveWithABadRunTime_IsRefused_NamingTheCollectorAndTheRule(string text) + { + Assert.False(Valid(Item(Daily, text), out var error)); + Assert.Contains(CollectorRunTime.InvalidRunAtMessage, error, StringComparison.Ordinal); + Assert.Contains($"'{Daily}'", error, StringComparison.Ordinal); + } + + [Theory] + [InlineData(1)] + [InlineData(60)] + [InlineData(1439)] + [InlineData(1441)] + [InlineData(2000)] + public void ARunTime_OnAnIntervalThatIsNotWholeDays_IsRefused_WithTheSharedWholeDayText(int minutes) + { + Assert.False(Valid(Item(Daily, "02:00", frequency: minutes), out var error)); + Assert.Contains(CollectorRunTime.IntervalRefusalMessage(Daily, minutes), error, StringComparison.Ordinal); + } + + [Fact] + public void TheHourlyRefusal_ReadsTheSharedSentence() + { + Assert.False(Valid(Item(Hourly, "02:00"), out var error)); + Assert.Contains( + "'wait_stats' runs every 1 minutes. A run time works only for a collector that runs once a day or less often " + + "(1440 minutes, or a multiple of 1440). Change the frequency or clear the run time.", + error, StringComparison.Ordinal); + } + + [Fact] + public void ARunTime_IsAllowedOnADailyCollector_AMultipleOfADay_AndAnOnLoadCollector() + { + Assert.True(Valid(Item(Daily, "02:00"), out _)); + Assert.True(Valid(Item(Daily, "02:00", frequency: 2880), out _)); + Assert.True(Valid(Item(Daily, "02:00", frequency: 10080), out _)); + /* An on-load collector (0) recaptures daily, so it is judged as a daily one. */ + Assert.True(Valid(Item(OnLoad, "02:00"), out _)); + } + + [Fact] + public void NoneAndUseDefault_AreAlwaysAllowed_EvenOnAnHourlyCollector_BecauseClearingIsTheRemedy() + { + Assert.True(Valid(Item(Hourly, "None"), out var error)); + Assert.Equal("", error); + Assert.True(Valid(Item(Hourly, "Use default"), out _)); + Assert.True(Valid(Item(Hourly, ""), out _)); + } + + /* ------------------------------ the conversion line ------------------------------ */ + + private static readonly DateTime Now = new(2026, 10, 2, 12, 0, 0, DateTimeKind.Utc); + + private const int ServerId = 41; + + private static string Describe( + string runAtText, int frequencyMinutes = 1440, int? serverId = ServerId, int? fleetRunAtMinute = null, + ServerClock? clock = null, bool azureSqlDatabase = false, string collector = Daily) => + CollectorScheduleRunAtText.Describe(collector, runAtText, frequencyMinutes, serverId, fleetRunAtMinute, clock, azureSqlDatabase, Now); + + [Fact] + public void AFleetRow_ShowsTheSixtyMinuteSpread() + { + var text = Describe("02:00", serverId: null, clock: null); + Assert.Contains("between 02:00 and 03:00", text, StringComparison.Ordinal); + Assert.Contains("server time", text, StringComparison.Ordinal); + + Assert.Contains("between 23:30 and 00:30", Describe("23:30", serverId: null), StringComparison.Ordinal); + } + + [Fact] + public void AFleetRowWithNoTime_SaysItHasNoFixedTime() + { + var text = Describe("Use default", serverId: null); + Assert.Contains("No fixed run time", text, StringComparison.Ordinal); + Assert.DoesNotContain("between", text, StringComparison.Ordinal); + } + + [Fact] + public void AServerRow_ShowsItsExactMinute_AndTheUtcConversion() + { + var clock = ServerClock.FixedOffset(-240); + var slot = CollectorRunTime.SlotUtc(new DateOnly(2026, 10, 2), 120, ServerId, clock.ToUtc); + var local = clock.ToServerLocal(slot); + + var text = Describe("02:00", clock: clock); + + /* The same shape as the plan's example, "02:00 server time (06:00 UTC today)", with this server's own spread + added to the time: the slot is run time + CadencePhaseOffset(serverId, 3600). */ + Assert.Contains($"{local:HH:mm} server time ({slot:HH:mm} UTC today)", text, StringComparison.Ordinal); + Assert.Equal(CollectorCadence.CadencePhaseOffset(ServerId, 3600), CollectorRunTime.Spread(ServerId)); + } + + [Fact] + public void AServerRow_ShowsTheNextRun_ForADailyCollector() + { + var clock = ServerClock.FixedOffset(-240); + /* Now is 12:00 UTC, after today's slot (about 06:xx UTC), so the next run is tomorrow's. */ + var next = CollectorRunTime.SlotUtc(new DateOnly(2026, 10, 3), 120, ServerId, clock.ToUtc); + + var text = Describe("02:00", clock: clock); + + Assert.Contains($"Next run: {next:yyyy-MM-dd HH:mm} UTC", text, StringComparison.Ordinal); + } + + [Fact] + public void NowBeforeTodaysSlot_PutsTheNextRunToday() + { + var clock = ServerClock.Utc; + var today = CollectorRunTime.SlotUtc(new DateOnly(2026, 10, 2), 18 * 60, ServerId, clock.ToUtc); + + var text = Describe("18:00", clock: clock); + + Assert.Contains($"Next run: {today:yyyy-MM-dd HH:mm} UTC", text, StringComparison.Ordinal); + } + + [Fact] + public void AZoneAheadOfUtc_SaysTheUtcDayTheSlotFallsOn() + { + /* 02:00 at +10:00 is 16:00 UTC the day before. */ + var clock = ServerClock.FixedOffset(600); + var slot = CollectorRunTime.SlotUtc(new DateOnly(2026, 10, 2), 120, ServerId, clock.ToUtc); + Assert.Equal(new DateTime(2026, 10, 1), slot.Date); + + Assert.Contains($"{slot:HH:mm} UTC yesterday", Describe("02:00", clock: clock), StringComparison.Ordinal); + } + + [Fact] + public void AServerWithNoClockYet_ReadsTheTimeAsUtc_AndSaysSo() + { + var text = Describe("02:00", clock: null); + + Assert.Contains("not known yet", text, StringComparison.Ordinal); + Assert.Contains("UTC", text, StringComparison.Ordinal); + var slot = CollectorRunTime.SlotUtc(new DateOnly(2026, 10, 2), 120, ServerId, CollectorRunTime.LocalIsUtc); + Assert.Contains($"{slot:HH:mm} server time", text, StringComparison.Ordinal); + } + + [Fact] + public void AzureSqlDatabase_SaysItAlwaysReportsUtc() + { + var text = Describe("02:00", clock: ServerClock.Utc, azureSqlDatabase: true); + + Assert.Contains("Azure SQL Database always reports UTC", text, StringComparison.Ordinal); + } + + [Fact] + public void AServerRow_UsingTheFleetsTime_NamesTheFleetTime_AndStillShowsItsOwnMinute() + { + var clock = ServerClock.Utc; + var slot = CollectorRunTime.SlotUtc(new DateOnly(2026, 10, 2), 120, ServerId, clock.ToUtc); + + var text = Describe("Use default", fleetRunAtMinute: 120, clock: clock); + + Assert.Contains("fleet-wide run time", text, StringComparison.Ordinal); + Assert.Contains($"{slot:HH:mm} server time", text, StringComparison.Ordinal); + } + + [Fact] + public void AServerRow_WithNoneOrNoTimeAnywhere_SaysThereIsNoFixedTime() + { + Assert.Contains("No fixed run time on this server", Describe("None", fleetRunAtMinute: 120), StringComparison.Ordinal); + Assert.DoesNotContain("Next run", Describe("None", fleetRunAtMinute: 120), StringComparison.Ordinal); + Assert.Contains("No fixed run time", Describe("Use default", fleetRunAtMinute: null), StringComparison.Ordinal); + } + + [Fact] + public void ATimeThatCannotApply_ShowsTheRefusalInsteadOfAConversion() + { + var text = Describe("02:00", frequencyMinutes: 60, collector: Hourly, clock: ServerClock.Utc); + + Assert.Contains(CollectorRunTime.IntervalRefusalMessage(Hourly, 60), text, StringComparison.Ordinal); + Assert.DoesNotContain("Next run", text, StringComparison.Ordinal); + } + + [Fact] + public void ABadTime_ShowsTheSharedMessage() + { + Assert.Equal(CollectorRunTime.InvalidRunAtMessage, Describe("25:61")); + } + + [Fact] + public void ACollectorThatRunsEveryFewDays_DoesNotClaimANextRunDate() + { + var text = Describe("02:00", frequencyMinutes: 2880, clock: ServerClock.Utc); + + Assert.Contains("every 2 days", text, StringComparison.Ordinal); + Assert.DoesNotContain("Next run:", text, StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/CollectorScheduleEditorSaveGuardTests.cs b/Darling/Darling.Tests/CollectorScheduleEditorSaveGuardTests.cs new file mode 100644 index 0000000000..789080befe --- /dev/null +++ b/Darling/Darling.Tests/CollectorScheduleEditorSaveGuardTests.cs @@ -0,0 +1,146 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.RegularExpressions; +using PerformanceMonitor.Analysis.Baselines; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// A Save of the Collector Schedules window that follows a failed read (#4938): no successful read, no write. A failed read leaves +/// the window holding nothing for it, which looks the same as a store with nothing stored, so a Save must not write what the window +/// never saw. The Save's contents come from a pure plan (), tested here without +/// a window or a store, and source pins on the window check that it records a failed read, keeps its warning and sends the plan. +/// +[Trait("Stage", "Guard")] +public sealed class CollectorScheduleEditorSaveGuardTests +{ + private const string Daily = "index_object_stats"; + private const int ServerId = 41; + + /// The grid of a scope the window read nothing for: every cell the code default, every Run at cell "Use default". + private static List Grid() => CollectorSchedulePresets.BuildDefaultSchedule(); + + private static string Window() => CSharpSourceWalker.StripCommentsAndStrings( + RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "CollectorScheduleEditorWindow.xaml.cs")); + + /// A server whose only override is a run-time row (set from the command line, say) opens on "Use default schedule" when + /// the run-time read fails, because the window then holds no run times. A plain Save of that window, nothing changed, must not + /// send the clear that "Use default schedule" otherwise sends: that clear deleted the run-time row it never showed. + [Fact] + public void ASave_AfterAFailedRunTimeRead_ThatChangedNothing_SendsNoRunTimeDeleteOrClear() + { + var plan = CollectorScheduleOverlay.BuildSavePlan( + Grid(), Array.Empty(), ServerId, usesDefault: true, resetToDefaults: false, schedulesRead: true, runTimesRead: false); + + Assert.Null(plan.Refusal); + Assert.Empty(plan.RunTimeChanges); + Assert.False(plan.ClearRunTimes); + Assert.Empty(plan.Rows); + } + + /// The control: the same window after a read that worked still clears the scope's run times on "Use default schedule", + /// which is what that checkbox means. The guard is for the unread case only. + [Fact] + public void ASave_AfterASuccessfulRead_OnUseDefaultSchedule_StillClearsTheServersRunTimes() + { + var plan = CollectorScheduleOverlay.BuildSavePlan( + Grid(), new[] { new CollectorRunTimeRow(ServerId, Daily, 300) }, ServerId, usesDefault: true, resetToDefaults: false, schedulesRead: true, runTimesRead: true); + + Assert.Null(plan.Refusal); + Assert.True(plan.ClearRunTimes); + } + + /// A server that does customize its schedule still saves it when the run-time read failed, and sends no run-time change + /// with it: not a delete, not an upsert, not a clear. + [Fact] + public void ASave_AfterAFailedRunTimeRead_StillSavesTheSchedule_WithNoRunTimeChange() + { + var grid = Grid(); + grid.Single(i => i.Name == "wait_stats").FrequencyMinutes = 10; + + var plan = CollectorScheduleOverlay.BuildSavePlan( + grid, Array.Empty(), ServerId, usesDefault: false, resetToDefaults: false, schedulesRead: true, runTimesRead: false); + + Assert.Null(plan.Refusal); + Assert.Contains(plan.Rows, r => r.CollectorName == "wait_stats" && r.FrequencyMinutes == 10); + Assert.Empty(plan.RunTimeChanges); + Assert.False(plan.ClearRunTimes); + } + + /// A run time typed into a cell, or a Reset to Defaults, is something the user asked for. With the run times unread it + /// cannot be written safely, so the Save is refused outright and says why, and sends nothing, instead of saving the rest and + /// quietly dropping that. + [Fact] + public void ASave_AfterAFailedRunTimeRead_IsRefused_WhenItWouldDropATypedRunTimeOrAReset() + { + var typed = Grid(); + typed.Single(i => i.Name == Daily).RunAtText = "02:00"; + + var withTime = CollectorScheduleOverlay.BuildSavePlan( + typed, Array.Empty(), ServerId, usesDefault: false, resetToDefaults: false, schedulesRead: true, runTimesRead: false); + var withReset = CollectorScheduleOverlay.BuildSavePlan( + Grid(), Array.Empty(), ServerId, usesDefault: false, resetToDefaults: true, schedulesRead: true, runTimesRead: false); + + foreach (var plan in new[] { withTime, withReset }) + { + Assert.Equal(CollectorScheduleOverlay.RunTimesUnreadRefusal, plan.Refusal); + Assert.Empty(plan.Rows); + Assert.Empty(plan.RunTimeChanges); + Assert.False(plan.ClearRunTimes); + } + } + + /// The schedule read has the same shape: a failed read leaves the window on the code defaults, and a Save of that + /// replaces the scope's stored rows with nothing. After a failed schedule read the Save is refused and sends nothing, for the + /// fleet scope and for a server alike. + [Theory] + [InlineData(null, false)] + [InlineData(ServerId, true)] + [InlineData(ServerId, false)] + public void ASave_AfterAFailedScheduleRead_IsRefused_AndSendsNothing(int? serverId, bool usesDefault) + { + var plan = CollectorScheduleOverlay.BuildSavePlan( + Grid(), Array.Empty(), serverId, usesDefault, resetToDefaults: false, schedulesRead: false, runTimesRead: true); + + Assert.Equal(CollectorScheduleOverlay.SchedulesUnreadRefusal, plan.Refusal); + Assert.Empty(plan.Rows); + Assert.Empty(plan.RunTimeChanges); + Assert.False(plan.ClearRunTimes); + } + + /// The window records a failed read, keeps its warning in front of every later status text, and builds its Save from + /// the plan. Its status line is written in ONE place, because any other assignment to it is what overwrote the warning. + [Fact] + public void TheWindow_KeepsAFailedReadsWarningVisible_AndBuildsItsSaveFromThePlan() + { + var window = Window(); + + Assert.Single(Regex.Matches(window, @"StatusText\.Text = ")); + Assert.Contains("CollectorScheduleOverlay.BuildSavePlan(", window, StringComparison.Ordinal); + Assert.Contains("schedulesRead: SchedulesRead", window, StringComparison.Ordinal); + Assert.Contains("runTimesRead: RunTimesRead", window, StringComparison.Ordinal); + Assert.Contains(".SaveCollectorScheduleAsync(_scopeServerId, plan.Rows, plan.RunTimeChanges, plan.ClearRunTimes)", window, StringComparison.Ordinal); + + /* Both reads set a warning of their own when they fail, and the same reads clear it when they work. */ + var runTimeRead = window[window.IndexOf("private async Task ReloadRunTimesAsync(", StringComparison.Ordinal)..window.IndexOf("private void ShowStatus(", StringComparison.Ordinal)]; + Assert.Contains("_runTimeReadWarning = ", runTimeRead, StringComparison.Ordinal); + var scheduleRead = window[window.IndexOf("private async Task ReloadSchedulesAsync(", StringComparison.Ordinal)..window.IndexOf("private async Task ReloadRunTimesAsync(", StringComparison.Ordinal)]; + Assert.Contains("_scheduleReadWarning = ", scheduleRead, StringComparison.Ordinal); + + /* "Apply Default to All Servers" re-reads through the same two steps, so a re-read that fails guards the next Save too. */ + var applyAll = window[window.IndexOf("private async void ApplyDefaultToAll_Click(", StringComparison.Ordinal)..window.IndexOf("private void CancelButton_Click(", StringComparison.Ordinal)]; + Assert.Contains("await ReloadSchedulesAsync();", applyAll, StringComparison.Ordinal); + Assert.Contains("await ReloadRunTimesAsync();", applyAll, StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/CollectorScheduleResetStatusTests.cs b/Darling/Darling.Tests/CollectorScheduleResetStatusTests.cs new file mode 100644 index 0000000000..a6efd2539b --- /dev/null +++ b/Darling/Darling.Tests/CollectorScheduleResetStatusTests.cs @@ -0,0 +1,46 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// The status line after "Apply Default to All Servers" (#4938): the per-server schedule overrides and the per-server run times the +/// call removed are counted apart and each named, singular or plural, so a reset of run times alone no longer reads as a reset of +/// schedule overrides. The store's own counts are checked in . +/// +public sealed class CollectorScheduleResetStatusTests +{ + private const string ResetTail = " — every server now follows the fleet default."; + + [Theory] + [InlineData(0, 0, "No per-server overrides to reset — every server already follows the fleet default.")] + [InlineData(1, 0, "Reset 1 per-server schedule override" + ResetTail)] + [InlineData(3, 0, "Reset 3 per-server schedule overrides" + ResetTail)] + [InlineData(0, 1, "Reset 1 per-server run time" + ResetTail)] + [InlineData(0, 2, "Reset 2 per-server run times" + ResetTail)] + [InlineData(1, 2, "Reset 1 per-server schedule override and 2 per-server run times" + ResetTail)] + [InlineData(2, 1, "Reset 2 per-server schedule overrides and 1 per-server run time" + ResetTail)] + public void TheResetStatus_CountsScheduleOverridesAndRunTimesApart_AndNamesEach(int scheduleOverrides, int runTimes, string expected) + { + Assert.Equal(expected, CollectorScheduleOverlay.FormatResetStatus(new CollectorScheduleResetCounts(scheduleOverrides, runTimes))); + } + + /// The window shows the text the formatter builds from the call's two counts, and no longer one number for both. + [Fact] + public void TheWindow_ShowsTheCountedResetStatus_NotOneNumberForBothKinds() + { + var source = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "CollectorScheduleEditorWindow.xaml.cs"); + + Assert.Contains("CollectorScheduleOverlay.FormatResetStatus(removed)", source, StringComparison.Ordinal); + Assert.DoesNotContain("schedule override(s)", source, StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/CollectorScheduleUseDefaultRunAtLineTests.cs b/Darling/Darling.Tests/CollectorScheduleUseDefaultRunAtLineTests.cs new file mode 100644 index 0000000000..fcdf64f8b3 --- /dev/null +++ b/Darling/Darling.Tests/CollectorScheduleUseDefaultRunAtLineTests.cs @@ -0,0 +1,81 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using PerformanceMonitor.Analysis.Baselines; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// The Run at line under the Collector Schedules grid, for a server that is on "Use default schedule" (#4938). The grid then shows +/// the FLEET's schedule, so the selected row's Run at cell holds the fleet's own time, and the line used to describe that as a +/// time the server had of its own. It now says the server uses the fleet-wide run time, the same sentence as for a server row set to +/// "Use default". All of it is pure, so it is tested without a window. +/// +public sealed class CollectorScheduleUseDefaultRunAtLineTests +{ + private const string Daily = "index_object_stats"; + private const int ServerId = 41; + private static readonly DateTime Now = new(2026, 10, 2, 12, 0, 0, DateTimeKind.Utc); + + [Fact] + public void UnderUseDefaultSchedule_TheRunAtLine_SaysTheServerUsesTheFleetWideRunTime() + { + var fleetGrid = CollectorScheduleOverlay.BuildEffectiveSchedule( + Array.Empty(), new[] { new CollectorRunTimeRow(null, Daily, 120) }, null); + var cell = fleetGrid.Single(i => i.Name == Daily); + Assert.Equal("02:00", cell.RunAtText); + + var line = CollectorScheduleRunAtText.Describe( + Daily, cell.RunAtText, cell.FrequencyMinutes, ServerId, 120, ServerClock.Utc, false, Now, usesDefaultSchedule: true); + var slot = CollectorRunTime.SlotUtc(new DateOnly(2026, 10, 2), 120, ServerId, ServerClock.Utc.ToUtc); + + Assert.StartsWith("Uses the fleet-wide run time (02:00). ", line, StringComparison.Ordinal); + Assert.Contains($"{slot:HH:mm} server time", line, StringComparison.Ordinal); + + /* The same cell on a server that customizes its schedule is that server's own time, and says nothing about the fleet. */ + var own = CollectorScheduleRunAtText.Describe( + Daily, cell.RunAtText, cell.FrequencyMinutes, ServerId, 120, ServerClock.Utc, false, Now); + Assert.DoesNotContain("fleet-wide", own, StringComparison.Ordinal); + } + + [Fact] + public void UnderUseDefaultSchedule_WithNoFleetWideRunTime_TheRunAtLine_SaysThereIsNoFixedTime() + { + var line = CollectorScheduleRunAtText.Describe( + Daily, "Use default", 1440, ServerId, null, ServerClock.Utc, false, Now, usesDefaultSchedule: true); + + Assert.StartsWith("No fixed run time", line, StringComparison.Ordinal); + Assert.DoesNotContain("fleet-wide", line, StringComparison.Ordinal); + } + + /// The fleet scope has no "Use default schedule", so the flag changes nothing on a fleet row. + [Fact] + public void TheFleetRow_IgnoresUseDefaultSchedule() + { + var plain = CollectorScheduleRunAtText.Describe(Daily, "02:00", 1440, null, 120, ServerClock.Utc, false, Now); + var flagged = CollectorScheduleRunAtText.Describe(Daily, "02:00", 1440, null, 120, ServerClock.Utc, false, Now, usesDefaultSchedule: true); + + Assert.Equal(plain, flagged); + } + + [Fact] + public void TheWindow_PassesUseDefaultScheduleToTheRunAtLine() + { + var window = CSharpSourceWalker.StripCommentsAndStrings( + RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "CollectorScheduleEditorWindow.xaml.cs")); + var refresh = window[window.IndexOf("private void RefreshRunAtDetail(", StringComparison.Ordinal)..window.IndexOf("private void RebuildEditingForScope(", StringComparison.Ordinal)]; + + Assert.Contains("usesDefaultSchedule:", refresh, StringComparison.Ordinal); + Assert.Contains("UseDefaultCheckBox.IsChecked == true", refresh, StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/CollectorStallProbeStoreTests.cs b/Darling/Darling.Tests/CollectorStallProbeStoreTests.cs index b7dab3f540..518a24a0ca 100644 --- a/Darling/Darling.Tests/CollectorStallProbeStoreTests.cs +++ b/Darling/Darling.Tests/CollectorStallProbeStoreTests.cs @@ -266,6 +266,19 @@ throughput half of the firing condition would be inert. */ Assert.Contains("using var stallProbeArm = StallProbeArm.Start(", runner, StringComparison.Ordinal); } + /// + /// The probe query matches this service's own session by APP_NAME() and HOST_NAME(), so + /// it binds no parameters: the runner must send the text as it ships, with none attached. + /// + [Fact] + public void TheProbeQuery_IsSentWithoutParameters() + { + var runner = ReadSource("Darling/PerformanceMonitor.Darling.Service/StallWaitProbeRunner.cs"); + + Assert.Contains("new CollectorQuery(StallWaitProbePolicy.QueryText)", runner, StringComparison.Ordinal); + Assert.DoesNotContain("new CollectorQuery(StallWaitProbePolicy.QueryText,", runner, StringComparison.Ordinal); + } + private static string RungSql() => PgMigrations.Scripts.Single(s => s.Version == RungVersion).Sql; private static bool IsTriggerSideColumn(string name) => diff --git a/Darling/Darling.Tests/CollectorStateContractTests.cs b/Darling/Darling.Tests/CollectorStateContractTests.cs index c58bd990ba..6349844003 100644 --- a/Darling/Darling.Tests/CollectorStateContractTests.cs +++ b/Darling/Darling.Tests/CollectorStateContractTests.cs @@ -38,6 +38,7 @@ namespace Darling.Tests; /// at source, the idiom this suite already uses for cross-artifact contracts. /// [Collection("live-postgres")] +[Trait("Reads", "Lite")] public sealed class CollectorStateContractTests { private const string RepoRootNotFound = "repo root not found -- the source pin cannot run"; diff --git a/Darling/Darling.Tests/ColumnFilterPersistenceTests.cs b/Darling/Darling.Tests/ColumnFilterPersistenceTests.cs new file mode 100644 index 0000000000..443612500d --- /dev/null +++ b/Darling/Darling.Tests/ColumnFilterPersistenceTests.cs @@ -0,0 +1,827 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Windows.Controls; +using System.Windows.Data; +using PerformanceMonitor.Common; +using PerformanceMonitor.Ui; +using Xunit; + +namespace Darling.Tests; + +/// +/// Column filters kept across a refresh and a restart (#5565): the store file (round trip, per-server separation, +/// least-recently-used bound, a bad file ignored, a cleared filter removed, no row data), and the grid manager that +/// loads them on the first scoped refresh, keeps them across UpdateData, and clears them on "Clear all filters". +/// +public sealed class ColumnFilterPersistenceTests : IDisposable +{ + private readonly string _dir = Path.Combine(Path.GetTempPath(), "colfilter-" + Guid.NewGuid().ToString("N")); + private string FilePath => Path.Combine(_dir, "column-filters.json"); + + public ColumnFilterPersistenceTests() => Directory.CreateDirectory(_dir); + + public void Dispose() + { + ColumnFilterStore.Current = null; + try { Directory.Delete(_dir, true); } catch (IOException) { } + } + + private ColumnFilterStore NewStore(List? warnings = null) => + new(FilePath, warnings is null ? null : warnings.Add, TimeSpan.FromHours(1)); + + private static ColumnFilterState Hide(string column, params string[] values) + { + var state = new ColumnFilterState { ColumnName = column, ValueMode = ColumnValueMode.Hide }; + foreach (var v in values) state.Values.Add(v); + return state; + } + + [Fact] + public void A_filter_saved_comes_back_after_a_restart_with_its_mode_values_and_text_match() + { + var first = NewStore(); + var saved = Hide("LoginName", "job_svc", "NT AUTHORITY\\SYSTEM"); + saved.ValueBlank = true; + saved.Operator = FilterOperator.StartsWith; + saved.Value = "sa"; + first.Save("srv-a", "QuerySnapshotsGrid", new[] { saved }); + first.Flush(); + + var restarted = NewStore(); + var loaded = Assert.Single(restarted.Load("srv-a", "QuerySnapshotsGrid")); + + Assert.Equal("LoginName", loaded.ColumnName); + Assert.Equal(ColumnValueMode.Hide, loaded.ValueMode); + Assert.True(loaded.Values.SetEquals(new[] { "job_svc", "NT AUTHORITY\\SYSTEM" })); + Assert.True(loaded.ValueBlank); + Assert.Equal(FilterOperator.StartsWith, loaded.Operator); + Assert.Equal("sa", loaded.Value); + } + + [Fact] + public void Filters_are_kept_apart_per_server_and_per_grid() + { + var store = NewStore(); + store.Save("srv-a", "GridOne", new[] { Hide("LoginName", "a") }); + store.Save("srv-b", "GridOne", new[] { Hide("LoginName", "b") }); + store.Save("srv-a", "GridTwo", new[] { Hide("LoginName", "c") }); + store.Flush(); + + var restarted = NewStore(); + Assert.Equal("a", Assert.Single(restarted.Load("srv-a", "GridOne")).Values.Single()); + Assert.Equal("b", Assert.Single(restarted.Load("srv-b", "GridOne")).Values.Single()); + Assert.Equal("c", Assert.Single(restarted.Load("srv-a", "GridTwo")).Values.Single()); + Assert.Empty(restarted.Load("srv-c", "GridOne")); + } + + [Fact] + public void A_cleared_filter_is_removed_from_the_file() + { + var store = NewStore(); + store.Save("srv-a", "GridOne", new[] { Hide("LoginName", "a") }); + store.Flush(); + store.Save("srv-a", "GridOne", new[] { new ColumnFilterState { ColumnName = "LoginName" } }); + store.Flush(); + + Assert.Empty(NewStore().Load("srv-a", "GridOne")); + Assert.DoesNotContain("srv-a", File.ReadAllText(FilePath), StringComparison.Ordinal); + } + + [Fact] + public void Only_the_200_most_recently_used_grids_are_kept() + { + var store = NewStore(); + for (var i = 0; i < 205; i++) + store.Save("srv", "Grid" + i, new[] { Hide("c", "v") }); + store.Load("srv", "Grid0"); // not evicted yet? it was: the first five went when 201..205 arrived + store.Flush(); + + var restarted = NewStore(); + Assert.Empty(restarted.Load("srv", "Grid0")); + Assert.Empty(restarted.Load("srv", "Grid4")); + Assert.Single(restarted.Load("srv", "Grid5")); + Assert.Single(restarted.Load("srv", "Grid204")); + } + + [Fact] + public void Using_a_grid_keeps_it_when_older_ones_are_dropped() + { + var store = NewStore(); + for (var i = 0; i < 200; i++) + store.Save("srv", "Grid" + i, new[] { Hide("c", "v") }); + store.Load("srv", "Grid0"); // the oldest becomes the newest + store.Save("srv", "Extra", new[] { Hide("c", "v") }); // 201st: Grid1 goes, not Grid0 + store.Flush(); + + var restarted = NewStore(); + Assert.Single(restarted.Load("srv", "Grid0")); + Assert.Empty(restarted.Load("srv", "Grid1")); + Assert.Single(restarted.Load("srv", "Extra")); + } + + [Fact] + public void A_column_naming_more_than_1000_values_keeps_its_text_match_and_loses_only_the_value_part() + { + var store = NewStore(); + var big = Hide("Big", Enumerable.Range(0, 1001).Select(i => "v" + i).ToArray()); + big.ValueBlank = true; + big.Value = "v1"; + var bigNoText = Hide("BigNoText", Enumerable.Range(0, 1001).Select(i => "v" + i).ToArray()); + var edge = Hide("Edge", Enumerable.Range(0, 1000).Select(i => "v" + i).ToArray()); + store.Save("srv", "Grid", new[] { big, bigNoText, edge }); + store.Flush(); + + var loaded = NewStore().Load("srv", "Grid"); + + /* the web page does the same: a cut set would change what it hides, but the text match fits and stays */ + Assert.Equal(new[] { "Big", "Edge" }, loaded.Select(f => f.ColumnName).OrderBy(n => n)); + var kept = loaded.Single(f => f.ColumnName == "Big"); + Assert.Equal(ColumnValueMode.None, kept.ValueMode); + Assert.Empty(kept.Values); + Assert.False(kept.ValueBlank); + Assert.Equal("v1", kept.Value); + Assert.Equal(1000, loaded.Single(f => f.ColumnName == "Edge").Values.Count); + } + + [Fact] + public void A_text_match_on_a_column_that_gets_no_list_is_never_written_but_one_on_a_list_column_is() + { + var store = NewStore(); + var statement = new ColumnFilterState { ColumnName = "QueryText", Value = "salary > 90000" }; + var script = new ColumnFilterState { ColumnName = "ImplementationScript", Value = "force_plan" }; + var error = new ColumnFilterState { ColumnName = "LastError", Value = "login failed for" }; + var login = new ColumnFilterState { ColumnName = "LoginName", Value = "svc_" }; + store.Save("srv", "Grid", new[] { statement, script, error, login }); + store.Flush(); + + var text = File.ReadAllText(FilePath); + Assert.DoesNotContain("salary", text, StringComparison.Ordinal); + Assert.DoesNotContain("force_plan", text, StringComparison.Ordinal); + Assert.DoesNotContain("login failed", text, StringComparison.Ordinal); + Assert.Equal("LoginName", Assert.Single(NewStore().Load("srv", "Grid")).ColumnName); + + /* and one planted in the file is not loaded either */ + File.WriteAllText(FilePath, "{\"Version\":1,\"Grids\":[{\"Server\":\"s\",\"Grid\":\"g\",\"Columns\":[" + + "{\"Column\":\"QueryText\",\"Operator\":\"Contains\",\"Value\":\"x\"},{\"Column\":\"LoginName\",\"Operator\":\"Contains\",\"Value\":\"y\"}]}]}"); + Assert.Equal("LoginName", Assert.Single(NewStore().Load("s", "g")).ColumnName); + } + + [Fact] + public void The_stored_limits_apply_when_the_file_is_read_too() + { + var many = string.Join(",", Enumerable.Range(0, 1001).Select(i => "\"v" + i + "\"")); + var longValue = new string('x', ColumnFilterStore.MaxValueLength + 1); + var grids = new List + { + /* too many values: the value part goes, the text match stays */ + "{\"Server\":\"s\",\"Grid\":\"many\",\"Columns\":[{\"Column\":\"LoginName\",\"Operator\":\"Contains\",\"Value\":\"t\",\"ValueMode\":\"Hide\",\"Values\":[" + many + "]}]}", + /* a value too long: the same */ + "{\"Server\":\"s\",\"Grid\":\"longvalue\",\"Columns\":[{\"Column\":\"LoginName\",\"Value\":\"t\",\"ValueMode\":\"Hide\",\"Values\":[\"" + longValue + "\"]}]}", + /* a text match too long: it goes, the value part stays */ + "{\"Server\":\"s\",\"Grid\":\"longtext\",\"Columns\":[{\"Column\":\"LoginName\",\"Value\":\"" + longValue + "\",\"ValueMode\":\"Hide\",\"Values\":[\"a\"]}]}", + /* a mode and an operator this version does not know: not match-all */ + "{\"Server\":\"s\",\"Grid\":\"badenum\",\"Columns\":[{\"Column\":\"LoginName\",\"Operator\":\"99\",\"ValueMode\":\"99\",\"Values\":[\"a\"]}]}", + /* a column name that is too long */ + "{\"Server\":\"s\",\"Grid\":\"longcolumn\",\"Columns\":[{\"Column\":\"" + longValue + "\",\"Value\":\"t\"}]}", + }; + File.WriteAllText(FilePath, "{\"Version\":1,\"Grids\":[" + string.Join(",", grids) + "]}"); + var warnings = new List(); + var store = NewStore(warnings); + + var many1 = Assert.Single(store.Load("s", "many")); + Assert.Equal(ColumnValueMode.None, many1.ValueMode); + Assert.Equal("t", many1.Value); + var longv = Assert.Single(store.Load("s", "longvalue")); + Assert.Equal(ColumnValueMode.None, longv.ValueMode); + Assert.Equal("t", longv.Value); + var longt = Assert.Single(store.Load("s", "longtext")); + Assert.Equal("", longt.Value); + Assert.Equal(ColumnValueMode.Hide, longt.ValueMode); + Assert.Empty(store.Load("s", "badenum")); + Assert.Empty(store.Load("s", "longcolumn")); + Assert.Empty(warnings); + } + + [Fact] + public void A_file_with_more_than_200_grids_or_200_columns_keeps_only_the_newest_grids_and_the_first_columns() + { + var grids = Enumerable.Range(0, 300).Select(i => + "{\"Server\":\"s\",\"Grid\":\"g" + i + "\",\"Columns\":[{\"Column\":\"LoginName\",\"Value\":\"t\"}]}"); + File.WriteAllText(FilePath, "{\"Version\":1,\"Grids\":[" + string.Join(",", grids) + "]}"); + var store = NewStore(); + + Assert.Empty(store.Load("s", "g99")); + Assert.Single(store.Load("s", "g100")); + Assert.Single(store.Load("s", "g299")); + + var columns = string.Join(",", Enumerable.Range(0, 250).Select(i => "{\"Column\":\"C" + i + "\",\"Value\":\"t\"}")); + File.WriteAllText(FilePath, "{\"Version\":1,\"Grids\":[{\"Server\":\"s\",\"Grid\":\"wide\",\"Columns\":[" + columns + "]}]}"); + Assert.Equal(ColumnFilterStore.MaxColumnsPerGrid, NewStore().Load("s", "wide").Count); + } + + [Fact] + public void A_null_or_wrong_typed_entry_is_skipped_and_the_rest_of_the_file_is_used() + { + File.WriteAllText(FilePath, "{\"Version\":1,\"Grids\":[null,5,\"x\",[1]," + + "{\"Server\":\"s\",\"Grid\":\"g\",\"Columns\":[null,7,\"x\",{\"Column\":5},{\"Column\":\"LoginName\",\"Value\":\"ok\",\"Values\":[null,1]}," + + "{\"Column\":\"HostName\",\"Value\":\"h\",\"ValueMode\":\"Hide\",\"Values\":[\"a\",null]}]}," + + "{\"Server\":5,\"Grid\":\"g2\",\"Columns\":[]},{\"Server\":\"s\",\"Grid\":\"g3\",\"Columns\":null}]}"); + var warnings = new List(); + var store = NewStore(warnings); + + var loaded = store.Load("s", "g"); + Assert.Equal(new[] { "HostName", "LoginName" }, loaded.Select(f => f.ColumnName).OrderBy(n => n)); + Assert.Equal("ok", loaded.Single(f => f.ColumnName == "LoginName").Value); + /* a null among the values makes that value part unusable, not the column */ + Assert.Equal(ColumnValueMode.None, loaded.Single(f => f.ColumnName == "HostName").ValueMode); + Assert.Empty(store.Load("s", "g2")); + Assert.Empty(store.Load("s", "g3")); + Assert.Empty(warnings); + } + + [Fact] + public void A_file_over_the_size_limit_or_nested_too_deep_is_ignored_whole_with_one_warning() + { + using (var stream = new FileStream(FilePath, FileMode.Create)) + stream.SetLength(ColumnFilterStore.MaxFileBytes + 1); + var warnings = new List(); + var big = NewStore(warnings); + Assert.Empty(big.Load("s", "g")); + Assert.Empty(big.Load("s", "g")); + Assert.Single(warnings); + + File.WriteAllText(FilePath, "{\"Version\":1,\"Grids\":" + new string('[', 100) + new string(']', 100) + "}"); + warnings.Clear(); + var deep = NewStore(warnings); + Assert.Empty(deep.Load("s", "g")); + Assert.Single(warnings); + } + + [Fact] + public void A_failed_save_is_retried_on_the_next_tick_and_warned_once_per_run() + { + var warnings = new List(); + var store = new ColumnFilterStore(FilePath, warnings.Add, TimeSpan.FromMilliseconds(50)); + Directory.CreateDirectory(FilePath + ".tmp"); // the temp file cannot be written + store.Save("srv", "Grid", new[] { Hide("c", "v") }); + + store.Flush(); + store.Flush(); + store.Flush(); + Assert.Single(warnings); + Assert.False(File.Exists(FilePath)); + + Directory.Delete(FilePath + ".tmp"); // the cause goes away: the debounce timer's own tick writes the file + var deadline = DateTime.UtcNow.AddSeconds(15); + while (!File.Exists(FilePath) && DateTime.UtcNow < deadline) + Thread.Sleep(50); + + Assert.True(File.Exists(FilePath), "the failed save was never retried"); + Assert.Single(warnings); + Assert.Single(NewStore().Load("srv", "Grid")); + store.Dispose(); + } + + [Fact] + public void A_save_that_keeps_failing_backs_off_from_the_debounce_up_to_a_minute_and_a_good_save_resets_it() + { + var failing = true; + var warnings = new List(); + var store = new ColumnFilterStore(FilePath, warnings.Add, TimeSpan.FromSeconds(10), + (path, text) => { if (failing) throw new IOException("the disk is full"); File.WriteAllText(path, text); }); + store.Save("srv", "Grid", new[] { Hide("c", "v") }); + Assert.Equal(TimeSpan.FromSeconds(10), store.RetryDelay); + + var seen = new List(); + for (var i = 0; i < 6; i++) + { + store.Flush(); + seen.Add(store.RetryDelay.TotalSeconds); + } + + // 10 s after the first failure, then doubling, never more than 60 s. + Assert.Equal(new[] { 10d, 20d, 40d, 60d, 60d, 60d }, seen); + Assert.Single(warnings); + Assert.Contains("is tried again in 10 seconds", warnings[0], StringComparison.Ordinal); + Assert.Contains("up to 60 seconds", warnings[0], StringComparison.Ordinal); + Assert.DoesNotContain("each change", warnings[0], StringComparison.Ordinal); + + failing = false; + store.Flush(); + Assert.True(File.Exists(FilePath)); + Assert.Equal(TimeSpan.FromSeconds(10), store.RetryDelay); + store.Dispose(); + } + + [Fact] + public void A_slow_write_does_not_hold_up_Load_or_Save() + { + using var writing = new ManualResetEventSlim(false); + using var release = new ManualResetEventSlim(false); + var store = new ColumnFilterStore(FilePath, null, TimeSpan.FromHours(1), (path, text) => + { + writing.Set(); + release.Wait(TimeSpan.FromSeconds(30)); + File.WriteAllText(path, text); + }); + store.Save("srv", "Grid", new[] { Hide("c", "v") }); + var flush = new Thread(store.Flush) { IsBackground = true }; + flush.Start(); + Assert.True(writing.Wait(TimeSpan.FromSeconds(15)), "the write never started"); + + // The file I/O is outside the store's lock: a grid's Load and Save on the UI thread do not wait for it. + var loadedCount = -1; + var meanwhile = new Thread(() => + { + loadedCount = store.Load("srv", "Grid").Count; + store.Save("srv", "Other", new[] { Hide("d", "w") }); + }) { IsBackground = true }; + meanwhile.Start(); + var finished = meanwhile.Join(TimeSpan.FromSeconds(5)); + release.Set(); + flush.Join(TimeSpan.FromSeconds(15)); + Assert.True(finished, "Load and Save waited for the file write"); + Assert.Equal(1, loadedCount); + + // The change made during the write is not lost: the store is still dirty, and the next flush writes it. + store.Flush(); + Assert.Single(NewStore().Load("srv", "Other")); + store.Dispose(); + } + + [Fact] + public void The_same_server_and_grid_twice_in_a_file_keeps_the_last_copy_so_Load_and_Save_agree() + { + File.WriteAllText(FilePath, "{\"Version\":1,\"Grids\":[" + + "{\"Server\":\"s\",\"Grid\":\"g\",\"Columns\":[{\"Column\":\"LoginName\",\"Value\":\"old\"}]}," + + "{\"Server\":\"s\",\"Grid\":\"other\",\"Columns\":[{\"Column\":\"HostName\",\"Value\":\"h\"}]}," + + "{\"Server\":\"s\",\"Grid\":\"g\",\"Columns\":[{\"Column\":\"LoginName\",\"Value\":\"first\"}," + + "{\"Column\":\"HostName\",\"Value\":\"h2\"},{\"Column\":\"LoginName\",\"Value\":\"last\"}]}]}"); + var store = NewStore(); + + var loaded = store.Load("s", "g"); + Assert.Equal(2, loaded.Count); + Assert.Equal("last", loaded.Single(f => f.ColumnName == "LoginName").Value); + Assert.Equal("h2", loaded.Single(f => f.ColumnName == "HostName").Value); + + // Saving the grid replaces the one entry: the older copy does not come back on the next read. + store.Save("s", "g", Array.Empty()); + store.Flush(); + var again = NewStore(); + Assert.Empty(again.Load("s", "g")); + Assert.Single(again.Load("s", "other")); + store.Dispose(); + } + + [Fact] + public void A_write_goes_through_a_temp_file_so_a_failed_write_leaves_the_old_file_whole() + { + var warnings = new List(); + var store = NewStore(warnings); + store.Save("srv", "One", new[] { Hide("c", "v") }); + store.Flush(); + var before = File.ReadAllBytes(FilePath); + + /* the temp file cannot be written: a write straight into the real file would change it, replacing the temp file does not */ + Directory.CreateDirectory(FilePath + ".tmp"); + store.Save("srv", "Two", new[] { Hide("c", "w") }); + store.Flush(); + + Assert.Single(warnings); + Assert.Equal(before, File.ReadAllBytes(FilePath)); + Assert.Single(NewStore().Load("srv", "One")); + Assert.Empty(NewStore().Load("srv", "Two")); + } + + [Fact] + public void A_file_left_in_the_old_place_is_moved_to_the_new_one_once_and_deleted() + { + var legacy = Path.Combine(_dir, "roaming", "column-filters.json"); + Directory.CreateDirectory(Path.GetDirectoryName(legacy)!); + var seed = NewStore(); + seed.Save("srv", "Grid", new[] { Hide("c", "v") }); + seed.Flush(); + File.Move(FilePath, legacy); + + var target = Path.Combine(_dir, "local", "column-filters.json"); + ColumnFilterStore.Install(target, null, legacy); + try + { + Assert.False(File.Exists(legacy)); + Assert.True(File.Exists(target)); + Assert.Single(ColumnFilterStore.Current!.Load("srv", "Grid")); + } + finally + { + ColumnFilterStore.Current = null; + } + + /* when the new place already has a file, the old one is only deleted */ + File.WriteAllText(legacy, "{\"Version\":1,\"Grids\":[]}"); + ColumnFilterStore.MigrateLegacyFile(legacy, target); + Assert.False(File.Exists(legacy)); + Assert.True(File.Exists(target)); + } + + [Theory] + [InlineData("this is not json {")] + [InlineData("{\"version\": 99, \"grids\": []}")] + [InlineData("[1,2,3]")] + [InlineData("")] + public void A_file_that_cannot_be_read_is_reported_once_and_ignored(string content) + { + File.WriteAllText(FilePath, content); + var warnings = new List(); + var store = NewStore(warnings); + + Assert.Empty(store.Load("srv", "Grid")); + Assert.Empty(store.Load("srv", "Grid")); + store.Save("srv", "Grid", new[] { Hide("c", "v") }); + Assert.Single(store.Load("srv", "Grid")); + Assert.Single(warnings); + } + + [Fact] + public void A_missing_file_is_a_first_run_and_says_nothing() + { + var warnings = new List(); + Assert.Empty(NewStore(warnings).Load("srv", "Grid")); + Assert.Empty(warnings); + } + + [Fact] + public void The_file_holds_the_ticked_values_and_the_keys_and_no_row_data() + { + var store = NewStore(); + store.Save("srv-a", "QuerySnapshotsGrid", new[] { Hide("LoginName", "job_svc") }); + store.Flush(); + + var text = File.ReadAllText(FilePath); + Assert.Contains("job_svc", text, StringComparison.Ordinal); + Assert.Contains("QuerySnapshotsGrid", text, StringComparison.Ordinal); + Assert.False(File.Exists(FilePath + ".tmp"), "the temp file is replaced away"); + } + + [Fact] + public void Writes_are_debounced_until_the_flush() + { + var store = NewStore(); // an hour's debounce + store.Save("srv", "Grid", new[] { Hide("c", "v") }); + + Assert.False(File.Exists(FilePath)); + store.Flush(); + Assert.True(File.Exists(FilePath)); + } + + [Fact] + public void Writes_replace_the_old_file_in_one_step() + { + var store = NewStore(); + store.Save("srv", "One", new[] { Hide("c", "v") }); + store.Flush(); + store.Save("srv", "Two", new[] { Hide("c", "v") }); + store.Flush(); + + var restarted = NewStore(); + Assert.Single(restarted.Load("srv", "One")); + Assert.Single(restarted.Load("srv", "Two")); + Assert.False(File.Exists(FilePath + ".tmp")); + } + + // ---- the grid manager ------------------------------------------------- + + private sealed class LoginRow + { + public string? LoginName { get; set; } + public int SessionId { get; set; } + public string? QueryText { get; set; } + public string? Detail { get; set; } + } + + private static DataGrid NewGrid(string? name, string? server) + { + var grid = new DataGrid { AutoGenerateColumns = false }; + if (name is not null) grid.Name = name; + if (server is not null) ColumnFilterScope.SetServer(grid, server); + foreach (var path in new[] { "LoginName", "SessionId", "QueryText" }) + grid.Columns.Add(new DataGridTextColumn { Header = path, Binding = new Binding(path) }); + return grid; + } + + private static List Rows(params string?[] logins) => + logins.Select((l, i) => new LoginRow { LoginName = l, SessionId = i }).ToList(); + + private static List Shown(DataGrid grid) => + grid.Items.OfType().Select(r => r.LoginName).ToList(); + + private static void Untick(IDataGridFilterManager manager, string column, string value) + { + var model = new ColumnValueListModel(manager.GetValueCatalog(column)!, manager.Filters.GetValueOrDefault(column)); + model.Entries.Single(e => e.Value == value).IsTicked = false; + var state = new ColumnFilterState { ColumnName = column }; + model.ApplyTo(state); + manager.SetFilter(state); + } + + [Fact] + public void A_text_match_on_a_query_text_column_lasts_the_session_but_is_not_stored() + { + var kept = StaTestThread.Run(() => + { + var store = NewStore(); + ColumnFilterStore.Current = store; + var grid = NewGrid("QuerySnapshotsGrid", "srv-a"); + var manager = new DataGridFilterManager(grid); + var rows = new List + { + new() { LoginName = "sa", QueryText = "select salary from pay" }, + new() { LoginName = "app", QueryText = "select 1" }, + }; + manager.UpdateData(rows); + manager.SetFilter(new ColumnFilterState { ColumnName = "QueryText", Value = "salary" }); + manager.SetFilter(new ColumnFilterState { ColumnName = "LoginName", Value = "sa" }); + + /* a refresh keeps both for the session */ + manager.UpdateData(rows); + var afterRefresh = Shown(grid); + store.Flush(); + store.Dispose(); + + var restarted = NewStore(); + ColumnFilterStore.Current = restarted; + var again = new DataGridFilterManager(NewGrid("QuerySnapshotsGrid", "srv-a")); + again.UpdateData(rows); + return (afterRefresh, again.Filters.Keys.OrderBy(k => k).ToList()); + }); + + Assert.Equal(new string?[] { "sa" }, kept.afterRefresh); + Assert.Equal(new[] { "LoginName" }, kept.Item2); + Assert.DoesNotContain("salary", File.ReadAllText(FilePath), StringComparison.Ordinal); + } + + [Fact] + public void A_filter_survives_a_refresh_and_a_value_that_first_appears_later_still_shows() + { + StaTestThread.Run(() => + { + var grid = NewGrid("G", null); + var manager = new DataGridFilterManager(grid); + manager.UpdateData(Rows("sa", "app", "job_svc")); + Untick(manager, "LoginName", "job_svc"); + + Assert.Equal(new string?[] { "sa", "app" }, Shown(grid)); + + manager.UpdateData(Rows("sa", "app", "job_svc", "newuser")); + Assert.Equal(new string?[] { "sa", "app", "newuser" }, Shown(grid)); + Assert.True(manager.AnyFilterActive); + return true; + }); + } + + [Fact] + public void The_value_list_comes_from_the_unfiltered_rows_not_the_ones_shown() + { + StaTestThread.Run(() => + { + var grid = NewGrid("G", null); + var manager = new DataGridFilterManager(grid); + manager.UpdateData(Rows("sa", "app", "job_svc")); + Untick(manager, "LoginName", "job_svc"); + + var catalog = manager.GetValueCatalog("LoginName")!; + Assert.Equal(new[] { "app", "job_svc", "sa" }, catalog.Values); + return true; + }); + } + + [Fact] + public void Number_query_text_and_prose_columns_and_columns_the_row_lacks_get_no_list() + { + StaTestThread.Run(() => + { + var grid = NewGrid("G", null); + var manager = new DataGridFilterManager(grid); + Assert.Null(manager.GetValueCatalog("LoginName")); // no rows yet + manager.UpdateData(Rows("sa")); + + Assert.NotNull(manager.GetValueCatalog("LoginName")); + Assert.Null(manager.GetValueCatalog("SessionId")); + Assert.Null(manager.GetValueCatalog("QueryText")); + Assert.Null(manager.GetValueCatalog("Detail")); + Assert.Null(manager.GetValueCatalog("NoSuchColumn")); + return true; + }); + } + + [Fact] + public void A_text_column_that_sorts_by_another_member_gets_no_list() + { + StaTestThread.Run(() => + { + var grid = NewGrid("G", null); + ((DataGridTextColumn)grid.Columns[0]).SortMemberPath = "SessionId"; + var manager = new DataGridFilterManager(grid); + manager.UpdateData(Rows("sa")); + + Assert.Null(manager.GetValueCatalog("LoginName")); + return true; + }); + } + + [Fact] + public void A_column_with_a_value_over_256_characters_gets_no_list() + { + StaTestThread.Run(() => + { + var grid = NewGrid("G", null); + var manager = new DataGridFilterManager(grid); + manager.UpdateData(Rows("sa", new string('x', 257))); + + Assert.Null(manager.GetValueCatalog("LoginName")); + return true; + }); + } + + [Fact] + public void A_filter_set_in_one_session_is_loaded_by_the_next_on_its_first_refresh() + { + var shown = StaTestThread.Run(() => + { + var store = NewStore(); + ColumnFilterStore.Current = store; + var grid = NewGrid("QuerySnapshotsGrid", "srv-a"); + var manager = new DataGridFilterManager(grid); + manager.UpdateData(Rows("sa", "app", "job_svc")); + Untick(manager, "LoginName", "job_svc"); + store.Flush(); + store.Dispose(); + + /* a restart: a new store over the same file, a new grid, a new manager */ + var restarted = NewStore(); + ColumnFilterStore.Current = restarted; + var grid2 = NewGrid("QuerySnapshotsGrid", "srv-a"); + var manager2 = new DataGridFilterManager(grid2); + manager2.UpdateData(Rows("sa", "app", "job_svc", "newuser")); + return Shown(grid2); + }); + + Assert.Equal(new string?[] { "sa", "app", "newuser" }, shown); + } + + [Fact] + public void A_cross_server_list_under_the_all_servers_scope_keeps_its_filters_across_a_restart() + { + var shown = StaTestThread.Run(() => + { + var store = NewStore(); + ColumnFilterStore.Current = store; + var manager = new DataGridFilterManager(NewGrid("AlertsDataGrid", ColumnFilterScope.AllServers)); + manager.UpdateData(Rows("sa", "app", "job_svc")); + Untick(manager, "LoginName", "job_svc"); + store.Flush(); + store.Dispose(); + + ColumnFilterStore.Current = NewStore(); + var grid = NewGrid("AlertsDataGrid", ColumnFilterScope.AllServers); + new DataGridFilterManager(grid).UpdateData(Rows("sa", "app", "job_svc")); + return Shown(grid); + }); + + Assert.Equal(new string?[] { "sa", "app" }, shown); + Assert.StartsWith("\u0001", ColumnFilterScope.AllServers, StringComparison.Ordinal); + } + + [Fact] + public void Another_servers_filters_do_not_apply_and_a_grid_without_a_name_or_scope_is_not_stored() + { + StaTestThread.Run(() => + { + var store = NewStore(); + ColumnFilterStore.Current = store; + + var grid = NewGrid("G", "srv-a"); + var manager = new DataGridFilterManager(grid); + manager.UpdateData(Rows("sa", "job_svc")); + Untick(manager, "LoginName", "job_svc"); + + var other = NewGrid("G", "srv-b"); + var otherManager = new DataGridFilterManager(other); + otherManager.UpdateData(Rows("sa", "job_svc")); + Assert.Equal(new string?[] { "sa", "job_svc" }, Shown(other)); + + var unnamed = NewGrid(null, "srv-a"); + var unnamedManager = new DataGridFilterManager(unnamed); + unnamedManager.UpdateData(Rows("sa", "job_svc")); + Untick(unnamedManager, "LoginName", "job_svc"); + + var unscoped = NewGrid("G", null); + var unscopedManager = new DataGridFilterManager(unscoped); + unscopedManager.UpdateData(Rows("sa", "job_svc")); + Untick(unscopedManager, "LoginName", "job_svc"); + + Assert.Single(store.Load("srv-a", "G")); + Assert.Empty(store.Load("srv-a", "")); + return true; + }); + } + + [Fact] + public void A_stored_filter_on_a_column_the_grid_no_longer_has_is_ignored() + { + StaTestThread.Run(() => + { + var store = NewStore(); + store.Save("srv-a", "G", new[] { Hide("RetiredColumn", "x"), Hide("LoginName", "job_svc") }); + ColumnFilterStore.Current = store; + + var grid = NewGrid("G", "srv-a"); + var manager = new DataGridFilterManager(grid); + manager.UpdateData(Rows("sa", "job_svc")); + + Assert.Equal(new[] { "LoginName" }, manager.Filters.Keys); + Assert.Equal(new string?[] { "sa" }, Shown(grid)); + return true; + }); + } + + [Fact] + public void Clear_all_filters_clears_the_grid_and_the_stored_copy_but_a_server_switch_does_not() + { + StaTestThread.Run(() => + { + var store = NewStore(); + ColumnFilterStore.Current = store; + var grid = NewGrid("G", "srv-a"); + var manager = new DataGridFilterManager(grid); + manager.UpdateData(Rows("sa", "job_svc")); + Untick(manager, "LoginName", "job_svc"); + + manager.ClearFilters(); // #2306: a context change + Assert.Equal(2, Shown(grid).Count); + Assert.Single(store.Load("srv-a", "G")); // still stored for that server + + manager.UpdateData(Rows("sa", "job_svc")); // the next refresh loads what is stored for the scope + Assert.Equal(new string?[] { "sa" }, Shown(grid)); + + manager.ClearAllFilters(); // the user + Assert.Equal(2, Shown(grid).Count); + Assert.False(manager.AnyFilterActive); + Assert.Empty(store.Load("srv-a", "G")); + + manager.UpdateData(Rows("sa", "job_svc")); + Assert.Equal(2, Shown(grid).Count); + return true; + }); + } + + [Fact] + public void A_server_switch_loads_the_new_servers_own_filters() + { + StaTestThread.Run(() => + { + var store = NewStore(); + store.Save("srv-b", "G", new[] { Hide("LoginName", "sa") }); + ColumnFilterStore.Current = store; + var grid = NewGrid("G", "srv-a"); + var manager = new DataGridFilterManager(grid); + manager.UpdateData(Rows("sa", "job_svc")); + Assert.Equal(2, Shown(grid).Count); + + ColumnFilterScope.SetServer(grid, "srv-b"); // the FinOps selector changes server + manager.ClearFilters(); + manager.UpdateData(Rows("sa", "job_svc")); + + Assert.Equal(new string?[] { "job_svc" }, Shown(grid)); + return true; + }); + } + + [Fact] + public void A_store_that_fails_never_blocks_the_grid() + { + StaTestThread.Run(() => + { + File.WriteAllText(FilePath, "garbage"); + ColumnFilterStore.Current = NewStore(); + var grid = NewGrid("G", "srv-a"); + var manager = new DataGridFilterManager(grid); + + manager.UpdateData(Rows("sa", "job_svc")); + Untick(manager, "LoginName", "job_svc"); + + Assert.Equal(new string?[] { "sa" }, Shown(grid)); + return true; + }); + } +} diff --git a/Darling/Darling.Tests/ColumnFilterPopupTests.cs b/Darling/Darling.Tests/ColumnFilterPopupTests.cs new file mode 100644 index 0000000000..0c0f2385a4 --- /dev/null +++ b/Darling/Darling.Tests/ColumnFilterPopupTests.cs @@ -0,0 +1,323 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Windows; +using System.Windows.Controls; +using System.Windows.Controls.Primitives; +using PerformanceMonitor.Common; +using PerformanceMonitor.Ui; +using Xunit; + +namespace Darling.Tests; + +/// +/// The column filter popup's value list (#5565) as the controls show it: the list section appears only for a column +/// with a catalog, search narrows it, Select All and the entries' ticks reach the applied filter, the notes, and +/// "Clear all filters" shows only when the grid has a filter and raises its own event. +/// +public sealed class ColumnFilterPopupTests +{ + private static T Find(DependencyObject root, Func where) where T : DependencyObject + { + foreach (var child in LogicalTreeHelper.GetChildren(root).OfType()) + { + if (child is T hit && where(hit)) + return hit; + try { return Find(child, where); } catch (InvalidOperationException) { } + } + throw new InvalidOperationException("not found"); + } + + private static Button ButtonNamed(ColumnFilterPopup popup, string content) => + Find
[^,\s]+),").Select(m => m.Groups["table"].Value).ToList(); + Assert.Equal(2, calls.Count); + Assert.All(calls, c => Assert.True(c is "table.Legacy" or "table.Default", $"PurgeOneAsync names {c}")); + } + + [Fact] + public void TheDefaultDeleteNamesTheDefaultPartition_AndIsAPlainCutoffDelete() + { + var sql = DarlingRetention.DefaultPartitionDeleteSql(QueryStoreIntervalPartitions.Wide); + + Assert.Equal("DELETE FROM collect.query_store_interval_wide_default WHERE first_execution_time < $1", sql); + Assert.DoesNotContain("ctid", sql, StringComparison.OrdinalIgnoreCase); + Assert.Equal( + "DELETE FROM collect.query_store_interval_latest_default WHERE first_execution_time < $1", + DarlingRetention.DefaultPartitionDeleteSql(QueryStoreIntervalPartitions.Latest)); + } + + [Fact] + public void NoPurgeStatementInRetentionNamesAnIntervalParentByLiteral() + { + var source = ReadSource("PerformanceMonitor.Darling.Service", "DarlingRetention.cs"); + + /* The parents are reachable only through the table descriptors (and only for logging); a literal parent name in a + purge call is the pre-partitioning shape. */ + var literal = new Regex(@"(RowCappedDeleteSql|PurgeOneAsync)\([^;]*""(collect\.)?query_store_interval_(wide|latest)"""); + Assert.DoesNotMatch(literal, source); + } + + [Fact] + public void TheLegacyDropIsGatedOnPromotion_AndTheLegacyTableCanNeverBeSelectedWhileUnbounded() + { + var source = ReadSource("PerformanceMonitor.Darling.Storage", "QueryStoreIntervalPartitions.cs"); + var start = source.IndexOf("public static async Task DropExpiredAsync(", StringComparison.Ordinal); + Assert.True(start >= 0); + var end = source.IndexOf("public static async Task RunMaintenanceAsync(", start, StringComparison.Ordinal); + var body = source.Substring(start, end - start); + + /* The first statement after reading the state: not promoted means not ready, and nothing is dropped. */ + var gate = body.IndexOf("if (!state.Promoted)", StringComparison.Ordinal); + var drop = body.IndexOf("DROP TABLE", StringComparison.Ordinal); + Assert.True(gate >= 0 && drop > gate, "DropExpiredAsync must check Promoted before any DROP"); + Assert.Contains("StepOutcome.NotReady", body.Substring(gate, drop - gate), StringComparison.Ordinal); + Assert.Contains("SET LOCAL lock_timeout", source, StringComparison.Ordinal); + Assert.Contains("IsLockTimeout(ex)", body, StringComparison.Ordinal); + + /* An unbounded legacy table and DEFAULT are never in the expired set. */ + var expired = source.IndexOf("public static IReadOnlyList ExpiredPartitions(", StringComparison.Ordinal); + var expiredBody = source.Substring(expired, 400); + Assert.Contains("!p.IsDefault", expiredBody, StringComparison.Ordinal); + Assert.Contains("!p.UpperUnbounded", expiredBody, StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/QueryStoreIntervalRetentionTestKit.cs b/Darling/Darling.Tests/QueryStoreIntervalRetentionTestKit.cs new file mode 100644 index 0000000000..157539a032 --- /dev/null +++ b/Darling/Darling.Tests/QueryStoreIntervalRetentionTestKit.cs @@ -0,0 +1,146 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Storage; + +namespace Darling.Tests; + +/// +/// Shared helpers for the #5571 retention and read-shape live tests on the day-partitioned Query Store interval tables: +/// a migrated scratch store, a promoted table, one-row inserts, and a logger that records what it was told. The test +/// classes using it each own their scratch database, so none of them can race another's DDL. +/// +/* #1776 own-store: deliberately NOT [Collection("live-postgres")]. This class only reads DARLING_TEST_PG for the + test classes that CREATE and DROP their own database through ScratchPostgres; it touches no shared database. */ +internal static class QueryStoreIntervalRetentionTestKit +{ + internal static readonly QueryStoreIntervalPartitions.IntervalTable Wide = QueryStoreIntervalPartitions.Wide; + internal static readonly QueryStoreIntervalPartitions.IntervalTable Latest = QueryStoreIntervalPartitions.Latest; + + internal const string SkipText = "Set DARLING_TEST_PG to a Postgres connection string to run the #5571 live tests."; + + internal static string? BaseConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + /// The fixed "now" the retention tests use: not midnight, so a day that straddles the cutoff exists. + internal static readonly DateTime Now = new(2026, 10, 8, 14, 55, 30, DateTimeKind.Utc); + + /// A logger that records every line, so a test can assert what was warned and what was not. + internal sealed class ListLogger : ILogger + { + internal List<(LogLevel Level, string Message)> Lines { get; } = new(); + + public IDisposable? BeginScope(TState state) where TState : notnull => null; + + public bool IsEnabled(LogLevel logLevel) => true; + + public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) + { + lock (Lines) + { + Lines.Add((logLevel, formatter(state, exception))); + } + } + + internal bool HasAtLeast(LogLevel level) + { + lock (Lines) + { + return Lines.Any(l => l.Level >= level); + } + } + + internal string Dump() + { + lock (Lines) + { + return string.Join(" | ", Lines.Select(l => $"{l.Level}: {l.Message}")); + } + } + } + + /// A scratch database run through every migration, so both interval tables are partitioned with an attached legacy table. + internal static async Task OpenStoreAsync(ScratchPostgres scratch, CancellationToken ct) + { + var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + return connection; + } + + internal static async Task ExecAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection) { CommandTimeout = 120 }; + await command.ExecuteNonQueryAsync(ct); + } + + internal static async Task CountAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection) { CommandTimeout = 120 }; + return Convert.ToInt64(await command.ExecuteScalarAsync(ct), CultureInfo.InvariantCulture); + } + + /// Whether a schema-qualified relation exists. + internal static async Task ExistsAsync(NpgsqlConnection connection, string qualifiedName, CancellationToken ct) + { + await using var command = new NpgsqlCommand("SELECT to_regclass($1) IS NOT NULL", connection); + command.Parameters.AddWithValue(qualifiedName); + return (bool)(await command.ExecuteScalarAsync(ct))!; + } + + /// One row with the given first_execution_time; the query id keys it. + internal static async Task InsertAsync( + NpgsqlConnection connection, QueryStoreIntervalPartitions.IntervalTable table, DateTime first, long query, CancellationToken ct) + { + var sql = table == Wide + ? "INSERT INTO collect.query_store_interval_wide (collection_time, server_id, database_name, query_id, plan_id, execution_type_desc, first_execution_time, last_execution_time, query_text, execution_count, avg_duration_us, runtime_stats_interval_id, interval_start_time_utc) " + + "VALUES ($1, 1, 'db', $2, $2, 'Regular', $1, $1, 'select 1', 1, 1000, 1, $1)" + : "INSERT INTO collect.query_store_interval_latest (collection_time, server_id, database_name, query_id, plan_id, first_execution_time, last_execution_time, query_text, execution_count, avg_duration_us, runtime_stats_interval_id) " + + "VALUES ($1, 1, 'db', $2, $2, $1, $1, 'select 1', 1, 1000, 1)"; + await using var command = new NpgsqlCommand(sql, connection); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = DateTime.SpecifyKind(first, DateTimeKind.Unspecified) }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Bigint, Value = query }); + await command.ExecuteNonQueryAsync(ct); + } + + /// The leaf a row lives in, schema-qualified, found by its query id. + internal static async Task PartitionOfAsync( + NpgsqlConnection connection, QueryStoreIntervalPartitions.IntervalTable table, long query, CancellationToken ct) + { + await using var command = new NpgsqlCommand( + $"SELECT 'collect.' || (SELECT c.relname FROM pg_class AS c WHERE c.oid = t.tableoid) FROM {table.Parent} AS t WHERE query_id = {query}", + connection); + return (string)(await command.ExecuteScalarAsync(ct))!; + } + + /// The leaves of a table by name, from the catalog. + internal static async Task> PartitionNamesAsync( + NpgsqlConnection connection, QueryStoreIntervalPartitions.IntervalTable table, CancellationToken ct) => + (await QueryStoreIntervalPartitions.ReadPartitionsAsync(connection, table, ct)).Select(p => p.Name).ToList(); + + /// Arms, validates and promotes a table as of (legacy bound S = that day + 2), with no waits between retries. + internal static async Task PromoteAsync( + NpgsqlConnection connection, QueryStoreIntervalPartitions.IntervalTable table, DateTime promotedAt, ILogger logger, CancellationToken ct) + { + var step = await QueryStoreIntervalPartitions.RunPromotionAsync( + connection, table, promotedAt, logger, TimeSpan.FromMilliseconds(50), TimeSpan.FromMilliseconds(50), ct); + if (step.Outcome is not (QueryStoreIntervalPartitions.StepOutcome.Done or QueryStoreIntervalPartitions.StepOutcome.NothingToDo)) + { + throw new InvalidOperationException($"{table.Name} did not promote: {step.Outcome} {step.Detail}"); + } + } +} diff --git a/Darling/Darling.Tests/QueryStoreIntervalWideBelowFloorLiveTests.cs b/Darling/Darling.Tests/QueryStoreIntervalWideBelowFloorLiveTests.cs index 9bb4d005d6..deeed16a20 100644 --- a/Darling/Darling.Tests/QueryStoreIntervalWideBelowFloorLiveTests.cs +++ b/Darling/Darling.Tests/QueryStoreIntervalWideBelowFloorLiveTests.cs @@ -33,6 +33,7 @@ namespace Darling.Tests; to CREATE and DROP its own database through ScratchPostgres and works entirely inside it (the chunk drops and the retention delete run against that database), so it cannot race live collection. */ [Collection("gap-cache-serial")] +[Trait("Cost", "Slow")] public sealed class QueryStoreIntervalWideBelowFloorLiveTests { internal const int ServerId = -4689001; @@ -441,6 +442,7 @@ public async Task Disclosure_EffectiveStart_IsTheExactBound_AndTheNoteNamesTheRe Assert.Equal(S.AddHours(12), DateTime.Parse(filled.GetProperty("effective_start").GetString()!, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind)); Assert.True(filled.GetProperty("window_truncated").GetBoolean()); Assert.Contains("began keeping complete history", filled.GetProperty("truncation_note").GetString(), StringComparison.Ordinal); + AssertTheNoteNamesTheField(filled); await ForceFilledSinceAsync(rig.Connection, S.AddDays(-1), ct); await PurgeTableAsync(rig.Connection, S.AddMinutes(90), ct); @@ -449,20 +451,40 @@ public async Task Disclosure_EffectiveStart_IsTheExactBound_AndTheNoteNamesTheRe Assert.Equal("interval_table", edge.GetProperty("history_source").GetString()); Assert.Equal(S.AddHours(2).Add(QueryStoreIntervalWide.PurgeEdgeMargin), DateTime.Parse(edge.GetProperty("effective_start").GetString()!, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind)); Assert.Contains("keeps 9 days", edge.GetProperty("truncation_note").GetString(), StringComparison.Ordinal); + AssertTheNoteNamesTheField(edge); + } + + /// #4966: effective_start prints in UTC with the Z, and the note names the start in that exact text. + /// The page shows the instant in the browser's zone by finding the field's text in the note; a note that spelled the + /// store's naive floor on its own carried no Z, so it was found nowhere and drawn in bare UTC above local times. + private static void AssertTheNoteNamesTheField(JsonElement answer) + { + var start = answer.GetProperty("effective_start").GetString()!; + Assert.EndsWith("Z", start, StringComparison.Ordinal); + Assert.Contains(start, answer.GetProperty("truncation_note").GetString(), StringComparison.Ordinal); } /* ---- helpers ------------------------------------------------------------------------------------------ */ - /// Runs the real retention delete for the interval table, slice by slice, until it deletes nothing. + /// + /// Runs the real retention delete for the interval table (#5569: the row-capped cursor form, batch by + /// batch, carrying the cursor the way DarlingRetention.PurgeOneAsync does) until it deletes nothing. + /// internal static async Task PurgeTableAsync(NpgsqlConnection connection, DateTime cutoff, CancellationToken ct) { - var sql = DarlingRetention.TimeSlicedDeleteSql("collect.query_store_interval_wide", "first_execution_time"); + var unspecified = DateTime.SpecifyKind(cutoff, DateTimeKind.Unspecified); + DateTime? cursor = null; int deleted; do { - await using var command = new NpgsqlCommand(sql, connection); - command.Parameters.Add(new NpgsqlParameter { TypedValue = DateTime.SpecifyKind(cutoff, DateTimeKind.Unspecified) }); - deleted = await command.ExecuteNonQueryAsync(ct); + var (rows, maxDeleted) = await DarlingRetention.ExecuteCursoredBatchAsync( + connection, "collect.query_store_interval_wide", "first_execution_time", 5_000, unspecified, + cursor, ct); + deleted = rows; + if (maxDeleted is not null) + { + cursor = maxDeleted; + } } while (deleted > 0); } @@ -475,11 +497,13 @@ internal static async Task SnapshotAsync(Rig rig, string name, string topSql, Da command.Parameters.Add(new NpgsqlParameter { TypedValue = DateTime.SpecifyKind(start, DateTimeKind.Unspecified) }); command.Parameters.Add(new NpgsqlParameter { TypedValue = DateTime.SpecifyKind(rig.End, DateTimeKind.Unspecified) }); command.Parameters.Add(new NpgsqlParameter { TypedValue = TestTop }); - for (var i = 0; i < 3; i++) + command.Parameters.Add(PerformanceMonitor.Darling.Storage.DatabaseFilter.All.Parameter()); /* #5245: the database list binds one text[] */ + for (var i = 0; i < 2; i++) { command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Text, Value = DBNull.Value }); } + command.Parameters.Add(new NpgsqlParameter { TypedValue = PerformanceMonitor.Darling.Storage.TopFill.FirstCandidates(TestTop) }); /* #5313: the round's candidate limit, bound last */ await command.ExecuteNonQueryAsync(ct); } diff --git a/Darling/Darling.Tests/QueryStoreIntervalWideBelowFloorRoutesLiveTests.cs b/Darling/Darling.Tests/QueryStoreIntervalWideBelowFloorRoutesLiveTests.cs index 4e2fbb8abd..bc57dc9bd4 100644 --- a/Darling/Darling.Tests/QueryStoreIntervalWideBelowFloorRoutesLiveTests.cs +++ b/Darling/Darling.Tests/QueryStoreIntervalWideBelowFloorRoutesLiveTests.cs @@ -123,6 +123,7 @@ private static async Task GridSnapshotAsync( command.Parameters.Add(new NpgsqlParameter { TypedValue = TestTop }); command.Parameters.Add(ViewerDataService.DatabaseFilterParameter(null)); + command.Parameters.Add(new NpgsqlParameter { TypedValue = PerformanceMonitor.Darling.Storage.TopFill.FirstCandidates(TestTop) }); /* #5313: the round's candidate limit, bound last */ await command.ExecuteNonQueryAsync(ct); } diff --git a/Darling/Darling.Tests/QueryStoreIntervalWideBrinIndexLiveTests.cs b/Darling/Darling.Tests/QueryStoreIntervalWideBrinIndexLiveTests.cs index 0c82074aae..43032b6ac2 100644 --- a/Darling/Darling.Tests/QueryStoreIntervalWideBrinIndexLiveTests.cs +++ b/Darling/Darling.Tests/QueryStoreIntervalWideBrinIndexLiveTests.cs @@ -45,13 +45,13 @@ private static async Task OpenMigratedAsync(ScratchPostgres sc return connection; } - private static async Task ScalarAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + internal static async Task ScalarAsync(NpgsqlConnection connection, string sql, CancellationToken ct) { await using var command = new NpgsqlCommand(sql, connection); return await command.ExecuteScalarAsync(ct); } - private static async Task ExecAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + internal static async Task ExecAsync(NpgsqlConnection connection, string sql, CancellationToken ct) { await using var command = new NpgsqlCommand(sql, connection) { CommandTimeout = 300 }; await command.ExecuteNonQueryAsync(ct); @@ -79,29 +79,35 @@ FROM generate_series(1, {rows}) AS g "collection_time = collection_time + interval '1 minute', last_execution_time = last_execution_time + interval '1 minute', " + "execution_count = execution_count + 1, avg_duration_us = avg_duration_us + 1"; - private static async Task<(long Updated, long Hot)> UpdateAndReadHotAsync( - NpgsqlConnection connection, string setList, string where, CancellationToken ct) + internal static async Task<(long Updated, long Hot)> UpdateAndReadHotAsync( + NpgsqlConnection connection, string setList, string where, CancellationToken ct, string? table = null) { + table ??= Table; /* The pg_stat_xact_* counters are the session's still-pending counts, so an earlier measurement on this connection is still in them after its rollback: take the delta around the update. */ await using var transaction = await connection.BeginTransactionAsync(ct); - var (updatedBefore, hotBefore) = await ReadXactCountersAsync(connection, transaction, ct); - await using (var update = new NpgsqlCommand($"UPDATE {Table} SET {setList} WHERE {where}", connection, transaction) { CommandTimeout = 300 }) + var (updatedBefore, hotBefore) = await ReadXactCountersAsync(connection, transaction, table, ct); + await using (var update = new NpgsqlCommand($"UPDATE {table} SET {setList} WHERE {where}", connection, transaction) { CommandTimeout = 300 }) { await update.ExecuteNonQueryAsync(ct); } - var (updatedAfter, hotAfter) = await ReadXactCountersAsync(connection, transaction, ct); + var (updatedAfter, hotAfter) = await ReadXactCountersAsync(connection, transaction, table, ct); await transaction.RollbackAsync(ct); return (updatedAfter - updatedBefore, hotAfter - hotBefore); } private static async Task<(long Updated, long Hot)> ReadXactCountersAsync( - NpgsqlConnection connection, NpgsqlTransaction transaction, CancellationToken ct) + NpgsqlConnection connection, NpgsqlTransaction transaction, string table, CancellationToken ct) { + /* #5571: the interval tables are partitioned by day, so the parent holds no rows or counters of its own; an update + is counted on the leaf partition that holds the row. The counters are summed over the table's leaves; the store + this test migrates is always partitioned (pg_partition_tree returns no row for a plain table). */ await using var command = new NpgsqlCommand( - "SELECT COALESCE(SUM(n_tup_upd), 0)::bigint, COALESCE(SUM(n_tup_hot_upd), 0)::bigint FROM pg_stat_xact_all_tables WHERE schemaname = 'collect' AND relname = 'query_store_interval_wide'", + "SELECT COALESCE(SUM(s.n_tup_upd), 0)::bigint, COALESCE(SUM(s.n_tup_hot_upd), 0)::bigint " + + "FROM pg_partition_tree(@table::regclass) AS p JOIN pg_stat_xact_all_tables AS s ON s.relid = p.relid WHERE p.isleaf", connection, transaction); + command.Parameters.AddWithValue("table", table); await using var reader = await command.ExecuteReaderAsync(ct); Assert.True(await reader.ReadAsync(ct)); return (reader.GetInt64(0), reader.GetInt64(1)); @@ -125,7 +131,7 @@ public async Task Ensure_BuildsAValidBrinIndexOnce_AndASecondCallChangesNothing( $"SELECT indisvalid FROM pg_index WHERE indexrelid = '{Index}'::regclass", ct))!); var definition = (string)(await ScalarAsync(connection, $"SELECT pg_get_indexdef('{Index}'::regclass)", ct))!; Assert.Contains("USING brin (collection_time)", definition); - Assert.Contains("autosummarize='on'", definition); + Assert.Contains("autosummarize=off", definition); var oid = await ScalarAsync(connection, $"SELECT '{Index}'::regclass::oid", ct); await EnsureAsync(connection, ct); @@ -165,9 +171,11 @@ public async Task TheUpsertsCollectionTimeRewrite_StaysHotWithTheBrin_AndNotWith } /// The column names the product's upsert sets in DO UPDATE SET, parsed from its own SQL. - private static HashSet UpsertSetColumns() + internal static HashSet UpsertSetColumns() => UpsertSetColumns(QueryStoreIntervalWide.UpsertSql); + + /// The same parse for another writer's upsert (the latest table's, #5507). + internal static HashSet UpsertSetColumns(string sql) { - var sql = QueryStoreIntervalWide.UpsertSql; var start = sql.IndexOf("DO UPDATE SET", StringComparison.Ordinal); Assert.True(start >= 0, "the upsert has no DO UPDATE SET; the parse below would read nothing"); var end = sql.IndexOf("WHERE (EXCLUDED.collection_time", start, StringComparison.Ordinal); @@ -230,7 +238,16 @@ public async Task NoNonBrinIndex_CoversAColumnTheUpsertSets_SoTheUpsertStaysHotF await using var scratch = await ScratchPostgres.CreateAsync(baseCs!, ct); await using var connection = await OpenMigratedAsync(scratch, ct); - await EnsureAsync(connection, ct); + + /* Every background index the service builds, not the BRIN alone: the census reads the catalog after the product's + own list has been ensured, so the next member of QueryStoreBackgroundIndexes.All is scanned without an edit here. */ + await QueryStoreBackgroundIndexesLiveTests.EnsureAllAsync(connection, ct); + foreach (var spec in QueryStoreBackgroundIndexes.All) + { + Assert.True( + (bool)(await ScalarAsync(connection, $"SELECT indisvalid FROM pg_index WHERE indexrelid = '{spec.IndexName}'::regclass", ct))!, + $"{spec.IndexName} must be built and valid before the census reads the catalog"); + } /* Data-independent: the catalog, not a seeded update. The scan must see the identity index and the first_execution_time index, or it reads nothing. */ @@ -332,19 +349,50 @@ public async Task AnInvalidLeftover_IsDroppedAndRebuilt() await using var scratch = await ScratchPostgres.CreateAsync(baseCs!, ct); await using var connection = await OpenMigratedAsync(scratch, ct); await EnsureAsync(connection, ct); - var oldOid = await ScalarAsync(connection, $"SELECT '{Index}'::regclass::oid", ct); - /* An interrupted CONCURRENTLY build leaves exactly this catalog state; the rig and CI roles are - superuser, so flipping indisvalid reproduces it deterministically. */ - await ExecAsync(connection, $"UPDATE pg_index SET indisvalid = false WHERE indexrelid = '{Index}'::regclass", ct); - Assert.False((bool)(await ScalarAsync(connection, $"SELECT indisvalid FROM pg_index WHERE indexrelid = '{Index}'::regclass", ct))!); + var oldLeafOid = await LeaveAnInterruptedLegacyBuildAsync(connection, Index, ct); await EnsureAsync(connection, ct); Assert.True((bool)(await ScalarAsync(connection, $"SELECT indisvalid FROM pg_index WHERE indexrelid = '{Index}'::regclass", ct))!); - Assert.NotEqual(oldOid, await ScalarAsync(connection, $"SELECT '{Index}'::regclass::oid", ct)); - var definition = (string)(await ScalarAsync(connection, $"SELECT pg_get_indexdef('{Index}'::regclass)", ct))!; + Assert.True((bool)(await ScalarAsync(connection, $"SELECT indisvalid FROM pg_index WHERE indexrelid = '{Index}_legacy'::regclass", ct))!); + Assert.True(await LeafIsAttachedAsync(connection, Index, ct)); + Assert.NotEqual(oldLeafOid, await ScalarAsync(connection, $"SELECT '{Index}_legacy'::regclass::oid", ct)); + var definition = (string)(await ScalarAsync(connection, $"SELECT pg_get_indexdef('{Index}_legacy'::regclass)", ct))!; Assert.Contains("USING brin (collection_time)", definition); - Assert.Contains("autosummarize='on'", definition); + Assert.Contains("autosummarize=off", definition); } + + /// + /// Leaves the catalog state an interrupted CREATE INDEX CONCURRENTLY leaves on a day-partitioned table (#5571): + /// the parent's ON ONLY index exists and is INVALID, and the legacy table's own index exists, is INVALID and is not + /// attached. (Flipping indisvalid on an attached parent, as the old plain-table test did, is not a state the build + /// can leave: nothing re-validates a parent whose children are all attached.) Returns the invalid leaf index's oid, which a + /// correct ensure drops and rebuilds. The rig and CI roles are superuser, so the flip is deterministic. + /// + internal static async Task LeaveAnInterruptedLegacyBuildAsync(NpgsqlConnection connection, string indexName, CancellationToken ct) + { + var shortName = indexName[(indexName.IndexOf('.', StringComparison.Ordinal) + 1)..]; + var parentDefinition = (string)(await ScalarAsync(connection, $"SELECT pg_get_indexdef('{indexName}'::regclass)", ct))!; + var table = Regex.Match(parentDefinition, @" ON ONLY (?
\S+) ", RegexOptions.CultureInvariant).Groups["table"].Value; + Assert.NotEqual(string.Empty, table); + + var leafDefinition = parentDefinition.Replace( + $"INDEX {shortName} ON ONLY {table} ", $"INDEX {shortName}_legacy ON {table}_legacy ", StringComparison.Ordinal); + Assert.NotEqual(parentDefinition, leafDefinition); + + await ExecAsync(connection, $"DROP INDEX {indexName}", ct); + await ExecAsync(connection, parentDefinition, ct); + await ExecAsync(connection, leafDefinition, ct); + await ExecAsync(connection, $"UPDATE pg_index SET indisvalid = false WHERE indexrelid = '{indexName}_legacy'::regclass", ct); + + Assert.False((bool)(await ScalarAsync(connection, $"SELECT indisvalid FROM pg_index WHERE indexrelid = '{indexName}'::regclass", ct))!); + Assert.False((bool)(await ScalarAsync(connection, $"SELECT indisvalid FROM pg_index WHERE indexrelid = '{indexName}_legacy'::regclass", ct))!); + return await ScalarAsync(connection, $"SELECT '{indexName}_legacy'::regclass::oid", ct); + } + + /// Whether the legacy leaf's index is attached to the parent index (a pg_inherits row). + internal static async Task LeafIsAttachedAsync(NpgsqlConnection connection, string indexName, CancellationToken ct) => + (bool)(await ScalarAsync(connection, + $"SELECT EXISTS (SELECT 1 FROM pg_inherits WHERE inhparent = '{indexName}'::regclass AND inhrelid = '{indexName}_legacy'::regclass)", ct))!; } diff --git a/Darling/Darling.Tests/QueryStoreIntervalWideBrinIndexTests.cs b/Darling/Darling.Tests/QueryStoreIntervalWideBrinIndexTests.cs index f62c0ce070..916aaa9dbf 100644 --- a/Darling/Darling.Tests/QueryStoreIntervalWideBrinIndexTests.cs +++ b/Darling/Darling.Tests/QueryStoreIntervalWideBrinIndexTests.cs @@ -58,7 +58,10 @@ public void TheBuildSql_IsConcurrent_Idempotent_AndABrinOnCollectionTime() var sql = QueryStoreIntervalWideBrinIndex.CreateSql; Assert.Contains("CREATE INDEX CONCURRENTLY IF NOT EXISTS", sql); Assert.Contains("USING brin (collection_time)", sql); - Assert.Contains("autosummarize = on", sql); + /* #5594: off. An autosummarize work item waits for the table's lock and cancels a running autovacuum. */ + Assert.Contains("autosummarize = off", sql); + Assert.DoesNotContain("autosummarize = on", sql); + Assert.Contains("autosummarize = off", QueryStoreIntervalWideBrinIndex.Spec.IndexDefinition); Assert.Contains("ix_query_store_interval_wide_collection_time_brin", sql); Assert.Contains("DROP INDEX CONCURRENTLY IF EXISTS", QueryStoreIntervalWideBrinIndex.DropSql); } @@ -66,7 +69,8 @@ public void TheBuildSql_IsConcurrent_Idempotent_AndABrinOnCollectionTime() [Fact] public void TheEnsure_ReadsValidityFirst_AndGuardsHypertablesThroughAViewCheck() { - var source = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "QueryStoreIntervalWideBrinIndex.cs"); + /* The machinery moved to QueryStoreBackgroundIndexes (#4952) so the two new btrees share it; the pins moved with it. */ + var source = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "QueryStoreBackgroundIndexes.cs"); /* The validity read is what tells an INVALID leftover from a good index; IF NOT EXISTS alone keeps both. */ Assert.Contains("i.indisvalid", source); @@ -75,7 +79,7 @@ public void TheEnsure_ReadsValidityFirst_AndGuardsHypertablesThroughAViewCheck() /* The hypertable catalog view is queried only after to_regclass says it exists (no TimescaleDB, no view). */ Assert.Contains("to_regclass('timescaledb_information.hypertables')", source); Assert.Contains("if (hasHypertableView)", source); - Assert.Contains("BrinAction.SkipHypertable", source); + Assert.Contains("IndexAction.SkipHypertable", source); Assert.Contains("LogWarning", source); } @@ -85,14 +89,16 @@ public void TheWorker_SchedulesTheEnsureAfterMigrations_WithoutAwaitingItOnTheSt var source = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "DarlingWorker.cs").Replace("\r\n", "\n"); var migrate = source.IndexOf("PgMigrations.MigrateAsync(migrateConnection", System.StringComparison.Ordinal); - var launch = source.IndexOf("var intervalWideBrin = QueryStoreIntervalWideBrinIndex.RunDelayedAsync(", System.StringComparison.Ordinal); - var drain = source.IndexOf("await intervalWideBrin;", System.StringComparison.Ordinal); + /* #5571: the launch is the day-partition task, which runs Phase A and then the index ensures on one task (the + VALIDATE and a CREATE INDEX CONCURRENTLY on the legacy table conflict), so the pin names that entry point. */ + var launch = source.IndexOf("var queryStoreIndexes = QueryStoreIntervalPartitions.RunDelayedAsync(", System.StringComparison.Ordinal); + var drain = source.IndexOf("await queryStoreIndexes;", System.StringComparison.Ordinal); var loopStop = source.IndexOf("PerformanceMonitor Darling collection loop stopped", System.StringComparison.Ordinal); Assert.True(migrate > 0 && launch > migrate, "the ensure must launch after migrations"); Assert.True(drain > launch && drain < loopStop, "the ensure is drained only at shutdown, after the collection loop"); - Assert.Single(Regex.Matches(source, @"await intervalWideBrin;")); - Assert.Contains("QueryStoreIntervalWideBrinIndex.StartDelay", source); + Assert.Single(Regex.Matches(source, @"await queryStoreIndexes;")); + Assert.Contains("QueryStoreBackgroundIndexes.StartDelay", source); } [Fact] @@ -103,7 +109,7 @@ public async Task RunDelayed_CancelledDuringTheDelay_LogsAtDebug_NotInformation( using var cts = new CancellationTokenSource(); cts.CancelAfter(TimeSpan.FromMilliseconds(200)); - await QueryStoreIntervalWideBrinIndex.RunDelayedAsync(dataSource, logger, TimeSpan.FromMinutes(20), cts.Token); + await QueryStoreBackgroundIndexes.RunDelayedAsync(dataSource, logger, TimeSpan.FromMinutes(20), QueryStoreBackgroundIndexes.All, cts.Token); var entry = Assert.Single(logger.Entries); Assert.Equal(LogLevel.Debug, entry.Level); @@ -116,7 +122,7 @@ public void RunDelayed_KeepsTheInformationLineOnlyForACancelAfterTheDelay() /* A cancel that lands after the delay needs a connection held open mid-handshake; Npgsql then may report the stop as its own exception, so that branch is pinned on the source: the flag is set right after the delay, the Debug branch is filtered on it, and the Information branch is the unfiltered catch. */ - var source = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "QueryStoreIntervalWideBrinIndex.cs") + var source = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "QueryStoreBackgroundIndexes.cs") .Replace("\r\n", "\n"); var delay = source.IndexOf("await Task.Delay(delay, cancellationToken)", System.StringComparison.Ordinal); diff --git a/Darling/Darling.Tests/QueryStoreIntervalWideFirstExecFloorTests.cs b/Darling/Darling.Tests/QueryStoreIntervalWideFirstExecFloorTests.cs index fc1b7a1520..9b71f2b7d2 100644 --- a/Darling/Darling.Tests/QueryStoreIntervalWideFirstExecFloorTests.cs +++ b/Darling/Darling.Tests/QueryStoreIntervalWideFirstExecFloorTests.cs @@ -17,12 +17,14 @@ namespace Darling.Tests; /// -/// #4605: among the indexes of collect.query_store_interval_wide are the unique key (it leads with -/// server_id and holds first_execution_time as a key column) and -/// idx_query_store_interval_wide_first_exec. The three per-server reads of the table (the MCP Query Store top, +/// #4605: among the btrees of collect.query_store_interval_wide are the unique key (it leads with +/// server_id and holds first_execution_time as a key column), +/// idx_query_store_interval_wide_first_exec, and, where the service has built it, the wide btree on +/// (server_id, first_execution_time) (#4952), which the service builds in the background rather than a +/// migration. The three per-server reads of the table (the MCP Query Store top, /// the Queries grid and the Trends chart, all WHERE server_id = $1) filter collection_time (the Trends -/// interval arm interval_start_time_utc), which neither serves, so each walked all of the server's rows. Each -/// now also carries +/// interval arm interval_start_time_utc), which none of them serves, so each walked all of the server's rows. +/// Each now also carries /// first_execution_time >= <window start> - . /// The Custom Views route, for all servers or some, carries no floor; its compose text is pinned in /// DarlingComposeTests.Compile_QueryStoreWideEligible_CarriesNoFirstExecutionTimeFloor. This class pins the @@ -95,23 +97,86 @@ public void ViewerQueryStoreTopTableSql_FloorsFirstExecutionTimeOnItsReadStart_A } [Fact] - public void ViewerQueryStoreDurationTrendTableSql_FloorsFirstExecutionTimeOnBothArms_AndTheRawTrendIsUntouched() + public void ViewerQueryStoreDurationTrendTableSql_BoundsFirstExecutionTimeOnBothArms_AndTheRawTrendIsUntouched() { var sql = ViewerDataService.QueryStoreDurationTrendTableSql; - var floor = Floor("$2"); - - /* Both arms, each after its own time filter: arm 1 (interval_start_time_utc) and arm 2 (the legacy - rows, collection_time). The margin appears exactly twice, once per arm. */ - var floors = Regex.Matches(sql, Regex.Escape(floor)); - Assert.Equal(2, floors.Count); - Assert.Equal(2, Regex.Matches(sql, Regex.Escape(QueryStoreIntervalWide.PurgeEdgeMarginSql)).Count); - - var arm1End = sql.IndexOf("interval_start_time_utc <= $3", StringComparison.Ordinal); - var arm2End = sql.IndexOf("collection_time <= $4", StringComparison.Ordinal); - Assert.True(arm1End >= 0 && arm1End < floors[0].Index, "arm 1's floor follows its interval_start_time_utc bounds"); - Assert.True(arm2End >= 0 && arm2End < floors[1].Index && floors[0].Index < arm2End, "arm 2's floor follows its collection_time bounds"); + var arms = sql.Split("UNION ALL", StringSplitOptions.None); + Assert.Equal(2, arms.Length); + + /* Arm 1 (#5523) is placed by interval_start_time_utc, so its range comes from the window: the first execution + lies inside the interval, which starts in [$2, $3] and spans at most a day. The collector-derived floor + (PurgeEdgeMarginSql) is arm 2's alone, which is placed by collection_time, and that arm takes the upper twin. */ + Assert.Contains("first_execution_time >= $2 - " + QueryStoreIntervalWide.IntervalStartSlackSql, arms[0], StringComparison.Ordinal); + Assert.Contains("first_execution_time <= $3 + " + QueryStoreIntervalWide.IntervalStartFirstExecMarginSql, arms[0], StringComparison.Ordinal); + Assert.DoesNotContain(QueryStoreIntervalWide.PurgeEdgeMarginSql, arms[0], StringComparison.Ordinal); + Assert.Contains(Floor("$2"), arms[1], StringComparison.Ordinal); + Assert.Contains("first_execution_time <= $4 + " + QueryStoreIntervalWide.FirstExecUpperSlackSql, arms[1], StringComparison.Ordinal); + Assert.Single(Regex.Matches(sql, Regex.Escape(QueryStoreIntervalWide.PurgeEdgeMarginSql))); + + /* Each bound follows its own arm's time filter. */ + var arm1End = arms[0].IndexOf("interval_start_time_utc <= $3", StringComparison.Ordinal); + Assert.True(arm1End >= 0 && arm1End < arms[0].IndexOf("first_execution_time >=", StringComparison.Ordinal), "arm 1's bounds follow its interval_start_time_utc bounds"); + var arm2End = arms[1].IndexOf("collection_time <= $4", StringComparison.Ordinal); + Assert.True(arm2End >= 0 && arm2End < arms[1].IndexOf("first_execution_time >=", StringComparison.Ordinal), "arm 2's bounds follow its collection_time bounds"); Assert.Contains("WHERE server_id = $1", sql, StringComparison.Ordinal); Assert.DoesNotContain(QueryStoreIntervalWide.PurgeEdgeMarginSql, ViewerDataService.QueryStoreDurationTrendSql, StringComparison.Ordinal); + Assert.DoesNotContain(QueryStoreIntervalWide.FirstExecUpperSlackSql, ViewerDataService.QueryStoreDurationTrendSql, StringComparison.Ordinal); + Assert.DoesNotContain(QueryStoreIntervalWide.IntervalStartSlackSql, ViewerDataService.QueryStoreDurationTrendSql, StringComparison.Ordinal); + } + + /// + /// #5523: the floor alone made the first_execution_time range run from the window start to now, so a window that + /// ended days ago, or the edge of a long one, scanned every newer row of the server. Each per-server read now also + /// carries the upper twin, placed after its floor. The raw reads carry neither. + /// + [Fact] + public void EveryPerServerTableRead_CarriesTheUpperBoundBesideItsFloor() + { + var slack = QueryStoreIntervalWide.FirstExecUpperSlackSql; + + var mcp = DarlingDataReader.QueryStoreTopTableSql; + Assert.Single(Regex.Matches(mcp, Regex.Escape("first_execution_time <= $3 + " + slack))); + Assert.True(mcp.IndexOf(Floor("$2"), StringComparison.Ordinal) < mcp.IndexOf("first_execution_time <= $3 + ", StringComparison.Ordinal)); + Assert.DoesNotContain(slack, DarlingDataReader.QueryStoreTopSql, StringComparison.Ordinal); + + /* The long-window read's two edge arms, each ending at its own window end: [$2, $8) and [$9, $3]. */ + var daily = DarlingDataReader.QueryStoreTopDailyTableSql; + Assert.Single(Regex.Matches(daily, Regex.Escape("first_execution_time < $8::date + " + slack))); + Assert.Single(Regex.Matches(daily, Regex.Escape("first_execution_time <= $3 + " + slack))); + Assert.Equal(2, Regex.Matches(daily, Regex.Escape("first_execution_time >= ")).Count); + + var grid = ViewerDataService.QueryStoreTopTableSql; + Assert.Single(Regex.Matches(grid, Regex.Escape("($3::timestamp IS NULL OR first_execution_time <= $3 + " + slack + ")"))); + Assert.DoesNotContain(slack, ViewerDataService.QueryStoreTopSql, StringComparison.Ordinal); + + var history = DarlingMcpQueryStoreHistoryTools.HistoryTableSql; + Assert.Single(Regex.Matches(history, Regex.Escape("first_execution_time <= $5 + " + slack))); + Assert.True(history.IndexOf("first_execution_time >= $4 - ", StringComparison.Ordinal) < history.IndexOf("first_execution_time <= $5 + ", StringComparison.Ordinal)); + Assert.DoesNotContain(slack, DarlingMcpQueryStoreHistoryTools.HistorySql, StringComparison.Ordinal); + } + + [Fact] + public void TheUpperSlack_IsNeverShorterThanTheDailyBuilders_AndItsSqlFormsAreNeverShorter() + { + /* The reads were exact before #5523 and the daily summary is documented as approximate, so the reads' slack (twelve + hours: clock set to local time, a long collection cycle) may be longer than the builder's hour but never shorter. */ + Assert.True(QueryStoreIntervalWide.FirstExecUpperSlack >= QueryStoreTopDaily.SkewSlack); + Assert.Equal(TimeSpan.FromHours(12), QueryStoreIntervalWide.FirstExecUpperSlack); + + var slack = Regex.Match(QueryStoreIntervalWide.FirstExecUpperSlackSql, @"^interval '(?\d+) minutes'$"); + Assert.True(slack.Success, QueryStoreIntervalWide.FirstExecUpperSlackSql); + Assert.True(TimeSpan.FromMinutes(long.Parse(slack.Groups["m"].Value, System.Globalization.CultureInfo.InvariantCulture)) >= QueryStoreIntervalWide.FirstExecUpperSlack); + + /* An interval placed by its start can hold a first execution a whole interval later, plus its own hour of slack + (both columns are the monitored clock, so no cross-clock skew applies to it). */ + Assert.Equal(TimeSpan.FromHours(1), QueryStoreIntervalWide.IntervalStartSlack); + var floor = Regex.Match(QueryStoreIntervalWide.IntervalStartSlackSql, @"^interval '(?\d+) minutes'$"); + Assert.True(floor.Success, QueryStoreIntervalWide.IntervalStartSlackSql); + Assert.True(TimeSpan.FromMinutes(long.Parse(floor.Groups["m"].Value, System.Globalization.CultureInfo.InvariantCulture)) >= QueryStoreIntervalWide.IntervalStartSlack); + Assert.Equal(QueryStoreIntervalWide.IntervalSpanMargin + QueryStoreIntervalWide.IntervalStartSlack, QueryStoreIntervalWide.IntervalStartFirstExecMargin); + var start = Regex.Match(QueryStoreIntervalWide.IntervalStartFirstExecMarginSql, @"^interval '(?\d+) minutes'$"); + Assert.True(start.Success, QueryStoreIntervalWide.IntervalStartFirstExecMarginSql); + Assert.True(TimeSpan.FromMinutes(long.Parse(start.Groups["m"].Value, System.Globalization.CultureInfo.InvariantCulture)) >= QueryStoreIntervalWide.IntervalStartFirstExecMargin); } } diff --git a/Darling/Darling.Tests/QueryStoreIntervalWideGateTests.cs b/Darling/Darling.Tests/QueryStoreIntervalWideGateTests.cs index d8bcede618..1dd21b9cf3 100644 --- a/Darling/Darling.Tests/QueryStoreIntervalWideGateTests.cs +++ b/Darling/Darling.Tests/QueryStoreIntervalWideGateTests.cs @@ -55,18 +55,52 @@ public void Clause2_FilledSinceAfterMaxOfRawFloorAndWindowStart_ReadsRaw() => Assert.False(Rule(filledSince: WindowEnd)); [Fact] - public void Clause3_TableFloorShortOfTheOneDayMargin_ReadsRaw() => - // table floor only reaches 23h before window start — short of the full one-day margin clause 3 needs — - // and raw has no floor (a plain store): the table cannot answer for the earliest part of the window. + public void Clause3_TableFloorShortOfThePurgeEdgeMargin_ReadsRaw() + { + /* #5541: the raw read counts a snapshot by collection_time alone, so a snapshot collected at or after the window + start can belong to an interval that began down to S - PurgeEdgeMargin (a 1440-minute interval collected up to + an hour plus the catch-up cap after it ended). A table floor between S - 26 h and S - 24 h used to pass the + old S - 1 day arm and then miss that interval. Raw has no floor here (a plain store), so only 3b can pass. */ + Assert.False(Rule(tableFloor: WindowStart.AddHours(-24))); + Assert.False(Rule(tableFloor: WindowStart - QueryStoreIntervalWide.PurgeEdgeMargin + TimeSpan.FromMinutes(1))); Assert.False(Rule(tableFloor: WindowStart.AddHours(-23))); + } [Fact] - public void Clause3_TableFloorAtOrBeyondTheOneDayMargin_ReadsTheTable() => - Assert.True(Rule(tableFloor: WindowStart.AddDays(-1))); + public void Clause3_ReviewFailingInput_AOneDayIntervalCollectedJustAfterTheWindowStart_ReadsRaw() + { + /* The round-1 review's pure form: default retention (raw keeps 30 days, so R is far below H and 3a fails), + H = S - 24 h, a 12 h minimum window. It returned true before the fix. */ + var s = WindowStart; + Assert.False(QueryStoreIntervalWide.UseTable( + s.AddDays(-1), false, s.AddDays(-10), s, s.AddDays(1), null, s.AddDays(1), s.AddHours(-24), TimeSpan.FromHours(12))); + } [Fact] - public void Clause3_RawFloorAtOrAboveTableFloor_ReadsTheTable_EvenOutsideTheMargin() => - Assert.True(Rule(rawFloor: WindowStart.AddDays(-3), tableFloor: WindowStart.AddDays(-3))); + public void Clause3_TableFloorAtOrBeyondThePurgeEdgeMargin_ReadsTheTable() + { + Assert.True(Rule(tableFloor: WindowStart - QueryStoreIntervalWide.PurgeEdgeMargin)); + Assert.True(Rule(tableFloor: WindowStart - QueryStoreIntervalWide.PurgeEdgeMargin - TimeSpan.FromMinutes(1))); + } + + [Fact] + public void Clause3_RawFloorAtOrAboveTableFloorPlusMargin_ReadsTheTable_EvenOutsideTheMargin() + { + /* The table floor sits 12h under the window start, short of the purge-edge margin, so only 3a can pass. */ + var h = WindowStart.AddHours(-12); + Assert.True(Rule(rawFloor: h + QueryStoreIntervalWide.PurgeEdgeMargin, tableFloor: h)); + Assert.True(Rule(rawFloor: h + QueryStoreIntervalWide.PurgeEdgeMargin + TimeSpan.FromMinutes(1), tableFloor: h)); + } + + [Fact] + public void Clause3_RawFloorWithinTheMarginOfTheTableFloor_ReadsRaw() + { + /* #5541: raw drops a snapshot by collection_time, the table drops an interval by first_execution_time, so + raw at or just above the table floor can still hold a snapshot of an interval the table dropped. */ + var h = WindowStart.AddHours(-12); + Assert.False(Rule(rawFloor: h, tableFloor: h)); + Assert.False(Rule(rawFloor: h + QueryStoreIntervalWide.PurgeEdgeMargin - TimeSpan.FromMinutes(1), tableFloor: h)); + } [Fact] public void Clause4_OpenEnd_SkipsTheAppliedThroughComparison() => diff --git a/Darling/Darling.Tests/QueryStoreIntervalWideGridLiveTests.cs b/Darling/Darling.Tests/QueryStoreIntervalWideGridLiveTests.cs index 21c989dec2..d059709a79 100644 --- a/Darling/Darling.Tests/QueryStoreIntervalWideGridLiveTests.cs +++ b/Darling/Darling.Tests/QueryStoreIntervalWideGridLiveTests.cs @@ -124,7 +124,7 @@ await ExecAsync(connection, /// /// Review D4R H1: a window under must issue ZERO /// round trips against collect.query_store_interval_wide — no ReadSourceInputsSql, no - /// ChunkFloorsSql, and above all no PlainTableFloorSql (the unindexed + /// RawChunkFloorSql, and above all no PlainTableFloorSql (the unindexed /// MIN(first_execution_time) WHERE server_id = $1 scan) — counted directly off /// pg_stat_user_tables.seq_scan/idx_scan before and after the call, since PostgreSQL's own /// catalog is the only seam that tells "no statement ran" apart from "a statement ran and returned @@ -438,6 +438,7 @@ private static async Task AssertRawEqualsTableAsync( ViewerDataService.AddServerWindowParameters(raw, ServerId, windowStart, windowEnd); raw.Parameters.Add(new NpgsqlParameter { TypedValue = top }); raw.Parameters.Add(ViewerDataService.DatabaseFilterParameter(null)); + raw.Parameters.Add(new NpgsqlParameter { TypedValue = PerformanceMonitor.Darling.Storage.TopFill.FirstCandidates(top) }); /* #5313: the round's candidate limit, bound last */ await raw.ExecuteNonQueryAsync(ct); } @@ -452,6 +453,7 @@ private static async Task AssertRawEqualsTableAsync( }); table.Parameters.Add(new NpgsqlParameter { TypedValue = top }); table.Parameters.Add(ViewerDataService.DatabaseFilterParameter(null)); + table.Parameters.Add(new NpgsqlParameter { TypedValue = PerformanceMonitor.Darling.Storage.TopFill.FirstCandidates(top) }); /* #5313: the round's candidate limit, bound last */ await table.ExecuteNonQueryAsync(ct); } @@ -475,6 +477,7 @@ private static async Task AssertRawEqualsTableAsync( ViewerDataService.AddServerWindowParameters(command, ServerId, windowStart, windowEnd); command.Parameters.Add(new NpgsqlParameter { TypedValue = top }); command.Parameters.Add(ViewerDataService.DatabaseFilterParameter(null)); + command.Parameters.Add(new NpgsqlParameter { TypedValue = PerformanceMonitor.Darling.Storage.TopFill.FirstCandidates(top) }); /* #5313: the round's candidate limit, bound last */ await using var reader = await command.ExecuteReaderAsync(ct); while (await reader.ReadAsync(ct)) { diff --git a/Darling/Darling.Tests/QueryStoreIntervalWideHistoryNoteTests.cs b/Darling/Darling.Tests/QueryStoreIntervalWideHistoryNoteTests.cs index 8780b04faa..bbb721f384 100644 --- a/Darling/Darling.Tests/QueryStoreIntervalWideHistoryNoteTests.cs +++ b/Darling/Darling.Tests/QueryStoreIntervalWideHistoryNoteTests.cs @@ -9,6 +9,9 @@ using System; using System.IO; using System.Linq; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; using PerformanceMonitor.Darling.Storage; using Xunit; @@ -35,9 +38,9 @@ public sealed class QueryStoreIntervalWideHistoryNoteTests public void HistoryNote_SlowCadence_IsExact() { var bound = QueryStoreIntervalWide.WideStartBound.RawFloorSlowCadence; - Assert.Equal(Lead1 + "this server" + Lead2 + "This server's" + Slow, QueryStoreIntervalWide.HistoryNote(Start, bound, false)); - Assert.Equal(Lead1 + "the servers in scope" + Lead2 + "alpha's" + Slow, QueryStoreIntervalWide.HistoryNote(Start, bound, true, "alpha")); - Assert.Equal(Lead1 + "the servers in scope" + Lead2 + "A server's" + Slow, QueryStoreIntervalWide.HistoryNote(Start, bound, true)); + Assert.Equal(Lead1 + "this server" + Lead2 + "This server's" + Slow, QueryStoreIntervalWide.HistoryNote(StartText, bound, false)); + Assert.Equal(Lead1 + "the servers in scope" + Lead2 + "alpha's" + Slow, QueryStoreIntervalWide.HistoryNote(StartText, bound, true, "alpha")); + Assert.Equal(Lead1 + "the servers in scope" + Lead2 + "A server's" + Slow, QueryStoreIntervalWide.HistoryNote(StartText, bound, true)); } [Fact] @@ -45,8 +48,8 @@ public void HistoryNote_LogNotYetCovering_IsExact() { const string Reason = "The collection log does not yet cover the interval table's purge edge, so older intervals are not read from the interval table."; var bound = QueryStoreIntervalWide.WideStartBound.RawFloorLogNotYetCovering; - Assert.Equal(Lead1 + "this server" + Lead2 + Reason, QueryStoreIntervalWide.HistoryNote(Start, bound, false)); - Assert.Equal(Lead1 + "the servers in scope" + Lead2 + Reason, QueryStoreIntervalWide.HistoryNote(Start, bound, true, "alpha")); + Assert.Equal(Lead1 + "this server" + Lead2 + Reason, QueryStoreIntervalWide.HistoryNote(StartText, bound, false)); + Assert.Equal(Lead1 + "the servers in scope" + Lead2 + Reason, QueryStoreIntervalWide.HistoryNote(StartText, bound, true, "alpha")); } [Theory] @@ -59,22 +62,41 @@ public void HistoryNote_LogNotYetCovering_IsExact() [InlineData(QueryStoreIntervalWide.WideStartBound.Window, false, Lead1 + "this server" + Lead2 + Clamp)] [InlineData(QueryStoreIntervalWide.WideStartBound.Window, true, Lead1 + "the servers in scope" + Lead2 + Clamp)] public void HistoryNote_IsExact_ForEveryBoundAndScope(QueryStoreIntervalWide.WideStartBound bound, bool manyServers, string expected) => - Assert.Equal(expected, QueryStoreIntervalWide.HistoryNote(Start, bound, manyServers)); + Assert.Equal(expected, QueryStoreIntervalWide.HistoryNote(StartText, bound, manyServers)); [Fact] public void HistoryNote_NamesTheSettingServer_OnlyForManyServers() { var expected = Lead1 + "the servers in scope" + Lead2 + "The interval table began keeping complete history for alpha at " + StartText + "."; - Assert.Equal(expected, QueryStoreIntervalWide.HistoryNote(Start, QueryStoreIntervalWide.WideStartBound.FilledSince, manyServers: true, settingServer: "alpha")); + Assert.Equal(expected, QueryStoreIntervalWide.HistoryNote(StartText, QueryStoreIntervalWide.WideStartBound.FilledSince, manyServers: true, settingServer: "alpha")); Assert.Equal( - QueryStoreIntervalWide.HistoryNote(Start, QueryStoreIntervalWide.WideStartBound.FilledSince, manyServers: false), - QueryStoreIntervalWide.HistoryNote(Start, QueryStoreIntervalWide.WideStartBound.FilledSince, manyServers: false, settingServer: "alpha")); + QueryStoreIntervalWide.HistoryNote(StartText, QueryStoreIntervalWide.WideStartBound.FilledSince, manyServers: false), + QueryStoreIntervalWide.HistoryNote(StartText, QueryStoreIntervalWide.WideStartBound.FilledSince, manyServers: false, settingServer: "alpha")); /* The purge-edge and floor reasons name no server: the same text with or without one. */ Assert.Equal( - QueryStoreIntervalWide.HistoryNote(Start, QueryStoreIntervalWide.WideStartBound.TablePurgeEdge, manyServers: true), - QueryStoreIntervalWide.HistoryNote(Start, QueryStoreIntervalWide.WideStartBound.TablePurgeEdge, manyServers: true, settingServer: "alpha")); + QueryStoreIntervalWide.HistoryNote(StartText, QueryStoreIntervalWide.WideStartBound.TablePurgeEdge, manyServers: true), + QueryStoreIntervalWide.HistoryNote(StartText, QueryStoreIntervalWide.WideStartBound.TablePurgeEdge, manyServers: true, settingServer: "alpha")); } + /// #4966: the sentence names the window's start in the one text effective_start prints (UTC, with + /// the Z), for an instant that came in naive, as the interval table's floor does. The web finds the instant in the + /// note by the field's exact text to show it in the browser's zone; a plain "o" of a naive instant has no Z, so it + /// found nothing and the note stayed in bare UTC above a grid of local times. + [Theory] + [MemberData(nameof(EveryBound))] + public void TheNote_NamesTheStartInTheTextEffectiveStartPrints_ForEveryBound(QueryStoreIntervalWide.WideStartBound bound) + { + var naive = new DateTime(2026, 8, 3, 12, 0, 0, DateTimeKind.Unspecified); + var printed = McpHelpers.FormatEffectiveStart(naive); + Assert.EndsWith("Z", printed, StringComparison.Ordinal); + /* Each surface formats the start where it calls the builder: the MCP top-queries table route, and Compose's panel. */ + Assert.Contains(printed, DarlingMcpDataTools.QueryStoreTableNote(naive, bound), StringComparison.Ordinal); + Assert.Contains(printed, DarlingWebEndpoints.QueryStoreHistoryNote(naive, bound), StringComparison.Ordinal); + } + + public static System.Collections.Generic.IEnumerable EveryBound() => + Enum.GetValues().Select(b => new object[] { b }); + [Theory] [InlineData(QueryStoreIntervalWide.WideStartBound.Window, null)] [InlineData(QueryStoreIntervalWide.WideStartBound.RawFloor, null)] @@ -88,7 +110,7 @@ public void BannerReason_IsExact_ForEveryBound(QueryStoreIntervalWide.WideStartB [Fact] public void ComposeForwarder_CarriesTheManyServersText() => Assert.Equal( - QueryStoreIntervalWide.HistoryNote(Start, QueryStoreIntervalWide.WideStartBound.TablePurgeEdge, manyServers: true), + QueryStoreIntervalWide.HistoryNote(StartText, QueryStoreIntervalWide.WideStartBound.TablePurgeEdge, manyServers: true), PerformanceMonitor.Darling.Service.DarlingWebEndpoints.QueryStoreHistoryNote(Start, QueryStoreIntervalWide.WideStartBound.TablePurgeEdge)); /// The reason phrase is built from two halves so this file never matches its own scan, and the scan diff --git a/Darling/Darling.Tests/QueryStoreIntervalWideMinSchemaVersionPinTests.cs b/Darling/Darling.Tests/QueryStoreIntervalWideMinSchemaVersionPinTests.cs index 5bbcdf502a..39f55a0159 100644 --- a/Darling/Darling.Tests/QueryStoreIntervalWideMinSchemaVersionPinTests.cs +++ b/Darling/Darling.Tests/QueryStoreIntervalWideMinSchemaVersionPinTests.cs @@ -30,6 +30,7 @@ namespace Darling.Tests; /// finds that migration by what it creates, not by today's version number, and asserts every constant still /// equals it. /// +[Trait("Stage", "Guard")] public sealed class QueryStoreIntervalWideMinSchemaVersionPinTests { [Fact] diff --git a/Darling/Darling.Tests/QueryStoreLegacyRowIndexLiveTests.cs b/Darling/Darling.Tests/QueryStoreLegacyRowIndexLiveTests.cs new file mode 100644 index 0000000000..45fbda193f --- /dev/null +++ b/Darling/Darling.Tests/QueryStoreLegacyRowIndexLiveTests.cs @@ -0,0 +1,369 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.RegularExpressions; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4972: the legacy-row check (, clause 6 of +/// ReadsTableAsync) gets an index read of the partial index +/// , which the start-path tuning pass builds. Own scratch +/// database per test. +/// +/* #1776 own-store: each live fact mints its own scratch database through ScratchPostgres. */ +public sealed class QueryStoreLegacyRowIndexLiveTests +{ + private const int ServerA = 4972001; + private const int ServerB = 4972002; + + private static readonly Regex NodeLabel = new(@"^\s*(->\s*)?(?.+)$", RegexOptions.Compiled); + + /// + /// Two servers' interval-stamped rows over three days, the older chunks compressed where TimescaleDB is + /// present. The shipped check, EXPLAINed with its real bound parameter types, reads the partial index + /// (Index Only Scan, Index Scan or Bitmap Index Scan) for every uncompressed chunk and never filters a heap scan on the NULL test; and it + /// answers false until a NULL-start row lands for one server. + /// + [Fact] + public async Task TheShippedCheck_PlansOnThePartialIndex_AndAnswersExactly() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the legacy-row index live test."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var timescaleEnabled = await LiveTimescaleProbe.TryEnableAsync(scratch.ConnectionString, ct); + if (timescaleEnabled) + { + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + await Exec(connection, "SELECT _timescaledb_functions.stop_background_workers()", ct); + /* Narrow chunks so the three-day seed spans several, with the older ones compressible. */ + await Exec(connection, "SELECT set_chunk_time_interval('collect.query_store_stats', INTERVAL '1 day')", ct); + } + + /* A fixed anchor, 23:50 UTC on a fixed past day, not the wall clock. Chunks align to UTC midnight and + the newest chunk is the one left uncompressed, so a seed ending at the real "now" leaves that chunk + holding only the hours since midnight. A near-empty chunk is a single heap page, where the Index Only + Scan and the Seq Scan cost within a fraction of a point of each other (about 1.7 against 2.1), so a + little statistics or visibility-map drift can flip the plan. Ending at 23:50 gives it a full day. */ + var utcNow = new DateTime(2026, 1, 15, 23, 50, 0, DateTimeKind.Unspecified); + var windowStart = utcNow.AddDays(-3); + + /* Per-pass contiguous batches, one server after the other, every 30 minutes for three days. */ + for (var pass = 143; pass >= 0; pass--) + { + var collectionTime = utcNow.AddMinutes(-2 - pass * 30); + await SeedPassAsync(connection, ServerA, collectionTime, 30, ct); + await SeedPassAsync(connection, ServerB, collectionTime, 30, ct); + } + + var compressed = 0; + if (timescaleEnabled) + { + compressed = await CompressOlderChunksAsync(connection, ct); + Assert.True(compressed > 0, "no chunk was compressed; the compressed-chunk half of the pin did not run"); + Assert.True(await CountChunksAsync(connection, compressedOnly: false, ct) > 0, + "every chunk was compressed; the uncompressed-chunk half of the pin did not run"); + } + + if (timescaleEnabled) + { + /* The uncompressed chunk is the one the plan assertion is about; keep it a realistic size so a seed + change cannot bring back the near-empty chunk, where a Seq Scan beats the index. */ + await using var newest = new NpgsqlCommand( + "SELECT count(*) FROM collect.query_store_stats WHERE collection_time >= $1", connection); + newest.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = utcNow.Date }); + var newestRows = Convert.ToInt64(await newest.ExecuteScalarAsync(ct), System.Globalization.CultureInfo.InvariantCulture); + Assert.True(newestRows >= 1000, + "the newest (uncompressed) chunk holds only " + newestRows + " rows; it must be a realistic day, not a near-empty chunk"); + } + + await Exec(connection, "ANALYZE collect.query_store_stats", ct); + + /* The production order: the upgraded store already holds data, compressed chunks included, when the + start-path tuning pass first builds the index. */ + await PgTableTuning.ApplyAsync(connection, NullLogger.Instance, ct); + + if (timescaleEnabled) + { + var sizes = await ChunkIndexSizesAsync(connection, compressedOnly: true, ct); + Assert.Equal(compressed, sizes.Count); + Assert.All(sizes, size => Assert.Equal(8192L, size)); + Assert.Equal(0, await CompressedRelationIndexCountAsync(connection, ct)); + } + + var plan = await ExplainShippedCheckAsync(connection, ServerA, windowStart, utcNow, ct); + AssertPlanUsesThePartialIndex(plan, timescaleEnabled, compressed); + + Assert.False(await RunShippedCheckAsync(connection, ServerA, windowStart, utcNow, ct)); + Assert.False(await RunShippedCheckAsync(connection, ServerB, windowStart, utcNow, ct)); + + await InsertNullStartRowAsync(connection, ServerA, utcNow.AddMinutes(-10), ct); + Assert.True(await RunShippedCheckAsync(connection, ServerA, windowStart, utcNow, ct)); + Assert.False(await RunShippedCheckAsync(connection, ServerB, windowStart, utcNow, ct)); + + if (timescaleEnabled) + { + /* Inheritance: a chunk created after the index existed carries it. */ + var before = (await ChunkIndexSizesAsync(connection, compressedOnly: false, ct)).Count; + var chunksBefore = await CountChunksAsync(connection, compressedOnly: null, ct); + await SeedPassAsync(connection, ServerA, utcNow.AddDays(5), 1, ct); + Assert.Equal(chunksBefore + 1, await CountChunksAsync(connection, compressedOnly: null, ct)); + Assert.Equal(before + 1, (await ChunkIndexSizesAsync(connection, compressedOnly: false, ct)).Count); + } + } + + /// + /// The first start-path pass leaves the index with its exact definition and valid; the second-pass + /// no-CREATE-under-writers pin lives in TuningHourlyCreateLiveTests; and with the index dropped, + /// the hourly pass does not build it and warns once. + /// + [Fact] + public async Task TheStartPath_BuildsTheIndexOnce_AndTheHourlyPassNeverBuildsIt() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the legacy-row index live test."); + + var ct = TestContext.Current.CancellationToken; + PgTableTuning.ResetHourlyMissingWarnings(); + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var body = new NpgsqlConnection(scratch.ConnectionString); + await body.OpenAsync(ct); + await PgMigrations.MigrateAsync(body, ct); + await TuningStartPasses.ConvergeAsync(body, ct); + + await using (var def = new NpgsqlCommand( + "SELECT i.indisvalid, pg_get_indexdef(i.indexrelid) FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid " + + "JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'collect' AND c.relname = $1", body)) + { + def.Parameters.AddWithValue(PgTableTuning.LegacyRowIndexName); + await using var reader = await def.ExecuteReaderAsync(ct); + Assert.True(await reader.ReadAsync(ct), "the start path should have built the index"); + Assert.True(reader.GetBoolean(0), "the index should be valid"); + var text = reader.GetString(1); + Assert.Contains("ON collect.query_store_stats", text, StringComparison.Ordinal); + Assert.Contains("(server_id, collection_time)", text, StringComparison.Ordinal); + Assert.Contains("WHERE (interval_start_time_utc IS NULL)", text, StringComparison.Ordinal); + } + + await Exec(body, "DROP INDEX collect." + PgTableTuning.LegacyRowIndexName, ct); + var logger = new CapturingTestLogger(); + var hourly = await PgTableTuning.ApplyAsync(body, logger, hourly: true, ct); + Assert.Equal(PgTableTuning.Statements.Count - 1, hourly); + var warnings = logger.Lines.Where(l => l.StartsWith("Warning:", StringComparison.Ordinal) + && l.Contains(PgTableTuning.LegacyRowIndexName, StringComparison.Ordinal) + && l.Contains("collect.query_store_stats", StringComparison.Ordinal)).ToList(); + Assert.True(warnings.Count == 1, "expected exactly one warning naming the index and its table: " + logger.Joined); + await using var exists = new NpgsqlCommand("SELECT 1 FROM pg_indexes WHERE schemaname = 'collect' AND indexname = $1", body); + exists.Parameters.AddWithValue(PgTableTuning.LegacyRowIndexName); + Assert.Null(await exists.ExecuteScalarAsync(ct)); + } + + private static void AssertPlanUsesThePartialIndex(string plan, bool timescaleEnabled, int compressedChunks) + { + var lines = plan.ReplaceLineEndings("\n").Split('\n', StringSplitOptions.RemoveEmptyEntries); + var nodes = new List<(string Label, List Details)>(); + foreach (var line in lines) + { + var text = NodeLabel.Match(line).Groups["text"].Value.TrimEnd(); + var isDetail = text.Contains(": ", StringComparison.Ordinal) && !line.TrimStart().StartsWith("->", StringComparison.Ordinal) + && nodes.Count > 0; + if (isDetail) + { + nodes[^1].Details.Add(text); + } + else + { + nodes.Add((text, new List())); + } + } + + var scans = nodes.Where(n => n.Label.Contains("Scan", StringComparison.Ordinal) + && (n.Label.Contains("query_store_stats", StringComparison.Ordinal) + || n.Label.Contains("_hyper_", StringComparison.Ordinal)) + && !n.Label.Contains("_compressed", StringComparison.Ordinal)).ToList(); + Assert.True(scans.Count > 0, "no scan of query_store_stats found in the plan:\n" + plan); + + /* Which index read the planner picks depends on visibility-map state and the platform; any read of + the partial index is the fix. A Bitmap Heap Scan counts when its child is a Bitmap Index Scan on it. */ + var idx = PgTableTuning.LegacyRowIndexName; + /* Chunk indexes are renamed with the chunk's prefix, so the label carries the name, not starts with it. */ + var indexReads = scans.Where(n => n.Label.Contains(idx, StringComparison.Ordinal) + && (n.Label.StartsWith("Index Only Scan using ", StringComparison.Ordinal) + || n.Label.StartsWith("Index Scan using ", StringComparison.Ordinal) + || n.Label.StartsWith("Bitmap Index Scan on ", StringComparison.Ordinal))).ToList(); + Assert.True(indexReads.Count > 0, "no scan reads the partial index " + idx + ":\n" + plan); + + foreach (var scan in scans) + { + var columnar = scan.Label.Contains("ColumnarScan", StringComparison.Ordinal); + var isIndexRead = indexReads.Contains(scan); + var isBitmapHeap = scan.Label.StartsWith("Bitmap Heap Scan on ", StringComparison.Ordinal) + && IsFollowedByBitmapIndexScan(nodes, scan, idx); + Assert.True(columnar || isIndexRead || isBitmapHeap, "a chunk is read by a heap scan instead of the partial index:\n" + plan); + /* A row Filter carrying the NULL test is the heap-filtered shape this index replaces. The columnar + scan's vectorized filter is a different line label and is fine. */ + Assert.DoesNotContain(scan.Details, d => d.StartsWith("Filter:", StringComparison.Ordinal) + && d.Contains("interval_start_time_utc IS NULL", StringComparison.Ordinal)); + } + + if (timescaleEnabled && compressedChunks > 0) + { + Assert.Contains(scans, n => n.Label.Contains("ColumnarScan", StringComparison.Ordinal)); + } + } + + private static bool IsFollowedByBitmapIndexScan(List<(string Label, List Details)> nodes, (string Label, List Details) heap, string idx) + { + var at = nodes.IndexOf(heap); + return at >= 0 && at + 1 < nodes.Count + && nodes[at + 1].Label.StartsWith("Bitmap Index Scan on ", StringComparison.Ordinal) + && nodes[at + 1].Label.Contains(idx, StringComparison.Ordinal); + } + + private static async Task CountChunksAsync(NpgsqlConnection connection, bool? compressedOnly, CancellationToken ct) + { + var filter = compressedOnly switch { true => " AND is_compressed", false => " AND NOT is_compressed", _ => string.Empty }; + await using var cmd = new NpgsqlCommand( + "SELECT count(*) FROM timescaledb_information.chunks WHERE hypertable_schema = 'collect' AND hypertable_name = 'query_store_stats'" + filter, connection); + return Convert.ToInt32(await cmd.ExecuteScalarAsync(ct), System.Globalization.CultureInfo.InvariantCulture); + } + + /// Sizes of the partial index's copy on each chunk (compressed ones only, or all), from the catalog. + private static async Task> ChunkIndexSizesAsync(NpgsqlConnection connection, bool compressedOnly, CancellationToken ct) + { + var sizes = new List(); + await using var cmd = new NpgsqlCommand( + "SELECT pg_relation_size(i.indexrelid) FROM timescaledb_information.chunks ch " + + "JOIN pg_class t ON t.relname = ch.chunk_name JOIN pg_namespace tn ON tn.oid = t.relnamespace AND tn.nspname = ch.chunk_schema " + + "JOIN pg_index i ON i.indrelid = t.oid " + + "WHERE ch.hypertable_schema = 'collect' AND ch.hypertable_name = 'query_store_stats' " + + "AND ($1 = false OR ch.is_compressed) " + + "AND pg_get_indexdef(i.indexrelid) LIKE '%WHERE (interval_start_time_utc IS NULL)'", connection); + cmd.Parameters.AddWithValue(compressedOnly); + await using var reader = await cmd.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + sizes.Add(reader.GetInt64(0)); + } + + return sizes; + } + + private static async Task CompressedRelationIndexCountAsync(NpgsqlConnection connection, CancellationToken ct) + { + await using var cmd = new NpgsqlCommand( + "SELECT count(*) FROM pg_indexes WHERE (tablename LIKE '%\\_compressed%' OR tablename LIKE 'compress\\_hyper\\_%') " + + "AND indexdef LIKE '%interval_start_time_utc IS NULL%'", connection); + return Convert.ToInt32(await cmd.ExecuteScalarAsync(ct), System.Globalization.CultureInfo.InvariantCulture); + } + + private static async Task CompressOlderChunksAsync(NpgsqlConnection connection, CancellationToken ct) + { + await Exec(connection, "ALTER TABLE collect.query_store_stats SET (timescaledb.compress, timescaledb.compress_segmentby = 'server_id')", ct); + var chunks = new List(); + await using (var list = new NpgsqlCommand( + "SELECT format('%I.%I', chunk_schema, chunk_name) FROM timescaledb_information.chunks " + + "WHERE hypertable_schema = 'collect' AND hypertable_name = 'query_store_stats' AND NOT is_compressed " + + "ORDER BY range_end DESC OFFSET 1", connection)) + await using (var reader = await list.ExecuteReaderAsync(ct)) + { + while (await reader.ReadAsync(ct)) + { + chunks.Add(reader.GetString(0)); + } + } + + var done = 0; + foreach (var chunk in chunks) + { + await using var compress = new NpgsqlCommand("SELECT compress_chunk($1::regclass, if_not_compressed => true)", connection); + compress.Parameters.AddWithValue(chunk); + await compress.ExecuteNonQueryAsync(ct); + done++; + } + + return done; + } + + private static async Task SeedPassAsync(NpgsqlConnection connection, int serverId, DateTime collectionTime, int rows, CancellationToken ct) + { + await using var insert = new NpgsqlCommand( + "INSERT INTO collect.query_store_stats (collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, " + + "execution_count, avg_duration_us, interval_start_time_utc) " + + "SELECT $1, $2, $3, 'legacy-row-index-e2e', 'db_' || (g % 4), 1000 + g, (1000 + g) * 10, 10 + g, 500 + g, $2 " + + "FROM generate_series(1, $4) AS g", connection); + insert.Parameters.AddWithValue(1L); + insert.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = collectionTime }); + insert.Parameters.AddWithValue(serverId); + insert.Parameters.AddWithValue(rows); + await insert.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertNullStartRowAsync(NpgsqlConnection connection, int serverId, DateTime collectionTime, CancellationToken ct) + { + await using var insert = new NpgsqlCommand( + "INSERT INTO collect.query_store_stats (collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, " + + "execution_count, avg_duration_us) VALUES ($1, $2, $3, 'legacy-row-index-e2e', 'db_0', 1, 10, 1, 1)", connection); + insert.Parameters.AddWithValue(1L); + insert.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = collectionTime }); + insert.Parameters.AddWithValue(serverId); + await insert.ExecuteNonQueryAsync(ct); + } + + private static NpgsqlCommand BindShippedCheck(string sql, NpgsqlConnection connection, int serverId, DateTime start, DateTime end) + { + var command = new NpgsqlCommand(sql, connection); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = serverId }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = DateTime.SpecifyKind(start, DateTimeKind.Unspecified) }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = DateTime.SpecifyKind(end, DateTimeKind.Unspecified) }); + return command; + } + + private static async Task RunShippedCheckAsync(NpgsqlConnection connection, int serverId, DateTime start, DateTime end, CancellationToken ct) + { + await using var command = BindShippedCheck(QueryStoreIntervalWide.HasLegacyRowSql, connection, serverId, start, end); + return (bool)(await command.ExecuteScalarAsync(ct))!; + } + + private static async Task ExplainShippedCheckAsync(NpgsqlConnection connection, int serverId, DateTime start, DateTime end, CancellationToken ct) + { + await using var command = BindShippedCheck("EXPLAIN (COSTS OFF) " + QueryStoreIntervalWide.HasLegacyRowSql, connection, serverId, start, end); + var lines = new List(); + await using var reader = await command.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + lines.Add(reader.GetString(0)); + } + + return string.Join("\n", lines); + } + + private static async Task Exec(NpgsqlConnection c, string sql, CancellationToken ct) + { + await using var cmd = new NpgsqlCommand(sql, c); + await cmd.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/QueryStorePlanViaMapLiveTests.cs b/Darling/Darling.Tests/QueryStorePlanViaMapLiveTests.cs new file mode 100644 index 0000000000..016f5683a0 --- /dev/null +++ b/Darling/Darling.Tests/QueryStorePlanViaMapLiveTests.cs @@ -0,0 +1,363 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5257: the collector ships query_store_stats.query_plan_text as a NULL placeholder since #2210; the plan +/// lives once in query_plan_dim, reached through collect.query_store_plan_map. The Service +/// (analyze_query_store_plan) and Viewer readers must find a plan there, keep reading old inline rows, and +/// treat a NULL-digest map row (a marker for a plan that will never exist) as absent. +/// +[Collection("live-postgres")] +public sealed class QueryStorePlanViaMapLiveTests +{ + private const string ServerName = "darling-qs-map-e2e"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + private const string Db = "MapPlanDb"; + + private static string Plan(string label) => + "" + + "" + + "" + + $"" + + ""; + + private static readonly string TextPlan = Plan("map-text"); + private static readonly string GzPlan = Plan("map-gz"); + private static readonly string OlderPlan = Plan("map-older"); + private static readonly string InlineOnly = Plan("inline-only"); + private static readonly string InlineBehindMarker = Plan("inline-behind-marker"); + private static readonly string MapNewer = Plan("map-newer"); + private static readonly string InlineOlder = Plan("inline-older"); + private static readonly string MapQ7 = Plan("map-q7"); + private static readonly string OrphanDigestPlan = Plan("orphan-digest-no-dim-row"); + private static readonly string InlineBehindOrphan = Plan("inline-behind-orphan"); + private static readonly string Stale = Plan("map-stale-8d"); + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + [Fact] + public void ViewerAndServiceReaders_ShareOneSqlShape_AndTheMapReadNeverTouchesTheFactTable() + { + static string Norm(string s) => string.Join(' ', s.Split((char[]?)null, StringSplitOptions.RemoveEmptyEntries)); + Assert.Equal(Norm(DarlingStoredPlanReader.QueryStorePlanViaMapSql), Norm(ViewerDataService.QueryStorePlanViaMapSql)); + foreach (var sql in new[] { DarlingStoredPlanReader.QueryStorePlanViaMapSql, ViewerDataService.QueryStorePlanViaMapSql }) + { + Assert.Contains("FROM collect.query_store_plan_map", sql, StringComparison.Ordinal); + Assert.Contains("LEFT JOIN query_plan_dim", sql, StringComparison.Ordinal); + Assert.Contains("m.plan_id = $3", sql, StringComparison.Ordinal); + /* Two index lookups: the fact table is never read once a plan_id is known. */ + Assert.DoesNotContain("query_store_stats", sql, StringComparison.Ordinal); + } + } + + [Fact] + public void TheUnpinnedCandidateRead_IsTimeBounded_NewestFirst_WithAPlanIdTiebreak() + { + var bounded = DarlingStoredPlanReader.QueryStorePlanCandidatesSql; + Assert.Contains("collection_time >= $4", bounded, StringComparison.Ordinal); + Assert.Contains("ORDER BY last_collected DESC, plan_id DESC", bounded, StringComparison.Ordinal); + + var unbounded = DarlingStoredPlanReader.QueryStorePlanCandidatesUnboundedSql; + Assert.DoesNotContain("collection_time >=", unbounded, StringComparison.Ordinal); + Assert.Contains("ORDER BY last_collected DESC, plan_id DESC", unbounded, StringComparison.Ordinal); + } + + [Fact] + public async Task BothReaders_FindMapOnlyPlans_ReadInlineRows_AndTreatMarkersAsAbsent_AgainstDevPostgres() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live query-store plan-map test."); + var ct = TestContext.Current.CancellationToken; + + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + await using var viewer = new ViewerDataService(cs!); + + var bodySucceeded = false; + try + { + /* Fixed anchors, never the wall clock. */ + var t0 = new DateTime(2026, 3, 1, 12, 0, 0, DateTimeKind.Unspecified); + var t1 = t0.AddHours(1); + + await RegisterServerAsync(connection, ct); + + /* 1: text-stored plan. */ + await InsertFactAsync(connection, t0, queryId: 1, planId: 11, inline: null, ct); + await InsertDimAsync(connection, TextPlan, gz: false, ct); + await InsertMapAsync(connection, 11, TextPlan, ct); + + /* 2: gzip-stored plan. */ + await InsertFactAsync(connection, t0, queryId: 2, planId: 21, inline: null, ct); + await InsertDimAsync(connection, GzPlan, gz: true, ct); + await InsertMapAsync(connection, 21, GzPlan, ct); + + /* 3: an older real plan, then a newer plan_id whose map row is a NULL-digest marker. */ + await InsertFactAsync(connection, t0, queryId: 3, planId: 31, inline: null, ct); + await InsertDimAsync(connection, OlderPlan, gz: false, ct); + await InsertMapAsync(connection, 31, OlderPlan, ct); + await InsertFactAsync(connection, t1, queryId: 3, planId: 32, inline: null, ct); + await InsertMapAsync(connection, 32, null, ct); + + /* 4: an old row that carries the text inline and has no map row. */ + await InsertFactAsync(connection, t0, queryId: 4, planId: 41, inline: InlineOnly, ct); + + /* 5: a marker on the only plan_id, with the text still inline: the inline value answers. */ + await InsertFactAsync(connection, t0, queryId: 5, planId: 51, inline: InlineBehindMarker, ct); + await InsertMapAsync(connection, 51, null, ct); + + /* 6: an OLDER plan that lives inline and a NEWER plan that lives in the map, for one query. */ + await InsertFactAsync(connection, t0, queryId: 6, planId: 61, inline: InlineOlder, ct); + await InsertFactAsync(connection, t1, queryId: 6, planId: 62, inline: null, ct); + await InsertDimAsync(connection, MapNewer, gz: false, ct); + await InsertMapAsync(connection, 62, MapNewer, ct); + + /* 7: a real older map plan, then a newer plan_id whose map row points at a digest with NO dimension + row (the dimension GC removed it). The fact row still has inline text: it must not be read. */ + await InsertFactAsync(connection, t0, queryId: 7, planId: 71, inline: null, ct); + await InsertDimAsync(connection, MapQ7, gz: false, ct); + await InsertMapAsync(connection, 71, MapQ7, ct); + await InsertFactAsync(connection, t1, queryId: 7, planId: 72, inline: InlineBehindOrphan, ct); + await InsertMapAsync(connection, 72, OrphanDigestPlan, ct); + + /* 8: the only plan was last seen 8 days before asOf, outside the 7-day bound. */ + var asOf = new DateTime(2026, 3, 2, 12, 0, 0, DateTimeKind.Unspecified); + await InsertFactAsync(connection, asOf.AddDays(-8), queryId: 8, planId: 81, inline: null, ct); + await InsertDimAsync(connection, Stale, gz: false, ct); + await InsertMapAsync(connection, 81, Stale, ct); + + /* ---- Service reader (plan_id optional). */ + Task Svc(long q, long? p) => DarlingStoredPlanReader.GetQueryStorePlanTextAsync(postgres, ServerId, Db, q, planId: p, asOf: asOf, cancellationToken: ct); + Task SvcRead(long q, long? p) => DarlingStoredPlanReader.ResolveQueryStorePlanAsync(postgres, ServerId, Db, q, planId: p, asOf: asOf, cancellationToken: ct); + Assert.Equal(TextPlan, await Svc(1, null)); + Assert.Equal(TextPlan, await Svc(1, 11)); + Assert.Equal(GzPlan, await Svc(2, null)); + /* Newest plan_id is a marker: the unpinned read falls to the older plan; the pinned one has no content. */ + Assert.Equal(OlderPlan, await Svc(3, null)); + Assert.Equal(OlderPlan, await Svc(3, 31)); + Assert.Null(await Svc(3, 32)); + Assert.Equal(InlineOnly, await Svc(4, null)); + /* A map row with no content is ABSENT: the inline text behind a marker is not read. */ + Assert.Null(await Svc(5, 51)); + Assert.Null(await Svc(5, null)); + Assert.Null(await Svc(999, null)); + Assert.Null(await Svc(1, 999)); + /* The plan_id names the plan; the pinned read goes by the map's key alone. */ + Assert.Equal(TextPlan, await Svc(999, 11)); + + /* Older plan inline, newer plan in the map: the map plan wins unpinned; each pinned read answers its own. */ + Assert.Equal(MapNewer, await Svc(6, null)); + Assert.Equal(MapNewer, await Svc(6, 62)); + Assert.Equal(InlineOlder, await Svc(6, 61)); + + /* A map row whose digest has no dimension row is absent; the unpinned read moves to the next plan. */ + Assert.Null(await Svc(7, 72)); + Assert.Equal(MapQ7, await Svc(7, null)); + + /* Last seen 8 days before asOf: the bounded pass finds nothing and the one unbounded pass does. */ + Assert.Equal(Stale, await Svc(8, null)); + Assert.Equal(Stale, await Svc(8, 81)); + + /* The resolved plan_id comes back with the plan. */ + Assert.Equal(11, (await SvcRead(1, null))!.PlanId); + Assert.Equal(31, (await SvcRead(3, null))!.PlanId); + Assert.Equal(62, (await SvcRead(6, null))!.PlanId); + Assert.Equal(71, (await SvcRead(7, null))!.PlanId); + Assert.Equal(81, (await SvcRead(8, null))!.PlanId); + Assert.Equal(61, (await SvcRead(6, 61))!.PlanId); + Assert.Null(await SvcRead(5, null)); + + /* ---- Viewer reader (plan_id required). */ + Task Vw(long q, long p) => viewer.GetQueryStorePlanTextAsync(ServerId, Db, q, p, cancellationToken: ct); + Assert.Equal(TextPlan, await Vw(1, 11)); + Assert.Equal(GzPlan, await Vw(2, 21)); + Assert.Equal(OlderPlan, await Vw(3, 31)); + Assert.Null(await Vw(3, 32)); + Assert.Equal(InlineOnly, await Vw(4, 41)); + Assert.Null(await Vw(5, 51)); + Assert.Equal(MapNewer, await Vw(6, 62)); + Assert.Equal(InlineOlder, await Vw(6, 61)); + Assert.Null(await Vw(7, 72)); + Assert.Equal(Stale, await Vw(8, 81)); + Assert.Null(await Vw(1, 999)); + /* The map is read by plan_id alone (a plan_id belongs to one query_id), so a query_id that does not + own the plan still gets it: the fact table is not consulted. */ + Assert.Equal(TextPlan, await Vw(999, 11)); + Assert.Null(await viewer.GetQueryStorePlanTextAsync(ServerId + 1, Db, 1, 11, cancellationToken: ct)); + + /* ---- Third reader: the actual-plan resolver. */ + async Task<(string? Text, string? Plan)> Resolve(long q) + { + await using var command = new NpgsqlCommand(DarlingWorker.ResolveStoredQueryStoreForActualPlanSql, connection); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(Db); + command.Parameters.AddWithValue(q); + await using var reader = await command.ExecuteReaderAsync(ct); + if (!await reader.ReadAsync(ct)) + { + return (null, null); + } + + return ( + reader.IsDBNull(0) ? null : reader.GetString(0), + PayloadDimensions.ResolveContent( + reader.IsDBNull(1) ? null : reader.GetString(1), + reader.IsDBNull(3) ? null : reader.GetFieldValue(3))); + } + + Assert.Equal(("SELECT /*q1*/", TextPlan), await Resolve(1)); + Assert.Equal(("SELECT /*q2*/", GzPlan), await Resolve(2)); + Assert.Equal(OlderPlan, (await Resolve(3)).Plan); + Assert.Equal(InlineOnly, (await Resolve(4)).Plan); + /* A marker: the query text still answers (it is what gets re-executed); the estimated plan is absent. */ + Assert.Equal(("SELECT /*q5*/", (string?)null), await Resolve(5)); + Assert.Equal(MapNewer, (await Resolve(6)).Plan); + Assert.Equal(MapQ7, (await Resolve(7)).Plan); + Assert.Equal(Stale, (await Resolve(8)).Plan); + Assert.Equal((null, null), await Resolve(999)); + + /* ---- analyze_query_store_plan end to end: the analysed statement and the resolved plan_id. */ + async Task<(string Identifier, string? Statement)> Analyse(long q, long? p) + { + var json = await DarlingMcpPlanTools.AnalyzeQueryStorePlan(postgres, Db, q, ServerName, p, cancellationToken: ct); + using var doc = JsonDocument.Parse(json); + Assert.False(doc.RootElement.TryGetProperty("status", out var status) && status.GetString() == "unavailable", json); + Assert.Equal("query_store", doc.RootElement.GetProperty("source").GetString()); + var statements = doc.RootElement.GetProperty("statements"); + return ( + doc.RootElement.GetProperty("identifier").GetString()!, + statements.GetArrayLength() == 0 ? null : statements[0].GetProperty("statement_text").GetString()); + } + + Assert.Equal(($"{Db}:1:11", "SELECT 'map-text'"), await Analyse(1, null)); + Assert.Equal(($"{Db}:2:21", "SELECT 'map-gz'"), await Analyse(2, null)); + /* The newest plan is a marker, so the analysed plan is the older one, and the identifier says which. */ + Assert.Equal(($"{Db}:3:31", "SELECT 'map-older'"), await Analyse(3, null)); + Assert.Equal(($"{Db}:6:62", "SELECT 'map-newer'"), await Analyse(6, null)); + Assert.Equal(($"{Db}:6:61", "SELECT 'inline-older'"), await Analyse(6, 61)); + Assert.Equal(($"{Db}:1:11", "SELECT 'map-text'"), await Analyse(1, 11)); + var missing = await DarlingMcpPlanTools.AnalyzeQueryStorePlan(postgres, Db, 3, ServerName, 32, cancellationToken: ct); + using (var doc = JsonDocument.Parse(missing)) + { + Assert.Equal("unavailable", doc.RootElement.GetProperty("status").GetString()); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, DeleteAsync); + } + } + + private static IEnumerable AllPlans() => new[] { TextPlan, GzPlan, OlderPlan, MapNewer, MapQ7, Stale }; + + private static async Task DeleteAsync(NpgsqlConnection connection, CancellationToken ct) + { + using (var cleanup = new NpgsqlCommand( + $"DELETE FROM query_store_stats WHERE server_id = {ServerId}; " + + $"DELETE FROM collect.query_store_plan_map WHERE server_id = {ServerId}; " + + $"DELETE FROM servers WHERE server_id = {ServerId};", connection)) + { + await cleanup.ExecuteNonQueryAsync(ct); + } + + foreach (var plan in AllPlans()) + { + using var dim = new NpgsqlCommand("DELETE FROM query_plan_dim WHERE digest = $1", connection); + dim.Parameters.AddWithValue(PayloadDimensions.Digest(plan)); + await dim.ExecuteNonQueryAsync(ct); + } + } + + private static async Task RegisterServerAsync(NpgsqlConnection connection, CancellationToken ct) + { + using var command = new NpgsqlCommand(@" +INSERT INTO servers (server_id, server_name, display_name, is_enabled, created_date, modified_date) +VALUES ($1, $2, $2, TRUE, $3, $3) +ON CONFLICT (server_id) DO UPDATE SET is_enabled = TRUE;", connection); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(ServerName); + command.Parameters.AddWithValue(new DateTime(2026, 3, 1, 0, 0, 0, DateTimeKind.Unspecified)); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertFactAsync( + NpgsqlConnection connection, DateTime collectionTime, long queryId, long planId, string? inline, CancellationToken ct) + { + using var command = new NpgsqlCommand(@" +INSERT INTO query_store_stats + (collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, query_plan_text, query_text) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)", connection); + command.Parameters.AddWithValue(CollectionIdGenerator.Next()); + command.Parameters.AddWithValue(collectionTime); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(ServerName); + command.Parameters.AddWithValue(Db); + command.Parameters.AddWithValue(queryId); + command.Parameters.AddWithValue(planId); + command.Parameters.AddWithValue((object?)inline ?? DBNull.Value); + command.Parameters.AddWithValue($"SELECT /*q{queryId}*/"); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertDimAsync(NpgsqlConnection connection, string plan, bool gz, CancellationToken ct) + { + using var command = new NpgsqlCommand( + gz + ? "INSERT INTO query_plan_dim (digest, query_plan_gz, last_seen) VALUES ($1, $2, $3) ON CONFLICT DO NOTHING" + : "INSERT INTO query_plan_dim (digest, query_plan_xml, last_seen) VALUES ($1, $2, $3) ON CONFLICT DO NOTHING", + connection); + command.Parameters.AddWithValue(PayloadDimensions.Digest(plan)); + if (gz) + { + command.Parameters.AddWithValue(PayloadDimensions.CompressContent(plan)); + } + else + { + command.Parameters.AddWithValue(plan); + } + + command.Parameters.AddWithValue(new DateTime(2026, 3, 1, 12, 0, 0, DateTimeKind.Unspecified)); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertMapAsync(NpgsqlConnection connection, long planId, string? plan, CancellationToken ct) + { + using var command = new NpgsqlCommand( + "INSERT INTO collect.query_store_plan_map (server_id, database_name, plan_id, digest, plan_hash, last_seen) VALUES ($1, $2, $3, $4, $5, $6)", + connection); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(Db); + command.Parameters.AddWithValue(planId); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Bytea, Value = plan is null ? DBNull.Value : PayloadDimensions.Digest(plan) }); + command.Parameters.AddWithValue("0x5257"); + command.Parameters.AddWithValue(new DateTime(2026, 3, 1, 12, 0, 0, DateTimeKind.Unspecified)); + await command.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/QueryStoreRankedHarness.cs b/Darling/Darling.Tests/QueryStoreRankedHarness.cs new file mode 100644 index 0000000000..59d54a787f --- /dev/null +++ b/Darling/Darling.Tests/QueryStoreRankedHarness.cs @@ -0,0 +1,346 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Text.Json.Nodes; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5582: the reusable half of the Query Store RankedTimeSeries exactness proof. It runs ONE panel through two compile +/// switches of (the candidate text and the oracle text) on the same stored rows +/// and says whether the result sets are equal. Part 3 of #5582 (an exact rollup) calls with its +/// own candidate delegate and the same oracle, over the same rows. +/// +/// What "equal" means. The same (bucket, group...) keys, and for every key the same value: exactly (bit for bit +/// after the final cast to double) when relativeTolerance is 0, within that relative tolerance otherwise. Every Query +/// Store fact column is bigint (V145), so every partial sum, count, minimum and maximum a candidate keeps is an exact +/// integer or numeric and the tolerance is 0; the parameter exists for a candidate that keeps a float partial. +/// +/// Ties at the top-N cutoff. Neither text orders the rank beyond value DESC NULLS LAST, so which of +/// several groups tied at the cutoff takes the last slot is the plan's choice. When the two member sets differ the harness +/// reads every group's true total from the ranked (non-time) panel, which neither switch changes, and accepts the difference +/// only if each side is a valid top N under those totals and the groups that differ all sit exactly at the cutoff value. The +/// rows of the groups both sides chose must still be equal. reports whether it +/// happened, so a test that expects the texts to agree can assert it. +/// +internal static class QueryStoreRankedHarness +{ + internal static readonly DateTime WindowStart = new(2026, 8, 1, 0, 0, 0, DateTimeKind.Unspecified); + internal static readonly DateTime WindowEnd = WindowStart.AddDays(3); + + /// The registered servers the seed writes, as (id, name). + internal static readonly (int Id, string Name)[] Servers = + { + (-58201, "rank-srv-1"), + (-58202, "rank-srv-2"), + (-58203, "rank-srv-3"), + }; + + /// Compiles a plan; the shape of with its switch bound. + internal delegate (ComposeCompiled? Compiled, string? Error) Compiler(PanelPlan plan, ComposeRunContext context); + + /// The compiler as the product runs it. + internal static readonly Compiler Product = (plan, context) => ComposeCompiler.CompileCore(plan, context, singleScanRankedTimeSeries: true); + + /// The compiler for the text every route emitted before #5582 (two scans of the fact rows). + internal static readonly Compiler Oracle = (plan, context) => ComposeCompiler.CompileCore(plan, context, singleScanRankedTimeSeries: false); + + /// The outcome of one comparison. + internal sealed record Comparison( + bool Equal, int CandidateRows, int OracleRows, bool MemberSetsDiffered, string? Difference, string CandidateSql, string OracleSql); + + /// The wide-route context for a window: Query Store reads the wide table, no rollups. is + /// the runner's member count of the group dimension (#5582); the default, 1,000, is a bounded group, so the single-scan gate's + /// member term passes and the bucket cap and the group column decide, as the seeded groups here are a handful. + internal static ComposeRunContext WideContext(DateTime start, DateTime end, IReadOnlyList? servers = null, long? groupMembers = 1_000) => + new(servers, start, end, ComposeRunContext.NoVariables, RollupAvailability.None, end, RollupCoverage.Unknown, QueryStoreWideEligible: true, QueryStoreGroupMembers: groupMembers); + + internal static PanelPlan Parse(string planJson) + { + var (plan, error) = ComposeSpec.TryParsePanel((JsonObject)JsonNode.Parse(planJson)!, Array.Empty()); + Assert.True(error is null, $"{planJson}: {error}"); + return plan!; + } + + /// Runs through both compilers over the same context and compares the rows. + internal static Task CompareAsync( + NpgsqlConnection connection, string planJson, ComposeRunContext context, Compiler candidate, Compiler oracle, + double relativeTolerance, CancellationToken ct) => + CompareAsync(connection, planJson, context, candidate, context, oracle, relativeTolerance, ct); + + /// The same comparison with a context of its own for each side (#5582 part 3): the stamp route's candidate carries a + /// QueryStoreStampThrough and its oracle, the wide-table route, does not. + internal static async Task CompareAsync( + NpgsqlConnection connection, string planJson, ComposeRunContext candidateContext, Compiler candidate, + ComposeRunContext context, Compiler oracle, double relativeTolerance, CancellationToken ct) + { + var plan = Parse(planJson); + var (candidateSql, candidateRows) = await RunAsync(connection, plan, candidateContext, candidate, ct); + var (oracleSql, oracleRows) = await RunAsync(connection, plan, context, oracle, ct); + /* The columns before the value: the groups, plus the bucket for a time series (#5582 part 3 widened this from the ranked time series only). */ + var width = plan.GroupBy.Count + 1 + (plan.Mode is PanelMode.TimeSeries or PanelMode.RankedTimeSeries ? 1 : 0); + var candidateByKey = ByKey(candidateRows, width, out var candidateDuplicate); + var oracleByKey = ByKey(oracleRows, width, out var oracleDuplicate); + + Comparison Make(bool equal, bool differed, string? difference) => + new(equal, candidateRows.Count, oracleRows.Count, differed, difference, candidateSql, oracleSql); + + if (candidateDuplicate is not null || oracleDuplicate is not null) + { + return Make(false, false, $"a result set repeats the key {candidateDuplicate ?? oracleDuplicate}"); + } + + var sameKeys = candidateByKey.Keys.ToHashSet(StringComparer.Ordinal).SetEquals(oracleByKey.Keys); + if (sameKeys) + { + var wrong = candidateByKey.FirstOrDefault(kv => !ValuesEqual(kv.Value, oracleByKey[kv.Key], relativeTolerance)); + return wrong.Key is null + ? Make(true, false, null) + : Make(false, false, $"key {wrong.Key}: candidate {Show(wrong.Value)}, oracle {Show(oracleByKey[wrong.Key])}"); + } + + /* The key sets differ. For a panel with no top-N rank that is a plain mismatch; for a ranked one it may be a tie at + the cutoff, which the totals decide. */ + if (plan.TopN is not > 0 || plan.Mode != PanelMode.RankedTimeSeries) + { + return Make(false, false, "the key sets differ and the panel has no rank to explain it"); + } + + var totals = await TotalsAsync(connection, planJson, context, oracle, ct); + var candidateMembers = GroupsOf(candidateByKey); + var oracleMembers = GroupsOf(oracleByKey); + var verdict = TiesExplain(totals, plan.TopN, candidateMembers, oracleMembers); + if (verdict is not null) + { + return Make(false, true, verdict); + } + + var common = candidateMembers.Intersect(oracleMembers, StringComparer.Ordinal).ToHashSet(StringComparer.Ordinal); + foreach (var (key, value) in candidateByKey.Where(kv => common.Contains(GroupOf(kv.Key)))) + { + if (!oracleByKey.TryGetValue(key, out var other) || !ValuesEqual(value, other, relativeTolerance)) + { + return Make(false, true, $"a group both sides chose differs at key {key}: candidate {Show(value)}, oracle {Show(other)}"); + } + } + + return Make(true, true, null); + } + + private static string? TiesExplain(Dictionary totals, int topN, HashSet candidate, HashSet oracle) + { + /* Best first: larger totals first, NULL last, as the rank orders them. */ + var ordered = totals.OrderByDescending(kv => kv.Value.HasValue).ThenByDescending(kv => kv.Value ?? 0d).ToList(); + var take = Math.Min(topN, ordered.Count); + if (take == 0) + { + return "no totals"; + } + + var cutoff = ordered[take - 1].Value; + bool Better(double? a, double? b) => a.HasValue && (!b.HasValue || a.Value > b.Value); + var mustHave = ordered.Where(kv => Better(kv.Value, cutoff)).Select(kv => kv.Key).ToHashSet(StringComparer.Ordinal); + foreach (var (name, members) in new[] { ("candidate", candidate), ("oracle", oracle) }) + { + if (members.Count != take) + { + return $"{name} chose {members.Count} groups, expected {take}"; + } + + if (!mustHave.IsSubsetOf(members)) + { + return $"{name} left out a group that beats the cutoff: {string.Join(", ", mustHave.Except(members))}"; + } + + var below = members.FirstOrDefault(m => totals.TryGetValue(m, out var t) && Better(cutoff, t)); + if (below is not null) + { + return $"{name} chose a group below the cutoff: {below}"; + } + } + + return null; + } + + private static async Task> TotalsAsync( + NpgsqlConnection connection, string planJson, ComposeRunContext context, Compiler compiler, CancellationToken ct) + { + var json = (JsonObject)JsonNode.Parse(planJson)!; + json.Remove("timeBucket"); + json.Remove("includeOther"); + json["topN"] = ComposeLimits.MaxTopN; + json["viz"] = "table"; + var plan = Parse(json.ToJsonString()); + var (_, rows) = await RunAsync(connection, plan, context, compiler, ct); + var totals = new Dictionary(StringComparer.Ordinal); + foreach (var row in rows) + { + totals[string.Join("|", row.Take(row.Length - 1).Select(Cell))] = row[^1] as double?; + } + + return totals; + } + + private static async Task<(string Sql, List Rows)> RunAsync( + NpgsqlConnection connection, PanelPlan plan, ComposeRunContext context, Compiler compiler, CancellationToken ct) + { + var (compiled, error) = compiler(plan, context); + Assert.True(error is null, error); + Assert.NotNull(compiled); + var rows = new List(); + await using var command = new NpgsqlCommand(compiled!.Sql, connection); + foreach (var parameter in compiled.Parameters) + { + command.Parameters.Add(parameter); + } + + await using var reader = await command.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + var values = new object?[reader.FieldCount]; + for (var i = 0; i < reader.FieldCount; i++) + { + values[i] = reader.IsDBNull(i) ? null : reader.GetValue(i); + } + + rows.Add(values); + } + + return (compiled.Sql, rows); + } + + private static string Cell(object? value) => value is null ? "" : Convert.ToString(value, CultureInfo.InvariantCulture)!; + + private static Dictionary ByKey(List rows, int width, out string? duplicate) + { + duplicate = null; + var byKey = new Dictionary(StringComparer.Ordinal); + foreach (var row in rows) + { + Assert.Equal(width, row.Length); + var key = string.Join("|", row.Take(width - 1).Select(Cell)); + if (!byKey.TryAdd(key, row[^1] is null ? null : Convert.ToDouble(row[^1], CultureInfo.InvariantCulture))) + { + duplicate = key; + } + } + + return byKey; + } + + /// The group part of a result key (everything after the bucket); null for a key with no group. + private static string GroupOf(string key) => key[(key.IndexOf('|', StringComparison.Ordinal) + 1)..]; + + private static HashSet GroupsOf(Dictionary byKey) => + byKey.Keys.Select(GroupOf).Where(g => !g.Contains(ComposeCompiler.OtherSeriesLabel, StringComparison.Ordinal)).ToHashSet(StringComparer.Ordinal); + + private static string Show(double? value) => value.HasValue ? value.Value.ToString("R", CultureInfo.InvariantCulture) : ""; + + /// Equal when both are NULL, or both are numbers within of each other (0: bit for bit). + internal static bool ValuesEqual(double? a, double? b, double relativeTolerance) + { + if (a is null || b is null) + { + return a is null && b is null; + } + + if (a.Value.Equals(b.Value)) + { + return true; + } + + if (relativeTolerance <= 0 || double.IsNaN(a.Value) || double.IsNaN(b.Value)) + { + return false; + } + + return Math.Abs(a.Value - b.Value) <= relativeTolerance * Math.Max(Math.Abs(a.Value), Math.Abs(b.Value)); + } + + /// + /// Writes the exactness rows into collect.query_store_interval_wide of a freshly migrated database and registers + /// . Deterministic (no random). 6,000 generated rows over three days and three servers, built to hold + /// every case the proof needs: + /// + /// hours 3, 9, 15, ... of the window hold no row (empty buckets); + /// module_name and database_name are NULL on some rows (a NULL group key), mod0 to mod16 + /// have many rows, only_one has one row, allnull has NULL in every measured column; + /// execution_count, avg_duration_us and avg_cpu_time_us are NULL on some rows; the duration and CPU columns reach 4e12, + /// so a per-group sum of their products exceeds a bigint (the numeric path); + /// the tb_ modules: two big ones, six that tie exactly, two small ones, each in one bucket, for the cutoff tie. + /// + /// + internal static async Task SeedAsync(NpgsqlConnection connection, CancellationToken ct) + { + foreach (var (id, name) in Servers) + { + await DarlingMcpTestData.RegisterServerAsync(connection, id, name, ct); + } + + await using var generated = new NpgsqlCommand( + @" +INSERT INTO collect.query_store_interval_wide + (collection_time, server_id, database_name, query_id, plan_id, execution_type_desc, first_execution_time, last_execution_time, + module_name, query_hash, execution_count, avg_duration_us, max_duration_us, avg_cpu_time_us, max_cpu_time_us, + query_plan_hash, runtime_stats_interval_id, interval_start_time_utc) +SELECT + $1::timestamp + make_interval(hours => s.h, mins => (s.i * 7) % 60, secs => (s.i % 13)), + CASE s.i % 3 WHEN 0 THEN $2 WHEN 1 THEN $3 ELSE $4 END, + CASE WHEN s.i % 29 = 0 THEN NULL ELSE 'db' || (s.i % 4) END, + s.i, s.i, 'Regular', $1::timestamp - interval '1 hour', $1::timestamp, + CASE WHEN s.i = 4242 THEN 'only_one' WHEN s.i % 101 = 7 THEN 'allnull' WHEN s.i % 5 = 0 THEN NULL ELSE 'mod' || (s.i % 17) END, + '0x' || lpad(to_hex(s.i % 400), 16, '0'), + CASE WHEN s.i % 11 = 0 OR s.i % 101 = 7 THEN NULL ELSE 1 + (s.i * 31) % 997 END, + CASE WHEN s.i % 9 = 0 OR s.i % 101 = 7 THEN NULL ELSE (s.i::bigint * 982451653) % 4000000000000 END, + CASE WHEN s.i % 13 = 0 OR s.i % 101 = 7 THEN NULL ELSE (s.i::bigint * 15485863) % 9000000 END, + CASE WHEN s.i % 7 = 0 OR s.i % 101 = 7 THEN NULL ELSE (s.i::bigint * 179424673) % 4000000000000 END, + CASE WHEN s.i % 17 = 0 OR s.i % 101 = 7 THEN NULL ELSE (s.i::bigint * 32452843) % 9000000 END, + '0x' || lpad(to_hex(s.i), 16, '0'), s.i, $1::timestamp - interval '1 hour' +FROM (SELECT i, (i * 13) % 72 AS h FROM generate_series(1, 6000) AS i) AS s +WHERE s.h % 6 <> 3;", connection); + generated.Parameters.AddWithValue(DateTime.SpecifyKind(WindowStart, DateTimeKind.Unspecified)); + generated.Parameters.AddWithValue(Servers[0].Id); + generated.Parameters.AddWithValue(Servers[1].Id); + generated.Parameters.AddWithValue(Servers[2].Id); + await generated.ExecuteNonQueryAsync(ct); + + /* The cutoff tie: tb_big1 and tb_big2 beat everything, tb_tie0..5 are identical in every measure, tb_small1/2 are below. + One row each, in the same hour, so every aggregate of a tied group is the same number. */ + await using var ties = new NpgsqlCommand( + @" +INSERT INTO collect.query_store_interval_wide + (collection_time, server_id, database_name, query_id, plan_id, execution_type_desc, first_execution_time, last_execution_time, + module_name, query_hash, execution_count, avg_duration_us, max_duration_us, avg_cpu_time_us, max_cpu_time_us, + query_plan_hash, runtime_stats_interval_id, interval_start_time_utc) +SELECT $1::timestamp + interval '10 hours 5 minutes', $2, 'dbTie', 900000 + t.n, 900000 + t.n, 'Regular', + $1::timestamp - interval '1 hour', $1::timestamp, t.module, '0x' || lpad(to_hex(900000 + t.n), 16, '0'), + t.executions, t.avg_us, t.max_us, t.avg_us, t.max_us, '0x' || lpad(to_hex(900000 + t.n), 16, '0'), 900000 + t.n, $1::timestamp - interval '1 hour' +FROM (VALUES + (1, 'tb_big1', 500, 900000, 950000), (2, 'tb_big2', 400, 800000, 850000), + (3, 'tb_tie0', 100, 1000, 2000), (4, 'tb_tie1', 100, 1000, 2000), (5, 'tb_tie2', 100, 1000, 2000), + (6, 'tb_tie3', 100, 1000, 2000), (7, 'tb_tie4', 100, 1000, 2000), (8, 'tb_tie5', 100, 1000, 2000), + (9, 'tb_small1', 1, 10, 20), (10, 'tb_small2', 1, 20, 30) +) AS t(n, module, executions, avg_us, max_us);", connection); + ties.Parameters.AddWithValue(DateTime.SpecifyKind(WindowStart, DateTimeKind.Unspecified)); + ties.Parameters.AddWithValue(Servers[0].Id); + await ties.ExecuteNonQueryAsync(ct); + + await using var analyze = new NpgsqlCommand("ANALYZE collect.query_store_interval_wide; ANALYZE collect.servers;", connection); + await analyze.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/QueryStoreReadsDatabaseFilterLiveTests.cs b/Darling/Darling.Tests/QueryStoreReadsDatabaseFilterLiveTests.cs new file mode 100644 index 0000000000..59a6775a17 --- /dev/null +++ b/Darling/Darling.Tests/QueryStoreReadsDatabaseFilterLiveTests.cs @@ -0,0 +1,302 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5245 (part of #5244), PR1: the database filter on the readers behind get_query_store_regressions and +/// get_query_heatmap. Each tool's internal overload takes a , so an overload call with +/// [A, B] over a seed of A, B and C must return only A's and B's rows, one name must read exactly as the public +/// (one-name) method does, and every one-name consumer on the empty answers must be list-aware: the echoed +/// database_name field, the three filtered empty answers of the regressions read, and the covered-claim sentence +/// of the heatmap's idle-grid answer. Both reads are one tier (query_store_stats and query_stats). +/// The public methods still pass one name, so no MCP schema, dispatch or tools/list byte changes. +/// +[Collection("live-postgres")] +public sealed class QueryStoreReadsDatabaseFilterLiveTests +{ + private const int ServerId = -950456; + private const string ServerName = "qs-reads-database-filter"; + + private const string A = "QsFilterA"; + private const string B = "QsFilterB"; + private const string C = "QsFilterC"; + // Both sides collected, nothing regressed. + private const string QuietH = "QsFilterQuietH"; + private const string QuietI = "QsFilterQuietI"; + // Baseline captures only. + private const string BaseOnlyF = "QsFilterBaseOnlyF"; + private const string BaseOnlyG = "QsFilterBaseOnlyG"; + // Recent captures only. + private const string RecentOnlyD = "QsFilterRecentOnlyD"; + private const string RecentOnlyE = "QsFilterRecentOnlyE"; + // Heatmap: databases whose every capture ran nothing. + private const string IdleX = "QsFilterIdleX"; + private const string IdleY = "QsFilterIdleY"; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static JsonElement Root(string json) => JsonDocument.Parse(json).RootElement; + + private static DateTime FloorToHour(DateTime value) => + new(value.Ticks - (value.Ticks % TimeSpan.TicksPerHour), value.Kind); + + private static DatabaseFilter Of(params string[] names) => DatabaseFilter.Of(names); + + private static List RegressionDatabases(JsonElement root) => + root.GetProperty("regressions").EnumerateArray().Select(r => r.GetProperty("database_name").GetString()!).OrderBy(x => x, StringComparer.Ordinal).ToList(); + + private static List HeatmapHashes(JsonElement root) => + root.GetProperty("cells").EnumerateArray().Select(c => c.GetProperty("top_query_hash").GetString()!).OrderBy(x => x, StringComparer.Ordinal).ToList(); + + private static Task Regressions(NpgsqlDataSource postgres, DatabaseFilter filter) => + DarlingMcpQueryStoreRegressionTools.GetQueryStoreRegressions( + postgres, ServerName, 24, filter, 30, false, null, null, CancellationToken.None); + + private static Task Heatmap(NpgsqlDataSource postgres, DatabaseFilter filter) => + DarlingMcpQueryHeatmapTools.GetQueryHeatmap( + postgres, ServerName, 24, null, filter, 5, 100, null, false, null, CancellationToken.None); + + [Fact] + public async Task TheRegressionsOverload_WithAListOfDatabases_ReturnsOnlyThoseDatabases_AndEveryOneNameConsumerIsListAware_AgainstDevPostgres() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), + "Set DARLING_TEST_PG to a Postgres connection string to run the live database filter test."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + + await using var postgres = NpgsqlDataSource.Create(cs!); + var bodySucceeded = false; + + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + var baseNow = DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow); + + // A, B, C each carry one query that got 4x slower in the window. + long queryId = 10; + foreach (var db in new[] { A, B, C }) + { + queryId++; + await SeedQsAsync(connection, ct, db, baseNow.AddHours(-40), 100, 1000, queryId, intervalId: 2); + await SeedQsAsync(connection, ct, db, baseNow.AddMinutes(-30), 200, 4000, queryId, intervalId: 1); + } + + // H and I: both sides collected, flat. + foreach (var db in new[] { QuietH, QuietI }) + { + await SeedQsAsync(connection, ct, db, baseNow.AddHours(-40), 100, 1000, 21, intervalId: 2); + await SeedQsAsync(connection, ct, db, baseNow.AddMinutes(-30), 100, 1000, 21, intervalId: 1); + } + + foreach (var db in new[] { BaseOnlyF, BaseOnlyG }) + await SeedQsAsync(connection, ct, db, baseNow.AddHours(-40), 100, 1000, 31, intervalId: 2); + + foreach (var db in new[] { RecentOnlyD, RecentOnlyE }) + await SeedQsAsync(connection, ct, db, baseNow.AddMinutes(-30), 100, 1000, 41, intervalId: 1); + + // The rows: [A, B] over A, B and C is A and B only; one name is that one database; no filter is all three. + var all = Root(await Regressions(postgres, DatabaseFilter.All)); + Assert.Equal([A, B, C], RegressionDatabases(all)); + Assert.Equal(JsonValueKind.Null, all.GetProperty("database_name").ValueKind); + + var two = Root(await Regressions(postgres, Of(A, B))); + Assert.Equal([A, B], RegressionDatabases(two)); + Assert.Equal(DatabaseFilter.ManyDatabasesDescription, two.GetProperty("database_name").GetString()); + + var reordered = Root(await Regressions(postgres, Of(C, A))); + Assert.Equal([A, C], RegressionDatabases(reordered)); + + // One name is what the public method returns for that name, byte for byte. + var oneOverload = await Regressions(postgres, DatabaseFilter.One(B)); + var onePublic = await DarlingMcpQueryStoreRegressionTools.GetQueryStoreRegressions(postgres, ServerName, 24, B); + Assert.Equal(StripWindowEcho(onePublic), StripWindowEcho(oneOverload)); + Assert.Equal([B], RegressionDatabases(Root(oneOverload))); + Assert.Equal(B, Root(oneOverload).GetProperty("database_name").GetString()); + + // The empty answers (EmptyAsync): each names the chosen databases, in the plural, and keeps its status. + var neither = Root(await Regressions(postgres, Of("QsNoSuchOne", "QsNoSuchTwo"))); + Assert.Equal("empty", neither.GetProperty("status").GetString()); + var neitherText = neither.GetProperty("message").GetString()!; + Assert.Contains("matched the chosen databases, so the filter matched nothing and this is NOT the all-clear: no query in those databases was compared. Check the database names,", neitherText, StringComparison.Ordinal); + + var noBaseline = Root(await Regressions(postgres, Of(RecentOnlyD, RecentOnlyE))); + Assert.Equal("unavailable", noBaseline.GetProperty("status").GetString()); + var noBaselineText = noBaseline.GetProperty("message").GetString()!; + Assert.Contains("no Query Store capture of the chosen databases in the 7-day baseline window", noBaselineText, StringComparison.Ordinal); + Assert.Contains("no baseline for those databases to compare against and no regression of their queries", noBaselineText, StringComparison.Ordinal); + Assert.Contains("Either those databases' whole collected history falls inside the last 24 hour(s), or they have none older", noBaselineText, StringComparison.Ordinal); + + var noRecent = Root(await Regressions(postgres, Of(BaseOnlyF, BaseOnlyG))); + Assert.Equal("empty", noRecent.GetProperty("status").GetString()); + var noRecentText = noRecent.GetProperty("message").GetString()!; + Assert.Contains("has Query Store history of the chosen databases from before this window but nothing of them collected IN the last 24 hour(s), so those databases' recent side is missing", noRecentText, StringComparison.Ordinal); + + // Both sides collected for every chosen database: the filtered answers fall through to the all-clear's own text. + var quiet = Root(await Regressions(postgres, Of(QuietH, QuietI))); + Assert.Equal("empty", quiet.GetProperty("status").GetString()); + Assert.DoesNotContain("NOT the all-clear", quiet.GetProperty("message").GetString()!, StringComparison.Ordinal); + + // One name: the three filtered answers read exactly as they did (the public method, which passes one name). + var oneNeither = Root(await DarlingMcpQueryStoreRegressionTools.GetQueryStoreRegressions(postgres, ServerName, 24, "QsNoSuchOne")); + Assert.Contains("matched database_name 'QsNoSuchOne', so the filter matched nothing and this is NOT the all-clear: no query in that database was compared. Check the database name, or drop", oneNeither.GetProperty("message").GetString()!, StringComparison.Ordinal); + var oneNoBaseline = Root(await DarlingMcpQueryStoreRegressionTools.GetQueryStoreRegressions(postgres, ServerName, 24, RecentOnlyD)); + Assert.Equal("unavailable", oneNoBaseline.GetProperty("status").GetString()); + Assert.Contains($"no Query Store capture of database_name '{RecentOnlyD}' in the 7-day baseline window before this window, so there is no baseline for that database to compare against and no regression of its queries", oneNoBaseline.GetProperty("message").GetString()!, StringComparison.Ordinal); + Assert.Contains("Either that database's whole collected history falls inside the last 24 hour(s), or it has none older", oneNoBaseline.GetProperty("message").GetString()!, StringComparison.Ordinal); + var oneNoRecent = Root(await DarlingMcpQueryStoreRegressionTools.GetQueryStoreRegressions(postgres, ServerName, 24, BaseOnlyF)); + Assert.Contains($"has Query Store history of database_name '{BaseOnlyF}' from before this window but nothing of it collected IN the last 24 hour(s), so that database's recent side is missing", oneNoRecent.GetProperty("message").GetString()!, StringComparison.Ordinal); + // ... and the overload with one name says the same thing. + Assert.Equal(oneNoRecent.GetProperty("message").GetString(), Root(await Regressions(postgres, DatabaseFilter.One(BaseOnlyF))).GetProperty("message").GetString()); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + [Fact] + public async Task TheHeatmapOverload_WithAListOfDatabases_ReturnsOnlyThoseDatabases_AndEveryOneNameConsumerIsListAware_AgainstDevPostgres() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), + "Set DARLING_TEST_PG to a Postgres connection string to run the live database filter test."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + + await using var postgres = NpgsqlDataSource.Create(cs!); + var bodySucceeded = false; + + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + var t1 = FloorToHour(DateTime.UtcNow.AddHours(-3)); + + // One hot query per database, each in its own 5-minute bin so they cannot share a cell. + await SeedQueryStatsAsync(connection, ct, A, t1, "0xHOTA", 5, 250_000); + await SeedQueryStatsAsync(connection, ct, B, t1.AddMinutes(10), "0xHOTB", 5, 250_000); + await SeedQueryStatsAsync(connection, ct, C, t1.AddMinutes(20), "0xHOTC", 5, 250_000); + + // Idle databases: captured, zero executions. + await SeedQueryStatsAsync(connection, ct, IdleX, t1.AddMinutes(30), "0xIDLEX", 0, 999_000); + await SeedQueryStatsAsync(connection, ct, IdleY, t1.AddMinutes(30), "0xIDLEY", 0, 999_000); + + var all = Root(await Heatmap(postgres, DatabaseFilter.All)); + Assert.Equal(["0xHOTA", "0xHOTB", "0xHOTC"], HeatmapHashes(all)); + Assert.Equal(JsonValueKind.Null, all.GetProperty("database_name").ValueKind); + + var two = Root(await Heatmap(postgres, Of(A, B))); + Assert.Equal(["0xHOTA", "0xHOTB"], HeatmapHashes(two)); + Assert.Equal(DatabaseFilter.ManyDatabasesDescription, two.GetProperty("database_name").GetString()); + + var oneOverload = await Heatmap(postgres, DatabaseFilter.One(C)); + var onePublic = await DarlingMcpQueryHeatmapTools.GetQueryHeatmap(postgres, ServerName, 24, database_name: C); + Assert.Equal(StripWindowEcho(onePublic), StripWindowEcho(oneOverload)); + Assert.Equal(["0xHOTC"], HeatmapHashes(Root(oneOverload))); + Assert.Equal(C, Root(oneOverload).GetProperty("database_name").GetString()); + + // The idle-grid answer: the covered claim names the filter, and the window floor does not speak for it. + var twoIdle = Root(await Heatmap(postgres, Of(IdleX, IdleY))); + Assert.Equal("empty", twoIdle.GetProperty("status").GetString()); + var twoIdleText = twoIdle.GetProperty("message").GetString()!; + Assert.Contains("and so does a filter on the chosen databases matching nothing collected.", twoIdleText, StringComparison.Ordinal); + Assert.DoesNotContain("database_name filter", twoIdleText, StringComparison.Ordinal); + + var oneIdleOverload = Root(await Heatmap(postgres, DatabaseFilter.One(IdleX))); + var oneIdlePublic = Root(await DarlingMcpQueryHeatmapTools.GetQueryHeatmap(postgres, ServerName, 24, database_name: IdleX)); + Assert.Contains("and so does a database_name filter matching nothing collected.", oneIdlePublic.GetProperty("message").GetString()!, StringComparison.Ordinal); + Assert.Equal(oneIdlePublic.GetProperty("message").GetString(), oneIdleOverload.GetProperty("message").GetString()); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// One name is the same JSON whichever method built it, except for the window instants, which each call + /// takes from its own clock. Those fields are dropped from both before they are compared. + private static string StripWindowEcho(string json) + { + using var doc = JsonDocument.Parse(json); + return RemoveVolatile(doc.RootElement); + } + + private static string RemoveVolatile(JsonElement root) + { + var parts = new List(); + foreach (var property in root.EnumerateObject()) + { + if (property.Name is "recent_window_start" or "recent_window_end" or "baseline_start" or "baseline_end" or "window_start" or "window_end") + continue; + parts.Add($"\"{property.Name}\":{property.Value.GetRawText()}"); + } + + return "{" + string.Join(",", parts) + "}"; + } + + private static async Task SeedQsAsync( + NpgsqlConnection connection, CancellationToken ct, string database, DateTime collectionTime, long executions, + long avgUs, long queryId, long intervalId) => + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO query_store_stats + (collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, + execution_type_desc, execution_count, avg_duration_us, avg_cpu_time_us, avg_logical_io_reads, + runtime_stats_interval_id, query_text, last_execution_time) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(collectionTime), ServerId, ServerName, + database, queryId, 9L, "Regular", executions, avgUs, avgUs, 100L, intervalId, + "SELECT * FROM dbo.Widgets", DarlingMcpTestData.Naive(collectionTime)); + + private static async Task SeedQueryStatsAsync( + NpgsqlConnection connection, CancellationToken ct, string database, DateTime collectionTime, string queryHash, + long deltaExec, long deltaElapsed) => + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO query_stats + (collection_id, collection_time, server_id, server_name, database_name, query_hash, + sample_interval_seconds, delta_execution_count, delta_worker_time, delta_elapsed_time, + delta_logical_reads, delta_logical_writes, query_text) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(collectionTime), ServerId, ServerName, + database, queryHash, 60, deltaExec, 0L, deltaElapsed, 0L, 0L, "SELECT * FROM dbo.Widgets"); + + private static async Task DeleteRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM query_store_stats WHERE server_id = $1", ServerId); + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM query_stats WHERE server_id = $1", ServerId); + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM servers WHERE server_id = $1", ServerId); + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM config_monitored_servers WHERE server_id = $1", ServerId); + } +} diff --git a/Darling/Darling.Tests/QueryStoreSliceTieBreakSourceTests.cs b/Darling/Darling.Tests/QueryStoreSliceTieBreakSourceTests.cs index 2abf21a9c3..4b83c1cb01 100644 --- a/Darling/Darling.Tests/QueryStoreSliceTieBreakSourceTests.cs +++ b/Darling/Darling.Tests/QueryStoreSliceTieBreakSourceTests.cs @@ -130,6 +130,20 @@ public void TheTieBreakNeverBecomesThePrimarySort() } } + [Fact] + public void ComposeCompiler_DedupePartitionByLiteral_EqualsTheRestrictionKey() + { + /* The dedupe spells its PARTITION BY list as a literal (this guard's regex has to see it); the + restriction joins on PerformanceMonitor.Darling.Service.ComposeCompiler.s_queryStorePartitionKey. + A drift between the two would make the restriction keep the wrong partitions. */ + var source = File.ReadAllText(SourcePath(Path.Combine("Darling", "PerformanceMonitor.Darling.Service", "Compose", "ComposeCompiler.cs"))); + var m = Regex.Match(source, @"ROW_NUMBER\(\) OVER \(PARTITION BY ""\s*\+\s*\$""(?[^""]+?)\s*""\s*\+\s*\$""ORDER BY"); + Assert.True(m.Success, "the Query Store dedupe's literal PARTITION BY list was not found in ComposeCompiler.cs"); + Assert.Equal( + string.Join(", ", PerformanceMonitor.Darling.Service.ComposeCompiler.s_queryStorePartitionKey), + m.Groups["cols"].Value); + } + /// Walks up from the test binary to the repo root so the pin works from any run directory. private static string SourcePath(string relative) { diff --git a/Darling/Darling.Tests/QueryStoreStatePruneTests.cs b/Darling/Darling.Tests/QueryStoreStatePruneTests.cs index c0cf12f65c..4ae16b6ec2 100644 --- a/Darling/Darling.Tests/QueryStoreStatePruneTests.cs +++ b/Darling/Darling.Tests/QueryStoreStatePruneTests.cs @@ -40,6 +40,7 @@ namespace Darling.Tests; /// itself lives in the shared so a prefix cannot end up pruned on /// one SKU and orphaning on the other. /// +[Trait("Reads", "Lite")] public sealed class QueryStoreStatePruneTests { private static string Qsowm(string database) => QueryStoreOpenIntervalState.WatermarkKeyPrefix + database; diff --git a/Darling/Darling.Tests/QueryStoreTopDailyBuilderLiveTests.cs b/Darling/Darling.Tests/QueryStoreTopDailyBuilderLiveTests.cs new file mode 100644 index 0000000000..f43eca5795 --- /dev/null +++ b/Darling/Darling.Tests/QueryStoreTopDailyBuilderLiveTests.cs @@ -0,0 +1,758 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// Pins the builder of the daily Query Store summary (#5094), . The summary is +/// approximate by design: a day is summarized after it ends, and a row that changes afterwards is missed or, for about a +/// day, counted twice. These facts pin what the builder does exactly (a built day equals a direct GROUP BY of the wide +/// table, to the second), when it builds (two passes, the oldest days first, a cap per tick), and the one residual the +/// second pass clears. Every seed is at a fixed past date and every call passes nowUtc explicitly, so no fact +/// reads the clock. Live facts run when DARLING_TEST_PG is set, each on its own scratch store. +/// +[Collection("live-postgres")] +public sealed class QueryStoreTopDailyBuilderLiveTests +{ + private const string SkipText = "Set DARLING_TEST_PG to a Postgres connection string to run the daily summary builder's live pins (each mints its own scratch database)."; + + private const int RetentionDays = 9; + + private static readonly DateTime Day = new(2026, 1, 10, 0, 0, 0, DateTimeKind.Unspecified); + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static async Task MigratedAsync(ScratchPostgres scratch, CancellationToken ct) + { + var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + return connection; + } + + private static async Task ScalarAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + return await command.ExecuteScalarAsync(ct); + } + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(ct); + } + + private static string At(DateTime value) => value.ToString("yyyy-MM-dd HH:mm:ss.ffffff", System.Globalization.CultureInfo.InvariantCulture); + + /// Enrolls the server (enabled unless told otherwise) and gives it a wide-table coverage claim. + private static async Task CoverAsync( + NpgsqlConnection connection, int serverId, CancellationToken ct, string filledSince = "2025-12-01 00:00:00", bool enabled = true) + { + await ExecAsync(connection, + $"INSERT INTO collect.servers (server_id, server_name, is_enabled) VALUES ({serverId}, 'srv{serverId}', {(enabled ? "TRUE" : "FALSE")}) ON CONFLICT (server_id) DO NOTHING", ct); + await ExecAsync(connection, + $"INSERT INTO collect.query_store_interval_wide_coverage (server_id, filled_since, applied_through) VALUES ({serverId}, TIMESTAMP '{filledSince}', TIMESTAMP '2026-03-01 00:00:00')", ct); + } + + private static int s_interval; + + /// One wide-table row. Unset average columns stay NULL; the interval began ten minutes before it was collected. + private static async Task SeedAsync( + NpgsqlConnection connection, int serverId, DateTime collectionTime, long queryId, CancellationToken ct, + string? moduleName = null, string? replicaRole = null, long? executionCount = null, long? duration = null, + long? cpu = null, long? reads = null, long? writes = null, long? physical = null, long? rowcount = null, + DateTime? lastExecution = null, string? planHash = null, DateTime? firstExecution = null) + { + await using var command = new NpgsqlCommand(@" +INSERT INTO collect.query_store_interval_wide +(collection_time, server_id, database_name, query_id, plan_id, execution_type_desc, first_execution_time, last_execution_time, + module_name, query_hash, execution_count, avg_duration_us, avg_cpu_time_us, avg_logical_io_reads, avg_logical_io_writes, + avg_physical_io_reads, avg_rowcount, query_plan_hash, replica_role, runtime_stats_interval_id, interval_start_time_utc) +VALUES +(@ct, @server, 'db1', @query, @query, 'Regular', @first, @last, + @module, 'h' || @query, @ec, @dur, @cpu, @reads, @writes, + @phys, @rows, @ph, @replica, @interval, @ct - interval '10 minutes')", connection); + command.Parameters.Add(new NpgsqlParameter("ct", NpgsqlDbType.Timestamp) { Value = collectionTime }); + command.Parameters.Add(new NpgsqlParameter("server", NpgsqlDbType.Integer) { Value = serverId }); + command.Parameters.Add(new NpgsqlParameter("query", NpgsqlDbType.Bigint) { Value = queryId }); + command.Parameters.Add(new NpgsqlParameter("first", NpgsqlDbType.Timestamp) { Value = firstExecution ?? collectionTime.AddMinutes(-10) }); + command.Parameters.Add(new NpgsqlParameter("last", NpgsqlDbType.Timestamp) { Value = (object?)lastExecution ?? collectionTime }); + command.Parameters.Add(new NpgsqlParameter("module", NpgsqlDbType.Text) { Value = (object?)moduleName ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter("ec", NpgsqlDbType.Bigint) { Value = (object?)executionCount ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter("dur", NpgsqlDbType.Bigint) { Value = (object?)duration ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter("cpu", NpgsqlDbType.Bigint) { Value = (object?)cpu ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter("reads", NpgsqlDbType.Bigint) { Value = (object?)reads ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter("writes", NpgsqlDbType.Bigint) { Value = (object?)writes ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter("phys", NpgsqlDbType.Bigint) { Value = (object?)physical ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter("rows", NpgsqlDbType.Bigint) { Value = (object?)rowcount ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter("ph", NpgsqlDbType.Text) { Value = (object?)planHash ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter("replica", NpgsqlDbType.Text) { Value = (object?)replicaRole ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter("interval", NpgsqlDbType.Bigint) { Value = (long)Interlocked.Increment(ref s_interval) }); + await command.ExecuteNonQueryAsync(ct); + } + + private static NpgsqlDataSource DataSource(ScratchPostgres scratch) => NpgsqlDataSource.Create(scratch.ConnectionString); + + private static async Task TickBuiltAsync(NpgsqlDataSource source, DateTime nowUtc) + { + var result = await QueryStoreTopDaily.RunTickAsync(source, nowUtc, RetentionDays, NullLogger.Instance, TestContext.Current.CancellationToken); + Assert.Equal(0, result.Failed); + return result.Built; + } + + private static async Task<(short Pass, DateTime BuiltAt, long SourceRows)?> BuiltRowAsync(NpgsqlConnection connection, int serverId, DateTime day, CancellationToken ct) + { + await using var command = new NpgsqlCommand( + $"SELECT pass, built_at, source_rows FROM collect.query_store_top_daily_built WHERE server_id = {serverId} AND day = DATE '{At(day)[..10]}'", connection); + await using var reader = await command.ExecuteReaderAsync(ct); + return await reader.ReadAsync(ct) ? (reader.GetInt16(0), reader.GetDateTime(1), reader.GetInt64(2)) : null; + } + + private static async Task SummarizedRowsAsync(NpgsqlConnection connection, int serverId, DateTime day, CancellationToken ct) => + Convert.ToInt64(await ScalarAsync(connection, + $"SELECT COALESCE(sum(interval_rows), 0) FROM collect.query_store_top_daily WHERE server_id = {serverId} AND day = DATE '{At(day)[..10]}'", ct)); + + private static async Task RunLiveAsync(Func body) + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + var bodySucceeded = false; + try + { + await body(scratch, connection, ct); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + [Fact] + public async Task ABuiltDay_EqualsADirectGroupByOfTheWideRows_EveryColumn_NullsIncluded() + { + await RunLiveAsync(async (scratch, connection, ct) => + { + /* Group 1: three intervals, one with every average NULL. Group 2: every average NULL on both rows (so + _n = 0 and _sum is NULL). Group 3: a module and a replica role. Group 4: another server, same day. */ + await SeedAsync(connection, 1, Day.AddHours(3), 1, ct, executionCount: 5, duration: 100, cpu: 10, reads: 1000, writes: 7, physical: 2, rowcount: 3, planHash: "a"); + await SeedAsync(connection, 1, Day.AddHours(4), 1, ct, executionCount: 7, duration: 200, cpu: 30, reads: 3000, writes: 9, physical: 4, rowcount: 5, planHash: "b", lastExecution: Day.AddHours(9)); + await SeedAsync(connection, 1, Day.AddHours(5), 1, ct); + await SeedAsync(connection, 1, Day.AddHours(6), 2, ct, executionCount: 11); + await SeedAsync(connection, 1, Day.AddHours(7), 2, ct); + await SeedAsync(connection, 1, Day.AddHours(8), 3, ct, moduleName: "dbo.p", replicaRole: "PRIMARY", executionCount: 2, duration: 50, cpu: 5); + await SeedAsync(connection, 2, Day.AddHours(8), 1, ct, executionCount: 99, duration: 1); + await CoverAsync(connection, 1, ct); + await CoverAsync(connection, 2, ct); + + var sourceRows = await QueryStoreTopDaily.BuildDayAsync(connection, 1, DateOnly.FromDateTime(Day), 1, Day.AddDays(1).AddHours(2), ct); + Assert.Equal(6L, sourceRows); + + const string Direct = @" +SELECT server_id, DATE '2026-01-10' AS day, database_name, query_id, plan_id, query_hash, execution_type_desc, replica_role, module_name, + count(*) AS interval_rows, sum(execution_count) AS execution_count_sum, + sum(avg_duration_us::numeric) AS avg_duration_us_sum, count(avg_duration_us) AS avg_duration_us_n, + sum(avg_cpu_time_us::numeric) AS avg_cpu_time_us_sum, count(avg_cpu_time_us) AS avg_cpu_time_us_n, + sum(avg_logical_io_reads::numeric) AS avg_logical_io_reads_sum, count(avg_logical_io_reads) AS avg_logical_io_reads_n, + sum(avg_logical_io_writes::numeric) AS avg_logical_io_writes_sum, count(avg_logical_io_writes) AS avg_logical_io_writes_n, + sum(avg_physical_io_reads::numeric) AS avg_physical_io_reads_sum, count(avg_physical_io_reads) AS avg_physical_io_reads_n, + sum(avg_rowcount::numeric) AS avg_rowcount_sum, count(avg_rowcount) AS avg_rowcount_n, + max(last_execution_time) AS last_execution_time_max, max(query_plan_hash) AS query_plan_hash_max, min(first_execution_time) AS first_execution_time_min +FROM collect.query_store_interval_wide +WHERE server_id = 1 AND collection_time >= TIMESTAMP '2026-01-10' AND collection_time < TIMESTAMP '2026-01-11' +GROUP BY server_id, database_name, query_id, plan_id, query_hash, execution_type_desc, replica_role, module_name"; + const string Stored = "SELECT * FROM collect.query_store_top_daily WHERE server_id = 1 AND day = DATE '2026-01-10'"; + + Assert.Equal(3L, await ScalarAsync(connection, $"SELECT count(*) FROM ({Stored}) s", ct)); + Assert.Equal(0L, await ScalarAsync(connection, $"SELECT count(*) FROM (({Stored}) EXCEPT ({Direct})) x", ct)); + Assert.Equal(0L, await ScalarAsync(connection, $"SELECT count(*) FROM (({Direct}) EXCEPT ({Stored})) x", ct)); + + /* The all-NULL group keeps its row: _n = 0, _sum NULL; and the key's NULLs stay NULL. */ + Assert.Equal(1L, await ScalarAsync(connection, + "SELECT count(*) FROM collect.query_store_top_daily WHERE server_id = 1 AND query_id = 2 AND avg_duration_us_n = 0 AND avg_duration_us_sum IS NULL AND avg_rowcount_n = 0 AND replica_role IS NULL AND module_name IS NULL AND execution_count_sum = 11 AND interval_rows = 2", ct)); + Assert.Equal(1L, await ScalarAsync(connection, + "SELECT count(*) FROM collect.query_store_top_daily WHERE server_id = 1 AND query_id = 1 AND avg_duration_us_sum = 300 AND avg_duration_us_n = 2 AND interval_rows = 3 AND execution_count_sum = 12 AND query_plan_hash_max = 'b'", ct)); + + Assert.Equal((short)1, (await BuiltRowAsync(connection, 1, Day, ct))!.Value.Pass); + Assert.Equal(6L, (await BuiltRowAsync(connection, 1, Day, ct))!.Value.SourceRows); + Assert.Null(await BuiltRowAsync(connection, 2, Day, ct)); + Assert.Equal(0L, await SummarizedRowsAsync(connection, 2, Day, ct)); + }); + } + + [Fact] + public async Task TheDayIsHalfOpen_RowsAtMidnightStartTheDay_AndNextMidnightIsNotIn() + { + await RunLiveAsync(async (scratch, connection, ct) => + { + await SeedAsync(connection, 1, Day.AddTicks(-10), 1, ct, executionCount: 1); + await SeedAsync(connection, 1, Day, 2, ct, executionCount: 1); + await SeedAsync(connection, 1, Day.AddDays(1).AddTicks(-10), 3, ct, executionCount: 1); + await SeedAsync(connection, 1, Day.AddDays(1), 4, ct, executionCount: 1); + await CoverAsync(connection, 1, ct); + + Assert.Equal(2L, await QueryStoreTopDaily.BuildDayAsync(connection, 1, DateOnly.FromDateTime(Day), 1, Day.AddDays(1).AddHours(2), ct)); + Assert.Equal(new long[] { 2, 3 }, + (await ScalarAsync(connection, "SELECT array_agg(query_id ORDER BY query_id) FROM collect.query_store_top_daily WHERE server_id = 1 AND day = DATE '2026-01-10'", ct) as long[])!); + }); + } + + [Fact] + public async Task TheSchedule_BuildsAtDayEndPlusTwoHours_RebuildsAtPlusTwentyFive_AndThenStops() + { + await RunLiveAsync(async (scratch, connection, ct) => + { + await SeedAsync(connection, 1, Day.AddHours(5), 1, ct, executionCount: 1, duration: 10); + await CoverAsync(connection, 1, ct); + await using var source = DataSource(scratch); + var dayEnd = Day.AddDays(1); + + await TickBuiltAsync(source, dayEnd.AddHours(1).AddMinutes(59)); + Assert.Null(await BuiltRowAsync(connection, 1, Day, ct)); + + var pass1At = dayEnd.AddHours(2); + await TickBuiltAsync(source, pass1At); + var built = await BuiltRowAsync(connection, 1, Day, ct); + Assert.Equal((short)1, built!.Value.Pass); + Assert.Equal(pass1At, built.Value.BuiltAt); + Assert.Equal(1L, built.Value.SourceRows); + + await TickBuiltAsync(source, dayEnd.AddHours(24).AddMinutes(59)); + built = await BuiltRowAsync(connection, 1, Day, ct); + Assert.Equal((short)1, built!.Value.Pass); + Assert.Equal(pass1At, built.Value.BuiltAt); + + var pass2At = dayEnd.AddHours(25); + await TickBuiltAsync(source, pass2At); + built = await BuiltRowAsync(connection, 1, Day, ct); + Assert.Equal((short)2, built!.Value.Pass); + Assert.Equal(pass2At, built.Value.BuiltAt); + + await TickBuiltAsync(source, dayEnd.AddDays(3)); + built = await BuiltRowAsync(connection, 1, Day, ct); + Assert.Equal((short)2, built!.Value.Pass); + Assert.Equal(pass2At, built.Value.BuiltAt); + + /* Nothing is planned for the day once it is final. */ + await using var planConnection = await source.OpenConnectionAsync(ct); + var plan = await QueryStoreTopDaily.PlanBuildsAsync(planConnection, dayEnd.AddDays(3), RetentionDays, ct); + Assert.DoesNotContain(plan, b => b.Day == DateOnly.FromDateTime(Day)); + }); + } + + [Fact] + public async Task TheDocumentedResidual_ARowMovedIntoTheNextDay_IsCountedTwiceUntilThePassTwoRebuild() + { + await RunLiveAsync(async (scratch, connection, ct) => + { + await SeedAsync(connection, 1, Day.AddHours(10), 1, ct, executionCount: 1); + await SeedAsync(connection, 1, Day.AddHours(23).AddMinutes(30), 2, ct, executionCount: 1); + await CoverAsync(connection, 1, ct); + await using var source = DataSource(scratch); + var dayEnd = Day.AddDays(1); + + await TickBuiltAsync(source, dayEnd.AddHours(2)); + Assert.Equal(2L, await SummarizedRowsAsync(connection, 1, Day, ct)); + + /* The interval resumed: its row now belongs to the next day. */ + await ExecAsync(connection, "UPDATE collect.query_store_interval_wide SET collection_time = TIMESTAMP '2026-01-11 00:30:00' WHERE query_id = 2", ct); + Assert.Equal(2L, await SummarizedRowsAsync(connection, 1, Day, ct)); + + await TickBuiltAsync(source, dayEnd.AddHours(25)); + Assert.Equal(1L, await SummarizedRowsAsync(connection, 1, Day, ct)); + Assert.Equal(1L, (await BuiltRowAsync(connection, 1, Day, ct))!.Value.SourceRows); + }); + } + + [Fact] + public async Task AnEmptyDay_GetsABuiltRowWithZeroSourceRows_AndNoSummaryRows() + { + await RunLiveAsync(async (scratch, connection, ct) => + { + await CoverAsync(connection, 1, ct); + await using var source = DataSource(scratch); + + await TickBuiltAsync(source, Day.AddDays(1).AddHours(2)); + + var built = await BuiltRowAsync(connection, 1, Day, ct); + Assert.Equal((short)1, built!.Value.Pass); + Assert.Equal(0L, built.Value.SourceRows); + Assert.Equal(0L, Convert.ToInt64(await ScalarAsync(connection, "SELECT count(*) FROM collect.query_store_top_daily", ct))); + }); + } + + [Fact] + public async Task AServerWithoutACoverageClaim_IsNotBuilt() + { + await RunLiveAsync(async (scratch, connection, ct) => + { + await SeedAsync(connection, 1, Day.AddHours(5), 1, ct, executionCount: 1); + await using var source = DataSource(scratch); + + Assert.Equal(0L, await TickBuiltAsync(source, Day.AddDays(1).AddHours(2))); + Assert.Equal(0L, Convert.ToInt64(await ScalarAsync(connection, "SELECT count(*) FROM collect.query_store_top_daily_built", ct))); + }); + } + + [Fact] + public async Task Gc_RemovesDaysBelowTheCutoff_AndOnlyThose() + { + await RunLiveAsync(async (scratch, connection, ct) => + { + /* now 2026-01-20 12:00, retention 9 days: the cutoff is (2026-01-11)::date - 1 = 2026-01-10. */ + await ExecAsync(connection, "INSERT INTO collect.servers (server_id, server_name) VALUES (1, 'srv1')", ct); + foreach (var d in new[] { 8, 9, 10, 11 }) + { + await ExecAsync(connection, + $"INSERT INTO collect.query_store_top_daily (server_id, day, interval_rows, avg_duration_us_n, avg_cpu_time_us_n, avg_logical_io_reads_n, avg_logical_io_writes_n, avg_physical_io_reads_n, avg_rowcount_n) VALUES (1, DATE '2026-01-{d:00}', 1, 0, 0, 0, 0, 0, 0)", ct); + await ExecAsync(connection, + $"INSERT INTO collect.query_store_top_daily_built (server_id, day, pass, built_at, source_rows) VALUES (1, DATE '2026-01-{d:00}', 2, TIMESTAMP '2026-01-15 00:00:00', 1)", ct); + } + + var removed = await QueryStoreTopDaily.GcAsync(connection, new DateTime(2026, 1, 20, 12, 0, 0, DateTimeKind.Unspecified), RetentionDays, ct); + + Assert.Equal(2L, removed); + Assert.Equal("2026-01-10,2026-01-11", await ScalarAsync(connection, "SELECT string_agg(day::text, ',' ORDER BY day) FROM collect.query_store_top_daily", ct)); + Assert.Equal("2026-01-10,2026-01-11", await ScalarAsync(connection, "SELECT string_agg(day::text, ',' ORDER BY day) FROM collect.query_store_top_daily_built", ct)); + }); + } + + [Fact] + public async Task Gc_RemovesADisabledServersDaysFromBothTables() + { + await RunLiveAsync(async (scratch, connection, ct) => + { + await ExecAsync(connection, "INSERT INTO collect.servers (server_id, server_name, is_enabled) VALUES (1, 'srv1', TRUE), (9, 'srv9', FALSE)", ct); + foreach (var (server, day) in new[] { (1, 19), (9, 19), (9, 18) }) + { + await InsertSummaryAndBuiltAsync(connection, server, day, ct); + } + + var removed = await QueryStoreTopDaily.GcAsync(connection, new DateTime(2026, 1, 20, 12, 0, 0, DateTimeKind.Unspecified), RetentionDays, ct); + + Assert.Equal(2L, removed); + Assert.Equal("1|2026-01-19", await ScalarAsync(connection, "SELECT string_agg(server_id || '|' || day, ';' ORDER BY server_id, day) FROM collect.query_store_top_daily", ct)); + Assert.Equal("1|2026-01-19", await ScalarAsync(connection, "SELECT string_agg(server_id || '|' || day, ';' ORDER BY server_id, day) FROM collect.query_store_top_daily_built", ct)); + }); + } + + /* #5507. The cleanup is one transaction: if the summary step fails, the built rows it already deleted come back, so + the two tables never disagree about which days exist. A trigger that refuses the summary delete forces the failure. */ + [Fact] + public async Task Gc_RollsTheBuiltRowsBack_WhenTheSummaryStepFails() + { + await RunLiveAsync(async (scratch, connection, ct) => + { + await ExecAsync(connection, "INSERT INTO collect.servers (server_id, server_name, is_enabled) VALUES (1, 'srv1', TRUE), (9, 'srv9', FALSE)", ct); + foreach (var (server, day) in new[] { (1, 19), (9, 19) }) + { + await InsertSummaryAndBuiltAsync(connection, server, day, ct); + } + + await ExecAsync(connection, @" +CREATE FUNCTION collect.refuse_summary_delete() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN RAISE EXCEPTION 'summary delete refused'; END $$; +CREATE TRIGGER refuse_summary_delete BEFORE DELETE ON collect.query_store_top_daily FOR EACH ROW EXECUTE FUNCTION collect.refuse_summary_delete();", ct); + + var failure = await Assert.ThrowsAsync(() => + QueryStoreTopDaily.GcAsync(connection, new DateTime(2026, 1, 20, 12, 0, 0, DateTimeKind.Unspecified), RetentionDays, ct)); + Assert.Contains("summary delete refused", failure.MessageText, StringComparison.Ordinal); + + Assert.Equal("1|2026-01-19;9|2026-01-19", await ScalarAsync(connection, "SELECT string_agg(server_id || '|' || day, ';' ORDER BY server_id, day) FROM collect.query_store_top_daily_built", ct)); + Assert.Equal("1|2026-01-19;9|2026-01-19", await ScalarAsync(connection, "SELECT string_agg(server_id || '|' || day, ';' ORDER BY server_id, day) FROM collect.query_store_top_daily", ct)); + }); + } + + private static async Task InsertSummaryAndBuiltAsync(NpgsqlConnection connection, int serverId, int day, CancellationToken ct) + { + await ExecAsync(connection, + $"INSERT INTO collect.query_store_top_daily (server_id, day, query_id, interval_rows, avg_duration_us_n, avg_cpu_time_us_n, avg_logical_io_reads_n, avg_logical_io_writes_n, avg_physical_io_reads_n, avg_rowcount_n) VALUES ({serverId}, DATE '2026-01-{day:00}', 1, 1, 0, 0, 0, 0, 0, 0)", ct); + await ExecAsync(connection, + $"INSERT INTO collect.query_store_top_daily_built (server_id, day, pass, built_at, source_rows) VALUES ({serverId}, DATE '2026-01-{day:00}', 2, TIMESTAMP '2026-01-15 00:00:00', 1)", ct); + } + + /* #5507. The pin that fails on the old single statement: a one-statement cleanup with its own predicate on the summary + reads all of it every tick. Here a disabled server's one day is due among 90,000 live summary rows. Two halves: + the EXPLAIN pins the shape of the statement GcAsync runs per due key (an index probe on both leading columns), and + the table's access counters around two GcAsync calls pin what GcAsync does (no sequential scan of the summary on + either call, at most the ten due rows fetched on the first, and no touch of the summary at all when nothing is due). */ + [Fact] + public async Task TheGc_ProbesTheSummaryByIndex_NeverScansIt_AndRunsNoSummaryDeleteWhenNothingIsDue() + { + await RunLiveAsync(async (scratch, connection, ct) => + { + await ExecAsync(connection, "INSERT INTO collect.servers (server_id, server_name, is_enabled) VALUES (1, 'srv1', TRUE), (9, 'srv9', FALSE)", ct); + await ExecAsync(connection, @" +INSERT INTO collect.query_store_top_daily_built (server_id, day, pass, built_at, source_rows) +SELECT 1, d::date, 2, TIMESTAMP '2026-01-15 00:00:00', 1 FROM generate_series(DATE '2026-01-16', DATE '2026-01-18', interval '1 day') AS d; +INSERT INTO collect.query_store_top_daily (server_id, day, query_id, interval_rows, avg_duration_us_n, avg_cpu_time_us_n, avg_logical_io_reads_n, avg_logical_io_writes_n, avg_physical_io_reads_n, avg_rowcount_n) +SELECT 1, d::date, q, 1, 0, 0, 0, 0, 0, 0 FROM generate_series(DATE '2026-01-16', DATE '2026-01-18', interval '1 day') AS d, generate_series(1, 30000) AS q; +INSERT INTO collect.query_store_top_daily_built (server_id, day, pass, built_at, source_rows) VALUES (9, DATE '2026-01-16', 2, TIMESTAMP '2026-01-15 00:00:00', 1); +INSERT INTO collect.query_store_top_daily (server_id, day, query_id, interval_rows, avg_duration_us_n, avg_cpu_time_us_n, avg_logical_io_reads_n, avg_logical_io_writes_n, avg_physical_io_reads_n, avg_rowcount_n) +SELECT 9, DATE '2026-01-16', q, 1, 0, 0, 0, 0, 0, 0 FROM generate_series(1, 10) AS q; +ANALYZE collect.query_store_top_daily; +ANALYZE collect.query_store_top_daily_built;", ct); + + /* The statement that removes one due key, run under EXPLAIN ANALYZE in a transaction that is rolled back. */ + await using (var transaction = await connection.BeginTransactionAsync(ct)) + { + string json; + await using (var command = new NpgsqlCommand("EXPLAIN (ANALYZE, FORMAT JSON) " + QueryStoreTopDaily.GcSummarySql, connection, transaction)) + { + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = 9 }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Date, Value = new DateOnly(2026, 1, 16) }); + json = (string)(await command.ExecuteScalarAsync(ct))!; + } + + await transaction.RollbackAsync(ct); + + var scans = new System.Collections.Generic.List<(string Node, string? Index, string? Cond, double Rows)>(); + void Walk(System.Text.Json.JsonElement node) + { + if (node.TryGetProperty("Relation Name", out var relation) && relation.GetString() == "query_store_top_daily" + && node.GetProperty("Node Type").GetString()!.Contains("Scan", StringComparison.Ordinal)) + { + scans.Add((node.GetProperty("Node Type").GetString()!, + node.TryGetProperty("Index Name", out var index) ? index.GetString() : null, + node.TryGetProperty("Index Cond", out var cond) ? cond.GetString() : null, + node.GetProperty("Actual Rows").GetDouble() * node.GetProperty("Actual Loops").GetDouble())); + } + + if (node.TryGetProperty("Plans", out var plans)) + { + foreach (var child in plans.EnumerateArray()) Walk(child); + } + + if (node.TryGetProperty("Plan", out var plan)) Walk(plan); + } + + using var document = System.Text.Json.JsonDocument.Parse(json); + Walk(document.RootElement[0]); + + Assert.NotEmpty(scans); + Assert.All(scans, scan => Assert.True(scan.Index == "ux_query_store_top_daily", $"{scan.Node} on {scan.Index ?? "no index"}")); + Assert.All(scans, scan => Assert.True(scan.Cond is not null && scan.Cond.Contains("server_id", StringComparison.Ordinal) + && scan.Cond.Contains("day", StringComparison.Ordinal), $"index condition: {scan.Cond ?? "none"}")); + Assert.True(scans.Sum(scan => scan.Rows) <= 10, $"the cleanup read {scans.Sum(scan => scan.Rows)} summary rows for ten due ones"); + } + + /* GcAsync ITSELF, observed through the table's own access counters: the EXPLAIN above runs the constant by name, + so on its own it would pass if GcAsync went back to one statement. pg_stat_force_next_flush() makes this + session's pending counts visible at the next statement. */ + async Task<(long SeqScan, long IdxScan, long IdxTupFetch)> CountersAsync() + { + await ExecAsync(connection, "SELECT pg_stat_force_next_flush()", ct); + await using var read = new NpgsqlCommand( + "SELECT seq_scan, COALESCE(idx_scan, 0), COALESCE(idx_tup_fetch, 0) FROM pg_stat_user_tables WHERE schemaname = 'collect' AND relname = 'query_store_top_daily'", connection); + await using var reader = await read.ExecuteReaderAsync(ct); + Assert.True(await reader.ReadAsync(ct)); + return (reader.GetInt64(0), reader.GetInt64(1), reader.GetInt64(2)); + } + + var now = new DateTime(2026, 1, 20, 12, 0, 0, DateTimeKind.Unspecified); + var before = await CountersAsync(); + Assert.Equal(1L, await QueryStoreTopDaily.GcAsync(connection, now, RetentionDays, ct)); + var afterDue = await CountersAsync(); + Assert.Equal(before.SeqScan, afterDue.SeqScan); + Assert.True(afterDue.IdxScan > before.IdxScan, "the due day's summary rows are removed through an index"); + Assert.True(afterDue.IdxTupFetch - before.IdxTupFetch <= 10, $"GcAsync fetched {afterDue.IdxTupFetch - before.IdxTupFetch} summary rows for ten due ones"); + Assert.Equal(90000L, await ScalarAsync(connection, "SELECT count(*) FROM collect.query_store_top_daily", ct)); + + /* The count above is a sequential scan of its own; measure the next pass from a fresh baseline. */ + var beforeIdle = await CountersAsync(); + Assert.Equal(0L, await QueryStoreTopDaily.GcAsync(connection, now, RetentionDays, ct)); + var afterIdle = await CountersAsync(); + Assert.Equal(beforeIdle.SeqScan, afterIdle.SeqScan); + Assert.Equal(beforeIdle.IdxScan, afterIdle.IdxScan); + Assert.Equal(beforeIdle.IdxTupFetch, afterIdle.IdxTupFetch); + Assert.Equal(90000L, await ScalarAsync(connection, "SELECT count(*) FROM collect.query_store_top_daily", ct)); + }); + } + + [Fact] + public async Task ABacklogLargerThanTheCap_BuildsTheOldestDaysFirst_AndExactlyTheCap() + { + await RunLiveAsync(async (scratch, connection, ct) => + { + await CoverAsync(connection, 1, ct); + await using var source = DataSource(scratch); + var now = new DateTime(2026, 6, 1, 12, 0, 0, DateTimeKind.Unspecified); + const int Retention = 100; + + var result = await QueryStoreTopDaily.RunTickAsync(source, now, Retention, NullLogger.Instance, ct); + + Assert.Equal(QueryStoreTopDaily.MaxBuildsPerTick, result.Built); + Assert.Equal(0, result.Failed); + Assert.Equal((long)QueryStoreTopDaily.MaxBuildsPerTick, Convert.ToInt64(await ScalarAsync(connection, "SELECT count(*) FROM collect.query_store_top_daily_built", ct))); + var first = (now - TimeSpan.FromDays(Retention)).Date.AddDays(2); + Assert.Equal(first, (DateTime)(await ScalarAsync(connection, "SELECT min(day)::timestamp FROM collect.query_store_top_daily_built", ct))!); + Assert.Equal(first.AddDays(QueryStoreTopDaily.MaxBuildsPerTick - 1), (DateTime)(await ScalarAsync(connection, "SELECT max(day)::timestamp FROM collect.query_store_top_daily_built", ct))!); + + /* Every one of those days was already past its final-build time, so each was built once, at pass 2. */ + Assert.Equal((long)QueryStoreTopDaily.MaxBuildsPerTick, Convert.ToInt64(await ScalarAsync(connection, "SELECT count(*) FROM collect.query_store_top_daily_built WHERE pass = 2", ct))); + Assert.Equal(0, result.BuiltPass1); + }); + } + + [Fact] + public async Task ABacklogPastTheFinalThreshold_BuildsEachDayOnceAtPassTwo_AndTheNextTickReachesNewerDays() + { + await RunLiveAsync(async (scratch, connection, ct) => + { + await CoverAsync(connection, 1, ct); + await using var source = DataSource(scratch); + var now = new DateTime(2026, 6, 1, 12, 0, 0, DateTimeKind.Unspecified); + const int Retention = 150; + var cap = QueryStoreTopDaily.MaxBuildsPerTick; + var first = (now - TimeSpan.FromDays(Retention)).Date.AddDays(2); + + var one = await QueryStoreTopDaily.RunTickAsync(source, now, Retention, NullLogger.Instance, ct); + Assert.Equal(cap, one.BuiltPass2); + Assert.Equal(0, one.BuiltPass1); + var firstBuiltAt = (DateTime)(await ScalarAsync(connection, "SELECT built_at FROM collect.query_store_top_daily_built WHERE day = DATE '" + At(first)[..10] + "'", ct))!; + + var two = await QueryStoreTopDaily.RunTickAsync(source, now, Retention, NullLogger.Instance, ct); + Assert.Equal(0, two.Failed); + Assert.Equal(0, two.BuiltPass1); + Assert.Equal(cap, two.BuiltPass2); + + /* The second tick built the next days; it did not redo the oldest ones. */ + Assert.Equal((long)(2 * cap), Convert.ToInt64(await ScalarAsync(connection, "SELECT count(*) FROM collect.query_store_top_daily_built WHERE pass = 2", ct))); + Assert.Equal(first.AddDays(2 * cap - 1), (DateTime)(await ScalarAsync(connection, "SELECT max(day)::timestamp FROM collect.query_store_top_daily_built", ct))!); + Assert.Equal(firstBuiltAt, (DateTime)(await ScalarAsync(connection, "SELECT built_at FROM collect.query_store_top_daily_built WHERE day = DATE '" + At(first)[..10] + "'", ct))!); + }); + } + + [Fact] + public async Task ALateWriteIntoABuiltDay_IsMissed_AsDocumented() + { + await RunLiveAsync(async (scratch, connection, ct) => + { + await SeedAsync(connection, 1, Day.AddHours(5), 1, ct, executionCount: 1); + await CoverAsync(connection, 1, ct); + await using var source = DataSource(scratch); + var dayEnd = Day.AddDays(1); + + await TickBuiltAsync(source, dayEnd.AddHours(2)); + await TickBuiltAsync(source, dayEnd.AddHours(25)); + var finalBuild = await BuiltRowAsync(connection, 1, Day, ct); + Assert.Equal((short)2, finalBuild!.Value.Pass); + Assert.Equal(1L, await SummarizedRowsAsync(connection, 1, Day, ct)); + + /* A backfill lands a wide row in the built day. Nothing plans the day again, so the summary does not see it. */ + await SeedAsync(connection, 1, Day.AddHours(6), 2, ct, executionCount: 1); + await TickBuiltAsync(source, dayEnd.AddDays(3)); + + Assert.Equal(1L, await SummarizedRowsAsync(connection, 1, Day, ct)); + Assert.Equal(finalBuild, await BuiltRowAsync(connection, 1, Day, ct)); + Assert.Equal(2L, Convert.ToInt64(await ScalarAsync(connection, + "SELECT count(*) FROM collect.query_store_interval_wide WHERE server_id = 1 AND collection_time >= TIMESTAMP '2026-01-10' AND collection_time < TIMESTAMP '2026-01-11'", ct))); + }); + } + + [Fact] + public async Task TheFirstExecutionRange_KeepsTheRowsAtItsEdges_AndDropsTheOneAtTheUpperEdge() + { + await RunLiveAsync(async (scratch, connection, ct) => + { + var collected = Day.AddHours(12); + var upper = Day.AddDays(1) + QueryStoreTopDaily.SkewSlack; + + /* 1: the oldest start a stored row can have (a second inside the span margin plus catch-up). 2 and 3: a start + just inside the upper edge, and exactly on it. 4: an ordinary row. */ + await SeedAsync(connection, 1, collected, 1, ct, executionCount: 1, duration: 10, + firstExecution: collected - QueryStoreTopDaily.FinalBuildAfter + TimeSpan.FromSeconds(1)); + await SeedAsync(connection, 1, collected, 2, ct, executionCount: 2, duration: 20, firstExecution: upper - TimeSpan.FromSeconds(1)); + await SeedAsync(connection, 1, collected, 3, ct, executionCount: 4, duration: 40, firstExecution: upper); + await SeedAsync(connection, 1, collected, 4, ct, executionCount: 8, duration: 80); + await CoverAsync(connection, 1, ct); + + Assert.Equal(3L, await QueryStoreTopDaily.BuildDayAsync(connection, 1, DateOnly.FromDateTime(Day), 1, Day.AddDays(1).AddHours(2), ct)); + + const string Direct = @" +SELECT server_id, DATE '2026-01-10' AS day, database_name, query_id, plan_id, query_hash, execution_type_desc, replica_role, module_name, + count(*) AS interval_rows, sum(execution_count) AS execution_count_sum, + sum(avg_duration_us::numeric) AS avg_duration_us_sum, count(avg_duration_us) AS avg_duration_us_n, + min(first_execution_time) AS first_execution_time_min +FROM collect.query_store_interval_wide +WHERE server_id = 1 AND collection_time >= TIMESTAMP '2026-01-10' AND collection_time < TIMESTAMP '2026-01-11' + AND first_execution_time < TIMESTAMP '2026-01-11 01:00:00' +GROUP BY server_id, database_name, query_id, plan_id, query_hash, execution_type_desc, replica_role, module_name"; + const string Stored = @" +SELECT server_id, day, database_name, query_id, plan_id, query_hash, execution_type_desc, replica_role, module_name, + interval_rows, execution_count_sum, avg_duration_us_sum, avg_duration_us_n, first_execution_time_min +FROM collect.query_store_top_daily WHERE server_id = 1 AND day = DATE '2026-01-10'"; + + Assert.Equal(new long[] { 1, 2, 4 }, + (await ScalarAsync(connection, "SELECT array_agg(query_id ORDER BY query_id) FROM collect.query_store_top_daily WHERE server_id = 1", ct) as long[])!); + Assert.Equal(0L, await ScalarAsync(connection, $"SELECT count(*) FROM (({Stored}) EXCEPT ({Direct})) x", ct)); + Assert.Equal(0L, await ScalarAsync(connection, $"SELECT count(*) FROM (({Direct}) EXCEPT ({Stored})) x", ct)); + Assert.Equal(3L, (await BuiltRowAsync(connection, 1, Day, ct))!.Value.SourceRows); + }); + } + + [Fact] + public async Task TheOldestPlannedDay_IsTwoDaysAfterTheRetentionHorizon() + { + await RunLiveAsync(async (scratch, connection, ct) => + { + await CoverAsync(connection, 1, ct); + await using var source = DataSource(scratch); + var now = new DateTime(2026, 1, 20, 12, 0, 0, DateTimeKind.Unspecified); + + await using var planConnection = await source.OpenConnectionAsync(ct); + var plan = await QueryStoreTopDaily.PlanBuildsAsync(planConnection, now, RetentionDays, ct); + + /* (now - 9 days)::date is 2026-01-11; the purge may already have cut part of 2026-01-12. */ + Assert.Equal(new DateOnly(2026, 1, 13), plan.Min(b => b.Day)); + Assert.Equal(new DateOnly(2026, 1, 19), plan.Max(b => b.Day)); + Assert.Equal(7, plan.Count); + }); + } + + [Fact] + public async Task OnlyEnabledServers_AndOnlyDaysAfterTheirCoverageClaim_ArePlanned_AndADisabledServersRowsAreCollected() + { + await RunLiveAsync(async (scratch, connection, ct) => + { + const int Retention = 20; + var now = new DateTime(2026, 1, 20, 12, 0, 0, DateTimeKind.Unspecified); + await CoverAsync(connection, 1, ct, filledSince: "2026-01-08 15:00:00"); + await CoverAsync(connection, 2, ct, enabled: false); + await SeedAsync(connection, 1, new DateTime(2026, 1, 9, 5, 0, 0), 1, ct, executionCount: 1); + foreach (var d in new[] { 10, 11 }) + { + await ExecAsync(connection, + $"INSERT INTO collect.query_store_top_daily (server_id, day, interval_rows, avg_duration_us_n, avg_cpu_time_us_n, avg_logical_io_reads_n, avg_logical_io_writes_n, avg_physical_io_reads_n, avg_rowcount_n) VALUES (2, DATE '2026-01-{d:00}', 1, 0, 0, 0, 0, 0, 0)", ct); + await ExecAsync(connection, + $"INSERT INTO collect.query_store_top_daily_built (server_id, day, pass, built_at, source_rows) VALUES (2, DATE '2026-01-{d:00}', 2, TIMESTAMP '2026-01-15 00:00:00', 1)", ct); + } + + await using var source = DataSource(scratch); + await using (var planConnection = await source.OpenConnectionAsync(ct)) + { + var plan = await QueryStoreTopDaily.PlanBuildsAsync(planConnection, now, Retention, ct); + Assert.DoesNotContain(plan, b => b.ServerId == 2); + Assert.All(plan, b => Assert.Equal(1, b.ServerId)); + Assert.Equal(new DateOnly(2026, 1, 9), plan.Min(b => b.Day)); + Assert.Equal(new DateOnly(2026, 1, 19), plan.Max(b => b.Day)); + } + + var result = await QueryStoreTopDaily.RunTickAsync(source, now, Retention, NullLogger.Instance, ct); + + Assert.Equal(0, result.Failed); + Assert.Equal(11, result.Built); + Assert.Equal(2L, result.DaysRemoved); + Assert.Equal(0L, Convert.ToInt64(await ScalarAsync(connection, "SELECT count(*) FROM collect.query_store_top_daily WHERE server_id = 2", ct))); + Assert.Equal(0L, Convert.ToInt64(await ScalarAsync(connection, "SELECT count(*) FROM collect.query_store_top_daily_built WHERE server_id = 2", ct))); + Assert.Equal(11L, Convert.ToInt64(await ScalarAsync(connection, "SELECT count(*) FROM collect.query_store_top_daily_built WHERE server_id = 1", ct))); + Assert.Equal(1L, await SummarizedRowsAsync(connection, 1, new DateTime(2026, 1, 9), ct)); + Assert.Null(await BuiltRowAsync(connection, 1, new DateTime(2026, 1, 8), ct)); + }); + } + + [Fact] + public async Task ATickPastItsTimeBudget_StopsStartingBuilds_AndReportsTheRestDeferred() + { + await RunLiveAsync(async (scratch, connection, ct) => + { + await CoverAsync(connection, 1, ct); + await using var source = DataSource(scratch); + var now = new DateTime(2026, 1, 20, 12, 0, 0, DateTimeKind.Unspecified); + + /* The elapsed time is read before each build: zero before the first, past the budget from then on. */ + var reads = 0; + var result = await QueryStoreTopDaily.RunTickAsync(source, now, RetentionDays, NullLogger.Instance, + () => reads++ == 0 ? TimeSpan.Zero : QueryStoreTopDaily.MaxTickDuration, ct); + + Assert.Equal(1, result.Built); + Assert.Equal(6, result.Deferred); + Assert.Equal(0, result.Failed); + Assert.Equal("2026-01-13", await ScalarAsync(connection, "SELECT string_agg(day::text, ',') FROM collect.query_store_top_daily_built", ct)); + + /* The next tick plans what was deferred. */ + var next = await QueryStoreTopDaily.RunTickAsync(source, now, RetentionDays, NullLogger.Instance, ct); + Assert.Equal(6, next.Built); + Assert.Equal(0, next.Deferred); + }); + } + + [Fact] + public void TheTenant_IsTheLastAwaitInsideTheTimescaleGate_WithItsOwnCatchAll() + { + var worker = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "DarlingWorker.cs"); + var code = CSharpSourceWalker.StripCommentsAndStrings(worker); + var tick = Body(code, "private async Task RunStoreMaintenanceTickAsync(CancellationToken stoppingToken)"); + Assert.False(string.IsNullOrEmpty(tick), "could not locate RunStoreMaintenanceTickAsync"); + + const string Converge = "await ConvergeStoreObjectsAsync(stoppingToken);"; + const string Build = "await BuildQueryStoreTopDailyAsync(stoppingToken);"; + var convergeAt = tick.IndexOf(Converge, StringComparison.Ordinal); + var buildAt = tick.IndexOf(Build, StringComparison.Ordinal); + Assert.True(convergeAt >= 0 && buildAt > convergeAt, "the summary builder is awaited after the store-object convergence"); + Assert.Equal(1, tick.Split(Build).Length - 1); + Assert.True(string.IsNullOrWhiteSpace(tick[(convergeAt + Converge.Length)..buildAt]), "nothing sits between the convergence and the builder"); + Assert.True(tick[(buildAt + Build.Length)..].TrimStart().StartsWith('}'), "the summary builder is the last await inside the TimescaleDB gate (only the closing brace of the gated block follows it)"); + Assert.Equal(';', tick[..buildAt].TrimEnd()[^1]); + + var builder = Body(code, "private async Task BuildQueryStoreTopDailyAsync(CancellationToken stoppingToken)"); + Assert.False(string.IsNullOrEmpty(builder), "could not locate BuildQueryStoreTopDailyAsync"); + Assert.Contains("catch (Exception ex) when (ex is not OperationCanceledException)", builder, StringComparison.Ordinal); + Assert.Contains("QueryStoreTopDaily.RunTickAsync(", builder, StringComparison.Ordinal); + Assert.Contains("DarlingRetention.QueryStoreIntervalWideRetentionDays", builder, StringComparison.Ordinal); + Assert.DoesNotContain("throw", builder, StringComparison.Ordinal); + } + + [Fact] + public void TheConstants_AreTheDocumentedOnes() + { + Assert.Equal(TimeSpan.FromHours(2), QueryStoreTopDaily.BuildAfter); + Assert.Equal(TimeSpan.FromHours(25), QueryStoreTopDaily.FinalBuildAfter); + Assert.Equal(60, QueryStoreTopDaily.MaxBuildsPerTick); + Assert.Equal(60, QueryStoreTopDaily.BuildStatementTimeoutSeconds); + Assert.Equal(TimeSpan.FromMinutes(10), QueryStoreTopDaily.MaxTickDuration); + Assert.Equal(TimeSpan.FromHours(1), QueryStoreTopDaily.SkewSlack); + Assert.Contains("60s", typeof(QueryStoreTopDaily).GetField("StatementTimeoutSql", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Static)!.GetValue(null)!.ToString(), StringComparison.Ordinal); + } + + private static string Body(string source, string signature) + { + var start = source.IndexOf(signature, StringComparison.Ordinal); + if (start < 0) + { + return string.Empty; + } + + var open = source.IndexOf('{', start); + var depth = 0; + for (var i = open; i < source.Length; i++) + { + if (source[i] == '{') + { + depth++; + } + else if (source[i] == '}' && --depth == 0) + { + return source[(open + 1)..i]; + } + } + + return string.Empty; + } +} diff --git a/Darling/Darling.Tests/QueryStoreTopDailyReadLiveTests.cs b/Darling/Darling.Tests/QueryStoreTopDailyReadLiveTests.cs new file mode 100644 index 0000000000..2efe066076 --- /dev/null +++ b/Darling/Darling.Tests/QueryStoreTopDailyReadLiveTests.cs @@ -0,0 +1,532 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// Pins the long-window read of get_query_store_top (#5094): whole built UTC days come from +/// collect.query_store_top_daily, the partial edge days and any unbuilt day from the wide table, in one statement +/// (). With no late change to a built day the answer equals the +/// interval-table read () value for value, doubles included +/// bit for bit; the one documented difference is a row written into a built day after that day was built. Every seed +/// is at a fixed past date, every clock is passed explicitly, and each live fact mints its own scratch database. +/// +/* #1776 own-store: deliberately NOT [Collection("live-postgres")]. Every live fact here reaches DARLING_TEST_PG only to + CREATE and DROP its own database through ScratchPostgres and then works entirely inside it, so it cannot race live + collection. */ +public sealed class QueryStoreTopDailyReadLiveTests +{ + private const string SkipText = "Set DARLING_TEST_PG to a Postgres connection string to run the daily summary read's live pins (each mints its own scratch database)."; + private const int ServerId = 1; + private const int Top = 30; + private const int RetentionDays = 9; + + private static readonly DateTime End = new(2026, 1, 14, 6, 0, 0, DateTimeKind.Unspecified); + private static readonly DateTime BuildNow = new(2026, 1, 15, 12, 0, 0, DateTimeKind.Unspecified); + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + /// SHA-256 of each statement, line endings normalized. They were taken from the pre-split text, before + /// QueryStoreTopSuffix was divided into its ranked head and its tail, and re-taken once for #5313, which + /// deliberately changed both statements: the over-fetch LIMIT $4 + 5 became the round's candidate limit and + /// the tail gained the page / count-row wrapper, and re-taken again for #5420, which deliberately bounded the tail's + /// inline-text fallback to the read's window (collection_time >= $2 and <= $3), and re-taken for #5523, which deliberately added the table read's upper first_execution_time bound (<= $3 + the slack, twelve hours). The split pin is the + /// tail-sharing assert below. + private const string RawSqlHash = "836D1C99489C7EC952E327EE1CE1C40D7B94DE3270EC67D9249D32D05634CC33"; + private const string TableSqlHash = "7B5AF0879E0AF1B75762318E4B41645D370939B2B0FFAEB768FCD9B589B2AB79"; + + private static string Hash(string sql) => + Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(sql.ReplaceLineEndings("\n")))); + + [Fact] + public void TheRawAndTableStatements_AreByteIdenticalToTheirPreSplitText() + { + Assert.Equal(RawSqlHash, Hash(DarlingDataReader.QueryStoreTopSql)); + Assert.Equal(TableSqlHash, Hash(DarlingDataReader.QueryStoreTopTableSql)); + } + + [Fact] + public void TheDailyStatement_SharesTheTableReadsTail_AndItsPredicates() + { + var daily = DarlingDataReader.QueryStoreTopDailyTableSql.ReplaceLineEndings("\n"); + var table = DarlingDataReader.QueryStoreTopTableSql.ReplaceLineEndings("\n"); + var tail = table[table.IndexOf("SELECT\n r.database_name,", StringComparison.Ordinal)..]; + Assert.EndsWith(tail, daily, StringComparison.Ordinal); + Assert.Contains("$7::text IS NULL OR module_name = $7", daily, StringComparison.Ordinal); + Assert.Contains("LIMIT $10", daily, StringComparison.Ordinal); /* #5313: the candidate limit, bound after the two dates */ + } + + [Fact] + public void TheLongestRun_TakesTheLongest_TiesToTheLatest_AndNothingFromNothing() + { + static DateOnly D(int day) => new(2026, 1, day); + Assert.Null(DarlingDataReader.LongestBuiltRun(Array.Empty())); + Assert.Equal((D(8), D(11)), DarlingDataReader.LongestBuiltRun(new[] { D(8), D(9), D(10), D(12), D(13) })); + Assert.Equal((D(11), D(14)), DarlingDataReader.LongestBuiltRun(new[] { D(8), D(9), D(11), D(12), D(13) })); + Assert.Equal((D(11), D(13)), DarlingDataReader.LongestBuiltRun(new[] { D(8), D(9), D(11), D(12) })); + Assert.Equal((D(9), D(10)), DarlingDataReader.LongestBuiltRun(new[] { D(9) })); + } + + [Fact] + public void TheWholeDays_AreTheMidnightsInsideTheWindow() + { + var (first, end) = DarlingDataReader.WholeDays(new DateTime(2026, 1, 11, 6, 0, 0), new DateTime(2026, 1, 14, 6, 0, 0)); + Assert.Equal((new DateOnly(2026, 1, 12), new DateOnly(2026, 1, 14)), (first, end)); + (first, end) = DarlingDataReader.WholeDays(new DateTime(2026, 1, 11, 0, 0, 0), new DateTime(2026, 1, 14, 0, 0, 0)); + Assert.Equal((new DateOnly(2026, 1, 11), new DateOnly(2026, 1, 14)), (first, end)); + } + + /* ───────────────────────────── live ───────────────────────────── */ + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(ct); + } + + private static string At(DateTime value) => value.ToString("yyyy-MM-dd HH:mm:ss.ffffff", CultureInfo.InvariantCulture); + + /// The wide table for server 1 from 2026-01-05 to 2026-01-14 12:00, one row per query per hour for most + /// hours (a row at every midnight among them), with nulls where the facts need them. 40 queries: query 40 has every + /// average NULL, queries divisible by 7 have a NULL cpu on odd hours, queries divisible by 4 change module + /// between hours, queries divisible by 5 are Aborted, and queries divisible by 6 carry a replica role. + private const string SeedSql = @" +INSERT INTO collect.query_store_interval_wide +(collection_time, server_id, database_name, query_id, plan_id, execution_type_desc, first_execution_time, last_execution_time, + module_name, query_hash, execution_count, avg_duration_us, avg_cpu_time_us, avg_logical_io_reads, avg_logical_io_writes, + avg_physical_io_reads, avg_rowcount, query_plan_hash, replica_role, runtime_stats_interval_id, interval_start_time_utc) +SELECT + ct, 1, 'db' || (q % 3), q, q, CASE WHEN q % 5 = 0 THEN 'Aborted' ELSE 'Regular' END, + ct - interval '10 minutes', ct - make_interval(mins => h % 7), + CASE WHEN q % 4 = 0 THEN CASE WHEN h % 2 = 0 THEN 'dbo.m1' ELSE NULL END WHEN q % 4 = 1 THEN 'dbo.m2' ELSE NULL END, + 'h' || q, 1 + (q * 13 + h * 7) % 23, + CASE WHEN q = 40 THEN NULL ELSE 1000 + (q * 7919 + h * 104729) % 90001 END, + CASE WHEN q = 40 OR (q % 7 = 0 AND h % 2 = 1) THEN NULL ELSE 10 + (q * 31 + h * 17) % 5003 END, + CASE WHEN q = 40 THEN NULL ELSE 100 + (q * 101 + h * 13) % 500009 END, + CASE WHEN q = 40 THEN NULL ELSE (q * 3 + h) % 97 END, + CASE WHEN q = 40 THEN NULL ELSE (q + h * 5) % 211 END, + CASE WHEN q = 40 THEN NULL ELSE 1 + (q * 29 + h * 3) % 1009 END, + 'ph' || (h % 5), CASE WHEN q % 6 = 0 THEN 'PRIMARY' ELSE NULL END, + q * 100000 + h, ct - interval '10 minutes' +FROM +( + SELECT q, h, TIMESTAMP '2026-01-05 00:00:00' + make_interval(hours => h) AS ct + FROM generate_series(1, 40) AS q + CROSS JOIN generate_series(0, 24 * 9 + 12) AS h + WHERE (q * 7 + h) % 5 <> 0 OR h % 24 = 0 +) AS s;"; + + private static async Task SeedAsync(NpgsqlConnection connection, CancellationToken ct) + { + await ExecAsync(connection, SeedSql, ct); + await ExecAsync(connection, "INSERT INTO collect.servers (server_id, server_name, is_enabled) VALUES (1, 'srv1', TRUE)", ct); + /* applied_through at or before the read's literal end, or the gate refuses the table (clause 4). */ + await ExecAsync(connection, + "INSERT INTO collect.query_store_interval_wide_coverage (server_id, filled_since, applied_through) VALUES (1, TIMESTAMP '2026-01-01 00:00:00', TIMESTAMP '2026-01-01 00:00:00')", ct); + } + + /// One extra wide-table row for query with a duration large enough to rank first. + private static async Task InsertRowAsync(NpgsqlConnection connection, DateTime collectionTime, long queryId, CancellationToken ct, long duration = 90_000_000, DateTime? firstExecution = null) + { + await using var command = new NpgsqlCommand(@" +INSERT INTO collect.query_store_interval_wide +(collection_time, server_id, database_name, query_id, plan_id, execution_type_desc, first_execution_time, last_execution_time, + query_hash, execution_count, avg_duration_us, avg_cpu_time_us, avg_logical_io_reads, avg_logical_io_writes, avg_physical_io_reads, + avg_rowcount, query_plan_hash, runtime_stats_interval_id, interval_start_time_utc) +VALUES (@ct, 1, 'db0', @q, @q, 'Regular', COALESCE(@fet, @ct - interval '10 minutes'), @ct, 'h' || @q, 1000, @dur, 5, 5, 5, 5, 5, 'late', @q, @ct - interval '10 minutes')", connection); + command.Parameters.Add(new NpgsqlParameter("ct", NpgsqlDbType.Timestamp) { Value = collectionTime }); + command.Parameters.Add(new NpgsqlParameter("fet", NpgsqlDbType.Timestamp) { Value = (object?)firstExecution ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter("q", NpgsqlDbType.Bigint) { Value = queryId }); + command.Parameters.Add(new NpgsqlParameter("dur", NpgsqlDbType.Bigint) { Value = duration }); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task BuildAsync(NpgsqlDataSource source, CancellationToken ct) + { + var result = await QueryStoreTopDaily.RunTickAsync(source, BuildNow, RetentionDays, NullLogger.Instance, ct); + Assert.Equal(0, result.Failed); + Assert.Equal(7, result.Built); + } + + private static async Task RunLiveAsync(Func body) + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await using var source = NpgsqlDataSource.Create(scratch.ConnectionString); + var bodySucceeded = false; + try + { + await SeedAsync(connection, ct); + await BuildAsync(source, ct); + await body(connection, source, ct); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + private static DarlingDataReader.QueryStoreRow ToRow(object?[] v) + { + static T? Get(object? value) => value is null or DBNull ? default : (T)value; + return new DarlingDataReader.QueryStoreRow( + Get(v[0]) ?? "", Get(v[1]) ?? 0, Get(v[2]) ?? 0, Get(v[3]) ?? "", Get(v[4]) ?? "", + Get(v[5]) ?? "", Get(v[6]), Get(v[7]) ?? 0, Get(v[8]) ?? 0, Get(v[9]) ?? 0, + Get(v[10]) ?? 0, Get(v[11]) ?? 0, Get(v[12]) ?? 0, Get(v[13]) ?? 0, + Get(v[14]), Get(v[15]) ?? "", Get(v[16])); + } + + /// A row as text with every double as its bit pattern, so "equal" means bit-equal. + private static string Format(DarlingDataReader.QueryStoreRow r) + { + static string B(double d) => BitConverter.DoubleToInt64Bits(d).ToString("x", CultureInfo.InvariantCulture); + return string.Join('|', r.DatabaseName, r.QueryId, r.PlanId, r.QueryHash, r.QueryPlanHash, r.ExecutionTypeDesc, r.ModuleName ?? "", + r.TotalExecutions, B(r.AvgDurationMs), B(r.AvgCpuTimeMs), B(r.AvgLogicalReads), B(r.AvgLogicalWrites), B(r.AvgPhysicalReads), + B(r.AvgRowcount), r.LastExecutionTime is { } t ? At(t) : "", r.QueryText, r.ReplicaRole ?? ""); + } + + private static List Format(IEnumerable rows) => rows.Select(Format).ToList(); + + /// Runs (either statement) and returns its raw rows, NULLs kept. + private static async Task> RunSqlAsync( + NpgsqlConnection connection, string sql, DateTime readStart, DateTime end, string? db, string? outcome, string? module, + (DateOnly, DateOnly)? span, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + command.Parameters.Add(new NpgsqlParameter { TypedValue = ServerId }); + command.Parameters.Add(new NpgsqlParameter { TypedValue = readStart }); + command.Parameters.Add(new NpgsqlParameter { TypedValue = end }); + command.Parameters.Add(new NpgsqlParameter { TypedValue = Top }); + command.Parameters.Add(PerformanceMonitor.Darling.Storage.DatabaseFilter.One(db).Parameter()); /* #5245: the list predicate binds one text[] */ + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Text, Value = (object?)outcome ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Text, Value = (object?)module ?? DBNull.Value }); + if (span is var (s, e)) + { + command.Parameters.Add(new NpgsqlParameter { TypedValue = s, NpgsqlDbType = NpgsqlDbType.Date }); + command.Parameters.Add(new NpgsqlParameter { TypedValue = e, NpgsqlDbType = NpgsqlDbType.Date }); + } + + command.Parameters.Add(new NpgsqlParameter { TypedValue = PerformanceMonitor.Darling.Storage.TopFill.FirstCandidates(Top) }); /* #5313: the round's candidate limit, bound last */ + + var rows = new List(); + await using var reader = await command.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + var values = new object[reader.FieldCount]; + reader.GetValues(values); + rows.Add(values); + } + + return rows; + } + + /// Today's read () of the same window, on the same connection. + private static async Task> TodaysReadAsync( + NpgsqlConnection connection, DateTime readStart, DateTime end, string? db, string? outcome, string? module, CancellationToken ct) => + Format((await RunSqlAsync(connection, DarlingDataReader.QueryStoreTopTableSql, readStart, end, db, outcome, module, null, ct)).Select(ToRow)); + + private static async Task RouteAsync( + NpgsqlDataSource source, DateTime start, DateTime end, string? db, string? outcome, string? module, CancellationToken ct) => + await DarlingDataReader.GetQueryStoreTopWithReachAsync(source, ServerId, start, end, Top, db, outcome, module, ct); + + public static IEnumerable Cells() + { + foreach (var hours in new[] { 72, 168 }) + { + yield return new object?[] { hours, null, null, null }; + yield return new object?[] { hours, "db1", null, null }; + yield return new object?[] { hours, null, "Aborted", null }; + yield return new object?[] { hours, null, null, "dbo.m1" }; + yield return new object?[] { hours, "db0", "Regular", "dbo.m2" }; + } + } + + [Theory] + [MemberData(nameof(Cells))] + public async Task TheRoute_EqualsTodaysTableRead_BitForBit_WhenNothingChangedAfterTheBuild(int hours, string? db, string? outcome, string? module) + { + await RunLiveAsync(async (connection, source, ct) => + { + var start = End.AddHours(-hours); + var read = await RouteAsync(source, start, End, db, outcome, module, ct); + + Assert.NotNull(read.Table); + Assert.True(read.DailyDaysUsed > 0); + Assert.Equal((hours == 72 ? 2 : 6), read.DailyDaysUsed); + Assert.Equal(new DateOnly(2026, 1, 14), read.DailySpan!.Value.EndExclusive); + Assert.NotEmpty(read.Rows); + + var today = await TodaysReadAsync(connection, read.Table!.Value.ReadStart, End, db, outcome, module, ct); + Assert.Equal(today, Format(read.Rows)); + }); + } + + [Fact] + public async Task AFaultReadingTheBuiltDays_AfterTheGateChoseTheTable_NotesFallbackRaw_AndAnswersFromRaw() + { + await RunLiveAsync(async (connection, source, ct) => + { + await ExecAsync(connection, "DROP TABLE collect.query_store_top_daily_built", ct); + await ExecAsync(connection, @" +INSERT INTO collect.query_store_stats (collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, + execution_type_desc, first_execution_time, module_name, query_hash, execution_count, avg_duration_us, avg_cpu_time_us, + max_duration_us, max_cpu_time_us, replica_role, runtime_stats_interval_id, interval_start_time_utc) +SELECT 1, TIMESTAMP '2026-01-10 00:30:00' + (i * interval '1 hour'), 1, 'srv1', 'db0', 100 + i, 1000 + i, + 'Regular', TIMESTAMP '2026-01-10 00:00:00' + (i * interval '1 hour'), 'mod', md5('q' || i), 10 + i, 500 + i, 300, 900, 700, 'PRIMARY', 77 + i, + TIMESTAMP '2026-01-10 00:00:00' + (i * interval '1 hour') +FROM generate_series(0, 9) AS i", ct); + var start = End.AddHours(-168); + var logger = new CapturingTestLogger(); + using var scope = ReadScope.Open(logger); + + var read = await RouteAsync(source, start, End, null, null, null, ct); + + Assert.Null(read.Table); + Assert.Equal(0, read.DailyDaysUsed); + Assert.Equal(ReadFallback.FallbackRaw, scope.Fallback); + var raw = Format((await RunSqlAsync(connection, DarlingDataReader.QueryStoreTopSql, start, End, null, null, null, null, ct)).Select(ToRow)); + Assert.NotEmpty(raw); + Assert.Equal(raw, Format(read.Rows)); + }); + } + + [Fact] + public async Task WithNoBuiltDay_TheReadIsTodaysStatement_AndNoSummaryIsUsed() + { + await RunLiveAsync(async (connection, source, ct) => + { + await ExecAsync(connection, "DELETE FROM collect.query_store_top_daily; DELETE FROM collect.query_store_top_daily_built", ct); + var read = await RouteAsync(source, End.AddHours(-168), End, null, null, null, ct); + + Assert.Equal(0, read.DailyDaysUsed); + Assert.Null(read.DailySpan); + Assert.Equal(await TodaysReadAsync(connection, read.Table!.Value.ReadStart, End, null, null, null, ct), Format(read.Rows)); + }); + } + + private static async Task ToolAnswerAsync(NpgsqlDataSource source, int hours, CancellationToken ct) => + System.Text.Json.JsonDocument.Parse(await DarlingMcpDataTools.GetQueryStoreTop( + source, "srv1", hours, Top, null, "2026-01-14T06:00:00Z", null, null, false, 400, ct)).RootElement.Clone(); + + [Fact] + public async Task TheTool_SaysApproximate_WithTheSummaryDays_WhenABuiltDayWasUsed() + { + await RunLiveAsync(async (connection, source, ct) => + { + var answer = await ToolAnswerAsync(source, 72, ct); + + Assert.True(answer.GetProperty("approximate").GetBoolean()); + Assert.Equal(DarlingMcpDataTools.QueryStoreApproximationNote, answer.GetProperty("approximation_note").GetString()); + var days = answer.GetProperty("summary_days"); + Assert.Equal("2026-01-12", days.GetProperty("from").GetString()); + Assert.Equal("2026-01-14", days.GetProperty("to_exclusive").GetString()); + await Task.CompletedTask; + }); + } + + [Fact] + public async Task TheTool_SaysNotApproximate_AndCarriesNoNote_WhenNoBuiltDayWasUsed() + { + await RunLiveAsync(async (connection, source, ct) => + { + await ExecAsync(connection, "DELETE FROM collect.query_store_top_daily; DELETE FROM collect.query_store_top_daily_built", ct); + var answer = await ToolAnswerAsync(source, 72, ct); + + Assert.False(answer.GetProperty("approximate").GetBoolean()); + Assert.Equal(System.Text.Json.JsonValueKind.Null, answer.GetProperty("approximation_note").ValueKind); + Assert.Equal(System.Text.Json.JsonValueKind.Null, answer.GetProperty("summary_days").ValueKind); + }); + } + + [Fact] + public async Task AGapInTheBuiltDays_UsesTheLongestRun_AndStillEqualsTodaysRead() + { + await RunLiveAsync(async (connection, source, ct) => + { + /* 168 h from 2026-01-07 06:00: whole days 8..13 are built. Losing the 10th leaves 8-9 and 11-13, and the longer run is used. */ + await ExecAsync(connection, "DELETE FROM collect.query_store_top_daily WHERE day = DATE '2026-01-10'; DELETE FROM collect.query_store_top_daily_built WHERE day = DATE '2026-01-10'", ct); + var read = await RouteAsync(source, End.AddHours(-168), End, null, null, null, ct); + + Assert.Equal(3, read.DailyDaysUsed); + Assert.Equal((new DateOnly(2026, 1, 11), new DateOnly(2026, 1, 14)), read.DailySpan); + Assert.Equal(await TodaysReadAsync(connection, read.Table!.Value.ReadStart, End, null, null, null, ct), Format(read.Rows)); + }); + } + + [Fact] + public async Task ALateWriteIntoABuiltDay_IsMissedByTheRoute_AndPresentInTodaysRead() + { + await RunLiveAsync(async (connection, source, ct) => + { + var start = End.AddHours(-168); + var before = await RouteAsync(source, start, End, null, null, null, ct); + + await InsertRowAsync(connection, new DateTime(2026, 1, 10, 10, 30, 0), 9001, ct); + + var route = await RouteAsync(source, start, End, null, null, null, ct); + var today = await TodaysReadAsync(connection, route.Table!.Value.ReadStart, End, null, null, null, ct); + + /* The route still answers as it did before the write; today's read has the new query, ranked first. */ + Assert.Equal(Format(before.Rows), Format(route.Rows)); + Assert.DoesNotContain(route.Rows, r => r.QueryId == 9001); + Assert.True(today.Any(t => t.Contains("|9001|", StringComparison.Ordinal)), string.Join("\n", today.Take(3))); + Assert.Equal(route.Rows.Count, today.Count); + }); + } + + [Fact] + public async Task TheEdges_AreReadExactlyFromTheWideTable_AtBothBoundaries() + { + await RunLiveAsync(async (connection, source, ct) => + { + /* A 72 h window ending 2026-01-14 06:00 holds the whole days 12 and 13 (the summary's [S, E) is [12, 14)). + Written after the build, so a row the summary covers is missed and a row an edge covers is read. */ + var s = new DateTime(2026, 1, 12, 0, 0, 0); + await InsertRowAsync(connection, s.AddSeconds(-1), 9101, ct); // last second of the first edge: read + await InsertRowAsync(connection, s, 9102, ct); // exactly S: inside the summary's first day, missed + await InsertRowAsync(connection, new DateTime(2026, 1, 14, 0, 0, 0), 9103, ct); // exactly E: the trailing edge, read + await InsertRowAsync(connection, End, 9104, ct); // exactly $3, inclusive: read + await InsertRowAsync(connection, End.AddSeconds(1), 9105, ct); // past $3: in neither + await InsertRowAsync(connection, End.AddHours(-72).AddSeconds(-1), 9106, ct); // before ReadStart: in neither + + var readStart = End.AddHours(-72); + await InsertRowAsync(connection, readStart, 9107, ct); // exactly ReadStart ($2 inclusive): read, by both routes + /* The oldest first_execution_time the invariant allows for a row collected at E + 1 h (the stricter trailing floor). */ + var trailing = new DateTime(2026, 1, 14, 1, 0, 0); + await InsertRowAsync(connection, trailing, 9108, ct, + firstExecution: trailing - (QueryStoreIntervalWide.IntervalSpanMargin + PerformanceMonitor.Collectors.WatermarkPolicy.MaxCatchup) + TimeSpan.FromSeconds(1)); + + var route = await RouteAsync(source, readStart, End, null, null, null, ct); + Assert.Equal((new DateOnly(2026, 1, 12), new DateOnly(2026, 1, 14)), route.DailySpan); + Assert.Equal(readStart, route.Table!.Value.ReadStart); + var ids = route.Rows.Select(r => r.QueryId).ToHashSet(); + Assert.Contains(9101L, ids); + Assert.DoesNotContain(9102L, ids); + Assert.Contains(9103L, ids); + Assert.Contains(9104L, ids); + Assert.DoesNotContain(9105L, ids); + Assert.DoesNotContain(9106L, ids); + Assert.Contains(9107L, ids); + Assert.Contains(9108L, ids); + + /* Today's read has everything the window holds except the two outside it. */ + var today = (await RunSqlAsync(connection, DarlingDataReader.QueryStoreTopTableSql, route.Table!.Value.ReadStart, End, null, null, null, null, ct)) + .Select(v => (long)v[1]!).ToHashSet(); + Assert.Contains(9102L, today); + Assert.DoesNotContain(9105L, today); + Assert.DoesNotContain(9106L, today); + Assert.Contains(9107L, today); + Assert.Contains(9108L, today); + }); + } + + [Fact] + public async Task ARowMovedFromABuiltDayIntoTheTrailingEdge_IsCountedInBothByTheRoute_AndOnceByTodaysRead() + { + await RunLiveAsync(async (connection, source, ct) => + { + /* Day 13 is rebuilt with the row in it (pass 1), then the interval resumes and the row's collection_time moves to + the next day, which is the trailing edge (read from the wide table). The summary still holds it for day 13. */ + await InsertRowAsync(connection, new DateTime(2026, 1, 13, 23, 30, 0), 9201, ct); + await ExecAsync(connection, "DELETE FROM collect.query_store_top_daily WHERE day = DATE '2026-01-13'; DELETE FROM collect.query_store_top_daily_built WHERE day = DATE '2026-01-13'", ct); + var tick = await QueryStoreTopDaily.RunTickAsync(source, BuildNow, RetentionDays, NullLogger.Instance, ct); + Assert.Equal(0, tick.Failed); + Assert.Equal(1, tick.Built); + await ExecAsync(connection, "UPDATE collect.query_store_interval_wide SET collection_time = TIMESTAMP '2026-01-14 00:30:00' WHERE query_id = 9201", ct); + + var route = await RouteAsync(source, End.AddHours(-72), End, null, null, null, ct); + var today = (await RunSqlAsync(connection, DarlingDataReader.QueryStoreTopTableSql, route.Table!.Value.ReadStart, End, null, null, null, null, ct)) + .Select(ToRow).Single(r => r.QueryId == 9201); + + Assert.Equal(1000L, today.TotalExecutions); + Assert.Equal(2000L, route.Rows.Single(r => r.QueryId == 9201).TotalExecutions); + }); + } + + private static async Task FilteredToolAnswerAsync(NpgsqlDataSource source, string? outcome, string? module, CancellationToken ct) => + System.Text.Json.JsonDocument.Parse(await DarlingMcpDataTools.GetQueryStoreTop( + source, "srv1", 72, Top, null, "2026-01-14T06:00:00Z", outcome, module, false, 400, ct)).RootElement.Clone(); + + [Theory] + [InlineData(true)] + [InlineData(false)] + public async Task TheEmptyAnswers_CarryTheApproximateFlag_OnlyWhenBuiltDaysWereUsed(bool withBuiltDays) + { + await RunLiveAsync(async (connection, source, ct) => + { + if (!withBuiltDays) + await ExecAsync(connection, "DELETE FROM collect.query_store_top_daily; DELETE FROM collect.query_store_top_daily_built", ct); + + var module = await FilteredToolAnswerAsync(source, null, "dbo.nothing_matches_this", ct); + var outcome = await FilteredToolAnswerAsync(source, "Exception", null, ct); + + Assert.Equal("empty", module.GetProperty("status").GetString()); + Assert.Equal("empty", outcome.GetProperty("status").GetString()); + if (withBuiltDays) + { + var hints = module.GetProperty("hints"); + Assert.True(hints.GetProperty("approximate").GetBoolean()); + Assert.Equal(DarlingMcpDataTools.QueryStoreApproximationNote, hints.GetProperty("approximation_note").GetString()); + hints = outcome.GetProperty("hints"); + Assert.True(hints.GetProperty("approximate").GetBoolean()); + Assert.Equal(DarlingMcpDataTools.QueryStoreApproximationNote, hints.GetProperty("approximation_note").GetString()); + } + else + { + Assert.False(module.GetProperty("hints").TryGetProperty("approximate", out _)); + Assert.False(module.GetProperty("hints").TryGetProperty("approximation_note", out _)); + Assert.False(outcome.TryGetProperty("hints", out _)); + } + await Task.CompletedTask; + }); + } + + [Fact] + public async Task AnAllNullAverageColumn_IsNullInBothReads_WithoutADivisionError() + { + await RunLiveAsync(async (connection, source, ct) => + { + var start = End.AddHours(-168); + var plan = (await RouteAsync(source, start, End, null, null, null, ct)).Table!.Value; + var span = (new DateOnly(2026, 1, 8), new DateOnly(2026, 1, 14)); + var route = await RunSqlAsync(connection, DarlingDataReader.QueryStoreTopDailyTableSql, plan.ReadStart, End, null, null, null, span, ct); + var today = await RunSqlAsync(connection, DarlingDataReader.QueryStoreTopTableSql, plan.ReadStart, End, null, null, null, null, ct); + + var routeRow = route.Single(v => (long)v[1]! == 40); + var todayRow = today.Single(v => (long)v[1]! == 40); + for (var column = 8; column <= 13; column++) + { + Assert.True(routeRow[column] is DBNull, $"route column {column}"); + Assert.True(todayRow[column] is DBNull, $"today column {column}"); + } + + Assert.Equal(todayRow[7], routeRow[7]); + }); + } +} diff --git a/Darling/Darling.Tests/QueryStoreTopDailyRungTests.cs b/Darling/Darling.Tests/QueryStoreTopDailyRungTests.cs new file mode 100644 index 0000000000..9146c6e50b --- /dev/null +++ b/Darling/Darling.Tests/QueryStoreTopDailyRungTests.cs @@ -0,0 +1,457 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Reflection; +using System.Text.RegularExpressions; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// Pins the Darling rung that adds the daily summary tables for get_query_store_top's long windows (#5094): +/// collect.query_store_top_daily and collect.query_store_top_daily_built. Both are plain tables (a day is +/// deleted and rebuilt whole, which a hypertable chunk would complicate), the summary's key columns copy the types of +/// collect.query_store_interval_wide and its unique index is NULLS NOT DISTINCT because most key columns +/// are NULL for most rows. The summary is approximate by design; the rung only creates the tables, and no reader or +/// writer uses them yet. Every fact finds the rung by NAME, so a renumber is one edit to the registration, the constant +/// and the viewer's return. The live facts run when DARLING_TEST_PG is set, each on its own scratch store. +/// +[Collection("live-postgres")] +public sealed class QueryStoreTopDailyRungTests +{ + public const string RungName = "query-store-top-daily"; + + private const string Daily = "query_store_top_daily"; + + private const string Built = "query_store_top_daily_built"; + + /// This rung's sentinel; the ordinal is a fact of the probe's shape. A newer rung's sentinel follows it. + private const int ProbeOrdinal = 136; + + private const string SkipText = "Set DARLING_TEST_PG to a Postgres connection string to run the daily summary tables' live pins (each mints its own scratch database)."; + + private static readonly string[] KeyColumns = + { + "server_id", "day", "database_name", "query_id", "plan_id", "query_hash", "execution_type_desc", "replica_role", "module_name", + }; + + private static readonly string[] AveragedColumns = + { + "avg_duration_us", "avg_cpu_time_us", "avg_logical_io_reads", "avg_logical_io_writes", "avg_physical_io_reads", "avg_rowcount", + }; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + /// The rung's version, read off the migration registered under . + public static int RungVersion => Rung.Version; + + private static PgMigrations.Migration Rung => PgMigrations.Scripts.Single(m => m.Name == RungName); + + /// The rung's SQL with its comments removed, so a pin reads statements and not the prose around them. + private static string Statements() => + Regex.Replace(Rung.Sql, @"/\*.*?\*/", " ", RegexOptions.Singleline).Replace("\r\n", "\n", StringComparison.Ordinal); + + /// The columns of one CREATE TABLE in a script, name to type text, in order. + private static List<(string Name, string Type)> ColumnsOf(string sql, string table) + { + var stripped = Regex.Replace(sql, @"/\*.*?\*/", " ", RegexOptions.Singleline).Replace("\r\n", "\n", StringComparison.Ordinal); + var header = "CREATE TABLE IF NOT EXISTS " + table + "\n("; + var start = stripped.IndexOf(header, StringComparison.Ordinal); + Assert.True(start >= 0, "no CREATE TABLE for " + table); + var end = stripped.IndexOf("\n)", start, StringComparison.Ordinal); + var columns = new List<(string, string)>(); + foreach (var raw in stripped[(start + header.Length)..end].Split('\n')) + { + var line = raw.Trim().TrimEnd(','); + if (line.Length == 0 || line.StartsWith("CONSTRAINT", StringComparison.Ordinal)) + { + continue; + } + + var space = line.IndexOf(' ', StringComparison.Ordinal); + var type = line[(space + 1)..].Replace(" NOT NULL", string.Empty, StringComparison.Ordinal).Trim(); + columns.Add((line[..space], type)); + } + + return columns; + } + + private static string WideSql() => + PgMigrations.Scripts.Single(m => m.Sql.Contains("CREATE TABLE IF NOT EXISTS collect.query_store_interval_wide\n", StringComparison.Ordinal) + || m.Sql.Contains("CREATE TABLE IF NOT EXISTS collect.query_store_interval_wide\r\n", StringComparison.Ordinal)).Sql; + + [Fact] + public void TheCleanup_IsDrivenFromTheSmallBuiltTable_SoTheSummaryDeleteIsAnIndexProbeByKey() + { + /* #5507: the shape PlanRegressionDaily took in #5448. A delete on the summary with a day test OR a NOT IN of its own + cannot use ux_query_store_top_daily and reads every summary row on every hourly tick. */ + var built = Regex.Replace(QueryStoreTopDaily.GcBuiltSql, @"\s+", " "); + Assert.StartsWith("DELETE FROM collect.query_store_top_daily_built", built.TrimStart(), StringComparison.Ordinal); + Assert.Contains("RETURNING server_id, day", built, StringComparison.Ordinal); + + var summary = Regex.Replace(QueryStoreTopDaily.GcSummarySql, @"\s+", " "); + Assert.Contains("DELETE FROM collect.query_store_top_daily WHERE server_id = $1::integer AND day = $2::date;", summary, StringComparison.Ordinal); + Assert.DoesNotContain(" OR ", summary, StringComparison.Ordinal); + Assert.DoesNotContain("NOT IN", summary, StringComparison.Ordinal); + } + + [Fact] + public void TheRungIsRegisteredInADenseLadder_AtVersion161() + { + var versions = PgMigrations.Scripts.Select(s => s.Version).ToList(); + + Assert.Equal(RungName, Rung.Name); + Assert.Equal(161, Rung.Version); + /* No longer the top rung: V162 (the slow-read record) landed above it. */ + Assert.True(Rung.Version < StorageVersion.SchemaVersion); + Assert.Equal(StorageVersion.SchemaVersion, PgMigrations.Scripts[^1].Version); + Assert.Equal(versions.Distinct().OrderBy(v => v), versions); + Assert.Contains(Rung.Version - 1, versions); + } + + [Fact] + public void BothTablesArePlainSchemaQualifiedTables_NotHypertables() + { + var sql = Statements(); + + Assert.Contains("CREATE TABLE IF NOT EXISTS collect." + Daily + "\n", sql, StringComparison.Ordinal); + Assert.Contains("CREATE TABLE IF NOT EXISTS collect." + Built + "\n", sql, StringComparison.Ordinal); + Assert.DoesNotContain("create_hypertable", sql, StringComparison.OrdinalIgnoreCase); + Assert.DoesNotContain("timescaledb", sql, StringComparison.OrdinalIgnoreCase); + + /* Every object is schema-qualified: the migrate session's search_path puts collect first, so a bare name would + still land in collect, but the next reader should not have to know that. */ + foreach (var match in Regex.Matches(sql, @"(?:CREATE TABLE IF NOT EXISTS|ON)\s+(\S+)").Cast()) + { + Assert.StartsWith("collect.", match.Groups[1].Value, StringComparison.Ordinal); + } + } + + [Fact] + public void TheUniqueIndex_IsNullsNotDistinct_OnExactlyTheIdentityKey() + { + var sql = Statements(); + var index = Regex.Match(sql, @"CREATE UNIQUE INDEX IF NOT EXISTS ux_query_store_top_daily\s+ON collect\." + Daily + @"\s*\((?[^)]*)\)\s*(?[^;]*);"); + + Assert.True(index.Success, "the rung has no unique index named ux_query_store_top_daily on the summary table"); + var columns = index.Groups["cols"].Value.Split(',').Select(c => c.Trim()).Where(c => c.Length > 0).ToArray(); + Assert.Equal(KeyColumns, columns); + Assert.Equal("NULLS NOT DISTINCT", index.Groups["tail"].Value.Trim()); + } + + [Fact] + public void TheSummaryTable_HoldsTheExactlyRecombinableParts_AndTheRest() + { + var columns = ColumnsOf(Rung.Sql, "collect." + Daily); + var names = columns.Select(c => c.Name).ToList(); + + var expected = new List(KeyColumns) { "interval_rows", "execution_count_sum" }; + foreach (var column in AveragedColumns) + { + expected.Add(column + "_sum"); + expected.Add(column + "_n"); + } + + expected.AddRange(new[] { "last_execution_time_max", "query_plan_hash_max", "first_execution_time_min" }); + Assert.Equal(expected, names); + + var types = columns.ToDictionary(c => c.Name, c => c.Type); + Assert.Equal("bigint", types["interval_rows"]); + Assert.Equal("numeric", types["execution_count_sum"]); + foreach (var column in AveragedColumns) + { + Assert.Equal("numeric", types[column + "_sum"]); + Assert.Equal("bigint", types[column + "_n"]); + } + + /* Counts are NOT NULL (a day with no value has a zero count), sums are nullable (no value, no sum). */ + var sql = Statements(); + Assert.Contains("interval_rows bigint NOT NULL", sql, StringComparison.Ordinal); + foreach (var column in AveragedColumns) + { + Assert.Contains(column + "_n bigint NOT NULL", sql, StringComparison.Ordinal); + Assert.DoesNotContain(column + "_sum numeric NOT NULL", sql, StringComparison.Ordinal); + } + } + + [Fact] + public void EverySharedColumn_HasTheWideTablesType() + { + var wide = ColumnsOf(WideSql(), "collect.query_store_interval_wide").ToDictionary(c => c.Name, c => c.Type); + var daily = ColumnsOf(Rung.Sql, "collect." + Daily).ToDictionary(c => c.Name, c => c.Type); + + /* The key columns that the wide table also has, by name; day is the one new date. */ + foreach (var column in KeyColumns.Where(c => c != "day")) + { + Assert.True(wide.ContainsKey(column), column + " is not a wide-table column"); + Assert.Equal(wide[column], daily[column]); + } + + /* The three aggregates over a wide column take that column's type. */ + Assert.Equal(wide["query_plan_hash"], daily["query_plan_hash_max"]); + Assert.Equal(wide["last_execution_time"], daily["last_execution_time_max"]); + Assert.Equal(wide["first_execution_time"], daily["first_execution_time_min"]); + Assert.Equal("date", daily["day"]); + } + + [Fact] + public void TheBuiltTable_RecordsOneRowPerServerAndDay() + { + var columns = ColumnsOf(Rung.Sql, "collect." + Built); + + Assert.Equal( + new[] { ("server_id", "integer"), ("day", "date"), ("pass", "smallint"), ("built_at", "timestamp"), ("source_rows", "bigint") }, + columns.ToArray()); + Assert.Matches(@"PRIMARY KEY \(server_id, day\)", Statements()); + Assert.Contains("pass smallint NOT NULL", Statements(), StringComparison.Ordinal); + Assert.Contains("built_at timestamp NOT NULL", Statements(), StringComparison.Ordinal); + Assert.Contains("source_rows bigint NOT NULL", Statements(), StringComparison.Ordinal); + } + + [Fact] + public void TheRungNeedsNoGrant_BecauseTheCollectSchemaGrantsCoverANewTable() + { + var sql = Statements(); + + Assert.DoesNotContain("GRANT", sql, StringComparison.OrdinalIgnoreCase); + Assert.DoesNotContain("REVOKE", sql, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public void TheProbeCarriesTheTableAsAnArm_AndMapsItsRungOnce() + { + var probe = ViewerDataService.StoreSchemaProbeSql.Replace("\r\n", "\n", StringComparison.Ordinal); + var arm = $"to_regclass('collect.{Daily}') IS NOT NULL"; + Assert.Contains(arm, probe, StringComparison.Ordinal); + Assert.True(probe.IndexOf(arm, StringComparison.Ordinal) < probe.IndexOf("collect.slow_reads", StringComparison.Ordinal), + "the newer rung's sentinel follows this one"); + + var viewer = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.cs"); + Assert.Contains($"reader.GetBoolean({ProbeOrdinal})", viewer, StringComparison.Ordinal); + + var method = typeof(ViewerDataService).GetMethod("MapProbedSchemaVersion", BindingFlags.NonPublic | BindingFlags.Static)!; + var arity = method.GetParameters().Length; + Assert.True(ProbeOrdinal < arity - 1, "a newer rung's sentinel follows this one"); + Assert.Equal("hasQueryStoreTopDaily", method.GetParameters()[ProbeOrdinal].Name); + + var all = Enumerable.Range(0, arity).Select(i => (object)(i <= ProbeOrdinal)).ToArray(); + Assert.Equal(Rung.Version, (int)method.Invoke(null, all)!); + + var behind = (object[])all.Clone(); + behind[ProbeOrdinal] = false; + Assert.Equal(Rung.Version - 1, (int)method.Invoke(null, behind)!); + + var thisArm = viewer.IndexOf("if (hasQueryStoreTopDaily)", StringComparison.Ordinal); + var nextArm = viewer.IndexOf("if (hasSlowReads)", StringComparison.Ordinal); + var previousArm = viewer.IndexOf("if (hasCollectorRunAt)", StringComparison.Ordinal); + Assert.True(thisArm >= 0, "no sentinel arm: a fully-migrated store would map one rung short"); + Assert.True(nextArm >= 0 && nextArm < thisArm, "the newer rung's arm sits above this one's"); + Assert.True(thisArm < previousArm, "this arm sits above the previous rung's, so a current store maps one rung short"); + Assert.Contains($"return {Rung.Version};", viewer[thisArm..previousArm], StringComparison.Ordinal); + + /* The V71 finding: the table is named in the probe line and nowhere in the arm's prose. The comment block sits + ABOVE the `if`, and this probe line has no information_schema in it, so the prose must not repeat the name. */ + var armProseStart = viewer.LastIndexOf("/* V161 (#5094)", thisArm, StringComparison.Ordinal); + Assert.True(armProseStart >= 0, "the arm has no comment block saying why it exists"); + Assert.DoesNotContain(Daily, viewer[armProseStart..thisArm], StringComparison.Ordinal); + } + + private static async Task MigratedAsync(ScratchPostgres scratch, System.Threading.CancellationToken ct) + { + var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + return connection; + } + + private static async Task ScalarAsync(NpgsqlConnection connection, string sql, System.Threading.CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + return await command.ExecuteScalarAsync(ct); + } + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, System.Threading.CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(ct); + } + + /// Everything that describes the two tables, as one string, so a test can say "unchanged" in one assertion. + private const string ShapeSql = + "SELECT (SELECT string_agg(table_name || '.' || column_name || ':' || data_type || ':' || is_nullable, ',' ORDER BY table_name, ordinal_position) FROM information_schema.columns WHERE table_schema = 'collect' AND table_name IN ('query_store_top_daily', 'query_store_top_daily_built'))" + + " || '|' || (SELECT string_agg(indexname || ':' || indexdef, ';' ORDER BY indexname) FROM pg_indexes WHERE schemaname = 'collect' AND tablename IN ('query_store_top_daily', 'query_store_top_daily_built'))" + + " || '|' || (SELECT count(*) FROM pg_trigger WHERE tgrelid IN ('collect.query_store_top_daily'::regclass, 'collect.query_store_top_daily_built'::regclass) AND NOT tgisinternal)"; + + [Fact] + public async Task MigratingAFreshStore_CreatesBothTablesAndTheUniqueIndex_AsPlainTables() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + + var bodySucceeded = false; + try + { + Assert.Equal(StorageVersion.SchemaVersion, Convert.ToInt32(await ScalarAsync(connection, "SELECT MAX(version) FROM darling_schema_version", ct))); + Assert.True(await ScalarAsync(connection, "SELECT to_regclass('collect.query_store_top_daily') IS NOT NULL AND to_regclass('collect.query_store_top_daily_built') IS NOT NULL", ct) is true); + + /* Plain relations (relkind 'r'), not hypertables, in the collect schema. */ + Assert.Equal(2L, await ScalarAsync(connection, + "SELECT count(*) FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'collect' AND c.relname IN ('query_store_top_daily', 'query_store_top_daily_built') AND c.relkind = 'r'", ct)); + /* Plain-ness is asked of the catalog only when the store has no timescaledb: the hypertables view does not exist + there, and a relation in an OR still fails the parse. With the extension, neither table may be a hypertable. */ + var hasTimescale = await ScalarAsync(connection, "SELECT EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'timescaledb')", ct) is true; + if (hasTimescale) + { + Assert.True(await ScalarAsync(connection, + "SELECT NOT EXISTS (SELECT 1 FROM timescaledb_information.hypertables WHERE hypertable_schema = 'collect' AND hypertable_name IN ('query_store_top_daily', 'query_store_top_daily_built'))", ct) is true); + } + else + { + Assert.True(await ScalarAsync(connection, + "SELECT NOT EXISTS (SELECT 1 FROM pg_inherits i WHERE i.inhparent IN ('collect.query_store_top_daily'::regclass, 'collect.query_store_top_daily_built'::regclass) OR i.inhrelid IN ('collect.query_store_top_daily'::regclass, 'collect.query_store_top_daily_built'::regclass))", ct) is true); + } + + /* The unique index exists, is unique, and is NULLS NOT DISTINCT on the key. */ + var indexDef = (string?)await ScalarAsync(connection, + "SELECT indexdef FROM pg_indexes WHERE schemaname = 'collect' AND tablename = 'query_store_top_daily' AND indexname = 'ux_query_store_top_daily'", ct); + Assert.NotNull(indexDef); + Assert.Contains("UNIQUE INDEX", indexDef, StringComparison.Ordinal); + Assert.Contains("NULLS NOT DISTINCT", indexDef, StringComparison.Ordinal); + Assert.Contains("(server_id, day, database_name, query_id, plan_id, query_hash, execution_type_desc, replica_role, module_name)", indexDef, StringComparison.Ordinal); + + /* The built table's key is one row per server and day. */ + Assert.Equal("server_id,day", await ScalarAsync(connection, + "SELECT string_agg(a.attname, ',' ORDER BY k.ord) FROM pg_constraint c CROSS JOIN LATERAL unnest(c.conkey) WITH ORDINALITY AS k(attnum, ord) JOIN pg_attribute a ON a.attrelid = c.conrelid AND a.attnum = k.attnum WHERE c.conrelid = 'collect.query_store_top_daily_built'::regclass AND c.contype = 'p'", ct)); + + /* The live column types equal the wide table's for every shared column. */ + Assert.Equal(0L, await ScalarAsync(connection, + "SELECT count(*) FROM information_schema.columns d JOIN information_schema.columns w ON w.table_schema = 'collect' AND w.table_name = 'query_store_interval_wide' AND w.column_name = d.column_name WHERE d.table_schema = 'collect' AND d.table_name = 'query_store_top_daily' AND d.data_type <> w.data_type", ct)); + + /* NULLS NOT DISTINCT in effect: two rows whose nullable key columns are all NULL are one group. */ + const string Insert = + "INSERT INTO collect.query_store_top_daily (server_id, day, interval_rows, avg_duration_us_n, avg_cpu_time_us_n, avg_logical_io_reads_n, avg_logical_io_writes_n, avg_physical_io_reads_n, avg_rowcount_n) VALUES (1, DATE '2026-01-02', 1, 0, 0, 0, 0, 0, 0)"; + await ExecAsync(connection, Insert, ct); + var duplicate = await Assert.ThrowsAsync(() => ExecAsync(connection, Insert, ct)); + Assert.Equal("23505", duplicate.SqlState); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + [Fact] + public async Task TheReadOnlyRoles_CanSelectBothTables_ThroughTheCollectSchemaGrants() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + + /* Roles are cluster-wide, so each run mints its own pair. */ + var suffix = Guid.NewGuid().ToString("N")[..8]; + var roles = new[] { "qstd_viewer_" + suffix, "qstd_mcp_" + suffix }; + + var bodySucceeded = false; + try + { + /* The provisioning grant model for the read-only roles (DarlingManagedRoles sections 2 and 6): SELECT on + every table in collect, as it is re-run after each migration pass. The rung adds no grant of its own. */ + foreach (var role in roles) + { + await ExecAsync(connection, + $"CREATE ROLE {role} NOLOGIN NOSUPERUSER; GRANT USAGE ON SCHEMA collect TO {role}; GRANT SELECT ON ALL TABLES IN SCHEMA collect TO {role};", ct); + } + + foreach (var role in roles) + { + await ExecAsync(connection, $"SET ROLE {role}", ct); + var roleBodySucceeded = false; + try + { + Assert.Equal(0L, await ScalarAsync(connection, "SELECT count(*) FROM collect.query_store_top_daily", ct)); + Assert.Equal(0L, await ScalarAsync(connection, "SELECT count(*) FROM collect.query_store_top_daily_built", ct)); + + var refused = await Assert.ThrowsAsync(() => ExecAsync(connection, + "DELETE FROM collect.query_store_top_daily", ct)); + Assert.Equal("42501", refused.SqlState); + roleBodySucceeded = true; + } + finally + { + /* RESET ROLE is a session setting: it has to run on the very session that took SET ROLE. */ + await LiveStoreCleanup.RunOwnedAsync(roleBodySucceeded, async () => + await ExecAsync(connection, "RESET ROLE", ct)); + } + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (cleanup, cleanupCt) => + { + foreach (var role in roles) + { + await ExecAsync(cleanup, $"DROP OWNED BY {role}", cleanupCt); + await ExecAsync(cleanup, $"DROP ROLE IF EXISTS {role}", cleanupCt); + } + }); + } + } + + [Fact] + public async Task ASecondMigrateRun_IsANoOp_AndTheRungsSqlRunAgain_ChangesNothing_AndKeepsItsRows() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = await MigratedAsync(scratch, ct); + + var bodySucceeded = false; + try + { + await ExecAsync(connection, + "INSERT INTO collect.query_store_top_daily (server_id, day, database_name, query_id, interval_rows, avg_duration_us_n, avg_cpu_time_us_n, avg_logical_io_reads_n, avg_logical_io_writes_n, avg_physical_io_reads_n, avg_rowcount_n) VALUES (1, DATE '2026-01-02', 'db_a', 7, 3, 3, 3, 3, 3, 3, 3)", ct); + await ExecAsync(connection, + "INSERT INTO collect.query_store_top_daily_built (server_id, day, pass, built_at, source_rows) VALUES (1, DATE '2026-01-02', 1, TIMESTAMP '2026-01-03 02:00:00', 3)", ct); + + var shapeBefore = await ScalarAsync(connection, ShapeSql, ct); + var versionsBefore = await ScalarAsync(connection, "SELECT string_agg(version::text, ',' ORDER BY version) FROM darling_schema_version", ct); + + await PgMigrations.MigrateAsync(connection, ct); + await ExecAsync(connection, Rung.Sql, ct); + await ExecAsync(connection, Rung.Sql, ct); + + Assert.Equal(shapeBefore, await ScalarAsync(connection, ShapeSql, ct)); + Assert.Equal(versionsBefore, await ScalarAsync(connection, "SELECT string_agg(version::text, ',' ORDER BY version) FROM darling_schema_version", ct)); + Assert.Equal(1L, await ScalarAsync(connection, "SELECT count(*) FROM collect.query_store_top_daily", ct)); + Assert.Equal(1L, await ScalarAsync(connection, "SELECT count(*) FROM collect.query_store_top_daily_built", ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } +} diff --git a/Darling/Darling.Tests/QueryStoreTopLiteralEndStraddleLiveTests.cs b/Darling/Darling.Tests/QueryStoreTopLiteralEndStraddleLiveTests.cs index 68c53f049e..eebf380e8f 100644 --- a/Darling/Darling.Tests/QueryStoreTopLiteralEndStraddleLiveTests.cs +++ b/Darling/Darling.Tests/QueryStoreTopLiteralEndStraddleLiveTests.cs @@ -97,8 +97,13 @@ clause except 4 passes and the literal-end reads below isolate clause 4 alone. * async Task<(long SeqScan, long IdxScan)> WideTableScanCountsAsync() { + await QuiesceScanStatisticsAsync(connection, ct); + /* #5571: the wide table is partitioned by day, so the parent has no counters of its own; a scan of it is counted on + the leaf partitions it reads (the legacy table and the daily ones), summed here. */ await using var command = new NpgsqlCommand( - "SELECT COALESCE(seq_scan, 0), COALESCE(idx_scan, 0) FROM pg_stat_user_tables WHERE relname = 'query_store_interval_wide'", connection); + "SELECT COALESCE(SUM(s.seq_scan), 0)::bigint, COALESCE(SUM(s.idx_scan), 0)::bigint " + + "FROM pg_partition_tree('collect.query_store_interval_wide'::regclass) AS p " + + "JOIN pg_stat_user_tables AS s ON s.relid = p.relid WHERE p.isleaf", connection); await using var reader = await command.ExecuteReaderAsync(ct); if (!await reader.ReadAsync(ct)) { @@ -199,10 +204,10 @@ private static async Task SeedStraddleAsync(DarlingCollectorRunner runner, Cance /* Clause 3's own floor control: UseTable's tableFloor is MIN(first_execution_time) across the WHOLE server, not just this identity, and clause 3 refuses unless raw's chunk floor or WindowStart minus - IntervalSpanMargin ("skewFloor") reaches at or past it. Left with only the 7001 identity below, the - table's floor sits at day0 (WindowStart + 1h) - AFTER skewFloor (WindowStart - 1 day) - so clause 3 + PurgeEdgeMargin ("edgeFloor") reaches at or past it. Left with only the 7001 identity below, the + table's floor sits at day0 (WindowStart + 1h) - AFTER edgeFloor (WindowStart - 26 h) - so clause 3 alone refuses even the open-end positive control, before clause 4 is ever reached. A second, - unrelated identity anchored at or before skewFloor pulls the server's table floor down there, so + unrelated identity anchored at or before edgeFloor pulls the server's table floor down there, so clause 3 passes and only clause 4 (LiteralEnd vs applied_through) can still refuse the straddle read. */ var floorAnchorFirst = WindowStart.AddDays(-2); var floorAnchor = new QueryStoreCollector.Row @@ -290,6 +295,36 @@ private static async Task ScalarBoolAsync(NpgsqlConnection connection, str return (bool)(await command.ExecuteScalarAsync(ct))!; } + /// + /// The scan counters in pg_stat_user_tables are a backend's to report, and a backend reports late: a read's + /// connection closes before its backend exits, and the backend flushes its counters as it exits (a busy runner + /// stretches that, and a backend that reported a moment ago holds new counts for up to 10 seconds). A snapshot + /// read straight after a read therefore missed that read's scans, and the next snapshot saw them + /// (Expected (2, 5) Actual (2, 6)). Wait until no other client backend is left in this database (every read here + /// runs on its own unpooled connection, and a backend leaves pg_stat_activity only after its exit flush), then + /// flush this connection's own pending counters, so a snapshot holds every scan made before it, and only those. + /// + private static async Task QuiesceScanStatisticsAsync(NpgsqlConnection connection, CancellationToken ct) + { + var deadline = DateTime.UtcNow.AddSeconds(30); + long others; + do + { + await using var command = new NpgsqlCommand( + "SELECT COUNT(*) FROM pg_stat_activity WHERE datname = current_database() AND backend_type = 'client backend' AND pid <> pg_backend_pid()", connection); + others = (long)(await command.ExecuteScalarAsync(ct))!; + if (others > 0) + { + Assert.True(DateTime.UtcNow < deadline, $"{others} other client backend(s) are still connected 30 seconds after the reads ended, so their scan counters are not final"); + await Task.Delay(50, ct); + } + } + while (others > 0); + + await using var flush = new NpgsqlCommand("SELECT pg_stat_force_next_flush()", connection); + await flush.ExecuteScalarAsync(ct); + } + private static async Task OpenMigratedAsync(ScratchPostgres scratch, CancellationToken ct) { var connection = new NpgsqlConnection(scratch.ConnectionString); diff --git a/Darling/Darling.Tests/QueryStoreTopMcpLiveTests.cs b/Darling/Darling.Tests/QueryStoreTopMcpLiveTests.cs index 4f4ce17b50..965bf76c10 100644 --- a/Darling/Darling.Tests/QueryStoreTopMcpLiveTests.cs +++ b/Darling/Darling.Tests/QueryStoreTopMcpLiveTests.cs @@ -44,6 +44,52 @@ public sealed class QueryStoreTopMcpLiveTests private static string? BaseConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + [Fact] + public async Task ATableReadThatFaultsAfterTheGateSaidTable_AnswersFromRaw_AndNotesFallbackRaw() + { + var baseCs = BaseConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(baseCs), "Set DARLING_TEST_PG to a Postgres connection string to run the #5097 fallback live test."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseCs!, ct); + await using var connection = await OpenMigratedAsync(scratch, ct); + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var runner = new DarlingCollectorRunner(postgres, new CollectorDeltaCalculator()); + + await QueryStoreIntervalWideGridLiveTests.SeedGridAsync(runner, ServerId, WindowStart, ct); + await ForceFilledSinceAsync(connection, WindowStart.AddDays(-1), ct); + await RegisterServerAsync(connection, ct); + var appliedThrough = await ScalarDateTimeAsync(connection, + "SELECT applied_through FROM collect.query_store_interval_wide_coverage WHERE server_id = @server_id", ct); + var queryEnd = appliedThrough > WindowEnd ? appliedThrough : WindowEnd; + + var logger = new CapturingTestLogger(); + using var scope = ReadScope.Open(logger); + + var viaTable = await DarlingDataReader.GetQueryStoreTopWithReachAsync(postgres, ServerId, WindowStart, queryEnd, TestTop, null, null, null, ct); + Assert.NotNull(viaTable.Table); + Assert.Null(scope.Fallback); + + DarlingDataReader.TestOnlyTableReadFault.Value = () => new InvalidOperationException("injected table read fault"); + try + { + var viaFallback = await DarlingDataReader.GetQueryStoreTopWithReachAsync(postgres, ServerId, WindowStart, queryEnd, TestTop, null, null, null, ct); + + Assert.Null(viaFallback.Table); + var rawKeys = await RawTopKeysAsync(connection, WindowStart, queryEnd, null, null, ct); + Assert.True(rawKeys.Count > 0, "the seed produced no raw rows; the comparison would be vacuous"); + Assert.Equal( + rawKeys.OrderBy(k => k).ToList(), + viaFallback.Rows.Select(r => (r.DatabaseName, r.QueryId, r.PlanId, r.ExecutionTypeDesc, r.ReplicaRole, r.TotalExecutions)).OrderBy(k => k).ToList()); + Assert.Equal(ReadFallback.FallbackRaw, scope.Fallback); + Assert.True(logger.CountAtLevel(Microsoft.Extensions.Logging.LogLevel.Warning) >= 1, logger.Joined); + } + finally + { + DarlingDataReader.TestOnlyTableReadFault.Value = null; + } + } + [Fact] public async Task TheTableRead_EqualsRaw_UnfilteredAndFiltered_EndToEnd_ThroughMcpAndWebDispatch_AndTheLiveGate() { @@ -302,9 +348,10 @@ private static async Task AssertRawEqualsTableAsync( raw.Parameters.Add(new NpgsqlParameter { TypedValue = DateTime.SpecifyKind(windowStart, DateTimeKind.Unspecified) }); raw.Parameters.Add(new NpgsqlParameter { TypedValue = DateTime.SpecifyKind(WindowEnd, DateTimeKind.Unspecified) }); raw.Parameters.Add(new NpgsqlParameter { TypedValue = TestTop }); - AddNullableText(raw, null); + raw.Parameters.Add(PerformanceMonitor.Darling.Storage.DatabaseFilter.All.Parameter()); AddNullableText(raw, executionType); AddNullableText(raw, moduleName); + raw.Parameters.Add(new NpgsqlParameter { TypedValue = PerformanceMonitor.Darling.Storage.TopFill.FirstCandidates(TestTop) }); /* #5313: the round's candidate limit, bound last */ await raw.ExecuteNonQueryAsync(ct); } @@ -314,9 +361,10 @@ private static async Task AssertRawEqualsTableAsync( table.Parameters.Add(new NpgsqlParameter { TypedValue = DateTime.SpecifyKind(tableStart, DateTimeKind.Unspecified) }); table.Parameters.Add(new NpgsqlParameter { TypedValue = DateTime.SpecifyKind(WindowEnd, DateTimeKind.Unspecified) }); table.Parameters.Add(new NpgsqlParameter { TypedValue = TestTop }); - AddNullableText(table, null); + table.Parameters.Add(PerformanceMonitor.Darling.Storage.DatabaseFilter.All.Parameter()); AddNullableText(table, executionType); AddNullableText(table, moduleName); + table.Parameters.Add(new NpgsqlParameter { TypedValue = PerformanceMonitor.Darling.Storage.TopFill.FirstCandidates(TestTop) }); /* #5313: the round's candidate limit, bound last */ await table.ExecuteNonQueryAsync(ct); } @@ -341,9 +389,10 @@ private static async Task AssertRawEqualsTableAsync( command.Parameters.Add(new NpgsqlParameter { TypedValue = DateTime.SpecifyKind(windowStart, DateTimeKind.Unspecified) }); command.Parameters.Add(new NpgsqlParameter { TypedValue = DateTime.SpecifyKind(windowEnd, DateTimeKind.Unspecified) }); command.Parameters.Add(new NpgsqlParameter { TypedValue = TestTop }); - AddNullableText(command, null); + command.Parameters.Add(PerformanceMonitor.Darling.Storage.DatabaseFilter.All.Parameter()); AddNullableText(command, executionType); AddNullableText(command, moduleName); + command.Parameters.Add(new NpgsqlParameter { TypedValue = PerformanceMonitor.Darling.Storage.TopFill.FirstCandidates(TestTop) }); /* #5313: the round's candidate limit, bound last */ await using var reader = await command.ExecuteReaderAsync(ct); while (await reader.ReadAsync(ct)) { diff --git a/Darling/Darling.Tests/QueryStoreTopWebDefaultTests.cs b/Darling/Darling.Tests/QueryStoreTopWebDefaultTests.cs index ce7f030bb3..0d6b53d771 100644 --- a/Darling/Darling.Tests/QueryStoreTopWebDefaultTests.cs +++ b/Darling/Darling.Tests/QueryStoreTopWebDefaultTests.cs @@ -34,7 +34,9 @@ public void GetQueryStoreTop_WebRowKeepsTheOld2000CharacterPreview() { var web = StripComments(ReadSource(WebEndpoints)); - var marker = "[\"get_query_store_top\"] = (c, pg, an) => DarlingMcpDataTools.GetQueryStoreTop("; + /* #5245: the entry reads the chosen databases first (DatabaseNames(c) is { } databases ? ... : refusal), so the + marker is the call itself and the row ends at the refusal arm. */ + var marker = "? DarlingMcpDataTools.GetQueryStoreTop("; var start = web.IndexOf(marker, StringComparison.Ordinal); Assert.True(start >= 0, "get_query_store_top's /api/read row was not found (renamed or moved?)."); var end = web.IndexOf("),\r\n", start, StringComparison.Ordinal); diff --git a/Darling/Darling.Tests/QueryStoreTopWindowTests.cs b/Darling/Darling.Tests/QueryStoreTopWindowTests.cs index 8c287ea33e..a2986adb2e 100644 --- a/Darling/Darling.Tests/QueryStoreTopWindowTests.cs +++ b/Darling/Darling.Tests/QueryStoreTopWindowTests.cs @@ -39,19 +39,22 @@ public class QueryStoreTopWindowTests "Darling", "PerformanceMonitor.Darling.Service", "Mcp", "DarlingMcpDataTools.cs")); /// - /// The floor probe is bounded on BOTH sides of the window. Bounding the partitioning column is what lets - /// TimescaleDB exclude chunks; an unbounded MIN would read the retention window to answer a question - /// about it. + /// The floor probe walks the table's (server_id, collection_time) index in time order and stops at the + /// first row, so it reads one row, not the retention window, while being bounded only by the window's end. A + /// floor bounded below by the window's start called a quiet start a cut. The start bounds only the existence + /// check that keeps an empty window null. /// [Fact] - public void TheWindowFloorProbe_IsBoundedOnBothSides() + public void TheWindowFloorProbe_WalksTheIndexFromTheOldestRow_AndChecksTheWindowApart() { var sql = DarlingDataReader.QueryStoreWindowFloorSql; - Assert.Contains("MIN(collection_time)", sql, StringComparison.Ordinal); - Assert.Contains("FROM query_store_stats", sql, StringComparison.Ordinal); - Assert.Contains("collection_time >=", sql, StringComparison.Ordinal); - Assert.Contains("collection_time <=", sql, StringComparison.Ordinal); + Assert.Contains("FROM collect.query_store_stats AS f", sql, StringComparison.Ordinal); + Assert.Contains("AND f.collection_time <= $3", sql, StringComparison.Ordinal); + Assert.DoesNotContain("f.collection_time >=", sql, StringComparison.Ordinal); + Assert.Contains("ORDER BY f.collection_time", sql, StringComparison.Ordinal); + Assert.Contains("LIMIT 1", sql, StringComparison.Ordinal); + Assert.Contains("AND w.collection_time >= $2", sql, StringComparison.Ordinal); } /// diff --git a/Darling/Darling.Tests/QueryStoreTrendIntervalRateTests.cs b/Darling/Darling.Tests/QueryStoreTrendIntervalRateTests.cs index c92d2386e6..353b84aae4 100644 --- a/Darling/Darling.Tests/QueryStoreTrendIntervalRateTests.cs +++ b/Darling/Darling.Tests/QueryStoreTrendIntervalRateTests.cs @@ -54,7 +54,7 @@ public sealed class QueryStoreTrendIntervalRateTests { yield return ("ViewerDataService.QueryStoreDurationTrendSql", ViewerDataService.QueryStoreDurationTrendSql); yield return ("ViewerDataService.QueryStoreDurationTrendTableSql", ViewerDataService.QueryStoreDurationTrendTableSql); - yield return ("DarlingTrendReader.QueryStoreDurationTrendSql", DarlingTrendReader.QueryStoreDurationTrendSql); + yield return ("DarlingTrendReader.QueryStoreDurationTrendFilteredSql", DarlingTrendReader.QueryStoreDurationTrendFilteredSql); } private static IEnumerable<(string Name, string Sql)> AllCopies() diff --git a/Darling/Darling.Tests/QueryStoreTrendRoutingTests.cs b/Darling/Darling.Tests/QueryStoreTrendRoutingTests.cs index ea01d10cbf..0a2d484e13 100644 --- a/Darling/Darling.Tests/QueryStoreTrendRoutingTests.cs +++ b/Darling/Darling.Tests/QueryStoreTrendRoutingTests.cs @@ -268,8 +268,8 @@ depth plus a day. The +1 day ceiling is the engine's INTERVAL_LENGTH_MINUTES max [Fact] public void RawFallbackSql_KeepsItsSlab_Deliberately() { - Assert.Contains("collection_time >= $2 - interval '1 day'", DarlingTrendReader.QueryStoreDurationTrendSql, StringComparison.Ordinal); - Assert.Contains("collection_time <= $3 + interval '30 days'", DarlingTrendReader.QueryStoreDurationTrendSql, StringComparison.Ordinal); + Assert.Contains("collection_time >= $2 - interval '1 day'", DarlingTrendReader.QueryStoreDurationTrendFilteredSql, StringComparison.Ordinal); + Assert.Contains("collection_time <= $3 + interval '30 days'", DarlingTrendReader.QueryStoreDurationTrendFilteredSql, StringComparison.Ordinal); Assert.Contains("collection_time >= $2 - interval '1 day'", ViewerDataService.QueryStoreDurationTrendSql, StringComparison.Ordinal); Assert.Contains("collection_time <= $3 + interval '30 days'", ViewerDataService.QueryStoreDurationTrendSql, StringComparison.Ordinal); } @@ -285,7 +285,7 @@ public void RawFallbackSql_KeepsItsSlab_Deliberately() [Fact] public void BothApps_ServeTheBuilderOutput_DifferingOnlyByTheDatabaseFilter() { - Assert.Equal(QueryStoreTrendRouting.BuildRollupTrendSql(withDatabaseFilter: false), DarlingTrendReader.QueryStoreDurationTrendRollupSql); + Assert.Equal(QueryStoreTrendRouting.BuildRollupTrendSql(withDatabaseFilter: true), DarlingTrendReader.QueryStoreDurationTrendRollupFilteredSql); Assert.Equal(QueryStoreTrendRouting.BuildRollupTrendSql(withDatabaseFilter: true), ViewerDataService.QueryStoreDurationTrendRollupSql); /* Strip the filter lines from the viewer's copy and the two must be byte-identical. */ @@ -293,12 +293,13 @@ public void BothApps_ServeTheBuilderOutput_DifferingOnlyByTheDatabaseFilter() var stripped = viewerSql .Replace("\n AND ($5::text[] IS NULL OR database_name = ANY($5))", "", StringComparison.Ordinal) .Replace("\n AND ($5::text[] IS NULL OR database_name = ANY($5))", "", StringComparison.Ordinal); - Assert.Equal(DarlingTrendReader.QueryStoreDurationTrendRollupSql, stripped); + Assert.Equal(QueryStoreTrendRouting.BuildRollupTrendSql(withDatabaseFilter: false), stripped); /* Every arm of the viewer's copy carries the filter: the rollup scan, the identified-interval arm, and the legacy arm — three occurrences, or one arm silently ignores the user's selection. */ var occurrences = viewerSql.Split("$5::text[] IS NULL").Length - 1; Assert.Equal(3, occurrences); - Assert.DoesNotContain("$5", DarlingTrendReader.QueryStoreDurationTrendRollupSql, StringComparison.Ordinal); + Assert.Equal(viewerSql, DarlingTrendReader.QueryStoreDurationTrendRollupFilteredSql); + Assert.DoesNotContain("$5", QueryStoreTrendRouting.BuildRollupTrendSql(withDatabaseFilter: false), StringComparison.Ordinal); } } diff --git a/Darling/Darling.Tests/QueryStoreWideFleetEligibilityLiveTests.cs b/Darling/Darling.Tests/QueryStoreWideFleetEligibilityLiveTests.cs new file mode 100644 index 0000000000..4423af478f --- /dev/null +++ b/Darling/Darling.Tests/QueryStoreWideFleetEligibilityLiveTests.cs @@ -0,0 +1,227 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4605: a fleet Custom Views panel's table-eligibility check reads the store-wide chunk floors once, not once +/// per server, and answers exactly what checking each server one after another answers. +/// +/* #1776 own-store: deliberately NOT [Collection("live-postgres")]; works entirely inside its own scratch database. */ +public sealed class QueryStoreWideFleetEligibilityLiveTests +{ + private static readonly DateTime S = DateTime.SpecifyKind(DateTime.UtcNow.Date.AddDays(-6), DateTimeKind.Unspecified); + + private static async Task<(ScratchPostgres Scratch, NpgsqlDataSource Postgres, NpgsqlConnection Connection)> SeedAsync( + string baseCs, int[] eligibleIds, int? legacyId, CancellationToken ct) + { + var scratch = await ScratchPostgres.CreateAsync(baseCs, ct); + NpgsqlConnection? connection = null; + NpgsqlDataSource? postgres = null; + try + { + connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + Assert.True(await TimescaleSupport.TryEnableAsync(connection, null, ct), "TimescaleDB must be enabled on the test cluster"); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var runner = new DarlingCollectorRunner(postgres, new CollectorDeltaCalculator()); + var all = eligibleIds.Concat(legacyId is int l ? new[] { l } : Array.Empty()).ToArray(); + for (var i = 0; i < all.Length; i++) + { + var id = all[i]; + if (id == legacyId) + { + await QueryStoreIntervalWideGridLiveTests.SeedGridLegacyAsync(runner, id, S, ct); + } + else + { + await QueryStoreIntervalWideGridLiveTests.SeedGridAsync(runner, id, S, ct); + } + + await using var ins = new NpgsqlCommand( + "INSERT INTO servers (server_id, server_name, display_name, is_enabled, sql_major_version, created_date, modified_date) " + + $"VALUES ({id}, 'qsiw-fleetsrv-{(char)('a' + i)}', 'qsiw-fleetsrv-{(char)('a' + i)}', TRUE, 16, now(), now()) ON CONFLICT (server_id) DO UPDATE SET is_enabled = TRUE", connection); + await ins.ExecuteNonQueryAsync(ct); + await QueryStoreIntervalWideBelowFloorLiveTests.SeedQueryStoreLogAsync(connection, id, S.AddDays(-60), S.AddDays(4), null, null, ct); + } + + /* Different read starts: servers 0 and 3 share the latest claim, so the tie goes to the lower server_id, qsiw-fleetsrv-d. */ + for (var i = 0; i < eligibleIds.Length; i++) + { + var hours = i == 0 || i == 3 ? 14 : 12 + (i % 2); + await using var upd = new NpgsqlCommand( + $"UPDATE collect.query_store_interval_wide_coverage SET filled_since = TIMESTAMP '{S.AddHours(hours):yyyy-MM-dd HH:mm:ss}' WHERE server_id = {eligibleIds[i]}", connection); + await upd.ExecuteNonQueryAsync(ct); + } + + await using var drop = new NpgsqlCommand( + $"SELECT drop_chunks('collect.query_store_stats', older_than => TIMESTAMP '{S.AddDays(1):yyyy-MM-dd HH:mm:ss}')", connection); + await drop.ExecuteNonQueryAsync(ct); + return (scratch, postgres, connection); + } + catch + { + if (postgres is not null) + { + await postgres.DisposeAsync(); + } + + if (connection is not null) + { + await connection.DisposeAsync(); + } + + await scratch.DisposeAsync(); + throw; + } + } + + private static async Task EndAsync(NpgsqlConnection connection, CancellationToken ct) + { + await using var cmd = new NpgsqlCommand("SELECT MAX(applied_through) FROM collect.query_store_interval_wide_coverage", connection); + return (DateTime)(await cmd.ExecuteScalarAsync(ct))!; + } + + /// The pre-change answer: each server resolved one after another, re-reading the floors every time. + private static async Task<(bool Eligible, DateTime? WideStart, QueryStoreIntervalWide.WideStartBound Bound, string? SettingServer)> SerialAsync( + NpgsqlConnection connection, DateTime end, CancellationToken ct) + { + var servers = new List<(int Id, string Name)>(); + await using (var cmd = new NpgsqlCommand("SELECT server_id, server_name FROM collect.servers WHERE is_enabled ORDER BY server_id", connection)) + await using (var reader = await cmd.ExecuteReaderAsync(ct)) + { + while (await reader.ReadAsync(ct)) + { + servers.Add((reader.GetInt32(0), reader.GetString(1))); + } + } + + var wideStart = S; + var bound = QueryStoreIntervalWide.WideStartBound.Window; + string? setting = null; + foreach (var (id, name) in servers) + { + var plan = await QueryStoreIntervalWide.ResolveReadAsync(connection, id, S, end, end, TimeSpan.FromHours(12), 60, null, ct); + if (!plan.UseTable) + { + return default; + } + + if (plan.ReadStart > wideStart) + { + wideStart = plan.ReadStart; + bound = plan.StartBound; + setting = name; + } + } + + return (true, wideStart, bound, setting); + } + + [Fact] + public async Task FleetCheck_ReadsTheStoreWideFloorsOnce_AndEqualsTheSerialAnswer_AcrossRepeatedRuns() + { + var baseCs = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseCs), "Set DARLING_TEST_PG to a Postgres connection string to run the #4605 live test."); + var ct = TestContext.Current.CancellationToken; + var ids = new[] { -4605101, -4605102, -4605103, -4605104, -4605105, -4605106 }; + var (scratch, postgres, connection) = await SeedAsync(baseCs!, ids, null, ct); + await using var _s = scratch; + await using var _c = connection; + await using var _p = postgres; + var end = await EndAsync(connection, ct); + + var expected = await SerialAsync(connection, end, ct); + Assert.True(expected.Eligible); + Assert.Equal(S.AddHours(14), expected.WideStart); + Assert.Equal("qsiw-fleetsrv-d", expected.SettingServer); + + /* Scoped to this scratch database by name, so a class running in parallel against another database is + invisible to the count. */ + using var floorReads = new NpgsqlCommandCounter(scratch.DatabaseName, "raw_floor"); + using var controlReads = new NpgsqlCommandCounter(scratch.DatabaseName, "fleet_floor_count_control"); + await using (var control = postgres.CreateCommand("SELECT 1 AS fleet_floor_count_control")) + { + await control.ExecuteScalarAsync(ct); + } + + Assert.True(controlReads.Count == 1, + $"The Npgsql activity listener counted {controlReads.Count} control commands on '{scratch.DatabaseName}' instead of 1, so its floors count would be meaningless."); + + for (var run = 0; run < 20; run++) + { + var before = floorReads.Count; + var got = await DarlingWebEndpoints.ResolveQueryStoreWideEligibleAsync(postgres, null, S, end, end, ct); + Assert.Equal(expected, (got.Eligible, got.WideStart, got.Bound, got.SettingServer)); + Assert.Equal(1, floorReads.Count - before); + } + } + + [Fact] + public async Task FleetCheck_WhenTheFirstServerRefusesBeforeTheFloors_ReadsNoFloors_AndAnswersRaw() + { + var baseCs = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseCs), "Set DARLING_TEST_PG to a Postgres connection string to run the #4605 live test."); + var ct = TestContext.Current.CancellationToken; + var ids = new[] { -4605301, -4605302, -4605303 }; + var (scratch, postgres, connection) = await SeedAsync(baseCs!, ids, null, ct); + await using var _s = scratch; + await using var _c = connection; + await using var _p = postgres; + var end = await EndAsync(connection, ct); + + /* A literal end before applied_through refuses at the step ahead of the floors read. */ + var literalEnd = S.AddHours(13); + using var floorReads = new NpgsqlCommandCounter(scratch.DatabaseName, "raw_floor"); + using var controlReads = new NpgsqlCommandCounter(scratch.DatabaseName, "fleet_floor_count_control"); + await using (var control = postgres.CreateCommand("SELECT 1 AS fleet_floor_count_control")) + { + await control.ExecuteScalarAsync(ct); + } + + Assert.True(controlReads.Count == 1, + $"The Npgsql activity listener counted {controlReads.Count} control commands on '{scratch.DatabaseName}' instead of 1, so its floors count would be meaningless."); + + var got = await DarlingWebEndpoints.ResolveQueryStoreWideEligibleAsync(postgres, null, S, end, literalEnd, ct); + Assert.False(got.Eligible); + Assert.Equal(0, floorReads.Count); + } + + [Fact] + public async Task FleetCheck_WithOneServerHoldingLegacyRows_ReadsRaw_LikeTheSerialAnswer() + { + var baseCs = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseCs), "Set DARLING_TEST_PG to a Postgres connection string to run the #4605 live test."); + var ct = TestContext.Current.CancellationToken; + var ids = new[] { -4605201, -4605202, -4605203 }; + var (scratch, postgres, connection) = await SeedAsync(baseCs!, ids, -4605204, ct); + await using var _s = scratch; + await using var _c = connection; + await using var _p = postgres; + var end = await EndAsync(connection, ct); + + Assert.Equal(default, await SerialAsync(connection, end, ct)); + var got = await DarlingWebEndpoints.ResolveQueryStoreWideEligibleAsync(postgres, null, S, end, end, ct); + Assert.False(got.Eligible); + Assert.Null(got.WideStart); + } +} diff --git a/Darling/Darling.Tests/QueryStoreWideReadGuardTests.cs b/Darling/Darling.Tests/QueryStoreWideReadGuardTests.cs new file mode 100644 index 0000000000..c15307e0bb --- /dev/null +++ b/Darling/Darling.Tests/QueryStoreWideReadGuardTests.cs @@ -0,0 +1,385 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Text.Json.Nodes; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5582: the refusal of a composed Query Store panel whose wide-table read cannot finish inside the compose statement +/// timeout. The pure half (the estimate arithmetic and the message) first; the live half after it runs the refusal through +/// every path that runs a composed panel: the shared runner (behind the web endpoint) and the MCP tool +/// run_custom_view_panel. +/// +[Trait("Stage", "Guard")] +public sealed class QueryStoreWideReadGuardTests +{ + private static readonly DateTime Start = new(2026, 8, 1, 0, 0, 0, DateTimeKind.Unspecified); + + [Fact] + public void TheLimits_AreTheFieldColdRate_At75PercentOfTheStatementTimeout() + { + /* S4: 8,457,483 rows cold in 55.2 s, 6.53 us per row. The budget is 45 s, 75% of the 60 s timeout (three cold runs differed by 14%). */ + const double coldPerRow = 55.215d / 8_457_483d; + const double budget = 60d * 0.75d; + Assert.Equal("60s", ComposeLimits.StatementTimeout); + + var oneScan = budget / coldPerRow; + Assert.InRange(ComposeLimits.MaxQueryStoreWideRows, oneScan * 0.98d, oneScan); + Assert.Equal(6_800_000L, ComposeLimits.MaxQueryStoreWideRows); + + /* Two scans: 6.53 cold first scan + 0.13 hash aggregate + 1.54 warm second scan = 8.19 us per row. */ + var twoScans = budget / ((6.53d + 0.13d + 1.54d) / 1_000_000d); + Assert.InRange(ComposeLimits.MaxQueryStoreWideRowsTwoScans, twoScans * 0.98d, twoScans); + Assert.Equal(5_400_000L, ComposeLimits.MaxQueryStoreWideRowsTwoScans); + + /* The old limit, 9.4 million, takes 61.4 s cold at the field rate: over the timeout. */ + Assert.True(9_400_000d * coldPerRow > 60d, "the old limit is over the timeout at the field rate"); + Assert.True(ComposeLimits.MaxQueryStoreWideRows * coldPerRow < budget); + } + + [Fact] + public void TheWeightAndTheBaseRowBound_AreThePinnedFigures() + { + Assert.Equal(0.3d, ComposeLimits.StampRowWeight); + Assert.Equal(1_000_000L, ComposeLimits.MaxSingleScanBaseRows); + + /* 436 bytes of temp per base row against the 1 GB limit: the bound is 43% of it, and 3 million rows would pass it. */ + Assert.True(ComposeLimits.MaxSingleScanBaseRows * 436L < 1_000_000_000L * 0.5d); + Assert.True(100L * 30_000L * 436L > 1_000_000_000L); + } + + [Fact] + public void OverTheLimit_RefusesWithTheCountServersAndWindow() + { + var message = QueryStoreWideReadGuard.Refusal(new QueryStoreWideReadGuard.Estimate(12_000_000, 43), Start, Start.AddDays(1)); + + Assert.Equal( + "This panel needs about 12 million Query Store rows (43 servers, 1 day), over the limit of 6.8 million. Choose fewer servers or a shorter window.", + message); + } + + [Fact] + public void TheEstimate_ScalesWithTheCountedRange_NotTheWindowTheCallerAsked() + { + var estimate = new QueryStoreWideReadGuard.Estimate(3_000_000, 10); + + Assert.Null(QueryStoreWideReadGuard.Refusal(estimate, Start, Start.AddDays(2))); + Assert.Equal( + "This panel needs about 9 million Query Store rows (10 servers, 3 days), over the limit of 6.8 million. Choose fewer servers or a shorter window.", + QueryStoreWideReadGuard.Refusal(estimate, Start, Start.AddDays(3))); + + /* The same window with the counted range cut to the newest day: under the limit, so no refusal. */ + Assert.Null(QueryStoreWideReadGuard.Refusal(estimate, Start.AddDays(2), Start.AddDays(3))); + } + + [Fact] + public void AtOrUnderTheLimit_NeverRefuses() + { + var atLimit = new QueryStoreWideReadGuard.Estimate(ComposeLimits.MaxQueryStoreWideRows, 5); + Assert.Null(QueryStoreWideReadGuard.Refusal(atLimit, Start, Start.AddDays(1))); + Assert.NotNull(QueryStoreWideReadGuard.Refusal(new QueryStoreWideReadGuard.Estimate(ComposeLimits.MaxQueryStoreWideRows + 1, 5), Start, Start.AddDays(1))); + + /* The fleet day that took 55 s cold on the large store is refused by the one-scan limit (before the stamp covers it). */ + Assert.NotNull(QueryStoreWideReadGuard.Refusal(new QueryStoreWideReadGuard.Estimate(8_457_483, 43), Start, Start.AddDays(1))); + } + + [Fact] + public void NoEstimate_OrNoServerWithABuiltDay_NeverRefuses() + { + Assert.Null(QueryStoreWideReadGuard.Refusal(null, Start, Start.AddDays(30))); + Assert.Null(QueryStoreWideReadGuard.Refusal(new QueryStoreWideReadGuard.Estimate(0, 0), Start, Start.AddDays(30))); + Assert.Null(QueryStoreWideReadGuard.Refusal(new QueryStoreWideReadGuard.Estimate(long.MaxValue, 0), Start, Start.AddDays(30))); + } + + [Theory] + [InlineData(1, "1 server")] + [InlineData(2, "2 servers")] + public void TheServerCount_AgreesInNumber(int servers, string expected) + { + var message = QueryStoreWideReadGuard.Refusal(new QueryStoreWideReadGuard.Estimate(ComposeLimits.MaxQueryStoreWideRows * 2, servers), Start, Start.AddDays(1)); + Assert.Contains("(" + expected + ", ", message, StringComparison.Ordinal); + } + + [Theory] + [InlineData(12, "12 hours")] + [InlineData(36, "36 hours")] + [InlineData(24, "1 day")] + [InlineData(72, "3 days")] + public void TheWindow_IsWholeDaysOrHours(int hours, string expected) + { + var message = QueryStoreWideReadGuard.Refusal(new QueryStoreWideReadGuard.Estimate(ComposeLimits.MaxQueryStoreWideRows * 10, 3), Start, Start.AddHours(hours)); + Assert.Contains(", " + expected + ")", message, StringComparison.Ordinal); + } + + [Fact] + public void TheEstimateSql_TakesEachServersMostRecentBuiltDay_OfTheServersInScope_DisabledOnesIncluded() + { + var sql = QueryStoreWideReadGuard.PerServerRowsSql; + Assert.Contains("collect.query_store_top_daily_built", sql, StringComparison.Ordinal); + Assert.Contains("ORDER BY d.day DESC", sql, StringComparison.Ordinal); + Assert.Contains("LIMIT 1", sql, StringComparison.Ordinal); + /* The compiled wide read joins collect.servers with no is_enabled predicate, so the estimate counts a disabled server too. */ + Assert.DoesNotContain("is_enabled", sql, StringComparison.Ordinal); + Assert.Contains("s.server_name = ANY($1)", sql, StringComparison.Ordinal); + } + + [Fact] + public void ARankedTimeSeriesPanelThatScansTwice_GetsTheTwoScanLimit_AndTheMessageNamesIt() + { + Assert.Equal(ComposeLimits.MaxQueryStoreWideRows, QueryStoreWideReadGuard.LimitFor(false)); + Assert.Equal(ComposeLimits.MaxQueryStoreWideRowsTwoScans, QueryStoreWideReadGuard.LimitFor(true)); + var estimate = new QueryStoreWideReadGuard.Estimate(6_000_000, 43); + Assert.Null(QueryStoreWideReadGuard.Refusal(estimate, Start, Start.AddDays(1), QueryStoreWideReadGuard.LimitFor(false))); + Assert.Equal( + "This panel needs about 6 million Query Store rows (43 servers, 1 day), over the limit of 5.4 million. Choose fewer servers or a shorter window.", + QueryStoreWideReadGuard.Refusal(estimate, Start, Start.AddDays(1), QueryStoreWideReadGuard.LimitFor(true))); + } + + /* A fleet day after the build: 8.46 million wide rows in the day, the newest 2.5 hours not yet stamped. The first 21.5 hours + count at 0.3 and the tail at 1.0. */ + private static readonly QueryStoreWideReadGuard.Estimate FleetDay = new(8_457_483, 43); + private static readonly DateTime StampedThrough = Start.AddHours(21.5d); + + [Fact] + public void AFleetDayWithATail_CountsTheStampedHoursAtTheWeight_AndPassesBothLimits() + { + var rows = FleetDay.RowsOver(Start, Start.AddDays(1), StampedThrough); + /* 2.5/24 x 8.46 M = 0.88 M wide + 0.3 x 7.57 M = 3.15 M, as the design derived it. */ + Assert.InRange(rows, 3_100_000L, 3_200_000L); + Assert.Null(QueryStoreWideReadGuard.Refusal(FleetDay, Start, Start.AddDays(1), QueryStoreWideReadGuard.LimitFor(false), StampedThrough)); + Assert.Null(QueryStoreWideReadGuard.Refusal(FleetDay, Start, Start.AddDays(1), QueryStoreWideReadGuard.LimitFor(true), StampedThrough)); + } + + [Fact] + public void TwoStampedDays_PassOneScanAndFailTwo_AndThreeDaysFailBoth() + { + /* Two days with the same 2.5 h tail: 5.7 million equivalent rows. */ + var twoDays = Start.AddDays(2); + var throughTwo = twoDays.AddHours(-2.5d); + Assert.InRange(FleetDay.RowsOver(Start, twoDays, throughTwo), 5_600_000L, 5_800_000L); + Assert.Null(QueryStoreWideReadGuard.Refusal(FleetDay, Start, twoDays, QueryStoreWideReadGuard.LimitFor(false), throughTwo)); + Assert.NotNull(QueryStoreWideReadGuard.Refusal(FleetDay, Start, twoDays, QueryStoreWideReadGuard.LimitFor(true), throughTwo)); + + /* Three days: 8.2 million, refused by both. */ + var threeDays = Start.AddDays(3); + var throughThree = threeDays.AddHours(-2.5d); + Assert.InRange(FleetDay.RowsOver(Start, threeDays, throughThree), 8_100_000L, 8_300_000L); + Assert.NotNull(QueryStoreWideReadGuard.Refusal(FleetDay, Start, threeDays, QueryStoreWideReadGuard.LimitFor(false), throughThree)); + Assert.NotNull(QueryStoreWideReadGuard.Refusal(FleetDay, Start, threeDays, QueryStoreWideReadGuard.LimitFor(true), throughThree)); + } + + [Fact] + public void TheStampThrough_IsClamped_AndNoStampThroughIsTodaysFigure() + { + var end = Start.AddDays(1); + var unweighted = FleetDay.RowsOver(Start, end); + Assert.Equal(unweighted, FleetDay.RowsOver(Start, end, null)); + Assert.Equal(unweighted, FleetDay.RowsOver(Start, end, Start)); /* nothing stamped */ + Assert.Equal(unweighted, FleetDay.RowsOver(Start, end, Start.AddHours(-5))); /* before the range */ + var allStamped = FleetDay.RowsOver(Start, end, end); + Assert.Equal(FleetDay.RowsOver(Start, end, end.AddDays(3)), allStamped); /* past the range */ + Assert.InRange(allStamped, (long)(unweighted * 0.3d) - 1, (long)(unweighted * 0.3d) + 2); + } + + [Fact] + public void AWeightedRefusal_NeverCallsTheFigureRows_AndNamesTheServersTheWindowAndWhatToChange() + { + var through = Start.AddDays(3).AddHours(-2.5d); + var message = QueryStoreWideReadGuard.Refusal(FleetDay, Start, Start.AddDays(3), QueryStoreWideReadGuard.LimitFor(false), through)!; + Assert.Equal( + "This panel reads too much Query Store history to finish inside the time limit (43 servers, 3 days). Choose fewer servers or a shorter window.", + message); + Assert.DoesNotContain("rows", message, StringComparison.OrdinalIgnoreCase); + Assert.DoesNotContain("million", message, StringComparison.OrdinalIgnoreCase); + foreach (var sentence in message.Split(". ", StringSplitOptions.RemoveEmptyEntries)) + { + Assert.True(sentence.Split(' ').Length <= 20, "keep each sentence short: " + sentence); + } + } + + [Fact] + public void TheMessage_ReadsPlainly() + { + /* The plain-English checker's rules, pinned on the message the user reads: short sentences, no code words. */ + var message = QueryStoreWideReadGuard.Refusal(new QueryStoreWideReadGuard.Estimate(12_000_000, 43), Start, Start.AddDays(1))!; + foreach (var sentence in message.Split(". ", StringSplitOptions.RemoveEmptyEntries)) + { + Assert.True(sentence.Split(' ').Length <= 20, "keep each sentence short: " + sentence); + } + + Assert.DoesNotContain("wide", message, StringComparison.OrdinalIgnoreCase); + Assert.DoesNotContain("collect.", message, StringComparison.Ordinal); + } +} + +/// +/// #5582, live: the estimate reads the right rows, and the refusal answers on every path that runs a composed panel +/// (the shared runner behind /api/compose/run, and the MCP tool run_custom_view_panel). Seeds the same +/// wide-eligible store builds, then writes the #5094 daily summary's +/// built rows the estimate reads. +/// +/* #1776 own-store: deliberately NOT [Collection("live-postgres")]. The test reaches DARLING_TEST_PG only to CREATE and + DROP its own database through ScratchPostgres and works entirely inside it. */ +public sealed class QueryStoreWideReadRefusalLiveTests +{ + private const int ServerId = -5582002; + private const string ServerName = "qsiw-refusal"; + private const string OtherServerName = "qsiw-refusal-other"; + private const int OtherServerId = -5582003; + + private static readonly DateTime S = DateTime.SpecifyKind(DateTime.UtcNow.Date.AddDays(-6), DateTimeKind.Unspecified); + + private const string PanelJson = + "{\"source\":\"query_store_stats\",\"measure\":\"qs_executions\",\"aggregate\":\"sum\",\"timeBucket\":\"hour\",\"viz\":\"line\"}"; + + [Fact] + public async Task Estimate_ReadsTheNewestBuiltDayOfEachEnabledServerInScope() + { + var baseCs = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseCs), "Set DARLING_TEST_PG to a Postgres connection string to run the #5582 estimate live test."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseCs!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + await DarlingMcpTestData.RegisterServerAsync(connection, OtherServerId, OtherServerName, ct); + var noBuiltDayId = ServerId - 10; + await DarlingMcpTestData.RegisterServerAsync(connection, noBuiltDayId, "qsiw-refusal-nobuilt", ct); + var disabledId = ServerId - 11; + await DarlingMcpTestData.RegisterServerAsync(connection, disabledId, "qsiw-refusal-disabled", ct); + await ExecAsync(connection, $"UPDATE collect.servers SET is_enabled = FALSE WHERE server_id = {disabledId}", ct); + + /* Two built days for the first server: only the newest counts. A server with no built day adds nothing. A disabled + server with one counts, as the compiled wide read takes its retained rows too. */ + await InsertBuiltAsync(connection, ServerId, S.Date.AddDays(1), 1_000, ct); + await InsertBuiltAsync(connection, ServerId, S.Date.AddDays(3), 700, ct); + await InsertBuiltAsync(connection, OtherServerId, S.Date.AddDays(2), 50, ct); + await InsertBuiltAsync(connection, disabledId, S.Date.AddDays(2), 9_999_999, ct); + + var fleet = await QueryStoreWideReadGuard.EstimateAsync(postgres, null, null, ct); + Assert.Equal(new QueryStoreWideReadGuard.Estimate(750 + 9_999_999, 3), fleet); + + var scoped = await QueryStoreWideReadGuard.EstimateAsync(postgres, new[] { ServerName }, null, ct); + Assert.Equal(new QueryStoreWideReadGuard.Estimate(700, 1), scoped); + + var noBuilt = await QueryStoreWideReadGuard.EstimateAsync(postgres, new[] { "qsiw-refusal-nobuilt" }, null, ct); + Assert.Equal(new QueryStoreWideReadGuard.Estimate(0, 0), noBuilt); + Assert.Null(QueryStoreWideReadGuard.Refusal(noBuilt, S, S.AddDays(30))); + + } + + [Fact] + public async Task EveryPathThatRunsAComposedPanel_RefusesAnOversizedRead_AndRunsASmallOne() + { + var baseCs = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseCs), "Set DARLING_TEST_PG to a Postgres connection string to run the #5582 refusal live test."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseCs!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + Assert.True(await TimescaleSupport.TryEnableAsync(connection, null, ct), "TimescaleDB must be enabled on the test cluster"); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var runner = new DarlingCollectorRunner(postgres, new CollectorDeltaCalculator()); + await QueryStoreIntervalWideGridLiveTests.SeedGridAsync(runner, ServerId, S, ct); + await ExecAsync(connection, + "INSERT INTO servers (server_id, server_name, display_name, is_enabled, sql_major_version, created_date, modified_date) " + + $"VALUES ({ServerId}, '{ServerName}', '{ServerName}', TRUE, 16, now(), now()) ON CONFLICT (server_id) DO UPDATE SET is_enabled = TRUE", ct); + await QueryStoreIntervalWideBelowFloorLiveTests.SeedQueryStoreLogAsync(connection, ServerId, S.AddDays(-60), S.AddDays(4), null, null, ct); + await ExecAsync(connection, + $"UPDATE collect.query_store_interval_wide_coverage SET filled_since = TIMESTAMP '{S.AddHours(12):yyyy-MM-dd HH:mm:ss}' WHERE server_id = {ServerId}", ct); + var end = (DateTime)(await ScalarAsync(connection, + $"SELECT applied_through FROM collect.query_store_interval_wide_coverage WHERE server_id = {ServerId}", ct))!; + await ExecAsync(connection, $"SELECT drop_chunks('collect.query_store_stats', older_than => TIMESTAMP '{S.AddDays(1):yyyy-MM-dd HH:mm:ss}')", ct); + + var resolution = await DarlingWebEndpoints.ResolveQueryStoreWideEligibleAsync(postgres, null, S, end, end, ct); + Assert.True(resolution.Eligible, "the seeded store must route the panel to the wide table, or this test proves nothing"); + + /* The window end carries the microseconds applied_through has: a whole-second end can fall before it, and the gate + then reads raw (the literal-end rule). */ + var spec = new JsonObject + { + ["panel"] = JsonNode.Parse(PanelJson), + ["windowStart"] = S.ToString("yyyy-MM-dd'T'HH:mm:ss'Z'", System.Globalization.CultureInfo.InvariantCulture), + ["windowEnd"] = end.ToString("yyyy-MM-dd'T'HH:mm:ss.ffffff'Z'", System.Globalization.CultureInfo.InvariantCulture), + }; + + /* No built day: the guard fails open and the panel runs, on both paths. */ + await AssertRunsAsync(postgres, spec, ct); + + /* A small built day: still runs. */ + await InsertBuiltAsync(connection, ServerId, S.Date.AddDays(2), 1_000, ct); + await AssertRunsAsync(postgres, spec, ct); + + /* An oversized built day: refused with the message, on the shared runner and on the MCP tool. */ + await ExecAsync(connection, $"UPDATE collect.query_store_top_daily_built SET source_rows = {ComposeLimits.MaxQueryStoreWideRows * 4} WHERE server_id = {ServerId}", ct); + + var outcome = await DarlingWebEndpoints.RunComposedPanelAsync(postgres, (JsonObject)spec.DeepClone(), ct); + Assert.True(outcome.Payload is null, "the oversized read must be refused; the panel ran: " + outcome.Payload?["sql"]); + Assert.False(outcome.IsServerError); + Assert.NotNull(outcome.Error); + Assert.StartsWith("This panel needs about ", outcome.Error, StringComparison.Ordinal); + Assert.Contains("Query Store rows (1 server, ", outcome.Error, StringComparison.Ordinal); + Assert.EndsWith("Choose fewer servers or a shorter window.", outcome.Error, StringComparison.Ordinal); + + var tool = await DarlingMcpCustomViewTools.RunCustomViewPanel(postgres, spec.ToJsonString()); + Assert.Contains("\"status\":\"invalid\"", tool.Replace(" ", string.Empty), StringComparison.Ordinal); + Assert.Contains("This panel needs about ", tool, StringComparison.Ordinal); + Assert.Contains("Choose fewer servers or a shorter window.", tool, StringComparison.Ordinal); + + /* A scope that leaves out the oversized server is not refused. */ + var scoped = (JsonObject)spec.DeepClone(); + scoped["server"] = "no-such-server-5582"; + var scopedOutcome = await DarlingWebEndpoints.RunComposedPanelAsync(postgres, scoped, ct); + Assert.True(scopedOutcome.Error is null || !scopedOutcome.Error.StartsWith("This panel needs about ", StringComparison.Ordinal), "a scope with no built day must not be refused"); + + } + + private static async Task AssertRunsAsync(NpgsqlDataSource postgres, JsonObject spec, CancellationToken ct) + { + var outcome = await DarlingWebEndpoints.RunComposedPanelAsync(postgres, (JsonObject)spec.DeepClone(), ct); + Assert.True(outcome.Payload is not null, "the panel must run: " + outcome.Error); + + var tool = await DarlingMcpCustomViewTools.RunCustomViewPanel(postgres, spec.ToJsonString()); + Assert.DoesNotContain("This panel needs about", tool, StringComparison.Ordinal); + } + + private static Task InsertBuiltAsync(NpgsqlConnection connection, int serverId, DateTime day, long sourceRows, CancellationToken ct) => + ExecAsync(connection, + "INSERT INTO collect.query_store_top_daily_built (server_id, day, pass, built_at, source_rows) " + + $"VALUES ({serverId}, DATE '{day:yyyy-MM-dd}', 2, now() AT TIME ZONE 'UTC', {sourceRows})", ct); + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task ScalarAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + return await command.ExecuteScalarAsync(ct); + } +} diff --git a/Darling/Darling.Tests/QueryStoreWideUpperBoundLiveTests.cs b/Darling/Darling.Tests/QueryStoreWideUpperBoundLiveTests.cs new file mode 100644 index 0000000000..61040f6b86 --- /dev/null +++ b/Darling/Darling.Tests/QueryStoreWideUpperBoundLiveTests.cs @@ -0,0 +1,305 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.RegularExpressions; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5523: the per-server reads of collect.query_store_interval_wide now bound first_execution_time at the +/// window's end as well as its start, so ix_query_store_interval_wide_server_first_exec scans a finite range. This +/// pins that no row is lost: rows straddle every edge of the window (an interval that began before the window and is +/// still running, a row collected exactly at each edge, a row whose first execution is just past the end because the +/// monitored server's clock runs ahead, an interval that spans a whole day past the end of a window placed by its start), +/// and each statement returns exactly what the same statement returns with the new bound cut out. One row sits past the +/// slack on purpose, and each bound has a row one minute outside it: the bounds are live, and those are the only rows the new text drops. +/// +/* #1776 own-store: reaches DARLING_TEST_PG only to CREATE and DROP its own database through ScratchPostgres and then + works entirely inside it, the same shape as QueryStoreBackgroundIndexesLiveTests. */ +public sealed class QueryStoreWideUpperBoundLiveTests +{ + private const int TopServer = 51; + private const int TrendServer = 52; + + private static readonly DateTime W0 = new(2026, 9, 10, 6, 0, 0, DateTimeKind.Unspecified); + private static readonly DateTime W1 = new(2026, 9, 12, 18, 0, 0, DateTimeKind.Unspecified); + private static readonly DateTime S = new(2026, 9, 11, 0, 0, 0, DateTimeKind.Unspecified); + private static readonly DateTime E = new(2026, 9, 12, 0, 0, 0, DateTimeKind.Unspecified); + + private static string? BaseConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + /* (query_id, collection_time, first_execution_time) for TopServer. Every row is its own interval (distinct interval id, plan + id = query id), so one row is one group in the top reads. */ + private static readonly (int Q, DateTime C, DateTime F)[] TopRows = + { + (1, W0.AddHours(1), W0.AddHours(-20)), // began before the window, still running + (2, W0, W0.AddHours(-25).AddMinutes(-30)), // collected exactly at the start, inside the 26 h floor + (3, W1, W1.AddHours(-1)), // collected exactly at the end + (4, W1, W1.AddHours(12).AddMinutes(-1)), // monitored clock 11 h 59 min ahead: inside the 12 h slack + (5, W1.AddMinutes(-1), W1.AddMinutes(30)), + (6, W0.AddMinutes(-1), W0.AddHours(-1)), // collected just before the window + (7, W1.AddMinutes(1), W1), // collected just after the window + (8, W1.AddDays(3), W1.AddDays(3)), // newer rows the old range walked and filtered out + (9, W0.AddHours(5), W0.AddHours(4)), // ordinary + (10, new DateTime(2026, 9, 11, 12, 0, 0), new DateTime(2026, 9, 11, 11, 0, 0)), // the middle day + (11, W1, W1.AddHours(12).AddMinutes(1)), // past the slack: dropped by every upper bound + (12, S.AddMinutes(-1), S.AddHours(12).AddMinutes(-1)), // collected just before the long-window read's first edge, first execution inside the slack past it + (13, S.AddMinutes(-1), S.AddHours(12).AddMinutes(1)), // the same, one minute past the slack: only the long-window read's first edge drops it + }; + + /* (query_id, interval start, first_execution_time, collection_time) for TrendServer; start null is a legacy row. */ + private static readonly (int Q, DateTime? Start, DateTime F, DateTime C)[] TrendRows = + { + (21, W0, W0, W0.AddMinutes(30)), // starts at the window start + (22, W0.AddMinutes(10), W0.AddMinutes(-59), W0.AddHours(1)), // first execution 59 minutes below the window start: inside the 1 h floor + (23, W1, W1.AddHours(24), W1.AddHours(25)), // starts at the end, spans the whole day + (24, W1.AddMinutes(-10), W1.AddHours(23).AddMinutes(50), W1.AddHours(24)), + (25, W0.AddMinutes(-1), W0.AddMinutes(-1), W0.AddMinutes(10)), // starts just before the window + (26, W1.AddMinutes(1), W1.AddMinutes(1), W1.AddMinutes(5)), // starts just after the window + (27, W0.AddHours(1), W0.AddHours(1), W0.AddHours(1)), + (30, W0.AddMinutes(20), W0.AddMinutes(-61), W0.AddHours(1)), // first execution 61 minutes below the window start: past the floor, dropped + (28, null, W0.AddHours(1), W0.AddHours(2)), // legacy rows + (29, null, W1.AddHours(12).AddMinutes(-1), W1.AddMinutes(-5)), + (31, null, W1.AddHours(12).AddMinutes(1), W1.AddMinutes(-3)), // legacy, one minute past the slack: dropped + }; + + private static readonly int[] ExpectedTop = { 1, 2, 3, 4, 5, 9, 10, 12, 13 }; + private static readonly int[] ExpectedDaily = { 1, 2, 3, 4, 5, 9, 12 }; + private static readonly DateTime[] ExpectedTrendPoints = + { + W0, W0.AddMinutes(10), W0.AddHours(1), W0.AddHours(2), W1.AddMinutes(-10), W1.AddMinutes(-5), W1, + }; + + /* What the same trend statement returns with its new bounds cut out: the two rows the bounds exist to drop come back. */ + private static readonly DateTime[] OldTrendPoints = + { + W0, W0.AddMinutes(10), W0.AddMinutes(20), W0.AddHours(1), W0.AddHours(2), W1.AddMinutes(-10), W1.AddMinutes(-5), W1.AddMinutes(-3), W1, + }; + + [Fact] + public async Task EveryReadWithTheUpperBound_ReturnsWhatItReturnedWithoutIt_AndDropsOnlyARowPastTheSlack() + { + var baseCs = BaseConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(baseCs), "Set DARLING_TEST_PG to a Postgres connection string to run the #5523 live test."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseCs!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await SeedAsync(connection, ct); + + /* ---- MCP top: the table read. ---- */ + var mcpNew = DarlingDataReader.QueryStoreTopTableSql; + var mcpOld = Cut(mcpNew, @"\s*AND\s+first_execution_time <= \$3 \+ interval '\d+ minutes'"); + var mcpTop = await TopIdsAsync(connection, mcpNew, ct); + Assert.Equal(ExpectedTop, mcpTop); + Assert.Equal(ExpectedTop.Append(11).OrderBy(q => q), await TopIdsAsync(connection, mcpOld, ct)); + + /* ---- The long-window read: only the two edge arms read the wide table; no daily row is built, so the middle + days contribute nothing, and the edges are what the bound could clip. ---- */ + var dailyNew = DarlingDataReader.QueryStoreTopDailyTableSql; + var dailyOld = Cut(Cut(dailyNew, @"\s*AND\s+first_execution_time < \$8::date \+ interval '\d+ minutes'"), + @"\s*AND\s+first_execution_time <= \$3 \+ interval '\d+ minutes'"); + var dailyIds = await DailyIdsAsync(connection, dailyNew, ct); + Assert.Equal(ExpectedDaily, dailyIds); + Assert.Equal(ExpectedDaily.Append(11).Append(13).OrderBy(q => q), await DailyIdsAsync(connection, dailyOld, ct)); + + /* ---- The Queries grid's table read: a closed end, and the open end a preset window binds as NULL. ---- */ + var gridNew = ViewerDataService.QueryStoreTopTableSql; + var gridOld = Cut(gridNew, @"\s*AND\s+\(\$3::timestamp IS NULL OR first_execution_time <= \$3 \+ interval '\d+ minutes'\)"); + Assert.Equal(ExpectedTop, await GridIdsAsync(connection, gridNew, W1, ct)); + Assert.Equal(ExpectedTop.Append(11).OrderBy(q => q), await GridIdsAsync(connection, gridOld, W1, ct)); + var openEndNew = await GridIdsAsync(connection, gridNew, null, ct); + Assert.Equal(await GridIdsAsync(connection, gridOld, null, ct), openEndNew); + Assert.Contains(8, openEndNew); + Assert.Contains(11, openEndNew); + + /* ---- The history tool's table read, one query at a time. ---- */ + var historyNew = DarlingMcpQueryStoreHistoryTools.HistoryTableSql; + var historyOld = Cut(historyNew, @"\s*AND\s+first_execution_time <= \$5 \+ interval '\d+ minutes'"); + foreach (var q in TopRows.Select(r => r.Q)) + { + var viaNew = await HistoryPlanIdsAsync(connection, historyNew, q, ct); + var viaOld = await HistoryPlanIdsAsync(connection, historyOld, q, ct); + if (q == 11) + { + Assert.Empty(viaNew); + Assert.Single(viaOld); + } + else + { + Assert.Equal(viaOld, viaNew); + } + + Assert.Equal(ExpectedTop.Contains(q), viaNew.Count == 1); + } + + /* ---- The duration trend's table read: both arms. ---- */ + var trendNew = ViewerDataService.QueryStoreDurationTrendTableSql; + var trendOld = trendNew + .Replace($"first_execution_time >= $2 - {QueryStoreIntervalWide.IntervalStartSlackSql}", $"first_execution_time >= $2 - {QueryStoreIntervalWide.PurgeEdgeMarginSql}", StringComparison.Ordinal); + trendOld = Cut(Cut(trendOld, @"\s*AND\s+first_execution_time <= \$3 \+ interval '\d+ minutes'"), + @"\s*AND\s+first_execution_time <= \$4 \+ interval '\d+ minutes'"); + Assert.NotEqual(trendNew, trendOld); + var points = await TrendPointsAsync(connection, trendNew, ct); + Assert.Equal(ExpectedTrendPoints, points); + Assert.Equal(OldTrendPoints, await TrendPointsAsync(connection, trendOld, ct)); + } + + /// The statement with one regex match cut out; the match must exist, or the "old" text would silently equal the new. + private static string Cut(string sql, string pattern) + { + var regex = new Regex(pattern); + Assert.Single(regex.Matches(sql)); + var cut = regex.Replace(sql, string.Empty); + Assert.NotEqual(sql, cut); + return cut; + } + + private static async Task SeedAsync(NpgsqlConnection connection, CancellationToken ct) + { + foreach (var (q, c, f) in TopRows) + { + await InsertAsync(connection, TopServer, q, c, f, q, c.AddMinutes(-5), ct); + } + + foreach (var (q, start, f, c) in TrendRows) + { + await InsertAsync(connection, TrendServer, q, c, f, start is null ? -q : q, start, ct); + } + + await using var analyze = new NpgsqlCommand("ANALYZE collect.query_store_interval_wide", connection); + await analyze.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertAsync(NpgsqlConnection connection, int server, int q, DateTime c, DateTime f, long intervalId, DateTime? start, CancellationToken ct) + { + await using var command = new NpgsqlCommand(@" +INSERT INTO collect.query_store_interval_wide + (collection_time, server_id, database_name, query_id, plan_id, execution_type_desc, first_execution_time, + last_execution_time, query_text, execution_count, avg_duration_us, runtime_stats_interval_id, interval_start_time_utc) +VALUES (@c, @server, 'db', @q, @q, 'Regular', @f, @c, 'select 1', 10, 1000, @interval, @start)", connection); + command.Parameters.AddWithValue("c", NpgsqlDbType.Timestamp, c); + command.Parameters.AddWithValue("server", server); + command.Parameters.AddWithValue("q", (long)q); + command.Parameters.AddWithValue("f", NpgsqlDbType.Timestamp, f); + command.Parameters.AddWithValue("interval", intervalId); + command.Parameters.Add(new NpgsqlParameter("start", NpgsqlDbType.Timestamp) { Value = (object?)start ?? DBNull.Value }); + await command.ExecuteNonQueryAsync(ct); + } + + private static NpgsqlParameter Ts(DateTime value) => new NpgsqlParameter { TypedValue = DateTime.SpecifyKind(value, DateTimeKind.Unspecified), NpgsqlDbType = NpgsqlDbType.Timestamp }; + + private static NpgsqlParameter Text() => new() { NpgsqlDbType = NpgsqlDbType.Text, Value = DBNull.Value }; + + private static async Task> IdsAsync(NpgsqlCommand command, string column, CancellationToken ct) + { + await using var reader = await command.ExecuteReaderAsync(ct); + var ordinal = reader.GetOrdinal(column); + var ids = new List(); + while (await reader.ReadAsync(ct)) + { + ids.Add((int)reader.GetInt64(ordinal)); + } + + ids.Sort(); + return ids; + } + + private static async Task> TopIdsAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + command.Parameters.Add(new NpgsqlParameter { TypedValue = TopServer }); + command.Parameters.Add(Ts(W0)); + command.Parameters.Add(Ts(W1)); + command.Parameters.Add(new NpgsqlParameter { TypedValue = 500 }); + command.Parameters.Add(DatabaseFilter.All.Parameter()); + command.Parameters.Add(Text()); + command.Parameters.Add(Text()); + command.Parameters.Add(new NpgsqlParameter { TypedValue = TopFill.FirstCandidates(500) }); + return await IdsAsync(command, "query_id", ct); + } + + private static async Task> DailyIdsAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + command.Parameters.Add(new NpgsqlParameter { TypedValue = TopServer }); + command.Parameters.Add(Ts(W0)); + command.Parameters.Add(Ts(W1)); + command.Parameters.Add(new NpgsqlParameter { TypedValue = 500 }); + command.Parameters.Add(DatabaseFilter.All.Parameter()); + command.Parameters.Add(Text()); + command.Parameters.Add(Text()); + command.Parameters.Add(new NpgsqlParameter { TypedValue = DateOnly.FromDateTime(S), NpgsqlDbType = NpgsqlDbType.Date }); + command.Parameters.Add(new NpgsqlParameter { TypedValue = DateOnly.FromDateTime(E), NpgsqlDbType = NpgsqlDbType.Date }); + command.Parameters.Add(new NpgsqlParameter { TypedValue = TopFill.FirstCandidates(500) }); + return await IdsAsync(command, "query_id", ct); + } + + private static async Task> GridIdsAsync(NpgsqlConnection connection, string sql, DateTime? end, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + command.Parameters.Add(new NpgsqlParameter { TypedValue = TopServer }); + command.Parameters.Add(Ts(W0)); + command.Parameters.Add(end is DateTime e + ? Ts(e) + : new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter { TypedValue = 500 }); + command.Parameters.Add(ViewerDataService.DatabaseFilterParameter(null)); + command.Parameters.Add(new NpgsqlParameter { TypedValue = TopFill.FirstCandidates(500) }); + return await IdsAsync(command, "query_id", ct); + } + + private static async Task> HistoryPlanIdsAsync(NpgsqlConnection connection, string sql, int q, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + command.Parameters.Add(new NpgsqlParameter { TypedValue = TopServer }); + command.Parameters.Add(new NpgsqlParameter { TypedValue = "db" }); + command.Parameters.Add(new NpgsqlParameter { TypedValue = q }); + command.Parameters.Add(Ts(W0)); + command.Parameters.Add(Ts(W1)); + await using var reader = await command.ExecuteReaderAsync(ct); + var ids = new List(); + while (await reader.ReadAsync(ct)) + { + ids.Add(reader.GetInt64(0)); + } + + return ids; + } + + private static async Task> TrendPointsAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + command.Parameters.Add(new NpgsqlParameter { TypedValue = TrendServer }); + command.Parameters.Add(Ts(W0)); + command.Parameters.Add(Ts(W1)); + command.Parameters.Add(Ts(W1)); + command.Parameters.Add(ViewerDataService.DatabaseFilterParameter(null)); + await using var reader = await command.ExecuteReaderAsync(ct); + var points = new List(); + while (await reader.ReadAsync(ct)) + { + points.Add(reader.GetDateTime(0)); + } + + return points; + } +} diff --git a/Darling/Darling.Tests/QueryStoreWriteFencePinTests.cs b/Darling/Darling.Tests/QueryStoreWriteFencePinTests.cs index 608e1b0685..e6943fd427 100644 --- a/Darling/Darling.Tests/QueryStoreWriteFencePinTests.cs +++ b/Darling/Darling.Tests/QueryStoreWriteFencePinTests.cs @@ -18,6 +18,7 @@ namespace Darling.Tests; /// the begin precedes the transaction's open, and the end sits in a finally that follows the commit, so a /// throw or a cancel after the rows committed (or a rollback) still ends the write. /// +[Trait("Stage", "Guard")] public sealed class QueryStoreWriteFencePinTests { private static string Here([CallerFilePath] string thisFile = "") => thisFile; diff --git a/Darling/Darling.Tests/QueryStoreWriterSetPinTests.cs b/Darling/Darling.Tests/QueryStoreWriterSetPinTests.cs index f6e5e85bf9..7d120f55b5 100644 --- a/Darling/Darling.Tests/QueryStoreWriterSetPinTests.cs +++ b/Darling/Darling.Tests/QueryStoreWriterSetPinTests.cs @@ -23,6 +23,7 @@ namespace Darling.Tests; /// ({table}, collect.{Table}, the collector's TargetTable), so a search for the literal table /// name finds none of them; keys on the spellings the writers actually use. /// +[Trait("Stage", "Guard")] public sealed class QueryStoreWriterSetPinTests { /// How a source names the table: the literal, a run-time spelling, or a sweep over every collector's diff --git a/Darling/Darling.Tests/QuietServerWatermarkLadderLiveTests.cs b/Darling/Darling.Tests/QuietServerWatermarkLadderLiveTests.cs new file mode 100644 index 0000000000..60f274eaeb --- /dev/null +++ b/Darling/Darling.Tests/QuietServerWatermarkLadderLiveTests.cs @@ -0,0 +1,369 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5515: a server with no job run or trace event for hours missed the 6-hour probe and went straight to the +/// unbounded MAX, which opens every chunk of the server's slice (5,426 blocks a call on job_history, 3,758 on +/// default_trace_events, on the 43-server store). The ladder tries a week between the two. These pins run +/// against a live store and check both halves of the contract: the answer is the unbounded MAX's answer for the +/// columns tested (a watermark behind the store's re-reads stored rows, one ahead of it skips new ones; job_history's +/// run_datetime, which does not rise with collection_time, can read older, which only widens a re-read), and the whole-slice +/// read runs only when the week found nothing. Counted with Npgsql's own command log, as the cache's pins are +/// (). The server-scoped reads, the cpu_utilization pair read and the +/// Azure per-database read all use the ladder. +/// +[Trait("Stage", "Live")] +[Collection("live-postgres")] +public sealed class QuietServerWatermarkLadderLiveTests +{ + private const int ServerId = -880155; + + private static string? Pg => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static string TraceBounded => + DarlingCollectorRunner.BuildServerWatermarkSql("default_trace_events", "event_time", bounded: true); + + private static string TraceUnbounded => + DarlingCollectorRunner.BuildServerWatermarkSql("default_trace_events", "event_time", bounded: false); + + private static string JobBounded => + DarlingCollectorRunner.BuildServerWatermarkInstanceIdSql("job_history", "instance_id", bounded: true); + + private static string JobUnbounded => + DarlingCollectorRunner.BuildServerWatermarkInstanceIdSql("job_history", "instance_id", bounded: false); + + /// The unbounded text is a prefix of the bounded one, so its own count is what is left after the bounded ones. + private static int Unbounded(CommandCountingLoggerProvider log, string unbounded, string bounded) => + log.CountContaining(unbounded) - log.CountContaining(bounded); + + [Fact] + public async Task AQuietServer_IsAnsweredByTheWeek_WithTheSameValueAsTheUnboundedRead() + { + Assert.SkipWhen(string.IsNullOrEmpty(Pg), "Set DARLING_TEST_PG to run the #5515 quiet-server pins."); + var ct = TestContext.Current.CancellationToken; + + var loggerFactory = new CommandCountingLoggerFactory(); + await using var dataSource = new NpgsqlDataSourceBuilder(Pg!).UseLoggerFactory(loggerFactory).Build(); + await using (var migrate = await dataSource.OpenConnectionAsync(ct)) + { + await PgMigrations.MigrateAsync(migrate, ct); + } + + var log = loggerFactory.Provider; + var runner = new DarlingCollectorRunner(dataSource, new CollectorDeltaCalculator()); + await CleanAsync(dataSource, ct); + var bodySucceeded = false; + try + { + /* Newest event two days ago: past the 6-hour probe, inside the week. */ + var twoDays = DateTime.UtcNow.Date - TimeSpan.FromDays(2) + TimeSpan.FromHours(3); + await InsertTraceEventAsync(dataSource, 1, twoDays, ct); + await InsertTraceEventAsync(dataSource, 2, twoDays - TimeSpan.FromHours(5), ct); + await InsertJobAsync(dataSource, 1, 410, DateTime.UtcNow - TimeSpan.FromDays(3), ct); + await InsertJobAsync(dataSource, 2, 400, DateTime.UtcNow - TimeSpan.FromDays(4), ct); + + /* The store's own answer first: this read is itself an unbounded statement, so it runs before the counting starts. */ + var storeAnswer = await UnboundedMaxAsync(dataSource, ct); + log.Reset(); + var trace = await runner.GetLastCollectedTimeAsync(ServerId, "default_trace_events", "event_time", ct); + var job = await runner.GetLastCollectedInstanceIdAsync(ServerId, "job_history", "instance_id", ct); + + Assert.Equal(twoDays, trace); + Assert.Equal(410L, job); + Assert.Equal(storeAnswer, trace); + Assert.Equal(0, Unbounded(log, TraceUnbounded, TraceBounded)); + Assert.Equal(0, log.CountContaining(JobUnbounded)); + /* Two bounded reads each: the 6-hour probe, then the week. */ + Assert.Equal(2, log.CountContaining(TraceBounded)); + Assert.Equal(2, log.CountContaining(JobBounded)); + + /* A row lands since: the 6-hour probe answers, in one read, and it is the new row (never skipped). */ + /* Whole microseconds: the store keeps no finer tick, so a finer one here could never be read back equal. */ + var recentTicks = (DateTime.UtcNow - TimeSpan.FromMinutes(5)).Ticks; + var recent = new DateTime(recentTicks - recentTicks % TimeSpan.TicksPerMicrosecond, DateTimeKind.Utc); + await InsertTraceEventAsync(dataSource, 3, recent, ct); + await InsertJobAsync(dataSource, 3, 420, recent, ct); + log.Reset(); + Assert.Equal(recent, await runner.GetLastCollectedTimeAsync(ServerId, "default_trace_events", "event_time", ct)); + Assert.Equal(420L, await runner.GetLastCollectedInstanceIdAsync(ServerId, "job_history", "instance_id", ct)); + Assert.Equal(1, log.CountContaining(TraceBounded)); + Assert.Equal(1, log.CountContaining(JobBounded)); + Assert.Equal(0, Unbounded(log, TraceUnbounded, TraceBounded)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(Pg!, bodySucceeded, async (cleanup, cleanupCt) => + await CleanAsync(cleanup, cleanupCt)); + } + } + + [Fact] + public async Task AServerQuietForMoreThanAWeek_AndOneWithNoRow_StillGetTheUnboundedAnswer() + { + Assert.SkipWhen(string.IsNullOrEmpty(Pg), "Set DARLING_TEST_PG to run the #5515 quiet-server pins."); + var ct = TestContext.Current.CancellationToken; + + var loggerFactory = new CommandCountingLoggerFactory(); + await using var dataSource = new NpgsqlDataSourceBuilder(Pg!).UseLoggerFactory(loggerFactory).Build(); + await using (var migrate = await dataSource.OpenConnectionAsync(ct)) + { + await PgMigrations.MigrateAsync(migrate, ct); + } + + var log = loggerFactory.Provider; + var runner = new DarlingCollectorRunner(dataSource, new CollectorDeltaCalculator()); + await CleanAsync(dataSource, ct); + var bodySucceeded = false; + try + { + /* No row at all: null after all three reads, so a first run still reads as one. */ + log.Reset(); + Assert.Null(await runner.GetLastCollectedTimeAsync(ServerId, "default_trace_events", "event_time", ct)); + Assert.Null(await runner.GetLastCollectedInstanceIdAsync(ServerId, "job_history", "instance_id", ct)); + Assert.Equal(1, Unbounded(log, TraceUnbounded, TraceBounded)); + Assert.Equal(1, log.CountContaining(JobUnbounded)); + + /* Newest row twenty days ago: past the week, so the whole-slice read still finds it. */ + var old = DateTime.UtcNow.Date - TimeSpan.FromDays(20) + TimeSpan.FromHours(1); + await InsertTraceEventAsync(dataSource, 1, old, ct); + await InsertJobAsync(dataSource, 1, 90, old, ct); + log.Reset(); + Assert.Equal(old, await runner.GetLastCollectedTimeAsync(ServerId, "default_trace_events", "event_time", ct)); + Assert.Equal(90L, await runner.GetLastCollectedInstanceIdAsync(ServerId, "job_history", "instance_id", ct)); + Assert.Equal(1, Unbounded(log, TraceUnbounded, TraceBounded)); + Assert.Equal(1, log.CountContaining(JobUnbounded)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(Pg!, bodySucceeded, async (cleanup, cleanupCt) => + await CleanAsync(cleanup, cleanupCt)); + } + } + + private static string PairBounded => + DarlingCollectorRunner.BuildServerWatermarkPairSql("cpu_utilization_stats", "sample_time", "sample_time_utc", bounded: true); + + private static string PairUnbounded => + DarlingCollectorRunner.BuildServerWatermarkPairSql("cpu_utilization_stats", "sample_time", "sample_time_utc", bounded: false); + + private static string DbBounded => + DarlingCollectorRunner.BuildServerWatermarkForDatabaseSql("deadlocks", "deadlock_time", "database_name", bounded: true); + + private static string DbUnbounded => + DarlingCollectorRunner.BuildServerWatermarkForDatabaseSql("deadlocks", "deadlock_time", "database_name", bounded: false); + + private const string QuietDatabase = "QuietDb"; + + /// The #3778 pair read (cpu_utilization) takes the same week rung: a server whose newest sample is + /// days old is answered by the week, and only a server quiet longer, or with no row, runs the unbounded pair. + [Fact] + public async Task ThePairRead_OfAQuietServer_IsAnsweredByTheWeek_AndOnlyAnOlderOrEmptyOneRunsTheUnboundedPair() + { + Assert.SkipWhen(string.IsNullOrEmpty(Pg), "Set DARLING_TEST_PG to run the #5515 quiet-server pins."); + var ct = TestContext.Current.CancellationToken; + + var loggerFactory = new CommandCountingLoggerFactory(); + await using var dataSource = new NpgsqlDataSourceBuilder(Pg!).UseLoggerFactory(loggerFactory).Build(); + await using (var migrate = await dataSource.OpenConnectionAsync(ct)) + { + await PgMigrations.MigrateAsync(migrate, ct); + } + + var log = loggerFactory.Provider; + var runner = new DarlingCollectorRunner(dataSource, new CollectorDeltaCalculator()); + await CleanAsync(dataSource, ct); + var bodySucceeded = false; + try + { + /* No row at all: (null, false) after the 6-hour probe, the week and the unbounded pair. */ + log.Reset(); + Assert.Equal((null, false), await runner.GetLastCollectedTimeWithFrameAsync( + ServerId, "cpu_utilization_stats", "sample_time", "sample_time_utc", ct)); + Assert.Equal(2, log.CountContaining(PairBounded)); + Assert.Equal(1, log.CountContaining(PairUnbounded) - log.CountContaining(PairBounded)); + + /* Newest sample two days ago: past the 6-hour probe, inside the week. No unbounded read. */ + var twoDays = DateTime.UtcNow.Date - TimeSpan.FromDays(2) + TimeSpan.FromHours(3); + await InsertCpuAsync(dataSource, 1, twoDays, ct); + log.Reset(); + Assert.Equal((twoDays, true), await runner.GetLastCollectedTimeWithFrameAsync( + ServerId, "cpu_utilization_stats", "sample_time", "sample_time_utc", ct)); + Assert.Equal(2, log.CountContaining(PairBounded)); + Assert.Equal(0, log.CountContaining(PairUnbounded) - log.CountContaining(PairBounded)); + + /* Newest sample twenty days ago: past the week, so the unbounded pair still finds it. */ + await CleanAsync(dataSource, ct); + var old = DateTime.UtcNow.Date - TimeSpan.FromDays(20) + TimeSpan.FromHours(1); + await InsertCpuAsync(dataSource, 2, old, ct); + log.Reset(); + Assert.Equal((old, true), await runner.GetLastCollectedTimeWithFrameAsync( + ServerId, "cpu_utilization_stats", "sample_time", "sample_time_utc", ct)); + Assert.Equal(1, log.CountContaining(PairUnbounded) - log.CountContaining(PairBounded)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(Pg!, bodySucceeded, async (cleanup, cleanupCt) => + await CleanAsync(cleanup, cleanupCt)); + } + } + + /// The per-database read (the Azure deadlock, blocked-process and long-query collectors call it with no + /// floor, once per database per cycle) takes the same week rung: a quiet database whose last row is days old + /// stops paying the unbounded read every cycle. + [Fact] + public async Task ThePerDatabaseRead_OfAQuietDatabase_IsAnsweredByTheWeek_AndOnlyAnOlderOrEmptyOneRunsTheUnboundedRead() + { + Assert.SkipWhen(string.IsNullOrEmpty(Pg), "Set DARLING_TEST_PG to run the #5515 quiet-server pins."); + var ct = TestContext.Current.CancellationToken; + + var loggerFactory = new CommandCountingLoggerFactory(); + await using var dataSource = new NpgsqlDataSourceBuilder(Pg!).UseLoggerFactory(loggerFactory).Build(); + await using (var migrate = await dataSource.OpenConnectionAsync(ct)) + { + await PgMigrations.MigrateAsync(migrate, ct); + } + + var log = loggerFactory.Provider; + var runner = new DarlingCollectorRunner(dataSource, new CollectorDeltaCalculator()); + await CleanAsync(dataSource, ct); + var bodySucceeded = false; + try + { + /* No row for the database: null after all three reads. */ + log.Reset(); + Assert.Null(await runner.GetLastCollectedTimeForDatabaseAsync( + ServerId, "deadlocks", "deadlock_time", "database_name", QuietDatabase, ct)); + Assert.Equal(2, log.CountContaining(DbBounded)); + Assert.Equal(1, log.CountContaining(DbUnbounded) - log.CountContaining(DbBounded)); + + /* Newest deadlock two days ago: past the 6-hour probe, inside the week. No unbounded read. */ + var twoDays = DateTime.UtcNow.Date - TimeSpan.FromDays(2) + TimeSpan.FromHours(3); + await InsertDeadlockAsync(dataSource, 1, twoDays, ct); + log.Reset(); + Assert.Equal(twoDays, await runner.GetLastCollectedTimeForDatabaseAsync( + ServerId, "deadlocks", "deadlock_time", "database_name", QuietDatabase, ct)); + Assert.Equal(2, log.CountContaining(DbBounded)); + Assert.Equal(0, log.CountContaining(DbUnbounded) - log.CountContaining(DbBounded)); + + /* Newest deadlock twenty days ago: past the week, so the unbounded read still finds it. */ + await CleanAsync(dataSource, ct); + var old = DateTime.UtcNow.Date - TimeSpan.FromDays(20) + TimeSpan.FromHours(1); + await InsertDeadlockAsync(dataSource, 2, old, ct); + log.Reset(); + Assert.Equal(old, await runner.GetLastCollectedTimeForDatabaseAsync( + ServerId, "deadlocks", "deadlock_time", "database_name", QuietDatabase, ct)); + Assert.Equal(1, log.CountContaining(DbUnbounded) - log.CountContaining(DbBounded)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(Pg!, bodySucceeded, async (cleanup, cleanupCt) => + await CleanAsync(cleanup, cleanupCt)); + } + } + + private static async Task InsertCpuAsync(NpgsqlDataSource dataSource, long id, DateTime sampleTime, CancellationToken ct) + { + await using var connection = await dataSource.OpenConnectionAsync(ct); + using var command = new NpgsqlCommand(@" +INSERT INTO cpu_utilization_stats (collection_id, collection_time, server_id, server_name, sample_time, sample_time_utc, sqlserver_cpu_utilization, other_process_cpu_utilization) +VALUES ($1, $2, $3, $4, $5, $5, 1, 1)", connection); + command.Parameters.AddWithValue(id + 9_300_000); + command.Parameters.AddWithValue(DateTime.SpecifyKind(sampleTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue("QUIET-WM-SRV"); + command.Parameters.AddWithValue(DateTime.SpecifyKind(sampleTime, DateTimeKind.Unspecified)); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertDeadlockAsync(NpgsqlDataSource dataSource, long id, DateTime deadlockTime, CancellationToken ct) + { + await using var connection = await dataSource.OpenConnectionAsync(ct); + using var command = new NpgsqlCommand(@" +INSERT INTO deadlocks (deadlock_id, collection_time, server_id, server_name, deadlock_time, deadlock_graph_xml, database_name) +VALUES ($1, $2, $3, $4, $2, '', $5)", connection); + command.Parameters.AddWithValue(id + 9_300_000); + command.Parameters.AddWithValue(DateTime.SpecifyKind(deadlockTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue("QUIET-WM-SRV"); + command.Parameters.AddWithValue(QuietDatabase); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task UnboundedMaxAsync(NpgsqlDataSource dataSource, CancellationToken ct) + { + await using var connection = await dataSource.OpenConnectionAsync(ct); + using var command = new NpgsqlCommand( + "SELECT MAX(event_time) FROM default_trace_events WHERE server_id = $1", connection); + command.Parameters.AddWithValue(ServerId); + return await command.ExecuteScalarAsync(ct) as DateTime?; + } + + private static async Task InsertTraceEventAsync(NpgsqlDataSource dataSource, long id, DateTime eventTime, CancellationToken ct) + { + await using var connection = await dataSource.OpenConnectionAsync(ct); + using var command = new NpgsqlCommand(@" +INSERT INTO default_trace_events (default_trace_event_id, collection_time, server_id, server_name, event_time) +VALUES ($1, $2, $3, $4, $5)", connection); + command.Parameters.AddWithValue(id); + command.Parameters.AddWithValue(DateTime.SpecifyKind(eventTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue("QUIET-WM-SRV"); + command.Parameters.AddWithValue(DateTime.SpecifyKind(eventTime, DateTimeKind.Unspecified)); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertJobAsync( + NpgsqlDataSource dataSource, long id, long instanceId, DateTime collectionTime, CancellationToken ct) + { + await using var connection = await dataSource.OpenConnectionAsync(ct); + using var command = new NpgsqlCommand(@" +INSERT INTO job_history (job_history_id, collection_time, server_id, server_name, instance_id) +VALUES ($1, $2, $3, $4, $5)", connection); + command.Parameters.AddWithValue(id + 9_300_000); + command.Parameters.AddWithValue(DateTime.SpecifyKind(collectionTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue("QUIET-WM-SRV"); + command.Parameters.AddWithValue(instanceId); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task CleanAsync(NpgsqlDataSource dataSource, CancellationToken ct) + { + await using var connection = await dataSource.OpenConnectionAsync(ct); + await CleanAsync(connection, ct); + } + + private static async Task CleanAsync(NpgsqlConnection connection, CancellationToken ct) + { + foreach (var table in new[] { "default_trace_events", "job_history", "cpu_utilization_stats", "deadlocks" }) + { + using var command = new NpgsqlCommand($"DELETE FROM {table} WHERE server_id = $1", connection); + command.Parameters.AddWithValue(ServerId); + await command.ExecuteNonQueryAsync(ct); + } + } +} diff --git a/Darling/Darling.Tests/RawChunkIntervalReconcilerLiveTests.cs b/Darling/Darling.Tests/RawChunkIntervalReconcilerLiveTests.cs index 062c4ea574..713ff9c5e7 100644 --- a/Darling/Darling.Tests/RawChunkIntervalReconcilerLiveTests.cs +++ b/Darling/Darling.Tests/RawChunkIntervalReconcilerLiveTests.cs @@ -70,7 +70,7 @@ await ExecAsync(connection, "SELECT create_hypertable('collect.rci_test', by_range('ts', INTERVAL '24 hours'), if_not_exists => true, migrate_data => true)", ct); await ExecAsync(connection, "ALTER TABLE collect.rci_test SET (timescaledb.compress, timescaledb.compress_orderby = 'ts')", ct); - var utcNow = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + var utcNow = LiveClock.Now(); await PlantChunkAsync(connection, utcNow.AddDays(-3), ct); await PlantChunkAsync(connection, utcNow.AddDays(-2), ct); await ExecAsync(connection, "SELECT compress_chunk(c) FROM show_chunks('collect.rci_test') c", ct); @@ -78,7 +78,7 @@ await ExecAsync(connection, Assert.Equal(24, await CurrentIntervalHoursAsync(connection, ct)); /* ---- phase 1: narrows one rung, applies, records ---- */ - var changed1 = await RawChunkIntervalReconciler.ReconcileAsync(connection, budgetBytes: 1.0, DateTime.UtcNow, logger: null, ct); + var changed1 = await RawChunkIntervalReconciler.ReconcileAsync(connection, budgetBytes: 1.0, DateTime.SpecifyKind(LiveClock.Now(), DateTimeKind.Utc), logger: null, ct); Assert.Equal(1, changed1); Assert.Equal(12, await CurrentIntervalHoursAsync(connection, ct)); @@ -96,7 +96,7 @@ await ExecAsync(connection, Assert.Equal(1L, await ScalarLongAsync(connection, "SELECT COUNT(*) FROM collect.raw_chunk_interval_reconcile_runs", ct)); /* ---- phase 2: same day, held — proves IntervalLastChangedUtc came from the history table ---- */ - var changed2 = await RawChunkIntervalReconciler.ReconcileAsync(connection, budgetBytes: 1.0, DateTime.UtcNow, logger: null, ct); + var changed2 = await RawChunkIntervalReconciler.ReconcileAsync(connection, budgetBytes: 1.0, DateTime.SpecifyKind(LiveClock.Now(), DateTimeKind.Utc), logger: null, ct); Assert.Equal(0, changed2); Assert.Equal(12, await CurrentIntervalHoursAsync(connection, ct)); Assert.Equal(1L, await ScalarLongAsync(connection, "SELECT COUNT(*) FROM collect.raw_chunk_interval_rung_history", ct)); @@ -233,7 +233,7 @@ await ExecAsync(connection, "SELECT create_hypertable('collect.rated_test', by_range('ts', INTERVAL '24 hours'), if_not_exists => true, migrate_data => true)", ct); await ExecAsync(connection, "ALTER TABLE collect.rated_test SET (timescaledb.compress, timescaledb.compress_orderby = 'ts')", ct); - var utcNow = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + var utcNow = LiveClock.Now(); for (var day = 1; day <= 5; day++) { await PlantRatedChunkAsync(connection, utcNow.AddDays(-day), ct); @@ -263,7 +263,7 @@ JOIN timescaledb_information.chunks c var budgetBytes = openBytesAt24Hours / 2.55; var logger = new CapturingTestLogger(); - var changed = await RawChunkIntervalReconciler.ReconcileAsync(connection, budgetBytes, DateTime.UtcNow, logger, ct); + var changed = await RawChunkIntervalReconciler.ReconcileAsync(connection, budgetBytes, DateTime.SpecifyKind(LiveClock.Now(), DateTimeKind.Utc), logger, ct); Assert.Equal(1, changed); Assert.Equal(12, await CurrentIntervalHoursAsync(connection, "rated_test", ct)); @@ -282,7 +282,7 @@ summary line still logs at Information with the ratio under 1. The rated table i either pass runs — that held line is asserted at Information too, by exact prefix. ---- */ var loggerNoChange = new CapturingTestLogger(); var changedNoChange = await RawChunkIntervalReconciler.ReconcileAsync( - connection, budgetBytes: openBytesAt24Hours * 1_000, DateTime.UtcNow, loggerNoChange, ct); + connection, budgetBytes: openBytesAt24Hours * 1_000, DateTime.SpecifyKind(LiveClock.Now(), DateTimeKind.Utc), loggerNoChange, ct); Assert.Equal(0, changedNoChange); var summaryNoChange = loggerNoChange.Lines.FirstOrDefault(l => diff --git a/Darling/Darling.Tests/RawGatePerSlotSourceFloorLiveTests.cs b/Darling/Darling.Tests/RawGatePerSlotSourceFloorLiveTests.cs index c8948d40e4..59c25cfa9f 100644 --- a/Darling/Darling.Tests/RawGatePerSlotSourceFloorLiveTests.cs +++ b/Darling/Darling.Tests/RawGatePerSlotSourceFloorLiveTests.cs @@ -43,23 +43,38 @@ public sealed class RawGatePerSlotSourceFloorLiveTests "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live #4300 per-slot floor pins."); var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, default); - - var connectionString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { MaxAutoPrepare = 0 }.ConnectionString; - var connection = new NpgsqlConnection(connectionString); - await connection.OpenAsync(); - await PgMigrations.MigrateAsync(connection, default); - - var enabled = await TimescaleSupport.TryEnableAsync(connection, null, default); - Assert.SkipWhen(!enabled, "The live #4300 per-slot floor pins need TimescaleDB."); - await TimescaleSupport.ConvertToHypertablesAsync(connection, null, default); - await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, default); - - await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection) { CommandTimeout = SetupTimeoutSeconds }) + NpgsqlConnection? connection = null; + /* A skip (no TimescaleDB) or a failed setup throws before the caller owns the store, so the opener drops it + here. Left alone it would survive to the process-exit drain, which the test runner waits only 10 s for. */ + try { - await stop.ExecuteNonQueryAsync(); + var connectionString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { MaxAutoPrepare = 0 }.ConnectionString; + connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(); + await PgMigrations.MigrateAsync(connection, default); + + var enabled = await TimescaleSupport.TryEnableAsync(connection, null, default); + Assert.SkipWhen(!enabled, "The live #4300 per-slot floor pins need TimescaleDB."); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, default); + await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, default); + + await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection) { CommandTimeout = SetupTimeoutSeconds }) + { + await stop.ExecuteNonQueryAsync(); + } + + return (connection, scratch); } + catch + { + if (connection is not null) + { + await connection.DisposeAsync(); + } - return (connection, scratch); + await scratch.DisposeAsync(); + throw; + } } /// Hour 0 (the raw floor) holds ONLY CPU-unknown rows: delta_worker_time = NULL, @@ -139,6 +154,7 @@ public async Task FloorCase_DbSlotOwnFloorSkipsCpuUnknownHour_Covered() { await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, (_, _) => Task.CompletedTask); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -177,6 +193,7 @@ public async Task QueryGrainCase_HoleAtOwnFloor_ShortThenCovered() { await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, (_, _) => Task.CompletedTask); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } } diff --git a/Darling/Darling.Tests/RawPurgeConvergeLiveTests.cs b/Darling/Darling.Tests/RawPurgeConvergeLiveTests.cs index 00be7a67d4..0df21cb56e 100644 --- a/Darling/Darling.Tests/RawPurgeConvergeLiveTests.cs +++ b/Darling/Darling.Tests/RawPurgeConvergeLiveTests.cs @@ -38,23 +38,38 @@ public sealed class RawPurgeConvergeLiveTests "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live #4391 converge pins."); var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, default); + NpgsqlConnection? connection = null; + /* A skip (no TimescaleDB) or a failed setup throws before the caller owns the store, so the opener drops it + here. Left alone it would survive to the process-exit drain, which the test runner waits only 10 s for. */ + try + { + var connectionString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { MaxAutoPrepare = 0 }.ConnectionString; + connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(); + await PgMigrations.MigrateAsync(connection, default); - var connectionString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { MaxAutoPrepare = 0 }.ConnectionString; - var connection = new NpgsqlConnection(connectionString); - await connection.OpenAsync(); - await PgMigrations.MigrateAsync(connection, default); + var enabled = await TimescaleSupport.TryEnableAsync(connection, null, default); + Assert.SkipWhen(!enabled, "The live #4391 converge pins need TimescaleDB."); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, default); + await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, default); - var enabled = await TimescaleSupport.TryEnableAsync(connection, null, default); - Assert.SkipWhen(!enabled, "The live #4391 converge pins need TimescaleDB."); - await TimescaleSupport.ConvertToHypertablesAsync(connection, null, default); - await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, default); + await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection) { CommandTimeout = SetupTimeoutSeconds }) + { + await stop.ExecuteNonQueryAsync(); + } - await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection) { CommandTimeout = SetupTimeoutSeconds }) - { - await stop.ExecuteNonQueryAsync(); + return (connection, scratch); } + catch + { + if (connection is not null) + { + await connection.DisposeAsync(); + } - return (connection, scratch); + await scratch.DisposeAsync(); + throw; + } } private static async Task ArmRawJobAsync(NpgsqlConnection connection, string relation) @@ -195,6 +210,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -253,6 +269,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -336,6 +353,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } diff --git a/Darling/Darling.Tests/RawPurgeTriggerGateErrorTests.cs b/Darling/Darling.Tests/RawPurgeTriggerGateErrorTests.cs index fec31ee8ea..47a273ef55 100644 --- a/Darling/Darling.Tests/RawPurgeTriggerGateErrorTests.cs +++ b/Darling/Darling.Tests/RawPurgeTriggerGateErrorTests.cs @@ -60,26 +60,41 @@ public void RepairEpochStampAllowed_TrueOnlyWithZeroFailures() "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live #4391 gate_error pin."); var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, default); - - /* Max Auto Prepare=0: RunRetentionPurgeJobSql is a multi-statement string run through NpgsqlCommand, - and this rig's connection reruns other multi-statement SQL enough times over its life that - Npgsql's default auto-prepare threshold can trigger on it — same reasoning as RawPurgeTriggerLiveTests. */ - var connectionString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { MaxAutoPrepare = 0 }.ConnectionString; - var connection = new NpgsqlConnection(connectionString); - await connection.OpenAsync(); - await PgMigrations.MigrateAsync(connection, default); - - var enabled = await TimescaleSupport.TryEnableAsync(connection, null, default); - Assert.SkipWhen(!enabled, "The live #4391 gate_error pin needs TimescaleDB."); - await TimescaleSupport.ConvertToHypertablesAsync(connection, null, default); - await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, default); - - await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection) { CommandTimeout = SetupTimeoutSeconds }) + NpgsqlConnection? connection = null; + /* A skip (no TimescaleDB) or a failed setup throws before the caller owns the store, so the opener drops it + here. Left alone it would survive to the process-exit drain, which the test runner waits only 10 s for. */ + try { - await stop.ExecuteNonQueryAsync(); + /* Max Auto Prepare=0: RunRetentionPurgeJobSql is a multi-statement string run through NpgsqlCommand, + and this rig's connection reruns other multi-statement SQL enough times over its life that + Npgsql's default auto-prepare threshold can trigger on it — same reasoning as RawPurgeTriggerLiveTests. */ + var connectionString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { MaxAutoPrepare = 0 }.ConnectionString; + connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(); + await PgMigrations.MigrateAsync(connection, default); + + var enabled = await TimescaleSupport.TryEnableAsync(connection, null, default); + Assert.SkipWhen(!enabled, "The live #4391 gate_error pin needs TimescaleDB."); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, default); + await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, default); + + await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection) { CommandTimeout = SetupTimeoutSeconds }) + { + await stop.ExecuteNonQueryAsync(); + } + + return (connection, scratch); } + catch + { + if (connection is not null) + { + await connection.DisposeAsync(); + } - return (connection, scratch); + await scratch.DisposeAsync(); + throw; + } } private static async Task ReadPostmasterEpochAsync(NpgsqlConnection connection) @@ -246,6 +261,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } } diff --git a/Darling/Darling.Tests/RawPurgeTriggerLiveTests.cs b/Darling/Darling.Tests/RawPurgeTriggerLiveTests.cs index dbacf5fab7..403ffe678d 100644 --- a/Darling/Darling.Tests/RawPurgeTriggerLiveTests.cs +++ b/Darling/Darling.Tests/RawPurgeTriggerLiveTests.cs @@ -40,28 +40,43 @@ public sealed class RawPurgeTriggerLiveTests "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live #4299 trigger pins."); var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, default); - - /* Max Auto Prepare=0: RunRetentionPurgeJobSql is a multi-statement string (BEGIN; ...; COMMIT;) run - through NpgsqlCommand, and this rig's connection reruns other multi-statement SQL enough times over - its life that Npgsql's default auto-prepare threshold can trigger on it, which Npgsql rejects with - 42601 ("cannot insert multiple commands into a prepared statement") — a client-side caching - artifact of this test's connection reuse, not a product behavior. */ - var connectionString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { MaxAutoPrepare = 0 }.ConnectionString; - var connection = new NpgsqlConnection(connectionString); - await connection.OpenAsync(); - await PgMigrations.MigrateAsync(connection, default); - - var enabled = await TimescaleSupport.TryEnableAsync(connection, null, default); - Assert.SkipWhen(!enabled, "The live #4299 trigger pins need TimescaleDB."); - await TimescaleSupport.ConvertToHypertablesAsync(connection, null, default); - await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, default); - - await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection) { CommandTimeout = SetupTimeoutSeconds }) + NpgsqlConnection? connection = null; + /* A skip (no TimescaleDB) or a failed setup throws before the caller owns the store, so the opener drops it + here. Left alone it would survive to the process-exit drain, which the test runner waits only 10 s for. */ + try { - await stop.ExecuteNonQueryAsync(); + /* Max Auto Prepare=0: RunRetentionPurgeJobSql is a multi-statement string (BEGIN; ...; COMMIT;) run + through NpgsqlCommand, and this rig's connection reruns other multi-statement SQL enough times over + its life that Npgsql's default auto-prepare threshold can trigger on it, which Npgsql rejects with + 42601 ("cannot insert multiple commands into a prepared statement") — a client-side caching + artifact of this test's connection reuse, not a product behavior. */ + var connectionString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { MaxAutoPrepare = 0 }.ConnectionString; + connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(); + await PgMigrations.MigrateAsync(connection, default); + + var enabled = await TimescaleSupport.TryEnableAsync(connection, null, default); + Assert.SkipWhen(!enabled, "The live #4299 trigger pins need TimescaleDB."); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, default); + await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, default); + + await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection) { CommandTimeout = SetupTimeoutSeconds }) + { + await stop.ExecuteNonQueryAsync(); + } + + return (connection, scratch); } + catch + { + if (connection is not null) + { + await connection.DisposeAsync(); + } - return (connection, scratch); + await scratch.DisposeAsync(); + throw; + } } private static async Task ReadPostmasterEpochAsync(NpgsqlConnection connection) @@ -190,6 +205,63 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); + } + } + + /// #5329: a store that took the io rollups holds raw hours the io views never saw (they are created WITH + /// NO DATA and their refresh policy reaches back one day), and the repair closes 24 buckets a start. The io + /// views must not hold the raw purge (the #1661 rule), so a purge over a range the io views only cover for the + /// last day still drops chunks and records "ran". RED on 9aae0d75c: the hole gate walked the io views and held + /// query_stats for as long as the io floor sat above raw's. + [Fact] + public async Task IoRollupsFilledOnlyForTheLastDay_DoNotHoldThePurge() + { + var (connection, scratch) = await OpenAsync(); + var bodySucceeded = false; + try + { + await ArmRawJobAsync(connection, Raw); + await SeedRawAsync(connection); + + var dropFrom = TimescaleSupport.AlignDown(DateTime.UtcNow.AddDays(-11), TimeSpan.FromHours(1)); + var dropTo = TimescaleSupport.AlignDown(DateTime.UtcNow, TimeSpan.FromHours(1)); + await RefreshSuccessorsAsync(connection, dropFrom, dropTo); + + /* What the io views hold on an upgraded store: the policy's one-day window and nothing earlier. */ + foreach (var io in new[] { TimescaleSupport.QueryStatsIoHourlyView, TimescaleSupport.ProcedureStatsIoHourlyView }) + { + await using var refresh = new NpgsqlCommand($"CALL refresh_continuous_aggregate('collect.{io}'::regclass, $1::timestamp, $2::timestamp)", connection) { CommandTimeout = SetupTimeoutSeconds }; + refresh.Parameters.AddWithValue(DateTime.SpecifyKind(dropTo.AddDays(-1), DateTimeKind.Unspecified)); + refresh.Parameters.AddWithValue(DateTime.SpecifyKind(dropTo, DateTimeKind.Unspecified)); + await refresh.ExecuteNonQueryAsync(); + } + + var periodic = await TimescaleSupport.EnsureRetentionPoliciesAsync(connection, null, TimescaleSupport.RetentionSweepPass.Periodic, default); + Assert.True(periodic.Armed >= 1, "the seeded and fully-refreshed raw relation must read Covered this pass"); + await StampCurrentEpochAsync(connection, Raw); + + var before = await ChunkCountAsync(connection); + var logger = new CapturingTestLogger(); + await DarlingWorker.TriggerRawPurgeCoreAsync(connection, logger, default); + + var after = await ChunkCountAsync(connection); + Assert.True(after < before, $"an io rollup that is only filled for the last day must not hold the raw purge (before={before}, after={after}); log: {logger.Joined}"); + var rec = await TimescaleSupport.ReadRawLastPurgeOutcomeAsync(connection, Raw, null, default); + Assert.NotNull(rec); + Assert.Equal("ran", rec!.Outcome); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (cleanup, cleanupCt) => + { + var batch = new LiveCleanupBatch(cleanup); + await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); + }); + await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -241,6 +313,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -300,6 +373,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -365,6 +439,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -417,6 +492,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -504,6 +580,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -585,6 +662,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -728,6 +806,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( var batch = new LiveCleanupBatch(cleanup); await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); + await scratch.DisposeAsync(); } } } diff --git a/Darling/Darling.Tests/RawRepairEpochTriggerLiveTests.cs b/Darling/Darling.Tests/RawRepairEpochTriggerLiveTests.cs index a4ac60461e..78f6f03b1a 100644 --- a/Darling/Darling.Tests/RawRepairEpochTriggerLiveTests.cs +++ b/Darling/Darling.Tests/RawRepairEpochTriggerLiveTests.cs @@ -37,20 +37,36 @@ public sealed class RawRepairEpochTriggerLiveTests "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live #4299 trigger pins."); var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, default); - var connection = new NpgsqlConnection(scratch.ConnectionString); - await connection.OpenAsync(); - await PgMigrations.MigrateAsync(connection, default); + NpgsqlConnection? connection = null; + /* A skip (no TimescaleDB) or a failed setup throws before the caller owns the store, so the opener drops it + here. Left alone it would survive to the process-exit drain, which the test runner waits only 10 s for. */ + try + { + connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(); + await PgMigrations.MigrateAsync(connection, default); - var enabled = await TimescaleSupport.TryEnableAsync(connection, null, default); - Assert.SkipWhen(!enabled, "The live #4299 trigger pins need TimescaleDB."); - await TimescaleSupport.ConvertToHypertablesAsync(connection, null, default); + var enabled = await TimescaleSupport.TryEnableAsync(connection, null, default); + Assert.SkipWhen(!enabled, "The live #4299 trigger pins need TimescaleDB."); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, default); - await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection) { CommandTimeout = SetupTimeoutSeconds }) - { - await stop.ExecuteNonQueryAsync(); + await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection) { CommandTimeout = SetupTimeoutSeconds }) + { + await stop.ExecuteNonQueryAsync(); + } + + return (connection, scratch); } + catch + { + if (connection is not null) + { + await connection.DisposeAsync(); + } - return (connection, scratch); + await scratch.DisposeAsync(); + throw; + } } private static async Task ReadPostmasterEpochAsync(NpgsqlConnection connection) @@ -103,6 +119,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -149,6 +166,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -181,6 +199,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -232,6 +251,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } } diff --git a/Darling/Darling.Tests/RawRetentionPurgeJobLiveTests.cs b/Darling/Darling.Tests/RawRetentionPurgeJobLiveTests.cs index b7b2ad6c5d..884a281526 100644 --- a/Darling/Darling.Tests/RawRetentionPurgeJobLiveTests.cs +++ b/Darling/Darling.Tests/RawRetentionPurgeJobLiveTests.cs @@ -30,6 +30,7 @@ namespace Darling.Tests; /// #1776 own-store: deliberately NOT [Collection("live-postgres")]. Goes through /// and never touches the shared database's tables. /// +[Collection("timing")] public sealed class RawRetentionPurgeJobLiveTests { private const string Raw = "query_stats"; diff --git a/Darling/Darling.Tests/RawTablesLeaveCatalogSweepLiveTests.cs b/Darling/Darling.Tests/RawTablesLeaveCatalogSweepLiveTests.cs index d18945760a..45300f7554 100644 --- a/Darling/Darling.Tests/RawTablesLeaveCatalogSweepLiveTests.cs +++ b/Darling/Darling.Tests/RawTablesLeaveCatalogSweepLiveTests.cs @@ -45,25 +45,40 @@ public sealed class RawTablesLeaveCatalogSweepLiveTests "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live #4427 pins."); var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, default); - - /* Max Auto Prepare=0: see RawPurgeTriggerLiveTests — RunRetentionPurgeJobSql is multi-statement and - this rig's connection reuse can trip Npgsql's auto-prepare cache on it. */ - var connectionString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { MaxAutoPrepare = 0 }.ConnectionString; - var connection = new NpgsqlConnection(connectionString); - await connection.OpenAsync(); - await PgMigrations.MigrateAsync(connection, default); - - var enabled = await TimescaleSupport.TryEnableAsync(connection, null, default); - Assert.SkipWhen(!enabled, "The live #4427 pins need TimescaleDB."); - await TimescaleSupport.ConvertToHypertablesAsync(connection, null, default); - await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, default); - - await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection) { CommandTimeout = SetupTimeoutSeconds }) + NpgsqlConnection? connection = null; + /* A skip (no TimescaleDB) or a failed setup throws before the caller owns the store, so the opener drops it + here. Left alone it would survive to the process-exit drain, which the test runner waits only 10 s for. */ + try { - await stop.ExecuteNonQueryAsync(); + /* Max Auto Prepare=0: see RawPurgeTriggerLiveTests — RunRetentionPurgeJobSql is multi-statement and + this rig's connection reuse can trip Npgsql's auto-prepare cache on it. */ + var connectionString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { MaxAutoPrepare = 0 }.ConnectionString; + connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(); + await PgMigrations.MigrateAsync(connection, default); + + var enabled = await TimescaleSupport.TryEnableAsync(connection, null, default); + Assert.SkipWhen(!enabled, "The live #4427 pins need TimescaleDB."); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, default); + await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, default); + + await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection) { CommandTimeout = SetupTimeoutSeconds }) + { + await stop.ExecuteNonQueryAsync(); + } + + return (connection, scratch); } + catch + { + if (connection is not null) + { + await connection.DisposeAsync(); + } - return (connection, scratch); + await scratch.DisposeAsync(); + throw; + } } private static async Task ArmRawJobAsync(NpgsqlConnection connection, string relation) @@ -199,6 +214,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -277,6 +293,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await cleanupCommand.ExecuteNonQueryAsync(cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -338,6 +355,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -407,6 +425,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -475,6 +494,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } } diff --git a/Darling/Darling.Tests/RawWindowFloorMcpLiveTests.cs b/Darling/Darling.Tests/RawWindowFloorMcpLiveTests.cs index 7e8152ab21..6ef807495e 100644 --- a/Darling/Darling.Tests/RawWindowFloorMcpLiveTests.cs +++ b/Darling/Darling.Tests/RawWindowFloorMcpLiveTests.cs @@ -98,6 +98,77 @@ await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) = } } + /// + /// #4966: effective_start names its instant as UTC, with the trailing Z, on a cut window and on a covered + /// one alike. A cut window serves from the floor read off the store (a naive instant), a covered one from the + /// requested start (already UTC), and the zone marker used to come and go with window_truncated. Both + /// tools are read through the code path a caller reaches, so the text is the one a client parses. + /// + [Fact] + public async Task RawTopReads_PrintEffectiveStartInUtc_ForACutWindowAndACoveredOne() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live raw-window-floor test."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + var now = DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow).AddMinutes(-2); + + /* Cut: the raw tier holds one row from five minutes ago, so a 24-hour ask is served from that floor. The + instant is the row's own, never shifted, and it is named as UTC. */ + var recent = now.AddMinutes(-5); + await PlantQueryAsync(connection, ct, recent); + await PlantProcedureAsync(connection, ct, recent); + var floorText = DateTime.SpecifyKind(recent, DateTimeKind.Utc).ToString("o"); + + var cutQueries = JsonDocument.Parse(await DarlingMcpDataTools.GetTopQueriesByCpu(postgres, ServerName, 24)).RootElement; + Assert.True(cutQueries.GetProperty("window_truncated").GetBoolean()); + Assert.Equal(floorText, cutQueries.GetProperty("effective_start").GetString()); + + var cutProcedures = JsonDocument.Parse(await DarlingMcpDataTools.GetTopProceduresByCpu(postgres, ServerName, 24)).RootElement; + Assert.True(cutProcedures.GetProperty("window_truncated").GetBoolean()); + Assert.Equal(floorText, cutProcedures.GetProperty("effective_start").GetString()); + + /* Covered: a row from before the window's start (three hours back, a two-hour ask) means the window was + served whole, so the served start is the requested one. */ + var older = now.AddHours(-3); + await PlantQueryAsync(connection, ct, older); + await PlantProcedureAsync(connection, ct, older); + + var coveredQueries = JsonDocument.Parse(await DarlingMcpDataTools.GetTopQueriesByCpu(postgres, ServerName, 2)).RootElement; + Assert.False(coveredQueries.GetProperty("window_truncated").GetBoolean()); + AssertUtcStamp(coveredQueries.GetProperty("effective_start").GetString()); + + var coveredProcedures = JsonDocument.Parse(await DarlingMcpDataTools.GetTopProceduresByCpu(postgres, ServerName, 2)).RootElement; + Assert.False(coveredProcedures.GetProperty("window_truncated").GetBoolean()); + AssertUtcStamp(coveredProcedures.GetProperty("effective_start").GetString()); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// A round-trip ISO-8601 stamp that names UTC: it ends in Z and reads back as a UTC instant. + private static void AssertUtcStamp(string? stamp) + { + Assert.NotNull(stamp); + Assert.EndsWith("Z", stamp, StringComparison.Ordinal); + Assert.Equal(DateTimeKind.Utc, DateTime.Parse(stamp, System.Globalization.CultureInfo.InvariantCulture, System.Globalization.DateTimeStyles.RoundtripKind).Kind); + } + private static Task PlantQueryAsync(NpgsqlConnection connection, CancellationToken ct, DateTime at) => DarlingMcpTestData.ExecAsync(connection, ct, @"INSERT INTO query_stats (collection_id, collection_time, server_id, server_name, database_name, query_hash, query_plan_hash, sql_handle, plan_handle, query_text, delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, min_dop, max_dop) @@ -124,6 +195,7 @@ private static async Task DeleteRowsAsync(NpgsqlConnection connection, Cancellat /// their window floor through the ONE shared probe (), never a hand-copied SQL /// constant per table — the defect #2364 would have repeated twice more. /// +[Trait("Stage", "Guard")] public sealed class RawWindowFloorSharedHelperSourcePinTests { private static string ReaderSource => RepoFile.ReadRepoFile( diff --git a/Darling/Darling.Tests/RawWindowFloorQuietStartTests.cs b/Darling/Darling.Tests/RawWindowFloorQuietStartTests.cs new file mode 100644 index 0000000000..d956b86ae1 --- /dev/null +++ b/Darling/Darling.Tests/RawWindowFloorQuietStartTests.cs @@ -0,0 +1,405 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Npgsql; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// The Queries tab's "Showing since" banner and the MCP tools' window_truncated read the raw window floor. +/// A floor found inside the window called a quiet start a cut: a server with older rows, quiet for the first day of +/// the window, was reported as holding less than the window. The floor is now the oldest row at or before the +/// window's end, the same way the Custom Views panels find where their data starts, and it is still null when the +/// window itself holds nothing, which the MCP tools report as an empty window. +/// +/* #1776 own-store: deliberately NOT [Collection("live-postgres")]. Every test here reaches DARLING_TEST_PG only to + CREATE and DROP its own database through ScratchPostgres, then works entirely inside it. */ +[Trait("Reads", "Lite")] +public sealed class RawWindowFloorQuietStartLiveTests +{ + private const int QuietServerId = -497201; + private const string QuietServerName = "raw-floor-quiet-start"; + private const int EndedServerId = -497202; + private const string EndedServerName = "raw-floor-ended"; + private const int RecentServerId = -497203; + private const string RecentServerName = "raw-floor-recent"; + + [Fact] + public async Task AQuietStartInsideTheWindow_IsNotACut_AgainstDevPostgres() + { + var ct = TestContext.Current.CancellationToken; + await using var store = await SeededStore.CreateAsync(ct); + var start = store.End.AddDays(-7); + + var floor = await RawWindowFloor.GetAsync(store.DataSource, RawWindowFloor.Table.QueryStats, QuietServerId, start, store.End, cancellationToken: ct); + + Assert.Equal(store.End.AddDays(-8), floor); + Assert.False(RawWindowFloor.IsTruncated(floor, start)); + Assert.Equal(start, RawWindowFloor.EffectiveStart(floor, start)); + } + + [Fact] + public async Task AServerWhoseRowsEndBeforeTheWindow_ReadsAsNothing_AgainstDevPostgres() + { + var ct = TestContext.Current.CancellationToken; + await using var store = await SeededStore.CreateAsync(ct); + var start = store.End.AddDays(-7); + + var floor = await RawWindowFloor.GetAsync(store.DataSource, RawWindowFloor.Table.QueryStats, EndedServerId, start, store.End, cancellationToken: ct); + + Assert.Null(floor); + } + + [Fact] + public async Task ARecentServer_IsCutAtItsFirstRow_AgainstDevPostgres() + { + var ct = TestContext.Current.CancellationToken; + await using var store = await SeededStore.CreateAsync(ct); + var start = store.End.AddDays(-7); + + var floor = await RawWindowFloor.GetAsync(store.DataSource, RawWindowFloor.Table.QueryStats, RecentServerId, start, store.End, cancellationToken: ct); + + Assert.Equal(store.End.AddDays(-3), floor); + Assert.True(RawWindowFloor.IsTruncated(floor, start)); + Assert.Equal(store.End.AddDays(-3), RawWindowFloor.EffectiveStart(floor, start)); + } + + private sealed class SeededStore : IAsyncDisposable + { + private readonly ScratchPostgres _scratch; + + private SeededStore(ScratchPostgres scratch, NpgsqlDataSource dataSource, DateTime end) + { + _scratch = scratch; + DataSource = dataSource; + End = end; + } + + public NpgsqlDataSource DataSource { get; } + + public DateTime End { get; } + + public static async Task CreateAsync(CancellationToken ct) + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live raw window floor tests."); + + var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + try + { + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DarlingMcpTestData.RegisterServerAsync(connection, QuietServerId, QuietServerName, ct); + await DarlingMcpTestData.RegisterServerAsync(connection, EndedServerId, EndedServerName, ct); + await DarlingMcpTestData.RegisterServerAsync(connection, RecentServerId, RecentServerName, ct); + + var now = DateTime.UtcNow; + var end = new DateTime(now.Year, now.Month, now.Day, now.Hour, now.Minute, 0, DateTimeKind.Utc); + + /* One old row, then nothing until a day into a 7-day window. */ + await InsertEveryHalfHourAsync(connection, QuietServerId, QuietServerName, end.AddDays(-8), end.AddDays(-8), ct); + await InsertEveryHalfHourAsync(connection, QuietServerId, QuietServerName, end.AddDays(-6), end, ct); + /* Rows that all end before a 7-day window starts. */ + await InsertEveryHalfHourAsync(connection, EndedServerId, EndedServerName, end.AddDays(-12), end.AddDays(-9), ct); + /* A server added 3 days ago. */ + await InsertEveryHalfHourAsync(connection, RecentServerId, RecentServerName, end.AddDays(-3), end, ct); + + return new SeededStore(scratch, NpgsqlDataSource.Create(scratch.ConnectionString), end); + } + catch + { + await scratch.DisposeAsync(); + throw; + } + } + + private static async Task InsertEveryHalfHourAsync( + NpgsqlConnection connection, int serverId, string serverName, DateTime firstUtc, DateTime lastUtc, CancellationToken ct) + { + await using var insert = new NpgsqlCommand(@" +INSERT INTO collect.query_stats + (collection_id, collection_time, server_id, server_name, database_name, query_hash, sql_handle, + delta_worker_time, delta_elapsed_time, delta_execution_count, sample_interval_seconds) +SELECT + row_number() OVER (), + t, + $1, + $2, + 'FloorDb', + 'HASH1', + '0x01', + 1000, + 1000, + 10, + 300 +FROM generate_series($3::timestamp, $4::timestamp, interval '30 minutes') AS t", connection); + insert.Parameters.AddWithValue(serverId); + insert.Parameters.AddWithValue(serverName); + insert.Parameters.AddWithValue(DateTime.SpecifyKind(firstUtc, DateTimeKind.Unspecified)); + insert.Parameters.AddWithValue(DateTime.SpecifyKind(lastUtc, DateTimeKind.Unspecified)); + await insert.ExecuteNonQueryAsync(ct); + } + + public async ValueTask DisposeAsync() + { + await DataSource.DisposeAsync(); + await _scratch.DisposeAsync(); + } + } +} + +/// +/// The served window never starts before the one asked for. A floor older than the window's start means the rows +/// reach back past it, so the window was served whole. +/// +[Trait("Reads", "Lite")] +public sealed class RawWindowFloorEffectiveStartTests +{ + private static readonly DateTime s_start = new(2026, 9, 25, 12, 0, 0, DateTimeKind.Utc); + + [Fact] + public void AFloorBeforeTheStart_ServesTheWholeWindow() => + Assert.Equal(s_start, RawWindowFloor.EffectiveStart(s_start.AddDays(-1), s_start)); + + [Fact] + public void AFloorAfterTheStart_IsWhereTheWindowWasServedFrom() => + Assert.Equal(s_start.AddHours(2), RawWindowFloor.EffectiveStart(s_start.AddHours(2), s_start)); + + [Fact] + public void NoFloor_ReportsTheRequestedStart() => + Assert.Equal(s_start, RawWindowFloor.EffectiveStart(null, s_start)); + + private const string NotCoveredHead = + "The store holds no collection of query_snapshots for this server in this window, so nothing was read, and this empty answer is not a report that nothing happened. "; + + private const string NotCoveredTail = + "The window may reach further back than the store retains, this server may have been monitored for less time than that, or collection may have stopped; get_collection_health shows which."; + + private const string TruncatedSentence = + "The window reaches further back than this server's raw query_snapshots retains (or this server has been monitored for less time than that), so the older part of it was not read."; + + private static async Task<(McpWindowNotice Notice, int Probes)> ReadAsync( + TimeSpan window, DateTime? floor, bool emptyAnswer, string? tail = null) + { + var probes = 0; + var notice = await DarlingMcpWindowNotice.ReadAsync( + () => { probes++; return Task.FromResult(floor); }, s_start, s_start + window, "query_snapshots", tail, emptyAnswer); + return (notice, probes); + } + + /// #4966: a probe that throws costs the notice, never the answer: no verdict, no hints, and a Warning. + [Fact] + public async Task AProbeThatThrows_AnswersNoNotice_AndLogsAWarning() + { + var logger = new CapturingLogger(); + var notice = await DarlingMcpWindowNotice.ReadAsync( + () => throw new TimeoutException("deadline"), s_start, s_start + TimeSpan.FromDays(7), "query_snapshots", logger: logger); + + Assert.True(notice.IsUnavailable); + Assert.False(notice.WindowTruncated); + Assert.Null(notice.EffectiveStart); + Assert.Null(notice.AsHints()); + Assert.Equal([LogLevel.Warning], logger.Levels); + + /* An empty answer fails the same way. */ + Assert.True((await DarlingMcpWindowNotice.ReadAsync( + () => throw new TimeoutException("deadline"), s_start, s_start + TimeSpan.FromHours(1), "query_snapshots", emptyAnswer: true)).IsUnavailable); + } + + /// #4966: only the CALLER's cancellation goes through; the probe's own deadline is a failed probe. + [Fact] + public async Task ACancelledCaller_StillCancels_ButAProbeDeadlineDoesNot() + { + using var cts = new CancellationTokenSource(); + await cts.CancelAsync(); + await Assert.ThrowsAnyAsync(() => DarlingMcpWindowNotice.ReadAsync( + () => throw new OperationCanceledException(cts.Token), s_start, s_start + TimeSpan.FromDays(7), "query_snapshots", cancellationToken: cts.Token)); + + var notice = await DarlingMcpWindowNotice.ReadAsync( + () => throw new OperationCanceledException("the probe's own deadline"), s_start, s_start + TimeSpan.FromDays(7), "query_snapshots", + cancellationToken: CancellationToken.None); + Assert.True(notice.IsUnavailable); + } + + private sealed class CapturingLogger : ILogger + { + public List Levels { get; } = []; + + public IDisposable? BeginScope(TState state) where TState : notnull => null; + + public bool IsEnabled(LogLevel logLevel) => true; + + public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) => + Levels.Add(logLevel); + } + + /// #4966: an answer with rows over a window of 90 minutes or less never asks the store, and is covered at the start that was asked for. + [Theory] + [InlineData(60)] + [InlineData(90)] + public async Task ARowsAnswer_OverNinetyMinutesOrLess_MakesNoProbe_AndIsCoveredAtTheAskedStart(int minutes) + { + var (notice, probes) = await ReadAsync(TimeSpan.FromMinutes(minutes), s_start.AddMinutes(80), emptyAnswer: false); + + Assert.Equal(0, probes); + Assert.Equal(McpHelpers.FormatEffectiveStart(s_start), notice.EffectiveStart); + Assert.False(notice.WindowTruncated); + Assert.Null(notice.TruncationNote); + } + + [Fact] + public async Task ARowsAnswer_PastNinetyMinutes_IsProbed_AndAFloorWithinTheSlackIsStillCovered() + { + var (notice, probes) = await ReadAsync(TimeSpan.FromMinutes(91), s_start.AddMinutes(90), emptyAnswer: false); + + Assert.Equal(1, probes); + Assert.False(notice.WindowTruncated); + Assert.Null(notice.TruncationNote); + Assert.Equal(McpHelpers.FormatEffectiveStart(s_start.AddMinutes(90)), notice.EffectiveStart); + } + + /// An empty answer is probed at any length, and a probe that finds nothing says NOT covered, with the sentence Lite says. + [Fact] + public async Task AnEmptyAnswer_AtSixtyMinutes_IsProbed_AndANullFloorIsNotCovered() + { + var (notice, probes) = await ReadAsync(TimeSpan.FromMinutes(60), floor: null, emptyAnswer: true); + + Assert.Equal(1, probes); + Assert.Null(notice.EffectiveStart); + Assert.True(notice.WindowTruncated); + Assert.Equal(NotCoveredHead + NotCoveredTail, notice.TruncationNote); + } + + [Fact] + public async Task AnEmptyAnswer_WhoseProbeFindsCoverage_IsCoveredAtTheFloor_OrTheStartWhenTheFloorIsEarlier() + { + var (inside, _) = await ReadAsync(TimeSpan.FromMinutes(60), s_start.AddMinutes(35), emptyAnswer: true); + Assert.Equal(McpHelpers.FormatEffectiveStart(s_start.AddMinutes(35)), inside.EffectiveStart); + Assert.False(inside.WindowTruncated); + Assert.Null(inside.TruncationNote); + + var (before, _) = await ReadAsync(TimeSpan.FromMinutes(60), s_start.AddDays(-3), emptyAnswer: true); + Assert.Equal(McpHelpers.FormatEffectiveStart(s_start), before.EffectiveStart); + Assert.False(before.WindowTruncated); + } + + [Fact] + public async Task AFloorPastTheSlack_IsTruncated_NamesTheTable_AndCarriesTheTail() + { + var floor = s_start.AddDays(2); + var (notice, probes) = await ReadAsync(TimeSpan.FromDays(7), floor, emptyAnswer: false, tail: "More."); + + Assert.Equal(1, probes); + Assert.True(notice.WindowTruncated); + Assert.Equal(McpHelpers.FormatEffectiveStart(floor), notice.EffectiveStart); + Assert.EndsWith("Z", notice.EffectiveStart, StringComparison.Ordinal); + Assert.Equal(TruncatedSentence + " More.", notice.TruncationNote); + Assert.Equal(TruncatedSentence, (await ReadAsync(TimeSpan.FromDays(7), floor, emptyAnswer: false)).Notice.TruncationNote); + } + + [Fact] + public void TheHints_CarryTheThreeKeys_WithTheSameValues() + { + var notice = DarlingMcpWindowNotice.Build(null, s_start, "waiting_tasks", emptyAnswer: true); + var json = System.Text.Json.JsonSerializer.SerializeToElement(notice.AsHints()); + + Assert.Equal(System.Text.Json.JsonValueKind.Null, json.GetProperty("effective_start").ValueKind); + Assert.True(json.GetProperty("window_truncated").GetBoolean()); + Assert.Equal(notice.TruncationNote, json.GetProperty("truncation_note").GetString()); + } + + /// + /// The two sentences are copies of Lite's (#4966), so a client reads one wording from either app. Lite's source is + /// read here and must hold each sentence as a string literal, and Darling's notice must be exactly those literals + /// with the table put in. A reword on one side fails until the other follows. + /// + [Fact] + public void TheNoticeSentences_EqualLitesText() + { + var lite = ReadRepoFile("Lite", "Mcp", "McpQueryTools.cs").ReplaceLineEndings("\n"); + + const string head = "The store holds no collection of {table} for this server in this window, so nothing was read, and this empty answer is not a report that nothing happened. "; + Assert.Contains("$\"" + head + "\"", lite, StringComparison.Ordinal); + Assert.Contains("\"" + NotCoveredTail + "\"", lite, StringComparison.Ordinal); + Assert.Equal(NotCoveredHead + NotCoveredTail, DarlingMcpWindowNotice.Build(null, s_start, "query_snapshots", emptyAnswer: true).TruncationNote); + + const string listHead = "The store holds no collection of {table} for this server in this window, so no windowed rows were read, and a list with no rows is not a report that nothing happened. "; + Assert.Contains("$\"" + listHead + "\"", lite, StringComparison.Ordinal); + Assert.Equal( + listHead.Replace("{table}", "query_snapshots", StringComparison.Ordinal) + NotCoveredTail, + DarlingMcpWindowNotice.Build(null, s_start, "query_snapshots", listOnly: true).TruncationNote); + + const string truncated = "The window reaches further back than this server's raw {table} retains (or this server has been monitored for less time than that), so the older part of it was not read."; + Assert.Contains("$\"" + truncated + "\"", lite, StringComparison.Ordinal); + Assert.Equal(TruncatedSentence, DarlingMcpWindowNotice.Build(s_start.AddDays(1), s_start, "query_snapshots").TruncationNote); + } + + /// The helper probes with GetAsync: GetForServerAsync answers null for a window of 90 minutes or less, which an empty answer would read as "not covered". + [Fact] + public void TheHelperProbes_WithGetAsync_NeverGetForServerAsync() + { + var source = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "Mcp", "DarlingMcpWindowNotice.cs"); + Assert.Contains("DataWindowFloor.GetAsync(", source, StringComparison.Ordinal); + Assert.DoesNotContain("GetForServerAsync(", source.Replace("", ""), StringComparison.Ordinal); + } + + /// The floor walks each table's index in time order, so the walk stops at the first row it meets. + [Theory] + [InlineData("query_stats")] + [InlineData("procedure_stats")] + [InlineData("query_store_stats")] + public void EveryRawFloorTable_HasTheIndexTheWalkUses(string table) => + Assert.True(DataWindowFloor.Source.TryForCollectorTable(table, out _)); + + /// + /// Every MCP tool takes its served start from and its truncation + /// verdict from , so none reports a start before the window it was asked + /// about, and none keeps a tolerance of its own. The clutter tool once restated the 90 minutes as a private + /// constant and compared the floor with requestedStart + by hand, a copy that went stale when the floor + /// stopped being the first row inside the window. The lines below are the ways a tool computes either by hand. + /// + [Fact] + public void NoMcpTool_ComputesTheServedStartByHand() + { + var byHand = new[] + { + "floor ?? requestedStart", + "WindowFloorTolerance", + "FromMinutes(90)", + "requestedStart +", + "requestedStart.Add(", + }; + + var found = new List(); + foreach (var file in new[] { "DarlingMcpDataTools.cs", "DarlingMcpQueryStoreClutterTools.cs", "DarlingMcpQueryStoreHistoryTools.cs" }) + { + var text = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "Mcp", file); + foreach (var line in byHand) + { + if (text.Contains(line, StringComparison.Ordinal)) + { + found.Add(file + " has `" + line + "`"); + } + } + } + + Assert.True(found.Count == 0, "an MCP tool computes the served start or the truncation verdict by hand: " + string.Join("; ", found)); + } +} diff --git a/Darling/Darling.Tests/RawWindowFloorViewerPortTests.cs b/Darling/Darling.Tests/RawWindowFloorViewerPortTests.cs index 6382d17b02..527784c1cd 100644 --- a/Darling/Darling.Tests/RawWindowFloorViewerPortTests.cs +++ b/Darling/Darling.Tests/RawWindowFloorViewerPortTests.cs @@ -33,7 +33,7 @@ public sealed class RawWindowFloorViewerPortTests [Fact] public void UpdateTruncationBanner_ShowsSinceEffectiveStart_WhenTheFloorIsPastTheSlack() { - OnStaThread(() => + StaTestThread.Run(() => { var banner = new TextBlock(); var floor = RequestedStart.AddDays(3); @@ -48,7 +48,7 @@ public void UpdateTruncationBanner_ShowsSinceEffectiveStart_WhenTheFloorIsPastTh [Fact] public void UpdateTruncationBanner_StaysCollapsed_WhenTheFloorIsInsideTheSlack() { - OnStaThread(() => + StaTestThread.Run(() => { /* 30 minutes after the requested start — inside DurationTrendRouting.TruncationSlack's 90-minute allowance, so a normal cadence-start lag, not a retention cut (the #2364 / #4231 ruling). Seeded @@ -65,7 +65,7 @@ public void UpdateTruncationBanner_StaysCollapsed_WhenTheFloorIsInsideTheSlack() [Fact] public void UpdateTruncationBanner_StaysCollapsed_WhenTheProbeFoundNoFloorAtAll() { - OnStaThread(() => + StaTestThread.Run(() => { var banner = new TextBlock(); @@ -75,26 +75,6 @@ public void UpdateTruncationBanner_StaysCollapsed_WhenTheProbeFoundNoFloorAtAll( }); } - /// WPF objects require STA; same shape as Lite.Tests' MainWindowAccessKeyTests / Dashboard.Tests' - /// DataGridExport tests. - private static void OnStaThread(Action body) - { - Exception? error = null; - var thread = new Thread(() => - { - try { body(); } - catch (Exception ex) { error = ex; } - }); - thread.SetApartmentState(ApartmentState.STA); - thread.Start(); - thread.Join(); - - if (error is not null) - { - throw error; - } - } - private static string ViewerFile(string file) => ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", file); [Theory] @@ -114,7 +94,7 @@ makes the same check on the MCP reader. */ [Fact] public void UpdateTruncationBanner_TableServed_NamesTheEffectiveStartAndTheBound_AndTheRawSlicerFloor() { - OnStaThread(() => + StaTestThread.Run(() => { var wideStart = RequestedStart.AddDays(1); var plan = new QueryStoreIntervalWide.WideReadPlan( @@ -145,11 +125,75 @@ public void UpdateTruncationBanner_TableServed_NamesTheEffectiveStartAndTheBound }); } + /* #5329: an hourly-served grid starts at this server's first bucket, so "Showing since" names THAT start, and raw's floor (which + only the slicer and the comparison read) is the slicer's. The hourly tier suffix still follows. */ + [Fact] + public void UpdateTruncationBanner_HourlyServed_NamesTheGridsStart_AndRawsFloorAsTheSlicers() + { + StaTestThread.Run(() => + { + const string Suffix = " - aggregated hourly"; + var banner = new TextBlock(); + + /* Hourly keeps more than raw: the grid starts a day in, raw's floor is four days in. */ + ViewerServerTab.UpdateTruncationBanner(banner, RequestedStart.AddDays(4), RequestedStart, Suffix, + hourly: new ViewerServerTab.HourlyServed(RequestedStart.AddDays(1))); + Assert.StartsWith("Showing since ", banner.Text, StringComparison.Ordinal); + Assert.Contains(" · slicer since ", banner.Text, StringComparison.Ordinal); + Assert.EndsWith(Suffix, banner.Text, StringComparison.Ordinal); + var gridOnly = new TextBlock(); + ViewerServerTab.UpdateTruncationBanner(gridOnly, null, RequestedStart, Suffix, + hourly: new ViewerServerTab.HourlyServed(RequestedStart.AddDays(1))); + Assert.StartsWith("Showing since ", gridOnly.Text, StringComparison.Ordinal); + Assert.DoesNotContain("slicer", gridOnly.Text, StringComparison.Ordinal); + + /* A late hourly start after an aligned window start is named even when raw reaches the start. */ + var late = new TextBlock(); + ViewerServerTab.UpdateTruncationBanner(late, RequestedStart, RequestedStart, Suffix, + hourly: new ViewerServerTab.HourlyServed(RequestedStart.AddHours(5))); + Assert.StartsWith("Showing since ", late.Text, StringComparison.Ordinal); + Assert.DoesNotContain("slicer", late.Text, StringComparison.Ordinal); + + /* A start inside the slack is the hour alignment alone: the banner names the window, not a "since". */ + var aligned = new TextBlock(); + ViewerServerTab.UpdateTruncationBanner(aligned, null, RequestedStart, Suffix, + hourly: new ViewerServerTab.HourlyServed(RequestedStart.AddHours(1))); + Assert.StartsWith("Showing 2", aligned.Text, StringComparison.Ordinal); + Assert.DoesNotContain("since", aligned.Text, StringComparison.Ordinal); + + /* No bucket at all (an empty grid) with a cut raw floor: the slicer's shortfall only. */ + var none = new TextBlock(); + ViewerServerTab.UpdateTruncationBanner(none, RequestedStart.AddDays(4), RequestedStart, Suffix, + hourly: new ViewerServerTab.HourlyServed(null)); + Assert.StartsWith("Slicer since ", none.Text, StringComparison.Ordinal); + Assert.DoesNotContain("full window", none.Text, StringComparison.Ordinal); + + /* #5329: the usual hourly route (a cut raw floor, a first bucket inside the slack, and an edges note that says the + grid stops short) must not say "the grid shows the full window": that is the wide plan's claim. */ + const string Edges = " - aggregated hourly. Window edges: the data from a to b is not included"; + var usual = new TextBlock(); + ViewerServerTab.UpdateTruncationBanner(usual, RequestedStart.AddDays(4), RequestedStart, Edges, + hourly: new ViewerServerTab.HourlyServed(RequestedStart.AddHours(1))); + Assert.DoesNotContain("full window", usual.Text, StringComparison.Ordinal); + Assert.StartsWith("Showing 2", usual.Text, StringComparison.Ordinal); + Assert.Contains(" · slicer since ", usual.Text, StringComparison.Ordinal); + Assert.EndsWith(Edges, usual.Text, StringComparison.Ordinal); + + /* An empty grid (no bucket) names no start for it, and makes no full-window claim, with or without a cut raw floor. */ + var emptyNoSlicer = new TextBlock(); + ViewerServerTab.UpdateTruncationBanner(emptyNoSlicer, null, RequestedStart, Edges, + hourly: new ViewerServerTab.HourlyServed(null)); + Assert.DoesNotContain("full window", emptyNoSlicer.Text, StringComparison.Ordinal); + Assert.DoesNotContain("Showing", emptyNoSlicer.Text, StringComparison.Ordinal); + Assert.StartsWith("No rollup bucket in the window", emptyNoSlicer.Text, StringComparison.Ordinal); + }); + } + [Fact] public void UpdateTruncationBanner_WideBranch_NamesTheSlicerFloor_WhetherOrNotTheGridWasCut() { var tab = ViewerFile("ViewerServerTab.Queries.cs"); - var start = tab.IndexOf("if (widePlan?.EffectiveStart is DateTime wideStart)", StringComparison.Ordinal); + var start = tab.IndexOf("if (servedStart is not null || hourlyServed)", StringComparison.Ordinal); Assert.True(start >= 0); var branch = tab[start..tab.IndexOf("return;", start, StringComparison.Ordinal)]; Assert.Contains("var slicerTruncated = RawWindowFloor.IsTruncated(floor, requestedStartUtc);", branch, StringComparison.Ordinal); @@ -171,8 +215,8 @@ public void QueryStoreGrid_BindsThePlansReadStart_AndPassesThePlanToTheBanner() var tab = ViewerFile("ViewerServerTab.Queries.cs"); Assert.Contains("GetQueryStoreTopQueriesWithReachAsync(", tab, StringComparison.Ordinal); - Assert.Contains("UpdateTruncationBanner(QueryStoreTruncationBanner, await floorTask, startUtc, widePlan: widePlan)", tab, StringComparison.Ordinal); - Assert.Contains("QueryStoreIntervalWide.BannerReason(widePlan.Value.StartBound)", tab, StringComparison.Ordinal); + Assert.Contains("UpdateTruncationBanner(QueryStoreTruncationBanner, await DataStartOrNullAsync(floorTask, \"Query Store\"), startUtc, widePlan: widePlan)", tab, StringComparison.Ordinal); + Assert.Contains("QueryStoreIntervalWide.BannerReason(widePlan!.Value.StartBound)", tab, StringComparison.Ordinal); Assert.DoesNotContain("interval table keeps 9 days", tab, StringComparison.Ordinal); Assert.Contains(" · slicer since ", tab, StringComparison.Ordinal); } diff --git a/Darling/Darling.Tests/RdsAssumeRoleSeamTests.cs b/Darling/Darling.Tests/RdsAssumeRoleSeamTests.cs new file mode 100644 index 0000000000..c2582e0063 --- /dev/null +++ b/Darling/Darling.Tests/RdsAssumeRoleSeamTests.cs @@ -0,0 +1,391 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Amazon; +using Amazon.RDS; +using Amazon.RDS.Model; +using Amazon.Runtime; +using Npgsql; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Targets; +using Xunit; + +namespace Darling.Tests; + +/// +/// Where a server's own AWS role meets the RDS readers (#5452): the clients every call uses, the host check that runs +/// under the same role, a role edit that reconnects, and the worker arm that records a role the service cannot use. Each +/// test builds the real ingestor or log source over a fake STS and a recording RDS client, so no AWS call is made. +/// +public sealed class RdsAssumeRoleSeamTests +{ + private const string Role = "arn:aws:iam::123456789012:role/darling-monitor"; + private const string OtherRole = "arn:aws:iam::123456789012:role/darling-other"; + private const string ExternalId = "ext-7Hq2mZ9vLx"; + private const string InstanceHost = "solo.abc123.us-east-1.rds.amazonaws.com"; + private const string ConnectionString = "Host=solo.abc123.us-east-1.rds.amazonaws.com;Username=u;Database=d"; + private const string ProcessAccessKey = "AKIAPROCESSEXAMPLE"; + private const string AssumedAccessKey = "ASIAEXAMPLEASSUMED"; + + private static readonly AwsRoleKey KeyA = new(Role, ExternalId); + + private static RdsEndpointVerifier Enforcing() + => RdsEndpointVerifier.ForTests(enforce: true, loginProbe: (_, _) => Task.FromResult(null)); + + private static NpgsqlDataSource UnusedDataSource() => NpgsqlDataSource.Create("Host=localhost;Port=1;Database=unused"); + + private static AwsRoleCredentialCache Cache(FakeSts sts, params string[] allowed) + => new(AwsRoleAllowlist.From(allowed), () => "install-1", sts.Factory, FakeSts.Source, new ManualTimeProvider(), null, null); + + /// An RDS client that records the access key each Describe call was signed with. + private sealed class RecordingRds : AmazonRDSClient + { + private readonly AWSCredentials _credentials; + + public RecordingRds(AWSCredentials credentials) : base(credentials, RegionEndpoint.USEast1) => _credentials = credentials; + + public List InstanceDescribeKeys { get; } = []; + + public List OtherKeys { get; } = []; + + public override async Task DescribeDBInstancesAsync( + DescribeDBInstancesRequest request, CancellationToken cancellationToken = default) + { + InstanceDescribeKeys.Add((await _credentials.GetCredentialsAsync()).AccessKey); + + return new DescribeDBInstancesResponse + { + DBInstances = [new DBInstance { Endpoint = new Endpoint { Address = InstanceHost }, DbiResourceId = "db-EXAMPLE" }], + }; + } + + public override async Task DescribeDBLogFilesAsync( + DescribeDBLogFilesRequest request, CancellationToken cancellationToken = default) + { + OtherKeys.Add((await _credentials.GetCredentialsAsync()).AccessKey); + + return new DescribeDBLogFilesResponse + { + DescribeDBLogFiles = [new DescribeDBLogFilesDetails { LogFileName = "error/postgresql.log.2026-08-25-18", LastWritten = 1000 }], + }; + } + + public override async Task DownloadDBLogFilePortionAsync( + DownloadDBLogFilePortionRequest request, CancellationToken cancellationToken = default) + { + OtherKeys.Add((await _credentials.GetCredentialsAsync()).AccessKey); + + return new DownloadDBLogFilePortionResponse { LogFileData = string.Empty, Marker = "M" }; + } + } + + /// The role-aware factory a test hands the log source: process credentials for no role, the cache's handle for a role. + private sealed class Clients + { + private readonly AwsRoleCredentialCache _cache; + + public Clients(AwsRoleCredentialCache cache) => _cache = cache; + + public List Built { get; } = []; + + public Func Factory => (region, role) => + { + var client = new RecordingRds(role is null + ? new BasicAWSCredentials(ProcessAccessKey, "example-process-secret") + : _cache.For(role.Value, region)); + Built.Add(client); + return client; + }; + + public IEnumerable InstanceDescribeKeys => Built.SelectMany(c => c.InstanceDescribeKeys); + } + + private static Task ReadAsync(RdsLogSource source, AwsRoleKey? role) + => source.ReadNewestAsync(InstanceHost, RdsLogSource.LogFileKind.Stderr, 7, CancellationToken.None, ConnectionString, role); + + /* ---- the host check runs under the role ----------------------------------------------------------------- */ + + [Fact] + public async Task HostCheck_UnderARole_UsesTheAssumedCredentials_NotTheProcessOnes() + { + var sts = new FakeSts(); + var clients = new Clients(Cache(sts, Role)); + var source = new RdsLogSource(roleClientFactory: clients.Factory, verifier: Enforcing()); + + await ReadAsync(source, KeyA); + + Assert.Equal([AssumedAccessKey], clients.InstanceDescribeKeys.ToArray()); + Assert.All(clients.Built.SelectMany(c => c.OtherKeys), key => Assert.Equal(AssumedAccessKey, key)); + var request = Assert.Single(sts.Requests); + Assert.Equal(Role, request.RoleArn); + Assert.Equal(ExternalId, request.ExternalId); + } + + [Fact] + public async Task HostCheck_WithNoRole_KeepsTheProcessCredentials_AndMakesNoStsCall() + { + var sts = new FakeSts(); + var clients = new Clients(Cache(sts, Role)); + var source = new RdsLogSource(roleClientFactory: clients.Factory, verifier: Enforcing()); + + await ReadAsync(source, role: null); + + Assert.Equal([ProcessAccessKey], clients.InstanceDescribeKeys.ToArray()); + Assert.Equal(0, sts.Calls); + } + + [Fact] + public async Task HostCheck_TheVerdictIsKeptPerRole_AndPerExternalIdPresence() + { + var sts = new FakeSts(); + var clients = new Clients(Cache(sts, Role, OtherRole)); + var source = new RdsLogSource(roleClientFactory: clients.Factory, verifier: Enforcing()); + + await ReadAsync(source, KeyA); + Assert.Single(clients.InstanceDescribeKeys); + + // Same server, host and role: the held verdict answers, no second Describe. + await ReadAsync(source, KeyA); + Assert.Single(clients.InstanceDescribeKeys); + + // A different role is a different scope: the check asks AWS again. + await ReadAsync(source, new AwsRoleKey(OtherRole, ExternalId)); + Assert.Equal(2, clients.InstanceDescribeKeys.Count()); + + // The same role with the external ID removed is another scope too. + await ReadAsync(source, new AwsRoleKey(Role, null)); + Assert.Equal(3, clients.InstanceDescribeKeys.Count()); + + // And back to no role at all. + await ReadAsync(source, role: null); + Assert.Equal(4, clients.InstanceDescribeKeys.Count()); + } + + [Fact] + public void CredentialScope_NamesTheRoleAndWhetherAnIdIsSet_NeverTheId() + { + Assert.Equal(Role + "|ext=set", KeyA.CredentialScope); + Assert.Equal(Role + "|ext=none", new AwsRoleKey(Role, null).CredentialScope); + Assert.Equal(Role + "|ext=none", new AwsRoleKey(Role, string.Empty).CredentialScope); + Assert.Equal(string.Empty, AwsRoleKey.ScopeOf(null)); + Assert.DoesNotContain(ExternalId, KeyA.CredentialScope, StringComparison.Ordinal); + } + + /* ---- a role edit reconnects ----------------------------------------------------------------------------- */ + + private static MonitoredServer Server(string? role = null, string? externalId = null) => new() + { + Name = "alpha-pg-01", + Host = InstanceHost, + Engine = "postgresql", + AwsRoleArn = role, + AwsExternalId = externalId, + }; + + [Fact] + public void ServerDefinitionEquals_ARoleOnlyEdit_IsADifferentDefinition() + { + Assert.True(DarlingWorker.ServerDefinitionEquals(Server(Role, ExternalId), Server(Role, ExternalId))); + Assert.False(DarlingWorker.ServerDefinitionEquals(Server(), Server(Role))); + Assert.False(DarlingWorker.ServerDefinitionEquals(Server(Role), Server())); + Assert.False(DarlingWorker.ServerDefinitionEquals(Server(Role, ExternalId), Server(OtherRole, ExternalId))); + Assert.False(DarlingWorker.ServerDefinitionEquals(Server(Role, ExternalId), Server(Role, "ext-another-id"))); + Assert.False(DarlingWorker.ServerDefinitionEquals(Server(Role, ExternalId), Server(Role))); + } + + [Fact] + public void ServerDefinitionEquals_BlankAndNullRoleAreTheSame_AndAnIdWithNoRoleIsIgnored() + { + Assert.True(DarlingWorker.ServerDefinitionEquals(Server(null), Server(" "))); + Assert.True(DarlingWorker.ServerDefinitionEquals(Server(null, ExternalId), Server())); + } + + /* ---- a role the service cannot use never reaches AWS ---------------------------------------------------- */ + + private static async Task NotAllowedFrom(string reader, AwsRoleCredentialCache cache) + { + var verifier = Enforcing(); + Task call = reader switch + { + "cpu" => new RdsCpuIngestor(UnusedDataSource(), verifier: verifier, roles: cache) + .IngestAsync(1, "s", InstanceHost, ConnectionString, KeyA, CancellationToken.None), + "plan" => new RdsPlanIngestor(UnusedDataSource(), verifier: verifier, roles: cache) + .IngestAsync(1, "s", InstanceHost, false, ConnectionString, KeyA, CancellationToken.None), + "deadlock" => new RdsDeadlockIngestor(UnusedDataSource(), verifier: verifier, roles: cache) + .IngestAsync(1, "s", InstanceHost, false, false, ConnectionString, KeyA, CancellationToken.None), + "events" => new RdsLogEventIngestor(UnusedDataSource(), TestLogHashKeys.Fixed, verifier: verifier, roles: cache) + .IngestAsync(1, "s", InstanceHost, false, false, ConnectionString, KeyA, CancellationToken.None), + _ => throw new ArgumentOutOfRangeException(nameof(reader)), + }; + + var thrown = await Record.ExceptionAsync(() => call); + + // The exact type: wrapped in an "unavailable" exception it would be read by its text, not by its kind. + return Assert.IsType(thrown); + } + + [Theory] + [InlineData("cpu")] + [InlineData("plan")] + [InlineData("deadlock")] + [InlineData("events")] + public async Task EveryReader_ARoleTheListDoesNotAllow_ThrowsTheRoleExceptionItself_AndMakesNoStsCall(string reader) + { + var sts = new FakeSts(); + + var ex = await NotAllowedFrom(reader, Cache(sts)); + + Assert.Equal(AwsRoleAssumeKind.NotAllowed, ex.Kind); + Assert.Equal(0, sts.Calls); + Assert.NotNull(DarlingWorker.AwsRoleConfigurationFault(ex)); + Assert.DoesNotContain(ExternalId, ex.Message, StringComparison.Ordinal); + } + + [Theory] + [InlineData("cpu")] + [InlineData("plan")] + [InlineData("deadlock")] + [InlineData("events")] + public async Task EveryReader_AnStsDenial_ReachesTheWorkerAsAPermissionsFault_NotAsAnIamGrantGap(string reader) + { + var sts = new FakeSts((_, _) => throw FakeSts.Error( + "AccessDenied", + "User: arn:aws:sts::123456789012:assumed-role/monitoring-host/i-0abc is not authorized to perform: sts:AssumeRole on resource: " + Role)); + var verifier = Enforcing(); + var cache = Cache(sts, Role); + + // A real AmazonRDSClient over the cache's credentials: the denial surfaces from the first signed request. + Task call = reader switch + { + "cpu" => new RdsCpuIngestor(UnusedDataSource(), verifier: verifier, roles: cache) + .IngestAsync(1, "s", InstanceHost, ConnectionString, KeyA, CancellationToken.None), + "plan" => new RdsPlanIngestor(UnusedDataSource(), verifier: verifier, roles: cache) + .IngestAsync(1, "s", InstanceHost, false, ConnectionString, KeyA, CancellationToken.None), + "deadlock" => new RdsDeadlockIngestor(UnusedDataSource(), verifier: verifier, roles: cache) + .IngestAsync(1, "s", InstanceHost, false, false, ConnectionString, KeyA, CancellationToken.None), + _ => new RdsLogEventIngestor(UnusedDataSource(), TestLogHashKeys.Fixed, verifier: verifier, roles: cache) + .IngestAsync(1, "s", InstanceHost, false, false, ConnectionString, KeyA, CancellationToken.None), + }; + + var ex = Assert.IsType(await Record.ExceptionAsync(() => call)); + + Assert.Equal(AwsRoleAssumeKind.Denied, ex.Kind); + Assert.Same(ex, DarlingWorker.AwsRoleConfigurationFault(ex)); + + // The role arm sees through a wrapper that the arms after it would claim as a missing grant on the monitoring host's + // own IAM role, so the order of the arms is what keeps this row naming the role. + Assert.Same(ex, DarlingWorker.AwsRoleConfigurationFault(new RdsLogUnavailableException("wrapped", true, ex))); + Assert.Same(ex, DarlingWorker.AwsRoleConfigurationFault(new PiMetricsUnavailableException("wrapped", true, ex))); + + var logger = new CapturingAwsLogger(); + var note = DarlingWorker.AwsRoleFaultNote(logger, "alpha-pg-01", "pg_plan_capture", ex); + + Assert.Equal(ex.Message, note); + Assert.Contains(Role, note, StringComparison.Ordinal); + Assert.Contains("with the external ID set on this server", note, StringComparison.Ordinal); + Assert.DoesNotContain("lacks a grant", note, StringComparison.Ordinal); + Assert.Contains("PERMISSIONS", logger.All, StringComparison.Ordinal); + Assert.Contains(Role, logger.All, StringComparison.Ordinal); + Assert.DoesNotContain(ExternalId, note + logger.All, StringComparison.Ordinal); + } + + [Fact] + public async Task AnStsFailureThatIsNotAConfiguration_IsNotClaimedByTheRoleArm_ButStillSurfacesUnwrapped() + { + var sts = new FakeSts((_, _) => throw FakeSts.Error("ExpiredToken", "The security token included in the request is expired")); + var ingestor = new RdsCpuIngestor(UnusedDataSource(), verifier: Enforcing(), roles: Cache(sts, Role)); + + var ex = Assert.IsType(await Record.ExceptionAsync( + () => ingestor.IngestAsync(1, "s", InstanceHost, ConnectionString, KeyA, CancellationToken.None))); + + Assert.Equal(AwsRoleAssumeKind.SourceCredentials, ex.Kind); + Assert.False(ex.IsConfiguration); + Assert.Null(DarlingWorker.AwsRoleConfigurationFault(ex)); + Assert.Null(DarlingWorker.AwsRoleConfigurationFault(new InvalidOperationException("unrelated"))); + } + + /* ---- the public factory parameters --------------------------------------------------------------------- */ + + [Fact] + public async Task APublicFactory_ServesAServerWithNoRole_AndRefusesAServerWithOne_WithoutCallingIt() + { + var called = 0; + var source = new RdsLogSource(_ => + { + called++; + return new RecordingRds(new BasicAWSCredentials(ProcessAccessKey, "example-process-secret")); + }, verifier: Enforcing()); + + await Assert.ThrowsAsync(() => ReadAsync(source, KeyA)); + Assert.Equal(0, called); + + await ReadAsync(source, role: null); + Assert.Equal(1, called); + } + + [Fact] + public async Task ADefaultFactoryWithNoCache_RefusesARole_Too() + { + var ingestor = new RdsCpuIngestor(UnusedDataSource(), verifier: Enforcing()); + + var ex = await Record.ExceptionAsync(() => ingestor.IngestAsync(1, "s", InstanceHost, ConnectionString, KeyA, CancellationToken.None)); + + var wrapped = Assert.IsType(ex); + Assert.IsType(wrapped.InnerException); + } + + /* ---- the wiring, pinned in the source ------------------------------------------------------------------- */ + + [Fact] + public void TheWorkersRoleArm_ComesBeforeEveryArmThatReadsTheTextOfAnAwsFailure() + { + var worker = RepoFile.ReadRepoFileLf("Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs"); + + var roleArm = worker.IndexOf("catch (Exception ex) when (AwsRoleConfigurationFault(ex) is { } assume)", StringComparison.Ordinal); + var mismatch = worker.IndexOf("catch (RdsEndpointMismatchException ex)", StringComparison.Ordinal); + var logArm = worker.IndexOf("catch (RdsLogUnavailableException ex) when (ex.IsAuthorizationFailure)", StringComparison.Ordinal); + var piArm = worker.IndexOf("catch (PiMetricsUnavailableException ex) when (ex.IsAuthorizationFailure)", StringComparison.Ordinal); + var general = worker.IndexOf(" catch (Exception ex)\n {\n /* #3095", StringComparison.Ordinal); + + Assert.True(roleArm > 0, "the role arm is missing"); + Assert.True(roleArm < mismatch, "the role arm must come before the endpoint arm"); + Assert.True(roleArm < logArm, "the role arm must come before the RDS log arm"); + Assert.True(roleArm < piArm, "the role arm must come before the Performance Insights arm"); + Assert.True(general < 0 || roleArm < general, "the role arm must come before the general arm"); + + var armBody = worker.Substring(roleArm, mismatch - roleArm); + Assert.Contains("\"PERMISSIONS\"", armBody, StringComparison.Ordinal); + Assert.Contains("AwsRoleFaultNote(", armBody, StringComparison.Ordinal); + } + + [Fact] + public void TheRunner_HandsTheRoleCacheToEveryReader_AndEveryCallPassesTheServersRole() + { + var runner = RepoFile.ReadRepoFileLf("Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs"); + + Assert.Equal(4, CountOf(runner, "roles: _awsRoles));")); + Assert.Equal(4, CountOf(runner, "server.Config.AwsRoleKey, cancellationToken);")); + Assert.Contains("RetainAwsRoles", RepoFile.ReadRepoFileLf("Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs"), StringComparison.Ordinal); + } + + private static int CountOf(string text, string needle) + { + var count = 0; + for (var at = text.IndexOf(needle, StringComparison.Ordinal); at >= 0; at = text.IndexOf(needle, at + needle.Length, StringComparison.Ordinal)) + { + count++; + } + + return count; + } +} diff --git a/Darling/Darling.Tests/RdsDeadlockReportResumeTests.cs b/Darling/Darling.Tests/RdsDeadlockReportResumeTests.cs index 52303be2e0..d3042956bf 100644 --- a/Darling/Darling.Tests/RdsDeadlockReportResumeTests.cs +++ b/Darling/Darling.Tests/RdsDeadlockReportResumeTests.cs @@ -76,7 +76,7 @@ private static (Func> Ingest, List ingestor.IngestAsync(1, "s", Host, false, false, CancellationToken.None), stored); + return (() => ingestor.IngestAsync(1, "s", Host, false, false, cancellationToken: CancellationToken.None), stored); } private static string SavedPosition(RdsFakeCollectorState state) => state.Value("pg_deadlocks", "log_resume") ?? "(none)"; diff --git a/Darling/Darling.Tests/RdsEndpointVerifierTests.Login.cs b/Darling/Darling.Tests/RdsEndpointVerifierTests.Login.cs new file mode 100644 index 0000000000..58ed25c774 --- /dev/null +++ b/Darling/Darling.Tests/RdsEndpointVerifierTests.Login.cs @@ -0,0 +1,462 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor Lite. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text.RegularExpressions; +using System.Threading; +using System.Threading.Tasks; +using Amazon.RDS.Model; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service.Targets; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: these cases never connect to any store; the login probe is an in-memory fake. */ +public partial class RdsEndpointVerifierTests +{ + private static IEnumerable ServiceFiles() + { + var service = Path.Combine(RepoRoot(), "Darling", "PerformanceMonitor.Darling.Service"); + + return Directory.EnumerateFiles(service, "*.cs", SearchOption.AllDirectories) + .Where(path => !path.Contains(Path.DirectorySeparatorChar + "obj" + Path.DirectorySeparatorChar, StringComparison.Ordinal) + && !path.Contains(Path.DirectorySeparatorChar + "bin" + Path.DirectorySeparatorChar, StringComparison.Ordinal)); + } + + private static string ServiceFile(string name) + => File.ReadAllText(Path.Combine(RepoRoot(), "Darling", "PerformanceMonitor.Darling.Service", name)).Replace("\r\n", "\n"); + + /// + /// The skip switch the older fixtures rely on stays out of product code: it may be named in the verifier's own file + /// and nowhere else in the service. + /// + [Fact] + public void TheTestSkipSwitch_IsNamedInNoOtherServiceFile() + { + var offenders = ServiceFiles() + .Where(path => Path.GetFileName(path) != "RdsEndpointVerifier.cs" + && File.ReadAllText(path).Contains("TestOnlySkipCheck", StringComparison.Ordinal)) + .ToList(); + + Assert.Empty(offenders); + } + + private const string TargetTypedConstruction = + @"(?:RdsEndpointVerifier\??\s+\w+\s*=|[Vv]erifier\w*\s*(?:\?\?=?|=))\s*new\s*\(\s*[^)\s]"; + + private const string TestFactoryCall = @"RdsEndpointVerifier\s*\.\s*ForTests\b"; + + /// + /// No service file builds a verifier that skips the check: outside the verifier's file the only construction forms are + /// the parameterless one (the runner's shared verifier) and the log source's clock-only one. None names + /// enforce or the test factory, and none passes arguments through a target-typed new(. The constructor + /// that takes enforce and the login probe is private. + /// + [Fact] + public void NoServiceFile_BuildsAVerifierThatSkipsTheCheck() + { + var construction = new Regex(@"new\s+RdsEndpointVerifier\s*\(([^)]*)\)"); + var offenders = new List(); + + foreach (var path in ServiceFiles().Where(path => Path.GetFileName(path) != "RdsEndpointVerifier.cs")) + { + var code = File.ReadAllText(path); + + if (code.Contains("enforce:", StringComparison.Ordinal) || code.Contains("enforce =", StringComparison.Ordinal)) + { + offenders.Add(path + " names enforce"); + } + + if (Regex.IsMatch(code, TestFactoryCall)) + { + offenders.Add(path + " calls the test factory"); + } + + foreach (Match match in construction.Matches(code)) + { + var arguments = match.Groups[1].Value.Trim(); + + if (arguments.Length > 0 && arguments != "clock") + { + offenders.Add(path + ": " + match.Value); + } + } + + foreach (Match match in Regex.Matches(code, TargetTypedConstruction)) + { + offenders.Add(path + ": " + match.Value); + } + } + + Assert.Empty(offenders); + + var constructors = typeof(RdsEndpointVerifier).GetConstructors( + System.Reflection.BindingFlags.Public | System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance); + + Assert.All( + constructors.Where(c => c.GetParameters().Any(p => p.Name == "enforce" || p.Name == "loginProbe")), + c => Assert.True(c.IsPrivate, "the constructor that takes enforce or the login probe is private")); + } + + /// The pin's patterns see the forms a planted construction would take. + [Theory] + [InlineData("RdsEndpointVerifier v = new(null, false);")] + [InlineData("RdsEndpointVerifier? v = new(null, null, (_, _) => Task.FromResult(null));")] + [InlineData("_verifier = verifier ?? new(clock, false);")] + [InlineData("var x = RdsEndpointVerifier.ForTests(enforce: false);")] + public void ThePin_SeesAPlantedConstruction(string planted) + { + Assert.True(Regex.IsMatch(planted, TargetTypedConstruction) || Regex.IsMatch(planted, TestFactoryCall)); + } + + /// The runner builds one verifier, hands it to the four RDS readers, and gives each the target's connection string and role (#5452). + [Fact] + public void TheRunner_SharesOneVerifierWithTheFourRdsReaders() + { + var runner = ServiceFile("DarlingCollectorRunner.cs"); + + Assert.Single(Regex.Matches(runner, @"RdsEndpointVerifier _rdsVerifier = new\(\);")); + Assert.Equal(4, Regex.Matches(runner, @"verifier: _rdsVerifier").Count); + Assert.Equal(4, Regex.Matches(runner, @"server\.ConnectionString, server\.Config\.AwsRoleKey, cancellationToken\);").Count); + } + + /// The reload clears a server's verdicts when its definition changed, and the dispatch arm for the refusal records PERMISSIONS. + [Fact] + public void TheWorker_ClearsVerdictsOnAChangedDefinition_AndHasTheRefusalArm() + { + var worker = ServiceFile("DarlingWorker.cs"); + + var changed = worker.IndexOf("if (connectionChanged)", StringComparison.Ordinal); + var forget = worker.IndexOf("_runner?.ForgetRdsVerdicts(id);", changed, StringComparison.Ordinal); + + Assert.True(changed > 0 && forget > changed, "the changed-definition branch clears the verdicts"); + Assert.True(forget - changed < 6000, "and it does so inside that branch"); + + var arm = worker.IndexOf("catch (RdsEndpointMismatchException ex)", StringComparison.Ordinal); + var general = worker.IndexOf("catch (RdsLogUnavailableException", arm, StringComparison.Ordinal); + + Assert.True(arm > 0 && general > arm); + Assert.Contains("\"PERMISSIONS\"", worker[arm..general], StringComparison.Ordinal); + Assert.Contains("ex.Message", worker[arm..general], StringComparison.Ordinal); + + /* The removal branch forgets the same state, so a re-added server is checked afresh. */ + var removed = worker.IndexOf("Removed from the monitored set (disabled/deleted)", StringComparison.Ordinal); + var removedForget = worker.IndexOf("_runner?.ForgetRdsVerdicts(id);", removed, StringComparison.Ordinal); + var removedEnd = worker.IndexOf("servers.RemoveAt(i);", removed, StringComparison.Ordinal); + + Assert.True(removed > 0 && removedForget > removed && removedForget < removedEnd, "the removal branch clears the verdicts"); + + /* The denied-Describe text on the log route names both Describe actions, as the AWS refusal does. */ + Assert.Contains("the role needs rds:DescribeDBInstances, rds:DescribeDBClusters, \"", worker, StringComparison.Ordinal); + } + + /// A refused login (28000, 28P01) is a connection-level fault, so the runtime is dropped and reconnects. + [Theory] + [InlineData("28000")] + [InlineData("28P01")] + [InlineData("57P01")] + public void ARefusedLogin_IsAConnectionFatalFault(string sqlState) + { + var fault = new PostgresException("boom", "FATAL", "FATAL", sqlState); + + Assert.Equal(CollectorTargetFault.ConnectionFatal, PostgresTargetProvider.Instance.Classify(fault, yieldsOnLockTimeout: false)); + } + + [Fact] + public void AnOtherwiseUnclassifiedFault_StaysUnclassified() + { + var fault = new PostgresException("boom", "ERROR", "ERROR", "42000"); + + Assert.Equal(CollectorTargetFault.Unclassified, PostgresTargetProvider.Instance.Classify(fault, yieldsOnLockTimeout: false)); + } + + [Fact] + public async Task ACachedMatch_NeedsAFreshLogin_NoOlderThanTheMatchLifetime() + { + var now = new DateTime(2026, 10, 7, 12, 0, 0, DateTimeKind.Utc); + var probes = 0; + string? answer = null; + var verifier = Enforcing(() => now, (_, _) => { probes++; return Task.FromResult(answer); }); + var rds = new FakeRds(); + + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None, ConnectionString); + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None, ConnectionString); + + Assert.Equal(1, rds.InstanceDescribes); + Assert.Equal(1, probes); + + /* Inside the hour the login holds. Past it the next read logs in again, and a refused login ends the reads. */ + now = now.AddMinutes(59); + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None, ConnectionString); + Assert.Equal(1, probes); + + now = now.AddMinutes(2); + answer = "SQLSTATE 28P01"; + + var ex = await Assert.ThrowsAsync( + () => verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None, ConnectionString)); + + Assert.Equal(2, probes); + Assert.Contains(InstanceHost, ex.Message, StringComparison.Ordinal); + Assert.Contains("SQLSTATE 28P01", ex.Message, StringComparison.Ordinal); + Assert.IsAssignableFrom(ex); + + /* The refusal is kept for the mismatch lifetime and the read still ends, with no new login in that window. */ + answer = null; + now = now.AddMinutes(4); + + var again = await Assert.ThrowsAsync( + () => verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None, ConnectionString)); + + Assert.Equal(2, probes); + Assert.Contains("SQLSTATE 28P01", again.Message, StringComparison.Ordinal); + + /* After it the next read logs in again, and goes on when the login is accepted. */ + now = now.AddMinutes(2); + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None, ConnectionString); + Assert.Equal(3, probes); + Assert.Equal(0, verifier.CachedLoginRefusalCount); + } + + [Fact] + public async Task ARefusedLogin_IsKeptPerServer_AndAClearedServerLogsInAgain() + { + var now = new DateTime(2026, 10, 7, 12, 0, 0, DateTimeKind.Utc); + var probes = 0; + var verifier = Enforcing(() => now, (_, _) => { probes++; return Task.FromResult("SQLSTATE 28P01"); }); + var rds = new FakeRds(); + + for (var read = 0; read < 3; read++) + { + await Assert.ThrowsAsync( + () => verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None, ConnectionString)); + } + + Assert.Equal(1, probes); + Assert.Equal(1, rds.InstanceDescribes); + + await Assert.ThrowsAsync( + () => verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 2, CancellationToken.None, ConnectionString)); + Assert.Equal(2, probes); + + verifier.ClearServer(1); + Assert.Equal(1, verifier.CachedLoginRefusalCount); + + await Assert.ThrowsAsync( + () => verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None, ConnectionString)); + Assert.Equal(3, probes); + } + + [Fact] + public async Task TheRealLoginProbe_NamesNoAddress() + { + var result = await RdsEndpointVerifier.ProbeLoginAsync("Host=127.0.0.1;Port=1;Timeout=1", CancellationToken.None); + + Assert.NotNull(result); + Assert.DoesNotContain("127.0.0.1", result, StringComparison.Ordinal); + Assert.Matches(@"^(SQLSTATE [0-9A-Z]{5}|[A-Za-z][A-Za-z0-9]*)$", result); + } + + [Fact] + public async Task NoConnectionString_IsNoFreshLogin_AndNothingIsRead() + { + var rds = new FakeRds(); + var verifier = RdsEndpointVerifier.ForTests(enforce: true); + + var ex = await Assert.ThrowsAsync( + () => verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None)); + + Assert.Contains("no connection string", ex.Message, StringComparison.Ordinal); + Assert.NotNull(await RdsEndpointVerifier.ProbeLoginAsync(null, CancellationToken.None)); + } + + [Fact] + public async Task ARefusedLogin_ThroughTheLogSource_ReadsNothing() + { + var rds = new FakeRds(); + var source = new RdsLogSource( + _ => rds, verifier: Enforcing(loginProbe: (_, _) => Task.FromResult("SQLSTATE 28000"))); + + await Assert.ThrowsAsync( + () => source.ReadNewestAsync(InstanceHost, RdsLogSource.LogFileKind.Stderr, 7, CancellationToken.None, ConnectionString)); + + Assert.Equal(0, rds.LogListings); + Assert.Equal(0, rds.Downloads); + } + + [Fact] + public async Task TheCredentialScope_IsPartOfTheVerdictKey_AndAClearedServerAsksAgain() + { + var rds = new FakeRds(); + var verifier = Enforcing(); + + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None, ConnectionString); + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None, ConnectionString); + Assert.Equal(1, rds.InstanceDescribes); + + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None, ConnectionString, "scope-b"); + Assert.Equal(2, rds.InstanceDescribes); + + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 2, CancellationToken.None, ConnectionString); + Assert.Equal(3, rds.InstanceDescribes); + + verifier.ClearServer(1); + + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None, ConnectionString); + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None, ConnectionString, "scope-b"); + Assert.Equal(5, rds.InstanceDescribes); + + /* Another server's verdict was kept. */ + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 2, CancellationToken.None, ConnectionString); + Assert.Equal(5, rds.InstanceDescribes); + } + + private static async Task MismatchTextAsync(FakeRds rds, string host, RdsEndpointVerifier? verifier = null) + { + var ex = await Assert.ThrowsAsync( + () => (verifier ?? Enforcing()).EnsureAsync(rds, Parse(host), host, 1, CancellationToken.None, ConnectionString)); + + return ex.Message; + } + + [Fact] + public async Task MissingId_ReadsAsMismatch() + { + var mismatched = await MismatchTextAsync(new FakeRds { InstanceAddress = ReportedElsewhere }, InstanceHost); + + Assert.Equal(mismatched, await MismatchTextAsync(new FakeRds { NoResults = true }, InstanceHost)); + Assert.Equal(mismatched, await MismatchTextAsync( + new FakeRds { DescribeFailure = new DBInstanceNotFoundException("DBInstance solo not found.") }, InstanceHost)); + + var clusterMismatch = await MismatchTextAsync( + new FakeRds { ClusterEndpoint = ReportedElsewhere, ReaderEndpoint = null, CustomEndpoints = null }, WriterHost); + + Assert.Equal(clusterMismatch, await MismatchTextAsync(new FakeRds { NoResults = true }, WriterHost)); + Assert.Equal(clusterMismatch, await MismatchTextAsync( + new FakeRds { DescribeFailure = new DBClusterNotFoundException("DBCluster shared not found.") }, WriterHost)); + } + + [Fact] + public async Task MissingId_IsCachedAsAMismatch() + { + var rds = new FakeRds { DescribeFailure = new DBInstanceNotFoundException("not found") }; + var verifier = Enforcing(); + + await MismatchTextAsync(rds, InstanceHost, verifier); + await MismatchTextAsync(rds, InstanceHost, verifier); + + Assert.Equal(1, rds.InstanceDescribes); + } + + [Theory] + [InlineData("hosK.abc123.us-east-1.rds.amazonaws.com", "hosk.abc123.us-east-1.rds.amazonaws.com")] + [InlineData("höst.abc123.us-east-1.rds.amazonaws.com", "höst.abc123.us-east-1.rds.amazonaws.com")] + [InlineData("solo.abc123.us-east-1.rds.amazonaws.comİ", "solo.abc123.us-east-1.rds.amazonaws.comİ")] + public void ANonAsciiHost_MatchesNothing(string host, string reported) + { + Assert.False(RdsEndpointVerifier.Matches(host, [reported])); + Assert.False(RdsEndpointVerifier.Matches(reported, [host])); + Assert.Equal(string.Empty, RdsEndpointVerifier.Normalize(host)); + } + + [Theory] + [InlineData("solo.abc123.us-east-1.rds.amazonaws.com\u00A0")] + [InlineData("solo.abc123.us-east-1.rds.amazonaws.com\u3000")] + [InlineData("\u00A0solo.abc123.us-east-1.rds.amazonaws.com")] + [InlineData("solo.abc123.us-east-1.rds.amazonaws.com\t")] + [InlineData("solo.abc123.us-east-1.rds.amazonaws.com\n")] + public async Task AHostWithAUnicodeSpaceOrControlCharacter_ReadsAsAMismatch(string host) + { + Assert.False(RdsEndpointVerifier.Matches(host, [InstanceHost])); + Assert.Equal(string.Empty, RdsEndpointVerifier.Normalize(host)); + + var rds = new FakeRds(); + + await Assert.ThrowsAsync( + () => Enforcing().EnsureAsync(rds, Parse(InstanceHost), host, 1, CancellationToken.None, ConnectionString)); + } + + [Fact] + public void OrdinarySpacesAtTheEnds_AreStillTrimmed() + { + Assert.True(RdsEndpointVerifier.Matches(" " + InstanceHost + " ", [InstanceHost])); + } + + [Fact] + public void AnAsciiHost_StillComparesIgnoringCase() + { + Assert.True(RdsEndpointVerifier.Matches("SOLO.ABC123.us-east-1.RDS.amazonaws.com", [InstanceHost])); + } + + [Theory] + [InlineData("solo.abc123.us-east-1.rds.amazonaws.com,other.abc123.us-east-1.rds.amazonaws.com")] + [InlineData("solo.abc123.us-east-1.rds.amazonaws.com:5432")] + [InlineData("solo.abc123.us-east-1.rds.amazonaws.comK")] + [InlineData("sölo.abc123.us-east-1.rds.amazonaws.com")] + [InlineData("solo.abc123.us-east-1.rds.amazonaws.com extra")] + public async Task AHostThatIsNotOneEndpointName_ReadsNothing(string host) + { + var rds = new FakeRds(); + var source = new RdsLogSource(_ => rds, verifier: Enforcing()); + + try + { + var chunk = await source.ReadNewestAsync(host, RdsLogSource.LogFileKind.Stderr, 7, CancellationToken.None, ConnectionString); + + Assert.Null(chunk); + } + catch (RdsEndpointMismatchException) + { + /* The other acceptable end: refused as a mismatch. Either way nothing below was read. */ + } + + Assert.Equal(0, rds.LogListings); + Assert.Equal(0, rds.Downloads); + } + + [Fact] + public async Task AnExpiredEntry_IsDroppedOnRead_AndEachCacheIsCapped() + { + var now = new DateTime(2026, 10, 7, 12, 0, 0, DateTimeKind.Utc); + var verifier = Enforcing(() => now); + + await MismatchTextAsync(new FakeRds { InstanceAddress = ReportedElsewhere }, InstanceHost, verifier); + Assert.Equal(1, verifier.CachedVerdictCount); + + now = now.AddMinutes(6); + var rds = new FakeRds(); + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None, ConnectionString); + + /* The expired mismatch was replaced by the new match under the same key, not kept beside it. */ + Assert.Equal(1, verifier.CachedVerdictCount); + + var last = 100 + RdsEndpointVerifier.MaxEntries + 49; + + for (var serverId = 100; serverId <= last; serverId++) + { + now = now.AddSeconds(1); + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, serverId, CancellationToken.None, ConnectionString); + } + + Assert.Equal(RdsEndpointVerifier.MaxEntries, verifier.CachedVerdictCount); + Assert.Equal(RdsEndpointVerifier.MaxEntries, verifier.CachedLoginCount); + + /* The oldest entry went first: the newest is kept, and server 1 (the oldest) asks AWS again. */ + var before = rds.InstanceDescribes; + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, last, CancellationToken.None, ConnectionString); + Assert.Equal(before, rds.InstanceDescribes); + + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None, ConnectionString); + Assert.Equal(before + 1, rds.InstanceDescribes); + } +} diff --git a/Darling/Darling.Tests/RdsEndpointVerifierTests.cs b/Darling/Darling.Tests/RdsEndpointVerifierTests.cs new file mode 100644 index 0000000000..8e9b749deb --- /dev/null +++ b/Darling/Darling.Tests/RdsEndpointVerifierTests.cs @@ -0,0 +1,387 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor Lite. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Runtime.CompilerServices; +using System.Threading; +using System.Threading.Tasks; +using Amazon.RDS; +using Amazon.RDS.Model; +using Npgsql; +using PerformanceMonitor.Darling.Service.Targets; +using Xunit; + +namespace Darling.Tests; + +/// +/// An RDS target's host has to be the endpoint AWS reports for the instance or cluster id parsed from it before any RDS +/// or Performance Insights read runs. Every case here builds the verifier with enforce: true, because the test +/// assembly's default verifier skips the check for the older fixtures that predate it. +/// +public partial class RdsEndpointVerifierTests +{ + private const string InstanceHost = "solo.abc123.us-east-1.rds.amazonaws.com"; + private const string WriterHost = "shared.cluster-abc123.us-east-1.rds.amazonaws.com"; + private const string ReaderHost = "shared.cluster-ro-abc123.us-east-1.rds.amazonaws.com"; + private const string CustomHost = "shared.cluster-custom-abc.us-east-1.rds.amazonaws.com"; + private const string HostNotReportedByAws = "solo.abc123.us-east-1.rds.example.net"; + private const string ReportedElsewhere = "elsewhere.zzz999.us-east-1.rds.amazonaws.com"; + + private const string ConnectionString = "Host=solo.abc123.us-east-1.rds.amazonaws.com;Username=u;Database=d"; + + private static RdsEndpointVerifier Enforcing( + Func? clock = null, Func>? loginProbe = null) + => RdsEndpointVerifier.ForTests(clock, enforce: true, loginProbe ?? ((_, _) => Task.FromResult(null))); + + private static RdsEndpoint.Parsed Parse(string host) => RdsEndpoint.TryParse(host)!.Value; + + private static NpgsqlDataSource UnusedDataSource() + => NpgsqlDataSource.Create("Host=localhost;Port=1;Database=unused"); + + private sealed class FakeRds : AmazonRDSClient + { + public FakeRds() : base(new Amazon.Runtime.BasicAWSCredentials("a", "b"), Amazon.RegionEndpoint.USEast1) { } + + public string? InstanceAddress { get; init; } = InstanceHost; + public string? ClusterEndpoint { get; init; } = WriterHost; + public string? ReaderEndpoint { get; init; } = ReaderHost; + public List? CustomEndpoints { get; init; } = [CustomHost]; + public Exception? DescribeFailure { get; init; } + + /// Describe answers with an empty list. + public bool NoResults { get; init; } + + public int InstanceDescribes; + public int ClusterDescribes; + public int LogListings; + public int Downloads; + + public override Task DescribeDBInstancesAsync( + DescribeDBInstancesRequest request, CancellationToken cancellationToken = default) + { + InstanceDescribes++; + + if (DescribeFailure is not null) + { + throw DescribeFailure; + } + + if (NoResults) + { + return Task.FromResult(new DescribeDBInstancesResponse { DBInstances = [] }); + } + + return Task.FromResult(new DescribeDBInstancesResponse + { + DBInstances = [new DBInstance { Endpoint = new Endpoint { Address = InstanceAddress }, DbiResourceId = "db-EXAMPLE" }], + }); + } + + public override Task DescribeDBClustersAsync( + DescribeDBClustersRequest request, CancellationToken cancellationToken = default) + { + ClusterDescribes++; + + if (DescribeFailure is not null) + { + throw DescribeFailure; + } + + if (NoResults) + { + return Task.FromResult(new DescribeDBClustersResponse { DBClusters = [] }); + } + + return Task.FromResult(new DescribeDBClustersResponse + { + DBClusters = + [ + new DBCluster + { + Endpoint = ClusterEndpoint, + ReaderEndpoint = ReaderEndpoint, + CustomEndpoints = CustomEndpoints, + DBClusterMembers = [new DBClusterMember { DBInstanceIdentifier = "writer-1", IsClusterWriter = true }], + }, + ], + }); + } + + public override Task DescribeDBLogFilesAsync( + DescribeDBLogFilesRequest request, CancellationToken cancellationToken = default) + { + LogListings++; + + return Task.FromResult(new DescribeDBLogFilesResponse + { + DescribeDBLogFiles = [new DescribeDBLogFilesDetails { LogFileName = "error/postgresql.log.2026-08-25-18", LastWritten = 1000 }], + }); + } + + public override Task DownloadDBLogFilePortionAsync( + DownloadDBLogFilePortionRequest request, CancellationToken cancellationToken = default) + { + Downloads++; + + return Task.FromResult(new DownloadDBLogFilePortionResponse { LogFileData = string.Empty, Marker = "M" }); + } + } + + [Theory] + [InlineData("SOLO.ABC123.US-EAST-1.RDS.AMAZONAWS.COM")] + [InlineData("solo.abc123.us-east-1.rds.amazonaws.com.")] + [InlineData(" Solo.Abc123.us-east-1.rds.amazonaws.com.. ")] + public void Matches_IgnoresCaseAndTrailingDot(string host) + { + Assert.True(RdsEndpointVerifier.Matches(host, [InstanceHost])); + Assert.True(RdsEndpointVerifier.Matches(InstanceHost, [host])); + } + + [Fact] + public void Matches_RefusesAnEmptyOrAbsentReportedAddress() + { + Assert.False(RdsEndpointVerifier.Matches(InstanceHost, [null, string.Empty, " ", "."])); + Assert.False(RdsEndpointVerifier.Matches(string.Empty, [string.Empty])); + Assert.False(RdsEndpointVerifier.Matches(InstanceHost, [])); + } + + [Fact] + public async Task Instance_HostEqualToTheReportedAddress_Proceeds() + { + var rds = new FakeRds(); + + await Enforcing().EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None); + + Assert.Equal(1, rds.InstanceDescribes); + } + + [Fact] + public async Task Instance_HostThatIsNotTheReportedAddress_Throws() + { + var rds = new FakeRds { InstanceAddress = ReportedElsewhere }; + + var ex = await Assert.ThrowsAsync( + () => Enforcing().EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None)); + + Assert.Contains(InstanceHost, ex.Message, StringComparison.Ordinal); + Assert.Contains("'solo'", ex.Message, StringComparison.Ordinal); + Assert.DoesNotContain(ReportedElsewhere, ex.Message, StringComparison.Ordinal); + } + + [Theory] + [InlineData(WriterHost)] + [InlineData(ReaderHost)] + [InlineData(CustomHost)] + public async Task Cluster_HostEqualToAnyReportedEndpoint_Proceeds(string host) + { + var rds = new FakeRds(); + + await Enforcing().EnsureAsync(rds, Parse(host), host, 1, CancellationToken.None); + + Assert.Equal(1, rds.ClusterDescribes); + } + + [Theory] + [InlineData(WriterHost)] + [InlineData(ReaderHost)] + [InlineData(CustomHost)] + public async Task Cluster_HostThatIsNotAReportedEndpoint_Throws(string host) + { + var rds = new FakeRds + { + ClusterEndpoint = "other.cluster-zzz.us-east-1.rds.amazonaws.com", + ReaderEndpoint = "other.cluster-ro-zzz.us-east-1.rds.amazonaws.com", + CustomEndpoints = null, + }; + + var ex = await Assert.ThrowsAsync( + () => Enforcing().EnsureAsync(rds, Parse(host), host, 1, CancellationToken.None)); + + Assert.Contains("cluster 'shared'", ex.Message, StringComparison.Ordinal); + Assert.DoesNotContain("zzz", ex.Message, StringComparison.Ordinal); + } + + [Fact] + public async Task AMatchIsCachedPerServerHostAndId_AndAnEditedHostAsksAgain() + { + var rds = new FakeRds(); + var verifier = Enforcing(); + + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None); + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost + ".", 1, CancellationToken.None); + + Assert.Equal(1, rds.InstanceDescribes); + + /* Another server id, with the same host, is its own key. */ + await verifier.EnsureAsync(rds, Parse(InstanceHost), InstanceHost, 2, CancellationToken.None); + + Assert.Equal(2, rds.InstanceDescribes); + + /* The host edited to one AWS does not report: a new key, so AWS is asked again, and now it does not match. */ + await Assert.ThrowsAsync( + () => verifier.EnsureAsync(rds, Parse(HostNotReportedByAws), HostNotReportedByAws, 1, CancellationToken.None)); + + Assert.Equal(3, rds.InstanceDescribes); + } + + [Fact] + public async Task AMismatchIsKeptBriefly_ThenAsksAgainAndCanRecover() + { + var now = new DateTime(2026, 10, 7, 12, 0, 0, DateTimeKind.Utc); + var verifier = Enforcing(() => now); + var wrong = new FakeRds { InstanceAddress = ReportedElsewhere }; + + await Assert.ThrowsAsync( + () => verifier.EnsureAsync(wrong, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None)); + await Assert.ThrowsAsync( + () => verifier.EnsureAsync(wrong, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None)); + + Assert.Equal(1, wrong.InstanceDescribes); + + now = now.AddMinutes(6); + var right = new FakeRds(); + + await verifier.EnsureAsync(right, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None); + + Assert.Equal(1, right.InstanceDescribes); + } + + [Fact] + public async Task ADeniedDescribe_IsNotCached_AndNamesTheAction() + { + var denied = new FakeRds + { + DescribeFailure = new AmazonRDSException( + "User: arn:aws:sts::1:assumed-role/x is not authorized to perform: rds:DescribeDBInstances"), + }; + var verifier = Enforcing(); + + var ex = await Assert.ThrowsAsync( + () => verifier.EnsureAsync(denied, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None)); + + Assert.Contains("rds:DescribeDBInstances", ex.Message, StringComparison.Ordinal); + Assert.True(RdsLogUnavailableException.IsAuthorizationRefusal(ex)); + + await Assert.ThrowsAsync( + () => verifier.EnsureAsync(denied, Parse(InstanceHost), InstanceHost, 1, CancellationToken.None)); + + Assert.Equal(2, denied.InstanceDescribes); + } + + [Fact] + public async Task LogSource_HostEqualToTheReportedAddress_ReadsTheLog() + { + var rds = new FakeRds(); + var source = new RdsLogSource(_ => rds, verifier: Enforcing()); + + var chunk = await source.ReadNewestAsync(InstanceHost, RdsLogSource.LogFileKind.Stderr, 7, CancellationToken.None); + + Assert.NotNull(chunk); + Assert.Equal(1, rds.Downloads); + } + + [Fact] + public async Task HostThatIsNotTheReportedEndpoint_ReadsNothing() + { + var rds = new FakeRds(); + var source = new RdsLogSource(_ => rds, verifier: Enforcing()); + + var ex = await Assert.ThrowsAsync( + () => source.ReadNewestAsync(HostNotReportedByAws, RdsLogSource.LogFileKind.Stderr, 7, CancellationToken.None)); + + Assert.Contains(HostNotReportedByAws, ex.Message, StringComparison.Ordinal); + Assert.Equal(0, rds.LogListings); + Assert.Equal(0, rds.Downloads); + } + + [Fact] + public async Task HostThatIsNotTheReportedEndpoint_ReachesTheRunnerUnwrapped_ThroughEachLogIngestor() + { + var rds = new FakeRds(); + var source = new RdsLogSource(_ => rds, verifier: Enforcing()); + + await Assert.ThrowsAsync( + () => new RdsPlanIngestor(UnusedDataSource(), source).IngestAsync(7, "srv", HostNotReportedByAws)); + await Assert.ThrowsAsync( + () => new RdsDeadlockIngestor(UnusedDataSource(), source).IngestAsync(7, "srv", HostNotReportedByAws)); + await Assert.ThrowsAsync( + () => new RdsLogEventIngestor(UnusedDataSource(), TestLogHashKeys.Fixed, source).IngestAsync(7, "srv", HostNotReportedByAws)); + + Assert.Equal(0, rds.LogListings); + Assert.Equal(0, rds.Downloads); + } + + [Fact] + public async Task ADeniedDescribe_ThroughALogIngestor_IsTheAuthorizationOutcome() + { + var rds = new FakeRds + { + DescribeFailure = new AmazonRDSException( + "User: arn:aws:sts::1:assumed-role/x is not authorized to perform: rds:DescribeDBInstances"), + }; + var source = new RdsLogSource(_ => rds, verifier: Enforcing()); + + var ex = await Assert.ThrowsAsync( + () => new RdsPlanIngestor(UnusedDataSource(), source).IngestAsync(7, "srv", InstanceHost)); + + Assert.True(ex.IsAuthorizationFailure); + Assert.Contains("rds:DescribeDBInstances", ex.Message, StringComparison.Ordinal); + Assert.Equal(0, rds.LogListings); + } + + [Fact] + public async Task Cpu_HostThatIsNotTheReportedEndpoint_MakesNoPerformanceInsightsCall() + { + var rds = new FakeRds(); + var piCalled = false; + var ingestor = new RdsCpuIngestor( + UnusedDataSource(), + rdsClientFactory: _ => rds, + piClientFactory: _ => { piCalled = true; return null!; }, + verifier: Enforcing()); + + var ex = await Assert.ThrowsAsync( + () => ingestor.IngestAsync(7, "srv", HostNotReportedByAws)); + + Assert.Contains(HostNotReportedByAws, ex.Message, StringComparison.Ordinal); + Assert.False(piCalled); + } + + [Fact] + public async Task Cpu_ADeniedDescribe_IsTheAuthorizationOutcome() + { + var rds = new FakeRds + { + DescribeFailure = new AmazonRDSException( + "User: arn:aws:sts::1:assumed-role/x is not authorized to perform: rds:DescribeDBInstances"), + }; + var ingestor = new RdsCpuIngestor( + UnusedDataSource(), + rdsClientFactory: _ => rds, + piClientFactory: _ => null!, + verifier: Enforcing()); + + var ex = await Assert.ThrowsAsync(() => ingestor.IngestAsync(7, "srv", InstanceHost)); + + Assert.True(ex.IsAuthorizationFailure); + } + + internal static string RepoRoot([CallerFilePath] string thisFile = "") + => Path.GetFullPath(Path.Combine(Path.GetDirectoryName(thisFile)!, "..", "..")); +} + +/* #1776 own-store: this class never connects to any store. It sets one in-memory switch when the assembly loads, so the + older RDS fixtures (whose fakes answer neither Describe call) build a verifier that skips the check. */ +internal static class RdsEndpointVerifierTestSwitch +{ + [ModuleInitializer] + internal static void SkipTheCheckForOlderFixtures() => RdsEndpointVerifier.TestOnlySkipCheck = true; +} diff --git a/Darling/Darling.Tests/RdsLogStateConcurrencyTests.cs b/Darling/Darling.Tests/RdsLogStateConcurrencyTests.cs new file mode 100644 index 0000000000..bfd97b0407 --- /dev/null +++ b/Darling/Darling.Tests/RdsLogStateConcurrencyTests.cs @@ -0,0 +1,477 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor Lite. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Diagnostics; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Amazon.RDS; +using Amazon.RDS.Model; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Targets; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5003: a store with several RDS targets runs the same collector for each of them at once, and every one of those +/// runs goes through ONE ingestor, which owns ONE . A pg_log_events run and a +/// pg_deadlocks run each failed once with .NET's "Collection was modified; enumeration operation may not +/// execute", and both landed on the hour, when RDS rotates its log and every target adds a position for the new file. +/// +/// These tests drive that sharing the way the ingestors do: one source, a different instance id for each +/// thread, all of them started together. They assert nothing about timing. A run that tears the shared tables throws +/// out of the thread that was caught enumerating, or leaves a position or a debounce stamp or a carry missing, and +/// both are failures here. They are bounded stress tests rather than a fixed interleaving, because the source has no +/// hook to pause an enumeration part way and one is not worth adding to production code for this: the table is seeded +/// with a few hundred other targets' positions so a scan is long, and every round adds a key, which is what breaks a +/// scan that is already running on another thread. +/// +public class RdsLogStateConcurrencyTests +{ + /// Threads that share the one source or book, each with its own instance id. + private const int Workers = 4; + + /// Other targets' positions already in the table. A prune or a position read walks all of them, so the + /// window a thread can be caught in is wide enough to hit inside a bounded run. + private const int IdleInstances = 600; + + /// Rotations each thread makes: a new log file every round, as the hourly rotation does. + private const int Rotations = 1500; + + /// Rounds for the books and the debounce table, whose operations are short. + private const int ShortRounds = 20_000; + + private static string FileFor(int hour) => "error/postgresql.log.2026-10-03-" + hour.ToString("D5"); + + private static string HostFor(string instance) => instance + ".abc123.us-east-1.rds.amazonaws.com"; + + /// + /// A client that answers from the instance id on each request, so one object can serve every thread. The source + /// disposes the client it is handed after each read, which a client shared by four threads must survive. + /// + private sealed class SharedRds : AmazonRDSClient + { + public SharedRds() : base(new Amazon.Runtime.BasicAWSCredentials("a", "b"), Amazon.RegionEndpoint.USEast1) { } + + /// The newest log file's hour for each instance. The listing also carries the hour before it, as RDS + /// does until the older file ages out. + public ConcurrentDictionary Hour { get; } = new(); + + /// Instances whose stderr file reads ten minutes newer than their csv file right now. + public ConcurrentDictionary StaleCsv { get; } = new(); + + protected override void Dispose(bool disposing) + { + /* Shared by every thread, and the source disposes whatever the factory hands it. */ + } + + public override Task DescribeDBLogFilesAsync( + DescribeDBLogFilesRequest request, CancellationToken cancellationToken = default) + { + var instance = request.DBInstanceIdentifier; + var hour = Hour.TryGetValue(instance, out var h) ? h : 1; + var stale = StaleCsv.TryGetValue(instance, out var s) && s; + + var files = new List + { + new() { LogFileName = FileFor(hour), LastWritten = hour * 10L + (stale ? 10 * 60 * 1000 : 0) }, + new() { LogFileName = FileFor(hour) + ".csv", LastWritten = hour * 10L }, + }; + + if (hour > 1) + { + files.Add(new() { LogFileName = FileFor(hour - 1), LastWritten = (hour - 1) * 10L }); + } + + return Task.FromResult(new DescribeDBLogFilesResponse { DescribeDBLogFiles = files }); + } + + public override Task DownloadDBLogFilePortionAsync( + DownloadDBLogFilePortionRequest request, CancellationToken cancellationToken = default) + => Task.FromResult(new DownloadDBLogFilePortionResponse + { + LogFileData = string.Empty, + Marker = "m-" + request.LogFileName, + AdditionalDataPending = false, + }); + } + + /// Runs on a thread of its own for each of workers, all + /// released together, and completes when every one has finished. A worker that throws fails the run with its + /// exception. + private static async Task RunTogetherAsync(int workers, Action body) + { + using var barrier = new Barrier(workers); + + var tasks = Enumerable.Range(0, workers) + .Select(worker => Task.Factory.StartNew( + () => + { + barrier.SignalAndWait(); + body(worker); + }, + CancellationToken.None, + TaskCreationOptions.LongRunning, + TaskScheduler.Default)) + .ToArray(); + + await Task.WhenAll(tasks).WaitAsync(TimeSpan.FromSeconds(60)); + } + + /// + /// Four targets read their position and commit a chunk against one source while every one of them rotates onto a + /// new file each round: the hourly rotation, run until it collides. A commit adds the new file's key and prunes the + /// old one by walking the table, and a position read walks it too, so one thread's add lands inside another + /// thread's walk. Nothing may throw, and afterwards every target still holds exactly the position it last + /// committed (and the targets that were idle all along still hold theirs). + /// + [Fact] + public async Task TargetsRotatingOntoNewLogFilesAtOnceEachKeepTheirOwnPosition() + { + var client = new SharedRds(); + var source = new RdsLogSource(_ => client); + + for (var i = 0; i < IdleInstances; i++) + { + Assert.True(source.RestorePosition(RdsLogSource.LogFileKind.Stderr, "idle-" + i, FileFor(0), "0")); + } + + var failed = 0; + + await RunTogetherAsync(Workers, worker => + { + var instance = "live-" + worker; + + try + { + for (var hour = 1; hour <= Rotations && Volatile.Read(ref failed) == 0; hour++) + { + client.Hour[instance] = hour; + + var chunk = source.ReadNewestAsync(HostFor(instance), RdsLogSource.LogFileKind.Stderr) + .GetAwaiter().GetResult(); + + source.CommitResume(chunk!.Value.Resume); + } + } + catch + { + Volatile.Write(ref failed, 1); + throw; + } + }); + + for (var worker = 0; worker < Workers; worker++) + { + var instance = "live-" + worker; + + Assert.True(source.HasMarkerForKey(instance + "|" + FileFor(Rotations)), instance + " lost its last position."); + Assert.False(source.HasMarkerForKey(instance + "|" + FileFor(Rotations - 1)), instance + " kept a position for a file it had moved past."); + } + + for (var i = 0; i < IdleInstances; i++) + { + Assert.True(source.HasMarkerForKey("idle-" + i + "|" + FileFor(0)), "idle-" + i + " lost its position."); + } + } + + /// + /// The csv route keeps, for each instance, when the stale-csv condition was first seen, and clears it the moment + /// the condition goes. Four targets flip between the two states against one source. Afterwards each of them is + /// stale again, so once the debounce has passed every one of them must raise ; + /// a stamp lost to a torn table restarts the debounce instead, and the read answers as if csvlog were fine. + /// + [Fact] + public async Task TargetsFlippingTheStaleCsvConditionAtOnceEachKeepTheirOwnDebounce() + { + var client = new SharedRds(); + var start = new DateTime(2026, 10, 3, 5, 0, 0, DateTimeKind.Utc); + var now = start; + var source = new RdsLogSource(_ => client, () => now); + + var failed = 0; + + await RunTogetherAsync(Workers, worker => + { + var instance = "live-" + worker; + + try + { + /* An even round is stale and an odd one is not, and the last round is even: the stamp exists at the end. */ + for (var round = 0; round <= ShortRounds && Volatile.Read(ref failed) == 0; round++) + { + client.StaleCsv[instance] = round % 2 == 0; + + source.ReadNewestAsync(HostFor(instance), RdsLogSource.LogFileKind.Csv).GetAwaiter().GetResult(); + } + } + catch + { + Volatile.Write(ref failed, 1); + throw; + } + }); + + now = start.AddMinutes(6); + + for (var worker = 0; worker < Workers; worker++) + { + var instance = "live-" + worker; + + await Assert.ThrowsAsync( + () => source.ReadNewestAsync(HostFor(instance), RdsLogSource.LogFileKind.Csv)); + } + } + + /// + /// The csvlog partial-record carry is one book for the ingestor, keyed by instance, and every target's run goes + /// through it. Four targets set and clear their own entries at once; afterwards each must still hold the carry it + /// set last. + /// + [Fact] + public async Task TargetsCommittingCsvlogCarriesAtOnceEachKeepTheirOwnCarry() + { + var book = new RdsCsvlogCarryBook(); + var failed = 0; + + await RunTogetherAsync(Workers, worker => + { + var instance = "live-" + worker; + + try + { + /* An even round holds a partial record, an odd one finishes it (the entry is cleared), and the last round is even. */ + for (var round = 0; round <= ShortRounds && Volatile.Read(ref failed) == 0; round++) + { + var (_, key, dropped, file) = book.CarryFor(instance + "|" + FileFor(round), startsAtFileStart: false); + + var next = round % 2 == 0 + ? new RdsCsvlogCarry.CsvCarry("partial-" + round, StartKnown: true) + : new RdsCsvlogCarry.CsvCarry(string.Empty, StartKnown: false); + + book.Commit(key, file, next, dropped); + } + } + catch + { + Volatile.Write(ref failed, 1); + throw; + } + }); + + for (var worker = 0; worker < Workers; worker++) + { + var (carry, _, _, _) = book.CarryFor("live-" + worker + "|" + FileFor(ShortRounds), startsAtFileStart: false); + + Assert.Equal("partial-" + ShortRounds, carry.Partial); + } + } + + /// The same as the csvlog book, for the deadlock report held back until the rest of it arrives. + [Fact] + public async Task TargetsCommittingHeldDeadlockReportsAtOnceEachKeepTheirOwnReport() + { + var book = new RdsDeadlockCarryBook(); + var failed = 0; + + await RunTogetherAsync(Workers, worker => + { + var instance = "live-" + worker; + + try + { + for (var round = 0; round <= ShortRounds && Volatile.Read(ref failed) == 0; round++) + { + var (_, key, file, _) = book.CarryFor(instance + "|" + FileFor(round)); + + book.Commit(key, file, round % 2 == 0 ? "report-" + round : string.Empty); + } + } + catch + { + Volatile.Write(ref failed, 1); + throw; + } + }); + + for (var worker = 0; worker < Workers; worker++) + { + var (held, _, _, _) = book.CarryFor("live-" + worker + "|" + FileFor(ShortRounds)); + + Assert.Equal("report-" + ShortRounds, held); + } + } + + /// Entries each writer adds in the read tests: enough that the book's table is rebuilt several times, up to a + /// size where one rebuild takes long enough for a reader to be inside it. + private const int GrowthKeysPerWriter = 20_000; + + /// Fresh books each read test grows from empty, so every one of them goes through the rebuilds again. + private const int GrowthRepeats = 5; + + /// + /// One run of the read tests against one book, through the two operations the readers and writers use: + /// holds a value for a target by its resume key, and returns what the + /// book holds for a resume key ("" for nothing). Each of readers has an entry stored before the + /// run starts and reads it in a loop until every writer is done. The same number of writers add thousands of entries + /// for other targets, which grows the table and rebuilds it again and again. A reader's entry is never touched, so + /// every read must find it holding what was stored. + /// + private static async Task ReadsKeepFindingTheirEntryWhileOthersAddTheirsAsync( + Action store, Func read) + { + for (var reader = 0; reader < Workers; reader++) + { + store("reader-" + reader + "|" + FileFor(1), "held-" + reader); + } + + var writersLeft = Workers; + var failed = 0; + + await RunTogetherAsync(Workers * 2, worker => + { + try + { + if (worker < Workers) + { + var resumeKey = "reader-" + worker + "|" + FileFor(1); + var expected = "held-" + worker; + + while (Volatile.Read(ref writersLeft) > 0 && Volatile.Read(ref failed) == 0) + { + var held = read(resumeKey); + + if (!string.Equals(held, expected, StringComparison.Ordinal)) + { + Assert.Equal(expected, held); + } + } + } + else + { + for (var i = 0; i < GrowthKeysPerWriter && Volatile.Read(ref failed) == 0; i++) + { + store("writer-" + worker + "-" + i + "|" + FileFor(1), "added"); + } + } + } + catch + { + Volatile.Write(ref failed, 1); + throw; + } + finally + { + if (worker >= Workers) + { + Interlocked.Decrement(ref writersLeft); + } + } + }); + } + + /// + /// A read of the csvlog carry book takes the book's lock like a write does. A read that does not can walk the table + /// while another thread is rebuilding it larger and answer "nothing held" for an entry that has been there all along + /// (or throw), and a carry that is missed is a partial record whose tail is never glued back. The other tests of the + /// book read only the entry their own thread writes, in a table that never grows, which a missing lock does not show. + /// + [Fact] + public async Task TargetsReadingCsvlogCarriesWhileOthersAddTheirsStillFindTheirOwn() + { + for (var repeat = 0; repeat < GrowthRepeats; repeat++) + { + var book = new RdsCsvlogCarryBook(); + + await ReadsKeepFindingTheirEntryWhileOthersAddTheirsAsync( + (resumeKey, held) => + { + var (_, key, dropped, file) = book.CarryFor(resumeKey, startsAtFileStart: false); + + book.Commit(key, file, new RdsCsvlogCarry.CsvCarry(held, StartKnown: true), dropped); + }, + resumeKey => book.CarryFor(resumeKey, startsAtFileStart: false).Carry.Partial); + } + } + + /// The same as the csvlog book, for the deadlock report held back until the rest of it arrives. + [Fact] + public async Task TargetsReadingHeldDeadlockReportsWhileOthersAddTheirsStillFindTheirOwn() + { + for (var repeat = 0; repeat < GrowthRepeats; repeat++) + { + var book = new RdsDeadlockCarryBook(); + + await ReadsKeepFindingTheirEntryWhileOthersAddTheirsAsync( + (resumeKey, held) => + { + var (_, key, file, _) = book.CarryFor(resumeKey); + + book.Commit(key, file, held); + }, + resumeKey => book.CarryFor(resumeKey).Held); + } + } + + /// Threads that ask one runner for its ingestors at once. + private const int PublishThreads = 8; + + /// + /// The runner is one object for every target, and it builds each RDS ingestor the first time a target needs it. Two + /// targets' first runs get there together: a plain ??= lets each build an ingestor and keep its own, so one of + /// the two (and the positions and carries it holds) is dropped while its run is still going. Eight threads ask a fresh + /// runner for every ingestor at the same moment, over and over for a second, and all eight must be handed the same + /// object of each kind every time. + /// + [Fact] + public async Task TheRunnerHandsEveryTargetTheSameIngestorForEachRdsCollector() + { + await using var store = NpgsqlDataSource.Create("Host=127.0.0.1;Port=1;Database=unused;Username=unused"); + var logHashKey = new PgLogHashKey(Enumerable.Repeat((byte)7, PgLogHashKey.KeyLength).ToArray()); + var clock = Stopwatch.StartNew(); + var rounds = 0; + + while (clock.Elapsed < TimeSpan.FromSeconds(1)) + { + var runner = new DarlingCollectorRunner(store, new CollectorDeltaCalculator()); + var plans = new object[PublishThreads]; + var deadlocks = new object[PublishThreads]; + var logEvents = new object[PublishThreads]; + var cpus = new object[PublishThreads]; + + await RunTogetherAsync(PublishThreads, worker => + { + plans[worker] = runner.PublishedRdsPlanIngestor(); + deadlocks[worker] = runner.PublishedRdsDeadlockIngestor(); + logEvents[worker] = runner.PublishedRdsLogEventIngestor(logHashKey); + cpus[worker] = runner.PublishedRdsCpuIngestor(); + }); + + AssertOneInstance("plan", plans); + AssertOneInstance("deadlock", deadlocks); + AssertOneInstance("log event", logEvents); + AssertOneInstance("cpu", cpus); + + rounds++; + } + + Assert.True(rounds > 0); + } + + private static void AssertOneInstance(string collector, object[] handed) + => Assert.True( + handed.All(item => ReferenceEquals(item, handed[0])), + "The " + collector + " ingestor: the threads of one runner were handed " + + handed.Distinct(ReferenceEqualityComparer.Instance).Count() + " different objects."); +} diff --git a/Darling/Darling.Tests/RdsResumePersistenceTests.cs b/Darling/Darling.Tests/RdsResumePersistenceTests.cs index 1e61e9183a..98be968fa8 100644 --- a/Darling/Darling.Tests/RdsResumePersistenceTests.cs +++ b/Darling/Darling.Tests/RdsResumePersistenceTests.cs @@ -145,19 +145,19 @@ private static Func> Ingestor( case "plans": { var ingestor = new RdsPlanIngestor(store, logs, resume: resume); - return () => ingestor.IngestAsync(1, "s", Host, csv, CancellationToken.None); + return () => ingestor.IngestAsync(1, "s", Host, csv, cancellationToken: CancellationToken.None); } case "deadlocks": { var ingestor = new RdsDeadlockIngestor(store, logs, resume: resume); - return () => ingestor.IngestAsync(1, "s", Host, false, csv, CancellationToken.None); + return () => ingestor.IngestAsync(1, "s", Host, false, csv, cancellationToken: CancellationToken.None); } default: { var ingestor = new RdsLogEventIngestor(store, TestLogHashKeys.Fixed, logs, resume: resume); - return () => ingestor.IngestAsync(1, "s", Host, false, csv, CancellationToken.None); + return () => ingestor.IngestAsync(1, "s", Host, false, csv, cancellationToken: CancellationToken.None); } } } @@ -450,7 +450,7 @@ public async Task APositionSavedThroughTheRunnerSurvivesANewProcess() var first = new RdsPlanIngestor( postgres, new RdsLogSource(_ => client), resume: runner.RdsResumeStoreFor("pg_plan_capture")); - await first.IngestAsync(TestServerId, "s", Host, false, TestContext.Current.CancellationToken); + await first.IngestAsync(TestServerId, "s", Host, false, cancellationToken: TestContext.Current.CancellationToken); using (var connection = new NpgsqlConnection(connectionString)) { @@ -471,7 +471,7 @@ public async Task APositionSavedThroughTheRunnerSurvivesANewProcess() var restarted = new RdsPlanIngestor( postgres, new RdsLogSource(_ => client), resume: runner.RdsResumeStoreFor("pg_plan_capture")); - await restarted.IngestAsync(TestServerId, "s", Host, false, TestContext.Current.CancellationToken); + await restarted.IngestAsync(TestServerId, "s", Host, false, cancellationToken: TestContext.Current.CancellationToken); Assert.Equal(Newest, client.Downloads[0].LogFileName); Assert.Equal("M1", client.Downloads[0].Marker); diff --git a/Darling/Darling.Tests/ReadLatencyAccumulatorTests.cs b/Darling/Darling.Tests/ReadLatencyAccumulatorTests.cs index 1d91ae52ba..170ae9ad6d 100644 --- a/Darling/Darling.Tests/ReadLatencyAccumulatorTests.cs +++ b/Darling/Darling.Tests/ReadLatencyAccumulatorTests.cs @@ -6,6 +6,7 @@ * Licensed under the MIT License. See LICENSE file in the project root for full license information. */ +using System; using System.Linq; using System.Threading.Tasks; using PerformanceMonitor.Darling.Service; @@ -18,6 +19,22 @@ namespace Darling.Tests; /// drain-resets, and bucket-edge placement. public sealed class ReadLatencyAccumulatorTests { + [Fact] + public void OutcomeLabels_AreTheClosedSet_AndTheOldFiveKeepTheirStoredSpelling() + { + var all = Enum.GetValues(); + + Assert.Equal( + new[] { "ok", "timeout", "cancelled", "error", "limit", "fallback_raw", "gate_failed" }, + all.Select(ReadLatencyAccumulator.OutcomeLabel).ToArray()); + foreach (var old in all.Where(o => o <= ReadOutcome.Limit)) + { + Assert.Equal(old.ToString().ToLowerInvariant(), ReadLatencyAccumulator.OutcomeLabel(old)); + } + + Assert.Throws(() => ReadLatencyAccumulator.OutcomeLabel((ReadOutcome)99)); + } + [Fact] public async Task Record_ManyConcurrentThreads_DrainTotalsMatch() { diff --git a/Darling/Darling.Tests/ReadLatencyBudgetEdgeTests.cs b/Darling/Darling.Tests/ReadLatencyBudgetEdgeTests.cs new file mode 100644 index 0000000000..8fa709dfe5 --- /dev/null +++ b/Darling/Darling.Tests/ReadLatencyBudgetEdgeTests.cs @@ -0,0 +1,43 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System.Collections.Generic; +using System.Linq; +using System.Text; +using PerformanceMonitor.Darling.Service.Mcp; +using Xunit; + +namespace Darling.Tests; + +/// +/// The get_read_latency page never serializes past its byte budget, even when the greedy fill lands within a few +/// bytes of it (#5135): the envelope's reads_returned digits and truncated spelling depend on the final page. +/// +public sealed class ReadLatencyBudgetEdgeTests +{ + private static object[] Rows(int n) => Enumerable.Range(0, n) + .Select(i => (object)new { surface = "web", route = "route_" + i.ToString("D4"), count = 1000 - i, pad = new string('x', 120) }) + .ToArray(); + + [Fact] + public void Response_NeverExceedsBudget_WhenTheFillLandsWithinAFewBytesOfIt() + { + var rows = Rows(150); + var full = Encoding.UTF8.GetByteCount(DarlingMcpReadLatencyTools.BuildResponse(24, null, null, rows, 1000, int.MaxValue)); + + var trimmedAtLeastOnce = false; + for (var budget = full - 12; budget <= full + 2; budget++) + { + var json = DarlingMcpReadLatencyTools.BuildResponse(24, null, null, rows, 1000, budget); + Assert.True(Encoding.UTF8.GetByteCount(json) <= budget, $"budget {budget}: got {Encoding.UTF8.GetByteCount(json)}"); + trimmedAtLeastOnce |= json.Contains("\"truncated\":true"); + } + + Assert.True(trimmedAtLeastOnce); + } +} diff --git a/Darling/Darling.Tests/ReadLatencyFlushLiveTests.cs b/Darling/Darling.Tests/ReadLatencyFlushLiveTests.cs index 865b8dbe97..3d546cd1b8 100644 --- a/Darling/Darling.Tests/ReadLatencyFlushLiveTests.cs +++ b/Darling/Darling.Tests/ReadLatencyFlushLiveTests.cs @@ -172,6 +172,35 @@ public async Task ARecordAndFlushRoundTrip_MatchesCountsTotalsMaxesAndBucketsEle Assert.Equal(3, rows); } + [Fact] + public async Task AFallbackAndAGateFailure_AreStoredWithTheirOwnOutcomeLabels() + { + var baseConnectionString = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the read-latency outcome pin."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var accumulator = new ReadLatencyAccumulator(); + accumulator.Record(ReadSurface.Mcp, "get_query_store_top", ReadOutcome.FallbackRaw, 40); + accumulator.Record(ReadSurface.Mcp, "get_query_store_top", ReadOutcome.GateFailed, 40); + await accumulator.FlushAsync(connection, new DateTime(2026, 1, 1, 5, 0, 0, DateTimeKind.Utc), logger: null, ct); + + using var read = new NpgsqlCommand("SELECT outcome FROM collect.read_latency ORDER BY outcome", connection); + await using var reader = await read.ExecuteReaderAsync(ct); + var labels = new System.Collections.Generic.List(); + while (await reader.ReadAsync(ct)) + { + labels.Add(reader.GetString(0)); + } + + Assert.Equal(new[] { "fallback_raw", "gate_failed" }, labels); + } + [Fact] public async Task ASecondFlushInTheSameHour_AddsToTheFirst() { diff --git a/Darling/Darling.Tests/ReadLatencyWebRecordingTests.cs b/Darling/Darling.Tests/ReadLatencyWebRecordingTests.cs index d8e4843a54..0ac5f3385a 100644 --- a/Darling/Darling.Tests/ReadLatencyWebRecordingTests.cs +++ b/Darling/Darling.Tests/ReadLatencyWebRecordingTests.cs @@ -196,6 +196,36 @@ public async Task ADispatchEntryThatThrowsA57014PostgresException_RecordsOneWebS Assert.Equal(1, sample.Count); } + [Fact] + public async Task ADispatchEntryThatNotesAFallback_RecordsOneWebSample_WithOutcomeFallbackRaw() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to run the read-latency web-recording pin."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + + const string routeName = "__test_fallback_raw"; + using var seam = new ExtraDispatchEntryScope( + (routeName, (_, _, _) => + { + ReadScope.NoteFallback(ReadFallback.FallbackRaw, "test dispatch entry", null); + return Task.FromResult("{\"ok\":true}"); + })); + + var readLatency = new ReadLatencyAccumulator(); + var server = await BuildServer(postgres, readLatency); + + var response = await Send(server, "/api/read/" + routeName); + Assert.Equal(HttpStatusCode.OK, response.StatusCode); + + var sample = Assert.Single(readLatency.Drain(), d => d.Surface == ReadSurface.Web && d.Route == routeName); + Assert.Equal(ReadOutcome.FallbackRaw, sample.Outcome); + } + /// /// Sets for the scope's lifetime and /// clears it on . The entry is held per async flow (#4782): only this fact, and the diff --git a/Darling/Darling.Tests/ReadScopeGateFaultLiveTests.cs b/Darling/Darling.Tests/ReadScopeGateFaultLiveTests.cs new file mode 100644 index 0000000000..6893bb18ea --- /dev/null +++ b/Darling/Darling.Tests/ReadScopeGateFaultLiveTests.cs @@ -0,0 +1,92 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Npgsql; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: deliberately NOT [Collection("live-postgres")]. Every test mints its own database through + ScratchPostgres and touches nothing on the shared one. */ + +/// +/// #5097: the source decision itself faulting on a connection that opens and accepts the read-only step. The +/// scratch database is migrated and then loses the wide-table coverage table, so the gate's own input read +/// faults. Both readers must note GateFailed and answer from raw, and the gate's single Warning is the +/// only line logged. +/// +public sealed class ReadScopeGateFaultLiveTests +{ + private const int ServerId = -5097001; + private const string ServerName = "gate-fault-live"; + + private static string? BaseConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static async Task FaultedStoreAsync(CancellationToken ct) + { + var scratch = await ScratchPostgres.CreateAsync(BaseConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + foreach (var sql in new[] + { + "INSERT INTO servers (server_id, server_name, display_name, is_enabled, sql_major_version, created_date, modified_date) " + + $"VALUES ({ServerId}, '{ServerName}', '{ServerName}', TRUE, 16, now(), now()) ON CONFLICT (server_id) DO UPDATE SET is_enabled = TRUE", + "DROP TABLE collect.query_store_interval_wide_coverage CASCADE", + }) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(ct); + } + + return scratch; + } + + [Fact] + public async Task TheMcpTopRead_WhenTheGateInputReadFaults_NotesGateFailed_AndReadsRaw() + { + Assert.SkipWhen(string.IsNullOrEmpty(BaseConnectionString), "Set DARLING_TEST_PG to a Postgres connection string to run the #5097 gate-fault live test."); + var ct = TestContext.Current.CancellationToken; + await using var scratch = await FaultedStoreAsync(ct); + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var end = new DateTime(2026, 9, 15, 0, 0, 0, DateTimeKind.Unspecified); + + var logger = new CapturingTestLogger(); + using var scope = ReadScope.Open(logger); + var result = await DarlingDataReader.GetQueryStoreTopWithReachAsync(postgres, ServerId, end.AddDays(-3), end, 10, null, null, null, ct); + + Assert.Null(result.Table); + Assert.Equal(ReadFallback.GateFailed, scope.Fallback); + Assert.Equal(1, logger.CountAtLevel(LogLevel.Warning)); + } + + [Fact] + public async Task TheComposeWideGate_WhenTheGateInputReadFaults_NotesGateFailed_AndReadsRaw() + { + Assert.SkipWhen(string.IsNullOrEmpty(BaseConnectionString), "Set DARLING_TEST_PG to a Postgres connection string to run the #5097 gate-fault live test."); + var ct = TestContext.Current.CancellationToken; + await using var scratch = await FaultedStoreAsync(ct); + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var end = new DateTime(2026, 9, 15, 0, 0, 0, DateTimeKind.Unspecified); + + var logger = new CapturingTestLogger(); + using var scope = ReadScope.Open(logger); + var resolution = await DarlingWebEndpoints.ResolveQueryStoreWideEligibleAsync(postgres, null, end.AddDays(-3), end, end, ct); + + Assert.False(resolution.Eligible); + Assert.Equal(ReadFallback.GateFailed, scope.Fallback); + Assert.Equal(1, logger.CountAtLevel(LogLevel.Warning)); + } +} diff --git a/Darling/Darling.Tests/ReadScopeStatementCaptureLiveTests.cs b/Darling/Darling.Tests/ReadScopeStatementCaptureLiveTests.cs new file mode 100644 index 0000000000..dfdd45c12b --- /dev/null +++ b/Darling/Darling.Tests/ReadScopeStatementCaptureLiveTests.cs @@ -0,0 +1,241 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Linq; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Service; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5097: per-statement capture inside a read. The live facts: the scope is visible inside the Npgsql activity +/// callbacks, when the activity stops, which tag carries a row count, what a command outside a scope costs, and +/// that fan-out loses no entry. +/// +[Collection("live-postgres")] +public sealed class ReadScopeStatementCaptureLiveTests +{ + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static NpgsqlDataSource Source() => NpgsqlDataSource.Create(ConnectionString!); + + private static void RequirePg() => + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), "Set DARLING_TEST_PG to run the #5097 statement-capture live tests."); + + [Fact] + public async Task ARealRead_InsideACapturingScope_DeliversItsTimings_ThroughTheAsyncLocal() + { + RequirePg(); + ReadStatementCapture.Register(); + await using var source = Source(); + /* Warm the pool so the read below runs on a pooled connection. */ + await using (var warm = await source.OpenConnectionAsync()) { } + + using var opened = ReadScope.Open(null); + opened.Scope.CaptureStatements = true; + await using (var command = source.CreateCommand("-- a comment\nSELECT 5097 AS capture_probe")) + { + await using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) { } + } + + var seen = opened.Scope.Snapshot(); + var probe = Assert.Single(seen, t => t.Label == "SELECT 5097 AS capture_probe"); + Assert.True(probe.DurationMs >= 0); + Assert.Equal(1, probe.Ordinal > 0 ? 1 : 0); + Assert.Equal(8, probe.Hash.Length); + } + + [Fact] + public async Task TheActivity_StopsAtReaderClose_SoTheDurationIncludesTheHeldReader() + { + RequirePg(); + ReadStatementCapture.Register(); + await using var source = Source(); + using var opened = ReadScope.Open(null); + opened.Scope.CaptureStatements = true; + + /* Postgres buffers the server's output, so a slow generate_series delivers its first row late; the + stop point is therefore probed from the client: read ONE row, hold the reader open for 600 ms, then + close it. A stop at first result would be a few ms; a stop at reader close includes the 600 ms. */ + var watch = Stopwatch.StartNew(); + long firstRowMs; + await using (var command = source.CreateCommand("SELECT g FROM generate_series(1, 5) AS g")) + { + await using var reader = await command.ExecuteReaderAsync(); + Assert.True(await reader.ReadAsync()); + firstRowMs = watch.ElapsedMilliseconds; + await Task.Delay(600); + } + + var totalMs = watch.ElapsedMilliseconds; + var timing = Assert.Single(opened.Scope.Snapshot(), t => t.Label.StartsWith("SELECT g FROM generate_series", StringComparison.Ordinal)); + System.IO.File.AppendAllText(System.IO.Path.Combine(System.IO.Path.GetTempPath(), "pm5097-capture-evidence.txt"), + $"(b) firstRowMs={firstRowMs} totalMs={totalMs} activityMs={timing.DurationMs:F0} rows={timing.Rows?.ToString() ?? "null"}\n"); + Assert.True(timing.DurationMs >= 550, $"duration {timing.DurationMs} ms stops before the reader closed (first row at {firstRowMs} ms)"); + } + + [Fact] + public async Task TheStoppedActivity_TagNames_AreListed_AndTheRowCountTagIsReadWhenPresent() + { + RequirePg(); + await using var source = Source(); + var tags = new List(); + using var listener = new ActivityListener + { + ShouldListenTo = s => s.Name == "Npgsql", + Sample = (ref ActivityCreationOptions _) => ActivitySamplingResult.AllDataAndRecorded, + ActivityStopped = a => + { + if (a.TagObjects.Any(t => t.Key == "db.query.text" && (t.Value as string)?.Contains("tag_probe_5097") == true)) + { + lock (tags) { tags.AddRange(a.TagObjects.Select(t => t.Key + "=" + t.Value)); } + } + }, + }; + ActivitySource.AddActivityListener(listener); + + await using (var command = source.CreateCommand("SELECT g AS tag_probe_5097 FROM generate_series(1, 7) AS g")) + { + await using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) { } + } + + string[] snapshot; + lock (tags) { snapshot = tags.ToArray(); } + System.IO.File.AppendAllText(System.IO.Path.Combine(System.IO.Path.GetTempPath(), "pm5097-capture-evidence.txt"), + "(c) tags: " + string.Join(" | ", snapshot.Select(t => t.Length > 90 ? t[..90] : t)) + "\n"); + Assert.NotEmpty(snapshot); + Assert.Contains(snapshot, t => t.StartsWith("db.query.text=", StringComparison.Ordinal)); + } + + [Fact] + public async Task ACommandOutsideAnyScope_IsNotSampled_AndCreatesNoActivity() + { + RequirePg(); + ReadStatementCapture.Register(); + await using var source = Source(); + + var options = default(ActivityCreationOptions); + Assert.Null(ReadScope.Current); + Assert.Equal(ActivitySamplingResult.None, ReadStatementCapture.Sample(ref options)); + + using (var opened = ReadScope.Open(null)) + { + /* A scope that did not ask for capture is also not sampled. */ + Assert.Equal(ActivitySamplingResult.None, ReadStatementCapture.Sample(ref options)); + opened.Scope.CaptureStatements = true; + Assert.Equal(ActivitySamplingResult.AllDataAndRecorded, ReadStatementCapture.Sample(ref options)); + } + + /* The command still runs normally outside any scope. */ + await using var plain = source.CreateCommand("SELECT 1"); + Assert.Equal(1, await plain.ExecuteScalarAsync()); + } + + [Fact] + public async Task ConcurrentCommands_InsideOneScope_AreAllCaptured() + { + RequirePg(); + ReadStatementCapture.Register(); + await using var source = Source(); + using var opened = ReadScope.Open(null); + opened.Scope.CaptureStatements = true; + + const int Fan = 32; + await Task.WhenAll(Enumerable.Range(0, Fan).Select(async i => + { + await using var command = source.CreateCommand($"SELECT {i} AS fan_probe_5097, pg_sleep(0.05)"); + await command.ExecuteScalarAsync(); + })); + + var seen = opened.Scope.Snapshot().Where(t => t.Label.Contains("fan_probe_5097", StringComparison.Ordinal)).ToList(); + Assert.Equal(Fan, seen.Count); + Assert.Equal(Fan, seen.Select(t => t.Label).Distinct().Count()); + Assert.Equal(Fan, seen.Select(t => t.Ordinal).Distinct().Count()); + } +} + +/// #5097: the label function and the queue cap (no database). +public sealed class ReadScopeStatementLabelTests +{ + [Fact] + public void TheLabel_SkipsLeadingCommentsAndBlankLines() + { + Assert.Equal("SELECT a FROM t", StatementLabel.Label("-- note\n\n -- more\n SELECT a FROM t\nWHERE x = 1")); + Assert.Equal("SELECT a FROM t", StatementLabel.Label("--c\r\nSELECT a FROM t\r\nWHERE x")); + } + + [Fact] + public void TheLabel_IsCutAt120Characters() + { + var label = StatementLabel.Label("SELECT " + new string('x', 400)); + Assert.Equal(120, label.Length); + } + + [Fact] + public void TheHash_IsStable_ShortAndDependsOnTheWholeText() + { + Assert.Equal(StatementLabel.Hash("SELECT 1"), StatementLabel.Hash("SELECT 1")); + Assert.NotEqual(StatementLabel.Hash("SELECT 1"), StatementLabel.Hash("SELECT 2")); + Assert.Equal(8, StatementLabel.Hash("SELECT 1").Length); + /* Known SHA-256 prefix, so a change of algorithm is noticed. */ + Assert.Equal("e1b0c442".Length, StatementLabel.Hash(string.Empty).Length); + Assert.Equal("e3b0c442", StatementLabel.Hash(string.Empty)); + } + + [Fact] + public void ANullOrEmptyText_GivesAnEmptyLabel() => Assert.Equal(string.Empty, StatementLabel.Label(null)); + + [Fact] + public void TheQueue_IsCapped_AndCountsTheOverflow() + { + using var opened = ReadScope.Open(null); + for (var i = 0; i < ReadScope.MaxStatements + 10; i++) + { + opened.Scope.AddStatement("SELECT " + i, 1, null); + } + + Assert.Equal(ReadScope.MaxStatements, opened.Scope.Snapshot().Count); + Assert.Equal(10, opened.Scope.StatementsDropped); + Assert.Equal(1, opened.Scope.Snapshot()[0].Ordinal); + } + + [Fact] + public void NoteSource_KeepsOnlyTheFixedVocabulary_AndNoteRows_Records() + { + using var opened = ReadScope.Open(null); + ReadScope.NoteSource("something_else"); + Assert.Null(opened.Scope.Source); + ReadScope.NoteSource(ReadScope.SourceIntervalTable, "gate"); + ReadScope.NoteRows(12); + Assert.Equal("interval_table", opened.Scope.Source); + Assert.Equal("gate", opened.Scope.SourceReason); + Assert.Equal(12, opened.Scope.Rows); + } + + [Fact] + public void NoteSourceAndRows_OutsideAScope_DoNothing() + { + Assert.Null(ReadScope.Current); + ReadScope.NoteSource(ReadScope.SourceRaw); + ReadScope.NoteRows(3); + } + + [Fact] + public void ARecorderOpenedScope_DoesNotCaptureUntilTheRecorderAsks() + { + using var opened = ReadScope.Open(null); + Assert.False(opened.Scope.CaptureStatements); + } +} diff --git a/Darling/Darling.Tests/ReadScopeTests.cs b/Darling/Darling.Tests/ReadScopeTests.cs new file mode 100644 index 0000000000..f4a3278ea9 --- /dev/null +++ b/Darling/Darling.Tests/ReadScopeTests.cs @@ -0,0 +1,260 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Npgsql; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5097: the ambient a recorded read runs inside, and the readers that note a silent +/// raw fallback into it. No database: every fault is a store that cannot be reached. +/// +public sealed class ReadScopeTests +{ + /// A port nothing listens on, so the open is refused at once. Never localhost: a shard may + /// have a live Postgres there. + private const string RefusedConnectionString = "Host=127.0.0.1;Port=1;Username=u;Password=p;Database=d;Timeout=2;Command Timeout=2;Pooling=false"; + + private static async Task NoteInChildAsync(ReadFallback kind) + { + await Task.Yield(); + ReadScope.NoteFallback(kind, "child", null); + } + + [Fact] + public async Task AFallbackNotedInAChildAsyncMethod_IsVisibleToTheParentAfterTheAwait() + { + using var scope = ReadScope.Open(null); + await NoteInChildAsync(ReadFallback.FallbackRaw); + + Assert.Equal(ReadFallback.FallbackRaw, scope.Fallback); + } + + [Fact] + public async Task TwoConcurrentScopes_DoNotCrossNotes() + { + async Task RunAsync(ReadFallback? note) + { + using var scope = ReadScope.Open(null); + await Task.Yield(); + if (note is { } kind) + { + await NoteInChildAsync(kind); + } + + await Task.Delay(20, TestContext.Current.CancellationToken); + return scope.Fallback; + } + + var results = await Task.WhenAll(RunAsync(ReadFallback.GateFailed), RunAsync(null), RunAsync(ReadFallback.FallbackRaw)); + + Assert.Equal(new ReadFallback?[] { ReadFallback.GateFailed, null, ReadFallback.FallbackRaw }, results); + } + + [Fact] + public void Resolve_ReplacesOnlyOk_AndGateFailedOutranksFallbackRaw() + { + Assert.Equal(ReadOutcome.FallbackRaw, ReadScope.Resolve(ReadOutcome.Ok, ReadFallback.FallbackRaw)); + Assert.Equal(ReadOutcome.GateFailed, ReadScope.Resolve(ReadOutcome.Ok, ReadFallback.GateFailed)); + Assert.Equal(ReadOutcome.Ok, ReadScope.Resolve(ReadOutcome.Ok, null)); + foreach (var bigger in new[] { ReadOutcome.Timeout, ReadOutcome.Cancelled, ReadOutcome.Error, ReadOutcome.Limit }) + { + Assert.Equal(bigger, ReadScope.Resolve(bigger, ReadFallback.GateFailed)); + Assert.Equal(bigger, ReadScope.Resolve(bigger, ReadFallback.FallbackRaw)); + } + + using var scope = ReadScope.Open(null); + ReadScope.NoteFallback(ReadFallback.GateFailed, "a", null); + ReadScope.NoteFallback(ReadFallback.FallbackRaw, "b", null); + Assert.Equal(ReadOutcome.GateFailed, ReadScope.Resolve(ReadOutcome.Ok, scope.Fallback)); + } + + [Fact] + public void ANoteWithNoScope_DoesNotThrow_AndDisposeRestoresThePreviousScope() + { + Assert.Null(ReadScope.Current); + ReadScope.NoteFallback(ReadFallback.GateFailed, "unscoped", new InvalidOperationException("x")); + ReadScope.Warn("unscoped", null); + + using var outer = ReadScope.Open(null); + using (var inner = ReadScope.Open(null)) + { + Assert.Same(inner.Scope, ReadScope.Current); + } + + Assert.Same(outer.Scope, ReadScope.Current); + } + + [Fact] + public void ANoteReachesTheScopeLoggerOnce_AtWarning_WithTheException() + { + var logger = new CapturingTestLogger(); + using var scope = ReadScope.Open(logger); + + var thrown = new InvalidOperationException("boom"); + ReadScope.NoteFallback(ReadFallback.FallbackRaw, "a site", thrown); + + Assert.Equal(1, logger.CountAtLevel(LogLevel.Warning)); + Assert.Contains("a site", logger.Joined, StringComparison.Ordinal); + var logged = Assert.Single(logger.ExceptionsAtLevel(LogLevel.Warning)); + Assert.Same(thrown, logged); + } + + [Fact] + public void ALedgerUncoveredNote_ResolvesToFallbackRaw_ReplacesOnlyOk_AndLogsNothingAtAnyLevel() + { + Assert.Equal(ReadOutcome.FallbackRaw, ReadScope.Resolve(ReadOutcome.Ok, ReadFallback.LedgerUncovered)); + foreach (var bigger in new[] { ReadOutcome.Timeout, ReadOutcome.Cancelled, ReadOutcome.Error, ReadOutcome.Limit }) + { + Assert.Equal(bigger, ReadScope.Resolve(bigger, ReadFallback.LedgerUncovered)); + } + + var logger = new CapturingTestLogger(); + using var scope = ReadScope.Open(logger); + ReadScope.Note(ReadFallback.LedgerUncovered); + + Assert.Equal(ReadFallback.LedgerUncovered, scope.Fallback); + Assert.Empty(logger.Lines); + } + + [Fact] + public void TheReasonsRank_GateFailedThenFallbackRaw_ThenLedgerUncovered_InAnyOrder_ThroughBothKindsOfNote() + { + /* weakest first: the held reason is whichever of the two noted ranks higher, and an equal one stays itself */ + var weakestFirst = new[] { ReadFallback.LedgerUncovered, ReadFallback.FallbackRaw, ReadFallback.GateFailed }; + for (var first = 0; first < weakestFirst.Length; first++) + { + for (var second = 0; second < weakestFirst.Length; second++) + { + foreach (var firstQuiet in new[] { true, false }) + { + foreach (var secondQuiet in new[] { true, false }) + { + using var scope = ReadScope.Open(null); + NoteEither(weakestFirst[first], firstQuiet); + NoteEither(weakestFirst[second], secondQuiet); + + Assert.Equal(weakestFirst[Math.Max(first, second)], scope.Fallback); + } + } + } + } + } + + private static void NoteEither(ReadFallback kind, bool quiet) + { + if (quiet) + { + ReadScope.Note(kind); + } + else + { + ReadScope.NoteFallback(kind, "rank test", null); + } + } + + [Fact] + public void ANoteWithNoLog_RecordsTheFallback_AndWritesNothing() + { + var logger = new CapturingTestLogger(); + using var scope = ReadScope.Open(logger); + + ReadScope.Note(ReadFallback.FallbackRaw); + Assert.Equal(ReadFallback.FallbackRaw, scope.Fallback); + ReadScope.Note(ReadFallback.GateFailed); + ReadScope.Note(ReadFallback.FallbackRaw); + + Assert.Equal(ReadFallback.GateFailed, scope.Fallback); + Assert.Equal(0, logger.CountAtLevel(LogLevel.Warning)); + } + + [Fact] + public async Task TheGateFaulting_ReturnsAPlanThatSaysTheDecisionFailed_AndWarnsOnce() + { + var logger = new CapturingTestLogger(); + await using var connection = new NpgsqlConnection(RefusedConnectionString); + var end = new DateTime(2026, 9, 15, 0, 0, 0, DateTimeKind.Utc); + + var plan = await QueryStoreIntervalWide.ResolveReadAsync( + connection, 1, end.AddHours(-48), end, end, TimeSpan.FromHours(24), 5, logger, TestContext.Current.CancellationToken); + + Assert.False(plan.UseTable); + Assert.True(plan.DecisionFailed); + Assert.Equal(1, logger.CountAtLevel(LogLevel.Warning)); + } + + [Fact] + public async Task TheMcpTopQueriesRead_OverAnUnreachableStore_NotesGateFailed_AndLogsThroughTheScope() + { + var logger = new CapturingTestLogger(); + await using var postgres = NpgsqlDataSource.Create(RefusedConnectionString); + var end = new DateTime(2026, 9, 15, 0, 0, 0, DateTimeKind.Utc); + + using var scope = ReadScope.Open(logger); + await Assert.ThrowsAnyAsync(() => DarlingDataReader.GetQueryStoreTopWithReachAsync( + postgres, 1, end.AddHours(-25), end, 10, null, null, null, TestContext.Current.CancellationToken)); + + Assert.Equal(ReadFallback.GateFailed, scope.Fallback); + Assert.True(logger.CountAtLevel(LogLevel.Warning) >= 1, logger.Joined); + } + + [Fact] + public async Task TheComposeWideGate_OverAnUnreachableStore_NotesGateFailed() + { + await using var postgres = NpgsqlDataSource.Create(RefusedConnectionString); + var end = new DateTime(2026, 9, 15, 0, 0, 0, DateTimeKind.Utc); + + using var scope = ReadScope.Open(new CapturingTestLogger()); + var resolution = await DarlingWebEndpoints.ResolveQueryStoreWideEligibleAsync( + postgres, null, end.AddDays(-3), end, null, TestContext.Current.CancellationToken); + + Assert.False(resolution.Eligible); + Assert.Equal(ReadFallback.GateFailed, scope.Fallback); + } + + [Fact] + public async Task TheHourlyEdgesGuard_OnAConnectionThatCannotRun_NotesGateFailed_AndRethrowsTheRollbackFault() + { + await using var connection = new NpgsqlConnection(RefusedConnectionString); + var hour = new DateTime(2026, 9, 15, 3, 0, 0, DateTimeKind.Utc); + var candidate = new ComposeHourlyEdgesCandidate("query_store_stats", "query_store_stats_hourly", hour, hour.AddHours(1)); + + using var scope = ReadScope.Open(new CapturingTestLogger()); + await Assert.ThrowsAnyAsync(() => DarlingWebEndpoints.ResolveHourlyEdgesVerdictAsync( + connection, candidate, null, 30, TestContext.Current.CancellationToken)); + + Assert.Equal(ReadFallback.GateFailed, scope.Fallback); + } + + [Fact] + public async Task AGuardFaultPlusARollbackFault_LogsTwoWarnings_AndTheSnapshotStartAddsNoThird() + { + var logger = new CapturingTestLogger(); + await using var connection = new NpgsqlConnection(RefusedConnectionString); + var hour = new DateTime(2026, 9, 15, 3, 0, 0, DateTimeKind.Utc); + var candidate = new ComposeHourlyEdgesCandidate("query_store_stats", "query_store_stats_hourly", hour, hour.AddHours(1)); + + using var scope = ReadScope.Open(logger); + var fault = await Assert.ThrowsAnyAsync(() => DarlingWebEndpoints.ResolveHourlyEdgesVerdictAsync( + connection, candidate, null, 30, TestContext.Current.CancellationToken)); + Assert.Equal(2, logger.CountAtLevel(LogLevel.Warning)); + + DarlingWebEndpoints.NoteHourlyEdgesSnapshotStartFault(fault); + + Assert.Equal(2, logger.CountAtLevel(LogLevel.Warning)); + Assert.Equal(ReadFallback.GateFailed, scope.Fallback); + } +} diff --git a/Darling/Darling.Tests/ReadmeDerivedCountPinTests.cs b/Darling/Darling.Tests/ReadmeDerivedCountPinTests.cs index f355f6ab5c..2a58f7904a 100644 --- a/Darling/Darling.Tests/ReadmeDerivedCountPinTests.cs +++ b/Darling/Darling.Tests/ReadmeDerivedCountPinTests.cs @@ -43,6 +43,7 @@ namespace Darling.Tests; /// bolded number, a line rewrapped — is otherwise indistinguishable from a clean tree by its result alone, /// which is the failure mode this whole family of source-parsing tests exists to catch. /// +[Trait("Stage", "Guard")] public sealed class ReadmeDerivedCountPinTests { /* The derived quantities, restated ONCE each. Everything below compares against these rather than against diff --git a/Darling/Darling.Tests/RecurrenceLabelStoreReadTests.cs b/Darling/Darling.Tests/RecurrenceLabelStoreReadTests.cs index 1fb1da2919..1280d6cb56 100644 --- a/Darling/Darling.Tests/RecurrenceLabelStoreReadTests.cs +++ b/Darling/Darling.Tests/RecurrenceLabelStoreReadTests.cs @@ -34,6 +34,7 @@ namespace Darling.Tests; /// expects and the labeler turns them into the sentence. /// [Collection("live-postgres")] +[Trait("Reads", "Lite")] public sealed class RecurrenceLabelStoreReadTests { private const string ServerName = "recurrence-label-e2e"; diff --git a/Darling/Darling.Tests/ReenterLineTests.cs b/Darling/Darling.Tests/ReenterLineTests.cs new file mode 100644 index 0000000000..9bec814db7 --- /dev/null +++ b/Darling/Darling.Tests/ReenterLineTests.cs @@ -0,0 +1,57 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using PerformanceMonitor.Darling.Service; +using Xunit; + +namespace Darling.Tests; + +/// +/// The line the service logs when saved passwords have to be entered again (#5366), and what it says instead while the +/// old-format passwords are only waiting for the one-time pin step (#5456). +/// +public sealed class ReenterLineTests +{ + [Theory] + [InlineData(0)] + [InlineData(-1)] + public void NothingToEnter_SaysNothing(int count) + { + Assert.Null(StoreConfigProvider.ReenterLine(count, waiting: false)); + Assert.Null(StoreConfigProvider.ReenterLine(count, waiting: true)); + } + + [Fact] + public void WhenNotWaiting_TheLineStaysTheEnterAgainText() + { + var one = StoreConfigProvider.ReenterLine(1, waiting: false); + Assert.NotNull(one); + Assert.StartsWith("1 saved password needs to be entered again.", one, StringComparison.Ordinal); + + var many = StoreConfigProvider.ReenterLine(3, waiting: false); + Assert.NotNull(many); + Assert.StartsWith("3 saved passwords need to be entered again.", many, StringComparison.Ordinal); + } + + [Fact] + public void WhileWaiting_SaysWait() + { + var one = StoreConfigProvider.ReenterLine(1, waiting: true); + Assert.NotNull(one); + Assert.Contains("for the one-time pin step. Do not enter it again yet.", one, StringComparison.Ordinal); + Assert.Contains("Do not enter", one, StringComparison.Ordinal); + Assert.DoesNotContain("needs to be entered again", one, StringComparison.Ordinal); + + var many = StoreConfigProvider.ReenterLine(3, waiting: true); + Assert.NotNull(many); + Assert.StartsWith("3 saved old-format passwords wait for the one-time pin step.", many, StringComparison.Ordinal); + Assert.Contains("Do not enter them again yet.", many, StringComparison.Ordinal); + Assert.NotEqual(StoreConfigProvider.ReenterLine(3, waiting: false), many); + } +} diff --git a/Darling/Darling.Tests/RefreshCeilingProvenancePinTests.cs b/Darling/Darling.Tests/RefreshCeilingProvenancePinTests.cs index 16bbbe658b..257ebf5e54 100644 --- a/Darling/Darling.Tests/RefreshCeilingProvenancePinTests.cs +++ b/Darling/Darling.Tests/RefreshCeilingProvenancePinTests.cs @@ -126,6 +126,7 @@ namespace Darling.Tests; /// file not to break the same rule. A rule stated in one paragraph binds nothing in the paragraphs a /// reader meets thousands of lines away, so where it is stated is not where it needs enforcing. /// +[Trait("Stage", "Guard")] public sealed class RefreshCeilingProvenancePinTests { /// @@ -561,11 +562,13 @@ four captured numbers are load-bearing and the drift sweep can reach every one o the grid, reducing LightHourlyRefreshCount to 12. The 3 successors that remain unmeasured are still argued by shape, but covered == total (12 == 12) so the scope sentence was deleted from the doc. The pin retires with the sentence: covered == total was the exit condition the comment named. */ - /* RETIRED: yield return ( + /* RE-ENABLED at #5329: the two io hourlies were registered after the census read, so the census covers + 12 of the 14 light views and the scope sentence is back, with this pin. */ + yield return ( "the light views the census covers and the ones registered after it", LightCeilingDeclaration, @"it covers ([0-9]+) of the ([0-9]+) light views the constant now bounds; the ([0-9]+) registered after the read are unmeasured", - true); */ + true); /* The exclusion CONTROL, which is what says the succeeded/finish filter is not selecting the population: both figures are the same number, and that number is the census count. A filter that @@ -1924,10 +1927,21 @@ population to widen rather than as three sentences to reconcile. */ + "the maximum, so one of the two is wrong"); /* A6: census now covers the full grid (covered == LightHourlyRefreshCount), so the view count in the census sentence is directly pinned here instead of via the scope sentence. */ - Require(lightCensus[3] == TimescaleSupport.LightHourlyRefreshCount, - $"the census sentence says it covers {lightCensus[3]} views against the " - + $"{TimescaleSupport.LightHourlyRefreshCount} non-heaviest hourly policies the product registers — " - + "the census covers the full grid; if a new view was added after the census re-read it"); + /* #5329: the census was read over twelve light views and the product now registers fourteen. The scope + sentence accounts for the difference: covered is the census's own count, total is the product's, and + unmeasured is the remainder the shape argument in the prose is made for. A census re-read over the + full layout states covered == total and unmeasured == 0, and retires the sentence again. */ + var scope = Read("the light views the census covers and the ones registered after it"); + Require(scope[0] == lightCensus[3], + $"the scope sentence says it covers {scope[0]} light views but the census sentence read {lightCensus[3]}"); + Require(scope[1] == TimescaleSupport.LightHourlyRefreshCount, + $"the scope sentence says the constant bounds {scope[1]} light views against the " + + $"{TimescaleSupport.LightHourlyRefreshCount} non-heaviest hourly policies the product registers"); + Require(scope[2] == TimescaleSupport.LightHourlyRefreshCount - lightCensus[3], + $"the scope sentence says {scope[2]} views are unmeasured; the product registers " + + $"{TimescaleSupport.LightHourlyRefreshCount} against a census of {lightCensus[3]}"); + Require(lightCensus[3] + scope[2] == TimescaleSupport.LightHourlyRefreshCount, + "covered plus unmeasured does not equal the product's light-policy count"); /* #3653 (Q12): the census was read over twelve light views and the product registers fifteen. The scope sentence has to account for every one of the difference by name-of-count — covered plus unmeasured equals the light-policy count, the covered count is the census's own, and the diff --git a/Darling/Darling.Tests/RefreshCeilingStalenessTests.cs b/Darling/Darling.Tests/RefreshCeilingStalenessTests.cs index 0b9e0f2e5c..7fcbafaee5 100644 --- a/Darling/Darling.Tests/RefreshCeilingStalenessTests.cs +++ b/Darling/Darling.Tests/RefreshCeilingStalenessTests.cs @@ -572,16 +572,18 @@ private static IReadOnlyList StaleDerivationClaims(string source) => /// existence is the concrete demonstration that the recorded ceiling was overtaken from inside the /// routine band (#3182). /// - /// This case was written to EXPIRE, expired at #3653 (Q12), and was RESTORED at #3653 (A6). - /// At Q12 the watch line fell from 1,050 s to 900 s — below this 952 s run — so the run classified - /// and was no longer invisible to the - /// slot watch. The A6 freeze re-derived the heaviest window back to 21 minutes, raising the five-sixths - /// line to 1,050 s again, so the 952 s run is back BELOW the watch line and back in - /// — invisible to the slot watch again. - /// The staleness line still fires on it (the ceiling finding is unchanged), and the band in which a run - /// can overtake the constant while the slot watch stays at Debug is [896, 1,050): 154 seconds wide, - /// the same width as at #3174 (before Q12 narrowed it to 4 s). The mechanism this file guards is still - /// reachable and this run still demonstrates it. + /// This case was written to EXPIRE, expired at #3653 (Q12), was RESTORED at #3653 (A6), and + /// expires AGAIN at #5329. + /// At Q12 the watch line fell from 1,050 s to 900 s, below this 952 s run, so the run classified + /// . The A6 freeze raised the line back to + /// 1,050 s and the run went back to . The two io + /// hourlies (#5329) joined the phase grid as light members, which re-derived the heaviest window from 21 to + /// 19 minutes and the five-sixths line from 1,050 s to 950 s, so the 952 s run is 2 s ABOVE the watch line + /// and visible to the slot watch again, still 188 s inside the 1,140 s wall. The staleness line still fires + /// on it (the ceiling finding is unchanged), and the band in which a run can overtake the constant while the + /// slot watch stays at Debug is [896, 950): 54 seconds wide (154 at A6, 4 at Q12). The mechanism this + /// file guards is still reachable, through a DERIVED reading inside that band, and this run still + /// demonstrates it from the other side: a ceiling-falsifying run that the slot watch also reports. /// [Fact] public void TheMeasuredRunThatDemonstratedTheDefect_StillFalsifiesTheRecordedCeiling() @@ -596,16 +598,16 @@ public void TheMeasuredRunThatDemonstratedTheDefect_StillFalsifiesTheRecordedCei + "still outside it, or drop this case and keep the derived ones — or the reading is being " + "cited against a constant it was not measured against (#3182)"); - /* #3653 A6: the A6 freeze raised the watch line back to 1,050 s, so the run is again BELOW it and - invisible to the slot watch — the InsideSlot case restored. Pinned in this direction with the - margin; a grid that moves the line BELOW 952 s (a narrower window) has to re-read this case. */ + /* #5329: the two io hourlies re-derived the watch line to 950 s, so the 952 s run is 2 s ABOVE it and + the slot watch sees it: the Q12 state, the ApproachingSlot case. Pinned in this direction with the + margin; a grid that moves the line back ABOVE 952 s (a wider window) has to re-read this case. */ Assert.True( - MeasuredCleanRunSeconds < WatchLine, - $"the quoted {MeasuredCleanRunSeconds} s run is now ABOVE the {WatchLine} s watch line — the " - + "run is no longer invisible to the slot watch; re-read this case for the ApproachingSlot band"); - Assert.Equal(98, WatchLine - MeasuredCleanRunSeconds); + MeasuredCleanRunSeconds > WatchLine, + $"the quoted {MeasuredCleanRunSeconds} s run is at or below the {WatchLine} s watch line, so the " + + "run is invisible to the slot watch again; re-read this case for the InsideSlot band"); + Assert.Equal(2, MeasuredCleanRunSeconds - WatchLine); Assert.Equal( - TimescaleSupport.RefreshSlotHeadroom.InsideSlot, + TimescaleSupport.RefreshSlotHeadroom.ApproachingSlot, TimescaleSupport.ClassifyRefreshSlotHeadroom(MeasuredCleanRunSeconds)); Assert.True( MeasuredCleanRunSeconds < TimescaleSupport.RefreshPhaseSlotSeconds, @@ -626,13 +628,13 @@ public void TheMeasuredRunThatDemonstratedTheDefect_StillFalsifiesTheRecordedCei Assert.StartsWith("Warning:", log.Joined, StringComparison.Ordinal); /* THE BAND THAT IS LEFT for #3182's mechanism — overtaking the constant while the slot watch logs at - Debug — is the gap between the two constants: 1,050 - 896 = 154 s at A6's grid (was 4 s at Q12). + Debug — is the gap between the two constants: 950 - 896 = 54 s at #5329's grid (154 s at A6, 4 s at Q12). A DERIVED reading one second above the ceiling still sits inside it, still classifies InsideSlot, and still falsifies the constant, so the mechanism this file guards is reachable and not vacuous; a grid that closed the band entirely (watch line at or below the ceiling) is red in TimescaleSupportTests, not here. */ var invisibleBandSeconds = WatchLine - Ceiling; - Assert.Equal(154, invisibleBandSeconds); + Assert.Equal(54, invisibleBandSeconds); Assert.True(invisibleBandSeconds > 0, "the watch line is at or below the recorded ceiling (#3653)"); var insideTheBand = Ceiling + 1d; diff --git a/Darling/Darling.Tests/RegisteredServerSettingDriftTests.cs b/Darling/Darling.Tests/RegisteredServerSettingDriftTests.cs index 887f1670b4..a06a3df81b 100644 --- a/Darling/Darling.Tests/RegisteredServerSettingDriftTests.cs +++ b/Darling/Darling.Tests/RegisteredServerSettingDriftTests.cs @@ -38,6 +38,7 @@ namespace Darling.Tests; /// DPAPI blob, so comparing them would report a WORKING configuration as drift on every start — quite apart /// from putting a secret one string interpolation away from a log line. /// +[Trait("Stage", "Guard")] public sealed class RegisteredServerSettingDriftTests { /// A darling.json entry: no stored id, so its ServerId is derived from its own address. @@ -66,6 +67,7 @@ private static MonitoredServer StoreRow(MonitoredServer like) AlertDeliveryModeOverride = like.AlertDeliveryModeOverride, Engine = like.Engine, Port = like.Port, + AwsRoleArn = like.AwsRoleArn, }; copy.StoredServerId = ServerIdHelper.GetDeterministicHashCode(copy.StorageName); @@ -196,6 +198,38 @@ public void AnUnsetDatabase_IsNotDriftAgainstTheEnginesImplicitDefault(string en Assert.Equal(explicitName, drift.StoreValue); } + /// + /// The AWS role a darling.json entry names is compared against the registered row's (#5452): a file role that differs + /// from the stored one shows as drift, a blank and an unset role agree, and the external ID beside it is never + /// compared or printed. + /// + [Fact] + public void TheAwsRole_IsCompared_AndTheExternalIdIsNot() + { + var file = FileEntry("pgtarget", "pgtarget-host"); + file.Engine = "postgres"; + file.Auth = "sql"; + file.Username = "darling_monitor"; + var store = StoreRow(file); + + Assert.Empty(Compare(file, store)); + + file.AwsRoleArn = " "; + Assert.Empty(Compare(file, store)); + + file.AwsRoleArn = "arn:aws:iam::123456789012:role/darling-monitor"; + file.AwsExternalId = "file-external-id"; + store.AwsExternalId = "stored-external-id"; + var drift = Assert.Single(Compare(file, store)); + Assert.Equal("awsRoleArn", drift.Field); + Assert.Equal("arn:aws:iam::123456789012:role/darling-monitor", drift.FileValue); + Assert.Equal("(none)", drift.StoreValue); + Assert.False(drift.AffectsConnection); + + store.AwsRoleArn = file.AwsRoleArn; + Assert.Empty(Compare(file, store)); + } + /// /// Port 0 means "the driver's default", which for Npgsql is 5432 — one value, not two. The message prints /// the effective port for the same reason: it is what was compared. @@ -885,6 +919,10 @@ collection on a server because someone rotated a credential collection never use { "password", "encryptedPassword", "remediationUsername", "remediationEncryptedPassword", + /* #5452: the external ID sent with a server's AWS role. Excluded for the credential's reasons: it is a + stored secret, and a stored secret is never read back for a compare (the comparison read does not + select it), so it cannot reach a log line. The role beside it, awsRoleArn, is compared. */ + "awsExternalId", }; var keys = typeof(MonitoredServer) @@ -916,6 +954,7 @@ collection on a server because someone rotated a credential collection never use AlertDeliveryModeOverride = null, Engine = engine, Port = 5432, + AwsRoleArn = "arn:aws:iam::123456789012:role/darling-store", }; store.StoredServerId = ServerIdHelper.GetDeterministicHashCode(store.StorageName); @@ -935,6 +974,8 @@ collection on a server because someone rotated a credential collection never use AlertDeliveryModeOverride = AlertNotificationMode.PerEvent, Engine = engine == "postgres" ? "sqlserver" : "postgres", Port = 5433, + AwsRoleArn = "arn:aws:iam::123456789012:role/darling-file", + AwsExternalId = "file-external-id", }; foreach (var d in StoreConfigProvider.CompareServerSettings(file, store)) @@ -960,9 +1001,10 @@ a field label that is not a real key sends the operator to edit something that i /* And that the exclusion list has not quietly grown past the credential. The literal IS the mechanism: an exclusion is only legitimate with an argument, and the argument belongs in the comment above beside the key, so growing this list has to show up in a diff. V113 (#2138 phase 1) - added the remediation credential's two keys, which is why it is four rather than two. */ + added the remediation credential's two keys, and V169 (#5452) the AWS external ID, which is why it is + five rather than two. */ Assert.Equal( - new[] { "encryptedPassword", "password", "remediationEncryptedPassword", "remediationUsername" }, + new[] { "awsExternalId", "encryptedPassword", "password", "remediationEncryptedPassword", "remediationUsername" }, excluded.OrderBy(k => k).ToArray()); /* The property behind the literal, so this is not purely a frozen list that the next lane bumps @@ -973,10 +1015,47 @@ a field label that is not a real key sends the operator to edit something that i { Assert.Matches( new System.Text.RegularExpressions.Regex( - "(?:password|username)$", + "(?:password|username|externalid)$", System.Text.RegularExpressions.RegexOptions.IgnoreCase | System.Text.RegularExpressions.RegexOptions.CultureInvariant), key); } } + + /// + /// A drifted connection setting on an entry that carries a password or a reference ends the file's password being + /// used (the backfill needs every connection setting to agree), and the warning says so: the server is named and the + /// sentence is the clause. An entry with no password, or a drift that is not a connection setting, adds nothing. + /// + [Fact] + public void ADriftedConnectionSettingOnAnEntryWithAPassword_SaysTheFilesPasswordIsNotUsedUntilTheyAgree() + { + var withPassword = FileEntry("pgtarget", "pgtarget.example.internal"); + withPassword.Auth = "sql"; + withPassword.Username = "darling_monitor"; + withPassword.EncryptedPassword = "env:PGTARGET_PASSWORD"; + var withoutPassword = FileEntry("other", "other.example.internal"); + var costOnly = FileEntry("costly", "costly.example.internal"); + costOnly.Auth = "sql"; + costOnly.Username = "darling_monitor"; + costOnly.Password = "typed-in-the-file"; + var stores = new[] { StoreRow(withPassword), StoreRow(withoutPassword), StoreRow(costOnly) } + .Select(r => Registered(r)).ToList(); + + withPassword.EncryptMode = "Strict"; + withoutPassword.EncryptMode = "Strict"; + costOnly.MonthlyCostUsd = 99m; + + var drifted = StoreConfigProvider.DescribeSettingDrift(new[] { withPassword, withoutPassword, costOnly }, stores); + Assert.Equal(3, drifted.Count); + + var note = StoreConfigProvider.DescribePasswordNotUsed(drifted, 10); + Assert.Contains("its password from darling.json is not used until the two agree", note, System.StringComparison.Ordinal); + Assert.Contains("pgtarget", note, System.StringComparison.Ordinal); + Assert.DoesNotContain("other", note, System.StringComparison.Ordinal); + Assert.DoesNotContain("costly", note, System.StringComparison.Ordinal); + Assert.DoesNotContain("typed-in-the-file", note, System.StringComparison.Ordinal); + Assert.DoesNotContain("PGTARGET_PASSWORD", note, System.StringComparison.Ordinal); + Assert.Equal("", StoreConfigProvider.DescribePasswordNotUsed(drifted.Where(d => d.Server != "pgtarget").ToList(), 10)); + } } diff --git a/Darling/Darling.Tests/RegressedFromProductiveTests.cs b/Darling/Darling.Tests/RegressedFromProductiveTests.cs index 31eb3dab18..7af4c8b469 100644 --- a/Darling/Darling.Tests/RegressedFromProductiveTests.cs +++ b/Darling/Darling.Tests/RegressedFromProductiveTests.cs @@ -244,7 +244,7 @@ let an ordinary bad window manufacture a regression. */ [InlineData("DarlingDataReader.CollectionHealthSql")] [InlineData("DarlingFleetReader.FleetCollectionHealthSql")] [InlineData("ViewerDataService.CollectionHealthSql")] - [InlineData("ViewerDataService.FleetCollectionHealthSql")] + [InlineData("ViewerDataService.FleetCollectionHealthByServerSql")] public void EveryBandingRead_SelectsBothRegressionInstants(string which) { var sql = which switch @@ -252,7 +252,7 @@ public void EveryBandingRead_SelectsBothRegressionInstants(string which) "DarlingDataReader.CollectionHealthSql" => DarlingDataReader.CollectionHealthSql, "DarlingFleetReader.FleetCollectionHealthSql" => DarlingFleetReader.FleetCollectionHealthSql, "ViewerDataService.CollectionHealthSql" => ViewerDataService.CollectionHealthSql, - "ViewerDataService.FleetCollectionHealthSql" => ViewerDataService.FleetCollectionHealthSql, + "ViewerDataService.FleetCollectionHealthByServerSql" => ViewerDataService.FleetCollectionHealthByServerSql, _ => throw new ArgumentOutOfRangeException(nameof(which), which, "unmapped banding read"), }; diff --git a/Darling/Darling.Tests/ReleaseStatementGateWorkflowTests.cs b/Darling/Darling.Tests/ReleaseStatementGateWorkflowTests.cs new file mode 100644 index 0000000000..d5f8733ff4 --- /dev/null +++ b/Darling/Darling.Tests/ReleaseStatementGateWorkflowTests.cs @@ -0,0 +1,179 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.RegularExpressions; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// The statement-filter release gate runs inside the release job, before the job packages anything (#5320, N4). +/// +/// The defect this closes was a gate nobody ran. StatementColumnCensusTests.StatementCensus_PendingEntries_BlockARelease +/// once failed while a statement-column census entry was pending only when DARLING_RELEASE_CUT=1, and the only +/// place that said to set it was a checklist step. A release that skipped the step shipped with a column still +/// unhooked and every check green. The first ruling (Erik, 2026-10-06) is that the release job runs the gate itself, and +/// that no other workflow sets the variable. Nothing is pending now, so the test fails on ANY pending entry whether or not +/// the variable is set (#5320): a pull request that adds an unjudged column fails in its own Darling test run, and the +/// release step is the backstop, not the only place the gate bites. +/// +public sealed class ReleaseStatementGateWorkflowTests +{ + private const string ReleaseWorkflow = ".github/workflows/build.yml"; + private const string NightlyWorkflow = ".github/workflows/nightly.yml"; + // Bound to the test with nameof (via StatementColumnCensusTests.ReleaseGateTestName), so renaming the test fails the + // workflow pin below rather than leaving the gate filtering on a name that matches nothing (review of #5367, B-M1). + private const string GateTest = StatementColumnCensusTests.ReleaseGateTestName; + + // The one summary line the runner prints for exactly one passing test. The gate step must demand this whole line. + private const string ExactGateSummary = "Total: 1, Errors: 0, Failed: 0, Skipped: 0, Not Run: 0"; + + private sealed record Step(string Job, string Name, string Text); + + /// Every step of every job in a workflow, comment lines dropped so a rationale that quotes a command is not judged as one. + private static List Steps(string workflow) + { + var steps = new List(); + var job = string.Empty; + string? name = null; + var text = new List(); + var inJobs = false; + + void Flush() + { + if (name is not null) + { + steps.Add(new Step(job, name, string.Join("\n", text))); + } + + name = null; + text.Clear(); + } + + foreach (var raw in ReadRepoFileLf(workflow).Split('\n')) + { + if (raw.TrimStart().StartsWith('#')) + { + continue; + } + + if (raw.StartsWith("jobs:", StringComparison.Ordinal)) + { + inJobs = true; + continue; + } + + var jobHeader = Regex.Match(raw, @"^ ([A-Za-z0-9_-]+):\s*$"); + if (inJobs && jobHeader.Success) + { + Flush(); + job = jobHeader.Groups[1].Value; + continue; + } + + var stepHeader = Regex.Match(raw, @"^ - name:\s*(.*)$"); + if (stepHeader.Success) + { + Flush(); + name = stepHeader.Groups[1].Value.Trim(); + } + + if (name is not null) + { + text.Add(raw); + } + } + + Flush(); + + // Non-vacuity: a scan that read nothing would satisfy the "no other workflow sets it" half by having nothing to judge. + Assert.True(steps.Count >= 10, $"only {steps.Count} steps found in {workflow}; the scan is not reading the file"); + return steps; + } + + private static bool IsPackaging(Step s) => + Regex.IsMatch(s.Text, @"dotnet publish |Compress-Archive|\bvpk\b"); + + private static bool SetsTheGate(Step s) => + Regex.IsMatch(s.Text, @"DARLING_RELEASE_CUT\s*:\s*['""]?1['""]?\s*$", RegexOptions.Multiline); + + [Fact] + public void TheReleaseJob_RunsTheStatementGateWithTheReleaseCutOn_BeforeItsFirstPackagingStep() + { + var steps = Steps(ReleaseWorkflow); + var gate = Assert.Single(steps.Where(SetsTheGate).ToList()); + + Assert.Contains("github.event_name == 'release'", gate.Text, StringComparison.Ordinal); + Assert.Contains(GateTest, gate.Text, StringComparison.Ordinal); + Assert.DoesNotContain("continue-on-error", gate.Text, StringComparison.Ordinal); + + // The gate runs the one test: with no filter the runner executes the whole suite a second time. + Assert.Matches(@"-method\s+""?\*?" + GateTest, gate.Text); + + // A -method filter that matches nothing exits 0 with "Total: 0", so a zero-test run would pass. The step must read the + // runner's summary and fail unless it shows exactly the one passing test, and must also fail on the runner's own exit code. + Assert.Contains(ExactGateSummary, gate.Text, StringComparison.Ordinal); + Assert.Matches(@"-not\s*\(\s*Select-String[^\r\n]*" + Regex.Escape(ExactGateSummary) + @"[^\r\n]*\)\s*\)\s*\{", gate.Text); + Assert.Matches(@"\$LASTEXITCODE\s+-ne\s+0", gate.Text); + Assert.Matches(@"Tee-Object\s+-FilePath\s+(\S+)[\s\S]*Select-String\s+-Path\s+\1\s", gate.Text); + + var jobSteps = steps.Where(s => s.Job == gate.Job).ToList(); + var firstPackaging = jobSteps.FindIndex(s => s.Name != gate.Name && IsPackaging(s)); + Assert.True(firstPackaging >= 0, "the gate's job has no packaging step: the scan is not reading it"); + + var gateIndex = jobSteps.FindIndex(s => s.Name == gate.Name); + Assert.True( + gateIndex < firstPackaging, + $"the statement gate step '{gate.Name}' (step {gateIndex}) must run before '{jobSteps[firstPackaging].Name}' (step {firstPackaging}), " + + "the job's first packaging step, so a release cannot publish while a statement column is pending (#5320)"); + } + + [Fact] + public void TheGateTest_FailsOnAnyPendingEntry_WhetherOrNotTheReleaseCutIsSet() + { + // #5320: the ratchet is part of every Darling test run, so the test must not branch on the variable. A copy of the + // old "only when DARLING_RELEASE_CUT=1" shape would let a pull request add an unjudged column and pass. + var source = ReadRepoFileLf("Darling/Darling.Tests/StatementColumnCensusTests.cs"); + var start = source.IndexOf("public void " + GateTest + "()", StringComparison.Ordinal); + Assert.True(start >= 0, "the gate test method was not found in StatementColumnCensusTests.cs"); + + var end = source.IndexOf("\n }\n", start, StringComparison.Ordinal); + Assert.True(end > start, "the end of the gate test method was not found"); + + var body = source[start..end]; + Assert.DoesNotContain("GetEnvironmentVariable", body, StringComparison.Ordinal); + Assert.DoesNotContain("DARLING_RELEASE_CUT", body, StringComparison.Ordinal); + Assert.Contains("Assert.True(", body, StringComparison.Ordinal); + Assert.Contains("pending.Length == 0", body, StringComparison.Ordinal); + } + + [Theory] + [InlineData(NightlyWorkflow)] + [InlineData(ReleaseWorkflow)] + public void OnlyTheReleaseStep_SetsTheReleaseCut(string workflow) + { + // The test no longer reads the variable, so setting it elsewhere is redundant; the release-only step stays the one place that + // names it, so the release backstop is a single pinned step rather than a variable scattered over workflows. + var offending = Steps(workflow) + .Where(s => s.Text.Contains("DARLING_RELEASE_CUT", StringComparison.Ordinal)) + .Where(s => !(workflow == ReleaseWorkflow + && SetsTheGate(s) + && s.Text.Contains("github.event_name == 'release'", StringComparison.Ordinal))) + .Select(s => s.Job + " / " + s.Name) + .ToList(); + + Assert.True( + offending.Count == 0, + "DARLING_RELEASE_CUT=1 fails the build while any statement column is pending; only the release-only gate step may set it (#5320):\n " + + string.Join("\n ", offending)); + } +} diff --git a/Darling/Darling.Tests/ReleaseWalkWordingTests.cs b/Darling/Darling.Tests/ReleaseWalkWordingTests.cs new file mode 100644 index 0000000000..c5b42738cf --- /dev/null +++ b/Darling/Darling.Tests/ReleaseWalkWordingTests.cs @@ -0,0 +1,191 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.ComponentModel; +using System.Diagnostics; +using System.Linq; +using System.Text.Json; +using PerformanceMonitor.PlanAnalysis; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// The release walk's web wording and count fixes, run against the shipped page modules under Node +/// (web-release-walk-harness.mjs; skipped when Node is not installed): W1 the offline fleet card's age, W5 the +/// Availability Group counts, W7 and W11 the words for a precondition and an answered-only answer, and W8 the Repro +/// script's time stamp. +/// +public sealed class ReleaseWalkWordingTests +{ + private static string Scenario(string scenario, string[]? inputs = null) + { + var psi = new ProcessStartInfo("node") { RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false }; + psi.ArgumentList.Add(PathTo("Darling", "Darling.Tests", "web-release-walk-harness.mjs")); + psi.ArgumentList.Add(PathTo("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js")); + psi.ArgumentList.Add(scenario); + if (inputs is not null) + { + psi.Environment["HARNESS_INPUT"] = JsonSerializer.Serialize(inputs); + } + + Process proc; + try + { + proc = Process.Start(psi)!; + } + catch (Win32Exception) + { + Assert.Skip("Node is not installed, so the shipped page code cannot be run."); + return ""; + } + + using (proc) + { + var error = proc.StandardError.ReadToEndAsync(); + var output = proc.StandardOutput.ReadToEnd().Trim(); + if (!proc.WaitForExit(30000)) + { + proc.Kill(entireProcessTree: true); + Assert.Fail("the release-walk harness did not finish in 30 s"); + } + + Assert.True(proc.ExitCode == 0, "the release-walk harness failed: " + error.Result); + return output; + } + } + + private static string[] Strings(JsonElement e) => e.EnumerateArray().Select(x => x.GetString()!).ToArray(); + + [Fact] + public void ADarkPastTheFleetWindowCard_SaysItLastCollectedMoreThanTwoDaysAgo() + { + using var doc = JsonDocument.Parse(Scenario("fleetDark")); + var texts = Strings(doc.RootElement.GetProperty("dark")); + Assert.Contains("Offline · last collected more than 2 days ago", texts); + Assert.Contains("no recent collection · last collected more than 2 days ago", texts); + Assert.Contains(texts, t => t == "last collected more than 2 days ago"); + Assert.DoesNotContain(texts, t => t == "no recent collection" || t.EndsWith("· no recent collection", StringComparison.Ordinal)); + } + + [Fact] + public void ADarkCardThePageGetsALastCollectionFor_ShowsTheRealAge_NotTheTwoDayBound() + { + using var doc = JsonDocument.Parse(Scenario("fleetDarkRealAge")); + var texts = Strings(doc.RootElement.GetProperty("dark")); + Assert.Contains("Offline · last collect 12d ago", texts); + Assert.Contains("last collected 12d ago", texts); + Assert.DoesNotContain(texts, t => t.Contains("more than 2 days", StringComparison.Ordinal)); + } + + [Fact] + public void AProblemListThatCouldNotBeRead_StaysOnThePageAndShowsTheError_NotHiddenAsIfThereWereNoProblems() + { + using var doc = JsonDocument.Parse(Scenario("serverTabReadFails")); + Assert.True(doc.RootElement.GetProperty("found").GetBoolean(), "the Analysis could not read these data families panel was not found"); + Assert.NotEqual("none", doc.RootElement.GetProperty("display").GetString()); + Assert.Contains("The store took too long to answer this read", doc.RootElement.GetProperty("text").GetString(), StringComparison.Ordinal); + } + + [Fact] + public void AnOfflineCardWithALastCollection_KeepsItsExactAge() + { + using var doc = JsonDocument.Parse(Scenario("fleetInWindow")); + var texts = Strings(doc.RootElement.GetProperty("dark")); + Assert.Contains("Offline · last collect 5h ago", texts); + Assert.Contains("last collected 5h ago", texts); + Assert.DoesNotContain(texts, t => t.Contains("more than 2 days", StringComparison.Ordinal)); + } + + [Fact] + public void OneGroupSeenByTwoServers_ReadsOneGroup_AndSaysWhatTheCardsAre() + { + using var doc = JsonDocument.Parse(Scenario("ag")); + Assert.Equal(new[] { "Group", "Reporting servers", "Views" }, Strings(doc.RootElement.GetProperty("labels"))); + Assert.Contains(Strings(doc.RootElement.GetProperty("notes")), n => n.StartsWith("Each card below is one reporting server's view of a group", StringComparison.Ordinal)); + } + + [Fact] + public void OneViewPerGroup_KeepsThePluralAndNeedsNoCardNote() + { + using var doc = JsonDocument.Parse(Scenario("agOnePerGroup")); + Assert.Equal(new[] { "Groups", "Reporting servers", "Views" }, Strings(doc.RootElement.GetProperty("labels"))); + Assert.DoesNotContain(Strings(doc.RootElement.GetProperty("notes")), n => n.StartsWith("Each card below", StringComparison.Ordinal)); + } + + private const string KcachePrecondition = + "The pg_kernel_stats collector is running against example-pg-01 but the PostgreSQL extension it reads is not installed there: " + + "its last run was recorded as a named non-fatal skip, so the per-query OS CPU is not being stored. This is a runtime PRECONDITION, " + + "not a permanent engine capability gap and not a collection outage: it can be satisfied on the monitored server."; + + private const string EngineGate = + "example-pg-01 runs PostgreSQL, so the sql_server_only collector does not run on it. This is a permanent engine capability gap, " + + "not a collection outage: checking collection health, enabling a collector or starting a capture cannot change it."; + + [Fact] + public void APreconditionSentence_IsNotTurnedIntoTheDoesNotApplyLine() + { + using var doc = JsonDocument.Parse(Scenario("plain", new[] { KcachePrecondition, EngineGate })); + var output = Strings(doc.RootElement.GetProperty("out")); + var line = doc.RootElement.GetProperty("line").GetString()!; + Assert.NotEqual(line, output[0]); + Assert.Contains("extension it reads is not installed there", output[0], StringComparison.Ordinal); + Assert.Equal(line, output[1]); + } + + [Fact] + public void TheAnsweredRankingsEmptyAnswer_DropsTheParameterName() + { + var message = + "answered_only excluded every row: example-pg-01 has 3 candidate btree index(es) in this window and NOT ONE of them has a trusted bloat answer. " + + "This is not a clean bill of health and it is not an empty server - the suppression breakdown below says why each index has no answer and which reasons are remediable. " + + "Re-run without answered_only to see the rows and their reasons. Coverage sentence."; + using var doc = JsonDocument.Parse(Scenario("plain", new[] { message })); + var o = Strings(doc.RootElement.GetProperty("out"))[0]; + Assert.DoesNotContain("answered_only", o, StringComparison.Ordinal); + Assert.StartsWith("Nothing to rank: example-pg-01 has 3 candidate btree index(es)", o, StringComparison.Ordinal); + Assert.Contains("the Index Bloat table above says why each index has no answer.", o, StringComparison.Ordinal); + Assert.DoesNotContain("breakdown below", o, StringComparison.Ordinal); + Assert.EndsWith("Coverage sentence.", o, StringComparison.Ordinal); + } + + [Fact] + public void TheKcachePanel_NamesTheMissingExtensionInPlainWords() + { + var tabs = System.IO.File.ReadAllText(PathTo("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "pages", "server-tabs.js")); + var panels = System.IO.File.ReadAllText(PathTo("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "panels.js")); + Assert.Contains("pg_stat_kcache is not installed on this server", tabs, StringComparison.Ordinal); + Assert.Contains("res.status === \"precondition\" && desc.extensionMissingLine", panels, StringComparison.Ordinal); + } + + [Fact] + public void ThePostgreSqlEmptyAnswers_NoLongerSayTheStoreCannotClassifyTheServer() + { + foreach (var file in new[] { "DarlingMcpPgWraparoundTools.cs", "DarlingMcpPgStatementTools.cs", "DarlingMcpPgWaitTools.cs" }) + { + var text = System.IO.File.ReadAllText(PathTo("Darling", "PerformanceMonitor.Darling.Service", "Mcp", file)); + Assert.DoesNotContain("cannot classify", text, StringComparison.Ordinal); + // the sentence is spread over concatenated literals; join them before looking for it + var joined = System.Text.RegularExpressions.Regex.Replace(text, "\"\\s*\\+\\s*\"", ""); + Assert.Contains("offline, nothing is being collected", joined, StringComparison.Ordinal); + } + } + + [Fact] + public void TheReproScriptsGeneratedLine_IsUtcAndSaysSo() + { + var before = DateTime.UtcNow.AddSeconds(-2); + var script = ReproScriptBuilder.BuildReproScript("SELECT 1", "master", planXml: null, isolationLevel: null); + var line = script.Split('\n').Select(l => l.TrimEnd('\r')).First(l => l.StartsWith("Generated: ", StringComparison.Ordinal)); + Assert.EndsWith(" UTC", line, StringComparison.Ordinal); + var stamp = DateTime.ParseExact(line["Generated: ".Length..^" UTC".Length], "yyyy-MM-dd HH:mm:ss", System.Globalization.CultureInfo.InvariantCulture); + Assert.InRange(stamp, before, DateTime.UtcNow.AddSeconds(2)); + } +} diff --git a/Darling/Darling.Tests/RemoteCollectorServiceCancellationCensusTests.cs b/Darling/Darling.Tests/RemoteCollectorServiceCancellationCensusTests.cs index 3839fab6b7..385d927633 100644 --- a/Darling/Darling.Tests/RemoteCollectorServiceCancellationCensusTests.cs +++ b/Darling/Darling.Tests/RemoteCollectorServiceCancellationCensusTests.cs @@ -20,6 +20,8 @@ namespace Darling.Tests; /// Lite.Tests (which cannot run in-process on macOS; discovery dies on WindowsBase) because /// reading source text needs nothing Windows-only. This runs on macOS and in CI alike. /// +[Trait("Stage", "Guard")] +[Trait("Reads", "Lite")] public sealed class RemoteCollectorServiceCancellationCensusTests { /// diff --git a/Darling/Darling.Tests/RepoFileAdoptionTests.cs b/Darling/Darling.Tests/RepoFileAdoptionTests.cs index aee5d863de..60d9e45818 100644 --- a/Darling/Darling.Tests/RepoFileAdoptionTests.cs +++ b/Darling/Darling.Tests/RepoFileAdoptionTests.cs @@ -71,6 +71,7 @@ namespace Darling.Tests; /// without a reader on top of it. That is the same duplication one layer down, it is a larger population /// than this one was, and it is deliberately not in scope. /// +[Trait("Stage", "Guard")] public sealed class RepoFileAdoptionTests { private const string Authority = "RepoFile.cs"; @@ -127,14 +128,27 @@ not tripped by the file's own prose comment saying "never innerHTML". */ chart counts are told apart from the constant's prose and the function from its callers. */ "ChartIntegerTicksPinTests.cs", "ChartWindowDomainTests.cs", + /* #4605: its wiring pins read the endpoint source with comments stripped and whitespace collapsed, so a call + that wraps across lines compares as one spelling. */ + "ComposeClientTimeoutAttributionTests.cs", /* #4887: its StoredEventCopies census takes each helper's table from that class's own source, on an anchor that runs from the helper's `=>` across the line break to the `Read("v_` on the next line. */ "ConsumedTimestampFrameDisciplineTests.cs", + /* #5288: its host-source pins slice TryStartServerAsync and the release methods out of + DarlingMcpHostService.cs and anchor on the line break and indent that close a block (the + `if (networkMode)` gate), which a CRLF checkout spells differently. */ + "DarlingMcpHostTests.cs", "DarlingPathFilterGateTests.cs", /* #3653 (A8e, PostgreSQL host): its per-arm fire-site pin anchors `Severity: null,` on the line break that follows it, so the CODE spelling is told apart from the same words inside the arms' own comments, which continue on the same line. */ "DarlingPgOperationalAlertTests.cs", + /* #5627: its pins slice the workflow's `on:` and `jobs:` blocks and the nightly's job body by line break and + indent, which a CRLF checkout spells differently. */ + "DarlingUpgradeShapesWorkflowTests.cs", + "FinOpsPageShellTests.cs", + "FinOpsTabLockingPageTests.cs", + "FinOpsTabUtilizationRawPageTests.cs", "FleetCardCollectionStaleNamesItsPopulationTests.cs", /* #3735: its source pin anchors the collection-health statement's tail across three line breaks (`FROM v_collection_log` / `WHERE` / `AND` / `GROUP BY`) and the CreateCommand line on the @@ -145,6 +159,9 @@ chart counts are told apart from the constant's prose and the function from its CreateCommand-plus-CommandTimeout shape, and the statement's own WHERE/AND tail) — the same reasoning as the fleet twin above, which is the file this one was written beside. */ "PerServerCollectionHealthMemoTests.cs", + /* #5366: its Linux-build-job pin slices the `darling-linux:` job out of the workflow starting at a line break + (`\n darling-linux:`), so the text must be LF whatever the checkout's line endings. */ + "PasswordKeySelfCheckTests.cs", "FleetPageAttentionFilterTests.cs", /* Its every-site pins match each `new AlertMuteContext { ... };` initializer across its line breaks in AlertEngine.cs and the Darling producers, and assert each one sets ServerId. */ @@ -157,8 +174,18 @@ chart counts are told apart from the constant's prose and the function from its /* #3653 item 12 (Q6): its cross-SKU pin slices the multi-line `keyed` CTE out of the Darling provider's SOURCE (from `keyed AS (` to `FROM clean` + `)` on the next line) and asserts Lite's source carries the identical text; the clock-read twin pin likewise compares a multi-line const body one token apart. */ + /* #4961: its no-drop pin looks for a statement block that runs across a line break in DarlingXeSessions.cs. */ + "AlwaysOnXeSessionsTests.cs", + /* #5459: it cuts build.yml into jobs, steps and arms on the line breaks between them. */ + "GuardStageWorkflowTests.cs", "LocalClockBucketKeyTests.cs", "LockedModeRestoreCoverageTests.cs", + /* Its always-on guard pin slices EnsureDatabaseScopedAsync's body up to the method's closing brace, on an + anchor that runs from the line break before that brace to the line break after it. */ + "LongQueryTraceLifecycleTests.cs", + /* #4961: its pin slices RunOneAsync's body up to the method's closing brace, on an anchor that starts at the line + break before that brace. */ + "LongQueryTraceRestartTests.cs", /* #3541 A10: its top-level-key discriminator anchors on the line break BEFORE the key (a per-row collection_time inside a Select is indented deeper and must not match), and its tool-body slicing keys on attribute text either side of one. */ @@ -195,10 +222,19 @@ loader call on the NEXT line — anchors that a CRLF read would miss on every ar /* #4427: its wiring pins slice RunPurgeNowBackgroundAsync's body by brace balance and matches calls that span line breaks. */ "RawPurgeNowWiringTests.cs", "RawWindowFloorViewerPortTests.cs", + /* #5452: its worker pin anchors the general catch arm on the line break and indent that follow `catch (Exception ex)`, + so the role arm is proven to come before it. */ + "RdsAssumeRoleSeamTests.cs", /* #3653 item 3: its twin pin extracts the multi-line GetPriorOccurrencesSql const body from BOTH finding stores' sources and asserts them equal up to one token, and its wiring pin slices each pass method by call sites that sit on their own lines — anchors spanning line breaks on every arm. */ "RecurrenceLabelStoreReadTests.cs", + /* #5320: its gate pins split build.yml into steps by line and anchor the gate step's PowerShell check (the + Tee-Object line through the Select-String line that reads the same log) across the lines between them. */ + "ReleaseStatementGateWorkflowTests.cs", + /* #5374: its start pin slices the provisioning chain of DarlingWorker.cs, comments removed, and matches the rules call + inside each branch of it, across line breaks. */ + "SelfManagedStoreRulesStartPinTests.cs", /* Its heading pin slices fillServerHead's body out of server.js, from its declaration to the closing brace on its own line, and asserts the display-name assignment sits inside it and ahead of the dot update. */ "ServerPageTitlePinTests.cs", @@ -212,9 +248,19 @@ call sites that sit on their own lines — anchors spanning line breaks on every and the pre-dispatch `if (…LongQueryTraceFault is { } traceFault) { throw … }` in RunOneAsync - so the CODE shape is told apart from the same words inside the comments beside it. */ "SwallowedItemFailureTests.cs", + "TuningHourlyCreateLiveTests.cs", "ViewTemplatesTests.cs", "ViewerFinOpsPickerRefreshPinTests.cs", "ViewerSidebarDotRendersTheCardStatusTests.cs", + /* #5586: its compose.js pin slices each renderer call up to the `break;` that ends its switch case, and its halo pin + splits app.css on line breaks to take the `.chart .bar-value` rule as one line. */ + "WebChartWidthBehaviourTests.cs", + /* #5245: pins that the filtered reads' catalog rows, dispatch entries and FILTERED list agree, over + DarlingWebEndpoints.cs and the js tree. */ + "WebDatabaseFilterReadsTests.cs", + /* #4961: its RunOneAsync pin takes the body from the method's declaration to the closing brace on its own line + (`\n }\n`), which only matches once the line endings are one spelling. */ + "XeSessionMissingWarningTests.cs", }; /// diff --git a/Darling/Darling.Tests/RepoFileResolutionEquivalenceTests.cs b/Darling/Darling.Tests/RepoFileResolutionEquivalenceTests.cs index 723460b066..ccf2c16004 100644 --- a/Darling/Darling.Tests/RepoFileResolutionEquivalenceTests.cs +++ b/Darling/Darling.Tests/RepoFileResolutionEquivalenceTests.cs @@ -41,6 +41,7 @@ namespace Darling.Tests; /// asserts that, and a pin moved into a /// subdirectory reds there rather than silently resolving from a different starting point. /// +[Trait("Reads", "Lite")] public sealed class RepoFileResolutionEquivalenceTests { /// A call site that really exists, paired with the resolver that used to serve it. diff --git a/Darling/Darling.Tests/ReproScriptLivePostgresTests.cs b/Darling/Darling.Tests/ReproScriptLivePostgresTests.cs new file mode 100644 index 0000000000..289896157d --- /dev/null +++ b/Darling/Darling.Tests/ReproScriptLivePostgresTests.cs @@ -0,0 +1,177 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Globalization; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5233: get_query_repro_script builds each kind's script from STORED text and plan, and nothing else. Seeded through +/// the inline query_plan_text column for Query Store (the plan map read arrives separately), the inline +/// query_text on query_stats and query_store_stats, a collect.query_store_text row for the precedence +/// check, and a query_snapshots row with an isolation level. Fixed anchors only. +/// +[Collection("live-postgres")] +public sealed class ReproScriptLivePostgresTests +{ + private const string ServerName = "darling-repro-script-e2e"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + + private const string Db = "ReproDb"; + private const string StatsHash = "0xA0B1C2D3E4F50617"; + + private static readonly DateTime Anchor = new DateTime(2026, 3, 4, 5, 6, 7, DateTimeKind.Unspecified).AddTicks(1_234_560); + + private static string PlanXml(string marker) => + "" + + $""; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static JsonElement Parse(string json, out JsonDocument doc) + { + doc = JsonDocument.Parse(json); + return doc.RootElement; + } + + [Fact] + public async Task EachKind_BuildsAScriptFromStoredText_AndAWrongKeyIsUnavailable() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live repro-script test."); + var ct = TestContext.Current.CancellationToken; + + await using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + + var bodySucceeded = false; + try + { + await RegisterServerAsync(connection, ct); + + await ExecAsync(connection, ct, + "INSERT INTO query_stats (collection_id, collection_time, server_id, server_name, database_name, query_hash, query_text, query_plan_xml) " + + "VALUES ($1, $2, $3, $4, $5, $6, $7, $8)", + CollectionIdGenerator.Next(), Anchor, ServerId, ServerName, Db, StatsHash, "SELECT 'stats-text';", PlanXml("stats")); + + /* Query Store 4242: collector text AND an older inline text, with a plan. 4243: inline text only, no plan. */ + await ExecAsync(connection, ct, + "INSERT INTO query_store_stats (collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, query_text, query_plan_text) " + + "VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)", + CollectionIdGenerator.Next(), Anchor, ServerId, ServerName, Db, 4242L, 7L, "SELECT 'inline-text';", PlanXml("query-store")); + await ExecAsync(connection, ct, + "INSERT INTO query_store_stats (collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, query_text) " + + "VALUES ($1, $2, $3, $4, $5, $6, $7, $8)", + CollectionIdGenerator.Next(), Anchor, ServerId, ServerName, Db, 4243L, 8L, "SELECT 'no-plan-text';"); + await ExecAsync(connection, ct, + "INSERT INTO collect.query_store_text (server_id, database_name, query_id, query_sql_text, last_seen) VALUES ($1, $2, $3, $4, $5)", + ServerId, Db, 4242L, "SELECT 'collector-text';", Anchor); + + await ExecAsync(connection, ct, + "INSERT INTO query_snapshots (collection_id, collection_time, server_id, server_name, session_id, request_id, database_name, query_text, transaction_isolation_level, query_plan) " + + "VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)", + CollectionIdGenerator.Next(), Anchor, ServerId, ServerName, 51, 3, Db, "SELECT 'snapshot-text';", "SERIALIZABLE", PlanXml("snapshot")); + + /* ---- query_hash */ + var root = Parse(await DarlingMcpPlanTools.GetQueryReproScript(postgres, "query_hash", ServerName, query_hash: StatsHash, cancellationToken: ct), out var d1); + using (d1) + { + Assert.True(root.GetProperty("plan_found").GetBoolean()); + Assert.Contains("SELECT 'stats-text';", root.GetProperty("script").GetString(), StringComparison.Ordinal); + } + + /* ---- query_store: the collector's text wins over the inline text. */ + root = Parse(await DarlingMcpPlanTools.GetQueryReproScript(postgres, "query_store", ServerName, Db, query_id: 4242, plan_id: 7, cancellationToken: ct), out var d2); + using (d2) + { + var script = root.GetProperty("script").GetString()!; + Assert.True(root.GetProperty("plan_found").GetBoolean()); + Assert.Contains("SELECT 'collector-text';", script, StringComparison.Ordinal); + Assert.DoesNotContain("inline-text", script, StringComparison.Ordinal); + } + + /* ---- query_store: text but no plan builds a script with plan_found false. */ + root = Parse(await DarlingMcpPlanTools.GetQueryReproScript(postgres, "query_store", ServerName, Db, query_id: 4243, cancellationToken: ct), out var d3); + using (d3) + { + Assert.False(root.GetProperty("plan_found").GetBoolean()); + Assert.Contains("SELECT 'no-plan-text';", root.GetProperty("script").GetString(), StringComparison.Ordinal); + } + + /* ---- active_snapshot: the collection_time the grid hands back, with the stored isolation level. */ + var key = Anchor.ToString("o", CultureInfo.InvariantCulture); + root = Parse(await DarlingMcpPlanTools.GetQueryReproScript(postgres, "active_snapshot", ServerName, collection_time: key, session_id: 51, request_id: 3, cancellationToken: ct), out var d4); + using (d4) + { + var script = root.GetProperty("script").GetString()!; + Assert.True(root.GetProperty("plan_found").GetBoolean()); + Assert.Contains("SELECT 'snapshot-text';", script, StringComparison.Ordinal); + Assert.Contains("SET TRANSACTION ISOLATION LEVEL", script, StringComparison.Ordinal); + } + + /* ---- wrong keys are unavailable, never an empty script. */ + await AssertUnavailableAsync(await DarlingMcpPlanTools.GetQueryReproScript(postgres, "query_hash", ServerName, query_hash: "0xDEADBEEF", cancellationToken: ct)); + await AssertUnavailableAsync(await DarlingMcpPlanTools.GetQueryReproScript(postgres, "query_store", ServerName, Db, query_id: 9999, cancellationToken: ct)); + await AssertUnavailableAsync(await DarlingMcpPlanTools.GetQueryReproScript(postgres, "active_snapshot", ServerName, collection_time: key, session_id: 999, request_id: 3, cancellationToken: ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + private static Task AssertUnavailableAsync(string json) + { + using var doc = JsonDocument.Parse(json); + Assert.Equal("unavailable", doc.RootElement.GetProperty("status").GetString()); + return Task.CompletedTask; + } + + private static async Task ExecAsync(NpgsqlConnection connection, CancellationToken ct, string sql, params object[] values) + { + await using var command = new NpgsqlCommand(sql, connection); + foreach (var v in values) + command.Parameters.AddWithValue(v is DateTime t ? DateTime.SpecifyKind(t, DateTimeKind.Unspecified) : v); + await command.ExecuteNonQueryAsync(ct); + } + + private static Task RegisterServerAsync(NpgsqlConnection connection, CancellationToken ct) => + ExecAsync(connection, ct, + @"INSERT INTO servers (server_id, server_name, display_name, is_enabled, created_date, modified_date) + VALUES ($1, $2, $3, TRUE, $4, $4) + ON CONFLICT (server_id) DO UPDATE SET server_name = EXCLUDED.server_name, display_name = EXCLUDED.display_name, + is_enabled = TRUE, modified_date = EXCLUDED.modified_date;", + ServerId, ServerName, ServerName, Anchor); + + private static async Task DeleteRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + await using var cleanup = new NpgsqlCommand( + $"DELETE FROM query_snapshots WHERE server_id = {ServerId}; " + + $"DELETE FROM query_stats WHERE server_id = {ServerId}; " + + $"DELETE FROM query_store_stats WHERE server_id = {ServerId}; " + + $"DELETE FROM collect.query_store_text WHERE server_id = {ServerId}; " + + $"DELETE FROM servers WHERE server_id = {ServerId};", connection); + await cleanup.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/RestartRowIndexLiveTests.cs b/Darling/Darling.Tests/RestartRowIndexLiveTests.cs new file mode 100644 index 0000000000..0f398e11fe --- /dev/null +++ b/Darling/Darling.Tests/RestartRowIndexLiveTests.cs @@ -0,0 +1,274 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4605: the restart-row partial indexes on collect.query_stats and collect.procedure_stats +/// (, ) +/// exist after the start-path tuning pass, hold only the restart rows, are read by +/// on an uncompressed chunk, and the pass is idempotent. Own scratch +/// database per test. +/// +/* #1776 own-store: each live fact mints its own scratch database through ScratchPostgres. */ +public sealed class RestartRowIndexLiveTests +{ + private const int RestartEvery = 500; + private const int RowsPerTable = 5000; + + [Fact] + public async Task BothIndexes_ExistWithThePartialPredicate_AndTheSetupIsIdempotent() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the restart-row index live test."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + await TuningStartPasses.ConvergeAsync(connection, ct); + + await AssertIndexAsync(connection, PgTableTuning.QueryStatsRestartRowIndexName, "collect.query_stats", ct); + await AssertIndexAsync(connection, PgTableTuning.ProcedureStatsRestartRowIndexName, "collect.procedure_stats", ct); + + var oidsBefore = await IndexOidsAsync(connection, ct); + var second = await PgTableTuning.ApplyAsync(connection, NullLogger.Instance, ct); + Assert.True(second > 0, "the second pass should run"); + Assert.Equal(oidsBefore, await IndexOidsAsync(connection, ct)); + Assert.Equal(2, oidsBefore.Count); + } + + [Fact] + public async Task EachReader_PlansOnItsPartialIndex_OnAnUncompressedChunk_AndReturnsExactlyTheRestartRows() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the restart-row index live test."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var timescaleEnabled = await LiveTimescaleProbe.TryEnableAsync(scratch.ConnectionString, ct); + Assert.SkipUnless(timescaleEnabled, "TimescaleDB is not available; the chunk-level plan needs a hypertable."); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + await Exec(connection, "SELECT _timescaledb_functions.stop_background_workers()", ct); + + /* A fixed anchor, 23:50 UTC on a fixed past day, not the wall clock. The seed reaches back 5,000 seconds + from this instant, and chunks are one day aligned to UTC midnight, so a seed ending at the real "now" + straddles two daily chunks between 00:00 and about 01:23 UTC. On either chunk a Seq Scan can then + cost less than the partial index, and the no-heap-scan assertion fails. Ending at 23:50 keeps the + whole seed inside one day. */ + var utcNow = new DateTime(2026, 1, 15, 23, 50, 0, DateTimeKind.Unspecified); + var windowStart = utcNow.AddHours(-2); + + await SeedAsync(connection, "collect.query_stats", utcNow, ct); + await SeedAsync(connection, "collect.procedure_stats", utcNow, ct); + + /* The whole seed must sit in one daily chunk: no row may fall before the anchor's date, and the + chunk must hold a realistic number of rows. */ + foreach (var table in new[] { "collect.query_stats", "collect.procedure_stats" }) + { + await using var older = new NpgsqlCommand("SELECT count(*) FROM " + table + " WHERE collection_time < $1", connection); + older.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = utcNow.Date }); + var olderRows = Convert.ToInt64(await older.ExecuteScalarAsync(ct), CultureInfo.InvariantCulture); + Assert.True(olderRows == 0, table + " has " + olderRows + " seeded rows before " + utcNow.Date.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture) + "; the seed must sit inside one daily chunk"); + + await using var newest = new NpgsqlCommand("SELECT count(*) FROM " + table + " WHERE collection_time >= $1", connection); + newest.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = utcNow.Date }); + var newestRows = Convert.ToInt64(await newest.ExecuteScalarAsync(ct), CultureInfo.InvariantCulture); + Assert.True(newestRows >= 1000, table + " holds only " + newestRows + " seeded rows; the seed must be a realistic day"); + } + + await TuningStartPasses.ConvergeAsync(connection, ct); + await Exec(connection, "ANALYZE collect.query_stats", ct); + await Exec(connection, "ANALYZE collect.procedure_stats", ct); + + var expected = RowsPerTable / RestartEvery; + await AssertReaderAsync(connection, IntervalRollupRestartRows.QueryStatsRestartRowsSql, PgTableTuning.QueryStatsRestartRowIndexName, windowStart, utcNow, expected, ct); + await AssertReaderAsync(connection, IntervalRollupRestartRows.ProcedureStatsRestartRowsSql, PgTableTuning.ProcedureStatsRestartRowIndexName, windowStart, utcNow, expected, ct); + + /* The end is exclusive: a window ending at the newest restart row's own time drops it. */ + var newestRestart = await ScalarAsync(connection, "SELECT max(collection_time) FROM collect.query_stats WHERE sample_interval_seconds = 0", ct); + var clipped = await RunAsync(connection, IntervalRollupRestartRows.QueryStatsRestartRowsSql, windowStart, (DateTime)newestRestart!, ct); + Assert.Equal(expected - 1, clipped); + } + + [Fact] + public void TheRestartRowReads_CarryPresenceOnly_NoWorkerTimeColumn() + { + foreach (var sql in new[] { IntervalRollupRestartRows.QueryStatsRestartRowsSql, IntervalRollupRestartRows.ProcedureStatsRestartRowsSql }) + { + Assert.DoesNotContain("delta_worker_time", sql, StringComparison.Ordinal); + Assert.Contains("r.collection_time, r.sample_interval_seconds", sql, StringComparison.Ordinal); + } + } + + private static async Task AssertIndexAsync(NpgsqlConnection connection, string name, string table, CancellationToken ct) + { + await using var def = new NpgsqlCommand( + "SELECT i.indisvalid, pg_get_indexdef(i.indexrelid) FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid " + + "JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'collect' AND c.relname = $1", connection); + def.Parameters.AddWithValue(name); + await using var reader = await def.ExecuteReaderAsync(ct); + Assert.True(await reader.ReadAsync(ct), "the start path should have built " + name); + Assert.True(reader.GetBoolean(0), name + " should be valid"); + var text = reader.GetString(1); + Assert.Contains("ON " + table, text, StringComparison.Ordinal); + Assert.Contains("(collection_time)", text, StringComparison.Ordinal); + Assert.Contains("WHERE (sample_interval_seconds = 0)", text, StringComparison.Ordinal); + } + + private static async Task> IndexOidsAsync(NpgsqlConnection connection, CancellationToken ct) + { + var oids = new List(); + await using var cmd = new NpgsqlCommand( + "SELECT c.oid::oid FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace " + + "WHERE n.nspname = 'collect' AND c.relname IN ($1, $2) ORDER BY c.relname", connection); + cmd.Parameters.AddWithValue(PgTableTuning.QueryStatsRestartRowIndexName); + cmd.Parameters.AddWithValue(PgTableTuning.ProcedureStatsRestartRowIndexName); + await using var reader = await cmd.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + oids.Add(reader.GetFieldValue(0)); + } + + return oids; + } + + /// + /// Mostly normal rows (sample_interval_seconds 60) with every 500th a restart row, all inside the last two + /// hours, so they land in the newest chunk, which stays uncompressed. + /// + private static async Task SeedAsync(NpgsqlConnection connection, string table, DateTime utcNow, CancellationToken ct) + { + var extra = table.EndsWith("procedure_stats", StringComparison.Ordinal) + ? "object_name, schema_name" + : "sql_handle, database_name"; + var extraValues = table.EndsWith("procedure_stats", StringComparison.Ordinal) + ? "'proc_' || (g % 50), 'dbo'" + : "'0x' || md5(g::text), 'db_' || (g % 4)"; + await using var insert = new NpgsqlCommand( + "INSERT INTO " + table + " (collection_id, collection_time, server_id, server_name, " + extra + ", delta_worker_time, sample_interval_seconds) " + + "SELECT 1, $1 - make_interval(secs => (g % 7000)), 1 + g % 3, 'restart-row-index-e2eA', " + extraValues + ", g, " + + "CASE WHEN g % " + RestartEvery.ToString(CultureInfo.InvariantCulture) + " = 0 THEN 0 ELSE 60 END " + + "FROM generate_series(1, " + RowsPerTable.ToString(CultureInfo.InvariantCulture) + ") AS g", connection); + insert.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = utcNow }); + await insert.ExecuteNonQueryAsync(ct); + } + + private static NpgsqlCommand Bind(string sql, NpgsqlConnection connection, DateTime start, DateTime end) + { + var command = new NpgsqlCommand(sql, connection); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = DateTime.SpecifyKind(start, DateTimeKind.Unspecified) }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = DateTime.SpecifyKind(end, DateTimeKind.Unspecified) }); + return command; + } + + private static async Task RunAsync(NpgsqlConnection connection, string sql, DateTime start, DateTime end, CancellationToken ct) + { + await using var command = Bind(sql, connection, start, end); + await using var reader = await command.ExecuteReaderAsync(ct); + var rows = 0; + while (await reader.ReadAsync(ct)) + { + Assert.Equal(0, reader.GetInt32(reader.FieldCount - 1)); + rows++; + } + + return rows; + } + + private static async Task AssertReaderAsync(NpgsqlConnection connection, string sql, string indexName, DateTime start, DateTime end, int expectedRows, CancellationToken ct) + { + Assert.Equal(expectedRows, await RunAsync(connection, sql, start, end, ct)); + + await using var explain = Bind("EXPLAIN (FORMAT JSON) " + sql, connection, start, end); + var json = (string)(await explain.ExecuteScalarAsync(ct))!; + using var doc = JsonDocument.Parse(json); + var reads = new List<(string NodeType, string Index)>(); + var heapScans = new List(); + Collect(doc.RootElement[0].GetProperty("Plan"), reads, heapScans); + + Assert.True(reads.Count > 0, "no index read found in the plan:\n" + json); + Assert.All(reads, r => Assert.Contains(indexName, r.Index, StringComparison.Ordinal)); + Assert.True(heapScans.Count == 0, "the plan scans the heap (" + string.Join(", ", heapScans) + "):\n" + json); + } + + /// + /// Nodes a plan of a restart-row read may contain: index reads, the bitmap heap fetch that follows a bitmap + /// index scan, and the plumbing that joins chunks. Any other node, a Seq Scan, a Tid Scan, a Sample Scan or a + /// Custom Scan that decompresses a chunk, fails the plan. This test seeds one uncompressed chunk, so it says + /// nothing about compressed chunks. + /// + private static readonly HashSet AllowedNodes = new(StringComparer.Ordinal) + { + "Index Scan", "Index Only Scan", "Bitmap Index Scan", "Bitmap Heap Scan", "Append", "Result", "Custom Scan (ChunkAppend)", + }; + + /// + /// Walks the plan. Every node must be on the allow-list; every index node must read the chunk's copy of the + /// partial index (the chunk index name carries the index name). A Bitmap Heap Scan is fine when its child is + /// the Bitmap Index Scan on that index. + /// + private static void Collect(JsonElement node, List<(string, string)> reads, List heapScans) + { + var type = node.GetProperty("Node Type").GetString()!; + if (node.TryGetProperty("Custom Plan Provider", out var provider) && type == "Custom Scan") + { + type += " (" + provider.GetString() + ")"; + } + + if (!AllowedNodes.Contains(type)) + { + heapScans.Add(type + " on " + (node.TryGetProperty("Relation Name", out var r) ? r.GetString() : "?")); + } + else if (type is "Index Scan" or "Index Only Scan" or "Bitmap Index Scan") + { + reads.Add((type, node.GetProperty("Index Name").GetString()!)); + } + + if (node.TryGetProperty("Plans", out var children)) + { + foreach (var child in children.EnumerateArray()) + { + Collect(child, reads, heapScans); + } + } + } + + private static async Task ScalarAsync(NpgsqlConnection c, string sql, CancellationToken ct) + { + await using var cmd = new NpgsqlCommand(sql, c); + return await cmd.ExecuteScalarAsync(ct); + } + + private static async Task Exec(NpgsqlConnection c, string sql, CancellationToken ct) + { + await using var cmd = new NpgsqlCommand(sql, c); + await cmd.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/RetentionCollectionYieldLiveTests.cs b/Darling/Darling.Tests/RetentionCollectionYieldLiveTests.cs new file mode 100644 index 0000000000..11037db819 --- /dev/null +++ b/Darling/Darling.Tests/RetentionCollectionYieldLiveTests.cs @@ -0,0 +1,186 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5592's live test: a real purge, through the product's own sweep, whose wall budget runs out part-way through a +/// table. The pass has to leave the rest of that table's rows, say in its run record that it stopped on its budget, +/// and leave the next pass to take the rows that are left. The clock is derived from the rows deleted so far (20 s +/// per 1,000), so the stop lands on the same batch on every run. Gated on DARLING_TEST_PG; CI sets it, a bare +/// machine skips. +/// +/* #1776 own-store: mints its own scratch database through ScratchPostgres and never touches the shared + store's tables, so it cannot race the live collection. */ +public sealed class RetentionCollectionYieldLiveTests +{ + private const int SeededRows = 5_000; + private const int BatchRows = 1_000; + private const string MapTable = "collect.query_store_plan_map"; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private sealed class NeverBehind : ICollectionPressure + { + public string? BehindReason() => null; + } + + private sealed class MustNotBeRead : ICollectionPressure + { + public string? BehindReason() => throw new InvalidOperationException("an unpaced pass read the collection signal"); + } + + private static async Task SeedExpiredMapRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + var expiredStamp = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified).AddDays(-400); + await using var seed = new NpgsqlCommand( + "INSERT INTO collect.query_store_plan_map (server_id, database_name, plan_id, digest, plan_hash, last_seen) " + + "SELECT 1, 'db', gs, ('\\x' || lpad(to_hex(gs), 8, '0'))::bytea, 'h' || gs, $2 " + + "FROM generate_series(1, $1) gs", connection); + seed.Parameters.AddWithValue(SeededRows); + seed.Parameters.AddWithValue(expiredStamp); + await seed.ExecuteNonQueryAsync(ct); + } + + private static async Task MapRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + await using var count = new NpgsqlCommand($"SELECT COUNT(*) FROM {MapTable}", connection); + return (long)(await count.ExecuteScalarAsync(ct))!; + } + + private static RetentionWalPacer NoWaitPacer() => + new(rateBytesPerSecond: 1_000_000_000, delay: (_, _) => Task.CompletedTask); + + [Fact] + public async Task ABudgetStoppedPass_LeavesTheRemainingRows_AndTheNextPassRemovesThem() + { + var baseConnectionString = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the #5592 live budget test."); + + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await SeedExpiredMapRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + + /* Fake time that follows the work: 20 s per 1,000 rows deleted from the map. A 50 s budget is spent after the + third full batch (60 s), so the fourth batch is the one the budget stops. Every table before the map is + empty, so it costs no time, and every table after it is not reached. */ + double WorkClock() + { + using var count = new NpgsqlCommand($"SELECT COUNT(*) FROM {MapTable}", connection); + var remaining = (long)count.ExecuteScalar()!; + return (SeededRows - remaining) / (double)BatchRows * 20; + } + + var pacerOne = NoWaitPacer(); + pacerOne.Yield = new RetentionCollectionYield( + new NeverBehind(), TimeSpan.FromSeconds(50), logger: null, + delay: (_, _) => Task.CompletedTask, secondsClock: WorkClock); + var firstLog = new CapturingTestLogger(); + + var first = await DarlingRetention.PurgeWithPacerAsync( + postgres, timescaleAvailable: false, firstLog, ct, + retentionDaysFor: null, planContentRetentionDays: 0, + livenessTouchedTablePruneRowCap: BatchRows, walPacer: pacerOne); + + Assert.Equal(3 * BatchRows, first.RowsDeleted); + Assert.Equal(SeededRows - 3 * BatchRows, await MapRowsAsync(connection, ct)); + Assert.True(pacerOne.Yield.StoppedOnBudget, firstLog.Joined); + Assert.Equal("collect.query_store_plan_map", pacerOne.Yield.FirstStoppedTable); + Assert.Contains(firstLog.Lines, l => l.Contains("Retention purge stopped at its", StringComparison.Ordinal) + && l.Contains("query_store_plan_map", StringComparison.Ordinal)); + + await using (var record = new NpgsqlCommand( + "SELECT error_message FROM collect.collection_log WHERE collector_name = $1 ORDER BY collection_time DESC LIMIT 1", + connection)) + { + record.Parameters.AddWithValue("data_retention"); + var text = (string?)await record.ExecuteScalarAsync(ct); + Assert.NotNull(text); + Assert.Contains("time budget in collect.query_store_plan_map", text, StringComparison.Ordinal); + Assert.Contains("the next pass continues from the rows left", text, StringComparison.Ordinal); + } + + /* The next pass: a fresh gate with room to finish. It takes exactly the rows that were left. */ + var pacerTwo = NoWaitPacer(); + pacerTwo.Yield = new RetentionCollectionYield( + new NeverBehind(), TimeSpan.FromHours(1), logger: null, + delay: (_, _) => Task.CompletedTask); + var secondLog = new CapturingTestLogger(); + + var second = await DarlingRetention.PurgeWithPacerAsync( + postgres, timescaleAvailable: false, secondLog, ct, + retentionDaysFor: null, planContentRetentionDays: 0, + livenessTouchedTablePruneRowCap: BatchRows, walPacer: pacerTwo); + + Assert.Equal(SeededRows - 3 * BatchRows, second.RowsDeleted); + Assert.Equal(0L, await MapRowsAsync(connection, ct)); + Assert.False(pacerTwo.Yield.StoppedOnBudget, secondLog.Joined); + } + + /// + /// A paced with a signal and a spent budget attaches the rule to the + /// pacer it builds: nothing is deleted and the record says why. The same call with paceWal false ignores the + /// signal altogether and deletes everything, without reading it (the unpaced callers keep today's behavior). + /// + [Fact] + public async Task PurgeAsync_AttachesTheRuleOnAPacedRun_AndIgnoresItOnAnUnpacedOne() + { + var baseConnectionString = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the #5592 live wiring test."); + + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await SeedExpiredMapRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var log = new CapturingTestLogger(); + + /* Paced, budget already spent: the first table is not started, so nothing is deleted. */ + var stopped = await DarlingRetention.PurgeAsync( + postgres, timescaleAvailable: false, log, ct, + retentionDaysFor: null, planContentRetentionDays: 0, + paceWal: true, collectionPressure: new NeverBehind(), wallBudget: TimeSpan.Zero); + + /* The caveats prune is a direct statement, not a paced batch, so the budget does not gate it. It is the one table + this pass purged: every gated table the spent budget kept from starting is taken back out of the count (#5595 + L4: without that subtraction the count is the 80-odd tables the loop walked, and without the entry skip the + record would say the pass stopped "in" the first table instead of before it). */ + Assert.Equal(0, stopped.RowsDeleted); + Assert.Equal((long)SeededRows, await MapRowsAsync(connection, ct)); + Assert.Equal(1, stopped.TablesPurged); + Assert.Contains(log.Lines, l => l.Contains("Retention purge stopped at its", StringComparison.Ordinal) + && l.Contains("time budget before ", StringComparison.Ordinal)); + + /* Unpaced: the same signal and budget are ignored, and the signal is never read. */ + var unpaced = await DarlingRetention.PurgeAsync( + postgres, timescaleAvailable: false, log, ct, + retentionDaysFor: null, planContentRetentionDays: 0, + paceWal: false, collectionPressure: new MustNotBeRead(), wallBudget: TimeSpan.Zero); + + Assert.Equal(SeededRows, unpaced.RowsDeleted); + Assert.Equal(0L, await MapRowsAsync(connection, ct)); + } +} diff --git a/Darling/Darling.Tests/RetentionCollectionYieldTests.cs b/Darling/Darling.Tests/RetentionCollectionYieldTests.cs new file mode 100644 index 0000000000..d33f7ff9c9 --- /dev/null +++ b/Darling/Darling.Tests/RetentionCollectionYieldTests.cs @@ -0,0 +1,922 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using PerformanceMonitor.Darling.Service; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5592: the retention drain gives way to collection. Every test runs on a fake signal, a fake clock and a delay +/// that only moves that clock, so nothing sleeps. The three rules each have a test that fails without its part: the +/// wait while collection is behind, the pause scaled to the batch, and the wall budget. +/// +public sealed class RetentionCollectionYieldTests +{ + private sealed class FakeTime + { + public double Now; + public readonly List Events = new(); + + public double Clock() => Now; + + /// A 5 s delay is a re-check wait ("w"); anything else is a post-batch pause ("p"). + public Task Delay(TimeSpan wait, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + Events.Add(Math.Abs(wait.TotalSeconds - RetentionCollectionYield.RecheckSeconds) < 1e-9 ? "w" : "p"); + Now += wait.TotalSeconds; + return Task.CompletedTask; + } + } + + private sealed class ScriptedPressure : ICollectionPressure + { + private readonly Queue _script; + private readonly string? _afterScript; + + public ScriptedPressure(string? afterScript, params string?[] script) + { + _script = new Queue(script); + _afterScript = afterScript; + } + + public int Reads { get; private set; } + + public string? BehindReason() + { + Reads++; + return _script.Count > 0 ? _script.Dequeue() : _afterScript; + } + } + + /// A table of expired rows: each batch takes up to Cap of them and costs BatchSeconds of fake time. + private sealed class FakeTable + { + private readonly FakeTime _time; + private readonly Queue? _durations; + + public FakeTable(FakeTime time, int rows, int cap = 100, double batchSeconds = 1, double[]? durations = null) + { + _time = time; + Rows = rows; + Cap = cap; + BatchSeconds = batchSeconds; + _durations = durations is null ? null : new Queue(durations); + } + + public int Rows { get; private set; } + public int Cap { get; } + public double BatchSeconds { get; } + public int Batches { get; private set; } + + public Task<(int Deleted, int Cap, long WalBytes)> Execute(CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + Batches++; + _time.Events.Add("b"); + _time.Now += _durations is { Count: > 0 } ? _durations.Dequeue() : BatchSeconds; + var deleted = Math.Min(Cap, Rows); + Rows -= deleted; + return Task.FromResult((deleted, Cap, 0L)); + } + } + + private static RetentionCollectionYield GateOn( + FakeTime time, ICollectionPressure pressure, double budgetSeconds = 1_800, double pauseFactor = RetentionCollectionYield.PauseFactor) => + new(pressure, TimeSpan.FromSeconds(budgetSeconds), logger: null, time.Delay, time.Clock, pauseFactor); + + /// A WAL pacer that never waits and writes no WAL, carrying the gate the way a real paced pass does. + private static RetentionWalPacer PacerCarrying(RetentionCollectionYield? gate) => + new(rateBytesPerSecond: 1_000_000_000, delay: (_, _) => Task.CompletedTask) { Yield = gate }; + + private static Task Drain(FakeTable table, RetentionWalPacer? pacer, CancellationToken cancellationToken) => + DarlingRetention.DrainBatchesAsync(table.Execute, pacer, cancellationToken); + + /* ---- rule 1: before each batch, wait while collection is behind ---- */ + + [Fact] + public async Task Drain_WaitsWhileCollectionIsBehind_ThenRunsTheBatch() + { + var time = new FakeTime(); + /* The first three reads say behind (the check before the batch, then two re-checks), the fourth says caught up. */ + var pressure = new ScriptedPressure(null, "behind", "behind", "behind"); + var gate = GateOn(time, pressure); + var table = new FakeTable(time, rows: 50); + + var deleted = await Drain(table, PacerCarrying(gate), TestContext.Current.CancellationToken); + + Assert.Equal(50, deleted); + Assert.Equal(new[] { "w", "w", "w", "b", "p" }, time.Events); + Assert.Equal(15, gate.TotalWaitSeconds); + Assert.Equal(4, pressure.Reads); + } + + [Fact] + public async Task Drain_ChecksTheSignalBeforeEveryBatch_NotOnlyTheFirst() + { + var time = new FakeTime(); + /* Batch 1 waits once, batch 2 twice, batch 3 once. */ + var pressure = new ScriptedPressure(null, "x", null, "x", "x", null, "x", null); + var gate = GateOn(time, pressure); + var table = new FakeTable(time, rows: 250); + + var deleted = await Drain(table, PacerCarrying(gate), TestContext.Current.CancellationToken); + + Assert.Equal(250, deleted); + Assert.Equal(3, table.Batches); + Assert.Equal(new[] { "w", "b", "p", "w", "w", "b", "p", "w", "b", "p" }, time.Events); + Assert.Equal(20, gate.TotalWaitSeconds); + } + + [Fact] + public async Task Drain_NeverReadsTheSignalWhenCollectionIsKeepingUp() + { + var time = new FakeTime(); + var pressure = new ScriptedPressure(null); + var table = new FakeTable(time, rows: 250); + + await Drain(table, PacerCarrying(GateOn(time, pressure)), TestContext.Current.CancellationToken); + + Assert.DoesNotContain("w", time.Events); + Assert.Equal(3, pressure.Reads); + } + + [Fact] + public async Task Drain_AFleetThatStaysBehindStillGetsOneBatchPerMaxWait() + { + var time = new FakeTime(); + var gate = GateOn(time, new ScriptedPressure("behind"), budgetSeconds: 100_000); + var table = new FakeTable(time, rows: 50); + + var deleted = await Drain(table, PacerCarrying(gate), TestContext.Current.CancellationToken); + + Assert.Equal(50, deleted); + Assert.Equal(RetentionCollectionYield.MaxWaitSeconds, gate.TotalWaitSeconds); + Assert.Equal(60, time.Events.Count(e => e == "w")); + Assert.Equal(1, table.Batches); + } + + /// One table's purge the way PurgeOneAsync does it: the entry check, the drain, then the table's record. + private static async Task PurgeLikeTheSweep(RetentionCollectionYield gate, FakeTable table, string name, CancellationToken cancellationToken) + { + if (gate.SkipTableOnBudget(name)) + { + return 0; + } + + var drained = await Drain(table, PacerCarrying(gate), cancellationToken); + gate.FinishTable(name, drained); + return drained; + } + + [Fact] + public async Task Pass_UnderASignalThatAlwaysReadsBehind_StillGivesEveryTableItsBatches_AndTheWaitsStopAtHalfTheBudget() + { + /* #5595 H1: 40 tables of one batch each, a signal that never clears, the real 30-minute budget. Uncapped, each + table's first batch waits 300 s and the pass starts 6 tables. Capped, the waits stop at 900 s (half the budget) + and the rest of the pass keeps only the pauses. */ + var time = new FakeTime(); + var gate = GateOn(time, new ScriptedPressure("behind")); + var tables = Enumerable.Range(0, 40).Select(_ => new FakeTable(time, rows: 50)).ToList(); + + for (var i = 0; i < tables.Count; i++) + { + await PurgeLikeTheSweep(gate, tables[i], "t" + i, TestContext.Current.CancellationToken); + } + + Assert.All(tables, t => Assert.Equal(1, t.Batches)); + Assert.All(tables, t => Assert.Equal(0, t.Rows)); + Assert.False(gate.StoppedOnBudget); + Assert.Equal(0, gate.TablesNotReached); + Assert.Equal(900, gate.TotalWaitSeconds); + Assert.Equal(180, time.Events.Count(e => e == "w")); + Assert.True(gate.WaitsCapped); + + /* After the cap there is no wait at all, only batch and pause. */ + var lastWait = time.Events.FindLastIndex(e => e == "w"); + Assert.All(time.Events.Skip(lastWait + 1), e => Assert.NotEqual("w", e)); + Assert.Equal(40, gate.TotalPauseSeconds); + } + + [Fact] + public async Task Pass_TheCapCutsAWaitShort_WhenTheTotalReachesHalfTheBudgetInTheMiddleOfOne() + { + /* A 700 s budget caps the waits at 350 s: the first table waits its 300 s, the second only 50 s of its 300, and + the third none. */ + var time = new FakeTime(); + var gate = GateOn(time, new ScriptedPressure("behind"), budgetSeconds: 700); + var tables = Enumerable.Range(0, 3).Select(_ => new FakeTable(time, rows: 50)).ToList(); + + for (var i = 0; i < tables.Count; i++) + { + await PurgeLikeTheSweep(gate, tables[i], "t" + i, TestContext.Current.CancellationToken); + } + + Assert.All(tables, t => Assert.Equal(1, t.Batches)); + Assert.Equal(350, gate.WaitCapSeconds); + Assert.Equal(350, gate.TotalWaitSeconds); + Assert.Equal(70, time.Events.Count(e => e == "w")); + } + + [Fact] + public async Task Pass_SaysOncePerPassThatItStoppedWaiting() + { + var time = new FakeTime(); + var log = new CapturingTestLogger(); + var gate = new RetentionCollectionYield( + new ScriptedPressure("behind"), TimeSpan.FromSeconds(1_800), log, time.Delay, time.Clock, RetentionCollectionYield.PauseFactor); + + for (var i = 0; i < 10; i++) + { + await PurgeLikeTheSweep(gate, new FakeTable(time, rows: 50), "t" + i, TestContext.Current.CancellationToken); + } + + Assert.Equal(1, log.Lines.Count(l => l.Contains("stops waiting and keeps only the pause", StringComparison.Ordinal))); + var note = gate.Describe(); + Assert.NotNull(note); + Assert.Contains("stopped waiting at 15 minutes, half the time budget", note, StringComparison.Ordinal); + } + + /* ---- rule 2: after each batch, pause for the batch's own run time ---- */ + + [Fact] + public async Task Drain_PausesForTheBatchsOwnRunTime() + { + var time = new FakeTime(); + var gate = GateOn(time, new ScriptedPressure(null)); + var table = new FakeTable(time, rows: 250, durations: new[] { 2.0, 8.0, 30.0 }); + + await Drain(table, PacerCarrying(gate), TestContext.Current.CancellationToken); + + /* 2 + 8 + 30 s of batches, and a pause equal to each: the drain ran half the time. */ + Assert.Equal(40, gate.TotalPauseSeconds); + Assert.Equal(80, time.Now); + Assert.Equal(new[] { "b", "p", "b", "p", "b", "p" }, time.Events); + } + + [Fact] + public async Task Drain_ThePauseScalesWithTheFactor() + { + var time = new FakeTime(); + var gate = GateOn(time, new ScriptedPressure(null), pauseFactor: 0.5); + var table = new FakeTable(time, rows: 200, durations: new[] { 4.0, 10.0 }); + + await Drain(table, PacerCarrying(gate), TestContext.Current.CancellationToken); + + /* 4 s and 10 s batches, then the empty third batch that proves the table is drained (1 s): half of 15. */ + Assert.Equal(7.5, gate.TotalPauseSeconds); + } + + [Fact] + public async Task Drain_AMinutesLongBatchDoesNotStallThePassForAsLong() + { + var time = new FakeTime(); + var gate = GateOn(time, new ScriptedPressure(null)); + var table = new FakeTable(time, rows: 50, durations: new[] { 200.0 }); + + await Drain(table, PacerCarrying(gate), TestContext.Current.CancellationToken); + + Assert.Equal(RetentionCollectionYield.MaxPauseSeconds, gate.TotalPauseSeconds); + } + + [Fact] + public async Task Drain_ThePauseIsBeforeTheWalWait_SoTheTwoOverlap() + { + /* The WAL pacer refills by elapsed time, so a pause taken before its wait is time its bucket already refilled. + A 4 MiB/s pacer whose bucket (40 MiB) is already spent, then a 3 s batch that writes 24 MiB: the batch alone + leaves a 12 MiB debt after its own 3 s of refill, a 3 s wait. With the 3 s pause before the wait the refill is + 24 MiB, so the debt is gone and the pacer waits 0 s: the pause absorbed the wait instead of adding to it. */ + const long MiB = 1_048_576; + async Task<(double WalWait, List Events)> Run(bool withPause) + { + var time = new FakeTime(); + var pacer = new RetentionWalPacer( + rateBytesPerSecond: 4 * MiB, + delay: (wait, _) => { time.Events.Add("wal"); time.Now += wait.TotalSeconds; return Task.CompletedTask; }, + secondsClock: time.Clock); + if (withPause) + { + pacer.Yield = GateOn(time, new ScriptedPressure(null)); + } + + await pacer.AfterBatchAsync(40 * MiB, TestContext.Current.CancellationToken); + var spentWait = pacer.TotalWaitSeconds; + var deleted = await DarlingRetention.DrainBatchesAsync( + ct => + { + time.Events.Add("b"); + time.Now += 3; + return Task.FromResult((10, 100, 24 * MiB)); + }, + pacer, + TestContext.Current.CancellationToken); + + Assert.Equal(10, deleted); + return (pacer.TotalWaitSeconds - spentWait, time.Events); + } + + var without = await Run(withPause: false); + Assert.Equal(3, without.WalWait); + Assert.Equal(new[] { "b", "wal" }, without.Events); + + var with = await Run(withPause: true); + Assert.Equal(0, with.WalWait); + Assert.Equal(new[] { "b", "p" }, with.Events); + } + + [Fact] + public async Task Drain_ABatchThatThrowsStillGetsItsPause_ThenTheErrorReachesTheCaller() + { + var time = new FakeTime(); + var gate = GateOn(time, new ScriptedPressure(null)); + + await Assert.ThrowsAsync(() => DarlingRetention.DrainBatchesAsync( + ct => { time.Events.Add("b"); time.Now += 3; throw new InvalidOperationException("statement timeout"); }, + PacerCarrying(gate), + TestContext.Current.CancellationToken)); + + Assert.Equal(new[] { "b", "p" }, time.Events); + Assert.Equal(3, gate.TotalPauseSeconds); + } + + [Fact] + public async Task Drain_CancellationDuringThePauseEndsTheDrain() + { + var time = new FakeTime(); + var gate = GateOn(time, new ScriptedPressure(null)); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(TestContext.Current.CancellationToken); + var batches = 0; + + await Assert.ThrowsAnyAsync(() => DarlingRetention.DrainBatchesAsync( + ct => { batches++; time.Events.Add("b"); time.Now += 3; cts.Cancel(); return Task.FromResult((100, 100, 0L)); }, + PacerCarrying(gate), + cts.Token)); + + /* The pause's delay saw the cancelled token: no pause was taken and no second batch ran. */ + Assert.Equal(1, batches); + Assert.Equal(new[] { "b" }, time.Events); + Assert.Equal(0, gate.TotalPauseSeconds); + } + + [Fact] + public async Task Drain_ABatchThatThrowsOnACancelledTokenGetsNoPause() + { + var time = new FakeTime(); + var gate = GateOn(time, new ScriptedPressure(null)); + using var cts = new CancellationTokenSource(); + + await Assert.ThrowsAnyAsync(() => DarlingRetention.DrainBatchesAsync( + ct => { cts.Cancel(); ct.ThrowIfCancellationRequested(); return Task.FromResult((0, 100, 0L)); }, + PacerCarrying(gate), + cts.Token)); + + Assert.Equal(0, gate.TotalPauseSeconds); + Assert.Empty(time.Events); + } + + /* ---- rule 3: a wall budget per pass ---- */ + + [Fact] + public async Task Drain_AStoppedPassLeavesTheRowsAndTheNextPassFinishesThem() + { + var time = new FakeTime(); + var table = new FakeTable(time, rows: 1_000, cap: 100, batchSeconds: 10); + + /* Pass 1: 50 s budget; each batch is 10 s plus a 10 s pause, so batches start at 0, 20 and 40 s. */ + var gate1 = GateOn(time, new ScriptedPressure(null), budgetSeconds: 50); + var first = await Drain(table, PacerCarrying(gate1), TestContext.Current.CancellationToken); + gate1.FinishTable("t", first); + + Assert.Equal(300, first); + Assert.Equal(700, table.Rows); + Assert.True(gate1.StoppedOnBudget); + Assert.Equal("t", gate1.FirstStoppedTable); + Assert.Equal(0, gate1.TablesNotReached); + + /* Pass 2: a fresh gate, the same table: the rest. */ + var gate2 = GateOn(time, new ScriptedPressure(null), budgetSeconds: 100_000); + var second = await Drain(table, PacerCarrying(gate2), TestContext.Current.CancellationToken); + gate2.FinishTable("t", second); + + Assert.Equal(700, second); + Assert.Equal(0, table.Rows); + Assert.False(gate2.StoppedOnBudget); + } + + [Fact] + public async Task Drain_TimeSpentWaitingCountsAgainstTheBudget() + { + var time = new FakeTime(); + /* A 60 s budget: the wait before the first batch is capped at half of it (30 s), then each batch is 10 s plus a + 10 s pause. Without the wait three batches would start (at 0, 20 and 40 s); with it the second starts at 50 s + and the third never does. */ + var gate = GateOn(time, new ScriptedPressure("behind"), budgetSeconds: 60); + var table = new FakeTable(time, rows: 1_000, cap: 100, batchSeconds: 10); + + var deleted = await Drain(table, PacerCarrying(gate), TestContext.Current.CancellationToken); + gate.FinishTable("t", deleted); + + Assert.Equal(200, deleted); + Assert.Equal(2, table.Batches); + Assert.Equal(30, gate.TotalWaitSeconds); + Assert.True(gate.StoppedOnBudget); + Assert.Equal(0, gate.TablesNotReached); + } + + [Fact] + public async Task Drain_ASpentBudgetSkipsThePauseSoThePassEndsAtOnce() + { + var time = new FakeTime(); + var gate = GateOn(time, new ScriptedPressure(null), budgetSeconds: 10); + var table = new FakeTable(time, rows: 1_000, batchSeconds: 10); + + await Drain(table, PacerCarrying(gate), TestContext.Current.CancellationToken); + + Assert.Equal(new[] { "b" }, time.Events); + Assert.Equal(0, gate.TotalPauseSeconds); + } + + [Fact] + public void SkipTableOnBudget_StartsNothingOnceTheBudgetIsSpent_AndCountsTheTable() + { + var time = new FakeTime(); + var gate = GateOn(time, new ScriptedPressure(null), budgetSeconds: 30); + + Assert.False(gate.SkipTableOnBudget("early")); + Assert.False(gate.StoppedOnBudget); + + time.Now = 30; + Assert.True(gate.SkipTableOnBudget("first-skipped")); + Assert.True(gate.SkipTableOnBudget("second-skipped")); + + Assert.True(gate.StoppedOnBudget); + Assert.Equal("first-skipped", gate.FirstStoppedTable); + Assert.Equal(2, gate.TablesNotReached); + Assert.Equal(2, gate.TablesLeft); + } + + [Fact] + public async Task Describe_AStopAtATablesEntryReadsBefore_NotIn_AndAStopInsideATableReadsIn() + { + /* #5595 L1: the budget runs out during table A's last batch, A ends normally, and B is the first table the pass + never started. The record must not say the pass stopped "in" B. */ + var time = new FakeTime(); + var gate = GateOn(time, new ScriptedPressure(null), budgetSeconds: 20); + await PurgeLikeTheSweep(gate, new FakeTable(time, rows: 50, batchSeconds: 25), "table_a", TestContext.Current.CancellationToken); + await PurgeLikeTheSweep(gate, new FakeTable(time, rows: 50), "table_b", TestContext.Current.CancellationToken); + await PurgeLikeTheSweep(gate, new FakeTable(time, rows: 50), "table_c", TestContext.Current.CancellationToken); + + Assert.Equal("before table_b", gate.StoppedPlace); + Assert.Equal(2, gate.TablesNotReached); + Assert.Equal(2, gate.TablesLeft); + Assert.Contains("time budget before table_b, with 2 table(s) not reached", gate.Describe(), StringComparison.Ordinal); + + /* Stopped part-way through a table: "in". */ + var time2 = new FakeTime(); + var gate2 = GateOn(time2, new ScriptedPressure(null), budgetSeconds: 20); + await PurgeLikeTheSweep(gate2, new FakeTable(time2, rows: 1_000, batchSeconds: 25), "table_x", TestContext.Current.CancellationToken); + Assert.Equal("in table_x", gate2.StoppedPlace); + Assert.Equal(1, gate2.TablesLeft); /* #5592: stopped part-way still has rows left, though it is not "not reached" */ + Assert.Equal(0, gate2.TablesNotReached); + Assert.Contains("time budget in table_x", gate2.Describe(), StringComparison.Ordinal); + } + + [Fact] + public async Task Describe_APassThatOnlyPausedReadsAsBefore_AndAPassThatWaitedSaysSo() + { + /* #5595 L2: the pause after a batch happens on every paced pass; it is not pressure and does not write the sentence. */ + var time = new FakeTime(); + var calm = GateOn(time, new ScriptedPressure(null)); + await PurgeLikeTheSweep(calm, new FakeTable(time, rows: 250, batchSeconds: 2), "t", TestContext.Current.CancellationToken); + Assert.True(calm.TotalPauseSeconds > 0); + Assert.Null(calm.Describe()); + Assert.Equal(0, calm.TablesLeft); + + var time2 = new FakeTime(); + var pushed = GateOn(time2, new ScriptedPressure(null, "behind", "behind")); + await PurgeLikeTheSweep(pushed, new FakeTable(time2, rows: 50), "t", TestContext.Current.CancellationToken); + Assert.Contains("yielded to collection: waited 10 s while it was behind, paused 1 s between batches", pushed.Describe(), StringComparison.Ordinal); + } + + [Fact] + public void FinishTable_ATableStoppedAfterDeletingRowsIsNotCountedAsNotReached() + { + var gate = GateOn(new FakeTime(), new ScriptedPressure(null)); + + gate.FinishTable("untouched", 0); + Assert.False(gate.StoppedOnBudget); + + gate.NoteBudgetStop(); + gate.FinishTable("part-way", 5_000); + + Assert.True(gate.StoppedOnBudget); + Assert.Equal("part-way", gate.FirstStoppedTable); + Assert.Equal(0, gate.TablesNotReached); + } + + /* ---- cancellation ---- */ + + [Fact] + public async Task Drain_CancellationDuringAWaitEndsPromptly() + { + /* The real delay and clock: the wait is 5 s long, and the cancel must end it well inside that. */ + var gate = new RetentionCollectionYield(new ScriptedPressure("behind"), TimeSpan.FromMinutes(30)); + var batches = 0; + using var cts = CancellationTokenSource.CreateLinkedTokenSource(TestContext.Current.CancellationToken); + + var drain = DarlingRetention.DrainBatchesAsync( + ct => { batches++; return Task.FromResult((0, 100, 0L)); }, PacerCarrying(gate), cts.Token); + cts.CancelAfter(TimeSpan.FromMilliseconds(100)); + + var finished = await Task.WhenAny(drain, Task.Delay(TimeSpan.FromSeconds(4), TestContext.Current.CancellationToken)); + Assert.Same(drain, finished); + await Assert.ThrowsAnyAsync(() => drain); + Assert.Equal(0, batches); + } + + [Fact] + public async Task Drain_CancellationBeforeTheFirstBatchRunsNoBatch() + { + var time = new FakeTime(); + var table = new FakeTable(time, rows: 500); + using var cts = new CancellationTokenSource(); + await cts.CancelAsync(); + + await Assert.ThrowsAnyAsync( + () => Drain(table, PacerCarrying(GateOn(time, new ScriptedPressure(null))), cts.Token)); + + Assert.Equal(0, table.Batches); + } + + /* ---- unpaced callers keep today's behavior ---- */ + + [Fact] + public async Task Drain_ACallerWithoutAGateRunsBatchesBackToBack() + { + var time = new FakeTime(); + var table = new FakeTable(time, rows: 250); + + var withPacerNoGate = await Drain(table, PacerCarrying(null), TestContext.Current.CancellationToken); + + Assert.Equal(250, withPacerNoGate); + Assert.Equal(new[] { "b", "b", "b" }, time.Events); + + var unpacedTable = new FakeTable(time, rows: 250); + var unpaced = await DarlingRetention.DrainBatchesAsync( + async ct => { var (d, c, _) = await unpacedTable.Execute(ct); return (d, c); }, + TestContext.Current.CancellationToken); + + Assert.Equal(250, unpaced); + Assert.DoesNotContain("p", time.Events); + Assert.DoesNotContain("w", time.Events); + } + + [Fact] + public void Describe_IsNullWhenThePassNeitherWaitedNorStopped_AndTheRunRecordTextIsUnchanged() + { + var gate = GateOn(new FakeTime(), new ScriptedPressure(null)); + Assert.Null(gate.Describe()); + + var plain = DarlingRetention.BuildRunRecordSummary(89, 18_007_201, 0, 0, paced: true, walBytes: 9_230L * 1_048_576, pacedSeconds: 596); + var withNoNote = DarlingRetention.BuildRunRecordSummary( + 89, 18_007_201, 0, 0, paced: true, walBytes: 9_230L * 1_048_576, pacedSeconds: 596, yieldNote: null); + Assert.Equal(plain, withNoNote); + } + + [Fact] + public async Task Describe_AStoppedPassSaysSo_InTheRunRecordWithoutBecomingAWarning() + { + var time = new FakeTime(); + var gate = GateOn(time, new ScriptedPressure(null), budgetSeconds: 50); + var table = new FakeTable(time, rows: 1_000, batchSeconds: 10); + gate.FinishTable("collect.query_store_interval_wide_legacy", await Drain(table, PacerCarrying(gate), TestContext.Current.CancellationToken)); + + var note = gate.Describe(); + Assert.NotNull(note); + Assert.Contains("paused 20 s between batches", note, StringComparison.Ordinal); + Assert.Contains("stopped at its 1-minute time budget in collect.query_store_interval_wide_legacy", note, StringComparison.Ordinal); + Assert.Contains("the next pass continues from the rows left", note, StringComparison.Ordinal); + + var (status, message) = DarlingRetention.BuildRunRecordSummary(12, 300, 0, 0, paced: true, yieldNote: note); + Assert.Equal("SUCCESS", status); + Assert.EndsWith(note!, message, StringComparison.Ordinal); + } + + /* ---- the real signal ---- */ + + [Fact] + public void Pressure_CollectionIsSettlingForTheFirstTenMinutesAfterCollectionStarts() + { + double seconds = 100_000; + var pressure = new CollectionPressure(stats: null, () => seconds); + pressure.MarkCollectionStarted(); + + seconds += 5; + Assert.Contains("still settling", pressure.BehindReason(), StringComparison.Ordinal); + + seconds = 100_000 + CollectionPressure.SettleWindow.TotalSeconds - 1; + Assert.NotNull(pressure.BehindReason()); + + seconds = 100_000 + CollectionPressure.SettleWindow.TotalSeconds; + Assert.Null(pressure.BehindReason()); + Assert.Equal(TimeSpan.FromMinutes(10), CollectionPressure.SettleWindow); + + /* The first mark wins: a second call does not restart the window. */ + pressure.MarkCollectionStarted(); + Assert.Null(pressure.BehindReason()); + } + + [Fact] + public void Pressure_TheSettleWindowStartsAtTheMark_NotWhenThePressureWasBuilt() + { + /* #5595 M2: the worker is built, then the store retries and migrations take 25 minutes, then the loop starts + collecting. The window runs from the loop's start, not the construction. Until then it reads settling. */ + double seconds = 0; + var pressure = new CollectionPressure(stats: null, () => seconds); + + seconds = 1_500; + Assert.Contains("has not started", pressure.BehindReason(), StringComparison.Ordinal); + + pressure.MarkCollectionStarted(); + seconds = 1_500 + 599; + Assert.Contains("still settling", pressure.BehindReason(), StringComparison.Ordinal); + + seconds = 1_500 + 600; + Assert.Null(pressure.BehindReason()); + } + + [Fact] + public void Pressure_TheSettleClockIsMonotonic_AndTheWorkerMarksTheStartWhereTheLoopBegins() + { + /* No wall clock in the pressure read: a backward step of the clock must not hold the drain, so it carries no DateTime. */ + var pressureSource = ReadSource("RetentionCollectionYield.cs"); + Assert.DoesNotContain("DateTime", pressureSource, StringComparison.Ordinal); + + var worker = ReadWorkerSource().Replace("\r\n", "\n", StringComparison.Ordinal); + Assert.Equal(1, CountOf(worker, "_collectionPressure.MarkCollectionStarted();")); + Assert.Equal(1, CountOf(worker, "new CollectionPressure(_fleetGateStats);")); + var loopStarted = worker.IndexOf("PerformanceMonitor Darling collection loop started", StringComparison.Ordinal); + var mark = worker.IndexOf("_collectionPressure.MarkCollectionStarted();", StringComparison.Ordinal); + var loop = worker.IndexOf("while (!stoppingToken.IsCancellationRequested)", mark, StringComparison.Ordinal); + Assert.True(loopStarted > 0 && mark > loopStarted && loop > mark, "the mark must sit between the loop's start log and the sweep loop"); + Assert.DoesNotContain("while (", worker[mark..loop], StringComparison.Ordinal); + } + + /// The start of the fake hour: well past the settle window, on a half-minute so a minute never straddles a read. + private static readonly DateTime HourStart = new(2026, 10, 8, 10, 0, 30, DateTimeKind.Utc); + + /// A pressure read whose collection started two hours ago, so the settle window is long over. + private static CollectionPressure Settled(FleetGateStats? stats) + { + var seconds = 0.0; + var pressure = new CollectionPressure(stats, () => seconds); + pressure.MarkCollectionStarted(); + seconds = 7_200; + return pressure; + } + + /// + /// Plays one hour a minute at a time and re-checks the signal once a minute (the drain re-checks every 5 s, so + /// this is a coarser read of the same buckets). and + /// give each minute's slots; returns how many of the 60 re-checks read "behind". + /// + private static int PlayHour(Func runPerMinute, Func skippedPerMinute, out FleetGateSnapshot hour) + { + var now = HourStart; + var stats = new FleetGateStats(() => now); + var pressure = Settled(stats); + var behind = 0; + for (var minute = 0; minute < 60; minute++) + { + now = HourStart.AddMinutes(minute); + var skipped = skippedPerMinute(minute); + for (var i = 0; i < runPerMinute(minute); i++) + { + /* The first slot of the minute carries that minute's skips, as a late run does. */ + stats.RecordSlot(i == 0 ? skipped : 0); + } + + if (pressure.BehindReason() is not null) + { + behind++; + } + } + + hour = stats.Snapshot(); + return behind; + } + + [Fact] + public void Pressure_ANormalHourWithThreeHangEpisodes_NeverReadsBehind() + { + /* The field's normal day: about 3 "skipping relaunch" episodes an hour, each stepping over a few slots, about + 1,250 slots due an hour (21 a minute). One episode alone never reaches the 5-slot minimum, and the hour + stays well under the alert's 5%. The old rule (any skipped slot in 5 minutes) read behind for 15 of these 60 re-checks. */ + var episodes = new Dictionary { [8] = 3, [31] = 4, [52] = 3 }; + var behind = PlayHour(_ => 21, m => episodes.GetValueOrDefault(m), out var hour); + + Assert.Equal(0, behind); + Assert.Equal(10, hour.Skipped); + Assert.InRange(hour.Due, 1_200, 1_300); + Assert.False(new DarlingSelfAlertEvaluator.FleetGateReport(hour.Run, hour.Skipped, 0, TimeSpan.Zero, TimeSpan.Zero, 1, HourStart).IsBehind); + } + + [Fact] + public void Pressure_ANormalHourWhereTwoEpisodesLandTogether_ReadsBehindOnlyWhileBothAreInTheWindow() + { + /* Episodes at minutes 5 and 12 (3 slots each) put 6 skipped slots, 2.8% of about 216 due, in one 10-minute + window from minute 12 until the first one ages out at minute 15: 3 of 60 re-checks, not a fifth of them. */ + var episodes = new Dictionary { [5] = 3, [12] = 3, [30] = 3, [45] = 3 }; + var behind = PlayHour(_ => 21, m => episodes.GetValueOrDefault(m), out _); + + Assert.Equal(3, behind); + } + + [Fact] + public void Pressure_TheIncidentHour_ReadsBehindThroughout() + { + /* The pre-fix drain hour: 97 of 1,252 due slots skipped (7.7%). 2 skipped a minute for 37 minutes, then 1. */ + var behind = PlayHour(m => m < 15 ? 20 : 19, m => m < 37 ? 2 : 1, out var hour); + + Assert.Equal(97, hour.Skipped); + Assert.Equal(1_252, hour.Due); + Assert.True(hour.SkippedPercent > 7.5); + Assert.True(new DarlingSelfAlertEvaluator.FleetGateReport(hour.Run, hour.Skipped, 0, TimeSpan.Zero, TimeSpan.Zero, 1, HourStart).IsBehind); + /* Behind from the third minute on (the 5-slot minimum is reached after three minutes of skips at 2 a minute). */ + Assert.True(behind >= 57, $"the incident hour read behind in only {behind} of 60 re-checks"); + } + + [Theory] + [InlineData(195, 5, true)] /* 5 of 200 = 2.5% exactly: counts */ + [InlineData(196, 5, false)] /* 5 of 201 = 2.49%: under the share */ + [InlineData(194, 6, true)] /* 6 of 200 = 3% */ + [InlineData(203, 5, false)] /* 5 of 208 = 2.4%: at the field's rate the share, not the minimum, decides */ + [InlineData(202, 6, true)] /* 6 of 208 = 2.88% */ + [InlineData(18, 2, false)] /* 2 of 20 = 10%: a big share of a thin window, but under the 3-slot minimum */ + [InlineData(17, 3, true)] /* 3 of 20 = 15%: at the minimum */ + [InlineData(16, 4, true)] /* 4 of 20 = 20% */ + [InlineData(0, 0, false)] /* nothing due */ + public void IsBehind_NeedsTheMinimumCountAndTheShare(long run, long skipped, bool expected) + { + Assert.Equal(expected, CollectionPressure.IsBehind(run, skipped)); + } + + [Fact] + public void Pressure_ASustainedRateJustUnderTheMinimum_NeverHoldsTheDrain_AndStaysUnderTheAlert() + { + /* A thin fleet (2 slots a minute, 20 in a window) with (minimum - 1) = 2 skipped slots landing in each of the + hour's six 10-minute windows: never behind, and the hour's total (12) is under the alert's minimum. One more + slot in a window holds the drain. */ + var justUnder = PlayHour(_ => 2, minute => minute % 10 == 0 ? (int)(CollectionPressure.SkipMinCount - 1) : 0, out var hour); + Assert.Equal(0, justUnder); + Assert.Equal(6 * (CollectionPressure.SkipMinCount - 1), hour.Skipped); + Assert.True(hour.Skipped < DarlingSelfAlertEvaluator.FleetGateBehindMinSkipped); + + var atMinimum = PlayHour(_ => 2, minute => minute % 10 == 0 ? (int)CollectionPressure.SkipMinCount : 0, out _); + Assert.True(atMinimum > 0); + } + + [Fact] + public void Pressure_TheSkippedSlotsAgeOutOfTheWindowAfterTenMinutes() + { + var now = HourStart; + var stats = new FleetGateStats(() => now); + var pressure = Settled(stats); + for (var i = 0; i < 200; i++) + { + stats.RecordSlot(0); + } + + /* The held-slot path the self-alert also counts (#5479). */ + stats.RecordSkippedSlots(6); + var reason = pressure.BehindReason(); + Assert.NotNull(reason); + Assert.Contains("6 of 206 collector slots were skipped in the last 10 minutes", reason, StringComparison.Ordinal); + + now = now.AddMinutes(9); + Assert.NotNull(pressure.BehindReason()); + + now = now.AddMinutes(1); + Assert.Null(pressure.BehindReason()); + } + + [Fact] + public void Pressure_AWorkerBuiltWithoutStatsReadsNoSkips() + { + Assert.Null(Settled(stats: null).BehindReason()); + } + + [Fact] + public void Pressure_TheConstantsAreTheDocumentedOnes() + { + Assert.Equal(10, CollectionPressure.SkipWindowMinutes); + Assert.Equal(3, CollectionPressure.SkipMinCount); + /* Half the alert's 5%: the drain backs off before the hour reaches the alert. */ + Assert.Equal(25, CollectionPressure.SkipSharePerMille); + Assert.Equal(DarlingSelfAlertEvaluator.FleetGateBehindPercent * 10 / 2, CollectionPressure.SkipSharePerMille); + /* #5595 M1: the alert's 60 buckets are six disjoint windows. A drain held just under the minimum lets + (minimum - 1) slots skip per window, plus one slot of slop between reads: that hour must stay under the alert's + minimum, and the next minimum up must not (so 3 is the largest). */ + var windowsPerHour = 60 / CollectionPressure.SkipWindowMinutes; + Assert.Equal(6, windowsPerHour); + Assert.True( + (CollectionPressure.SkipMinCount - 1 + 1) * windowsPerHour < DarlingSelfAlertEvaluator.FleetGateBehindMinSkipped, + "the worst hour a drain held just under the minimum can sustain must stay under the alert's minimum"); + Assert.True( + (CollectionPressure.SkipMinCount + 1) * windowsPerHour >= DarlingSelfAlertEvaluator.FleetGateBehindMinSkipped, + "the minimum is the largest that stays under the alert's"); + Assert.Equal(0.5, RetentionCollectionYield.WaitBudgetFraction); + /* One wait is half the window: a burst can hold the signal for the whole window, but a batch goes every 5 minutes. */ + Assert.Equal(300, RetentionCollectionYield.MaxWaitSeconds); + Assert.Equal(CollectionPressure.SkipWindowMinutes * 60 / 2, RetentionCollectionYield.MaxWaitSeconds); + } + + [Fact] + public void FleetGateStats_SlotsInLastMinutes_ReadsTheSameBucketsTheSelfAlertReads() + { + var now = new DateTime(2026, 10, 8, 12, 0, 30, DateTimeKind.Utc); + var stats = new FleetGateStats(() => now); + stats.RecordSlot(2); + now = now.AddMinutes(7); + stats.RecordSlot(5); + stats.RecordSlot(0); + + Assert.Equal((2L, 5L), stats.SlotsInLastMinutes(5)); + Assert.Equal((3L, 7L), stats.SlotsInLastMinutes(8)); + var all = stats.Snapshot(); + Assert.Equal((all.Run, all.Skipped), stats.SlotsInLastMinutes(FleetGateStats.WindowMinutes)); + Assert.Equal((all.Run, all.Skipped), stats.SlotsInLastMinutes(10_000)); + Assert.Equal((0L, 0L), stats.SlotsInLastMinutes(0)); + Assert.Equal((0L, 0L), stats.SlotsInLastMinutes(-3)); + } + + [Fact] + public void ABudgetStopIsNormalControlFlow_TheWarmUpAfterTheDrainStillRuns() + { + /* #5585's warm-up runs for the interval tables whose purge deleted rows, and a pass stopped part-way through a + table did delete rows. The stop must therefore fall through to the run record and the warm-up: no return + and no throw between the budget's log line and the warm-up call. */ + var source = ReadSource("DarlingRetention.cs").Replace("\r\n", "\n", StringComparison.Ordinal); + var stopLog = source.IndexOf("if (yieldGate is { StoppedOnBudget: true })", StringComparison.Ordinal); + var warmUp = source.IndexOf("QueryStoreIntervalFloorWarmUp.RunAfterDrainAsync(", StringComparison.Ordinal); + Assert.True(stopLog > 0 && warmUp > stopLog, "the budget's log line must come before the warm-up call"); + + var between = source[stopLog..warmUp]; + Assert.DoesNotContain("return ", between, StringComparison.Ordinal); + Assert.DoesNotContain("throw ", between, StringComparison.Ordinal); + Assert.DoesNotContain("throw;", between, StringComparison.Ordinal); + Assert.Contains("LogRetentionRunAsync", between, StringComparison.Ordinal); + } + + [Fact] + public void Worker_BothPurgeCallersPassTheSignal() + { + var source = ReadWorkerSource().Replace("\r\n", "\n", StringComparison.Ordinal); + + /* The daily sweep (RunScheduledPurgeAsync) and the on-demand purge_now (RunPurgeNowBackgroundAsync). */ + Assert.Equal(2, CountOf(source, "collectionPressure: _collectionPressure")); + Assert.Contains("paceWal: true,\n collectionPressure: _collectionPressure", source, StringComparison.Ordinal); + Assert.Contains("runLabel: runLabel,\n collectionPressure: _collectionPressure", source, StringComparison.Ordinal); + } + + private static int CountOf(string text, string needle) + { + var count = 0; + for (var at = text.IndexOf(needle, StringComparison.Ordinal); at >= 0; at = text.IndexOf(needle, at + needle.Length, StringComparison.Ordinal)) + { + count++; + } + + return count; + } + + private static string ReadWorkerSource([System.Runtime.CompilerServices.CallerFilePath] string thisFile = "") => + ReadSource("DarlingWorker.cs", thisFile); + + private static string ReadSource(string fileName, [System.Runtime.CompilerServices.CallerFilePath] string thisFile = "") + { + var relative = Path.Combine("Darling", "PerformanceMonitor.Darling.Service", fileName); + for (var dir = new DirectoryInfo(Path.GetDirectoryName(thisFile)!); dir is not null; dir = dir.Parent) + { + var candidate = Path.Combine(dir.FullName, relative); + if (File.Exists(candidate)) + { + return File.ReadAllText(candidate); + } + } + + throw new FileNotFoundException($"Could not locate {relative}"); + } +} diff --git a/Darling/Darling.Tests/RetentionHoldAndCadenceRawRedirectLiveTests.cs b/Darling/Darling.Tests/RetentionHoldAndCadenceRawRedirectLiveTests.cs index e2de7c9e50..e9aee0106e 100644 --- a/Darling/Darling.Tests/RetentionHoldAndCadenceRawRedirectLiveTests.cs +++ b/Darling/Darling.Tests/RetentionHoldAndCadenceRawRedirectLiveTests.cs @@ -37,22 +37,37 @@ public sealed class RetentionHoldAndCadenceRawRedirectLiveTests "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live #4299 redirect pins."); var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, default); + NpgsqlConnection? connection = null; + /* A skip (no TimescaleDB) or a failed setup throws before the caller owns the store, so the opener drops it + here. Left alone it would survive to the process-exit drain, which the test runner waits only 10 s for. */ + try + { + var connectionString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { MaxAutoPrepare = 0 }.ConnectionString; + connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(); + await PgMigrations.MigrateAsync(connection, default); - var connectionString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { MaxAutoPrepare = 0 }.ConnectionString; - var connection = new NpgsqlConnection(connectionString); - await connection.OpenAsync(); - await PgMigrations.MigrateAsync(connection, default); + var enabled = await TimescaleSupport.TryEnableAsync(connection, null, default); + Assert.SkipWhen(!enabled, "The live #4299 redirect pins need TimescaleDB."); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, default); - var enabled = await TimescaleSupport.TryEnableAsync(connection, null, default); - Assert.SkipWhen(!enabled, "The live #4299 redirect pins need TimescaleDB."); - await TimescaleSupport.ConvertToHypertablesAsync(connection, null, default); + await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection) { CommandTimeout = SetupTimeoutSeconds }) + { + await stop.ExecuteNonQueryAsync(); + } - await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection) { CommandTimeout = SetupTimeoutSeconds }) - { - await stop.ExecuteNonQueryAsync(); + return (connection, scratch); } + catch + { + if (connection is not null) + { + await connection.DisposeAsync(); + } - return (connection, scratch); + await scratch.DisposeAsync(); + throw; + } } private static async Task ArmRawJobAsync(NpgsqlConnection connection, string relation) @@ -100,6 +115,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -134,6 +150,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } @@ -180,6 +197,7 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async ( await batch.RemoveRetentionPolicyAsync(Raw, cleanupCt); }); await connection.DisposeAsync(); + await scratch.DisposeAsync(); } } } diff --git a/Darling/Darling.Tests/RetentionReevaluationTests.cs b/Darling/Darling.Tests/RetentionReevaluationTests.cs index 63411c78f9..11f0e13235 100644 --- a/Darling/Darling.Tests/RetentionReevaluationTests.cs +++ b/Darling/Darling.Tests/RetentionReevaluationTests.cs @@ -310,23 +310,26 @@ about this tick. */ var guardAt = worker.IndexOf("if (StampIsDue(_nextCompressionCheckUtc, CompressionCheckSpan, DateTime.UtcNow))", StringComparison.Ordinal); var stampAt = worker.IndexOf("_nextCompressionCheckUtc = TimescaleSupport.NextCompressionCheckUtc(DateTime.UtcNow, s_compressionCheckInterval);", StringComparison.Ordinal); Assert.True(guardAt > 0 && stampAt > guardAt, "the tick's guard is the due time, stamped forward inside it"); - var flagGateAt = worker.IndexOf("if (_timescaleAvailable)", stampAt, StringComparison.Ordinal); - var compressionAt = worker.IndexOf("await EvaluateCompressionJobHealthAsync(stoppingToken);", StringComparison.Ordinal); - var retentionAt = worker.IndexOf("await ReevaluateRetentionPoliciesAsync(stoppingToken);", StringComparison.Ordinal); - /* #3817 is the FOURTH tenant and sits after this pass, so the "next thing" that used to bound the - retention call's position is now the convergence call, and the store-metrics block bounds THAT. - Both bounds are kept: dropping the outer one would let a later edit hoist the whole gated group out - of the tick without this pin noticing. */ - var convergenceAt = worker.IndexOf("await ConvergeStoreObjectsAsync(stoppingToken);", StringComparison.Ordinal); + /* #4970: the tenants run in RunStoreMaintenanceTickAsync, launched from the loop after the stamp. + Their order is read inside that method's body, so the flag gate is found from the body's start. */ + var launchAt = worker.IndexOf("TryStartStoreMaintenanceTick(token => RunStoreMaintenanceTickAsync(token), stoppingToken);", stampAt, StringComparison.Ordinal); + Assert.True(launchAt > stampAt, "the loop launches the tick after stamping its due time"); var nextBlockAt = worker.IndexOf("/* #2068: the store self-metrics sweep.", StringComparison.Ordinal); - Assert.True(flagGateAt > stampAt && compressionAt > flagGateAt && retentionAt > compressionAt - && convergenceAt > retentionAt && nextBlockAt > convergenceAt, + Assert.True(nextBlockAt > launchAt, "the launch stays ahead of the store self-metrics block"); + var tickBody = MethodBody(worker, "private async Task RunStoreMaintenanceTickAsync(CancellationToken stoppingToken)"); + Assert.False(string.IsNullOrEmpty(tickBody), "could not locate RunStoreMaintenanceTickAsync"); + var flagGateAt = tickBody.IndexOf("if (_timescaleAvailable)", StringComparison.Ordinal); + var compressionAt = tickBody.IndexOf("await EvaluateCompressionJobHealthAsync(stoppingToken);", StringComparison.Ordinal); + var retentionAt = tickBody.IndexOf("await ReevaluateRetentionPoliciesAsync(stoppingToken);", StringComparison.Ordinal); + var convergenceAt = tickBody.IndexOf("await ConvergeStoreObjectsAsync(stoppingToken);", StringComparison.Ordinal); + Assert.True(flagGateAt >= 0 && compressionAt > flagGateAt && retentionAt > compressionAt + && convergenceAt > retentionAt, "the retention re-evaluation is awaited after the compression check inside the tick's _timescaleAvailable gate, " + "and the store-object convergence after it"); Assert.Equal(1, CountOf(worker, "await ReevaluateRetentionPoliciesAsync(stoppingToken);")); Assert.Equal(1, CountOf(worker, "await ConvergeStoreObjectsAsync(stoppingToken);")); - var between = worker[compressionAt..retentionAt]; + var between = tickBody[compressionAt..retentionAt]; Assert.DoesNotContain("if (", between, StringComparison.Ordinal); Assert.DoesNotContain("return", between, StringComparison.Ordinal); Assert.DoesNotContain("}", between, StringComparison.Ordinal); diff --git a/Darling/Darling.Tests/RetiredBaselineAggregateTests.cs b/Darling/Darling.Tests/RetiredBaselineAggregateTests.cs index 5c1adc8557..0834ea0798 100644 --- a/Darling/Darling.Tests/RetiredBaselineAggregateTests.cs +++ b/Darling/Darling.Tests/RetiredBaselineAggregateTests.cs @@ -145,7 +145,11 @@ FROM collect.cpu_utilization_stats GROUP BY server_id, bucket, collection_time WITH NO DATA", ct); await ExecuteAsync(connection, - "SELECT add_continuous_aggregate_policy('collect.cpu_utilization_baseline', start_offset => INTERVAL '3 days', end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour', if_not_exists => true)", ct); + /* #5416: initial_start a day out, so TimescaleDB's scheduler never runs this policy's refresh job while + the test is alive. A refresh job running at the moment of the sweep's DROP fails it with XX000 + "tuple concurrently deleted" and left a test connection Closed (one CI flake, reproduced locally). + The policy still exists for the cascade assertions. */ + "SELECT add_continuous_aggregate_policy('collect.cpu_utilization_baseline', start_offset => INTERVAL '3 days', end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour', initial_start => now() + INTERVAL '1 day', if_not_exists => true)", ct); await ExecuteAsync(connection, "SELECT add_retention_policy('collect.cpu_utilization_baseline', drop_after => INTERVAL '35 days', if_not_exists => true)", ct); await ExecuteAsync(connection, @@ -278,7 +282,8 @@ await ExecuteAsync(connection, await ExecuteAsync(connection, TimescaleSupport.LegacyCreateWaitStatsBaselineSql, ct); await ExecuteAsync(connection, TimescaleSupport.CreateWaitStatsIntervalBaselineSql, ct); await ExecuteAsync(connection, - $"SELECT add_continuous_aggregate_policy('collect.{legacy}', start_offset => INTERVAL '1 day', end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour', if_not_exists => true)", ct); + /* #5416: initial_start a day out, so no refresh job races the day-forty DROP (see the first test). */ + $"SELECT add_continuous_aggregate_policy('collect.{legacy}', start_offset => INTERVAL '1 day', end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour', initial_start => now() + INTERVAL '1 day', if_not_exists => true)", ct); await RefreshFromAsync(connection, legacy, hour1, ct); await RefreshFromAsync(connection, successor, hour2, ct); @@ -363,15 +368,31 @@ zero is gone from the statistic. */ asserts the outcome (the sweep removes the legacy), not that the first call does. */ var sweepLog = new CapturingTestLogger(); var sweepDropped = 0; - for (var attempt = 0; - attempt < 5 && await ScalarAsync(connection, TimescaleSupport.BaselineRelationExistsSql(legacy), null, ct); - attempt++) + /* DropRetiredBaselineAggregatesAsync catches every non-cancel exception into its logger, so a drop + that broke the connection surfaced here as "Connection is not open" from the loop's next + statement while the real error sat in sweepLog, which only the assert below prints and a thrown + loop never reaches. Any failure in the loop, and a connection no longer open after a drop, now + fails with the sweep's log in the message. */ + try { - if (attempt > 0) + for (var attempt = 0; + attempt < 5 && await ScalarAsync(connection, TimescaleSupport.BaselineRelationExistsSql(legacy), null, ct); + attempt++) { - await Task.Delay(TimeSpan.FromSeconds(1), ct); + if (attempt > 0) + { + await Task.Delay(TimeSpan.FromSeconds(1), ct); + } + sweepDropped += await TimescaleSupport.DropRetiredBaselineAggregatesAsync(connection, sweepLog, dayForty, ct); + if (connection.State != System.Data.ConnectionState.Open) + { + throw new InvalidOperationException($"the connection is {connection.State} after sweep attempt {attempt + 1}"); + } } - sweepDropped += await TimescaleSupport.DropRetiredBaselineAggregatesAsync(connection, sweepLog, dayForty, ct); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + throw new Xunit.Sdk.XunitException($"the sweep loop failed with {ex.GetType().Name}: {ex.Message} -- the sweep's log: {sweepLog.Joined}", ex); } Assert.True(sweepDropped >= 1, $"the sweep never dropped the legacy aggregate: {sweepLog.Joined}"); Assert.False(await ScalarAsync(connection, TimescaleSupport.BaselineRelationExistsSql(legacy), null, ct), "the legacy aggregate must be gone"); diff --git a/Darling/Darling.Tests/RetiredBaselineSweepRecoveryTests.cs b/Darling/Darling.Tests/RetiredBaselineSweepRecoveryTests.cs new file mode 100644 index 0000000000..cb15680cee --- /dev/null +++ b/Darling/Darling.Tests/RetiredBaselineSweepRecoveryTests.cs @@ -0,0 +1,251 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Data; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Darling.Tests; +using Microsoft.Extensions.Logging; +using Npgsql; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace PerformanceMonitor.Darling.Tests; + +/// +/// #5416: the retirement sweep on a connection that a failed drop has just broken. A refresh job that is +/// running when the sweep's DROP MATERIALIZED VIEW ... CASCADE lands makes the drop fail with +/// XX000: tuple concurrently deleted (severity ERROR, not FATAL). Npgsql 10 closes the connection on an +/// ERROR in SQLSTATE classes XX, 58 and 53 (measured: State goes to Closed, FullState to +/// Broken), and the sweep's one connection is shared by every relation it walks and by every convergence step +/// that follows it. Before this, one lost race left the next relation's attempts, and the baseline fallback views +/// and statement statistics steps behind it, failing on "Connection is not open" until the next hourly pass. +/// The sweep now reopens its own connection after a failed attempt. +/// +[Collection("live-postgres")] +public sealed class RetiredBaselineSweepRecoveryLiveTests +{ + /// + /// Deterministic: an event trigger raises XX000 from inside the cpu aggregate's DROP, which is the + /// error (and the broken connection) the refresh-job race produces, on demand. The file_io view behind it in + /// the sweep must still drop in the SAME pass, the failure must be logged once, and the connection must be + /// Open when the sweep returns. + /// + [Fact] + public async Task Sweep_ReopensItsConnection_AfterADropBreaksIt_AndDropsTheNextRelation_AgainstDevPostgres() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live sweep recovery test."); + + var ct = TestContext.Current.CancellationToken; + var bodySucceeded = false; + try + { + /* The sweep runs on the connection shape the product's service uses: the product search path, so an + unqualified TimescaleDB call in the sweep (alter_job) resolves on a fresh store too, where the extension + lives in collect and a pooled backend started before the migration has no path to it. */ + using var connection = new NpgsqlConnection(WithProductSearchPath(connectionString!)); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await CreateRetiredFixtureAsync(connection, ct); + + await ExecuteAsync(connection, @" +CREATE OR REPLACE FUNCTION collect.w5416_break_drop() RETURNS event_trigger LANGUAGE plpgsql AS $fn$ +BEGIN + IF EXISTS (SELECT 1 FROM pg_event_trigger_dropped_objects() WHERE object_name = 'cpu_utilization_baseline') + THEN + RAISE EXCEPTION 'planted #5416 drop failure' USING ERRCODE = 'XX000'; + END IF; +END +$fn$", ct); + await ExecuteAsync(connection, "CREATE EVENT TRIGGER w5416_break_drop ON sql_drop EXECUTE FUNCTION collect.w5416_break_drop()", ct); + + /* Sweep 1: cpu's drop fails and breaks the connection; file_io must still go. */ + var firstLog = new CapturingTestLogger(); + var dropped = await TimescaleSupport.DropRetiredBaselineAggregatesAsync(connection, firstLog, ct); + Assert.True(connection.State == ConnectionState.Open, + $"the sweep left its connection {connection.State} after a failed drop: {firstLog.Joined}"); + Assert.True(dropped == 1, $"the sweep dropped {dropped} relations, wanted 1 (file_io, behind the failed cpu drop): {firstLog.Joined}"); + Assert.Equal(1, firstLog.Lines.Count(l => l.StartsWith("Warning:", StringComparison.Ordinal) + && l.Contains("cpu_utilization_baseline", StringComparison.Ordinal))); + Assert.Equal(1, firstLog.CountAtLevel(LogLevel.Warning)); + Assert.True(await ScalarAsync(connection, "SELECT to_regclass('collect.file_io_baseline') IS NULL", ct), "file_io_baseline must be gone"); + Assert.True(await ScalarAsync(connection, "SELECT to_regclass('collect.cpu_utilization_baseline') IS NOT NULL", ct), "cpu_utilization_baseline survives the failed drop"); + + /* The same connection object keeps working afterwards: the retry on the next pass. */ + await ExecuteAsync(connection, "DROP EVENT TRIGGER w5416_break_drop", ct); + var secondLog = new CapturingTestLogger(); + Assert.Equal(1, await TimescaleSupport.DropRetiredBaselineAggregatesAsync(connection, secondLog, ct)); + Assert.Equal(0, secondLog.CountAtLevel(LogLevel.Warning)); + Assert.True(await ScalarAsync(connection, "SELECT to_regclass('collect.cpu_utilization_baseline') IS NULL", ct), "cpu_utilization_baseline must go on the retry"); + Assert.Equal(0, await TimescaleSupport.DropRetiredBaselineAggregatesAsync(connection, null, ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, bodySucceeded, async (cleanup, cleanupCt) => + { + await ExecuteAsync(cleanup, "DROP EVENT TRIGGER IF EXISTS w5416_break_drop", cleanupCt); + await ExecuteAsync(cleanup, "DROP FUNCTION IF EXISTS collect.w5416_break_drop()", cleanupCt); + await ExecuteAsync(cleanup, TimescaleSupport.DropRetiredBaselineRelationSql("cpu_utilization_baseline"), cleanupCt); + await ExecuteAsync(cleanup, TimescaleSupport.DropRetiredBaselineRelationSql("file_io_baseline"), cleanupCt); + }); + } + } + + /// + /// A refresh job in flight when the sweep's drop lands: the drop loses, the connection must be Open when the + /// sweep returns, and the retry loop must finish the job once the refresh is done. + /// + /// #5549: this used to race a real CALL run_job against the drop, 40 times, with the policy also + /// due on the scheduler. On CI the client backend running that refresh died with 0xC0000005 while the drop was + /// running, and the postmaster restarted every process on the shared cluster, so the tests that were running + /// at the time failed too. A TimescaleDB job must not run while its aggregate is being dropped, so the race is + /// now staged instead. The refresh job still runs for real, to completion, before the sweep starts. Then the + /// runner holds what a refresh holds until it commits, ROW EXCLUSIVE on the materialization hypertable + /// (its DELETE and INSERT), in an open transaction. A lock_timeout on the test's own sweeper session makes + /// the first drop give up behind that lock instead of waiting it out (the limit is reset right after that attempt, + /// so the retries run with the server's own). That is the drop losing to the job with the connection still Open, + /// the shape of the deadlock arm (40P01), on every iteration. The broken-connection arm + /// (XX000) is pinned by the deterministic test above. The policy stays a day out, so the scheduler never + /// launches its own refresh beside the drop either. + /// + [Fact] + public async Task Sweep_LeavesItsConnectionOpen_WhenARefreshHoldsALockDuringTheDrop_AgainstDevPostgres() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live sweep race test."); + + var ct = TestContext.Current.CancellationToken; + var bodySucceeded = false; + try + { + using var setup = new NpgsqlConnection(connectionString); + await setup.OpenAsync(ct); + await PgMigrations.MigrateAsync(setup, ct); + + /* run_job is qualified with the extension's schema. A pooled connection opened before a fresh store's + migration has no search path to it, and the old test swallowed that 42883 with every other error, so on + a fresh database its refresh job never ran at all. */ + var runJob = await ScalarAsync(setup, + "SELECT format('%I.run_job', n.nspname) FROM pg_extension e JOIN pg_namespace n ON n.oid = e.extnamespace WHERE e.extname = 'timescaledb'", ct); + + for (var i = 0; i < 5; i++) + { + await CreateRetiredFixtureAsync(setup, ct); + var jobId = await ScalarAsync(setup, + "SELECT job_id FROM timescaledb_information.jobs WHERE proc_name = 'policy_refresh_continuous_aggregate' AND hypertable_schema = 'collect' AND hypertable_name = 'cpu_utilization_baseline'", ct); + var materialization = await ScalarAsync(setup, + "SELECT format('%I.%I', materialization_hypertable_schema, materialization_hypertable_name) FROM timescaledb_information.continuous_aggregates WHERE view_schema = 'collect' AND view_name = 'cpu_utilization_baseline'", ct); + + /* Opened with the product search path, like the service's own connections; run_job below stays qualified. */ + using var sweeper = new NpgsqlConnection(WithProductSearchPath(connectionString!)); + await sweeper.OpenAsync(ct); + using var runner = new NpgsqlConnection(WithProductSearchPath(connectionString!)); + await runner.OpenAsync(ct); + + /* The refresh runs for real and finishes before any drop starts (#5549). */ + await ExecuteAsync(runner, $"CALL {runJob}({jobId})", ct); + + /* The next refresh, in flight: the lock its materialization writes hold until it commits. */ + using var inFlight = await runner.BeginTransactionAsync(ct); + await ExecuteAsync(runner, $"LOCK TABLE {materialization} IN ROW EXCLUSIVE MODE", ct); + await ExecuteAsync(sweeper, "SET lock_timeout = '500ms'", ct); + + var log = new CapturingTestLogger(); + for (var attempt = 0; attempt < 5; attempt++) + { + await TimescaleSupport.DropRetiredBaselineAggregatesAsync(sweeper, log, ct); + Assert.True(sweeper.State == ConnectionState.Open, + $"iteration {i}: the connection is {sweeper.State} after sweep attempt {attempt + 1}: {log.Joined}"); + if (attempt == 0) + { + /* The drop lost to the job: lock_timeout (55P03), logged once, and the aggregate is still there. */ + Assert.True(await ScalarAsync(sweeper, "SELECT to_regclass('collect.cpu_utilization_baseline') IS NOT NULL", ct), + $"iteration {i}: the first sweep dropped the aggregate through the in-flight job's lock: {log.Joined}"); + Assert.Equal(1, log.Lines.Count(l => l.StartsWith("Warning:", StringComparison.Ordinal) + && l.Contains("cpu_utilization_baseline", StringComparison.Ordinal) + && l.Contains("55P03", StringComparison.Ordinal))); + /* The 500 ms limit was for the attempt that has to lose. Later attempts run with the server's own + lock_timeout, so autovacuum or another job holding a lock briefly does not fail the test. */ + await ExecuteAsync(sweeper, "RESET lock_timeout", ct); + await inFlight.CommitAsync(ct); + } + + if (await ScalarAsync(sweeper, "SELECT to_regclass('collect.cpu_utilization_baseline') IS NULL AND to_regclass('collect.file_io_baseline') IS NULL", ct)) + { + break; + } + + await Task.Delay(TimeSpan.FromMilliseconds(250), ct); + } + + Assert.True(await ScalarAsync(sweeper, "SELECT to_regclass('collect.cpu_utilization_baseline') IS NULL AND to_regclass('collect.file_io_baseline') IS NULL", ct), + $"iteration {i}: the retired relations survived five sweep attempts: {log.Joined}"); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, bodySucceeded, async (cleanup, cleanupCt) => + { + await ExecuteAsync(cleanup, TimescaleSupport.DropRetiredBaselineRelationSql("cpu_utilization_baseline"), cleanupCt); + await ExecuteAsync(cleanup, TimescaleSupport.DropRetiredBaselineRelationSql("file_io_baseline"), cleanupCt); + }); + } + } + + /// The connection string with Search Path set to the product's (), as DarlingWorker sets it. + private static string WithProductSearchPath(string connectionString) => + new NpgsqlConnectionStringBuilder(connectionString) { SearchPath = PgSchemaGenerator.SearchPath }.ConnectionString; + + /// The pre-#2007 shapes the sweep retires: cpu as a continuous aggregate with its policies, file_io as a plain view. + private static async Task CreateRetiredFixtureAsync(NpgsqlConnection connection, CancellationToken ct) + { + await ExecuteAsync(connection, + "SELECT create_hypertable('collect.cpu_utilization_stats', by_range('collection_time', INTERVAL '1 day'), if_not_exists => true, migrate_data => true)", ct); + await ExecuteAsync(connection, @" +CREATE MATERIALIZED VIEW IF NOT EXISTS collect.cpu_utilization_baseline +WITH (timescaledb.continuous, timescaledb.materialized_only = false) AS +SELECT server_id, time_bucket('1 hour', collection_time) AS bucket, collection_time, + sum(sqlserver_cpu_utilization) AS cpu_sum, + sum(power(sqlserver_cpu_utilization, 2)) AS cpu_sumsq, + count(sqlserver_cpu_utilization) AS cpu_count +FROM collect.cpu_utilization_stats +GROUP BY server_id, bucket, collection_time +WITH NO DATA", ct); + /* The refresh policy of this fixture's aggregate is parked a day out in both tests, so the scheduler never launches + that policy's refresh while the sweep is dropping the aggregate (#5549). It stops only this policy: the other + jobs the migration scheduled in the test database still run on their own schedules. */ + const string initialStart = "now() + INTERVAL '1 day'"; + await ExecuteAsync(connection, + $"SELECT add_continuous_aggregate_policy('collect.cpu_utilization_baseline', start_offset => INTERVAL '3 days', end_offset => INTERVAL '1 hour', schedule_interval => INTERVAL '1 hour', initial_start => {initialStart}, if_not_exists => true)", ct); + await ExecuteAsync(connection, + "CREATE OR REPLACE VIEW collect.file_io_baseline AS SELECT server_id, date_trunc('hour', collection_time) AS bucket, collection_time, count(*) AS row_count FROM collect.file_io_stats GROUP BY 1, 2, 3", ct); + } + + private static async Task ScalarAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + using var command = new NpgsqlCommand(sql, connection) { CommandTimeout = 120 }; + var value = await command.ExecuteScalarAsync(ct); + Assert.NotNull(value); + return (T)Convert.ChangeType(value, typeof(T), System.Globalization.CultureInfo.InvariantCulture)!; + } + + private static async Task ExecuteAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + using var command = new NpgsqlCommand(sql, connection) { CommandTimeout = 120 }; + await command.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/RightSizingWindowTests.cs b/Darling/Darling.Tests/RightSizingWindowTests.cs index afee1f81e5..af630e2c4d 100644 --- a/Darling/Darling.Tests/RightSizingWindowTests.cs +++ b/Darling/Darling.Tests/RightSizingWindowTests.cs @@ -43,24 +43,32 @@ public void Describe_RendersTheCountAndSpan_CapsAtSevenDays_NeverTheLast(long co [InlineData("ViewerDataService.FinOps.Recommendations.cs")] public void ViewerRightSizing_NamesTheCoveredWindow_AndSkipsAnUnchangedSize(string file) { - var source = CSharpSourceWalker.StripCommentsAndStrings( - RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", file)); + // The viewer keeps the reads and the window text; the thresholds and the advice wording moved to Darling.Storage. var raw = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", file); + var storageRaw = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "FinOps", "FinOpsRecommendationFigures.cs"); + var source = CSharpSourceWalker.StripCommentsAndStrings(storageRaw); + var readerRaw = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "FinOps", "DarlingFinOpsRecommendationsReader.cs"); + Assert.DoesNotContain("\"the last ", readerRaw, StringComparison.Ordinal); Assert.DoesNotContain("\"the last ", raw, StringComparison.Ordinal); - Assert.Contains("From {cpuWindow}, P95 CPU utilization was", raw, StringComparison.Ordinal); - Assert.Contains("P95 SQL Server memory from {window} is", raw, StringComparison.Ordinal); - Assert.Contains("P95 SQL Server memory from {memWindow} is", raw, StringComparison.Ordinal); + Assert.DoesNotContain("\"the last ", storageRaw, StringComparison.Ordinal); + Assert.Contains("From {cpuWindow}, P95 CPU utilization was", storageRaw, StringComparison.Ordinal); + Assert.Contains("P95 SQL Server memory from {window} is", storageRaw, StringComparison.Ordinal); + Assert.Contains("P95 SQL Server memory from {memWindow} is", storageRaw, StringComparison.Ordinal); + Assert.DoesNotContain("memory over 7 days is {p95MemMb", storageRaw, StringComparison.Ordinal); + Assert.DoesNotContain("memory over 7 days is {p95Mb", storageRaw, StringComparison.Ordinal); Assert.DoesNotContain("memory over 7 days is {p95MemMb", raw, StringComparison.Ordinal); Assert.DoesNotContain("memory over 7 days is {p95Mb", raw, StringComparison.Ordinal); - Assert.Contains("RightSizingWindow.Describe(", raw, StringComparison.Ordinal); + Assert.DoesNotContain("memory over 7 days is {p95MemMb", readerRaw, StringComparison.Ordinal); + Assert.DoesNotContain("memory over 7 days is {p95Mb", readerRaw, StringComparison.Ordinal); + Assert.Contains("RightSizingWindow.Describe(", readerRaw, StringComparison.Ordinal); Assert.Matches(new Regex(@"targetMb\s*/\s*1024\s*<\s*physMb\s*/\s*1024"), source); Assert.Matches(new Regex(@"memRatio\s*<\s*0\.50m\s*&&\s*targetMb\s*/\s*1024\s*<\s*util\.PhysicalMemoryMb\s*/\s*1024"), source); Assert.Matches(new Regex(@"targetCores\s*>\s*0\s*&&\s*targetCores\s*<\s*cpuCount"), source); Assert.DoesNotMatch(new Regex(@"targetMb\s*<\s*physMb\s*&&"), source); - Assert.Contains("under 3ms across {storageWindow}", raw, StringComparison.Ordinal); - Assert.DoesNotContain("under 3ms over 7 days", raw, StringComparison.Ordinal); - Assert.Contains("HasQueryStatsCoverageAsync(", raw, StringComparison.Ordinal); + Assert.Contains("under 3ms across {storageWindow}", storageRaw, StringComparison.Ordinal); + Assert.DoesNotContain("under 3ms over 7 days", storageRaw, StringComparison.Ordinal); + Assert.Contains("HasQueryStatsCoverageAsync(", readerRaw, StringComparison.Ordinal); } [Theory] @@ -78,30 +86,37 @@ public void WindowedReads_SelectTheSamplesOwnFirstAndLastTime(string sqlName) [Fact] public void Readers_TakeTheWindowFromThoseColumns_AndTheCpuFallbackStaysNeutral() { - var raw = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.FinOps.Recommendations.cs"); + var raw = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "FinOps", "DarlingFinOpsRecommendationsReader.cs"); Assert.Matches(new Regex(@"reader\.IsDBNull\(4\)\s*\?\s*0L\s*:\s*Convert\.ToInt64\(reader\.GetValue\(4\),\s*CultureInfo\.InvariantCulture\),\s*reader\.IsDBNull\(2\)\s*\|\|\s*reader\.IsDBNull\(3\)\s*\?\s*TimeSpan\.Zero\s*:\s*reader\.GetDateTime\(3\)\s*-\s*reader\.GetDateTime\(2\)"), raw); Assert.Matches(new Regex(@"cpuReader\.IsDBNull\(3\)\s*\?\s*0L\s*:\s*Convert\.ToInt64\(cpuReader\.GetValue\(3\),\s*CultureInfo\.InvariantCulture\),\s*cpuReader\.IsDBNull\(1\)\s*\|\|\s*cpuReader\.IsDBNull\(2\)\s*\?\s*TimeSpan\.Zero\s*:\s*cpuReader\.GetDateTime\(2\)\s*-\s*cpuReader\.GetDateTime\(1\)"), raw); Assert.Contains("var cpuWindow = \"recent samples\"", raw, StringComparison.Ordinal); - Assert.Contains("storageSamples += reader.IsDBNull(7)", raw, StringComparison.Ordinal); + Assert.Contains("reader.IsDBNull(7) ? 0L : Convert.ToInt64(reader.GetValue(7)", raw, StringComparison.Ordinal); + var storageRaw = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "FinOps", "FinOpsRecommendationFigures.cs"); + Assert.Contains("storageSamples += row.WindowSamples", storageRaw, StringComparison.Ordinal); } [Fact] - public void QueryStatsFirstSampleSql_IsOneScalarOverTheServersCutoffWindow() + public void IdleCoverageSql_CountsTheDistinctUtcDaysWithAQueryStatsSample() { - var sql = ViewerDataService.RecommendationsQueryStatsFirstSampleSql; - Assert.Contains("SELECT MIN(collection_time)", sql, StringComparison.Ordinal); + var sql = ViewerDataService.RecommendationsIdleCoverageSql; + /* One EXISTS probe per complete UTC day (#5492), the day starts D-7 .. D-1: $2 through $3 - 1 day. */ + Assert.Contains("generate_series(CAST($2 AS timestamp), CAST($3 AS timestamp) - INTERVAL '1 day', INTERVAL '1 day')", sql, StringComparison.Ordinal); Assert.Contains("FROM v_query_stats", sql, StringComparison.Ordinal); - Assert.Contains("$1", sql, StringComparison.Ordinal); - Assert.DoesNotContain("$2", sql, StringComparison.Ordinal); - Assert.DoesNotContain("collection_time >=", sql, StringComparison.Ordinal); + Assert.Contains("q.server_id = $1", sql, StringComparison.Ordinal); + Assert.Contains("q.collection_time >= d.day_start", sql, StringComparison.Ordinal); + /* Each day stops at its own end, and the oldest sample is read beside them (the rule needs both). */ + Assert.Contains("q.collection_time < d.day_start + INTERVAL '1 day'", sql, StringComparison.Ordinal); + Assert.Contains("ORDER BY collection_time LIMIT 1", sql, StringComparison.Ordinal); } [Fact] public void QueryStatsCoverage_IsExactlyTheSevenDayCutoff_WithNoSlack() { - var raw = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.FinOps.Recommendations.cs"); - Assert.Contains("firstSample <= cutoff", raw, StringComparison.Ordinal); + var raw = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "FinOps", "DarlingFinOpsRecommendationsReader.cs"); + var viewerRaw = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.FinOps.Recommendations.cs"); + Assert.DoesNotContain("firstSample <= cutoff", raw, StringComparison.Ordinal); + Assert.DoesNotContain("TimeSpan.FromDays(1)", viewerRaw, StringComparison.Ordinal); Assert.DoesNotContain("TimeSpan.FromDays(1)", raw, StringComparison.Ordinal); } } diff --git a/Darling/Darling.Tests/RollupBackfillLiveTests.cs b/Darling/Darling.Tests/RollupBackfillLiveTests.cs index ed66237e1e..554b380934 100644 --- a/Darling/Darling.Tests/RollupBackfillLiveTests.cs +++ b/Darling/Darling.Tests/RollupBackfillLiveTests.cs @@ -66,7 +66,7 @@ public async Task RollupCreatedOverExistingHistory_ReadsEmptyBelowItsFloor_Until await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); /* All timestamps Kind-Unspecified — naive-UTC storage, see PgCollectorRowWriter. */ - var now = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + var now = LiveClock.Now(); var rawOldest = now.AddDays(-HistoryDays); /* ── 1. PRE-EXISTING HISTORY: raw rows going back well before any rollup exists. ── */ @@ -234,7 +234,7 @@ public async Task HealthyStore_WhereRawIsShallowerThanTheRollup_DoesNotFallBackT Assert.True(await TimescaleSupport.TryEnableAsync(connection, null, ct)); await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); - var now = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + var now = LiveClock.Now(); /* Plant history, materialize ALL of it, then purge raw back to a short horizon — the steady state a healthy store reaches once its purges are armed. */ @@ -304,7 +304,7 @@ public async Task BackfillRollupsVerb_DryRunsThenBackfills_AndReportsCoverageItA Assert.True(await TimescaleSupport.TryEnableAsync(setup, null, ct)); await TimescaleSupport.ConvertToHypertablesAsync(setup, null, ct); - var seedNow = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + var seedNow = LiveClock.Now(); await SeedHourlyQueryStatsAsync(setup, seedNow.AddDays(-HistoryDays), seedNow, ct); await TimescaleSupport.EnsureContinuousAggregatesAsync(setup, null, ct); } @@ -431,7 +431,7 @@ public async Task BackfillRollupsVerb_ConvergesTheThreeLevelQueryStoreChain_Incl Assert.True(await TimescaleSupport.TryEnableAsync(setup, null, ct)); await TimescaleSupport.ConvertToHypertablesAsync(setup, null, ct); - var seedNow = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + var seedNow = LiveClock.Now(); await SeedHourlyQueryStoreStatsAsync(setup, seedNow.AddDays(-HistoryDays), seedNow, ct); await TimescaleSupport.EnsureContinuousAggregatesAsync(setup, null, ct); } @@ -519,7 +519,7 @@ public async Task Probe_CalibrationSampleCountsBuckets_NotRows() Assert.True(await TimescaleSupport.TryEnableAsync(connection, null, ct)); await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); - var now = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + var now = LiveClock.Now(); await SeedHourlyQueryStatsAsync(connection, now.AddDays(-HistoryDays), now, ct); await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, ct); @@ -586,7 +586,7 @@ public async Task MidSliceCancellation_LeavesTheFloorInsideTheRange_WithNoGapsAb /* Wide enough that the raw hypertable holds several chunks, so that one in the MIDDLE of the range exists to hold the refresh at. */ - var now = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + var now = LiveClock.Now(); var rangeFrom = now.Date.AddDays(-CancellationHistoryDays); var rangeTo = now.Date; @@ -654,6 +654,7 @@ returned rather than dropped so that a refresh which failed BEFORE it was held c }, CancellationToken.None); + var cancelDidNotStopTheRefresh = false; try { await WaitUntilBackendWaitsOnAsync(connection, refreshConnection.ProcessID, heldChunk, refresh, ct); @@ -663,7 +664,19 @@ returned rather than dropped so that a refresh which failed BEFORE it was held c try { await cancellation.CancelAsync(); - _ = await refresh; + + /* #5587: bounded. The lock above is still held here, and the refresh's own command timeout is an hour, + so a cancel that did not reach the backend would leave this wait with nothing to end it (a CI shard + hung to its job timeout on a wait of this shape). On the limit the lock is released below, the refresh + is let finish, and the test fails by name. */ + try + { + _ = await refresh.WaitAsync(CancelledRefreshLimit, CancellationToken.None); + } + catch (TimeoutException) + { + cancelDidNotStopTheRefresh = true; + } /* Release the cancelled backend BEFORE anything else touches this store. `await refresh` returns when the client-side task completes, which is not the same instant the server-side backend finishes @@ -672,7 +685,10 @@ Closing explicitly removes that window rather than relying on disposal order to the same lifecycle-hardening the arming tests got: the shared-store flake class on this rig is connection-level, so a test that deliberately aborts a statement should not leave the cleanup to chance. */ - await refreshConnection.CloseAsync(); + if (!cancelDidNotStopTheRefresh) + { + await refreshConnection.CloseAsync(); + } } finally { @@ -682,6 +698,14 @@ Closing explicitly removes that window rather than relying on disposal order to } } + if (cancelDidNotStopTheRefresh) + { + _ = await refresh; + Assert.Fail( + $"the cancelled refresh was still waiting on the held chunk {heldChunk} {CancelledRefreshLimit.TotalSeconds:0} s after the cancel, " + + "so the cancel did not reach its backend. The lock was released so the refresh could end."); + } + var floor = await RollupBackfill.ReadCoverageFloorAsync(connection, view, ct); Assert.True(floor is not null, @@ -729,6 +753,9 @@ proved nothing. */ /// seconds, so reaching this means the refresh is stuck somewhere else. private static readonly TimeSpan HeldChunkWaitLimit = TimeSpan.FromSeconds(120); + /// #5587: how long a cancelled refresh may take to end while the chunk lock is still held; a cancel that works ends it in milliseconds. + private static readonly TimeSpan CancelledRefreshLimit = TimeSpan.FromSeconds(60); + /// /// The raw collect.query_stats chunk whose time range holds , as a /// schema-qualified, quoted name that both LOCK TABLE and ::regclass accept. Found by RANGE in @@ -836,7 +863,7 @@ public async Task InterruptedBackfill_DoesNotLookComplete_AndTheArmingGateRefuse Assert.True(await TimescaleSupport.TryEnableAsync(connection, null, ct)); await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); - var now = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + var now = LiveClock.Now(); await SeedHourlyQueryStatsAsync(connection, now.AddDays(-HistoryDays), now, ct); await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, ct); @@ -953,7 +980,7 @@ public async Task BackfillRollupsVerb_RefusesAnAbsurdPlan_WithoutClaimingSuccess Assert.True(await TimescaleSupport.TryEnableAsync(setup, null, ct)); await TimescaleSupport.ConvertToHypertablesAsync(setup, null, ct); - var seedNow = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + var seedNow = LiveClock.Now(); await SeedHourlyQueryStatsAsync(setup, seedNow.AddDays(-2), seedNow, ct); /* ONE corrupt row, which is all it takes. */ @@ -1079,7 +1106,7 @@ public async Task HierarchicalDaily_ConvergesToItsHourly_NotToRaw_SoTheHourlyTie Assert.True(await TimescaleSupport.TryEnableAsync(connection, null, ct)); await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); - var now = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + var now = LiveClock.Now(); var historyFrom = now.Date.AddDays(-HistoryDays); await SeedHourlyQueryStatsAsync(connection, historyFrom, now, ct); diff --git a/Darling/Darling.Tests/RollupCoverageCeilingLiveTests.cs b/Darling/Darling.Tests/RollupCoverageCeilingLiveTests.cs index c3846eaed9..271dc0ae4b 100644 --- a/Darling/Darling.Tests/RollupCoverageCeilingLiveTests.cs +++ b/Darling/Darling.Tests/RollupCoverageCeilingLiveTests.cs @@ -112,8 +112,21 @@ RollupFloorMaxReuse documents. ── */ Assert.NotNull(laterWatermark); Assert.True(laterWatermark > engineWatermark, "the second refresh must move the engine's own watermark later"); + /* #4957: past the hour with the oldest chunk unchanged, the caller is served the cached floor AND ceiling and one + background re-measure replaces them, so the first call after the hour still reads the old ceiling and the + ceiling advances once that re-measure lands. (Before #4957 this call measured inline.) */ var pastTtl = now + TimescaleSupport.RollupFloorMaxReuse + TimeSpan.FromMinutes(1); - var secondCall = await TimescaleSupport.DetectRollupCoverageAsync(dataSource, availability, pastTtl, ct); + var servedFromCache = await TimescaleSupport.DetectRollupCoverageAsync(dataSource, availability, pastTtl, ct); + Assert.Equal(firstCall.CeilingOf(view), servedFromCache.CeilingOf(view)); + + var secondCall = servedFromCache; + var deadline = DateTime.UtcNow.AddSeconds(30); + while (secondCall.CeilingOf(view) != laterWatermark && DateTime.UtcNow < deadline) + { + await Task.Delay(50, ct); + secondCall = await TimescaleSupport.DetectRollupCoverageAsync(dataSource, availability, pastTtl, ct); + } + Assert.Equal(laterWatermark, secondCall.CeilingOf(view)); Assert.True(secondCall.CeilingOf(view) > firstCall.CeilingOf(view), "the ceiling must advance once the rollup materializes further"); diff --git a/Darling/Darling.Tests/RollupCoverageRoutingTests.cs b/Darling/Darling.Tests/RollupCoverageRoutingTests.cs index c464191cb2..89ab2774ed 100644 --- a/Darling/Darling.Tests/RollupCoverageRoutingTests.cs +++ b/Darling/Darling.Tests/RollupCoverageRoutingTests.cs @@ -12,6 +12,7 @@ using System.Linq; using System.Text.RegularExpressions; using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Service.Mcp; using Xunit; namespace Darling.Tests; @@ -202,7 +203,7 @@ public void CoverageProbe_CoversEveryRoutedViewAndEveryRolledRawTable() TimescaleSupport.QueryStoreStatsHourlyView, TimescaleSupport.QueryStoreStatsDailyView, }) { - Assert.Contains($"(SELECT min(bucket) FROM collect.{view})", sql, StringComparison.Ordinal); + Assert.Contains($"(SELECT {TimescaleSupport.ColdFloorMarker} min(bucket) FROM collect.{view})", sql, StringComparison.Ordinal); /* And every routed view must resolve to a raw table, or its ladder would have no floor to fall to. */ Assert.NotNull(RollupCoverage.RawTableFor(view)); @@ -747,4 +748,98 @@ private static string StripComments(string source) return null; } + + /// + /// #5329: the routing the Top Queries reader derives from what the store measured, with the io rollup ABSENT (not on + /// the store, even with a stale floor cached for its name), EMPTY (present, no floor), PARTIAL (a floor after the + /// window's start) and COVERING (a floor at or before it). The window starts ten days back and raw keeps four. + /// + [Theory] + [InlineData("absent", TopRanking.Reads, HourlyRoute.Raw)] + [InlineData("absent", TopRanking.Cpu, HourlyRoute.Stitched)] + [InlineData("empty", TopRanking.Reads, HourlyRoute.Raw)] + [InlineData("empty", TopRanking.Duration, HourlyRoute.Stitched)] + [InlineData("partial", TopRanking.Reads, HourlyRoute.Io)] // io (6 days) reaches further back than raw (4 days) + [InlineData("partial", TopRanking.Executions, HourlyRoute.Stitched)] + [InlineData("covering", TopRanking.Reads, HourlyRoute.Io)] + [InlineData("covering", TopRanking.Cpu, HourlyRoute.Io)] + [InlineData("covering", TopRanking.Duration, HourlyRoute.Io)] + [InlineData("covering", TopRanking.Executions, HourlyRoute.Io)] + public void TheQueriesReader_RoutesOnTheIoRollupsAvailabilityAndFloor(string io, TopRanking ranking, HourlyRoute expected) + { + var floors = new Dictionary(StringComparer.Ordinal) + { + [TimescaleSupport.QueryStatsHourlyView] = DaysAgo(30), + }; + switch (io) + { + case "absent": + floors[TimescaleSupport.QueryStatsIoHourlyView] = DaysAgo(30); /* a stale floor under a name the store does not have */ + break; + case "partial": + floors[TimescaleSupport.QueryStatsIoHourlyView] = DaysAgo(6); + break; + case "covering": + floors[TimescaleSupport.QueryStatsIoHourlyView] = DaysAgo(30); + break; + } + + var availability = io == "absent" ? RollupAvailability.WithoutIoHourlies : RollupAvailability.All; + var coverage = new RollupCoverage( + floors, + new Dictionary(StringComparer.Ordinal) { ["query_stats"] = DaysAgo(4) }, + availability); + + Assert.Equal(expected, DarlingDataReader.ChooseQueriesHourlyRoute(availability, coverage, ranking, DaysAgo(10))); + } + + /// + /// #5329 lane B2: the same table for the Top Procedures reader, over procedure_stats_io_hourly and + /// procedure_stats's oldest row. Each case also plants the QUERIES io floor the other way round, so a procedures + /// route that read the queries relation's floor would answer differently and fail. + /// + [Theory] + [InlineData("absent", TopRanking.Reads, HourlyRoute.Raw)] + [InlineData("absent", TopRanking.Cpu, HourlyRoute.Stitched)] + [InlineData("empty", TopRanking.Reads, HourlyRoute.Raw)] + [InlineData("empty", TopRanking.Duration, HourlyRoute.Stitched)] + [InlineData("partial", TopRanking.Reads, HourlyRoute.Io)] // io (6 days) reaches further back than raw (4 days) + [InlineData("partial", TopRanking.Executions, HourlyRoute.Stitched)] + [InlineData("partialShallow", TopRanking.Reads, HourlyRoute.Raw)] // raw (4 days) reaches further back than io (2 days) + [InlineData("covering", TopRanking.Reads, HourlyRoute.Io)] + [InlineData("covering", TopRanking.Cpu, HourlyRoute.Io)] + [InlineData("covering", TopRanking.Duration, HourlyRoute.Io)] + [InlineData("covering", TopRanking.Executions, HourlyRoute.Io)] + public void TheProceduresReader_RoutesOnTheIoRollupsAvailabilityAndFloor(string io, TopRanking ranking, HourlyRoute expected) + { + var floors = new Dictionary(StringComparer.Ordinal) + { + [TimescaleSupport.ProcedureStatsHourlyView] = DaysAgo(30), + /* the queries io rollup says the opposite of every case below: covering, where procedures is not */ + [TimescaleSupport.QueryStatsIoHourlyView] = DaysAgo(30), + }; + switch (io) + { + case "absent": + floors[TimescaleSupport.ProcedureStatsIoHourlyView] = DaysAgo(30); /* a stale floor under a name the store does not have */ + break; + case "partial": + floors[TimescaleSupport.ProcedureStatsIoHourlyView] = DaysAgo(6); + break; + case "partialShallow": + floors[TimescaleSupport.ProcedureStatsIoHourlyView] = DaysAgo(2); + break; + case "covering": + floors[TimescaleSupport.ProcedureStatsIoHourlyView] = DaysAgo(30); + break; + } + + var availability = io == "absent" ? RollupAvailability.WithoutIoHourlies : RollupAvailability.All; + var coverage = new RollupCoverage( + floors, + new Dictionary(StringComparer.Ordinal) { ["procedure_stats"] = DaysAgo(4), ["query_stats"] = DaysAgo(4) }, + availability); + + Assert.Equal(expected, DarlingDataReader.ChooseProceduresHourlyRoute(availability, coverage, ranking, DaysAgo(10))); + } } diff --git a/Darling/Darling.Tests/RollupCoverageWarmupTests.cs b/Darling/Darling.Tests/RollupCoverageWarmupTests.cs new file mode 100644 index 0000000000..98dfda5936 --- /dev/null +++ b/Darling/Darling.Tests/RollupCoverageWarmupTests.cs @@ -0,0 +1,50 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System.Text.RegularExpressions; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4957: where the rollup-floor warm is launched. The warm itself runs against a store (see +/// RollupFloorCacheLiveTests); these pin that the two processes that pay the cold sort inline actually start it — +/// the service after its migrations, the Viewer once its store has answered — and that neither waits for it. +/// +public sealed class RollupCoverageWarmupTests +{ + [Fact] + public void TheWorker_LaunchesTheCoverageWarmAfterMigrations_WithoutAwaitingItOnTheStartupPath() + { + var source = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "DarlingWorker.cs").Replace("\r\n", "\n"); + + var migrate = source.IndexOf("PgMigrations.MigrateAsync(migrateConnection", System.StringComparison.Ordinal); + var launch = source.IndexOf("var rollupCoverageWarm = RollupCoverageWarmup.RunDelayedAsync(", System.StringComparison.Ordinal); + var drain = source.IndexOf("await rollupCoverageWarm;", System.StringComparison.Ordinal); + var loopStop = source.IndexOf("PerformanceMonitor Darling collection loop stopped", System.StringComparison.Ordinal); + + Assert.True(migrate > 0 && launch > migrate, "the warm must launch after migrations"); + Assert.True(drain > launch && drain < loopStop, "the warm is drained only at shutdown, after the collection loop"); + Assert.Single(Regex.Matches(source, @"await rollupCoverageWarm;")); + Assert.Contains("RollupCoverageWarmup.ServiceStartDelay", source); + } + + [Fact] + public void TheViewer_WarmsTheCoverageOnceItsStoreHasAnswered_AndDoesNotWaitForIt() + { + var main = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "MainWindow.xaml.cs").Replace("\r\n", "\n"); + var readOnlyProbe = main.IndexOf("await _dataService.DetectReadOnlyAsync();", System.StringComparison.Ordinal); + var warm = main.IndexOf("_ = _dataService.WarmRollupCoverageAsync(", System.StringComparison.Ordinal); + + Assert.True(readOnlyProbe > 0 && warm > readOnlyProbe, "the Viewer warms only after the store answered its reachability, schema and seat probes"); + + var dataService = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.RollupAvailability.cs").Replace("\r\n", "\n"); + Assert.Contains("RollupCoverageWarmup.RunDelayedAsync(_dataSource", dataService); + Assert.Contains("RollupCoverageWarmup.ViewerStartDelay", dataService); + } +} diff --git a/Darling/Darling.Tests/RollupFloorCacheLiveTests.cs b/Darling/Darling.Tests/RollupFloorCacheLiveTests.cs index 80c613b529..f0e9f5c1ac 100644 --- a/Darling/Darling.Tests/RollupFloorCacheLiveTests.cs +++ b/Darling/Darling.Tests/RollupFloorCacheLiveTests.cs @@ -8,6 +8,7 @@ using System; using System.Collections.Generic; +using System.Linq; using System.Threading; using System.Threading.Tasks; using Npgsql; @@ -57,7 +58,7 @@ public async Task DetectRollupCoverageAsync_ReusesACompressedChunksFloor_UntilTh await stop.ExecuteNonQueryAsync(ct); } - var now = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + var now = LiveClock.Now(); var oldest = now.AddDays(-3); /* ── 1. Plant three days of collect.query_store_stats and materialize the interval-honest hourly @@ -132,9 +133,11 @@ on every call. ── */ var floorSecond = secondCall.FloorOf(TimescaleSupport.QueryStoreStatsIntervalHourlyView); Assert.Equal(floorFirst, floorSecond); - var minBucketExecutions = loggerFactory.Provider.CountContaining( - $"min(bucket) FROM collect.{TimescaleSupport.QueryStoreStatsIntervalHourlyView}"); + /* #5329: counted by the probes' fixed markers (ColdFloorMarker, EarlierFloorMarker), not by how a statement spells + min(bucket): one cold probe across both calls, and the warm call's bounded earlier-floor read exactly once. */ + var minBucketExecutions = loggerFactory.Provider.CountContaining(MinBucketStatement); Assert.Equal(1, minBucketExecutions); + Assert.Equal(1, loggerFactory.Provider.CountContaining(EarlierFloorStatement)); /* ── (c) drop_chunks the oldest materialization chunk: the floor must RE-measure and move later. drop_chunks is called against the AGGREGATE VIEW (the only relation name that stays stable @@ -160,6 +163,358 @@ drop_chunks is called against the AGGREGATE VIEW (the only relation name that st Assert.True(floorThird > floorSecond, "the cache must re-measure and move the floor later once the chunk it was keyed on is gone"); } + /* ──────────── #4957: one floor cache per STORE, warmed at start, re-measured past the hour in the background ──────────── */ + + private static readonly string HourlyView = TimescaleSupport.QueryStoreStatsIntervalHourlyView; + + /// The statement text whose executions the #4957 tests count: the hourly rollup's min(bucket). + private static readonly string MinBucketStatement = $"{TimescaleSupport.ColdFloorMarker} min(bucket) FROM collect.{TimescaleSupport.QueryStoreStatsIntervalHourlyView}"; + + /// The bounded earlier-floor probe's fixed marker (#5329); counted beside the cold one, so neither count leans on + /// how a statement spells min(bucket). + private static readonly string EarlierFloorStatement = TimescaleSupport.EarlierFloorMarker; + + private static string? BaseConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + /// The hour two UTC days before today at 02:00: inside ONE one-day materialization chunk with 22 buckets + /// after it, so a delete of its first two buckets can never empty that chunk (a 23:00 start would). + /* #5496: the start comes from the test's ONE clock reading (SeedEnd), never a second read: two reads straddling + midnight UTC gave the two stores of one test different seed starts. */ + private static DateTime SeedStart(DateTime now) => DateTime.SpecifyKind(now.Date.AddDays(-3).AddHours(2), DateTimeKind.Unspecified); + + private static DateTime SeedEnd() => LiveClock.Now(); + + /// #5496: the seed start is a pure function of the one instant the test read, so it can be proved on both sides of + /// midnight UTC without waiting for one. A second clock read would have moved the start a whole day when it crossed. + [Theory] + [InlineData(2026, 10, 6, 23, 59, 59, 2026, 10, 3)] + [InlineData(2026, 10, 7, 0, 0, 0, 2026, 10, 4)] + [InlineData(2026, 10, 7, 0, 0, 1, 2026, 10, 4)] + public void TheSeedStart_IsDerivedFromTheOneInstantItIsGiven(int y, int mo, int d, int h, int mi, int s, int ey, int emo, int ed) + { + var start = SeedStart(new DateTime(y, mo, d, h, mi, s, DateTimeKind.Unspecified)); + Assert.Equal(new DateTime(ey, emo, ed, 2, 0, 0), start); + } + + [Fact] + public async Task TwoDataSourcesOnOneStore_MeasureTheFloorOnce_AndTwoStoresNeverShareAFloor() + { + Assert.SkipWhen(string.IsNullOrEmpty(BaseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live #4957 per-store floor cache test (it mints its own scratch databases)."); + var ct = TestContext.Current.CancellationToken; + var now = SeedEnd(); + + await using var storeA = await ScratchPostgres.CreateAsync(BaseConnectionString!, ct); + await using var storeB = await ScratchPostgres.CreateAsync(BaseConnectionString!, ct); + await PopulateStoreAsync(storeA.ConnectionString, SeedStart(now), now, deleteRawBefore: null, oneDayChunks: false, ct); + await PopulateStoreAsync(storeB.ConnectionString, SeedStart(now), now, deleteRawBefore: SeedStart(now).AddHours(7), oneDayChunks: false, ct); + + var truthA = await TruthAsync(storeA.ConnectionString, ct); + var truthB = await TruthAsync(storeB.ConnectionString, ct); + Assert.NotNull(truthA); + Assert.NotNull(truthB); + Assert.True(truthB > truthA, "the two stores must hold different floors, or sharing one cache would go unnoticed"); + + var factoryA1 = new CommandCountingLoggerFactory(); + var factoryA2 = new CommandCountingLoggerFactory(); + var factoryB = new CommandCountingLoggerFactory(); + await using var dataSourceA1 = new NpgsqlDataSourceBuilder(storeA.ConnectionString).UseLoggerFactory(factoryA1).Build(); + await using var dataSourceA2 = new NpgsqlDataSourceBuilder(storeA.ConnectionString).UseLoggerFactory(factoryA2).Build(); + await using var dataSourceB = new NpgsqlDataSourceBuilder(storeB.ConnectionString).UseLoggerFactory(factoryB).Build(); + var availability = await TimescaleSupport.DetectRollupsAsync(dataSourceA1, ct); + Assert.True(availability.Has(HourlyView)); + + var viaA1 = await TimescaleSupport.DetectRollupCoverageAsync(dataSourceA1, availability, ct); + var viaA2 = await TimescaleSupport.DetectRollupCoverageAsync(dataSourceA2, availability, ct); + var viaB = await TimescaleSupport.DetectRollupCoverageAsync(dataSourceB, availability, ct); + + Assert.Equal(truthA, viaA1.FloorOf(HourlyView)); + Assert.Equal(truthA, viaA2.FloorOf(HourlyView)); + Assert.Equal(truthB, viaB.FloorOf(HourlyView)); + + /* The first data source on a store pays the sort; the second data source on the SAME store does not. */ + Assert.Equal(1, factoryA1.Provider.CountContaining(MinBucketStatement)); + Assert.Equal(0, factoryA2.Provider.CountContaining(MinBucketStatement)); + + /* A different store has its own floor: it measured for itself, and never read store A's. */ + Assert.Equal(1, factoryB.Provider.CountContaining(MinBucketStatement)); + } + + [Fact] + public async Task ADatabaseDroppedAndRecreatedUnderTheSameName_IsMeasuredAgain_NotServedItsOldFloor() + { + Assert.SkipWhen(string.IsNullOrEmpty(BaseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live #4957 recreated-database floor cache test (it mints its own scratch database)."); + var ct = TestContext.Current.CancellationToken; + var now = SeedEnd(); + + await using var store = await ScratchPostgres.CreateAsync(BaseConnectionString!, ct); + await PopulateStoreAsync(store.ConnectionString, SeedStart(now), now, deleteRawBefore: null, oneDayChunks: false, ct); + var truthBefore = await TruthAsync(store.ConnectionString, ct); + var oldestChunkBefore = await OldestChunkAsync(store.ConnectionString, ct); + + /* One data source across the drop: the scratch connection string does not pool, so each command opens a + fresh connection and the same data source reaches the second incarnation. */ + var factory = new CommandCountingLoggerFactory(); + await using var dataSource = new NpgsqlDataSourceBuilder(store.ConnectionString).UseLoggerFactory(factory).Build(); + var availability = await TimescaleSupport.DetectRollupsAsync(dataSource, ct); + var first = await TimescaleSupport.DetectRollupCoverageAsync(dataSource, availability, now, ct); + Assert.Equal(truthBefore, first.FloorOf(HourlyView)); + + /* Drop the database and build it again under the same name, laid out identically but with its oldest five + hours removed: the same chunk and hypertable names, a later floor. */ + await RecreateDatabaseAsync(BaseConnectionString!, store.DatabaseName, ct); + await PopulateStoreAsync(store.ConnectionString, SeedStart(now), now, deleteRawBefore: SeedStart(now).AddHours(5), oneDayChunks: false, ct); + var truthAfter = await TruthAsync(store.ConnectionString, ct); + Assert.True(truthAfter > truthBefore, "the recreated store's floor must be later, or a stale floor would go unnoticed"); + Assert.Equal(oldestChunkBefore, await OldestChunkAsync(store.ConnectionString, ct)); + + var second = await TimescaleSupport.DetectRollupCoverageAsync(dataSource, availability, now, ct); + + Assert.Equal(truthAfter, second.FloorOf(HourlyView)); + Assert.Equal(2, factory.Provider.CountContaining(MinBucketStatement)); + } + + [Fact] + public async Task APastTheHourFloorWithItsChunkUnchanged_IsServedFromTheCache_AndRemeasuredOnceInTheBackground_AChangedChunkIsMeasuredInline() + { + Assert.SkipWhen(string.IsNullOrEmpty(BaseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live #4957 background re-measure test (it mints its own scratch database)."); + var ct = TestContext.Current.CancellationToken; + var now = SeedEnd(); + + await using var store = await ScratchPostgres.CreateAsync(BaseConnectionString!, ct); + await PopulateStoreAsync(store.ConnectionString, SeedStart(now), now, deleteRawBefore: null, oneDayChunks: true, ct); + + var factory = new CommandCountingLoggerFactory(); + await using var dataSource = new NpgsqlDataSourceBuilder(store.ConnectionString).UseLoggerFactory(factory).Build(); + var availability = await TimescaleSupport.DetectRollupsAsync(dataSource, ct); + + var cold = await TimescaleSupport.DetectRollupCoverageAsync(dataSource, availability, now, ct); + var floorCold = cold.FloorOf(HourlyView); + Assert.Equal(await TruthAsync(store.ConnectionString, ct), floorCold); + Assert.Equal(1, factory.Provider.CountContaining(MinBucketStatement)); + + /* A retention delete inside the oldest chunk: the true floor moves later and the chunk's identity does not. */ + var oldestChunk = await OldestChunkAsync(store.ConnectionString, ct); + await DeleteMaterializedBucketsBeforeAsync(store.ConnectionString, floorCold!.Value.AddHours(2), ct); + var floorAfterDelete = await TruthAsync(store.ConnectionString, ct); + Assert.True(floorAfterDelete > floorCold); + Assert.Equal(oldestChunk, await OldestChunkAsync(store.ConnectionString, ct)); + + /* Past the hour: served from the cache (the floor as it stood), NOT measured inline. */ + var pastTheHour = now + TimescaleSupport.RollupFloorMaxReuse + TimeSpan.FromMinutes(1); + var served = await TimescaleSupport.DetectRollupCoverageAsync(dataSource, availability, pastTheHour, ct); + Assert.Equal(floorCold, served.FloorOf(HourlyView)); + + /* Several callers at once, while that re-measure runs or just after it: each gets the old or the new floor, never an error. */ + var burst = await Task.WhenAll(Enumerable.Range(0, 6).Select(_ => + Task.Run(() => TimescaleSupport.DetectRollupCoverageAsync(dataSource, availability, pastTheHour, ct), ct))); + Assert.All(burst, c => Assert.True(c.FloorOf(HourlyView) == floorCold || c.FloorOf(HourlyView) == floorAfterDelete)); + + /* The one background re-measure lands: callers see the new floor, and the seven past-the-hour callers + between them ran the sort once (plus the cold measure), not seven times. */ + var refreshed = await PollForFloorAsync(dataSource, availability, pastTheHour, floorAfterDelete, ct); + Assert.Equal(floorAfterDelete, refreshed.FloorOf(HourlyView)); + Assert.Equal(2, factory.Provider.CountContaining(MinBucketStatement)); + + /* A changed chunk is wrong, not merely old: dropping the oldest chunk is measured inline, on the calling thread, + even though this entry is past the hour too. */ + var (_, oldestRangeEnd) = await OldestChunkWithRangeEndAsync(store.ConnectionString, ct); + await DropChunksOlderThanAsync(store.ConnectionString, oldestRangeEnd, ct); + var floorAfterDrop = await TruthAsync(store.ConnectionString, ct); + Assert.True(floorAfterDrop > floorAfterDelete); + + var pastTheHourAgain = pastTheHour + TimescaleSupport.RollupFloorMaxReuse + TimeSpan.FromMinutes(1); + var inline = await TimescaleSupport.DetectRollupCoverageAsync(dataSource, availability, pastTheHourAgain, ct); + Assert.Equal(floorAfterDrop, inline.FloorOf(HourlyView)); + Assert.Equal(3, factory.Provider.CountContaining(MinBucketStatement)); + } + + [Fact] + public async Task AFloorPastTwiceTheHourWithItsChunkUnchanged_IsMeasuredInline_NotServedFromTheCache() + { + Assert.SkipWhen(string.IsNullOrEmpty(BaseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live #4957 serve-cap test (it mints its own scratch database)."); + var ct = TestContext.Current.CancellationToken; + var now = SeedEnd(); + + await using var store = await ScratchPostgres.CreateAsync(BaseConnectionString!, ct); + await PopulateStoreAsync(store.ConnectionString, SeedStart(now), now, deleteRawBefore: null, oneDayChunks: true, ct); + + var factory = new CommandCountingLoggerFactory(); + await using var dataSource = new NpgsqlDataSourceBuilder(store.ConnectionString).UseLoggerFactory(factory).Build(); + var availability = await TimescaleSupport.DetectRollupsAsync(dataSource, ct); + + var cold = await TimescaleSupport.DetectRollupCoverageAsync(dataSource, availability, now, ct); + var floorCold = cold.FloorOf(HourlyView); + Assert.Equal(await TruthAsync(store.ConnectionString, ct), floorCold); + Assert.Equal(1, factory.Provider.CountContaining(MinBucketStatement)); + + /* A retention delete inside the oldest chunk: the true floor moves later and the chunk's identity does not, + which is the one change the cache cannot see and the cap exists to bound. */ + var oldestChunk = await OldestChunkAsync(store.ConnectionString, ct); + await DeleteMaterializedBucketsBeforeAsync(store.ConnectionString, floorCold!.Value.AddHours(2), ct); + var floorAfterDelete = await TruthAsync(store.ConnectionString, ct); + Assert.True(floorAfterDelete > floorCold); + Assert.Equal(oldestChunk, await OldestChunkAsync(store.ConnectionString, ct)); + + /* At the serve cap the entry is no longer served: the first caller measures it inline and gets the true floor + in the same call, with the sort already run when it returns (no polling for a background pass). */ + var pastTheCap = now + TimescaleSupport.RollupFloorMaxServeAge + TimeSpan.FromMinutes(1); + var inline = await TimescaleSupport.DetectRollupCoverageAsync(dataSource, availability, pastTheCap, ct); + Assert.Equal(floorAfterDelete, inline.FloorOf(HourlyView)); + Assert.Equal(2, factory.Provider.CountContaining(MinBucketStatement)); + + /* It replaced the entry, so the next call is current and measures nothing. */ + var next = await TimescaleSupport.DetectRollupCoverageAsync(dataSource, availability, pastTheCap, ct); + Assert.Equal(floorAfterDelete, next.FloorOf(HourlyView)); + Assert.Equal(2, factory.Provider.CountContaining(MinBucketStatement)); + } + + [Fact] + public async Task AfterTheStartUpWarm_TheFirstCoverageCallOnAnotherDataSource_RunsNoMinBucket() + { + Assert.SkipWhen(string.IsNullOrEmpty(BaseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live #4957 start-up warm test (it mints its own scratch database)."); + var ct = TestContext.Current.CancellationToken; + var now = SeedEnd(); + + await using var store = await ScratchPostgres.CreateAsync(BaseConnectionString!, ct); + await PopulateStoreAsync(store.ConnectionString, SeedStart(now), now, deleteRawBefore: null, oneDayChunks: false, ct); + var truth = await TruthAsync(store.ConnectionString, ct); + + /* The service's own data source warms; the MCP or web host's data source is the first CALLER. */ + await using var workerSource = NpgsqlDataSource.Create(store.ConnectionString); + await RollupCoverageWarmup.RunDelayedAsync(workerSource, logger: null, TimeSpan.Zero, ct); + + var factory = new CommandCountingLoggerFactory(); + await using var callerSource = new NpgsqlDataSourceBuilder(store.ConnectionString).UseLoggerFactory(factory).Build(); + var availability = await TimescaleSupport.DetectRollupsAsync(callerSource, ct); + var coverage = await TimescaleSupport.DetectRollupCoverageAsync(callerSource, availability, ct); + + Assert.Equal(truth, coverage.FloorOf(HourlyView)); + Assert.Equal(0, factory.Provider.CountContaining(MinBucketStatement)); + } + + /// Builds a TimescaleDB store in the database names: migrated, hypertables, + /// the continuous aggregates, and the hourly Query Store rollup materialized over .. + /// — less any raw rows older than , which is how a second incarnation of the same + /// layout gets a later floor. + private static async Task PopulateStoreAsync( + string connectionString, DateTime from, DateTime to, DateTime? deleteRawBefore, bool oneDayChunks, CancellationToken ct) + { + await using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var timescaleEnabled = await LiveTimescaleProbe.TryEnableAsync(connectionString, ct); + Assert.True(timescaleEnabled, "TimescaleDB must be available on CI for the live #4957 floor cache tests"); + + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + + /* No background worker racing the fixture's own refresh calls below. */ + await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection)) + { + await stop.ExecuteNonQueryAsync(ct); + } + + await SeedHourlyQueryStoreStatsAsync(connection, from, to, ct); + if (deleteRawBefore is DateTime cut) + { + await using var trim = new NpgsqlCommand("DELETE FROM collect.query_store_stats WHERE collection_time < $1", connection); + trim.Parameters.AddWithValue(cut); + await trim.ExecuteNonQueryAsync(ct); + } + + await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, ct); + + if (oneDayChunks) + { + await using var setInterval = new NpgsqlCommand(TimescaleSupport.SetMaterializationChunkIntervalSql(HourlyView), connection); + await setInterval.ExecuteNonQueryAsync(ct); + } + + await using var refresh = new NpgsqlCommand(TimescaleSupport.RefreshContinuousAggregateSql(HourlyView, force: true), connection); + refresh.Parameters.AddWithValue(from); + await refresh.ExecuteNonQueryAsync(ct); + } + + private static async Task TruthAsync(string connectionString, CancellationToken ct) + { + await using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + return await ScalarInstantAsync(connection, $"SELECT min(bucket) FROM collect.{HourlyView}", ct); + } + + private static async Task OldestChunkAsync(string connectionString, CancellationToken ct) + => (await OldestChunkWithRangeEndAsync(connectionString, ct)).ChunkName; + + private static async Task<(string? ChunkName, DateTime RangeEnd)> OldestChunkWithRangeEndAsync(string connectionString, CancellationToken ct) + { + await using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + return await OldestMaterializationChunkAsync(connection, HourlyView, ct); + } + + private static async Task DeleteMaterializedBucketsBeforeAsync(string connectionString, DateTime before, CancellationToken ct) + { + await using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await using var find = new NpgsqlCommand($@" +SELECT format('%I.%I', materialization_hypertable_schema, materialization_hypertable_name) +FROM timescaledb_information.continuous_aggregates +WHERE view_schema = 'collect' AND view_name = '{HourlyView}'", connection); + var table = (string)(await find.ExecuteScalarAsync(ct))!; + + await using var delete = new NpgsqlCommand($"DELETE FROM {table} WHERE bucket < $1", connection); + delete.Parameters.AddWithValue(before); + await delete.ExecuteNonQueryAsync(ct); + } + + private static async Task DropChunksOlderThanAsync(string connectionString, DateTime olderThan, CancellationToken ct) + { + await using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await using var drop = new NpgsqlCommand($"SELECT drop_chunks('collect.{HourlyView}', older_than => $1::timestamp)", connection); + drop.Parameters.AddWithValue(olderThan); + await drop.ExecuteNonQueryAsync(ct); + } + + private static async Task RecreateDatabaseAsync(string baseConnectionString, string databaseName, CancellationToken ct) + { + /* #5549: unpooled, because this connection runs the drop and the create after it. */ + await using var admin = new NpgsqlConnection(ScratchPostgres.UnpooledAdminConnectionString(baseConnectionString)); + await admin.OpenAsync(ct); + /* No TimescaleDB job worker is left in the database the FORCE drop below kills (#5480). */ + await ScratchPostgres.QuiesceTimescaleJobsAsync(baseConnectionString, databaseName); + await using (var drop = new NpgsqlCommand($"DROP DATABASE \"{databaseName}\" WITH (FORCE)", admin)) + { + await drop.ExecuteNonQueryAsync(ct); + } + + await using var create = new NpgsqlCommand($"CREATE DATABASE \"{databaseName}\"", admin); + await create.ExecuteNonQueryAsync(ct); + } + + /// Calls coverage with the same wall clock until the hourly rollup's floor reads + /// (a background re-measure has landed) or thirty seconds pass. + private static async Task PollForFloorAsync( + NpgsqlDataSource dataSource, RollupAvailability availability, DateTime now, DateTime? expected, CancellationToken ct) + { + var deadline = DateTime.UtcNow.AddSeconds(30); + while (true) + { + var coverage = await TimescaleSupport.DetectRollupCoverageAsync(dataSource, availability, now, ct); + if (coverage.FloorOf(HourlyView) == expected || DateTime.UtcNow > deadline) + { + return coverage; + } + + await Task.Delay(50, ct); + } + } + private static async Task SeedHourlyQueryStoreStatsAsync( NpgsqlConnection connection, DateTime from, DateTime to, CancellationToken cancellationToken) { diff --git a/Darling/Darling.Tests/RollupFloorCacheTests.cs b/Darling/Darling.Tests/RollupFloorCacheTests.cs index 8bd7bd0d2b..741e78b89d 100644 --- a/Darling/Darling.Tests/RollupFloorCacheTests.cs +++ b/Darling/Darling.Tests/RollupFloorCacheTests.cs @@ -344,7 +344,7 @@ public void RollupCoverageProbeSql_TwoArgument_NamesOnlyTheMeasuredViews() var sql = TimescaleSupport.RollupCoverageProbeSql(RollupAvailability.All, measure); - Assert.Contains($"(SELECT min(bucket) FROM collect.{TimescaleSupport.QueryStoreStatsHourlyView})", sql, StringComparison.Ordinal); + Assert.Contains($"(SELECT {TimescaleSupport.ColdFloorMarker} min(bucket) FROM collect.{TimescaleSupport.QueryStoreStatsHourlyView})", sql, StringComparison.Ordinal); Assert.DoesNotContain($"collect.{TimescaleSupport.QueryStatsHourlyView})", sql, StringComparison.Ordinal); /* Column count is unchanged — a non-measured present view still contributes a placeholder column, @@ -371,7 +371,7 @@ public void RollupCoverageProbeSql_OneArgumentForm_MatchesTodaysLiteralOutputFor /* A literal copy of today's (pre-#4539) output for RollupAvailability.All, so a change to this overload's SHAPE (not just its selectivity) cannot slip through unnoticed. */ var expected = "SELECT " + string.Join(", ", TimescaleSupport.RollupViews - .Select(r => $"(SELECT min(bucket) FROM collect.{r.View})") + .Select(r => $"(SELECT /* coverage-floor:cold */ min(bucket) FROM collect.{r.View})") .Concat(TimescaleSupport.RolledRawTables.Select(t => $"(SELECT min(collection_time) FROM collect.{t})"))); Assert.Equal(expected, TimescaleSupport.RollupCoverageProbeSql(RollupAvailability.All), StringComparer.Ordinal); @@ -405,4 +405,74 @@ oldest chunk's name — this SQL is the source of that column. */ Assert.Contains("materialization_hypertable_schema", sql, StringComparison.Ordinal); Assert.Contains("materialization_hypertable_name", sql, StringComparison.Ordinal); } + + /* ─────────── #5329: a reused floor is still asked whether anything is earlier ─────────── */ + + [Fact] + public void RollupViewsToCheckForAnEarlierFloor_AReusedView_IsACandidate() + { + var view = TimescaleSupport.QueryStatsIoHourlyView; + var cached = new Dictionary(StringComparer.Ordinal) { [view] = Entry("_hyper_1_1_chunk", SomeFloor) }; + var oldestNow = new Dictionary(StringComparer.Ordinal) { [view] = new("_hyper_1_1_chunk", SomeHypertable) }; + + var views = TimescaleSupport.RollupViewsToCheckForAnEarlierFloor( + cached, oldestNow, RollupAvailability.All, new HashSet(StringComparer.Ordinal), deferred: null); + + Assert.Equal(new[] { view }, views); + } + + [Fact] + public void RollupViewsToCheckForAnEarlierFloor_AViewBeingMeasuredADeferredViewAnUncachedViewAndAChangedChunk_AreNotCandidates() + { + var measured = TimescaleSupport.QueryStatsIoHourlyView; + var deferred = TimescaleSupport.ProcedureStatsIoHourlyView; + var uncached = TimescaleSupport.QueryStoreStatsHourlyView; + var changed = TimescaleSupport.QueryStatsHourlyView; + var cached = new Dictionary(StringComparer.Ordinal) + { + [measured] = Entry("_hyper_1_1_chunk", SomeFloor), + [deferred] = Entry("_hyper_1_1_chunk", SomeFloor), + [changed] = Entry("_hyper_1_1_chunk", SomeFloor), + }; + var oldestNow = new Dictionary(StringComparer.Ordinal) + { + [measured] = new("_hyper_1_1_chunk", SomeHypertable), + [deferred] = new("_hyper_1_1_chunk", SomeHypertable), + [uncached] = new("_hyper_1_1_chunk", SomeHypertable), + [changed] = new("_hyper_1_2_chunk", SomeHypertable), + }; + + var views = TimescaleSupport.RollupViewsToCheckForAnEarlierFloor( + cached, oldestNow, RollupAvailability.All, + new HashSet(new[] { measured }, StringComparer.Ordinal), + new HashSet(new[] { deferred }, StringComparer.Ordinal)); + + Assert.Empty(views); + } + + [Fact] + public void RollupViewsToCheckForAnEarlierFloor_AViewTheStoreLacks_IsNotACandidate() + { + var view = TimescaleSupport.QueryStatsIoHourlyView; + var cached = new Dictionary(StringComparer.Ordinal) { [view] = Entry("_hyper_1_1_chunk", SomeFloor) }; + var oldestNow = new Dictionary(StringComparer.Ordinal) { [view] = new("_hyper_1_1_chunk", SomeHypertable) }; + + var views = TimescaleSupport.RollupViewsToCheckForAnEarlierFloor( + cached, oldestNow, RollupAvailability.None, new HashSet(StringComparer.Ordinal), deferred: null); + + Assert.Empty(views); + } + + [Fact] + public void RollupEarlierFloorProbeSql_IsABoundedReadPerView_AndNeverTheColdMinBucketStatement() + { + var sql = TimescaleSupport.RollupEarlierFloorProbeSql(new[] { TimescaleSupport.QueryStatsIoHourlyView, TimescaleSupport.ProcedureStatsIoHourlyView }); + + Assert.Contains($"FROM collect.{TimescaleSupport.QueryStatsIoHourlyView} AS m WHERE m.bucket < $1::timestamp", sql, StringComparison.Ordinal); + Assert.Contains($"FROM collect.{TimescaleSupport.ProcedureStatsIoHourlyView} AS m WHERE m.bucket < $2::timestamp", sql, StringComparison.Ordinal); + /* The floor-cache live tests count the markers (ColdFloorMarker for the cold sort, EarlierFloorMarker for this bounded + read) to prove the cold sort is not re-run; the bounded read carries the earlier marker and never the cold one. */ + Assert.DoesNotContain(TimescaleSupport.ColdFloorMarker, sql, StringComparison.Ordinal); + Assert.Contains(TimescaleSupport.EarlierFloorMarker, sql, StringComparison.Ordinal); + } } diff --git a/Darling/Darling.Tests/RollupFloorStoreCacheTests.cs b/Darling/Darling.Tests/RollupFloorStoreCacheTests.cs new file mode 100644 index 0000000000..5f89c71034 --- /dev/null +++ b/Darling/Darling.Tests/RollupFloorStoreCacheTests.cs @@ -0,0 +1,213 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using Npgsql; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4957, with no database: the pure halves of the per-store rollup floor cache — which store a data source belongs +/// to, which views a cycle measures inline versus in the background, and how a cycle's result is written back. +/// The end-to-end behavior is in RollupFloorCacheLiveTests. +/// +public sealed class RollupFloorStoreCacheTests +{ + private const string View = TimescaleSupport.QueryStoreStatsIntervalHourlyView; + private static readonly DateTime Now = new(2026, 10, 2, 12, 0, 0, DateTimeKind.Unspecified); + private static readonly DateTime Floor = Now.AddDays(-30); + + private static string KeyOf(string connectionString) + { + using var dataSource = NpgsqlDataSource.Create(connectionString); + return TimescaleSupport.RollupStoreKey(dataSource); + } + + [Fact] + public void RollupStoreKey_IsTheSameForAnyUserPasswordOrSessionOption_OnOneHostPortAndDatabase() + { + var first = KeyOf("Host=Store-A;Port=5432;Username=owner;Password=one;Database=darling"); + var second = KeyOf("Host=store-a;Port=5432;Username=mcp;Password=two;Database=darling;Options=-c TimeZone=UTC"); + + Assert.Equal(first, second); + } + + [Theory] + [InlineData("Host=store-b;Port=5432;Username=owner;Database=darling")] + [InlineData("Host=store-a;Port=5433;Username=owner;Database=darling")] + [InlineData("Host=store-a;Port=5432;Username=owner;Database=darling_other")] + public void RollupStoreKey_DiffersWhenTheHostPortOrDatabaseDiffers(string other) + { + Assert.NotEqual(KeyOf("Host=store-a;Port=5432;Username=owner;Database=darling"), KeyOf(other)); + } + + private static TimescaleSupport.RollupFloorCacheEntry Entry(DateTime measuredAt, long oid = 7) + => new("chunk_1", "hyper_1", Floor, measuredAt, Ceiling: null, DatabaseOid: oid); + + private static TimescaleSupport.RollupFloorPlan Plan( + TimescaleSupport.RollupFloorCacheEntry? entry, long oidNow = 7, string chunkNow = "chunk_1", string hypertableNow = "hyper_1") + { + var cached = new Dictionary(StringComparer.Ordinal); + if (entry is { } e) + { + cached[View] = e; + } + + var oldestNow = new Dictionary(StringComparer.Ordinal) + { + [View] = new(chunkNow, hypertableNow, oidNow), + }; + + return TimescaleSupport.PlanRollupFloorMeasurements(cached, oldestNow, RollupAvailability.All, Now); + } + + [Fact] + public void Plan_AFloorPastTheHourWithItsIdentityUnchanged_IsDueInTheBackground_NotInline() + { + var plan = Plan(Entry(Now - TimescaleSupport.RollupFloorMaxReuse)); + + Assert.Empty(plan.Inline); + Assert.Contains(View, plan.Background); + } + + [Fact] + public void Plan_AFreshFloorWithItsIdentityUnchanged_IsNeitherInlineNorDue() + { + var plan = Plan(Entry(Now - TimeSpan.FromMinutes(5))); + + Assert.Empty(plan.Inline); + Assert.Empty(plan.Background); + } + + [Fact] + public void Plan_NoEntryYet_IsInline() + { + var plan = Plan(entry: null); + + Assert.Contains(View, plan.Inline); + Assert.Empty(plan.Background); + } + + [Fact] + public void Plan_ADifferentDatabaseOid_IsInline_EvenWhenTheEntryIsPastTheHour() + { + var plan = Plan(Entry(Now - TimescaleSupport.RollupFloorMaxReuse - TimeSpan.FromMinutes(5), oid: 7), oidNow: 8); + + Assert.Contains(View, plan.Inline); + Assert.Empty(plan.Background); + } + + /* The second threshold. Between the reuse window and twice the window a floor is served from the cache and + re-measured once in the background (the tests above). At twice the window it is measured inline again, so no + floor is served older than that, whether or not callers kept arriving. */ + [Fact] + public void ServeCap_IsTwiceTheReuseWindow() + { + Assert.Equal(TimescaleSupport.RollupFloorMaxReuse * 2, TimescaleSupport.RollupFloorMaxServeAge); + Assert.True(TimescaleSupport.RollupFloorMaxServeAge > TimescaleSupport.RollupFloorMaxReuse); + } + + [Fact] + public void Plan_AFloorJustUnderTheServeCap_WithItsIdentityUnchanged_IsStillServed_AndDueInTheBackground() + { + var plan = Plan(Entry(Now - TimescaleSupport.RollupFloorMaxServeAge + TimeSpan.FromMinutes(1))); + + Assert.Empty(plan.Inline); + Assert.Contains(View, plan.Background); + } + + [Theory] + [InlineData(0)] + [InlineData(1)] + [InlineData(60 * 24 * 30)] + public void Plan_AFloorAtOrPastTheServeCap_WithItsIdentityUnchanged_IsInline_NotServedFromTheCache(int minutesPast) + { + var plan = Plan(Entry(Now - TimescaleSupport.RollupFloorMaxServeAge - TimeSpan.FromMinutes(minutesPast))); + + Assert.Contains(View, plan.Inline); + Assert.Empty(plan.Background); + } + + [Theory] + [InlineData(5)] + [InlineData(61)] + [InlineData(121)] + [InlineData(60 * 24 * 30)] + public void Plan_AChangedChunkHypertableOrDatabase_IsInline_AtAnyAge(int ageMinutes) + { + var measuredAt = Now - TimeSpan.FromMinutes(ageMinutes); + + foreach (var plan in new[] + { + Plan(Entry(measuredAt), chunkNow: "chunk_2"), + Plan(Entry(measuredAt), hypertableNow: "hyper_2"), + Plan(Entry(measuredAt), oidNow: 8), + }) + { + Assert.Contains(View, plan.Inline); + Assert.Empty(plan.Background); + } + } + + [Fact] + public void Plan_RollupFloorsToMeasure_IsTheUnionOfBoth() + { + var cached = new Dictionary(StringComparer.Ordinal) + { + [View] = Entry(Now - TimescaleSupport.RollupFloorMaxReuse), + }; + var oldestNow = new Dictionary(StringComparer.Ordinal) + { + [View] = new("chunk_1", "hyper_1", 7), + }; + + Assert.Contains(View, TimescaleSupport.RollupFloorsToMeasure(cached, oldestNow, RollupAvailability.All, Now)); + } + + [Fact] + public void Apply_ReplacesWhatThisCycleMeasured_DropsAnEmptyView_AndLeavesTheRestAsTheCacheHoldsThemNow() + { + var other = TimescaleSupport.QueryStoreStatsHourlyView; + var newerThanTheCycleSnapshot = Entry(Now); + var cache = new Dictionary(StringComparer.Ordinal) + { + [View] = Entry(Now.AddHours(-2)), + [other] = newerThanTheCycleSnapshot, + }; + var measured = new Dictionary(StringComparer.Ordinal) { [View] = null }; + var newEntries = new Dictionary(StringComparer.Ordinal) + { + [other] = Entry(Now.AddHours(-5)), + }; + + TimescaleSupport.ApplyRollupFloorMeasurements(cache, measured, newEntries, RollupAvailability.All); + + Assert.False(cache.ContainsKey(View)); + Assert.Equal(newerThanTheCycleSnapshot, cache[other]); + } + + [Fact] + public void Apply_DropsAViewWhoseRollupIsGone() + { + var cache = new Dictionary(StringComparer.Ordinal) + { + [View] = Entry(Now), + }; + + TimescaleSupport.ApplyRollupFloorMeasurements( + cache, + new Dictionary(StringComparer.Ordinal), + new Dictionary(StringComparer.Ordinal), + RollupAvailability.None); + + Assert.Empty(cache); + } +} diff --git a/Darling/Darling.Tests/RollupIoFloorFillLiveTests.cs b/Darling/Darling.Tests/RollupIoFloorFillLiveTests.cs new file mode 100644 index 0000000000..d6177c54a2 --- /dev/null +++ b/Darling/Darling.Tests/RollupIoFloorFillLiveTests.cs @@ -0,0 +1,125 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Tests; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: the test mints its own scratch database, because it materializes continuous aggregates and stops the + scheduler, which no other class sharing the store may see. */ +/// +/// #5329: the rollup floor cache sees an io rollup filled in place. A new io hourly rollup is filled recent hours first +/// by the refresh policy and the earlier hours later, usually into the materialization chunk the floor cache already +/// keys on, so neither the chunk name nor the hypertable moves. The cache used to keep the later floor until the hour +/// was up, so stayed false and Top Queries stayed on the partial +/// route. After the fix the next coverage probe reports the earlier floor. +/// +public sealed class RollupIoFloorFillLiveTests +{ + private const int ServerId = -953291; + private const string ServerName = "a5329-io-floor-fill"; + private static readonly DateTime WindowStart = new(2026, 1, 5, 0, 0, 0, DateTimeKind.Unspecified); + + [Fact] + public async Task AnIoRollupFilledEarlierInsideItsOldestChunk_IsSeenByTheNextProbe_AndTheRouteBecomesCovering() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live #5329 io floor fill test (it mints its own scratch database)."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var timescaleEnabled = await TimescaleSupport.TryEnableAsync(connection, null, ct); + Assert.SkipWhen(!timescaleEnabled, "The live #5329 io floor fill test needs TimescaleDB."); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + Assert.True(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, null, ct)); + + await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection)) + { + await stop.ExecuteNonQueryAsync(ct); + } + + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, ct); + + var bodySucceeded = false; + try + { + var io = TimescaleSupport.QueryStatsIoHourlyView; + var late = WindowStart.AddHours(8); + + /* Raw rows for an early window and a late one, eight hours apart (the seed is four hourly buckets each). */ + await IoRollupOracleSeed.PlantAsync(connection, ct, ServerId, ServerName, WindowStart); + await IoRollupOracleSeed.PlantAsync(connection, ct, ServerId, ServerName, late); + + /* The refresh policy's first fill: the late hours only. The floor is inside the oldest chunk. */ + await RefreshAsync(connection, io, late, late.AddHours(5), ct); + var chunkBefore = await OldestChunkAsync(connection, io, ct); + + await using var dataSource = NpgsqlDataSource.Create(scratch.ConnectionString); + var availability = await TimescaleSupport.DetectRollupsAsync(dataSource, ct); + Assert.True(availability.Has(io)); + + var partial = await TimescaleSupport.DetectRollupCoverageAsync(dataSource, availability, ct); + Assert.Equal(late, partial.FloorOf(io)); + Assert.False(ViewerDataService.IoHourlyCoversWindow(availability, partial, io, WindowStart), + "while only the late hours are materialized the io route must stay on the partial route"); + + /* The later fill: the early hours, into the same chunk. */ + await RefreshAsync(connection, io, WindowStart, WindowStart.AddHours(5), ct); + Assert.Equal(chunkBefore, await OldestChunkAsync(connection, io, ct)); + + var covering = await TimescaleSupport.DetectRollupCoverageAsync(dataSource, availability, ct); + Assert.Equal(WindowStart, covering.FloorOf(io)); + Assert.True(ViewerDataService.IoHourlyCoversWindow(availability, covering, io, WindowStart), + "after the early hours are materialized the next probe must route to the io rollup"); + + /* A probe with nothing new keeps the floor. */ + var again = await TimescaleSupport.DetectRollupCoverageAsync(dataSource, availability, ct); + Assert.Equal(WindowStart, again.FloorOf(io)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, (_, _) => Task.CompletedTask); + } + } + + private static async Task RefreshAsync(NpgsqlConnection connection, string view, DateTime from, DateTime to, System.Threading.CancellationToken ct) + { + await using var refresh = new NpgsqlCommand($"CALL refresh_continuous_aggregate('collect.{view}'::regclass, $1::timestamp, $2::timestamp)", connection); + refresh.Parameters.AddWithValue(from); + refresh.Parameters.AddWithValue(to); + await refresh.ExecuteNonQueryAsync(ct); + } + + private static async Task OldestChunkAsync(NpgsqlConnection connection, string view, System.Threading.CancellationToken ct) + { + await using var command = new NpgsqlCommand(@" +SELECT c.chunk_name +FROM timescaledb_information.continuous_aggregates AS ca +JOIN timescaledb_information.chunks AS c + ON c.hypertable_schema = ca.materialization_hypertable_schema AND c.hypertable_name = ca.materialization_hypertable_name +WHERE ca.view_schema = 'collect' AND ca.view_name = $1 +ORDER BY c.range_start +LIMIT 1", connection); + command.Parameters.AddWithValue(view); + return (string?)await command.ExecuteScalarAsync(ct); + } +} diff --git a/Darling/Darling.Tests/RollupTileDescriptionTests.cs b/Darling/Darling.Tests/RollupTileDescriptionTests.cs index 34e283a726..6cd57f6074 100644 --- a/Darling/Darling.Tests/RollupTileDescriptionTests.cs +++ b/Darling/Darling.Tests/RollupTileDescriptionTests.cs @@ -158,7 +158,7 @@ public void TheAgNumbers_DescribeThemselvesWithTheirLabelsId() Assert.Contains( @"el(""div"", { class: ""num"", text: fmtInt(num), ""aria-describedby"": lblId })", ag, StringComparison.Ordinal); - Assert.Contains(@"el(""div"", { class: ""lbl"", id: lblId, text: lbl })", ag, StringComparison.Ordinal); + Assert.Contains(@"el(""div"", { class: ""lbl"", id: lblId, text: countLabel(num, lbl) })", ag, StringComparison.Ordinal); /* The scope decision, pinned. The fleet tile takes a sub-line and a title because that rollup qualifies one of its numbers with coverage and hovers a long note; these three counts qualify diff --git a/Darling/Darling.Tests/RollupWindowEndBoundTests.cs b/Darling/Darling.Tests/RollupWindowEndBoundTests.cs index 92401db479..09203e1a72 100644 --- a/Darling/Darling.Tests/RollupWindowEndBoundTests.cs +++ b/Darling/Darling.Tests/RollupWindowEndBoundTests.cs @@ -194,13 +194,17 @@ public void ComposeRawRoute_KeepsTheInclusiveEndOnItsRawTimeColumn() Assert.DoesNotContain("f.collection_time < $2", compiled.Sql, StringComparison.Ordinal); } - /* The three first-bucket probes are NOT part of the change: they only locate the first bucket the window - holds (where the served span starts), and a bucket at the very end can be that first bucket only when the - window holds no other, in which case the read returns no rows either way. */ + /* #5329: the first-bucket probes take the grids' exclusive end. They once kept `<=` (a bucket at the very end can be the + first bucket only when the window holds no other, and then the read returned no rows either way). That reason no + longer holds: the probe's answer now names the start of the grid in the window-edges note and the viewer banner, so + a server whose only bucket begins AT an hour-aligned end would be told a start for a grid that read nothing. */ [Fact] - public void FirstBucketProbes_AreUnchanged() + public void FirstBucketProbes_StopBeforeTheWindowEnd_LikeTheGrids() { - Assert.Contains("f.bucket <= $3$CEIL$", DarlingDataReader.HourlyFirstBucketSql, StringComparison.Ordinal); - Assert.Contains("f.bucket <= $3$CEIL$", DarlingDataReader.HourlyFirstBucketSingleRelationSql, StringComparison.Ordinal); + foreach (var sql in new[] { RollupCoverage.HourlyFirstBucketSql, RollupCoverage.HourlyFirstBucketSingleRelationSql }) + { + Assert.Contains("f.bucket < $3$CEIL$", sql, StringComparison.Ordinal); + Assert.False(Regex.IsMatch(sql, @"\bbucket\s*<="), "a `bucket <=` bound is back in a first-bucket probe."); + } } } diff --git a/Darling/Darling.Tests/RowPlanReadLivePostgresTests.cs b/Darling/Darling.Tests/RowPlanReadLivePostgresTests.cs new file mode 100644 index 0000000000..3e667a995d --- /dev/null +++ b/Darling/Darling.Tests/RowPlanReadLivePostgresTests.cs @@ -0,0 +1,750 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Globalization; +using System.Linq; +using System.Text.Json; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5228: the three row-keyed plan reads the web grids call (get_active_query_plan_xml, +/// get_query_store_plan_xml, get_procedure_plan_xml) return the EXACT stored XML for the row's own key and +/// answer unavailable on a key that matches nothing. The Active Queries read is keyed by the row's +/// natural key, so its collection_time must survive the trip through ToString("o") with the +/// microseconds Postgres keeps, and a snapshot row collected with no request_id must answer to request_id 0. +/// Query Store plans are seeded through the inline query_plan_text column. Fixed anchors only. +/// +[Collection("live-postgres")] +public sealed class RowPlanReadLivePostgresTests +{ + private const string ServerName = "darling-row-plan-e2e"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + + /// A second registered server for the cross-server proof. Its name is not a substring of + /// (or the reverse), so neither can resolve as the other's partial match. + private const string OtherServerName = "darling-twin-plan-e2e"; + private static readonly int OtherServerId = ServerIdHelper.GetDeterministicHashCode(OtherServerName); + + private const string Db = "RowPlanDb"; + private const string SqlHandle = "0xA0B1C2D3E4F50617"; + + /// Fixed anchor with a non-zero microsecond part (123456 us), naive UTC as the store keeps it. + private static readonly DateTime Anchor = new DateTime(2026, 3, 4, 5, 6, 7, DateTimeKind.Unspecified).AddTicks(1_234_560); + + private static string PlanXml(string marker) => + "" + + $""; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static string StatusOf(string json) + { + using var doc = JsonDocument.Parse(json); + return doc.RootElement.GetProperty("status").GetString()!; + } + + /// The answer's message, decoded (the JSON writer escapes an apostrophe, so a raw substring test would miss one). + private static string MessageOf(string json) + { + using var doc = JsonDocument.Parse(json); + return doc.RootElement.GetProperty("message").GetString()!; + } + + [Fact] + public async Task EachRowPlanTool_ReturnsTheExactStoredXml_AndUnavailableOnAWrongKey() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live row-plan test."); + var ct = TestContext.Current.CancellationToken; + + await using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + + var bodySucceeded = false; + try + { + await RegisterServerAsync(connection, ct); + + var estimated = PlanXml("estimated"); + var live = PlanXml("live"); + var noRequest = PlanXml("no-request-id"); + var queryStore = PlanXml("query-store"); + var procedure = PlanXml("procedure"); + + await InsertSnapshotAsync(connection, Anchor, 51, 3, estimated, live, ct); + await InsertSnapshotAsync(connection, Anchor, 52, null, noRequest, null, ct); + await InsertQueryStoreAsync(connection, Anchor, 4242, 7, queryStore, ct); + await InsertProcedureAsync(connection, Anchor, procedure, ct); + + /* ---- get_active_query_plan_xml: the collection_time the grid hands back is Anchor.ToString("o"). */ + var key = Anchor.ToString("o", CultureInfo.InvariantCulture); + Assert.Equal(estimated, await DarlingMcpPlanTools.GetActiveQueryPlanXml(postgres, key, 51, ServerName, 3, cancellationToken: ct)); + Assert.Equal(live, await DarlingMcpPlanTools.GetActiveQueryPlanXml(postgres, key, 51, ServerName, 3, live: true, cancellationToken: ct)); + + /* The microseconds are the key: the same instant rounded to the second matches nothing. */ + var rounded = Anchor.AddTicks(-(Anchor.Ticks % TimeSpan.TicksPerSecond)).ToString("o", CultureInfo.InvariantCulture); + Assert.Equal("unavailable", StatusOf(await DarlingMcpPlanTools.GetActiveQueryPlanXml(postgres, rounded, 51, ServerName, 3, cancellationToken: ct))); + + /* A row collected with a NULL request_id answers to request_id 0 (the grids carry it as 0). */ + Assert.Equal(noRequest, await DarlingMcpPlanTools.GetActiveQueryPlanXml(postgres, key, 52, ServerName, 0, cancellationToken: ct)); + Assert.Equal(noRequest, await DarlingMcpPlanTools.GetActiveQueryPlanXml(postgres, key, 52, ServerName, cancellationToken: ct)); + + /* Wrong keys, and a row with no live plan, are unavailable. */ + Assert.Equal("unavailable", StatusOf(await DarlingMcpPlanTools.GetActiveQueryPlanXml(postgres, key, 999, ServerName, 3, cancellationToken: ct))); + Assert.Equal("unavailable", StatusOf(await DarlingMcpPlanTools.GetActiveQueryPlanXml(postgres, key, 51, ServerName, 4, cancellationToken: ct))); + Assert.Equal("unavailable", StatusOf(await DarlingMcpPlanTools.GetActiveQueryPlanXml(postgres, key, 52, ServerName, 0, live: true, cancellationToken: ct))); + + /* ---- get_query_store_plan_xml, through the inline query_plan_text column. */ + Assert.Equal(queryStore, await DarlingMcpPlanTools.GetQueryStorePlanXml(postgres, Db, 4242, ServerName, cancellationToken: ct)); + Assert.Equal(queryStore, await DarlingMcpPlanTools.GetQueryStorePlanXml(postgres, Db, 4242, ServerName, plan_id: 7, cancellationToken: ct)); + Assert.Equal("unavailable", StatusOf(await DarlingMcpPlanTools.GetQueryStorePlanXml(postgres, Db, 9999, ServerName, cancellationToken: ct))); + Assert.Equal("unavailable", StatusOf(await DarlingMcpPlanTools.GetQueryStorePlanXml(postgres, Db, 4242, ServerName, plan_id: 8, cancellationToken: ct))); + + /* ---- get_procedure_plan_xml. */ + Assert.Equal(procedure, await DarlingMcpPlanTools.GetProcedurePlanXml(postgres, SqlHandle, ServerName, cancellationToken: ct)); + Assert.Equal("unavailable", StatusOf(await DarlingMcpPlanTools.GetProcedurePlanXml(postgres, "0xDEADBEEF", ServerName, cancellationToken: ct))); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + private static Task RegisterServerAsync(NpgsqlConnection connection, System.Threading.CancellationToken ct) => + RegisterServerAsync(connection, ServerId, ServerName, ct); + + private static async Task RegisterServerAsync( + NpgsqlConnection connection, int serverId, string serverName, System.Threading.CancellationToken ct) + { + await using var command = new NpgsqlCommand(@" +INSERT INTO servers (server_id, server_name, display_name, is_enabled, created_date, modified_date) +VALUES ($1, $2, $3, TRUE, $4, $4) +ON CONFLICT (server_id) DO UPDATE SET server_name = EXCLUDED.server_name, display_name = EXCLUDED.display_name, + is_enabled = TRUE, modified_date = EXCLUDED.modified_date;", connection); + command.Parameters.AddWithValue(serverId); + command.Parameters.AddWithValue(serverName); + command.Parameters.AddWithValue(serverName); + command.Parameters.AddWithValue(DateTime.SpecifyKind(Anchor, DateTimeKind.Unspecified)); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertSnapshotAsync( + NpgsqlConnection connection, DateTime collectionTime, int sessionId, int? requestId, string? plan, string? livePlan, + System.Threading.CancellationToken ct) + { + await using var command = new NpgsqlCommand(@" +INSERT INTO query_snapshots + (collection_id, collection_time, server_id, server_name, session_id, request_id, query_plan, live_query_plan) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8)", connection); + command.Parameters.AddWithValue(CollectionIdGenerator.Next()); + command.Parameters.AddWithValue(DateTime.SpecifyKind(collectionTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(ServerName); + command.Parameters.AddWithValue(sessionId); + command.Parameters.AddWithValue((object?)requestId ?? DBNull.Value); + command.Parameters.AddWithValue((object?)plan ?? DBNull.Value); + command.Parameters.AddWithValue((object?)livePlan ?? DBNull.Value); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertQueryStoreAsync( + NpgsqlConnection connection, DateTime collectionTime, long queryId, long planId, string planText, + System.Threading.CancellationToken ct) + { + await using var command = new NpgsqlCommand(@" +INSERT INTO query_store_stats + (collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, query_plan_text) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8)", connection); + command.Parameters.AddWithValue(CollectionIdGenerator.Next()); + command.Parameters.AddWithValue(DateTime.SpecifyKind(collectionTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(ServerName); + command.Parameters.AddWithValue(Db); + command.Parameters.AddWithValue(queryId); + command.Parameters.AddWithValue(planId); + command.Parameters.AddWithValue(planText); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertProcedureAsync( + NpgsqlConnection connection, DateTime collectionTime, string planXml, System.Threading.CancellationToken ct) + { + await using var command = new NpgsqlCommand(@" +INSERT INTO procedure_stats + (collection_id, collection_time, server_id, server_name, database_name, schema_name, object_name, sql_handle, query_plan_xml) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)", connection); + command.Parameters.AddWithValue(CollectionIdGenerator.Next()); + command.Parameters.AddWithValue(DateTime.SpecifyKind(collectionTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(ServerName); + command.Parameters.AddWithValue(Db); + command.Parameters.AddWithValue("dbo"); + command.Parameters.AddWithValue("usp_RowPlan"); + command.Parameters.AddWithValue(SqlHandle); + command.Parameters.AddWithValue(planXml); + await command.ExecuteNonQueryAsync(ct); + } + + /// + /// #5236: the blocking and deadlock plan reads fetch the plan of the row the LIST printed, by the strings the list + /// emitted (as_of is set, so the window is fixed). Blocked-process reports: one with both plans, one with the blocked plan + /// only, one with none, and one whose plan is an empty string (the flag and the read must agree it is absent). Two + /// deadlocks share BOTH stamps and differ in victim, so victim_process_id picks the right twin; a third has no plan and a + /// fourth an empty one. The flags are written only when true. The event_time carries microseconds, and the key is matched + /// exactly. The DMV-snapshot arm has no plan columns and is covered by the reader mapping test. + /// + [Fact] + public async Task BlockingAndDeadlockRows_FetchTheirOwnPlans_ByTheKeysTheListReadsEmit() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live row-plan test."); + var ct = TestContext.Current.CancellationToken; + + await using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + + var bodySucceeded = false; + try + { + await RegisterServerAsync(connection, ct); + + var bothBlocked = PlanXml("both-blocked"); + var bothBlocking = PlanXml("both-blocking"); + var onlyBlocked = PlanXml("only-blocked"); + var victimOne = PlanXml("victim-one"); + var victimTwo = PlanXml("victim-two"); + + await InsertReportAsync(connection, Anchor, 51, 52, bothBlocked, bothBlocking, ct); + await InsertReportAsync(connection, Anchor.AddMinutes(1), 53, 54, onlyBlocked, null, ct); + await InsertReportAsync(connection, Anchor.AddMinutes(2), 55, 56, null, null, ct); + await InsertReportAsync(connection, Anchor.AddMinutes(3), 57, 58, "", "", ct); + + var twinCollected = Anchor.AddMinutes(10); + var twinStamp = twinCollected.AddSeconds(-7); + await InsertDeadlockAsync(connection, twinCollected, twinStamp, "process1a2b", victimOne, ct); + await InsertDeadlockAsync(connection, twinCollected, twinStamp, "process3c4d", victimTwo, ct); + await InsertDeadlockAsync(connection, Anchor.AddMinutes(20), Anchor.AddMinutes(20).AddSeconds(-7), "process5e6f", null, ct); + await InsertDeadlockAsync(connection, Anchor.AddMinutes(30), Anchor.AddMinutes(30).AddSeconds(-7), "process7a8b", "", ct); + + var asOf = Anchor.AddHours(1).ToString("yyyy-MM-dd'T'HH:mm:ss'Z'", CultureInfo.InvariantCulture); + + /* ---- blocking: the list's own strings feed the tool. */ + using var blockingList = JsonDocument.Parse(await DarlingMcpBlockingTools.GetBlocking( + postgres, ServerName, hours_back: 24, as_of: asOf, cancellationToken: ct)); + var events = blockingList.RootElement.GetProperty("events"); + + JsonElement Event(int blockedSpid) => events.EnumerateArray().Single(e => e.GetProperty("blocked_spid").GetInt32() == blockedSpid); + + /* The row's own strings, its database_name included (the web sends the row's database when it has one). */ + async Task BlockingPlan(JsonElement e, string? side = null) => await DarlingMcpPlanTools.GetBlockingPlanXml( + postgres, e.GetProperty("event_time").GetString()!, e.GetProperty("blocked_spid").GetInt32(), e.GetProperty("blocking_spid").GetInt32(), + ServerName, e.GetProperty("blocked_ecid").GetInt32(), e.GetProperty("blocking_ecid").GetInt32(), side ?? "blocked", DatabaseOf(e), ct); + + var both = Event(51); + Assert.True(both.GetProperty("has_blocked_plan").GetBoolean()); + Assert.True(both.GetProperty("has_blocking_plan").GetBoolean()); + Assert.Equal(bothBlocked, await BlockingPlan(both)); + Assert.Equal(bothBlocking, await BlockingPlan(both, "blocking")); + Assert.Equal(bothBlocking, await BlockingPlan(both, "BLOCKING")); + + /* The microseconds are the key: the same instant rounded to the second matches nothing. */ + var rounded = Anchor.AddTicks(-(Anchor.Ticks % TimeSpan.TicksPerSecond)).ToString("o", CultureInfo.InvariantCulture); + Assert.Equal("unavailable", StatusOf(await DarlingMcpPlanTools.GetBlockingPlanXml(postgres, rounded, 51, 52, ServerName, cancellationToken: ct))); + + var blockedOnly = Event(53); + Assert.True(blockedOnly.GetProperty("has_blocked_plan").GetBoolean()); + Assert.False(blockedOnly.TryGetProperty("has_blocking_plan", out _)); + Assert.Equal(onlyBlocked, await BlockingPlan(blockedOnly)); + Assert.Equal("unavailable", StatusOf(await BlockingPlan(blockedOnly, "blocking"))); + + /* No plan, and an empty-string plan: neither flag, and the reads agree. */ + foreach (var spid in new[] { 55, 57 }) + { + var none = Event(spid); + Assert.False(none.TryGetProperty("has_blocked_plan", out _)); + Assert.False(none.TryGetProperty("has_blocking_plan", out _)); + Assert.Equal("unavailable", StatusOf(await BlockingPlan(none))); + Assert.Equal("unavailable", StatusOf(await BlockingPlan(none, "blocking"))); + } + + /* ---- deadlocks: twins share both stamps, so the victim picks the plan. */ + using var deadlockList = JsonDocument.Parse(await DarlingMcpBlockingTools.GetDeadlocks( + postgres, ServerName, hours_back: 24, as_of: asOf, cancellationToken: ct)); + var deadlocks = deadlockList.RootElement.GetProperty("deadlocks"); + + JsonElement Deadlock(string victim) => deadlocks.EnumerateArray().Single(d => d.GetProperty("victim_process_id").GetString() == victim); + + async Task VictimPlan(JsonElement d, string? victim) => await DarlingMcpPlanTools.GetDeadlockPlanXml( + postgres, d.GetProperty("collection_time").GetString()!, d.GetProperty("deadlock_time").GetString()!, ServerName, victim, DatabaseOf(d), ct); + + var one = Deadlock("process1a2b"); + var two = Deadlock("process3c4d"); + Assert.Equal(one.GetProperty("collection_time").GetString(), two.GetProperty("collection_time").GetString()); + Assert.Equal(one.GetProperty("deadlock_time").GetString(), two.GetProperty("deadlock_time").GetString()); + Assert.True(one.GetProperty("has_victim_plan").GetBoolean()); + Assert.True(two.GetProperty("has_victim_plan").GetBoolean()); + Assert.Equal(victimOne, await VictimPlan(one, "process1a2b")); + Assert.Equal(victimTwo, await VictimPlan(two, "process3c4d")); + Assert.Equal("unavailable", StatusOf(await VictimPlan(one, "process-nobody"))); + + /* Without the tiebreak the stamps cannot say which twin was meant, so the read picks neither: it refuses, and the + refusal names victim_process_id (the new test below pins the whole shape). */ + var untied = await VictimPlan(one, null); + Assert.Equal("invalid", StatusOf(untied)); + Assert.Contains("victim_process_id", untied, StringComparison.Ordinal); + + foreach (var victim in new[] { "process5e6f", "process7a8b" }) + { + var none = Deadlock(victim); + Assert.False(none.TryGetProperty("has_victim_plan", out _)); + Assert.Equal("unavailable", StatusOf(await VictimPlan(none, victim))); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// + /// #5236: a plan read for one server never returns another server's plan. A second server holds a blocked-process report + /// with the same event_time, spids and ecids as the first server's, collected a minute EARLIER (so it is the earliest copy + /// of that key), and a deadlock with the same stamps and the same victim, inserted FIRST (so it has the lower deadlock_id). + /// Each server's reads return its own plan, whichever side or victim form is asked. The first server's reads are the ones + /// that would return the other server's plan if a read lost its server predicate or bound the wrong id. + /// + [Fact] + public async Task BlockingAndDeadlockPlanReads_NeverReturnAnotherServersPlan() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live row-plan test."); + var ct = TestContext.Current.CancellationToken; + + await using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + + var bodySucceeded = false; + try + { + await RegisterServerAsync(connection, ServerId, ServerName, ct); + await RegisterServerAsync(connection, OtherServerId, OtherServerName, ct); + + var ownBlocked = PlanXml("own-blocked"); + var ownBlocking = PlanXml("own-blocking"); + var ownVictim = PlanXml("own-victim"); + var otherBlocked = PlanXml("other-blocked"); + var otherBlocking = PlanXml("other-blocking"); + var otherVictim = PlanXml("other-victim"); + + /* The same event on both servers: equal event_time, spids and ecids. The other server's copy is collected a minute + earlier than this server's, so without the server predicate it would be the earliest copy of the key. */ + var ownCollected = Anchor.AddSeconds(5); + await InsertReportAsync(connection, OtherServerId, OtherServerName, ownCollected.AddMinutes(-1), Anchor, Db, 51, 52, otherBlocked, otherBlocking, ct); + await InsertReportAsync(connection, ServerId, ServerName, ownCollected, Anchor, Db, 51, 52, ownBlocked, ownBlocking, ct); + + /* The same deadlock stamps and victim on both servers, the other server's inserted first so its deadlock_id is lower. */ + var collected = Anchor.AddMinutes(10); + var stamp = collected.AddSeconds(-7); + await InsertDeadlockAsync(connection, OtherServerId, OtherServerName, collected, stamp, "process1a2b", otherVictim, Db, ct); + await InsertDeadlockAsync(connection, ServerId, ServerName, collected, stamp, "process1a2b", ownVictim, Db, ct); + + var eventTime = Anchor.ToString("o", CultureInfo.InvariantCulture); + var collectionTime = collected.ToString("o", CultureInfo.InvariantCulture); + var deadlockTime = stamp.ToString("o", CultureInfo.InvariantCulture); + + foreach (var (server, blocked, blocking, victim) in new[] + { + (ServerName, ownBlocked, ownBlocking, ownVictim), + (OtherServerName, otherBlocked, otherBlocking, otherVictim), + }) + { + Assert.Equal(blocked, await DarlingMcpPlanTools.GetBlockingPlanXml(postgres, eventTime, 51, 52, server, cancellationToken: ct)); + Assert.Equal(blocking, await DarlingMcpPlanTools.GetBlockingPlanXml(postgres, eventTime, 51, 52, server, side: "blocking", cancellationToken: ct)); + Assert.Equal(victim, await DarlingMcpPlanTools.GetDeadlockPlanXml(postgres, collectionTime, deadlockTime, server, "process1a2b", cancellationToken: ct)); + Assert.Equal(victim, await DarlingMcpPlanTools.GetDeadlockPlanXml(postgres, collectionTime, deadlockTime, server, cancellationToken: ct)); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// + /// #5236: without a victim_process_id, two deadlocks that share both stamps and name different victims are never guessed + /// between. The read refuses with an invalid answer that names victim_process_id and carries neither plan, for a + /// missing victim and an empty one alike; naming the victim picks either twin. A deadlock that is alone for its stamps still + /// answers with no victim, and so do two stored copies of ONE deadlock (they name the same victim), by the lower deadlock_id. + /// + [Fact] + public async Task DeadlockPlanRead_WithoutAVictim_RefusesTwinsThatNameDifferentVictims_AndStillAnswersAloneDeadlocks() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live row-plan test."); + var ct = TestContext.Current.CancellationToken; + + await using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + + var bodySucceeded = false; + try + { + await RegisterServerAsync(connection, ct); + + var twinOne = PlanXml("twin-one"); + var twinTwo = PlanXml("twin-two"); + var lone = PlanXml("lone"); + var copyFirst = PlanXml("copy-first"); + var copySecond = PlanXml("copy-second"); + + var twinCollected = Anchor.AddMinutes(10); + var twinStamp = twinCollected.AddSeconds(-7); + await InsertDeadlockAsync(connection, twinCollected, twinStamp, "process1a2b", twinOne, ct); + await InsertDeadlockAsync(connection, twinCollected, twinStamp, "process3c4d", twinTwo, ct); + + var loneCollected = Anchor.AddMinutes(20); + var loneStamp = loneCollected.AddSeconds(-7); + await InsertDeadlockAsync(connection, loneCollected, loneStamp, "process5e6f", lone, ct); + + var copyCollected = Anchor.AddMinutes(30); + var copyStamp = copyCollected.AddSeconds(-7); + await InsertDeadlockAsync(connection, copyCollected, copyStamp, "process7a8b", copyFirst, ct); + await InsertDeadlockAsync(connection, copyCollected, copyStamp, "process7a8b", copySecond, ct); + + async Task Read(DateTime collected, DateTime stamp, string? victim) => await DarlingMcpPlanTools.GetDeadlockPlanXml( + postgres, collected.ToString("o", CultureInfo.InvariantCulture), stamp.ToString("o", CultureInfo.InvariantCulture), ServerName, victim, + cancellationToken: ct); + + /* Twins with no victim: refused, naming victim_process_id, with neither plan in the answer. */ + foreach (var none in new string?[] { null, "" }) + { + var refused = await Read(twinCollected, twinStamp, none); + using var doc = JsonDocument.Parse(refused); + Assert.Equal("invalid", doc.RootElement.GetProperty("status").GetString()); + Assert.Equal("victim_process_id", doc.RootElement.GetProperty("hints").GetProperty("parameter").GetString()); + Assert.Contains("victim_process_id", doc.RootElement.GetProperty("message").GetString(), StringComparison.Ordinal); + Assert.DoesNotContain("ShowPlanXML", refused, StringComparison.Ordinal); + } + + /* Naming the victim picks either twin, and a victim that names neither is a plain miss, not a refusal. */ + Assert.Equal(twinOne, await Read(twinCollected, twinStamp, "process1a2b")); + Assert.Equal(twinTwo, await Read(twinCollected, twinStamp, "process3c4d")); + Assert.Equal("unavailable", StatusOf(await Read(twinCollected, twinStamp, "process-nobody"))); + + /* A deadlock alone for its stamps answers with or without its victim. */ + Assert.Equal(lone, await Read(loneCollected, loneStamp, null)); + Assert.Equal(lone, await Read(loneCollected, loneStamp, "process5e6f")); + + /* Two copies of one deadlock name the same victim: they are not twins, so the lower deadlock_id answers. */ + Assert.Equal(copyFirst, await Read(copyCollected, copyStamp, null)); + Assert.Equal(copyFirst, await Read(copyCollected, copyStamp, "process7a8b")); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// + /// #5236: both reads take an optional database_name, because one server_id can collect several databases (an Azure master + /// target), and each database numbers its own sessions. Two databases hold a report with the same event_time, spids and + /// ecids, and a deadlock with the same stamps AND the same victim, and each database's plans differ. With the row's database + /// sent, each read returns that database's plan, through the tools and through the web read route (which echoes it). With + /// none sent, or an empty one, the answer is the one the reads gave before the filter existed (the earliest copy of the + /// report, the lower deadlock_id). A database that has no such row is a miss that says which database. + /// + [Fact] + public async Task PlanReads_NarrowToTheRowsDatabase_WhenItIsSent_AndKeepTheOldAnswerWhenItIsNot() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live row-plan test."); + var ct = TestContext.Current.CancellationToken; + + await using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + + var bodySucceeded = false; + try + { + await RegisterServerAsync(connection, ct); + + const string dbOne = "RowPlanDbOne"; + const string dbTwo = "RowPlanDbTwo"; + var oneBlocked = PlanXml("one-blocked"); + var oneBlocking = PlanXml("one-blocking"); + var oneVictim = PlanXml("one-victim"); + var twoBlocked = PlanXml("two-blocked"); + var twoBlocking = PlanXml("two-blocking"); + var twoVictim = PlanXml("two-victim"); + + /* The same event key in both databases; the first database's copy is collected first, so it is the earliest copy. */ + await InsertReportAsync(connection, ServerId, ServerName, Anchor.AddSeconds(5), Anchor, dbOne, 51, 52, oneBlocked, oneBlocking, ct); + await InsertReportAsync(connection, ServerId, ServerName, Anchor.AddSeconds(8), Anchor, dbTwo, 51, 52, twoBlocked, twoBlocking, ct); + + /* The same stamps and the same victim in both databases, the first database's inserted first (the lower deadlock_id). */ + var collected = Anchor.AddMinutes(10); + var stamp = collected.AddSeconds(-7); + await InsertDeadlockAsync(connection, ServerId, ServerName, collected, stamp, "process1a2b", oneVictim, dbOne, ct); + await InsertDeadlockAsync(connection, ServerId, ServerName, collected, stamp, "process1a2b", twoVictim, dbTwo, ct); + + var eventTime = Anchor.ToString("o", CultureInfo.InvariantCulture); + var collectionTime = collected.ToString("o", CultureInfo.InvariantCulture); + var deadlockTime = stamp.ToString("o", CultureInfo.InvariantCulture); + + async Task Blocked(string? database, string? side = null) => await DarlingMcpPlanTools.GetBlockingPlanXml( + postgres, eventTime, 51, 52, ServerName, side: side, database_name: database, cancellationToken: ct); + async Task Victim(string? database, string? victim = "process1a2b") => await DarlingMcpPlanTools.GetDeadlockPlanXml( + postgres, collectionTime, deadlockTime, ServerName, victim, database, ct); + + /* The row's database sent: that database's plan. */ + Assert.Equal(oneBlocked, await Blocked(dbOne)); + Assert.Equal(oneBlocking, await Blocked(dbOne, "blocking")); + Assert.Equal(twoBlocked, await Blocked(dbTwo)); + Assert.Equal(twoBlocking, await Blocked(dbTwo, "blocking")); + Assert.Equal(oneVictim, await Victim(dbOne)); + Assert.Equal(twoVictim, await Victim(dbTwo)); + Assert.Equal(twoVictim, await Victim(dbTwo, victim: null)); + + /* No database, or an empty one: the answer the reads gave before the filter existed. */ + foreach (var none in new string?[] { null, "" }) + { + Assert.Equal(oneBlocked, await Blocked(none)); + Assert.Equal(oneBlocking, await Blocked(none, "blocking")); + Assert.Equal(oneVictim, await Victim(none)); + } + + /* A database with no such row is a miss that names it, not another database's plan. */ + var blockedMiss = await Blocked("NoSuchDb"); + Assert.Equal("unavailable", StatusOf(blockedMiss)); + Assert.Contains("in database 'NoSuchDb'", MessageOf(blockedMiss), StringComparison.Ordinal); + var victimMiss = await Victim("NoSuchDb"); + Assert.Equal("unavailable", StatusOf(victimMiss)); + Assert.Contains("in database 'NoSuchDb'", MessageOf(victimMiss), StringComparison.Ordinal); + + /* The web read route forwards the row's database_name to the tool and echoes it with the plan. */ + async Task Web(string tool, string query) + { + var context = new Microsoft.AspNetCore.Http.DefaultHttpContext(); + context.Request.QueryString = new Microsoft.AspNetCore.Http.QueryString("?" + query); + using var doc = JsonDocument.Parse(await DarlingWebEndpoints.BuildReadDispatch()[tool](context, postgres, null!)); + return doc.RootElement.Clone(); + } + + var blockedKey = $"server={ServerName}&event_time={eventTime}&blocked_spid=51&blocking_spid=52"; + var webTwo = await Web("get_blocking_plan_xml", blockedKey + "&database_name=" + dbTwo); + Assert.Equal(twoBlocked, webTwo.GetProperty("plan_xml").GetString()); + Assert.Equal(dbTwo, webTwo.GetProperty("database_name").GetString()); + var webNone = await Web("get_blocking_plan_xml", blockedKey); + Assert.Equal(oneBlocked, webNone.GetProperty("plan_xml").GetString()); + Assert.Equal(JsonValueKind.Null, webNone.GetProperty("database_name").ValueKind); + + var victimKey = $"server={ServerName}&collection_time={collectionTime}&deadlock_time={deadlockTime}&victim_process_id=process1a2b"; + var webVictimTwo = await Web("get_deadlock_plan_xml", victimKey + "&database_name=" + dbTwo); + Assert.Equal(twoVictim, webVictimTwo.GetProperty("plan_xml").GetString()); + Assert.Equal(dbTwo, webVictimTwo.GetProperty("database_name").GetString()); + Assert.Equal(oneVictim, (await Web("get_deadlock_plan_xml", victimKey)).GetProperty("plan_xml").GetString()); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// The row's database_name as the list printed it, or null when the row has none (the web then sends none). + private static string? DatabaseOf(JsonElement row) => + row.TryGetProperty("database_name", out var value) && value.ValueKind == JsonValueKind.String ? value.GetString() : null; + + private static Task InsertReportAsync( + NpgsqlConnection connection, DateTime eventTime, int blockedSpid, int blockingSpid, string? blockedPlan, string? blockingPlan, + System.Threading.CancellationToken ct) => + InsertReportAsync(connection, ServerId, ServerName, eventTime.AddSeconds(5), eventTime, Db, blockedSpid, blockingSpid, blockedPlan, blockingPlan, ct); + + private static async Task InsertReportAsync( + NpgsqlConnection connection, int serverId, string serverName, DateTime collectionTime, DateTime eventTime, string? database, + int blockedSpid, int blockingSpid, string? blockedPlan, string? blockingPlan, System.Threading.CancellationToken ct) + { + await using var command = new NpgsqlCommand(@" +INSERT INTO blocked_process_reports + (blocked_report_id, collection_time, server_id, server_name, event_time, wait_time_ms, blocking_spid, blocked_spid, + blocked_ecid, blocking_ecid, blocking_status, database_name, blocked_query_plan_xml, blocking_query_plan_xml) +VALUES ($1, $2, $3, $4, $5, 12000, $6, $7, 0, 0, 'suspended', $8, $9, $10)", connection); + command.Parameters.AddWithValue(CollectionIdGenerator.Next()); + command.Parameters.AddWithValue(DateTime.SpecifyKind(collectionTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(serverId); + command.Parameters.AddWithValue(serverName); + command.Parameters.AddWithValue(DateTime.SpecifyKind(eventTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(blockingSpid); + command.Parameters.AddWithValue(blockedSpid); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Text, Value = (object?)database ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Text, Value = (object?)blockedPlan ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Text, Value = (object?)blockingPlan ?? DBNull.Value }); + await command.ExecuteNonQueryAsync(ct); + } + + private static Task InsertDeadlockAsync( + NpgsqlConnection connection, DateTime collectionTime, DateTime deadlockTime, string victimProcessId, string? victimPlan, + System.Threading.CancellationToken ct) => + InsertDeadlockAsync(connection, ServerId, ServerName, collectionTime, deadlockTime, victimProcessId, victimPlan, null, ct); + + private static async Task InsertDeadlockAsync( + NpgsqlConnection connection, int serverId, string serverName, DateTime collectionTime, DateTime deadlockTime, + string victimProcessId, string? victimPlan, string? database, System.Threading.CancellationToken ct) + { + await using var command = new NpgsqlCommand(@" +INSERT INTO deadlocks + (deadlock_id, collection_time, server_id, server_name, deadlock_time, victim_process_id, victim_query_plan_xml, database_name) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8)", connection); + command.Parameters.AddWithValue(CollectionIdGenerator.Next()); + command.Parameters.AddWithValue(DateTime.SpecifyKind(collectionTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(serverId); + command.Parameters.AddWithValue(serverName); + command.Parameters.AddWithValue(DateTime.SpecifyKind(deadlockTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(victimProcessId); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Text, Value = (object?)victimPlan ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Text, Value = (object?)database ?? DBNull.Value }); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task DeleteRowsAsync(NpgsqlConnection connection, System.Threading.CancellationToken ct) + { + await using var cleanup = new NpgsqlCommand( + $"DELETE FROM blocked_process_reports WHERE server_id IN ({ServerId}, {OtherServerId}); " + + $"DELETE FROM deadlocks WHERE server_id IN ({ServerId}, {OtherServerId}); " + + $"DELETE FROM query_snapshots WHERE server_id = {ServerId}; " + + $"DELETE FROM query_store_stats WHERE server_id = {ServerId}; " + + $"DELETE FROM procedure_stats WHERE server_id = {ServerId}; " + + $"DELETE FROM servers WHERE server_id IN ({ServerId}, {OtherServerId});", connection); + await cleanup.ExecuteNonQueryAsync(ct); + } + + /// + /// #5236: the twin check looks at EVERY deadlock the stamps match, not only the ones that captured a plan. Two deadlocks that + /// share both stamps and name different victims are refused when no victim is sent, whichever of them captured a plan (the + /// plan of the other one must not answer for it); with each victim sent, the one that captured a plan answers with it and + /// the other is the plain not-captured miss. Two stored copies of ONE deadlock (the same victim) still answer from the copy + /// that captured a plan, even when the lower deadlock_id is the copy that did not. + /// + [Fact] + public async Task DeadlockPlanRead_TwinsRefuse_WhetherOrNotEachCapturedAPlan_AndACopyWithoutAPlanNeverHidesItsSibling() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live row-plan test."); + var ct = TestContext.Current.CancellationToken; + + await using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + + var bodySucceeded = false; + try + { + await RegisterServerAsync(connection, ct); + + var firstPlan = PlanXml("first-captured"); + var secondPlan = PlanXml("second-captured"); + var copyPlan = PlanXml("copy-captured"); + + /* Only the first twin captured a plan. */ + var firstOnlyCollected = Anchor.AddMinutes(40); + var firstOnlyStamp = firstOnlyCollected.AddSeconds(-7); + await InsertDeadlockAsync(connection, firstOnlyCollected, firstOnlyStamp, "process1a2b", firstPlan, ct); + await InsertDeadlockAsync(connection, firstOnlyCollected, firstOnlyStamp, "process3c4d", null, ct); + + /* Only the second twin captured a plan. */ + var secondOnlyCollected = Anchor.AddMinutes(50); + var secondOnlyStamp = secondOnlyCollected.AddSeconds(-7); + await InsertDeadlockAsync(connection, secondOnlyCollected, secondOnlyStamp, "process1a2b", "", ct); + await InsertDeadlockAsync(connection, secondOnlyCollected, secondOnlyStamp, "process3c4d", secondPlan, ct); + + /* Two copies of one deadlock: the lower deadlock_id captured no plan, the higher one did. */ + var copyCollected = Anchor.AddMinutes(60); + var copyStamp = copyCollected.AddSeconds(-7); + await InsertDeadlockAsync(connection, copyCollected, copyStamp, "process7a8b", null, ct); + await InsertDeadlockAsync(connection, copyCollected, copyStamp, "process7a8b", copyPlan, ct); + + async Task Read(DateTime collected, DateTime stamp, string? victim) => await DarlingMcpPlanTools.GetDeadlockPlanXml( + postgres, collected.ToString("o", CultureInfo.InvariantCulture), stamp.ToString("o", CultureInfo.InvariantCulture), ServerName, victim, + cancellationToken: ct); + + void AssertRefused(string answer) + { + using var doc = JsonDocument.Parse(answer); + Assert.Equal("invalid", doc.RootElement.GetProperty("status").GetString()); + Assert.Equal("victim_process_id", doc.RootElement.GetProperty("hints").GetProperty("parameter").GetString()); + Assert.Contains("victim_process_id", doc.RootElement.GetProperty("message").GetString(), StringComparison.Ordinal); + Assert.DoesNotContain("ShowPlanXML", answer, StringComparison.Ordinal); + } + + /* No victim: refused when only the first twin captured a plan, and when only the second did. */ + foreach (var none in new string?[] { null, "" }) + { + AssertRefused(await Read(firstOnlyCollected, firstOnlyStamp, none)); + AssertRefused(await Read(secondOnlyCollected, secondOnlyStamp, none)); + } + + /* Each victim sent: the twin that captured a plan answers with it, the other is the not-captured miss. */ + Assert.Equal(firstPlan, await Read(firstOnlyCollected, firstOnlyStamp, "process1a2b")); + Assert.Equal("unavailable", StatusOf(await Read(firstOnlyCollected, firstOnlyStamp, "process3c4d"))); + Assert.Equal("unavailable", StatusOf(await Read(secondOnlyCollected, secondOnlyStamp, "process1a2b"))); + Assert.Equal(secondPlan, await Read(secondOnlyCollected, secondOnlyStamp, "process3c4d")); + + /* Copies of one deadlock answer from the copy that captured a plan, with no victim or with theirs. */ + Assert.Equal(copyPlan, await Read(copyCollected, copyStamp, null)); + Assert.Equal(copyPlan, await Read(copyCollected, copyStamp, "process7a8b")); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } +} diff --git a/Darling/Darling.Tests/RuntimePreconditionMissTests.cs b/Darling/Darling.Tests/RuntimePreconditionMissTests.cs index 99cc901bbb..0c74e9a46d 100644 --- a/Darling/Darling.Tests/RuntimePreconditionMissTests.cs +++ b/Darling/Darling.Tests/RuntimePreconditionMissTests.cs @@ -34,6 +34,7 @@ namespace Darling.Tests; /// only property that matters: that a state somebody can fix is never reported as one they cannot, and the /// reverse. /// +[Trait("Reads", "Lite")] public sealed class CollectorRuntimePreconditionTests { private const string Server = "precondition-probe"; @@ -789,20 +790,29 @@ public void BothGatedOffMessages_SayAReconnectIsNeeded_NotThatNothingNeedsRestar } /// - /// The contradiction this change also fixes, which predates #2559 and shipped for two issues. The - /// SESSION_MISSING arm has always said in its own sentence that a dropped session "stays missing until - /// the next connect", and then appended a shared epilogue promising "nothing to restart on the monitoring - /// side". Both sentences were in the same operator-facing string. + /// The SESSION_MISSING arm names the app's own recovery. The app creates its capture sessions when it + /// connects and ensures them again at least once an hour, so a session that was dropped or stopped comes back within + /// the hour once the login holds the grant, and nobody has to reconnect the server. The arm used to say the + /// session "stays missing until the next connect", told the reader to let the server reconnect, and ended on + /// the connect-scoped epilogue. All three are false now. The message says the hourly create, asks for no + /// reconnect, and ends on the general epilogue that promises nothing to restart. The interval is pinned + /// beside the wording, so a change to the interval has to change the sentence too. /// [Fact] - public void TheCaptureSessionMessage_NoLongerContradictsItself() + public void TheCaptureSessionMessage_SaysTheSessionComesBackWithinTheHour_AndAsksForNoReconnect() { var message = CollectorRuntimePrecondition.CollectionOutcomeMessage( Server, "deadlocks", CollectorRuntimePrecondition.CaptureSessionMissingStatus, "The session was not found.", DateTime.UtcNow.AddMinutes(-3))!; - Assert.Contains("next connect", message, StringComparison.Ordinal); - Assert.DoesNotContain("nothing to restart", message, StringComparison.OrdinalIgnoreCase); + Assert.Equal(TimeSpan.FromHours(1), AlwaysOnXeSessions.EnsureInterval); + Assert.Contains("at least once an hour", message, StringComparison.Ordinal); + Assert.Contains("within the hour", message, StringComparison.Ordinal); + Assert.Contains("creates a missing session and starts a stopped one", message, StringComparison.Ordinal); + Assert.Contains("nothing to restart", message, StringComparison.OrdinalIgnoreCase); + Assert.DoesNotContain("next connect", message, StringComparison.Ordinal); + Assert.DoesNotContain("reconnect", message, StringComparison.OrdinalIgnoreCase); + Assert.DoesNotContain("read once when the service connects", message, StringComparison.Ordinal); } /// @@ -905,6 +915,7 @@ public void ACollectorWithNoNounPhrase_StillGetsACorrectSentence() /// on an engine that can never have the surface would re-introduce the defect #2511 closed, one layer down. /// Nothing in the type system enforces that, and a ?? chain is trivially reorderable. /// +[Trait("Reads", "Lite")] public sealed class RuntimePreconditionReadWiringTests { private const string DarlingMcp = "Darling/PerformanceMonitor.Darling.Service/Mcp"; @@ -1165,6 +1176,7 @@ private static string[] RepoFilesIn(string relativeDirectory, [CallerFilePath] s /// everyone. /// [Collection("live-postgres")] +[Trait("Reads", "Lite")] public sealed class RuntimePreconditionMissLivePostgresTests { private const string DeniedServerName = "darling-precondition-denied"; diff --git a/Darling/Darling.Tests/SameStatementPileupSourceCensusTests.cs b/Darling/Darling.Tests/SameStatementPileupSourceCensusTests.cs index ab69ab49c4..7266699fce 100644 --- a/Darling/Darling.Tests/SameStatementPileupSourceCensusTests.cs +++ b/Darling/Darling.Tests/SameStatementPileupSourceCensusTests.cs @@ -34,6 +34,8 @@ namespace Darling.Tests; /// follow-on enrichment where QS happens to be readable. That is a different code path on the /// scheduled pass, and nothing here restricts it. /// +[Trait("Stage", "Guard")] +[Trait("Reads", "Lite")] public sealed class SameStatementPileupSourceCensusTests { /// @@ -181,7 +183,8 @@ public void TheStatementIdentity_ComesFromTheSnapshotRow() ElapsedMs: 29_124, CpuTimeMs: 1_351, LogicalReads: 287_049, - PhysicalReads: 29_505); + PhysicalReads: 29_505, + PreviewText: "select 1"); Assert.Equal("0xabcdef0123456789", SameStatementPileupDetector.StatementIdentity(row)); diff --git a/Darling/Darling.Tests/ScopedLoadOrderingTests.cs b/Darling/Darling.Tests/ScopedLoadOrderingTests.cs index 8fda165cad..5e12d2e871 100644 --- a/Darling/Darling.Tests/ScopedLoadOrderingTests.cs +++ b/Darling/Darling.Tests/ScopedLoadOrderingTests.cs @@ -27,7 +27,8 @@ namespace Darling.Tests; /// a drop rather than a silent loss only because something is certain to repaint — in the Darling viewer the /// coalescing replay in RefreshVisibleAsync / RefreshActiveInnerTabAsync / /// RefreshActiveSubTabAsync, which re-reads the scope at the re-entered load's own entry. Lite has -/// no replay anywhere: ServerTab._isRefreshing and RecommendationsTab._isBusy are bail-only +/// no replay at these sites (the server tab's main-tab loads gained one in #5371, through +/// RefreshCoordinator): RecommendationsTab._isBusy is bail-only /// and FinOpsTab has no load guard at all, so the same drop there suppresses a paint nothing will redo. /// A generation cannot do that: the only condition under which it drops is that a newer load for the same /// surface has already started, so the surface always has a live writer. @@ -54,13 +55,13 @@ namespace Darling.Tests; /// silent loss this mechanism exists to avoid, reached from the other direction. /// /// What is deliberately NOT in this table. ServerTab's twenty-one main-tab -/// Refresh*Async loaders. Their races route through the bail-only _isRefreshing, where the -/// user-visible half is the DROPPED trigger — a time-range change mid-pass leaves the charts on the old -/// window while the combo shows the new one — and no drop-based mechanism can fix a load that never started. -/// That wants the viewer's replay, which needs _isRefreshing separated from the event-suppression duty -/// it also serves in TimeDisplayMode_SelectionChanged, ServerTab.DrillDown.cs and -/// ServerTab.Grids.cs, and that is a load-architecture change rather than a guard. +/// Refresh*Async loaders. Their races used to route through the bail-only _isRefreshing, where the +/// user-visible half was the DROPPED trigger — a time-range change mid-pass left the charts on the old +/// window while the combo showed the new one — and no drop-based mechanism can fix a load that never started. +/// #5371 gave them the viewer's replay (RefreshCoordinator), with the event-suppression duty split off +/// into _suppressRangeRefresh, which was a load-architecture change rather than a guard. /// +[Trait("Reads", "Lite")] public sealed class ScopedLoadOrderingTests { // ══ The mechanism ═════════════════════════════════════════════════════════════════════════ @@ -555,7 +556,9 @@ private static int FirstAwait(Site site, string body) /// /// Offsets of the paints: an assignment into a XAML-generated control's UI property, or a call to one of - /// the two helpers that push a list into a grid. Derived from the shape rather than from a per-site list + /// the two helpers that push a list into a grid, or ShowAlerts, which holds the Alerts history tab's whole + /// paint (#5542 L6 moved it out of LoadAlertsAsync so a test can drive it with rows): the call sits below + /// the supersession check with no await between, which keeps the rule true for every paint inside it. Derived from the shape rather than from a per-site list /// of surface names, so a paint added to a guarded method is covered without a second edit here. /// 's chrome allowance is subtracted — a button re-enabled in a finally has /// to fire whoever won the race. @@ -567,7 +570,7 @@ private static List Paints(Site site, string body) return Regex.Matches( body, PaintTarget + @"\s*=(?!=)|\.\s*UpdateData\s*\(|\bApplyViewModel\s*\(" - + @"|\bSet\w*ComparisonMode\s*\(") + + @"|\bSet\w*ComparisonMode\s*\(|\bShowAlerts\s*\(") .Where(m => !chrome.Any(c => m.Value.Replace(" ", "", StringComparison.Ordinal) .StartsWith(c, StringComparison.Ordinal))) .Select(m => m.Index) diff --git a/Darling/Darling.Tests/ScratchDatabaseDisposalCensusTests.cs b/Darling/Darling.Tests/ScratchDatabaseDisposalCensusTests.cs new file mode 100644 index 0000000000..9720402f65 --- /dev/null +++ b/Darling/Darling.Tests/ScratchDatabaseDisposalCensusTests.cs @@ -0,0 +1,378 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text.RegularExpressions; +using Xunit; + +namespace Darling.Tests; + +/// +/// Every test that mints a scratch database drops it before the test ends. +/// +/// The defect this closes. Seven live test classes (28 tests) minted a scratch database with +/// ScratchPostgres.CreateAsync and disposed only their connection, so each database lived until the process-exit +/// drain. Since #5480 each drop unschedules the database's TimescaleDB jobs and waits for their workers, about 0.4 +/// seconds a database on a CI runner, so the drain of those 29 databases took 11 seconds. The test runner (xunit.v3 +/// under Microsoft.Testing.Platform) forces the process out with exit code 1 ("Foreground threads were left running, +/// forcing process exit") 10 seconds after the run returns, so the nightly's Darling PostgreSQL step failed with 28,763 +/// tests passed, and its publish job was skipped. The drain was not a thread: it was the process-exit handler holding +/// the exit past the runner's wait. +/// +/// What this pins. A method that mints a scratch database, directly or through a helper that returns it, +/// disposes it in the same method (await using or DisposeAsync). A helper that returns the store and can +/// skip or throw after minting it disposes the store before it lets the skip or the failure out, since the caller +/// never owns a store the helper did not return. Source-level on purpose: a live check needs a PostgreSQL server, and +/// the databases a leak leaves are only counted after every test has finished. +/// +[Trait("Stage", "Guard")] +public sealed class ScratchDatabaseDisposalCensusTests +{ + /* Spelled in two parts so this file's own samples are not mistaken for a test that mints a database. */ + private const string CreateCall = "ScratchPostgres" + ".CreateAsync("; + + private static readonly Regex MemberStart = new( + @"^ (?:\[[^\]]*\]\s*)*(?:public|private|internal|protected|static|async)\b", + RegexOptions.Compiled); + + private static readonly Regex DisposeCall = new(@"\b\w*[sS]cratch\w*\??\.DisposeAsync\(", RegexOptions.Compiled); + + [Fact] + public void NoTestLeavesAScratchDatabaseForTheExitDrain() + { + var directory = FindTestProjectDirectory(); + Assert.NotNull(directory); + + var leaks = new List(); + var scanned = 0; + foreach (var file in Directory.EnumerateFiles(directory!, "*.cs", SearchOption.TopDirectoryOnly).OrderBy(f => f, StringComparer.Ordinal)) + { + if (Path.GetFileName(file) is "ScratchDatabaseDisposalCensusTests.cs" or "ScratchPostgres.cs") + { + continue; + } + + var source = File.ReadAllText(file); + if (!source.Contains(CreateCall, StringComparison.Ordinal)) + { + continue; + } + + scanned++; + leaks.AddRange(FindLeaks(source).Select(method => $"{Path.GetFileName(file)}: {method}")); + } + + Assert.True(scanned > 100, $"the census read {scanned} files that mint a scratch database; the walk to the test sources is broken"); + Assert.True( + leaks.Count == 0, + "These methods mint a scratch database and never drop it, so it waits for the process-exit drain, which the test runner " + + "cuts off after 10 seconds (exit code 1 with every test passed). Put `await using var scratch = ...` on it, or call " + + "`await scratch.DisposeAsync()` in the finally:" + Environment.NewLine + string.Join(Environment.NewLine, leaks)); + } + + [Fact] + public void TheDetector_FlagsAMethodThatMintsAndNeverDisposes() + { + var leak = $@" +public sealed class Sample +{{ + [Fact] + public async Task Leaks() + {{ + var scratch = await {CreateCall}cs, default); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + }} +}}"; + Assert.Equal(new[] { "Leaks" }, FindLeaks(leak)); + } + + [Fact] + public void TheDetector_FlagsAHelperTestThatOnlyDisposesItsConnection() + { + /* The shape the seven classes had: the opener returns the store, and the test's finally closed the connection. */ + var leak = $@" +public sealed class Sample +{{ + private static async Task<(NpgsqlConnection Connection, ScratchPostgres Scratch)> OpenAsync() + {{ + var scratch = await {CreateCall}cs, default); + return (new NpgsqlConnection(scratch.ConnectionString), scratch); + }} + + [Fact] + public async Task Leaks() + {{ + var (connection, scratch) = await OpenAsync(); + try + {{ + }} + finally + {{ + await connection.DisposeAsync(); + }} + }} +}}"; + Assert.Equal(new[] { "Leaks" }, FindLeaks(leak)); + } + + [Fact] + public void TheDetector_FlagsAnOpenerThatCanSkipAfterMintingAndDoesNotDispose() + { + var leak = $@" +public sealed class Sample +{{ + private static async Task<(NpgsqlConnection Connection, ScratchPostgres Scratch)> OpenAsync() + {{ + var scratch = await {CreateCall}cs, default); + var connection = new NpgsqlConnection(scratch.ConnectionString); + Assert.SkipWhen(true, ""needs TimescaleDB""); + return (connection, scratch); + }} +}}"; + Assert.Equal(new[] { "OpenAsync" }, FindLeaks(leak)); + } + + [Fact] + public void TheDetector_AcceptsEveryWayTheRepoDisposesAStore() + { + var clean = $@" +public sealed class Sample +{{ + private static async Task<(NpgsqlConnection Connection, ScratchPostgres Scratch)> OpenAsync() + {{ + var scratch = await {CreateCall}cs, default); + NpgsqlConnection? connection = null; + try + {{ + connection = new NpgsqlConnection(scratch.ConnectionString); + Assert.SkipWhen(true, ""needs TimescaleDB""); + return (connection, scratch); + }} + catch + {{ + await scratch.DisposeAsync(); + throw; + }} + }} + + private static async Task SeedAsync(string cs) => await {CreateCall}cs, default); + + [Fact] + public async Task DisposesInFinally() + {{ + var (connection, scratch) = await OpenAsync(); + try {{ }} + finally + {{ + await connection.DisposeAsync(); + await scratch.DisposeAsync(); + }} + }} + + [Fact] + public async Task UsingDeclaration() + {{ + await using var scratch = await {CreateCall}cs, default); + }} + + [Fact] + public async Task UsingDeclarationThroughAHelper() + {{ + await using var store = await SeedAsync(""cs""); + }} + + [Fact] + public async Task UsingAnAlias() + {{ + var scratch = await SeedAsync(""cs""); + await using var _ = scratch; + }} +}}"; + Assert.Empty(FindLeaks(clean)); + } + + /// The methods in that mint a scratch database and do not drop it, by name. + internal static IReadOnlyList FindLeaks(string source) + { + source = Regex.Replace(source, @"/\*.*?\*/", string.Empty, RegexOptions.Singleline); + source = Regex.Replace(source, @"//[^\r\n]*", string.Empty); + + var chunks = SplitIntoMembers(source); + + /* A helper whose return type carries the store, or a small handle type whose own DisposeAsync drops it, hands + the store to its caller, who then owns the drop. */ + var owners = new List { "ScratchPostgres" }; + owners.AddRange(FindHandleTypes(source)); + var ownerType = new Regex(@"\b(?:Task|ValueTask)<[^;{]*\b(?:" + string.Join("|", owners.Select(Regex.Escape)) + @")\b"); + var creators = chunks + .Where(c => ownerType.IsMatch(ReturnType(c))) + .Select(c => c.Name) + .ToHashSet(StringComparer.Ordinal); + + var lifetimeDropsTheStore = chunks.Any(c => c.Name == "DisposeAsync" && DisposeCall.IsMatch(c.Text)); + var creatorCall = creators.Count == 0 ? null : new Regex(@"(?(); + foreach (var chunk in chunks) + { + var body = chunk.Text[FirstLine(chunk.Text).Length..]; + var minted = body.Contains(CreateCall, StringComparison.Ordinal); + var calledAHelper = creatorCall is not null && creatorCall.IsMatch(body); + if (!minted && !calledAHelper) + { + continue; + } + + if (creators.Contains(chunk.Name)) + { + /* The helper itself: only a skip or a throw after minting is a leak, because the caller never got the store. */ + var afterMint = minted ? body[body.IndexOf(CreateCall, StringComparison.Ordinal)..] : body; + if (afterMint.Contains("Skip", StringComparison.Ordinal) && !DisposeCall.IsMatch(afterMint)) + { + leaks.Add(chunk.Name); + } + + continue; + } + + if (chunk.Name == "InitializeAsync" && lifetimeDropsTheStore) + { + continue; + } + + if (!Disposes(body, creators)) + { + leaks.Add(chunk.Name); + } + } + + return leaks; + } + + /// The nested types whose DisposeAsync drops a scratch database: a test hands one out and disposes it as a unit. + private static IEnumerable FindHandleTypes(string source) + { + var typeDeclaration = new Regex(@"^\s+(?:(?:public|private|internal|protected|sealed|static|readonly|file|partial)\s+)*(?:class|record\s+struct|record|struct)\s+(\w+)"); + var lines = source.Replace("\r\n", "\n").Split('\n'); + string? lastType = null; + var disposeWindow = 0; + var found = new HashSet(StringComparer.Ordinal); + foreach (var line in lines) + { + var declaration = typeDeclaration.Match(line); + if (declaration.Success) + { + lastType = declaration.Groups[1].Value; + } + + if (Regex.IsMatch(line, @"\bValueTask\s+DisposeAsync\(\)")) + { + disposeWindow = 12; + } + else if (disposeWindow > 0) + { + disposeWindow--; + if (lastType is not null && DisposeCall.IsMatch(line)) + { + found.Add(lastType); + } + } + } + + return found; + } + + private static bool Disposes(string body, HashSet creators) + { + /* `await (await Mint(...)).DisposeAsync()` mints and drops in one statement. */ + if (DisposeCall.IsMatch(body) + || body.Contains("DropRememberedAsync(", StringComparison.Ordinal) + || Regex.IsMatch(body, @"CreateAsync\([^;]*\)\)\s*\.DisposeAsync\(")) + { + return true; + } + + /* `await using var x = ...` on a variable named for the store, or on the result of the mint or a helper. */ + if (Regex.IsMatch(body, @"await\s+using\s*\(?\s*(?:var\s+)?\w*[sS]cratch\w*\s*=")) + { + return true; + } + + if (Regex.IsMatch(body, @"await\s+using\s*\(?\s*(?:var\s+)?\w+\s*=\s*[\w.]*[sS]cratch\w*\s*[;)]")) + { + return true; + } + + var sources = new List { Regex.Escape("ScratchPostgres" + ".CreateAsync") }; + sources.AddRange(creators.Select(Regex.Escape)); + return Regex.IsMatch(body, @"await\s+using\s*\(?\s*(?:var\s+)?\w+\s*=\s*\(?\s*await\s+(?:" + string.Join("|", sources) + @")\b"); + } + + /// The part of a member's first line before its name: its return type, so a parameter of the store's type does not count. + private static string ReturnType(Member member) + { + var line = FirstLine(member.Text); + var at = line.IndexOf(member.Name + "(", StringComparison.Ordinal); + return at < 0 ? line : line[..at]; + } + + private static string FirstLine(string text) + { + var end = text.IndexOf('\n'); + return end < 0 ? text : text[..(end + 1)]; + } + + private readonly record struct Member(string Name, string Text); + + /// Cuts a class into its members: each starts on a line at the class-member indent (4 spaces) that opens with a modifier. + private static List SplitIntoMembers(string source) + { + var members = new List(); + var lines = source.Replace("\r\n", "\n").Split('\n'); + var start = -1; + for (var i = 0; i <= lines.Length; i++) + { + var atEnd = i == lines.Length; + if (!atEnd && !MemberStart.IsMatch(lines[i])) + { + continue; + } + + if (start >= 0) + { + var text = string.Join("\n", lines[start..i]) + "\n"; + var name = Regex.Match(lines[start], @"(\w+)\s*(?:<[^>(]*>)?\s*\("); + members.Add(new Member(name.Success ? name.Groups[1].Value : lines[start].Trim(), text)); + } + + start = i; + } + + return members; + } + + private static string? FindTestProjectDirectory() + { + var directory = new DirectoryInfo(AppContext.BaseDirectory); + for (var i = 0; i < 10 && directory is not null; i++) + { + if (File.Exists(Path.Combine(directory.FullName, "PerformanceMonitor.sln"))) + { + var source = Path.Combine(directory.FullName, "Darling", "Darling.Tests"); + return Directory.Exists(source) ? source : null; + } + + directory = directory.Parent; + } + + return null; + } +} diff --git a/Darling/Darling.Tests/ScratchDatabaseSweep.cs b/Darling/Darling.Tests/ScratchDatabaseSweep.cs new file mode 100644 index 0000000000..40b4b88691 --- /dev/null +++ b/Darling/Darling.Tests/ScratchDatabaseSweep.cs @@ -0,0 +1,193 @@ +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Text.RegularExpressions; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; + +namespace Darling.Tests; + +/// +/// The naming scheme of databases, and the start-of-run sweep that drops the ones a +/// killed run left behind (#4981). +/// +/// Why a sweep exists at all. ScratchPostgres.DisposeAsync drops its own database, but a run +/// that is killed (a stopped debugger, a closed terminal, a machine restart) never reaches it. On a long-lived local +/// cluster those databases pile up, and each one that has the TimescaleDB extension holds one of the cluster's few +/// background-worker slots for its scheduler. Four hundred of them used every slot, and later tests failed with +/// "limit of 8 exceeded" and "out of background workers" with nothing wrong in the code under test. +/// +/// Why the sweep is allowed to drop anything. It is aimed at whatever DARLING_TEST_PG names, so +/// the rules that keep it from touching anything but its own debris are the whole design, and every one is pinned by +/// : +/// +/// The name must match the factory's exact pattern, anchored at both ends: the prefix, a 14-digit UTC creation +/// stamp, and 12 lowercase hex characters. A near miss (another prefix, a different suffix length, uppercase hex, a +/// trailing extra) is never dropped. Databases minted before the stamp existed carry no age, so they match nothing +/// here and are left alone. +/// The creation stamp in the name must be older than . A run in flight, this +/// process's or a neighbour's, owns only young databases. +/// No client session may be connected to the database. Background workers (the TimescaleDB scheduler) do not +/// count: every database with the extension has one, and an abandoned database would otherwise never look idle. +/// The drop is a plain DROP DATABASE, never WITH (FORCE), so a database that gained a session +/// between the check and the drop refuses to go instead of being pulled out from under a test. +/// Each drop is preceded by (#5549): a database is never +/// dropped while one of its TimescaleDB job workers is still attached, the same rule every in-test drop follows. +/// The quiesce is best-effort and bounded, and it never throws. It opens one short client session of its own in the +/// database, and the drop needs no retry for that session: PostgreSQL's own DROP DATABASE waits up to 5 +/// seconds for other backends to exit before it raises SQLSTATE 55006 (CountOtherDBBackends), which covers the +/// closing session. Do not wrap the drop in a retry loop: when a session really stays, each attempt costs that full +/// 5 seconds, and 30 of them (about 153 s) ran past the sweep's 60 s limit and left every later database in place. +/// A plain drop is tried once per database. The quiesce also unschedules the database's TimescaleDB jobs +/// (alter_job(scheduled => false)) before the drop, so a drop the server refuses leaves that database +/// with its jobs unscheduled. That is accepted: the sweep only drops a database it has already judged abandoned +/// (past , no client session), and nothing is rescheduled. +/// The database the connection string itself names is never a candidate. +/// +/// +/* #1776 own-store: this class never touches a store's tables. It lists databases by name and drops the factory's + own abandoned ones, so it cannot race the shared store. */ +internal static class ScratchDatabaseSweep +{ + internal const string NamePrefix = "darling_scratch_"; + + /// The creation stamp's layout: UTC, to the second. + internal const string StampFormat = "yyyyMMddHHmmss"; + + /// + /// How old a scratch database must be before the sweep may drop it. No test keeps a database for anything near + /// this long, so a database past it belongs to a run that is gone; the margin is what keeps a slow neighbour's + /// databases safe. + /// + internal static readonly TimeSpan AbandonedAfter = TimeSpan.FromMinutes(30); + + /// + /// The one pattern a droppable name matches. \z, not $: $ also matches before a trailing + /// newline, and [0-9], not \d: \d accepts other scripts' digits. + /// + private static readonly Regex NamePattern = new( + "^darling_scratch_(?[0-9]{14})_(?[0-9a-f]{12})\\z", + RegexOptions.CultureInvariant | RegexOptions.ExplicitCapture); + + /// + /// Databases with the factory's prefix that have no client session, cheapest filter first. This only narrows the + /// list; is the exact match that decides. + /// + private const string IdleCandidatesSql = @" +SELECT d.datname +FROM pg_database AS d +WHERE d.datname LIKE 'darling\_scratch\_%' + AND NOT d.datistemplate + AND NOT EXISTS ( + SELECT 1 FROM pg_stat_activity AS a + WHERE a.datid = d.oid AND a.backend_type = 'client backend')"; + + /// A fresh name: the prefix, the creation time to the second, then 12 hex characters. + internal static string NewName(DateTime utcNow) => + NamePrefix + + utcNow.ToUniversalTime().ToString(StampFormat, CultureInfo.InvariantCulture) + + "_" + + Guid.NewGuid().ToString("N")[..12]; + + /// True when is exactly a name can produce. + internal static bool IsFactoryName(string? name) => TryParseCreatedUtc(name, out _); + + /// + /// The creation time read out of a factory name; false for any other name, and for a stamp that is not a real + /// date (a month of 13 is a near miss, not a database to drop). + /// + internal static bool TryParseCreatedUtc(string? name, out DateTime createdUtc) + { + createdUtc = default; + if (name is null) + { + return false; + } + + var match = NamePattern.Match(name); + return match.Success + && DateTime.TryParseExact( + match.Groups["stamp"].Value, + StampFormat, + CultureInfo.InvariantCulture, + DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, + out createdUtc); + } + + /// + /// The pure decision: of , the factory-pattern ones whose creation stamp is at least + /// before . A stamp in the future (a clock that ran ahead) + /// is not old, so it stays. + /// + internal static IReadOnlyList SelectAbandoned(IEnumerable names, DateTime utcNow, TimeSpan olderThan) => + names + .Where(name => TryParseCreatedUtc(name, out var createdUtc) && utcNow - createdUtc >= olderThan) + .Distinct(StringComparer.Ordinal) + .ToList(); + + /// + /// Drops the abandoned scratch databases on the cluster reaches and + /// returns their names, logging each drop and each refusal. + /// + internal static async Task> SweepAsync( + string baseConnectionString, + DateTime utcNow, + TimeSpan olderThan, + Action log, + CancellationToken cancellationToken) + { + var ownDatabase = new NpgsqlConnectionStringBuilder(baseConnectionString).Database; + var dropped = new List(); + + /* #5549: unpooled, like ScratchPostgres's own create and drop, so the backend that ran a DROP DATABASE ends + with this connection instead of going back into the pool every live test draws from. */ + await using var admin = new NpgsqlConnection(ScratchPostgres.UnpooledAdminConnectionString(baseConnectionString)); + await admin.OpenAsync(cancellationToken); + + var idle = new List(); + await using (var list = new NpgsqlCommand(IdleCandidatesSql, admin)) + await using (var reader = await list.ExecuteReaderAsync(cancellationToken)) + { + while (await reader.ReadAsync(cancellationToken)) + { + idle.Add(reader.GetString(0)); + } + } + + foreach (var name in SelectAbandoned(idle, utcNow, olderThan)) + { + if (string.Equals(name, ownDatabase, StringComparison.Ordinal)) + { + continue; + } + + /* #5549: no TimescaleDB job worker is left in the database the drop below removes. Best-effort: it never + throws, so a database that cannot be opened, or one without the extension, goes straight to the drop; and + it ends at once when nothing runs there, so a sweep over idle databases stays quick. */ + await ScratchPostgres.QuiesceTimescaleJobsAsync(baseConnectionString, name, cancellationToken); + + try + { + /* One attempt. The server itself waits up to 5 s for the quiesce's closing session (and any other + backend) to leave before it raises 55006, so a retry here only multiplies the wait for a session + that really stays. The name matched the anchored pattern above, so it is hex and digits only: safe + as a quoted identifier. */ + await using var drop = new NpgsqlCommand($"DROP DATABASE IF EXISTS \"{name}\"", admin); + await drop.ExecuteNonQueryAsync(cancellationToken); + + dropped.Add(name); + log($"Dropped abandoned scratch database {name}."); + } + catch (PostgresException ex) + { + /* Typically a session that connected after the check: not abandoned after all. Leave it. Its TimescaleDB + jobs stay unscheduled (the quiesce ran first); see the class comment. */ + log($"Left scratch database {name} in place: {ex.MessageText}"); + } + } + + return dropped; + } +} diff --git a/Darling/Darling.Tests/ScratchDatabaseSweepLiveTests.cs b/Darling/Darling.Tests/ScratchDatabaseSweepLiveTests.cs new file mode 100644 index 0000000000..5682b0aad1 --- /dev/null +++ b/Darling/Darling.Tests/ScratchDatabaseSweepLiveTests.cs @@ -0,0 +1,431 @@ +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4981 against a real cluster: the start-of-run sweep drops a scratch database a killed run left behind, and +/// nothing else; and a scratch database that was never disposed does not outlive the process. +/// +/// The sweep cases plant databases with names that are exact factory names and names that only look like +/// them, one of each reason the sweep must refuse (too young, a client session connected, a different shape), runs +/// the sweep, and reads pg_database back. The sweep may also drop abandoned databases that other runs left on +/// the same cluster, which is its job, so the assertions are about the planted names and about every dropped name +/// being a factory name. +/// +/* #1776 own-store: every database this class creates carries a name unique to the run, and the only rows it reads + are the cluster's own database list, so it does not race the shared store's tables. */ +public sealed class ScratchDatabaseSweepLiveTests +{ + private const string OldStamp = "20010101000000"; + + [Fact] + public async Task TheSweep_DropsAnOldIdleFactoryDatabase_AndLeavesEveryOtherNameAlone() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to a Postgres connection string to run the live scratch sweep test."); + + var ct = TestContext.Current.CancellationToken; + var now = DateTime.UtcNow; + /* 12 lowercase hex characters, fresh per call, so concurrent runs never collide on a name. */ + static string Id() => Guid.NewGuid().ToString("N")[..12]; + + var abandoned = $"darling_scratch_{OldStamp}_{Id()}"; + var busy = $"darling_scratch_{OldStamp}_{Id()}"; + var young = $"darling_scratch_{now.AddMinutes(-5):yyyyMMddHHmmss}_{Id()}"; + var nearMisses = new[] + { + $"darling_scratch_{Id()}", // the earlier shape, no stamp + $"darling_scratch_{OldStamp}_{Id()}0", // 13 hex + $"darling_scratch_{OldStamp}_{Id()[..11]}", // 11 hex + $"darling_scratch_{OldStamp}_ABCDEF{Id()[..6]}", // uppercase hex + $"darling_scratch_{OldStamp}_{Id()}_x", // trailing extra + $"darling_scratch_20011301000000_{Id()}", // month 13 + $"xdarling_scratch_{OldStamp}_{Id()}", // prefixed + $"darlingtest_scratch_{OldStamp}_{Id()}", // another family's prefix + }; + + var planted = new List { abandoned, busy, young }; + planted.AddRange(nearMisses); + + /* The once-per-cluster sweep that ScratchPostgres.CreateAsync waits on runs from whichever test in this process + creates a scratch database first, in parallel with this class. A database stamped 2001 with no session is + exactly what that sweep drops, so planting one while it is still listing could lose it before the checks + below run. Creating and disposing one scratch database here waits for that sweep to finish, and it never + runs a second time in the process, so nothing planted below can be taken by it. */ + await (await ScratchPostgres.CreateAsync(baseConnectionString!, ct)).DisposeAsync(); + + NpgsqlConnection? busySession = null; + var bodySucceeded = false; + try + { + /* #5549: unpooled, like every connection in this class that creates or drops a database. */ + await using (var admin = new NpgsqlConnection(ScratchPostgres.UnpooledAdminConnectionString(baseConnectionString))) + { + await admin.OpenAsync(ct); + + /* A client session on an old database: a run that is still using it, whatever its name says. It is + connected the moment the database exists, because old with no session is what any sweep drops, + including one from another process on the same cluster, which this class cannot wait for. */ + await CreateDatabaseAsync(admin, busy, ct); + var busyBuilder = new NpgsqlConnectionStringBuilder(baseConnectionString) { Database = busy, Pooling = false }; + busySession = new NpgsqlConnection(busyBuilder.ConnectionString); + await busySession.OpenAsync(ct); + + foreach (var name in planted.Where(name => name != busy)) + { + await CreateDatabaseAsync(admin, name, ct); + } + } + + var log = new List(); + var dropped = await ScratchDatabaseSweep.SweepAsync( + baseConnectionString!, now, ScratchDatabaseSweep.AbandonedAfter, log.Add, ct); + + /* Nothing here depends on THIS sweep being the one that dropped the abandoned database: a sweep from another + process on the same cluster may have done it first. What must hold either way is that it is gone, that + every drop this sweep made was a factory name and was logged, and that it refused the rest. */ + Assert.All(dropped, name => Assert.True(ScratchDatabaseSweep.IsFactoryName(name), name)); + Assert.All(dropped, name => Assert.Contains(log, line => line.Contains(name, StringComparison.Ordinal))); + Assert.DoesNotContain(busy, dropped); + Assert.DoesNotContain(young, dropped); + Assert.Empty(dropped.Intersect(nearMisses, StringComparer.Ordinal)); + + var remaining = await ExistingAsync(baseConnectionString!, planted, ct); + Assert.DoesNotContain(abandoned, remaining); + Assert.Equal( + planted.Where(name => name != abandoned).OrderBy(name => name, StringComparer.Ordinal), + remaining.OrderBy(name => name, StringComparer.Ordinal)); + + bodySucceeded = true; + } + finally + { + if (busySession is not null) + { + await busySession.DisposeAsync(); + } + + await LiveStoreCleanup.RunAsync(ScratchPostgres.UnpooledAdminConnectionString(baseConnectionString!), bodySucceeded, async (cleanup, cleanupCt) => + { + foreach (var name in planted) + { + /* No TimescaleDB job worker is left in the database the FORCE drop below kills (#5480). */ + await ScratchPostgres.QuiesceTimescaleJobsAsync(baseConnectionString!, name); + await using var drop = new NpgsqlCommand($"DROP DATABASE IF EXISTS \"{name}\" WITH (FORCE)", cleanup); + await drop.ExecuteNonQueryAsync(cleanupCt); + } + }); + } + } + + /// + /// #5549: an abandoned database with TimescaleDB jobs is quiesced before the sweep drops it. A custom job that + /// sleeps 8 seconds is running in the database when the sweep starts. The quiesce unschedules the jobs and waits for + /// the running worker to leave (its cap is 10 seconds). TimescaleDB's scheduler ends an unscheduled job's running + /// worker itself, a few seconds after the stop (faster on some machines), so a correct sweep can finish well before + /// the job's 8 seconds are up. Without the quiesce, a plain DROP DATABASE does not wait for the worker: + /// TimescaleDB ends it as part of the drop, which is the kill the quiesce exists to avoid, and the drop returns in a + /// fraction of a second (0.1 to 0.3 seconds on a rig). The two differ in ORDER, not in elapsed time (a clock floor + /// failed a correct 1.6 second sweep on CI): with the quiesce the worker is gone before the sweep's + /// DROP DATABASE starts, without it the drop runs while the worker is alive. So a monitor session polls + /// pg_stat_activity during the sweep and the test fails if any one look sees both at once. It also fails, rather + /// than passing silently, if the monitor never saw the worker alive after the sweep began. + /// + [Fact] + public async Task TheSweep_QuiescesAnAbandonedDatabasesTimescaleJobs_BeforeDroppingIt() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to a Postgres connection string to run the live scratch sweep test."); + + var ct = TestContext.Current.CancellationToken; + var now = DateTime.UtcNow; + var name = $"darling_scratch_{OldStamp}_{Guid.NewGuid().ToString("N")[..12]}"; + + /* Same wait as the first sweep test: the once-per-cluster sweep must have finished before an old, idle database + is planted, or it could be the one to drop it before the checks below run. */ + await (await ScratchPostgres.CreateAsync(baseConnectionString!, ct)).DisposeAsync(); + + var bodySucceeded = false; + var workerAge = 0.0; + try + { + var unpooledAdmin = ScratchPostgres.UnpooledAdminConnectionString(baseConnectionString); + await using var admin = new NpgsqlConnection(unpooledAdmin); + await admin.OpenAsync(ct); + await CreateDatabaseAsync(admin, name, ct); + + var inDatabase = new NpgsqlConnectionStringBuilder(baseConnectionString) { Database = name, Pooling = false }.ConnectionString; + await using (var connection = new NpgsqlConnection(inDatabase)) + { + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + Assert.SkipUnless(await LiveTimescaleProbe.TryEnableAsync(inDatabase, ct), + "TimescaleDB is not available on this PostgreSQL, so there are no jobs for the sweep to quiesce."); + + /* The product's own policies, then one job that is still running when the sweep arrives. */ + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + foreach (var sql in new[] + { + "CREATE PROCEDURE public.sweep_quiesce_slow_job(job_id int, config jsonb) LANGUAGE plpgsql AS $$ BEGIN PERFORM pg_sleep(8); END $$", + "SELECT add_job('public.sweep_quiesce_slow_job', interval '1 hour', initial_start => now())", + }) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(ct); + } + } + + /* The slow job's worker (a "User-Defined Action", unlike the quick policy workers that run first) is attached to + the database, and no client session is: that is what the sweep looks for. */ + var workers = 0L; + for (var poll = 0; poll < 100 && workers == 0; poll++) + { + workers = await SlowJobWorkerCountAsync(admin, name, ct); + if (workers == 0) + { + await Task.Delay(TimeSpan.FromMilliseconds(100), ct); + } + } + + Assert.True(workers > 0, "the slow job never started, so this test would prove nothing about the quiesce."); + for (var poll = 0; poll < 50 && await ClientSessionCountAsync(admin, name, ct) > 0; poll++) + { + await Task.Delay(TimeSpan.FromMilliseconds(100), ct); + } + + Assert.Equal(0L, await ClientSessionCountAsync(admin, name, ct)); + + /* How long the worker has run, by the server's clock: it has 8 seconds minus this left. */ + await using (var age = new NpgsqlCommand( + @"SELECT extract(epoch FROM now() - min(backend_start))::float8 FROM pg_stat_activity + WHERE datname = $1 AND backend_type LIKE 'User-Defined Action%'", admin)) + { + age.Parameters.AddWithValue(name); + workerAge = (double)(await age.ExecuteScalarAsync(ct))!; + } + + Assert.True(workerAge < 5, $"the job worker is already {workerAge:F1} s into its 8 s sleep, too little left of it for the sweep to find the worker still alive."); + + var workerPid = await WorkerPidAsync(admin, name, ct); + + /* Watch the order, not the clock: a monitor session polls pg_stat_activity (about every 20 ms) for the whole + sweep. Each look is one statement, so it sees the worker and the sweeper's DROP DATABASE together or not + at all. A correct sweep never has both alive at one look: the worker is gone before the DROP starts. */ + var watch = new SweepWatch(); + using var watchStop = CancellationTokenSource.CreateLinkedTokenSource(ct); + await using var monitor = new NpgsqlConnection(unpooledAdmin); + await monitor.OpenAsync(ct); + var watcher = WatchAsync(monitor, workerPid, name, watch, watchStop.Token); + + var log = new List(); + IReadOnlyList dropped; + try + { + watch.SweepStarted = true; + dropped = await ScratchDatabaseSweep.SweepAsync( + baseConnectionString!, now, ScratchDatabaseSweep.AbandonedAfter, log.Add, ct); + } + finally + { + await watchStop.CancelAsync(); + await watcher; + } + + Assert.Contains(name, dropped); + Assert.True(watch.WorkerAliveAfterStart > 0, + $"the monitor never saw the job worker (pid {workerPid}) alive after the sweep started ({watch.Looks} looks), so this run proves nothing about the quiesce: the scheduler ended the worker before the sweep began."); + Assert.True(watch.Overlaps == 0, + $"the sweep dropped the database while the job worker (pid {workerPid}) was still alive, at {watch.Overlaps} of {watch.Looks} looks: it did not wait the worker out."); + Assert.Contains(log, line => line.Contains(name, StringComparison.Ordinal) && line.StartsWith("Dropped", StringComparison.Ordinal)); + Assert.Empty(await ExistingAsync(baseConnectionString!, new[] { name }, ct)); + Assert.Equal(0L, await ScratchPostgres.JobWorkerCountAsync(admin, name, ct)); + bodySucceeded = true; + } + finally + { + await DropIfStillThereAsync(baseConnectionString!, name, bodySucceeded); + } + } + + /// What the monitor session saw while the sweep ran. The test sets ; only the watcher writes the counts. + private sealed class SweepWatch + { + public volatile bool SweepStarted; + public int Looks; + public int WorkerAliveAfterStart; + public int Overlaps; + } + + private static async Task WorkerPidAsync(NpgsqlConnection admin, string name, CancellationToken ct) + { + await using var pid = new NpgsqlCommand( + "SELECT min(pid) FROM pg_stat_activity WHERE datname = $1 AND backend_type LIKE 'User-Defined Action%'", admin); + pid.Parameters.AddWithValue(name); + return await pid.ExecuteScalarAsync(ct) is int found ? found : 0; + } + + /// + /// Polls until fires. One statement per look, so both facts come from one snapshot: the job + /// worker with this pid is alive, and another backend is running the sweep's DROP DATABASE for this name. + /// + private static async Task WatchAsync(NpgsqlConnection monitor, int workerPid, string name, SweepWatch watch, CancellationToken stop) + { + /* The pattern is written DROP%DATABASE% so the drop-site census (UnpooledDropConnectionCensusTests) does not take this look-up for a drop. */ + const string Look = @"SELECT + EXISTS (SELECT 1 FROM pg_stat_activity WHERE pid = $1 AND backend_type LIKE 'User-Defined Action%'), + EXISTS (SELECT 1 FROM pg_stat_activity WHERE pid <> pg_backend_pid() AND state = 'active' + AND query LIKE 'DROP%DATABASE%' AND strpos(query, $2) > 0)"; + try + { + while (!stop.IsCancellationRequested) + { + var started = watch.SweepStarted; + await using (var look = new NpgsqlCommand(Look, monitor)) + { + look.Parameters.AddWithValue(workerPid); + look.Parameters.AddWithValue(name); + await using var reader = await look.ExecuteReaderAsync(stop); + await reader.ReadAsync(stop); + var workerAlive = reader.GetBoolean(0); + var dropRunning = reader.GetBoolean(1); + watch.Looks++; + if (started && workerAlive) + { + watch.WorkerAliveAfterStart++; + } + + if (workerAlive && dropRunning) + { + watch.Overlaps++; + } + } + + /* No pause between looks: without the quiesce, the stretch where the DROP is running and the worker is not yet + gone lasts only the milliseconds TimescaleDB takes to end the worker, and a 20 ms pause missed it. */ + await Task.Yield(); + } + } + catch (OperationCanceledException) + { + /* The sweep returned; the last look was cut short. */ + } + } + + private static async Task SlowJobWorkerCountAsync(NpgsqlConnection admin, string name, CancellationToken ct) + { + await using var count = new NpgsqlCommand( + "SELECT count(*) FROM pg_stat_activity WHERE datname = $1 AND backend_type LIKE 'User-Defined Action%'", admin); + count.Parameters.AddWithValue(name); + return (long)(await count.ExecuteScalarAsync(ct))!; + } + + private static async Task ClientSessionCountAsync(NpgsqlConnection admin, string name, CancellationToken ct) + { + await using var count = new NpgsqlCommand( + "SELECT count(*) FROM pg_stat_activity WHERE datname = $1 AND backend_type = 'client backend'", admin); + count.Parameters.AddWithValue(name); + return (long)(await count.ExecuteScalarAsync(ct))!; + } + + [Fact] + public async Task ADisposedScratchDatabase_IsGone_AndNoLongerRemembered() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to a Postgres connection string to run the live scratch sweep test."); + + var ct = TestContext.Current.CancellationToken; + var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var name = scratch.DatabaseName; + var bodySucceeded = false; + try + { + Assert.True(ScratchDatabaseSweep.IsFactoryName(name), name); + Assert.True(ScratchPostgres.IsRemembered(name)); + Assert.Single(await ExistingAsync(baseConnectionString!, new[] { name }, ct)); + + await scratch.DisposeAsync(); + + Assert.Empty(await ExistingAsync(baseConnectionString!, new[] { name }, ct)); + Assert.False(ScratchPostgres.IsRemembered(name)); + bodySucceeded = true; + } + finally + { + await DropIfStillThereAsync(baseConnectionString!, name, bodySucceeded); + } + } + + /// + /// A test that fails before it can dispose, or a setup helper that throws between the create and the return, leaves + /// its database undisposed. The process-exit drain exists for exactly that; this leaves one undisposed on purpose + /// and runs the drain's own drop for just that name (the drain itself would also drop the databases other tests, + /// running in parallel, still use). + /// + [Fact] + public async Task ADatabaseNobodyDisposed_IsDroppedByTheDrain_AndForgottenAfterwards() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to a Postgres connection string to run the live scratch sweep test."); + + var ct = TestContext.Current.CancellationToken; + var leaked = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); // deliberately never disposed + var name = leaked.DatabaseName; + var bodySucceeded = false; + try + { + Assert.True(ScratchPostgres.IsRemembered(name), "a database nobody disposed must still be queued for the drain"); + Assert.Single(await ExistingAsync(baseConnectionString!, new[] { name }, ct)); + + await ScratchPostgres.DropRememberedAsync(name); + + Assert.Empty(await ExistingAsync(baseConnectionString!, new[] { name }, ct)); + Assert.False(ScratchPostgres.IsRemembered(name)); + bodySucceeded = true; + } + finally + { + await DropIfStillThereAsync(baseConnectionString!, name, bodySucceeded); + } + } + + /* The names are built in this class from digits and lowercase hex (or are deliberate near misses of that shape), + never from input, so they are safe as quoted identifiers. */ + private static async Task CreateDatabaseAsync(NpgsqlConnection admin, string name, CancellationToken ct) + { + await using var create = new NpgsqlCommand($"CREATE DATABASE \"{name}\"", admin); + await create.ExecuteNonQueryAsync(ct); + } + + private static async Task> ExistingAsync(string connectionString, IEnumerable names, CancellationToken ct) + { + var found = new List(); + await using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await using var command = new NpgsqlCommand("SELECT datname FROM pg_database WHERE datname = ANY($1)", connection); + command.Parameters.AddWithValue(names.ToArray()); + await using var reader = await command.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + found.Add(reader.GetString(0)); + } + + return found; + } + + private static Task DropIfStillThereAsync(string connectionString, string name, bool bodySucceeded) => + LiveStoreCleanup.RunAsync(ScratchPostgres.UnpooledAdminConnectionString(connectionString), bodySucceeded, async (cleanup, cleanupCt) => + { + /* No TimescaleDB job worker is left in the database the FORCE drop below kills (#5480). */ + await ScratchPostgres.QuiesceTimescaleJobsAsync(connectionString, name); + await using var drop = new NpgsqlCommand($"DROP DATABASE IF EXISTS \"{name}\" WITH (FORCE)", cleanup); + await drop.ExecuteNonQueryAsync(cleanupCt); + }); +} diff --git a/Darling/Darling.Tests/ScratchDatabaseSweepTests.cs b/Darling/Darling.Tests/ScratchDatabaseSweepTests.cs new file mode 100644 index 0000000000..763a3e4364 --- /dev/null +++ b/Darling/Darling.Tests/ScratchDatabaseSweepTests.cs @@ -0,0 +1,129 @@ +using System; +using System.Globalization; +using System.Linq; +using Xunit; + +namespace Darling.Tests; + +/// +/// The decision half of the start-of-run sweep (#4981): which database names it may ever drop. The sweep is aimed at +/// whatever the test connection string names, so the property that matters is the negative one, that a name that merely +/// looks like a scratch database is never selected. The live half, against a real cluster, is +/// . +/// +public sealed class ScratchDatabaseSweepTests +{ + private static readonly DateTime Now = new(2026, 10, 3, 12, 0, 0, DateTimeKind.Utc); + + private const string OldStamp = "20260101000000"; + + private const string Id = "0123456789ab"; + + private static string Name(string stamp = OldStamp, string id = Id) => "darling_scratch_" + stamp + "_" + id; + + [Fact] + public void AFreshName_IsAFactoryName_AndCarriesItsCreationTime() + { + var created = new DateTime(2026, 10, 3, 11, 59, 58, DateTimeKind.Utc); + + var name = ScratchDatabaseSweep.NewName(created); + + Assert.True(ScratchDatabaseSweep.IsFactoryName(name), name); + Assert.True(ScratchDatabaseSweep.TryParseCreatedUtc(name, out var parsed)); + Assert.Equal(created, parsed); + Assert.Equal(DateTimeKind.Utc, parsed.Kind); + Assert.True(name.Length <= 63, "a PostgreSQL identifier is at most 63 bytes"); + } + + [Fact] + public void TwoFreshNames_DifferEvenInTheSameSecond() + { + Assert.NotEqual(ScratchDatabaseSweep.NewName(Now), ScratchDatabaseSweep.NewName(Now)); + } + + /// + /// Every one of these looks like a scratch database to a person skimming a database list, and none may be + /// dropped, however old it looks. The first is what the factory minted before names carried a stamp: it has no + /// age to compare, so the sweep leaves it. + /// + [Theory] + [InlineData("darling_scratch_0123456789ab")] // the earlier shape, no stamp + [InlineData("darling_scratch_20260101000000_0123456789a")] // 11 hex + [InlineData("darling_scratch_20260101000000_0123456789abc")] // 13 hex + [InlineData("darling_scratch_20260101000000_0123456789AB")] // uppercase hex + [InlineData("darling_scratch_20260101000000_0123456789ag")] // not hex + [InlineData("darling_scratch_20260101000000_0123456789ab_x")] // trailing extra + [InlineData("darling_scratch_20260101000000_0123456789ab\n")] // trailing newline + [InlineData("darling_scratch_2026010100000_0123456789ab")] // 13-digit stamp + [InlineData("darling_scratch_202601010000000_0123456789ab")] // 15-digit stamp + [InlineData("darling_scratch_20261301000000_0123456789ab")] // month 13 is not a date + [InlineData("darling_scratch_20260230000000_0123456789ab")] // 30 February is not a date + [InlineData("darling_scratch_20260101250000_0123456789ab")] // hour 25 is not a time + [InlineData("darling_scratch_2026010100000a_0123456789ab")] // a letter in the stamp + [InlineData("darling_scratch_\u0662\u0660\u0662\u0666\u0660\u0661\u0660\u0661\u0660\u0660\u0660\u0660\u0660\u0660_0123456789ab")] // non-ASCII digits in the stamp + [InlineData("Darling_Scratch_20260101000000_0123456789ab")] // different case + [InlineData("xdarling_scratch_20260101000000_0123456789ab")] // prefixed + [InlineData(" darling_scratch_20260101000000_0123456789ab")] // leading space + [InlineData("darling_scratch__0123456789ab")] // no stamp, extra underscore + [InlineData("darling_scratch_20260101000000-0123456789ab")] // wrong separator + [InlineData("darling_scratch20260101000000_0123456789ab")] // missing separator + [InlineData("darlingtest_scratch_20260101000000_0123456789ab")] // another family's prefix + [InlineData("darling_upgrade_ladder_test_3_7_0")] // another test fixture's database + [InlineData("darlingtest_4981")] + [InlineData("darling")] + [InlineData("postgres")] + [InlineData("template0")] + [InlineData("template1")] + [InlineData("")] + public void ANameThatLooksCloseButIsNotTheFactoryPattern_IsNeverSelected(string name) + { + Assert.False(ScratchDatabaseSweep.IsFactoryName(name), name); + Assert.Empty(ScratchDatabaseSweep.SelectAbandoned(new[] { name }, Now, TimeSpan.Zero)); + Assert.Empty(ScratchDatabaseSweep.SelectAbandoned(new[] { name }, DateTime.MaxValue.AddDays(-1), TimeSpan.Zero)); + } + + [Fact] + public void ANullName_IsNotAFactoryName() + { + Assert.False(ScratchDatabaseSweep.IsFactoryName(null)); + } + + [Fact] + public void OnlyFactoryNamesOlderThanTheThreshold_AreSelected() + { + var threshold = ScratchDatabaseSweep.AbandonedAfter; + string Stamped(TimeSpan age) => Name(Now.Subtract(age).ToString(ScratchDatabaseSweep.StampFormat, CultureInfo.InvariantCulture)); + + var exactlyAtTheThreshold = Stamped(threshold); + var justPast = Stamped(threshold + TimeSpan.FromSeconds(1)); + var daysOld = Stamped(TimeSpan.FromDays(9)); + var justInside = Stamped(threshold - TimeSpan.FromSeconds(1)); + var young = Stamped(TimeSpan.FromMinutes(2)); + var aheadOfTheClock = Name(Now.AddHours(3).ToString(ScratchDatabaseSweep.StampFormat, CultureInfo.InvariantCulture), "fedcba987654"); + + var selected = ScratchDatabaseSweep.SelectAbandoned( + new[] { young, justInside, aheadOfTheClock, exactlyAtTheThreshold, justPast, daysOld, "postgres", "darling_scratch_0123456789ab" }, + Now, + threshold); + + Assert.Equal( + new[] { exactlyAtTheThreshold, justPast, daysOld }.OrderBy(n => n, StringComparer.Ordinal), + selected.OrderBy(n => n, StringComparer.Ordinal)); + } + + [Fact] + public void ARepeatedName_IsSelectedOnce() + { + var selected = ScratchDatabaseSweep.SelectAbandoned(new[] { Name(), Name() }, Now, ScratchDatabaseSweep.AbandonedAfter); + + Assert.Equal(new[] { Name() }, selected); + } + + [Fact] + public void TheThreshold_IsLongerThanAnyTestRunsAScratchDatabase() + { + /* A test keeps its database for the length of the test. The longest live tests here run for minutes, so a + margin under ten would put a slow neighbour's databases at risk. */ + Assert.True(ScratchDatabaseSweep.AbandonedAfter >= TimeSpan.FromMinutes(10)); + } +} diff --git a/Darling/Darling.Tests/ScratchPostgres.cs b/Darling/Darling.Tests/ScratchPostgres.cs index e65d5aa72d..9de51d3496 100644 --- a/Darling/Darling.Tests/ScratchPostgres.cs +++ b/Darling/Darling.Tests/ScratchPostgres.cs @@ -1,7 +1,11 @@ using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Linq; using System.Threading; using System.Threading.Tasks; using Npgsql; +using Xunit; namespace Darling.Tests; @@ -14,6 +18,21 @@ namespace Darling.Tests; /// dispose with WITH (FORCE) so a lingering connection can never wedge the drop. /// Pooling=false on the scratch connection string keeps in-process pooled connections from /// pinning the database in the first place. +/// +/// A scratch database must not outlive its run (#4981). Each one that has the TimescaleDB extension +/// holds a background-worker slot for its scheduler, and a long-lived local cluster has only a few, so databases +/// left behind eventually fail unrelated tests with "limit of 8 exceeded". Three layers keep that from happening. +/// +/// The drop on dispose is retried and a failure is reported, no longer swallowed. +/// Every database this process creates is remembered until its drop succeeds. Whatever is still remembered +/// when the process exits (a test that failed before it could dispose, a drop that kept failing) is dropped then, +/// and named on stderr together with the test that created it. +/// The first create against a cluster sweeps that cluster for databases a killed run left behind +/// (), because a killed process runs neither of the layers above. The sweep runs on +/// whatever cluster DARLING_TEST_PG names, and that is safe because it drops only a database whose name matches +/// the factory's anchored pattern, that is at least 30 minutes old and idle, and that is not its own, with a plain +/// DROP DATABASE. +/// /// internal sealed class ScratchPostgres : IAsyncDisposable { @@ -37,17 +56,44 @@ private ScratchPostgres(string adminConnectionString, string databaseName, strin ConnectionString = connectionString; } + /// + /// Every database this process created and has not yet dropped: name to the connection string that drops it, and + /// the test that created it. An entry leaves only when its drop succeeds. + /// + private static readonly ConcurrentDictionary Remembered = new(); + + /// One start-of-run sweep per cluster per process, keyed by host and port. + private static readonly ConcurrentDictionary> Sweeps = new(); + + private static int _exitDrainHooked; + + /// Drop attempts on dispose: a transient refusal (a session that has not finished closing) should not leak. + private const int DropAttempts = 3; + public static async Task CreateAsync(string baseConnectionString, CancellationToken cancellationToken) { - /* Hex-only generated name: safe to interpolate as a quoted identifier. */ - var databaseName = "darling_scratch_" + Guid.NewGuid().ToString("N")[..12]; + HookExitDrain(); + await SweepOncePerClusterAsync(baseConnectionString, cancellationToken); - await using (var admin = new NpgsqlConnection(baseConnectionString)) + /* The factory's pattern (prefix, creation stamp, 12 hex): safe to interpolate as a quoted identifier. */ + var databaseName = ScratchDatabaseSweep.NewName(DateTime.UtcNow); + + /* Remembered BEFORE the create, so a failure or cancellation anywhere after it still has a drop queued; the + exit drain uses IF EXISTS, so a name whose create never ran costs nothing. */ + Remembered[databaseName] = (baseConnectionString, CurrentTestName()); + + try { + await using var admin = new NpgsqlConnection(UnpooledAdminConnectionString(baseConnectionString)); await admin.OpenAsync(cancellationToken); await using var create = new NpgsqlCommand($"CREATE DATABASE \"{databaseName}\"", admin); await create.ExecuteNonQueryAsync(cancellationToken); } + catch + { + await TryDropAsync(baseConnectionString, databaseName); + throw; + } var builder = new NpgsqlConnectionStringBuilder(baseConnectionString) { @@ -55,6 +101,10 @@ public static async Task CreateAsync(string baseConnectionStrin Pooling = false, }; + /* The base string can already carry the session pin (LiveStoreSessionTimeZone adds it to DARLING_TEST_PG), + so the unpinned string drops it. */ + builder.Remove("Timezone"); + /* Pinned to UTC the way every product store connection is, so a cluster whose default zone is behind UTC cannot skew timestamp round trips (a chunk's range_end read back through a ::timestamp cast). */ return new ScratchPostgres( @@ -64,19 +114,483 @@ cannot skew timestamp round trips (a chunk's range_end read back through a ::tim builder.ConnectionString); } + /// + /// Drops the database, retrying a transient refusal. A drop that still fails is reported and the database stays + /// remembered for the exit drain; it does not fail the test, because failing a passing test in its cleanup would + /// invert the signal, and the layers behind this one still remove it. + /// public async ValueTask DisposeAsync() { + for (var attempt = 1; attempt <= DropAttempts; attempt++) + { + try + { + await DropAsync(_adminConnectionString, DatabaseName); + return; + } + catch (Exception ex) when (attempt < DropAttempts) + { + Report($"Scratch database {DatabaseName}: drop attempt {attempt} of {DropAttempts} failed ({ex.GetType().Name}: {ex.Message}); retrying."); + await Task.Delay(TimeSpan.FromMilliseconds(250 * attempt)); + } + catch (Exception ex) + { + Report($"Scratch database {DatabaseName}: drop failed after {DropAttempts} attempts ({ex.GetType().Name}: {ex.Message}); it stays queued for the exit drain."); + } + } + } + + /// True while this process still has a drop pending for . + internal static bool IsRemembered(string databaseName) => Remembered.ContainsKey(databaseName); + + /// + /// Drops the named databases if this process still remembers them, as the exit drain does for everything left. + /// Exposed by name so a test can prove the drain on a database it leaked on purpose without dropping the databases + /// that other tests, running in parallel, are still using. + /// + internal static async Task DropRememberedAsync(params string[] databaseNames) + { + foreach (var name in databaseNames) + { + if (Remembered.TryGetValue(name, out var entry)) + { + await DropAsync(entry.AdminConnectionString, name); + } + } + } + + /// Total time may spend before it lets the drop go ahead. + internal static readonly TimeSpan QuiesceCap = TimeSpan.FromSeconds(10); + + /// + /// Unschedules every TimescaleDB job in , then waits (up to + /// ) for the job workers already running there to finish, so the + /// DROP DATABASE ... WITH (FORCE) that follows does not SIGTERM a policy worker mid-run. CI saw the + /// PostgreSQL postmaster log "terminated by exception 0xC0000005" for a TimescaleDB worker, twice, both times + /// with a scratch-database FORCE drop in flight (PR #5480); no worker is running in the dropped database + /// once this returns, so the drop has nothing to kill. + /// Best-effort and bounded: it never throws, never fails a test, and never skips the drop. A caller's + /// ends it early too, without a message (the exit drain passes its deadline). A database + /// without the extension, or one already gone, returns at once. It does NOT call + /// _timescaledb_functions.stop_background_workers(), which SIGTERMs running jobs, the very kill this + /// avoids. + /// + internal static async Task QuiesceTimescaleJobsAsync(string adminConnectionString, string databaseName, CancellationToken cancellationToken = default) + { + using var cap = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cap.CancelAfter(QuiesceCap); try { - await using var admin = new NpgsqlConnection(_adminConnectionString); - await admin.OpenAsync(); - await using var drop = new NpgsqlCommand($"DROP DATABASE IF EXISTS \"{DatabaseName}\" WITH (FORCE)", admin); - await drop.ExecuteNonQueryAsync(); + var scratchBuilder = new NpgsqlConnectionStringBuilder(adminConnectionString) + { + Database = databaseName, + Pooling = false, + Timeout = 5, + }; + var adminBuilder = new NpgsqlConnectionStringBuilder(adminConnectionString) { Pooling = false, Timeout = 5 }; + + await using (var scratch = new NpgsqlConnection(scratchBuilder.ConnectionString)) + { + await scratch.OpenAsync(cap.Token); + await using var probe = new NpgsqlCommand("SELECT to_regclass('_timescaledb_config.bgw_job') IS NOT NULL", scratch); + if (!(bool)(await probe.ExecuteScalarAsync(cap.Token))!) + { + return; + } + + await using var unschedule = new NpgsqlCommand( + "SELECT alter_job(job_id, scheduled => false) FROM timescaledb_information.jobs", scratch); + await unschedule.ExecuteNonQueryAsync(cap.Token); + } + + await using var admin = new NpgsqlConnection(adminBuilder.ConnectionString); + await admin.OpenAsync(cap.Token); + /* Two empty polls in a row: a worker the scheduler registered just before the unschedule can still be + starting up and not yet show in pg_stat_activity on the first look. */ + var emptyPolls = 0; + while (!cap.IsCancellationRequested) + { + emptyPolls = await JobWorkerCountAsync(admin, databaseName, cap.Token) == 0 ? emptyPolls + 1 : 0; + if (emptyPolls >= 2) + { + return; + } + + await Task.Delay(TimeSpan.FromMilliseconds(100), cap.Token); + } + } + catch (PostgresException ex) when (ex.SqlState == PostgresErrorCodes.InvalidCatalogName) + { + /* The database is already gone: nothing to quiesce, and the drop below is IF EXISTS. */ + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + /* The caller's own deadline (the exit drain's), not the cap: the caller decides whether the drop still runs. */ + } + catch (Exception ex) + { + /* The cap, a missing database or a refused connect all land here. The drop that follows is the + test's cleanup and it still runs; the retries behind it are unchanged. */ + Console.Error.WriteLine($"Scratch database {databaseName}: TimescaleDB jobs not quiesced before the drop ({ex.GetType().Name}: {ex.Message})."); + } + } + + /// + /// How many TimescaleDB job workers (and any other non-client, non-scheduler, non-autovacuum backend) are + /// attached to . Job workers show up as e.g. Retention Policy [1038]. + /// + internal static async Task JobWorkerCountAsync(NpgsqlConnection admin, string databaseName, CancellationToken cancellationToken) + { + await using var count = new NpgsqlCommand( + @"SELECT count(*) FROM pg_stat_activity + WHERE datname = $1 + AND backend_type NOT IN ('client backend', 'TimescaleDB Background Worker Scheduler', 'autovacuum worker')", + admin); + count.Parameters.AddWithValue(databaseName); + return (long)(await count.ExecuteScalarAsync(cancellationToken))!; + } + + private static async Task DropAsync(string adminConnectionString, string databaseName) + { + await QuiesceTimescaleJobsAsync(adminConnectionString, databaseName); + await using var admin = new NpgsqlConnection(UnpooledAdminConnectionString(adminConnectionString)); + await admin.OpenAsync(); + await using var drop = new NpgsqlCommand($"DROP DATABASE IF EXISTS \"{databaseName}\" WITH (FORCE)", admin); + await drop.ExecuteNonQueryAsync(); + Remembered.TryRemove(databaseName, out _); + } + + private static async Task TryDropAsync(string adminConnectionString, string databaseName) + { + try + { + await DropAsync(adminConnectionString, databaseName); } catch { - /* Best-effort: a leaked scratch database on a throwaway test cluster is harmless, - and failing a passing test in its cleanup would invert the signal. */ + /* The create itself failed, so the original exception is the story; the exit drain retries this drop. */ } } + + /// + /// Subscribes the exit drain once. A process-exit handler is the one place that runs for a failed test, a test + /// that never disposed, and an interrupted run alike; it cannot run for a killed process, which is what the + /// start-of-run sweep is for. + /// + private static void HookExitDrain() + { + if (Interlocked.Exchange(ref _exitDrainHooked, 1) == 0) + { + AppDomain.CurrentDomain.ProcessExit += (_, _) => DropEverythingRemembered(); + } + } + + private static void DropEverythingRemembered() + { + var pending = new List(); + foreach (var (name, entry) in Remembered.ToArray()) + { + pending.Add(new RememberedDatabase(name, entry.AdminConnectionString, entry.Creator)); + } + + DropRememberedAtExit(pending); + } + + /// One database the exit drain was asked to drop: its name, the connection string that drops it, and the test that created it. + internal readonly record struct RememberedDatabase(string Name, string AdminConnectionString, string Creator); + + /// + /// What an exit drain did: connections it tried to open, databases it dropped, drops that failed, databases it left + /// alone because their cluster did not answer, and databases it had not finished when its deadline passed. + /// + internal readonly record struct ExitDrainOutcome(int ConnectAttempts, int Dropped, int Failed, int Skipped, int Unfinished); + + /// + /// How long the exit drain waits to connect to a cluster, in seconds. The default is 15, and the drain ran once per + /// remembered database, so a cluster stopped before the process exited cost 15 seconds for each database the run + /// still held (#4981). + /// + internal const int ExitDrainConnectTimeoutSeconds = 3; + + /// + /// How long one drop at exit may run, in seconds. This stays at the library default on purpose: a dead cluster + /// never gets as far as a command (its connect fails first), and a drop on a live cluster ends with an immediate + /// checkpoint that took 11 seconds on a loaded local cluster, so a shorter limit cancelled a drop that was working + /// and left the very database the drain exists to remove. + /// + internal const int ExitDrainCommandTimeoutSeconds = 30; + + /// + /// #5549: the admin connection string with pooling off, for the CREATE DATABASE and the DROP DATABASE. The admin + /// string is the caller's DARLING_TEST_PG, so a pooled admin connection went back into the pool every live + /// test opens, and the next test took the backend that had just run the drop. CI's log showed one backend PID run + /// DROP DATABASE ... WITH (FORCE) on a scratch database and, half a second later, the + /// CALL run_job(...) of the retired-baseline race test. With pooling off, the backend that ran a create or a + /// drop ends when the connection closes, and no other test ever runs on it. + /// + internal static string UnpooledAdminConnectionString(string adminConnectionString) => + new NpgsqlConnectionStringBuilder(adminConnectionString) { Pooling = false }.ConnectionString; + + /// The admin connection string with the exit drain's own connect and command timeouts, and no pooling: one connection per drop. + internal static string ExitDrainConnectionString(string adminConnectionString) => + new NpgsqlConnectionStringBuilder(adminConnectionString) + { + Timeout = ExitDrainConnectTimeoutSeconds, + CommandTimeout = ExitDrainCommandTimeoutSeconds, + Pooling = false, + }.ConnectionString; + + /// + /// How long the whole exit drain may take, in seconds, in-flight drops included. The drain is a ProcessExit handler, + /// and xUnit v3 under Microsoft.Testing.Platform gives that stage 11 seconds after the run returns (one second, then + /// its shutdownForegroundThreadWaitSeconds default of 10) before it prints "[FATAL ERROR] Foreground threads + /// were left running, forcing process exit" and exits with code 1. No thread was left running: the 2026-10-08 + /// nightly passed every test and still failed because this drain, then dropping 39 leaked databases one at a time + /// with a TimescaleDB quiesce before each (#5480), was still at work when that window closed. The drain returns when + /// this passes even with a drop in flight (one DROP DATABASE waited 26 seconds for its checkpoint on a local rig), and + /// names every database it did not finish; the next run's start-of-run sweep removes those, or they go with the + /// throwaway cluster. Tests that mint a database must still drop it themselves (see + /// ScratchDatabaseDisposalCensusTests); this budget only keeps a leak from failing a run. + /// + internal const int ExitDrainBudgetSeconds = 6; + + /// How many databases the exit drain quiesces and drops at once. Each holds at most two connections at a time. + internal const int ExitDrainParallelism = 16; + + /// + /// The exit drain's body, taking its list as a parameter so a test can run it without a process exit and without + /// touching the databases that other tests, running in parallel, still hold. + /// + /// + /// Clusters drain side by side, and within a live cluster up to databases + /// drop at once, all under one (default seconds). + /// When it passes, the drain returns and names every + /// database it did not finish. A drop already sent is not cancelled (an interrupted DROP DATABASE can leave an + /// invalid database behind), but no new connect, quiesce or drop starts. + /// The first database on each cluster goes alone and probes it. A connect that fails without the server + /// answering (a stopped cluster, a refused or timed-out connection) marks the cluster dead, so N databases on it + /// cost one short wait, not N, and the rest are skipped by name. A server that answers with an error (a bad + /// password, a missing database) is not a dead cluster, and a failed drop on a live cluster never stops the + /// others. + /// + internal static ExitDrainOutcome DropRememberedAtExit(IEnumerable remembered, TimeSpan? budget = null) + { + var deadline = budget ?? TimeSpan.FromSeconds(ExitDrainBudgetSeconds); + var databases = remembered.ToList(); + var drain = new ExitDrain(databases.Count); + using var stop = new CancellationTokenSource(deadline); + using var gate = new SemaphoreSlim(ExitDrainParallelism); + + var clusters = databases + .Select((db, index) => (Db: db, Index: index)) + .GroupBy(entry => ClusterKeyOrOwn(entry.Db), StringComparer.Ordinal) + .Select(cluster => Task.Run(() => DrainClusterAsync(cluster.ToList(), drain, gate, stop.Token))) + .ToArray(); + + try + { + Task.WaitAll(clusters, deadline); + } + catch (AggregateException) + { + /* Every per-database step catches its own failure; nothing here should throw, and nothing may stop the exit. */ + } + + /* One snapshot: a drop sent before the deadline can still finish after this line, and the counts and the names + printed below must agree. */ + var states = drain.Snapshot(); + for (var i = 0; i < databases.Count; i++) + { + if (states[i] == ExitDrain.Unsettled) + { + var db = databases[i]; + Console.Error.WriteLine($"Scratch database {db.Name} outlived its test ({db.Creator}) and was not dropped within the exit drain's {deadline.TotalSeconds:0.#}-second limit; the next run's start-of-run sweep removes it."); + } + } + + return new ExitDrainOutcome( + Volatile.Read(ref drain.ConnectAttempts), + states.Count(state => state == ExitDrain.Dropped), + states.Count(state => state == ExitDrain.Failed), + states.Count(state => state == ExitDrain.Skipped), + states.Count(state => state == ExitDrain.Unsettled)); + } + + /// The database's cluster by host and port, or a key of its own when its connection string does not parse, so it fails alone. + private static string ClusterKeyOrOwn(RememberedDatabase db) + { + try + { + return ClusterKey(db.AdminConnectionString); + } + catch (Exception) + { + return "unparsed:" + db.Name; + } + } + + private static async Task DrainClusterAsync(List<(RememberedDatabase Db, int Index)> cluster, ExitDrain drain, SemaphoreSlim gate, CancellationToken stop) + { + if (await DropOneAtExitAsync(cluster[0].Db, cluster[0].Index, drain, gate, stop) == ExitDrop.ClusterDead) + { + foreach (var (db, index) in cluster.Skip(1)) + { + drain.Settle(index, ExitDrain.Skipped); + Console.Error.WriteLine($"Scratch database {db.Name} outlived its test ({db.Creator}) and was not dropped at exit: its cluster did not answer an earlier connect."); + } + + return; + } + + await Task.WhenAll(cluster.Skip(1).Select(entry => DropOneAtExitAsync(entry.Db, entry.Index, drain, gate, stop))); + } + + private static async Task DropOneAtExitAsync(RememberedDatabase db, int index, ExitDrain drain, SemaphoreSlim gate, CancellationToken stop) + { + try + { + await gate.WaitAsync(stop); + } + catch (OperationCanceledException) + { + return ExitDrop.OutOfTime; + } + + try + { + Interlocked.Increment(ref drain.ConnectAttempts); + var connectionString = ExitDrainConnectionString(db.AdminConnectionString); + await using var admin = new NpgsqlConnection(connectionString); + try + { + await admin.OpenAsync(stop); + } + catch (OperationCanceledException) when (stop.IsCancellationRequested) + { + return ExitDrop.OutOfTime; + } + catch (Exception ex) when (ex is not PostgresException) + { + drain.Settle(index, ExitDrain.Failed); + Console.Error.WriteLine($"Scratch database {db.Name} outlived its test ({db.Creator}) and could not be dropped at exit: {ex.Message}"); + return ExitDrop.ClusterDead; + } + + /* Bounded by its own cap and by the drain's deadline, and it never throws. */ + await QuiesceTimescaleJobsAsync(connectionString, db.Name, stop); + if (stop.IsCancellationRequested) + { + return ExitDrop.OutOfTime; + } + + await using var drop = new NpgsqlCommand($"DROP DATABASE IF EXISTS \"{db.Name}\" WITH (FORCE)", admin); + await drop.ExecuteNonQueryAsync(CancellationToken.None); + Remembered.TryRemove(db.Name, out _); + drain.Settle(index, ExitDrain.Dropped); + Console.Error.WriteLine($"Scratch database {db.Name} outlived its test ({db.Creator}); dropped at process exit."); + return ExitDrop.Done; + } + catch (Exception ex) + { + drain.Settle(index, ExitDrain.Failed); + Console.Error.WriteLine($"Scratch database {db.Name} outlived its test ({db.Creator}) and could not be dropped at exit: {ex.Message}"); + return ExitDrop.Done; + } + finally + { + gate.Release(); + } + } + + /// How one database's exit drop ended, as far as the rest of its cluster is concerned. + private enum ExitDrop + { + Done, + ClusterDead, + OutOfTime, + } + + /// Each database's outcome in the drain, set once, and the connect count, safe to update from every drop at once. + private sealed class ExitDrain(int count) + { + public const int Unsettled = 0; + public const int Dropped = 1; + public const int Failed = 2; + public const int Skipped = 3; + + private readonly int[] _states = new int[count]; + + public int ConnectAttempts; + + public void Settle(int index, int state) => Interlocked.CompareExchange(ref _states[index], state, Unsettled); + + public int[] Snapshot() + { + var copy = new int[_states.Length]; + for (var i = 0; i < copy.Length; i++) + { + copy[i] = Volatile.Read(ref _states[i]); + } + + return copy; + } + } + + /// The cluster a connection string names, by host and port: what the start-of-run sweep and the exit drain each treat as one cluster. + private static string ClusterKey(string connectionString) + { + var builder = new NpgsqlConnectionStringBuilder(connectionString); + return $"{builder.Host}:{builder.Port}".ToUpperInvariant(); + } + + /// + /// The first create against a cluster sweeps it once. Every later caller waits on the same task, so no scratch + /// database is created mid-sweep, but a failed sweep never fails a test: it only means debris stays for the next run. + /// + private static async Task SweepOncePerClusterAsync(string baseConnectionString, CancellationToken cancellationToken) + { + var key = ClusterKey(baseConnectionString); + var sweep = Sweeps.GetOrAdd(key, _ => new Lazy(() => RunSweepAsync(baseConnectionString))); + await sweep.Value.WaitAsync(cancellationToken); + } + + private static async Task RunSweepAsync(string baseConnectionString) + { + try + { + using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(60)); + await ScratchDatabaseSweep.SweepAsync( + baseConnectionString, DateTime.UtcNow, ScratchDatabaseSweep.AbandonedAfter, Report, timeout.Token); + } + catch (Exception ex) + { + Report($"The start-of-run sweep for abandoned scratch databases failed and was skipped: {ex.Message}"); + } + } + + private static string CurrentTestName() + { + try + { + return TestContext.Current.Test?.TestDisplayName ?? "no test context"; + } + catch + { + return "no test context"; + } + } + + private static void Report(string message) + { + try + { + TestContext.Current.SendDiagnosticMessage(message); + } + catch + { + /* No test context to report to; the text is advisory. */ + } + + Console.Error.WriteLine(message); + } } diff --git a/Darling/Darling.Tests/ScratchPostgresExitDrainTests.cs b/Darling/Darling.Tests/ScratchPostgresExitDrainTests.cs new file mode 100644 index 0000000000..8f7f1ed352 --- /dev/null +++ b/Darling/Darling.Tests/ScratchPostgresExitDrainTests.cs @@ -0,0 +1,329 @@ +using System; +using System.Diagnostics; +using System.Net; +using System.Net.Sockets; +using System.Threading.Tasks; +using Npgsql; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4981: the exit drain must not stall the end of a test run when the cluster it was asked to clean is gone. It is +/// the last thing a run does, so with the default 15-second connect wait per remembered database, a cluster that was +/// stopped before the process exited cost 15 seconds for each database the run still held. +/// +/// The drain body takes its list as a parameter, so none of these tests needs a process exit, and none +/// touches the databases that other tests in the same process are still using. +/// +/* #1776 own-store: the dead-cluster cases never connect to a store, and the live case creates one scratch database + through ScratchPostgres, so it does not read or write the shared store's tables. */ +[Collection("timing")] +public sealed class ScratchPostgresExitDrainTests +{ + /// A port nothing listens on: bind to port 0, read the port the OS chose, release it. + private static int ClosedPort() + { + var listener = new TcpListener(IPAddress.Loopback, 0); + listener.Start(); + try + { + return ((IPEndPoint)listener.LocalEndpoint).Port; + } + finally + { + listener.Stop(); + } + } + + private static string StoppedClusterConnectionString(int port) => + new NpgsqlConnectionStringBuilder + { + Host = "127.0.0.1", + Port = port, + Username = "darling", + Database = "postgres", + Pooling = false, + }.ConnectionString; + + private static ScratchPostgres.RememberedDatabase Remembered(string connectionString, string creator) => + new($"darling_scratch_20010101000000_{Guid.NewGuid():N}"[..43], connectionString, creator); + + /// + /// A cluster that hangs: a listener that never accepts, so the OS completes each TCP handshake and the startup + /// message then waits for an answer that never comes, until the drain's connect timeout. Stop it when done. + /// + private static (TcpListener Listener, string ConnectionString) HangingCluster() + { + var listener = new TcpListener(IPAddress.Loopback, 0); + listener.Start(); + return (listener, StoppedClusterConnectionString(((IPEndPoint)listener.LocalEndpoint).Port)); + } + + /// + /// The 2026-10-08 nightly passed all 28,763 tests and still failed with "[FATAL ERROR] Foreground threads were + /// left running": this drain, a ProcessExit handler, was still dropping 39 leaked databases one at a time when + /// xUnit's 11-second end-of-run window closed. Four clusters that each hang for the full connect timeout cost + /// four timeouts in a row on the old drain (about 12 seconds, past that window) and one when they drain side by side. + /// + [Fact] + public void TheDrain_OfSeveralHangingClusters_WaitsOutTheirConnectsSideBySide() + { + var clusters = new[] { HangingCluster(), HangingCluster(), HangingCluster(), HangingCluster() }; + try + { + var databases = Array.ConvertAll(clusters, cluster => Remembered(cluster.ConnectionString, "a hanging one")); + + var clock = Stopwatch.StartNew(); + var outcome = ScratchPostgres.DropRememberedAtExit(databases); + clock.Stop(); + + Assert.True( + clock.Elapsed < TimeSpan.FromSeconds(ScratchPostgres.ExitDrainConnectTimeoutSeconds * 2), + $"the drain of 4 hanging clusters took {clock.Elapsed.TotalSeconds:0.0} seconds; one at a time costs 4 connect timeouts of {ScratchPostgres.ExitDrainConnectTimeoutSeconds} seconds"); + Assert.Equal(4, outcome.ConnectAttempts); + Assert.Equal(4, outcome.Failed); + Assert.Equal(0, outcome.Dropped); + } + finally + { + foreach (var cluster in clusters) + { + cluster.Listener.Stop(); + } + } + } + + /// + /// Whatever the clusters do, the drain returns by its deadline and names each database it did not finish, so a + /// leak, a slow drop or a hung cluster can never again run the exit past xUnit's window and fail a green run. + /// + [Fact] + public void TheDrain_ReturnsAtItsDeadline_AndCountsWhatItDidNotFinish() + { + var (listener, hanging) = HangingCluster(); + try + { + var deadline = TimeSpan.FromSeconds(1); + var clock = Stopwatch.StartNew(); + var outcome = ScratchPostgres.DropRememberedAtExit( + new[] { Remembered(hanging, "hung one"), Remembered(hanging, "hung two"), Remembered(hanging, "hung three") }, + deadline); + clock.Stop(); + + /* Under the 3-second connect timeout, so it was the deadline, not the timeout, that ended the drain. */ + Assert.True( + clock.Elapsed < TimeSpan.FromSeconds(ScratchPostgres.ExitDrainConnectTimeoutSeconds) - TimeSpan.FromMilliseconds(500), + $"the drain took {clock.Elapsed.TotalSeconds:0.0} seconds against a {deadline.TotalSeconds:0}-second deadline"); + Assert.Equal(3, outcome.Unfinished); + Assert.Equal(0, outcome.Dropped); + Assert.Equal(0, outcome.Failed); + Assert.Equal(0, outcome.Skipped); + } + finally + { + listener.Stop(); + } + } + + [Fact] + public void TheDrain_OfThreeDatabases_OnAStoppedCluster_TriesTheClusterOnce_AndEndsQuickly() + { + var stopped = StoppedClusterConnectionString(ClosedPort()); + var databases = new[] { Remembered(stopped, "test one"), Remembered(stopped, "test two"), Remembered(stopped, "test three") }; + + var clock = Stopwatch.StartNew(); + var outcome = ScratchPostgres.DropRememberedAtExit(databases); + clock.Stop(); + + /* One short wait, not three. A refused connect on Windows takes about 2 seconds, so the old drain took about 6 + here (and 45 where the connect waited out its default), against one attempt capped at the 3-second timeout. */ + Assert.True( + clock.Elapsed < TimeSpan.FromSeconds(ScratchPostgres.ExitDrainConnectTimeoutSeconds + 2), + $"the drain of 3 databases on a stopped cluster took {clock.Elapsed.TotalSeconds:0.0} seconds"); + + /* Three databases on one dead cluster cost one failed connect, not three: the other two are named and left. */ + Assert.Equal(1, outcome.ConnectAttempts); + Assert.Equal(1, outcome.Failed); + Assert.Equal(2, outcome.Skipped); + Assert.Equal(0, outcome.Dropped); + } + + [Fact] + public void TheDrain_OutOfBudget_StartsNoDrop_AndNamesEveryDatabaseItLeaves() + { + /* The test runner gives the process 10 seconds to exit once the run returns, then forces exit code 1. The + nightly of 2026-10-08 lost its publish to that with 28,763 tests green. A drain whose budget is spent + starts no connect at all: nothing is attempted, nothing is dropped, and every database is counted as + unfinished, so the exit ends on time. */ + var stopped = StoppedClusterConnectionString(ClosedPort()); + var databases = new[] { Remembered(stopped, "budget one"), Remembered(stopped, "budget two"), Remembered(stopped, "budget three") }; + + var outcome = ScratchPostgres.DropRememberedAtExit(databases, TimeSpan.Zero); + + Assert.Equal(0, outcome.ConnectAttempts); + Assert.Equal(0, outcome.Dropped); + Assert.Equal(0, outcome.Failed); + Assert.Equal(0, outcome.Skipped); + Assert.Equal(3, outcome.Unfinished); + } + + [Fact] + public void TheDrainBudget_StaysInsideTheRunnersExitWait_WithRoomForTheProcessToLeave() + { + /* The runner waits 10 seconds (shutdownForegroundThreadWaitSeconds) and the wait starts when the run returns. + The budget bounds the whole drain, a drop in flight included, and still leaves headroom for the process to + leave. */ + Assert.InRange(ScratchPostgres.ExitDrainBudgetSeconds, 1, 7); + } + + [Fact] + public void TheDrain_ConnectsWithTheShortTimeouts_AndKeepsTheRestOfTheAdminConnectionString() + { + var admin = new NpgsqlConnectionStringBuilder + { + Host = "db.example.test", + Port = 5544, + Username = "darling", + Database = "postgres", + Pooling = false, + }.ConnectionString; + + var drain = new NpgsqlConnectionStringBuilder(ScratchPostgres.ExitDrainConnectionString(admin)); + + /* The connect wait is the one that stalled the exit: 15 seconds by default, 3 for the drain. Both values are + named in one place, and the drain sets them whatever the base string carries. */ + Assert.Equal(ScratchPostgres.ExitDrainConnectTimeoutSeconds, drain.Timeout); + Assert.Equal(ScratchPostgres.ExitDrainCommandTimeoutSeconds, drain.CommandTimeout); + Assert.True(drain.Timeout < new NpgsqlConnectionStringBuilder().Timeout); + Assert.False(drain.Pooling); + Assert.Equal("db.example.test", drain.Host); + Assert.Equal(5544, drain.Port); + Assert.Equal("darling", drain.Username); + Assert.Equal("postgres", drain.Database); + } + + [Fact] + public void TheDrain_StopsOnlyTheClusterThatDidNotAnswer_NotAnotherOne() + { + /* Two different ports, so the two stopped clusters are two clusters even if the OS hands the same port out twice. */ + var firstPort = ClosedPort(); + int secondPort; + do + { + secondPort = ClosedPort(); + } + while (secondPort == firstPort); + + var firstStopped = StoppedClusterConnectionString(firstPort); + var secondStopped = StoppedClusterConnectionString(secondPort); + + var outcome = ScratchPostgres.DropRememberedAtExit(new[] + { + Remembered(firstStopped, "first one"), + Remembered(secondStopped, "second one"), + Remembered(firstStopped, "first two"), + Remembered(secondStopped, "second two"), + }); + + /* Two clusters, each tried once: a dead cluster does not cause the other to be skipped. */ + Assert.Equal(2, outcome.ConnectAttempts); + Assert.Equal(2, outcome.Failed); + Assert.Equal(2, outcome.Skipped); + } + + [Fact] + public async Task TheDrain_OnALiveCluster_DropsWhatItWasGiven_EvenWhenAStoppedClusterIsInTheSameList() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to a Postgres connection string to run the live exit-drain test."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var stopped = StoppedClusterConnectionString(ClosedPort()); + + /* A deadline as long as one drop's command timeout: this pins the dead-and-live mix, not the deadline, and a drop + mid-suite waits for a checkpoint that a busy cluster can take seconds to finish. */ + var outcome = ScratchPostgres.DropRememberedAtExit( + new[] + { + Remembered(stopped, "dead one"), + new ScratchPostgres.RememberedDatabase(scratch.DatabaseName, baseConnectionString!, "the live test"), + Remembered(stopped, "dead two"), + }, + TimeSpan.FromSeconds(ScratchPostgres.ExitDrainConnectTimeoutSeconds + ScratchPostgres.ExitDrainCommandTimeoutSeconds) + ScratchPostgres.QuiesceCap); + + /* One attempt on the stopped cluster, one on the live one; the second database on the stopped cluster is left. */ + Assert.Equal(2, outcome.ConnectAttempts); + Assert.Equal(1, outcome.Dropped); + Assert.Equal(1, outcome.Failed); + Assert.Equal(1, outcome.Skipped); + Assert.False(ScratchPostgres.IsRemembered(scratch.DatabaseName), "a database the drain dropped must leave the queue"); + + await using var probe = new NpgsqlConnection(baseConnectionString); + await probe.OpenAsync(ct); + await using var exists = new NpgsqlCommand("SELECT count(*) FROM pg_database WHERE datname = $1", probe); + exists.Parameters.AddWithValue(scratch.DatabaseName); + Assert.Equal(0L, Convert.ToInt64(await exists.ExecuteScalarAsync(ct))); + } + + /// + /// A run that leaks many databases on a live cluster: the drain returns by its deadline and accounts for every one, + /// dropped or not finished, with none failed. The 2026-10-08 nightly leaked 39, and dropping them one at a time + /// outran xUnit's end-of-run window. + /// How many it drops in time is not pinned. Each DROP DATABASE waits for an immediate checkpoint, and on a + /// cluster busy with other tests' writes one checkpoint took 30 seconds on a local rig (2026-10-08), so a drop can + /// legitimately miss the deadline mid-suite; at the real process exit the cluster is idle. + /// + [Fact] + public async Task TheDrain_OfManyLeakedDatabases_OnALiveCluster_ReturnsByItsDeadline_AndAccountsForEveryOne() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to a Postgres connection string to run the live exit-drain test."); + + var ct = TestContext.Current.CancellationToken; + const int Leaked = 6; + var scratches = new ScratchPostgres[Leaked]; + try + { + for (var i = 0; i < Leaked; i++) + { + scratches[i] = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + } + + var clock = Stopwatch.StartNew(); + var outcome = ScratchPostgres.DropRememberedAtExit( + Array.ConvertAll(scratches, scratch => new ScratchPostgres.RememberedDatabase(scratch.DatabaseName, baseConnectionString!, "the leak test"))); + clock.Stop(); + + Assert.True( + clock.Elapsed < TimeSpan.FromSeconds(ScratchPostgres.ExitDrainBudgetSeconds) + TimeSpan.FromSeconds(1), + $"the drain of {Leaked} databases took {clock.Elapsed.TotalSeconds:0.0} seconds against its {TimeSpan.FromSeconds(ScratchPostgres.ExitDrainBudgetSeconds).TotalSeconds:0}-second deadline"); + Assert.Equal(Leaked, outcome.Dropped + outcome.Unfinished); + Assert.Equal(0, outcome.Failed); + Assert.Equal(0, outcome.Skipped); + Assert.True(outcome.ConnectAttempts >= 1, "the drain never tried the live cluster"); + + /* A database the drain dropped is gone from the cluster and from the queue. A drop it sent before the + deadline may finish after it returned, so the queue can hold fewer than the outcome's unfinished count. */ + var dropped = Array.FindAll(scratches, scratch => !ScratchPostgres.IsRemembered(scratch.DatabaseName)); + Assert.True(dropped.Length >= outcome.Dropped, $"the drain reported {outcome.Dropped} dropped, and only {dropped.Length} left the queue"); + await using var probe = new NpgsqlConnection(baseConnectionString); + await probe.OpenAsync(ct); + await using var exists = new NpgsqlCommand("SELECT count(*) FROM pg_database WHERE datname = ANY($1)", probe); + exists.Parameters.AddWithValue(Array.ConvertAll(dropped, scratch => scratch.DatabaseName)); + Assert.Equal(0L, Convert.ToInt64(await exists.ExecuteScalarAsync(ct))); + } + finally + { + /* A drop the drain already made makes this a no-op; one it missed (a failing run) is dropped here, not at exit. */ + foreach (var scratch in scratches) + { + if (scratch is not null) + { + await scratch.DisposeAsync(); + } + } + } + } +} diff --git a/Darling/Darling.Tests/ScratchPostgresQuiesceTests.cs b/Darling/Darling.Tests/ScratchPostgresQuiesceTests.cs new file mode 100644 index 0000000000..56b514144c --- /dev/null +++ b/Darling/Darling.Tests/ScratchPostgresQuiesceTests.cs @@ -0,0 +1,310 @@ +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.IO; +using System.Linq; +using System.Text.RegularExpressions; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: this class mints its own scratch databases through ScratchPostgres and never touches another test's + rows, so it is deliberately NOT [Collection("live-postgres")]. */ + +/// +/// PR #5480: runs before a scratch database is dropped +/// WITH (FORCE), so no TimescaleDB policy worker is mid-run in the database being dropped (CI saw a worker die with +/// 0xC0000005 twice, both with such a drop in flight). It is best-effort: it never throws and never blocks the drop. +/// +[Collection("timing")] +public sealed class ScratchPostgresQuiesceLiveTests +{ + [Fact] + public async Task AfterTheHelper_EveryJobIsUnscheduled_AndNoJobWorkerIsLeftInTheDatabase() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live scratch-quiesce test (it mints its own scratch database)."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + Assert.SkipUnless(await LiveTimescaleProbe.TryEnableAsync(scratch.ConnectionString, ct), + "TimescaleDB is not available on this PostgreSQL, so there are no policy jobs to quiesce."); + + /* The product's policies (retention, compression, continuous-aggregate refresh), installed the way the + product installs them: they are scheduled on creation, which is what makes workers run when a test ends. */ + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + + var scheduledBefore = await ScalarAsync(connection, "SELECT count(*) FROM timescaledb_information.jobs WHERE scheduled", ct); + Assert.True(scheduledBefore > 0, "the product policies installed no scheduled job, so this test would prove nothing."); + + /* Start some workers right now, the state a test's dispose meets. */ + await using (var kick = new NpgsqlCommand( + "SELECT alter_job(job_id, next_start => now()) FROM timescaledb_information.jobs WHERE scheduled", connection)) + { + await kick.ExecuteNonQueryAsync(ct); + } + + await ScratchPostgres.QuiesceTimescaleJobsAsync(scratch.UnpinnedConnectionString, scratch.DatabaseName); + + Assert.Equal(0L, await ScalarAsync(connection, "SELECT count(*) FROM timescaledb_information.jobs WHERE scheduled", ct)); + + await using var admin = new NpgsqlConnection(new NpgsqlConnectionStringBuilder(scratch.UnpinnedConnectionString) { Database = "postgres" }.ConnectionString); + await admin.OpenAsync(ct); + Assert.Equal(0L, await ScratchPostgres.JobWorkerCountAsync(admin, scratch.DatabaseName, ct)); + } + + [Fact] + public async Task ADatabaseWithoutTheExtension_ReturnsAtOnce_AndDoesNotThrow() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live scratch-quiesce test (it mints its own scratch database)."); + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, TestContext.Current.CancellationToken); + + var clock = Stopwatch.StartNew(); + await ScratchPostgres.QuiesceTimescaleJobsAsync(scratch.UnpinnedConnectionString, scratch.DatabaseName); + + Assert.True(clock.Elapsed < ScratchPostgres.QuiesceCap, $"took {clock.Elapsed}, past the cap {ScratchPostgres.QuiesceCap}."); + } + + [Fact] + public async Task ADatabaseThatIsAlreadyGone_ReturnsWithinTheCap_AndDoesNotThrow() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live scratch-quiesce test (it mints its own scratch database)."); + + var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, TestContext.Current.CancellationToken); + var unpinned = scratch.UnpinnedConnectionString; + var name = scratch.DatabaseName; + await scratch.DisposeAsync(); + + var clock = Stopwatch.StartNew(); + await ScratchPostgres.QuiesceTimescaleJobsAsync(unpinned, name); + + Assert.True(clock.Elapsed < ScratchPostgres.QuiesceCap, $"took {clock.Elapsed}, past the cap {ScratchPostgres.QuiesceCap}."); + } + + /// + /// #5549: the backend that ran a scratch database's CREATE DATABASE and DROP DATABASE must end with its + /// connection. The admin string is the caller's DARLING_TEST_PG, so a pooled admin connection went back + /// into the pool every live test draws from, and on CI the next test's CALL run_job ran on the backend + /// that had just run the drop. The base string here carries its own application name, so its pool and its + /// sessions belong to this test alone: once the scratch database is gone, none of them may still be open. + /// + [Fact] + public async Task TheCreateAndTheDrop_LeaveNoBackendOpenInTheCallersPool() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live scratch-pool test (it mints its own scratch database)."); + + var ct = TestContext.Current.CancellationToken; + var tag = "w5549-" + Guid.NewGuid().ToString("N")[..12]; + var tagged = new NpgsqlConnectionStringBuilder(baseConnectionString) { ApplicationName = tag }.ConnectionString; + + var scratch = await ScratchPostgres.CreateAsync(tagged, ct); + await scratch.DisposeAsync(); + + await using var probe = new NpgsqlConnection(baseConnectionString); + await probe.OpenAsync(ct); + var left = -1L; + var lastQuery = ""; + /* An unpooled connection's backend leaves pg_stat_activity a moment after the close, not at it. A pooled one + stays for the pool's idle lifetime (300 seconds by default), far past this wait. */ + for (var poll = 0; poll < 25; poll++) + { + await using var command = new NpgsqlCommand( + "SELECT count(*), coalesce(max(query), '') FROM pg_stat_activity WHERE application_name = $1", probe); + command.Parameters.AddWithValue(tag); + await using var reader = await command.ExecuteReaderAsync(ct); + await reader.ReadAsync(ct); + left = reader.GetInt64(0); + lastQuery = reader.GetString(1); + if (left == 0) + { + break; + } + + await reader.DisposeAsync(); + await Task.Delay(TimeSpan.FromMilliseconds(200), ct); + } + + Assert.True(left == 0, + $"{left} backend(s) from the caller's pool are still open after the scratch database's create and drop; the last one ran: {lastQuery}"); + } + + private static async Task ScalarAsync(NpgsqlConnection connection, string sql, System.Threading.CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + return (long)(await command.ExecuteScalarAsync(ct))!; + } +} + +/// +/// PR #5480: every DROP DATABASE ... WITH (FORCE) statement in Darling.Tests is preceded, in the same method, by +/// QuiesceTimescaleJobsAsync, so a new drop site cannot reintroduce the kill of a running TimescaleDB job worker. +/// A statement is a string literal holding both DROP DATABASE and WITH (FORCE); comments are blanked first by . +/// +[Trait("Stage", "Guard")] +public sealed class ScratchPostgresQuiesceCensusTests +{ + private const string Helper = "QuiesceTimescaleJobsAsync"; + + /// How many lines above a drop the helper call may sit. The farthest real site has it 3 lines above. + private const int Window = 12; + + private static readonly Regex DropStatement = new("\".*DROP DATABASE.*WITH \\(FORCE\\)", RegexOptions.Compiled); + + /// + /// Blanks comments (to spaces, keeping every newline so line numbers hold) and keeps string literals, quotes + /// included, because the drop statement is itself a literal. Built on , so a + /// line that merely starts with * inside a literal is still seen, and a comment is never. + /// + internal static string BlankComments(string text) + { + var keep = CSharpSourceWalker.CodeMask(text); + foreach (var (start, body) in CSharpSourceWalker.StringLiteralBodies(text)) + { + for (var i = start; i < start + body.Length; i++) + { + keep[i] = true; + } + + if (start > 0 && text[start - 1] == '"') + { + keep[start - 1] = true; + } + + if (start + body.Length < text.Length && text[start + body.Length] == '"') + { + keep[start + body.Length] = true; + } + } + + var sb = new System.Text.StringBuilder(text.Length); + for (var i = 0; i < text.Length; i++) + { + sb.Append(keep[i] ? text[i] : text[i] == '\n' ? '\n' : ' '); + } + + return sb.ToString(); + } + + private static (int Sites, List Unguarded) Scan(IEnumerable<(string Name, string Source)> files) + { + var sites = 0; + var unguarded = new List(); + foreach (var (name, source) in files) + { + var lines = BlankComments(source).Split('\n'); + for (var i = 0; i < lines.Length; i++) + { + if (!DropStatement.IsMatch(lines[i])) + { + continue; + } + + sites++; + var from = Math.Max(0, i - Window); + if (!lines.Skip(from).Take(i - from).Any(line => line.Contains(Helper, StringComparison.Ordinal))) + { + unguarded.Add($"{name}:{i + 1}"); + } + } + } + + return (sites, unguarded); + } + + [Fact] + public void EveryForceDropInTheTestProject_IsPrecededByTheQuiesceHelper() + { + var directory = FindTestProjectDirectory(); + Assert.True(directory is not null, "could not find Darling/Darling.Tests from the test binary."); + + var files = Directory.EnumerateFiles(directory!, "*.cs", SearchOption.AllDirectories) + .Select(path => (Name: Path.GetRelativePath(directory!, path).Replace('\\', '/'), Path: path)) + .Where(found => !found.Name.Split('/').Any(segment => segment is "bin" or "obj")) + .Where(found => found.Name != "ScratchPostgresQuiesceTests.cs") + .Select(found => (found.Name, File.ReadAllText(found.Path))) + .ToList(); + + var (sites, unguarded) = Scan(files); + + Assert.True(sites >= 7, $"the scan found {sites} WITH (FORCE) drop site(s); the project has at least 7, so the match found nothing."); + Assert.True(unguarded.Count == 0, + "These WITH (FORCE) drops are not preceded by ScratchPostgres.QuiesceTimescaleJobsAsync (PR #5480): " + + string.Join(", ", unguarded)); + } + + [Fact] + public void ADropWithoutTheHelper_IsReported_AndOneWithItIsNot() + { + const string guarded = + "await ScratchPostgres.QuiesceTimescaleJobsAsync(cs, name);\n" + + "await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {name} WITH (FORCE)\", admin);\n"; + const string bare = "await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {name} WITH (FORCE)\", admin);\n"; + const string commentOnly = "/// the drop is DROP DATABASE ... WITH (FORCE)\n// \"DROP DATABASE x WITH (FORCE)\"\n"; + + var clean = Scan([("G.cs", guarded)]); + Assert.Equal(1, clean.Sites); + Assert.Empty(clean.Unguarded); + var offending = Scan([("B.cs", bare)]); + Assert.Equal(1, offending.Sites); + Assert.Equal(["B.cs:1"], offending.Unguarded); + Assert.Equal(0, Scan([("C.cs", commentOnly)]).Sites); + } + + /// + /// The CI side of PR #5480: both workflows warn when the PostgreSQL log holds "terminated by exception", on every + /// outcome (if: always()) and without ever failing the job, ahead of the step that stops the server. + /// + [Theory] + [InlineData("build.yml")] + [InlineData("nightly.yml")] + public void TheWorkflow_WarnsOnAWorkerCrashInThePostgresLog_AlwaysAndOnlyAsAWarning(string workflow) + { + var path = Path.Combine(AppContext.BaseDirectory, "Fixtures", workflow); + Assert.True(File.Exists(path), $"{workflow} was not copied beside the test binary (Darling.Tests.csproj links it into Fixtures\\)."); + var text = File.ReadAllText(path); + + var start = text.IndexOf("- name: Warn if a PostgreSQL worker crashed", StringComparison.Ordinal); + Assert.True(start >= 0, $"{workflow} has no 'Warn if a PostgreSQL worker crashed' step."); + var stop = text.IndexOf("- name: Stop PostgreSQL", start, StringComparison.Ordinal); + Assert.True(stop > start, $"{workflow}: the warning step must come before 'Stop PostgreSQL'."); + + var step = text[start..stop]; + Assert.Contains("if: always()", step, StringComparison.Ordinal); + Assert.Contains("darling-pg.log", step, StringComparison.Ordinal); + Assert.Contains("terminated by exception", step, StringComparison.Ordinal); + Assert.Contains("::warning", step, StringComparison.Ordinal); + Assert.DoesNotContain("::error", step, StringComparison.Ordinal); + Assert.Contains("exit 0", step, StringComparison.Ordinal); + } + + private static string? FindTestProjectDirectory() + { + var directory = new DirectoryInfo(AppContext.BaseDirectory); + for (var i = 0; i < 10 && directory is not null; i++) + { + if (File.Exists(Path.Combine(directory.FullName, "PerformanceMonitor.sln"))) + { + var source = Path.Combine(directory.FullName, "Darling", "Darling.Tests"); + return Directory.Exists(source) ? source : null; + } + + directory = directory.Parent; + } + + return null; + } +} diff --git a/Darling/Darling.Tests/SealedSecretResolverTests.cs b/Darling/Darling.Tests/SealedSecretResolverTests.cs new file mode 100644 index 0000000000..aab46bf2e5 --- /dev/null +++ b/Darling/Darling.Tests/SealedSecretResolverTests.cs @@ -0,0 +1,393 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// How the service opens a saved password (#5366): a reference first, then a sealed value through the key ring, then an +/// old-format (DPAPI) value on Windows only when darling.json declares it or a pin taken at upgrade still matches the row. +/// Every refusal throws before the password is used, with one fixed sentence that never carries the value. +/// +/// The old-format opener is a counting stand-in, so a test can say that it was never called. +/// +public sealed class SealedSecretResolverTests +{ + private const string Plain = "p@ss-not-real"; + + private static readonly Lazy s_key = new(PasswordPrivateKey.Generate); + private static readonly Lazy s_otherKey = new(PasswordPrivateKey.Generate); + + private sealed class Probe + { + public int Calls; + public string Result = Plain; + + public LegacyDpapi AsWindows() => new(true, _ => { Calls++; return Result; }); + + public LegacyDpapi AsLinux() => new(false, _ => { Calls++; return Result; }); + } + + private static IPasswordKeyRing Ring => DarlingPasswordKey.FromPrivateKey(s_key.Value); + + private static MonitoredServer Server(string host = "alpha-host.example.test", string? stored = null) => new() + { + Name = "alpha", + Host = host, + Auth = "sql", + Username = "monitor", + EncryptedPassword = stored, + RemediationUsername = "remediator", + }; + + private static string SealFor(MonitoredServer server, PasswordPrivateKey? key = null) => + PasswordSeal.Seal(Plain, (key ?? s_key.Value).PublicKey, server.SecretBinding); + + private static LegacyPin PinFor(string stored, PasswordBinding binding) => + new(SHA256.HashData(Encoding.UTF8.GetBytes(stored)), binding.LegacyPinHash()); + + /* ═══════════ sealed values ═══════════ */ + + [Fact] + public void ASealedPasswordOpensForTheConnectionItWasSavedFor() + { + var server = Server(); + server.EncryptedPassword = SealFor(server); + var probe = new Probe(); + + Assert.Equal(Plain, DarlingSecrets.ResolvePassword(server, out var usedPlaintext, Ring, probe.AsWindows())); + Assert.False(usedPlaintext); + Assert.Equal(0, probe.Calls); + } + + [Fact] + public void ASealedPasswordCopiedToARowWithAnotherHostIsRefusedWithTheBindingMessage() + { + var saved = Server(); + var sealedText = SealFor(saved); + var copy = Server(host: "beta-host.example.test", stored: sealedText); + var probe = new Probe(); + + var ex = Assert.Throws(() => DarlingSecrets.ResolvePassword(copy, out _, Ring, probe.AsWindows())); + + Assert.Equal( + "The saved password for server 'alpha' was saved for a different connection, or was changed. Enter the password again.", + ex.Message); + Assert.Equal(0, probe.Calls); + Assert.DoesNotContain(Plain, ex.Message, StringComparison.Ordinal); + } + + [Fact] + public void ASealedPasswordWithAnUnknownKeyIdGivesTheUnknownKeySentence() + { + var server = Server(); + server.EncryptedPassword = SealFor(server, s_otherKey.Value); + var probe = new Probe(); + + var ex = Assert.Throws(() => DarlingSecrets.ResolvePassword(server, out _, Ring, probe.AsWindows())); + + var expectedId = PasswordSeal.DisplayKeyId(s_otherKey.Value.PublicKey.KeyId); + Assert.Equal( + $"The saved password for server 'alpha' was sealed to password key {expectedId}, which this service does not have. " + + "Restore the credentials volume, or enter the password again.", + ex.Message); + Assert.Equal(0, probe.Calls); + } + + [Fact] + public void ANewerSealedVersionNeverReachesTheOldFormatOpener() + { + var server = Server(stored: "sealed:v2:" + Convert.ToBase64String(new byte[64])); + server.EncryptedPasswordDeclaredByFile = true; + var probe = new Probe(); + + var ex = Assert.Throws(() => DarlingSecrets.ResolvePassword(server, out _, Ring, probe.AsWindows())); + + Assert.Equal("The saved password for server 'alpha' was saved by a newer version of Darling.", ex.Message); + Assert.Equal(0, probe.Calls); + } + + [Fact] + public void ANotReadyRingGivesTheNotReadySentenceNotTheUnknownKeySentence() + { + var server = Server(); + server.EncryptedPassword = SealFor(server); + var probe = new Probe(); + + var ex = Assert.Throws(() => + DarlingSecrets.ResolvePassword(server, out _, DarlingPasswordKey.Refusing(DarlingPasswordKey.NotReadyReason), probe.AsWindows())); + + Assert.Equal(DarlingPasswordKey.NotReadyReason, ex.Message); + Assert.Equal(0, probe.Calls); + } + + [Fact] + public void ARefusingRingGivesItsOwnReason() + { + var server = Server(); + server.EncryptedPassword = SealFor(server); + const string reason = "The password key is missing from the credentials directory."; + + var ex = Assert.Throws(() => + DarlingSecrets.ResolvePassword(server, out _, DarlingPasswordKey.Refusing(reason), new Probe().AsWindows())); + + Assert.Equal(reason, ex.Message); + } + + [Fact] + public void AConnectionFieldThatIsNotValidTextGivesTheBindingMessage() + { + var saved = Server(); + var server = Server(host: "alpha\ud800.example.test", stored: SealFor(saved)); + + var ex = Assert.Throws(() => DarlingSecrets.ResolvePassword(server, out _, Ring, new Probe().AsWindows())); + + Assert.Contains("saved for a different connection, or was changed", ex.Message, StringComparison.Ordinal); + Assert.DoesNotContain("\ud800", ex.Message, StringComparison.Ordinal); + } + + [Fact] + public void ASealedPasswordIsOpenedWhateverThePlatform() + { + var server = Server(); + server.EncryptedPassword = SealFor(server); + + Assert.Equal(Plain, DarlingSecrets.ResolvePassword(server, out _, Ring, new Probe().AsLinux())); + } + + [Fact] + public void ARowReadWithANullDatabaseBindsToTheEmptyDatabase() + { + /* A row's NULL database and an empty one are the same text in the binding, and the identity read from the row is what the + sealed value was bound to. */ + var identity = ServerConnectionIdentity.FromStoredColumns("alpha-host.example.test", null, null, null, false, "sql", "monitor", "Mandatory", false, false); + var sealedText = PasswordSeal.Seal(Plain, s_key.Value.PublicKey, PasswordBinding.ForServer(identity)); + var server = Server(stored: sealedText); + server.StoredIdentity = identity; + + Assert.Equal(Plain, DarlingSecrets.ResolvePassword(server, out _, Ring, new Probe().AsWindows())); + } + + [Fact] + public void ASealedRemediationPasswordOpensOnlyForItsOwnConnectionAndLogin() + { + var server = Server(); + server.RemediationEncryptedPassword = PasswordSeal.Seal(Plain, s_key.Value.PublicKey, server.RemediationBinding); + Assert.Equal(Plain, DarlingSecrets.ResolveRemediationPassword(server, Ring, new Probe().AsWindows())); + + server.RemediationUsername = "another-login"; + var ex = Assert.Throws(() => DarlingSecrets.ResolveRemediationPassword(server, Ring, new Probe().AsWindows())); + Assert.Equal( + "The saved remediation password for server 'alpha' was saved for a different connection, or was changed. Enter the password again.", + ex.Message); + } + + [Fact] + public void ASealedSmtpPasswordOpensOnlyForItsOwnSmtpConnection() + { + var smtp = new SmtpConfig { Host = "mail.example.test", Port = 587, UseSsl = true, Username = "mailer" }; + smtp.EncryptedPassword = PasswordSeal.Seal(Plain, s_key.Value.PublicKey, smtp.SecretBinding); + Assert.Equal(Plain, DarlingSecrets.ResolveSmtpPassword(smtp, Ring, new Probe().AsLinux())); + + smtp.Host = "other-mail.example.test"; + var ex = Assert.Throws(() => DarlingSecrets.ResolveSmtpPassword(smtp, Ring, new Probe().AsLinux())); + Assert.Equal( + "The saved SMTP password was saved for a different connection, or was changed. Enter the password again.", ex.Message); + } + + /* ═══════════ old-format (DPAPI) values ═══════════ */ + + [Fact] + public void AnUnpinnedOldFormatPasswordIsRefusedAndNeverOpened() + { + var server = Server(stored: "AQIDBAUGBwgJCg=="); + var probe = new Probe(); + + var ex = Assert.Throws(() => DarlingSecrets.ResolvePassword(server, out _, Ring, probe.AsWindows())); + + Assert.Equal( + "The saved password for server 'alpha' is in the old format and was saved after this service was upgraded, or its connection changed. " + + "Update the Darling Viewer, then enter the password again.", + ex.Message); + Assert.Equal(0, probe.Calls); + } + + [Fact] + public void APinnedOldFormatPasswordOpensWhileItsRowIsUnchanged() + { + const string stored = "AQIDBAUGBwgJCg=="; + var server = Server(stored: stored); + server.SecretPin = PinFor(stored, server.SecretBinding); + var probe = new Probe(); + + Assert.Equal(Plain, DarlingSecrets.ResolvePassword(server, out var usedPlaintext, Ring, probe.AsWindows())); + Assert.False(usedPlaintext); + Assert.Equal(1, probe.Calls); + } + + [Fact] + public void APinnedRowWhoseHostWasChangedByADirectUpdateIsRefused() + { + const string stored = "AQIDBAUGBwgJCg=="; + var server = Server(stored: stored); + server.SecretPin = PinFor(stored, server.SecretBinding); + server.Host = "beta-host.example.test"; + var probe = new Probe(); + + var ex = Assert.Throws(() => DarlingSecrets.ResolvePassword(server, out _, Ring, probe.AsWindows())); + + Assert.Contains("is in the old format", ex.Message, StringComparison.Ordinal); + Assert.Equal(0, probe.Calls); + } + + [Fact] + public void APinForAnotherValueDoesNotOpenAChangedPassword() + { + var server = Server(stored: "AQIDBAUGBwgJCg=="); + server.SecretPin = PinFor("AAAAAAAAAAAAAA==", server.SecretBinding); + var probe = new Probe(); + + Assert.Throws(() => DarlingSecrets.ResolvePassword(server, out _, Ring, probe.AsWindows())); + Assert.Equal(0, probe.Calls); + } + + [Fact] + public void AnOldFormatPasswordTheFileDeclaresOpensWithoutAPin() + { + var server = Server(stored: "AQIDBAUGBwgJCg=="); + server.EncryptedPasswordDeclaredByFile = true; + var probe = new Probe(); + + Assert.Equal(Plain, DarlingSecrets.ResolvePassword(server, out _, Ring, probe.AsWindows())); + Assert.Equal(1, probe.Calls); + } + + [Fact] + public void AnOldFormatPasswordOnAnotherPlatformGivesTheOtherMachineSentenceEvenWhenDeclaredOrPinned() + { + const string stored = "AQIDBAUGBwgJCg=="; + var server = Server(stored: stored); + server.EncryptedPasswordDeclaredByFile = true; + server.SecretPin = PinFor(stored, server.SecretBinding); + var probe = new Probe(); + + var ex = Assert.Throws(() => DarlingSecrets.ResolvePassword(server, out _, Ring, probe.AsLinux())); + + Assert.Equal( + "The saved password for server 'alpha' was saved with Windows DPAPI on another machine and cannot be read here. Enter it again.", + ex.Message); + Assert.Equal(0, probe.Calls); + } + + [Fact] + public void ARemediationOldFormatPasswordNeedsItsOwnPin() + { + const string stored = "AQIDBAUGBwgJCg=="; + var server = Server(); + server.RemediationEncryptedPassword = stored; + var probe = new Probe(); + + Assert.Throws(() => DarlingSecrets.ResolveRemediationPassword(server, Ring, probe.AsWindows())); + Assert.Equal(0, probe.Calls); + + server.RemediationPin = PinFor(stored, server.RemediationBinding); + Assert.Equal(Plain, DarlingSecrets.ResolveRemediationPassword(server, Ring, probe.AsWindows())); + Assert.Equal(1, probe.Calls); + } + + [Fact] + public void AnSmtpOldFormatPasswordNeedsAPinOrADeclaration() + { + const string stored = "AQIDBAUGBwgJCg=="; + var smtp = new SmtpConfig { Host = "mail.example.test", Username = "mailer", EncryptedPassword = stored }; + var probe = new Probe(); + + Assert.Throws(() => DarlingSecrets.ResolveSmtpPassword(smtp, Ring, probe.AsWindows())); + Assert.Equal(0, probe.Calls); + + smtp.SecretPin = PinFor(stored, smtp.SecretBinding); + Assert.Equal(Plain, DarlingSecrets.ResolveSmtpPassword(smtp, Ring, probe.AsWindows())); + } + + /* ═══════════ what darling.json declares ═══════════ */ + + [Fact] + public void TheFileDeclaresAnOldFormatValueAndAReferenceButNotASealedValue() + { + const string json = """ + { + "servers": [ + { "name": "alpha", "host": "alpha-host.example.test", "auth": "sql", "username": "monitor", "encryptedPassword": "AQIDBAUGBwgJCg==" }, + { "name": "beta", "host": "beta-host.example.test", "auth": "sql", "username": "monitor", "encryptedPassword": "sealed:v1:0123456789abcdef:AAAA" }, + { "name": "gamma", "host": "gamma-host.example.test", "auth": "sql", "username": "monitor", "encryptedPassword": "env:SOME_NOT_REAL_VARIABLE" } + ], + "smtp": { "host": "mail.example.test", "encryptedPassword": "AQIDBAUGBwgJCg==" } + } + """; + var config = DarlingConfig.Parse(json); + + Assert.True(config.Servers.Single(s => s.Name == "alpha").EncryptedPasswordDeclaredByFile); + Assert.False(config.Servers.Single(s => s.Name == "beta").EncryptedPasswordDeclaredByFile); + Assert.True(config.Servers.Single(s => s.Name == "gamma").EncryptedPasswordDeclaredByFile); + Assert.True(config.Smtp.EncryptedPasswordDeclaredByFile); + } + + [Fact] + public void AStoreRowIsMarkedOnlyWhenTheFileDeclaresTheSameValueForTheSameConnection() + { + const string stored = "AQIDBAUGBwgJCg=="; + var config = DarlingConfig.Parse( + """{ "servers": [ { "name": "alpha", "host": "alpha-host.example.test", "auth": "sql", "username": "monitor", "encryptedPassword": "AQIDBAUGBwgJCg==" } ] }"""); + + var sameRow = Server(stored: stored); + StoreConfigProvider.MarkSlotsTheFileDeclares(sameRow, config); + Assert.True(sameRow.EncryptedPasswordDeclaredByFile); + + var movedRow = Server(host: "beta-host.example.test", stored: stored); + StoreConfigProvider.MarkSlotsTheFileDeclares(movedRow, config); + Assert.False(movedRow.EncryptedPasswordDeclaredByFile); + + var changedValueRow = Server(stored: "AAAAAAAAAAAAAA=="); + StoreConfigProvider.MarkSlotsTheFileDeclares(changedValueRow, config); + Assert.False(changedValueRow.EncryptedPasswordDeclaredByFile); + } + + /* ═══════════ the load-time count ═══════════ */ + + [Fact] + public void TheCountNamesEachOldFormatPasswordThatCannotBeOpened() + { + const string stored = "AQIDBAUGBwgJCg=="; + var pinned = Server(stored: stored); + pinned.SecretPin = PinFor(stored, pinned.SecretBinding); + + var unpinned = Server(stored: stored); + unpinned.RemediationEncryptedPassword = stored; + + var declared = Server(stored: stored); + declared.EncryptedPasswordDeclaredByFile = true; + + var sealedServer = Server(); + sealedServer.EncryptedPassword = SealFor(sealedServer); + + var smtp = new SmtpConfig { Host = "mail.example.test", EncryptedPassword = stored }; + var servers = new[] { pinned, unpinned, declared, sealedServer }; + + Assert.Equal(3, StoreConfigProvider.CountPasswordsToEnterAgain(servers, smtp, new Probe().AsWindows())); + Assert.Equal(0, StoreConfigProvider.CountPasswordsToEnterAgain(new[] { sealedServer }, null, new Probe().AsWindows())); + /* Off Windows no old-format value can be opened, pinned or declared. */ + Assert.Equal(5, StoreConfigProvider.CountPasswordsToEnterAgain(servers, smtp, new Probe().AsLinux())); + } +} diff --git a/Darling/Darling.Tests/SecondaryReplicaScopeLiveTests.cs b/Darling/Darling.Tests/SecondaryReplicaScopeLiveTests.cs new file mode 100644 index 0000000000..6c5a060d2c --- /dev/null +++ b/Darling/Darling.Tests/SecondaryReplicaScopeLiveTests.cs @@ -0,0 +1,533 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Analysis; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Analysis; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: this fact mints its own scratch database through ScratchPostgres for every test and never touches + another test's rows, so it is deliberately NOT [Collection("live-postgres")]. */ + +/// +/// #5558 on a seeded PostgreSQL store, the twin of Lite's SecondaryReplicaFactTests: the replicated facts drop the +/// database this node holds as a secondary copy and the node-local ones keep it. Three databases: SecDb (the local +/// copy of group AG1 is SECONDARY), PrimDb (the local copy of group AG2 is PRIMARY) and StandDb (in no group). +/// Skipped when DARLING_TEST_PG is not set. +/// +public sealed class SecondaryReplicaScopeLiveTests +{ + private const int ServerId = -558_201; + private const string ServerName = "darling-secondary-replica-scope"; + private static readonly string[] Databases = ["SecDb", "PrimDb", "StandDb"]; + + private static string? BaseConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static DateTime Now() => DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow); + + /// Runs against a fresh migrated scratch store with the test server registered, then + /// drops the store whether the body passed or not. + private static async Task WithStoreAsync(Func body) + { + var baseConnectionString = BaseConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to run the live secondary replica scope tests."); + + var ct = TestContext.Current.CancellationToken; + var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var bodySucceeded = false; + try + { + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + await body(connection, postgres, ct); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => await Task.CompletedTask); + await scratch.DisposeAsync(); + } + } + + private static Task ExecAsync(NpgsqlConnection connection, CancellationToken ct, string sql, params object?[] values) => + DarlingMcpTestData.ExecAsync(connection, ct, sql, values); + + /// AG1: this node (NODE1) is for it and holds SecDb; AG2: this node is PRIMARY and + /// holds PrimDb. NODE2 is the other replica of AG1 and holds SecDb too (not local). + private static async Task SeedAgAsync(NpgsqlConnection c, CancellationToken ct, DateTime at, string localRole = "SECONDARY") + { + const string Replica = "INSERT INTO ag_replica_states (collection_id, collection_time, server_id, server_name, ag_name, replica_server_name, role_desc, is_local) VALUES ($1,$2,$3,$4,$5,$6,$7,$8)"; + var when = DarlingMcpTestData.Naive(at); + await ExecAsync(c, ct, Replica, CollectionIdGenerator.Next(), when, ServerId, ServerName, "AG1", "NODE1", localRole, true); + await ExecAsync(c, ct, Replica, CollectionIdGenerator.Next(), when, ServerId, ServerName, "AG1", "NODE2", "PRIMARY", false); + await ExecAsync(c, ct, Replica, CollectionIdGenerator.Next(), when, ServerId, ServerName, "AG2", "NODE1", "PRIMARY", true); + const string Db = "INSERT INTO ag_database_replica_states (collection_id, collection_time, server_id, server_name, ag_name, database_name, replica_server_name, is_local) VALUES ($1,$2,$3,$4,$5,$6,$7,$8)"; + await ExecAsync(c, ct, Db, CollectionIdGenerator.Next(), when, ServerId, ServerName, "AG1", "SecDb", "NODE1", true); + await ExecAsync(c, ct, Db, CollectionIdGenerator.Next(), when, ServerId, ServerName, "AG1", "SecDb", "NODE2", false); + await ExecAsync(c, ct, Db, CollectionIdGenerator.Next(), when, ServerId, ServerName, "AG2", "PrimDb", "NODE1", true); + } + + /// One sys.databases capture for each of the three databases: auto_shrink on and RCSI off, so every one is a + /// DB_CONFIG offender, plus one 20 GB percent-growth data file each for FILE_AUTOGROWTH_PERCENT. + private static async Task SeedDatabasesAsync(NpgsqlConnection c, CancellationToken ct, DateTime end) + { + foreach (var db in Databases) + { + await ExecAsync(c, ct, @" +INSERT INTO database_config + (config_id, capture_time, server_id, server_name, database_name, + state_desc, compatibility_level, collation_name, recovery_model, is_read_only, + is_auto_close_on, is_auto_shrink_on, is_auto_create_stats_on, is_auto_update_stats_on, + is_auto_update_stats_async_on, is_read_committed_snapshot_on, snapshot_isolation_state, + is_parameterization_forced, is_query_store_on, is_encrypted, is_trustworthy_on, is_db_chaining_on, + is_broker_enabled, is_cdc_enabled, is_mixed_page_allocation_on, log_reuse_wait_desc, page_verify_option, + target_recovery_time_seconds, delayed_durability, is_accelerated_database_recovery_on, + is_memory_optimized_enabled, is_optimized_locking_on) +VALUES ($1, $2, $3, $4, $5, + 'ONLINE', 160, 'SQL_Latin1_General_CP1_CI_AS', 'FULL', FALSE, + FALSE, TRUE, TRUE, TRUE, + FALSE, FALSE, 'OFF', + FALSE, TRUE, FALSE, FALSE, FALSE, + FALSE, FALSE, FALSE, 'NOTHING', 'CHECKSUM', + 60, 'DISABLED', FALSE, + FALSE, FALSE)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(end.AddHours(-1)), ServerId, ServerName, db); + + await ExecAsync(c, ct, @" +INSERT INTO database_size_stats + (collection_id, collection_time, server_id, server_name, database_name, database_id, file_id, file_type_desc, file_name, + physical_name, total_size_mb, used_size_mb, is_percent_growth, growth_pct) +VALUES ($1, $2, $3, $4, $5, 7, 1, 'ROWS', $6, 'D:\data.mdf', 20480, 10000, TRUE, 10)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(end.AddMinutes(-30)), ServerId, ServerName, db, db + "_data"); + } + } + + private static AnalysisContext Context(DateTime end) => new() + { + ServerId = ServerId, + ServerName = ServerName, + TimeRangeStart = DarlingMcpTestData.Naive(end.AddHours(-4)), + TimeRangeEnd = DarlingMcpTestData.Naive(end), + ServerUtcOffset = TimeSpan.Zero, + }; + + private static async Task<(List Facts, AnalysisContext Context)> FactsAsync(NpgsqlDataSource postgres, DateTime end, CancellationToken ct) + { + var context = Context(end); + context.CancellationToken = ct; + await PgSecondaryReplicaScope.EnsureAsync(postgres, context, logger: null); + var facts = (await new PgFactCollector(postgres).CollectFactsAsync(context)).ToList(); + return (facts, context); + } + + private static Fact One(List facts, string key) => Assert.Single(facts, f => f.Key == key); + + private static async Task DrillDownJsonAsync(NpgsqlDataSource postgres, AnalysisContext context, string rootFactKey, double severity = 0.3) + { + var finding = new AnalysisFinding + { + ServerId = ServerId, RootFactKey = rootFactKey, StoryPath = rootFactKey, PathKeys = [rootFactKey], Severity = severity, + }; + await new PgDrillDownCollector(postgres).EnrichFindingsAsync([finding], context); + return JsonSerializer.Serialize(finding.DrillDown); + } + + [Fact] + public async Task OnASecondary_DbConfigAndAutogrowthDropTheSecondaryDatabase_TheDrillDownsAgree_AndNodeLocalFactsKeepIt() + { + await WithStoreAsync(async (c, postgres, ct) => + { + var end = Now(); + await SeedDatabasesAsync(c, ct, end); + await SeedAgAsync(c, ct, end.AddMinutes(-1)); + + var (facts, context) = await FactsAsync(postgres, end, ct); + + Assert.Equal(["SecDb"], context.SecondaryReplicaDatabases!.ToArray()); + + var config = One(facts, "DB_CONFIG"); + Assert.Equal(2.0, config.Value); + Assert.Equal(2.0, config.Metadata["database_count"]); + Assert.Equal(2.0, config.Metadata["auto_shrink_on_count"]); + Assert.Equal(2.0, config.Metadata["rcsi_off_count"]); + + var autogrowth = One(facts, "FILE_AUTOGROWTH_PERCENT"); + Assert.Equal(2.0, autogrowth.Value); + Assert.Equal(2.0, autogrowth.Metadata["database_count"]); + + /* The drill-downs list the same databases the facts counted. */ + var configJson = await DrillDownJsonAsync(postgres, context, "DB_CONFIG"); + Assert.Contains("PrimDb", configJson); + Assert.Contains("StandDb", configJson); + Assert.DoesNotContain("SecDb", configJson); + + var growthJson = await DrillDownJsonAsync(postgres, context, "FILE_AUTOGROWTH_PERCENT"); + Assert.Contains("PrimDb", growthJson); + Assert.Contains("StandDb", growthJson); + Assert.DoesNotContain("SecDb", growthJson); + }); + } + + [Fact] + public async Task OnThePrimary_OrWithNoOrStaleAgRows_NothingIsSkipped() + { + await WithStoreAsync(async (c, postgres, ct) => + { + var end = Now(); + await SeedDatabasesAsync(c, ct, end); + + /* No AG rows at all. */ + var (noRows, noRowsContext) = await FactsAsync(postgres, end, ct); + Assert.Empty(noRowsContext.SecondaryReplicaDatabases!); + Assert.Equal(3.0, One(noRows, "DB_CONFIG").Metadata["database_count"]); + Assert.Equal(3.0, One(noRows, "FILE_AUTOGROWTH_PERCENT").Metadata["database_count"]); + + /* Rows exist, but the newest is three hours old: stale, so the role vouches for nothing. */ + await SeedAgAsync(c, ct, end.AddHours(-3)); + var (stale, staleContext) = await FactsAsync(postgres, end, ct); + Assert.Empty(staleContext.SecondaryReplicaDatabases!); + Assert.Equal(3.0, One(stale, "DB_CONFIG").Metadata["database_count"]); + + /* A fresh snapshot where the local role for the group is PRIMARY. */ + await SeedAgAsync(c, ct, end.AddMinutes(-1), localRole: "PRIMARY"); + var (primary, primaryContext) = await FactsAsync(postgres, end, ct); + Assert.Empty(primaryContext.SecondaryReplicaDatabases!); + Assert.Equal(3.0, One(primary, "DB_CONFIG").Metadata["database_count"]); + }); + } + + [Fact] + public async Task AnAsOfRead_UsesTheRoleAtItsOwnEnd_NotTheCurrentOne_AndTheNoteFollows() + { + await WithStoreAsync(async (c, postgres, ct) => + { + var now = Now(); + /* Primary three hours ago, secondary a minute ago (a failover in between). */ + await SeedAgAsync(c, ct, now.AddHours(-3).AddMinutes(-1), localRole: "PRIMARY"); + await SeedAgAsync(c, ct, now.AddMinutes(-1)); + + var current = await PgSecondaryReplicaScope.ReadAsync(postgres, ServerId, now, logger: null, ct); + Assert.Equal(["SecDb"], current.ToArray()); + Assert.Equal(AgReplicaScope.SkippedNote(1), await PgSecondaryReplicaScope.NoteAsync(postgres, ServerId, logger: null, ct)); + + var earlier = await PgSecondaryReplicaScope.ReadAsync(postgres, ServerId, now.AddHours(-3), logger: null, ct); + Assert.Empty(earlier); + Assert.Null(await PgSecondaryReplicaScope.NoteAsync(postgres, ServerId, logger: null, ct, asOfUtc: now.AddHours(-3))); + + /* An anchor before any snapshot knows nothing, so it skips nothing. */ + Assert.Empty(await PgSecondaryReplicaScope.ReadAsync(postgres, ServerId, now.AddDays(-30), logger: null, ct)); + }); + } + + /// + /// #5558 round 2 (L1): compare_analysis and get_analysis_facts build secondary_replica_note from the set their own context + /// filtered with, not from a second read of the role. The baseline window ends before the only snapshot (nothing known, + /// nothing skipped); the comparison window ends after it (SecDb skipped). Each returned set is its own. + /// + [Fact] + public async Task CompareAndFactsReads_CarryTheSetTheirContextFilteredWith_SoTheNoteCannotDisagree() + { + await WithStoreAsync(async (c, postgres, ct) => + { + var end = Now(); + await SeedAgAsync(c, ct, end.AddMinutes(-1)); + var service = new DarlingAnalysisService(postgres); + + var (_, _, _, _, _, baselineSecondaries, comparisonSecondaries) = await service.ComparePeriodsAsync( + ServerId, ServerName, DarlingMcpTestData.Naive(end.AddHours(-30)), DarlingMcpTestData.Naive(end.AddHours(-26)), + DarlingMcpTestData.Naive(end.AddHours(-4)), DarlingMcpTestData.Naive(end), ct); + Assert.Empty(baselineSecondaries!); + Assert.Null(AgReplicaScope.SkippedNote(baselineSecondaries)); + Assert.Equal(["SecDb"], comparisonSecondaries!.ToArray()); + Assert.Equal(AgReplicaScope.SkippedNote(1), AgReplicaScope.SkippedNote(comparisonSecondaries)); + + var (_, _, caveats) = await service.CollectAndScoreFactsAsync(ServerId, ServerName, 4, DarlingMcpTestData.Naive(end), ct); + Assert.Equal(["SecDb"], caveats.SecondaryReplicaDatabases!.ToArray()); + }); + } + + private static Task SeedRegressionAsync(NpgsqlConnection c, CancellationToken ct, DateTime end, string db, long planId, string hash, + long avgCpu, DateTime firstExec, DateTime lastExec, long queryId = 101) => + ExecAsync(c, ct, @" +INSERT INTO query_store_stats + (collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, execution_type_desc, + first_execution_time, last_execution_time, query_text, query_hash, execution_count, avg_cpu_time_us, + avg_duration_us, query_plan_hash, is_forced_plan, force_failure_count, runtime_stats_interval_id) +VALUES ($1,$2,$3,$4,$5,$11,$6,'Regular',$7,$8,'SELECT 1','0xQH',100,$9,$9,$10,FALSE,0,$6)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(end), ServerId, ServerName, db, planId, + DarlingMcpTestData.Naive(firstExec), DarlingMcpTestData.Naive(lastExec), avgCpu, hash, queryId); + + [Fact] + public async Task PlanRegression_DropsTheSecondaryDatabase() + { + await WithStoreAsync(async (c, postgres, ct) => + { + var end = Now(); + await SeedAgAsync(c, ct, end.AddMinutes(-1)); + foreach (var db in new[] { "SecDb", "StandDb" }) + { + await SeedRegressionAsync(c, ct, end, db, planId: 1, hash: "0xGOOD", avgCpu: 100_000, firstExec: end.AddDays(-6), lastExec: end.AddDays(-5)); + await SeedRegressionAsync(c, ct, end, db, planId: 2, hash: "0xBAD", avgCpu: 1_200_000, firstExec: end.AddDays(-1), lastExec: end); + } + + var (facts, context) = await FactsAsync(postgres, end, ct); + + Assert.Equal(1.0, One(facts, "PLAN_REGRESSION").Metadata["offender_count"]); + Assert.NotEmpty(context.PlanRegressionOffenders!); + Assert.All(context.PlanRegressionOffenders!, o => Assert.Equal("StandDb", o.DatabaseName)); + }); + } + + /// + /// L1 (round 1): the regressed-queries drill-down filters the secondary database in its source CTE, BEFORE its LIMIT 5, as + /// Lite's twin does. The pass where the fact did not run has no offender list, so every query is read; when the secondary + /// database holds the five worst regressions, a filter applied after the LIMIT would leave nothing for the primary's own. + /// + [Fact] + public async Task TheRegressedQueriesDrillDown_FiltersTheSecondaryDatabaseBeforeItsLimit() + { + await WithStoreAsync(async (c, postgres, ct) => + { + var end = Now(); + await SeedAgAsync(c, ct, end.AddMinutes(-1)); + for (var q = 0; q < 5; q++) + { + var queryId = 201 + q; + await SeedRegressionAsync(c, ct, end, "SecDb", planId: queryId * 10 + 1, hash: "0xGOOD", avgCpu: 100_000, firstExec: end.AddDays(-6), lastExec: end.AddDays(-5), queryId: queryId); + await SeedRegressionAsync(c, ct, end, "SecDb", planId: queryId * 10 + 2, hash: "0xBAD", avgCpu: 5_000_000, firstExec: end.AddDays(-1), lastExec: end, queryId: queryId); + } + foreach (var queryId in new long[] { 101, 102 }) + { + await SeedRegressionAsync(c, ct, end, "StandDb", planId: queryId * 10 + 1, hash: "0xGOOD", avgCpu: 100_000, firstExec: end.AddDays(-6), lastExec: end.AddDays(-5), queryId: queryId); + await SeedRegressionAsync(c, ct, end, "StandDb", planId: queryId * 10 + 2, hash: "0xBAD", avgCpu: 1_200_000, firstExec: end.AddDays(-1), lastExec: end, queryId: queryId); + } + + /* No fact ran, so PlanRegressionOffenders is null; the set itself is resolved as a pass would. */ + var context = Context(end); + context.CancellationToken = ct; + await PgSecondaryReplicaScope.EnsureAsync(postgres, context, logger: null); + Assert.Null(context.PlanRegressionOffenders); + + var json = await DrillDownJsonAsync(postgres, context, "PLAN_REGRESSION", severity: 0.6); + using var doc = JsonDocument.Parse(json); + var rows = doc.RootElement.GetProperty("regressed_queries").EnumerateArray().ToList(); + Assert.Equal(2, rows.Count); + Assert.All(rows, r => Assert.Equal("StandDb", r.GetProperty("database").GetString())); + }); + } + + private static Task SeedObjectAsync(NpgsqlConnection c, CancellationToken ct, DateTime at, string db, decimal reservedMb) => + ExecAsync(c, ct, @" +INSERT INTO index_object_stats + (collection_id, collection_time, server_id, server_name, database_name, database_id, schema_name, object_id, table_name, + index_id, index_name, reserved_mb, row_lock_wait_in_ms, index_lock_promotion_count) +VALUES ($1,$2,$3,$4,$5,7,'dbo',100,'Big',1,'PK_Big',$6,0,0)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(at), ServerId, ServerName, db, reservedMb); + + [Fact] + public async Task ObjectGrowth_DropsTheSecondaryDatabase() + { + await WithStoreAsync(async (c, postgres, ct) => + { + var end = Now(); + await SeedAgAsync(c, ct, end.AddMinutes(-1)); + foreach (var (db, grownMb) in new[] { ("SecDb", 900m), ("StandDb", 500m) }) + { + await SeedObjectAsync(c, ct, end.AddDays(-1), db, 100m); + await SeedObjectAsync(c, ct, end.AddHours(-1), db, 100m + grownMb); + } + + /* The baseline gate: anomaly detection sits out a server with no history to compare against. */ + await ExecAsync(c, ct, + "INSERT INTO cpu_utilization_stats (collection_id, collection_time, server_id, server_name, sample_time, sqlserver_cpu_utilization, other_process_cpu_utilization) VALUES ($1, $2, $3, $4, $2, 10, 5)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(end.AddDays(-2)), ServerId, ServerName); + + var context = Context(end); + context.CancellationToken = ct; + await PgSecondaryReplicaScope.EnsureAsync(postgres, context, logger: null); + var detector = new PgAnomalyDetector(postgres, new PgBaselineProvider(postgres)); + var anomalies = await detector.DetectAnomaliesAsync(context); + + var growth = Assert.Single(anomalies, f => f.Key == "ANOMALY_OBJECT_GROWTH"); + Assert.Equal("StandDb", growth.DatabaseName); + }); + } + + /// + /// get_analysis_findings carries the shared one-sentence note beside a non-empty list and inside the empty answer when + /// this node holds a secondary copy. Lite's twin is in McpAnalysisFindingsCommandTests. + /// + [Fact] + public async Task GetAnalysisFindings_CarriesTheSecondaryReplicaNote_OnlyWhenThisNodeHoldsASecondaryCopy() + { + await WithStoreAsync(async (c, postgres, ct) => + { + var now = Now(); + var service = new DarlingAnalysisService(postgres); + + /* Nothing stored, no AG rows: the plain empty answer, no note. */ + using (var plain = JsonDocument.Parse(await DarlingMcpTools.GetAnalysisFindings(service, postgres, ServerName))) + { + Assert.Equal("empty", plain.RootElement.GetProperty("status").GetString()); + Assert.DoesNotContain("secondary copy", plain.RootElement.GetProperty("message").GetString(), StringComparison.Ordinal); + } + + /* A secondary copy, still no findings: the empty answer names the skipped database. */ + await SeedAgAsync(c, ct, now.AddMinutes(-1)); + using (var empty = JsonDocument.Parse(await DarlingMcpTools.GetAnalysisFindings(service, postgres, ServerName))) + { + Assert.Contains(AgReplicaScope.SkippedNote(1)!, empty.RootElement.GetProperty("message").GetString(), StringComparison.Ordinal); + } + + /* With a finding stored the field rides on the payload. */ + var t = now.AddMinutes(-10); + await ExecAsync(c, ct, @" +INSERT INTO analysis_findings + (finding_id, analysis_time, server_id, server_name, database_name, time_range_start, time_range_end, severity, confidence, + category, story_path, story_path_hash, story_text, root_fact_key, root_fact_value, leaf_fact_key, leaf_fact_value, fact_count, + incident_id, remediation_action_json, drill_down_json) +VALUES ($1, $2, $3, $4, 'StandDb', $5, $2, 0.72, 0.55, 'config', 'DB_CONFIG', 'AG5558_HASH', '', 'DB_CONFIG', 2, 'DB_CONFIG', 2, 1, + 'AG5558_INCIDENT', NULL, NULL)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(t), ServerId, ServerName, DarlingMcpTestData.Naive(t.AddHours(-4))); + + using (var withNote = JsonDocument.Parse(await DarlingMcpTools.GetAnalysisFindings(service, postgres, ServerName))) + { + Assert.Equal(AgReplicaScope.SkippedNote(1), withNote.RootElement.GetProperty("secondary_replica_note").GetString()); + } + }); + } + + private static Task PlantWaitAsync(NpgsqlConnection c, CancellationToken ct, DateTime at, long waitMs) => ExecAsync(c, ct, @" +INSERT INTO wait_stats + (collection_id, collection_time, server_id, server_name, wait_type, delta_waiting_tasks, delta_wait_time_ms) +VALUES ($1, $2, $3, $4, 'AN5558_WAIT', 50, $5)", CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(at), ServerId, ServerName, waitMs); + + /// The note from wherever the answer's shape puts it: the root of a data-bearing answer, or the hints of a miss. + private static string? NoteOf(string json) + { + using var doc = JsonDocument.Parse(json); + var root = doc.RootElement; + var carrier = root.TryGetProperty("secondary_replica_note", out _) ? root : root.GetProperty("hints"); + var note = carrier.GetProperty("secondary_replica_note"); + return note.ValueKind == JsonValueKind.Null ? null : note.GetString(); + } + + /// + /// M4 (round 1): analyze_server carries the same note get_analysis_findings does, under the same field name, always present + /// and null when nothing is skipped; the service keeps the sentence the pass used. Lite's twin is in + /// SecondaryReplicaMcpAnalysisTests. + /// + [Fact] + public async Task AnalyzeServer_CarriesTheNote_OnlyWhenThisNodeHoldsASecondaryCopy() + { + await WithStoreAsync(async (c, postgres, ct) => + { + var service = new DarlingAnalysisService(postgres) { MinimumDataHours = 0 }; + + Assert.Null(NoteOf(await DarlingMcpTools.AnalyzeServer(service, postgres, ServerName, 4))); + Assert.Null(service.LastSecondaryReplicaNote); + + await SeedAgAsync(c, ct, Now().AddMinutes(-1)); + var answer = await DarlingMcpTools.AnalyzeServer(service, postgres, ServerName, 4); + Assert.Equal(AgReplicaScope.SkippedNote(1), NoteOf(answer)); + Assert.Equal(AgReplicaScope.SkippedNote(1), service.LastSecondaryReplicaNote); + + using var doc = JsonDocument.Parse(answer); + if (doc.RootElement.GetProperty("status").GetString() == "empty") + Assert.Contains(AgReplicaScope.SkippedNote(1)!, doc.RootElement.GetProperty("message").GetString(), StringComparison.Ordinal); + }); + } + + /// + /// Round 2: get_analysis_facts returns facts the secondary filter already trimmed, so it carries the same note, in + /// every envelope it can answer with, always present and null when nothing is skipped. Lite's twin is in + /// SecondaryReplicaMcpAnalysisTests. + /// + [Fact] + public async Task GetAnalysisFacts_CarriesTheNote_OnlyWhenThisNodeHoldsASecondaryCopy() + { + await WithStoreAsync(async (c, postgres, ct) => + { + var service = new DarlingAnalysisService(postgres) { MinimumDataHours = 0 }; + + /* Nothing collected, no AG rows: an unavailable answer, the field there and null. */ + Assert.Null(NoteOf(await DarlingMcpTools.GetAnalysisFacts(service, postgres, ServerName, 4, cancellationToken: ct))); + + /* A secondary copy, still nothing collected: the sentence is in the field and in the prose. */ + await SeedAgAsync(c, ct, Now().AddMinutes(-1)); + var bare = await DarlingMcpTools.GetAnalysisFacts(service, postgres, ServerName, 4, cancellationToken: ct); + Assert.Equal(AgReplicaScope.SkippedNote(1), NoteOf(bare)); + using (var bareDoc = JsonDocument.Parse(bare)) + Assert.Contains(AgReplicaScope.SkippedNote(1)!, bareDoc.RootElement.GetProperty("message").GetString(), StringComparison.Ordinal); + + /* Facts in the window: the facts payload carries it at the root. */ + for (var m = 5; m <= 55; m += 10) await PlantWaitAsync(c, ct, Now().AddMinutes(-m), 150_000L); + var withFacts = await DarlingMcpTools.GetAnalysisFacts(service, postgres, ServerName, 4, cancellationToken: ct); + using var factsDoc = JsonDocument.Parse(withFacts); + Assert.True(factsDoc.RootElement.TryGetProperty("facts", out _), "expected the facts payload, got: " + withFacts[..Math.Min(withFacts.Length, 200)]); + Assert.Equal(AgReplicaScope.SkippedNote(1), factsDoc.RootElement.GetProperty("secondary_replica_note").GetString()); + }); + } + + [Fact] + public async Task GetAnalysisFacts_OnAPrimary_CarriesANullNote() + { + await WithStoreAsync(async (c, postgres, ct) => + { + await SeedAgAsync(c, ct, Now().AddMinutes(-1), localRole: "PRIMARY"); + for (var m = 5; m <= 55; m += 10) await PlantWaitAsync(c, ct, Now().AddMinutes(-m), 150_000L); + var answer = await DarlingMcpTools.GetAnalysisFacts( + new DarlingAnalysisService(postgres) { MinimumDataHours = 0 }, postgres, ServerName, 4, cancellationToken: ct); + Assert.Null(NoteOf(answer)); + }); + } + + /// M4 (round 1): compare_analysis carries one note per window, each from the role at that window's own end. + [Fact] + public async Task CompareAnalysis_CarriesOneNotePerWindow_EachFromItsOwnRole() + { + await WithStoreAsync(async (c, postgres, ct) => + { + var now = Now(); + var baselineEnd = now.AddHours(-24); + + /* Secondary when the baseline window ended, primary after a failover by the time the comparison window ends. */ + await SeedAgAsync(c, ct, baselineEnd.AddMinutes(-1)); + await SeedAgAsync(c, ct, now.AddMinutes(-1), localRole: "PRIMARY"); + await PlantWaitAsync(c, ct, baselineEnd.AddHours(-1), 500_000L); + await PlantWaitAsync(c, ct, now.AddHours(-1), 900_000L); + + var answer = await DarlingMcpTools.CompareAnalysis( + new DarlingAnalysisService(postgres) { MinimumDataHours = 0 }, postgres, ServerName, 4, 28, cancellationToken: ct); + + using var doc = JsonDocument.Parse(answer); + var root = doc.RootElement; + var (baseline, comparison) = root.TryGetProperty("baseline", out var b) + ? (b.GetProperty("secondary_replica_note"), root.GetProperty("comparison").GetProperty("secondary_replica_note")) + : (root.GetProperty("hints").GetProperty("baseline_secondary_replica_note"), root.GetProperty("hints").GetProperty("comparison_secondary_replica_note")); + Assert.Equal(AgReplicaScope.SkippedNote(1), baseline.GetString()); + Assert.Equal(JsonValueKind.Null, comparison.ValueKind); + }); + } +} diff --git a/Darling/Darling.Tests/SecondaryReplicaScopeTests.cs b/Darling/Darling.Tests/SecondaryReplicaScopeTests.cs new file mode 100644 index 0000000000..99bc1a4b94 --- /dev/null +++ b/Darling/Darling.Tests/SecondaryReplicaScopeTests.cs @@ -0,0 +1,277 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Runtime.CompilerServices; +using System.Text.RegularExpressions; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Analysis; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Analysis; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5558: 's predicate builder and its fail-open read, with no database. The seeded +/// twins of Lite's fact tests are in . +/// +[Trait("Reads", "Lite")] +public sealed class PgSecondaryReplicaScopeTests +{ + private static AnalysisContext ContextWith(params string[] secondaries) => new() + { + SecondaryReplicaDatabases = new HashSet(secondaries, StringComparer.Ordinal), + }; + + [Fact] + public void Apply_WithNothingToSkip_JustRemovesTheMarker_AndAddsNoParameter() + { + using var cmd = new NpgsqlCommand("SELECT 1 FROM t WHERE a = $1 /*SEC*/ ORDER BY 1"); + cmd.Parameters.AddWithValue(1); + + PgSecondaryReplicaScope.Apply(cmd, ContextWith(), "DB_CONFIG", "database_name"); + + Assert.DoesNotContain("/*SEC*/", cmd.CommandText, StringComparison.Ordinal); + Assert.Equal("SELECT 1 FROM t WHERE a = $1 ORDER BY 1", cmd.CommandText); + Assert.Single(cmd.Parameters); + } + + [Fact] + public void Apply_WithASecondary_BindsTheNamesAsTheyAreAsOneArray_NumberedAfterTheExistingParameters() + { + using var cmd = new NpgsqlCommand("SELECT 1 FROM t WHERE a = $1 /*SEC*/ ORDER BY 1"); + cmd.Parameters.AddWithValue(1); + + PgSecondaryReplicaScope.Apply(cmd, ContextWith("SecDb", "OtherDb"), "DB_CONFIG", "database_name"); + + Assert.Contains("AND (database_name IS NULL OR database_name <> ALL($2::text[]))", cmd.CommandText, StringComparison.Ordinal); + Assert.Equal(2, cmd.Parameters.Count); + var names = Assert.IsType(cmd.Parameters[1].Value); + Assert.Equal(["OtherDb", "SecDb"], names.OrderBy(n => n, StringComparer.Ordinal).ToArray()); + } + + [Fact] + public void Apply_MatchesTheNameExactly_ASecondarySalesDbDoesNotHideASeparateSalesdb() + { + /* #5558 round 2: on a case-sensitive server collation, SalesDb (a secondary copy) and salesdb (a standalone + database) are two databases. The set carries only the secondary's exact name and the predicate compares + it as is, so the standalone one keeps its rows. */ + using var cmd = new NpgsqlCommand("SELECT 1 FROM t WHERE a = $1 /*SEC*/"); + cmd.Parameters.AddWithValue(1); + + PgSecondaryReplicaScope.Apply(cmd, ContextWith("SalesDb"), "DB_CONFIG", "database_name"); + + Assert.Equal(["SalesDb"], Assert.IsType(cmd.Parameters[1].Value)); + Assert.DoesNotContain("lower(", cmd.CommandText, StringComparison.Ordinal); + } + + /// + /// #5558 round 2 (L4): the regressed-queries drill-down has two SQL variants, the raw one and the interval-table one, and + /// the live drill-down test runs whichever the seeded store picks. Each must carry the marker exactly once and before its + /// LIMIT 5; a variant that lost it would leave Apply binding a parameter nothing references and the filter + /// would silently not run there. + /// + [Theory] + [InlineData("raw")] + [InlineData("table")] + public void EachRegressedQueriesVariant_CarriesTheMarkerExactlyOnce_BeforeItsLimit(string variant) + { + var sql = variant == "raw" ? PgDrillDownCollector.RegressedQueriesSql : PgDrillDownCollector.RegressedQueriesTableSql; + + var first = sql.IndexOf(PgSecondaryReplicaScope.Marker, StringComparison.Ordinal); + Assert.True(first >= 0, variant + " variant lost the secondary-replica marker"); + Assert.Equal(-1, sql.IndexOf(PgSecondaryReplicaScope.Marker, first + PgSecondaryReplicaScope.Marker.Length, StringComparison.Ordinal)); + + var limit = sql.IndexOf("LIMIT 5", first, StringComparison.Ordinal); + Assert.True(limit > first, variant + " variant: the marker must come before its LIMIT 5, so the filter runs before the cut"); + } + + [Fact] + public void Apply_UsesTheKeywordItIsGiven_ForAClauseThatOpensTheWhere() + { + using var cmd = new NpgsqlCommand("SELECT 1 FROM t /*SEC*/ GROUP BY 1"); + PgSecondaryReplicaScope.Apply(cmd, ContextWith("SecDb"), "PLAN_REGRESSION", "database_name", "WHERE"); + Assert.Contains(" WHERE (database_name IS NULL OR database_name <> ALL($1::text[]))", cmd.CommandText, StringComparison.Ordinal); + } + + [Fact] + public void Apply_LeavesANodeLocalKeyUnfiltered_EvenWithASecondarySet() + { + using var cmd = new NpgsqlCommand("SELECT 1 FROM t WHERE a = $1 /*SEC*/"); + cmd.Parameters.AddWithValue(1); + + PgSecondaryReplicaScope.Apply(cmd, ContextWith("SecDb"), "MISSING_INDEX", "database_name"); + + Assert.DoesNotContain("/*SEC*/", cmd.CommandText, StringComparison.Ordinal); + Assert.DoesNotContain("<> ALL(", cmd.CommandText, StringComparison.Ordinal); + Assert.Single(cmd.Parameters); + } + + [Fact] + public void Apply_WithNoSetOnTheContext_SkipsNothing() + { + using var cmd = new NpgsqlCommand("SELECT 1 FROM t WHERE a = $1 /*SEC*/"); + cmd.Parameters.AddWithValue(1); + + PgSecondaryReplicaScope.Apply(cmd, new AnalysisContext(), "DB_CONFIG", "database_name"); + + Assert.DoesNotContain("<> ALL(", cmd.CommandText, StringComparison.Ordinal); + Assert.Single(cmd.Parameters); + } + + [Fact] + public async Task ReadAsync_FailsOpen_WhenTheStoreCannotBeReached() + { + /* A port nothing listens on: the connection fails at once, and the answer is the empty set, not a throw. */ + await using var unreachable = NpgsqlDataSource.Create( + "Host=127.0.0.1;Port=1;Username=x;Database=x;Timeout=2;Pooling=false"); + + var set = await PgSecondaryReplicaScope.ReadAsync(unreachable, 1, DateTime.UtcNow, logger: null, CancellationToken.None); + Assert.Empty(set); + + Assert.Null(await PgSecondaryReplicaScope.NoteAsync(unreachable, 1, logger: null, CancellationToken.None)); + } + + [Fact] + public async Task EnsureAsync_FillsTheSetOnce_AndLeavesAnAlreadyResolvedSetAlone() + { + await using var unreachable = NpgsqlDataSource.Create( + "Host=127.0.0.1;Port=1;Username=x;Database=x;Timeout=2;Pooling=false"); + + var context = new AnalysisContext { ServerId = 1, TimeRangeEnd = DateTime.UtcNow }; + await PgSecondaryReplicaScope.EnsureAsync(unreachable, context, logger: null); + Assert.NotNull(context.SecondaryReplicaDatabases); + Assert.Empty(context.SecondaryReplicaDatabases!); + + var resolved = new HashSet(StringComparer.Ordinal) { "SecDb" }; + var already = new AnalysisContext { ServerId = 1, SecondaryReplicaDatabases = resolved }; + await PgSecondaryReplicaScope.EnsureAsync(unreachable, already, logger: null); + Assert.Same(resolved, already.SecondaryReplicaDatabases); + } + + [Fact] + public void NoneSkipped_IsEmpty() + { + Assert.Empty(PgSecondaryReplicaScope.NoneSkipped); + } +} + +/// +/// #5558: every new AnalysisContext in the product code resolves the secondary replica set, either in its own +/// initializer or through EnsureAsync on the variable it is assigned to. A context built without it leaves +/// null, which skips nothing, so the replicated findings of a +/// secondary copy would quietly come back for that one pass. A pass that reads only node-local facts says so by setting the +/// empty set on purpose. Scans Lite and Darling; test projects build contexts freely and are not scanned. +/// +[Trait("Reads", "Lite")] +public sealed class AnalysisContextSecondaryScopeSourceScanTests +{ + private static string RepoRoot([CallerFilePath] string thisFile = "") + { + var dir = Path.GetDirectoryName(thisFile)!; + while (dir is not null && !File.Exists(Path.Combine(dir, "Directory.Build.props"))) + dir = Path.GetDirectoryName(dir); + Assert.False(dir is null, "could not locate the repo root from the test source path"); + return dir!; + } + + private static IEnumerable ProductSources(string root) + { + foreach (var top in new[] { "Lite", "Darling" }) + { + foreach (var file in Directory.EnumerateFiles(Path.Combine(root, top), "*.cs", SearchOption.AllDirectories)) + { + var relative = Path.GetRelativePath(root, file).Replace('\\', '/'); + if (relative.Contains("/bin/") || relative.Contains("/obj/") || relative.Contains(".Tests/")) continue; + yield return file; + } + } + } + + [Fact] + public void EveryAnalysisContextInTheProduct_ResolvesTheSecondaryReplicaSet() + { + var root = RepoRoot(); + var construction = new Regex(@"(?:var\s+(?\w+)\s*=\s*)?new\s+AnalysisContext\b", RegexOptions.CultureInvariant); + var missing = new List(); + var sites = 0; + + /* The end of the member a construction sits in: the next closing brace at member indentation (4 spaces). + Each site is judged on its own text, so a second context in a file that already ensures a variable of the + same name cannot borrow the first one's call. */ + var memberEnd = new Regex(@"\n \}[ \t]*\n", RegexOptions.CultureInvariant); + + foreach (var file in ProductSources(root)) + { + var text = File.ReadAllText(file).Replace("\r\n", "\n"); + foreach (Match m in construction.Matches(text)) + { + sites++; + var end = memberEnd.Match(text, m.Index); + var limit = end.Success ? end.Index : text.Length; + var member = text[m.Index..limit]; + /* The initializer's closing brace is searched only up to the member's end: an inline construction + such as Foo(new AnalysisContext { ... }); closes with "})", and an unbounded search would run on into + a LATER construction's initializer and borrow its SecondaryReplicaDatabases setting (#5558 round 2). */ + var close = text.IndexOf("};", m.Index, limit - m.Index, StringComparison.Ordinal); + var initializer = close < 0 ? member : text[m.Index..close]; + var name = m.Groups["name"].Value; + var inInitializer = initializer.Contains("SecondaryReplicaDatabases", StringComparison.Ordinal); + var ensured = name.Length > 0 && Regex.IsMatch(member, @"SecondaryReplicaScope\.EnsureAsync\([^)]*\b" + Regex.Escape(name) + @"\b"); + + /* A context handed straight to AnalyzeAsync(context) is resolved by that overload, provided the overload + itself ensures the set it receives (checked on the same file's declaration, so the hand-off cannot + be a loophole). */ + if (!inInitializer && !ensured && name.Length > 0 && Regex.IsMatch(member, @"\bAnalyzeAsync\(\s*" + Regex.Escape(name) + @"\s*\)")) + { + var overload = Regex.Match(text, @"AnalyzeAsync\(AnalysisContext (?

\w+)\)"); + if (overload.Success) + { + var overloadEnd = memberEnd.Match(text, overload.Index); + var body = overloadEnd.Success ? text[overload.Index..overloadEnd.Index] : text[overload.Index..]; + ensured = Regex.IsMatch(body, @"SecondaryReplicaScope\.EnsureAsync\([^)]*\b" + overload.Groups["p"].Value + @"\b"); + } + } + if (!inInitializer && !ensured) + { + var line = text[..m.Index].Count(c => c == '\n') + 1; + missing.Add(Path.GetRelativePath(root, file).Replace('\\', '/') + ":" + line); + } + } + } + + Assert.True(sites >= 10, "the scan found only " + sites + " construction sites, so its pattern no longer matches the code"); + Assert.True(missing.Count == 0, + "an AnalysisContext that never resolves SecondaryReplicaDatabases would bring back a secondary copy's findings: " + string.Join(", ", missing)); + } +} + +///

#5558: the viewer Recommendations view model carries the note only where a list or the all-clear shows. +[Trait("Reads", "Lite")] +public sealed class RecommendationsReplicaNoteTests +{ + [Fact] + public void TheNote_IsCarriedOnTheEmptyAndLoadedStates_AndNullWhenNotSet() + { + var empty = RecommendationsViewModel.FromFindings(Array.Empty(), "SQL2022"); + Assert.Null(empty.ReplicaNote); + + var note = AgReplicaScope.SkippedNote(2); + Assert.Equal(note, empty.WithReplicaNote(note).ReplicaNote); + Assert.Same(empty, empty.WithReplicaNote(" ")); + Assert.Null(empty.ReplicaNote); + Assert.Equal(RecommendationsState.Empty, empty.State); + } +} diff --git a/Darling/Darling.Tests/SelfAlertDeliveryStampTests.cs b/Darling/Darling.Tests/SelfAlertDeliveryStampTests.cs index dc001f3178..5ec157dbc4 100644 --- a/Darling/Darling.Tests/SelfAlertDeliveryStampTests.cs +++ b/Darling/Darling.Tests/SelfAlertDeliveryStampTests.cs @@ -427,7 +427,7 @@ public void TheWorkerStampsTheHourlyTickAndTheFleetSweep_OnTheGrid() var worker = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "DarlingWorker.cs") .Replace("\r\n", "\n", StringComparison.Ordinal); - const string hourly = "_nextStoreMetricsUtc = NextGridStamp(_nextStoreMetricsUtc, DateTime.UtcNow, s_storeMetricsInterval);"; + const string hourly = "_nextStoreMetricsUtc = NextGridStamp(_nextStoreMetricsUtc, nowUtc, s_storeMetricsInterval);"; const string sweep = "_nextFleetSweepUtc = NextGridStamp(_nextFleetSweepUtc, DateTime.UtcNow, TimeSpan.FromMinutes(fleetSweepMinutes));"; Assert.Equal(1, worker.Split(hourly).Length - 1); Assert.Equal(1, worker.Split(sweep).Length - 1); diff --git a/Darling/Darling.Tests/SelfAlertFailedSendCensusTests.cs b/Darling/Darling.Tests/SelfAlertFailedSendCensusTests.cs index e280492a6c..241afe7987 100644 --- a/Darling/Darling.Tests/SelfAlertFailedSendCensusTests.cs +++ b/Darling/Darling.Tests/SelfAlertFailedSendCensusTests.cs @@ -34,6 +34,7 @@ namespace Darling.Tests; /// AfterSelfFire arm pins the metric and the interval the arm passes, because the interval is the one /// argument a wrong copy would leave compiling: the back-dated stamp then opens the arm's gate at the wrong time. /// +[Trait("Stage", "Guard")] public sealed class SelfAlertFailedSendCensusTests { private const string RestoreNotice = @@ -41,6 +42,10 @@ public sealed class SelfAlertFailedSendCensusTests private const string ArmHandOff = "AfterSelfFire"; private const string RetryHandOff = "NoteRetrySend"; + + /* #5493: the lasting-state self-alerts hand their answer to the state alert tracker (NoteStateSend(metric, grain, answer)), + which is NoteRetrySend over the shared state retries. */ + private const string StateHandOff = "NoteStateSend"; private const string DocumentHandOff = "RecordDocumentDeliveredAsync"; /// @@ -56,10 +61,14 @@ public sealed class SelfAlertFailedSendCensusTests ["ApplyAgReplicaHealthAsync"] = (1, RestoreNotice), ["EvaluateStoreUpgradeAsync"] = (2, "one notice per service start about the start's own upgrade: an event, never re-evaluated, so nothing to fire again"), + ["EvaluateCollectionGapAtStartAsync"] = + (1, "one notice per service start about the gap before the start: an event, never re-evaluated, so nothing to fire again"), ["EvaluateStoreTimescaleAsync"] = (1, "one notice per service start about the start's own extension update: an event, never re-evaluated"), ["ApplyNotificationChannelsAsync"] = (1, "the failing-channel notice is stated once when the policy decides it; the next decision is the channel's recovery"), + ["GiveUpOnHistoryAuditDayAsync"] = + (1, "the history audit gave up on a day at the next day's slot: the day is not audited again, so there is no later tick to send it on, and the stamp that matters is the new slot's own"), ["ApplyPolicyJobsStuckAsync"] = (1, "the auto-re-armed notice is a state-machine edge: the next pass escalates or records the recovery, and neither is gated by a stamp"), }; @@ -74,6 +83,7 @@ public sealed class SelfAlertFailedSendCensusTests "ApplyCollectorCostDigestAsync", "ApplyFleetSweepRollupAsync", "ApplyAnalysisSinglesDigestAsync", + "ApplyCollectionHistoryAuditAsync", }; /// The evaluator's source with comments and literal text blanked (offsets and newlines kept), read @@ -91,7 +101,7 @@ public sealed class SelfAlertFailedSendCensusTests @"\bawait\s+FireAsync\(", RegexOptions.CultureInvariant); private static readonly Regex HandOffCall = new( - @"\b(?" + ArmHandOff + "|" + RetryHandOff + "|" + DocumentHandOff + @")\s*\(", + @"\b(?" + ArmHandOff + "|" + RetryHandOff + "|" + StateHandOff + "|" + DocumentHandOff + @")\s*\(", RegexOptions.CultureInvariant); /// One await FireAsync( in code (not in a comment or a literal): where it is, its enclosing @@ -235,7 +245,9 @@ private static List CallArguments(string code, int open) /// Whether is the call that acts on a send's answer in /// : the document stamp for a daily document, the retry hand-offs for every arm. private static bool IsHandOffFor(string method, string callee) => - Digests.Contains(method) ? callee == DocumentHandOff : callee is ArmHandOff or RetryHandOff; + /* #5493: NoteStateSend is the state alerts' hand-off and forwards to NoteRetrySend; that forwarding is not a send's hand-off. */ + method != StateHandOff + && (Digests.Contains(method) ? callee == DocumentHandOff : callee is ArmHandOff or RetryHandOff or StateHandOff); /// The hand-off that receives 's answer: the first one in the same method /// after the call and before the method's next FireAsync, or null when there is none. A hand-off that @@ -268,7 +280,8 @@ public void TheCensusFindsTheSends_AndTheMethodsThatMakeThem() var handOffs = HandOffs(EvaluatorCode.Value); Assert.True( - handOffs.Count(h => h.Callee == ArmHandOff) >= 15 && handOffs.Count(h => h.Callee == RetryHandOff) >= 3, + handOffs.Count(h => h.Callee == ArmHandOff) >= 6 && handOffs.Count(h => h.Callee == RetryHandOff) >= 3 + && handOffs.Count(h => h.Callee == StateHandOff) >= 11, $"expected the evaluator's AfterSelfFire and NoteRetrySend calls, found {handOffs.Count} hand-off(s) in all"); Assert.DoesNotContain(handOffs, h => h.Method == ""); Assert.All(handOffs, h => Assert.NotEmpty(h.Arguments)); @@ -373,7 +386,7 @@ a hand-off whose send was dropped leaves it long. */ if (kept.Count != handed.Count) { - var callees = Digests.Contains(method) ? DocumentHandOff : $"{ArmHandOff} or {RetryHandOff}"; + var callees = Digests.Contains(method) ? DocumentHandOff : $"{ArmHandOff}, {RetryHandOff} or {StateHandOff}"; problems.Add( $"{method} keeps {kept.Count} FireAsync answer(s) (lines {string.Join(", ", kept.Select(s => s.Line))}) " @@ -417,22 +430,15 @@ the local the send landed in. A local that nothing reads is a kept answer nobody /// hand-off with the wrong interval still compiles, and back-dates the stamp so the gate opens early or never. /// [Theory] - [InlineData("ApplyAgDatabaseHealthAsync", 1, "AgSyncFellBehindMetric", "SharedCooldown")] [InlineData("ApplyDiskPressureAsync", 0, "DiskPressureMetric", "SharedCooldown")] - [InlineData("ApplyCustomRuleHealthAsync", 0, "CustomRuleHealthMetric", "SharedCooldown")] [InlineData("ApplyStaleMuteRulesAsync", 0, "StaleMuteMetric", "StaleMuteRefire")] [InlineData("ApplyStoreSettingsAsync", 0, "StoreSettingsMetric", "StoreSettingsRefire")] - [InlineData("ApplyWebTlsCertificateAsync", 0, "WebTlsCertExpiryMetric", "WebTlsCertRefire")] - [InlineData("ApplyFleetGateAsync", 0, "FleetGateMetric", "SharedCooldown")] - [InlineData("ApplyStoreJobCadenceAsync", 0, "JobCadenceMetric", "SharedCooldown")] - [InlineData("ApplyRetentionHoldsAsync", 0, "RetentionHoldMetric", "SharedCooldown")] - [InlineData("ApplyRawPurgeOverHorizonAsync", 0, "RawPurgeOverHorizonMetric", "SharedCooldown")] + /* #5288: the web and MCP certificate alerts are one body, ApplyListenerTlsCertificateAsync, which fires and + reports under its listener descriptor's metric (the band.Metric precedent below), so ONE row covers both + listeners. The interval is still the shared daily WebTlsCertRefire. */ + [InlineData("ApplyListenerTlsCertificateAsync", 0, "tls.ExpiryMetric", "WebTlsCertRefire")] [InlineData("ApplyToastSlackAsync", 0, "ToastSlackMetric", "ToastSlackRefire")] [InlineData("ApplyCheckpointerPressureAsync", 0, "CheckpointerPressureMetric", "SharedCooldown")] - [InlineData("ApplyPolicyJobsStuckAsync", 0, "band.Metric", "SharedCooldown")] - [InlineData("ApplyPolicyJobsStuckAsync", 1, "band.Metric", "SharedCooldown")] - [InlineData("ApplyPolicyJobsStuckAsync", 2, "band.Metric", "SharedCooldown")] - [InlineData("ApplyPolicyJobsStuckAsync", 3, "band.Metric", "SharedCooldown")] public void ArmHandsItsSendsAnswerToAfterSelfFire_WithItsOwnMetricAndInterval( string method, int send, string metric, string interval) { @@ -459,4 +465,47 @@ public void ArmHandsItsSendsAnswerToAfterSelfFire_WithItsOwnMetricAndInterval( Assert.True(arguments[4] == interval, $"{method} line {handOff.Line}: AfterSelfFire passes {arguments[4]} as the interval, not {interval}"); Assert.True(arguments[5] == site.Variable, $"{method} line {handOff.Line}: AfterSelfFire is given {arguments[5]}, not the answer kept at line {site.Line} ({site.Variable})"); } + + /// + /// #5493: one case per lasting-state self-alert. Its send's answer goes to NoteStateSend(metric, grain, + /// answer), which keeps the failed-send retry on the shared state retries; there is no cooldown stamp to + /// back-date, because the alert repeats only per connection_refire_minutes (or to send again an alert no + /// channel took). The send carries the metric, and the hand-off reports under that same metric and takes that + /// send's own answer. + /// + [Theory] + [InlineData("ApplyCaptureDownAsync", 0, "CaptureDownMetric")] + [InlineData("ApplyAgentNotRunningAsync", 0, "AgentDownMetric")] + [InlineData("ApplyAgDatabaseHealthAsync", 1, "AgSyncFellBehindMetric")] + [InlineData("ApplyCustomRuleHealthAsync", 0, "CustomRuleHealthMetric")] + [InlineData("ApplyFleetGateAsync", 0, "FleetGateMetric")] + [InlineData("ApplyStoreJobCadenceAsync", 0, "JobCadenceMetric")] + [InlineData("ApplyRetentionHoldsAsync", 0, "RetentionHoldMetric")] + [InlineData("ApplyRawPurgeOverHorizonAsync", 0, "RawPurgeOverHorizonMetric")] + [InlineData("ApplyPolicyJobsStuckAsync", 0, "band.Metric")] + [InlineData("ApplyPolicyJobsStuckAsync", 1, "band.Metric")] + [InlineData("ApplyPolicyJobsStuckAsync", 2, "band.Metric")] + [InlineData("ApplyPolicyJobsStuckAsync", 3, "band.Metric")] + public void StateAlertHandsItsSendsAnswerToNoteStateSend_WithItsOwnMetric(string method, int send, string metric) + { + var code = EvaluatorCode.Value; + var sites = FireSites(code); + var kept = sites.Where(s => s.Method == method && s.KeepsResult).ToList(); + + Assert.True(send < kept.Count, $"{method} keeps {kept.Count} FireAsync answer(s); this case names send {send}"); + + var site = kept[send]; + var sent = CallArguments(code, code.IndexOf('(', site.Index)); + Assert.True(sent.Count > 2 && sent[2] == metric, $"{method} send {send} (line {site.Line}) carries {(sent.Count > 2 ? sent[2] : "no metric")}, not {metric}"); + + var handOff = HandOffFollowing(site, sites, HandOffs(code)); + Assert.True(handOff is not null, $"{method} send {send} (line {site.Line}) is not handed to NoteStateSend before the method's next send"); + Assert.True(handOff.Callee == StateHandOff, $"{method} send {send} (line {site.Line}) is handed to {handOff.Callee}, not NoteStateSend"); + + /* NoteStateSend(metric, grainKey, delivery) */ + var arguments = handOff.Arguments; + Assert.True(arguments.Count == 3, $"{method} line {handOff.Line}: NoteStateSend takes 3 arguments, found {arguments.Count}"); + Assert.True(arguments[0] == metric, $"{method} line {handOff.Line}: NoteStateSend reports under {arguments[0]}, not {metric}"); + Assert.True(arguments[2] == site.Variable, $"{method} line {handOff.Line}: NoteStateSend is given {arguments[2]}, not the answer kept at line {site.Line} ({site.Variable})"); + } } diff --git a/Darling/Darling.Tests/SelfAlertFailedSendRetryTests.cs b/Darling/Darling.Tests/SelfAlertFailedSendRetryTests.cs index 80543bc62f..ca428c962d 100644 --- a/Darling/Darling.Tests/SelfAlertFailedSendRetryTests.cs +++ b/Darling/Darling.Tests/SelfAlertFailedSendRetryTests.cs @@ -8,6 +8,7 @@ using System; using System.Collections.Generic; +using System.Linq; using System.Threading; using System.Threading.Tasks; using PerformanceMonitor.Alerting; @@ -53,6 +54,15 @@ public enum Arm Arm.CollectionStopped, Arm.CaptureDown, Arm.AgentDown, Arm.CostRegression, }; + /* #5489: Collection Stopped is a state alert now, so a delivered fire is NOT repeated after the cooldown (see + CollectionStoppedStateAlertTests). #5493 did the same for Capture Down and Agent Not Running (see + StateSelfAlertTests). The two tests below that pin "a delivered fire waits the whole cooldown and then fires + again" run for the one arm left that repeats per cooldown: the cost regression, a threshold alert. */ + public static TheoryData CooldownRepeatArms => new() + { + Arm.CostRegression, + }; + /* The delivery shapes the channels really produce, as PgAlertFailedSendRetryTests and AlertEngineTests build them: the deliverer answers these same values for a self-alert. */ private static AlertDelivery FailedByWebhook() => AlertDelivery.FromFanout( @@ -103,10 +113,16 @@ private sealed class Rig public Rig() { Evaluator = new DarlingSelfAlertEvaluator( - Settings, Deliverer, new DarlingSelfAlertTests.FakeHistoryStore(), _ => false, - logger: new DarlingSelfAlertTests.CapturingLogger(), utcNow: () => Now); + Settings, Deliverer, History, _ => false, + logger: new DarlingSelfAlertTests.CapturingLogger(), utcNow: () => Now, + connectionRefireMinutes: () => ConnectionRefireMinutes); } + public DarlingSelfAlertTests.FakeHistoryStore History { get; } = new(); + + /// #5489: connection_refire_minutes, read live per sweep like the shipped seam. 0 = off. + public int ConnectionRefireMinutes { get; set; } + public DarlingSelfAlertTests.FakeSettings Settings { get; } = new() { CooldownMinutes = 5 }; public ScriptedDeliverer Deliverer { get; } = new(); @@ -183,7 +199,7 @@ public async Task EachFurtherFailure_DoublesTheWait_AndNeverPassesTheCooldown(Ar } [Theory] - [MemberData(nameof(Arms))] + [MemberData(nameof(CooldownRepeatArms))] public async Task ADeliveredFire_WaitsTheFullCooldown(Arm arm) { var rig = new Rig { Deliverer = { Answer = DeliveredByWebhook() } }; @@ -202,7 +218,7 @@ public async Task ADeliveredFire_WaitsTheFullCooldown(Arm arm) } [Theory] - [MemberData(nameof(Arms))] + [MemberData(nameof(CooldownRepeatArms))] public async Task ARetryThatIsDelivered_WaitsTheFullCooldown_AndTheNextFailureStartsAtAMinute(Arm arm) { var rig = new Rig { Deliverer = { Answer = FailedByWebhook() } }; @@ -281,4 +297,120 @@ public async Task ACostRegressionMarker_GoesBackToItsPriorValue_NotToNothing_Whe await rig.SweepAsync(Arm.CostRegression, FirstDataPoint); Assert.Equal(3, rig.Fires); } + + /* ---------------- #5489: Collection Stopped is a state alert ---------------- */ + + [Fact] + public async Task CollectionStopped_StoppedAcrossManyCooldowns_WithRefireOff_AlertsExactlyOnce() + { + var rig = new Rig { Deliverer = { Answer = DeliveredByWebhook() } }; + + /* 12 cooldowns (an hour) of a standing outage, one sweep a minute. */ + for (var minute = 0; minute <= 60; minute++) + { + rig.Now = Start.AddMinutes(minute); + await rig.SweepAsync(Arm.CollectionStopped); + } + + Assert.Equal(1, rig.Fires); + } + + [Fact] + public async Task CollectionStopped_WithRefireOn_RepeatsOnTheRefireInterval_NotTheCooldown() + { + var rig = new Rig { Deliverer = { Answer = DeliveredByWebhook() }, ConnectionRefireMinutes = 30 }; + + /* 95 minutes stopped, swept every minute, cooldown 5: alerts at 0, 30, 60 and 90, not one per cooldown. */ + var firedAtMinute = new List(); + for (var minute = 0; minute <= 95; minute++) + { + rig.Now = Start.AddMinutes(minute); + var before = rig.Fires; + await rig.SweepAsync(Arm.CollectionStopped); + if (rig.Fires > before) + { + firedAtMinute.Add(minute); + } + } + + Assert.Equal(new[] { 0, 30, 60, 90 }, firedAtMinute.ToArray()); + } + + [Fact] + public async Task CollectionStopped_RecoverThenStoppedAgain_IsANewOutageWithItsOwnAlert() + { + var rig = new Rig { Deliverer = { Answer = DeliveredByWebhook() } }; + + await rig.SweepAsync(Arm.CollectionStopped); + Assert.Equal(1, rig.Fires); + + /* Recovery: one "Collection Resumed" row. */ + rig.Now = Start.AddMinutes(1); + await rig.Evaluator.ApplyCollectionStoppedAsync(ServerId, Name, stopped: false, "", Ct); + Assert.Equal("Collection Resumed", Assert.Single(rig.History.Records).MetricName); + + /* Stopped again a minute later, inside the cooldown: a new outage, a new alert. */ + rig.Now = Start.AddMinutes(2); + await rig.SweepAsync(Arm.CollectionStopped); + Assert.Equal(2, rig.Fires); + + /* And that outage is a state alert too. */ + rig.Now = Start.AddMinutes(60); + await rig.SweepAsync(Arm.CollectionStopped); + Assert.Equal(2, rig.Fires); + } + + [Fact] + public async Task CollectionStopped_AnAlertNoChannelTook_IsSentAgainAfterTheBackOff_ThenStopsOnceDelivered_WithRefireOff() + { + var rig = new Rig { Deliverer = { Answer = FailedByWebhook() } }; + + await rig.SweepAsync(Arm.CollectionStopped); + Assert.Equal(1, rig.Fires); + + rig.Now = Start.AddSeconds(59); + await rig.SweepAsync(Arm.CollectionStopped); + Assert.Equal(1, rig.Fires); + + /* The retry, a minute on, is delivered. */ + rig.Deliverer.Answer = DeliveredByWebhook(); + rig.Now = Start.AddSeconds(60); + await rig.SweepAsync(Arm.CollectionStopped); + Assert.Equal(2, rig.Fires); + + /* Delivered: nothing more for the rest of the outage. */ + for (var minute = 2; minute <= 60; minute++) + { + rig.Now = Start.AddMinutes(minute); + await rig.SweepAsync(Arm.CollectionStopped); + } + + Assert.Equal(2, rig.Fires); + } + + [Fact] + public async Task CollectionStopped_RefireIsReadLive_ChangedWhileTheOutageStands() + { + var rig = new Rig { Deliverer = { Answer = DeliveredByWebhook() } }; + + await rig.SweepAsync(Arm.CollectionStopped); + rig.Now = Start.AddMinutes(45); + await rig.SweepAsync(Arm.CollectionStopped); + Assert.Equal(1, rig.Fires); + + /* Turned on to 30 minutes at run time: the 45 minutes since the alert are already past the interval. */ + rig.ConnectionRefireMinutes = 30; + await rig.SweepAsync(Arm.CollectionStopped); + Assert.Equal(2, rig.Fires); + + rig.Now = Start.AddMinutes(60); + await rig.SweepAsync(Arm.CollectionStopped); + Assert.Equal(2, rig.Fires); + + /* Turned off again: the standing outage stops repeating. */ + rig.ConnectionRefireMinutes = 0; + rig.Now = Start.AddMinutes(240); + await rig.SweepAsync(Arm.CollectionStopped); + Assert.Equal(2, rig.Fires); + } } diff --git a/Darling/Darling.Tests/SelfAlertFleetPassTests.cs b/Darling/Darling.Tests/SelfAlertFleetPassTests.cs new file mode 100644 index 0000000000..06eb282691 --- /dev/null +++ b/Darling/Darling.Tests/SelfAlertFleetPassTests.cs @@ -0,0 +1,877 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using PerformanceMonitor.Alerting; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Analysis; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Notifications; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5479: the per-server self-alerts (Collection Stopped and its siblings) run in a fleet-level pass the sweep loop +/// launches every tick, not at the top of each server's collection body. A body the memory launch guard held off, or one +/// that never finishes, used to silence them: a fleet that stopped collecting paged nobody for 25 hours. +/// +public sealed class SelfAlertFleetPassTests +{ + /* Only ever the limit on something that never happens: no assertion reads how long a step took. */ + private static readonly TimeSpan Patience = TimeSpan.FromSeconds(30); + + /* A gate that waits for the fleet pass to reach its width: generous, because it only ever fails a pass that never gets there. */ + private static readonly TimeSpan GatePatience = TimeSpan.FromSeconds(30); + + private sealed class WorkerLogger(CapturingTestLogger inner) : ILogger + { + public IDisposable? BeginScope(TState state) where TState : notnull => null; + + public bool IsEnabled(LogLevel logLevel) => true; + + public void Log( + LogLevel logLevel, EventId eventId, TState state, Exception? exception, + Func formatter) => inner.Log(logLevel, eventId, state, exception, formatter); + } + + internal static DarlingWorker MakeWorker(CapturingTestLogger? logger = null) => + new( + logger is null ? NullLogger.Instance : new WorkerLogger(logger), + NullLoggerFactory.Instance, + new McpRuntimeState(), + new WebRuntimeState(), + new MonitoredServerRegistryState(), + new CollectorRuntimeState(), + new WebTlsCertificateState(), + new BaselineCache(), + new ReadLatencyAccumulator()); + + internal static DarlingWorker.ServerLoopState MakeServer(string name, bool connected = false) + { + var config = new MonitoredServer { Name = name, Host = name + ".invalid" }; + return new DarlingWorker.ServerLoopState + { + Config = config, + Runtime = connected + ? new ServerRuntime + { + Config = config, + ConnectionString = "Server=tcp:" + name + ",1433;Initial Catalog=master;Encrypt=True", + Target = new CollectorTargetInfo(), + StorageName = name, + ServerId = 7, + } + : null, + }; + } + + /// + /// Waits for the worker's current self-alert pass to finish, failing the test with a message if it does not within + /// . A test that counts evaluations must wait on this, not on the counter, before it starts the + /// next pass: the counter moves inside the pass, before the pass task completes. + /// + internal static async Task WaitForPassToEndAsync(DarlingWorker worker, int round) + { + var pass = worker.SelfAlertPassForTests; + Assert.NotNull(pass); + try + { + await pass.WaitAsync(Patience, TestContext.Current.CancellationToken); + } + catch (TimeoutException) + { + Assert.Fail($"the self-alert pass of round {round} did not end within {Patience}"); + } + } + + internal static async Task BecomesTrueAsync(Func condition) + { + var deadline = DateTime.UtcNow + Patience; + while (DateTime.UtcNow < deadline) + { + if (condition()) + { + return true; + } + + await Task.Delay(10, TestContext.Current.CancellationToken); + } + + return condition(); + } + + /// + /// Collection Stopped during a hold: no body is launched and none is ever called, yet the server is evaluated. On the + /// old shape the evaluation was the first thing the body did, so a held body meant no evaluation at all. + /// + [Fact] + public async Task AServerWithNoBodyLaunched_IsStillEvaluatedByTheFleetPass() + { + var worker = MakeWorker(); + var evaluated = new List(); + worker.SelfAlertServerOverride = (server, _) => + { + lock (evaluated) + { + evaluated.Add(server.Config.DisplayName); + } + + return Task.CompletedTask; + }; + var servers = new[] { MakeServer("example-sql-01"), MakeServer("example-sql-02") }; + + Assert.True(worker.TryStartSelfAlertPass(servers, TestContext.Current.CancellationToken)); + + Assert.True(await BecomesTrueAsync(() => { lock (evaluated) { return evaluated.Count == 2; } })); + Assert.All(servers, s => Assert.Null(s.InFlightSweep)); + } + + /// + /// The two halves of #5479 together: the memory launch guard is forced to hold (through its test seam, a sampler that + /// reads over the line and a collection that frees nothing), a server has had no success for longer than the + /// Collection Stopped window, and Collection Stopped fires from the fleet pass while the guard still holds and no body + /// has been launched or called. On the old shape the alert was the first statement of the body the guard held off. + /// The store read is the one piece stood in for: the evaluator's own judgement and edge apply run on a last success + /// three hours back, as the evaluator's restart pins do. + /// + [Fact] + public async Task CollectionStopped_FiresFromTheFleetPass_WhileTheLaunchGuardStillHolds_AndNoBodyRuns() + { + var harness = new DarlingSelfAlertTests.Harness(); + var start = harness.Now; + var evaluator = harness.Build(); + var lastSuccess = start.AddHours(-3); + + var worker = MakeWorker(); + var clock = TimeSpan.Zero; + worker.LaunchGuard = new LaunchMemoryGuard( + () => new LaunchMemoryReading(1500L * 1024 * 1024, 1536L * 1024 * 1024, "test metric", "test limit"), + () => { }, + () => clock, + NullLogger.Instance); + const int serverId = 424242; + worker.SelfAlertServerOverride = async (server, ct) => + { + var stopped = evaluator.JudgeCollectionStopped(serverId, lastSuccess, 10, 9, out var reason); + await evaluator.ApplyCollectionStoppedAsync(serverId, server.Config.DisplayName, stopped, reason, ct); + }; + var servers = new[] { MakeServer("example-sql-01", connected: true) }; + + /* The window has not passed after the start: silent. Then it has, and the guard has held the whole time. */ + Assert.False(worker.LaunchGuard.MayLaunch(0)); + Assert.True(worker.TryStartSelfAlertPass(servers, TestContext.Current.CancellationToken)); + Assert.True(await BecomesTrueAsync(() => worker.TryStartSelfAlertPass(servers, TestContext.Current.CancellationToken))); + Assert.Empty(harness.Deliverer.Outcomes); + + harness.Now = start.AddMinutes(30); + servers[0].NextSelfAlertSweep = DateTime.MinValue; + clock = TimeSpan.FromMinutes(30); + Assert.False(worker.LaunchGuard.MayLaunch(0)); + Assert.True(await BecomesTrueAsync(() => worker.TryStartSelfAlertPass(servers, TestContext.Current.CancellationToken))); + + Assert.True(await BecomesTrueAsync(() => harness.Deliverer.Outcomes.Any(o => o.MetricName == "Collection Stopped"))); + var fired = Assert.Single(harness.Deliverer.Outcomes, o => o.MetricName == "Collection Stopped"); + Assert.StartsWith("No successful collection in 3 hours", fired.CurrentValue, StringComparison.Ordinal); + Assert.True(worker.LaunchGuard.IsHolding, "the guard must still be holding when the alert fires"); + Assert.Null(servers[0].InFlightSweep); + } + + /// + /// A server whose body never finishes is evaluated on its cadence all the same: the pass does not look at the body. + /// + [Fact] + public async Task AServerWhoseBodyNeverFinishes_StillGetsItsEvaluationOnCadence() + { + var worker = MakeWorker(); + var evaluations = 0; + worker.SelfAlertServerOverride = async (_, _) => + { + Interlocked.Increment(ref evaluations); + + /* A slow tail: the counter is visible well before the pass ends, so a test that starts the next pass on the + counter alone gets a pass still running every time (the CI race). */ + await Task.Delay(200); + }; + var server = MakeServer("example-sql-01", connected: true); + server.InFlightSweep = new TaskCompletionSource().Task; + + for (var round = 1; round <= 3; round++) + { + server.NextSelfAlertSweep = DateTime.MinValue; + var previous = worker.TryStartSelfAlertPass([server], TestContext.Current.CancellationToken); + Assert.True(previous); + Assert.True(await BecomesTrueAsync(() => Volatile.Read(ref evaluations) == round), $"round {round} was not evaluated"); + + /* The counter moves inside the pass, before the pass task ends: wait for the pass itself, or the next + TryStartSelfAlertPass sees a pass still running and returns false (the CI race). */ + await WaitForPassToEndAsync(worker, round); + } + + /* On its cadence: a tick inside the 30 second window evaluates nothing. */ + Assert.True(worker.TryStartSelfAlertPass([server], TestContext.Current.CancellationToken)); + await WaitForPassToEndAsync(worker, 4); + Assert.Equal(3, Volatile.Read(ref evaluations)); + } + + /// + /// Two passes at once never evaluate the same server: while one pass is still running the next tick is skipped, and + /// the server is evaluated once even though its stamp is due again. + /// + [Fact] + public async Task APassStillRunning_MakesTheNextTickSkip_SoNoServerIsEvaluatedByTwoThreads() + { + var worker = MakeWorker(); + var release = new TaskCompletionSource(); + var started = new TaskCompletionSource(); + var concurrent = 0; + var maxConcurrent = 0; + var evaluations = 0; + worker.SelfAlertServerOverride = async (_, _) => + { + var now = Interlocked.Increment(ref concurrent); + InterlockedMax(ref maxConcurrent, now); + Interlocked.Increment(ref evaluations); + started.TrySetResult(); + await release.Task; + Interlocked.Decrement(ref concurrent); + }; + var server = MakeServer("example-sql-01"); + + Assert.True(worker.TryStartSelfAlertPass([server], TestContext.Current.CancellationToken)); + await started.Task.WaitAsync(Patience, TestContext.Current.CancellationToken); + + /* The stamp is due again, and the pass is still inside the first evaluation. */ + server.NextSelfAlertSweep = DateTime.MinValue; + Assert.False(worker.TryStartSelfAlertPass([server], TestContext.Current.CancellationToken)); + Assert.False(worker.TryStartSelfAlertPass([server], TestContext.Current.CancellationToken)); + Assert.Equal(1, Volatile.Read(ref evaluations)); + Assert.Equal(1, Volatile.Read(ref maxConcurrent)); + + release.SetResult(); + Assert.True(await BecomesTrueAsync(() => worker.TryStartSelfAlertPass([server], TestContext.Current.CancellationToken))); + Assert.True(await BecomesTrueAsync(() => Volatile.Read(ref evaluations) == 2)); + Assert.Equal(1, Volatile.Read(ref maxConcurrent)); + } + + /// + /// Runs 20 servers through the fleet pass at , where every evaluation counts itself in + /// flight, records the peak, and waits at a gate that opens once evaluations are in + /// flight together. The test asserts concurrency, never elapsed time: a loaded runner stretches every evaluation + /// and cannot change the peak. A pass that never reaches the width fails the test with a message after + /// , and a pass that exceeds it shows in the peak. + /// + private static async Task<(int Peak, Dictionary PerServer, string? GateFailure)> RunTwentyServersAsync( + int sweepWidth, int expectedPeak) + { + var worker = MakeWorker(); + var concurrent = 0; + var maxConcurrent = 0; + var perServer = new Dictionary(); + var gate = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + string? gateFailure = null; + worker.SelfAlertServerOverride = async (server, ct) => + { + var inFlight = Interlocked.Increment(ref concurrent); + InterlockedMax(ref maxConcurrent, inFlight); + lock (perServer) + { + perServer[server.Config.DisplayName] = perServer.GetValueOrDefault(server.Config.DisplayName) + 1; + } + + if (inFlight >= expectedPeak) + { + gate.TrySetResult(); + } + + try + { + await gate.Task.WaitAsync(GatePatience, ct); + } + catch (TimeoutException) + { + gateFailure = $"only {Volatile.Read(ref maxConcurrent)} evaluations were in flight together; the width is {expectedPeak}"; + gate.TrySetResult(); + } + + Interlocked.Decrement(ref concurrent); + }; + var servers = Enumerable.Range(1, 20).Select(i => MakeServer($"example-sql-{i:00}")).ToArray(); + + Assert.True(worker.TryStartSelfAlertPass(servers, TestContext.Current.CancellationToken, sweepWidth: sweepWidth)); + await WaitForPassToEndAsync(worker, 1); + + lock (perServer) + { + return (Volatile.Read(ref maxConcurrent), new Dictionary(perServer), gateFailure); + } + } + + /// + /// The fleet pass runs the servers at the fleet gate's width, not one at a time (#5481 round 1): twenty servers at + /// width 4 are evaluated four at a time, each exactly once. The test counts what is in flight together, not how long + /// the pass takes. At 500 servers the one-at-a-time pass overran every tick. + /// + [Fact] + public async Task TheFleetPass_EvaluatesAtTheSweepWidth_NotOneServerAtATime() + { + var (peak, perServer, gateFailure) = await RunTwentyServersAsync(sweepWidth: 4, expectedPeak: 4); + + Assert.Null(gateFailure); + Assert.Equal(20, perServer.Count); + Assert.All(perServer.Values, n => Assert.Equal(1, n)); + Assert.Equal(DarlingWorker.SelfAlertPassWidthFor(4), peak); + } + + /// + /// M1 (#5481 round 2): the pass's width comes from , the smaller of + /// the sweep width and half of , so its store reads spend at most half + /// of the reserve the web viewer's and the MCP tools' reads share. + /// + [Theory] + [InlineData(1, 1)] + [InlineData(2, 2)] + [InlineData(4, 4)] + [InlineData(9, 4)] + [InlineData(16, 4)] + [InlineData(0, 1)] + [InlineData(int.MaxValue, 4)] + public void TheSelfAlertPassWidth_IsTheSweepWidthCappedAtHalfTheReserve(int sweepWidth, int expected) + { + Assert.Equal(expected, DarlingWorker.SelfAlertPassWidthFor(sweepWidth)); + Assert.True(DarlingWorker.SelfAlertPassWidthFor(sweepWidth) <= DarlingWorker.DailyRunPoolReserve / 2); + } + + /// + /// M1: a fleet at max_concurrent_sweeps = 16 evaluates at most 4 servers at once. RED on the round-1 shape, + /// where the pass took the full sweep width and ran 16 store reads beside 16 sweep bodies on a pool of 24. + /// + [Fact] + public async Task TheFleetPass_AtSweepWidthSixteen_NeverEvaluatesMoreThanTheCappedWidthAtOnce() + { + var capped = DarlingWorker.SelfAlertPassWidthFor(16); + Assert.Equal(DarlingWorker.DailyRunPoolReserve / 2, capped); + + var (peak, perServer, gateFailure) = await RunTwentyServersAsync(sweepWidth: 16, expectedPeak: capped); + + Assert.Null(gateFailure); + Assert.Equal(20, perServer.Count); + Assert.All(perServer.Values, n => Assert.Equal(1, n)); + Assert.Equal(capped, peak); + } + + /// + /// L1 (#5481 round 2): a pass that has run 20 seconds is past one 15-second tick but inside the 30-second server + /// cadence, so the tick that skips logs nothing. RED on round 1, which warned on every skip. + /// + [Fact] + public async Task APassPastOneTickButInsideTheCadence_LogsNoWarning() + { + var logger = new CapturingTestLogger(); + var worker = MakeWorker(logger); + var now = new DateTime(2026, 1, 1, 12, 0, 0, DateTimeKind.Utc); + worker.SelfAlertPassClock = () => now; + var started = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + worker.SelfAlertServerOverride = async (_, _) => + { + started.TrySetResult(); + await release.Task; + }; + var server = MakeServer("example-sql-01"); + + Assert.True(worker.TryStartSelfAlertPass([server], TestContext.Current.CancellationToken)); + await started.Task.WaitAsync(Patience, TestContext.Current.CancellationToken); + + now += TimeSpan.FromSeconds(15); + Assert.False(worker.TryStartSelfAlertPass([server], TestContext.Current.CancellationToken)); + now += TimeSpan.FromSeconds(5); + Assert.False(worker.TryStartSelfAlertPass([server], TestContext.Current.CancellationToken)); + + Assert.Equal(0, logger.CountAtLevel(LogLevel.Warning)); + release.SetResult(); + Assert.True(await BecomesTrueAsync(() => worker.TryStartSelfAlertPass([server], TestContext.Current.CancellationToken))); + } + + /// + /// L1: a pass that has run 70 seconds has outrun the 30-second cadence, so exactly one Warning is logged for it, + /// with how long it has run, however many ticks skip. A pass after it that also overruns warns once again. + /// + [Fact] + public async Task APassThatOutrunsTheCadence_LogsExactlyOneWarning_WithHowLongItHasRun() + { + var logger = new CapturingTestLogger(); + var worker = MakeWorker(logger); + var now = new DateTime(2026, 1, 1, 12, 0, 0, DateTimeKind.Utc); + worker.SelfAlertPassClock = () => now; + var started = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + worker.SelfAlertServerOverride = async (_, _) => + { + started.TrySetResult(); + await release.Task; + }; + var server = MakeServer("example-sql-01"); + + Assert.True(worker.TryStartSelfAlertPass([server], TestContext.Current.CancellationToken)); + await started.Task.WaitAsync(Patience, TestContext.Current.CancellationToken); + + now += TimeSpan.FromSeconds(20); + Assert.False(worker.TryStartSelfAlertPass([server], TestContext.Current.CancellationToken)); + now += TimeSpan.FromSeconds(50); + Assert.False(worker.TryStartSelfAlertPass([server], TestContext.Current.CancellationToken)); + now += TimeSpan.FromSeconds(15); + Assert.False(worker.TryStartSelfAlertPass([server], TestContext.Current.CancellationToken)); + + Assert.Equal(1, logger.CountAtLevel(LogLevel.Warning)); + Assert.Contains("running for 70s", logger.Joined); + release.SetResult(); + Assert.True(await BecomesTrueAsync(() => worker.TryStartSelfAlertPass([server], TestContext.Current.CancellationToken))); + } + + /// + /// A cancellation inside one server's evaluation that is not the service shutting down (the custom-alert evaluator + /// rethrows every ) is logged and the pass goes on to the servers after it + /// (#5481 round 1). It used to end the pass, and the servers behind it starved if it repeated. + /// + [Fact] + public async Task ANonShutdownCancellationInOneServer_IsLogged_AndTheRestOfThePassGoesOn() + { + var logger = new CapturingTestLogger(); + var worker = MakeWorker(logger); + var evaluated = new List(); + worker.SelfAlertServerOverride = (server, _) => + { + lock (evaluated) + { + evaluated.Add(server.Config.DisplayName); + } + + if (server.Config.DisplayName == "example-sql-01") + { + throw new OperationCanceledException("a timeout inside one evaluator"); + } + + return Task.CompletedTask; + }; + + Assert.True(worker.TryStartSelfAlertPass( + [MakeServer("example-sql-01"), MakeServer("example-sql-02"), MakeServer("example-sql-03")], + TestContext.Current.CancellationToken, sweepWidth: 1)); + + Assert.True(await BecomesTrueAsync(() => { lock (evaluated) { return evaluated.Count == 3; } })); + Assert.Equal(["example-sql-01", "example-sql-02", "example-sql-03"], evaluated); + Assert.Equal(1, logger.CountAtLevel(LogLevel.Error)); + Assert.Contains("self-alert pass failed", logger.Joined); + } + + private static void InterlockedMax(ref int target, int value) + { + int seen; + do + { + seen = Volatile.Read(ref target); + } + while (value > seen && Interlocked.CompareExchange(ref target, value, seen) != seen); + } + + /// + /// A retired server is skipped, and one server's throw is logged without skipping the servers after it. + /// + [Fact] + public async Task ARetiredServerIsSkipped_AndOneServersThrowDoesNotSkipTheRest() + { + var worker = MakeWorker(); + var evaluated = new List(); + worker.SelfAlertServerOverride = (server, _) => + { + lock (evaluated) + { + evaluated.Add(server.Config.DisplayName); + } + + if (server.Config.DisplayName == "example-sql-01") + { + throw new InvalidOperationException("store read failed"); + } + + return Task.CompletedTask; + }; + var retired = MakeServer("example-sql-00"); + retired.Retired = true; + + Assert.True(worker.TryStartSelfAlertPass([retired, MakeServer("example-sql-01"), MakeServer("example-sql-02")], TestContext.Current.CancellationToken)); + + Assert.True(await BecomesTrueAsync(() => { lock (evaluated) { return evaluated.Count == 2; } })); + lock (evaluated) + { + Assert.Equal(["example-sql-01", "example-sql-02"], evaluated.Order().ToArray()); + } + } + + /// + /// An operator pause still evaluates nothing, as before: the launch sits after the pause's continue, before the + /// launch loop and outside the memory guard's branch, and the body no longer evaluates anything itself. + /// + [Fact] + public void TheLoopLaunchesThePass_AfterThePauseContinue_BeforeTheLaunchLoop_AndTheBodyEvaluatesNothing() + { + var source = ServerConnectBackoffTests.ReadWorkerSource(); + var loop = source.IndexOf("private async Task RunCollectionLoopAsync(", StringComparison.Ordinal); + Assert.True(loop > 0); + + var pause = source.IndexOf("if (!ShouldRunCollection(_paused))", loop, StringComparison.Ordinal); + var pauseContinue = source.IndexOf("continue;", pause, StringComparison.Ordinal); + var launch = source.IndexOf("TryStartSelfAlertPass(sweepTargets, stoppingToken, StoreConfigProvider.ClampConcurrentSweeps(config.MaxConcurrentSweeps));", loop, StringComparison.Ordinal); + var guard = source.IndexOf("var mayLaunchSweeps = _launchMemoryGuard.MayLaunch(", loop, StringComparison.Ordinal); + var launchLoop = source.IndexOf("foreach (var server in sweepTargets)", loop, StringComparison.Ordinal); + Assert.True(pause > 0 && pauseContinue > pause, "the pause gate was not found"); + Assert.True(launch > pauseContinue, "the self-alert pass must sit after the operator pause's continue"); + Assert.True(launch < guard && launch < launchLoop, "the self-alert pass must not depend on the guard or the launch loop"); + Assert.Single(System.Text.RegularExpressions.Regex.Matches(source, @"TryStartSelfAlertPass\(sweepTargets")); + + var bodyStart = source.IndexOf("private async Task ProcessServerSweepAsync(", StringComparison.Ordinal); + var nextMember = System.Text.RegularExpressions.Regex.Match( + source.Substring(bodyStart + 100), @"\r?\n (?:private|internal|public|protected)[^\r\n]*\("); + Assert.True(nextMember.Success, "the member after the body was not found"); + var body = source.Substring(bodyStart, 100 + nextMember.Index); + Assert.Contains("RunDueCollectorsAsync(server, runner", body, StringComparison.Ordinal); + Assert.DoesNotContain("EvaluateStoreAlertsAsync", body, StringComparison.Ordinal); + Assert.DoesNotContain("EvaluateServerAsync", body, StringComparison.Ordinal); + Assert.DoesNotContain("NextSelfAlertSweep", body, StringComparison.Ordinal); + Assert.DoesNotContain("NextCustomAlertSweep", body, StringComparison.Ordinal); + + /* Shutdown waits for the tracked pass like it does for the ticks. */ + Assert.Contains("inFlightSweeps.Add(_selfAlertPass);", source, StringComparison.Ordinal); + } + + /// + /// The held-slot count sits at the guard's break, and the first pass that launches again raises the skip-credit floor + /// the way the first pass after an operator pause does, before any body launches. + /// + [Fact] + public void TheLoop_CountsHeldSlotsAtTheGuardBreak_AndResumesTheFloorBeforeTheFirstLaunch() + { + var source = ServerConnectBackoffTests.ReadWorkerSource(); + var loop = source.IndexOf("private async Task RunCollectionLoopAsync(", StringComparison.Ordinal); + var held = source.IndexOf("if (!mayLaunchSweeps)", loop, StringComparison.Ordinal); + var count = source.IndexOf("CountHeldSlots(sweepTargets, DateTime.UtcNow);", held, StringComparison.Ordinal); + var brk = source.IndexOf("break;", count, StringComparison.Ordinal); + var resumeIf = source.IndexOf("if (heldSinceLastLaunch)", brk, StringComparison.Ordinal); + var resume = source.IndexOf("_skipCreditFloor.Resume(DateTime.UtcNow);", resumeIf, StringComparison.Ordinal); + var launch = source.IndexOf("server.InFlightSweep = ProcessServerSweepAsync(", loop, StringComparison.Ordinal); + + Assert.True(held > 0 && count > held && brk > count, "the held-slot count must come before the guard's break"); + Assert.True(resumeIf > brk && resume > resumeIf && resume < launch, "the floor must be raised before any body launches"); + } + + /// + /// Held slots: 10 minutes of a 1-minute collector's slots come due while the guard holds the pass off, and each is + /// counted once, within a pass of its due time, spread over the passes. The first launch after the release counts none + /// of them again, because the loop raises the floor. On the old shape the hold counted 0 and the release counted 10. + /// + [Fact] + public async Task HeldSlots_AreCountedAsTheyComeDue_AndTheFirstLaunchAfterTheReleaseCountsNoneOfThemAgain() + { + var worker = MakeWorker(); + var server = MakeServer("example-sql-01", connected: true); + var held = DateTime.UtcNow.AddMinutes(-10); + server.NextDue["wait_stats"] = held; + var stats = worker.FleetGateStatsForTest!; + + long previous = 0; + var perPass = new List(); + for (var pass = 1; pass * 15 <= 600; pass++) + { + worker.CountHeldSlots([server], held.AddSeconds(pass * 15)); + var skipped = stats.Snapshot().Skipped; + perPass.Add(skipped - previous); + previous = skipped; + } + + /* 10 minutes, 1-minute slots: the slot at the stamp is served by the first run, and the 10 after it are lost. */ + Assert.Equal(10, previous); + Assert.All(perPass, n => Assert.InRange(n, 0, 1)); + Assert.Equal(40, perPass.Count); + Assert.Equal(10, perPass.Count(n => n == 1)); + + /* A pass that counts again at the same time counts nothing. */ + Assert.Equal(0, worker.CountHeldSlots([server], held.AddMinutes(10))); + + /* The release: the loop raises the floor, then the first body runs the slot. It counts no skipped slot. */ + worker.SkipCreditFloorForTest.Resume(DateTime.UtcNow); + worker.RunOneBodyOverride = (_, _, _, _) => Task.FromResult(1); + var before = stats.Snapshot(); + await worker.RunDueCollectorsAsync(server, null!, TestContext.Current.CancellationToken); + var after = stats.Snapshot(); + + Assert.Equal(before.Skipped, after.Skipped); + Assert.True(after.Run > before.Run, "the slot at the stamp must run after the release"); + } + + /// + /// Without the floor raise the same release would count the held slots a second time: this is the burst the field saw. + /// It pins that the raise is what stops it, so a loop that stopped calling it would show here. + /// + [Fact] + public async Task WithoutTheFloorRaise_TheFirstLaunchAfterTheHoldWouldCountTheHeldSlotsAgain() + { + var worker = MakeWorker(); + var server = MakeServer("example-sql-01", connected: true); + server.NextDue["wait_stats"] = DateTime.UtcNow.AddMinutes(-10); + worker.RunOneBodyOverride = (_, _, _, _) => Task.FromResult(1); + var stats = worker.FleetGateStatsForTest!; + + await worker.RunDueCollectorsAsync(server, null!, TestContext.Current.CancellationToken); + + Assert.InRange(stats.Snapshot().Skipped, 9, 10); + } + + [Fact] + public void AServerWithABodyInFlight_ADisconnectedServer_AndARetiredServer_AreNotCountedByTheHold() + { + var worker = MakeWorker(); + var due = DateTime.UtcNow.AddMinutes(-10); + var inFlight = MakeServer("example-sql-01", connected: true); + inFlight.InFlightSweep = new TaskCompletionSource().Task; + var disconnected = MakeServer("example-sql-02"); + var retired = MakeServer("example-sql-03", connected: true); + retired.Retired = true; + foreach (var s in new[] { inFlight, disconnected, retired }) + { + s.NextDue["wait_stats"] = due; + } + + Assert.Equal(0, worker.CountHeldSlots([inFlight, disconnected, retired], DateTime.UtcNow)); + } +} + +/// +/// The live half of the fleet pass tests (#5479, #5481 round 1): the production evaluator reads a real store, so the +/// class joins the shared-store collection instead of serializing the pure tests above with it. +/// +[Collection("live-postgres")] +public sealed class SelfAlertFleetPassLiveTests +{ + private const int LivePassServerId = -770179; + + /// + /// The production evaluator through the fleet pass, with the memory guard forced to hold (#5481 round 1). Every other + /// test here stands in for the evaluation; this one reads the store. A server's last SUCCESS collection_log row is + /// three hours old, the guard holds and no body is launched, and Collection Stopped is delivered. + /// + [Fact] + public async Task LiveStore_CollectionStopped_FiresFromTheProductionEvaluator_ThroughTheFleetPass_WhileTheGuardHolds() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live fleet pass."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteLivePassRowsAsync(connection, ct); + + await using var postgres = NpgsqlDataSource.Create(connectionString!); + var bodySucceeded = false; + try + { + var utcNow = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + using (var insert = new NpgsqlCommand(@" +INSERT INTO collection_log (log_id, server_id, server_name, collector_name, collection_time, duration_ms, status, error_message, rows_collected, sql_duration_ms, duckdb_duration_ms) +VALUES ($1, $2, 'example-sql-live', 'wait_stats', $3, 0, 'SUCCESS', NULL, 0, 0, 0)", connection)) + { + insert.Parameters.AddWithValue(9_100_000L); + insert.Parameters.AddWithValue(LivePassServerId); + insert.Parameters.AddWithValue(utcNow.AddHours(-3)); + await insert.ExecuteNonQueryAsync(ct); + } + + var harness = new DarlingSelfAlertTests.Harness { Now = DateTime.UtcNow }; + + /* The alert reaches the store through the production deliverer and history writer, so the test can read it + back from the alert history as well as from what the recording deliverer received (#5481 round 2). */ + var alertSettings = new DarlingAlertSettings(new DarlingConfig()); + var historyWriter = new PgAlertHistoryStore(postgres); + harness.Deliverer.Inner = new DarlingAlertDeliverer( + alertSettings, + historyWriter, + new WebhookAlertService(alertSettings, DarlingAlertDeliverer.Branding, NullLogger.Instance, historyWriter), + NullLogger.Instance); + var worker = SelfAlertFleetPassTests.MakeWorker(); + worker.StoreForTests = postgres; + worker.SelfAlertsForTests = harness.Build(); + var clock = TimeSpan.Zero; + worker.LaunchGuard = new LaunchMemoryGuard( + () => new LaunchMemoryReading(1500L * 1024 * 1024, 1536L * 1024 * 1024, "test metric", "test limit"), + () => { }, + () => clock, + NullLogger.Instance); + var server = SelfAlertFleetPassTests.MakeServer("example-sql-live"); + server.Config.StoredServerId = LivePassServerId; + + /* The service has just started: the stale row is judged from the start, so the first pass is silent. */ + Assert.False(worker.LaunchGuard.MayLaunch(0)); + Assert.True(worker.TryStartSelfAlertPass([server], ct)); + Assert.True(await SelfAlertFleetPassTests.BecomesTrueAsync(() => worker.TryStartSelfAlertPass([server], ct))); + Assert.DoesNotContain(harness.Deliverer.Outcomes, o => o.MetricName == "Collection Stopped"); + + /* Past the window since the start, with the guard still holding and no body launched or called. */ + harness.Now = harness.Now.AddMinutes(31); + clock = TimeSpan.FromMinutes(31); + server.NextSelfAlertSweep = DateTime.MinValue; + Assert.False(worker.LaunchGuard.MayLaunch(0)); + Assert.True(await SelfAlertFleetPassTests.BecomesTrueAsync(() => worker.TryStartSelfAlertPass([server], ct))); + + Assert.True(await SelfAlertFleetPassTests.BecomesTrueAsync(() => harness.Deliverer.Outcomes.Any(o => o.MetricName == "Collection Stopped"))); + Assert.Single(harness.Deliverer.Outcomes, o => o.MetricName == "Collection Stopped"); + + /* And the history: the production writer put exactly one Collection Stopped row in config_alert_log. The first + pass's silence is asserted above through the deliverer. */ + Assert.True(await SelfAlertFleetPassTests.BecomesTrueAsync(() => CountHistoryRowsAsync(connection, ct).GetAwaiter().GetResult() >= 1)); + Assert.Equal(1, await CountHistoryRowsAsync(connection, ct)); + Assert.True(worker.LaunchGuard.IsHolding, "the guard must still be holding when the alert fires"); + Assert.Null(server.InFlightSweep); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, bodySucceeded, async (cleanup, cleanupCt) => + { + await DeleteLivePassRowsAsync(cleanup, cleanupCt); + }); + } + } + + private static async Task CountHistoryRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + using var read = new NpgsqlCommand( + "SELECT count(*) FROM config_alert_log WHERE server_name = 'example-sql-live' AND metric_name = 'Collection Stopped';", connection); + return (long)(await read.ExecuteScalarAsync(ct))!; + } + + private static async Task DeleteLivePassRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + using var cleanup = new NpgsqlCommand( + $"DELETE FROM collection_log WHERE server_id = {LivePassServerId}; " + + "DELETE FROM config_alert_log WHERE server_name = 'example-sql-live' AND metric_name = 'Collection Stopped';", connection); + await cleanup.ExecuteNonQueryAsync(ct); + } +} + +/// +/// #5479: the pure rule behind the held-slot count. +/// +public sealed class HeldSlotsTests +{ + private static readonly DateTime T0 = new(2026, 10, 7, 12, 0, 0, DateTimeKind.Utc); + private static readonly TimeSpan Minute = TimeSpan.FromMinutes(1); + + [Fact] + public void AGridCollector_LosesASlotOnceTheNextOneIsDue_TheNumberALateRunWouldStepOver() + { + Assert.Equal(0, HeldSlots.Lost(T0, T0, DateTime.MinValue, Minute, Minute)); + Assert.Equal(0, HeldSlots.Lost(T0, T0.AddSeconds(59), DateTime.MinValue, Minute, Minute)); + Assert.Equal(1, HeldSlots.Lost(T0, T0.AddSeconds(60), DateTime.MinValue, Minute, Minute)); + + for (var seconds = 0; seconds <= 6 * 3600; seconds += 15) + { + var now = T0.AddSeconds(seconds); + Assert.Equal(CollectorCadence.SkippedSlots(T0, now, Minute), HeldSlots.Lost(T0, now, DateTime.MinValue, Minute, Minute)); + } + } + + [Fact] + public void ASlotIsCountedOnce_WhateverTheNumberOfPasses_AndSpreadOverThePassesAsTheyComeDue() + { + var mark = default(HeldSlotMark); + long counted = 0; + var perPass = new List(); + for (var pass = 1; pass <= 4 * 60; pass++) + { + var n = HeldSlots.Newly(T0, T0.AddSeconds(pass * 15), DateTime.MinValue, Minute, Minute, ref mark); + counted += n; + perPass.Add(n); + } + + Assert.Equal(60, counted); + Assert.All(perPass, n => Assert.InRange(n, 0, 1)); + + /* The same instant again, and an earlier one, add nothing. */ + Assert.Equal(0, HeldSlots.Newly(T0, T0.AddHours(1), DateTime.MinValue, Minute, Minute, ref mark)); + Assert.Equal(0, HeldSlots.Newly(T0, T0.AddMinutes(30), DateTime.MinValue, Minute, Minute, ref mark)); + } + + [Fact] + public void AMovedStamp_StartsTheCountAgain_BecauseABodyRanAndCountedItsOwnSlots() + { + var mark = default(HeldSlotMark); + Assert.Equal(5, HeldSlots.Newly(T0, T0.AddMinutes(5), DateTime.MinValue, Minute, Minute, ref mark)); + + var moved = T0.AddMinutes(6); + Assert.Equal(0, HeldSlots.Newly(moved, T0.AddMinutes(6), DateTime.MinValue, Minute, Minute, ref mark)); + Assert.Equal(2, HeldSlots.Newly(moved, T0.AddMinutes(8), DateTime.MinValue, Minute, Minute, ref mark)); + } + + /// The interval changes mid-hold (a reload from 60 to 1 minute during a 2 hour hold): the due stamp stays, so + /// counting at the new interval from the stamp would invent 119 slots at once. The mark restarts at the change and only + /// slots after it count (#5481 round 1). + [Fact] + public void AnIntervalChangeMidHold_RestartsTheMark_AndInventsNoSlots() + { + var hour = TimeSpan.FromMinutes(60); + var mark = default(HeldSlotMark); + Assert.Equal(2, HeldSlots.Newly(T0, T0.AddHours(2), DateTime.MinValue, hour, hour, ref mark)); + + var change = T0.AddHours(2); + Assert.Equal(0, HeldSlots.Newly(T0, change, DateTime.MinValue, Minute, Minute, ref mark)); + Assert.Equal(0, HeldSlots.Newly(T0, change.AddSeconds(59), DateTime.MinValue, Minute, Minute, ref mark)); + + /* From the change on: the slot at the change is lost a minute later, then one a minute. */ + Assert.Equal(1, HeldSlots.Newly(T0, change.AddMinutes(1), DateTime.MinValue, Minute, Minute, ref mark)); + Assert.Equal(4, HeldSlots.Newly(T0, change.AddMinutes(5), DateTime.MinValue, Minute, Minute, ref mark)); + } + + [Fact] + public void ARaisedFloor_CountsFromTheFloor_AndNeverCountsASlotBeforeIt() + { + var mark = default(HeldSlotMark); + Assert.Equal(10, HeldSlots.Newly(T0, T0.AddMinutes(10), DateTime.MinValue, Minute, Minute, ref mark)); + + /* The loop was not running for a while (a sleep): the floor moves to now, and only slots after it count. */ + var floor = T0.AddMinutes(10); + Assert.Equal(0, HeldSlots.Newly(T0, T0.AddMinutes(10), floor, Minute, Minute, ref mark)); + Assert.Equal(3, HeldSlots.Newly(T0, T0.AddMinutes(13), floor, Minute, Minute, ref mark)); + + /* The release raises the floor to now: nothing before it counts, so the first body counts none of the hold. */ + var release = T0.AddMinutes(13).AddSeconds(30); + Assert.Equal(0, HeldSlots.Lost(T0, release, release, Minute, Minute)); + } + + [Fact] + public void ACollectorWithARunTime_LosesItsDayOnceTheGraceHasPassed_NotWhenTheNextDayIsDue() + { + var day = TimeSpan.FromDays(1); + var grace = CollectorRunTime.Grace; + + Assert.Equal(0, HeldSlots.Lost(T0, T0.Add(grace).AddSeconds(-1), DateTime.MinValue, day, grace)); + Assert.Equal(1, HeldSlots.Lost(T0, T0.Add(grace), DateTime.MinValue, day, grace)); + Assert.Equal(1, HeldSlots.Lost(T0, T0.AddDays(1), DateTime.MinValue, day, grace)); + Assert.Equal(2, HeldSlots.Lost(T0, T0.AddDays(1).Add(grace), DateTime.MinValue, day, grace)); + } +} diff --git a/Darling/Darling.Tests/SelfManagedStoreRulesStartPinTests.cs b/Darling/Darling.Tests/SelfManagedStoreRulesStartPinTests.cs new file mode 100644 index 0000000000..5547cd4d34 --- /dev/null +++ b/Darling/Darling.Tests/SelfManagedStoreRulesStartPinTests.cs @@ -0,0 +1,67 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Text.RegularExpressions; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// The service's start creates the store's password rules on a store it does not provision itself. The start is one large +/// method on a live worker, so no unit seam reaches it; this reads the worker's source, comments removed, and fails when +/// any of the three calls is gone from the provisioning chain or moves out of its branch. +/// +[Trait("Stage", "Guard")] +public sealed class SelfManagedStoreRulesStartPinTests +{ + private static string ProvisioningChain() + { + var source = Regex.Replace( + ReadRepoFileLf("Darling", "PerformanceMonitor.Darling.Service", "DarlingWorker.cs"), + @"/\*.*?\*/|//[^\n]*", "", RegexOptions.Singleline); + + var begin = source.IndexOf("DarlingManagedRoles.EnsureProvisionedAsync(", StringComparison.Ordinal); + Assert.True(begin > 0, "the managed provisioning call was not found"); + + /* The chain ends at the TimescaleDB block's first call. */ + var end = source.IndexOf("TimescaleSupport.TryEnableAsync(", begin, StringComparison.Ordinal); + Assert.True(end > begin, "the end of the provisioning chain was not found"); + return source[begin..end]; + } + + [Fact] + public void ASelfManagedStoreStart_CreatesThePasswordRules_InEveryBranchThatDoesNotProvisionTheRulesItself() + { + var chain = ProvisioningChain(); + + /* A container store whose compose provisioning did not provision it. */ + Assert.Matches( + new Regex( + @"ProvisionComposeStoreAsync\(.*?if \(verdict\.Provisioned\)\s*\{.*?\}\s*else\s*\{\s*await DarlingManagedRoles\.EnsureServerPasswordRulesAsync\(postgres, _logger, stoppingToken\);\s*\}", + RegexOptions.Singleline), + chain); + + /* Any other store that is not managed. */ + Assert.Matches( + new Regex( + @"else if \(!config\.Postgres\.Managed\)\s*\{\s*await DarlingManagedRoles\.EnsureServerPasswordRulesAsync\(postgres, _logger, stoppingToken\);\s*\}", + RegexOptions.Singleline), + chain); + + /* The managed branch, when its provisioning throws before its own rules section. */ + Assert.Matches( + new Regex( + @"EnsureProvisionedAsync\(.*?ex\.Message\);\s*await DarlingManagedRoles\.EnsureServerPasswordRulesAsync\(postgres, _logger, stoppingToken\);\s*\}\s*\}\s*else if \(!config\.Postgres\.Managed && ", + RegexOptions.Singleline), + chain); + + Assert.Equal(3, Regex.Matches(chain, @"EnsureServerPasswordRulesAsync\(").Count); + } +} diff --git a/Darling/Darling.Tests/SensitiveStatementAutoParamTests.cs b/Darling/Darling.Tests/SensitiveStatementAutoParamTests.cs new file mode 100644 index 0000000000..d3449daaff --- /dev/null +++ b/Darling/Darling.Tests/SensitiveStatementAutoParamTests.cs @@ -0,0 +1,767 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Text; +using System.Text.RegularExpressions; +using System.Xml.Linq; +using PerformanceMonitor.Common; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348: the auto-parameter scope of the XML walk. SQL Server stores a simple- or forced-parameterized statement +/// under its parameterized text (set [password] = @1) and moves the literal into the statement's +/// ParameterList, so the stored text alone names nothing. The walk puts each literal back (the probe), judges +/// that, and a named probe withholds the statement's parameter values while the statement text itself is kept. +/// +/// The judge here is a STAND-IN for the .NET evaluation that lands with the pattern work (A1): it mirrors the two +/// T-SQL alternatives that matter here (a credential-shaped name, optionally bracketed or quoted, then = and a +/// string or binary literal; and pwd= as a connection-string key), with ASCII word boundaries. The real judge +/// runs over the captured fixtures in the wiring wave. +/// +[Collection("timing")] +public class SensitiveStatementAutoParamTests +{ + private const string P = "<

>"; + private const string PText = "<<P>>"; + private static readonly XNamespace Ns = "http://schemas.microsoft.com/sqlserver/2004/07/showplan"; + + // Stand-in for the .NET evaluation of T4 and T9 (see the class comment). + private static readonly Regex T4 = new(@"(password|pwd|secret)[\]""]?\s*=\s*(n?'|0x)", + RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + private static readonly Regex T9 = new(@"(? T4.IsMatch(s) || T9.IsMatch(s); + + private readonly ITestOutputHelper _output; + + public SensitiveStatementAutoParamTests(ITestOutputHelper output) => _output = output; + + private static string Fixture(string name) => + File.ReadAllText(Path.Combine(AppContext.BaseDirectory, "Fixtures", "StatementScrub", name)); + + private static string? Run(string xml, Func? judge = null, int max = int.MaxValue) => + SensitiveStatements.XmlCore(xml, judge ?? StandIn, () => false, P, null, max); + + private static XElement Col(string column, string? compiled, string? runtime = null) + { + var e = new XElement(Ns + "ColumnReference", new XAttribute("Column", column)); + if (compiled is not null) + e.Add(new XAttribute("ParameterCompiledValue", compiled)); + if (runtime is not null) + e.Add(new XAttribute("ParameterRuntimeValue", runtime)); + return e; + } + + private static XElement Stmt(string element, string text, params XElement[] columns) + { + var plan = new XElement(Ns + "QueryPlan"); + if (columns.Length > 0) + plan.Add(new XElement(Ns + "ParameterList", columns)); + return new XElement(Ns + element, new XAttribute("StatementText", text), new XAttribute("StatementId", "1"), plan); + } + + private static string Plan(params XElement[] statements) => + new XElement(Ns + "ShowPlanXML", + new XElement(Ns + "BatchSequence", new XElement(Ns + "Batch", new XElement(Ns + "Statements", statements)))) + .ToString(SaveOptions.DisableFormatting); + + private static List Attrs(string xml, string name) => + XDocument.Parse(xml).Descendants().Select(e => (string?)e.Attribute(name)).Where(v => v is not null).Select(v => v!).ToList(); + + private static string OnlyStatementText(string xml) => Attrs(xml, "StatementText").Single(); + + // ── the RED on L2's code (no probe): the stored form names nothing, so nothing was withheld ── + + [Theory] + [InlineData("(@1 nvarchar(4000),@2 tinyint)UPDATE [dbo].[t] set [password] = @1 WHERE [id]=@2")] + [InlineData("(@1 nvarchar(4000),@2 tinyint)UPDATE [dbo].[t] set [pwd] = @1 WHERE [id]=@2")] + [InlineData("(@1 nvarchar(4000),@2 tinyint)UPDATE [dbo].[t] set [secret] = @1 WHERE [id]=@2")] + public void SetCredentialColumnToToken_WithholdsTheCompiledValues_AndKeepsTheStatementText(string text) + { + string xml = Plan(Stmt("StmtSimple", text, Col("@1", "N'S3cret-value'"), Col("@2", "(7)"))); + + string? result = Run(xml); + + Assert.NotNull(result); + Assert.NotSame(xml, result); + Assert.Equal(new[] { P, P }, Attrs(result!, "ParameterCompiledValue")); + Assert.Equal(text, OnlyStatementText(result!)); + Assert.DoesNotContain("S3cret-value", result); + } + + [Fact] + public void WhereClauseOfTwoTokens_WithholdsBothValues_AndKeepsTheStatementText() + { + const string text = "(@1 nvarchar(4000),@2 nvarchar(4000))SELECT [id] FROM [dbo].[t] WHERE [username]=@1 AND [password]=@2"; + string xml = Plan(Stmt("StmtSimple", text, Col("@2", "N'S3cret-value'"), Col("@1", "N'app'"))); + + string? result = Run(xml); + + Assert.NotNull(result); + Assert.Equal(new[] { P, P }, Attrs(result!, "ParameterCompiledValue")); + Assert.Equal(text, OnlyStatementText(result!)); + } + + [Theory] + [InlineData("SELECT [id] FROM [dbo].[t] WHERE [secret_id]=@1")] + [InlineData("SELECT [id] FROM [dbo].[t] WHERE [PasswordHash]=@1")] + [InlineData("SELECT @@ROWCOUNT WHERE [id]=@@SPID")] + public void NameOnlyLooksSimilar_KeepsItsValues_AndReturnsTheSameInstance(string text) + { + string xml = Plan(Stmt("StmtSimple", text, Col("@1", "N'plain-value'"))); + + Assert.Same(xml, Run(xml)); + } + + [Fact] + public void TokensAreMatchedByOrdinal_AtTenNeverTakesTheValueOfOne() + { + // @10 is a different parameter from @1. A prefix replace would read [pwd]=N'a'0 and withhold; the right + // probe is [pwd]=(5), a number, which names nothing. + string xml = Plan(Stmt("StmtSimple", "SELECT [id] FROM [dbo].[t] WHERE [pwd]=@10", Col("@1", "N'a'"), Col("@10", "(5)"))); + + Assert.Same(xml, Run(xml)); + } + + [Fact] + public void TokenFollowedByWordCharacter_OrPrecededByOne_IsNotAToken() + { + string xml = Plan(Stmt("StmtSimple", "SELECT [id] FROM [dbo].[t] WHERE [pwd]=@1abc OR [pwd]=a@1", + Col("@1", "N'S3cret-value'"))); + + Assert.Same(xml, Run(xml)); + } + + [Fact] + public void ForcedParameterization_TheZeroTokenIsCovered() + { + string xml = Plan(Stmt("StmtSimple", "(@0 varchar(8000))SELECT [id] FROM [dbo].[t] WHERE [password] = @0", + Col("@0", "'S3cret-value'"))); + + string? result = Run(xml); + + Assert.NotNull(result); + Assert.Equal(new[] { P }, Attrs(result!, "ParameterCompiledValue")); + } + + [Fact] + public void ApplicationNamedParameter_KeepsItsValue() + { + // sp_executesql: the statement text keeps the application's own parameter name, which is not a token. + string xml = Plan(Stmt("StmtSimple", "(@pwd nvarchar(50))UPDATE [dbo].[t] SET [password] = @pwd", + Col("@pwd", "N'S3cret-value'"))); + + Assert.Same(xml, Run(xml)); + } + + [Fact] + public void ValueRulesFireOnThePutBackLiteral_AndTheQuestionMarkFormWouldNotName() + { + const string text = "(@1 nvarchar(4000))INSERT [dbo].[cfg] VALUES (@1)"; + string withValue = Plan(Stmt("StmtSimple", text, Col("@1", "N'Server=h;PWD=x'"))); + string noValue = Plan(Stmt("StmtSimple", text)); + + string? result = Run(withValue); + + Assert.NotNull(result); + Assert.Equal(new[] { P }, Attrs(result!, "ParameterCompiledValue")); + Assert.Equal(text, OnlyStatementText(result!)); + // with no value to put back the probe reads VALUES (N'?'), which names nothing + Assert.Same(noValue, Run(noValue)); + } + + [Fact] + public void NumericValue_NamesNothing() + { + string xml = Plan(Stmt("StmtSimple", "(@1 int)INSERT [dbo].[cfg] VALUES (@1)", Col("@1", "(7)"))); + + Assert.Same(xml, Run(xml)); + } + + [Fact] + public void RuntimeValue_IsUsedWhenThereIsNoCompiledValue() + { + string xml = Plan(Stmt("StmtSimple", "(@1 nvarchar(4000))UPDATE [dbo].[t] set [password] = @1", + Col("@1", null, "N'S3cret-value'"))); + + string? result = Run(xml); + + Assert.NotNull(result); + Assert.Equal(new[] { P }, Attrs(result!, "ParameterRuntimeValue")); + } + + [Fact] + public void TokenWithNoParameterListEntry_FallsBackToTheQuestionMark_AndACredentialNameStillNames() + { + const string text = "(@1 nvarchar(4000))UPDATE [dbo].[t] set [password] = @1"; + var stmt = Stmt("StmtSimple", text); + stmt.Element(Ns + "QueryPlan")!.Add(new XElement(Ns + "ScalarOperator", new XAttribute("ScalarString", "[plain-text]"))); + string xml = Plan(stmt); + + string? result = Run(xml); + + // [password] = N'?' is still named, so the scope opens (the ScalarString inside is withheld); the text is kept + Assert.NotNull(result); + Assert.Equal(P, Attrs(result!, "ScalarString").Single()); + Assert.Equal(text, OnlyStatementText(result!)); + } + + [Fact] + public void KeptStatementTextOfAProbeScope_SurvivesWhenItHoldsAQuote() + { + // P13: inside a scope any value holding a quote is withheld (L-J), except the probe scope's own text. + const string text = "(@1 nvarchar(4000))UPDATE [dbo].[t] set [password] = @1, [note] = 'x'"; + string xml = Plan(Stmt("StmtSimple", text, Col("@1", "N'S3cret-value'"))); + + string? result = Run(xml); + + Assert.NotNull(result); + Assert.Equal(text, OnlyStatementText(result!)); + Assert.Equal(new[] { P }, Attrs(result!, "ParameterCompiledValue")); + } + + [Fact] + public void NestedStatement_EachParameterListBelongsToItsOwnStatement() + { + // Outer StmtCond: [pwd]=@1 with a number (names nothing). Inner StmtSimple: [pwd]=@1 with a literal (named). + // A ParameterList read for the wrong statement would withhold the outer one (N'?' fallback) or keep the inner. + var inner = Stmt("StmtSimple", "SELECT [id] FROM [dbo].[t] WHERE [pwd]=@1", Col("@1", "N'S3cret-inner'")); + var outer = new XElement(Ns + "StmtCond", + new XAttribute("StatementText", "IF EXISTS (SELECT 1 FROM [dbo].[t] WHERE [pwd]=@1)"), + new XElement(Ns + "Condition", new XElement(Ns + "QueryPlan", + new XElement(Ns + "ParameterList", Col("@1", "(5)")))), + new XElement(Ns + "Then", new XElement(Ns + "Statements", inner))); + string xml = Plan(outer); + + string? result = Run(xml); + + Assert.NotNull(result); + var values = Attrs(result!, "ParameterCompiledValue"); + Assert.Equal(new[] { "(5)", P }, values); + Assert.DoesNotContain("S3cret-inner", result); + Assert.Equal(2, Attrs(result!, "StatementText").Count); + } + + [Fact] + public void SiblingStatements_AreJudgedOnTheirOwn() + { + var named = Stmt("StmtSimple", "(@1 nvarchar(4000))UPDATE [dbo].[t] set [password] = @1", Col("@1", "N'S3cret-value'")); + var plain = Stmt("StmtSimple", "(@1 nvarchar(4000))SELECT [id] FROM [dbo].[t] WHERE [name]=@1", Col("@1", "N'plain-value'")); + string xml = Plan(named, plain); + + string? result = Run(xml); + + Assert.NotNull(result); + Assert.Equal(new[] { P, "N'plain-value'" }, Attrs(result!, "ParameterCompiledValue")); + } + + [Fact] + public void ParameterListPastTheCut_StillWithholdsTheStatementByName() + { + var plan = new XElement(Ns + "QueryPlan", + new XElement(Ns + "RelOp", new XAttribute("NodeId", "0"), + new XElement(Ns + "ScalarOperator", new XAttribute("ScalarString", "[plain-text]"))), + new XElement(Ns + "RelOp", new XAttribute("NodeId", "1"), new XAttribute("Filler", new string('x', 4000))), + new XElement(Ns + "ParameterList", Col("@1", "N'S3cret-value'"))); + var stmt = new XElement(Ns + "StmtSimple", + new XAttribute("StatementText", "(@1 nvarchar(4000))UPDATE [dbo].[t] set [password] = @1"), plan); + string xml = Plan(stmt); + Assert.True(xml.IndexOf("", StringComparison.Ordinal) > 1500); + + // 1,000 characters in: the compiled value lies past the cut, so the probe reads [password] = N'?' (named), + // and the scope withholds the ScalarString that sits inside the cut. + string? result = Run(xml, max: 1000); + + Assert.NotNull(result); + Assert.DoesNotContain("plain-text", result); + Assert.Contains(PText, result); + } + + // ── L2c review fixes (H1, M1, L3) ── + + private static bool CanaryOrStandIn(string s) => StandIn(s) || s.Contains("CANARY", StringComparison.Ordinal); + + [Fact] + public void WithAMaxOutput_ValuesPastPassOnesStop_AreWithheldEvenWhenTheOutputIsShorter_LongNamedStatementFirst() + { + // statement 1 is named and long: its text becomes the short placeholder, so pass 2 reaches input that pass 1 + // (which stops by INPUT offset) never read + string xml = Plan( + Stmt("StmtSimple", "CANARY " + new string('x', 3000)), + Stmt("StmtSimple", "(@1 nvarchar(4000),@2 tinyint)UPDATE [dbo].[u] set [password] = @1 WHERE [id]=@2", + Col("@1", "N'S3cret-drift'"), Col("@2", "(7)"))); + const int max = 2000; + + string? cut = Run(xml, CanaryOrStandIn, max); + string? whole = Run(xml, CanaryOrStandIn); + + Assert.NotNull(cut); + Assert.NotNull(whole); + Assert.DoesNotContain("S3cret-drift", whole); + Assert.DoesNotContain("S3cret-drift", cut![..Math.Min(max, cut.Length)]); + } + + [Fact] + public void WithAMaxOutput_ValuesPastPassOnesStop_AreWithheldEvenWhenTheOutputIsShorter_LargeInList() + { + // ' is five characters in the input and one in the output, so pass 2 outruns pass 1 on a long IN-list + var sb = new StringBuilder(); + sb.Append(""); + sb.Append(""); + for (int i = 0; i < 119; i++) + sb.Append(""); + sb.Append(""); + sb.Append(""); + sb.Append(""); + string xml = sb.ToString(); + int last = xml.IndexOf("PWD=drift2", StringComparison.Ordinal); + int max = last - 200; + Assert.True(max > 1000, "fixture shape"); + + string? cut = Run(xml, CanaryOrStandIn, max); + + Assert.NotNull(cut); + Assert.DoesNotContain("drift2", cut![..Math.Min(max, cut.Length)]); + } + + [Fact] + public void RuntimeValueThatDiffersFromTheCompiledValue_IsJudgedToo() + { + // an actual plan: the compiled value is whichever run compiled the cached plan, the runtime value is this run's + const string text = "(@1 nvarchar(50))INSERT [dbo].[cfg] VALUES (@1)"; + string xml = Plan(Stmt("StmtSimple", text, Col("@1", "N'benign'", "N'Server=h;PWD=rt3'"))); + + string? result = Run(xml); + + Assert.NotNull(result); + Assert.NotSame(xml, result); + Assert.DoesNotContain("rt3", result); + Assert.Equal(new[] { P }, Attrs(result!, "ParameterCompiledValue")); + Assert.Equal(new[] { P }, Attrs(result!, "ParameterRuntimeValue")); + Assert.Equal(text, OnlyStatementText(result!)); + + // both benign (equal, then different): nothing is named and the same instance comes back + string same = Plan(Stmt("StmtSimple", text, Col("@1", "N'benign'", "N'benign'"))); + string differ = Plan(Stmt("StmtSimple", text, Col("@1", "N'benign'", "N'other'"))); + Assert.Same(same, Run(same)); + Assert.Same(differ, Run(differ)); + } + + [Fact] + public void AValueWhoseTokenIsNotInTheStatementText_IsJudgedOnItsOwn() + { + // the stored text is shorter than the statement, so no token puts the value back into the probe + const string text = "INSERT [dbo].[cfg] VALUES ("; + string xml = Plan(Stmt("StmtSimple", text, Col("@1", "N'Server=h;PWD=s7'"))); + + string? result = Run(xml); + + Assert.NotNull(result); + Assert.DoesNotContain("s7", result); + Assert.Equal(new[] { P }, Attrs(result!, "ParameterCompiledValue")); + Assert.Equal(text, OnlyStatementText(result!)); + + // a plain value, and an application-named parameter, name nothing + string plain = Plan(Stmt("StmtSimple", text, Col("@1", "N'plain'"))); + string appNamed = Plan(Stmt("StmtSimple", text, Col("@pwd", "N'Server=h;PWD=s7'"))); + Assert.Same(plain, Run(plain)); + Assert.Same(appNamed, Run(appNamed)); + } + + + [Fact] + public void ElementFormParameterizedText_NamedStatementIsWithheld_EvenAfterAnEarlierHit() + { + // Statement 1 is named by its own text (the first hit); statement 2 only by an element-form ParameterizedText, + // which pass 2 reads after the start tag, so pass 1 has to record it. + var first = Stmt("StmtSimple", "UPDATE [dbo].[t] SET password = N'S3cret-one'", Col("@1", "N'a'")); + var second = Stmt("StmtSimple", "SELECT 1", Col("@1", "N'keep-or-withhold'")); + second.Element(Ns + "QueryPlan")!.AddFirst(new XElement(Ns + "ParameterizedText", + "(@1 nvarchar(50))UPDATE [dbo].[t] SET [password] = N'S3cret-two'")); + string xml = Plan(first, second); + + string? result = Run(xml); + + Assert.NotNull(result); + Assert.Equal(new[] { P, P }, Attrs(result!, "ParameterCompiledValue")); + Assert.DoesNotContain("S3cret-two", result); + } + + // ── the captured fixtures: what SQL Server stored on a real instance ── + + [Theory] + [InlineData("autoparam_update_prepared.xml")] + [InlineData("autoparam_select_prepared.xml")] + public void CapturedPreparedPlan_WithholdsTheLiteral_AndKeepsTheStatementText(string file) + { + string xml = Fixture(file); + string original = OnlyStatementText(xml); + + string? result = Run(xml); + + Assert.NotNull(result); + Assert.NotSame(xml, result); + Assert.DoesNotContain("S3cret-fixture", result); + Assert.Equal(original, OnlyStatementText(result!)); + Assert.All(Attrs(result!, "ParameterCompiledValue"), v => Assert.Equal(P, v)); + Assert.NotEmpty(Attrs(result!, "ParameterCompiledValue")); + } + + [Theory] + [InlineData("autoparam_update_adhoc_shell.xml")] + [InlineData("autoparam_select_adhoc_shell.xml")] + public void CapturedAdhocShell_WithholdsItsLiteralStatementText_AndItsParameterizedTextAttribute(string file) + { + string xml = Fixture(file); + Assert.Contains("S3cret-fixture", OnlyStatementText(xml)); + Assert.Single(Attrs(xml, "ParameterizedText")); + + string? result = Run(xml); + + Assert.NotNull(result); + Assert.Equal(P, OnlyStatementText(result!)); + Assert.Equal(P, Attrs(result!, "ParameterizedText").Single()); + Assert.DoesNotContain("S3cret-fixture", result); + } + + // ── a plan that does not parse: withheld whole when the probe could have mattered ── + + [Fact] + public void CutPreparedPlan_InsideItsParameterList_IsWithheldWhole() + { + string xml = Fixture("autoparam_update_prepared.xml"); + int at = xml.IndexOf("ParameterCompiledValue=\"N'S3", StringComparison.Ordinal) + 30; + string cut = xml.Substring(0, at); + + // the judge alone names nothing in the cut text (the stored text is [password] = @1) + Assert.False(StandIn(System.Net.WebUtility.HtmlDecode(cut))); + Assert.Equal(P, Run(cut)); + } + + [Fact] + public void CutPlan_WithNoToken_TakesTheWholeTextPath() + { + const string plain = "", StringComparison.Ordinal) + 8); + Assert.Contains("@1", cut); + Assert.DoesNotContain("ParameterCompiledValue", cut); + Assert.DoesNotContain("ParameterRuntimeValue", cut); + + // a column the judge does not name: with every token put back as N'?' it still names nothing + string neutral = cut.Replace("[password]", "[note]", StringComparison.Ordinal); + Assert.Contains("[note] = @1", neutral); + Assert.Same(neutral, Run(neutral)); + } + + [Fact] + public void CutPlan_WithATokenAndNoParameterValues_IsJudgedWithTheTokensPutBackAsAPlaceholderValue() + { + // The stored text names nothing on its own (set [password] = @1); the parsed twin judges + // set [password] = N'?', names it, and withholds the statement's residual literals. The cut text has no + // parameter value to put back, but the same judgment applies. + string xml = Fixture("autoparam_update_prepared.xml"); + string cut = xml.Substring(0, xml.IndexOf("", StringComparison.Ordinal) + 8); + Assert.Contains("[password] = @1", cut); + Assert.False(StandIn(System.Net.WebUtility.HtmlDecode(cut))); + Assert.Equal(P, Run(cut)); + + const string issue = ""; + + private static string BigPlan(int targetChars, string firstStatementText, string? firstCompiled, bool tokensInRest) + { + var sb = new StringBuilder(targetChars + 4096); + sb.Append(""); + int n = 0; + while (sb.Length < targetChars) + { + string text = n == 0 ? firstStatementText + : tokensInRest ? "(@1 nvarchar(50))SELECT [c] FROM [dbo].[t] WHERE [name]=@1" : "SELECT [c] FROM [dbo].[t] WHERE [name] = 1"; + string? compiled = n == 0 ? firstCompiled : tokensInRest ? "N'plain-value'" : null; + sb.Append(""); + for (int i = 0; i < 20; i++) + sb.Append(FillerRelOp); + if (compiled is not null) + sb.Append(""); + sb.Append("\n"); + } + sb.Append(""); + return sb.ToString(); + } + + private (long Ms, long Bytes, string? Result) Measure(string label, string xml, int max) + { + // the judge matches the earlier walk measurements (a plain substring test), so the numbers compare + Func substring = s => s.Contains("CANARY", StringComparison.OrdinalIgnoreCase); + GC.Collect(); + long before = GC.GetAllocatedBytesForCurrentThread(); + var sw = Stopwatch.StartNew(); + string? result = SensitiveStatements.XmlCore(xml, substring, () => false, P, null, max); + sw.Stop(); + long bytes = GC.GetAllocatedBytesForCurrentThread() - before; + string line = string.Create(CultureInfo.InvariantCulture, + $"SSF-L2B-MEASURE {label}: {xml.Length / 1_000_000.0:F1} M chars, {sw.ElapsedMilliseconds} ms, {bytes / 1_048_576.0:F1} MB allocated, {xml.Length / 1_000_000.0 / Math.Max(sw.Elapsed.TotalSeconds, 0.0001):F0} M chars/s"); + _output.WriteLine(line); + TestContext.Current.SendDiagnosticMessage(line); + return (sw.ElapsedMilliseconds, bytes, result); + } + + /// + /// The fastest of runs of on the same document (#5561). A busy machine only + /// ever adds time, so the minimum is the run it touched least: one stalled run (8 s on a shared runner for a walk that + /// takes 0.6 s) no longer fails a pin whose purpose is the walk's cost, while a walk that is really slow is slow in every + /// run and still fails. The bytes are the smallest too, and the result is the last run's. + /// + private (long Ms, long Bytes, string? Result) FastestOf(int runs, string label, string xml, int max) + { + long bestMs = long.MaxValue; + long bestBytes = long.MaxValue; + string? result = null; + for (int run = 0; run < runs; run++) + { + var m = Measure(label, xml, max); + bestMs = Math.Min(bestMs, m.Ms); + bestBytes = Math.Min(bestBytes, m.Bytes); + result = m.Result; + } + + return (bestMs, bestBytes, result); + } + + [Fact] + public void Measure_PassOne_27MbPlan_NoHit_AndWithAThousandAutoParameterizedStatements() + { + string plain = BigPlan(27_000_000, "SELECT [c] FROM [dbo].[t] WHERE [name] = 1", null, tokensInRest: false); + // fastest of three (#5561): the ceiling below is the walk's cost, not the worst stall a shared runner dealt it + var noToken = FastestOf(3, "27 MB no hit, no token", plain, int.MaxValue); + Assert.Same(plain, noToken.Result); + + // the same shape with a token and a ParameterList in every statement + string tokens = BigPlan(27_000_000, "(@1 nvarchar(50))SELECT [c] FROM [dbo].[t] WHERE [name]=@1", "N'plain-value'", tokensInRest: true); + var withTokens = FastestOf(3, "27 MB no hit, token in every statement", tokens, int.MaxValue); + Assert.Same(tokens, withTokens.Result); + + // probe cost per 1,000 statements: a plan of just 1,000 statements, with and without the tokens + string thousandPlain = BigPlan(1000 * (FillerRelOp.Length * 20 + 130), "SELECT 1", null, tokensInRest: false); + string thousandTokens = BigPlan(1000 * (FillerRelOp.Length * 20 + 230), "(@1 nvarchar(50))SELECT 1 WHERE [x]=@1", "N'plain-value'", tokensInRest: true); + int statements = Regex.Matches(thousandTokens, ""; + private const string ShowplanClose = ""; + + // needs the real value: neither the stored text (a token) nor the placeholder value (N'?') names it + private static bool NeedsTheRealValue(string s) => + s.Contains("password", StringComparison.Ordinal) && s.Contains("S3cretSt", StringComparison.Ordinal); + + [Fact] + public void AnElementThatIsNotAStmtButCarriesAStatementText_DoesNotTakeTheParameterValuesBelowIt() + { + string xml = ShowplanOpen + + "" + + "" + + "" + ShowplanClose; + + string? result = Run(xml, NeedsTheRealValue); + + Assert.NotNull(result); + Assert.DoesNotContain("S3cretSt", result); + } + + [Fact] + public void AStatementElementWithNoStatementText_HasItsAutoParameterValuesJudged() + { + string xml = ShowplanOpen + + "" + + "" + + ShowplanClose; + + string? result = Run(xml); + + Assert.NotNull(result); + Assert.DoesNotContain("PWD=nt", result); + } + + [Theory] + [InlineData("@")] + [InlineData("@")] + [InlineData("@")] + [InlineData("@")] + public void AZeroPaddedCharacterReferenceForTheAtSign_StillBuildsTheProbe(string at) + { + string xml = ShowplanOpen + + "" + + "" + + "" + ShowplanClose; + + string? result = Run(xml, NeedsTheRealValue); + + Assert.NotNull(result); + Assert.DoesNotContain("S3cretSt", result); + } + + [Fact] + public void ANestedPlanInAnEdcShowplanXmlAttribute_IsJudgedAsAPlan() + { + const string nested = "<ShowPlanXML><StmtSimple StatementText="ALTER LOGIN x WITH PASSWORD = &apos;edc1&apos;"/></ShowPlanXML>"; + string outside = ""; + // the same nested plan inside a withheld statement + const string nestedWhere = "<ShowPlanXML><StmtSimple StatementText="SELECT 1 WHERE c = &apos;edc2&apos;"/></ShowPlanXML>"; + string inside = ShowplanOpen + "" + + "" + ShowplanClose; + + string? a = Run(outside, CanaryOrStandIn); + string? b = Run(inside, CanaryOrStandIn); + + Assert.NotNull(a); + Assert.NotNull(b); + Assert.DoesNotContain("edc1", a); + Assert.DoesNotContain("edc2", b); + } + + // ── L2g: a plan nested in an EdcShowplanXml attribute is judged one level deep (#5320) ── + + /// escaped times as attribute text, written directly + /// (an entity escaped k times is &, amp; k-1 times, then its name). + private static void AppendEscaped(StringBuilder sb, string s, int times) + { + foreach (char c in s) + { + string? name = c switch { '<' => "lt;", '>' => "gt;", '"' => "quot;", '&' => "amp;", _ => null }; + if (name is null || times == 0) + { + sb.Append(c); + continue; + } + sb.Append('&'); + for (int i = 1; i < times; i++) + sb.Append("amp;"); + sb.Append(name); + } + } + + /// A document holding plans, each in the EdcShowplanXml attribute of + /// the one above, with at the bottom. + private static string NestedPlans(int levels, string innermost) + { + const string open = ""; + var sb = new StringBuilder(); + for (int j = 0; j < levels; j++) + AppendEscaped(sb, open, j); + AppendEscaped(sb, innermost, levels); + for (int j = levels - 1; j >= 0; j--) + AppendEscaped(sb, close, j); + return sb.ToString(); + } + + /// The EdcShowplanXml value of the plan held in the top document's own EdcShowplanXml: + /// the level-2 attribute. + private static string LevelTwoAttribute(string result) + { + string level1 = (string)XElement.Parse(result).Attribute("EdcShowplanXml")!; + return (string)XElement.Parse(level1).Attribute("EdcShowplanXml")!; + } + + [Fact] + public void ANestedPlanTwoLevelsDeep_IsWithheldEvenWhenItHoldsOnlyOrdinaryText() + { + string xml = NestedPlans(2, "lvl2plain"); + + string? result = Run(xml, CanaryOrStandIn); + + Assert.NotNull(result); + Assert.DoesNotContain("lvl2plain", result); + Assert.Equal(P, LevelTwoAttribute(result)); + } + + [Fact] + public void ANestedPlanOneLevelDeep_HoldingOnlyOrdinaryText_IsKeptAsTheSameInstance() + { + string xml = NestedPlans(1, "lvl1plain"); + + string? result = Run(xml, CanaryOrStandIn); + + Assert.Same(xml, result); + } + + [Fact] + public void ADeeplyNestedPlan_ReturnsWithoutOverflowingTheStack_AndIsWithheldPastTheFirstLevel() + { + string xml = NestedPlans(2000, "deepplain"); + + // a cheap judge: this pins the walk, not the pattern. The document is 32 MB (escaping makes the size grow with + // the square of the depth) and each scan of it costs about half a second, so the bound is a hang guard, not a speed pin. + var timer = Stopwatch.StartNew(); + string? result = Run(xml, v => v.Contains("CANARY", StringComparison.Ordinal)); + timer.Stop(); + + Assert.NotNull(result); + Assert.DoesNotContain("deepplain", result); + Assert.Equal(P, LevelTwoAttribute(result)); + Assert.True(timer.Elapsed < TimeSpan.FromSeconds(10), $"took {timer.Elapsed.TotalMilliseconds:F0} ms; size {xml.Length}"); + } +} diff --git a/Darling/Darling.Tests/SensitiveStatementCorpus.cs b/Darling/Darling.Tests/SensitiveStatementCorpus.cs new file mode 100644 index 0000000000..8b36ce1d8d --- /dev/null +++ b/Darling/Darling.Tests/SensitiveStatementCorpus.cs @@ -0,0 +1,165 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System.Linq; + +namespace Darling.Tests; + +/// +/// The statement corpus shared by the .NET evaluation tests and the PostgreSQL parity test (#4348): what the +/// filter names, what it leaves alone, and the two strings only the .NET engine bounds with a time limit. +/// +internal static class SensitiveStatementCorpus +{ + /// Statements the filter must name. Each is also judged lower-cased and with a leading block + /// comment (). + public static readonly string[] Named = + { + "CREATE LOGIN [app] WITH PASSWORD = N'S3cret'", + "ALTER LOGIN [app] WITH PASSWORD = 0x0200AB HASHED", + "ALTER LOGIN [app] DISABLE", + "CREATE/* x */LOGIN [a] WITH PASSWORD = 'p'", + "CREATE\r\nLOGIN [a] FROM WINDOWS", + "EXEC sp_addlogin 'app', 'S3cret'", + "EXEC master..sp_password NULL, 'S3cret', 'app'", + "EXEC sp_addlinkedsrvlogin 'SRV', 'false', NULL, 'u', 'S3cret'", + "EXEC sp_setapprole 'r', 'S3cret'", + "OPEN SYMMETRIC KEY k DECRYPTION BY PASSWORD = 'S3cret'", + "OPEN MASTER KEY DECRYPTION BY PASSWORD = N'S3cret'", + "CREATE MASTER KEY ENCRYPTION BY PASSWORD = N'S3cret'", + "CREATE CREDENTIAL c WITH IDENTITY = 'u', SECRET = 'S3cret'", + "CREATE DATABASE SCOPED CREDENTIAL c WITH IDENTITY = 'SHARED ACCESS SIGNATURE', SECRET = 'sv=1&sig=x'", + "BACKUP CERTIFICATE c TO FILE = 'f' WITH PRIVATE KEY (FILE = 'k', ENCRYPTION BY PASSWORD = 'S3cret')", + "RESTORE DATABASE d FROM DISK = 'f' WITH MEDIAPASSWORD = 'S3cret'", + "CREATE SYMMETRIC KEY k WITH KEY_SOURCE = 'phrase', ALGORITHM = AES_256 ENCRYPTION BY CERTIFICATE c", + "SELECT * FROM OPENROWSET('MSOLEDBSQL', 'Server=h;UID=u;PWD=S3cret;', 'SELECT 1')", + "SELECT * FROM OPENDATASOURCE('MSOLEDBSQL', 'Data Source=h;User ID=u;Pwd=S3cret').db.dbo.t", + "EXEC sp_addlinkedserver @server = 'S', @provider = 'MSOLEDBSQL', @provstr = 'UID=u;PWD=S3cret'", + "EXEC sp_addpushsubscription_agent @publication = N'p', @job_password = N'S3cret'", + "ALTER SERVICE MASTER KEY WITH OLD_ACCOUNT = 'a', OLD_PASSWORD = 'S3cret'", + "ALTER APPLICATION ROLE r WITH PASSWORD = N'S3cret'", + "SELECT DECRYPTBYPASSPHRASE('phrase', @c)", + "exec sp_executesql N'UPDATE dbo.t SET c = @pwd', N'@pwd nvarchar(50)', @pwd = N'S3cret'", + "UPDATE dbo.Creds SET [password] = N'S3cret'", + "UPDATE t SET \"password\" = 'S3cret'", + "UPDATE dbo.Users SET pwd = N'S3cret' WHERE id = 7", + "UPDATE dbo.Users SET password = @p", + "SELECT 1 WHERE client_secret = 'S3cret'", + + // a typed declaration of a password variable (#5320): the name, a type with an optional length, then the literal + "DECLARE @mypassword nvarchar(20) = N'S3cret'", + "DECLARE @pwd varchar(max) = 'S3cret'", + "DECLARE @secret sysname = N'S3cret'", + "DECLARE @a int = 1, @pwd varchar(10) = 'S3cret'", + "DECLARE @a int = 1, @b nvarchar(20) = N'x', @db_passwd nvarchar(20) = N'S3cret', @c int", + "DECLARE @mypassword varbinary(64) = 0x0200AB", + "DECLARE @mypassword AS nvarchar(20) = N'S3cret'", + "DECLARE @mypassword [nvarchar](20) = N'S3cret'", + "DECLARE @mypassword sys.sysname = N'S3cret'", + "DECLARE @mypassword nvarchar (20) = N'S3cret'", + "DECLARE @mypassword decimal(10, 2) = N'S3cret'", + "DECLARE @pwd nvarchar(20) /* x */ = -- y\r\n N'S3cret'", + "CREATE PROCEDURE dbo.p @password nvarchar(20) = N'S3cret' AS SELECT 1", + "exec sp_executesql N'SELECT 1', N'@password nvarchar(20) = N''S3cret'''", + "DECLARE pwd text := 'secret-x'; BEGIN NULL; END", + "DECLARE db_secret varchar(20) = 'secret-x'", + "DECLARE api_secret constant text := e'secret-x'", + "DECLARE pwd text DEFAULT 'secret-x'", + "DECLARE pwd text := $q$secret-x$q$", + + // the typed declaration names exactly T3's name set, key_source included (#5320 L1) + "DECLARE @key_source nvarchar(100) = N'S3cret'", + "DECLARE key_source text := 'secret-x'", + "CREATE PROCEDURE dbo.p @key_source varchar(50) = 'S3cret' AS SELECT 1", + + // the PostgreSQL forms + "ALTER ROLE app PASSWORD 'secret-x'", + "CREATE ROLE r LOGIN PASSWORD 'secret-x'", + "CREATE USER MAPPING FOR u SERVER s OPTIONS (user 'u', password 'secret-x')", + "ALTER SYSTEM SET primary_conninfo = 'host=h password=secret-x'", + "CREATE SUBSCRIPTION sub CONNECTION 'host=h password=secret-x' PUBLICATION p", + "ALTER ROLE app SET work_mem = '64MB'", + "CREATE USER MAPPING FOR u SERVER s OPTIONS (user 'u', secret_access_key 'secret-x')", + "ALTER SERVER s OPTIONS (ADD token 'secret-x')", + "CREATE SERVER s FOREIGN DATA WRAPPER w OPTIONS (api_key 'secret-x')", + }; + + /// Statements the filter must leave alone. + public static readonly string[] NotNamed = + { + "SELECT name FROM sys.sql_logins", + "EXECUTE AS LOGIN = N'app'", + "SELECT * FROM dbo.Users WHERE PasswordHash = @h", + "UPDATE dbo.Users SET PasswordHash = HASHBYTES('SHA2_256', @p)", + "SELECT * FROM OPENROWSET(BULK N'f.json', SINGLE_CLOB) AS j", + "EXEC sp_helplogins", + "SELECT secret_id FROM dbo.t WHERE secret_id = 5", + "SELECT * FROM t WHERE pwd = @p", + "SELECT * FROM t WHERE pwd = $1", + + // typed declarations that carry no literal, and neighbours of the typed-declaration shape (#5320) + "DECLARE @password nvarchar(20) = @p", + "DECLARE @mypassword nvarchar(20)", + "DECLARE pwd text := $1", + "CREATE PROCEDURE dbo.p @password nvarchar(20) = NULL AS SELECT 1", + "CREATE TABLE dbo.u (id int, password_hash varbinary(64) NOT NULL)", + "SELECT * FROM t WHERE pwd_len int", + "DECLARE @pwd_len int = 8", + "DECLARE @pwd int = 8", + "DECLARE @mykey_source nvarchar(100) = N'x'", + "DECLARE @key_sources nvarchar(100) = N'x'", + "SELECT password varchar FROM t WHERE x = 'a'", + "SELECT * FROM t WHERE is_secret AND state = 'x'", + "SELECT [password] FROM t", + "SET password_encryption = 'scram-sha-256'", + "ALTER INDEX ix ON dbo.t REBUILD", + "BACKUP DATABASE d TO URL = 'https://storage.example/c/d.bak' WITH CREDENTIAL = 'cred'", + "ALTER AVAILABILITY GROUP ag FAILOVER", + + // the PostgreSQL neighbours + "SELECT * FROM t WHERE password_changed_at > $1", + "SET work_mem = '64MB'", + "SELECT rolname FROM pg_roles", + "SELECT 1", + "CREATE TABLE passwords (id int)", + }; + + /// The strings the plan once expected the .NET judge to time out on: many comment tokens after a + /// keyword. The linear-time pre-check (#5320) now answers Clean for them, as PostgreSQL does. They are not + /// part of the parity set, because the full .NET judge alone still backtracks on them. + public static readonly string[] Adversarial = + { + "create" + string.Concat(Enumerable.Repeat(" --", 40)) + "x", + "password" + string.Concat(Enumerable.Repeat(" --", 40)) + "x", + }; + + /// Strings aimed at the typed-declaration alternative (#5320): a password word, then many comment + /// tokens, a type and a long length part, but no literal. The pre-check answers Clean for each of them. Kept + /// apart from , which the output-filter tests prefix with a pre-check hit so the + /// full judge runs and times out; these do not make the full judge slow. + public static readonly string[] TypedDeclarationAdversarial = + { + "pwd" + string.Concat(Enumerable.Repeat(" --", 40)) + " nvarchar(20) = x", + "secret" + string.Concat(Enumerable.Repeat(" /* c */", 40)) + " text :", + "pwd varchar(" + new string(' ', 100_000) + "x", + "declare @pwd " + string.Concat(Enumerable.Repeat("nvarchar ", 5_000)) + "= @p", + }; + + /// A typed secret-named variable followed by a long dash banner, in a value where another part passes + /// the pre-check (hashed_pwd = HASHBYTES(1, @p) has no word start, so only the pre-check's superset hits + /// it, and the full judge then runs). The typed-declaration alternative backtracks on the run of dashes (the known + /// comment-run case, #5320 L2), so the .NET judge answers TimedOut, which counts as named, while PostgreSQL + /// answers Clean. It errs toward withholding. Not part of the parity set; the tests pin both answers. + public static readonly string[] NamedByTimeoutInDotNetOnly = + { + "UPDATE u SET hashed_pwd = HASHBYTES(1, @p); DECLARE @pwd nvarchar(20) " + new string('-', 60) + "\nSELECT 1", + }; + + /// A named statement as written, lower-cased, and with a leading block comment. + public static string[] Variants(string text) => new[] { text, text.ToLowerInvariant(), "/* c */ " + text }; +} diff --git a/Darling/Darling.Tests/SensitiveStatementHostFilterTests.cs b/Darling/Darling.Tests/SensitiveStatementHostFilterTests.cs new file mode 100644 index 0000000000..2d4480bed3 --- /dev/null +++ b/Darling/Darling.Tests/SensitiveStatementHostFilterTests.cs @@ -0,0 +1,314 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.IO; +using System.Linq; +using System.Text.Json; +using System.Text.Json.Nodes; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348: the statement filter as the Darling host REGISTERS it. +/// runs the filter alone around a fake tool; these run the host's real filter list (the unknown-argument guard, the +/// latency filter, GCF, the statement filter) through a real server, with and without +/// DARLING_OUTPUT_FORMAT=gcf, so the slot the filter holds in that list is part of what is pinned: placed +/// before GCF it would read a GCF wire instead of the tool's JSON, and the canary would come back. +/// +[Collection("DarlingOutputFormatEnv")] +public sealed class SensitiveStatementHostFilterTests : IDisposable +{ + private const string Marker = SensitiveStatements.PlaceholderText; + + public SensitiveStatementHostFilterTests() => + Environment.SetEnvironmentVariable("DARLING_OUTPUT_FORMAT", null); + + public void Dispose() => Environment.SetEnvironmentVariable("DARLING_OUTPUT_FORMAT", null); + + /// Eight uniform rows, so a GCF wire is smaller than the JSON and GCF really re-encodes the answer. + /// Row 1 carries the canary, row 2 a canary with a line feed between CREATE and LOGIN (a GCF wire escapes the + /// line feed, which defeats the filter's token match), the rest are plain. + private static string CanaryRows() + { + var rows = new JsonArray(); + for (int i = 0; i < 8; i++) + { + string text = i switch + { + 1 => StatementScrubCanary.CanaryStatement, + 2 => "CREATE\nLOGIN [canary_split_ssf] WITH PASSWORD = N'S3cret-canary-ssf'", + _ => StatementScrubCanary.PlainStatement, + }; + rows.Add(new JsonObject { ["query_text"] = text, ["executions"] = 100 + i, ["avg_ms"] = 10 + i }); + } + + return new JsonObject { ["rows"] = rows }.ToJsonString(); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task TheHostsRealFilterList_WithholdsTheCanaryPrecisely_WithAndWithoutGcf(bool gcf) + { + Environment.SetEnvironmentVariable("DARLING_OUTPUT_FORMAT", gcf ? "gcf" : null); + // gcf: null = the production wiring, the environment variable (#5320); the census default pins plain output + using var server = await StatementFilterCensus.BuildHostAsync(gcf: null); + string raw = CanaryRows(); + StatementFilterCensus.AssertRawHoldsTheCanary(raw); + if (gcf) + { + // the precondition: with nothing swept, GCF really re-encodes this answer, so the order is under test + Assert.NotNull(GcfOutput.TryEncode(raw)); + } + + var (text, isError) = await StatementFilterCensus.FilterThroughHostAsync(server, raw); + + Assert.False(isError); + StatementFilterCensus.AssertFilteredWithholdsTheCanary(raw, text); + Assert.DoesNotContain("canary_ssf", text); + Assert.DoesNotContain("canary_split_ssf", text); + Assert.Equal(gcf, !LooksLikeJson(text)); + + // precise: the two canary rows are the marker and the six plain rows are untouched, in either encoding + Assert.Equal(2, Count(text, Marker)); + Assert.Equal(6, Count(text, "canary_plain_ssf")); + if (!gcf) + { + var rows = JsonNode.Parse(text)!["rows"]!.AsArray(); + Assert.Equal(8, rows.Count); + Assert.Equal(Marker, (string?)rows[1]!["query_text"]); + Assert.Equal(Marker, (string?)rows[2]!["query_text"]); + foreach (int i in new[] { 0, 3, 4, 5, 6, 7 }) + { + Assert.Equal(StatementScrubCanary.PlainStatement, (string?)rows[i]!["query_text"]); + } + } + } + + /// #5320: the census host's format is pinned, so the environment variable (which another class in this + /// collection, or a live class running beside it, may have set) cannot change what a census answer looks like. Pinned + /// plain output returns a page with no hit byte for byte, while the variable says gcf. + [Fact] + public async Task APinnedPlainHost_ReturnsAPageWithNoHitUnchanged_EvenWhileTheEnvironmentSaysGcf() + { + Environment.SetEnvironmentVariable("DARLING_OUTPUT_FORMAT", "gcf"); + using var server = await StatementFilterCensus.BuildHostAsync(gcf: false); + string clean = CleanRows(); + Assert.NotNull(GcfOutput.TryEncode(clean)); // the precondition: an unpinned GCF host WOULD re-encode this page + + var (text, isError) = await StatementFilterCensus.FilterThroughHostAsync(server, clean); + + Assert.False(isError); + Assert.Equal(clean, text); + } + + /// #5320: the other pin. GCF pinned ON re-encodes the answer whatever the variable says, the filter still + /// reads the tool's own JSON first (it sits next to the tool), and so a hit is still the marker after GCF encoding. + [Fact] + public async Task APinnedGcfHost_StillWithholdsTheCanary_AfterGcfEncoding_WhileTheEnvironmentIsUnset() + { + Environment.SetEnvironmentVariable("DARLING_OUTPUT_FORMAT", null); + using var server = await StatementFilterCensus.BuildHostAsync(gcf: true); + string raw = CanaryRows(); + StatementFilterCensus.AssertRawHoldsTheCanary(raw); + Assert.NotNull(GcfOutput.TryEncode(raw)); + + var (text, isError) = await StatementFilterCensus.FilterThroughHostAsync(server, raw); + + Assert.False(isError); + Assert.False(LooksLikeJson(text), "the pinned-on host did not GCF-encode the answer"); + StatementFilterCensus.AssertFilteredWithholdsTheCanary(raw, text); + Assert.DoesNotContain("canary_ssf", text); + Assert.DoesNotContain("canary_split_ssf", text); + Assert.Equal(2, Count(text, Marker)); + Assert.Equal(6, Count(text, "canary_plain_ssf")); + } + + /// Eight uniform plain rows: nothing for the filter to withhold, and big enough for GCF to be smaller. + private static string CleanRows() + { + var rows = new JsonArray(); + for (int i = 0; i < 8; i++) + { + rows.Add(new JsonObject + { + ["id"] = i, + ["database_name"] = "db_" + i, + ["query_text"] = StatementScrubCanary.PlainStatement, + ["duration_ms"] = 100 + i, + }); + } + + return new JsonObject { ["rows"] = rows }.ToJsonString(); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task AnErrorResultThroughTheHostsRealFilterList_IsSwept_AndStaysAnError(bool gcf) + { + Environment.SetEnvironmentVariable("DARLING_OUTPUT_FORMAT", gcf ? "gcf" : null); + // gcf: null = the production wiring, the environment variable (#5320); the census default pins plain output + using var server = await StatementFilterCensus.BuildHostAsync(gcf: null); + string raw = new JsonObject + { + ["status"] = "error", + ["message"] = StatementScrubCanary.CanaryStatement, + }.ToJsonString(); + + var (text, isError) = await StatementFilterCensus.FilterThroughHostAsync(server, raw, isError: true); + + Assert.True(isError); + StatementFilterCensus.AssertFilteredWithholdsTheCanary(raw, text); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task AnMcpExceptionThatNamesAStatement_ThroughTheHostsRealFilterList_ReachesTheClientSwept(bool gcf) + { + // L1: the SDK builds an error result from a thrown McpException's message OUTSIDE the result sweep, so the + // filter has to catch the exception itself. Without that catch the client reads the canary in the message. + Environment.SetEnvironmentVariable("DARLING_OUTPUT_FORMAT", gcf ? "gcf" : null); + using var server = await StatementFilterCensus.BuildHostAsync(); + string message = StatementScrubCanary.CanaryStatement; + + var (text, isError) = await StatementFilterCensus.CallRealToolAsync( + server, StatementFilterCensus.ThrowProbeName, new JsonObject { ["message"] = message }); + + Assert.True(isError); + StatementFilterCensus.AssertFilteredWithholdsTheCanary(message, text); + } + + [Fact] + public async Task AnMcpExceptionThatNamesNoStatement_ThroughTheHostsRealFilterList_KeepsItsMessage() + { + using var server = await StatementFilterCensus.BuildHostAsync(); + + var (text, isError) = await StatementFilterCensus.CallRealToolAsync( + server, StatementFilterCensus.ThrowProbeName, new JsonObject { ["message"] = "server_name is required" }); + + Assert.True(isError); + Assert.Contains("server_name is required", text); + Assert.DoesNotContain(Marker, text); + } + + [Fact] + public async Task AnalyzePlanXmlOnTheCanaryPlan_ThroughTheHost_WithholdsStatementOneAndKeepsStatementTwo() + { + using var server = await StatementFilterCensus.BuildHostAsync(); + string plan = StatementScrubCanary.CanaryPlan(); + + var (text, isError) = await StatementFilterCensus.CallRealToolAsync( + server, "analyze_plan_xml", new JsonObject { ["plan_xml"] = plan }); + + Assert.False(isError); + StatementFilterCensus.AssertFilteredWithholdsTheCanary(plan, text); + JsonNode.Parse(text); + } + + [Fact] + public async Task TheFilterAlsoCoversTheCoreEndpoint_AndTheToolsAddedSincePlanning() + { + using var server = await StatementFilterCensus.BuildHostAsync(); + + // The filter list is global to the host, so a tool added after the plan is swept by the same registration. + // Pin that the newer tool is served at all, so the claim is about a tool this host really dispatches. + Assert.Contains("get_query_store_query_history", await StatementFilterCensus.ListToolNamesAsync(server)); + + var (text, isError) = await StatementFilterCensus.CallRealToolAsync( + server, "get_tool_guide", new JsonObject(), "/core"); + Assert.False(isError); + JsonNode.Parse(text); + } + + [Fact] + public async Task EveryGetToolGuideTopic_IsByteIdenticalThroughTheSweep_AndThroughTheHost() + { + using var server = await StatementFilterCensus.BuildHostAsync(); + var catalog = server.Services.GetRequiredService(); + var topics = McpToolGuideTopics.All; + Assert.NotEmpty(topics); + + foreach (string? name in topics.Select(t => t.Name).Prepend(null)) + { + string[]? names = name is null ? null : new[] { name }; + string rendered = McpToolGuide.Render(catalog, topics, null, names); + + // through the sweep alone: the very same text, not an equal copy + var swept = SensitiveStatementOutputFilter.Sweep(new ModelContextProtocol.Protocol.CallToolResult + { + Content = new System.Collections.Generic.List { new ModelContextProtocol.Protocol.TextContentBlock { Text = rendered } }, + }); + Assert.Same(rendered, ((ModelContextProtocol.Protocol.TextContentBlock)swept.Content[0]).Text); + + // through the host's whole registered list + var args = new JsonObject(); + if (names is not null) args["topics"] = new JsonArray(JsonValue.Create(name)); + var (text, isError) = await StatementFilterCensus.CallRealToolAsync(server, "get_tool_guide", args); + Assert.False(isError); + Assert.Equal(rendered, text); + } + } + + [Fact] + public void TheHostRegistersTheStatementFilterLast_AfterGcf() + { + string source = File.ReadAllText(HostSourcePath()); + int gcf = source.IndexOf("AddCallToolFilter(GcfCallToolFilter.Instance)", StringComparison.Ordinal); + int sweep = source.IndexOf("AddCallToolFilter(SensitiveStatementOutputFilter.Instance)", StringComparison.Ordinal); + + Assert.True(gcf > 0, "the host no longer registers GCF through AddCallToolFilter(GcfCallToolFilter.Instance)"); + Assert.True(sweep > gcf, "the statement filter must be registered AFTER GCF: the last-added filter sits next to the tool"); + Assert.Equal(sweep, source.LastIndexOf(".AddCallToolFilter(", StringComparison.Ordinal) + 1); + Assert.Equal(2, source.Split("AddCallToolFilter(SensitiveStatementOutputFilter.Instance)").Length); + } + + private static bool LooksLikeJson(string text) + { + try + { + JsonDocument.Parse(text).Dispose(); + return true; + } + catch (JsonException) + { + return false; + } + } + + private static int Count(string text, string needle) + { + int count = 0; + for (int at = text.IndexOf(needle, StringComparison.Ordinal); at >= 0; + at = text.IndexOf(needle, at + needle.Length, StringComparison.Ordinal)) + { + count++; + } + + return count; + } + + private static string HostSourcePath() + { + string? dir = AppContext.BaseDirectory; + while (dir is not null && !Directory.Exists(Path.Combine(dir, "Darling", "PerformanceMonitor.Darling.Service"))) + { + dir = Path.GetDirectoryName(dir); + } + + if (dir is null) throw new InvalidOperationException("The repository root was not found above the test binaries."); + return Path.Combine(dir, "Darling", "PerformanceMonitor.Darling.Service", "Mcp", "DarlingMcpHostService.cs"); + } +} diff --git a/Darling/Darling.Tests/SensitiveStatementJsonTests.cs b/Darling/Darling.Tests/SensitiveStatementJsonTests.cs new file mode 100644 index 0000000000..432f7c2fd8 --- /dev/null +++ b/Darling/Darling.Tests/SensitiveStatementJsonTests.cs @@ -0,0 +1,299 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Linq; +using System.Text; +using System.Text.Json.Nodes; +using System.Xml.Linq; +using PerformanceMonitor.Common; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348: the JSON walk of the statement filter, run with fake judges so only the mechanics are under test: which +/// strings reach which judge, what a replaced key looks like, when the original instance comes back, and what +/// happens when a judge throws. +/// +public sealed class SensitiveStatementJsonTests +{ + private readonly ITestOutputHelper _output; + + public SensitiveStatementJsonTests(ITestOutputHelper output) => _output = output; + + private const string Marker = "[withheld]"; + private const string Refusal = "Output withheld: test refusal."; + + private sealed class Recorder + { + public readonly List TextCalls = new(); + public readonly List XmlCalls = new(); + + public string? Text(string? s) + { + TextCalls.Add(s ?? ""); + return s != null && s.Contains("SECRET", StringComparison.Ordinal) ? Marker : s; + } + + public string? Xml(string? s) + { + XmlCalls.Add(s ?? ""); + return s != null && s.Contains("SECRET", StringComparison.Ordinal) ? "" : s; + } + + public string Run(string output) => SensitiveStatements.JsonCore(output, Text, Xml, Refusal); + } + + [Fact] + public void NestedObjectsAndArraysAreWalked() + { + var r = new Recorder(); + string output = "{\"a\":{\"b\":[\"first SECRET value\",{\"c\":\"keep this one\",\"d\":[\"deep SECRET text\"]}],\"n\":7},\"flag\":true,\"nothing\":null}"; + + string result = r.Run(output); + + Assert.Equal( + "{\"a\":{\"b\":[\"" + Marker + "\",{\"c\":\"keep this one\",\"d\":[\"" + Marker + "\"]}],\"n\":7},\"flag\":true,\"nothing\":null}", + result); + } + + [Fact] + public void AngleFirstValuesGoToXmlAndOthersGoToText() + { + var r = new Recorder(); + string output = "{\"plan\":\" SECRET\",\"short_xml\":\"\",\"stmt\":\"select SECRET from t\",\"tiny\":\"abcd\",\"number\":12345}"; + + string result = r.Run(output); + + Assert.Equal(new[] { " SECRET", "" }, r.XmlCalls); + Assert.DoesNotContain(r.TextCalls, c => c.StartsWith(" <", StringComparison.Ordinal) || c == ""); + Assert.Contains("select SECRET from t", r.TextCalls); + Assert.DoesNotContain("abcd", r.TextCalls); + Assert.DoesNotContain("12345", r.TextCalls); + var node = JsonNode.Parse(result)!.AsObject(); + Assert.Equal("", (string?)node["plan"]); + Assert.Equal("", (string?)node["short_xml"]); + Assert.Equal(Marker, (string?)node["stmt"]); + Assert.Equal("abcd", (string?)node["tiny"]); + Assert.Equal(12345, (int?)node["number"]); + } + + [Fact] + public void NamedKeyBecomesMarkerWithPositionAndKeysStayUnique() + { + var r = new Recorder(); + string output = "{\"keep\":1,\"SECRET_key_one\":\"v1\",\"SECRET_key_two\":\"v2\",\"plain_key\":\"v3\"}"; + + string result = r.Run(output); + + Assert.Equal("{\"keep\":1,\"" + Marker + "#1\":\"v1\",\"" + Marker + "#2\":\"v2\",\"plain_key\":\"v3\"}", result); + var keys = JsonNode.Parse(result)!.AsObject().Select(p => p.Key).ToList(); + Assert.Equal(keys.Count, keys.Distinct(StringComparer.Ordinal).Count()); + } + + [Fact] + public void ValueOfANamedKeyIsStillJudged() + { + var r = new Recorder(); + + string result = r.Run("{\"SECRET_key_name\":\"also SECRET value\"}"); + + Assert.Equal("{\"" + Marker + "#0\":\"" + Marker + "\"}", result); + } + + [Fact] + public void NamedKeysInNestedObjectsUseTheirOwnPosition() + { + var r = new Recorder(); + + string result = r.Run("{\"x\":[{\"a\":1,\"b\":2,\"SECRET_nested_key\":3}]}"); + + Assert.Equal("{\"x\":[{\"a\":1,\"b\":2,\"" + Marker + "#2\":3}]}", result); + } + + [Fact] + public void RootStringIsJudged() + { + var r = new Recorder(); + + Assert.Equal("\"" + Marker + "\"", r.Run("\"root SECRET text\"")); + } + + [Fact] + public void NothingChangedReturnsTheOriginalInstance() + { + var r = new Recorder(); + // Pretty-printed on purpose: unchanged output must not be re-serialized into the compact form. + string output = "{\n \"a\": [ \"nothing to see here\", 1, 2 ],\n \"b\": { \"c\": \"\" }\n}"; + + string result = r.Run(output); + + Assert.Same(output, result); + } + + [Fact] + public void ChangedOutputIsReserializedWithTheMcpOptions() + { + var r = new Recorder(); + string output = "{\n \"a\" : \"SECRET text here\",\n \"b\" : [ 1, 2 ]\n}"; + + string result = r.Run(output); + + Assert.Equal("{\"a\":\"" + Marker + "\",\"b\":[1,2]}", result); + } + + [Fact] + public void JsonThatDoesNotParse_GivesTheRefusal_NotAJudgmentOfTheEscapedText() + { + // A cut object, and an array nested past the parser's default depth: both start like JSON and are not. + // Judged as text they would be the escaped form (a line break is the two characters \r\n), which a pattern + // written for the decoded statement does not match, so the raw output would come back. + string deep = new string('[', 70) + "\"x\"" + new string(']', 70); + foreach (string output in new[] { "{\"a\":\"CREATE\\r\\nSECRET x\"", " [\"CREATE\\r\\nSECRET x\"", deep }) + { + var r = new Recorder(); + Assert.Equal(Refusal, r.Run(output)); + Assert.Empty(r.TextCalls); + Assert.Empty(r.XmlCalls); + } + } + + [Fact] + public void NonJsonIsJudgedWhole() + { + var r = new Recorder(); + + string plainText = "Error: SECRET statement text, not json"; + Assert.Equal(Marker, r.Run(plainText)); + Assert.Equal(new[] { plainText }, r.TextCalls); + Assert.Empty(r.XmlCalls); + + var r2 = new Recorder(); + string plan = "\r\n SECRET"; + Assert.Equal("", r2.Run(plan)); + Assert.Equal(new[] { plan }, r2.XmlCalls); + Assert.Empty(r2.TextCalls); + } + + [Fact] + public void NonJsonThatIsCleanComesBackAsTheSameInstance() + { + var r = new Recorder(); + string plain = "No server named that is registered."; + string plan = "clean"; + + Assert.Same(plain, r.Run(plain)); + Assert.Same(plan, r.Run(plan)); + } + + [Fact] + public void AThrowingJudgeGivesExactlyTheRefusalArgument() + { + Func boom = _ => throw new InvalidOperationException("judge failed"); + Func ok = s => s; + + // text judge: a value, a key, and a whole non-JSON output. + Assert.Equal(Refusal, SensitiveStatements.JsonCore("{\"a\":\"some statement text\"}", boom, ok, Refusal)); + Assert.Equal(Refusal, SensitiveStatements.JsonCore("{\"some_long_key\":1}", boom, ok, Refusal)); + Assert.Equal(Refusal, SensitiveStatements.JsonCore("plain text output", boom, ok, Refusal)); + // xml judge: a value and a whole output. + Assert.Equal(Refusal, SensitiveStatements.JsonCore("{\"a\":\"\"}", ok, boom, Refusal)); + Assert.Equal(Refusal, SensitiveStatements.JsonCore("", ok, boom, Refusal)); + // a different refusal argument comes back verbatim. + Assert.Equal("other sentence", SensitiveStatements.JsonCore("", ok, boom, "other sentence")); + } + + [Fact] + public void AJudgeThatReturnsNullForWholeOutputGivesTheRefusal() + { + Func nul = _ => null; + + Assert.Equal(Refusal, SensitiveStatements.JsonCore("plain text output", nul, nul, Refusal)); + } + + [Fact] + public void EmptyOutputIsReturnedAsIs() + { + var r = new Recorder(); + + Assert.Same(string.Empty, r.Run(string.Empty)); + Assert.Empty(r.TextCalls); + } + + [Fact] + public void CanaryPlanParsesWithThreeStatementsAndEveryNeedle() + { + string plan = StatementScrubCanary.CanaryPlan(); + + var doc = XDocument.Parse(plan); + Assert.Equal(3, doc.Descendants().Count(e => e.Name.LocalName == "StmtSimple")); + foreach (string needle in StatementScrubCanary.SecretNeedles) Assert.Contains(needle, plan); + foreach (string needle in StatementScrubCanary.KeptNeedles) Assert.Contains(needle, plan); + + var texts = doc.Descendants().Where(e => e.Name.LocalName == "StmtSimple") + .Select(e => (string?)e.Attribute("StatementText")).ToList(); + Assert.Equal(StatementScrubCanary.CanaryStatement, texts[0]); + Assert.Equal(StatementScrubCanary.PlainStatement, texts[1]); + Assert.Equal(StatementScrubCanary.AutoParamStatement, texts[2]); + Assert.Contains("S3cret-canary-ssf", StatementScrubCanary.CanaryStatement); + Assert.Contains("canary_plain_ssf", StatementScrubCanary.PlainStatement); + } + + [Fact] + public void CanaryPlanInsideJsonReachesTheXmlJudge() + { + var r = new Recorder(); + string plan = StatementScrubCanary.CanaryPlan(); + string output = new JsonObject { ["plan_xml"] = plan, ["note"] = "no statement here" }.ToJsonString(); + + Assert.Same(output, r.Run(output)); + Assert.Equal(new[] { plan }, r.XmlCalls); + } + + /// + /// Measurement for the lane report, not a gate: the walk's own cost with identity judges on a 1 MB, 200-row + /// response. It asserts only that identity judges leave the output untouched. + /// + [Fact] + public void MeasureWalkCostWithIdentityJudges() + { + var rows = new JsonArray(); + string filler = new string('x', 5000); + for (int i = 0; i < 200; i++) + { + rows.Add(new JsonObject + { + ["id"] = i, + ["query_text"] = "select " + filler + " from dbo.t where c = " + i, + ["query_plan"] = "" + filler.Substring(0, 100) + "", + ["database_name"] = "db" + i, + }); + } + string output = new JsonObject { ["rows"] = rows }.ToJsonString(); + Assert.InRange(Encoding.UTF8.GetByteCount(output), 1_000_000, 1_300_000); + + Func identity = s => s; + Assert.Same(output, SensitiveStatements.JsonCore(output, identity, identity, Refusal)); + + var times = new List(); + for (int run = 0; run < 20; run++) + { + long start = Stopwatch.GetTimestamp(); + string result = SensitiveStatements.JsonCore(output, identity, identity, Refusal); + times.Add(Stopwatch.GetElapsedTime(start).TotalMilliseconds); + Assert.Same(output, result); + } + times.Sort(); + double median = (times[9] + times[10]) / 2; + string line = $"JsonCore identity walk, {Encoding.UTF8.GetByteCount(output)} bytes, 200 rows, 20 runs: median {median:F2} ms, min {times[0]:F2} ms, max {times[19]:F2} ms"; + _output.WriteLine(line); + } +} diff --git a/Darling/Darling.Tests/SensitiveStatementOutputFilterTests.cs b/Darling/Darling.Tests/SensitiveStatementOutputFilterTests.cs new file mode 100644 index 0000000000..0910dadcd4 --- /dev/null +++ b/Darling/Darling.Tests/SensitiveStatementOutputFilterTests.cs @@ -0,0 +1,629 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.IO; +using System.Linq; +using System.Text.Json; +using System.Text.Json.Nodes; +using System.Threading; +using System.Threading.Tasks; +using System.Xml.Linq; +using ModelContextProtocol; +using ModelContextProtocol.Protocol; +using ModelContextProtocol.Server; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.PlanAnalysis; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348: the MCP output filter, run through its real call-tool filter around a fake tool, so what is under test is +/// what a client would receive: which strings come back as the marker, where the marker sits in the structure, that +/// an error result is swept, how long a result that tries to run the judge out of time takes, and that the real +/// plan analyzer's output is withheld for the canary statement and kept for the plain one. +/// +public sealed class SensitiveStatementOutputFilterTests +{ + private readonly ITestOutputHelper _output; + + public SensitiveStatementOutputFilterTests(ITestOutputHelper output) => _output = output; + + private const string Marker = SensitiveStatements.PlaceholderText; + + private static readonly string s_plainRow = + new JsonObject { ["query_text"] = StatementScrubCanary.PlainStatement, ["rows"] = 3 }.ToJsonString(); + + /// Runs through the real filter as a tool's return value. + private static async Task ThroughFilterAsync(CallToolResult result) + { + McpRequestHandler handler = + SensitiveStatementOutputFilter.Instance((_, _) => ValueTask.FromResult(result)); + return await handler(null!, CancellationToken.None); + } + + private static async Task ThrownThroughFilterAsync(McpException thrown) + { + McpRequestHandler handler = + SensitiveStatementOutputFilter.Instance((_, _) => throw thrown); + return await Assert.ThrowsAnyAsync(async () => await handler(null!, CancellationToken.None)); + } + + [Fact] + public async Task AThrownMcpException_IsRethrownWithItsMessageSwept_KeepingTypeCodeAndInnerException() + { + // L1: the SDK writes a thrown McpException's message into an error result outside the result sweep + var inner = new InvalidOperationException("inner"); + var plain = await ThrownThroughFilterAsync(new McpException("server_name is required", inner)); + Assert.Equal("server_name is required", plain.Message); + + var swept = await ThrownThroughFilterAsync(new McpException(StatementScrubCanary.CanaryStatement, inner)); + Assert.Equal(typeof(McpException), swept.GetType()); + Assert.Same(inner, swept.InnerException); + Assert.Equal(Marker, swept.Message); + + var protocol = await ThrownThroughFilterAsync( + new McpProtocolException(StatementScrubCanary.CanaryStatement, inner, McpErrorCode.InvalidParams)); + var rethrown = Assert.IsType(protocol); + Assert.Equal(McpErrorCode.InvalidParams, rethrown.ErrorCode); + Assert.Same(inner, rethrown.InnerException); + Assert.Equal(Marker, rethrown.Message); + } + + private static CallToolResult Text(string text, bool? isError = null) => new() + { + Content = new List { new TextContentBlock { Text = text } }, + IsError = isError, + }; + + private static string TextOf(CallToolResult r) => ((TextContentBlock)r.Content[0]).Text; + + private static string Json(params (string Key, string Value)[] pairs) + { + var obj = new JsonObject(); + foreach (var (key, value) in pairs) obj[key] = value; + return obj.ToJsonString(); + } + + // ── precise: the marker lands inside the structure, plain text is untouched ── + + [Fact] + public async Task ACanaryStatementBecomesTheMarkerInsideTheStructure_APlainOneIsUnchanged() + { + string json = Json( + ("named", StatementScrubCanary.CanaryStatement), + ("plain", StatementScrubCanary.PlainStatement)); + + var result = await ThroughFilterAsync(Text(json)); + + var parsed = JsonNode.Parse(TextOf(result))!.AsObject(); + Assert.Equal(Marker, (string?)parsed["named"]); + Assert.Equal(StatementScrubCanary.PlainStatement, (string?)parsed["plain"]); + Assert.Equal(2, parsed.Count); + Assert.DoesNotContain("S3cret-canary-ssf", TextOf(result)); + } + + [Fact] + public async Task ACreateAndLoginSplitByALineFeedIsWithheldToo() + { + string statement = "CREATE\nLOGIN [canary_lf_ssf] WITH PASSWORD = N'S3cret-lf-ssf'"; + string json = Json(("sql_text", statement), ("note", "second value kept as is")); + + var result = await ThroughFilterAsync(Text(json)); + + var parsed = JsonNode.Parse(TextOf(result))!.AsObject(); + Assert.Equal(Marker, (string?)parsed["sql_text"]); + Assert.Equal("second value kept as is", (string?)parsed["note"]); + Assert.DoesNotContain("S3cret-lf-ssf", TextOf(result)); + } + + // ── blocks the filter cannot read are refused, and every block's _meta is swept ── + + private static void AssertTheFixedRefusal(CallToolResult result) + { + Assert.True(result.IsError); + Assert.Single(result.Content); + Assert.Equal(SensitiveStatements.JsonRefusal, TextOf(result)); + Assert.Null(result.StructuredContent); + } + + [Fact] + public async Task ABlobResourceBlockBecomesTheFixedRefusal() + { + var blob = new BlobResourceContents + { + Uri = "file:///x", + MimeType = "text/plain", + Blob = System.Text.Encoding.UTF8.GetBytes(StatementScrubCanary.CanaryStatement), + }; + var original = new CallToolResult + { + Content = new List + { + new TextContentBlock { Text = s_plainRow }, + new EmbeddedResourceBlock { Resource = blob }, + }, + }; + + AssertTheFixedRefusal(await ThroughFilterAsync(original)); + } + + [Fact] + public async Task AResourceLinkBlockBecomesTheFixedRefusal() + { + var original = new CallToolResult + { + Content = new List + { + new ResourceLinkBlock + { + Uri = "file:///x", + Name = StatementScrubCanary.CanaryStatement, + Description = StatementScrubCanary.CanaryStatement, + }, + }, + }; + + var result = await ThroughFilterAsync(original); + + AssertTheFixedRefusal(result); + Assert.DoesNotContain("S3cret-canary-ssf", TextOf(result)); + } + + [Fact] + public async Task ATextBlockCarryingACanaryInItsMetaHasTheCanaryWithheld() + { + var original = new CallToolResult + { + Content = new List + { + new TextContentBlock + { + Text = s_plainRow, + Meta = new JsonObject { ["note"] = StatementScrubCanary.CanaryStatement, ["keep"] = "plain" }, + }, + }, + }; + + var result = await ThroughFilterAsync(original); + + var block = Assert.IsType(result.Content[0]); + Assert.Equal(s_plainRow, block.Text); + Assert.Equal(Marker, (string?)block.Meta!["note"]); + Assert.Equal("plain", (string?)block.Meta["keep"]); + Assert.DoesNotContain("S3cret-canary-ssf", block.Meta.ToJsonString()); + } + + [Fact] + public async Task ACanaryInAnEmbeddedTextResourcesMetaOrTheResultsMetaIsWithheld() + { + var original = new CallToolResult + { + Content = new List + { + new EmbeddedResourceBlock + { + Meta = new JsonObject { ["a"] = StatementScrubCanary.CanaryStatement }, + Resource = new TextResourceContents + { + Uri = "file:///x", + Text = s_plainRow, + Meta = new JsonObject { ["b"] = StatementScrubCanary.CanaryStatement }, + }, + }, + }, + Meta = new JsonObject { ["c"] = StatementScrubCanary.CanaryStatement }, + }; + + var result = await ThroughFilterAsync(original); + + var embedded = Assert.IsType(result.Content[0]); + var resource = Assert.IsType(embedded.Resource); + Assert.Equal(Marker, (string?)embedded.Meta!["a"]); + Assert.Equal(Marker, (string?)resource.Meta!["b"]); + Assert.Equal(Marker, (string?)result.Meta!["c"]); + Assert.Equal(s_plainRow, resource.Text); + } + + // A URI or a MIME type that carries a named statement is useless once the statement is replaced, so the + // whole result becomes the fixed refusal (the block keeps a valid shape that way). + + [Fact] + public async Task ACanaryInAnEmbeddedTextResourcesUriBecomesTheFixedRefusal() + { + var original = new CallToolResult + { + Content = new List + { + new EmbeddedResourceBlock + { + Resource = new TextResourceContents + { + Uri = "x:" + StatementScrubCanary.CanaryStatement, + MimeType = "text/plain", + Text = "plain row", + }, + }, + }, + }; + + var result = await ThroughFilterAsync(original); + + AssertTheFixedRefusal(result); + Assert.DoesNotContain("S3cret-canary-ssf", TextOf(result)); + } + + [Fact] + public async Task ACanaryInAnEmbeddedTextResourcesMimeTypeBecomesTheFixedRefusal() + { + var original = new CallToolResult + { + Content = new List + { + new EmbeddedResourceBlock + { + Resource = new TextResourceContents + { + Uri = "file:///x", + MimeType = StatementScrubCanary.CanaryStatement, + Text = "plain row", + }, + }, + }, + }; + + var result = await ThroughFilterAsync(original); + + AssertTheFixedRefusal(result); + Assert.DoesNotContain("S3cret-canary-ssf", TextOf(result)); + } + + [Fact] + public async Task AnEmbeddedTextResourceWithAPlainUriAndMimeTypeKeepsTheResultAsTheSameInstance() + { + var original = new CallToolResult + { + Content = new List + { + new EmbeddedResourceBlock + { + Resource = new TextResourceContents + { + Uri = "file:///x", + MimeType = "text/plain", + Text = "plain row", + }, + }, + }, + }; + + var result = await ThroughFilterAsync(original); + + Assert.Same(original, result); + } + + [Fact] + public async Task AMetaThatNamesNothingKeepsTheResultAsTheSameInstance() + { + var original = new CallToolResult + { + Content = new List + { + new TextContentBlock { Text = s_plainRow, Meta = new JsonObject { ["k"] = "v" } }, + }, + Meta = new JsonObject { ["k"] = "v" }, + }; + + Assert.Same(original, await ThroughFilterAsync(original)); + } + + [Fact] + public async Task AResultThatNamesNothingComesBackAsTheSameInstance() + { + var original = Text(s_plainRow); + + var result = await ThroughFilterAsync(original); + + Assert.Same(original, result); + Assert.Same(s_plainRow, TextOf(result)); + } + + [Fact] + public async Task AnErrorResultIsSweptAndStaysAnError() + { + string json = Json(("error", "failed near: " + StatementScrubCanary.CanaryStatement)); + + var result = await ThroughFilterAsync(Text(json, isError: true)); + + Assert.True(result.IsError); + Assert.DoesNotContain("S3cret-canary-ssf", TextOf(result)); + Assert.Contains(Marker, TextOf(result)); + } + + [Fact] + public async Task APlainTextErrorSentenceIsSwept() + { + var result = await ThroughFilterAsync( + Text("could not run: " + StatementScrubCanary.CanaryStatement, isError: true)); + + Assert.True(result.IsError); + Assert.Equal(Marker, TextOf(result)); + } + + [Fact] + public async Task EveryTextBlockOfAResultIsSwept_AndTheOtherFieldsCarryOver() + { + var original = new CallToolResult + { + Content = new List + { + new TextContentBlock { Text = s_plainRow }, + new TextContentBlock { Text = Json(("q", StatementScrubCanary.CanaryStatement)) }, + }, + IsError = false, + }; + + var result = await ThroughFilterAsync(original); + + Assert.Equal(2, result.Content.Count); + Assert.Same(s_plainRow, ((TextContentBlock)result.Content[0]).Text); + Assert.DoesNotContain("S3cret-canary-ssf", ((TextContentBlock)result.Content[1]).Text); + Assert.False(result.IsError); + } + + [Fact] + public async Task StructuredContentIsSweptToo() + { + using var doc = JsonDocument.Parse(Json(("q", StatementScrubCanary.CanaryStatement), ("n", "keep this value"))); + var original = Text(s_plainRow); + original.StructuredContent = doc.RootElement.Clone(); + + var result = await ThroughFilterAsync(original); + + Assert.NotNull(result.StructuredContent); + string raw = result.StructuredContent!.Value.GetRawText(); + Assert.DoesNotContain("S3cret-canary-ssf", raw); + Assert.Equal(Marker, result.StructuredContent.Value.GetProperty("q").GetString()); + Assert.Equal("keep this value", result.StructuredContent.Value.GetProperty("n").GetString()); + } + + [Fact] + public async Task ANamedKeyBecomesTheMarkerWithItsPosition_AndItsValueIsKept() + { + string json = new JsonObject + { + ["id"] = "row one", + [StatementScrubCanary.CanaryStatement] = "the value under a named key", + }.ToJsonString(); + + var result = await ThroughFilterAsync(Text(json)); + + var parsed = JsonNode.Parse(TextOf(result))!.AsObject(); + Assert.Equal(2, parsed.Count); + Assert.Equal("row one", (string?)parsed["id"]); + Assert.Equal("the value under a named key", (string?)parsed[Marker + "#1"]); + Assert.DoesNotContain("S3cret-canary-ssf", TextOf(result)); + } + + // ── the clock: a result built to run the judge out of time ── + + /// A near-hit the linear-time pre-check passes and the full judge times out on (see + /// SensitiveStatementsTests): each costs one 250 ms match timeout until the budget is spent. + private static IEnumerable NearHits() => + SensitiveStatementCorpus.Adversarial.Select(adversarial => "xcreate login; " + adversarial); + + /// Every string value in every row is the marker. A key is judged like a value, so once the budget is + /// spent a key is the marker plus its position; before that it is kept. + private static void AssertEveryStringIsTheMarker(JsonArray rows) + { + foreach (var row in rows) + { + foreach (var pair in row!.AsObject()) + { + Assert.True(pair.Key is "i" or "query_text" || pair.Key.StartsWith(Marker, StringComparison.Ordinal), pair.Key); + if (pair.Value is JsonValue value && value.GetValueKind() == JsonValueKind.String) + Assert.Equal(Marker, value.GetValue()); + } + } + } + + [Fact] + public async Task AHundredValuesThatTimeOutFinishWithinTheBudgetAndEveryOneIsWithheld() + { + var hits = NearHits().ToArray(); + var rows = new JsonArray(); + for (int i = 0; i < 100; i++) + rows.Add(new JsonObject { ["i"] = i, ["query_text"] = hits[i % hits.Length] }); + string json = rows.ToJsonString(); + SensitiveStatements.Judge("warm the regex up"); + await ThroughFilterAsync(Text(Json(("q", "warm the filter up")))); + + // The bound belongs to the code, not to the runner's load: assert the BEST of three runs and record the worst. + double worst = 0; + double best = double.MaxValue; + CallToolResult? last = null; + for (int attempt = 0; attempt < 3; attempt++) + { + var started = Stopwatch.GetTimestamp(); + last = await ThroughFilterAsync(Text(json)); + double ms = Stopwatch.GetElapsedTime(started).TotalMilliseconds; + worst = Math.Max(worst, ms); + best = Math.Min(best, ms); + } + + _output.WriteLine($"100-value case over 3 runs: worst {worst:F0} ms, best {best:F0} ms"); + TimingClaim.AtMost(best, 1950, $"100 timing-out values, best of 3 (worst {worst:F0} ms)"); + var parsed = JsonNode.Parse(TextOf(last!))!.AsArray(); + Assert.Equal(100, parsed.Count); + AssertEveryStringIsTheMarker(parsed); + } + + [Fact] + public async Task ABudgetSharedByTwoBlocksIsSpentOnceNotTwice() + { + var hits = NearHits().ToArray(); + var rows = new JsonArray(); + for (int i = 0; i < 50; i++) + rows.Add(new JsonObject { ["query_text"] = hits[i % hits.Length] }); + string json = rows.ToJsonString(); + var original = new CallToolResult + { + Content = new List + { + new TextContentBlock { Text = json }, + new TextContentBlock { Text = json }, + }, + }; + SensitiveStatements.Judge("warm the regex up"); + + // Best of three, worst recorded (see the 100-value case above). + double worst = 0; + double best = double.MaxValue; + CallToolResult? result = null; + for (int attempt = 0; attempt < 3; attempt++) + { + var started = Stopwatch.GetTimestamp(); + result = await ThroughFilterAsync(original); + double ms = Stopwatch.GetElapsedTime(started).TotalMilliseconds; + worst = Math.Max(worst, ms); + best = Math.Min(best, ms); + } + + _output.WriteLine($"two blocks of 50 timing-out values over 3 runs: worst {worst:F0} ms, best {best:F0} ms"); + TimingClaim.AtMost(best, 1950, $"two blocks of 50 timing-out values, best of 3 (worst {worst:F0} ms)"); + foreach (var block in result!.Content.Cast()) + AssertEveryStringIsTheMarker(JsonNode.Parse(block.Text)!.AsArray()); + } + + /// Measurement for the lane report, not a gate: the sweep's cost on a 1 MB response that names + /// nothing (200 rows of 5000 characters plus a plain statement each). + [Fact] + public async Task MeasureSweepOverheadOnAOneMegabyteResponseThatNamesNothing() + { + await StatementFilterWarmUp.EnsureAsync(); + var rows = new JsonArray(); + string filler = string.Concat(Enumerable.Repeat("SELECT c FROM dbo.t WHERE id = @id; ", 140)); + for (int i = 0; i < 200; i++) + rows.Add(new JsonObject { ["id"] = i, ["query_text"] = filler + i, ["database_name"] = "db" + i }); + string json = rows.ToJsonString(); + var original = Text(json); + await ThroughFilterAsync(original); + + var times = new List(); + for (int i = 0; i < 9; i++) + { + var started = Stopwatch.GetTimestamp(); + var result = await ThroughFilterAsync(original); + times.Add(Stopwatch.GetElapsedTime(started).TotalMilliseconds); + Assert.Same(original, result); + } + + times.Sort(); + _output.WriteLine($"{json.Length:N0} characters; median sweep {times[times.Count / 2]:F1} ms (min {times[0]:F1}, max {times[^1]:F1})"); + Assert.True(json.Length >= 1_000_000, $"{json.Length} characters"); + } + + // ── the real plan analyzer ── + + [Fact] + public async Task AnalyzePlanXmlOnTheCanaryPlanWithholdsStatementOneAndKeepsStatementTwo() + { + // the control: the analysis of the unfiltered plan holds the parameter and constant values + string control = McpPlanAnalysisFormatter.BuildAnalysisResult( + StatementScrubCanary.CanaryPlan(), null, "xml", null, (AnalyzerConfig?)null, CancellationToken.None); + Assert.Contains("param-secret-ssf", control); + + string analysis = DarlingMcpPlanTools.AnalyzePlanXml(StatementScrubCanary.CanaryPlan()); + var result = await ThroughFilterAsync(Text(analysis)); + string text = TextOf(result); + + _output.WriteLine(text.Length > 600 ? text.Substring(0, 600) : text); + Assert.DoesNotContain("S3cret-canary-ssf", text); + Assert.DoesNotContain("param-secret-ssf", text); + Assert.DoesNotContain("const-secret-ssf", text); + Assert.Contains(Marker, text); + Assert.Contains("canary_plain_ssf", text); + Assert.Contains("param-canary-ssf", text); + JsonNode.Parse(text); + } + + // ── the real XML read ── + + [Fact] + public void TheCanaryPlanThroughTheRealXmlWithholdsEverySecretNeedleAndKeepsStatementTwoAndThree() + { + string plan = StatementScrubCanary.CanaryPlan(); + + string? result = SensitiveStatements.Xml(plan); + + Assert.NotNull(result); + Assert.NotSame(plan, result); + foreach (string needle in StatementScrubCanary.SecretNeedles) Assert.DoesNotContain(needle, result); + foreach (string needle in StatementScrubCanary.KeptNeedles) Assert.Contains(needle, result); + var texts = XDocument.Parse(result!).Descendants().Where(e => e.Name.LocalName == "StmtSimple") + .Select(e => (string?)e.Attribute("StatementText")).ToList(); + Assert.Equal(3, texts.Count); + Assert.Equal(Marker, texts[0]); + Assert.Equal(StatementScrubCanary.PlainStatement, texts[1]); + Assert.Equal(StatementScrubCanary.AutoParamStatement, texts[2]); + } + + [Theory] + [InlineData("autoparam_update_prepared.xml")] + [InlineData("autoparam_select_prepared.xml")] + [InlineData("autoparam_update_adhoc_shell.xml")] + [InlineData("autoparam_select_adhoc_shell.xml")] + public void ACapturedPlanThroughTheRealXmlWithholdsTheLiteral(string file) + { + string xml = File.ReadAllText(Path.Combine(AppContext.BaseDirectory, "Fixtures", "StatementScrub", file)); + Assert.Contains("S3cret-fixture", xml); + + string? result = SensitiveStatements.Xml(xml); + + Assert.NotNull(result); + Assert.DoesNotContain("S3cret-fixture", result); + XDocument.Parse(result!); + } + + [Fact] + public void ACapturedPreparedPlanKeepsItsStatementText() + { + string xml = File.ReadAllText( + Path.Combine(AppContext.BaseDirectory, "Fixtures", "StatementScrub", "autoparam_update_prepared.xml")); + string original = (string)XDocument.Parse(xml).Descendants().First(e => e.Attribute("StatementText") != null) + .Attribute("StatementText")!; + + string? result = SensitiveStatements.Xml(xml); + + string kept = (string)XDocument.Parse(result!).Descendants().First(e => e.Attribute("StatementText") != null) + .Attribute("StatementText")!; + Assert.Equal(original, kept); + } + + [Fact] + public void XmlOnAPlanThatNamesNothingReturnsTheSameInstance_AndOnNullOrEmptyTheInput() + { + string plain = ""; + + Assert.Same(plain, SensitiveStatements.Xml(plain)); + Assert.Null(SensitiveStatements.Xml(null)); + Assert.Same(string.Empty, SensitiveStatements.Xml(string.Empty)); + } + + [Fact] + public void JsonOnANonJsonSentenceJudgesItWhole_AndRefusesAnUnreadableOpener() + { + Assert.Equal(Marker, SensitiveStatements.Json("error in " + StatementScrubCanary.CanaryStatement)); + Assert.Equal(SensitiveStatements.JsonRefusal, SensitiveStatements.Json("{\"q\": \"" + StatementScrubCanary.CanaryStatement)); + Assert.Same("plain words", SensitiveStatements.Json("plain words")); + } +} diff --git a/Darling/Darling.Tests/SensitiveStatementRawCleanTests.cs b/Darling/Darling.Tests/SensitiveStatementRawCleanTests.cs new file mode 100644 index 0000000000..697a606119 --- /dev/null +++ b/Darling/Darling.Tests/SensitiveStatementRawCleanTests.cs @@ -0,0 +1,248 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Net; +using PerformanceMonitor.Alerting; +using PerformanceMonitor.Common; +using PerformanceMonitor.Notifications; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5477: a document the linear-time pre-check clears as raw text skips the walk (two parses and a judge call per +/// value). The clear must be exact, so every document here is run through the walk itself +/// (XmlCore with the production judge, which never takes the shortcut) and the answers must match; the +/// documents that must NOT take the shortcut are the ones where raw text and decoded values differ. +/// +public class SensitiveStatementRawCleanTests +{ + private const string Secret = "CREATE LOGIN [canary_raw] WITH PASSWORD = N'S3cret-canary-raw'"; + private const string Plain = "SELECT canary_plain_raw FROM dbo.t WHERE c = 1"; + + /// A declared variable whose type length runs across a CR LF: the reader turns the pair into one character, so the + /// value it hands the judge is a character shorter than the raw text. + private const string LineEndStatement = "DECLARE @pwd nvarchar(\r\n1234567890123456789) = N'placeholder-value'"; + + private const string TwoLineEndStatement = "DECLARE @secret varchar(\r\n\r\n123456789012345678) = 'placeholder-value'"; + + private static bool Named(string value) => + SensitiveStatements.Names(value) + || (value.Contains('&', StringComparison.Ordinal) && SensitiveStatements.Names(WebUtility.HtmlDecode(value))); + + /// What the walk answers, with no shortcut in front of it. + private static string? Walk(string xml) => + SensitiveStatements.XmlCore(xml, Named, static () => false, SensitiveStatements.PlaceholderText); + + private static string? Filtered(string xml, out bool tookTheShortcut) + { + var before = SensitiveStatements.RawCleanClears; + var result = SensitiveStatements.Xml(xml, new SensitiveStatements.JudgeBudget(SensitiveStatements.ReadBudget)); + tookTheShortcut = SensitiveStatements.RawCleanClears != before; + return result; + } + + [Fact] + public void ACleanDocument_IsClearedWithoutAWalk_AndComesBackAsTheSameInstance() + { + var xml = "" + Plain + "more text"; + + var result = Filtered(xml, out var tookTheShortcut); + + Assert.True(tookTheShortcut); + Assert.Same(xml, result); + Assert.Same(xml, Walk(xml)); + } + + [Fact] + public void TheClear_IsChargedToTheBudget() + { + var budget = new SensitiveStatements.JudgeBudget(SensitiveStatements.ReadBudget); + + SensitiveStatements.Xml("" + new string('x', 400_000) + "", budget); + + Assert.True(budget.Elapsed > TimeSpan.Zero); + } + + public static TheoryData DocumentsThatMustBeWalked() => new() + { + // A named value: the walk replaces it. + "" + Secret + "" + Plain + "", + // A named value in an attribute. + "" + Plain + "", + // A named value in a CDATA section. + "", + // The secret spelled with a character reference: only decoding shows it, so it is no raw-text clear. + "CREATE LOGIN x WITH PASSWORD = N'S3cret'", + // Entity-encoded quote around the literal. + "password = 'hunter2'", + // A keyword spelled with a character reference: the raw text never holds the word, only the decoded value does. + "password = 'placeholder-value'", + "", + // A line comment in an attribute value: the reader turns the line break into a space, so the comment runs + // on to the literal in the value and not in the raw text. + "" + Plain + "", + "", + // An XML comment holds a double hyphen: no shortcut, and the comment is judged by the walk. + "" + Plain + "", + // An auto-parameter token: the walk judges the statement with its parameter value put back. + "" + + "", + // An element-form ParameterizedText. + "select 1", + // Not well formed, with and without a secret. + "" + Plain, + "" + Secret, + "just some text, not xml", + "pwd = 'x' and not xml", + // A carriage return in a text node, an attribute, a CDATA section, with two pairs, and in a document that is not + // well formed: the reader shortens each CR LF, so the raw text is no stand-in for the value. + "" + LineEndStatement + "", + "", + "", + "" + TwoLineEndStatement + "", + "" + LineEndStatement, + "", + }; + + [Theory] + [MemberData(nameof(DocumentsThatMustBeWalked))] + public void EveryDocument_GetsTheWalksAnswer(string xml) + { + var expected = Walk(xml); + + var actual = Filtered(xml, out _); + + Assert.Equal(expected, actual); + } + + [Theory] + [InlineData("CREATE LOGIN x WITH PASSWORD = N'S3cret'")] + [InlineData("password = 'hunter2'")] + [InlineData("password = 'placeholder-value'")] + [InlineData("")] + [InlineData("x")] + [InlineData("x")] + [InlineData("")] + [InlineData("select 1")] + [InlineData("DECLARE @pwd nvarchar(\r\n1234567890123456789) = N'placeholder-value'")] + [InlineData("")] + [InlineData("")] + [InlineData("DECLARE @secret varchar(\r\n\r\n123456789012345678) = 'placeholder-value'")] + [InlineData("DECLARE @pwd nvarchar(\r\n1234567890123456789) = N'placeholder-value'")] + public void ADocumentWhereRawTextAndValuesDiffer_IsNeverClearedAsRawText(string xml) + { + Filtered(xml, out var tookTheShortcut); + + Assert.False(tookTheShortcut); + } + + [Theory] + [InlineData(LineEndStatement)] + [InlineData(TwoLineEndStatement)] + public void ALineEndInsideATypeLength_StillGetsTheStatementWithheld(string statement) + { + var xml = "" + statement + ""; + + var result = Filtered(xml, out var tookTheShortcut); + + Assert.False(tookTheShortcut); + Assert.DoesNotContain("placeholder-value", result, StringComparison.Ordinal); + Assert.Equal(Walk(xml), result); + } + + [Fact] + public void AJsonValueHoldingAnXmlDocumentWithALineEnd_GetsTheWalksAnswer() + { + var json = "{\"plan\":\"DECLARE @pwd nvarchar(\\r\\n1234567890123456789) = N'placeholder-value'\"}"; + + var result = SensitiveStatements.Json(json, new SensitiveStatements.JudgeBudget(SensitiveStatements.ReadBudget)); + + Assert.DoesNotContain("placeholder-value", result, StringComparison.Ordinal); + Assert.Contains(SensitiveStatements.PlaceholderText, result, StringComparison.Ordinal); + } + + [Fact] + public void AnAlertAttachmentWithALineEndInAStatement_GetsTheStatementWithheld() + { + var context = new AlertContext + { + AttachmentXml = "" + + LineEndStatement + + "", + AttachmentFileName = "blocked_process_report.xml", + }; + + var filtered = AlertStatementFilter.Apply(context); + + Assert.NotNull(filtered?.AttachmentXml); + Assert.DoesNotContain("placeholder-value", filtered!.AttachmentXml, StringComparison.Ordinal); + } + + /// The pre-check stands in for the walk only while the pattern reads a value the way it reads the raw text. A + /// counted repetition, or a class that tells a tab, a line feed and a space apart, outside the one place that + /// already sends the document to the walk, could break that, so appending either to the pattern must be a + /// deliberate act with a new exit in the shortcut. + [Fact] + public void ThePattern_HasNoCountedRepetitionBeyondTheOneTheShortcutCovers_AndNoClassThatSeparatesWhitespace() + { + var counted = System.Text.RegularExpressions.Regex.Matches(SensitiveStatements.Pattern, @"\{\d+(,\d*)?\}"); + Assert.Single(counted); + Assert.Equal("{0,20}", counted[0].Value); + + // The line-comment alternative is the one place a control character class is allowed: the shortcut exits on "--". + var rest = SensitiveStatements.Pattern.Replace("--[^[:cntrl:]]*", string.Empty, StringComparison.Ordinal); + foreach (var banned in new[] { "[:cntrl:]", "[:blank:]", "[:print:]", "[:graph:]", "\t", "\n", "\r", " " }) + { + Assert.DoesNotContain(banned, rest, StringComparison.Ordinal); + } + } + + [Fact] + public void ADocumentThatNamesAValue_IsNotCleared() + { + var xml = "" + Plain + "" + Secret + ""; + + var result = Filtered(xml, out var tookTheShortcut); + + Assert.False(tookTheShortcut); + Assert.DoesNotContain("S3cret", result, StringComparison.Ordinal); + } + + [Fact] + public void ABudgetWithAFakeJudge_IsNeverShortCircuited() + { + var calls = 0; + var budget = new SensitiveStatements.JudgeBudget( + SensitiveStatements.ReadBudget, + _ => + { + calls++; + return SensitiveStatements.Verdict.Named; + }); + var before = SensitiveStatements.RawCleanClears; + + var result = SensitiveStatements.Xml("" + Plain + "", budget); + + Assert.Equal(before, SensitiveStatements.RawCleanClears); + Assert.True(calls > 0); + Assert.NotEqual("" + Plain + "", result); + } + + [Fact] + public void ASpentBudget_WithholdsTheDocument_AsBefore() + { + var budget = new SensitiveStatements.JudgeBudget(SensitiveStatements.ReadBudget); + budget.AddElapsed(TimeSpan.FromSeconds(60)); + + var result = SensitiveStatements.Xml("" + Plain + "", budget); + + Assert.Equal(SensitiveStatements.PlaceholderText, result); + } +} diff --git a/Darling/Darling.Tests/SensitiveStatementSessionTests.cs b/Darling/Darling.Tests/SensitiveStatementSessionTests.cs new file mode 100644 index 0000000000..041c7808fa --- /dev/null +++ b/Darling/Darling.Tests/SensitiveStatementSessionTests.cs @@ -0,0 +1,503 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348, #5320 (PR B, lane R1a): the collection-time statement filter session. The budget tests use a fake judge and +/// a fake clock, so no test waits on real time; the identity and canary tests use the real judge. +/// +public sealed class SensitiveStatementSessionTests +{ + private const string Marker = SensitiveStatements.PlaceholderText; + + private static readonly TimeSpan Fifteen = TimeSpan.FromSeconds(15); + + /// A judge that counts its calls, charges a fixed cost to a fake clock, and answers by a rule. + private sealed class FakeJudge(TimeSpan cost, Func? rule = null) + { + public TimeSpan Now { get; private set; } + + public List Seen { get; } = new(); + + public Func Clock => () => Now; + + public SensitiveStatements.Verdict Judge(string text) + { + Seen.Add(text); + Now += cost; + return rule?.Invoke(text) ?? SensitiveStatements.Verdict.Clean; + } + } + + private static SensitiveStatements.Session NewSession( + FakeJudge fake, + SensitiveStatements.TimedOutMemo? memo = null, + TimeSpan? limit = null) => + new(fake.Judge, fake.Clock, limit ?? Fifteen, memo ?? new SensitiveStatements.TimedOutMemo()); + + // ---- budget -------------------------------------------------------------------------------------------- + + [Fact] + public void ValuesAfterTheBudgetIsSpentAreUnjudgedMarkers() + { + var fake = new FakeJudge(TimeSpan.FromSeconds(8)); + var session = NewSession(fake); + + Assert.Equal("a", session.Text("a")); + Assert.Equal("b", session.Text("b")); + Assert.True(session.Spent); + + var third = session.Text("c"); + + Assert.Equal(Marker, third); + Assert.Equal(2, fake.Seen.Count); + Assert.Equal(1, session.Unjudged); + Assert.Equal(0, session.Named); + } + + [Fact] + public void TryTextReturnsFalseForAValueLeftUnjudgedByASpentBudget_AndTrueForANamedOne() + { + var fake = new FakeJudge( + TimeSpan.FromSeconds(8), + text => text == "named" ? SensitiveStatements.Verdict.Named : SensitiveStatements.Verdict.Clean); + var session = NewSession(fake); + + Assert.True(session.TryText("named", out var namedResult)); + Assert.Equal(Marker, namedResult); + Assert.True(session.TryText("ok", out _)); + + Assert.False(session.TryText("late", out var lateResult)); + Assert.Equal(Marker, lateResult); + Assert.Equal(1, session.Named); + Assert.Equal(1, session.Unjudged); + } + + [Fact] + public void AnUnjudgedValueIsNotRememberedSoALaterSessionJudgesIt() + { + var spent = new FakeJudge(TimeSpan.FromSeconds(20)); + var first = NewSession(spent); + first.Text("x"); + Assert.False(first.TryText("y", out _)); + + var fresh = new FakeJudge(TimeSpan.FromMilliseconds(1)); + var second = NewSession(fresh); + + Assert.True(second.TryText("y", out var result)); + Assert.Equal("y", result); + Assert.Equal(new[] { "y" }, fresh.Seen); + } + + [Fact] + public void ABudgetCrossedMidXmlGivesTheWholeMarker_AndTheNextSessionGivesTheScrubbedPlan() + { + // r2 Q5. The fake judge calls the real judge and charges 5 s a value, so the plan runs out of budget + // part way through. + var plan = StatementScrubCanary.CanaryPlan(); + var slow = new FakeJudge(TimeSpan.FromSeconds(5), SensitiveStatements.Judge); + var first = NewSession(slow); + + Assert.False(first.TryXml(plan, out var withheldWhole)); + Assert.Equal(Marker, withheldWhole); + Assert.Equal(1, first.Unjudged); + + var next = new SensitiveStatements.Session(); + Assert.True(next.TryXml(plan, out var scrubbed)); + + Assert.NotEqual(Marker, scrubbed); + Assert.NotEqual(plan, scrubbed); + Assert.Contains(Marker, scrubbed, StringComparison.Ordinal); + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.DoesNotContain(needle, scrubbed, StringComparison.Ordinal); + } + + Assert.Contains("canary_plain_ssf", scrubbed, StringComparison.Ordinal); + Assert.Equal(0, next.Unjudged); + } + + [Fact] + public void ABudgetAlreadySpentGivesTheWholeMarkerForADocumentThatNeedsJudging() + { + var fake = new FakeJudge(TimeSpan.FromSeconds(20)); + var session = NewSession(fake); + session.Text("x"); + + Assert.False(session.TryXml("v1v2", out var result)); + Assert.Equal(Marker, result); + Assert.Equal(1, session.Unjudged); + } + + // ---- failure and identity ------------------------------------------------------------------------------ + + [Fact] + public void AThrowingJudgeGivesTheMarker_ForTextAndForXml_AndNeverThrows() + { + SensitiveStatements.Session NewThrowing() => + new(_ => throw new InvalidOperationException("judge failed"), null, Fifteen, new SensitiveStatements.TimedOutMemo()); + + Assert.Equal(Marker, NewThrowing().Text("SELECT 1")); + // A throwing judge names every value, so the document keeps its shape and no value survives. + var xml = NewThrowing().Xml("SELECT 1"); + Assert.Contains(Marker, xml, StringComparison.Ordinal); + Assert.DoesNotContain("SELECT 1", xml, StringComparison.Ordinal); + } + + [Fact] + public void PlainTextAndPlainXmlComeBackAsTheSameInstance() + { + foreach (var session in new[] { new SensitiveStatements.Session(), NewSession(new FakeJudge(TimeSpan.Zero)) }) + { + var text = new string(StatementScrubCanary.PlainStatement.ToCharArray()); + Assert.Same(text, session.Text(text)); + + var withAmpersand = new string("SELECT 1 WHERE a = 1 AND b < 2".ToCharArray()); + Assert.Same(withAmpersand, session.Text(withAmpersand)); + + var xml = new string("SELECT 1".ToCharArray()); + Assert.Same(xml, session.Xml(xml)); + + Assert.Null(session.Text(null)); + Assert.Same(string.Empty, session.Text(string.Empty)); + Assert.Null(session.Xml(null)); + Assert.Same(string.Empty, session.Xml(string.Empty)); + } + } + + [Fact] + public void ANamedValueIsTheMarker_IncludingOneNamedOnlyAfterHtmlDecoding() + { + var session = new SensitiveStatements.Session(); + + Assert.Equal(Marker, session.Text(StatementScrubCanary.CanaryStatement)); + Assert.Equal(Marker, session.Text("CREATE LOGIN [a] WITH PASSWORD = N'S3cret'")); + Assert.Equal(2, session.Named); + } + + [Fact] + public void TheCanaryPlanScrubbedInTwoSessionsGivesEqualStringsAndEqualHashes() + { + var plan = StatementScrubCanary.CanaryPlan(); + + var first = new SensitiveStatements.Session().Xml(plan)!; + var second = new SensitiveStatements.Session().Xml(plan)!; + + Assert.Equal(first, second); + Assert.Equal(SHA256.HashData(Encoding.UTF8.GetBytes(first)), SHA256.HashData(Encoding.UTF8.GetBytes(second))); + Assert.NotEqual(plan, first); + Assert.Contains(Marker, first, StringComparison.Ordinal); + foreach (var needle in StatementScrubCanary.SecretNeedles.Take(3)) + { + Assert.DoesNotContain(needle, first, StringComparison.Ordinal); + } + + Assert.Contains("canary_plain_ssf", first, StringComparison.Ordinal); + Assert.Contains("param-canary-ssf", first, StringComparison.Ordinal); + } + + // ---- memo ---------------------------------------------------------------------------------------------- + + [Fact] + public void TheSessionMemoJudgesARepeatedValueOnce() + { + var fake = new FakeJudge(TimeSpan.FromMilliseconds(1)); + var session = NewSession(fake); + + for (var i = 0; i < 5; i++) + { + Assert.Equal("SELECT 1", session.Text(new string("SELECT 1".ToCharArray()))); + } + + Assert.Single(fake.Seen); + Assert.Equal(4, session.MemoHits); + Assert.Equal(5, session.Values); + } + + [Fact] + public void TheSessionMemoKeepsANamedVerdictAndCountsEachValueWithheld() + { + var fake = new FakeJudge( + TimeSpan.FromMilliseconds(1), + _ => SensitiveStatements.Verdict.Named); + var session = NewSession(fake); + + Assert.Equal(Marker, session.Text("s")); + Assert.Equal(Marker, session.Text("s")); + + Assert.Single(fake.Seen); + Assert.Equal(2, session.Named); + } + + [Fact] + public void ThereIsNoTimeoutCount_ASessionWithTenTimeoutsUnderTheBudgetStillJudgesTheEleventhValue() + { + var fake = new FakeJudge( + TimeSpan.FromMilliseconds(250), + text => text.StartsWith("slow", StringComparison.Ordinal) + ? SensitiveStatements.Verdict.TimedOut + : SensitiveStatements.Verdict.Clean); + var session = NewSession(fake); + + for (var i = 0; i < 10; i++) + { + Assert.Equal(Marker, session.Text("slow " + i)); + } + + Assert.False(session.Spent); + var value = "SELECT 11"; + Assert.True(session.TryText(value, out var result)); + Assert.Same(value, result); + Assert.Equal(11, fake.Seen.Count); + Assert.Equal(10, session.TimedOut); + Assert.Equal(0, session.Unjudged); + } + + [Fact] + public void ASecondSessionWithTheSameSlowValueSpendsNoTimeOnIt() + { + // r2 M-G, A-5: a value whose match timed out once is remembered process-wide (the memo is shared here + // the way the public sessions share theirs). + var memo = new SensitiveStatements.TimedOutMemo(); + var slow = "slow-" + Guid.NewGuid(); + var firstFake = new FakeJudge(TimeSpan.FromMilliseconds(250), _ => SensitiveStatements.Verdict.TimedOut); + var first = NewSession(firstFake, memo); + Assert.Equal(Marker, first.Text(slow)); + Assert.Equal(250, first.ElapsedMs); + Assert.Equal(1, memo.Count); + + var secondFake = new FakeJudge(TimeSpan.FromMilliseconds(250), _ => SensitiveStatements.Verdict.TimedOut); + var second = NewSession(secondFake, memo); + + Assert.Equal(Marker, second.Text(slow)); + Assert.Empty(secondFake.Seen); + Assert.Equal(0, second.ElapsedMs); + Assert.Equal(1, second.TimedOutMemoHits); + Assert.Equal(0, second.TimedOut); + Assert.Equal(1, second.Named); + } + + [Fact] + public void OnlyTimedOutVerdictsAreRememberedProcessWide() + { + var memo = new SensitiveStatements.TimedOutMemo(); + var fake = new FakeJudge( + TimeSpan.FromMilliseconds(1), + text => text == "named" ? SensitiveStatements.Verdict.Named : SensitiveStatements.Verdict.Clean); + + var session = NewSession(fake, memo); + session.Text("named"); + session.Text("clean"); + + Assert.Equal(0, memo.Count); + } + + [Fact] + public void TheProcessWideMemoIsClearedWhenFull_AndStaysUnderItsLimit() + { + var memo = new SensitiveStatements.TimedOutMemo(); + for (var i = 0; i < 5000; i++) + { + memo.Add("v" + i); + } + + Assert.InRange(memo.Count, 1, 4096); + Assert.True(memo.Contains("v4999")); + Assert.False(memo.Contains("v0")); + } + + [Fact] + public void ThePublicSessionSharesOneProcessWideMemo() + { + var value = "shared-" + Guid.NewGuid(); + SensitiveStatements.TimedOutMemo.Shared.Add(value); + + var session = new SensitiveStatements.Session(); + + Assert.Equal(Marker, session.Text(value)); + Assert.Equal(1, session.TimedOutMemoHits); + Assert.Equal(0, session.ElapsedMs); + } + + // ---- the cycle's measurements -------------------------------------------------------------------------- + + private static CollectorContext NewContext() => new() + { + ServerId = 1, + ServerName = "target-a", + CollectionTime = new DateTime(2026, 10, 6, 12, 0, 0, DateTimeKind.Utc), + Deltas = new CollectorDeltaCalculator(), + }; + + [Fact] + public void ACycleThatJudgedNothingCarriesNoScrubMeasurements() + { + var context = NewContext(); + var session = context.BeginStatementScrub(); + session.Text(null); + session.Text(string.Empty); + + Assert.Empty(context.Measurements); + Assert.Empty(NewContext().Measurements); + } + + [Fact] + public void ACycleThatJudgedAValueCarriesTheFourMeasurements_AndASecondReadAddsNothingTwice() + { + var context = NewContext(); + var first = context.BeginStatementScrub(); + first.Text(StatementScrubCanary.CanaryStatement); + first.Text(StatementScrubCanary.PlainStatement); + var second = context.BeginStatementScrub(); + second.Text(StatementScrubCanary.CanaryStatement.Replace("canary_ssf", "other_ssf", StringComparison.Ordinal)); + + long Get(string label) => context.Measurements.Single(m => m.Label == label).Value; + + Assert.Equal(2, Get(CollectorContext.StatementScrubNamedMeasurement)); + Assert.Equal(0, Get(CollectorContext.StatementScrubTimeoutsMeasurement)); + Assert.Equal(0, Get(CollectorContext.StatementScrubUnjudgedMeasurement)); + Assert.True(Get(CollectorContext.StatementScrubMsMeasurement) >= 0); + Assert.Equal(4, context.Measurements.Count); + + // Reading again adds only what a session did since. + Assert.Equal(2, Get(CollectorContext.StatementScrubNamedMeasurement)); + second.Text("CREATE LOGIN [again] WITH PASSWORD = N'x'"); + Assert.Equal(3, Get(CollectorContext.StatementScrubNamedMeasurement)); + Assert.Equal(4, context.Measurements.Count); + } + + [Fact] + public void TheScrubMeasurementLabelsAreLegalAndDistinct() + { + var labels = new[] + { + CollectorContext.StatementScrubNamedMeasurement, + CollectorContext.StatementScrubTimeoutsMeasurement, + CollectorContext.StatementScrubUnjudgedMeasurement, + CollectorContext.StatementScrubMsMeasurement, + }; + + Assert.All(labels, label => Assert.True(CollectorMeasurementNote.IsValidLabel(label), label)); + Assert.Equal(4, labels.Distinct(StringComparer.Ordinal).Count()); + } + + // ---- measured (recorded, not compared) ----------------------------------------------------------------- + + private static string Plan(int targetBytes) + { + var node = "" + + "" + + ""; + var builder = new StringBuilder(targetBytes + 1024); + builder.Append("") + .Append("") + .Append(""); + for (var i = 0; builder.Length < targetBytes; i++) + { + builder.AppendFormat(node, i); + } + + return builder.Append("").ToString(); + } + + private static void Record(string message) => TestContext.Current.SendDiagnosticMessage(message); + + [Fact] + public void SessionTextThroughputOnTenThousandGeneratedStatementsIsRecorded() + { + var fragments = new[] + { + "SELECT a.col_a, b.col_b FROM dbo.TableA AS a JOIN dbo.TableB AS b ON a.id = b.a_id ", + "WHERE a.created >= @from AND a.created < @to AND b.status IN (1, 2, 3) ", + "INSERT INTO dbo.Audit (kind, detail) VALUES (N'x', @d); UPDATE dbo.T SET c = c + 1 WHERE k = @k; ", + "GROUP BY a.col_a ORDER BY COUNT(*) DESC OPTION (RECOMPILE) ", + }; + var random = new Random(5320); + var statements = new string[10_000]; + long chars = 0; + for (var i = 0; i < statements.Length; i++) + { + var length = random.Next(200, 4001); + var builder = new StringBuilder(length + 120); + while (builder.Length < length) + { + builder.Append(fragments[random.Next(fragments.Length)]); + } + + builder.Append(" /* ").Append(i).Append(" */"); + statements[i] = builder.ToString(); + chars += statements[i].Length; + } + + var session = new SensitiveStatements.Session(); + var watch = Stopwatch.StartNew(); + foreach (var statement in statements) + { + Assert.Same(statement, session.Text(statement)); + } + + watch.Stop(); + var megabytes = chars / 1_048_576.0; + Record($"Session.Text over 10,000 statements: {megabytes:F1} MB in {watch.ElapsedMilliseconds} ms = {megabytes / watch.Elapsed.TotalSeconds:F1} MB/s, named={session.Named}, unjudged={session.Unjudged}"); + Assert.Equal(0, session.Named); + Assert.Equal(0, session.Unjudged); + } + + [Theory] + [InlineData(512 * 1024)] + [InlineData(27 * 1024 * 1024)] + public void SessionXmlThroughputOnAPlanIsRecorded(int bytes) + { + var plan = Plan(bytes); + var session = new SensitiveStatements.Session(); + + var watch = Stopwatch.StartNew(); + var result = session.Xml(plan); + watch.Stop(); + + var megabytes = plan.Length / 1_048_576.0; + Record($"Session.Xml on a {megabytes:F1} MB plan: {watch.ElapsedMilliseconds} ms = {megabytes / watch.Elapsed.TotalSeconds:F1} MB/s, unjudged={session.Unjudged}, spent={session.Spent}"); + Assert.Same(plan, result); + } + + [Fact] + public void TheMemoHitRateOnFiftyThousandQueryStoreShapedRowsWithTwoThousandDistinctTextsIsRecorded() + { + var random = new Random(4348); + var distinct = new string[2_000]; + for (var i = 0; i < distinct.Length; i++) + { + distinct[i] = "SELECT c" + i + " FROM dbo.t" + i + " WHERE k = @k" + new string('x', random.Next(100, 2000)); + } + + var session = new SensitiveStatements.Session(); + var watch = Stopwatch.StartNew(); + for (var row = 0; row < 50_000; row++) + { + // A fresh instance per row, as a reader materializes one. + session.Text(new string(distinct[random.Next(distinct.Length)].AsSpan())); + } + + watch.Stop(); + var rate = 100.0 * session.MemoHits / session.Values; + Record($"Memo over 50,000 rows with 2,000 distinct texts: hits={session.MemoHits} of {session.Values} = {rate:F1}% in {watch.ElapsedMilliseconds} ms"); + Assert.Equal(50_000, session.Values); + Assert.True(session.MemoHits >= 50_000 - 2_000, $"hits {session.MemoHits}"); + } +} diff --git a/Darling/Darling.Tests/SensitiveStatementXmlTests.cs b/Darling/Darling.Tests/SensitiveStatementXmlTests.cs new file mode 100644 index 0000000000..ec9dfc9427 --- /dev/null +++ b/Darling/Darling.Tests/SensitiveStatementXmlTests.cs @@ -0,0 +1,636 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.IO; +using System.Linq; +using System.Text; +using System.Xml.Linq; +using PerformanceMonitor.Common; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348: the XML walk of the statement filter (SensitiveStatements.XmlCore). The judge here is a plain +/// substring test on CANARY, so these pin the mechanics (what is judged, what is exempt, what a withheld +/// statement writes, the failure rules and the budget), not the pattern. +/// +public class SensitiveStatementXmlTests +{ + private const string P = "<

>"; + + ///

The placeholder as it sits in serialized text (it holds angle brackets, so it is escaped). + private const string PText = "<<P>>"; + private static readonly XNamespace Ns = "http://schemas.microsoft.com/sqlserver/2004/07/showplan"; + + private static bool Canary(string s) => s.Contains("CANARY", StringComparison.OrdinalIgnoreCase); + + private static string Fixture(string name) => + File.ReadAllText(Path.Combine(AppContext.BaseDirectory, "Fixtures", "StatementScrub", name)); + + private static string? Run(string? xml, Func? named = null, Func? spent = null, + Action? charge = null, int max = int.MaxValue) => + SensitiveStatements.XmlCore(xml, named ?? Canary, spent ?? (() => false), P, charge, max); + + /// Every element and attribute name, in document order: the shape a rewrite must keep. + private static List Shape(string xml) + { + var shape = new List(); + foreach (var e in XDocument.Parse(xml).Descendants()) + { + shape.Add(e.Name.LocalName + "[" + string.Join(",", e.Attributes().Select(a => a.Name.LocalName)) + "]"); + } + return shape; + } + + private static List Stmts(string xml) => XDocument.Parse(xml).Descendants(Ns + "StmtSimple").ToList(); + + private static string?[] Values(XElement stmt, string attribute) => + stmt.Descendants().Select(e => (string?)e.Attribute(attribute)).Where(v => v is not null).ToArray(); + + // ── the RED: a named statement withholds its own values, and only its own ── + + [Fact] + public void NamedStatementWithholdsItsParametersAndLiterals_AndTheNextStatementIsJudgedOnItsOwn() + { + string xml = Fixture("two_statements.xml"); + + string? result = Run(xml); + + Assert.NotNull(result); + Assert.NotSame(xml, result); + Assert.Equal(Shape(xml), Shape(result!)); + + var s = Stmts(result!); + Assert.Equal(2, s.Count); + + // statement 1 (StatementText holds CANARY): the whole H2 list is the caller's placeholder + Assert.All(Values(s[0], "ParameterCompiledValue"), v => Assert.Equal(P, v)); + Assert.All(Values(s[0], "ParameterRuntimeValue"), v => Assert.Equal(P, v)); + Assert.All(Values(s[0], "ScalarString"), v => Assert.Equal(P, v)); + Assert.All(Values(s[0], "ConstValue"), v => Assert.Equal(P, v)); + Assert.Equal(P, (string?)s[0].Attribute("ParameterizedText")); + Assert.Equal(P, (string?)s[0].Attribute("StatementText")); + // ... including a numeric parameter value, which names nothing on its own + Assert.DoesNotContain("(7)", result); + + // statement 2: ParameterCompiledValue holding CANARY is exempt and KEPT; ScalarString and ConstValue holding + // CANARY are judged on their own and become the placeholder; the plain runtime value stays + Assert.Equal("N'CANARY-keep'", Values(s[1], "ParameterCompiledValue").Single()); + Assert.Equal("N'plain-runtime'", Values(s[1], "ParameterRuntimeValue").Single()); + Assert.Contains("CANARY-keep", result); + Assert.DoesNotContain("CANARY-two", result); + Assert.Contains(Values(s[1], "ScalarString"), v => v == P); + Assert.Equal(P, Values(s[1], "ConstValue").Single()); + Assert.Equal("SELECT name FROM dbo.people WHERE name = @n", (string?)s[1].Attribute("StatementText")); + + // the statement-1 text itself never survives, and the document still parses (Parse above) + Assert.DoesNotContain("CANARY-one", result); + Assert.DoesNotContain("CANARY-compiled", result); + Assert.DoesNotContain("CANARY-runtime", result); + } + + [Fact] + public void ThePlaceholderIsTheParameter_NotAConstantOfTheFile() + { + string? result = SensitiveStatements.XmlCore(Fixture("two_statements.xml"), Canary, () => false, "[[other]]"); + + Assert.Contains("[[other]]", result); + Assert.DoesNotContain(P, result); + } + + [Fact] + public void NoHit_ReturnsTheSameInstance() + { + string xml = Fixture("two_statements.xml").Replace("CANARY", "plain", StringComparison.Ordinal); + + Assert.Same(xml, Run(xml)); + Assert.Null(Run(null)); + Assert.Same(string.Empty, Run(string.Empty)); + } + + [Fact] + public void ParameterValuesAreExemptOutsideAScope_EvenWhenTheyHoldTheName() + { + const string xml = ""; + + Assert.Same(xml, Run(xml)); + } + + // ── scope opening and the L-J rule ── + + [Fact] + public void AParameterizedTextAttributeHoldingTheName_OpensTheScope_AndInsideItAnyQuotedValueIsWithheld() + { + const string xml = + "" + + "" + + "" + + "" + + ""; + + string result = Run("" + xml + "")!; + var stmts = XDocument.Parse(result).Descendants("StmtSimple").ToList(); + + Assert.Equal(P, (string?)stmts[0].Attribute("ParameterizedText")); + Assert.Equal("UPDATE t SET c = @1", (string?)stmts[0].Attribute("StatementText")); + Assert.Equal("0.5", (string?)stmts[0].Attribute("StatementSubTreeCost")); + var convert0 = stmts[0].Descendants("PlanAffectingConvert").Single(); + Assert.Equal(P, (string?)convert0.Attribute("Expression")); + Assert.Equal("n", (string?)convert0.Attribute("Note")); + Assert.Equal(P, (string?)stmts[0].Descendants("ColumnReference").Single().Attribute("ParameterCompiledValue")); + Assert.Equal("@1", (string?)stmts[0].Descendants("ColumnReference").Single().Attribute("Column")); + + // the same attribute outside any scope is kept + Assert.Equal("x = 'lit-ssf'", (string?)stmts[1].Descendants("PlanAffectingConvert").Single().Attribute("Expression")); + Assert.Equal("(8)", (string?)stmts[1].Descendants("ColumnReference").Single().Attribute("ParameterCompiledValue")); + } + + [Fact] + public void TheScopeEndsAtTheStatementsEndTag_NestedStatementsAreInsideIt() + { + const string xml = + "" + + ""; + + string result = Run(xml)!; + var doc = XDocument.Parse(result); + + Assert.Equal(P, (string?)doc.Descendants("StmtSimple").Single().Attribute("Lit")); + Assert.Equal(P, (string?)doc.Descendants("Cond").Single().Attribute("Lit")); + Assert.Equal("'c'", (string?)doc.Descendants("Other").Single().Attribute("Lit")); + Assert.Equal("(1)", (string?)doc.Descendants("Other").Single().Attribute("ParameterRuntimeValue")); + } + + [Fact] + public void AnEmptyStatementElementOpensTheScopeOnlyForItsOwnAttributes() + { + const string xml = ""; + + var doc = XDocument.Parse(Run(xml)!); + + Assert.Equal(P, (string?)doc.Descendants("StmtUseDb").Single().Attribute("Lit")); + Assert.Equal("1", (string?)doc.Descendants("StmtUseDb").Single().Attribute("Cost")); + Assert.Equal("'c'", (string?)doc.Descendants("Other").Single().Attribute("Lit")); + } + + [Fact] + public void InsideAScope_ElementFormsOfTheListAndQuotedTextAreWithheld() + { + const string xml = + "(@1 int)select 1" + + "it's here7 rowsit's fine"; + + var doc = XDocument.Parse(Run(xml)!); + + Assert.Equal(P, doc.Descendants("ParameterizedText").Single().Value); + Assert.Equal(P, doc.Descendants("Note").Single().Value); + Assert.Equal("7 rows", doc.Descendants("Plain").Single().Value); + Assert.Equal("it's fine", doc.Descendants("After").Single().Value); + } + + [Fact] + public void InsideAScope_AValueHoldingABinaryLiteralIsWithheld_ExceptTheHashAndHandleAttributes() + { + const string xml = + "" + + "" + + "SELECT 1 WHERE h = 0xDEADBEEF" + + "0x05000100CD" + + "h = 0xDEADBEEF"; + + var doc = XDocument.Parse(Run(xml)!); + var stmt = doc.Descendants("StmtSimple").Single(); + + Assert.Equal(P, (string?)doc.Descendants("PlanAffectingConvert").First().Attribute("Expression")); + Assert.Equal("Seek Plan", (string?)doc.Descendants("PlanAffectingConvert").First().Attribute("ConvertIssue")); + Assert.Equal(P, doc.Descendants("RemoteQuery").First().Value); + Assert.Equal("index 0x is empty", (string?)doc.Descendants("Plain").Single().Attribute("Note")); + Assert.Equal("0x0A1B2C3D4E5F6071", (string?)stmt.Attribute("QueryHash")); + Assert.Equal("0x0A1B2C3D4E5F6072", (string?)stmt.Attribute("QueryPlanHash")); + Assert.Equal("0x09000000AB", (string?)stmt.Attribute("StatementSqlHandle")); + Assert.Equal("0x06000100AB", (string?)stmt.Attribute("ParameterizedPlanHandle")); + Assert.Equal("0x05000100CD", doc.Descendants("PlanHandle").Single().Value); + // outside a withheld scope the value is kept (the judge does not name it) + Assert.Equal("CONVERT_IMPLICIT(varbinary(64),[t].[h],0)=0x0200ABCD", + (string?)doc.Descendants("PlanAffectingConvert").Last().Attribute("Expression")); + Assert.Equal("h = 0xDEADBEEF", doc.Descendants("RemoteQuery").Last().Value); + } + + [Fact] + public void AnElementThatIsNotNamedStmtButCarriesAStatementText_OpensTheScope() + { + // The showplan schema names every statement element Stmt*; an element of another name that carries the + // statement attributes is read the same way, in both passes (the ordinals stay in step). + const string xml = + "7" + + ""; + + var doc = XDocument.Parse(Run(xml)!); + + Assert.Equal(P, (string?)doc.Descendants("Wrapper").Single().Attribute("StatementText")); + Assert.Equal(P, (string?)doc.Descendants("Wrapper").Single().Attribute("Lit")); + Assert.Equal(P, (string?)doc.Descendants("Inner").Single().Attribute("Lit")); + Assert.Equal(P, doc.Descendants("ConstValue").Single().Value); + Assert.Equal("'c'", (string?)doc.Descendants("StmtSimple").Single().Attribute("Lit")); + Assert.Equal("'d'", (string?)doc.Descendants("Other").Single().Attribute("Lit")); + } + + // ── other XML: reports, graphs, events ── + + [Fact] + public void ADecodedLineBreakInsideAnAttribute_IsJudgedDecoded() + { + const string xml = ""; + + string? result = Run(xml, s => s.Contains("CREATE\r\nLOGIN", StringComparison.Ordinal)); + + Assert.Contains("StatementText=\"" + PText + "\"", result); + Assert.Contains("Cost=\"1\"", result); + } + + [Fact] + public void AlineBreakCharacterInAValueSurvivesTheRewriteAsACharacterReference() + { + const string xml = ""; + + string result = Run(xml)!; + + Assert.Equal("x\r\ny", (string?)XDocument.Parse(result).Root!.Attribute("a")); + Assert.Equal(P, (string?)XDocument.Parse(result).Root!.Attribute("b")); + } + + [Fact] + public void ABlockedProcessReport_WithholdsOnlyTheNamedInputBuffer() + { + const string xml = + "" + + "UPDATE t SET c = 1" + + "UPDATE t SET c = 'CANARY'" + + "SELECT 1"; + + var doc = XDocument.Parse(Run(xml)!); + + Assert.Equal(P, doc.Descendants("inputbuf").First().Value); + Assert.Equal("SELECT 1", doc.Descendants("inputbuf").Last().Value); + Assert.Equal("UPDATE t SET c = 1", doc.Descendants("frame").Single().Value); + Assert.Equal("1", (string?)doc.Descendants("blocked-process-report").Single().Attribute("monitorLoop")); + } + + [Fact] + public void ADeadlockGraph_WithholdsTheNamedInputBufferAndFrameText() + { + const string xml = + "" + + "DECLARE @c nvarchar(10) = N'CANARY'" + + "EXEC s 'CANARY'" + + "SELECT 2SELECT 2" + + ""; + + var doc = XDocument.Parse(Run(xml)!); + var frames = doc.Descendants("frame").ToList(); + var bufs = doc.Descendants("inputbuf").ToList(); + + Assert.Equal(P, frames[0].Value); + Assert.Equal("SELECT 2", frames[1].Value); + Assert.Equal(P, bufs[0].Value); + Assert.Equal("SELECT 2", bufs[1].Value); + Assert.Equal("adhoc", (string?)frames[0].Attribute("procname")); + } + + [Fact] + public void ASystemHealthSqlTextAction_IsWithheldWhenNamed() + { + const string xml = + "102" + + "SELECT 'CANARY'" + + "5"; + + var doc = XDocument.Parse(Run(xml)!); + + Assert.Equal(P, doc.Descendants("action").First().Element("value")!.Value); + Assert.Equal("5", doc.Descendants("action").Last().Element("value")!.Value); + Assert.Equal("102", doc.Descendants("data").Single().Element("value")!.Value); + } + + [Fact] + public void ACdataSection_IsJudgedAndKeptAsCdata() + { + const string xml = "]]>]]>"; + + string result = Run(xml)!; + + Assert.Contains("]]>", result); + Assert.Contains("", result); + } + + // ── preservation in the rewrite ── + + [Fact] + public void TheRewriteKeepsTheDeclarationOnlyWhenOnePresent() + { + string withDecl = "\r\n"; + string without = ""; + + string r1 = Run(withDecl)!; + string r2 = Run(without)!; + + Assert.StartsWith("", r1); + Assert.DoesNotContain(""; + + string result = Run(xml)!; + var doc = XDocument.Parse(result); + + Assert.Equal("urn:one", doc.Root!.Name.NamespaceName); + Assert.Equal("urn:two", doc.Root.Element(XNamespace.Get("urn:two") + "item")!.Name.NamespaceName); + Assert.Contains("xmlns:p=\"urn:two\"", result); + Assert.Contains("", result); + Assert.Contains("", result); + Assert.Contains("", result); + Assert.Contains("", result); + Assert.Contains("", result); + Assert.Equal(P, (string?)doc.Root.Attribute("v")); + Assert.Equal("x", (string?)doc.Root.Attribute("CANARY")); + } + + [Fact] + public void TheRewriteKeepsTheWhitespaceBetweenElements() + { + const string xml = "\r\n \r\n \r\n \r\n"; + + string result = Run(xml)!; + + // the reader normalizes a literal CR LF between elements to LF (XML 1.0 section 2.11); that is the only + // change besides the withheld value and the space the writer puts before an empty element's slash + Assert.Equal("\n \n \n \n", result); + } + + [Theory] + [InlineData("spill_plan.sqlplan")] + [InlineData("key_lookup_plan.sqlplan")] + [InlineData("udf_plan.sqlplan")] + public void ARealPlan_ComesBackEqualToItself_ExceptTheWithheldValues(string file) + { + string xml = File.ReadAllText(Path.Combine(AppContext.BaseDirectory, "Fixtures", "OriginPlans", file)); + const string hitValue = "false"; + string? result = Run(xml, s => s == hitValue); + Assert.NotSame(xml, result); // every plan holds a false flag + + var expected = XDocument.Parse(xml); + foreach (var a in expected.Descendants().Attributes().Where(a => a.Value == hitValue).ToList()) + a.Value = P; + var actual = XDocument.Parse(result!); + + Assert.True(XNode.DeepEquals(expected, actual), "only the withheld values may differ"); + // multi-line statement text survives as the same characters (CR LF inside an attribute is a character reference) + Assert.Equal( + expected.Descendants().Select(e => (string?)e.Attribute("StatementText")).Where(v => v is not null), + actual.Descendants().Select(e => (string?)e.Attribute("StatementText")).Where(v => v is not null)); + } + + // ── failure rules and the budget ── + + [Fact] + public void CutXml_IsJudgedWhole_DecodedBeforeJudging() + { + const string cutNamed = "CREATE LOGIN s.Contains("CREATE\r\nLOGIN", StringComparison.Ordinal))); + Assert.Same(cutClean, Run(cutClean)); + Assert.Equal(P, Run("CANARY]>&e;"; + const string clean = "]>&e;"; + + Assert.Equal(P, Run(named)); + Assert.Same(clean, Run(clean)); + } + + [Fact] + public void ASpentBudgetAfterTheFirstNode_GivesTheWholePlaceholder() + { + string xml = Fixture("two_statements.xml").Replace("CANARY", "plain", StringComparison.Ordinal); + int asked = 0; + + string? pass1 = Run(xml, spent: () => ++asked > 1); + + Assert.Equal(P, pass1); + + // spent only once pass 2 is running: still the whole placeholder, never a half-written document + string named = Fixture("two_statements.xml"); + int asks = 0; + int asksAtFirstHit = 0; + Run(named, v => + { + bool hit = Canary(v); + if (hit && asksAtFirstHit == 0) + asksAtFirstHit = asks; + return hit; + }, () => { asks++; return false; }); + Assert.True(asksAtFirstHit > 0 && asks > asksAtFirstHit + 10, "pass 2 must ask the budget after pass 1 stopped"); + int calls = 0; + + string? pass2 = Run(named, spent: () => ++calls > asksAtFirstHit + 10); + + Assert.Equal(P, pass2); + } + + [Fact] + public void AJudgeThatThrows_FailsClosedToTheWholePlaceholder() + { + string? result = Run("", _ => throw new InvalidOperationException("boom")); + + Assert.Equal(P, result); + } + + [Fact] + public void TheParseTimeIsChargedToTheCaller() + { + var charged = new List(); + + Run(Fixture("two_statements.xml"), charge: charged.Add); + + Assert.NotEmpty(charged); + Assert.All(charged, t => Assert.True(t >= TimeSpan.Zero)); + } + + // ── maxOutputChars (P3, M5, L-I) ── + + private static string PrettyPlan(int statements, int bodyLinesPerStatement, int canaryStatement) + { + var sb = new StringBuilder(); + sb.Append("\r\n"); + for (int i = 0; i < statements; i++) + { + string text = i == canaryStatement ? "SELECT CANARY" : "SELECT " + i; + sb.Append(" \r\n"); + for (int l = 0; l < bodyLinesPerStatement; l++) + sb.Append(" \r\n"); + sb.Append(" \r\n"); + } + sb.Append(""); + return sb.ToString(); + } + + [Fact] + public void WithAMaxOutput_ANamedStatementTextThatStarts2KbBeforeTheCutIsAHit() + { + // each statement is ~2 KB of body lines; the named one starts well before the cut at ~8 KB + string xml = PrettyPlan(statements: 20, bodyLinesPerStatement: 20, canaryStatement: 3); + int named = xml.IndexOf("SELECT CANARY", StringComparison.Ordinal); + int cutAt = named + 2048; + Assert.True(named > 0 && cutAt < xml.Length - 4096, "fixture shape"); + + string? result = Run(xml, max: cutAt); + + Assert.NotSame(xml, result); + Assert.DoesNotContain("CANARY", result); + Assert.Contains("StatementText=\"" + PText + "\"", result); + // the rewrite goes at least to the cut, and stops short of the whole document + Assert.True(result!.Length >= cutAt, "result must reach the cut: " + result.Length + " vs " + cutAt); + Assert.True(result.Length < xml.Length, "work is bounded by the cut"); + } + + [Fact] + public void WithAMaxOutput_AValueWhoseNodeStartsPastTheCutIsNotJudged() + { + string xml = PrettyPlan(statements: 20, bodyLinesPerStatement: 20, canaryStatement: 18); + int named = xml.IndexOf("SELECT CANARY", StringComparison.Ordinal); + + Assert.Same(xml, Run(xml, max: named - 4096)); + Assert.NotSame(xml, Run(xml, max: named + 4096)); + Assert.NotSame(xml, Run(xml)); + } + + [Fact] + public void WithAMaxOutput_ALongNamedAttributeThatStartsBeforeTheCutIsStillJudgedWhole() + { + string longValue = new string('x', 100_000) + " CANARY"; + string xml = "\r\n\r\n"; + + string? result = Run(xml, max: 200); + + Assert.NotSame(xml, result); + Assert.Contains(PText, result); + Assert.DoesNotContain("CANARY", result); + } + + [Fact] + public void WithAMaxOutput_ACleanDocumentIsTheSameInstance() + { + string xml = PrettyPlan(statements: 20, bodyLinesPerStatement: 20, canaryStatement: -1); + + Assert.Same(xml, Run(xml, max: 1000)); + } + + // ── measurements (assert correctness only; the timings are printed for the part file) ── + + [Fact] + public void Measurement_PlanPaddedTo512Kb() + { + string plan = File.ReadAllText(Path.Combine(AppContext.BaseDirectory, "Fixtures", "OriginPlans", "serially-parallel.sqlplan")); + int start = plan.IndexOf("", start, StringComparison.Ordinal) + "".Length; + Assert.True(start > 0 && end > start); + string block = plan.Substring(start, end - start); + Assert.DoesNotContain("", StringComparison.Ordinal); + Assert.True(insertAt > 0); + var clones = new StringBuilder(); + while (plan.Length + clones.Length < 512 * 1024) + clones.Append(block); + padded.Insert(insertAt, clones.ToString()); + string big = padded.ToString(); + + // a hit in statement 1 only: the canary goes in the first statement's text + string firstTag = "StatementText=\""; + int at = big.IndexOf(firstTag, start, StringComparison.Ordinal) + firstTag.Length; + string hit = big.Insert(at, "CANARY "); + + Assert.Same(big, Run(big)); + double best1 = double.MaxValue; + for (int i = 0; i < 7; i++) + { + var sw = Stopwatch.StartNew(); + Run(big); + best1 = Math.Min(best1, sw.Elapsed.TotalSeconds); + } + + string? rewritten = Run(hit); + Assert.NotSame(hit, rewritten); + Assert.DoesNotContain("CANARY", rewritten); + double best2 = double.MaxValue; + for (int i = 0; i < 7; i++) + { + var sw = Stopwatch.StartNew(); + Run(hit); + best2 = Math.Min(best2, sw.Elapsed.TotalMilliseconds); + } + + string line = string.Create(System.Globalization.CultureInfo.InvariantCulture, + $"SSF-L2-MEASURE chars={big.Length} pass1_no_hit_ms={best1 * 1000:F1} pass1_Mchars_per_s={big.Length / 1e6 / best1:F1} pass2_hit_ms={best2:F1} out_chars={rewritten!.Length}"); + TestContext.Current.SendDiagnosticMessage(line); + Assert.True(best1 > 0); + } + + // ── comments and processing instructions are judged like a text node ── + + [Fact] + public void ANamedComment_IsWithheld_AndAPlainOneIsByteIdentical() + { + const string plain = ""; + Assert.Same(plain, Run(plain)); + + string? named = Run(""); + + Assert.NotNull(named); + Assert.DoesNotContain("CANARY", named); + Assert.Contains("", named); + Assert.Contains("", named); + } + + [Fact] + public void ANamedProcessingInstruction_IsWithheld_AndAPlainOneIsByteIdentical() + { + const string plain = ""; + Assert.Same(plain, Run(plain)); + + string? named = Run(""); + + Assert.NotNull(named); + Assert.DoesNotContain("CANARY", named); + Assert.Contains("", named); + } + + [Fact] + public void ANamedComment_IsAHitEvenWhenNothingElseIs_AfterTheFirstElement() + { + string? named = Run(""); + + Assert.NotNull(named); + Assert.DoesNotContain("CANARY", named); + } +} diff --git a/Darling/Darling.Tests/SensitiveStatementsParityLiveTests.cs b/Darling/Darling.Tests/SensitiveStatementsParityLiveTests.cs new file mode 100644 index 0000000000..d513102ea8 --- /dev/null +++ b/Darling/Darling.Tests/SensitiveStatementsParityLiveTests.cs @@ -0,0 +1,190 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Common; +using Xunit; + +namespace Darling.Tests; + +/// +/// Parity of the two engines (#4348): PostgreSQL's ~* against the shared pattern and +/// must agree on every string of the corpus. The adversarial strings +/// are not part of the parity set: PostgreSQL's answer and time are recorded for them, without an equality +/// assertion. +/// +/// #1776 own-store - mints its own scratch database () rather than +/// sharing the live fixture, so it is deliberately NOT in the live-postgres collection. +/// +[Collection("timing")] +public sealed class SensitiveStatementsParityLiveTests +{ + [Fact] + public async Task PostgresAndTheDotNetEvaluationAgreeOnTheCorpus() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrWhiteSpace(connectionString), + "Set DARLING_TEST_PG to a connection string to compare the #4348 pattern in both engines."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(connectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + + var disagreements = new List(); + var checkedCount = 0; + var all = new List<(string Text, bool Expected)>(); + foreach (var text in SensitiveStatementCorpus.Named) + { + foreach (var variant in SensitiveStatementCorpus.Variants(text)) + { + all.Add((variant, true)); + } + } + + foreach (var text in SensitiveStatementCorpus.NotNamed) + { + all.Add((text, false)); + } + + foreach (var (text, expected) in all) + { + var pg = await IsNamedAsync(connection, text, ct); + var dotnet = SensitiveStatements.Names(text); + checkedCount++; + if (pg != dotnet || pg != expected) + { + disagreements.Add($"pg={pg} dotnet={dotnet} expected={expected}: {text}"); + } + } + + TestContext.Current.SendDiagnosticMessage(string.Create(CultureInfo.InvariantCulture, + $"parity: {checkedCount} strings, {disagreements.Count} disagreements")); + Assert.True(disagreements.Count == 0, string.Join(Environment.NewLine, disagreements)); + } + + [Fact] + public async Task PostgresAnswersTheAdversarialStringsQuickly_RecordedNotCompared() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrWhiteSpace(connectionString), + "Set DARLING_TEST_PG to a connection string to time the #4348 adversarial strings in PostgreSQL."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(connectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + + foreach (var text in SensitiveStatementCorpus.Adversarial) + { + var watch = Stopwatch.StartNew(); + var named = await IsNamedAsync(connection, text, ct); + watch.Stop(); + TestContext.Current.SendDiagnosticMessage(string.Create(CultureInfo.InvariantCulture, + $"pg ~* on adversarial '{text.Substring(0, 8)}...': named={named}, {watch.ElapsedMilliseconds} ms")); + } + } + + /// The strings aimed at the typed-declaration alternative (#5320) carry no literal, so both engines + /// answer Clean, and PostgreSQL answers them quickly. + [Fact] + public async Task PostgresAndTheDotNetEvaluationAgreeOnTheTypedDeclarationAdversarialStrings() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrWhiteSpace(connectionString), + "Set DARLING_TEST_PG to a connection string to compare the #5320 typed-declaration strings in both engines."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(connectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + + foreach (var text in SensitiveStatementCorpus.TypedDeclarationAdversarial) + { + var watch = Stopwatch.StartNew(); + var pg = await IsNamedAsync(connection, text, ct); + watch.Stop(); + var dotnet = SensitiveStatements.Names(text); + TestContext.Current.SendDiagnosticMessage(string.Create(CultureInfo.InvariantCulture, + $"pg ~* on typed-declaration string '{text.Substring(0, 8)}...': named={pg}, {watch.ElapsedMilliseconds} ms")); + + Assert.False(pg, text.Substring(0, 24)); + Assert.Equal(dotnet, pg); + Assert.True(watch.ElapsedMilliseconds < 2000, $"{watch.ElapsedMilliseconds} ms: {text.Substring(0, 24)}"); + } + } + + /// The one known divergence (#5320 L2): a typed secret-named variable followed by a dash banner, in a + /// value that also passes the pre-check. PostgreSQL answers Clean, quickly; the .NET judge runs out of its time + /// on the backtracking comment run and names the value (TimedOut). The row pins both answers. + [Fact] + public async Task PostgresAnswersTheDashBannerRowCleanWhereTheDotNetJudgeNamesItByTimeout() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrWhiteSpace(connectionString), + "Set DARLING_TEST_PG to a connection string to compare the #5320 dash-banner row in both engines."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(connectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + + foreach (var text in SensitiveStatementCorpus.NamedByTimeoutInDotNetOnly) + { + var watch = Stopwatch.StartNew(); + var pg = await IsNamedAsync(connection, text, ct); + watch.Stop(); + + Assert.False(pg, text.Substring(0, 24)); + Assert.True(watch.ElapsedMilliseconds < 2000, $"{watch.ElapsedMilliseconds} ms: {text.Substring(0, 24)}"); + Assert.Equal(SensitiveStatements.Verdict.TimedOut, SensitiveStatements.Judge(text)); + Assert.True(SensitiveStatements.Names(text)); + } + } + + /// A U+212A (Kelvin sign) next to a named keyword is not a word character under the C collation, + /// and the .NET word class is case-sensitive so it is not one there either (#5320 L1). PostgreSQL's + /// answer depends on the cluster's locale, so the match runs with COLLATE "C" and the fact runs + /// on every cluster, whatever its character type (#5320 N2). Not part of the corpus: lower-casing the + /// sign gives an ASCII k. + [Fact] + public async Task AKelvinSignNextToANamedKeywordGetsTheSameVerdictInBothEngines_UnderTheCCollation() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrWhiteSpace(connectionString), + "Set DARLING_TEST_PG to a connection string to compare the #5320 Kelvin-sign case in both engines."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(connectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + + const string text = "x \u212Asp_addlogin 'a'"; + var pg = await IsNamedAsync(connection, text, ct, cCollation: true); + var dotnet = SensitiveStatements.Names(text); + TestContext.Current.SendDiagnosticMessage(string.Create(CultureInfo.InvariantCulture, + $"kelvin sign: pg={pg} dotnet={dotnet}")); + + Assert.True(dotnet); + Assert.Equal(dotnet, pg); + } + + private static async Task IsNamedAsync(NpgsqlConnection connection, string text, CancellationToken ct, bool cCollation = false) + { + await using var command = new NpgsqlCommand(cCollation ? "SELECT $1 COLLATE \"C\" ~* $2" : "SELECT $1 ~* $2", connection); + command.Parameters.AddWithValue(text); + command.Parameters.AddWithValue(SensitiveStatements.Pattern); + return (bool)(await command.ExecuteScalarAsync(ct))!; + } +} diff --git a/Darling/Darling.Tests/SensitiveStatementsTests.cs b/Darling/Darling.Tests/SensitiveStatementsTests.cs new file mode 100644 index 0000000000..0145520c0f --- /dev/null +++ b/Darling/Darling.Tests/SensitiveStatementsTests.cs @@ -0,0 +1,1153 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Globalization; +using System.Linq; +using System.Text; +using System.Threading; +using PerformanceMonitor.Common; +using Xunit; + +namespace Darling.Tests; + +/// +/// The .NET evaluation of the shared statement filter (#4348): the translation of the pattern, its guards, the +/// verdicts, the per-value timeout and the elapsed-time budget. The corpus is shared with the PostgreSQL +/// parity test in . +/// +public sealed class SensitiveStatementsTests +{ + [Fact] + public void TheCorpusIsNamedInEveryVariant() + { + var failures = new List(); + foreach (var text in SensitiveStatementCorpus.Named) + { + foreach (var variant in SensitiveStatementCorpus.Variants(text)) + { + if (!SensitiveStatements.Names(variant)) + { + failures.Add("should be named: " + variant); + } + } + } + + Assert.True(failures.Count == 0, string.Join(Environment.NewLine, failures)); + } + + [Fact] + public void TheNeighboursAreNotNamed() + { + var failures = SensitiveStatementCorpus.NotNamed + .Where(SensitiveStatements.Names) + .Select(text => "should NOT be named: " + text) + .ToList(); + + Assert.True(failures.Count == 0, string.Join(Environment.NewLine, failures)); + } + + [Fact] + public void JudgeIsNotCleanExactlyWhenNamesIsTrue_OnTheCorpus() + { + var all = SensitiveStatementCorpus.Named.SelectMany(SensitiveStatementCorpus.Variants) + .Concat(SensitiveStatementCorpus.NotNamed); + foreach (var text in all) + { + Assert.Equal(SensitiveStatements.Judge(text) != SensitiveStatements.Verdict.Clean, SensitiveStatements.Names(text)); + } + } + + [Fact] + public void NullAndEmptyAreNeverNamed() + { + Assert.False(SensitiveStatements.Names(null)); + Assert.False(SensitiveStatements.Names(string.Empty)); + Assert.Equal(SensitiveStatements.Verdict.Clean, SensitiveStatements.Judge(string.Empty)); + Assert.Null(SensitiveStatements.Text(null)); + Assert.Equal(string.Empty, SensitiveStatements.Text(string.Empty)); + } + + /// The translated pattern is pinned as a literal: a change to the shared pattern or to the + /// translation has to show up here on purpose. No POSIX class survives it. + [Fact] + public void TheTranslatedPatternIsPinnedAndHasNoPosixClassLeft() + { + var translated = SensitiveStatements.TranslatedPattern; + + Assert.NotNull(translated); + Assert.DoesNotContain("[:", translated, StringComparison.Ordinal); + Assert.Equal(PinnedTranslation, translated); + } + + private const string PinnedTranslation = """ + (?:(?An anchor or a stray ] that the guard must understand: ^ and $ inside a + /// bracket are fine, and a ] that closes no bracket is a literal (T9 needs it). + [Fact] + public void BracketsAndALiteralCloseBracketPassTheGuard() + { + Assert.True(SensitiveStatements.TryTranslate("[^$[:space:]]+(]|\")x", out var translated)); + Assert.Equal("[^$\\s]+(]|\")x", translated); + Assert.True(SensitiveStatements.TryTranslate("[]^]x", out _)); + Assert.False(SensitiveStatements.TryTranslate("[abc", out _)); + } + + /// The word-start assertion is hoisted out of the alternatives that share it (a speed-up only), + /// and an alternative without it stays outside the group. + [Fact] + public void TheSharedWordStartIsHoistedOutOfTheAlternatives() + { + Assert.True(SensitiveStatements.TryTranslate("[[:<:]]ab|[[:<:]]cd|ef", out var translated)); + + Assert.Equal("(?:(?The .NET judge compiles the pattern as two regexes (#5320 M1), both cut out of the one shared + /// constant. Put back together with | they are exactly . The + /// first is the thirteen alternatives before T10, and its translation is the literal the whole judge was before + /// T10 existed; the second is T10 alone. PostgreSQL still reads the whole constant. + [Fact] + public void TheTwoJudgePartsPutBackTogetherAreExactlyThePattern() + { + var count = SensitiveStatements.JudgeHeadAlternatives; + Assert.True(SensitiveStatements.TrySplitAlternatives(SensitiveStatements.Pattern, count, out var head, out var tail)); + + Assert.Equal(SensitiveStatements.Pattern, head + "|" + tail); + Assert.False(SensitiveStatements.TrySplitAlternatives(tail, 1, out _, out _), "the second part is one alternative"); + Assert.True(SensitiveStatements.TrySplitAlternatives(head, count - 1, out _, out _)); + Assert.False(SensitiveStatements.TrySplitAlternatives(head, count, out _, out _), "the first part is exactly " + count + " alternatives"); + Assert.StartsWith("[[:<:]]([a-z0-9_]*(password|passwd|pwd|secret)|key_source)", tail, StringComparison.Ordinal); + Assert.True(SensitiveStatements.TryTranslate(head, out var headTranslated)); + Assert.True(SensitiveStatements.TryTranslate(tail, out var tailTranslated)); + Assert.Equal(PinnedHeadTranslation, headTranslated); + Assert.Equal(PinnedTailTranslation, tailTranslated); + } + + /// The first judge part, A1-T9, translated: byte for byte the literal + /// pinned before T10 was appended. + private const string PinnedHeadTranslation = """ + (?:(?The second judge part, T10 alone, translated. + private const string PinnedTailTranslation = """ + (?Two regexes are a speed change only (#5320 M1): over the corpus and the seeded fuzz set the split + /// judge and the one-regex judge give the same verdict, so "named when either matches" is exact OR. The + /// first part alone is built too, so the test also shows the fuzz set reaches strings only the second part names. + [Fact] + public void TheSplitJudgeGivesTheSameVerdictAsTheOneRegexJudge_OverTheCorpusAndAFuzzSet() + { + var timeout = TimeSpan.FromSeconds(5); + Assert.True(SensitiveStatements.TrySplitAlternatives( + SensitiveStatements.Pattern, SensitiveStatements.JudgeHeadAlternatives, out var headSource, out _)); + var one = SensitiveStatements.CreateJudge(SensitiveStatements.Pattern, timeout, prefilter: false); + var two = SensitiveStatements.CreateJudge( + SensitiveStatements.Pattern, timeout, prefilter: false, headAlternatives: SensitiveStatements.JudgeHeadAlternatives); + var headOnly = SensitiveStatements.CreateJudge(headSource, timeout, prefilter: false); + + var named = 0; + var tailOnly = 0; + var failures = new List(); + foreach (var text in CorpusAndFuzz()) + { + var verdict = one(text); + named += verdict == SensitiveStatements.Verdict.Named ? 1 : 0; + tailOnly += verdict == SensitiveStatements.Verdict.Named && headOnly(text) == SensitiveStatements.Verdict.Clean ? 1 : 0; + if (two(text) != verdict) + { + failures.Add("split and single differ: " + text); + } + } + + Assert.True(named > 1_000, $"the fuzz set names only {named} strings, too few to pin anything"); + Assert.True(tailOnly > 50, $"only {tailOnly} strings are named by the second part alone, too few to pin anything"); + Assert.True(failures.Count == 0, string.Join(Environment.NewLine, failures.Take(20))); + } + + /// A runs at the largest limit that fits the time + /// left (#5320 M1), so the second judge part never runs past what the first left of the one budget. With + /// 250 ms the limits are 187, 93, 46 and 23 ms. The text backtracks without end, so each call ends at the + /// limit it ran with: every reading is at least 60% of the limit, and the best of three is within a generous + /// overshoot of it. + [Theory] + [InlineData(200, 187)] + [InlineData(150, 93)] + [InlineData(90, 46)] + [InlineData(45, 23)] + public void ABudgetedRegexRunsAtTheLargestLimitThatFitsWhatIsLeft(int remainingMs, int expectedLimitMs) + { + var regex = new SensitiveStatements.BudgetedRegex("(a|aa)+b", TimeSpan.FromMilliseconds(250)); + var text = new string('a', 60) + "c"; + + var best = long.MaxValue; + for (var attempt = 0; attempt < 3; attempt++) + { + var watch = Stopwatch.StartNew(); + Assert.Throws( + () => regex.IsMatch(text, TimeSpan.FromMilliseconds(remainingMs))); + watch.Stop(); + Assert.True(watch.ElapsedMilliseconds >= expectedLimitMs * 0.6, $"stopped after {watch.ElapsedMilliseconds} ms, below the {expectedLimitMs} ms limit"); + best = Math.Min(best, watch.ElapsedMilliseconds); + } + + TimingClaim.AtMost(best, expectedLimitMs + 100, $"best of 3 for a {expectedLimitMs} ms limit"); + } + + [Fact] + public void ABudgetedRegexWithLessTimeLeftThanItsSmallestLimitTimesOutWithoutRunning() + { + var regex = new SensitiveStatements.BudgetedRegex("b", TimeSpan.FromMilliseconds(250)); + + Assert.True(regex.IsMatch("xb", TimeSpan.FromMilliseconds(250))); + Assert.False(regex.IsMatch("xa", TimeSpan.FromMilliseconds(30))); + Assert.Throws(() => regex.IsMatch("xb", TimeSpan.FromMilliseconds(20))); + Assert.Throws(() => regex.IsMatch("xb", TimeSpan.Zero)); + Assert.Throws(() => regex.IsMatch("xb", TimeSpan.FromMilliseconds(-5))); + } + + /// The two parts share the one per-value timeout (#5320 M1). A fake clock says the first part took a + /// given time; the second part then gets only what is left. The first part is a plain x; the second + /// backtracks without end on the text, so its real time shows the limit it was given: nothing used runs it at + /// 187 ms, 100 ms used at 93 ms, 200 ms used at 46 ms, and 240 ms used (under the smallest 23 ms limit left) is + /// TimedOut at once. + [Theory] + [InlineData(0, 187)] + [InlineData(100, 93)] + [InlineData(200, 46)] + [InlineData(240, 0)] + public void TheSecondJudgePartRunsOnlyInsideWhatTheFirstLeftOfTheBudget(int firstPartMs, int limitMs) + { + var text = new string('a', 60) + "d"; + var best = long.MaxValue; + for (var attempt = 0; attempt < 3; attempt++) + { + var calls = 0; + var judge = SensitiveStatements.CreateJudge( + "x|(a|aa)+c", + TimeSpan.FromMilliseconds(250), + prefilter: false, + headAlternatives: 1, + clock: () => TimeSpan.FromMilliseconds(calls++ == 0 ? 0 : firstPartMs)); + var watch = Stopwatch.StartNew(); + var verdict = judge(text); + watch.Stop(); + + Assert.Equal(SensitiveStatements.Verdict.TimedOut, verdict); + Assert.True(watch.ElapsedMilliseconds >= limitMs * 0.6, $"stopped after {watch.ElapsedMilliseconds} ms"); + best = Math.Min(best, watch.ElapsedMilliseconds); + } + + TimingClaim.AtMost(best, limitMs + 100, $"best of 3; the second part had {limitMs} ms"); + } + + /// A part that matches names the value whichever part it is, and a text neither matches is clean. + [Fact] + public void ATwoPartJudgeNamesAValueEitherPartMatches() + { + var judge = SensitiveStatements.CreateJudge("x|y|z", SensitiveStatements.MatchTimeout, headAlternatives: 2); + + Assert.Equal(SensitiveStatements.Verdict.Named, judge("--x--")); + Assert.Equal(SensitiveStatements.Verdict.Named, judge("--y--")); + Assert.Equal(SensitiveStatements.Verdict.Named, judge("--z--")); + Assert.Equal(SensitiveStatements.Verdict.Clean, judge("--w--")); + } + + /// The production judge is the split judge (#5320 N2), pinned without a stopwatch. A fake clock says the + /// first part used 240 ms of the 250 ms budget, so the second part has too little time left and the value is + /// TimedOut. x_pwd = HASHBYTES(1) reaches the full judge (the pre-check's superset matches it) and no + /// part names it, so a judge that has gone back to one regex answers Clean here and fails this test. + [Fact] + public void TheProductionJudgeRunsItsSecondPartInsideWhatTheFirstLeft() + { + var calls = 0; + var judge = SensitiveStatements.CreateProductionJudge( + clock: () => TimeSpan.FromMilliseconds(calls++ == 0 ? 0 : 240)); + + Assert.Equal(SensitiveStatements.Verdict.TimedOut, judge("x_pwd = HASHBYTES(1)")); + } + + /// The warm-up builds the shared judge, can run again, and does not throw (#5320 N1). + [Fact] + public void WarmUpBuildsTheSharedJudgeAndCanRunAgain() + { + SensitiveStatements.WarmUp(); + SensitiveStatements.WarmUp(); + + Assert.True(SensitiveStatements.JudgeIsBuilt); + Assert.Equal(SensitiveStatements.Verdict.Clean, SensitiveStatements.Judge("SELECT 1")); + } + + /// 1 MB of ordinary SQL with a trailing near miss that only the pre-check's superset hits (#5320 M1). + /// x_pwd = HASHBYTES(1) has no word start, so the pre-check matches it (T4 without its assertion) and the + /// full judge must run and answer Clean. With the whole pattern as one compiled regex that took about 226 ms + /// a megabyte and the value timed out and was withheld; as two regexes it is Clean inside the 250 ms timeout. + [Fact] + public void AMegabyteOfOrdinarySqlWithATrailingNearMissIsCleanInsideTheTimeout() + { + StatementFilterWarmUp.Ensure(); + SensitiveStatements.Judge("warm the regex up"); + var unit = "SELECT col_a, col_b FROM dbo.t WHERE c = 1 AND d <> 2 "; + var text = string.Concat(Enumerable.Repeat(unit, 1_000_000 / unit.Length + 1)).Substring(0, 1_000_000) + + " x_pwd = HASHBYTES(1)"; + + // The gate (#5369): fail-closed holds whatever the load (a slow machine may time out, never name nothing + // wrongly), and with no clock to run out the same split judge answers Clean. + Assert.NotEqual(SensitiveStatements.Verdict.Named, SensitiveStatements.Judge(text)); + Assert.Equal(SensitiveStatements.Verdict.Clean, UnboundedJudge.Value(text)); + + // The wall-clock claim, a measurement unless DARLING_TIMING_TESTS=1. + var best = long.MaxValue; + for (var attempt = 0; attempt < 5 && best >= 250; attempt++) + { + var watch = Stopwatch.StartNew(); + var verdict = SensitiveStatements.Judge(text); + watch.Stop(); + + if (verdict == SensitiveStatements.Verdict.Clean) + { + best = Math.Min(best, watch.ElapsedMilliseconds); + } + } + + TimingClaim.AtMost(best == long.MaxValue ? double.PositiveInfinity : best, 249, "1 MB near miss came back Clean"); + } + + /// A typed secret-named variable followed by a long dash banner, in a value where something else + /// passes the pre-check (#5320 L2). The typed-declaration alternative's gaps after the type backtrack on a run of + /// dashes (the known comment-run case); the second judge part runs out of its time and the value is named by the + /// timeout. PostgreSQL answers Clean. That errs toward withholding, and an atomic rewrite of the gap is ruled + /// out, so the row pins the .NET answer: TimedOut, never Clean, back inside about the match timeout. + [Fact] + public void ATypedSecretVariableBehindADashBannerIsNamedByTimeout_WhenAnotherPartPassesThePreCheck() + { + SensitiveStatements.Judge("warm the regex up"); + foreach (var text in SensitiveStatementCorpus.NamedByTimeoutInDotNetOnly) + { + var best = long.MaxValue; + for (var attempt = 0; attempt < 3; attempt++) + { + var watch = Stopwatch.StartNew(); + var verdict = SensitiveStatements.Judge(text); + watch.Stop(); + + Assert.Equal(SensitiveStatements.Verdict.TimedOut, verdict); + best = Math.Min(best, watch.ElapsedMilliseconds); + } + + TimingClaim.AtMost(best, 400, "best of 3 for a banner named by timeout"); + Assert.True(SensitiveStatements.Names(text)); + } + } + + [Fact] + public void AMatchThatRunsPastItsTimeoutGivesTimedOut() + { + // Nested alternation under a quantifier backtracks exponentially on a near miss. + // Without the pre-check: its linear-time engine would answer this toy pattern (which has no assertion to drop) Clean. + var judge = SensitiveStatements.CreateJudge("(a|aa)+b", TimeSpan.FromMilliseconds(50), prefilter: false); + + var verdict = judge(new string('a', 60) + "c"); + + Assert.Equal(SensitiveStatements.Verdict.TimedOut, verdict); + Assert.Equal(SensitiveStatements.Verdict.Named, judge("xaab")); + Assert.Equal(SensitiveStatements.Verdict.Clean, judge("xyz")); + } + + // The timing tests take the best of several attempts: the test assembly runs many classes in parallel + // and a loaded machine can stretch one wall-clock reading; the verdict itself must hold on every attempt. + + /// The two comment-token strings the plan once pinned as TimedOut. The linear-time pre-check + /// (#5320 M1) finds no role, user or literal in them, so they are Clean, which is also what + /// PostgreSQL answers. + [Fact] + public void EachAdversarialStringIsCleanWithin50Ms() + { + SensitiveStatements.Judge("warm the regex up"); + foreach (var text in SensitiveStatementCorpus.Adversarial) + { + var best = BestOfJudging(text, SensitiveStatements.Verdict.Clean, 5); + + TimingClaim.AtMost(best, 49, $"best of 5: {text}"); + Assert.False(SensitiveStatements.Names(text)); + } + } + + /// The same for the strings aimed at the typed-declaration alternative (#5320): a password word, many + /// comment tokens or a long length part, and no literal. The pre-check finds no literal, so they are Clean. + [Fact] + public void EachTypedDeclarationAdversarialStringIsCleanWithin50Ms() + { + SensitiveStatements.Judge("warm the regex up"); + foreach (var text in SensitiveStatementCorpus.TypedDeclarationAdversarial) + { + var best = BestOfJudging(text, SensitiveStatements.Verdict.Clean, 5); + + TimingClaim.AtMost(best, 49, $"best of 5: {text.Substring(0, 24)}"); + Assert.False(SensitiveStatements.Names(text)); + } + } + + /// Ordinary banner comments: a run of dashes after create, a comment after a column named + /// pwd, and so on. The full judge splits a dash run exponentially on these, so only the pre-check + /// keeps them Clean. + [Fact] + public void OrdinaryDashBannersAreCleanWithin50Ms() + { + var dashes = new string('-', 60); + var banners = new[] + { + "----" + dashes + "\n-- Create\n----" + dashes + "\nCREATE TABLE dbo.t (id int);", + "-- Create " + dashes + "\nSELECT 1", + "SELECT 1 AS pwd -- " + dashes, + "UPDATE t SET secret -- " + dashes + "\n= 1", + "create " + dashes + "x", + "create" + string.Concat(Enumerable.Repeat(" --", 20)), + }; + SensitiveStatements.Judge("warm the regex up"); + foreach (var text in banners) + { + var best = BestOfJudging(text, SensitiveStatements.Verdict.Clean, 5); + + TimingClaim.AtMost(best, 49, $"best of 5: {text.Substring(0, Math.Min(40, text.Length))}"); + } + } + + /// A string the pre-check still hits (xcreate login has no word start, so only the superset + /// matches) but the full judge cannot finish: it keeps the TimedOut verdict, which counts as named. A + /// planted near-hit can still force a timeout, which is what the process-wide memo is for. + [Fact] + public void ANearHitThatReachesTheFullJudgeStillTimesOutAndStaysNamed() + { + var text = "xcreate login; create" + string.Concat(Enumerable.Repeat(" --", 40)) + "x"; + SensitiveStatements.Judge("warm the regex up"); + + var best = BestOfJudging(text, SensitiveStatements.Verdict.TimedOut, 3); + + TimingClaim.AtMost(best, 300, "best of 3 for a near hit that times out"); + Assert.True(SensitiveStatements.Names(text)); + } + + /// A typed declaration of a password variable behind many comment tokens (#5320). The full judge may + /// backtrack on the second string, so the answer is Named or TimedOut (named either way), never Clean, and the + /// call returns inside the match timeout. + [Fact] + public void ATypedDeclarationBehindManyCommentTokensIsNeverClean_AndReturnsInsideTheTimeout() + { + var comments = string.Concat(Enumerable.Repeat(" --", 40)); + SensitiveStatements.Judge("warm the regex up"); + foreach (var text in new[] + { + "declare @pwd" + comments + "\n nvarchar(20) = N'x'", + "declare @pwd" + comments + " nvarchar(20) = N'x'", + }) + { + var watch = Stopwatch.StartNew(); + var verdict = SensitiveStatements.Judge(text); + watch.Stop(); + + Assert.NotEqual(SensitiveStatements.Verdict.Clean, verdict); + TimingClaim.AtMost(watch.ElapsedMilliseconds, 999, $"typed declaration behind comment tokens: {text.Substring(0, 24)}"); + } + } + + /// A 100 KB binary literal (200,000 hex characters) in an INSERT, and the other long single-run + /// shapes that made the credential-URL branch quadratic in the full judge (#5320 M2). + [Fact] + public void ALongHexLiteralAndOtherLongRunsAreCleanAndFast() + { + var texts = new[] + { + "INSERT INTO dbo.Files (Id, Body) VALUES (7, 0x" + new string('A', 200_000) + ")", + "INSERT INTO dbo.Files (Id, Body) VALUES (7, 0x" + new string('a', 1_000_000) + ")", + "SELECT '" + new string('a', 1_000_000) + "'", + }; + SensitiveStatements.Judge("warm the regex up"); + foreach (var text in texts) + { + var best = BestOfJudging(text, SensitiveStatements.Verdict.Clean, 5, UnboundedJudge.Value); + + TimingClaim.AtMost(best, 99, $"best of 5 for {text.Length} characters"); + } + } + + /// The production pattern and split, with a one-minute timeout and a clock that never advances: no + /// load can time it out, so a verdict asserted through it cannot flake (#5369). + private static readonly Lazy> UnboundedJudge = new(() => + SensitiveStatements.CreateJudge( + SensitiveStatements.Pattern, + TimeSpan.FromMinutes(1), + headAlternatives: SensitiveStatements.JudgeHeadAlternatives, + clock: static () => TimeSpan.Zero)); + + private static long BestOfJudging( + string text, SensitiveStatements.Verdict expected, int attempts, Func? judge = null) + { + judge ??= SensitiveStatements.Judge; + var best = long.MaxValue; + for (var attempt = 0; attempt < attempts; attempt++) + { + var watch = Stopwatch.StartNew(); + var verdict = judge(text); + watch.Stop(); + + Assert.Equal(expected, verdict); + best = Math.Min(best, watch.ElapsedMilliseconds); + } + + return best; + } + + private static IEnumerable CorpusAndFuzz() + { + var corpus = SensitiveStatementCorpus.Named.SelectMany(SensitiveStatementCorpus.Variants) + .Concat(SensitiveStatementCorpus.NotNamed); + return corpus.Concat(FuzzStrings()); + } + + private static readonly string[] s_fuzzTokens = + { + "create", "alter", "role", "user", "group", "subscription", "server", "login", "credential", "scoped", + "password", "passwd", "pwd", "secret", "key_source", "x_pwd", "xpassword", "pgpassword", "sp_addlogin", + "sp_password", "encryptbypassphrase", "pwdcompare", "opendatasource", "openrowset", "select", "from", + "update", "set", "dbo.t", "col", "bulk", "(", ")", "'", "n'", "e'", "u&'", "0x", "0xff", "$1", "$a", "=", + "to", ":", "@", "/", "://", "user:pw@", "http://", "]", "[", "\"", "[pwd]", "\"secret\"", ";", ",", "_", + "9", "a", " ", " ", " ", "\n", "\t", "--", "-- c", "/* c */", "/*", "*/", "*", + "nvarchar", "varchar(20)", "(max)", "text", "int", "as", "constant", "default", ":=", "[nvarchar]", "sys.sysname", + }; + + /// A seeded set of short strings built from the tokens the pattern is made of, so a large share of + /// them reach a branch of the pattern. + private static IEnumerable FuzzStrings() + { + var random = new Random(5324); + for (var i = 0; i < 30_000; i++) + { + var count = random.Next(1, 14); + var builder = new StringBuilder(); + for (var j = 0; j < count; j++) + { + builder.Append(s_fuzzTokens[random.Next(s_fuzzTokens.Length)]); + if (random.Next(3) == 0) + { + builder.Append(' '); + } + } + + yield return builder.ToString(); + } + } + + /// The pre-check is a superset of the full pattern (#5320 M1): every string the full judge names, + /// the pre-check also hits, and the pre-checked judge gives the same verdict as the full judge alone. Run + /// over the corpus and a seeded fuzz set; no PostgreSQL needed. + [Fact] + public void ThePrefilterHitsEveryStringTheFullJudgeNames_OverTheCorpusAndAFuzzSet() + { + var timeout = TimeSpan.FromSeconds(5); + var prefilter = SensitiveStatements.CreatePrefilter(SensitiveStatements.Pattern, timeout); + Assert.NotNull(prefilter); + var full = SensitiveStatements.CreateJudge(SensitiveStatements.Pattern, timeout, prefilter: false); + var combined = SensitiveStatements.CreateJudge(SensitiveStatements.Pattern, timeout); + + var named = 0; + var failures = new List(); + foreach (var text in CorpusAndFuzz()) + { + var verdict = full(text); + if (verdict == SensitiveStatements.Verdict.Named) + { + named++; + if (!prefilter!.IsMatch(text)) + { + failures.Add("the pre-check missed a named string: " + text); + } + } + + if (combined(text) != verdict) + { + failures.Add("the pre-checked judge disagrees with the full judge: " + text); + } + } + + Assert.True(named > 1_000, $"the fuzz set names only {named} strings, too few to pin anything"); + Assert.True(failures.Count == 0, string.Join(Environment.NewLine, failures.Take(20))); + } + + /// The pre-check uses no lookaround and is built from the one definition: it is the translation + /// with only the word-boundary assertions removed. + [Fact] + public void ThePrefilterSourceIsTheTranslationWithoutItsAssertions() + { + Assert.True(SensitiveStatements.TryTranslate(SensitiveStatements.Pattern, factor: false, dropAssertions: true, out var superset)); + + Assert.DoesNotContain("(?<", superset, StringComparison.Ordinal); + Assert.DoesNotContain("(?=", superset, StringComparison.Ordinal); + Assert.DoesNotContain("(?!", superset, StringComparison.Ordinal); + Assert.DoesNotContain("[:", superset, StringComparison.Ordinal); + Assert.StartsWith("(create|alter)", superset, StringComparison.Ordinal); + } + + [Fact] + public void AFailedGuardGivesNoPrefilter_AndAPatternItCannotBuildKeepsTheFullJudge() + { + Assert.Null(SensitiveStatements.CreatePrefilter("a\\b", TimeSpan.FromSeconds(1))); + Assert.Null(SensitiveStatements.CreatePrefilter("[[:bogus:]]", TimeSpan.FromSeconds(1))); + // A pattern with a lookahead of its own cannot run NonBacktracking, so there is no pre-check and the + // full judge still answers. + Assert.Null(SensitiveStatements.CreatePrefilter("a(?=b)", TimeSpan.FromSeconds(1))); + var judge = SensitiveStatements.CreateJudge("a(?=b)", TimeSpan.FromSeconds(1)); + Assert.Equal(SensitiveStatements.Verdict.Named, judge("xab")); + Assert.Equal(SensitiveStatements.Verdict.Clean, judge("xac")); + } + + /// Hoisting the shared word start out of the alternatives is only a search-order change (#5320 L5). + /// The factored and unfactored forms give the same verdict over the corpus and a seeded fuzz set, so the + /// factoring is pinned without a PostgreSQL rig. Both run without the pre-check, so it is the regex that is + /// compared. + [Fact] + public void TheFactoredAndUnfactoredFormsGiveTheSameVerdict_OverTheCorpusAndAFuzzSet() + { + var timeout = TimeSpan.FromSeconds(5); + Assert.True(SensitiveStatements.TryTranslate(SensitiveStatements.Pattern, factor: true, dropAssertions: false, out var factoredSource)); + Assert.True(SensitiveStatements.TryTranslate(SensitiveStatements.Pattern, factor: false, dropAssertions: false, out var unfactoredSource)); + Assert.NotEqual(factoredSource, unfactoredSource); + var factored = SensitiveStatements.CreateJudge(SensitiveStatements.Pattern, timeout, factor: true, prefilter: false); + var unfactored = SensitiveStatements.CreateJudge(SensitiveStatements.Pattern, timeout, factor: false, prefilter: false); + + var named = 0; + var failures = new List(); + foreach (var text in CorpusAndFuzz()) + { + var verdict = factored(text); + named += verdict == SensitiveStatements.Verdict.Named ? 1 : 0; + if (unfactored(text) != verdict) + { + failures.Add("factored and unfactored differ: " + text); + } + } + + Assert.True(named > 1_000, $"the fuzz set names only {named} strings, too few to pin anything"); + Assert.True(failures.Count == 0, string.Join(Environment.NewLine, failures.Take(20))); + } + + [Fact] + public void AMillionCharOrdinaryStatementIsCleanAndFast() + { + StatementFilterWarmUp.Ensure(); + SensitiveStatements.Judge("warm the regex up"); + var unit = "SELECT col_a, col_b FROM dbo.t WHERE c = 1 AND d <> 2 "; + var text = string.Concat(Enumerable.Repeat(unit, 1_000_000 / unit.Length + 1)).Substring(0, 1_000_000); + + // The gate (#5369): Clean with no time to run out. The speed is a measurement unless DARLING_TIMING_TESTS=1. + Assert.Equal(SensitiveStatements.Verdict.Clean, UnboundedJudge.Value(text)); + + var best = long.MaxValue; + for (var attempt = 0; attempt < 5 && best >= 250; attempt++) + { + var watch = Stopwatch.StartNew(); + var verdict = SensitiveStatements.Judge(text); + watch.Stop(); + + if (verdict == SensitiveStatements.Verdict.Clean) + { + best = Math.Min(best, watch.ElapsedMilliseconds); + } + } + + TimingClaim.AtMost(best == long.MaxValue ? double.PositiveInfinity : best, 249, "1,000,000 ordinary characters came back Clean"); + } + + [Fact] + public void TextReturnsThePlaceholderForANamedValue_AndTheSameInstanceOtherwise() + { + var clean = new string("SELECT 1 FROM dbo.t".ToCharArray()); + + Assert.Equal(SensitiveStatements.PlaceholderText, SensitiveStatements.Text("CREATE LOGIN [a] WITH PASSWORD = 'p'")); + Assert.Same(clean, SensitiveStatements.Text(clean)); + } + + [Fact] + public void TextWithholdsAValueThatIsNamedOnlyOnceHtmlDecoded() + { + // " " and " " decode to a space, so the encoded form hides the statement from a plain read. + var encoded = "CREATE LOGIN [a] FROM WINDOWS"; + Assert.False(SensitiveStatements.Names(encoded)); + + Assert.Equal(SensitiveStatements.PlaceholderText, SensitiveStatements.Text(encoded)); + var harmless = "SELECT 'a & b' FROM dbo.t"; + Assert.Same(harmless, SensitiveStatements.Text(harmless)); + } + + /// With IgnoreCase .NET reads U+212A (the Kelvin sign) as a word character, which PostgreSQL's + /// C-locale word-start check does not (#5320 L1). The word class is case-sensitive, so a Kelvin sign next + /// to a named keyword is a boundary here too, and the pre-check still hits it. + [Fact] + public void AKelvinSignNextToANamedKeywordIsAWordBoundary_AndThePrefilterStillHitsIt() + { + const string text = "x \u212Asp_addlogin 'a'"; + var timeout = TimeSpan.FromSeconds(5); + var prefilter = SensitiveStatements.CreatePrefilter(SensitiveStatements.Pattern, timeout); + + Assert.Equal(SensitiveStatements.Verdict.Named, SensitiveStatements.Judge(text)); + Assert.True(SensitiveStatements.Names(text)); + Assert.NotNull(prefilter); + Assert.Matches(prefilter!, text); + Assert.Equal(SensitiveStatements.Verdict.Named, SensitiveStatements.CreateJudge(SensitiveStatements.Pattern, timeout, prefilter: false)(text)); + } + + // ---- JudgeBudget ------------------------------------------------------------------------------------- + + [Fact] + public void ABudgetStopsRunningTheJudgeOnceSpent_AndCountsTheRestUnjudged() + { + // A fake clock, not a sleep: each judge call advances it 30 ms, so a 100 ms budget is spent after + // exactly four calls (0, 30, 60 and 90 ms elapsed all leave it unspent; 120 ms spends it) (#5320). + var calls = 0; + var now = TimeSpan.Zero; + var budget = new SensitiveStatements.JudgeBudget(TimeSpan.FromMilliseconds(100), _ => + { + calls++; + now += TimeSpan.FromMilliseconds(30); + return SensitiveStatements.Verdict.Clean; + }, clock: () => now); + + var verdicts = Enumerable.Range(0, 20).Select(i => budget.Judge("v" + i.ToString(CultureInfo.InvariantCulture))).ToList(); + + Assert.True(budget.Spent); + Assert.Equal(4, calls); + Assert.Equal(20 - calls, budget.Unjudged); + Assert.All(verdicts.Skip(calls), v => Assert.Equal(SensitiveStatements.Verdict.Named, v)); + Assert.All(verdicts.Take(calls), v => Assert.Equal(SensitiveStatements.Verdict.Clean, v)); + Assert.Equal(TimeSpan.FromMilliseconds(120), budget.Elapsed); + + var before = calls; + budget.Judge("one more"); + Assert.Equal(before, calls); + Assert.Equal(20 - calls + 1, budget.Unjudged); + } + + [Fact] + public void ABudgetCountsNamedAndTimedOutVerdicts() + { + var verdicts = new Queue(new[] + { + SensitiveStatements.Verdict.Named, + SensitiveStatements.Verdict.TimedOut, + SensitiveStatements.Verdict.Clean, + }); + var budget = new SensitiveStatements.JudgeBudget(TimeSpan.FromSeconds(30), _ => verdicts.Dequeue()); + + budget.Judge("a"); + budget.Judge("b"); + budget.Judge("c"); + + Assert.Equal(1, budget.Named); + Assert.Equal(1, budget.TimedOut); + Assert.Equal(0, budget.Unjudged); + Assert.False(budget.Spent); + } + + [Fact] + public void AddElapsedChargesTheBudget() + { + var ran = false; + var budget = new SensitiveStatements.JudgeBudget(TimeSpan.FromSeconds(1), _ => + { + ran = true; + return SensitiveStatements.Verdict.Clean; + }); + + budget.AddElapsed(TimeSpan.FromSeconds(1)); + + Assert.True(budget.Spent); + Assert.Equal(SensitiveStatements.Verdict.Named, budget.Judge("x")); + Assert.False(ran); + Assert.Equal(1, budget.Unjudged); + } + + [Fact] + public void AJudgeThatThrowsIsNamed() + { + var budget = new SensitiveStatements.JudgeBudget(TimeSpan.FromSeconds(1), _ => throw new InvalidOperationException("x")); + + Assert.Equal(SensitiveStatements.Verdict.Named, budget.Judge("x")); + Assert.Equal(1, budget.Named); + } + + /// The build of the shared judge (the regex compile, the pre-check and a warm-up match) is not + /// charged to a budget (#5320 L3): the first call is charged its match time only. The fake clock moves + /// 100 ms inside the build and 1 ms inside the match. + [Fact] + public void TheFirstCallOfABudgetIsChargedItsMatchTimeNotTheBuild() + { + var now = TimeSpan.Zero; + var built = 0; + var shared = new Lazy>(() => + { + built++; + now += TimeSpan.FromMilliseconds(100); + return _ => + { + now += TimeSpan.FromMilliseconds(1); + return SensitiveStatements.Verdict.Clean; + }; + }); + var budget = new SensitiveStatements.JudgeBudget(TimeSpan.FromSeconds(1), judge: null, clock: () => now, shared: shared); + + Assert.Equal(SensitiveStatements.Verdict.Clean, budget.Judge("SELECT 1")); + Assert.Equal(SensitiveStatements.Verdict.Clean, budget.Judge("SELECT 2")); + + Assert.Equal(1, built); + Assert.Equal(TimeSpan.FromMilliseconds(2), budget.Elapsed); + Assert.False(budget.Spent); + } + + [Fact] + public void ABudgetWithoutAFakeJudgeUsesTheSharedPattern() + { + var budget = new SensitiveStatements.JudgeBudget(TimeSpan.FromSeconds(5)); + + Assert.Equal(SensitiveStatements.Verdict.Named, budget.Judge("CREATE LOGIN [a] FROM WINDOWS")); + Assert.Equal(SensitiveStatements.Verdict.Clean, budget.Judge("SELECT 1")); + } + + // ---- Budget growth and memo (#5477) ------------------------------------------------------------------ + + private const int Megabyte = 1_048_576; + + /// An XML document of exactly characters that holds one plain statement. + private static string PaddedXml(int length, char pad = 'x') => + "" + new string(pad, length - "".Length) + ""; + + [Fact] + public void ABudgetGrowsHalfASecondPerMegabyteOfEachDistinctDocument_UpToTheCap() + { + var floor = TimeSpan.FromMilliseconds(1500); + + var one = new SensitiveStatements.JudgeBudget(floor); + one.Earn(4 * Megabyte); + Assert.Equal(TimeSpan.FromSeconds(3.5), one.Limit); + + var three = new SensitiveStatements.JudgeBudget(floor); + for (var i = 0; i < 3; i++) + { + three.Earn(4 * Megabyte); + } + + Assert.Equal(TimeSpan.FromSeconds(7.5), three.Limit); + + var half = new SensitiveStatements.JudgeBudget(floor); + half.Earn(Megabyte / 2); + Assert.Equal(TimeSpan.FromSeconds(1.75), half.Limit); + + var huge = new SensitiveStatements.JudgeBudget(floor); + huge.Earn(40 * Megabyte); + Assert.Equal(TimeSpan.FromSeconds(10), huge.Limit); + + // A floor above the cap (the 15 s session budget) is never cut down. + var session = new SensitiveStatements.JudgeBudget(TimeSpan.FromSeconds(15)); + session.Earn(40 * Megabyte); + Assert.Equal(TimeSpan.FromSeconds(15), session.Limit); + } + + [Fact] + public void TheEntryPointsEarnForADocumentOnce_AndAMemoHitEarnsNothing() + { + StatementFilterWarmUp.Ensure(); + var calls = 0; + var budget = new SensitiveStatements.JudgeBudget(TimeSpan.FromSeconds(1.5), _ => + { + calls++; + return SensitiveStatements.Verdict.Clean; + }); + var doc = PaddedXml(Megabyte); + + var first = SensitiveStatements.Xml(doc, budget); + var callsAfterFirst = calls; + Assert.Same(doc, first); + Assert.Equal(TimeSpan.FromSeconds(2), budget.Limit); + Assert.Equal(1, budget.DocumentPasses); + + // The same instance, then an equal copy: both are hits that run no judge and earn nothing, and the copy + // comes back as the caller's own instance, not the first one. + Assert.Same(doc, SensitiveStatements.Xml(doc, budget)); + var copy = PaddedXml(Megabyte); + Assert.NotSame(doc, copy); + Assert.Same(copy, SensitiveStatements.Xml(copy, budget)); + Assert.Equal(callsAfterFirst, calls); + Assert.Equal(TimeSpan.FromSeconds(2), budget.Limit); + Assert.Equal(1, budget.DocumentPasses); + + // A different document of the same size earns again. + SensitiveStatements.Xml(PaddedXml(Megabyte, 'y'), budget); + Assert.Equal(TimeSpan.FromSeconds(2.5), budget.Limit); + Assert.Equal(2, budget.DocumentPasses); + } + + [Fact] + public void TwoEqualStringsUnderOneBudget_EachComesBackAsItsOwnInstance_WhenNothingIsNamed() + { + var budget = new SensitiveStatements.JudgeBudget(TimeSpan.FromSeconds(1.5), _ => SensitiveStatements.Verdict.Clean); + var first = "{\"k\":\"v\"}"; + var second = new string(first.AsSpan()); + Assert.NotSame(first, second); + + Assert.Same(first, SensitiveStatements.Json(first, budget)); + Assert.Same(second, SensitiveStatements.Json(second, budget)); + Assert.Equal(1, budget.DocumentPasses); + } + + [Fact] + public void ACappedXmlCall_EarnsOnlyForTheTextItCanJudge() + { + StatementFilterWarmUp.Ensure(); + var floor = TimeSpan.FromSeconds(1.5); + var budget = new SensitiveStatements.JudgeBudget(floor, _ => SensitiveStatements.Verdict.Clean); + + SensitiveStatements.Xml(PaddedXml(4 * Megabyte), budget, 1000); + + // Well under a millisecond for about a thousand characters, not the two seconds a 4 MB document earns in full. + Assert.InRange(budget.Limit, floor, floor + TimeSpan.FromMilliseconds(5)); + } + + [Fact] + public void AMemoHitReturnsTheWithheldMarkerTheFirstCallReturned() + { + var calls = 0; + var budget = new SensitiveStatements.JudgeBudget(TimeSpan.FromSeconds(1.5), _ => + { + calls++; + return SensitiveStatements.Verdict.Named; + }); + var doc = PaddedXml(1000); + + var first = SensitiveStatements.Xml(doc, budget); + Assert.Contains(SensitiveStatements.PlaceholderText, first, StringComparison.Ordinal); + var callsAfterFirst = calls; + Assert.Same(first, SensitiveStatements.Xml(doc, budget)); + Assert.Equal(callsAfterFirst, calls); + + var text = new string('t', 500); + Assert.Equal(SensitiveStatements.PlaceholderText, SensitiveStatements.TextUnder(budget, text)); + var callsAfterText = calls; + Assert.Equal(SensitiveStatements.PlaceholderText, SensitiveStatements.TextUnder(budget, text)); + Assert.Equal(callsAfterText, calls); + } + + [Fact] + public void AJsonResultEarnsOnceForItsWholeLength_AndTheValuesInsideEarnNothing() + { + StatementFilterWarmUp.Ensure(); + var budget = new SensitiveStatements.JudgeBudget(TimeSpan.FromSeconds(1.5), _ => SensitiveStatements.Verdict.Clean); + var json = "{\"plan\":\"" + new string('x', Megabyte) + "\",\"other\":\"" + new string('y', Megabyte) + "\"}"; + + Assert.Same(json, SensitiveStatements.Json(json, budget)); + + var expected = new SensitiveStatements.JudgeBudget(TimeSpan.FromSeconds(1.5)); + expected.Earn(json.Length); + Assert.Equal(expected.Limit, budget.Limit); + Assert.Equal(1, budget.DocumentPasses); + + Assert.Same(json, SensitiveStatements.Json(json, budget)); + Assert.Equal(expected.Limit, budget.Limit); + Assert.Equal(1, budget.DocumentPasses); + } + + // ---- Measurements (recorded in the test diagnostics on every run) ------------------------------------- + + [Fact] + public void Measurements_AreRecordedAndWithinTheBounds() + { + StatementFilterWarmUp.Ensure(); + SensitiveStatements.Judge("warm the regex up"); + + // Throughput over 10,000 generated ordinary statements, 200-4,000 chars each. + var fragments = new[] + { + "SELECT col_a, col_b, COUNT(*) FROM dbo.Orders AS o JOIN dbo.Customers AS c ON c.id = o.customer_id ", + "WHERE o.created_at >= @from AND o.status IN (1, 2, 3) AND c.region <> N'north' ", + "GROUP BY col_a, col_b HAVING COUNT(*) > 10 ORDER BY 3 DESC; ", + "UPDATE dbo.Inventory SET qty = qty - @n, touched = SYSUTCDATETIME() WHERE sku = @sku; ", + "INSERT INTO dbo.AuditTrail (kind, detail) VALUES (N'order', N'shipped'); ", + "/* a block comment that explains the next line */ -- and a line comment ", + }; + var random = new Random(4348); + var statements = new string[10_000]; + long totalChars = 0; + for (var i = 0; i < statements.Length; i++) + { + var length = random.Next(200, 4001); + var builder = new StringBuilder(length + 120); + while (builder.Length < length) + { + builder.Append(fragments[random.Next(fragments.Length)]); + } + + statements[i] = builder.ToString(0, length); + totalChars += length; + } + + var watch = Stopwatch.StartNew(); + var named = 0; + foreach (var statement in statements) + { + if (SensitiveStatements.Judge(statement) != SensitiveStatements.Verdict.Clean) + { + named++; + } + } + + watch.Stop(); + var megabytes = totalChars / 1_048_576.0; + Record($"Judge over 10,000 statements: {megabytes:F1} MB in {watch.ElapsedMilliseconds} ms = {megabytes / watch.Elapsed.TotalSeconds:F1} MB/s, named={named}"); + Assert.Equal(0, named); + + // The 1,000,000-character guard. + var unit = "SELECT col_a, col_b FROM dbo.t WHERE c = 1 AND d <> 2 "; + var big = string.Concat(Enumerable.Repeat(unit, 1_000_000 / unit.Length + 1)).Substring(0, 1_000_000); + watch.Restart(); + var bigVerdict = SensitiveStatements.Judge(big); + watch.Stop(); + Record($"Judge on 1,000,000 chars: {bigVerdict}, {watch.Elapsed.TotalMilliseconds:F1} ms"); + + // The two judge parts and the whole pattern as one regex, on the same 1,000,000 characters (#5320 M1), without + // the pre-check and with a timeout that cannot fire, so the numbers are the cost of the matching alone. + Assert.True(SensitiveStatements.TrySplitAlternatives( + SensitiveStatements.Pattern, SensitiveStatements.JudgeHeadAlternatives, out var headSource, out var tailSource)); + var longTimeout = TimeSpan.FromSeconds(30); + foreach (var (label, source, parts) in new[] + { + ("part 1 (A1-T9)", headSource, 0), + ("part 2 (T10)", tailSource, 0), + ("both parts", SensitiveStatements.Pattern, SensitiveStatements.JudgeHeadAlternatives), + ("whole pattern as one regex", SensitiveStatements.Pattern, 0), + }) + { + var judge = SensitiveStatements.CreateJudge(source, longTimeout, prefilter: false, headAlternatives: parts); + judge(big); + var bestMs = double.MaxValue; + for (var attempt = 0; attempt < 3; attempt++) + { + watch.Restart(); + judge(big); + watch.Stop(); + bestMs = Math.Min(bestMs, watch.Elapsed.TotalMilliseconds); + } + + Record($"Judge on 1,000,000 chars, {label}, no pre-check: best of 3 {bestMs:F1} ms"); + } + + // Each adversarial string. + foreach (var text in SensitiveStatementCorpus.Adversarial) + { + watch.Restart(); + var verdict = SensitiveStatements.Judge(text); + watch.Stop(); + Record($"Judge on adversarial '{text.Substring(0, 8)}...': {verdict}, {watch.Elapsed.TotalMilliseconds:F1} ms"); + } + + // The worst overrun past the limit with a 250 ms judge (read-time limit 1.5 s, and a limit that is not + // a multiple of the judge's time). + foreach (var limitMs in new[] { 1500, 1400, 1260 }) + { + var budget = new SensitiveStatements.JudgeBudget(TimeSpan.FromMilliseconds(limitMs), _ => + { + Thread.Sleep(250); + return SensitiveStatements.Verdict.TimedOut; + }); + watch.Restart(); + while (!budget.Spent) + { + budget.Judge("x"); + } + + watch.Stop(); + var overrun = budget.Elapsed - TimeSpan.FromMilliseconds(limitMs); + Record($"JudgeBudget limit {limitMs} ms, 250 ms judge: stopped at {budget.Elapsed.TotalMilliseconds:F0} ms, overrun {overrun.TotalMilliseconds:F0} ms"); + Assert.True(overrun < TimeSpan.FromMilliseconds(300), $"overrun {overrun.TotalMilliseconds} ms"); + } + } + + private static void Record(string message) => TestContext.Current.SendDiagnosticMessage(message); +} diff --git a/Darling/Darling.Tests/SeparatelyMonitoredListNoteTests.cs b/Darling/Darling.Tests/SeparatelyMonitoredListNoteTests.cs index a33b969cc2..e89b79f15f 100644 --- a/Darling/Darling.Tests/SeparatelyMonitoredListNoteTests.cs +++ b/Darling/Darling.Tests/SeparatelyMonitoredListNoteTests.cs @@ -63,8 +63,8 @@ public void TheWebPanels_DeclareTheNoteKey_AndTheHostsPassTheRegistry() var end = js.IndexOf("),", at, StringComparison.Ordinal); Assert.Contains("\"separately_monitored_note\"", js.Substring(at, end - at), StringComparison.Ordinal); } - Assert.Contains("WebSqlTextPreviewLength, registryState, c.RequestAborted)", web, StringComparison.Ordinal); - Assert.Contains("as_of: AsOf(c), registryState: registryState, cancellationToken: c.RequestAborted)", web, StringComparison.Ordinal); + Assert.Contains("WebSqlTextPreviewLength, registryState, logger, c.RequestAborted)", web, StringComparison.Ordinal); + Assert.Contains("as_of: AsOf(c), registryState: registryState, logger: logger, cancellationToken: c.RequestAborted)", web, StringComparison.Ordinal); } [Fact] @@ -180,7 +180,7 @@ public void TheLockingPanel_CarriesBothNotes_AndTheViewerAndWebHostPassTheirSour Assert.Contains("getPath(res.data, desc.noteKey)", panels, StringComparison.Ordinal); Assert.Contains("desc.moreNoteKeys", panels, StringComparison.Ordinal); var web = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "DarlingWebEndpoints.cs"); - Assert.Contains("Rows(c, \"limit\", 200), registryState, c.RequestAborted)", web, StringComparison.Ordinal); + Assert.Contains("Rows(c, \"limit\", 200), registryState, ", web, StringComparison.Ordinal); var viewer = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "FinOpsTab.Locking.cs"); Assert.Contains("SeparatelyMonitoredListNoteFor(await _dataService.GetSeparatelyMonitoredAsync(", viewer, StringComparison.Ordinal); Assert.Contains("FinOpsSeparatelyMonitoredNote.Visibility", viewer, StringComparison.Ordinal); diff --git a/Darling/Darling.Tests/SerialLoopStoreSizeSourceTests.cs b/Darling/Darling.Tests/SerialLoopStoreSizeSourceTests.cs index 6d6190aba1..2cd04f4b93 100644 --- a/Darling/Darling.Tests/SerialLoopStoreSizeSourceTests.cs +++ b/Darling/Darling.Tests/SerialLoopStoreSizeSourceTests.cs @@ -42,13 +42,14 @@ namespace Darling.Tests; /// its SQL from another type: it pins the reference in source AND the referent's shipped value at /// runtime, so neither a retyped copy nor a changed constant can slip past. /// — total for SQL written INLINE in any of -/// the ten members. A member that referenced a size-scaling constant declared elsewhere would not match, +/// the eleven members. A member that referenced a size-scaling constant declared elsewhere would not match, /// which is what the tree census below is for. /// — total for /// pg_database_size anywhere in shipped source, wherever the string lives, so no indirection /// evades it. Narrower in function coverage, wider in reach; the two overlap deliberately. /// /// +[Trait("Reads", "Lite")] public sealed class SerialLoopStoreSizeSourceTests { /// @@ -207,7 +208,7 @@ public void TheDiskCheckExecutesTheRecordedSizeConstant() Assert.Contains("StoreSelfMetrics.LatestStoreSizeSql", code, StringComparison.Ordinal); - /* And the deadline regime is unchanged — this member is still one of the ten, so the chain + /* And the deadline regime is unchanged — this member is still one of the eleven, so the chain arithmetic StartupCommandTimeoutTests defends still describes the same population. Asserted here too because the fix would look equally "done" if the site had been moved out of the regime instead, and that is a different change with a different justification. */ @@ -237,7 +238,7 @@ heap happened to be in. A LIMIT-less version returns every whole-store row ever } /// - /// The category invariant, over the ten members + /// The category invariant, over the eleven members /// bounds: none of them may INLINE a read whose /// cost scales with the store. /// @@ -266,7 +267,7 @@ public void NoSerialLoopMemberInlinesAStoreScalingRead() back empty — or short by a member — would make the assertion below pass by scanning less than it claims, which is the failure this file exists to make impossible for the deadline census. Asserted as the site TOTAL rather than as a member count, and tied back to the census's own - independently-asserted number instead of to a literal: nine members hold the ten commands, so a + independently-asserted number instead of to a literal: eleven members hold the twelve commands, so a count of members is arithmetic nobody can check by eye. */ Assert.Equal(StartupCommandTimeoutTests.ExpectedSerialLoopSites, sites); Assert.True(members > 0, "the serial-loop member projection came back empty"); diff --git a/Darling/Darling.Tests/ServerAddRouteTests.cs b/Darling/Darling.Tests/ServerAddRouteTests.cs new file mode 100644 index 0000000000..d5549bb2cd --- /dev/null +++ b/Darling/Darling.Tests/ServerAddRouteTests.cs @@ -0,0 +1,586 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Net; +using System.Net.Http; +using System.Text; +using System.Text.Json; +using System.Text.Json.Nodes; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.TestHost; +using Microsoft.Extensions.Logging; +using Npgsql; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Hosting; +using PerformanceMonitor.Darling.Service.Mcp; +using Xunit; + +namespace Darling.Tests; + +/// +/// POST /api/servers (#4843) through a real host over a store that is never opened: the route's gates, +/// limits, audit line and credential handling, with the add core stood in by a stub so no database is needed. +/// The core itself (validation, dedupe, probe, write) is covered by DarlingMcpServerAdminToolsTests, and the +/// grant it writes under by ServerAddViewerRoleLiveTests. Every test that sends a credential sends a +/// recognisable fake one and asserts it appears in no response and no log line. +/// +public sealed class ServerAddRouteTests +{ + private const string FakeSecret = "Zq9-fake-secret-must-never-appear-7XK"; + + private const string AddedAnswer = + "{\"requested\":1,\"added\":1,\"skipped\":0,\"collided\":0,\"failed\":0,\"results\":[{\"server\":\"sql01\",\"status\":\"added\",\"detail\":\"Connected\"}]}"; + + private sealed class Rig : IAsyncDisposable + { + public required WebApplication App { get; init; } + public required HttpClient Client { get; init; } + public required CapturingTestLogger Log { get; init; } + public required NpgsqlDataSource Source { get; init; } + public required List Bodies { get; init; } + + public async ValueTask DisposeAsync() + { + Client.Dispose(); + await App.DisposeAsync(); + await Source.DisposeAsync(); + } + } + + /// A host with the route mapped over a stub core. A request's seat comes from its X-Seat header + /// (viewer is read-only, admin or absent edits); also runs the + /// host's group-level write gate ahead of the route, as the production pipeline does. + private static async Task StartAsync(Func> core, bool useWriteGate = false, TimeSpan? addTimeout = null) + { + var builder = WebApplication.CreateBuilder(); + builder.WebHost.UseTestServer(); + builder.Logging.ClearProviders(); + var app = builder.Build(); + var log = new CapturingTestLogger(); + var source = NpgsqlDataSource.Create("Host=localhost;Database=never_opened;Username=nobody"); + var bodies = new List(); + + app.Use(async (context, next) => + { + var seat = context.Request.Headers["X-Seat"] == "viewer" + ? new DarlingWebSeat("bob", false) + : new DarlingWebSeat(Uri.UnescapeDataString(context.Request.Headers["X-Principal"].FirstOrDefault() ?? "alice"), true); + context.Items[DarlingWebSeat.HttpContextItemKey] = seat; + if (useWriteGate && !DarlingWebSeat.IsRequestAllowed(seat, context.Request.Method, context.Request.Path.Value ?? "/")) + { + context.Response.StatusCode = StatusCodes.Status403Forbidden; + return; + } + + await next(context); + }); + DarlingWebEndpoints.MapServers(app, source, log, body => + { + bodies.Add(body); + return core(body); + }, addTimeout); + await app.StartAsync(TestContext.Current.CancellationToken); + return new Rig { App = app, Client = app.GetTestClient(), Log = log, Source = source, Bodies = bodies }; + } + + private static async Task<(HttpStatusCode Status, string Body)> PostAsync( + Rig rig, string body, string mediaType = "application/json", string? seat = null, string? principal = null) + { + var ct = TestContext.Current.CancellationToken; + using var request = new HttpRequestMessage(HttpMethod.Post, "/api/servers") + { + Content = new StringContent(body, Encoding.UTF8, mediaType), + }; + if (seat is not null) + { + request.Headers.Add("X-Seat", seat); + } + + if (principal is not null) + { + request.Headers.Add("X-Principal", Uri.EscapeDataString(principal)); + } + + using var response = await rig.Client.SendAsync(request, ct); + return (response.StatusCode, await response.Content.ReadAsStringAsync(ct)); + } + + /// Waits for a captured log line that contains . A late audit line is written by + /// a thread-pool continuation after the 503 was answered, so a test waits for it and never asserts the instant it + /// releases the core. + private static async Task WaitForLogLineAsync(Rig rig, string fragment) + { + var deadline = DateTime.UtcNow.AddSeconds(10); + while (!rig.Log.Lines.Any(l => l.Contains(fragment, StringComparison.Ordinal))) + { + Assert.True(DateTime.UtcNow < deadline, $"no log line contained \"{fragment}\"; the log held: {rig.Log.Joined}"); + await Task.Delay(20, TestContext.Current.CancellationToken); + } + } + + private static string Entries(int count, string password = FakeSecret) => + "[" + string.Join(",", Enumerable.Range(0, count).Select(i => + $"{{\"host\":\"sql{i:D2}\",\"auth\":\"SQL\",\"username\":\"monitor\",\"password\":\"{password}\"}}")) + "]"; + + private static void AssertNoSecret(Rig rig, string responseBody) + { + Assert.DoesNotContain(FakeSecret, responseBody, StringComparison.Ordinal); + Assert.DoesNotContain(FakeSecret, rig.Log.Joined, StringComparison.Ordinal); + } + + /* ═══════════════════════════ gates ═══════════════════════════ */ + + [Theory] + [InlineData("text/plain")] + [InlineData("application/x-www-form-urlencoded")] + [InlineData("multipart/form-data")] + public async Task ANonJsonContentType_Answers415_AndReachesNoCore(string mediaType) + { + await using var rig = await StartAsync(_ => Task.FromResult(AddedAnswer)); + var (status, body) = await PostAsync(rig, Entries(1), mediaType); + Assert.Equal(HttpStatusCode.UnsupportedMediaType, status); + Assert.Empty(rig.Bodies); + AssertNoSecret(rig, body); + } + + [Fact] + public async Task AReadOnlySeat_Answers403_AndReachesNoCore_WithOrWithoutTheHostWriteGate() + { + foreach (var gate in new[] { false, true }) + { + await using var rig = await StartAsync(_ => Task.FromResult(AddedAnswer), useWriteGate: gate); + var (status, body) = await PostAsync(rig, Entries(1), seat: "viewer"); + Assert.Equal(HttpStatusCode.Forbidden, status); + Assert.Empty(rig.Bodies); + AssertNoSecret(rig, body); + } + } + + [Fact] + public async Task AnEditingSeat_ReachesTheCore_WithTheBodyUnchanged_AndGetsTheCoresAnswer() + { + await using var rig = await StartAsync(_ => Task.FromResult(AddedAnswer), useWriteGate: true); + var request = Entries(1); + var (status, body) = await PostAsync(rig, request); + Assert.Equal(HttpStatusCode.OK, status); + Assert.Equal(request, Assert.Single(rig.Bodies)); + Assert.Equal("added", JsonNode.Parse(body)!["results"]![0]!["status"]!.GetValue()); + AssertNoSecret(rig, body); + } + + /* ═══════════════════════════ limits ═══════════════════════════ */ + + [Fact] + public async Task TwentyEntriesPass_AndTwentyOneAreRefused400_BeforeTheCore() + { + await using var rig = await StartAsync(_ => Task.FromResult(AddedAnswer)); + + Assert.Equal(20, DarlingWebEndpoints.MaxServersPerAddRequest); + var ok = await PostAsync(rig, Entries(20)); + Assert.Equal(HttpStatusCode.OK, ok.Status); + Assert.Single(rig.Bodies); + + var tooMany = await PostAsync(rig, Entries(21)); + Assert.Equal(HttpStatusCode.BadRequest, tooMany.Status); + Assert.Single(rig.Bodies); + AssertNoSecret(rig, tooMany.Body); + Assert.Contains("20", tooMany.Body, StringComparison.Ordinal); + } + + [Fact] + public async Task ASecondAdd_WhileOneIsRunning_Answers429_AndTheFirstStillCompletes() + { + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await using var rig = await StartAsync(async _ => + { + entered.TrySetResult(); + await release.Task; + return AddedAnswer; + }); + + var first = PostAsync(rig, Entries(1)); + await entered.Task.WaitAsync(TimeSpan.FromSeconds(30), TestContext.Current.CancellationToken); + + var second = await PostAsync(rig, Entries(1)); + Assert.Equal((HttpStatusCode)429, second.Status); + AssertNoSecret(rig, second.Body); + + release.SetResult(); + Assert.Equal(HttpStatusCode.OK, (await first).Status); + + /* The gate is released: a later add runs. */ + Assert.Equal(HttpStatusCode.OK, (await PostAsync(rig, Entries(1))).Status); + } + + [Fact] + public async Task TheInFlightGate_IsReleased_WhenTheCoreFaults() + { + var calls = 0; + await using var rig = await StartAsync(_ => ++calls == 1 + ? throw new InvalidOperationException("boom " + FakeSecret) + : Task.FromResult(AddedAnswer)); + + var faulted = await PostAsync(rig, Entries(1)); + Assert.Equal(HttpStatusCode.InternalServerError, faulted.Status); + AssertNoSecret(rig, faulted.Body); + Assert.Equal(HttpStatusCode.OK, (await PostAsync(rig, Entries(1))).Status); + } + + [Fact] + public async Task AnOversizedBody_Answers400_WithoutTheBody() + { + await using var rig = await StartAsync(_ => Task.FromResult(AddedAnswer)); + var huge = "[{\"host\":\"h\",\"password\":\"" + FakeSecret + new string('x', DarlingWebEndpoints.MaxServerAddBodyBytes) + "\"}]"; + var (status, body) = await PostAsync(rig, huge); + Assert.Equal(HttpStatusCode.BadRequest, status); + Assert.Empty(rig.Bodies); + AssertNoSecret(rig, body); + } + + /* ═══════════════════════════ credentials ═══════════════════════════ */ + + [Theory] + [InlineData("[{\"host\":\"h\",\"password\":\"" + FakeSecret + "\"")] // truncated + [InlineData("[{\"host\":\"h\",\"password\":\"" + FakeSecret + "\"} nope]")] // junk after a secret + [InlineData("{\"host\":\"h\",\"password\":\"" + FakeSecret + "\"}")] // an object, not an array + [InlineData("[\"" + FakeSecret + "\"]")] // an entry that is not an object + [InlineData("[{\"host\":\"h\",\"password\":\"" + FakeSecret + "\",\"password\":\"" + FakeSecret + "\"}]")] // duplicate field + [InlineData("[]")] + public async Task EveryMalformedBody_Answers400_ThatEchoesNothingOfTheBody(string malformed) + { + await using var rig = await StartAsync(_ => Task.FromResult(AddedAnswer)); + var (status, body) = await PostAsync(rig, malformed); + Assert.Equal(HttpStatusCode.BadRequest, status); + Assert.Empty(rig.Bodies); + AssertNoSecret(rig, body); + Assert.Equal(0, rig.Log.CountAtLevel(LogLevel.Error)); + using var doc = JsonDocument.Parse(body); + Assert.Equal(JsonValueKind.String, doc.RootElement.GetProperty("error").ValueKind); + } + + [Fact] + public async Task TheCoresCaughtExceptionEnvelope_NeverReachesTheWire_NorTheLog() + { + var envelope = McpHelpers.FormatError("add_servers", new InvalidOperationException("could not use password " + FakeSecret)); + await using var rig = await StartAsync(_ => Task.FromResult(envelope)); + + var (status, body) = await PostAsync(rig, Entries(1)); + Assert.Equal(HttpStatusCode.InternalServerError, status); + AssertNoSecret(rig, body); + } + + [Fact] + public async Task ASecretThatSomehowReachesTheCoresAnswer_IsRedactedBeforeItIsWritten() + { + var leaky = "{\"requested\":1,\"added\":0,\"skipped\":0,\"collided\":0,\"failed\":1,\"results\":[" + + "{\"server\":\"sql00\",\"status\":\"connection_failed\",\"detail\":\"login failed for password " + FakeSecret + "\"}]}"; + await using var rig = await StartAsync(_ => Task.FromResult(leaky)); + + var (status, body) = await PostAsync(rig, Entries(1)); + Assert.Equal(HttpStatusCode.OK, status); + AssertNoSecret(rig, body); + Assert.Equal("login failed for password [redacted]", DetailOf(body)); + Assert.Contains("login failed for password [redacted]", rig.Log.Joined, StringComparison.Ordinal); + } + + [Fact] + public async Task TheWholeRequestRefusal_IsA400_AndCarriesNoSecret() + { + var refusal = "{\"status\":\"invalid\",\"message\":\"servers_json is not valid JSON near " + FakeSecret + "\"}"; + await using var rig = await StartAsync(_ => Task.FromResult(refusal)); + + var (status, body) = await PostAsync(rig, Entries(1)); + Assert.Equal(HttpStatusCode.BadRequest, status); + AssertNoSecret(rig, body); + } + + [Fact] + public void RedactAddAnswer_KeepsTheEnvelopeShape_AndReplacesOnlyTheSecret() + { + var answer = DarlingWebEndpoints.RedactAddAnswer( + "{\"requested\":1,\"results\":[{\"server\":\"a\",\"status\":\"invalid\",\"detail\":\"x " + FakeSecret + " y\"}]}", + new[] { FakeSecret }); + var row = JsonNode.Parse(answer)!["results"]![0]!; + Assert.Equal("x [redacted] y", row["detail"]!.GetValue()); + Assert.Equal("a", row["server"]!.GetValue()); + Assert.Equal("invalid", row["status"]!.GetValue()); + + const string untouched = "{\"requested\":1}"; + Assert.Equal(untouched, DarlingWebEndpoints.RedactAddAnswer(untouched, Array.Empty())); + } + + /* ═══════════════════════════ audit ═══════════════════════════ */ + + [Fact] + public async Task EachAddedServer_IsLoggedOnce_WithThePrincipalTheNameAndTheAuthMode_AndNeverTheSecret() + { + var answer = "{\"requested\":3,\"added\":2,\"skipped\":0,\"collided\":0,\"failed\":1,\"results\":[" + + "{\"server\":\"sql00\",\"status\":\"added\",\"detail\":\"ok\"}," + + "{\"server\":\"win01\",\"status\":\"added\",\"detail\":\"ok\"}," + + "{\"server\":\"sql02\",\"status\":\"connection_failed\",\"detail\":\"no\"}]}"; + await using var rig = await StartAsync(_ => Task.FromResult(answer)); + var request = "[{\"host\":\"sql00\",\"auth\":\"SQL\",\"username\":\"u\",\"password\":\"" + FakeSecret + "\"}," + + "{\"host\":\"win01\"}," + + "{\"host\":\"sql02\",\"auth\":\"SQL\",\"username\":\"u\",\"password\":\"" + FakeSecret + "\"}]"; + + var (status, body) = await PostAsync(rig, request); + Assert.Equal(HttpStatusCode.OK, status); + + var added = rig.Log.Lines.Where(l => l.StartsWith("Information: Server added by", StringComparison.Ordinal)).ToList(); + Assert.Equal(2, added.Count); + Assert.Contains("Server added by alice: sql00, auth SQL", added[0], StringComparison.Ordinal); + Assert.Contains("Server added by alice: win01, auth Windows", added[1], StringComparison.Ordinal); + Assert.DoesNotContain(added, l => l.Contains("sql02", StringComparison.Ordinal)); + AssertNoSecret(rig, body); + } + + [Fact] + public async Task ANothingAddedAnswer_LogsNoAuditLine() + { + var answer = "{\"requested\":1,\"added\":0,\"skipped\":1,\"collided\":0,\"failed\":0,\"results\":[{\"server\":\"sql00\",\"status\":\"duplicate\",\"detail\":\"d\"}]}"; + await using var rig = await StartAsync(_ => Task.FromResult(answer)); + Assert.Equal(HttpStatusCode.OK, (await PostAsync(rig, Entries(1))).Status); + Assert.Empty(rig.Log.Lines); + } + + /* ═══════════════════════ failure text ═══════════════════════ */ + + private static string Failed(string status, string detail) => + "{\"requested\":1,\"added\":0,\"skipped\":0,\"collided\":0,\"failed\":1,\"results\":[{\"server\":\"sql00\",\"status\":\"" + status + + "\",\"detail\":" + JsonSerializer.Serialize(detail) + "}]}"; + + private static string DetailOf(string body) => + JsonNode.Parse(body)!["results"]![0]!["detail"]!.GetValue(); + + [Theory] + [InlineData("connection_failed", "Could not connect to sql00,1433. The service log has the details.")] + [InlineData("connection_failed", "Could not connect to sql01,1433. The service log has the details.")] + [InlineData("not_saved", "Not saved: the write was refused")] + public async Task AFailureRow_CarriesTheCoresOwnText_ToTheCaller_AndTheLog(string rowStatus, string detail) + { + await using var rig = await StartAsync(_ => Task.FromResult(Failed(rowStatus, detail))); + var (status, body) = await PostAsync(rig, Entries(1)); + Assert.Equal(HttpStatusCode.OK, status); + Assert.Equal(detail, DetailOf(body)); + Assert.Contains(rig.Log.Lines, l => l.StartsWith("Information: Server add failed", StringComparison.Ordinal) + && l.Contains(detail, StringComparison.Ordinal)); + AssertNoSecret(rig, body); + } + + [Theory] + [InlineData("duplicate", "Already monitored.")] + [InlineData("invalid", "host is required.")] + [InlineData("added", "Connected to PostgreSQL 16")] + public async Task ValidationAndSuccessDetails_PassThrough(string status, string detail) + { + await using var rig = await StartAsync(_ => Task.FromResult(Failed(status, detail))); + var (_, body) = await PostAsync(rig, Entries(1)); + Assert.Equal(detail, DetailOf(body)); + } + + /* ═══════════════════════ slot timeout ═══════════════════════ */ + + [Fact] + public async Task ACoreThatNeverFinishes_Answers503_AndKeepsTheSlot_UntilTheAddFinishes() + { + var hang = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var calls = 0; + await using var rig = await StartAsync( + _ => ++calls == 1 ? hang.Task : Task.FromResult(AddedAnswer), addTimeout: TimeSpan.FromMilliseconds(200)); + + var (status, body) = await PostAsync(rig, Entries(1)); + Assert.Equal(HttpStatusCode.ServiceUnavailable, status); + Assert.Contains(DarlingWebEndpoints.ServerAddTimedOutText, body, StringComparison.Ordinal); + AssertNoSecret(rig, body); + + /* The first core call is still outstanding: the slot is still held. */ + Assert.Equal(HttpStatusCode.TooManyRequests, (await PostAsync(rig, Entries(1))).Status); + Assert.Equal(1, calls); + + hang.SetResult(AddedAnswer); + var accepted = HttpStatusCode.TooManyRequests; + for (var i = 0; i < 50 && accepted == HttpStatusCode.TooManyRequests; i++) + { + accepted = (await PostAsync(rig, Entries(1))).Status; + if (accepted == HttpStatusCode.TooManyRequests) + { + await Task.Delay(50, TestContext.Current.CancellationToken); + } + } + + Assert.Equal(HttpStatusCode.OK, accepted); + } + + /// The 503 told the client nothing about the add, but the core commits when it finishes: its audit lines + /// (the added server, the failed row with the secret redacted) are written then, for the principal who sent the + /// add, and not before. + [Fact] + public async Task AnAddThatOutlivesTheTimeout_StillWritesItsAuditLines_ForThePrincipalWhoSentIt_WhenItFinishes() + { + var hang = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await using var rig = await StartAsync(_ => hang.Task, addTimeout: TimeSpan.FromMilliseconds(200)); + + var (status, body) = await PostAsync(rig, Entries(2), principal: "dana"); + Assert.Equal(HttpStatusCode.ServiceUnavailable, status); + AssertNoSecret(rig, body); + Assert.DoesNotContain(rig.Log.Lines, l => l.Contains("Server added by", StringComparison.Ordinal)); + + /* sql00 committed after the 503, and sql01's probe failed with the secret in the driver's text. */ + hang.SetResult("{\"requested\":2,\"added\":1,\"skipped\":0,\"collided\":0,\"failed\":1,\"results\":[" + + "{\"server\":\"sql00\",\"status\":\"added\",\"detail\":\"Connected\"}," + + "{\"server\":\"sql01\",\"status\":\"connection_failed\",\"detail\":\"login failed for password " + FakeSecret + "\"}]}"); + await WaitForLogLineAsync(rig, "Server added by"); + + var added = Assert.Single(rig.Log.Lines, l => l.Contains("Server added by", StringComparison.Ordinal)); + Assert.Contains("Server added by dana: sql00, auth SQL", added, StringComparison.Ordinal); + var failed = Assert.Single(rig.Log.Lines, l => l.Contains("Server add failed for", StringComparison.Ordinal)); + Assert.Contains("Server add failed for dana: sql01, connection_failed: login failed for password [redacted]", failed, StringComparison.Ordinal); + AssertNoSecret(rig, body); + } + + /* ═══════════════════════ slot release ═══════════════════════ */ + + /// Cycles for the slot-release stress pins. The defect is a race between the handler's answer and the + /// slot release, so one pass proves little; the fixed code frees the slot before the answer exists, so every + /// cycle passes by construction and a regression shows up as a refusal within the loop. + private const int SlotCycles = 400; + + [Fact] + public async Task AnAddThatFinishedInsideTheWait_FreesTheSlotBeforeItsAnswer_SoAnImmediateSecondAddIsNotRefused() + { + await using var rig = await StartAsync(_ => Task.FromResult(AddedAnswer)); + + for (var i = 0; i < SlotCycles; i++) + { + var status = (await PostAsync(rig, Entries(1))).Status; + Assert.True(status == HttpStatusCode.OK, $"add {i + 1} of {SlotCycles} answered {(int)status} right after a finished add"); + } + } + + [Fact] + public async Task AnAddThatFinishedInsideTheWait_ReleasesTheSlotExactlyOnce_SoALaterHeldAddStillBlocksTheNext() + { + var calls = 0; + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await using var rig = await StartAsync(_ => + { + if (Interlocked.Increment(ref calls) % 2 == 1) + { + return Task.FromResult(AddedAnswer); + } + + entered.TrySetResult(); + return release.Task.ContinueWith(_ => AddedAnswer, TaskScheduler.Default); + }); + + for (var i = 0; i < SlotCycles; i++) + { + /* A finishes inside the wait: its handler and its own continuation both reach for the release. */ + Assert.Equal(HttpStatusCode.OK, (await PostAsync(rig, Entries(1))).Status); + + /* B takes the freed slot and holds it. A's continuation may still be in flight; a second release + from it would free B's slot while B runs, and C would be let in. */ + entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var held = PostAsync(rig, Entries(1)); + await entered.Task.WaitAsync(TimeSpan.FromSeconds(30), TestContext.Current.CancellationToken); + + Assert.Equal(HttpStatusCode.TooManyRequests, (await PostAsync(rig, Entries(1))).Status); + Assert.Equal(2 * i + 2, Volatile.Read(ref calls)); + + release.SetResult(); + Assert.Equal(HttpStatusCode.OK, (await held).Status); + } + } + + [Fact] + public async Task AnAddThatOutlivesTheTimeout_KeepsTheSlotForEveryLaterAdd_AndFreesItOnce_WhenItFinishes() + { + var hang = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var heldEntered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var heldRelease = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var calls = 0; + await using var rig = await StartAsync(_ => + { + switch (Interlocked.Increment(ref calls)) + { + case 1: + return hang.Task; + case 3: + heldEntered.TrySetResult(); + return heldRelease.Task; + default: + return Task.FromResult(AddedAnswer); + } + }, addTimeout: TimeSpan.FromMilliseconds(200)); + + Assert.Equal(HttpStatusCode.ServiceUnavailable, (await PostAsync(rig, Entries(1))).Status); + + /* The handler has answered and gone: the add is still running, so the slot is still held. */ + for (var i = 0; i < 5; i++) + { + Assert.Equal(HttpStatusCode.TooManyRequests, (await PostAsync(rig, Entries(1))).Status); + } + + Assert.Equal(1, Volatile.Read(ref calls)); + + /* The add finishes: the slot is freed, by the add's own continuation. */ + hang.SetResult(AddedAnswer); + var accepted = HttpStatusCode.TooManyRequests; + for (var i = 0; i < 100 && accepted == HttpStatusCode.TooManyRequests; i++) + { + accepted = (await PostAsync(rig, Entries(1))).Status; + if (accepted == HttpStatusCode.TooManyRequests) + { + await Task.Delay(50, TestContext.Current.CancellationToken); + } + } + + Assert.Equal(HttpStatusCode.OK, accepted); + + /* Freed once: the next add takes the slot and a further one is refused, not let in beside it. */ + var held = PostAsync(rig, Entries(1)); + await heldEntered.Task.WaitAsync(TimeSpan.FromSeconds(30), TestContext.Current.CancellationToken); + Assert.Equal(HttpStatusCode.TooManyRequests, (await PostAsync(rig, Entries(1))).Status); + + heldRelease.SetResult(AddedAnswer); + Assert.Equal(HttpStatusCode.OK, (await held).Status); + } + + /* ═══════════════════════ principal on one line ═══════════════════════ */ + + [Fact] + public async Task APrincipalWithControlCharacters_IsLoggedOnOneLine() + { + await using var rig = await StartAsync(_ => Task.FromResult(AddedAnswer)); + var ct = TestContext.Current.CancellationToken; + using var request = new HttpRequestMessage(HttpMethod.Post, "/api/servers") + { + Content = new StringContent(Entries(1), Encoding.UTF8, "application/json"), + }; + request.Headers.TryAddWithoutValidation("X-Principal", "eve%0D%0Aevil%09x"); + using var response = await rig.Client.SendAsync(request, ct); + Assert.Equal(HttpStatusCode.OK, response.StatusCode); + + var line = Assert.Single(rig.Log.Lines, l => l.Contains("Server added by", StringComparison.Ordinal)); + Assert.DoesNotContain('\n', line); + Assert.DoesNotContain('\r', line); + Assert.DoesNotContain('\t', line); + Assert.Contains("eve", line, StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/ServerAddViewerRoleLiveTests.cs b/Darling/Darling.Tests/ServerAddViewerRoleLiveTests.cs new file mode 100644 index 0000000000..39da10eba1 --- /dev/null +++ b/Darling/Darling.Tests/ServerAddViewerRoleLiveTests.cs @@ -0,0 +1,514 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Security.Cryptography; +using System.Text.Json.Nodes; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: this class mints its own scratch database through ScratchPostgres and a role with a per-run + name, and touches nothing on the shared one, so it is not serialized against the live-postgres collection. */ + +/// +/// The server-onboarding write as a non-superuser holding exactly the grants the managed viewer role gets +/// (#4843): the web dashboard's POST /api/servers runs the add_servers core on the viewer-role pool, and +/// every other live test connects as the owner, which can write anything, so a missing grant would pass them all +/// and fail only in a real deployment with a 42501. The statements are TAKEN FROM +/// 's output with the role name swapped, so removing the +/// grant from the product's SQL makes the add test fail. +/// +[Collection("darling-owned-secrets")] +public sealed class ServerAddViewerRoleLiveTests : IDisposable +{ + private readonly DarlingOwnedSet _ownedBefore = DarlingOwnedSecrets.Current; + + /* Configuration loaded, owning nothing: an unpopulated set refuses every env:/file: reference (R3-3), and the + entries here use one. */ + public ServerAddViewerRoleLiveTests() => DarlingOwnedSecrets.Set(DarlingOwnedSet.Empty); + + public void Dispose() => DarlingOwnedSecrets.Set(_ownedBefore); + + internal static readonly string RolePassword = Convert.ToHexString(RandomNumberGenerator.GetBytes(16)); + + private const string ServerGrant = "GRANT INSERT ON config.config_monitored_servers TO viewer;"; + + /// Pass as skip to provision a role WITHOUT the edit function (a store whose roles predate it). + internal const string MissingEditFunction = ""; + + private static readonly DarlingMcpServerAdminTools.ServerProbe Reachable = (_, _) => Task.FromResult( + new ConnectionProbeResult( + Success: true, MajorVersion: 15, EngineEdition: 3, EngineEditionDescription: "Enterprise", + IsAzureSqlDb: false, IsAzureManagedInstance: false, IsAwsRds: false, HasMsdbAccess: true, Error: null)); + + /// The scratch database a connection string names (#5560): what a replayed database-level GRANT is aimed at, + /// so no replay rewrites the shared cluster's own darling database. + private static string DatabaseOf(string connectionString) => + new NpgsqlConnectionStringBuilder(connectionString).Database!; + + /// Every GRANT / REVOKE in the managed batch that names viewer (alone or in a list), in order, + /// retargeted at and aimed at . + /// drops statements by exact text. + internal static List ViewerStatements(string roleName, string database, string? skip) + { + var provisioning = DarlingManagedRoles.BuildProvisioningSql( + ProvisioningTestSecrets.Admin, ProvisioningTestSecrets.Viewer, ProvisioningTestSecrets.Mcp); + var statements = new List(); + + /* Drop the comment lines BEFORE splitting: the provisioning prose contains semicolons. */ + var uncommented = string.Join('\n', provisioning.Split('\n').Where(l => !l.TrimStart().StartsWith("--", StringComparison.Ordinal))); + foreach (var raw in uncommented.Split(';')) + { + var statement = raw.Trim(); + var isGrant = statement.StartsWith("GRANT ", StringComparison.Ordinal); + if (!isGrant && !statement.StartsWith("REVOKE ", StringComparison.Ordinal)) + { + continue; + } + + var marker = isGrant ? " TO " : " FROM "; + var at = statement.LastIndexOf(marker, StringComparison.Ordinal); + if (at < 0 || statement.Contains("EXECUTE ON FUNCTION", StringComparison.Ordinal)) + { + continue; + } + + if (!statement[(at + marker.Length)..].Split(',').Select(t => t.Trim()).Contains("viewer", StringComparer.Ordinal)) + { + continue; + } + + if (skip is not null && string.Equals(statement + ";", skip, StringComparison.Ordinal)) + { + continue; + } + + statements.Add(ViewerGrantReplay.OnDatabase(statement[..at], database) + marker + roleName); + } + + return statements; + } + + internal static async Task<(ScratchPostgres Scratch, NpgsqlDataSource Owner, string OwnerString)> OpenAsync(CancellationToken ct) + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live viewer-role server-add test (it mints its own scratch database and role)."); + + var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var ownerString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { SearchPath = PgSchemaGenerator.SearchPath }.ConnectionString; + await using (var connection = new NpgsqlConnection(ownerString)) + { + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, null, ct); + } + + return (scratch, NpgsqlDataSource.Create(ownerString), ownerString); + } + + internal static async Task ProvisionAsync( + NpgsqlDataSource owner, string ownerString, string roleName, string? skip, CancellationToken ct) + { + await ExecAsync(owner, $"CREATE ROLE {roleName} LOGIN NOSUPERUSER PASSWORD '{RolePassword}'", ct); + foreach (var statement in ViewerStatements(roleName, DatabaseOf(ownerString), skip)) + { + await ExecAsync(owner, statement, ct); + } + + /* The edit function (#5240), from the shared builder so it is the real one, owned by the owner; EXECUTE for the role. */ + if (!string.Equals(skip, MissingEditFunction, StringComparison.Ordinal)) + { + await ExecAsync(owner, DarlingManagedRoles.BuildEditMonitoredServerFunctionSql("config"), ct); + await ExecAsync(owner, DarlingManagedRoles.BuildEditMonitoredServerLegacyWrapperSql("config"), ct); + await ExecAsync(owner, $"GRANT EXECUTE ON FUNCTION config.edit_monitored_server({DarlingManagedRoles.EditMonitoredServerSignature}) TO {roleName}", ct); + await ExecAsync(owner, $"GRANT EXECUTE ON FUNCTION config.edit_monitored_server({DarlingManagedRoles.EditMonitoredServerLegacySignature}) TO {roleName}", ct); + } + + return NpgsqlDataSource.Create(new NpgsqlConnectionStringBuilder(ownerString) { Username = roleName, Password = RolePassword }.ConnectionString); + } + + private const string OneServer = + "[{\"host\":\"added-by-viewer\",\"auth\":\"SQL\",\"username\":\"monitor\",\"password\":\"Viewer-Add-Pw-1\"}]"; + + [Fact] + public async Task TheShippedViewerGrants_LetTheAddCoreInsertAServer_AndTheBeaconTriggerFire() + { + var ct = TestContext.Current.CancellationToken; + var roleName = "srv_add_" + Guid.NewGuid().ToString("N")[..8]; + var (scratch, owner, ownerString) = await OpenAsync(ct); + await using var _ = scratch; + await using var __ = owner; + + Assert.Contains(ServerGrant.Replace(" TO viewer;", " TO " + roleName, StringComparison.Ordinal), ViewerStatements(roleName, DatabaseOf(ownerString), null)); + await using var asViewer = await ProvisionAsync(owner, ownerString, roleName, null, ct); + var bodySucceeded = false; + try + { + /* The beacon row is seeded by the running service, not by migration; the trigger updates nothing without it. */ + await ExecAsync(owner, "INSERT INTO config_service (id) VALUES (1) ON CONFLICT DO NOTHING", ct); + long versionBefore; + await using (var read = owner.CreateCommand("SELECT config_version FROM config_service")) + { + versionBefore = Convert.ToInt64(await read.ExecuteScalarAsync(ct)); + } + + var answer = JsonNode.Parse(await DarlingMcpServerAdminTools.AddServersAsync(asViewer, OneServer, Reachable, ct, TestKeyRings.Healthy))!; + Assert.Equal(1, answer["added"]!.GetValue()); + Assert.Equal("added", answer["results"]![0]!["status"]!.GetValue()); + + await using (var count = owner.CreateCommand("SELECT count(*) FROM config_monitored_servers WHERE host = 'added-by-viewer'")) + { + Assert.Equal(1L, Convert.ToInt64(await count.ExecuteScalarAsync(ct))); + } + + /* The write fired trg_bump_monitored_servers AS viewer, through the two-column config_service grant. */ + await using (var read = owner.CreateCommand("SELECT config_version FROM config_service")) + { + Assert.True(Convert.ToInt64(await read.ExecuteScalarAsync(ct)) > versionBefore); + } + + /* The grant is INSERT only: the viewer role cannot delete the definition it inserted (no web route removes one yet). */ + var deleteDenied = await Assert.ThrowsAsync(async () => + { + await using var delete = asViewer.CreateCommand("DELETE FROM config_monitored_servers WHERE host = 'added-by-viewer'"); + await delete.ExecuteNonQueryAsync(ct); + }); + Assert.Equal("42501", deleteDenied.SqlState); + + /* The credential column stays unreadable to this role: INSERT needed no read of it. */ + var denied = await Assert.ThrowsAsync(async () => + { + await using var peek = asViewer.CreateCommand("SELECT encrypted_password FROM config_monitored_servers"); + await peek.ExecuteScalarAsync(ct); + }); + Assert.Equal("42501", denied.SqlState); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(bodySucceeded, () => ExecAsync(owner, $"DROP OWNED BY {roleName}; DROP ROLE IF EXISTS {roleName};", CancellationToken.None)); + } + } + + [Fact] + public async Task WithoutTheServerGrant_TheAddWrite_Fails42501_AndNothingIsSaved() + { + var ct = TestContext.Current.CancellationToken; + var roleName = "srv_nogrant_" + Guid.NewGuid().ToString("N")[..8]; + var (scratch, owner, ownerString) = await OpenAsync(ct); + await using var _ = scratch; + await using var __ = owner; + + await using var asViewer = await ProvisionAsync(owner, ownerString, roleName, ServerGrant, ct); + var bodySucceeded = false; + try + { + /* The statement the core runs, as a role that holds every viewer grant EXCEPT this one. */ + var denied = await Assert.ThrowsAsync(async () => + { + await using var insert = asViewer.CreateCommand(DarlingMcpServerAdminTools.InsertServerSql); + insert.Parameters.Add(new NpgsqlParameter { TypedValue = 7 }); + insert.Parameters.Add(new NpgsqlParameter { TypedValue = "n" }); + insert.Parameters.Add(new NpgsqlParameter { TypedValue = "h" }); + insert.Parameters.Add(new NpgsqlParameter { Value = DBNull.Value, NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Text }); + insert.Parameters.Add(new NpgsqlParameter { TypedValue = "sql" }); + insert.Parameters.Add(new NpgsqlParameter { Value = DBNull.Value, NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Text }); + insert.Parameters.Add(new NpgsqlParameter { Value = DBNull.Value, NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Text }); + insert.Parameters.Add(new NpgsqlParameter { TypedValue = "Mandatory" }); + insert.Parameters.Add(new NpgsqlParameter { TypedValue = false }); + insert.Parameters.Add(new NpgsqlParameter { TypedValue = false }); + insert.Parameters.Add(new NpgsqlParameter { TypedValue = false }); + insert.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Array | NpgsqlTypes.NpgsqlDbType.Text, Value = Array.Empty() }); + insert.Parameters.Add(new NpgsqlParameter { TypedValue = 0m }); + insert.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Timestamp, Value = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified) }); + insert.Parameters.Add(new NpgsqlParameter { TypedValue = "sqlserver" }); + insert.Parameters.Add(new NpgsqlParameter { TypedValue = 0 }); + insert.Parameters.Add(new NpgsqlParameter { Value = DBNull.Value, NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Text }); + insert.Parameters.Add(new NpgsqlParameter { Value = DBNull.Value, NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Text }); + await insert.ExecuteNonQueryAsync(ct); + }); + Assert.Equal("42501", denied.SqlState); + + /* Through the core, the same refusal is a typed not_saved, with no row written. */ + var answer = JsonNode.Parse(await DarlingMcpServerAdminTools.AddServersAsync(asViewer, OneServer, Reachable, ct, TestKeyRings.Healthy))!; + Assert.Equal(0, answer["added"]!.GetValue()); + Assert.Equal("not_saved", answer["results"]![0]!["status"]!.GetValue()); + await using (var count = owner.CreateCommand("SELECT count(*) FROM config_monitored_servers WHERE host IN ('h', 'added-by-viewer')")) + { + Assert.Equal(0L, Convert.ToInt64(await count.ExecuteScalarAsync(ct))); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(bodySucceeded, () => ExecAsync(owner, $"DROP OWNED BY {roleName}; DROP ROLE IF EXISTS {roleName};", CancellationToken.None)); + } + } + + [Fact] + public async Task TheViewerRole_HoldsNoUpdate_OnTheMonitoredServersTable_AndNoWriteOnTheCredentialColumn() + { + var ct = TestContext.Current.CancellationToken; + var roleName = "srv_upd_" + Guid.NewGuid().ToString("N")[..8]; + var (scratch, owner, ownerString) = await OpenAsync(ct); + await using var _ = scratch; + await using var __ = owner; + + await using var asViewer = await ProvisionAsync(owner, ownerString, roleName, null, ct); + var bodySucceeded = false; + try + { + await using (var seed = owner.CreateCommand("INSERT INTO config_monitored_servers (server_id, name, host) VALUES (-4843, 'seed', 'seed-host')")) + { + await seed.ExecuteNonQueryAsync(ct); + } + + foreach (var sql in new[] + { + "UPDATE config_monitored_servers SET is_enabled = FALSE WHERE server_id = -4843", + "UPDATE config_monitored_servers SET encrypted_password = 'x' WHERE server_id = -4843", + }) + { + var denied = await Assert.ThrowsAsync(async () => + { + await using var update = asViewer.CreateCommand(sql); + await update.ExecuteNonQueryAsync(ct); + }); + Assert.Equal("42501", denied.SqlState); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(bodySucceeded, () => ExecAsync(owner, $"DROP OWNED BY {roleName}; DROP ROLE IF EXISTS {roleName};", CancellationToken.None)); + } + } + + private static DarlingMcpServerAdminTools.EditColumnValue Col(string column, NpgsqlTypes.NpgsqlDbType type, object? value) => new(column, column, type, value); + + private static async Task<(string Outcome, DateTime? Token)> CallEditAsync( + NpgsqlDataSource source, int id, DateTime expected, IReadOnlyList sets, CancellationToken ct) + { + await using var call = source.CreateCommand(DarlingMcpServerAdminTools.EditFunctionSql); + foreach (var parameter in DarlingMcpServerAdminTools.BuildEditFunctionParameters(id, expected, sets)) + { + call.Parameters.Add(parameter); + } + + await using var reader = await call.ExecuteReaderAsync(ct); + Assert.True(await reader.ReadAsync(ct)); + return (reader.GetString(0), reader.IsDBNull(1) ? null : reader.GetDateTime(1)); + } + + private static async Task RowAsync(NpgsqlDataSource owner, int id, string columns, CancellationToken ct) + { + await using var read = owner.CreateCommand($"SELECT concat_ws('|', {columns}) FROM config_monitored_servers WHERE server_id = {id}"); + return (string)(await read.ExecuteScalarAsync(ct))!; + } + + /// + /// The edit's one write as the viewer role (#5240): a direct UPDATE of host is denied, and the function refuses a move + /// of host that keeps the stored secret on a SQL or service-principal row (the row is unchanged), saves it with a new + /// secret, answers a stale token as a conflict, saves a name change on a secret row and a Windows-auth host move with no secret. + /// + [Fact] + public async Task AsTheViewerRole_TheEditFunction_RefusesAMoveThatKeepsTheSecret_AndSavesTheRest() + { + var ct = TestContext.Current.CancellationToken; + var roleName = "srv_fn_" + Guid.NewGuid().ToString("N")[..8]; + var (scratch, owner, ownerString) = await OpenAsync(ct); + await using var _ = scratch; + await using var __ = owner; + await using var asViewer = await ProvisionAsync(owner, ownerString, roleName, null, ct); + var bodySucceeded = false; + try + { + await ExecAsync(owner, "INSERT INTO config_service (id) VALUES (1) ON CONFLICT DO NOTHING", ct); + await ExecAsync(owner, "INSERT INTO config_monitored_servers (server_id, name, host, auth, username, encrypted_password) VALUES (7101, 'a', 'a.example.test', 'sql', 'monitor', 'blob-a'), (7102, 'b', 'b.example.test', 'serviceprincipal', 'client-id', 'blob-b'), (7103, 'c', 'c.example.test', 'integrated', NULL, NULL)", ct); + var host = new[] { Col("host", NpgsqlTypes.NpgsqlDbType.Text, "moved.example.test") }; + var secret = Col("encrypted_password", NpgsqlTypes.NpgsqlDbType.Text, "blob-new"); + + var denied = await Assert.ThrowsAsync(async () => + { + await using var update = asViewer.CreateCommand("UPDATE config_monitored_servers SET host = 'moved.example.test' WHERE server_id = 7101"); + await update.ExecuteNonQueryAsync(ct); + }); + Assert.Equal("42501", denied.SqlState); + + foreach (var id in new[] { 7101, 7102 }) + { + var before = await RowAsync(owner, id, "host, encrypted_password, modified_at", ct); + var token = DateTime.Parse(before.Split('|')[2], System.Globalization.CultureInfo.InvariantCulture); + Assert.Equal("password_needed", (await CallEditAsync(asViewer, id, token, host, ct)).Outcome); + Assert.Equal(before, await RowAsync(owner, id, "host, encrypted_password, modified_at", ct)); + } + + await using var stamp = owner.CreateCommand("SELECT modified_at FROM config_monitored_servers WHERE server_id = 7101"); + var read = (DateTime)(await stamp.ExecuteScalarAsync(ct))!; + var saved = await CallEditAsync(asViewer, 7101, read, [host[0], secret], ct); + Assert.Equal("saved", saved.Outcome); + Assert.True(saved.Token > read); + Assert.Equal("moved.example.test|blob-new", await RowAsync(owner, 7101, "host, encrypted_password", ct)); + + Assert.Equal("conflict", (await CallEditAsync(asViewer, 7101, read, [Col("name", NpgsqlTypes.NpgsqlDbType.Text, "late")], ct)).Outcome); + Assert.Equal("not_found", (await CallEditAsync(asViewer, 999999, read, host, ct)).Outcome); + + await using var stamp2 = owner.CreateCommand("SELECT modified_at FROM config_monitored_servers WHERE server_id = 7102"); + var read2 = (DateTime)(await stamp2.ExecuteScalarAsync(ct))!; + Assert.Equal("password_needed", (await CallEditAsync(asViewer, 7102, read2, [Col("encrypt_mode", NpgsqlTypes.NpgsqlDbType.Text, "Optional")], ct)).Outcome); + Assert.Equal("Mandatory|blob-b", await RowAsync(owner, 7102, "encrypt_mode, encrypted_password", ct)); + Assert.Equal("saved", (await CallEditAsync(asViewer, 7102, read2, [Col("name", NpgsqlTypes.NpgsqlDbType.Text, "b-renamed")], ct)).Outcome); + Assert.Equal("b-renamed|blob-b", await RowAsync(owner, 7102, "name, encrypted_password", ct)); + + await using var stamp3 = owner.CreateCommand("SELECT modified_at FROM config_monitored_servers WHERE server_id = 7103"); + var read3 = (DateTime)(await stamp3.ExecuteScalarAsync(ct))!; + Assert.Equal("saved", (await CallEditAsync(asViewer, 7103, read3, host, ct)).Outcome); + Assert.Equal("moved.example.test", await RowAsync(owner, 7103, "host", ct)); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(bodySucceeded, () => ExecAsync(owner, $"DROP OWNED BY {roleName}; DROP ROLE IF EXISTS {roleName};", CancellationToken.None)); + } + } + + /// + /// The edit function refuses the same connection changes the route refuses (#5240): on a secret row, a direct call as the + /// viewer role with no new secret answers password_needed and leaves the row unchanged for each of port, database, + /// read-only intent, authentication, username, encrypt mode, certificate trust and multi-subnet failover. A change of + /// only the display name, and a change that comes with a new secret, still save. + /// + [Fact] + public async Task AsTheViewerRole_TheEditFunction_RefusesEveryConnectionChangeThatKeepsTheSecret() + { + var ct = TestContext.Current.CancellationToken; + var roleName = "srv_fnc_" + Guid.NewGuid().ToString("N")[..8]; + var (scratch, owner, ownerString) = await OpenAsync(ct); + await using var _ = scratch; + await using var __ = owner; + await using var asViewer = await ProvisionAsync(owner, ownerString, roleName, null, ct); + var bodySucceeded = false; + try + { + await ExecAsync(owner, "INSERT INTO config_service (id) VALUES (1) ON CONFLICT DO NOTHING", ct); + await ExecAsync(owner, "INSERT INTO config_monitored_servers (server_id, name, host, port, database, auth, username, encrypted_password, encrypt_mode, trust_server_certificate, read_only_intent, multi_subnet_failover) VALUES (7201, 'a', 'a.example.test', 1433, 'master', 'sql', 'monitor', 'blob-a', 'Mandatory', false, false, false)", ct); + const string cols = "name, host, port, database, auth, username, encrypted_password, encrypt_mode, trust_server_certificate, read_only_intent, multi_subnet_failover, modified_at"; + + var changes = new (string Label, DarlingMcpServerAdminTools.EditColumnValue Change)[] + { + ("port", Col("port", NpgsqlTypes.NpgsqlDbType.Integer, 1444)), + ("database", Col("database", NpgsqlTypes.NpgsqlDbType.Text, "other")), + ("read_only_intent", Col("read_only_intent", NpgsqlTypes.NpgsqlDbType.Boolean, true)), + ("auth", Col("auth", NpgsqlTypes.NpgsqlDbType.Text, "serviceprincipal")), + ("username", Col("username", NpgsqlTypes.NpgsqlDbType.Text, "someone-else")), + ("encrypt_mode", Col("encrypt_mode", NpgsqlTypes.NpgsqlDbType.Text, "Optional")), + ("trust_server_certificate", Col("trust_server_certificate", NpgsqlTypes.NpgsqlDbType.Boolean, true)), + ("multi_subnet_failover", Col("multi_subnet_failover", NpgsqlTypes.NpgsqlDbType.Boolean, true)), + }; + + foreach (var (label, change) in changes) + { + var before = await RowAsync(owner, 7201, cols, ct); + var token = DateTime.Parse(before.Split('|')[^1], System.Globalization.CultureInfo.InvariantCulture); + Assert.True("password_needed" == (await CallEditAsync(asViewer, 7201, token, [change], ct)).Outcome, $"{label} change with no new secret was not refused"); + Assert.Equal(before, await RowAsync(owner, 7201, cols, ct)); + } + + var current = await RowAsync(owner, 7201, cols, ct); + var currentToken = DateTime.Parse(current.Split('|')[^1], System.Globalization.CultureInfo.InvariantCulture); + var renamed = await CallEditAsync(asViewer, 7201, currentToken, [Col("name", NpgsqlTypes.NpgsqlDbType.Text, "renamed")], ct); + Assert.Equal("saved", renamed.Outcome); + Assert.Equal("renamed|blob-a", await RowAsync(owner, 7201, "name, encrypted_password", ct)); + + var withSecret = await CallEditAsync(asViewer, 7201, renamed.Token!.Value, + [Col("database", NpgsqlTypes.NpgsqlDbType.Text, "other"), Col("encrypted_password", NpgsqlTypes.NpgsqlDbType.Text, "blob-new")], ct); + Assert.Equal("saved", withSecret.Outcome); + Assert.Equal("other|blob-new", await RowAsync(owner, 7201, "database, encrypted_password", ct)); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(bodySucceeded, () => ExecAsync(owner, $"DROP OWNED BY {roleName}; DROP ROLE IF EXISTS {roleName};", CancellationToken.None)); + } + } + + /// The batch's REVOKE (#5240) takes away any UPDATE an earlier build gave viewer (column-level or table-level), + /// every run, and a second run changes nothing. + [Fact] + public async Task EveryRunOfTheBatch_TakesAnyViewerUpdateOnTheMonitoredServersTableAway_AndIsIdempotent() + { + var ct = TestContext.Current.CancellationToken; + var roleName = "srv_rst_" + Guid.NewGuid().ToString("N")[..8]; + var (scratch, owner, ownerString) = await OpenAsync(ct); + await using var _ = scratch; + await using var __ = owner; + await using var asViewer = await ProvisionAsync(owner, ownerString, roleName, null, ct); + var bodySucceeded = false; + try + { + Assert.Equal("", await GrantedUpdateColumnsAsync(owner, roleName, ct)); + await ExecAsync(owner, $"GRANT UPDATE (name, host, encrypted_password, modified_at) ON config.config_monitored_servers TO {roleName}", ct); + Assert.NotEqual("", await GrantedUpdateColumnsAsync(owner, roleName, ct)); + await ExecAsync(owner, $"GRANT UPDATE ON config.config_monitored_servers TO {roleName}", ct); + Assert.True(await HasTableLevelUpdateAsync(owner, roleName, ct), "setup: the table-level UPDATE did not take"); + + foreach (var run in new[] { 1, 2 }) + { + await RunViewerStatementsAsync(owner, roleName, ct); + Assert.Equal("", await GrantedUpdateColumnsAsync(owner, roleName, ct)); + Assert.False(await HasTableLevelUpdateAsync(owner, roleName, ct), $"run {run} left a table-level UPDATE"); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(bodySucceeded, () => ExecAsync(owner, $"DROP OWNED BY {roleName}; DROP ROLE IF EXISTS {roleName};", CancellationToken.None)); + } + } + + private static async Task GrantedUpdateColumnsAsync(NpgsqlDataSource owner, string roleName, CancellationToken ct) + { + await using var granted = owner.CreateCommand( + "SELECT COALESCE(string_agg(column_name, ',' ORDER BY column_name COLLATE \"C\"), '') FROM information_schema.column_privileges " + + $"WHERE grantee = '{roleName}' AND table_schema = 'config' AND table_name = 'config_monitored_servers' AND privilege_type = 'UPDATE'"); + return (string)(await granted.ExecuteScalarAsync(ct))!; + } + + private static async Task RunViewerStatementsAsync(NpgsqlDataSource owner, string roleName, CancellationToken ct) + { + foreach (var statement in ViewerStatements(roleName, DatabaseOf(owner.ConnectionString), null)) + { + await ExecAsync(owner, statement, ct); + } + } + + private static async Task HasTableLevelUpdateAsync(NpgsqlDataSource owner, string roleName, CancellationToken ct) + { + await using var tableLevel = owner.CreateCommand($"SELECT has_table_privilege('{roleName}', 'config.config_monitored_servers', 'UPDATE')"); + return (bool)(await tableLevel.ExecuteScalarAsync(ct))!; + } + + private static async Task ExecAsync(NpgsqlDataSource source, string sql, CancellationToken ct) + { + await using var command = source.CreateCommand(sql); + await command.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/ServerConnectPolicyTests.cs b/Darling/Darling.Tests/ServerConnectPolicyTests.cs index f116dd1ee6..a098548429 100644 --- a/Darling/Darling.Tests/ServerConnectPolicyTests.cs +++ b/Darling/Darling.Tests/ServerConnectPolicyTests.cs @@ -69,6 +69,8 @@ public void ADefinitionEdit_ResetsTheBackoff() var worker = (DarlingWorker)RuntimeHelpers.GetUninitializedObject(typeof(DarlingWorker)); typeof(DarlingWorker).GetField("_logger", BindingFlags.NonPublic | BindingFlags.Instance)! .SetValue(worker, NullLogger.Instance); + typeof(DarlingWorker).GetField("_collectorFaultStacks", BindingFlags.NonPublic | BindingFlags.Instance)! + .SetValue(worker, new CollectorFaultStackLog()); var list = Activator.CreateInstance(typeof(List<>).MakeGenericType(loopState))!; list.GetType().GetMethod("Add")!.Invoke(list, new[] { state }); @@ -125,6 +127,8 @@ public async Task DeadServers_DoNotHoldFleetPermits_WhileTheirConnectsAreInFligh var worker = (DarlingWorker)RuntimeHelpers.GetUninitializedObject(typeof(DarlingWorker)); typeof(DarlingWorker).GetField("_logger", BindingFlags.NonPublic | BindingFlags.Instance)! .SetValue(worker, NullLogger.Instance); + typeof(DarlingWorker).GetField("_collectorFaultStacks", BindingFlags.NonPublic | BindingFlags.Instance)! + .SetValue(worker, new CollectorFaultStackLog()); typeof(DarlingWorker).GetField("_connectProbeGate", BindingFlags.NonPublic | BindingFlags.Instance)! .SetValue(worker, probeGate); /* A dead server: the connect parks (SqlClient's 15 s) and never answers. */ @@ -249,6 +253,8 @@ private static DarlingWorker NewWorker(SemaphoreSlim probeGate, Func.Instance); + typeof(DarlingWorker).GetField("_collectorFaultStacks", BindingFlags.NonPublic | BindingFlags.Instance)! + .SetValue(worker, new CollectorFaultStackLog()); typeof(DarlingWorker).GetField("_connectProbeGate", BindingFlags.NonPublic | BindingFlags.Instance)! .SetValue(worker, probeGate); worker.ConnectOverride = connect; @@ -375,6 +381,8 @@ private static DarlingWorker NewWorker() var worker = (DarlingWorker)RuntimeHelpers.GetUninitializedObject(typeof(DarlingWorker)); typeof(DarlingWorker).GetField("_logger", BindingFlags.NonPublic | BindingFlags.Instance)! .SetValue(worker, NullLogger.Instance); + typeof(DarlingWorker).GetField("_collectorFaultStacks", BindingFlags.NonPublic | BindingFlags.Instance)! + .SetValue(worker, new CollectorFaultStackLog()); typeof(DarlingWorker).GetField("_connectProbeGate", BindingFlags.NonPublic | BindingFlags.Instance)! .SetValue(worker, new SemaphoreSlim(ServerConnectProbe.GateWidth, ServerConnectProbe.GateWidth)); return worker; @@ -464,8 +472,6 @@ public async Task ADefinitionEditDuringTheAttempt_IsConnectedAgainWithTheNewDefi var original = new MonitoredServer { Name = "s", Host = "old.invalid", StoredServerId = 7 }; var edited = new MonitoredServer { Name = "s", Host = "new.invalid", StoredServerId = 7 }; var state = ServerConnectBackoffTests.NewLoopState(LoopState, original); - /* Skip the self-alert pass: it needs the store, and this test is about the connect. */ - Set(state, "NextSelfAlertSweep", DateTime.UtcNow.AddSeconds(20)); var list = Activator.CreateInstance(typeof(List<>).MakeGenericType(LoopState))!; list.GetType().GetMethod("Add")!.Invoke(list, new[] { state }); diff --git a/Darling/Darling.Tests/ServerDatabasesEndpointLiveTests.cs b/Darling/Darling.Tests/ServerDatabasesEndpointLiveTests.cs new file mode 100644 index 0000000000..e8a5cb1871 --- /dev/null +++ b/Darling/Darling.Tests/ServerDatabasesEndpointLiveTests.cs @@ -0,0 +1,555 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Net; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.TestHost; +using Microsoft.Extensions.Logging; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5245: GET /api/server-databases through a real host over a live store. The list is the user databases the +/// store has collected for ONE server, from BOTH views the desktop picker reads (v_database_config and +/// v_database_size_stats), de-duplicated, with the system databases removed, and a name that is markup travels as +/// JSON data. Skipped without DARLING_TEST_PG, like its siblings. +/// +/* #1776: seeds and reads registry + database rows on the SHARED store, so it serializes with the other live classes + that do. */ +[Collection("live-postgres")] +public sealed class ServerDatabasesEndpointLiveTests +{ + private const int ServerId = 5_245_001; + private const string ServerName = "srv5245-endpoint-live"; + private const int OtherServerId = 5_245_002; + private const string OtherServerName = "srv5245-endpoint-other"; + + /// A name that is markup: it must come back as a string value, never be interpreted. + private const string MarkupName = ""; + + [Fact] + public async Task TheRoute_AnswersTheUserDatabasesOfOneServer_FromBothViews_WithSystemDatabasesRemoved_AndMarkupAsData() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live #5245 server-databases route test."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await CleanupAsync(connection, ct); + + var bodySucceeded = false; + try + { + await InsertServerAsync(connection, ServerId, ServerName, ct); + await InsertServerAsync(connection, OtherServerId, OtherServerName, ct); + var now = DateTime.UtcNow; + + /* v_database_config only, v_database_size_stats only, and both (AppB); two system databases. */ + await InsertAsync(connection, "database_config", "config_id", "capture_time", ServerId, ServerName, "AppA", now, ct); + await InsertAsync(connection, "database_config", "config_id", "capture_time", ServerId, ServerName, "AppB", now, ct); + await InsertAsync(connection, "database_config", "config_id", "capture_time", ServerId, ServerName, "master", now, ct); + await InsertAsync(connection, "database_size_stats", "collection_id", "collection_time", ServerId, ServerName, "AppB", now, ct); + await InsertAsync(connection, "database_size_stats", "collection_id", "collection_time", ServerId, ServerName, "AppC", now, ct); + await InsertAsync(connection, "database_size_stats", "collection_id", "collection_time", ServerId, ServerName, "tempdb", now, ct); + await InsertAsync(connection, "database_size_stats", "collection_id", "collection_time", ServerId, ServerName, MarkupName, now, ct); + /* Another server's database must not leak into this server's list. */ + await InsertAsync(connection, "database_config", "config_id", "capture_time", OtherServerId, OtherServerName, "OtherOnly", now, ct); + + await using var postgres = NpgsqlDataSource.Create(connectionString!); + var builder = WebApplication.CreateBuilder(); + builder.WebHost.UseTestServer(); + builder.Logging.ClearProviders(); + await using var app = builder.Build(); + DarlingServerDatabasesEndpoint.Map(app, postgres, app.Logger); + await app.StartAsync(ct); + using var client = app.GetTestClient(); + + using var response = await client.GetAsync("/api/server-databases?server=" + Uri.EscapeDataString(ServerName), ct); + var text = await response.Content.ReadAsStringAsync(ct); + Assert.Equal(HttpStatusCode.OK, response.StatusCode); + Assert.Equal("application/json", response.Content.Headers.ContentType?.MediaType); + + using var doc = JsonDocument.Parse(text); + Assert.Equal(ServerName, doc.RootElement.GetProperty("server").GetString()); + Assert.False(doc.RootElement.GetProperty("truncated").GetBoolean()); + var names = doc.RootElement.GetProperty("databases").EnumerateArray().Select(e => e.GetString()!).ToArray(); + + /* Both views, de-duplicated, system databases gone, the other server's database absent. The ORDER BY is + the database collation's, so the set is compared in ordinal order. */ + Assert.Equal( + new[] { "AppA", "AppB", "AppC", MarkupName }.OrderBy(n => n, StringComparer.Ordinal).ToArray(), + names.OrderBy(n => n, StringComparer.Ordinal).ToArray()); + + /* Markup is data: the angle brackets are escaped in the body, so the raw text holds no tag at all. */ + Assert.DoesNotContain(" + await CleanupAsync(cleanup, cleanupCt)); + } + } + + [Fact] + public async Task TheRoute_StopsAtItsCap_AndSaysTruncated_OnlyWhenOneMoreDatabaseExists() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live #5245 server-databases cap test."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await CleanupAsync(connection, ct); + + var bodySucceeded = false; + try + { + await InsertServerAsync(connection, ServerId, ServerName, ct); + var cap = DarlingServerDatabasesEndpoint.MaxDatabases; + Assert.Equal(5000, cap); + + /* Exactly cap databases: every name comes back and nothing is cut. */ + await InsertManyAsync(connection, ServerId, ServerName, 1, cap, ct); + + await using var postgres = NpgsqlDataSource.Create(connectionString!); + var builder = WebApplication.CreateBuilder(); + builder.WebHost.UseTestServer(); + builder.Logging.ClearProviders(); + await using var app = builder.Build(); + DarlingServerDatabasesEndpoint.Map(app, postgres, app.Logger); + await app.StartAsync(ct); + using var client = app.GetTestClient(); + var url = "/api/server-databases?server=" + Uri.EscapeDataString(ServerName); + + using (var exact = await client.GetAsync(url, ct)) + { + Assert.Equal(HttpStatusCode.OK, exact.StatusCode); + using var doc = JsonDocument.Parse(await exact.Content.ReadAsStringAsync(ct)); + Assert.Equal(cap, doc.RootElement.GetProperty("databases").GetArrayLength()); + Assert.False(doc.RootElement.GetProperty("truncated").GetBoolean()); + } + + /* One more (cap + 1): the cap's worth of names, in name order, and truncated. The extra name is the last in + name order, so it is the one left out. */ + await InsertManyAsync(connection, ServerId, ServerName, cap + 1, cap + 1, ct); + using (var cut = await client.GetAsync(url, ct)) + { + Assert.Equal(HttpStatusCode.OK, cut.StatusCode); + using var doc = JsonDocument.Parse(await cut.Content.ReadAsStringAsync(ct)); + var names = doc.RootElement.GetProperty("databases").EnumerateArray().Select(e => e.GetString()!).ToArray(); + Assert.Equal(cap, names.Length); + Assert.True(doc.RootElement.GetProperty("truncated").GetBoolean()); + Assert.DoesNotContain(CapName(cap + 1), names); + Assert.Contains(CapName(1), names); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, bodySucceeded, async (cleanup, cleanupCt) => + await CleanupAsync(cleanup, cleanupCt)); + } + } + + /// #5314: with more databases than the route's cap, a database past the cap is absent from the plain list + /// (truncated is true), and the same route with search finds it. Seeded at the real cap (5,001 numbered names + /// plus one name that sorts after all of them), so the whole path runs at its real size. + [Fact] + public async Task TheRoute_Search_FindsADatabasePastTheCap_ThePlainListLeavesOut() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live #5314 server-databases search test."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await CleanupAsync(connection, ct); + + var bodySucceeded = false; + try + { + await InsertServerAsync(connection, ServerId, ServerName, ct); + var cap = DarlingServerDatabasesEndpoint.MaxDatabases; + await InsertManyAsync(connection, ServerId, ServerName, 1, cap + 1, ct); + const string Past = "zz_past_the_cap_5314"; + await InsertAsync(connection, "database_config", "config_id", "capture_time", ServerId, ServerName, Past, DateTime.UtcNow, ct); + + await using var postgres = NpgsqlDataSource.Create(connectionString!); + var builder = WebApplication.CreateBuilder(); + builder.WebHost.UseTestServer(); + builder.Logging.ClearProviders(); + await using var app = builder.Build(); + DarlingServerDatabasesEndpoint.Map(app, postgres, app.Logger); + await app.StartAsync(ct); + using var client = app.GetTestClient(); + var url = "/api/server-databases?server=" + Uri.EscapeDataString(ServerName); + + var (plainNames, plainCut) = await GetAsync(client, url, ct); + Assert.Equal(cap, plainNames.Length); + Assert.True(plainCut); + Assert.DoesNotContain(Past, plainNames); + + // A blank search reads as no search: the same cut list. + var (blankNames, blankCut) = await GetAsync(client, url + "&search=%20%20", ct); + Assert.Equal(cap, blankNames.Length); + Assert.True(blankCut); + + // The search reaches past the cap, case-insensitively, and the answer is whole (nothing more to say). + var (found, foundCut) = await GetAsync(client, url + "&search=" + Uri.EscapeDataString("PAST_THE"), ct); + Assert.Equal(new[] { Past }, found); + Assert.False(foundCut); + + // A search that matches more than the cap is cut at the cap, with the flag. + var (many, manyCut) = await GetAsync(client, url + "&search=cap5245_", ct); + Assert.Equal(cap, many.Length); + Assert.True(manyCut); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, bodySucceeded, async (cleanup, cleanupCt) => + await CleanupAsync(cleanup, cleanupCt)); + } + } + + /// #5314: %, _ and the escape character in a search match only themselves. + [Fact] + public async Task TheRoute_Search_MatchesPercentUnderscoreAndBackslashOnlyAsThemselves() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live #5314 server-databases wildcard test."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await CleanupAsync(connection, ct); + + var bodySucceeded = false; + try + { + await InsertServerAsync(connection, ServerId, ServerName, ct); + var now = DateTime.UtcNow; + foreach (var name in new[] { "pct%name", "pctXname", "und_name", "undXname", "back\\slash", "backXslash", "Plain" }) + { + await InsertAsync(connection, "database_config", "config_id", "capture_time", ServerId, ServerName, name, now, ct); + } + + await using var postgres = NpgsqlDataSource.Create(connectionString!); + var builder = WebApplication.CreateBuilder(); + builder.WebHost.UseTestServer(); + builder.Logging.ClearProviders(); + await using var app = builder.Build(); + DarlingServerDatabasesEndpoint.Map(app, postgres, app.Logger); + await app.StartAsync(ct); + using var client = app.GetTestClient(); + var url = "/api/server-databases?server=" + Uri.EscapeDataString(ServerName) + "&search="; + + Assert.Equal(new[] { "pct%name" }, (await GetAsync(client, url + Uri.EscapeDataString("pct%"), ct)).Names); + Assert.Equal(new[] { "und_name" }, (await GetAsync(client, url + Uri.EscapeDataString("und_"), ct)).Names); + Assert.Equal(new[] { "back\\slash" }, (await GetAsync(client, url + Uri.EscapeDataString("back\\"), ct)).Names); + // A lone wildcard is a character to find, not "everything": no name holds a bare percent sign before "X". + Assert.Empty((await GetAsync(client, url + Uri.EscapeDataString("%X"), ct)).Names); + Assert.Equal(new[] { "und_name" }, (await GetAsync(client, url + Uri.EscapeDataString("UND_NAME"), ct)).Names); + // Without a search every name comes back. + Assert.Equal(7, (await GetAsync(client, "/api/server-databases?server=" + Uri.EscapeDataString(ServerName), ct)).Names.Length); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, bodySucceeded, async (cleanup, cleanupCt) => + await CleanupAsync(cleanup, cleanupCt)); + } + } + + /// #5314: the route's row counts and its truncated flag, asserted over a handful of rows through a small cap, + /// so they fail on their own and not only through the constant (an off-by-one in the extra-row probe, or a search that + /// limits after instead of before, changes a count here). + [Fact] + public async Task TheRoute_RowCountsAndTruncatedFlag_HoldAtASmallCap_WithAndWithoutSearch() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live #5314 server-databases small-cap test."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await CleanupAsync(connection, ct); + + var bodySucceeded = false; + try + { + await InsertServerAsync(connection, ServerId, ServerName, ct); + var now = DateTime.UtcNow; + // Five names: three "alpha" ones, then "beta1" and "beta2". + foreach (var name in new[] { "alpha1", "alpha2", "alpha3", "beta1", "beta2" }) + { + await InsertAsync(connection, "database_config", "config_id", "capture_time", ServerId, ServerName, name, now, ct); + } + + await using var postgres = NpgsqlDataSource.Create(connectionString!); + var builder = WebApplication.CreateBuilder(); + builder.WebHost.UseTestServer(); + builder.Logging.ClearProviders(); + await using var app = builder.Build(); + DarlingServerDatabasesEndpoint.Map(app, postgres, app.Logger, 3); + await app.StartAsync(ct); + using var client = app.GetTestClient(); + var url = "/api/server-databases?server=" + Uri.EscapeDataString(ServerName); + + // Five names, a cap of three: the first three by name, cut. + var plain = await GetAsync(client, url, ct); + Assert.Equal(new[] { "alpha1", "alpha2", "alpha3" }, plain.Names); + Assert.True(plain.Truncated); + + // The search narrows BEFORE the limit: both "beta" names come back (they are past the cap in the plain list), not cut. + var beta = await GetAsync(client, url + "&search=BETA", ct); + Assert.Equal(new[] { "beta1", "beta2" }, beta.Names); + Assert.False(beta.Truncated); + + // Exactly the cap's worth of matches is not cut; one more is. + var alpha = await GetAsync(client, url + "&search=alpha", ct); + Assert.Equal(new[] { "alpha1", "alpha2", "alpha3" }, alpha.Names); + Assert.False(alpha.Truncated); + var a = await GetAsync(client, url + "&search=a", ct); + Assert.Equal(new[] { "alpha1", "alpha2", "alpha3" }, a.Names); + Assert.True(a.Truncated); + + // No match is an empty list, not an error. + var none = await GetAsync(client, url + "&search=nothing-matches", ct); + Assert.Empty(none.Names); + Assert.False(none.Truncated); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, bodySucceeded, async (cleanup, cleanupCt) => + await CleanupAsync(cleanup, cleanupCt)); + } + } + + /// #5314: a NUL, a repeated search key and a value over 128 characters are refused with the route's + /// invalid envelope (a 400), before the store is read, and the body never holds the value sent. + [Fact] + public async Task TheRoute_Search_RefusesANul_ARepeatedKey_AndAnOverLongValue_WithoutEchoingThem() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live #5314 server-databases refusal test."); + + var ct = TestContext.Current.CancellationToken; + await using var postgres = NpgsqlDataSource.Create(connectionString!); + var builder = WebApplication.CreateBuilder(); + builder.WebHost.UseTestServer(); + builder.Logging.ClearProviders(); + await using var app = builder.Build(); + DarlingServerDatabasesEndpoint.Map(app, postgres, app.Logger); + await app.StartAsync(ct); + using var client = app.GetTestClient(); + var url = "/api/server-databases?server=" + Uri.EscapeDataString(ServerName) + "&search="; + + foreach (var (query, echoed) in new[] + { + ("ab%00cd", "cd"), + ("first&search=second", "second"), + (new string('x', DarlingServerDatabasesEndpoint.MaxSearchLength + 1), "xxxxxxxx"), + }) + { + using var response = await client.GetAsync(url + query, ct); + var text = await response.Content.ReadAsStringAsync(ct); + Assert.Equal(HttpStatusCode.BadRequest, response.StatusCode); + using var doc = JsonDocument.Parse(text); + Assert.Equal("invalid", doc.RootElement.GetProperty("status").GetString()); + Assert.Equal("search", doc.RootElement.GetProperty("hints").GetProperty("parameter").GetString()); + Assert.DoesNotContain(echoed, text, StringComparison.Ordinal); + } + } + + private static async Task<(string[] Names, bool Truncated)> GetAsync( + System.Net.Http.HttpClient client, string url, CancellationToken ct) + { + using var response = await client.GetAsync(url, ct); + Assert.Equal(HttpStatusCode.OK, response.StatusCode); + using var doc = JsonDocument.Parse(await response.Content.ReadAsStringAsync(ct)); + return ( + doc.RootElement.GetProperty("databases").EnumerateArray().Select(e => e.GetString()!).ToArray(), + doc.RootElement.GetProperty("truncated").GetBoolean()); + } + + /// The seeded name for number ; the zero padding keeps name order equal to number order + /// under any collation. + private static string CapName(int n) => "cap5245_" + n.ToString("D5", System.Globalization.CultureInfo.InvariantCulture); + + private static async Task InsertManyAsync( + NpgsqlConnection connection, int serverId, string serverName, int first, int last, CancellationToken ct) + { + using var command = new NpgsqlCommand( + "INSERT INTO database_config (config_id, capture_time, server_id, server_name, database_name) " + + "SELECT $1 + g, $2, $3, $4, 'cap5245_' || lpad(g::text, 5, '0') FROM generate_series($5::int, $6::int) AS g", + connection); + command.Parameters.AddWithValue(5_245_000_000_000L); + command.Parameters.AddWithValue(DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(serverId); + command.Parameters.AddWithValue(serverName); + command.Parameters.AddWithValue(first); + command.Parameters.AddWithValue(last); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertServerAsync(NpgsqlConnection connection, int serverId, string serverName, CancellationToken ct) + { + using var command = new NpgsqlCommand( + "INSERT INTO servers (server_id, server_name, display_name, is_enabled) VALUES ($1, $2, $3, TRUE)", connection); + command.Parameters.AddWithValue(serverId); + command.Parameters.AddWithValue(serverName); + command.Parameters.AddWithValue(serverName); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertAsync( + NpgsqlConnection connection, string table, string idColumn, string timeColumn, + int serverId, string serverName, string databaseName, DateTime whenUtc, CancellationToken ct) + { + using var command = new NpgsqlCommand( + $"INSERT INTO {table} ({idColumn}, {timeColumn}, server_id, server_name, database_name) VALUES ($1, $2, $3, $4, $5)", + connection); + command.Parameters.AddWithValue(CollectionIdGenerator.Next()); + command.Parameters.AddWithValue(DateTime.SpecifyKind(whenUtc, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(serverId); + command.Parameters.AddWithValue(serverName); + command.Parameters.AddWithValue(databaseName); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task CleanupAsync(NpgsqlConnection connection, CancellationToken ct) + { + foreach (var table in new[] { "database_config", "database_size_stats", "servers" }) + { + using var command = new NpgsqlCommand($"DELETE FROM {table} WHERE server_id = ANY($1)", connection); + command.Parameters.AddWithValue(new[] { ServerId, OtherServerId }); + await command.ExecuteNonQueryAsync(ct); + } + } +} + +/// +/// #5245 L7: the inventory SQL deduplicates inside EACH arm. Over 500 databases, hourly for 90 days (2.2 million +/// v_database_size_stats rows) the bare UNION shape sorted every row and took a 4.4 s median; hashing each +/// arm down first took 0.26 s with the same rows. A pin, because the shape is invisible in the result. +/// +public sealed class CollectedDatabasesSqlTests +{ + [Fact] + public void NamesSql_DeduplicatesInsideEachArm_SoTheUnionSortsNamesNotRows() + { + var sql = string.Join(" ", CollectedDatabases.NamesSql.Split(['\r', '\n', ' '], StringSplitOptions.RemoveEmptyEntries)); + + Assert.Contains("SELECT DISTINCT database_name FROM v_database_config WHERE server_id = $1", sql, StringComparison.Ordinal); + Assert.Contains("SELECT DISTINCT database_name FROM v_database_size_stats WHERE server_id = $1", sql, StringComparison.Ordinal); + } + + /// #5314: the search read is the shared list narrowed by one ILIKE against $3 with a backslash escape, ahead of + /// the ORDER BY and the LIMIT, so the limit counts matches; the pattern builder escapes the three characters that mean + /// something to LIKE. + [Fact] + public void NamesSearchLimitedSql_NarrowsBeforeTheLimit_WithOneEscapedPatternParameter() + { + var sql = CollectedDatabases.NamesSearchLimitedSql; + var like = sql.IndexOf("database_name ILIKE $3 ESCAPE '\\'", StringComparison.Ordinal); + var order = sql.IndexOf("ORDER BY database_name", StringComparison.Ordinal); + var limit = sql.IndexOf("LIMIT $2", StringComparison.Ordinal); + Assert.True(like > 0 && like < order && order < limit, sql); + Assert.DoesNotContain("$4", sql, StringComparison.Ordinal); + Assert.StartsWith(CollectedDatabases.NamesSql[..CollectedDatabases.NamesSql.IndexOf("\nORDER BY", StringComparison.Ordinal)], sql, StringComparison.Ordinal); + + Assert.Equal("%a\\%b\\_c\\\\d%", CollectedDatabases.SearchPattern("a%b_c\\d")); + Assert.Equal("%plain%", CollectedDatabases.SearchPattern("plain")); + } + + /// #5314: the search value's binding rules, pure. Blank is no search; a NUL, a second key and a value over 128 + /// characters are refused, and the refusal sentence never carries what was sent. + [Fact] + public void TryBindSearch_ReadsBlankAsNoSearch_AndRefusesNulRepeatsAndOverLongValues() + { + static (bool Ok, string? Search, string? Refusal) Bind(string query) + { + var ok = DarlingServerDatabasesEndpoint.TryBindSearch( + new Microsoft.AspNetCore.Http.QueryCollection(Microsoft.AspNetCore.WebUtilities.QueryHelpers.ParseQuery(query)), + out var search, out var refusal); + return (ok, search, refusal); + } + + Assert.Equal((true, null, null), Bind("")); + Assert.Equal((true, null, null), Bind("?search=")); + Assert.Equal((true, null, null), Bind("?search=%20%09")); + Assert.Equal((true, " ab ", null), Bind("?search=%20ab%20")); + Assert.Equal((true, new string('x', 128), null), Bind("?search=" + new string('x', 128))); + + foreach (var bad in new[] { "?search=a%00b", "?search=one&search=two", "?search=" + new string('y', 129) }) + { + var (ok, search, refusal) = Bind(bad); + Assert.False(ok); + Assert.Null(search); + Assert.Contains("\"status\":\"invalid\"", refusal, StringComparison.Ordinal); + Assert.DoesNotContain("two", refusal, StringComparison.Ordinal); + Assert.DoesNotContain("yyyy", refusal, StringComparison.Ordinal); + } + } + + /// The web route reads : the shared list plus one trailing LIMIT. + /// The desktop Excluded Databases picker keeps the unlimited , so its list + /// is every collected name as before. + [Fact] + public void NamesLimitedSql_IsTheSharedListPlusALimit_AndTheDesktopPickerStaysUnlimited() + { + Assert.Equal(CollectedDatabases.NamesSql + "\nLIMIT $2", CollectedDatabases.NamesLimitedSql); + Assert.DoesNotContain("LIMIT", CollectedDatabases.NamesSql, StringComparison.Ordinal); + + var desktop = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.ExcludedDatabases.cs"); + Assert.Contains("CollectedDatabases.NamesSql", desktop, StringComparison.Ordinal); + Assert.DoesNotContain("NamesLimitedSql", desktop, StringComparison.Ordinal); + + var route = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "DarlingServerDatabasesEndpoint.cs"); + Assert.Contains("CollectedDatabases.NamesLimitedSql", route, StringComparison.Ordinal); + Assert.Contains("CollectedDatabases.NamesSearchLimitedSql", route, StringComparison.Ordinal); + } +} \ No newline at end of file diff --git a/Darling/Darling.Tests/ServerEditCoreTests.cs b/Darling/Darling.Tests/ServerEditCoreTests.cs new file mode 100644 index 0000000000..3ed7bcac89 --- /dev/null +++ b/Darling/Darling.Tests/ServerEditCoreTests.cs @@ -0,0 +1,743 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.Json.Nodes; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using NpgsqlTypes; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Service.Mcp; +using Xunit; +using Edit = PerformanceMonitor.Darling.Service.Mcp.DarlingMcpServerAdminTools; + +namespace Darling.Tests; + +/// +/// The edit core (#5240) over a stand-in store and probe: field validation, the refused keys, the optimistic token, +/// the credential rules, the audit line and the rule that no secret value reaches an answer. The statements the core +/// really runs are proved against a store in . +/// +[Collection("darling-owned-secrets")] +public sealed class ServerEditCoreTests : IDisposable +{ + private const string Secret = "Synth-Edit-Pw-1"; + private const string LiteralSecret = "Lit3ral-Sw0rd-Value"; + + private readonly DarlingOwnedSet _ownedBefore = DarlingOwnedSecrets.Current; + + public ServerEditCoreTests() => DarlingOwnedSecrets.Set(DarlingOwnedSet.Empty); + + public void Dispose() => DarlingOwnedSecrets.Set(_ownedBefore); + + private static readonly DateTime Stamp = new(2026, 10, 5, 12, 0, 0, 123, DateTimeKind.Unspecified); + + private static Edit.ServerEditRow SqlRow(string host = "alpha-01.example.test", string auth = "sql", string engine = "sqlserver") => new( + ServerId: 41, Name: "alpha-01", Host: host, Port: 0, Database: null, ReadOnlyIntent: false, Engine: engine, + Auth: auth, Username: auth == "integrated" ? null : "monitor", EncryptMode: "Mandatory", TrustServerCertificate: false, + MultiSubnetFailover: false, MonthlyCostUsd: 10m, ModifiedAt: Stamp.AddTicks(7)); + + private static readonly Edit.ServerProbe Reachable = (_, _) => Task.FromResult( + new ConnectionProbeResult(true, 15, 3, "Enterprise", false, false, false, true, null)); + + private static Edit.ServerProbe Unreachable(string error) => (_, _) => Task.FromResult( + new ConnectionProbeResult(false, 0, 0, null, false, false, false, false, error)); + + private static readonly Edit.ServerProbe ThrowingProbe = + (_, _) => throw new InvalidOperationException("the probe must not run for this request"); + + private sealed class FakeStore : Edit.IServerEditStore + { + public Edit.ServerEditRow? Row { get; set; } + public List OtherKeys { get; } = []; + public int Reads { get; private set; } + public int Writes { get; private set; } + public List LastSets { get; private set; } = []; + public string? LastNewKey { get; private set; } + public string? LastActualKey { get; private set; } + public Edit.ServerEditWriteKind WriteResult { get; set; } = Edit.ServerEditWriteKind.Written; + + public Task ReadRowAsync(int serverId, CancellationToken cancellationToken) + { + Reads++; + return Task.FromResult(Row); + } + + public Task> LoadOtherStorageKeysAsync(int serverId, CancellationToken cancellationToken) => + Task.FromResult(OtherKeys.ToList()); + + public Task WriteAsync( + int serverId, DateTime expectedModifiedAt, IReadOnlyList sets, string? newStorageKey, string? actualStorageKey, CancellationToken cancellationToken) + { + Writes++; + LastSets = sets.ToList(); + LastNewKey = newStorageKey; + LastActualKey = actualStorageKey; + return Task.FromResult(new Edit.ServerEditWrite(WriteResult, Stamp.AddSeconds(1))); + } + } + + private sealed class CapturingLogger : ILogger + { + public List Lines { get; } = []; + + public IDisposable? BeginScope(TState state) where TState : notnull => null; + + public bool IsEnabled(LogLevel logLevel) => true; + + public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) => + Lines.Add(formatter(state, exception)); + } + + private static Task Run(FakeStore store, string changes, Edit.ServerProbe? probe = null, IPasswordKeyRing? ring = null, ILogger? logger = null) => + Edit.EditServerCoreAsync(store, 41, changes, probe ?? ThrowingProbe, ring ?? TestKeyRings.Healthy, logger, CancellationToken.None); + + private static string Status(string answer) => JsonNode.Parse(answer)!["status"]!.GetValue(); + + /* ---------------- parse ---------------- */ + + [Theory] + [InlineData("engine")] + [InlineData("is_enabled")] + [InlineData("excluded_databases")] + [InlineData("server_id")] + [InlineData("capture_plans")] + [InlineData("alert_delivery_mode_override")] + [InlineData("plan_force_bot_enabled")] + [InlineData("remediation_username")] + [InlineData("remediation_encrypted_password")] + public async Task ARefusedKey_IsInvalid_AndNothingIsReadOrWritten(string key) + { + var store = new FakeStore { Row = SqlRow() }; + var answer = await Run(store, $"{{\"display_name\":\"x\",\"{key}\":1}}"); + + Assert.Equal("invalid", Status(answer)); + Assert.Contains(key.StartsWith("remediation_", StringComparison.Ordinal) ? "remediation_*" : key, answer, StringComparison.Ordinal); + Assert.Contains("cannot be changed here", answer, StringComparison.Ordinal); + Assert.Equal(0, store.Reads); + Assert.Equal(0, store.Writes); + } + + [Fact] + public void AnUnknownKey_IsRefused_NamingTheEditableKeys_AndNeverEchoingAValue() + { + var (changes, error) = Edit.ParseEditChanges("{\"passwrd\":\"" + LiteralSecret + "\"}"); + + Assert.Null(changes); + Assert.Contains("not an editable field", error, StringComparison.Ordinal); + Assert.Contains("display_name", error, StringComparison.Ordinal); + Assert.DoesNotContain(LiteralSecret, error, StringComparison.Ordinal); + } + + [Theory] + [InlineData("")] + [InlineData(" ")] + [InlineData("not json")] + [InlineData("[]")] + [InlineData("{}")] + [InlineData("{\"display_name\":\"a\",\"display_name\":\"b\"}")] + [InlineData("{\"port\":70000}")] + [InlineData("{\"port\":\"abc\"}")] + [InlineData("{\"host\":\" \"}")] + [InlineData("{\"host\":5}")] + [InlineData("{\"read_only_intent\":\"yes\"}")] + [InlineData("{\"trust_server_certificate\":1}")] + [InlineData("{\"multi_subnet_failover\":null}")] + [InlineData("{\"auth\":\"EntraMFA\"}")] + [InlineData("{\"auth\":\"\"}")] + [InlineData("{\"encrypt_mode\":\"Sometimes\"}")] + [InlineData("{\"monthly_cost_usd\":-1}")] + [InlineData("{\"monthly_cost_usd\":\"lots\"}")] + [InlineData("{\"expected_modified_at\":\"\"}")] + public void ABadBody_IsInvalid_BeforeAnyStoreAccess(string json) + { + var (changes, error) = Edit.ParseEditChanges(json); + + Assert.Null(changes); + Assert.False(string.IsNullOrWhiteSpace(error)); + } + + [Fact] + public void AGoodBody_ParsesEveryField_AndLeavesTheUnnamedOnesUntouched() + { + var (c, error) = Edit.ParseEditChanges( + "{\"display_name\":\" Orders \",\"host\":\"b.example.test\",\"port\":\"5433\",\"database\":null,\"read_only_intent\":true," + + "\"auth\":\"sql\",\"username\":\"u\",\"password\":\"" + Secret + "\",\"encrypt_mode\":\"strict\"," + + "\"trust_server_certificate\":true,\"multi_subnet_failover\":true,\"monthly_cost_usd\":12.5,\"expected_modified_at\":\"t\"}"); + + Assert.Null(error); + Assert.Equal("Orders", c!.DisplayName); + Assert.Equal("b.example.test", c.Host); + Assert.Equal(5433, c.Port); + Assert.True(c.HasDatabase); + Assert.Null(c.Database); + Assert.Equal("sql", c.StoreAuth); + Assert.Equal("Strict", c.EncryptMode); + Assert.Equal(12.5m, c.MonthlyCostUsd); + + var (only, _) = Edit.ParseEditChanges("{\"monthly_cost_usd\":1}"); + Assert.False(only!.HasDatabase); + Assert.False(only.HasUsername); + Assert.Null(only.Host); + Assert.Null(only.Port); + } + + /* ---------------- the SET list ---------------- */ + + [Fact] + public void TheWritableColumns_NeverIncludeAnIdentityOrAnOwnerOnlyColumn() + { + var forbidden = new[] + { + "server_id", "engine", "is_enabled", "excluded_databases", "capture_plans", "alert_delivery_mode_override", + "plan_force_bot_enabled", "remediation_username", "remediation_encrypted_password", "created_at", + }; + + Assert.Empty(Edit.EditColumnOfField.Values.Intersect(forbidden)); + Assert.Equal(14, Edit.EditColumnOfField.Count); + } + + [Fact] + public void TheFunctionCall_NamesOnlyTheColumnsGiven_PassesTheTokenUnrounded_AndLeavesTheRestNull() + { + var token = Stamp.AddTicks(7); + var parameters = Edit.BuildEditFunctionParameters( + 41, token, + [ + new Edit.EditColumnValue("display_name", "name", NpgsqlDbType.Text, "x"), + new Edit.EditColumnValue("password", "encrypted_password", NpgsqlDbType.Text, "blob"), + ]); + + Assert.Equal(17, parameters.Count); + Assert.Equal(["name", "encrypted_password"], (string[])parameters[2].Value!); + Assert.Equal(token, (DateTime)parameters[1].Value!); + Assert.Equal("x", parameters[3].Value); + Assert.Equal("blob", parameters[10].Value); + Assert.Equal(DBNull.Value, parameters[4].Value); + Assert.DoesNotContain("encrypted_password", Edit.EditFunctionSql, StringComparison.Ordinal); + Assert.Equal( + Edit.EditColumnOfField.Values.Order(StringComparer.Ordinal).ToArray(), + Edit.EditFunctionColumns.Select(c => c.Column).Order(StringComparer.Ordinal).ToArray()); + } + + [Fact] + public void ASetOnAnUnlistedColumn_ThrowsBeforeAnySql() + { + var ex = Assert.Throws(() => + Edit.BuildEditFunctionParameters(41, Stamp, [new Edit.EditColumnValue("x", "is_enabled", NpgsqlDbType.Boolean, false)])); + Assert.Contains("is_enabled", ex.Message, StringComparison.Ordinal); + } + + /* ---------------- plan: the credential rules ---------------- */ + + private static (Edit.ServerEditPlan? Plan, string? Error) Plan(Edit.ServerEditRow row, string json, PasswordKeyStatus? key = null) + { + var (changes, error) = Edit.ParseEditChanges(json); + Assert.Null(error); + return Edit.PlanEdit(row, changes!, key ?? TestKeyRings.Healthy.Status); + } + + [Fact] + public void ANonConnectionEdit_OnASqlRow_NeedsNoSecret_NoProbe_AndNoReconnectForCost() + { + var (cost, error) = Plan(SqlRow(), "{\"monthly_cost_usd\":99}"); + Assert.Null(error); + Assert.Equal(["monthly_cost_usd"], cost!.ChangedFields); + Assert.False(cost.NeedsProbe); + Assert.False(cost.Reconnects); + Assert.DoesNotContain(cost.Sets, s => s.Column == "encrypted_password"); + + var (rename, _) = Plan(SqlRow(), "{\"display_name\":\"renamed\"}"); + Assert.False(rename!.NeedsProbe); + Assert.True(rename.Reconnects, "the worker compares the display name, so a rename reconnects"); + } + + [Theory] + [InlineData("{\"host\":\"beta-01.example.test\"}")] + [InlineData("{\"database\":\"Orders\"}")] + [InlineData("{\"read_only_intent\":true}")] + [InlineData("{\"username\":\"someone-else\"}")] + [InlineData("{\"encrypt_mode\":\"Strict\"}")] + [InlineData("{\"trust_server_certificate\":true}")] + [InlineData("{\"multi_subnet_failover\":true}")] + public void AConnectionEdit_OnASqlRow_WithoutTheSecret_IsRefused_WithTheDocumentedSentence(string json) + { + var (plan, error) = Plan(SqlRow(), json); + + Assert.Null(plan); + Assert.Equal(Edit.EditPasswordNeededText, error); + } + + [Fact] + public void AConnectionEdit_WithTheSecret_IsProbed_AndWritesTheSecretColumn() + { + var (plan, error) = Plan(SqlRow(), "{\"host\":\"beta-01.example.test\",\"password\":\"" + Secret + "\"}"); + + Assert.Null(error); + Assert.True(plan!.NeedsProbe); + Assert.True(plan.Reconnects); + Assert.True(plan.AddressChanged); + Assert.Equal(["host", "password"], plan.ChangedFields); + Assert.Equal(Secret, plan.PlaintextSecret); + } + + [Fact] + public void APortOnASqlServerRow_IsRefused_ThePortGoesInTheHost() + { + var (plan, error) = Plan(SqlRow(), "{\"port\":1444}"); + + Assert.Null(plan); + Assert.Equal("A SQL Server port goes in host, as host,port.", error); + } + + [Fact] + public void APortOnAPostgresRow_IsAccepted_AndNeedsTheSecret() + { + var (needs, needsError) = Plan(SqlRow(engine: "postgres"), "{\"port\":5433}"); + Assert.Null(needs); + Assert.Equal(Edit.EditPasswordNeededText, needsError); + + var (plan, error) = Plan(SqlRow(engine: "postgres"), "{\"port\":5433,\"password\":\"" + Secret + "\"}"); + Assert.Null(error); + Assert.Contains("port", plan!.ChangedFields); + } + + [Fact] + public void ACaseOnlyEncryptModeChange_IsUnchanged_NotAWriteOrAReconnect() + { + var (plan, error) = Plan(SqlRow(), "{\"encrypt_mode\":\"mandatory\"}"); + + Assert.Null(error); + Assert.Empty(plan!.Sets); + Assert.False(plan.NeedsProbe); + } + + [Theory] + [InlineData("integrated")] + [InlineData("managedidentity")] + public void ACostOnlyEdit_OnANonSecretAuthRow_DoesNotReconnect(string auth) + { + var (plan, error) = Plan(SqlRow(auth: auth), "{\"monthly_cost_usd\":99}"); + + Assert.Null(error); + Assert.Equal(["monthly_cost_usd"], plan!.ChangedFields); + Assert.False(plan.Reconnects); + } + + [Fact] + public void ASwitchFromASecretModeToANonSecretMode_StillReconnects() + { + var (plan, error) = Plan(SqlRow(), "{\"auth\":\"Windows\"}"); + + Assert.Null(error); + Assert.True(plan!.Reconnects); + } + + [Fact] + public async Task ARefusedKey_IsTruncated_AndARemediationKeyNamesOnlyThePrefix() + { + var longKey = new string('k', 60); + var answer = await Run(new FakeStore { Row = SqlRow() }, "{\"remediation_password_hint_" + longKey + "\":1}"); + Assert.Equal("invalid", Status(answer)); + Assert.Contains("remediation_*", answer, StringComparison.Ordinal); + Assert.DoesNotContain(longKey, answer, StringComparison.Ordinal); + } + + [Fact] + public async Task AConflictInsideTheWrite_CarriesTheCurrentValues_LikeThePreProbeConflict() + { + var store = new FakeStore { Row = SqlRow(), WriteResult = Edit.ServerEditWriteKind.Conflict }; + + var answer = await Run(store, "{\"monthly_cost_usd\":5}"); + + Assert.Equal("conflict", Status(answer)); + var current = JsonNode.Parse(answer)!["current"]!; + Assert.Equal("alpha-01", current["display_name"]!.GetValue()); + Assert.Equal(Edit.ModifiedAtToken(store.Row!.ModifiedAt), current["modified_at"]!.GetValue()); + } + + [Fact] + public void AWindowsRow_NeedsNoSecret_ToChangeItsHost_ButIsProbed() + { + var (plan, error) = Plan(SqlRow(auth: "integrated"), "{\"host\":\"beta-01.example.test\"}"); + + Assert.Null(error); + Assert.True(plan!.NeedsProbe); + Assert.DoesNotContain(plan.Sets, s => s.Column == "encrypted_password"); + } + + [Fact] + public void ASwitchIntoSql_RequiresThePassword_AndASwitchOutClearsTheStoredSecret() + { + var (refused, error) = Plan(SqlRow(auth: "integrated"), "{\"auth\":\"SQL\",\"username\":\"u\"}"); + Assert.Null(refused); + Assert.Contains("needs the password", error, StringComparison.Ordinal); + + var (into, _) = Plan(SqlRow(auth: "integrated"), "{\"auth\":\"SQL\",\"username\":\"u\",\"password\":\"" + Secret + "\"}"); + Assert.Equal(["auth", "username", "password"], into!.ChangedFields); + + var (spRefused, spError) = Plan(SqlRow(), "{\"auth\":\"ServicePrincipal\",\"username\":\"app-id\"}"); + Assert.Null(spRefused); + Assert.Contains("client secret", spError, StringComparison.Ordinal); + + var (outOf, outError) = Plan(SqlRow(), "{\"auth\":\"Windows\"}"); + Assert.Null(outError); + var cleared = Assert.Single(outOf!.Sets, s => s.Column == "encrypted_password"); + Assert.Null(cleared.Value); + Assert.Contains(outOf.Sets, s => s.Column == "username" && s.Value is null); + Assert.Null(outOf.PlaintextSecret); + } + + [Fact] + public void APasswordSentToAWindowsRow_IsIgnored_AsAddIgnoresIt() + { + var (plan, error) = Plan(SqlRow(auth: "integrated"), "{\"monthly_cost_usd\":1,\"password\":\"" + Secret + "\"}"); + + Assert.Null(error); + Assert.Null(plan!.PlaintextSecret); + Assert.DoesNotContain(plan.Sets, s => s.Column == "encrypted_password"); + } + + [Fact] + public void ALiteralSecret_WhileTheKeyIsNotReady_IsRefused_AsAddRefusesIt_AndTheTextNeverHoldsIt() + { + var (plan, error) = Plan(SqlRow(), "{\"host\":\"b.example.test\",\"password\":\"" + LiteralSecret + "\"}", TestKeyRings.NotReady.Status); + + Assert.Null(plan); + Assert.Equal(Edit.LiteralSecretRefusal(LiteralSecret, TestKeyRings.NotReady.Status, isServicePrincipal: false), error); + Assert.DoesNotContain(LiteralSecret, error, StringComparison.Ordinal); + } + + [Fact] + public void AReferenceToDarlingsOwnSecrets_IsRefused_WithTheSentenceEveryReferenceGets() + { + DarlingOwnedSecrets.Set(new DarlingOwnedSet(Array.Empty(), new[] { "DARLING_CONFIG" })); + + var (plan, error) = Plan(SqlRow(), "{\"host\":\"b.example.test\",\"password\":\"env:DARLING_CONFIG\"}"); + + Assert.Null(plan); + Assert.Equal(DarlingSecretSource.RequestReferenceRefusalText, error); + } + + [Fact] + public void APostgresRow_CannotLeaveSqlAuth() + { + var row = SqlRow() with { Engine = "postgres", Port = 5432 }; + + var (plan, error) = Plan(row, "{\"auth\":\"Windows\"}"); + + Assert.Null(plan); + Assert.Contains("PostgreSQL target requires", error, StringComparison.Ordinal); + } + + [Fact] + public void ABlankDisplayName_FallsBackToTheHost_AsAddDoes() + { + var (plan, _) = Plan(SqlRow(), "{\"display_name\":\" \"}"); + + Assert.Equal("alpha-01.example.test", plan!.NewName); + } + + /* ---------------- the core ---------------- */ + + [Fact] + public async Task AStaleToken_IsAConflict_WithTheCurrentValues_AndNothingIsWritten() + { + var store = new FakeStore { Row = SqlRow() }; + + var answer = await Run(store, "{\"monthly_cost_usd\":5,\"expected_modified_at\":\"2026-10-05T11:00:00.0000000\"}"); + + Assert.Equal("conflict", Status(answer)); + Assert.Equal("alpha-01", JsonNode.Parse(answer)!["current"]!["display_name"]!.GetValue()); + Assert.Equal(Edit.ModifiedAtToken(store.Row!.ModifiedAt), JsonNode.Parse(answer)!["current"]!["modified_at"]!.GetValue()); + Assert.Equal(0, store.Writes); + } + + [Fact] + public async Task TheCurrentToken_KeepsMicroseconds_AndIsAccepted() + { + var store = new FakeStore { Row = SqlRow() }; + var token = Edit.ModifiedAtToken(store.Row!.ModifiedAt); + Assert.EndsWith("1230007", token, StringComparison.Ordinal); + + var answer = await Run(store, $"{{\"monthly_cost_usd\":5,\"expected_modified_at\":\"{token}\"}}"); + + Assert.Equal("updated", Status(answer)); + Assert.Equal(1, store.Writes); + } + + [Fact] + public async Task ARequestEqualToTheStoredValues_IsUnchanged_WithZeroWrites_AndNoProbe() + { + var store = new FakeStore { Row = SqlRow() }; + + var answer = await Run(store, "{\"monthly_cost_usd\":10,\"host\":\"alpha-01.example.test\"}"); + + Assert.Equal("unchanged", Status(answer)); + Assert.Equal(0, store.Writes); + } + + [Fact] + public async Task AMissingRow_IsNotFound() + { + var answer = await Run(new FakeStore { Row = null }, "{\"monthly_cost_usd\":5}"); + + Assert.Equal("not_found", Status(answer)); + } + + [Fact] + public async Task AnAddressAnotherServerHolds_Collides_AsOccupied_BeforeTheProbe() + { + var store = new FakeStore { Row = SqlRow() }; + store.OtherKeys.Add("BETA-01.example.test"); + + var answer = await Run(store, "{\"host\":\"beta-01.example.test\",\"password\":\"" + Secret + "\"}"); + + Assert.Equal("collides", Status(answer)); + Assert.Equal("occupied", JsonNode.Parse(answer)!["reason"]!.GetValue()); + Assert.Equal(0, store.Writes); + } + + [Fact] + public async Task AProbeThatLandsInADatabaseAnotherServerCovers_Collides_AndNothingIsWritten() + { + var store = new FakeStore { Row = SqlRow() }; + store.OtherKeys.Add("alpha-01.example.test:Covered"); + Edit.ServerProbe lands = (_, _) => Task.FromResult(new ConnectionProbeResult( + true, 15, 3, "Enterprise", false, false, false, true, null, ConnectedDatabase: "Covered")); + + var answer = await Run(store, "{\"read_only_intent\":false,\"database\":\"Wrong\",\"password\":\"" + Secret + "\"}", lands); + + Assert.Equal("collides", Status(answer)); + Assert.Equal(0, store.Writes); + } + + [Fact] + public async Task AFailedProbe_IsConnectionFailed_NothingIsWritten_AndTheSecretIsRedactedFromTheDetail() + { + var store = new FakeStore { Row = SqlRow() }; + + var answer = await Run(store, "{\"host\":\"b.example.test\",\"password\":\"" + Secret + "\"}", Unreachable("login failed for " + Secret)); + + Assert.Equal("connection_failed", Status(answer)); + Assert.DoesNotContain(Secret, answer, StringComparison.Ordinal); + Assert.Equal(0, store.Writes); + } + + [Theory] + [InlineData(0, "updated")] + [InlineData(1, "not_found")] + [InlineData(2, "conflict")] + [InlineData(3, "collides")] + public async Task TheStoresWriteVerdict_IsTheAnswer(int verdict, string expected) + { + var store = new FakeStore { Row = SqlRow(), WriteResult = (Edit.ServerEditWriteKind)verdict }; + + var answer = await Run(store, "{\"monthly_cost_usd\":5}"); + + Assert.Equal(expected, Status(answer)); + } + + [Fact] + public async Task AnUpdated_NamesWhatChanged_TheToken_AndTheOneTimeAddressSentence() + { + var store = new FakeStore { Row = SqlRow() }; + + var answer = JsonNode.Parse(await Run(store, "{\"host\":\"b.example.test\",\"display_name\":\"Beta\",\"password\":\"" + Secret + "\"}", Reachable))!; + + Assert.Equal("updated", answer["status"]!.GetValue()); + Assert.Equal(41, answer["server_id"]!.GetValue()); + Assert.Equal(["display_name", "host", "password"], answer["changed"]!.AsArray().Select(n => n!.GetValue()).ToArray()); + Assert.True(answer["reconnects"]!.GetValue()); + Assert.True(answer["tested"]!.GetValue()); + Assert.Equal(Edit.ModifiedAtToken(Stamp.AddSeconds(1)), answer["modified_at"]!.GetValue()); + Assert.Contains("keeps its id and history", answer["note"]!.GetValue(), StringComparison.Ordinal); + Assert.Equal("b.example.test", store.LastNewKey); + + var costOnly = JsonNode.Parse(await Run(new FakeStore { Row = SqlRow() }, "{\"monthly_cost_usd\":5}"))!; + Assert.DoesNotContain("keeps its id", costOnly["note"]!.GetValue(), StringComparison.Ordinal); + Assert.False(costOnly["tested"]!.GetValue()); + Assert.False(costOnly["reconnects"]!.GetValue()); + } + + [Fact] + public async Task TheColumnsWritten_AreExactlyTheChangedOnes_NeverAnIdentityOrOwnerColumn() + { + var store = new FakeStore { Row = SqlRow(engine: "postgres") }; + + await Run(store, "{\"host\":\"b.example.test\",\"port\":1444,\"database\":\"D\",\"read_only_intent\":true,\"monthly_cost_usd\":3," + + "\"encrypt_mode\":\"Strict\",\"trust_server_certificate\":true,\"multi_subnet_failover\":true,\"username\":\"u2\"," + + "\"display_name\":\"n\",\"password\":\"" + Secret + "\"}", Reachable); + + var columns = store.LastSets.Select(s => s.Column).OrderBy(c => c, StringComparer.Ordinal).ToArray(); + Assert.Equal( + ["database", "encrypt_mode", "encrypted_password", "host", "monthly_cost_usd", "multi_subnet_failover", "name", "port", + "read_only_intent", "trust_server_certificate", "username"], + columns); + var listed = (string[])Edit.BuildEditFunctionParameters(41, Stamp, store.LastSets)[2].Value!; + foreach (var banned in new[] { "server_id", "engine", "is_enabled", "excluded_databases", "remediation_username", "remediation_encrypted_password", "plan_force_bot_enabled", "capture_plans" }) + { + Assert.DoesNotContain(banned, listed); + } + } + + [Fact] + public async Task TheStoredSecret_IsTheProtectedForm_NotThePlaintextForm() + { + var store = new FakeStore { Row = SqlRow() }; + + await Run(store, "{\"host\":\"b.example.test\",\"password\":\"" + Secret + "\"}", Reachable); + + var stored = Assert.Single(store.LastSets, s => s.Column == "encrypted_password"); + /* Sealed for the connection the row has after the edit: the new host, everything else as the row held it. */ + var row = SqlRow(); + var binding = PasswordBinding.ForServer(ServerConnectionIdentity.FromStoredColumns( + "b.example.test", row.Port, row.Engine, row.Database, row.ReadOnlyIntent, row.Auth, row.Username, row.EncryptMode, + row.TrustServerCertificate, row.MultiSubnetFailover)); + Assert.Equal(Secret, TestKeyRings.Healthy.Open((string)stored.Value!, binding)); + Assert.NotEqual(Secret, stored.Value); + var movedAgain = PasswordBinding.ForServer(ServerConnectionIdentity.FromStoredColumns( + row.Host, row.Port, row.Engine, row.Database, row.ReadOnlyIntent, row.Auth, row.Username, row.EncryptMode, + row.TrustServerCertificate, row.MultiSubnetFailover)); + Assert.Throws(() => { _ = TestKeyRings.Healthy.Open((string)stored.Value!, movedAgain); }); + } + + [Fact] + public async Task NoAnswerAndNoLogLine_ForAnyStatus_HoldsTheSubmittedSecret() + { + var logger = new CapturingLogger(); + var answers = new List(); + var secretBody = "\"password\":\"" + Secret + "\""; + + answers.Add(await Run(new FakeStore { Row = SqlRow() }, "{\"host\":\"b.example.test\"," + secretBody + "}", Reachable, logger: logger)); + answers.Add(await Run(new FakeStore { Row = SqlRow() }, "{\"host\":\"b.example.test\"," + secretBody + ",\"expected_modified_at\":\"stale\"}", Reachable, logger: logger)); + answers.Add(await Run(new FakeStore { Row = SqlRow() }, "{\"host\":\"b.example.test\"," + secretBody + "}", Unreachable("bad " + Secret), logger: logger)); + answers.Add(await Run(new FakeStore { Row = null }, "{\"host\":\"b.example.test\"," + secretBody + "}", Reachable, logger: logger)); + answers.Add(await Run(new FakeStore { Row = SqlRow(), OtherKeys = { "b.example.test" } }, "{\"host\":\"b.example.test\"," + secretBody + "}", Reachable, logger: logger)); + answers.Add(await Run(new FakeStore { Row = SqlRow() }, "{\"engine\":\"postgres\"," + secretBody + "}", Reachable, logger: logger)); + answers.Add(await Run(new FakeStore { Row = SqlRow() }, "{\"passwrd\":\"" + Secret + "\"}", Reachable, logger: logger)); + answers.Add(await Run(new FakeStore { Row = SqlRow(), WriteResult = Edit.ServerEditWriteKind.Conflict }, "{\"host\":\"b.example.test\"," + secretBody + "}", Reachable, logger: logger)); + answers.Add(await Run(new FakeStore { Row = SqlRow() }, "{\"host\":\"b.example.test\",\"password\":\"" + LiteralSecret + "\"}", Reachable, ring: TestKeyRings.NotReady, logger: logger)); + + Assert.Equal(9, answers.Count); + Assert.All(answers, a => + { + Assert.DoesNotContain(Secret, a, StringComparison.Ordinal); + Assert.DoesNotContain(LiteralSecret, a, StringComparison.Ordinal); + }); + Assert.All(logger.Lines, l => + { + Assert.DoesNotContain(Secret, l, StringComparison.Ordinal); + Assert.DoesNotContain("SYNTH_EDIT_SECRET_REF", l, StringComparison.Ordinal); + }); + } + + [Fact] + public async Task TheAuditLine_NamesTheIdAndTheFieldNames_NeverAValue() + { + var logger = new CapturingLogger(); + var store = new FakeStore { Row = SqlRow() }; + + await Run(store, "{\"display_name\":\"Distinct-New-Name\",\"host\":\"distinct-new-host.example.test\",\"password\":\"" + Secret + "\"}", Reachable, logger: logger); + + var line = Assert.Single(logger.Lines); + Assert.Contains("41", line, StringComparison.Ordinal); + Assert.Contains("display_name,host,password", line, StringComparison.Ordinal); + Assert.DoesNotContain("Distinct-New-Name", line, StringComparison.Ordinal); + Assert.DoesNotContain("distinct-new-host", line, StringComparison.Ordinal); + Assert.DoesNotContain("SYNTH_EDIT_SECRET_REF", line, StringComparison.Ordinal); + + var quiet = new CapturingLogger(); + await Run(new FakeStore { Row = SqlRow() }, "{\"monthly_cost_usd\":10}", logger: quiet); + await Run(new FakeStore { Row = SqlRow(), WriteResult = Edit.ServerEditWriteKind.Conflict }, "{\"monthly_cost_usd\":11}", logger: quiet); + Assert.Empty(quiet.Lines); + } + + /* ---------------- a move of host or port asks for the password BEFORE any connection test (#5240) ---------------- */ + + [Theory] + [InlineData("sql", "sqlserver", "{\"host\":\"beta-01.example.test\"}")] + [InlineData("serviceprincipal", "sqlserver", "{\"host\":\"beta-01.example.test\"}")] + [InlineData("sql", "postgres", "{\"port\":5433}")] + [InlineData("sql", "postgres", "{\"host\":\"beta-01.example.test\"}")] + public async Task AMoveOfHostOrPort_OnASecretRow_AsksForThePasswordBeforeTheTest_WithNoProbeAndNoWrite(string auth, string engine, string changes) + { + var probed = 0; + Edit.ServerProbe counting = (config, ct) => + { + probed++; + return Reachable(config, ct); + }; + var store = new FakeStore { Row = SqlRow(auth: auth, engine: engine) }; + + var answer = JsonNode.Parse(await Run(store, changes, counting))!; + + Assert.Equal("invalid", answer["status"]!.GetValue()); + Assert.Equal(Edit.EditPasswordNeededText, answer["message"]!.GetValue()); + Assert.Equal(0, probed); + Assert.Equal(0, store.Writes); + } + + [Fact] + public async Task AMoveOfHost_WithThePassword_TestsWithTheNewOne_AndSavesIt() + { + string? testedWith = null; + Edit.ServerProbe capturing = (config, ct) => + { + testedWith = config.Password; + return Reachable(config, ct); + }; + var store = new FakeStore { Row = SqlRow() }; + + var answer = await Run(store, "{\"host\":\"beta-01.example.test\",\"password\":\"" + Secret + "\"}", capturing); + + Assert.Equal("updated", Status(answer)); + Assert.Equal(Secret, testedWith); + Assert.Contains(store.LastSets, s => s.Column == "encrypted_password"); + } + + [Fact] + public async Task AStoreThatRefusesAMoveWithoutASecret_GivesTheSameAnswerThePlanGives() + { + var store = new FakeStore { Row = SqlRow(), WriteResult = Edit.ServerEditWriteKind.PasswordNeeded }; + + var answer = JsonNode.Parse(await Run(store, "{\"monthly_cost_usd\":5}"))!; + + Assert.Equal("invalid", answer["status"]!.GetValue()); + Assert.Equal(Edit.EditPasswordNeededText, answer["message"]!.GetValue()); + } + + [Fact] + public async Task AStoreThatRefusesAReferenceAsThePassword_GivesTheOneSentenceForIt() + { + var store = new FakeStore { Row = SqlRow(), WriteResult = Edit.ServerEditWriteKind.ReferenceRefused }; + + var answer = JsonNode.Parse(await Run(store, "{\"monthly_cost_usd\":5}"))!; + + Assert.Equal("invalid", answer["status"]!.GetValue()); + Assert.Equal("Enter the password itself. References (env: or file:) can only be set in the configuration file.", answer["message"]!.GetValue()); + Assert.Equal(Edit.EditReferenceRefusedText, answer["message"]!.GetValue()); + } + + [Fact] + public async Task AStoreThatKeepsARemediationLoginOnAMove_GivesTheOneSentenceForIt() + { + var store = new FakeStore { Row = SqlRow(), WriteResult = Edit.ServerEditWriteKind.RemediationKept }; + + var answer = JsonNode.Parse(await Run(store, "{\"monthly_cost_usd\":5}"))!; + + Assert.Equal("invalid", answer["status"]!.GetValue()); + Assert.Equal("This server has a remediation login stored. Change how it is reached on the service host, in the configuration file or with --add-server.", answer["message"]!.GetValue()); + Assert.Equal(Edit.EditRemediationKeptText, answer["message"]!.GetValue()); + } +} diff --git a/Darling/Darling.Tests/ServerEditGateLiveTests.cs b/Darling/Darling.Tests/ServerEditGateLiveTests.cs new file mode 100644 index 0000000000..07dce38ce2 --- /dev/null +++ b/Darling/Darling.Tests/ServerEditGateLiveTests.cs @@ -0,0 +1,154 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Net; +using System.Net.Http; +using System.Security.Cryptography; +using System.Text; +using System.Threading.Tasks; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.TestHost; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using PerformanceMonitor.Darling.Analysis; +using PerformanceMonitor.Darling.Service.Hosting; +using PerformanceMonitor.Darling.Service.Mcp; +using Xunit; + +namespace Darling.Tests; + +/// +/// The #5240 edit routes through the REAL web pipeline (DarlingWebHostService.ConfigurePipeline, as +/// does), with session cookies minted under a key the test chose: an +/// anonymous caller is 401, a read-only sign-in is 403 on the unsafe edit method and on the by-id admin read (the +/// host's group gate lets that GET through, so the route refuses it itself), an editing sign-in reaches the route, and +/// only PATCH is routed on the edit path. The store pool is never opened: every request that reaches the route is +/// one the route refuses before it touches the store. +/// +public sealed class ServerEditGateLiveTests +{ + private const string ListenIp = "192.168.1.205"; + private const string Token = "correct-token-value"; + private static readonly IPAddress InCidrRemote = IPAddress.Parse("192.168.1.50"); + private static readonly byte[] Key = RandomNumberGenerator.GetBytes(32); + + private static string Cookie(string? subject, WebOidcRole? role) => + DarlingWebHostService.SessionCookieName + "=" + DarlingWebHostService.BuildSessionCookieValue( + Key, DateTimeOffset.UtcNow.AddHours(1), subject is null ? null : DarlingWebSeat.EncodeCookieSubject(subject, role!.Value)); + + private static async Task BuildServer() + { + var builder = WebApplication.CreateBuilder(); + builder.WebHost.UseTestServer(); + builder.Logging.ClearProviders(); + var postgres = NpgsqlDataSource.Create("Host=localhost;Database=postgres;Username=darling"); + builder.Services.AddSingleton(postgres); + DarlingWebHostService.ConfigureResponseCompression(builder.Services); + var app = builder.Build(); + var host = new DarlingWebHostService( + NullLogger.Instance, new WebRuntimeState(), new CollectorRuntimeState(), new WebTlsCertificateState(), new BaselineCache()); + host.ConfigurePipeline( + app, postgres, networkMode: true, networkListenIp: IPAddress.Parse(ListenIp), allowedCidr: IPNetwork.Parse("192.168.1.0/24"), + accessToken: Token, oidcClient: null, publicBaseUrlHost: null, sessionSigningKeyForTests: Key); + await app.StartAsync(); + return app.GetTestServer(); + } + + private static async Task<(int Status, string Body)> SendAsync(TestServer server, string method, string path, string? cookie, string? body = null) + { + var ctx = await server.SendAsync(c => + { + c.Request.Method = method; + c.Request.Path = path; + c.Request.Headers.Host = ListenIp; + c.Connection.RemoteIpAddress = InCidrRemote; + if (cookie is not null) + { + c.Request.Headers["Cookie"] = cookie; + } + + if (body is not null) + { + c.Request.ContentType = "application/json"; + c.Request.Body = new System.IO.MemoryStream(Encoding.UTF8.GetBytes(body)); + } + }); + return (ctx.Response.StatusCode, await new System.IO.StreamReader(ctx.Response.Body).ReadToEndAsync()); + } + + private const string Edit = "{\"display_name\":\"x\",\"password\":\"Zq9-fake-secret-must-never-appear-7XK\"}"; + + [Theory] + [InlineData("PATCH", "/api/servers/41")] + [InlineData("GET", "/api/admin/servers/41")] + public async Task AnAnonymousCaller_IsRefused401Json_OnBothEditRoutes(string method, string path) + { + using var server = await BuildServer(); + var (status, body) = await SendAsync(server, method, path, cookie: null, method == "PATCH" ? Edit : null); + Assert.Equal(StatusCodes.Status401Unauthorized, status); + Assert.Contains("\"error\"", body, StringComparison.Ordinal); + Assert.DoesNotContain("Zq9", body, StringComparison.Ordinal); + } + + [Fact] + public async Task AReadOnlySignIn_IsRefused403_OnPatch_BeforeTheRoute_WithoutEchoingTheBody() + { + using var server = await BuildServer(); + var (status, body) = await SendAsync(server, "PATCH", "/api/servers/41", Cookie("bob", WebOidcRole.Viewer), Edit); + Assert.Equal(StatusCodes.Status403Forbidden, status); + Assert.Contains("read-only", body, StringComparison.Ordinal); + Assert.DoesNotContain("Zq9", body, StringComparison.Ordinal); + } + + [Fact] + public async Task AReadOnlySignIn_IsRefused403_OnTheByIdAdminRead_BeforeTheStore() + { + using var server = await BuildServer(); + var (status, body) = await SendAsync(server, "GET", "/api/admin/servers/41", Cookie("bob", WebOidcRole.Viewer)); + Assert.Equal(StatusCodes.Status403Forbidden, status); + Assert.Contains("read-only", body, StringComparison.Ordinal); + } + + [Fact] + public async Task AnEditingSignIn_ReachesThePatchRoute_WhichRefusesAMissingTokenWith400() + { + using var server = await BuildServer(); + var (status, body) = await SendAsync(server, "PATCH", "/api/servers/41", Cookie("alice", WebOidcRole.Admin), Edit); + Assert.Equal(StatusCodes.Status400BadRequest, status); + Assert.Contains("expected_modified_at", body, StringComparison.Ordinal); + Assert.DoesNotContain("Zq9", body, StringComparison.Ordinal); + } + + [Theory] + [InlineData("POST")] + [InlineData("PUT")] + [InlineData("DELETE")] + public async Task OnlyPatchIsRoutedOnTheEditPath_ForAnEditingSignIn(string method) + { + using var server = await BuildServer(); + var (status, _) = await SendAsync(server, method, "/api/servers/41", Cookie("alice", WebOidcRole.Admin), Edit); + Assert.Equal(StatusCodes.Status405MethodNotAllowed, status); + } + + [Theory] + [InlineData("POST")] + [InlineData("PUT")] + [InlineData("PATCH")] + [InlineData("DELETE")] + public async Task EveryUnsafeMethod_OnTheAdminReadPath_IsRefusedForAReadOnlySignIn_AndNotRoutedForAnEditingOne(string method) + { + using var server = await BuildServer(); + Assert.Equal(StatusCodes.Status403Forbidden, (await SendAsync(server, method, "/api/admin/servers/41", Cookie("bob", WebOidcRole.Viewer), "{}")).Status); + Assert.Equal(StatusCodes.Status405MethodNotAllowed, (await SendAsync(server, method, "/api/admin/servers/41", Cookie("alice", WebOidcRole.Admin), "{}")).Status); + } +} diff --git a/Darling/Darling.Tests/ServerEditLiveTests.cs b/Darling/Darling.Tests/ServerEditLiveTests.cs new file mode 100644 index 0000000000..f15a2cd563 --- /dev/null +++ b/Darling/Darling.Tests/ServerEditLiveTests.cs @@ -0,0 +1,583 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Security.Cryptography; +using System.Text.Json.Nodes; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; +using Edit = PerformanceMonitor.Darling.Service.Mcp.DarlingMcpServerAdminTools; + +namespace Darling.Tests; + +/* #1776 own-store: each fact mints its own scratch database through ScratchPostgres and never touches another + one. The role it connects as is created for the fact and dropped in its cleanup. */ + +/// +/// edit_server (#5240) against a store, connected as the mcp role: the statements are TAKEN FROM +/// with the role name swapped, so removing a grant the edit +/// needs makes these fail with 42501. The owner-side reads prove what the write left alone. +/// +[Collection("darling-owned-secrets")] +public sealed class ServerEditLiveTests : IDisposable +{ + private readonly DarlingOwnedSet _ownedBefore = DarlingOwnedSecrets.Current; + + public ServerEditLiveTests() => DarlingOwnedSecrets.Set(DarlingOwnedSet.Empty); + + public void Dispose() => DarlingOwnedSecrets.Set(_ownedBefore); + + private static readonly string RolePassword = Convert.ToHexString(RandomNumberGenerator.GetBytes(16)); + + private const string SecretRef = "Synth-Edit-Live-Pw-1"; + + private static readonly Edit.ServerProbe Reachable = (_, _) => Task.FromResult( + new ConnectionProbeResult(true, 15, 3, "Enterprise", false, false, false, true, null)); + + private sealed record Rig(ScratchPostgres Scratch, NpgsqlDataSource Owner, NpgsqlDataSource Mcp, string Role) : IAsyncDisposable + { + public async ValueTask DisposeAsync() + { + await Mcp.DisposeAsync(); + await Owner.DisposeAsync(); + await Scratch.DisposeAsync(); + } + } + + private static List McpStatements(string roleName, string database) + { + var provisioning = DarlingManagedRoles.BuildProvisioningSql( + ProvisioningTestSecrets.Admin, ProvisioningTestSecrets.Viewer, ProvisioningTestSecrets.Mcp); + var statements = new List(); + var uncommented = string.Join('\n', provisioning.Split('\n').Where(l => !l.TrimStart().StartsWith("--", StringComparison.Ordinal))); + foreach (var raw in uncommented.Split(';')) + { + var statement = raw.Trim(); + var isGrant = statement.StartsWith("GRANT ", StringComparison.Ordinal); + if (!isGrant && !statement.StartsWith("REVOKE ", StringComparison.Ordinal)) + { + continue; + } + + var marker = isGrant ? " TO " : " FROM "; + var at = statement.LastIndexOf(marker, StringComparison.Ordinal); + if (at < 0 || statement.Contains("EXECUTE ON FUNCTION", StringComparison.Ordinal)) + { + continue; + } + + if (!statement[(at + marker.Length)..].Split(',').Select(t => t.Trim()).Contains("mcp", StringComparer.Ordinal)) + { + continue; + } + + statements.Add(ViewerGrantReplay.OnDatabase(statement[..at], database) + marker + roleName); + } + + return statements; + } + + private static async Task OpenAsync(CancellationToken ct) + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live edit_server tests (each mints its own scratch database and role)."); + + var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var ownerString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { SearchPath = PgSchemaGenerator.SearchPath }.ConnectionString; + await using (var connection = new NpgsqlConnection(ownerString)) + { + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, null, ct); + } + + var owner = NpgsqlDataSource.Create(ownerString); + var role = "srv_edit_" + Guid.NewGuid().ToString("N")[..8]; + await ExecAsync(owner, $"CREATE ROLE {role} LOGIN NOSUPERUSER PASSWORD '{RolePassword}'", ct); + foreach (var statement in McpStatements(role, scratch.DatabaseName)) + { + await ExecAsync(owner, statement, ct); + } + + /* The edit write is a function (#5240) the provisioning batch creates and grants: the real one, EXECUTE for this role. */ + await ExecAsync(owner, DarlingManagedRoles.BuildEditMonitoredServerFunctionSql("config"), ct); + await ExecAsync(owner, DarlingManagedRoles.BuildEditMonitoredServerLegacyWrapperSql("config"), ct); + await ExecAsync(owner, $"GRANT EXECUTE ON FUNCTION config.edit_monitored_server({DarlingManagedRoles.EditMonitoredServerSignature}) TO {role}", ct); + await ExecAsync(owner, $"GRANT EXECUTE ON FUNCTION config.edit_monitored_server({DarlingManagedRoles.EditMonitoredServerLegacySignature}) TO {role}", ct); + await ExecAsync(owner, "INSERT INTO config_service (id) VALUES (1) ON CONFLICT DO NOTHING", ct); + var mcp = NpgsqlDataSource.Create(new NpgsqlConnectionStringBuilder(ownerString) { Username = role, Password = RolePassword }.ConnectionString); + return new Rig(scratch, owner, mcp, role); + } + + private static async Task DropRoleAsync(Rig rig, bool bodySucceeded) => + await LiveStoreCleanup.RunOwnedAsync(bodySucceeded, () => + ExecAsync(rig.Owner, $"DROP OWNED BY {rig.Role}; DROP ROLE IF EXISTS {rig.Role};", CancellationToken.None)); + + private static async Task ExecAsync(NpgsqlDataSource source, string sql, CancellationToken ct) + { + await using var command = source.CreateCommand(sql); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task ScalarAsync(NpgsqlDataSource source, string sql, CancellationToken ct) + { + await using var command = source.CreateCommand(sql); + return (T)Convert.ChangeType((await command.ExecuteScalarAsync(ct))!, typeof(T)); + } + + private static Task SeedServerAsync( + NpgsqlDataSource owner, int id, string name, string host, string auth, string? username, string? blob, CancellationToken ct, + string? remediationBlob = null) => + ExecAsync(owner, $@"INSERT INTO config_monitored_servers + (server_id, name, host, auth, username, encrypted_password, excluded_databases, capture_plans, is_enabled, + alert_delivery_mode_override, plan_force_bot_enabled, remediation_username, remediation_encrypted_password, monthly_cost_usd) + VALUES ({id}, '{name}', '{host}', '{auth}', {(username is null ? "NULL" : "'" + username + "'")}, {(blob is null ? "NULL" : "'" + blob + "'")}, + ARRAY['tempdb','model'], TRUE, FALSE, 'PerEvent', TRUE, 'rem-user', {(remediationBlob is null ? "NULL" : "'" + remediationBlob + "'")}, 7)", ct); + + private static async Task RowSignatureAsync(NpgsqlDataSource owner, int id, CancellationToken ct) => + await ScalarAsync(owner, $@"SELECT concat_ws('|', is_enabled, array_to_string(excluded_databases, ','), capture_plans, + alert_delivery_mode_override, plan_force_bot_enabled, remediation_username, remediation_encrypted_password, engine, created_at) + FROM config_monitored_servers WHERE server_id = {id}", ct); + + private static JsonNode Parse(string answer) => JsonNode.Parse(answer)!; + + /// Opens the stored value of a row with , through the connection settings the row holds. + private static async Task OpenStoredAsync(NpgsqlDataSource owner, int id, IPasswordKeyRing ring, CancellationToken ct) + { + await using var command = owner.CreateCommand( + "SELECT " + ServerConnectionIdentity.StoredColumns + ", encrypted_password FROM config_monitored_servers WHERE server_id = " + id); + await using var reader = await command.ExecuteReaderAsync(ct); + Assert.True(await reader.ReadAsync(ct)); + var identity = ServerConnectionIdentity.FromStoredColumns( + reader.IsDBNull(0) ? null : reader.GetString(0), reader.IsDBNull(1) ? null : reader.GetInt32(1), + reader.IsDBNull(2) ? null : reader.GetString(2), reader.IsDBNull(3) ? null : reader.GetString(3), + !reader.IsDBNull(4) && reader.GetBoolean(4), reader.IsDBNull(5) ? null : reader.GetString(5), + reader.IsDBNull(6) ? null : reader.GetString(6), reader.IsDBNull(7) ? null : reader.GetString(7), + !reader.IsDBNull(8) && reader.GetBoolean(8), !reader.IsDBNull(9) && reader.GetBoolean(9)); + return ring.Open(reader.GetString(10), PasswordBinding.ForServer(identity)); + } + + [Fact] + public async Task AsTheMcpRole_AnEditOfNameCostAndAddress_KeepsTheIdTagsAndSettings_AndBumpsTheBeaconOnce() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + var ok = false; + try + { + await SeedServerAsync(rig.Owner, 5101, "alpha-01", "alpha-01.example.test", "integrated", null, null, ct); + await SeedServerAsync(rig.Owner, 5102, "alpha-02", "alpha-02.example.test", "integrated", null, null, ct); + var tags = new ServerTagStore(rig.Owner, 30); + var tag = Assert.IsType(await tags.CreateAsync("East", null, null, ct)).Tag!; + await tags.AssignAsync(tag.Id, [5101, 5102], ct); + var before = await RowSignatureAsync(rig.Owner, 5101, ct); + var versionBefore = await ScalarAsync(rig.Owner, "SELECT config_version FROM config_service", ct); + var modifiedBefore = await ScalarAsync(rig.Owner, "SELECT modified_at::text FROM config_monitored_servers WHERE server_id = 5101", ct); + + var name = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "alpha-01.example.test", "{\"display_name\":\"Orders\",\"monthly_cost_usd\":120}", Reachable, TestKeyRings.Healthy, null, ct)); + Assert.Equal("updated", name["status"]!.GetValue()); + Assert.Equal(5101, name["server_id"]!.GetValue()); + Assert.Equal(versionBefore + 1, await ScalarAsync(rig.Owner, "SELECT config_version FROM config_service", ct)); + + var address = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "Orders", "{\"host\":\"alpha-09.example.test\"}", Reachable, TestKeyRings.Healthy, null, ct)); + Assert.Equal("updated", address["status"]!.GetValue()); + Assert.Equal(5101, address["server_id"]!.GetValue()); + Assert.Contains("keeps its id", address["note"]!.GetValue(), StringComparison.Ordinal); + + Assert.Equal("Orders", await ScalarAsync(rig.Owner, "SELECT name FROM config_monitored_servers WHERE server_id = 5101", ct)); + Assert.Equal("alpha-09.example.test", await ScalarAsync(rig.Owner, "SELECT host FROM config_monitored_servers WHERE server_id = 5101", ct)); + Assert.Equal(120L, await ScalarAsync(rig.Owner, "SELECT monthly_cost_usd::bigint FROM config_monitored_servers WHERE server_id = 5101", ct)); + Assert.NotEqual(modifiedBefore, await ScalarAsync(rig.Owner, "SELECT modified_at::text FROM config_monitored_servers WHERE server_id = 5101", ct)); + Assert.Equal(before, await RowSignatureAsync(rig.Owner, 5101, ct)); + + /* The tags stayed, on the same id, for the edited server and its neighbour. */ + Assert.Equal(1L, await ScalarAsync(rig.Owner, "SELECT count(*) FROM server_tag_map WHERE server_id = 5101", ct)); + Assert.Equal(1L, await ScalarAsync(rig.Owner, "SELECT count(*) FROM server_tag_map WHERE server_id = 5102", ct)); + Assert.Equal(2L, await ScalarAsync(rig.Owner, "SELECT count(*) FROM config_monitored_servers", ct)); + + /* A request equal to the stored values writes nothing: no statement, so no beacon bump. */ + var versionMid = await ScalarAsync(rig.Owner, "SELECT config_version FROM config_service", ct); + var same = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "Orders", "{\"display_name\":\"Orders\"}", Reachable, TestKeyRings.Healthy, null, ct)); + Assert.Equal("unchanged", same["status"]!.GetValue()); + Assert.Equal(versionMid, await ScalarAsync(rig.Owner, "SELECT config_version FROM config_service", ct)); + ok = true; + } + finally + { + await DropRoleAsync(rig, ok); + } + } + + [Fact] + public async Task AsTheMcpRole_AnAddressEditOnARowWithARemediationLogin_IsRefused_AndWritesNothing_WhileANameEditSaves() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + var ok = false; + try + { + await SeedServerAsync(rig.Owner, 5191, "alpha-91", "alpha-91.example.test", "integrated", null, null, ct, remediationBlob: "rem-blob"); + var before = await RowSignatureAsync(rig.Owner, 5191, ct); + var versionBefore = await ScalarAsync(rig.Owner, "SELECT config_version FROM config_service", ct); + + var moved = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "alpha-91.example.test", "{\"host\":\"alpha-99.example.test\"}", Reachable, TestKeyRings.Healthy, null, ct)); + Assert.Equal("invalid", moved["status"]!.GetValue()); + Assert.Equal(Edit.EditRemediationKeptText, moved["message"]!.GetValue()); + Assert.Equal("alpha-91.example.test", await ScalarAsync(rig.Owner, "SELECT host FROM config_monitored_servers WHERE server_id = 5191", ct)); + Assert.Equal(before, await RowSignatureAsync(rig.Owner, 5191, ct)); + Assert.Equal(versionBefore, await ScalarAsync(rig.Owner, "SELECT config_version FROM config_service", ct)); + + var renamed = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "alpha-91.example.test", "{\"display_name\":\"Renamed\"}", Reachable, TestKeyRings.Healthy, null, ct)); + Assert.Equal("updated", renamed["status"]!.GetValue()); + Assert.Equal(before, await RowSignatureAsync(rig.Owner, 5191, ct)); + ok = true; + } + finally + { + await DropRoleAsync(rig, ok); + } + } + + [Fact] + public async Task AsTheMcpRole_RoleAndExternalIdEditsGoThroughTheEditFunction_AndEveryOtherColumnStaysUpdatable() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + var ok = false; + try + { + await SeedServerAsync(rig.Owner, 5192, "alpha-92", "alpha-92.example.test", "integrated", null, null, ct); + await ExecAsync(rig.Owner, + "UPDATE config_monitored_servers SET engine = 'postgres', port = 5432, aws_role_arn = 'arn:aws:iam::123456789012:role/a', aws_external_id = 'tenant-1234' WHERE server_id = 5192", ct); + + /* A direct UPDATE of the role or the external ID is not allowed for this login; the edit function is the way. */ + var roleUpdate = await Assert.ThrowsAsync(() => + ExecAsync(rig.Mcp, "UPDATE config_monitored_servers SET aws_role_arn = 'arn:aws:iam::123456789012:role/b' WHERE server_id = 5192", ct)); + Assert.Equal("42501", roleUpdate.SqlState); + var externalUpdate = await Assert.ThrowsAsync(() => + ExecAsync(rig.Mcp, "UPDATE config_monitored_servers SET aws_external_id = 'tenant-5678' WHERE server_id = 5192", ct)); + Assert.Equal("42501", externalUpdate.SqlState); + Assert.Equal("arn:aws:iam::123456789012:role/a|tenant-1234", + await ScalarAsync(rig.Owner, "SELECT concat_ws('|', aws_role_arn, aws_external_id) FROM config_monitored_servers WHERE server_id = 5192", ct)); + + /* Every other column is still updatable: the grant names each column of the table but the three AWS ones. */ + await ExecAsync(rig.Mcp, "UPDATE config_monitored_servers SET name = 'alpha-92b', is_enabled = TRUE, excluded_databases = ARRAY['model'] WHERE server_id = 5192", ct); + var columns = await ScalarAsync(rig.Owner, + "SELECT string_agg(attname, ',' ORDER BY attname) FROM pg_attribute WHERE attrelid = 'config.config_monitored_servers'::regclass " + + "AND attnum > 0 AND NOT attisdropped AND attname NOT IN ('aws_role_arn', 'aws_external_id', 'aws_external_id_set')", ct); + Assert.Equal( + string.Join(',', DarlingManagedRoles.McpMonitoredServerUpdateColumns.Split(',', StringSplitOptions.TrimEntries).Order(StringComparer.Ordinal)), + columns); + + /* An add still names the role, and the edit function still changes it, with the external ID sent or cleared. */ + await ExecAsync(rig.Mcp, + "INSERT INTO config_monitored_servers (server_id, name, host, auth, engine, port, aws_role_arn, aws_external_id) " + + "VALUES (5193, 'alpha-93', 'alpha-93.example.test', 'sql', 'postgres', 5432, 'arn:aws:iam::123456789012:role/a', 'tenant-1234')", ct); + var token = await ScalarAsync(rig.Owner, "SELECT to_char(modified_at, 'YYYY-MM-DD HH24:MI:SS.US') FROM config_monitored_servers WHERE server_id = 5192", ct); + var outcome = await ScalarAsync(rig.Mcp, + $"SELECT outcome FROM config.edit_monitored_server(5192, '{token}'::timestamp, ARRAY['aws_role_arn','aws_external_id']::text[], " + + "NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'arn:aws:iam::123456789012:role/b', NULL)", ct); + Assert.Equal("saved", outcome); + Assert.Equal("arn:aws:iam::123456789012:role/b|", + await ScalarAsync(rig.Owner, "SELECT concat_ws('|', aws_role_arn, aws_external_id, '') FROM config_monitored_servers WHERE server_id = 5192", ct)); + ok = true; + } + finally + { + await DropRoleAsync(rig, ok); + } + } + + [Fact] + public async Task AsTheMcpRole_EmptyRequiredFields_AnswerInvalidValue_OnBothSignatures_AndWriteNothing() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + var ok = false; + try + { + await SeedServerAsync(rig.Owner, 5194, "alpha-94", "alpha-94.example.test", "sql", "monitor", "blob-94", ct); + await ExecAsync(rig.Owner, + "UPDATE config_monitored_servers SET engine = 'postgres', port = 5432, aws_role_arn = 'arn:aws:iam::123456789012:role/a', aws_external_id = 'tenant-1234' WHERE server_id = 5194", ct); + var before = await ScalarAsync(rig.Owner, + "SELECT concat_ws('|', name, host, port, auth, encrypted_password, aws_role_arn, aws_external_id, modified_at) FROM config_monitored_servers WHERE server_id = 5194", ct); + + foreach (var columns in new[] { "'name'", "'host','encrypted_password'", "'port'", "'auth'", "'encrypt_mode'", "'read_only_intent'", "'monthly_cost_usd'" }) + { + var token = await ScalarAsync(rig.Owner, "SELECT to_char(modified_at, 'YYYY-MM-DD HH24:MI:SS.US') FROM config_monitored_servers WHERE server_id = 5194", ct); + var seventeen = await ScalarAsync(rig.Mcp, + $"SELECT outcome FROM config.edit_monitored_server(5194, '{token}'::timestamp, ARRAY[{columns}]::text[], " + + "NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'typed-secret', NULL, NULL, NULL, NULL, NULL, NULL)", ct); + Assert.Equal("invalid_value", seventeen); + var fifteen = await ScalarAsync(rig.Mcp, + $"SELECT outcome FROM config.edit_monitored_server(5194, '{token}'::timestamp, ARRAY[{columns}]::text[], " + + "NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'typed-secret', NULL, NULL, NULL, NULL)", ct); + Assert.Equal("invalid_value", fifteen); + } + + /* A role the table's format check does not accept is the same outcome, and nothing changed. */ + var roleToken = await ScalarAsync(rig.Owner, "SELECT to_char(modified_at, 'YYYY-MM-DD HH24:MI:SS.US') FROM config_monitored_servers WHERE server_id = 5194", ct); + Assert.Equal("invalid_value", await ScalarAsync(rig.Mcp, + $"SELECT outcome FROM config.edit_monitored_server(5194, '{roleToken}'::timestamp, ARRAY['aws_role_arn','aws_external_id']::text[], " + + "NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'not-an-arn', 'tenant-1234')", ct)); + + Assert.Equal(before, await ScalarAsync(rig.Owner, + "SELECT concat_ws('|', name, host, port, auth, encrypted_password, aws_role_arn, aws_external_id, modified_at) FROM config_monitored_servers WHERE server_id = 5194", ct)); + ok = true; + } + finally + { + await DropRoleAsync(rig, ok); + } + } + + [Fact] + public async Task AStaleToken_IsAConflict_AndALostRace_IsAConflict_WithNothingWritten() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + var ok = false; + try + { + await SeedServerAsync(rig.Owner, 5111, "alpha-11", "alpha-11.example.test", "integrated", null, null, ct); + var first = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "alpha-11", "{\"monthly_cost_usd\":1}", Reachable, TestKeyRings.Healthy, null, ct)); + var token = first["modified_at"]!.GetValue(); + + /* The token an earlier answer returned is accepted once, then it is stale. */ + var good = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "alpha-11", $"{{\"monthly_cost_usd\":2,\"expected_modified_at\":\"{token}\"}}", Reachable, TestKeyRings.Healthy, null, ct)); + Assert.Equal("updated", good["status"]!.GetValue()); + Assert.NotEqual(token, good["modified_at"]!.GetValue()); + var stale = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "alpha-11", $"{{\"monthly_cost_usd\":3,\"expected_modified_at\":\"{token}\"}}", Reachable, TestKeyRings.Healthy, null, ct)); + Assert.Equal("conflict", stale["status"]!.GetValue()); + Assert.Equal(2L, await ScalarAsync(rig.Owner, "SELECT monthly_cost_usd::bigint FROM config_monitored_servers WHERE server_id = 5111", ct)); + + /* The race: a write lands between the core's read and its UPDATE. The probe stands in the gap. */ + Edit.ServerProbe racing = async (_, c) => + { + await ExecAsync(rig.Owner, "UPDATE config_monitored_servers SET modified_at = modified_at + interval '1 second' WHERE server_id = 5111", c); + return new ConnectionProbeResult(true, 15, 3, "Enterprise", false, false, false, true, null); + }; + var lost = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "alpha-11", "{\"host\":\"alpha-12.example.test\"}", racing, TestKeyRings.Healthy, null, ct)); + Assert.Equal("conflict", lost["status"]!.GetValue()); + Assert.Equal("alpha-11", lost["current"]!["display_name"]!.GetValue()); + Assert.Equal("alpha-11.example.test", lost["current"]!["host"]!.GetValue()); + Assert.Equal("alpha-11.example.test", await ScalarAsync(rig.Owner, "SELECT host FROM config_monitored_servers WHERE server_id = 5111", ct)); + ok = true; + } + finally + { + await DropRoleAsync(rig, ok); + } + } + + [Fact] + public async Task AnAddressAnotherServerHolds_Collides_AndNothingIsWritten() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + var ok = false; + try + { + await SeedServerAsync(rig.Owner, 5121, "alpha-21", "alpha-21.example.test", "integrated", null, null, ct); + await SeedServerAsync(rig.Owner, 5122, "alpha-22", "alpha-22.example.test", "integrated", null, null, ct); + + var answer = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "alpha-21", "{\"host\":\"ALPHA-22.example.test\"}", Reachable, TestKeyRings.Healthy, null, ct)); + + Assert.Equal("collides", answer["status"]!.GetValue()); + Assert.Equal("occupied", answer["reason"]!.GetValue()); + Assert.Equal("alpha-21.example.test", await ScalarAsync(rig.Owner, "SELECT host FROM config_monitored_servers WHERE server_id = 5121", ct)); + + var ambiguous = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "alpha-2", "{\"monthly_cost_usd\":1}", Reachable, TestKeyRings.Healthy, null, ct)); + Assert.Equal("ambiguous", ambiguous["status"]!.GetValue()); + ok = true; + } + finally + { + await DropRoleAsync(rig, ok); + } + } + + [Fact] + public async Task ASecretEdit_StoresTheProtectedSecret_AndTheMcpRoleStillCannotReadItBack() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + var ok = false; + try + { + await SeedServerAsync(rig.Owner, 5131, "alpha-31", "alpha-31.example.test", "sql", "monitor", "env:OLD_REF", ct); + + var refused = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "alpha-31", "{\"host\":\"alpha-32.example.test\"}", Reachable, TestKeyRings.Healthy, null, ct)); + Assert.Equal("invalid", refused["status"]!.GetValue()); + Assert.Equal("env:OLD_REF", await ScalarAsync(rig.Owner, "SELECT encrypted_password FROM config_monitored_servers WHERE server_id = 5131", ct)); + + var saved = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "alpha-31", $"{{\"host\":\"alpha-32.example.test\",\"password\":\"{SecretRef}\"}}", Reachable, TestKeyRings.Healthy, null, ct)); + Assert.Equal("updated", saved["status"]!.GetValue()); + Assert.True(saved["tested"]!.GetValue()); + Assert.DoesNotContain(SecretRef, saved.ToJsonString(), StringComparison.Ordinal); + var storedAfterSave = await ScalarAsync(rig.Owner, "SELECT encrypted_password FROM config_monitored_servers WHERE server_id = 5131", ct); + /* The value is sealed for the row's own connection settings: it opens with the key through them. */ + Assert.Equal(SecretRef, await OpenStoredAsync(rig.Owner, 5131, TestKeyRings.Healthy, ct)); + + var denied = await Assert.ThrowsAsync(async () => await ScalarAsync(rig.Mcp, "SELECT encrypted_password FROM config_monitored_servers", ct)); + Assert.Equal("42501", denied.SqlState); + + /* A probe that throws with the submitted secret in its message: the outer catch redacts it. */ + Edit.ServerProbe throwing = (_, _) => throw new InvalidOperationException("driver said: " + SecretRef); + var thrown = await Edit.EditServerByNameAsync(rig.Mcp, "alpha-31", $"{{\"host\":\"alpha-33.example.test\",\"password\":\"{SecretRef}\"}}", throwing, TestKeyRings.Healthy, null, ct); + Assert.DoesNotContain(SecretRef, thrown, StringComparison.Ordinal); + + /* A name-only edit leaves the stored secret alone, with no password in the request. */ + var rename = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "alpha-31", "{\"display_name\":\"Beta\"}", Reachable, TestKeyRings.Healthy, null, ct)); + Assert.Equal("updated", rename["status"]!.GetValue()); + Assert.Equal(storedAfterSave, await ScalarAsync(rig.Owner, "SELECT encrypted_password FROM config_monitored_servers WHERE server_id = 5131", ct)); + ok = true; + } + finally + { + await DropRoleAsync(rig, ok); + } + } + + /* ---------------- sealing at add and edit, read back the way the service reads it (#5366) ---------------- */ + + private const string TypedPassword = "p@ss-not-real"; + private const string RetypedPassword = "p@ss-not-real-2"; + + /// The enabled rows, read the way the service reads them at startup. + private static async Task> ReadServersAsync(NpgsqlDataSource owner, CancellationToken ct) + { + await using var connection = await owner.OpenConnectionAsync(ct); + return await StoreConfigProvider.ReadMonitoredServersAsync(connection, new DarlingConfig(), ct); + } + + [Fact] + public async Task AnAdd_WithALiteralPassword_StoresASealedValue_ThatTheResolverOpensForTheRowsOwnSettings() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + var ok = false; + try + { + var answer = await Edit.AddServersAsync(rig.Mcp, + $"[{{\"host\":\"alpha-41.example.test\",\"auth\":\"SQL\",\"username\":\"monitor\",\"password\":\"{TypedPassword}\"}}]", + Reachable, ct, TestKeyRings.Healthy); + Assert.Contains("\"added\"", answer, StringComparison.Ordinal); + Assert.DoesNotContain(TypedPassword, answer, StringComparison.Ordinal); + + var server = Assert.Single(await ReadServersAsync(rig.Owner, ct), s => s.Host == "alpha-41.example.test"); + Assert.StartsWith("sealed:", server.EncryptedPassword, StringComparison.Ordinal); + Assert.DoesNotContain(TypedPassword, server.EncryptedPassword, StringComparison.Ordinal); + + /* The service's own resolver opens it with the key, through the settings the row stores. */ + Assert.Equal(TypedPassword, DarlingSecrets.ResolvePassword(server, out var usedPlaintext, TestKeyRings.Healthy)); + Assert.False(usedPlaintext); + + /* A ring that does not hold the key cannot open it, and the same value does not open for another host. */ + Assert.ThrowsAny(() => DarlingSecrets.ResolvePassword( + server, out _, DarlingPasswordKey.FromPrivateKey(PasswordPrivateKey.Generate()))); + var otherHost = server.ConnectionIdentity with { Host = "alpha-43.example.test" }; + Assert.Throws(() => TestKeyRings.Healthy.Open(server.EncryptedPassword!, PasswordBinding.ForServer(otherHost))); + ok = true; + } + finally + { + await DropRoleAsync(rig, ok); + } + } + + [Fact] + public async Task AnEdit_WithARetypedPasswordAndANewHost_OpensWithTheNewBinding_AndTheOldSealedValueNoLongerOpensForTheNewHost() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + var ok = false; + try + { + await Edit.AddServersAsync(rig.Mcp, + $"[{{\"host\":\"alpha-51.example.test\",\"auth\":\"SQL\",\"username\":\"monitor\",\"password\":\"{TypedPassword}\"}}]", + Reachable, ct, TestKeyRings.Healthy); + var before = Assert.Single(await ReadServersAsync(rig.Owner, ct), s => s.Host == "alpha-51.example.test"); + var oldSealed = before.EncryptedPassword; + Assert.Equal(TypedPassword, DarlingSecrets.ResolvePassword(before, out _, TestKeyRings.Healthy)); + + /* The host moves and the password is not typed again: refused, nothing written (the old value is bound to the old host). */ + var refused = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "alpha-51.example.test", "{\"host\":\"alpha-52.example.test\"}", Reachable, TestKeyRings.Healthy, null, ct)); + Assert.Equal("invalid", refused["status"]!.GetValue()); + Assert.Equal(oldSealed, (await ReadServersAsync(rig.Owner, ct)).Single(s => s.ServerId == before.ServerId).EncryptedPassword); + + var saved = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "alpha-51.example.test", + $"{{\"host\":\"alpha-52.example.test\",\"password\":\"{RetypedPassword}\"}}", Reachable, TestKeyRings.Healthy, null, ct)); + Assert.Equal("updated", saved["status"]!.GetValue()); + + var after = Assert.Single(await ReadServersAsync(rig.Owner, ct), s => s.ServerId == before.ServerId); + Assert.Equal("alpha-52.example.test", after.Host); + Assert.NotEqual(oldSealed, after.EncryptedPassword); + Assert.Equal(RetypedPassword, DarlingSecrets.ResolvePassword(after, out _, TestKeyRings.Healthy)); + + /* The value sealed for the old host does not open for the new one. */ + after.EncryptedPassword = oldSealed; + Assert.ThrowsAny(() => DarlingSecrets.ResolvePassword(after, out _, TestKeyRings.Healthy)); + ok = true; + } + finally + { + await DropRoleAsync(rig, ok); + } + } + + /* ---------------- the probe, against a real SQL Server ---------------- */ + + [Fact] + public async Task AgainstARealSqlServer_ACorrectSecretSaves_AndAWrongOneIsConnectionFailed_WithNoWrite() + { + var host = Environment.GetEnvironmentVariable("DARLING_TEST_SQL"); + Assert.SkipWhen(string.IsNullOrEmpty(host), "Set DARLING_TEST_SQL (and _USER / _PASSWORD) to run the live probe check."); + var user = Environment.GetEnvironmentVariable("DARLING_TEST_SQL_USER") ?? "sa"; + var password = Environment.GetEnvironmentVariable("DARLING_TEST_SQL_PASSWORD"); + Assert.SkipWhen(string.IsNullOrEmpty(password), "Set DARLING_TEST_SQL_PASSWORD to run the live probe check."); + + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + var ok = false; + try + { + var wrong = "not-the-password-" + Guid.NewGuid().ToString("N")[..6]; + Environment.SetEnvironmentVariable("SYNTH_EDIT_LIVE_GOOD", password); + await SeedServerAsync(rig.Owner, 5141, "alpha-41", host!, "sql", user, "env:SYNTH_EDIT_LIVE_GOOD", ct); + await ExecAsync(rig.Owner, "UPDATE config_monitored_servers SET trust_server_certificate = TRUE WHERE server_id = 5141", ct); + var before = await ScalarAsync(rig.Owner, "SELECT concat_ws('|', database, encrypted_password, modified_at::text) FROM config_monitored_servers WHERE server_id = 5141", ct); + + Edit.ServerProbe real = (config, c) => DarlingServerConnector.ProbeAsync(config, null, c); + + var bad = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "alpha-41", "{\"database\":\"master\",\"password\":\"" + wrong + "\"}", real, TestKeyRings.Healthy, null, ct)); + Assert.Equal("connection_failed", bad["status"]!.GetValue()); + Assert.DoesNotContain(wrong, bad.ToJsonString(), StringComparison.Ordinal); + Assert.Equal(before, await ScalarAsync(rig.Owner, "SELECT concat_ws('|', database, encrypted_password, modified_at::text) FROM config_monitored_servers WHERE server_id = 5141", ct)); + + var good = Parse(await Edit.EditServerByNameAsync(rig.Mcp, "alpha-41", "{\"database\":\"master\",\"password\":\"" + password + "\"}", real, TestKeyRings.Healthy, null, ct)); + Assert.Equal("updated", good["status"]!.GetValue()); + Assert.True(good["tested"]!.GetValue()); + Assert.Equal("master", await ScalarAsync(rig.Owner, "SELECT database FROM config_monitored_servers WHERE server_id = 5141", ct)); + ok = true; + } + finally + { + Environment.SetEnvironmentVariable("SYNTH_EDIT_LIVE_GOOD", null); + await DropRoleAsync(rig, ok); + } + } +} diff --git a/Darling/Darling.Tests/ServerEditPasswordRuleViewerTests.cs b/Darling/Darling.Tests/ServerEditPasswordRuleViewerTests.cs new file mode 100644 index 0000000000..482ae31804 --- /dev/null +++ b/Darling/Darling.Tests/ServerEditPasswordRuleViewerTests.cs @@ -0,0 +1,422 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; +using Edit = PerformanceMonitor.Darling.Service.Mcp.DarlingMcpServerAdminTools; + +namespace Darling.Tests; + +/* #1776 own-store: each live fact mints its own scratch database through ScratchPostgres and never touches another one. */ + +/// +/// The desktop viewer's server edit follows the same password rule as the web and MCP edit (#5240): when a SQL or +/// service-principal server's host or port changes, the stored password is not reused, and the save is refused until a +/// newly entered password comes with it. The rule lives in , +/// so any caller of the upsert gets it, and the dialog asks for the password again before the save and the connection +/// test. The viewer cannot reference the service, so the rule and its sentence are copies; the first facts hold them to +/// the service's. The last facts need a store. +/// +[Collection("live-postgres")] +public sealed class ServerEditPasswordRuleViewerTests +{ + private const int ServerId = 7501; + private const string StoredHost = "edit-rule.example.test"; + private const string MovedHost = "edit-rule-moved.example.test"; + private const string StoredBlob = "stored-blob"; + private const string NewBlob = "newly-protected-blob"; + + [Fact] + public void TheViewersSentence_IsTheWebAndMcpEditsSentence() + { + Assert.Equal(Edit.EditPasswordNeededText, ViewerDataService.EditPasswordNeededText); + Assert.Equal(Edit.EditPasswordNeededText, new MonitoredServerPasswordNeededException().Message); + } + + /// The rule the viewer's edit applies (, with the + /// form's host trimmed of spaces as the dialog trims it) to a form that changes only the host and port. + private static bool ViewerMoves(string storedHost, int storedPort, string newHost, int newPort) => + ServerConnectionIdentity.Differ( + ServerConnectionIdentity.FromStoredColumns(newHost.Trim(' '), newPort, null, null, false, null, null, null, false, false), + ServerConnectionIdentity.FromStoredColumns(storedHost, storedPort, null, null, false, null, null, null, false, false)); + + [Theory] + [InlineData("edit-rule.example.test", 0, "edit-rule.example.test", 0, false)] + [InlineData("edit-rule.example.test", 0, " edit-rule.example.test ", 0, false)] + [InlineData("edit-rule.example.test", 0, "edit-rule-moved.example.test", 0, true)] + [InlineData("edit-rule.example.test", 0, "EDIT-RULE.example.test", 0, true)] + [InlineData("edit-rule.example.test\\inst", 0, "edit-rule.example.test", 0, true)] + [InlineData("edit-rule.example.test", 0, "edit-rule.example.test", 5433, true)] + [InlineData("edit-rule.example.test", 5433, "edit-rule.example.test", 5433, false)] + /* Only the incoming host is trimmed; the stored text is compared exactly, as the core's SameAddress does. */ + [InlineData("edit-rule.example.test ", 0, "edit-rule.example.test", 0, true)] + [InlineData("edit-rule.example.test ", 0, "edit-rule.example.test ", 0, true)] + public void TheViewersMoveRule_IsTheCoresHostAndPortRule(string storedHost, int storedPort, string newHost, int newPort, bool moved) + { + Assert.Equal(moved, ViewerMoves(storedHost, storedPort, newHost, newPort)); + } + + [Theory] + [InlineData("edit-rule.example.test", 0, "edit-rule.example.test", 0)] + [InlineData("edit-rule.example.test", 0, " edit-rule.example.test ", 0)] + [InlineData("edit-rule.example.test", 0, "EDIT-RULE.example.test", 0)] + [InlineData("edit-rule.example.test ", 0, "edit-rule.example.test", 0)] + [InlineData("edit-rule.example.test ", 0, "edit-rule.example.test ", 0)] + [InlineData("edit-rule.example.test", 0, "edit-rule.example.test", 5433)] + public void TheViewersMoveRule_AgreesWithTheCoresSameAddress_OnTheTrimmedIncomingHost(string storedHost, int storedPort, string newHost, int newPort) + { + /* The core trims a request's host when it reads it, then calls SameAddress with the stored text as it is. */ + Assert.Equal( + !Edit.SameAddress(newHost.Trim(' '), newPort, storedHost, storedPort), + ViewerMoves(storedHost, storedPort, newHost, newPort)); + } + + [Theory] + [InlineData("serviceprincipal", "sql")] + [InlineData("sql", "serviceprincipal")] + [InlineData("integrated", "sql")] + public void TheViewersSwitchSentences_AreTheCoresSentences(string storedAuth, string newAuth) + { + var row = new Edit.ServerEditRow( + ServerId: ServerId, Name: "edit-rule", Host: StoredHost, Port: 0, Database: null, ReadOnlyIntent: false, Engine: "sqlserver", + Auth: storedAuth, Username: "app-user", EncryptMode: "Mandatory", TrustServerCertificate: false, + MultiSubnetFailover: false, MonthlyCostUsd: 0m, ModifiedAt: new DateTime(2026, 10, 5, 12, 0, 0, DateTimeKind.Unspecified)); + var (changes, parseError) = Edit.ParseEditChanges("{\"auth\":\"" + (newAuth == "sql" ? "SQL" : "ServicePrincipal") + "\",\"username\":\"app-user\"}"); + Assert.Null(parseError); + + var (plan, error) = Edit.PlanEdit(row, changes!, TestKeyRings.Healthy.Status); + + Assert.Null(plan); + Assert.Equal( + newAuth == "sql" ? ViewerDataService.EditSwitchToSqlNeedsPasswordText : ViewerDataService.EditSwitchToServicePrincipalNeedsSecretText, + error); + } + + [Fact] + public void TheDialogDoesNotPrefillTheStoredSecret_AndAsksAgainWhenTheReachMoves() + { + var source = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "AddServerDialog.xaml.cs"); + Assert.DoesNotContain("TryUnprotect(existing.EncryptedPassword)", source, StringComparison.Ordinal); + Assert.DoesNotContain("PasswordBox.Password = decrypted", source, StringComparison.Ordinal); + Assert.DoesNotContain("AzureClientSecretBox.Password = decrypted", source, StringComparison.Ordinal); + + /* Both keep-the-stored-blob branches (SQL and service principal) ask first. */ + var guard = "if (ReachMovedFromForm("; + var first = source.IndexOf(guard, StringComparison.Ordinal); + Assert.True(first >= 0, "missing: " + guard); + Assert.True(source.IndexOf(guard, first + 1, StringComparison.Ordinal) > first, "the second keep-blob branch has no guard"); + Assert.Contains("catch (MonitoredServerPasswordNeededException ex)", source, StringComparison.Ordinal); + Assert.True(source.IndexOf("catch (MonitoredServerPasswordNeededException ex)", StringComparison.Ordinal) + < source.IndexOf("Error saving server:", StringComparison.Ordinal), "the refusal is caught before the general catch"); + } + + private sealed record Rig(ScratchPostgres Scratch, NpgsqlDataSource Owner) : IAsyncDisposable + { + public async ValueTask DisposeAsync() + { + await Owner.DisposeAsync(); + await Scratch.DisposeAsync(); + } + } + + private static async Task OpenAsync(string auth, string? blob, CancellationToken ct, string? remediationBlob = null) + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live viewer edit password tests (each mints its own scratch database)."); + + var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var ownerString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { SearchPath = PgSchemaGenerator.SearchPath }.ConnectionString; + await using (var connection = new NpgsqlConnection(ownerString)) + { + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, null, ct); + } + + var owner = NpgsqlDataSource.Create(ownerString); + await using (var command = owner.CreateCommand(@"INSERT INTO config_monitored_servers + (server_id, name, host, auth, username, encrypted_password, encrypt_mode, excluded_databases, is_enabled, monthly_cost_usd) + VALUES ($1, 'edit-rule', $2, $3, 'app-user', $4, 'Mandatory', ARRAY[]::text[], TRUE, 0)")) + { + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(StoredHost); + command.Parameters.AddWithValue(auth); + command.Parameters.AddWithValue(blob is null ? DBNull.Value : blob); + await command.ExecuteNonQueryAsync(ct); + } + + if (remediationBlob is not null) + { + await using var remediation = owner.CreateCommand( + "UPDATE config_monitored_servers SET remediation_username = 'fix-login', remediation_encrypted_password = $2 WHERE server_id = $1"); + remediation.Parameters.AddWithValue(ServerId); + remediation.Parameters.AddWithValue(remediationBlob); + await remediation.ExecuteNonQueryAsync(ct); + } + + return new Rig(scratch, owner); + } + + private static MonitoredServerRow Row(string auth, string host, string? blob, string encryptMode = "Mandatory", int port = 0) => new() + { + ServerId = ServerId, + Name = "edit-rule", + Host = host, + Port = port, + Auth = auth, + Username = "app-user", + EncryptedPassword = blob, + EncryptMode = encryptMode, + }; + + private static async Task<(string Host, string? Blob, string EncryptMode)> StoredAsync(Rig rig, CancellationToken ct) + { + await using var command = rig.Owner.CreateCommand( + "SELECT host, encrypted_password, encrypt_mode FROM config_monitored_servers WHERE server_id = $1"); + command.Parameters.AddWithValue(ServerId); + await using var reader = await command.ExecuteReaderAsync(ct); + Assert.True(await reader.ReadAsync(ct)); + return (reader.GetString(0), reader.IsDBNull(1) ? null : reader.GetString(1), reader.GetString(2)); + } + + [Theory] + [InlineData("sql")] + [InlineData("serviceprincipal")] + public async Task AHostChange_ThatKeepsTheStoredBlob_IsRefused_WithTheWebsSentence_AndNothingIsWritten(string auth) + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(auth, StoredBlob, ct); + await using var viewer = new ViewerDataService(rig.Scratch.ConnectionString); + + var refused = await Assert.ThrowsAsync( + () => viewer.UpsertMonitoredServerAsync(Row(auth, MovedHost, StoredBlob), ct)); + Assert.Equal(Edit.EditPasswordNeededText, refused.Message); + + Assert.Equal((StoredHost, StoredBlob, "Mandatory"), await StoredAsync(rig, ct)); + } + + [Fact] + public async Task AHostChange_WithNoBlobAtAll_AndAPortChange_AreRefusedToo() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync("sql", StoredBlob, ct); + await using var viewer = new ViewerDataService(rig.Scratch.ConnectionString); + + await Assert.ThrowsAsync( + () => viewer.UpsertMonitoredServerAsync(Row("sql", MovedHost, null), ct)); + await Assert.ThrowsAsync( + () => viewer.UpsertMonitoredServerAsync(Row("sql", StoredHost, StoredBlob, port: 5433), ct)); + + Assert.Equal((StoredHost, StoredBlob, "Mandatory"), await StoredAsync(rig, ct)); + } + + [Fact] + public async Task AHostChange_WithANewlyEnteredPassword_Saves() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync("sql", StoredBlob, ct); + await using var viewer = new ViewerDataService(rig.Scratch.ConnectionString); + + await viewer.UpsertMonitoredServerAsync(Row("sql", MovedHost, NewBlob), ct); + + Assert.Equal((MovedHost, NewBlob, "Mandatory"), await StoredAsync(rig, ct)); + } + + [Fact] + public async Task ANonReachChange_KeepsTheStoredPassword() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync("sql", StoredBlob, ct); + await using var viewer = new ViewerDataService(rig.Scratch.ConnectionString); + + var renamed = Row("sql", StoredHost, StoredBlob); + renamed.Name = "edit-rule-renamed"; + renamed.MonthlyCostUsd = 5m; + await viewer.UpsertMonitoredServerAsync(renamed, ct); + + Assert.Equal((StoredHost, StoredBlob, "Mandatory"), await StoredAsync(rig, ct)); + await using var read = rig.Owner.CreateCommand("SELECT name FROM config_monitored_servers WHERE server_id = $1"); + read.Parameters.AddWithValue(ServerId); + Assert.Equal("edit-rule-renamed", (string?)await read.ExecuteScalarAsync(ct)); + } + + [Fact] + public async Task AWindowsAuthHostChange_Saves() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync("integrated", null, ct); + await using var viewer = new ViewerDataService(rig.Scratch.ConnectionString); + + await viewer.UpsertMonitoredServerAsync(Row("integrated", MovedHost, null), ct); + + Assert.Equal((MovedHost, null, "Mandatory"), await StoredAsync(rig, ct)); + } + + [Theory] + [InlineData("sql", "serviceprincipal", "Switching to ServicePrincipal authentication needs the client secret as password.")] + [InlineData("serviceprincipal", "sql", "Switching to SQL authentication needs the password.")] + [InlineData("integrated", "sql", "Switching to SQL authentication needs the password.")] + public async Task AnAuthSwitch_ThatKeepsTheStoredBlob_OrCarriesNone_IsRefused_WithTheCoresSentence_AndNothingIsWritten( + string storedAuth, string newAuth, string sentence) + { + var ct = TestContext.Current.CancellationToken; + var storedBlob = storedAuth == "integrated" ? null : StoredBlob; + await using var rig = await OpenAsync(storedAuth, storedBlob, ct); + await using var viewer = new ViewerDataService(rig.Scratch.ConnectionString); + + if (storedBlob is not null) + { + var kept = await Assert.ThrowsAsync( + () => viewer.UpsertMonitoredServerAsync(Row(newAuth, StoredHost, StoredBlob), ct)); + Assert.Equal(sentence, kept.Message); + } + + var none = await Assert.ThrowsAsync( + () => viewer.UpsertMonitoredServerAsync(Row(newAuth, StoredHost, null), ct)); + Assert.Equal(sentence, none.Message); + + Assert.Equal((StoredHost, storedBlob, "Mandatory"), await StoredAsync(rig, ct)); + } + + [Theory] + [InlineData("sql", "serviceprincipal")] + [InlineData("serviceprincipal", "sql")] + [InlineData("integrated", "sql")] + public async Task AnAuthSwitch_WithANewlyEnteredSecret_Saves(string storedAuth, string newAuth) + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(storedAuth, storedAuth == "integrated" ? null : StoredBlob, ct); + await using var viewer = new ViewerDataService(rig.Scratch.ConnectionString); + + await viewer.UpsertMonitoredServerAsync(Row(newAuth, StoredHost, NewBlob), ct); + + Assert.Equal((StoredHost, NewBlob, "Mandatory"), await StoredAsync(rig, ct)); + } + + [Fact] + public async Task ASwitchOutOfASecretMode_NeedsNoSecret_AndSaves() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync("sql", StoredBlob, ct); + await using var viewer = new ViewerDataService(rig.Scratch.ConnectionString); + + await viewer.UpsertMonitoredServerAsync(Row("integrated", StoredHost, null), ct); + + Assert.Equal((StoredHost, null, "Mandatory"), await StoredAsync(rig, ct)); + } + + public static IEnumerable ConnectionFields() => + new[] { "encrypt mode", "trust certificate", "multi-subnet", "username", "database", "read-only", "engine", "port", "host" } + .Select(f => new object[] { f }); + + private static MonitoredServerRow Changed(string field, string auth, string? blob) + { + var row = Row(auth, StoredHost, blob); + switch (field) + { + case "encrypt mode": row.EncryptMode = "Optional"; break; + case "trust certificate": row.TrustServerCertificate = true; break; + case "multi-subnet": row.MultiSubnetFailover = true; break; + case "username": row.Username = "other-user"; break; + case "database": row.Database = "tempdb"; break; + case "read-only": row.ReadOnlyIntent = true; break; + case "engine": row.Engine = "postgres"; break; + case "port": row.Port = 5433; break; + case "host": row.Host = MovedHost; break; + default: throw new ArgumentException(field); + } + + return row; + } + + /// The sentence the web and MCP edit give for a row with a remediation login. Held here as text because the + /// two apps carry it as separate constants. + private const string RemediationSentence = + "This server has a remediation login stored. Change how it is reached on the service host, in the configuration file or with --add-server."; + + [Fact] + public void TheViewersRemediationSentence_IsTheSharedSentence() + { + Assert.Equal(RemediationSentence, ViewerDataService.RemediationKeptText); + Assert.Equal("PW003", ViewerDataService.StoreRemediationKeptSqlState); + } + + [Theory] + [MemberData(nameof(ConnectionFields))] + public async Task AnyConnectionChange_ThatKeepsTheStoredBlob_IsRefusedBeforeTheWrite(string field) + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync("sql", StoredBlob, ct); + await using var viewer = new ViewerDataService(rig.Scratch.ConnectionString); + + var refused = await Assert.ThrowsAsync( + () => viewer.UpsertMonitoredServerAsync(Changed(field, "sql", StoredBlob), ct)); + Assert.Equal(Edit.EditPasswordNeededText, refused.Message); + var blank = await Assert.ThrowsAsync( + () => viewer.UpsertMonitoredServerAsync(Changed(field, "sql", null), ct)); + Assert.Equal(Edit.EditPasswordNeededText, blank.Message); + + Assert.Equal((StoredHost, StoredBlob, "Mandatory"), await StoredAsync(rig, ct)); + } + + [Theory] + [MemberData(nameof(ConnectionFields))] + public async Task AnyConnectionChange_WithANewlyEnteredPassword_Saves(string field) + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync("sql", StoredBlob, ct); + await using var viewer = new ViewerDataService(rig.Scratch.ConnectionString); + + await viewer.UpsertMonitoredServerAsync(Changed(field, "sql", NewBlob), ct); + + var stored = await StoredAsync(rig, ct); + Assert.Equal(NewBlob, stored.Blob); + } + + [Theory] + [InlineData("sql", "stored-blob")] + [InlineData("integrated", null)] + public async Task AConnectionChange_OnARowWithARemediationSecret_IsRefusedBeforeTheWrite_WithTheSharedSentence(string auth, string? blob) + { + var ct = TestContext.Current.CancellationToken; + foreach (var field in ConnectionFields().Select(f => (string)f[0])) + { + await using var rig = await OpenAsync(auth, blob, ct, remediationBlob: "remediation-blob"); + await using var viewer = new ViewerDataService(rig.Scratch.ConnectionString); + + /* A password typed with the change does not help: the refusal does not depend on it. */ + var typed = await Assert.ThrowsAsync( + () => viewer.UpsertMonitoredServerAsync(Changed(field, auth, auth == "sql" ? NewBlob : null), ct)); + Assert.Equal(RemediationSentence, typed.Message); + + Assert.Equal((StoredHost, blob, "Mandatory"), await StoredAsync(rig, ct)); + } + } + + [Fact] + public async Task ANonConnectionChange_OnARowWithARemediationSecret_Saves() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync("sql", StoredBlob, ct, remediationBlob: "remediation-blob"); + await using var viewer = new ViewerDataService(rig.Scratch.ConnectionString); + + var renamed = Row("sql", StoredHost, StoredBlob); + renamed.MonthlyCostUsd = 7m; + await viewer.UpsertMonitoredServerAsync(renamed, ct); + + Assert.Equal((StoredHost, StoredBlob, "Mandatory"), await StoredAsync(rig, ct)); + } +} diff --git a/Darling/Darling.Tests/ServerEditRouteTests.cs b/Darling/Darling.Tests/ServerEditRouteTests.cs new file mode 100644 index 0000000000..01fb6d3893 --- /dev/null +++ b/Darling/Darling.Tests/ServerEditRouteTests.cs @@ -0,0 +1,822 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Net; +using System.Net.Http; +using System.Text; +using System.Text.Json.Nodes; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.TestHost; +using Microsoft.Extensions.Logging; +using Npgsql; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Hosting; +using PerformanceMonitor.Darling.Service.Mcp; +using Xunit; + +namespace Darling.Tests; + +/// +/// PATCH /api/servers/{id} and GET /api/admin/servers/{id} (#5240) through a real host over a store that +/// is never opened: the gates, limits, status mapping, audit line, credential handling and the slot SHARED with +/// POST /api/servers, with the edit and add cores stood in by stubs. The edit core itself is covered by +/// ServerEditCoreTests / ServerEditLiveTests, and the grant it writes under by +/// ServerEditViewerRoleLiveTests. +/// +public sealed class ServerEditRouteTests +{ + private const string FakeSecret = "Zq9-fake-secret-must-never-appear-7XK"; + private const string Token = "2026-10-05T12:00:00.1234567"; + + private const string UpdatedAnswer = + "{\"status\":\"updated\",\"server\":\"sql01\",\"display_name\":\"Orders\",\"server_id\":41,\"changed\":[\"display_name\",\"monthly_cost_usd\"],\"reconnects\":true,\"tested\":false,\"modified_at\":\"2026-10-05T12:00:01.0000001\",\"note\":\"n\"}"; + + private const string AddedAnswer = + "{\"requested\":1,\"added\":1,\"skipped\":0,\"collided\":0,\"failed\":0,\"results\":[{\"server\":\"sql01\",\"status\":\"added\",\"detail\":\"Connected\"}]}"; + + private static string Changes(string extra = "\"display_name\":\"Orders\"") => + "{" + extra + ",\"expected_modified_at\":\"" + Token + "\"}"; + + private sealed class Rig : IAsyncDisposable + { + public required WebApplication App { get; init; } + public required HttpClient Client { get; init; } + public required CapturingTestLogger Log { get; init; } + public required NpgsqlDataSource Source { get; init; } + public required List<(int Id, string Body)> Edits { get; init; } + + public async ValueTask DisposeAsync() + { + Client.Dispose(); + await App.DisposeAsync(); + await Source.DisposeAsync(); + } + } + + private static async Task StartAsync( + Func>? edit = null, Func>? add = null, bool useWriteGate = false, + TimeSpan? editTimeout = null, TimeSpan? addTimeout = null, + Func>? read = null) + { + var builder = WebApplication.CreateBuilder(); + builder.WebHost.UseTestServer(); + builder.Logging.ClearProviders(); + var app = builder.Build(); + var log = new CapturingTestLogger(); + var source = NpgsqlDataSource.Create("Host=localhost;Database=never_opened;Username=nobody"); + var edits = new List<(int, string)>(); + + app.Use(async (context, next) => + { + var seat = context.Request.Headers["X-Seat"] == "viewer" + ? new DarlingWebSeat("bob", false) + : new DarlingWebSeat(Uri.UnescapeDataString(context.Request.Headers["X-Principal"].FirstOrDefault() ?? "alice"), true); + context.Items[DarlingWebSeat.HttpContextItemKey] = seat; + if (useWriteGate && !DarlingWebSeat.IsRequestAllowed(seat, context.Request.Method, context.Request.Path.Value ?? "/")) + { + context.Response.StatusCode = StatusCodes.Status403Forbidden; + return; + } + + await next(context); + }); + DarlingWebEndpoints.MapServers( + app, source, log, + add ?? (_ => Task.FromResult(AddedAnswer)), addTimeout, + (id, body) => + { + lock (edits) + { + edits.Add((id, body)); + } + + return (edit ?? ((_, _) => Task.FromResult(UpdatedAnswer)))(id, body); + }, + editTimeout, read); + await app.StartAsync(TestContext.Current.CancellationToken); + return new Rig { App = app, Client = app.GetTestClient(), Log = log, Source = source, Edits = edits }; + } + + private static async Task<(HttpStatusCode Status, string Body)> SendAsync( + Rig rig, HttpMethod method, string path, string? body = null, string mediaType = "application/json", string? seat = null, string? principal = null) + { + var ct = TestContext.Current.CancellationToken; + using var request = new HttpRequestMessage(method, path); + if (body is not null) + { + request.Content = new StringContent(body, Encoding.UTF8, mediaType); + } + + if (seat is not null) + { + request.Headers.Add("X-Seat", seat); + } + + if (principal is not null) + { + request.Headers.Add("X-Principal", Uri.EscapeDataString(principal)); + } + + using var response = await rig.Client.SendAsync(request, ct); + return (response.StatusCode, await response.Content.ReadAsStringAsync(ct)); + } + + private static Task<(HttpStatusCode Status, string Body)> PatchAsync( + Rig rig, string body, int id = 41, string mediaType = "application/json", string? seat = null, string? principal = null) => + SendAsync(rig, HttpMethod.Patch, "/api/servers/" + id, body, mediaType, seat, principal); + + /// Waits for a captured log line that contains . A late audit line is written by + /// a thread-pool continuation after the 503 was answered, so a test waits for it and never asserts the instant it + /// releases the core. + private static async Task WaitForLogLineAsync(Rig rig, string fragment) + { + var deadline = DateTime.UtcNow.AddSeconds(10); + while (!rig.Log.Lines.Any(l => l.Contains(fragment, StringComparison.Ordinal))) + { + Assert.True(DateTime.UtcNow < deadline, $"no log line contained \"{fragment}\"; the log held: {rig.Log.Joined}"); + await Task.Delay(20, TestContext.Current.CancellationToken); + } + } + + private static Task<(HttpStatusCode Status, string Body)> PostAddAsync(Rig rig) => + SendAsync(rig, HttpMethod.Post, "/api/servers", "[{\"host\":\"sql01\"}]"); + + private static void AssertNoSecret(Rig rig, string responseBody) + { + Assert.DoesNotContain(FakeSecret, responseBody, StringComparison.Ordinal); + Assert.DoesNotContain(FakeSecret, rig.Log.Joined, StringComparison.Ordinal); + } + + /* ═══════════════════════ status mapping ═══════════════════════ */ + + [Theory] + [InlineData("{\"status\":\"updated\"}", 200)] + [InlineData("{\"status\":\"unchanged\"}", 200)] + [InlineData("{\"status\":\"invalid\",\"message\":\"m\"}", 400)] + [InlineData("{\"status\":\"not_found\",\"message\":\"m\"}", 404)] + [InlineData("{\"status\":\"conflict\",\"message\":\"m\",\"current\":{}}", 409)] + [InlineData("{\"status\":\"collides\",\"message\":\"m\"}", 409)] + [InlineData("{\"status\":\"connection_failed\",\"message\":\"m\"}", 200)] + [InlineData("{\"status\":\"error\",\"message\":\"boom\"}", 500)] + [InlineData("{\"status\":\"somethingNew\"}", 500)] + [InlineData("not json at all", 400)] + public void ServerEditEnvelopeStatus_MapsTheEditVocabularyOntoHttp(string envelope, int expected) => + Assert.Equal(expected, DarlingWebEndpoints.ServerEditEnvelopeStatus(envelope)); + + [Fact] + public async Task EveryCoreStatus_ReachesTheWireUnderItsMappedCode_WithTheEnvelopeAsTheBody() + { + var cases = new (string Envelope, HttpStatusCode Expected)[] + { + (UpdatedAnswer, HttpStatusCode.OK), + ("{\"status\":\"unchanged\",\"message\":\"m\"}", HttpStatusCode.OK), + ("{\"status\":\"invalid\",\"message\":\"m\"}", HttpStatusCode.BadRequest), + ("{\"status\":\"not_found\",\"message\":\"m\"}", HttpStatusCode.NotFound), + ("{\"status\":\"conflict\",\"message\":\"m\",\"current\":{\"modified_at\":\"t\"}}", HttpStatusCode.Conflict), + ("{\"status\":\"collides\",\"message\":\"m\"}", HttpStatusCode.Conflict), + ("{\"status\":\"connection_failed\",\"message\":\"Could not connect. Nothing was saved.\"}", HttpStatusCode.OK), + }; + foreach (var (envelope, expected) in cases) + { + await using var rig = await StartAsync(edit: (_, _) => Task.FromResult(envelope)); + var (status, body) = await PatchAsync(rig, Changes()); + Assert.Equal(expected, status); + Assert.Equal(JsonNode.Parse(envelope)!["status"]!.GetValue(), JsonNode.Parse(body)!["status"]!.GetValue()); + } + } + + [Fact] + public async Task AConflict_CarriesTheCurrentValues_ToTheCaller() + { + const string conflict = "{\"status\":\"conflict\",\"message\":\"m\",\"current\":{\"display_name\":\"Orders\",\"modified_at\":\"t2\"}}"; + await using var rig = await StartAsync(edit: (_, _) => Task.FromResult(conflict)); + var (status, body) = await PatchAsync(rig, Changes()); + Assert.Equal(HttpStatusCode.Conflict, status); + Assert.Equal("t2", JsonNode.Parse(body)!["current"]!["modified_at"]!.GetValue()); + } + + [Fact] + public async Task TheCoresCaughtExceptionEnvelope_NeverReachesTheWire_NorTheLog() + { + var error = "{\"status\":\"error\",\"message\":\"Error during edit_server: 28P01 password authentication failed for user 'x' " + FakeSecret + "\"}"; + await using var rig = await StartAsync(edit: (_, _) => Task.FromResult(error)); + var (status, body) = await PatchAsync(rig, Changes("\"password\":\"" + FakeSecret + "\"")); + Assert.True((int)status >= 500); + AssertNoSecret(rig, body); + Assert.DoesNotContain("28P01", body, StringComparison.Ordinal); + } + + /* ═══════════════════════ gates ═══════════════════════ */ + + [Fact] + public async Task AReadOnlySeat_Answers403_AndReachesNoCore_WithOrWithoutTheHostWriteGate() + { + foreach (var gate in new[] { false, true }) + { + await using var rig = await StartAsync(useWriteGate: gate); + var (status, body) = await PatchAsync(rig, Changes("\"password\":\"" + FakeSecret + "\""), seat: "viewer"); + Assert.Equal(HttpStatusCode.Forbidden, status); + Assert.Empty(rig.Edits); + AssertNoSecret(rig, body); + } + } + + [Theory] + [InlineData("text/plain")] + [InlineData("application/x-www-form-urlencoded")] + [InlineData("multipart/form-data")] + public async Task ANonJsonContentType_Answers415_AndReachesNoCore(string mediaType) + { + await using var rig = await StartAsync(); + var (status, _) = await PatchAsync(rig, Changes(), mediaType: mediaType); + Assert.Equal(HttpStatusCode.UnsupportedMediaType, status); + Assert.Empty(rig.Edits); + } + + [Fact] + public async Task OnlyPatchIsRoutedOnTheEditPath_EveryOtherUnsafeMethodIsRefused405() + { + await using var rig = await StartAsync(); + foreach (var method in new[] { HttpMethod.Post, HttpMethod.Put, HttpMethod.Delete }) + { + var (status, _) = await SendAsync(rig, method, "/api/servers/41", Changes()); + Assert.Equal(HttpStatusCode.MethodNotAllowed, status); + } + + Assert.Empty(rig.Edits); + } + + [Fact] + public async Task ANonNumericId_IsNotRouted() + { + await using var rig = await StartAsync(); + var (status, _) = await SendAsync(rig, HttpMethod.Patch, "/api/servers/abc", Changes()); + Assert.Equal(HttpStatusCode.NotFound, status); + Assert.Empty(rig.Edits); + } + + [Theory] + [InlineData("PATCH", "/api/servers/41", false)] + [InlineData("PATCH", "/api/servers/41", true)] + [InlineData("GET", "/api/admin/servers/41", false)] + [InlineData("GET", "/api/admin/servers/41", true)] + public void TheHostWriteGate_RefusesAReadOnlySeatOnlyTheUnsafeEditMethod(string method, string path, bool canEdit) => + Assert.Equal(method == "GET" || canEdit, DarlingWebSeat.IsRequestAllowed(new DarlingWebSeat("who", canEdit), method, path)); + + /* ═══════════════════════ body ═══════════════════════ */ + + [Fact] + public async Task TheBodyIsPassedToTheCoreUnchanged_WithTheRouteId() + { + await using var rig = await StartAsync(); + var body = Changes("\"monthly_cost_usd\":12.5"); + var (status, answer) = await PatchAsync(rig, body, id: 77); + Assert.Equal(HttpStatusCode.OK, status); + Assert.Equal((77, body), Assert.Single(rig.Edits)); + Assert.Equal("updated", JsonNode.Parse(answer)!["status"]!.GetValue()); + } + + [Fact] + public async Task AnOversizedBody_Answers400_AtTheCap_WithoutTheBody() + { + await using var rig = await StartAsync(); + var padding = new string('x', DarlingWebEndpoints.MaxServerEditBodyBytes); + var (status, body) = await PatchAsync(rig, Changes("\"display_name\":\"" + padding + "\"")); + Assert.Equal(HttpStatusCode.BadRequest, status); + Assert.Empty(rig.Edits); + Assert.DoesNotContain("xxxxxxxx", body, StringComparison.Ordinal); + + /* A body just under the cap still reaches the core. */ + var fits = Changes("\"display_name\":\"" + new string('x', DarlingWebEndpoints.MaxServerEditBodyBytes - 200) + "\""); + Assert.True(Encoding.UTF8.GetByteCount(fits) < DarlingWebEndpoints.MaxServerEditBodyBytes); + Assert.Equal(HttpStatusCode.OK, (await PatchAsync(rig, fits)).Status); + } + + [Theory] + [InlineData("")] + [InlineData("not json")] + [InlineData("[]")] + [InlineData("\"text\"")] + [InlineData("null")] + [InlineData("{\"display_name\":\"a\",\"display_name\":\"b\",\"expected_modified_at\":\"t\"}")] + [InlineData("{\"display_name\":\"a\"}")] + [InlineData("{\"display_name\":\"a\",\"expected_modified_at\":\"\"}")] + [InlineData("{\"display_name\":\"a\",\"expected_modified_at\":123}")] + [InlineData("{\"display_name\":\"a\",\"expected_modified_at\":null}")] + public async Task EveryMalformedBody_OrAMissingToken_Answers400_BeforeTheCore_AndEchoesNothing(string malformed) + { + await using var rig = await StartAsync(); + var (status, body) = await PatchAsync(rig, malformed); + Assert.Equal(HttpStatusCode.BadRequest, status); + Assert.Empty(rig.Edits); + Assert.DoesNotContain("display_name", body, StringComparison.Ordinal); + } + + [Fact] + public void TheMissingTokenRefusal_NamesTheToken_AndTheDuplicateRefusal_NamesNoValue() + { + Assert.False(DarlingWebEndpoints.TryReadServerEditBody("{\"host\":\"h\"}", out _, out var missing)); + Assert.Contains("expected_modified_at", missing, StringComparison.Ordinal); + Assert.False(DarlingWebEndpoints.TryReadServerEditBody("{\"host\":\"" + FakeSecret + "\",\"host\":\"b\"}", out _, out var duplicate)); + Assert.DoesNotContain(FakeSecret, duplicate, StringComparison.Ordinal); + Assert.True(DarlingWebEndpoints.TryReadServerEditBody(Changes(), out var parsed, out var none)); + Assert.Null(none); + Assert.Equal(Token, parsed["expected_modified_at"]!.GetValue()); + } + + /* ═══════════════════════ credentials and audit ═══════════════════════ */ + + [Fact] + public async Task ASecretThatSomehowReachesTheCoresAnswer_IsRedactedBeforeItIsWritten() + { + var leaky = "{\"status\":\"connection_failed\",\"message\":\"Could not connect: login failed for " + FakeSecret + ". Nothing was saved.\"}"; + await using var rig = await StartAsync(edit: (_, _) => Task.FromResult(leaky)); + var (status, body) = await PatchAsync(rig, Changes("\"password\":\"" + FakeSecret + "\"")); + Assert.Equal(HttpStatusCode.OK, status); + Assert.Contains("[redacted]", body, StringComparison.OrdinalIgnoreCase); + AssertNoSecret(rig, body); + } + + [Fact] + public void RedactEditAnswer_RemovesTheSecretInItsRawAndJsonEscapedSpelling() + { + const string secret = "p\"w\\d"; + /* Serialized the way the core writes it: on the wire the secret's quote and backslash are escaped, so the text + holds only the escaped spelling and the redaction has to work on the parsed value. */ + var escaped = System.Text.Json.JsonSerializer.Serialize(secret)[1..^1]; + var answer = new JsonObject { ["status"] = "connection_failed", ["message"] = "bad " + secret + " and again " + secret }.ToJsonString(); + Assert.Contains(escaped, answer, StringComparison.Ordinal); + Assert.DoesNotContain(secret, answer, StringComparison.Ordinal); + + var redacted = DarlingWebEndpoints.RedactEditAnswer(answer, new[] { secret }); + Assert.DoesNotContain(secret, redacted, StringComparison.Ordinal); + Assert.DoesNotContain(escaped, redacted, StringComparison.Ordinal); + var envelope = JsonNode.Parse(redacted)!.AsObject(); + Assert.Equal("bad [redacted] and again [redacted]", envelope["message"]!.GetValue()); + Assert.Equal("connection_failed", envelope["status"]!.GetValue()); + Assert.Equal(answer, DarlingWebEndpoints.RedactEditAnswer(answer, Array.Empty())); + } + + /// The whole-text replace this pins against turned a secret that spelled a key, a number or an engine word + /// into a hole in the answer's structure, so a committed edit read as a failure and its audit line was dropped. + [Theory] + [InlineData("41")] + [InlineData("true")] + [InlineData("false")] + [InlineData("postgres")] + [InlineData("status")] + [InlineData("updated")] + [InlineData("sql01")] + [InlineData("current")] + public void RedactEditAnswer_NeverRewritesAStructuredField_WhateverTheSecretSpells(string secret) + { + var stored = "{\"status\":\"conflict\",\"message\":\"The server changed since it was read.\",\"current\":{\"engine\":\"postgres\"," + + "\"username\":\"postgres\",\"host\":\"sql01\",\"display_name\":\"Orders\",\"server_id\":41,\"trust_server_certificate\":true,\"read_only_intent\":false}}"; + foreach (var answer in new[] { UpdatedAnswer, stored }) + { + var redacted = DarlingWebEndpoints.RedactEditAnswer(answer, new[] { secret }); + Assert.True( + JsonNode.DeepEquals(JsonNode.Parse(answer), JsonNode.Parse(redacted)), + $"a secret spelled \"{secret}\" rewrote a structured field: {redacted}"); + } + } + + [Fact] + public void RedactEditAnswer_ReplacesAnAnswerThatDoesNotParse_WithAFixedBody_AndRedactsAnythingThatIsNotAnObject() + { + var broken = DarlingWebEndpoints.RedactEditAnswer("{\"status\":\"updated\" " + FakeSecret + " ", new[] { FakeSecret }); + Assert.DoesNotContain(FakeSecret, broken, StringComparison.Ordinal); + Assert.Equal("error", JsonNode.Parse(broken)!["status"]!.GetValue()); + + Assert.Equal("[\"x [redacted] y\"]", DarlingWebEndpoints.RedactEditAnswer("[\"x " + FakeSecret + " y\"]", new[] { FakeSecret })); + } + + [Theory] + [InlineData("41")] + [InlineData("true")] + [InlineData("updated")] + [InlineData("sql01")] + public async Task ASecretThatSpellsAStructuredValue_LeavesACommittedEditsAnswerIntact_AndItsAuditLineWritten(string secret) + { + await using var rig = await StartAsync(); + var (status, body) = await PatchAsync(rig, Changes("\"display_name\":\"Orders\",\"password\":\"" + secret + "\"")); + Assert.Equal(HttpStatusCode.OK, status); + Assert.True(JsonNode.DeepEquals(JsonNode.Parse(UpdatedAnswer), JsonNode.Parse(body)), $"the answer was rewritten: {body}"); + Assert.Equal(41, JsonNode.Parse(body)!["server_id"]!.GetValue()); + var line = Assert.Single(rig.Log.Lines, l => l.StartsWith("Information: Server edited by", StringComparison.Ordinal)); + Assert.Contains("Server edited by alice: id 41, fields display_name,monthly_cost_usd", line, StringComparison.Ordinal); + } + + [Fact] + public async Task AConflict_ForAPostgresTargetWhoseSecretIsTheEngineWord_StillCarriesTheStoredValues() + { + var conflict = "{\"status\":\"conflict\",\"message\":\"The server changed since it was read.\",\"current\":{\"engine\":\"postgres\"," + + "\"username\":\"postgres\",\"host\":\"db01\",\"display_name\":\"postgres\",\"modified_at\":\"" + Token + "\"}}"; + await using var rig = await StartAsync(edit: (_, _) => Task.FromResult(conflict)); + var (status, body) = await PatchAsync(rig, Changes("\"host\":\"db02\",\"password\":\"postgres\"")); + Assert.Equal(HttpStatusCode.Conflict, status); + var current = JsonNode.Parse(body)!["current"]!.AsObject(); + Assert.Equal("postgres", current["engine"]!.GetValue()); + Assert.Equal("postgres", current["username"]!.GetValue()); + Assert.Equal("db01", current["host"]!.GetValue()); + } + + [Fact] + public async Task ACoreThatThrows_AnswersTheFixedErrorBody_AndFreesTheSlot_WithoutTheSecret() + { + var first = true; + await using var rig = await StartAsync(edit: (_, _) => + { + if (first) + { + first = false; + throw new InvalidOperationException("driver said " + FakeSecret); + } + + return Task.FromResult(UpdatedAnswer); + }); + var (status, body) = await PatchAsync(rig, Changes("\"password\":\"" + FakeSecret + "\"")); + Assert.Equal(HttpStatusCode.InternalServerError, status); + AssertNoSecret(rig, body); + Assert.Equal(HttpStatusCode.OK, (await PatchAsync(rig, Changes())).Status); + } + + [Fact] + public async Task AnUpdatedEdit_IsLoggedOnce_WithThePrincipalTheIdAndTheFieldNames_AndNeverAValueOrTheSecret() + { + await using var rig = await StartAsync(); + var (status, body) = await PatchAsync(rig, Changes("\"display_name\":\"Orders-Secret-Name\",\"monthly_cost_usd\":99,\"password\":\"" + FakeSecret + "\"")); + Assert.Equal(HttpStatusCode.OK, status); + var edited = rig.Log.Lines.Where(l => l.StartsWith("Information: Server edited by", StringComparison.Ordinal)).ToList(); + var line = Assert.Single(edited); + Assert.Contains("Server edited by alice: id 41, fields display_name,monthly_cost_usd", line, StringComparison.Ordinal); + Assert.DoesNotContain("Orders-Secret-Name", rig.Log.Joined, StringComparison.Ordinal); + Assert.DoesNotContain("99", line, StringComparison.Ordinal); + AssertNoSecret(rig, body); + } + + [Theory] + [InlineData("{\"status\":\"unchanged\"}")] + [InlineData("{\"status\":\"conflict\",\"message\":\"m\",\"current\":{}}")] + [InlineData("{\"status\":\"invalid\",\"message\":\"m\"}")] + public async Task ANothingSavedAnswer_LogsNoEditedLine(string envelope) + { + await using var rig = await StartAsync(edit: (_, _) => Task.FromResult(envelope)); + await PatchAsync(rig, Changes()); + Assert.DoesNotContain(rig.Log.Lines, l => l.Contains("Server edited by", StringComparison.Ordinal)); + } + + [Fact] + public async Task AFailedProbe_IsLoggedOnce_WithTheCoresRedactedDetail() + { + var failed = "{\"status\":\"connection_failed\",\"message\":\"Could not connect to alpha-01.example.test. The service log has the details. Nothing was saved.\"}"; + await using var rig = await StartAsync(edit: (_, _) => Task.FromResult(failed)); + await PatchAsync(rig, Changes("\"password\":\"" + FakeSecret + "\"")); + var line = Assert.Single(rig.Log.Lines, l => l.Contains("Server edit failed for alice: id 41", StringComparison.Ordinal)); + Assert.Contains("Could not connect to alpha-01.example.test", line, StringComparison.Ordinal); + Assert.DoesNotContain(FakeSecret, rig.Log.Joined, StringComparison.Ordinal); + } + + [Fact] + public async Task APrincipalWithControlCharacters_IsLoggedOnOneLine() + { + await using var rig = await StartAsync(); + using var request = new HttpRequestMessage(HttpMethod.Patch, "/api/servers/41") + { + Content = new StringContent(Changes(), Encoding.UTF8, "application/json"), + }; + request.Headers.Add("X-Principal", Uri.EscapeDataString("eve\r\nForged line")); + using var response = await rig.Client.SendAsync(request, TestContext.Current.CancellationToken); + Assert.Equal(HttpStatusCode.OK, response.StatusCode); + var line = Assert.Single(rig.Log.Lines, l => l.Contains("Server edited by", StringComparison.Ordinal)); + Assert.DoesNotContain('\n', line); + Assert.DoesNotContain('\r', line); + } + + /* ═══════════════════════ the shared slot ═══════════════════════ */ + + [Fact] + public async Task AnEdit_WhileAnAddHoldsTheSlot_Answers429_AndTheAddStillCompletes() + { + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await using var rig = await StartAsync(add: _ => + { + entered.TrySetResult(); + return release.Task; + }); + + var add = PostAddAsync(rig); + await entered.Task.WaitAsync(TimeSpan.FromSeconds(30), TestContext.Current.CancellationToken); + var (status, body) = await PatchAsync(rig, Changes()); + Assert.Equal(HttpStatusCode.TooManyRequests, status); + Assert.Contains("already running", body, StringComparison.Ordinal); + Assert.Empty(rig.Edits); + + release.SetResult(AddedAnswer); + Assert.Equal(HttpStatusCode.OK, (await add).Status); + Assert.Equal(HttpStatusCode.OK, (await PatchAsync(rig, Changes())).Status); + } + + [Fact] + public async Task AnAdd_WhileAnEditHoldsTheSlot_Answers429_AndTheEditStillCompletes() + { + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var addCalls = 0; + await using var rig = await StartAsync( + edit: (_, _) => + { + entered.TrySetResult(); + return release.Task; + }, + add: _ => + { + Interlocked.Increment(ref addCalls); + return Task.FromResult(AddedAnswer); + }); + + var edit = PatchAsync(rig, Changes()); + await entered.Task.WaitAsync(TimeSpan.FromSeconds(30), TestContext.Current.CancellationToken); + Assert.Equal(HttpStatusCode.TooManyRequests, (await PostAddAsync(rig)).Status); + Assert.Equal(0, Volatile.Read(ref addCalls)); + + release.SetResult(UpdatedAnswer); + Assert.Equal(HttpStatusCode.OK, (await edit).Status); + Assert.Equal(HttpStatusCode.OK, (await PostAddAsync(rig)).Status); + } + + [Fact] + public async Task ASecondEdit_WhileOneIsRunning_Answers429() + { + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var calls = 0; + await using var rig = await StartAsync(edit: (_, _) => + { + if (Interlocked.Increment(ref calls) == 1) + { + entered.TrySetResult(); + return release.Task; + } + + return Task.FromResult(UpdatedAnswer); + }); + var first = PatchAsync(rig, Changes()); + await entered.Task.WaitAsync(TimeSpan.FromSeconds(30), TestContext.Current.CancellationToken); + Assert.Equal(HttpStatusCode.TooManyRequests, (await PatchAsync(rig, Changes(), id: 42)).Status); + release.SetResult(UpdatedAnswer); + Assert.Equal(HttpStatusCode.OK, (await first).Status); + } + + private const int SlotCycles = 400; + + [Fact] + public async Task AnEditThatFinishedInsideTheWait_FreesTheSlotBeforeItsAnswer_SoAnImmediateSecondEditOrAddIsNotRefused() + { + await using var rig = await StartAsync(); + /* Back to back, nothing between the two requests: a release that waited for the continuation would be refused. */ + for (var i = 0; i < SlotCycles; i++) + { + var (editStatus, _) = await PatchAsync(rig, Changes()); + Assert.True(editStatus == HttpStatusCode.OK, $"edit {i + 1} of {SlotCycles} answered {(int)editStatus} right after a finished edit"); + } + + for (var i = 0; i < SlotCycles; i++) + { + Assert.Equal(HttpStatusCode.OK, (await PatchAsync(rig, Changes())).Status); + var (addStatus, _) = await PostAddAsync(rig); + Assert.True(addStatus == HttpStatusCode.OK, $"add {i + 1} of {SlotCycles} answered {(int)addStatus} right after a finished edit"); + var (again, _) = await PatchAsync(rig, Changes()); + Assert.True(again == HttpStatusCode.OK, $"edit {i + 1} of {SlotCycles} answered {(int)again} right after a finished add"); + } + } + + [Fact] + public async Task AnEditThatFinishedInsideTheWait_ReleasesTheSlotExactlyOnce_SoALaterHeldEditStillBlocksTheNext() + { + var calls = 0; + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await using var rig = await StartAsync(edit: (_, _) => + { + if (Interlocked.Increment(ref calls) % 2 == 1) + { + return Task.FromResult(UpdatedAnswer); + } + + entered.TrySetResult(); + return release.Task.ContinueWith(_ => UpdatedAnswer, TaskScheduler.Default); + }); + + for (var i = 0; i < SlotCycles; i++) + { + Assert.Equal(HttpStatusCode.OK, (await PatchAsync(rig, Changes())).Status); + entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var held = PatchAsync(rig, Changes()); + await entered.Task.WaitAsync(TimeSpan.FromSeconds(30), TestContext.Current.CancellationToken); + + Assert.Equal(HttpStatusCode.TooManyRequests, (await PostAddAsync(rig)).Status); + Assert.Equal(HttpStatusCode.TooManyRequests, (await PatchAsync(rig, Changes())).Status); + Assert.Equal(2 * i + 2, Volatile.Read(ref calls)); + + release.SetResult(); + Assert.Equal(HttpStatusCode.OK, (await held).Status); + } + } + + [Fact] + public async Task AnEditThatOutlivesTheTimeout_Answers503_KeepsTheSlotForAddAndEdit_AndFreesItOnce_WhenItFinishes() + { + var hang = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var calls = 0; + await using var rig = await StartAsync( + edit: (_, _) => Interlocked.Increment(ref calls) == 1 ? hang.Task : Task.FromResult(UpdatedAnswer), + editTimeout: TimeSpan.FromMilliseconds(200)); + + var (status, body) = await PatchAsync(rig, Changes("\"password\":\"" + FakeSecret + "\""), principal: "dana"); + Assert.Equal(HttpStatusCode.ServiceUnavailable, status); + Assert.Equal(DarlingWebEndpoints.ServerEditTimedOutText, JsonNode.Parse(body)!["error"]!.GetValue()); + AssertNoSecret(rig, body); + + Assert.Equal(HttpStatusCode.TooManyRequests, (await PatchAsync(rig, Changes())).Status); + Assert.Equal(HttpStatusCode.TooManyRequests, (await PostAddAsync(rig)).Status); + Assert.Equal(1, Volatile.Read(ref calls)); + Assert.DoesNotContain(rig.Log.Lines, l => l.Contains("Server edited by", StringComparison.Ordinal)); + + /* The client was told 503, but the core commits when it finishes: its audit line is written then, once, for the + principal who sent the edit (not whoever sends the next one). */ + hang.SetResult(UpdatedAnswer); + await WaitForLogLineAsync(rig, "Server edited by"); + var late = Assert.Single(rig.Log.Lines, l => l.Contains("Server edited by", StringComparison.Ordinal)); + Assert.Contains("Server edited by dana: id 41, fields display_name,monthly_cost_usd", late, StringComparison.Ordinal); + AssertNoSecret(rig, body); + + var accepted = HttpStatusCode.TooManyRequests; + for (var i = 0; i < 100 && accepted == HttpStatusCode.TooManyRequests; i++) + { + accepted = (await PatchAsync(rig, Changes())).Status; + if (accepted == HttpStatusCode.TooManyRequests) + { + await Task.Delay(50, TestContext.Current.CancellationToken); + } + } + + Assert.Equal(HttpStatusCode.OK, accepted); + Assert.Single(rig.Log.Lines, l => l.Contains("Server edited by dana", StringComparison.Ordinal)); + Assert.Single(rig.Log.Lines, l => l.Contains("Server edited by alice", StringComparison.Ordinal)); + } + + [Fact] + public void TheEditTimeout_IsSixtySeconds_AndAddKeepsItsHundredAndTwenty() + { + Assert.Equal(TimeSpan.FromSeconds(60), DarlingWebEndpoints.ServerEditSlotTimeout); + Assert.Equal(TimeSpan.FromSeconds(120), DarlingWebEndpoints.ServerAddSlotTimeout); + } + + /* ═══════════════════════ GET /api/admin/servers/{id} ═══════════════════════ */ + + private static DarlingMcpServerAdminTools.ServerEditRow Row(int id = 41) => new( + id, "Orders", "sql01.example.test", 1433, "orders", true, "sqlserver", "sql", "monitor", "Strict", true, false, 12.5m, + new DateTime(2026, 10, 5, 12, 0, 0, 123, DateTimeKind.Unspecified).AddTicks(4567)); + + [Fact] + public async Task TheAdminRead_ReturnsTheEditableNonSecretValues_AndAnOpaqueToken_AndNoSecretKey() + { + await using var rig = await StartAsync(read: id => Task.FromResult(Row(id))); + var response = await rig.Client.GetAsync("/api/admin/servers/41", TestContext.Current.CancellationToken); + var body = await response.Content.ReadAsStringAsync(TestContext.Current.CancellationToken); + Assert.Equal(HttpStatusCode.OK, response.StatusCode); + Assert.Contains("no-store", response.Headers.CacheControl?.ToString() ?? "", StringComparison.Ordinal); + + var json = JsonNode.Parse(body)!.AsObject(); + Assert.Equal("Orders", json["display_name"]!.GetValue()); + Assert.Equal("SQL", json["auth"]!.GetValue()); + Assert.Equal("monitor", json["username"]!.GetValue()); + Assert.Equal(12.5m, json["monthly_cost_usd"]!.GetValue()); + Assert.Equal("2026-10-05T12:00:00.1234567", json["modified_at"]!.GetValue()); + Assert.DoesNotContain(json.Select(p => p.Key), k => k.Contains("password", StringComparison.OrdinalIgnoreCase) || k.Contains("secret", StringComparison.OrdinalIgnoreCase) || k.Contains("encrypted", StringComparison.OrdinalIgnoreCase)); + Assert.DoesNotContain("encrypted_password", body, StringComparison.Ordinal); + } + + [Fact] + public async Task TheAdminReadsToken_IsAcceptedByTheEditRouteAsIs() + { + await using var rig = await StartAsync(read: id => Task.FromResult(Row(id))); + var json = JsonNode.Parse(await rig.Client.GetStringAsync("/api/admin/servers/41", TestContext.Current.CancellationToken))!; + var token = json["modified_at"]!.GetValue(); + Assert.Equal(DarlingMcpServerAdminTools.ModifiedAtToken(Row().ModifiedAt), token); + var (status, _) = await PatchAsync(rig, "{\"display_name\":\"x\",\"expected_modified_at\":\"" + token + "\"}"); + Assert.Equal(HttpStatusCode.OK, status); + Assert.Contains(token, Assert.Single(rig.Edits).Body, StringComparison.Ordinal); + } + + [Fact] + public async Task TheAdminRead_OfAMissingServer_Answers404() + { + await using var rig = await StartAsync(read: _ => Task.FromResult(null)); + Assert.Equal(HttpStatusCode.NotFound, (await rig.Client.GetAsync("/api/admin/servers/9", TestContext.Current.CancellationToken)).StatusCode); + } + + [Fact] + public async Task TheAdminRead_WhenTheStoreFaults_AnswersTheFixedErrorBody_NotTheExceptionText() + { + await using var rig = await StartAsync(read: _ => throw new InvalidOperationException("password authentication failed for host db.internal.example")); + var response = await rig.Client.GetAsync("/api/admin/servers/9", TestContext.Current.CancellationToken); + var body = await response.Content.ReadAsStringAsync(TestContext.Current.CancellationToken); + Assert.True((int)response.StatusCode >= 500); + Assert.DoesNotContain("db.internal.example", body, StringComparison.Ordinal); + Assert.DoesNotContain("db.internal.example", rig.Log.Joined, StringComparison.Ordinal); + } + + /// The host's group gate lets every GET through for a read-only seat, so the route's own edit-right check + /// is the only thing between that seat and the form's pre-fill (username, TLS posture). The gate on or off, the + /// seat gets 403 and the read never reaches the store. + [Fact] + public async Task TheAdminRead_IsRefused403ForAReadOnlySeat_AndNeverReachesTheStore_WithOrWithoutTheHostWriteGate() + { + foreach (var gate in new[] { false, true }) + { + var reads = 0; + await using var rig = await StartAsync(useWriteGate: gate, read: id => + { + Interlocked.Increment(ref reads); + return Task.FromResult(Row(id)); + }); + var (status, body) = await SendAsync(rig, HttpMethod.Get, "/api/admin/servers/41", seat: "viewer"); + Assert.Equal(HttpStatusCode.Forbidden, status); + Assert.Contains("read-only", body, StringComparison.Ordinal); + Assert.DoesNotContain("monitor", body, StringComparison.Ordinal); + Assert.Equal(0, Volatile.Read(ref reads)); + + var (post, _) = await SendAsync(rig, HttpMethod.Post, "/api/admin/servers/41", "{}", seat: "viewer"); + Assert.NotEqual(HttpStatusCode.OK, post); + Assert.Equal(0, Volatile.Read(ref reads)); + + /* The same rig still serves an editing seat: the refusal is the seat's, not the route's. */ + var (editing, _) = await SendAsync(rig, HttpMethod.Get, "/api/admin/servers/41"); + Assert.Equal(HttpStatusCode.OK, editing); + Assert.Equal(1, Volatile.Read(ref reads)); + } + } + + [Fact] + public async Task AStoreWhoseRolesPredateTheEditFunction_Answers500_WithABodyThatSaysToRerunTheProvisionScript() + { + var envelope = PerformanceMonitor.Common.McpHelpers.FormatError("edit_server", new InvalidOperationException(DarlingMcpServerAdminTools.EditStoreNeedsRolesText)); + await using var rig = await StartAsync(edit: (_, _) => Task.FromResult(envelope)); + + var (status, body) = await PatchAsync(rig, Changes()); + + Assert.Equal(HttpStatusCode.InternalServerError, status); + Assert.Equal(DarlingMcpServerAdminTools.EditStoreNeedsRolesText, JsonNode.Parse(body)!["error"]!.GetValue()); + Assert.Contains("provision-roles.sql", body, StringComparison.Ordinal); + } + + /* A server id is a signed hash of the server's storage name, so about half of all ids are negative. The routes take them. */ + private const int NegativeServerId = -1862834905; + + [Fact] + public async Task AnEditRouteWithANegativeServerId_ReachesTheCoreWithThatId() + { + await using var rig = await StartAsync(); + var body = Changes("\"monthly_cost_usd\":12.5"); + + var (status, answer) = await PatchAsync(rig, body, id: NegativeServerId); + + Assert.Equal(HttpStatusCode.OK, status); + Assert.Equal((NegativeServerId, body), Assert.Single(rig.Edits)); + Assert.Equal("updated", JsonNode.Parse(answer)!["status"]!.GetValue()); + } + + [Fact] + public async Task TheAdminReadWithANegativeServerId_ReadsThatRow() + { + var asked = new List(); + await using var rig = await StartAsync(read: id => + { + asked.Add(id); + return Task.FromResult(Row(id)); + }); + + var response = await rig.Client.GetAsync("/api/admin/servers/" + NegativeServerId, TestContext.Current.CancellationToken); + var body = await response.Content.ReadAsStringAsync(TestContext.Current.CancellationToken); + + Assert.Equal(HttpStatusCode.OK, response.StatusCode); + Assert.Equal(NegativeServerId, Assert.Single(asked)); + Assert.Equal("Orders", JsonNode.Parse(body)!["display_name"]!.GetValue()); + } +} diff --git a/Darling/Darling.Tests/ServerEditViewerRoleLiveTests.cs b/Darling/Darling.Tests/ServerEditViewerRoleLiveTests.cs new file mode 100644 index 0000000000..469f131749 --- /dev/null +++ b/Darling/Darling.Tests/ServerEditViewerRoleLiveTests.cs @@ -0,0 +1,389 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Net; +using System.Net.Http; +using System.Text; +using System.Text.Json.Nodes; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.TestHost; +using Microsoft.Extensions.Logging; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Hosting; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// The web edit routes (#5240) end to end against a store, on a pool connected as a role holding exactly the grants the +/// managed viewer role gets (taken from , so removing the +/// EXECUTE grant on config.edit_monitored_server makes the edit fail with 42501; the role holds no UPDATE on the +/// table): an edit succeeds, a stale token is a 409 carrying +/// current, an edit while an add holds the shared slot is a 429, and the by-id read returns no secret. Edits here +/// change the name and cost only, which need no probe. +/// +[Collection("darling-owned-secrets")] +public sealed class ServerEditViewerRoleLiveTests : IDisposable +{ + private readonly DarlingOwnedSet _ownedBefore = DarlingOwnedSecrets.Current; + + public ServerEditViewerRoleLiveTests() => DarlingOwnedSecrets.Set(DarlingOwnedSet.Empty); + + public void Dispose() => DarlingOwnedSecrets.Set(_ownedBefore); + + private const string StoredBlob = "stored-blob-must-never-be-returned-Q7"; + + private static async Task ExecAsync(NpgsqlDataSource source, string sql, CancellationToken ct) + { + await using var command = source.CreateCommand(sql); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task ScalarAsync(NpgsqlDataSource source, string sql, CancellationToken ct) + { + await using var command = source.CreateCommand(sql); + return Convert.ToString(await command.ExecuteScalarAsync(ct), System.Globalization.CultureInfo.InvariantCulture)!; + } + + private static async Task<(WebApplication App, HttpClient Client)> HostAsync( + NpgsqlDataSource viewer, Func>? add, CancellationToken ct) + { + var builder = WebApplication.CreateBuilder(); + builder.WebHost.UseTestServer(); + builder.Logging.ClearProviders(); + var app = builder.Build(); + app.Use(async (context, next) => + { + context.Items[DarlingWebSeat.HttpContextItemKey] = new DarlingWebSeat("alice", true); + await next(context); + }); + DarlingWebEndpoints.MapServers(app, viewer, new CapturingTestLogger(), add); + await app.StartAsync(ct); + return (app, app.GetTestClient()); + } + + private static async Task<(HttpStatusCode Status, string Body)> PatchAsync(HttpClient client, int id, string body, CancellationToken ct) + { + using var request = new HttpRequestMessage(HttpMethod.Patch, "/api/servers/" + id) + { + Content = new StringContent(body, Encoding.UTF8, "application/json"), + }; + using var response = await client.SendAsync(request, ct); + return (response.StatusCode, await response.Content.ReadAsStringAsync(ct)); + } + + [Fact] + public async Task AsTheViewerRole_TheWebEdit_Succeeds_KeepsTheStoredSecret_AndAStaleTokenIsA409WithTheCurrentValues() + { + var ct = TestContext.Current.CancellationToken; + var roleName = "srv_web_" + Guid.NewGuid().ToString("N")[..8]; + var (scratch, owner, ownerString) = await ServerAddViewerRoleLiveTests.OpenAsync(ct); + await using var _ = scratch; + await using var __ = owner; + await using var viewer = await ServerAddViewerRoleLiveTests.ProvisionAsync(owner, ownerString, roleName, null, ct); + var ok = false; + try + { + await ExecAsync(owner, "INSERT INTO config_service (id) VALUES (1) ON CONFLICT DO NOTHING", ct); + await ExecAsync(owner, $"INSERT INTO config_monitored_servers (server_id, name, host, auth, username, encrypted_password, monthly_cost_usd) VALUES (5201, 'alpha', 'alpha.example.test', 'sql', 'monitor', '{StoredBlob}', 3)", ct); + var (app, client) = await HostAsync(viewer, null, ct); + await using var __app = app; + using var ___client = client; + + /* The read: editable non-secret values plus the opaque token, and no secret anywhere in it. */ + var read = await client.GetAsync("/api/admin/servers/5201", ct); + var readBody = await read.Content.ReadAsStringAsync(ct); + Assert.Equal(HttpStatusCode.OK, read.StatusCode); + Assert.DoesNotContain(StoredBlob, readBody, StringComparison.Ordinal); + Assert.DoesNotContain("encrypted", readBody, StringComparison.OrdinalIgnoreCase); + var token = JsonNode.Parse(readBody)!["modified_at"]!.GetValue(); + Assert.Equal("alpha", JsonNode.Parse(readBody)!["display_name"]!.GetValue()); + Assert.Equal((HttpStatusCode.NotFound, true), ((await client.GetAsync("/api/admin/servers/999999", ct)).StatusCode, true)); + + var versionBefore = await ScalarAsync(owner, "SELECT config_version FROM config_service", ct); + + /* The edit, as viewer: the SET list names only granted columns, FOR UPDATE works, the beacon fires. */ + var (status, body) = await PatchAsync(client, 5201, "{\"display_name\":\"Orders\",\"monthly_cost_usd\":120,\"expected_modified_at\":\"" + token + "\"}", ct); + Assert.True(status == HttpStatusCode.OK, "edit answered " + (int)status + ": " + body); + var answer = JsonNode.Parse(body)!; + Assert.Equal("updated", answer["status"]!.GetValue()); + Assert.Equal(5201, answer["server_id"]!.GetValue()); + Assert.NotEqual(token, answer["modified_at"]!.GetValue()); + Assert.Equal("Orders", await ScalarAsync(owner, "SELECT name FROM config_monitored_servers WHERE server_id = 5201", ct)); + Assert.Equal("120", await ScalarAsync(owner, "SELECT monthly_cost_usd::bigint FROM config_monitored_servers WHERE server_id = 5201", ct)); + Assert.Equal(StoredBlob, await ScalarAsync(owner, "SELECT encrypted_password FROM config_monitored_servers WHERE server_id = 5201", ct)); + Assert.Equal(long.Parse(versionBefore, System.Globalization.CultureInfo.InvariantCulture) + 1, + long.Parse(await ScalarAsync(owner, "SELECT config_version FROM config_service", ct), System.Globalization.CultureInfo.InvariantCulture)); + Assert.DoesNotContain(StoredBlob, body, StringComparison.Ordinal); + + /* The same (now stale) token again: 409 with the current non-secret values, and nothing written. */ + var (staleStatus, staleBody) = await PatchAsync(client, 5201, "{\"display_name\":\"Other\",\"expected_modified_at\":\"" + token + "\"}", ct); + Assert.Equal(HttpStatusCode.Conflict, staleStatus); + var stale = JsonNode.Parse(staleBody)!; + Assert.Equal("conflict", stale["status"]!.GetValue()); + Assert.Equal("Orders", stale["current"]!["display_name"]!.GetValue()); + Assert.Equal(answer["modified_at"]!.GetValue(), stale["current"]!["modified_at"]!.GetValue()); + Assert.DoesNotContain(StoredBlob, staleBody, StringComparison.Ordinal); + Assert.Equal("Orders", await ScalarAsync(owner, "SELECT name FROM config_monitored_servers WHERE server_id = 5201", ct)); + + /* The fresh token works, a stale id is a 404, and an unchanged request writes nothing. */ + var fresh = answer["modified_at"]!.GetValue(); + Assert.Equal(HttpStatusCode.OK, (await PatchAsync(client, 5201, "{\"display_name\":\"Orders\",\"expected_modified_at\":\"" + fresh + "\"}", ct)).Status); + Assert.Equal(HttpStatusCode.NotFound, (await PatchAsync(client, 999999, "{\"display_name\":\"x\",\"expected_modified_at\":\"" + fresh + "\"}", ct)).Status); + ok = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(ok, () => ExecAsync(owner, $"DROP OWNED BY {roleName}; DROP ROLE IF EXISTS {roleName};", CancellationToken.None)); + } + } + + [Fact] + public async Task AsTheViewerRole_AnEditWhileAnAddHoldsTheSlot_Is429_AndWritesNothing() + { + var ct = TestContext.Current.CancellationToken; + var roleName = "srv_slot_" + Guid.NewGuid().ToString("N")[..8]; + var (scratch, owner, ownerString) = await ServerAddViewerRoleLiveTests.OpenAsync(ct); + await using var _ = scratch; + await using var __ = owner; + await using var viewer = await ServerAddViewerRoleLiveTests.ProvisionAsync(owner, ownerString, roleName, null, ct); + var ok = false; + try + { + await ExecAsync(owner, "INSERT INTO config_service (id) VALUES (1) ON CONFLICT DO NOTHING", ct); + await ExecAsync(owner, "INSERT INTO config_monitored_servers (server_id, name, host) VALUES (5202, 'beta', 'beta.example.test')", ct); + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var (app, client) = await HostAsync(viewer, _ => + { + entered.TrySetResult(); + return release.Task; + }, ct); + await using var __app = app; + using var ___client = client; + + var token = JsonNode.Parse(await client.GetStringAsync("/api/admin/servers/5202", ct))!["modified_at"]!.GetValue(); + var held = client.PostAsync("/api/servers", new StringContent("[{\"host\":\"gamma\"}]", Encoding.UTF8, "application/json"), ct); + await entered.Task.WaitAsync(TimeSpan.FromSeconds(30), ct); + + var (status, _) = await PatchAsync(client, 5202, "{\"display_name\":\"Blocked\",\"expected_modified_at\":\"" + token + "\"}", ct); + Assert.Equal(HttpStatusCode.TooManyRequests, status); + Assert.Equal("beta", await ScalarAsync(owner, "SELECT name FROM config_monitored_servers WHERE server_id = 5202", ct)); + + release.SetResult("{\"requested\":1,\"added\":0,\"skipped\":0,\"collided\":0,\"failed\":0,\"results\":[]}"); + using (await held) + { + } + + Assert.Equal(HttpStatusCode.OK, (await PatchAsync(client, 5202, "{\"display_name\":\"Allowed\",\"expected_modified_at\":\"" + token + "\"}", ct)).Status); + ok = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(ok, () => ExecAsync(owner, $"DROP OWNED BY {roleName}; DROP ROLE IF EXISTS {roleName};", CancellationToken.None)); + } + } + + [Fact] + public async Task WithoutTheEditFunction_TheWebEdit_IsA500ThatSaysToRerunTheProvisionScript_AndNothingIsSaved() + { + var ct = TestContext.Current.CancellationToken; + var roleName = "srv_nogr_" + Guid.NewGuid().ToString("N")[..8]; + var (scratch, owner, ownerString) = await ServerAddViewerRoleLiveTests.OpenAsync(ct); + await using var _ = scratch; + await using var __ = owner; + await using var viewer = await ServerAddViewerRoleLiveTests.ProvisionAsync(owner, ownerString, roleName, ServerAddViewerRoleLiveTests.MissingEditFunction, ct); + var ok = false; + try + { + await ExecAsync(owner, "INSERT INTO config_service (id) VALUES (1) ON CONFLICT DO NOTHING", ct); + await ExecAsync(owner, "INSERT INTO config_monitored_servers (server_id, name, host) VALUES (5203, 'delta', 'delta.example.test')", ct); + var (app, client) = await HostAsync(viewer, null, ct); + await using var __app = app; + using var ___client = client; + + var token = JsonNode.Parse(await client.GetStringAsync("/api/admin/servers/5203", ct))!["modified_at"]!.GetValue(); + var (status, body) = await PatchAsync(client, 5203, "{\"display_name\":\"Nope\",\"expected_modified_at\":\"" + token + "\"}", ct); + Assert.Equal(HttpStatusCode.InternalServerError, status); + Assert.Equal(DarlingMcpServerAdminTools.EditStoreNeedsRolesText, JsonNode.Parse(body)!["error"]!.GetValue()); + Assert.Contains("provision-roles.sql", body, StringComparison.Ordinal); + foreach (var driverText in new[] { "42883", "42501", "edit_monitored_server", "does not exist", "permission denied" }) + { + Assert.DoesNotContain(driverText, body, StringComparison.OrdinalIgnoreCase); + } + Assert.Equal("delta", await ScalarAsync(owner, "SELECT name FROM config_monitored_servers WHERE server_id = 5203", ct)); + ok = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(ok, () => ExecAsync(owner, $"DROP OWNED BY {roleName}; DROP ROLE IF EXISTS {roleName};", CancellationToken.None)); + } + } + + /// + /// The edit core's REAL write with the credential in the SET list, as the viewer role (#5240): UPDATE + /// config_monitored_servers SET encrypted_password = $3, modified_at = ... WHERE server_id = $1 AND modified_at = $2 + /// RETURNING modified_at. The web edits above change the name and cost only, and the column fact in + /// runs hand-written statements, so nothing else ran this exact statement under + /// the viewer's grants: a write to a column viewer can set and cannot read, a predicate and a RETURNING that read only + /// modified_at (which viewer may read), and the statement-level beacon trigger firing as viewer. + /// + [Fact] + public async Task AsTheViewerRole_TheEditStoresRealUpdate_WithThePasswordInTheSetList_WritesUnderTheTokenPredicate_AndReturnsTheNewToken() + { + var ct = TestContext.Current.CancellationToken; + var roleName = "srv_pwd_" + Guid.NewGuid().ToString("N")[..8]; + var (scratch, owner, ownerString) = await ServerAddViewerRoleLiveTests.OpenAsync(ct); + await using var _ = scratch; + await using var __ = owner; + await using var viewer = await ServerAddViewerRoleLiveTests.ProvisionAsync(owner, ownerString, roleName, null, ct); + var ok = false; + try + { + await ExecAsync(owner, "INSERT INTO config_service (id) VALUES (1) ON CONFLICT DO NOTHING", ct); + await ExecAsync(owner, $"INSERT INTO config_monitored_servers (server_id, name, host, auth, username, encrypted_password) VALUES (5204, 'echo', 'echo.example.test', 'sql', 'monitor', '{StoredBlob}')", ct); + + var store = new DarlingMcpServerAdminTools.PostgresServerEditStore(viewer); + var row = await store.ReadRowAsync(5204, ct); + Assert.NotNull(row); + var versionBefore = long.Parse(await ScalarAsync(owner, "SELECT config_version FROM config_service", ct), System.Globalization.CultureInfo.InvariantCulture); + + /* The write: the credential column in the SET list, under the modified_at the edit read. */ + var write = await store.WriteAsync( + 5204, row!.ModifiedAt, [new DarlingMcpServerAdminTools.EditColumnValue("password", "encrypted_password", NpgsqlDbType.Text, "blob-2")], null, null, ct); + Assert.Equal(DarlingMcpServerAdminTools.ServerEditWriteKind.Written, write.Kind); + Assert.True(write.ModifiedAt > row.ModifiedAt, "RETURNING modified_at did not move past the token the edit read"); + + /* The new blob is stored, the new token is what the row now carries (read as owner and through the by-id read as + viewer), and the beacon moved exactly once. */ + Assert.Equal("blob-2", await ScalarAsync(owner, "SELECT encrypted_password FROM config_monitored_servers WHERE server_id = 5204", ct)); + await using (var stamp = owner.CreateCommand("SELECT modified_at FROM config_monitored_servers WHERE server_id = 5204")) + { + Assert.Equal(write.ModifiedAt, (DateTime)(await stamp.ExecuteScalarAsync(ct))!); + } + + Assert.Equal(write.ModifiedAt, (await store.ReadRowAsync(5204, ct))!.ModifiedAt); + Assert.Equal(versionBefore + 1, long.Parse(await ScalarAsync(owner, "SELECT config_version FROM config_service", ct), System.Globalization.CultureInfo.InvariantCulture)); + + /* The token just spent is stale: the same write is a conflict and changes nothing (the blob, the token, the beacon). */ + var stale = await store.WriteAsync( + 5204, row.ModifiedAt, [new DarlingMcpServerAdminTools.EditColumnValue("password", "encrypted_password", NpgsqlDbType.Text, "blob-3")], null, null, ct); + Assert.Equal(DarlingMcpServerAdminTools.ServerEditWriteKind.Conflict, stale.Kind); + Assert.Equal("blob-2", await ScalarAsync(owner, "SELECT encrypted_password FROM config_monitored_servers WHERE server_id = 5204", ct)); + Assert.Equal(versionBefore + 1, long.Parse(await ScalarAsync(owner, "SELECT config_version FROM config_service", ct), System.Globalization.CultureInfo.InvariantCulture)); + ok = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(ok, () => ExecAsync(owner, $"DROP OWNED BY {roleName}; DROP ROLE IF EXISTS {roleName};", CancellationToken.None)); + } + } + + /// + /// The MCP side of the same sentence: edit_server on a store whose roles predate the edit function answers + /// the error envelope carrying (re-run + /// provision-roles.sql), with none of the driver's own text, and saves nothing. The tool runs the same core and the + /// same store write the web route does, so this is the call a token holder makes against such a store. + /// + [Fact] + public async Task WithoutTheEditFunction_TheMcpEditServerTool_AnswersToRerunTheProvisionScript_AndNothingIsSaved() + { + var ct = TestContext.Current.CancellationToken; + var roleName = "srv_mcpn_" + Guid.NewGuid().ToString("N")[..8]; + var (scratch, owner, ownerString) = await ServerAddViewerRoleLiveTests.OpenAsync(ct); + await using var _ = scratch; + await using var __ = owner; + await using var role = await ServerAddViewerRoleLiveTests.ProvisionAsync(owner, ownerString, roleName, ServerAddViewerRoleLiveTests.MissingEditFunction, ct); + var ok = false; + try + { + await ExecAsync(owner, "INSERT INTO config_service (id) VALUES (1) ON CONFLICT DO NOTHING", ct); + await ExecAsync(owner, "INSERT INTO config_monitored_servers (server_id, name, host) VALUES (5205, 'echo', 'echo.example.test')", ct); + + /* A rename needs no connection test, so the probe never runs; the write is the first thing that meets the missing function. */ + var answer = await DarlingMcpServerAdminTools.EditServerByNameAsync( + role, "echo", "{\"display_name\":\"Nope\"}", (_, _) => throw new InvalidOperationException("a rename must not be probed"), TestKeyRings.Healthy, null, ct); + + var envelope = JsonNode.Parse(answer)!; + Assert.Equal("error", envelope["status"]!.GetValue()); + Assert.Contains(DarlingMcpServerAdminTools.EditStoreNeedsRolesText, envelope["message"]!.GetValue(), StringComparison.Ordinal); + Assert.Contains("provision-roles.sql", answer, StringComparison.Ordinal); + foreach (var driverText in new[] { "42883", "42501", "edit_monitored_server", "does not exist", "permission denied" }) + { + Assert.DoesNotContain(driverText, answer, StringComparison.OrdinalIgnoreCase); + } + + Assert.Equal("echo", await ScalarAsync(owner, "SELECT name FROM config_monitored_servers WHERE server_id = 5205", ct)); + ok = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(ok, () => ExecAsync(owner, $"DROP OWNED BY {roleName}; DROP ROLE IF EXISTS {roleName};", CancellationToken.None)); + } + } + + /// + /// A field an edit cannot change is refused by the real core on the way through the web route: 400, a message that + /// names the key (any remediation_ key is named remediation_*), and the row is untouched. The refused key + /// comes FIRST in the body because the core names the first refused key in body order, and is_enabled is itself + /// refused. The row's id is negative, as about half of all ids are, so this also runs the live read and write routes on one. + /// + [Fact] + public async Task AsTheViewerRole_ARefusedField_Is400ThroughTheRealCore_NamesTheKey_AndLeavesTheRowAlone() + { + var ct = TestContext.Current.CancellationToken; + var roleName = "srv_ref_" + Guid.NewGuid().ToString("N")[..8]; + var (scratch, owner, ownerString) = await ServerAddViewerRoleLiveTests.OpenAsync(ct); + await using var _ = scratch; + await using var __ = owner; + await using var viewer = await ServerAddViewerRoleLiveTests.ProvisionAsync(owner, ownerString, roleName, null, ct); + var ok = false; + try + { + await ExecAsync(owner, "INSERT INTO config_service (id) VALUES (1) ON CONFLICT DO NOTHING", ct); + await ExecAsync(owner, "INSERT INTO config_monitored_servers (server_id, name, host) VALUES (-5206, 'foxtrot', 'foxtrot.example.test')", ct); + var (app, client) = await HostAsync(viewer, null, ct); + await using var __app = app; + using var ___client = client; + + var token = JsonNode.Parse(await client.GetStringAsync("/api/admin/servers/-5206", ct))!["modified_at"]!.GetValue(); + var before = await ScalarAsync(owner, "SELECT modified_at::text FROM config_monitored_servers WHERE server_id = -5206", ct); + + foreach (var (key, value, named) in new[] + { + ("server_id", "7", "server_id"), + ("remediation_username", "\"someone\"", "remediation_*"), + }) + { + var body = $"{{\"{key}\":{value},\"is_enabled\":false,\"expected_modified_at\":\"{token}\"}}"; + var (status, answer) = await PatchAsync(client, -5206, body, ct); + + Assert.Equal(HttpStatusCode.BadRequest, status); + Assert.Contains($"'{named}' cannot be changed here", JsonNode.Parse(answer)!["message"]!.GetValue(), StringComparison.Ordinal); + Assert.Equal(before, await ScalarAsync(owner, "SELECT modified_at::text FROM config_monitored_servers WHERE server_id = -5206", ct)); + Assert.Equal("foxtrot", await ScalarAsync(owner, "SELECT name FROM config_monitored_servers WHERE server_id = -5206", ct)); + } + + ok = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(ok, () => ExecAsync(owner, $"DROP OWNED BY {roleName}; DROP ROLE IF EXISTS {roleName};", CancellationToken.None)); + } + } +} diff --git a/Darling/Darling.Tests/ServerIdentityLockLiveTests.cs b/Darling/Darling.Tests/ServerIdentityLockLiveTests.cs new file mode 100644 index 0000000000..2619248c0a --- /dev/null +++ b/Darling/Darling.Tests/ServerIdentityLockLiveTests.cs @@ -0,0 +1,775 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.Json.Nodes; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; +using Edit = PerformanceMonitor.Darling.Service.Mcp.DarlingMcpServerAdminTools; + +namespace Darling.Tests; + +/* #1776 own-store: each fact mints its own scratch database through ScratchPostgres and never touches another one; + the advisory lock it contends on is scoped to that database. */ + +/// +/// Every write that gives a monitored server an address takes ONE store lock and re-checks the address under it +/// (#5240, review finding 6). server_id is the hash of the storage key, an edit keeps its row's old id, and +/// the table has no unique index on the key, so without the lock an add_servers INSERT and an +/// edit_server UPDATE (or two edits) can each find an address free during their connection probe and each +/// take it: two definitions with one storage key, both monitored. +/// +/// How the race is made to happen. Both callers are parked inside their probe, which both reach after +/// their first read of the table showed the address free. The test then takes the lock from a second connection and +/// lets the probes finish, so both callers reach their write with the address still free in the table. With the +/// lock in both write paths, both queue behind the held lock and the winner's commit is what the loser re-checks +/// against. Without it, both writes land: the assertions on the table fail, and the queue assertion with them. +/// +/// The desktop viewer's writes (#5240, PR 2). The viewer's add (AddMonitoredServerAsync) and edit +/// (UpsertMonitoredServerAsync) take the same lock and re-read the addresses under it. The viewer has no probe +/// to park in, so its facts make the race the other way round: the lock is taken from a second connection FIRST, the +/// two callers are started one at a time (each waits until the one before it is queued, so the lock is granted in call +/// order), and the lock is released once both are queued. The caller that queued first wins; the second's re-check +/// runs against the first's committed write and must refuse. Each viewer fact runs in both orders, so the viewer's +/// re-check is what refuses in one of them. A viewer write that took no lock would not queue (the queue assertion +/// fails); one that did not re-check would write over the other caller (the answers and the table assertions fail). +/// The startup seed's INSERT loop takes the lock too, and skips an address that was claimed while it waited. +/// +[Collection("live-postgres")] +public sealed class ServerIdentityLockLiveTests +{ + private const string AddressK = "race-k.example.test"; + + private static readonly ConnectionProbeResult Reached = new(true, 15, 3, "Enterprise", false, false, false, true, null); + + /// The probe both callers share: it counts arrivals and waits for . + private sealed class ParkedProbe + { + private readonly TaskCompletionSource _release = new(TaskCreationOptions.RunContinuationsAsynchronously); + private int _arrived; + + public int Arrived => Volatile.Read(ref _arrived); + + public Edit.ServerProbe Probe => async (_, probeToken) => + { + Interlocked.Increment(ref _arrived); + await _release.Task.WaitAsync(probeToken); + return Reached; + }; + + public void Release() => _release.TrySetResult(); + } + + private sealed record Rig(ScratchPostgres Scratch, NpgsqlDataSource Owner, string OwnerString) : IAsyncDisposable + { + public async ValueTask DisposeAsync() + { + await Owner.DisposeAsync(); + await Scratch.DisposeAsync(); + } + } + + private sealed record Race(string FirstAnswer, string SecondAnswer, bool BothQueuedOnTheLock, long LocksLeftAfterwards); + + private static async Task OpenAsync(CancellationToken ct) + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live server identity lock tests (each mints its own scratch database)."); + + var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var ownerString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { SearchPath = PgSchemaGenerator.SearchPath }.ConnectionString; + await using (var connection = new NpgsqlConnection(ownerString)) + { + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, null, ct); + } + + var owner = NpgsqlDataSource.Create(ownerString); + + /* The edit write is a function (#5240) the provisioning batch creates, not a migration: this rig creates the real one. */ + await ExecAsync(owner, DarlingManagedRoles.BuildEditMonitoredServerFunctionSql("config"), ct); + await ExecAsync(owner, DarlingManagedRoles.BuildEditMonitoredServerLegacyWrapperSql("config"), ct); + await ExecAsync(owner, "INSERT INTO config_service (id) VALUES (1) ON CONFLICT DO NOTHING", ct); + return new Rig(scratch, owner, ownerString); + } + + private static async Task ExecAsync(NpgsqlDataSource source, string sql, CancellationToken ct) + { + await using var command = source.CreateCommand(sql); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task CountAsync(NpgsqlDataSource source, string sql, CancellationToken ct) + { + await using var command = source.CreateCommand(sql); + return Convert.ToInt64(await command.ExecuteScalarAsync(ct)); + } + + private static Task SeedServerAsync(NpgsqlDataSource owner, int id, string name, string host, CancellationToken ct) => + ExecAsync(owner, $@"INSERT INTO config_monitored_servers + (server_id, name, host, auth, username, encrypted_password, excluded_databases, capture_plans, is_enabled, + alert_delivery_mode_override, plan_force_bot_enabled, remediation_username, remediation_encrypted_password, monthly_cost_usd) + VALUES ({id}, '{name}', '{host}', 'integrated', NULL, NULL, + ARRAY['tempdb','model'], TRUE, FALSE, 'PerEvent', TRUE, 'rem-user', NULL, 7)", ct); + + /// The storage key of every definition, mapped from the same five columns the product reads (a NULL + /// engine or port is the default one), in server_id order. + private static async Task> StorageKeysAsync(NpgsqlDataSource owner, CancellationToken ct) + { + var keys = new List(); + await using var command = owner.CreateCommand( + "SELECT host, database, read_only_intent, engine, port FROM config_monitored_servers ORDER BY server_id"); + await using var reader = await command.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + keys.Add(ServerIdHelper.BuildStorageName( + reader.GetString(0), reader.IsDBNull(1) ? null : reader.GetString(1), !reader.IsDBNull(2) && reader.GetBoolean(2), + reader.IsDBNull(3) ? null : reader.GetString(3), reader.IsDBNull(4) ? 0 : reader.GetInt32(4))); + } + + return keys; + } + + /// Waiters (granted = false) or holders (true) of advisory locks in THIS database. + private static Task AdvisoryLocksAsync(NpgsqlDataSource owner, bool granted, CancellationToken ct) => + CountAsync(owner, @"SELECT count(*) FROM pg_locks + WHERE locktype = 'advisory' AND granted = " + (granted ? "true" : "false") + @" + AND database = (SELECT oid FROM pg_database WHERE datname = current_database())", ct); + + private static async Task WaitUntilAsync(Func> condition, TimeSpan timeout, CancellationToken ct) + { + var deadline = DateTime.UtcNow + timeout; + while (true) + { + if (await condition()) + { + return true; + } + + if (DateTime.UtcNow >= deadline) + { + return false; + } + + await Task.Delay(TimeSpan.FromMilliseconds(50), ct); + } + } + + private static JsonNode Parse(string answer) => JsonNode.Parse(answer)!; + + private static string AddJson(string host) => "[{\"host\":\"" + host + "\"}]"; + + private static string AddStatusOf(string answer) => Parse(answer)["results"]![0]!["status"]!.GetValue(); + + /// + /// Parks both callers in their probe, takes the identity lock from a second connection, lets the probes finish, + /// waits until both callers are queued on the lock (or one of them is already done, which is what a write path + /// without the lock looks like), then releases the lock and returns both answers. + /// + private static async Task RaceAsync( + Rig rig, Func> first, Func> second, CancellationToken ct) + { + var parked = new ParkedProbe(); + var firstCall = Task.Run(() => first(parked.Probe), ct); + var secondCall = Task.Run(() => second(parked.Probe), ct); + + var holder = new NpgsqlConnection(rig.OwnerString); + NpgsqlTransaction? held = null; + var bodySucceeded = false; + try + { + Assert.True( + await WaitUntilAsync(() => Task.FromResult(parked.Arrived == 2), TimeSpan.FromSeconds(30), ct), + $"Both callers should reach their probe (the address read as free for each); {parked.Arrived} did."); + + await holder.OpenAsync(ct); + held = await holder.BeginTransactionAsync(ct); + await using (var take = new NpgsqlCommand("SELECT pg_advisory_xact_lock(hashtext('config_monitored_servers.identity'))", holder, held)) + { + await take.ExecuteNonQueryAsync(ct); + } + + parked.Release(); + var bothQueued = false; + await WaitUntilAsync(async () => + { + if (await AdvisoryLocksAsync(rig.Owner, false, ct) >= 2) + { + bothQueued = true; + return true; + } + + return firstCall.IsCompleted || secondCall.IsCompleted; + }, TimeSpan.FromSeconds(30), ct); + + await held.RollbackAsync(ct); + var answers = await Task.WhenAll(firstCall, secondCall).WaitAsync(TimeSpan.FromSeconds(60), ct); + var left = await AdvisoryLocksAsync(rig.Owner, true, ct) + await AdvisoryLocksAsync(rig.Owner, false, ct); + bodySucceeded = true; + return new Race(answers[0], answers[1], bothQueued, left); + } + finally + { + parked.Release(); + + /* The holder's own rollback: only the session holding the lock can end its transaction. */ + await LiveStoreCleanup.RunOwnedAsync(bodySucceeded, async () => + { + if (held is not null) + { + await held.DisposeAsync(); + } + + await holder.DisposeAsync(); + }); + } + } + + [Fact] + public async Task AnAddAndAnEditThatClaimOneAddress_QueueOnTheLock_AndExactlyOneRowEndsWithIt() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + await SeedServerAsync(rig.Owner, 7301, "race-x", "race-x.example.test", ct); + + var race = await RaceAsync(rig, + probe => Edit.AddServersAsync(rig.Owner, AddJson(AddressK), probe, ct), + probe => Edit.EditServerByNameAsync(rig.Owner, "race-x", "{\"host\":\"" + AddressK + "\"}", probe, TestKeyRings.Healthy, null, ct), + ct); + + var addStatus = AddStatusOf(race.FirstAnswer); + var edit = Parse(race.SecondAnswer); + var editStatus = edit["status"]!.GetValue(); + var addWon = addStatus == "added" && editStatus == "collides"; + var editWon = addStatus == "duplicate" && editStatus == "updated"; + Assert.True(addWon ^ editWon, $"Exactly one caller should win. add: {addStatus}, edit: {editStatus}."); + if (addWon) + { + Assert.Equal("occupied", edit["reason"]!.GetValue()); + } + else + { + Assert.Contains("claimed this address", Parse(race.FirstAnswer)["results"]![0]!["detail"]!.GetValue(), StringComparison.Ordinal); + } + + var keys = await StorageKeysAsync(rig.Owner, ct); + Assert.Equal(keys.Count, keys.Distinct(StringComparer.OrdinalIgnoreCase).Count()); + Assert.Single(keys, key => key.Contains(AddressK, StringComparison.OrdinalIgnoreCase)); + Assert.Equal(addWon ? 2 : 1, keys.Count); + Assert.Equal(0, race.LocksLeftAfterwards); + Assert.True(race.BothQueuedOnTheLock, "Both the add and the edit should have queued behind the held identity lock."); + } + + [Fact] + public async Task TwoEditsThatMoveTwoServersOntoOneAddress_QueueOnTheLock_AndExactlyOneRowEndsWithIt() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + await SeedServerAsync(rig.Owner, 7311, "race-x", "race-x.example.test", ct); + await SeedServerAsync(rig.Owner, 7312, "race-y", "race-y.example.test", ct); + + var race = await RaceAsync(rig, + probe => Edit.EditServerByNameAsync(rig.Owner, "race-x", "{\"host\":\"" + AddressK + "\"}", probe, TestKeyRings.Healthy, null, ct), + probe => Edit.EditServerByNameAsync(rig.Owner, "race-y", "{\"host\":\"" + AddressK + "\"}", probe, TestKeyRings.Healthy, null, ct), + ct); + + var statuses = new[] { Parse(race.FirstAnswer), Parse(race.SecondAnswer) }.Select(a => a["status"]!.GetValue()).Order().ToArray(); + Assert.Equal(new[] { "collides", "updated" }, statuses); + + var keys = await StorageKeysAsync(rig.Owner, ct); + Assert.Equal(2, keys.Count); + Assert.Equal(keys.Count, keys.Distinct(StringComparer.OrdinalIgnoreCase).Count()); + Assert.Single(keys, key => key.Contains(AddressK, StringComparison.OrdinalIgnoreCase)); + Assert.Equal(0, race.LocksLeftAfterwards); + Assert.True(race.BothQueuedOnTheLock, "Both edits should have queued behind the held identity lock."); + } + + [Fact] + public async Task AnAddAndAnEditOnDifferentAddresses_BothSucceed_AfterQueueingOnTheLock() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + await SeedServerAsync(rig.Owner, 7321, "race-x", "race-x.example.test", ct); + + var race = await RaceAsync(rig, + probe => Edit.AddServersAsync(rig.Owner, AddJson("race-added.example.test"), probe, ct), + probe => Edit.EditServerByNameAsync(rig.Owner, "race-x", "{\"host\":\"race-moved.example.test\"}", probe, TestKeyRings.Healthy, null, ct), + ct); + + Assert.Equal("added", AddStatusOf(race.FirstAnswer)); + Assert.Equal("updated", Parse(race.SecondAnswer)["status"]!.GetValue()); + + var keys = await StorageKeysAsync(rig.Owner, ct); + Assert.Equal(2, keys.Count); + Assert.Single(keys, key => key.Contains("race-added.example.test", StringComparison.OrdinalIgnoreCase)); + Assert.Single(keys, key => key.Contains("race-moved.example.test", StringComparison.OrdinalIgnoreCase)); + Assert.Equal(0, race.LocksLeftAfterwards); + Assert.True(race.BothQueuedOnTheLock, "Both the add and the edit should have queued behind the held identity lock."); + } + + [Fact] + public async Task TheConnectionProbeRunsOutsideTheLock_SoASlowProbeNeverHoldsUpOtherWrites() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + await SeedServerAsync(rig.Owner, 7331, "race-x", "race-x.example.test", ct); + + var parked = new ParkedProbe(); + var add = Task.Run(() => Edit.AddServersAsync(rig.Owner, AddJson("race-added.example.test"), parked.Probe, ct), ct); + var edit = Task.Run(() => Edit.EditServerByNameAsync(rig.Owner, "race-x", "{\"host\":\"race-moved.example.test\"}", parked.Probe, TestKeyRings.Healthy, null, ct), ct); + var bodySucceeded = false; + try + { + Assert.True( + await WaitUntilAsync(() => Task.FromResult(parked.Arrived == 2), TimeSpan.FromSeconds(30), ct), + $"Both callers should be waiting in their probe; {parked.Arrived} were."); + + /* While both sit in a probe (up to about 45 seconds in the field) neither holds the lock: nothing in the + database holds or waits on an advisory lock, and a third session takes it at once. */ + Assert.Equal(0, await AdvisoryLocksAsync(rig.Owner, true, ct)); + Assert.Equal(0, await AdvisoryLocksAsync(rig.Owner, false, ct)); + await using (var probeConnection = await rig.Owner.OpenConnectionAsync(ct)) + await using (var transaction = await probeConnection.BeginTransactionAsync(ct)) + await using (var tryLock = new NpgsqlCommand("SELECT pg_try_advisory_xact_lock(hashtext('config_monitored_servers.identity'))", probeConnection, transaction)) + { + Assert.True((bool)(await tryLock.ExecuteScalarAsync(ct))!, "The identity lock should be free while both callers are in their probe."); + await transaction.RollbackAsync(ct); + } + + parked.Release(); + Assert.Equal("added", AddStatusOf(await add.WaitAsync(TimeSpan.FromSeconds(60), ct))); + Assert.Equal("updated", Parse(await edit.WaitAsync(TimeSpan.FromSeconds(60), ct))["status"]!.GetValue()); + Assert.Equal(0, await AdvisoryLocksAsync(rig.Owner, true, ct)); + bodySucceeded = true; + } + finally + { + /* A body that failed before the release must not leave both callers parked in their probe. */ + await LiveStoreCleanup.RunOwnedAsync(bodySucceeded, () => + { + parked.Release(); + return Task.CompletedTask; + }); + } + } + + /// A probe that answers at once: the viewer's callers have none, and the MCP caller in the same race + /// must reach its write without waiting. + private static readonly Edit.ServerProbe ReachedAtOnce = (_, _) => Task.FromResult(Reached); + + private static MonitoredServerRow ViewerRow(int id, string name, string host) => new() { ServerId = id, Name = name, Host = host }; + + /// The viewer's edit save, as one word: Updated, or Claimed when it refused the address. + private static async Task ViewerEditAsync(ViewerDataService viewer, MonitoredServerRow row, CancellationToken ct) + { + try + { + await viewer.UpsertMonitoredServerAsync(row, ct); + return "Updated"; + } + catch (MonitoredServerAddressClaimedException) + { + return "Claimed"; + } + } + + /// + /// The race the viewer's writes are held to: takes the SERVICE's identity lock () + /// from a second connection first, starts the callers one at a time, waits until both are queued on it (or one is + /// already done, which is what a write path without the lock looks like), then releases it and returns both answers + /// in call order. The first caller wins the lock. + /// + private static async Task QueuedRaceAsync(Rig rig, Func> first, Func> second, CancellationToken ct) + { + var holder = new NpgsqlConnection(rig.OwnerString); + NpgsqlTransaction? held = null; + var bodySucceeded = false; + try + { + await holder.OpenAsync(ct); + held = await holder.BeginTransactionAsync(ct); + await using (var take = new NpgsqlCommand(Edit.IdentityLockSql, holder, held)) + { + await take.ExecuteNonQueryAsync(ct); + } + + /* One at a time: the lock is granted to its waiters in the order they queued, so the caller started first + wins it and the second one's re-check is what runs against the first one's committed write. Starting both + at once would leave the winner to the scheduler, and a viewer write whose re-check was gone could pass by + luck whenever it happened to queue first. */ + var firstCall = Task.Run(first, ct); + await WaitUntilAsync( + async () => await AdvisoryLocksAsync(rig.Owner, false, ct) >= 1 || firstCall.IsCompleted, TimeSpan.FromSeconds(30), ct); + var secondCall = Task.Run(second, ct); + var bothQueued = false; + await WaitUntilAsync(async () => + { + if (await AdvisoryLocksAsync(rig.Owner, false, ct) >= 2) + { + bothQueued = true; + return true; + } + + return firstCall.IsCompleted || secondCall.IsCompleted; + }, TimeSpan.FromSeconds(30), ct); + + await held.RollbackAsync(ct); + var answers = await Task.WhenAll(firstCall, secondCall).WaitAsync(TimeSpan.FromSeconds(60), ct); + var left = await AdvisoryLocksAsync(rig.Owner, true, ct) + await AdvisoryLocksAsync(rig.Owner, false, ct); + bodySucceeded = true; + return new Race(answers[0], answers[1], bothQueued, left); + } + finally + { + /* The holder's own rollback: only the session holding the lock can end its transaction. */ + await LiveStoreCleanup.RunOwnedAsync(bodySucceeded, async () => + { + if (held is not null) + { + await held.DisposeAsync(); + } + + await holder.DisposeAsync(); + }); + } + } + + /// Whose write queues on the lock first, and so wins it: the viewer's add (the MCP edit's + /// re-check then refuses), or the MCP edit (the VIEWER's re-check then refuses the add). + [Theory] + [InlineData(true)] + [InlineData(false)] + public async Task AViewerAddAndAnMcpEditThatClaimOneAddress_QueueOnTheLock_AndTheSecondToQueueIsRefused(bool viewerFirst) + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + await using var viewer = new ViewerDataService(rig.Scratch.ConnectionString); + await SeedServerAsync(rig.Owner, 7341, "race-x", "race-x.example.test", ct); + + Func> viewerAdd = async () => (await viewer.AddMonitoredServerAsync( + ViewerRow(ViewerDataService.ComputeServerId(AddressK, null, false), "viewer-added", AddressK), ct)).Outcome.ToString(); + Func> mcpEdit = async () => Parse(await Edit.EditServerByNameAsync( + rig.Owner, "race-x", "{\"host\":\"" + AddressK + "\"}", ReachedAtOnce, TestKeyRings.Healthy, null, ct))["status"]!.GetValue(); + + var race = viewerFirst + ? await QueuedRaceAsync(rig, viewerAdd, mcpEdit, ct) + : await QueuedRaceAsync(rig, mcpEdit, viewerAdd, ct); + + if (viewerFirst) + { + Assert.Equal(nameof(MonitoredServerAddOutcome.Added), race.FirstAnswer); + Assert.Equal("collides", race.SecondAnswer); + } + else + { + Assert.Equal("updated", race.FirstAnswer); + Assert.Equal(nameof(MonitoredServerAddOutcome.Duplicate), race.SecondAnswer); + } + + var keys = await StorageKeysAsync(rig.Owner, ct); + Assert.Equal(keys.Count, keys.Distinct(StringComparer.OrdinalIgnoreCase).Count()); + Assert.Single(keys, key => key.Contains(AddressK, StringComparison.OrdinalIgnoreCase)); + Assert.Equal(viewerFirst ? 2 : 1, keys.Count); + Assert.Equal(0, race.LocksLeftAfterwards); + Assert.True(race.BothQueuedOnTheLock, "Both the viewer's add and the MCP edit should have queued behind the held identity lock."); + } + + /// Whose write queues on the lock first, and so wins it: the viewer's edit (the MCP add's + /// re-check then answers duplicate), or the MCP add (the VIEWER's re-check then refuses the edit). + [Theory] + [InlineData(true)] + [InlineData(false)] + public async Task AViewerEditAndAnMcpAddThatClaimOneAddress_QueueOnTheLock_AndTheSecondToQueueIsRefused(bool viewerFirst) + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + await using var viewer = new ViewerDataService(rig.Scratch.ConnectionString); + await SeedServerAsync(rig.Owner, 7351, "race-x", "race-x.example.test", ct); + + Func> viewerEdit = () => ViewerEditAsync(viewer, ViewerRow(7351, "race-x", AddressK), ct); + Func> mcpAdd = async () => AddStatusOf(await Edit.AddServersAsync(rig.Owner, AddJson(AddressK), ReachedAtOnce, ct)); + + var race = viewerFirst + ? await QueuedRaceAsync(rig, viewerEdit, mcpAdd, ct) + : await QueuedRaceAsync(rig, mcpAdd, viewerEdit, ct); + + if (viewerFirst) + { + Assert.Equal("Updated", race.FirstAnswer); + Assert.Equal("duplicate", race.SecondAnswer); + } + else + { + Assert.Equal("added", race.FirstAnswer); + Assert.Equal("Claimed", race.SecondAnswer); + } + + var keys = await StorageKeysAsync(rig.Owner, ct); + Assert.Equal(keys.Count, keys.Distinct(StringComparer.OrdinalIgnoreCase).Count()); + Assert.Single(keys, key => key.Contains(AddressK, StringComparison.OrdinalIgnoreCase)); + Assert.Equal(viewerFirst ? 1 : 2, keys.Count); + Assert.Equal(0, race.LocksLeftAfterwards); + Assert.True(race.BothQueuedOnTheLock, "Both the viewer's edit and the MCP add should have queued behind the held identity lock."); + } + + [Fact] + public async Task TwoViewerEditsThatMoveTwoServersOntoOneAddress_QueueOnTheLock_AndTheSecondToQueueIsRefused() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + await using var viewer = new ViewerDataService(rig.Scratch.ConnectionString); + await SeedServerAsync(rig.Owner, 7361, "race-x", "race-x.example.test", ct); + await SeedServerAsync(rig.Owner, 7362, "race-y", "race-y.example.test", ct); + + var race = await QueuedRaceAsync(rig, + () => ViewerEditAsync(viewer, ViewerRow(7361, "race-x", AddressK), ct), + () => ViewerEditAsync(viewer, ViewerRow(7362, "race-y", AddressK), ct), + ct); + + Assert.Equal("Updated", race.FirstAnswer); + Assert.Equal("Claimed", race.SecondAnswer); + + var keys = await StorageKeysAsync(rig.Owner, ct); + Assert.Equal(2, keys.Count); + Assert.Equal(keys.Count, keys.Distinct(StringComparer.OrdinalIgnoreCase).Count()); + Assert.Single(keys, key => key.Contains(AddressK, StringComparison.OrdinalIgnoreCase)); + Assert.Equal(0, race.LocksLeftAfterwards); + Assert.True(race.BothQueuedOnTheLock, "Both viewer edits should have queued behind the held identity lock."); + } + + /// The viewer's insert-if-absent (the migrate-in's old write) is an identity write too: an address an edit + /// left a definition at, under an older id, is refused (false, nothing written); a free address is written. + [Fact] + public async Task TheViewersInsertIfAbsent_RefusesAnAddressHeldUnderAnotherId_AndWritesAFreeOne() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + await using var viewer = new ViewerDataService(rig.Scratch.ConnectionString); + await SeedServerAsync(rig.Owner, 7391, "moved", AddressK, ct); + + Assert.False(await viewer.InsertMonitoredServerIfAbsentAsync( + ViewerRow(ViewerDataService.ComputeServerId(AddressK, null, false), "again", AddressK), ct)); + Assert.True(await viewer.InsertMonitoredServerIfAbsentAsync( + ViewerRow(ViewerDataService.ComputeServerId("free.example.test", null, false), "free", "free.example.test"), ct)); + + var keys = await StorageKeysAsync(rig.Owner, ct); + Assert.Equal(2, keys.Count); + Assert.Single(keys, key => key.Contains(AddressK, StringComparison.OrdinalIgnoreCase)); + Assert.Single(keys, key => key.Contains("free.example.test", StringComparison.OrdinalIgnoreCase)); + Assert.Equal(0, await AdvisoryLocksAsync(rig.Owner, true, ct) + await AdvisoryLocksAsync(rig.Owner, false, ct)); + } + + [Fact] + public async Task AViewerAddAndAViewerEditOnDifferentAddresses_BothSucceed_AfterQueueingOnTheLock() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + await using var viewer = new ViewerDataService(rig.Scratch.ConnectionString); + await SeedServerAsync(rig.Owner, 7371, "race-x", "race-x.example.test", ct); + + var race = await QueuedRaceAsync(rig, + async () => (await viewer.AddMonitoredServerAsync( + ViewerRow(ViewerDataService.ComputeServerId("race-added.example.test", null, false), "viewer-added", "race-added.example.test"), ct)).Outcome.ToString(), + () => ViewerEditAsync(viewer, ViewerRow(7371, "race-x", "race-moved.example.test"), ct), + ct); + + Assert.Equal(nameof(MonitoredServerAddOutcome.Added), race.FirstAnswer); + Assert.Equal("Updated", race.SecondAnswer); + + var keys = await StorageKeysAsync(rig.Owner, ct); + Assert.Equal(2, keys.Count); + Assert.Single(keys, key => key.Contains("race-added.example.test", StringComparison.OrdinalIgnoreCase)); + Assert.Single(keys, key => key.Contains("race-moved.example.test", StringComparison.OrdinalIgnoreCase)); + Assert.Equal(0, race.LocksLeftAfterwards); + Assert.True(race.BothQueuedOnTheLock, "Both the viewer's add and its edit should have queued behind the held identity lock."); + } + + /// + /// The startup seed's INSERT loop takes the identity lock, and reads the addresses again under it. The lock is held + /// from a second connection; the seed (two servers from darling.json, the second at address K) queues behind it and + /// writes nothing while it waits; the holder then commits a definition that an edit would have left at K under + /// another id, and releases the lock. The seed's own INSERT for K would land under hash(K), a different id, + /// so nothing but the re-read keeps the registry at one row for K. + /// + [Fact] + public async Task TheStartupSeed_QueuesOnTheLock_WritesNothingWhileItWaits_AndSkipsAnAddressClaimedMeanwhile() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + + var config = new DarlingConfig(); + config.Servers.Add(new MonitoredServer { Name = "seed-a", Host = "seed-a.example.test", Auth = "integrated" }); + config.Servers.Add(new MonitoredServer { Name = "seed-k", Host = AddressK, Auth = "integrated" }); + var provider = new StoreConfigProvider(rig.Owner); + + var holder = new NpgsqlConnection(rig.OwnerString); + NpgsqlTransaction? held = null; + var bodySucceeded = false; + try + { + await holder.OpenAsync(ct); + held = await holder.BeginTransactionAsync(ct); + await using (var take = new NpgsqlCommand(Edit.IdentityLockSql, holder, held)) + { + await take.ExecuteNonQueryAsync(ct); + } + + var seed = Task.Run(() => provider.SeedIfEmptyAsync(config, ct), ct); + Assert.True( + await WaitUntilAsync(async () => await AdvisoryLocksAsync(rig.Owner, false, ct) >= 1, TimeSpan.FromSeconds(30), ct), + "The seed should queue behind the held identity lock."); + Assert.Equal(0, await CountAsync(rig.Owner, "SELECT count(*) FROM config_monitored_servers", ct)); + + /* The definition an edit left at K under an older id, committed in the lock's own transaction. */ + await using (var claim = new NpgsqlCommand( + $@"INSERT INTO config_monitored_servers (server_id, name, host, auth, excluded_databases, is_enabled, monthly_cost_usd) + VALUES (7381, 'mover', '{AddressK}', 'integrated', ARRAY[]::text[], TRUE, 0)", holder, held)) + { + await claim.ExecuteNonQueryAsync(ct); + } + + await held.CommitAsync(ct); + await seed.WaitAsync(TimeSpan.FromSeconds(60), ct); + + var keys = await StorageKeysAsync(rig.Owner, ct); + Assert.Equal(2, keys.Count); + Assert.Equal(keys.Count, keys.Distinct(StringComparer.OrdinalIgnoreCase).Count()); + Assert.Single(keys, key => key.Contains(AddressK, StringComparison.OrdinalIgnoreCase)); + Assert.Single(keys, key => key.Contains("seed-a.example.test", StringComparison.OrdinalIgnoreCase)); + Assert.Equal(1, await CountAsync(rig.Owner, "SELECT count(*) FROM config_monitored_servers WHERE server_id = 7381 AND name = 'mover'", ct)); + Assert.Equal(0, await AdvisoryLocksAsync(rig.Owner, true, ct) + await AdvisoryLocksAsync(rig.Owner, false, ct)); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(bodySucceeded, async () => + { + if (held is not null) + { + await held.DisposeAsync(); + } + + await holder.DisposeAsync(); + }); + } + } + + /* ---------------- the database the probe reached (#5240, round 1) ---------------- + An edit or add that names no database is probed, and the probe says which database the connection reached. The + check before the write compares that database's key with the keys read BEFORE the probe; a definition committed + DURING the probe was invisible to it. The write is handed the same key and compares it under the lock. */ + + private const string AddressH = "race-h.example.test"; + + private static ConnectionProbeResult ReachedDatabase(string database) => Reached with { ConnectedDatabase = database }; + + private static string KeyOfHostAndDatabase(string host, string database) => + Edit.ParseRequest("[{\"host\":\"" + host + "\",\"database\":\"" + database + "\"}]").Entries.Single().StorageKey; + + /// + /// An edit whose probe lands in database X while a second writer (a real add) commits H/X during that probe is + /// refused under the lock with the same collides answer the check before the write gives, and the table is + /// left as the second writer made it. The edit that keeps its address (a changed certificate setting) is the case + /// the address re-check never covered: it handed the write no key at all. + /// + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task AnEditWhoseProbeLandsInADatabaseASecondWriterTookMeanwhile_IsRefusedUnderTheLock_AndSavesNothing(bool movesTheAddress) + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + var originalHost = movesTheAddress ? "race-h-old.example.test" : AddressH; + await SeedServerAsync(rig.Owner, 7401, "race-a", originalHost, ct); + + var secondWriter = ""; + Edit.ServerProbe probe = async (_, probeToken) => + { + secondWriter = await Edit.AddServersAsync( + rig.Owner, "[{\"host\":\"" + AddressH + "\",\"database\":\"X\"}]", ReachedAtOnce, probeToken); + return ReachedDatabase("X"); + }; + + var changes = movesTheAddress ? "{\"host\":\"" + AddressH + "\"}" : "{\"trust_server_certificate\":true}"; + var answer = Parse(await Edit.EditServerByNameAsync(rig.Owner, "race-a", changes, probe, TestKeyRings.Healthy, null, ct)); + + Assert.Equal("added", AddStatusOf(secondWriter)); + Assert.Equal("collides", answer["status"]!.GetValue()); + var message = answer["message"]!.GetValue(); + Assert.StartsWith("Not saved:", message, StringComparison.Ordinal); + Assert.Contains("lands in 'X'", message, StringComparison.Ordinal); + + var keys = await StorageKeysAsync(rig.Owner, ct); + Assert.Equal(2, keys.Count); + Assert.Single(keys, key => string.Equals(key, KeyOfHostAndDatabase(AddressH, "X"), StringComparison.OrdinalIgnoreCase)); + Assert.Equal(1, await CountAsync(rig.Owner, + "SELECT count(*) FROM config_monitored_servers WHERE server_id = 7401 AND host = '" + originalHost + "' AND database IS NULL AND trust_server_certificate = FALSE", ct)); + Assert.Equal(0, await AdvisoryLocksAsync(rig.Owner, true, ct) + await AdvisoryLocksAsync(rig.Owner, false, ct)); + } + + /// The add's twin of the edit fact above: the add names no database, its probe lands in X, and a second + /// writer commits H/X during the probe. The add is refused under the lock as collides and writes nothing; only + /// the second writer's definition collects X. + [Fact] + public async Task AnAddWhoseProbeLandsInADatabaseASecondWriterTookMeanwhile_IsRefusedUnderTheLock_AndSavesNothing() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + + var secondWriter = ""; + Edit.ServerProbe probe = async (_, probeToken) => + { + secondWriter = await Edit.AddServersAsync( + rig.Owner, "[{\"host\":\"" + AddressH + "\",\"database\":\"X\"}]", ReachedAtOnce, probeToken); + return ReachedDatabase("X"); + }; + + var answer = await Edit.AddServersAsync(rig.Owner, AddJson(AddressH), probe, ct); + + Assert.Equal("added", AddStatusOf(secondWriter)); + Assert.Equal("collides", AddStatusOf(answer)); + var detail = Parse(answer)["results"]![0]!["detail"]!.GetValue(); + Assert.StartsWith("Not added:", detail, StringComparison.Ordinal); + Assert.Contains("lands in 'X'", detail, StringComparison.Ordinal); + + var keys = await StorageKeysAsync(rig.Owner, ct); + Assert.Single(keys); + Assert.Equal(KeyOfHostAndDatabase(AddressH, "X"), keys[0], StringComparer.OrdinalIgnoreCase); + Assert.Equal(0, await AdvisoryLocksAsync(rig.Owner, true, ct) + await AdvisoryLocksAsync(rig.Owner, false, ct)); + } + + /// The control for the two refusals above: a probe that lands in a database nobody holds is saved, for an + /// add (stored under its declared key) and for an edit, so the re-check under the lock refuses only a database that + /// another definition really took. + [Fact] + public async Task AProbeThatLandsInADatabaseNobodyHolds_IsNotRefused_ForAnAddOrAnEdit() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpenAsync(ct); + Edit.ServerProbe probe = (_, _) => Task.FromResult(ReachedDatabase("X")); + + Assert.Equal("added", AddStatusOf(await Edit.AddServersAsync(rig.Owner, AddJson(AddressH), probe, ct))); + var edit = Parse(await Edit.EditServerByNameAsync(rig.Owner, AddressH, "{\"trust_server_certificate\":true}", probe, TestKeyRings.Healthy, null, ct)); + Assert.Equal("updated", edit["status"]!.GetValue()); + + var keys = await StorageKeysAsync(rig.Owner, ct); + Assert.Single(keys); + Assert.Equal(1, await CountAsync(rig.Owner, + "SELECT count(*) FROM config_monitored_servers WHERE host = '" + AddressH + "' AND database IS NULL AND trust_server_certificate = TRUE", ct)); + } +} diff --git a/Darling/Darling.Tests/ServerIdentityLockSourcePinTests.cs b/Darling/Darling.Tests/ServerIdentityLockSourcePinTests.cs new file mode 100644 index 0000000000..525d654ccc --- /dev/null +++ b/Darling/Darling.Tests/ServerIdentityLockSourcePinTests.cs @@ -0,0 +1,203 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Text.RegularExpressions; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// The desktop viewer takes the same server identity lock as the web and MCP writers (#5240, PR 2), and these facts +/// hold the parts of that a live race cannot: the viewer's COPY of the lock text is the service's text, each +/// identity write takes the lock first, in the transaction that writes, and the Add/Edit dialog shows a refused +/// (claimed) address as an expected refusal rather than an error. The live facts are in +/// ; these run without a store. +/// +/// The Viewer project cannot reference the service, so +/// is a copy of DarlingMcpServerAdminTools.IdentityLockSql. A copy that drifted would take a different advisory +/// key and serialise with nothing, and every write in the viewer would look locked while racing the web and MCP +/// writers freely. +/// +[Trait("Stage", "Guard")] +public sealed class ServerIdentityLockSourcePinTests +{ + private static readonly Regex LockLiteral = new( + @"const\s+string\s+\w*IdentityLock\w*Sql\s*=\s*""(?[^""]*)""", + RegexOptions.CultureInvariant); + + private static string ServiceSource() => + RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "Mcp", "DarlingMcpServerAdminTools.cs"); + + private static string ViewerSource() => + RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.MonitoredServers.cs"); + + private static string StoreConfigSource() => + RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "StoreConfigProvider.cs"); + + private static string DialogSource() => + RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "AddServerDialog.xaml.cs"); + + /// The text between two markers, which must both be there and in that order. + private static string Between(string source, string startMarker, string endMarker) + { + var start = source.IndexOf(startMarker, StringComparison.Ordinal); + Assert.True(start >= 0, "missing: " + startMarker); + var end = source.IndexOf(endMarker, start + startMarker.Length, StringComparison.Ordinal); + Assert.True(end > start, "missing, or not after the start: " + endMarker); + return source[start..end]; + } + + /// Asserts each needle is in , in the order given. + private static void AssertInOrder(string body, params string[] needles) + { + var at = -1; + foreach (var needle in needles) + { + var found = body.IndexOf(needle, at + 1, StringComparison.Ordinal); + Assert.True(found > at, "expected, after the step before it: " + needle); + at = found; + } + } + + [Fact] + public void TheViewersLockText_IsTheServicesLockText_InSourceAndAsCompiled() + { + var service = LockLiteral.Match(ServiceSource()); + var viewer = LockLiteral.Match(ViewerSource()); + + /* Positive control: both declarations are found, so an empty equals-empty cannot pass. */ + Assert.True(service.Success, "the service's lock constant was not found in DarlingMcpServerAdminTools.cs"); + Assert.True(viewer.Success, "the viewer's lock constant was not found in ViewerDataService.MonitoredServers.cs"); + Assert.Contains("pg_advisory_xact_lock", service.Groups["text"].Value, StringComparison.Ordinal); + + Assert.Equal(service.Groups["text"].Value, viewer.Groups["text"].Value); + Assert.Equal(DarlingMcpServerAdminTools.IdentityLockSql, ViewerDataService.MonitoredServerIdentityLockSql); + } + + [Fact] + public void TheViewersAdd_TakesTheLockAndReadsTheAddressesFirst_ThenInsertsAndCommitsInOneTransaction() + { + var body = Between( + ViewerSource(), + "public async Task AddMonitoredServerAsync(", + "public static MonitoredServerAddOutcome ClassifyAddAgainstOccupant("); + + AssertInOrder( + body, + "BeginTransactionAsync(", + "TakeIdentityLockAndFindClaimantAsync(", + "MonitoredServerAddOutcome.Duplicate", + "new NpgsqlCommand(MonitoredServerInsertIfAbsentSql, connection, transaction)", + "CommitAsync("); + Assert.DoesNotContain("_dataSource.CreateCommand(", body, StringComparison.Ordinal); + } + + [Fact] + public void TheViewersEdit_TakesTheLockAndReadsTheOtherAddressesFirst_ThenUpsertsAndCommitsInOneTransaction() + { + var body = Between( + ViewerSource(), + "public async Task UpsertMonitoredServerAsync(", + "private static string StorageKeyOf(MonitoredServerRow row)"); + + AssertInOrder( + body, + "BeginTransactionAsync(", + "TakeIdentityLockAndFindClaimantAsync(connection, transaction, row, row.ServerId, cancellationToken)", + "throw new MonitoredServerAddressClaimedException(", + "new NpgsqlCommand(MonitoredServerUpsertSql, connection, transaction)", + "CommitAsync("); + Assert.DoesNotContain("_dataSource.CreateCommand(", body, StringComparison.Ordinal); + } + + /// + /// A claimed address is an expected refusal of the dialog's save, like a read-only seat or a schema skew: the + /// dialog shows the exception's message as it is and logs nothing at error level. Without its own catch the + /// save handler's catch-all would prefix the message with "Error saving server:" and log an error for what is + /// the identity lock working as designed. + /// + [Fact] + public void TheDialogsSaveHandler_CatchesAClaimedAddressBeforeTheGeneralCatch_ShowsItAsIs_AndLogsNothing() + { + var handler = Between( + DialogSource(), + "private async void SaveButton_Click(", + "private async void TestConnectionButton_Click("); + + /* The type has its own catch ahead of the catch-all, and the catch-all is still where the error is logged: + the positive control that makes "this catch logs nothing" mean something. */ + AssertInOrder( + handler, + "catch (MonitoredServerAddressClaimedException ex)", + "catch (Exception ex)", + "ViewerLogger.Error("); + + /* The claimed-address catch's own block, as code: comments and literal text blanked, so its explanation + cannot satisfy or trip the checks below. */ + var claimedCatch = Between(handler, "catch (MonitoredServerAddressClaimedException ex)", "catch (Exception ex)"); + var code = CSharpSourceWalker.StripCommentsAndStrings(claimedCatch); + + AssertInOrder(code, "StatusText.Text = ex.Message;", "SaveButton.IsEnabled = true;"); + Assert.Empty(CSharpSourceWalker.StringLiteralBodies(claimedCatch)); + Assert.DoesNotContain("ViewerLogger", code, StringComparison.Ordinal); + } + + [Fact] + public void TheViewersInsertIfAbsent_IsAnIdentityWriteToo_LockFirstThenInsertInOneTransaction() + { + var body = Between( + ViewerSource(), + "public async Task InsertMonitoredServerIfAbsentAsync(", + "public async Task AddMonitoredServerAsync("); + + AssertInOrder( + body, + "BeginTransactionAsync(", + "TakeIdentityLockAndFindClaimantAsync(connection, transaction, row, null, cancellationToken)", + "new NpgsqlCommand(MonitoredServerInsertIfAbsentSql, connection, transaction)", + "CommitAsync("); + Assert.DoesNotContain("_dataSource.CreateCommand(", body, StringComparison.Ordinal); + } + + [Fact] + public void TheViewersHelper_TakesTheLockBeforeItReadsAnyAddress_BothOnTheCallersTransaction() + { + var body = Between( + ViewerSource(), + "private static async Task TakeIdentityLockAndFindClaimantAsync(", + "public async Task InsertMonitoredServerIfAbsentAsync("); + + AssertInOrder( + body, + "new NpgsqlCommand(MonitoredServerIdentityLockSql, connection, transaction)", + "new NpgsqlCommand(MonitoredServersSelectSql, connection, transaction)", + "StringComparison.OrdinalIgnoreCase"); + } + + [Fact] + public void TheStartupSeed_TakesTheSameLockFirst_ReadsTheHeldAddresses_ThenInsertsInOneTransaction() + { + var body = Between( + StoreConfigSource(), + "private static async Task SeedMonitoredServersAsync(", + "/* ---------------- read (store -> in-memory view) ---------------- */"); + + AssertInOrder( + body, + "BeginTransactionAsync(", + "DarlingMcpServerAdminTools.IdentityLockSql, connection, transaction", + "DarlingMcpServerAdminTools.ExistingServersSql, connection, transaction", + "heldAddresses.Add(server.StorageName)", + "INSERT INTO config_monitored_servers", + "ON CONFLICT (server_id) DO NOTHING\", connection, transaction)", + "CommitAsync("); + } +} diff --git a/Darling/Darling.Tests/ServerInventoryFleetMetricsLiveTests.cs b/Darling/Darling.Tests/ServerInventoryFleetMetricsLiveTests.cs index c6b19d22bb..6380bd9e88 100644 --- a/Darling/Darling.Tests/ServerInventoryFleetMetricsLiveTests.cs +++ b/Darling/Darling.Tests/ServerInventoryFleetMetricsLiveTests.cs @@ -168,6 +168,15 @@ watermarkTarget on the first run of this test. */ await InsertQueryStatsAsync(connection, ServerId, ServerName, "TailDb", watermarkTarget.AddMinutes(10), 2, ct); // NeverExecutedDb: no query_stats rows at all. + /* The idle coverage rule: 7 days of history and a sample on each complete UTC day before "now" (idleCutoff + 7 days, so + 2020-06-01 through 2020-06-07). One zero-execution sample before the cutoff hour (read raw, as the rollup may not hold + it) and one at noon on each day; the days after the watermark are read raw, the first day from the rollup. */ + await FinOpsIdleCoverageSeed.InsertAsync(connection, ct, ServerId, ServerName, idleCutoff.AddHours(-1), "oldest"); + for (var day = 0; day < 7; day++) + { + await FinOpsIdleCoverageSeed.InsertAsync(connection, ct, ServerId, ServerName, anchorHour.Date.AddDays(day).AddHours(12), "day" + day); + } + /* The "known databases" universe: one size snapshot per candidate, all at the SAME collection_time so the latest-snapshot join (exact timestamp equality) picks up every one of them together. */ var snapshotTime = watermarkTarget.AddMinutes(20); @@ -256,6 +265,7 @@ await ExecAsync(connection, "INSERT INTO servers (server_id, server_name) VALUES await InsertDatabaseSizeAsync(connection, ServerA, "old-new-A", "StaleDb", now.AddDays(-2), 9999, ct); // not latest — must be ignored await InsertQueryStatsAsync(connection, ServerA, "old-new-A", "AppDb1", now.AddDays(-2), 5, ct); // AppDb2: no executions -> idle. + await FinOpsIdleCoverageSeed.SeedAsync(connection, ct, ServerA, "old-new-A", now); // Server B: both databases active, no memory-grant evidence at all (grants CTE empty, must COALESCE to 0). await InsertCpuAsync(connection, ServerB, now.AddHours(-1), 20, ct); @@ -265,14 +275,17 @@ await ExecAsync(connection, "INSERT INTO servers (server_id, server_name) VALUES await InsertDatabaseSizeAsync(connection, ServerB, "old-new-B", "Db2", now.AddMinutes(-15), 150, ct); await InsertQueryStatsAsync(connection, ServerB, "old-new-B", "Db1", now.AddDays(-1), 2, ct); await InsertQueryStatsAsync(connection, ServerB, "old-new-B", "Db2", now.AddDays(-6), 7, ct); // inside the 7d window + await FinOpsIdleCoverageSeed.SeedAsync(connection, ct, ServerB, "old-new-B", now); // Server C: both databases idle, no CPU/memory evidence at all (every left-joined CTE empty). await InsertDatabaseSizeAsync(connection, ServerC, "old-new-C", "Db1", now.AddMinutes(-45), 50, ct); await InsertDatabaseSizeAsync(connection, ServerC, "old-new-C", "Db2", now.AddMinutes(-45), 75, ct); + await FinOpsIdleCoverageSeed.SeedAsync(connection, ct, ServerC, "old-new-C", now); // Server D: low CPU and nothing else — the verdict every server with CPU samples kept: OVER_PROVISIONED. await InsertCpuAsync(connection, ServerD, now.AddHours(-1), 5, ct); await InsertCpuAsync(connection, ServerD, now.AddHours(-2), 7, ct); + await FinOpsIdleCoverageSeed.SeedAsync(connection, ct, ServerD, "old-new-D", now); await using var viewer = new ViewerDataService(scratch.ConnectionString); var metrics = await viewer.GetServerMetricsAsync(ct); @@ -359,6 +372,10 @@ private static async Task ReadIdleDbCountAsync( await using var command = dataSource.CreateCommand(sql); command.Parameters.Add(new NpgsqlParameter { TypedValue = idleCutoff }); // $1 cpu cutoff — unused by this test command.Parameters.Add(new NpgsqlParameter { TypedValue = idleCutoff }); // $2 idle cutoff + var now = idleCutoff.AddDays(7); + command.Parameters.Add(new NpgsqlParameter { TypedValue = PerformanceMonitor.Darling.Storage.FinOps.DarlingFinOpsOptimizationReader.IdleCoverageStartUtc(now) }); // $3 first complete day + command.Parameters.Add(new NpgsqlParameter { TypedValue = PerformanceMonitor.Darling.Storage.FinOps.DarlingFinOpsOptimizationReader.IdleCoverageDays }); // $4 days + command.Parameters.Add(new NpgsqlParameter { TypedValue = PerformanceMonitor.Darling.Storage.FinOps.DarlingFinOpsOptimizationReader.IdleCoverageEndUtc(now) }); // $5 today await using var reader = await command.ExecuteReaderAsync(ct); while (await reader.ReadAsync(ct)) { diff --git a/Darling/Darling.Tests/ServerInventoryFleetMetricsTests.cs b/Darling/Darling.Tests/ServerInventoryFleetMetricsTests.cs index 15eb639f7f..7ea3aafe38 100644 --- a/Darling/Darling.Tests/ServerInventoryFleetMetricsTests.cs +++ b/Darling/Darling.Tests/ServerInventoryFleetMetricsTests.cs @@ -86,10 +86,15 @@ NULL through the outer LEFT JOIN (the bug the live parity test caught). */ Assert.Contains("FILTER (WHERE ad.database_name IS NULL)", sql, StringComparison.Ordinal); Assert.Contains("delta_execution_count > 0", sql, StringComparison.Ordinal); - /* Only two binds now: cpu cutoff, idle cutoff — server_id is gone. */ + /* Five binds: cpu cutoff, idle cutoff, the idle-coverage start, the days it must cover and the coverage end (today 00:00, exclusive) — server_id is gone. A server without + a query-stats sample on each of those days has no idle_dbs row, so its count is NULL (a dash), not a count from unwatched days. */ Assert.Contains("$1", sql, StringComparison.Ordinal); Assert.Contains("$2", sql, StringComparison.Ordinal); - Assert.DoesNotContain("$3", sql, StringComparison.Ordinal); + Assert.Contains("$3", sql, StringComparison.Ordinal); + Assert.Contains("$4", sql, StringComparison.Ordinal); + Assert.Contains("$5", sql, StringComparison.Ordinal); + Assert.DoesNotContain("$6", sql, StringComparison.Ordinal); + Assert.Contains("JOIN idle_coverage ic ON ic.server_id = s.server_id", sql, StringComparison.Ordinal); } [Fact] @@ -109,7 +114,7 @@ public void ServerMetricsSqlFor_RoutesTheIdleCheckThroughTheStitchedRollup_PlusT Assert.Contains("execution_count_sum > 0", sql, StringComparison.Ordinal); Assert.Contains("bucket >=", sql, StringComparison.Ordinal); var unionCount = System.Text.RegularExpressions.Regex.Matches(sql, "UNION").Count; - Assert.Equal(2, unionCount); // three arms, two UNIONs joining them + Assert.Equal(3, unionCount); // activity: three arms, two UNIONs; idle coverage: the rollup days UNION the raw days after the watermark /* The two edge literals: the ceiling hour of the idle cutoff (05:00 — 04:27 rounds UP to the next whole hour, since the partial hour [04:27, 05:00) is exactly the slice the rollup can't answer) diff --git a/Darling/Darling.Tests/ServerLocalReadFrameDisciplineTests.cs b/Darling/Darling.Tests/ServerLocalReadFrameDisciplineTests.cs index bce97994ef..a326878c4a 100644 --- a/Darling/Darling.Tests/ServerLocalReadFrameDisciplineTests.cs +++ b/Darling/Darling.Tests/ServerLocalReadFrameDisciplineTests.cs @@ -50,6 +50,7 @@ namespace Darling.Tests; /// additionally asserts its ANCHOR is present before judging the file: the bare-read check keys on the /// dte alias, so a rename would otherwise turn it into an assertion about nothing. /// +[Trait("Reads", "Lite")] public sealed class ServerLocalReadFrameDisciplineTests { /* ───────────────────────── the literal-SQL reads of default_trace_events ───────────────────────── */ @@ -300,12 +301,14 @@ public void CompiledAnnotationSql_ScopesTheServerClockRead_ToThePanelsOwnServerA var servers = new[] { "SERVER-A", "SERVER-B" }; var clockRead = ComposeCompiler.CompileServerClockRead(RunContext(servers)); - Assert.Contains("AND server_name = ANY($1)", clockRead.Sql, StringComparison.Ordinal); + Assert.Contains("AND server_id = ANY(ARRAY(SELECT reg.server_id FROM collect.servers AS reg WHERE reg.server_name = ANY($1)))", clockRead.Sql, StringComparison.Ordinal); Assert.Single(clockRead.Parameters); var scoped = CompiledAnnotation(serverLocal, servers); - Assert.Contains("f.server_name = ANY($3)", scoped.Sql, StringComparison.Ordinal); - Assert.Single(Regex.Matches(scoped.Sql, @"ANY\(")); + Assert.Contains("f.server_id = ANY(ARRAY(SELECT reg.server_id FROM collect.servers AS reg WHERE reg.server_name = ANY($3)))", scoped.Sql, StringComparison.Ordinal); + /* #5525: the id list is resolved from the names inside the SQL, so the scope spells ANY( twice and still binds the names once. */ + Assert.Equal(2, Regex.Matches(scoped.Sql, @"ANY\(").Count); + Assert.Single(Regex.Matches(scoped.Sql, @"\$3(?!\d)")); Assert.Equal(7, scoped.Parameters.Count); Assert.DoesNotContain("SERVER-A", scoped.Sql, StringComparison.Ordinal); diff --git a/Darling/Darling.Tests/ServerPageTabsTests.cs b/Darling/Darling.Tests/ServerPageTabsTests.cs index 99a3395bc7..01f75b2ea5 100644 --- a/Darling/Darling.Tests/ServerPageTabsTests.cs +++ b/Darling/Darling.Tests/ServerPageTabsTests.cs @@ -127,6 +127,18 @@ it contributes nothing to the auroraOnly set below. It is mapped anyway because /// which is the point. A comment that names a read the dispatch no longer has is stale documentation about /// the one thing here that is impossible to verify by eye. /// + [Fact] + public void TheQueryStoreTopGrid_DrawsTheApproximationNote_AsAFurtherNote() + { + var js = ServerTabsJs; + var at = js.IndexOf("\"get_query_store_top\",", StringComparison.Ordinal); + Assert.True(at >= 0); + var end = js.IndexOf("\n ),", at, StringComparison.Ordinal); + var call = js[at..end]; + Assert.Contains("\"truncation_note\",", call, StringComparison.Ordinal); + Assert.Contains("[\"approximation_note\"]", call, StringComparison.Ordinal); + } + [Fact] public void EveryReadTheServerPageNames_ExistsInTheDispatch() { @@ -794,7 +806,7 @@ correct without it until two renders overlap. */ the render. A repeat render paints from the remembered card synchronously; a card that does not arrive leaves a painted page alone, because a failed fleet read is not evidence the engine changed. */ Assert.Contains("const remembered = lastCard.get(server);", ServerJs, StringComparison.Ordinal); - Assert.Contains("painted = paintTabs(tabsSlot, server, tabId, remembered.card);", ServerJs, StringComparison.Ordinal); + Assert.Contains("painted = paintTabsKeeping(keep, tabsSlot, server, tabId, remembered.card);", ServerJs, StringComparison.Ordinal); Assert.Contains("if (card) lastCard.set(server, { card, reason });", ServerJs, StringComparison.Ordinal); Assert.Contains("if (!painted || (card && serverTabsFor(card) !== painted)) {", ServerJs, StringComparison.Ordinal); Assert.DoesNotContain("card.is_postgres", ServerJs, StringComparison.Ordinal); @@ -885,8 +897,8 @@ public void TheTabsThatCannotDoWhatTheDesktopDoes_SaySo() Assert.Contains("block-chain view", js, StringComparison.Ordinal); /* The note renders — a `note` field with no renderer is the same silence in a different place. */ - Assert.Contains("return tab.note ? noticeStrip(tab.note) : null;", js, StringComparison.Ordinal); - Assert.Contains("tabNote(tab)", ServerJs, StringComparison.Ordinal); + Assert.Contains("return note ? noticeStrip(note) : null;", js, StringComparison.Ordinal); + Assert.Contains("tabNote(tab, tabReach())", ServerJs, StringComparison.Ordinal); } /// @@ -922,11 +934,12 @@ public void EveryDataPanel_ExplainsItsOwnEmptyState() /* The WHOLE signature, so emptyText is asserted to be a declared parameter rather than something read off an options object. #3278 appended `noteKey = null` - an opt-in server-supplied caveat, unrelated to this guard - and #4925 appended `moreNoteKeys = null` after it (further caveat fields - rendered the same way). The literal is spelled out here rather than truncated at emptyText + rendered the same way), and #4843 appended `columnGroups = null` (the opt-in column groups of a wide grid), and #5226 appended `control = null` (a node drawn under the title: + the ranking selector of the Top Queries / Top Procedures cards), and #5245 appended `dbScope = null` (the panel's own database-scope chip, for a panel whose scope differs from its read's class; unrelated to this guard)., and the release walk appended `extensionMissingLine = null` (the plain line a precondition panel shows when its optional extension is not installed; unrelated to this guard). The literal is spelled out here rather than truncated at emptyText because a prefix match would stop noticing a parameter inserted BEFORE it. */ Assert.Contains( "function table(title, read, params, rowsKey, columns, subtitle, emptyText, span = 2, " - + "noteKey = null, moreNoteKeys = null)", + + "noteKey = null, moreNoteKeys = null, columnGroups = null, control = null, dbScope = null, extensionMissingLine = null)", js, StringComparison.Ordinal); Assert.Contains( @@ -1209,7 +1222,7 @@ private static Dictionary DescriptorArraysIn(string js) /// private static IEnumerable<(string Read, string[] Arrays)> DescriptorArraysBoundIn(string js) { - foreach (Match open in Regex.Matches(js, @"(? pattern). The browser tab title is the static "Darling Web" in /// index.html and no script sets document.title, so there is no second place that printed the key. /// +[Trait("Stage", "Guard")] public sealed class ServerPageTitlePinTests { private static string ServerJs => ReadRepoFileLf(Path.Combine( @@ -81,7 +82,7 @@ public void TheRouteKey_StaysTheIdentity_OnlyTheHeadingChanges() Assert.Contains("const matches = (c) => c.server_name === server || c.display_name === server;", js, StringComparison.Ordinal); Assert.Contains("current = { server, tab: null };", js, StringComparison.Ordinal); - Assert.Contains("mount(main, [head, whySlot, tabsSlot, gridNode]);", js, StringComparison.Ordinal); + Assert.Contains("mount(main, [head, whySlot, tabsSlot, reachSlot, gridNode]);", js, StringComparison.Ordinal); Assert.DoesNotContain("server = card.display_name", js, StringComparison.Ordinal); Assert.DoesNotContain("current = { server: card.display_name", js, StringComparison.Ordinal); } diff --git a/Darling/Darling.Tests/ServerPasswordFromRequestTests.cs b/Darling/Darling.Tests/ServerPasswordFromRequestTests.cs new file mode 100644 index 0000000000..c36cb1fd11 --- /dev/null +++ b/Darling/Darling.Tests/ServerPasswordFromRequestTests.cs @@ -0,0 +1,586 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Net; +using System.Net.Http; +using System.Text; +using System.Text.Json; +using System.Text.Json.Nodes; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.TestHost; +using Microsoft.Extensions.Logging; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Hosting; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; +using Core = PerformanceMonitor.Darling.Service.Mcp.DarlingMcpServerAdminTools; + +namespace Darling.Tests; + +/// +/// The password a request carries is the password itself. The web add and edit routes, the MCP add and edit tools and +/// the test_connect command all ask one question of it (), +/// and it is the resolver's own question, so what is refused is exactly what would be dereferenced. +/// +/// An edit that sends the password blank keeps whatever is stored, and a test of a stored value only runs at the +/// address that stored it. Each request path is exercised through the real core, with a probe that counts its calls. +/// +[Collection("darling-owned-secrets")] +public sealed class ServerPasswordFromRequestTests : IDisposable +{ + private const string Typed = "Typed-Pw-Q7-7XK"; + private const string StoredReference = "env:SYNTH_STORED_PASSWORD_REF"; + private static readonly DateTime Stamp = new(2026, 10, 5, 12, 0, 0, 123, DateTimeKind.Unspecified); + + private readonly DarlingOwnedSet _ownedBefore = DarlingOwnedSecrets.Current; + + public ServerPasswordFromRequestTests() => DarlingOwnedSecrets.Set(DarlingOwnedSet.Empty); + + public void Dispose() => DarlingOwnedSecrets.Set(_ownedBefore); + + /* Every shape the resolver dereferences: the prefix at the very start, any text after it (the name or path is trimmed). */ + public static TheoryData References => new() + { + "env:X", + "env:SOME_VARIABLE", + "env: SPACED_NAME ", + "file:C:\\x", + "file:C:\\secrets\\sql.txt", + "file:/run/secrets/sql_password", + "file: C:\\spaced path\\x ", + }; + + private static string RefusalSentence => DarlingSecretSource.RequestReferenceRefusalText; + + /* ---------------- the one predicate ---------------- */ + + [Theory] + [MemberData(nameof(References))] + public void TheRefusal_IsExactlyWhatTheResolverWouldDereference(string reference) + { + Assert.True(DarlingSecretSource.IsReference(reference)); + Assert.Equal(RefusalSentence, DarlingSecretSource.RequestReferenceRefusal(reference)); + Assert.DoesNotContain(reference.Trim(), RefusalSentence, StringComparison.Ordinal); + } + + [Theory] + [InlineData("hunter2")] + [InlineData("pa-env:-ss")] + [InlineData("my file:pw")] + [InlineData("xenv:X")] + [InlineData(" env:X")] + [InlineData("ENV:X")] + [InlineData("File:C:\\x")] + [InlineData("")] + [InlineData(" ")] + public void ALiteral_IsNotRefused_BecauseTheResolverWouldNotDereferenceIt(string literal) + { + Assert.False(DarlingSecretSource.IsReference(literal)); + Assert.Null(DarlingSecretSource.RequestReferenceRefusal(literal)); + Assert.Equal(literal, DarlingSecretSource.Resolve(literal, "test.setting")); + } + + [Fact] + public void TheNullPassword_IsNotRefused() => Assert.Null(DarlingSecretSource.RequestReferenceRefusal(null)); + + /* ---------------- MCP add_servers (and the web add: the route hands the body to this same core) ---------------- */ + + private static string AddBody(string password, string auth = "SQL") => + JsonSerializer.Serialize(new[] { new { host = "sql01", auth, username = "monitor", password } }); + + private sealed class UntouchedDefinitions : Core.IServerDefinitions + { + public Task> LoadStorageKeysAsync(CancellationToken cancellationToken) => + throw new InvalidOperationException("the store must not be opened for a refused request"); + + public Task ReadStorageKeyAsync(int serverId, CancellationToken cancellationToken) => + throw new InvalidOperationException("the store must not be read for a refused request"); + + public Task InsertAsync(Core.ParsedServerEntry entry, string? encryptedPassword, string? actualStorageKey, CancellationToken cancellationToken) => + throw new InvalidOperationException("nothing may be written for a refused request"); + } + + private sealed class CountingProbe + { + public int Calls { get; private set; } + + public MonitoredServer? Last { get; private set; } + + public Core.ServerProbe Probe => (server, _) => + { + Calls++; + Last = server; + return Task.FromResult(new ConnectionProbeResult(true, 15, 3, "Enterprise", false, false, false, true, null)); + }; + } + + [Theory] + [MemberData(nameof(References))] + public async Task McpAdd_RefusesAReference_WithTheSentence_NoEcho_NoProbe_NoStoreAccess(string reference) + { + foreach (var auth in new[] { "SQL", "ServicePrincipal" }) + { + var probe = new CountingProbe(); + + var answer = await Core.AddServersAsync(new UntouchedDefinitions(), AddBody(reference, auth), probe.Probe, CancellationToken.None); + + var node = JsonNode.Parse(answer)!; + Assert.Equal(1, node["failed"]!.GetValue()); + Assert.Equal(0, node["added"]!.GetValue()); + var result = node["results"]![0]!; + Assert.Equal("invalid", result["status"]!.GetValue()); + Assert.Equal(RefusalSentence, result["detail"]!.GetValue()); + Assert.DoesNotContain(reference.Trim(), answer, StringComparison.Ordinal); + Assert.Equal(0, probe.Calls); + } + } + + [Fact] + public void McpAdd_RefusesTheReference_OnEveryPlatform_BeforeTheLiteralRule() + { + foreach (var isWindows in new[] { true, false }) + { + var (entries, invalid, whole) = Core.ParseRequest(AddBody("file:C:\\x"), isWindows ? TestKeyRings.Healthy.Status : TestKeyRings.NotReady.Status); + + Assert.Null(whole); + Assert.Empty(entries); + Assert.Equal(RefusalSentence, Assert.Single(invalid).Detail); + } + } + + [Fact] + public void McpAdd_AcceptsAPassword_ThatOnlyContainsAPrefixInTheMiddle() + { + var (entries, invalid, _) = Core.ParseRequest(AddBody("pa-env:-ss"), TestKeyRings.Healthy.Status); + + Assert.Empty(invalid); + Assert.Equal("pa-env:-ss", Assert.Single(entries).PlaintextPassword); + } + + [Fact] + public void TheHostCommandLineAdd_StillAcceptsAReference_ThatIsNotDarlingsOwn() + { + var (entries, invalid, _) = Core.ParseRequest(AddBody("env:SOME_OTHER_VAR"), TestKeyRings.NotReady.Status, allowSecretReferences: true); + + Assert.Empty(invalid); + Assert.Equal("env:SOME_OTHER_VAR", Assert.Single(entries).PlaintextPassword); + } + + [Fact] + public void ARequestAdd_OfALiteral_WhileTheKeyIsNotReady_GivesTheKeysReason_NotAReference() + { + var (_, invalid, _) = Core.ParseRequest(AddBody(Typed), TestKeyRings.NotReady.Status); + + var detail = Assert.Single(invalid).Detail; + Assert.Contains(DarlingPasswordKey.NotReadyReason, detail, StringComparison.Ordinal); + Assert.DoesNotContain("env:", detail, StringComparison.Ordinal); + Assert.DoesNotContain(Typed, detail, StringComparison.Ordinal); + } + + /* ---------------- MCP edit_server (and the web edit: the route hands the body to this same core) ---------------- */ + + private static Core.ServerEditRow Row(string host = "alpha-01.example.test", string auth = "sql", string? database = null) => new( + ServerId: 41, Name: "alpha-01", Host: host, Port: 0, Database: database, ReadOnlyIntent: false, Engine: "sqlserver", + Auth: auth, Username: auth == "integrated" ? null : "monitor", EncryptMode: "Mandatory", TrustServerCertificate: false, + MultiSubnetFailover: false, MonthlyCostUsd: 10m, ModifiedAt: Stamp.AddTicks(7)); + + /* A store whose row holds a REFERENCE in the secret slot, as an installation seeded from the configuration file does. The + secret column is the one thing this fake keeps: a write that does not name it leaves the reference byte for byte. */ + private sealed class StoreHoldingAReference : Core.IServerEditStore + { + public Core.ServerEditRow Row { get; set; } = ServerPasswordFromRequestTests.Row(); + + public string? StoredSecret { get; private set; } = StoredReference; + + public int Writes { get; private set; } + + public Task ReadRowAsync(int serverId, CancellationToken cancellationToken) => Task.FromResult(Row); + + public Task> LoadOtherStorageKeysAsync(int serverId, CancellationToken cancellationToken) => Task.FromResult(new List()); + + public Task WriteAsync( + int serverId, DateTime expectedModifiedAt, IReadOnlyList sets, string? newStorageKey, string? actualStorageKey, CancellationToken cancellationToken) + { + Writes++; + if (sets.FirstOrDefault(s => s.Column == "encrypted_password") is { } secret) + { + StoredSecret = (string?)secret.Value; + } + + return Task.FromResult(new Core.ServerEditWrite(Core.ServerEditWriteKind.Written, Stamp.AddSeconds(1))); + } + } + + private static Task Edit(StoreHoldingAReference store, string changes, CountingProbe probe) => + Core.EditServerCoreAsync(store, 41, changes, probe.Probe, TestKeyRings.Healthy, logger: null, CancellationToken.None); + + private static string Status(string answer) => JsonNode.Parse(answer)!["status"]!.GetValue(); + + [Theory] + [MemberData(nameof(References))] + public async Task McpEdit_RefusesATypedReference_WithTheSentence_NoEcho_NoProbe_NoWrite(string reference) + { + foreach (var body in new[] + { + "{\"host\":\"beta.example.test\",\"password\":" + JsonSerializer.Serialize(reference) + "}", + "{\"display_name\":\"Renamed\",\"password\":" + JsonSerializer.Serialize(reference) + "}", + }) + { + var store = new StoreHoldingAReference(); + var probe = new CountingProbe(); + + var answer = await Edit(store, body, probe); + + Assert.Equal("invalid", Status(answer)); + Assert.Equal(RefusalSentence, JsonNode.Parse(answer)!["message"]!.GetValue()); + Assert.DoesNotContain(reference.Trim(), answer, StringComparison.Ordinal); + Assert.Equal(0, probe.Calls); + Assert.Equal(0, store.Writes); + Assert.Equal(StoredReference, store.StoredSecret); + } + } + + [Fact] + public async Task McpEdit_AcceptsATypedPassword_ThatOnlyContainsAPrefixInTheMiddle() + { + var store = new StoreHoldingAReference(); + var probe = new CountingProbe(); + + var answer = await Edit(store, "{\"display_name\":\"Renamed\",\"password\":\"pa-env:-ss\"}", probe); + + Assert.Equal("updated", Status(answer)); + Assert.Equal("pa-env:-ss", probe.Last!.Password); + } + + [Theory] + [InlineData("{\"display_name\":\"Renamed\"}")] + [InlineData("{\"display_name\":\"Renamed\",\"password\":\"\"}")] + [InlineData("{\"monthly_cost_usd\":99,\"password\":\"\"}")] + public async Task McpEdit_WithTheStoredSecretAReference_AbsentOrBlankKeepsItByteForByte(string body) + { + var store = new StoreHoldingAReference(); + var probe = new CountingProbe(); + + var answer = await Edit(store, body, probe); + + Assert.Equal("updated", Status(answer)); + Assert.Equal(1, store.Writes); + Assert.Equal(StoredReference, store.StoredSecret); + Assert.Equal(0, probe.Calls); + } + + [Fact] + public async Task McpEdit_ATypedLiteral_ReplacesTheStoredReference() + { + var store = new StoreHoldingAReference(); + var probe = new CountingProbe(); + + var answer = await Edit(store, "{\"display_name\":\"Renamed\",\"password\":\"" + Typed + "\"}", probe); + + Assert.Equal("updated", Status(answer)); + Assert.NotEqual(StoredReference, store.StoredSecret); + var row = store.Row; + var binding = PasswordBinding.ForServer(ServerConnectionIdentity.FromStoredColumns( + row.Host, row.Port, row.Engine, row.Database, row.ReadOnlyIntent, row.Auth, row.Username, row.EncryptMode, + row.TrustServerCertificate, row.MultiSubnetFailover)); + Assert.Equal(Typed, TestKeyRings.Healthy.Open(store.StoredSecret!, binding)); + Assert.Equal(Typed, probe.Last!.Password); + Assert.DoesNotContain(Typed, answer, StringComparison.Ordinal); + } + + /* The stored reference is never read back by this surface, so a test that has no typed password cannot use it: any change to + how the server is reached answers with the plain ask, and nothing is probed. */ + [Theory] + [InlineData("{\"host\":\"elsewhere.example.test\"}")] + [InlineData("{\"host\":\"elsewhere.example.test\",\"password\":\"\"}")] + [InlineData("{\"host\":\"alpha-01.example.test\\\\OTHERINSTANCE\"}")] + [InlineData("{\"database\":\"Other\"}")] + [InlineData("{\"username\":\"someone_else\"}")] + public async Task McpEdit_AChangeOfHowTheServerIsReached_WithNoTypedPassword_AsksForIt_AndProbesNothing(string body) + { + var store = new StoreHoldingAReference(); + var probe = new CountingProbe(); + + var answer = await Edit(store, body, probe); + + Assert.Equal("invalid", Status(answer)); + Assert.Equal(Core.EditPasswordNeededText, JsonNode.Parse(answer)!["message"]!.GetValue()); + Assert.Equal(0, probe.Calls); + Assert.Equal(0, store.Writes); + Assert.Equal(StoredReference, store.StoredSecret); + } + + /* ---------------- the web routes, with the real cores behind them ---------------- */ + + private sealed class Web : IAsyncDisposable + { + public required WebApplication App { get; init; } + + public required HttpClient Client { get; init; } + + public required NpgsqlDataSource Source { get; init; } + + public required CountingProbe Probe { get; init; } + + public required StoreHoldingAReference Store { get; init; } + + public async ValueTask DisposeAsync() + { + Client.Dispose(); + await App.DisposeAsync(); + await Source.DisposeAsync(); + } + } + + private static async Task StartWebAsync() + { + var builder = WebApplication.CreateBuilder(); + builder.WebHost.UseTestServer(); + builder.Logging.ClearProviders(); + var app = builder.Build(); + var source = NpgsqlDataSource.Create("Host=localhost;Database=never_opened;Username=nobody"); + var probe = new CountingProbe(); + var store = new StoreHoldingAReference(); + + app.Use(async (context, next) => + { + context.Items[DarlingWebSeat.HttpContextItemKey] = new DarlingWebSeat("alice", true); + await next(context); + }); + DarlingWebEndpoints.MapServers( + app, source, new CapturingTestLogger(), + addServers: body => Core.AddServersAsync(new UntouchedDefinitions(), body, probe.Probe, CancellationToken.None), + editServer: (id, body) => Core.EditServerCoreAsync(store, id, body, probe.Probe, TestKeyRings.Healthy, logger: null, CancellationToken.None)); + await app.StartAsync(TestContext.Current.CancellationToken); + return new Web { App = app, Client = app.GetTestClient(), Source = source, Probe = probe, Store = store }; + } + + private static async Task<(HttpStatusCode Status, string Body)> SendAsync(Web web, HttpMethod method, string path, string body) + { + var ct = TestContext.Current.CancellationToken; + using var request = new HttpRequestMessage(method, path) { Content = new StringContent(body, Encoding.UTF8, "application/json") }; + using var response = await web.Client.SendAsync(request, ct); + return (response.StatusCode, await response.Content.ReadAsStringAsync(ct)); + } + + [Theory] + [MemberData(nameof(References))] + public async Task WebAdd_RefusesAReference_WithTheSentence_NoEcho_NoProbe(string reference) + { + await using var web = await StartWebAsync(); + + var (status, body) = await SendAsync(web, HttpMethod.Post, "/api/servers", AddBody(reference)); + + Assert.Contains(RefusalSentence, body, StringComparison.Ordinal); + Assert.DoesNotContain(reference.Trim(), body, StringComparison.Ordinal); + Assert.Equal("invalid", JsonNode.Parse(body)!["results"]![0]!["status"]!.GetValue()); + Assert.True(status is HttpStatusCode.OK or HttpStatusCode.BadRequest); + Assert.Equal(0, web.Probe.Calls); + } + + [Theory] + [MemberData(nameof(References))] + public async Task WebEdit_RefusesAReference_WithTheSentence_NoEcho_NoProbe_NoWrite(string reference) + { + await using var web = await StartWebAsync(); + var json = "{\"display_name\":\"Renamed\",\"password\":" + JsonSerializer.Serialize(reference) + + ",\"expected_modified_at\":\"" + Core.ModifiedAtToken(Stamp.AddTicks(7)) + "\"}"; + + var (_, body) = await SendAsync(web, HttpMethod.Patch, "/api/servers/41", json); + + Assert.Contains(RefusalSentence, body, StringComparison.Ordinal); + Assert.DoesNotContain(reference.Trim(), body, StringComparison.Ordinal); + Assert.Equal(0, web.Probe.Calls); + Assert.Equal(0, web.Store.Writes); + Assert.Equal(StoredReference, web.Store.StoredSecret); + } + + [Fact] + public async Task WebEdit_ABlankPassword_KeepsTheStoredReference() + { + await using var web = await StartWebAsync(); + var json = "{\"display_name\":\"Renamed\",\"password\":\"\",\"expected_modified_at\":\"" + Core.ModifiedAtToken(Stamp.AddTicks(7)) + "\"}"; + + var (status, body) = await SendAsync(web, HttpMethod.Patch, "/api/servers/41", json); + + Assert.Equal(HttpStatusCode.OK, status); + Assert.Equal("updated", Status(body)); + Assert.Equal(StoredReference, web.Store.StoredSecret); + Assert.Equal(0, web.Probe.Calls); + } + + /* ---------------- the test_connect command ---------------- */ + + private static MonitoredServer Tested(string host, string? encrypted, int port = 0, string? password = null) => new() + { + Name = "alpha-01", Host = host, Port = port, Auth = "sql", Username = "monitor", EncryptedPassword = encrypted, Password = password, + }; + + /// The stored row the tested server () matches on every setting. + private static ServerConnectionIdentity StoredRow(string host, int port = 0) => + ServerConnectionIdentity.FromStoredColumns(host, port, "sqlserver", null, false, "sql", "monitor", "Mandatory", false, false); + + private sealed class StoredAddresses + { + public List Rows { get; } = []; + + public List Asked { get; } = []; + + public Task> ReadAsync(string reference, CancellationToken cancellationToken) + { + Asked.Add(reference); + return Task.FromResult>(Rows.ToList()); + } + } + + [Fact] + public async Task TestConnect_AStoredReference_IsResolvedOnlyWhereHostPortAndInstanceAllMatch() + { + var stored = new StoredAddresses(); + stored.Rows.Add(StoredRow("alpha-01.example.test\\INST1")); + var ct = CancellationToken.None; + + Assert.Null(await DarlingCommandExecutor.TestConnectReferenceRefusalAsync( + Tested("alpha-01.example.test\\INST1", StoredReference), stored.ReadAsync, ct)); + + foreach (var (host, port) in new[] + { + ("elsewhere.example.test\\INST1", 0), + ("alpha-01.example.test\\INST2", 0), + ("alpha-01.example.test", 0), + ("alpha-01.example.test\\INST1", 1434), + ("ALPHA-01.example.test\\INST1", 0), + }) + { + Assert.Equal( + DarlingCommandExecutor.TestConnectPasswordNeededText, + await DarlingCommandExecutor.TestConnectReferenceRefusalAsync(Tested(host, StoredReference, port), stored.ReadAsync, ct)); + } + } + + /// The tested server flipped on one connection setting away from . + private static MonitoredServer TestedWith(string setting) + { + var server = Tested("alpha-01.example.test", StoredReference); + switch (setting) + { + case "engine": server.Engine = "postgresql"; break; + case "encryptMode": server.EncryptMode = "Optional"; break; + case "trustServerCertificate": server.TrustServerCertificate = true; break; + case "auth": server.Auth = "serviceprincipal"; break; + case "username": server.Username = "someone-else"; break; + case "database": server.Database = "other_db"; break; + case "readOnlyIntent": server.ReadOnlyIntent = true; break; + case "multiSubnetFailover": server.MultiSubnetFailover = true; break; + case "host": server.Host = "elsewhere.example.test"; break; + case "port": server.Port = 1434; break; + case "none": break; + default: throw new ArgumentOutOfRangeException(nameof(setting), setting, null); + } + + return server; + } + + [Theory] + [InlineData("engine")] + [InlineData("encryptMode")] + [InlineData("trustServerCertificate")] + [InlineData("auth")] + [InlineData("username")] + [InlineData("database")] + [InlineData("readOnlyIntent")] + [InlineData("multiSubnetFailover")] + [InlineData("host")] + [InlineData("port")] + public async Task TestConnect_AStoredReference_IsRefusedWhenAnySingleConnectionSettingDiffersFromTheStoredServer(string setting) + { + var stored = new StoredAddresses(); + stored.Rows.Add(StoredRow("alpha-01.example.test")); + + var refusal = await DarlingCommandExecutor.TestConnectReferenceRefusalAsync(TestedWith(setting), stored.ReadAsync, CancellationToken.None); + + Assert.Equal(DarlingCommandExecutor.TestConnectPasswordNeededText, refusal); + Assert.Contains("different connection settings", refusal, StringComparison.Ordinal); + Assert.DoesNotContain(StoredReference, refusal, StringComparison.Ordinal); + } + + [Fact] + public async Task TestConnect_AStoredReference_IsResolvedWhenEveryConnectionSettingMatches_IgnoringCaseWhereTheEditCoreDoes() + { + var stored = new StoredAddresses(); + stored.Rows.Add(StoredRow("alpha-01.example.test")); + var server = TestedWith("none"); + server.Auth = "SQL"; + server.EncryptMode = "MANDATORY"; + server.Engine = "SqlServer"; + + Assert.Null(await DarlingCommandExecutor.TestConnectReferenceRefusalAsync(TestedWith("none"), stored.ReadAsync, CancellationToken.None)); + Assert.Null(await DarlingCommandExecutor.TestConnectReferenceRefusalAsync(server, stored.ReadAsync, CancellationToken.None)); + } + + + [Fact] + public async Task TestConnect_AReferenceNoStoredServerHolds_IsRefused() + { + var stored = new StoredAddresses(); + + var refusal = await DarlingCommandExecutor.TestConnectReferenceRefusalAsync( + Tested("alpha-01.example.test", "file:C:\\x"), stored.ReadAsync, CancellationToken.None); + + Assert.Equal(DarlingCommandExecutor.TestConnectPasswordNeededText, refusal); + Assert.DoesNotContain("C:\\x", refusal, StringComparison.Ordinal); + } + + [Fact] + public async Task TestConnect_AReferenceInThePlainPasswordSlot_IsAlwaysRefused_WithoutAskingTheStore() + { + var stored = new StoredAddresses(); + stored.Rows.Add(StoredRow("alpha-01.example.test")); + + var refusal = await DarlingCommandExecutor.TestConnectReferenceRefusalAsync( + Tested("alpha-01.example.test", null, password: "env:X"), stored.ReadAsync, CancellationToken.None); + + Assert.Equal(RefusalSentence, refusal); + Assert.Empty(stored.Asked); + } + + [Theory] + [InlineData(null)] + [InlineData("Dpapi-Blob-Base64+/==")] + public async Task TestConnect_ADpapiBlobOrNoPassword_IsNotAskedAbout_AndTheStoreIsNotRead(string? encrypted) + { + var stored = new StoredAddresses(); + + Assert.Null(await DarlingCommandExecutor.TestConnectReferenceRefusalAsync( + Tested("anywhere.example.test", encrypted), stored.ReadAsync, CancellationToken.None)); + Assert.Empty(stored.Asked); + } + + /* ---------------- the configuration file still takes references ---------------- */ + + [Fact] + public void TheConfigurationFile_StillAcceptsAReference_AndMarksItDeclaredByTheFile() + { + var config = DarlingConfig.Parse( + "{ \"servers\": [ { \"host\": \"sql01\", \"auth\": \"sql\", \"username\": \"monitor\", \"encryptedPassword\": \"env:SOME_FILE_DECLARED_REF\" } ] }"); + + var server = Assert.Single(config.Servers); + Assert.Equal("env:SOME_FILE_DECLARED_REF", server.EncryptedPassword); + Assert.True(server.EncryptedPasswordDeclaredByFile); + } +} diff --git a/Darling/Darling.Tests/ServerRemovalClearsTagsLiveTests.cs b/Darling/Darling.Tests/ServerRemovalClearsTagsLiveTests.cs new file mode 100644 index 0000000000..aea6b6cf82 --- /dev/null +++ b/Darling/Darling.Tests/ServerRemovalClearsTagsLiveTests.cs @@ -0,0 +1,110 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Text.Json; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: this class mints its own scratch database through ScratchPostgres and touches nothing on the + shared one. It seeds fixed server ids and reads no wall clock. */ + +/// Removing a server also removes its fleet-tag assignments (#5085). server_id is a fixed hash of +/// the connection, so assignments left behind would come back when the same server is added again and would put +/// it back inside tag-scoped alert rules. Both remove paths are driven through their product entry points: the +/// MCP remove_server tool and the Viewer's DeleteMonitoredServerAsync. +public sealed class ServerRemovalClearsTagsLiveTests +{ + private const int KeptId = 7101; + private const int RemovedId = 7102; + + private static async Task<(ScratchPostgres Scratch, NpgsqlDataSource Source, string ConnectionString)?> SeedAsync() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live server-removal tag test (it mints its own scratch database)."); + + var ct = TestContext.Current.CancellationToken; + var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var connectionString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { SearchPath = PgSchemaGenerator.SearchPath }.ConnectionString; + await using (var connection = new NpgsqlConnection(connectionString)) + { + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, null, ct); + } + + var source = NpgsqlDataSource.Create(connectionString); + await ExecAsync(source, $@"INSERT INTO config_monitored_servers (server_id, name, host) VALUES + ({KeptId}, 'tagkeep-host', 'tagkeep-host'), ({RemovedId}, 'tagdrop-host', 'tagdrop-host')"); + var store = new ServerTagStore(source, 30); + var ct2 = TestContext.Current.CancellationToken; + var prod = Assert.IsType(await store.CreateAsync("Prod", null, null, ct2)).Tag!; + var east = Assert.IsType(await store.CreateAsync("East", null, null, ct2)).Tag!; + foreach (var tag in new[] { prod, east }) + { + await store.AssignAsync(tag.Id, [KeptId, RemovedId], ct2); + } + + return (scratch, source, connectionString); + } + + private static async Task ExecAsync(NpgsqlDataSource source, string sql) + { + await using var command = source.CreateCommand(sql); + await command.ExecuteNonQueryAsync(TestContext.Current.CancellationToken); + } + + private static async Task MapRowsAsync(NpgsqlDataSource source, int serverId) + { + await using var command = source.CreateCommand($"SELECT count(*) FROM server_tag_map WHERE server_id = {serverId}"); + return Convert.ToInt32(await command.ExecuteScalarAsync(TestContext.Current.CancellationToken)); + } + + [Fact] + public async Task TheMcpRemoveServerTool_DropsTheServersTagAssignments_AndNoOtherServers() + { + var seeded = await SeedAsync(); + var (scratch, source, _) = seeded!.Value; + await using var _s = scratch; + await using var _d = source; + Assert.Equal(2, await MapRowsAsync(source, RemovedId)); + + using var doc = JsonDocument.Parse(await DarlingMcpServerAdminTools.RemoveServer(source, "tagdrop-host")); + Assert.Equal("removed", doc.RootElement.GetProperty("status").GetString()); + + Assert.Equal(0, await MapRowsAsync(source, RemovedId)); + Assert.Equal(2, await MapRowsAsync(source, KeptId)); + } + + [Fact] + public async Task TheViewersRemove_DropsTheServersTagAssignments_AndNoOtherServers() + { + var seeded = await SeedAsync(); + var (scratch, source, connectionString) = seeded!.Value; + await using var _s = scratch; + await using var _d = source; + Assert.Equal(2, await MapRowsAsync(source, RemovedId)); + + await using (var viewer = new ViewerDataService(connectionString)) + { + await viewer.DeleteMonitoredServerAsync(RemovedId, TestContext.Current.CancellationToken); + } + + Assert.Equal(0, await MapRowsAsync(source, RemovedId)); + Assert.Equal(2, await MapRowsAsync(source, KeptId)); + Assert.Equal(0L, Convert.ToInt64(await (source.CreateCommand($"SELECT count(*) FROM config_monitored_servers WHERE server_id = {RemovedId}")).ExecuteScalarAsync(TestContext.Current.CancellationToken))); + } +} diff --git a/Darling/Darling.Tests/ServerTabColumnGroupsPinTests.cs b/Darling/Darling.Tests/ServerTabColumnGroupsPinTests.cs new file mode 100644 index 0000000000..b2b95023e1 --- /dev/null +++ b/Darling/Darling.Tests/ServerTabColumnGroupsPinTests.cs @@ -0,0 +1,47 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System.IO; +using System.Linq; +using System.Text.RegularExpressions; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// The two System Events grids with the most columns on the web dashboard use the shared column groups (#4843): every +/// group named in a grid's group list marks at least one column, every column group is named in the list, and the +/// default groups are among the listed ones. +/// +[Trait("Stage", "Guard")] +public sealed class ServerTabColumnGroupsPinTests +{ + private static string Js() => File.ReadAllText(PathTo("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "pages", "server-tabs.js")); + + [Theory] + [InlineData("Memory Conditions", "MEMORY_CONDITION_COLUMNS", "MEMORY_CONDITION_GROUPS")] + [InlineData("Memory Node OOM", "MEMORY_OOM_COLUMNS", "MEMORY_OOM_GROUPS")] + public void TheWideGrid_GroupsItsColumns_AndItsGroupListMatches(string title, string columns, string groups) + { + var js = Js(); + var cols = Regex.Match(js, "const " + columns + @" = \[(.*?)\n\];", RegexOptions.Singleline).Groups[1].Value; + var used = Regex.Matches(cols, "group: \"([^\"]+)\"").Select(m => m.Groups[1].Value).Distinct().ToArray(); + var decl = Regex.Match(js, "const " + groups + @" = \{ groups: \[([^\]]*)\], defaultGroups: \[([^\]]*)\] \};"); + Assert.True(decl.Success, groups + " must declare groups and defaultGroups."); + var listed = Regex.Matches(decl.Groups[1].Value, "\"([^\"]+)\"").Select(m => m.Groups[1].Value).ToArray(); + var defaults = Regex.Matches(decl.Groups[2].Value, "\"([^\"]+)\"").Select(m => m.Groups[1].Value).ToArray(); + Assert.Equal(listed.OrderBy(x => x), used.OrderBy(x => x)); + Assert.NotEmpty(defaults); + Assert.All(defaults, d => Assert.Contains(d, listed)); + Assert.Contains("group: ", cols); + // The constant is the last argument of THIS grid's own table() call, next to its own columns constant. + var call = Regex.Match(js, "table\\(\\s*\"" + title + "\",(?:(?!\\btable\\().)*?\\b" + columns + @",(?:(?!\btable\().)*?,\s*" + groups + @"\s*\)", RegexOptions.Singleline); + Assert.True(call.Success, title + "'s table() call must pass " + groups + " as its last argument."); + } +} diff --git a/Darling/Darling.Tests/ServerTabEmptyStatesAndHeadersTests.cs b/Darling/Darling.Tests/ServerTabEmptyStatesAndHeadersTests.cs new file mode 100644 index 0000000000..b8ac2b5de1 --- /dev/null +++ b/Darling/Darling.Tests/ServerTabEmptyStatesAndHeadersTests.cs @@ -0,0 +1,124 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor Lite. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.RegularExpressions; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// The server tabs of Lite and of the Darling Viewer (a copy of Lite's) say why a grid or chart is empty, and their +/// Configuration grids show whole column headers. The walk of the release found a blank Trace Flags grid, a Query Store +/// Regressions grid that said "No data for the selected time range." at Last 7 days while Last 24 hours listed rows, Configuration +/// headers cut off at the default width ("Configured Valu", "Dynami", "Compat Le"), and two Blocking charts that looked +/// broken because they hold points only where a collection found a waiting task. +/// +[Trait("Reads", "Lite")] +public sealed class ServerTabEmptyStatesAndHeadersTests +{ + private static string LiteXaml() => RepoFile.ReadRepoFile("Lite", "Controls", "ServerTab.xaml"); + + private static string ViewerXaml() => RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerServerTab.xaml"); + + public static IEnumerable BothTabs() => new[] { new object[] { "Lite" }, new object[] { "Viewer" } }; + + private static string Xaml(string app) => app == "Lite" ? LiteXaml() : ViewerXaml(); + + /// One text column of the Configuration tab: its header words and its Width in the XAML. + private static List<(string Header, int Width)> ConfigurationColumns(string xaml) + { + var start = xaml.IndexOf("x:Name=\"ServerConfigGrid\"", StringComparison.Ordinal); + var end = xaml.IndexOf("x:Name=\"TraceFlagsNoDataMessage\"", StringComparison.Ordinal); + Assert.True(start > 0 && end > start, "the Configuration tab's grids were not found"); + + return Regex.Matches( + xaml[start..end], + @"]*?\bWidth=""(\d+)""[^>]*>\s*.*? (m.Groups[2].Value, int.Parse(m.Groups[1].Value, System.Globalization.CultureInfo.InvariantCulture))) + .ToList(); + } + + /// + /// A bold header with a filter button needs about 6 pixels a character plus 50 for the button and the padding. The walk saw + /// "Configured Value" cut to 15 characters at 130, "Dynamic" to 6 at 80 and "Compat Level" to 9 at 100. + /// + [Theory] + [MemberData(nameof(BothTabs))] + public void ConfigurationHeaders_FitTheirColumns(string app) + { + var columns = ConfigurationColumns(Xaml(app)); + + Assert.True(columns.Count >= 30, $"{app}: expected the Configuration grids' columns, found {columns.Count}"); + + var clipped = columns + .Where(c => c.Width < (c.Header.Length * 6) + 50) + .Select(c => $"{c.Header} ({c.Width})") + .ToList(); + + Assert.True(clipped.Count == 0, $"{app}: headers wider than their columns: {string.Join(", ", clipped)}"); + } + + [Theory] + [MemberData(nameof(BothTabs))] + public void TraceFlagsGrid_SaysNoTraceFlagsAreEnabled_WhenItHasNoRows(string app) + { + var xaml = Xaml(app); + var at = xaml.IndexOf("x:Name=\"TraceFlagsNoDataMessage\"", StringComparison.Ordinal); + Assert.True(at > 0); + var element = xaml[at..xaml.IndexOf("/>", at, StringComparison.Ordinal)]; + + Assert.Contains("Text=\"No trace flags are enabled on this server.\"", element, StringComparison.Ordinal); + + var code = app == "Lite" + ? RepoFile.ReadRepoFile("Lite", "Controls", "ServerTab.Refresh.cs") + : RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerServerTab.Filters.cs"); + var call = Regex.Match(code, @"ShowEngineGap(?:Async)?\(TraceFlagsNoDataMessage[^;]*;"); + + Assert.True(call.Success, $"{app}: the Trace Flags load no longer calls the engine-gap step"); + Assert.Contains("keepsOwnEmptyText: true", call.Value, StringComparison.Ordinal); + } + + [Fact] + public void QueryStoreRegressionsGrid_SaysWhyItCanBeEmpty() + { + var text = ViewerServerTab.QueryStoreRegressionsEmptyText; + + Assert.Contains("7 days before", text, StringComparison.Ordinal); + Assert.DoesNotContain("No data for the selected time range", text, StringComparison.Ordinal); + + var filters = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerServerTab.Filters.cs"); + Assert.Contains("EmptyState.SetText(QueryStoreRegressionsGrid, QueryStoreRegressionsEmptyText)", filters, StringComparison.Ordinal); + } + + /// Lite has no Query Store Regressions grid (only the MCP tool reads that comparison), so only the Viewer needs the words. + [Fact] + public void Lite_HasNoQueryStoreRegressionsGrid() + { + Assert.DoesNotContain("QueryStoreRegressionsGrid", LiteXaml(), StringComparison.Ordinal); + } + + [Theory] + [MemberData(nameof(BothTabs))] + public void CurrentWaitsCharts_SayPointsAppearOnlyWhereTasksWereWaiting(string app) + { + var xaml = Xaml(app); + var at = xaml.IndexOf("x:Name=\"CurrentWaitsSparseNote\"", StringComparison.Ordinal); + Assert.True(at > 0, $"{app}: the Current Waits tab has no sparse-data note"); + + var tab = xaml.IndexOf("", StringComparison.Ordinal); + var tabEnd = xaml.IndexOf("", tab, StringComparison.Ordinal); + Assert.InRange(at, tab, tabEnd); + + Assert.Contains("Points appear only when a collection found waiting or blocked tasks.", xaml[at..], StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/ServerTabGridColumnPinTests.cs b/Darling/Darling.Tests/ServerTabGridColumnPinTests.cs new file mode 100644 index 0000000000..64ff87ede3 --- /dev/null +++ b/Darling/Darling.Tests/ServerTabGridColumnPinTests.cs @@ -0,0 +1,78 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text.RegularExpressions; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// Pins the Query Store Regressions, Plan Corrections and Long Queries grids on the server page to the field +/// names their reads return: a column key the read does not emit renders an empty column, so each key is +/// checked against the tool source. +/// +[Trait("Stage", "Guard")] +public sealed class ServerTabGridColumnPinTests +{ + private static string ServerTabsJs => ReadRepoFile(Path.Combine( + "Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "pages", "server-tabs.js")).ReplaceLineEndings("\n"); + + private static string Tool(string file) => ReadRepoFile(Path.Combine( + "Darling", "PerformanceMonitor.Darling.Service", "Mcp", file)).ReplaceLineEndings("\n"); + + private static List<(string Key, string Line)> Columns(string constant) + { + var js = ServerTabsJs; + var start = js.IndexOf("const " + constant + " = [", StringComparison.Ordinal); + Assert.True(start >= 0, constant); + var end = js.IndexOf("\n];", start, StringComparison.Ordinal); + return Regex.Matches(js[start..end], @"^\s*\{ key: ""([a-z_.]+)""[^\n]*$", RegexOptions.Multiline) + .Select(m => (m.Groups[1].Value, m.Value)).ToList(); + } + + private static void EveryKeyIsReturnedBy(string constant, string toolFile, params string[] expected) + { + var tool = Tool(toolFile); + var cols = Columns(constant); + foreach (var (key, _) in cols) + Assert.True(Regex.IsMatch(tool, @"\b" + key + @"\s*="), $"{constant}: '{key}' is not a field of {toolFile}"); + foreach (var key in expected) + Assert.Contains(cols, c => c.Key == key); + Assert.Equal(cols.Count, cols.Select(c => c.Key).Distinct().Count()); + // Numeric columns use numeric formats, never pre-formatted *_text strings. + Assert.DoesNotContain(cols, c => c.Key.EndsWith("_text", StringComparison.Ordinal) && c.Key != "query_text"); + } + + [Fact] + public void QueryStoreRegressions_CarriesBaseExecsAndBothReadsColumns() => + EveryKeyIsReturnedBy("QUERY_STORE_REGRESSION_COLUMNS", "DarlingMcpQueryStoreRegressionTools.cs", + "baseline_exec_count", "baseline_reads", "recent_reads"); + + [Fact] + public void PlanCorrections_CarriesEveryFieldTheReadReturnsExceptTheDuplicatedFlags() => + EveryKeyIsReturnedBy("PLAN_CORRECTION_COLUMNS", "DarlingMcpPlanCorrectionTools.cs", + "regressed_plan_id", "last_good_plan_id", "last_good_plan_forcing_type", + "last_good_plan_force_failure_reason", "regressed_plan_execution_count", + "regressed_plan_cpu_time_average_ms", "last_good_plan_execution_count", + "last_good_plan_cpu_time_average_ms", "valid_since", "last_refresh", + "execute_action_initiated_by", "execute_action_initiated_time", + "revert_action_initiated_by", "revert_action_initiated_time", "recommendation_state_reason"); + + [Fact] + public void PlanCorrections_TimestampColumnsAreInstants() + { + foreach (var (key, line) in Columns("PLAN_CORRECTION_COLUMNS")) + if (key is "valid_since" or "last_refresh" || key.EndsWith("_time", StringComparison.Ordinal)) + Assert.Contains("format: \"time\"", line); + } + + [Fact] + public void LongQueries_CarriesEventTypeReadsWritesHashAndLoginButNoHost() + { + EveryKeyIsReturnedBy("LONG_QUERY_COLUMNS", "DarlingMcpLongQueryTools.cs", + "event_type", "logical_reads", "physical_reads", "writes", "query_hash", "server_principal_name"); + Assert.DoesNotContain(Columns("LONG_QUERY_COLUMNS"), c => c.Key.Contains("host", StringComparison.Ordinal)); + } +} diff --git a/Darling/Darling.Tests/ServerTabToolbarAndTooltipsTests.cs b/Darling/Darling.Tests/ServerTabToolbarAndTooltipsTests.cs new file mode 100644 index 0000000000..5ae2751fc7 --- /dev/null +++ b/Darling/Darling.Tests/ServerTabToolbarAndTooltipsTests.cs @@ -0,0 +1,223 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor Lite. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.RegularExpressions; +using System.Threading; +using System.Windows.Automation.Peers; +using PerformanceMonitor.Darling.Viewer; +using PerformanceMonitor.PlanAnalysis; +using PerformanceMonitor.Ui; +using Xunit; + +namespace Darling.Tests; + +/// +/// Release walk V10, D20, D21 and D22 in the server tab of Lite and of the Darling Viewer (a copy of Lite's): the four toolbar +/// combos had no name for a screen reader, the Compare box was greyed on most tabs with a tooltip that WPF never shows on a +/// disabled control, the Running Jobs tab with no running job was an empty grid with no words, the plan viewer's operator tooltips +/// could not be read by a screen reader, and two FinOps column headers had no tooltip while their cells did. +/// +[Trait("Reads", "Lite")] +public sealed class ServerTabToolbarAndTooltipsTests +{ + private static string Xaml(string app) => app == "Lite" + ? RepoFile.ReadRepoFile("Lite", "Controls", "ServerTab.xaml") + : RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerServerTab.xaml"); + + private static string FinOpsXaml(string app) => app == "Lite" + ? RepoFile.ReadRepoFile("Lite", "Controls", "FinOpsTab.xaml") + : RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "FinOpsTab.xaml"); + + public static IEnumerable BothTabs() => new[] { new object[] { "Lite" }, new object[] { "Viewer" } }; + + /// The opening tag of the element with the given x:Name, up to its first '>'. + private static string OpeningTag(string xaml, string name) + { + var at = xaml.IndexOf($"x:Name=\"{name}\"", StringComparison.Ordinal); + Assert.True(at > 0, $"{name} not found"); + var start = xaml.LastIndexOf('<', at); + return xaml[start..(xaml.IndexOf('>', at) + 1)]; + } + + [Theory] + [MemberData(nameof(BothTabs))] + public void ToolbarCombos_HaveANameForAScreenReader(string app) + { + var expected = new Dictionary + { + ["RangePicker"] = "Time range", + ["CompareToCombo"] = "Compare to", + ["AutoRefreshIntervalCombo"] = "Auto-refresh interval", + ["TimeDisplayModeBox"] = "Time display", + }; + + foreach (var (control, name) in expected) + { + Assert.Contains($"AutomationProperties.Name=\"{name}\"", OpeningTag(Xaml(app), control), StringComparison.Ordinal); + } + } + + [Theory] + [MemberData(nameof(BothTabs))] + public void CompareBox_ShowsItsToolTipEvenWhenDisabled(string app) + { + Assert.Contains("ToolTipService.ShowOnDisabled=\"True\"", OpeningTag(Xaml(app), "CompareToCombo"), StringComparison.Ordinal); + } + + [Fact] + public void Viewer_CompareToolTip_NamesTheTabsItWorksOn() + { + var tip = ViewerServerTab.CompareUnavailableToolTip; + + Assert.Contains("Top Queries", tip, StringComparison.Ordinal); + Assert.Contains("Top Procedures", tip, StringComparison.Ordinal); + Assert.Contains("Query Store", tip, StringComparison.Ordinal); + Assert.DoesNotContain("not available", tip, StringComparison.Ordinal); + } + + [Fact] + public void Lite_CompareToolTip_NamesTheTabsItWorksOn() + { + var code = RepoFile.ReadRepoFile("Lite", "Controls", "ServerTab.Comparison.cs"); + var constant = Regex.Match(code, @"CompareUnavailableToolTip\s*=\s*""([^""]+)"""); + + Assert.True(constant.Success); + foreach (var tab in new[] { "Overview", "Top Queries", "Top Procedures", "Query Store" }) + { + Assert.Contains(tab, constant.Groups[1].Value, StringComparison.Ordinal); + } + + Assert.Contains("CompareToCombo.ToolTip = CompareUnavailableToolTip;", code, StringComparison.Ordinal); + } + + /// The Viewer's Compare box used to stay enabled on every other tab once the Queries sub-tab had been on a supported one. + [Theory] + [InlineData(ViewerServerTab.QueriesInnerTabIndex, 3, true)] + [InlineData(ViewerServerTab.QueriesInnerTabIndex, 4, true)] + [InlineData(ViewerServerTab.QueriesInnerTabIndex, 5, true)] + [InlineData(ViewerServerTab.QueriesInnerTabIndex, 0, false)] + [InlineData(ViewerServerTab.QueriesInnerTabIndex, 9, false)] + [InlineData(0, 3, false)] + [InlineData(1, 3, false)] + [InlineData(ViewerServerTab.QueriesInnerTabIndex + 1, 4, false)] + public void Viewer_CompareWorksOnlyOnTheThreeQueriesGrids(int innerTab, int queriesSubTab, bool expected) + { + Assert.Equal(expected, ViewerServerTab.IsComparisonSupported(innerTab, queriesSubTab)); + } + + [Fact] + public void Viewer_InnerTabSwitch_UpdatesTheCompareBox() + { + var code = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerServerTab.xaml.cs"); + var handler = code[code.IndexOf("private async void InnerTabs_SelectionChanged", StringComparison.Ordinal)..]; + + Assert.Contains("UpdateCompareDropdownState();", handler[..handler.IndexOf("RefreshActiveInnerTabAsync", StringComparison.Ordinal)], StringComparison.Ordinal); + } + + [Theory] + [MemberData(nameof(BothTabs))] + public void RunningJobsGrid_SaysNoJobIsRunning_WhenItHasNoRows(string app) + { + var xaml = Xaml(app); + Assert.Contains("Text=\"No SQL Agent jobs are running.\"", OpeningTag(xaml, "RunningJobsNoDataMessage"), StringComparison.Ordinal); + + var code = app == "Lite" + ? RepoFile.ReadRepoFile("Lite", "Controls", "ServerTab.Refresh.cs") + : RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerServerTab.RunningJobs.cs"); + var call = Regex.Match(code, @"ShowEngineGap(?:Async)?\(RunningJobsNoDataMessage[^;]*;", RegexOptions.Singleline); + + Assert.True(call.Success, $"{app}: the Running Jobs load no longer calls the engine-gap step"); + Assert.Contains("keepsOwnEmptyText:", call.Value, StringComparison.Ordinal); + Assert.DoesNotContain("keepsOwnEmptyText: false", call.Value, StringComparison.Ordinal); + } + + [Theory] + [MemberData(nameof(BothTabs))] + public void FinOpsFindingsGrid_DetailAndSavingsHeaders_HaveATooltip(string app) + { + var xaml = FinOpsXaml(app); + + foreach (var header in new[] { "Detail", "Est. Savings ($/mo)" }) + { + var at = xaml.IndexOf($" 0, $"{app}: column {header} not found"); + var column = xaml[at..xaml.IndexOf("", at, StringComparison.Ordinal)]; + + Assert.Matches(@".* new() + { + PhysicalOp = "Clustered Index Scan", + LogicalOp = "Clustered Index Scan", + CostPercent = 62, + EstimatedOperatorCost = 1.5, + EstimatedTotalSubtreeCost = 2.5, + EstimateRows = 1200, + ObjectName = "Orders", + Warnings = { new PlanWarning { WarningType = "Missing Index", Message = "An index would help." } }, + }; + + [Fact] + public void PlanOperatorBox_ExposesItsTooltipSummaryToAScreenReader() + { + var node = Scan(); + + var (name, help, isControl) = StaTestThread.Run(() => + { + var box = new PlanNodeBorder(); + System.Windows.Automation.AutomationProperties.SetName(box, PlanNodeBorder.OperatorName(node)); + System.Windows.Automation.AutomationProperties.SetHelpText(box, PlanNodeBorder.Summary(node, null)); + var peer = UIElementAutomationPeer.CreatePeerForElement(box); + Assert.NotNull(peer); + return (peer.GetName(), peer.GetHelpText(), peer.IsControlElement()); + }); + + Assert.Equal("Clustered Index Scan", name); + Assert.True(isControl); + Assert.Contains("62%", help, StringComparison.Ordinal); + Assert.Contains("Estimated rows", help, StringComparison.Ordinal); + Assert.Contains("Orders", help, StringComparison.Ordinal); + Assert.Contains("1 warning: Missing Index", help, StringComparison.Ordinal); + } + + [Fact] + public void PlanOperatorSummary_PrefersActualRows_AndListsEachWarningTypeOnce() + { + var node = Scan(); + node.HasActualStats = true; + node.ActualRows = 40; + node.ActualExecutions = 3; + var warnings = new List + { + new() { WarningType = "Spill" }, + new() { WarningType = "Spill" }, + new() { WarningType = "Implicit Conversion" }, + }; + + var text = PlanNodeBorder.Summary(node, warnings); + + Assert.Contains("actual rows 40", text, StringComparison.Ordinal); + Assert.Contains("actual executions 3", text, StringComparison.Ordinal); + Assert.Contains("3 warnings: Spill, Implicit Conversion.", text, StringComparison.Ordinal); + } + + [Fact] + public void PlanViewer_BuildsItsOperatorBoxesAsPlanNodeBorders_WithHelpText() + { + var code = RepoFile.ReadRepoFile("PerformanceMonitor.Ui", "PlanViewerControl.Rendering.cs"); + + Assert.Contains("var border = new PlanNodeBorder", code, StringComparison.Ordinal); + Assert.Equal(2, Regex.Matches(code, @"AutomationProperties\.SetHelpText\(border, PlanNodeBorder\.Summary").Count); + Assert.Contains("AutomationProperties.SetName(border, PlanNodeBorder.OperatorName(node))", code, StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/ServerTagCoverageTests.cs b/Darling/Darling.Tests/ServerTagCoverageTests.cs new file mode 100644 index 0000000000..0139d9e566 --- /dev/null +++ b/Darling/Darling.Tests/ServerTagCoverageTests.cs @@ -0,0 +1,106 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System.Collections.Generic; +using System.Linq; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// Which custom alert rules a tag change affects (#5085), by diffing two snapshots. +public sealed class ServerTagCoverageTests +{ + private static ServerTagSnapshot Snap( + IEnumerable tags, + IEnumerable<(int Server, int Tag)> assignments, + IEnumerable rules) => + new(tags.ToList(), assignments.Select(a => new ServerTagAssignmentRow(a.Server, a.Tag)).ToList(), rules.ToList()); + + [Fact] + public void Members_IsTheWholeSubtree() + { + var tags = new[] { new ServerTagRow(1, "Prod", null, 0, null), new ServerTagRow(2, "East", 1, 0, null) }; + var snap = Snap(tags, [(100, 1), (101, 2), (102, 99)], []); + Assert.Equal([100, 101], ServerTagCoverage.Members(snap, 1).OrderBy(i => i)); + Assert.Equal([101], ServerTagCoverage.Members(snap, 2).OrderBy(i => i)); + } + + [Fact] + public void Unassign_OfAServerStillReachableThroughASiblingUnderTheScope_ReportsNothing() + { + var tags = new[] { new ServerTagRow(1, "Prod", null, 0, null), new ServerTagRow(2, "East", 1, 0, null), new ServerTagRow(3, "West", 1, 0, null) }; + var rules = new[] { new TagScopedRule(7, "r", true, 1) }; + var before = Snap(tags, [(100, 2), (100, 3)], rules); + var after = Snap(tags, [(100, 3)], rules); + Assert.Empty(ServerTagCoverage.Diff(before, after)); + } + + [Fact] + public void Move_ReportsOldAncestorsLosing_NewAncestorsGaining_AndNotTheCommonOne() + { + /* Root(1) > A(2) > M(5); Root(1) > B(3). Move M under B: A loses, B gains, Root unchanged. */ + var rules = new[] { new TagScopedRule(1, "root", true, 1), new TagScopedRule(2, "a", true, 2), new TagScopedRule(3, "b", false, 3), new TagScopedRule(4, "m", true, 5) }; + var beforeTags = new[] { new ServerTagRow(1, "Root", null, 0, null), new ServerTagRow(2, "A", 1, 0, null), new ServerTagRow(3, "B", 1, 0, null), new ServerTagRow(5, "M", 2, 0, null) }; + var afterTags = new[] { new ServerTagRow(1, "Root", null, 0, null), new ServerTagRow(2, "A", 1, 0, null), new ServerTagRow(3, "B", 1, 0, null), new ServerTagRow(5, "M", 3, 0, null) }; + var assignments = new[] { (100, 5) }; + + var diff = ServerTagCoverage.Diff(Snap(beforeTags, assignments, rules), Snap(afterTags, assignments, rules)); + + Assert.Equal(2, diff.Count); + var a = diff.Single(d => d.ScopeTagId == 2); + Assert.Equal("loses_servers", a.Effect); + Assert.Equal([100], a.ServersLost); + Assert.Equal(1, a.ServersLostCount); + var b = diff.Single(d => d.ScopeTagId == 3); + Assert.Equal("gains_servers", b.Effect); + Assert.False(b.Enabled); + Assert.DoesNotContain(diff, d => d.ScopeTagId is 1 or 5); + } + + [Fact] + public void ARuleWhoseScopeTagWasAlreadyGoneBeforeTheWrite_IsNotListed() + { + var tags = new[] { new ServerTagRow(1, "Prod", null, 0, null), new ServerTagRow(2, "East", 1, 0, null) }; + var rules = new[] { new TagScopedRule(7, "dangling", true, 99), new TagScopedRule(8, "live", true, 2) }; + var before = Snap(tags, [(100, 2)], rules); + + var diff = ServerTagCoverage.SimulateDelete(before, 2); + + Assert.DoesNotContain(diff, d => d.RuleId == 7); + Assert.Equal("orphaned", diff.Single(d => d.RuleId == 8).Effect); + Assert.Empty(ServerTagCoverage.Diff(before, before)); + } + + [Fact] + public void ADeletedScopeTag_IsOrphaned_WithEveryOldServerLost() + { + var tags = new[] { new ServerTagRow(1, "Prod", null, 0, null), new ServerTagRow(2, "East", 1, 0, null) }; + var rules = new[] { new TagScopedRule(7, "r", true, 2), new TagScopedRule(8, "up", true, 1) }; + var before = Snap(tags, [(100, 2), (101, 1)], rules); + + var diff = ServerTagCoverage.SimulateDelete(before, 2); + + var orphan = diff.Single(d => d.RuleId == 7); + Assert.Equal("orphaned", orphan.Effect); + Assert.Equal([100], orphan.ServersLost); + var up = diff.Single(d => d.RuleId == 8); + Assert.Equal("loses_servers", up.Effect); + Assert.Equal([100], up.ServersLost); + } + + [Fact] + public void RulesScopedInSubtree_IncludesDisabledRules_AndExcludesAncestors() + { + var tags = new[] { new ServerTagRow(1, "Prod", null, 0, null), new ServerTagRow(2, "East", 1, 0, null) }; + var rules = new[] { new TagScopedRule(7, "r", false, 2), new TagScopedRule(8, "up", true, 1) }; + var impact = ServerTagCoverage.RulesScopedInSubtree(Snap(tags, [], rules), 1); + Assert.Equal(2, impact.Count); + Assert.Equal([7L], ServerTagCoverage.RulesScopedInSubtree(Snap(tags, [], rules), 2).Select(r => r.RuleId)); + } +} diff --git a/Darling/Darling.Tests/ServerTagDeleteWarningTests.cs b/Darling/Darling.Tests/ServerTagDeleteWarningTests.cs new file mode 100644 index 0000000000..806a5eb805 --- /dev/null +++ b/Darling/Darling.Tests/ServerTagDeleteWarningTests.cs @@ -0,0 +1,49 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System.Collections.Generic; +using System.Linq; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// The sentence both tag-delete dialogs append when custom alert rules are scoped to the tag or a child. +public sealed class ServerTagDeleteWarningTests +{ + private static List Rules(int n) => + Enumerable.Range(1, n).Select(i => new TagScopedRule(10 + i, "Rule" + i, true, 3)).ToList(); + + [Fact] + public void NoRules_AddNothing() => + Assert.Equal(string.Empty, ServerTagCoverage.DeleteWarning(Rules(0))); + + [Fact] + public void OneRule_NamesItWithItsId() => + Assert.Equal( + " Custom alert rules scoped to this tag or a child tag will stop matching any server: 'Rule1' (#11)", + ServerTagCoverage.DeleteWarning(Rules(1))); + + [Fact] + public void FiveRules_NameAllFive_WithNoMoreSuffix() + { + var text = ServerTagCoverage.DeleteWarning(Rules(5)); + + Assert.EndsWith("'Rule4' (#14), 'Rule5' (#15)", text); + Assert.DoesNotContain("more", text); + } + + [Fact] + public void SevenRules_NameFive_ThenSayTwoMore() + { + var text = ServerTagCoverage.DeleteWarning(Rules(7)); + + Assert.EndsWith("'Rule5' (#15), and 2 more", text); + Assert.DoesNotContain("Rule6", text); + } +} diff --git a/Darling/Darling.Tests/ServerTagMcpRoleLiveTests.cs b/Darling/Darling.Tests/ServerTagMcpRoleLiveTests.cs new file mode 100644 index 0000000000..aa17e548c3 --- /dev/null +++ b/Darling/Darling.Tests/ServerTagMcpRoleLiveTests.cs @@ -0,0 +1,111 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Security.Cryptography; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: this class mints its own scratch database through ScratchPostgres and a role with a per-run + name, and touches nothing on the shared one, so it is not serialized against the live-postgres collection. */ + +/// The fleet-tag writes as a non-superuser holding exactly the grants the managed mcp role gets +/// for them (#5085). Every other live test connects as the owner, which can write anything, so a missing +/// mcp grant would pass them all and fail only in a real deployment with a 42501. +/// +/// The test role is a LOGIN NOSUPERUSER role with a per-run name (the DarlingMcpStoreHostToolsLiveTests +/// pattern; a role belongs to the whole cluster, so the literal mcp would collide with a real run). It gets +/// the schema reach the real role has (USAGE and SELECT on config) and the two tag-table write grants +/// TAKEN FROM 's output with the role name swapped, so +/// removing a grant from the product's SQL makes this test fail. +public sealed class ServerTagMcpRoleLiveTests +{ + private static readonly string RoleName = "tag_mcp_" + Guid.NewGuid().ToString("N")[..8]; + private static readonly string RolePassword = Convert.ToHexString(RandomNumberGenerator.GetBytes(16)); + + [Fact] + public async Task TheMcpRoleGrants_LetServerTagStoreCreateAssignUpdateAndDelete() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live mcp-role server-tag test (it mints its own scratch database and role)."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var ownerString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { SearchPath = PgSchemaGenerator.SearchPath }.ConnectionString; + await using var owner = NpgsqlDataSource.Create(ownerString); + await using (var connection = new NpgsqlConnection(ownerString)) + { + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, null, ct); + } + + var provisioning = DarlingManagedRoles.BuildProvisioningSql( + ProvisioningTestSecrets.Admin, ProvisioningTestSecrets.Viewer, ProvisioningTestSecrets.Mcp); + var tagGrants = provisioning + .Split('\n') + .Select(line => line.Trim()) + .Where(line => line.StartsWith("GRANT ", StringComparison.Ordinal) + && line.EndsWith(" TO mcp;", StringComparison.Ordinal) + && (line.Contains("config.server_tags ", StringComparison.Ordinal) || line.Contains("config.server_tag_map ", StringComparison.Ordinal))) + .Select(line => line[..^" TO mcp;".Length] + " TO " + RoleName + ";") + .ToList(); + + await ExecAsync(owner, $"CREATE ROLE {RoleName} LOGIN NOSUPERUSER PASSWORD '{RolePassword}'", ct); + try + { + await ExecAsync(owner, $"GRANT USAGE ON SCHEMA config TO {RoleName}; GRANT SELECT ON ALL TABLES IN SCHEMA config TO {RoleName};", ct); + + /* Exactly the two tag-table write grants come out of the product's SQL; a renamed or dropped line + would otherwise leave this test granting nothing and still passing. */ + Assert.Equal(2, tagGrants.Count); + + var asMcp = new NpgsqlConnectionStringBuilder(ownerString) { Username = RoleName, Password = RolePassword }.ConnectionString; + await using var source = NpgsqlDataSource.Create(asMcp); + var store = new ServerTagStore(source, 30); + + /* Before the grants the same role can read but not write: the insufficient-privilege answer proves the + grants below are what lets the writes through. */ + var denied = await Assert.ThrowsAsync(() => store.CreateAsync("Prod", null, null, ct)); + Assert.Equal("42501", denied.SqlState); + + foreach (var grant in tagGrants) + { + await ExecAsync(owner, grant, ct); + } + + var created = Assert.IsType(await store.CreateAsync("Prod", null, null, ct)); + var tagId = created.Tag!.Id; + Assert.Equal(new[] { 4, 5 }, await store.AssignAsync(tagId, new[] { 4, 5 }, ct)); + var updated = Assert.IsType( + await store.UpdateAsync(tagId, new ServerTagEdit(true, "Production", false, null, false, null), ct)); + Assert.Equal("Production", updated.Tag!.Name); + Assert.Equal(new[] { 4 }, await store.UnassignAsync(tagId, new[] { 4 }, ct)); + var deleted = Assert.IsType(await store.DeleteAsync(tagId, confirm: true, ct)); + Assert.Equal(1, deleted.RemovedAssignments); + Assert.Empty(await store.ReadTagsAsync(ct)); + } + finally + { + await ExecAsync(owner, $"DROP OWNED BY {RoleName}; DROP ROLE IF EXISTS {RoleName};", ct); + } + } + + private static async Task ExecAsync(NpgsqlDataSource source, string sql, System.Threading.CancellationToken ct) + { + await using var command = source.CreateCommand(sql); + await command.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/ServerTagRulesTests.cs b/Darling/Darling.Tests/ServerTagRulesTests.cs new file mode 100644 index 0000000000..5e649d34ad --- /dev/null +++ b/Darling/Darling.Tests/ServerTagRulesTests.cs @@ -0,0 +1,151 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System.Collections.Generic; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// The fleet-tag write rules (#5085): name, duplicate, depth, cycle, parent and colour. +public sealed class ServerTagRulesTests +{ + /* A chain Prod(1) > East(2) > Web(3) > Edge(4), plus a root Test(10) with child Lab(11). */ + private static readonly List Tree = + [ + new(1, "Prod", null, 0, null), + new(2, "East", 1, 0, null), + new(3, "Web", 2, 0, null), + new(4, "Edge", 3, 0, null), + new(10, "Test", null, 0, null), + new(11, "Lab", 10, 0, null), + ]; + + private static ServerTagEdit Move(int? parent) => new(false, null, false, null, true, parent); + + [Fact] + public void Create_UnderADepthThreeParent_IsRefused_AndUnderDepthTwoIsAllowed() + { + var refused = Assert.IsType(ServerTagRules.CheckCreate(Tree, "Deep", 4, null, out _, out _)); + Assert.Equal("depth_limit", refused.Code); + Assert.Equal("Tags can nest at most four levels deep.", refused.Message); + + Assert.Null(ServerTagRules.CheckCreate(Tree, "Ok", 3, null, out _, out _)); + } + + [Fact] + public void Move_OfAHeightOneSubtreeUnderDepthTwo_IsRefused_AndMoveToRootIsAllowed() + { + /* Test(10) has height 1; under Web(3, depth 2) it would reach depth 4 (0-based). */ + var refused = Assert.IsType(ServerTagRules.CheckUpdate(Tree, 10, Move(3), out _, out _)); + Assert.Equal("depth_limit", refused.Code); + + Assert.Null(ServerTagRules.CheckUpdate(Tree, 4, Move(null), out _, out _)); + } + + [Fact] + public void Move_UnderItselfOrAGrandchild_IsACycle() + { + var self = Assert.IsType(ServerTagRules.CheckUpdate(Tree, 1, Move(1), out _, out _)); + Assert.Equal("cycle", self.Code); + var grand = Assert.IsType(ServerTagRules.CheckUpdate(Tree, 1, Move(3), out _, out _)); + Assert.Equal("cycle", grand.Code); + } + + [Fact] + public void Duplicates_AreRefusedIgnoringCase_OnCreateRenameAndMove() + { + Assert.IsType(ServerTagRules.CheckCreate(Tree, " prod ", null, null, out _, out _)); + + var rename = new ServerTagEdit(true, "TEST", false, null, false, null); + Assert.IsType(ServerTagRules.CheckUpdate(Tree, 1, rename, out _, out _)); + + /* Moving a tag named Lab into a parent that already has a Lab. */ + var withTwin = new List(Tree) { new(12, "Lab", 1, 0, null) }; + Assert.IsType(ServerTagRules.CheckUpdate(withTwin, 12, Move(10), out _, out _)); + } + + [Fact] + public void ACaseOnlySelfRename_IsAllowed() + { + var rename = new ServerTagEdit(true, "PROD", false, null, false, null); + Assert.Null(ServerTagRules.CheckUpdate(Tree, 1, rename, out var name, out _)); + Assert.Equal("PROD", name); + } + + [Theory] + [InlineData("")] + [InlineData(" ")] + [InlineData("a\tb")] + [InlineData("Prod\u202E")] + [InlineData("Pr\u200Bod")] + public void BadNames_AreRefused(string name) + { + var refused = Assert.IsType(ServerTagRules.CheckName(name, out _)); + Assert.Equal("bad_name", refused.Code); + } + + [Fact] + public void Names_AreTrimmed_AndCappedAtOneHundred() + { + Assert.Null(ServerTagRules.CheckName(" Prod ", out var clean)); + Assert.Equal("Prod", clean); + Assert.Null(ServerTagRules.CheckName(new string('a', 100), out _)); + Assert.Equal("bad_name", ServerTagRules.CheckName(new string('a', 101), out _)!.Code); + } + + [Fact] + public void AFormatCharacterName_SaysItIsAnInvisibleFormattingCharacter() + { + var refused = Assert.IsType(ServerTagRules.CheckName("Pr\u200Bod", out _)); + Assert.Equal("Tag name contains an invisible formatting character.", refused.Message); + } + + [Fact] + public void ACreateColour_IsTrimmedThenValidated() + { + Assert.Null(ServerTagRules.CheckCreate(Tree, "Fresh", null, " #416fa6 ", out _, out var colour)); + Assert.Equal("#416FA6", colour); + Assert.IsType(ServerTagRules.CheckCreate(Tree, "Fresh", null, " ", out _, out _)); + } + + [Fact] + public void Colours_AreUpperCased_AndMalformedOnesRefused() + { + Assert.Null(ServerTagRules.CheckColour("#416fa6", out var colour)); + Assert.Equal("#416FA6", colour); + Assert.Null(ServerTagRules.CheckColour(null, out var none)); + Assert.Null(none); + + foreach (var bad in new[] { "#12345", "red", string.Empty }) + { + var refused = Assert.IsType(ServerTagRules.CheckColour(bad, out _)); + Assert.Equal("bad_colour", refused.Code); + Assert.Equal("colour must be #RRGGBB, e.g. #416FA6, or null to clear it.", refused.Message); + } + } + + [Fact] + public void AMissingParent_IsUnknownParent_AndAMissingTagIsNotFound() + { + var nf = Assert.IsType(ServerTagRules.CheckCreate(Tree, "X", 99, null, out _, out _)); + Assert.Equal("unknown_parent", nf.Code); + var missing = Assert.IsType(ServerTagRules.CheckUpdate(Tree, 99, Move(null), out _, out _)); + Assert.Null(missing.Code); + Assert.IsType(ServerTagRules.CheckDelete(Tree, 99)); + } + + [Fact] + public void AMalformedCycleInTheData_DoesNotHangTheWalks() + { + var cyclic = new List { new(1, "A", 2, 0, null), new(2, "B", 1, 0, null) }; + Assert.True(ServerTagRules.Depth(cyclic, 1) <= ServerTagRules.MaxWalkSteps); + Assert.True(ServerTagRules.Height(cyclic, 1) <= ServerTagRules.MaxWalkSteps); + Assert.Contains(2, ServerTagRules.Descendants(cyclic, 1)); + } +} diff --git a/Darling/Darling.Tests/ServerTagStoreLiveTests.cs b/Darling/Darling.Tests/ServerTagStoreLiveTests.cs new file mode 100644 index 0000000000..51387eb59f --- /dev/null +++ b/Darling/Darling.Tests/ServerTagStoreLiveTests.cs @@ -0,0 +1,296 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: this class mints its own scratch database through ScratchPostgres and touches nothing on the + shared one, so it is not serialized against the live-postgres collection. It seeds fixed tags, servers and + rules and reads no wall clock. */ + +/// The fleet-tag store against a real Postgres (#5085): the colour stamp, the lock, the 23505 backstop, +/// the delete cascade, the rule read and the assignment RETURNING. +public sealed class ServerTagStoreLiveTests +{ + private async Task<(ScratchPostgres Scratch, NpgsqlDataSource Source, ServerTagStore Store)?> OpenAsync() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live server-tag store test (it mints its own scratch database)."); + + var ct = TestContext.Current.CancellationToken; + var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var connectionString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { SearchPath = PgSchemaGenerator.SearchPath }.ConnectionString; + await using (var connection = new NpgsqlConnection(connectionString)) + { + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, null, ct); + } + + var source = NpgsqlDataSource.Create(connectionString); + return (scratch, source, new ServerTagStore(source, 30)); + } + + private static async Task ExecAsync(NpgsqlDataSource source, string sql) + { + await using var command = source.CreateCommand(sql); + await command.ExecuteNonQueryAsync(TestContext.Current.CancellationToken); + } + + [Fact] + public async Task CreateStampsThePaletteColour_AndTwoConcurrentSameNameCreatesGiveOneOkAndOneConflict() + { + var opened = await OpenAsync(); + var (scratch, source, store) = opened!.Value; + await using var _ = scratch; + await using var __ = source; + var ct = TestContext.Current.CancellationToken; + + var created = Assert.IsType(await store.CreateAsync("Prod", null, null, ct)); + Assert.Equal(TagColours.ForTagId(created.Tag!.Id), created.Tag.Colour); + + var results = await Task.WhenAll( + store.CreateAsync("Twin", null, null, ct), + store.CreateAsync("TWIN", null, null, ct)); + Assert.Equal(1, results.Count(r => r is ServerTagWriteResult.Ok)); + var conflict = Assert.Single(results.OfType()); + /* The lock makes the loser see the winner's committed row, so the in-memory duplicate check answers with + the precise root message; without it the loser would only be caught by the unique index. */ + Assert.Equal("A tag named 'Twin' already exists at the root.", conflict.Message.Replace("'TWIN'", "'Twin'")); + } + + [Fact] + public async Task ARawDuplicateThatBypassedTheLock_SurfacesAsConflict_OnRename() + { + var opened = await OpenAsync(); + var (scratch, source, store) = opened!.Value; + await using var _ = scratch; + await using var __ = source; + var ct = TestContext.Current.CancellationToken; + + var a = Assert.IsType(await store.CreateAsync("Alpha", null, null, ct)); + Assert.IsType(await store.CreateAsync("Beta", null, null, ct)); + + /* The in-memory check compares with ToLowerInvariant; a name that differs only by a character the two + folds treat differently would reach the index. Simulate a writer that skipped the check by renaming + straight onto the sibling's name through the unique index. */ + var edit = new ServerTagEdit(true, "Beta", false, null, false, null); + Assert.IsType(await store.UpdateAsync(a.Tag!.Id, edit, ct)); + await Assert.ThrowsAsync(() => ExecAsync(source, "UPDATE server_tags SET name = 'Beta' WHERE name = 'Alpha'")); + } + + [Fact] + public async Task Delete_CascadesTheSubtree_AndReportsTheIdsAndAssignments() + { + var opened = await OpenAsync(); + var (scratch, source, store) = opened!.Value; + await using var _ = scratch; + await using var __ = source; + var ct = TestContext.Current.CancellationToken; + + var root = Assert.IsType(await store.CreateAsync("Prod", null, null, ct)).Tag!; + var child = Assert.IsType(await store.CreateAsync("East", root.Id, null, ct)).Tag!; + await store.AssignAsync(root.Id, [9001], ct); + await store.AssignAsync(child.Id, [9001, 9002], ct); + + var deleted = Assert.IsType(await store.DeleteAsync(root.Id, confirm: true, ct)); + Assert.Equal(3, deleted.RemovedAssignments); + Assert.Equal(new[] { root.Id, child.Id }.OrderBy(i => i), deleted.RemovedTagIds!); + + var snapshot = await store.ReadSnapshotAsync(ct); + Assert.Empty(snapshot.Tags); + Assert.Empty(snapshot.Assignments); + Assert.IsType(await store.DeleteAsync(root.Id, confirm: true, ct)); + } + + [Fact] + public async Task ACreateColourWithSpaces_IsStoredTrimmedAndUpperCased() + { + var opened = await OpenAsync(); + var (scratch, source, store) = opened!.Value; + await using var _ = scratch; + await using var __ = source; + var ct = TestContext.Current.CancellationToken; + + var created = Assert.IsType(await store.CreateAsync("Padded", null, " #416fa6 ", ct)); + Assert.Equal("#416FA6", created.Tag!.Colour); + } + + [Fact] + public async Task AnUnconfirmedDelete_ReReadsTheRulesUnderTheLock_AndDeletesNothing() + { + var opened = await OpenAsync(); + var (scratch, source, store) = opened!.Value; + await using var _ = scratch; + await using var __ = source; + var ct = TestContext.Current.CancellationToken; + + var root = Assert.IsType(await store.CreateAsync("Prod", null, null, ct)).Tag!; + var child = Assert.IsType(await store.CreateAsync("East", root.Id, null, ct)).Tag!; + await store.AssignAsync(child.Id, [9001], ct); + + /* A rule scoped into the subtree after the caller's own pre-check: the store call is made directly, so + only the check under the delete lock can stop it. */ + await ExecAsync(source, $@"INSERT INTO custom_alert_rules (name, definition, enabled, version, created_at, updated_at, updated_by) VALUES + ('late-rule', '{{""scope"":{{""mode"":""tag"",""tagId"":{child.Id}}}}}', true, 1, now(), now(), 'test')"); + + var refused = Assert.IsType(await store.DeleteAsync(root.Id, confirm: false, ct)); + Assert.Equal("late-rule", Assert.Single(refused.Rules).Name); + Assert.Contains("'late-rule'", refused.Message, StringComparison.Ordinal); + + var snapshot = await store.ReadSnapshotAsync(ct); + Assert.Equal(2, snapshot.Tags.Count); + Assert.Single(snapshot.Assignments); + + Assert.IsType(await store.DeleteAsync(root.Id, confirm: true, ct)); + Assert.Empty((await store.ReadSnapshotAsync(ct)).Tags); + } + + [Fact] + public async Task AnAssignToATagDeletedInTheMeantime_IsReportedGone_NotAGenericError() + { + var opened = await OpenAsync(); + var (scratch, source, store) = opened!.Value; + await using var _ = scratch; + await using var __ = source; + var ct = TestContext.Current.CancellationToken; + + var tag = Assert.IsType(await store.CreateAsync("Doomed", null, null, ct)).Tag!; + Assert.IsType(await store.DeleteAsync(tag.Id, confirm: true, ct)); + + /* The core's pre-read is bypassed: the store is asked to assign to a tag id that is already gone, which + is what an assign racing a delete sees (a foreign-key violation). */ + var gone = await Assert.ThrowsAsync(() => store.AssignAsync(tag.Id, [9001], ct)); + Assert.Equal(tag.Id, gone.TagId); + + /* And through the core, the same race (the snapshot still lists the tag) answers not_found. */ + var racing = new RacingAssignStore(store, tag); + var json = await DarlingMcpServerTagTools.AssignServerTagCore(racing, tag.Id, [9001], ct); + Assert.Contains("\"status\":\"not_found\"", json.Replace(" ", string.Empty), StringComparison.Ordinal); + } + + /// A store whose snapshot still lists a tag that is gone and that registers every server: the state + /// an assign sees when a delete lands between its pre-read and its insert. + private sealed class RacingAssignStore(ServerTagStore inner, ServerTagRow stale) : IServerTagStore + { + public async Task ReadSnapshotAsync(System.Threading.CancellationToken ct = default) + { + var real = await inner.ReadSnapshotAsync(ct); + return real with { Tags = real.Tags.Append(stale).ToList() }; + } + + public Task> FindRegisteredServerIdsAsync(System.Collections.Generic.IReadOnlyCollection serverIds, System.Threading.CancellationToken ct = default) => + Task.FromResult>(serverIds.ToHashSet()); + + public Task CreateAsync(string name, int? parentId, string? colour, System.Threading.CancellationToken ct = default) => inner.CreateAsync(name, parentId, colour, ct); + + public Task UpdateAsync(int tagId, ServerTagEdit edit, System.Threading.CancellationToken ct = default) => inner.UpdateAsync(tagId, edit, ct); + + public Task DeleteAsync(int tagId, bool confirm, System.Threading.CancellationToken ct = default) => inner.DeleteAsync(tagId, confirm, ct); + + public Task> AssignAsync(int tagId, System.Collections.Generic.IReadOnlyList serverIds, System.Threading.CancellationToken ct = default) => inner.AssignAsync(tagId, serverIds, ct); + + public Task> UnassignAsync(int tagId, System.Collections.Generic.IReadOnlyList serverIds, System.Threading.CancellationToken ct = default) => inner.UnassignAsync(tagId, serverIds, ct); + + public Task HasChildrenAsync(int tagId, System.Threading.CancellationToken ct = default) => inner.HasChildrenAsync(tagId, ct); + + public Task ClearForServerAsync(int serverId, System.Threading.CancellationToken ct = default) => inner.ClearForServerAsync(serverId, ct); + } + + [Fact] + public async Task TagScopedRules_AreReadSkippingBadIds_AndRegisteredServersFilter() + { + var opened = await OpenAsync(); + var (scratch, source, store) = opened!.Value; + await using var _ = scratch; + await using var __ = source; + var ct = TestContext.Current.CancellationToken; + + await ExecAsync(source, @"INSERT INTO custom_alert_rules (name, definition, enabled, version, created_at, updated_at, updated_by) VALUES + ('tag-rule', '{""scope"":{""mode"":""tag"",""tagId"":7}}', true, 1, now(), now(), 'test'), + ('bad-rule', '{""scope"":{""mode"":""tag"",""tagId"":""x""}}', true, 1, now(), now(), 'test'), + ('fleet-rule', '{""scope"":{""mode"":""fleet""}}', true, 1, now(), now(), 'test')"); + + var snapshot = await store.ReadSnapshotAsync(ct); + var rule = Assert.Single(snapshot.Rules); + Assert.Equal("tag-rule", rule.Name); + Assert.Equal(7, rule.ScopeTagId); + + var found = await store.FindRegisteredServerIdsAsync([1, 2, 3], ct); + Assert.Empty(found); + } + + /// The five tool cores against a real (the product path, no fake): the + /// depth and cycle refusals, then the delete that needs confirm because a tag-scoped rule points at the + /// subtree, then the confirmed delete that leaves the rule orphaned. + [Fact] + public async Task ServerTagToolCores_OverTheRealStore_RefuseDepthAndCycle_AndGateTheDeleteOnTheRules() + { + var opened = await OpenAsync(); + var (scratch, source, store) = opened!.Value; + await using var _ = scratch; + await using var __ = source; + var ct = TestContext.Current.CancellationToken; + + static System.Text.Json.Nodes.JsonObject Parse(string json) => (System.Text.Json.Nodes.JsonObject)System.Text.Json.Nodes.JsonNode.Parse(json)!; + + var ids = new System.Collections.Generic.List(); + int? parent = null; + foreach (var name in new[] { "Level1", "Level2", "Level3", "Level4" }) + { + var made = Parse(await DarlingMcpServerTagTools.CreateServerTagCore(store, name, parent, null, ct)); + Assert.Equal("created", (string?)made["status"]); + parent = (int)made["tag"]!["tag_id"]!; + ids.Add(parent.Value); + } + + var tooDeep = Parse(await DarlingMcpServerTagTools.CreateServerTagCore(store, "Level5", ids[3], null, ct)); + Assert.Equal("depth_limit", (string?)tooDeep["refusal"]); + + var cycle = Parse(await DarlingMcpServerTagTools.UpdateServerTagCore(store, ids[0], $"{{\"parent_id\":{ids[2]}}}", ct)); + Assert.Equal("cycle", (string?)cycle["refusal"]); + + await ExecAsync(source, "INSERT INTO custom_alert_rules (name, definition, enabled, version, created_at, updated_at, updated_by) VALUES " + + $"('tag-rule', '{{\"scope\":{{\"mode\":\"tag\",\"tagId\":{ids[2]}}}}}', true, 1, now(), now(), 'test')"); + + var needsConfirm = Parse(await DarlingMcpServerTagTools.DeleteServerTagCore(store, ids[1], false, ct)); + Assert.Equal("confirm_required", (string?)needsConfirm["status"]); + Assert.Equal("tag-rule", (string?)needsConfirm["affected_rules"]![0]!["name"]); + Assert.Equal(4, (await store.ReadTagsAsync(ct)).Count); + + var deleted = Parse(await DarlingMcpServerTagTools.DeleteServerTagCore(store, ids[1], true, ct)); + Assert.Equal("deleted", (string?)deleted["status"]); + Assert.Equal("orphaned", (string?)deleted["affected_rules"]![0]!["effect"]); + Assert.Single(await store.ReadTagsAsync(ct)); + } + + [Fact] + public async Task AssignAndUnassign_ReturnExactlyTheChangedIds() + { + var opened = await OpenAsync(); + var (scratch, source, store) = opened!.Value; + await using var _ = scratch; + await using var __ = source; + var ct = TestContext.Current.CancellationToken; + + var tag = Assert.IsType(await store.CreateAsync("Prod", null, null, ct)).Tag!; + Assert.Equal([1, 2], await store.AssignAsync(tag.Id, [1, 2], ct)); + Assert.Equal([3], await store.AssignAsync(tag.Id, [2, 3], ct)); + Assert.Equal([1, 3], await store.UnassignAsync(tag.Id, [1, 3, 4], ct)); + Assert.True(await store.HasChildrenAsync(tag.Id, ct) == false); + } +} diff --git a/Darling/Darling.Tests/ServerTagViewerRoleLiveTests.cs b/Darling/Darling.Tests/ServerTagViewerRoleLiveTests.cs new file mode 100644 index 0000000000..93af6c82eb --- /dev/null +++ b/Darling/Darling.Tests/ServerTagViewerRoleLiveTests.cs @@ -0,0 +1,182 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Text.Json.Nodes; +using System.Security.Cryptography; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: this class mints its own scratch database through ScratchPostgres and a role with a per-run + name, and touches nothing on the shared one, so it is not serialized against the live-postgres collection. */ + +/// The fleet-tag writes as a non-superuser holding exactly the grants the managed viewer role gets +/// for them (#5085): the web dashboard's /api/server-tags routes run on the viewer-role pool, and every +/// other live test connects as the owner, which can write anything, so a missing grant would pass them all and +/// fail only in a real deployment with a 42501. The two grant lines are TAKEN FROM +/// 's output with the role name swapped, so removing one from +/// the product's SQL makes this test fail. +public sealed class ServerTagViewerRoleLiveTests +{ + private static readonly string RoleName = "tag_viewer_" + Guid.NewGuid().ToString("N")[..8]; + private static readonly string RolePassword = Convert.ToHexString(RandomNumberGenerator.GetBytes(16)); + + [Fact] + public async Task TheViewerRoleGrants_LetServerTagStoreCreateAssignUpdateUnassignAndDelete() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live viewer-role server-tag test (it mints its own scratch database and role)."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var ownerString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { SearchPath = PgSchemaGenerator.SearchPath }.ConnectionString; + await using var owner = NpgsqlDataSource.Create(ownerString); + await using (var connection = new NpgsqlConnection(ownerString)) + { + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, null, ct); + } + + var provisioning = DarlingManagedRoles.BuildProvisioningSql( + ProvisioningTestSecrets.Admin, ProvisioningTestSecrets.Viewer, ProvisioningTestSecrets.Mcp); + var tagGrants = provisioning + .Split('\n') + .Select(line => line.Trim()) + .Where(line => line.StartsWith("GRANT ", StringComparison.Ordinal) + && line.EndsWith(" TO " + "viewer" + ";", StringComparison.Ordinal) + && (line.Contains("config.server_tags ", StringComparison.Ordinal) || line.Contains("config.server_tag_map ", StringComparison.Ordinal))) + .ToList(); + Assert.Equal(2, tagGrants.Count); + tagGrants = tagGrants + .Select(line => line[..^(" TO " + "viewer" + ";").Length] + " TO " + RoleName + ";") + .ToList(); + + await ExecAsync(owner, $"CREATE ROLE {RoleName} LOGIN NOSUPERUSER PASSWORD '{RolePassword}'", ct); + try + { + await ExecAsync(owner, $"GRANT USAGE ON SCHEMA config TO {RoleName}; GRANT SELECT ON ALL TABLES IN SCHEMA config TO {RoleName};", ct); + foreach (var grant in tagGrants) + { + await ExecAsync(owner, grant, ct); + } + + var asViewer = new NpgsqlConnectionStringBuilder(ownerString) { Username = RoleName, Password = RolePassword }.ConnectionString; + await using var source = NpgsqlDataSource.Create(asViewer); + var store = new ServerTagStore(source, 30); + + var created = Assert.IsType(await store.CreateAsync("Prod", null, null, ct)); + var tagId = created.Tag!.Id; + Assert.Equal(new[] { 4, 5 }, await store.AssignAsync(tagId, new[] { 4, 5 }, ct)); + var updated = Assert.IsType( + await store.UpdateAsync(tagId, new ServerTagEdit(true, "Production", false, null, false, null), ct)); + Assert.Equal("Production", updated.Tag!.Name); + Assert.Equal(new[] { 4 }, await store.UnassignAsync(tagId, new[] { 4 }, ct)); + var deleted = Assert.IsType(await store.DeleteAsync(tagId, true, ct)); + Assert.Equal(1, deleted.RemovedAssignments); + Assert.Empty(await store.ReadTagsAsync(ct)); + } + finally + { + await ExecAsync(owner, $"DROP OWNED BY {RoleName}; DROP ROLE IF EXISTS {RoleName};", ct); + } + } + + /// L3 (#5137 review): the cores, not just the store, as a NOSUPERUSER role holding EXACTLY the viewer + /// role's provisioned grants. Every GRANT / REVOKE statement that names viewer is taken from + /// in order (so the column-carved SELECT on + /// config_monitored_servers applies, not a blanket SELECT), then retargeted at the scratch role. + /// Seeded as owner: one monitored server and one custom alert rule scoped to the tag. + [Fact] + public async Task TheExactViewerGrants_LetTheWebCoresCreateAssignUpdateUnassignAndDeleteWithAScopedRule() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live viewer-role server-tag test (it mints its own scratch database and role)."); + + var ct = TestContext.Current.CancellationToken; + var roleName = "tag_core_" + Guid.NewGuid().ToString("N")[..8]; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var ownerString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { SearchPath = PgSchemaGenerator.SearchPath }.ConnectionString; + await using var owner = NpgsqlDataSource.Create(ownerString); + await using (var connection = new NpgsqlConnection(ownerString)) + { + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, null, ct); + } + + var viewerStatements = ViewerGrantReplay.StatementsFor(roleName, scratch.DatabaseName); + + /* The viewer's table-wide config SELECT and its column carve must both be in what we replay. */ + Assert.Contains(viewerStatements, x => x.Contains("SELECT ON ALL TABLES IN SCHEMA config", StringComparison.Ordinal)); + Assert.Contains(viewerStatements, x => x.StartsWith("GRANT SELECT (", StringComparison.Ordinal) && x.Contains("config_monitored_servers", StringComparison.Ordinal)); + Assert.Contains(viewerStatements, x => x.EndsWith("config.server_tags " + "TO " + roleName, StringComparison.Ordinal)); + Assert.Contains(viewerStatements, x => x.EndsWith("config.server_tag_map " + "TO " + roleName, StringComparison.Ordinal)); + + await ExecAsync(owner, $"CREATE ROLE {roleName} LOGIN NOSUPERUSER PASSWORD '{RolePassword}'", ct); + try + { + foreach (var statement in viewerStatements) + { + await ExecAsync(owner, statement, ct); + } + + await using (var seed = await owner.OpenConnectionAsync(ct)) + { + await DarlingMcpTestData.RegisterServerAsync(seed, 41, "core-viewer-a", ct); + } + + var asViewer = new NpgsqlConnectionStringBuilder(ownerString) { Username = roleName, Password = RolePassword }.ConnectionString; + await using var source = NpgsqlDataSource.Create(asViewer); + var store = new ServerTagStore(source, 30); + + var created = JsonNode.Parse(await DarlingMcpServerTagTools.CreateServerTagCore(store, "Prod", null, null, ct))!; + Assert.Equal("created", created["status"]!.GetValue()); + var tagId = created["tag"]!["tag_id"]!.GetValue(); + + var assigned = JsonNode.Parse(await DarlingMcpServerTagTools.AssignServerTagCore(store, tagId, new[] { 41 }, ct))!; + Assert.Equal("assigned", assigned["status"]!.GetValue()); + + var updated = JsonNode.Parse(await DarlingMcpServerTagTools.UpdateServerTagCore(store, tagId, "{\"name\":\"Production\"}", ct))!; + Assert.Equal("updated", updated["status"]!.GetValue()); + + await using (var command = owner.CreateCommand("INSERT INTO custom_alert_rules (name, definition, enabled, version, created_at, updated_at, updated_by) VALUES " + + $"('tag-rule', '{{\"scope\":{{\"mode\":\"tag\",\"tagId\":{tagId}}}}}', true, 1, now(), now(), 'test')")) + { + await command.ExecuteNonQueryAsync(ct); + } + + var unassigned = JsonNode.Parse(await DarlingMcpServerTagTools.UnassignServerTagCore(store, tagId, new[] { 41 }, ct))!; + Assert.Equal("unassigned", unassigned["status"]!.GetValue()); + + var needsConfirm = JsonNode.Parse(await DarlingMcpServerTagTools.DeleteServerTagCore(store, tagId, false, ct))!; + Assert.Equal("confirm_required", needsConfirm["status"]!.GetValue()); + + var deleted = JsonNode.Parse(await DarlingMcpServerTagTools.DeleteServerTagCore(store, tagId, true, ct))!; + Assert.Equal("deleted", deleted["status"]!.GetValue()); + } + finally + { + await ExecAsync(owner, $"DROP OWNED BY {roleName}; DROP ROLE IF EXISTS {roleName};", ct); + } + } + + private static async Task ExecAsync(NpgsqlDataSource source, string sql, System.Threading.CancellationToken ct) + { + await using var command = source.CreateCommand(sql); + await command.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/ServerTagWebRoutesLiveTests.cs b/Darling/Darling.Tests/ServerTagWebRoutesLiveTests.cs new file mode 100644 index 0000000000..a5d2bdbf22 --- /dev/null +++ b/Darling/Darling.Tests/ServerTagWebRoutesLiveTests.cs @@ -0,0 +1,160 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Net; +using System.Net.Http; +using System.Text; +using System.Text.Json; +using System.Threading.Tasks; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.TestHost; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: this class mints its own scratch database through ScratchPostgres and touches nothing on the + shared one, so it is not serialized against the live-postgres collection. */ + +/// The server-tag web routes through a real host over a scratch store (#5085). +public sealed class ServerTagWebRoutesLiveTests +{ + private static async Task<(ScratchPostgres Scratch, NpgsqlDataSource Source)> OpenAsync() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live server-tag route tests (they mint their own scratch database)."); + var ct = TestContext.Current.CancellationToken; + var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var connectionString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { SearchPath = PgSchemaGenerator.SearchPath }.ConnectionString; + await using (var connection = new NpgsqlConnection(connectionString)) + { + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, null, ct); + } + + return (scratch, NpgsqlDataSource.Create(connectionString)); + } + + private static async Task StartAsync(NpgsqlDataSource source) + { + var builder = WebApplication.CreateBuilder(); + builder.WebHost.UseTestServer(); + builder.Logging.ClearProviders(); + var app = builder.Build(); + DarlingWebEndpoints.MapServerTags(app, source, NullLogger.Instance); + await app.StartAsync(TestContext.Current.CancellationToken); + return app; + } + + private static StringContent Json(string body, string mediaType = "application/json") => + new(body, Encoding.UTF8, mediaType); + + private static async Task<(HttpStatusCode Status, JsonElement Body)> SendAsync(HttpClient client, HttpMethod method, string url, string? body, string mediaType = "application/json") + { + var ct = TestContext.Current.CancellationToken; + using var request = new HttpRequestMessage(method, url) { Content = body is null ? Json("", mediaType) : Json(body, mediaType) }; + using var response = await client.SendAsync(request, ct); + var text = await response.Content.ReadAsStringAsync(ct); + return (response.StatusCode, JsonDocument.Parse(text).RootElement.Clone()); + } + + [Fact] + public async Task TheRoutes_AnswerCreateUpdateAssignConflictConfirmAndDelete() + { + var (scratch, source) = await OpenAsync(); + await using var _ = scratch; + await using var __ = source; + var ct = TestContext.Current.CancellationToken; + await using var app = await StartAsync(source); + using var client = app.GetTestClient(); + + var created = await SendAsync(client, HttpMethod.Post, "/api/server-tags", "{\"name\":\"Prod\"}"); + Assert.Equal(HttpStatusCode.Created, created.Status); + var tagId = created.Body.GetProperty("tag").GetProperty("tag_id").GetInt32(); + + Assert.Equal(HttpStatusCode.Conflict, (await SendAsync(client, HttpMethod.Post, "/api/server-tags", "{\"name\":\"prod\"}")).Status); + Assert.Equal(HttpStatusCode.BadRequest, (await SendAsync(client, HttpMethod.Post, "/api/server-tags", "{\"name\":\"\"}")).Status); + Assert.Equal(HttpStatusCode.BadRequest, (await SendAsync(client, HttpMethod.Post, "/api/server-tags", "{\"nope\":1}")).Status); + Assert.Equal(HttpStatusCode.UnsupportedMediaType, (await SendAsync(client, HttpMethod.Post, "/api/server-tags", "{\"name\":\"X\"}", "text/plain")).Status); + + var updated = await SendAsync(client, new HttpMethod("PATCH"), $"/api/server-tags/{tagId}", "{\"name\":\"Production\"}"); + Assert.Equal(HttpStatusCode.OK, updated.Status); + Assert.Equal(HttpStatusCode.NotFound, (await SendAsync(client, new HttpMethod("PATCH"), "/api/server-tags/9999", "{\"name\":\"Z\"}")).Status); + + await using (var connection = await source.OpenConnectionAsync(ct)) + { + await DarlingMcpTestData.RegisterServerAsync(connection, 41, "route-test-a", ct); + } + + var assigned = await SendAsync(client, HttpMethod.Post, $"/api/server-tags/{tagId}/servers", "{\"server_ids\":[41]}"); + Assert.Equal(HttpStatusCode.OK, assigned.Status); + Assert.Equal(HttpStatusCode.NotFound, (await SendAsync(client, HttpMethod.Post, $"/api/server-tags/{tagId}/servers", "{\"server_ids\":[42]}")).Status); + Assert.Equal(HttpStatusCode.BadRequest, (await SendAsync(client, HttpMethod.Post, $"/api/server-tags/{tagId}/servers", "{\"server_ids\":\"x\"}")).Status); + + /* A rule scoped to the tag turns an unconfirmed delete into a 409 that carries the rule. */ + await using (var command = source.CreateCommand("INSERT INTO custom_alert_rules (name, definition, enabled, version, created_at, updated_at, updated_by) VALUES " + + $"('tag-rule', '{{\"scope\":{{\"mode\":\"tag\",\"tagId\":{tagId}}}}}', true, 1, now(), now(), 'test')")) + { + await command.ExecuteNonQueryAsync(ct); + } + + var needsConfirm = await SendAsync(client, HttpMethod.Delete, $"/api/server-tags/{tagId}", null); + Assert.Equal(HttpStatusCode.Conflict, needsConfirm.Status); + Assert.Equal("confirm_required", needsConfirm.Body.GetProperty("status").GetString()); + Assert.True(needsConfirm.Body.GetProperty("affected_rules").GetArrayLength() > 0); + + var unassigned = await SendAsync(client, HttpMethod.Delete, $"/api/server-tags/{tagId}/servers", "{\"server_ids\":[41]}"); + Assert.Equal(HttpStatusCode.OK, unassigned.Status); + + var deleted = await SendAsync(client, HttpMethod.Delete, $"/api/server-tags/{tagId}?confirm=true", null); + Assert.Equal(HttpStatusCode.OK, deleted.Status); + Assert.Equal("deleted", deleted.Body.GetProperty("status").GetString()); + } + + [Fact] + public async Task TheRoutes_AnswerMalformedBodiesAndEdgesWith4xxNeverA500() + { + var (scratch, source) = await OpenAsync(); + await using var _ = scratch; + await using var __ = source; + await using var app = await StartAsync(source); + using var client = app.GetTestClient(); + var patch = new HttpMethod("PATCH"); + + var created = await SendAsync(client, HttpMethod.Post, "/api/server-tags", "{\"name\":\"Prod\"}"); + var tagId = created.Body.GetProperty("tag").GetProperty("tag_id").GetInt32(); + Assert.Equal(HttpStatusCode.Created, (await SendAsync(client, HttpMethod.Post, "/api/server-tags", "{\"name\":\"Other\"}")).Status); + + /* M1: a duplicate JSON key is the caller's malformed body. */ + Assert.Equal(HttpStatusCode.BadRequest, (await SendAsync(client, HttpMethod.Post, "/api/server-tags", "{\"name\":\"a\",\"name\":\"b\"}")).Status); + Assert.Equal(HttpStatusCode.BadRequest, (await SendAsync(client, patch, $"/api/server-tags/{tagId}", "{\"name\":\"a\",\"name\":\"b\"}")).Status); + Assert.Equal(HttpStatusCode.BadRequest, (await SendAsync(client, HttpMethod.Post, $"/api/server-tags/{tagId}/servers", "{\"server_ids\":[1],\"server_ids\":[2]}")).Status); + + /* M2: a body past the 64 KB cap is refused, on every body-reading route. */ + var pad = new string(' ', 70 * 1024); // valid JSON either way, so only the cap can refuse it + var big = "{\"name\":\"Padded\"}" + pad; + Assert.Equal(HttpStatusCode.BadRequest, (await SendAsync(client, HttpMethod.Post, "/api/server-tags", big)).Status); + Assert.Equal(HttpStatusCode.BadRequest, (await SendAsync(client, patch, $"/api/server-tags/{tagId}", "{\"name\":\"Prod\"}" + pad)).Status); + + /* L4: 415 on both DELETE routes, a non-"true" confirm, and a PATCH to a sibling's name. */ + Assert.Equal(HttpStatusCode.UnsupportedMediaType, (await SendAsync(client, HttpMethod.Delete, $"/api/server-tags/{tagId}", null, "text/plain")).Status); + Assert.Equal(HttpStatusCode.UnsupportedMediaType, (await SendAsync(client, HttpMethod.Delete, $"/api/server-tags/{tagId}/servers", "{\"server_ids\":[1]}", "text/plain")).Status); + Assert.Equal(HttpStatusCode.BadRequest, (await SendAsync(client, HttpMethod.Delete, $"/api/server-tags/{tagId}?confirm=1", null)).Status); + Assert.Equal(HttpStatusCode.BadRequest, (await SendAsync(client, HttpMethod.Delete, $"/api/server-tags/{tagId}?confirm=TRUE", null)).Status); + Assert.Equal(HttpStatusCode.Conflict, (await SendAsync(client, patch, $"/api/server-tags/{tagId}", "{\"name\":\"other\"}")).Status); + } +} diff --git a/Darling/Darling.Tests/SharedBaselineCacheTests.cs b/Darling/Darling.Tests/SharedBaselineCacheTests.cs index 83f9999228..831eb6bf3e 100644 --- a/Darling/Darling.Tests/SharedBaselineCacheTests.cs +++ b/Darling/Darling.Tests/SharedBaselineCacheTests.cs @@ -145,7 +145,7 @@ public void TheService_HandsOneBaselineCacheToEveryHostsAnalysis() Assert.Contains("RegisterAnalysisService(builder.Services, postgres, planFetcher, _logger, _baselineCache, config.Analyzer, _registryState)", mcp, StringComparison.Ordinal); var web = Code("Darling", "PerformanceMonitor.Darling.Service", "Mcp", "DarlingWebHostService.cs"); - Assert.Contains("DarlingWebEndpoints.MapAll(app, postgres, _collectorState, _logger, _baselineCache, postgresConfig, _readLatency, analyzerConfig, _registryState);", web, StringComparison.Ordinal); + Assert.Contains("DarlingWebEndpoints.MapAll(app, postgres, _collectorState, _logger, _baselineCache, postgresConfig, _readLatency, analyzerConfig, _registryState, _slowReads);", web, StringComparison.Ordinal); var endpoints = Code("Darling", "PerformanceMonitor.Darling.Service", "DarlingWebEndpoints.cs"); Assert.Contains("new DarlingAnalysisService(postgres, logger: logger, baselineCache: baselineCache, analyzerConfig: analyzerConfig)", endpoints, StringComparison.Ordinal); diff --git a/Darling/Darling.Tests/SharedBaselineModelPinTests.cs b/Darling/Darling.Tests/SharedBaselineModelPinTests.cs index 48c8061c5a..2b02cc1e6b 100644 --- a/Darling/Darling.Tests/SharedBaselineModelPinTests.cs +++ b/Darling/Darling.Tests/SharedBaselineModelPinTests.cs @@ -25,6 +25,8 @@ namespace Darling.Tests; /// Dashboard keeps its OWN copy on purpose (PerformanceMonitorDashboard.Analysis) — this pin is /// Lite<->Darling only. Lite.Tests carries the mirror-image pin for the Lite half. /// +[Trait("Stage", "Guard")] +[Trait("Reads", "Lite")] public sealed class SharedBaselineModelPinTests { private const string SharedAssembly = "PerformanceMonitor.Analysis"; diff --git a/Darling/Darling.Tests/ShowPlanParserParameterizedTextTests.cs b/Darling/Darling.Tests/ShowPlanParserParameterizedTextTests.cs new file mode 100644 index 0000000000..bf5a5ebba4 --- /dev/null +++ b/Darling/Darling.Tests/ShowPlanParserParameterizedTextTests.cs @@ -0,0 +1,66 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.IO; +using System.Linq; +using PerformanceMonitor.PlanAnalysis; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348 (side find): SQL Server records ParameterizedText as an ATTRIBUTE of the statement element. An ad hoc +/// shell statement has no QueryPlan, so the parser's element-only read left the plan viewer's "Parameterized" +/// row empty. The attribute is read with the other statement attributes; the element form stays as the fallback. +/// The attribute fixture is a real capture (Fixtures/StatementScrub/autoparam_select_adhoc_shell.xml). +/// +public sealed class ShowPlanParserParameterizedTextTests +{ + private const string Ns = "http://schemas.microsoft.com/sqlserver/2004/07/showplan"; + + private static string Fixture(string name) => + File.ReadAllText(Path.Combine(AppContext.BaseDirectory, "Fixtures", "StatementScrub", name)); + + private static PlanStatement OnlyStatement(string xml) => + ShowPlanParser.Parse(xml).Batches.SelectMany(b => b.Statements).Single(); + + [Fact] + public void AdhocShellStatement_ReadsTheParameterizedTextAttribute() + { + var stmt = OnlyStatement(Fixture("autoparam_select_adhoc_shell.xml")); + + Assert.Equal( + "(@1 nvarchar(4000),@2 nvarchar(4000))SELECT [id] FROM [tempdb].[dbo].[ssf_l2b] WHERE [username]=@1 AND [password]=@2", + stmt.ParameterizedText); + } + + [Fact] + public void ElementFormParameterizedText_StillParses() + { + string xml = $""" + + (@1 int)SELECT @1 + + """; + + Assert.Equal("(@1 int)SELECT @1", OnlyStatement(xml).ParameterizedText); + } + + [Fact] + public void AttributeWinsOverTheElementForm() + { + string xml = $""" + + (@1 int)ELEMENT + + """; + + Assert.Equal("(@1 int)ATTR", OnlyStatement(xml).ParameterizedText); + } +} diff --git a/Darling/Darling.Tests/SidebarSearchGroupTests.cs b/Darling/Darling.Tests/SidebarSearchGroupTests.cs new file mode 100644 index 0000000000..d7cda4540b --- /dev/null +++ b/Darling/Darling.Tests/SidebarSearchGroupTests.cs @@ -0,0 +1,249 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.ComponentModel; +using System.Diagnostics; +using System.Linq; +using System.Text.Json; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// The web sidebar's server search and tag grouping (#5243): the shipped sidebar.js and the grouping rules it +/// shares with the Fleet page (fleet-groups.js), run under Node (sidebar-harness.mjs) against a fake DOM +/// and a stubbed localStorage. Skipped when Node is not installed. +/// +public sealed class SidebarSearchGroupTests +{ + private static bool TryRun(string scenario, out JsonElement result) + { + result = default; + var psi = new ProcessStartInfo("node") { RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false }; + psi.ArgumentList.Add(PathTo("Darling", "Darling.Tests", "sidebar-harness.mjs")); + psi.ArgumentList.Add(PathTo("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js")); + psi.ArgumentList.Add(scenario); + + Process proc; + try + { + proc = Process.Start(psi)!; + } + catch (Win32Exception) + { + Assert.Skip("Node is not installed, so the shipped page script cannot be run."); + return false; + } + + using (proc) + { + var error = proc.StandardError.ReadToEndAsync(); + var output = proc.StandardOutput.ReadToEnd().Trim(); + if (!proc.WaitForExit(30000)) + { + proc.Kill(entireProcessTree: true); + Assert.Fail("the sidebar harness did not finish in 30 s for scenario " + scenario); + } + + Assert.True(proc.ExitCode == 0, "the sidebar harness failed for scenario " + scenario + ": " + error.Result); + using var doc = JsonDocument.Parse(output); + result = doc.RootElement.Clone(); + return true; + } + } + + private static string[] Strings(JsonElement array) => array.EnumerateArray().Select(e => e.ValueKind == JsonValueKind.String ? e.GetString()! : e.ToString()).ToArray(); + + private static readonly string[] AllFive = { "Alpha", "Bravo", "Charlie", "Delta", "Echo" }; + + [Fact] + public void TheSearch_MatchesDisplayName_ServerName_AndTagName_IgnoringCaseAndEdgeSpaces() + { + if (!TryRun("filter", out var r)) return; + + Assert.Equal(new[] { "Alpha" }, Strings(r.GetProperty("byDisplay"))); + Assert.Equal(new[] { "Charlie" }, Strings(r.GetProperty("byServerName"))); + Assert.Equal(new[] { "Bravo", "Echo" }, Strings(r.GetProperty("byTag"))); + Assert.Equal(new[] { "Alpha", "Bravo" }, Strings(r.GetProperty("byTagCase"))); + Assert.Equal(new[] { "Delta" }, Strings(r.GetProperty("padded"))); + Assert.Equal(AllFive, Strings(r.GetProperty("empty"))); + Assert.True(r.GetProperty("none").GetProperty("empty").GetBoolean()); + } + + [Fact] + public void TheGroupedList_IsFavourites_ThenTheTagTreeDepthFirst_ThenUntagged_AndAMultiTagServerAppearsUnderEach() + { + if (!TryRun("groupedOrder", out var r)) return; + + // Bravo is in Prod (under East) and West: it is listed under both. + Assert.Equal( + new[] { "G:East(1)@0", "S:Charlie@1", "G:Prod(2)@1", "S:Alpha@2", "S:Bravo@2", "G:West(2)@0", "S:Bravo@1", "S:Echo@1", "G:Untagged(1)@0", "S:Delta@1" }, + Strings(r.GetProperty("plain"))); + + // Favourites lead; the same servers still appear under their tags. The empty Spare tag takes no line. + Assert.Equal("G:Favourites(2)@0", Strings(r.GetProperty("withFavourites"))[0]); + Assert.Equal(new[] { "S:Delta@1", "S:Echo@1", "G:East(1)@0" }, Strings(r.GetProperty("withFavourites")).Skip(1).Take(3)); + + // Collapsing East hides its servers and the whole Prod subtree; collapsing Untagged hides only its server. + Assert.Equal( + new[] { "G:East(1,c)@0", "G:West(2)@0", "S:Bravo@1", "S:Echo@1", "G:Untagged(1)@0", "S:Delta@1" }, + Strings(r.GetProperty("collapsedEast"))); + Assert.Equal("G:Untagged(1,c)@0", Strings(r.GetProperty("collapsedUntagged")).Last()); + + // A filter keeps a parent whose child matches (East holds nothing itself), and drops the groups that have no match. + Assert.Equal( + new[] { "G:East(0)@0", "G:Prod(1)@1", "S:Bravo@2", "G:West(2)@0", "S:Bravo@1", "S:Echo@1" }, + Strings(r.GetProperty("filtered"))); + Assert.True(r.GetProperty("noMatch").GetProperty("empty").GetBoolean()); + } + + [Fact] + public void TheSearchTerm_SurvivesTheRepaint_TheNoMatchLineIsOne_AndTheGroupedToggleIsStoredByIdOnly() + { + if (!TryRun("paint", out var r)) return; + + Assert.Equal(AllFive, Strings(r.GetProperty("flat"))); + Assert.True(r.GetProperty("inputFirst").GetBoolean()); + Assert.True(r.GetProperty("inputFirstAfterRepaints").GetBoolean()); // the search box element survives every repaint (focus and caret) + Assert.Equal(new[] { "Alpha", "Bravo" }, Strings(r.GetProperty("typed"))); + + // The 60 s poll hands a fresh payload (here in another order) to the same list; the term is still applied. + Assert.Equal(new[] { "Alpha", "Bravo" }, Strings(r.GetProperty("afterPoll"))); + Assert.Equal("Prod", r.GetProperty("termAfterPoll").GetString()); + Assert.Equal(new[] { "Bravo" }, Strings(r.GetProperty("activeAfterPoll"))); + + Assert.Equal("No servers match", r.GetProperty("noMatchText").GetString()); + Assert.Equal(1, r.GetProperty("noMatchRows").GetInt32()); + Assert.Equal(AllFive, Strings(r.GetProperty("cleared"))); + + Assert.Equal( + new[] { "G:East", "S:Charlie", "G:Prod", "S:Alpha", "S:Bravo", "G:West", "S:Bravo", "S:Echo", "G:Untagged", "S:Delta" }, + Strings(r.GetProperty("groupedLines"))); + Assert.True(r.GetProperty("stored").GetProperty("grouped").GetBoolean()); + Assert.Equal(new[] { "G:East", "G:West", "S:Bravo", "S:Echo", "G:Untagged", "S:Delta" }, Strings(r.GetProperty("afterCollapse"))); + Assert.Equal(new[] { "1" }, Strings(r.GetProperty("storedAfterCollapse").GetProperty("collapsedGroups"))); + Assert.DoesNotContain("East", r.GetProperty("storedText").GetString()); + } + + [Fact] + public void TypingAfterARouteChange_PaintsTheCurrentRoutesServerAsActive_NotTheOneOfTheLastFullPaint() + { + if (!TryRun("activeRoute", out var r)) return; + + Assert.Equal(new[] { "Alpha" }, Strings(r.GetProperty("before"))); + Assert.Equal(new[] { "Echo" }, Strings(r.GetProperty("afterTyping"))); + } + + [Fact] + public void ASearch_IgnoresCollapsedGroups_SoAMatchIsNeverHidden_AndClearingItRestoresTheCollapsedView() + { + if (!TryRun("searchOpensCollapsed", out var r)) return; + + // East is collapsed (Prod, with Alpha, sits under it). + Assert.Equal(new[] { "G:East", "G:West", "S:Bravo", "S:Echo", "G:Untagged", "S:Delta" }, Strings(r.GetProperty("collapsed"))); + + // The term "alpha" shows Alpha even though East is collapsed; the stored choice is untouched. + Assert.Equal(new[] { "G:East", "G:Prod", "S:Alpha" }, Strings(r.GetProperty("searching"))); + Assert.Equal(new[] { "1" }, Strings(r.GetProperty("storedWhileSearching"))); + + // A blank term is not a search; clearing it brings the collapsed view back. + Assert.Equal(Strings(r.GetProperty("collapsed")), Strings(r.GetProperty("blankTerm"))); + Assert.Equal(Strings(r.GetProperty("collapsed")), Strings(r.GetProperty("cleared"))); + + // The shared helper: with no term the collapse still applies (the Fleet page's rule is unchanged). + Assert.Equal( + new[] { "G:East(1,c)@0", "G:West(2)@0", "S:Bravo@1", "S:Echo@1", "G:Untagged(1)@0", "S:Delta@1" }, + Strings(r.GetProperty("helperNoTerm"))); + Assert.Equal(new[] { "G:East(0)@0", "G:Prod(1)@1", "S:Alpha@2" }, Strings(r.GetProperty("helperTerm"))); + } + + [Fact] + public void AfterAHeaderToggle_FocusReturnsToTheSameGroupsHeader_AndASearchBoxKeepsItsOwnFocus() + { + if (!TryRun("headerFocus", out var r)) return; + + Assert.True(r.GetProperty("replaced").GetBoolean()); // the repaint did build a new element + Assert.True(r.GetProperty("focusIsSameGroup").GetBoolean()); + Assert.Equal("false", r.GetProperty("collapsed").GetString()); + Assert.True(r.GetProperty("secondToggle").GetBoolean()); + Assert.True(r.GetProperty("searchKeepsFocus").GetBoolean()); + } + + [Fact] + public void ACollapsedHeader_ShowsItsSubtreesDistinctServerCount_AnExpandedOneItsDirectCount() + { + if (!TryRun("collapsedCount", out var r)) return; + + // East holds Charlie directly; Prod under it holds Alpha and Bravo. + Assert.Equal(new[] { "1", "3" }, Strings(r.GetProperty("eastExpanded"))); + Assert.Equal(new[] { "1", "3" }, Strings(r.GetProperty("eastCollapsed"))); + // A parent with no servers of its own; a server in both children counts once. + Assert.Equal(new[] { "0", "2" }, Strings(r.GetProperty("parentOnly"))); + Assert.Equal("East (3)", r.GetProperty("collapsedHeader").GetString()); + Assert.Equal("East (1)", r.GetProperty("expandedHeader").GetString()); + } + + [Fact] + public void TheGroupedView_AndCollapsedGroups_RoundTripThroughStorage_WithoutDroppingTheCollapsedSidebarFlag() + { + if (!TryRun("roundTrip", out var r)) return; + + Assert.True(r.GetProperty("grouped").GetBoolean()); + Assert.Equal(new[] { "2", "untagged", "favourites" }, Strings(r.GetProperty("collapsed"))); + Assert.True(r.GetProperty("sidebarCollapsed").GetBoolean()); + Assert.Equal(1, r.GetProperty("stored").GetProperty("v").GetInt32()); + Assert.Equal(new[] { "untagged" }, Strings(r.GetProperty("afterExpand"))); + Assert.Empty(r.GetProperty("fleetKeys").EnumerateArray()); + Assert.True(r.GetProperty("storesNoNames").GetBoolean()); + } + + [Fact] + public void CorruptOrForeignStoredSidebarState_FallsBackToTheFlatListWithNothingCollapsed() + { + if (!TryRun("corruptStorage", out var r)) return; + + foreach (var name in new[] { "garbage", "arrayRoot", "foreignVersion", "wrongTypes", "badEntries" }) + { + var c = r.GetProperty("results").GetProperty(name); + Assert.False(c.GetProperty("grouped").GetBoolean(), name); + Assert.Equal(AllFive, Strings(c.GetProperty("painted"))); + } + + foreach (var name in new[] { "garbage", "arrayRoot", "foreignVersion", "wrongTypes" }) + Assert.Empty(r.GetProperty("results").GetProperty(name).GetProperty("collapsed").EnumerateArray()); + + // Entries that are not a tag id or a fixed group id are dropped; a valid one beside them is kept. + Assert.Equal(new[] { "untagged" }, Strings(r.GetProperty("results").GetProperty("badEntries").GetProperty("collapsed"))); + } + + [Fact] + public void TheFleetPage_AndTheSidebar_UseTheSharedHelpers_NeitherKeepsACopy() + { + var fleet = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "pages", "fleet.js"); + var shared = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "fleet-groups.js"); + var sidebar = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "sidebar.js"); + var app = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "app.js"); + + Assert.Contains("import { buildTagGroups, cardMatches } from \"../fleet-groups.js\";", fleet, StringComparison.Ordinal); + Assert.DoesNotContain("function buildTagGroups(", fleet, StringComparison.Ordinal); + Assert.DoesNotContain("function cardMatches(", fleet, StringComparison.Ordinal); + Assert.Contains("export function buildTagGroups(", shared, StringComparison.Ordinal); + Assert.Contains("export function cardMatches(", shared, StringComparison.Ordinal); + Assert.Contains("cardMatches(c, fleetFilter)", fleet, StringComparison.Ordinal); + + Assert.Contains("from \"./fleet-groups.js\"", sidebar, StringComparison.Ordinal); + Assert.DoesNotContain("function cardMatches(", sidebar, StringComparison.Ordinal); + Assert.Contains("paintServerList(serverList,", app, StringComparison.Ordinal); + Assert.Contains("initSidebarSearch(document.getElementById(\"server-search\"), paintSidebar);", app, StringComparison.Ordinal); + Assert.DoesNotContain("activeParam }", sidebar, StringComparison.Ordinal); // no cached active param + Assert.DoesNotContain("darling.fleet.", sidebar, StringComparison.Ordinal); + Assert.DoesNotContain("innerHTML", sidebar, StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/SingleInstanceGuardTests.cs b/Darling/Darling.Tests/SingleInstanceGuardTests.cs index 453e70cf21..94d7e98026 100644 --- a/Darling/Darling.Tests/SingleInstanceGuardTests.cs +++ b/Darling/Darling.Tests/SingleInstanceGuardTests.cs @@ -22,6 +22,7 @@ namespace Darling.Tests; /// are Windows-gated because named-mutex namespace semantics are Windows-specific. The production name and the /// release-on-dispose (SCM stop→start) contract are pinned too. /// +[Trait("Stage", "Guard")] public sealed class SingleInstanceGuardTests { private static string UniqueName() => @"Local\Darling-Test-" + Guid.NewGuid().ToString("N"); diff --git a/Darling/Darling.Tests/SlowClassCensusTests.cs b/Darling/Darling.Tests/SlowClassCensusTests.cs new file mode 100644 index 0000000000..35d71d4bd9 --- /dev/null +++ b/Darling/Darling.Tests/SlowClassCensusTests.cs @@ -0,0 +1,348 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.IO; +using System.Linq; +using System.Runtime.CompilerServices; +using System.Text.RegularExpressions; +using Xunit; + +namespace Darling.Tests; + +/// +/// Pins the list of classes that carry the Cost trait with the value Slow in Darling.Tests and Lite.Tests, and the CI wiring +/// that acts on the tag (#5459, change 5). +/// +/// What the tag does. A pull request run leaves a tagged class out unless the change reaches it +/// (ci-select.py slow-skip: the class's own source file, a file whose name the class's source mentions, a +/// script or style file for the Node-driven classes, a test-project or build input). A push to dev, a merge-queue run, +/// the nightly and a release run every class. So the tag moves a class's cost off the pull request and onto the next +/// push, which is a fair trade only for a class that is slow, has never failed, and guards nothing a pull request +/// must stop at the door. +/// +/// Why a pinned list. A class must not gain or lose the tag silently. Adding a class means adding it +/// to with the reason, in the same change; the failure message names the class. The classes in +/// were measured with the rest and stay on pull requests; tagging one fails +/// here, because each guards a security boundary, an upgrade path or a shared path. +/// +[Trait("Stage", "Guard")] +[Trait("Reads", "Lite")] +public sealed class SlowClassCensusTests +{ + private const string Darling = "darling"; + private const string Lite = "lite"; + + /// The tagged classes, each with the suite and why it may wait for the push to dev. + private static readonly Dictionary s_slow = new(StringComparer.Ordinal) + { + ["WebDataStartNoteTests"] = (Darling, "no store: the note's list and the page code under Node; 0 failures in 30 days"), + ["WebDataStartNoteLiveTests"] = (Darling, "the note's store side; 0 failures in 30 days"), + ["ViewerTextLookupWindowBoundLiveTests"] = (Darling, "a plan-shape read bound on a scratch store; 0 failures in 30 days"), + ["AdminServerEditBehaviourTests"] = (Darling, "the Admin page's edit flow under Node; 0 failures in 30 days"), + ["DeadlockProcessRowsTests"] = (Darling, "a pure read shaper; 0 failures in 30 days"), + ["ComposeHourlyRawEdgesGuardLiveTests"] = (Darling, "compose routing edges on a scratch store; 0 failures in 30 days"), + ["QueryStoreIntervalWideBelowFloorLiveTests"] = (Darling, "a Query Store read bound on a scratch store; 0 failures in 30 days"), + ["WebPerfmonMultiSelectBehaviourTests"] = (Darling, "a web picker under Node; 0 failures in 30 days"), + ["WebChartAtTimeBehaviourTests"] = (Darling, "web chart drill-downs under Node; 0 failures in 30 days"), + ["WebWaitStatsMultiSelectBehaviourTests"] = (Darling, "a web picker under Node; 0 failures in 30 days"), + ["ComposeTimeoutPhaseLiveTests"] = (Darling, "both compose timeout phases on a real store; 0 failures in 30 days"), + ["PgServerLogTailCsvJsonRotationLiveTests"] = (Darling, "log-tail rotation on dedicated clusters; 0 failures in 30 days"), + ["ProcedureTrendRunModelLiveTests"] = (Darling, "a trend read on a scratch store; 0 failures in 30 days"), + ["GridAdoptionBehaviourTests"] = (Darling, "web grid column filters under Node; 0 failures in 30 days"), + ["PlanViewerRuntimeSummaryNestingTests"] = (Darling, "WPF margins read back through reflection; 0 failures in 30 days"), + ["McpWindowNoticeSystemHealthToolTests"] = (Lite, "a window notice on one MCP tool family; 0 failures in 30 days"), + ["McpWindowNoticeConfigAndLogToolTests"] = (Lite, "a window notice on one MCP tool family; 0 failures in 30 days"), + ["McpWindowNoticeEventToolTests"] = (Lite, "a window notice on one MCP tool family; 0 failures in 30 days"), + ["McpWindowNoticeToolTests"] = (Lite, "a window notice on one MCP tool family; 0 failures in 30 days"), + ["McpWindowNoticeAggregateToolTests"] = (Lite, "a window notice on one MCP tool family; 0 failures in 30 days"), + ["ServerClockDstReadTests"] = (Lite, "four read sites across a DST change; 0 failures in 30 days"), + ["QueryStatsTextAfterRankingTests"] = (Lite, "the memory shape of two reads; 0 failures in 30 days"), + ["ProcedureStatsIdleRowsReaderTests"] = (Lite, "idle-row handling in the readers; 0 failures in 30 days"), + ["LongQueryTraceLifecycleLiteTests"] = (Lite, "trace session lifecycle on Azure SQL Database; 0 failures in 30 days"), + ["DataStartBannerSurfaceTests"] = (Lite, "the Showing-since note per surface; 0 failures in 30 days"), + ["DataStartBannerQueriesTabTests"] = (Lite, "the Showing-since note on the Queries tab; 0 failures in 30 days"), + ["BlockingChartsDataStartTests"] = (Lite, "the Showing-since note on the Blocking charts; 0 failures in 30 days"), + }; + + /// Slow classes measured with the rest that stay on pull requests, each with the reason. + private static readonly Dictionary s_keptOnPullRequests = new(StringComparer.Ordinal) + { + ["DarlingPasswordKeyStoreLiveTests"] = "passwords and keys", + ["ServerEditPasswordRuleViewerTests"] = "the password rules of the server edit", + ["DarlingStoreUpgradeTests"] = "the store upgrade path", + ["ManagedConfUpgradePathTests"] = "the managed configuration upgrade path", + ["DarlingManagedPostgresTests"] = "the managed runtime, 13 failures in 30 days", + ["AlertNotebookEndpointTests"] = "a census over the shared read-dispatch table", + ["ArchiveResetRestoresStateTests"] = "archive reset, a data-loss path; 3 failures in 30 days", + ["ArchiveGroupedWatermarkReadTests"] = "archive reads, a data-loss path", + ["StoredEventCopiesTests"] = "1 failure in 30 days", + ["FactCollectorTests"] = "1 failure in 30 days", + ["ScenarioTests"] = "6 failures in 30 days", + ["AnomalyDetectorTests"] = "4 failures in 30 days", + }; + + private static readonly Regex s_classDecl = new( + @"^[ \t]*(?:(?:public|internal|private|protected|sealed|static|abstract|partial)\s+)*class\s+(\w+)", + RegexOptions.Compiled); + + /// A line of the attribute and comment block above a declaration (the same lines ci-select.py reads). + private static readonly Regex s_attributeOrComment = new(@"^(\[|//|/\*|\*)", RegexOptions.Compiled); + + private static string Root([CallerFilePath] string thisFile = "") + => Path.GetFullPath(Path.Combine(Path.GetDirectoryName(thisFile)!, "..", "..")); + + /// {class: (suite, trait values on its declaration)} for every class of both suites that carries any + /// Cost trait; the attribute block above the declaration is read the way ci-select.py reads it. + private static Dictionary Traits)> Scan() + { + var found = new Dictionary)>(StringComparer.Ordinal); + foreach (var (suite, rel) in new[] { (Darling, Path.Combine("Darling", "Darling.Tests")), (Lite, "Lite.Tests") }) + { + var dir = Path.Combine(Root(), rel); + foreach (var file in Directory.EnumerateFiles(dir, "*.cs", SearchOption.AllDirectories) + .Where(f => !f.Contains(Path.DirectorySeparatorChar + "obj" + Path.DirectorySeparatorChar) + && !f.Contains(Path.DirectorySeparatorChar + "bin" + Path.DirectorySeparatorChar))) + { + var lines = File.ReadAllLines(file); + for (var n = 0; n < lines.Length; n++) + { + var m = s_classDecl.Match(lines[n]); + if (!m.Success) + { + continue; + } + + var traits = new HashSet(StringComparer.Ordinal); + for (var j = n - 1; j >= 0; j--) + { + var s = lines[j].Trim(); + if (!(s_attributeOrComment.IsMatch(s) || s.EndsWith("*/", StringComparison.Ordinal))) + { + break; + } + + foreach (Match t in Regex.Matches(s, "Trait\\(\\s*\"(\\w+)\"\\s*,\\s*\"(\\w+)\"\\s*\\)")) + { + traits.Add(t.Groups[1].Value + "=" + t.Groups[2].Value); + } + } + + if (found.TryGetValue(m.Groups[1].Value, out var prev)) + { + prev.Item2.UnionWith(traits); + } + else + { + found[m.Groups[1].Value] = (suite, traits); + } + } + } + } + + return found; + } + + private static (int ExitCode, string Output) RunScript(params string[] args) + { + var psi = new ProcessStartInfo("python") + { + WorkingDirectory = Root(), + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false, + }; + psi.ArgumentList.Add("-I"); + psi.ArgumentList.Add(Path.Combine(Root(), ".github", "scripts", "ci-select.py")); + foreach (var a in args) + { + psi.ArgumentList.Add(a); + } + + using var p = Process.Start(psi) ?? throw new InvalidOperationException("python did not start"); + var stdout = p.StandardOutput.ReadToEndAsync(); + var stderr = p.StandardError.ReadToEndAsync(); + Assert.True(p.WaitForExit(120_000), "ci-select.py did not finish in two minutes"); + return (p.ExitCode, stdout.Result); + } + + private static string[] Lines(string output) => + output.Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries); + + [Fact] + public void TheTaggedClasses_AreExactlyThePinnedList() + { + var tagged = Scan().Where(kv => kv.Value.Traits.Contains("Cost=Slow")).ToDictionary(kv => kv.Key, kv => kv.Value.Suite); + var pinned = s_slow.ToDictionary(kv => kv.Key, kv => kv.Value.Suite); + + var gained = tagged.Keys.Except(pinned.Keys).OrderBy(x => x, StringComparer.Ordinal).ToList(); + var lost = pinned.Keys.Except(tagged.Keys).OrderBy(x => x, StringComparer.Ordinal).ToList(); + Assert.True(gained.Count == 0, + "classes carry the Cost=Slow trait without a row in SlowClassCensusTests.s_slow (add the row with the reason, " + + "or remove the tag): " + string.Join(", ", gained)); + Assert.True(lost.Count == 0, + "classes are pinned as Cost=Slow but no longer carry the tag (restore it, or remove the row): " + string.Join(", ", lost)); + foreach (var (name, suite) in tagged) + { + Assert.True(pinned[name] == suite, name + " is pinned under the wrong suite"); + } + + Assert.True(s_slow.Count >= 20, "anti-vacuity: the pinned list shrank below 20 classes"); + } + + [Fact] + public void ATaggedClass_IsNeverAGuardClass_NorOneTheDarlingReadsLegChooses() + { + // The Guard stage runs on every pull request, and the Lite reads leg exists to run the Reads=Darling classes + // when a Darling read changes. A Slow tag on either would be dropped or contradicted by the other rule. + var scan = Scan(); + foreach (var name in s_slow.Keys) + { + var traits = scan[name].Traits; + Assert.DoesNotContain("Stage=Guard", traits); + Assert.DoesNotContain("Reads=Darling", traits); + } + } + + [Fact] + public void TheKeptClasses_AreNotTagged() + { + var scan = Scan(); + foreach (var (name, why) in s_keptOnPullRequests) + { + Assert.True(scan.ContainsKey(name), name + " is listed as kept on pull requests but no such class exists; update the list"); + Assert.False(scan[name].Traits.Contains("Cost=Slow"), name + " must stay on pull requests (" + why + ") but carries the Slow tag"); + } + } + + [Fact] + public void ThePullRequestRule_LeavesOutEveryTaggedClass_ForAnUnrelatedChange_AndNoneOtherwise() + { + foreach (var suite in new[] { Darling, Lite }) + { + var expected = s_slow.Where(kv => kv.Value.Suite == suite).Select(kv => kv.Key).OrderBy(x => x, StringComparer.Ordinal).ToArray(); + + var (code, output) = RunScript("slow-skip", "--suite", suite, "--event", "pull_request", "SECURITY.md"); + Assert.Equal(0, code); + Assert.Equal(expected, Lines(output).OrderBy(x => x, StringComparer.Ordinal).ToArray()); + + // Every event but a pull request runs every class. + foreach (var evt in new[] { "push", "merge_group", "release" }) + { + Assert.Empty(Lines(RunScript("slow-skip", "--suite", suite, "--event", evt, "SECURITY.md").Output)); + } + + // An unknown change (no file list) skips nothing. + Assert.Empty(Lines(RunScript("slow-skip", "--suite", suite, "--event", "pull_request").Output)); + + // The workflow, the selector and a project file reach every class. + foreach (var input in new[] { ".github/workflows/build.yml", "Lite/PerformanceMonitorLite.csproj" }) + { + Assert.Empty(Lines(RunScript("slow-skip", "--suite", suite, "--event", "pull_request", input).Output)); + } + } + } + + [Fact] + public void APullRequestIntoMain_SkipsNothing_AndOneIntoDevSkipsTheSameAsBefore() + { + foreach (var suite in new[] { Darling, Lite }) + { + var expected = s_slow.Where(kv => kv.Value.Suite == suite).Select(kv => kv.Key).OrderBy(x => x, StringComparer.Ordinal).ToArray(); + + // #5459: the release gate (a pull request into main) runs every class, Cost=Slow ones too. + Assert.Empty(Lines(RunScript("slow-skip", "--suite", suite, "--event", "pull_request", "--base-ref", "main", "SECURITY.md").Output)); + + // The same change into dev, and one with no base branch at all (the replay corpus has none), skip as before. + foreach (var baseRef in new[] { "dev", "" }) + { + var (code, output) = RunScript("slow-skip", "--suite", suite, "--event", "pull_request", "--base-ref", baseRef, "SECURITY.md"); + Assert.Equal(0, code); + Assert.Equal(expected, Lines(output).OrderBy(x => x, StringComparer.Ordinal).ToArray()); + } + } + } + + [Fact] + public void AChange_ThatReachesATaggedClass_PutsItBack() + { + // Its own test file. + var own = Lines(RunScript("slow-skip", "--suite", Darling, "--event", "pull_request", + "Darling/Darling.Tests/DeadlockProcessRowsTests.cs").Output); + Assert.DoesNotContain("DeadlockProcessRowsTests", own); + Assert.Contains("WebDataStartNoteTests", own); + + // A product file the test source names. + var product = Lines(RunScript("slow-skip", "--suite", Lite, "--event", "pull_request", + "Lite/Services/LocalDataService.cs").Output); + Assert.DoesNotContain("McpWindowNoticeSystemHealthToolTests", product); + Assert.Contains("LongQueryTraceLifecycleLiteTests", product); + + // A page script reaches the classes that run the shipped scripts under Node. + var script = Lines(RunScript("slow-skip", "--suite", Darling, "--event", "pull_request", + "Darling/PerformanceMonitor.Darling.Web/wwwroot/js/never-named-anywhere.js").Output); + Assert.DoesNotContain("WebPerfmonMultiSelectBehaviourTests", script); + Assert.DoesNotContain("GridAdoptionBehaviourTests", script); + Assert.Contains("ComposeTimeoutPhaseLiveTests", script); + + // A non-source file inside the test project (a fixture, a harness) reaches every class of that suite. + Assert.Empty(Lines(RunScript("slow-skip", "--suite", Darling, "--event", "pull_request", + "Darling/Darling.Tests/web-perfmon-multi-harness.mjs").Output)); + } + + [Fact] + public void TheWorkflow_AppliesTheRuleOnPullRequestsOnly_InEveryLegThatRunsTheseSuites() + { + var yml = File.ReadAllText(Path.Combine(Root(), ".github", "workflows", "build.yml")); + + // #5616: the shard cut moved into one script per suite, which build.yml's legs and nightly.yml's legs both call. + // The build job's whole-suite pass (Darling) stays inline; the Darling shards and the Lite shards are the scripts. + // The whole-tree guards job is deliberately NOT here: CrossAppGuardCiGateTests pins its invocation whole, and the + // backstop exemptions rest on it. + var darlingShard = File.ReadAllText(Path.Combine(Root(), ".github", "scripts", "run-darling-pg-shard.ps1")); + var liteShard = File.ReadAllText(Path.Combine(Root(), ".github", "scripts", "run-lite-shard.ps1")); + Assert.Single(Regex.Matches(yml, @"slow-skip --suite darling --event \$env:SLOW_EVENT ""--base-ref=\$env:SLOW_BASE_REF"" --repo \$env:SLOW_REPO --pr \$env:SLOW_PR")); + Assert.Single(Regex.Matches(darlingShard, @"slow-skip --suite darling --event \$EventName ""--base-ref=\$env:SLOW_BASE_REF"" --repo \$env:SLOW_REPO --pr \$env:SLOW_PR")); + Assert.Single(Regex.Matches(liteShard, @"slow-skip --suite lite --event \$EventName ""--base-ref=\$env:SLOW_BASE_REF"" --repo \$env:SLOW_REPO --pr \$env:SLOW_PR")); + Assert.DoesNotContain("slow-skip", File.ReadAllText(Path.Combine(Root(), ".github", "workflows", "nightly.yml")), StringComparison.Ordinal); + + // Each use sits inside a `pull_request` condition, so a push, a merge-queue run, the nightly (whose legs pass no + // event) and a release run every class. + // The test map selection (#5459) has its own pull_request condition beside it, in the build job's step and in each + // shard script. A usable selection replaces the skip, so in the scripts the skip sits in the `elseif` of the + // selection's `if`: the skip never runs on a pull request that was narrowed, and never on anything but a pull request. + Assert.Single(Regex.Matches(yml, @"\$env:SLOW_EVENT -eq 'pull_request' -and \$scopeArgs\.Count -eq 0")); + Assert.Single(Regex.Matches(yml, @"\$env:SLOW_EVENT -eq 'pull_request' -and \$env:TEST_MAP_SELECTION")); + Assert.Equal(2, Regex.Matches(yml, @"\$env:SLOW_EVENT -eq 'pull_request'").Count); + foreach (var shard in new[] { darlingShard, liteShard }) + { + Assert.Single(Regex.Matches(shard, @"\$EventName -eq 'pull_request' -and \$SelectionFile")); + Assert.Single(Regex.Matches(shard, @"elseif \(\$EventName -eq 'pull_request'\) \{")); + Assert.Equal(2, Regex.Matches(shard, @"\$EventName -eq 'pull_request'").Count); + Assert.True(shard.IndexOf("elseif ($EventName -eq 'pull_request')", StringComparison.Ordinal) + < shard.IndexOf("slow-skip --suite", StringComparison.Ordinal), "the skip must sit inside the elseif"); + } + + // The two shard steps hand the event on from the environment they set, and the nightly passes none. The build + // job's selected Darling pass (#5459) is the third caller of the Darling script, and passes it the same way. + Assert.Equal(3, Regex.Matches(yml, @"run-(darling-pg|lite)-shard\.ps1 [^\r\n]*-EventName \$env:SLOW_EVENT").Count); + + // And the environment each of the three steps reads is set from the run. + Assert.Equal(3, Regex.Matches(yml, @"SLOW_EVENT: \$\{\{ github\.event_name \}\}").Count); + Assert.Equal(3, Regex.Matches(yml, @"SLOW_PR: \$\{\{ github\.event\.pull_request\.number \}\}").Count); + Assert.Equal(3, Regex.Matches(yml, @"SLOW_REPO: \$\{\{ github\.repository \}\}").Count); + + // #5459: the base branch reaches all three slow-skip callers, so a pull request into main skips nothing. + Assert.Equal(3, Regex.Matches(yml, @"SLOW_BASE_REF: \$\{\{ github\.base_ref \}\}").Count); + Assert.Single(Regex.Matches(yml, @"slow-skip [^\r\n]*""--base-ref=\$env:SLOW_BASE_REF""")); + Assert.Single(Regex.Matches(darlingShard, @"slow-skip [^\r\n]*""--base-ref=\$env:SLOW_BASE_REF""")); + Assert.Single(Regex.Matches(liteShard, @"slow-skip [^\r\n]*""--base-ref=\$env:SLOW_BASE_REF""")); + } +} diff --git a/Darling/Darling.Tests/SlowReadLogTests.cs b/Darling/Darling.Tests/SlowReadLogTests.cs new file mode 100644 index 0000000000..0ad826737f --- /dev/null +++ b/Darling/Darling.Tests/SlowReadLogTests.cs @@ -0,0 +1,538 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; +using System.Text.Json.Nodes; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using Xunit; + +namespace Darling.Tests; + +/// +/// Pins for the slow-read record's write rule and shaping (#5097): which reads are recorded, how arguments and +/// statements are normalised and bounded, the bounded queue, and the read tool's payload trim. The threshold is a +/// process-wide seam, so the class shares the live collection and runs serially with the acceptance test that lowers it. +/// +[Collection("live-postgres")] +public sealed class SlowReadLogTests +{ + private static ReadScope NewScope() + { + using var opened = ReadScope.Open(null); + return opened.Scope; + } + + [Fact] + public void AReadIsRecorded_OverTheThreshold_OrOnATimeoutErrorOrLimit_AndNotOtherwise() + { + var log = new SlowReadLog(); + Assert.Equal(5_000, log.ThresholdMs); + + Assert.True(log.ShouldRecord(ReadOutcome.Ok, 5_000)); + Assert.False(log.ShouldRecord(ReadOutcome.Ok, 4_999)); + Assert.True(log.ShouldRecord(ReadOutcome.Timeout, 10)); + Assert.True(log.ShouldRecord(ReadOutcome.Error, 10)); + Assert.True(log.ShouldRecord(ReadOutcome.Limit, 10)); + Assert.False(log.ShouldRecord(ReadOutcome.Cancelled, 10)); + Assert.False(log.ShouldRecord(ReadOutcome.FallbackRaw, 10)); + Assert.False(log.ShouldRecord(ReadOutcome.GateFailed, 10)); + Assert.True(log.ShouldRecord(ReadOutcome.Cancelled, 6_000)); + Assert.True(log.ShouldRecord(ReadOutcome.FallbackRaw, 6_000)); + } + + [Fact] + public void Offer_QueuesASlowRead_AndNothingForAFastOkRead() + { + var log = new SlowReadLog(); + var scope = NewScope(); + + log.Offer(scope, ReadSurface.Mcp, "fast_tool", ReadOutcome.Ok, 40, null, null, null); + Assert.False(log.TryRead(out _)); + + log.Offer(scope, ReadSurface.Mcp, "slow_tool", ReadOutcome.Ok, 7_000, null, null, null); + Assert.True(log.TryRead(out var record)); + Assert.Equal("slow_tool", record!.Route); + Assert.Equal("mcp", record.Surface); + Assert.Equal("ok", record.Outcome); + Assert.Equal(7_000, record.TotalMs); + Assert.Equal(DateTimeKind.Unspecified, record.ReadTimeUtc.Kind); + } + + [Fact] + public void Arguments_AreNormalised_ServerToIdAndHoursToAWindow_AndSecretKeysBlanked() + { + var arguments = new JsonObject + { + ["server_name"] = "some-host", + ["hours_back"] = 48, + ["top"] = 20, + ["api_key"] = "do-not-store", + }; + var started = new DateTime(2026, 10, 1, 12, 0, 0, DateTimeKind.Utc); + + var (json, truncated, start, end) = SlowReadLog.NormaliseArguments(arguments, 7, started); + + Assert.False(truncated); + var parsed = JsonNode.Parse(json)!.AsObject(); + Assert.Equal(48, (int)parsed["hours_back"]!); + Assert.Equal(20, (int)parsed["top"]!); + Assert.Equal(7, (int)parsed["server_id"]!); + Assert.Equal("[redacted]", (string)parsed["api_key"]!); + Assert.False(parsed.ContainsKey("server_name")); + Assert.DoesNotContain("some-host", json, StringComparison.Ordinal); + Assert.Equal(new DateTime(2026, 10, 1, 12, 0, 0), end); + Assert.Equal(new DateTime(2026, 9, 29, 12, 0, 0), start); + } + + [Fact] + public void ARepeatedHoursKey_RecordedAsAnArray_KeepsTheWindowOfItsFirstValue() + { + var started = new DateTime(2026, 10, 1, 12, 0, 0, DateTimeKind.Utc); + + var (_, _, start, end) = SlowReadLog.NormaliseArguments( + new JsonObject { ["hours"] = new JsonArray("2", "24") }, null, started); + Assert.Equal(new DateTime(2026, 10, 1, 12, 0, 0), end); + Assert.Equal(new DateTime(2026, 10, 1, 10, 0, 0), start); + + var (_, _, backStart, _) = SlowReadLog.NormaliseArguments( + new JsonObject { ["hours_back"] = new JsonArray(6, 24) }, null, started); + Assert.Equal(new DateTime(2026, 10, 1, 6, 0, 0), backStart); + } + + [Fact] + public void ARepeatedAsOfKey_RecordedAsAnArray_EndsTheWindowAtItsFirstValue() + { + var started = new DateTime(2026, 10, 1, 12, 0, 0, DateTimeKind.Utc); + + var (_, _, start, end) = SlowReadLog.NormaliseArguments( + new JsonObject { ["hours"] = "4", ["as_of"] = new JsonArray("2026-09-30T08:00:00Z", "2026-01-01T00:00:00Z") }, + null, started); + + Assert.Equal(new DateTime(2026, 9, 30, 8, 0, 0), end); + Assert.Equal(new DateTime(2026, 9, 30, 4, 0, 0), start); + } + + [Fact] + public void APlainHoursValue_IsReadAsBefore_AndAnEmptyArrayDerivesNoWindow() + { + var started = new DateTime(2026, 10, 1, 12, 0, 0, DateTimeKind.Utc); + + var (_, _, plainStart, plainEnd) = SlowReadLog.NormaliseArguments( + new JsonObject { ["hours"] = "2" }, null, started); + Assert.Equal(new DateTime(2026, 10, 1, 10, 0, 0), plainStart); + Assert.Equal(new DateTime(2026, 10, 1, 12, 0, 0), plainEnd); + + var (_, _, emptyStart, emptyEnd) = SlowReadLog.NormaliseArguments( + new JsonObject { ["hours"] = new JsonArray() }, null, started); + Assert.Null(emptyStart); + Assert.Null(emptyEnd); + + var (_, _, noAsOfStart, noAsOfEnd) = SlowReadLog.NormaliseArguments( + new JsonObject { ["hours"] = "2", ["as_of"] = new JsonArray() }, null, started); + Assert.Equal(new DateTime(2026, 10, 1, 12, 0, 0), noAsOfEnd); + Assert.Equal(new DateTime(2026, 10, 1, 10, 0, 0), noAsOfStart); + } + + [Fact] + public void Arguments_OverFourKilobytes_AreReplacedByATruncatedObject() + { + /* Strings over 128 characters are omitted, so the size bound is reached by many short keys. */ + var arguments = new JsonObject { ["top"] = 5 }; + for (var i = 0; i < 300; i++) + { + arguments["query_" + i.ToString("D3", System.Globalization.CultureInfo.InvariantCulture)] = new string('x', 100); + } + + var (json, truncated, _, _) = SlowReadLog.NormaliseArguments(arguments, null, DateTime.UtcNow); + + Assert.True(truncated); + Assert.True(Encoding.UTF8.GetByteCount(json) <= SlowReadLog.MaxArgumentsBytes); + var parsed = JsonNode.Parse(json)!.AsObject(); + Assert.True((bool)parsed["truncated"]!); + Assert.Contains("query_000", parsed["keys"]!.AsArray().Select(k => (string)k!)); + } + + [Fact] + public void Statements_AreCappedAtFifty_AndSayMoreRan() + { + var log = new SlowReadLog(); + using var opened = ReadScope.Open(null); + var scope = opened.Scope; + scope.CaptureStatements = true; + for (var i = 0; i < 60; i++) + { + scope.AddStatement("SELECT " + i, 1.25 + i, null); + } + + log.Offer(scope, ReadSurface.Web, "route", ReadOutcome.Timeout, 100, null, "57014", null); + Assert.True(log.TryRead(out var record)); + Assert.Equal(50, JsonNode.Parse(record!.StatementsJson)!.AsArray().Count); + Assert.Equal(60, record.StatementCount); + Assert.True(record.StatementsTruncated); + Assert.Equal("57014", record.ErrorClass); + } + + [Fact] + public void TheQueue_HoldsAtMost256_DropsTheOldest_AndCountsTheDrops() + { + var log = new SlowReadLog(); + var scope = NewScope(); + for (var i = 0; i < SlowReadLog.Capacity + 10; i++) + { + log.Offer(scope, ReadSurface.Mcp, "r" + i, ReadOutcome.Error, 1, null, "X", null); + } + + Assert.Equal(10, log.Dropped); + Assert.True(log.TryRead(out var first)); + Assert.Equal("r10", first!.Route); + } + + [Fact] + public void ErrorClass_IsATypeNameOrSqlState_NeverMessageText() + { + Assert.Equal("InvalidOperationException", SlowReadLog.ErrorClassOf(new InvalidOperationException("secret text"))); + Assert.Equal("57014", SlowReadLog.ErrorClassOf(new PostgresException("canceling statement", "ERROR", "ERROR", "57014"))); + } + + private static JsonObject Stored(JsonObject arguments, out string json) + { + (json, _, _, _) = SlowReadLog.NormaliseArguments(arguments, null, DateTime.UtcNow); + return JsonNode.Parse(json)!.AsObject(); + } + + [Fact] + public void AServerBatchArgument_CarryingAPassword_IsNeverStored() + { + var arguments = new JsonObject + { + ["servers_json"] = "[{\"server_name\":\"x\",\"password\":\"Hunter2!\"}]", + }; + + var stored = Stored(arguments, out var json); + + Assert.DoesNotContain("Hunter2!", json, StringComparison.Ordinal); + Assert.DoesNotContain("password\":\"", json, StringComparison.Ordinal); + Assert.Equal(SlowReadLog.OmittedMarker, (string)stored["servers_json"]!); + } + + [Fact] + public void ASettingsJsonArgument_CarryingAWebhookSecret_IsOmitted() + { + var arguments = new JsonObject + { + ["settings_json"] = "{\"smtp\":{\"notes\":\"x\"},\"hook\":\"https://hooks.example.test/T000/B000/s3cr3tvalue\"}", + }; + + var stored = Stored(arguments, out var json); + + Assert.DoesNotContain("s3cr3tvalue", json, StringComparison.Ordinal); + Assert.Equal(SlowReadLog.OmittedMarker, (string)stored["settings_json"]!); + } + + [Fact] + public void ANestedSecretKey_InAComposeBody_IsRedactedAtAnyDepth() + { + var body = JsonNode.Parse("{\"panel\":{\"options\":{\"api_key\":\"abc123\",\"top\":5}},\"list\":[{\"Password\":\"p\"}]}")!.AsObject(); + + var stored = Stored(body, out var json); + + Assert.DoesNotContain("abc123", json, StringComparison.Ordinal); + Assert.Equal(SlowReadLog.RedactedMarker, (string)stored["panel"]!["options"]!["api_key"]!); + Assert.Equal(5, (int)stored["panel"]!["options"]!["top"]!); + Assert.Equal(SlowReadLog.RedactedMarker, (string)stored["list"]![0]!["Password"]!); + } + + [Fact] + public void AWebQueryString_RedactsTheToken_AndKeepsTheScalars() + { + var query = new JsonObject { ["token"] = "abc", ["hours_back"] = "48" }; + + var stored = Stored(query, out _); + + Assert.Equal(SlowReadLog.RedactedMarker, (string)stored["token"]!); + Assert.Equal("48", (string)stored["hours_back"]!); + } + + [Fact] + public void NumbersBooleansAndShortStrings_AreKept() + { + var stored = Stored(new JsonObject { ["hours_back"] = 48, ["top"] = 20, ["flag"] = true, ["view"] = "waits" }, out _); + + Assert.Equal(48, (int)stored["hours_back"]!); + Assert.Equal(20, (int)stored["top"]!); + Assert.True((bool)stored["flag"]!); + Assert.Equal("waits", (string)stored["view"]!); + } + + [Fact] + public void AnyNonJsonString_OfMoreThan128Characters_IsOmitted_AndExactly128IsKept() + { + var stored = Stored(new JsonObject { ["long"] = new string('a', 129), ["edge"] = new string('b', 128) }, out _); + + Assert.Equal(SlowReadLog.OmittedMarker, (string)stored["long"]!); + Assert.Equal(128, ((string)stored["edge"]!).Length); + } + + [Fact] + public void AnyShortStringWithABrace_OrBracket_IsOmitted_ParsedOrNot() + { + var stored = Stored(new JsonObject { ["a"] = "{\"k\":1}", ["b"] = "[1,2]", ["c"] = "[not json", ["d"] = "{also not" }, out _); + + Assert.Equal(SlowReadLog.OmittedMarker, (string)stored["a"]!); + Assert.Equal(SlowReadLog.OmittedMarker, (string)stored["b"]!); + Assert.Equal(SlowReadLog.OmittedMarker, (string)stored["c"]!); + Assert.Equal(SlowReadLog.OmittedMarker, (string)stored["d"]!); + } + + [Fact] + public void Nesting_StopsAtDepthEight_AndAnArrayAtFiftyElements() + { + JsonNode inner = new JsonObject { ["leaf"] = 1 }; + for (var i = 0; i < 10; i++) + { + inner = new JsonObject { ["n"] = inner }; + } + + var many = new JsonArray(); + for (var i = 0; i < 60; i++) + { + many.Add(i); + } + + var stored = Stored(new JsonObject { ["deep"] = inner, ["many"] = many }, out var json); + + Assert.Contains("\"[omitted]\"", json, StringComparison.Ordinal); + Assert.DoesNotContain("leaf", json, StringComparison.Ordinal); + var array = stored["many"]!.AsArray(); + Assert.Equal(51, array.Count); + Assert.Equal("[omitted 10 more]", (string)array[50]!); + } + + [Theory] + [InlineData("password")] + [InlineData("passwd")] + [InlineData("pwd")] + [InlineData("client_secret")] + [InlineData("access_token")] + [InlineData("apikey")] + [InlineData("api_key")] + [InlineData("license_key")] + [InlineData("credential")] + [InlineData("connectionstring")] + [InlineData("connection_string")] + [InlineData("authorization")] + public void EverySecretFragment_IsRedacted(string key) + { + var stored = Stored(new JsonObject { [key] = "v", ["x"] = new JsonObject { [key.ToUpperInvariant()] = "v" } }, out _); + + Assert.Equal(SlowReadLog.RedactedMarker, (string)stored[key]!); + Assert.Equal(SlowReadLog.RedactedMarker, (string)stored["x"]![key.ToUpperInvariant()]!); + } + + [Fact] + public void TheStatementCapture_ReadsOnlyTheQueryTextAndReturnedRowsTags() + { + var source = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "ReadStatementCapture.cs"); + var tagKeys = System.Text.RegularExpressions.Regex.Matches(source, "\"(db\\.[A-Za-z0-9_.]+)\"") + .Select(m => m.Groups[1].Value).Distinct().OrderBy(k => k, StringComparer.Ordinal).ToArray(); + + Assert.Equal(new[] { "db.query.text", "db.response.returned_rows" }, tagKeys); + Assert.DoesNotContain("data_source", source, StringComparison.Ordinal); + } + + [Fact] + public void ABuiltRecord_CarriesTheRowCountTheReadNoted() + { + using var opened = ReadScope.Open(null); + ReadScope.NoteRows(1234); + + var record = SlowReadLog.Build(opened.Scope, ReadSurface.Mcp, "r", ReadOutcome.Ok, 9_000, null, null, DateTime.UtcNow); + Assert.Equal(1234L, record.RowCount); + + using var none = ReadScope.Open(null); + Assert.Null(SlowReadLog.Build(none.Scope, ReadSurface.Mcp, "r", ReadOutcome.Ok, 9_000, null, null, DateTime.UtcNow).RowCount); + } + + [Fact] + public void ThePurge_RunsFirst_ThenAtMostOncePerMinuteOrPerHundredInserts() + { + var log = new SlowReadLog(); + var t0 = new DateTime(2026, 10, 1, 12, 0, 0, DateTimeKind.Utc); + + Assert.True(log.PurgeDue(t0)); + for (var i = 0; i < 50; i++) + { + Assert.False(log.PurgeDue(t0.AddSeconds(1))); + } + + Assert.True(log.PurgeDue(t0.AddSeconds(61)), "a minute on"); + for (var i = 1; i < SlowReadLog.PurgeEveryInserts; i++) + { + Assert.False(log.PurgeDue(t0.AddSeconds(62))); + } + + Assert.True(log.PurgeDue(t0.AddSeconds(62)), "the hundredth insert"); + } + + [Fact] + public void TheThreshold_IsInjectedPerInstance() + { + var instant = new SlowReadLog(0); + var normal = new SlowReadLog(); + + Assert.True(instant.ShouldRecord(ReadOutcome.Ok, 0)); + Assert.False(normal.ShouldRecord(ReadOutcome.Ok, 0)); + } + + private static DarlingSlowReadReader.SlowReadRow Row(long id, int statements) + { + var array = new JsonArray(); + for (var i = 1; i <= statements; i++) + { + array.Add(new JsonObject { ["ordinal"] = i, ["label"] = new string('q', 120), ["hash"] = "abcd1234", ["ms"] = 10.25 * i, ["rows"] = null }); + } + + return new DarlingSlowReadReader.SlowReadRow( + id, new DateTime(2026, 10, 1, 0, 0, 0).AddSeconds(id), "mcp", "get_x", "timeout", 9_000, null, null, null, null, + new JsonObject { ["top"] = 20 }, false, "raw", null, array, statements, false, "57014"); + } + + [Fact] + public void TheResponse_FitsTheBudget_DropsRowsFromTheEnd_AndSaysSo() + { + var reads = Enumerable.Range(1, 100).Select(i => Row(i, 50)).ToList(); + var page = new DarlingSlowReadReader.Page(reads, new[] { new DarlingSlowReadReader.SummaryRow("mcp", "get_x", "timeout", 100) }); + + var json = DarlingMcpSlowReadTools.BuildResponse(24, null, null, null, page, 100, fullDetail: true, McpResponseBudgetBytes); + + Assert.True(Encoding.UTF8.GetByteCount(json) <= McpResponseBudgetBytes, "the payload is over the budget"); + var parsed = JsonNode.Parse(json)!.AsObject(); + Assert.True((bool)parsed["truncated"]!); + var returned = (int)parsed["reads_returned"]!; + Assert.InRange(returned, 1, 99); + Assert.Equal(100, (int)parsed["reads_total"]!); + /* Newest first: the dropped rows are the OLDEST, so the first row kept is the newest id. */ + Assert.Equal(returned, parsed["reads"]!.AsArray().Count); + Assert.Contains("2026-10-01T00:01:40", (string)parsed["reads"]![0]!["read_time"]!, StringComparison.Ordinal); + } + + [Fact] + public void TheDefaultRowShowsTheFiveSlowestStatements_SlowestFirst_AndCountsTheOmitted() + { + var shaped = DarlingMcpSlowReadTools.ShapeStatements(Row(1, 8).Statements, fullDetail: false); + Assert.Equal(5, shaped.Shown.Count); + Assert.Equal(3, shaped.Omitted); + var first = JsonNode.Parse(System.Text.Json.JsonSerializer.Serialize(shaped.Shown[0]))!; + Assert.Equal(8, (int)first["ordinal"]!); + Assert.Equal(82.0, (double)first["ms"]!); + + var full = DarlingMcpSlowReadTools.ShapeStatements(Row(1, 8).Statements, fullDetail: true); + Assert.Equal(8, full.Shown.Count); + Assert.Equal(0, full.Omitted); + } + + [Fact] + public async Task AWriteThatFails_IsSwallowed() + { + await using var dead = NpgsqlDataSource.Create("Host=127.0.0.1;Port=1;Username=x;Password=x;Timeout=1"); + var log = new SlowReadLog(); + var scope = NewScope(); + log.Offer(scope, ReadSurface.Mcp, "r", ReadOutcome.Error, 1, null, "X", null); + Assert.True(log.TryRead(out var record)); + + await log.StoreAsync(dead, record!, null); + } + + [Fact] + public void ABomPrefixedOrTrailingCommaJsonString_IsOmitted() + { + var stored = Stored(new JsonObject { ["a"] = "\uFEFF{\"password\":\"x\"}", ["b"] = "{\"password\":\"x\",}", ["c"] = "/*c*/{\"a\":1}" }, out _); + + Assert.Equal(SlowReadLog.OmittedMarker, (string)stored["a"]!); + Assert.Equal(SlowReadLog.OmittedMarker, (string)stored["b"]!); + Assert.Equal(SlowReadLog.OmittedMarker, (string)stored["c"]!); + } + + [Fact] + public void AUrlCredentialOrPasswordBearingString_IsOmitted_AndAPlainUrlIsKept() + { + var stored = Stored(new JsonObject { ["a"] = "https://u:p@h/x", ["b"] = "Server=a;Password=b", ["c"] = "Server=a;PWD = b", ["d"] = "https://h/x" }, out _); + + Assert.Equal(SlowReadLog.OmittedMarker, (string)stored["a"]!); + Assert.Equal(SlowReadLog.OmittedMarker, (string)stored["b"]!); + Assert.Equal(SlowReadLog.OmittedMarker, (string)stored["c"]!); + Assert.Equal("https://h/x", (string)stored["d"]!); + } + + [Fact] + public void DedupKey_IsKept_WhileApiKeyIsRedacted_AndOrdinaryArgumentsSurvive() + { + var stored = Stored(new JsonObject { ["dedup_key"] = "abc", ["api_key"] = "abc", ["hours_back"] = 4, ["top"] = 5, ["server_name"] = "s" }, out _); + + Assert.Equal("abc", (string)stored["dedup_key"]!); + Assert.Equal(SlowReadLog.RedactedMarker, (string)stored["api_key"]!); + Assert.Equal(4, (int)stored["hours_back"]!); + Assert.Equal(5, (int)stored["top"]!); + } + + [Fact] + public void AWebQueryKeyWithOddCharacters_IsDroppedAndCounted_AndALongKeyIsCappedAt64() + { + var args = DarlingWebEndpoints.WebQueryArguments(new[] + { + new KeyValuePair("bad key!", "v"), + new KeyValuePair(new string('k', 100), "v"), + new KeyValuePair("top", "5"), + }); + + Assert.False(args.ContainsKey("bad key!")); + Assert.Equal(1, (int)args["_dropped_keys"]!); + Assert.True(args.ContainsKey(new string('k', 64))); + Assert.False(args.ContainsKey(new string('k', 100))); + Assert.Equal("5", (string)args["top"]!); + Assert.False(DarlingWebEndpoints.WebQueryArguments(new[] { new KeyValuePair("top", "5") }).ContainsKey("_dropped_keys")); + } + + /// A repeated key is recorded as a JSON array of its values, in the order sent; a key sent once stays a + /// plain string, and the 64-character key rule still applies to a repeated key (#5245). + [Fact] + public void ARepeatedWebQueryKey_KeepsEveryValueAsAnArray_AndASingleKeyIsUnchanged() + { + var args = DarlingWebEndpoints.WebQueryArguments(new[] + { + new KeyValuePair("database_name", "A"), + new KeyValuePair("top", "5"), + new KeyValuePair("database_name", "B"), + new KeyValuePair("server", ""), + new KeyValuePair("server", "S2"), + new KeyValuePair(new string('k', 100), "x"), + new KeyValuePair(new string('k', 100), "y"), + }); + + var databases = Assert.IsType(args["database_name"]); + Assert.Equal(new[] { "A", "B" }, databases.Select(v => (string?)v).ToArray()); + Assert.Equal(new[] { "", "S2" }, Assert.IsType(args["server"]).Select(v => (string?)v).ToArray()); + Assert.Equal(new[] { "x", "y" }, Assert.IsType(args[new string('k', 64)]).Select(v => (string?)v).ToArray()); + Assert.Equal("5", Assert.IsAssignableFrom(args["top"]).GetValue()); + } + + [Fact] + public void AnUnknownComposeSource_RecordsComposeUnknown_AndAKnownOneKeepsItsName() + { + Assert.Equal("compose:unknown", DarlingWebEndpoints.ComposeRouteLabel(new JsonObject { ["panel"] = new JsonObject { ["source"] = new string('x', 500) } })); + Assert.Equal("compose:wait_stats", DarlingWebEndpoints.ComposeRouteLabel(new JsonObject { ["panel"] = new JsonObject { ["source"] = "wait_stats" } })); + } + + private const int McpResponseBudgetBytes = 32 * 1024; +} diff --git a/Darling/Darling.Tests/SlowReadsLiveTests.cs b/Darling/Darling.Tests/SlowReadsLiveTests.cs new file mode 100644 index 0000000000..9defc01e13 --- /dev/null +++ b/Darling/Darling.Tests/SlowReadsLiveTests.cs @@ -0,0 +1,269 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Net; +using System.Text; +using System.Text.Json.Nodes; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.TestHost; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using PerformanceMonitor.Darling.Analysis; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// Live pins for the slow-read record (#5097): the acceptance path (a slowed get_query_store_top call is read back +/// with get_slow_reads, arguments and statements intact), the writer's retention and row cap, and the read tool's +/// empty answer. Each fact mints its own scratch database. +/// +[Collection("live-postgres")] +public sealed class SlowReadsLiveTests +{ + private const string Token = "slow-read-pin-token"; + private const int ServerId = 5097001; + private const string ServerName = "slow-read-pin"; + + private static string? BaseConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static async Task OpenMigratedAsync(ScratchPostgres scratch, CancellationToken ct) + { + var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + return connection; + } + + private static async Task RegisterServerAsync(NpgsqlConnection connection, CancellationToken ct) + { + await using var command = new NpgsqlCommand(@" +INSERT INTO servers (server_id, server_name, display_name, is_enabled, sql_major_version, created_date, modified_date) +VALUES ($1, $2, $2, TRUE, 16, now(), now()) +ON CONFLICT (server_id) DO UPDATE SET is_enabled = TRUE;", connection); + command.Parameters.Add(new NpgsqlParameter { TypedValue = ServerId }); + command.Parameters.Add(new NpgsqlParameter { TypedValue = ServerName }); + await command.ExecuteNonQueryAsync(ct); + } + + private static SlowReadRecord Record(DateTime time, string route) => new( + time, "mcp", route, "timeout", 9_000, null, null, null, "{}", false, null, null, "[]", 0, false, "57014"); + + private static async Task BuildServerAsync(NpgsqlDataSource postgres, SlowReadLog slowReads) + { + var builder = WebApplication.CreateBuilder(); + builder.WebHost.UseTestServer(); + builder.Logging.ClearProviders(); + builder.Services.AddSingleton(postgres); + var sharedBaselines = new BaselineCache(); + builder.Services.AddTransient(_ => new DarlingAnalysisService( + postgres, planFetcher: null, logger: NullLogger.Instance, baselineCache: sharedBaselines)); + builder.Services.AddSingleton(NullLogger.Instance); + + DarlingMcpHostService.ConfigureMcpServices( + builder.Services, DarlingPeerDirectory.Snapshot.Empty, new ReadLatencyAccumulator(), NullLogger.Instance, slowReads); + + var app = builder.Build(); + var host = new DarlingMcpHostService(NullLogger.Instance, new McpRuntimeState(), new MonitoredServerRegistryState()); + host.ConfigurePipeline(app, networkMode: false, networkListenIp: null, allowedCidr: System.Net.IPNetwork.Parse("127.0.0.1/32"), bearerToken: Token); + await app.StartAsync(); + return app.GetTestServer(); + } + + private static async Task CallToolAsync(TestServer server, string tool, JsonObject arguments) + { + var request = new JsonObject + { + ["jsonrpc"] = "2.0", + ["id"] = 1, + ["method"] = "tools/call", + ["params"] = new JsonObject { ["name"] = tool, ["arguments"] = arguments }, + }.ToJsonString(); + + var ctx = await server.SendAsync(c => + { + c.Request.Method = "POST"; + c.Request.Path = "/"; + c.Request.Headers.Host = "localhost"; + c.Request.Headers.Accept = "application/json, text/event-stream"; + c.Request.ContentType = "application/json"; + var bytes = Encoding.UTF8.GetBytes(request); + c.Request.Body = new System.IO.MemoryStream(bytes); + c.Request.ContentLength = bytes.Length; + c.Connection.RemoteIpAddress = IPAddress.Loopback; + }); + + using var reader = new System.IO.StreamReader(ctx.Response.Body); + var body = await reader.ReadToEndAsync(); + var data = body.Split('\n').Select(l => l.TrimEnd('\r')).FirstOrDefault(l => l.StartsWith("data:", StringComparison.Ordinal)); + var envelope = JsonNode.Parse(data is null ? body : data["data:".Length..].Trim())!.AsObject(); + var text = (string)envelope["result"]!["content"]![0]!["text"]!; + return JsonNode.Parse(text)!.AsObject(); + } + + [Fact] + public async Task ASlowedQueryStoreTopCall_IsRecordedWithItsArguments_SourceAndStatements_AndReadBackByGetSlowReads() + { + var baseCs = BaseConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(baseCs), "Set DARLING_TEST_PG to a Postgres connection string to run the #5097 slow-read live test."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseCs!, ct); + await using var connection = await OpenMigratedAsync(scratch, ct); + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + await RegisterServerAsync(connection, ct); + + var bodySucceeded = false; + var log = new SlowReadLog(0); + using var writerStop = new CancellationTokenSource(); + Task? writer = null; + TestServer? server = null; + try + { + /* Program.cs registers the process listener; a test host does not. */ + ReadStatementCapture.Register(); + writer = log.RunAsync(postgres, null, writerStop.Token); + server = await BuildServerAsync(postgres, log); + + await CallToolAsync(server, "get_query_store_top", new JsonObject { ["server_name"] = ServerName, ["hours_back"] = 48, ["top"] = 20 }); + + JsonObject? answer = null; + for (var attempt = 0; attempt < 50; attempt++) + { + answer = await CallToolAsync(server, "get_slow_reads", new JsonObject { ["route"] = "get_query_store_top" }); + if (answer["reads"] is JsonArray { Count: > 0 }) + { + break; + } + + await Task.Delay(100, ct); + } + + var read = Assert.Single(answer!["reads"]!.AsArray())!.AsObject(); + Assert.Equal("mcp", (string)read["surface"]!); + Assert.Equal(48, (int)read["arguments"]!["hours_back"]!); + Assert.Equal(20, (int)read["arguments"]!["top"]!); + Assert.Equal(ServerName, (string)read["server_name"]!); + Assert.Equal(ServerId, (int)read["arguments"]!["server_id"]!); + Assert.False(read["arguments"]!.AsObject().ContainsKey("server_name")); + Assert.Contains((string)read["source"]!, new[] { "raw", "interval_table" }); + var statements = read["statements"]!.AsArray(); + Assert.NotEmpty(statements); + Assert.All(statements, s => Assert.True((double)s!["ms"]! >= 0)); + Assert.True((int)read["statement_count"]! >= statements.Count); + var storedStatements = statements.ToJsonString(); + foreach (var leak in new[] { "Host=", "Username=", "Password=" }) + { + Assert.DoesNotContain(leak, storedStatements, StringComparison.Ordinal); + } + + Assert.NotNull(read["rows"]); + Assert.True((long)read["rows"]! >= 0); + Assert.EndsWith("Z", (string)read["read_time"]!, StringComparison.Ordinal); + Assert.NotNull(read["window_start"]); + + bodySucceeded = true; + } + finally + { + server?.Dispose(); + writerStop.Cancel(); + if (writer is not null) + { + await writer; + } + + await LiveStoreCleanupFinishAsync(scratch, bodySucceeded); + } + } + + private static Task LiveStoreCleanupFinishAsync(ScratchPostgres scratch, bool bodySucceeded) => + LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, static (_, _) => Task.CompletedTask); + + [Fact] + public async Task TheWriter_KeepsTheNewestTenThousandRows_AndPurgesRowsOlderThanThirtyDays() + { + var baseCs = BaseConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(baseCs), "Set DARLING_TEST_PG to a Postgres connection string to run the #5097 slow-read live test."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseCs!, ct); + await using var connection = await OpenMigratedAsync(scratch, ct); + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + + var bodySucceeded = false; + try + { + var now = new DateTime(2026, 10, 1, 12, 0, 0, DateTimeKind.Unspecified); + await using (var seed = new NpgsqlCommand(@" +INSERT INTO collect.slow_reads (read_time, surface, route, outcome, total_ms, arguments, arguments_truncated, statements, statement_count, statements_truncated) +SELECT TIMESTAMP '2026-10-01 11:00:00' - make_interval(secs => g), 'mcp', 'seed', 'timeout', 9000, '{}'::jsonb, FALSE, '[]'::jsonb, 0, FALSE +FROM generate_series(1, 10000) AS g; +INSERT INTO collect.slow_reads (read_time, surface, route, outcome, total_ms, arguments, arguments_truncated, statements, statement_count, statements_truncated) +VALUES (TIMESTAMP '2026-08-01 00:00:00', 'mcp', 'ancient', 'timeout', 9000, '{}'::jsonb, FALSE, '[]'::jsonb, 0, FALSE);", connection)) + { + await seed.ExecuteNonQueryAsync(ct); + } + + await new SlowReadLog().StoreAsync(postgres, Record(now, "newest"), null); + + await using var count = new NpgsqlCommand("SELECT count(*) FROM collect.slow_reads", connection); + Assert.Equal((long)SlowReadLog.RowCap, await count.ExecuteScalarAsync(ct)); + await using var ancient = new NpgsqlCommand("SELECT count(*) FROM collect.slow_reads WHERE route = 'ancient'", connection); + Assert.Equal(0L, await ancient.ExecuteScalarAsync(ct)); + await using var newest = new NpgsqlCommand("SELECT count(*) FROM collect.slow_reads WHERE route = 'newest'", connection); + Assert.Equal(1L, await newest.ExecuteScalarAsync(ct)); + await using var oldest = new NpgsqlCommand("SELECT min(read_time) FROM collect.slow_reads", connection); + Assert.Equal(new DateTime(2026, 10, 1, 11, 0, 0).AddSeconds(-9_999), (DateTime)(await oldest.ExecuteScalarAsync(ct))!); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanupFinishAsync(scratch, bodySucceeded); + } + } + + [Fact] + public async Task GetSlowReads_OnAnEmptyStore_AnswersEmpty_AndRefusesABadSurface() + { + var baseCs = BaseConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(baseCs), "Set DARLING_TEST_PG to a Postgres connection string to run the #5097 slow-read live test."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseCs!, ct); + await using var connection = await OpenMigratedAsync(scratch, ct); + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + + var bodySucceeded = false; + try + { + var empty = await DarlingMcpSlowReadTools.GetSlowReads(postgres, cancellationToken: ct); + Assert.Equal("empty", (string)JsonNode.Parse(empty)!["status"]!); + + var refused = await DarlingMcpSlowReadTools.GetSlowReads(postgres, surface: "nope", cancellationToken: ct); + Assert.Contains("surface", refused, StringComparison.Ordinal); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanupFinishAsync(scratch, bodySucceeded); + } + } +} diff --git a/Darling/Darling.Tests/SlowReadsRungTests.cs b/Darling/Darling.Tests/SlowReadsRungTests.cs new file mode 100644 index 0000000000..8fdeba1830 --- /dev/null +++ b/Darling/Darling.Tests/SlowReadsRungTests.cs @@ -0,0 +1,171 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Reflection; +using System.Text.RegularExpressions; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// Pins the Darling rung that adds collect.slow_reads (#5097), the record of reads that ran long or ended in a +/// timeout, error or limit. Internal self-telemetry: one plain table and one index, every statement guarded by +/// IF NOT EXISTS, nothing that alters or removes existing objects. Every fact finds the rung by NAME. The live facts run +/// when DARLING_TEST_PG is set, each on its own scratch store. +/// +[Collection("live-postgres")] +public sealed class SlowReadsRungTests +{ + public const string RungName = "slow-reads"; + + private const string Table = "slow_reads"; + + public static int RungVersion => Rung.Version; + + /// This rung's sentinel; the ordinal is a fact of the probe's shape, and a newer rung's sentinel follows it. + private const int ProbeOrdinal = 137; + + private const string SkipText = "Set DARLING_TEST_PG to a Postgres connection string to run the slow-read table's live pins (each mints its own scratch database)."; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static PgMigrations.Migration Rung => PgMigrations.Scripts.Single(m => m.Name == RungName); + + private static string Statements() => Rung.Sql.Replace("\r\n", "\n", StringComparison.Ordinal); + + [Fact] + public void TheRungIsRegisteredInADenseLadder_AtVersion162() + { + var versions = PgMigrations.Scripts.Select(s => s.Version).ToList(); + + Assert.Equal(162, Rung.Version); + /* No longer the top rung: V163 (the store's statement history) landed above it. */ + Assert.True(Rung.Version < StorageVersion.SchemaVersion); + Assert.Equal(StorageVersion.SchemaVersion, PgMigrations.Scripts[^1].Version); + Assert.Equal(versions.Distinct().OrderBy(v => v), versions); + Assert.Contains(Rung.Version - 1, versions); + } + + [Fact] + public void TheSql_CreatesTheTableAndTheTimeIndex_AndOnlyCreatesWithGuards() + { + var sql = Statements(); + + Assert.Contains("CREATE TABLE IF NOT EXISTS collect.slow_reads\n", sql, StringComparison.Ordinal); + Assert.Contains(" error_class text,\n row_count bigint\n);", sql, StringComparison.Ordinal); + Assert.Contains("CREATE INDEX IF NOT EXISTS idx_slow_reads_time\n ON collect.slow_reads(read_time, slow_read_id);", sql, StringComparison.Ordinal); + Assert.DoesNotContain("create_hypertable", sql, StringComparison.OrdinalIgnoreCase); + Assert.DoesNotContain("CHECK", sql, StringComparison.Ordinal); + + foreach (var create in Regex.Matches(sql, @"CREATE\s+(?:UNIQUE\s+)?(TABLE|INDEX)\s+(?!IF NOT EXISTS)", RegexOptions.IgnoreCase).Cast()) + { + Assert.Fail("an unguarded CREATE: " + create.Value); + } + + foreach (var verb in new[] { "ALTER ", "UPDATE ", "DELETE ", "GRANT ", "DROP ", "INSERT " }) + { + Assert.DoesNotContain(verb, sql, StringComparison.OrdinalIgnoreCase); + } + } + + [Fact] + public void TheProbeCarriesTheTable_AndMapsAStoreThroughThisRungToThisRung() + { + var probe = ViewerDataService.StoreSchemaProbeSql.Replace("\r\n", "\n", StringComparison.Ordinal); + var arm = $"to_regclass('collect.{Table}') IS NOT NULL"; + Assert.Contains(arm, probe, StringComparison.Ordinal); + + var viewer = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.cs"); + Assert.Contains($"reader.GetBoolean({ProbeOrdinal})", viewer, StringComparison.Ordinal); + Assert.Contains($"reader.GetBoolean({ProbeOrdinal + 1})", viewer, StringComparison.Ordinal); + + var method = typeof(ViewerDataService).GetMethod("MapProbedSchemaVersion", BindingFlags.NonPublic | BindingFlags.Static)!; + var arity = method.GetParameters().Length; + Assert.True(ProbeOrdinal < arity - 1, "a newer rung's sentinel follows this one"); + Assert.Equal("hasSlowReads", method.GetParameters()[ProbeOrdinal].Name); + + var all = Enumerable.Range(0, arity).Select(i => (object)(i <= ProbeOrdinal)).ToArray(); + Assert.Equal(Rung.Version, (int)method.Invoke(null, all)!); + Assert.Equal(StorageVersion.SchemaVersion, ViewerDataService.RequiredStoreSchemaVersion); + + var behind = (object[])all.Clone(); + behind[ProbeOrdinal] = false; + Assert.Equal(Rung.Version - 1, (int)method.Invoke(null, behind)!); + + var thisArm = viewer.IndexOf("if (hasSlowReads)", StringComparison.Ordinal); + var previousArm = viewer.IndexOf("if (hasQueryStoreTopDaily)", StringComparison.Ordinal); + Assert.True(thisArm >= 0, "no sentinel arm: a fully-migrated store would map one rung short"); + Assert.True(thisArm < previousArm, "this arm sits above the previous rung's, so a current store maps one rung short"); + Assert.Contains($"return {Rung.Version};", viewer[thisArm..previousArm], StringComparison.Ordinal); + + /* The V71 finding: the table is named in the probe line and nowhere in the arm's prose. */ + var armProseStart = viewer.LastIndexOf("/* V162 (#5097)", thisArm, StringComparison.Ordinal); + Assert.True(armProseStart >= 0, "the arm has no comment block saying why it exists"); + Assert.DoesNotContain(Table, viewer[armProseStart..thisArm], StringComparison.Ordinal); + } + + [Fact] + public async Task MigratingAFreshStore_CreatesThePlainTableAndTheIndex_AndRerunningTheRungChangesNothing() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var bodySucceeded = false; + try + { + Assert.Equal(StorageVersion.SchemaVersion, Convert.ToInt32(await ScalarAsync(connection, "SELECT MAX(version) FROM darling_schema_version", ct))); + Assert.Equal(1L, await ScalarAsync(connection, + "SELECT count(*) FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'collect' AND c.relname = 'slow_reads' AND c.relkind = 'r'", ct)); + Assert.Equal(1L, await ScalarAsync(connection, + "SELECT count(*) FROM pg_indexes WHERE schemaname = 'collect' AND tablename = 'slow_reads' AND indexname = 'idx_slow_reads_time'", ct)); + + await ExecAsync(connection, + "INSERT INTO collect.slow_reads (read_time, surface, route, outcome, total_ms, arguments, arguments_truncated, statements, statement_count, statements_truncated) VALUES (TIMESTAMP '2026-01-01 00:00:00', 'mcp', 'r', 'ok', 5000, '{}'::jsonb, FALSE, '[]'::jsonb, 0, FALSE)", ct); + var shape = "SELECT string_agg(column_name || ':' || data_type, ',' ORDER BY ordinal_position) FROM information_schema.columns WHERE table_schema = 'collect' AND table_name = 'slow_reads'"; + var before = await ScalarAsync(connection, shape, ct); + + /* A store stamped below the rung migrates again: the stamp goes, the SQL runs a second time. */ + await ExecAsync(connection, "DELETE FROM darling_schema_version WHERE version >= 162", ct); + await PgMigrations.MigrateAsync(connection, ct); + + Assert.Equal(StorageVersion.SchemaVersion, Convert.ToInt32(await ScalarAsync(connection, "SELECT MAX(version) FROM darling_schema_version", ct))); + Assert.Equal(before, await ScalarAsync(connection, shape, ct)); + Assert.Equal(1L, await ScalarAsync(connection, "SELECT count(*) FROM collect.slow_reads", ct)); + Assert.Equal(2L, await ScalarAsync(connection, "SELECT count(*) FROM pg_indexes WHERE schemaname = 'collect' AND tablename = 'slow_reads'", ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, static (_, _) => Task.CompletedTask); + } + } + + private static async Task ScalarAsync(NpgsqlConnection connection, string sql, System.Threading.CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + return await command.ExecuteScalarAsync(ct); + } + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, System.Threading.CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/SqlTextPinTests.cs b/Darling/Darling.Tests/SqlTextPinTests.cs index a0dbf3ccb7..8bc3225c1c 100644 --- a/Darling/Darling.Tests/SqlTextPinTests.cs +++ b/Darling/Darling.Tests/SqlTextPinTests.cs @@ -17,6 +17,7 @@ namespace Darling.Tests; /// brittle form it replaces; one that normalised too much would make a removed predicate pass, which is /// strictly worse than the brittleness. Neither shows up as a failure anywhere else. /// +[Trait("Stage", "Guard")] public sealed class SqlTextPinTests { /// The exact #3212 rewrite: a table alias added, then the column qualified with it. diff --git a/Darling/Darling.Tests/StaTestThread.cs b/Darling/Darling.Tests/StaTestThread.cs new file mode 100644 index 0000000000..41585d1229 --- /dev/null +++ b/Darling/Darling.Tests/StaTestThread.cs @@ -0,0 +1,277 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Runtime.ExceptionServices; +using System.Threading; +using System.Threading.Tasks; +using System.Windows; +using System.Windows.Threading; +using Xunit; + +/// +/// #5602: every test that needs WPF runs its body on ONE shared STA thread per test process, instead of starting a thread of its +/// own. WPF keeps process-wide state that is not thread-safe (#5559: the DependencyPropertyDescriptor cache and its +/// unsynchronised AddValueChanged dictionary) and per-thread state that is expensive to leave behind (#5596: a weak-event +/// table and a dispatcher per thread, 300 ms each at process exit). Fifty test files each starting their own thread meant both. +/// +/// One body at a time, end to end. A body that pumps (Dispatcher.PushFrame, a modal loop) lets a second queued +/// Invoke run inside it, which is #5559 again. So the CALLER takes a process-wide gate before it queues its body and +/// releases it after the body and its drain have returned; the dispatcher never has two bodies queued. The wait for the gate is +/// not charged to the hang guard. +/// +/// +/// Drain. After each body the thread closes any window the body left open and runs the dispatcher to application-idle, +/// so work one test queued does not run during the next. An exception thrown by queued work (a timer tick, a Loaded handler) +/// does not end the thread and the process: it is handed to the test whose body or drain ran it. +/// +/// +/// Hang guard. A body that does not return within fails the calling test, naming it. The shared +/// thread is then stuck inside that body and cannot be freed, so every later WPF test in the process fails at once with a +/// message naming the stuck test, instead of each waiting out its own 60 s. +/// +/// +/// Tests that must own a thread (a dispatcher shutdown, a thread that has to END while the test looks at the process) stay on +/// their own thread and take around it, so they cannot overlap a body running here. The allow list +/// is in StaThreadCensusTests, with a reason per file. +/// +/// This file lives in Darling.Tests and is compiled into Lite.Tests by a link (one host, two suites). It is in the global +/// namespace so the Lite classes, which sit in a different namespace, need no extra using. +/// +internal static class StaTestThread +{ + private static readonly StaHost s_host = new(); + + /// How long a body (and its drain) may take before the calling test fails. Settable so a test can shorten it. + internal static TimeSpan HangGuard { get => s_host.HangGuard; set => s_host.HangGuard = value; } + + /// Runs on the shared STA thread and returns when it and its drain are done. Its exception comes back here with its own stack. + internal static void Run(Action body) => s_host.Run(body); + + /// Runs on the shared STA thread and returns its result. + internal static T Run(Func body) => s_host.Run(body); + + /// Runs an async on the shared STA thread: its continuations come back to that thread's dispatcher. + internal static void Run(Func body) => s_host.Run(body); + + /// + /// For a test that must own its thread: waits for the gate and returns the token that releases it. Not re-entrant, and not to + /// be called from a body already running in . + /// + internal static IDisposable EnterExclusive() => s_host.EnterExclusive(); +} + +/// +/// The machinery behind : one lazily started background STA thread running a dispatcher, and the gate. +/// The suites use the single instance behind ; the host's own tests make their own, so the hang guard +/// can be proven without leaving the shared thread stuck for the rest of the process. +/// +internal sealed class StaHost +{ + internal TimeSpan HangGuard { get; set; } = TimeSpan.FromSeconds(60); + + /* A SemaphoreSlim, not a Monitor: a test may be async, so the releasing thread is not promised to be the taking one. */ + private readonly SemaphoreSlim _gate = new(1, 1); + private readonly object _startLock = new(); + private readonly List _strays = new(); + private Dispatcher? _dispatcher; + private volatile string? _stuckTest; + + internal string? StuckTest => _stuckTest; + + /// The dispatcher of the host thread (starting the thread if it is not yet). For the host's own tests. + internal Dispatcher HostDispatcher => EnsureThread(); + + internal void Run(Action body) => RunCore(() => { WithoutSyncContext(body); return Task.CompletedTask; }); + + internal T Run(Func body) + { + T result = default!; + RunCore(() => { WithoutSyncContext(() => result = body()); return Task.CompletedTask; }); + return result; + } + + /// + /// A synchronous body runs with no synchronization context, as it did on a thread of its own that never ran a dispatcher. Inside the + /// dispatcher's own callback the current context is the dispatcher's, and a body that blocks on a task (.GetAwaiter().GetResult()) + /// whose awaits capture it would wait for the very thread it is blocking. A frame the body pushes installs its own context for the + /// frame's life and restores this one, so a pumping body is unaffected. + /// + private static void WithoutSyncContext(Action body) + { + var prior = SynchronizationContext.Current; + SynchronizationContext.SetSynchronizationContext(null); + try + { + body(); + } + finally + { + SynchronizationContext.SetSynchronizationContext(prior); + } + } + + internal void Run(Func body) => RunCore(body); + + internal IDisposable EnterExclusive() + { + _gate.Wait(); + return new Releaser(_gate); + } + + /// Ends the host thread's dispatcher. Only for a host a test made for itself. + internal void Shutdown() => _dispatcher?.BeginInvokeShutdown(DispatcherPriority.Send); + + private void RunCore(Func body) + { + FailIfStuck(); + _gate.Wait(); + try + { + FailIfStuck(); + var dispatcher = EnsureThread(); + ExceptionDispatchInfo? error = null; + lock (_strays) + { + _strays.Clear(); + } + + var finished = dispatcher.InvokeAsync(() => Job(body, e => error = e)).Task.Unwrap(); + if (!finished.Wait(HangGuard)) + { + var name = CurrentTestName(); + _stuckTest = name; + throw new TimeoutException( + $"WPF test body did not return within {HangGuard.TotalSeconds:0} s: {name}. The shared STA test thread is stuck inside it, " + + "so every later WPF test in this process fails at once naming this test (#5602)."); + } + + error?.Throw(); + lock (_strays) + { + if (_strays.Count > 0) + { + var strays = _strays.ToArray(); + _strays.Clear(); + throw new AggregateException("Work the test queued on the dispatcher threw while the shared STA thread ran it (#5602):", strays); + } + } + } + finally + { + _gate.Release(); + } + } + + /// The body, then the cleanup. Never throws: the body's exception is handed back for the caller to rethrow with its stack. + private async Task Job(Func body, Action fail) + { + try + { + await body(); + } + catch (Exception ex) + { + fail(ExceptionDispatchInfo.Capture(ex)); + } + + try + { + CloseLeftoverWindows(); + await Dispatcher.Yield(DispatcherPriority.ApplicationIdle); + } + catch (Exception ex) + { + fail(ExceptionDispatchInfo.Capture(ex)); + } + } + + /// A test that failed before its own Close() leaves a window (and its HWND) behind; the next test must not meet it. + private static void CloseLeftoverWindows() + { + foreach (PresentationSource source in PresentationSource.CurrentSources) + { + if (source.RootVisual is Window window && window.Dispatcher == Dispatcher.CurrentDispatcher) + { + window.Close(); + } + } + } + + private void FailIfStuck() + { + var stuck = _stuckTest; + if (stuck is not null) + { + throw new InvalidOperationException( + $"The shared WPF STA test thread is stuck inside a body that never returned: {stuck}. This test did not run (#5602)."); + } + } + + private static string CurrentTestName() => TestContext.Current.Test?.TestDisplayName ?? "(test name unavailable)"; + + private Dispatcher EnsureThread() + { + var existing = _dispatcher; + if (existing is not null) + { + return existing; + } + + lock (_startLock) + { + if (_dispatcher is not null) + { + return _dispatcher; + } + + using var ready = new ManualResetEventSlim(); + Dispatcher? made = null; + var thread = new Thread(() => + { + var dispatcher = Dispatcher.CurrentDispatcher; + dispatcher.UnhandledException += (_, e) => + { + lock (_strays) + { + _strays.Add(e.Exception); + } + + e.Handled = true; + }; + made = dispatcher; + ready.Set(); + Dispatcher.Run(); + }) + { + Name = "StaTestThread", + /* Background: a stuck body must not keep the test process alive after the run returns. */ + IsBackground = true, + }; + thread.SetApartmentState(ApartmentState.STA); + thread.Start(); + ready.Wait(); + _dispatcher = made; + return made!; + } + } + + private sealed class Releaser(SemaphoreSlim gate) : IDisposable + { + private int _released; + + public void Dispose() + { + if (Interlocked.Exchange(ref _released, 1) == 0) + { + gate.Release(); + } + } + } +} diff --git a/Darling/Darling.Tests/StaTestThreadTests.cs b/Darling/Darling.Tests/StaTestThreadTests.cs new file mode 100644 index 0000000000..a3ac451cbd --- /dev/null +++ b/Darling/Darling.Tests/StaTestThreadTests.cs @@ -0,0 +1,195 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Runtime.CompilerServices; +using System.Threading; +using System.Threading.Tasks; +using System.Windows; +using System.Windows.Threading; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5602: the shared STA test thread. Bodies run in order on one thread, an exception keeps its stack, a body that pumps does not +/// let a second one in, the hang guard fires and names the test, and the drain runs the work a body queued before the next body +/// starts. The hang-guard test uses a host of its own: it leaves that host's thread stuck, which the shared one must never be. +/// In the timing collection because it judges 200 to 300 ms limits (the hang guard, a gate that must hold a body out), which a +/// parallel run's load would stretch (#5602). +/// +[Collection("timing")] +public sealed class StaTestThreadTests +{ + [Fact] + public void Bodies_RunOneAfterAnother_OnTheSameBackgroundStaThread() + { + var first = StaTestThread.Run(Describe); + var second = StaTestThread.Run(Describe); + + Assert.Equal(first.Id, second.Id); + Assert.NotEqual(Environment.CurrentManagedThreadId, first.Id); + Assert.Equal(ApartmentState.STA, first.Apartment); + Assert.True(first.Background, "the shared thread must be a background thread, or a stuck body keeps the test process alive"); + } + + [Fact] + public void ABodysException_ComesBackToTheCaller_WithItsOwnStack() + { + var thrown = Assert.Throws(() => StaTestThread.Run(() => ThrowFromNamedFrame())); + + Assert.Equal("boom from the body", thrown.Message); + Assert.Contains(nameof(ThrowFromNamedFrame), thrown.StackTrace); + + /* The thread survives the failure: the next body still runs. */ + Assert.Equal(5, StaTestThread.Run(() => 5)); + } + + [Fact] + public void ABodyThatPumps_DoesNotLetASecondBodyIn() + { + var inside = 0; + var most = 0; + var errors = new List(); + var callers = Enumerable.Range(0, 3).Select(_ => new Thread(() => + { + try + { + for (var i = 0; i < 3; i++) + { + StaTestThread.Run(() => + { + var now = Interlocked.Increment(ref inside); + InterlockedMax(ref most, now); + + /* A nested frame: while it runs, anything else queued on this dispatcher would run inside it. */ + var frame = new DispatcherFrame(); + var timer = new DispatcherTimer(TimeSpan.FromMilliseconds(15), DispatcherPriority.Normal, (_, _) => frame.Continue = false, Dispatcher.CurrentDispatcher); + timer.Start(); + Dispatcher.PushFrame(frame); + timer.Stop(); + + Interlocked.Decrement(ref inside); + }); + } + } + catch (Exception ex) + { + lock (errors) + { + errors.Add(ex); + } + } + })).ToList(); + + callers.ForEach(t => t.Start()); + callers.ForEach(t => Assert.True(t.Join(TimeSpan.FromSeconds(60)), "a caller never finished")); + + Assert.Empty(errors); + Assert.Equal(1, most); + } + + [Fact] + public void ABodyThatNeverReturns_FailsItsOwnTest_ByName_AndEveryLaterCallFailsAtOnce() + { + var host = new StaHost { HangGuard = TimeSpan.FromMilliseconds(300) }; + using var release = new ManualResetEventSlim(); + try + { + var timedOut = Assert.Throws(() => host.Run(() => release.Wait())); + Assert.Contains(nameof(ABodyThatNeverReturns_FailsItsOwnTest_ByName_AndEveryLaterCallFailsAtOnce), timedOut.Message); + Assert.Contains(nameof(ABodyThatNeverReturns_FailsItsOwnTest_ByName_AndEveryLaterCallFailsAtOnce), host.StuckTest); + + var ran = false; + var started = Environment.TickCount64; + var refused = Assert.Throws(() => host.Run(() => ran = true)); + Assert.Contains(nameof(ABodyThatNeverReturns_FailsItsOwnTest_ByName_AndEveryLaterCallFailsAtOnce), refused.Message); + Assert.False(ran); + Assert.True(Environment.TickCount64 - started < 250, "a later call must fail at once, not wait out its own guard"); + } + finally + { + release.Set(); + host.Shutdown(); + } + } + + [Fact] + public void AfterABody_TheDispatcherIsDrained_AndAWindowItLeftOpenIsClosed() + { + var queuedRan = false; + Window? left = null; + StaTestThread.Run(() => + { + Dispatcher.CurrentDispatcher.BeginInvoke(DispatcherPriority.ApplicationIdle, new Action(() => queuedRan = true)); + left = new Window { Width = 50, Height = 50, ShowInTaskbar = false }; + left.Show(); + }); + + Assert.True(queuedRan, "work the body queued must run before the next test, not during it"); + Assert.False(StaTestThread.Run(() => left!.IsVisible), "a window the body left open must be closed by the drain"); + } + + [Fact] + public void AnAsyncBody_ResumesOnTheSharedThread_AfterEachAwait() + { + var ids = new List(); + StaTestThread.Run(async () => + { + ids.Add(Environment.CurrentManagedThreadId); + await Task.Delay(10); + ids.Add(Environment.CurrentManagedThreadId); + await Dispatcher.Yield(DispatcherPriority.Background); + ids.Add(Environment.CurrentManagedThreadId); + }); + + Assert.Single(ids.Distinct()); + Assert.NotEqual(Environment.CurrentManagedThreadId, ids[0]); + } + + [Fact] + public void ExceptionFromWorkTheBodyQueued_ComesBackToTheTest_AndDoesNotEndTheThread() + { + var thrown = Assert.Throws(() => StaTestThread.Run(() => + Dispatcher.CurrentDispatcher.BeginInvoke(DispatcherPriority.Background, new Action(() => throw new FormatException("queued work failed"))))); + + Assert.Contains(thrown.InnerExceptions, e => e is FormatException { Message: "queued work failed" }); + Assert.Equal(7, StaTestThread.Run(() => 7)); + } + + [Fact] + public async Task EnterExclusive_KeepsABodyOut_UntilItIsDisposed() + { + var ran = new ManualResetEventSlim(); + Task caller; + using (StaTestThread.EnterExclusive()) + { + caller = Task.Run(() => StaTestThread.Run(() => ran.Set())); + Assert.False(ran.Wait(TimeSpan.FromMilliseconds(200)), "a body ran while a thread that owns its own thread held the gate"); + } + + await caller.WaitAsync(TimeSpan.FromSeconds(30), TestContext.Current.CancellationToken); + Assert.True(ran.IsSet); + } + + private static (int Id, ApartmentState Apartment, bool Background) Describe() => + (Environment.CurrentManagedThreadId, Thread.CurrentThread.GetApartmentState(), Thread.CurrentThread.IsBackground); + + [MethodImpl(MethodImplOptions.NoInlining)] + private static void ThrowFromNamedFrame() => throw new InvalidOperationException("boom from the body"); + + private static void InterlockedMax(ref int target, int value) + { + int seen; + while ((seen = Volatile.Read(ref target)) < value && Interlocked.CompareExchange(ref target, value, seen) != seen) + { + } + } +} diff --git a/Darling/Darling.Tests/StaThreadCensusTests.cs b/Darling/Darling.Tests/StaThreadCensusTests.cs new file mode 100644 index 0000000000..cbacbd6f44 --- /dev/null +++ b/Darling/Darling.Tests/StaThreadCensusTests.cs @@ -0,0 +1,139 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Runtime.CompilerServices; +using System.Text.RegularExpressions; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5602: every test that needs WPF runs its body on the one shared STA thread (StaTestThread.Run), not on a thread of its +/// own. About fifty test files used to start their own STA thread behind a gate (#5559): each left a dispatcher and a weak-event +/// table behind (#5596's 300 ms per thread at process exit), and a body that pumped could still overlap another. A new test that +/// starts its own thread, defines its own OnStaThread-style helper, or runs Dispatcher.Run() brings that back, so +/// this fails and names the file and the rule. A test that really must own its thread goes on with the +/// reason, and must take StaTestThread.EnterExclusive() around the thread so it cannot overlap a body on the shared one. +/// +[Trait("Stage", "Guard")] +public sealed class StaThreadCensusTests +{ + /* Built from parts so this file does not contain the text it scans for. */ + private static readonly string s_sta = "ApartmentState" + ".STA"; + private static readonly string s_run = "Dispatcher" + ".Run("; + private const string Exclusive = "StaTestThread.EnterExclusive()"; + + /// + /// The host and its own tests start the thread by design. Every other entry is a test that must own a thread, with the reason. + /// + private static readonly Dictionary Allowed = new(StringComparer.Ordinal) + { + ["StaTestThread.cs"] = "the host: the one place the shared STA thread is started", + ["StaTestThreadTests.cs"] = "the host's own tests assert the thread's apartment state", + ["StaThreadCensusTests.cs"] = "this census names the text it scans for", + ["WpfExitSwitchTests.cs"] = "#5598: it needs a WPF STA thread that has ENDED, to time what the process-exit handler does for it; the shared " + + "thread never ends. It takes StaTestThread.EnterExclusive() so it cannot overlap a body on the shared thread.", + }; + + /// Anti-vacuity: the suite has well over this many files that use the host, so a scan that stopped seeing them fails. + private const int HostUserFloor = 20; + + /// A method named like the per-file helpers this replaced: OnStaThread, OnStaThread<T>, RunOnStaThread, OnDispatcher. + private static readonly Regex s_helperDefinition = new( + @"\bstatic\s+[\w<>\[\]?,\.\s()]+?\b(?:Run)?On(?:Sta(?:Thread)?|Dispatcher|UiThread)\s*(?:<\w+>)?\s*\(", RegexOptions.Compiled); + + [Fact] + public void NoTestFileOutsideTheAllowList_StartsItsOwnStaThread_DefinesAnStaHelper_OrRunsADispatcher() + { + var violations = new List(); + var hostUsers = 0; + foreach (var (rel, text) in TestFiles()) + { + if (text.Contains("StaTestThread.Run", StringComparison.Ordinal)) + { + hostUsers++; + } + + if (Allowed.ContainsKey(Path.GetFileName(rel))) + { + continue; + } + + /* Code only: a comment or a string that names the old helper, the dispatcher or the apartment state is not a use of it. */ + var code = CSharpSourceWalker.StripCommentsAndStrings(text); + if (code.Contains(s_sta, StringComparison.Ordinal)) + { + violations.Add(rel + ": starts its own STA thread; run the body in StaTestThread.Run(...) instead"); + } + + if (s_helperDefinition.IsMatch(code)) + { + violations.Add(rel + ": defines its own OnStaThread-style helper; call StaTestThread.Run(...) directly"); + } + + if (code.Contains(s_run, StringComparison.Ordinal)) + { + violations.Add(rel + ": runs its own dispatcher; use the async overload StaTestThread.Run(async () => ...)"); + } + } + + Assert.True( + violations.Count == 0, + "WPF tests share ONE STA thread per test process (#5602). Fix, or add the file to StaThreadCensusTests.Allowed with the reason it must " + + "own a thread:" + Environment.NewLine + string.Join(Environment.NewLine, violations)); + Assert.True(hostUsers >= HostUserFloor, $"only {hostUsers} test files use StaTestThread.Run; the scan has stopped seeing them"); + } + + [Fact] + public void ATestThatOwnsItsThread_TakesTheExclusiveGateAroundIt() + { + var missing = new List(); + foreach (var (rel, text) in TestFiles()) + { + var name = Path.GetFileName(rel); + if (Allowed.ContainsKey(name) && name is not ("StaTestThread.cs" or "StaTestThreadTests.cs" or "StaThreadCensusTests.cs") + && !text.Contains(Exclusive, StringComparison.Ordinal)) + { + missing.Add(rel); + } + } + + Assert.True( + missing.Count == 0, + "These tests own their STA thread but do not call " + Exclusive + " first, so they can overlap a body on the shared thread (#5559's race): " + + string.Join(", ", missing)); + } + + [Fact] + public void EveryAllowListEntry_SaysWhy() + { + Assert.All(Allowed, entry => Assert.True(entry.Value.Length >= 20, entry.Key + " is on the allow list without a reason")); + } + + private static IEnumerable<(string Rel, string Text)> TestFiles() + { + var dir = ProjectDir(); + foreach (var file in Directory.EnumerateFiles(dir, "*.cs", SearchOption.AllDirectories)) + { + var rel = Path.GetRelativePath(dir, file); + if (rel.StartsWith("bin" + Path.DirectorySeparatorChar, StringComparison.Ordinal) + || rel.StartsWith("obj" + Path.DirectorySeparatorChar, StringComparison.Ordinal)) + { + continue; + } + + yield return (rel, File.ReadAllText(file)); + } + } + + private static string ProjectDir([CallerFilePath] string thisFile = "") => Path.GetDirectoryName(thisFile)!; +} diff --git a/Darling/Darling.Tests/StallWaitProbeLiveTests.cs b/Darling/Darling.Tests/StallWaitProbeLiveTests.cs new file mode 100644 index 0000000000..5463248d3e --- /dev/null +++ b/Darling/Darling.Tests/StallWaitProbeLiveTests.cs @@ -0,0 +1,243 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Text.RegularExpressions; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Data.SqlClient; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using Xunit; + +namespace Darling.Tests; + +/// +/// The stall probe's query against a real SQL Server (#5097). Skipped unless DARLING_TEST_SQL is set, +/// so CI does not run it. +/// +public class StallWaitProbeLiveTests +{ + [Fact] + public async Task TheProbe_CountsAPlantedWaitforAsIdle_AndStillSeesOurOwnSessionWaiting() + { + var sqlHost = Environment.GetEnvironmentVariable("DARLING_TEST_SQL"); + Assert.SkipWhen(string.IsNullOrEmpty(sqlHost), + "Set DARLING_TEST_SQL to run the live stall probe test."); + + var ct = TestContext.Current.CancellationToken; + var sqlUser = Environment.GetEnvironmentVariable("DARLING_TEST_SQL_USER"); + var config = new MonitoredServer + { + Name = "darling-stall-probe-e2e", + Host = sqlHost!, + Auth = string.IsNullOrEmpty(sqlUser) ? "integrated" : "sql", + Username = sqlUser, + Password = Environment.GetEnvironmentVariable("DARLING_TEST_SQL_PASSWORD"), + TrustServerCertificate = true, + }; + + var runtime = await DarlingServerConnector.ConnectAsync(config, logger: null, ct); + Assert.SkipWhen(runtime.Target.IsAzureSqlDb, "The planted WAITFOR session is not visible across an Azure SQL Database pool."); + + /* One session of ours sits in a fresh WAITFOR, a user-task wait. */ + await using var waiter = new SqlConnection(runtime.ConnectionString); + await waiter.OpenAsync(ct); + await using var waitCommand = new SqlCommand("WAITFOR DELAY '00:00:20';", waiter) { CommandTimeout = 60 }; + var waiting = waitCommand.ExecuteNonQueryAsync(ct); + + try + { + await Task.Delay(TimeSpan.FromSeconds(2), ct); + + /* A second connection with the same application name runs the probe query. */ + await using var probe = new SqlConnection(runtime.ConnectionString); + await probe.OpenAsync(ct); + await using var command = new SqlCommand(StallWaitProbePolicy.QueryText, probe) { CommandTimeout = 10 }; + await using var reader = await command.ExecuteReaderAsync(ct); + var sample = await StallWaitProbePolicy.ReadAsync(reader, ct); + + Assert.NotNull(sample); + Assert.True(sample!.BackgroundWaitingTasks > 0, "every instance has idle background waiters when the login holds VIEW SERVER STATE; a login without that permission sees none"); + Assert.NotNull(sample.WaitSummary); + + /* WAITFOR is on the shared idle list: the planted session is counted in the tail, not ranked. */ + Assert.DoesNotContain("WAITFOR:", sample.WaitSummary, StringComparison.Ordinal); + Assert.NotEqual("WAITFOR", sample.TopWaitType); + Assert.Matches(@"idle [1-9]\d* tasks/[1-9]\d* types", sample.WaitSummary); + Assert.True(sample.CollectorSessions >= 1, "our planted session should be visible by program and host"); + Assert.Equal("WAITFOR", sample.CollectorWaitType); + } + finally + { + try + { + await waiting; + } + catch (SqlException) + { + /* The 20-second delay ending is all this waits for. */ + } + } + } + + /// + /// An idle wait does not rank as the top user-task wait (#5267). One connection runs + /// sp_server_diagnostics, which repeats and sits in SP_SERVER_DIAGNOSTICS_SLEEP on an + /// is_user_process = 1 session, exactly as the availability-group health check does. A second + /// connection blocks on a lock a third holds. The real probe query must rank the lock wait first and count + /// the idle wait in the tail rather than in the ranking. + /// + [Fact] + public async Task TheProbe_RanksALockWait_AboveSpServerDiagnosticsSleep_AndCountsTheIdleWait() + { + var sqlHost = Environment.GetEnvironmentVariable("DARLING_TEST_SQL"); + Assert.SkipWhen(string.IsNullOrEmpty(sqlHost), + "Set DARLING_TEST_SQL to run the live stall probe test."); + + var ct = TestContext.Current.CancellationToken; + var sqlUser = Environment.GetEnvironmentVariable("DARLING_TEST_SQL_USER"); + var config = new MonitoredServer + { + Name = "darling-stall-probe-idle-e2e", + Host = sqlHost!, + Auth = string.IsNullOrEmpty(sqlUser) ? "integrated" : "sql", + Username = sqlUser, + Password = Environment.GetEnvironmentVariable("DARLING_TEST_SQL_PASSWORD"), + TrustServerCertificate = true, + }; + + var runtime = await DarlingServerConnector.ConnectAsync(config, logger: null, ct); + Assert.SkipWhen(runtime.Target.IsAzureSqlDb, "The planted sessions are not visible across an Azure SQL Database pool."); + + await using var holder = new SqlConnection(runtime.ConnectionString); + await holder.OpenAsync(ct); + await using var blocked = new SqlConnection(runtime.ConnectionString); + await blocked.OpenAsync(ct); + await using var diagnostics = new SqlConnection(runtime.ConnectionString); + await diagnostics.OpenAsync(ct); + + var table = "tempdb.dbo.stall_probe_5267_" + Guid.NewGuid().ToString("N"); + await using var block = new SqlCommand("SELECT id FROM " + table + " WITH (READCOMMITTEDLOCK);", blocked) { CommandTimeout = 120 }; + await using var diag = new SqlCommand("EXEC sp_server_diagnostics 5;", diagnostics) { CommandTimeout = 0 }; + Task? blockedTask = null; + Task? diagnosticsTask = null; + SqlTransaction? held = null; + var diagnosticsSpid = 0; + var blockedSpid = 0; + await using (var spid = new SqlCommand("SELECT CONVERT(integer, @@SPID);", diagnostics)) + { + diagnosticsSpid = (int)(await spid.ExecuteScalarAsync(ct))!; + } + + await using (var spid = new SqlCommand("SELECT CONVERT(integer, @@SPID);", blocked)) + { + blockedSpid = (int)(await spid.ExecuteScalarAsync(ct))!; + } + + try + { + await using (var create = new SqlCommand("CREATE TABLE " + table + " (id integer NOT NULL); INSERT " + table + " VALUES (1);", holder)) + { + await create.ExecuteNonQueryAsync(ct); + } + + /* The third connection holds an exclusive lock for the rest of the test. */ + held = holder.BeginTransaction(); + await using (var hold = new SqlCommand("UPDATE " + table + " SET id = 2;", holder, held)) + { + await hold.ExecuteNonQueryAsync(ct); + } + + blockedTask = block.ExecuteScalarAsync(CancellationToken.None); + diagnosticsTask = diag.ExecuteNonQueryAsync(CancellationToken.None); + + /* Long enough for the lock wait to build and for the diagnostics procedure to be in its sleep. */ + await Task.Delay(TimeSpan.FromSeconds(3), ct); + + await using var probe = new SqlConnection(runtime.ConnectionString); + await probe.OpenAsync(ct); + await using var command = new SqlCommand(StallWaitProbePolicy.QueryText, probe) { CommandTimeout = 10 }; + await using var reader = await command.ExecuteReaderAsync(ct); + var sample = await StallWaitProbePolicy.ReadAsync(reader, ct); + + Assert.NotNull(sample); + Assert.NotNull(sample!.TopWaitType); + Assert.StartsWith("LCK_M_", sample.TopWaitType, StringComparison.Ordinal); + Assert.NotNull(sample.WaitSummary); + Assert.DoesNotContain("SP_SERVER_DIAGNOSTICS_SLEEP:", sample.WaitSummary, StringComparison.Ordinal); + + var idle = Regex.Match(sample.WaitSummary!, @"idle (\d+) tasks/(\d+) types"); + Assert.True(idle.Success, "the tail names the idle waits: " + sample.WaitSummary); + Assert.True(long.Parse(idle.Groups[1].Value, System.Globalization.CultureInfo.InvariantCulture) >= 1, "the diagnostics session's idle wait is counted: " + sample.WaitSummary); + Assert.True(int.Parse(idle.Groups[2].Value, System.Globalization.CultureInfo.InvariantCulture) >= 1); + + /* The total counts every waiting task, and covers the three disjoint buckets: the background tasks, + the idle ones the ranking dropped, and the ranked waits. Anything else waiting only adds to it. */ + var idleTasks = long.Parse(idle.Groups[1].Value, System.Globalization.CultureInfo.InvariantCulture); + var rankedTasks = Regex.Matches(sample.WaitSummary!, @"(?:^|; )[A-Z0-9_]+:(\d+)x/\d+ms") + .Cast().Sum(m => long.Parse(m.Groups[1].Value, System.Globalization.CultureInfo.InvariantCulture)); + Assert.True(rankedTasks >= 1, "the lock wait is ranked: " + sample.WaitSummary); + Assert.True( + sample.WaitingTaskCount >= sample.BackgroundWaitingTasks + idleTasks + rankedTasks, + $"total {sample.WaitingTaskCount} covers background {sample.BackgroundWaitingTasks} + idle {idleTasks} + ranked {rankedTasks}: {sample.WaitSummary}"); + } + finally + { + held?.Rollback(); + + /* KILL ends both waiting sessions at once; sp_server_diagnostics repeats until it is stopped. */ + foreach (var victim in new[] { diagnosticsSpid, blockedSpid }) + { + try + { + await using var kill = new SqlCommand("KILL " + victim.ToString(System.Globalization.CultureInfo.InvariantCulture) + ";", holder); + await kill.ExecuteNonQueryAsync(CancellationToken.None); + } + catch (SqlException) + { + /* The session may already be gone. */ + } + } + + foreach (var task in new[] { blockedTask, diagnosticsTask }) + { + if (task is null) + { + continue; + } + + try + { + await task; + } + catch (Exception ex) when (ex is SqlException or OperationCanceledException) + { + /* Cancelling the two waiting sessions is all this waits for. */ + } + } + + try + { + await using var drop = new SqlCommand("DROP TABLE IF EXISTS " + table + ";", holder); + await drop.ExecuteNonQueryAsync(CancellationToken.None); + } + catch (SqlException) + { + /* The table lives in tempdb and goes with the next restart. */ + } + + /* The two KILLed connections are dead on the server. Disposed as they are, they return to the pool + and a later test with the same connection string can draw one. ClearPool dooms the pooled + connections and discards the in-use ones when they close, so none is reused. */ + SqlConnection.ClearPool(diagnostics); + SqlConnection.ClearPool(blocked); + } + } +} diff --git a/Darling/Darling.Tests/StallWaitProbeQueryTextPinTests.cs b/Darling/Darling.Tests/StallWaitProbeQueryTextPinTests.cs new file mode 100644 index 0000000000..2c1dc59a37 --- /dev/null +++ b/Darling/Darling.Tests/StallWaitProbeQueryTextPinTests.cs @@ -0,0 +1,139 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.ComponentModel; +using System.Linq; +using System.Reflection; +using System.Text.RegularExpressions; +using ModelContextProtocol.Server; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using Xunit; + +namespace Darling.Tests; + +/// +/// The stall probe's idle-wait list (#5267) against the shared IgnoredWaitDefaults.All set. The list is +/// rendered into the probe's T-SQL as literals, and a name that is not a plain wait name is refused. Lite.Tests +/// pins the same facts, but CI does not run Lite.Tests on a Darling-only change, so this class is what stops a +/// bad entry added to the shared set from reaching a release. +/// +[Trait("Stage", "Guard")] +public sealed class StallWaitProbeQueryTextPinTests +{ + /// + /// Every name in the shared set renders: it is a plain upper-case wait name, and the rendered text holds it + /// as a literal exactly once, because the list is rendered a single time. + /// + [Fact] + public void EveryIgnoredWaitName_IsAPlainWaitName_AndIsRenderedIntoTheProbeText() + { + var sql = StallWaitProbePolicy.QueryText; + + /* The list is the leading derived set; the scheduler filter further down has a literal of its own. */ + var list = sql.Substring(0, sql.IndexOf("SELECT /* PerformanceMonitorDarling", StringComparison.Ordinal)); + + Assert.NotEmpty(IgnoredWaitDefaults.All); + Assert.All(IgnoredWaitDefaults.All, name => Assert.Matches("^[A-Z0-9_]+$", name)); + + foreach (var name in IgnoredWaitDefaults.All) + { + Assert.Single(Regex.Matches(sql, "N'" + Regex.Escape(name) + "'")); + Assert.Contains("N'" + name + "'", list, StringComparison.Ordinal); + } + + Assert.Equal( + IgnoredWaitDefaults.All.OrderBy(n => n, StringComparer.Ordinal).ToList(), + Regex.Matches(list, "N'([^']*)'").Select(m => m.Groups[1].Value).ToList()); + } + + /// + /// A bad entry fails the BUILDER, with the documented exception, for a quote, a lower-case name, a trailing + /// space, an empty name, a null and an empty list. + /// + [Theory] + [InlineData("BAD'; DROP")] + [InlineData("lower_case")] + [InlineData("TRAILING_SPACE ")] + [InlineData("")] + [InlineData(null)] + public void ABadEntry_IsRefusedByTheBuilder_WithTheDocumentedException(string? bad) + { + Assert.Throws(() => StallWaitProbePolicy.BuildQueryText(new[] { "OK_WAIT", bad! })); + Assert.Throws(() => StallWaitProbePolicy.BuildQueryText(Array.Empty())); + } + + /// + /// The refusal reaches only the probe-text path. The policy's own members, which every budgeted collector + /// uses, do not render the list, so a bad entry cannot make them throw. This is the property the text + /// being a lazy holder (and not a static field of the policy) exists to give. + /// + [Fact] + public void ABadEntry_FailsOnlyTheTextPath_NotThePolicy() + { + /* The builder throws... */ + Assert.Throws(() => StallWaitProbePolicy.BuildQueryText(new[] { "bad name" })); + + /* ...and the policy is still usable beside it: nothing but QueryText calls the builder with the shared + set, and QueryText is read inside the probe's try block. */ + Assert.True(StallWaitProbePolicy.HardBudget > TimeSpan.Zero); + Assert.NotEmpty(StallWaitProbePolicy.Outcomes); + Assert.Contains(StallWaitProbePolicy.OutcomeQueryFailed, StallWaitProbePolicy.Outcomes); + + /* Structurally: the policy's type initialiser never calls the builder, so no entry in the shared set + can fail it. A static field initialised with BuildQueryText() would put the call back in it. */ + Assert.False(TypeInitialiserCallsBuilder(), "the policy's type initialiser renders the query text"); + } + + /// + /// The tool text names the three generations of stall-probe row (#5267): the guide tail, which tools/list + /// does not serve, says a tail without the idle part is from a build that still ranked idle waits. The head + /// (the part tools/list serves) is pinned in length by McpToolsListBudgetTests and here by content. + /// + [Fact] + public void TheGuideTail_SaysARowWithoutTheIdlePart_RankedIdleWaits_AndTheHeadIsUntouched() + { + var method = typeof(DarlingMcpStallProbeTools) + .GetMethods(BindingFlags.Public | BindingFlags.Static) + .Single(m => m.GetCustomAttribute()?.Name == "get_collector_stall_probes"); + var (head, tail) = McpToolGuide.Split(method.GetCustomAttribute()!.Description); + + Assert.NotNull(tail); + Assert.Contains("A tail without 'idle K tasks/J types' is from a build that still ranked idle waits.", tail, StringComparison.Ordinal); + + /* The head is byte-identical to dev's: same length (the budget file pins 595 with the schema) and the + same text, and it carries nothing about idle waits. */ + Assert.StartsWith("Out-of-band DMV samples taken when this service's own collector stalls", head, StringComparison.Ordinal); + Assert.EndsWith("Unbanded, untrended: one row is evidence about a moment.", head, StringComparison.Ordinal); + Assert.Equal(564, head.Length); + Assert.DoesNotContain("idle", head, StringComparison.OrdinalIgnoreCase); + } + + private static bool TypeInitialiserCallsBuilder() + { + var builder = typeof(StallWaitProbePolicy).GetMethod( + "BuildQueryText", + BindingFlags.Static | BindingFlags.NonPublic | BindingFlags.Public); + Assert.NotNull(builder); + + var il = typeof(StallWaitProbePolicy).TypeInitializer!.GetMethodBody()!.GetILAsByteArray()!; + var token = BitConverter.GetBytes(builder!.MetadataToken); + + for (var i = 0; i <= il.Length - token.Length; i++) + { + if (il.AsSpan(i, token.Length).SequenceEqual(token)) + { + return true; + } + } + + return false; + } +} diff --git a/Darling/Darling.Tests/StartupCommandTimeoutTests.cs b/Darling/Darling.Tests/StartupCommandTimeoutTests.cs index 76f339403c..bc36c852e8 100644 --- a/Darling/Darling.Tests/StartupCommandTimeoutTests.cs +++ b/Darling/Darling.Tests/StartupCommandTimeoutTests.cs @@ -24,14 +24,14 @@ namespace Darling.Tests; /// deadline (#2874) — the managed-Postgres bootstrap and network reconcile, the in-place major-upgrade /// path, least-privilege role provisioning, the delta-baseline re-seed, and the config store's first-run /// seed — and so must the commands the collection loop's own SERIAL thread runs ahead of every launch, -/// which share seven of their call sites with that path and cannot share its number. +/// which share eight of their call sites with that path and cannot share its number. /// /// Three constants, because the failure modes differ rather than the magnitudes. /// covers the un-retried bootstrap sites /// ( of them); /// the one inside the bootstrap's six-attempt first-connection retry, where the deadline MULTIPLIES; and -/// the ten on the collection loop's serial -/// thread, where ten sequential commands have to fit inside one watchdog window. +/// the twelve on the collection loop's serial +/// thread, where twelve sequential commands have to fit inside one watchdog window. /// /// This is the regime whose value goes UP, and the pin has to defend that as much as the /// number. Two of the four failure handlers on this path are LogCritical followed by @@ -85,15 +85,17 @@ public sealed class StartupCommandTimeoutTests /// What is deliberately absent, and why each is a different budget — the token test, member /// by member. /// - /// StoreConfigProvider.ReadConfigVersionAsync (1 site) and LoadViewAsync's five + /// StoreConfigProvider.ReadConfigVersionAsync (1 site) and LoadViewAsync's six /// reads (ReadServiceRowAsync, ReadAlertSettingsAsync, ReadNotificationAsync, - /// ReadMonitoredServersAsync, ReadScheduleOverridesAsync), + /// ReadNotificationRoutesAsync, ReadMonitoredServersAsync, and + /// ReadScheduleOverridesAsync, whose two commands are ReadScheduleRowsAsync and + /// ReadRunTimesAsync — seven commands in all), /// DarlingObservability.SyncServerEnabledStatesAsync's two, and /// DarlingManagedRoles.ReassertComposeStatementTimeoutAsync's one are the CONTROL-PLANE - /// regime: ten sites awaited inline on the serial collection-loop thread, ahead of every per-server + /// regime: eleven sites awaited inline on the serial collection-loop thread, ahead of every per-server /// launch, on the 15 s s_sweepInterval tick. Their blast radius is the whole fleet while their /// floor is a single-row lookup, they fail OPEN (the live config stands), and their ceiling is the - /// tick — a strictly TIGHTER bound on both sides than this group's. Seven of them also run once on + /// tick — a strictly TIGHTER bound on both sides than this group's. Nine of them also run once on /// the bootstrap path, where DarlingWorker awaits them during startup, and that is exactly why they /// are excluded rather than claimed: a deadline is a property of the command, so the tighter of a /// dual-caller site's two bounds has to win, and it is not this one. @@ -154,17 +156,26 @@ private static readonly (string File, string Member, int Bootstrap, int ConnectP ("StoreConfigProvider.cs", "SeedServiceRowAsync", 1, 0, 0, 0), ("StoreConfigProvider.cs", "SeedAlertSettingsAsync", 1, 0, 0, 0), ("StoreConfigProvider.cs", "SeedNotificationAsync", 1, 0, 0, 0), - ("StoreConfigProvider.cs", "SeedMonitoredServersAsync", 1, 0, 0, 0), + /* #5240: the seed's INSERT loop runs in one transaction behind the identity lock, so the member gained two + more bootstrap commands: the lock itself and the read of the addresses already held, both un-retried, + both on this deadline. The INSERT is the third. */ + ("StoreConfigProvider.cs", "SeedMonitoredServersAsync", 3, 0, 0, 0), /* The serial collection-loop thread: the control-plane reload body plus the disk-check store-size - read. Seven of these ten ALSO run once on the bootstrap path above, and take this constant - rather than BootstrapSeconds because a dual-caller site has to take the tighter of its two - bounds. */ + read. Nine of these twelve ALSO run once on the bootstrap path above (LoadViewAsync's seven + commands and SyncServerEnabledStatesAsync's two), and take this constant rather than + BootstrapSeconds because a dual-caller site has to take the tighter of its two bounds. */ ("StoreConfigProvider.cs", "ReadServiceRowAsync", 0, 0, 1, 0), ("StoreConfigProvider.cs", "ReadAlertSettingsAsync", 0, 0, 1, 0), ("StoreConfigProvider.cs", "ReadNotificationAsync", 0, 0, 1, 0), + /* The notification routes LoadViewAsync reads between the notification settings and the monitored + servers: one command on this constant, and it runs once on the bootstrap path too. */ + ("StoreConfigProvider.cs", "ReadNotificationRoutesAsync", 0, 0, 1, 0), ("StoreConfigProvider.cs", "ReadMonitoredServersAsync", 0, 0, 1, 0), - ("StoreConfigProvider.cs", "ReadScheduleOverridesAsync", 0, 0, 1, 0), + /* #4938: the schedule read is two commands, the sparse schedule rows and the run times from their own table, + each on this constant. ReadScheduleOverridesAsync only layers one onto the other and builds no command. */ + ("StoreConfigProvider.cs", "ReadScheduleRowsAsync", 0, 0, 1, 0), + ("StoreConfigProvider.cs", "ReadRunTimesAsync", 0, 0, 1, 0), ("DarlingObservability.cs", "SyncServerEnabledStatesAsync", 0, 0, 2, 0), ("DarlingManagedRoles.cs", "ReassertComposeStatementTimeoutAsync", 0, 0, 1, 0), ("DarlingWorker.cs", "ReadStoreSizeBytesAsync", 0, 0, 1, 0), @@ -181,11 +192,11 @@ hold. It is a single-row INSERT into the same collect.collection_log #2928 bound }; /// The group's own totals, so a member that stops creating commands fails loudly. - private const int ExpectedBootstrapSites = 34; + private const int ExpectedBootstrapSites = 36; private const int ExpectedConnectProbeSites = 1; - internal const int ExpectedSerialLoopSites = 10; + internal const int ExpectedSerialLoopSites = 12; /// /// The serial-loop regime's members, projected out of the census above so a second pin over the same @@ -193,10 +204,10 @@ hold. It is a single-row INSERT into the same collect.collection_log #2928 bound /// /// Read by , which asserts a property of what these /// members RUN rather than of the deadline they set. Exposed as a projection rather than as a second - /// list because a hand-copied enumeration of ten members is a counted claim with a numeral welded on: + /// list because a hand-copied enumeration of eleven members is a counted claim with a numeral welded on: /// it would keep passing over nine of them after the tenth moved, and the pin's message would name the - /// wrong population. The per-member SITE COUNT travels with it for the same reason: nine members - /// hold the regime's ten commands (SyncServerEnabledStatesAsync builds two), so a consumer + /// wrong population. The per-member SITE COUNT travels with it for the same reason: eleven members + /// hold the regime's twelve commands (SyncServerEnabledStatesAsync builds two), so a consumer /// counting members would silently disagree with and could /// not tell a dropped member from that arithmetic. /// @@ -529,7 +540,7 @@ public void TheConnectProbeDeadline_SurvivesItsOwnRetryMultiplier() /// /// What this no longer claims. The old product forbade 6; nothing in the service does. /// The answer to "why 5, and what would justify 6" is that 5 is a measured floor with three ceilings - /// over it — the tick, Npgsql's default, and the bootstrap's number for the seven dual-caller sites — + /// over it — the tick, Npgsql's default, and the bootstrap's number for the eight dual-caller sites — /// and 6 clears all three as well. Raising it costs delivered cadence in proportion, which is a trade /// to argue rather than an inequality to lose. /// @@ -588,13 +599,13 @@ the worst tick and cannot be re-derived as its sum. */ Assert.True( seconds < ServiceCommandDeadlines.BootstrapSeconds, $"serial-loop deadline {seconds}s is not tighter than the bootstrap's " - + $"{ServiceCommandDeadlines.BootstrapSeconds}s. Seven of these ten sites ALSO run once on " + + $"{ServiceCommandDeadlines.BootstrapSeconds}s. Nine of these twelve sites ALSO run once on " + "the bootstrap path, and a dual-caller site must take the tighter of its two bounds — if " - + "this is not the tighter one, those seven are on the wrong constant"); + + "this is not the tighter one, those nine are on the wrong constant"); } /// - /// is what thirty-seven sites' correctness is asserted through, so its + /// is what forty-eight sites' correctness is asserted through, so its /// own edges are pinned. The fixtures are shapes that OCCUR in the members above, and the fourth is /// the one that matters: the using (...) { ...; } block whose neighbour a two-statement window /// would borrow, which is the real shape of BridgeTimescaleAsync. The last fixture covers the diff --git a/Darling/Darling.Tests/StateSelfAlertTests.cs b/Darling/Darling.Tests/StateSelfAlertTests.cs new file mode 100644 index 0000000000..305aed064f --- /dev/null +++ b/Darling/Darling.Tests/StateSelfAlertTests.cs @@ -0,0 +1,491 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using PerformanceMonitor.Alerting; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Notifications; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5493: a self-alert for a LASTING state sends one alert per occurrence, not one per cooldown. Collection Stopped +/// took the "Server Unreachable" shape in #5489 (: one alert on entry, a repeat only +/// per connection_refire_minutes, a send again only for an alert no channel took); these are the other alerts +/// whose condition stays until someone acts. Driven through the real arms with a scripted deliverer and a clock the +/// test moves, one [Theory] over the alerts so each gets the same four cases: +/// one alert across many cooldowns with repeats off, a repeat per interval with repeats on, a new alert after the +/// condition ended and started again, and a send again for an alert no channel took. +/// +public sealed class StateSelfAlertTests +{ + private const int ServerId = 424242; + private const string Name = "STATE-ALERT-SRV"; + + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + private static readonly DateTime Start = new(2026, 7, 1, 12, 0, 0, DateTimeKind.Utc); + + public enum State + { + CaptureDown, + AgentNotRunning, + AgSyncBehind, + CustomRuleHealth, + FleetGate, + StoreJobOverCadence, + RetentionHeld, + RawPurgeOverHorizon, + PolicyJobStuckEscalated, + } + + /// The alerts with severity bands: a rise inside one occurrence is a new alert (#5493). + public enum BandedState + { + StoreJobOverCadence, + RetentionHeld, + RawPurgeOverHorizon, + } + + public static TheoryData BandedStates + { + get + { + var data = new TheoryData(); + foreach (var state in Enum.GetValues()) + { + data.Add(state); + } + + return data; + } + } + + public static TheoryData States + { + get + { + var data = new TheoryData(); + foreach (var state in Enum.GetValues()) + { + data.Add(state); + } + + return data; + } + } + + private static AlertDelivery FailedByWebhook() => AlertDelivery.FromFanout( + new EmailFanoutResult( + EmailOutcome: AlertChannelOutcome.NotAttempted, SendError: null, + WebhookOutcome: AlertChannelOutcome.Failed, WebhookSendError: "Slack: 429 Too Many Requests", AnyChannelConfigured: true), + muted: false, trayChannelPresent: false); + + private static AlertDelivery DeliveredByWebhook() => AlertDelivery.FromFanout( + new EmailFanoutResult( + EmailOutcome: AlertChannelOutcome.NotAttempted, SendError: null, + WebhookOutcome: AlertChannelOutcome.Delivered, WebhookSendError: null, AnyChannelConfigured: true), + muted: false, trayChannelPresent: false); + + private sealed class ScriptedDeliverer : IAlertDeliverer + { + public List Outcomes { get; } = new(); + + public AlertDelivery? Answer { get; set; } = DeliveredByWebhook(); + + public Task DeliverAsync(AlertOutcome outcome, CancellationToken cancellationToken = default) + { + Outcomes.Add(outcome); + return Task.CompletedTask; + } + + public Task DeliverAndReportAsync(AlertOutcome outcome, CancellationToken cancellationToken = default) + { + Outcomes.Add(outcome); + return Task.FromResult(Answer); + } + } + + private sealed class Rig + { + public Rig(State state) + { + State = state; + Evaluator = new DarlingSelfAlertEvaluator( + Settings, Deliverer, History, _ => false, + logger: new DarlingSelfAlertTests.CapturingLogger(), utcNow: () => Now, + connectionRefireMinutes: () => ConnectionRefireMinutes); + } + + public State State { get; } + + public DarlingSelfAlertTests.FakeHistoryStore History { get; } = new(); + + public int ConnectionRefireMinutes { get; set; } + + public DarlingSelfAlertTests.FakeSettings Settings { get; } = new() { CooldownMinutes = 5 }; + + public ScriptedDeliverer Deliverer { get; } = new(); + + public DateTime Now { get; set; } = Start; + + public DarlingSelfAlertEvaluator Evaluator { get; } + + public int Fires => Deliverer.Outcomes.Count; + + public void At(int minute) => Now = Start.AddMinutes(minute); + + /// One check of a banded alert's condition at the warning or the critical band. + public Task HoldBandAsync(BandedState state, bool critical) => state switch + { + BandedState.StoreJobOverCadence => Evaluator.ApplyStoreJobCadenceAsync( + new[] { new StoreJobCadenceReading(1028, "policy_compression t", critical ? 3_600_000 : 1_800_000, 3_600_000) }, Ct), + BandedState.RetentionHeld => Evaluator.ApplyRetentionHoldsAsync( + new[] { new RetentionHoldReading(1, "t", false, "4 days", 19, critical ? 1_561_449 : 800_000, 345_600) }, Ct), + BandedState.RawPurgeOverHorizon => Evaluator.ApplyRawPurgeOverHorizonAsync( + new[] { new RawPurgeOverHorizonReading(1, "t", "4 days", critical ? 4.5 : 2.5, null) }, Ct), + _ => throw new ArgumentOutOfRangeException(nameof(state)), + }; + + public Task EndBandAsync(BandedState state) => state switch + { + BandedState.StoreJobOverCadence => Evaluator.ApplyStoreJobCadenceAsync( + new[] { new StoreJobCadenceReading(1028, "policy_compression t", 60_000, 3_600_000) }, Ct), + BandedState.RetentionHeld => Evaluator.ApplyRetentionHoldsAsync(new[] { Retention(armed: true) }, Ct), + BandedState.RawPurgeOverHorizon => Evaluator.ApplyRawPurgeOverHorizonAsync( + new[] { new RawPurgeOverHorizonReading(1, "t", "4 days", 0.5, new RawLastPurgeRecord(Now, "ran", null, null)) }, Ct), + _ => throw new ArgumentOutOfRangeException(nameof(state)), + }; + + /// One check with the condition holding. + public Task HoldAsync() => State switch + { + State.CaptureDown => Evaluator.ApplyCaptureDownAsync(ServerId, Name, new[] { "Blocking" }, Ct), + State.AgentNotRunning => Evaluator.ApplyAgentNotRunningAsync(ServerId, Name, agentRunningFresh: false, agentEverSeenRunning: true, Ct), + State.AgSyncBehind => Evaluator.ApplyAgDatabaseHealthAsync(ServerId, Name, new[] { AgDatabase(lagSeconds: 900) }, Ct), + State.CustomRuleHealth => Evaluator.ApplyCustomRuleHealthAsync(UnhealthyRules(), Ct), + State.FleetGate => Evaluator.ApplyFleetGateAsync(GateReport(skipped: 500), Ct), + State.StoreJobOverCadence => Evaluator.ApplyStoreJobCadenceAsync(new[] { new StoreJobCadenceReading(1028, "policy_compression t", 3_600_000, 3_600_000) }, Ct), + State.RetentionHeld => Evaluator.ApplyRetentionHoldsAsync(new[] { Retention(armed: false) }, Ct), + State.RawPurgeOverHorizon => Evaluator.ApplyRawPurgeOverHorizonAsync(new[] { new RawPurgeOverHorizonReading(1, "t", "4 days", 4.5, null) }, Ct), + State.PolicyJobStuckEscalated => Evaluator.ApplyPolicyJobsStuckAsync(StuckJob(), RearmFails, Ct), + _ => throw new ArgumentOutOfRangeException(nameof(State)), + }; + + /// One check with the condition gone (the fleet gate also needs its quiet hour, so that one + /// moves the clock past it). + public async Task EndAsync() + { + switch (State) + { + case State.CaptureDown: + await Evaluator.ApplyCaptureDownAsync(ServerId, Name, Array.Empty(), Ct); + break; + case State.AgentNotRunning: + await Evaluator.ApplyAgentNotRunningAsync(ServerId, Name, agentRunningFresh: true, agentEverSeenRunning: true, Ct); + break; + case State.AgSyncBehind: + await Evaluator.ApplyAgDatabaseHealthAsync(ServerId, Name, new[] { AgDatabase(lagSeconds: 0) }, Ct); + break; + case State.CustomRuleHealth: + await Evaluator.ApplyCustomRuleHealthAsync(new CustomAlertHealthReport( + Array.Empty(), Array.Empty()), Ct); + break; + case State.FleetGate: + /* Caught up starts the quiet clock; a whole quiet hour later it resolves. */ + await Evaluator.ApplyFleetGateAsync(GateReport(skipped: 0), Ct); + Now += DarlingSelfAlertEvaluator.FleetGateQuietHold; + await Evaluator.ApplyFleetGateAsync(GateReport(skipped: 0), Ct); + break; + case State.StoreJobOverCadence: + await Evaluator.ApplyStoreJobCadenceAsync(new[] { new StoreJobCadenceReading(1028, "policy_compression t", 60_000, 3_600_000) }, Ct); + break; + case State.RetentionHeld: + await Evaluator.ApplyRetentionHoldsAsync(new[] { Retention(armed: true) }, Ct); + break; + case State.RawPurgeOverHorizon: + await Evaluator.ApplyRawPurgeOverHorizonAsync( + new[] { new RawPurgeOverHorizonReading(1, "t", "4 days", 0.5, new RawLastPurgeRecord(Now, "ran", null, null)) }, Ct); + break; + case State.PolicyJobStuckEscalated: + await Evaluator.ApplyPolicyJobsStuckAsync(StorePolicyJobHealth.Empty, RearmFails, Ct); + break; + default: + throw new ArgumentOutOfRangeException(nameof(State)); + } + } + + private static Task RearmFails(long jobId) => Task.FromResult(false); + + private static AgDatabaseReading AgDatabase(int lagSeconds) => + new("AG1", "Sales", "NODE2", SecondaryLagSeconds: lagSeconds, RedoQueueSizeKb: 0, IsSuspended: false, SuspendReasonDesc: null); + + private static CustomAlertHealthReport UnhealthyRules() => new( + new[] { new CustomAlertRuleHealthIssue(1, "broken 1", "invalid metric: unknown measure 'x'") }, + Array.Empty()); + + private DarlingSelfAlertEvaluator.FleetGateReport GateReport(long skipped) => + new(Run: 500, Skipped: skipped, QueueWaits: 0, QueueWaitTotal: TimeSpan.Zero, QueueWaitMax: TimeSpan.Zero, + GateWidth: 4, WindowEndUtc: Now); + + private static RetentionHoldReading Retention(bool armed) => + new(1, "t", armed, "4 days", 19, 1_561_449, 345_600); + + private static StorePolicyJobHealth StuckJob() + { + var stuck = new[] + { + new StuckPolicyJob( + 7, "wait_stats", "next_start is -infinity — the scheduler will never run it again", + Arm: StuckPolicyJobArm.NextStartNegativeInfinity), + }; + return new StorePolicyJobHealth( + stuck, + new[] { new PolicyJobRunReading(7, stuck[0].Family, "wait_stats", Held: false, LastRunStatus: null, TotalFailures: 0) }); + } + } + + [Theory] + [MemberData(nameof(States))] + public async Task AStandingCondition_AcrossManyCooldowns_WithRefireOff_AlertsExactlyOnce(State state) + { + var rig = new Rig(state); + + /* 12 cooldowns (an hour) of a standing condition, one check a minute. */ + for (var minute = 0; minute <= 60; minute++) + { + rig.At(minute); + await rig.HoldAsync(); + } + + Assert.Equal(1, rig.Fires); + } + + [Theory] + [MemberData(nameof(States))] + public async Task AStandingCondition_WithRefireOn_RepeatsOnTheRefireInterval_NotTheCooldown(State state) + { + var rig = new Rig(state) { ConnectionRefireMinutes = 30 }; + + var firedAtMinute = new List(); + for (var minute = 0; minute <= 95; minute++) + { + rig.At(minute); + var before = rig.Fires; + await rig.HoldAsync(); + if (rig.Fires > before) + { + firedAtMinute.Add(minute); + } + } + + Assert.Equal(new[] { 0, 30, 60, 90 }, firedAtMinute.ToArray()); + } + + [Theory] + [MemberData(nameof(States))] + public async Task TheConditionEndingAndStartingAgain_IsANewOccurrenceWithItsOwnAlert_AndOneResolution(State state) + { + var rig = new Rig(state); + + await rig.HoldAsync(); + Assert.Equal(1, rig.Fires); + + rig.At(1); + await rig.EndAsync(); + Assert.Single(rig.History.Records); + + /* Back a minute later, inside the cooldown: a new occurrence, a new alert, and again no repeats. */ + rig.Now += TimeSpan.FromMinutes(1); + await rig.HoldAsync(); + Assert.Equal(2, rig.Fires); + + rig.Now += TimeSpan.FromMinutes(60); + await rig.HoldAsync(); + Assert.Equal(2, rig.Fires); + } + + [Theory] + [MemberData(nameof(States))] + public async Task AnAlertNoChannelTook_IsSentAgainAfterTheBackOff_ThenStopsOnceDelivered_WithRefireOff(State state) + { + var rig = new Rig(state) { Deliverer = { Answer = FailedByWebhook() } }; + + await rig.HoldAsync(); + Assert.Equal(1, rig.Fires); + + rig.Now = Start.AddSeconds(59); + await rig.HoldAsync(); + Assert.Equal(1, rig.Fires); + + rig.Deliverer.Answer = DeliveredByWebhook(); + rig.Now = Start.AddSeconds(60); + await rig.HoldAsync(); + Assert.Equal(2, rig.Fires); + + for (var minute = 2; minute <= 60; minute++) + { + rig.At(minute); + await rig.HoldAsync(); + } + + Assert.Equal(2, rig.Fires); + } + + [Theory] + [MemberData(nameof(BandedStates))] + public async Task ARiseToTheCriticalBand_InsideOneOccurrence_SendsOneMoreAlert_WithRefireOff(BandedState state) + { + var rig = new Rig(State.StoreJobOverCadence); + + await rig.HoldBandAsync(state, critical: false); + Assert.Equal(1, rig.Fires); + + rig.At(1); + await rig.HoldBandAsync(state, critical: true); + Assert.Equal(2, rig.Fires); + Assert.Equal(AlertSeverityLevel.Warning, rig.Deliverer.Outcomes[0].Severity); + Assert.Equal(AlertSeverityLevel.Critical, rig.Deliverer.Outcomes[1].Severity); + + /* The rise is sent once, and a standing critical band does not repeat. */ + for (var minute = 2; minute <= 60; minute++) + { + rig.At(minute); + await rig.HoldBandAsync(state, critical: true); + } + + Assert.Equal(2, rig.Fires); + } + + [Theory] + [MemberData(nameof(BandedStates))] + public async Task AFallToTheWarningBand_SendsNothing_AndSoDoesRisingBackToCriticalInTheSameOccurrence(BandedState state) + { + var rig = new Rig(State.StoreJobOverCadence); + + await rig.HoldBandAsync(state, critical: true); + rig.At(1); + await rig.HoldBandAsync(state, critical: false); + Assert.Equal(1, rig.Fires); + + rig.At(2); + await rig.HoldBandAsync(state, critical: true); + Assert.Equal(1, rig.Fires); + } + + [Theory] + [MemberData(nameof(BandedStates))] + public async Task Recovery_ForgetsTheSeverity_SoCriticalThenWarningAfterRecoveryIsANewEntry(BandedState state) + { + var rig = new Rig(State.StoreJobOverCadence); + + await rig.HoldBandAsync(state, critical: true); + rig.At(1); + await rig.EndBandAsync(state); + Assert.Single(rig.History.Records); + + rig.At(2); + await rig.HoldBandAsync(state, critical: false); + Assert.Equal(2, rig.Fires); + Assert.Equal(AlertSeverityLevel.Warning, rig.Deliverer.Outcomes[1].Severity); + + /* ... and the warning-to-critical rise of that new occurrence alerts again. */ + rig.At(3); + await rig.HoldBandAsync(state, critical: true); + Assert.Equal(3, rig.Fires); + } + + [Theory] + [MemberData(nameof(BandedStates))] + public async Task ARepeat_WithRefireOn_CarriesTheCurrentSeverity(BandedState state) + { + var rig = new Rig(State.StoreJobOverCadence) { ConnectionRefireMinutes = 30 }; + + await rig.HoldBandAsync(state, critical: true); + rig.At(30); + await rig.HoldBandAsync(state, critical: false); + + Assert.Equal(2, rig.Fires); + Assert.Equal(AlertSeverityLevel.Critical, rig.Deliverer.Outcomes[0].Severity); + Assert.Equal(AlertSeverityLevel.Warning, rig.Deliverer.Outcomes[1].Severity); + } + + private static CustomAlertHealthReport Rules(params long[] brokenIds) => new( + brokenIds.Select(id => new CustomAlertRuleHealthIssue(id, "rule-" + id, "invalid metric")).ToArray(), + Array.Empty()); + + [Fact] + public async Task ARuleJoiningTheUnhealthySet_SendsOneAlert_NamingTheSetAndTheNewRule_WithRefireOff() + { + var rig = new Rig(State.CustomRuleHealth); + + await rig.Evaluator.ApplyCustomRuleHealthAsync(Rules(1, 2), Ct); + Assert.Equal(1, rig.Fires); + Assert.DoesNotContain("New since", rig.Deliverer.Outcomes[0].DetailText!); + + rig.At(10); + await rig.Evaluator.ApplyCustomRuleHealthAsync(Rules(1, 2), Ct); + Assert.Equal(1, rig.Fires); + + rig.At(20); + await rig.Evaluator.ApplyCustomRuleHealthAsync(Rules(1, 2, 3), Ct); + Assert.Equal(2, rig.Fires); + var detail = rig.Deliverer.Outcomes[1].DetailText!; + Assert.Contains("New since the previous alert were rule 3 \"rule-3\"", detail); + Assert.Contains("- Rule 1 ", detail); + Assert.Contains("- Rule 2 ", detail); + Assert.Contains("- Rule 3 ", detail); + + /* The enlarged set is now the known one: no further alert while it stands. */ + rig.At(90); + await rig.Evaluator.ApplyCustomRuleHealthAsync(Rules(1, 2, 3), Ct); + Assert.Equal(2, rig.Fires); + } + + [Fact] + public async Task ARuleLeavingTheUnhealthySet_SendsNothing_AndRejoiningItIsNewAgain() + { + var rig = new Rig(State.CustomRuleHealth); + + await rig.Evaluator.ApplyCustomRuleHealthAsync(Rules(1, 2), Ct); + rig.At(10); + await rig.Evaluator.ApplyCustomRuleHealthAsync(Rules(1), Ct); + Assert.Equal(1, rig.Fires); + Assert.Empty(rig.History.Records); + + rig.At(20); + await rig.Evaluator.ApplyCustomRuleHealthAsync(Rules(1, 2), Ct); + Assert.Equal(2, rig.Fires); + Assert.Contains("New since the previous alert were rule 2", rig.Deliverer.Outcomes[1].DetailText!); + } + + [Fact] + public async Task AnEmptySet_ResolvesOnce_AndTheNextRuleToBreakIsAnEntryWithNoNewRuleSentence() + { + var rig = new Rig(State.CustomRuleHealth); + + await rig.Evaluator.ApplyCustomRuleHealthAsync(Rules(1), Ct); + rig.At(1); + await rig.Evaluator.ApplyCustomRuleHealthAsync(Rules(), Ct); + rig.At(2); + await rig.Evaluator.ApplyCustomRuleHealthAsync(Rules(), Ct); + Assert.Single(rig.History.Records); + + rig.At(3); + await rig.Evaluator.ApplyCustomRuleHealthAsync(Rules(7), Ct); + Assert.Equal(2, rig.Fires); + Assert.DoesNotContain("New since", rig.Deliverer.Outcomes[1].DetailText!); + } +} diff --git a/Darling/Darling.Tests/StatementAlertReadCutLiveTests.cs b/Darling/Darling.Tests/StatementAlertReadCutLiveTests.cs new file mode 100644 index 0000000000..995acb7ab4 --- /dev/null +++ b/Darling/Darling.Tests/StatementAlertReadCutLiveTests.cs @@ -0,0 +1,87 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Alerting; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5320: the long-running-query alert read returns the WHOLE statement, the statement filter judges it, and the cut to +/// the alert's 300 characters comes after. The read used to cut in SQL (SUBSTRING(r.query_text, 1, 300)), so a +/// value that sat early in a batch with the text that names it past the 300th character read clean and the alert +/// carried the start of the secret. #1776 own-store: this class registers one server of its own and removes its rows. +/// +[Collection("live-postgres")] +public sealed class StatementAlertReadCutLiveTests +{ + private const string ServerName = "statement-alert-cut-e2e"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + /// A plain statement of 500 characters: kept as the first 300 and nothing more. + private static readonly string Plain = "SELECT plain_ssf " + new string('x', 483); + + [Fact] + public async Task LongRunningQueries_JudgeTheWholeStatement_ThenCutAtThreeHundred() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live alert cut test."); + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + var collected = DarlingMcpTestData.Naive(DateTime.UtcNow).AddMinutes(-1); + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO query_snapshots (collection_id, collection_time, server_id, server_name, session_id, database_name, query_text, total_elapsed_time_ms, cpu_time_ms, reads, writes, query_hash, wait_type) +VALUES ($1, $2, $3, $4, 71, 'UriDb', $5, 900000, 1, 1, 0, '0x01', 'CXPACKET'), ($6, $2, $3, $4, 72, 'UriDb', $7, 800000, 1, 1, 0, '0x02', 'CXPACKET')", + CollectionIdGenerator.Next(), collected, ServerId, ServerName, StatementScrubCanary.UriStatement(300), + CollectionIdGenerator.Next(), Plain); + + var adapter = new DarlingAlertReadAdapter(postgres); + var read = await adapter.GetLongRunningQueriesAsync( + ServerId.ToString(System.Globalization.CultureInfo.InvariantCulture), thresholdMinutes: 5, maxResults: 5, + excludeSpServerDiagnostics: true, excludeWaitFor: true, excludeBackups: true, excludeMiscWaits: true, + excludeCdc: true, excludedDatabases: Array.Empty(), LongRunningQueryExclusions.None, ct); + + var named = read.Sessions.Single(q => q.SessionId == 71).QueryText; + Assert.DoesNotContain(StatementScrubCanary.UriSecretPartial, named, StringComparison.Ordinal); + Assert.Contains(SensitiveStatements.PlaceholderText, named, StringComparison.Ordinal); + + var plain = read.Sessions.Single(q => q.SessionId == 72).QueryText; + Assert.Equal(Plain[..300], plain); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + private static async Task DeleteRowsAsync(NpgsqlConnection connection, System.Threading.CancellationToken ct) + { + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM query_snapshots WHERE server_id = $1", ServerId); + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM servers WHERE server_id = $1", ServerId); + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM config_monitored_servers WHERE server_id = $1", ServerId); + } +} diff --git a/Darling/Darling.Tests/StatementAnalysisReadCutLiveTests.cs b/Darling/Darling.Tests/StatementAnalysisReadCutLiveTests.cs new file mode 100644 index 0000000000..b8bbb4a1e0 --- /dev/null +++ b/Darling/Darling.Tests/StatementAnalysisReadCutLiveTests.cs @@ -0,0 +1,302 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Analysis; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Analysis; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5320: Darling's analysis readers return the WHOLE statement, the statement filter judges it, and the cut comes +/// after. Each case plants a batch whose named value (a URI's user:secret@) sits inside the cut and whose +/// naming at-sign sits past it, so a reader that cuts in SQL first hands the filter a prefix that reads clean and +/// the answer holds the start of the secret. A plain statement past the cut is cut exactly as before. +/// One case per reader family: blocking, queries, activity (the bad-actor read), pileup, target deadlocks. +/// +[Collection("live-postgres")] +public sealed class StatementAnalysisReadCutLiveTests +{ + private const string ServerName = "statement-analysis-cut-e2e"; + private const string PgServerName = "statement-analysis-cut-pg-e2e"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + private static readonly int PgServerId = ServerIdHelper.GetDeterministicHashCode(PgServerName); + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + /// A plain statement of 700 characters: kept as the first 500 and nothing more. + private static readonly string Plain = "SELECT plain_ssf " + new string('x', 683); + + private static string Json(object? value) => JsonSerializer.Serialize(value); + + private static void AssertNamedWithheldAndPlainCutAtFiveHundred(string json) + { + Assert.DoesNotContain(StatementScrubCanary.UriSecretPartial, json, StringComparison.Ordinal); + Assert.Contains(SensitiveStatements.PlaceholderText, json, StringComparison.Ordinal); + Assert.Contains(Plain[..500], json, StringComparison.Ordinal); + Assert.DoesNotContain(Plain[..501], json, StringComparison.Ordinal); + } + + private sealed record Run(NpgsqlConnection Connection, NpgsqlDataSource Postgres, AnalysisContext Context, CancellationToken Ct); + + private static async Task RunAsync(Func body) + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live analysis cut test."); + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + await DarlingMcpTestData.RegisterServerAsync(connection, PgServerId, PgServerName, ct); + var end = DarlingMcpTestData.Naive(DateTime.UtcNow).AddMinutes(1); + var context = new AnalysisContext + { + ServerId = ServerId, ServerName = ServerName, TimeRangeStart = end.AddHours(-4), TimeRangeEnd = end, + ServerUtcOffset = TimeSpan.Zero, CancellationToken = ct + }; + await body(new Run(connection, postgres, context, ct)); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + private static async Task EnrichAsync(PgDrillDownCollector collector, Run run, string rootKey) + { + var finding = new AnalysisFinding { RootFactKey = rootKey, StoryPath = rootKey, PathKeys = [rootKey], Severity = 1.0 }; + await collector.EnrichFindingsAsync([finding], run.Context); + Assert.NotNull(finding.DrillDown); + return finding; + } + + [Fact] + public async Task BlockingReaders_JudgeTheWholeStatement_ThenCutAtFiveHundred() + { + await RunAsync(async run => + { + var at = run.Context.TimeRangeStart.AddMinutes(30); + await DarlingMcpTestData.ExecAsync(run.Connection, run.Ct, @" +INSERT INTO blocked_process_reports + (blocked_report_id, collection_time, server_id, server_name, event_time, database_name, blocked_spid, blocking_spid, wait_time_ms, lock_mode, blocked_sql_text, blocking_sql_text) +VALUES ($1, $2, $3, $4, $2, 'UriDb', 71, 60, 12000, 'X', $5, $6)", + CollectionIdGenerator.Next(), at, ServerId, ServerName, StatementScrubCanary.UriStatement(500), Plain); + await DarlingMcpTestData.ExecAsync(run.Connection, run.Ct, @" +INSERT INTO deadlocks (deadlock_id, collection_time, server_id, server_name, deadlock_time, victim_process_id, victim_sql_text, deadlock_graph_xml, database_name) +VALUES ($1, $2, $3, $4, $2, 'process1', $5, '', 'UriDb'), ($6, $2, $3, $4, $2, 'process2', $7, '', 'UriDb')", + CollectionIdGenerator.Next(), at, ServerId, ServerName, StatementScrubCanary.UriStatement(500), CollectionIdGenerator.Next(), Plain); + + var collector = new PgDrillDownCollector(run.Postgres); + var top = await EnrichAsync(collector, run, "BLOCKING_EVENTS"); + AssertNamedWithheldAndPlainCutAtFiveHundred(Json(top.DrillDown!["top_blocking_chains"])); + + var chains = await EnrichAsync(collector, run, "BLOCKING_CHAIN"); + AssertNamedWithheldAndPlainCutAtFiveHundred(Json(chains.DrillDown!["reconstructed_blocking_chains"])); + + var deadlocks = await EnrichAsync(collector, run, "DEADLOCKS"); + AssertNamedWithheldAndPlainCutAtFiveHundred(Json(deadlocks.DrillDown!["top_deadlocks"])); + }); + } + + [Fact] + public async Task QueryReaders_JudgeTheWholeStatement_ThenCutAtFiveHundred() + { + await RunAsync(async run => + { + var at = run.Context.TimeRangeStart.AddMinutes(30); + foreach (var (hash, text) in new[] { ("0xURICUT", StatementScrubCanary.UriStatement(500)), ("0xPLAINCUT", Plain) }) + { + await DarlingMcpTestData.ExecAsync(run.Connection, run.Ct, @" +INSERT INTO query_stats + (collection_id, collection_time, server_id, server_name, database_name, query_hash, query_plan_hash, sql_handle, plan_handle, + creation_time, query_text, delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, delta_spills, min_dop, max_dop) +VALUES ($1, $2, $3, $4, 'UriDb', $5, '0xPLAN', '0xSQLH', '0xPLANH', $6, $7, 10, 500000, 900000, 1000, 5, 1, 1)", + CollectionIdGenerator.Next(), at, ServerId, ServerName, hash, at.AddDays(-1), text); + } + + var collector = new PgDrillDownCollector(run.Postgres); + var spills = await EnrichAsync(collector, run, "QUERY_SPILLS"); + AssertNamedWithheldAndPlainCutAtFiveHundred(Json(spills.DrillDown!["top_spilling_queries"])); + + var cpu = await EnrichAsync(collector, run, "CPU_SQL_PERCENT"); + AssertNamedWithheldAndPlainCutAtFiveHundred(Json(cpu.DrillDown!["top_cpu_queries"])); + }); + } + + [Fact] + public async Task BadActorFactRead_CarriesNoStatementText() + { + await RunAsync(async run => + { + var at = run.Context.TimeRangeStart.AddMinutes(30); + await DarlingMcpTestData.ExecAsync(run.Connection, run.Ct, @" +INSERT INTO query_stats + (collection_id, collection_time, server_id, server_name, database_name, query_hash, query_plan_hash, sql_handle, plan_handle, + creation_time, query_text, delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, delta_spills, min_dop, max_dop) +VALUES ($1, $2, $3, $4, 'UriDb', '0xBADACTOR', '0xPLAN', '0xSQLH', '0xPLANH', $5, $6, 500, 90000000, 90000000, 900000, 0, 1, 1)", + CollectionIdGenerator.Next(), at, ServerId, ServerName, at.AddDays(-1), StatementScrubCanary.UriStatement(200)); + + /* The fact is keyed on the hash and carries numbers only: its read must not fetch a cut statement it never prints. */ + Assert.DoesNotContain("query_text", PgFactCollector.BadActorSql, StringComparison.Ordinal); + Assert.DoesNotContain("LEFT(", PgFactCollector.BadActorSql, StringComparison.Ordinal); + await using var cmd = new NpgsqlCommand(PgFactCollector.BadActorSql, run.Connection); + cmd.Parameters.AddWithValue(ServerId); + cmd.Parameters.AddWithValue(run.Context.TimeRangeStart); + cmd.Parameters.AddWithValue(run.Context.TimeRangeEnd); + await using var reader = await cmd.ExecuteReaderAsync(run.Ct); + Assert.True(await reader.ReadAsync(run.Ct)); + Assert.Equal(10, reader.FieldCount); + for (var i = 0; i < reader.FieldCount; i++) + Assert.NotEqual("query_text", reader.GetName(i)); + }); + } + + [Fact] + public async Task PileupReader_JudgesTheWholeStatement_AndKeepsTheIdentityTextCutTheOldWay() + { + await RunAsync(async run => + { + var at = DarlingMcpTestData.Naive(DateTime.UtcNow).AddMinutes(-5); + var named = StatementScrubCanary.UriStatement(1500); + var plain = "SELECT plain_ssf " + new string('x', 1983); + foreach (var (session, text) in new[] { (91, named), (92, plain) }) + { + await DarlingMcpTestData.ExecAsync(run.Connection, run.Ct, @" +INSERT INTO query_snapshots (collection_id, collection_time, server_id, server_name, session_id, database_name, query_text, status, total_elapsed_time_ms) +VALUES ($1, $2, $3, $4, $5, 'UriDb', $6, 'running', 9000)", + CollectionIdGenerator.Next(), at, ServerId, ServerName, session, text); + } + + var rows = await new PgPileupSnapshotReader(run.Postgres).ReadWindowAsync(ServerId, at.AddMinutes(-10), run.Ct); + var namedRow = Assert.Single(rows, r => r.SessionId == 91); + var plainRow = Assert.Single(rows, r => r.SessionId == 92); + + /* The shown text is judged whole: no start of the secret, the marker instead. */ + Assert.DoesNotContain(StatementScrubCanary.UriSecretPartial, namedRow.PreviewText, StringComparison.Ordinal); + Assert.Equal(SensitiveStatements.PlaceholderText, namedRow.PreviewText); + /* The identity text is the old SQL cut (1500 characters), unjudged, so a null-hash row groups as it did. */ + Assert.Equal(named[..1500], namedRow.QueryText); + Assert.Equal(plain[..1500], plainRow.QueryText); + Assert.Equal(plain[..1500], plainRow.PreviewText); + Assert.Equal(PgPileupSnapshotReader.StatementTextCharacters, plainRow.PreviewText!.Length); + }); + } + + [Fact] + public async Task TargetDeadlockReader_JudgesTheWholeNormalizedStatement_ThenCuts() + { + await RunAsync(async run => + { + /* The marker sits inside the 2,000-character cut; the statement that names it (CREATE ROLE) sits past it. */ + var marker = "SELECT marker_pg_ssf" + string.Concat(Enumerable.Repeat(", 1", 800)) + " ; CREATE ROLE zz_ssf"; + var plain = "SELECT plain_pg_ssf" + string.Concat(Enumerable.Repeat(", 1", 800)); + var at = run.Context.TimeRangeStart.AddMinutes(30); + foreach (var (shape, statement) in new[] { ("relation a", marker), ("relation b", plain) }) + { + await DarlingMcpTestData.ExecAsync(run.Connection, run.Ct, @" +INSERT INTO pg_deadlocks + (collection_id, collection_time, server_id, server_name, occurred_at, victim_pid, participant_count, deadlock_hash, lock_modes, resources, victim_statement, graph_text) +VALUES ($1, $2, $3, $4, $2, 4242, 2, upper(left(encode(sha256(convert_to($5, 'UTF8')), 'hex'), 32)), 'ShareLock', $6, $5, 'graph')", + CollectionIdGenerator.Next(), at, PgServerId, PgServerName, statement, shape); + } + + var context = new AnalysisContext + { + ServerId = PgServerId, ServerName = PgServerName, TimeRangeStart = run.Context.TimeRangeStart, TimeRangeEnd = run.Context.TimeRangeEnd, + ServerUtcOffset = TimeSpan.Zero, CancellationToken = run.Ct + }; + var finding = new AnalysisFinding + { + RootFactKey = PgTargetFactKeys.DeadlockRate, StoryPath = PgTargetFactKeys.DeadlockRate, + PathKeys = [PgTargetFactKeys.DeadlockRate], Severity = 1.0 + }; + await new PgTargetDrillDownCollector(run.Postgres).EnrichFindingsAsync([finding], context); + var json = Json(finding.DrillDown![PgTargetDrillDownCollector.DeadlockExemplarsSection]); + + Assert.DoesNotContain("marker_pg_ssf", json, StringComparison.Ordinal); + Assert.Contains(SensitiveStatements.PlaceholderText, json, StringComparison.Ordinal); + Assert.Contains("plain_pg_ssf", json, StringComparison.Ordinal); + }); + } + + private static string Graph(string inputBuffer, int pad = 0) => + " 0 ? " note=\"" + new string('n', pad) + "\"" : "") + ">" + + inputBuffer + ""; + + private static async Task<(string Xml, bool Truncated)> DeadlockPreviewOfAsync(Run run, string graph) + { + await DarlingMcpTestData.ExecAsync(run.Connection, run.Ct, "DELETE FROM deadlocks WHERE server_id = $1", ServerId); + await DarlingMcpTestData.ExecAsync(run.Connection, run.Ct, @" +INSERT INTO deadlocks (deadlock_id, collection_time, server_id, server_name, deadlock_time, deadlock_graph_xml, victim_sql_text) +VALUES ($1, $2, $3, $4, $2, $5, 'x')", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(DateTime.UtcNow.AddMinutes(-5)), ServerId, ServerName, graph); + var json = await PerformanceMonitor.Darling.Service.Mcp.DarlingMcpBlockingTools.GetDeadlockDetail(run.Postgres, ServerName); + var row = System.Text.Json.Nodes.JsonNode.Parse(json)!["deadlocks"]![0]!; + return ((string)row["deadlock_graph_xml"]!, (bool)row["deadlock_graph_xml_truncated"]!); + } + + [Fact] + public async Task DeadlockDetail_PreviewTruncatedFlag_IsReadOffTheFilteredGraph_NotTheRawOne() + { + await RunAsync(async run => + { + /* Shrink: a raw graph past the 2000-character preview whose long statement the filter replaces by the marker comes + back SHORT, so nothing was cut and the flag is false (the raw length said true). */ + var longSecret = "CREATE LOGIN [shrink_ssf] WITH PASSWORD = N'" + new string('s', 1500) + "'"; + var shrinking = Graph(longSecret + new string(' ', 400)); + Assert.True(shrinking.Length > 2000); + var shrunk = await DeadlockPreviewOfAsync(run, shrinking); + Assert.False(shrunk.Truncated); + Assert.DoesNotContain("PASSWORD", shrunk.Xml, StringComparison.OrdinalIgnoreCase); + Assert.True(shrunk.Xml.Length < 2000); + + /* Growth: a raw graph of exactly 2000 characters whose short statement becomes the longer marker is over 2000 + filtered, so the preview cuts it and the flag is true (the raw length said false). */ + var shortSecret = "CREATE LOGIN a WITH PASSWORD='x'"; + var padded = Graph(shortSecret, pad: 2000 - Graph(shortSecret).Length - " note=\"\"".Length); + Assert.Equal(2000, padded.Length); + Assert.True(SensitiveStatements.Xml(padded)!.Length > 2000); + var grown = await DeadlockPreviewOfAsync(run, padded); + Assert.True(grown.Truncated); + Assert.EndsWith("... (truncated)", grown.Xml, StringComparison.Ordinal); + Assert.DoesNotContain("PASSWORD", grown.Xml, StringComparison.OrdinalIgnoreCase); + + /* A plain graph is cut once and flagged once, on both lengths. */ + Assert.True((await DeadlockPreviewOfAsync(run, Graph(new string('p', 2500)))).Truncated); + Assert.False((await DeadlockPreviewOfAsync(run, Graph("SELECT 1"))).Truncated); + }); + } + + private static async Task DeleteRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + foreach (var table in new[] { "blocked_process_reports", "deadlocks", "query_stats", "query_snapshots", "pg_deadlocks" }) + { + await DarlingMcpTestData.ExecAsync(connection, ct, $"DELETE FROM {table} WHERE server_id = ANY($1)", new[] { ServerId, PgServerId }); + } + + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM servers WHERE server_id = ANY($1)", new[] { ServerId, PgServerId }); + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM config_monitored_servers WHERE server_id = ANY($1)", new[] { ServerId, PgServerId }); + } +} diff --git a/Darling/Darling.Tests/StatementAnalysisReadCutPinTests.cs b/Darling/Darling.Tests/StatementAnalysisReadCutPinTests.cs new file mode 100644 index 0000000000..c75a2b4aa3 --- /dev/null +++ b/Darling/Darling.Tests/StatementAnalysisReadCutPinTests.cs @@ -0,0 +1,109 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Analysis; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5320: no Darling analysis read cuts statement text in SQL. The text comes back whole, the statement filter +/// judges it, and cuts it after. Plans are PR C's and relation names are not +/// statement text, so neither is scanned for. +/// +[Trait("Stage", "Guard")] +public sealed class StatementAnalysisReadCutPinTests +{ + [Fact] + public void NoAnalysisReadCutsStatementTextInSql() + { + var dir = Path.GetDirectoryName(typeof(PgDrillDownCollector).Assembly.Location)!; + var root = dir; + while (root is not null && !Directory.Exists(Path.Combine(root, "Darling", "PerformanceMonitor.Darling.Analysis"))) + root = Path.GetDirectoryName(root); + Assert.NotNull(root); + + var offenders = Directory.GetFiles(Path.Combine(root!, "Darling", "PerformanceMonitor.Darling.Analysis"), "*.cs") + .Where(f => !f.EndsWith("PgDrillDownCollector.Plans.cs", StringComparison.Ordinal)) + .SelectMany(f => CutSites(File.ReadAllText(f)).Select(line => Path.GetFileName(f) + ":" + line)) + .ToList(); + + Assert.Empty(offenders); + } + + /// + /// The 1-based line of every SQL cut of a statement-named column in one C# source. Reads the string literals + /// through rather than splitting lines on a comment prefix, + /// so a block comment's continuation lines (no leading asterisk) are not read as SQL, and the SQL in a + /// verbatim or raw literal is (#3052). + /// + private static List CutSites(string source) + { + var sites = new List(); + + foreach (var (start, body) in CSharpSourceWalker.StringLiteralBodies(source)) + { + foreach (System.Text.RegularExpressions.Match m in System.Text.RegularExpressions.Regex.Matches(body, + @"LEFT\(\s*(MAX\(|COALESCE\()?\s*\w*\.?(query_text|query_sql_text|victim_sql_text|blocked_sql_text|blocking_sql_text|latest_victim_statement|latest_graph_text)\b")) + { + var line = 1; + for (var i = 0; i < start && i < source.Length; i++) + { + if (source[i] == '\n') line++; + } + + sites.Add(line + body.Take(m.Index).Count(c => c == '\n')); + } + } + + return sites; + } + + [Fact] + public void TheCutScan_ReadsLiteralsNotComments_AndFindsACutInEitherLiteralForm() + { + const string fixture = """ + class Probe + { + /* A note that mentions LEFT(query_text, 200) in a block comment + LEFT(MAX(t.query_text), 5) on a line with no asterisk. */ + // LEFT(query_text, 9) in a line comment + const string Plain = "SELECT LEFT(MAX(t.query_text), 200) FROM q"; + const string Verbatim = @"SELECT 1, + LEFT(COALESCE(d.query_text, ''), 10) FROM q"; + } + """; + + var sites = CutSites(fixture); + + Assert.Equal(2, sites.Count); + Assert.Equal(new[] { 6, 8 }, sites.ToArray()); + } + + [Fact] + public void Preview_JudgesTheWholeText_ThenCutsByCodePoint() + { + var named = StatementScrubCanary.UriStatement(500); + Assert.Equal(SensitiveStatements.PlaceholderText, AnalysisStatementText.Preview(named, 500)); + + var plain = "SELECT " + new string('x', 700); + Assert.Equal(plain[..500], AnalysisStatementText.Preview(plain, 500)); + Assert.Null(AnalysisStatementText.Preview(null, 500)); + + /* PostgreSQL's LEFT counts code points: 3 astral characters are 6 UTF-16 units and still 3 characters. */ + var astral = "\U0001F600\U0001F600\U0001F600\U0001F600"; + Assert.Equal("\U0001F600\U0001F600\U0001F600", AnalysisStatementText.CutCodePoints(astral, 3)); + Assert.Equal("ab", AnalysisStatementText.CutCodePoints("abcd", 2)); + Assert.Equal("abcd", AnalysisStatementText.CutCodePoints("abcd", 4)); + } +} diff --git a/Darling/Darling.Tests/StatementCollectionCensusCases.cs b/Darling/Darling.Tests/StatementCollectionCensusCases.cs new file mode 100644 index 0000000000..d38560ba9e --- /dev/null +++ b/Darling/Darling.Tests/StatementCollectionCensusCases.cs @@ -0,0 +1,53 @@ +// Copyright (c) Erik Darling Data. All rights reserved. +// Licensed under the terms in the LICENSE file in the repository root. + +using System; +using System.Collections.Generic; + +namespace Darling.Tests; + +/// +/// #4348 (statement filter): the registry of collection-time census cases. A case is a test that plants the canary +/// in a collector's input, runs the read and the write, and asserts that no written value holds a secret needle +/// (StatementScrubRecordingWriter is the harness). Each lane that hooks a column through the statement filter +/// adds one line here, naming the test that proves it: the key is "definition.column", exactly as +/// StatementColumnCensusTests.Listed spells it, and the value is "StatementCollectionCensusTests.TestMethod": a +/// [Fact] or [Theory] with no skip, in the class StatementCollectionCensusTests (partial, so each +/// lane can add its file), whose source drives StatementScrubRecordingWriter and names the column. +/// StatementColumnCensusTests.EveryNonPendingHookedColumn_HasACollectionCase reads this registry: a Hooked +/// entry cannot be flipped to non-pending without a case here, and a case here that names a missing test or a column +/// that is not a Hooked entry fails too. +/// +internal static class StatementCollectionCensusCases +{ + public static readonly IReadOnlyDictionary ByColumn = + new Dictionary(StringComparer.Ordinal) + { + // Filled by the lanes that hook a column, one line each, for example: + // ["query_stats.query_text"] = "StatementCollectionCensusTests.QueryStats_QueryText_IsWithheld", + ["long_query_completions.statement_text"] = "StatementCollectionCensusTests.LongQueryCompletions_StatementText_IsWithheldAndThePlainStatementIsUntouched", + ["default_trace_events.text_data"] = "StatementCollectionCensusTests.DefaultTraceEvents_TextData_IsWithheldAndThePlainStatementIsUntouched", + ["system_health_events.event_xml"] = "StatementCollectionCensusTests.SystemHealthEvents_EventXml_WithholdsTheSqlTextActionAndKeepsTheRestOfTheEvent", + ["job_history.message"] = "StatementCollectionCensusTests.JobHistory_Message_IsWithheldWhenItEchoesAStatement_AndAPlainMessageIsUntouched", + ["query_store.query_text"] = "StatementCollectionCensusTests.QueryStore_QueryText_IsWithheld", + ["query_snapshots.query_text"] = "StatementCollectionCensusTests.QuerySnapshots_QueryText_IsWithheld", + ["query_snapshots.query_plan"] = "StatementCollectionCensusTests.QuerySnapshots_QueryPlan_IsFiltered", + ["query_snapshots.live_query_plan"] = "StatementCollectionCensusTests.QuerySnapshots_LiveQueryPlan_IsFiltered", + ["plan_correction.query_text"] = "StatementCollectionCensusTests.PlanCorrection_QueryText_IsWithheld", + ["plan_correction.implementation_script"] = "StatementCollectionCensusTests.PlanCorrection_ImplementationScript_IsWithheld", + ["query_stats.query_text"] = "StatementCollectionCensusTests.QueryStats_QueryText_IsWithheld", + ["query_stats.query_plan_xml"] = "StatementCollectionCensusTests.QueryStats_QueryPlanXml_InlinePlanIsFiltered", + ["procedure_stats.query_plan_xml"] = "StatementCollectionCensusTests.ProcedureStats_QueryPlanXml_InlinePlanIsFiltered", + // R4 (blocking and deadlocks), StatementScrubBlockingTests.cs: + ["blocked_process_report.blocked_sql_text"] = "StatementCollectionCensusTests.BlockedProcessReport_blocked_sql_text_blocking_sql_text_blocked_process_report_xml_and_both_plans_AreWithheld", + ["blocked_process_report.blocking_sql_text"] = "StatementCollectionCensusTests.BlockedProcessReport_blocked_sql_text_blocking_sql_text_blocked_process_report_xml_and_both_plans_AreWithheld", + ["blocked_process_report.blocked_process_report_xml"] = "StatementCollectionCensusTests.BlockedProcessReport_blocked_sql_text_blocking_sql_text_blocked_process_report_xml_and_both_plans_AreWithheld", + ["blocked_process_report.blocked_query_plan_xml"] = "StatementCollectionCensusTests.BlockedProcessReport_blocked_sql_text_blocking_sql_text_blocked_process_report_xml_and_both_plans_AreWithheld", + ["blocked_process_report.blocking_query_plan_xml"] = "StatementCollectionCensusTests.BlockedProcessReport_blocked_sql_text_blocking_sql_text_blocked_process_report_xml_and_both_plans_AreWithheld", + ["deadlocks.victim_sql_text"] = "StatementCollectionCensusTests.Deadlocks_victim_sql_text_deadlock_graph_xml_and_victim_query_plan_xml_AreWithheld", + ["deadlocks.deadlock_graph_xml"] = "StatementCollectionCensusTests.Deadlocks_victim_sql_text_deadlock_graph_xml_and_victim_query_plan_xml_AreWithheld", + ["deadlocks.victim_query_plan_xml"] = "StatementCollectionCensusTests.Deadlocks_victim_sql_text_deadlock_graph_xml_and_victim_query_plan_xml_AreWithheld", + ["dmv_blocking_snapshot.blocked_sql_text"] = "StatementCollectionCensusTests.DmvBlockingSnapshot_blocked_sql_text_and_blocking_sql_text_AreWithheld", + ["dmv_blocking_snapshot.blocking_sql_text"] = "StatementCollectionCensusTests.DmvBlockingSnapshot_blocked_sql_text_and_blocking_sql_text_AreWithheld", + }; +} diff --git a/Darling/Darling.Tests/StatementColumnCensusTests.cs b/Darling/Darling.Tests/StatementColumnCensusTests.cs new file mode 100644 index 0000000000..ce04873644 --- /dev/null +++ b/Darling/Darling.Tests/StatementColumnCensusTests.cs @@ -0,0 +1,972 @@ +// Copyright (c) Erik Darling Data. All rights reserved. +// Licensed under the terms in the LICENSE file in the repository root. + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Reflection; +using System.Text; +using System.Text.RegularExpressions; +using PerformanceMonitor.Collectors; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348 (statement filter, PR B): a census over every SQL Server collector definition's payload columns. A column +/// whose name looks like it can hold statement text, a plan, an event document or a script (the +/// ) must be listed below as one of three things, so a new column cannot reach the +/// store without somebody deciding what happens to it: +/// +/// Hooked: the collector passes the value through the statement filter before the write. Each entry +/// names its row in the plan's collection-write-path table and the lane that lands the hook. A row whose hook has +/// not landed yet is "pending: Rn"; the integration branch flips to false per row. +/// NullByDesign: the collector writes NULL there on purpose. +/// Exempt: the name matches but the value is not statement content, with the reason spelled out. +/// +/// The pattern also matches input_buffer, batch_text, command and definition. +/// A Hooked entry that is not must have a collection-time case registered in +/// (), so +/// flipping an entry cannot pass without a hook that a test exercises. The census sees definition payload columns +/// only; lists the writers that do not go through a definition (a plan or text store +/// fed outside the collector pipeline, the slow-read log) and a source scan checks each one for a +/// SensitiveStatements call once it is no longer pending. +/// A new matching column fails until it is listed. A listed +/// column that no longer exists or no longer matches fails , +/// so the list cannot rot. PostgreSQL definitions are out of scope: they are a separate engine with their own +/// filter (PgSensitiveStatementFilter). +/// +[Trait("Stage", "Guard")] +public sealed class StatementColumnCensusTests +{ + private static readonly Regex PayloadPattern = new( + "sql_text$|query_text$|statement_text$|text_data$|_xml$|query_plan|event_xml|implementation_script|^message$|input_buffer|batch_text|command|definition", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + + private enum Kind + { + Hooked, + NullByDesign, + Exempt, + } + + private sealed record Entry(Kind Kind, string Detail, string? Lane = null, bool Pending = false); + + private static Entry Hooked(string row, string lane, bool pending = true) => + new(Kind.Hooked, "6.7 row " + row, lane, pending); + + private static Entry Null(string reason) => new(Kind.NullByDesign, reason); + + private static Entry Exempt(string reason) => new(Kind.Exempt, reason); + + /// Keyed "definition.column" (the definition's Name, which is what schedules and logs call it). + private static readonly Dictionary Listed = new(StringComparer.Ordinal) + { + /* Rows 1-4: query_stats and procedure_stats text and plans, inline and deferred fetch (R2). */ + ["query_stats.query_text"] = Hooked("1", "R2", pending: false), + ["query_stats.query_plan_xml"] = Hooked("2 and 3 (deferred plan fetch)", "R2", pending: false), + ["procedure_stats.query_plan_xml"] = Hooked("3 and 4 (deferred plan fetch, inline)", "R2", pending: false), + ["query_stats.query_plan_hash"] = Exempt("a hash of the plan, not plan text; the filter has nothing to read"), + ["query_stats.query_plan_xml_bytes"] = Exempt("the byte length of the stored plan, a number"), + ["procedure_stats.query_plan_xml_bytes"] = Exempt("the byte length of the stored plan, a number"), + + /* Row 5: Query Store. Lite stores it live; Darling stores it live and from backfill (R3). */ + ["query_store.query_text"] = Hooked("5", "R3", pending: false), + ["query_store.query_plan_hash"] = Exempt("a hash of the plan, not plan text; the filter has nothing to read"), + ["query_store.query_plan_text"] = Null("the main Query Store query writes a typed NULL here (QueryStoreCollector.cs:738); plan text reaches the store only through the separate by-ids fetch and the plan writer (6.7 row 7, R3)"), + + /* Row 8: the live snapshot of running requests (R6). */ + ["query_snapshots.query_text"] = Hooked("8", "R6", pending: false), + ["query_snapshots.query_plan"] = Hooked("8", "R6", pending: false), + ["query_snapshots.live_query_plan"] = Hooked("8", "R6", pending: false), + + /* Rows 9-11: blocked process reports (R4). */ + ["blocked_process_report.blocked_sql_text"] = Hooked("9", "R4", pending: false), + ["blocked_process_report.blocking_sql_text"] = Hooked("9", "R4", pending: false), + ["blocked_process_report.blocked_process_report_xml"] = Hooked("10", "R4", pending: false), + ["blocked_process_report.blocked_query_plan_xml"] = Hooked("11", "R4", pending: false), + ["blocked_process_report.blocking_query_plan_xml"] = Hooked("11", "R4", pending: false), + + /* Rows 12-14: deadlocks (R4). */ + ["deadlocks.victim_sql_text"] = Hooked("12", "R4", pending: false), + ["deadlocks.deadlock_graph_xml"] = Hooked("13", "R4", pending: false), + ["deadlocks.victim_query_plan_xml"] = Hooked("14", "R4", pending: false), + + /* Row 15: the DMV blocking snapshot (R4). */ + ["dmv_blocking_snapshot.blocked_sql_text"] = Hooked("15", "R4", pending: false), + ["dmv_blocking_snapshot.blocking_sql_text"] = Hooked("15", "R4", pending: false), + + /* Rows 16-18 and 21: the event and history collectors (R5). */ + ["long_query_completions.statement_text"] = Hooked("16", "R5", pending: false), + ["default_trace_events.text_data"] = Hooked("17", "R5", pending: false), + ["system_health_events.event_xml"] = Hooked("18", "R5", pending: false), + ["job_history.message"] = Hooked("21", "R5", pending: false), + + /* The widened pattern (input_buffer, batch_text, command, definition) found one more column. Plan 6.7 names it. */ + ["index_object_stats.filter_definition"] = Exempt("an index filter predicate over bracketed column names (IndexObjectStatsCollector.cs), not a statement"), + + /* Row 19: plan correction (R6). */ + ["plan_correction.query_text"] = Hooked("19", "R6", pending: false), + ["plan_correction.implementation_script"] = Hooked("19", "R6", pending: false), + }; + + /// + /// Columns the pattern does not match but the plan names as exempt, so a later rename into the pattern, or a + /// removal, is noticed. Each must exist in its definition and must NOT match the pattern (a match belongs in + /// ). + /// + private static readonly Dictionary WatchedExempt = new(StringComparer.Ordinal) + { + ["waiting_tasks.resource_description"] = "the collector writes a literal NULL there (WaitingTasksCollector.cs)", + }; + + private static IEnumerable<(string Definition, string Column)> SqlServerPayloadColumns() => + CollectorCatalog.All + .Where(d => d.TargetEngine == CollectorTargetEngine.SqlServer) + .SelectMany(d => d.PayloadColumns.Select(c => (d.Name, c.Name))); + + /// The census itself, over any set of (definition, column) pairs; the plant test feeds it a fake one. + private static string[] Unlisted(IEnumerable<(string Definition, string Column)> columns) => + columns + .Where(c => PayloadPattern.IsMatch(c.Column)) + .Select(c => c.Definition + "." + c.Column) + .Where(key => !Listed.ContainsKey(key)) + .OrderBy(k => k, StringComparer.Ordinal) + .ToArray(); + + [Fact] + public void EveryMatchingPayloadColumn_IsListed() + { + var unlisted = Unlisted(SqlServerPayloadColumns()); + + Assert.True( + unlisted.Length == 0, + "These SQL Server payload columns look like statement, plan or event content but are not in " + + "StatementColumnCensusTests.Listed. Hook each one through the statement filter (and list it as Hooked " + + "with its plan row and lane), or list it as NullByDesign or Exempt with the reason: " + + string.Join(", ", unlisted)); + } + + [Fact] + public void ANewMatchingColumn_IsReportedUntilListed() + { + var planted = SqlServerPayloadColumns() + .Append(("query_stats", "extra_statement_text")) + .Append(("wait_stats", "wait_type")) + .ToArray(); + + var unlisted = Unlisted(planted); + + Assert.Equal(new[] { "query_stats.extra_statement_text" }, unlisted); + } + + [Fact] + public void EveryListedColumn_StillMatchesADefinition() + { + var actual = SqlServerPayloadColumns() + .Where(c => PayloadPattern.IsMatch(c.Column)) + .Select(c => c.Definition + "." + c.Column) + .ToHashSet(StringComparer.Ordinal); + + var stale = Listed.Keys.Where(k => !actual.Contains(k)).OrderBy(k => k, StringComparer.Ordinal).ToArray(); + + Assert.True( + stale.Length == 0, + "These census entries name a column that no longer exists in a SQL Server definition or no longer " + + "matches the payload pattern; remove or fix them: " + string.Join(", ", stale)); + } + + [Fact] + public void EveryWatchedExemptColumn_ExistsAndStaysOutsideThePattern() + { + var all = SqlServerPayloadColumns().Select(c => c.Definition + "." + c.Column).ToHashSet(StringComparer.Ordinal); + + foreach (var (key, reason) in WatchedExempt) + { + Assert.False(string.IsNullOrWhiteSpace(reason), key + " needs a reason"); + Assert.True(all.Contains(key), key + " is no longer a SQL Server payload column; remove it from WatchedExempt"); + + var column = key[(key.IndexOf('.', StringComparison.Ordinal) + 1)..]; + Assert.False(PayloadPattern.IsMatch(column), key + " now matches the payload pattern; move it into Listed"); + } + } + + [Fact] + public void EveryEntry_CarriesItsReasonRowAndLane() + { + foreach (var (key, entry) in Listed) + { + Assert.False(string.IsNullOrWhiteSpace(entry.Detail), key + " needs a plan row or a reason"); + + if (entry.Kind == Kind.Hooked) + { + Assert.Matches("^R[0-9]+$", entry.Lane ?? string.Empty); + Assert.StartsWith("6.7 row ", entry.Detail, StringComparison.Ordinal); + } + else + { + Assert.Null(entry.Lane); + Assert.False(entry.Pending, key + ": only a Hooked entry can be pending"); + } + } + } + + [Fact] + public void TheCensusCountsWhatItReports() + { + var matched = SqlServerPayloadColumns().Count(c => PayloadPattern.IsMatch(c.Column)); + Assert.Equal(matched, Listed.Count); + + Assert.Equal(matched, Listed.Values.Count(e => e.Kind == Kind.Hooked) + + Listed.Values.Count(e => e.Kind == Kind.NullByDesign) + + Listed.Values.Count(e => e.Kind == Kind.Exempt)); + } + + // ── non-pending Hooked entries need a collection-time case ── + + /// + /// The one class a registered collection case must sit in: the collection census, the tests that plant the canary + /// in a collector's input and read what StatementScrubRecordingWriter recorded. It may be a partial class + /// spread over several files. + /// + internal const string CollectionCensusClass = "StatementCollectionCensusTests"; + + /// A real test: a [Fact] or [Theory] that is not skipped, conditionally skipped or explicit. + private static bool IsRunnableTest(MethodInfo method) + { + var facts = method.GetCustomAttributes(inherit: true).ToArray(); + return facts.Length > 0 + && facts.All(f => string.IsNullOrEmpty(f.Skip) + && string.IsNullOrEmpty(f.SkipUnless) + && string.IsNullOrEmpty(f.SkipWhen) + && !f.Explicit); + } + + /// The source of every file that declares , joined (empty when none is found). + private static string CensusClassSource(Type type) + { + var dir = Path.GetDirectoryName(RepoFile.PathTo("Darling", "Darling.Tests", "StatementColumnCensusTests.cs"))!; + var declaration = new Regex(@"\bclass\s+" + Regex.Escape(type.Name) + @"\b"); + return string.Join("\n", Directory.EnumerateFiles(dir, "*.cs", SearchOption.AllDirectories) + .Where(f => !f.Contains(Path.DirectorySeparatorChar + "obj" + Path.DirectorySeparatorChar, StringComparison.Ordinal) + && !f.Contains(Path.DirectorySeparatorChar + "bin" + Path.DirectorySeparatorChar, StringComparison.Ordinal)) + .Select(File.ReadAllText) + .Where(t => declaration.IsMatch(t))); + } + + /// + /// The problems with a registry: a key that is not a Hooked entry, or a case that does not name a runnable test + /// (a method that exists, carries [Fact]/[Theory] with no skip) in the collection census class whose + /// source drives StatementScrubRecordingWriter, names the column, and does not skip at run time. + /// + private static string[] RegistryProblems(IReadOnlyDictionary cases) => + RegistryProblems(cases, typeof(StatementColumnCensusTests).Assembly.GetTypes(), CollectionCensusClass, CensusClassSource); + + private static string[] RegistryProblems( + IReadOnlyDictionary cases, IEnumerable types, string censusClass, Func sourceOf) + { + var problems = new List(); + var typeList = types.ToArray(); + + foreach (var (key, test) in cases.OrderBy(c => c.Key, StringComparer.Ordinal)) + { + if (!Listed.TryGetValue(key, out var entry) || entry.Kind != Kind.Hooked) + { + problems.Add(key + " is not a Hooked entry of Listed"); + continue; + } + + var parts = test.Split('.'); + if (parts.Length != 2 || parts[0] != censusClass) + { + problems.Add(key + " names " + test + ", which is not a test in " + censusClass); + continue; + } + + var type = typeList.FirstOrDefault(t => t.Name == censusClass); + var method = type? + .GetMethods(BindingFlags.Public | BindingFlags.Instance | BindingFlags.Static) + .FirstOrDefault(m => m.Name == parts[1] && IsRunnableTest(m)); + if (type is null || method is null) + { + problems.Add(key + " names " + test + ", which is not a runnable [Fact] or [Theory] (none, skipped, or explicit)"); + continue; + } + + var source = sourceOf(type); + var column = key[(key.IndexOf('.') + 1)..]; + if (!source.Contains("StatementScrubRecordingWriter", StringComparison.Ordinal) + || !source.Contains(column, StringComparison.Ordinal) + || source.Contains("Assert.Skip", StringComparison.Ordinal)) + { + problems.Add(key + " names " + test + ", but " + censusClass + " does not drive StatementScrubRecordingWriter " + + "for " + column + " (or it skips at run time)"); + } + } + + return problems.ToArray(); + } + + /// The Hooked entries that claim their hook landed (not pending) and have no registered case. + private static string[] NonPendingWithoutACase(IReadOnlyDictionary cases) => + Listed.Where(e => e.Value.Kind == Kind.Hooked && !e.Value.Pending && !cases.ContainsKey(e.Key)) + .Select(e => e.Key) + .OrderBy(k => k, StringComparer.Ordinal) + .ToArray(); + + [Fact] + public void EveryNonPendingHookedColumn_HasACollectionCase() + { + var missing = NonPendingWithoutACase(StatementCollectionCensusCases.ByColumn); + + Assert.True( + missing.Length == 0, + "These Hooked entries are not pending but have no collection-time case in " + + "StatementCollectionCensusCases.ByColumn (a test that plants the canary in the collector's input and " + + "asserts what is written): " + string.Join(", ", missing)); + } + + [Fact] + public void EveryRegisteredCollectionCase_NamesARunnableTestOnAHookedColumn() + { + var problems = RegistryProblems(StatementCollectionCensusCases.ByColumn); + + Assert.True(problems.Length == 0, string.Join("; ", problems)); + } + + // Fixtures for the registry self-test. Not public, so xunit does not run them (xUnit1000 asks for public). +#pragma warning disable xUnit1000 + private sealed class FixtureCensus + { + [Fact] + public void Runs() { } + + [Fact(Skip = "fixture")] + public void Skipped() { } + + [Fact(Explicit = true)] + public void NotRunByDefault() { } + + [Theory(Skip = "fixture")] + [InlineData(1)] + public void SkippedTheory(int x) => _ = x; + + public void NoAttribute() { } + } + + private sealed class FixtureOtherClass + { + [Fact] + public void Runs() { } + } +#pragma warning restore xUnit1000 + + [Fact] + public void TheRegistryChecks_CatchABrokenCaseAndAnUncoveredEntry() + { + // With no cases at all, every non-pending Hooked entry is named (none today: all are pending). + var empty = new Dictionary(StringComparer.Ordinal); + Assert.Equal( + Listed.Count(e => e.Value.Kind == Kind.Hooked && !e.Value.Pending), + NonPendingWithoutACase(empty).Length); + + // The real registry check: a missing class, a column that is not Hooked, and a runnable test in a class that is + // not the collection census, each fail. + var broken = new Dictionary(StringComparer.Ordinal) + { + ["query_stats.query_text"] = "NoSuchClass.NoSuchMethod", + ["query_stats.query_plan_hash"] = "StatementColumnCensusTests.EveryEntry_CarriesItsReasonRowAndLane", + }; + Assert.Equal(2, RegistryProblems(broken).Length); + + // Over the fixture census class (named FixtureCensus here): only a runnable test whose source drives the + // recording writer for the column is accepted. + var types = new[] { typeof(FixtureCensus), typeof(FixtureOtherClass) }; + string Driving(Type _) => "var w = new StatementScrubRecordingWriter(); // query_text"; + string Unrelated(Type _) => "plain test with no harness"; + + string[] Check(string test, Func source) => + RegistryProblems( + new Dictionary(StringComparer.Ordinal) { ["query_stats.query_text"] = test }, + types, nameof(FixtureCensus), source); + + Assert.Empty(Check("FixtureCensus.Runs", Driving)); + Assert.Single(Check("FixtureCensus.Skipped", Driving)); + Assert.Single(Check("FixtureCensus.NotRunByDefault", Driving)); + Assert.Single(Check("FixtureCensus.SkippedTheory", Driving)); + Assert.Single(Check("FixtureCensus.NoAttribute", Driving)); + Assert.Single(Check("FixtureCensus.Missing", Driving)); + Assert.Single(Check("FixtureOtherClass.Runs", Driving)); + Assert.Single(Check("FixtureCensus.Runs", Unrelated)); + Assert.Single(Check("FixtureCensus.Runs", _ => "new StatementScrubRecordingWriter(); query_text; Assert.SkipWhen(x, y);")); + } + + // ── writers outside the definitions ── + + /// + /// A writer that can put statement text, a plan or a caller's arguments into a store without going through a + /// collector definition's payload columns. Sites are the (file, type, method) places the hook may live; + /// every site must still exist, and in a non-pending entry EACH site must judge (a SensitiveStatements call, + /// a session it received) or call another site of the entry that does. + /// + private sealed record Watched( + string Name, string Row, string Lane, bool Pending, params (string File, string Type, string Method)[] Sites); + + private const string Runner = "Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs"; + + private static readonly Watched[] WatchedWriters = + { + // 6.7 row 7: Query Store plans by id. The runner fetches and hands them to the plan writer. + new("query store plans by id", "6.7 row 7", "R3", false, + (Runner, "DarlingCollectorRunner", "FetchAndStorePlansAsync"), + ("Darling/PerformanceMonitor.Darling.Storage/QueryStorePlanWriter.cs", "QueryStorePlanWriter", "WriteAsync")), + + // 6.7 row 6: the Query Store text store. + new("query store text store", "6.7 row 6", "R3", false, + (Runner, "DarlingCollectorRunner", "FetchAndStoreQueryTextAsync"), + ("Darling/PerformanceMonitor.Darling.Storage/QueryStoreTextWriter.cs", "QueryStoreTextWriter", "WriteAsync")), + + // 6.7 row 20: the oversized-plan sweep fetches one plan and records it in the backlog. + new("oversized plan sweep", "6.7 row 20", "R6", false, + ("Darling/PerformanceMonitor.Darling.Service/OversizedPlanBacklogSweep.cs", "OversizedPlanBacklogSweep", "FetchOnePlanAsync"), + ("Darling/PerformanceMonitor.Darling.Service/OversizedPlanBacklogSweep.cs", "OversizedPlanBacklogSweep", "JudgeFetchedPlan")), + + // The fetch_plan and execute_actual_plan command results store a plan in result_json. Both handlers must hand + // the plan to PlanResultOutcome (the judge): the per-method rule fails a handler that serializes planXml + // itself (#5367 review round 2, N1; the live test's fake host calls PlanResultOutcome on its own, so it + // cannot see such a revert). + new("plan command results", "plan commands (fetch_plan, execute_actual_plan)", "E", false, + ("Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs", "DarlingWorker", "RunFetchPlanAsync"), + ("Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs", "DarlingWorker", "RunExecuteActualPlanAsync"), + ("Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs", "DarlingWorker", "PlanResultOutcome")), + + // The slow-read log stores up to 4 KB of a slow call's arguments (a plan or a statement a caller passed). + // L7: every surface's offer ends in SlowReadLog.Offer, which judges the arguments before they are stored. + new("slow-read log: the MCP filter's offer", "plan section 1 (slow-read log)", "L7 and L8", false, + ("Darling/PerformanceMonitor.Darling.Service/Mcp/McpToolLatencyFilter.cs", "McpToolLatencyFilter", "OfferSlow"), + ("Darling/PerformanceMonitor.Darling.Service/SlowReadLog.cs", "SlowReadLog", "Offer")), + new("slow-read log: the log's offer", "plan section 1 (slow-read log)", "L7 and L8", false, + ("Darling/PerformanceMonitor.Darling.Service/SlowReadLog.cs", "SlowReadLog", "Offer")), + }; + + private static readonly Regex SessionCall = new( + @"SensitiveStatements\s*\.\s*(Session|Text|Xml|Json)\b", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + + /// The source with comments, and the inside of string and character literals, blanked to spaces + /// (newlines kept), so a brace or a name in a comment or a string is not read as code. + internal static string CodeOnly(string source) + { + var sb = new StringBuilder(source.Length); + var i = 0; + while (i < source.Length) + { + var c = source[i]; + var next = i + 1 < source.Length ? source[i + 1] : '\0'; + + if (c == '/' && next == '/') + { + while (i < source.Length && source[i] != '\n') + { + sb.Append(' '); + i++; + } + } + else if (c == '/' && next == '*') + { + var end = source.IndexOf("*/", i + 2, StringComparison.Ordinal); + end = end < 0 ? source.Length : end + 2; + for (; i < end; i++) + { + sb.Append(source[i] == '\n' ? '\n' : ' '); + } + } + else if (c == '"' && source.AsSpan(i).StartsWith("\"\"\"", StringComparison.Ordinal)) + { + var end = source.IndexOf("\"\"\"", i + 3, StringComparison.Ordinal); + end = end < 0 ? source.Length : end + 3; + for (; i < end; i++) + { + sb.Append(source[i] == '\n' ? '\n' : ' '); + } + } + else if (c == '"' || c == '\'') + { + var verbatim = c == '"' && i > 0 && (source[i - 1] == '@' || (i > 1 && source[i - 1] == '$' && source[i - 2] == '@')); + sb.Append(' '); + i++; + while (i < source.Length) + { + var d = source[i]; + if (!verbatim && d == '\\') + { + sb.Append(" "); + i += 2; + continue; + } + + if (d == c) + { + if (verbatim && i + 1 < source.Length && source[i + 1] == c) + { + sb.Append(" "); + i += 2; + continue; + } + + sb.Append(' '); + i++; + break; + } + + if (d == '\n' && !verbatim) + { + break; + } + + sb.Append(d == '\n' ? '\n' : ' '); + i++; + } + } + else + { + sb.Append(c); + i++; + } + } + + return sb.ToString(); + } + + /// The bodies of every declaration of in (empty + /// when the type or the method is not there). A declaration is a line that starts with modifiers or a return + /// type and then the name and "(", followed by a block body. + internal static List BodiesOf(string source, string type, string method) => + DeclarationsOf(source, type, method).Select(d => d.Body).ToList(); + + /// As , with each declaration's parameter list beside its body, so a writer that + /// is HANDED a SensitiveStatements.Session (the batch's, rather than one it makes) can be recognised. + internal static List<(string Parameters, string Body)> DeclarationsOf(string source, string type, string method) + { + var bodies = new List<(string Parameters, string Body)>(); + var code = CodeOnly(source); + if (!Regex.IsMatch(code, @"\b(class|struct|record|interface)\s+" + Regex.Escape(type) + @"\b")) + { + return bodies; + } + + var declaration = new Regex( + @"^[ \t]*(?!(?:return|await|new|else|throw|yield)\b)(?:[\w<>\[\],.?()]+[ \t]+)+" + Regex.Escape(method) + @"[ \t]*(?:<[^>\r\n]*>)?[ \t]*\(", + RegexOptions.Multiline | RegexOptions.CultureInvariant); + + foreach (Match m in declaration.Matches(code)) + { + var depth = 0; + var i = m.Index + m.Length - 1; + for (; i < code.Length; i++) + { + if (code[i] == '(') + { + depth++; + } + else if (code[i] == ')' && --depth == 0) + { + break; + } + } + + var open = i; + while (open < code.Length && code[open] != '{' && code[open] != ';' + && !(code[open] == '=' && open + 1 < code.Length && code[open + 1] == '>')) + { + open++; + } + + if (open >= code.Length || code[open] != '{') + { + continue; // an abstract, interface or expression-bodied member: no block body to scan + } + + var braces = 0; + var close = open; + for (; close < code.Length; close++) + { + if (code[close] == '{') + { + braces++; + } + else if (code[close] == '}' && --braces == 0) + { + break; + } + } + + var parametersStart = m.Index + m.Length - 1; + bodies.Add(( + code[parametersStart..Math.Max(parametersStart, Math.Min(open, code.Length))], + code[open..Math.Min(close + 1, code.Length)])); + } + + return bodies; + } + + private static bool HasSessionCall(string body) => SessionCall.IsMatch(body); + + private static readonly Regex SessionParameter = new( + @"\bSensitiveStatements\s*\.\s*Session\??\s+(?\w+)", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + + /// + /// A writer that is passed the batch's session rather than making one: it names a + /// SensitiveStatements.Session parameter and its body calls a judging method (Text, TryText, + /// Xml, TryXml) on that parameter. Merely receiving a session, or judging through some other + /// object, is not hooking. + /// + private static bool JudgesThroughAReceivedSession((string Parameters, string Body) declaration) + { + foreach (Match p in SessionParameter.Matches(declaration.Parameters)) + { + var call = new Regex( + @"(? + /// Whether (of a method on ) calls the site + /// : an unqualified call when it is on the same type, Type.Method( otherwise, or a + /// call on a private instance field (_field.Method() that declares with the + /// target's type: the MCP filter hands a slow call's arguments to the slow-read log through its + /// _slowReads field (#5320, dev's #5362 merged into #5367). A field of any other type does not count + /// (#5367 review: a writer calling _queue.Offer( on an unrelated queue is not hooked). + /// + private static bool CallsSite(string body, string ownType, (string Type, string Method) target, string fileSource) + { + if (target.Type == ownType) + { + return Regex.IsMatch(body, @"(?_\w+)\s*\.\s*" + Regex.Escape(target.Method) + @"\s*\(", + RegexOptions.CultureInvariant)) + { + // The field must be declared in this file with the target's type, readonly or not, nullable or not. + var declared = @"\b" + Regex.Escape(target.Type) + @"\??\s+" + Regex.Escape(call.Groups["field"].Value) + @"\b"; + if (Regex.IsMatch(fileSource, declared, RegexOptions.CultureInvariant)) + { + return true; + } + } + + return false; + } + + /// The problems with a watched list, reading each site's file through . + private static string[] WatchedProblems(IEnumerable writers, Func read) + { + var problems = new List(); + foreach (var w in writers) + { + if (string.IsNullOrWhiteSpace(w.Lane) || string.IsNullOrWhiteSpace(w.Row)) + { + problems.Add(w.Name + " needs a plan row and an owning lane"); + } + + // Per method (review of #5367, A-L2): every site is a method that returns or hands on plan or statement + // text, and each one must filter. It filters when it judges directly, judges through a session it + // received, or calls another site of the same writer that does (the runner's fetch methods hand a session + // to the store writer; the sweep's fetch returns what its judging step returns). A second method that + // does none of these fails even when a sibling is hooked, so one hooked method cannot vouch for the rest. + var sites = new List<(string File, string Type, string Method, List<(string Parameters, string Body)> Bodies)>(); + foreach (var (file, type, method) in w.Sites) + { + var source = read(file); + var bodies = source is null ? new List<(string Parameters, string Body)>() : DeclarationsOf(source, type, method); + if (bodies.Count == 0) + { + problems.Add(w.Name + ": " + type + "." + method + " in " + file + " no longer exists; update WatchedWriters"); + continue; + } + + sites.Add((file, type, method, bodies)); + } + + var filtering = new HashSet<(string Type, string Method)>(); + foreach (var site in sites) + { + if (site.Bodies.Any(d => HasSessionCall(d.Body) || JudgesThroughAReceivedSession(d))) + { + filtering.Add((site.Type, site.Method)); + } + } + + bool changed; + do + { + changed = false; + foreach (var site in sites.Where(x => !filtering.Contains((x.Type, x.Method)))) + { + if (filtering.Any(f => site.Bodies.Any(d => CallsSite(d.Body, site.Type, f, read(site.File) ?? "")))) + { + filtering.Add((site.Type, site.Method)); + changed = true; + } + } + } + while (changed); + + if (!w.Pending) + { + foreach (var site in sites.Where(x => !filtering.Contains((x.Type, x.Method)))) + { + problems.Add(w.Name + ": " + site.Type + "." + site.Method + " is not pending but neither calls SensitiveStatements nor hands the text to a method that does"); + } + } + } + + return problems.ToArray(); + } + + private static string? ReadSiteFile(string relative) + { + var path = RepoFile.PathTo(relative); + return File.Exists(path) ? File.ReadAllText(path) : null; + } + + [Fact] + public void EveryWatchedWriter_StillExists_AndANonPendingOneCallsTheFilter() + { + var problems = WatchedProblems(WatchedWriters, ReadSiteFile); + + Assert.True(problems.Length == 0, string.Join("; ", problems)); + Assert.Equal(WatchedWriters.Length, WatchedWriters.Select(w => w.Name).Distinct(StringComparer.Ordinal).Count()); + } + + [Fact] + public void TheWatchedScan_SeesACallAMissingMethodAndACommentedCall() + { + const string Source = @" +namespace N +{ + public sealed class W + { + // SensitiveStatements.Session() in a comment is not a call + public void Plain(string s) + { + var braces = ""{ not a block }""; /* SensitiveStatements.Text(s) */ + } + + internal async Task Hooked(string s) + { + var session = new SensitiveStatements.Session(); + Use(session.Text(s)); + } + } +}"; + Assert.Single(BodiesOf(Source, "W", "Plain")); + Assert.False(HasSessionCall(BodiesOf(Source, "W", "Plain")[0])); + Assert.True(HasSessionCall(BodiesOf(Source, "W", "Hooked")[0])); + Assert.Empty(BodiesOf(Source, "W", "Gone")); + Assert.Empty(BodiesOf(Source, "Other", "Hooked")); + + var one = new Watched("planted", "6.7 row 0", "R0", false, ("f.cs", "W", "Plain")); + Assert.Single(WatchedProblems(new[] { one }, _ => Source)); + Assert.Empty(WatchedProblems(new[] { one with { Sites = new[] { ("f.cs", "W", "Hooked") } } }, _ => Source)); + Assert.Single(WatchedProblems(new[] { one with { Pending = true, Sites = new[] { ("f.cs", "W", "Gone") } } }, _ => Source)); + Assert.Single(WatchedProblems(new[] { one with { Pending = true } }, _ => null)); + } + + [Fact] + public void TheWatchedScan_SeesACallOnAPrivateFieldOfAnotherType_ButNotOnALocalOrAnUnrelatedMethod() + { + var target = ("SlowReadLog", "Offer"); + const string declares = "private readonly SlowReadLog _slowReads = new();\nprivate SlowReadLog? _optional;"; + Assert.True(CallsSite("_slowReads.Offer(a, b);", "McpToolLatencyFilter", target, declares)); + Assert.True(CallsSite("_optional.Offer(a);", "McpToolLatencyFilter", target, declares)); + Assert.True(CallsSite("SlowReadLog.Offer(a);", "McpToolLatencyFilter", target, "")); + Assert.False(CallsSite("local.Offer(a);", "McpToolLatencyFilter", target, declares)); + Assert.False(CallsSite("_slowReads.ShouldRecord(a);", "McpToolLatencyFilter", target, declares)); + // A private field of any OTHER type is not the site (#5367 review): the queue is not the slow-read log. + Assert.False(CallsSite("_queue.Offer(a);", "McpToolLatencyFilter", target, "private readonly Queue _queue = new();")); + Assert.False(CallsSite("_queue.Offer(a);", "McpToolLatencyFilter", target, declares)); + Assert.False(CallsSite("Offer(a);", "McpToolLatencyFilter", target, declares)); + } + + [Fact] + public void TheWatchedScan_RecognisesTheSessionAWriterReceives_AndAWriterThatNeverJudgesStillFails() + { + const string Source = @" +namespace N +{ + internal static class R + { + internal static (int, string) Judges(SensitiveStatements.Session scrub, string xml, int attempts) + { + if (!scrub.TryXml(xml, out var judged)) + { + return (0, null); + } + + return (1, judged); + } + + internal static void ReceivesButNeverJudges(SensitiveStatements.Session scrub, string xml) + { + Use(xml); + } + + internal static void JudgesThroughSomethingElse(SensitiveStatements.Session scrub, Other other, string xml) + { + other.Xml(xml); + } + + internal static void JudgesThroughAParameterThatIsNotASession(Other scrub, string xml) + { + scrub.Xml(xml); + } + + internal static void ACommentIsNotAJudgement(SensitiveStatements.Session scrub, string xml) + { + // scrub.Xml(xml) + var s = ""scrub.Xml(x)""; + } + } +}"; + Watched Site(string method) => new("planted", "6.7 row 0", "R0", false, ("f.cs", "R", method)); + + Assert.Empty(WatchedProblems(new[] { Site("Judges") }, _ => Source)); + Assert.Single(WatchedProblems(new[] { Site("ReceivesButNeverJudges") }, _ => Source)); + Assert.Single(WatchedProblems(new[] { Site("JudgesThroughSomethingElse") }, _ => Source)); + Assert.Single(WatchedProblems(new[] { Site("JudgesThroughAParameterThatIsNotASession") }, _ => Source)); + Assert.Single(WatchedProblems(new[] { Site("ACommentIsNotAJudgement") }, _ => Source)); + } + + [Fact] + public void TheWatchedScan_JudgesEveryMethodOfAWriter_NotJustTheOnesThatFilter() + { + // Review of #5367 (A-L2): the scan passed a writer when any one of its methods filtered, so a method added + // beside a hooked one (a second fetch that returns plan text) went unseen. + const string Source = @" +namespace N +{ + internal static class Sweep + { + internal static async Task<(int, string)> Fetch(SensitiveStatements.Session scrub, string xml) + { + return Judge(scrub, xml); + } + + internal static async Task<(int, string)> FetchAnother(SensitiveStatements.Session scrub, string xml) + { + return (1, xml); + } + + internal static (int, string) Judge(SensitiveStatements.Session scrub, string xml) + { + scrub.TryXml(xml, out var judged); + return (1, judged); + } + + internal static void Mentions(string xml) + { + // Judge(scrub, xml) in a comment is not a call + Use(xml); + } + } + + internal static class Runner + { + internal static void Hands(string xml) + { + Sweep.Judge(null, xml); + } + + internal static void Skips(string xml) + { + Other.Judge(null, xml); + } + } +}"; + Watched Writer(params (string Type, string Method)[] sites) => + new("planted", "6.7 row 0", "R0", false, sites.Select(s => ("f.cs", s.Type, s.Method)).ToArray()); + + // A method that hands the text on to a hooked sibling filters through it (the sweep's fetch, the runner's fetches). + Assert.Empty(WatchedProblems(new[] { Writer(("Sweep", "Fetch"), ("Sweep", "Judge")) }, _ => Source)); + Assert.Empty(WatchedProblems(new[] { Writer(("Runner", "Hands"), ("Sweep", "Judge")) }, _ => Source)); + + // RED on the any-method rule: a second method that does not filter fails although a sibling is hooked. + var unfiltered = WatchedProblems(new[] { Writer(("Sweep", "Fetch"), ("Sweep", "Judge"), ("Sweep", "FetchAnother")) }, _ => Source); + Assert.Single(unfiltered); + Assert.Contains("Sweep.FetchAnother", unfiltered[0], StringComparison.Ordinal); + + // A comment naming a hooked sibling, or a call to another type's method of the same name, is not delegation. + Assert.Single(WatchedProblems(new[] { Writer(("Sweep", "Mentions"), ("Sweep", "Judge")) }, _ => Source)); + Assert.Single(WatchedProblems(new[] { Writer(("Runner", "Skips"), ("Sweep", "Judge")) }, _ => Source)); + + // A pending writer is not judged, as before. + Assert.Empty(WatchedProblems(new[] { Writer(("Sweep", "FetchAnother")) with { Pending = true } }, _ => Source)); + } + + [Fact] + public void TheOversizedSweepWatchedWriter_IsHookedThroughTheSessionItReceives() + { + var source = ReadSiteFile("Darling/PerformanceMonitor.Darling.Service/OversizedPlanBacklogSweep.cs"); + Assert.NotNull(source); + + var judge = DeclarationsOf(source!, "OversizedPlanBacklogSweep", "JudgeFetchedPlan"); + Assert.Single(judge); + Assert.False(HasSessionCall(judge[0].Body), "the judging step makes no session of its own"); + Assert.True(JudgesThroughAReceivedSession(judge[0])); + } + + // ── the release gate ── + + private static string[] PendingEntries() => + Listed.Where(e => e.Value.Kind == Kind.Hooked && e.Value.Pending) + .Select(e => "column " + e.Key + " (" + e.Value.Lane + ")") + .Concat(WatchedWriters.Where(w => w.Pending).Select(w => "writer " + w.Name + " (" + w.Lane + ")")) + .OrderBy(k => k, StringComparer.Ordinal) + .ToArray(); + + /// + /// The gate test's name, bound with nameof so a rename moves it. The release workflow's gate step filters on this + /// name, and ReleaseStatementGateWorkflowTests reads this constant, so a rename that leaves the workflow behind + /// fails a pin instead of leaving the gate matching zero tests (#5320, review of #5367). + /// + public const string ReleaseGateTestName = nameof(StatementCensus_PendingEntries_BlockARelease); + + /// + /// The statement-column ratchet (see the release-checklist skill). It fails while ANY statement-column census entry or + /// watched writer is pending, and its message is the pending count and the list. Nothing is pending now, so a pull + /// request that adds a statement column or writer without judging it fails here on its own, in every normal Darling + /// test run, instead of at the release cut (#5320, ruling 2026-10-06). The name stays because the release workflow's + /// gate step and ReleaseStatementGateWorkflowTests bind to it. That step still sets DARLING_RELEASE_CUT=1 + /// and runs this one test ahead of every publish step; the variable no longer changes what the test asserts. + /// + [Fact] + public void StatementCensus_PendingEntries_BlockARelease() + { + var pending = PendingEntries(); + var summary = "Statement-column census pending entries: " + pending.Length + + (pending.Length == 0 ? string.Empty : " -> " + string.Join("; ", pending)); + + Console.WriteLine(summary); + + Assert.True( + pending.Length == 0, + "Every statement-column census entry and watched writer must be hooked; a pending one fails every test run, not only a release cut. " + summary); + } +} diff --git a/Darling/Darling.Tests/StatementCutSourceScanTests.cs b/Darling/Darling.Tests/StatementCutSourceScanTests.cs new file mode 100644 index 0000000000..bd70422aca --- /dev/null +++ b/Darling/Darling.Tests/StatementCutSourceScanTests.cs @@ -0,0 +1,875 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Text.RegularExpressions; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5320: statement text is judged WHOLE, then cut (McpHelpers.TruncateStatement / StatementPreview / +/// AnalysisStatementText.Preview). Never cut, then judge: a cut BEFORE the judge lets a value early in a batch +/// pass while the text that names it sits past the cut. This scan fails when either shape appears in the Lite MCP, +/// service and analysis code or in the Darling service and analysis code: +/// +/// C#: Truncate(x, n), x.Substring(...), x[..n], CutCodePoints(x, n), RawCut(x) or +/// lines.Take(n) (lines split out of a graph) where x is named like statement text (see +/// ). A local that was assigned from a judge call in the same block, before the cut, is +/// judged text and its cut is the sanctioned order (). +/// SQL in a C# string literal: LEFT(, SUBSTRING( or SUBSTR( over such a column, including a +/// column written as an interpolation hole (LEFT({col}, n)) whose name, or the constant it names, is statement text. +/// C# (#5361 round 2): x.AsSpan(a, b) (also inside string.Concat), x.AsSpan()[..n], +/// x.Slice(...), x.Remove(n) and a chained x.Split(...).Take(n). SQL: a DuckDB slice +/// (text[1:n], array_slice) and a PostgreSQL ::varchar(n) or CAST(... AS varchar(n)). +/// +/// The scanner is pinned with planted sources; the real trees are held to a short, named list of exceptions. +/// Known blind spots (a regex over source text cannot read these reliably; they are named here on purpose, +/// not left silent, so a reviewer of a new statement-text cut checks them by eye): +/// (1) an alias local: var t = row.QueryText; t[..300] cuts under a name that is not statement-like, and the scan +/// reads names, not data flow. (2) a ternary, ?? arm or later re-assignment around a judged local: +/// var s = flag ? SensitiveStatements.Text(a) : raw; s[..300] is excused because the right side holds a judge call, +/// and so is a raw re-assignment after a judge. Also not seen: SQL built by concatenation ("LEFT(" + col + ", 500)") +/// and a JSON-generic local. +/// +[Trait("Reads", "Lite")] +public sealed class StatementCutSourceScanTests +{ + /// Folders scanned, repo-root relative. Lite MCP, services and analysis; the shared alerting, notification and analysis projects; Darling service and analysis. + private static readonly string[] ScannedTrees = + { + "Lite/Mcp", + "Lite/Services", + "Lite/Analysis", + "PerformanceMonitor.Alerting", + "PerformanceMonitor.Notifications", + "PerformanceMonitor.Analysis", + "Darling/PerformanceMonitor.Darling.Service", + "Darling/PerformanceMonitor.Darling.Analysis", + }; + + /// A name that says the value is statement text. Lower-cased identifier or column, substring match. + private static readonly Regex StatementName = new( + @"sql|query_?text|statement|inputbuf|graph|stmt|normalized|preview_?text|raw_?text|fetched_?text|text_?data" + + @"|blocked_?process|current_?query|root_?query|query_?sample|sample_?text|(?:blocked|blocking|victim)_?query|\bquery\b|_query\b", + RegexOptions.Compiled | RegexOptions.IgnoreCase); + + /// Truncate(x, n) and the alert builders' TruncateText(x, n) (#5320: it cuts the alert + /// context's statement text). TruncateStatement is the judge-then-cut helper and is not matched. + private static readonly Regex TruncateCall = new(@"\bTruncate(?:Text)?\s*\(", RegexOptions.Compiled); + private static readonly Regex SubstringCall = new(@"([\w][\w\.\?!]*)\s*\.\s*Substring\s*\(", RegexOptions.Compiled); + private static readonly Regex RangeCut = new(@"([\w][\w\.\?!]*)\s*\[[^\]\[]*\.\.[^\]\[]*\]", RegexOptions.Compiled); + private static readonly Regex SqlCut = new(@"\b(left|substring|substr|array_slice|list_slice)\s*\(", RegexOptions.Compiled | RegexOptions.IgnoreCase); + + /// The cut helpers that are not Truncate: the code-point cut and the pileup readers' RawCut. + private static readonly Regex CutHelperCall = new(@"\b(CutCodePoints|RawCut)\s*\(", RegexOptions.Compiled); + + /// x.Take(n): the cut of a line list. Only a receiver that is statement text, or lines split out of it, counts. + private static readonly Regex TakeCall = new(@"([\w][\w\.\?!]*)\s*\.\s*Take\s*\(", RegexOptions.Compiled); + + /// + /// x.AsSpan(a, b) / x.AsSpan(n) (a sliced span, also inside string.Concat), and the whole-text + /// x.AsSpan() followed by a range index or .Slice(...). #5361 F2. + /// + private static readonly Regex AsSpanCall = new(@"([\w][\w\.\?!]*)\s*\.\s*AsSpan\s*\(", RegexOptions.Compiled); + + /// x.Remove(n): drops the tail (or a middle run) of a string. + private static readonly Regex RemoveCall = new(@"([\w][\w\.\?!]*)\s*\.\s*Remove\s*\(", RegexOptions.Compiled); + + /// x.Slice(a, b) on a span or memory of statement text. + private static readonly Regex SliceCall = new(@"([\w][\w\.\?!]*)\s*\.\s*Slice\s*\(", RegexOptions.Compiled); + + /// The .Take( of any chain, including one after a call (x.Split('\n').Take(5)). + private static readonly Regex AnyTakeCall = new(@"\.\s*Take\s*\(", RegexOptions.Compiled); + + /// DuckDB slice in SQL: col[1:n], col[:n]. + private static readonly Regex SqlSlice = new(@"([\w][\w\.]*)\s*\[[^\]\[]*:[^\]\[]*\]", RegexOptions.Compiled); + + /// The fixed-length character types that truncate in PostgreSQL: varchar(n), char(n), character varying(n). + private const string LengthLimitedType = @"(?:character\s+varying|character|varchar|bpchar|char)\s*\(\s*\d+\s*\)"; + + /// PostgreSQL operand::varchar(n). The operand is a name, or a parenthesised expression read by . + private static readonly Regex SqlCastOperator = new(@"::\s*" + LengthLimitedType, RegexOptions.Compiled | RegexOptions.IgnoreCase); + + /// PostgreSQL / DuckDB CAST(operand AS varchar(n)). + private static readonly Regex SqlCastCall = new(@"\bcast\s*\(", RegexOptions.Compiled | RegexOptions.IgnoreCase); + + private static readonly Regex SqlCastTail = new(@"\bas\s+" + LengthLimitedType + @"\s*$", RegexOptions.Compiled | RegexOptions.IgnoreCase); + + /// A call that judges statement text whole (the judge itself, or a helper that judges then cuts). + private static readonly Regex JudgeCall = new( + @"\b(?:SensitiveStatements\s*\.\s*(?:Text|Xml|Json)|AnalysisStatementText\s*\.\s*Preview|TruncateStatement|StatementPreview|FilterStoredPlan)\s*\(", + RegexOptions.Compiled); + + private static readonly Regex Assignment = new(@"\b(\w+)\s*=(?![=>])", RegexOptions.Compiled); + + /// A string local, field or constant initialised from one literal: lets LEFT({col}, n) be read as the column col names. + private static readonly Regex StringConstant = new( + @"\b(?:string|var)\s+(\w+)\s*=\s*(\$?@?""[^;]*);", RegexOptions.Compiled); + + private static readonly Regex Identifier = new(@"[A-Za-z_]\w*", RegexOptions.Compiled); + + internal sealed record Hit(string Kind, int Line, string Expression); + + /// True for an exception the same wave retires: the count is then a ceiling, so it + /// passes at 0 as well and cannot go stale when the owning change lands. The default is exact. + private sealed record Allowed(string File, string Expression, int Count, string Reason, bool Ceiling = false); + + /// + /// Cuts of statement-named values that are allowed to stay, each by file and expression. The count is exact, so + /// an exception that stops being needed (or doubles) fails here and is removed or re-argued on purpose, unless it + /// is marked a ceiling. + /// + private static readonly Allowed[] Exceptions = + { + new("Lite/Services/LocalDataService.FinOps.Recommendations.cs", "left(@sql,len(@sql)-10)", 1, + "trims the trailing UNION ALL off the dynamic script this method is building; the script is never shown as a statement"), + new("Lite/Services/LocalDataService.FinOps.Workload.cs", "left(query_text,200)", 1, + "QueryPreview: the desktop grid cell; the same row carries FullQueryText whole and no Lite MCP tool reads QueryPreview"), + new("Lite/Services/LocalDataService.FinOps.Workload.cs", "left(qs2.query_text,200)", 1, + "SampleQueryText: the desktop grid cell; the same row carries FullQueryText whole and no Lite MCP tool reads SampleQueryText"), + new("Lite/Services/LocalDataService.QueryStats.cs", "left(query_text,120)", 1, + "the desktop heatmap hover (GetQueryHeatmapAsync); the get_query_heatmap tool reads LocalDataService.QueryHeatmap.cs, which fetches whole text and cuts after the filter"), + new("Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs", "row.victimstatement!", 1, + "BuildPgDeadlockIncident cuts a PostgreSQL victim statement the collector already judged whole: PgDeadlockLogParser redacts the whole report (PgLogTextRedactor.RedactDetail) before it takes the victim's statement out of it, and PgDeadlockLogParser.NormalizeStatement re-reads a stored one. Nothing here sees raw text"), + new("Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs", "row.rootquery!", 1, + "BuildPgBlockingIncident cuts a PostgreSQL root query the target already judged whole: PgBlockingCollector wraps blocker.query in PgSensitiveStatementFilter.SqlPredicate on the target before the row leaves it (PgSensitiveStatementFilterTests), and PgStatementTextScrub covers rows stored before that. Nothing here sees raw text"), + new("PerformanceMonitor.Notifications/AnalysisNotificationService.cs", "element.getrawtext()", 1, + "ScalarText shortens a nested drill-down object or array for the finding notification's field. The drill-down JSON is what DrillDownCollector stores: its statement values are judge-then-cut there (AnalysisStatementText.Preview), so this cuts judged values and never sees raw statement text"), + new("PerformanceMonitor.Notifications/WebhookAlertService.cs", "paragraph", 2, + "SplitProseLabel splits a synthesized advice paragraph on its first ': ' into label and value; both halves are kept, nothing is cut off (the name matches 'graph')"), + new("Darling/PerformanceMonitor.Darling.Service/Mcp/TopRanking.cs", "sql", 2, + "ReplaceOnce splices a ranking clause into the top-N query's own SQL script with string.Concat over AsSpan; the script is our text, not a statement read from a monitored server"), + new("PerformanceMonitor.Analysis/SameStatementPileupDetector.cs", "normalized", 1, + "Preview normalizes whitespace and cuts the finding's printed statement; its only caller passes SnapshotRow.PreviewText, which the reader judged whole before it cut (a null PreviewText prints an empty string, #5361 F1). Nothing here sees raw text"), + new("Lite/Analysis/BaselineProvider.cs", "eventbaselinesql", 3, + "slices the event-baseline SQL script (our own query text) around its events CTE to swap one column; it is a script, not a statement read from a monitored server"), + new("Lite/Analysis/PileupSnapshotReader.cs", "rawcut(rawtext)", 1, + "identity only: the cut text is the same-statement pileup's grouping key and noise test, built from the raw read and never printed; the shown text is cut after the judge"), + new("Darling/PerformanceMonitor.Darling.Analysis/PgPileupSnapshotReader.cs", "cutcodepoints(rawtext", 1, + "identity only: the cut text is the same-statement pileup's grouping key and noise test (a null-hash surrogate), built from the raw read and never printed; the shown text is cut after the judge"), + new("Darling/PerformanceMonitor.Darling.Analysis/PgTargetDrillDownCollector.Deadlocks.cs", + "left(encode(sha256(convert_to(latest_graph_text,'utf8')),'hex'),32)", 1, + "an identity hash of the raw graph, compared with latest_hash and never shown"), + new("Darling/PerformanceMonitor.Darling.Analysis/PgTargetDrillDownCollector.Deadlocks.cs", "lines.take", 1, + "BoundGraphText bounds a graph its caller already judged whole (graphNormalized = SensitiveStatements.Text(...)); it cuts whole lines of judged text and never sees raw"), + }; + + // ------------------------------------------------------------------ the scanner + + /// Every cut of statement-named text in one C# source: the C# shapes and the SQL-in-literal shape. + internal static List Scan(string source) + { + var hits = new List(); + + var code = CSharpSourceWalker.StripCommentsAndStrings(source); + var judged = JudgedLocals(code); + + foreach (Match m in TruncateCall.Matches(code)) + { + var arg = FirstArgument(code, m.Index + m.Length - 1); + if (StatementName.IsMatch(arg) && !IsJudgedLocal(judged, arg, m.Index)) + hits.Add(new Hit("C# Truncate", LineOf(code, m.Index), Normalize(arg))); + } + + foreach (Match m in CutHelperCall.Matches(code)) + { + var arg = FirstArgument(code, m.Index + m.Length - 1); + if (StatementName.IsMatch(arg) && !IsJudgedLocal(judged, arg, m.Index)) + hits.Add(new Hit("C# " + m.Groups[1].Value, LineOf(code, m.Index), Normalize(m.Groups[1].Value + "(" + arg + ")"))); + } + + foreach (Match m in SubstringCall.Matches(code)) + { + if (StatementName.IsMatch(m.Groups[1].Value) && !IsJudgedLocal(judged, m.Groups[1].Value, m.Index)) + hits.Add(new Hit("C# Substring", LineOf(code, m.Index), Normalize(m.Groups[1].Value))); + } + + foreach (Match m in RangeCut.Matches(code)) + { + if (StatementName.IsMatch(m.Groups[1].Value) && !IsJudgedLocal(judged, m.Groups[1].Value, m.Index)) + hits.Add(new Hit("C# range", LineOf(code, m.Index), Normalize(m.Groups[1].Value))); + } + + foreach (Match m in TakeCall.Matches(code)) + { + var receiver = m.Groups[1].Value; + if (!TakesStatementLines(code, receiver, m.Index) || IsJudgedLocal(judged, receiver, m.Index)) continue; + hits.Add(new Hit("C# Take", LineOf(code, m.Index), Normalize(receiver + ".Take"))); + } + + foreach (Match m in AsSpanCall.Matches(code)) + { + var receiver = m.Groups[1].Value; + if (!StatementName.IsMatch(receiver) || IsJudgedLocal(judged, receiver, m.Index)) continue; + + var open = m.Index + m.Length - 1; + var args = CallBody(code, open); + if (args.Trim().Length == 0) + { + /* x.AsSpan() on its own is not a cut; x.AsSpan()[..n] and x.AsSpan().Slice(a, b) are. */ + var after = open + args.Length + 2; + while (after < code.Length && char.IsWhiteSpace(code[after])) after++; + var ranged = after < code.Length && code[after] == '[' + && BracketBody(code, after).Contains("..", StringComparison.Ordinal); + var sliced = Regex.IsMatch(code.Substring(after, Math.Min(code.Length - after, 12)), @"^\.\s*Slice\s*\("); + if (!ranged && !sliced) continue; + } + + hits.Add(new Hit("C# AsSpan", LineOf(code, m.Index), Normalize(receiver))); + } + + foreach (Match m in SliceCall.Matches(code)) + { + var receiver = m.Groups[1].Value; + if (StatementName.IsMatch(receiver) && !IsJudgedLocal(judged, receiver, m.Index)) + hits.Add(new Hit("C# Slice", LineOf(code, m.Index), Normalize(receiver))); + } + + foreach (Match m in RemoveCall.Matches(code)) + { + var receiver = m.Groups[1].Value; + if (StatementName.IsMatch(receiver) && !IsJudgedLocal(judged, receiver, m.Index)) + hits.Add(new Hit("C# Remove", LineOf(code, m.Index), Normalize(receiver))); + } + + foreach (Match m in AnyTakeCall.Matches(code)) + { + /* A Take after a call: x.Split('\n').Take(5). The chain's own names count, its arguments do not. */ + var before = m.Index; + while (before > 0 && char.IsWhiteSpace(code[before - 1])) before--; + if (before == 0 || code[before - 1] != ')') continue; + + var chain = ChainBefore(code, before); + var root = Identifier.Match(chain); + if (!StatementName.IsMatch(chain) || root.Success && IsJudgedLocal(judged, root.Value, m.Index)) continue; + hits.Add(new Hit("C# Take", LineOf(code, m.Index), Normalize(chain + ".Take"))); + } + + var constants = StringConstant.Matches(source) + .GroupBy(c => c.Groups[1].Value, StringComparer.Ordinal) + .ToDictionary(g => g.Key, g => g.First().Groups[2].Value, StringComparer.Ordinal); + + foreach (var (start, body) in CSharpSourceWalker.StringLiteralBodies(source)) + { + foreach (Match m in SqlCut.Matches(body)) + { + var open = m.Index + m.Length - 1; + var arg = FirstArgument(body, open); + + /* An interpolation hole is blank in the body. Read the same call out of the source (the body keeps the + source's offsets) so LEFT({col}, n) is judged by the hole's own name, or by the constant it names. */ + var inSource = start + open < source.Length && source[start + open] == '('; + var rawArg = inSource ? FirstArgument(source, start + open) : arg; + var viaHole = inSource && !StatementName.IsMatch(arg) && SameButForHoles(arg, rawArg) + && HoleNamesStatementText(rawArg, constants); + if (!StatementName.IsMatch(arg) && !viaHole) continue; + + var whole = Normalize(m.Groups[1].Value + "(" + (viaHole ? CallBody(source, start + open) : CallBody(body, open)) + ")"); + hits.Add(new Hit("SQL " + m.Groups[1].Value.ToLowerInvariant(), LineOf(source, start + m.Index), whole)); + } + } + + /* #5361 F2: the SQL cuts that are not LEFT/SUBSTRING: a DuckDB slice and a PostgreSQL fixed-length cast. */ + foreach (var (start, body) in CSharpSourceWalker.StringLiteralBodies(source)) + { + foreach (Match m in SqlSlice.Matches(body)) + { + if (StatementName.IsMatch(m.Groups[1].Value)) + hits.Add(new Hit("SQL slice", LineOf(source, start + m.Index), Normalize(m.Value))); + } + + foreach (Match m in SqlCastOperator.Matches(body)) + { + var operand = OperandBefore(body, m.Index); + if (StatementName.IsMatch(operand)) + hits.Add(new Hit("SQL cast", LineOf(source, start + m.Index), Normalize(operand + m.Value))); + } + + foreach (Match m in SqlCastCall.Matches(body)) + { + var inner = CallBody(body, m.Index + m.Length - 1); + var tail = SqlCastTail.Match(inner); + if (tail.Success && StatementName.IsMatch(inner.Substring(0, tail.Index))) + hits.Add(new Hit("SQL cast", LineOf(source, start + m.Index), Normalize("cast(" + inner + ")"))); + } + } + + return hits; + } + + /// The operand of a :: at : the name before it, or the parenthesised expression. + private static string OperandBefore(string text, int at) + { + var end = at; + while (end > 0 && char.IsWhiteSpace(text[end - 1])) end--; + if (end == 0) return ""; + + if (text[end - 1] == ')') + { + var depth = 0; + for (var i = end - 1; i >= 0; i--) + { + if (text[i] == ')') depth++; + else if (text[i] == '(' && --depth == 0) return text.Substring(i, end - i); + } + + return text.Substring(0, end); + } + + var begin = end; + while (begin > 0 && (char.IsLetterOrDigit(text[begin - 1]) || text[begin - 1] is '_' or '.')) begin--; + return text.Substring(begin, end - begin); + } + + /// The call chain that ends at (just after a closing paren), with every call's arguments blanked. + private static string ChainBefore(string code, int end) + { + var sb = new StringBuilder(); + var i = end; + while (i > 0) + { + var c = code[i - 1]; + if (c == ')') + { + var depth = 0; + var j = i - 1; + for (; j >= 0; j--) + { + if (code[j] == ')') depth++; + else if (code[j] == '(' && --depth == 0) break; + } + + if (j < 0) break; + sb.Insert(0, "()"); + i = j; + } + else if (char.IsLetterOrDigit(c) || c is '_' or '.' or '?' or '!') + { + sb.Insert(0, c); + i--; + } + else if (char.IsWhiteSpace(c)) + { + i--; + } + else break; + } + + return sb.ToString(); + } + + /// Everything between the '[' at and its closing bracket. + private static string BracketBody(string text, int open) + { + var depth = 0; + for (var i = open; i < text.Length; i++) + { + if (text[i] == '[') depth++; + else if (text[i] == ']' && --depth == 0) return text.Substring(open + 1, i - open - 1); + } + + return text.Substring(open + 1); + } + + /// + /// True when is with its blanked interpolation holes filled in: the same + /// length, and every non-blank character equal. A raw read that runs past the literal's end (a C# concatenation) is not. + /// + private static bool SameButForHoles(string body, string raw) + { + if (body.Length != raw.Length) return false; + for (var i = 0; i < body.Length; i++) + { + if (body[i] != ' ' && body[i] != raw[i]) return false; + } + + return true; + } + + /// True when an interpolation hole in a SQL column list names statement text, directly or through a string constant. + private static bool HoleNamesStatementText(string rawArg, Dictionary constants) + { + if (StatementName.IsMatch(rawArg)) return true; + + foreach (Match id in Identifier.Matches(rawArg)) + { + if (constants.TryGetValue(id.Value, out var value) && StatementName.IsMatch(value)) return true; + } + + return false; + } + + /// A local assigned from a judge call, and the block it is in: the span where its cut is judge-then-cut. + private sealed record JudgedLocal(string Name, int From, int To); + + /// + /// Every name = ... judge(...) in the source, scoped to the block the assignment sits in and to the text after + /// it. A judged local that is cut in that span is the sanctioned order; the same name cut in another method, or before + /// the assignment, is not excused. + /// + private static List JudgedLocals(string code) + { + var locals = new List(); + foreach (Match m in Assignment.Matches(code)) + { + var from = m.Index + m.Length; + var statement = code.Substring(from, StatementEnd(code, from) - from); + if (JudgeCall.IsMatch(statement)) locals.Add(new JudgedLocal(m.Groups[1].Value, from, BlockEnd(code, from))); + } + + return locals; + } + + private static bool IsJudgedLocal(List judged, string operand, int at) + { + var name = operand.Trim().TrimEnd('!'); + return Identifier.Match(name) is { Success: true } id && id.Value == name + && judged.Any(j => j.Name == name && j.From <= at && at < j.To); + } + + /// The end of the statement or initialiser that starts at : a top-level ; or ,, or the bracket that closes the enclosing one. + private static int StatementEnd(string code, int from) + { + var depth = 0; + for (var i = from; i < code.Length; i++) + { + var c = code[i]; + if (c is '(' or '[' or '{') depth++; + else if (c is ')' or ']' or '}') + { + if (--depth < 0) return i; + } + else if (c is ';' or ',' && depth == 0) return i; + } + + return code.Length; + } + + /// The close of the innermost block that holds (the end of the text when braces never balance). + private static int BlockEnd(string code, int from) + { + var depth = 0; + for (var i = from; i < code.Length; i++) + { + if (code[i] == '{') depth++; + else if (code[i] == '}' && --depth < 0) return i; + } + + return code.Length; + } + + /// + /// True for x.Take(n) when x is named like statement text, or is a line list declared from statement text + /// (var lines = graphText.Split('\n')): a cut of a graph's lines is a cut of statement text. + /// + private static bool TakesStatementLines(string code, string receiver, int at) + { + if (StatementName.IsMatch(receiver)) return true; + if (!receiver.Contains("line", StringComparison.OrdinalIgnoreCase) || !Identifier.IsMatch(receiver)) return false; + + var declaration = new Regex(@"\b" + Regex.Escape(receiver) + @"\s*=(?![=>])\s*([^;]*);"); + return declaration.Matches(code.Substring(0, at)).Any(d => StatementName.IsMatch(d.Groups[1].Value)); + } + + private static string Normalize(string s) => + Regex.Replace(s, @"\s+", "").ToLowerInvariant(); + + private static int LineOf(string text, int index) + { + var line = 1; + for (var i = 0; i < index && i < text.Length; i++) + { + if (text[i] == '\n') line++; + } + + return line; + } + + /// The first argument of the call whose '(' is at : up to a top-level comma or the closing paren. + private static string FirstArgument(string text, int open) + { + var depth = 0; + for (var i = open; i < text.Length; i++) + { + var c = text[i]; + if (c == '(') depth++; + else if (c == ')') + { + depth--; + if (depth == 0) return text.Substring(open + 1, i - open - 1); + } + else if (c == ',' && depth == 1) return text.Substring(open + 1, i - open - 1); + } + + return text.Substring(open + 1); + } + + /// Everything between the '(' at and its closing paren. + private static string CallBody(string text, int open) + { + var depth = 0; + for (var i = open; i < text.Length; i++) + { + if (text[i] == '(') depth++; + else if (text[i] == ')' && --depth == 0) return text.Substring(open + 1, i - open - 1); + } + + return text.Substring(open + 1); + } + + private static IEnumerable ScannedFiles() => + ScannedTrees + .SelectMany(t => Directory.EnumerateFiles(Path.Combine(RepoFile.Root, t), "*.cs", SearchOption.AllDirectories)) + .Select(f => Path.GetRelativePath(RepoFile.Root, f).Replace('\\', '/')) + .Where(f => !f.Contains("/obj/", StringComparison.Ordinal) && !f.Contains("/bin/", StringComparison.Ordinal)) + .OrderBy(f => f, StringComparer.Ordinal); + + /// The problems left after the named exceptions are applied to what the scan found, per file. + private static List Reconcile(Dictionary> perFile, IEnumerable exceptions) + { + var allExceptions = exceptions.ToList(); + var problems = new List(); + + foreach (var (file, hits) in perFile.OrderBy(p => p.Key, StringComparer.Ordinal)) + { + var remaining = new List(hits); + + foreach (var allowed in allExceptions.Where(e => e.File == file)) + { + var matched = remaining.Where(h => h.Expression.Contains(allowed.Expression, StringComparison.Ordinal)).ToList(); + var wrong = allowed.Ceiling ? matched.Count > allowed.Count : matched.Count != allowed.Count; + if (wrong) + problems.Add($"{file}: exception '{allowed.Expression}' expected {(allowed.Ceiling ? "at most " : "")}{allowed.Count} site(s), found {matched.Count}"); + foreach (var h in matched) remaining.Remove(h); + } + + foreach (var h in remaining) + problems.Add($"{file}:{h.Line}: {h.Kind} cuts statement text before the filter judges it: {h.Expression}"); + } + + foreach (var allowed in allExceptions.Where(e => !e.Ceiling)) + { + if (!perFile.ContainsKey(allowed.File)) + problems.Add($"{allowed.File}: exception '{allowed.Expression}' expected {allowed.Count} site(s), found 0 (remove it)"); + } + + return problems; + } + + // ------------------------------------------------------------------ the real trees + + [Fact] + public void NoStatementTextIsCutBeforeItIsJudged_ExceptTheNamedSites() + { + var files = ScannedFiles().ToList(); + Assert.True(files.Count > 200, "the scan walked the trees: " + files.Count); + Assert.Contains(files, f => f.StartsWith("Lite/Analysis/", StringComparison.Ordinal)); + + var perFile = new Dictionary>(); + foreach (var file in files) + { + var hits = Scan(RepoFile.ReadRepoFile(file.Split('/'))); + if (hits.Count > 0) perFile[file] = hits; + } + + var problems = Reconcile(perFile, Exceptions); + + Assert.True(problems.Count == 0, string.Join(Environment.NewLine, problems)); + } + + // ------------------------------------------------------------------ the scanner itself, on planted sources + + [Theory] + [InlineData("var p = McpHelpers.Truncate(row.QueryText, 400);", "C# Truncate")] + [InlineData("var p = McpHelpers.Truncate(sqlText, 400);", "C# Truncate")] + [InlineData("var p = AlertContextBuilders.TruncateText(row.QueryText);", "C# Truncate")] + [InlineData("var p = McpHelpers.Truncate(query, 400);", "C# Truncate")] + [InlineData("var p = McpHelpers.Truncate(row.blocking_query, 400);", "C# Truncate")] + [InlineData("var p = TruncateText(g.BlockedQuery, 300);", "C# Truncate")] + [InlineData("var p = TruncateText(g.QueryText, 300);", "C# Truncate")] + [InlineData("var p = TruncateText(blockedQuery, 300);", "C# Truncate")] + [InlineData("var p = AlertContextBuilders.TruncateText(statement, 80);", "C# Truncate")] + [InlineData("var p = McpHelpers.Truncate(r.victim_sql_text ?? \"\", 400);", "C# Truncate")] + [InlineData("var p = McpHelpers.Truncate(inputbuf, 400);", "C# Truncate")] + [InlineData("var p = McpHelpers.Truncate(deadlockGraph, 400);", "C# Truncate")] + [InlineData("var p = statement.Substring(0, 200);", "C# Substring")] + [InlineData("var p = row.Statement?.Substring(0, 200);", "C# Substring")] + [InlineData("var p = sql_text[..200];", "C# range")] + [InlineData("var p = batch.QueryText[0..200];", "C# range")] + public void TheScanner_CatchesACSharpCutOfStatementText(string source, string kind) + { + var hits = Scan("class C { void M() { " + source + " } }"); + + Assert.Contains(hits, h => h.Kind == kind); + } + + [Theory] + [InlineData("SELECT LEFT(query_text, 500) AS q FROM t")] + [InlineData("SELECT left(blocked_sql_text, 5) FROM t")] + [InlineData("SELECT SUBSTRING(r.query_text, 1, 300) FROM t r")] + [InlineData("SELECT substr(query_text, 1, 200) FROM t")] + [InlineData("SELECT LEFT(MAX(v.query_text), 500) FROM t v")] + [InlineData("SELECT LEFT(COALESCE(x.query_sql_text, l.query_text), 500) FROM t")] + [InlineData("SELECT LEFT(latest_graph_text, 4000) FROM t")] + [InlineData("SELECT LEFT(inputbuf, 4000) FROM t")] + [InlineData("SELECT LEFT(victim_statement, $5) FROM t")] + public void TheScanner_CatchesASqlCutOfStatementText(string sql) + { + var hits = Scan("class C { const string Q = @\"" + sql + "\"; }"); + + Assert.Contains(hits, h => h.Kind.StartsWith("SQL ", StringComparison.Ordinal)); + } + + [Theory] + [InlineData("var p = McpHelpers.Truncate(relationName, 63);")] + [InlineData("var p = McpHelpers.Truncate(serverName, 40);")] + [InlineData("var p = tableName.Substring(0, 10);")] + [InlineData("var p = schema_name[..8];")] + [InlineData("var p = McpHelpers.TruncateStatement(sqlText, 400);")] + [InlineData("var p = AlertContextBuilders.TruncateStatement(row.QueryText);")] + [InlineData("var p = AlertContextBuilders.TruncateText(j.Message, 300);")] + [InlineData("var p = McpHelpers.StatementPreview(row.QueryText, 400);")] + [InlineData("// McpHelpers.Truncate(sqlText, 400) is what this used to be")] + [InlineData("var note = \"McpHelpers.Truncate(sqlText, 400)\";")] + public void TheScanner_LeavesAnythingThatIsNotACutOfStatementText(string source) + { + var hits = Scan("class C { void M() { " + source + " } }"); + + Assert.Empty(hits); + } + + [Theory] + [InlineData("SELECT LEFT(relname, 63) FROM pg_class")] + [InlineData("SELECT LEFT(server_name, 20) FROM servers")] + [InlineData("SELECT LEFT(query_hash, 8) FROM t")] + [InlineData("SELECT a FROM t LEFT JOIN (SELECT query_text FROM u) x ON x.id = t.id")] + [InlineData("SELECT query_text FROM t")] + public void TheScanner_LeavesSqlThatDoesNotCutStatementText(string sql) + { + var hits = Scan("class C { const string Q = @\"" + sql + "\"; }"); + + Assert.Empty(hits); + } + + [Fact] + public void TheScanner_ReadsAnInterpolatedAndMultiLineLiteral_AndReportsTheLine() + { + var source = new StringBuilder() + .AppendLine("class C") + .AppendLine("{") + .AppendLine(" string Q(int n) => $@\"SELECT a,") + .AppendLine(" LEFT(query_text, {n}) AS q FROM t\";") + .AppendLine("}") + .ToString(); + + var hits = Scan(source); + + var hit = Assert.Single(hits); + Assert.Equal(4, hit.Line); + } + + // ------------------------------------------------------------------ the rules added in #5361, each with its planted RED + + private static List ScanBody(string body) => Scan("class C { void M() { " + body + " } }"); + + [Theory] + [InlineData("var p = AnalysisStatementText.CutCodePoints(rawText, 500);", "C# CutCodePoints")] + [InlineData("var p = RawCut(rawText);", "C# RawCut")] + [InlineData("var p = victimNormalized[..400];", "C# range")] + [InlineData("var p = McpHelpers.Truncate(row.PreviewText, 400);", "C# Truncate")] + [InlineData("var p = McpHelpers.Truncate(r.TextData, 400);", "C# Truncate")] + [InlineData("var p = row.RootQuery!.Substring(0, 200);", "C# Substring")] + [InlineData("var lines = deadlockGraph.Split(','); var p = string.Join(',', lines.Take(40));", "C# Take")] + [InlineData("var p = string.Join(',', graphLines.Take(40));", "C# Take")] + public void TheScanner_CatchesTheWiderStatementNamesAndCutShapes(string body, string kind) + { + Assert.Contains(ScanBody(body), h => h.Kind == kind); + } + + // ------------------------------------------------------------------ the shapes added in #5361 round 2 (F2), each with its planted RED + + [Theory] + [InlineData("var p = row.QueryText.AsSpan(0, 200).ToString();", "C# AsSpan")] + [InlineData("var p = sqlText.AsSpan(200).ToString();", "C# AsSpan")] + [InlineData("var p = string.Concat(sqlText.AsSpan(0, 200), \"...\");", "C# AsSpan")] + [InlineData("var p = sqlText.AsSpan()[..200].ToString();", "C# AsSpan")] + [InlineData("var p = sqlText.AsSpan().Slice(0, 200).ToString();", "C# AsSpan")] + [InlineData("var p = statementText.Slice(0, 200);", "C# Slice")] + [InlineData("var p = row.Statement.Remove(200);", "C# Remove")] + [InlineData("var p = string.Join('\\n', deadlockGraph.Split('\\n').Take(40));", "C# Take")] + [InlineData("var p = deadlockGraph.Split('\\n', StringSplitOptions.RemoveEmptyEntries).Take(40).ToList();", "C# Take")] + public void TheScanner_CatchesTheCutShapesAddedInRoundTwo(string body, string kind) + { + Assert.Contains(ScanBody(body), h => h.Kind == kind); + } + + [Theory] + [InlineData("var p = serverName.AsSpan(0, 3).ToString();")] + [InlineData("var n = int.Parse(sqlText.AsSpan());")] + [InlineData("var p = tableName.Remove(3);")] + [InlineData("var p = names.Split(',').Take(3).ToList();")] + [InlineData("var p = string.Concat(label.AsSpan(0, 3), \"x\");")] + [InlineData("var sqlText = SensitiveStatements.Text(raw); var p = string.Concat(sqlText.AsSpan(0, 5), \"x\");")] + [InlineData("var sqlText = SensitiveStatements.Text(raw); var p = sqlText.Remove(5);")] + [InlineData("var graph = SensitiveStatements.Xml(raw, 100); var p = graph.Split('\\n').Take(5).ToList();")] + public void TheScanner_LeavesTheRoundTwoShapesThatAreNotACutOfStatementText(string body) + { + Assert.Empty(ScanBody(body)); + } + + [Theory] + [InlineData("SELECT query_text[1:500] AS q FROM t")] + [InlineData("SELECT sql_text[:500] FROM t")] + [InlineData("SELECT r.blocked_sql_text[1:200] FROM t r")] + [InlineData("SELECT array_slice(query_text, 1, 500) FROM t")] + [InlineData("SELECT query_text::varchar(500) FROM t")] + [InlineData("SELECT r.query_text :: character varying(500) FROM t r")] + [InlineData("SELECT (COALESCE(a.query_text, b.query_text))::varchar(500) FROM t")] + [InlineData("SELECT query_text::char(80) FROM t")] + [InlineData("SELECT CAST(query_text AS varchar(500)) FROM t")] + [InlineData("SELECT CAST(COALESCE(x.statement, y.statement) AS VARCHAR(300)) FROM t")] + public void TheScanner_CatchesASqlSliceOrFixedLengthCastOfStatementText(string sql) + { + var hits = Scan("class C { const string Q = @\"" + sql + "\"; }"); + + Assert.Contains(hits, h => h.Kind.StartsWith("SQL ", StringComparison.Ordinal)); + } + + [Theory] + [InlineData("SELECT relname[1:63] FROM t")] + [InlineData("SELECT query_hash::varchar(16) FROM t")] + [InlineData("SELECT query_text::text FROM t")] + [InlineData("SELECT CAST(query_text AS text) FROM t")] + [InlineData("SELECT CAST(query_hash AS varchar(16)) FROM t")] + [InlineData("SELECT query_text::varchar FROM t")] + [InlineData("SELECT x[1:2] FROM t")] + public void TheScanner_LeavesASqlSliceOrCastThatIsNotAFixedLengthCutOfStatementText(string sql) + { + Assert.Empty(Scan("class C { const string Q = @\"" + sql + "\"; }")); + } + + [Fact] + public void TheScansDocComment_NamesItsKnownBlindSpots() + { + var doc = RepoFile.ReadRepoFile("Darling", "Darling.Tests", "StatementCutSourceScanTests.cs"); + Assert.Contains("Known blind spots", doc, StringComparison.Ordinal); + Assert.Contains("alias local", doc, StringComparison.Ordinal); + Assert.Contains("ternary", doc, StringComparison.Ordinal); + } + + [Theory] + [InlineData("var lines = packageNames.Split(','); var p = lines.Take(5);")] + [InlineData("var p = ordered.Take(limit);")] + [InlineData("var p = McpHelpers.Truncate(serverName, 40);")] + public void TheScanner_LeavesALineListThatIsNotStatementText(string body) + { + Assert.Empty(ScanBody(body)); + } + + [Fact] + public void TheScanner_CutThenJudge_IsCaught_AndJudgeThenCut_IsNot() + { + var cutThenJudge = ScanBody("var cut = McpHelpers.Truncate(sqlText, 100); var j = SensitiveStatements.Text(cut);"); + Assert.Contains(cutThenJudge, h => h.Kind == "C# Truncate"); + + var judgeThenCut = ScanBody("var sqlText = SensitiveStatements.Text(raw); var p = McpHelpers.Truncate(sqlText, 100);"); + Assert.Empty(judgeThenCut); + } + + [Theory] + [InlineData("var graph = SensitiveStatements.Xml(raw, 100); var p = McpHelpers.Truncate(graph, 100);")] + [InlineData("var graphNormalized = SensitiveStatements.Text(Normalize(raw)); var p = graphNormalized![..400];")] + [InlineData("var judged = AnalysisStatementText.Preview(raw, 900); var p = judged.Substring(0, 5);")] + [InlineData("var filteredGraph = full ? r.Graph : SensitiveStatements.Xml(r.Graph, 400) ?? \"x\"; var p = full ? filteredGraph : McpHelpers.Truncate(filteredGraph, 400);")] + [InlineData("var fetchedText = SensitiveStatements.Text(reader.GetString(4)) ?? \"\"; var p = fetchedText[..McpHelpers.TextElementCutLength(fetchedText, 9)];")] + [InlineData("var queryText = SensitiveStatements.Text(raw) ?? \"\"; var p = AlertContextBuilders.TruncateText(queryText, 80);")] + public void TheScanner_LeavesALocalThatWasJudgedBeforeItsCut(string body) + { + Assert.Empty(ScanBody(body)); + } + + [Theory] + // judged in another method: the name is not excused there + [InlineData("class C { void A() { var sqlText = SensitiveStatements.Text(raw); } void B(string sqlText) { var p = McpHelpers.Truncate(sqlText, 5); } }")] + // judged AFTER the cut: the cut still ran on raw text + [InlineData("class C { void A() { var p = McpHelpers.Truncate(sqlText, 5); sqlText = SensitiveStatements.Text(raw); } }")] + // judged inside an inner block that has closed + [InlineData("class C { void A() { if (x) { var sqlText = SensitiveStatements.Text(raw); } var p = McpHelpers.Truncate(sqlText, 5); } }")] + // the judge call belongs to a different local + [InlineData("class C { void A() { var other = SensitiveStatements.Text(raw); var p = McpHelpers.Truncate(sqlText, 5); } }")] + // a member of an object initializer is not judged by its neighbour + [InlineData("class C { void A() { var o = new { sqlText = raw, b = SensitiveStatements.Text(raw) }; var p = McpHelpers.Truncate(sqlText, 5); } }")] + public void TheScanner_DoesNotExcuseAJudgedNameOutsideItsBlockOrBeforeItsAssignment(string source) + { + Assert.Contains(Scan(source), h => h.Kind == "C# Truncate"); + } + + [Theory] + [InlineData("const string Col = \"query_text\"; string Q() => $@\"SELECT LEFT({Col}, 500) FROM t\";")] + [InlineData("string Q(string queryTextColumn) => $@\"SELECT SUBSTRING({queryTextColumn}, 1, 300) FROM t\";")] + [InlineData("string Q(string a) => $@\"SELECT LEFT({a}.query_text, 500) FROM t {a}\";")] + [InlineData("string Q() { var col = \"blocked_sql_text\"; return $@\"SELECT substr({col}, 1, 200) FROM t\"; }")] + [InlineData("string Q(string statementCol) => $\"SELECT id, LEFT({statementCol}, 100) AS q, hash FROM t\";")] + public void TheScanner_CatchesASqlCutOverAnInterpolatedColumn(string members) + { + var hits = Scan("class C { " + members + " }"); + + Assert.Contains(hits, h => h.Kind.StartsWith("SQL ", StringComparison.Ordinal)); + } + + [Theory] + [InlineData("const string Col = \"relname\"; string Q() => $@\"SELECT LEFT({Col}, 63) FROM t\";")] + [InlineData("string Q(string nameColumn) => $@\"SELECT LEFT({nameColumn}, 63) FROM t\";")] + [InlineData("string Q(string cols) => $@\"SELECT {cols} FROM t\";")] + public void TheScanner_LeavesASqlCutOverAnInterpolatedColumnThatIsNotStatementText(string members) + { + Assert.Empty(Scan("class C { " + members + " }")); + } + + [Fact] + public void ACeilingException_PassesAtZero_AndFailsAboveItsCount_WhileAnExactOneFailsStale() + { + var none = new Dictionary>(); + var one = new Dictionary> { ["F.cs"] = new() { new Hit("C# range", 3, "p.sqltext") } }; + var two = new Dictionary> + { + ["F.cs"] = new() { new Hit("C# range", 3, "p.sqltext"), new Hit("C# range", 9, "p.sqltext") }, + }; + + var ceiling = new[] { new Allowed("F.cs", "p.sqltext", 1, "retires with its PR", Ceiling: true) }; + var exact = new[] { new Allowed("F.cs", "p.sqltext", 1, "stays") }; + + Assert.Empty(Reconcile(none, ceiling)); + Assert.Empty(Reconcile(one, ceiling)); + Assert.NotEmpty(Reconcile(two, ceiling)); + Assert.NotEmpty(Reconcile(none, exact)); + Assert.Empty(Reconcile(one, exact)); + Assert.NotEmpty(Reconcile(two, exact)); + } + + [Fact] + public void TheSurfaceOfTheScan_ReachesLiteAnalysis_AndTheIdentityOnlyExceptionsAreNamed() + { + Assert.Contains("Lite/Analysis", ScannedTrees); + + /* #5361 F1: the shared analysis project is scanned, and the pileup finding's printed statement is the reader's + judged PreviewText with an empty fallback, never the raw identity cut in QueryText. */ + Assert.Contains("PerformanceMonitor.Analysis", ScannedTrees); + var detector = RepoFile.ReadRepoFile("PerformanceMonitor.Analysis", "SameStatementPileupDetector.cs"); + Assert.Contains("Preview(leader.PreviewText ?? \"\")", detector, StringComparison.Ordinal); + Assert.DoesNotContain("PreviewText ?? leader.QueryText", detector, StringComparison.Ordinal); + Assert.DoesNotContain("string? PreviewText = null", detector, StringComparison.Ordinal); + Assert.Contains(Exceptions, e => e.File == "Lite/Analysis/PileupSnapshotReader.cs" && !e.Ceiling); + Assert.Contains(Exceptions, e => e.File.EndsWith("/PgPileupSnapshotReader.cs", StringComparison.Ordinal) && !e.Ceiling); + } +} diff --git a/Darling/Darling.Tests/StatementDrillDownTextFetchLiveTests.cs b/Darling/Darling.Tests/StatementDrillDownTextFetchLiveTests.cs new file mode 100644 index 0000000000..d16b990d11 --- /dev/null +++ b/Darling/Darling.Tests/StatementDrillDownTextFetchLiveTests.cs @@ -0,0 +1,481 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Analysis; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Analysis; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5361: the reconstructed-chain drill-down reads its 5,000 pair-rows WITHOUT statement text and fetches the whole text +/// of only the levels it shows, by event key, from the source each level's row came from (the blocked-process-report +/// view, or the DMV-snapshot view for an edge no report holds). What it prints must be what it printed when the pair +/// read carried the whole text: the same chains, the same levels in the same order, the same judged and cut text. +/// The oracle is that old flow, run in the test on the same rows: the pair read with the text columns restored, the +/// whole-text DMV append, the same reconstruction, per text. +/// The seeded rows repeat an edge with different text per row (so the wrong row is visible), mix both sources, include a +/// plain statement past the cut, a named statement the filter withholds, an astral character at the cut, a NULL text +/// and a NULL ecid. +/// +[Collection("live-postgres")] +public sealed class StatementDrillDownTextFetchLiveTests +{ + private const string ServerName = "statement-drilldown-textfetch-e2e"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static readonly string Plain = "SELECT plain_ssf " + new string('x', 683); + + /// An astral character (a surrogate pair) whose first half is the 500th character: the cut must not split it. + private static readonly string AstralAtCut = new string('a', 499) + "\U0001F600" + new string('b', 100); + + private static string Json(object? value) => JsonSerializer.Serialize(value); + + private static async Task InsertBprAsync( + NpgsqlConnection connection, CancellationToken ct, DateTime at, int blockedSpid, int blockingSpid, long waitMs, + string? blockedText, string? blockingText, int? blockedEcid = 0, int? blockingEcid = 0) + { + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO blocked_process_reports + (blocked_report_id, collection_time, server_id, server_name, event_time, database_name, blocked_spid, blocking_spid, wait_time_ms, + lock_mode, blocked_sql_text, blocking_sql_text, blocked_ecid, blocking_ecid, monitor_loop) +VALUES ($1, $2, $3, $4, $2, 'FetchDb', $5, $6, $7, 'X', $8, $9, $10, $11, 7)", + CollectionIdGenerator.Next(), at, ServerId, ServerName, blockedSpid, blockingSpid, waitMs, + (object?)blockedText ?? DBNull.Value, (object?)blockingText ?? DBNull.Value, + (object?)blockedEcid ?? DBNull.Value, (object?)blockingEcid ?? DBNull.Value); + } + + private static async Task InsertDmvAsync( + NpgsqlConnection connection, CancellationToken ct, DateTime at, int blockedSpid, int blockingSpid, long waitMs, + string? blockedText, string? blockingText) + { + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO dmv_blocking_snapshots + (collection_id, collection_time, server_id, server_name, event_time, database_name, blocked_spid, blocking_spid, wait_time_ms, + blocking_status, blocked_sql_text, blocking_sql_text, blocked_ecid, blocking_ecid) +VALUES ($1, $2, $3, $4, $2, 'FetchDb', $5, $6, $7, 'sleeping', $8, $9, 0, 0)", + CollectionIdGenerator.Next(), at, ServerId, ServerName, blockedSpid, blockingSpid, waitMs, + (object?)blockedText ?? DBNull.Value, (object?)blockingText ?? DBNull.Value); + } + + /// The flow of the drill-down before #5361, run on the same store: whole text in every pair-row. + private static async Task OracleChainsJsonAsync(NpgsqlConnection connection, AnalysisContext context, CancellationToken ct) + { + var oracleSql = PgDrillDownCollector.ReconstructedChainsSql + .Replace("''::text AS blocked_sql", "blocked_sql_text AS blocked_sql", StringComparison.Ordinal) + .Replace("''::text AS blocking_sql", "blocking_sql_text AS blocking_sql", StringComparison.Ordinal); + Assert.NotEqual(PgDrillDownCollector.ReconstructedChainsSql, oracleSql); + + using var cmd = new NpgsqlCommand(oracleSql, connection); + cmd.Parameters.AddWithValue(context.ServerId); + cmd.Parameters.AddWithValue(context.TimeRangeStart); + cmd.Parameters.AddWithValue(context.TimeRangeEnd); + cmd.Parameters.AddWithValue(EventWindowFloor.For(context.TimeRangeStart)); + var rows = new List(); + await using (var reader = await cmd.ExecuteReaderAsync(ct)) + { + while (await reader.ReadAsync(ct)) + rows.Add(PgBlockingPairRowQuery.Read(reader)); + } + + await PgBlockingPairRowQuery.AppendDmvSnapshotRowsAsync( + connection.CreateCommand, rows, context.ServerId, context.TimeRangeStart, context.TimeRangeEnd, ct, includeText: true); + Assert.Contains(rows, r => r.BlockedSqlText.Length > 0); + + var reconstruction = BlockingChainReconstructor.Reconstruct( + rows, maxDepth: 50, maxPairs: 5000, stepBudget: 100_000, scopeByMonitorLoop: false); + return Json(reconstruction.Chains.Take(3).Select(chain => (object)new + { + apex_spid = chain.ApexSpid, + apex_sleeping = chain.ApexSleeping, + depth = chain.Depth, + victim_count = chain.VictimCount, + max_wait_ms = chain.MaxWaitMs, + levels = chain.Levels.Select(l => new + { + level = l.Level, + blocking_spid = l.BlockingSpid, + blocked_spid = l.BlockedSpid, + lock_mode = l.LockMode, + wait_time_ms = l.WaitTimeMs, + blocking_sql = PgDrillDownCollector.StatementPreview(l.BlockingSqlText), + blocked_sql = PgDrillDownCollector.StatementPreview(l.BlockedSqlText) + }).ToList() + }).ToList()); + } + + [Fact] + public async Task ReconstructedChains_PrintTheSameLevelsAndText_AsTheWholeTextPairRead_FromBothSources() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live drill-down text test."); + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + var end = DarlingMcpTestData.Naive(DateTime.UtcNow).AddMinutes(1); + var context = new AnalysisContext + { + ServerId = ServerId, ServerName = ServerName, TimeRangeStart = end.AddHours(-4), TimeRangeEnd = end, + ServerUtcOffset = TimeSpan.Zero, CancellationToken = ct + }; + var t0 = new DateTime(context.TimeRangeStart.AddMinutes(30).Ticks / TimeSpan.TicksPerMinute * TimeSpan.TicksPerMinute, DateTimeKind.Unspecified); + var named = StatementScrubCanary.UriStatement(500); + + /* Chain A, blocked-process reports: 60 -> 71 -> 72 -> 73. The edge 60 -> 71 repeats three times with different + text per row; the longest wait (12,000, newest of the two) is the row the chain keeps. */ + await InsertBprAsync(connection, ct, t0.AddSeconds(1), 71, 60, 5_000, "SELECT older_a1", "SELECT older_b1"); + await InsertBprAsync(connection, ct, t0.AddSeconds(2), 71, 60, 12_000, named, Plain); + await InsertBprAsync(connection, ct, t0.AddSeconds(3), 71, 60, 12_000, AstralAtCut, AstralAtCut + "tail"); + await InsertBprAsync(connection, ct, t0.AddSeconds(4), 72, 71, 3_000, "SELECT long_" + new string('y', 2_000), "SELECT blocking_72"); + await InsertBprAsync(connection, ct, t0.AddSeconds(5), 73, 72, 2_000, null, ""); + await InsertBprAsync(connection, ct, t0.AddSeconds(6), 74, 72, 1_000, "SELECT null_ecid", "SELECT null_ecid_blocker", blockedEcid: null, blockingEcid: null); + + /* Chain B, DMV snapshots only (no report holds it): 80 -> 81 -> 82. */ + await InsertDmvAsync(connection, ct, t0.AddMinutes(5).AddSeconds(1), 81, 80, 4_000, "SELECT dmv_old", "SELECT dmv_old_blocker"); + await InsertDmvAsync(connection, ct, t0.AddMinutes(5).AddSeconds(30), 81, 80, 9_000, named, Plain); + await InsertDmvAsync(connection, ct, t0.AddMinutes(5).AddSeconds(31), 82, 81, 2_500, "SELECT dmv_82", AstralAtCut); + + /* Chain C: an edge both sources hold in one minute. The report wins and the DMV edge is dropped, so the text + printed is the report's, not the snapshot's. */ + await InsertBprAsync(connection, ct, t0.AddMinutes(10).AddSeconds(5), 91, 90, 6_000, "SELECT from_report", "SELECT from_report_blocker"); + await InsertDmvAsync(connection, ct, t0.AddMinutes(10).AddSeconds(25), 91, 90, 7_000, "SELECT from_snapshot", "SELECT from_snapshot_blocker"); + + var oracle = await OracleChainsJsonAsync(connection, context, ct); + + var finding = new AnalysisFinding { RootFactKey = "BLOCKING_CHAIN", StoryPath = "BLOCKING_CHAIN", PathKeys = ["BLOCKING_CHAIN"], Severity = 1.0 }; + await new PgDrillDownCollector(postgres).EnrichFindingsAsync([finding], context); + Assert.NotNull(finding.DrillDown); + var shipped = Json(finding.DrillDown!["reconstructed_blocking_chains"]); + + Assert.Equal(oracle, shipped); + + /* Positive controls: the comparison saw all three chains, both sources, the cut, the withheld marker and the + text of the row the reconstruction kept (not an older or a snapshot row). */ + var chains = JsonSerializer.Deserialize(shipped).EnumerateArray().ToList(); + Assert.Equal(3, chains.Count); + Assert.Contains(SensitiveStatements.PlaceholderText, shipped, StringComparison.Ordinal); + Assert.DoesNotContain(StatementScrubCanary.UriSecretPartial, shipped, StringComparison.Ordinal); + Assert.Contains(Plain[..500], shipped, StringComparison.Ordinal); + Assert.DoesNotContain(Plain[..501], shipped, StringComparison.Ordinal); + Assert.Contains("SELECT dmv_82", shipped, StringComparison.Ordinal); + Assert.Contains("SELECT from_report", shipped, StringComparison.Ordinal); + Assert.DoesNotContain("SELECT from_snapshot", shipped, StringComparison.Ordinal); + Assert.DoesNotContain("SELECT older_a1", shipped, StringComparison.Ordinal); + Assert.Contains("SELECT null_ecid", shipped, StringComparison.Ordinal); + + /* The pair read itself carries no statement text. */ + using var cmd = new NpgsqlCommand(PgDrillDownCollector.ReconstructedChainsSql, connection); + cmd.Parameters.AddWithValue(ServerId); + cmd.Parameters.AddWithValue(context.TimeRangeStart); + cmd.Parameters.AddWithValue(context.TimeRangeEnd); + cmd.Parameters.AddWithValue(EventWindowFloor.For(context.TimeRangeStart)); + var pairRows = new List(); + await using (var reader = await cmd.ExecuteReaderAsync(ct)) + { + while (await reader.ReadAsync(ct)) + pairRows.Add(PgBlockingPairRowQuery.Read(reader)); + } + + Assert.True(pairRows.Count >= 7, "the BPR pair read returned " + pairRows.Count + " rows"); + Assert.All(pairRows, r => Assert.True(r.BlockedSqlText.Length == 0 && r.BlockingSqlText.Length == 0)); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// + /// The parameter-sensitive drill-down ranks plans WITHOUT text (has_inline_text) and reads the whole inline text of + /// the five it keeps by (database, query hash, plan hash), then the dimension's text by digest for a kept plan with no + /// inline text. It must print what it printed when the ranking read carried each row's inline text. The seeded + /// plans: a newest row whose inline text differs from the older rows' (the newest wins), a plain statement past the + /// cut, a named statement the filter withholds, an astral character at the cut under a NULL database, an EMPTY inline + /// text (inline wins even when empty), a plan with no inline text on its newest row but some on older rows (the + /// dimension's text), and two plans sharing one query hash with different plan hashes. + /// + [Fact] + public async Task ParameterSensitivePlans_PrintTheSameText_AsTheRankingReadThatCarriedIt() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live drill-down text test."); + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + var end = DarlingMcpTestData.Naive(DateTime.UtcNow).AddMinutes(1); + var windowStart = end.AddHours(-4); + var context = new AnalysisContext + { + ServerId = ServerId, ServerName = ServerName, TimeRangeStart = windowStart, TimeRangeEnd = end, + ServerUtcOffset = TimeSpan.Zero, CancellationToken = ct + }; + var named = StatementScrubCanary.UriStatement(500); + + /* plan: (database, query hash, plan hash, newest inline text, older inline text, dimension text or null). */ + var plans = new (string? Db, string Hash, string PlanHash, string? Newest, string? Older, string? Dim)[] + { + ("FetchDb", "0xQH_SAME", "0xPH_0", Plain, "SELECT older_0", null), + ("FetchDb", "0xQH_SAME", "0xPH_1", named, "SELECT older_1", null), + (null, "0xQH_2", "0xPH_2", AstralAtCut, null, null), + ("FetchDb", "0xQH_3", "0xPH_3", "", "SELECT older_3", null), + ("FetchDb", "0xQH_4", "0xPH_4", null, "SELECT older_4", "SELECT dimension_4 " + new string('d', 600)), + ("FetchDb", "0xQH_5", "0xPH_5", "SELECT beyond_the_cap_5", null, null), + ("FetchDb", "0xQH_6", "0xPH_6", "SELECT beyond_the_cap_6", null, null), + }; + + var compiledBeforeWindow = DateTime.SpecifyKind(windowStart.AddDays(-2), DateTimeKind.Unspecified); + for (var p = 0; p < plans.Length; p++) + { + var plan = plans[p]; + for (var snapshot = 0; snapshot < 3; snapshot++) + { + var newest = snapshot == 2; + var digest = System.Security.Cryptography.SHA256.HashData(System.Text.Encoding.UTF8.GetBytes($"d6 {p} {newest}")); + if (newest && plan.Dim is not null) + { + await DarlingMcpTestData.ExecAsync(connection, ct, + "INSERT INTO query_text_dim (digest, query_text, last_seen) VALUES ($1, $2, $3) ON CONFLICT (digest) DO NOTHING", + digest, plan.Dim, DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified)); + } + + var inline = newest ? plan.Newest : (plan.Older is null ? null : plan.Older + " snapshot " + snapshot); + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO query_stats + (collection_id, collection_time, server_id, server_name, database_name, query_hash, query_plan_hash, + creation_time, execution_count, min_worker_time, max_worker_time, min_grant_kb, max_grant_kb, + min_spills, max_spills, query_text, query_text_digest, delta_execution_count) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, 20000, $10, 1024, 2048, 0, 0, $11, $12, 25)", + CollectionIdGenerator.Next(), DateTime.SpecifyKind(windowStart.AddMinutes(30 + (snapshot * 60)), DateTimeKind.Unspecified), + ServerId, ServerName, (object?)plan.Db ?? DBNull.Value, plan.Hash, plan.PlanHash, compiledBeforeWindow, + 500L + snapshot, 20_000L * (60 - p), (object?)inline ?? DBNull.Value, digest); + } + } + + /* The oracle: the ranking read with the text restored (the shape before #5361), the reader's cap, and the + dimension read for the kept plans. */ + var oracleSql = PgDrillDownCollector.ParameterSensitiveSql + .Replace("query_text IS NOT NULL AS has_inline_text,", "query_text,", StringComparison.Ordinal) + .Replace(" has_inline_text,", " query_text,", StringComparison.Ordinal) + .Replace(" o.has_inline_text,", " o.query_text,", StringComparison.Ordinal); + Assert.DoesNotContain("has_inline_text", oracleSql, StringComparison.Ordinal); + var kept = new List<(object?[] Row, string? Inline, byte[]? Digest)>(); + using (var cmd = new NpgsqlCommand(oracleSql, connection)) + { + cmd.Parameters.AddWithValue(ServerId); + cmd.Parameters.AddWithValue(windowStart); + cmd.Parameters.AddWithValue(end); + await using var reader = await cmd.ExecuteReaderAsync(ct); + while (kept.Count < PgDrillDownCollector.ParameterSensitiveMaxOffenders && await reader.ReadAsync(ct)) + { + var row = new object?[10]; + for (var i = 0; i < 9; i++) + row[i] = reader.IsDBNull(i) ? null : reader.GetValue(i); + kept.Add((row, reader.IsDBNull(9) ? null : reader.GetString(9), reader.IsDBNull(13) ? null : reader.GetFieldValue(13))); + } + } + + Assert.Equal(PgDrillDownCollector.ParameterSensitiveMaxOffenders, kept.Count); + var dimension = new Dictionary(StringComparer.Ordinal); + await using (var dimCmd = new NpgsqlCommand(PgDrillDownCollector.ParameterSensitiveTextSql, connection)) + { + dimCmd.Parameters.Add(new NpgsqlParameter + { + NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Array | NpgsqlTypes.NpgsqlDbType.Bytea, + Value = kept.Where(k => k.Inline is null && k.Digest is not null).Select(k => k.Digest!).ToArray() + }); + await using var dimReader = await dimCmd.ExecuteReaderAsync(ct); + while (await dimReader.ReadAsync(ct)) + dimension[PgDrillDownCollector.DigestKey(dimReader.GetFieldValue(0))] = dimReader.GetString(1); + } + + var oracle = Json(kept.Select(k => (object)new + { + database = k.Row[0] as string ?? "", + query_hash = k.Row[1] as string ?? "", + query_plan_hash = k.Row[2] as string ?? "", + execution_count = k.Row[3] is null ? 0L : Convert.ToInt64(k.Row[3]), + min_worker_time_us = k.Row[4] is null ? 0L : Convert.ToInt64(k.Row[4]), + max_worker_time_us = k.Row[5] is null ? 0L : Convert.ToInt64(k.Row[5]), + worker_ratio = k.Row[6] is null ? 0.0 : Convert.ToDouble(k.Row[6]), + grant_ratio = k.Row[7] is null ? 0.0 : Convert.ToDouble(k.Row[7]), + spills_on_some_inputs = k.Row[8] is not null && Convert.ToInt32(k.Row[8]) == 1, + query_text = PgDrillDownCollector.StatementPreview(PgDrillDownCollector.SettledQueryText(k.Inline, k.Digest, dimension)) + }).ToList()); + + var finding = new AnalysisFinding { RootFactKey = "PARAMETER_SENSITIVITY", StoryPath = "PARAMETER_SENSITIVITY", PathKeys = ["PARAMETER_SENSITIVITY"], Severity = 1.0 }; + await new PgDrillDownCollector(postgres).EnrichFindingsAsync([finding], context); + Assert.NotNull(finding.DrillDown); + var shipped = Json(finding.DrillDown!["parameter_sensitive_queries"]); + + Assert.Equal(oracle, shipped); + + /* Positive controls: the newest row's text and not an older one's, the cut, the withheld marker, the empty inline + text, the dimension's text, the plan cap, and a ranking read that carries no text. */ + Assert.Contains(Plain[..500], shipped, StringComparison.Ordinal); + Assert.DoesNotContain(Plain[..501], shipped, StringComparison.Ordinal); + Assert.Contains(SensitiveStatements.PlaceholderText, shipped, StringComparison.Ordinal); + Assert.DoesNotContain(StatementScrubCanary.UriSecretPartial, shipped, StringComparison.Ordinal); + Assert.DoesNotContain("older_", shipped, StringComparison.Ordinal); + Assert.Contains("SELECT dimension_4", shipped, StringComparison.Ordinal); + Assert.DoesNotContain("beyond_the_cap", shipped, StringComparison.Ordinal); + Assert.Contains("\"query_text\":\"\"", shipped, StringComparison.Ordinal); + + using var rankCmd = new NpgsqlCommand(PgDrillDownCollector.ParameterSensitiveSql, connection); + rankCmd.Parameters.AddWithValue(ServerId); + rankCmd.Parameters.AddWithValue(windowStart); + rankCmd.Parameters.AddWithValue(end); + await using var rankReader = await rankCmd.ExecuteReaderAsync(ct); + Assert.Equal("boolean", rankReader.GetDataTypeName(9)); + var ranked = 0; + while (await rankReader.ReadAsync(ct)) + ranked++; + Assert.Equal(plans.Length, ranked); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// + /// #1767 / #5361: since the collectors write the fact row's query_text as NULL and keep the text in + /// query_text_dim, the inline-text read must resolve the dimension too, the way the view's + /// COALESCE(f.query_text, qtd.query_text) does. The two seeded rows above write inline text, so they could + /// not see a read of the bare fact column. Here the newest row of one plan has NULL inline text and a digest + /// whose text lives only in the dimension; a second plan has inline text AND a dimension row (inline wins, even + /// when empty); a third has neither (no text, so the plan is left out of the answer). + /// + [Fact] + public async Task ParameterSensitiveInlineText_ResolvesADimensionStoredText_WhenTheFactPayloadIsNull() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live drill-down text test."); + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + var digests = new[] + { + System.Security.Cryptography.SHA256.HashData(System.Text.Encoding.UTF8.GetBytes("d10 dim only")), + System.Security.Cryptography.SHA256.HashData(System.Text.Encoding.UTF8.GetBytes("d10 inline wins")), + System.Security.Cryptography.SHA256.HashData(System.Text.Encoding.UTF8.GetBytes("d10 neither")), + }; + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + var end = DarlingMcpTestData.Naive(DateTime.UtcNow).AddMinutes(1); + var windowStart = end.AddHours(-4); + var dimensionText = "SELECT dimension_only " + new string('d', 800); + + foreach (var (digest, dimText) in new[] { (digests[0], dimensionText), (digests[1], "SELECT dimension_loses") }) + { + await DarlingMcpTestData.ExecAsync(connection, ct, + "INSERT INTO query_text_dim (digest, query_text, last_seen) VALUES ($1, $2, $3) ON CONFLICT (digest) DO NOTHING", + digest, dimText, DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified)); + } + + /* plan: (plan hash, older inline text, newest inline text, digest). The newest row is the one the read keeps. */ + var plans = new (string PlanHash, string? Older, string? Newest, byte[] Digest)[] + { + ("0xPH_DIM", "SELECT older_inline_is_not_the_newest", null, digests[0]), + ("0xPH_INLINE", null, "", digests[1]), + ("0xPH_NONE", null, null, digests[2]), + }; + + foreach (var plan in plans) + { + for (var snapshot = 0; snapshot < 2; snapshot++) + { + var newest = snapshot == 1; + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO query_stats + (collection_id, collection_time, server_id, server_name, database_name, query_hash, query_plan_hash, + creation_time, execution_count, min_worker_time, max_worker_time, min_grant_kb, max_grant_kb, + min_spills, max_spills, query_text, query_text_digest, delta_execution_count) +VALUES ($1, $2, $3, $4, 'FetchDb', '0xQH_DIM', $5, $6, 500, 20000, 400000, 1024, 2048, 0, 0, $7, $8, 25)", + CollectionIdGenerator.Next(), + DateTime.SpecifyKind(windowStart.AddMinutes(30 + (snapshot * 60)), DateTimeKind.Unspecified), + ServerId, ServerName, plan.PlanHash, DateTime.SpecifyKind(windowStart.AddDays(-2), DateTimeKind.Unspecified), + (object?)(newest ? plan.Newest : plan.Older) ?? DBNull.Value, plan.Digest); + } + } + + var text = new Dictionary(); + using (var cmd = new NpgsqlCommand(PgDrillDownCollector.ParameterSensitiveInlineTextSql, connection)) + { + cmd.Parameters.AddWithValue(ServerId); + cmd.Parameters.AddWithValue(windowStart); + cmd.Parameters.AddWithValue(end); + cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Array | NpgsqlTypes.NpgsqlDbType.Text, Value = plans.Select(_ => "FetchDb").ToArray() }); + cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Array | NpgsqlTypes.NpgsqlDbType.Text, Value = plans.Select(_ => "0xQH_DIM").ToArray() }); + cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlTypes.NpgsqlDbType.Array | NpgsqlTypes.NpgsqlDbType.Text, Value = plans.Select(p => p.PlanHash).ToArray() }); + await using var reader = await cmd.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + text[reader.GetString(2)] = reader.IsDBNull(3) ? null : reader.GetString(3); + } + + Assert.Equal(3, text.Count); + /* Fact payload NULL, text only in the dimension: the whole dimension text, uncut. */ + Assert.Equal(dimensionText, text["0xPH_DIM"]); + /* Inline text wins over the dimension's, even when it is empty (the view's COALESCE). */ + Assert.Equal("", text["0xPH_INLINE"]); + /* No inline text and no dimension row for the digest: NULL, which the reader leaves out. */ + Assert.Null(text["0xPH_NONE"]); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + { + await DeleteRowsAsync(cleanup, cleanupCt); + foreach (var digest in digests) + await DarlingMcpTestData.ExecAsync(cleanup, cleanupCt, "DELETE FROM query_text_dim WHERE digest = $1", digest); + }); + } + } + + private static async Task DeleteRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + foreach (var table in new[] { "blocked_process_reports", "dmv_blocking_snapshots", "query_stats" }) + await DarlingMcpTestData.ExecAsync(connection, ct, $"DELETE FROM {table} WHERE server_id = $1", ServerId); + + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM servers WHERE server_id = $1", ServerId); + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM config_monitored_servers WHERE server_id = $1", ServerId); + } +} diff --git a/Darling/Darling.Tests/StatementFilterAlertTests.cs b/Darling/Darling.Tests/StatementFilterAlertTests.cs new file mode 100644 index 0000000000..6c0e41e08f --- /dev/null +++ b/Darling/Darling.Tests/StatementFilterAlertTests.cs @@ -0,0 +1,602 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using PerformanceMonitor.Alerting; +using PerformanceMonitor.Common; +using PerformanceMonitor.Notifications; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5320 (part of #4348): alert notifications go through the statement filter. The engine's FireAsync +/// filters every outcome before it logs or delivers it; the finding senders filter a finding before they send or +/// record it. These pins run the real engine against a fake deliverer, so an alert that carries a statement or a +/// report in its details, its attachments or its short message reaches the deliverer withheld. +/// +[Collection("timing")] +public sealed class StatementFilterAlertTests +{ + private const string Marker = SensitiveStatements.PlaceholderText; + + /// A blocked-process report whose two input buffers hold the canary and the plain statement. + internal static string ReportXml() => + "" + + "" + StatementScrubCanary.CanaryStatement + "" + + "" + StatementScrubCanary.PlainStatement + + ""; + + /// Every string an outcome would put in front of a channel, joined, so one needle search covers it. + internal static string Everything(AlertOutcome outcome) + { + var parts = new List { outcome.DetailText, outcome.ShortMessage, outcome.CurrentValue, outcome.ThresholdValue }; + parts.AddRange(Everything(outcome.Context)); + return string.Join("\n", parts.Where(p => p is not null)); + } + + internal static IEnumerable Everything(AlertContext? context) + { + if (context is null) + { + yield break; + } + + yield return context.AttachmentXml; + foreach (var item in context.Details) + { + yield return item.Heading; + yield return item.Body; + foreach (var (label, value) in item.Fields) + { + yield return label; + yield return value; + } + + foreach (var record in item.Records) + { + yield return record.Summary; + foreach (var (label, text) in record.Texts) + { + yield return label; + yield return text; + } + } + } + + foreach (var incident in (context.Incidents ?? new List()).Where(i => i is not null)) + { + foreach (var involved in incident.InvolvedObjects) + { + yield return involved; + } + + yield return incident.Attachment?.Xml; + foreach (var field in incident.DetailFields ?? Array.Empty()) + { + yield return field.Value; + } + } + } + + internal static void AssertNoSecret(string text) + { + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.DoesNotContain(needle, text, StringComparison.Ordinal); + } + } + + internal static AlertEngineTests.Harness BlockingHarness(out AlertEngineTests.Harness h) + { + h = new AlertEngineTests.Harness(); + h.Settings.BlockingEnabled = true; + var row = AlertEngineTests.BlockingRow(55); + row.BlockedSqlText = StatementScrubCanary.CanaryStatement; + row.BlockingSqlText = StatementScrubCanary.PlainStatement; + row.BlockedProcessReportXml = ReportXml(); + h.Adapter.Blocking.Add(row); + return h; + } + + [Fact] + public async Task Engine_BlockingAlert_DeliversDetailsAndReportWithTheStatementWithheld() + { + var h = BlockingHarness(out _); + + await h.Build().EvaluateServerAsync(AlertEngineTests.Harness.Snapshot()); + + var outcome = Assert.Single(h.Deliverer.Outcomes); + var everything = Everything(outcome); + AssertNoSecret(everything); + Assert.Contains(Marker, everything, StringComparison.Ordinal); + /* The rest of the alert is unchanged: the plain statement, the report's other buffer and the database. */ + Assert.Contains(StatementScrubCanary.PlainStatement, everything, StringComparison.Ordinal); + Assert.Contains("StackOverflow", everything, StringComparison.Ordinal); + Assert.NotNull(Assert.Single(outcome.Context!.Incidents!).Attachment); + var attachment = outcome.Context.Incidents![0].Attachment!; + Assert.Contains(Marker, attachment.Xml, StringComparison.Ordinal); + Assert.Contains(StatementScrubCanary.PlainStatement, attachment.Xml, StringComparison.Ordinal); + Assert.NotNull(System.Xml.Linq.XDocument.Parse(attachment.Xml)); + } + + [Fact] + public async Task Engine_BlockingAlert_FlatDetailTextIsRebuiltFromTheFilteredContext() + { + var h = BlockingHarness(out _); + + await h.Build().EvaluateServerAsync(AlertEngineTests.Harness.Snapshot()); + + var outcome = Assert.Single(h.Deliverer.Outcomes); + /* The engine's flat text is the flattening of its context; after the filter it still is, so the two + surfaces a channel renders (structured details and flat text) say the same thing. */ + Assert.False(string.IsNullOrEmpty(outcome.DetailText)); + Assert.Equal(AlertContextBuilders.ContextToDetailText(outcome.Context), outcome.DetailText); + Assert.Contains(Marker, outcome.DetailText, StringComparison.Ordinal); + } + + [Fact] + public async Task Engine_LongRunningQueryAlert_WithholdsTheQueryTextAndTheToastBody() + { + var h = new AlertEngineTests.Harness(); + h.Settings.LongRunningQueryEnabled = true; + h.Adapter.LongRunning.Add(new LongRunningQueryInfo + { + SessionId = 71, + DatabaseName = "StackOverflow", + QueryText = StatementScrubCanary.CanaryStatement, + ElapsedSeconds = 45 * 60, + CpuTimeMs = 1000, + QueryHash = "0x9AAF0129E4E9AD07", + }); + + await h.Build().EvaluateServerAsync(AlertEngineTests.Harness.Snapshot()); + + var outcome = Assert.Single(h.Deliverer.Outcomes); + var everything = Everything(outcome); + AssertNoSecret(everything); + Assert.Contains(Marker, everything, StringComparison.Ordinal); + Assert.Contains("71", everything, StringComparison.Ordinal); + } + + [Fact] + public async Task Engine_LongRunningQueryAlert_APlainQueryReachesTheDelivererUnchanged() + { + var h = new AlertEngineTests.Harness(); + h.Settings.LongRunningQueryEnabled = true; + h.Adapter.LongRunning.Add(new LongRunningQueryInfo + { + SessionId = 71, + DatabaseName = "StackOverflow", + QueryText = StatementScrubCanary.PlainStatement, + ElapsedSeconds = 45 * 60, + CpuTimeMs = 1000, + QueryHash = "0x9AAF0129E4E9AD07", + }); + + await h.Build().EvaluateServerAsync(AlertEngineTests.Harness.Snapshot()); + + var everything = Everything(Assert.Single(h.Deliverer.Outcomes)); + Assert.Contains("canary_plain_ssf", everything, StringComparison.Ordinal); + Assert.DoesNotContain(Marker, everything, StringComparison.Ordinal); + } + + [Fact] + public async Task Engine_MuteRuleKeyedOnTheRawText_StillMutes_WhileTheDeliveredOutcomeIsFiltered() + { + var h = new AlertEngineTests.Harness + { + /* A rule that matches the statement as stored (before the filter existed): it is asked about the + RAW text, so it still matches and the alert is recorded as muted. */ + IsMuted = context => context.QueryText?.Contains("canary_ssf", StringComparison.Ordinal) == true, + }; + h.Settings.LongRunningQueryEnabled = true; + h.Adapter.LongRunning.Add(new LongRunningQueryInfo + { + SessionId = 71, + DatabaseName = "StackOverflow", + QueryText = StatementScrubCanary.CanaryStatement, + ElapsedSeconds = 45 * 60, + CpuTimeMs = 1000, + QueryHash = "0x9AAF0129E4E9AD07", + }); + + await h.Build().EvaluateServerAsync(AlertEngineTests.Harness.Snapshot()); + + var outcome = Assert.Single(h.Deliverer.Outcomes); + Assert.True(outcome.Muted); + AssertNoSecret(Everything(outcome)); + } + + [Fact] + public void Apply_AnAlertWithNothingNamed_ComesBackAsTheSameInstance() + { + var context = new AlertContext(); + var item = new AlertDetailItem { Heading = "Blocking Chain #1" }; + item.Fields.Add(("Blocked Query", StatementScrubCanary.PlainStatement)); + context.Details.Add(item); + context.AttachmentXml = ReportXml().Replace(StatementScrubCanary.CanaryStatement, StatementScrubCanary.PlainStatement, StringComparison.Ordinal); + var outcome = new AlertOutcome( + "1", "SRV", "Blocking Detected", "1", "1", context, + AlertContextBuilders.ContextToDetailText(context), null, null, false, null, "short"); + + Assert.Same(outcome, AlertStatementFilter.Apply(outcome)); + Assert.Same(context, AlertStatementFilter.Apply(context)); + } + + [Fact] + public void Apply_JudgesEveryValueAnAlertCarries() + { + var context = new AlertContext { AttachmentXml = ReportXml(), AttachmentFileName = "blocked_process_report.xml" }; + var item = new AlertDetailItem { Heading = StatementScrubCanary.CanaryStatement, Body = StatementScrubCanary.CanaryStatement }; + item.Fields.Add(("Blocked Query", StatementScrubCanary.CanaryStatement)); + item.Fields.Add(("Database", "StackOverflow")); + item.Records.Add(new AlertDetailRecord(1, StatementScrubCanary.CanaryStatement, + new List<(string Label, string Text)> { ("Query Text", StatementScrubCanary.CanaryStatement), ("Other", StatementScrubCanary.PlainStatement) })); + context.Details.Add(item); + context.Incidents = new List + { + new("key", new[] { "dbo.t" }, DetailFields: new[] { new AlertIncidentField("Victim SQL", StatementScrubCanary.CanaryStatement) }, + Attachment: new AlertIncidentAttachment(ReportXml(), AlertIncidentAttachment.BlockedProcessReportFileName)), + }; + + var filtered = AlertStatementFilter.Apply(context)!; + + Assert.NotSame(context, filtered); + var everything = string.Join("\n", Everything(filtered).Where(p => p is not null)); + AssertNoSecret(everything); + Assert.Contains(StatementScrubCanary.PlainStatement, everything, StringComparison.Ordinal); + Assert.Equal("StackOverflow", filtered.Details[0].Fields[1].Value); + Assert.Equal("blocked_process_report.xml", filtered.AttachmentFileName); + Assert.Equal("key", filtered.Incidents![0].DedupKey); + Assert.Equal(AlertIncidentAttachment.BlockedProcessReportFileName, filtered.Incidents[0].Attachment!.FileName); + + /* The original is untouched: a context is shared with the caller's own state. */ + Assert.Contains(StatementScrubCanary.CanaryStatement, context.Details[0].Fields[0].Value, StringComparison.Ordinal); + Assert.Contains("S3cret-canary-ssf", context.AttachmentXml, StringComparison.Ordinal); + } + + [Fact] + public void Apply_DetailTextThatIsNotTheFlatteningIsJudgedWhole() + { + var context = new AlertContext(); + var outcome = new AlertOutcome( + "1", "SRV", "Self Alert", "1", "1", context, + "The collector failed on " + StatementScrubCanary.CanaryStatement, null, null, false, null, + "short " + StatementScrubCanary.CanaryStatement); + + var filtered = AlertStatementFilter.Apply(outcome); + + Assert.Equal(Marker, filtered.DetailText); + Assert.Equal(Marker, filtered.ShortMessage); + } + + [Fact] + public void Apply_AFailureDeliversTheAlertWithItsDetailsAndAttachmentsDropped() + { + var context = new AlertContext { AttachmentXml = ReportXml(), AttachmentFileName = "blocked_process_report.xml" }; + /* A null item cannot be read: the filter must fail closed, not pass the alert through. */ + context.Details.Add(new AlertDetailItem { Heading = "ok" }); + context.Details.Add(null!); + context.Incidents = new List + { + new("key", new[] { "dbo.t" }, DetailFields: new[] { new AlertIncidentField("Victim SQL", StatementScrubCanary.CanaryStatement) }, + Attachment: new AlertIncidentAttachment(ReportXml(), AlertIncidentAttachment.BlockedProcessReportFileName)), + }; + var outcome = new AlertOutcome( + "1", "SRV", "Blocking Detected", "1", "1", context, "flat " + StatementScrubCanary.CanaryStatement, + null, null, false, null, "short " + StatementScrubCanary.CanaryStatement); + + var filtered = AlertStatementFilter.Apply(outcome); + + Assert.Empty(filtered.Context!.Details); + Assert.Null(filtered.Context.AttachmentXml); + Assert.Null(filtered.Context.AttachmentFileName); + var incident = Assert.Single(filtered.Context.Incidents!); + Assert.Equal("key", incident.DedupKey); + Assert.Null(incident.Attachment); + Assert.Null(incident.DetailFields); + Assert.Equal(Marker, filtered.DetailText); + Assert.Equal(Marker, filtered.ShortMessage); + AssertNoSecret(Everything(filtered)); + } + + [Fact] + public void Apply_AQuerySetThatSlowsTheJudge_FinishesInsideTheBudget() + { + /* Many long values built to make the judge work: nested gaps and unterminated comments. Past the budget a value + is withheld unjudged, so the work is bounded however many arrive. + + #5459: this asserted `took < 1950 ms` on the real clock, so a runner that stalled for a moment (a collection, + a descheduled thread) failed it though the budget logic was fine. The budget's clock is now a stepped one: + every judge call is charged 50 ms, so the budget (1.5 s, plus a little earned per value) is spent after a few + dozen of the 800 calls, on any machine. The real judge still runs on every value it is given. The wall-clock + bound is kept at 10x only to catch a hang. */ + var context = new AlertContext(); + for (var i = 0; i < 400; i++) + { + var item = new AlertDetailItem { Heading = "Query " + i }; + var gap = string.Concat(Enumerable.Repeat("/* " + i + " */ -- x\n ", 600)); + item.Fields.Add(("Query Text", "create " + gap + " /* never closed " + new string('a', 20_000))); + context.Details.Add(item); + } + + var step = TimeSpan.FromMilliseconds(50); + var budget = SteppingBudget(step); + var stopwatch = Stopwatch.StartNew(); + AlertContext filtered; + using (AlertStatementFilter.UseBudget(() => budget)) + { + filtered = AlertStatementFilter.Apply(context)!; + } + + stopwatch.Stop(); + + Assert.True(stopwatch.ElapsedMilliseconds < 19_500, "took " + stopwatch.ElapsedMilliseconds + " ms"); + Assert.Equal(400, filtered.Details.Count); + Assert.True(budget.Spent, "the budget did not run out"); + Assert.True(budget.Unjudged > 0, "nothing was withheld unjudged"); + // The judge ran only while the budget was open: it is charged exactly one step per call, so the time charged + // can pass the limit by at most the one call that crossed the line. That is the bound, in the budget's own time. + Assert.True(budget.Elapsed < budget.Limit + step, $"charged {budget.Elapsed} against a limit of {budget.Limit}"); + } + + [Fact] + public void ApplyFinding_JudgesTheContextAndTheProse() + { + var context = new AlertContext(); + var item = new AlertDetailItem { Heading = "Top Cpu Queries" }; + item.Fields.Add(("Query Text", StatementScrubCanary.CanaryStatement)); + context.Details.Add(item); + var alert = new FindingAlert( + "Analysis: High CPU", "SRV", "1", "1", "101", context, 0.9, 0.5, + "prose " + StatementScrubCanary.CanaryStatement, true); + + var filtered = AlertStatementFilter.Apply(alert); + + Assert.NotSame(alert, filtered); + Assert.Equal(Marker, filtered.DetailText); + AssertNoSecret(string.Join("\n", Everything(filtered.Context))); + Assert.Equal("Analysis: High CPU", filtered.MetricName); + Assert.Equal(0.9, filtered.Severity); + + var clean = alert with { DetailText = "prose", Context = new AlertContext() }; + Assert.Same(clean, AlertStatementFilter.Apply(clean)); + } + + [Fact] + public void ApplyFindings_ReturnsTheSameListWhenNothingIsNamed_AndOnlyTheNamedAlertIsReplaced() + { + var clean = new FindingAlert("A", "SRV", "1", "1", "101", new AlertContext(), 0.9, 0.5, "plain", true); + var named = new FindingAlert("B", "SRV", "1", "1", "101", new AlertContext(), 0.9, 0.5, StatementScrubCanary.CanaryStatement, true); + + var untouched = new List { clean }; + Assert.Same(untouched, AlertStatementFilter.Apply(untouched)); + + var both = new List { clean, named }; + var filtered = AlertStatementFilter.Apply(both); + Assert.Same(clean, filtered[0]); + Assert.Equal(Marker, filtered[1].DetailText); + Assert.Equal(StatementScrubCanary.CanaryStatement, both[1].DetailText); + } + + [Fact] + public async Task Engine_AFourMegabyteReport_ReachesTheDelivererFilteredNotWithheldWhole() + { + await StatementFilterWarmUp.EnsureAsync(); + using var pinned = UnspendableBudget(); + var h = BlockingHarness(out _); + /* One report with a 4 MB tail of harmless elements: the walk reads all of it, withholds only the + canary statement, and does not run out of budget. */ + h.Adapter.Blocking[0].BlockedProcessReportXml = ReportXml().Replace( + "", + string.Concat(Enumerable.Repeat("" + new string('x', 400) + "", 10_000)) + "", + StringComparison.Ordinal); + + await h.Build().EvaluateServerAsync(AlertEngineTests.Harness.Snapshot()); + + var outcome = Assert.Single(h.Deliverer.Outcomes); + AssertNoSecret(Everything(outcome)); + var xml = Assert.Single(outcome.Context!.Incidents!).Attachment!.Xml; + Assert.True(xml.Length > 4_000_000, "the report was withheld whole: " + xml.Length); + Assert.Contains(StatementScrubCanary.PlainStatement, xml, StringComparison.Ordinal); + } + + [Fact] + public void Apply_AFourMegabyteReport_ComesBackFilteredNotWithheldWhole() + { + StatementFilterWarmUp.Ensure(); + using var pinned = UnspendableBudget(); + var report = ReportXml().Replace( + "", + string.Concat(Enumerable.Repeat("" + new string('x', 400) + "", 10_000)) + "", + StringComparison.Ordinal); + var context = new AlertContext { AttachmentXml = report }; + + var filtered = AlertStatementFilter.Apply(context)!; + + /* #5459: this asserted `took < 1950 ms` under the real 1.5 s + 0.5 s per MB wall-clock budget, so a slow runner + withheld the document whole and failed it. The budget is pinned so it cannot run out; the proof that a + document past the budget is withheld is Apply_ABudgetAlreadySpent_WithholdsTheDocumentWholeAndEveryValue. */ + Assert.Contains(StatementScrubCanary.PlainStatement, filtered.AttachmentXml!, StringComparison.Ordinal); + Assert.True(filtered.AttachmentXml!.Length > 4_000_000, "the report was withheld whole: " + filtered.AttachmentXml.Length); + AssertNoSecret(filtered.AttachmentXml!); + } + + /// A budget that cannot run out on a slow runner (#5459): one hour, which is above the 10 s cap a budget + /// grows to, so it stays one hour. Every clean value and document comes back as it went in. + internal static IDisposable UnspendableBudget() => + AlertStatementFilter.UseBudget(() => new SensitiveStatements.JudgeBudget(TimeSpan.FromHours(1))); + + /// A budget that is already spent when the filter first looks at it (#5459): an hour of elapsed time is + /// charged up front, which no document can earn back (the cap is 10 s). The real judge still runs, so nothing is + /// faked except the time the budget has already been charged. + private static SensitiveStatements.JudgeBudget SpentBudget() + { + var budget = new SensitiveStatements.JudgeBudget(SensitiveStatements.ReadBudget); + budget.AddElapsed(TimeSpan.FromHours(1)); + Assert.True(budget.Spent); + return budget; + } + + [Fact] + public void Apply_ABudgetAlreadySpent_WithholdsTheDocumentWholeAndEveryValue() + { + // #5459: the other half of the pinned budget. Past the budget a value is withheld unjudged, never passed through. + StatementFilterWarmUp.Ensure(); + var report = ReportXml().Replace( + "", + string.Concat(Enumerable.Repeat("" + new string('x', 400) + "", 10_000)) + "", + StringComparison.Ordinal); + var context = new AlertContext { AttachmentXml = report }; + var item = new AlertDetailItem { Heading = "Plain" }; + item.Fields.Add(("Query Text", StatementScrubCanary.PlainStatement)); + context.Details.Add(item); + + SensitiveStatements.JudgeBudget? made = null; + using (AlertStatementFilter.UseBudget(() => made = SpentBudget())) + { + var filtered = AlertStatementFilter.Apply(context)!; + + Assert.Equal(Marker, filtered.AttachmentXml); + var field = Assert.Single(Assert.Single(filtered.Details).Fields); + Assert.Equal(Marker, field.Value); + Assert.DoesNotContain(StatementScrubCanary.PlainStatement, string.Join(Environment.NewLine, Everything(filtered)), StringComparison.Ordinal); + } + + Assert.NotNull(made); + Assert.True(made!.Unjudged > 0, "nothing was counted unjudged"); + Assert.Equal(0, made.Named); + } + + [Fact] + public void ApplyFinding_ABudgetAlreadySpent_WithholdsThePlainProse() + { + StatementFilterWarmUp.Ensure(); + var alert = new FindingAlert("Analysis: High CPU", "SRV", "1", "1", "101", new AlertContext(), 0.9, 0.5, "plain prose", true); + + using (AlertStatementFilter.UseBudget(SpentBudget)) + { + Assert.Equal(Marker, AlertStatementFilter.Apply(alert).DetailText); + } + + Assert.Equal("plain prose", AlertStatementFilter.Apply(alert).DetailText); + } + + /// A report with a 4 MB tail of harmless elements, made different per + /// so no two share a string (#5477). + private static string BigReport(int distinct) => + ReportXml().Replace( + "", + string.Concat(Enumerable.Repeat("" + distinct.ToString(System.Globalization.CultureInfo.InvariantCulture) + new string('x', 399) + "", 10_000)) + + "", + StringComparison.Ordinal); + + private static AlertEngineTests.Harness ThreeIncidentHarness() + { + var h = new AlertEngineTests.Harness(); + h.Settings.BlockingEnabled = true; + for (var i = 0; i < 3; i++) + { + var row = AlertEngineTests.BlockingRow(55 + i); + row.ContentiousObject = "StackOverflow.dbo.Table" + i.ToString(System.Globalization.CultureInfo.InvariantCulture); + row.BlockedSqlText = StatementScrubCanary.CanaryStatement; + row.BlockingSqlText = StatementScrubCanary.PlainStatement; + row.BlockedProcessReportXml = BigReport(i); + h.Adapter.Blocking.Add(row); + } + + return h; + } + + [Fact] + public async Task Engine_ThreeIncidentsWithDifferentFourMegabyteReports_EachReachesTheDelivererFilteredNotWithheldWhole() + { + // #5477: the budget grows with the distinct documents it judges, so the third report is not starved by the first two. + await StatementFilterWarmUp.EnsureAsync(); + using var pinned = UnspendableBudget(); + var h = ThreeIncidentHarness(); + + await h.Build().EvaluateServerAsync(AlertEngineTests.Harness.Snapshot()); + + var outcome = Assert.Single(h.Deliverer.Outcomes); + AssertNoSecret(Everything(outcome)); + var incidents = outcome.Context!.Incidents!; + Assert.Equal(3, incidents.Count); + foreach (var incident in incidents) + { + var xml = incident.Attachment!.Xml; + Assert.True(xml.Length > 4_000_000, "a report was withheld whole: " + xml.Length); + Assert.Contains(StatementScrubCanary.PlainStatement, xml, StringComparison.Ordinal); + Assert.Contains(Marker, xml, StringComparison.Ordinal); + } + } + + [Fact] + public void ApplyCore_ThreeIncidentsWithOneBlockingAlertContext_JudgesEachDistinctReportOnce() + { + // The engine's own context: the alert's attachment is the first incident's report (the same string). + StatementFilterWarmUp.Ensure(); + var h = ThreeIncidentHarness(); + var context = AlertContextBuilders.BuildBlockingContext("SRV", h.Adapter.Blocking, Array.Empty())!; + Assert.Equal(3, context.Incidents!.Count); + Assert.Same(context.AttachmentXml, context.Incidents[0].Attachment!.Xml); + + // #5459: the filtered result is read under a budget that cannot run out (a slow runner withheld the third report). + var pinned = new SensitiveStatements.JudgeBudget(TimeSpan.FromHours(1)); + var filtered = AlertStatementFilter.ApplyCore(context, pinned)!; + Assert.Equal(3, filtered.Incidents!.Count(i => i.Attachment!.Xml.Length > 4_000_000)); + + // The accounting is read off a budget that starts where production's does. Earning and the memo do not depend on + // whether the time ran out, so a slow runner changes neither number. + var budget = new SensitiveStatements.JudgeBudget(SensitiveStatements.ReadBudget); + AlertStatementFilter.ApplyCore(context, budget); + + // Three distinct reports: the alert-level copy of the first one was a memo hit, not a second walk. + Assert.Equal(3, budget.DocumentPasses); + Assert.Equal(3, pinned.DocumentPasses); + // And the budget earned 0.5 s per MB of each of them (a little under 2.0 s each, so about 7.4 s), not 1.5 s in all. + Assert.InRange(budget.Limit, TimeSpan.FromSeconds(7.25), TimeSpan.FromSeconds(8.5)); + } + + /// A budget whose clock moves on every read, so each judge call is charged exactly + /// however fast or slow the machine is (#5459). The real judge still runs. + private static SensitiveStatements.JudgeBudget SteppingBudget(TimeSpan step) + { + var now = TimeSpan.Zero; + return new SensitiveStatements.JudgeBudget(SensitiveStatements.ReadBudget, clock: () => now += step); + } + + [Fact] + public void ApplyCore_ThreeIncidents_ABudgetThatRunsOutMidway_WithholdsWholeAndKeepsTheAccounting() + { + // #5459: the cause of the old "expected 3, actual 2". The filter judges under a wall-clock budget, and a runner + // slow enough to spend it withholds a report whole. Here the clock charges 50 ms per judge call, so the budget + // (1.5 s plus about 2 s a report, against about 10,000 values a report) is spent part of the way in, on any + // machine. A report the budget was spent on comes back as the marker, nothing named leaks, and the numbers the + // test above reads (passes, limit) are the same whether the time ran out or not. + StatementFilterWarmUp.Ensure(); + var h = ThreeIncidentHarness(); + var context = AlertContextBuilders.BuildBlockingContext("SRV", h.Adapter.Blocking, Array.Empty())!; + + var budget = SteppingBudget(TimeSpan.FromMilliseconds(50)); + var filtered = AlertStatementFilter.ApplyCore(context, budget)!; + + Assert.True(budget.Spent, "the budget did not run out"); + Assert.True(filtered.Incidents!.Count(i => i.Attachment!.Xml.Length > 4_000_000) < 3, "no report was withheld"); + Assert.Contains(filtered.Incidents!, i => i.Attachment!.Xml == Marker); + AssertNoSecret(string.Join(Environment.NewLine, Everything(filtered))); + Assert.Equal(3, budget.DocumentPasses); + Assert.InRange(budget.Limit, TimeSpan.FromSeconds(7.25), TimeSpan.FromSeconds(8.5)); + } +} diff --git a/Darling/Darling.Tests/StatementFilterAnalysisReadsLiveTests.cs b/Darling/Darling.Tests/StatementFilterAnalysisReadsLiveTests.cs new file mode 100644 index 0000000000..a2cc20314b --- /dev/null +++ b/Darling/Darling.Tests/StatementFilterAnalysisReadsLiveTests.cs @@ -0,0 +1,164 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Text.RegularExpressions; +using System.Threading.Tasks; +using Microsoft.AspNetCore.TestHost; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Analysis; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Notifications; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348, L8: the read-time census for the derived reads. The persisted finding's drill-down JSON (rows 24 and 25: +/// get_analysis_findings, and the stored findings analyze_server answers from), an alert history row's detail +/// and context (row 26) and a mute rule's query-text pattern (row 27) are planted with the canary statement, read RAW through +/// the tool method (which must still hold the canary: the control), and then sent through the host's registered output +/// filter, which must withhold it and keep the plain statement beside it. Rows 28 to 30 (get_analysis_facts, +/// compare_analysis, get_sweep_reports) carry numbers, fact keys and object names, never statement text, and a source +/// scan pins that. +/// +/// #1776 own-store: mints its own scratch database (), never the shared +/// live-postgres collection. In the DarlingOutputFormatEnv collection because the host filter list changes with +/// DARLING_OUTPUT_FORMAT. +/// +[Collection("DarlingOutputFormatEnv")] +public sealed class StatementFilterAnalysisReadsLiveTests : IDisposable +{ + private const int ServerId = -453_301; + private const string ServerName = "SsfAnalysisReadsSrv"; + + public StatementFilterAnalysisReadsLiveTests() => + Environment.SetEnvironmentVariable("DARLING_OUTPUT_FORMAT", null); + + public void Dispose() => Environment.SetEnvironmentVariable("DARLING_OUTPUT_FORMAT", null); + + private static async Task AssertFilteredAsync(TestServer server, string label, string raw) + { + try + { + StatementFilterCensus.AssertRawHoldsTheCanary(raw); + var (filtered, isError) = await StatementFilterCensus.FilterThroughHostAsync(server, raw); + Assert.False(isError); + StatementFilterCensus.AssertFilteredWithholdsTheCanary(raw, filtered); + } + catch (Exception ex) when (ex is not Xunit.Sdk.XunitException || !ex.Message.StartsWith(label, StringComparison.Ordinal)) + { + throw new Xunit.Sdk.XunitException(label + ": " + ex.Message); + } + } + + [Fact] + public async Task PersistedFindingsAlertHistoryAndMuteRules_WithholdTheCanaryStatement_ThroughTheRegisteredFilter() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live analysis-reads filter test."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(connectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + + var now = DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow).AddMinutes(-5); + + /* Row 25 (and the stored findings behind row 24): a persisted finding whose drill-down JSON carries the statements. */ + var drillDown = new System.Text.Json.Nodes.JsonObject + { + ["top_queries"] = new System.Text.Json.Nodes.JsonArray( + new System.Text.Json.Nodes.JsonObject { ["query_text"] = StatementScrubCanary.CanaryStatement, ["executions"] = 5 }, + new System.Text.Json.Nodes.JsonObject { ["query_text"] = StatementScrubCanary.PlainStatement, ["executions"] = 7 }), + }.ToJsonString(); + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO analysis_findings + (finding_id, analysis_time, server_id, server_name, database_name, + time_range_start, time_range_end, severity, confidence, category, + story_path, story_path_hash, story_text, + root_fact_key, root_fact_value, leaf_fact_key, leaf_fact_value, fact_count, + incident_id, remediation_action_json, drill_down_json) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, NULL, $20)", + CollectionIdGenerator.Next(), now, ServerId, ServerName, "SsfDb", + now.AddHours(-4), now, 0.72, 0.55, "waits", + "HIGH_CPU → TOP_QUERY", "SSFHASH1", "story", "HIGH_CPU", 92.5, "TOP_QUERY", 4.1, 2, "SSF_INCIDENT_1", drillDown); + + /* Row 26: an alert history row, detail text and context both naming the statements. */ + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO config_alert_log + (alert_time, server_id, server_name, metric_name, current_value, threshold_value, + alert_sent, notification_type, send_error, muted, dismissed, detail_text, context_json) +VALUES ($1, $2, $3, 'High CPU', 1, 1, FALSE, 'none', NULL, FALSE, FALSE, $4, $5)", + DarlingMcpTestData.Naive(now), ServerId, ServerName, + "Top statement: " + StatementScrubCanary.CanaryStatement, + "{\"queries\":[{\"query_text\":\"" + StatementScrubCanary.CanaryStatement + "\"}]}"); + + /* Row 27: a mute rule whose pattern is the statement text. */ + await new PgMuteRuleStore(postgres).InsertAsync(new MuteRule + { + Id = "ssf-mute-1", + ServerName = ServerName, + QueryTextPattern = StatementScrubCanary.CanaryStatement, + Reason = "census " + StatementScrubCanary.PlainStatement, + }); + + using var server = await StatementFilterCensus.BuildHostAsync(); + + var findings = await DarlingMcpTools.GetAnalysisFindings( + new DarlingAnalysisService(postgres), postgres, ServerName, hours_back: 24, include_drilldown: true, as_of: null, cancellationToken: ct); + await AssertFilteredAsync(server, "get_analysis_findings", findings); + + var history = await DarlingMcpAlertTools.GetAlertHistory(postgres, ServerName, 24, 50, null, false, ct); + await AssertFilteredAsync(server, "get_alert_history", history); + + var rules = await DarlingMcpAlertTools.GetMuteRules(postgres, true, ct); + await AssertFilteredAsync(server, "get_mute_rules", rules); + } + + /// Rows 28 to 30: the three tools answer from fact keys, numbers, object names and sweep check results. A + /// statement-text column or a drill-down read appearing in one of them fails here until it is judged. + [Theory] + [InlineData("DarlingMcpTools.cs", "GetAnalysisFacts")] + [InlineData("DarlingMcpTools.cs", "CompareAnalysis")] + [InlineData("DarlingMcpFleetSweepTools.cs", "GetSweepReports")] + public void TheFactAndSweepReadsCarryNoStatementTextColumn(string file, string method) + { + var path = System.IO.Path.Combine(RepoRoot(), "Darling", "PerformanceMonitor.Darling.Service", "Mcp", file); + var source = StatementColumnCensusTests.CodeOnly(System.IO.File.ReadAllText(path)); + + var start = source.IndexOf("Task " + method + "(", StringComparison.Ordinal); + Assert.True(start >= 0, method + " is gone from " + file + "; update this census"); + var next = source.IndexOf("[McpServerTool", start, StringComparison.Ordinal); + var body = source.Substring(start, (next < 0 ? source.Length : next) - start); + + var hit = Regex.Match(body, @"query_text|sql_text|statement_text|batch_text|text_data|QueryText|StatementText|query_plan|drill_?down", + RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + Assert.False(hit.Success, method + " now mentions '" + hit.Value + "': judge what it reads, then add a case to this census"); + } + + private static string RepoRoot() + { + var dir = new System.IO.DirectoryInfo(AppContext.BaseDirectory); + while (dir is not null && !System.IO.File.Exists(System.IO.Path.Combine(dir.FullName, "PerformanceMonitor.sln"))) + { + dir = dir.Parent; + } + + return dir?.FullName ?? throw new InvalidOperationException("repository root not found"); + } +} diff --git a/Darling/Darling.Tests/StatementFilterBlockingReadsLiveTests.cs b/Darling/Darling.Tests/StatementFilterBlockingReadsLiveTests.cs new file mode 100644 index 0000000000..3cd05bffd4 --- /dev/null +++ b/Darling/Darling.Tests/StatementFilterBlockingReadsLiveTests.cs @@ -0,0 +1,450 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Globalization; +using System.Linq; +using System.Text; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using System.Xml.Linq; +using Microsoft.AspNetCore.TestHost; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348, rows 16-23 of the read inventory: the blocking, deadlock and Extended Events family. Each case plants +/// text in the rows the tool reads, runs the tool RAW (the control: the answer must +/// still HOLD the canary, so a clean filtered answer proves the filter and not an empty plant) and then sends that answer +/// through the host's output filter (, the instance +/// DarlingMcpHostService registers last), and asserts what a client would receive: no secret needle, the marker +/// where the statement was, and the plain statement beside it unchanged. +/// +/// The tools: get_blocking, get_deadlocks, get_deadlock_detail, get_blocked_process_xml, +/// get_long_query_completions, get_default_trace_events, get_health_parser_significant_waits and +/// get_health_parser_severe_errors. Each answer goes through , the host's whole registered filter list. That +/// host is built with GCF pinned off (#5320), so DARLING_OUTPUT_FORMAT, which classes in the DarlingOutputFormatEnv +/// collection set while this live class may be running, cannot re-encode an answer. The filter's slot in the list is also pinned by +/// SensitiveStatementHostFilterTests. +/// +/// The deadlock graph is the one field with two shapes: at full_graph: true the XML parses and only the named +/// statement text is replaced (precise); at the default the reader judges the whole graph and then cuts it to 2000 characters, so the +/// cut text carries the marker where the statement was and holds no secret (#5320). Five reads (get_blocking, get_deadlocks, +/// get_deadlock_detail, get_long_query_completions, get_default_trace_events) judge in the reader too; the census holds them to the +/// reader's own answer and checks the store still holds the canary. +/// +[Collection("live-postgres")] +public sealed class StatementFilterBlockingReadsLiveTests +{ + private const string ServerName = "darling-ssf-blocking-reads"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + private const string Db = "SsfBlockDb"; + private const string Marker = StatementFilterCensus.Marker; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + /// The tool's answer through the host's whole filter list, as a client receives it. + private static async Task FilteredAsync(TestServer host, string raw) + { + var (text, isError) = await StatementFilterCensus.FilterThroughHostAsync(host, raw); + Assert.False(isError); + return text; + } + + private static void AssertFiltered(string label, string raw, string filtered) + { + try + { + StatementFilterCensus.AssertRawHoldsTheCanary(raw); + StatementFilterCensus.AssertFilteredWithholdsTheCanary(raw, filtered); + using var parsed = JsonDocument.Parse(filtered); + } + catch (Exception ex) + { + throw new Xunit.Sdk.XunitException(label + ": " + ex.Message); + } + } + + /// + /// The census check for a read whose READER judges the whole statement before it cuts it (#5320). The reader's own answer + /// holds the withheld marker and not one secret needle, still parses, and the host filter has nothing left to change (so the + /// reader did the whole job). The control that the plant is real moves from the raw answer to the STORE: the stored column + /// still holds the canary (), so an answer without it is the reader's doing and not an empty plant. + /// + private static void AssertReaderJudged(string label, string raw, string filtered) + { + try + { + foreach (string needle in StatementScrubCanary.SecretNeedles) + { + Assert.DoesNotContain(needle, raw); + } + + Assert.Contains(Marker, raw); + using var parsed = JsonDocument.Parse(raw); + Assert.True(string.Equals(raw, filtered, StringComparison.Ordinal), "the host filter changed an answer the reader had already judged"); + } + catch (Exception ex) + { + throw new Xunit.Sdk.XunitException(label + ": " + ex.Message); + } + } + + /// True when of holds the canary for this test's server. + private static async Task StoredCanaryAsync(NpgsqlConnection connection, string table, string column, CancellationToken ct) + { + await using var cmd = new NpgsqlCommand( + $"SELECT EXISTS (SELECT 1 FROM {table} WHERE server_id = $1 AND position('S3cret-canary-ssf' IN {column}) > 0)", connection); + cmd.Parameters.AddWithValue(ServerId); + return (bool)(await cmd.ExecuteScalarAsync(ct))!; + } + /// Every string value in that opens with < (an embedded XML document). + private static List EmbeddedXml(string json) + { + var found = new List(); + using var doc = JsonDocument.Parse(json); + void Walk(JsonElement e) + { + switch (e.ValueKind) + { + case JsonValueKind.String: + var s = e.GetString(); + if (s is not null && s.TrimStart().StartsWith('<')) found.Add(s); + break; + case JsonValueKind.Object: + foreach (var p in e.EnumerateObject()) Walk(p.Value); + break; + case JsonValueKind.Array: + foreach (var i in e.EnumerateArray()) Walk(i); + break; + } + } + + Walk(doc.RootElement); + return found; + } + + private static string DeadlockGraph(int paddingProcesses) + { + var sb = new StringBuilder(); + sb.Append(""); + sb.Append("") + .Append("").Append(StatementScrubCanary.CanaryStatement).Append("") + .Append("").Append(StatementScrubCanary.CanaryStatement).Append(""); + sb.Append("") + .Append("").Append(StatementScrubCanary.PlainStatement).Append("") + .Append("").Append(StatementScrubCanary.PlainStatement).Append(""); + for (int i = 0; i < paddingProcesses; i++) + { + sb.Append("") + .Append("SELECT canary_pad_ssf FROM dbo.pad WHERE id = @c") + .Append("SELECT canary_pad_ssf FROM dbo.pad WHERE id = @c"); + } + + sb.Append("") + .Append(""); + return sb.ToString(); + } + + private static string BlockedProcessReport() => + "" + + "" + + "" + StatementScrubCanary.CanaryStatement + "" + + "" + StatementScrubCanary.CanaryStatement + "" + + "" + + "" + StatementScrubCanary.PlainStatement + ""; + + [Fact] + public async Task TheEightBlockingAndEventReads_AnswerWithTheCanaryStatementWithheld() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live blocking-read filter test."); + var ct = TestContext.Current.CancellationToken; + + await using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + using var host = await StatementFilterCensus.BuildHostAsync(); + + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + var t = DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow.AddMinutes(-9)); + string wait = System.IO.File.ReadAllText(System.IO.Path.Combine(AppContext.BaseDirectory, "Fixtures", "SystemHealth", "wait_info.xml")) + .Replace("SELECT * FROM dbo.big_table WHERE id = @p1;", StatementScrubCanary.CanaryStatement, StringComparison.Ordinal); + string error = System.IO.File.ReadAllText(System.IO.Path.Combine(AppContext.BaseDirectory, "Fixtures", "SystemHealth", "error_reported.xml")) + .Replace("The operating system returned error 21 to SQL Server during a read at offset 0x00000c60000 in file 'D:\\data\\prod.mdf'.", + "The statement failed: " + StatementScrubCanary.CanaryStatement, StringComparison.Ordinal); + Assert.Contains("S3cret-canary-ssf", wait, StringComparison.Ordinal); + Assert.Contains("S3cret-canary-ssf", error, StringComparison.Ordinal); + + /* row 16 and row 19: one blocked-process report with the canary on the blocked side. */ + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO blocked_process_reports + (blocked_report_id, collection_time, server_id, server_name, event_time, wait_time_ms, blocking_spid, blocked_spid, + blocked_ecid, blocking_ecid, blocking_status, database_name, blocked_sql_text, blocking_sql_text, blocked_process_report_xml) +VALUES ($1, $2, $3, $4, $5, 12000, 62, 61, 0, 0, 'suspended', $6, $7, $8, $9)", + CollectionIdGenerator.Next(), t.AddSeconds(5), ServerId, ServerName, t, Db, + StatementScrubCanary.CanaryStatement, StatementScrubCanary.PlainStatement, BlockedProcessReport()); + + /* rows 17 and 18: one deadlock with the canary as the victim's statement and in its graph. */ + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO deadlocks (deadlock_id, collection_time, server_id, server_name, database_name, deadlock_time, victim_process_id, victim_sql_text, deadlock_graph_xml) +VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)", + CollectionIdGenerator.Next(), t.AddSeconds(5), ServerId, ServerName, Db, t, "process1a", StatementScrubCanary.CanaryStatement, DeadlockGraph(0)); + + /* row 20 */ + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO long_query_completions (long_query_completion_id, collection_time, server_id, server_name, event_time, event_type, database_name, duration_microseconds, statement_text) +VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)", + CollectionIdGenerator.Next(), t.AddSeconds(5), ServerId, ServerName, t, "rpc_completed", Db, 5_000_000L, StatementScrubCanary.CanaryStatement); + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO long_query_completions (long_query_completion_id, collection_time, server_id, server_name, event_time, event_type, database_name, duration_microseconds, statement_text) +VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)", + CollectionIdGenerator.Next(), t.AddSeconds(5), ServerId, ServerName, t, "rpc_completed", Db, 4_000_000L, StatementScrubCanary.PlainStatement); + + /* row 21: a severe ErrorLog entry whose text_data is the canary, and a plain one beside it. */ + foreach (var (text, offset) in new[] { (StatementScrubCanary.CanaryStatement, 0), (StatementScrubCanary.PlainStatement, 1) }) + { + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO default_trace_events + (default_trace_event_id, collection_time, server_id, server_name, event_time, event_name, severity, text_data) +VALUES ($1,$2,$3,$4,$5,$6,$7,$8)", + CollectionIdGenerator.Next(), t.AddSeconds(5), ServerId, ServerName, t.AddSeconds(offset), "ErrorLog", 20, text); + } + + /* rows 22 and 23 */ + foreach (var (xml, type) in new[] { (wait, SystemHealthParser.WaitInfoEvent), (error, SystemHealthParser.ErrorReportedEvent) }) + { + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO system_health_events + (system_health_event_id, collection_time, server_id, server_name, event_time, event_type, event_xml) +VALUES ($1, $2, $3, $4, $5, $6, $7)", + CollectionIdGenerator.Next(), t.AddSeconds(5), ServerId, ServerName, t, type, xml); + } + + /* Every row is checked before the test fails, so one run names each row that leaks. */ + var failures = new List(); + /* A row whose reader judges the statement itself (#5320: the cut-and-judge reads) names its stored column in + `readerJudges`; the store must still hold the canary there, and the reader's own answer is held to the marker. The + other rows return the stored text raw and the host filter is what withholds it. */ + async Task Check(string label, string raw, Action? more = null, (string Table, string Column)? readerJudges = null) + { + try + { + string filtered = await FilteredAsync(host, raw); + if (readerJudges is { } stored) + { + Assert.True(await StoredCanaryAsync(connection, stored.Table, stored.Column, ct), "the stored " + stored.Table + "." + stored.Column + " lost the canary, so the plant never reached the store"); + AssertReaderJudged(label, raw, filtered); + } + else + { + AssertFiltered(label, raw, filtered); + } + more?.Invoke(filtered); + } + catch (Exception ex) + { + failures.Add(label + ": " + ex.Message.Split('\n')[0]); + } + } + + await Check("16 get_blocking", await DarlingMcpBlockingTools.GetBlocking(postgres, ServerName, cancellationToken: ct), readerJudges: ("blocked_process_reports", "blocked_sql_text")); + await Check("17 get_deadlocks", await DarlingMcpBlockingTools.GetDeadlocks(postgres, ServerName, cancellationToken: ct), readerJudges: ("deadlocks", "victim_sql_text")); + await Check("18 get_deadlock_detail", await DarlingMcpBlockingTools.GetDeadlockDetail(postgres, ServerName, cancellationToken: ct), readerJudges: ("deadlocks", "victim_sql_text")); + + /* row 19: the report XML still parses, the canary's inputbuf and frame hold the marker, and the plain side is intact. */ + await Check("19 get_blocked_process_xml", await DarlingMcpBlockingTools.GetBlockedProcessXml(postgres, ServerName, cancellationToken: ct), filtered => + { + var reports = EmbeddedXml(filtered); + Assert.Single(reports); + var bprDoc = XDocument.Parse(reports[0]); + var inputbufs = bprDoc.Descendants("inputbuf").Select(e => e.Value).ToList(); + Assert.Equal(new[] { Marker, StatementScrubCanary.PlainStatement }, inputbufs); + Assert.DoesNotContain(bprDoc.Descendants("frame"), f => f.Value.Contains("S3cret", StringComparison.Ordinal)); + }); + + await Check("20 get_long_query_completions", await DarlingMcpLongQueryTools.GetLongQueryCompletions(postgres, ServerName, cancellationToken: ct), readerJudges: ("long_query_completions", "statement_text")); + await Check("21 get_default_trace_events", await DarlingMcpDefaultTraceTools.GetDefaultTraceEvents(postgres, ServerName, cancellationToken: ct), readerJudges: ("default_trace_events", "text_data")); + await Check("22 get_health_parser_significant_waits", await DarlingMcpHealthParserTools.GetSignificantWaits(postgres, ServerName, cancellationToken: ct)); + await Check("23 get_health_parser_severe_errors", await DarlingMcpHealthParserTools.GetSevereErrors(postgres, ServerName, cancellationToken: ct)); + Assert.True(failures.Count == 0, string.Join(" | ", failures)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// + /// get_deadlock_detail's graph, both ways. With full_graph true the answer carries the whole graph: it parses, + /// the victim's frame and inputbuf hold the marker, and the other process, its statement and the lock resource are the + /// stored ones (precise). By default the reader judges the whole graph (#5320) and then cuts it to 2000 characters: the cut graph holds the marker where + /// the victim's statement was, keeps the other process's statement, says it is truncated, and the host filter has nothing left to change. + /// + [Fact] + public async Task GetDeadlockDetail_FullGraphIsPrecise_AndTheDefaultCutGraphIsJudgedWholeThenCut() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live blocking-read filter test."); + var ct = TestContext.Current.CancellationToken; + + await using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + using var host = await StatementFilterCensus.BuildHostAsync(); + + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + var t = DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow.AddMinutes(-9)); + /* 20 padding processes put the graph well past the 2000-character preview; the canary processes come first, so it sits inside the cut. */ + string graph = DeadlockGraph(20); + Assert.True(graph.Length > 2500, "the graph is " + graph.Length + " characters"); + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO deadlocks (deadlock_id, collection_time, server_id, server_name, database_name, deadlock_time, victim_process_id, victim_sql_text, deadlock_graph_xml) +VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)", + CollectionIdGenerator.Next(), t.AddSeconds(5), ServerId, ServerName, Db, t, "process1a", StatementScrubCanary.CanaryStatement, graph); + + string full = await DarlingMcpBlockingTools.GetDeadlockDetail(postgres, ServerName, full_graph: true, cancellationToken: ct); + string fullFiltered = await FilteredAsync(host, full); + AssertFiltered("full_graph true", full, fullFiltered); + var fullGraphs = EmbeddedXml(fullFiltered); + Assert.Single(fullGraphs); + var doc = XDocument.Parse(fullGraphs[0]); + var victim = doc.Descendants("process").Single(p => (string?)p.Attribute("id") == "process1a"); + Assert.Equal(Marker, victim.Element("inputbuf")!.Value); + Assert.Equal(Marker, victim.Descendants("frame").Single().Value); + var other = doc.Descendants("process").Single(p => (string?)p.Attribute("id") == "process2b"); + Assert.Equal(StatementScrubCanary.PlainStatement, other.Element("inputbuf")!.Value); + Assert.Equal(StatementScrubCanary.PlainStatement, other.Descendants("frame").Single().Value); + Assert.Equal(22, doc.Descendants("process").Count()); + Assert.Equal(Db + ".dbo.t", (string?)doc.Descendants("keylock").Single().Attribute("objectname")); + + /* The default form (#5320): the reader judges the WHOLE graph, then cuts it to 2000 characters, so its own answer is + already safe. The stored graph still holds the canary (the plant is real, and is past the cut length), the answer's + graph carries the marker where the victim's statement was, keeps the other process's statement, is cut and says so, + and the host filter finds nothing left to change. */ + Assert.True(await StoredCanaryAsync(connection, "deadlocks", "deadlock_graph_xml", ct), "the stored graph lost the canary, so the plant never reached the store"); + string cut = await DarlingMcpBlockingTools.GetDeadlockDetail(postgres, ServerName, cancellationToken: ct); + string cutFiltered = await FilteredAsync(host, cut); + AssertReaderJudged("full_graph false", cut, cutFiltered); + using var cutDoc = JsonDocument.Parse(cut); + var cutRow = cutDoc.RootElement.GetProperty("deadlocks")[0]; + string cutGraph = cutRow.GetProperty("deadlock_graph_xml").GetString()!; + Assert.True(cutRow.GetProperty("deadlock_graph_xml_truncated").GetBoolean()); + Assert.EndsWith("... (truncated)", cutGraph); + Assert.True(cutGraph.Length > 1500 && cutGraph.Length < graph.Length, "the cut graph is " + cutGraph.Length + " characters of " + graph.Length); + Assert.Contains("" + Marker + "", cutGraph); + Assert.Contains("" + Marker + "", cutGraph); + Assert.Contains("" + StatementScrubCanary.PlainStatement + "", cutGraph); + Assert.Equal("process1a", cutRow.GetProperty("victim_process_id").GetString()); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// + /// The filter's cost on get_deadlock_detail at full_graph: true over a graph of about 4 MB (the answer carries the + /// graph and the per-process rows built from it). The ceiling is loose (a slow runner is not a failure); it catches a read that + /// spends the filter's whole budget and so answers with every value withheld. The answer must still hold the precise shape. + /// Set SSF_L6_TIMINGS=1 to fail with the measured milliseconds. + /// + [Fact] + public async Task GetDeadlockDetail_OnAFourMegabyteGraph_StaysPreciseInsideTheCeiling() + { + await StatementFilterWarmUp.EnsureAsync(); + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live blocking-read filter test."); + var ct = TestContext.Current.CancellationToken; + + await using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + using var host = await StatementFilterCensus.BuildHostAsync(); + + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + var t = DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow.AddMinutes(-9)); + string graph = DeadlockGraph(14_000); + Assert.InRange(graph.Length, 4_000_000, 5_500_000); + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO deadlocks (deadlock_id, collection_time, server_id, server_name, database_name, deadlock_time, victim_process_id, victim_sql_text, deadlock_graph_xml) +VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)", + CollectionIdGenerator.Next(), t.AddSeconds(5), ServerId, ServerName, Db, t, "process1a", StatementScrubCanary.CanaryStatement, graph); + + var read = Stopwatch.StartNew(); + string raw = await DarlingMcpBlockingTools.GetDeadlockDetail(postgres, ServerName, full_graph: true, cancellationToken: ct); + read.Stop(); + var filter = Stopwatch.StartNew(); + string filtered = await FilteredAsync(host, raw); + filter.Stop(); + string timings = string.Create(CultureInfo.InvariantCulture, + $"graph {graph.Length} chars, answer {raw.Length} chars: tool {read.ElapsedMilliseconds} ms, filter {filter.ElapsedMilliseconds} ms"); + + Assert.True(filter.Elapsed < TimeSpan.FromSeconds(10), timings); + AssertFiltered("4 MB graph", raw, filtered); + var graphs = EmbeddedXml(filtered); + Assert.Single(graphs); + var doc = XDocument.Parse(graphs[0]); + Assert.Equal(Marker, doc.Descendants("process").First(p => (string?)p.Attribute("id") == "process1a").Element("inputbuf")!.Value); + Assert.Equal(StatementScrubCanary.PlainStatement, + doc.Descendants("process").First(p => (string?)p.Attribute("id") == "process2b").Element("inputbuf")!.Value); + Assert.True(doc.Descendants("process").Count() >= 14_000, timings); + + if (Environment.GetEnvironmentVariable("SSF_L6_TIMINGS") == "1") + Assert.Fail(timings); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + private static async Task DeleteRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + foreach (var table in new[] + { + "blocked_process_reports", "deadlocks", "long_query_completions", "default_trace_events", "system_health_events", "servers", + }) + { + await DarlingMcpTestData.ExecAsync(connection, ct, $"DELETE FROM {table} WHERE server_id = $1", ServerId); + } + } +} diff --git a/Darling/Darling.Tests/StatementFilterCensus.cs b/Darling/Darling.Tests/StatementFilterCensus.cs new file mode 100644 index 0000000000..5598524744 --- /dev/null +++ b/Darling/Darling.Tests/StatementFilterCensus.cs @@ -0,0 +1,257 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.ComponentModel; +using System.Linq; +using System.Net; +using System.Net.Http; +using System.Text; +using System.Text.Json; +using System.Text.Json.Nodes; +using System.Threading.Tasks; +using System.Xml.Linq; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.TestHost; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using ModelContextProtocol; +using ModelContextProtocol.Protocol; +using ModelContextProtocol.Server; +using Npgsql; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Analysis; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348: the read-time census harness. A read's census plants values in the +/// read's source rows, runs the read RAW (the tool method alone, which must still HOLD the canary: the control that +/// proves the plant reached the output) and FILTERED (the same text sent through the host's REAL registered +/// call-tool filter list, with or without DARLING_OUTPUT_FORMAT=gcf), and then asserts what the filtered +/// text may hold. The harness is read-only for its users: a per-read test calls +/// (or for a tool that needs no store) and the Assert* helpers, and builds +/// no host of its own. +/// +/// The host is the real DarlingMcpHostService.ConfigureMcpServices plus ConfigurePipeline +/// pair behind an in-memory server, with one test-only tool () added after it. The probe +/// answers whatever text a test registered under a key, so the text a real read produced reaches the registered +/// filters exactly as a tool's return value does. Tests that call it and change the environment variable belong to +/// the DarlingOutputFormatEnv collection. +/// +internal static class StatementFilterCensus +{ + /// The statement filter's marker, the only thing a withheld value may be replaced by. + public const string Marker = SensitiveStatements.PlaceholderText; + + private const string ProbeName = "ssf_census_probe"; + + private static readonly ConcurrentDictionary s_payloads = new(); + + /// The one test-only tool, registered ONLY on a census host (never in production's + /// ConfigureMcpServices). It answers the payload registered under key, as a text block, the way a + /// tool that returned that string would, error flag included. + [McpServerToolType] + public sealed class ProbeTools + { + [McpServerTool(Name = ProbeName), Description("Test-only: answers the text a census test registered.")] + public static CallToolResult Probe([Description("The payload key.")] string key) + { + var (text, isError) = s_payloads[key]; + return new CallToolResult + { + Content = new List { new TextContentBlock { Text = text } }, + IsError = isError ? true : null, + }; + } + } + + /// Test-only: a tool that THROWS an McpException carrying message, the way a tool that + /// reports a failure through the SDK's exception does. Its text reaches the client through the SDK's own error + /// result, so a filter has to see the exception to sweep it. + [McpServerToolType] + public sealed class ThrowProbeTools + { + [McpServerTool(Name = ThrowProbeName), Description("Test-only: throws an McpException with the given message.")] + public static CallToolResult Throw([Description("The exception message.")] string message) => + throw new McpException(message); + } + + public const string ThrowProbeName = "ssf_census_throw"; + + /// A real host: the production tool and filter registration plus the probe tool, behind an in-memory + /// server. Dispose it with the test. + /// pins the host's GCF format (#5320): false (the default) is plain output and + /// true is GCF on. Neither reads DARLING_OUTPUT_FORMAT, so a class in the DarlingOutputFormatEnv + /// collection that sets it cannot reach this host. null keeps the production default (the environment variable, + /// read on each call), for the test that pins the real wiring. + public static async Task BuildHostAsync(bool? gcf = false) + { + var builder = WebApplication.CreateBuilder(); + builder.WebHost.UseTestServer(); + builder.Logging.ClearProviders(); + + var postgres = NpgsqlDataSource.Create("Host=localhost;Database=postgres;Username=darling"); + builder.Services.AddSingleton(postgres); + if (gcf is bool pinned) + { + builder.Services.AddSingleton(new FixedGcfOutputFormat(pinned)); + } + + var sharedBaselines = new BaselineCache(); + builder.Services.AddTransient(_ => new DarlingAnalysisService( + postgres, planFetcher: null, logger: Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance, + baselineCache: sharedBaselines)); + builder.Services.AddSingleton( + Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance); + + DarlingMcpHostService.ConfigureMcpServices( + builder.Services, DarlingPeerDirectory.Snapshot.Empty, new ReadLatencyAccumulator(), + Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance, null); + + // Chains onto the SAME builder ConfigureMcpServices created, so every real tool and every real filter stays. + builder.Services.AddMcpServer().WithTools().WithTools(); + + var app = builder.Build(); + var host = new DarlingMcpHostService( + Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance, + new McpRuntimeState(), + new MonitoredServerRegistryState()); + host.ConfigurePipeline( + app, networkMode: false, networkListenIp: null, + allowedCidr: IPNetwork.Parse("127.0.0.1/32"), bearerToken: "ssf-census-token"); + + await app.StartAsync(); + return app.GetTestServer(); + } + + /// Sends through the host's registered filters as a tool's result and + /// returns what a client would receive. + public static async Task<(string Text, bool IsError)> FilterThroughHostAsync( + TestServer server, string rawToolText, bool isError = false, string path = "/") + { + string key = Guid.NewGuid().ToString("N"); + s_payloads[key] = (rawToolText, isError); + try + { + return await CallRealToolAsync(server, ProbeName, new JsonObject { ["key"] = key }, path); + } + finally + { + s_payloads.TryRemove(key, out _); + } + } + + /// A tools/call for a real tool, through the host's registered filters. Returns the first text + /// block and the error flag; throws when the server answered a protocol error, so a test fails loudly instead of + /// reading an empty answer as a clean one. + public static async Task<(string Text, bool IsError)> CallRealToolAsync( + TestServer server, string toolName, JsonObject arguments, string path = "/") + { + string body = new JsonObject + { + ["jsonrpc"] = "2.0", + ["id"] = 1, + ["method"] = "tools/call", + ["params"] = new JsonObject { ["name"] = toolName, ["arguments"] = arguments }, + }.ToJsonString(); + + JsonElement root = (await RpcAsync(server, path, body)).RootElement; + if (root.TryGetProperty("error", out var error)) + { + throw new InvalidOperationException("The server answered a protocol error: " + error.GetRawText()); + } + + var result = root.GetProperty("result"); + bool isError = result.TryGetProperty("isError", out var flag) && flag.ValueKind == JsonValueKind.True; + string text = result.GetProperty("content")[0].GetProperty("text").GetString() ?? ""; + return (text, isError); + } + + /// The names tools/list advertises on . + public static async Task> ListToolNamesAsync(TestServer server, string path = "/") + { + using var doc = await RpcAsync(server, path, "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/list\"}"); + return doc.RootElement.GetProperty("result").GetProperty("tools").EnumerateArray() + .Select(t => t.GetProperty("name").GetString()!).ToList(); + } + + private static async Task RpcAsync(TestServer server, string path, string requestBody) + { + var ctx = await server.SendAsync(c => + { + c.Request.Method = "POST"; + c.Request.Path = path; + c.Request.Headers.Host = "localhost"; + c.Request.Headers.Accept = "application/json, text/event-stream"; + c.Request.ContentType = "application/json"; + var bytes = Encoding.UTF8.GetBytes(requestBody); + c.Request.Body = new System.IO.MemoryStream(bytes); + c.Request.ContentLength = bytes.Length; + c.Connection.RemoteIpAddress = IPAddress.Loopback; + }); + + using var reader = new System.IO.StreamReader(ctx.Response.Body); + string raw = await reader.ReadToEndAsync(); + Assert.Equal(200, ctx.Response.StatusCode); + + // A stateless answer is one SSE event ("data: {json}") or a bare JSON body, depending on the Accept match. + string json = raw.TrimStart().StartsWith('{') + ? raw + : string.Join("\n", raw.Split('\n').Where(l => l.StartsWith("data:", StringComparison.Ordinal)) + .Select(l => l.Substring(5).Trim())); + return JsonDocument.Parse(json); + } + + // ── the census assertions ── + + /// The control: the RAW read holds the canary, so a filtered answer that does not proves the filter and + /// not an empty plant. + public static void AssertRawHoldsTheCanary(string raw) + { + Assert.Contains("S3cret-canary-ssf", raw); + } + + /// The FILTERED answer of a read: no secret needle, the marker present, and every kept needle the raw + /// text held still there, unchanged. + public static void AssertFilteredWithholdsTheCanary(string raw, string filtered) + { + foreach (string needle in StatementScrubCanary.SecretNeedles) + { + Assert.DoesNotContain(needle, filtered); + } + + Assert.Contains(Marker, filtered); + foreach (string kept in StatementScrubCanary.KeptNeedles.Where(k => raw.Contains(k, StringComparison.Ordinal))) + { + Assert.Contains(kept, filtered); + } + } + + /// The filtered text of a plan read in XML form: it still parses, statement 1 is the marker, and + /// statements 2 and 3 keep their text, operators and parameters. + public static void AssertPlanFilteredKeepsTheRest(string rawPlan, string filteredPlan) + { + AssertFilteredWithholdsTheCanary(rawPlan, filteredPlan); + var statements = XDocument.Parse(filteredPlan).Descendants().Where(e => e.Name.LocalName == "StmtSimple") + .Select(e => (string?)e.Attribute("StatementText")).ToList(); + Assert.Equal(Marker, statements[0]); + Assert.Equal(StatementScrubCanary.PlainStatement, statements[1]); + Assert.Equal(StatementScrubCanary.AutoParamStatement, statements[2]); + Assert.Contains( + XDocument.Parse(filteredPlan).Descendants(), + e => (string?)e.Attribute("ParameterCompiledValue") == "N'param-canary-ssf'"); + } +} diff --git a/Darling/Darling.Tests/StatementFilterDeliveryTests.cs b/Darling/Darling.Tests/StatementFilterDeliveryTests.cs new file mode 100644 index 0000000000..667bead9c7 --- /dev/null +++ b/Darling/Darling.Tests/StatementFilterDeliveryTests.cs @@ -0,0 +1,434 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging.Abstractions; +using PerformanceMonitor.Alerting; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Notifications; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5320 (part of #4348), the delivery choke point: DarlingAlertDeliverer.DeliverAndReportAsync runs the statement +/// filter, so the eight callers that hand an outcome straight to it (the PostgreSQL families in DarlingWorker, a +/// self alert, a custom alert rule) are filtered like an engine alert. These run the REAL deliverer over a recording +/// history store, so what is asserted is what an operator would be sent and what the history row stores. +/// +public sealed class StatementFilterDeliveryTests +{ + private const string Marker = SensitiveStatements.PlaceholderText; + + private const string DefinitionJson = + "{\"metric\":{\"source\":\"wait_stats\",\"measure\":\"wait_time_ms\",\"aggregate\":\"sum\",\"hours\":1}," + + "\"predicate\":{\"op\":\"ge\",\"warnThreshold\":1000}}"; + + private static (DarlingAlertDeliverer Deliverer, RecordingHistoryStore History) Build(DarlingConfig? config = null) + { + var settings = new DarlingAlertSettings(config ?? new DarlingConfig()); + var history = new RecordingHistoryStore(); + var webhooks = new WebhookAlertService( + settings, DarlingAlertDeliverer.Branding, NullLogger.Instance, history); + return (new DarlingAlertDeliverer(settings, history, webhooks, NullLogger.Instance), history); + } + + private static string RowText(AlertHistoryRecord record) => + string.Join("\n", new[] { record.CurrentValueText, record.ThresholdValueText, record.DetailText, record.ContextJson } + .Where(p => p is not null)); + + private static void AssertNoSecret(string text) + { + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.DoesNotContain(needle, text, StringComparison.Ordinal); + } + } + + /// The shape a PostgreSQL long-running-query family builds: one detail item with the query text. + private static AlertOutcome PgFamilyOutcome(string queryText) + { + var context = new AlertContext(); + var item = new AlertDetailItem { Heading = "Long-running query" }; + item.Fields.Add(("Database", "stackoverflow")); + item.Fields.Add(("Query", queryText)); + context.Details.Add(item); + return new AlertOutcome( + "pg1", "pg-server", "PG Long-Running Query", "45", "30", context, + AlertContextBuilders.ContextToDetailText(context), 45, 30, false, null, + ShortMessage: "Session 71 running 45m - " + queryText); + } + + private static LongRunningQueryInfo LongRunning(string queryText) => new() + { + SessionId = 71, + DatabaseName = "StackOverflow", + QueryText = queryText, + ElapsedSeconds = 45 * 60, + CpuTimeMs = 1000, + QueryHash = "0x9AAF0129E4E9AD07", + }; + + [Fact] + public async Task APostgreSqlFamilyAlert_HandedStraightToTheDeliverer_IsDeliveredWithTheStatementWithheld() + { + var (deliverer, history) = Build(); + + await deliverer.DeliverAndReportAsync(PgFamilyOutcome(StatementScrubCanary.CanaryStatement), TestContext.Current.CancellationToken); + + var row = Assert.Single(history.Records); + var text = RowText(row); + AssertNoSecret(text); + Assert.Contains(Marker, text, StringComparison.Ordinal); + Assert.Contains("stackoverflow", text, StringComparison.Ordinal); + } + + [Fact] + public async Task ASelfAlert_WhoseProseNamesAStatement_IsDeliveredWithTheMarker() + { + var (deliverer, history) = Build(); + var outcome = new AlertOutcome( + "1", "srv", "Collection Failing", "3", "1", Context: null, + DetailText: "Collector failed on: " + StatementScrubCanary.CanaryStatement, + NumericCurrentValue: 3, NumericThresholdValue: 1, Muted: false, Severity: null, + ShortMessage: "Collector failed on: " + StatementScrubCanary.CanaryStatement); + + await deliverer.DeliverAndReportAsync(outcome, TestContext.Current.CancellationToken); + + var text = RowText(Assert.Single(history.Records)); + AssertNoSecret(text); + Assert.Contains(Marker, text, StringComparison.Ordinal); + } + + [Fact] + public async Task ACustomRuleAlert_WhoseNameAndDescriptionNameAStatement_IsDeliveredWithTheMarker() + { + var (definition, error) = CustomAlertRuleDefinition.TryParse(DefinitionJson); + Assert.Null(error); + var rule = new CustomAlertRule( + 7, StatementScrubCanary.CanaryStatement, DefinitionJson, StatementScrubCanary.CanaryStatement, + Enabled: true, Version: 1, CreatedAt: DateTime.UtcNow, UpdatedAt: DateTime.UtcNow, UpdatedBy: null); + var outcome = CustomAlertEvaluator.BuildFireOutcome( + rule, definition!, serverKey: "7", serverDisplayName: "srv", value: 1500, + severity: AlertSeverityLevel.Warning, muted: false); + Assert.Contains("S3cret-canary-ssf", outcome.DetailText!, StringComparison.Ordinal); + var (deliverer, history) = Build(); + + await deliverer.DeliverAndReportAsync(outcome, TestContext.Current.CancellationToken); + + var row = Assert.Single(history.Records); + AssertNoSecret(RowText(row)); + Assert.Contains(Marker, row.DetailText!, StringComparison.Ordinal); + var filtered = AlertStatementFilter.Apply(outcome); + AssertNoSecret(filtered.DisplayName!); + } + + [Fact] + public async Task APerEventSplit_OfADirectlyDeliveredAlert_IsFilteredToo() + { + var config = new DarlingConfig(); + config.Alerts.DeliveryMode = AlertNotificationMode.PerEvent; + var (deliverer, history) = Build(config); + var context = new AlertContext(); + AlertIncidentRenderer.Apply(context, new List + { + new("k1", new[] { "dbo.t" }, DetailFields: new[] { new AlertIncidentField("Victim SQL", StatementScrubCanary.CanaryStatement) }), + new("k2", new[] { "dbo.u" }, DetailFields: new[] { new AlertIncidentField("Victim SQL", StatementScrubCanary.CanaryStatement) }), + }); + var outcome = new AlertOutcome( + "pg1", "pg-server", "PG Blocking Detected", "2", "1", context, "detail", 2, 1, false, null); + + await deliverer.DeliverAndReportAsync(outcome, TestContext.Current.CancellationToken); + + Assert.Equal(2, history.Records.Count); + Assert.All(history.Records, r => AssertNoSecret(RowText(r))); + } + + [Fact] + public async Task AnOutcomeTheEngineAlreadyFiltered_IsNotJudgedASecondTime() + { + /* The marker is the whole contract: a marked outcome is trusted as it is, so text planted in one passes + through, and the same text in an unmarked outcome does not. That is what "judged once" means here. */ + var (deliverer, history) = Build(); + var planted = PgFamilyOutcome(StatementScrubCanary.CanaryStatement); + var unmarked = PgFamilyOutcome(StatementScrubCanary.CanaryStatement); + + await deliverer.DeliverAndReportAsync(AlertStatementFilter.MarkJudged(planted), TestContext.Current.CancellationToken); + await deliverer.DeliverAndReportAsync(unmarked, TestContext.Current.CancellationToken); + + Assert.Contains("S3cret-canary-ssf", RowText(history.Records[0]), StringComparison.Ordinal); + AssertNoSecret(RowText(history.Records[1])); + } + + [Fact] + public async Task AWithCopyThatAddsNewText_OfAJudgedOutcome_IsJudgedAgain() + { + /* M2 (#5360): the mark lives in the filter's own table, keyed by instance, so a `with` copy of a judged + outcome is a new instance and its new text is judged. At the base the mark was a record parameter that a + `with` copy kept, so this text reached the deliverer unjudged. */ + var judged = AlertStatementFilter.MarkJudged(AlertStatementFilter.Apply(PgFamilyOutcome(StatementScrubCanary.PlainStatement))); + var edited = judged with { DetailText = "Query: " + StatementScrubCanary.CanaryStatement, ShortMessage = StatementScrubCanary.CanaryStatement }; + + Assert.NotSame(edited, AlertStatementFilter.Apply(edited)); + Assert.DoesNotContain("S3cret-canary-ssf", StatementFilterAlertTests.Everything(AlertStatementFilter.Apply(edited)), StringComparison.Ordinal); + + var (deliverer, history) = Build(); + await deliverer.DeliverAndReportAsync(edited, TestContext.Current.CancellationToken); + AssertNoSecret(RowText(Assert.Single(history.Records))); + } + + [Fact] + public void AnOutcomeACallerBuilt_IsJudged_AndTheFilterHasNoPublicWayToMarkOne() + { + var built = PgFamilyOutcome(StatementScrubCanary.CanaryStatement); + + Assert.NotSame(built, AlertStatementFilter.Apply(built)); + Assert.DoesNotContain("S3cret-canary-ssf", StatementFilterAlertTests.Everything(AlertStatementFilter.Apply(built)), StringComparison.Ordinal); + + /* No public parameter, property or method can say "already filtered" (M2, #5360). */ + Assert.Null(typeof(AlertOutcome).GetProperty("StatementFiltered")); + Assert.Null(typeof(AlertStatementFilter).GetMethod("MarkJudged", System.Reflection.BindingFlags.Public | System.Reflection.BindingFlags.Static)); + } + + [Fact] + public void Apply_SetsTheMarkerOnTheCopyItReturns_AndNeverJudgesAMarkedOutcomeAgain() + { + var named = PgFamilyOutcome(StatementScrubCanary.CanaryStatement); + + var filtered = AlertStatementFilter.Apply(named); + + Assert.NotSame(named, filtered); + Assert.Same(filtered, AlertStatementFilter.Apply(filtered)); + Assert.Same(filtered, AlertStatementFilter.MarkJudged(filtered)); + } + + [Fact] + public void APlainAlert_IsTheSameInstanceThroughTheFilter_AndMarkedOnce() + { + var plain = PgFamilyOutcome(StatementScrubCanary.PlainStatement); + + Assert.Same(plain, AlertStatementFilter.Apply(plain)); + var marked = AlertStatementFilter.MarkJudged(plain); + Assert.Same(marked, AlertStatementFilter.Apply(marked)); + Assert.Same(marked, AlertStatementFilter.MarkJudged(marked)); + Assert.Same(plain.Context, marked.Context); + } + + [Fact] + public async Task APlainAlertThroughTheEngine_ReachesTheDelivererMarked_WithItsContextUntouched() + { + var h = new AlertEngineTests.Harness(); + h.Settings.LongRunningQueryEnabled = true; + h.Adapter.LongRunning.Add(LongRunning(StatementScrubCanary.PlainStatement)); + + await h.Build().EvaluateServerAsync(AlertEngineTests.Harness.Snapshot()); + + var outcome = Assert.Single(h.Deliverer.Outcomes); + Assert.Same(outcome, AlertStatementFilter.Apply(outcome)); + Assert.Contains("canary_plain_ssf", StatementFilterAlertTests.Everything(outcome), StringComparison.Ordinal); + } + + [Fact] + public async Task AnEngineAlert_IsFilteredByTheEngineAndMarked_SoTheRealDelivererStoresItFiltered() + { + var h = new AlertEngineTests.Harness(); + h.Settings.LongRunningQueryEnabled = true; + h.Adapter.LongRunning.Add(LongRunning(StatementScrubCanary.CanaryStatement)); + + await h.Build().EvaluateServerAsync(AlertEngineTests.Harness.Snapshot()); + + var outcome = Assert.Single(h.Deliverer.Outcomes); + Assert.Same(outcome, AlertStatementFilter.Apply(outcome)); + var (deliverer, history) = Build(); + await deliverer.DeliverAndReportAsync(outcome, TestContext.Current.CancellationToken); + AssertNoSecret(RowText(Assert.Single(history.Records))); + } + + private static DarlingPgDeadlockReader.PgDeadlockRow DeadlockRow(string? victimStatement) => + new( + OccurredAtUtc: new DateTime(2026, 8, 31, 0, 0, 0, DateTimeKind.Utc), + VictimPid: 111, + ParticipantCount: 2, + DeadlockHash: "hash-1", + LockModes: null, + Resources: null, + VictimStatement: victimStatement, + TimesSeen: 1); + + private static DarlingPgBlockingReader.PgBlockingChainRow BlockingRow(string? rootQuery) => + new( + CapturedAt: new DateTime(2026, 8, 31, 0, 0, 0, DateTimeKind.Utc), + RootBackendId: 424242, + RootPid: 200, + Databases: new[] { "sales" }, + RootUsername: "app", + RootApplicationName: "webapi", + RootState: "active", + RootQuery: rootQuery, + RootIsIdleInTransaction: false, + RootXactDurationMs: 5000, + RootQueryDurationMs: 5000, + TotalVictims: 3, + DirectVictims: 3, + MaxDepth: 1, + WorstVictimWaitMs: 4000, + WorstVictimQuery: "SELECT 2", + SamplesAsRoot: 1, + QueryTextMayBeTruncated: false, + ChainMayBeTruncated: false); + + /// The outcome the PostgreSQL deadlock and blocking families hand the deliverer: incidents only, no detail text. + private static AlertOutcome PgIncidentOutcome(string metric, params AlertIncident[] incidents) => + new( + "pg1", "pg-server", metric, "1", "1", + Context: new AlertContext { Incidents = incidents.ToList() }, + DetailText: null, NumericCurrentValue: 1, NumericThresholdValue: 1, Muted: false, Severity: null, + ShortMessage: "1 event in the last hour"); + + [Theory] + [InlineData(true, false)] + [InlineData(true, true)] + [InlineData(false, false)] + [InlineData(false, true)] + public async Task APostgreSqlIncident_ThatNamesAStatementAmongItsObjects_IsDeliveredWithTheStatementWithheld(bool blocking, bool perEvent) + { + /* The REAL incident builders (the objects hold the root query or the victim statement), the real deliverer + over a recording history store, a capturing webhook endpoint and a capturing mail endpoint. */ + using var endpoint = new CapturingWebhookEndpoint(); + using var smtp = new CapturingSmtpEndpoint(); + var config = new DarlingConfig(); + config.Webhooks.TeamsUrl = endpoint.Url; + config.Webhooks.SlackUrl = endpoint.Url; + config.Webhooks.GenericUrl = endpoint.Url; + config.Smtp.Host = "127.0.0.1"; + config.Smtp.Port = smtp.Port; + config.Smtp.UseSsl = false; + config.Smtp.From = "monitor@example.invalid"; + config.Smtp.To = "operator@example.invalid"; + if (perEvent) + { + config.Alerts.DeliveryMode = AlertNotificationMode.PerEvent; + } + + var settings = new DarlingAlertSettings(config); + var history = new RecordingHistoryStore(); + var webhooks = new WebhookAlertService( + settings, DarlingAlertDeliverer.Branding, NullLogger.Instance, history); + var deliverer = new DarlingAlertDeliverer(settings, history, webhooks, NullLogger.Instance); + var incident = blocking + ? DarlingWorker.BuildPgBlockingIncident(BlockingRow(StatementScrubCanary.CanaryStatement)) + : DarlingWorker.BuildPgDeadlockIncident(DeadlockRow(StatementScrubCanary.CanaryStatement)); + Assert.Contains("S3cret-canary-ssf", string.Join("\n", incident.InvolvedObjects), StringComparison.Ordinal); + + await deliverer.DeliverAndReportAsync( + PgIncidentOutcome(blocking ? "PG Blocking Detected" : "PG Deadlocks Detected", incident), + TestContext.Current.CancellationToken); + + var row = Assert.Single(history.Records); + Assert.NotEmpty(endpoint.Bodies); + Assert.NotEmpty(smtp.Messages); + AssertNoSecret(RowText(row)); + foreach (var body in endpoint.Bodies) + { + AssertNoSecret(body); + } + + foreach (var message in smtp.Messages) + { + AssertNoSecret(message); + } + + Assert.Contains(Marker, string.Join("\n", endpoint.Bodies), StringComparison.Ordinal); + Assert.Contains(Marker, row.ContextJson!, StringComparison.Ordinal); + } + + [Fact] + public void AFailureWhileJudging_ForAnAlertWithNoDetailText_StillSaysWhyItHasNoDetail() + { + /* A null detail item cannot be read, which forces the failure path. The per-event PostgreSQL families pass + no detail text, so the reason has to ride the context. */ + var context = new AlertContext + { + Incidents = new List { DarlingWorker.BuildPgBlockingIncident(BlockingRow(StatementScrubCanary.CanaryStatement)) }, + }; + context.Details.Add(null!); + var outcome = new AlertOutcome( + "pg1", "pg-server", "PG Blocking Detected", "1", "1", context, + DetailText: null, NumericCurrentValue: 1, NumericThresholdValue: 1, Muted: false, Severity: null); + + var filtered = AlertStatementFilter.Apply(outcome); + + Assert.Contains(Marker, AlertContextBuilders.ContextToDetailText(filtered.Context)!, StringComparison.Ordinal); + AssertNoSecret(StatementFilterAlertTests.Everything(filtered)); + var incident = Assert.Single(filtered.Context!.Incidents!); + Assert.Empty(incident.InvolvedObjects); + Assert.Equal("424242", incident.DedupKey); + } + + [Fact] + public void AFailureWhileJudging_ForAnAlertWithANullIncident_ReturnsTheClearedAlertInsteadOfThrowing() + { + var failing = PgIncidentOutcome( + "PG Blocking Detected", + DarlingWorker.BuildPgBlockingIncident(BlockingRow(StatementScrubCanary.CanaryStatement)), + null!); + failing.Context!.Details.Add(null!); + + var filtered = AlertStatementFilter.Apply(failing); + + Assert.Same(filtered, AlertStatementFilter.Apply(filtered)); // judged: a second Apply returns it unchanged + Assert.Contains(Marker, AlertContextBuilders.ContextToDetailText(filtered.Context)!, StringComparison.Ordinal); + Assert.Equal(2, filtered.Context!.Incidents!.Count); + Assert.Empty(filtered.Context.Incidents[0].InvolvedObjects); + Assert.Null(filtered.Context.Incidents[1]); + } + + [Fact] + public async Task AFailureWhileJudging_ForAnAlertWithNoIncident_StillDeliversTheClearedAlert() + { + var (deliverer, history) = Build(); + var context = new AlertContext(); + context.Details.Add(null!); + var failing = new AlertOutcome( + "1", "srv", "Collection Failing", "3", "1", context, + DetailText: null, NumericCurrentValue: 3, NumericThresholdValue: 1, Muted: false, Severity: null); + + await deliverer.DeliverAndReportAsync(failing, TestContext.Current.CancellationToken); + + var row = Assert.Single(history.Records); + Assert.Contains(Marker, RowText(row), StringComparison.Ordinal); + } + + private sealed class RecordingHistoryStore : IAlertHistoryStore + { + public List Records { get; } = new(); + + public Task RecordAlertAsync(AlertHistoryRecord record) + { + Records.Add(record); + return Task.CompletedTask; + } + + public Task GetLastEmailSentUtcAsync(string serverId, string metricName, string? dedupKey = null) => + Task.FromResult(null); + + public Task GetLastWebhookSentUtcAsync(string serverId, string metricName, string? dedupKey = null) => + Task.FromResult(null); + + public Task GetLastAlertTimeAsync(string serverId, string metricName, string? dedupKey = null) => + Task.FromResult(null); + + public Task GetLastDeliveredPageUtcAsync(string serverId, string metricName) => + Task.FromResult(null); + } +} diff --git a/Darling/Darling.Tests/StatementFilterDrillDownWarningsLiveTests.cs b/Darling/Darling.Tests/StatementFilterDrillDownWarningsLiveTests.cs new file mode 100644 index 0000000000..512be66a2d --- /dev/null +++ b/Darling/Darling.Tests/StatementFilterDrillDownWarningsLiveTests.cs @@ -0,0 +1,109 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Net; +using System.Text.Json; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Analysis; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Analysis; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348, L8: a drill-down plan warning quotes up to 200 characters of a predicate from the plan it analyzed, and the +/// finding carrying it is persisted and read back by analyze_server and get_analysis_findings. The collector +/// judges each stored plan before the analyzer sees it, so a predicate that belongs to a withheld statement is withheld +/// with it, while the warning for a statement that is kept still quotes its predicate. +/// +/// #1776 own-store: mints its own scratch database (), never the shared +/// live-postgres collection. +/// +public sealed class StatementFilterDrillDownWarningsLiveTests +{ + private const int ServerId = -453_201; + private const string ServerName = "SsfDrillDownSrv"; + + private static string Statement(string text, string predicate, string constant) => + "" + + "" + + "" + + "" + + "" + + ""; + + private static string TwoStatementPlan() => + "" + + "" + + Statement(StatementScrubCanary.CanaryStatement, "upper([db].[dbo].[t].[c])=N'const-secret-ssf'", "N'const-secret-ssf'") + + Statement(StatementScrubCanary.PlainStatement, "upper([db].[dbo].[t].[c])=N'kept-predicate-ssf'", "N'kept-predicate-ssf'") + + ""; + + [Fact] + public async Task APredicateQuotedInAPlanWarning_IsWithheldWithItsStatement_AndKeptWithAKeptOne() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live drill-down warning filter test."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(connectionString!, ct); + await using (var connection = new NpgsqlConnection(scratch.ConnectionString)) + { + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await using var command = new NpgsqlCommand(@" +INSERT INTO query_stats + (collection_id, collection_time, server_id, server_name, database_name, query_hash, query_plan_xml, delta_worker_time) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8)", connection); + command.Parameters.AddWithValue(CollectionIdGenerator.Next()); + command.Parameters.AddWithValue(DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(ServerName); + command.Parameters.AddWithValue("SsfDb"); + command.Parameters.AddWithValue("0xSSF0DRILL"); + command.Parameters.AddWithValue(TwoStatementPlan()); + command.Parameters.AddWithValue(500_000L); + await command.ExecuteNonQueryAsync(ct); + } + + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var periodEnd = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Unspecified); + var context = new AnalysisContext + { + ServerId = ServerId, + ServerName = ServerName, + TimeRangeStart = periodEnd.AddHours(-4), + TimeRangeEnd = periodEnd, + ServerUtcOffset = TimeSpan.Zero, + }; + var finding = new AnalysisFinding + { + RootFactKey = "PLAN_WARNING", + StoryPath = "PLAN_WARNING", + PathKeys = ["PLAN_WARNING"], + Severity = 1.0, + }; + + await new PgDrillDownCollector(postgres).EnrichFindingsAsync([finding], context); + + Assert.True(finding.DrillDown!.TryGetValue("plan_warnings", out var warnings), "plan_warnings was not collected"); + var text = JsonSerializer.Serialize(warnings); + Assert.Contains("kept-predicate-ssf", text, StringComparison.Ordinal); + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.DoesNotContain(needle, text, StringComparison.Ordinal); + } + } +} diff --git a/Darling/Darling.Tests/StatementFilterPlanReaderSourceScanTests.cs b/Darling/Darling.Tests/StatementFilterPlanReaderSourceScanTests.cs new file mode 100644 index 0000000000..f98697d019 --- /dev/null +++ b/Darling/Darling.Tests/StatementFilterPlanReaderSourceScanTests.cs @@ -0,0 +1,169 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Reflection; +using System.Runtime.CompilerServices; +using System.Text.RegularExpressions; +using PerformanceMonitor.Darling.Service.Mcp; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348, Layer 1: the stored-plan reads are filtered at ONE seam, DarlingStoredPlanReader. Three pins keep it that way. +/// (1) Every public method of the reader is listed here, takes the optional maxOutputChars and answers through +/// SensitiveStatements.Xml (or through another listed method that does); a new public read fails until it is listed, +/// wrapped and planted in StatementFilterPlanReadsLiveTests. (2) No other file under Mcp names a plan column in +/// SQL it runs, so a tool cannot read a plan around the seam. (3) The scan itself is proven against planted sources. +/// +public sealed class StatementFilterPlanReaderSourceScanTests +{ + /// Each public read, and how many times its own body calls SensitiveStatements.Xml (a method that only + /// delegates to another listed read has none). + private static readonly Dictionary PublicReads = new(StringComparer.Ordinal) + { + ["GetQueryStatsPlanXmlByHashAsync"] = 1, + ["GetProcedurePlanXmlBySqlHandleAsync"] = 1, + ["GetQueryStorePlanTextAsync"] = 0, + ["ResolveQueryStorePlanAsync"] = 1, + ["ReadQueryStorePlanByIdAsync"] = 1, + ["GetQuerySnapshotPlanXmlAsync"] = 1, + ["GetBlockingPlanXmlAsync"] = 1, + ["GetDeadlockVictimPlanXmlAsync"] = 1, + }; + + [Fact] + public void EveryPublicPlanRead_IsListed_TakesMaxOutputChars_AndIsPlantedInTheLiveTests() + { + string[] actual = typeof(DarlingStoredPlanReader) + .GetMethods(BindingFlags.Public | BindingFlags.Static | BindingFlags.DeclaredOnly) + .Select(m => m.Name) + .OrderBy(n => n, StringComparer.Ordinal) + .ToArray(); + Assert.Equal(PublicReads.Keys.OrderBy(n => n, StringComparer.Ordinal).ToArray(), actual); + + string live = File.ReadAllText(Path.Combine(RepoRoot(), "Darling", "Darling.Tests", "StatementFilterPlanReadsLiveTests.cs")); + foreach (MethodInfo method in typeof(DarlingStoredPlanReader).GetMethods(BindingFlags.Public | BindingFlags.Static | BindingFlags.DeclaredOnly)) + { + ParameterInfo? cut = method.GetParameters().SingleOrDefault(p => p.Name == "maxOutputChars"); + Assert.True(cut is not null && cut.HasDefaultValue && Equals(cut.DefaultValue, int.MaxValue), + method.Name + " must take an optional maxOutputChars that defaults to int.MaxValue"); + ParameterInfo[] parameters = method.GetParameters(); + Assert.True(parameters[^1].ParameterType == typeof(System.Threading.CancellationToken) && parameters[^2] == cut, + method.Name + ": maxOutputChars sits just before the CancellationToken (CA1068)"); + Assert.Contains("DarlingStoredPlanReader." + method.Name + "(", live, StringComparison.Ordinal); + } + } + + [Fact] + public void EveryPublicPlanRead_AnswersThroughTheStatementFilter_AndNoRawTwinIsPublic() + { + string source = ReaderCode(); + foreach ((string name, int expected) in PublicReads) + { + string body = MemberSource(source, name); + Assert.Equal(expected, Regex.Matches(body, @"\bSensitiveStatements\s*\.\s*Xml\s*\(").Count); + if (expected == 0) + Assert.Matches(@"\bResolveQueryStorePlanAsync\s*\(", body); + else + Assert.Contains("maxOutputChars", body, StringComparison.Ordinal); + } + + // The unfiltered twins are private, so nothing outside the file can reach a raw plan. + MatchCollection twins = Regex.Matches(source, @"^ (?\w+) static [^\n]*?\b\w+RawAsync\(", RegexOptions.Multiline); + Assert.Equal(PublicReads.Count - 1, twins.Count); + foreach (Match twin in twins) + Assert.Equal("private", twin.Groups["access"].Value); + } + + /// The columns that hold a plan: the stored XML, its gzip form, Query Store's text, the snapshot columns, and every + /// *_plan_xml (the blocked, blocking and victim plans). query_plan_hash is a hash, not a plan. + private static readonly Regex PlanColumn = new( + @"(?The reads of a plan column that are not plan reads: col IS [NOT] NULL and col <> '', and the + /// has_* aliases they are printed under: the has-a-plan flags the lists print. + private static readonly Regex PresenceOnly = new( + @"(?\s*'')|(?The names of the plan columns that SQL string literals in read, with the presence-only + /// idiom taken out first. Comments are not code, and a string that is not SQL (a tool description) is not a read. + internal static List PlanColumnReads(string source) + { + var found = new List(); + foreach ((int _, string text) in CSharpSourceWalker.StringLiteralBodies(source)) + { + if (!LooksLikeSql.IsMatch(text)) continue; + foreach (Match m in PlanColumn.Matches(PresenceOnly.Replace(text, " "))) + found.Add(m.Value); + } + + return found; + } + + [Fact] + public void NoFileUnderMcp_ReadsAPlanColumn_ExceptTheStoredPlanReader() + { + string mcp = Path.Combine(RepoRoot(), "Darling", "PerformanceMonitor.Darling.Service", "Mcp"); + var offenders = new List(); + foreach (string file in Directory.EnumerateFiles(mcp, "*.cs", SearchOption.AllDirectories)) + { + if (string.Equals(Path.GetFileName(file), "DarlingStoredPlanReader.cs", StringComparison.Ordinal)) continue; + foreach (string column in PlanColumnReads(File.ReadAllText(file)).Distinct()) + offenders.Add(Path.GetFileName(file) + ": " + column); + } + + Assert.Empty(offenders); + // The reader itself is the one place the scan must find them: a scan that finds none there reads nothing. + Assert.NotEmpty(PlanColumnReads(File.ReadAllText(Path.Combine(mcp, "DarlingStoredPlanReader.cs")))); + } + + [Fact] + public void ThePlanColumnScan_FlagsEachPlanColumn_AndIgnoresHashesFlagsCommentsAndDescriptions() + { + foreach (string column in new[] { "query_plan_xml", "query_plan_text", "query_plan_gz", "query_plan", "live_query_plan", "victim_query_plan_xml", "blocked_query_plan_xml", "plan_xml" }) + { + Assert.Equal(new[] { column }, PlanColumnReads("const string S = \"SELECT " + column + " FROM t\";")); + Assert.Equal(new[] { column }, PlanColumnReads("const string S = \"\"\"\n SELECT 1\n FROM t WHERE x = (SELECT " + column + " FROM u)\n \"\"\";")); + } + + // Not reads of a plan: a plan HASH, a presence flag, a comment, and a tool description that is not SQL. + Assert.Empty(PlanColumnReads("const string S = \"SELECT query_plan_hash FROM t\";")); + Assert.Empty(PlanColumnReads("const string S = \"SELECT (blocked_query_plan_xml IS NOT NULL AND blocked_query_plan_xml <> '') AS f FROM t\";")); + Assert.Empty(PlanColumnReads("// SELECT query_plan_xml FROM t\nconst string S = \"SELECT 1 FROM t\";")); + Assert.Empty(PlanColumnReads("[Description(\"Returns the raw stored query_plan_xml, see get_plan_xml\")] void M() { }")); + Assert.Empty(PlanColumnReads("const string S = \"SELECT (query_plan IS NOT NULL) AS has_query_plan, (b.plan_xml IS NOT NULL) AS has_plan_xml FROM t\";")); + // A flag does not excuse a read beside it. + Assert.Equal(new[] { "blocked_query_plan_xml" }, + PlanColumnReads("const string S = \"SELECT blocked_query_plan_xml, (blocking_query_plan_xml IS NOT NULL) AS f FROM t\";")); + } + + // ── helpers ── + + private static string ReaderCode() => CSharpSourceWalker.StripCommentsAndStrings(File.ReadAllText( + Path.Combine(RepoRoot(), "Darling", "PerformanceMonitor.Darling.Service", "Mcp", "DarlingStoredPlanReader.cs"))); + + /// The text of the member that declares as a method, from its declaration line to the next + /// member at class level (four spaces of indent). + private static string MemberSource(string code, string name) + { + Match start = Regex.Match(code, @"^ public static [^\n]*?\b" + Regex.Escape(name) + @"\(", RegexOptions.Multiline); + Assert.True(start.Success, name + " has no public declaration"); + Match next = Regex.Match(code.Substring(start.Index + start.Length), @"^ (?:public|private|internal)\b", RegexOptions.Multiline); + return next.Success ? code.Substring(start.Index, start.Length + next.Index) : code.Substring(start.Index); + } + + private static string RepoRoot([CallerFilePath] string thisFile = "") + => Path.GetFullPath(Path.Combine(Path.GetDirectoryName(thisFile)!, "..", "..")); +} diff --git a/Darling/Darling.Tests/StatementFilterPlanReadsLiveTests.cs b/Darling/Darling.Tests/StatementFilterPlanReadsLiveTests.cs new file mode 100644 index 0000000000..99690ba30c --- /dev/null +++ b/Darling/Darling.Tests/StatementFilterPlanReadsLiveTests.cs @@ -0,0 +1,608 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Diagnostics; +using System.Globalization; +using System.Text; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348, Layer 1: every stored-plan read in DarlingStoredPlanReader answers with the plan the statement filter judged, so +/// a tool that reads a stored plan cannot hand the canary statement, or the values its plan carries, to a caller. Each case is +/// planted with in the form the store keeps it, then read through the public +/// method: the stored query_stats and procedure_stats plan inline and in the gzip query_plan_dim row, a Query Store plan +/// through query_store_plan_map (text and gzip) and through the inline fallback, the snapshot's query_plan and +/// live_query_plan, and the blocked, blocking and deadlock-victim plans. The cut cases read a plan larger than the +/// tools' 500 KB transport limit and check the cut is made on the filtered plan. +/// +[Collection("live-postgres")] +public sealed class StatementFilterPlanReadsLiveTests +{ + private const string ServerName = "darling-ssf-plan-reads"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + private const string Db = "SsfPlanDb"; + private static readonly DateTime Anchor = new DateTime(2026, 3, 4, 5, 6, 7, DateTimeKind.Unspecified).AddTicks(1_234_560); + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + /// The canary plan, made distinct per case (two cases that stored the same plan would share one dimension row, + /// and a text row and a gzip row cannot both be the one row). + private static string Variant(int n) => StatementScrubCanary.CanaryPlan().Replace("16.0.4000.1\"", "16.0.4000." + n + "\"", StringComparison.Ordinal); + + private static void AssertFiltered(string label, string raw, string? filtered) + { + Assert.True(filtered is not null, label + ": the read found no plan (seeding or key is wrong)"); + StatementFilterCensus.AssertRawHoldsTheCanary(raw); + try + { + StatementFilterCensus.AssertPlanFilteredKeepsTheRest(raw, filtered!); + } + catch (Exception ex) + { + throw new Xunit.Sdk.XunitException(label + ": " + ex.Message); + } + } + + [Fact] + public async Task EveryStoredPlanRead_ReturnsThePlanWithTheCanaryStatementWithheld() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live plan-read filter test."); + var ct = TestContext.Current.CancellationToken; + + await using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + + var bodySucceeded = false; + try + { + await RegisterServerAsync(connection, ct); + + string statsInline = Variant(1), statsGz = Variant(2), procInline = Variant(3), procGz = Variant(4); + string qsMapText = Variant(5), qsMapGz = Variant(6), qsInline = Variant(7); + string snapshot = Variant(8), snapshotLive = Variant(9); + string blocked = Variant(10), blocking = Variant(11), victim = Variant(12); + + /* query_stats and procedure_stats: the plan inline on the row, and in the gzip query_plan_dim row the row's digest names. */ + await InsertStatsAsync(connection, "query_stats", Anchor, "0xQSINLINE", null, statsInline, null, ct); + await InsertDimAsync(connection, statsGz, gz: true, ct); + await InsertStatsAsync(connection, "query_stats", Anchor, "0xQSGZ", null, null, statsGz, ct); + await InsertStatsAsync(connection, "procedure_stats", Anchor, null, "0xA0A0A001", procInline, null, ct); + await InsertDimAsync(connection, procGz, gz: true, ct); + await InsertStatsAsync(connection, "procedure_stats", Anchor, null, "0xA0A0A002", null, procGz, ct); + + /* Query Store: through the map to a text and a gzip dimension row, and the inline column with no map row. */ + await InsertQueryStoreAsync(connection, Anchor, queryId: 1, planId: 11, inline: null, ct); + await InsertDimAsync(connection, qsMapText, gz: false, ct); + await InsertMapAsync(connection, 11, qsMapText, ct); + await InsertQueryStoreAsync(connection, Anchor, queryId: 2, planId: 21, inline: null, ct); + await InsertDimAsync(connection, qsMapGz, gz: true, ct); + await InsertMapAsync(connection, 21, qsMapGz, ct); + await InsertQueryStoreAsync(connection, Anchor, queryId: 3, planId: 31, inline: qsInline, ct); + + await InsertSnapshotAsync(connection, Anchor, 51, 3, snapshot, snapshotLive, ct); + await InsertReportAsync(connection, Anchor, 61, 62, blocked, blocking, ct); + var deadlockCollected = Anchor.AddMinutes(10); + var deadlockStamp = deadlockCollected.AddSeconds(-7); + await InsertDeadlockAsync(connection, deadlockCollected, deadlockStamp, "process1a2b", victim, ct); + + AssertFiltered("query_stats inline", statsInline, + await DarlingStoredPlanReader.GetQueryStatsPlanXmlByHashAsync(postgres, ServerId, "0xQSINLINE", Db, cancellationToken: ct)); + AssertFiltered("query_stats gz dim", statsGz, + await DarlingStoredPlanReader.GetQueryStatsPlanXmlByHashAsync(postgres, ServerId, "0xQSGZ", null, cancellationToken: ct)); + AssertFiltered("procedure inline", procInline, + await DarlingStoredPlanReader.GetProcedurePlanXmlBySqlHandleAsync(postgres, ServerId, "0xA0A0A001", cancellationToken: ct)); + AssertFiltered("procedure gz dim", procGz, + await DarlingStoredPlanReader.GetProcedurePlanXmlBySqlHandleAsync(postgres, ServerId, "0xA0A0A002", cancellationToken: ct)); + + AssertFiltered("query store map text (text read)", qsMapText, + await DarlingStoredPlanReader.GetQueryStorePlanTextAsync(postgres, ServerId, Db, 1, null, cancellationToken: ct)); + AssertFiltered("query store map gz (text read, pinned)", qsMapGz, + await DarlingStoredPlanReader.GetQueryStorePlanTextAsync(postgres, ServerId, Db, 2, 21, cancellationToken: ct)); + AssertFiltered("query store inline fallback (text read)", qsInline, + await DarlingStoredPlanReader.GetQueryStorePlanTextAsync(postgres, ServerId, Db, 3, null, cancellationToken: ct)); + var resolved = await DarlingStoredPlanReader.ResolveQueryStorePlanAsync(postgres, ServerId, Db, 1, null, cancellationToken: ct); + AssertFiltered("query store map text (resolve)", qsMapText, resolved?.PlanXml); + Assert.Equal(11, resolved!.PlanId); + AssertFiltered("query store inline fallback (resolve)", qsInline, + (await DarlingStoredPlanReader.ResolveQueryStorePlanAsync(postgres, ServerId, Db, 3, 31, cancellationToken: ct))?.PlanXml); + AssertFiltered("query store map gz (by id)", qsMapGz, + await DarlingStoredPlanReader.ReadQueryStorePlanByIdAsync(postgres, ServerId, Db, 2, 21, cancellationToken: ct)); + AssertFiltered("query store inline fallback (by id)", qsInline, + await DarlingStoredPlanReader.ReadQueryStorePlanByIdAsync(postgres, ServerId, Db, 3, 31, cancellationToken: ct)); + + AssertFiltered("snapshot query_plan", snapshot, + await DarlingStoredPlanReader.GetQuerySnapshotPlanXmlAsync(postgres, ServerId, Anchor, 51, 3, live: false, cancellationToken: ct)); + AssertFiltered("snapshot live_query_plan", snapshotLive, + await DarlingStoredPlanReader.GetQuerySnapshotPlanXmlAsync(postgres, ServerId, Anchor, 51, 3, live: true, cancellationToken: ct)); + + AssertFiltered("blocked plan", blocked, + await DarlingStoredPlanReader.GetBlockingPlanXmlAsync(postgres, ServerId, Anchor, 61, 0, 62, 0, blockingSide: false, cancellationToken: ct)); + AssertFiltered("blocking plan", blocking, + await DarlingStoredPlanReader.GetBlockingPlanXmlAsync(postgres, ServerId, Anchor, 61, 0, 62, 0, blockingSide: true, cancellationToken: ct)); + var deadlock = await DarlingStoredPlanReader.GetDeadlockVictimPlanXmlAsync(postgres, ServerId, deadlockCollected, deadlockStamp, "process1a2b", cancellationToken: ct); + Assert.False(deadlock.Ambiguous); + AssertFiltered("deadlock victim plan", victim, deadlock.PlanXml); + + /* The tools that read through the seam: the same plan, filtered, and the analysis of it holds no secret. */ + AssertFiltered("get_plan_xml", statsInline, await DarlingMcpPlanTools.GetPlanXml(postgres, "0xQSINLINE", ServerName, cancellationToken: ct)); + string analysis = await DarlingMcpPlanTools.AnalyzeQueryPlan(postgres, "0xQSINLINE", ServerName, cancellationToken: ct); + foreach (string needle in StatementScrubCanary.SecretNeedles) + Assert.DoesNotContain(needle, analysis, StringComparison.Ordinal); + using (var doc = JsonDocument.Parse(analysis)) + Assert.Contains(StatementFilterCensus.Marker, doc.RootElement.ToString(), StringComparison.Ordinal); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// + /// A plan over the tools' 500 KB limit whose FIRST statement is the canary. The filter runs on the plan and the cut is made on + /// what it returns, so the answer is a recognizable cut plan, ending in the truncation marker, with that statement withheld + /// by its own placeholder. It is not the whole-output placeholder a sweep of the cut text would have to fall back to. The + /// same plan read with the reader's default (no cut) comes back whole, with the same statement withheld. + /// + [Fact] + public async Task APlanOverTheTransportLimit_WithTheCanaryFirst_IsCutOnTheFilteredPlan_NotWithheldWhole() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live plan-read filter test."); + var ct = TestContext.Current.CancellationToken; + + await using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + + var bodySucceeded = false; + try + { + await RegisterServerAsync(connection, ct); + + string big = BigPlan(600_000, canaryFirst: true); + Assert.True(big.Length > 600_000); + await InsertStatsAsync(connection, "query_stats", Anchor, "0xBIGHIT", null, big, null, ct); + + string cut = await UntilNotStarvedAsync("cut read", + () => DarlingMcpPlanTools.GetPlanXml(postgres, "0xBIGHIT", ServerName, cancellationToken: ct), SensitiveStatements.IsWithheldPlan); + Assert.NotEqual(SensitiveStatements.PlaceholderText, cut); + Assert.StartsWith(" (await DarlingStoredPlanReader.GetQueryStatsPlanXmlByHashAsync(postgres, ServerId, "0xBIGHIT", null, cancellationToken: ct))!, SensitiveStatements.IsWithheldPlan); + Assert.True(whole.Length > 512_000); + foreach (string needle in StatementScrubCanary.SecretNeedles) + Assert.DoesNotContain(needle, whole, StringComparison.Ordinal); + Assert.Contains(StatementFilterCensus.Marker, whole, StringComparison.Ordinal); + Assert.EndsWith("", whole, StringComparison.Ordinal); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// + /// The answer a client receives for a stored plan over the tools' 500 KB limit, at the two wire seams: the host's registered + /// filters (MCP) and the web read's ToHttpResult. A plan with no auto-parameter token comes back as a cut, filtered plan + /// ending in the truncation marker, with the named statement's placeholder inside. The same plan padded with auto-parameter + /// tokens (@1) comes back as the whole-output placeholder: the cut plan does not parse, so the second check cannot + /// probe it and withholds it whole (fails closed, accepted behaviour). + /// + [Fact] + public async Task APlanOverTheTransportLimit_OnTheWire_IsACutFilteredPlanWithoutAutoParameterTokens_AndThePlaceholderWithThem() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live plan-read filter test."); + var ct = TestContext.Current.CancellationToken; + + await using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + using var host = await StatementFilterCensus.BuildHostAsync(); + + /* #5440: pay the filter's one-time costs (the judge build and the first parse, which run before and inside its + time limits) on a small plan of the same shape, as the service does at startup, so the 600,000-character reads + below are timed on a warm filter. */ + SensitiveStatements.WarmUp(); + string primed = SensitiveStatements.Xml(BigPlan(30_000, canaryFirst: true), 500_000)!; + _ = await StatementFilterCensus.FilterThroughHostAsync(host, primed); + + var bodySucceeded = false; + try + { + await RegisterServerAsync(connection, ct); + await InsertStatsAsync(connection, "query_stats", Anchor, "0xWIRENOTOKEN", null, BigPlan(600_000, canaryFirst: true), null, ct); + await InsertStatsAsync(connection, "query_stats", Anchor, "0xWIRETOKENS", null, BigPlan(600_000, canaryFirst: true, autoParameterTokens: true), null, ct); + + /* No tokens: the cut, filtered plan, on both wires. */ + string noTokens = await UntilNotStarvedAsync("tool read, no tokens", + () => DarlingMcpPlanTools.GetPlanXml(postgres, "0xWIRENOTOKEN", ServerName, cancellationToken: ct), SensitiveStatements.IsWithheldPlan); + string mcpCut = await UntilNotStarvedAsync("host filter, no tokens", + async () => (await StatementFilterCensus.FilterThroughHostAsync(host, noTokens)).Text, SensitiveStatements.IsWithheldPlan); + AssertCutFilteredPlan("host filter, no tokens", mcpCut, markerSuffix: true); + /* The web wrap moves the "... (truncated)" suffix into a truncated flag, so the plan_xml text is the cut plan without it. */ + using var webCutDoc = JsonDocument.Parse(await UntilNotStarvedAsync("web read, no tokens", + () => WebAnswerAsync(noTokens, "0xWIRENOTOKEN"), + answer => !JsonDocument.Parse(answer).RootElement.TryGetProperty("plan_xml", out var planXml) || SensitiveStatements.IsWithheldPlan(planXml.GetString()))); + Assert.True(webCutDoc.RootElement.GetProperty("truncated").GetBoolean()); + AssertCutFilteredPlan("web read, no tokens", webCutDoc.RootElement.GetProperty("plan_xml").GetString()!, markerSuffix: false); + + /* Tokens: the tool's own answer is the cut plan; the second check withholds it whole on both wires. */ + string withTokens = await UntilNotStarvedAsync("tool read, with tokens", + () => DarlingMcpPlanTools.GetPlanXml(postgres, "0xWIRETOKENS", ServerName, cancellationToken: ct), SensitiveStatements.IsWithheldPlan); + Assert.EndsWith("... (truncated)", withTokens, StringComparison.Ordinal); + string mcpHeld = (await StatementFilterCensus.FilterThroughHostAsync(host, withTokens)).Text; + Assert.Equal(SensitiveStatements.PlaceholderText, mcpHeld); + string webHeld = JsonDocument.Parse(await WebAnswerAsync(withTokens, "0xWIRETOKENS")).RootElement.GetProperty("plan_xml").GetString()!; + Assert.Equal(SensitiveStatements.PlaceholderText, webHeld); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// How many times one read of a 600,000-character plan is tried before a withheld answer counts. + private const int StarvedAttempts = 3; + + /// + /// #5440: runs one filtered read of a plan far over the transport limit and gives back its answer. The filter + /// withholds a plan it cannot finish judging inside its limits: (1.5 s for the + /// whole read) and (250 ms for one value, which the cut plan's text is, + /// because it no longer parses). Both run on the wall clock, and a read of this plan takes 35 to 100 ms on an idle + /// machine, so a runner that stalls the thread for a quarter of a second gets the placeholder instead of the plan. The + /// product keeps both limits (a slow judge must still fail closed). This test measures the read and, only when the + /// answer is withheld AND the read took at least (the smallest time + /// the filter can give up in), reads again, up to times, and reports each time. + /// A withheld answer that came back fast is never retried, and the caller still asserts a cut, filtered plan on the + /// answer it gets, so a filter that withholds a plan it should cut still fails on every attempt. + /// + private static async Task UntilNotStarvedAsync(string label, Func> read, Func isWithheld) + { + for (int attempt = 1; ; attempt++) + { + long started = Stopwatch.GetTimestamp(); + string answer = await read(); + TimeSpan took = Stopwatch.GetElapsedTime(started); + bool withheld = isWithheld(answer); + bool starved = withheld && took >= SensitiveStatements.MatchTimeout; + TestContext.Current.SendDiagnosticMessage( + $"#5440 {label}: attempt {attempt} took {took.TotalMilliseconds:F0} ms, withheld={withheld}, starved={starved}"); + if (!starved || attempt >= StarvedAttempts) + { + return answer; + } + } + } + + private static void AssertCutFilteredPlan(string label, string answer, bool markerSuffix) + { + Assert.True(answer.StartsWith("= 500_000, label + ": not a cut plan (" + answer.Length + " chars)"); + Assert.False(answer.EndsWith("", StringComparison.Ordinal), label + ": the whole plan came back"); + Assert.Contains(StatementFilterCensus.Marker, answer, StringComparison.Ordinal); + foreach (string needle in StatementScrubCanary.SecretNeedles) + Assert.DoesNotContain(needle, answer, StringComparison.Ordinal); + } + + /// What the web plan read sends for a tool answer: the wrapped answer through ToHttpResult. + private static async Task WebAnswerAsync(string toolAnswer, string queryHash) + { + var result = DarlingWebEndpoints.ToHttpResult( + DarlingWebEndpoints.WrapPlanXml(toolAnswer, queryHash, null), "/api/read/get_plan_xml", + Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance, 1); + var services = new Microsoft.Extensions.DependencyInjection.ServiceCollection(); + Microsoft.Extensions.DependencyInjection.LoggingServiceCollectionExtensions.AddLogging(services); + var context = new Microsoft.AspNetCore.Http.DefaultHttpContext + { + RequestServices = Microsoft.Extensions.DependencyInjection.ServiceCollectionContainerBuilderExtensions.BuildServiceProvider(services), + }; + context.Response.Body = new System.IO.MemoryStream(); + await result.ExecuteAsync(context); + return Encoding.UTF8.GetString(((System.IO.MemoryStream)context.Response.Body).ToArray()); + } + + /// + /// A 20 MB stored plan read through get_plan_xml, with the sensitive statement at the start and with none: the filter + /// is told where the tool will cut, so neither shape walks the whole document. The ceiling is loose on purpose (a slow + /// runner is not a failure); it catches a read that goes back to filtering all 20 MB. + /// + [Fact] + public async Task ATwentyMegabytePlan_ReadThroughGetPlanXml_StaysInsideTheCeiling_WithAndWithoutAHit() + { + await StatementFilterWarmUp.EnsureAsync(); + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live plan-read filter test."); + var ct = TestContext.Current.CancellationToken; + + await using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + + var bodySucceeded = false; + try + { + await RegisterServerAsync(connection, ct); + await InsertStatsAsync(connection, "query_stats", Anchor, "0xTWENTYHIT", null, BigPlan(20_000_000, canaryFirst: true), null, ct); + await InsertStatsAsync(connection, "query_stats", Anchor, "0xTWENTYMISS", null, BigPlan(20_000_000, canaryFirst: false), null, ct); + + var timings = new StringBuilder(); + foreach (string hash in new[] { "0xTWENTYHIT", "0xTWENTYMISS" }) + { + /* The same read before this change: the stored text and the tool's own cut, with nothing between. */ + var before = Stopwatch.StartNew(); + string raw; + await using (var command = new NpgsqlCommand("SELECT query_plan_xml FROM query_stats WHERE server_id = $1 AND query_hash = $2", connection)) + { + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(hash); + raw = (string)(await command.ExecuteScalarAsync(ct))!; + } + + string unfiltered = McpHelpers.Truncate(raw, 512_000)!; + before.Stop(); + + var after = Stopwatch.StartNew(); + string answer = await DarlingMcpPlanTools.GetPlanXml(postgres, hash, ServerName, cancellationToken: ct); + after.Stop(); + + timings.Append(CultureInfo.InvariantCulture, $"{hash}: unfiltered {before.ElapsedMilliseconds} ms, filtered {after.ElapsedMilliseconds} ms; "); + Assert.EndsWith("... (truncated)", answer, StringComparison.Ordinal); + Assert.NotEqual(SensitiveStatements.PlaceholderText, answer); + Assert.True(after.Elapsed < TimeSpan.FromSeconds(15), timings.ToString()); + Assert.Equal(hash == "0xTWENTYHIT", answer.Contains(StatementFilterCensus.Marker, StringComparison.Ordinal)); + Assert.Equal(hash == "0xTWENTYMISS", answer == unfiltered); + } + + if (Environment.GetEnvironmentVariable("SSF_L5A_TIMINGS") == "1") + Assert.Fail(timings.ToString()); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// A plan of at least characters: the canary plan's statements first (when + /// ; the plain statement otherwise), then plain padding statements. + private static string BigPlan(int minLength, bool canaryFirst, bool autoParameterTokens = false) + { + string canary = StatementScrubCanary.CanaryPlan(); + int open = canary.IndexOf(""); + return sb.Append(Tail).ToString(); + } + + // ── seeding ── + + private static async Task RegisterServerAsync(NpgsqlConnection connection, CancellationToken ct) + { + await using var command = new NpgsqlCommand(@" +INSERT INTO servers (server_id, server_name, display_name, is_enabled, created_date, modified_date) +VALUES ($1, $2, $2, TRUE, $3, $3) +ON CONFLICT (server_id) DO UPDATE SET server_name = EXCLUDED.server_name, display_name = EXCLUDED.display_name, + is_enabled = TRUE, modified_date = EXCLUDED.modified_date;", connection); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(ServerName); + command.Parameters.AddWithValue(DateTime.SpecifyKind(Anchor, DateTimeKind.Unspecified)); + await command.ExecuteNonQueryAsync(ct); + } + + /// One query_stats or procedure_stats row carrying the plan inline () or by the digest of a + /// dimension row (, which stored). + private static async Task InsertStatsAsync( + NpgsqlConnection connection, string table, DateTime collectionTime, string? queryHash, string? sqlHandle, + string? inline, string? dimPlan, CancellationToken ct) + { + string columns = table == "query_stats" + ? "query_hash, sql_handle, plan_handle" + : "schema_name, object_name, sql_handle"; + string values = table == "query_stats" ? "$5, $5, $5" : "'dbo', 'usp_ssf', $5"; + await using var command = new NpgsqlCommand($@" +INSERT INTO {table} + (collection_id, collection_time, server_id, server_name, database_name, {columns}, query_plan_xml, query_plan_digest) +VALUES ($1, $2, $3, $4, '{Db}', {values}, $6, $7)", connection); + command.Parameters.AddWithValue(CollectionIdGenerator.Next()); + command.Parameters.AddWithValue(DateTime.SpecifyKind(collectionTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(ServerName); + command.Parameters.AddWithValue((object?)queryHash ?? (object?)sqlHandle ?? DBNull.Value); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Text, Value = (object?)inline ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter + { + NpgsqlDbType = NpgsqlDbType.Bytea, + Value = dimPlan is null ? DBNull.Value : PayloadDimensions.Digest(dimPlan), + }); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertDimAsync(NpgsqlConnection connection, string plan, bool gz, CancellationToken ct) + { + await using var command = new NpgsqlCommand( + gz + ? "INSERT INTO query_plan_dim (digest, query_plan_gz, last_seen) VALUES ($1, $2, $3) ON CONFLICT DO NOTHING" + : "INSERT INTO query_plan_dim (digest, query_plan_xml, last_seen) VALUES ($1, $2, $3) ON CONFLICT DO NOTHING", + connection); + command.Parameters.AddWithValue(PayloadDimensions.Digest(plan)); + if (gz) + command.Parameters.AddWithValue(PayloadDimensions.CompressContent(plan)); + else + command.Parameters.AddWithValue(plan); + command.Parameters.AddWithValue(DateTime.SpecifyKind(Anchor, DateTimeKind.Unspecified)); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertQueryStoreAsync( + NpgsqlConnection connection, DateTime collectionTime, long queryId, long planId, string? inline, CancellationToken ct) + { + await using var command = new NpgsqlCommand(@" +INSERT INTO query_store_stats + (collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, query_plan_text) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8)", connection); + command.Parameters.AddWithValue(CollectionIdGenerator.Next()); + command.Parameters.AddWithValue(DateTime.SpecifyKind(collectionTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(ServerName); + command.Parameters.AddWithValue(Db); + command.Parameters.AddWithValue(queryId); + command.Parameters.AddWithValue(planId); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Text, Value = (object?)inline ?? DBNull.Value }); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertMapAsync(NpgsqlConnection connection, long planId, string plan, CancellationToken ct) + { + await using var command = new NpgsqlCommand( + "INSERT INTO collect.query_store_plan_map (server_id, database_name, plan_id, digest, plan_hash, last_seen) VALUES ($1, $2, $3, $4, $5, $6)", + connection); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(Db); + command.Parameters.AddWithValue(planId); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Bytea, Value = PayloadDimensions.Digest(plan) }); + command.Parameters.AddWithValue("0x5257"); + command.Parameters.AddWithValue(DateTime.SpecifyKind(Anchor, DateTimeKind.Unspecified)); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertSnapshotAsync( + NpgsqlConnection connection, DateTime collectionTime, int sessionId, int requestId, string plan, string livePlan, CancellationToken ct) + { + await using var command = new NpgsqlCommand(@" +INSERT INTO query_snapshots + (collection_id, collection_time, server_id, server_name, session_id, request_id, query_plan, live_query_plan) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8)", connection); + command.Parameters.AddWithValue(CollectionIdGenerator.Next()); + command.Parameters.AddWithValue(DateTime.SpecifyKind(collectionTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(ServerName); + command.Parameters.AddWithValue(sessionId); + command.Parameters.AddWithValue(requestId); + command.Parameters.AddWithValue(plan); + command.Parameters.AddWithValue(livePlan); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertReportAsync( + NpgsqlConnection connection, DateTime eventTime, int blockedSpid, int blockingSpid, string blockedPlan, string blockingPlan, CancellationToken ct) + { + await using var command = new NpgsqlCommand(@" +INSERT INTO blocked_process_reports + (blocked_report_id, collection_time, server_id, server_name, event_time, wait_time_ms, blocking_spid, blocked_spid, + blocked_ecid, blocking_ecid, blocking_status, database_name, blocked_query_plan_xml, blocking_query_plan_xml) +VALUES ($1, $2, $3, $4, $5, 12000, $6, $7, 0, 0, 'suspended', $8, $9, $10)", connection); + command.Parameters.AddWithValue(CollectionIdGenerator.Next()); + command.Parameters.AddWithValue(DateTime.SpecifyKind(eventTime.AddSeconds(5), DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(ServerName); + command.Parameters.AddWithValue(DateTime.SpecifyKind(eventTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(blockingSpid); + command.Parameters.AddWithValue(blockedSpid); + command.Parameters.AddWithValue(Db); + command.Parameters.AddWithValue(blockedPlan); + command.Parameters.AddWithValue(blockingPlan); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task InsertDeadlockAsync( + NpgsqlConnection connection, DateTime collectionTime, DateTime deadlockTime, string victimProcessId, string victimPlan, CancellationToken ct) + { + await using var command = new NpgsqlCommand(@" +INSERT INTO deadlocks + (deadlock_id, collection_time, server_id, server_name, deadlock_time, victim_process_id, victim_query_plan_xml, database_name) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8)", connection); + command.Parameters.AddWithValue(CollectionIdGenerator.Next()); + command.Parameters.AddWithValue(DateTime.SpecifyKind(collectionTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(ServerName); + command.Parameters.AddWithValue(DateTime.SpecifyKind(deadlockTime, DateTimeKind.Unspecified)); + command.Parameters.AddWithValue(victimProcessId); + command.Parameters.AddWithValue(victimPlan); + command.Parameters.AddWithValue(Db); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task DeleteRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + await using (var cleanup = new NpgsqlCommand( + $"DELETE FROM blocked_process_reports WHERE server_id = {ServerId}; " + + $"DELETE FROM deadlocks WHERE server_id = {ServerId}; " + + $"DELETE FROM query_snapshots WHERE server_id = {ServerId}; " + + $"DELETE FROM query_store_stats WHERE server_id = {ServerId}; " + + $"DELETE FROM collect.query_store_plan_map WHERE server_id = {ServerId}; " + + $"DELETE FROM query_stats WHERE server_id = {ServerId}; " + + $"DELETE FROM procedure_stats WHERE server_id = {ServerId}; " + + $"DELETE FROM servers WHERE server_id = {ServerId};", connection)) + { + await cleanup.ExecuteNonQueryAsync(ct); + } + + /* The dimension rows are keyed by digest, so they go by the plans this class planted. */ + for (int n = 1; n <= 12; n++) + { + await using var dim = new NpgsqlCommand("DELETE FROM query_plan_dim WHERE digest = $1", connection); + dim.Parameters.AddWithValue(PayloadDimensions.Digest(Variant(n))); + await dim.ExecuteNonQueryAsync(ct); + } + } +} diff --git a/Darling/Darling.Tests/StatementFilterQueryReadsLiveTests.cs b/Darling/Darling.Tests/StatementFilterQueryReadsLiveTests.cs new file mode 100644 index 0000000000..15a2081d6f --- /dev/null +++ b/Darling/Darling.Tests/StatementFilterQueryReadsLiveTests.cs @@ -0,0 +1,310 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348, read-time census for the query-text family (inventory rows 1-7): get_active_queries, get_top_queries_by_cpu, +/// get_query_store_top, get_query_store_regressions, get_query_heatmap, get_plan_corrections and both get_finops views that +/// print statement text (high_impact's sample_query_text, optimization's query_preview). Each case plants rows DIRECTLY in +/// the store tables, so the rows look like ones stored before the collection hook existed: a canary statement beside a plain +/// one, and a second server holding only plain rows. The read runs RAW (the tool method alone: it must still hold the +/// canary, the control that proves the plant reached the output) and FILTERED (the same text through the host's registered +/// filter list). The filtered answer holds no secret needle, carries the marker, and keeps the plain statement. A page with +/// no canary comes back byte-identical to the raw read (whenever the output format variable is unset, which is the +/// production default). Each case reads the preview form and the full_text=true form where the tool has one. +/// +[Collection("live-postgres")] +public sealed class StatementFilterQueryReadsLiveTests +{ + private const string Db = "SsfQueryDb"; + private const string CanaryServer = "darling-ssf-query-reads"; + private const string CleanServer = "darling-ssf-query-reads-clean"; + private static readonly int CanaryId = ServerIdHelper.GetDeterministicHashCode(CanaryServer); + private static readonly int CleanId = ServerIdHelper.GetDeterministicHashCode(CleanServer); + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private readonly ITestOutputHelper _output; + + public StatementFilterQueryReadsLiveTests(ITestOutputHelper output) => _output = output; + + public static IEnumerable> Cases() => new[] + { + new TheoryDataRow("get_active_queries"), + new TheoryDataRow("get_top_queries_by_cpu"), + new TheoryDataRow("get_query_store_top"), + new TheoryDataRow("get_query_store_regressions"), + new TheoryDataRow("get_query_heatmap"), + new TheoryDataRow("get_plan_corrections"), + new TheoryDataRow("get_finops high_impact"), + new TheoryDataRow("get_finops optimization"), + }; + + [Theory] + [MemberData(nameof(Cases))] + public async Task EveryQueryTextRead_WithTheCanaryStoredBeforeTheHook_WithholdsItThroughTheHost_AndLeavesACleanPageUntouched(string tool) + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live query-text filter census."); + var ct = TestContext.Current.CancellationToken; + + await using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + using var host = await StatementFilterCensus.BuildHostAsync(); + + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, CanaryId, CanaryServer, ct); + await DarlingMcpTestData.RegisterServerAsync(connection, CleanId, CleanServer, ct); + await SeedAsync(connection, tool, CanaryId, CanaryServer, withCanary: true, ct); + await SeedAsync(connection, tool, CleanId, CleanServer, withCanary: false, ct); + + var reads = ReadsFor(postgres, tool, ct); + Assert.NotEmpty(reads); + foreach (var (form, readerJudges, read) in reads) + { + string label = tool + " (" + form + ")"; + + /* The plant is real: the stored rows still hold the canary, whatever the read does with it. */ + Assert.True(await StoreHoldsTheCanaryAsync(connection, CanaryId, ct), label + ": the stored rows lost the canary, so the plant never reached the store"); + + string raw = await read(CanaryServer); + string filtered; + if (readerJudges) + { + /* #5320: this form's reader judges the WHOLE statement before it cuts it, so the reader's own answer already + holds the marker and not one secret needle, and the plain statement beside it is kept. The host filter then + has nothing left to withhold (the answer is unchanged), which proves the reader did the whole job. */ + foreach (string needle in StatementScrubCanary.SecretNeedles) + Assert.False(raw.Contains(needle, StringComparison.Ordinal), label + ": the reader's own answer still holds a secret needle"); + Assert.True(raw.Contains(StatementFilterCensus.Marker, StringComparison.Ordinal), label + ": the reader's own answer does not carry the withheld marker"); + Assert.True(raw.Contains("canary_plain_ssf", StringComparison.Ordinal), label + ": the reader's own answer lost the plain statement beside the canary"); + filtered = (await StatementFilterCensus.FilterThroughHostAsync(host, raw)).Text; + Assert.True(string.Equals(raw, filtered, StringComparison.Ordinal), label + ": the host filter changed an answer the reader had already judged"); + } + else + { + Assert.True(raw.Contains("S3cret-canary-ssf", StringComparison.Ordinal), label + ": the raw read lost the canary, so the plant did not reach the output"); + filtered = (await StatementFilterCensus.FilterThroughHostAsync(host, raw)).Text; + try + { + StatementFilterCensus.AssertFilteredWithholdsTheCanary(raw, filtered); + } + catch (Exception ex) + { + throw new Xunit.Sdk.XunitException(label + ": " + ex.Message); + } + } + /* The no-hit page: nothing to withhold, so the answer is the raw read, byte for byte. The host is built with GCF + pinned off (#5320), so no other class's DARLING_OUTPUT_FORMAT reaches it and the identity holds every run. */ + string clean = await read(CleanServer); + Assert.DoesNotContain("S3cret-canary-ssf", clean, StringComparison.Ordinal); + string cleanFiltered = (await StatementFilterCensus.FilterThroughHostAsync(host, clean)).Text; + Assert.True(string.Equals(clean, cleanFiltered, StringComparison.Ordinal), label + ": a page with no canary changed under the filter"); + + _output.WriteLine($"{label}: no-hit page {Encoding.UTF8.GetByteCount(clean):N0} bytes raw, {Encoding.UTF8.GetByteCount(cleanFiltered):N0} bytes filtered; canary page {Encoding.UTF8.GetByteCount(raw):N0} raw, {Encoding.UTF8.GetByteCount(filtered):N0} filtered."); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// Every form of the tool the census reads: the default (preview) and the full_text form where the tool has one. ReaderJudges is true for a form whose reader judges the whole statement itself (#5320: the preview forms, which judge then cut, and the heatmap, whose reader judges both), so the census holds that form to the reader's own answer; a false form returns the stored text raw and the host filter is the one that withholds it. + private static List<(string Form, bool ReaderJudges, Func> Read)> ReadsFor(NpgsqlDataSource postgres, string tool, CancellationToken ct) + { + var reads = new List<(string, bool, Func>)>(); + switch (tool) + { + case "get_active_queries": + reads.Add(("preview", true, s => DarlingMcpSessionTools.GetActiveQueries(postgres, s, cancellationToken: ct))); + reads.Add(("full_text", false, s => DarlingMcpSessionTools.GetActiveQueries(postgres, s, full_text: true, cancellationToken: ct))); + break; + case "get_top_queries_by_cpu": + reads.Add(("default", true, s => DarlingMcpDataTools.GetTopQueriesByCpu(postgres, s, cancellationToken: ct))); + reads.Add(("detail=full", true, s => DarlingMcpDataTools.GetTopQueriesByCpu(postgres, s, detail: "full", cancellationToken: ct))); + reads.Add(("group_by=host_object", true, s => DarlingMcpDataTools.GetTopQueriesByCpu(postgres, s, group_by: "host_object", cancellationToken: ct))); + break; + case "get_query_store_top": + reads.Add(("preview", true, s => DarlingMcpDataTools.GetQueryStoreTop(postgres, s, cancellationToken: ct))); + reads.Add(("full_text", false, s => DarlingMcpDataTools.GetQueryStoreTop(postgres, s, full_text: true, cancellationToken: ct))); + break; + case "get_query_store_regressions": + reads.Add(("preview", true, s => DarlingMcpQueryStoreRegressionTools.GetQueryStoreRegressions(postgres, s, cancellationToken: ct))); + reads.Add(("full_text", false, s => DarlingMcpQueryStoreRegressionTools.GetQueryStoreRegressions(postgres, s, full_text: true, cancellationToken: ct))); + break; + case "get_query_heatmap": + reads.Add(("preview", true, s => DarlingMcpQueryHeatmapTools.GetQueryHeatmap(postgres, s, cancellationToken: ct))); + reads.Add(("full_text", true, s => DarlingMcpQueryHeatmapTools.GetQueryHeatmap(postgres, s, full_text: true, cancellationToken: ct))); + break; + case "get_plan_corrections": + reads.Add(("preview", true, s => DarlingMcpPlanCorrectionTools.GetPlanCorrections(postgres, s, cancellationToken: ct))); + reads.Add(("full_text", false, s => DarlingMcpPlanCorrectionTools.GetPlanCorrections(postgres, s, full_text: true, cancellationToken: ct))); + break; + case "get_finops high_impact": + reads.Add(("high_impact", true, s => DarlingMcpFinOpsTools.GetFinOps(postgres, "high_impact", s, cancellationToken: ct))); + break; + case "get_finops optimization": + reads.Add(("optimization", true, s => DarlingMcpFinOpsTools.GetFinOps(postgres, "optimization", s, cancellationToken: ct))); + break; + default: + throw new ArgumentOutOfRangeException(nameof(tool), tool, null); + } + + return reads; + } + + private static async Task SeedAsync(NpgsqlConnection connection, string tool, int serverId, string serverName, bool withCanary, CancellationToken ct) + { + var now = DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow); + var texts = new List<(string Text, string Tag)>(); + if (withCanary) texts.Add((StatementScrubCanary.CanaryStatement, "C")); + texts.Add((StatementScrubCanary.PlainStatement, "P")); + + switch (tool) + { + case "get_active_queries": + for (int i = 0; i < texts.Count; i++) + { + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO query_snapshots (collection_id, collection_time, server_id, server_name, session_id, database_name, elapsed_time_formatted, query_text, status, blocking_session_id, wait_type, wait_time_ms, cpu_time_ms, total_elapsed_time_ms, reads, writes, logical_reads, granted_query_memory_gb, transaction_isolation_level, dop, parallel_worker_count, login_name, host_name, program_name, open_transaction_count, request_id, wait_resource, percent_complete, query_hash) +VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(now.AddMinutes(-5 - i)), serverId, serverName, 100 + i, Db, + "00 00:00:05.125", texts[i].Text, "running", 0, null, 0L, 1000L, 1500L, 10_000L, 50L, 7L, 0m, + "Read Committed", 1, 0, "app_svc", "APPSRV01", "MyOrderService.Worker", 0, 0, + "KEY: 5:72057594043432960 (a1b2c3d4e5f6)", 12.5m, "0x" + (0x1A2B3C4D5E6F7080L + i).ToString("X16")); + } + + break; + + case "get_top_queries_by_cpu": + case "get_finops high_impact": + case "get_finops optimization": + for (int i = 0; i < texts.Count; i++) + { + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO query_stats (collection_id, collection_time, server_id, server_name, database_name, query_hash, query_plan_hash, sql_handle, plan_handle, query_text, + delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, min_dop, max_dop, sample_interval_seconds) +VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(now.AddMinutes(-30 - i)), serverId, serverName, Db, + "0xQH" + texts[i].Tag, "0xPLAN" + texts[i].Tag, "0xSQLH" + texts[i].Tag, "0xPLANH" + texts[i].Tag, texts[i].Text, + 10L, 1_000_000L, 2_000_000L, 5_000L, 1, 1, 60); + } + + break; + + case "get_query_heatmap": + { + var t0 = new DateTime(now.Ticks - (now.Ticks % TimeSpan.TicksPerHour), now.Kind).AddHours(-1); + long[] elapsedMicros = { 500, 5_000_000 }; + for (int i = 0; i < texts.Count; i++) + { + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO query_stats (collection_id, collection_time, server_id, server_name, database_name, query_hash, sample_interval_seconds, delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, delta_logical_writes, query_text) +VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(t0), serverId, serverName, Db, "0xHM" + texts[i].Tag, + 60, 1L, 0L, elapsedMicros[i % 2], 0L, 0L, texts[i].Text); + } + + break; + } + + case "get_query_store_top": + for (int i = 0; i < texts.Count; i++) + { + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO query_store_stats (collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, execution_type_desc, last_execution_time, module_name, query_text, query_hash, query_plan_hash, execution_count, avg_duration_us, avg_cpu_time_us, avg_logical_io_reads, avg_logical_io_writes, avg_physical_io_reads, avg_rowcount) +VALUES ($1,$2,$3,$4,$5,$6,$7,'Regular',$2,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(now.AddMinutes(-30 - i)), serverId, serverName, Db, + 100L + i, 200L + i, null, texts[i].Text, "0xQSH" + texts[i].Tag, "0xQSP" + texts[i].Tag, + 100L, 5000L, 4000L, 100L, 10L, 5L, 10L); + } + + break; + + case "get_query_store_regressions": + for (int i = 0; i < texts.Count; i++) + { + long queryId = 1000 + i; + foreach (var (at, executions, avg, interval, text) in new[] + { + (now.AddHours(-40), 50L, 1000L, 2L * i + 1, "SELECT 1"), + (now.AddMinutes(-30), 200L, 4000L, 2L * i + 2, texts[i].Text), + }) + { + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO query_store_stats (collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, execution_type_desc, execution_count, avg_duration_us, avg_cpu_time_us, avg_logical_io_reads, runtime_stats_interval_id, query_text, last_execution_time) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(at), serverId, serverName, Db, queryId, 9L, "Regular", + executions, avg, avg, 100L, interval, text, DarlingMcpTestData.Naive(at)); + } + } + + break; + + case "get_plan_corrections": + for (int i = 0; i < texts.Count; i++) + { + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO plan_correction (collection_id, collection_time, server_id, server_name, database_name, recommendation_name, recommendation_state, query_id, query_text, regressed_plan_id, last_good_plan_id, score) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(now.AddMinutes(-20 - i)), serverId, serverName, Db, + "PlanRegression_ssf_" + texts[i].Tag, "Active", 5000L + i, texts[i].Text, 6000L + i, 7000L + i, 80); + } + + break; + + default: + throw new ArgumentOutOfRangeException(nameof(tool), tool, null); + } + } + + /// True when a stored statement column of the server's rows still holds the canary (the tables the cases plant in). + private static async Task StoreHoldsTheCanaryAsync(NpgsqlConnection connection, int serverId, CancellationToken ct) + { + await using var cmd = new NpgsqlCommand( + @"SELECT EXISTS (SELECT 1 FROM query_snapshots WHERE server_id = $1 AND position('S3cret-canary-ssf' IN query_text) > 0) + OR EXISTS (SELECT 1 FROM query_stats WHERE server_id = $1 AND position('S3cret-canary-ssf' IN query_text) > 0) + OR EXISTS (SELECT 1 FROM query_store_stats WHERE server_id = $1 AND position('S3cret-canary-ssf' IN query_text) > 0) + OR EXISTS (SELECT 1 FROM plan_correction WHERE server_id = $1 AND position('S3cret-canary-ssf' IN query_text) > 0)", + connection); + cmd.Parameters.AddWithValue(serverId); + return (bool)(await cmd.ExecuteScalarAsync(ct))!; + } + private static async Task DeleteRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + foreach (int id in new[] { CanaryId, CleanId }) + { + await DarlingMcpTestData.ExecAsync(connection, ct, + $"DELETE FROM query_snapshots WHERE server_id = {id}; DELETE FROM query_stats WHERE server_id = {id}; " + + $"DELETE FROM query_store_stats WHERE server_id = {id}; DELETE FROM plan_correction WHERE server_id = {id}; " + + $"DELETE FROM servers WHERE server_id = {id}; DELETE FROM config_monitored_servers WHERE server_id = {id};"); + } + } +} diff --git a/Darling/Darling.Tests/StatementFilterWarmUp.cs b/Darling/Darling.Tests/StatementFilterWarmUp.cs new file mode 100644 index 0000000000..b3fd1cc605 --- /dev/null +++ b/Darling/Darling.Tests/StatementFilterWarmUp.cs @@ -0,0 +1,37 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Threading; +using System.Threading.Tasks; +using PerformanceMonitor.Alerting; + +namespace Darling.Tests; + +/// +/// Runs the statement filter's start-up warm-up once per test process, for the tests that judge a document of +/// 1 MB or more (#5478, following #5477). +/// The service (DarlingWorker.ExecuteAsync) and Lite (MainWindow) both run +/// at start, so no real alert walks a large report in a process whose +/// filter code is still unoptimized (tiered JIT). A test process never runs that start path, so the first large +/// document a test judged paid for the cold code: 0.5 to 2 s for 4 MB on the laptop, against 22 to 72 ms warm. The +/// first test in the process to walk a 4 MB report could therefore use up its whole judging budget, which depended +/// on test order and on what ran beside it. This helper gives the tests the same start path the product has. It is +/// not a budget change: no limit, constant or assertion differs, and no product code knows about tests. +/// +internal static class StatementFilterWarmUp +{ + private static readonly Lazy s_once = new( + () => AlertStatementFilter.WarmUpAsync(), LazyThreadSafetyMode.ExecutionAndPublication); + + /// Awaits the warm-up with its default probe; the first caller starts it, later callers share it. + public static Task EnsureAsync() => s_once.Value; + + /// The same for a test that is not async. The warm-up never throws, and it runs on the thread pool. + public static void Ensure() => s_once.Value.GetAwaiter().GetResult(); +} diff --git a/Darling/Darling.Tests/StatementFilterWarmUpSourceTests.cs b/Darling/Darling.Tests/StatementFilterWarmUpSourceTests.cs new file mode 100644 index 0000000000..0f569900a5 --- /dev/null +++ b/Darling/Darling.Tests/StatementFilterWarmUpSourceTests.cs @@ -0,0 +1,54 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System.IO; +using System.Runtime.CompilerServices; +using System.Text.RegularExpressions; +using Xunit; + +namespace Darling.Tests; + +/// #5320 N1: the statement filter's judge takes 250-900 ms to build, and the first value that reaches it pays +/// that. Each host that judges on a UI thread or on a request a user waits on starts the build on a background thread +/// at startup. This pins that each start path still does: Lite's desktop app (its connection and AG alerts run the +/// filter on the UI thread, and its MCP server shares the process) and the Darling service (its MCP tools, plan +/// analysis and alerts). The Darling viewer and the web host judge through the PostgreSQL predicate and never build +/// the .NET judge, so they have nothing to warm. +[Trait("Reads", "Lite")] +public sealed class StatementFilterWarmUpSourceTests +{ + private static readonly Regex s_warmUpCall = new( + @"^\s*_\s*=\s*Task\.Run\(\s*SensitiveStatements\.WarmUp\s*\)\s*;", RegexOptions.CultureInvariant | RegexOptions.Multiline); + + [Theory] + [InlineData("Lite/App.xaml.cs")] + [InlineData("Darling/PerformanceMonitor.Darling.Service/Program.cs")] + public void EachStartPathStartsTheJudgeWarmUpOffTheCallingThread(string relative) + { + var source = File.ReadAllText(Path.Combine(RepoRoot(), relative.Replace('/', Path.DirectorySeparatorChar))); + + // A statement on its own line, so a call inside a comment does not count. + Assert.True( + s_warmUpCall.IsMatch(source), + $"{relative} does not start SensitiveStatements.WarmUp on a background task (#5320 N1)"); + } + + private static string RepoRoot([CallerFilePath] string thisFile = "") + { + var dir = Path.GetDirectoryName(thisFile)!; + while (dir is not null + && !File.Exists(Path.Combine(dir, "PerformanceMonitor.sln")) + && !Directory.Exists(Path.Combine(dir, ".git"))) + { + dir = Path.GetDirectoryName(dir); + } + + Assert.NotNull(dir); + return dir!; + } +} diff --git a/Darling/Darling.Tests/StatementFilterWarmUpWaitTests.cs b/Darling/Darling.Tests/StatementFilterWarmUpWaitTests.cs new file mode 100644 index 0000000000..c9f8665d15 --- /dev/null +++ b/Darling/Darling.Tests/StatementFilterWarmUpWaitTests.cs @@ -0,0 +1,475 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Threading; +using System.Threading.Tasks; +using ModelContextProtocol.Protocol; +using PerformanceMonitor.Alerting; +using PerformanceMonitor.Common; +using PerformanceMonitor.Notifications; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5478, #5484: a call that judges a large document waits for a running start-up warm-up, but only until one deadline +/// (3 s after the warm-up started), only when it judges at least 256 KB, and the alert engine's fire path awaits instead +/// of blocking. The MCP and web sweep waits the same way. +/// +[Collection("timing")] +public sealed class StatementFilterWarmUpWaitTests +{ + private static long Ticks(TimeSpan span) => (long)(span.TotalSeconds * Stopwatch.Frequency); + + /// A fake warm-up that has been running for . Disposing it finishes it and clears the + /// registration, so a failing assertion cannot leave a running fake for the next test to wait on. + private sealed class StagedWarmUp : IDisposable + { + public StagedWarmUp(TimeSpan age) + { + Source = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + Started = Stopwatch.GetTimestamp() - Ticks(age); + SensitiveStatements.SetWarmUp(Source.Task, Started); + } + + public TaskCompletionSource Source { get; } + + public long Started { get; } + + public void Dispose() + { + Source.TrySetResult(); + SensitiveStatements.SetWarmUp(null, 0); + } + } + + /// A blocked-process report of about 300,000 characters (above the 256 KB gate) that still names the canary. + private static string LargeReportXml() => + StatementFilterAlertTests.ReportXml().Replace( + "", + "" + new string('x', 300_000) + "", + StringComparison.Ordinal); + + private static AlertContext LargeContext() => new() { AttachmentXml = LargeReportXml() }; + + private static CallToolResult LargeToolResult() => new() + { + Content = new List { new TextContentBlock { Text = "{\"v\":\"" + new string('x', 300_000) + "\"}" } }, + }; + + // ---- the primitive: one deadline for every caller ---- + + [Fact] + public void WaitForWarmUp_ARunningWarmUp_IsWaitedForOnlyUpToTheLimit() + { + var running = new TaskCompletionSource(); + Assert.Equal(TimeSpan.FromSeconds(3), SensitiveStatements.WarmUpWaitLimit); + Assert.Equal(256 * 1024, SensitiveStatements.WarmUpWaitGateChars); + + var watch = Stopwatch.StartNew(); + var finished = SensitiveStatements.WaitForWarmUp(running.Task, Stopwatch.GetTimestamp(), TimeSpan.FromMilliseconds(300)); + watch.Stop(); + + Assert.False(finished); + Assert.InRange(watch.ElapsedMilliseconds, 250, 2500); + + // The same task, finished a moment later, ends the wait early. + _ = Task.Run(async () => { await Task.Delay(200); running.SetResult(); }); + watch.Restart(); + Assert.True(SensitiveStatements.WaitForWarmUp(running.Task, Stopwatch.GetTimestamp(), TimeSpan.FromSeconds(20))); + Assert.InRange(watch.ElapsedMilliseconds, 0, 5000); + } + + [Fact] + public void WaitForWarmUp_NoWarmUp_AFinishedOne_OrOneThatAlreadyFailed_DelaysNoOne() + { + var failed = Task.FromException(new InvalidOperationException("boom")); + var watch = Stopwatch.StartNew(); + Assert.True(SensitiveStatements.WaitForWarmUp(null, Stopwatch.GetTimestamp(), TimeSpan.FromSeconds(20))); + Assert.True(SensitiveStatements.WaitForWarmUp(Task.CompletedTask, Stopwatch.GetTimestamp(), TimeSpan.FromSeconds(20))); + Assert.True(SensitiveStatements.WaitForWarmUp(failed, Stopwatch.GetTimestamp(), TimeSpan.FromSeconds(20))); + watch.Stop(); + Assert.True(watch.ElapsedMilliseconds < 1000, "took " + watch.ElapsedMilliseconds + " ms"); + _ = failed.Exception; + } + + [Fact] + public void WaitForWarmUp_AWarmUpThatFaultsWhileWaitedOn_ReturnsFalseAtOnce_AndNeverThrows() + { + var source = new TaskCompletionSource(); + _ = Task.Run(async () => + { + await Task.Delay(100); + source.SetException(new InvalidOperationException("boom")); + }); + + var watch = Stopwatch.StartNew(); + var finished = SensitiveStatements.WaitForWarmUp(source.Task, Stopwatch.GetTimestamp(), TimeSpan.FromSeconds(20)); + watch.Stop(); + + Assert.False(finished); + Assert.InRange(watch.ElapsedMilliseconds, 0, 5000); + Assert.NotNull(source.Task.Exception); + } + + [Fact] + public void WaitForWarmUp_AfterTheDeadline_NobodyWaits_AndTheFleetPaysTheLimitOnce() + { + var running = new TaskCompletionSource(); + var limit = TimeSpan.FromMilliseconds(500); + var started = Stopwatch.GetTimestamp(); + + // Three callers one after another: the first waits out the deadline, the other two find it already passed. + var watch = Stopwatch.StartNew(); + Assert.False(SensitiveStatements.WaitForWarmUp(running.Task, started, limit)); + var afterFirst = watch.ElapsedMilliseconds; + Assert.False(SensitiveStatements.WaitForWarmUp(running.Task, started, limit)); + Assert.False(SensitiveStatements.WaitForWarmUp(running.Task, started, limit)); + watch.Stop(); + + Assert.InRange(afterFirst, 400, 2500); + Assert.True(watch.ElapsedMilliseconds - afterFirst < 250, + "the callers after the deadline waited " + (watch.ElapsedMilliseconds - afterFirst) + " ms"); + + // A warm-up that started long ago and never finishes costs a new caller nothing. + var old = Stopwatch.GetTimestamp() - Ticks(TimeSpan.FromSeconds(10)); + watch.Restart(); + Assert.False(SensitiveStatements.WaitForWarmUp(running.Task, old, TimeSpan.FromSeconds(3))); + Assert.True(watch.ElapsedMilliseconds < 250, "took " + watch.ElapsedMilliseconds + " ms"); + running.SetResult(); + } + + [Fact] + public async Task WhenWarmAsync_ATimedOutAsyncWait_IsNotFollowedByASecondWaitInTheSyncOne() + { + var running = new TaskCompletionSource(); + var limit = TimeSpan.FromMilliseconds(500); + var started = Stopwatch.GetTimestamp(); + + var watch = Stopwatch.StartNew(); + await SensitiveStatements.WhenWarmAsync(running.Task, started, limit); + var afterAsync = watch.ElapsedMilliseconds; + Assert.InRange(afterAsync, 400, 2500); + + Assert.False(SensitiveStatements.WaitForWarmUp(running.Task, started, limit)); + watch.Stop(); + Assert.True(watch.ElapsedMilliseconds - afterAsync < 250, + "the sync wait after the async one took " + (watch.ElapsedMilliseconds - afterAsync) + " ms"); + running.SetResult(); + } + + [Fact] + public async Task WhenWarmAsync_IsCompleteWithNoWarmUp_AFinishedOne_AFailedOne_OrAnExpiredDeadline_AndNeverThrows() + { + var failed = new TaskCompletionSource(); + failed.SetException(new InvalidOperationException("boom")); + var stuck = new TaskCompletionSource(); + + Assert.True(SensitiveStatements.WhenWarmAsync(null, 0, TimeSpan.FromSeconds(3)).IsCompleted); + Assert.True(SensitiveStatements.WhenWarmAsync(Task.CompletedTask, 0, TimeSpan.FromSeconds(3)).IsCompleted); + Assert.True(SensitiveStatements.WhenWarmAsync(failed.Task, Stopwatch.GetTimestamp(), TimeSpan.FromSeconds(3)).IsCompleted); + Assert.True(SensitiveStatements.WhenWarmAsync( + stuck.Task, Stopwatch.GetTimestamp() - Ticks(TimeSpan.FromSeconds(10)), TimeSpan.FromSeconds(3)).IsCompleted); + + // A warm-up that faults while it is awaited ends the wait without throwing. + var faulting = new TaskCompletionSource(); + _ = Task.Run(async () => { await Task.Delay(100); faulting.SetException(new InvalidOperationException("boom")); }); + await SensitiveStatements.WhenWarmAsync(faulting.Task, Stopwatch.GetTimestamp(), TimeSpan.FromSeconds(20)) + .WaitAsync(TimeSpan.FromSeconds(20)); + + _ = failed.Task.Exception; + _ = faulting.Task.Exception; + stuck.SetResult(); + } + + // ---- the size gate ---- + + [Fact] + public async Task TheGate_BelowTheThreshold_NobodyWaits_AtTheThreshold_TheCallWaits() + { + await StatementFilterWarmUp.EnsureAsync(); + using var staged = new StagedWarmUp(TimeSpan.Zero); + + var watch = Stopwatch.StartNew(); + SensitiveStatements.WaitForWarmUp(SensitiveStatements.WarmUpWaitGateChars - 1); + watch.Stop(); + Assert.True(watch.ElapsedMilliseconds < 500, "a call below the gate waited " + watch.ElapsedMilliseconds + " ms"); + + _ = Task.Run(async () => { await Task.Delay(400); staged.Source.SetResult(); }); + watch.Restart(); + SensitiveStatements.WaitForWarmUp(SensitiveStatements.WarmUpWaitGateChars); + watch.Stop(); + Assert.InRange(watch.ElapsedMilliseconds, 300, 2900); + } + + [Fact] + public async Task Apply_OfASmallAlert_DuringARunningWarmUp_DoesNotWait() + { + await StatementFilterWarmUp.EnsureAsync(); + using var staged = new StagedWarmUp(TimeSpan.Zero); + + var watch = Stopwatch.StartNew(); + var filtered = AlertStatementFilter.Apply(new AlertContext { AttachmentXml = StatementFilterAlertTests.ReportXml() }); + watch.Stop(); + + Assert.True(watch.ElapsedMilliseconds < 1000, "took " + watch.ElapsedMilliseconds + " ms"); + StatementFilterAlertTests.AssertNoSecret(filtered!.AttachmentXml!); + } + + // ---- the alert path through the product's own warm-up ---- + + [Fact] + public async Task Apply_DuringASlowWarmUp_WaitsForItAndThenJudgesTheContext() + { + // Settle the process's own warm-up first, so no other test replaces the one this test staged. + await StatementFilterWarmUp.EnsureAsync(); + using var release = new ManualResetEventSlim(false); + var slow = AlertStatementFilter.WarmUpAsync(() => release.Wait(TimeSpan.FromSeconds(20))); + try + { + var entered = new TaskCompletionSource(); + long returnedAt = 0; + var apply = Task.Run(() => + { + entered.SetResult(); + var result = AlertStatementFilter.Apply(LargeContext()); + returnedAt = Stopwatch.GetTimestamp(); + return result; + }); + + // Apply has started; had it not waited, a judged 300 KB document would be back long before this. + await entered.Task.WaitAsync(TimeSpan.FromSeconds(20)); + await Task.Delay(500); + Assert.False(apply.IsCompleted, "Apply did not wait for the warm-up that was still running"); + + var releasedAt = Stopwatch.GetTimestamp(); + release.Set(); + var filtered = await apply.WaitAsync(TimeSpan.FromSeconds(20)); + Assert.True(returnedAt >= releasedAt, "Apply returned before the warm-up was released"); + + StatementFilterAlertTests.AssertNoSecret(filtered!.AttachmentXml!); + Assert.Contains(StatementScrubCanary.PlainStatement, filtered.AttachmentXml!, StringComparison.Ordinal); + } + finally + { + release.Set(); + await slow; + } + } + + [Fact] + public async Task TheWarmUpsOwnApplyCalls_NeverWaitOnTheWarmUp() + { + await StatementFilterWarmUp.EnsureAsync(); + long insideMs = -1; + var warmUp = AlertStatementFilter.WarmUpAsync(() => + { + // Large enough to be gated: only the warm-up's own flag keeps these from waiting the whole limit on themselves. + var watch = Stopwatch.StartNew(); + _ = AlertStatementFilter.Apply(LargeContext()); + _ = AlertStatementFilter.Apply(new FindingAlert("A", "SRV", "1", "1", "101", LargeContext(), 0.9, 0.5, "plain", true)); + insideMs = watch.ElapsedMilliseconds; + }); + + await warmUp.WaitAsync(TimeSpan.FromSeconds(20)); + Assert.InRange(insideMs, 0, 2000); + } + + [Fact] + public async Task TheRegistrationIsWrittenBeforeTheWarmUpCanRun() + { + await StatementFilterWarmUp.EnsureAsync(); + for (var i = 0; i < 50; i++) + { + // The first thing the work does is ask who is registered: it must be this warm-up, never the previous one. + Task? seenByTheWork = null; + var warmUp = SensitiveStatements.StartWarmUp(() => + { + seenByTheWork = SensitiveStatements.RegisteredWarmUp; + return Task.CompletedTask; + }); + await warmUp.WaitAsync(TimeSpan.FromSeconds(20)); + Assert.Same(warmUp, seenByTheWork); + } + } + + [Fact] + public async Task Apply_AfterAWarmUpThatThrew_IsNotDelayed() + { + await StatementFilterWarmUp.EnsureAsync(); + await AlertStatementFilter.WarmUpAsync(() => throw new InvalidOperationException("boom")); + + var watch = Stopwatch.StartNew(); + var filtered = AlertStatementFilter.Apply(LargeContext()); + watch.Stop(); + + Assert.True(watch.ElapsedMilliseconds < 2000, "took " + watch.ElapsedMilliseconds + " ms"); + StatementFilterAlertTests.AssertNoSecret(filtered!.AttachmentXml!); + } + + [Fact] + public async Task TheAlertWarmUpSwallowsAThrownException_AndRunsTheDefaultProbe() + { + await AlertStatementFilter.WarmUpAsync(() => throw new InvalidOperationException("boom")); + await AlertStatementFilter.WarmUpAsync(); + } + + // ---- the engine's fire path awaits ---- + + [Fact] + public async Task TheEnginesFirePath_DuringARunningWarmUp_AwaitsInsteadOfBlockingTheCaller() + { + await StatementFilterWarmUp.EnsureAsync(); + var h = StatementFilterAlertTests.BlockingHarness(out _); + h.Adapter.Blocking[0].BlockedProcessReportXml = LargeReportXml(); + using var staged = new StagedWarmUp(TimeSpan.Zero); + + var engine = h.Build(); + var watch = Stopwatch.StartNew(); + var evaluation = engine.EvaluateServerAsync(AlertEngineTests.Harness.Snapshot()); + watch.Stop(); + + // A blocking wait would have held this thread for the rest of the warm-up before the call returned. + Assert.True(watch.ElapsedMilliseconds < 1500, "the fire path blocked the caller for " + watch.ElapsedMilliseconds + " ms"); + await Task.Delay(300); + Assert.False(evaluation.IsCompleted, "the alert was delivered while the warm-up was still running"); + Assert.Empty(h.Deliverer.Outcomes); + + staged.Source.SetResult(); + await evaluation.WaitAsync(TimeSpan.FromSeconds(30)); + + var outcome = Assert.Single(h.Deliverer.Outcomes); + StatementFilterAlertTests.AssertNoSecret(StatementFilterAlertTests.Everything(outcome)); + } + + // ---- the MCP and web sweep ---- + + [Fact] + public async Task TheSweep_OfALargeResult_WaitsForARunningWarmUp_AndASmallOneDoesNot() + { + await StatementFilterWarmUp.EnsureAsync(); + using var staged = new StagedWarmUp(TimeSpan.Zero); + + var small = new CallToolResult { Content = new List { new TextContentBlock { Text = "{\"v\":1}" } } }; + var watch = Stopwatch.StartNew(); + Assert.Same(small, SensitiveStatementOutputFilter.Sweep(small)); + watch.Stop(); + Assert.True(watch.ElapsedMilliseconds < 500, "a small result waited " + watch.ElapsedMilliseconds + " ms"); + + _ = Task.Run(async () => { await Task.Delay(400); staged.Source.SetResult(); }); + var large = LargeToolResult(); + watch.Restart(); + var swept = SensitiveStatementOutputFilter.Sweep(large); + watch.Stop(); + + Assert.InRange(watch.ElapsedMilliseconds, 300, 2900); + Assert.Same(large, swept); + } + + // ---- #5484: sizing runs only while a warm-up can still make the call wait ---- + + [Fact] + public async Task WarmUpPending_IsTrueOnlyWhileAWarmUpRunsInsideItsDeadline() + { + await StatementFilterWarmUp.EnsureAsync(); + + // No warm-up registered. + SensitiveStatements.SetWarmUp(null, 0); + Assert.False(SensitiveStatements.WarmUpPending); + + // One running before its 3 s deadline. + using (var staged = new StagedWarmUp(TimeSpan.Zero)) + { + Assert.True(SensitiveStatements.WarmUpPending); + + // The same warm-up once it finished. + staged.Source.SetResult(); + Assert.False(SensitiveStatements.WarmUpPending); + } + + // One still running, but past the deadline. + using (var late = new StagedWarmUp(TimeSpan.FromSeconds(4))) + { + Assert.False(late.Source.Task.IsCompleted); + Assert.False(SensitiveStatements.WarmUpPending); + } + + // Inside the warm-up itself: it never waits on itself, so it is never pending there. + var insideSaw = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var started = SensitiveStatements.StartWarmUp(() => + { + insideSaw.SetResult(SensitiveStatements.WarmUpPending); + return Task.CompletedTask; + }); + Assert.False(await insideSaw.Task); + await started; + SensitiveStatements.SetWarmUp(null, 0); + } + + [Fact] + public void TheSizedWait_SizesTheCallOnlyWhileAWarmUpIsPending() + { + var sized = 0; + long Size() { sized++; return 10; } + + SensitiveStatements.SetWarmUp(null, 0); + SensitiveStatements.WaitForWarmUp(Size); + Assert.Equal(0, sized); + + using (var staged = new StagedWarmUp(TimeSpan.Zero)) + { + SensitiveStatements.WaitForWarmUp(Size); + Assert.Equal(1, sized); + + staged.Source.SetResult(); + SensitiveStatements.WaitForWarmUp(Size); + Assert.Equal(1, sized); + } + + using (var late = new StagedWarmUp(TimeSpan.FromSeconds(4))) + { + SensitiveStatements.WaitForWarmUp(Size); + Assert.Equal(1, sized); + } + } + + [Fact] + public async Task TheSweep_SizesTheResultOnlyWhileAWarmUpIsPending() + { + await StatementFilterWarmUp.EnsureAsync(); + var result = LargeToolResult(); + + // Seam: SensitiveStatementOutputFilter.SizingCalls counts SweptChars calls, because the structured content's + // GetRawText (the copy that matters) is not observable from a test. The collection is serial, so no other + // test moves the counter. + SensitiveStatements.SetWarmUp(null, 0); + var before = SensitiveStatementOutputFilter.SizingCalls; + SensitiveStatementOutputFilter.Sweep(result); + Assert.Equal(before, SensitiveStatementOutputFilter.SizingCalls); + + using (var staged = new StagedWarmUp(TimeSpan.Zero)) + { + _ = Task.Run(async () => { await Task.Delay(100); staged.Source.SetResult(); }); + SensitiveStatementOutputFilter.Sweep(result); + Assert.Equal(before + 1, SensitiveStatementOutputFilter.SizingCalls); + + // Finished: sized no more. + SensitiveStatementOutputFilter.Sweep(result); + Assert.Equal(before + 1, SensitiveStatementOutputFilter.SizingCalls); + } + + using (var late = new StagedWarmUp(TimeSpan.FromSeconds(4))) + { + SensitiveStatementOutputFilter.Sweep(result); + Assert.Equal(before + 1, SensitiveStatementOutputFilter.SizingCalls); + } + } +} diff --git a/Darling/Darling.Tests/StatementFilterWebTests.cs b/Darling/Darling.Tests/StatementFilterWebTests.cs new file mode 100644 index 0000000000..dca3a93ebb --- /dev/null +++ b/Darling/Darling.Tests/StatementFilterWebTests.cs @@ -0,0 +1,323 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Diagnostics; +using System.IO; +using System.Linq; +using System.Text; +using System.Text.Json; +using System.Text.Json.Nodes; +using System.Threading.Tasks; +using Microsoft.AspNetCore.Http; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging.Abstractions; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348: the web side of the statement filter. Every /api/read/* route answers through +/// DarlingWebEndpoints.ToHttpResult, the mute-rule routes through MuteRuleToolResult, and triage, the +/// alert notebook and the fleet sweep feed write their own body through DarlingWebStatementSweep.JsonText. Each +/// case hands the writer a tool answer that still holds the canary (the raw answer: the MCP host's filter is not in the +/// web path) and reads back what a browser would get. The tests are RED on a writer with no sweep, because the canary +/// then comes through. +/// +[Collection("timing")] +public sealed class StatementFilterWebTests +{ + private static readonly string Canary = StatementScrubCanary.CanaryStatement; + private static readonly string Plain = StatementScrubCanary.PlainStatement; + + private static async Task<(int Status, string Body)> RunAsync(IResult result) + { + var context = new DefaultHttpContext + { + RequestServices = new ServiceCollection().AddLogging().BuildServiceProvider(), + }; + context.Response.Body = new MemoryStream(); + await result.ExecuteAsync(context); + context.Response.Body.Position = 0; + return (context.Response.StatusCode, Encoding.UTF8.GetString(((MemoryStream)context.Response.Body).ToArray())); + } + + private static Task<(int Status, string Body)> ReadRoute(string tool, string raw) => + RunAsync(DarlingWebEndpoints.ToHttpResult(raw, "/api/read/" + tool, NullLogger.Instance, 1)); + + private static void AssertWithheld(string body) + { + foreach (string needle in StatementScrubCanary.SecretNeedles) Assert.DoesNotContain(needle, body); + Assert.Contains(StatementFilterCensus.Marker, body); + } + + [Fact] + public async Task GetActiveQueries_WithholdsANamedStatement_AndKeepsAPlainOne() + { + string raw = new JsonObject + { + ["server"] = "srv-a", + ["queries"] = new JsonArray( + new JsonObject { ["session_id"] = 55, ["query_text"] = Canary }, + new JsonObject { ["session_id"] = 56, ["query_text"] = Plain }), + }.ToJsonString(); + StatementFilterCensus.AssertRawHoldsTheCanary(raw); + + var (status, body) = await ReadRoute("get_active_queries", raw); + + Assert.Equal(200, status); + AssertWithheld(body); + Assert.Contains(Plain, body); + Assert.Equal(55, JsonNode.Parse(body)!["queries"]![0]!["session_id"]!.GetValue()); + } + + [Fact] + public async Task GetPlanXml_WrappedJson_WithholdsStatementOne_AndKeepsStatementTwo() + { + string plan = StatementScrubCanary.CanaryPlan(); + string raw = DarlingWebEndpoints.WrapPlanXml(plan, "0xABCD", "Orders"); + StatementFilterCensus.AssertRawHoldsTheCanary(raw); + + var (status, body) = await ReadRoute("get_plan_xml", raw); + + Assert.Equal(200, status); + string filteredPlan = JsonNode.Parse(body)!["plan_xml"]!.GetValue(); + StatementFilterCensus.AssertPlanFilteredKeepsTheRest(plan, filteredPlan); + Assert.Equal("0xABCD", JsonNode.Parse(body)!["query_hash"]!.GetValue()); + } + + [Fact] + public async Task GetDeadlockDetail_CardsAreWithheld_EvenWhenTheStatementIsCutAtTheCap() + { + // A statement that is named only AFTER the card's cap: the card holds an innocent-looking prefix unless the + // text is judged whole before it is cut. + string longCanary = "/* " + new string('x', DarlingWebDeadlockGraph.SqlTextCap) + " */ " + Canary; + string graph = "" + + "" + longCanary + "" + + "" + Plain + "" + + "" + + "" + + "" + + ""; + string tool = new JsonObject + { + ["deadlocks"] = new JsonArray(new JsonObject { ["deadlock_graph_xml"] = graph }), + }.ToJsonString(); + + string withCards = DarlingWebDeadlockGraph.AddGraphs(tool); + Assert.Contains("\"graph\"", withCards); + + var (status, body) = await ReadRoute("get_deadlock_detail", withCards); + + Assert.Equal(200, status); + AssertWithheld(body); + Assert.Contains(Plain, body); + var processes = JsonNode.Parse(body)!["deadlocks"]![0]!["graph"]!["processes"]!.AsArray(); + Assert.Equal(StatementFilterCensus.Marker, processes[0]!["sql_text"]!.GetValue()); + } + + [Fact] + public async Task AlertsTest_Answer_IsSwept() + { + string raw = new JsonObject + { + ["status"] = "ok", + ["sample_rows"] = new JsonArray(new JsonObject { ["query_text"] = Canary, ["wait_type"] = "LCK_M_X" }), + }.ToJsonString(); + + var (_, body) = await RunAsync(DarlingWebEndpoints.ToHttpResult(raw, "/api/alerts/test", NullLogger.Instance, 1)); + + AssertWithheld(body); + Assert.Contains("LCK_M_X", body); + } + + [Fact] + public async Task MuteRuleEnabled_EchoOfACanaryPattern_IsWithheld_AndKeepsItsStatus() + { + string raw = new JsonObject + { + ["status"] = "ok", + ["rule"] = new JsonObject { ["id"] = "r1", ["enabled"] = true, ["pattern"] = Canary }, + }.ToJsonString(); + + var (status, body) = await RunAsync(DarlingWebEndpoints.MuteRuleToolResult( + raw, "/api/mute-rules/{id}/enabled", NullLogger.Instance, 1)); + + Assert.Equal(200, status); + AssertWithheld(body); + Assert.True(JsonNode.Parse(body)!["rule"]!["enabled"]!.GetValue()); + } + + [Fact] + public async Task AnAnswerThatNamesNothing_IsWrittenByteForByte() + { + string raw = "{\"rows\":[{\"query_text\":\"" + Plain + "\",\"n\":3}]}"; + + var (status, body) = await ReadRoute("get_active_queries", raw); + + Assert.Equal(200, status); + Assert.Equal(raw, body); + } + + [Fact] + public async Task AnAnswerTheSweepCannotRead_IsNeverWritten_ItBecomesTheRefusalSentence() + { + // Opens like JSON and does not parse: the sweep refuses it instead of judging its escaped text. + string raw = "{\"query_text\":\"" + Canary + "\""; + + var (status, body) = await ReadRoute("get_active_queries", raw); + + Assert.Equal(500, status); + Assert.Contains(SensitiveStatements.JsonRefusal, body); + foreach (string needle in StatementScrubCanary.SecretNeedles) Assert.DoesNotContain(needle, body); + } + + [Fact] + public async Task ASlowReadRow_WithACanaryInItsArguments_IsWithheldOnTheWebRead() + { + string raw = new JsonObject + { + ["slow_reads"] = new JsonArray(new JsonObject + { + ["route"] = "analyze_plan_xml", + ["arguments"] = new JsonObject { ["statement"] = Canary }.ToJsonString(), + }), + }.ToJsonString(); + + var (_, body) = await ReadRoute("get_slow_reads", raw); + + AssertWithheld(body); + Assert.Contains("analyze_plan_xml", body); + } + + [Theory] + [InlineData("/api/triage")] + [InlineData("/api/alert-notebook")] + [InlineData("/api/sweeps")] + public async Task AnEndpointsOwnBody_IsSweptBeforeItIsWritten(string route) + { + var body = new JsonObject + { + ["alert"] = new JsonObject { ["detail"] = Canary, ["metric"] = "blocking" }, + ["sections"] = new JsonArray(new JsonObject { ["rows"] = new JsonArray(new JsonObject { ["query_text"] = Plain }) }), + }; + + var (status, written) = await RunAsync(DarlingWebStatementSweep.JsonText(body, route, NullLogger.Instance, 1)); + + Assert.Equal(200, status); + AssertWithheld(written); + Assert.Contains(Plain, written); + Assert.Equal("blocking", JsonNode.Parse(written)!["alert"]!["metric"]!.GetValue()); + } + + [Fact] + public void SlowReadArguments_AreJudgedBeforeTheyAreStored() + { + var named = new JsonObject { ["statement"] = Canary, ["hours"] = 4 }; + string swept = SensitiveStatements.Json(named.ToJsonString()); + + var stored = SlowReadLog.ArgumentsAfterSweep(named, swept)!; + + foreach (string needle in StatementScrubCanary.SecretNeedles) Assert.DoesNotContain(needle, stored.ToJsonString()); + Assert.Contains(StatementFilterCensus.Marker, stored.ToJsonString()); + Assert.Equal(4, stored["hours"]!.GetValue()); + + var plain = new JsonObject { ["statement"] = Plain }; + Assert.Same(plain, SlowReadLog.ArgumentsAfterSweep(plain, SensitiveStatements.Json(plain.ToJsonString()))); + + var refused = SlowReadLog.ArgumentsAfterSweep(plain, SensitiveStatements.JsonRefusal)!; + Assert.DoesNotContain(Plain, refused.ToJsonString()); + Assert.Same(plain, SlowReadLog.ArgumentsAfterSweep(plain, null)); + } + + /// + /// The page's plan panel draws what the read route answered. The answer is the real one (the wrapped canary plan + /// through ToHttpResult); the shipped plan-viewer.js runs on it under Node, and the text it shows is + /// checked: statement 1 (header and parameter values) shows the marker, statement 2's parameters are intact. + /// + [Fact] + public async Task ThePlanPanel_ShowsStatementOneAsTheMarker_AndStatementTwoIntact() + { + var (_, answered) = await ReadRoute("get_plan_xml", DarlingWebEndpoints.WrapPlanXml(StatementScrubCanary.CanaryPlan(), "0xABCD", "Orders")); + string file = Path.Combine(Path.GetTempPath(), "ssf-l7-plan-answer-" + Guid.NewGuid().ToString("N") + ".json"); + File.WriteAllText(file, answered); + try + { + var psi = new ProcessStartInfo("node") { RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false }; + psi.ArgumentList.Add(RepoFile.PathTo("Darling", "Darling.Tests", "web-plan-viewer-harness.mjs")); + psi.ArgumentList.Add(RepoFile.PathTo("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js")); + psi.ArgumentList.Add("answeredPlan"); + psi.ArgumentList.Add(file); + + Process? started = null; + try + { + started = Process.Start(psi); + } + catch (Win32Exception) + { + Assert.Skip("Node is not installed, so the shipped page script cannot be run."); + } + + /* Assert.Skip throws, so a missing Node never reaches here; the null is only a Start that returned no process. */ + using var proc = started ?? throw new InvalidOperationException("Starting the plan viewer harness returned no process."); + { + var error = proc.StandardError.ReadToEndAsync(); + string output = proc.StandardOutput.ReadToEnd().Trim(); + Assert.True(proc.WaitForExit(20000), "the plan viewer harness did not finish in 20 s"); + Assert.True(proc.ExitCode == 0, "the plan viewer harness failed: " + await error); + using var doc = JsonDocument.Parse(output.Split('\n').First(l => l.StartsWith('{'))); + string shown = doc.RootElement.GetProperty("pre").GetString()!; + + foreach (string needle in StatementScrubCanary.SecretNeedles) Assert.DoesNotContain(needle, shown); + Assert.Contains("StatementText=\"" + StatementFilterCensus.Marker + "\"", shown); + Assert.Contains(StatementScrubCanary.PlainStatement, shown); + Assert.Contains("ParameterCompiledValue=\"N'param-canary-ssf'\"", shown); + Assert.Contains("ssf_autoparam_canary", shown); + } + } + finally + { + File.Delete(file); + } + } + + /// The sweep's cost on a read that names nothing: a 1 MB answer of plain statements and numbers. The + /// bound is loose on purpose (the sweep's own budget is 1.5 s); the median is printed for the record. + [Fact] + public async Task AOneMegabyteAnswerThatNamesNothing_IsWrittenUnchanged_AndSweptInsideTheBudget() + { + await StatementFilterWarmUp.EnsureAsync(); + var rows = new JsonArray(); + for (int i = 0; rows.Count < 12000; i++) + { + rows.Add(new JsonObject { ["id"] = i, ["query_text"] = Plain + " /* " + i + " */", ["wait_type"] = "PAGEIOLATCH_SH", ["ms"] = i * 3.5 }); + } + + string raw = new JsonObject { ["rows"] = rows }.ToJsonString(); + Assert.InRange(raw.Length, 1_000_000, 2_000_000); + + var times = new List(); + for (int run = 0; run < 7; run++) + { + var clock = System.Diagnostics.Stopwatch.StartNew(); + var (status, body) = await ReadRoute("get_active_queries", raw); + times.Add(clock.Elapsed.TotalMilliseconds); + Assert.Equal(200, status); + Assert.Equal(raw.Length, body.Length); + } + + times.Sort(); + Console.WriteLine("web sweep, 1 MB no-hit response: median " + times[times.Count / 2].ToString("F0", System.Globalization.CultureInfo.InvariantCulture) + " ms"); + Assert.True(times[times.Count / 2] < 1500, "median " + times[times.Count / 2] + " ms"); + } +} diff --git a/Darling/Darling.Tests/StatementMarkerReaderTests.cs b/Darling/Darling.Tests/StatementMarkerReaderTests.cs new file mode 100644 index 0000000000..5cfee550cc --- /dev/null +++ b/Darling/Darling.Tests/StatementMarkerReaderTests.cs @@ -0,0 +1,425 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Diagnostics; +using System.IO; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using PerformanceMonitor.Analysis; +using PerformanceMonitor.Common; +using PerformanceMonitor.Notifications; +using PerformanceMonitor.PlanAnalysis; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// #4348 / #5320 (R7): the places that READ stored statement text and treat it as a statement must know the +/// withheld marker is not one. A collector that withholds a statement stores +/// in its place, so a reader that re-runs it, seeds a mute rule +/// with it, or groups on it would act on the marker as if it were the text: +/// +/// "Get Actual Plan" re-submits the stored text (, shared by Lite, the +/// Dashboard and Darling), 14 call sites in all; +/// "Mute this alert" pre-fills a query-text pattern from the alert's text (3 desktop dialogs and the web +/// mute-context.js); +/// the same-statement pileup detector falls back to a text hash for a row with no query hash, and the +/// blocking incident grouper keys a chain on its normalized query pair: two DIFFERENT withheld statements +/// would be one. +/// +/// +public sealed class StatementMarkerReaderTests +{ + private const string Marker = SensitiveStatements.PlaceholderText; + private const string Plain = "SELECT canary_plain_ssf FROM dbo.t WHERE c = @c"; + + // ---- "Get Actual Plan" ------------------------------------------------------------------------------- + + private static async Task<(string? Plan, List Scripts, Exception? Error)> RunExecutorAsync(string queryText) + { + var scripts = new List(); + Exception? error = null; + string? plan = null; + try + { + plan = await ActualPlanExecutor.ExecuteForActualPlanAsync( + "Server=unused", "db1", queryText, planXml: null, isolationLevel: null, isAzureSqlDb: false, + timeoutSeconds: 0, ActualPlanExecutor.DefaultProductName, + (cs, db, script, azure, timeout, ct) => + { + scripts.Add(script); + return Task.FromResult(""); + }, + CancellationToken.None); + } + catch (Exception ex) + { + error = ex; + } + + return (plan, scripts, error); + } + + [Fact] + public async Task TheExecutorSendsAPlainStatementToTheServer() + { + var (plan, scripts, error) = await RunExecutorAsync(Plain); + + Assert.Null(error); + Assert.Equal("", plan); + var script = Assert.Single(scripts); + Assert.Contains(Plain, script, StringComparison.Ordinal); + } + + [Theory] + [InlineData(Marker)] + [InlineData(" " + Marker + "\r\n")] + public async Task TheExecutorRefusesTheMarker_AndSendsNothing(string queryText) + { + var (plan, scripts, error) = await RunExecutorAsync(queryText); + + Assert.Empty(scripts); + Assert.Null(plan); + var refusal = Assert.IsType(error); + Assert.Equal( + "This statement's text was withheld (#4348), so it cannot be run to capture an actual plan.", + refusal.Message); + } + + [Fact] + public async Task TheExecutorRefusesAnOldStoredStatementTheFilterWithholds_AndSendsNothing() + { + /* B-L1: a row stored before the filter was on holds raw text, not the marker. */ + const string old = "CREATE LOGIN [a] WITH PASSWORD = 'p'"; + Assert.NotEqual(Marker, old); + Assert.Equal(Marker, SensitiveStatements.Text(old)); + + var (plan, scripts, error) = await RunExecutorAsync(old); + + Assert.Empty(scripts); + Assert.Null(plan); + var refusal = Assert.IsType(error); + Assert.Equal( + "This statement's text was withheld (#4348), so it cannot be run to capture an actual plan.", + refusal.Message); + } + + // ---- "Mute this alert" ------------------------------------------------------------------------------- + + [Fact] + public void AMuteDialogSeedsAPlainStatementTruncatedToTheLimit() + { + var context = new AlertMuteContext { QueryText = Plain }; + Assert.Equal(Plain, context.SeedQueryTextPattern(SensitiveStatements.Text)); + + context.QueryText = new string('x', 500); + Assert.Equal(AlertMuteContext.SeedQueryTextMax, context.SeedQueryTextPattern(SensitiveStatements.Text)!.Length); + } + + [Fact] + public void AMuteDialogSeedsNoPatternFromTheMarker() + { + Assert.Null(new AlertMuteContext { QueryText = Marker }.SeedQueryTextPattern(SensitiveStatements.Text)); + Assert.Null(new AlertMuteContext { QueryText = " " + Marker + " " }.SeedQueryTextPattern(SensitiveStatements.Text)); + Assert.Null(new AlertMuteContext { QueryText = null }.SeedQueryTextPattern(SensitiveStatements.Text)); + + /* The same value arrives through the alert's detail text. */ + var parsed = new AlertMuteContext(); + parsed.PopulateFromDetailText(" Database: db1\n Blocked Query: " + Marker + "\n Wait Type: LCK_M_X\n"); + Assert.Null(parsed.SeedQueryTextPattern(SensitiveStatements.Text)); + Assert.Equal("LCK_M_X", parsed.WaitType); + } + + [Fact] + public void AMuteDialogSeedsNothingFromAPreFilterAlertWhoseStatementTheFilterNames() + { + /* #5320: an alert stored before the filter was on holds raw text. The seed judges the whole text first, so + a statement the filter names seeds nothing, however its secret sits against the 200-character cut. */ + const string old = "CREATE LOGIN [a] WITH PASSWORD = 'p'"; + Assert.Equal(Marker, SensitiveStatements.Text(old)); + + var parsed = new AlertMuteContext(); + parsed.PopulateFromDetailText(" Database: db1\n Blocked Query: " + old + "\n Wait Type: LCK_M_X\n"); + Assert.Equal(old, parsed.QueryText); + Assert.Null(parsed.SeedQueryTextPattern(SensitiveStatements.Text)); + + /* The secret lies past the cut: a cut-then-judge seed would be the clean first 200 characters. */ + var late = new AlertMuteContext { QueryText = new string('x', 250) + " " + old }; + Assert.Null(late.SeedQueryTextPattern(SensitiveStatements.Text)); + + /* A plain statement still seeds its first 200 characters. */ + var plain = new AlertMuteContext { QueryText = "SELECT " + new string('y', 400) }; + Assert.Equal(plain.QueryText![..AlertMuteContext.SeedQueryTextMax], plain.SeedQueryTextPattern(SensitiveStatements.Text)); + + /* A judge that returns null seeds nothing. */ + Assert.Null(new AlertMuteContext { QueryText = Plain }.SeedQueryTextPattern(_ => null)); + } + + private static string RunNode(string script) + { + var psi = new ProcessStartInfo("node") { RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false }; + psi.ArgumentList.Add("--input-type=module"); + psi.ArgumentList.Add("-e"); + psi.ArgumentList.Add(script); + Process proc; + try + { + proc = Process.Start(psi)!; + } + catch (Win32Exception) + { + Assert.Skip("Node is not installed, so the shipped page script cannot be run."); + return ""; + } + + using (proc) + { + var error = proc.StandardError.ReadToEndAsync(); + var output = proc.StandardOutput.ReadToEnd().Trim(); + if (!proc.WaitForExit(20000)) + { + proc.Kill(entireProcessTree: true); + Assert.Fail("node did not finish in 20 s"); + } + + Assert.True(proc.ExitCode == 0, "node failed: " + error.Result); + return output; + } + } + + [Fact] + public void TheWebMuteContextSeedsNoPatternFromTheMarker() + { + /* The shipped file has no package.json beside it, so import a copy named .mjs. */ + var dir = Path.Combine(Path.GetTempPath(), "statement-marker-reader-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(dir); + try + { + var copy = Path.Combine(dir, "mute-context.mjs"); + File.Copy(PathTo("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "mute-context.js"), copy); + var script = + "import { mutePrefillParams } from " + JsonSerializer.Serialize(new Uri(copy).AbsoluteUri) + ";" + + "const row = (d) => ({ server_id: 1, server_name: 'S', metric_name: 'Blocking Detected', detail_text: d });" + + "console.log(JSON.stringify({" + + " marker: mutePrefillParams(row(' Blocked Query: " + Marker + "\\n Wait Type: LCK_M_X\\n'))," + + " plain: mutePrefillParams(row(' Blocked Query: " + Plain + "\\n')) }));"; + using var doc = JsonDocument.Parse(RunNode(script)); + + var marker = doc.RootElement.GetProperty("marker"); + Assert.True( + !marker.TryGetProperty("query_text_pattern", out var seeded) || seeded.ValueKind == JsonValueKind.Null, + "the marker was seeded as a query-text pattern"); + Assert.Equal("LCK_M_X", marker.GetProperty("wait_type_pattern").GetString()); + Assert.Equal(Plain, doc.RootElement.GetProperty("plain").GetProperty("query_text_pattern").GetString()); + } + finally + { + Directory.Delete(dir, recursive: true); + } + } + + [Fact] + public void TheWebMuteContextReadsAPaddedMarkerAsTheMarker() + { + /* B-L3: the C# check (WithheldStatementMarker.IsMarker) trims before it compares; the page script must too. */ + var dir = Path.Combine(Path.GetTempPath(), "statement-marker-reader-pad-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(dir); + try + { + var copy = Path.Combine(dir, "mute-context.mjs"); + File.Copy(PathTo("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "mute-context.js"), copy); + var script = + "import { isWithheldMarker } from " + JsonSerializer.Serialize(new Uri(copy).AbsoluteUri) + ";" + + "console.log(JSON.stringify([isWithheldMarker(" + JsonSerializer.Serialize(Marker) + ")," + + " isWithheldMarker(" + JsonSerializer.Serialize(" " + Marker + "\r\n") + ")," + + " isWithheldMarker(" + JsonSerializer.Serialize(Plain) + "), isWithheldMarker(null)]));"; + using var doc = JsonDocument.Parse(RunNode(script)); + + Assert.Equal([true, true, false, false], doc.RootElement.EnumerateArray().Select(e => e.GetBoolean()).ToArray()); + } + finally + { + Directory.Delete(dir, recursive: true); + } + } + + [Fact] + public void TheWebMuteContextSeedsItsQueryPatternThroughTheMarkerHelper() + { + /* B-L3 (round 2): the test above pins the helper, this one pins its call site. mutePrefillParams must decide + "is this the marker" with isWithheldMarker (which trims), not an exact-equality check of its own that a padded + marker would slip past. */ + var source = File.ReadAllText(PathTo("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "mute-context.js")); + var start = source.IndexOf("export function mutePrefillParams", StringComparison.Ordinal); + Assert.True(start >= 0, "mutePrefillParams was not found in mute-context.js"); + var end = source.IndexOf("\nexport function ", start + 1, StringComparison.Ordinal); + var body = end < 0 ? source[start..] : source[start..end]; + + Assert.Contains("query_text_pattern: isWithheldMarker(d.queryText)", body, StringComparison.Ordinal); + Assert.DoesNotContain("WITHHELD_MARKER", body, StringComparison.Ordinal); + } + + // ---- Same-statement pileup --------------------------------------------------------------------------- + + private static readonly DateTime PileupAt = new(2026, 10, 5, 12, 0, 0, DateTimeKind.Utc); + + private static SameStatementPileupDetector.SnapshotRow PileupRow(DateTime at, int session, long elapsedMs, string text) => + new(at, session, QueryHash: null, text, "db1", "suspended", "PAGEIOLATCH_SH", 5, elapsedMs, 100, 1_000, 10, PreviewText: text); + + private static List Window(params string[] texts) + { + var rows = texts + .Select((t, i) => PileupRow(PileupAt, 100 + i, 20_000, t)) + .ToList(); + /* One sub-second sighting of the first statement's text, a few minutes earlier: the baseline the detector + requires of a statement that is "normally fast". */ + rows.Add(PileupRow(PileupAt.AddMinutes(-3), 90, 50, texts[0])); + return rows; + } + + [Fact] + public void ThreePlainSessionsOnOneTextWithNoHashStillFireAPileup() + { + var detections = SameStatementPileupDetector.Evaluate("srv", Window(Plain, Plain, Plain), PileupAt.AddSeconds(9)); + Assert.Single(detections); + } + + [Fact] + public void ThreeWithheldStatementsWithNoHashAreNotOnePileup() + { + /* Three sessions each running a different withheld statement have nothing in common but the marker. */ + var detections = SameStatementPileupDetector.Evaluate("srv", Window(Marker, Marker, Marker), PileupAt.AddSeconds(9)); + Assert.Empty(detections); + } + + [Fact] + public void AWithheldTextGivesNoStatementIdentity_AndAHashStillWins() + { + Assert.Null(SameStatementPileupDetector.StatementIdentity(PileupRow(PileupAt, 1, 20_000, Marker))); + Assert.Null(SameStatementPileupDetector.StatementIdentity(PileupRow(PileupAt, 1, 20_000, " " + Marker + "\r\n"))); + Assert.NotNull(SameStatementPileupDetector.StatementIdentity(PileupRow(PileupAt, 1, 20_000, Plain))); + + var hashed = PileupRow(PileupAt, 1, 20_000, Marker) with { QueryHash = "0xABCDEF0123456789" }; + Assert.Equal("0xabcdef0123456789", SameStatementPileupDetector.StatementIdentity(hashed)); + } + + // ---- Blocking incident grouping ---------------------------------------------------------------------- + + private static BlockingIncidentGrouper.BlockedEvent Chain(string? blocked, string? blocking, long wait = 5_000) => + new("db1", ContentiousObject: null, blocked, blocking, wait); + + [Fact] + public void TwoSamplesOfOnePlainChainGroupAsOneIncident() + { + var groups = BlockingIncidentGrouper.Group("srv", [Chain(Plain, "UPDATE dbo.t SET c = 1"), Chain(Plain, "UPDATE dbo.t SET c = 1", 9_000)]); + var group = Assert.Single(groups); + Assert.Equal(2, group.OccurrenceCount); + } + + private static BlockingIncidentGrouper.BlockedEvent ModeChain(string? blocked, string? blocking, string? lockMode, long wait = 5_000) => + new("db1", ContentiousObject: null, blocked, blocking, wait, LockMode: lockMode); + + private static string KeyOf(BlockingIncidentGrouper.BlockedEvent e) => + Assert.Single(BlockingIncidentGrouper.Group("srv", [e])).Incident.DedupKey; + + [Theory] + [InlineData(Marker, Marker)] + [InlineData(Marker, "UPDATE dbo.t SET c = 1")] + [InlineData(Plain, Marker)] + public void TheSameWithheldChainKeepsOneDedupKeyWhateverOrderItsSamplesArriveIn(string blocked, string blocking) + { + /* Two cycles carry the same two incidents (X and Y, told apart by lock mode) with their samples in a + different order. The key must come from what the sample says, never from where it sits in the call. */ + var x1 = ModeChain(blocked, blocking, "X", 5_000); + var x2 = ModeChain(blocked, blocking, "X", 9_000); + var y1 = ModeChain(blocked, blocking, "S", 6_000); + + var cycle1 = BlockingIncidentGrouper.Group("srv", [x1, y1, x2]); + var cycle2 = BlockingIncidentGrouper.Group("srv", [y1, x2, x1]); + + Assert.Equal(2, cycle1.Count); + Assert.Equal( + cycle1.Select(g => g.Incident.DedupKey).OrderBy(k => k, StringComparer.Ordinal), + cycle2.Select(g => g.Incident.DedupKey).OrderBy(k => k, StringComparer.Ordinal)); + Assert.Equal(KeyOf(x1), KeyOf(x2)); + } + + [Fact] + public void WithheldChainsThatDifferInNothingVisibleAreOneIncident() + { + /* The stated limit: with the text withheld, the same database, partner statement and lock mode is all + there is to tell chains apart, so those samples are one incident. */ + var group = Assert.Single(BlockingIncidentGrouper.Group("srv", + [ModeChain(Marker, Marker, "X"), ModeChain(Marker, Marker, "X", 9_000)])); + Assert.Equal(2, group.OccurrenceCount); + } + + [Fact] + public void WithheldChainsWithDifferentLockModesAreDifferentIncidents() + { + var groups = BlockingIncidentGrouper.Group("srv", [ModeChain(Marker, Marker, "X"), ModeChain(Marker, Marker, "S")]); + + Assert.Equal(2, groups.Count); + Assert.NotEqual(groups[0].Incident.DedupKey, groups[1].Incident.DedupKey); + } + + [Fact] + public void WithheldChainsWithDifferentVisiblePartnersAreDifferentIncidents() + { + var groups = BlockingIncidentGrouper.Group("srv", + [ModeChain(Marker, "UPDATE dbo.a SET c = 1", "X"), ModeChain(Marker, "UPDATE dbo.b SET c = 1", "X")]); + + Assert.Equal(2, groups.Count); + Assert.NotEqual(groups[0].Incident.DedupKey, groups[1].Incident.DedupKey); + } + + [Fact] + public void AWithheldSideAndAVisibleSideAreNotInterchangeable() + { + const string other = "UPDATE dbo.t SET c = 1"; + var blockedWithheld = KeyOf(ModeChain(Marker, other, "X")); + var blockingWithheld = KeyOf(ModeChain(other, Marker, "X")); + var bothWithheld = KeyOf(ModeChain(Marker, Marker, "X")); + + Assert.NotEqual(blockedWithheld, blockingWithheld); + Assert.NotEqual(blockedWithheld, bothWithheld); + Assert.NotEqual(blockingWithheld, bothWithheld); + } + + [Fact] + public void WithheldChainsThatDifferOnlyByAVisibleLiteralAreOneIncident() + { + var group = Assert.Single(BlockingIncidentGrouper.Group("srv", + [ModeChain(Marker, "UPDATE dbo.t SET c = 1", "X"), ModeChain(Marker, "UPDATE dbo.t SET c = 2", "X", 9_000)])); + Assert.Equal(2, group.OccurrenceCount); + } + + [Fact] + public void TheAnalysisCopyOfTheMarkerEqualsTheCommonOne() + { + /* PerformanceMonitor.Analysis does not reference Common (a reference would change six packages.lock.json + files), so SameStatementPileupDetector carries its own copy of the marker; this pins the two equal. */ + Assert.Equal(SensitiveStatements.PlaceholderText, WithheldStatementMarker.Text); + } + + [Fact] + public void AChainWithAResolvedObjectStillGroupsOnTheObject() + { + /* The object key never reads the text, so a withheld statement on a resolved object is still one incident. */ + var a = new BlockingIncidentGrouper.BlockedEvent("db1", "dbo.Orders", Marker, Marker, 5_000); + var b = new BlockingIncidentGrouper.BlockedEvent("db1", "dbo.Orders", Marker, Marker, 9_000); + var group = Assert.Single(BlockingIncidentGrouper.Group("srv", [a, b])); + Assert.Equal(2, group.OccurrenceCount); + } +} diff --git a/Darling/Darling.Tests/StatementPreviewCutLiveTests.cs b/Darling/Darling.Tests/StatementPreviewCutLiveTests.cs new file mode 100644 index 0000000000..f7d6c8aec8 --- /dev/null +++ b/Darling/Darling.Tests/StatementPreviewCutLiveTests.cs @@ -0,0 +1,125 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Text.Json; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5320: statement previews are filtered, then cut, on the rig. A URI whose secret straddles the preview cut +/// () must reach a client as the marker, never as the first characters +/// of the secret that a cut before the host's sweep leaves behind. +/// +[Collection("live-postgres")] +public sealed class StatementPreviewCutLiveTests +{ + private const string ServerName = "statement-preview-cut-e2e"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + [Fact] + public async Task GetActiveQueries_ThroughTheHost_WithholdsAUriSecretThatStraddlesTheFourHundredCharacterCut() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live preview-cut test."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO query_snapshots (collection_id, collection_time, server_id, server_name, session_id, database_name, query_text, status) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(DateTime.UtcNow.AddMinutes(-5)), ServerId, ServerName, + 91, "UriDb", StatementScrubCanary.UriStatement(400), "running"); + + string raw = await DarlingMcpSessionTools.GetActiveQueries(postgres, ServerName); + Assert.Contains("query_text", raw, StringComparison.Ordinal); + + using var server = await StatementFilterCensus.BuildHostAsync(); + var (text, isError) = await StatementFilterCensus.FilterThroughHostAsync(server, raw); + + Assert.False(isError); + Assert.DoesNotContain(StatementScrubCanary.UriSecretPartial, text, StringComparison.Ordinal); + Assert.Contains(SensitiveStatements.PlaceholderText, text, StringComparison.Ordinal); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + [Fact] + public async Task QueryHeatmap_WithholdsAUriSecret_ThatStraddlesThePreviewCut() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live preview-cut test."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + + var bodySucceeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + var t = DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow).AddMinutes(-20); + await DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO query_stats + (collection_id, collection_time, server_id, server_name, database_name, query_hash, + sample_interval_seconds, delta_execution_count, delta_worker_time, delta_elapsed_time, + delta_logical_reads, delta_logical_writes, query_text) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(t), ServerId, ServerName, + "UriDb", "0xURICUT", 60, 5L, 0L, 250_000L, 0L, 0L, StatementScrubCanary.UriStatement(80)); + + var rows = await DarlingQueryHeatmapReader.GetQueryHeatmapAsync( + postgres, ServerId, HeatmapMetric.Duration, t.AddMinutes(-5), t.AddMinutes(30), + DatabaseFilter.One(null), bucketMinutes: 5, limit: 50, previewLength: 80, ct); + + var row = Assert.Single(rows); + Assert.DoesNotContain(StatementScrubCanary.UriSecretPartial, row.TopQueryText, StringComparison.Ordinal); + Assert.Equal(SensitiveStatements.PlaceholderText, row.TopQueryText); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + private static async Task DeleteRowsAsync(NpgsqlConnection connection, System.Threading.CancellationToken ct) + { + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM query_snapshots WHERE server_id = $1", ServerId); + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM query_stats WHERE server_id = $1", ServerId); + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM servers WHERE server_id = $1", ServerId); + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM config_monitored_servers WHERE server_id = $1", ServerId); + } +} diff --git a/Darling/Darling.Tests/StatementPreviewCutPinTests.cs b/Darling/Darling.Tests/StatementPreviewCutPinTests.cs new file mode 100644 index 0000000000..988a5978d3 --- /dev/null +++ b/Darling/Darling.Tests/StatementPreviewCutPinTests.cs @@ -0,0 +1,102 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using PerformanceMonitor.Common; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5320: a statement preview is filtered, THEN cut. A URI's user:secret@ is named by the at-sign that closes +/// it, so a preview cut BEFORE the at-sign is judged clean and returns part of the secret. These pins walk every cut +/// length over the shipping helper and . +/// +[Trait("Stage", "Guard")] +public sealed class StatementPreviewCutPinTests +{ + /// Text in the corpus's named statements that must never leave in any preview of them. + private static readonly string[] CorpusSecrets = { "S3cret", "secret-x", "sig=x", "0200AB", "'phrase'" }; + + private static IEnumerable<(string Statement, string[] Needles)> Cases() + { + foreach (string statement in SensitiveStatementCorpus.Named) + { + foreach (string variant in SensitiveStatementCorpus.Variants(statement)) + { + string[] needles = CorpusSecrets.Where(n => variant.Contains(n, StringComparison.Ordinal)).ToArray(); + if (needles.Length > 0) yield return (variant, needles); + } + } + + foreach (string scheme in new[] { "https", "ftp", "postgresql" }) + { + yield return ( + "EXEC x @u = N'" + scheme + "://svc:" + StatementScrubCanary.UriSecret + "@host.example/path'", + new[] { StatementScrubCanary.UriSecret, StatementScrubCanary.UriSecretPartial }); + } + + yield return (StatementScrubCanary.UriStatement(400), new[] { StatementScrubCanary.UriSecret, StatementScrubCanary.UriSecretPartial }); + } + + [Fact] + public void ThePremise_APreviewCutBeforeTheFilterHoldsPartOfAUriSecret() + { + string statement = StatementScrubCanary.UriStatement(400); + Assert.True(SensitiveStatements.Names(statement)); + + string cutFirst = McpHelpers.Truncate(statement, 400)!; + Assert.Contains(StatementScrubCanary.UriSecretPartial, cutFirst, StringComparison.Ordinal); + Assert.False(SensitiveStatements.Names(cutFirst), "the cut text is judged clean, so the host's sweep would pass it"); + } + + [Fact] + public void TruncateStatement_AtEveryCutLength_HoldsNoSecretOfTheStatement() + { + int checkedCuts = 0; + foreach (var (statement, needles) in Cases()) + { + Assert.True(SensitiveStatements.Names(statement), "every case is a named statement: " + statement.Trim()); + for (int cut = 0; cut <= statement.Length + 1; cut++) + { + string? preview = McpHelpers.TruncateStatement(statement, cut); + foreach (string needle in needles) + { + Assert.False( + preview != null && preview.Contains(needle, StringComparison.Ordinal), + $"TruncateStatement(cut {cut}) of '{statement.Trim()}' holds '{needle}'"); + } + + string? bare = McpHelpers.StatementPreview(statement, cut); + foreach (string needle in needles) + { + Assert.False( + bare != null && bare.Contains(needle, StringComparison.Ordinal), + $"StatementPreview(cut {cut}) of '{statement.Trim()}' holds '{needle}'"); + } + + checkedCuts++; + } + } + + Assert.True(checkedCuts > 1000, "the walk covered the corpus: " + checkedCuts); + } + + [Fact] + public void ACleanStatement_IsCutLikeTruncate_AndKeepsItsMarker() + { + string plain = StatementScrubCanary.PlainStatement; + Assert.Equal(McpHelpers.Truncate(plain, 10), McpHelpers.TruncateStatement(plain, 10)); + Assert.Equal(plain, McpHelpers.TruncateStatement(plain, plain.Length)); + Assert.Equal(plain[..10], McpHelpers.StatementPreview(plain, 10)); + Assert.Null(McpHelpers.TruncateStatement(null, 10)); + Assert.Equal("", McpHelpers.TruncateStatement("", 10)); + } +} diff --git a/Darling/Darling.Tests/StatementScrubBlockingTests.cs b/Darling/Darling.Tests/StatementScrubBlockingTests.cs new file mode 100644 index 0000000000..9c9ebf1bbe --- /dev/null +++ b/Darling/Darling.Tests/StatementScrubBlockingTests.cs @@ -0,0 +1,504 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Data; +using System.Diagnostics; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using System.Xml.Linq; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Notifications; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348, lane R4: the blocking and deadlock collectors judge each statement string where it first enters a row. The +/// census cases below plant the canary in the collector's INPUT (a DataTableReader), run the read and +/// WritePayload through , and check what was written. Columns covered: +/// blocked_process_report blocked_sql_text, blocking_sql_text, blocked_process_report_xml, blocked_query_plan_xml, +/// blocking_query_plan_xml; deadlocks victim_sql_text, deadlock_graph_xml, victim_query_plan_xml; dmv_blocking_snapshot +/// blocked_sql_text, blocking_sql_text. +/// +[Trait("Stage", "Guard")] +public sealed partial class StatementCollectionCensusTests +{ + private static readonly DateTime EventTime = new(2026, 10, 6, 11, 59, 0, DateTimeKind.Utc); + private const string Frame = ""; + + private static string BlockedReportXml(string blockedText, string blockingText) => + "" + + "" + + "" + Frame + System.Security.SecurityElement.Escape(blockedText) + "" + + "" + System.Security.SecurityElement.Escape(blockedText) + "" + + "" + + "" + System.Security.SecurityElement.Escape(blockingText) + "" + + ""; + + private static string DeadlockGraph(string victimText, string otherText) => + "" + + "" + Frame + + System.Security.SecurityElement.Escape(victimText) + "" + + "" + System.Security.SecurityElement.Escape(victimText) + "" + + "" + System.Security.SecurityElement.Escape(otherText) + "" + + "" + + "" + + "" + + ""; + + private static CollectorContext BlockingContext() => new() + { + ServerId = 42, + ServerName = "test-server", + CollectionTime = new DateTime(2026, 10, 6, 12, 0, 0, DateTimeKind.Utc), + Deltas = null!, + CapturePlanXml = true, + Target = new CollectorTargetInfo(), + }; + + private static DataTableReader BprReader(params (string Xml, string? BlockedPlan, string? BlockingPlan)[] reports) + { + var dataSet = new DataSet(); + var payload = new DataTable("payload"); + payload.Columns.Add("event_time", typeof(DateTime)); + payload.Columns.Add("blocked_process_report_xml", typeof(string)); + payload.Columns.Add("object_id", typeof(int)); + payload.Columns.Add("database_id", typeof(int)); + payload.Columns.Add("contentious_object", typeof(string)); + payload.Columns.Add("blocked_query_plan_xml", typeof(string)); + payload.Columns.Add("blocking_query_plan_xml", typeof(string)); + foreach (var (xml, blocked, blocking) in reports) + { + payload.Rows.Add(EventTime, xml, DBNull.Value, 6, DBNull.Value, (object?)blocked ?? DBNull.Value, (object?)blocking ?? DBNull.Value); + } + + dataSet.Tables.Add(payload); + var gate = new DataTable("gate"); + gate.Columns.Add("execution_count", typeof(long)); + gate.Columns.Add("gated", typeof(bool)); + gate.Rows.Add(1L, false); + dataSet.Tables.Add(gate); + return dataSet.CreateDataReader(); + } + + private static DataTableReader DeadlockReader(params (DateTime Time, string Graph, string? Plan)[] graphs) => + DeadlockReaderWithVictims(graphs.Select(g => (g.Time, g.Graph, g.Plan, "process1")).ToArray()); + + private static DataTableReader DeadlockReaderWithVictims(params (DateTime Time, string Graph, string? Plan, string Victim)[] graphs) + { + var dataSet = new DataSet(); + var payload = new DataTable("payload"); + payload.Columns.Add("deadlock_time", typeof(DateTime)); + payload.Columns.Add("victim_process_id", typeof(string)); + payload.Columns.Add("deadlock_graph_xml", typeof(string)); + payload.Columns.Add("victim_query_plan_xml", typeof(string)); + payload.Columns.Add("source_database_name", typeof(string)); + foreach (var (time, graph, plan, victim) in graphs) + { + payload.Rows.Add(time, victim, graph, (object?)plan ?? DBNull.Value, DBNull.Value); + } + + dataSet.Tables.Add(payload); + var gate = new DataTable("gate"); + gate.Columns.Add("execution_count", typeof(long)); + gate.Columns.Add("gated", typeof(bool)); + gate.Rows.Add(1L, false); + dataSet.Tables.Add(gate); + return dataSet.CreateDataReader(); + } + + private static async Task<(List Rows, StatementScrubRecordingWriter Writer)> RunBprAsync( + CollectorContext context, DataTableReader reader) + { + var rows = await BlockedProcessReportCollector.Instance.ReadAsync(reader, context, CancellationToken.None); + var writer = new StatementScrubRecordingWriter(); + foreach (var row in rows) + { + BlockedProcessReportCollector.Instance.WritePayload(row, writer, context); + } + + return (rows, writer); + } + + private static async Task<(List Rows, StatementScrubRecordingWriter Writer)> RunDeadlocksAsync( + CollectorContext context, DataTableReader reader) + { + var rows = await DeadlocksCollector.Instance.ReadAsync(reader, context, CancellationToken.None); + var writer = new StatementScrubRecordingWriter(); + foreach (var row in rows) + { + DeadlocksCollector.Instance.WritePayload(row, writer, context); + } + + return (rows, writer); + } + + private static void AssertNoNeedle(StatementScrubRecordingWriter writer) + { + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.False(writer.AnyStringContains(needle), "a written value still holds " + needle); + } + } + + private static void AssertPlanScrubbed(string? plan) + { + Assert.NotNull(plan); + XDocument.Parse(plan!); + Assert.Contains(SensitiveStatements.PlaceholderText, plan, StringComparison.Ordinal); + Assert.Contains("canary_plain_ssf", plan, StringComparison.Ordinal); + Assert.Contains("param-canary-ssf", plan, StringComparison.Ordinal); + } + + // ── blocked_process_report: blocked_sql_text, blocking_sql_text, blocked_process_report_xml, + // blocked_query_plan_xml, blocking_query_plan_xml ── + + [Fact] + public async Task BlockedProcessReport_blocked_sql_text_blocking_sql_text_blocked_process_report_xml_and_both_plans_AreWithheld() + { + var xml = BlockedReportXml(StatementScrubCanary.CanaryStatement, StatementScrubCanary.CanaryStatement); + var context = BlockingContext(); + + var (rows, writer) = await RunBprAsync(context, BprReader((xml, StatementScrubCanary.CanaryPlan(), StatementScrubCanary.CanaryPlan()))); + + var row = Assert.Single(rows); + AssertNoNeedle(writer); + Assert.Equal(SensitiveStatements.PlaceholderText, row.BlockedSqlText); + Assert.Equal(SensitiveStatements.PlaceholderText, row.BlockingSqlText); + AssertPlanScrubbed(row.BlockedQueryPlanXml); + AssertPlanScrubbed(row.BlockingQueryPlanXml); + + // The stored report still parses, holds the marker and keeps the frame attributes a reader uses. + var stored = XElement.Parse(row.ReportXml!); + Assert.Contains(SensitiveStatements.PlaceholderText, row.ReportXml, StringComparison.Ordinal); + var frame = stored.Descendants("frame").Single(); + Assert.Equal("0x0200AA01", (string?)frame.Attribute("sqlhandle")); + Assert.Equal("10", (string?)frame.Attribute("stmtstart")); + Assert.Equal("55", (string?)stored.Element("blocked-process")!.Element("process")!.Attribute("spid")); + + // A second run over the same input stores the same text (identity is computed from the filtered value). + var (_, again) = await RunBprAsync(BlockingContext(), BprReader((xml, StatementScrubCanary.CanaryPlan(), StatementScrubCanary.CanaryPlan()))); + Assert.Equal(writer.Strings.ToList(), again.Strings.ToList()); + + // The cycle reports what the filter did. + Assert.Contains(context.Measurements, m => m.Label == CollectorContext.StatementScrubNamedMeasurement && m.Value > 0); + } + + [Fact] + public async Task BlockedProcessReport_AReportWithPlainStatements_IsStoredByteForByte() + { + var xml = BlockedReportXml(StatementScrubCanary.PlainStatement, StatementScrubCanary.PlainStatement); + + var (rows, writer) = await RunBprAsync(BlockingContext(), BprReader((xml, null, null))); + + var row = Assert.Single(rows); + Assert.Same(xml, row.ReportXml); + Assert.Equal(StatementScrubCanary.PlainStatement, row.BlockedSqlText); + Assert.Equal(StatementScrubCanary.PlainStatement, row.BlockingSqlText); + Assert.Contains(StatementScrubCanary.PlainStatement, writer.Strings); + } + + [Fact] + public async Task BlockedProcessReport_AResolvedProcedureName_IsJudgedAgain() + { + // The resolved "schema.object" replaces a placeholder that was not statement text; it is a new string entering + // the row, so it is judged again. A name that reads as a statement is withheld. + var context = BlockingContext(); + var xml = BlockedReportXml("Proc [Database Id = 5 Object Id = 7]", "Proc [Database Id = 5 Object Id = 8]"); + var (rows, _) = await RunBprAsync(context, BprReader((xml, null, null))); + Assert.Equal("Proc [Database Id = 5 Object Id = 7]", rows[0].BlockedSqlText); + + var supplemental = new DataTable(); + supplemental.Columns.Add("database_id", typeof(int)); + supplemental.Columns.Add("object_id", typeof(int)); + supplemental.Columns.Add("database_name", typeof(string)); + supplemental.Columns.Add("schema_name", typeof(string)); + supplemental.Columns.Add("object_name", typeof(string)); + supplemental.Rows.Add(5, 7, "db1", "dbo", "usp_plain"); + supplemental.Rows.Add(5, 8, "db1", "dbo", "usp_other"); + await BlockedProcessReportCollector.Instance.ApplySupplementalAsync(rows, supplemental.CreateDataReader(), context, CancellationToken.None); + + Assert.Contains("usp_plain", rows[0].BlockedSqlText, StringComparison.Ordinal); + Assert.DoesNotContain("Proc [", rows[0].BlockedSqlText, StringComparison.Ordinal); + } + + // ── deadlocks: victim_sql_text, deadlock_graph_xml, victim_query_plan_xml ── + + [Fact] + public async Task Deadlocks_victim_sql_text_deadlock_graph_xml_and_victim_query_plan_xml_AreWithheld() + { + var graph = DeadlockGraph(StatementScrubCanary.CanaryStatement, StatementScrubCanary.PlainStatement); + var context = BlockingContext(); + + var (rows, writer) = await RunDeadlocksAsync(context, DeadlockReader((EventTime, graph, StatementScrubCanary.CanaryPlan()))); + + var row = Assert.Single(rows); + AssertNoNeedle(writer); + Assert.Equal(SensitiveStatements.PlaceholderText, row.VictimSqlText); + AssertPlanScrubbed(row.VictimQueryPlanXml); + + // The stored graph parses, holds the marker, keeps the frame handles, keeps the other process's statement, + // and the object extractor finds the same objects it finds in the raw graph. + var stored = XElement.Parse(row.GraphXml!); + Assert.Contains(SensitiveStatements.PlaceholderText, row.GraphXml, StringComparison.Ordinal); + Assert.Contains(StatementScrubCanary.PlainStatement, row.GraphXml, StringComparison.Ordinal); + var frame = stored.Descendants("frame").Single(); + Assert.Equal("0x0200AA01", (string?)frame.Attribute("sqlhandle")); + Assert.Equal("10", (string?)frame.Attribute("stmtstart")); + var objects = DeadlockObjectExtractor.FromGraphXml(row.GraphXml); + Assert.NotEmpty(objects); + Assert.Equal(DeadlockObjectExtractor.FromGraphXml(graph), objects); + } + + [Fact] + public async Task Deadlocks_ANamedDeadlockReadTwice_IsDroppedTheSecondTime() + { + var graph = DeadlockGraph(StatementScrubCanary.CanaryStatement, StatementScrubCanary.PlainStatement); + + var (first, _) = await RunDeadlocksAsync(BlockingContext(), DeadlockReader((EventTime, graph, null))); + var stored = new HashSet<(DateTime Time, string Graph)>( + first.Select(r => DeadlocksCollector.Instance.GetIdentity(r)!.Value)); + var (second, _) = await RunDeadlocksAsync(BlockingContext(), DeadlockReader((EventTime, graph, null))); + + Assert.Empty(DeadlocksCollector.Instance.DropAlreadyStored(second, stored)); + } + + [Fact] + public async Task Deadlocks_AWholeMarkerGraph_HasAnIdentityFromItsParsedColumns_SoReReadsAreDropped_AndDifferentOnesStay() + { + // A graph that does not parse and holds the canary is withheld whole, so the stored graph is only the marker. + // The identity then comes from the columns read before judging it: the event time, the victim process id and + // the database. Two different deadlocks at the same time (different victims) keep their own identities. + var truncatedA = "" + StatementScrubCanary.CanaryStatement; + var truncatedB = "" + StatementScrubCanary.CanaryStatement; + + var (rows, writer) = await RunDeadlocksAsync(BlockingContext(), + DeadlockReaderWithVictims((EventTime, truncatedA, null, "process1"), (EventTime, truncatedB, null, "process9"))); + + Assert.Equal(2, rows.Count); + Assert.All(rows, r => Assert.Equal(SensitiveStatements.PlaceholderText, r.GraphXml)); + Assert.All(rows, r => Assert.NotNull(DeadlocksCollector.Instance.GetIdentity(r))); + Assert.NotEqual(DeadlocksCollector.Instance.GetIdentity(rows[0]), DeadlocksCollector.Instance.GetIdentity(rows[1])); + AssertNoNeedle(writer); + Assert.Equal(2, DeadlocksCollector.Instance.DropAlreadyStored(rows, new HashSet<(DateTime Time, string Graph)>()).Count); + + // Three reads of the same whole-marker deadlock inside the 10-minute overlap leave one row (R4 recorded + // three before the identity existed). + var stored = new HashSet<(DateTime Time, string Graph)>(); + var kept = 0; + for (var cycle = 0; cycle < 3; cycle++) + { + var (again, _) = await RunDeadlocksAsync(BlockingContext(), DeadlockReader((EventTime, truncatedA, null))); + var survivors = DeadlocksCollector.Instance.DropAlreadyStored(again, stored); + kept += survivors.Count; + foreach (var survivor in survivors) + stored.Add(DeadlocksCollector.Instance.GetIdentity(survivor)!.Value); + } + + Assert.Equal(1, kept); + } + + [Fact] + public async Task Deadlocks_TwoWholeMarkerGraphsAtDifferentTimes_AreBothKept_AndOneWithNoVictimIdHasNoIdentity() + { + var truncated = "" + StatementScrubCanary.CanaryStatement; + + var (rows, _) = await RunDeadlocksAsync(BlockingContext(), + DeadlockReader((EventTime, truncated, null), (EventTime.AddSeconds(30), truncated, null))); + + Assert.Equal(2, DeadlocksCollector.Instance.DropAlreadyStored(rows, + new HashSet<(DateTime Time, string Graph)>()).Count); + var stored = new HashSet<(DateTime Time, string Graph)> { DeadlocksCollector.Instance.GetIdentity(rows[0])!.Value }; + Assert.Single(DeadlocksCollector.Instance.DropAlreadyStored(rows, stored)); + + // No victim process id: nothing tells this row apart from another whole-marker row at its time, so it has + // no identity and is never dropped. + var (noVictim, _) = await RunDeadlocksAsync(BlockingContext(), + DeadlockReaderWithVictims((EventTime, truncated, null, ""))); + Assert.Null(DeadlocksCollector.Instance.GetIdentity(noVictim[0])); + } + + [Fact] + public async Task Deadlocks_APlainGraph_IsStoredByteForByte_WithTheSameIdentityAcrossRuns() + { + var graph = DeadlockGraph(StatementScrubCanary.PlainStatement, StatementScrubCanary.PlainStatement); + + var (rows, _) = await RunDeadlocksAsync(BlockingContext(), DeadlockReader((EventTime, graph, null))); + var (again, _) = await RunDeadlocksAsync(BlockingContext(), DeadlockReader((EventTime, graph, null))); + + Assert.Same(graph, rows[0].GraphXml); + Assert.Equal(StatementScrubCanary.PlainStatement, rows[0].VictimSqlText); + Assert.Equal(DeadlocksCollector.Instance.GetIdentity(rows[0]), DeadlocksCollector.Instance.GetIdentity(again[0])); + } + + // ── dmv_blocking_snapshot: blocked_sql_text, blocking_sql_text ── + + [Fact] + public async Task DmvBlockingSnapshot_blocked_sql_text_and_blocking_sql_text_AreWithheld() + { + var table = new DataTable(); + foreach (var (name, type) in new (string, Type)[] + { + ("database_name", typeof(string)), ("blocked_spid", typeof(int)), ("blocked_ecid", typeof(int)), + ("blocked_last_tran_started", typeof(DateTime)), ("blocking_spid", typeof(int)), ("blocking_ecid", typeof(int)), + ("blocking_last_tran_started", typeof(DateTime)), ("wait_time_ms", typeof(long)), ("lock_mode", typeof(string)), + ("blocking_status", typeof(string)), ("contentious_object", typeof(string)), ("blocked_sql_text", typeof(string)), + ("blocking_sql_text", typeof(string)), ("blocked_login_name", typeof(string)), ("blocked_host_name", typeof(string)), + ("blocked_client_app", typeof(string)), ("blocking_login_name", typeof(string)), ("blocking_host_name", typeof(string)), + ("blocking_client_app", typeof(string)), + }) + { + table.Columns.Add(name, type); + } + + table.Rows.Add("db1", 55, 0, EventTime, 66, 0, EventTime, 9000L, "X", "sleeping", "t1", + StatementScrubCanary.CanaryStatement, StatementScrubCanary.PlainStatement, "l1", "h1", "a1", "l2", "h2", "a2"); + + var context = BlockingContext(); + var rows = await DmvBlockingSnapshotCollector.Instance.ReadAsync(table.CreateDataReader(), context, CancellationToken.None); + var writer = new StatementScrubRecordingWriter(); + foreach (var row in rows) + { + DmvBlockingSnapshotCollector.Instance.WritePayload(row, writer, context); + } + + AssertNoNeedle(writer); + Assert.Equal(SensitiveStatements.PlaceholderText, rows[0].BlockedSqlText); + Assert.Equal(StatementScrubCanary.PlainStatement, rows[0].BlockingSqlText); + Assert.Contains(StatementScrubCanary.PlainStatement, writer.Strings); + } + + // ── timing: a 4 MB ring buffer of reports and graphs ── + + /// The input a timing run judges: about of blocked-process reports, a fifth of + /// them carrying the canary statement. + private static (DataTableReader Reader, int Count, int Bytes) BprInput(int bytes) + { + var reports = new List<(string, string?, string?)>(); + var size = 0; + for (var i = 0; size < bytes; i++) + { + var text = i % 5 == 0 ? StatementScrubCanary.CanaryStatement : "SELECT col" + i + " FROM dbo.t" + i + " WHERE c = @c"; + var xml = BlockedReportXml(text, StatementScrubCanary.PlainStatement + new string('x', 1000)); + reports.Add((xml, null, null)); + size += xml.Length; + } + + return (BprReader(reports.ToArray()), reports.Count, size); + } + + /// The input a timing run judges: about of deadlock graphs, a fifth of them + /// carrying the canary statement. + private static (DataTableReader Reader, int Count, int Bytes) GraphInput(int bytes) + { + var graphs = new List<(DateTime, string, string?)>(); + var size = 0; + for (var i = 0; size < bytes; i++) + { + var text = i % 5 == 0 ? StatementScrubCanary.CanaryStatement : "SELECT col" + i + " FROM dbo.t" + i + " WHERE c = @c"; + var graph = DeadlockGraph(text, StatementScrubCanary.PlainStatement + new string('x', 1000)); + graphs.Add((EventTime.AddSeconds(i), graph, null)); + size += graph.Length; + } + + return (DeadlockReader(graphs.ToArray()), graphs.Count, size); + } + + /// Judges one blocked-process buffer of about and returns the seconds it took, + /// its size in megabytes, and the writer (for the no-leak check). The input is built before the clock starts. + private static async Task<(double Seconds, double Megabytes, int Count, StatementScrubRecordingWriter Writer)> TimeBprAsync(int bytes) + { + var (reader, count, size) = BprInput(bytes); + var watch = Stopwatch.StartNew(); + var (_, writer) = await RunBprAsync(BlockingContext(), reader); + return (watch.Elapsed.TotalSeconds, size / 1_000_000.0, count, writer); + } + + /// The deadlock-graph twin of . + private static async Task<(double Seconds, double Megabytes, int Count, StatementScrubRecordingWriter Writer)> TimeDeadlocksAsync(int bytes) + { + var (reader, count, size) = GraphInput(bytes); + var watch = Stopwatch.StartNew(); + var (_, writer) = await RunDeadlocksAsync(BlockingContext(), reader); + return (watch.Elapsed.TotalSeconds, size / 1_000_000.0, count, writer); + } + + /// + /// The seconds-per-megabyte of the fastest of runs at , with the + /// last run's count and writer. The fastest, not the mean: a busy machine only ever adds time, so the minimum is + /// the run least touched by it. + /// + private static async Task<(double SecondsPerMegabyte, double Seconds, int Count, StatementScrubRecordingWriter Writer)> FastestRateAsync( + Func> run, int bytes, int runs) + { + var best = double.MaxValue; + var bestSeconds = double.MaxValue; + var count = 0; + StatementScrubRecordingWriter? writer = null; + for (var i = 0; i < runs; i++) + { + var result = await run(bytes); + best = Math.Min(best, result.Seconds / result.Megabytes); + bestSeconds = Math.Min(bestSeconds, result.Seconds); + count = result.Count; + writer = result.Writer; + } + + return (best, bestSeconds, count, writer!); + } + + /// + /// A blocked-process ring buffer and a deadlock set of about 4 MB each are judged in time LINEAR in their size. + /// The guard's purpose is a judge that goes quadratic on a big buffer, and it is not a clock test: a fixed + /// seconds ceiling failed on a busy machine, because a busy machine slows a correct judge too. So the 4 MB + /// seconds-per-megabyte is compared with a 1 MB baseline measured in the SAME run, each the fastest of a few + /// runs: load scales both, a quadratic judge makes the 4 MB rate about four times the baseline's, and a linear + /// one keeps the two equal. The 2.5 limit sits between. Both sides take the fastest of three runs (#5561: the big side took two, + /// so a stall across both could fail a correct judge). The absolute ceiling is only a hang backstop, far above + /// anything a loaded machine produces. + /// + [Fact] + public async Task ABlockingRingBufferOfAboutFourMegabytes_IsJudgedInTimeLinearInItsSize() + { + const int Small = 1_000_000; + const int Big = 4_000_000; + const double RateGrowthLimit = 2.5; + const double BaselineFloorSecondsPerMegabyte = 0.02; + const double HangBackstopSeconds = 60; + + await StatementFilterWarmUp.EnsureAsync(); + await TimeBprAsync(Small); + await TimeDeadlocksAsync(Small); + + var bprBase = await FastestRateAsync(TimeBprAsync, Small, 3); + var bprBig = await FastestRateAsync(TimeBprAsync, Big, 3); + var deadlockBase = await FastestRateAsync(TimeDeadlocksAsync, Small, 3); + var deadlockBig = await FastestRateAsync(TimeDeadlocksAsync, Big, 3); + + TestContext.Current.SendDiagnosticMessage( + $"R4 timing: {bprBig.Count} reports in {bprBig.Seconds:F2} s ({bprBig.SecondsPerMegabyte:F3} s/MB; 1 MB baseline {bprBase.SecondsPerMegabyte:F3} s/MB), " + + $"{deadlockBig.Count} graphs in {deadlockBig.Seconds:F2} s ({deadlockBig.SecondsPerMegabyte:F3} s/MB; 1 MB baseline {deadlockBase.SecondsPerMegabyte:F3} s/MB)"); + AssertNoNeedle(bprBase.Writer); + AssertNoNeedle(bprBig.Writer); + AssertNoNeedle(deadlockBase.Writer); + AssertNoNeedle(deadlockBig.Writer); + Assert.True(bprBig.Seconds < HangBackstopSeconds && deadlockBig.Seconds < HangBackstopSeconds, + $"{bprBig.Seconds:F2} s for reports, {deadlockBig.Seconds:F2} s for graphs: past the hang backstop"); + Assert.True( + bprBig.SecondsPerMegabyte <= Math.Max(bprBase.SecondsPerMegabyte, BaselineFloorSecondsPerMegabyte) * RateGrowthLimit, + $"reports: {bprBig.SecondsPerMegabyte:F3} s/MB at 4 MB against {bprBase.SecondsPerMegabyte:F3} s/MB at 1 MB is not linear"); + Assert.True( + deadlockBig.SecondsPerMegabyte <= Math.Max(deadlockBase.SecondsPerMegabyte, BaselineFloorSecondsPerMegabyte) * RateGrowthLimit, + $"graphs: {deadlockBig.SecondsPerMegabyte:F3} s/MB at 4 MB against {deadlockBase.SecondsPerMegabyte:F3} s/MB at 1 MB is not linear"); + } +} diff --git a/Darling/Darling.Tests/StatementScrubCanary.cs b/Darling/Darling.Tests/StatementScrubCanary.cs new file mode 100644 index 0000000000..3e93eddd12 --- /dev/null +++ b/Darling/Darling.Tests/StatementScrubCanary.cs @@ -0,0 +1,131 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System.Security; + +namespace Darling.Tests; + +/// +/// #4348: the planted statements every statement-filter test uses. Plain constants and string builders only, with +/// no Darling types, so Lite.Tests can compile this file by link (precedent: the linked files in +/// Lite.Tests.csproj). +/// A secret needle is text that must be gone after the filter. A kept needle is text the filter must leave +/// alone, so a test also proves the filter did not withhold everything. +/// +internal static class StatementScrubCanary +{ + /// A statement the filter must withhold. + public const string CanaryStatement = "CREATE LOGIN [canary_ssf] WITH PASSWORD = N'S3cret-canary-ssf'"; + + /// A statement the filter must keep. + public const string PlainStatement = "SELECT canary_plain_ssf FROM dbo.t WHERE c = @c"; + + /// An auto-parameterized statement whose text is kept but whose parameter values are not. + public const string AutoParamStatement = + "(@1 nvarchar(4000),@2 tinyint)UPDATE [dbo].[ssf_autoparam_canary] set [password] = @1 WHERE [id]=@2"; + + /// Text that must not survive the filter anywhere in a result. + public static readonly string[] SecretNeedles = + { + "S3cret-canary-ssf", + "param-secret-ssf", + "const-secret-ssf", + "autoparam-secret-ssf", + }; + + /// The credential in . + public const string UriSecret = "Q4rTw8Nzk"; + + /// The first characters of : all a preview cut before the closing at-sign holds. + public const string UriSecretPartial = "Q4r"; + + /// + /// A statement that holds a URI's user:secret@, padded in front so a cut at + /// lands after the first three characters of the secret and before the at-sign that names it (#5320). A preview + /// cut BEFORE the statement is judged holds ; one cut after the judgement holds none. + /// + public static string UriStatement(int cutLength) + { + const string Head = "EXEC x @u = N'https://svc:"; + return new string(' ', cutLength - 3 - Head.Length) + Head + UriSecret + "@host.example/path'"; + } + + /// Text that must come through the filter unchanged. + public static readonly string[] KeptNeedles = + { + "canary_plain_ssf", + "param-canary-ssf", + "ssf_autoparam_canary", + }; + + /// + /// A ShowPlanXML with three StmtSimple elements. Statement 1 names the canary statement and carries + /// secret parameter and constant values. Statement 2 is the plain statement with its own parameter value. + /// Statement 3 is auto-parameterized: its StatementText is kept text, its parameter values are secret. + /// + public static string CanaryPlan() + { + return + "" + + "" + + // Statement 1: the canary statement. + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + // Statement 2: the plain statement. + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + // Statement 3: auto-parameterized; the statement text is kept, the parameter values are not. + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + "" + + + ""; + } + + private static string Esc(string value) => SecurityElement.Escape(value) ?? string.Empty; +} diff --git a/Darling/Darling.Tests/StatementScrubDeadlockLiveTests.cs b/Darling/Darling.Tests/StatementScrubDeadlockLiveTests.cs new file mode 100644 index 0000000000..5a9619c42d --- /dev/null +++ b/Darling/Darling.Tests/StatementScrubDeadlockLiveTests.cs @@ -0,0 +1,175 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348 (statement filter, collection of deadlocks): a deadlock graph the filter withheld WHOLE is stored as the +/// marker text, the same for every such graph, so its identity is built from the columns the collector parsed before +/// judging it: the event time, the victim process id and the database. The one-time duplicate cleanup must read two +/// different deadlocks at the same time as two, a re-read of one as a copy, and a real copy as a copy. +/// +/// #1776 own-store - mints its own scratch database () because it runs a +/// delete path over collect.deadlocks, which the shared fixture must never inherit. +/// +[Collection("live-postgres")] +public sealed class StatementScrubDeadlockLiveTests +{ + private const int Server = -445301; + private static readonly DateTime EventTime = DateTime.SpecifyKind(new DateTime(2026, 3, 10, 9, 30, 0), DateTimeKind.Unspecified); + private const string Plain = ""; + + [Fact] + public async Task WholeMarkerDeadlocks_ReReadsAreRemoved_DifferentVictimsAndTimesAreKept_AndARealCopyStillGoes() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live deadlock duplicate cleanup (it mints its own scratch database)."); + + var ct = TestContext.Current.CancellationToken; + var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var bodySucceeded = false; + try + { + await using (var setup = new NpgsqlConnection(scratch.ConnectionString)) + { + await setup.OpenAsync(ct); + await PgMigrations.MigrateAsync(setup, ct); + + /* 1, 2 and 3 are three reads of one whole-marker deadlock; 4 is a different victim at the same time; + 5 the same victim at another time; 6 and 7 carry no victim id, so nothing tells them apart. */ + await InsertAsync(setup, 1, EventTime.AddMinutes(1), SensitiveStatements.PlaceholderText, ct, "process1", "db1"); + await InsertAsync(setup, 2, EventTime.AddMinutes(2), SensitiveStatements.PlaceholderText, ct, "process1", "db1"); + await InsertAsync(setup, 3, EventTime.AddMinutes(3), SensitiveStatements.PlaceholderText, ct, "process1", "db1"); + await InsertAsync(setup, 4, EventTime.AddMinutes(4), SensitiveStatements.PlaceholderText, ct, "process9", "db1"); + await InsertAsync(setup, 5, EventTime.AddMinutes(5), SensitiveStatements.PlaceholderText, ct, "process1", "db1", EventTime.AddSeconds(30)); + await InsertAsync(setup, 6, EventTime.AddMinutes(6), SensitiveStatements.PlaceholderText, ct, null, "db1"); + await InsertAsync(setup, 7, EventTime.AddMinutes(7), SensitiveStatements.PlaceholderText, ct, null, "db1"); + await InsertAsync(setup, 8, EventTime.AddMinutes(8), Plain, ct, "process1", "db1"); + await InsertAsync(setup, 9, EventTime.AddMinutes(9), Plain, ct, "process9", "db2"); + } + + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + var summary = await DeadlockDuplicateCleanup.RunAsync(postgres, logger: null, ct); + + Assert.Equal(3, summary.RowsRemoved); + + await using var verify = new NpgsqlConnection(scratch.ConnectionString); + await verify.OpenAsync(ct); + var ids = new List(); + await using (var command = new NpgsqlCommand("SELECT deadlock_id FROM collect.deadlocks ORDER BY 1", verify)) + await using (var reader = await command.ExecuteReaderAsync(ct)) + { + while (await reader.ReadAsync(ct)) + { + ids.Add(reader.GetInt64(0)); + } + } + + Assert.Equal(new List { 1, 4, 5, 6, 7, 8 }, ids); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => await Task.CompletedTask); + await scratch.DisposeAsync(); + } + } + + /* The stored-copy read the pre-insert dedupe runs must hand back, for a whole-marker row, exactly the identity + the collector computes for the row it is about to write. */ + [Fact] + public async Task TheStoredIdentityRead_GivesAWholeMarkerRowTheIdentityTheCollectorComputes() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live deadlock duplicate cleanup (it mints its own scratch database)."); + + var ct = TestContext.Current.CancellationToken; + var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var bodySucceeded = false; + try + { + await using var setup = new NpgsqlConnection(scratch.ConnectionString); + await setup.OpenAsync(ct); + await PgMigrations.MigrateAsync(setup, ct); + + await InsertAsync(setup, 1, EventTime.AddMinutes(1), SensitiveStatements.PlaceholderText, ct, "process1", "db1"); + await InsertAsync(setup, 2, EventTime.AddMinutes(2), SensitiveStatements.PlaceholderText, ct, "process9", null); + await InsertAsync(setup, 3, EventTime.AddMinutes(3), SensitiveStatements.PlaceholderText, ct, null, "db1"); + await InsertAsync(setup, 4, EventTime.AddMinutes(4), Plain, ct, "process1", "db1"); + + var stored = new HashSet<(DateTime Time, string Graph)>(); + await using (var command = new NpgsqlCommand(DarlingCollectorRunner.StoredDeadlockIdentitySql, setup)) + { + command.Parameters.AddWithValue(Server); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Array | NpgsqlDbType.Timestamp, Value = new[] { EventTime } }); + command.Parameters.AddWithValue(EventTime.AddDays(-1)); + await using var reader = await command.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + stored.Add((reader.GetDateTime(0), reader.GetString(1))); + } + + static DeadlocksCollector.Row Row(string graph, string? victim, string? db) => + new() { DeadlockTime = EventTime, GraphXml = graph, VictimProcessId = victim, DatabaseName = db }; + + /* A re-read of a stored whole-marker row is dropped; another victim or database is not; a row with no + victim id has no identity; real text still matches by graph text alone. */ + var rows = new List + { + Row(SensitiveStatements.PlaceholderText, "process1", "db1"), + Row(SensitiveStatements.PlaceholderText, "process9", null), + Row(SensitiveStatements.PlaceholderText, "process1", "db2"), + Row(SensitiveStatements.PlaceholderText, "process7", "db1"), + Row(SensitiveStatements.PlaceholderText, null, "db1"), + Row(Plain, "anything", "elsewhere"), + }; + + var kept = DeadlocksCollector.Instance.DropAlreadyStored(rows, stored); + + Assert.Equal(new[] { 2, 3, 4 }, kept.Select(r => rows.IndexOf(r)).ToArray()); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => await Task.CompletedTask); + await scratch.DisposeAsync(); + } + } + + private static async Task InsertAsync(NpgsqlConnection connection, long id, DateTime collectionTime, string graph, CancellationToken ct, + string? victim = null, string? database = null, DateTime? deadlockTime = null) + { + await using var command = new NpgsqlCommand(@" +INSERT INTO collect.deadlocks (deadlock_id, collection_time, server_id, server_name, deadlock_time, deadlock_graph_xml, victim_process_id, database_name) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8)", connection); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Bigint, Value = id }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = collectionTime }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = Server }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Text, Value = "server-" + Server.ToString(CultureInfo.InvariantCulture) }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = deadlockTime ?? EventTime }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Text, Value = graph }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Text, Value = (object?)victim ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Text, Value = (object?)database ?? DBNull.Value }); + await command.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/StatementScrubDerivedStoreTests.cs b/Darling/Darling.Tests/StatementScrubDerivedStoreTests.cs new file mode 100644 index 0000000000..48983a4ad4 --- /dev/null +++ b/Darling/Darling.Tests/StatementScrubDerivedStoreTests.cs @@ -0,0 +1,312 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Runtime.CompilerServices; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Analysis; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Analysis; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Notifications; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348, #5320 (lane R10, Layer 0b): the derived stores judge their own text where they write it. A finding or an +/// alert is built from rows a collector stored, and a row stored before the collection-time filter existed can still +/// hold a statement, so story_text, drill_down_json, detail_text and context_json go +/// through at the INSERT. Clean input comes back as the same instance, so a clean row +/// is stored byte for byte as before. +/// +public sealed class StatementScrubDerivedStoreTests +{ + private static string Fresh(string text) => new(text.AsSpan()); + + private static void AssertNoSecret(string? value) + { + Assert.NotNull(value); + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.DoesNotContain(needle, value, StringComparison.Ordinal); + } + } + + [Fact] + public void Text_ReturnsTheSameInstanceOnCleanInput() + { + var clean = Fresh("Waits rose on " + StatementScrubCanary.PlainStatement); + Assert.Same(clean, DerivedStoreScrub.Text(clean)); + Assert.Equal(string.Empty, DerivedStoreScrub.Text(string.Empty)); + Assert.Null(DerivedStoreScrub.Text(null)); + } + + [Fact] + public void Text_WithholdsANamedStatement() + { + var result = DerivedStoreScrub.Text(Fresh("Blocked on " + StatementScrubCanary.CanaryStatement)); + Assert.Equal(SensitiveStatements.PlaceholderText, result); + } + + [Fact] + public void Json_ReturnsTheSameInstanceOnCleanInput() + { + var clean = Fresh("{\"rows\":[{\"query_text\":\"" + StatementScrubCanary.PlainStatement + "\",\"n\":3}]}"); + Assert.Same(clean, DerivedStoreScrub.Json(clean)); + Assert.Null(DerivedStoreScrub.Json(null)); + } + + [Fact] + public void Json_ReplacesOnlyTheNamedValueAndKeepsTheRest() + { + var json = JsonSerializer.Serialize(new + { + rows = new[] + { + new { query_text = StatementScrubCanary.CanaryStatement, n = 1 }, + new { query_text = StatementScrubCanary.PlainStatement, n = 2 }, + }, + }); + + var result = DerivedStoreScrub.Json(json); + + AssertNoSecret(result); + Assert.Contains(SensitiveStatements.PlaceholderText, result, StringComparison.Ordinal); + Assert.Contains("canary_plain_ssf", result, StringComparison.Ordinal); + using var parsed = JsonDocument.Parse(result!); + Assert.Equal(2, parsed.RootElement.GetProperty("rows").GetArrayLength()); + } + + [Fact] + public void Json_StoresNullForADocumentTheFilterCannotRead() + { + Assert.Null(DerivedStoreScrub.Json("{ this is not json")); + } + + /// The Darling writers call the helper; reverting either call fails this on the old shape. + [Fact] + public void TheDarlingWritersCallTheHelper() + { + var finding = File.ReadAllText(RepoPath("Darling/PerformanceMonitor.Darling.Analysis/PgFindingStore.cs")); + Assert.Contains("DerivedStoreScrub.Text(finding.StoryText)", finding, StringComparison.Ordinal); + Assert.Contains("DerivedStoreScrub.Json(DrillDownSerializer.Serialize(finding.DrillDown))", finding, StringComparison.Ordinal); + + var alert = File.ReadAllText(RepoPath("Darling/PerformanceMonitor.Darling.Service/PgAlertHistoryStore.cs")); + Assert.Contains("DerivedStoreScrub.Text(record.DetailText)", alert, StringComparison.Ordinal); + Assert.Contains("DerivedStoreScrub.Json(record.ContextJson)", alert, StringComparison.Ordinal); + } + + /// + /// The managed-role config.record_custom_alert_resolution function (R10 verdict: it cannot hold a + /// statement). Its only caller is CustomAlertEvaluator.WriteTeardownResolutionAsync, which builds + /// detail_text from the server name, the operator-typed rule name (newline-stripped and capped) and one of + /// three fixed reasons. Nothing there reads a collected row. If that message ever starts to carry collected text, + /// this pin fails and the write has to go through . + /// + [Fact] + public void TheCustomAlertResolutionRowCarriesNoCollectedText() + { + var evaluator = File.ReadAllText(RepoPath("Darling/PerformanceMonitor.Darling.Service/CustomAlertEvaluator.cs")); + Assert.Contains("$\"{serverName}: {safeName} resolved because {reason}\"", evaluator, StringComparison.Ordinal); + Assert.Contains("RecordCustomAlertResolutionAsync(serverId, serverName, title, message)", evaluator, StringComparison.Ordinal); + + var callers = Directory.EnumerateFiles(RepoPath("Darling"), "*.cs", SearchOption.AllDirectories) + .Where(f => !f.Contains($"{Path.DirectorySeparatorChar}obj{Path.DirectorySeparatorChar}", StringComparison.Ordinal) + && !f.Contains($"{Path.DirectorySeparatorChar}bin{Path.DirectorySeparatorChar}", StringComparison.Ordinal) + && !f.Contains($"{Path.DirectorySeparatorChar}Darling.Tests{Path.DirectorySeparatorChar}", StringComparison.Ordinal)) + .Where(f => File.ReadAllText(f).Contains(".RecordCustomAlertResolutionAsync(", StringComparison.Ordinal)) + .Select(Path.GetFileName) + .ToList(); + Assert.Equal(new[] { "CustomAlertEvaluator.cs" }, callers); + } + + private static string RepoPath(string relative, [CallerFilePath] string thisFile = "") => + Path.GetFullPath(Path.Combine(Path.GetDirectoryName(thisFile)!, "..", "..", relative)); +} + +/// +/// The live half: a finding and an alert built from a raw planted source row store the marker in the real tables. +/// +[Collection("live-postgres")] +public sealed class StatementScrubDerivedStoreLiveTests +{ + private const int FindingServer = -532010; + private const int AlertServer = -532011; + + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + private static async Task RunLiveAsync(Func body) + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live derived-store scrub test."); + + using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(Ct); + await PgMigrations.MigrateAsync(connection, Ct); + await DeleteTestRowsAsync(connection, Ct); + + await using var postgres = NpgsqlDataSource.Create(connectionString!); + var bodySucceeded = false; + try + { + await body(postgres, connection); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteTestRowsAsync(cleanup, cleanupCt)); + } + } + + private static async Task DeleteTestRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + using var cleanup = new NpgsqlCommand( + $"DELETE FROM analysis_findings WHERE server_id = {FindingServer}; DELETE FROM config_alert_log WHERE server_id = {AlertServer};", + connection); + await cleanup.ExecuteNonQueryAsync(ct); + } + + private static AnalysisFinding Finding(string hash, string story, Dictionary? drillDown) + { + return new AnalysisFinding + { + ServerId = FindingServer, + ServerName = "derived-scrub-finding", + Category = "queries", + StoryPath = "SCRUB_A -> SCRUB_B", + StoryPathHash = hash, + IncidentId = "inc-" + hash, + Severity = 1.2, + Confidence = StoryConfidence.Compute(1, 5, 2), + FactCount = 2, + RootFactKey = "SCRUB_A", + RootFactValue = 1.0, + AnalysisTime = DateTime.UtcNow, + StoryText = story, + DrillDown = drillDown, + }; + } + + private static Dictionary DrillDown(string statement) => new() + { + ["top_queries"] = new[] + { + new Dictionary { ["query_text"] = statement, ["executions"] = 7 }, + }, + }; + + private static async Task<(string Story, string? Drill)> ReadFindingAsync(NpgsqlConnection connection, string hash) + { + using var command = new NpgsqlCommand( + "SELECT story_text, drill_down_json FROM analysis_findings WHERE server_id = $1 AND story_path_hash = $2", connection); + command.Parameters.AddWithValue(FindingServer); + command.Parameters.AddWithValue(hash); + await using var reader = await command.ExecuteReaderAsync(Ct); + Assert.True(await reader.ReadAsync(Ct)); + return (reader.GetString(0), reader.IsDBNull(1) ? null : reader.GetString(1)); + } + + private static AnalysisContext Context() + { + var end = DateTime.UtcNow; + return new AnalysisContext + { + ServerId = FindingServer, + ServerName = "derived-scrub-finding", + TimeRangeStart = end.AddHours(-4), + TimeRangeEnd = end, + ServerUtcOffset = TimeSpan.Zero, + }; + } + + [Fact] + public async Task AFindingBuiltFromARawPlantedRowStoresTheMarker() + { + await RunLiveAsync(async (postgres, connection) => + { + var store = new PgFindingStore(postgres); + var raw = Finding("rawcanary01", "Blocked on " + StatementScrubCanary.CanaryStatement, + DrillDown(StatementScrubCanary.CanaryStatement)); + var plainDrill = DrillDown(StatementScrubCanary.PlainStatement); + var plain = Finding("plaincanary01", "Waits rose on " + StatementScrubCanary.PlainStatement, plainDrill); + + await store.InsertFindingsAsync(new List { raw, plain }, Context()); + + var (story, drill) = await ReadFindingAsync(connection, "rawcanary01"); + Assert.Equal(SensitiveStatements.PlaceholderText, story); + Assert.NotNull(drill); + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.DoesNotContain(needle, drill, StringComparison.Ordinal); + } + Assert.Contains(SensitiveStatements.PlaceholderText, drill, StringComparison.Ordinal); + using (JsonDocument.Parse(drill!)) { } + + /* The clean finding is stored byte for byte as the serializer wrote it. */ + var (plainStory, plainDrillStored) = await ReadFindingAsync(connection, "plaincanary01"); + Assert.Equal("Waits rose on " + StatementScrubCanary.PlainStatement, plainStory); + Assert.Equal(DrillDownSerializer.Serialize(plainDrill), plainDrillStored); + }); + } + + [Fact] + public async Task AnAlertBuiltFromARawPlantedRowStoresTheMarker() + { + await RunLiveAsync(async (postgres, connection) => + { + var store = new PgAlertHistoryStore(postgres); + var contextJson = JsonSerializer.Serialize(new { dedup = "abc123", query_text = StatementScrubCanary.CanaryStatement }); + var plainContext = JsonSerializer.Serialize(new { dedup = "abc124", query_text = StatementScrubCanary.PlainStatement }); + + await store.RecordAlertAsync(new AlertHistoryRecord( + AlertServer.ToString(System.Globalization.CultureInfo.InvariantCulture), "derived-scrub-alert", "Scrub Raw", + "1", "1", 1, 1, AlertDelivery.NoChannelApplies(), false, + "Long query: " + StatementScrubCanary.CanaryStatement, contextJson)); + await store.RecordAlertAsync(new AlertHistoryRecord( + AlertServer.ToString(System.Globalization.CultureInfo.InvariantCulture), "derived-scrub-alert", "Scrub Plain", + "1", "1", 1, 1, AlertDelivery.NoChannelApplies(), false, + "Long query: " + StatementScrubCanary.PlainStatement, plainContext)); + + using var read = new NpgsqlCommand( + "SELECT metric_name, detail_text, context_json FROM config_alert_log WHERE server_id = $1", connection); + read.Parameters.AddWithValue(AlertServer); + var rows = new Dictionary(); + await using (var reader = await read.ExecuteReaderAsync(Ct)) + { + while (await reader.ReadAsync(Ct)) + { + rows[reader.GetString(0)] = (reader.IsDBNull(1) ? null : reader.GetString(1), reader.IsDBNull(2) ? null : reader.GetString(2)); + } + } + + Assert.Equal(2, rows.Count); + var raw = rows["Scrub Raw"]; + Assert.Equal(SensitiveStatements.PlaceholderText, raw.Detail); + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.DoesNotContain(needle, raw.Context, StringComparison.Ordinal); + } + Assert.Contains(SensitiveStatements.PlaceholderText, raw.Context, StringComparison.Ordinal); + Assert.Contains("abc123", raw.Context, StringComparison.Ordinal); + + var plain = rows["Scrub Plain"]; + Assert.Equal("Long query: " + StatementScrubCanary.PlainStatement, plain.Detail); + Assert.Equal(plainContext, plain.Context); + }); + } +} diff --git a/Darling/Darling.Tests/StatementScrubEndToEndLiveTests.cs b/Darling/Darling.Tests/StatementScrubEndToEndLiveTests.cs new file mode 100644 index 0000000000..b33e598d22 --- /dev/null +++ b/Darling/Darling.Tests/StatementScrubEndToEndLiveTests.cs @@ -0,0 +1,639 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Data; +using System.Data.Common; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.AspNetCore.Http; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348 (statement filter), the end-to-end proof for the Darling store: a new row holds the marker, and an old row +/// reads back as the marker without being changed. +/// +/// Collect. The canary goes in as the rows a monitored server would return (a fake target provider +/// hands the runner a reader over a table), and runs the real +/// definition: read, filter, delta, dedupe, write. The stored row is then read straight from the rig store. +/// query_stats and deadlocks take the Azure per-database path of the runner, and query_snapshots the server-wide +/// path, so both loops are covered. +/// +/// Read. Rows are planted straight into the tables, as rows stored before the upgrade would be. The tool +/// reads RAW (the control: the canary must still be there, so the plant reached the output), then through the +/// host's real filter list (the MCP read) and through the web read's result writer (/api/read/*). Both must +/// show the marker and no secret needle. The planted table's bytes are hashed before the reads and after: a read +/// never rewrites a stored row. +/// +/// #1776 own-store - every row is keyed by this class's own fake server ids and removed in the cleanup. +/// +[Collection("live-postgres")] +public sealed class StatementScrubEndToEndLiveTests +{ + private const string SkipReason = "Set DARLING_TEST_PG to a Postgres connection string to run the statement filter end-to-end tests."; + private const string Db = "SsfE2eDb"; + private static readonly DateTime Anchor = new(2031, 6, 10, 12, 0, 0, DateTimeKind.Unspecified); + + private const int CollectStatsId = -495101; + private const int CollectSnapshotsId = -495102; + private const int CollectDeadlocksId = -495103; + private const int ReadStatsId = -495111; + private const int ReadSnapshotsId = -495112; + private const int ReadDeadlocksId = -495113; + + private static readonly int[] AllIds = + { + CollectStatsId, CollectSnapshotsId, CollectDeadlocksId, ReadStatsId, ReadSnapshotsId, ReadDeadlocksId, + }; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static string NameOf(int id) => "darling-ssf-e2e" + id.ToString(System.Globalization.CultureInfo.InvariantCulture); + + // ── collect ── + + [Fact] + public async Task QueryStats_TheCanaryThroughTheRunner_IsStoredAsTheMarker_ThePlainStatementIsKept_AndThePlanIsFiltered() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), SkipReason); + var ct = TestContext.Current.CancellationToken; + await using var connection = await OpenAsync(cs!, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + var bodySucceeded = false; + try + { + var runner = new DarlingCollectorRunner(postgres, new CollectorDeltaCalculator(), null, capturePlans: () => true); + runner.TargetProviderOverrideForTests = _ => new FakeTargetProvider(_ => QueryStatsReader()); + runner.AzureDatabaseListOverrideForTests = (_, _) => Task.FromResult(new List { "alpha" }); + + await runner.RunAsync(QueryStatsCollector.Instance, Server(CollectStatsId, azure: true), ct); + + var texts = await ScalarListAsync(connection, + "SELECT query_hash || '|' || COALESCE(query_text, '') FROM v_query_stats WHERE server_id = " + CollectStatsId + " ORDER BY query_hash", ct); + Assert.Equal(new[] { "0xC0|" + StatementFilterCensus.Marker, "0xP0|" + StatementScrubCanary.PlainStatement }, texts); + + string? plan = await StoredPlanAsync(connection, CollectStatsId, "0xC0", ct); + Assert.False(string.IsNullOrEmpty(plan), "the canary row's stored plan did not resolve, so the plan check proves nothing"); + StatementFilterCensus.AssertPlanFilteredKeepsTheRest(StatementScrubCanary.CanaryPlan(), plan!); + AssertNoSecret(await WholeTableTextAsync(connection, "query_stats", CollectStatsId, ct)); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + [Fact] + public async Task QuerySnapshots_TheCanaryThroughTheRunnersServerWidePath_IsStoredAsTheMarker_AndBothPlansAreFiltered() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), SkipReason); + var ct = TestContext.Current.CancellationToken; + await using var connection = await OpenAsync(cs!, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + var bodySucceeded = false; + try + { + /* #5320: the runner's server-wide path, driven through the same fake target provider the per-database cases + use (the runner opens its server-wide connection through that provider), so the whole run is covered: + read, filter, delta, dedupe, write. */ + var runner = new DarlingCollectorRunner(postgres, new CollectorDeltaCalculator(), null, capturePlans: () => true); + runner.TargetProviderOverrideForTests = _ => new FakeTargetProvider(_ => SnapshotsReader()); + + await runner.RunAsync(QuerySnapshotsCollector.Instance, Server(CollectSnapshotsId, azure: false), ct); + + var rows = await ScalarListAsync(connection, + "SELECT session_id::text || '|' || COALESCE(query_text, '') FROM query_snapshots WHERE server_id = " + CollectSnapshotsId + " ORDER BY session_id", ct); + Assert.Equal(new[] { "81|" + StatementFilterCensus.Marker, "82|" + StatementScrubCanary.PlainStatement }, rows); + + foreach (string column in new[] { "query_plan", "live_query_plan" }) + { + var plans = await ScalarListAsync(connection, + "SELECT " + column + " FROM query_snapshots WHERE server_id = " + CollectSnapshotsId + " AND session_id = 81 AND " + column + " IS NOT NULL", ct); + string plan = Assert.Single(plans); + StatementFilterCensus.AssertPlanFilteredKeepsTheRest(StatementScrubCanary.CanaryPlan(), plan); + } + + AssertNoSecret(await WholeTableTextAsync(connection, "query_snapshots", CollectSnapshotsId, ct)); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + [Fact] + public async Task Deadlocks_TheCanaryThroughTheRunner_IsStoredAsTheMarker_InTheVictimTextAndTheGraph() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), SkipReason); + var ct = TestContext.Current.CancellationToken; + await using var connection = await OpenAsync(cs!, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + var bodySucceeded = false; + try + { + var runner = new DarlingCollectorRunner(postgres, new CollectorDeltaCalculator()); + bool withPlan = runner.ShouldCapturePlanXmlFor(DeadlocksCollector.Instance.Name, CollectDeadlocksId); + runner.TargetProviderOverrideForTests = _ => new FakeTargetProvider(_ => DeadlocksReader(withPlan)); + runner.AzureDatabaseListOverrideForTests = (_, _) => Task.FromResult(new List { "alpha" }); + + await runner.RunAsync(DeadlocksCollector.Instance, Server(CollectDeadlocksId, azure: true), ct); + + var victims = await ScalarListAsync(connection, + "SELECT COALESCE(victim_sql_text, '') FROM deadlocks WHERE server_id = " + CollectDeadlocksId + " ORDER BY deadlock_time", ct); + Assert.Equal(new[] { StatementFilterCensus.Marker, StatementScrubCanary.PlainStatement }, victims); + + var graphs = await ScalarListAsync(connection, + "SELECT deadlock_graph_xml FROM deadlocks WHERE server_id = " + CollectDeadlocksId + " ORDER BY deadlock_time", ct); + foreach (string needle in StatementScrubCanary.SecretNeedles) Assert.DoesNotContain(needle, graphs[0], StringComparison.Ordinal); + Assert.Contains(StatementFilterCensus.Marker, graphs[0], StringComparison.Ordinal); + Assert.Contains(StatementScrubCanary.PlainStatement, graphs[1], StringComparison.Ordinal); + + AssertNoSecret(await WholeTableTextAsync(connection, "deadlocks", CollectDeadlocksId, ct)); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + // ── read ── + + [Fact] + public async Task QueryStats_ARowStoredBeforeTheUpgrade_ReadsBackAsTheMarker_OnTheMcpAndTheWebRead_AndTheStoredRowIsUnchanged() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), SkipReason); + var ct = TestContext.Current.CancellationToken; + await using var connection = await OpenAsync(cs!, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + using var host = await StatementFilterCensus.BuildHostAsync(); + var bodySucceeded = false; + try + { + string server = NameOf(ReadStatsId); + await DarlingMcpTestData.RegisterServerAsync(connection, ReadStatsId, server, ct); + var now = DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow); + var seeds = new[] { (StatementScrubCanary.CanaryStatement, "C", StatementScrubCanary.CanaryPlan()), (StatementScrubCanary.PlainStatement, "P", (string?)null) }; + for (int i = 0; i < seeds.Length; i++) + { + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO query_stats (collection_id, collection_time, server_id, server_name, database_name, query_hash, query_plan_hash, sql_handle, plan_handle, query_text, query_plan_xml, + delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, min_dop, max_dop, sample_interval_seconds) +VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(now.AddMinutes(-30 - i)), ReadStatsId, server, Db, + "0xQH" + seeds[i].Item2, "0xPLAN" + seeds[i].Item2, "0xSQLH" + seeds[i].Item2, "0xPLANH" + seeds[i].Item2, + seeds[i].Item1, seeds[i].Item3, + 10L, 1_000_000L, 2_000_000L, 5_000L, 1, 1, 60); + } + + string before = await TableHashAsync(connection, "query_stats", ReadStatsId, ct); + + string raw = await DarlingMcpDataTools.GetTopQueriesByCpu(postgres, server, cancellationToken: ct); + /* #5320: since PR C these tools judge the statement text at the read itself (McpHelpers.TruncateStatement), so the + tool answer already carries the marker and none of the named text; the host filter and the web sweep that + follow must still hold it. */ + foreach (string needle in StatementScrubCanary.SecretNeedles) Assert.DoesNotContain(needle, raw, StringComparison.Ordinal); + Assert.Contains(StatementFilterCensus.Marker, raw, StringComparison.Ordinal); + await AssertMcpAndWebAsync(host, "get_top_queries_by_cpu", raw); + + string rawPlan = await DarlingMcpPlanTools.GetPlanXml(postgres, "0xQHC", server, cancellationToken: ct); + string filteredPlan = (await StatementFilterCensus.FilterThroughHostAsync(host, rawPlan)).Text; + foreach (string needle in StatementScrubCanary.SecretNeedles) Assert.DoesNotContain(needle, filteredPlan, StringComparison.Ordinal); + Assert.Contains(StatementFilterCensus.Marker, filteredPlan, StringComparison.Ordinal); + Assert.Contains("canary_plain_ssf", filteredPlan, StringComparison.Ordinal); + /* The web route wraps the tool's plan in its JSON envelope before the sweep writes it. */ + var (planStatus, planBody) = await WebReadAsync("get_plan_xml", DarlingWebEndpoints.WrapPlanXml(rawPlan, "0xQHC", Db)); + Assert.Equal(200, planStatus); + foreach (string needle in StatementScrubCanary.SecretNeedles) Assert.DoesNotContain(needle, planBody, StringComparison.Ordinal); + Assert.Contains(StatementFilterCensus.Marker, planBody, StringComparison.Ordinal); + + Assert.Equal(before, await TableHashAsync(connection, "query_stats", ReadStatsId, ct)); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + [Fact] + public async Task QuerySnapshots_ARowStoredBeforeTheUpgrade_ReadsBackAsTheMarker_OnTheMcpAndTheWebRead_AndTheStoredRowIsUnchanged() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), SkipReason); + var ct = TestContext.Current.CancellationToken; + await using var connection = await OpenAsync(cs!, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + using var host = await StatementFilterCensus.BuildHostAsync(); + var bodySucceeded = false; + try + { + string server = NameOf(ReadSnapshotsId); + await DarlingMcpTestData.RegisterServerAsync(connection, ReadSnapshotsId, server, ct); + var now = DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow); + var texts = new[] { StatementScrubCanary.CanaryStatement, StatementScrubCanary.PlainStatement }; + for (int i = 0; i < texts.Length; i++) + { + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO query_snapshots (collection_id, collection_time, server_id, server_name, session_id, database_name, elapsed_time_formatted, query_text, status, blocking_session_id, wait_type, wait_time_ms, cpu_time_ms, total_elapsed_time_ms, reads, writes, logical_reads, granted_query_memory_gb, transaction_isolation_level, dop, parallel_worker_count, login_name, host_name, program_name, open_transaction_count, request_id, wait_resource, percent_complete, query_hash) +VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(now.AddMinutes(-5 - i)), ReadSnapshotsId, server, 100 + i, Db, + "00 00:00:05.125", texts[i], "running", 0, null, 0L, 1000L, 1500L, 10_000L, 50L, 7L, 0m, + "Read Committed", 1, 0, "app_svc", "APPSRV01", "MyOrderService.Worker", 0, 0, + "KEY: 5:72057594043432960 (a1b2c3d4e5f6)", 12.5m, "0x" + (0x1A2B3C4D5E6F7080L + i).ToString("X16")); + } + + string before = await TableHashAsync(connection, "query_snapshots", ReadSnapshotsId, ct); + + string raw = await DarlingMcpSessionTools.GetActiveQueries(postgres, server, cancellationToken: ct); + /* #5320: since PR C these tools judge the statement text at the read itself (McpHelpers.TruncateStatement), so the + tool answer already carries the marker and none of the named text; the host filter and the web sweep that + follow must still hold it. */ + foreach (string needle in StatementScrubCanary.SecretNeedles) Assert.DoesNotContain(needle, raw, StringComparison.Ordinal); + Assert.Contains(StatementFilterCensus.Marker, raw, StringComparison.Ordinal); + await AssertMcpAndWebAsync(host, "get_active_queries", raw); + + Assert.Equal(before, await TableHashAsync(connection, "query_snapshots", ReadSnapshotsId, ct)); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + [Fact] + public async Task Deadlocks_ARowStoredBeforeTheUpgrade_ReadsBackAsTheMarker_OnTheMcpAndTheWebRead_AndTheStoredRowIsUnchanged() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), SkipReason); + var ct = TestContext.Current.CancellationToken; + await using var connection = await OpenAsync(cs!, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + using var host = await StatementFilterCensus.BuildHostAsync(); + var bodySucceeded = false; + try + { + string server = NameOf(ReadDeadlocksId); + await DarlingMcpTestData.RegisterServerAsync(connection, ReadDeadlocksId, server, ct); + var now = DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow); + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO deadlocks (deadlock_id, collection_time, server_id, server_name, deadlock_time, deadlock_graph_xml, victim_process_id, victim_sql_text, database_name) +VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(now.AddMinutes(-5)), ReadDeadlocksId, server, + DarlingMcpTestData.Naive(now.AddMinutes(-6)), Graph(StatementScrubCanary.CanaryStatement, StatementScrubCanary.PlainStatement), + "p1", StatementScrubCanary.CanaryStatement, Db); + + string before = await TableHashAsync(connection, "deadlocks", ReadDeadlocksId, ct); + + string raw = await DarlingMcpBlockingTools.GetDeadlockDetail(postgres, server, full_graph: true, cancellationToken: ct); + Assert.Contains("S3cret-canary-ssf", raw, StringComparison.Ordinal); + await AssertMcpAndWebAsync(host, "get_deadlock_detail", raw); + + Assert.Equal(before, await TableHashAsync(connection, "deadlocks", ReadDeadlocksId, ct)); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + // ── plumbing ── + + /// The plan a stored row resolves to through the view, the way every reader sees it (inline, or diverted to the plan dimension). + private static async Task StoredPlanAsync(NpgsqlConnection connection, int serverId, string queryHash, CancellationToken ct) + { + await using var read = new NpgsqlCommand( + "SELECT query_plan_xml, query_plan_gz FROM v_query_stats WHERE server_id = $1 AND query_hash = $2", connection); + read.Parameters.AddWithValue(serverId); + read.Parameters.AddWithValue(queryHash); + await using var reader = await read.ExecuteReaderAsync(ct); + Assert.True(await reader.ReadAsync(ct), "the row must exist"); + return PayloadDimensions.ResolveContent( + reader.IsDBNull(0) ? null : reader.GetString(0), + reader.IsDBNull(1) ? null : reader.GetFieldValue(1)); + } + + private sealed class NoDeltas : ICollectorDeltaCalculator + { + public long CalculateDelta(int serverId, string collectorName, string key, long currentValue, + DateTime? collectionTime = null, int maxGapSeconds = 0) => currentValue; + + public long CalculateDeltaWithInterval(int serverId, string collectorName, string key, long currentValue, + out int intervalSeconds, DateTime? collectionTime = null, int maxGapSeconds = 0) + { + intervalSeconds = 0; + return currentValue; + } + } + + private static async Task WriteBatchAsync( + NpgsqlConnection connection, ICollectorDefinition definition, CollectorContext context, IReadOnlyList rows, CancellationToken ct) + { + var writer = new PgCollectorRowWriter(); + var dimensions = new PayloadDimensionBatch(); + writer.UseDimensions(PayloadDimensions.DiversionPlanFor(definition), dimensions); + + await using var transaction = await connection.BeginTransactionAsync(ct); + using (var importer = await connection.BeginBinaryImportAsync(PgCollectorRowWriter.CopyCommandFor(definition), ct)) + { + writer.Importer = importer; + foreach (var row in rows) + { + await importer.StartRowAsync(ct); + writer.Value(CollectionIdGenerator.Next()); + writer.Value(context.CollectionTime).Value(context.ServerId).Value(context.ServerName); + writer.BeginPayload(); + definition.WritePayload(row, writer, context); + writer.EndPayload(definition.PayloadColumns.Count); + } + + await importer.CompleteAsync(ct); + } + + await PayloadDimensionWriter.FlushAsync(connection, transaction, dimensions, context.CollectionTime, ct); + await transaction.CommitAsync(ct); + } + + private static async Task AssertMcpAndWebAsync(Microsoft.AspNetCore.TestHost.TestServer host, string tool, string raw) + { + string filtered = (await StatementFilterCensus.FilterThroughHostAsync(host, raw)).Text; + try + { + StatementFilterCensus.AssertFilteredWithholdsTheCanary(raw, filtered); + } + catch (Exception ex) + { + throw new Xunit.Sdk.XunitException(tool + " (MCP): " + ex.Message); + } + + var (status, body) = await WebReadAsync(tool, raw); + Assert.Equal(200, status); + foreach (string needle in StatementScrubCanary.SecretNeedles) Assert.DoesNotContain(needle, body, StringComparison.Ordinal); + Assert.Contains(StatementFilterCensus.Marker, body, StringComparison.Ordinal); + Assert.Contains("canary_plain_ssf", body, StringComparison.Ordinal); + } + + private static async Task<(int Status, string Body)> WebReadAsync(string tool, string raw) + { + var result = DarlingWebEndpoints.ToHttpResult(raw, "/api/read/" + tool, NullLogger.Instance, 1); + var context = new DefaultHttpContext { RequestServices = new ServiceCollection().AddLogging().BuildServiceProvider() }; + context.Response.Body = new MemoryStream(); + await result.ExecuteAsync(context); + context.Response.Body.Position = 0; + return (context.Response.StatusCode, Encoding.UTF8.GetString(((MemoryStream)context.Response.Body).ToArray())); + } + + private static void AssertNoSecret(string text) + { + foreach (string needle in StatementScrubCanary.SecretNeedles) Assert.DoesNotContain(needle, text, StringComparison.Ordinal); + } + + private static string Graph(string victimText, string otherText) => + "" + + "" + + System.Security.SecurityElement.Escape(victimText) + "" + + "" + + System.Security.SecurityElement.Escape(otherText) + "" + + ""; + + private static ServerRuntime Server(int id, bool azure) => new() + { + Config = new MonitoredServer { Name = NameOf(id), Host = "h" }, + ConnectionString = "Server=azure;Database=master", + Target = azure + ? new CollectorTargetInfo { IsAzureSqlDb = true, Engine = CollectorTargetEngine.SqlServer } + : new CollectorTargetInfo { SqlMajorVersion = 15, Engine = CollectorTargetEngine.SqlServer }, + StorageName = NameOf(id), + ServerId = id, + }; + + private static async Task OpenAsync(string cs, CancellationToken ct) + { + var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + return connection; + } + + private static async Task DeleteRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + foreach (int id in AllIds) + { + await DarlingMcpTestData.ExecAsync(connection, ct, + $"DELETE FROM query_snapshots WHERE server_id = {id}; DELETE FROM query_stats WHERE server_id = {id}; " + + $"DELETE FROM deadlocks WHERE server_id = {id}; DELETE FROM collector_state WHERE server_id = {id}; " + + $"DELETE FROM servers WHERE server_id = {id}; DELETE FROM config_monitored_servers WHERE server_id = {id};"); + } + } + + private static async Task> ScalarListAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + var list = new List(); + await using var command = new NpgsqlCommand(sql, connection); + await using var reader = await command.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + list.Add(reader.IsDBNull(0) ? "" : reader.GetString(0)); + } + + return list; + } + + /// Every column of every row of one server in a table, as one text, for a whole-row secret scan. + private static async Task WholeTableTextAsync(NpgsqlConnection connection, string table, int serverId, CancellationToken ct) => + string.Concat(await ScalarListAsync(connection, $"SELECT t::text FROM {table} t WHERE server_id = {serverId}", ct)); + + /// A hash of the bytes of every row of one server in a table: equal before and after means no read rewrote a row. + private static async Task TableHashAsync(NpgsqlConnection connection, string table, int serverId, CancellationToken ct) + { + var hash = await ScalarListAsync(connection, + $"SELECT md5(COALESCE(string_agg(t::text, E'\\n' ORDER BY t::text), '')) || '|' || count(*)::text FROM {table} t WHERE server_id = {serverId}", ct); + return Assert.Single(hash); + } + + // ── what the monitored server hands back ── + + private static Type QueryStatsColumnType(int ordinal) => ordinal switch + { + 3 or 4 => typeof(DateTime), + 0 or 1 or 2 or 36 or 37 or 38 or 42 => typeof(string), + 40 or 41 or 43 => typeof(int), + _ => typeof(long), + }; + + private static DbDataReader QueryStatsReader() + { + var table = new DataTable(); + for (var i = 0; i < 44; i++) table.Columns.Add("c" + i.ToString(System.Globalization.CultureInfo.InvariantCulture), QueryStatsColumnType(i)); + table.Columns.Add("query_plan_xml", typeof(string)); + table.Columns.Add("query_plan_xml_bytes", typeof(long)); + foreach (var (hash, text, plan) in new[] + { + ("0xC0", StatementScrubCanary.CanaryStatement, StatementScrubCanary.CanaryPlan()), + ("0xP0", StatementScrubCanary.PlainStatement, (string?)null), + }) + { + var values = new object[table.Columns.Count]; + for (var i = 0; i < values.Length; i++) values[i] = DBNull.Value; + values[1] = hash; + values[36] = hash + "A"; + values[37] = hash + "B"; + values[38] = text; + values[44] = (object?)plan ?? DBNull.Value; + values[45] = (long)(plan?.Length ?? 0); + table.Rows.Add(values); + } + + return table.CreateDataReader(); + } + + private static DbDataReader SnapshotsReader() + { + var table = new DataTable(); + var types = new Type[35]; + for (var i = 0; i < types.Length; i++) types[i] = typeof(string); + foreach (int i in new[] { 0, 7, 18, 19, 23, 34 }) types[i] = typeof(int); + foreach (int i in new[] { 9, 11, 12, 13, 14, 15 }) types[i] = typeof(long); + foreach (int i in new[] { 16, 24 }) types[i] = typeof(decimal); + types[25] = typeof(bool); + foreach (int i in new[] { 27, 28, 29, 30, 31, 32 }) types[i] = typeof(double); + types[33] = typeof(DateTime); + for (var i = 0; i < types.Length; i++) table.Columns.Add("c" + i.ToString(System.Globalization.CultureInfo.InvariantCulture), types[i]); + + foreach (var (session, text, plan) in new[] + { + (81, StatementScrubCanary.CanaryStatement, StatementScrubCanary.CanaryPlan()), + (82, StatementScrubCanary.PlainStatement, (string?)null), + }) + { + var values = new object[types.Length]; + for (var i = 0; i < values.Length; i++) values[i] = DBNull.Value; + values[0] = session; + values[1] = Db; + values[3] = text; + values[4] = (object?)plan ?? DBNull.Value; + values[5] = (object?)plan ?? DBNull.Value; + values[6] = "running"; + values[26] = "0x" + session.ToString(System.Globalization.CultureInfo.InvariantCulture); + table.Rows.Add(values); + } + + return table.CreateDataReader(); + } + + private static DbDataReader DeadlocksReader(bool withPlan) + { + var data = new DataSet(); + var rows = new DataTable(); + rows.Columns.Add("time", typeof(DateTime)); + rows.Columns.Add("victim", typeof(string)); + rows.Columns.Add("graph", typeof(string)); + if (withPlan) rows.Columns.Add("plan", typeof(string)); + rows.Columns.Add("source", typeof(string)); + foreach (var (minute, victimText, otherText) in new[] + { + (1, StatementScrubCanary.CanaryStatement, StatementScrubCanary.PlainStatement), + (2, StatementScrubCanary.PlainStatement, StatementScrubCanary.PlainStatement), + }) + { + var values = new List { Anchor.AddMinutes(minute), "p1", Graph(victimText, otherText) }; + if (withPlan) values.Add(minute == 1 ? StatementScrubCanary.CanaryPlan() : DBNull.Value); + values.Add(DBNull.Value); + rows.Rows.Add(values.ToArray()); + } + + var gate = new DataTable(); + gate.Columns.Add("count", typeof(long)); + gate.Columns.Add("gated", typeof(bool)); + gate.Rows.Add(100L, false); + data.Tables.Add(rows); + data.Tables.Add(gate); + return data.CreateDataReader(); + } + + private sealed class FakeTargetProvider(Func reader) : ITargetProvider + { + public CollectorTargetEngine Engine => CollectorTargetEngine.SqlServer; + + public DbConnection CreateConnection(string connectionString) => new FakeConnection(connectionString); + + public DbCommand CreateCommand(CollectorQuery query, DbConnection connection, int commandTimeoutSeconds) => + new FakeCommand(((FakeConnection)connection).Database1, reader); + + public CollectorTargetFault Classify(Exception exception, bool yieldsOnLockTimeout) => CollectorTargetFault.Unclassified; + + public string WithDatabase(string connectionString, string databaseName) => "db=" + databaseName; + + public (string ConnectionString, CollectorQuery Query) BuildDatabaseListPlan( + string connectionString, IReadOnlyList? excludedDatabases, IReadOnlyList? databaseScope) => + throw new NotSupportedException(); + } + + private sealed class FakeConnection(string connectionString) : DbConnection + { + private string _cs = connectionString; + public string Database1 => _cs.StartsWith("db=", StringComparison.Ordinal) ? _cs[3..] : "master"; + + [System.Diagnostics.CodeAnalysis.AllowNull] + public override string ConnectionString { get => _cs; set => _cs = value ?? string.Empty; } + public override string Database => string.Empty; + public override string DataSource => string.Empty; + public override string ServerVersion => string.Empty; + public override ConnectionState State => ConnectionState.Open; + public override void ChangeDatabase(string databaseName) { } + public override void Close() { } + public override void Open() { } + public override Task OpenAsync(CancellationToken cancellationToken) => Task.CompletedTask; + protected override DbTransaction BeginDbTransaction(IsolationLevel isolationLevel) => throw new NotSupportedException(); + protected override DbCommand CreateDbCommand() => throw new NotSupportedException(); + } + + private sealed class FakeCommand(string database, Func reader) : DbCommand + { + [System.Diagnostics.CodeAnalysis.AllowNull] + public override string CommandText { get; set; } = string.Empty; + public override int CommandTimeout { get; set; } + public override CommandType CommandType { get; set; } + public override bool DesignTimeVisible { get; set; } + public override UpdateRowSource UpdatedRowSource { get; set; } + protected override DbConnection? DbConnection { get; set; } + protected override DbParameterCollection DbParameterCollection => throw new NotSupportedException(); + protected override DbTransaction? DbTransaction { get; set; } + public override void Cancel() { } + public override int ExecuteNonQuery() => throw new NotSupportedException(); + public override object? ExecuteScalar() => throw new NotSupportedException(); + public override void Prepare() { } + protected override DbParameter CreateDbParameter() => throw new NotSupportedException(); + protected override DbDataReader ExecuteDbDataReader(CommandBehavior behavior) => reader(database); + protected override Task ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken) => + Task.FromResult(reader(database)); + } +} diff --git a/Darling/Darling.Tests/StatementScrubEventsTests.cs b/Darling/Darling.Tests/StatementScrubEventsTests.cs new file mode 100644 index 0000000000..0cdcabf1f8 --- /dev/null +++ b/Darling/Darling.Tests/StatementScrubEventsTests.cs @@ -0,0 +1,289 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Data; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348, collection census for the event and history collectors (R5): long-query completions +/// statement_text, default-trace text_data, system_health event_xml and job history +/// message. Each case plants the canary in the collector's INPUT (a reader over a ), +/// runs the definition's read and then its WritePayload into a , and +/// asserts on what was written: no secret needle, the marker where the statement was, and the plain statement is the +/// SAME instance the reader returned. The cases are registered in . +/// +public sealed partial class StatementCollectionCensusTests +{ + private static readonly DateTime Stamp = new(2026, 9, 22, 12, 0, 0, DateTimeKind.Utc); + + private static CollectorContext EventsContext() => new() + { + ServerId = 42, + ServerName = "test-server", + CollectionTime = Stamp, + Deltas = new EventsNoDeltas(), + Target = new CollectorTargetInfo(), + }; + + private sealed class EventsNoDeltas : ICollectorDeltaCalculator + { + public long CalculateDelta(int serverId, string collectorName, string key, long currentValue, + DateTime? collectionTime = null, int maxGapSeconds = 0) => currentValue; + + public long CalculateDeltaWithInterval(int serverId, string collectorName, string key, long currentValue, + out int intervalSeconds, DateTime? collectionTime = null, int maxGapSeconds = 0) + { + intervalSeconds = 0; + return currentValue; + } + } + + private static DataTable EventsTable(params (string Name, Type Type)[] columns) + { + var t = new DataTable("payload"); + foreach (var (name, type) in columns) + { + t.Columns.Add(name, type); + } + + return t; + } + + private static void AssertNoSecret(StatementScrubRecordingWriter writer) + { + foreach (var needle in StatementScrubCanary.SecretNeedles.Append("canary_ssf")) + { + Assert.False(writer.AnyStringContains(needle), "a written value still holds " + needle); + } + } + + // ── long_query_completions.statement_text ── + + private static DataTable LongQueryInput(string canary, string plain) + { + var t = EventsTable( + ("event_time", typeof(DateTime)), ("event_type", typeof(string)), ("duration", typeof(long)), + ("cpu_time", typeof(long)), ("physical_reads", typeof(long)), ("logical_reads", typeof(long)), + ("writes", typeof(long)), ("row_count", typeof(long)), ("result", typeof(string)), + ("statement_text", typeof(string)), ("object_name", typeof(string)), ("client_app_name", typeof(string)), + ("client_pid", typeof(int)), ("database_id", typeof(int)), ("database_name", typeof(string)), + ("event_sequence", typeof(long)), ("nt_username", typeof(string)), ("query_hash", typeof(string)), + ("server_principal_name", typeof(string)), ("session_id", typeof(int))); + foreach (var text in new[] { canary, plain }) + { + t.Rows.Add(Stamp, "sql_batch_completed", 5_000_000L, 4_000_000L, 1L, 2L, 3L, 4L, "OK", text, "obj", "app", + 100, 5, "db", 7L, "nt", "0xABCD", "login", 55); + } + + return t; + } + + private static async Task RunLongQuery(DataTable input, CollectorContext context) + { + var def = LongQueryCompletionsCollector.Instance; + using var reader = input.CreateDataReader(); + var rows = await def.ReadAsync(reader, context, CancellationToken.None); + var writer = new StatementScrubRecordingWriter(); + foreach (var row in rows) + { + def.WritePayload(row, writer, context); + } + + return writer; + } + + [Fact] + public async Task LongQueryCompletions_StatementText_IsWithheldAndThePlainStatementIsUntouched() + { + var input = LongQueryInput(StatementScrubCanary.CanaryStatement, StatementScrubCanary.PlainStatement); + var plainInstance = (string)input.Rows[1]["statement_text"]; + + var writer = await RunLongQuery(input, EventsContext()); + var again = await RunLongQuery(input, EventsContext()); + + AssertNoSecret(writer); + Assert.Contains(SensitiveStatements.PlaceholderText, writer.Strings); + Assert.Contains(writer.Strings, s => ReferenceEquals(s, plainInstance)); + Assert.Equal(writer.Values, again.Values); + } + + // ── default_trace_events.text_data ── + + private static DataTable DefaultTraceInput(string canary, string plain) + { + var t = EventsTable( + ("event_time", typeof(DateTime)), ("event_name", typeof(string)), ("event_class", typeof(int)), + ("spid", typeof(int)), ("database_name", typeof(string)), ("database_id", typeof(int)), + ("login_name", typeof(string)), ("host_name", typeof(string)), ("application_name", typeof(string)), + ("object_name", typeof(string)), ("filename", typeof(string)), ("integer_data", typeof(long)), + ("integer_data_2", typeof(long)), ("text_data", typeof(string)), ("session_login_name", typeof(string)), + ("error_number", typeof(int)), ("severity", typeof(int)), ("state", typeof(int)), + ("event_sequence", typeof(long)), ("duration_us", typeof(long)), ("end_time", typeof(DateTime))); + foreach (var text in new[] { canary, plain }) + { + t.Rows.Add(Stamp, "Object:Created", 164, 55, "db", 5, "login", "host", "app", "obj", "f.trc", 1L, 2L, + text, "slogin", 0, 0, 0, 9L, 100L, Stamp); + } + + return t; + } + + private static async Task RunDefaultTrace(DataTable input, CollectorContext context) + { + var def = DefaultTraceEventsCollector.Instance; + using var reader = input.CreateDataReader(); + var rows = await def.ReadAsync(reader, context, CancellationToken.None); + var writer = new StatementScrubRecordingWriter(); + foreach (var row in rows) + { + def.WritePayload(row, writer, context); + } + + return writer; + } + + [Fact] + public async Task DefaultTraceEvents_TextData_IsWithheldAndThePlainStatementIsUntouched() + { + var input = DefaultTraceInput(StatementScrubCanary.CanaryStatement, StatementScrubCanary.PlainStatement); + var plainInstance = (string)input.Rows[1]["text_data"]; + + var writer = await RunDefaultTrace(input, EventsContext()); + var again = await RunDefaultTrace(input, EventsContext()); + + AssertNoSecret(writer); + Assert.Contains(SensitiveStatements.PlaceholderText, writer.Strings); + Assert.Contains(writer.Strings, s => ReferenceEquals(s, plainInstance)); + Assert.Equal(writer.Values, again.Values); + } + + // ── system_health_events.event_xml ── + + private static string WaitInfoEvent(string sqlText) => + "" + + "11LCK_M_X" + + "9000" + + "3" + + "" + System.Security.SecurityElement.Escape(sqlText) + "" + + "55" + + ""; + + private static DataTable SystemHealthInput(string canaryXml, string plainXml) + { + var t = EventsTable(("event_time", typeof(DateTime)), ("event_type", typeof(string)), ("event_xml", typeof(string))); + t.Rows.Add(Stamp, "wait_info", canaryXml); + t.Rows.Add(Stamp, "wait_info", plainXml); + return t; + } + + private static async Task<(StatementScrubRecordingWriter Writer, CollectorContext Context)> RunSystemHealth(DataTable input) + { + var context = EventsContext(); + var def = SystemHealthEventsCollector.Instance; + using var reader = input.CreateDataReader(); + var rows = await def.ReadAsync(reader, context, CancellationToken.None); + var writer = new StatementScrubRecordingWriter(); + foreach (var row in rows) + { + def.WritePayload(row, writer, context); + } + + return (writer, context); + } + + [Fact] + public async Task SystemHealthEvents_EventXml_WithholdsTheSqlTextActionAndKeepsTheRestOfTheEvent() + { + var canaryXml = WaitInfoEvent(StatementScrubCanary.CanaryStatement); + var plainXml = WaitInfoEvent(StatementScrubCanary.PlainStatement); + var input = SystemHealthInput(canaryXml, plainXml); + var plainInstance = (string)input.Rows[1]["event_xml"]; + + var (writer, context) = await RunSystemHealth(input); + var (again, _) = await RunSystemHealth(input); + + AssertNoSecret(writer); + var written = writer.Strings.Where(s => s.StartsWith(" m.Label == CollectorContext.StatementScrubNamedMeasurement && m.Value == 1); + } + + // ── job_history.message ── + + private static DataTable JobHistoryInput(string canary, string plain) + { + var t = EventsTable( + ("instance_id", typeof(long)), ("job_id", typeof(string)), ("job_name", typeof(string)), + ("job_enabled", typeof(bool)), ("category_name", typeof(string)), ("step_id", typeof(int)), + ("step_name", typeof(string)), ("run_status", typeof(int)), ("run_status_desc", typeof(string)), + ("run_datetime", typeof(DateTime)), ("run_duration_seconds", typeof(long)), + ("retries_attempted", typeof(int)), ("message", typeof(string))); + long id = 1; + foreach (var text in new[] { canary, plain }) + { + t.Rows.Add(id++, "00000000-0000-0000-0000-000000000001", "job", true, "cat", 1, "step", 0, "Failed", + Stamp, 61L, 0, text); + } + + return t; + } + + private static async Task RunJobHistory(DataTable input, CollectorContext context) + { + var def = JobHistoryCollector.Instance; + using var reader = input.CreateDataReader(); + var rows = await def.ReadAsync(reader, context, CancellationToken.None); + var writer = new StatementScrubRecordingWriter(); + foreach (var row in rows) + { + def.WritePayload(row, writer, context); + } + + return writer; + } + + [Fact] + public async Task JobHistory_Message_IsWithheldWhenItEchoesAStatement_AndAPlainMessageIsUntouched() + { + var canaryMessage = "Executed as user: sa. " + StatementScrubCanary.CanaryStatement + " [SQLSTATE 42000] (Error 156)"; + var input = JobHistoryInput(canaryMessage, "The job succeeded. The Job was invoked by Schedule 3."); + var plainInstance = (string)input.Rows[1]["message"]; + + var writer = await RunJobHistory(input, EventsContext()); + var again = await RunJobHistory(input, EventsContext()); + + AssertNoSecret(writer); + Assert.Contains(SensitiveStatements.PlaceholderText, writer.Strings); + Assert.Contains(writer.Strings, s => ReferenceEquals(s, plainInstance)); + Assert.Equal(writer.Values, again.Values); + } +} diff --git a/Darling/Darling.Tests/StatementScrubPlanCommandResultTests.cs b/Darling/Darling.Tests/StatementScrubPlanCommandResultTests.cs new file mode 100644 index 0000000000..8abb0a6a4d --- /dev/null +++ b/Darling/Darling.Tests/StatementScrubPlanCommandResultTests.cs @@ -0,0 +1,177 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5320 (the plan viewers half of #4348), the Darling service's side. The fetch_plan and +/// execute_actual_plan commands read a plan live from the monitored server and report it in +/// config.config_command.result_json. The row stays in the store until the terminal-command purge when the +/// viewer never reads it, so the plan is judged by the statement filter BEFORE the result is written +/// (DarlingWorker.PlanResultOutcome): the stored result never holds the raw text. +/// +/* The round-trip rows go through config.config_command, the shared live store: the collection serializes this class + with the other live classes so its inserts and deletes cannot race another class's assertions. */ +[Collection("live-postgres")] +public sealed class StatementScrubPlanCommandResultTests +{ + /// Distinctive sentinel, like the other command round-trip tests: a real server_id is a hash. + private const int SentinelServerId = -515152; + + private static string PlanXmlOf(string? resultJson) + { + using var document = JsonDocument.Parse(resultJson!); + Assert.True(document.RootElement.GetProperty("success").GetBoolean()); + return document.RootElement.GetProperty("planXml").GetString()!; + } + + private static void AssertFiltered(string plan) + { + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.DoesNotContain(needle, plan, StringComparison.Ordinal); + } + + foreach (var needle in StatementScrubCanary.KeptNeedles) + { + Assert.Contains(needle, plan, StringComparison.Ordinal); + } + } + + [Theory] + [InlineData("plan fetched")] + [InlineData("actual plan captured")] + public void PlanResultOutcome_JudgesThePlanBeforeItIsSerialized(string resultStatus) + { + var raw = StatementScrubCanary.CanaryPlan(); + + var outcome = DarlingWorker.PlanResultOutcome(resultStatus, raw); + + Assert.True(outcome.Success); + Assert.Equal(resultStatus, outcome.ResultStatus); + AssertFiltered(PlanXmlOf(outcome.ResultJson)); + } + + [Fact] + public void PlanResultOutcome_ANamedPlanThatCannotBeJudged_IsStoredAsTheMarker_AndAPlainPlanIsUnchanged() + { + var plain = ""; + Assert.Equal(plain, PlanXmlOf(DarlingWorker.PlanResultOutcome("plan fetched", plain).ResultJson)); + + /* What the filter hands back for this text is what is stored, so whatever it withholds whole is the marker. */ + var odd = "A host that answers each plan command the way the worker's handlers do after the live read: through + /// DarlingWorker.PlanResultOutcome, with the raw canary plan the monitored server would have returned. + private sealed class RawPlanHost : IDarlingCommandHost + { + public Task FetchPlanAsync(int serverId, PlanFetchRequest request, CancellationToken cancellationToken) + => Task.FromResult(DarlingWorker.PlanResultOutcome("plan fetched", StatementScrubCanary.CanaryPlan())); + + public Task ExecuteActualPlanAsync(int serverId, ActualPlanRequest request, CancellationToken cancellationToken) + => Task.FromResult(DarlingWorker.PlanResultOutcome("actual plan captured", StatementScrubCanary.CanaryPlan())); + + public Task SnapshotNowAsync(int serverId, CancellationToken cancellationToken) + => throw new InvalidOperationException("not this command"); + + public Task AnalyzeNowAsync(int serverId, CancellationToken cancellationToken) + => throw new InvalidOperationException("not this command"); + + public Task PurgeNowAsync(int? customRetentionDays, CancellationToken cancellationToken) + => throw new InvalidOperationException("not this command"); + + public Task TestHypotheticalIndexAsync(int serverId, HypotheticalIndexRequest request, CancellationToken cancellationToken) + => throw new InvalidOperationException("not this command"); + + public Task FetchActiveQueriesLiveAsync(int serverId, CancellationToken cancellationToken) + => throw new InvalidOperationException("not this command"); + } + + [Theory] + [InlineData("fetch_plan", "{\"planHandle\":\"0x0600AB\",\"databaseName\":\"master\"}", "plan fetched")] + [InlineData("execute_actual_plan", "{\"queryHash\":\"0x0102030405060708\",\"databaseName\":\"master\"}", "actual plan captured")] + public async Task ThePlanCommandPath_NeverWritesTheRawPlanIntoResultJson(string commandType, string argsJson, string expectedStatus) + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the plan command result_json round-trip."); + + var ct = TestContext.Current.CancellationToken; + var bodySucceeded = false; + try + { + await using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteSentinelRowsAsync(connection, ct); + + long commandId; + using (var insert = new NpgsqlCommand( + "INSERT INTO config.config_command (command_type, target_server_id, args_json, status) " + + "VALUES ($1, $2, $3::jsonb, 'pending') RETURNING command_id", connection)) + { + insert.Parameters.AddWithValue(commandType); + insert.Parameters.AddWithValue(SentinelServerId); + insert.Parameters.AddWithValue(argsJson); + commandId = Convert.ToInt64(await insert.ExecuteScalarAsync(ct)); + } + + await using var postgres = NpgsqlDataSource.Create(connectionString!); + var executor = new DarlingCommandExecutor(postgres, new RawPlanHost(), "test-instance", null); + + var claimed = await executor.ClaimNextAsync(ct); + Assert.NotNull(claimed); + Assert.Equal(commandType, claimed!.CommandType); + await executor.ExecuteAndReportAsync(claimed, ct); + + using var read = new NpgsqlCommand( + "SELECT status, result_status, result_json::text FROM config.config_command WHERE command_id = $1", connection); + read.Parameters.AddWithValue(commandId); + using var reader = await read.ExecuteReaderAsync(ct); + Assert.True(await reader.ReadAsync(ct)); + Assert.Equal("succeeded", reader.GetString(0)); + Assert.Equal(expectedStatus, reader.GetString(1)); + + /* The stored result holds the filtered plan: the secret values the raw plan carried are not in the row. */ + var stored = reader.GetString(2); + AssertFiltered(PlanXmlOf(stored)); + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.DoesNotContain(needle, stored, StringComparison.Ordinal); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, bodySucceeded, async (cleanup, cleanupCt) => + { + await DeleteSentinelRowsAsync(cleanup, cleanupCt); + }); + } + } + + private static async Task DeleteSentinelRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + using var delete = new NpgsqlCommand( + $"DELETE FROM config.config_command WHERE target_server_id = {SentinelServerId}", connection); + await delete.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/StatementScrubQueryStatsLiveTests.cs b/Darling/Darling.Tests/StatementScrubQueryStatsLiveTests.cs new file mode 100644 index 0000000000..fb45d2fac7 --- /dev/null +++ b/Darling/Darling.Tests/StatementScrubQueryStatsLiveTests.cs @@ -0,0 +1,310 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Data; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348 (statement filter, lane R2) against a real PostgreSQL (gated on DARLING_TEST_PG): what query_stats +/// leaves in query_text_dim and query_plan_dim after the statement filter ran at collection time. +/// Rows go through the collector's own ReadAsync and WritePayload, the real +/// and , in one transaction as the host does. +/// +/// The cached digest of the deferred plan fetch is computed exactly as the host computes it (the digest of +/// the filtered plan), so a digest-only row written from it must resolve to the filtered dim row, and a plan the +/// filter withheld whole must resolve to the marker's dim row while the next cycle's filtered plan resolves to its +/// own (no digest-only write is ever left without a dim row). +/// +[Collection("live-postgres")] +public sealed class StatementScrubQueryStatsLiveTests +{ + private const string SkipReason = "Set DARLING_TEST_PG to a Postgres connection string to run the statement filter store tests."; + + private static readonly DateTime Anchor = new(2031, 4, 10, 12, 0, 0, DateTimeKind.Unspecified); + + private sealed class NoDeltas : ICollectorDeltaCalculator + { + public long CalculateDelta(int serverId, string collectorName, string key, long currentValue, + DateTime? collectionTime = null, int maxGapSeconds = 0) => currentValue; + + public long CalculateDeltaWithInterval(int serverId, string collectorName, string key, long currentValue, + out int intervalSeconds, DateTime? collectionTime = null, int maxGapSeconds = 0) + { + intervalSeconds = 0; + return currentValue; + } + } + + private static Type ColumnType(int ordinal) => ordinal switch + { + 3 or 4 => typeof(DateTime), + 0 or 1 or 2 or 36 or 37 or 38 or 42 => typeof(string), + 40 or 41 or 43 => typeof(int), + _ => typeof(long), + }; + + /// One inline-plan query_stats row, read through the collector (so the filter runs where it does live). + private static async Task ReadRowAsync( + string queryHash, string text, string plan, int serverId, string serverName) + { + var context = ContextFor(serverId, serverName, Anchor); + using var table = new DataTable(); + for (var i = 0; i < 44; i++) + { + table.Columns.Add("c" + i.ToString(CultureInfo.InvariantCulture), ColumnType(i)); + } + + table.Columns.Add("query_plan_xml", typeof(string)); + table.Columns.Add("query_plan_xml_bytes", typeof(long)); + var values = new object[table.Columns.Count]; + for (var i = 0; i < 44; i++) + { + values[i] = DBNull.Value; + } + + values[1] = queryHash; + values[36] = "0x" + queryHash; + values[37] = "0x" + queryHash; + values[38] = text; + values[44] = plan; + values[45] = (long)plan.Length; + table.Rows.Add(values); + + await using var reader = table.CreateDataReader(); + var rows = await QueryStatsCollector.Instance.ReadAsync(reader, context, CancellationToken.None); + return Assert.Single(rows); + } + + private static CollectorContext ContextFor(int serverId, string serverName, DateTime collectionTime) => new() + { + ServerId = serverId, + ServerName = serverName, + CollectionTime = collectionTime, + Deltas = new NoDeltas(), + Target = new CollectorTargetInfo(), + CapturePlanXml = true, + }; + + /// The digest the host caches for a plan it fetched: the digest of the filtered plan. + private static string CachedDigestOf(string filteredPlan) => + Convert.ToHexString(PayloadDimensions.Digest(PgCollectorRowWriter.StripEmbeddedNuls(filteredPlan))); + + private static async Task> WriteBatchAsync( + NpgsqlConnection connection, int serverId, string serverName, DateTime collectionTime, + IReadOnlyList rows, CancellationToken ct) + { + var definition = QueryStatsCollector.Instance; + var context = ContextFor(serverId, serverName, collectionTime); + var writer = new PgCollectorRowWriter(); + var dimensions = new PayloadDimensionBatch(); + writer.UseDimensions(PayloadDimensions.DiversionPlanFor(definition), dimensions); + + await using var transaction = await connection.BeginTransactionAsync(ct); + using (var importer = await connection.BeginBinaryImportAsync(PgCollectorRowWriter.CopyCommandFor(definition), ct)) + { + writer.Importer = importer; + foreach (var row in rows) + { + await importer.StartRowAsync(ct); + writer.Value(CollectionIdGenerator.Next()); + writer.Value(collectionTime).Value(serverId).Value(serverName); + writer.BeginPayload(); + definition.WritePayload(row, writer, context); + writer.EndPayload(definition.PayloadColumns.Count); + } + + await importer.CompleteAsync(ct); + } + + var absent = await PayloadDimensionWriter.FlushAsync(connection, transaction, dimensions, collectionTime, ct); + await transaction.CommitAsync(ct); + return absent; + } + + private static async Task OpenMigratedStoreAsync(string connectionString, CancellationToken ct) + { + var builder = new NpgsqlConnectionStringBuilder(connectionString) { SearchPath = PgSchemaGenerator.SearchPath }; + var connection = new NpgsqlConnection(builder.ConnectionString); + try + { + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + } + catch + { + await connection.DisposeAsync(); + throw; + } + + return connection; + } + + /// The text and plan a stored row resolves to through the view, the way every reader sees them. + private static async Task<(string? Text, string? Plan)> ResolvedAsync( + NpgsqlConnection connection, int serverId, string queryHash, CancellationToken ct) + { + await using var read = new NpgsqlCommand( + "SELECT query_text, query_plan_xml, query_plan_gz FROM v_query_stats WHERE server_id = $1 AND query_hash = $2", connection); + read.Parameters.AddWithValue(serverId); + read.Parameters.AddWithValue(queryHash); + await using var reader = await read.ExecuteReaderAsync(ct); + Assert.True(await reader.ReadAsync(ct), "the row must exist"); + return ( + reader.IsDBNull(0) ? null : reader.GetString(0), + PayloadDimensions.ResolveContent( + reader.IsDBNull(1) ? null : reader.GetString(1), + reader.IsDBNull(2) ? null : reader.GetFieldValue(2))); + } + + private static async Task CleanupAsync(NpgsqlConnection cleanup, int serverId, IEnumerable planDigests, CancellationToken ct) + { + await using (var facts = new NpgsqlCommand("DELETE FROM query_stats WHERE server_id = $1", cleanup)) + { + facts.Parameters.AddWithValue(serverId); + await facts.ExecuteNonQueryAsync(ct); + } + + foreach (var digest in planDigests) + { + await using var dim = new NpgsqlCommand("DELETE FROM query_plan_dim WHERE digest = $1", cleanup); + dim.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Bytea, Value = digest }); + await dim.ExecuteNonQueryAsync(ct); + } + + await using var text = new NpgsqlCommand("DELETE FROM query_text_dim WHERE digest = $1", cleanup); + text.Parameters.Add(new NpgsqlParameter + { + NpgsqlDbType = NpgsqlDbType.Bytea, + Value = PayloadDimensions.Digest(PgCollectorRowWriter.StripEmbeddedNuls(SensitiveStatements.PlaceholderText)), + }); + await text.ExecuteNonQueryAsync(ct); + } + + private static (int ServerId, string ServerName) NewServer() => + (-Random.Shared.Next(1_000_000, int.MaxValue), "pm4348-" + Guid.NewGuid().ToString("N")[..12]); + + [Fact] + public async Task TheDimensionsHoldTheMarker_AndTheCachedDigestResolvesToTheFilteredPlanOnTheNextCycle() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), SkipReason); + + var ct = TestContext.Current.CancellationToken; + var (serverId, serverName) = NewServer(); + var rawPlan = StatementScrubCanary.CanaryPlan(); + await using var connection = await OpenMigratedStoreAsync(connectionString!, ct); + var digests = new List(); + var bodySucceeded = false; + try + { + var first = await ReadRowAsync("0xA1", StatementScrubCanary.CanaryStatement, rawPlan, serverId, serverName); + var filteredDigest = CachedDigestOf(first.QueryPlanXml!); + digests.Add(Convert.FromHexString(filteredDigest)); + Assert.Equal(filteredDigest, CachedDigestOf((await ReadRowAsync("0xA2", StatementScrubCanary.CanaryStatement, rawPlan, serverId, serverName)).QueryPlanXml!)); + + await WriteBatchAsync(connection, serverId, serverName, Anchor, new[] { first }, ct); + + /* The second cycle: the host has the digest and sends no plan text. */ + var second = await ReadRowAsync("0xB1", StatementScrubCanary.PlainStatement, rawPlan, serverId, serverName); + second.QueryPlanXml = null; + second.KnownPlanDigest = filteredDigest; + var absent = await WriteBatchAsync(connection, serverId, serverName, Anchor.AddMinutes(1), new[] { second }, ct); + + Assert.Empty(absent); + var (text1, plan1) = await ResolvedAsync(connection, serverId, "0xA1", ct); + var (text2, plan2) = await ResolvedAsync(connection, serverId, "0xB1", ct); + Assert.Equal(SensitiveStatements.PlaceholderText, text1); + Assert.Equal(StatementScrubCanary.PlainStatement, text2); + Assert.Equal(first.QueryPlanXml, plan1); + Assert.Equal(first.QueryPlanXml, plan2); + StatementFilterCensus.AssertPlanFilteredKeepsTheRest(rawPlan, plan2!); + + /* Nothing the raw statement or the raw plan would have stored is in the dimensions: neither raw digest has a + row, and the stored plan holds no secret needle. */ + Assert.Equal(0L, await CountByDigestAsync( + connection, "query_text_dim", PayloadDimensions.Digest(PgCollectorRowWriter.StripEmbeddedNuls(StatementScrubCanary.CanaryStatement)), ct)); + Assert.Equal(0L, await CountByDigestAsync( + connection, "query_plan_dim", PayloadDimensions.Digest(PgCollectorRowWriter.StripEmbeddedNuls(rawPlan)), ct)); + Assert.Equal(1L, await CountByDigestAsync( + connection, "query_text_dim", PayloadDimensions.Digest(PgCollectorRowWriter.StripEmbeddedNuls(SensitiveStatements.PlaceholderText)), ct)); + Assert.Equal(1L, await CountByDigestAsync(connection, "query_plan_dim", digests[0], ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, bodySucceeded, async (cleanup, cleanupCt) => + await CleanupAsync(cleanup, serverId, digests, cleanupCt)); + } + } + + /// r2 Q5: a plan the filter withheld whole stores the marker's dim row; the next cycle's filtered plan stores + /// its own; a digest-only write resolves for each. + [Fact] + public async Task AWholeMarkerPlan_AndTheNextCyclesFilteredPlan_BothResolveToDimRows() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), SkipReason); + + var ct = TestContext.Current.CancellationToken; + var (serverId, serverName) = NewServer(); + var rawPlan = StatementScrubCanary.CanaryPlan(); + await using var connection = await OpenMigratedStoreAsync(connectionString!, ct); + var digests = new List(); + var bodySucceeded = false; + try + { + var whole = await ReadRowAsync("0xC1", StatementScrubCanary.PlainStatement, rawPlan, serverId, serverName); + whole.QueryPlanXml = SensitiveStatements.PlaceholderText; + var filtered = await ReadRowAsync("0xC2", StatementScrubCanary.PlainStatement, rawPlan, serverId, serverName); + var wholeDigest = CachedDigestOf(SensitiveStatements.PlaceholderText); + var filteredDigest = CachedDigestOf(filtered.QueryPlanXml!); + digests.Add(Convert.FromHexString(wholeDigest)); + digests.Add(Convert.FromHexString(filteredDigest)); + Assert.NotEqual(wholeDigest, filteredDigest); + + Assert.Empty(await WriteBatchAsync(connection, serverId, serverName, Anchor, new[] { whole }, ct)); + Assert.Empty(await WriteBatchAsync(connection, serverId, serverName, Anchor.AddMinutes(1), new[] { filtered }, ct)); + + var digestOnly = await ReadRowAsync("0xC3", StatementScrubCanary.PlainStatement, rawPlan, serverId, serverName); + digestOnly.QueryPlanXml = null; + digestOnly.KnownPlanDigest = filteredDigest; + Assert.Empty(await WriteBatchAsync(connection, serverId, serverName, Anchor.AddMinutes(2), new[] { digestOnly }, ct)); + + Assert.Equal(SensitiveStatements.PlaceholderText, (await ResolvedAsync(connection, serverId, "0xC1", ct)).Plan); + Assert.Equal(filtered.QueryPlanXml, (await ResolvedAsync(connection, serverId, "0xC2", ct)).Plan); + Assert.Equal(filtered.QueryPlanXml, (await ResolvedAsync(connection, serverId, "0xC3", ct)).Plan); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, bodySucceeded, async (cleanup, cleanupCt) => + await CleanupAsync(cleanup, serverId, digests, cleanupCt)); + } + } + + private static async Task CountByDigestAsync(NpgsqlConnection connection, string table, byte[] digest, CancellationToken ct) + { + await using var command = new NpgsqlCommand("SELECT count(*) FROM " + table + " WHERE digest = $1", connection); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Bytea, Value = digest }); + return (long)(await command.ExecuteScalarAsync(ct))!; + } +} diff --git a/Darling/Darling.Tests/StatementScrubQueryStatsTests.cs b/Darling/Darling.Tests/StatementScrubQueryStatsTests.cs new file mode 100644 index 0000000000..754470b01c --- /dev/null +++ b/Darling/Darling.Tests/StatementScrubQueryStatsTests.cs @@ -0,0 +1,449 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Data; +using System.Globalization; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348 (statement filter, lane R2): the collection census for query_stats and procedure_stats. The +/// canary is planted in the collector's INPUT (a reader over a , as +/// and do), the read and the write +/// run, and remembers what was written: no written value holds a secret +/// needle, the statement column is exactly the marker, the plan parses with the marker in statement 1 and the kept +/// needles in statements 2 and 3, and a plain statement is the same instance the reader returned. +/// +public sealed partial class StatementCollectionCensusTests +{ + private static readonly DateTime s_now = new(2026, 7, 2, 12, 0, 0, DateTimeKind.Utc); + + private sealed class NoDeltas : ICollectorDeltaCalculator + { + public long CalculateDelta(int serverId, string collectorName, string key, long currentValue, + DateTime? collectionTime = null, int maxGapSeconds = 0) => currentValue; + + public long CalculateDeltaWithInterval(int serverId, string collectorName, string key, long currentValue, + out int intervalSeconds, DateTime? collectionTime = null, int maxGapSeconds = 0) + { + intervalSeconds = 0; + return currentValue; + } + } + + private static CollectorContext Context(bool capture, bool defer = false, bool identity = false) => new() + { + ServerId = 42, + ServerName = "test-server", + CollectionTime = s_now, + Deltas = new NoDeltas(), + Target = new CollectorTargetInfo(), + CapturePlanXml = capture, + DeferPlanXmlFetch = defer, + PlanIdentityColumns = identity, + }; + + private static Type QueryStatsColumnType(int ordinal) => ordinal switch + { + 3 or 4 => typeof(DateTime), + 0 or 1 or 2 or 36 or 37 or 38 or 42 => typeof(string), + 40 or 41 or 43 => typeof(int), + _ => typeof(long), + }; + + /// A query_stats result with the given statement text and (optionally) inline plan columns. + private static async Task> ReadQueryStatsAsync( + CollectorContext context, params (string? Text, string? Plan)[] rows) + { + using var table = new DataTable(); + for (var i = 0; i < 44; i++) + { + table.Columns.Add("c" + i.ToString(CultureInfo.InvariantCulture), QueryStatsColumnType(i)); + } + + var withPlan = context.CapturePlanXml && !context.DeferPlanXmlFetch; + if (withPlan) + { + table.Columns.Add("query_plan_xml", typeof(string)); + table.Columns.Add("query_plan_xml_bytes", typeof(long)); + } + + var n = 0; + foreach (var (text, plan) in rows) + { + var values = new object[table.Columns.Count]; + for (var i = 0; i < 44; i++) + { + values[i] = DBNull.Value; + } + + values[1] = "0xQ" + (n++).ToString(CultureInfo.InvariantCulture); + values[36] = "0x01"; + values[37] = "0x02"; + values[38] = (object?)text ?? DBNull.Value; + if (withPlan) + { + values[44] = (object?)plan ?? DBNull.Value; + values[45] = plan is null ? DBNull.Value : (long)plan.Length; + } + + table.Rows.Add(values); + } + + await using var reader = table.CreateDataReader(); + return await QueryStatsCollector.Instance.ReadAsync(reader, context, CancellationToken.None); + } + + private static StatementScrubRecordingWriter Write(QueryStatsCollector.Row row, CollectorContext context) + { + var writer = new StatementScrubRecordingWriter(); + QueryStatsCollector.Instance.WritePayload(row, writer, context); + return writer; + } + + [Fact] + public async Task QueryStats_QueryText_IsWithheld() + { + var context = Context(capture: false); + var plain = new string(StatementScrubCanary.PlainStatement.ToCharArray()); + + var rows = await ReadQueryStatsAsync(context, (StatementScrubCanary.CanaryStatement, null), (plain, null)); + + var named = Write(rows[0], context); + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.False(named.AnyStringContains(needle), needle); + } + + /* the written value of the query_text column is exactly the marker */ + var textColumn = QueryStatsCollector.Instance.PayloadColumns.ToList().FindIndex(c => c.Name == "query_text"); + Assert.True(textColumn >= 0); + Assert.Equal(SensitiveStatements.PlaceholderText, named.Values[textColumn]); + Assert.Equal(SensitiveStatements.PlaceholderText, rows[0].QueryText); + + /* the plain statement is kept (its same-instance pin is the next test) */ + Assert.Equal(plain, rows[1].QueryText); + Assert.Contains(plain, Write(rows[1], context).Strings); + } + + [Fact] + public async Task QueryStats_QueryText_PlainStatementIsTheSameInstanceTheReaderReturned() + { + var context = Context(capture: false); + var plain = new string(StatementScrubCanary.PlainStatement.ToCharArray()); + using var table = new DataTable(); + for (var i = 0; i < 44; i++) + { + table.Columns.Add("c" + i.ToString(CultureInfo.InvariantCulture), QueryStatsColumnType(i)); + } + + var values = new object[44]; + for (var i = 0; i < 44; i++) + { + values[i] = DBNull.Value; + } + + values[38] = plain; + table.Rows.Add(values); + await using var reader = table.CreateDataReader(); + + var rows = await QueryStatsCollector.Instance.ReadAsync(reader, context, CancellationToken.None); + + Assert.Same(plain, Assert.Single(rows).QueryText); + } + + [Fact] + public async Task QueryStats_QueryPlanXml_InlinePlanIsFiltered() + { + var context = Context(capture: true); + var rows = await ReadQueryStatsAsync(context, (null, StatementScrubCanary.CanaryPlan())); + + var written = Write(rows[0], context); + + var plan = Assert.Single(written.Payloads).Content!; + StatementFilterCensus.AssertPlanFilteredKeepsTheRest(StatementScrubCanary.CanaryPlan(), plan); + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.False(written.AnyStringContains(needle), needle); + } + } + + [Fact] + public async Task QueryStats_QueryPlanXml_DeferredFetchPlanIsFiltered_AndTheDigestIsOfTheFilteredPlan() + { + var plan = StatementScrubCanary.CanaryPlan(); + var first = await FetchOneAsync(plan, new SensitiveStatements.Session()); + var second = await FetchOneAsync(plan, new SensitiveStatements.Session()); + + StatementFilterCensus.AssertPlanFilteredKeepsTheRest(plan, first!); + Assert.Equal(first, second); + Assert.Equal( + Convert.ToHexString(PayloadDimensions.Digest(first!)), + Convert.ToHexString(PayloadDimensions.Digest(second!))); + } + + [Fact] + public async Task QueryStats_QueryPlanXml_ASpentBudgetGivesTheWholeMarker_AndTheNextSessionTheFilteredPlan() + { + var plan = StatementScrubCanary.CanaryPlan(); + var spent = SpentSession(); + Assert.True(spent.Spent); + + var whole = await FetchOneAsync(plan, spent); + var next = await FetchOneAsync(plan, new SensitiveStatements.Session()); + + Assert.Equal(SensitiveStatements.PlaceholderText, whole); + Assert.NotEqual(SensitiveStatements.PlaceholderText, next); + StatementFilterCensus.AssertPlanFilteredKeepsTheRest(plan, next!); + } + + /// A session whose 15-second budget is already spent: its fake judge charges 20 seconds to a fake clock. + private static SensitiveStatements.Session SpentSession() + { + var now = TimeSpan.Zero; + var session = new SensitiveStatements.Session( + _ => + { + now += TimeSpan.FromSeconds(20); + return SensitiveStatements.Verdict.Clean; + }, + () => now, + TimeSpan.FromSeconds(15), + new SensitiveStatements.TimedOutMemo()); + session.Text("a value that spends the budget"); + return session; + } + + private static async Task FetchOneAsync(string plan, SensitiveStatements.Session session) + { + using var table = new DataTable(); + table.Columns.Add("ord", typeof(int)); + table.Columns.Add("query_plan_xml", typeof(string)); + table.Columns.Add("query_plan_xml_bytes", typeof(long)); + table.Rows.Add(0, plan, (long)plan.Length); + await using var reader = table.CreateDataReader(); + + var fetched = await QueryStatsCollector.ReadPlanFetchAsync(reader, session, CancellationToken.None); + return fetched[0].PlanXml; + } + + [Fact] + public async Task ProcedureStats_QueryPlanXml_InlinePlanIsFiltered() + { + var context = Context(capture: true); + using var table = new DataTable(); + for (var i = 0; i < 27; i++) + { + table.Columns.Add("c" + i.ToString(CultureInfo.InvariantCulture), i switch + { + 4 or 5 => typeof(DateTime), + 0 or 1 or 2 or 3 or 25 or 26 => typeof(string), + _ => typeof(long), + }); + } + + var values = new object[table.Columns.Count]; + for (var i = 0; i < 27; i++) + { + values[i] = table.Columns[i].DataType == typeof(long) && i is 6 or 7 or 8 or 9 or 10 or 11 or 22 or 23 or 24 ? 1L : DBNull.Value; + } + + table.Rows.Add(values); + + await using var reader = table.CreateDataReader(); + var rows = await ProcedureStatsCollector.Instance.ReadAsync(reader, context, CancellationToken.None); + + /* #5449: the plan enters the row in the plan phase, which is where the filter now runs. */ + using var planTable = new DataTable(); + planTable.Columns.Add("ord", typeof(int)); + planTable.Columns.Add("query_plan_xml", typeof(string)); + planTable.Columns.Add("query_plan_xml_bytes", typeof(long)); + planTable.Rows.Add(0, StatementScrubCanary.CanaryPlan(), 7L); + await using var planReader = planTable.CreateDataReader(); + var phase = await ProcedureStatsCollector.ReadPlanPhaseAsync(planReader, context, CancellationToken.None); + ProcedureStatsCollector.ApplyPlanPhase(rows, new[] { 0 }, phase); + + var writer = new StatementScrubRecordingWriter(); + ProcedureStatsCollector.Instance.WritePayload(Assert.Single(rows), writer, context); + + var plan = Assert.Single(writer.Payloads).Content!; + StatementFilterCensus.AssertPlanFilteredKeepsTheRest(StatementScrubCanary.CanaryPlan(), plan); + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.False(writer.AnyStringContains(needle), needle); + } + } + + /// r2 Q5: a plan the filter withheld whole (budget spent) is stored this cycle but never cached, so the next + /// cycle fetches the plan again and stores its filtered form; a filtered plan IS cached under its own digest. + [Fact] + public async Task ProcedureStats_QueryPlanXml_AWholeMarkerPlanFromTheDeferredFetchIsNotCached_AndTheNextCycleCachesTheFilteredPlan() + { + var cache = new PlanDigestCache(); + var plan = StatementScrubCanary.CanaryPlan(); + var row = default(ProcedureStatsCollector.Row) with + { + DatabaseName = "db1", + SchemaName = "dbo", + ObjectName = "p1", + ObjectType = "P", + CachedTime = s_now.AddHours(-4), + SqlHandle = "0x0301", + PlanHandle = "0x0501", + PlanStatementCount = 3, + PlanLastStatementCompile = s_now.AddHours(-3), + PlanGenerationSum = 7, + }; + + var unjudged = new HashSet(); + ProcedureStatsPlanReuse.PlanFetch FetchThrough(SensitiveStatements.Session session) => async (handles, token) => + { + using var table = new DataTable(); + table.Columns.Add("ord", typeof(int)); + table.Columns.Add("query_plan_xml", typeof(string)); + table.Columns.Add("query_plan_xml_bytes", typeof(long)); + table.Rows.Add(0, plan, (long)plan.Length); + await using var reader = table.CreateDataReader(); + return await QueryStatsCollector.ReadPlanFetchAsync(reader, session, token, unjudged); + }; + + var spent = SpentSession(); + var rows1 = new List { row }; + var first = await ProcedureStatsPlanReuse.ApplyOnAsync(1, cache, rows1, true, 1, s_now, FetchThrough(spent), CancellationToken.None, unjudged); + cache.ConfirmPending(first.Pending, s_now); + + Assert.Equal(SensitiveStatements.PlaceholderText, rows1[0].QueryPlanXml); + Assert.Empty(first.Pending); + + var rows2 = new List { row }; + unjudged.Clear(); + var second = await ProcedureStatsPlanReuse.ApplyOnAsync(1, cache, rows2, true, 2, s_now, FetchThrough(new SensitiveStatements.Session()), CancellationToken.None, unjudged); + cache.ConfirmPending(second.Pending, s_now); + + Assert.Equal(0, second.Hit); + StatementFilterCensus.AssertPlanFilteredKeepsTheRest(plan, rows2[0].QueryPlanXml!); + Assert.Single(second.Pending); + + var rows3 = new List { row }; + var third = await ProcedureStatsPlanReuse.ApplyOnAsync(1, cache, rows3, true, 3, s_now, FetchThrough(new SensitiveStatements.Session()), CancellationToken.None, unjudged); + Assert.Equal(1, third.Hit); + Assert.Equal(ProcedureStatsPlanReuse.DigestOf(rows2[0].QueryPlanXml!), rows3[0].KnownPlanDigest); + } + + /// A prepared plan cut inside its parameter list: it does not parse, and the auto-parameter probe could have + /// mattered, so the filter withholds the whole plan without a budget being involved (the same plan every cycle). + private const string CutPreparedPlan = "" + + "#5367 review, A-L4: a plan the filter withheld whole for its OWN sake (it cannot be parsed) is not a + /// budget skip. It fails the same way every cycle, so it is recorded like any fetched plan: the second cycle + /// answers from the cache instead of fetching and rendering the plan on the target again. + [Fact] + public async Task ProcedureStats_QueryPlanXml_APlanWithheldWholeForItsOwnSakeIsCached_SoTheNextCycleDoesNotFetchItAgain() + { + var cache = new PlanDigestCache(); + var plan = CutPreparedPlan; + var row = default(ProcedureStatsCollector.Row) with + { + DatabaseName = "db1", + SchemaName = "dbo", + ObjectName = "p1", + ObjectType = "P", + CachedTime = s_now.AddHours(-4), + SqlHandle = "0x0301", + PlanHandle = "0x0501", + PlanStatementCount = 3, + PlanLastStatementCompile = s_now.AddHours(-3), + PlanGenerationSum = 7, + }; + + var fetches = 0; + var unjudged = new HashSet(); + ProcedureStatsPlanReuse.PlanFetch FetchThrough(SensitiveStatements.Session session) => async (handles, token) => + { + fetches++; + using var table = new DataTable(); + table.Columns.Add("ord", typeof(int)); + table.Columns.Add("query_plan_xml", typeof(string)); + table.Columns.Add("query_plan_xml_bytes", typeof(long)); + table.Rows.Add(0, plan, (long)plan.Length); + await using var reader = table.CreateDataReader(); + return await QueryStatsCollector.ReadPlanFetchAsync(reader, session, token, unjudged); + }; + + SensitiveStatements.Session Ordinary() => new(); + + var rows1 = new List { row }; + var first = await ProcedureStatsPlanReuse.ApplyOnAsync(1, cache, rows1, true, 1, s_now, FetchThrough(Ordinary()), CancellationToken.None, unjudged); + cache.ConfirmPending(first.Pending, s_now); + + Assert.Equal(SensitiveStatements.PlaceholderText, rows1[0].QueryPlanXml); + Assert.Empty(unjudged); + Assert.Single(first.Pending); + + var rows2 = new List { row }; + var second = await ProcedureStatsPlanReuse.ApplyOnAsync(1, cache, rows2, true, 2, s_now, FetchThrough(Ordinary()), CancellationToken.None, unjudged); + + Assert.Equal(1, second.Hit); + Assert.Equal(1, fetches); + Assert.Equal(ProcedureStatsPlanReuse.DigestOf(SensitiveStatements.PlaceholderText), rows2[0].KnownPlanDigest); + } + + [Fact] + public async Task ReadPlanFetchAsync_ReportsOnlyTheBudgetSkippedPlansAsUnjudged() + { + var plan = StatementScrubCanary.CanaryPlan(); + using var table = new DataTable(); + table.Columns.Add("ord", typeof(int)); + table.Columns.Add("query_plan_xml", typeof(string)); + table.Columns.Add("query_plan_xml_bytes", typeof(long)); + table.Rows.Add(0, plan, (long)plan.Length); + using var cutTable = new DataTable(); + cutTable.Columns.Add("ord", typeof(int)); + cutTable.Columns.Add("query_plan_xml", typeof(string)); + cutTable.Columns.Add("query_plan_xml_bytes", typeof(long)); + cutTable.Rows.Add(0, CutPreparedPlan, (long)CutPreparedPlan.Length); + + var spentSet = new HashSet(); + await using (var reader = table.CreateDataReader()) + { + await QueryStatsCollector.ReadPlanFetchAsync(reader, SpentSession(), CancellationToken.None, spentSet); + } + + var ownSakeSet = new HashSet(); + await using (var reader = cutTable.CreateDataReader()) + { + var fetched = await QueryStatsCollector.ReadPlanFetchAsync(reader, new SensitiveStatements.Session(), CancellationToken.None, ownSakeSet); + Assert.Equal(SensitiveStatements.PlaceholderText, fetched[0].PlanXml); + } + + Assert.Equal(new[] { 0 }, spentSet); + Assert.Empty(ownSakeSet); + } + + [Fact] + public void QueryStats_TheDeferredFetch_CachesAWholeMarkerPlanUnlessTheBudgetSkippedIt() + { + var source = System.IO.File.ReadAllText(RepoFile.PathTo("Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs")); + var bodies = StatementColumnCensusTests.BodiesOf(source, "DarlingCollectorRunner", "FetchDeferredQueryStatsPlansAsync"); + + Assert.Single(bodies); + Assert.Contains("unjudged.Contains(ord)", bodies[0], StringComparison.Ordinal); + Assert.DoesNotContain("PlaceholderText", bodies[0], StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/StatementScrubQueryStoreLiveTests.cs b/Darling/Darling.Tests/StatementScrubQueryStoreLiveTests.cs new file mode 100644 index 0000000000..5d34369857 --- /dev/null +++ b/Darling/Darling.Tests/StatementScrubQueryStoreLiveTests.cs @@ -0,0 +1,320 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348, #5320 (lane R3): the two Darling Query Store writers against a REAL store. The text writer and the plan +/// writer judge each fetched value before it is stored, and a value the filter could not judge (its per-batch +/// budget is spent) is dropped from the batch, not stored as the marker: both stores are write-once per id, and an +/// absent row is exactly what the missing-set probe reads as "fetch me again". +/// +[Collection("live-postgres")] +public sealed class StatementScrubQueryStoreLiveTests +{ + private const string ServerName = "darling-statement-scrub-qs-e2e"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + private const string Db = "ScrubDb"; + private const int TestTimeoutSeconds = 30; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + /// A judge that charges a fixed cost to a fake clock for every value it is asked about. + private sealed class FakeJudge(TimeSpan cost) + { + public TimeSpan Now { get; private set; } + + public int Calls { get; private set; } + + public Func Clock => () => Now; + + public SensitiveStatements.Verdict Judge(string text) + { + Calls++; + Now += cost; + return SensitiveStatements.Verdict.Clean; + } + } + + private static SensitiveStatements.Session SessionSpentAfter(FakeJudge fake, double seconds) => + new(fake.Judge, fake.Clock, TimeSpan.FromSeconds(seconds), new SensitiveStatements.TimedOutMemo()); + + private static async Task> StoredTextsAsync(NpgsqlConnection connection, System.Threading.CancellationToken ct) + { + var result = new Dictionary(); + using var command = new NpgsqlCommand( + "SELECT query_id, query_sql_text FROM collect.query_store_text WHERE server_id = $1 AND database_name = $2", connection); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(Db); + await using var reader = await command.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + result[reader.GetInt64(0)] = reader.IsDBNull(1) ? null : reader.GetString(1); + } + + return result; + } + + private static async Task> MapDigestsAsync(NpgsqlConnection connection, System.Threading.CancellationToken ct) + { + var result = new Dictionary(); + using var command = new NpgsqlCommand( + "SELECT plan_id, digest FROM collect.query_store_plan_map WHERE server_id = $1 AND database_name = $2", connection); + command.Parameters.AddWithValue(ServerId); + command.Parameters.AddWithValue(Db); + await using var reader = await command.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + result[reader.GetInt64(0)] = reader.IsDBNull(1) ? null : (byte[])reader.GetValue(1); + } + + return result; + } + + private static async Task DimContentAsync(NpgsqlConnection connection, byte[] digest, System.Threading.CancellationToken ct) + { + using var command = new NpgsqlCommand("SELECT query_plan_gz FROM collect.query_plan_dim WHERE digest = $1", connection); + command.Parameters.AddWithValue(digest); + var value = await command.ExecuteScalarAsync(ct); + return value is byte[] compressed ? PayloadDimensions.DecompressContent(compressed) : null; + } + + [Fact] + public async Task QueryStoreText_StoresTheMarkerForANamedStatement_AndTheTextForAPlainOne() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live statement filter test."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + + var bodySucceeded = false; + try + { + var landed = await QueryStoreTextWriter.WriteAsync( + connection, ServerId, Db, + new[] + { + new FetchedQueryText(1, StatementScrubCanary.CanaryStatement, "0xAA"), + new FetchedQueryText(2, StatementScrubCanary.PlainStatement, "0xBB"), + new FetchedQueryText(3, null, "0xCC"), + }, + DateTime.UtcNow, TestTimeoutSeconds, cancellationToken: ct); + + Assert.Equal(new long[] { 1, 2, 3 }, landed); + var stored = await StoredTextsAsync(connection, ct); + Assert.Equal(SensitiveStatements.PlaceholderText, stored[1]); + Assert.Equal(StatementScrubCanary.PlainStatement, stored[2]); + Assert.Null(stored[3]); + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.DoesNotContain(stored.Values, v => v is not null && v.Contains(needle, StringComparison.Ordinal)); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// + /// r2 M-D: a session that spends its budget after three values leaves ids 4 and up ABSENT (not the marker), the + /// writer does not return them, and the next cycle's fresh session stores them with their real text. + /// + [Fact] + public async Task QueryStoreText_DropsTheRowsTheBudgetLeftUnjudged_AndTheNextCycleStoresThem() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live statement filter test."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + + var bodySucceeded = false; + try + { + var texts = Enumerable.Range(1, 6) + .Select(i => new FetchedQueryText(i, "SELECT " + i + " FROM dbo.t", "0x0" + i)) + .ToArray(); + + var fake = new FakeJudge(TimeSpan.FromSeconds(1)); + var landed = await QueryStoreTextWriter.WriteAsync( + connection, ServerId, Db, texts, DateTime.UtcNow, TestTimeoutSeconds, SessionSpentAfter(fake, 3), ct); + + Assert.Equal(new long[] { 1, 2, 3 }, landed); + var afterFirst = await StoredTextsAsync(connection, ct); + Assert.Equal(new long[] { 1, 2, 3 }, afterFirst.Keys.OrderBy(k => k).ToArray()); + Assert.DoesNotContain(afterFirst.Values, v => v == SensitiveStatements.PlaceholderText); + + /* The next cycle: the probe sees ids 4-6 as missing, the fetch ships them, a fresh session judges them. */ + var second = await QueryStoreTextWriter.WriteAsync( + connection, ServerId, Db, texts.Skip(3).ToArray(), DateTime.UtcNow, TestTimeoutSeconds, cancellationToken: ct); + + Assert.Equal(new long[] { 4, 5, 6 }, second); + var afterSecond = await StoredTextsAsync(connection, ct); + Assert.Equal(6, afterSecond.Count); + Assert.Equal("SELECT 5 FROM dbo.t", afterSecond[5]); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt)); + } + } + + /// + /// The plan map's digest resolves to a dimension row holding the FILTERED plan: the digest is of the filtered XML, + /// the raw plan's digest has no dimension row, the stored plan holds no secret needle and keeps statement 2. + /// + [Fact] + public async Task QueryStorePlan_DigestResolvesToTheFilteredPlan() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live statement filter test."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + var canary = StatementScrubCanary.CanaryPlan(); + var filtered = new SensitiveStatements.Session().Xml(canary)!; + var rawDigest = PayloadDimensions.Digest(canary); + var filteredDigest = PayloadDimensions.Digest(filtered); + await DeleteRowsAsync(connection, ct, rawDigest, filteredDigest); + + var bodySucceeded = false; + try + { + Assert.NotEqual(canary, filtered); + + var landed = await QueryStorePlanWriter.WriteAsync( + connection, ServerId, Db, new[] { new FetchedPlan(1, canary, "0xAAAA") }, DateTime.UtcNow, TestTimeoutSeconds, cancellationToken: ct); + + Assert.Equal(new long[] { 1 }, landed); + var map = await MapDigestsAsync(connection, ct); + Assert.Equal(filteredDigest, map[1]); + Assert.Null(await DimContentAsync(connection, rawDigest, ct)); + + var stored = await DimContentAsync(connection, filteredDigest, ct); + Assert.Equal(filtered, stored); + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.DoesNotContain(needle, stored, StringComparison.Ordinal); + } + + Assert.Contains(SensitiveStatements.PlaceholderText, stored, StringComparison.Ordinal); + foreach (var needle in StatementScrubCanary.KeptNeedles) + { + Assert.Contains(needle, stored, StringComparison.Ordinal); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt, rawDigest, filteredDigest)); + } + } + + /// + /// r2 M-D, plan side: a plan the budget left unjudged gets NO map row (so the probe reads it as missing and + /// refetches it), is not in the returned list, and the next cycle stores it. A plan with NO XML still gets its + /// content-less map row. + /// + [Fact] + public async Task QueryStorePlan_DropsThePlansTheBudgetLeftUnjudged_AndTheNextCycleStoresThem() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live statement filter test."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + var plans = Enumerable.Range(1, 4) + .Select(i => new FetchedPlan(i, "", "0x0" + i)) + .ToArray(); + var digests = plans.Select(p => PayloadDimensions.Digest(p.PlanXml!)).ToArray(); + await DeleteRowsAsync(connection, ct, digests); + + var bodySucceeded = false; + try + { + /* A fake judge that spends the budget after the first judged value: whatever a plan's document judges, the + first plan is the only one fully judged; the plans after it reach a spent budget. */ + var fake = new FakeJudge(TimeSpan.FromSeconds(1)); + var probe = SessionSpentAfter(fake, 1); + probe.TryXml(plans[0].PlanXml, out _); + var perPlan = fake.Calls; + Assert.True(perPlan >= 1, "the fake judge must be asked about at least one value per plan"); + + var spent = SessionSpentAfter(new FakeJudge(TimeSpan.FromSeconds(1)), perPlan * 2 + 0.5); + var landed = await QueryStorePlanWriter.WriteAsync( + connection, ServerId, Db, plans.Append(new FetchedPlan(5, null, "0x05")).ToArray(), + DateTime.UtcNow, TestTimeoutSeconds, spent, ct); + + Assert.Equal(new long[] { 1, 2, 5 }, landed); + var afterFirst = await MapDigestsAsync(connection, ct); + Assert.Equal(new long[] { 1, 2, 5 }, afterFirst.Keys.OrderBy(k => k).ToArray()); + Assert.Null(afterFirst[5]); + + var second = await QueryStorePlanWriter.WriteAsync( + connection, ServerId, Db, plans.Skip(2).ToArray(), DateTime.UtcNow, TestTimeoutSeconds, cancellationToken: ct); + + Assert.Equal(new long[] { 3, 4 }, second); + var afterSecond = await MapDigestsAsync(connection, ct); + Assert.Equal(5, afterSecond.Count); + Assert.Equal(digests[3], afterSecond[4]); + Assert.NotNull(await DimContentAsync(connection, digests[3], ct)); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, cleanupCt) => + await DeleteRowsAsync(cleanup, cleanupCt, digests)); + } + } + + private static async Task DeleteRowsAsync(NpgsqlConnection connection, System.Threading.CancellationToken ct, params byte[][] digests) + { + if (digests.Length > 0) + { + using var dimension = new NpgsqlCommand("DELETE FROM collect.query_plan_dim WHERE digest = ANY($1)", connection); + dimension.Parameters.AddWithValue(NpgsqlTypes.NpgsqlDbType.Array | NpgsqlTypes.NpgsqlDbType.Bytea, digests); + await dimension.ExecuteNonQueryAsync(ct); + } + + var sql = + $"DELETE FROM collect.query_store_plan_map WHERE server_id = {ServerId};" + + $"DELETE FROM collect.query_store_text WHERE server_id = {ServerId};"; + using var cleanup = new NpgsqlCommand(sql, connection); + await cleanup.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/StatementScrubQueryStoreTests.cs b/Darling/Darling.Tests/StatementScrubQueryStoreTests.cs new file mode 100644 index 0000000000..e934c5a45e --- /dev/null +++ b/Darling/Darling.Tests/StatementScrubQueryStoreTests.cs @@ -0,0 +1,181 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Data; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348, #5320 (lane R3): Query Store statement text goes through the statement filter where the collector first +/// materializes it, which is the one line both Lite's live read and Darling's backfill run. These tests plant the +/// canary in the collector's input (a DbDataReader from a DataTable), run the read and then +/// WritePayload through . They are the collection-time census case +/// for query_store.query_text (registered in ). +/// +public sealed partial class StatementCollectionCensusTests +{ + private const string QueryStoreDb = "QsCanaryDb"; + + private sealed class QsNoDeltas : ICollectorDeltaCalculator + { + public long CalculateDelta(int serverId, string key, string metric, long current, DateTime? at = null, int i = 0) => 0; + + public long CalculateDeltaWithInterval(int serverId, string key, string metric, long current, out int seconds, DateTime? at = null, int i = 0) + { + seconds = 60; + return 0; + } + } + + private static CollectorContext QueryStoreContext() => new() + { + ServerId = 42, + ServerName = "test-server", + CollectionTime = new DateTime(2026, 7, 2, 12, 0, 0, DateTimeKind.Utc), + Deltas = new QsNoDeltas(), + Target = new CollectorTargetInfo(), + CurrentDatabaseName = QueryStoreDb, + }; + + /// The 56 columns the Query Store read takes by position: only the types matter, the rest are NULL. + private static DataTable QueryStoreTable(params (long QueryId, string? Text)[] rows) + { + var table = new DataTable(); + for (var i = 0; i < 56; i++) + { + Type type = i switch + { + 2 or 5 or 6 or 7 or 44 or 45 or 48 or 50 or 51 or 52 => typeof(string), + 3 or 4 => typeof(DateTimeOffset), + 46 => typeof(bool), + 49 => typeof(int), + 54 or 55 => typeof(DateTime), + _ => typeof(long), + }; + table.Columns.Add("c" + i.ToString(System.Globalization.CultureInfo.InvariantCulture), type); + } + + foreach (var (queryId, text) in rows) + { + var values = new object?[56]; + for (var i = 0; i < 56; i++) + { + values[i] = DBNull.Value; + } + + values[0] = queryId; + values[1] = queryId; + values[6] = text is null ? DBNull.Value : text; + values[8] = 1L; + table.Rows.Add(values); + } + + return table; + } + + private static async Task> ReadQueryStoreAsync(DataTable table, bool perItem, CollectorContext context) + { + await using var reader = table.CreateDataReader(); + if (perItem) + { + var rows = new List(); + await QueryStoreCollector.Instance.ReadItemAsync(QueryStoreDb, reader, rows, context, CancellationToken.None); + return rows; + } + + return await QueryStoreCollector.Instance.ReadAsync(reader, context, CancellationToken.None); + } + + /// + /// Census case for query_store.query_text, in both read shapes (the per-database read and the per-item read that + /// Azure SQL Database and the backfill use): the written query_text of the canary row is exactly the marker, no + /// secret needle is written, and a plain statement is the SAME instance the reader returned. RED on the base: the + /// written value held the canary statement. + /// + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task QueryStore_QueryText_IsWithheld(bool perItem) + { + using var table = QueryStoreTable( + (1, StatementScrubCanary.CanaryStatement), + (2, StatementScrubCanary.PlainStatement)); + var context = QueryStoreContext(); + + var rows = await ReadQueryStoreAsync(table, perItem, context); + Assert.Equal(2, rows.Count); + + var recording = new StatementScrubRecordingWriter(); + foreach (var row in rows) + { + QueryStoreCollector.Instance.WritePayload(row, recording, context); + } + + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.False(recording.AnyStringContains(needle), "a written value holds " + needle); + } + + Assert.Equal(SensitiveStatements.PlaceholderText, rows[0].QueryText); + Assert.Contains(SensitiveStatements.PlaceholderText, recording.Strings); + Assert.Contains(recording.Strings, s => ReferenceEquals(s, rows[1].QueryText)); + Assert.Equal(StatementScrubCanary.PlainStatement, rows[1].QueryText); + Assert.True(recording.AnyStringContains("canary_plain_ssf")); + } + + /// + /// The byte budget (#1556) counts the RAW text, not the shorter marker, so the cut points do not move because the + /// filter ran: a canary row ships the same byte count it did before the filter existed. + /// + [Fact] + public async Task QueryStore_QueryText_ByteBudgetStillCountsTheRawText() + { + using var table = QueryStoreTable((1, StatementScrubCanary.CanaryStatement)); + var context = QueryStoreContext(); + + var rows = await ReadQueryStoreAsync(table, perItem: false, context); + + Assert.Equal(SensitiveStatements.PlaceholderText, Assert.Single(rows).QueryText); + Assert.Equal(StatementScrubCanary.CanaryStatement.Length * 2L, context.PerItemTextBytesShipped); + } + + /// The read runs under the cycle's session, so a withheld statement shows in the cycle's measurements. + [Fact] + public async Task QueryStore_QueryText_CountsTheWithheldValueInTheCycle() + { + using var table = QueryStoreTable( + (1, StatementScrubCanary.CanaryStatement), + (2, StatementScrubCanary.PlainStatement)); + var context = QueryStoreContext(); + + await ReadQueryStoreAsync(table, perItem: false, context); + + Assert.Equal(1, context.Measurements.Single(m => m.Label == CollectorContext.StatementScrubNamedMeasurement).Value); + } + + /// The collector's own queries are excluded BEFORE the filter, so the marker never hides the self-filter's text. + [Fact] + public async Task QueryStore_QueryText_SelfRowsAreStillSkippedBeforeTheFilter() + { + using var table = QueryStoreTable( + (1, QueryStoreCollector.SelfQueryMarker + " SELECT 1"), + (2, StatementScrubCanary.PlainStatement)); + + var rows = await ReadQueryStoreAsync(table, perItem: false, QueryStoreContext()); + + Assert.Equal(2, Assert.Single(rows).QueryId); + } +} diff --git a/Darling/Darling.Tests/StatementScrubRecordingWriter.cs b/Darling/Darling.Tests/StatementScrubRecordingWriter.cs new file mode 100644 index 0000000000..d45d1e8f6e --- /dev/null +++ b/Darling/Darling.Tests/StatementScrubRecordingWriter.cs @@ -0,0 +1,69 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using PerformanceMonitor.Collectors; + +namespace Darling.Tests; + +/// +/// #4348: an that remembers every value a definition's WritePayload wrote, in +/// order, so the collection census can plant a canary in the collector's input, run the read and the write, and +/// assert on what was written: no written value holds a secret needle, and a plain statement is the SAME instance the +/// reader returned. A payload written through is recorded as its +/// content (the interface's default), with the digest the caller offered kept beside it. +/// +internal sealed class StatementScrubRecordingWriter : ICollectorRowWriter +{ + private readonly List _values = new(); + private readonly List<(string? Content, string? KnownDigest)> _payloads = new(); + + /// Every value written, in order. A null is a null column. + public IReadOnlyList Values => _values; + + /// Every PayloadOrDigest call: the content and the digest the caller already knew. + public IReadOnlyList<(string? Content, string? KnownDigest)> Payloads => _payloads; + + /// The string values written, in order. + public IEnumerable Strings => _values.OfType(); + + /// True when any written string contains (ordinal). + public bool AnyStringContains(string needle) => + Strings.Any(s => s.Contains(needle, StringComparison.Ordinal)); + + private ICollectorRowWriter Add(object? v) + { + _values.Add(v); + return this; + } + + public ICollectorRowWriter Value(string? value) => Add(value); + public ICollectorRowWriter Value(long value) => Add(value); + public ICollectorRowWriter Value(long? value) => Add(value); + public ICollectorRowWriter Value(int value) => Add(value); + public ICollectorRowWriter Value(int? value) => Add(value); + public ICollectorRowWriter Value(short value) => Add(value); + public ICollectorRowWriter Value(short? value) => Add(value); + public ICollectorRowWriter Value(double value) => Add(value); + public ICollectorRowWriter Value(double? value) => Add(value); + public ICollectorRowWriter Value(decimal value) => Add(value); + public ICollectorRowWriter Value(decimal? value) => Add(value); + public ICollectorRowWriter Value(bool value) => Add(value); + public ICollectorRowWriter Value(bool? value) => Add(value); + public ICollectorRowWriter Value(DateTime value) => Add(value); + public ICollectorRowWriter Value(DateTime? value) => Add(value); + public ICollectorRowWriter NullValue() => Add(null); + + public ICollectorRowWriter PayloadOrDigest(string? content, string? knownDigest) + { + _payloads.Add((content, knownDigest)); + return Add(content); + } +} diff --git a/Darling/Darling.Tests/StatementScrubSnapshotTests.cs b/Darling/Darling.Tests/StatementScrubSnapshotTests.cs new file mode 100644 index 0000000000..32972915e0 --- /dev/null +++ b/Darling/Darling.Tests/StatementScrubSnapshotTests.cs @@ -0,0 +1,457 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Data; +using System.Diagnostics; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348, rows 8, 19 and 20 of the collection census: the live snapshot of running requests +/// (query_snapshots: query_text, query_plan, live_query_plan), plan correction (plan_correction: +/// query_text, implementation_script) and the oversized-plan sweep (oversized_plan_backlog: plan_xml). Each +/// test plants the canary in the collector's input, runs the read and then WritePayload into a +/// , and asserts on what was written: no secret needle, the statement +/// column exactly the marker, the plan parseable with the marker in statement 1 and the kept needles in statement 2, +/// and a plain statement the SAME instance the reader returned. +/// +public sealed partial class StatementCollectionCensusTests +{ + private const string Marker = SensitiveStatements.PlaceholderText; + + private static CollectorContext SnapshotContext() => new() + { + ServerId = 7, + ServerName = "ssf-test", + CollectionTime = new DateTime(2026, 10, 6, 12, 0, 0, DateTimeKind.Utc), + Deltas = new CollectorDeltaCalculator(), + CurrentDatabaseName = "ssf_db", + }; + + /// A reader shaped like query_snapshots' 35 columns: a session id, then the columns the read + /// guards with IsDBNull. Only the statement text and the two plans are set. + private static DataTableReader SnapshotReader(params (string? Text, string? Plan, string? Live)[] rows) + { + var table = new DataTable("query_snapshots"); + for (var i = 0; i < 35; i++) + { + table.Columns.Add("c" + i, typeof(object)); + } + + var id = 1; + foreach (var (text, plan, live) in rows) + { + var values = new object?[35]; + for (var i = 0; i < values.Length; i++) + { + values[i] = DBNull.Value; + } + + values[0] = id++; + values[3] = (object?)text ?? DBNull.Value; + values[4] = (object?)plan ?? DBNull.Value; + values[5] = (object?)live ?? DBNull.Value; + table.Rows.Add(values); + } + + return table.CreateDataReader(); + } + + private static async Task<(List Rows, StatementScrubRecordingWriter Writer)> ReadAndWriteSnapshotsAsync( + DataTableReader reader, Func? session = null) + { + var context = SnapshotContext(); + context.ScrubSessionFactory = session; + var rows = await QuerySnapshotsCollector.Instance.ReadAsync(reader, context, CancellationToken.None); + var writer = new StatementScrubRecordingWriter(); + foreach (var row in rows) + { + QuerySnapshotsCollector.Instance.WritePayload(row, writer, context); + } + + return (rows, writer); + } + + private static void AssertNoSecretNeedle(StatementScrubRecordingWriter writer) + { + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.False(writer.AnyStringContains(needle), "a written value still holds " + needle); + } + } + + private static void AssertFilteredCanaryPlan(string? written) + { + Assert.NotNull(written); + var parsed = System.Xml.Linq.XDocument.Parse(written!); + Assert.NotNull(parsed.Root); + Assert.Contains(Marker, written!, StringComparison.Ordinal); + foreach (var needle in StatementScrubCanary.SecretNeedles) + { + Assert.DoesNotContain(needle, written!, StringComparison.Ordinal); + } + + foreach (var kept in StatementScrubCanary.KeptNeedles) + { + Assert.Contains(kept, written!, StringComparison.Ordinal); + } + } + + [Fact] + public async Task QuerySnapshots_QueryText_IsWithheld() + { + var plain = new string(StatementScrubCanary.PlainStatement.AsSpan()); + var (rows, writer) = await ReadAndWriteSnapshotsAsync( + SnapshotReader((StatementScrubCanary.CanaryStatement, null, null), (plain, null, null))); + + Assert.Equal(Marker, rows[0].QueryText); + Assert.Same(plain, rows[1].QueryText); + AssertNoSecretNeedle(writer); + Assert.Contains(Marker, writer.Strings); + Assert.Contains(plain, writer.Strings); + + /* The same input judged twice reads the same (the family has no digest, so identity is the text itself). */ + var (again, _) = await ReadAndWriteSnapshotsAsync( + SnapshotReader((StatementScrubCanary.CanaryStatement, null, null), (plain, null, null))); + Assert.Equal(rows[0].QueryText, again[0].QueryText); + Assert.Equal(rows[1].QueryText, again[1].QueryText); + } + + [Fact] + public async Task QuerySnapshots_QueryPlan_IsFiltered() + { + var (rows, writer) = await ReadAndWriteSnapshotsAsync( + SnapshotReader((null, StatementScrubCanary.CanaryPlan(), null))); + + AssertNoSecretNeedle(writer); + AssertFilteredCanaryPlan(rows[0].QueryPlan); + Assert.Null(rows[0].LiveQueryPlan); + } + + [Fact] + public async Task QuerySnapshots_LiveQueryPlan_IsFiltered() + { + var (rows, writer) = await ReadAndWriteSnapshotsAsync( + SnapshotReader((null, null, StatementScrubCanary.CanaryPlan()))); + + AssertNoSecretNeedle(writer); + AssertFilteredCanaryPlan(rows[0].LiveQueryPlan); + Assert.Null(rows[0].QueryPlan); + } + + [Fact] + public async Task QuerySnapshots_APlainPlan_IsTheSameInstanceTheReaderReturned() + { + var plain = "" + + ""; + var (rows, _) = await ReadAndWriteSnapshotsAsync(SnapshotReader((null, plain, plain))); + + Assert.Same(plain, rows[0].QueryPlan); + Assert.Same(plain, rows[0].LiveQueryPlan); + } + + /// A reader shaped like plan correction's 38 columns (ordinals 0-37); only the statement text (14) and + /// the implementation script (37) are set. + private static DataTableReader PlanCorrectionReader(string? queryText, string? script) + { + var table = new DataTable("plan_correction"); + for (var i = 0; i < 38; i++) + { + table.Columns.Add("c" + i, typeof(object)); + } + + var values = new object?[38]; + for (var i = 0; i < values.Length; i++) + { + values[i] = DBNull.Value; + } + + values[14] = (object?)queryText ?? DBNull.Value; + values[37] = (object?)script ?? DBNull.Value; + table.Rows.Add(values); + return table.CreateDataReader(); + } + + [Fact] + public async Task PlanCorrection_QueryText_IsWithheld() + { + var context = SnapshotContext(); + var plain = new string(StatementScrubCanary.PlainStatement.AsSpan()); + + /* Both reads: the Azure per-database read (ReadAsync) and the per-database item read (ReadItemAsync). */ + var azure = await PlanCorrectionCollector.Instance.ReadAsync( + PlanCorrectionReader(StatementScrubCanary.CanaryStatement, null), context, CancellationToken.None); + var item = new List(); + await PlanCorrectionCollector.Instance.ReadItemAsync( + "ssf_db", PlanCorrectionReader(StatementScrubCanary.CanaryStatement, null), item, context, CancellationToken.None); + var kept = await PlanCorrectionCollector.Instance.ReadAsync( + PlanCorrectionReader(plain, null), context, CancellationToken.None); + + var writer = new StatementScrubRecordingWriter(); + foreach (var row in azure.Concat(item).Concat(kept)) + { + PlanCorrectionCollector.Instance.WritePayload(row, writer, context); + } + + Assert.Equal(Marker, azure[0].QueryText); + Assert.Equal(Marker, item[0].QueryText); + Assert.Same(plain, kept[0].QueryText); + AssertNoSecretNeedle(writer); + } + + [Fact] + public async Task PlanCorrection_ImplementationScript_IsWithheld() + { + var context = SnapshotContext(); + var item = new List(); + await PlanCorrectionCollector.Instance.ReadItemAsync( + "ssf_db", PlanCorrectionReader(null, StatementScrubCanary.CanaryStatement), item, context, CancellationToken.None); + var plain = new string(StatementScrubCanary.PlainStatement.AsSpan()); + var kept = await PlanCorrectionCollector.Instance.ReadAsync( + PlanCorrectionReader(null, plain), context, CancellationToken.None); + + var writer = new StatementScrubRecordingWriter(); + foreach (var row in item.Concat(kept)) + { + PlanCorrectionCollector.Instance.WritePayload(row, writer, context); + } + + Assert.Equal(Marker, item[0].ImplementationScript); + Assert.Same(plain, kept[0].ImplementationScript); + AssertNoSecretNeedle(writer); + } + + /* ---- oversized_plan_backlog.plan_xml: the sweep's judging step ---- */ + + [Fact] + public void OversizedPlanSweep_PlanXml_IsFilteredBeforeItIsStored() + { + var (verdict, stored, error) = OversizedPlanBacklogSweep.JudgeFetchedPlan( + new SensitiveStatements.Session(), StatementScrubCanary.CanaryPlan(), 0); + + Assert.Equal(OversizedPlanBacklogSweep.PlanFetchVerdict.Captured, verdict); + Assert.Null(error); + AssertFilteredCanaryPlan(stored); + } + + [Fact] + public void OversizedPlanSweep_PlanXml_APlainPlanIsTheSameInstance() + { + var plain = "" + + ""; + + var (verdict, stored, _) = OversizedPlanBacklogSweep.JudgeFetchedPlan(new SensitiveStatements.Session(), plain, 0); + + Assert.Equal(OversizedPlanBacklogSweep.PlanFetchVerdict.Captured, verdict); + Assert.Same(plain, stored); + } + + [Fact] + public void OversizedPlanSweep_PlanXml_APlanTheBudgetCannotCover_StaysClaimableInsteadOfStoringTheMarker() + { + /* A session whose budget is already spent: the fetch is Failed (the row stays claimable and the next pass, + with a fresh budget, fetches it again), and nothing is handed to the store. */ + var spent = new SensitiveStatements.Session(null, null, TimeSpan.Zero, null); + + var (verdict, stored, error) = OversizedPlanBacklogSweep.JudgeFetchedPlan(spent, StatementScrubCanary.CanaryPlan(), 0); + + Assert.Equal(OversizedPlanBacklogSweep.PlanFetchVerdict.Failed, verdict); + Assert.Null(stored); + Assert.False(string.IsNullOrEmpty(error)); + Assert.Equal(1, spent.Unjudged); + } + + /* A session whose budget has room when the plan starts and runs out inside the plan: each judged value costs a + second against a 1.5-second budget, so the plan itself is what spends it. */ + private static SensitiveStatements.Session SessionThePlanOverruns() + { + var now = TimeSpan.Zero; + return new SensitiveStatements.Session( + _ => + { + now += TimeSpan.FromSeconds(1); + return SensitiveStatements.Verdict.Clean; + }, + () => now, + TimeSpan.FromMilliseconds(1500), + new SensitiveStatements.TimedOutMemo()); + } + + [Fact] + public void OversizedPlanSweep_APlanThatOverrunsTheBudgetOnItsFirstAttempt_StaysClaimable() + { + var session = SessionThePlanOverruns(); + Assert.False(session.Spent); + + var (verdict, stored, error) = OversizedPlanBacklogSweep.JudgeFetchedPlan(session, StatementScrubCanary.CanaryPlan(), 0); + + Assert.Equal(OversizedPlanBacklogSweep.PlanFetchVerdict.JudgeTimedOut, verdict); + Assert.Null(stored); + Assert.False(string.IsNullOrEmpty(error)); + Assert.True(session.Spent); + } + + private const string OnePlainValuePlan = "" + + ""; + + [Fact] + public void OversizedPlanSweep_APlanThatOverrunsAPartlySpentBudget_StaysClaimable_EvenOnItsLastAttempt() + { + /* #5367 review, A-M1: an earlier plan in the batch used part of the budget (it is not spent), and this plan + overruns what is left. Alone it might judge in a second, so it says nothing about the plan: it must not be + retired as the marker, however many attempts the row has. */ + var session = SessionThePlanOverruns(); + Assert.True(OversizedPlanBacklogSweep.JudgeFetchedPlan(session, OnePlainValuePlan, 0).Verdict + == OversizedPlanBacklogSweep.PlanFetchVerdict.Captured); + Assert.False(session.Spent); + Assert.True(session.Values > 0); + + var (verdict, stored, error) = OversizedPlanBacklogSweep.JudgeFetchedPlan( + session, StatementScrubCanary.CanaryPlan(), OversizedPlanBacklogSweep.MaxJudgeAttempts - 1); + + Assert.True(session.Spent, "the plan must have overrun the budget the earlier plan left"); + Assert.Equal(OversizedPlanBacklogSweep.PlanFetchVerdict.Failed, verdict); + Assert.Null(stored); + Assert.False(string.IsNullOrEmpty(error)); + } + + [Fact] + public void OversizedPlanSweep_OnlyAJudgeTimeoutAddsAnAttempt() + { + /* #5367 review, A-M1: attempt_count is the retirement limit's count of judge timeouts, so a connect failure, + a budget skip and an expiry stamp the attempt time and leave the count alone. */ + Assert.Equal(OversizedPlanBacklog.RecordAttemptSql, OversizedPlanBacklogSweep.OutcomeSql(OversizedPlanBacklogSweep.PlanFetchVerdict.JudgeTimedOut)); + Assert.Equal(OversizedPlanBacklog.RecordFailureSql, OversizedPlanBacklogSweep.OutcomeSql(OversizedPlanBacklogSweep.PlanFetchVerdict.Failed)); + Assert.Equal(OversizedPlanBacklog.RecordExpirySql, OversizedPlanBacklogSweep.OutcomeSql(OversizedPlanBacklogSweep.PlanFetchVerdict.Expired)); + Assert.Equal(OversizedPlanBacklog.RecordCaptureSql, OversizedPlanBacklogSweep.OutcomeSql(OversizedPlanBacklogSweep.PlanFetchVerdict.Captured)); + + Assert.Contains("attempt_count = attempt_count + 1", OversizedPlanBacklog.RecordAttemptSql, StringComparison.Ordinal); + Assert.DoesNotContain("attempt_count", OversizedPlanBacklog.RecordFailureSql, StringComparison.Ordinal); + Assert.DoesNotContain("attempt_count", OversizedPlanBacklog.RecordExpirySql, StringComparison.Ordinal); + Assert.Contains("last_attempt_at = $7", OversizedPlanBacklog.RecordFailureSql, StringComparison.Ordinal); + } + + [Fact] + public void OversizedPlanSweep_APlanThatOverrunsTheBudgetAgain_IsStoredOnceAsTheWholePlanMarker() + { + var attempts = OversizedPlanBacklogSweep.MaxJudgeAttempts - 1; + Assert.True(attempts >= 1, "the first judging failure must always leave the row claimable"); + + var (verdict, stored, error) = OversizedPlanBacklogSweep.JudgeFetchedPlan( + SessionThePlanOverruns(), StatementScrubCanary.CanaryPlan(), attempts); + + /* Captured is the verdict that stamps captured_at, which the claim excludes: the row is resolved. */ + Assert.Equal(OversizedPlanBacklogSweep.PlanFetchVerdict.Captured, verdict); + Assert.Equal(SensitiveStatements.PlaceholderText, stored); + Assert.Null(error); + } + + [Fact] + public void OversizedPlanSweep_ASessionAnEarlierPlanSpent_NeverRetiresARow_HoweverManyAttemptsItHas() + { + /* The budget was gone before this plan was reached: that says nothing about the plan. */ + var spent = new SensitiveStatements.Session(null, null, TimeSpan.Zero, null); + + var (verdict, stored, _) = OversizedPlanBacklogSweep.JudgeFetchedPlan(spent, StatementScrubCanary.CanaryPlan(), 500); + + Assert.Equal(OversizedPlanBacklogSweep.PlanFetchVerdict.Failed, verdict); + Assert.Null(stored); + } + + [Fact] + public void OversizedPlanSweep_PassesItsBatchSessionIn_AndMakesNoSessionOfItsOwnInTheJudgingStep() + { + var source = System.IO.File.ReadAllText(RepoFile.PathTo("Darling/PerformanceMonitor.Darling.Service/OversizedPlanBacklogSweep.cs")); + var bodies = StatementColumnCensusTests.BodiesOf(source, "OversizedPlanBacklogSweep", "JudgeFetchedPlan"); + + Assert.Single(bodies); + Assert.DoesNotContain("new SensitiveStatements.Session", bodies[0], StringComparison.Ordinal); + Assert.DoesNotContain("??=", bodies[0], StringComparison.Ordinal); + } + + /* ---- the worst case, for the part file: 200 snapshot rows with 47 KB plans plus one 27 MB plan ---- */ + + /// The worst case: 200 distinct snapshot rows with 47 KB plans plus one 27 MB plan. + private static DataTableReader WorstCaseReader() + { + var rowPlan = SizedPlan(47 * 1024); + var rows = new List<(string? Text, string? Plan, string? Live)>(); + for (var i = 0; i < 200; i++) + { + /* Distinct values so the verdict memo cannot answer for the batch. */ + rows.Add(($"SELECT {i} FROM dbo.t{i}", rowPlan.Replace("ssf_pad", "ssf_pad" + i, StringComparison.Ordinal), null)); + } + + rows.Add((null, SizedPlan(27 * 1024 * 1024), null)); + return SnapshotReader(rows.ToArray()); + } + + [Fact] + public async Task QuerySnapshots_WorstCase_TwoHundredPlansAndAHugeOne_FinishesInsideTheSessionBudget() + { + /* #5459: this asserted `took < 20 s` under the session's real 15 s wall-clock budget. The budget is charged wall + time, so a loaded runner (the Guard stage runs a thousand tests at once) could spend it, withhold plans whole + and stretch the read past the bound, though nothing was wrong. The session is pinned to a budget of an hour + that cannot run out, so the answer no longer depends on the machine: every one of the 201 plans is judged + and none is withheld. The proof that a spent budget withholds and stays bounded is + QuerySnapshots_WorstCase_ASpentSessionWithholdsEveryPlanWholeAndNamesNothing. The wall bound is kept only as + a hang backstop: the read takes about two seconds alone, so two minutes means something is wrong. */ + var clock = Stopwatch.StartNew(); + var (read, writer) = await ReadAndWriteSnapshotsAsync( + WorstCaseReader(), () => new SensitiveStatements.Session(null, null, TimeSpan.FromHours(1), null)); + clock.Stop(); + + TestContext.Current.SendDiagnosticMessage( + "snapshot worst case: " + clock.ElapsedMilliseconds + " ms, " + read.Count + " rows, " + + read.Count(r => r.QueryPlan == Marker) + " plans withheld whole"); + + Assert.Equal(201, read.Count); + Assert.Equal(0, read.Count(r => r.QueryPlan == Marker)); + AssertNoSecretNeedle(writer); + Assert.True(clock.Elapsed < TimeSpan.FromMinutes(2), "the read took " + clock.Elapsed); + } + + [Fact] + public async Task QuerySnapshots_WorstCase_ASpentSessionWithholdsEveryPlanWholeAndNamesNothing() + { + /* #5459: the other half of the pinned budget. A session whose time is already used up (its limit is zero) judges + nothing: all 201 rows still come back, every plan is the marker, and nothing is written but the marker. */ + var (read, writer) = await ReadAndWriteSnapshotsAsync( + WorstCaseReader(), () => new SensitiveStatements.Session(null, null, TimeSpan.Zero, null)); + + Assert.Equal(201, read.Count); + Assert.Equal(201, read.Count(r => r.QueryPlan == Marker)); + AssertNoSecretNeedle(writer); + } + + private static string SizedPlan(int approximateChars) + { + const string head = ""; + const string tail = ""; + var sb = new System.Text.StringBuilder(approximateChars + 256); + sb.Append(head); + for (var n = 0; sb.Length < approximateChars; n++) + { + /* Each statement is distinct, so the verdict memo cannot answer for the document. */ + sb.Append(""); + } + + sb.Append(tail); + return sb.ToString(); + } +} diff --git a/Darling/Darling.Tests/StatementScrubViewerPlanWithheldTests.cs b/Darling/Darling.Tests/StatementScrubViewerPlanWithheldTests.cs new file mode 100644 index 0000000000..955ff4d97d --- /dev/null +++ b/Darling/Darling.Tests/StatementScrubViewerPlanWithheldTests.cs @@ -0,0 +1,153 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Windows; +using System.Windows.Controls; +using PerformanceMonitor.Common; +using PerformanceMonitor.Ui; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5320 (the plan viewers half of #4348), the Darling viewer's side. A stored or live plan the statement filter +/// withheld whole is the marker, not a plan. The viewer shows it as withheld +/// () instead of the parser's "The plan XML could not be read", +/// and no site that saves a plan to a .sqlplan file writes it. The control and the guard are the shared +/// PerformanceMonitor.Ui ones Lite uses too; Lite.Tests holds the same pins for Lite's own save sites. +/// +public sealed class StatementScrubViewerPlanWithheldTests +{ + private const string Marker = SensitiveStatements.PlaceholderText; + + [Fact] + public void APlanWithheldWhole_ShowsTheWithheldSentenceInTheViewer_NotAParseError() + { + StaTestThread.Run(() => + { + var control = new PlanViewerControl(); + try + { + control.LoadPlan(Marker, "Stored Plan").GetAwaiter().GetResult(); + + var title = (TextBlock)control.FindName("EmptyStateTitle"); + Assert.Equal("This plan was withheld by the statement filter (#4348).", title.Text); + Assert.DoesNotContain("could not be read", title.Text, StringComparison.Ordinal); + Assert.Equal(Visibility.Visible, ((UIElement)control.FindName("EmptyState")).Visibility); + Assert.Equal(Visibility.Collapsed, ((UIElement)control.FindName("EmptyStateDetail")).Visibility); + Assert.Equal(Visibility.Collapsed, ((UIElement)control.FindName("PlanScrollViewer")).Visibility); + } + finally + { + control.Cleanup(); + } + }); + } + + [Fact] + public void TheWithheldGuard_RefusesOnlyTheMarker_AndARealPlanIsUnchanged() + { + Assert.Equal(SensitiveStatements.WithheldPlanSentence, WithheldPlanGuard.WithheldSentence(Marker)); + Assert.Equal(SensitiveStatements.WithheldPlanSentence, WithheldPlanGuard.WithheldSentence(" " + Marker + "\r\n")); + + var plain = "" + + ""; + + Assert.Null(WithheldPlanGuard.WithheldSentence(plain)); + Assert.Null(WithheldPlanGuard.WithheldSentence(StatementScrubCanary.CanaryPlan())); + Assert.Null(WithheldPlanGuard.WithheldSentence(null)); + Assert.Null(WithheldPlanGuard.WithheldSentence(string.Empty)); + + /* A real plan is not refused, and nothing is shown for it (a refusal would put up a message box). */ + Assert.False(WithheldPlanGuard.RefuseSave(plain)); + Assert.False(WithheldPlanGuard.RefuseSave(null)); + } + + /// + /// Every place the Darling viewer (and the shared plan control it hosts) writes a plan to a .sqlplan file + /// asks WithheldPlanGuard.RefuseSave first. A new save site fails here until it does. + /// + [Fact] + public void EveryPlanSaveSite_RefusesTheWithheldMarkerFirst() + { + var problems = new List(); + var sites = 0; + + foreach (var root in new[] { "Darling/PerformanceMonitor.Darling.Viewer", "PerformanceMonitor.Ui" }) + { + foreach (var file in Directory.EnumerateFiles(RepoFile.PathTo(root), "*.cs", SearchOption.AllDirectories)) + { + var relative = file.Replace('\\', '/'); + if (relative.Contains("/obj/", StringComparison.Ordinal) || relative.Contains("/bin/", StringComparison.Ordinal)) + { + continue; + } + + var text = File.ReadAllText(file); + if (!text.Contains("new SaveFileDialog", StringComparison.Ordinal) || !text.Contains(".sqlplan\"", StringComparison.Ordinal)) + { + continue; + } + + sites++; + if (!text.Contains("WithheldPlanGuard.RefuseSave(", StringComparison.Ordinal)) + { + problems.Add(relative + " writes a .sqlplan without WithheldPlanGuard.RefuseSave"); + } + } + } + + /* The three history windows, the shared SavePlanFile helper and the plan control's own Save button. */ + Assert.True(sites >= 5, "expected at least 5 plan save sites, found " + sites); + Assert.Empty(problems); + } + + /// + /// #5367: a withheld deadlock graph or blocked process report is not "a plan", so its save refusal names what is + /// being saved; the plan viewers keep their existing sentence word for word. + /// + [Fact] + public void TheWithheldSentence_NamesWhatIsSaved_AndThePlanSentenceIsUnchanged() + { + Assert.Equal("This plan was withheld by the statement filter (#4348).", WithheldPlanGuard.WithheldSentenceFor("plan")); + Assert.Equal(SensitiveStatements.WithheldPlanSentence, WithheldPlanGuard.WithheldSentenceFor("plan")); + Assert.Equal("This deadlock graph was withheld by the statement filter (#4348).", WithheldPlanGuard.WithheldSentenceFor("deadlock graph")); + Assert.Equal("This blocked process report was withheld by the statement filter (#4348).", WithheldPlanGuard.WithheldSentenceFor("blocked process report")); + } + + /// + /// #5320: a deadlock graph or blocked process report the filter withheld whole is the marker, not XML. The viewer's + /// two XML downloads go through the shared FileSaveHelper.SaveXmlToFile, which asks the guard before it + /// opens the save dialog, so the marker is never written to an .xml file and the user is told why. + /// + [Fact] + public void SaveXmlToFile_RefusesAWithheldReportBeforeTheSaveDialog_AndTheViewersDownloadsUseIt() + { + var helper = File.ReadAllText(RepoFile.PathTo("PerformanceMonitor.Ui/FileSaveHelper.cs")); + var start = helper.IndexOf("public static void SaveXmlToFile(", StringComparison.Ordinal); + Assert.True(start >= 0, "SaveXmlToFile not found"); + var dialog = helper.IndexOf("new SaveFileDialog", start, StringComparison.Ordinal); + var guard = helper.IndexOf("WithheldPlanGuard.RefuseSave(xml, withheldSubject)", start, StringComparison.Ordinal); + Assert.True(guard >= 0 && dialog >= 0 && guard < dialog, "SaveXmlToFile must call WithheldPlanGuard.RefuseSave(xml, withheldSubject) before it opens the dialog"); + + var export = File.ReadAllText(RepoFile.PathTo("Darling/PerformanceMonitor.Darling.Viewer/ViewerServerTab.CopyExport.cs")); + Assert.Contains("using static PerformanceMonitor.Ui.FileSaveHelper;", export, StringComparison.Ordinal); + Assert.Contains("SaveXmlToFile(row.DeadlockGraphXml", export, StringComparison.Ordinal); + Assert.Contains("SaveXmlToFile(row.BlockedProcessReportXml", export, StringComparison.Ordinal); + + /* #5367: the sentence names what is saved. A deadlock graph is not "a plan"; each download passes its own subject. */ + Assert.Contains("SaveXmlToFile(row.DeadlockGraphXml, $\"deadlock_{row.DeadlockTime:yyyyMMdd_HHmmss}.xml\", \"deadlock XML\", \"deadlock graph\")", export, StringComparison.Ordinal); + Assert.Contains("\"blocked process XML\", \"blocked process report\")", export, StringComparison.Ordinal); + Assert.DoesNotContain("File.WriteAllText", export, StringComparison.Ordinal); + } + +} diff --git a/Darling/Darling.Tests/StatementScrubViewerSavePathsTests.cs b/Darling/Darling.Tests/StatementScrubViewerSavePathsTests.cs new file mode 100644 index 0000000000..9c230fe2cb --- /dev/null +++ b/Darling/Darling.Tests/StatementScrubViewerSavePathsTests.cs @@ -0,0 +1,108 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Runtime.CompilerServices; +using System.Text.RegularExpressions; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348, the Darling viewer's side of "what leaves the app". The viewer never opens a connection to a monitored +/// server: it reads rows the collectors already judged, and asks the service for anything live (the Active Queries +/// snapshot, a plan by sql_handle, an actual-plan re-run), whose handlers judge the text and the plans before they +/// answer. So a viewer save or copy of statement text or a plan carries what the screen shows. Pinned here, so a change +/// to that is noticed: no viewer file connects to SQL Server itself, every viewer site that writes a .sqlplan asks the +/// withheld guard first (Lite.Tests holds the same scan over Lite and the shared project), and the service handlers a live +/// answer comes from still judge it. +/// +public sealed class StatementScrubViewerSavePathsTests +{ + private static IEnumerable ViewerSources() => + Directory.EnumerateFiles(Path.Combine(RepoRoot(), "Darling", "PerformanceMonitor.Darling.Viewer"), "*.cs", SearchOption.AllDirectories) + .Where(f => + { + var relative = Path.GetRelativePath(RepoRoot(), f).Replace('\\', '/'); + return !relative.Contains("/obj/", StringComparison.Ordinal) && !relative.Contains("/bin/", StringComparison.Ordinal); + }); + + [Fact] + public void NoViewerFile_OpensItsOwnConnectionToASqlServer() + { + var offenders = new List(); + foreach (var file in ViewerSources()) + { + var code = CSharpSourceWalker.StripCommentsAndStrings(File.ReadAllText(file)); + if (Regex.IsMatch(code, @"new\s+SqlConnection\s*\(")) + { + offenders.Add(Path.GetFileName(file)); + } + } + + Assert.Empty(offenders); + } + + [Fact] + public void EveryViewerPlanSaveSite_RefusesTheWithheldMarkerFirst() + { + var problems = new List(); + var sites = 0; + foreach (var file in ViewerSources()) + { + var text = File.ReadAllText(file); + if (!text.Contains("new SaveFileDialog", StringComparison.Ordinal) || !text.Contains(".sqlplan\"", StringComparison.Ordinal)) + { + continue; + } + + sites++; + if (!text.Contains("WithheldPlanGuard.RefuseSave(", StringComparison.Ordinal)) + { + problems.Add(Path.GetFileName(file) + " writes a .sqlplan without WithheldPlanGuard.RefuseSave"); + } + } + + /* The three history windows. The tab's own plan saves go through the shared FileSaveHelper, which Lite.Tests scans. */ + Assert.True(sites >= 3, "expected at least 3 viewer plan save sites, found " + sites); + Assert.Empty(problems); + } + + [Fact] + public void TheServiceHandlers_ThatAnswerALiveRead_StillJudgeIt() + { + var collector = File.ReadAllText(Path.Combine(RepoRoot(), "PerformanceMonitor.Collectors", "QuerySnapshotsCollector.cs")); + var start = collector.IndexOf("public override async ValueTask> ReadAsync", StringComparison.Ordinal); + Assert.True(start > 0); + var body = collector[start..Math.Min(collector.Length, start + 3500)]; + Assert.Contains("BeginStatementScrub()", body, StringComparison.Ordinal); + Assert.Contains("scrub.Text(reader.GetString(3))", body, StringComparison.Ordinal); + Assert.Equal(2, Regex.Matches(body, @"scrub\.Xml\(").Count); + + var worker = File.ReadAllText(Path.Combine(RepoRoot(), "Darling", "PerformanceMonitor.Darling.Service", "DarlingWorker.cs")); + var outcome = worker.IndexOf("internal static CommandOutcome PlanResultOutcome", StringComparison.Ordinal); + Assert.True(outcome > 0); + Assert.Contains("SensitiveStatements.Xml(planXml)", worker[outcome..Math.Min(worker.Length, outcome + 900)], StringComparison.Ordinal); + Assert.Contains("PlanResultOutcome(\"actual plan captured\"", worker, StringComparison.Ordinal); + Assert.Contains("PlanResultOutcome(\"plan fetched\"", worker, StringComparison.Ordinal); + } + + private static string RepoRoot([CallerFilePath] string thisFile = "") + { + var dir = Path.GetDirectoryName(thisFile)!; + while (dir is not null && !File.Exists(Path.Combine(dir, "PerformanceMonitor.sln"))) + { + dir = Path.GetDirectoryName(dir); + } + + return dir ?? throw new InvalidOperationException("repository root not found"); + } +} diff --git a/Darling/Darling.Tests/StatusBarBehaviourTests.cs b/Darling/Darling.Tests/StatusBarBehaviourTests.cs new file mode 100644 index 0000000000..add546a886 --- /dev/null +++ b/Darling/Darling.Tests/StatusBarBehaviourTests.cs @@ -0,0 +1,153 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.ComponentModel; +using System.Diagnostics; +using System.Linq; +using System.Text.Json; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// What the web shell's status bar draws, from the shipped app.js status-bar code run under Node +/// (status-bar-harness.mjs): the seat label from the session probe, the store size from +/// get_store_host read at most once per five minutes, a collection state only when /api/ping is not +/// ok, and a bar that survives any of those reads failing. Node is skipped when it is not installed. +/// +public sealed class StatusBarBehaviourTests +{ + private static bool TryRun(string scenario, out JsonElement result) + { + result = default; + var psi = new ProcessStartInfo("node") { RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false }; + psi.ArgumentList.Add(PathTo("Darling", "Darling.Tests", "status-bar-harness.mjs")); + psi.ArgumentList.Add(PathTo("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "app.js")); + psi.ArgumentList.Add(scenario); + + Process proc; + try + { + proc = Process.Start(psi)!; + } + catch (Win32Exception) + { + Assert.Skip("Node is not installed, so the shipped status-bar code cannot be run."); + return false; + } + + using (proc) + { + var error = proc.StandardError.ReadToEndAsync(); + var output = proc.StandardOutput.ReadToEnd().Trim(); + if (!proc.WaitForExit(20000)) + { + proc.Kill(entireProcessTree: true); + Assert.Fail("the status-bar harness did not finish in 20 s for scenario " + scenario); + } + + Assert.True(proc.ExitCode == 0, "the status-bar harness failed for scenario " + scenario + ": " + error.Result); + using var doc = JsonDocument.Parse(output); + result = doc.RootElement.Clone(); + return true; + } + } + + private static string[] Items(JsonElement r) => + r.GetProperty("items").EnumerateArray().Where(e => e.ValueKind == JsonValueKind.String).Select(e => e.GetString()!).ToArray(); + + [Theory] + [InlineData("seatWrite", "Seat: read-write")] + [InlineData("seatRead", "Seat: read-only")] + [InlineData("seatProbeFailed", "Seat: not connected")] + public void TheSeatLabel_FollowsTheSessionProbe_InTheViewersWording(string scenario, string expected) + { + if (!TryRun(scenario, out var r)) return; + Assert.Contains(expected, Items(r)); + } + + [Theory] + [InlineData("storeSize", "Database: 5.0 GB")] + [InlineData("storeSizeMb", "Database: 300 MB")] + public void TheStoreSize_ComesFromTheStoreHostRead_InTheViewersUnits(string scenario, string expected) + { + if (!TryRun(scenario, out var r)) return; + Assert.Contains(expected, Items(r)); + } + + [Fact] + public void TheStoreSize_IsReadOncePerFiveMinutes_NotOnEveryPoll() + { + if (!TryRun("cached", out var r)) return; + Assert.Equal(1, r.GetProperty("reads").GetInt32()); + Assert.Contains("Database: 5.0 GB", Items(r)); + + if (!TryRun("refetched", out var later)) return; + Assert.Equal(2, later.GetProperty("reads").GetInt32()); + Assert.Contains("Database: 6.0 GB", Items(later)); + } + + [Theory] + [InlineData("ping_starting", "Collection: Starting")] + [InlineData("ping_degraded", "Collection: Degraded")] + [InlineData("ping_stopped", "Collection: Stopped")] + public void ANonOkPingStatus_IsShown(string scenario, string expected) + { + if (!TryRun(scenario, out var r)) return; + Assert.Contains(expected, Items(r)); + } + + [Fact] + public void AnOkPing_AddsNothing() + { + if (!TryRun("ping_ok", out var r)) return; + Assert.DoesNotContain(Items(r), i => i.StartsWith("Collection", StringComparison.Ordinal)); + } + + [Fact] + public void AFailedStoreSizeRead_NeverBreaksTheBar() + { + if (!TryRun("storeFails", out var r)) return; + Assert.True(r.GetProperty("hasBar").GetBoolean()); + Assert.DoesNotContain(Items(r), i => i.StartsWith("Database", StringComparison.Ordinal)); + Assert.Contains("Seat: read-write", Items(r)); + + if (!TryRun("storeFailsAfterGood", out var stale)) return; + Assert.Contains("Database: 5.0 GB (stale)", Items(stale)); + } + + /// + /// W3/W10 of the morning walk: the footer's "Updated" time stayed at the first load on the server, FinOps and + /// Job History pages, with auto-refresh on or after "Reload this page now", because it came only from the + /// shell's fleet read. It now follows the page's own latest load, and a later fleet poll does not pull it back. + /// + [Fact] + public void TheUpdatedTime_FollowsThePagesOwnLatestLoad() + { + if (!TryRun("pageUpdated", out var r)) return; + var stamp = r.GetProperty("stamp").GetString()!; + string[] Strings(string name) => r.GetProperty(name).EnumerateArray().Where(e => e.ValueKind == JsonValueKind.String).Select(e => e.GetString()!).ToArray(); + + Assert.Contains("Updated T", Strings("before")); + Assert.Contains("Updated " + stamp, Strings("afterPage")); + Assert.Contains("Updated " + stamp, Strings("afterNextFleetPoll")); + Assert.DoesNotContain("Updated T", Strings("afterNextFleetPoll")); + } + + [Theory] + [InlineData("pingFails")] + [InlineData("sessionFails")] + public void AFailedPingOrSessionRead_LeavesTheRestOfTheBar(string scenario) + { + if (!TryRun(scenario, out var r)) return; + Assert.Contains("Database: 5.0 GB", Items(r)); + Assert.Contains("1 server", Items(r)); + } +} diff --git a/Darling/Darling.Tests/StorageAndFileIoWebDispatchLiveTests.cs b/Darling/Darling.Tests/StorageAndFileIoWebDispatchLiveTests.cs new file mode 100644 index 0000000000..32ca702aea --- /dev/null +++ b/Darling/Darling.Tests/StorageAndFileIoWebDispatchLiveTests.cs @@ -0,0 +1,147 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.Json; +using System.Threading.Tasks; +using Darling.Tests; +using Microsoft.AspNetCore.Http; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using Xunit; + +namespace PerformanceMonitor.Darling.Tests; + +/// +/// #5244 PR5 (lane W5): the five storage and file I/O reads the web pages drive (get_database_sizes, get_table_index_sizes, +/// get_pvs_stats, get_file_io_stats and get_file_io_trend), called the way the page calls them, through the web read dispatch +/// with two REPEATED database_name keys over a seed of databases A, B and C. Each read must answer with A's and B's rows +/// and none of C's; on the base the dispatch ignored the key (or, for the trend, read only the first), so each answered with all +/// three databases. The same class pins the database_name echo on the answers each tool gained it on (blank is null) and +/// the three public MCP tools' new parameter. +/// +[Collection("live-postgres")] +public sealed class StorageAndFileIoWebDispatchLiveTests +{ + private const string A = StorageDatabaseFilterLiveTests.A; + private const string B = StorageDatabaseFilterLiveTests.B; + private const string C = StorageDatabaseFilterLiveTests.C; + private const string Cleanup = StorageDatabaseFilterLiveTests.Cleanup + "; DELETE FROM file_io_stats WHERE server_id = {0}"; + + private static async Task WebReadAsync(NpgsqlDataSource postgres, string serverName, string read, params string[] databases) + { + var query = new List> { new("server_name", serverName), new("hours_back", "24") }; + query.AddRange(databases.Select(d => new KeyValuePair("database_name", d))); + var context = new DefaultHttpContext(); + context.Request.QueryString = QueryString.Create(query); + return await DarlingWebEndpoints.BuildReadDispatch()[read](context, postgres, null!); + } + + private static string? Echo(string json) => + JsonDocument.Parse(json).RootElement.TryGetProperty("database_name", out var echo) && echo.ValueKind == JsonValueKind.String ? echo.GetString() : null; + + private static bool HasEchoKey(string json) => JsonDocument.Parse(json).RootElement.TryGetProperty("database_name", out _); + + private static string[] DatabasesIn(string json, string array) => + JsonDocument.Parse(json).RootElement.GetProperty(array).EnumerateArray() + .Select(e => e.GetProperty("database_name").GetString()!).Where(n => n != "(other)").Distinct().Order().ToArray(); + + private static Task InsertFileIoAsync(NpgsqlConnection connection, System.Threading.CancellationToken ct, int serverId, string serverName, DateTime t, string db) => + DarlingMcpTestData.ExecAsync(connection, ct, @" +INSERT INTO file_io_stats + (collection_id, collection_time, server_id, server_name, database_name, file_name, file_type, + physical_name, size_mb, delta_reads, delta_writes, delta_read_bytes, delta_write_bytes, + delta_stall_read_ms, delta_stall_write_ms, sample_interval_seconds) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(t), serverId, serverName, db, db + ".mdf", "ROWS", + "D:\\data\\" + db + ".mdf", 1000m, 100L, 50L, 819_200L, 409_600L, 500L, 100L, 60); + + [Fact] + public Task TheFiveReads_TwoRepeatedKeys_ReadOnlyAAndBs() => + DurationTrendDatabaseFilterLiveTests.WithSharedStoreAsync("a5244p5w-web-dispatch", async (connection, postgres, serverId, serverName, now, ct) => + { + foreach (var (db, mb) in new[] { (A, 100m), (B, 200m), (C, 300m) }) + { + await StorageDatabaseFilterLiveTests.InsertTableAsync(connection, ct, serverId, serverName, now.AddHours(-1), db, "T" + db, mb); + await StorageDatabaseFilterLiveTests.InsertFileAsync(connection, ct, serverId, serverName, now.AddHours(-1), db, mb); + await StorageDatabaseFilterLiveTests.InsertPvsAsync(connection, ct, serverId, serverName, now.AddHours(-1), db, mb); + await InsertFileIoAsync(connection, ct, serverId, serverName, now.AddMinutes(-30), db); + await InsertFileIoAsync(connection, ct, serverId, serverName, now.AddMinutes(-5), db); + } + + foreach (var (read, array) in new[] + { + ("get_database_sizes", "databases"), ("get_table_index_sizes", "tables"), ("get_pvs_stats", "databases"), + ("get_file_io_stats", "files"), ("get_file_io_trend", "trend"), + }) + { + var all = await WebReadAsync(postgres, serverName, read); + Assert.Equal(new[] { A, B, C }, DatabasesIn(all, array)); + Assert.True(HasEchoKey(all), $"{read} left the database_name echo off its unfiltered answer"); + Assert.Null(Echo(all)); + + var both = await WebReadAsync(postgres, serverName, read, A, B); + Assert.True(new[] { A, B }.SequenceEqual(DatabasesIn(both, array)), $"{read} over A and B read {string.Join(",", DatabasesIn(both, array))}: {both}"); + Assert.Equal("the chosen databases", Echo(both)); + + var one = await WebReadAsync(postgres, serverName, read, B); + Assert.Equal(new[] { B }, DatabasesIn(one, array)); + Assert.Equal(B, Echo(one)); + } + + /* The four public tools that gained database_name take it last, and a blank is every database. */ + var sizes = await DarlingMcpObjectStatsTools.GetDatabaseSizes(postgres, serverName, " ", ct); + Assert.Equal(new[] { A, B, C }, DatabasesIn(sizes, "databases")); + Assert.True(HasEchoKey(sizes)); + Assert.Null(Echo(sizes)); + Assert.Equal(new[] { A },DatabasesIn(await DarlingMcpObjectStatsTools.GetTableIndexSizes(postgres, serverName, A, ct), "tables")); + Assert.Equal(new[] { C }, DatabasesIn(await DarlingMcpPvsTools.GetPvsStats(postgres, serverName, 0, C, ct), "databases")); + Assert.Equal(new[] { B }, DatabasesIn(await DarlingMcpDataTools.GetFileIoStats(postgres, serverName, B, ct), "files")); + }, Cleanup); + + /// The empty answers of the three tools that gained database_name say which databases they were limited to. + [Fact] + public Task TheEmptyAnswers_CarryTheEcho() => + DurationTrendDatabaseFilterLiveTests.WithSharedStoreAsync("a5244p5w-echo", async (connection, postgres, serverId, serverName, now, ct) => + { + await StorageDatabaseFilterLiveTests.InsertTableAsync(connection, ct, serverId, serverName, now.AddHours(-1), A, "TA", 100m); + await StorageDatabaseFilterLiveTests.InsertFileAsync(connection, ct, serverId, serverName, now.AddHours(-1), A, 100m); + await StorageDatabaseFilterLiveTests.InsertPvsAsync(connection, ct, serverId, serverName, now.AddHours(-1), A, 100m); + + /* A snapshot exists, the chosen database is not in it: empty, naming the database. */ + foreach (var read in new[] { "get_database_sizes", "get_table_index_sizes", "get_pvs_stats" }) + { + var one = await WebReadAsync(postgres, serverName, read, "NoSuchDb"); + Assert.Equal("empty", DarlingMcpTestData.StatusOf(one)); + Assert.Equal("NoSuchDb", Echo(one)); + var many = await WebReadAsync(postgres, serverName, read, "NoX", "NoY"); + Assert.Equal("empty", DarlingMcpTestData.StatusOf(many)); + Assert.Equal("the chosen databases", Echo(many)); + } + }, Cleanup); + + /// A server with no data at all: the no-data answers carry the echo too, null for every database. + [Fact] + public Task TheNoDataAnswers_CarryTheEcho() => + DurationTrendDatabaseFilterLiveTests.WithSharedStoreAsync("a5244p5w-echo-none", async (connection, postgres, serverId, serverName, now, ct) => + { + await Task.CompletedTask; + foreach (var read in new[] { "get_database_sizes", "get_table_index_sizes", "get_pvs_stats" }) + { + var none = await WebReadAsync(postgres, serverName, read, A, B); + Assert.Equal("the chosen databases", Echo(none)); + var noneAll = await WebReadAsync(postgres, serverName, read); + Assert.True(HasEchoKey(noneAll), $"{read}'s no-data answer for every database has no database_name key: {noneAll}"); + Assert.Null(Echo(noneAll)); + } + }, Cleanup); +} diff --git a/Darling/Darling.Tests/StorageCommandTimeoutTests.cs b/Darling/Darling.Tests/StorageCommandTimeoutTests.cs index ac5e7dd362..16f22bc736 100644 --- a/Darling/Darling.Tests/StorageCommandTimeoutTests.cs +++ b/Darling/Darling.Tests/StorageCommandTimeoutTests.cs @@ -272,7 +272,8 @@ private static IEnumerable StorageSources() Assert.True(Directory.Exists(dir), $"storage project directory not found: {dir}"); - var paths = Directory.EnumerateFiles(dir, "*.cs", SearchOption.TopDirectoryOnly) + var paths = Directory.EnumerateFiles(dir, "*.cs", SearchOption.AllDirectories) + .Where(p => !IsBuildOutput(dir, p)) .OrderBy(p => p, System.StringComparer.Ordinal) .ToArray(); @@ -281,6 +282,44 @@ private static IEnumerable StorageSources() return paths; } + /// + /// True when a path sits under the project's bin or obj tree. Compared as PATH + /// SEGMENTS, so a source file that merely has "obj" in its name is not excluded. + /// + private static bool IsBuildOutput(string projectDir, string path) + { + var relative = Path.GetRelativePath(projectDir, path); + var segments = relative.Split(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar); + + return segments.Any(s => + string.Equals(s, "bin", System.StringComparison.OrdinalIgnoreCase) + || string.Equals(s, "obj", System.StringComparison.OrdinalIgnoreCase)); + } + + /// The sweep descends into Storage subfolders, so the FinOps readers' commands are covered. + [Fact] + public void TheStorageSweep_IncludesTheSubfolders_AndTheFinOpsReader() + { + var dir = Path.Combine(RepoRoot(), "Darling", "PerformanceMonitor.Darling.Storage"); + var swept = StorageSources().Select(p => p.Replace('\\', '/')).ToArray(); + + Assert.Contains(swept, p => p.EndsWith("/FinOps/DarlingFinOpsHighImpactReader.cs", System.StringComparison.Ordinal)); + Assert.DoesNotContain(StorageSources(), p => IsBuildOutput(dir, p)); + } + + /// The bin/obj rule compares whole path segments, relative to the project folder. + [Theory] + [InlineData("obj/Debug/x.cs", true)] + [InlineData("bin/Release/x.cs", true)] + [InlineData("Robj/x.cs", false)] + [InlineData("FinOps/objects.cs", false)] + public void IsBuildOutput_MatchesWholeSegmentsOnly(string relative, bool expected) + { + var dir = Path.Combine(Path.GetTempPath(), "proj"); + + Assert.Equal(expected, IsBuildOutput(dir, Path.Combine(dir, relative))); + } + private static string RepoRoot([CallerFilePath] string thisFile = "") { var dir = Path.GetDirectoryName(thisFile)!; diff --git a/Darling/Darling.Tests/StorageDatabaseFilterLiveTests.cs b/Darling/Darling.Tests/StorageDatabaseFilterLiveTests.cs new file mode 100644 index 0000000000..16504de63d --- /dev/null +++ b/Darling/Darling.Tests/StorageDatabaseFilterLiveTests.cs @@ -0,0 +1,353 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Text.Json; +using System.Threading.Tasks; +using Darling.Tests; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace PerformanceMonitor.Darling.Tests; + +/// +/// #5244 PR5, lane R1: the database filter on the storage reads get_table_index_sizes, get_database_sizes and get_pvs_stats, +/// run against a store. Every read takes a through an internal overload (the public MCP method +/// passes ). Each live test seeds databases A, B and C and chooses A and B, so a predicate +/// that is missing, or that matches one name only, shows as C (or a missing B) in the answer. The one-name consumers on each +/// read's empty path are covered: a filter that matches nothing is empty "for the chosen databases" when the server has +/// data, and stays unavailable or not_collected when it has none. +/// +[Collection("live-postgres")] +public sealed class StorageDatabaseFilterLiveTests +{ + internal const string Cleanup = + "DELETE FROM index_object_stats WHERE server_id = {0}; DELETE FROM database_size_stats WHERE server_id = {0}; " + + "DELETE FROM pvs_stats WHERE server_id = {0}"; + + internal const string A = "StorageFilterA"; + internal const string B = "StorageFilterB"; + internal const string C = "StorageFilterC"; + + private static DatabaseFilter Of(params string[] names) => DatabaseFilter.Of(names); + + private static JsonElement Parse(string json) => JsonDocument.Parse(json).RootElement.Clone(); + + private static string Message(string json) => Parse(json).GetProperty("message").GetString()!; + + private static string[] Names(JsonElement root, string array) => + root.GetProperty(array).EnumerateArray().Select(e => e.GetProperty("database_name").GetString()!).ToArray(); + + /* ───────────────────────────── seeding ───────────────────────────── */ + + internal static Task InsertTableAsync(Npgsql.NpgsqlConnection connection, System.Threading.CancellationToken ct, + int serverId, string serverName, DateTime t, string db, string table, decimal reservedMb) => + DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO index_object_stats (collection_id, collection_time, server_id, server_name, database_name, schema_name, object_id, table_name, index_id, index_name, index_type_desc, reserved_mb, used_mb, total_rows) +VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(t), serverId, serverName, db, "dbo", 100, table, 1, + "PK_" + table, "CLUSTERED", reservedMb, reservedMb / 2, 1_000L); + + internal static Task InsertFileAsync(Npgsql.NpgsqlConnection connection, System.Threading.CancellationToken ct, + int serverId, string serverName, DateTime t, string db, decimal sizeMb) => + DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO database_size_stats (collection_id, collection_time, server_id, server_name, database_name, database_id, file_id, file_type_desc, file_name, physical_name, total_size_mb, used_size_mb) +VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(t), serverId, serverName, db, 5, 1, "ROWS", db + "_data", + "C:\\data\\" + db + ".mdf", sizeMb, sizeMb / 2); + + internal static Task InsertPvsAsync(Npgsql.NpgsqlConnection connection, System.Threading.CancellationToken ct, + int serverId, string serverName, DateTime t, string db, decimal pvsMb) => + DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO pvs_stats (collection_id, collection_time, server_id, server_name, database_name, database_id, is_accelerated_database_recovery_on, persistent_version_store_size_mb, database_data_size_mb) +VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)", + CollectionIdGenerator.Next(), DarlingMcpTestData.Naive(t), serverId, serverName, db, 7, true, pvsMb, 1000m); + + /* ───────────────────────────── get_table_index_sizes ───────────────────────────── */ + + /// + /// Two of three databases: only the chosen databases' tables come back, the largest table (database C's) is not there, + /// and the history block is the server's own span, equal to the unfiltered call's, though the oldest snapshot + /// holds only database C (the snapshot anchors never move with the filter). + /// + [Fact] + public Task TableIndexSizes_TwoOfThree_OnlyTheChosenDatabases_HistoryStaysTheServers() => + DurationTrendDatabaseFilterLiveTests.WithSharedStoreAsync("a5244p5r1-tables", async (connection, postgres, serverId, serverName, now, ct) => + { + await InsertTableAsync(connection, ct, serverId, serverName, now.AddDays(-10), C, "TC", 800m); + await InsertTableAsync(connection, ct, serverId, serverName, now.AddHours(-1), A, "TA", 300m); + await InsertTableAsync(connection, ct, serverId, serverName, now.AddHours(-1), B, "TB", 200m); + await InsertTableAsync(connection, ct, serverId, serverName, now.AddHours(-1), C, "TC", 900m); + + var all = Parse(await DarlingMcpObjectStatsTools.GetTableIndexSizes(postgres, serverName, DatabaseFilter.All, ct)); + Assert.Equal(new[] { C, A, B }, Names(all, "tables")); + + var chosen = Parse(await DarlingMcpObjectStatsTools.GetTableIndexSizes(postgres, serverName, Of(A, B), ct)); + Assert.Equal(new[] { A, B }, Names(chosen, "tables")); + Assert.Equal(2, chosen.GetProperty("tables_returned").GetInt32()); + Assert.False(chosen.GetProperty("truncated").GetBoolean()); + Assert.Equal(all.GetProperty("history").GetRawText(), chosen.GetProperty("history").GetRawText()); + + var one = Parse(await DarlingMcpObjectStatsTools.GetTableIndexSizes(postgres, serverName, DatabaseFilter.One(B), ct)); + Assert.Equal(new[] { B }, Names(one, "tables")); + + /* The public MCP method passes every database, and a whitespace-only name is every database too (M3). */ + Assert.Equal(all.GetRawText(), Parse(await DarlingMcpObjectStatsTools.GetTableIndexSizes(postgres, serverName, cancellationToken: ct)).GetRawText()); + Assert.Equal(all.GetRawText(), Parse(await DarlingMcpObjectStatsTools.GetTableIndexSizes(postgres, serverName, DatabaseFilter.One(" "), ct)).GetRawText()); + }, Cleanup); + + /// + /// The cap of 100 applies after the filter: 105 big tables in database C and two small ones in A. Unfiltered, the page is + /// 100 tables of C and truncated; filtered to A and B, the page holds both small tables and is not truncated (a cap taken + /// before the filter would have returned nothing). + /// + [Fact] + public Task TableIndexSizes_CapAppliesAfterTheFilter() => + DurationTrendDatabaseFilterLiveTests.WithSharedStoreAsync("a5244p5r1-tables-cap", async (connection, postgres, serverId, serverName, now, ct) => + { + for (var i = 0; i < 105; i++) + { + await InsertTableAsync(connection, ct, serverId, serverName, now.AddHours(-1), C, "Big" + i, 1000m + i); + } + + await InsertTableAsync(connection, ct, serverId, serverName, now.AddHours(-1), A, "SmallA", 5m); + await InsertTableAsync(connection, ct, serverId, serverName, now.AddHours(-1), B, "SmallB", 4m); + + var all = Parse(await DarlingMcpObjectStatsTools.GetTableIndexSizes(postgres, serverName, DatabaseFilter.All, ct)); + Assert.Equal(100, all.GetProperty("tables_returned").GetInt32()); + Assert.True(all.GetProperty("truncated").GetBoolean()); + Assert.DoesNotContain(A, Names(all, "tables")); + + var chosen = Parse(await DarlingMcpObjectStatsTools.GetTableIndexSizes(postgres, serverName, Of(A, B), ct)); + Assert.Equal(new[] { A, B }, Names(chosen, "tables")); + Assert.False(chosen.GetProperty("truncated").GetBoolean()); + }, Cleanup); + + /// + /// The empty path. A filter that matches no table, on a server that has index rows, is empty and says so for the + /// database (one name) or the chosen databases (two or more), never unavailable. A server with no index rows at + /// all stays unavailable under a filter and without one, and a server whose engine cannot collect stays + /// not_collected. + /// + [Fact] + public Task TableIndexSizes_EmptyPaths_SayWhichKindOfNothing() => + DurationTrendDatabaseFilterLiveTests.WithSharedStoreAsync("a5244p5r1-tables-empty", async (connection, postgres, serverId, serverName, now, ct) => + { + var none = await DarlingMcpObjectStatsTools.GetTableIndexSizes(postgres, serverName, Of("Nope1", "Nope2"), ct); + Assert.Equal("unavailable", DarlingMcpTestData.StatusOf(none)); + Assert.Equal("unavailable", DarlingMcpTestData.StatusOf(await DarlingMcpObjectStatsTools.GetTableIndexSizes(postgres, serverName, DatabaseFilter.All, ct))); + + await InsertTableAsync(connection, ct, serverId, serverName, now.AddHours(-1), A, "TA", 300m); + + var one = await DarlingMcpObjectStatsTools.GetTableIndexSizes(postgres, serverName, DatabaseFilter.One("Nope1"), ct); + Assert.Equal("empty", DarlingMcpTestData.StatusOf(one)); + Assert.Contains("database 'Nope1'", Message(one), StringComparison.Ordinal); + Assert.DoesNotContain("chosen databases", Message(one), StringComparison.Ordinal); + + var many = await DarlingMcpObjectStatsTools.GetTableIndexSizes(postgres, serverName, Of("Nope1", "Nope2"), ct); + Assert.Equal("empty", DarlingMcpTestData.StatusOf(many)); + Assert.Contains("the chosen databases", Message(many), StringComparison.Ordinal); + Assert.DoesNotContain("Nope1", Message(many), StringComparison.Ordinal); + + await DarlingMcpTestData.ExecAsync(connection, ct, "UPDATE servers SET engine_kind = $2 WHERE server_id = $1", serverId, MonitoredEngineKind.Postgres); + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM index_object_stats WHERE server_id = $1", serverId); + Assert.Equal("not_collected", DarlingMcpTestData.StatusOf( + await DarlingMcpObjectStatsTools.GetTableIndexSizes(postgres, serverName, Of("Nope1", "Nope2"), ct))); + }, Cleanup); + + /* ───────────────────────────── get_database_sizes ───────────────────────────── */ + + /// + /// Two of three databases at the server's newest snapshot: only the chosen databases' files, the same + /// captured_at as the unfiltered call. A database that exists only in an OLDER snapshot is not found (the snapshot + /// never moves with the filter), and that is an empty answer for the database, not unavailable. + /// + [Fact] + public Task DatabaseSizes_TwoOfThree_OnlyTheChosenDatabases_SnapshotStaysTheServers() => + DurationTrendDatabaseFilterLiveTests.WithSharedStoreAsync("a5244p5r1-sizes", async (connection, postgres, serverId, serverName, now, ct) => + { + await InsertFileAsync(connection, ct, serverId, serverName, now.AddHours(-5), "OnlyOlder", 77m); + await InsertFileAsync(connection, ct, serverId, serverName, now.AddHours(-1), A, 100m); + await InsertFileAsync(connection, ct, serverId, serverName, now.AddHours(-1), B, 200m); + await InsertFileAsync(connection, ct, serverId, serverName, now.AddHours(-1), C, 300m); + + var all = Parse(await DarlingMcpObjectStatsTools.GetDatabaseSizes(postgres, serverName, DatabaseFilter.All, ct)); + Assert.Equal(new[] { A, B, C }, Names(all, "databases")); + + var chosen = Parse(await DarlingMcpObjectStatsTools.GetDatabaseSizes(postgres, serverName, Of(A, B), ct)); + Assert.Equal(new[] { A, B }, Names(chosen, "databases")); + Assert.Equal(2, chosen.GetProperty("file_count").GetInt32()); + Assert.Equal(all.GetProperty("captured_at").GetString(), chosen.GetProperty("captured_at").GetString()); + + /* The exact spelling: a differently cased name is not the database (the SQL arm compares with =). */ + Assert.Equal("empty", DarlingMcpTestData.StatusOf( + await DarlingMcpObjectStatsTools.GetDatabaseSizes(postgres, serverName, DatabaseFilter.One(A.ToUpperInvariant()), ct))); + + Assert.Equal(all.GetRawText(), Parse(await DarlingMcpObjectStatsTools.GetDatabaseSizes(postgres, serverName, cancellationToken: ct)).GetRawText()); + Assert.Equal(all.GetRawText(), Parse(await DarlingMcpObjectStatsTools.GetDatabaseSizes(postgres, serverName, DatabaseFilter.One(""), ct)).GetRawText()); + + var older = await DarlingMcpObjectStatsTools.GetDatabaseSizes(postgres, serverName, DatabaseFilter.One("OnlyOlder"), ct); + Assert.Equal("empty", DarlingMcpTestData.StatusOf(older)); + Assert.Contains("database 'OnlyOlder'", Message(older), StringComparison.Ordinal); + + var olderMany = await DarlingMcpObjectStatsTools.GetDatabaseSizes(postgres, serverName, Of("OnlyOlder", "Nope"), ct); + Assert.Equal("empty", DarlingMcpTestData.StatusOf(olderMany)); + Assert.Contains("the chosen databases", Message(olderMany), StringComparison.Ordinal); + }, Cleanup); + + /// The empty path with no snapshot at all stays unavailable (filtered or not), and a server whose engine + /// cannot collect stays not_collected. + [Fact] + public Task DatabaseSizes_NoSnapshot_StaysUnavailableOrNotCollected() => + DurationTrendDatabaseFilterLiveTests.WithSharedStoreAsync("a5244p5r1-sizes-empty", async (connection, postgres, serverId, serverName, now, ct) => + { + Assert.Equal("unavailable", DarlingMcpTestData.StatusOf( + await DarlingMcpObjectStatsTools.GetDatabaseSizes(postgres, serverName, Of("Nope1", "Nope2"), ct))); + Assert.Equal("unavailable", DarlingMcpTestData.StatusOf( + await DarlingMcpObjectStatsTools.GetDatabaseSizes(postgres, serverName, DatabaseFilter.All, ct))); + + await DarlingMcpTestData.ExecAsync(connection, ct, "UPDATE servers SET engine_kind = $2 WHERE server_id = $1", serverId, MonitoredEngineKind.Postgres); + Assert.Equal("not_collected", DarlingMcpTestData.StatusOf( + await DarlingMcpObjectStatsTools.GetDatabaseSizes(postgres, serverName, DatabaseFilter.One("Nope1"), ct))); + }, Cleanup); + + /* ───────────────────────────── get_pvs_stats ───────────────────────────── */ + + /// + /// Two of three databases at the server's newest snapshot: only the chosen databases' rows, the same as_of as the + /// unfiltered call, and a trend over the chosen databases only. + /// + [Fact] + public Task PvsStats_TwoOfThree_OnlyTheChosenDatabases_AsOfStaysTheServers() => + DurationTrendDatabaseFilterLiveTests.WithSharedStoreAsync("a5244p5r1-pvs", async (connection, postgres, serverId, serverName, now, ct) => + { + await InsertPvsAsync(connection, ct, serverId, serverName, now.AddHours(-1), A, 50m); + await InsertPvsAsync(connection, ct, serverId, serverName, now.AddHours(-1), B, 10m); + await InsertPvsAsync(connection, ct, serverId, serverName, now.AddHours(-1), C, 900m); + await InsertPvsAsync(connection, ct, serverId, serverName, now.AddHours(-2), A, 40m); + await InsertPvsAsync(connection, ct, serverId, serverName, now.AddHours(-2), B, 8m); + await InsertPvsAsync(connection, ct, serverId, serverName, now.AddHours(-2), C, 800m); + + var all = Parse(await DarlingMcpPvsTools.GetPvsStats(postgres, serverName, 24, DatabaseFilter.All, ct)); + Assert.Equal(new[] { C, A, B }, Names(all, "databases")); + Assert.Equal(new[] { A, B, C }, Names(all, "trend")); + + var chosen = Parse(await DarlingMcpPvsTools.GetPvsStats(postgres, serverName, 24, Of(A, B), ct)); + Assert.Equal(new[] { A, B }, Names(chosen, "databases")); + Assert.Equal(new[] { A, B }, Names(chosen, "trend")); + Assert.Equal(all.GetProperty("as_of").GetString(), chosen.GetProperty("as_of").GetString()); + + var one = Parse(await DarlingMcpPvsTools.GetPvsStats(postgres, serverName, 0, DatabaseFilter.One(B), ct)); + Assert.Equal(new[] { B }, Names(one, "databases")); + + Assert.Equal(all.GetRawText(), Parse(await DarlingMcpPvsTools.GetPvsStats(postgres, serverName, 24, cancellationToken: ct)).GetRawText()); + Assert.Equal(all.GetRawText(), Parse(await DarlingMcpPvsTools.GetPvsStats(postgres, serverName, 24, DatabaseFilter.One(" "), ct)).GetRawText()); + }, Cleanup); + + /// + /// The trend's top five is the top five of the CHOSEN databases: seven big databases outrank A and B, so an unfiltered + /// trend holds five of the big ones and none of A or B, while a trend filtered to A and B holds both. + /// + [Fact] + public Task PvsStats_TrendTopFiveAppliesAfterTheFilter() => + DurationTrendDatabaseFilterLiveTests.WithSharedStoreAsync("a5244p5r1-pvs-top5", async (connection, postgres, serverId, serverName, now, ct) => + { + for (var i = 1; i <= 7; i++) + { + await InsertPvsAsync(connection, ct, serverId, serverName, now.AddHours(-1), "Big" + i, 100m * i); + await InsertPvsAsync(connection, ct, serverId, serverName, now.AddHours(-2), "Big" + i, 90m * i); + } + + await InsertPvsAsync(connection, ct, serverId, serverName, now.AddHours(-1), A, 5m); + await InsertPvsAsync(connection, ct, serverId, serverName, now.AddHours(-1), B, 4m); + await InsertPvsAsync(connection, ct, serverId, serverName, now.AddHours(-2), A, 3m); + await InsertPvsAsync(connection, ct, serverId, serverName, now.AddHours(-2), B, 2m); + + var all = Parse(await DarlingMcpPvsTools.GetPvsStats(postgres, serverName, 24, DatabaseFilter.All, ct)); + Assert.Equal(5, Names(all, "trend").Length); + Assert.DoesNotContain(A, Names(all, "trend")); + + var chosen = Parse(await DarlingMcpPvsTools.GetPvsStats(postgres, serverName, 24, Of(A, B), ct)); + Assert.Equal(new[] { A, B }, Names(chosen, "trend")); + Assert.All(chosen.GetProperty("trend").EnumerateArray(), t => Assert.Equal(2, t.GetProperty("points").GetArrayLength())); + + /* The reader-level calls the tool makes: the same sets. */ + var rows = await DarlingPvsReader.GetPvsStatsLatestAsync(postgres, serverId, Of(A, B), ct); + Assert.Equal(new[] { A, B }, rows.Select(r => r.DatabaseName).ToArray()); + var points = await DarlingPvsReader.GetPvsTrendAsync(postgres, serverId, now.AddHours(-24), Of(A, B), ct); + Assert.Equal(new[] { A, B }, points.Select(p => p.DatabaseName).Distinct().ToArray()); + }, Cleanup); + + /// + /// The empty path. A filter that matches no database, on a server with a PVS snapshot, is empty for the database + /// (one name) or the chosen databases (two or more) and does not say the server has no PVS data. A server with no PVS rows + /// keeps its own sentence under a filter and without one, and a server whose engine cannot collect stays + /// not_collected. + /// + [Fact] + public Task PvsStats_EmptyPaths_SayWhichKindOfNothing() => + DurationTrendDatabaseFilterLiveTests.WithSharedStoreAsync("a5244p5r1-pvs-empty", async (connection, postgres, serverId, serverName, now, ct) => + { + var noData = await DarlingMcpPvsTools.GetPvsStats(postgres, serverName, 0, Of("Nope1", "Nope2"), ct); + Assert.Equal("empty", DarlingMcpTestData.StatusOf(noData)); + Assert.Contains("No PVS data collected for this server", Message(noData), StringComparison.Ordinal); + Assert.Equal(Message(noData), Message(await DarlingMcpPvsTools.GetPvsStats(postgres, serverName, 0, DatabaseFilter.All, ct))); + + await InsertPvsAsync(connection, ct, serverId, serverName, now.AddHours(-1), A, 50m); + await InsertPvsAsync(connection, ct, serverId, serverName, now.AddHours(-1), B, 10m); + + var one = await DarlingMcpPvsTools.GetPvsStats(postgres, serverName, 0, DatabaseFilter.One("Nope1"), ct); + Assert.Equal("empty", DarlingMcpTestData.StatusOf(one)); + Assert.Contains("database 'Nope1'", Message(one), StringComparison.Ordinal); + Assert.DoesNotContain("No PVS data collected for this server", Message(one), StringComparison.Ordinal); + + var many = await DarlingMcpPvsTools.GetPvsStats(postgres, serverName, 0, Of("Nope1", "Nope2"), ct); + Assert.Equal("empty", DarlingMcpTestData.StatusOf(many)); + Assert.Contains("the chosen databases", Message(many), StringComparison.Ordinal); + Assert.Contains("2 other database(s)", Message(many), StringComparison.Ordinal); + + await DarlingMcpTestData.ExecAsync(connection, ct, "DELETE FROM pvs_stats WHERE server_id = $1", serverId); + await DarlingMcpTestData.ExecAsync(connection, ct, "UPDATE servers SET engine_kind = $2 WHERE server_id = $1", serverId, MonitoredEngineKind.Postgres); + Assert.Equal("not_collected", DarlingMcpTestData.StatusOf( + await DarlingMcpPvsTools.GetPvsStats(postgres, serverName, 0, DatabaseFilter.One("Nope1"), ct))); + }, Cleanup); + + /* ───────────────────────────── the statements ───────────────────────────── */ + + /// + /// The list predicate is the desktop's, bound as ONE text[] at the index each statement documents: $5 on the four snapshot + /// CTEs of the table-size read (and not on boundaries, which carries the snapshot anchors and the store's span), $3 + /// on the database-size read, $2 on the PVS snapshot read, $3 on the PVS trend's top-five CTE. + /// + [Fact] + public void Statements_CarryTheListPredicate_OnTheRightRelations() + { + const string Predicate = "{0}::text[] IS NULL OR database_name = ANY({0})"; + + var growth = DarlingObjectStatsReader.ObjectSizeGrowthSql; + var predicate5 = string.Format(System.Globalization.CultureInfo.InvariantCulture, Predicate, "$5"); + Assert.Equal(4, growth.Split(predicate5).Length - 1); + var boundaries = growth[growth.IndexOf("boundaries AS (", StringComparison.Ordinal)..growth.IndexOf("latest AS (", StringComparison.Ordinal)]; + Assert.DoesNotContain("$5", boundaries, StringComparison.Ordinal); + + Assert.Contains(string.Format(System.Globalization.CultureInfo.InvariantCulture, Predicate, "$3"), DarlingObjectStatsReader.DatabaseSizeLatestSql, StringComparison.Ordinal); + Assert.Contains(string.Format(System.Globalization.CultureInfo.InvariantCulture, Predicate, "$2"), DarlingPvsReader.PvsStatsLatestSql, StringComparison.Ordinal); + + var trend = DarlingPvsReader.PvsTrendSql; + var topDbs = trend[..trend.IndexOf("JOIN top_dbs", StringComparison.Ordinal)]; + Assert.Contains(string.Format(System.Globalization.CultureInfo.InvariantCulture, Predicate, "$3"), topDbs, StringComparison.Ordinal); + Assert.Contains("LIMIT 5", topDbs, StringComparison.Ordinal); + + /* Byte-for-byte the predicate DatabaseFilter.Clause spells. */ + Assert.Contains(DatabaseFilter.All.Clause("database_name", 5).TrimStart().Substring("AND ".Length), growth, StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/StorageGrowthNewTableCutLiveTests.cs b/Darling/Darling.Tests/StorageGrowthNewTableCutLiveTests.cs new file mode 100644 index 0000000000..6248ca3bc6 --- /dev/null +++ b/Darling/Darling.Tests/StorageGrowthNewTableCutLiveTests.cs @@ -0,0 +1,126 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Storage.FinOps; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: the fixture seeds its own scratch database, so nothing here shares rows with another test. */ +/// +/// A table with no row at the database's earliest snapshot in the window was created inside it, so it counts its +/// whole size as growth. The grid's summary and the heatmap's series must agree on that, and cut the same top-N. +/// +[Trait("Reads", "Lite")] +public sealed class StorageGrowthNewTableCutLiveTests +{ + private const string ServerName = "darling-sg-newtable-cut"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + + private static readonly string[] Queries = + [ + DarlingFinOpsStorageGrowthReader.ObjectGrowthSummarySql, + DarlingFinOpsStorageGrowthReader.ObjectGrowthSeriesSql, + ]; + + [Fact] + public void BothQueries_CountAMissingEarliestRowAsZero_AndOrderTheSameWay() + { + foreach (var sql in Queries) + { + Assert.Contains("COALESCE(e.e_reserved_mb, 0)", sql, StringComparison.Ordinal); + Assert.DoesNotContain("COALESCE(e.e_reserved_mb, l.cur_reserved_mb)", sql, StringComparison.Ordinal); + Assert.Contains("growth_mb DESC NULLS LAST, l.schema_name, l.table_name", sql, StringComparison.Ordinal); + } + } + + [Fact] + public void LiteQueries_CountAMissingEarliestRowAsZero_AndOrderTheSameWay() + { + var dir = AppContext.BaseDirectory; + while (dir != null && !File.Exists(Path.Combine(dir, "Lite", "Services", "LocalDataService.FinOps.IndexObjects.cs"))) + dir = Path.GetDirectoryName(dir); + Assert.NotNull(dir); + var source = File.ReadAllText(Path.Combine(dir, "Lite", "Services", "LocalDataService.FinOps.IndexObjects.cs")); + + Assert.Equal(2, CountOf(source, "COALESCE(e.e_reserved_mb, 0)")); + Assert.DoesNotContain("COALESCE(e.e_reserved_mb, l.cur_reserved_mb)", source, StringComparison.Ordinal); + Assert.Equal(2, CountOf(source, "growth_mb DESC NULLS LAST, l.schema_name, l.table_name")); + } + + [Fact] + public async Task TheGridAndTheHeatmap_CutTheSameTopN_AndRankATableCreatedInTheWindowByItsWholeSize() + { + var cs = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live storage growth test."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(cs!, ct); + var earliest = new DateTime(2026, 3, 1, 0, 10, 0, DateTimeKind.Unspecified); + var latest = new DateTime(2026, 3, 31, 0, 10, 0, DateTimeKind.Unspecified); + await using (var c = new NpgsqlConnection(scratch.ConnectionString)) + { + await c.OpenAsync(ct); + await PgMigrations.MigrateAsync(c, ct); + await DarlingMcpTestData.RegisterServerAsync(c, ServerId, ServerName, ct); + + /* Two growing, three unchanged, two shrinking, one table that exists only at the latest instant and is the + biggest, and one whose latest size is unknown. */ + foreach (var (table, before, after) in new (string, decimal?, decimal?)[] + { + ("Grow1", 100m, 150m), ("Grow2", 100m, 130m), + ("Flat1", 50m, 50m), ("Flat2", 50m, 50m), ("Flat3", 50m, 50m), + ("Shrink1", 200m, 150m), ("Shrink2", 200m, 100m), + ("NewBig", null, 400m), + ("NoSize", 10m, null), + }) + { + if (before != null) await SeedAsync(c, table, earliest, before, ct); + await SeedAsync(c, table, latest, after, ct); + } + } + + await using var ds = NpgsqlDataSource.Create(scratch.ConnectionString); + var (objects, samples) = await DarlingFinOpsStorageGrowthReader.GetObjectGrowthHeatmapDataAsync( + ds, ServerId, "cutdb", earliest.Date, 30, 5, 30, ct); + + var summaryTables = objects.Select(o => o.TableName).ToArray(); + var seriesTables = samples.Select(s => s.ObjectKey.Split('.')[1]).Distinct().OrderBy(t => t, StringComparer.Ordinal).ToArray(); + Assert.Equal(new[] { "NewBig", "Grow1", "Grow2", "Flat1", "Flat2" }, summaryTables); + Assert.Equal(summaryTables.OrderBy(t => t, StringComparer.Ordinal).ToArray(), seriesTables); + + var added = objects[0]; + Assert.Equal("NewBig", added.TableName); + Assert.Equal(400m, added.Growth30dMb!.Value); + Assert.Null(added.GrowthPct30d); + Assert.DoesNotContain(summaryTables, t => t.StartsWith("Shrink", StringComparison.Ordinal)); + Assert.DoesNotContain(seriesTables, t => t.StartsWith("Shrink", StringComparison.Ordinal)); + } + + private static int CountOf(string text, string needle) + { + var count = 0; + for (var i = text.IndexOf(needle, StringComparison.Ordinal); i >= 0; i = text.IndexOf(needle, i + needle.Length, StringComparison.Ordinal)) count++; + return count; + } + + private static Task SeedAsync(NpgsqlConnection c, string table, DateTime at, decimal? reservedMb, CancellationToken ct) => + DarlingMcpTestData.ExecAsync(c, ct, + @"INSERT INTO index_object_stats (collection_id, collection_time, server_id, server_name, database_name, schema_name, object_id, table_name, index_id, index_name, index_type_desc, reserved_mb, used_mb, total_rows, user_seeks, user_scans, user_lookups, user_updates) + VALUES ($1,$2,$3,$4,'cutdb','dbo',$5,$6,1,$7,'CLUSTERED',$8,$8,1000,0,0,0,0)", + CollectionIdGenerator.Next(), at, ServerId, ServerName, 100 + table.Length, table, "PK_" + table, (object?)reservedMb ?? DBNull.Value); +} diff --git a/Darling/Darling.Tests/StorageGrowthNoHistoryLiveTests.cs b/Darling/Darling.Tests/StorageGrowthNoHistoryLiveTests.cs index e033a3ed9e..555ea6cb8c 100644 --- a/Darling/Darling.Tests/StorageGrowthNoHistoryLiveTests.cs +++ b/Darling/Darling.Tests/StorageGrowthNoHistoryLiveTests.cs @@ -48,16 +48,18 @@ public async Task ADatabaseWithNoPastRowReadsGrowthAsUnknown_NotZero() var now = DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow); await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); - /* fresh: only the newest snapshot. weekold: also a snapshot 8 days back. full: snapshots 31 and 8 days back. */ + /* fresh: only the newest snapshot. weekold: also a snapshot 7.5 days back. full: snapshots 30.5 and 7.5 days back. A baseline is + the snapshot nearest its 7 or 30 day mark within one day, so 7.5 and 30.5 sit inside the tolerance, and the daily rate + divides by the real span (7.5 and 30.5 days): a fixed 7 or 30 divisor would give a different figure. */ await SeedAsync(connection, "fresh", now, 500m, ct); await SeedAsync(connection, "weekold", now, 500m, ct); - await SeedAsync(connection, "weekold", now.AddDays(-8), 400m, ct); + await SeedAsync(connection, "weekold", now.AddDays(-7.5), 400m, ct); await SeedAsync(connection, "full", now, 500m, ct); - await SeedAsync(connection, "full", now.AddDays(-8), 450m, ct); - await SeedAsync(connection, "full", now.AddDays(-31), 300m, ct); + await SeedAsync(connection, "full", now.AddDays(-7.5), 450m, ct); + await SeedAsync(connection, "full", now.AddDays(-30.5), 300m, ct); await using var viewer = new ViewerDataService(cs!); - var rows = await viewer.GetStorageGrowthAsync(ServerId, ct); + var rows = await viewer.GetStorageGrowthAsync(ServerId, cancellationToken: ct); var fresh = Assert.Single(rows, r => r.DatabaseName == "fresh"); Assert.Equal(500m, fresh.CurrentSizeMb); @@ -69,13 +71,13 @@ public async Task ADatabaseWithNoPastRowReadsGrowthAsUnknown_NotZero() var weekOld = Assert.Single(rows, r => r.DatabaseName == "weekold"); Assert.Equal(100m, (decimal?)weekOld.Growth7dMb); Assert.Null((object?)weekOld.Growth30dMb); - Assert.Equal(100m / 8m, weekOld.DailyGrowthRateMb!.Value, 4); + Assert.Equal(100m / 7.5m, weekOld.DailyGrowthRateMb!.Value, 4); Assert.Null((object?)weekOld.GrowthPct30d); var full = Assert.Single(rows, r => r.DatabaseName == "full"); Assert.Equal(50m, (decimal?)full.Growth7dMb); Assert.Equal(200m, (decimal?)full.Growth30dMb); - Assert.Equal(200m / 31m, full.DailyGrowthRateMb!.Value, 4); + Assert.Equal(200m / 30.5m, full.DailyGrowthRateMb!.Value, 4); Assert.Equal(200m * 100m / 300m, full.GrowthPct30d!.Value, 4); /* Largest 30-day growth first; a database with no 30-day figure sorts after, by its 7-day growth. */ @@ -89,6 +91,57 @@ public async Task ADatabaseWithNoPastRowReadsGrowthAsUnknown_NotZero() } } + [Fact] + public async Task ASixDayHistoryGivesARealSevenDayBaseline_AndNothingFor30Days() + { + var cs = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live storage growth test."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var bodySucceeded = false; + try + { + await DeleteRowsAsync(connection, ct); + var now = DarlingMcpTestData.TruncateToSeconds(DateTime.UtcNow); + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + + /* A store that has run for 6 days: the oldest sample is 6 days back, one day from the 7-day mark and so inside the tolerance, so + it is the 7-day baseline. "flat" never changed, so its baseline equals its current size and its growth is a real 0. Neither + has a sample near 30 days ago, so the 30-day figures are unknown. */ + foreach (var (daysAgo, grown, flat) in new[] { (6, 600m, 500m), (4, 700m, 500m), (2, 900m, 500m), (0, 1000m, 500m) }) + { + await SeedAsync(connection, "grown", now.AddDays(-daysAgo), grown, ct); + await SeedAsync(connection, "flat", now.AddDays(-daysAgo), flat, ct); + } + + await using var viewer = new ViewerDataService(cs!); + var rows = await viewer.GetStorageGrowthAsync(ServerId, cancellationToken: ct); + + var grownRow = Assert.Single(rows, r => r.DatabaseName == "grown"); + Assert.Equal(600m, grownRow.Size7dAgoMb); + Assert.Equal(400m, grownRow.Growth7dMb); + Assert.Null(grownRow.Size7dAgoNote); + Assert.Null(grownRow.Size30dAgoMb); + Assert.Null(grownRow.Growth30dMb); + Assert.Equal("No sample from 30 days ago", grownRow.Size30dAgoNote); + + var flatRow = Assert.Single(rows, r => r.DatabaseName == "flat"); + Assert.Equal(500m, flatRow.Size7dAgoMb); + Assert.Equal(0m, flatRow.Growth7dMb); + Assert.Null(flatRow.Size30dAgoMb); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, DeleteRowsAsync); + } + } + [Fact] public async Task AStaleServerReadsGrowthAsUnknown_AndTheRateUsesTheRealElapsedDays() { @@ -111,21 +164,32 @@ public async Task AStaleServerReadsGrowthAsUnknown_AndTheRateUsesTheRealElapsedD /* Collection stopped 20 days ago: "at or before 7 days ago" is the latest snapshot itself, which is no comparison. */ await SeedAsync(connection, "stale", now.AddDays(-20), 500m, ct); - var stale = Assert.Single(await viewer.GetStorageGrowthAsync(ServerId, ct)); + var stale = Assert.Single(await viewer.GetStorageGrowthAsync(ServerId, cancellationToken: ct)); Assert.Null(stale.Size7dAgoMb); Assert.Null(stale.Growth7dMb); Assert.Null(stale.Growth30dMb); Assert.Null(stale.DailyGrowthRateMb); Assert.Null(stale.GrowthPct30d); - /* The "7-day" past is really 12 days old after a collection gap: the rate is over those 12 days. */ + /* The "7-day" past is really 12 days old after a collection gap: that is not within a day of the 7-day mark, so there is no + 7-day baseline at all (unknown, never a 12-day figure labelled 7 days). */ await DeleteRowsAsync(connection, ct); await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); await SeedAsync(connection, "gap", now, 500m, ct); await SeedAsync(connection, "gap", now.AddDays(-12), 380m, ct); - var gap = Assert.Single(await viewer.GetStorageGrowthAsync(ServerId, ct)); - Assert.Equal(120m, gap.Growth7dMb!.Value); - Assert.Equal(120m / 12m, gap.DailyGrowthRateMb!.Value, 4); + var gap = Assert.Single(await viewer.GetStorageGrowthAsync(ServerId, cancellationToken: ct)); + Assert.Null(gap.Size7dAgoMb); + Assert.Null(gap.Growth7dMb); + Assert.Null(gap.DailyGrowthRateMb); + + /* A past 7.5 days old (inside the tolerance) is the 7-day baseline, and the rate is over its real 7.5 days. */ + await DeleteRowsAsync(connection, ct); + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + await SeedAsync(connection, "near", now, 500m, ct); + await SeedAsync(connection, "near", now.AddDays(-7.5), 380m, ct); + var near = Assert.Single(await viewer.GetStorageGrowthAsync(ServerId, cancellationToken: ct)); + Assert.Equal(120m, near.Growth7dMb!.Value); + Assert.Equal(120m / 7.5m, near.DailyGrowthRateMb!.Value, 4); bodySucceeded = true; } @@ -136,7 +200,7 @@ public async Task AStaleServerReadsGrowthAsUnknown_AndTheRateUsesTheRealElapsedD } [Fact] - public async Task TheTableDrillReadsGrowthAsUnknown_ForATableWithNoEarlierSample() + public async Task TheTableDrillCountsATableCreatedInTheWindowByItsWholeSize_AndReadsOneSnapshotAsUnknown() { var cs = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live storage growth test."); @@ -163,7 +227,7 @@ public async Task TheTableDrillReadsGrowthAsUnknown_ForATableWithNoEarlierSample Assert.Null(only.DailyGrowthRateMb); Assert.Null(only.GrowthPct30d); - /* Two snapshots: a table in both has growth, a table added since has none. */ + /* Two snapshots: a table in both has growth, a table added since counts its whole size. */ await SeedTableAsync(connection, "Old", now.AddDays(-10), 100m, ct); await SeedTableAsync(connection, "Old", now, 150m, ct); await SeedTableAsync(connection, "NewTable", now, 30m, ct); @@ -173,9 +237,11 @@ public async Task TheTableDrillReadsGrowthAsUnknown_ForATableWithNoEarlierSample Assert.Equal(50m / 30m, old.DailyGrowthRateMb!.Value, 4); Assert.Equal(50m * 100m / 100m, old.GrowthPct30d!.Value, 4); var added = Assert.Single(objects, o => o.TableName == "NewTable"); - Assert.Null(added.Growth30dMb); + Assert.Equal(30m, added.Growth30dMb!.Value); + Assert.Equal(30m / 30m, added.DailyGrowthRateMb!.Value, 4); Assert.Null(added.GrowthPct30d); - Assert.Equal("Old", objects[0].TableName); + /* OnlyNow (100 MB) is absent from the earlier snapshot too, so it counts its whole size and ranks first. */ + Assert.Equal(new[] { "OnlyNow", "Old", "NewTable" }, objects.Select(x => x.TableName).ToArray()); bodySucceeded = true; } diff --git a/Darling/Darling.Tests/StoreApplicationNameCensusTests.cs b/Darling/Darling.Tests/StoreApplicationNameCensusTests.cs index 3ccbb79ed4..a044d4f2c8 100644 --- a/Darling/Darling.Tests/StoreApplicationNameCensusTests.cs +++ b/Darling/Darling.Tests/StoreApplicationNameCensusTests.cs @@ -33,6 +33,7 @@ namespace Darling.Tests; /// WithApplicationName( call on the same line, the same round-1-review discipline the TZ census /// applies to its own pin). /// +[Trait("Stage", "Guard")] public sealed class StoreApplicationNameCensusTests { /* Same floors as StoreSessionTimeZonePinCensusTests: this scan finds the identical 29 call sites (the @@ -63,6 +64,9 @@ makes no sense — a monitored target's own pg_stat_activity is the target's bus "DarlingWorker.cs:ReadPgStatementTextAsync", "DarlingWorker.cs:RunTestHypotheticalIndexAsync", + /* The RDS endpoint check's fresh unpooled login to the MONITORED target (runtime.ConnectionString), never the store. */ + "RdsEndpointVerifier.cs:ProbeLoginAsync", + /* DarlingWorker's custom-alert viewer source (#4479): its connection string comes from either DarlingManagedPostgres.TryBuildViewerConnectionStringFromStoredCredential (BuildRoleConnectionString, WebApplicationName) or DarlingStoreLogins.ResolveComposeCustomAlertViewerAsync (which returns diff --git a/Darling/Darling.Tests/StoreConfigProviderConnectionMatchTests.cs b/Darling/Darling.Tests/StoreConfigProviderConnectionMatchTests.cs new file mode 100644 index 0000000000..eea60dffad --- /dev/null +++ b/Darling/Darling.Tests/StoreConfigProviderConnectionMatchTests.cs @@ -0,0 +1,169 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using PerformanceMonitor.Darling.Service; +using Xunit; + +namespace Darling.Tests; + +/// +/// A darling.json entry and a store row are the same connection only when they agree on all ten connection settings plus +/// engine (ConnectionSettingsDiffer, authentication compared ignoring case). The file entry's secret is copied into a +/// row that has none, and a reference the file declares is marked on a row, only for such a row. +/// +public sealed class StoreConfigProviderConnectionMatchTests +{ + private const string FileSecret = "file-secret-value-Q7"; + private const string DeclaredReference = "env:DARLING_TEST_DECLARED_REFERENCE_5374"; + + private static MonitoredServer Entry() => new() + { + Name = "alpha", + Host = "alpha.example.test", + Port = 1433, + Database = "master", + Auth = "sql", + Username = "monitor", + EncryptMode = "Mandatory", + TrustServerCertificate = false, + MultiSubnetFailover = false, + ReadOnlyIntent = false, + Engine = "sqlserver", + EncryptedPassword = FileSecret, + EncryptedPasswordDeclaredByFile = true, + }; + + private static MonitoredServer RowLike(MonitoredServer entry, string? secret) => new() + { + Name = entry.Name, + StoredServerId = entry.ServerId, + Host = entry.Host, + Port = entry.Port, + Database = entry.Database, + Auth = entry.Auth, + Username = entry.Username, + EncryptMode = entry.EncryptMode, + TrustServerCertificate = entry.TrustServerCertificate, + MultiSubnetFailover = entry.MultiSubnetFailover, + ReadOnlyIntent = entry.ReadOnlyIntent, + Engine = entry.Engine, + EncryptedPassword = secret, + }; + + public static IEnumerable OneFieldDiffers() => + new[] { "encrypt mode", "trust certificate", "multi-subnet", "auth kind", "username", "database", "read-only", "engine", "port", "host" } + .Select(f => new object[] { f }); + + private static void Change(string field, MonitoredServer r) + { + switch (field) + { + case "encrypt mode": r.EncryptMode = "Optional"; break; + case "trust certificate": r.TrustServerCertificate = true; break; + case "multi-subnet": r.MultiSubnetFailover = true; break; + case "auth kind": r.Auth = "serviceprincipal"; break; + case "username": r.Username = "other"; break; + case "database": r.Database = "tempdb"; break; + case "read-only": r.ReadOnlyIntent = true; break; + case "engine": r.Engine = "postgresql"; break; + case "port": r.Port = 1434; break; + case "host": r.Host = "beta.example.test"; break; + default: throw new ArgumentException(field); + } + } + + [Fact] + public void TheEqualRow_GetsTheFileSecret() + { + var entry = Entry(); + var row = RowLike(entry, null); + + StoreConfigProvider.BackfillSecretFromFile(row, new DarlingConfig { Servers = [entry] }); + + Assert.Equal(FileSecret, row.EncryptedPassword); + } + + [Theory] + [MemberData(nameof(OneFieldDiffers))] + public void ARowThatDiffersInOneField_GetsNoFileSecret(string field) + { + var entry = Entry(); + var row = RowLike(entry, null); + Change(field, row); + + StoreConfigProvider.BackfillSecretFromFile(row, new DarlingConfig { Servers = [entry] }); + + Assert.True(string.IsNullOrEmpty(row.EncryptedPassword), field); + Assert.Null(row.Password); + } + + [Fact] + public void TheEqualRow_GetsTheDeclaredMark() + { + var entry = Entry(); + entry.EncryptedPassword = DeclaredReference; + var row = RowLike(entry, DeclaredReference); + + StoreConfigProvider.MarkSlotsTheFileDeclares(row, new DarlingConfig { Servers = [entry] }); + + Assert.True(row.EncryptedPasswordDeclaredByFile); + } + + [Theory] + [MemberData(nameof(OneFieldDiffers))] + public void ARowThatDiffersInOneField_GetsNoDeclaredMark(string field) + { + var entry = Entry(); + entry.EncryptedPassword = DeclaredReference; + var row = RowLike(entry, DeclaredReference); + Change(field, row); + + StoreConfigProvider.MarkSlotsTheFileDeclares(row, new DarlingConfig { Servers = [entry] }); + + Assert.False(row.EncryptedPasswordDeclaredByFile, field); + } + + [Fact] + public void TheEqualRow_GetsTheRemediationDeclaredMark() + { + var entry = Entry(); + entry.RemediationUsername = "fixer"; + entry.RemediationEncryptedPassword = DeclaredReference; + entry.RemediationEncryptedPasswordDeclaredByFile = true; + var row = RowLike(entry, null); + row.RemediationUsername = "fixer"; + row.RemediationEncryptedPassword = DeclaredReference; + + StoreConfigProvider.MarkSlotsTheFileDeclares(row, new DarlingConfig { Servers = [entry] }); + + Assert.True(row.RemediationEncryptedPasswordDeclaredByFile); + } + + [Theory] + [InlineData("other-fixer")] + [InlineData("")] + [InlineData(null)] + public void ARowWithAnotherRemediationUsername_GetsNoRemediationDeclaredMark(string? rowUsername) + { + /* #5366: the remediation reference is declared for one remediation login; a row that names another (or none) is not it. */ + var entry = Entry(); + entry.RemediationUsername = "fixer"; + entry.RemediationEncryptedPassword = DeclaredReference; + entry.RemediationEncryptedPasswordDeclaredByFile = true; + var row = RowLike(entry, null); + row.RemediationUsername = rowUsername; + row.RemediationEncryptedPassword = DeclaredReference; + + StoreConfigProvider.MarkSlotsTheFileDeclares(row, new DarlingConfig { Servers = [entry] }); + + Assert.False(row.RemediationEncryptedPasswordDeclaredByFile); + } +} diff --git a/Darling/Darling.Tests/StoreConnectionStringAppendPinTests.cs b/Darling/Darling.Tests/StoreConnectionStringAppendPinTests.cs index 58249e91be..60b451d2cb 100644 --- a/Darling/Darling.Tests/StoreConnectionStringAppendPinTests.cs +++ b/Darling/Darling.Tests/StoreConnectionStringAppendPinTests.cs @@ -28,6 +28,7 @@ namespace Darling.Tests; /// including an explicit empty one, must survive byte for byte, and the output must literally start with the /// caller's own string. /// +[Trait("Stage", "Guard")] public sealed class StoreConnectionStringAppendPinTests { /// Each row is independently valid Npgsql connection-string content and exercises one of the diff --git a/Darling/Darling.Tests/StoreCopyPhaseLivePostgresTests.cs b/Darling/Darling.Tests/StoreCopyPhaseLivePostgresTests.cs index f2aa211557..6a40569577 100644 --- a/Darling/Darling.Tests/StoreCopyPhaseLivePostgresTests.cs +++ b/Darling/Darling.Tests/StoreCopyPhaseLivePostgresTests.cs @@ -229,8 +229,8 @@ fault the runner itself raised. */ /// deadline is a bare carrying /// . Remove the bound and this test does not fail /// slowly or ambiguously: it waits for the connection's timeout and then fails on the type. The - /// elapsed range is asserted on both sides as well, so a deadline accidentally written in - /// milliseconds cannot satisfy the type and message alone. + /// elapsed floor is asserted as well, so a deadline accidentally written in + /// milliseconds cannot satisfy the type and message alone (#5602: the ceiling is the type and message). /// /// It really does cost the sweep deadline in wall clock, and that is the price of driving /// the SHIPPED private body rather than a copy: the production call site takes @@ -295,13 +295,17 @@ arrives as an NpgsqlException wrapping one of these and would satisfy an assigna from the re-attempt gate, and would be reported as an orderly stop. */ Assert.IsNotAssignableFrom(fault); - /* It fired on its own clock rather than the connection's. Both sides: the upper bound is what says - Npgsql's timeout did not do this, and the lower bound is what a deadline written in - milliseconds would fail. */ - Assert.InRange( - clock.Elapsed, - StoreCopyStartDeadline.Deadline - TimeSpan.FromSeconds(1), - TimeSpan.FromSeconds(commandTimeoutSeconds)); + /* It fired on its own clock rather than the connection's. #5602: the LOWER bound only, as a clock check. + What a deadline written in milliseconds would fail is this floor, and load can only lengthen an + elapsed time, never shorten it, so it cannot flake. The upper bound this used to carry (the + connection's command timeout) is now held by the work: Npgsql's own timeout arrives as an + NpgsqlException wrapping a TimeoutException, which the exact-type and exact-message assertions above + already reject, so a late-firing deadline on a loaded runner no longer fails and the bug the upper + bound was written for, "the connection's bound ended the stall", still does. */ + Assert.True( + clock.Elapsed >= StoreCopyStartDeadline.Deadline - TimeSpan.FromSeconds(1), + $"The start phase gave up after {clock.Elapsed.TotalSeconds:F1}s, before the sweep deadline " + + $"({StoreCopyStartDeadline.Deadline.TotalSeconds:F0}s) less a second: the deadline is not the sweep deadline."); /* And the consequences the phase axis carries, on this fault: it is a transport fault, it is start-phase, so the gate accepts it — a re-attempt after a start-phase stall is exactly-once and diff --git a/Darling/Darling.Tests/StoreDimensionRowCountLiveTests.cs b/Darling/Darling.Tests/StoreDimensionRowCountLiveTests.cs new file mode 100644 index 0000000000..55b171ed58 --- /dev/null +++ b/Darling/Darling.Tests/StoreDimensionRowCountLiveTests.cs @@ -0,0 +1,134 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: this class mints its own scratch database through ScratchPostgres and never touches another + test's rows, so it is deliberately NOT [Collection("live-postgres")]. */ + +/// +/// #5520: the dimension rows of collect.store_metrics take row_count from pg_class.reltuples, +/// not from a count(*) over the dimension table (149,562 blocks and 25.9 s a call on a large store). The +/// estimate is NULL where PostgreSQL reports -1 (never vacuumed or analysed), like the table kind's, and the +/// scan is gone from the statement. +/// +public sealed class StoreDimensionRowCountLiveTests +{ + private const string SkipText = "Set DARLING_TEST_PG to a Postgres connection string to run the dimension row_count pins (each mints its own scratch database)."; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + [Fact] + public void TheDimensionSweep_TakesRowCountFromReltuples_AndNeverScansTheDimension() + { + var sql = StoreSelfMetrics.DimensionInsertSql; + + Assert.DoesNotContain("count(*)", sql, StringComparison.OrdinalIgnoreCase); + foreach (var dim in new[] { PayloadDimensions.QueryTextDimTable, PayloadDimensions.QueryPlanDimTable }) + { + /* -1 (never vacuumed or analysed, PostgreSQL 14+) is NULL, never a count of minus one. */ + Assert.Contains( + $"(SELECT CASE WHEN c.reltuples >= 0 THEN c.reltuples::bigint END FROM pg_class AS c WHERE c.oid = 'collect.{dim}'::regclass),", + sql, StringComparison.Ordinal); + Assert.DoesNotContain($"FROM collect.{dim})", sql, StringComparison.Ordinal); + } + } + + [Fact] + public async Task TheDimensionRowCount_IsTheAnalyzedEstimate_AndNullWhereTheTableWasNeverAnalyzed() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + var bodySucceeded = false; + try + { + await ExecAsync(connection, $"TRUNCATE collect.{PayloadDimensions.QueryTextDimTable}, collect.{PayloadDimensions.QueryPlanDimTable}", ct); + for (var i = 0; i < 5; i++) + { + await ExecAsync(connection, + $"INSERT INTO collect.{PayloadDimensions.QueryTextDimTable} ({PayloadDimensions.DigestColumn}, {PayloadDimensions.PayloadColumnOf(PayloadDimensions.QueryTextDimTable)}, {PayloadDimensions.LastSeenColumn}) " + + $"VALUES (decode('{i:x2}', 'hex'), 'text {i}', now() AT TIME ZONE 'UTC')", ct); + } + + for (var i = 0; i < 3; i++) + { + await ExecAsync(connection, + $"INSERT INTO collect.{PayloadDimensions.QueryPlanDimTable} ({PayloadDimensions.DigestColumn}, {PayloadDimensions.PayloadColumnOf(PayloadDimensions.QueryPlanDimTable)}, {PayloadDimensions.LastSeenColumn}) " + + $"VALUES (decode('{i:x2}', 'hex'), '', now() AT TIME ZONE 'UTC')", ct); + } + + /* Never analysed: reltuples is -1 on PostgreSQL 14+, so the figure is NULL. An autovacuum that got there + first would have set it, so the expectation follows the catalog rather than assuming it. */ + var beforeAnalyze = await SweepAsync(connection, DateTime.UtcNow, ct); + foreach (var dim in new[] { PayloadDimensions.QueryTextDimTable, PayloadDimensions.QueryPlanDimTable }) + { + var reltuples = await ReltuplesAsync(connection, dim, ct); + Assert.Equal(reltuples >= 0 ? (long?)reltuples : null, beforeAnalyze[dim]); + } + + /* Analysed: the estimate of a table this small is its row count. */ + await ExecAsync(connection, $"ANALYZE collect.{PayloadDimensions.QueryTextDimTable}", ct); + await ExecAsync(connection, $"ANALYZE collect.{PayloadDimensions.QueryPlanDimTable}", ct); + var afterAnalyze = await SweepAsync(connection, DateTime.UtcNow.AddMinutes(1), ct); + Assert.Equal(5L, afterAnalyze[PayloadDimensions.QueryTextDimTable]); + Assert.Equal(3L, afterAnalyze[PayloadDimensions.QueryPlanDimTable]); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + private static async Task> SweepAsync(NpgsqlConnection connection, DateTime metricTimeUtc, CancellationToken ct) + { + var metricTime = DateTime.SpecifyKind(new DateTime(metricTimeUtc.Ticks - (metricTimeUtc.Ticks % TimeSpan.TicksPerSecond)), DateTimeKind.Unspecified); + await using (var sweep = new NpgsqlCommand(StoreSelfMetrics.DimensionInsertSql, connection)) + { + sweep.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = metricTime }); + Assert.Equal(2, await sweep.ExecuteNonQueryAsync(ct)); + } + + var rows = new Dictionary(); + await using var read = new NpgsqlCommand( + "SELECT object_name, row_count FROM collect.store_metrics WHERE object_kind = 'dimension' AND metric_time = $1", connection); + read.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = metricTime }); + await using var reader = await read.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + rows[reader.GetString(0)] = reader.IsDBNull(1) ? null : reader.GetInt64(1); + } + + return rows; + } + + private static async Task ReltuplesAsync(NpgsqlConnection connection, string dim, CancellationToken ct) + { + await using var cmd = new NpgsqlCommand($"SELECT reltuples::float8 FROM pg_class WHERE oid = 'collect.{dim}'::regclass", connection); + return (double)(await cmd.ExecuteScalarAsync(ct))!; + } + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var cmd = new NpgsqlCommand(sql, connection); + await cmd.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/StoreInstallIdBindingRuleTests.cs b/Darling/Darling.Tests/StoreInstallIdBindingRuleTests.cs new file mode 100644 index 0000000000..c769223fc1 --- /dev/null +++ b/Darling/Darling.Tests/StoreInstallIdBindingRuleTests.cs @@ -0,0 +1,71 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// The rule that says whether a stored install id row belongs to the store it is read from (#4961), as a truth table. +/// A managed major upgrade makes a new cluster id, keeps the database's OID and the install id table's OID, and raises the +/// server's major version. So a row that has its table OID keeps its id through a changed cluster id only when both OIDs +/// match and the major rose: a copy of the row into a fresh install of the same version also matches both OIDs (a new +/// cluster numbers its objects from the same start) and has a new cluster id, but the major did not rise, so it gets a new +/// id. A row made before the table OID was kept has none, and is compared as it was before: by the database's OID, and by +/// the cluster id when the row and the current read both have one. +/// +public sealed class StoreInstallIdBindingRuleTests +{ + // stored cluster id, database OID, table OID, major; current cluster id, database OID, table OID, major; same store? + [Theory] + /* The same id: nothing changed. */ + [InlineData(1L, 5L, 7L, 16, 1L, 5L, 7L, 16, true)] + /* A major upgrade: new cluster id, both OIDs kept, the major rose. */ + [InlineData(1L, 5L, 7L, 16, 2L, 5L, 7L, 17, true)] + [InlineData(1L, 5L, 7L, 15, 2L, 5L, 7L, 17, true)] // two majors in one step is still a rise + /* A copy of the row into a fresh install: new cluster id, both OIDs match, but the major did not rise. */ + [InlineData(1L, 5L, 7L, 17, 2L, 5L, 7L, 17, false)] // the same major + [InlineData(1L, 5L, 7L, 17, 2L, 5L, 7L, 16, false)] // a lower major + /* The OIDs always count, with or without a changed cluster id. */ + [InlineData(1L, 5L, 7L, 16, 1L, 5L, 8L, 16, false)] // another table OID + [InlineData(1L, 5L, 7L, 16, 1L, 6L, 7L, 16, false)] // another database OID + [InlineData(1L, 5L, 7L, 16, 2L, 5L, 8L, 17, false)] // shaped like an upgrade, but the table OID changed + [InlineData(1L, 5L, 7L, 16, 2L, 6L, 7L, 17, false)] // shaped like an upgrade, but the database OID changed + /* A row with its table OID and no major (the column was added to a store that already had rows): the cluster id must match. */ + [InlineData(1L, 5L, 7L, null, 1L, 5L, 7L, 17, true)] + [InlineData(1L, 5L, 7L, null, 2L, 5L, 7L, 17, false)] // a changed cluster id with no stored major cannot be shown to be an upgrade + /* A row made before the table OID was kept (no table OID, no major): the old rule, the cluster id counting when both sides have one. */ + [InlineData(1L, 5L, null, null, 1L, 5L, 7L, 17, true)] + [InlineData(1L, 5L, null, null, 2L, 5L, 7L, 17, false)] // another cluster id is another store, as before + [InlineData(1L, 5L, null, null, 2L, 5L, 7L, 16, false)] + [InlineData(1L, 5L, null, null, 1L, 6L, 7L, 17, false)] // the database's OID always counts + [InlineData(1L, 5L, null, 16, 2L, 5L, 7L, 17, false)] // the table OID decides which rule applies, not the major + /* An unknown cluster id, on either side or both: the two OIDs decide, so a grant that changes later never makes a new id. */ + [InlineData(null, 5L, 7L, 16, 2L, 5L, 7L, 16, true)] + [InlineData(1L, 5L, 7L, 16, null, 5L, 7L, 16, true)] + [InlineData(null, 5L, 7L, 16, null, 5L, 7L, 16, true)] + [InlineData(null, 5L, 7L, null, 2L, 5L, 7L, 17, true)] + [InlineData(1L, 5L, 7L, null, null, 5L, 7L, 17, true)] + [InlineData(null, 5L, 7L, 17, null, 5L, 7L, 16, true)] // no cluster id to have changed, so the major is no reason + [InlineData(null, 5L, 7L, 16, null, 5L, 8L, 16, false)] // another table OID, and no cluster id to say otherwise + [InlineData(null, 5L, 7L, 16, null, 6L, 7L, 16, false)] // another database OID + [InlineData(null, 5L, null, null, 2L, 5L, 7L, 17, true)] + [InlineData(1L, 5L, null, null, null, 5L, 7L, 17, true)] + [InlineData(null, 5L, null, null, null, 5L, 7L, 17, true)] + [InlineData(null, 5L, null, null, null, 6L, 7L, 17, false)] + public void TheRuleIsTheTwoOidsAndThenTheMajor_AndTheOldRuleForARowWithoutATableOid( + long? storedCluster, long storedDatabase, long? storedTable, int? storedMajor, + long? currentCluster, long currentDatabase, long currentTable, int currentMajor, + bool expectedSame) + { + Assert.Equal(expectedSame, StoreInstallId.IsSameStore( + storedCluster, storedDatabase, storedTable, storedMajor, + currentCluster, currentDatabase, currentTable, currentMajor)); + } +} diff --git a/Darling/Darling.Tests/StoreInstallIdLiveTests.cs b/Darling/Darling.Tests/StoreInstallIdLiveTests.cs new file mode 100644 index 0000000000..63f2745dae --- /dev/null +++ b/Darling/Darling.Tests/StoreInstallIdLiveTests.cs @@ -0,0 +1,797 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: every fact runs in the one scratch database the fixture mints through ScratchPostgres, resets + the install id table to its freshly migrated shape first, and never touches another test's rows, so the class is + deliberately NOT [Collection("live-postgres")]. */ + +/// The scratch store behind : a database of its own, migrated once +/// through the real ladder, dropped when the class is done. Nothing here enables TimescaleDB, so there is no +/// scheduler to race. +public sealed class InstallIdScratchStore : IAsyncLifetime +{ + private ScratchPostgres? _scratch; + + /// The scratch store's connection string, or null when DARLING_TEST_PG is unset (the normal + /// ungated run), in which case every fact skips. + public string? ConnectionString => _scratch?.ConnectionString; + + public async ValueTask InitializeAsync() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + if (string.IsNullOrEmpty(baseConnectionString)) + { + return; + } + + _scratch = await ScratchPostgres.CreateAsync(baseConnectionString, CancellationToken.None); + await using var connection = new NpgsqlConnection(_scratch.ConnectionString); + await connection.OpenAsync(CancellationToken.None); + await PgMigrations.MigrateAsync(connection, CancellationToken.None); + } + + public async ValueTask DisposeAsync() + { + if (_scratch is not null) + { + await _scratch.DisposeAsync(); + } + } +} + +/// +/// The install id store against a real PostgreSQL (#4961). The service makes the id's row at start and the CLI and +/// the Viewer only read it, so what matters is that the make is safe to run twice at once, that a stored id which +/// does not belong to this store (a mismatched binding, or a value that is not eight lowercase hex digits) is +/// replaced rather than trusted, and that the reader writes nothing. +/// +public sealed class StoreInstallIdLiveTests : IClassFixture +{ + private const string Table = "config.config_install_id"; + + private static readonly string[] BadIds = + { + "ABCDEF12", // upper case: on a case-sensitive collation it would name a different session + "abcdef1", // one short + "abcdef123", // one long + "abcdefgh", // not hex + "abcdef12\n", // a trailing line ending: a `$` anchor in some engines would let it through + " abcdef12", // padding + }; + + private readonly InstallIdScratchStore _store; + + public StoreInstallIdLiveTests(InstallIdScratchStore store) + { + _store = store; + } + + public static TheoryData BadIdData() => new(BadIds); + + private string ConnectionString + { + get + { + Assert.SkipWhen(string.IsNullOrEmpty(_store.ConnectionString), "Set DARLING_TEST_PG to run the live install id store pins."); + return _store.ConnectionString!; + } + } + + private static string RungSql => PgMigrations.Scripts.Single(m => m.Name == InstallIdRungTests.RungName).Sql; + + private static string TableOidRungSql => PgMigrations.Scripts.Single(m => m.Name == InstallIdTableOidRungTests.RungName).Sql; + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, CancellationToken ct, params object[] parameters) + { + await using var command = new NpgsqlCommand(sql, connection); + foreach (var parameter in parameters) + { + command.Parameters.Add(new NpgsqlParameter { Value = parameter }); + } + + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task ScalarAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + return (T)(await command.ExecuteScalarAsync(ct))!; + } + + /// The table as the rung leaves it: dropped and made again from the product's own SQL. + private async Task ResetAsync(CancellationToken ct) + { + var connection = new NpgsqlConnection(ConnectionString); + await connection.OpenAsync(ct); + await ExecAsync(connection, $"DROP TABLE IF EXISTS {Table}", ct); + await ExecAsync(connection, RungSql, ct); + await ExecAsync(connection, TableOidRungSql, ct); + return connection; + } + + private static async Task<(string InstallId, long? SystemIdentifier, long DatabaseOid, long? TableOid, int? ServerMajor)> ReadRowAsync(NpgsqlConnection connection, CancellationToken ct) + { + await using var command = new NpgsqlCommand($"SELECT install_id, system_identifier, database_oid, table_oid, server_major FROM {Table} WHERE id = 1", connection); + await using var reader = await command.ExecuteReaderAsync(ct); + Assert.True(await reader.ReadAsync(ct), "the install id table has no row"); + return (reader.GetString(0), reader.IsDBNull(1) ? null : reader.GetInt64(1), reader.GetInt64(2), + reader.IsDBNull(3) ? null : reader.GetInt64(3), reader.IsDBNull(4) ? null : reader.GetInt32(4)); + } + + /// The binding this store really has, read the way the product's make reads it. + private static async Task<(long SystemIdentifier, long DatabaseOid, long TableOid, int ServerMajor)> RealBindingAsync(NpgsqlConnection connection, CancellationToken ct) + { + await using var command = new NpgsqlCommand( + "SELECT s.system_identifier, d.oid::bigint, 'config.config_install_id'::regclass::oid::bigint, current_setting('server_version_num')::int / 10000 FROM pg_control_system() AS s CROSS JOIN pg_database AS d WHERE d.datname = current_database()", + connection); + await using var reader = await command.ExecuteReaderAsync(ct); + Assert.True(await reader.ReadAsync(ct)); + return (reader.GetInt64(0), reader.GetInt64(1), reader.GetInt64(2), reader.GetInt32(3)); + } + + [Fact] + public async Task Ensure_OnAnEmptyTable_MakesOneValidIdBoundToThisStore_AndLogsNoWarning() + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + + var logger = new CapturingTestLogger(); + var id = await StoreInstallId.EnsureAsync(connection, logger, ct); + + Assert.True(InstallId.IsValid(id), $"'{id}' is not eight lowercase hex digits"); + Assert.Equal(1L, await ScalarAsync(connection, $"SELECT count(*) FROM {Table}", ct)); + var row = await ReadRowAsync(connection, ct); + Assert.Equal(id, row.InstallId); + var binding = await RealBindingAsync(connection, ct); + Assert.Equal(binding.SystemIdentifier, row.SystemIdentifier); + Assert.Equal(binding.DatabaseOid, row.DatabaseOid); + Assert.Equal(binding.TableOid, row.TableOid); + Assert.Equal(binding.ServerMajor, row.ServerMajor); + Assert.Equal(0, logger.CountAtLevel(LogLevel.Warning)); + + /* A start after that one finds the row and keeps the id: nothing changes and nothing is logged. */ + var again = new CapturingTestLogger(); + Assert.Equal(id, await StoreInstallId.EnsureAsync(connection, again, ct)); + Assert.Equal(0, again.CountAtLevel(LogLevel.Warning)); + Assert.Equal(1L, await ScalarAsync(connection, $"SELECT count(*) FROM {Table}", ct)); + } + + [Fact] + public async Task Ensure_ConcurrentStartsOnOneStore_MakeOneRowAndTheSameId() + { + var ct = TestContext.Current.CancellationToken; + await using var setup = await ResetAsync(ct); + + const int starts = 12; + var gate = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var loggers = Enumerable.Range(0, starts).Select(_ => new CapturingTestLogger()).ToArray(); + var tasks = loggers.Select(async logger => + { + await using var connection = new NpgsqlConnection(ConnectionString); + await connection.OpenAsync(ct); + await gate.Task; + return await StoreInstallId.EnsureAsync(connection, logger, ct); + }).ToArray(); + + gate.SetResult(); + var ids = await Task.WhenAll(tasks); + + Assert.Single(ids.Distinct()); + Assert.True(InstallId.IsValid(ids[0])); + Assert.Equal(1L, await ScalarAsync(setup, $"SELECT count(*) FROM {Table}", ct)); + Assert.Equal(ids[0], (await ReadRowAsync(setup, ct)).InstallId); + Assert.All(loggers, logger => Assert.Equal(0, logger.CountAtLevel(LogLevel.Warning))); + } + + [Theory] + [InlineData("database_oid", "database OID")] + [InlineData("table_oid", "table's OID")] + public async Task Ensure_AMismatchedBinding_MakesANewId_AndLogsOneWarningNamingBothIdsAndTheBindingThatDiffered(string boundColumn, string namedBinding) + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + + var oldId = await StoreInstallId.EnsureAsync(connection, new CapturingTestLogger(), ct); + await ExecAsync(connection, $"UPDATE {Table} SET {boundColumn} = {boundColumn} + 1", ct); + + var logger = new CapturingTestLogger(); + var newId = await StoreInstallId.EnsureAsync(connection, logger, ct); + + Assert.True(InstallId.IsValid(newId)); + Assert.NotEqual(oldId, newId); + var warning = Assert.Single(logger.Lines, l => l.StartsWith("Warning:", StringComparison.Ordinal)); + Assert.Contains(oldId, warning, StringComparison.Ordinal); + Assert.Contains(newId, warning, StringComparison.Ordinal); + Assert.Contains(namedBinding, warning, StringComparison.Ordinal); + + var row = await ReadRowAsync(connection, ct); + var binding = await RealBindingAsync(connection, ct); + Assert.Equal(newId, row.InstallId); + Assert.Equal(binding.SystemIdentifier, row.SystemIdentifier); + Assert.Equal(binding.DatabaseOid, row.DatabaseOid); + Assert.Equal(binding.TableOid, row.TableOid); + Assert.Equal(binding.ServerMajor, row.ServerMajor); + + /* The next start is an ordinary one: same id, no second warning. */ + var after = new CapturingTestLogger(); + Assert.Equal(newId, await StoreInstallId.EnsureAsync(connection, after, ct)); + Assert.Equal(0, after.CountAtLevel(LogLevel.Warning)); + } + + [Theory] + [MemberData(nameof(BadIdData))] + public async Task Ensure_ABadStoredId_MakesANewId_AndLogsOneWarningNamingBothIds(string badId) + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + + /* The CHECK is what keeps a bad value out, so a fact that plants one has to take it away first (in a + store of its own, so nothing else sees the table without it). The binding stays the real one: only + the id is wrong. */ + await ExecAsync(connection, $"ALTER TABLE {Table} DROP CONSTRAINT ck_config_install_id_format", ct); + var binding = await RealBindingAsync(connection, ct); + await ExecAsync(connection, + $"INSERT INTO {Table} (id, install_id, system_identifier, database_oid) VALUES (1, $1, $2, $3)", ct, + badId, binding.SystemIdentifier, binding.DatabaseOid); + + var logger = new CapturingTestLogger(); + var newId = await StoreInstallId.EnsureAsync(connection, logger, ct); + + Assert.True(InstallId.IsValid(newId)); + Assert.NotEqual(badId, newId); + var warning = Assert.Single(logger.Lines, l => l.StartsWith("Warning:", StringComparison.Ordinal)); + Assert.Contains(badId, warning, StringComparison.Ordinal); + Assert.Contains(newId, warning, StringComparison.Ordinal); + Assert.Equal(newId, (await ReadRowAsync(connection, ct)).InstallId); + } + + [Fact] + public async Task Ensure_ConcurrentStartsOnAMismatchedRow_MakeOneNewId_AndLogOneWarningInTotal() + { + var ct = TestContext.Current.CancellationToken; + await using var setup = await ResetAsync(ct); + + var oldId = await StoreInstallId.EnsureAsync(setup, new CapturingTestLogger(), ct); + await ExecAsync(setup, $"UPDATE {Table} SET table_oid = table_oid + 1", ct); + + const int starts = 12; + var gate = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var loggers = Enumerable.Range(0, starts).Select(_ => new CapturingTestLogger()).ToArray(); + var tasks = loggers.Select(async logger => + { + await using var connection = new NpgsqlConnection(ConnectionString); + await connection.OpenAsync(ct); + await gate.Task; + return await StoreInstallId.EnsureAsync(connection, logger, ct); + }).ToArray(); + + gate.SetResult(); + var ids = await Task.WhenAll(tasks); + + var newId = Assert.Single(ids.Distinct()); + Assert.NotEqual(oldId, newId); + Assert.True(InstallId.IsValid(newId)); + Assert.Equal(1, loggers.Sum(logger => logger.CountAtLevel(LogLevel.Warning))); + Assert.Equal(1L, await ScalarAsync(setup, $"SELECT count(*) FROM {Table}", ct)); + Assert.Equal(newId, (await ReadRowAsync(setup, ct)).InstallId); + } + + [Theory] + [MemberData(nameof(BadIdData))] + public async Task TheTable_RefusesAnIdThatIsNotEightLowercaseHexDigits(string badId) + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + var binding = await RealBindingAsync(connection, ct); + + var ex = await Assert.ThrowsAsync(() => ExecAsync(connection, + $"INSERT INTO {Table} (id, install_id, system_identifier, database_oid) VALUES (1, $1, $2, $3)", ct, + badId, binding.SystemIdentifier, binding.DatabaseOid)); + + Assert.Equal("23514", ex.SqlState); + Assert.Equal("ck_config_install_id_format", ex.ConstraintName); + } + + [Fact] + public async Task TheTable_HoldsOneRowOnly() + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + var binding = await RealBindingAsync(connection, ct); + + var ex = await Assert.ThrowsAsync(() => ExecAsync(connection, + $"INSERT INTO {Table} (id, install_id, system_identifier, database_oid) VALUES (2, 'abcdef12', $1, $2)", ct, + binding.SystemIdentifier, binding.DatabaseOid)); + + Assert.Equal("23514", ex.SqlState); + } + + [Fact] + public async Task TryRead_ReadsTheStoredId_AndNeverWrites() + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + + /* No row yet: nothing comes back and the read makes none. */ + Assert.Null(await StoreInstallId.TryReadAsync(connection, ct)); + Assert.Equal(0L, await ScalarAsync(connection, $"SELECT count(*) FROM {Table}", ct)); + + var id = await StoreInstallId.EnsureAsync(connection, new CapturingTestLogger(), ct); + var before = await ScalarAsync(connection, $"SELECT xmin::text FROM {Table}", ct); + + Assert.Equal(id, await StoreInstallId.TryReadAsync(connection, ct)); + await using (var dataSource = NpgsqlDataSource.Create(ConnectionString)) + { + Assert.Equal(id, await StoreInstallId.TryReadAsync(dataSource, ct)); + } + + /* A read leaves the row exactly as it found it (a write would have given it a new xmin). */ + Assert.Equal(before, await ScalarAsync(connection, $"SELECT xmin::text FROM {Table}", ct)); + } + + /// The CLI can read a store before the upgraded service has migrated it: a store still at the rung below + /// the table OID has neither that column nor the major's, and the reader's statement must not name either. + [Fact] + public async Task TryRead_OnAStoreStillBelowTheTableOidRung_ReadsTheId_AndWritesNothing() + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + var id = await StoreInstallId.EnsureAsync(connection, new CapturingTestLogger(), ct); + + await ExecAsync(connection, $"ALTER TABLE {Table} DROP COLUMN table_oid, DROP COLUMN server_major", ct); + var before = await ScalarAsync(connection, $"SELECT xmin::text FROM {Table}", ct); + + Assert.Equal(id, await StoreInstallId.TryReadAsync(connection, ct)); + await using (var dataSource = NpgsqlDataSource.Create(ConnectionString)) + { + Assert.Equal(id, await StoreInstallId.TryReadAsync(dataSource, ct)); + } + + Assert.Equal(before, await ScalarAsync(connection, $"SELECT xmin::text FROM {Table}", ct)); + Assert.DoesNotContain("table_oid", StoreInstallId.ReadSql, StringComparison.Ordinal); + Assert.DoesNotContain("server_major", StoreInstallId.ReadSql, StringComparison.Ordinal); + } + + [Fact] + public async Task TryRead_OnAStoreWithoutTheTable_IsNull_AndOnABadStoredValue_IsNull_WithoutRepairingIt() + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + + await ExecAsync(connection, $"DROP TABLE {Table}", ct); + Assert.Null(await StoreInstallId.TryReadAsync(connection, ct)); + + await ExecAsync(connection, RungSql, ct); + await ExecAsync(connection, $"ALTER TABLE {Table} DROP CONSTRAINT ck_config_install_id_format", ct); + var binding = await RealBindingAsync(connection, ct); + await ExecAsync(connection, + $"INSERT INTO {Table} (id, install_id, system_identifier, database_oid) VALUES (1, 'ABCDEF12', $1, $2)", ct, + binding.SystemIdentifier, binding.DatabaseOid); + + Assert.Null(await StoreInstallId.TryReadAsync(connection, ct)); + Assert.Equal("ABCDEF12", await ScalarAsync(connection, $"SELECT install_id FROM {Table} WHERE id = 1", ct)); + } + + /// + /// The race-safety rule at the statement itself, deterministically: two starts that read the same row both try to + /// replace it, and only the first one's UPDATE finds the id it read. The timing-dependent fact above can pass + /// when one start finishes before the others begin, so this one pins the guard without depending on timing. + /// + [Fact] + public async Task TheReplacement_IsGuardedByTheIdTheStartRead_SoOnlyOneOfTwoStartsThatSawTheSameRowMakesTheNewId() + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + var staleId = await StoreInstallId.EnsureAsync(connection, new CapturingTestLogger(), ct); + var binding = await RealBindingAsync(connection, ct); + + async Task ReplaceAsync(string newId) + { + await using var command = new NpgsqlCommand(StoreInstallId.ReplaceSql, connection); + command.Parameters.Add(new NpgsqlParameter { Value = newId }); + command.Parameters.Add(new NpgsqlParameter { Value = binding.SystemIdentifier }); + command.Parameters.Add(new NpgsqlParameter { Value = binding.DatabaseOid }); + command.Parameters.Add(new NpgsqlParameter { Value = binding.TableOid }); + command.Parameters.Add(new NpgsqlParameter { Value = binding.ServerMajor }); + command.Parameters.Add(new NpgsqlParameter { Value = staleId }); + return await command.ExecuteNonQueryAsync(ct); + } + + Assert.Equal(1, await ReplaceAsync("aaaaaaaa")); + Assert.Equal(0, await ReplaceAsync("bbbbbbbb")); + Assert.Equal("aaaaaaaa", (await ReadRowAsync(connection, ct)).InstallId); + } + + /// + /// The rebind's guard at the statement itself: it matches a row exactly when it would change something in it, so of + /// several starts that saw the same row exactly one writes and logs. When the cluster id is not known, a row that has + /// one keeps its major, so the statement finds nothing to write although the major differs (0 rows); a row that has + /// none takes the major (1 row, then none left), and a row that has none of its table OID takes that and keeps the + /// major. When the cluster id is known the row takes it and the major together. + /// + [Fact] + public async Task TheRebind_MatchesARowExactlyWhenItWouldChangeSomethingInIt_AndKeepsTheMajorOfARowThatHasAClusterIdWhenNoneIsKnown() + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + var id = await StoreInstallId.EnsureAsync(connection, new CapturingTestLogger(), ct); + var binding = await RealBindingAsync(connection, ct); + var oldCluster = binding.SystemIdentifier + 1; + var oldMajor = binding.ServerMajor - 1; + + async Task RebindAsync(long? knownCluster) + { + await using var command = new NpgsqlCommand(StoreInstallId.RebindSql, connection); + command.Parameters.Add(new NpgsqlParameter { Value = binding.TableOid }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Bigint, Value = (object?)knownCluster ?? DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter { Value = binding.ServerMajor }); + command.Parameters.Add(new NpgsqlParameter { Value = id }); + return await command.ExecuteNonQueryAsync(ct); + } + + /* A row that has a cluster id, with none known: nothing to write, although the major differs. */ + await ExecAsync(connection, $"UPDATE {Table} SET system_identifier = {oldCluster}, server_major = {oldMajor}", ct); + Assert.Equal(0, await RebindAsync(null)); + var row = await ReadRowAsync(connection, ct); + Assert.Equal(oldCluster, row.SystemIdentifier); + Assert.Equal(oldMajor, row.ServerMajor); + + /* The same row with the cluster id known: both are written, once. */ + Assert.Equal(1, await RebindAsync(binding.SystemIdentifier)); + row = await ReadRowAsync(connection, ct); + Assert.Equal(binding.SystemIdentifier, row.SystemIdentifier); + Assert.Equal(binding.ServerMajor, row.ServerMajor); + Assert.Equal(0, await RebindAsync(binding.SystemIdentifier)); + + /* A row with no cluster id takes the major although none is known, once. */ + await ExecAsync(connection, $"UPDATE {Table} SET system_identifier = NULL, server_major = {oldMajor}", ct); + Assert.Equal(1, await RebindAsync(null)); + row = await ReadRowAsync(connection, ct); + Assert.Null(row.SystemIdentifier); + Assert.Equal(binding.ServerMajor, row.ServerMajor); + Assert.Equal(0, await RebindAsync(null)); + + /* A row that has a cluster id and no table OID: the table OID is written, and the major stays. */ + await ExecAsync(connection, $"UPDATE {Table} SET table_oid = NULL, system_identifier = {oldCluster}, server_major = {oldMajor}", ct); + Assert.Equal(1, await RebindAsync(null)); + row = await ReadRowAsync(connection, ct); + Assert.Equal(binding.TableOid, row.TableOid); + Assert.Equal(oldCluster, row.SystemIdentifier); + Assert.Equal(oldMajor, row.ServerMajor); + Assert.Equal(0, await RebindAsync(null)); + } + + /// A major upgrade makes a new cluster id, keeps both OIDs and raises the major (the row here is the one the + /// older version wrote: its cluster id, and a lower major): the id stays, the row is rebound to the new cluster id and + /// the current major, and one Information line names both cluster ids and both majors. Nothing is a Warning, and a + /// start after it changes nothing. + [Fact] + public async Task Ensure_AChangedClusterIdWithAHigherMajor_KeepsTheId_RebindsIt_AndLogsOneInformationNamingBothClusterIds() + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + var id = await StoreInstallId.EnsureAsync(connection, new CapturingTestLogger(), ct); + var binding = await RealBindingAsync(connection, ct); + var oldCluster = binding.SystemIdentifier + 1; + var oldMajor = binding.ServerMajor - 1; + await ExecAsync(connection, $"UPDATE {Table} SET system_identifier = {oldCluster}, server_major = {oldMajor}", ct); + + var logger = new CapturingTestLogger(); + Assert.Equal(id, await StoreInstallId.EnsureAsync(connection, logger, ct)); + + Assert.Equal(0, logger.CountAtLevel(LogLevel.Warning)); + var line = Assert.Single(logger.Lines, l => l.StartsWith("Information:", StringComparison.Ordinal)); + Assert.Contains(oldCluster.ToString(System.Globalization.CultureInfo.InvariantCulture), line, StringComparison.Ordinal); + Assert.Contains(binding.SystemIdentifier.ToString(System.Globalization.CultureInfo.InvariantCulture), line, StringComparison.Ordinal); + Assert.Contains($"from {oldMajor} to {binding.ServerMajor}", line, StringComparison.Ordinal); + Assert.Contains("id stays", line, StringComparison.Ordinal); + var row = await ReadRowAsync(connection, ct); + Assert.Equal(id, row.InstallId); + Assert.Equal(binding.SystemIdentifier, row.SystemIdentifier); + Assert.Equal(binding.TableOid, row.TableOid); + Assert.Equal(binding.ServerMajor, row.ServerMajor); + + var before = await ScalarAsync(connection, $"SELECT xmin::text FROM {Table}", ct); + var after = new CapturingTestLogger(); + Assert.Equal(id, await StoreInstallId.EnsureAsync(connection, after, ct)); + Assert.Empty(after.Lines); + Assert.Equal(before, await ScalarAsync(connection, $"SELECT xmin::text FROM {Table}", ct)); + } + + /// A changed cluster id with both OIDs matching and no rise in the major is not an upgrade. It is what a copy of + /// the row into a fresh install of the same version looks like (a fresh cluster numbers its objects from the same start, + /// so the OIDs can match), or into an install of a lower one, and the copy gets an id of its own: one Warning naming both + /// ids, the cluster ids and the majors, and a row bound to this store. + [Theory] + [InlineData(0)] // the stored major is this server's: the same major + [InlineData(1)] // the stored major is higher than this server's: a lower major + public async Task Ensure_AChangedClusterIdWithoutAHigherMajor_MakesANewId_AndLogsOneWarningNamingTheClusterIdsAndMajors(int storedMajorAbove) + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + var oldId = await StoreInstallId.EnsureAsync(connection, new CapturingTestLogger(), ct); + var binding = await RealBindingAsync(connection, ct); + var oldCluster = binding.SystemIdentifier + 1; + var oldMajor = binding.ServerMajor + storedMajorAbove; + await ExecAsync(connection, $"UPDATE {Table} SET system_identifier = {oldCluster}, server_major = {oldMajor}", ct); + + var logger = new CapturingTestLogger(); + var newId = await StoreInstallId.EnsureAsync(connection, logger, ct); + + Assert.True(InstallId.IsValid(newId)); + Assert.NotEqual(oldId, newId); + Assert.Equal(0, logger.Lines.Count(l => l.StartsWith("Information:", StringComparison.Ordinal))); + var warning = Assert.Single(logger.Lines, l => l.StartsWith("Warning:", StringComparison.Ordinal)); + Assert.Contains(oldId, warning, StringComparison.Ordinal); + Assert.Contains(newId, warning, StringComparison.Ordinal); + Assert.Contains("cluster id", warning, StringComparison.Ordinal); + Assert.Contains(oldCluster.ToString(System.Globalization.CultureInfo.InvariantCulture), warning, StringComparison.Ordinal); + Assert.Contains("major version", warning, StringComparison.Ordinal); + + var row = await ReadRowAsync(connection, ct); + Assert.Equal(newId, row.InstallId); + Assert.Equal(binding.SystemIdentifier, row.SystemIdentifier); + Assert.Equal(binding.TableOid, row.TableOid); + Assert.Equal(binding.ServerMajor, row.ServerMajor); + + var after = new CapturingTestLogger(); + Assert.Equal(newId, await StoreInstallId.EnsureAsync(connection, after, ct)); + Assert.Empty(after.Lines); + } + + /// Several starts that all see the changed cluster id and a lower stored major end on the same id, and one of + /// them says so. + [Fact] + public async Task Ensure_ConcurrentStartsOnAChangedClusterIdWithAHigherMajor_KeepTheId_AndLogOneInformationInTotal() + { + var ct = TestContext.Current.CancellationToken; + await using var setup = await ResetAsync(ct); + var id = await StoreInstallId.EnsureAsync(setup, new CapturingTestLogger(), ct); + await ExecAsync(setup, $"UPDATE {Table} SET system_identifier = system_identifier + 1, server_major = server_major - 1", ct); + + const int starts = 12; + var gate = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var loggers = Enumerable.Range(0, starts).Select(_ => new CapturingTestLogger()).ToArray(); + var tasks = loggers.Select(async logger => + { + await using var connection = new NpgsqlConnection(ConnectionString); + await connection.OpenAsync(ct); + await gate.Task; + return await StoreInstallId.EnsureAsync(connection, logger, ct); + }).ToArray(); + + gate.SetResult(); + var ids = await Task.WhenAll(tasks); + + Assert.Equal(id, Assert.Single(ids.Distinct())); + Assert.Equal(0, loggers.Sum(logger => logger.CountAtLevel(LogLevel.Warning))); + Assert.Equal(1, loggers.Sum(logger => logger.Lines.Count(l => l.StartsWith("Information:", StringComparison.Ordinal)))); + var binding = await RealBindingAsync(setup, ct); + var row = await ReadRowAsync(setup, ct); + Assert.Equal(binding.SystemIdentifier, row.SystemIdentifier); + Assert.Equal(binding.ServerMajor, row.ServerMajor); + } + + /// A stored cluster id of NULL (made while the login was refused it) is filled in once the current one is + /// known, and the id stays. The cluster id was unknown, so nothing says it changed and the two OIDs decide, as they + /// did before the major was kept. + [Fact] + public async Task Ensure_AStoredNullClusterId_IsFilledIn_AndKeepsTheId() + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + var id = await StoreInstallId.EnsureAsync(connection, new CapturingTestLogger(), ct); + await ExecAsync(connection, $"UPDATE {Table} SET system_identifier = NULL", ct); + + var logger = new CapturingTestLogger(); + Assert.Equal(id, await StoreInstallId.EnsureAsync(connection, logger, ct)); + + Assert.Equal(0, logger.CountAtLevel(LogLevel.Warning)); + Assert.Equal((await RealBindingAsync(connection, ct)).SystemIdentifier, (await ReadRowAsync(connection, ct)).SystemIdentifier); + } + + /// An unknown cluster id stays unknown to the rule whatever the stored major is: a row stored with a NULL + /// cluster id and a major that differs from this server's (higher or lower) keeps its id, takes the cluster id and the + /// current major, and logs no Warning. + [Theory] + [InlineData(-1)] + [InlineData(1)] + public async Task Ensure_AStoredNullClusterId_WithAnotherMajor_KeepsTheId_AndWritesTheCurrentMajor(int storedMajorOffset) + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + var id = await StoreInstallId.EnsureAsync(connection, new CapturingTestLogger(), ct); + var binding = await RealBindingAsync(connection, ct); + await ExecAsync(connection, $"UPDATE {Table} SET system_identifier = NULL, server_major = {binding.ServerMajor + storedMajorOffset}", ct); + + var logger = new CapturingTestLogger(); + Assert.Equal(id, await StoreInstallId.EnsureAsync(connection, logger, ct)); + + Assert.Equal(0, logger.CountAtLevel(LogLevel.Warning)); + var row = await ReadRowAsync(connection, ct); + Assert.Equal(id, row.InstallId); + Assert.Equal(binding.SystemIdentifier, row.SystemIdentifier); + Assert.Equal(binding.ServerMajor, row.ServerMajor); + } + + /// A row that has its table OID but no major (the column was added to a store that already had rows) must still + /// match the cluster id. With the cluster id unchanged it keeps its id and takes the current major, quietly. With a + /// changed one nothing shows the server was upgraded, so the row gets a new id and one Warning. + [Fact] + public async Task Ensure_ARowWithATableOidAndNoMajor_MustMatchTheClusterId_ThenAdoptsTheMajor() + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + var id = await StoreInstallId.EnsureAsync(connection, new CapturingTestLogger(), ct); + var binding = await RealBindingAsync(connection, ct); + + /* The cluster id matches: kept, and the major is filled in without a line in the log. */ + await ExecAsync(connection, $"UPDATE {Table} SET server_major = NULL", ct); + var kept = new CapturingTestLogger(); + Assert.Equal(id, await StoreInstallId.EnsureAsync(connection, kept, ct)); + Assert.Empty(kept.Lines); + Assert.Equal(binding.ServerMajor, (await ReadRowAsync(connection, ct)).ServerMajor); + + /* The cluster id changed and there is no stored major to show a rise: a new id. */ + await ExecAsync(connection, $"UPDATE {Table} SET server_major = NULL, system_identifier = system_identifier + 1", ct); + var replaced = new CapturingTestLogger(); + var newId = await StoreInstallId.EnsureAsync(connection, replaced, ct); + Assert.NotEqual(id, newId); + var warning = Assert.Single(replaced.Lines, l => l.StartsWith("Warning:", StringComparison.Ordinal)); + Assert.Contains(id, warning, StringComparison.Ordinal); + Assert.Contains(newId, warning, StringComparison.Ordinal); + Assert.Contains("major version", warning, StringComparison.Ordinal); + var row = await ReadRowAsync(connection, ct); + Assert.Equal(newId, row.InstallId); + Assert.Equal(binding.SystemIdentifier, row.SystemIdentifier); + Assert.Equal(binding.ServerMajor, row.ServerMajor); + } + + /// The column the major is kept in: nullable integer, made by the rung together with the table OID. Running the + /// rung again, as a store that has it already would, changes nothing and does not fail. + [Fact] + public async Task TheMajorColumn_IsANullableInteger_AndTheRungIsSafeToRunAgain() + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + + async Task<(string Type, string Nullable)> ColumnAsync(string name) + { + await using var command = new NpgsqlCommand( + "SELECT data_type, is_nullable FROM information_schema.columns WHERE table_schema = 'config' AND table_name = 'config_install_id' AND column_name = $1", + connection); + command.Parameters.Add(new NpgsqlParameter { Value = name }); + await using var reader = await command.ExecuteReaderAsync(ct); + Assert.True(await reader.ReadAsync(ct), $"the install id table has no {name} column"); + return (reader.GetString(0), reader.GetString(1)); + } + + Assert.Equal(("integer", "YES"), await ColumnAsync("server_major")); + Assert.Equal(("bigint", "YES"), await ColumnAsync("table_oid")); + + var id = await StoreInstallId.EnsureAsync(connection, new CapturingTestLogger(), ct); + var before = await ScalarAsync(connection, $"SELECT xmin::text FROM {Table}", ct); + await ExecAsync(connection, TableOidRungSql, ct); + Assert.Equal(("integer", "YES"), await ColumnAsync("server_major")); + Assert.Equal(before, await ScalarAsync(connection, $"SELECT xmin::text FROM {Table}", ct)); + Assert.Equal(id, (await ReadRowAsync(connection, ct)).InstallId); + } + + /// The upgrade of a store that holds the id the previous rung's service made: the rung adds both columns and + /// leaves the row alone, and the next start keeps the id and fills in the table OID and the major. + [Fact] + public async Task TheRung_OnAStoreWithTheRowThePreviousRungMade_KeepsTheId_AndTheNextStartFillsInBothValues() + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + await ExecAsync(connection, $"DROP TABLE {Table}", ct); + await ExecAsync(connection, RungSql, ct); + var binding = await RealBindingAsync(connection, ct); + await ExecAsync(connection, + $"INSERT INTO {Table} (id, install_id, system_identifier, database_oid) VALUES (1, 'abcdef12', $1, $2)", ct, + binding.SystemIdentifier, binding.DatabaseOid); + + await ExecAsync(connection, TableOidRungSql, ct); + var migrated = await ReadRowAsync(connection, ct); + Assert.Equal("abcdef12", migrated.InstallId); + Assert.Null(migrated.TableOid); + Assert.Null(migrated.ServerMajor); + + var logger = new CapturingTestLogger(); + Assert.Equal("abcdef12", await StoreInstallId.EnsureAsync(connection, logger, ct)); + Assert.Empty(logger.Lines); + var row = await ReadRowAsync(connection, ct); + var real = await RealBindingAsync(connection, ct); + Assert.Equal("abcdef12", row.InstallId); + Assert.Equal(real.TableOid, row.TableOid); + Assert.Equal(real.ServerMajor, row.ServerMajor); + } + + /// A changed table OID is a different store, and it is one with a NULL cluster id too: the OIDs decide, so a + /// refused or unknown cluster id cannot hide it. The new row carries the real binding. + [Fact] + public async Task Ensure_AChangedTableOid_WithANullClusterId_MakesANewId_AndOneWarning() + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + var oldId = await StoreInstallId.EnsureAsync(connection, new CapturingTestLogger(), ct); + await ExecAsync(connection, $"UPDATE {Table} SET system_identifier = NULL, table_oid = table_oid + 1", ct); + + var logger = new CapturingTestLogger(); + var newId = await StoreInstallId.EnsureAsync(connection, logger, ct); + + Assert.NotEqual(oldId, newId); + var warning = Assert.Single(logger.Lines, l => l.StartsWith("Warning:", StringComparison.Ordinal)); + Assert.Contains(oldId, warning, StringComparison.Ordinal); + Assert.Contains(newId, warning, StringComparison.Ordinal); + Assert.Contains("table's OID", warning, StringComparison.Ordinal); + var binding = await RealBindingAsync(connection, ct); + var row = await ReadRowAsync(connection, ct); + Assert.Equal(newId, row.InstallId); + Assert.Equal(binding.TableOid, row.TableOid); + Assert.Equal(binding.SystemIdentifier, row.SystemIdentifier); + } + + /// A row made before the table OID was kept has none (and no major). It is compared by the old rule once (the + /// cluster id counts when both sides have one) and, when it is this store's, has the table OID and the major filled in + /// and keeps its id. + [Fact] + public async Task Ensure_ARowWithNoTableOid_FollowsTheOldRuleOnce_ThenHasTheOidAndTheMajorFilledIn() + { + var ct = TestContext.Current.CancellationToken; + await using var connection = await ResetAsync(ct); + var binding = await RealBindingAsync(connection, ct); + + /* This store's row: kept, and filled in. */ + await ExecAsync(connection, + $"INSERT INTO {Table} (id, install_id, system_identifier, database_oid) VALUES (1, 'abcdef12', $1, $2)", ct, + binding.SystemIdentifier, binding.DatabaseOid); + var kept = new CapturingTestLogger(); + Assert.Equal("abcdef12", await StoreInstallId.EnsureAsync(connection, kept, ct)); + Assert.Equal(0, kept.CountAtLevel(LogLevel.Warning)); + var adopted = await ReadRowAsync(connection, ct); + Assert.Equal(binding.TableOid, adopted.TableOid); + Assert.Equal(binding.ServerMajor, adopted.ServerMajor); + + /* The old rule still holds for such a row: another cluster id with no table OID is another store, once, even + when the server's major is not what it was. */ + await ExecAsync(connection, $"UPDATE {Table} SET table_oid = NULL, server_major = NULL, system_identifier = system_identifier + 1", ct); + var replaced = new CapturingTestLogger(); + var newId = await StoreInstallId.EnsureAsync(connection, replaced, ct); + Assert.NotEqual("abcdef12", newId); + var warning = Assert.Single(replaced.Lines, l => l.StartsWith("Warning:", StringComparison.Ordinal)); + Assert.Contains("cluster id", warning, StringComparison.Ordinal); + var row = await ReadRowAsync(connection, ct); + Assert.Equal(newId, row.InstallId); + Assert.Equal(binding.TableOid, row.TableOid); + Assert.Equal(binding.SystemIdentifier, row.SystemIdentifier); + Assert.Equal(binding.ServerMajor, row.ServerMajor); + + /* After that the table OID and the major are there, and the new rule applies: a changed cluster id at the same major + is another store, and at a lower stored major it is an upgrade. */ + await ExecAsync(connection, $"UPDATE {Table} SET system_identifier = system_identifier + 1", ct); + var copy = await StoreInstallId.EnsureAsync(connection, new CapturingTestLogger(), ct); + Assert.NotEqual(newId, copy); + await ExecAsync(connection, $"UPDATE {Table} SET system_identifier = system_identifier + 1, server_major = server_major - 1", ct); + Assert.Equal(copy, await StoreInstallId.EnsureAsync(connection, new CapturingTestLogger(), ct)); + } +} diff --git a/Darling/Darling.Tests/StoreInstallIdRefusedClusterLiveTests.cs b/Darling/Darling.Tests/StoreInstallIdRefusedClusterLiveTests.cs new file mode 100644 index 0000000000..1c01989c03 --- /dev/null +++ b/Darling/Darling.Tests/StoreInstallIdRefusedClusterLiveTests.cs @@ -0,0 +1,454 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Globalization; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: every fact runs in the one scratch database the fixture mints through ScratchPostgres, empties the + install id table first, and puts the cluster-id grant back to its default, so the class is deliberately NOT + [Collection("live-postgres")]. The REVOKE the facts make is on a function's ACL, which lives in the database's own + catalog, so it never reaches another database on the cluster. */ + +/// The scratch store behind : a database of its own, +/// migrated once through the real ladder, and a login that has no more than the store's runtime role has on the +/// install id table (it is not a superuser, so a REVOKE on pg_control_system() reaches it). A second +/// connection string puts a function that always fails ahead of the real one on the login's search path, to make a +/// failure that is not a refusal. The login is dropped with the database. +public sealed class RefusedClusterScratchStore : IAsyncLifetime +{ + private const string LoginPassword = "install-id-login-pw"; + + private readonly string _loginName = "install_id_login_" + Guid.NewGuid().ToString("N")[..12]; + private string? _baseConnectionString; + private ScratchPostgres? _scratch; + + /// The scratch store as its owner, or null when DARLING_TEST_PG is unset (the normal ungated + /// run), in which case every fact skips. + public string? OwnerConnectionString => _scratch?.ConnectionString; + + /// The same store as the login without superuser rights. + public string? LoginConnectionString => _scratch is null ? null : LoginBuilder().ConnectionString; + + /// The login's connection with a failing pg_control_system() found before the real one. + public string? FailingFunctionConnectionString + { + get + { + if (_scratch is null) + { + return null; + } + + var builder = LoginBuilder(); + builder.SearchPath = "shadow,pg_catalog"; + return builder.ConnectionString; + } + } + + private NpgsqlConnectionStringBuilder LoginBuilder() => + new(_scratch!.ConnectionString) { Username = _loginName, Password = LoginPassword, Pooling = false }; + + public async ValueTask InitializeAsync() + { + _baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + if (string.IsNullOrEmpty(_baseConnectionString)) + { + return; + } + + _scratch = await ScratchPostgres.CreateAsync(_baseConnectionString, CancellationToken.None); + await using var connection = new NpgsqlConnection(_scratch.ConnectionString); + await connection.OpenAsync(CancellationToken.None); + await PgMigrations.MigrateAsync(connection, CancellationToken.None); + + /* What the runtime role has on this table: the schema's USAGE and SELECT, INSERT, UPDATE and DELETE (the + blanket grants provisioning makes on the config schema). Nothing on pg_control_system() but the + default every login has until a server revokes it. */ + var sql = $@" +CREATE ROLE {_loginName} LOGIN NOSUPERUSER PASSWORD '{LoginPassword}'; +GRANT USAGE ON SCHEMA config TO {_loginName}; +GRANT SELECT, INSERT, UPDATE, DELETE ON config.config_install_id TO {_loginName}; +CREATE SCHEMA shadow; +CREATE FUNCTION shadow.pg_control_system() RETURNS TABLE(system_identifier bigint) LANGUAGE plpgsql +AS $body$ BEGIN RAISE EXCEPTION 'the cluster id read failed' USING ERRCODE = '55000'; END $body$; +GRANT USAGE ON SCHEMA shadow TO {_loginName}; +GRANT EXECUTE ON FUNCTION shadow.pg_control_system() TO {_loginName};"; + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(CancellationToken.None); + } + + public async ValueTask DisposeAsync() + { + if (_scratch is null) + { + return; + } + + await _scratch.DisposeAsync(); + + try + { + await using var admin = new NpgsqlConnection(_baseConnectionString); + await admin.OpenAsync(CancellationToken.None); + await using var drop = new NpgsqlCommand($"DROP ROLE IF EXISTS {_loginName}", admin); + await drop.ExecuteNonQueryAsync(CancellationToken.None); + } + catch + { + /* Best-effort: a leaked login on a throwaway test cluster is harmless, and failing a passing test in + its cleanup would invert the signal. */ + } + } +} + +/// +/// The install id store on a server that refuses a login pg_control_system() (#4961). A managed or hardened +/// server may revoke it, and the service has to start there anyway: the id is then bound to the database alone, the +/// stored cluster id is left empty, and a later start that can read the cluster id (or can no longer) keeps the same id +/// rather than making a new one on its own. Only a refusal falls back; any other failure of that read still fails the +/// start. +/// +public sealed class StoreInstallIdRefusedClusterLiveTests : IClassFixture +{ + private const string Table = "config.config_install_id"; + + private readonly RefusedClusterScratchStore _store; + + public StoreInstallIdRefusedClusterLiveTests(RefusedClusterScratchStore store) + { + _store = store; + } + + private string OwnerConnectionString + { + get + { + Assert.SkipWhen(string.IsNullOrEmpty(_store.OwnerConnectionString), "Set DARLING_TEST_PG to run the live install id fallback pins."); + return _store.OwnerConnectionString!; + } + } + + private static async Task OpenAsync(string connectionString, CancellationToken ct) + { + var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + return connection; + } + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task ScalarAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + return (T)(await command.ExecuteScalarAsync(ct))!; + } + + /// The owner's connection to a store with no install id row and the cluster id readable, the state a + /// fact then changes to the one it needs. + private async Task OwnerAsync(CancellationToken ct) + { + var owner = await OpenAsync(OwnerConnectionString, ct); + await ExecAsync(owner, $"DELETE FROM {Table}", ct); + await AllowClusterIdAsync(owner, ct); + return owner; + } + + private static Task RefuseClusterIdAsync(NpgsqlConnection owner, CancellationToken ct) => + ExecAsync(owner, "REVOKE EXECUTE ON FUNCTION pg_control_system() FROM PUBLIC", ct); + + private static Task AllowClusterIdAsync(NpgsqlConnection owner, CancellationToken ct) => + ExecAsync(owner, "GRANT EXECUTE ON FUNCTION pg_control_system() TO PUBLIC", ct); + + private static async Task<(string InstallId, long? SystemIdentifier, long DatabaseOid)> ReadRowAsync(NpgsqlConnection connection, CancellationToken ct) + { + await using var command = new NpgsqlCommand($"SELECT install_id, system_identifier, database_oid FROM {Table} WHERE id = 1", connection); + await using var reader = await command.ExecuteReaderAsync(ct); + Assert.True(await reader.ReadAsync(ct), "the install id table has no row"); + return (reader.GetString(0), reader.IsDBNull(1) ? null : reader.GetInt64(1), reader.GetInt64(2)); + } + + private static Task DatabaseOidAsync(NpgsqlConnection connection, CancellationToken ct) => + ScalarAsync(connection, "SELECT oid::bigint FROM pg_database WHERE datname = current_database()", ct); + + private static Task ClusterIdAsync(NpgsqlConnection owner, CancellationToken ct) => + ScalarAsync(owner, "SELECT system_identifier FROM pg_control_system()", ct); + + private static Task ServerMajorAsync(NpgsqlConnection connection, CancellationToken ct) => + ScalarAsync(connection, "SELECT current_setting('server_version_num')::int / 10000", ct); + + private static Task StoredMajorAsync(NpgsqlConnection owner, CancellationToken ct) => + ScalarAsync(owner, $"SELECT server_major FROM {Table}", ct); + + /// The row's version: a write, even one that sets the values it already has, gives the row a new one. + private static Task RowVersionAsync(NpgsqlConnection owner, CancellationToken ct) => + ScalarAsync(owner, $"SELECT xmin::text FROM {Table}", ct); + + /// Makes the stored row the one the cluster before a major upgrade left: another cluster id and a major below + /// this server's, with both OIDs as they are. This server then reads as the same store after an upgrade. + private static async Task<(long Cluster, int Major)> StoreTheRowOfTheClusterBeforeAnUpgradeAsync(NpgsqlConnection owner, CancellationToken ct) + { + var cluster = await ClusterIdAsync(owner, ct) + 1; + var major = await ServerMajorAsync(owner, ct) - 1; + await ExecAsync(owner, $"UPDATE {Table} SET system_identifier = {cluster}, server_major = {major}", ct); + return (cluster, major); + } + + private static bool IsTheDatabaseAloneLine(string line) => + line.StartsWith("Information:", StringComparison.Ordinal) && line.Contains("database alone", StringComparison.Ordinal); + + [Fact] + public async Task Ensure_WhenTheLoginMayNotReadTheClusterId_StoresARowBoundToTheDatabaseAlone_AndLogsOneInformation() + { + var ct = TestContext.Current.CancellationToken; + await using var owner = await OwnerAsync(ct); + await RefuseClusterIdAsync(owner, ct); + await using var login = await OpenAsync(_store.LoginConnectionString!, ct); + + var logger = new CapturingTestLogger(); + var id = await StoreInstallId.EnsureAsync(login, logger, ct); + + Assert.True(InstallId.IsValid(id), $"'{id}' is not eight lowercase hex digits"); + Assert.Equal(1L, await ScalarAsync(owner, $"SELECT count(*) FROM {Table}", ct)); + var row = await ReadRowAsync(owner, ct); + Assert.Equal(id, row.InstallId); + Assert.Null(row.SystemIdentifier); + Assert.Equal(await DatabaseOidAsync(owner, ct), row.DatabaseOid); + Assert.Equal(0, logger.CountAtLevel(LogLevel.Warning)); + Assert.Single(logger.Lines, IsTheDatabaseAloneLine); + + /* The read path (the CLI's and the Viewer's) sees that row too. */ + Assert.Equal(id, await StoreInstallId.TryReadAsync(login, ct)); + } + + [Fact] + public async Task TryRead_OfARowStoredWithoutAClusterId_ReadsTheId_AndWritesNothing() + { + var ct = TestContext.Current.CancellationToken; + await using var owner = await OwnerAsync(ct); + await RefuseClusterIdAsync(owner, ct); + var oid = await DatabaseOidAsync(owner, ct); + await ExecAsync(owner, + $"INSERT INTO {Table} (id, install_id, system_identifier, database_oid) VALUES (1, 'abcdef12', NULL, {oid})", ct); + var before = await ScalarAsync(owner, $"SELECT xmin::text FROM {Table}", ct); + + await using var login = await OpenAsync(_store.LoginConnectionString!, ct); + Assert.Equal("abcdef12", await StoreInstallId.TryReadAsync(login, ct)); + await using (var dataSource = NpgsqlDataSource.Create(_store.LoginConnectionString!)) + { + Assert.Equal("abcdef12", await StoreInstallId.TryReadAsync(dataSource, ct)); + } + + /* A read leaves the row exactly as it found it (a write would have given it a new xmin). */ + Assert.Equal(before, await ScalarAsync(owner, $"SELECT xmin::text FROM {Table}", ct)); + } + + [Fact] + public async Task Ensure_AfterTheClusterIdWasRefused_ThenAllowed_KeepsTheSameId_AndLogsNoWarning() + { + var ct = TestContext.Current.CancellationToken; + await using var owner = await OwnerAsync(ct); + await using var login = await OpenAsync(_store.LoginConnectionString!, ct); + + await RefuseClusterIdAsync(owner, ct); + var first = await StoreInstallId.EnsureAsync(login, new CapturingTestLogger(), ct); + + await AllowClusterIdAsync(owner, ct); + var logger = new CapturingTestLogger(); + var second = await StoreInstallId.EnsureAsync(login, logger, ct); + + Assert.Equal(first, second); + Assert.Equal(0, logger.CountAtLevel(LogLevel.Warning)); + Assert.Equal(1L, await ScalarAsync(owner, $"SELECT count(*) FROM {Table}", ct)); + Assert.Equal(first, (await ReadRowAsync(owner, ct)).InstallId); + } + + [Fact] + public async Task Ensure_AfterTheClusterIdWasAllowed_ThenRefused_KeepsTheSameId_AndLogsNoWarning() + { + var ct = TestContext.Current.CancellationToken; + await using var owner = await OwnerAsync(ct); + await using var login = await OpenAsync(_store.LoginConnectionString!, ct); + + var first = await StoreInstallId.EnsureAsync(login, new CapturingTestLogger(), ct); + var stored = await ReadRowAsync(owner, ct); + Assert.Equal(await ClusterIdAsync(owner, ct), stored.SystemIdentifier); + + await RefuseClusterIdAsync(owner, ct); + var logger = new CapturingTestLogger(); + var second = await StoreInstallId.EnsureAsync(login, logger, ct); + + Assert.Equal(first, second); + Assert.Equal(0, logger.CountAtLevel(LogLevel.Warning)); + Assert.Equal(1L, await ScalarAsync(owner, $"SELECT count(*) FROM {Table}", ct)); + Assert.Equal(stored, await ReadRowAsync(owner, ct)); + } + + /// The server was upgraded to a new major version (the row says cluster A and major M, the server now has + /// cluster B and major M+1, and both OIDs are as they were), and the first start after it can't read the cluster id. + /// The row keeps the id, and it keeps the major that goes with the cluster id it has: the new major written beside the + /// old cluster id would make a later start that can read the new cluster id see a changed cluster id at an equal + /// major, which is what a copy of the row looks like. The start writes nothing, so the row is as it found it. + [Fact] + public async Task Ensure_AfterAMajorUpgrade_AStartThatCannotReadTheClusterId_KeepsTheId_AndLeavesTheRowAsItWas() + { + var ct = TestContext.Current.CancellationToken; + await using var owner = await OwnerAsync(ct); + await using var login = await OpenAsync(_store.LoginConnectionString!, ct); + var id = await StoreInstallId.EnsureAsync(login, new CapturingTestLogger(), ct); + var (oldCluster, oldMajor) = await StoreTheRowOfTheClusterBeforeAnUpgradeAsync(owner, ct); + await RefuseClusterIdAsync(owner, ct); + var before = await RowVersionAsync(owner, ct); + + var logger = new CapturingTestLogger(); + Assert.Equal(id, await StoreInstallId.EnsureAsync(login, logger, ct)); + + Assert.Equal(0, logger.CountAtLevel(LogLevel.Warning)); + var row = await ReadRowAsync(owner, ct); + Assert.Equal(id, row.InstallId); + Assert.Equal(oldCluster, row.SystemIdentifier); + Assert.Equal(oldMajor, await StoredMajorAsync(owner, ct)); + Assert.Equal(before, await RowVersionAsync(owner, ct)); + } + + /// The same upgrade, and a later start can read the cluster id. The id stays, one Information line says the + /// cluster id changed and the major rose, and the row is rebound to the new cluster id and major: the start that could + /// not read the cluster id did not make the later one look like a copy. + [Fact] + public async Task Ensure_AfterAMajorUpgrade_AStartThatCannotReadTheClusterId_ThenOneThatCan_KeepsTheId_AndRebindsTheRow() + { + var ct = TestContext.Current.CancellationToken; + await using var owner = await OwnerAsync(ct); + await using var login = await OpenAsync(_store.LoginConnectionString!, ct); + var id = await StoreInstallId.EnsureAsync(login, new CapturingTestLogger(), ct); + var (oldCluster, oldMajor) = await StoreTheRowOfTheClusterBeforeAnUpgradeAsync(owner, ct); + var newCluster = await ClusterIdAsync(owner, ct); + var newMajor = await ServerMajorAsync(owner, ct); + + await RefuseClusterIdAsync(owner, ct); + var refused = new CapturingTestLogger(); + Assert.Equal(id, await StoreInstallId.EnsureAsync(login, refused, ct)); + + await AllowClusterIdAsync(owner, ct); + var allowed = new CapturingTestLogger(); + var later = await StoreInstallId.EnsureAsync(login, allowed, ct); + + Assert.Equal(id, later); + Assert.Equal(0, refused.CountAtLevel(LogLevel.Warning)); + Assert.Equal(0, allowed.CountAtLevel(LogLevel.Warning)); + var line = Assert.Single(allowed.Lines, l => l.StartsWith("Information:", StringComparison.Ordinal)); + Assert.Contains(oldCluster.ToString(CultureInfo.InvariantCulture), line, StringComparison.Ordinal); + Assert.Contains(newCluster.ToString(CultureInfo.InvariantCulture), line, StringComparison.Ordinal); + Assert.Contains($"from {oldMajor} to {newMajor}", line, StringComparison.Ordinal); + Assert.Contains("id stays", line, StringComparison.Ordinal); + var row = await ReadRowAsync(owner, ct); + Assert.Equal(id, row.InstallId); + Assert.Equal(newCluster, row.SystemIdentifier); + Assert.Equal(newMajor, await StoredMajorAsync(owner, ct)); + Assert.Equal(1L, await ScalarAsync(owner, $"SELECT count(*) FROM {Table}", ct)); + } + + /// A row with no cluster id has no cluster id for a major to stay beside, so a start that can't read the cluster + /// id still writes the current major into it, whether the stored one was lower or higher. The next such start finds + /// nothing left to write. + [Theory] + [InlineData(-1)] + [InlineData(1)] + public async Task Ensure_ARowWithNoClusterId_AStartThatCannotReadTheClusterId_StillWritesTheMajor(int storedMajorOffset) + { + var ct = TestContext.Current.CancellationToken; + await using var owner = await OwnerAsync(ct); + await using var login = await OpenAsync(_store.LoginConnectionString!, ct); + await RefuseClusterIdAsync(owner, ct); + var id = await StoreInstallId.EnsureAsync(login, new CapturingTestLogger(), ct); + var major = await ServerMajorAsync(owner, ct); + await ExecAsync(owner, $"UPDATE {Table} SET server_major = {major + storedMajorOffset}", ct); + + var logger = new CapturingTestLogger(); + Assert.Equal(id, await StoreInstallId.EnsureAsync(login, logger, ct)); + + Assert.Equal(0, logger.CountAtLevel(LogLevel.Warning)); + var row = await ReadRowAsync(owner, ct); + Assert.Equal(id, row.InstallId); + Assert.Null(row.SystemIdentifier); + Assert.Equal(major, await StoredMajorAsync(owner, ct)); + + var written = await RowVersionAsync(owner, ct); + Assert.Equal(id, await StoreInstallId.EnsureAsync(login, new CapturingTestLogger(), ct)); + Assert.Equal(written, await RowVersionAsync(owner, ct)); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Ensure_WhenTheClusterIdIsRefused_ADifferentDatabaseOid_MakesANewId_AndLogsOneWarningNamingBothIds(bool firstStartReadTheClusterId) + { + var ct = TestContext.Current.CancellationToken; + await using var owner = await OwnerAsync(ct); + await using var login = await OpenAsync(_store.LoginConnectionString!, ct); + + if (!firstStartReadTheClusterId) + { + await RefuseClusterIdAsync(owner, ct); + } + + var oldId = await StoreInstallId.EnsureAsync(login, new CapturingTestLogger(), ct); + await RefuseClusterIdAsync(owner, ct); + await ExecAsync(owner, $"UPDATE {Table} SET database_oid = database_oid + 1", ct); + + var logger = new CapturingTestLogger(); + var newId = await StoreInstallId.EnsureAsync(login, logger, ct); + + Assert.True(InstallId.IsValid(newId)); + Assert.NotEqual(oldId, newId); + var warning = Assert.Single(logger.Lines, l => l.StartsWith("Warning:", StringComparison.Ordinal)); + Assert.Contains(oldId, warning, StringComparison.Ordinal); + Assert.Contains(newId, warning, StringComparison.Ordinal); + + var row = await ReadRowAsync(owner, ct); + Assert.Equal(newId, row.InstallId); + Assert.Null(row.SystemIdentifier); + Assert.Equal(await DatabaseOidAsync(owner, ct), row.DatabaseOid); + + /* The next start is an ordinary one: same id, no second warning. */ + var after = new CapturingTestLogger(); + Assert.Equal(newId, await StoreInstallId.EnsureAsync(login, after, ct)); + Assert.Equal(0, after.CountAtLevel(LogLevel.Warning)); + } + + /// Only a refusal (permission denied) falls back. A read that fails for any other reason is a store that + /// is not answering as it should, so the start fails and its retry takes over, as it does for a failed migration. + /// Guards against a fallback written as a catch of every database error. + [Fact] + public async Task Ensure_AFailureOfTheClusterIdReadOtherThanARefusal_IsNotSwallowed_AndNothingIsStored() + { + var ct = TestContext.Current.CancellationToken; + await using var owner = await OwnerAsync(ct); + await using var login = await OpenAsync(_store.FailingFunctionConnectionString!, ct); + + var logger = new CapturingTestLogger(); + var ex = await Assert.ThrowsAsync(() => StoreInstallId.EnsureAsync(login, logger, ct)); + + Assert.Equal("55000", ex.SqlState); + Assert.Equal(0L, await ScalarAsync(owner, $"SELECT count(*) FROM {Table}", ct)); + Assert.DoesNotContain(logger.Lines, IsTheDatabaseAloneLine); + } +} diff --git a/Darling/Darling.Tests/StoreInstallIdSourcePinTests.cs b/Darling/Darling.Tests/StoreInstallIdSourcePinTests.cs new file mode 100644 index 0000000000..db1d957f25 --- /dev/null +++ b/Darling/Darling.Tests/StoreInstallIdSourcePinTests.cs @@ -0,0 +1,127 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// Source pins for who may make the install id's row (#4961). The service makes it once at start, after the store's +/// migrations and before any worker runs; the CLI and the Viewer only read it. A reader that quietly inserted would +/// make an id for a store nobody started, and two makers racing is what the row's one-row CHECK and the make's +/// race-safe update exist to survive, not to invite. +/// +[Trait("Stage", "Guard")] +public sealed class StoreInstallIdSourcePinTests +{ + private static readonly string[] ProductProjects = + { + "PerformanceMonitor.Darling.Service", + "PerformanceMonitor.Darling.Viewer", + "PerformanceMonitor.Darling.Storage", + "PerformanceMonitor.Darling.Analysis", + }; + + private static IEnumerable ProductSourceFiles() => + ProductProjects + .SelectMany(project => Directory.EnumerateFiles(Path.Combine(RepoFile.Root, "Darling", project), "*.cs", SearchOption.AllDirectories)) + .Where(path => !path.Contains(Path.DirectorySeparatorChar + "obj" + Path.DirectorySeparatorChar, StringComparison.Ordinal) + && !path.Contains(Path.DirectorySeparatorChar + "bin" + Path.DirectorySeparatorChar, StringComparison.Ordinal)); + + [Fact] + public void OnlyTheServiceWorkerCallsTheEnsure_SoTheCliAndTheViewerNeverMakeTheRow() + { + var callers = ProductSourceFiles() + .Where(path => CSharpSourceWalker.StripCommentsAndStrings(File.ReadAllText(path)) + .Contains("StoreInstallId.EnsureAsync(", StringComparison.Ordinal)) + .Select(Path.GetFileName) + .ToList(); + + Assert.Equal(new[] { "DarlingWorker.cs" }, callers); + + /* The two readers, named, so a failure says which surface broke the rule. */ + var viewerCallers = Directory.EnumerateFiles(Path.Combine(RepoFile.Root, "Darling", "PerformanceMonitor.Darling.Viewer"), "*.cs", SearchOption.AllDirectories) + .Where(path => File.ReadAllText(path).Contains("StoreInstallId.EnsureAsync", StringComparison.Ordinal)) + .ToList(); + Assert.Empty(viewerCallers); + + foreach (var cliFile in new[] { "DarlingCliCommands.cs", "Program.cs" }) + { + var cli = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", cliFile); + Assert.DoesNotContain("StoreInstallId.EnsureAsync", cli, StringComparison.Ordinal); + } + } + + [Fact] + public void TheReadPathIsASelect_AndItsMethodsReferenceNoWriteStatement() + { + var sql = StoreInstallId.ReadSql.TrimStart(); + Assert.StartsWith("SELECT", sql, StringComparison.OrdinalIgnoreCase); + foreach (var verb in new[] { "INSERT", "UPDATE", "DELETE", "TRUNCATE", "DROP", "ALTER", "CREATE", "GRANT" }) + { + Assert.DoesNotContain(verb, sql, StringComparison.OrdinalIgnoreCase); + } + + var code = CSharpSourceWalker.StripCommentsAndStrings( + RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "StoreInstallId.cs")); + var searchFrom = 0; + var readers = 0; + while (true) + { + var at = code.IndexOf("TryReadAsync(", searchFrom, StringComparison.Ordinal); + if (at < 0) + { + break; + } + + var open = code.IndexOf('{', at); + var semicolon = code.IndexOf(';', at); + searchFrom = at + 1; + if (open < 0 || (semicolon >= 0 && semicolon < open)) + { + continue; // an expression-bodied member or a call: the declarations with a block are the ones to read + } + + var body = CSharpSourceWalker.BraceBalanced(code, open); + readers++; + foreach (var write in new[] { "InsertSql", "ReplaceSql", "ExecuteNonQueryAsync", "EnsureAsync" }) + { + Assert.DoesNotContain(write, body, StringComparison.Ordinal); + } + } + + Assert.True(readers > 0, "no TryReadAsync body was found to pin"); + } + + [Fact] + public void TheWorkerMakesTheIdAfterTheMigrationsAndBeforeTheWorkers_AndKeepsIt() + { + var worker = CSharpSourceWalker.StripCommentsAndStrings( + RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "DarlingWorker.cs")); + + var migrate = worker.IndexOf("PgMigrations.MigrateAsync(migrateConnection", StringComparison.Ordinal); + var ensure = worker.IndexOf("StoreInstallId.EnsureAsync(", StringComparison.Ordinal); + Assert.True(migrate >= 0, "the worker no longer migrates the store at start, so this pin is comparing against nothing"); + Assert.True(ensure > migrate, "the install id is made after the store's migrations, never before them"); + + /* Inside the startup retry's try, before its `break`: a failure to make the id is retried and triaged like a + failed migration, and no worker has started by then. */ + var leave = worker.IndexOf("break;", migrate, StringComparison.Ordinal); + Assert.True(leave > ensure, "the install id is made inside the startup retry's try, before the loop is left"); + var catches = worker.IndexOf("catch (", migrate, StringComparison.Ordinal); + Assert.True(catches > ensure, "the make sits in the same try as the migration"); + + Assert.Contains("_installId =", worker, StringComparison.Ordinal); + Assert.Contains("internal string? CurrentInstallId", worker, StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/StoreIsOnThisMachineTests.cs b/Darling/Darling.Tests/StoreIsOnThisMachineTests.cs index 24b8379d4d..6e3896c79b 100644 --- a/Darling/Darling.Tests/StoreIsOnThisMachineTests.cs +++ b/Darling/Darling.Tests/StoreIsOnThisMachineTests.cs @@ -129,37 +129,16 @@ public void AParseableHostWinsOverTrailingJunk() } /// - /// The dialog surfaces the machine-bound-secret hint for the two secret-bearing modes — SQL auth and (as of - /// #3484) a service principal — on a non-local store, and CLEARS its own message when the mode changes away, - /// the same self-clearing discipline the interactive-Entra arm uses, which stops a stale hint sitting under - /// an unrelated auth mode. - /// - /// Pinned at the source: the alternative is standing up a WPF dialog with a live store. - /// - [Fact] - public void TheDialogShowsTheHintForSecretBearingModesOnANonLocalStore() - { - var source = ReadDialogSource(); - - Assert.Contains("SqlAuthRadio.IsChecked == true || ServicePrincipalAuthRadio.IsChecked == true", source, StringComparison.Ordinal); - Assert.Contains("_dataService is { StoreIsOnThisMachine: false }", source, StringComparison.Ordinal); - Assert.Contains("StatusText.Text = SqlCredentialMachineBoundHint;", source, StringComparison.Ordinal); - /* Self-clearing, exactly like the interactive-Entra message above it. */ - Assert.Contains("else if (StatusText.Text == SqlCredentialMachineBoundHint)", source, StringComparison.Ordinal); - } - - /// - /// The hint has to be actionable, not just alarming: it names all three ways to produce a credential the - /// service can use, which is what turns it from a warning into an instruction. + /// A password is sealed to the service's published key (#5366), so it can be saved from a viewer on any machine: the + /// dialog no longer warns that a secret is readable only on the machine that wrote it. /// [Fact] - public void TheHintNamesEveryWayToProduceAUsableCredential() + public void TheDialogDoesNotWarnThatASecretIsBoundToThisMachine() { var source = ReadDialogSource(); - Assert.Contains("viewer on the service's host", source, StringComparison.Ordinal); - Assert.Contains("--add-server", source, StringComparison.Ordinal); - Assert.Contains("env:/file:", source, StringComparison.Ordinal); + Assert.DoesNotContain("SqlCredentialMachineBoundHint", source, StringComparison.Ordinal); + Assert.DoesNotContain("StoreIsOnThisMachine", source, StringComparison.Ordinal); } private static string ReadDialogSource([CallerFilePath] string thisFile = "") diff --git a/Darling/Darling.Tests/StoreLogClassifierTests.cs b/Darling/Darling.Tests/StoreLogClassifierTests.cs index 95f580cb4d..9a2e925f2d 100644 --- a/Darling/Darling.Tests/StoreLogClassifierTests.cs +++ b/Darling/Darling.Tests/StoreLogClassifierTests.cs @@ -123,6 +123,16 @@ not an administrator command. The floor rule names the whole administrative phra "\tSELECT DISTINCT ON (server_id) server_id", "\tFROM v_cpu_utilization_stats", + /* slow_plan (#5097) - an auto_explain JSON plan under the same duration rule. */ + "2026-09-05 14:07:20.500 UTC [5336] LOG: duration: 11000.100 ms plan:", + "\t{", + "\t \"Query Text\": \"SELECT 1 WHERE x = 'FixtureLeak'\",", + "\t \"Plan\": {", + "\t \"Node Type\": \"Seq Scan\",", + "\t \"Relation Name\": \"t\"", + "\t }", + "\t}", + /* lock_timeout, under the MANAGED prefix - proving the classifier does not care which family. */ "2026-09-05 14:07:30 UTC:192.0.2.10(52345):app_user@app_db:[5334]:ERROR: canceling statement due to lock timeout", @@ -256,7 +266,9 @@ public void EveryRetainedClass_KeepsItsProseAsWritten_AndNormalizesItsSql() var duplicate = retained.Single(g => g.MessageText!.StartsWith("duplicate key", StringComparison.Ordinal)); Assert.Equal("duplicate key value violates unique constraint \"?\"", duplicate.MessageText); Assert.Contains("ERROR: duplicate key value violates unique constraint \"t_pkey\"", duplicate.SampleLine, StringComparison.Ordinal); - Assert.Contains("DETAIL: Key (id)=(Kept3944c) already exists.", duplicate.SampleLine, StringComparison.Ordinal); + /* The key value is withheld from the stored text (#5452): the constraint and the column stay, the stored value does not. */ + Assert.Contains("DETAIL: Key (id)=(...) already exists.", duplicate.SampleLine, StringComparison.Ordinal); + Assert.DoesNotContain("Kept3944c", duplicate.SampleLine, StringComparison.Ordinal); Assert.Contains("STATEMENT: INSERT INTO t VALUES ('?')", duplicate.SampleLine, StringComparison.Ordinal); Assert.Contains( "DETAIL: Process 5360 waits for ShareLock on transaction 809; blocked by process 5361.", @@ -861,11 +873,12 @@ public void RepeatsOfOneSlowStatement_GroupIntoOneRow_WhateverTheirDurations() /// /// A duration line that names no statement is not a slow statement this class can name: log_duration's - /// bare line, and auto_explain's plan, whose Query Text is the statement VERBATIM. Both are counted as - /// routine and keep no text. + /// bare line is counted as routine and keeps no text. auto_explain's plan, whose Query Text is the + /// statement VERBATIM, is no longer routine (#5097): a plan in text format cannot be redacted, so it is kept + /// as its head alone, with the plan withheld. /// [Fact] - public void ADurationLineWithNoStatement_IsRoutine_AndKeepsNoText() + public void ADurationLineWithNoStatement_IsRoutine_AndATextPlanIsKeptAsItsHeadOnly() { var census = StoreLogClassifier.Classify(string.Join("\n", [ @@ -876,11 +889,244 @@ public void ADurationLineWithNoStatement_IsRoutine_AndKeepsNoText() "", ])); - var only = Assert.Single(census.Groups); - Assert.Equal(StoreLogClassifier.RoutineClass, only.EventClass); - Assert.Equal(2, only.Occurrences); - Assert.Null(only.MessageText); - Assert.Null(only.SampleLine); + var routine = Assert.Single(census.Groups, g => g.EventClass == StoreLogClassifier.RoutineClass); + Assert.Equal(1, routine.Occurrences); + Assert.Null(routine.MessageText); + Assert.Null(routine.SampleLine); + + var plan = Assert.Single(census.Groups, g => g.EventClass == StoreLogClassifier.SlowPlanClass); + Assert.Equal(StoreLogClassifier.WithheldPlanMessage, plan.MessageText); + Assert.Equal( + DefaultPrefix + "LOG: duration: 8123.001 ms plan:\n\t" + PgLogTextRedactor.WithheldPlan, + plan.SampleLine); + Assert.DoesNotContain("Secret3904f", plan.SampleLine, StringComparison.Ordinal); + } + + /* A PostgreSQL 16+ auto_explain JSON report, as the log writes it: the brackets EXPLAIN adds are stripped, + so the root is an object, and every line of it is tab-led under the duration line. The planted values are + distinct literals, one in each place a value can ride. */ + private static string[] JsonPlanEntry(string duration, string parameter, string literal, long? queryId = 4242) + { + var json = new[] + { + "{", + " \"Query Text\": \"SELECT * FROM orders WHERE code = '" + literal + "' AND id = $1\",", + " \"Query Parameters\": \"$1 = '" + parameter + "'\",", + " \"Plan\": {", + " \"Node Type\": \"Index Scan\",", + " \"Relation Name\": \"orders\",", + " \"Alias\": \"orders\",", + " \"Index Cond\": \"(orders.id = " + (parameter.Length + 770001) + ")\",", + " \"Filter\": \"((orders.code)::text = '" + literal + "'::text)\"", + " }" + (queryId is null ? string.Empty : ","), + }.ToList(); + if (queryId is not null) + { + json.Add(" \"Query Identifier\": " + queryId); + } + + json.Add("}"); + return [DefaultPrefix + "LOG: duration: " + duration + " ms plan:", .. json.Select(l => "\t" + l)]; + } + + /// + /// #5097: an auto_explain JSON plan is KEPT as slow_plan, as the redacted JSON the monitored-target plan + /// route stores. Every planted value is asserted absent by its exact string, in the message and in the sample, + /// and the sample is asserted to BE the compact redacted JSON, so a classifier that stored the raw text (or that + /// dropped the redaction) fails here. + /// + [Fact] + public void AJsonPlan_IsKeptAsSlowPlan_WithNoStatementTextAndNoValue() + { + var census = StoreLogClassifier.Classify(string.Join("\n", JsonPlanEntry("12345.678", "BindLeak5097a", "LitLeak5097b")) + "\n"); + + var plan = Assert.Single(census.Groups); + Assert.Equal(StoreLogClassifier.SlowPlanClass, plan.EventClass); + Assert.Equal("LOG", plan.Severity); + Assert.StartsWith("plan ", plan.MessageText, StringComparison.Ordinal); + Assert.EndsWith(" Index Scan queryid=4242", plan.MessageText, StringComparison.Ordinal); + + foreach (var planted in new[] { "BindLeak5097a", "LitLeak5097b", "Query Text", "Query Parameters", "770014", "SELECT * FROM orders" }) + { + Assert.DoesNotContain(planted, plan.MessageText, StringComparison.Ordinal); + Assert.DoesNotContain(planted, plan.SampleLine, StringComparison.Ordinal); + } + + var firstLine = DefaultPrefix + "LOG: duration: 12345.678 ms plan:"; + var parsed = PgPlanLogParser.FromBlock(0, 0, "{\"Plan\":{\"Node Type\":\"Index Scan\",\"Relation Name\":\"orders\",\"Alias\":\"orders\",\"Index Cond\":\"(orders.id = 770014)\",\"Filter\":\"((orders.code)::text = 'LitLeak5097b'::text)\"},\"Query Identifier\":4242}"); + Assert.NotNull(parsed); + Assert.Equal(firstLine + "\n\t" + parsed!.Value.PlanJson, plan.SampleLine); + Assert.Equal("plan " + parsed.Value.PlanHash + " Index Scan queryid=4242", plan.MessageText); + Assert.Contains("(orders.id = ?)", plan.SampleLine, StringComparison.Ordinal); + /* System.Text.Json writes the apostrophe as \u0027, the same bytes the plan route stores. */ + Assert.Contains("\\u0027?\\u0027::text", plan.SampleLine, StringComparison.Ordinal); + } + + /// One plan shape is one row however often it ran and whatever it ran with (the parameter value no longer + /// splits the hash, #5103); a different shape is another row; a plan with no query identifier says ?. + [Fact] + public void TwoRunsOfOnePlanShape_AreOneRow_AndAnotherShapeIsAnother() + { + var census = StoreLogClassifier.Classify(string.Join("\n", + JsonPlanEntry("11000.1", "ParamOneA", "LitOneA") + .Concat(JsonPlanEntry("19000.9", "ParamTwoB", "LitTwoB")) + .Concat(JsonPlanEntry("15000.5", "ParamThreeC", "LitThreeC", queryId: null))) + "\n"); + + var plans = census.Groups.Where(g => g.EventClass == StoreLogClassifier.SlowPlanClass).ToList(); + Assert.Equal(2, plans.Count); + var withId = Assert.Single(plans, g => g.MessageText!.EndsWith("queryid=4242", StringComparison.Ordinal)); + Assert.Equal(2, withId.Occurrences); + var withoutId = Assert.Single(plans, g => g.MessageText!.EndsWith("queryid=?", StringComparison.Ordinal)); + Assert.Equal(1, withoutId.Occurrences); + foreach (var planted in new[] { "ParamOneA", "ParamTwoB", "ParamThreeC", "LitOneA", "LitTwoB", "LitThreeC" }) + { + Assert.DoesNotContain(planted, string.Concat(plans.Select(p => p.MessageText + p.SampleLine)), StringComparison.Ordinal); + } + } + + /// #5114: two runs of one plan whose estimates and runtime counters differ are one row, and the kept sample + /// still carries a capture's own estimates. + [Fact] + public void TwoRunsOfOneShape_WithDifferentEstimates_AreOneRow() + { + static string[] Entry(string duration, string cost, string rows) => + [ + DefaultPrefix + "LOG: duration: " + duration + " ms plan:", + "\t{\"Plan\":{\"Node Type\":\"Index Scan\",\"Relation Name\":\"orders\",\"Alias\":\"orders\",\"Total Cost\":" + cost + + ",\"Plan Rows\":" + rows + ",\"Actual Rows\":" + rows + ",\"Shared Hit Blocks\":" + rows + "},\"Query Identifier\":4242,\"Execution Time\":" + duration + "}", + ]; + + var census = StoreLogClassifier.Classify(string.Join("\n", Entry("11000.1", "8.5", "1").Concat(Entry("19000.9", "91234.5", "880000"))) + "\n"); + + var plan = Assert.Single(census.Groups, g => g.EventClass == StoreLogClassifier.SlowPlanClass); + Assert.Equal(2, plan.Occurrences); + Assert.EndsWith(" Index Scan queryid=4242", plan.MessageText, StringComparison.Ordinal); + Assert.Contains("\"Total Cost\":", plan.SampleLine, StringComparison.Ordinal); + } + + /// A plan that parsed but is over the sample cap keeps its hash, top node and query id, and a size marker + /// in place of the JSON: it still groups, and nothing of it is kept. + [Fact] + public void AParsedPlanOverTheCap_KeepsItsKey_AndASizeMarker() + { + var wide = string.Join(",", Enumerable.Range(0, 900).Select(i => "\"t.column_" + i + "\"")); + var tooBig = DefaultPrefix + "LOG: duration: 14000.0 ms plan:\n\t{\"Plan\": {\"Node Type\": \"Seq Scan\", \"Output\": [" + wide + + "]}, \"Query Identifier\": 777, \"Query Text\": \"TooBigLeak5097\"}\n"; + + var plan = Assert.Single(StoreLogClassifier.Classify(tooBig).Groups); + + Assert.Equal(StoreLogClassifier.SlowPlanClass, plan.EventClass); + Assert.Matches("^plan [0-9A-F]{32} Seq Scan queryid=777$", plan.MessageText); + Assert.Contains(StoreLogClassifier.PlanTooLargeMarker, plan.SampleLine, StringComparison.Ordinal); + Assert.Matches(@"plan too large to keep: \d+ characters$", plan.SampleLine); + Assert.True(plan.SampleLine!.Length <= StoreLogClassifier.MaxSampleLength); + foreach (var planted in new[] { "TooBigLeak5097", "Seq Scan", "Query Text", "column_5" }) + { + Assert.DoesNotContain(planted, plan.SampleLine, StringComparison.Ordinal); + } + } + + /// A tab-led body that parses as JSON but breaks an assumption of the reader (a non-string + /// Node Type) is withheld and throws nothing. + [Fact] + public void APlanBodyThatThrowsInTheReader_IsWithheld_NotThrown() + { + var slab = DefaultPrefix + "LOG: duration: 15000.0 ms plan:\n\t{\"Plan\": {\"Node Type\": 1}}\n"; + + var plan = Assert.Single(StoreLogClassifier.Classify(slab).Groups); + + Assert.Equal(StoreLogClassifier.WithheldPlanMessage, plan.MessageText); + Assert.EndsWith("\n\t" + PgLogTextRedactor.WithheldPlan, plan.SampleLine, StringComparison.Ordinal); + } + + /// A legacy slow_statement row whose sample is a plan head is emptied, as it was before plans were + /// kept; it is not relabelled while staying in the slow-statement class. + [Fact] + public void ALegacySlowStatementRowHoldingAPlanHead_RemasksToNothing() + { + var sample = JsonPlanEntry("12345.678", "BindLeak5097c", "LitLeak5097d").Aggregate((a, b) => a + "\n" + b); + + Assert.Equal( + ((string?)null, (string?)null), + StoreLogClassifier.MaskStoredEvent(StoreLogClassifier.SlowStatementClass, "duration: 12345.678 ms plan:", sample)); + } + + /// A plan that cannot be read whole keeps its head and the withheld marker, and nothing of the plan: JSON + /// cut by a read boundary (the slab ends inside it), JSON that is not a plan, and JSON whose redacted form would + /// pass the sample cap. Each stays slow_plan, because the first line decides the class. + [Fact] + public void APlanThatCannotBeReadWhole_KeepsOnlyItsHead() + { + var entry = JsonPlanEntry("12000.0", "CutParam5097", "CutLit5097"); + var cut = string.Join("\n", entry.Take(6)) + "\n"; + var notAPlan = string.Join("\n", + [ + DefaultPrefix + "LOG: duration: 13000.0 ms plan:", + "\t{\"Anything\": \"NotAPlanLeak5097\"}", + "", + ]); + + foreach (var slab in new[] { cut, notAPlan }) + { + var census = StoreLogClassifier.Classify(slab); + var plan = Assert.Single(census.Groups); + Assert.Equal(StoreLogClassifier.SlowPlanClass, plan.EventClass); + Assert.Equal(StoreLogClassifier.WithheldPlanMessage, plan.MessageText); + Assert.EndsWith("\n\t" + PgLogTextRedactor.WithheldPlan, plan.SampleLine, StringComparison.Ordinal); + foreach (var planted in new[] { "CutParam5097", "CutLit5097", "NotAPlanLeak5097", "Seq Scan", "Query Text" }) + { + Assert.DoesNotContain(planted, plan.SampleLine, StringComparison.Ordinal); + } + } + + /* A slab whose first line is the tail of an entry the previous read classified opens no entry here. */ + var orphan = StoreLogClassifier.Classify(string.Join("\n", entry.Skip(1)) + "\n"); + Assert.Empty(orphan.Groups); + } + + /// A forged label inside a plan is only a string in a tab-led line: it opens no entry, so the plan stays + /// one slow_plan row and the forged severity is not read. + [Fact] + public void AForgedSeverityInsideAPlan_OpensNoEntry() + { + var entry = JsonPlanEntry("12000.0", "P", "ERROR: forged 5097 FATAL: x"); + var census = StoreLogClassifier.Classify(string.Join("\n", entry) + "\n"); + + Assert.Equal(1, census.EntriesRead); + var plan = Assert.Single(census.Groups); + Assert.Equal(StoreLogClassifier.SlowPlanClass, plan.EventClass); + Assert.DoesNotContain("forged 5097", plan.SampleLine, StringComparison.Ordinal); + } + + /// The remask pass and the read path run over rows already + /// stored: a slow_plan row comes back unchanged, one stored with a value in it (a build that kept raw + /// text) comes back without it, and a withheld row stays withheld. + [Fact] + public void MaskStoredEvent_IsIdempotentOnSlowPlanRows_AndRedactsAnUnredactedOne() + { + var census = StoreLogClassifier.Classify(string.Join("\n", JsonPlanEntry("12000.0", "IdemParam5097", "IdemLit5097")) + "\n"); + var plan = Assert.Single(census.Groups); + + var once = StoreLogClassifier.MaskStoredEvent(StoreLogClassifier.SlowPlanClass, plan.MessageText, plan.SampleLine); + Assert.Equal((plan.MessageText, plan.SampleLine), once); + Assert.Equal(once, StoreLogClassifier.MaskStoredEvent(StoreLogClassifier.SlowPlanClass, once.Message, once.Sample)); + + var raw = string.Join("\n", JsonPlanEntry("12000.0", "IdemParam5097", "IdemLit5097")); + var remasked = StoreLogClassifier.MaskStoredEvent(StoreLogClassifier.SlowPlanClass, "duration: 12000.0 ms plan:", raw); + Assert.Equal(plan.MessageText, remasked.Message); + Assert.DoesNotContain("IdemParam5097", remasked.Sample, StringComparison.Ordinal); + Assert.DoesNotContain("IdemLit5097", remasked.Sample, StringComparison.Ordinal); + + var withheld = StoreLogClassifier.Classify(DefaultPrefix + "LOG: duration: 8000.0 ms plan:\n\tQuery Text: SELECT 1\n"); + var kept = Assert.Single(withheld.Groups); + Assert.Equal( + (kept.MessageText, kept.SampleLine), + StoreLogClassifier.MaskStoredEvent(StoreLogClassifier.SlowPlanClass, kept.MessageText, kept.SampleLine)); + + /* A slow_plan row whose sample is not a plan head at all keeps no text. */ + Assert.Equal( + (null, null), + StoreLogClassifier.MaskStoredEvent(StoreLogClassifier.SlowPlanClass, "x", DefaultPrefix + "LOG: checkpoint starting: time")); } /// @@ -904,7 +1150,7 @@ census is exactly what a broken line split produces. */ Assert.True(census.EntriesRead > 0, "the fixture produced no entries at all"); Assert.True(census.Groups.Count > 0, "the fixture produced no groups at all"); - Assert.Equal(14, StoreLogClassifier.ClassNames.Count); + Assert.Equal(15, StoreLogClassifier.ClassNames.Count); var covered = census.Groups.Select(g => g.EventClass).ToHashSet(StringComparer.Ordinal); Assert.Equal( @@ -1616,6 +1862,99 @@ private static SortedSet Fingerprint(StoreLogClassifier.Census census) return set; } + [Fact] + public void MaskEntry_NamesTheConstraint_AndKeepsTheRowTupleOutOfTheStoredText() + { + var raw = DefaultPrefix + "ERROR: null value in column \"name\" of relation \"config_monitored_servers\" violates not-null constraint\n" + + DefaultPrefix + "DETAIL: Failing row contains (1, x, host-1, arn:aws:iam::123456789012:role/r, ext-secret-123, t).\n" + + DefaultPrefix + "STATEMENT: SELECT 1"; + + var masked = StoreLogClassifier.MaskEntry(raw); + + Assert.DoesNotContain("ext-secret-123", masked, StringComparison.Ordinal); + Assert.DoesNotContain("host-1", masked, StringComparison.Ordinal); + Assert.Contains("violates not-null constraint", masked, StringComparison.Ordinal); + Assert.Contains("DETAIL: Failing row contains (...).", masked, StringComparison.Ordinal); + Assert.Equal(masked, StoreLogClassifier.MaskEntry(masked)); + } + + [Theory] + [InlineData("Key (name)=(alpha) already exists.", "Key (name)=(...) already exists.")] + [InlineData("Key (host, port)=(db-1.example.test, 5432) already exists.", "Key (host, port)=(...) already exists.")] + [InlineData("Key (server_id)=(5) is not present in table \"config_monitored_servers\".", "Key (server_id)=(...) is not present in table \"config_monitored_servers\".")] + [InlineData("Key (name)=(a) already exists. and more (b) already exists.", "Key (name)=(...) already exists.")] + [InlineData("Key (name)=(...) already exists.", "Key (name)=(...) already exists.")] + [InlineData("Key (COALESCE(parent_id, 0), lower(name))=(0, prod) already exists.", "Key (COALESCE(parent_id, 0), lower(name))=(...) already exists.")] + [InlineData("Key (COALESCE(parent_id, 0), lower(name))=(0, prod", "Key (COALESCE(parent_id, 0), lower(name))=(...)")] + [InlineData("Key (COALESCE(parent_id, 0), lower(name))=(...) already exists.", "Key (COALESCE(parent_id, 0), lower(name))=(...) already exists.")] + [InlineData("Failing row contains (1, x, host-1).", "Failing row contains (...).")] + [InlineData("Partition key of the failing row contains (ts) = (2026-10-07 00:00:00).", "Partition key of the failing row contains (...).")] + [InlineData("FAILING ROW CONTAINS (1, x).", "FAILING ROW CONTAINS (...).")] + [InlineData("Process 12 waits for ShareLock on transaction 99; blocked by process 13.", "Process 12 waits for ShareLock on transaction 99; blocked by process 13.")] + public void MaskRowValues_ReplacesTheKeyValue_AndLeavesOtherProseAlone(string detail, string expected) + { + Assert.Equal(expected, StoreLogClassifier.MaskRowValues(detail)); + } + + [Fact] + public void MaskEntry_KeepsNoDetail_WhenTheStoresMessagesAreNotEnglish() + { + var raw = DefaultPrefix + "ERROR: duplicate key value violates unique constraint \"config_monitored_servers_name_key\"\n" + + DefaultPrefix + "DETAIL: Key (name)=(prod-1) already exists.\n" + + DefaultPrefix + "STATEMENT: SELECT 1"; + + var english = StoreLogClassifier.MaskEntry(raw, cut: false, detailInEnglish: true); + Assert.Contains("DETAIL: Key (name)=(...) already exists.", english, StringComparison.Ordinal); + Assert.Equal(StoreLogClassifier.MaskEntry(raw), english); + + var other = StoreLogClassifier.MaskEntry(raw, cut: false, detailInEnglish: false); + Assert.DoesNotContain("prod-1", other, StringComparison.Ordinal); + Assert.Contains("DETAIL: " + StoreLogClassifier.WithheldDetail, other, StringComparison.Ordinal); + Assert.Contains("violates unique constraint", other, StringComparison.Ordinal); + Assert.Equal(other, StoreLogClassifier.MaskEntry(other, cut: false, detailInEnglish: false)); + } + + [Fact] + public void Classify_KeepsNoDetailInTheSample_WhenTheStoresMessagesAreNotEnglish() + { + var slab = DefaultPrefix + "ERROR: duplicate key value violates unique constraint \"config_monitored_servers_name_key\"\n" + + DefaultPrefix + "DETAIL: Key (name)=(prod-1) already exists.\n"; + + var english = StoreLogClassifier.Classify(slab); + var other = StoreLogClassifier.Classify(slab, detailInEnglish: false); + + Assert.Contains(english.Groups, g => g.SampleLine?.Contains("Key (name)=(...)", StringComparison.Ordinal) == true); + Assert.DoesNotContain(other.Groups, g => g.SampleLine?.Contains("prod-1", StringComparison.Ordinal) == true); + Assert.Contains(other.Groups, g => g.SampleLine?.Contains(StoreLogClassifier.WithheldDetail, StringComparison.Ordinal) == true); + } + + [Theory] + [InlineData("C", true)] + [InlineData("POSIX", true)] + [InlineData("C.UTF-8", true)] + [InlineData("en_US.UTF-8", true)] + [InlineData("English_United States.1252", true)] + [InlineData("ja_JP.UTF-8", false)] + [InlineData("es_ES.UTF-8", false)] + [InlineData("German_Germany.1252", false)] + [InlineData("", false)] + [InlineData(null, false)] + public void IsEnglishLcMessages_ReadsTheLocaleName(string? value, bool expected) + { + Assert.Equal(expected, StoreLogClassifier.IsEnglishLcMessages(value)); + } + + [Fact] + public void TheSweep_ReadsTheStoresMessageLocaleOnce_AndPassesItToTheClassifier() + { + var sweep = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "StoreLogSweep.cs"); + + Assert.Single(System.Text.RegularExpressions.Regex.Matches(sweep, "SHOW lc_messages")); + Assert.Contains("StoreLogClassifier.Classify(slab.Text, detailInEnglish)", sweep, StringComparison.Ordinal); + Assert.Contains("StoreLogClassifier.IsEnglishLcMessages(value as string)", sweep, StringComparison.Ordinal); + Assert.Contains("catch (Exception ex) when (ex is not OperationCanceledException)", sweep[sweep.IndexOf("SHOW lc_messages", StringComparison.Ordinal)..], StringComparison.Ordinal); + } + private static string Render(string prefix, IEnumerable messages) { var slab = new StringBuilder(); diff --git a/Darling/Darling.Tests/StoreLogSeverityLocaleTests.cs b/Darling/Darling.Tests/StoreLogSeverityLocaleTests.cs index 3b99cae516..7874484adf 100644 --- a/Darling/Darling.Tests/StoreLogSeverityLocaleTests.cs +++ b/Darling/Darling.Tests/StoreLogSeverityLocaleTests.cs @@ -291,7 +291,7 @@ private static string ManagedConfBlocks() /// A stand-in value for a conf factory's parameter, or a failure naming the signature this scan /// cannot fill — never a skip, which would drop the block from the scan without saying so. - private static object Placeholder(MethodInfo factory, ParameterInfo parameter) + private static object? Placeholder(MethodInfo factory, ParameterInfo parameter) { if (parameter.ParameterType == typeof(int)) { @@ -311,6 +311,15 @@ still fails loudly here rather than being fed a value that means nothing to it. return "timescaledb"; } + /* #5459: BuildWalSizingConfAppend restates the max_wal_size already in force so a restart near a WAL size + step does not rewrite the file. A fresh store has no file in force, and that is the path the block's text + comes from, so null is the right stand-in. Keyed on type and name, so a different long? parameter still + fails loudly here. */ + if (parameter.ParameterType == typeof(long?) && parameter.Name == "inForceMaxWalSizeMb") + { + return null; + } + Assert.Fail( $"{factory.Name} takes a {parameter.ParameterType.Name} parameter '{parameter.Name}' this scan cannot " + "supply, so its block would go unscanned. Extend the placeholders rather than excluding the block."); diff --git a/Darling/Darling.Tests/StoreMaintenanceTickLaunchTests.cs b/Darling/Darling.Tests/StoreMaintenanceTickLaunchTests.cs new file mode 100644 index 0000000000..604419b812 --- /dev/null +++ b/Darling/Darling.Tests/StoreMaintenanceTickLaunchTests.cs @@ -0,0 +1,269 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using PerformanceMonitor.Darling.Analysis; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// #4970: the hourly store-maintenance tick (availability re-probe, compression job health, retention +/// re-evaluation, store-object convergence) used to be awaited inline in the collection launch loop, so a +/// slow pass held every sweep launch back once an hour. It now runs as a tracked background task, one at a +/// time, through . A never-signalled +/// stands in for the tick so "does not await" and "still running" are +/// directly observable. +/// +public sealed class StoreMaintenanceTickLaunchTests +{ + private static (DarlingWorker Worker, RecordingLogger Logger) MakeWorker() + { + var logger = new RecordingLogger(); + var worker = new DarlingWorker( + logger, + NullLoggerFactory.Instance, + new McpRuntimeState(), + new WebRuntimeState(), + new MonitoredServerRegistryState(), + new CollectorRuntimeState(), + new WebTlsCertificateState(), + new BaselineCache(), + new ReadLatencyAccumulator()); + return (worker, logger); + } + + [Fact] + public void TheLaunchLoop_DoesNotAwaitTheHourlyTenantsInline() + { + var worker = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "DarlingWorker.cs"); + var start = worker.IndexOf("while (!stoppingToken.IsCancellationRequested)", StringComparison.Ordinal); + var drain = worker.IndexOf("Shutdown drain for the fire-and-track bodies", StringComparison.Ordinal); + Assert.True(start >= 0 && drain > start, "could not locate the launch loop span"); + var loop = worker[start..drain]; + + foreach (var inline in new[] + { + "await ReprobeTimescaleAvailabilityAsync", + "await EvaluateCompressionJobHealthAsync", + "await ReevaluateRetentionPoliciesAsync", + "await ConvergeStoreObjectsAsync", + "await SweepStoreSelfMetricsAsync", + "_selfAlerts.EvaluateCollectorCostAsync", + "_selfAlerts.EvaluateToastSlackAsync", + "_selfAlerts.EvaluateCheckpointerPressureAsync", + "_selfAlerts.EvaluateFleetSweepRollupAsync", + "_selfAlerts.EvaluateAnalysisSinglesDigestAsync", + }) + { + Assert.DoesNotContain(inline, loop, StringComparison.Ordinal); + } + + Assert.Contains("TryStartStoreMaintenanceTick(", loop, StringComparison.Ordinal); + Assert.Contains("TryStartStoreMetricsTick(", loop, StringComparison.Ordinal); + + /* The deadlock re-mask key is assigned on the loop, ahead of the launch. */ + var key = loop.IndexOf("_pgDeadlockRemaskKey = runner.LogHashKey;", StringComparison.Ordinal); + Assert.True(key >= 0 && key < loop.IndexOf("TryStartStoreMetricsTick(", StringComparison.Ordinal)); + } + + [Fact] + public void TheMetricsLauncher_TakesTheWindowOnlyWhenItLaunches_AndDefersBehindMaintenance() + { + var worker = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "DarlingWorker.cs") + .ReplaceLineEndings("\n"); + var launcher = worker.IndexOf("internal bool TryStartStoreMetricsTick(", StringComparison.Ordinal); + Assert.True(launcher > 0); + var end = worker.IndexOf("\n }\n", launcher, StringComparison.Ordinal); + var body = worker[launcher..end]; + var defer = body.IndexOf("_storeMaintenanceTick is { IsCompleted: false }", StringComparison.Ordinal); + var stamp = body.IndexOf("_nextStoreMetricsUtc = NextGridStamp(_nextStoreMetricsUtc, nowUtc, s_storeMetricsInterval);", StringComparison.Ordinal); + var window = body.IndexOf("takeWindow()", StringComparison.Ordinal); + var launch = body.IndexOf("RunTrackedTickAsync(", StringComparison.Ordinal); + Assert.True(defer >= 0 && stamp > defer && window > stamp && launch > window, "defer < stamp < window < launch"); + Assert.Single(System.Text.RegularExpressions.Regex.Matches(worker, @"_checkpointSyncSampler\.TakeWindowMax")); + } + + private static DateTime NextStoreMetrics(DarlingWorker w) => + (DateTime)typeof(DarlingWorker).GetField("_nextStoreMetricsUtc", System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic)!.GetValue(w)!; + + private static readonly DateTime Now = new(2026, 10, 2, 12, 34, 5, DateTimeKind.Utc); + + [Fact] + public void MetricsLauncher_ReturnsImmediately_WhileTheTickIsStillRunning() + { + var (worker, _) = MakeWorker(); + var gate = new TaskCompletionSource(); + Assert.True(worker.TryStartStoreMetricsTick(Now, () => null, (_, _) => gate.Task, CancellationToken.None)); + gate.SetResult(); + } + + [Fact] + public void MetricsLauncher_DoesNotStartASecondTick_ButAdvancesTheStamp_AndWarns() + { + var (worker, logger) = MakeWorker(); + var gate = new TaskCompletionSource(); + Assert.True(worker.TryStartStoreMetricsTick(Now, () => null, (_, _) => gate.Task, CancellationToken.None)); + var taken = 0; + var secondRan = false; + var second = worker.TryStartStoreMetricsTick( + Now.AddHours(1), () => { taken++; return null; }, (_, _) => { secondRan = true; return Task.CompletedTask; }, CancellationToken.None); + + Assert.False(second); + Assert.False(secondRan); + Assert.Equal(0, taken); + Assert.True(NextStoreMetrics(worker) > Now); + Assert.Contains(logger.Entries, e => e.Level == LogLevel.Warning && e.Message.Contains("skipping this hour", StringComparison.Ordinal)); + gate.SetResult(); + } + + [Fact] + public async Task MetricsLauncher_StartsTheNextTick_AfterTheFirstCompletes() + { + var (worker, _) = MakeWorker(); + var gate = new TaskCompletionSource(); + Assert.True(worker.TryStartStoreMetricsTick(Now, () => null, (_, _) => gate.Task, CancellationToken.None)); + gate.SetResult(); + await Task.Delay(TimeSpan.FromMilliseconds(50)); + + var secondRan = false; + Assert.True(worker.TryStartStoreMetricsTick( + Now.AddHours(1), () => null, (_, _) => { secondRan = true; return Task.CompletedTask; }, CancellationToken.None)); + await Task.Delay(TimeSpan.FromMilliseconds(50)); + Assert.True(secondRan); + } + + [Fact] + public async Task AFaultingMetricsTick_IsObservedAndDoesNotBlockTheNext() + { + var (worker, logger) = MakeWorker(); + Assert.True(worker.TryStartStoreMetricsTick( + Now, () => null, (_, _) => throw new InvalidOperationException("boom"), CancellationToken.None)); + await Task.Delay(TimeSpan.FromMilliseconds(50)); + + Assert.Contains(logger.Entries, e => e.Level == LogLevel.Error && e.Exception is InvalidOperationException); + Assert.True(worker.TryStartStoreMetricsTick(Now.AddHours(1), () => null, (_, _) => Task.CompletedTask, CancellationToken.None)); + } + + [Fact] + public void DefersStoreMetrics_WhileTheMaintenanceTickRuns() + { + var (worker, _) = MakeWorker(); + var gate = new TaskCompletionSource(); + Assert.True(worker.TryStartStoreMaintenanceTick(_ => gate.Task, CancellationToken.None)); + var before = NextStoreMetrics(worker); + var taken = 0; + var ran = false; + + var launched = worker.TryStartStoreMetricsTick( + Now, () => { taken++; return null; }, (_, _) => { ran = true; return Task.CompletedTask; }, CancellationToken.None); + + /* Deferred: no launch, the window is not taken, and the stamp is unmoved so the next 15 s pass retries. */ + Assert.False(launched); + Assert.False(ran); + Assert.Equal(0, taken); + Assert.Equal(before, NextStoreMetrics(worker)); + + gate.SetResult(); + } + + [Fact] + public void ReturnsImmediately_WhileTheTickIsStillRunning() + { + var (worker, _) = MakeWorker(); + var gate = new TaskCompletionSource(); + + /* The gate is never signalled: returning at all proves the launcher did not await the tick. */ + Assert.True(worker.TryStartStoreMaintenanceTick(_ => gate.Task, CancellationToken.None)); + + gate.SetResult(); + } + + [Fact] + public void DoesNotStartASecondTick_WhileTheFirstIsRunning() + { + var (worker, logger) = MakeWorker(); + var gate = new TaskCompletionSource(); + var secondRan = false; + + Assert.True(worker.TryStartStoreMaintenanceTick(_ => gate.Task, CancellationToken.None)); + var second = worker.TryStartStoreMaintenanceTick( + _ => + { + secondRan = true; + return Task.CompletedTask; + }, + CancellationToken.None); + + Assert.False(second); + Assert.False(secondRan); + Assert.Contains(logger.Entries, e => e.Level == LogLevel.Warning && e.Message.Contains("skipping this hour", StringComparison.Ordinal)); + + gate.SetResult(); + } + + [Fact] + public async Task StartsTheNextTick_AfterTheFirstCompletes() + { + var (worker, _) = MakeWorker(); + var gate = new TaskCompletionSource(); + Assert.True(worker.TryStartStoreMaintenanceTick(_ => gate.Task, CancellationToken.None)); + gate.SetResult(); + await Task.Delay(TimeSpan.FromMilliseconds(50)); + + var secondRan = false; + Assert.True(worker.TryStartStoreMaintenanceTick( + _ => + { + secondRan = true; + return Task.CompletedTask; + }, + CancellationToken.None)); + await Task.Delay(TimeSpan.FromMilliseconds(50)); + Assert.True(secondRan); + } + + [Fact] + public async Task AFaultingTick_IsObservedAndDoesNotBlockTheNext() + { + var (worker, logger) = MakeWorker(); + Assert.True(worker.TryStartStoreMaintenanceTick( + _ => throw new InvalidOperationException("boom"), CancellationToken.None)); + await Task.Delay(TimeSpan.FromMilliseconds(50)); + + Assert.Contains(logger.Entries, e => e.Level == LogLevel.Error && e.Exception is InvalidOperationException); + Assert.True(worker.TryStartStoreMaintenanceTick(_ => Task.CompletedTask, CancellationToken.None)); + } + + private sealed class RecordingLogger : ILogger + { + public List<(LogLevel Level, string Message, Exception? Exception)> Entries { get; } = []; + + public IDisposable? BeginScope(TState state) where TState : notnull => null; + + public bool IsEnabled(LogLevel logLevel) => true; + + public void Log( + LogLevel logLevel, EventId eventId, TState state, Exception? exception, + Func formatter) + { + lock (Entries) + { + Entries.Add((logLevel, formatter(state, exception), exception)); + } + } + } +} diff --git a/Darling/Darling.Tests/StoreObjectConvergenceTests.cs b/Darling/Darling.Tests/StoreObjectConvergenceTests.cs index b7c4da166f..bca4a57ef2 100644 --- a/Darling/Darling.Tests/StoreObjectConvergenceTests.cs +++ b/Darling/Darling.Tests/StoreObjectConvergenceTests.cs @@ -598,6 +598,7 @@ await Assert.ThrowsAsync(() => DarlingWorker.RunStor /// (the positive control, without which "zero" could mean the log was not recording). The figures are in /// the PR body; what lives here is the shape that keeps them true. /// +[Trait("Stage", "Guard")] public sealed class CompressionEnableGuardTests { /// @@ -617,6 +618,12 @@ public void TheGuardsRead_AsksForEnabledAndSegmentBy_FromTheLongStableView_Scope Assert.Contains("cs.segmentby_column_index IS NOT NULL", sql, StringComparison.Ordinal); Assert.DoesNotContain("hypertable_compression_settings", sql, StringComparison.Ordinal); + /* The columns are joined with the separator the statement writes them with (#4951: collection_log's + segmentby has two columns), so a converged two-column setting reads back exactly as it was written. + Any other separator would read every converged pass as "not converged" and re-issue the ALTER, and its + ACCESS EXCLUSIVE lock, every hour. */ + Assert.Contains("string_agg(cs.attname, ', ' ORDER BY cs.segmentby_column_index)", sql, StringComparison.Ordinal); + /* Scoped, for the reason CompressionPolicyStateSql gives: a bring-your-own store may carry its own wait_stats hypertable in another schema, and this product must not read — let alone ALTER — it. */ Assert.Contains("h.hypertable_schema = 'collect'", sql, StringComparison.Ordinal); @@ -642,6 +649,12 @@ public void TheStatementAndTheComparison_ShareOneSegmentByConstant() TimescaleSupport.EnableCompressionSql("query_stats"), StringComparison.Ordinal); + /* #4951: collection_log alone segments by collector as well, and every other table keeps server_id. */ + Assert.Contains( + "timescaledb.compress_segmentby = 'server_id, collector_name'", + TimescaleSupport.EnableCompressionSql("collection_log"), + StringComparison.Ordinal); + /* The two halves are tied THROUGH the constant, not through a shared spelling: the statement interpolates it, and the guard compares against it. Both are asserted on the SOURCE, because a rendered statement cannot tell you whether the name came from the constant or from a literal that @@ -652,11 +665,13 @@ happens to agree with it today — which is the whole failure mode. A file-wide var raw = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "TimescaleSupport.cs"); /* The statement's half is read off the RAW source: the interpolation lives inside a string literal, which StripCommentsAndStrings blanks by design. */ - Assert.Contains("timescaledb.compress_segmentby = '{CompressionSegmentByColumn}'", raw, StringComparison.Ordinal); + /* #4951: the value is per table now (collection_log has its own), so both halves go through the one + per-table lookup rather than the shared column constant. */ + Assert.Contains("timescaledb.compress_segmentby = '{CompressionSegmentByFor(table)}'", raw, StringComparison.Ordinal); /* The guard's half is code, so it is read off the stripped source — prose about the comparison is not the comparison. */ Assert.Contains( - "string.Equals(segmentBy, CompressionSegmentByColumn, StringComparison.Ordinal)", + "string.Equals(segmentBy, CompressionSegmentByFor(table), StringComparison.Ordinal)", CSharpSourceWalker.StripCommentsAndStrings(raw), StringComparison.Ordinal); } @@ -706,6 +721,29 @@ applies to it as much as to the collector tables. */ Assert.Contains("ReadTablesNeedingCompressionEnableAsync(connection, logger, cancellationToken);", collectionLog, StringComparison.Ordinal); Assert.Contains("if (converged is null || !converged.Contains(CollectionLogTable))", collectionLog, StringComparison.Ordinal); + /* #4951: and collection_log's ALTER goes only through the bounded helper, which runs it inside its own + transaction and treats a lock it could not get as "try next pass" (the live + CollectionLogSegmentByLiveTests prove the wait is bounded; this keeps the shape in the unit tier). */ + Assert.Contains("TryRunBoundedDdlAsync(", collectionLog, StringComparison.Ordinal); + Assert.Contains("new[] { EnableCompressionSql(CollectionLogTable) }", collectionLog, StringComparison.Ordinal); + Assert.DoesNotContain("new NpgsqlCommand(EnableCompressionSql(CollectionLogTable)", collectionLog, StringComparison.Ordinal); + + /* Below TimescaleDB 2.14 the change cannot run while compressed chunks exist, so the version check comes + first and the ALTER is not attempted there. No live tier here runs a TimescaleDB that old, so the order + is pinned in this one. */ + var versionCheckAt = collectionLog.IndexOf("CollectionLogSettingsChangeBlockedAsync(connection, logger, cancellationToken)", StringComparison.Ordinal); + var alterAt = collectionLog.IndexOf("TryRunBoundedDdlAsync(", StringComparison.Ordinal); + Assert.True(versionCheckAt >= 0 && versionCheckAt < alterAt, + "EnsureCollectionLogHypertableAsync must check the TimescaleDB version and compressed chunks before it attempts the settings change"); + Assert.Contains("BoundedDdlOutcome.LockBusy", collectionLog, StringComparison.Ordinal); + /* The label is a string literal, which the stripped text above cannot hold: read it from the raw source of the same method. */ + var rawStorage = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "TimescaleSupport.cs").Replace("\r\n", "\n", StringComparison.Ordinal); + var rawStart = rawStorage.IndexOf("public static async Task EnsureCollectionLogHypertableAsync(", StringComparison.Ordinal); + Assert.True(rawStart >= 0, "could not locate EnsureCollectionLogHypertableAsync in the raw source"); + var rawEnd = rawStorage.IndexOf("\n }\n", rawStart, StringComparison.Ordinal); + Assert.True(rawEnd > rawStart, "could not find the end of EnsureCollectionLogHypertableAsync in the raw source"); + Assert.Contains("\"collection_log's compression settings\"", rawStorage[rawStart..rawEnd], StringComparison.Ordinal); + /* A read failure must issue every ALTER rather than skip every ALTER: the conservative direction, because a needless ALTER costs one lock and a skipped one costs a table that never compresses. Both call sites treat null as "not converged", which is what the two conditions above encode. */ @@ -861,3 +899,114 @@ private static async Task ExecuteAsync(NpgsqlConnection connection, string sql, await command.ExecuteNonQueryAsync(ct); } } + +/// +/// #5444, live: a convergence step whose failure makes Npgsql close the shared connection (XX000 is in the +/// classes Npgsql 10 closes on) must not fail the steps behind it on the same connection. +/// +[Collection("live-postgres")] +public sealed class StoreObjectConvergenceReopenLiveTests +{ + private static DarlingWorker.StoreObjectConvergenceStep Step(string name, Func body) => + new(name, DarlingWorker.StoreObjectConvergenceStage.Timescale, DarlingWorker.StoreObjectChangeSignal.Delta, + async (c, _, _) => { await body(c); return 0; }); + + private static async Task ExecAsync(NpgsqlConnection c, string sql) + { + await using var cmd = new NpgsqlCommand(sql, c); + await cmd.ExecuteNonQueryAsync(); + } + + [Theory] + [InlineData("XX000", true)] + [InlineData("P0001", false)] + public async Task AStepThatBreaksTheConnection_DoesNotFailTheStepAfterIt_AgainstDevPostgres(string sqlState, bool breaksConnection) + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live convergence reopen test."); + + var ct = TestContext.Current.CancellationToken; + var bodySucceeded = false; + try + { + using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + var logger = new CapturingTestLogger(); + var tally = new DarlingWorker.StoreObjectConvergenceTally(); + var stateAfterA = (System.Data.ConnectionState?)null; + + var a = Step("step A", async c => + { + try { await ExecAsync(c, $"DO $$ BEGIN RAISE EXCEPTION 'planted #5444' USING ERRCODE = '{sqlState}'; END $$"); } + catch { stateAfterA = c.State; throw; } + }); + var bRan = false; + var b = Step("step B", async c => { await ExecAsync(c, "SELECT 1"); bRan = true; }); + + await DarlingWorker.RunStoreObjectConvergenceStepAsync(connection, a, tally, logger, ct); + await DarlingWorker.RunStoreObjectConvergenceStepAsync(connection, b, tally, logger, ct); + + Assert.Equal(breaksConnection, stateAfterA != System.Data.ConnectionState.Open); + Assert.True(bRan, "step B must run on the reopened connection: " + logger.Joined); + Assert.Equal(new[] { "step A" }, tally.Failed); + Assert.Equal(2, tally.Steps); + Assert.Equal(System.Data.ConnectionState.Open, connection.State); + Assert.Equal(breaksConnection ? 1 : 0, logger.Lines.Count(l => l.Contains("it was reopened", StringComparison.Ordinal))); + Assert.DoesNotContain("Connection is not open", logger.Joined, StringComparison.Ordinal); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, bodySucceeded, (_, _) => Task.CompletedTask); + } + } + + /// + /// The common shape: a step that isolates its own statements (as the reshape drop does) catches the + /// connection-breaking error and RETURNS normally with the connection closed. The runner must reopen it + /// anyway, so the next step runs, and the step is not counted failed because it did not throw. + /// + [Fact] + public async Task AStepThatSwallowsAConnectionBreakingError_StillLeavesAnOpenConnectionForTheNextStep_AgainstDevPostgres() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live convergence reopen test."); + + var ct = TestContext.Current.CancellationToken; + var bodySucceeded = false; + try + { + using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + var logger = new CapturingTestLogger(); + var tally = new DarlingWorker.StoreObjectConvergenceTally(); + var stateAfterA = (System.Data.ConnectionState?)null; + + var a = Step("step A", async c => + { + try { await ExecAsync(c, "DO $$ BEGIN RAISE EXCEPTION 'planted #5444' USING ERRCODE = 'XX000'; END $$"); } + catch (PostgresException) { stateAfterA = c.State; } + }); + var bRan = false; + var b = Step("step B", async c => { await ExecAsync(c, "SELECT 1"); bRan = true; }); + + await DarlingWorker.RunStoreObjectConvergenceStepAsync(connection, a, tally, logger, ct); + await DarlingWorker.RunStoreObjectConvergenceStepAsync(connection, b, tally, logger, ct); + + Assert.NotEqual(System.Data.ConnectionState.Open, stateAfterA); + Assert.True(bRan, "step B must run on the reopened connection: " + logger.Joined); + Assert.Empty(tally.Failed); + Assert.Equal(2, tally.Steps); + Assert.Equal(System.Data.ConnectionState.Open, connection.State); + Assert.Equal(1, logger.Lines.Count(l => l.Contains("step 'step A' left the store connection closed; it was reopened", StringComparison.Ordinal))); + Assert.DoesNotContain("Connection is not open", logger.Joined, StringComparison.Ordinal); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, bodySucceeded, (_, _) => Task.CompletedTask); + } + } +} diff --git a/Darling/Darling.Tests/StoreObjectHourlyStepSelectionTests.cs b/Darling/Darling.Tests/StoreObjectHourlyStepSelectionTests.cs new file mode 100644 index 0000000000..e43e3ff64f --- /dev/null +++ b/Darling/Darling.Tests/StoreObjectHourlyStepSelectionTests.cs @@ -0,0 +1,73 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using PerformanceMonitor.Darling.Service; +using Xunit; + +namespace Darling.Tests; + +/// +/// The hourly convergence pass runs a step's HourlyEnsureAsync when it has one and the step's +/// EnsureAsync otherwise; the start path always runs EnsureAsync. Each call runs its own +/// delegate and not the other. +/// +public sealed class StoreObjectHourlyStepSelectionTests +{ + private static DarlingWorker.StoreObjectConvergenceStep Step(List ran, bool withHourly) => + new("fake ensure", DarlingWorker.StoreObjectConvergenceStage.Tuning, DarlingWorker.StoreObjectChangeSignal.InPlace, + (_, _, _) => { ran.Add("start"); return Task.FromResult(0); }, + withHourly ? (_, _, _) => { ran.Add("hourly"); return Task.FromResult(0); } : null); + + [Fact] + public async Task EachPass_RunsItsOwnDelegate_AndNotTheOther() + { + var ran = new List(); + var step = Step(ran, withHourly: true); + var logger = new CapturingTestLogger(); + var ct = TestContext.Current.CancellationToken; + + await DarlingWorker.RunStoreObjectConvergenceStepAsync(null!, step, new DarlingWorker.StoreObjectConvergenceTally(), logger, ct, hourly: true); + Assert.Equal(new[] { "hourly" }, ran); + + ran.Clear(); + await DarlingWorker.RunStoreObjectConvergenceStepAsync(null!, step, new DarlingWorker.StoreObjectConvergenceTally(), logger, ct); + Assert.Equal(new[] { "start" }, ran); + } + + [Fact] + public async Task AStepWithNoHourlyDelegate_RunsItsEnsureOnTheHourlyPass() + { + var ran = new List(); + await DarlingWorker.RunStoreObjectConvergenceStepAsync( + null!, Step(ran, withHourly: false), new DarlingWorker.StoreObjectConvergenceTally(), + new CapturingTestLogger(), TestContext.Current.CancellationToken, hourly: true); + Assert.Equal(new[] { "start" }, ran); + } + + [Fact] + public void TheHourlyTickPassesHourlyTrue_AndThePerformanceTuningStepCarriesAnHourlyDelegate() + { + var src = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "DarlingWorker.cs") + .ReplaceLineEndings("\n"); + + var start = src.IndexOf("private async Task ConvergeStoreObjectsAsync(", StringComparison.Ordinal); + Assert.True(start >= 0); + var end = src.IndexOf("\n }\n", start, StringComparison.Ordinal); + var body = src.Substring(start, end - start); + Assert.Contains("RunStoreObjectConvergenceStepAsync(connection, step, tally, _logger, budget.Token, hourly: true)", body, StringComparison.Ordinal); + + var entry = src.IndexOf("new(\"composer performance tuning\"", StringComparison.Ordinal); + Assert.True(entry >= 0); + var entryEnd = src.IndexOf("\n };", entry, StringComparison.Ordinal); + Assert.Contains("PgTableTuning.ApplyAsync(connection, logger, hourly: true, ct)", src.Substring(entry, entryEnd - entry), StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/StoreQueryHistoryToolLiveTests.cs b/Darling/Darling.Tests/StoreQueryHistoryToolLiveTests.cs new file mode 100644 index 0000000000..126e3afcee --- /dev/null +++ b/Darling/Darling.Tests/StoreQueryHistoryToolLiveTests.cs @@ -0,0 +1,507 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Globalization; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// get_store_query_history (#5097) against a real server. The history is planted directly in the three tables, so +/// the figures are exactly what each fact says. Each fact mints its own scratch database. +/// +[Collection("live-postgres")] +public sealed class StoreQueryHistoryToolLiveTests +{ + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static readonly DateTime Now = DateTime.UtcNow; + + private static string Ts(double hoursAgo) => + "'" + DateTime.SpecifyKind(Now.AddHours(-hoursAgo), DateTimeKind.Unspecified).ToString("yyyy-MM-dd HH:mm:ss.ffffff", CultureInfo.InvariantCulture) + "'"; + + private static async Task<(ScratchPostgres Scratch, NpgsqlDataSource Source)> StartAsync(CancellationToken ct, bool migrate = true) + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), "Set DARLING_TEST_PG to run the statement history reader's live pins (each mints its own scratch database)."); + var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + if (migrate) + { + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + } + + return (scratch, NpgsqlDataSource.Create(scratch.ConnectionString)); + } + + private static async Task ExecAsync(NpgsqlDataSource source, string sql, CancellationToken ct) + { + await using var command = source.CreateCommand(sql); + await command.ExecuteNonQueryAsync(ct); + } + + private static Task Capture(NpgsqlDataSource s, double hoursAgo, string outcome, CancellationToken ct, int dealloc = 0, int seen = 3, int kept = 3) => + ExecAsync(s, $"INSERT INTO collect.store_statement_captures VALUES ({Ts(hoursAgo)}, 3600, NULL, 0, {dealloc}, {seen}, {kept}, 0, '{outcome}')", ct); + + private static Task Row(NpgsqlDataSource s, double hoursAgo, long queryId, long calls, double totalMs, string flags, CancellationToken ct, string role = "owner") => + ExecAsync(s, $"INSERT INTO collect.store_statement_history VALUES ({Ts(hoursAgo)}, 3600, '{role}', {queryId}, {calls}, {totalMs.ToString(CultureInfo.InvariantCulture)}, 1, 2, 3, 4, 9.5, {flags})", ct); + + private const string None = "false, false, false"; + + [Fact] + public async Task TheSeries_IsOldestFirst_WithTheFlags_AndAMeanOfDeltaTotalOverDeltaCalls() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, source) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = source; + var ok = false; + try + { + await Capture(source, 3.5, "ok", ct); + await Capture(source, 2.5, "ok", ct); + await Capture(source, 1.5, "ok", ct); + await Row(source, 1.5, 9007199254740993, 4, 40, None, ct); + await Row(source, 3.5, 9007199254740993, 10, 100, "true, false, false", ct); + await Row(source, 2.5, 9007199254740993, 5, 150, "false, false, true", ct); + + var json = await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(source, query_id: "9007199254740993", cancellationToken: ct); + using var doc = JsonDocument.Parse(json); + var root = doc.RootElement; + + Assert.Equal("series", root.GetProperty("mode").GetString()); + Assert.Equal("9007199254740993", root.GetProperty("query_id").GetString()); + var series = root.GetProperty("series").EnumerateArray().ToArray(); + Assert.Equal(3, series.Length); + Assert.Equal(new[] { 10L, 5L, 4L }, series.Select(r => r.GetProperty("calls").GetInt64()).ToArray()); + Assert.Equal(new[] { 10d, 30d, 10d }, series.Select(r => r.GetProperty("mean_ms").GetDouble()).ToArray()); + Assert.True(series[0].GetProperty("first_seen").GetBoolean()); + Assert.True(series[1].GetProperty("reset_in_interval").GetBoolean()); + Assert.False(series[2].GetProperty("entry_restarted").GetBoolean()); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task TheRanking_IsByTotalTime_Truncates_AndPrintsTheQueryIdAsAString() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, source) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = source; + var ok = false; + try + { + await Capture(source, 2.5, "ok", ct); + await Capture(source, 1.5, "ok", ct); + await Row(source, 2.5, 1, 10, 10, None, ct); + await Row(source, 1.5, 1, 10, 10, None, ct); + await Row(source, 2.5, 2, 1, 500, None, ct); + await Row(source, 2.5, 3, 100, 90, None, ct, role: "viewer"); + + var json = await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(source, top: 2, cancellationToken: ct); + using var doc = JsonDocument.Parse(json); + var root = doc.RootElement; + + Assert.Equal("ranked", root.GetProperty("mode").GetString()); + Assert.True(root.GetProperty("truncated").GetBoolean()); + var rows = root.GetProperty("statements").EnumerateArray().ToArray(); + Assert.Equal(2, rows.Length); + Assert.Equal(JsonValueKind.String, rows[0].GetProperty("query_id").ValueKind); + Assert.Equal("2", rows[0].GetProperty("query_id").GetString()); + Assert.Equal("3", rows[1].GetProperty("query_id").GetString()); + Assert.Equal("viewer", rows[1].GetProperty("role").GetString()); + + var all = JsonDocument.Parse(await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(source, top: 5, cancellationToken: ct)).RootElement; + Assert.False(all.GetProperty("truncated").GetBoolean()); + var one = all.GetProperty("statements").EnumerateArray().First(r => r.GetProperty("query_id").GetString() == "1"); + Assert.Equal(20L, one.GetProperty("calls").GetInt64()); + Assert.Equal(1d, one.GetProperty("mean_ms").GetDouble()); + Assert.Equal(2, one.GetProperty("hours_present").GetInt32()); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task AStoreWithoutTheRung_AnswersPrecondition() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, source) = await StartAsync(ct, migrate: false); + await using var _ = scratch; + await using var __ = source; + var ok = false; + try + { + var json = await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(source, cancellationToken: ct); + + Assert.StartsWith("{\"status\":\"precondition\"", json, StringComparison.Ordinal); + Assert.DoesNotContain("effective_start", json, StringComparison.Ordinal); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task AStoreWithNoCapturesYet_AnswersPrecondition_Bare() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, source) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = source; + var ok = false; + try + { + var json = await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(source, cancellationToken: ct); + + Assert.StartsWith("{\"status\":\"precondition\"", json, StringComparison.Ordinal); + Assert.DoesNotContain("window_truncated", json, StringComparison.Ordinal); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task TheCapturesSummary_CountsRebaselinedHours_AndTheEvictions() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, source) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = source; + var ok = false; + try + { + await Capture(source, 3.5, "ok", ct, dealloc: 2); + await Capture(source, 2.5, "rebaselined", ct, seen: 0, kept: 0); + await Capture(source, 1.5, "ok", ct, dealloc: 3, seen: 150, kept: 100); + await Row(source, 1.5, 7, 1, 1, None, ct); + + var root = JsonDocument.Parse(await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(source, cancellationToken: ct)).RootElement; + var captures = root.GetProperty("captures"); + + Assert.Equal(3L, captures.GetProperty("count").GetInt64()); + Assert.Equal(1L, captures.GetProperty("rebaselined").GetInt64()); + Assert.Equal(5L, captures.GetProperty("dealloc_delta_total").GetInt64()); + Assert.Equal(1L, captures.GetProperty("capped_hours").GetInt64()); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task HistoryStartingInsideTheWindow_IsTruncated_AtTheFirstCapture_AndTheKeysComeAfterHoursBack() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, source) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = source; + var ok = false; + try + { + await Capture(source, 5.5, "rebaselined", ct, seen: 0, kept: 0); + await Capture(source, 4.5, "ok", ct); + await Row(source, 4.5, 7, 1, 1, None, ct); + + var json = await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(source, hours_back: 48, cancellationToken: ct); + var root = JsonDocument.Parse(json).RootElement; + + Assert.True(root.GetProperty("window_truncated").GetBoolean()); + var start = DateTime.Parse(root.GetProperty("effective_start").GetString()!, CultureInfo.InvariantCulture, DateTimeStyles.AdjustToUniversal); + Assert.InRange((Now.AddHours(-5.5) - start).Duration().TotalSeconds, 0, 2); + Assert.EndsWith("Z", root.GetProperty("effective_start").GetString(), StringComparison.Ordinal); + Assert.Contains("History is kept 90 days and taken hourly.", root.GetProperty("truncation_note").GetString(), StringComparison.Ordinal); + + var names = root.EnumerateObject().Select(p => p.Name).ToArray(); + Assert.Equal("hours_back", names[1]); + Assert.Equal(new[] { "effective_start", "window_truncated", "truncation_note" }, names.Skip(2).Take(3).ToArray()); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task HistoryFromBeforeTheWindow_IsNotTruncated() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, source) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = source; + var ok = false; + try + { + await Capture(source, 100, "ok", ct); + await Capture(source, 1.5, "ok", ct); + await Row(source, 1.5, 7, 1, 1, None, ct); + + var root = JsonDocument.Parse(await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(source, hours_back: 24, cancellationToken: ct)).RootElement; + + Assert.False(root.GetProperty("window_truncated").GetBoolean()); + Assert.Equal(JsonValueKind.Null, root.GetProperty("truncation_note").ValueKind); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task AnEmptyWindow_CarriesTheKeysUnderHints() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, source) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = source; + var ok = false; + try + { + await Capture(source, 100, "ok", ct); + + var root = JsonDocument.Parse(await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(source, hours_back: 24, cancellationToken: ct)).RootElement; + + Assert.Equal("empty", root.GetProperty("status").GetString()); + var hints = root.GetProperty("hints"); + Assert.False(hints.GetProperty("window_truncated").GetBoolean()); + Assert.Contains(hints.EnumerateObject(), p => p.Name == "effective_start"); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task AFailedEarliestCaptureRead_CostsTheKeys_NotTheAnswer() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, source) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = source; + var ok = false; + try + { + await Capture(source, 1.5, "ok", ct); + await Row(source, 1.5, 7, 1, 1, None, ct); + + DarlingMcpStoreQueryHistoryTools.TestOnlyFailEarliestRead = true; + /* The switch is per async flow, so it dies with this test; it is cleared right after the call all the same. */ + var json = await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(source, cancellationToken: ct); + DarlingMcpStoreQueryHistoryTools.TestOnlyFailEarliestRead = false; + + var root = JsonDocument.Parse(json).RootElement; + Assert.Equal("ranked", root.GetProperty("mode").GetString()); + Assert.DoesNotContain(root.EnumerateObject(), p => p.Name == "effective_start"); + Assert.DoesNotContain(root.EnumerateObject(), p => p.Name == "window_truncated"); + Assert.Single(root.GetProperty("statements").EnumerateArray()); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task TheRecentAndEarlierMeans_SplitTheNewestQuarter_AndAreNullUnderFourHours() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, source) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = source; + var ok = false; + try + { + for (var h = 8; h >= 1; h--) + { + await Capture(source, h + 0.5, "ok", ct); + } + + /* Statement 1: eight hours; the newest two (25% of 8) cost 50 ms/call, the six before cost 10 ms/call. */ + for (var h = 8; h >= 3; h--) + { + await Row(source, h + 0.5, 1, 10, 100, None, ct); + } + + await Row(source, 2.5, 1, 10, 500, None, ct); + await Row(source, 1.5, 1, 10, 500, None, ct); + /* Statement 2: three hours, too few to split. */ + for (var h = 3; h >= 1; h--) + { + await Row(source, h + 0.5, 2, 1, 1, None, ct); + } + + var root = JsonDocument.Parse(await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(source, cancellationToken: ct)).RootElement; + var rows = root.GetProperty("statements").EnumerateArray().ToArray(); + var one = rows.First(r => r.GetProperty("query_id").GetString() == "1"); + Assert.Equal(50d, one.GetProperty("recent_mean_ms").GetDouble()); + Assert.Equal(10d, one.GetProperty("earlier_mean_ms").GetDouble()); + var two = rows.First(r => r.GetProperty("query_id").GetString() == "2"); + Assert.Equal(JsonValueKind.Null, two.GetProperty("recent_mean_ms").ValueKind); + Assert.Equal(JsonValueKind.Null, two.GetProperty("earlier_mean_ms").ValueKind); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task AFirstSeenHour_IsCounted_ButNeverFakesASlowdown_AndTheFlagsAreCountedPerStatement() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, source) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = source; + var ok = false; + try + { + for (var h = 8; h >= 1; h--) + { + await Capture(source, h + 0.5, "ok", ct); + } + + /* Steady 10 ms/call for seven hours; the first hour is a first_seen upper bound that credits a lifetime at 1000 ms/call. */ + await Row(source, 8.5, 7, 10, 10000, "true, false, false", ct); + for (var h = 7; h >= 3; h--) + { + await Row(source, h + 0.5, 7, 10, 100, h == 5 ? "false, true, false" : None, ct); + } + + await Row(source, 2.5, 7, 10, 100, "false, false, true", ct); + await Row(source, 1.5, 7, 10, 100, None, ct); + + var root = JsonDocument.Parse(await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(source, cancellationToken: ct)).RootElement; + var s7 = root.GetProperty("statements").EnumerateArray().First(r => r.GetProperty("query_id").GetString() == "7"); + Assert.Equal(1, s7.GetProperty("first_seen_hours").GetInt32()); + Assert.Equal(1, s7.GetProperty("restarted_hours").GetInt32()); + Assert.Equal(1, s7.GetProperty("reset_hours").GetInt32()); + Assert.Equal(8, s7.GetProperty("hours_present").GetInt32()); + Assert.Equal(10d, s7.GetProperty("recent_mean_ms").GetDouble()); + Assert.Equal(10d, s7.GetProperty("earlier_mean_ms").GetDouble()); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task TheRoleFilter_KeepsOnlyThatRolesRows_InBothModes() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, source) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = source; + var ok = false; + try + { + await Capture(source, 1.5, "ok", ct); + await Row(source, 1.5, 5, 1, 10, None, ct, role: "owner"); + await Row(source, 1.5, 5, 3, 30, None, ct, role: "viewer"); + + var ranked = JsonDocument.Parse(await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(source, role: "viewer", cancellationToken: ct)).RootElement; + var only = Assert.Single(ranked.GetProperty("statements").EnumerateArray()); + Assert.Equal("viewer", only.GetProperty("role").GetString()); + Assert.Equal(3L, only.GetProperty("calls").GetInt64()); + + var series = JsonDocument.Parse(await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(source, query_id: "5", role: "owner", cancellationToken: ct)).RootElement; + var row = Assert.Single(series.GetProperty("series").EnumerateArray()); + Assert.Equal(1L, row.GetProperty("calls").GetInt64()); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task AStoreWhoseCapturesAllCouldNotReadTheExtension_AnswersPrecondition_WithTheReason() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, source) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = source; + var ok = false; + try + { + await Capture(source, 2.5, "precondition", ct); + await Capture(source, 1.5, "precondition", ct); + + var json = await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(source, cancellationToken: ct); + Assert.StartsWith("{\"status\":\"precondition\"", json, StringComparison.Ordinal); + Assert.Contains("could not read pg_stat_statements", json, StringComparison.Ordinal); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task AnUnusableReaderNow_StillAnswersTheHistory_WithTextUnavailable_AndATextNote() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, source) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = source; + var ok = false; + try + { + await Capture(source, 1.5, "ok", ct); + await Row(source, 1.5, 11, 2, 20, None, ct); + + /* A scratch database has no pg_stat_statements loaded, so the reader is not usable now. */ + var ranked = JsonDocument.Parse(await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(source, cancellationToken: ct)).RootElement; + Assert.Equal("ranked", ranked.GetProperty("mode").GetString()); + var note = ranked.GetProperty("text_note").GetString(); + Assert.False(string.IsNullOrEmpty(note)); + Assert.Equal("text unavailable", ranked.GetProperty("statements")[0].GetProperty("query").GetString()); + + var series = JsonDocument.Parse(await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(source, query_id: "11", cancellationToken: ct)).RootElement; + Assert.Equal("text unavailable", series.GetProperty("query").GetString()); + Assert.Equal(note, series.GetProperty("text_note").GetString()); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } +} diff --git a/Darling/Darling.Tests/StoreQueryHistoryToolTests.cs b/Darling/Darling.Tests/StoreQueryHistoryToolTests.cs new file mode 100644 index 0000000000..b9bd3a8703 --- /dev/null +++ b/Darling/Darling.Tests/StoreQueryHistoryToolTests.cs @@ -0,0 +1,68 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service.Mcp; +using Xunit; + +namespace Darling.Tests; + +/// The refusals of get_store_query_history (#5097), which are decided before any read. +public sealed class StoreQueryHistoryToolTests +{ + [Theory] + [InlineData(0, 24, "top")] + [InlineData(1001, 24, "top")] + [InlineData(20, 0, "hours_back")] + [InlineData(20, 2161, "hours_back")] + public async Task ABadTopOrWindow_IsRefused_NamingTheParameter(int top, int hours, string parameter) + { + await using var unreachable = NpgsqlDataSource.Create("Host=127.0.0.1;Port=1;Username=nobody;Password=nothing;Timeout=1"); + + var result = await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(unreachable, top: top, hours_back: hours, cancellationToken: TestContext.Current.CancellationToken); + + Assert.StartsWith("{\"status\":\"invalid\"", result, StringComparison.Ordinal); + Assert.Contains($"\"parameter\":\"{parameter}\"", result, StringComparison.Ordinal); + } + + [Fact] + public void TheTextRead_KeepsTheSharedSensitiveStatementPredicate_OnTopOfTheReaderFunctionsOwnFilter() + { + /* #5097: the diagnostics bundle applied the shared predicate in its own statement-text read and now reads the text through + this tool, so the tool's read applies it: a store whose reader function is older than the pattern still withholds. The + predicate wraps the text the read chooses (max(query) per query id), so it is the shown text that is tested. */ + Assert.Contains(PgSensitiveStatementFilter.SqlPredicate("max(f.query)"), DarlingMcpStoreQueryHistoryTools.TextSql, StringComparison.Ordinal); + } + + [Fact] + public async Task ABadRole_IsRefused_ListingTheRoles() + { + await using var unreachable = NpgsqlDataSource.Create("Host=127.0.0.1;Port=1;Username=nobody;Password=nothing;Timeout=1"); + + var result = await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(unreachable, role: "darling", cancellationToken: TestContext.Current.CancellationToken); + + Assert.Contains("\"parameter\":\"role\"", result, StringComparison.Ordinal); + Assert.Contains("viewer", result, StringComparison.Ordinal); + } + + [Theory] + [InlineData("abc")] + [InlineData("1.5")] + [InlineData("99999999999999999999")] + public async Task ANonNumericQueryId_IsRefused(string queryId) + { + await using var unreachable = NpgsqlDataSource.Create("Host=127.0.0.1;Port=1;Username=nobody;Password=nothing;Timeout=1"); + + var result = await DarlingMcpStoreQueryHistoryTools.GetStoreQueryHistory(unreachable, query_id: queryId, cancellationToken: TestContext.Current.CancellationToken); + + Assert.Contains("\"parameter\":\"query_id\"", result, StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/StoreSelfMetricsTests.cs b/Darling/Darling.Tests/StoreSelfMetricsTests.cs index 7b8c973d2b..eafc931d3b 100644 --- a/Darling/Darling.Tests/StoreSelfMetricsTests.cs +++ b/Darling/Darling.Tests/StoreSelfMetricsTests.cs @@ -189,7 +189,7 @@ TimescaleSupport detection and a plain-PG store skips it silently. */ } [Fact] - public void DimensionInsertSql_CoversBothPayloadDims_WithTotalRelationSizeAndExactRowCount() + public void DimensionInsertSql_CoversBothPayloadDims_WithTotalRelationSizeAndTheEstimatedRowCount() { var sql = StoreSelfMetrics.DimensionInsertSql; @@ -201,7 +201,9 @@ biggest forecasting term a stored series. pg_total_relation_size because the pla Assert.Contains(PayloadDimensions.QueryPlanDimTable, sql, StringComparison.Ordinal); Assert.Contains("pg_total_relation_size", sql, StringComparison.Ordinal); Assert.Contains("'dimension'", sql, StringComparison.Ordinal); - Assert.Contains("count(*)", sql, StringComparison.Ordinal); + /* #5520: the planner's estimate, NULL where it is -1, never a count(*) over a table this size every hour. */ + Assert.Contains("CASE WHEN c.reltuples >= 0 THEN c.reltuples::bigint END", sql, StringComparison.Ordinal); + Assert.DoesNotContain("count(*)", sql, StringComparison.Ordinal); } [Fact] @@ -302,9 +304,19 @@ public void TableInsertSql_NamesTheSevenProductTables_AndTheCensusExcludesExactl foreach (var table in qualified) { Assert.Contains($"'{table}',", sql, StringComparison.Ordinal); - Assert.Contains($"pg_total_relation_size('{table}')", sql, StringComparison.Ordinal); - /* The planner's estimate, NULL where it is -1 (never analysed) — never a 15 GiB count(*) an hour. */ - Assert.Contains($"(SELECT CASE WHEN c.reltuples >= 0 THEN c.reltuples::bigint END FROM pg_class c WHERE c.oid = '{table}'::regclass)", sql, StringComparison.Ordinal); + if (table is "collect." + QueryStoreIntervalLatest.TableName or "collect." + QueryStoreIntervalWide.TableName) + { + /* #5571: a day-partitioned parent has no storage of its own (its size reads 0), so its row sums the + LEAF partitions through pg_partition_tree, bytes and the reltuples estimate alike. */ + Assert.Contains($"COALESCE((SELECT sum(pg_total_relation_size(t.relid)) FROM pg_partition_tree('{table}'::regclass) AS t WHERE t.isleaf), pg_total_relation_size('{table}'))::bigint", sql, StringComparison.Ordinal); + Assert.Contains($"(SELECT sum(c.reltuples)::bigint FROM pg_class AS c WHERE c.reltuples >= 0 AND (c.oid IN (SELECT t.relid FROM pg_partition_tree('{table}'::regclass) AS t WHERE t.isleaf) OR", sql, StringComparison.Ordinal); + } + else + { + Assert.Contains($"pg_total_relation_size('{table}')", sql, StringComparison.Ordinal); + /* The planner's estimate, NULL where it is -1 (never analysed) — never a 15 GiB count(*) an hour. */ + Assert.Contains($"(SELECT CASE WHEN c.reltuples >= 0 THEN c.reltuples::bigint END FROM pg_class c WHERE c.oid = '{table}'::regclass)", sql, StringComparison.Ordinal); + } /* The census names the same table by the same compound constant, compared against the concatenated schema.relation — no hand-typed (schema, relation) tuple to drift (review catch). */ @@ -380,6 +392,15 @@ without updating one of the three lists fails here — the census's whole point. ["collect.store_log_captures"] = "self-telemetry about the store's own logging, not collected monitoring data", ["collect.read_latency"] = "internal self-telemetry (the read-path histogram), not collected monitoring data (V148 doc)", ["collect.pg_statement_text"] = "dimension-shaped content keyed to facts rather than collected as facts, pruned on last_seen (V73 doc)", + ["collect.query_stats_hour_ledger"] = "the hourly row count of query_stats the count guard reads, one row per server per hour and pruned to the hourly rollup's horizon, not collected monitoring data (V164 doc)", + ["collect.query_stats_hour_ledger_state"] = "one row saying from which hour the ledger is complete, bytes too small to matter (V164 doc)", + ["collect.store_statement_captures"] = "internal self-telemetry (the store's statement history captures), not collected monitoring data (V163 doc)", + ["collect.store_statement_history"] = "internal self-telemetry (the store's statement history), pruned to 90 days by its own writer, not collected monitoring data (V163 doc)", + ["collect.slow_reads"] = "internal self-telemetry (the slow-read record), not collected monitoring data (V162 doc)", + ["collect.query_store_top_daily"] = "the daily summary the Query Store top read uses, approximate by design and rebuilt from the interval table (V161 doc)", + ["collect.plan_regression_daily"] = "the per-day plan totals PLAN_REGRESSION reads for closed days, rebuilt from the interval table (V168 doc)", + ["collect.plan_regression_daily_built"] = "per-day build and validity bookkeeping for the per-day plan totals, bytes too small to matter (V168 doc)", + ["collect.query_store_top_daily_built"] = "per-day build bookkeeping for the daily summary, bytes too small to matter (V161 doc)", ["collect.plan_force_actions"] = "the force-plan bot's append-only audit ledger, not collected monitoring data (V107 doc)", }; @@ -563,6 +584,98 @@ public void NoDarlingProductSqlReadsTheChunkCatalog() Assert.Empty(offenders); } + /// + /// #5520: the dimension rows' row_count is the planner's estimate, and the comment on the dimension + /// insert in says no reader computes from it. This pins that claim: in the + /// product source, a use of a store-metrics row's RowCount is a record parameter, a verbatim pass + /// through to the tool answer, or the owner-job-history evidence (a job row's own count, never a dimension's), + /// and the only reader of the column from the store is the metrics reader. A new use that does arithmetic, a + /// comparison or a threshold on it fails here, so the author decides whether an estimate that moves in steps + /// is fit for it. + /// + [Fact] + public void NoReaderComputesFromADimensionRowCount() + { + var root = RepoRoot(); + var offenders = new List(); + var scanned = 0; + var passThroughs = 0; + var allowed = new[] + { + @"^long\? RowCount,$", + @"^row_count = \w+\.RowCount,$", + @"^var status = row\.RowCount is null \? OwnerJobHistoryEvidenceStatus\.Filtered$", + @"^row\.RowCount,$", + }; + + foreach (var project in new[] + { + "PerformanceMonitor.Darling.Storage", "PerformanceMonitor.Darling.Service", + "PerformanceMonitor.Darling.Analysis", "PerformanceMonitor.Darling.Viewer", + }) + { + var dir = Path.Combine(root, "Darling", project); + Assert.True(Directory.Exists(dir), $"product project not found: {dir}"); + + foreach (var path in Directory.EnumerateFiles(dir, "*.cs", SearchOption.AllDirectories)) + { + var relative = Path.GetRelativePath(dir, path); + var top = relative.Split(Path.DirectorySeparatorChar)[0]; + if (string.Equals(top, "bin", StringComparison.OrdinalIgnoreCase) + || string.Equals(top, "obj", StringComparison.OrdinalIgnoreCase)) + { + continue; + } + + scanned++; + var text = File.ReadAllText(path); + var name = Path.GetFileName(path); + + /* The store's own column: read by the metrics reader, written by the self-metrics collector, created by a migration. */ + if (text.Contains("collect.store_metrics", StringComparison.Ordinal) + && Regex.IsMatch(text, @"\brow_count\b") + && name is not ("DarlingStoreMetricsReader.cs" or "StoreSelfMetrics.cs" or "PgMigrations.cs")) + { + offenders.Add($"{project}/{relative}: reads store_metrics.row_count outside the metrics reader"); + } + + if (!text.Contains("StoreMetricRow", StringComparison.Ordinal) + && !text.Contains("StoreMetricDailyPoint", StringComparison.Ordinal)) + { + continue; + } + + /* Comments are blanked by the shared walker, not a line-prefix filter, so a // note or a block comment that names RowCount is not read as a use. */ + var lines = CSharpSourceWalker.StripCommentsAndStrings(text).Split('\n'); + for (var i = 0; i < lines.Length; i++) + { + var line = lines[i].Trim(); + if (!Regex.IsMatch(line, @"\bRowCount\b")) + { + continue; + } + + if (allowed.Any(a => Regex.IsMatch(line, a))) + { + if (line.StartsWith("row_count = ", StringComparison.Ordinal)) + { + passThroughs++; + } + + continue; + } + + offenders.Add($"{project}/{relative}:{i + 1}: {line}"); + } + } + } + + /* An empty scan is how a guard starts reporting clean, so a moved tree or a renamed pass-through fails here. */ + Assert.True(scanned >= 100, $"the product scan found only {scanned} source files; the projects have moved"); + Assert.True(passThroughs >= 2, $"expected the two tool pass-throughs (the latest row and the daily point), found {passThroughs}"); + Assert.True(offenders.Count == 0, string.Join(Environment.NewLine, offenders)); + } + private static string RepoRoot([CallerFilePath] string thisFile = "") { var dir = Path.GetDirectoryName(thisFile); diff --git a/Darling/Darling.Tests/StoreServerPasswordRulesLiveTests.cs b/Darling/Darling.Tests/StoreServerPasswordRulesLiveTests.cs new file mode 100644 index 0000000000..59a41d5604 --- /dev/null +++ b/Darling/Darling.Tests/StoreServerPasswordRulesLiveTests.cs @@ -0,0 +1,644 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: this class mints its own scratch database through ScratchPostgres and touches nothing on the + shared one. The managed role names are cluster-wide, so it runs in the pg-cluster-roles collection (#5602), alone, + with the other classes that provision them, and it skips when a set of those names already exists on the rig. */ + +/// +/// The store's own password rules, as the roles that meet them. The shipped provisioning batch runs whole on a scratch +/// store, then each role connects as itself: the viewer, admin and MCP roles hand the store a password and never a +/// reference (env: or file:), and a change of how a server is reached keeps no stored password. The store owner is held +/// to neither, so a configuration-file seed and --add-server keep working, and a row already in the table is never +/// touched. Each scenario runs against the managed batch's text and against the self-managed script's. +/// +[Collection("pg-cluster-roles")] +public sealed class StoreServerPasswordRulesLiveTests +{ + private const string ReferenceSentence = "Enter the password itself. References (env: or file:) can only be set in the configuration file."; + private const string MoveSentence = "Changing how this server is reached needs its password again: it is stored encrypted and this surface cannot read it back."; + private const string RemediationSentence = "This server has a remediation login stored. Change how it is reached on the service host, in the configuration file or with --add-server."; + private const string ReferenceState = "PW001"; + private const string MoveState = "PW002"; + private const string RemediationState = "PW003"; + + private static string RequireLivePostgres() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the store password rule tests (each mints its own scratch database)."); + return connectionString!; + } + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task TextAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + return Convert.ToString(await command.ExecuteScalarAsync(ct), System.Globalization.CultureInfo.InvariantCulture); + } + + private static string OwnerRoleOf(NpgsqlConnection owner) + => new NpgsqlConnectionStringBuilder(owner.ConnectionString).Username ?? "darling"; + + private static async Task ConnectAsync(string ownerString, string role, string password, CancellationToken ct) + { + var connection = new NpgsqlConnection(new NpgsqlConnectionStringBuilder(ownerString) + { + Username = role, + Password = password, + SearchPath = "collect,config,public", + Pooling = false, + }.ConnectionString); + await connection.OpenAsync(ct); + return connection; + } + + /// The function and trigger as the self-managed script carries them, cut out of the script's text. + private static string ByoRulesSql() + { + var script = RepoFile.ReadRepoFile("Darling", "tools", "provision-roles.sql").Replace("\r\n", "\n", StringComparison.Ordinal); + var start = script.IndexOf("CREATE OR REPLACE FUNCTION config.monitored_server_password_rules()", StringComparison.Ordinal); + Assert.True(start >= 0, "provision-roles.sql carries no password rules function"); + const string triggerEnd = "EXECUTE FUNCTION config.monitored_server_password_rules();"; + var end = script.IndexOf(triggerEnd, start, StringComparison.Ordinal); + Assert.True(end >= 0, "provision-roles.sql carries no password rules trigger"); + return script[start..(end + triggerEnd.Length)]; + } + + /// The edit function as the self-managed script carries it, cut out of the script's text: the 17-argument + /// function and the 15-argument wrapper that follows it (#5452). + private static string ByoEditFunctionSql() + { + var script = RepoFile.ReadRepoFile("Darling", "tools", "provision-roles.sql").Replace("\r\n", "\n", StringComparison.Ordinal); + var start = script.IndexOf("CREATE OR REPLACE FUNCTION config.edit_monitored_server(", StringComparison.Ordinal); + Assert.True(start >= 0, "provision-roles.sql carries no edit function"); + const string functionEnd = "$fn$;"; + var end = script.IndexOf(functionEnd, start, StringComparison.Ordinal); + Assert.True(end >= 0, "provision-roles.sql carries no end for the edit function"); + var wrapperEnd = script.IndexOf(functionEnd, end + functionEnd.Length, StringComparison.Ordinal); + Assert.True(wrapperEnd >= 0, "provision-roles.sql carries no end for the edit function's legacy wrapper"); + return script[start..(wrapperEnd + functionEnd.Length)]; + } + + private static async Task InsertServerAsync( + NpgsqlConnection connection, int id, string host, string? secret, string? remediationSecret, CancellationToken ct) + { + await using var command = new NpgsqlCommand( + "INSERT INTO config_monitored_servers (server_id, name, host, database, auth, username, encrypted_password, remediation_encrypted_password, engine, port) " + + "VALUES ($1, $2, $3, 'master', 'sql', 'monitor', $4, $5, 'sqlserver', 0)", connection); + command.Parameters.Add(new NpgsqlParameter { TypedValue = id }); + command.Parameters.AddWithValue("srv-" + id); + command.Parameters.AddWithValue(host); + command.Parameters.AddWithValue(secret is null ? DBNull.Value : secret); + command.Parameters.AddWithValue(remediationSecret is null ? DBNull.Value : remediationSecret); + await command.ExecuteNonQueryAsync(ct); + } + + private static async Task RefusedAsync(Func write, string sqlState, string sentence) + { + var ex = await Assert.ThrowsAsync(write); + Assert.Equal(sqlState, ex.SqlState); + Assert.Equal(sentence, ex.MessageText); + return ex; + } + + private static async Task EditOutcomeAsync( + NpgsqlConnection role, int id, string columns, string? host, string? secret, NpgsqlConnection ownerConnection, CancellationToken ct) + { + var token = await TextAsync(ownerConnection, $"SELECT to_char(modified_at, 'YYYY-MM-DD HH24:MI:SS.US') FROM config_monitored_servers WHERE server_id = {id}", ct); + string Lit(string? value) => value is null ? "NULL" : "'" + value.Replace("'", "''", StringComparison.Ordinal) + "'"; + return await TextAsync(role, + $"SELECT outcome FROM config.edit_monitored_server({id}, '{token}'::timestamp, ARRAY[{columns}]::text[], NULL, {Lit(host)}, NULL, NULL, NULL, NULL, NULL, {Lit(secret)}, NULL, NULL, NULL, NULL)", ct); + } + + private static async Task RunScenarioAsync(bool selfManagedText, Func body) + { + var connectionString = RequireLivePostgres(); + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(connectionString, ct); + + await using (var owner = new NpgsqlConnection(scratch.ConnectionString)) + { + await owner.OpenAsync(ct); + await PgMigrations.MigrateAsync(owner, ct); + Assert.SkipWhen( + await TextAsync(owner, "SELECT EXISTS (SELECT 1 FROM pg_roles WHERE rolname IN ('admin', 'viewer', 'mcp'))", ct) == "True", + "A cluster-wide admin/viewer/mcp role already exists on this rig; the managed provisioning batch would adopt it."); + } + + var bodySucceeded = false; + try + { + await using var owner = new NpgsqlConnection(new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { SearchPath = "collect,config,public" }.ConnectionString); + await owner.OpenAsync(ct); + var batch = DarlingManagedRoles.BuildProvisioningSql( + ProvisioningTestSecrets.Admin, ProvisioningTestSecrets.Viewer, ProvisioningTestSecrets.Mcp, + 15, PasswordReassert.All, ProvisioningTarget.ComposeStore(OwnerRoleOf(owner), scratch.DatabaseName)); + await ExecAsync(owner, batch, ct); + if (selfManagedText) + { + await ExecAsync(owner, ByoEditFunctionSql(), ct); + await ExecAsync(owner, ByoRulesSql(), ct); + } + + await body(scratch.ConnectionString, owner, ct); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (cleanup, cleanupCt) => + await ExecAsync(cleanup, + "DROP OWNED BY admin, viewer, mcp; DROP ROLE IF EXISTS admin; DROP ROLE IF EXISTS viewer; DROP ROLE IF EXISTS mcp", cleanupCt)); + } + } + + private const string ProtectedBlob = "AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAA-a-protected-blob"; + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task TheViewerRole_GivesThePasswordItself_NeverAReference(bool selfManagedText) + { + await RunScenarioAsync(selfManagedText, async (ownerString, owner, ct) => + { + await using var viewer = await ConnectAsync(ownerString, "viewer", ProvisioningTestSecrets.ViewerPassword, ct); + + await RefusedAsync(() => InsertServerAsync(viewer, 7301, "h-env", "env:SOME_SECRET", null, ct), ReferenceState, ReferenceSentence); + await RefusedAsync(() => InsertServerAsync(viewer, 7302, "h-file", "file:/run/secrets/sql_password", null, ct), ReferenceState, ReferenceSentence); + await RefusedAsync(() => InsertServerAsync(viewer, 7303, "h-rem", ProtectedBlob, "file:/run/secrets/remediation", ct), ReferenceState, ReferenceSentence); + Assert.Equal("0", await TextAsync(owner, "SELECT count(*) FROM config_monitored_servers", ct)); + + /* The rule is the resolver's own: case-sensitive, at the start of the text. A password that merely contains the + prefix, or spells it another way, is a password. */ + await InsertServerAsync(viewer, 7304, "h-plain", ProtectedBlob, null, ct); + await InsertServerAsync(viewer, 7305, "h-upper", "ENV:NOT_A_REFERENCE", null, ct); + await InsertServerAsync(viewer, 7306, "h-lead", " env:NOT_A_REFERENCE", null, ct); + await InsertServerAsync(viewer, 7307, "h-inside", "xenv:NOT_A_REFERENCE", null, ct); + await InsertServerAsync(viewer, 7308, "h-none", null, null, ct); + Assert.Equal("5", await TextAsync(owner, "SELECT count(*) FROM config_monitored_servers", ct)); + + /* The edit function: a reference is refused with its own outcome and nothing is written. */ + Assert.Equal("reference_refused", await EditOutcomeAsync(viewer, 7304, "'encrypted_password'", null, "env:SOME_SECRET", owner, ct)); + Assert.Equal("reference_refused", await EditOutcomeAsync(viewer, 7304, "'encrypted_password'", null, "file:/run/secrets/x", owner, ct)); + Assert.Equal("reference_refused", await EditOutcomeAsync(viewer, 7304, "'host','encrypted_password'", "moved-host", "file:/run/secrets/x", owner, ct)); + Assert.Equal(ProtectedBlob, await TextAsync(owner, "SELECT encrypted_password FROM config_monitored_servers WHERE server_id = 7304", ct)); + Assert.Equal("h-plain", await TextAsync(owner, "SELECT host FROM config_monitored_servers WHERE server_id = 7304", ct)); + + /* What the function does for the web edit still works: a move with a new password is saved (its own UPDATE + runs as the function's owner), a move without one is still asked for the password, an ordinary password is + stored as given. */ + Assert.Equal("password_needed", await EditOutcomeAsync(viewer, 7304, "'host'", "moved-host", null, owner, ct)); + Assert.Equal("saved", await EditOutcomeAsync(viewer, 7304, "'host','encrypted_password'", "moved-host", "a-new-protected-blob", owner, ct)); + Assert.Equal("moved-host", await TextAsync(owner, "SELECT host FROM config_monitored_servers WHERE server_id = 7304", ct)); + Assert.Equal("a-new-protected-blob", await TextAsync(owner, "SELECT encrypted_password FROM config_monitored_servers WHERE server_id = 7304", ct)); + }); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task TheAdminRole_CannotSetAReference_OrMoveAServerThatKeepsItsStoredPassword(bool selfManagedText) + { + await RunScenarioAsync(selfManagedText, async (ownerString, owner, ct) => + { + await InsertServerAsync(owner, 7401, "keep-host", ProtectedBlob, null, ct); + await InsertServerAsync(owner, 7402, "ref-host", "file:/run/secrets/sql_password", null, ct); + await InsertServerAsync(owner, 7403, "rem-host", ProtectedBlob, "AQAAANCMnd8-a-remediation-blob", ct); + await using var admin = await ConnectAsync(ownerString, "admin", ProvisioningTestSecrets.AdminPassword, ct); + + /* A reference, set by an update or inserted. */ + await RefusedAsync(() => ExecAsync(admin, "UPDATE config_monitored_servers SET encrypted_password = 'file:/run/secrets/other' WHERE server_id = 7401", ct), ReferenceState, ReferenceSentence); + await RefusedAsync(() => ExecAsync(admin, "UPDATE config_monitored_servers SET encrypted_password = 'env:OTHER', host = 'own-host' WHERE server_id = 7401", ct), ReferenceState, ReferenceSentence); + await RefusedAsync(() => ExecAsync(admin, "UPDATE config_monitored_servers SET remediation_encrypted_password = 'env:OTHER' WHERE server_id = 7401", ct), ReferenceState, ReferenceSentence); + await RefusedAsync(() => ExecAsync(admin, "UPDATE config_monitored_servers SET encrypted_password = 'file:/run/secrets/changed' WHERE server_id = 7402", ct), ReferenceState, ReferenceSentence); + await RefusedAsync(() => InsertServerAsync(admin, 7404, "new-env", "env:SOME_SECRET", null, ct), ReferenceState, ReferenceSentence); + await RefusedAsync(() => InsertServerAsync(admin, 7405, "new-file", "file:/etc/secret", null, ct), ReferenceState, ReferenceSentence); + + /* A move that keeps the stored password: the host, the port, the engine, and each of the other connection fields. */ + foreach (var change in new[] + { + "host = 'own-host'", "port = 5432", "engine = 'postgresql'", "database = 'other'", "read_only_intent = TRUE", + "auth = 'serviceprincipal'", "username = 'someone-else'", "encrypt_mode = 'Optional'", + "trust_server_certificate = TRUE", "multi_subnet_failover = TRUE", + }) + { + await RefusedAsync(() => ExecAsync(admin, $"UPDATE config_monitored_servers SET {change} WHERE server_id = 7401", ct), MoveState, MoveSentence); + } + + /* A new password is not enough while a remediation password is still stored with it, and a move that keeps a + stored reference is refused like any other. */ + await RefusedAsync(() => ExecAsync(admin, "UPDATE config_monitored_servers SET host = 'own-host', encrypted_password = 'a-new-protected-blob' WHERE server_id = 7403", ct), RemediationState, RemediationSentence); + await RefusedAsync(() => ExecAsync(admin, "UPDATE config_monitored_servers SET host = 'own-host' WHERE server_id = 7403", ct), RemediationState, RemediationSentence); + await RefusedAsync(() => ExecAsync(admin, "UPDATE config_monitored_servers SET host = 'own-host' WHERE server_id = 7402", ct), MoveState, MoveSentence); + Assert.Equal("keep-host", await TextAsync(owner, "SELECT host FROM config_monitored_servers WHERE server_id = 7401", ct)); + Assert.Equal(ProtectedBlob, await TextAsync(owner, "SELECT encrypted_password FROM config_monitored_servers WHERE server_id = 7401", ct)); + + /* What an admin write may still do. */ + await ExecAsync(admin, "UPDATE config_monitored_servers SET host = 'new-host', encrypted_password = 'a-new-protected-blob' WHERE server_id = 7401", ct); + Assert.Equal("new-host", await TextAsync(owner, "SELECT host FROM config_monitored_servers WHERE server_id = 7401", ct)); + await ExecAsync(admin, "UPDATE config_monitored_servers SET host = 'cleared-host', encrypted_password = NULL, auth = 'integrated' WHERE server_id = 7401", ct); + await ExecAsync(admin, "UPDATE config_monitored_servers SET name = 'renamed', is_enabled = FALSE, monthly_cost_usd = 12, excluded_databases = '{a}' WHERE server_id = 7402", ct); + await ExecAsync(admin, "UPDATE config_monitored_servers SET trust_server_certificate = trust_server_certificate WHERE server_id = 7402", ct); + Assert.Equal("file:/run/secrets/sql_password", await TextAsync(owner, "SELECT encrypted_password FROM config_monitored_servers WHERE server_id = 7402", ct)); + await InsertServerAsync(admin, 7406, "admin-plain", ProtectedBlob, null, ct); + + /* The Viewer's edit is an upsert that sends the stored value back: an unchanged reference passes, a changed one + does not, and neither lets a move keep the stored password. */ + const string upsert = + "INSERT INTO config_monitored_servers (server_id, name, host, database, auth, username, encrypted_password, engine, port) " + + "VALUES (7402, 'renamed again', {0}, 'master', 'sql', 'monitor', {1}, 'sqlserver', 0) " + + "ON CONFLICT (server_id) DO UPDATE SET name = EXCLUDED.name, host = EXCLUDED.host, encrypted_password = EXCLUDED.encrypted_password"; + await ExecAsync(admin, string.Format(System.Globalization.CultureInfo.InvariantCulture, upsert, "'ref-host'", "'file:/run/secrets/sql_password'"), ct); + Assert.Equal("renamed again", await TextAsync(owner, "SELECT name FROM config_monitored_servers WHERE server_id = 7402", ct)); + await RefusedAsync(() => ExecAsync(admin, string.Format(System.Globalization.CultureInfo.InvariantCulture, upsert, "'ref-host'", "'file:/run/secrets/changed'"), ct), ReferenceState, ReferenceSentence); + await RefusedAsync(() => ExecAsync(admin, string.Format(System.Globalization.CultureInfo.InvariantCulture, upsert, "'own-host'", "'file:/run/secrets/sql_password'"), ct), MoveState, MoveSentence); + await RefusedAsync(() => ExecAsync(admin, string.Format(System.Globalization.CultureInfo.InvariantCulture, upsert.Replace("7402", "7499", StringComparison.Ordinal), "'ref-host'", "'file:/run/secrets/sql_password'"), ct), ReferenceState, ReferenceSentence); + Assert.Equal("ref-host", await TextAsync(owner, "SELECT host FROM config_monitored_servers WHERE server_id = 7402", ct)); + }); + } + + /// + /// A role that may create a temporary table puts one named like a catalog or a table the store's code reads first in its + /// own session: the rules and the edit function name what they read, and keep the temporary schema last. + /// + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task ATemporaryTableNamedLikeACatalogOrAStoreTable_ChangesNothingTheRulesOrTheEditFunctionRead(bool selfManagedText) + { + await RunScenarioAsync(selfManagedText, async (ownerString, owner, ct) => + { + await InsertServerAsync(owner, 7901, "keep-host", ProtectedBlob, null, ct); + await using var viewer = await ConnectAsync(ownerString, "viewer", ProvisioningTestSecrets.ViewerPassword, ct); + await using var mcp = await ConnectAsync(ownerString, "mcp", ProvisioningTestSecrets.McpPassword, ct); + + static async Task ShadowCatalogAsync(NpgsqlConnection role, CancellationToken token) + { + await ExecAsync(role, "CREATE TEMP TABLE pg_class (oid oid, relowner oid)", token); + await ExecAsync(role, + "INSERT INTO pg_class VALUES ('config.config_monitored_servers'::regclass, (SELECT oid FROM pg_catalog.pg_roles WHERE rolname = current_user))", token); + } + + await ShadowCatalogAsync(viewer, ct); + await RefusedAsync(() => InsertServerAsync(viewer, 7902, "h-env", "env:SOME_SECRET", null, ct), ReferenceState, ReferenceSentence); + + await ShadowCatalogAsync(mcp, ct); + await RefusedAsync(() => ExecAsync(mcp, "UPDATE config_monitored_servers SET host = 'moved' WHERE server_id = 7901", ct), MoveState, MoveSentence); + Assert.Equal("keep-host", await TextAsync(owner, "SELECT host FROM config_monitored_servers WHERE server_id = 7901", ct)); + + /* The edit function reads and writes the store's table, not a temporary table of the same name. */ + await ExecAsync(viewer, "CREATE TEMP TABLE config_monitored_servers (server_id integer)", ct); + Assert.Equal("saved", await EditOutcomeAsync(viewer, 7901, "'host','encrypted_password'", "moved-host", "a-new-protected-blob", owner, ct)); + Assert.Equal("moved-host", await TextAsync(owner, "SELECT host FROM config_monitored_servers WHERE server_id = 7901", ct)); + }); + } + + /// + /// A role that may create temporary objects puts a table named like each table the two definer functions use + /// (config.edit_monitored_server and config.record_custom_alert_resolution) in its own session, with a + /// trigger on it that would record who runs it. Called as that role, each function writes the store's table and the trigger + /// never fires. + /// + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task ATemporaryTableAndTriggerNamedLikeAStoreTable_IsNeverUsedByTheDefinerFunctions(bool selfManagedText) + { + await RunScenarioAsync(selfManagedText, async (ownerString, owner, ct) => + { + var next = 7950; + foreach (var (roleName, password) in new[] { ("viewer", ProvisioningTestSecrets.ViewerPassword), ("mcp", ProvisioningTestSecrets.McpPassword) }) + { + var serverId = next++; + var metric = "shadow-metric-" + roleName; + await InsertServerAsync(owner, serverId, "keep-host", ProtectedBlob, null, ct); + await using var role = await ConnectAsync(ownerString, roleName, password, ct); + + await ExecAsync(role, "CREATE TEMP TABLE trigger_log (who text)", ct); + await ExecAsync(role, + "CREATE FUNCTION pg_temp.shadow_trigger() RETURNS trigger LANGUAGE plpgsql AS $fn$ " + + "BEGIN INSERT INTO pg_temp.trigger_log VALUES (current_user); RETURN NEW; END $fn$", ct); + + await ExecAsync(role, + "CREATE TEMP TABLE config_monitored_servers (server_id integer, name text, host text, port integer, database text, " + + "read_only_intent boolean, auth text, username text, encrypted_password text, remediation_encrypted_password text, " + + "encrypt_mode text, trust_server_certificate boolean, multi_subnet_failover boolean, monthly_cost_usd numeric, modified_at timestamp)", ct); + var token = await TextAsync(owner, $"SELECT to_char(modified_at, 'YYYY-MM-DD HH24:MI:SS.US') FROM config_monitored_servers WHERE server_id = {serverId}", ct); + await ExecAsync(role, + $"INSERT INTO config_monitored_servers (server_id, host, auth, modified_at) VALUES ({serverId}, 'keep-host', 'sql', '{token}'::timestamp)", ct); + await ExecAsync(role, + "CREATE TRIGGER shadow BEFORE UPDATE ON config_monitored_servers FOR EACH ROW EXECUTE FUNCTION pg_temp.shadow_trigger()", ct); + + await ExecAsync(role, + "CREATE TEMP TABLE config_alert_log (alert_time timestamp, server_id integer, server_name text, metric_name text, " + + "current_value double precision, threshold_value double precision, alert_sent boolean, notification_type text, " + + "send_error text, muted boolean, detail_text text, context_json text)", ct); + await ExecAsync(role, + "CREATE TRIGGER shadow BEFORE INSERT ON config_alert_log FOR EACH ROW EXECUTE FUNCTION pg_temp.shadow_trigger()", ct); + + Assert.Equal("saved", await TextAsync(role, + $"SELECT outcome FROM config.edit_monitored_server({serverId}, '{token}'::timestamp, ARRAY['host','encrypted_password']::text[], " + + "NULL, 'moved-host', NULL, NULL, NULL, NULL, NULL, 'a-new-protected-blob', NULL, NULL, NULL, NULL)", ct)); + await ExecAsync(role, $"SELECT config.record_custom_alert_resolution({serverId}, 'srv', '{metric}', 'detail')", ct); + + Assert.Equal("0", await TextAsync(role, "SELECT count(*) FROM pg_temp.trigger_log", ct)); + Assert.Equal("moved-host", await TextAsync(owner, $"SELECT host FROM config_monitored_servers WHERE server_id = {serverId}", ct)); + Assert.Equal("1", await TextAsync(owner, $"SELECT count(*) FROM config_alert_log WHERE server_id = {serverId} AND metric_name = '{metric}'", ct)); + Assert.Equal("keep-host", await TextAsync(role, $"SELECT host FROM pg_temp.config_monitored_servers WHERE server_id = {serverId}", ct)); + Assert.Equal("0", await TextAsync(role, "SELECT count(*) FROM pg_temp.config_alert_log", ct)); + } + }); + } + + private const string RemediationBlob = "AQAAANCMnd8-a-remediation-blob"; + + /// + /// A change to how a server is reached on a row that holds a remediation login is refused by the edit function whatever + /// else the call sends, and nothing is written; the same call on a row without one saves. The store's own trigger refuses + /// the same change made directly, with the same sentence. + /// + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task AChangeToHowAServerIsReached_OnARowWithARemediationLogin_IsRefusedAndWritesNothing(bool selfManagedText) + { + await RunScenarioAsync(selfManagedText, async (ownerString, owner, ct) => + { + await InsertServerAsync(owner, 8001, "rem-viewer", ProtectedBlob, RemediationBlob, ct); + await InsertServerAsync(owner, 8002, "plain-viewer", ProtectedBlob, null, ct); + await InsertServerAsync(owner, 8003, "rem-mcp", ProtectedBlob, RemediationBlob, ct); + await InsertServerAsync(owner, 8004, "plain-mcp", ProtectedBlob, null, ct); + await using var viewer = await ConnectAsync(ownerString, "viewer", ProvisioningTestSecrets.ViewerPassword, ct); + await using var mcp = await ConnectAsync(ownerString, "mcp", ProvisioningTestSecrets.McpPassword, ct); + + foreach (var (role, remediationId, plainId, host) in new[] { (viewer, 8001, 8002, "rem-viewer"), (mcp, 8003, 8004, "rem-mcp") }) + { + Assert.Equal("remediation_kept", await EditOutcomeAsync(role, remediationId, "'host','encrypted_password'", "evil-host", "typed-anything", owner, ct)); + Assert.Equal("remediation_kept", await EditOutcomeAsync(role, remediationId, "'host'", "evil-host", null, owner, ct)); + Assert.Equal(host + "|" + ProtectedBlob + "|" + RemediationBlob, + await TextAsync(owner, $"SELECT host || '|' || encrypted_password || '|' || remediation_encrypted_password FROM config_monitored_servers WHERE server_id = {remediationId}", ct)); + + /* A change that is not to how the row is reached still saves, and leaves the remediation login alone. */ + Assert.Equal("saved", await EditOutcomeAsync(role, remediationId, "'encrypted_password'", null, "a-new-protected-blob", owner, ct)); + Assert.Equal(RemediationBlob, await TextAsync(owner, $"SELECT remediation_encrypted_password FROM config_monitored_servers WHERE server_id = {remediationId}", ct)); + + Assert.Equal("saved", await EditOutcomeAsync(role, plainId, "'host','encrypted_password'", "moved-host", "a-new-protected-blob", owner, ct)); + Assert.Equal("moved-host", await TextAsync(owner, $"SELECT host FROM config_monitored_servers WHERE server_id = {plainId}", ct)); + } + + await RefusedAsync(() => ExecAsync(mcp, "UPDATE config_monitored_servers SET host = 'evil-host' WHERE server_id = 8003", ct), RemediationState, RemediationSentence); + await RefusedAsync(() => ExecAsync(mcp, "UPDATE config_monitored_servers SET host = 'evil-host', encrypted_password = 'typed-anything' WHERE server_id = 8003", ct), RemediationState, RemediationSentence); + Assert.Equal("rem-mcp", await TextAsync(owner, "SELECT host FROM config_monitored_servers WHERE server_id = 8003", ct)); + + /* The login name the remediation secret was stored for is part of how the row is reached: it does not change + while the secret stays, and does on a row that holds none. */ + var before = await TextAsync(owner, "SELECT COALESCE(remediation_username, '(none)') FROM config_monitored_servers WHERE server_id = 8003", ct); + await RefusedAsync(() => ExecAsync(mcp, "UPDATE config_monitored_servers SET remediation_username = 'another-login' WHERE server_id = 8003", ct), RemediationState, RemediationSentence); + Assert.Equal(before, await TextAsync(owner, "SELECT COALESCE(remediation_username, '(none)') FROM config_monitored_servers WHERE server_id = 8003", ct)); + await ExecAsync(mcp, "UPDATE config_monitored_servers SET remediation_username = 'another-login' WHERE server_id = 8004", ct); + Assert.Equal("another-login", await TextAsync(owner, "SELECT remediation_username FROM config_monitored_servers WHERE server_id = 8004", ct)); + }); + } + + /// + /// A self-managed store's rules are created on the service's start, as the role that owns its tables, when the script was + /// not re-run after an upgrade; running it again changes nothing, and a login that may not create them gets one warning + /// that names the script and fails nothing. Rules already in place from another role raise no warning; a function that + /// differs raises one. + /// + [Fact] + public async Task ASelfManagedStoreWithNoRules_GetsThemFromTheServicesStart_AndALoginThatMayNotCreateThemGetsOneWarning() + { + var connectionString = RequireLivePostgres(); + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(connectionString, ct); + var login = "rules_probe_" + Guid.NewGuid().ToString("N")[..8]; + const string triggerCount = "SELECT count(*) FROM pg_trigger WHERE tgrelid = 'config.config_monitored_servers'::regclass AND tgname = 'trg_monitored_server_password_rules'"; + const string functionCount = "SELECT count(*) FROM pg_proc WHERE oid = 'config.monitored_server_password_rules()'::regprocedure"; + + var bodySucceeded = false; + try + { + await using var owner = new NpgsqlConnection(scratch.ConnectionString); + await owner.OpenAsync(ct); + await PgMigrations.MigrateAsync(owner, ct); + Assert.Equal("0", await TextAsync(owner, triggerCount, ct)); + + var log = new CapturingTestLogger(); + await using (var ownerSource = NpgsqlDataSource.Create(scratch.ConnectionString)) + { + Assert.True(await DarlingManagedRoles.EnsureServerPasswordRulesAsync(ownerSource, log, ct), log.Joined); + Assert.True(await DarlingManagedRoles.EnsureServerPasswordRulesAsync(ownerSource, log, ct), log.Joined); + } + + Assert.Equal("1", await TextAsync(owner, triggerCount, ct)); + Assert.Equal("1", await TextAsync(owner, functionCount, ct)); + Assert.Equal(0, log.CountAtLevel(Microsoft.Extensions.Logging.LogLevel.Warning)); + + await ExecAsync(owner, "DROP TRIGGER trg_monitored_server_password_rules ON config.config_monitored_servers", ct); + await ExecAsync(owner, $"CREATE ROLE {login} LOGIN PASSWORD 'rules-probe-1'", ct); + var restricted = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { Username = login, Password = "rules-probe-1" }.ConnectionString; + var warned = new CapturingTestLogger(); + await using (var restrictedSource = NpgsqlDataSource.Create(restricted)) + { + Assert.False(await DarlingManagedRoles.EnsureServerPasswordRulesAsync(restrictedSource, warned, ct), warned.Joined); + } + + Assert.Equal(1, warned.CountAtLevel(Microsoft.Extensions.Logging.LogLevel.Warning)); + Assert.Contains("provision-roles.sql", warned.Joined, StringComparison.Ordinal); + Assert.Equal("0", await TextAsync(owner, triggerCount, ct)); + + /* Rules that are already in place, made by a role this login is not, are left alone and warn nothing: the login + could not replace the function, and nothing needs replacing. A function that differs in its setting or in its + body gets one warning each. */ + await ExecAsync(owner, $"GRANT USAGE ON SCHEMA config TO {login}", ct); + await using (var ownerSource = NpgsqlDataSource.Create(scratch.ConnectionString)) + await using (var restrictedSource = NpgsqlDataSource.Create(restricted)) + { + Assert.True(await DarlingManagedRoles.EnsureServerPasswordRulesAsync(ownerSource, new CapturingTestLogger(), ct)); + Assert.Equal("1", await TextAsync(owner, triggerCount, ct)); + + var inPlace = new CapturingTestLogger(); + Assert.True(await DarlingManagedRoles.EnsureServerPasswordRulesAsync(restrictedSource, inPlace, ct), inPlace.Joined); + Assert.Equal(0, inPlace.CountAtLevel(Microsoft.Extensions.Logging.LogLevel.Warning)); + + await ExecAsync(owner, "ALTER FUNCTION config.monitored_server_password_rules() SET search_path = public", ct); + var otherSetting = new CapturingTestLogger(); + Assert.False(await DarlingManagedRoles.EnsureServerPasswordRulesAsync(restrictedSource, otherSetting, ct), otherSetting.Joined); + Assert.Equal(1, otherSetting.CountAtLevel(Microsoft.Extensions.Logging.LogLevel.Warning)); + + Assert.True(await DarlingManagedRoles.EnsureServerPasswordRulesAsync(ownerSource, new CapturingTestLogger(), ct)); + await ExecAsync(owner, "CREATE OR REPLACE FUNCTION config.monitored_server_password_rules() RETURNS trigger LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $x$ BEGIN RETURN NEW; END; $x$", ct); + var otherBody = new CapturingTestLogger(); + Assert.False(await DarlingManagedRoles.EnsureServerPasswordRulesAsync(restrictedSource, otherBody, ct), otherBody.Joined); + Assert.Equal(1, otherBody.CountAtLevel(Microsoft.Extensions.Logging.LogLevel.Warning)); + Assert.Contains("provision-roles.sql", otherBody.Joined, StringComparison.Ordinal); + + Assert.True(await DarlingManagedRoles.EnsureServerPasswordRulesAsync(ownerSource, new CapturingTestLogger(), ct)); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (cleanup, cleanupCt) => + await ExecAsync(cleanup, $"DROP OWNED BY {login}; DROP ROLE IF EXISTS {login}", cleanupCt)); + } + } + + [Fact] + public async Task TheMcpRole_GivesThePasswordItself_NeverAReference() + { + await RunScenarioAsync(selfManagedText: false, async (ownerString, owner, ct) => + { + await using var mcp = await ConnectAsync(ownerString, "mcp", ProvisioningTestSecrets.McpPassword, ct); + + await RefusedAsync(() => InsertServerAsync(mcp, 7501, "mcp-env", "env:SOME_SECRET", null, ct), ReferenceState, ReferenceSentence); + await RefusedAsync(() => InsertServerAsync(mcp, 7502, "mcp-file", "file:/run/secrets/sql_password", null, ct), ReferenceState, ReferenceSentence); + await InsertServerAsync(mcp, 7503, "mcp-plain", ProtectedBlob, null, ct); + await InsertServerAsync(owner, 7504, "seeded", "file:/run/secrets/seeded", null, ct); + + /* The upsert shape's pass is the admin's alone: mcp reads no secret column, so the reference it would send + back is refused. */ + await RefusedAsync(() => ExecAsync(mcp, + "INSERT INTO config_monitored_servers (server_id, name, host, database, auth, username, encrypted_password, engine, port) " + + "VALUES (7504, 'x', 'seeded', 'master', 'sql', 'monitor', 'file:/run/secrets/seeded', 'sqlserver', 0) " + + "ON CONFLICT (server_id) DO UPDATE SET name = EXCLUDED.name", ct), ReferenceState, ReferenceSentence); + + Assert.Equal("reference_refused", await EditOutcomeAsync(mcp, 7503, "'encrypted_password'", null, "env:SOME_SECRET", owner, ct)); + Assert.Equal("reference_refused", await EditOutcomeAsync(mcp, 7503, "'encrypted_password'", null, "file:/run/secrets/x", owner, ct)); + Assert.Equal("password_needed", await EditOutcomeAsync(mcp, 7503, "'host'", "moved-host", null, owner, ct)); + Assert.Equal("saved", await EditOutcomeAsync(mcp, 7503, "'host','encrypted_password'", "moved-host", "a-new-protected-blob", owner, ct)); + Assert.Equal("a-new-protected-blob", await TextAsync(owner, "SELECT encrypted_password FROM config_monitored_servers WHERE server_id = 7503", ct)); + + /* mcp may still remove a server, as before. */ + await ExecAsync(mcp, "DELETE FROM config_monitored_servers WHERE server_id = 7503", ct); + }); + } + + [Fact] + public async Task TheViewersEditAsAdmin_Saves_AndAChangeThatKeepsTheStoredPassword_ShowsTheMoveSentence() + { + await RunScenarioAsync(selfManagedText: false, async (ownerString, owner, ct) => + { + await InsertServerAsync(owner, 7801, "viewer-host", ProtectedBlob, null, ct); + await InsertServerAsync(owner, 7802, "viewer-ref-host", "file:/run/secrets/sql_password", null, ct); + var adminString = new NpgsqlConnectionStringBuilder(ownerString) + { + Username = "admin", + Password = ProvisioningTestSecrets.AdminPassword, + SearchPath = "collect,config,public", + Pooling = false, + }.ConnectionString; + await using var viewer = new PerformanceMonitor.Darling.Viewer.ViewerDataService(adminString); + + PerformanceMonitor.Darling.Viewer.MonitoredServerRow Row(int id, string host, string blob, string encryptMode, string name) => new() + { + ServerId = id, + Name = name, + Host = host, + Database = "master", + Auth = "sql", + Username = "monitor", + EncryptedPassword = blob, + EncryptMode = encryptMode, + }; + + /* A rename saves, a new blob with a change saves, and the stored reference sent back unchanged saves. */ + await viewer.UpsertMonitoredServerAsync(Row(7801, "viewer-host", ProtectedBlob, "Mandatory", "renamed"), ct); + await viewer.UpsertMonitoredServerAsync(Row(7802, "viewer-ref-host", "file:/run/secrets/sql_password", "Mandatory", "renamed ref"), ct); + await viewer.UpsertMonitoredServerAsync(Row(7801, "viewer-host", "a-new-protected-blob", "Optional", "renamed"), ct); + Assert.Equal("Optional", await TextAsync(owner, "SELECT encrypt_mode FROM config_monitored_servers WHERE server_id = 7801", ct)); + + /* A change the Viewer's own check does not look at, with the stored password kept: the store's refusal, in the + sentence the Viewer shows for a moved host. */ + var refused = await Assert.ThrowsAsync( + () => viewer.UpsertMonitoredServerAsync(Row(7801, "viewer-host", "a-new-protected-blob", "Mandatory", "renamed"), ct)); + Assert.Equal(PerformanceMonitor.Darling.Viewer.ViewerDataService.EditPasswordNeededText, refused.Message); + Assert.Equal("Optional", await TextAsync(owner, "SELECT encrypt_mode FROM config_monitored_servers WHERE server_id = 7801", ct)); + }); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task TheStoreOwner_StillSetsReferencesAndMovesServers(bool selfManagedText) + { + await RunScenarioAsync(selfManagedText, async (ownerString, owner, ct) => + { + /* The configuration-file seed and --add-server: references go in, and a server moves with the stored value kept. */ + await InsertServerAsync(owner, 7601, "seed-env", "env:SQL_PASSWORD", null, ct); + await InsertServerAsync(owner, 7602, "seed-file", "file:/run/secrets/sql_password", "file:/run/secrets/remediation", ct); + await ExecAsync(owner, "UPDATE config_monitored_servers SET encrypted_password = 'file:/run/secrets/rotated' WHERE server_id = 7601", ct); + await ExecAsync(owner, "UPDATE config_monitored_servers SET host = 'moved', port = 1433, engine = 'sqlserver', database = 'other' WHERE server_id = 7602", ct); + Assert.Equal("moved", await TextAsync(owner, "SELECT host FROM config_monitored_servers WHERE server_id = 7602", ct)); + Assert.Equal("file:/run/secrets/rotated", await TextAsync(owner, "SELECT encrypted_password FROM config_monitored_servers WHERE server_id = 7601", ct)); + }); + } + + [Fact] + public async Task ARowAlreadyHoldingAReference_IsNotTouchedByProvisioning_AndStillReads() + { + var connectionString = RequireLivePostgres(); + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(connectionString, ct); + await using (var probe = new NpgsqlConnection(scratch.ConnectionString)) + { + await probe.OpenAsync(ct); + await PgMigrations.MigrateAsync(probe, ct); + Assert.SkipWhen( + await TextAsync(probe, "SELECT EXISTS (SELECT 1 FROM pg_roles WHERE rolname IN ('admin', 'viewer', 'mcp'))", ct) == "True", + "A cluster-wide admin/viewer/mcp role already exists on this rig; the managed provisioning batch would adopt it."); + } + + var bodySucceeded = false; + try + { + await using var owner = new NpgsqlConnection(new NpgsqlConnectionStringBuilder(scratch.ConnectionString) { SearchPath = "collect,config,public" }.ConnectionString); + await owner.OpenAsync(ct); + + /* The reference was stored before the rules existed, with a modified_at that provisioning must leave alone. */ + await InsertServerAsync(owner, 7701, "old-ref", "file:/run/secrets/sql_password", "env:REMEDIATION", ct); + var before = await TextAsync(owner, "SELECT encrypted_password || '|' || remediation_encrypted_password || '|' || host || '|' || modified_at::text FROM config_monitored_servers WHERE server_id = 7701", ct); + + var batch = DarlingManagedRoles.BuildProvisioningSql( + ProvisioningTestSecrets.Admin, ProvisioningTestSecrets.Viewer, ProvisioningTestSecrets.Mcp, + 15, PasswordReassert.All, ProvisioningTarget.ComposeStore(OwnerRoleOf(owner), scratch.DatabaseName)); + await ExecAsync(owner, batch, ct); + await ExecAsync(owner, batch, ct); + + Assert.Equal(before, await TextAsync(owner, "SELECT encrypted_password || '|' || remediation_encrypted_password || '|' || host || '|' || modified_at::text FROM config_monitored_servers WHERE server_id = 7701", ct)); + Assert.Equal("1", await TextAsync(owner, "SELECT count(*) FROM pg_trigger WHERE tgrelid = 'config.config_monitored_servers'::regclass AND tgname = 'trg_monitored_server_password_rules'", ct)); + + /* The collection read the owner makes still resolves the stored reference. */ + Assert.Equal("file:/run/secrets/sql_password", await TextAsync(owner, "SELECT encrypted_password FROM config_monitored_servers WHERE server_id = 7701", ct)); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (cleanup, cleanupCt) => + await ExecAsync(cleanup, + "DROP OWNED BY admin, viewer, mcp; DROP ROLE IF EXISTS admin; DROP ROLE IF EXISTS viewer; DROP ROLE IF EXISTS mcp", cleanupCt)); + } + } +} diff --git a/Darling/Darling.Tests/StoreSessionTimeZonePinCensusTests.cs b/Darling/Darling.Tests/StoreSessionTimeZonePinCensusTests.cs index 5fd2f4eddd..5492d675b6 100644 --- a/Darling/Darling.Tests/StoreSessionTimeZonePinCensusTests.cs +++ b/Darling/Darling.Tests/StoreSessionTimeZonePinCensusTests.cs @@ -40,6 +40,7 @@ namespace Darling.Tests; /// inline on the NpgsqlDataSource.Create(...) line itself, so every STORE site the scan finds must show /// the pin on its own line, with no indirection rule left to fool. /// +[Trait("Stage", "Guard")] public sealed class StoreSessionTimeZonePinCensusTests { /* Measured on dev at #4277: 562 .cs files across the four store-side projects, 29 call sites (25 STORE, @@ -67,6 +68,9 @@ glob or a desynchronised regex fails loudly instead of reporting a vacuous clean connection, not the store's. */ "DarlingWorker.cs:ReadPgStatementTextAsync", "DarlingWorker.cs:RunTestHypotheticalIndexAsync", + + /* The RDS endpoint check's fresh unpooled login to a MONITORED target (runtime.ConnectionString), never the store. */ + "RdsEndpointVerifier.cs:ProbeLoginAsync", }; [Fact] diff --git a/Darling/Darling.Tests/StoreSqlClockDisciplineTests.cs b/Darling/Darling.Tests/StoreSqlClockDisciplineTests.cs index 921d9b5335..57e107f5be 100644 --- a/Darling/Darling.Tests/StoreSqlClockDisciplineTests.cs +++ b/Darling/Darling.Tests/StoreSqlClockDisciplineTests.cs @@ -45,6 +45,7 @@ namespace Darling.Tests; /// columns back. Widening this to the write side is a separate change with a data-migration question /// attached, so it is deliberately out of scope here rather than silently allowlisted. /// +[Trait("Reads", "Lite")] public sealed class StoreSqlClockDisciplineTests { /* Floors, so the scan cannot pass by finding nothing. Measured on dev: 602 files, 20,843 string literal diff --git a/Darling/Darling.Tests/StoreStatementHistoryLiveTests.cs b/Darling/Darling.Tests/StoreStatementHistoryLiveTests.cs new file mode 100644 index 0000000000..5b1acf016e --- /dev/null +++ b/Darling/Darling.Tests/StoreStatementHistoryLiveTests.cs @@ -0,0 +1,383 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// The store statement history snapshot (#5097) against a real server. Each fact mints its own scratch database, migrates +/// it, and stands in for the reader functions with plain SQL functions over planted tables, so the epoch and the counters +/// are exactly what the fact says and the shared cluster's own statement counters are never reset. +/// +[Collection("live-postgres")] +public sealed class StoreStatementHistoryLiveTests +{ + private static readonly DateTime T0 = new(2026, 3, 1, 10, 0, 0, DateTimeKind.Unspecified); + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private const string Fakes = @" +CREATE TABLE public.fake_stats +( + role_name text, queryid bigint, calls bigint, total_exec_ms double precision, max_exec_ms double precision, + rows_returned bigint, shared_blks_hit bigint, shared_blks_read bigint, temp_blks_written bigint +); +CREATE TABLE public.fake_info (stats_reset timestamptz, dealloc bigint); +INSERT INTO public.fake_info VALUES (NULL, 0); +CREATE FUNCTION config.store_statement_stats() +RETURNS TABLE (role_name text, queryid bigint, calls bigint, total_exec_ms double precision, mean_exec_ms double precision, + max_exec_ms double precision, total_plan_ms double precision, rows_returned bigint, shared_blks_hit bigint, + shared_blks_read bigint, temp_blks_written bigint, query text) +LANGUAGE sql AS $$ SELECT f.role_name, f.queryid, f.calls, f.total_exec_ms, f.total_exec_ms / NULLIF(f.calls, 0), f.max_exec_ms, + 0::double precision, f.rows_returned, f.shared_blks_hit, f.shared_blks_read, f.temp_blks_written, NULL::text + FROM public.fake_stats f $$; +CREATE FUNCTION config.store_statement_stats_info() +RETURNS TABLE (stats_reset timestamptz, dealloc bigint) +LANGUAGE sql AS $$ SELECT i.stats_reset, i.dealloc FROM public.fake_info i $$; +-- The rig may not preload the extension, and the ready check asks pg_settings (unqualified on purpose, so this +-- stand-in, first on the path, answers for it). +CREATE SCHEMA fakeset; +CREATE TABLE fakeset.pg_settings (name text, setting text); +INSERT INTO fakeset.pg_settings VALUES ('pg_stat_statements.max', '5000'); +SET search_path = fakeset, pg_catalog, public;"; + + private static async Task<(ScratchPostgres Scratch, NpgsqlConnection Connection)> StartAsync(CancellationToken ct, bool withFakes = true) + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), "Set DARLING_TEST_PG to run the statement history snapshot's live pins (each mints its own scratch database)."); + var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + if (withFakes) + { + await ExecAsync(connection, Fakes, ct); + } + + return (scratch, connection); + } + + private static async Task SetStats(NpgsqlConnection c, string rows, CancellationToken ct) + { + await ExecAsync(c, "TRUNCATE public.fake_stats", ct); + if (rows.Length > 0) + { + await ExecAsync(c, "INSERT INTO public.fake_stats VALUES " + rows, ct); + } + } + + [Fact] + public async Task TheFirstCapture_Rebaselines_WritesNoHistory_AndFillsTheBaseline() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, c) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = c; + var ok = false; + try + { + await SetStats(c, "('owner', 1, 10, 100, 20, 10, 5, 1, 0), ('owner', 2, 4, 40, 15, 4, 1, 0, 0)", ct); + var result = await StoreStatementHistory.SnapshotAsync(c, T0, null, ct); + + Assert.Equal(StoreStatementHistory.OutcomeRebaselined, result.Outcome); + Assert.Equal(0L, await ScalarAsync(c, "SELECT count(*) FROM collect.store_statement_history", ct)); + Assert.Equal(2L, await ScalarAsync(c, "SELECT count(*) FROM config.store_statement_baseline", ct)); + Assert.Equal(1L, await ScalarAsync(c, "SELECT count(*) FROM collect.store_statement_captures WHERE outcome = 'rebaselined' AND interval_seconds IS NULL", ct)); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task ASameEpochDelta_IsCurrentMinusBaseline_IdleStatementsWriteNothing_AndAServiceRestartChangesNothing() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, c) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = c; + var ok = false; + try + { + await SetStats(c, "('owner', 1, 10, 100, 20, 10, 5, 1, 0), ('owner', 2, 4, 40, 15, 4, 1, 0, 0)", ct); + await StoreStatementHistory.SnapshotAsync(c, T0, null, ct); + + /* Statement 1 ran 5 more times for 50 ms; statement 2 was idle; statement 3 appeared. */ + await SetStats(c, "('owner', 1, 15, 150, 25, 15, 8, 1, 0), ('owner', 2, 4, 40, 15, 4, 1, 0, 0), ('owner', 3, 2, 8, 5, 2, 0, 0, 0)", ct); + + /* A new connection stands in for a restarted service: nothing but the store is carried over. */ + await using var second = new NpgsqlConnection(scratch.ConnectionString); + await second.OpenAsync(ct); + await ExecAsync(second, "SET search_path = fakeset, pg_catalog, public", ct); + var result = await StoreStatementHistory.SnapshotAsync(second, T0.AddHours(1), null, ct); + + Assert.Equal(StoreStatementHistory.OutcomeOk, result.Outcome); + Assert.Equal(2, result.StatementsSeen); + Assert.Equal(2, result.StatementsKept); + Assert.Equal(5L, await ScalarAsync(c, "SELECT delta_calls FROM collect.store_statement_history WHERE queryid = 1", ct)); + Assert.Equal(50d, await ScalarAsync(c, "SELECT delta_total_exec_ms FROM collect.store_statement_history WHERE queryid = 1", ct)); + Assert.Equal(3600, await ScalarAsync(c, "SELECT interval_seconds FROM collect.store_statement_history WHERE queryid = 1", ct)); + Assert.Equal(0L, await ScalarAsync(c, "SELECT count(*) FROM collect.store_statement_history WHERE queryid = 2", ct)); + Assert.Equal(true, await ScalarAsync(c, "SELECT first_seen FROM collect.store_statement_history WHERE queryid = 3", ct)); + Assert.Equal(false, await ScalarAsync(c, "SELECT first_seen OR entry_restarted OR reset_in_interval FROM collect.store_statement_history WHERE queryid = 1", ct)); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task AFallenCounter_CreditsTheCurrentValue_AsAnEntryRestart() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, c) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = c; + var ok = false; + try + { + await SetStats(c, "('owner', 1, 100, 1000, 20, 10, 5, 1, 0)", ct); + await StoreStatementHistory.SnapshotAsync(c, T0, null, ct); + await SetStats(c, "('owner', 1, 3, 30, 20, 3, 1, 0, 0)", ct); + await StoreStatementHistory.SnapshotAsync(c, T0.AddHours(1), null, ct); + + Assert.Equal(3L, await ScalarAsync(c, "SELECT delta_calls FROM collect.store_statement_history WHERE queryid = 1", ct)); + Assert.Equal(true, await ScalarAsync(c, "SELECT entry_restarted FROM collect.store_statement_history WHERE queryid = 1", ct)); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task AResetInsideTheInterval_CreditsTheCurrentValue_AndARebaselineWritesNoHistory() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, c) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = c; + var ok = false; + try + { + await ExecAsync(c, "UPDATE public.fake_info SET stats_reset = TIMESTAMPTZ '2026-02-20 00:00:00+00'", ct); + await SetStats(c, "('owner', 1, 100, 1000, 20, 10, 5, 1, 0)", ct); + await StoreStatementHistory.SnapshotAsync(c, T0, null, ct); + + /* The reset lands 20 minutes after the previous capture. */ + await ExecAsync(c, "UPDATE public.fake_info SET stats_reset = TIMESTAMPTZ '2026-03-01 10:20:00+00'", ct); + await SetStats(c, "('owner', 1, 7, 70, 20, 7, 1, 0, 0)", ct); + var inside = await StoreStatementHistory.SnapshotAsync(c, T0.AddHours(1), null, ct); + Assert.Equal(StoreStatementHistory.OutcomeOk, inside.Outcome); + Assert.Equal(7L, await ScalarAsync(c, "SELECT delta_calls FROM collect.store_statement_history", ct)); + Assert.Equal(true, await ScalarAsync(c, "SELECT reset_in_interval FROM collect.store_statement_history", ct)); + + /* A stamp that moved to a time BEFORE the previous capture cannot be placed. */ + await ExecAsync(c, "UPDATE public.fake_info SET stats_reset = TIMESTAMPTZ '2026-01-01 00:00:00+00'", ct); + await SetStats(c, "('owner', 1, 9, 90, 20, 9, 1, 0, 0)", ct); + var rebase = await StoreStatementHistory.SnapshotAsync(c, T0.AddHours(2), null, ct); + Assert.Equal(StoreStatementHistory.OutcomeRebaselined, rebase.Outcome); + Assert.Equal(1L, await ScalarAsync(c, "SELECT count(*) FROM collect.store_statement_history", ct)); + Assert.Equal(9L, await ScalarAsync(c, "SELECT calls FROM config.store_statement_baseline WHERE queryid = 1", ct)); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task TheCap_KeepsTheTopStatementsByTime_AndTheCaptureCountsWhatItSaw_AndHiddenStatements() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, c) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = c; + var ok = false; + try + { + await StoreStatementHistory.SnapshotAsync(c, T0, null, ct); + await ExecAsync(c, + "INSERT INTO public.fake_stats SELECT 'owner', g, 1, g, g, 1, 0, 0, 0 FROM generate_series(1, " + (StoreStatementHistory.TopStatements + 25) + ") g", ct); + await ExecAsync(c, "INSERT INTO public.fake_stats VALUES ('other', NULL, 5, 5, 5, 5, 0, 0, 0)", ct); + var result = await StoreStatementHistory.SnapshotAsync(c, T0.AddHours(1), null, ct); + + Assert.Equal(StoreStatementHistory.TopStatements + 25, result.StatementsSeen); + Assert.Equal(StoreStatementHistory.TopStatements, result.StatementsKept); + Assert.Equal(1, result.HiddenStatements); + Assert.Equal((long)StoreStatementHistory.TopStatements, await ScalarAsync(c, "SELECT count(*) FROM collect.store_statement_history", ct)); + Assert.Equal(26L, await ScalarAsync(c, "SELECT min(queryid) FROM collect.store_statement_history", ct)); + Assert.Equal(1, await ScalarAsync(c, "SELECT hidden_statements FROM collect.store_statement_captures ORDER BY capture_time DESC LIMIT 1", ct)); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task Retention_DeletesOldHistoryAndCaptures_AndNeverTheBaseline() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, c) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = c; + var ok = false; + try + { + var old = T0.AddDays(-(StoreStatementHistory.RetentionDays + 5)); + await ExecAsync(c, $"INSERT INTO collect.store_statement_captures (capture_time, statements_seen, statements_kept, hidden_statements, outcome) VALUES ('{old:yyyy-MM-dd HH:mm:ss}', 0, 0, 0, 'ok')", ct); + await ExecAsync(c, $"INSERT INTO collect.store_statement_history VALUES ('{old:yyyy-MM-dd HH:mm:ss}', 3600, 'owner', 9, 1, 1, 1, 0, 0, 0, 1, false, false, false)", ct); + await ExecAsync(c, $"INSERT INTO config.store_statement_baseline VALUES ('owner', 77, 1, 1, 1, 0, 0, 0, '{old:yyyy-MM-dd HH:mm:ss}')", ct); + await SetStats(c, "('owner', 77, 1, 1, 1, 1, 0, 0, 0)", ct); + + await StoreStatementHistory.SnapshotAsync(c, T0, null, ct); + + Assert.Equal(0L, await ScalarAsync(c, "SELECT count(*) FROM collect.store_statement_history", ct)); + Assert.Equal(1L, await ScalarAsync(c, "SELECT count(*) FROM collect.store_statement_captures", ct)); + Assert.Equal(1L, await ScalarAsync(c, "SELECT count(*) FROM config.store_statement_baseline WHERE queryid = 77", ct)); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task WhenTheReaderIsNotThere_ThePassRecordsAPreconditionCapture_AndDoesNotThrow() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, c) = await StartAsync(ct, withFakes: false); + await using var _ = scratch; + await using var __ = c; + var ok = false; + try + { + var result = await StoreStatementHistory.SnapshotAsync(c, T0, null, ct); + + Assert.Equal(StoreStatementHistory.OutcomePrecondition, result.Outcome); + Assert.Equal(1L, await ScalarAsync(c, "SELECT count(*) FROM collect.store_statement_captures WHERE outcome = 'precondition'", ct)); + Assert.Equal(0L, await ScalarAsync(c, "SELECT count(*) FROM collect.store_statement_history", ct)); + Assert.Equal(0L, await ScalarAsync(c, "SELECT count(*) FROM config.store_statement_baseline", ct)); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + /// The planted pg_settings has no pg_stat_statements.max row. + [Fact] + public async Task AnInstalledButUnloadedExtension_WritesAPreconditionCapture_AndNoHistory() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, c) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = c; + var ok = false; + try + { + await ExecAsync(c, "DELETE FROM fakeset.pg_settings", ct); + await SetStats(c, "('owner', 1, 100, 1000, 20, 10, 5, 1, 0)", ct); + var result = await StoreStatementHistory.SnapshotAsync(c, T0, null, ct); + + Assert.Equal(StoreStatementHistory.OutcomePrecondition, result.Outcome); + Assert.Equal(1L, await ScalarAsync(c, "SELECT count(*) FROM collect.store_statement_captures WHERE outcome = 'precondition'", ct)); + Assert.Equal(0L, await ScalarAsync(c, "SELECT count(*) FROM config.store_statement_baseline", ct)); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task AReaderThatRaises_RollsTheWholePassBack_AndLeavesTheBaselineAlone() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, c) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = c; + var ok = false; + try + { + await SetStats(c, "('owner', 1, 100, 1000, 20, 10, 5, 1, 0)", ct); + await StoreStatementHistory.SnapshotAsync(c, T0, null, ct); + await SetStats(c, "('owner', 1, 130, 1300, 20, 10, 5, 1, 0)", ct); + await ExecAsync(c, "DROP FUNCTION config.store_statement_stats(); CREATE FUNCTION config.store_statement_stats() RETURNS TABLE (role_name text, queryid bigint, calls bigint, total_exec_ms double precision, mean_exec_ms double precision, max_exec_ms double precision, total_plan_ms double precision, rows_returned bigint, shared_blks_hit bigint, shared_blks_read bigint, temp_blks_written bigint, query text) LANGUAGE plpgsql AS $$ BEGIN RAISE EXCEPTION 'planted' USING ERRCODE = '55000'; END $$", ct); + + var ex = await Assert.ThrowsAsync(() => StoreStatementHistory.SnapshotAsync(c, T0.AddHours(1), null, ct)); + Assert.Equal("55000", ex.SqlState); + Assert.Equal(0L, await ScalarAsync(c, "SELECT count(*) FROM collect.store_statement_history", ct)); + Assert.Equal(1L, await ScalarAsync(c, "SELECT count(*) FROM collect.store_statement_captures", ct)); + Assert.Equal(100L, await ScalarAsync(c, "SELECT calls FROM config.store_statement_baseline WHERE queryid = 1", ct)); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + [Fact] + public async Task TheSameEpoch_WithStampsOnBothSides_SubtractsTheBaseline_AndTheIntervalAgreesOnBothRows() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, c) = await StartAsync(ct); + await using var _ = scratch; + await using var __ = c; + var ok = false; + try + { + await ExecAsync(c, "UPDATE public.fake_info SET stats_reset = TIMESTAMPTZ '2026-02-20 00:00:00+00'", ct); + await SetStats(c, "('owner', 1, 100, 1000, 20, 10, 5, 1, 0)", ct); + await StoreStatementHistory.SnapshotAsync(c, T0, null, ct); + await SetStats(c, "('owner', 1, 130, 1300, 25, 12, 6, 1, 0)", ct); + var second = await StoreStatementHistory.SnapshotAsync(c, T0.AddSeconds(3599.6), null, ct); + + Assert.Equal(StoreStatementHistory.OutcomeOk, second.Outcome); + Assert.Equal(30L, await ScalarAsync(c, "SELECT delta_calls FROM collect.store_statement_history", ct)); + Assert.Equal(false, await ScalarAsync(c, "SELECT reset_in_interval FROM collect.store_statement_history", ct)); + Assert.Equal( + await ScalarAsync(c, "SELECT interval_seconds FROM collect.store_statement_history", ct), + await ScalarAsync(c, "SELECT interval_seconds FROM collect.store_statement_captures WHERE outcome = 'ok'", ct)); + ok = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, ok, static (_, _) => Task.CompletedTask); + } + } + + private static async Task ScalarAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + return await command.ExecuteScalarAsync(ct); + } + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/StoreStatementHistoryRungTests.cs b/Darling/Darling.Tests/StoreStatementHistoryRungTests.cs new file mode 100644 index 0000000000..96eb0d97c7 --- /dev/null +++ b/Darling/Darling.Tests/StoreStatementHistoryRungTests.cs @@ -0,0 +1,222 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Reflection; +using System.Text.RegularExpressions; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// Pins the Darling rung that adds the store's statement history (#5097): two plain collect tables and one config +/// baseline table. Every fact finds the rung by NAME. The live fact mints its own scratch store. +/// +[Collection("live-postgres")] +public sealed class StoreStatementHistoryRungTests +{ + public const string RungName = "store-statement-history"; + + private const int ProbeOrdinal = 138; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + public static int RungVersion => Rung.Version; + + private static PgMigrations.Migration Rung => PgMigrations.Scripts.Single(m => m.Name == RungName); + + private static string Statements() => + Regex.Replace(Rung.Sql, @"/\*.*?\*/", " ", RegexOptions.Singleline).Replace("\r\n", "\n", StringComparison.Ordinal); + + private static List<(string Name, string Type)> ColumnsOf(string table) + { + var sql = Statements(); + var header = "CREATE TABLE IF NOT EXISTS " + table + "\n("; + var start = sql.IndexOf(header, StringComparison.Ordinal); + Assert.True(start >= 0, "no CREATE TABLE for " + table); + var end = sql.IndexOf("\n)", start, StringComparison.Ordinal); + var columns = new List<(string, string)>(); + foreach (var raw in sql[(start + header.Length)..end].Split('\n')) + { + var line = raw.Trim().TrimEnd(','); + if (line.Length == 0 || line.StartsWith("CONSTRAINT", StringComparison.Ordinal)) + { + continue; + } + + var space = line.IndexOf(' ', StringComparison.Ordinal); + columns.Add((line[..space], line[(space + 1)..].Replace(" NOT NULL", string.Empty, StringComparison.Ordinal).Trim())); + } + + return columns; + } + + [Fact] + public void TheRungIsRegisteredOnce_InADenseLadder() + { + var versions = PgMigrations.Scripts.Select(s => s.Version).ToList(); + + Assert.Equal(163, Rung.Version); + Assert.Single(PgMigrations.Scripts, m => m.Name == RungName); + Assert.Equal(StorageVersion.SchemaVersion, PgMigrations.Scripts[^1].Version); + Assert.Equal(versions.Distinct().OrderBy(v => v), versions); + Assert.Contains(Rung.Version - 1, versions); + } + + [Fact] + public void TheTablesArePlain_SchemaQualified_AndGuarded() + { + var sql = Statements(); + + Assert.DoesNotContain("create_hypertable", sql, StringComparison.OrdinalIgnoreCase); + Assert.DoesNotContain("timescaledb", sql, StringComparison.OrdinalIgnoreCase); + foreach (var match in Regex.Matches(sql, @"(?:CREATE TABLE IF NOT EXISTS|ON)\s+(\S+)").Cast()) + { + Assert.Matches(@"^(collect|config)\.", match.Groups[1].Value); + } + + foreach (var create in Regex.Matches(sql, @"CREATE\s+(?:UNIQUE\s+)?(TABLE|INDEX)\s+(?!IF NOT EXISTS)", RegexOptions.IgnoreCase).Cast()) + { + Assert.Fail("an unguarded CREATE: " + create.Value); + } + + foreach (var verb in new[] { "ALTER ", "UPDATE ", "DELETE ", "GRANT ", "DROP ", "INSERT " }) + { + Assert.DoesNotContain(verb, sql, StringComparison.OrdinalIgnoreCase); + } + + foreach (var index in new[] + { + "idx_store_statement_captures_time\n ON collect.store_statement_captures(capture_time)", + "idx_store_statement_history_time\n ON collect.store_statement_history(capture_time)", + "idx_store_statement_history_query\n ON collect.store_statement_history(queryid, capture_time)", + }) + { + Assert.Contains("CREATE INDEX IF NOT EXISTS " + index, sql, StringComparison.Ordinal); + } + + Assert.Contains("CONSTRAINT pk_store_statement_baseline PRIMARY KEY (role_name, queryid)", sql, StringComparison.Ordinal); + } + + [Fact] + public void TheColumnsAndTypes_ArePinned() + { + Assert.Equal( + ["capture_time:timestamp", "interval_seconds:integer", "stats_reset:timestamp", "dealloc:bigint", "dealloc_delta:bigint", + "statements_seen:integer", "statements_kept:integer", "hidden_statements:integer", "outcome:text"], + ColumnsOf("collect.store_statement_captures").Select(c => c.Name + ":" + c.Type).ToArray()); + + Assert.Equal( + ["capture_time:timestamp", "interval_seconds:integer", "role_name:text", "queryid:bigint", "delta_calls:bigint", + "delta_total_exec_ms:double precision", "delta_rows:bigint", "delta_shared_blks_hit:bigint", "delta_shared_blks_read:bigint", + "delta_temp_blks_written:bigint", "max_exec_ms:double precision", "first_seen:boolean", "entry_restarted:boolean", + "reset_in_interval:boolean"], + ColumnsOf("collect.store_statement_history").Select(c => c.Name + ":" + c.Type).ToArray()); + + Assert.Equal( + ["role_name:text", "queryid:bigint", "calls:bigint", "total_exec_ms:double precision", "rows_returned:bigint", + "shared_blks_hit:bigint", "shared_blks_read:bigint", "temp_blks_written:bigint", "captured_at:timestamp"], + ColumnsOf("config.store_statement_baseline").Select(c => c.Name + ":" + c.Type).ToArray()); + } + + [Fact] + public void NoneOfTheTables_IsACollectorTable() + { + foreach (var table in new[] { "store_statement_captures", "store_statement_history", "store_statement_baseline" }) + { + Assert.DoesNotContain(CollectorCatalog.All, c => c.TargetTable == table); + } + } + + [Fact] + public void TheProbeCarriesTheHistoryTable_AndMapsAStoreThroughThisRungToThisRung() + { + var probe = ViewerDataService.StoreSchemaProbeSql.Replace("\r\n", "\n", StringComparison.Ordinal); + var arm = "to_regclass('collect.store_statement_history') IS NOT NULL"; + Assert.Contains(arm, probe, StringComparison.Ordinal); + + var viewer = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.cs"); + Assert.Contains($"reader.GetBoolean({ProbeOrdinal})", viewer, StringComparison.Ordinal); + /* No longer the probe's last arm: V164 (the hourly row-count ledger) landed above it. */ + Assert.Contains($"reader.GetBoolean({ProbeOrdinal + 1})", viewer, StringComparison.Ordinal); + + var method = typeof(ViewerDataService).GetMethod("MapProbedSchemaVersion", BindingFlags.NonPublic | BindingFlags.Static)!; + var parameters = method.GetParameters(); + Assert.True(ProbeOrdinal < parameters.Length - 1, "a newer rung's sentinel follows this one"); + Assert.Equal("hasStoreStatementHistory", parameters[ProbeOrdinal].Name); + + var all = Enumerable.Range(0, parameters.Length).Select(i => (object)(i <= ProbeOrdinal)).ToArray(); + Assert.Equal(Rung.Version, (int)method.Invoke(null, all)!); + Assert.Equal(StorageVersion.SchemaVersion, ViewerDataService.RequiredStoreSchemaVersion); + + var behind = (object[])all.Clone(); + behind[ProbeOrdinal] = false; + Assert.Equal(Rung.Version - 1, (int)method.Invoke(null, behind)!); + + var thisArm = viewer.IndexOf("if (hasStoreStatementHistory)", StringComparison.Ordinal); + var previousArm = viewer.IndexOf("if (hasSlowReads)", StringComparison.Ordinal); + Assert.True(thisArm >= 0 && thisArm < previousArm, "this arm sits above the previous rung's"); + Assert.Contains($"return {Rung.Version};", viewer[thisArm..previousArm], StringComparison.Ordinal); + + var armProseStart = viewer.LastIndexOf("/* V163 (#5097)", thisArm, StringComparison.Ordinal); + Assert.True(armProseStart >= 0, "the arm has no comment block saying why it exists"); + Assert.DoesNotContain("store_statement_history", viewer[armProseStart..thisArm], StringComparison.Ordinal); + } + + [Fact] + public async Task MigratingAFreshStore_CreatesThePlainTablesAndIndexes_AndRerunningChangesNothing() + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), "Set DARLING_TEST_PG to run the statement history rung's live pin (it mints its own scratch database)."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var bodySucceeded = false; + try + { + Assert.Equal(StorageVersion.SchemaVersion, Convert.ToInt32(await ScalarAsync(connection, "SELECT MAX(version) FROM darling_schema_version", ct))); + Assert.Equal(3L, await ScalarAsync(connection, + "SELECT count(*) FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind = 'r' AND c.relname IN ('store_statement_captures','store_statement_history','store_statement_baseline')", ct)); + Assert.Equal(3L, await ScalarAsync(connection, + "SELECT count(*) FROM pg_indexes WHERE indexname IN ('idx_store_statement_captures_time','idx_store_statement_history_time','idx_store_statement_history_query')", ct)); + + await ExecAsync(connection, "DELETE FROM darling_schema_version WHERE version >= 163", ct); + await PgMigrations.MigrateAsync(connection, ct); + Assert.Equal(StorageVersion.SchemaVersion, Convert.ToInt32(await ScalarAsync(connection, "SELECT MAX(version) FROM darling_schema_version", ct))); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, static (_, _) => Task.CompletedTask); + } + } + + private static async Task ScalarAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + return await command.ExecuteScalarAsync(ct); + } + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/StoreStatementHistoryTests.cs b/Darling/Darling.Tests/StoreStatementHistoryTests.cs new file mode 100644 index 0000000000..3ed9b90d37 --- /dev/null +++ b/Darling/Darling.Tests/StoreStatementHistoryTests.cs @@ -0,0 +1,116 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Text.RegularExpressions; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// The store statement history's pure parts (#5097): the epoch decision and the SQL text's shape. No store. +public sealed class StoreStatementHistoryTests +{ + private static readonly DateTime Previous = new(2026, 1, 1, 10, 0, 0, DateTimeKind.Unspecified); + + [Fact] + public void NoPreviousCapture_Rebaselines() => + Assert.Equal(StoreStatementHistory.Epoch.Rebaseline, StoreStatementHistory.DecideEpoch(null, null, Previous)); + + [Fact] + public void TheSameStamp_IsTheSameEpoch() => + Assert.Equal(StoreStatementHistory.Epoch.SameEpoch, StoreStatementHistory.DecideEpoch(Previous, Previous.AddDays(-3), Previous.AddDays(-3))); + + [Fact] + public void BothStampsAbsent_IsTheSameEpoch() => + Assert.Equal(StoreStatementHistory.Epoch.SameEpoch, StoreStatementHistory.DecideEpoch(Previous, null, null)); + + [Fact] + public void AStampAfterThePreviousCapture_IsAResetInsideTheInterval() => + Assert.Equal(StoreStatementHistory.Epoch.ResetInsideInterval, StoreStatementHistory.DecideEpoch(Previous, Previous.AddDays(-3), Previous.AddMinutes(20))); + + [Fact] + public void AStampBeforeThePreviousCapture_ThatDiffers_Rebaselines() => + Assert.Equal(StoreStatementHistory.Epoch.Rebaseline, StoreStatementHistory.DecideEpoch(Previous, Previous.AddDays(-3), Previous.AddDays(-1))); + + [Fact] + public void AStampThatVanished_Rebaselines() => + Assert.Equal(StoreStatementHistory.Epoch.Rebaseline, StoreStatementHistory.DecideEpoch(Previous, Previous.AddDays(-3), null)); + + [Fact] + public void TheSnapshot_ReadsOnlyTheReaderFunction_RanksByTimeAndBindsTheCap() + { + Assert.DoesNotContain("pg_stat_statements", StoreStatementHistory.SnapshotSql + StoreStatementHistory.CaptureCurrentSql, StringComparison.Ordinal); + Assert.Contains("config.store_statement_stats()", StoreStatementHistory.CaptureCurrentSql, StringComparison.Ordinal); + Assert.Contains("config.store_statement_stats_info()", StoreStatementHistory.InfoSql, StringComparison.Ordinal); + Assert.Contains("queryid IS NOT NULL", StoreStatementHistory.SnapshotSql, StringComparison.Ordinal); + Assert.Contains("ORDER BY a.delta_total_exec_ms DESC", StoreStatementHistory.SnapshotSql, StringComparison.Ordinal); + Assert.Contains("LIMIT $2", StoreStatementHistory.SnapshotSql, StringComparison.Ordinal); + Assert.DoesNotMatch(@"LIMIT\s+\d", StoreStatementHistory.SnapshotSql); + } + + [Fact] + public void TheRetentionDeletes_AreOnCaptureTime_AndTheBaselineIsNeverPurged() + { + Assert.Matches(@"WHERE capture_time < \$1", StoreStatementHistory.HistoryRetentionDeleteSql); + Assert.Matches(@"WHERE capture_time < \$1", StoreStatementHistory.CaptureRetentionDeleteSql); + Assert.Equal(90, StoreStatementHistory.RetentionDays); + Assert.Equal(100, StoreStatementHistory.TopStatements); + } + + [Fact] + public void EveryCommandInTheFile_SetsADeadline_ThroughOneHelper() + { + var source = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "StoreStatementHistory.cs"); + var constructions = Regex.Matches(source, @"new\s+NpgsqlCommand\b|\bnew\(sql, connection, transaction\)"); + Assert.Single(constructions); + Assert.Contains("CommandTimeout = SnapshotTimeoutSeconds", source, StringComparison.Ordinal); + } + + [Fact] + public void TheWorkerRunsTheSnapshot_InItsOwnCatch_BeforeTheCollectorCostFlush() + { + var worker = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "DarlingWorker.cs"); + var call = worker.IndexOf("await StoreStatementHistory.SnapshotAsync(", StringComparison.Ordinal); + var flush = worker.IndexOf("await _collectorCost.FlushAsync(", StringComparison.Ordinal); + Assert.True(call > 0 && call < flush, "the snapshot sits before the collector-cost flush"); + + var tail = worker[call..flush]; + Assert.Contains("catch (Exception ex) when (ex is not OperationCanceledException", tail, StringComparison.Ordinal); + Assert.Contains("historyBudget.IsCancellationRequested && !budget.IsCancellationRequested", tail, StringComparison.Ordinal); + Assert.Contains("_storeStatementHistoryWarned = false;", tail, StringComparison.Ordinal); + Assert.Contains("SqlState", tail, StringComparison.Ordinal); + Assert.Contains("_storeStatementHistoryWarned", tail, StringComparison.Ordinal); + Assert.Contains("LogWarning", tail, StringComparison.Ordinal); + Assert.Contains("LogDebug", tail, StringComparison.Ordinal); + } + + /// + /// The snapshot runs under its own linked slice of the tick's budget, like the re-mask, and a slice that expired + /// while the budget did not is caught, so the flushes after it still run. + /// + [Fact] + public void TheWorkerGivesTheSnapshotItsOwnSlice_SoAnExpiredSliceDoesNotSkipTheFlushes() + { + var worker = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "DarlingWorker.cs"); + Assert.Contains("using var historyBudget = CancellationTokenSource.CreateLinkedTokenSource(budget.Token);", worker, StringComparison.Ordinal); + Assert.Contains("historyBudget.CancelAfter(StoreStatementHistory.SliceBudget);", worker, StringComparison.Ordinal); + Assert.Contains("await StoreStatementHistory.SnapshotAsync(connection, DateTime.UtcNow, _logger, historyBudget.Token);", worker, StringComparison.Ordinal); + Assert.DoesNotContain("SnapshotAsync(connection, DateTime.UtcNow, _logger, budget.Token)", worker, StringComparison.Ordinal); + Assert.True(StoreStatementHistory.SliceBudget < TimeSpan.FromSeconds(300)); + } + + [Fact] + public void TheReadyCheck_AlsoRequiresTheExtensionLoadedAndTheTempPrivilege() + { + Assert.Contains("name = 'pg_stat_statements.max'", StoreStatementHistory.ReaderReadySql, StringComparison.Ordinal); + Assert.Contains("has_database_privilege(current_database(), 'TEMP')", StoreStatementHistory.ReaderReadySql, StringComparison.Ordinal); + Assert.Contains("a.queryid, a.role_name", StoreStatementHistory.SnapshotSql, StringComparison.Ordinal); + Assert.Contains("$6::integer", StoreStatementHistory.SnapshotSql, StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/StoreStatementStatsLiveTests.cs b/Darling/Darling.Tests/StoreStatementStatsLiveTests.cs index 56946e81a9..bed04ad82a 100644 --- a/Darling/Darling.Tests/StoreStatementStatsLiveTests.cs +++ b/Darling/Darling.Tests/StoreStatementStatsLiveTests.cs @@ -54,6 +54,13 @@ public sealed class StoreStatementStatsLiveTests private const string PlainOwnerPassword = "PlainOwnerLiteral3899"; private const string ScrubPassword = "ScrubLiteral3899"; + /* #5602: roles and databases are cluster-wide, so each run names its own and no other test can see them. */ + private static readonly string RunSuffix = Guid.NewGuid().ToString("N")[..8]; + private static readonly string Reader = "pgss_reader_" + RunSuffix; + private static readonly string Outsider = "pgss_outsider_" + RunSuffix; + private static readonly string PlainOwner = "pgss_plainowner_" + RunSuffix; + private static readonly string PlainDb = "pgss_plain_" + RunSuffix; + [Fact] public async Task OnTheProductsOwnConf_StatementsRankByRole_AndARolePasswordNeverReachesTheView_Gated() { @@ -88,22 +95,22 @@ tracking off. */ Assert.Equal("off", await ScalarAsync(c, "SHOW pg_stat_statements.track_utility", ct)); await ExecAsync(c, "CREATE SCHEMA IF NOT EXISTS config", ct); - await ExecAsync(c, $"CREATE ROLE pgss_reader LOGIN PASSWORD '{ReaderPassword}'", ct); - await ExecAsync(c, $"CREATE ROLE pgss_outsider LOGIN PASSWORD '{OutsiderPassword}'", ct); + await ExecAsync(c, $"CREATE ROLE {Reader} LOGIN PASSWORD '{ReaderPassword}'", ct); + await ExecAsync(c, $"CREATE ROLE {Outsider} LOGIN PASSWORD '{OutsiderPassword}'", ct); /* Schema USAGE for both, as provisioning gives the product's roles: the outsider's refusal below must come from the missing EXECUTE grant, not from the schema. */ - await ExecAsync(c, "GRANT USAGE ON SCHEMA config TO pgss_reader, pgss_outsider", ct); + await ExecAsync(c, $"GRANT USAGE ON SCHEMA config TO {Reader}, {Outsider}", ct); Assert.Equal( StoreStatementStats.SetupOutcome.Ready, - await StoreStatementStats.EnsureAsync(c, "config", ["pgss_reader", "pgss_absent"], NullLogger.Instance, ct)); + await StoreStatementStats.EnsureAsync(c, "config", [Reader, "pgss_absent"], NullLogger.Instance, ct)); /* Both CREATE ROLEs above carried a password literal, and neither was recorded. */ Assert.Equal(0L, await CountRecordedAsync(c, "Literal3899", ct)); } - var readerCs = new NpgsqlConnectionStringBuilder(ownerCs.ConnectionString) { Username = "pgss_reader", Password = ReaderPassword }; + var readerCs = new NpgsqlConnectionStringBuilder(ownerCs.ConnectionString) { Username = Reader, Password = ReaderPassword }; await using var reader = NpgsqlDataSource.Create(readerCs.ConnectionString); await using (var r = await reader.OpenConnectionAsync(ct)) { @@ -121,14 +128,14 @@ tracking off. */ var root0 = doc.RootElement; var statements = root0.GetProperty("statements").EnumerateArray().ToList(); var probe = Assert.Single(statements, s => QueryOf(s).Contains("pgss_probe_marker", StringComparison.Ordinal)); - Assert.Equal("pgss_reader", probe.GetProperty("role").GetString()); + Assert.Equal(Reader, probe.GetProperty("role").GetString()); Assert.Equal(3L, probe.GetProperty("calls").GetInt64()); Assert.Contains("$1 AS pgss_probe_marker", QueryOf(probe), StringComparison.Ordinal); Assert.Equal(JsonValueKind.String, probe.GetProperty("query_id").ValueKind); Assert.True(long.TryParse(probe.GetProperty("query_id").GetString(), out _)); Assert.DoesNotContain(statements, s => QueryOf(s).Contains("Literal3899", StringComparison.Ordinal)); Assert.Contains(root0.GetProperty("by_role").EnumerateArray(), row => row.GetProperty("role").GetString() == "owner"); - Assert.Contains(root0.GetProperty("by_role").EnumerateArray(), row => row.GetProperty("role").GetString() == "pgss_reader"); + Assert.Contains(root0.GetProperty("by_role").EnumerateArray(), row => row.GetProperty("role").GetString() == Reader); Assert.False(string.IsNullOrEmpty(root0.GetProperty("stats_since").GetString())); Assert.False(root0.GetProperty("utility_statements_tracked").GetBoolean()); Assert.False(root0.GetProperty("connected_as_owner").GetBoolean()); @@ -146,7 +153,7 @@ tracking off. */ } /* A role outside the grant is told what it lacks, from the catalog, not handed an error. */ - var outsiderCs = new NpgsqlConnectionStringBuilder(ownerCs.ConnectionString) { Username = "pgss_outsider", Password = OutsiderPassword }; + var outsiderCs = new NpgsqlConnectionStringBuilder(ownerCs.ConnectionString) { Username = Outsider, Password = OutsiderPassword }; await using (var outsider = NpgsqlDataSource.Create(outsiderCs.ConnectionString)) { var denied = await DarlingMcpStoreQueryStatsTools.GetStoreQueryStats(outsider); @@ -187,7 +194,7 @@ await DarlingMcpStoreQueryStatsTools.GetStoreQueryStats(reader, top: 1000), await using (var c = await OpenAsync(ownerCs.ConnectionString, ct)) { await ExecAsync(c, "SET pg_stat_statements.track_utility = on", ct); - await ExecAsync(c, $"ALTER ROLE pgss_outsider PASSWORD '{ScrubPassword}'", ct); + await ExecAsync(c, $"ALTER ROLE {Outsider} PASSWORD '{ScrubPassword}'", ct); await ExecAsync(c, "RESET pg_stat_statements.track_utility", ct); Assert.Equal(1L, await CountRecordedAsync(c, ScrubPassword, ct)); } @@ -226,11 +233,11 @@ await DarlingMcpStoreQueryStatsTools.GetStoreQueryStats(reader, top: 1000, full_ await ExecAsync(c, "SET pg_stat_statements.track_utility = on", ct); Assert.Equal( StoreStatementStats.SetupOutcome.Ready, - await StoreStatementStats.EnsureAsync(c, "config", ["pgss_reader"], firstPass, ct)); + await StoreStatementStats.EnsureAsync(c, "config", [Reader], firstPass, ct)); Assert.Equal(0L, await CountRecordedAsync(c, ScrubPassword, ct)); Assert.Equal( StoreStatementStats.SetupOutcome.Ready, - await StoreStatementStats.EnsureAsync(c, "config", ["pgss_reader"], secondPass, ct)); + await StoreStatementStats.EnsureAsync(c, "config", [Reader], secondPass, ct)); } Assert.Contains("Warning: Statement statistics: pg_stat_statements.track_utility is on", firstPass.Joined, StringComparison.Ordinal); @@ -244,29 +251,29 @@ await DarlingMcpStoreQueryStatsTools.GetStoreQueryStats(reader, top: 1000, full_ and what fixes it, until pg_read_all_stats is granted. */ await using (var c = await OpenAsync(ownerCs.ConnectionString, ct)) { - await ExecAsync(c, $"CREATE ROLE pgss_plainowner LOGIN NOSUPERUSER PASSWORD '{PlainOwnerPassword}'", ct); - await ExecAsync(c, "CREATE DATABASE pgss_plain OWNER pgss_plainowner", ct); + await ExecAsync(c, $"CREATE ROLE {PlainOwner} LOGIN NOSUPERUSER PASSWORD '{PlainOwnerPassword}'", ct); + await ExecAsync(c, $"CREATE DATABASE {PlainDb} OWNER {PlainOwner}", ct); } - var plainSuperCs = new NpgsqlConnectionStringBuilder(ownerCs.ConnectionString) { Database = "pgss_plain" }; + var plainSuperCs = new NpgsqlConnectionStringBuilder(ownerCs.ConnectionString) { Database = PlainDb }; await using (var c = await OpenAsync(plainSuperCs.ConnectionString, ct)) { await ExecAsync(c, "CREATE EXTENSION pg_stat_statements", ct); - await ExecAsync(c, "CREATE SCHEMA config AUTHORIZATION pgss_plainowner", ct); - await ExecAsync(c, "GRANT USAGE ON SCHEMA config TO pgss_reader", ct); + await ExecAsync(c, $"CREATE SCHEMA config AUTHORIZATION {PlainOwner}", ct); + await ExecAsync(c, $"GRANT USAGE ON SCHEMA config TO {Reader}", ct); await ScalarAsync(c, "SELECT 3899 AS pgss_hidden_marker", ct); } - var plainOwnerCs = new NpgsqlConnectionStringBuilder(plainSuperCs.ConnectionString) { Username = "pgss_plainowner", Password = PlainOwnerPassword }; + var plainOwnerCs = new NpgsqlConnectionStringBuilder(plainSuperCs.ConnectionString) { Username = PlainOwner, Password = PlainOwnerPassword }; await using (var c = await OpenAsync(plainOwnerCs.ConnectionString, ct)) { Assert.False(await ScalarAsync(c, "SELECT pg_catalog.pg_has_role(current_user, 'pg_read_all_settings', 'USAGE')", ct)); Assert.Equal( StoreStatementStats.SetupOutcome.Ready, - await StoreStatementStats.EnsureAsync(c, "config", ["pgss_reader"], NullLogger.Instance, ct)); + await StoreStatementStats.EnsureAsync(c, "config", [Reader], NullLogger.Instance, ct)); } - var plainReaderCs = new NpgsqlConnectionStringBuilder(readerCs.ConnectionString) { Database = "pgss_plain" }; + var plainReaderCs = new NpgsqlConnectionStringBuilder(readerCs.ConnectionString) { Database = PlainDb }; await using (var plainReader = NpgsqlDataSource.Create(plainReaderCs.ConnectionString)) { using (var doc = JsonDocument.Parse(await DarlingMcpStoreQueryStatsTools.GetStoreQueryStats(plainReader, top: 1000, full_text: true))) @@ -277,7 +284,7 @@ await DarlingMcpStoreQueryStatsTools.GetStoreQueryStats(reader, top: 1000, full_ await using (var c = await OpenAsync(ownerCs.ConnectionString, ct)) { - await ExecAsync(c, "GRANT pg_read_all_stats TO pgss_plainowner", ct); + await ExecAsync(c, $"GRANT pg_read_all_stats TO {PlainOwner}", ct); } using (var doc = JsonDocument.Parse(await DarlingMcpStoreQueryStatsTools.GetStoreQueryStats(plainReader, top: 1000, full_text: true))) diff --git a/Darling/Darling.Tests/StoreStatementStatsTests.cs b/Darling/Darling.Tests/StoreStatementStatsTests.cs index 43aa271fcb..37538c57f3 100644 --- a/Darling/Darling.Tests/StoreStatementStatsTests.cs +++ b/Darling/Darling.Tests/StoreStatementStatsTests.cs @@ -11,6 +11,7 @@ using System.Text.RegularExpressions; using System.Threading.Tasks; using Npgsql; +using PerformanceMonitor.Collectors; using PerformanceMonitor.Darling.Service; using PerformanceMonitor.Darling.Service.Mcp; using PerformanceMonitor.Darling.Storage; @@ -46,7 +47,12 @@ public void TheReaderFunctions_AreDefinersWithAPinnedPathAndStringMode_RevokedFr does not name; everything else keeps its timings and reads as withheld, and no row is filtered out on its text any more. */ Assert.Contains($"WHEN s.query ~* {Literal(StoreStatementStats.ReadableStatementPattern)}", sql, StringComparison.Ordinal); - Assert.Contains($"AND s.query !~* {Literal(StoreStatementStats.SensitiveStatementPattern)}", sql, StringComparison.Ordinal); + /* #5320: the sensitive pattern rides hex-encoded, so the function's recorded text does not name itself; + what the server decodes is byte-identical to the shared pattern. */ + var operand = Regex.Match(sql, @"AND s\.query !~\* \(SELECT pg_catalog\.convert_from\(pg_catalog\.decode\('([0-9a-f]+)', 'hex'\), 'UTF8'\)\)"); + Assert.True(operand.Success, "the reader function must test the sensitive pattern through the hex operand"); + Assert.Equal(StoreStatementStats.SensitiveStatementPattern, System.Text.Encoding.UTF8.GetString(Convert.FromHexString(operand.Groups[1].Value))); + Assert.DoesNotContain($"!~* {Literal(StoreStatementStats.SensitiveStatementPattern)}", sql, StringComparison.Ordinal); /* #3920's review: before PostgreSQL 16, pg_stat_statements keeps a SELECT ... INTO's constants as typed. */ Assert.Contains("AND NOT (pg_catalog.current_setting('server_version_num')::integer < 160000", sql, StringComparison.Ordinal); Assert.Contains($"AND s.query ~* {Literal(StoreStatementStats.SelectIntoPattern)}) THEN s.query", sql, StringComparison.Ordinal); @@ -147,6 +153,32 @@ public void TheGrant_CoversBothFunctions_OnlyForRolesThatExist() Assert.Throws(() => StoreStatementStats.BuildGrantSql("config", null!)); } + /// + /// #5320: the setup statements a pass runs are utility statements, and pg_stat_statements records them as typed + /// while track_utility is on. The scrub removes whatever the shared statement filter names, so a setup + /// statement the filter names would be recorded, removed and reported as a credential-bearing statement on + /// every pass. The filter's keyword-only alternatives name a few procedure and function names on their own, + /// which the reader function's text used to carry inside its pattern literal. + /// + [Fact] + public void TheSetupStatements_AreNotNamedByTheStatementFilter_SoTheScrubNeverRemovesTheProductsOwnDdl() + { + foreach (var withInfoView in new[] { true, false }) + { + Assert.False(PerformanceMonitor.Common.SensitiveStatements.Names(StoreStatementStats.BuildFunctionSql("config", "public", withInfoView)), + $"the reader function's text (withInfoView={withInfoView}) must not be named by the statement filter"); + } + + Assert.False(PerformanceMonitor.Common.SensitiveStatements.Names(StoreStatementStats.BuildGrantSql("config", ["pgss_reader", "pgss_other"])), + "the grant statements must not be named by the statement filter"); + Assert.False(PerformanceMonitor.Common.SensitiveStatements.Names(StoreStatementStats.CreateExtensionSql)); + + /* The guard bites: the pattern's own text, spelled as a literal, IS named. */ + var literal = "AND s.query !~* " + Literal(StoreStatementStats.SensitiveStatementPattern); + Assert.True(PerformanceMonitor.Common.SensitiveStatements.Names(literal), + "the shared pattern's own text is expected to be named, which is why the function embeds it encoded"); + } + /// The scrub resets only the sensitive statements of THIS database (#3904's review: without the dbid /// predicate a superuser service login on a shared cluster reset other databases' entries every pass), and /// never through query id 0, which pg_stat_statements_reset reads as "every statement of the role". @@ -398,4 +430,27 @@ public void TheToolMasksEveryTextItShows_AndLeavesNormalizedTextAsItWas() Assert.Equal(StoreStatementStats.InsufficientPrivilegeText, DarlingMcpStoreQueryStatsTools.ShownText(StoreStatementStats.InsufficientPrivilegeText)); Assert.Equal("", DarlingMcpStoreQueryStatsTools.ShownText("")); } + + /// + /// #5097: a text the shared sensitive-statement filter withheld in SQL arrives as its placeholder, which is a SQL line comment + /// that the lexer strips to an empty string. It must read as withheld instead, so a statement withheld by the tools' second + /// layer is not mistaken for one that has no text. + /// + [Fact] + public void ATextTheSharedFilterWithheld_ReadsAsWithheld_NotAsAnEmptyString() + { + Assert.Equal(StoreStatementStats.WithheldText, DarlingMcpStoreQueryStatsTools.ShownText(PgSensitiveStatementFilter.PlaceholderText)); + Assert.NotEqual("", DarlingMcpStoreQueryStatsTools.ShownText(PgSensitiveStatementFilter.PlaceholderText)); + } + + /// + /// #5097: the ranked read applies the shared predicate on top of the reader function's own filter, as the history tool's text + /// read does, so the diagnostics bundle's two statement-text members agree on a store whose function body is older than the + /// pattern. + /// + [Fact] + public void TheRankedRead_WrapsTheReaderFunctionsText_InTheSharedSensitiveStatementPredicate() + { + Assert.Contains(PgSensitiveStatementFilter.SqlPredicate("ranked.query"), DarlingMcpStoreQueryStatsTools.BuildStatementsSql("total_exec_ms"), StringComparison.Ordinal); + } } diff --git a/Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs b/Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs index d81df0c88f..50d6d6d8df 100644 --- a/Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs +++ b/Darling/Darling.Tests/StoreToastAndCheckpointerTests.cs @@ -1343,19 +1343,28 @@ public void TheWorker_EvaluatesBothOnTheStoreSelfMetricsTick_AfterTheSweep() .Replace("\r\n", "\n", StringComparison.Ordinal); var tick = worker.IndexOf("if (StampIsDue(_nextStoreMetricsUtc, s_storeMetricsInterval, DateTime.UtcNow))", StringComparison.Ordinal); - var window = worker.IndexOf("var checkpointWindow = _checkpointSyncSampler.TakeWindowMax();", tick, StringComparison.Ordinal); - var sweep = worker.IndexOf("await SweepStoreSelfMetricsAsync(checkpointWindow, stoppingToken);", tick, StringComparison.Ordinal); - var toast = worker.IndexOf("await _selfAlerts.EvaluateToastSlackAsync(_postgres!, stoppingToken);", sweep, StringComparison.Ordinal); - var checkpointer = worker.IndexOf("await _selfAlerts.EvaluateCheckpointerPressureAsync(_postgres!, stoppingToken);", toast, StringComparison.Ordinal); var nextTick = worker.IndexOf("await Task.Delay(s_sweepInterval, stoppingToken);", tick, StringComparison.Ordinal); - Assert.True(tick >= 0 && sweep > tick && toast > sweep && checkpointer > toast && checkpointer < nextTick, + /* #4970: the loop only launches the tick; the window is taken inside the launcher, ahead of the launch, and + the sweep and its evaluators run in the tick body, the sweep first. */ + var launch = worker.IndexOf("TryStartStoreMetricsTick(DateTime.UtcNow, _checkpointSyncSampler.TakeWindowMax,", tick, StringComparison.Ordinal); + Assert.True(tick >= 0 && launch > tick && launch < nextTick, "the loop must launch the store self-metrics tick"); + + var launcher = worker.IndexOf("internal bool TryStartStoreMetricsTick(", StringComparison.Ordinal); + var window = worker.IndexOf("var checkpointWindow = takeWindow();", launcher, StringComparison.Ordinal); + var run = worker.IndexOf("RunTrackedTickAsync(\"hourly store self-metrics tick\"", window, StringComparison.Ordinal); + Assert.True(launcher > 0 && window > launcher && run > window, "the launcher must take the sampler's window before the tick that stores it"); + + var body = worker.IndexOf("private async Task RunStoreMetricsTickAsync(", run, StringComparison.Ordinal); + var sweep = worker.IndexOf("await SweepStoreSelfMetricsAsync(checkpointWindow, stoppingToken);", body, StringComparison.Ordinal); + var toast = worker.IndexOf("await _selfAlerts.EvaluateToastSlackAsync(_postgres!, stoppingToken);", sweep, StringComparison.Ordinal); + var checkpointer = worker.IndexOf("await _selfAlerts.EvaluateCheckpointerPressureAsync(_postgres!, stoppingToken);", toast, StringComparison.Ordinal); + Assert.True(body > 0 && sweep > body && toast > sweep && checkpointer > toast, "the two #3783 conditions must be evaluated on the store self-metrics tick, after the sweep that writes their evidence"); - /* #4834: the window is taken ONCE, at the top of the tick and ahead of the sweep, and handed to the sweep - that stores it. The evaluator reads it back from the stored row, so it is never handed a window. */ - Assert.True(window > tick && window < sweep, "the tick must take the sampler's window before the sweep that stores it"); - Assert.Single(Regex.Matches(worker, @"TakeWindowMax\(")); + /* #4834: the window is taken ONCE, ahead of the sweep, and handed to the sweep that stores it. The + evaluator reads it back from the stored row, so it is never handed a window. */ + Assert.Single(Regex.Matches(worker, @"TakeWindowMax\b")); Assert.Contains("checkpointLongestSync?.SyncMs, checkpointLongestSync?.SampledUtc", worker, StringComparison.Ordinal); /* Exactly one call site each — the census in DarlingSelfAlertTests asserts they exist; this asserts where. */ diff --git a/Darling/Darling.Tests/StoredPlanAnalysisFilterTests.cs b/Darling/Darling.Tests/StoredPlanAnalysisFilterTests.cs new file mode 100644 index 0000000000..9df30c7957 --- /dev/null +++ b/Darling/Darling.Tests/StoredPlanAnalysisFilterTests.cs @@ -0,0 +1,182 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Runtime.CompilerServices; +using System.Security; +using System.Text.Json; +using System.Text.RegularExpressions; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.PlanAnalysis; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348: the three stored-plan analysis tools (analyze_query_plan, analyze_procedure_plan, +/// analyze_query_store_plan) lift parameter values and statement text out of the plan into fields of their own, +/// so the statement filter has to judge the stored XML before the analysis reads it, as analyze_plan_xml does. +/// A plan the filter withholds whole is answered with one plain sentence, not a parse error. +/// +public sealed class StoredPlanAnalysisFilterTests +{ + [Theory] + [InlineData("query_stats", "abc123")] + [InlineData("procedure_stats", "0x0200abcd")] + [InlineData("query_store", "Db:7")] + public void TheAnalysisOfTheCanaryPlan_HoldsNoSecret_ForEachStoredPlanSource(string source, string identifier) + { + string result = DarlingMcpPlanTools.AnalyzeFilteredPlan( + StatementScrubCanary.CanaryPlan(), "srv", source, identifier, null, null, default); + + foreach (string needle in StatementScrubCanary.SecretNeedles) + Assert.DoesNotContain(needle, result); + JsonDocument.Parse(result).Dispose(); + } + + [Theory] + [InlineData("query_stats")] + [InlineData("procedure_stats")] + [InlineData("query_store")] + public void ACleanPlansAnalysis_IsByteIdenticalToTheUnfilteredAnalysis(string source) + { + string plan = DarlingPlanFixtures.SamplePlanXml; + + string filtered = DarlingMcpPlanTools.AnalyzeFilteredPlan(plan, "srv", source, "id", null, null, default); + string direct = McpPlanAnalysisFormatter.BuildAnalysisResult(plan, "srv", source, "id", null, null, default); + + Assert.Equal(direct, filtered); + } + + [Theory] + [InlineData("query_stats")] + [InlineData("procedure_stats")] + [InlineData("query_store")] + [InlineData("xml")] + public void APlanTheFilterWithholdsWhole_IsAnsweredWithOnePlainSentence_NotAParseError(string source) + { + // A document that does not parse is judged as decoded text and withheld whole when it names a sensitive + // statement (the same outcome as a spent read budget). + string plan = " + /// Every product .cs file under (not bin, obj or + /// deprecated) is read with comments and string contents blanked by CSharpSourceWalker. The formatter's BuildAnalysisResult may appear exactly + /// once, qualified, in : a direct call in any other file, a call through + /// using static or an alias, and a method group all name it again and so fail. + /// + private static List FormatterCallProblems(string appRoot, string helperRelativePath) + { + var problems = new List(); + string helper = Path.GetFullPath(Path.Combine(appRoot, helperRelativePath)); + int mentions = 0; + foreach (string file in Directory.EnumerateFiles(appRoot, "*.cs", SearchOption.AllDirectories)) + { + string relative = Path.GetRelativePath(appRoot, file); + string[] parts = relative.Split(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar); + if (parts.Take(parts.Length - 1).Any(p => p is "bin" or "obj" or "deprecated")) continue; + + bool isHelper = string.Equals(Path.GetFullPath(file), helper, StringComparison.OrdinalIgnoreCase); + // Read as code, not by line prefix (#3052): the walker blanks line, block and doc comments and string + // contents wherever they sit, so a mention inside a block comment's continuation lines or after code on a + // line is judged correctly, and code that follows a closing block comment on its line is still read. + foreach (string line in CSharpSourceWalker.StripCommentsAndStrings(File.ReadAllText(file)).Split('\n')) + { + if (Regex.IsMatch(line, @"\busing\s+static\s+[\w.]*McpPlanAnalysisFormatter\b")) + problems.Add(relative + ": using static of the formatter"); + + int here = Regex.Matches(line, @"\bBuildAnalysisResult\b").Count; + if (here == 0) continue; + mentions += here; + bool qualified = Regex.IsMatch(line, @"\bMcpPlanAnalysisFormatter\s*\.\s*BuildAnalysisResult\s*\("); + if (!isHelper || !qualified || here != 1) + problems.Add(relative + ": a call to BuildAnalysisResult outside the filtered helper"); + } + } + + if (mentions != 1) problems.Add("expected exactly one call to BuildAnalysisResult in the helper, found " + mentions); + return problems; + } + + [Fact] + public void EveryAnalysisToolHere_ReachesTheAnalyzerOnlyThroughTheFilteredHelper() + { + string appRoot = Path.Combine(RepoRoot(), "Darling", "PerformanceMonitor.Darling.Service"); + string helperRelativePath = Path.Combine("Mcp", "DarlingMcpPlanTools.cs"); + string source = File.ReadAllText(Path.Combine(appRoot, helperRelativePath)); + + // One direct call to the formatter across the whole app: the helper's own. The four tools call the helper. + Assert.Empty(FormatterCallProblems(appRoot, helperRelativePath)); + Assert.Equal(4, Regex.Matches(source, @"return AnalyzeFilteredPlan\(").Count); + Assert.Contains("xml = SensitiveStatements.Xml(xml) ?? SensitiveStatements.PlaceholderText;", source); + } + + [Fact] + public void TheFormatterCallScan_FlagsADirectCallInAnotherFile_AUsingStaticCall_AndIgnoresBinObjAndDeprecated() + { + string root = Path.Combine(Path.GetTempPath(), "spaf-" + Guid.NewGuid().ToString("N")); + try + { + Directory.CreateDirectory(root); + string helperLine = "return McpPlanAnalysisFormatter.BuildAnalysisResult(xml);"; + File.WriteAllText(Path.Combine(root, "Helper.cs"), + "// McpPlanAnalysisFormatter.BuildAnalysisResult( is named here\n" + helperLine); + Assert.Empty(FormatterCallProblems(root, "Helper.cs")); + + // Comments are not code wherever they sit: a block comment's unprefixed continuation lines, a trailing + // comment after code, and a string that spells the name are all ignored. + File.WriteAllText(Path.Combine(root, "Prose.cs"), + "/* McpPlanAnalysisFormatter.BuildAnalysisResult(xml) is the old shape,\n" + + " and this line was once read as code */\n" + + "class P { int x = 1; // BuildAnalysisResult( here\n" + + "string s = \"BuildAnalysisResult(\"; }"); + Assert.Empty(FormatterCallProblems(root, "Helper.cs")); + // ...but code after a closing block comment on the same line is still code. + File.WriteAllText(Path.Combine(root, "Prose.cs"), "/* note */ class P { object o = BuildAnalysisResult(x); }"); + Assert.NotEmpty(FormatterCallProblems(root, "Helper.cs")); + File.Delete(Path.Combine(root, "Prose.cs")); + + foreach (string ignored in new[] { "bin", "obj", "deprecated" }) + { + Directory.CreateDirectory(Path.Combine(root, ignored)); + File.WriteAllText(Path.Combine(root, ignored, "Skipped.cs"), helperLine); + } + Assert.Empty(FormatterCallProblems(root, "Helper.cs")); + + string other = Path.Combine(root, "Other.cs"); + File.WriteAllText(other, "class T { string M() { " + helperLine + " } }"); + Assert.NotEmpty(FormatterCallProblems(root, "Helper.cs")); + + File.WriteAllText(other, + "using static PerformanceMonitor.PlanAnalysis.McpPlanAnalysisFormatter;\nclass T { string M() { return BuildAnalysisResult(xml); } }"); + Assert.True(FormatterCallProblems(root, "Helper.cs").Count >= 2); + } + finally + { + Directory.Delete(root, recursive: true); + } + } + + private static string RepoRoot([CallerFilePath] string thisFile = "") + => Path.GetFullPath(Path.Combine(Path.GetDirectoryName(thisFile)!, "..", "..")); +} diff --git a/Darling/Darling.Tests/StragglerCommandTimeoutTests.cs b/Darling/Darling.Tests/StragglerCommandTimeoutTests.cs index eb24927102..90479cfad2 100644 --- a/Darling/Darling.Tests/StragglerCommandTimeoutTests.cs +++ b/Darling/Darling.Tests/StragglerCommandTimeoutTests.cs @@ -28,7 +28,8 @@ namespace Darling.Tests; /// true of these three: measured on the pre-change tree, Targets/HypotheticalIndexExperiment.cs holds /// 6 command sites and all 6 are this group's, PgPlanForceActionStore.cs holds 4 and all 4 are, and /// DarlingCliCommands.cs held 3 and all 3 were (4 since #3752 added the collector toggle verbs' -/// schedule read-back, which shares the CLI store-read budget). So a file here IS a regime, and the per-file census +/// schedule read-back, which shares the CLI store-read budget, and 6 since #4961 added the two store reads +/// --drop-xe-sessions makes before it connects to a server, which share it too). So a file here IS a regime, and the per-file census /// counts below are what keep it one — a fourth site added to any of these files fails on the count and asks /// a person whether it shares the budget, rather than inheriting the answer by adjacency. The reusable /// rule for the remaining groups is to scope at the smallest unit that is EXACT, and to check which unit @@ -78,7 +79,7 @@ private static readonly (string File, Hop Hop, int Sites, string Constant, Regex ("PgPlanForceActionStore.cs", Hop.Store, 4, nameof(ServiceCommandDeadlines.PostAnalysisForcePlanSeconds), Assigns(@"ServiceCommandDeadlines\.PostAnalysisForcePlanSeconds")), - ("DarlingCliCommands.cs", Hop.Store, 4, + ("DarlingCliCommands.cs", Hop.Store, 8, /* 7 since #4938: the collector run-time read-back is a store read of its own; 8 since #5366: the old-format password pin-table check */ "ServiceCommandDeadlines.Cli{StoreRead,BudgetBackstop}Seconds", Assigns(@"ServiceCommandDeadlines\.Cli(?:StoreRead|BudgetBackstop)Seconds")), }; @@ -153,7 +154,7 @@ Every pin landed in this sweep before #2940 matches its value regex over the raw } /// - /// The same fourteen sites, judged a SECOND way — through the shared + /// The same sixteen sites, judged a SECOND way — through the shared /// that #2938 extracted and #2874's other pins route through. /// /// Additive rather than a replacement, because the two ask different questions. The @@ -168,7 +169,7 @@ Every pin landed in this sweep before #2940 matches its value regex over the raw /// by side on the command plane. If they ever disagree, one of them is wrong and the build says so. /// /// Every site in these three files writes its deadline as an object initializer ON the - /// construction, so it is the scanner's CONSTRUCTION-span half that carries all fourteen; the + /// construction, so it is the scanner's CONSTRUCTION-span half that carries all sixteen; the /// name-qualified assignment half is inert here and is exercised instead by /// , whose nine /// timed sites are all assignments. diff --git a/Darling/Darling.Tests/SwallowedItemFailureTests.cs b/Darling/Darling.Tests/SwallowedItemFailureTests.cs index 95b67dd594..b5d29de919 100644 --- a/Darling/Darling.Tests/SwallowedItemFailureTests.cs +++ b/Darling/Darling.Tests/SwallowedItemFailureTests.cs @@ -66,8 +66,10 @@ public void TheRunner_StampsTheDatabase_OnTheAllFailedRethrow() /// /// The Azure arm of the long-query reconcile keeps the #2623 account over the databases it tried to /// ENABLE in, throws the first refusal when every database refused (so the worker does not latch and - /// the run classifies SESSION_MISSING), and returns the partial note when some did. The DROP arm is - /// deliberately not scored: no run is dispatched while disabled. + /// the run classifies SESSION_MISSING), and returns the partial note when some did. Drops are + /// deliberately not scored: no run is dispatched while disabled, and a drop outside the monitored set is + /// not a capture outage. The scored loop runs only on the ENABLE side; drops go through the shared + /// LongQueryTraceDatabases.DropEachAsync. /// [Fact] public void TheAzureReconcile_ThrowsWhenEveryDatabaseRefused_AndReturnsThePartialNoteWhenSomeDid() @@ -77,19 +79,22 @@ public void TheAzureReconcile_ThrowsWhenEveryDatabaseRefused_AndReturnsThePartia Assert.Contains("public static async Task ReconcileLongQueryCompletionsAsync(", source, StringComparison.Ordinal); Assert.Contains("private static async Task ReconcileLongQueryCompletionsAzureAsync(", source, StringComparison.Ordinal); - /* Scored only while enabling. */ - Assert.Contains("if (enabled)\n {\n attempted++;\n }", source, StringComparison.Ordinal); - Assert.Contains("if (enabled)\n {\n failed++;\n failedDatabases.Add(databaseName);\n CollectorFaultDatabase.Stamp(ex, databaseName);\n firstFailure ??= ex;\n }", source, StringComparison.Ordinal); + /* Scored only while enabling: the loop over the plan's create list counts, the drops do not. */ + Assert.Contains("foreach (var databaseName in LongQueryTraceDatabases.Plan(enabled: true, Array.Empty(), monitored, separatelyMonitored, keptElsewhere: Array.Empty()).Create)\n {\n cancellationToken.ThrowIfCancellationRequested();\n attempted++;", source, StringComparison.Ordinal); + Assert.Contains("failed++;\n failedDatabases.Add(databaseName);\n CollectorFaultDatabase.Stamp(ex, databaseName);\n firstFailure ??= ex;", source, StringComparison.Ordinal); + Assert.Contains("LongQueryTraceDatabases.DropEachAsync(", source, StringComparison.Ordinal); /* All refused: the first failure, raw, after the summary line. Same predicate as the runners'. */ Assert.Contains("if (attempted > 0 && failed == attempted && firstFailure is not null)", source, StringComparison.Ordinal); Assert.Contains("System.Runtime.ExceptionServices.ExceptionDispatchInfo.Capture(firstFailure).Throw();", source, StringComparison.Ordinal); - /* Some refused: the shared composer, so the wording is the runners'. */ + /* Some refused: the shared composer, so the wording is the runners'. The first failure is worded as the per-database + line words it, so a create or start refused on Azure carries the caps sentence (#4961). */ Assert.Contains( - "return EnumeratedCollectorDriver.BuildPartialFailureNote(failed, attempted, failedDatabases, firstFailure?.Message);", + "var partialNote = EnumeratedCollectorDriver.BuildPartialFailureNote(failed, attempted, failedDatabases, firstFailure is null ? null : AlwaysOnXeSessions.DescribeFailure(firstFailure));", source, StringComparison.Ordinal); + Assert.Contains("return partialNote;", source, StringComparison.Ordinal); /* The per-database warning that names the refusing database is KEPT - the account rides beside it, it does not replace it. */ @@ -109,17 +114,21 @@ public void TheWorker_ClassifiesARefusedSessionAsSessionMissing_BeforeDispatchin { var source = ReadRepoFileLf("Darling", "PerformanceMonitor.Darling.Service", "DarlingWorker.cs"); - /* The two slots on the loop state. */ - Assert.Contains("public string? LongQueryTraceFault { get; set; }", source, StringComparison.Ordinal); + /* The slots on the loop state. #5378: the fault became a property over a backing field, so that clearing + it also clears the permission flag that rides beside it (a stale flag would mislabel the next fault). */ + Assert.Contains("public string? LongQueryTraceFault\n", source, StringComparison.Ordinal); + Assert.Contains("_longQueryTraceFault = value;\n if (value is null)\n {\n LongQueryTraceFaultIsPermission = false;", source, StringComparison.Ordinal); + Assert.Contains("public bool LongQueryTraceFaultIsPermission { get; set; }", source, StringComparison.Ordinal); Assert.Contains("public string? LongQueryTracePartialNote { get; set; }", source, StringComparison.Ordinal); /* The reconcile fills them: partial note on success (enabled only), fault on a throw (enabled only). */ Assert.Contains("var partialNote = await DarlingXeSessions.ReconcileLongQueryCompletionsAsync(", source, StringComparison.Ordinal); Assert.Contains("server.LongQueryTracePartialNote = enabled ? partialNote : null;", source, StringComparison.Ordinal); var reconcileCatch = source.IndexOf("Failed to reconcile the long-query completion XE session: {Message}", StringComparison.Ordinal); - var faultSet = source.IndexOf("server.LongQueryTraceFault = refusedIn is null", StringComparison.Ordinal); + var faultSet = source.IndexOf("server.LongQueryTraceFault = permissionDenied", StringComparison.Ordinal); Assert.True(reconcileCatch > 0 && faultSet > reconcileCatch, "the fault is recorded in the reconcile's catch"); Assert.Contains("var refusedIn = CollectorFaultDatabase.For(ex, fallback: null);", source, StringComparison.Ordinal); + Assert.Contains("server.LongQueryTraceFaultIsPermission = permissionDenied;", source, StringComparison.Ordinal); /* Reset with the latch on every (re)connect. */ var latchReset = source.IndexOf("server.LongQueryTraceApplied = null;", StringComparison.Ordinal); @@ -131,18 +140,24 @@ public void TheWorker_ClassifiesARefusedSessionAsSessionMissing_BeforeDispatchin /* RunOneAsync: the fault throws the SESSION_MISSING type BEFORE the dispatch; the partial note is merged onto HostNote AFTER it. Both gated on the collector's own declared name. */ var runOne = source.IndexOf("private async Task RunOneAsync(", StringComparison.Ordinal); + /* #5378: the fault throws the PERMISSIONS type when the reconcile's refusal was a permission denial, and + the SESSION_MISSING type for any other refusal - both before the dispatch. */ var faultThrow = source.IndexOf( - "if (IsLongQueryCompletionsCollector(collectorName) && server.LongQueryTraceFault is { } traceFault)\n {\n throw new DarlingXeSessionMissingException(traceFault);\n }", + "if (IsLongQueryCompletionsCollector(collectorName) && server.LongQueryTraceFault is { } traceFault)\n {", + StringComparison.Ordinal); + var permissionThrow = source.IndexOf( + "if (server.LongQueryTraceFaultIsPermission)\n {\n throw new DarlingXeSessionDeniedException(traceFault);\n }\n\n throw new DarlingXeSessionMissingException(traceFault);\n }", StringComparison.Ordinal); var dispatch = source.IndexOf("var result = await run(runner, runtime, cancellationToken);", StringComparison.Ordinal); var noteMerge = source.IndexOf( "result = result with { HostNote = EnumeratedCollectorDriver.MergeNotes(result.HostNote, partialNote) };", StringComparison.Ordinal); - Assert.True(runOne > 0 && faultThrow > runOne && dispatch > faultThrow && noteMerge > dispatch, - "fault check, then dispatch, then partial-note merge, all inside RunOneAsync"); + Assert.True(runOne > 0 && faultThrow > runOne && permissionThrow > faultThrow && dispatch > permissionThrow && noteMerge > dispatch, + "fault check (permission refusal, else session missing), then dispatch, then partial-note merge, all inside RunOneAsync"); /* The message-only constructor the pre-dispatch throw needs, beside the wrapping one. */ Assert.Contains("public DarlingXeSessionMissingException(string message) : base(message) { }", source, StringComparison.Ordinal); + Assert.Contains("public DarlingXeSessionDeniedException(string message) : base(message) { }", source, StringComparison.Ordinal); /* Name-guarded on the collector's OWN name, not a literal. */ Assert.Contains( diff --git a/Darling/Darling.Tests/SweepBodyDetachPolicyTests.cs b/Darling/Darling.Tests/SweepBodyDetachPolicyTests.cs index 883835a7f6..665fee5026 100644 --- a/Darling/Darling.Tests/SweepBodyDetachPolicyTests.cs +++ b/Darling/Darling.Tests/SweepBodyDetachPolicyTests.cs @@ -25,8 +25,10 @@ namespace Darling.Tests; /// /// /// query_store — p50 3,350ms, p90 65,053ms, max 364,202ms -/// index_object_stats — p90 15,640ms, but 1440-minute cadence (42 runs/day -/// fleet-wide, one per server per day), so its body impact is amortised to nil +/// index_object_stats — p90 15,640ms, 1440-minute cadence (42 runs/day +/// fleet-wide, one per server per day). Its run is detached from the body too (#4938), by cadence rather +/// than by name, so it is not in the by-name set pinned below. How rarely it runs says nothing about how +/// long the body would wait on it when it does (43 minutes across 72 databases in a field case) /// procedure_stats — p50 5,982ms, p90 11,964ms, but 1-minute tier /// query_stats — p50 2,070ms, p90 5,062ms, 1-minute tier /// plan_correction — p50 2,681ms, max 133,934ms — the same bimodal diff --git a/Darling/Darling.Tests/TargetProviderTests.cs b/Darling/Darling.Tests/TargetProviderTests.cs index a0a9781e29..d4544957a3 100644 --- a/Darling/Darling.Tests/TargetProviderTests.cs +++ b/Darling/Darling.Tests/TargetProviderTests.cs @@ -12,6 +12,7 @@ using Microsoft.Data.SqlClient; using Npgsql; using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; using PerformanceMonitor.Darling.Service.Targets; using Xunit; @@ -151,6 +152,18 @@ public void TreatsALockTimeoutAsAYieldOnlyForCollectorsThatOptIn() PostgresTargetProvider.Instance.Classify(pgLockTimeout, yieldsOnLockTimeout: false)); } + /// + /// #5378: 15247, what CREATE or ALTER EVENT SESSION raises for a login without ALTER ANY EVENT SESSION, is a permission + /// denial for the SQL Server provider, so a collector run that meets it records PERMISSIONS and not ERROR. + /// + [Fact] + public void ClassifiesAnEventSessionPermissionDenialOnSqlServerAsPermissions() + { + var denied = LongQueryTraceReadOnlyIntentTests.SqlExceptionFactory.Create(15247, 14, "User does not have permission to perform this action."); + + Assert.Equal(CollectorTargetFault.Permissions, SqlServerTargetProvider.Instance.Classify(denied, false)); + } + [Fact] public void ClassifiesUnrecognizedExceptionsAsUnclassifiedSoTheyStayLoud() { @@ -288,6 +301,32 @@ even though nobody thought to assert against that particular name. */ Assert.Contains("survivors ONLY", note, StringComparison.Ordinal); } + /// + /// The runner's Azure database list is the provider's plan with the server's exclusions only when asked for. The + /// long-query trace's drops ask for none, so a session created before a database was excluded is still found there. + /// Both lifecycle rigs replace the listing, so this reads the plan the real listing runs. + /// + [Fact] + public void TheRunnersAzureDatabaseList_AppliesTheServersExclusionsOnlyWhenAsked() + { + var server = new ServerRuntime + { + Config = new MonitoredServer { Name = "excl", Host = "excl.database.windows.net", ExcludedDatabases = ["scratch", "tempdb_clone"] }, + ConnectionString = "Server=tcp:excl.database.windows.net,1433;Initial Catalog=master;Encrypt=True", + Target = new CollectorTargetInfo { IsAzureSqlDb = true }, + StorageName = "excl.database.windows.net", + ServerId = 1, + }; + + var (_, every) = DarlingCollectorRunner.AzureDatabaseListPlan(server, databaseScope: null, applyExclusions: false); + Assert.DoesNotContain("NOT IN", every.Text, StringComparison.Ordinal); + Assert.Empty(every.Parameters); + + var (_, monitored) = DarlingCollectorRunner.AzureDatabaseListPlan(server, databaseScope: null, applyExclusions: true); + Assert.Contains("name NOT IN (@excl_db_0, @excl_db_1)", monitored.Text, StringComparison.Ordinal); + Assert.Equal(2, monitored.Parameters.Count); + } + /// /// The exclusion list has to reach the right column name on each engine, parameterized rather than /// interpolated — a database named with a quote must not be able to alter the enumeration query. diff --git a/Darling/Darling.Tests/TargetSessionIdCacheTests.cs b/Darling/Darling.Tests/TargetSessionIdCacheTests.cs new file mode 100644 index 0000000000..6dec68e555 --- /dev/null +++ b/Darling/Darling.Tests/TargetSessionIdCacheTests.cs @@ -0,0 +1,238 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + + +using System; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Data.SqlClient; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// The per-physical-connection session id cache (#5132): the unit halves run anywhere, the live halves need a +/// real SQL Server (DARLING_TEST_SQL, optional _USER / _PASSWORD) and a store (DARLING_TEST_PG) and skip +/// when those are unset, as CI does. +/// +public sealed class TargetSessionIdCacheTests +{ + [Fact] + public async Task TheSameConnectionIdIsLookedUpOnce() + { + var cache = new TargetSessionIdCache(8); + var id = Guid.NewGuid(); + var calls = 0; + Task Lookup(CancellationToken _) { calls++; return Task.FromResult(57); } + + Assert.Equal(57, (await cache.ResolveAsync(id, Lookup, null, TestContext.Current.CancellationToken)).Spid); + Assert.Equal(57, (await cache.ResolveAsync(id, Lookup, null, TestContext.Current.CancellationToken)).Spid); + Assert.Equal(1, calls); + Assert.Equal(1, cache.LookupCount); + } + + [Fact] + public async Task ADifferentConnectionIdIsANewLookup() + { + var cache = new TargetSessionIdCache(8); + var calls = 0; + Task Lookup(CancellationToken _) { calls++; return Task.FromResult(50 + calls); } + + Assert.Equal(51, (await cache.ResolveAsync(Guid.NewGuid(), Lookup, null, TestContext.Current.CancellationToken)).Spid); + Assert.Equal(52, (await cache.ResolveAsync(Guid.NewGuid(), Lookup, null, TestContext.Current.CancellationToken)).Spid); + Assert.Equal(2, calls); + } + + [Fact] + public async Task TheBoundIsHonoured_AndTheOldestEntryIsEvicted() + { + var cache = new TargetSessionIdCache(3); + var ids = new[] { Guid.NewGuid(), Guid.NewGuid(), Guid.NewGuid(), Guid.NewGuid() }; + var calls = 0; + Task Lookup(CancellationToken _) { calls++; return Task.FromResult(60 + calls); } + + foreach (var id in ids) + { + await cache.ResolveAsync(id, Lookup, null, TestContext.Current.CancellationToken); + } + + Assert.Equal(3, cache.Count); + Assert.Equal(4, calls); + + /* The newest three still hit; the first was evicted and asks again. */ + await cache.ResolveAsync(ids[3], Lookup, null, TestContext.Current.CancellationToken); + Assert.Equal(4, calls); + await cache.ResolveAsync(ids[0], Lookup, null, TestContext.Current.CancellationToken); + Assert.Equal(5, calls); + Assert.Equal(3, cache.Count); + Assert.Equal(4096, TargetSessionIdCache.Capacity); + } + + [Fact] + public async Task AFailedLookupIsNullAndNotCached() + { + var cache = new TargetSessionIdCache(8); + var id = Guid.NewGuid(); + + Assert.Null((await cache.ResolveAsync(id, _ => throw new InvalidOperationException("boom"), null, TestContext.Current.CancellationToken)).Spid); + Assert.Equal(0, cache.Count); + + /* A non-positive answer is not a session id either. */ + Assert.Null((await cache.ResolveAsync(id, _ => Task.FromResult(0), null, TestContext.Current.CancellationToken)).Spid); + Assert.Equal(0, cache.Count); + + /* The next run tries again and succeeds. */ + Assert.Equal(44, (await cache.ResolveAsync(id, _ => Task.FromResult(44), null, TestContext.Current.CancellationToken)).Spid); + Assert.Equal(3, cache.LookupCount); + } + + [Fact] + public async Task AnEmptyConnectionIdIsNeverLookedUp() + { + var cache = new TargetSessionIdCache(8); + Assert.Null((await cache.ResolveAsync(Guid.Empty, _ => Task.FromResult(9), null, TestContext.Current.CancellationToken)).Spid); + Assert.Equal(0, cache.LookupCount); + } + + [Fact] + public async Task AClosedConnection_HasAnEmptyConnectionId_SoTheLookupIsSkipped() + { + var cache = new TargetSessionIdCache(8); + using var connection = new SqlConnection("Server=127.0.0.1,1;Connect Timeout=1;Encrypt=false"); + Assert.Null((await cache.ResolveAsync(connection, null, TestContext.Current.CancellationToken)).Spid); + Assert.Equal(0, cache.Count); + Assert.Equal(0, cache.LookupCount); + } + + [Fact] + public async Task ARealCancellationPropagates_AndTheCacheIsUnchanged() + { + var cache = new TargetSessionIdCache(8); + using var cts = new CancellationTokenSource(); + await cts.CancelAsync(); + await Assert.ThrowsAnyAsync(async () => + await cache.ResolveAsync(Guid.NewGuid(), ct => { ct.ThrowIfCancellationRequested(); return Task.FromResult(5); }, null, cts.Token)); + Assert.Equal(0, cache.Count); + } + + [Fact] + public async Task ALookupTimeout_WithoutACancelledToken_IsStillNullNotAThrow() + { + var cache = new TargetSessionIdCache(8); + var result = await cache.ResolveAsync(Guid.NewGuid(), _ => throw new OperationCanceledException(), null, TestContext.Current.CancellationToken); + Assert.Null(result.Spid); + Assert.Equal(0, cache.Count); + } + + [Fact] + public async Task TheAnswerIsFiledUnderTheConnectionIdReadAfterTheLookup() + { + var cache = new TargetSessionIdCache(8); + var before = Guid.NewGuid(); + var after = Guid.NewGuid(); + + var first = await cache.ResolveAsync(before, _ => Task.FromResult(61), null, TestContext.Current.CancellationToken, () => after); + Assert.Equal(61, first.Spid); + Assert.Equal(after, first.Key); + Assert.Equal(1, cache.Count); + + /* The old id has no entry (a lookup is issued again); the new id hits. */ + var calls = 0; + Task Lookup(CancellationToken _) { calls++; return Task.FromResult(99); } + Assert.Equal(61, (await cache.ResolveAsync(after, Lookup, null, TestContext.Current.CancellationToken)).Spid); + Assert.Equal(0, calls); + Assert.Equal(99, (await cache.ResolveAsync(before, Lookup, null, TestContext.Current.CancellationToken)).Spid); + Assert.Equal(1, calls); + } + + [Fact] + public async Task AnEvictedConnectionIdAsksAgain_AndAReplacedConnectionIsDiscarded() + { + var cache = new TargetSessionIdCache(2); + var id = Guid.NewGuid(); + var other = Guid.NewGuid(); + var calls = 0; + Task Lookup(CancellationToken _) { calls++; return Task.FromResult(70 + calls); } + + await cache.ResolveAsync(id, Lookup, null, TestContext.Current.CancellationToken); + await cache.ResolveAsync(other, Lookup, null, TestContext.Current.CancellationToken); + + Assert.False(cache.DiscardIfReplaced(id, id)); + Assert.False(cache.DiscardIfReplaced(Guid.Empty, id)); + Assert.Equal(2, cache.Count); + + Assert.True(cache.DiscardIfReplaced(id, Guid.NewGuid())); + Assert.Equal(1, cache.Count); + Assert.Equal(73, (await cache.ResolveAsync(id, Lookup, null, TestContext.Current.CancellationToken)).Spid); + Assert.Equal(3, calls); + + /* The queue stays consistent with the dictionary: the bound still holds after an eviction. */ + await cache.ResolveAsync(Guid.NewGuid(), Lookup, null, TestContext.Current.CancellationToken); + Assert.Equal(2, cache.Count); + } + + internal static MonitoredServer LiveServer(string host) + { + var user = Environment.GetEnvironmentVariable("DARLING_TEST_SQL_USER"); + return new MonitoredServer + { + Name = "darling-spid-e2e", + Host = host, + Auth = string.IsNullOrEmpty(user) ? "integrated" : "sql", + Username = user, + Password = Environment.GetEnvironmentVariable("DARLING_TEST_SQL_PASSWORD"), + TrustServerCertificate = true, + }; + } + + [Fact] + public async Task Live_TheProductConnection_ReturnsTheTrueSpid_AndAPooledReopenIssuesNoSecondLookup() + { + var host = Environment.GetEnvironmentVariable("DARLING_TEST_SQL"); + Assert.SkipWhen(string.IsNullOrEmpty(host), "Set DARLING_TEST_SQL to run the live session id check."); + var ct = TestContext.Current.CancellationToken; + + var connectionString = new SqlConnectionStringBuilder(MonitoredServerConnection.BuildConnectionString(LiveServer(host!), Environment.GetEnvironmentVariable("DARLING_TEST_SQL_PASSWORD"))) + { + Pooling = true, + MaxPoolSize = 1, + }.ConnectionString; + Assert.True(new SqlConnectionStringBuilder(connectionString).MultipleActiveResultSets, "the product string is MARS"); + + var cache = new TargetSessionIdCache(8); + int first; + Guid firstId; + await using (var one = new SqlConnection(connectionString)) + { + await one.OpenAsync(ct); + Assert.Equal(0, one.ServerProcessId); + first = ((await cache.ResolveAsync(one, null, ct)).Spid).GetValueOrDefault(); + firstId = one.ClientConnectionId; + + using var truth = new SqlCommand("SELECT @@SPID;", one); + Assert.Equal((int)(short)first, Convert.ToInt32(await truth.ExecuteScalarAsync(ct))); + } + + Assert.True(first > 0); + Assert.Equal(1, cache.LookupCount); + + await using (var two = new SqlConnection(connectionString)) + { + await two.OpenAsync(ct); + Assert.Equal(firstId, two.ClientConnectionId); + Assert.Equal(first, (await cache.ResolveAsync(two, null, ct)).Spid); + } + + Assert.Equal(1, cache.LookupCount); + SqlConnection.ClearAllPools(); + } +} diff --git a/Darling/Darling.Tests/TargetSessionIdLiveStoreTests.cs b/Darling/Darling.Tests/TargetSessionIdLiveStoreTests.cs new file mode 100644 index 0000000000..30399d861d --- /dev/null +++ b/Darling/Darling.Tests/TargetSessionIdLiveStoreTests.cs @@ -0,0 +1,75 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: this fact mints its own scratch database through ScratchPostgres and never touches another + test's rows, so it is deliberately NOT [Collection("live-postgres")]. */ + +/// +/// The live end-to-end half of the target session id work (#5132): a real collector run against a real SQL +/// Server records a positive target_session_id in a scratch store. Skips without DARLING_TEST_SQL and +/// DARLING_TEST_PG, as CI does. +/// +public sealed class TargetSessionIdLiveStoreTests +{ + [Fact] + public async Task Live_AFullCollectorRun_RecordsANonNullTargetSessionId_InTheStore() + { + var pg = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + var host = Environment.GetEnvironmentVariable("DARLING_TEST_SQL"); + Assert.SkipWhen(string.IsNullOrEmpty(pg) || string.IsNullOrEmpty(host), + "Set DARLING_TEST_PG and DARLING_TEST_SQL to run the live session id E2E."); + var ct = TestContext.Current.CancellationToken; + + var scratch = await ScratchPostgres.CreateAsync(pg!, ct); + var bodySucceeded = false; + try + { + await using var dataSource = NpgsqlDataSource.Create(scratch.ConnectionString); + await using (var migrate = await dataSource.OpenConnectionAsync(ct)) + { + await PgMigrations.MigrateAsync(migrate, ct); + } + + var runtime = await DarlingServerConnector.ConnectAsync(TargetSessionIdCacheTests.LiveServer(host!), null, ct); + var runner = new DarlingCollectorRunner(dataSource, new CollectorDeltaCalculator()); + var result = await runner.RunAsync(WaitStatsCollector.Instance, runtime, ct); + + Assert.NotNull(result.Drain); + Assert.True(result.Drain!.Value.TargetSessionId > 0, "the run must carry the target's session id"); + + await DarlingObservability.LogCollectionAsync( + dataSource, runtime, "wait_stats", "SUCCESS", result.Rows, result.SqlMs, result.StorageMs, null, + fanout: null, phases: null, drain: result.Drain, fetchPhases: null, sweepPeerMaxMs: null, null, ct); + + await using var verify = await dataSource.OpenConnectionAsync(ct); + using var read = new NpgsqlCommand( + "SELECT target_session_id FROM collection_log WHERE server_id = $1 AND collector_name = 'wait_stats' ORDER BY collection_time DESC LIMIT 1", verify); + read.Parameters.AddWithValue(runtime.ServerId); + var stored = await read.ExecuteScalarAsync(ct); + Assert.IsType(stored); + Assert.True((int)stored! > 0); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => await Task.CompletedTask); + await scratch.DisposeAsync(); + } + } +} diff --git a/Darling/Darling.Tests/TempDbReservedLabelProvenanceTests.cs b/Darling/Darling.Tests/TempDbReservedLabelProvenanceTests.cs index 9dcb4f7746..1224217145 100644 --- a/Darling/Darling.Tests/TempDbReservedLabelProvenanceTests.cs +++ b/Darling/Darling.Tests/TempDbReservedLabelProvenanceTests.cs @@ -61,6 +61,8 @@ namespace Darling.Tests; /// And a mislabel on some OTHER tempdb number (a field naming reserved MB as "used") is outside the /// property: this pin is about the percentage. /// +[Trait("Stage", "Guard")] +[Trait("Reads", "Lite")] public class TempDbReservedLabelProvenanceTests { /// diff --git a/Darling/Darling.Tests/TestData/password-seal-v1-vector.json b/Darling/Darling.Tests/TestData/password-seal-v1-vector.json new file mode 100644 index 0000000000..99bc32d3c7 --- /dev/null +++ b/Darling/Darling.Tests/TestData/password-seal-v1-vector.json @@ -0,0 +1,22 @@ +{ + "note": "test-only key, used nowhere else", + "algorithm": "RSA3072-OAEP-SHA256/A256GCM", + "keyId": "27eff058f1387123", + "publicKeySpki": "MIIBojANBgkqhkiG9w0BAQEFAAOCAY8AMIIBigKCAYEA39XnMrJShL4h3yhffHxE7P2t1TGmcdn1dbXddZZqCcsZIiKvG5\u002BtoxACYtIkG0LMtlOM0sw2caqtJ2l51tvXE9cTh9801z//vYXQSVfwc3bd0g2iG\u002BoUrtbR/lYJN23lwMw\u002B3cnk54mNR10Drg5O01eKHUWbptY9rv6zZNdLxCZDll0G/4/ur9CQoNhxpK88RVIjy9oYiBlxLrZ2\u002Bj2Ld4cLy\u002BtWCHiK4JvPjEVHGuOASJlMrx6l5sDeTKRNLkVrym8qQBgU92rIWtyOjxkKvxrnTVONvQdsTonrqMQPguy4BDV0cNJsZDAUQP0Se5lMdci4cSw1nQBELgqokjhxzw85140IZimZZ8QV88lTZIUEH4Hg3cu8/Vj7z0uHtz4ynImzcowk4osZU6rUNzONcAmpYBsQKoSKlarhYbuCqwTZ1uBQky9y/nJQa4sYiw5bWiB91UbEXYmSRZ\u002Bf1XIN1dw9274K84C7Z7MqJofB6XLG1ItWIu2eN8ltVZdhtcxNAgMBAAE=", + "privateKeyPkcs8": "MIIG/QIBADANBgkqhkiG9w0BAQEFAASCBucwggbjAgEAAoIBgQDf1ecyslKEviHfKF98fETs/a3VMaZx2fV1td11lmoJyxkiIq8bn62jEAJi0iQbQsy2U4zSzDZxqq0naXnW29cT1xOH3zTXP/\u002B9hdBJV/Bzdt3SDaIb6hSu1tH\u002BVgk3beXAzD7dyeTniY1HXQOuDk7TV4odRZum1j2u/rNk10vEJkOWXQb/j\u002B6v0JCg2HGkrzxFUiPL2hiIGXEutnb6PYt3hwvL61YIeIrgm8\u002BMRUca44BImUyvHqXmwN5MpE0uRWvKbypAGBT3asha3I6PGQq/GudNU429B2xOieuoxA\u002BC7LgENXRw0mxkMBRA/RJ7mUx1yLhxLDWdAEQuCqiSOHHPDznXjQhmKZlnxBXzyVNkhQQfgeDdy7z9WPvPS4e3PjKcibNyjCTiixlTqtQ3M41wCalgGxAqhIqVquFhu4KrBNnW4FCTL3L\u002BclBrixiLDltaIH3VRsRdiZJFn5/Vcg3V3D3bvgrzgLtnsyomh8HpcsbUi1Yi7Z43yW1Vl2G1zE0CAwEAAQKCAYAVUag9meJmJj3PpWbC0os2D0tZPOtfeKT6WyIGqxbaa\u002B\u002Bu4qLO/LxrBYFUrDMoLYqV6wtEoJKm7NylHlSudC3eA\u002BQmUsoTnlLbWD1oXunV9YJXGVlRU1A4/tjtYoBqORti65EpSqt0qhM8ss1A7LfxRSswcorKrykc28F2X1MmEYbqcV3TRYA0WqcLytG4H8Cb0aD7L4CapeGz00ktuGLqNYCYIYgGMoT3s59m\u002Bso082oqf/9fxomCmugRYKzBNXdLIY1/n46Q1JTHn4aEmej8Lt7i5Jsrh0fZ\u002BCxsavQROEFXr68L3MUv5vIYKRQf4Qha6iedHGOHySe9hdjkagdRxgpMZKjcI11lEClKuEoKC6HVWLO3MVJYq4ZQqK4Waht0sxDzs5LfvAABpSuYtlrCpulue5wnlRr2HqGsZQpq6A9Vq/0U2KjExuHPW2FIzGpZNOU3ZoEVCWNfXsVxOe/mwVDRwejaSybGUOLGuNYhcacrTCc8\u002Boq9qSZeql8wmfkCgcEA5OwciBsBSo8803o6otCveJF8LMgJf0/xbTcP8UsVg8QYFu3FwOYvH2hSypfbhpaxe2oz2RAelD4GjcwHWSIT03\u002B4ATaAuACx3O5UTnsXfdZIPpUxOYFLELWTwrdI9QPeYkCFUm//ezIbFjT1aTwC4mARqFGvVT0hao5JIwmSpe0Fnqd73FdWIDDj2REMCQy0gYfVNMtqBVgh0AXRzdZaAid7eWV\u002B2iXJd/YokfTofs6VaZjzsGTxbWellquAzhunAoHBAPpPwxwZCiT6KAYupXNcOTs5J3dZJpAiI07odKtGweJvzKIFYEo4JsQo35oxMvsIVmhDBJVSHj7I/HhlvRnMTzntA0FCYeqFXaQI7Os1RGsAmY1RsoRXLYQPzZarGXl2tGjvyYgDmNdboOkCKga68kYRbs/w\u002BF3Mg9u6a1XL/PtlAS4pIhyQ2U1\u002B6zV\u002BXrCG5TNY026KQvx4SsdfsqIrMy7zMqHipv6ybALyMhK\u002Bc5yvfMJ\u002ByhSf0UiKdqFxqs\u002BG6wKBwQDFcXQmXAxgD2zqBXk/ZniBqBGO2jV\u002BziewCBbDCfyTenVAiXHPr3kjQKpvvEyGJ8EKbg4sAIAvYdPrRvdrhD3PZFADE8wvZ8IAiLrEOdyIF2KWP72GGKLy1XmNLuJJGT4cRgP3l0umKRA9tg2Y5z6hmzR/lXyBK0mACxq/kokefXEkZtYotFc/Souqc86Km5\u002BwPMTCVVOrKll3fx12yJM3WnBfww\u002BG/aJnyjMxuCcy/\u002BDUhCxijp3Au\u002B2RJ3jfy\u002B8CgcBhYhC9aLx6oFxb81/pHrMhaF4DX8rR/oYQF/3z\u002BMZLq0VIr6DDn07tuB0uJv/3jEnZrGEnGtirgpQ\u002B3w4P8F3OYniGtiXUEOjafaJ267Vfj88tnOA//o61t1Q7zvtNNzJ/NPMKGKQo/PwASxhAoBDL09Jx0gU\u002BsOeH3\u002BIMch1NTQGFjPk5dd0k\u002BkDWAgEi2ohnTqKLef0PrlkhcgjUcMwr8u/67Wwc4RuGleWHR1NBXUDDhoJmPbs1ZrE6pDdZH9sCgcAMYBhFqQcHJRrV5f\u002BQxFYD\u002Bk\u002BGlIKyQxMWF2MHpd996ISqEctrmuqfTFRAGfMIokXfd0rMgECUtrw8I2NXXf4KFMi83ZzY1SOCnSBbDRtf1jY6BcB0TDMR3H4R9bvcPPWxN7CEKFNoqmi36ICmGxL9RHhkI/NcNbPwqJcSvXAU25as3mqS5jq6IPiwbtH6t46xqWRozitEE3GyvIBJ5UNCZfvUwXv1Bkhw\u002BsqjpNzJCSvzBz75uwdqbPwk\u002BCe/bZo=", + "binding": { + "purpose": "server", + "host": "example-sql-01", + "port": 1433, + "engine": "sqlserver", + "database": "example_db", + "readOnlyIntent": false, + "auth": "sql", + "username": "example_login", + "encryptMode": "Mandatory", + "trustServerCertificate": false, + "multiSubnetFailover": false + }, + "plaintext": "p@ss-not-real", + "sealed": "sealed:v1:27eff058f1387123:pmT7uFFA1leKrM2F8g52qqW1E4nX7\u002BUhg4eqMnIkASD/5PCkURyx4MAQrODiWtz8WdqyO3ZK3/Qo2fv0Hh7ad2ZXTe4wX49YIHnYYBLyMKrqt0eAs/TCH8bHa0f8pgjs/BrzP3MYT8t0MyNgfVr416xrfAB2Mv/fH0oof9z\u002Bu7rBn/jhXPJ91UB8fjIpQpyYVPemADre9aVj2TQOywDHycpnnZ6itAZ\u002Bajc1y/vmm6oDWuCJmv/lCEptRlrUz4O51DHvh70px6s3APxbq0EhXfw44xPfRu\u002B320M5VWuR2aLdlWBy/tOzJ0SIp7if09RsWXz2iWSlBv6ih0O505eJsHFFi9WgAef\u002B1GEnWi1U5v5z\u002BT41khDqlOq4FW/a43ADjsVLUBQMRDR\u002BnBWeEE4suwOcrvbyhNa1DKexrS2N0rK4z1\u002BSE7IwYyw3PLQFj5dtKfLd6zgk6TCmxyrW2FEhXF29sKq9Q1nivrsTP7vwriQ\u002B2Z3LSX6X/n8g9GpZpmK8929pzoOEgbbchiX9SswmI95xkov\u002BxW09VtunwpWlvLLQ28m5LOGkCHw/9\u002BtMlAwA0TrIth5nE5g691j5" +} diff --git a/Darling/Darling.Tests/TestKeyRings.cs b/Darling/Darling.Tests/TestKeyRings.cs new file mode 100644 index 0000000000..12cc020813 --- /dev/null +++ b/Darling/Darling.Tests/TestKeyRings.cs @@ -0,0 +1,23 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; + +namespace Darling.Tests; + +/// The two password key rings the write-path tests ask for (#5366): one that seals and opens with a key made for +/// the test run, and one that refuses with the not-ready reason. Tests pass a ring; none sets the service's own. +internal static class TestKeyRings +{ + /// A ring over a key generated once for the test run. + public static readonly IPasswordKeyRing Healthy = DarlingPasswordKey.FromPrivateKey(PasswordPrivateKey.Generate()); + + /// A ring that cannot seal, with . + public static readonly IPasswordKeyRing NotReady = DarlingPasswordKey.Refusing(DarlingPasswordKey.NotReadyReason); +} diff --git a/Darling/Darling.Tests/TestsNoSilentPassGuardTests.cs b/Darling/Darling.Tests/TestsNoSilentPassGuardTests.cs new file mode 100644 index 0000000000..efcf13358b --- /dev/null +++ b/Darling/Darling.Tests/TestsNoSilentPassGuardTests.cs @@ -0,0 +1,126 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text.RegularExpressions; +using Xunit; + +namespace Darling.Tests; + +/// +/// No [Fact] or [Theory] in either test project may return early from a catch block. +/// +/// A test that does try { setup(); } catch { return; } is reported as a PASS, not a skip. On any +/// machine where the setup cannot run it asserts nothing and still reads as coverage: a Windows run of a +/// symlink test reported "0 skipped" for exactly this reason, when creating a symbolic link needs a privilege +/// an ordinary session lacks. The honest forms are Assert.Skip(reason) (or Assert.SkipUnless) +/// when the setup can legitimately be impossible, and letting the exception fail the test when it should +/// always work. +/// +/// The scan runs over output, so prose and +/// literals cannot match. A return; in a catch of a helper that is not itself a test (a listener +/// shutdown loop, say) is not flagged. There is no exemption list: none was needed. +/// +[Trait("Stage", "Guard")] +[Trait("Reads", "Lite")] +public sealed class TestsNoSilentPassGuardTests +{ + private static readonly Regex TestAttribute = new(@"\[\s*(?:Fact|Theory)\b", RegexOptions.Compiled); + private static readonly Regex CatchKeyword = new(@"\bcatch\b", RegexOptions.Compiled); + private static readonly Regex BareReturn = new(@"\breturn\s*;", RegexOptions.Compiled); + + /// The line (1-based) of every bare return; inside a catch block of a test method. + internal static List SilentPassLines(string text) + { + var code = CSharpSourceWalker.StripCommentsAndStrings(text); + var lines = new List(); + + foreach (Match attribute in TestAttribute.Matches(code)) + { + var brace = code.IndexOf('{', attribute.Index); + var arrow = code.IndexOf("=>", attribute.Index, StringComparison.Ordinal); + if (brace < 0 || (arrow >= 0 && arrow < brace)) + { + continue; + } + + var body = CSharpSourceWalker.BraceBalanced(code, brace); + + foreach (Match handler in CatchKeyword.Matches(body)) + { + var open = body.IndexOf('{', handler.Index); + if (open < 0) + { + continue; + } + + var block = CSharpSourceWalker.BraceBalanced(body, open); + foreach (Match ret in BareReturn.Matches(block)) + { + var offset = brace + open + ret.Index; + lines.Add(code.Take(offset).Count(c => c == '\n') + 1); + } + } + } + + return lines.Distinct().ToList(); + } + + [Fact] + public void NoTestInEitherProject_ReturnsEarlyFromACatchBlock() + { + var offenders = new List(); + var scanned = 0; + + foreach (var project in new[] { "Darling/Darling.Tests", "Lite.Tests" }) + { + var root = RepoFile.PathTo(project); + Assert.True(Directory.Exists(root), $"test project folder not found: {root}"); + + foreach (var file in Directory.EnumerateFiles(root, "*.cs", SearchOption.AllDirectories)) + { + var relative = Path.GetRelativePath(RepoFile.Root, file); + var parts = relative.Split(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar); + if (parts.Contains("bin") || parts.Contains("obj")) + { + continue; + } + + scanned++; + foreach (var line in SilentPassLines(File.ReadAllText(file))) + { + offenders.Add($"{relative.Replace('\\', '/')}:{line}"); + } + } + } + + Assert.True(scanned > 100, $"the scan found only {scanned} files, so it is reading the wrong place"); + Assert.True( + offenders.Count == 0, + "A test that returns from a catch block is reported as a PASS, not a skip, so on a machine where its " + + "setup cannot run it asserts nothing and still reads as coverage. Replace the return with " + + "Assert.Skip(reason) (or Assert.SkipUnless) when the setup can legitimately be impossible, or let " + + "the exception fail the test. Offenders: " + string.Join(", ", offenders)); + } + + [Fact] + public void TheScan_FlagsABareReturnInACatch_AndNothingElse() + { + const string flagged = "public class T\n{\n [Fact]\n public void A()\n {\n try { X(); }\n catch (Exception)\n {\n return;\n }\n }\n}\n"; + Assert.Equal(new[] { 9 }, SilentPassLines(flagged)); + + const string notATest = "public class T\n{\n private static void Helper()\n {\n try { X(); }\n catch (Exception) { return; }\n }\n}\n"; + Assert.Empty(SilentPassLines(notATest)); + + const string explicitSkip = "public class T\n{\n [Fact]\n public void A()\n {\n try { X(); }\n catch (Exception ex) { Assert.Skip(ex.Message); }\n return;\n }\n}\n"; + Assert.Empty(SilentPassLines(explicitSkip)); + } +} diff --git a/Darling/Darling.Tests/TimeHonestyRungTests.cs b/Darling/Darling.Tests/TimeHonestyRungTests.cs index 45f35ba55a..1acefd785d 100644 --- a/Darling/Darling.Tests/TimeHonestyRungTests.cs +++ b/Darling/Darling.Tests/TimeHonestyRungTests.cs @@ -58,6 +58,7 @@ namespace Darling.Tests; /// after the writer has stored a twin, and after the post-rung rows are gone again); the dedup is pinned in /// Lite.Tests/CpuUtilizationCollectorDefinitionTests. /// +[Trait("Reads", "Lite")] public sealed class TimeHonestyRungTests { private const int RungVersion = 134; @@ -506,6 +507,7 @@ private static string Dedent(string sql) /// against every other live class because it shares the store. /// [Collection("live-postgres")] +[Trait("Reads", "Lite")] public sealed class TimeHonestyRungLivePostgresTests { private const int ServerId = -134134; diff --git a/Darling/Darling.Tests/TimeRangePickerControlTests.cs b/Darling/Darling.Tests/TimeRangePickerControlTests.cs new file mode 100644 index 0000000000..5666656333 --- /dev/null +++ b/Darling/Darling.Tests/TimeRangePickerControlTests.cs @@ -0,0 +1,199 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Threading; +using System.Windows; +using System.Windows.Media; +using System.Windows.Threading; +using PerformanceMonitor.Ui; +using Xunit; + +/* Darling.Tests only: a Darling file Lite.Tests compiles must be named in build.yml's Lite path filter. */ +namespace Darling.Tests; + +/// #5562: the shared WPF time range picker constructs, renders and raises its event on an STA thread. +public sealed class TimeRangePickerControlTests +{ + private static readonly TimeZoneInfo NewYork = TimeZoneInfo.FindSystemTimeZoneById("America/New_York"); + private static readonly DateTime Now = new(2026, 10, 8, 11, 1, 0, DateTimeKind.Unspecified); + + private static TimeRangePicker Build() + => new() { ZoneProvider = () => NewYork, NowProvider = () => Now }; + + [Fact] + public void Constructs_WithTheShortPresetsAndTheCalendarPeriods() + { + StaTestThread.Run(() => + { + var picker = Build(); + Assert.Equal(9, picker.RollingPanel.Children.Count); + Assert.Equal(8, picker.PeriodPanel.Children.Count); + Assert.Equal("4h", picker.Value.Id); + Assert.Equal("Past 4 hours", picker.ButtonText.Text); + Assert.Equal("Oct 8, 3:01 am - Oct 8, 7:01 am (UTC-04:00)", picker.ResolvedText.Text); + Assert.False(picker.IsPopupOpen); + }); + } + + [Fact] + public void Value_RendersWithoutRaisingTheEvent_AndSelectRaisesIt() + { + StaTestThread.Run(() => + { + var picker = Build(); + var raised = 0; + TimeRangeChangedEventArgs? last = null; + picker.RangeChanged += (_, e) => { raised++; last = e; }; + + picker.Value = TimeRangeSpec.ForPeriod(CalendarPeriod.WeekToDate); + Assert.Equal(0, raised); + Assert.Equal("Week to Date", picker.ButtonText.Text); + Assert.Equal("Oct 5, 12:00 am - Oct 8, 7:01 am (UTC-04:00)", picker.ResolvedText.Text); + + Assert.True(picker.Select(TimeRangePresets.Find("1w")!)); + Assert.Equal(1, raised); + Assert.Equal("1w", last!.Spec.Id); + Assert.Equal(TimeSpan.FromDays(7), last.Range.Span); + Assert.Equal("1w", picker.Value.Id); + }); + } + + [Fact] + public void Select_RefusesARangeUnderTheMinimum_AndHoldsTheOldOne() + { + StaTestThread.Run(() => + { + var picker = Build(); + picker.NowProvider = () => new DateTime(2026, 10, 8, 4, 2, 0, DateTimeKind.Unspecified); + var raised = false; + picker.RangeChanged += (_, _) => raised = true; + + Assert.False(picker.Select(TimeRangeSpec.Relative(TimeSpan.FromMinutes(4)))); + Assert.False(raised); + Assert.Equal("4h", picker.Value.Id); + + /* A calendar period is exempt from the floor: Today at 00:02 is held, never replaced by another range. */ + Assert.True(picker.Select(TimeRangeSpec.ForPeriod(CalendarPeriod.Today))); + Assert.True(raised); + Assert.Equal("today", picker.Value.Id); + }); + } + + [Fact] + public void Resolve_FollowsTheZoneProvider() + { + StaTestThread.Run(() => + { + var picker = Build(); + picker.Value = TimeRangeSpec.ForPeriod(CalendarPeriod.Yesterday); + var eastern = picker.Resolve()!; + picker.ZoneProvider = () => TimeZoneInfo.Utc; + picker.Refresh(); + var utc = picker.Resolve()!; + Assert.NotEqual(eastern.StartUtc, utc.StartUtc); + Assert.Equal("Oct 7, 12:00 am - Oct 8, 12:00 am (UTC+00:00)", picker.ResolvedText.Text); + }); + } + + [Fact] + public void Notes_ShowTheDataStartAndTheCollectorInterval() + { + StaTestThread.Run(() => + { + var picker = Build(); + picker.Value = TimeRangePresets.Find("1w")!; + Assert.Equal(Visibility.Collapsed, picker.NoteText.Visibility); + + picker.DataStartUtc = new DateTime(2026, 10, 3, 18, 0, 0); + Assert.Equal(Visibility.Visible, picker.NoteText.Visibility); + Assert.Equal("Data starts Oct 3, 2:00 pm", picker.NoteText.Text); + + picker.Value = TimeRangePresets.Find("5m")!; + picker.DataStartUtc = null; + picker.SampleInterval = TimeSpan.FromMinutes(5); + Assert.Equal("Data here is collected every 5 minutes.", picker.NoteText.Text); + + picker.SampleInterval = TimeSpan.FromMinutes(1); + Assert.Equal(Visibility.Collapsed, picker.NoteText.Visibility); + }); + } + + [Fact] + public void Compact_HidesTheDetailBesideTheButton_AndKeepsItInTheTooltip() + { + StaTestThread.Run(() => + { + var picker = Build(); + picker.Compact = true; + Assert.Equal(Visibility.Collapsed, picker.DetailPanel.Visibility); + Assert.Contains("(UTC-04:00)", (string)picker.PickerButton.ToolTip, StringComparison.Ordinal); + picker.Compact = false; + Assert.Equal(Visibility.Visible, picker.DetailPanel.Visibility); + }); + } + + [Fact] + public void TypingARange_PreviewsItBeforeApply_AndAnErrorDisablesApply() + { + StaTestThread.Run(() => + { + var picker = Build(); + picker.InputBox.Text = "last month"; + Assert.True(picker.ApplyButton.IsEnabled); + Assert.Equal("30d Sep 1, 12:00 am - Oct 1, 12:00 am (UTC-04:00)", picker.EchoText.Text); + + picker.InputBox.Text = "3m"; + Assert.False(picker.ApplyButton.IsEnabled); + Assert.Contains("5 minutes", picker.EchoText.Text, StringComparison.Ordinal); + }); + } + + [Fact] + public void OpeningThePopup_ThemesTheCalendarFromTheHostBrushes_AndAPickedRangeFillsTheTextBox() + { + StaTestThread.Run(() => + { + var dark = new SolidColorBrush(Color.FromRgb(0x11, 0x12, 0x17)); + var light = new SolidColorBrush(Color.FromRgb(0xE4, 0xE6, 0xEB)); + var picker = Build(); + picker.Resources["BackgroundBrush"] = dark; + picker.Resources["ForegroundBrush"] = light; + var window = new Window { Content = picker, Width = 300, Height = 200, ShowActivated = false, WindowStyle = WindowStyle.None, ShowInTaskbar = false, Left = -4000, Top = -4000 }; + try + { + window.Show(); + picker.PickerPopup.IsOpen = true; + // Let the Opened handler and its Loaded-priority theming pass run. + window.Dispatcher.Invoke(() => { }, DispatcherPriority.ApplicationIdle); + + Assert.Same(dark, picker.PickCalendar.Background); + Assert.Same(light, picker.PickCalendar.Foreground); + Assert.Equal("Examples:", picker.EchoText.Text.Substring(0, 9)); + Assert.NotEmpty(picker.PeriodPanel.Children); + + picker.PickCalendar.SelectedDates.AddRange(new DateTime(2026, 10, 1), new DateTime(2026, 10, 3)); + Assert.Equal("Oct 1 - Oct 3", picker.InputBox.Text); + Assert.True(picker.ApplyButton.IsEnabled); + + TimeRangeChangedEventArgs? picked = null; + picker.RangeChanged += (_, e) => picked = e; + picker.ApplyButton.RaiseEvent(new RoutedEventArgs(System.Windows.Controls.Primitives.ButtonBase.ClickEvent)); + Assert.NotNull(picked); + Assert.Equal(TimeRangeKind.Fixed, picked!.Spec.Kind); + Assert.Equal(new DateTime(2026, 10, 1, 4, 0, 0), picked.Range.StartUtc); + Assert.Equal(new DateTime(2026, 10, 4, 4, 0, 0), picked.Range.EndUtc); + Assert.False(picker.IsPopupOpen); + } + finally + { + window.Close(); + } + }); + } +} diff --git a/Darling/Darling.Tests/TimeRangeTests.cs b/Darling/Darling.Tests/TimeRangeTests.cs new file mode 100644 index 0000000000..eb77bb62ae --- /dev/null +++ b/Darling/Darling.Tests/TimeRangeTests.cs @@ -0,0 +1,332 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Text.Json; +using PerformanceMonitor.Ui; +using Xunit; + +/* Darling.Tests covers the shared picker model (Lite and the Viewer both reference PerformanceMonitor.Ui). + Lite.Tests does not compile this file: a linked Darling file must be named in build.yml's Lite path filter. */ +namespace Darling.Tests; + +/// +/// #5562: the shared time range grammar. Every row of Fixtures/time-range-cases.json must parse to the +/// stated instants, liveness and echo text (or the stated error code); the web module runs the same file. +/// +public sealed class TimeRangeFixtureTests +{ + private static readonly string FixturePath = Path.Combine(AppContext.BaseDirectory, "Fixtures", "time-range-cases.json"); + + public static IEnumerable Cases() + { + using var doc = JsonDocument.Parse(File.ReadAllText(FixturePath)); + foreach (var c in doc.RootElement.GetProperty("cases").EnumerateArray()) + { + yield return new object[] { c.GetProperty("name").GetString()! + " [" + c.GetProperty("input").GetString() + "]" , c.GetRawText() }; + } + } + + [Fact] + public void Fixture_UsesZoneIdsThisRuntimeResolves() + { + using var doc = JsonDocument.Parse(File.ReadAllText(FixturePath)); + var zones = doc.RootElement.GetProperty("cases").EnumerateArray().Select(c => c.GetProperty("zone").GetString()!).Distinct().ToList(); + Assert.NotEmpty(zones); + foreach (var id in zones) + { + Assert.NotNull(TimeZoneInfo.FindSystemTimeZoneById(id)); + } + } + + [Theory] + [MemberData(nameof(Cases))] + public void Parse_MatchesTheSharedFixture(string label, string caseJson) + { + Assert.False(string.IsNullOrEmpty(label)); + using var doc = JsonDocument.Parse(caseJson); + var c = doc.RootElement; + var input = c.GetProperty("input").GetString(); + var now = ParseZ(c.GetProperty("now").GetString()!); + var zone = TimeZoneInfo.FindSystemTimeZoneById(c.GetProperty("zone").GetString()!); + + var result = TimeRangeParser.Parse(input, now, zone); + + var expectedError = c.GetProperty("error"); + if (expectedError.ValueKind == JsonValueKind.String) + { + Assert.False(result.Ok, "expected an error but got " + result.Echo); + Assert.Equal(expectedError.GetString(), result.ErrorCode); + Assert.False(string.IsNullOrWhiteSpace(result.Error)); + if (c.TryGetProperty("errorMessage", out var message)) + { + // A row that states its words holds both front ends to them (the midnight refusal of a calendar period). + Assert.Equal(message.GetString(), result.Error); + } + + Assert.Equal(string.Empty, result.Echo); + return; + } + + Assert.True(result.Ok, result.ErrorCode + ": " + result.Error); + Assert.Equal(c.GetProperty("start").GetString(), Z(result.Range!.StartUtc)); + Assert.Equal(c.GetProperty("end").GetString(), Z(result.Range!.EndUtc)); + Assert.Equal(c.GetProperty("live").GetBoolean(), result.Range.IsLive); + Assert.Equal(c.GetProperty("echo").GetString(), result.Echo); + } + + internal static DateTime ParseZ(string iso) + => DateTime.SpecifyKind(DateTime.ParseExact(iso, "yyyy-MM-dd'T'HH:mm:ss'Z'", CultureInfo.InvariantCulture, DateTimeStyles.None), DateTimeKind.Unspecified); + + internal static string Z(DateTime naiveUtc) + => naiveUtc.ToString("yyyy-MM-dd'T'HH:mm:ss'Z'", CultureInfo.InvariantCulture); +} + +/// #5562: the preset lists, calendar periods, ids, legacy hour mapping and the notes beside the picker. +public sealed class TimeRangeModelTests +{ + private static readonly TimeZoneInfo NewYork = TimeZoneInfo.FindSystemTimeZoneById("America/New_York"); + private static readonly DateTime Now = new(2026, 10, 8, 11, 1, 0, DateTimeKind.Unspecified); + + [Fact] + public void RollingPresets_AreTheNineShortOnesInOrder() + { + Assert.Equal(new[] { "5m", "15m", "30m", "1h", "4h", "1d", "2d", "1w", "1mo" }, TimeRangePresets.Rolling.Select(p => p.Id)); + } + + [Fact] + public void CalendarPeriods_AreTheEightInOrder() + { + Assert.Equal( + new[] { "Today", "Yesterday", "Week to Date", "Previous Week", "Month to Date", "Previous Month", "Year to Date", "Previous Year" }, + TimeRangePresets.CalendarPeriods.Select(p => p.Name)); + } + + [Fact] + public void CalendarPeriods_ReportTheirCurrentLength() + { + // Thursday 7:01 am in New York: Week to Date is Monday 00:00 to now. + Assert.Equal("3d", TimeRangePresets.CurrentLength(TimeRangeSpec.ForPeriod(CalendarPeriod.WeekToDate), Now, NewYork)); + Assert.Equal("7h 1m", TimeRangePresets.CurrentLength(TimeRangeSpec.ForPeriod(CalendarPeriod.Today), Now, NewYork)); + Assert.Equal("1d", TimeRangePresets.CurrentLength(TimeRangeSpec.ForPeriod(CalendarPeriod.Yesterday), Now, NewYork)); + Assert.Equal("7d", TimeRangePresets.CurrentLength(TimeRangeSpec.ForPeriod(CalendarPeriod.PreviousWeek), Now, NewYork)); + Assert.Equal("30d", TimeRangePresets.CurrentLength(TimeRangeSpec.ForPeriod(CalendarPeriod.PreviousMonth), Now, NewYork)); + Assert.Equal("365d", TimeRangePresets.CurrentLength(TimeRangeSpec.ForPeriod(CalendarPeriod.PreviousYear), Now, NewYork)); + // Just after midnight Today is exempt from the 5-minute floor (seat ruling): it reads 00:00 to now, never another range. + Assert.Equal("2m", TimeRangePresets.CurrentLength(TimeRangeSpec.ForPeriod(CalendarPeriod.Today), new DateTime(2026, 10, 8, 4, 2, 0), NewYork)); + // Exactly at midnight nothing has happened yet, so a zero-length period is still refused. + Assert.Null(TimeRangePresets.CurrentLength(TimeRangeSpec.ForPeriod(CalendarPeriod.Today), new DateTime(2026, 10, 8, 4, 0, 0), NewYork)); + } + + [Theory] + [InlineData("400000mo")] + [InlineData("1000000d")] + [InlineData("999999w")] + [InlineData("1000000h")] + public void AnId_PastTheHundredYearLimit_IsNotAnIdAndNeverThrows(string id) + { + // #5562 review H1: a hand-edited settings value used to overflow TimeSpan and crash startup. + Assert.False(TimeRangeSpec.TryFromId(id, out var spec)); + Assert.Null(spec); + } + + [Fact] + public void AnId_AtTheLimit_StillParses() + { + Assert.True(TimeRangeSpec.TryFromId("36500d", out var spec)); + Assert.Equal(TimeSpan.FromDays(36500), spec!.Span); + Assert.False(TimeRangeSpec.TryFromId("36501d", out _)); + } + + [Theory] + [InlineData(int.MaxValue)] + [InlineData(300_000_000)] + [InlineData(876_001)] + public void LegacyHours_PastTheLimit_AreNull_NotAnOverflow(int hours) + { + Assert.Null(TimeRangePresets.FromLegacyHours(hours)); + } + + [Fact] + public void ACalendarPeriodIsExemptFromTheFloor_ButARelativeOrFixedRangeIsNot() + { + var justAfterMidnight = new DateTime(2026, 10, 8, 4, 2, 0); + Assert.True(TimeRangeSpec.ForPeriod(CalendarPeriod.Today).TryResolve(justAfterMidnight, NewYork, out var today, out _)); + Assert.Equal(new DateTime(2026, 10, 8, 4, 0, 0), today!.StartUtc); + Assert.Equal(justAfterMidnight, today.EndUtc); + Assert.False(TimeRangeSpec.FixedRange(justAfterMidnight.AddMinutes(-2), justAfterMidnight).TryResolve(justAfterMidnight, NewYork, out _, out var error)); + Assert.Equal("too_short", error!.Code); + } + + [Fact] + public void ShortDaysOnTheChangeDays_ReadAs23And25Hours() + { + var spec = TimeRangeSpec.ForPeriod(CalendarPeriod.Yesterday); + Assert.Equal("23h", TimeRangePresets.CurrentLength(spec, new DateTime(2026, 3, 9, 16, 0, 0), NewYork)); + Assert.Equal("1d", TimeRangePresets.CurrentLength(spec, new DateTime(2026, 11, 2, 17, 0, 0), NewYork)); + Assert.True(spec.TryResolve(new DateTime(2026, 11, 2, 17, 0, 0), NewYork, out var longDay, out _)); + Assert.Equal(TimeSpan.FromHours(25), longDay!.Span); + } + + [Fact] + public void ConsecutivePeriods_TileWithNoGap() + { + var now = new DateTime(2026, 3, 9, 16, 0, 0); + Assert.True(TimeRangeSpec.ForPeriod(CalendarPeriod.PreviousWeek).TryResolve(now, NewYork, out var lastWeek, out _)); + Assert.True(TimeRangeSpec.ForPeriod(CalendarPeriod.WeekToDate).TryResolve(now, NewYork, out var thisWeek, out _)); + Assert.Equal(lastWeek!.EndUtc, thisWeek!.StartUtc); + } + + [Theory] + [InlineData(1, "1h")] + [InlineData(4, "4h")] + [InlineData(12, "12h")] + [InlineData(24, "1d")] + [InlineData(168, "1w")] + [InlineData(48, "2d")] + [InlineData(720, "1mo")] + [InlineData(36, "36h")] + [InlineData(2160, "90d")] + public void FromLegacyHours_MapsWholeHours(int hours, string expectedId) + { + var spec = TimeRangePresets.FromLegacyHours(hours); + Assert.NotNull(spec); + Assert.Equal(expectedId, spec!.Id); + Assert.Equal(hours, spec.WholeHours); + Assert.True(spec.TryResolve(Now, NewYork, out var range, out _)); + Assert.Equal(TimeSpan.FromHours(hours), range!.Span); + } + + [Theory] + [InlineData(0)] + [InlineData(-4)] + public void FromLegacyHours_RefusesZeroAndNegative(int hours) + => Assert.Null(TimeRangePresets.FromLegacyHours(hours)); + + [Fact] + public void Ids_RoundTrip() + { + var specs = TimeRangePresets.Rolling.Concat(TimeRangePresets.CalendarPeriods).ToList(); + specs.Add(TimeRangeSpec.Relative(TimeSpan.FromMinutes(90))); + specs.Add(TimeRangeSpec.FixedRange(new DateTime(2026, 10, 1, 4, 0, 0), new DateTime(2026, 10, 3, 4, 0, 0))); + specs.Add(TimeRangeSpec.SinceInstant(new DateTime(2026, 10, 1, 4, 0, 0))); + foreach (var spec in specs) + { + Assert.True(TimeRangeSpec.TryFromId(spec.Id, out var back), spec.Id); + Assert.Equal(spec, back); + } + + Assert.False(TimeRangeSpec.TryFromId("", out _)); + Assert.False(TimeRangeSpec.TryFromId("tomorrow", out _)); + Assert.False(TimeRangeSpec.TryFromId("0h", out _)); + Assert.False(TimeRangeSpec.TryFromId("h", out _)); + Assert.Equal(TimeSpan.FromDays(60), TimeRangeSpec.TryFromId("2mo", out var twoMonths) ? twoMonths!.Span : default); + } + + [Fact] + public void TheMinimumIsFiveMinutes_AndNothingIsWidened() + { + Assert.True(TimeRangeSpec.Relative(TimeSpan.FromMinutes(5)).TryResolve(Now, NewYork, out _, out _)); + Assert.False(TimeRangeSpec.Relative(TimeSpan.FromMinutes(4)).TryResolve(Now, NewYork, out var none, out var error)); + Assert.Null(none); + Assert.Equal("too_short", error!.Code); + Assert.Contains("5 minutes", error.Message, StringComparison.Ordinal); + } + + [Fact] + public void NoUpperCap_AYearsLongRangeResolves() + { + var result = TimeRangeParser.Parse("1000d", Now, NewYork); + Assert.True(result.Ok); + Assert.Equal(TimeSpan.FromDays(1000), result.Range!.Span); + } + + [Fact] + public void TypedAndPresetRanges_AgreeOnTheSameInstants() + { + Assert.Equal(TimeRangeSpec.Relative(TimeSpan.FromDays(7)), TimeRangeParser.Parse("7d", Now, NewYork).Spec); + Assert.Equal(TimeRangePresets.Find("1mo"), TimeRangeParser.Parse("30 days", Now, NewYork).Spec); + Assert.Equal(TimeRangeSpec.ForPeriod(CalendarPeriod.PreviousMonth), TimeRangeParser.Parse("last month", Now, NewYork).Spec); + Assert.Equal(TimeRangeSpec.Relative(TimeSpan.FromDays(30)), TimeRangeParser.Parse("last 30 days", Now, NewYork).Spec); + } + + [Fact] + public void Resolve_FollowsTheZone_ForCalendarPeriodsButNotForFixedRanges() + { + var today = TimeRangeSpec.ForPeriod(CalendarPeriod.Yesterday); + Assert.True(today.TryResolve(Now, NewYork, out var ny, out _)); + Assert.True(today.TryResolve(Now, TimeZoneInfo.Utc, out var utc, out _)); + Assert.NotEqual(ny!.StartUtc, utc!.StartUtc); + + var fixedRange = TimeRangeParser.Parse("Oct 1", Now, NewYork).Spec!; + Assert.True(fixedRange.TryResolve(Now, NewYork, out var a, out _)); + Assert.True(fixedRange.TryResolve(Now, TimeZoneInfo.Utc, out var b, out _)); + Assert.Equal(a!.StartUtc, b!.StartUtc); + Assert.Equal(a.EndUtc, b.EndUtc); + } + + [Fact] + public void LiveRanges_SlideWithNow() + { + var spec = TimeRangeParser.Parse("since 10/1", Now, NewYork).Spec!; + Assert.True(spec.TryResolve(Now, NewYork, out var first, out _)); + Assert.True(spec.TryResolve(Now.AddHours(3), NewYork, out var later, out _)); + Assert.Equal(first!.StartUtc, later!.StartUtc); + Assert.Equal(first.EndUtc.AddHours(3), later.EndUtc); + Assert.True(later.IsLive); + } + + [Fact] + public void Label_ForTheIssueExample() + { + var result = TimeRangeParser.Parse("week to date", new DateTime(2026, 10, 8, 11, 1, 0), NewYork); + Assert.Equal("3d Oct 5, 12:00 am - Oct 8, 7:01 am (UTC-04:00)", result.Echo); + Assert.Equal("Oct 5, 12:00 am", result.Range!.StartText); + Assert.Equal("UTC-04:00", result.Range.ZoneText); + } + + [Fact] + public void DataStartNote_AppearsOnlyWhenTheRangeStartsWellBeforeTheData() + { + Assert.True(TimeRangeSpec.Relative(TimeSpan.FromDays(7)).TryResolve(Now, NewYork, out var week, out _)); + // Data begins Oct 3 2:00 pm EDT = 18:00Z; the week starts Oct 1 7:01 am EDT. + Assert.Equal("Data starts Oct 3, 2:00 pm", TimeRangeNotes.DataStartNote(week!, new DateTime(2026, 10, 3, 18, 0, 0))); + // Starting inside the 90 minute slack of the data's start says nothing. + Assert.Null(TimeRangeNotes.DataStartNote(week!, new DateTime(2026, 10, 1, 12, 0, 0))); + Assert.Null(TimeRangeNotes.DataStartNote(week!, null)); + Assert.Null(TimeRangeNotes.DataStartNote(week!, new DateTime(2026, 9, 1))); + } + + [Theory] + [InlineData(10, 5, "Data here is collected every 5 minutes.")] + [InlineData(14, 5, "Data here is collected every 5 minutes.")] + [InlineData(15, 5, null)] + [InlineData(5, 1, null)] + [InlineData(5, 2, "Data here is collected every 2 minutes.")] + [InlineData(120, 60, "Data here is collected every hour.")] + [InlineData(60, 0, null)] + public void SampleIntervalNote_FiresUnderThreeSamples(int spanMinutes, int intervalMinutes, string? expected) + { + TimeSpan? interval = intervalMinutes == 0 ? null : TimeSpan.FromMinutes(intervalMinutes); + Assert.Equal(expected, TimeRangeNotes.SampleIntervalNote(TimeSpan.FromMinutes(spanMinutes), interval)); + } + + [Fact] + public void FormatLength_FloorsAndDropsHoursPastADay() + { + Assert.Equal("45m", TimeRangeSpec.FormatLength(TimeSpan.FromMinutes(45))); + Assert.Equal("1h 30m", TimeRangeSpec.FormatLength(TimeSpan.FromMinutes(90))); + Assert.Equal("2h", TimeRangeSpec.FormatLength(TimeSpan.FromHours(2))); + Assert.Equal("1d", TimeRangeSpec.FormatLength(TimeSpan.FromHours(47))); + Assert.Equal("3d", TimeRangeSpec.FormatLength(new TimeSpan(3, 7, 1, 0))); + } +} diff --git a/Darling/Darling.Tests/TimescaleAggregateCompressionTests.cs b/Darling/Darling.Tests/TimescaleAggregateCompressionTests.cs index fd6c5e8ee0..f98b4bab3f 100644 --- a/Darling/Darling.Tests/TimescaleAggregateCompressionTests.cs +++ b/Darling/Darling.Tests/TimescaleAggregateCompressionTests.cs @@ -1,4 +1,4 @@ -/* +/* * Copyright (c) 2026 Erik Darling, Darling Data LLC * * This file is part of the SQL Server Performance Monitor. @@ -100,13 +100,15 @@ public void EveryAggregate_IsRegisteredOnce_WithItsTier_ABucketColumn_AndServerI 7 (into FrozenRollupAggregates, off the band entirely); LC then removed the deferral that had held the three interval-honest successor dailies out of AggregateCompressionTargets, so the list is now every member of all three source lists, with nothing subtracted — 6 + 7 + 7. */ - Assert.Equal(6, TimescaleSupport.HourlyAggregates.Length); + /* #5329 appended the two io hourlies, so the hourly list is 8 and the band is 8 + 7 + 7 = 22 of the 23 + hours available. */ + Assert.Equal(8, TimescaleSupport.HourlyAggregates.Length); Assert.Equal(7, TimescaleSupport.DailyAggregates.Length); Assert.Equal(7, TimescaleSupport.BaselineAggregates.Length); Assert.Equal( TimescaleSupport.HourlyAggregates.Length + TimescaleSupport.DailyAggregates.Length + TimescaleSupport.BaselineAggregates.Length, targets.Count); - Assert.Equal(20, targets.Count); + Assert.Equal(22, targets.Count); Assert.Equal(targets.Count, targets.Select(t => t.View).Distinct(StringComparer.Ordinal).Count()); @@ -199,7 +201,9 @@ three interval-honest successors — the legacy trio dropped out when LC froze i tiers (the L1 dedup layer and the interval-grain daily) and the seven baselines: 5 + 2 + 7. Was 17 before LC (8 + 2 + 7, with the legacy trio still counted); three fewer now. Zero here is a filter that matched nothing. */ - Assert.Equal(17 - 3, checkedTiers); + /* #5329: the two io hourlies are leaves with a retention policy and a compression target, so two + more: 14 + 2 = 16. */ + Assert.Equal(17 - 3 + 2, checkedTiers); } /// @@ -236,7 +240,9 @@ immediately before the raw compression band opens — the tiling identity from t sits past the ceiling inside that window, and a ceiling that grew to meet it fails here — 304 s of margin. */ var secondsPastHeaviest = (minute - TimescaleSupport.HeaviestRefreshStartMinute) * 60; - Assert.Equal(1200, secondsPastHeaviest); + /* #5329: (35 - 17) * 60 = 1,080 s past the heaviest start (1,200 s at :15 before the two io hourlies), + 184 s over the 896 s ceiling (was 304 s). */ + Assert.Equal(1080, secondsPastHeaviest); Assert.True( secondsPastHeaviest > TimescaleSupport.HeaviestHourlyRefreshObservedCeilingSeconds, $"the daily band's minute is {secondsPastHeaviest}s past the heaviest refresh's start against a " @@ -604,7 +610,8 @@ that were wide. */ registry's, not a literal chosen to match; this pin catches the day either side of that trade moves without the other. */ Assert.Contains($"{TimescaleSupport.AggregateCompressionTargets.Count}/{TimescaleSupport.AggregateCompressionTargets.Count} materializations chunked at {TimescaleSupport.MaterializationChunkInterval}", firstLog.Joined, StringComparison.Ordinal); - Assert.Equal(20, TimescaleSupport.AggregateCompressionTargets.Count); + /* Twenty-two since #5329 appended the two io hourlies to the twenty of LC. */ + Assert.Equal(22, TimescaleSupport.AggregateCompressionTargets.Count); Assert.Contains($"{wideBefore} changed this start", firstLog.Joined, StringComparison.Ordinal); /* A settled store issues no set_chunk_time_interval at all: the direct call returns zero changes and diff --git a/Darling/Darling.Tests/TimescaleAvailabilityReprobeTests.cs b/Darling/Darling.Tests/TimescaleAvailabilityReprobeTests.cs index c333b76abd..5d63dce5c5 100644 --- a/Darling/Darling.Tests/TimescaleAvailabilityReprobeTests.cs +++ b/Darling/Darling.Tests/TimescaleAvailabilityReprobeTests.cs @@ -54,6 +54,8 @@ public sealed class TimescaleAvailabilityReprobeTests private const string TickGuard = "if (StampIsDue(_nextCompressionCheckUtc, CompressionCheckSpan, DateTime.UtcNow))"; private const string Stamp = "_nextCompressionCheckUtc = TimescaleSupport.NextCompressionCheckUtc(DateTime.UtcNow, s_compressionCheckInterval);"; private const string Latch = "_timescaleAvailable"; + private const string LaunchCall = "TryStartStoreMaintenanceTick(token => RunStoreMaintenanceTickAsync(token)"; + private const string TickSignature = "private async Task RunStoreMaintenanceTickAsync(CancellationToken stoppingToken)"; private const string ReprobeSignature = "private async Task ReprobeTimescaleAvailabilityAsync(CancellationToken cancellationToken)"; /// @@ -84,11 +86,26 @@ public void TheReprobe_IsNotReachableOnlyFromBehindTheLatchItCorrects() /* INSTRUMENT FIRST. The probe is required to be under exactly two conditions of its own - the tick's due time, and the negated latch - so a walk that returned nothing, or that lost the innermost frame, fails here rather than reporting the absence of a positive guard it never looked for. */ - Assert.True( + Assert.False( reprobeGuards.Any(g => g.Contains("_nextCompressionCheckUtc", StringComparison.Ordinal)), - "the guard walk did not find the store-maintenance tick's due-time condition around the re-probe. " - + "Either the probe has been given a cadence of its own - which is a second timer on a tick that " - + $"exists to hold them - or the walk is not reading this block. Guards seen: {Describe(reprobeGuards)}"); + "the re-probe has been given a cadence of its own - which is a second timer on a tick that " + + $"exists to hold them. Guards seen: {Describe(reprobeGuards)}"); + + /* The probe lives in the tick's own method body (#4970); the tick's due-time condition now guards the + single launch call site, and that condition must not test the latch positively either. */ + var tickBody = MethodBody(code, TickSignature); + Assert.False(string.IsNullOrEmpty(tickBody), "could not locate RunStoreMaintenanceTickAsync"); + Assert.Contains(ReprobeCall, tickBody, StringComparison.Ordinal); + Assert.Equal(1, CountOf(code, LaunchCall)); + var launchGuards = EnclosingGuards(code, code.IndexOf(LaunchCall, StringComparison.Ordinal)); + Assert.True( + launchGuards.Any(g => g.Contains("_nextCompressionCheckUtc", StringComparison.Ordinal)), + "the guard walk did not find the tick's due-time condition around the launch call. " + + $"Guards seen: {Describe(launchGuards)}"); + Assert.True( + !launchGuards.Any(TestsTheLatchPositively), + "the hourly tick is launched from behind a guard that requires _timescaleAvailable, so a falsely-degraded " + + $"service could never re-probe (#3815): {Describe(launchGuards)}"); Assert.True( reprobeGuards.Any(g => g.Replace(" ", string.Empty, StringComparison.Ordinal).Contains("!" + Latch, StringComparison.Ordinal)), @@ -146,6 +163,7 @@ public void TheTick_StampsItsDueTimeWhateverTheLatchSays_AndTheProbeRunsAheadOfT would break without breaking anything else. */ Assert.Equal(1, CountOf(code, TickGuard)); Assert.Equal(1, CountOf(code, Stamp)); + Assert.Equal(1, CountOf(code, LaunchCall)); Assert.Equal(1, CountOf(code, CompressionCall)); Assert.Equal(1, CountOf(code, RetentionCall)); /* #3817's store-object convergence is the fourth tenant, and the count is here for the same reason @@ -155,18 +173,30 @@ would break without breaking anything else. */ /* The latch is gone from the tick's OUTER condition; it now sits one level in. */ Assert.DoesNotContain(Latch + " && StampIsDue(_nextCompressionCheckUtc", code, StringComparison.Ordinal); + /* In the loop: due-time guard, then the stamp, then the launch. */ var guardAt = code.IndexOf(TickGuard, StringComparison.Ordinal); var stampAt = code.IndexOf(Stamp, StringComparison.Ordinal); - var reprobeAt = code.IndexOf(ReprobeCall, StringComparison.Ordinal); - var compressionAt = code.IndexOf(CompressionCall, StringComparison.Ordinal); - var retentionAt = code.IndexOf(RetentionCall, StringComparison.Ordinal); - var convergenceAt = code.IndexOf(ConvergenceCall, StringComparison.Ordinal); + var launchAt = code.IndexOf(LaunchCall, StringComparison.Ordinal); + Assert.True( + guardAt > 0 && stampAt > guardAt && launchAt > stampAt, + "the loop runs: due-time guard, stamp, launch of the tick - in that order (got " + + $"{guardAt}, {stampAt}, {launchAt})"); + + /* In the tick's method body: availability re-probe, flag gate, compression read, retention pass, + store-object convergence - in that order. */ + var tickBody = MethodBody(code, TickSignature); + Assert.False(string.IsNullOrEmpty(tickBody), "could not locate RunStoreMaintenanceTickAsync"); + var reprobeAt = tickBody.IndexOf(ReprobeCall, StringComparison.Ordinal); + var flagGateAt = tickBody.IndexOf("if (" + Latch + ")", StringComparison.Ordinal); + var compressionAt = tickBody.IndexOf(CompressionCall, StringComparison.Ordinal); + var retentionAt = tickBody.IndexOf(RetentionCall, StringComparison.Ordinal); + var convergenceAt = tickBody.IndexOf(ConvergenceCall, StringComparison.Ordinal); Assert.True( - guardAt > 0 && stampAt > guardAt && reprobeAt > stampAt && compressionAt > reprobeAt && retentionAt > compressionAt + reprobeAt >= 0 && flagGateAt > reprobeAt && compressionAt > flagGateAt && retentionAt > compressionAt && convergenceAt > retentionAt, - "the tick runs: due-time guard, stamp, availability re-probe, compression read, retention pass, " + "the tick body runs: availability re-probe, flag gate, compression read, retention pass, " + "store-object convergence - in that order (got " - + $"{guardAt}, {stampAt}, {reprobeAt}, {compressionAt}, {retentionAt}, {convergenceAt})"); + + $"{reprobeAt}, {flagGateAt}, {compressionAt}, {retentionAt}, {convergenceAt})"); /* And the stamp itself is not behind the latch, which is the cadence claim proper. The instrument check comes first for the reason it does above: "no guard requires the latch" is satisfied just as @@ -183,13 +213,6 @@ would break without breaking anything else. */ "the due time is stamped forward from inside a guard that requires _timescaleAvailable, so on a " + "store in plain-PostgreSQL mode it never advances and the re-probe fires on every 15-second sweep " + $"pass instead of hourly: {Describe(stampGuards)}"); - - /* The flag gate is still there, between the probe and the two tenants that need it. Searched FROM the - stamp: the start-path block carries the same text thousands of lines earlier. */ - var flagGateAt = code.IndexOf("if (" + Latch + ")", stampAt, StringComparison.Ordinal); - Assert.True(flagGateAt > reprobeAt && flagGateAt < compressionAt, - "the compression read, the retention pass and the store-object convergence must still be behind the " - + "latch, one level in from the tick"); } /// diff --git a/Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs b/Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs index e2973b7306..901472df9b 100644 --- a/Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs +++ b/Darling/Darling.Tests/TimescaleContinuousAggregateTests.cs @@ -1,4 +1,4 @@ -/* +/* * Copyright (c) 2026 Erik Darling, Darling Data LLC * * This file is part of the SQL Server Performance Monitor. @@ -198,10 +198,12 @@ public void HourlyRefreshPolicies_AreStaggeredAcrossTheHour_OnAFixedSchedule() /* Thirteen hourly refreshes since #3653 (A6): the six registered hourly rollups — the three Query Store views and the three interval-honest successors that REPLACED the legacy trio on the phase grid — and the seven baseline aggregates. Twelve of them light; the heaviest is answered by identity. */ - Assert.Equal(6, TimescaleSupport.HourlyAggregates.Length); + /* #5329 appended the two io hourlies: eight registered hourly rollups, fifteen hourly refreshes, fourteen + of them light. */ + Assert.Equal(8, TimescaleSupport.HourlyAggregates.Length); Assert.Equal(7, TimescaleSupport.BaselineAggregates.Length); - Assert.Equal(13, TimescaleSupport.HourlyRefreshPhaseOrder.Count); - Assert.Equal(12, TimescaleSupport.LightHourlyRefreshCount); + Assert.Equal(15, TimescaleSupport.HourlyRefreshPhaseOrder.Count); + Assert.Equal(14, TimescaleSupport.LightHourlyRefreshCount); var phases = TimescaleSupport.HourlyRefreshPhaseOrder .Select(TimescaleSupport.RefreshPhaseMinutesFor) @@ -466,9 +468,11 @@ actually uses. */ collect.query_stats feeds THREE hourly policies since #3653 (A6): the two interval-honest rollups (query-grain and per-database) and the query_stats baseline — the widest contention group on the grid after the legacy trio left the phase grid at LC, every one on its own minute. */ - Assert.Equal(3, sourceOf.Count(kv => string.Equals(kv.Value, "query_stats", StringComparison.Ordinal))); + /* Four since #5329: query_stats_io_hourly joins the two interval-honest rollups and the baseline. */ + Assert.Equal(4, sourceOf.Count(kv => string.Equals(kv.Value, "query_stats", StringComparison.Ordinal))); /* procedure_stats has ONE consumer since A6: only ProcedureStatsIntervalHourlyView (the legacy left the grid). */ - Assert.Equal(1, sourceOf.Count(kv => string.Equals(kv.Value, "procedure_stats", StringComparison.Ordinal))); + /* Two since #5329: procedure_stats_io_hourly sits beside it. */ + Assert.Equal(2, sourceOf.Count(kv => string.Equals(kv.Value, "procedure_stats", StringComparison.Ordinal))); Assert.Equal(2, sourceOf.Count(kv => string.Equals(kv.Value, "query_store_stats", StringComparison.Ordinal))); var views = new HashSet(definitions.Select(a => a.View), StringComparer.Ordinal); @@ -1326,24 +1330,46 @@ an interior repair can move below the seam and read back Covered. */ } /// - /// PIN (#4300): when the stitched slot's fallback fires (no successor bucket above the legacy's last), - /// the probe's upper bound must be now() - HourlyRefreshStartOffset, not a bare now(). - /// A bare now() lets a healthy, upgrading store with an EMPTY successor read every hour back - /// to the legacy's freeze as unprobed and therefore Short, RE-HOLDING the raw purge for rows the - /// successor's own first refresh will reach within - /// anyway. RED on dev: the fallback ends at a bare time_bucket(INTERVAL '1 hour', now()::timestamp)) - /// with no offset subtraction. + /// PIN (#4300, #4981): when the stitched slot's fallback fires (no successor bucket above the legacy's + /// last), the probe's upper bound must be the service clock minus + /// , not a bare now(). A bare now() + /// lets a healthy, upgrading store with an EMPTY successor read every hour back to the legacy's freeze as + /// unprobed and therefore Short, RE-HOLDING the raw purge for rows the successor's own first refresh will + /// reach within the offset anyway. And the clock is BOUND ($1), never now() in the text: + /// now()::timestamp renders in the store session's time zone against buckets that are naive UTC. + /// RED at 9441f6b6c: the fallback ends at + /// time_bucket(INTERVAL '1 hour', now()::timestamp - INTERVAL '1 day'), so the statement carries a + /// session clock and no placeholder. /// [Fact] - public void RetentionArmSafetySql_StitchedSlot_FallbackUpperBoundUsesHourlyRefreshStartOffset() + public void RetentionArmSafetySql_StitchedSlot_FallbackUpperBoundIsTheBoundHorizon_NotTheSessionClock() { var sql = TimescaleSupport.RetentionArmSafetySql( "query_stats", "collection_time", new[] { TimescaleSupport.QueryStatsIntervalHourlyView }); + Assert.Equal("$1", TimescaleSupport.RetentionArmSafetyHorizonPlaceholder); Assert.Contains( - $"time_bucket(INTERVAL '1 hour', now()::timestamp - INTERVAL '{TimescaleSupport.HourlyRefreshStartOffset}')", + $"time_bucket(INTERVAL '1 hour', {TimescaleSupport.RetentionArmSafetyHorizonPlaceholder})) - INTERVAL '1 hour'", sql, StringComparison.Ordinal); - Assert.DoesNotContain("time_bucket(INTERVAL '1 hour', now()::timestamp))", sql, StringComparison.Ordinal); + + /* No session clock anywhere in the statement, in any spelling. */ + Assert.DoesNotContain("now()", sql, StringComparison.OrdinalIgnoreCase); + Assert.DoesNotContain("localtimestamp", sql, StringComparison.OrdinalIgnoreCase); + Assert.DoesNotContain("current_timestamp", sql, StringComparison.OrdinalIgnoreCase); + Assert.DoesNotContain("clock_timestamp", sql, StringComparison.OrdinalIgnoreCase); + + /* What is bound: the UTC clock minus the same offset the SQL used to subtract, as a naive timestamp, so + Npgsql sends timestamp and not timestamptz. */ + var utcNow = new DateTime(2026, 10, 3, 6, 30, 0, DateTimeKind.Utc); + var horizon = TimescaleSupport.RetentionArmSafetyHorizon(utcNow); + Assert.Equal(utcNow - TimescaleSupport.HourlyRefreshStartSpan, horizon); + Assert.Equal(DateTimeKind.Unspecified, horizon.Kind); + + /* A statement with no stitched slot never names the placeholder. */ + var plainSql = TimescaleSupport.RetentionArmSafetySql( + "query_store_stats", "collection_time", + new[] { TimescaleSupport.QueryStoreStatsHourlyView, TimescaleSupport.QueryStoreStatsIntervalHourlyView }); + Assert.DoesNotContain(TimescaleSupport.RetentionArmSafetyHorizonPlaceholder, plainSql, StringComparison.Ordinal); } /// diff --git a/Darling/Darling.Tests/TimescaleSupportTests.cs b/Darling/Darling.Tests/TimescaleSupportTests.cs index 5431eb06f3..4869439721 100644 --- a/Darling/Darling.Tests/TimescaleSupportTests.cs +++ b/Darling/Darling.Tests/TimescaleSupportTests.cs @@ -1,4 +1,4 @@ -/* +/* * Copyright (c) 2026 Erik Darling, Darling Data LLC * * This file is part of the SQL Server Performance Monitor. @@ -142,7 +142,9 @@ other pin in this file still expects. CompressAfterFor is the single source both foreach (var schema in CollectorCatalog.All) { - Assert.Contains("timescaledb.compress_segmentby = 'server_id'", + /* #5574: perfmon_stats segments by counter as well; every other collector table keeps server_id alone. */ + var wantedSegmentBy = schema.TargetTable == TimescaleSupport.PerfmonStatsTable ? "server_id, counter_name" : "server_id"; + Assert.Contains($"timescaledb.compress_segmentby = '{wantedSegmentBy}'", TimescaleSupport.EnableCompressionSql(schema), StringComparison.Ordinal); Assert.Contains("if_not_exists => true", TimescaleSupport.AddCompressionPolicySql(schema), StringComparison.Ordinal); @@ -150,10 +152,33 @@ other pin in this file still expects. CompressAfterFor is the single source both TimescaleSupport.AddCompressionPolicySql(schema), StringComparison.Ordinal); } - /* collection_log gets the identical compression via the raw-name overloads (the runtime path). */ + /* collection_log gets its compression via the raw-name overloads (the runtime path), segmented by + collector as well as by server (#4951): its reads ask for one collector on one server, so a + server_id-only segment made every such read decompress every collector's runs on that server. */ Assert.Equal( - "ALTER TABLE collection_log SET (timescaledb.compress, timescaledb.compress_segmentby = 'server_id')", + "ALTER TABLE collection_log SET (timescaledb.compress, timescaledb.compress_segmentby = 'server_id, collector_name')", TimescaleSupport.EnableCompressionSql(TimescaleSupport.CollectionLogTable)); + + /* The one lookup behind both the statement and the convergence read: collection_log alone gets the + collector column, every collector table keeps server_id, and collection_log's ALTER waits a bounded + time for its lock. */ + Assert.Equal("server_id, collector_name", TimescaleSupport.CollectionLogSegmentBy); + Assert.Equal(TimescaleSupport.CollectionLogSegmentBy, TimescaleSupport.CompressionSegmentByFor(TimescaleSupport.CollectionLogTable)); + /* #5574: perfmon_stats alone among the collector tables also segments by counter. */ + Assert.Equal("server_id, counter_name", TimescaleSupport.PerfmonStatsSegmentBy); + Assert.All(CollectorCatalog.All.Where(schema => schema.TargetTable != TimescaleSupport.PerfmonStatsTable), schema => Assert.Equal("server_id", TimescaleSupport.CompressionSegmentByFor(schema.TargetTable))); + Assert.Equal(TimescaleSupport.PerfmonStatsSegmentBy, TimescaleSupport.CompressionSegmentByFor(TimescaleSupport.PerfmonStatsTable)); + + /* Callers spell tables both ways ("collect.x" and bare "x"). A schema-qualified collection_log must get the + same value as the bare name the convergence read compares against, or the read never sees the table as + converged and every hourly pass issues the ALTER (#3817's divergence, reached through a spelling). */ + Assert.Equal(TimescaleSupport.CollectionLogSegmentBy, TimescaleSupport.CompressionSegmentByFor("collect." + TimescaleSupport.CollectionLogTable)); + Assert.Equal( + "ALTER TABLE collect.collection_log SET (timescaledb.compress, timescaledb.compress_segmentby = 'server_id, collector_name')", + TimescaleSupport.EnableCompressionSql("collect." + TimescaleSupport.CollectionLogTable)); + Assert.All(CollectorCatalog.All.Where(schema => schema.TargetTable != TimescaleSupport.PerfmonStatsTable), schema => Assert.Equal("server_id", TimescaleSupport.CompressionSegmentByFor("collect." + schema.TargetTable))); + Assert.Equal(TimescaleSupport.PerfmonStatsSegmentBy, TimescaleSupport.CompressionSegmentByFor("collect." + TimescaleSupport.PerfmonStatsTable)); + Assert.Equal("3s", TimescaleSupport.HourlyDdlLockTimeout); Assert.True(TimescaleSupport.TryCompressionPhaseMinutesFor(TimescaleSupport.CollectionLogTable, out var logPhase)); Assert.Equal( "SELECT add_compression_policy('collection_log', compress_after => INTERVAL '1 days', schedule_interval => INTERVAL '1 hour', if_not_exists => true, " @@ -289,6 +314,40 @@ the point is that last_run_started_at is never read raw. */ TimescaleSupport.StuckPolicyJobsSql, StringComparison.Ordinal); } + /* ---------------- collection_log's settings change, by TimescaleDB version (#4951) ---------------- */ + + [Theory] + [InlineData("2.13.0")] + [InlineData("2.13.1")] + [InlineData("2.13.1-dev")] + public void CollectionLogSettingsChange_BelowTheRelease_WaitsWhileAChunkIsCompressed(string extversion) + { + /* Before 2.14 TimescaleDB refuses a compression-settings change while any chunk is compressed, so the + ensure skips it there; with no compressed chunk, any 2.x takes the ALTER. */ + var version = TimescaleSupport.ParseTimescaleVersion(extversion); + Assert.NotNull(version); + Assert.True(TimescaleSupport.CompressionSettingsChangeBlocked(version, hasCompressedChunks: true)); + Assert.False(TimescaleSupport.CompressionSettingsChangeBlocked(version, hasCompressedChunks: false)); + } + + [Theory] + [InlineData("2.14")] + [InlineData("2.14.0")] + [InlineData("2.17.2")] + [InlineData("2.30.1")] + [InlineData(null)] + [InlineData("not-a-version")] + public void CollectionLogSettingsChange_FromTheRelease_OrUnknown_IsAttempted(string? extversion) + { + /* From 2.14 the change applies to chunks compressed after it. An unknown version is attempted too: skipping + it would keep a new store on the old setting forever, and a refused ALTER only rolls back and logs. A + two-part "2.14" must not rank below the floor, which is why the floor has two parts. */ + Assert.Equal(new Version(2, 14), TimescaleSupport.CompressionSettingsChangeWithCompressedChunksFrom); + var version = TimescaleSupport.ParseTimescaleVersion(extversion); + Assert.False(TimescaleSupport.CompressionSettingsChangeBlocked(version, hasCompressedChunks: true)); + Assert.False(TimescaleSupport.CompressionSettingsChangeBlocked(version, hasCompressedChunks: false)); + } + /* ---------------- the -infinity arm's sentence, by TimescaleDB version (#3591) ---------------- */ [Theory] @@ -2133,6 +2192,8 @@ and the corrected hourly on the standard 21-day one. The corrected DAILY is kept legacies they supersede, and are DERIVED from the supersession registry so a fourth pair cannot leave an armed policy behind on the shared fixture. 20 relations since Q12 (17 before). */ .Concat(TimescaleSupport.SupersededHourlyRollups.Select(s => s.Successor)) + /* #5329: the two io hourlies carry the leaf rule's 90-day policy too (Coverage = themselves): 23 relations. */ + .Concat(new[] { TimescaleSupport.QueryStatsIoHourlyView, TimescaleSupport.ProcedureStatsIoHourlyView }) .Concat(TimescaleSupport.BaselineAggregates.Select(a => a.View)) /* #3893: the off-grid fleet collection-health rollup carries its own 8-day leaf policy. 21 relations. */ .Concat(TimescaleSupport.OffGridAggregates.Select(a => a.View)) @@ -2517,13 +2578,17 @@ width and a rounded measurement certifies a derivation the code does not have as 14 at Q12, 11 before it), the heaviest refresh starts at 11 + the 4-minute guard = :15 (was :18, :15 before Q12), and its window is the remainder 60 - 15 - 24 = 21 minutes (was 18, 21 before Q12). The literals move WITH the identities beside them; nothing here was renumbered by hand. */ - Assert.Equal(12, TimescaleSupport.LightHourlyRefreshCount); - Assert.Equal(11, TimescaleSupport.LightBandSpanMinutes); - Assert.Equal(15, TimescaleSupport.HeaviestRefreshStartMinute); + /* RE-DERIVED AGAIN at #5329 by appending the two io hourlies (query_stats_io_hourly and + procedure_stats_io_hourly): fifteen hourly policies, fourteen of them light, so the light band spans + (14 - 1) * 1 = 13 minutes (was 11), the heaviest refresh starts at 13 + the 4-minute guard = :17 (was + :15), and its window is 60 - 17 - 24 = 19 minutes (was 21). */ + Assert.Equal(14, TimescaleSupport.LightHourlyRefreshCount); + Assert.Equal(13, TimescaleSupport.LightBandSpanMinutes); + Assert.Equal(17, TimescaleSupport.HeaviestRefreshStartMinute); Assert.Equal( TimescaleSupport.LightBandSpanMinutes + TimescaleSupport.CompressionPhaseGuardMinutes, TimescaleSupport.HeaviestRefreshStartMinute); - Assert.Equal(21, TimescaleSupport.HeaviestRefreshWindowMinutes); + Assert.Equal(19, TimescaleSupport.HeaviestRefreshWindowMinutes); Assert.Equal( TimescaleSupport.MinutesInHourlyCadence - TimescaleSupport.HeaviestRefreshStartMinute @@ -2569,7 +2634,9 @@ next. A recorded ceiling that outgrew the window therefore has to FAIL here rath #3653's Q12, before the freeze moved the legacy trio off this grid). The compression band does NOT move: it is the hour's remainder after the heaviest window, and the window grew back by exactly what the light band shrank, so :36-:59 stays where #3174 left it. */ - Assert.Equal(new[] { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 15 }, refreshMinutes); + /* #5329: fourteen light minutes :00-:13 and the heaviest at :17 (the two io hourlies); the compression + band still does NOT move, :36-:59, because the window is the hour's remainder (17 + 19 = :36). */ + Assert.Equal(new[] { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 17 }, refreshMinutes); Assert.Equal(TimescaleSupport.HourlyRefreshPhaseOrder.Count, refreshMinutes.Length); var heaviestMinute = TimescaleSupport.RefreshPhaseMinutesFor(TimescaleSupport.HeaviestHourlyRefreshView); @@ -2621,8 +2688,10 @@ filter produced output". */ /* Twelve light starts at :00-:11, each excluding its own minute and the three after it: :00-:14, fifteen minutes (was :00-:17, eighteen, during Q12). The heaviest window is the 21 minutes :15-:35 (was 18, :18-:35, during Q12). */ - Assert.Equal(15, excludedByGuard.Length); - Assert.Equal(21, excludedByHeaviestWindow.Length); + /* #5329: fourteen light starts at :00-:13 exclude :00-:16, seventeen minutes, and the heaviest window is + the 19 minutes :17-:35. */ + Assert.Equal(17, excludedByGuard.Length); + Assert.Equal(19, excludedByHeaviestWindow.Length); Assert.Contains(heaviestMinute + 1, excludedByHeaviestWindow); Assert.Empty(excludedByGuard.Intersect(TimescaleSupport.CompressionPhaseMinutes)); Assert.Empty(excludedByHeaviestWindow.Intersect(TimescaleSupport.CompressionPhaseMinutes)); @@ -2699,8 +2768,12 @@ its own two terms have moved. */ GAP is stated in seconds beside the percentage because a percentage that truncates to 0 can read as "none" when it is not. A fourth light member, or a ceiling re-derived one run higher, turns this red — which is the ruling the grid asks for at that point rather than a hand-tuned band. */ - Assert.Equal(17, (TimescaleSupport.RefreshSlotWarningSeconds - ceiling) * 100 / ceiling); - Assert.Equal(154, TimescaleSupport.RefreshSlotWarningSeconds - ceiling); + /* RE-DERIVED at #5329 (the two io hourlies, window 21 -> 19 minutes): the line is 1,140 * 5 / 6 = + 950 s, and the ceiling sits 54 s under it, 6% of itself (was 154 s, 17%). Still ordered, so the + pin holds; the headroom is what the io views spent. The next light member turns the ordering + red at the same ceiling, which is the ruling the grid asks for then. */ + Assert.Equal(6, (TimescaleSupport.RefreshSlotWarningSeconds - ceiling) * 100 / ceiling); + Assert.Equal(54, TimescaleSupport.RefreshSlotWarningSeconds - ceiling); } /* ─────────────────── #3044: the watch on the LIVE figure, not the constant ─────────────────── */ @@ -2726,10 +2799,12 @@ public void TheRefreshSlotWatchLines_AreDerivedFromTheWindow_NotWrittenDown() for the derivation) — the slot is 21 * 60 = 1,260 s (was 18 * 60 = 1,080 during Q12, when the successors were briefly appended behind the legacy trio instead) and the five-sixths line is 1,260 * 5 / 6 = 1,050 s (was 900). The identities beside the literals are what moved them. */ - Assert.Equal(1260, TimescaleSupport.RefreshPhaseSlotSeconds); + /* #5329: the two io hourlies take the window to 19 minutes, so the slot is 19 * 60 = 1,140 s and the + five-sixths line 1,140 * 5 / 6 = 950 s (was 1,260 and 1,050). */ + Assert.Equal(1140, TimescaleSupport.RefreshPhaseSlotSeconds); Assert.Equal(TimescaleSupport.HeaviestRefreshWindowMinutes * 60, TimescaleSupport.RefreshPhaseSlotSeconds); - Assert.Equal(1050, TimescaleSupport.RefreshSlotWarningSeconds); + Assert.Equal(950, TimescaleSupport.RefreshSlotWarningSeconds); Assert.Equal( TimescaleSupport.RefreshPhaseSlotSeconds * 5 / 6, TimescaleSupport.RefreshSlotWarningSeconds); @@ -2748,8 +2823,8 @@ a second line rather than only the wall. */ Assert.True( TimescaleSupport.RefreshSlotWarningSeconds < TimescaleSupport.RefreshPhaseSlotSeconds, "the watch line is at or past the slot it is meant to give warning of"); - /* The remaining sixth: 1,260 / 6 = 210 s of lead (was 1,080 / 6 = 180 during Q12). */ - Assert.Equal(210, TimescaleSupport.RefreshPhaseSlotSeconds - TimescaleSupport.RefreshSlotWarningSeconds); + /* The remaining sixth: 1,140 / 6 = 190 s of lead (was 1,260 / 6 = 210 before #5329). */ + Assert.Equal(190, TimescaleSupport.RefreshPhaseSlotSeconds - TimescaleSupport.RefreshSlotWarningSeconds); } /// @@ -2872,7 +2947,7 @@ public void TheRejectedWatchLineAlternative_NoLongerSitsBelowTheRecordedCeiling_ agrees with any derivation, including one frozen at 480 or at Q12's 840. */ var alternative = TimescaleSupport.RefreshPhaseSlotSeconds - (TimescaleSupport.CompressionPhaseGuardMinutes * 60); - Assert.Equal(1020, alternative); + Assert.Equal(900, alternative); Assert.True( alternative < TimescaleSupport.RefreshSlotWarningSeconds, "the alternative is no longer the LOWER of the two lines, so the paragraph rejecting it as the " @@ -2889,7 +2964,10 @@ than editing this assertion. */ $"the {alternative} s alternative sits at or below the {ceiling} s recorded ceiling again, so the " + "ordering no longer discriminates and the coupling argument is the load-bearing one alone — " + "re-read #3107 and #3174 rather than editing this assertion"); - Assert.Equal(124, alternative - ceiling); + /* #5329: 900 - 896 = 4 s (was 124 s at the 1,260 s window): the alternative is still the higher of the + two numbers, by almost nothing, so the ordering argument is nearly gone and the coupling is the + load-bearing one. The next light member would reverse it, and this assertion is where that shows. */ + Assert.Equal(4, alternative - ceiling); /* The chosen line still clears the ceiling; the alternative would too. Stated as the verdicts the classifier produces where it can be (the ceiling against the shipped line) and as the inequality @@ -2903,7 +2981,7 @@ where it cannot (a line is a compile-time value and the classifier cannot be re- ceiling < TimescaleSupport.RefreshSlotWarningSeconds, $"the chosen {TimescaleSupport.RefreshSlotWarningSeconds} s line no longer sits above the " + $"{ceiling} s recorded ceiling"); - Assert.Equal(30, TimescaleSupport.RefreshSlotWarningSeconds - alternative); + Assert.Equal(50, TimescaleSupport.RefreshSlotWarningSeconds - alternative); /* THE COUPLING, as the derivations rather than as prose. The alternative is the slot less one guard band, so it moves with the light class's declared width; the chosen line tracks only the window. @@ -2936,7 +3014,7 @@ below it does not. Without this the ordering would be arithmetic with no stated leaves the larger margin, so lead time argues FOR it and cannot be part of its rejection; at LC's restored window the guard band (240 s) is unchanged and the sixth is 210 s (was 180 during Q12). */ Assert.Equal(240, TimescaleSupport.RefreshPhaseSlotSeconds - alternative); - Assert.Equal(210, TimescaleSupport.RefreshPhaseSlotSeconds - TimescaleSupport.RefreshSlotWarningSeconds); + Assert.Equal(190, TimescaleSupport.RefreshPhaseSlotSeconds - TimescaleSupport.RefreshSlotWarningSeconds); } /// @@ -2966,8 +3044,9 @@ 154 s. That band assertion went RED on #3166's census against a 15-minute slot a it with geometry rather than with a renumbered band — see the classifier test for the ordering it restored; the freeze re-derived the geometry back to that same shape, and the band held. */ Assert.Equal(TimescaleSupport.RefreshSlotHeadroom.InsideSlot, atTheCeiling.Headroom); - Assert.Equal(364, atTheCeiling.ClearOfSlotSeconds); - Assert.Equal(71.1, atTheCeiling.PercentOfSlot, 1); + /* #5329: the 1,140 s window leaves 1,140 - 896 = 244 s clear, 78.6% of the window. */ + Assert.Equal(244, atTheCeiling.ClearOfSlotSeconds); + Assert.Equal(78.6, atTheCeiling.PercentOfSlot, 1); /* The watch line, which is where APPROACHING starts: 83.3% of the window, 1,260 / 6 = 210 s clear (was 180 s of Q12's 1,080 s window; the percentage is the same five-sixths fraction at any window @@ -2975,7 +3054,8 @@ 154 s. That band assertion went RED on #3166's census against a 15-minute slot a var atTheWatchLine = new HeaviestRefreshSlotReading( TimescaleSupport.HeaviestHourlyRefreshView, TimescaleSupport.RefreshSlotWarningSeconds); Assert.Equal(TimescaleSupport.RefreshSlotHeadroom.ApproachingSlot, atTheWatchLine.Headroom); - Assert.Equal(210, atTheWatchLine.ClearOfSlotSeconds); + /* #5329: the window is 1,140 s, so the sixth is 190 s clear. */ + Assert.Equal(190, atTheWatchLine.ClearOfSlotSeconds); Assert.Equal(83.3, atTheWatchLine.PercentOfSlot, 1); /* OVERRUN, derived: a run one sixth of the window past the wall. The sixth is the same fraction the @@ -3165,7 +3245,8 @@ public void TheJobCadenceKnob_FiresInsideTheWindow_WhichIsWhyTheSlotWatchIsSepar + "(V57), so the repair is the grid"); /* #3653's LC freeze restores the 1,260 s window, leaving the knob 360 s inside the wall (was 180 inside Q12's 1,080 s window, before the freeze moved the legacy trio off this grid). */ - Assert.Equal(360, TimescaleSupport.RefreshPhaseSlotSeconds - (hourlyCadenceSeconds * ShippedWarnPercent / 100)); + /* #5329: the two io hourlies take the window to 1,140 s, so the knob sits 240 s inside the wall. */ + Assert.Equal(240, TimescaleSupport.RefreshPhaseSlotSeconds - (hourlyCadenceSeconds * ShippedWarnPercent / 100)); /* And where the knob's own clamp lets an operator move it to — well past the wall, with the grid's precondition broken and nothing said. */ @@ -3191,7 +3272,8 @@ the assertion below says so by failing on that width. */ $"the knob's {hourlyCadenceSeconds * ShippedWarnPercent / 100} s line no longer fires strictly " + $"before the {TimescaleSupport.RefreshSlotWarningSeconds} s watch line, so the two signals have " + "collapsed to a coincidence again — re-read #3174 and Q12 rather than editing this assertion"); - Assert.Equal(150, TimescaleSupport.RefreshSlotWarningSeconds - (hourlyCadenceSeconds * ShippedWarnPercent / 100)); + /* #5329: 950 - 900 = 50 s (was 150 s at the 1,260 s window). */ + Assert.Equal(50, TimescaleSupport.RefreshSlotWarningSeconds - (hourlyCadenceSeconds * ShippedWarnPercent / 100)); Assert.True( hourlyCadenceSeconds * ShippedWarnPercent / 100 <= TimescaleSupport.RefreshSlotWarningSeconds && TimescaleSupport.RefreshSlotWarningSeconds < TimescaleSupport.RefreshPhaseSlotSeconds, @@ -3608,18 +3690,23 @@ public void TheRefreshGridIsUnchanged_AndTheCompressionGridsOneInputFromItIsPinn successors in the three positions their legacies held — not appended behind the family, which is what keeps the grid at thirteen policies instead of sixteen. */ (TimescaleSupport.QueryStoreStatsHourlyView, 0), - (TimescaleSupport.QueryStoreStatsIntervalHourlyView, 15), + (TimescaleSupport.QueryStoreStatsIntervalHourlyView, 17), (TimescaleSupport.QueryStoreStatsCorrectedHourlyView, 4), (TimescaleSupport.QueryStatsIntervalHourlyView, 8), (TimescaleSupport.ProcedureStatsIntervalHourlyView, 1), (TimescaleSupport.QueryStatsDbIntervalHourlyView, 2), - (TimescaleSupport.PerfmonIntervalBaselineView, 3), - (TimescaleSupport.WaitStatsIntervalBaselineView, 5), - (TimescaleSupport.SessionStatsBaselineView, 6), - (TimescaleSupport.QueryStatsBaselineView, 7), - (TimescaleSupport.BlockedProcessBaselineView, 9), - (TimescaleSupport.DeadlockBaselineView, 10), - (TimescaleSupport.MemoryBaselineView, 11), + /* #5329: the two io hourlies, appended. The query one groups by statement, so it is the FOURTH + unbounded member and takes position 3 * 4 = 12; the procedure one is deployment-bounded and + takes the next free position, 3. The baseline members shift up past it. */ + (TimescaleSupport.QueryStatsIoHourlyView, 12), + (TimescaleSupport.ProcedureStatsIoHourlyView, 3), + (TimescaleSupport.PerfmonIntervalBaselineView, 5), + (TimescaleSupport.WaitStatsIntervalBaselineView, 6), + (TimescaleSupport.SessionStatsBaselineView, 7), + (TimescaleSupport.QueryStatsBaselineView, 9), + (TimescaleSupport.BlockedProcessBaselineView, 10), + (TimescaleSupport.DeadlockBaselineView, 11), + (TimescaleSupport.MemoryBaselineView, 13), }; Assert.Equal(expected.Length, TimescaleSupport.HourlyRefreshPhaseOrder.Count); diff --git a/Darling/Darling.Tests/TimingClaim.cs b/Darling/Darling.Tests/TimingClaim.cs new file mode 100644 index 0000000000..9caf10239d --- /dev/null +++ b/Darling/Darling.Tests/TimingClaim.cs @@ -0,0 +1,41 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using Xunit; + +namespace Darling.Tests; + +/// A claim about real elapsed time (#5369): "this came back inside N ms". A wall-clock bound fails on a +/// loaded machine for reasons that have nothing to do with the code, so it must not be able to fail a normal run. +/// Without DARLING_TIMING_TESTS=1 the measurement is only written to the test output; with it set the bound +/// is asserted. The deterministic claim (the verdict, or a fake clock's arithmetic) stays a plain assertion in the +/// test that calls this. +internal static class TimingClaim +{ + /// The opt-in variable. Set it to 1 on a quiet machine to enforce the wall-clock bounds. + internal const string EnvironmentVariable = "DARLING_TIMING_TESTS"; + + internal static bool Enforced => + string.Equals(Environment.GetEnvironmentVariable(EnvironmentVariable), "1", StringComparison.Ordinal); + + /// Asserts is at most when the opt-in is set; + /// otherwise records the reading and passes. + internal static void AtMost(double actualMs, double limitMs, string what) + { + var message = $"{what}: {actualMs:F0} ms against a {limitMs:F0} ms bound"; + if (Enforced) + { + Assert.True(actualMs <= limitMs, message); + } + else + { + TestContext.Current.SendDiagnosticMessage($"timing claim not enforced ({EnvironmentVariable} is not 1): {message}"); + } + } +} diff --git a/Darling/Darling.Tests/TimingCollection.cs b/Darling/Darling.Tests/TimingCollection.cs new file mode 100644 index 0000000000..989f86a08b --- /dev/null +++ b/Darling/Darling.Tests/TimingCollection.cs @@ -0,0 +1,32 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using Xunit; + +namespace Darling.Tests; + +/// +/// The one collection for every test class that times its own work and judges the time against a limit (#5602). +/// xunit runs a collection on its own, after the +/// parallel classes have finished, so the timed calls get a quiet process instead of sharing CPU with the other +/// 17,000 tests (#4788: the setting redactor's scaling check; #5484: the statement filter's warm-up waits; #5561: a +/// timing test on a 27 MB plan; #5600: the cloud identity probe's step limit). The bars stay where they were: a +/// real regression still fails them, and a busy runner no longer does. +/// +/// This collection replaced pg-setting-redactor-timing and statement-filter-warm-up. The warm-up +/// wait tests stage a process-wide fake warm-up; running them in this collection keeps them apart from every other +/// class too, which is all their own collection did. +/// +/// A class that times work and cannot join it (it needs the live-postgres collection's shared store) or +/// does not judge the time (a hang guard, a diagnostic) is named on the allow list of +/// with its reason. +/// +[CollectionDefinition("timing", DisableParallelization = true)] +public sealed class TimingCollection +{ +} diff --git a/Darling/Darling.Tests/TimingTestCensusTests.cs b/Darling/Darling.Tests/TimingTestCensusTests.cs new file mode 100644 index 0000000000..425f6029db --- /dev/null +++ b/Darling/Darling.Tests/TimingTestCensusTests.cs @@ -0,0 +1,168 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Reflection; +using System.Runtime.CompilerServices; +using Xunit; + +namespace Darling.Tests; + +/// +/// A Darling.Tests class that starts a clock runs in the timing collection or is on the allow list with its +/// reason (#5602). xunit runs test classes in parallel, so a class that times its work and fails past a limit is +/// judging the whole runner's load: #5561 (a 27 MB plan) and #5600 (the cloud probe's step limit, 1083 ms against a +/// 1000 ms bar in the whole-suite guards job) were that. is declared with +/// DisableParallelization, so xunit runs it alone, after the parallel classes. +/// +/// "Starts a clock" is 's rule: Stopwatch.StartNew, GetTimestamp, +/// GetElapsedTime, new Stopwatch, a DateTime.UtcNow subtraction or Environment.TickCount in +/// code (comments and string literals do not count). That is wider than "asserts on the elapsed time", so a class that +/// only reports a time is named too, and one more assertion cannot slip in unseen. The allow list is per file; every +/// entry carries its reason, and an entry whose file no longer starts a clock outside the collection fails, so the +/// list only shrinks. +/// +[Trait("Stage", "Guard")] +public sealed class TimingTestCensusTests +{ + private const string HangGuard = "hang guard: the bound is a generous ceiling that only stops a stuck test, so load cannot trip it and a regression is not what it measures"; + + private const string PollLoop = "poll loop: the clock only bounds a wait for a state that the test then asserts on; no assertion is made on the time"; + + private const string Diagnostic = "diagnostic: the time is written to the test output and nothing asserts on it"; + + private const string LiveStore = "live-postgres member (#5602): it needs that collection's shared migrated store, and a class is in one collection only. The bound is wide (a server-side timeout or a hang guard of 10 s and up, against work that takes milliseconds), so runner load does not trip it. It stays: a second copy of the shared-store fixture in the timing collection would read the process-wide LiveCleanupBatch ledger again and report every live class's residue twice"; + + /// Anti-vacuity: the timing collection holds well over this many classes, so a scan that stopped seeing it fails. + private const int TimingMemberFloor = 15; + + private static readonly Dictionary s_allowed = new(StringComparer.Ordinal) + { + ["CaggGroupIndexDropUpgradeLiveTests.cs"] = "lower bound only: it asserts the migration waited out a ~8 s lock hold (elapsed >= 7 s); load makes the wait longer, never shorter", + ["CollectionLogDrainForensicsStoreTests.cs"] = "the clock is handed to a counting reader under test; the assertions are on the readings it records (rows, bytes, a last-read stamp >= 0), with no upper bound on any time", + ["CollectionLogSegmentByLiveTests.cs"] = LiveStore + "; the 30 s ceiling is a hang guard on a lock wait", + ["ComposeTimeoutPhaseLiveTests.cs"] = HangGuard + " (60 s against a 5 s compose timeout)", + ["DarlingManagedPostgresTests.cs"] = PollLoop + " (a retry of a file operation the OS may still hold)", + ["MoveDirectoryOnceReleasedTests.cs"] = "lower bound only: it asserts the retry waited its patience before giving up; load makes the wait longer, never shorter", + ["PerDatabaseFaultPathSplitTests.cs"] = "the clock is a running slice the code under test reads; the test spins until it passes a minimum and asserts it keeps running and freezes when stopped, with no upper bound", + ["PgWaitSamplerLiveTests.cs"] = LiveStore + "; the bound is the sampling window plus 30 s, with the window as the lower bound", + ["PlanRegressionDailyTriggerCostRig.cs"] = Diagnostic + " (a cost rig, never asserted)", + ["ProvisioningSerializationTests.cs"] = "lower bounds on a deliberate wait plus ceilings of 20 s and 90 s over it, which are hang guards; the same file also holds a live-postgres member", + ["QueryStoreBackfillCutChunkLiveTests.cs"] = Diagnostic, + ["QueryStoreIntervalConvergenceLiveTests.cs"] = Diagnostic + " (each step's duration is recorded for the failure message)", + ["QueryStoreIntervalPurgeRowCappedTests.cs"] = HangGuard + " (120 s for a purge drain)", + ["RdsLogStateConcurrencyTests.cs"] = "the clock sets how long the test hammers the code (rounds for one second); no assertion is made on a time", + ["ReadScopeStatementCaptureLiveTests.cs"] = "live-postgres member; lower bound only (the recorded duration must reach 550 ms after a 600 ms delay)", + ["RollupBackfillLiveTests.cs"] = PollLoop + " (a held-chunk wait with a limit)", + ["SensitiveStatementJsonTests.cs"] = Diagnostic + " (median of 20 identity walks)", + ["SensitiveStatementOutputFilterTests.cs"] = "every time bound goes through TimingClaim.AtMost, which asserts only when DARLING_TIMING_TESTS=1 and otherwise records the reading; the best of three is what it judges", + ["SensitiveStatementSessionTests.cs"] = Diagnostic + " (throughput records)", + ["SensitiveStatementXmlTests.cs"] = Diagnostic + " (best-of-seven records; the only assertion is that the pass took time)", + ["SensitiveStatementsTests.cs"] = "time bounds go through TimingClaim.AtMost (asserts only when DARLING_TIMING_TESTS=1, best of several runs); the real assertions are lower bounds (a budgeted judge ran at least 60% of its limit) and fake-clock arithmetic", + ["StatementFilterBlockingReadsLiveTests.cs"] = LiveStore + "; the 10 s bound is a hang guard on a filter that judges 1 MB in tens of milliseconds", + ["StatementFilterPlanReadsLiveTests.cs"] = LiveStore + "; the 15 s bound is a hang guard on a filter that judges a plan in tens of milliseconds", + ["StatementScrubBlockingTests.cs"] = Diagnostic + " (scrub throughput records)", + ["StatementScrubSnapshotTests.cs"] = HangGuard + " (2 minutes for a worst-case snapshot read)", + ["StoreCopyPhaseLivePostgresTests.cs"] = "live-postgres member; lower bound only (the start phase must not give up before the sweep deadline less a second); the upper bound moved to the exact fault type and message asserts, which tell the deadline from Npgsql's own timeout (#5602)", + ["TrendStaleStatsLiveTests.cs"] = Diagnostic + " (the assertions are on the answer status)", + ["TuningDropLockRecoveryLiveTests.cs"] = HangGuard + " (60 s against a single blocked drop)", + ["WebExceptionTextCensusTests.cs"] = "the stopwatch's reading is passed as the elapsed-milliseconds argument of a fake tool's timeout result; no assertion is made on a time", + ["TuningReaderBudgetLiveTests.cs"] = LiveStore + "; the bound is half of each reader's 15 s or 30 s command deadline", + }; + + [Fact] + public void TheDetector_SeesAClockAndTheCollection_AndIgnoresProseAndStrings() + { + const string text = "namespace N;\n\n" + + "[Collection(\"timing\")]\npublic sealed class Timed\n{\n void A() { var w = Stopwatch.StartNew(); }\n}\n\n" + + "public sealed class Bare\n{\n void B() { long t = Stopwatch.GetTimestamp(); }\n}\n\n" + + "[Trait(\"Stage\", \"Guard\")]\n[Collection(\"timing\")]\npublic sealed class TwoAttributes\n{\n void C() { var d = DateTime.UtcNow - start; }\n}\n\n" + + "public sealed class Prose\n{\n // var w = Stopwatch.StartNew();\n /* new Stopwatch() */\n" + + " void D() { Assert.Contains(\"Stopwatch.GetTimestamp()\", source); var s = @\"a \"\"Stopwatch.StartNew()\"\" b\"; }\n" + + " /// Environment.TickCount\n void E() { }\n}\n\n" + + "public sealed class Interpolated\n{\n void F() => Log($\"x {Stopwatch.StartNew()}\");\n}\n"; + + var rows = TimingTestScanner.ScanText("t.cs", text).ToDictionary(r => r.Name); + + Assert.True(rows["Timed"].StartsClock); + Assert.True(rows["Timed"].InTimingCollection); + Assert.True(rows["Bare"].StartsClock); + Assert.False(rows["Bare"].InTimingCollection); + Assert.True(rows["TwoAttributes"].StartsClock); + Assert.True(rows["TwoAttributes"].InTimingCollection); + Assert.False(rows["Prose"].StartsClock); + Assert.False(rows["Interpolated"].StartsClock); + } + + [Fact] + public void TheDetector_ReportsAClockOutsideTheCollection_AndAStaleAllowEntry() + { + var rows = TimingTestScanner.ScanText( + "Bare.cs", + "public sealed class Bare\n{\n void B() { var w = new Stopwatch(); }\n}\n"); + + var problems = TimingTestScanner.Problems(rows, new Dictionary()); + Assert.Single(problems); + Assert.Contains("Bare.cs", problems[0], StringComparison.Ordinal); + + Assert.Empty(TimingTestScanner.Problems(rows, new Dictionary { ["Bare.cs"] = "reason" })); + Assert.Equal(new[] { "Gone.cs" }, TimingTestScanner.Stale(rows, new Dictionary { ["Bare.cs"] = "reason", ["Gone.cs"] = "reason" })); + } + + [Fact] + public void EveryClassThatStartsAClock_RunsInTheTimingCollection_OrIsAllowedWithAReason() + { + var rows = TimingTestScanner.Scan(TestsRoot()); + + var problems = TimingTestScanner.Problems(rows, s_allowed); + Assert.True( + problems.Count == 0, + "A test class starts a clock outside the `timing` collection (#5602). xunit runs classes in parallel, so a time limit " + + "judged there is judged against the runner's load. Add [Collection(\"timing\")] above the class (TimingCollection runs " + + "alone, after the parallel classes), assert on the work instead of the clock, or, for a hang guard, a poll loop or a " + + "diagnostic, name the file in TimingTestCensusTests.s_allowed with the reason:\n " + string.Join("\n ", problems)); + + var stale = TimingTestScanner.Stale(rows, s_allowed); + Assert.True( + stale.Count == 0, + "These allow-list entries no longer name a file that starts a clock outside the timing collection; delete them:\n " + + string.Join("\n ", stale)); + + var members = rows.Count(r => r.InTimingCollection); + Assert.True( + members >= TimingMemberFloor, + $"only {members} Darling.Tests classes are in the timing collection; the scan has stopped seeing them"); + } + + [Fact] + public void EveryAllowListEntry_NamesARealFile_AndGivesAReason() + { + foreach (var (file, reason) in s_allowed) + { + Assert.True(File.Exists(Path.Combine(TestsRoot(), file)), $"allow-list entry {file} is not a file under Darling.Tests"); + Assert.False(string.IsNullOrWhiteSpace(reason), $"allow-list entry {file} has no reason"); + } + } + + [Fact] + public void TheTimingCollection_IsNotParallel_AndIsTheOnlyOneOfItsName() + { + var definitions = typeof(TimingTestCensusTests).Assembly.GetTypes() + .Select(t => (Type: t, Attribute: t.GetCustomAttribute())) + .Where(d => d.Attribute is not null && string.Equals(d.Attribute.Name, TimingTestScanner.CollectionName, StringComparison.Ordinal)) + .ToList(); + + var definition = Assert.Single(definitions); + Assert.True(definition.Attribute!.DisableParallelization, "the timing collection must run alone, or it protects nothing"); + } + + private static string TestsRoot([CallerFilePath] string thisFile = "") => Path.GetDirectoryName(thisFile)!; +} diff --git a/Darling/Darling.Tests/TimingTestScanner.cs b/Darling/Darling.Tests/TimingTestScanner.cs new file mode 100644 index 0000000000..539f55bc89 --- /dev/null +++ b/Darling/Darling.Tests/TimingTestScanner.cs @@ -0,0 +1,275 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Text.RegularExpressions; + +namespace Darling.Tests; + +/// +/// Finds the test classes that start a clock, for the timing census of each test suite (#5602). +/// +/// Why. xunit runs test classes in parallel, so a class that times its own work and fails when the +/// time passes a limit is judging the whole runner's load, not the code. #5561 and #5600 were that. The fix is a +/// single collection, timing, declared with DisableParallelization: xunit runs it alone, after the +/// parallel classes. The census (TimingTestCensusTests in each project) fails when a class starts a clock +/// outside that collection and is not on the allow list with its reason. +/// +/// What counts as starting a clock. A statement in a class body, after comments and string contents are +/// blanked, that calls Stopwatch.StartNew, Stopwatch.GetTimestamp or Stopwatch.GetElapsedTime, +/// constructs a Stopwatch, subtracts a DateTime.UtcNow/Now reading from a variable named like a start time (started, start, began; a heuristic, so "N minutes ago" arithmetic is not caught), or reads +/// Environment.TickCount. Source pins that quote those calls inside a string literal do not count. The rule is +/// wider than "asserts on the elapsed time": a class that only reports a time has to say so on the allow list, which +/// keeps one more place from growing an assertion unnoticed. +/// +/// Linked into Lite.Tests (see Lite.Tests.csproj and the lite_linked_shard filter in build.yml). +/// +internal static class TimingTestScanner +{ + /// The collection name every timing class carries. + internal const string CollectionName = "timing"; + + internal const string CollectionAttributeText = "[Collection(\"" + CollectionName + "\")]"; + + private static readonly Regex s_clock = new( + @"\bStopwatch\s*\.\s*(?:StartNew|GetTimestamp|GetElapsedTime)\s*\(|\bnew\s+(?:System\.Diagnostics\.)?Stopwatch\s*\(|\b(?:DateTime|DateTimeOffset)\s*\.\s*(?:UtcNow|Now)\s*-\s*(?:started|startedAt|start|startUtc|began|begun|t0)\b|\bEnvironment\s*\.\s*TickCount(?:64)?\b", + RegexOptions.CultureInvariant); + + private static readonly Regex s_classHeader = new( + @"^(?:(?:public|internal)\s+)?(?:(?:sealed|static|abstract|partial)\s+)*(?:class|record)\s+(?\w+)", + RegexOptions.CultureInvariant | RegexOptions.Multiline); + + /// One top-level class: its file, its name, whether it starts a clock, whether it is in timing. + internal sealed record Row(string File, string Name, bool StartsClock, bool InTimingCollection); + + /// Every top-level class under (build output skipped), one row per declaration. + internal static IReadOnlyList Scan(string testsRoot) + { + var rows = new List(); + foreach (var path in Directory.EnumerateFiles(testsRoot, "*.cs", SearchOption.AllDirectories).OrderBy(p => p, StringComparer.Ordinal)) + { + var relative = Path.GetRelativePath(testsRoot, path).Replace('\\', '/'); + if (relative.StartsWith("bin/", StringComparison.Ordinal) || relative.StartsWith("obj/", StringComparison.Ordinal)) + { + continue; + } + + rows.AddRange(ScanText(relative, File.ReadAllText(path))); + } + + return rows; + } + + /// The rows of one file's text. + internal static IReadOnlyList ScanText(string file, string text) + { + var withStrings = Blank(text, blankStrings: false); + var code = Blank(text, blankStrings: true); + var headers = s_classHeader.Matches(code); + var rows = new List(); + for (var i = 0; i < headers.Count; i++) + { + var start = headers[i].Index; + var end = i + 1 < headers.Count ? headers[i + 1].Index : code.Length; + + // The attribute lines directly above the declaration belong to it; the clock lines never sit there. + var attrStart = start; + while (attrStart > 0) + { + var lineEnd = attrStart - 1; + var lineStart = withStrings.LastIndexOf('\n', Math.Max(lineEnd - 1, 0)) + 1; + var line = withStrings.Substring(lineStart, lineEnd - lineStart).Trim(); + if (line.StartsWith('[') && line.EndsWith(']')) + { + attrStart = lineStart; + continue; + } + + break; + } + + var attributes = withStrings.Substring(attrStart, start - attrStart); + rows.Add(new Row( + file, + headers[i].Groups["name"].Value, + s_clock.IsMatch(code.AsSpan(start, end - start)), + attributes.Contains(CollectionAttributeText, StringComparison.Ordinal))); + } + + return rows; + } + + /// The problems with : a clock outside timing, off the allow list. + internal static List Problems(IReadOnlyList rows, IReadOnlyDictionary allowed) + { + var problems = new List(); + foreach (var row in rows.Where(r => r.StartsClock && !r.InTimingCollection)) + { + if (!allowed.ContainsKey(row.File)) + { + problems.Add($"{row.File}: {row.Name} starts a clock outside {CollectionAttributeText} and is not on the allow list"); + } + } + + return problems; + } + + /// The allow-list entries that no longer name a file with a clock outside timing: the list only shrinks. + internal static List Stale(IReadOnlyList rows, IReadOnlyDictionary allowed) + { + var live = rows.Where(r => r.StartsClock && !r.InTimingCollection).Select(r => r.File).ToHashSet(StringComparer.Ordinal); + return allowed.Keys.Where(k => !live.Contains(k)).OrderBy(k => k, StringComparer.Ordinal).ToList(); + } + + /// + /// with comments, and optionally the contents of string and character literals, replaced by + /// spaces. Length and line breaks are kept, so an index into the result is an index into the source. + /// + internal static string Blank(string text, bool blankStrings) + { + var output = new StringBuilder(text.Length); + var i = 0; + while (i < text.Length) + { + var c = text[i]; + var next = i + 1 < text.Length ? text[i + 1] : '\0'; + + if (c == '/' && next == '/') + { + while (i < text.Length && text[i] != '\n') + { + output.Append(text[i] == '\r' ? '\r' : ' '); + i++; + } + + continue; + } + + if (c == '/' && next == '*') + { + output.Append(" "); + i += 2; + while (i < text.Length && !(text[i] == '*' && i + 1 < text.Length && text[i + 1] == '/')) + { + output.Append(text[i] is '\n' or '\r' ? text[i] : ' '); + i++; + } + + if (i < text.Length) + { + output.Append(" "); + i += 2; + } + + continue; + } + + if (c == '"' && next == '"' && i + 2 < text.Length && text[i + 2] == '"') + { + // Raw string literal: three or more quotes to the same count. + var quotes = 0; + while (i + quotes < text.Length && text[i + quotes] == '"') + { + quotes++; + } + + var closer = new string('"', quotes); + var close = text.IndexOf(closer, i + quotes, StringComparison.Ordinal); + var stop = close < 0 ? text.Length : close + quotes; + CopyString(text, i, stop, output, blankStrings, keep: quotes); + i = stop; + continue; + } + + if (c == '"' || ((c == '@' || c == '$') && IsStringStart(text, i))) + { + var verbatim = false; + var j = i; + while (text[j] is '@' or '$') + { + verbatim |= text[j] == '@'; + j++; + } + + // text[j] is the opening quote. + var k = j + 1; + while (k < text.Length) + { + if (verbatim && text[k] == '"' && k + 1 < text.Length && text[k + 1] == '"') + { + k += 2; + continue; + } + + if (!verbatim && text[k] == '\\') + { + k += 2; + continue; + } + + if (text[k] == '"' || (!verbatim && text[k] == '\n')) + { + break; + } + + k++; + } + + var stop = Math.Min(k + 1, text.Length); + CopyString(text, i, stop, output, blankStrings, keep: (j - i) + 1); + i = stop; + continue; + } + + if (c == '\'') + { + var k = i + 1; + while (k < text.Length && text[k] != '\'' && text[k] != '\n') + { + k += text[k] == '\\' ? 2 : 1; + } + + var stop = Math.Min(k + 1, text.Length); + CopyString(text, i, stop, output, blankStrings, keep: 1); + i = stop; + continue; + } + + output.Append(c); + i++; + } + + return output.ToString(); + } + + private static bool IsStringStart(string text, int index) + { + var j = index; + while (j < text.Length && text[j] is '@' or '$') + { + j++; + } + + return j < text.Length && text[j] == '"'; + } + + /// Appends text[from..to); with the interior (past opening characters and the closing one) becomes spaces. + private static void CopyString(string text, int from, int to, StringBuilder output, bool blank, int keep) + { + for (var p = from; p < to; p++) + { + var ch = text[p]; + var isEdge = p - from < keep || p == to - 1; + output.Append(!blank || isEdge || ch is '\n' or '\r' ? ch : ' '); + } + } +} diff --git a/Darling/Darling.Tests/TopCpuQueriesTextLiveTests.cs b/Darling/Darling.Tests/TopCpuQueriesTextLiveTests.cs index 459efd524b..c143812aa4 100644 --- a/Darling/Darling.Tests/TopCpuQueriesTextLiveTests.cs +++ b/Darling/Darling.Tests/TopCpuQueriesTextLiveTests.cs @@ -140,7 +140,7 @@ public async Task TheDrillDown_ReturnsTheOraclesRows_AndResolvesTextForThePrinte try { - var windowEnd = TruncateToSeconds(DateTime.UtcNow); + var windowEnd = TruncateToSeconds(DateTime.SpecifyKind(LiveClock.Now(), DateTimeKind.Utc)); var windowStart = windowEnd.AddHours(-4); var context = new AnalysisContext { diff --git a/Darling/Darling.Tests/TopCpuQueriesTextShapeTests.cs b/Darling/Darling.Tests/TopCpuQueriesTextShapeTests.cs index 62493b5ba4..3cb2bdd05b 100644 --- a/Darling/Darling.Tests/TopCpuQueriesTextShapeTests.cs +++ b/Darling/Darling.Tests/TopCpuQueriesTextShapeTests.cs @@ -35,7 +35,7 @@ window nor the ranking may project it... */ Assert.DoesNotContain("query_text", sql[..cut], StringComparison.Ordinal); /* ...it is read after the cut: equality for a keyed group, NULL-safe for a group with a NULL key. */ - Assert.Contains("SELECT LEFT(MAX(v.query_text), 500)", sql[cut..], StringComparison.Ordinal); + Assert.Contains("SELECT MAX(v.query_text)", sql[cut..], StringComparison.Ordinal); Assert.Contains("v.query_hash = t.query_hash", sql[cut..], StringComparison.Ordinal); Assert.Contains("CASE WHEN t.database_name IS NULL OR t.query_hash IS NULL THEN", sql[cut..], StringComparison.Ordinal); Assert.Contains("v.query_hash IS NOT DISTINCT FROM t.query_hash", sql[cut..], StringComparison.Ordinal); diff --git a/Darling/Darling.Tests/TopFillEmptyRoundLiveTests.cs b/Darling/Darling.Tests/TopFillEmptyRoundLiveTests.cs new file mode 100644 index 0000000000..49a3519321 --- /dev/null +++ b/Darling/Darling.Tests/TopFillEmptyRoundLiveTests.cs @@ -0,0 +1,147 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Darling.Tests; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace PerformanceMonitor.Darling.Tests; + +/// +/// #5313: a window whose top groups are ALL WAITFOR shells, more of them than the first round's top + 5 candidates. +/// The first round's page is then trimmed to nothing, so it returns no page row. It used to carry no +/// candidate_count either, and an empty round read as "the candidates ran out": the list came back empty although +/// real groups existed below the shells. Each statement now returns its count on a row of its own (page_ord NULL), +/// so the fill asks again. +/// +/// Each test seeds twelve shells that outrank six real groups and reads a page of three, so round one (8 candidates) +/// sees only shells and the answer is only right if a later round runs. +/// +[Collection("live-postgres")] +public sealed class TopFillEmptyRoundLiveTests +{ + private const int Top = 3; + private const int Shells = 12; + private const int Real = 6; + private const string WaitforText = "WAITFOR DELAY '00:00:30'"; + + [Fact] + public Task TopQueries_AFirstRoundOfOnlyShells_StillFillsThePage() => + WithSharedStoreAsync("top-fill-empty-round-queries", async (connection, postgres, serverId, serverName, now, ct) => + { + for (var i = 1; i <= Shells + Real; i++) + { + var shell = i <= Shells; + var text = shell ? WaitforText : "SELECT real " + i; + var weight = shell ? 1_000_000L - i : 10_000L - i * 100; + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO query_stats + (collection_id, collection_time, server_id, server_name, database_name, query_hash, query_plan_hash, sql_handle, plan_handle, + query_text, query_text_digest, delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, + delta_logical_writes, delta_physical_reads, min_worker_time, max_worker_time, min_elapsed_time, max_elapsed_time, + min_dop, max_dop, sample_interval_seconds) + VALUES ($1, $2, $3, $4, 'FillDb', $5, $6, $7, $8, $9, $10, 10, $11, $11, 100, 0, 0, 1, $11, 1, $11, 1, 1, 3600)", + CollectionIdGenerator.Next(), DarlingMcpTestData.TruncateToSeconds(now.AddHours(-2).AddSeconds(i)), serverId, serverName, + "0xQ" + i, "0xP" + i, "0xS" + i, "0xL" + i, text, SHA256.HashData(Encoding.UTF8.GetBytes(text)), weight); + } + + var (start, end) = (now.AddHours(-24), now.AddMinutes(5)); + foreach (var ranking in new[] { TopRanking.Cpu, TopRanking.Duration }) + { + foreach (var rollUp in new[] { false, true }) + { + var result = await DarlingDataReader.GetTopQueriesByCpuRoutedAsync(postgres, serverId, start, end, Top, null, rollUp, 0, ranking, ct); + var label = $"{ranking}, roll-up {rollUp}"; + Assert.True(result.Rows.Count == Top, $"{label}: {result.Rows.Count} rows, expected {Top}"); + Assert.DoesNotContain(result.Rows, r => (r.QueryText ?? "").StartsWith("WAITFOR", StringComparison.Ordinal)); + Assert.Equal(new[] { "SELECT real 13", "SELECT real 14", "SELECT real 15" }, result.Rows.Select(r => r.QueryText).ToArray()); + } + } + }, "DELETE FROM query_stats WHERE server_id = {0}"); + + [Fact] + public Task QueryStoreTop_Mcp_AFirstRoundOfOnlyShells_StillFillsThePage() => + WithSharedStoreAsync("top-fill-empty-round-qs-mcp", async (connection, postgres, serverId, serverName, now, ct) => + { + await SeedQueryStoreAsync(connection, serverId, serverName, now, ct); + var rows = await DarlingDataReader.GetQueryStoreTopAsync(postgres, serverId, now.AddHours(-3), now.AddMinutes(5), Top, null, null, null, ct); + Assert.Equal(Top, rows.Count); + Assert.DoesNotContain(rows, r => (r.QueryText ?? "").StartsWith("WAITFOR", StringComparison.Ordinal)); + Assert.Equal(new long[] { 113, 114, 115 }, rows.Select(r => r.QueryId).ToArray()); + }, "DELETE FROM query_store_stats WHERE server_id = {0}"); + + [Fact] + public Task QueryStoreTop_Viewer_AFirstRoundOfOnlyShells_StillFillsThePage() => + WithSharedStoreAsync("top-fill-empty-round-qs-viewer", async (connection, postgres, serverId, serverName, now, ct) => + { + await SeedQueryStoreAsync(connection, serverId, serverName, now, ct); + await using var viewer = new ViewerDataService(Environment.GetEnvironmentVariable("DARLING_TEST_PG")!); + var rows = await viewer.GetQueryStoreTopQueriesAsync(serverId, now.AddHours(-3), now.AddMinutes(5), Top, cancellationToken: ct); + Assert.Equal(Top, rows.Count); + Assert.DoesNotContain(rows, r => (r.QueryText ?? "").StartsWith("WAITFOR", StringComparison.Ordinal)); + Assert.Equal(new long[] { 113, 114, 115 }, rows.Select(r => r.QueryId).ToArray()); + }, "DELETE FROM query_store_stats WHERE server_id = {0}"); + + /// Twelve shell plans that outrank six real ones on total duration (executions times average). + private static async Task SeedQueryStoreAsync(NpgsqlConnection connection, int serverId, string serverName, DateTime now, CancellationToken ct) + { + for (var i = 1; i <= Shells + Real; i++) + { + var shell = i <= Shells; + var queryId = shell ? i : 100 + i; + var duration = shell ? 50_000_000L - i : 10_000L - i * 10; + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO query_store_stats (collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, query_hash, query_plan_hash, query_text, execution_count, avg_duration_us, avg_cpu_time_us, last_execution_time) + VALUES ($1, $2, $3, $4, 'FillQsDb', $5, $5, $6, $7, $8, 10, $9, 800, $2)", + CollectionIdGenerator.Next(), DarlingMcpTestData.TruncateToSeconds(now.AddHours(-2).AddSeconds(i)), serverId, serverName, (long)queryId, + "0xQ" + queryId, "0xP" + queryId, shell ? WaitforText : "SELECT real " + queryId, duration); + } + } + + private static async Task WithSharedStoreAsync( + string serverName, Func body, string cleanupSql) + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), "Set DARLING_TEST_PG to run the live fill tests."); + var ct = TestContext.Current.CancellationToken; + var serverId = ServerIdHelper.GetDeterministicHashCode(serverName); + var cleanup = string.Format(System.Globalization.CultureInfo.InvariantCulture, cleanupSql, serverId) + + string.Format(System.Globalization.CultureInfo.InvariantCulture, "; DELETE FROM servers WHERE server_id = {0}", serverId); + + using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DarlingMcpTestData.ExecAsync(connection, ct, cleanup); + + await using var postgres = NpgsqlDataSource.Create(connectionString!); + var succeeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, serverId, serverName, ct); + await body(connection, postgres, serverId, serverName, DarlingMcpTestData.Naive(DateTime.UtcNow), ct); + succeeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, succeeded, async (cleanupConnection, cleanupCt) => + await DarlingMcpTestData.ExecAsync(cleanupConnection, cleanupCt, cleanup)); + } + } +} diff --git a/Darling/Darling.Tests/TopProceduresDetailLiveTests.cs b/Darling/Darling.Tests/TopProceduresDetailLiveTests.cs new file mode 100644 index 0000000000..1fbde8de80 --- /dev/null +++ b/Darling/Darling.Tests/TopProceduresDetailLiveTests.cs @@ -0,0 +1,130 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Darling.Tests; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace PerformanceMonitor.Darling.Tests; + +/// +/// The widened get_top_procedures_by_cpu against live PostgreSQL: detail='full' adds the desktop Top Procedures grid's +/// remaining fields from procedure_stats, converts the two timestamps from the monitored server's clock, omits the +/// ones the store holds no value for, and the default call stays as lean as before. +/// +[Collection("live-postgres")] +public sealed class TopProceduresDetailLiveTests +{ + private const string ServerName = "darling-top-procedures-detail-e2e"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + + [Fact] + public async Task FullDetail_CarriesTheDesktopFields_ConvertsTheClock_AndOmitsNulls() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), "Set DARLING_TEST_PG to run the live top-procedures detail test."); + var ct = TestContext.Current.CancellationToken; + + using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await CleanupAsync(connection, ct); + + await using var postgres = NpgsqlDataSource.Create(connectionString!); + var succeeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + /* A non-zero offset on purpose: at 0 the conversion asserted below would pass whether or not it ran. */ + await DarlingMcpTestData.ExecAsync(connection, ct, + "INSERT INTO server_properties (collection_id, collection_time, server_id, server_name, utc_offset_minutes) " + + $"VALUES ({CollectionIdGenerator.Next()}, now() AT TIME ZONE 'UTC', {ServerId}, '{ServerName}', -240)"); + var now = DarlingMcpTestData.Naive(DateTime.UtcNow); + var lastExec = new DateTime(2026, 3, 4, 5, 6, 7, DateTimeKind.Unspecified); + var cached = new DateTime(2026, 3, 1, 2, 3, 4, DateTimeKind.Unspecified); + + await PlantAsync(connection, ct, now.AddHours(-3), "usp_Full", 500, full: true, lastExec, cached, 2, 40); + await PlantAsync(connection, ct, now.AddHours(-2), "usp_Full", 400, full: true, lastExec, cached, 1, 80); + await PlantAsync(connection, ct, now.AddHours(-1), "usp_Lean", 10, full: false, null, null, 1, 1); + + var rows = await DarlingDataReader.GetTopProceduresByCpuAsync(postgres, ServerId, now.AddDays(-7), now.AddMinutes(5), top: 20, databaseName: null, cancellationToken: ct); + var detail = rows.First(r => r.ObjectName == "usp_Full").Detail; + Assert.NotNull(detail); + /* Stored on the monitored server's clock (UTC-4); the read hands back naive UTC. */ + Assert.Equal(lastExec.AddHours(4), detail!.LastExecutionTime); + Assert.Equal(cached.AddHours(4), detail.CachedTime); + Assert.Equal(1L, detail.MinLogicalReads); + Assert.Equal(80L, detail.MaxLogicalReads); + Assert.Equal(5_000_000_000L, detail.MaxPhysicalReads); + Assert.Equal(1L, detail.MinSpills); + Assert.Null(rows.First(r => r.ObjectName == "usp_Lean").Detail!.MaxLogicalReads); + + var summary = await DarlingMcpDataTools.GetTopProceduresByCpu(postgres, ServerName, hours_back: 168, top: 20); + using (var s = JsonDocument.Parse(summary)) + { + var q = s.RootElement.GetProperty("procedures").EnumerateArray().First(x => x.GetProperty("full_name").GetString() == "dbo.usp_Full"); + Assert.False(q.TryGetProperty("max_logical_reads", out _)); + Assert.False(q.TryGetProperty("last_execution_time", out _)); + } + + var full = await DarlingMcpDataTools.GetTopProceduresByCpu(postgres, ServerName, hours_back: 168, top: 20, detail: "full"); + using var f = JsonDocument.Parse(full); + var procs = f.RootElement.GetProperty("procedures").EnumerateArray().ToList(); + var fp = procs.First(x => x.GetProperty("full_name").GetString() == "dbo.usp_Full"); + Assert.Equal("2026-03-04T09:06:07.0000000", fp.GetProperty("last_execution_time").GetString()); + Assert.Equal("2026-03-01T06:03:04.0000000", fp.GetProperty("cached_time").GetString()); + Assert.Equal(80, fp.GetProperty("max_logical_reads").GetInt64()); + Assert.Equal(5_000_000_000L, fp.GetProperty("max_physical_reads").GetInt64()); + Assert.Equal(1, fp.GetProperty("min_spills").GetInt64()); + Assert.True(fp.GetProperty("avg_spills").GetDouble() > 0); + var lean = procs.First(x => x.GetProperty("full_name").GetString() == "dbo.usp_Lean"); + Assert.False(lean.TryGetProperty("max_logical_reads", out _)); + Assert.False(lean.TryGetProperty("last_execution_time", out _)); + Assert.Equal(JsonValueKind.Null, f.RootElement.GetProperty("precision_note").ValueKind); + + var refused = await DarlingMcpDataTools.GetTopProceduresByCpu(postgres, ServerName, hours_back: 168, top: 20, detail: "everything"); + Assert.Contains("detail must be", refused, StringComparison.Ordinal); + succeeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, succeeded, async (cleanup, cleanupCt) => + await CleanupAsync(cleanup, cleanupCt)); + } + } + + private static async Task PlantAsync( + NpgsqlConnection connection, CancellationToken ct, DateTime at, string objectName, long weight, bool full, + DateTime? lastExec, DateTime? cached, long minSpills, long maxReads) + { + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO procedure_stats (collection_id, collection_time, server_id, server_name, database_name, schema_name, object_name, + sql_handle, delta_worker_time, delta_elapsed_time, delta_execution_count, delta_spills, + sample_interval_seconds, last_execution_time, cached_time, + min_logical_reads, max_logical_reads, max_physical_reads, min_spills, max_spills) + VALUES ($1,$2,$3,$4,$5,'dbo',$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19)", + CollectionIdGenerator.Next(), at, ServerId, ServerName, "StackOverflow", objectName, "0xSQLH" + objectName, + weight * 1000L, weight * 2000L, weight, 4L, 60, + (object?)lastExec ?? DBNull.Value, (object?)cached ?? DBNull.Value, + full ? minSpills : DBNull.Value, full ? maxReads : DBNull.Value, full ? 5_000_000_000L : DBNull.Value, + full ? minSpills : DBNull.Value, full ? minSpills + 5 : DBNull.Value); + } + + private static async Task CleanupAsync(NpgsqlConnection connection, CancellationToken ct) => + await DarlingMcpTestData.ExecAsync(connection, ct, + $"DELETE FROM procedure_stats WHERE server_id = {ServerId}; DELETE FROM servers WHERE server_id = {ServerId}"); +} diff --git a/Darling/Darling.Tests/TopProceduresHourlyRoutingLiveTests.cs b/Darling/Darling.Tests/TopProceduresHourlyRoutingLiveTests.cs index 21c3d0903c..af8429c5a3 100644 --- a/Darling/Darling.Tests/TopProceduresHourlyRoutingLiveTests.cs +++ b/Darling/Darling.Tests/TopProceduresHourlyRoutingLiveTests.cs @@ -228,6 +228,18 @@ public async Task GetTopProceduresByCpu_HourlyRouted_CarriesTierUsedAndPrecision Assert.All(procedures, p => Assert.Equal(System.Text.Json.JsonValueKind.Null, p.GetProperty(column).ValueKind)); } + /* detail=full on the rollup: no detail field exists there, and the note says so. */ + using var fullDoc = System.Text.Json.JsonDocument.Parse(await DarlingMcpDataTools.GetTopProceduresByCpu( + hourlyDataSource, ServerName, hours_back: hoursBack, top: 10, as_of: asOf, detail: "full")); + Assert.Equal("hourly", fullDoc.RootElement.GetProperty("tier_used").GetString()); + Assert.Contains("detail=full was asked", fullDoc.RootElement.GetProperty("precision_note").GetString()!, StringComparison.Ordinal); + Assert.All(fullDoc.RootElement.GetProperty("procedures").EnumerateArray(), p => + { + Assert.False(p.TryGetProperty("last_execution_time", out _)); + Assert.False(p.TryGetProperty("max_logical_reads", out _)); + }); + Assert.DoesNotContain("detail=full was asked", hourlyNote.GetString()!, StringComparison.Ordinal); + bodySucceeded = true; } finally diff --git a/Darling/Darling.Tests/TopProceduresHourlyRoutingTests.cs b/Darling/Darling.Tests/TopProceduresHourlyRoutingTests.cs index aa0c85e2c6..32d955488c 100644 --- a/Darling/Darling.Tests/TopProceduresHourlyRoutingTests.cs +++ b/Darling/Darling.Tests/TopProceduresHourlyRoutingTests.cs @@ -36,7 +36,9 @@ public void TopProceduresHourlySql_GroupsByDatabaseSchemaObject_RanksByWorkerTim Assert.Contains("GROUP BY database_name, schema_name, object_name", sql, StringComparison.Ordinal); Assert.DoesNotContain("object_type", sql, StringComparison.Ordinal); - Assert.Contains("ORDER BY SUM(worker_time_sum) DESC", sql, StringComparison.Ordinal); + /* The ranking is the const's anchor; the CPU read is the const expanded to the worker-time sum (#5226). */ + Assert.Contains("ORDER BY rank_metric DESC NULLS LAST", sql, StringComparison.Ordinal); + Assert.Contains("SUM(worker_time_sum) AS rank_metric", TopRankings.Apply(sql, TopRanking.Cpu, hourly: true), StringComparison.Ordinal); Assert.Contains(DarlingDataReader.TopProceduresHourlyFromPlaceholder, sql, StringComparison.Ordinal); } @@ -71,6 +73,42 @@ public void TopProceduresHourlySql_NeverNamesRollupRelationDirectly() Assert.DoesNotContain("FROM procedure_stats_hourly", methodBody, StringComparison.Ordinal); } + /// + /// #5329 lane B2 source pin: the procedures read takes the io route through ChooseProceduresHourlyRoute (the one route + /// table, TopRankings.ChooseHourlyRoute, with no second copy), the hourly arm names procedure_stats_io_hourly and + /// projects its three sums only on that route, the reads ranking carries the raw retention notice only when it does not read io, + /// and the procedures tool no longer says only raw carries reads. + /// + [Fact] + public void TheIoRoute_IsChosenInTheRoutedRead_NamedInTheHourlyArm_AndDisclosedByTheTool() + { + var source = File.ReadAllText(FindReaderSourcePath()); + var routed = source.Substring(source.IndexOf("public static async Task GetTopProceduresByCpuRoutedAsync", StringComparison.Ordinal)); + routed = routed[..routed.IndexOf("GetTopProceduresByCpuHourlyAsync(postgres, coverage", StringComparison.Ordinal)]; + Assert.Contains("ChooseProceduresHourlyRoute(rollups, coverage, ranking, startUtc)", routed, StringComparison.Ordinal); + Assert.Contains("ranking != TopRanking.Reads || ioRoute", routed, StringComparison.Ordinal); + Assert.DoesNotContain("TopRankings.HourlyCarries(ranking)", routed, StringComparison.Ordinal); + + var chooser = source.Substring(source.IndexOf("public static HourlyRoute ChooseProceduresHourlyRoute", StringComparison.Ordinal)); + chooser = chooser[..chooser.IndexOf("return TopRankings.ChooseHourlyRoute", StringComparison.Ordinal)]; + Assert.Contains("return TopRankings.ChooseHourlyRoute(ranking, startUtc, ioFloor, rawOldest);", source, StringComparison.Ordinal); + Assert.Contains("TimescaleSupport.ProcedureStatsIoHourlyView", chooser, StringComparison.Ordinal); + Assert.DoesNotContain("QueryStats", chooser, StringComparison.Ordinal); + + var arm = source.Substring(source.IndexOf("private static async Task<(List Rows, DateTime? FirstBucket)> GetTopProceduresByCpuHourlyAsync", StringComparison.Ordinal)); + arm = arm[..arm.IndexOf("/* ─────────────────────────── query store", StringComparison.Ordinal)]; + /* #5329: the io FROM goes through StitchedRelationSql like every other hourly relation (no literal relation name). */ + Assert.Contains("coverage.StitchedRelationSql(\n io ? TimescaleSupport.ProcedureStatsIoHourlyView : TimescaleSupport.ProcedureStatsHourlyView,", arm.Replace("\r\n", "\n"), StringComparison.Ordinal); + Assert.DoesNotContain("collect.{TimescaleSupport.ProcedureStatsIoHourlyView} AS f", arm, StringComparison.Ordinal); + Assert.Contains("io ? TimescaleSupport.ProcedureStatsIoHourlyView : TimescaleSupport.ProcedureStatsHourlyView", arm, StringComparison.Ordinal); + Assert.Contains("TopRankings.HourlyIoSums", arm, StringComparison.Ordinal); + + var tool = File.ReadAllText(Path.Combine(RepoRoot(), "Darling", "PerformanceMonitor.Darling.Service", "Mcp", "DarlingMcpDataTools.cs")); + Assert.DoesNotContain("which only raw procedure_stats carries (the hourly rollup keeps CPU", tool, StringComparison.Ordinal); + Assert.Contains("procedure_stats_io_hourly", tool, StringComparison.Ordinal); + Assert.Contains("hourly && !routed.IoRoute ? (long?)null : r.TotalLogicalReads", tool, StringComparison.Ordinal); + } + private static string FindReaderSourcePath() => Path.Combine(RepoRoot(), "Darling", "PerformanceMonitor.Darling.Service", "Mcp", "DarlingDataReader.cs"); diff --git a/Darling/Darling.Tests/TopProceduresIoRouteLiveTests.cs b/Darling/Darling.Tests/TopProceduresIoRouteLiveTests.cs new file mode 100644 index 0000000000..2892f7bb62 --- /dev/null +++ b/Darling/Darling.Tests/TopProceduresIoRouteLiveTests.cs @@ -0,0 +1,289 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Darling.Tests; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace PerformanceMonitor.Darling.Tests; + +/* #1776 own-store: each test mints a scratch database, because it materializes continuous aggregates (and one drops + one), which no other class sharing the store may see. */ +/// +/// #5329 lane B2, live: the Top Procedures reader on an Hourly-tier window with the io hourly rollup absent, empty, partial +/// and covering (the twin of TopQueriesIoRouteLiveTests). The oracle seed is lane A's +/// (a NULL-reads group, a reads tie, two databases, a zero-interval row the rollups drop and a NULL-interval row they keep). +/// +/// The oracle: over the SAME window, the reads ranking answered from procedure_stats_io_hourly equals the raw route +/// (keys, order and every total, the reads, physical reads and writes included), and the CPU, duration and executions rankings +/// answered from it equal the same ranking over the interval-honest sibling. Raw is read FIRST (its rows still exist), then +/// the raw rows are deleted so the router must read a rollup. Each read goes through a FRESH data source, because coverage +/// is cached per data source. +/// +public sealed class TopProceduresIoRouteLiveTests +{ + private const int ServerId = -943862; + private const string ServerName = "a5329-top-procedures-io-route"; + private const int Top = 10; + + private static readonly DateTime WindowStart = new(2026, 1, 5, 0, 0, 0, DateTimeKind.Unspecified); + private static readonly DateTime WindowEnd = WindowStart.AddHours(IoRollupOracleSeed.Hours + 1); + private static readonly DateTime SurvivorAt = WindowStart.AddHours(IoRollupOracleSeed.Hours).AddMinutes(30); + + private sealed record Total(string Db, string Name, long Executions, long CpuUs, long ElapsedUs, long LogicalReads, long PhysicalReads, long LogicalWrites); + + private static Total[] Totals(IEnumerable rows, bool withIo = true) => + rows.Select(r => new Total( + r.DatabaseName, r.ObjectName, r.TotalExecutions, r.TotalCpuUs, r.TotalElapsedUs, + withIo ? r.TotalLogicalReads : 0, withIo ? r.TotalPhysicalReads : 0, withIo ? r.TotalLogicalWrites : 0)).ToArray(); + + [Fact] + public Task Covering_EveryRankingReadsIo_ReadsEqualsTheRawRoute_AndTheOtherRankingsEqualTheIntervalSibling() => + WithStoreAsync(async (scratch, connection, ct) => + { + await IoRollupOracleSeed.PlantAsync(connection, ct, ServerId, ServerName, WindowStart); + await TopRankingLiveTests.PlantProcedureAsync(connection, ct, "collect.procedure_stats", ServerId, ServerName, SurvivorAt, "QSURV", 1_000, 1_000, 7, 1); + await RefreshAsync(connection, TimescaleSupport.ProcedureStatsIntervalHourlyView, WindowStart, ct); + await RefreshAsync(connection, TimescaleSupport.ProcedureStatsIoHourlyView, WindowStart, ct); + + /* The raw route, read FIRST while raw still holds the rows: the raw reads statement itself over the same window. The + router cannot supply it: a window this old is Hourly-tier by age, and the reader's raw arm runs only on a + recent one. */ + var raw = await RunRawReadsAsync(connection, ct); + Assert.True(raw.Length >= 5, "the seed should rank at least five groups, or the comparison below proves nothing"); + Assert.DoesNotContain(raw, r => r.Name == "usp_D"); /* the zero-interval row's 9,999,999 reads are in neither route */ + + /* Now the raw rows go, so the window can only be answered by a rollup. */ + await PurgeRawBeforeAsync(connection, WindowStart.AddHours(IoRollupOracleSeed.Hours), ct); + + await using var ioSource = NpgsqlDataSource.Create(scratch.ConnectionString); + var io = await DarlingDataReader.GetTopProceduresByCpuRoutedAsync( + ioSource, ServerId, WindowStart, WindowEnd, Top, databaseName: null, ranking: TopRanking.Reads, cancellationToken: ct); + Assert.Equal(RetentionTier.Hourly, io.Tier); + Assert.True(io.IoRoute); + Assert.False(io.RawForced); + Assert.Null(io.RetentionNotice); /* the io rollup reaches the start: nothing partial to disclose */ + Assert.Equal(raw, Totals(io.Rows)); + Assert.Contains(io.Rows, r => r.TotalLogicalReads > 0 && r.TotalPhysicalReads > 0 && r.TotalLogicalWrites > 0); + Assert.Contains(io.Rows, r => r.ObjectName == "usp_C" && r.TotalLogicalReads == 0); /* the NULL-reads group stays 0, never dropped */ + + /* Every other ranking reads the io relation too, and equals the interval sibling over the same window. */ + foreach (var ranking in new[] { TopRanking.Cpu, TopRanking.Duration, TopRanking.Executions }) + { + var label = TopRankings.WireName(ranking); + var viaIo = await DarlingDataReader.GetTopProceduresByCpuRoutedAsync( + ioSource, ServerId, WindowStart, WindowEnd, Top, databaseName: null, ranking: ranking, cancellationToken: ct); + Assert.True(viaIo.IoRoute, $"{label}: covering io takes every ranking"); + Assert.Equal(RetentionTier.Hourly, viaIo.Tier); + var sibling = await RunHourlyAsync(connection, TimescaleSupport.ProcedureStatsIntervalHourlyView, ranking, ct); + Assert.Equal(sibling, Totals(viaIo.Rows, withIo: false).Select(t => (t.Db, t.Name, t.Executions, t.CpuUs, t.ElapsedUs)).ToArray()); + } + + /* The tool: the rows carry the sums, and the note names the relation instead of saying only raw has reads. */ + var query = "?server=" + ServerName + "&hours=24&as_of=" + Uri.EscapeDataString(WindowEnd.ToString("o")) + "&order_by=reads"; + using var doc = JsonDocument.Parse(await TopRankingLiveTests.WebReadAsync(ioSource, "get_top_procedures_by_cpu", query)); + Assert.Equal("hourly", doc.RootElement.GetProperty("tier_used").GetString()); + Assert.Contains("procedure_stats_io_hourly", doc.RootElement.GetProperty("precision_note").GetString(), StringComparison.Ordinal); + Assert.NotEqual(JsonValueKind.Null, doc.RootElement.GetProperty("procedures")[0].GetProperty("total_logical_reads").ValueKind); + }); + + [Fact] + public Task EmptyThenPartial_RoutesByWhichRelationReachesFurtherBack() => + WithStoreAsync(async (scratch, connection, ct) => + { + await IoRollupOracleSeed.PlantAsync(connection, ct, ServerId, ServerName, WindowStart); + await TopRankingLiveTests.PlantProcedureAsync(connection, ct, "collect.procedure_stats", ServerId, ServerName, SurvivorAt, "QSURV", 1_000, 1_000, 7, 1); + await RefreshAsync(connection, TimescaleSupport.ProcedureStatsIntervalHourlyView, WindowStart, ct); + await PurgeRawBeforeAsync(connection, WindowStart.AddHours(1), ct); /* raw now starts at hour 1 */ + + /* io EMPTY: reads is raw (forced, with raw's retention notice); the other rankings keep the stitched rollup. */ + var empty = await ReadAsync(scratch, TopRanking.Reads, ct); + Assert.Equal(RetentionTier.Raw, empty.Tier); + Assert.True(empty.RawForced); + Assert.False(empty.IoRoute); + Assert.NotNull(empty.RetentionNotice); + var cpuEmpty = await ReadAsync(scratch, TopRanking.Cpu, ct); + Assert.Equal(RetentionTier.Hourly, cpuEmpty.Tier); + Assert.False(cpuEmpty.IoRoute); + Assert.All(cpuEmpty.Rows, r => Assert.Equal(0, r.TotalLogicalReads)); + + /* Covering is the other test: the floor cache trusts a floor while the rollup's oldest chunk is unchanged, so a refresh + that only moves the floor earlier inside that chunk is not seen by the next read of the same store. */ + /* io PARTIAL (starts at hour 2) while raw reaches back to hour 1: raw is deeper, so reads stays raw. */ + await RefreshAsync(connection, TimescaleSupport.ProcedureStatsIoHourlyView, WindowStart.AddHours(2), ct); + var rawDeeper = await ReadAsync(scratch, TopRanking.Reads, ct); + Assert.Equal(RetentionTier.Raw, rawDeeper.Tier); + Assert.False(rawDeeper.IoRoute); + Assert.NotNull(rawDeeper.RetentionNotice); + Assert.False((await ReadAsync(scratch, TopRanking.Cpu, ct)).IoRoute); + + /* raw is purged to hour 4: now the partial io reaches further back than raw, and reads takes it, disclosed as + a hourly window that starts at io's first bucket rather than with raw's retention notice. */ + await PurgeRawBeforeAsync(connection, WindowStart.AddHours(4), ct); + var partial = await ReadAsync(scratch, TopRanking.Reads, ct); + Assert.Equal(RetentionTier.Hourly, partial.Tier); + Assert.True(partial.IoRoute); + Assert.Null(partial.RetentionNotice); + Assert.Equal(WindowStart.AddHours(2), partial.HourlyFirstBucket); + Assert.DoesNotContain(partial.Rows, r => r.ObjectName == "usp_A" && r.TotalLogicalReads == 0); + Assert.False((await ReadAsync(scratch, TopRanking.Cpu, ct)).IoRoute); /* partial io never takes a non-reads ranking */ + }); + + [Fact] + public Task Absent_AStoreWithoutTheIoRollup_ReadsRawForReads_AndNeverNamesTheMissingRelation() => + WithStoreAsync(async (scratch, connection, ct) => + { + await IoRollupOracleSeed.PlantAsync(connection, ct, ServerId, ServerName, WindowStart); + await TopRankingLiveTests.PlantProcedureAsync(connection, ct, "collect.procedure_stats", ServerId, ServerName, SurvivorAt, "QSURV", 1_000, 1_000, 7, 1); + await RefreshAsync(connection, TimescaleSupport.ProcedureStatsIntervalHourlyView, WindowStart, ct); + await PurgeRawBeforeAsync(connection, WindowStart.AddHours(IoRollupOracleSeed.Hours), ct); + await using (var drop = new NpgsqlCommand($"DROP MATERIALIZED VIEW collect.{TimescaleSupport.ProcedureStatsIoHourlyView}", connection)) + { + await drop.ExecuteNonQueryAsync(ct); + } + + var reads = await ReadAsync(scratch, TopRanking.Reads, ct); + Assert.Equal(RetentionTier.Raw, reads.Tier); + Assert.True(reads.RawForced); + Assert.Equal("usp_QSURV", Assert.Single(reads.Rows).ObjectName); + + var cpu = await ReadAsync(scratch, TopRanking.Cpu, ct); + Assert.Equal(RetentionTier.Hourly, cpu.Tier); + Assert.False(cpu.IoRoute); + Assert.NotEmpty(cpu.Rows); + }); + + // ---------------------------------------------------------------- plumbing + + private static async Task ReadAsync(ScratchPostgres scratch, TopRanking ranking, CancellationToken ct) + { + await using var source = NpgsqlDataSource.Create(scratch.ConnectionString); + return await DarlingDataReader.GetTopProceduresByCpuRoutedAsync( + source, ServerId, WindowStart, WindowEnd, Top, databaseName: null, ranking: ranking, cancellationToken: ct); + } + + /// The hourly statement over directly (the sibling the io rollup must equal), as + /// (database, object, executions, cpu, elapsed) in page order. + private static async Task<(string Db, string Name, long Executions, long CpuUs, long ElapsedUs)[]> RunHourlyAsync( + NpgsqlConnection connection, string view, TopRanking ranking, CancellationToken ct) + { + var sql = TopRankings.Apply(DarlingDataReader.TopProceduresHourlySql, ranking, hourly: true) + .Replace(DarlingDataReader.TopProceduresHourlyFromPlaceholder, $"collect.{view} AS f", StringComparison.Ordinal) + .Replace(DarlingDataReader.TopProceduresHourlyIoSumsPlaceholder, TopRankings.HourlyNoIoSums, StringComparison.Ordinal) + .Replace("$CEIL$", "", StringComparison.Ordinal); + await using var command = new NpgsqlCommand(sql, connection); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = ServerId }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = WindowStart }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = WindowEnd }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = Top }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Array | NpgsqlDbType.Text, Value = DBNull.Value }); + var rows = new List<(string, string, long, long, long)>(); + await using var reader = await command.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + /* database, schema, object, executions, cpu, elapsed (0-5), then the three io sums (6-8). */ + rows.Add((reader.GetString(0), reader.GetString(2), reader.GetInt64(3), reader.GetInt64(4), reader.GetInt64(5))); + } + + return rows.ToArray(); + } + + /// The raw reads statement ( ranked by reads) over the window, as totals in page order. + private static async Task RunRawReadsAsync(NpgsqlConnection connection, CancellationToken ct) + { + var sql = TopRankings.Apply(DarlingDataReader.TopProceduresSql, TopRanking.Reads, hourly: false); + await using var command = new NpgsqlCommand(sql, connection); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = ServerId }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = WindowStart }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = WindowEnd }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = Top }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Array | NpgsqlDbType.Text, Value = DBNull.Value }); + var rows = new List(); + await using var reader = await command.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + /* database, schema, object, type, sql_handle, plan_handle (0-5), executions, cpu, elapsed (6-8), logical reads (9), + logical writes (10), physical reads (11). */ + rows.Add(new Total( + reader.GetString(0), reader.GetString(2), reader.GetInt64(6), reader.GetInt64(7), reader.GetInt64(8), + reader.IsDBNull(9) ? 0 : reader.GetInt64(9), reader.IsDBNull(11) ? 0 : reader.GetInt64(11), reader.IsDBNull(10) ? 0 : reader.GetInt64(10))); + } + + return rows.ToArray(); + } + + private static async Task PurgeRawBeforeAsync(NpgsqlConnection connection, DateTime before, CancellationToken ct) + { + await using var purge = new NpgsqlCommand("DELETE FROM collect.procedure_stats WHERE server_id = $1 AND collection_time < $2", connection); + purge.Parameters.AddWithValue(ServerId); + purge.Parameters.AddWithValue(before); + await purge.ExecuteNonQueryAsync(ct); + } + + private static async Task RefreshAsync(NpgsqlConnection connection, string view, DateTime from, CancellationToken ct) + { + await using var refresh = new NpgsqlCommand($"CALL refresh_continuous_aggregate('collect.{view}'::regclass, $1::timestamp, $2::timestamp)", connection); + refresh.Parameters.AddWithValue(from); + refresh.Parameters.AddWithValue(WindowEnd.AddHours(1)); + await refresh.ExecuteNonQueryAsync(ct); + } + + /// Mints a scratch TimescaleDB store with the continuous aggregates and this class's server registered, and runs the body on it. + private static async Task WithStoreAsync(Func body) + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live #5329 procedures io route tests."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var timescaleEnabled = await TimescaleSupport.TryEnableAsync(connection, null, ct); + Assert.SkipWhen(!timescaleEnabled, "The live #5329 procedures io route tests need TimescaleDB."); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + Assert.True(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, null, ct)); + + await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection)) + { + await stop.ExecuteNonQueryAsync(ct); + } + + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, ct); + + var bodySucceeded = false; + try + { + await body(scratch, connection, ct); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (cleanup, cleanupCt) => + { + await using var probe = new NpgsqlCommand( + "SELECT count(*) FROM pg_catalog.pg_stat_activity WHERE datname = pg_catalog.current_database() " + + "AND backend_type LIKE 'TimescaleDB Background Worker Scheduler%'", cleanup); + var schedulers = Convert.ToInt64(await probe.ExecuteScalarAsync(cleanupCt)); + Assert.Equal(0L, schedulers); + }); + } + } +} diff --git a/Darling/Darling.Tests/TopQueriesDatabaseFilterLiveTests.cs b/Darling/Darling.Tests/TopQueriesDatabaseFilterLiveTests.cs new file mode 100644 index 0000000000..6408e6f7f1 --- /dev/null +++ b/Darling/Darling.Tests/TopQueriesDatabaseFilterLiveTests.cs @@ -0,0 +1,625 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Darling.Tests; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace PerformanceMonitor.Darling.Tests; + +/// +/// #5245 (part of #5244), PR1: the database filter on the three top reads (get_top_queries_by_cpu, +/// get_top_procedures_by_cpu, get_query_store_top). Each read has an internal overload that takes a +/// , with the list predicate ($5::text[] IS NULL OR database_name = ANY($5)) in every +/// statement of every tier, so an overload call with [A, B] over a seed of A, B and C returns only A's and B's rows, one +/// name reads as the public method does, and a refill round can never pull a row from C. +/// The public methods still pass one name, so there is no MCP schema, dispatch or tools/list byte change. +/// +/// This class is the shared-store half (the raw tiers and Query Store's raw route, which the store's own +/// precondition health table feeds); is the half that mints its own +/// database for the hourly rollup tier and Query Store's interval-table and daily routes. +/// +[Collection("live-postgres")] +public sealed class TopQueriesDatabaseFilterLiveTests +{ + private const string A = "TopFilterA"; + private const string B = "TopFilterB"; + private const string C = "TopFilterC"; + private const string WaitforText = "WAITFOR DELAY '00:00:30'"; + + /// The names of the databases a list of rows covers, in a stable order. + private static List Databases(IEnumerable names) => names.Distinct().OrderBy(n => n, StringComparer.Ordinal).ToList(); + + private static List JsonDatabases(string json, string property) => + Databases(JsonDocument.Parse(json).RootElement.GetProperty(property).EnumerateArray().Select(r => r.GetProperty("database_name").GetString()!)); + + private static DatabaseFilter Of(params string[] names) => DatabaseFilter.Of(names); + + /// One query_stats row with a text of its own; is its CPU and elapsed time. + private static Task PlantQueryAsync( + NpgsqlConnection connection, CancellationToken ct, int serverId, string serverName, DateTime at, string db, string tag, string text, long weight) => + DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO query_stats + (collection_id, collection_time, server_id, server_name, database_name, query_hash, query_plan_hash, sql_handle, plan_handle, + query_text, query_text_digest, delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, + delta_logical_writes, delta_physical_reads, min_worker_time, max_worker_time, min_elapsed_time, max_elapsed_time, + min_dop, max_dop, sample_interval_seconds) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, 10, $12, $12, 100, 0, 0, 1, $12, 1, $12, 1, 1, 3600)", + CollectionIdGenerator.Next(), DarlingMcpTestData.TruncateToSeconds(at), serverId, serverName, db, + "0xQ" + tag, "0xP" + tag, "0xS" + tag, "0xL" + tag, text, SHA256.HashData(Encoding.UTF8.GetBytes(text)), weight); + + private static Task PlantProcedureAsync( + NpgsqlConnection connection, CancellationToken ct, int serverId, string serverName, DateTime at, string db, string name, long weight) => + DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO procedure_stats + (collection_id, collection_time, server_id, server_name, database_name, schema_name, object_name, sql_handle, + delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, sample_interval_seconds) + VALUES ($1, $2, $3, $4, $5, 'dbo', $6, $7, 10, $8, $8, 100, 3600)", + CollectionIdGenerator.Next(), DarlingMcpTestData.TruncateToSeconds(at), serverId, serverName, db, name, "0xSQLH" + name, weight); + + private static Task PlantQueryStoreAsync( + NpgsqlConnection connection, CancellationToken ct, int serverId, string serverName, DateTime at, string db, long queryId, string text, long durationUs) => + DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO query_store_stats (collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, query_hash, query_plan_hash, query_text, execution_count, avg_duration_us, avg_cpu_time_us, last_execution_time) + VALUES ($1, $2, $3, $4, $5, $6, $6, $7, $8, $9, 10, $10, 800, $2)", + CollectionIdGenerator.Next(), DarlingMcpTestData.TruncateToSeconds(at), serverId, serverName, db, queryId, + "0xQ" + queryId, "0xP" + queryId, text, durationUs); + + private const string Cleanup = + "DELETE FROM query_stats WHERE server_id = {0}; DELETE FROM procedure_stats WHERE server_id = {0}; " + + "DELETE FROM query_store_stats WHERE server_id = {0}; DELETE FROM query_store_health WHERE server_id = {0}"; + + [Fact] + public Task TheRawTiers_AListOfDatabases_ReturnsOnlyThoseDatabases_AndOneNameReadsAsThePublicMethodDoes() => + WithSharedStoreAsync("top-filter-raw-tiers", async (connection, postgres, serverId, serverName, now, ct) => + { + foreach (var (db, weight) in new[] { (A, 3_000L), (B, 2_000L), (C, 9_000L) }) + { + await PlantQueryAsync(connection, ct, serverId, serverName, now.AddHours(-2), db, "Q" + db, "SELECT " + db, weight); + await PlantProcedureAsync(connection, ct, serverId, serverName, now.AddHours(-2), db, "usp_" + db, weight); + await PlantQueryStoreAsync(connection, ct, serverId, serverName, now.AddHours(-2), db, (long)(10 + db[^1]), "SELECT qs " + db, weight * 1000); + } + + var (start, end) = (now.AddHours(-24), now.AddMinutes(5)); + foreach (var rollUp in new[] { false, true }) + { + var queries = await DarlingDataReader.GetTopQueriesByCpuRoutedAsync(postgres, serverId, start, end, 10, Of(A, B), rollUp, 0, TopRanking.Cpu, ct); + Assert.Equal(RetentionTier.Raw, queries.Tier); + Assert.Equal([A, B], Databases(queries.Rows.Select(r => r.DatabaseName))); + Assert.Equal([A, B], Databases((await DarlingDataReader.GetTopQueriesByCpuAsync(postgres, serverId, start, end, 10, Of(B, A), rollUp, 0, TopRanking.Cpu, ct)).Select(r => r.DatabaseName))); + Assert.Equal([A, B, C], Databases((await DarlingDataReader.GetTopQueriesByCpuAsync(postgres, serverId, start, end, 10, DatabaseFilter.All, rollUp, 0, TopRanking.Cpu, ct)).Select(r => r.DatabaseName))); + /* One name: the public one-name method and the overload give the same page. */ + var one = await DarlingDataReader.GetTopQueriesByCpuAsync(postgres, serverId, start, end, 10, DatabaseFilter.One(B), rollUp, 0, TopRanking.Cpu, ct); + var oneByName = await DarlingDataReader.GetTopQueriesByCpuAsync(postgres, serverId, start, end, 10, B, rollUp, 0, TopRanking.Cpu, ct); + Assert.Equal(["SELECT " + B], one.Select(r => r.QueryText)); + Assert.Equal(one.Select(r => r.QueryHash), oneByName.Select(r => r.QueryHash)); + } + + var procedures = await DarlingDataReader.GetTopProceduresByCpuRoutedAsync(postgres, serverId, start, end, 10, Of(A, B), TopRanking.Cpu, ct); + Assert.Equal(RetentionTier.Raw, procedures.Tier); + Assert.Equal([A, B], Databases(procedures.Rows.Select(r => r.DatabaseName))); + Assert.Equal([A, B, C], Databases((await DarlingDataReader.GetTopProceduresByCpuAsync(postgres, serverId, start, end, 10, DatabaseFilter.All, TopRanking.Cpu, ct)).Select(r => r.DatabaseName))); + Assert.Equal([B], Databases((await DarlingDataReader.GetTopProceduresByCpuAsync(postgres, serverId, start, end, 10, B, TopRanking.Cpu, ct)).Select(r => r.DatabaseName))); + + var store = await DarlingDataReader.GetQueryStoreTopWithReachAsync(postgres, serverId, now.AddHours(-3), end, 10, Of(A, B), null, null, ct); + Assert.Null(store.Table); + Assert.Equal([A, B], Databases(store.Rows.Select(r => r.DatabaseName))); + Assert.Equal([A, B, C], Databases((await DarlingDataReader.GetQueryStoreTopAsync(postgres, serverId, now.AddHours(-3), end, 10, DatabaseFilter.All, null, null, ct)).Select(r => r.DatabaseName))); + Assert.Equal([B], Databases((await DarlingDataReader.GetQueryStoreTopAsync(postgres, serverId, now.AddHours(-3), end, 10, B, ct)).Select(r => r.DatabaseName))); + + /* The tools, end to end: the page is the top N of the chosen databases. */ + Assert.Equal([A, B], JsonDatabases(await DarlingMcpDataTools.GetTopQueriesRanked(postgres, serverName, 24, 10, Of(A, B), false, 0, "query_hash", null, "summary", null, ct), "queries")); + Assert.Equal([A, B], JsonDatabases(await DarlingMcpDataTools.GetTopProceduresRanked(postgres, serverName, 24, 10, Of(A, B), null, "summary", null, ct), "procedures")); + Assert.Equal([A, B], JsonDatabases(await DarlingMcpDataTools.GetQueryStoreTop(postgres, serverName, 3, 10, Of(A, B), null, null, null, false, 400, ct), "queries")); + }, Cleanup); + + /// + /// Twelve WAITFOR shells (six in A, six in B) outrank everything, so the first round (top + 5 = 8 candidates) sees only + /// shells and a refill must run. C's real groups outrank the real groups of A and B, so a refill that dropped the filter + /// would fill the page from C. Under the filter the page is A's and B's real groups. + /// + [Fact] + public Task ARefillRound_UnderTheFilter_NeverReturnsAnUnchosenDatabase() => + WithSharedStoreAsync("top-filter-refill", async (connection, postgres, serverId, serverName, now, ct) => + { + var tick = 0; + foreach (var db in new[] { A, B }) + { + for (var i = 1; i <= 6; i++) + { + tick++; + await PlantQueryAsync(connection, ct, serverId, serverName, now.AddHours(-2).AddSeconds(tick), db, "Shell" + db + i, WaitforText, 1_000_000 - tick); + await PlantQueryStoreAsync(connection, ct, serverId, serverName, now.AddHours(-2).AddSeconds(tick), db, tick, WaitforText, 50_000_000 - tick); + } + + for (var i = 1; i <= 2; i++) + { + tick++; + await PlantQueryAsync(connection, ct, serverId, serverName, now.AddHours(-2).AddSeconds(tick), db, "Real" + db + i, "SELECT real " + db + i, 10_000 - tick * 100); + await PlantQueryStoreAsync(connection, ct, serverId, serverName, now.AddHours(-2).AddSeconds(tick), db, tick, "SELECT real " + db + i, 10_000 - tick * 10); + } + } + + for (var i = 1; i <= 6; i++) + { + tick++; + await PlantQueryAsync(connection, ct, serverId, serverName, now.AddHours(-2).AddSeconds(tick), C, "Real" + C + i, "SELECT real " + C + i, 500_000 - tick); + await PlantQueryStoreAsync(connection, ct, serverId, serverName, now.AddHours(-2).AddSeconds(tick), C, tick, "SELECT real " + C + i, 20_000_000 - tick); + } + + var (start, end) = (now.AddHours(-24), now.AddMinutes(5)); + foreach (var rollUp in new[] { false, true }) + { + var rows = await DarlingDataReader.GetTopQueriesByCpuAsync(postgres, serverId, start, end, 3, Of(A, B), rollUp, 0, TopRanking.Cpu, ct); + Assert.True(rows.Count == 3, $"roll-up {rollUp}: {rows.Count} rows, expected 3"); + Assert.DoesNotContain(rows, r => (r.QueryText ?? "").StartsWith("WAITFOR", StringComparison.Ordinal)); + Assert.Equal([A, B], Databases(rows.Select(r => r.DatabaseName))); + Assert.DoesNotContain(rows, r => r.DatabaseName == C); + + /* Control: with no filter the same refill fills the page from C, which is what the filter has to stop. */ + var unfiltered = await DarlingDataReader.GetTopQueriesByCpuAsync(postgres, serverId, start, end, 3, DatabaseFilter.All, rollUp, 0, TopRanking.Cpu, ct); + Assert.All(unfiltered, r => Assert.Equal(C, r.DatabaseName)); + } + + var store = await DarlingDataReader.GetQueryStoreTopAsync(postgres, serverId, now.AddHours(-3), end, 3, Of(A, B), null, null, ct); + Assert.True(store.Count == 3, $"query store: {store.Count} rows, expected 3"); + Assert.DoesNotContain(store, r => (r.QueryText ?? "").StartsWith("WAITFOR", StringComparison.Ordinal)); + Assert.Equal([A, B], Databases(store.Select(r => r.DatabaseName))); + var storeUnfiltered = await DarlingDataReader.GetQueryStoreTopAsync(postgres, serverId, now.AddHours(-3), end, 3, DatabaseFilter.All, null, null, ct); + Assert.All(storeUnfiltered, r => Assert.Equal(C, r.DatabaseName)); + }, Cleanup); + + private static Task PlantHealthAsync(NpgsqlConnection connection, CancellationToken ct, int serverId, string serverName, DateTime at, string db, string actualState) => + DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO query_store_health (config_id, capture_time, server_id, server_name, database_name, actual_state, desired_state, readonly_reason, current_storage_size_mb, max_storage_size_mb, size_based_cleanup_mode, stale_query_threshold_days, max_plans_per_query, interval_length_minutes) + VALUES ($1,$2,$3,$4,$5,$6,'READ_WRITE',0,50,1000,'AUTO',30,200,60)", + CollectionIdGenerator.Next(), DarlingMcpTestData.TruncateToSeconds(at), serverId, serverName, db, actualState); + + /// + /// Every one-name consumer on get_query_store_top's empty paths is list-aware: Query Store off in one of two chosen + /// databases hands back the other's rows with no precondition envelope; off in both, an empty window answers with the + /// precondition envelope; off in one and recording in the other stays silent (the rule is "any database in scope + /// recording", not a per-database verdict). The execution_type and module_name empty answers name the chosen databases + /// for two or more names and keep today's text for one. + /// + [Fact] + public Task QueryStoreTop_TheEmptyPaths_AreListAware() => + WithSharedStoreAsync("top-filter-qs-precondition", async (connection, postgres, serverId, serverName, now, ct) => + { + await PlantHealthAsync(connection, ct, serverId, serverName, now.AddMinutes(-10), A, "OFF"); + await PlantHealthAsync(connection, ct, serverId, serverName, now.AddMinutes(-10), B, "READ_WRITE"); + await PlantHealthAsync(connection, ct, serverId, serverName, now.AddMinutes(-10), C, "OFF"); + await PlantQueryStoreAsync(connection, ct, serverId, serverName, now.AddHours(-2), B, 7, "SELECT qs " + B, 5_000); + + /* Off in A, on in B: B's rows come back, with no precondition envelope. */ + var rows = await DarlingMcpDataTools.GetQueryStoreTop(postgres, serverName, 3, 10, Of(A, B), null, null, null, false, 400, ct); + Assert.Equal([B], JsonDatabases(rows, "queries")); + Assert.False(JsonDocument.Parse(rows).RootElement.TryGetProperty("status", out var rowsStatus) && rowsStatus.GetString() == "precondition"); + + /* Off in both A and C, nothing recorded for them in the window: the precondition envelope. */ + var offBoth = JsonDocument.Parse(await DarlingMcpDataTools.GetQueryStoreTop(postgres, serverName, 3, 10, Of(A, C), null, null, null, false, 400, ct)).RootElement; + Assert.Equal("precondition", offBoth.GetProperty("status").GetString()); + Assert.Contains("Query Store is not enabled", offBoth.GetProperty("message").GetString(), StringComparison.Ordinal); + + /* Only A is in the snapshot, and A is off: the envelope. Once a database in scope records, the envelope is silent. */ + var onlyOff = JsonDocument.Parse(await DarlingMcpDataTools.GetQueryStoreTop(postgres, serverName, 1, 10, Of(A, "TopFilterNoRows"), null, null, null, false, 400, ct)).RootElement; + Assert.Equal("precondition", onlyOff.GetProperty("status").GetString()); + await PlantHealthAsync(connection, ct, serverId, serverName, now.AddMinutes(-10), "TopFilterNoRows", "READ_WRITE"); + var anyRecording = JsonDocument.Parse(await DarlingMcpDataTools.GetQueryStoreTop(postgres, serverName, 1, 10, Of(A, "TopFilterNoRows"), null, null, null, false, 400, ct)).RootElement; + Assert.NotEqual("precondition", anyRecording.GetProperty("status").GetString()); + + /* The execution_type empty answer: B has Regular rows only. */ + var two = JsonDocument.Parse(await DarlingMcpDataTools.GetQueryStoreTop(postgres, serverName, 3, 10, Of(A, B), null, "Aborted", null, false, 400, ct)).RootElement; + Assert.Equal("empty", two.GetProperty("status").GetString()); + Assert.Contains("No Aborted executions for the chosen databases in the 3-hour window", two.GetProperty("message").GetString(), StringComparison.Ordinal); + var one = JsonDocument.Parse(await DarlingMcpDataTools.GetQueryStoreTop(postgres, serverName, 3, 10, DatabaseFilter.One(B), null, "Aborted", null, false, 400, ct)).RootElement; + Assert.Contains("No Aborted executions in database '" + B + "' in the 3-hour window", one.GetProperty("message").GetString(), StringComparison.Ordinal); + Assert.Equal(one.GetProperty("message").GetString(), + JsonDocument.Parse(await DarlingMcpDataTools.GetQueryStoreTop(postgres, serverName, 3, 10, B, null, "Aborted", null, false, 400, ct)).RootElement.GetProperty("message").GetString()); + + /* The module_name empty answer, the same two phrasings. */ + var moduleTwo = JsonDocument.Parse(await DarlingMcpDataTools.GetQueryStoreTop(postgres, serverName, 3, 10, Of(A, B), null, null, "dbo.usp_Missing", false, 400, ct)).RootElement; + Assert.Contains("matched module_name 'dbo.usp_Missing' for the chosen databases in the 3-hour window", moduleTwo.GetProperty("message").GetString(), StringComparison.Ordinal); + var moduleOne = JsonDocument.Parse(await DarlingMcpDataTools.GetQueryStoreTop(postgres, serverName, 3, 10, DatabaseFilter.One(B), null, null, "dbo.usp_Missing", false, 400, ct)).RootElement; + Assert.Contains("matched module_name 'dbo.usp_Missing' in database '" + B + "' in the 3-hour window", moduleOne.GetProperty("message").GetString(), StringComparison.Ordinal); + }, Cleanup); + + /// The statement text pins: every pass of every tier carries the list predicate, and none keeps the one-name shape. + [Theory] + [InlineData(nameof(DarlingDataReader.TopQueriesSql), 2)] + [InlineData(nameof(DarlingDataReader.TopQueriesByHostObjectSql), 2)] + [InlineData(nameof(DarlingDataReader.TopQueriesHourlySql), 1)] + [InlineData(nameof(DarlingDataReader.TopProceduresSql), 2)] + [InlineData(nameof(DarlingDataReader.TopProceduresHourlySql), 1)] + public void EveryStatement_CarriesTheListPredicate_InEveryPass(string statement, int passes) + { + var sql = (string)typeof(DarlingDataReader).GetField(statement, System.Reflection.BindingFlags.Public | System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Static)!.GetValue(null)!; + Assert.DoesNotContain("$5::text IS NULL", sql, StringComparison.Ordinal); + Assert.Equal(passes, System.Text.RegularExpressions.Regex.Matches(sql, System.Text.RegularExpressions.Regex.Escape("($5::text[] IS NULL OR database_name = ANY($5))")).Count); + } + + [Fact] + public void TheQueryStoreStatements_CarryTheListPredicate_OnEveryRoute() + { + foreach (var (name, sql) in new[] + { + ("QueryStoreTopSql", DarlingDataReader.QueryStoreTopSql), + ("QueryStoreTopTableSql", DarlingDataReader.QueryStoreTopTableSql), + ("QueryStoreTopDailyTableSql", DarlingDataReader.QueryStoreTopDailyTableSql), + }) + { + Assert.DoesNotContain("$5::text IS NULL", sql, StringComparison.Ordinal); + Assert.True( + System.Text.RegularExpressions.Regex.Matches(sql, System.Text.RegularExpressions.Regex.Escape("($5::text[] IS NULL OR database_name = ANY($5))")).Count >= 1, + name + " must carry the list predicate"); + } + + Assert.Contains("($2::text[] IS NULL OR database_name = ANY($2))", DarlingRuntimePrecondition.LatestQueryStoreStatesSql, StringComparison.Ordinal); + } + + private static async Task WithSharedStoreAsync( + string serverName, Func body, string cleanupSql) + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), "Set DARLING_TEST_PG to run the live top-reads database filter tests."); + var ct = TestContext.Current.CancellationToken; + var serverId = ServerIdHelper.GetDeterministicHashCode(serverName); + var cleanup = string.Format(System.Globalization.CultureInfo.InvariantCulture, cleanupSql, serverId) + + string.Format(System.Globalization.CultureInfo.InvariantCulture, "; DELETE FROM servers WHERE server_id = {0}", serverId); + + using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DarlingMcpTestData.ExecAsync(connection, ct, cleanup); + + await using var postgres = NpgsqlDataSource.Create(connectionString!); + var succeeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, serverId, serverName, ct); + await body(connection, postgres, serverId, serverName, DarlingMcpTestData.Naive(DateTime.UtcNow), ct); + succeeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, succeeded, async (cleanupConnection, cleanupCt) => + await DarlingMcpTestData.ExecAsync(cleanupConnection, cleanupCt, cleanup)); + } + } +} + +/* #1776 own-store: this class mints its own scratch database through ScratchPostgres for every test and never touches + another test's rows, so it is deliberately NOT [Collection("live-postgres")] (it also stops the scratch database's + TimescaleDB background workers, which the shared store must never inherit). */ + +/// +/// #5245: the other half of : the hourly rollup tier of the queries and +/// procedures reads (a TimescaleDB scratch store; the raw rows are purged after the rollup refreshes, so the router reads +/// the rollup), and get_query_store_top's interval-table route and daily-summary route. +/// +public sealed class TopQueriesDatabaseFilterScratchLiveTests +{ + private const int ServerId = -943852; + private const string ServerName = "a5245-top-filter-hourly"; + private const string A = "TopFilterA"; + private const string B = "TopFilterB"; + private const string C = "TopFilterC"; + + private static readonly DateTime WindowStart = new(2026, 1, 5, 0, 0, 0, DateTimeKind.Unspecified); + private static readonly DateTime SurvivorAt = WindowStart.AddHours(5); + + private static List Databases(IEnumerable names) => names.Distinct().OrderBy(n => n, StringComparer.Ordinal).ToList(); + + private static DatabaseFilter Of(params string[] names) => DatabaseFilter.Of(names); + + [Fact] + public async Task TheHourlyTier_AListOfDatabases_ReturnsOnlyThoseDatabases_ForQueriesAndProcedures() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live hourly database filter test."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var timescaleEnabled = await TimescaleSupport.TryEnableAsync(connection, null, ct); + Assert.SkipWhen(!timescaleEnabled, "The live hourly database filter test needs TimescaleDB."); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + Assert.True(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, null, ct)); + await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection)) + { + await stop.ExecuteNonQueryAsync(ct); + } + + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, ct); + + var bodySucceeded = false; + try + { + /* C is the hottest, so an unfiltered hourly read would put it first. */ + var windowEnd = WindowStart.AddDays(1); + var at = WindowStart.AddHours(1); + foreach (var (db, weight) in new[] { (A, 3_000L), (B, 2_000L), (C, 9_000L) }) + { + at = at.AddMinutes(10); + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO collect.query_stats (collection_id, collection_time, server_id, server_name, database_name, query_hash, sql_handle, + delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, total_logical_reads, sample_interval_seconds) + VALUES ($1, $2, $3, $4, $5, $6, $7, 10, $8, $8, 100, 1000, 3600)", + CollectionIdGenerator.Next(), at, ServerId, ServerName, db, "0xQ" + db, "0xSQLH" + db, weight); + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO collect.procedure_stats (collection_id, collection_time, server_id, server_name, database_name, schema_name, object_name, sql_handle, + delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, total_logical_reads, sample_interval_seconds) + VALUES ($1, $2, $3, $4, $5, 'dbo', $6, $7, 10, $8, $8, 100, 1000, 3600)", + CollectionIdGenerator.Next(), at, ServerId, ServerName, db, "usp_" + db, "0xSQLH" + db, weight); + } + + /* A survivor in raw keeps the raw floor later than the window start, so the router sends the read to the rollup. */ + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO collect.query_stats (collection_id, collection_time, server_id, server_name, database_name, query_hash, sql_handle, + delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, total_logical_reads, sample_interval_seconds) + VALUES ($1, $2, $3, $4, $5, '0xQSURV', '0xSQLHSURV', 1, 1000, 1000, 7, 1000, 3600)", + CollectionIdGenerator.Next(), SurvivorAt, ServerId, ServerName, A); + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO collect.procedure_stats (collection_id, collection_time, server_id, server_name, database_name, schema_name, object_name, sql_handle, + delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, total_logical_reads, sample_interval_seconds) + VALUES ($1, $2, $3, $4, $5, 'dbo', 'usp_SURV', '0xSQLHSURV', 1, 1000, 1000, 7, 1000, 3600)", + CollectionIdGenerator.Next(), SurvivorAt, ServerId, ServerName, A); + + foreach (var view in new[] { TimescaleSupport.QueryStatsIntervalHourlyView, TimescaleSupport.ProcedureStatsIntervalHourlyView }) + { + await using var refresh = new NpgsqlCommand($"CALL refresh_continuous_aggregate('collect.{view}'::regclass, $1::timestamp, $2::timestamp)", connection); + refresh.Parameters.AddWithValue(WindowStart); + refresh.Parameters.AddWithValue(windowEnd.AddHours(1)); + await refresh.ExecuteNonQueryAsync(ct); + } + + foreach (var table in new[] { "collect.query_stats", "collect.procedure_stats" }) + { + await using var purge = new NpgsqlCommand($"DELETE FROM {table} WHERE server_id = $1 AND collection_time >= $2 AND collection_time < $3", connection); + purge.Parameters.AddWithValue(ServerId); + purge.Parameters.AddWithValue(WindowStart); + purge.Parameters.AddWithValue(WindowStart.AddHours(3)); + await purge.ExecuteNonQueryAsync(ct); + } + + await using var hourly = NpgsqlDataSource.Create(scratch.ConnectionString); + + var all = await DarlingDataReader.GetTopQueriesByCpuRoutedAsync(hourly, ServerId, WindowStart, windowEnd, 10, DatabaseFilter.All, ranking: TopRanking.Cpu, cancellationToken: ct); + Assert.True(all.Tier == RetentionTier.Hourly, $"queries: expected the hourly tier but read {all.Tier}"); + Assert.Equal([A, B, C], Databases(all.Rows.Select(r => r.DatabaseName))); + + var two = await DarlingDataReader.GetTopQueriesByCpuRoutedAsync(hourly, ServerId, WindowStart, windowEnd, 10, Of(A, B), ranking: TopRanking.Cpu, cancellationToken: ct); + Assert.Equal(RetentionTier.Hourly, two.Tier); + Assert.Equal([A, B], Databases(two.Rows.Select(r => r.DatabaseName))); + Assert.DoesNotContain(two.Rows, r => r.QueryHash == "0xQ" + C); + + var one = await DarlingDataReader.GetTopQueriesByCpuRoutedAsync(hourly, ServerId, WindowStart, windowEnd, 10, B, ranking: TopRanking.Cpu, cancellationToken: ct); + Assert.Equal([B], Databases(one.Rows.Select(r => r.DatabaseName))); + Assert.Equal(one.Rows.Select(r => r.QueryHash), + (await DarlingDataReader.GetTopQueriesByCpuRoutedAsync(hourly, ServerId, WindowStart, windowEnd, 10, DatabaseFilter.One(B), ranking: TopRanking.Cpu, cancellationToken: ct)).Rows.Select(r => r.QueryHash)); + + var procedures = await DarlingDataReader.GetTopProceduresByCpuRoutedAsync(hourly, ServerId, WindowStart, windowEnd, 10, Of(A, B), TopRanking.Cpu, ct); + Assert.True(procedures.Tier == RetentionTier.Hourly, $"procedures: expected the hourly tier but read {procedures.Tier}"); + Assert.Equal([A, B], Databases(procedures.Rows.Select(r => r.DatabaseName))); + Assert.Equal([A, B, C], Databases((await DarlingDataReader.GetTopProceduresByCpuRoutedAsync(hourly, ServerId, WindowStart, windowEnd, 10, DatabaseFilter.All, TopRanking.Cpu, ct)).Rows.Select(r => r.DatabaseName))); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (cleanup, cleanupCt) => + { + await using var probe = new NpgsqlCommand( + "SELECT count(*) FROM pg_catalog.pg_stat_activity WHERE datname = pg_catalog.current_database() " + + "AND backend_type LIKE 'TimescaleDB Background Worker Scheduler%'", cleanup); + Assert.Equal(0L, Convert.ToInt64(await probe.ExecuteScalarAsync(cleanupCt))); + }); + } + } + + /// + /// The hourly route's refill (#5313 with #5245): twelve WAITFOR shells (six in A, six in B) outrank every real group, so + /// round one (top + 5 = 8 candidates) holds only shells and TopFill asks again with a larger candidate limit. C's real + /// groups outrank the real groups of A and B, so a refill round that dropped the filter would fill the page from C. + /// Every round of the hourly statement binds the list ($5), so the page is A's and B's real groups; the unfiltered + /// control shows the same refill filling the page from C, which is what the filter has to stop. + /// + [Fact] + public async Task TheHourlyTier_ARefillRound_UnderTheFilter_NeverReturnsAnUnchosenDatabase() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live hourly refill database filter test."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var timescaleEnabled = await TimescaleSupport.TryEnableAsync(connection, null, ct); + Assert.SkipWhen(!timescaleEnabled, "The live hourly refill database filter test needs TimescaleDB."); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + Assert.True(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, null, ct)); + await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection)) + { + await stop.ExecuteNonQueryAsync(ct); + } + + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, ct); + + var bodySucceeded = false; + try + { + var windowEnd = WindowStart.AddDays(1); + const string WaitforText = "WAITFOR DELAY '00:00:30'"; + var tick = 0; + + async Task PlantAsync(DateTime at, string db, string tag, string text, long weight) => + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO collect.query_stats (collection_id, collection_time, server_id, server_name, database_name, query_hash, sql_handle, query_text, + delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, total_logical_reads, sample_interval_seconds) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, 10, $9, $9, 100, 1000, 3600)", + CollectionIdGenerator.Next(), at, ServerId, ServerName, db, "0xQ" + tag, "0xSQLH" + tag, text, weight); + + foreach (var db in new[] { A, B }) + { + for (var i = 1; i <= 6; i++) + { + tick++; + var tag = "Shell" + db + i; + await PlantAsync(WindowStart.AddHours(1).AddSeconds(tick), db, tag, WaitforText, 1_000_000 - tick); + /* The text lookup reads raw and the purge below removes the first hours: this later row keeps the shell's text findable. */ + await PlantAsync(SurvivorAt.AddSeconds(tick), db, tag, WaitforText, 1); + } + + for (var i = 1; i <= 2; i++) + { + tick++; + await PlantAsync(WindowStart.AddHours(1).AddSeconds(tick), db, "Real" + db + i, "SELECT real " + db + i, 10_000 - tick * 100); + } + } + + for (var i = 1; i <= 6; i++) + { + tick++; + await PlantAsync(WindowStart.AddHours(1).AddSeconds(tick), C, "Real" + C + i, "SELECT real " + C + i, 500_000 - tick); + } + + await using (var refresh = new NpgsqlCommand($"CALL refresh_continuous_aggregate('collect.{TimescaleSupport.QueryStatsIntervalHourlyView}'::regclass, $1::timestamp, $2::timestamp)", connection)) + { + refresh.Parameters.AddWithValue(WindowStart); + refresh.Parameters.AddWithValue(windowEnd.AddHours(1)); + await refresh.ExecuteNonQueryAsync(ct); + } + + await using (var purge = new NpgsqlCommand("DELETE FROM collect.query_stats WHERE server_id = $1 AND collection_time >= $2 AND collection_time < $3", connection)) + { + purge.Parameters.AddWithValue(ServerId); + purge.Parameters.AddWithValue(WindowStart); + purge.Parameters.AddWithValue(WindowStart.AddHours(3)); + await purge.ExecuteNonQueryAsync(ct); + } + + await using var hourly = NpgsqlDataSource.Create(scratch.ConnectionString); + + var filtered = await DarlingDataReader.GetTopQueriesByCpuRoutedAsync(hourly, ServerId, WindowStart, windowEnd, 3, Of(A, B), ranking: TopRanking.Cpu, cancellationToken: ct); + Assert.True(filtered.Tier == RetentionTier.Hourly, $"expected the hourly tier but read {filtered.Tier}"); + Assert.True(filtered.Rows.Count == 3, $"{filtered.Rows.Count} rows, expected 3 (the refill must reach past the shells)"); + Assert.DoesNotContain(filtered.Rows, r => (r.QueryText ?? "").StartsWith("WAITFOR", StringComparison.Ordinal)); + Assert.DoesNotContain(filtered.Rows, r => r.DatabaseName == C); + Assert.Equal([A, B], Databases(filtered.Rows.Select(r => r.DatabaseName))); + + /* Control: the same refill with no filter fills the page from C. */ + var unfiltered = await DarlingDataReader.GetTopQueriesByCpuRoutedAsync(hourly, ServerId, WindowStart, windowEnd, 3, DatabaseFilter.All, ranking: TopRanking.Cpu, cancellationToken: ct); + Assert.Equal(RetentionTier.Hourly, unfiltered.Tier); + Assert.All(unfiltered.Rows, r => Assert.Equal(C, r.DatabaseName)); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (cleanup, cleanupCt) => + { + await using var probe = new NpgsqlCommand( + "SELECT count(*) FROM pg_catalog.pg_stat_activity WHERE datname = pg_catalog.current_database() " + + "AND backend_type LIKE 'TimescaleDB Background Worker Scheduler%'", cleanup); + Assert.Equal(0L, Convert.ToInt64(await probe.ExecuteScalarAsync(cleanupCt))); + }); + } + } + + private static readonly DateTime QsEnd = new(2026, 1, 14, 6, 0, 0, DateTimeKind.Unspecified); + private static readonly DateTime QsBuildNow = new(2026, 1, 15, 12, 0, 0, DateTimeKind.Unspecified); + + /// Nine queries over db0, db1 and db2 (query q lives in db(q % 3)), one row per query per hour from 2026-01-05 to 2026-01-14 12:00. + private const string QsSeedSql = @" +INSERT INTO collect.query_store_interval_wide +(collection_time, server_id, database_name, query_id, plan_id, execution_type_desc, first_execution_time, last_execution_time, + module_name, query_hash, execution_count, avg_duration_us, avg_cpu_time_us, avg_logical_io_reads, avg_logical_io_writes, + avg_physical_io_reads, avg_rowcount, query_plan_hash, replica_role, runtime_stats_interval_id, interval_start_time_utc) +SELECT + ct, 1, 'db' || (q % 3), q, q, 'Regular', ct - interval '10 minutes', ct, NULL, + 'h' || q, 5, 1000 + q * 100, 500 + q, 100, 10, 5, 3, 'ph', NULL, q * 100000 + h, ct - interval '10 minutes' +FROM +( + SELECT q, h, TIMESTAMP '2026-01-05 00:00:00' + make_interval(hours => h) AS ct + FROM generate_series(1, 9) AS q + CROSS JOIN generate_series(0, 24 * 9 + 12) AS h +) AS s;"; + + [Fact] + public async Task QueryStoreTop_TheIntervalTableAndDailyRoutes_AListOfDatabases_ReturnsOnlyThoseDatabases() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the live Query Store routes database filter test."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await using var source = NpgsqlDataSource.Create(scratch.ConnectionString); + + var bodySucceeded = false; + try + { + await DarlingMcpTestData.ExecAsync(connection, ct, QsSeedSql); + await DarlingMcpTestData.ExecAsync(connection, ct, "INSERT INTO collect.servers (server_id, server_name, is_enabled) VALUES (1, 'srv1', TRUE)"); + await DarlingMcpTestData.ExecAsync(connection, ct, + "INSERT INTO collect.query_store_interval_wide_coverage (server_id, filled_since, applied_through) VALUES (1, TIMESTAMP '2026-01-01 00:00:00', TIMESTAMP '2026-01-01 00:00:00')"); + + var start = QsEnd.AddHours(-72); + + /* No built day yet: the interval-table statement. */ + var table = await DarlingDataReader.GetQueryStoreTopWithReachAsync(source, 1, start, QsEnd, 30, Of("db0", "db1"), null, null, ct); + Assert.NotNull(table.Table); + Assert.Equal(0, table.DailyDaysUsed); + Assert.Equal(["db0", "db1"], Databases(table.Rows.Select(r => r.DatabaseName))); + Assert.Equal(6, table.Rows.Count); + Assert.Equal(["db2"], Databases((await DarlingDataReader.GetQueryStoreTopWithReachAsync(source, 1, start, QsEnd, 30, "db2", null, null, ct)).Rows.Select(r => r.DatabaseName))); + Assert.Equal(["db0", "db1", "db2"], Databases((await DarlingDataReader.GetQueryStoreTopWithReachAsync(source, 1, start, QsEnd, 30, DatabaseFilter.All, null, null, ct)).Rows.Select(r => r.DatabaseName))); + + /* Built days: the daily-summary statement. */ + var result = await QueryStoreTopDaily.RunTickAsync(source, QsBuildNow, 9, NullLogger.Instance, ct); + Assert.Equal(0, result.Failed); + var daily = await DarlingDataReader.GetQueryStoreTopWithReachAsync(source, 1, start, QsEnd, 30, Of("db0", "db1"), null, null, ct); + Assert.True(daily.DailyDaysUsed > 0, "the read must use built days"); + Assert.Equal(["db0", "db1"], Databases(daily.Rows.Select(r => r.DatabaseName))); + Assert.Equal(6, daily.Rows.Count); + Assert.Equal(["db0", "db2"], Databases((await DarlingDataReader.GetQueryStoreTopWithReachAsync(source, 1, start, QsEnd, 30, Of("db2", "db0"), null, null, ct)).Rows.Select(r => r.DatabaseName))); + var oneDaily = await DarlingDataReader.GetQueryStoreTopWithReachAsync(source, 1, start, QsEnd, 30, DatabaseFilter.One("db1"), null, null, ct); + Assert.Equal(["db1"], Databases(oneDaily.Rows.Select(r => r.DatabaseName))); + Assert.Equal(oneDaily.Rows.Select(r => r.QueryId), + (await DarlingDataReader.GetQueryStoreTopWithReachAsync(source, 1, start, QsEnd, 30, "db1", null, null, ct)).Rows.Select(r => r.QueryId)); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } +} diff --git a/Darling/Darling.Tests/TopQueriesDetailLiveTests.cs b/Darling/Darling.Tests/TopQueriesDetailLiveTests.cs new file mode 100644 index 0000000000..d281f02a5d --- /dev/null +++ b/Darling/Darling.Tests/TopQueriesDetailLiveTests.cs @@ -0,0 +1,167 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Text; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Darling.Tests; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace PerformanceMonitor.Darling.Tests; + +/// +/// The widened get_top_queries_by_cpu against live PostgreSQL: detail='full' adds the desktop Top Queries grid's remaining +/// fields from query_stats, omits the ones the store holds no value for, and the default call stays as lean as before. +/// A 20-row, 168-hour call stays under the shared response-size target in both forms. +/// +[Collection("live-postgres")] +public sealed class TopQueriesDetailLiveTests +{ + private const string ServerName = "darling-top-queries-detail-e2e"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + + [Fact] + public async Task FullDetail_CarriesTheDesktopFields_OmitsNulls_AndStaysUnderTheBudgetAt168Hours() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), "Set DARLING_TEST_PG to run the live top-queries detail test."); + var ct = TestContext.Current.CancellationToken; + + using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await CleanupAsync(connection, ct); + + await using var postgres = NpgsqlDataSource.Create(connectionString!); + var succeeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + /* A non-zero offset on purpose: at 0 the conversion asserted below would pass whether or not it ran. */ + await DarlingMcpTestData.ExecAsync(connection, ct, + "INSERT INTO server_properties (collection_id, collection_time, server_id, server_name, utc_offset_minutes) " + + $"VALUES ({CollectionIdGenerator.Next()}, now() AT TIME ZONE 'UTC', {ServerId}, '{ServerName}', -240)"); + var now = DarlingMcpTestData.Naive(DateTime.UtcNow); + var lastExec = new DateTime(2026, 3, 4, 5, 6, 7, DateTimeKind.Unspecified); + var created = new DateTime(2026, 3, 1, 2, 3, 4, DateTimeKind.Unspecified); + + /* One fully populated group (two collections, so the extremes and the max-of columns aggregate) and, below it, + twenty-two lean groups holding none of the detail columns: the null-omission case and the page size. */ + await PlantAsync(connection, ct, now.AddHours(-3), "0xFULLHASH", 500, full: true, lastExec, created, 2, 4); + await PlantAsync(connection, ct, now.AddHours(-2), "0xFULLHASH", 400, full: true, lastExec, created, 1, 8); + for (var i = 0; i < 22; i++) + { + await PlantAsync(connection, ct, now.AddHours(-1).AddMinutes(-i), "0xLEAN" + i.ToString("D2"), 10 + i, full: false, null, null, 1, 1); + } + + var window = (now.AddDays(-7), now.AddMinutes(5)); + var rows = await DarlingDataReader.GetTopQueriesByCpuAsync(postgres, ServerId, window.Item1, window.Item2, top: 20, databaseName: null, cancellationToken: ct); + var detail = rows.First(r => r.QueryHash == "0xFULLHASH").Detail; + Assert.NotNull(detail); + /* Stored on the monitored server's clock (UTC-4); the read hands back naive UTC. */ + Assert.Equal(lastExec.AddHours(4), detail!.LastExecutionTime); + Assert.Equal(created.AddHours(4), detail.CreationTime); + Assert.Equal(7L, detail.MinGrantKb); + Assert.Equal(900L, detail.MaxGrantKb); + Assert.Equal(1L, detail.MinReservedThreads); + Assert.Equal(8L, detail.MaxUsedThreads); + Assert.Equal(3L, detail.PlanGenerationNum); + Assert.Equal(1000L, detail.TotalClrTimeUs); + Assert.NotNull(detail.WorkerTimePerSecond); + Assert.Null(rows.First(r => r.QueryHash == "0xLEAN21").Detail!.MaxGrantKb); + + var summary = await DarlingMcpDataTools.GetTopQueriesByCpu(postgres, ServerName, hours_back: 168, top: 20); + var full = await DarlingMcpDataTools.GetTopQueriesByCpu(postgres, ServerName, hours_back: 168, top: 20, detail: "full"); + Console.WriteLine($"get_top_queries_by_cpu 168h top 20 (short query text): summary={Encoding.UTF8.GetByteCount(summary)} bytes, full={Encoding.UTF8.GetByteCount(full)} bytes"); + Assert.True(Encoding.UTF8.GetByteCount(summary) < McpResponseBudget.DefaultBytes); + + using (var s = JsonDocument.Parse(summary)) + { + var q = s.RootElement.GetProperty("queries").EnumerateArray().First(x => x.GetProperty("query_hash").GetString() == "0xFULLHASH"); + Assert.False(q.TryGetProperty("max_grant_kb", out _)); + Assert.False(q.TryGetProperty("last_execution_time", out _)); + } + + using var f = JsonDocument.Parse(full); + var queries = f.RootElement.GetProperty("queries").EnumerateArray().ToList(); + var fq = queries.First(x => x.GetProperty("query_hash").GetString() == "0xFULLHASH"); + Assert.Equal("2026-03-04T09:06:07.0000000", fq.GetProperty("last_execution_time").GetString()); + Assert.Equal("2026-03-01T06:03:04.0000000", fq.GetProperty("creation_time").GetString()); + Assert.Equal(900, fq.GetProperty("max_grant_kb").GetInt64()); + Assert.Equal(1.0, fq.GetProperty("total_clr_ms").GetDouble(), 3); + Assert.Equal(3, fq.GetProperty("plan_generation_num").GetInt64()); + Assert.Equal(5_000_000_000L, fq.GetProperty("max_physical_reads").GetInt64()); + var lean = queries.First(x => x.GetProperty("query_hash").GetString() == "0xLEAN21"); + Assert.False(lean.TryGetProperty("max_grant_kb", out _)); + Assert.False(lean.TryGetProperty("last_execution_time", out _)); + Assert.True(Encoding.UTF8.GetByteCount(full) < McpResponseBudget.DefaultBytes); + + /* A fully populated row (every detail field present) against the same row without them: the per-row cost of + detail='full' on a real SQL Server target, where the grant and thread columns are populated. */ + using var sd = JsonDocument.Parse(summary); + var sRow = sd.RootElement.GetProperty("queries").EnumerateArray().First(x => x.GetProperty("query_hash").GetString() == "0xFULLHASH"); + var perRow = Encoding.UTF8.GetByteCount(fq.GetRawText()) - Encoding.UTF8.GetByteCount(sRow.GetRawText()); + Console.WriteLine($"fully populated row: detail='full' adds {perRow} bytes per row; 20 rows add {perRow * 20} bytes"); + Assert.InRange(perRow, 250, 900); + + /* group_by=host_object runs the second statement: the same detail fields, same conversion. */ + var rolled = await DarlingMcpDataTools.GetTopQueriesByCpu(postgres, ServerName, hours_back: 168, top: 20, group_by: "host_object", detail: "full"); + using (var r = JsonDocument.Parse(rolled)) + { + var rq = r.RootElement.GetProperty("queries").EnumerateArray().First(x => x.GetProperty("query_hash").GetString() == "0xFULLHASH"); + Assert.Equal("2026-03-04T09:06:07.0000000", rq.GetProperty("last_execution_time").GetString()); + Assert.Equal(900, rq.GetProperty("max_grant_kb").GetInt64()); + } + + var refused = await DarlingMcpDataTools.GetTopQueriesByCpu(postgres, ServerName, hours_back: 168, top: 20, detail: "everything"); + Assert.Contains("detail must be", refused, StringComparison.Ordinal); + succeeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, succeeded, async (cleanup, cleanupCt) => + await CleanupAsync(cleanup, cleanupCt)); + } + } + + private static async Task PlantAsync( + NpgsqlConnection connection, CancellationToken ct, DateTime at, string queryHash, long weight, bool full, + DateTime? lastExec, DateTime? created, int minThreads, int maxThreads) + { + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO query_stats (collection_id, collection_time, server_id, server_name, database_name, + query_hash, query_plan_hash, sql_handle, plan_handle, query_text, + delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, + min_dop, max_dop, sample_interval_seconds, + last_execution_time, creation_time, min_grant_kb, max_grant_kb, + min_reserved_threads, max_reserved_threads, min_used_threads, max_used_threads, + total_clr_time, plan_generation_num, max_physical_reads) + VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28)", + CollectionIdGenerator.Next(), at, ServerId, ServerName, "StackOverflow", + queryHash, "0xPLAN", "0xSQLH" + queryHash, "0xPLANH", "SELECT 1", + weight, weight * 1000L, weight * 2000L, weight * 10L, 1, 1, 60, + (object?)lastExec ?? DBNull.Value, (object?)created ?? DBNull.Value, + full ? 7L : DBNull.Value, full ? (weight == 500 ? 900L : 800L) : DBNull.Value, + full ? (long)minThreads : DBNull.Value, full ? (long)maxThreads : DBNull.Value, + full ? (long)minThreads : DBNull.Value, full ? (long)maxThreads : DBNull.Value, + full ? (weight == 500 ? 1000L : 500L) : DBNull.Value, full ? 3L : DBNull.Value, + full ? 5_000_000_000L : DBNull.Value); + } + + private static async Task CleanupAsync(NpgsqlConnection connection, CancellationToken ct) => + await DarlingMcpTestData.ExecAsync(connection, ct, + $"DELETE FROM query_stats WHERE server_id = {ServerId}; DELETE FROM servers WHERE server_id = {ServerId}"); +} diff --git a/Darling/Darling.Tests/TopQueriesHourlyRoutingLiveTests.cs b/Darling/Darling.Tests/TopQueriesHourlyRoutingLiveTests.cs index 9eeaed1791..a2f70581f2 100644 --- a/Darling/Darling.Tests/TopQueriesHourlyRoutingLiveTests.cs +++ b/Darling/Darling.Tests/TopQueriesHourlyRoutingLiveTests.cs @@ -39,6 +39,11 @@ public sealed class TopQueriesHourlyRoutingLiveTests /// once we delete raw's rows for it, which is what actually drives the router, not calendar time. private static readonly DateTime WindowStart = new(2026, 1, 5, 0, 0, 0, DateTimeKind.Unspecified); + /// #4966: effective_start prints in UTC with the trailing Z on every path, a floor read off the + /// store included. The floors these tests plant are naive store instants, so the text expected names the same + /// instant as UTC (written out here, not through the product's formatter). + private static string UtcText(DateTime naive) => DateTime.SpecifyKind(naive, DateTimeKind.Utc).ToString("o"); + [Fact] public async Task GetTopQueriesByCpuRoutedAsync_RoutesToHourlyOnceRawIsPurged_AndTotalsAgree() { @@ -239,7 +244,7 @@ public async Task GetTopQueriesByCpu_AgedWindow_WithMinDopOrRollUp_ReadsRaw_AndD var root = doc.RootElement; Assert.Equal("raw", root.GetProperty("tier_used").GetString()); Assert.True(root.GetProperty("window_truncated").GetBoolean()); - Assert.Equal(DarlingMcpTestData.TruncateToSeconds(survivor).ToString("o"), root.GetProperty("effective_start").GetString()); + Assert.Equal(UtcText(DarlingMcpTestData.TruncateToSeconds(survivor)), root.GetProperty("effective_start").GetString()); Assert.Contains("stayed on raw", root.GetProperty("precision_note").GetString()); } @@ -253,6 +258,11 @@ public async Task GetTopQueriesByCpu_AgedWindow_WithMinDopOrRollUp_ReadsRaw_AndD using var emptyDoc = System.Text.Json.JsonDocument.Parse(await DarlingMcpDataTools.GetTopQueriesByCpu( hourlyDataSource, ServerName, hours_back: hoursBack, top: 10, min_dop: 5, as_of: asOf)); Assert.Contains("still holds (from", emptyDoc.RootElement.GetProperty("message").GetString() ?? emptyDoc.RootElement.ToString()); + + /* #4966: the miss message names the floor in the text effective_start prints beside it (UTC, with the Z). */ + var missStart = emptyDoc.RootElement.GetProperty("hints").GetProperty("effective_start").GetString()!; + Assert.EndsWith("Z", missStart, StringComparison.Ordinal); + Assert.Contains("still holds (from " + missStart + ";", emptyDoc.RootElement.GetProperty("message").GetString(), StringComparison.Ordinal); Assert.True(emptyDoc.RootElement.ToString().Contains("window_truncated", StringComparison.Ordinal)); bodySucceeded = true; @@ -333,6 +343,14 @@ public async Task HourlyRouted_MissingColumnsAreNull_AndSqlHandleIsCarried() } Assert.Equal("0xTOPQ1", row.GetProperty("sql_handle").GetString()); + + /* detail=full on the rollup: no detail field exists there, and the note says so. */ + using var fullDoc = System.Text.Json.JsonDocument.Parse(await DarlingMcpDataTools.GetTopQueriesByCpu( + hourlyDataSource, ServerName, hours_back: hoursBack, top: 10, as_of: asOf, detail: "full")); + Assert.Equal("hourly", fullDoc.RootElement.GetProperty("tier_used").GetString()); + Assert.Contains("detail=full was asked", fullDoc.RootElement.GetProperty("precision_note").GetString()!, StringComparison.Ordinal); + Assert.False(fullDoc.RootElement.GetProperty("queries")[0].TryGetProperty("max_grant_kb", out _)); + Assert.DoesNotContain("detail=full was asked", doc.RootElement.GetProperty("precision_note").GetString()!, StringComparison.Ordinal); Assert.Equal(System.Text.Json.JsonValueKind.String, row.GetProperty("text_note").ValueKind); Assert.Contains("may be a WAITFOR shell", row.GetProperty("text_note").GetString(), StringComparison.Ordinal); @@ -404,7 +422,7 @@ public async Task HourlyRouted_YoungServer_ReportsWindowTruncated() hourlyDataSource, ServerName + "-young", hours_back: hoursBack, top: 10, as_of: asOf)); Assert.Equal("hourly", young.RootElement.GetProperty("tier_used").GetString()); Assert.True(young.RootElement.GetProperty("window_truncated").GetBoolean()); - Assert.Equal(WindowStart.AddHours(10).ToString("o"), young.RootElement.GetProperty("effective_start").GetString()); + Assert.Equal(UtcText(WindowStart.AddHours(10)), young.RootElement.GetProperty("effective_start").GetString()); Assert.Contains("hourly rollup", young.RootElement.GetProperty("truncation_note").GetString()); using var covered = System.Text.Json.JsonDocument.Parse(await DarlingMcpDataTools.GetTopQueriesByCpu( @@ -494,16 +512,16 @@ and an ordinary row at +13h. Its first bucket at or after the stitch floor must using var both = System.Text.Json.JsonDocument.Parse(await DarlingMcpDataTools.GetTopQueriesByCpu( hourlyDataSource, ServerName, hours_back: hoursBack, top: 10, as_of: asOf)); Assert.Equal("hourly", both.RootElement.GetProperty("tier_used").GetString()); - Assert.Equal(WindowStart.AddHours(1).ToString("o"), both.RootElement.GetProperty("effective_start").GetString()); + Assert.Equal(UtcText(WindowStart.AddHours(1)), both.RootElement.GetProperty("effective_start").GetString()); using var successorOnly = System.Text.Json.JsonDocument.Parse(await DarlingMcpDataTools.GetTopQueriesByCpu( hourlyDataSource, ServerName + "-successor", hours_back: hoursBack, top: 10, as_of: asOf)); Assert.Equal("hourly", successorOnly.RootElement.GetProperty("tier_used").GetString()); - Assert.Equal(WindowStart.AddHours(12).ToString("o"), successorOnly.RootElement.GetProperty("effective_start").GetString()); + Assert.Equal(UtcText(WindowStart.AddHours(12)), successorOnly.RootElement.GetProperty("effective_start").GetString()); using var legacyOnly = System.Text.Json.JsonDocument.Parse(await DarlingMcpDataTools.GetTopQueriesByCpu( hourlyDataSource, ServerName + "-legacyonly", hours_back: hoursBack, top: 10, as_of: asOf)); - Assert.Equal(WindowStart.AddHours(13).ToString("o"), legacyOnly.RootElement.GetProperty("effective_start").GetString()); + Assert.Equal(UtcText(WindowStart.AddHours(13)), legacyOnly.RootElement.GetProperty("effective_start").GetString()); bodySucceeded = true; } @@ -568,7 +586,8 @@ public async Task HourlyRouted_EndBeyondTheMaterializationCeiling_SaysNothingAft hourlyDataSource, ServerName, hours_back: hoursBack, top: 10, as_of: asOf)); Assert.Equal("hourly", doc.RootElement.GetProperty("tier_used").GetString()); var note = doc.RootElement.GetProperty("precision_note").GetString()!; - Assert.Contains("the hourly rollup is materialized only to " + WindowStart.AddHours(2).ToString("o") + "; nothing after it was read", note, StringComparison.Ordinal); + /* #4966: the instants the note names are UTC with the Z, as effective_start is. */ + Assert.Contains("the hourly rollup is materialized only to " + PerformanceMonitor.Common.McpHelpers.FormatEffectiveStart(WindowStart.AddHours(2)) + "; nothing after it was read", note, StringComparison.Ordinal); Assert.DoesNotContain("included whole", note, StringComparison.Ordinal); /* An end cut is not a start cut: the window flag stays about the start. */ Assert.False(doc.RootElement.GetProperty("window_truncated").GetBoolean()); diff --git a/Darling/Darling.Tests/TopQueriesHourlyRoutingTests.cs b/Darling/Darling.Tests/TopQueriesHourlyRoutingTests.cs index a15a33ce8e..4b90986f88 100644 --- a/Darling/Darling.Tests/TopQueriesHourlyRoutingTests.cs +++ b/Darling/Darling.Tests/TopQueriesHourlyRoutingTests.cs @@ -10,6 +10,7 @@ using System.IO; using System.Runtime.CompilerServices; using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; using Xunit; namespace Darling.Tests; @@ -36,7 +37,9 @@ public void TopQueriesHourlySql_GroupsByDatabaseAndQueryHash_RanksByWorkerTimeSu Assert.Contains("GROUP BY database_name, query_hash", sql, StringComparison.Ordinal); Assert.DoesNotContain("host_object_name", sql, StringComparison.Ordinal); - Assert.Contains("ORDER BY SUM(worker_time_sum) DESC", sql, StringComparison.Ordinal); + /* The ranking is the const's anchor; the CPU read is the const expanded to the worker-time sum (#5226). */ + Assert.Contains("ORDER BY rank_metric DESC NULLS LAST", sql, StringComparison.Ordinal); + Assert.Contains("SUM(worker_time_sum) AS rank_metric", TopRankings.Apply(sql, TopRanking.Cpu, hourly: true), StringComparison.Ordinal); Assert.Contains(DarlingDataReader.TopQueriesHourlyFromPlaceholder, sql, StringComparison.Ordinal); } @@ -80,12 +83,17 @@ public void TopQueriesHourlySql_NeverNamesRollupRelationDirectly() public void TopQueriesHourlySql_LooksUpTextInStatement_AndSelectsNoDeltaExtremes() { var sql = DarlingDataReader.TopQueriesHourlySql; - Assert.Contains("LEFT JOIN LATERAL", sql, StringComparison.Ordinal); + /* #5309: one text lookup, probed once per key by index, not one per row and not a read of the window's raw rows. It has two + stages, each one lateral: latest_in_window for the ranked keys, then (#5299 round 2, N1) latest_any for the keys the + window had no text for. */ + Assert.Equal(2, System.Text.RegularExpressions.Regex.Matches(sql, "LATERAL").Count); + Assert.Contains("latest_in_window AS (", sql, StringComparison.Ordinal); Assert.Contains("MAX(sql_handle)", sql, StringComparison.Ordinal); Assert.Contains("NOT LIKE 'WAITFOR%'", sql, StringComparison.Ordinal); - Assert.Contains("LIMIT $4 + 5", sql, StringComparison.Ordinal); + Assert.Contains("LIMIT $6", sql, StringComparison.Ordinal); /* #5313: the candidate limit; the ceiling moved to $7 */ + Assert.DoesNotContain("+ 5", sql, StringComparison.Ordinal); Assert.DoesNotContain("worker_time_min", sql, StringComparison.Ordinal); - Assert.DoesNotContain("$6", sql, StringComparison.Ordinal); + Assert.DoesNotContain("$7", sql, StringComparison.Ordinal); /* the ceiling is a $CEIL$ placeholder, bound only when known */ Assert.Null(typeof(DarlingDataReader).GetField("TopQueriesHourlyTextLookupSql", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Static)); @@ -102,7 +110,7 @@ public void TopQueriesHourlySql_LooksUpTextInStatement_AndSelectsNoDeltaExtremes [Fact] public void HourlyFirstBucketSql_IsLeastOfTwoOrderedFirstRowProbes_WithNoUnion() { - var sql = DarlingDataReader.HourlyFirstBucketSql; + var sql = RollupCoverage.HourlyFirstBucketSql; Assert.Contains("least(", sql, StringComparison.Ordinal); Assert.Contains("$LEGACY$", sql, StringComparison.Ordinal); Assert.Contains("$SUCCESSOR$", sql, StringComparison.Ordinal); @@ -115,27 +123,39 @@ public void HourlyFirstBucketSql_IsLeastOfTwoOrderedFirstRowProbes_WithNoUnion() [Fact] public void HourlyFirstBucketSingleRelationSql_ReadsThePlaceholderRelation_OrderedWithLimitOne() { - var sql = DarlingDataReader.HourlyFirstBucketSingleRelationSql; + var sql = RollupCoverage.HourlyFirstBucketSingleRelationSql; Assert.Contains(DarlingDataReader.TopQueriesHourlyFromPlaceholder, sql, StringComparison.Ordinal); Assert.Contains("ORDER BY f.bucket LIMIT 1", sql, StringComparison.Ordinal); Assert.DoesNotContain("UNION", sql, StringComparison.OrdinalIgnoreCase); } - /// The seam splits at the same floor the stitch uses, and both hourly readers go through it. + /// The seam splits at the same floor the stitch uses, and the service's readers and the Viewer's grids go through + /// the ONE copy in Storage (#5329). [Fact] public void GetHourlyFirstBucketAsync_SplitsAtStitchFloor_AndBothReadersUseIt() { - var source = File.ReadAllText(FindReaderSourcePath()); - var start = source.IndexOf("private static async Task GetHourlyFirstBucketAsync(", StringComparison.Ordinal); + var storage = File.ReadAllText(Path.Combine(RepoRoot(), "Darling", "PerformanceMonitor.Darling.Storage", "TimescaleSupport.cs")); + var start = storage.IndexOf("public async Task GetHourlyFirstBucketAsync(", StringComparison.Ordinal); Assert.True(start >= 0); - var end = source.IndexOf("return value is DateTime bucket", start, StringComparison.Ordinal); - var body = source[start..end]; - Assert.Contains(".StitchFloor(", body, StringComparison.Ordinal); + var end = storage.IndexOf("return value is DateTime bucket", start, StringComparison.Ordinal); + var body = storage[start..end]; + var source = File.ReadAllText(FindReaderSourcePath()); + Assert.DoesNotContain("HourlyFirstBucketSql =", source, StringComparison.Ordinal); + Assert.DoesNotContain("Task GetHourlyFirstBucketAsync(", source, StringComparison.Ordinal); + foreach (var viewerFile in new[] { "ViewerDataService.QueryStats.cs", "ViewerDataService.ProcedureStats.cs" }) + { + Assert.Contains("coverage.GetHourlyFirstBucketAsync(", + File.ReadAllText(Path.Combine(RepoRoot(), "Darling", "PerformanceMonitor.Darling.Viewer", viewerFile)), StringComparison.Ordinal); + } + + Assert.Contains("StitchFloor(", body, StringComparison.Ordinal); Assert.Contains("SuccessorOf(", body, StringComparison.Ordinal); Assert.Contains("HourlyFirstBucketSingleRelationSql", body, StringComparison.Ordinal); Assert.Contains("\"UNION\"", body, StringComparison.Ordinal); - Assert.Contains("GetHourlyFirstBucketAsync(postgres, coverage, TimescaleSupport.QueryStatsHourlyView,", source, StringComparison.Ordinal); - Assert.Contains("GetHourlyFirstBucketAsync(postgres, coverage, TimescaleSupport.ProcedureStatsHourlyView,", source, StringComparison.Ordinal); + Assert.Contains("coverage.GetHourlyFirstBucketAsync(", source, StringComparison.Ordinal); + Assert.Contains("postgres, io ? TimescaleSupport.QueryStatsIoHourlyView : TimescaleSupport.QueryStatsHourlyView,", source, StringComparison.Ordinal); + /* #5329: the procedures arm picks its relation (io or the stitched legacy view) inside the call. */ + Assert.Contains("postgres, io ? TimescaleSupport.ProcedureStatsIoHourlyView : TimescaleSupport.ProcedureStatsHourlyView,", source, StringComparison.Ordinal); } private static string FindReaderSourcePath() @@ -143,4 +163,33 @@ private static string FindReaderSourcePath() private static string RepoRoot([CallerFilePath] string thisFile = "") => Path.GetFullPath(Path.Combine(Path.GetDirectoryName(thisFile)!, "..", "..")); + + /// + /// #5329 source pin: the routed read takes the io route through ChooseQueriesHourlyRoute, the hourly arm names + /// query_stats_io_hourly (and projects its three sums) only on that route, a reads ranking carries the raw + /// retention notice only when it does not read io, and the queries tool no longer says only raw carries reads. + /// + [Fact] + public void TheIoRoute_IsChosenInTheRoutedRead_NamedInTheHourlyArm_AndDisclosedByTheTool() + { + var source = File.ReadAllText(FindReaderSourcePath()); + var routed = source.Substring(source.IndexOf("public static async Task GetTopQueriesByCpuRoutedAsync", StringComparison.Ordinal)); + routed = routed[..routed.IndexOf("GetTopQueriesByCpuHourlyAsync(postgres, coverage", StringComparison.Ordinal)]; + Assert.Contains("ChooseQueriesHourlyRoute(rollups, coverage, ranking, startUtc)", routed, StringComparison.Ordinal); + Assert.Contains("ranking != TopRanking.Reads || ioRoute", routed, StringComparison.Ordinal); + Assert.DoesNotContain("!TopRankings.HourlyCarries(ranking)", routed, StringComparison.Ordinal); + + var arm = source.Substring(source.IndexOf("private static async Task<(List Rows, DateTime? FirstBucket)> GetTopQueriesByCpuHourlyAsync", StringComparison.Ordinal)); + arm = arm[..arm.IndexOf("/* ─────────────────────────── top procedures", StringComparison.Ordinal)]; + /* #5329: the io FROM goes through StitchedRelationSql like every other hourly relation (no literal relation name). */ + Assert.Contains("coverage.StitchedRelationSql(\n io ? TimescaleSupport.QueryStatsIoHourlyView : TimescaleSupport.QueryStatsHourlyView,", arm.Replace("\r\n", "\n"), StringComparison.Ordinal); + Assert.DoesNotContain("collect.{TimescaleSupport.QueryStatsIoHourlyView} AS f", arm, StringComparison.Ordinal); + Assert.Contains("io ? TimescaleSupport.QueryStatsIoHourlyView : TimescaleSupport.QueryStatsHourlyView,", arm, StringComparison.Ordinal); + Assert.Contains("coverage.GetHourlyFirstBucketAsync(", arm, StringComparison.Ordinal); + Assert.Contains("TopRankings.HourlyIoSums", arm, StringComparison.Ordinal); + + var tool = File.ReadAllText(Path.Combine(RepoRoot(), "Darling", "PerformanceMonitor.Darling.Service", "Mcp", "DarlingMcpDataTools.cs")); + Assert.DoesNotContain("which only raw query_stats carries (the hourly rollup keeps CPU", tool, StringComparison.Ordinal); + Assert.Contains("query_stats_io_hourly", tool, StringComparison.Ordinal); + } } diff --git a/Darling/Darling.Tests/TopQueriesIoRouteLiveTests.cs b/Darling/Darling.Tests/TopQueriesIoRouteLiveTests.cs new file mode 100644 index 0000000000..14061de1c7 --- /dev/null +++ b/Darling/Darling.Tests/TopQueriesIoRouteLiveTests.cs @@ -0,0 +1,354 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Darling.Tests; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace PerformanceMonitor.Darling.Tests; + +/* #1776 own-store: each test mints a scratch database, because it materializes continuous aggregates (and one drops + one), which no other class sharing the store may see. */ +/// +/// #5329 lane B1, live: the Top Queries reader on an Hourly-tier window with the io hourly rollup absent, empty, partial +/// and covering. The oracle seed is lane A's (a NULL-reads group, a reads tie, two +/// sql_handles under one query_hash, two databases, a zero-interval row the rollups drop and a NULL-interval row they keep). +/// +/// The oracle: over the SAME window, the reads ranking answered from query_stats_io_hourly equals the raw route +/// (keys, order and every total, the reads, physical reads and writes included), and the CPU, duration and executions rankings +/// answered from it equal the same ranking over the interval-honest sibling. Raw is read FIRST (its rows still exist), then +/// the raw rows are deleted so the router must read a rollup. Each read goes through a FRESH data source, because coverage +/// is cached per data source. +/// +public sealed class TopQueriesIoRouteLiveTests +{ + private const int ServerId = -943861; + private const string ServerName = "a5329-top-queries-io-route"; + private const int Top = 10; + + private static readonly DateTime WindowStart = new(2026, 1, 5, 0, 0, 0, DateTimeKind.Unspecified); + private static readonly DateTime WindowEnd = WindowStart.AddHours(IoRollupOracleSeed.Hours + 1); + private static readonly DateTime SurvivorAt = WindowStart.AddHours(IoRollupOracleSeed.Hours).AddMinutes(30); + + private sealed record Total(string Db, string Hash, long Executions, long CpuUs, long ElapsedUs, long LogicalReads, long PhysicalReads, long LogicalWrites); + + private static Total[] Totals(IEnumerable rows, bool withIo = true) => + rows.Select(r => new Total( + r.DatabaseName, r.QueryHash, r.TotalExecutions, r.TotalCpuUs, r.TotalElapsedUs, + withIo ? r.TotalLogicalReads : 0, withIo ? r.TotalPhysicalReads : 0, withIo ? r.TotalLogicalWrites : 0)).ToArray(); + + [Fact] + public Task Covering_EveryRankingReadsIo_ReadsEqualsTheRawRoute_AndTheOtherRankingsEqualTheIntervalSibling() => + WithStoreAsync(async (scratch, connection, ct) => + { + await IoRollupOracleSeed.PlantAsync(connection, ct, ServerId, ServerName, WindowStart); + await TopRankingLiveTests.PlantQueryAsync(connection, ct, "collect.query_stats", ServerId, ServerName, SurvivorAt, "QSURV", 1_000, 1_000, 7, 1); + await RefreshAsync(connection, TimescaleSupport.QueryStatsIntervalHourlyView, WindowStart, ct); + await RefreshAsync(connection, TimescaleSupport.QueryStatsIoHourlyView, WindowStart, ct); + + /* The raw route, read FIRST while raw still holds the rows: the raw reads statement itself over the same window. The + router cannot supply it: a window this old is Hourly-tier by age, and the reader's raw arm runs only on a + recent one. */ + var raw = await RunRawReadsAsync(connection, ct); + Assert.True(raw.Length >= 5, "the seed should rank at least five groups, or the comparison below proves nothing"); + Assert.DoesNotContain(raw, r => r.Hash == "0xQHD"); /* the zero-interval row's 9,999,999 reads are in neither route */ + + /* Now the raw rows go, so the window can only be answered by a rollup. */ + await PurgeRawBeforeAsync(connection, WindowStart.AddHours(IoRollupOracleSeed.Hours), ct); + + await using var ioSource = NpgsqlDataSource.Create(scratch.ConnectionString); + var io = await DarlingDataReader.GetTopQueriesByCpuRoutedAsync( + ioSource, ServerId, WindowStart, WindowEnd, Top, databaseName: null, ranking: TopRanking.Reads, cancellationToken: ct); + Assert.Equal(RetentionTier.Hourly, io.Tier); + Assert.True(io.IoRoute); + Assert.False(io.RawForced); + Assert.Null(io.RetentionNotice); /* the io rollup reaches the start: nothing partial to disclose */ + Assert.Equal(raw, Totals(io.Rows)); + Assert.Contains(io.Rows, r => r.TotalLogicalReads > 0 && r.TotalPhysicalReads > 0 && r.TotalLogicalWrites > 0); + Assert.Contains(io.Rows, r => r.QueryHash == "0xQHC" && r.TotalLogicalReads == 0); /* the NULL-reads group stays 0, never dropped */ + + /* Every other ranking reads the io relation too, and equals the interval sibling over the same window. */ + foreach (var ranking in new[] { TopRanking.Cpu, TopRanking.Duration, TopRanking.Executions }) + { + var label = TopRankings.WireName(ranking); + var viaIo = await DarlingDataReader.GetTopQueriesByCpuRoutedAsync( + ioSource, ServerId, WindowStart, WindowEnd, Top, databaseName: null, ranking: ranking, cancellationToken: ct); + Assert.True(viaIo.IoRoute, $"{label}: covering io takes every ranking"); + Assert.Equal(RetentionTier.Hourly, viaIo.Tier); + var sibling = await RunHourlyAsync(connection, TimescaleSupport.QueryStatsIntervalHourlyView, ranking, ct); + Assert.Equal(sibling, Totals(viaIo.Rows, withIo: false).Select(t => (t.Db, t.Hash, t.Executions, t.CpuUs, t.ElapsedUs)).ToArray()); + } + + /* The tool: the rows carry the sums, and the note names the relation instead of saying only raw has reads. */ + var query = "?server=" + ServerName + "&hours=24&as_of=" + Uri.EscapeDataString(WindowEnd.ToString("o")) + "&order_by=reads"; + using var doc = JsonDocument.Parse(await TopRankingLiveTests.WebReadAsync(ioSource, "get_top_queries_by_cpu", query)); + Assert.Equal("hourly", doc.RootElement.GetProperty("tier_used").GetString()); + var ioNote = doc.RootElement.GetProperty("precision_note").GetString()!; + Assert.Contains("query_stats_io_hourly", ioNote, StringComparison.Ordinal); + Assert.NotEqual(JsonValueKind.Null, doc.RootElement.GetProperty("queries")[0].GetProperty("total_logical_reads").ValueKind); + + /* The window-edges note rides on the io route too: ?: binds looser than +, and an unparenthesized form dropped it here, + leaving the answer silently cut at io's materialization ceiling. The expected text is the shared note over the same + read the tool made (24 hours back from as_of). The stitched route's sql_handle caveat is vacuous on io, whose WHERE + excludes the zero-interval rows, and is not claimed. */ + var toolStart = WindowEnd.AddHours(-24); + var toolRead = await DarlingDataReader.GetTopQueriesByCpuRoutedAsync( + ioSource, ServerId, toolStart, WindowEnd, Top, databaseName: null, ranking: TopRanking.Reads, cancellationToken: ct); + Assert.True(toolRead.IoRoute); + var edges = HourlyWindowEdges.Note(toolStart, toolRead.HourlyFirstBucket, WindowEnd, toolRead.HourlyCeiling); + Assert.False(string.IsNullOrEmpty(edges), "the seed must leave an edge to note (a ceiling before the end), or this pin proves nothing"); + Assert.Contains(edges, ioNote, StringComparison.Ordinal); + Assert.DoesNotContain("sql_handle is the rollup's MAX(sql_handle)", ioNote, StringComparison.Ordinal); + }); + + /* A reads ranking forced to raw by min_dop is forced by the FILTER: the io rollup covers the window and was never consulted, so the + answer must not say it was absent, empty or too shallow. ReadsForcedRaw is true only when the reads ranking itself sent the read to + raw. */ + [Fact] + public Task ARaw_ForcedByMinDop_NamesTheFilter_NotAnAbsentIoRollup() => + WithStoreAsync(async (scratch, connection, ct) => + { + await IoRollupOracleSeed.PlantAsync(connection, ct, ServerId, ServerName, WindowStart); + await TopRankingLiveTests.PlantQueryAsync(connection, ct, "collect.query_stats", ServerId, ServerName, SurvivorAt, "QSURV", 1_000, 1_000, 7, 1); + await DarlingMcpTestData.ExecAsync(connection, ct, "UPDATE collect.query_stats SET max_dop = 8 WHERE query_hash = '0xQSURV'"); + await RefreshAsync(connection, TimescaleSupport.QueryStatsIntervalHourlyView, WindowStart, ct); + await RefreshAsync(connection, TimescaleSupport.QueryStatsIoHourlyView, WindowStart, ct); + + await using var ioSource = NpgsqlDataSource.Create(scratch.ConnectionString); + var forced = await DarlingDataReader.GetTopQueriesByCpuRoutedAsync( + ioSource, ServerId, WindowEnd.AddHours(-24), WindowEnd, Top, databaseName: null, minMaxDop: 4, ranking: TopRanking.Reads, cancellationToken: ct); + Assert.Equal(RetentionTier.Raw, forced.Tier); + Assert.True(forced.RawForced); + Assert.False(forced.ReadsForcedRaw); /* the filter forced raw; io covers and was not consulted */ + Assert.Equal("0xQSURV", Assert.Single(forced.Rows).QueryHash); + + var json = await DarlingMcpDataTools.GetTopQueriesRanked( + ioSource, ServerName, 24, Top, null, false, 4, "query_hash", WindowEnd.ToString("o"), "summary", "reads", ct); + using (var doc = JsonDocument.Parse(json)) + { + var note = doc.RootElement.GetProperty("precision_note").GetString()!; + Assert.Contains("parallel_only / min_dop / group_by=host_object", note, StringComparison.Ordinal); + Assert.DoesNotContain("absent, empty or did not reach back", note, StringComparison.Ordinal); + Assert.DoesNotContain("order_by=reads needs per-query logical reads", note, StringComparison.Ordinal); + } + + /* Raw gone: the empty answer names the filter as the thing the rollup cannot apply, and does not claim the io rollup misses the window. */ + await PurgeRawBeforeAsync(connection, WindowEnd.AddDays(1), ct); + var empty = await DarlingMcpDataTools.GetTopQueriesRanked( + ioSource, ServerName, 24, Top, null, false, 4, "query_hash", WindowEnd.ToString("o"), "summary", "reads", ct); + Assert.Contains("apply parallel_only/min_dop/group_by=host_object", empty, StringComparison.Ordinal); + Assert.DoesNotContain("does not reach this window's start", empty, StringComparison.Ordinal); + Assert.DoesNotContain("rank by reads", empty, StringComparison.Ordinal); + }); + + [Fact] + public Task EmptyThenPartial_RoutesByWhichRelationReachesFurtherBack() => + WithStoreAsync(async (scratch, connection, ct) => + { + await IoRollupOracleSeed.PlantAsync(connection, ct, ServerId, ServerName, WindowStart); + await TopRankingLiveTests.PlantQueryAsync(connection, ct, "collect.query_stats", ServerId, ServerName, SurvivorAt, "QSURV", 1_000, 1_000, 7, 1); + await RefreshAsync(connection, TimescaleSupport.QueryStatsIntervalHourlyView, WindowStart, ct); + await PurgeRawBeforeAsync(connection, WindowStart.AddHours(1), ct); /* raw now starts at hour 1 */ + + /* io EMPTY: reads is raw (forced, with raw's retention notice); the other rankings keep the stitched rollup. */ + var empty = await ReadAsync(scratch, TopRanking.Reads, ct); + Assert.Equal(RetentionTier.Raw, empty.Tier); + Assert.True(empty.RawForced); + Assert.False(empty.IoRoute); + Assert.NotNull(empty.RetentionNotice); + var cpuEmpty = await ReadAsync(scratch, TopRanking.Cpu, ct); + Assert.Equal(RetentionTier.Hourly, cpuEmpty.Tier); + Assert.False(cpuEmpty.IoRoute); + Assert.All(cpuEmpty.Rows, r => Assert.Equal(0, r.TotalLogicalReads)); + + /* Covering is the other test: the floor cache trusts a floor while the rollup's oldest chunk is unchanged, so a refresh + that only moves the floor earlier inside that chunk is not seen by the next read of the same store. */ + /* io PARTIAL (starts at hour 2) while raw reaches back to hour 1: raw is deeper, so reads stays raw. */ + await RefreshAsync(connection, TimescaleSupport.QueryStatsIoHourlyView, WindowStart.AddHours(2), ct); + var rawDeeper = await ReadAsync(scratch, TopRanking.Reads, ct); + Assert.Equal(RetentionTier.Raw, rawDeeper.Tier); + Assert.False(rawDeeper.IoRoute); + Assert.NotNull(rawDeeper.RetentionNotice); + Assert.False((await ReadAsync(scratch, TopRanking.Cpu, ct)).IoRoute); + + /* raw is purged to hour 4: now the partial io reaches further back than raw, and reads takes it, disclosed as + a hourly window that starts at io's first bucket rather than with raw's retention notice. */ + await PurgeRawBeforeAsync(connection, WindowStart.AddHours(4), ct); + var partial = await ReadAsync(scratch, TopRanking.Reads, ct); + Assert.Equal(RetentionTier.Hourly, partial.Tier); + Assert.True(partial.IoRoute); + Assert.Null(partial.RetentionNotice); + Assert.Equal(WindowStart.AddHours(2), partial.HourlyFirstBucket); + Assert.DoesNotContain(partial.Rows, r => r.QueryHash == "0xQHA" && r.TotalLogicalReads == 0); + Assert.False((await ReadAsync(scratch, TopRanking.Cpu, ct)).IoRoute); /* partial io never takes a non-reads ranking */ + }); + + [Fact] + public Task Absent_AStoreWithoutTheIoRollup_ReadsRawForReads_AndNeverNamesTheMissingRelation() => + WithStoreAsync(async (scratch, connection, ct) => + { + await IoRollupOracleSeed.PlantAsync(connection, ct, ServerId, ServerName, WindowStart); + await TopRankingLiveTests.PlantQueryAsync(connection, ct, "collect.query_stats", ServerId, ServerName, SurvivorAt, "QSURV", 1_000, 1_000, 7, 1); + await RefreshAsync(connection, TimescaleSupport.QueryStatsIntervalHourlyView, WindowStart, ct); + await PurgeRawBeforeAsync(connection, WindowStart.AddHours(IoRollupOracleSeed.Hours), ct); + await using (var drop = new NpgsqlCommand($"DROP MATERIALIZED VIEW collect.{TimescaleSupport.QueryStatsIoHourlyView}", connection)) + { + await drop.ExecuteNonQueryAsync(ct); + } + + var reads = await ReadAsync(scratch, TopRanking.Reads, ct); + Assert.Equal(RetentionTier.Raw, reads.Tier); + Assert.True(reads.RawForced); + Assert.True(reads.ReadsForcedRaw); /* here the reads ranking itself sent the read to raw: no io rollup */ + Assert.Equal("0xQSURV", Assert.Single(reads.Rows).QueryHash); + + var cpu = await ReadAsync(scratch, TopRanking.Cpu, ct); + Assert.Equal(RetentionTier.Hourly, cpu.Tier); + Assert.False(cpu.IoRoute); + Assert.NotEmpty(cpu.Rows); + }); + + // ---------------------------------------------------------------- plumbing + + private static async Task ReadAsync(ScratchPostgres scratch, TopRanking ranking, CancellationToken ct) + { + await using var source = NpgsqlDataSource.Create(scratch.ConnectionString); + return await DarlingDataReader.GetTopQueriesByCpuRoutedAsync( + source, ServerId, WindowStart, WindowEnd, Top, databaseName: null, ranking: ranking, cancellationToken: ct); + } + + /// The hourly statement over directly (the sibling the io rollup must equal), as + /// (database, hash, executions, cpu, elapsed) in page order. + private static async Task<(string Db, string Hash, long Executions, long CpuUs, long ElapsedUs)[]> RunHourlyAsync( + NpgsqlConnection connection, string view, TopRanking ranking, CancellationToken ct) + { + var sql = TopRankings.Apply(DarlingDataReader.TopQueriesHourlySql, ranking, hourly: true) + .Replace(DarlingDataReader.TopQueriesHourlyFromPlaceholder, $"collect.{view} AS f", StringComparison.Ordinal) + .Replace(DarlingDataReader.TopQueriesHourlyIoSumsPlaceholder, TopRankings.HourlyNoIoSums, StringComparison.Ordinal) + .Replace("$CEIL$", "", StringComparison.Ordinal); + await using var command = new NpgsqlCommand(sql, connection); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = ServerId }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = WindowStart }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = WindowEnd }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = Top }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Array | NpgsqlDbType.Text, Value = DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = Top + 5 }); + var rows = new List<(string, string, long, long, long)>(); + await using var reader = await command.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + if (reader.IsDBNull(10)) + { + continue; /* the candidate-count row */ + } + + rows.Add((reader.GetString(0), reader.GetString(1), reader.GetInt64(2), reader.GetInt64(3), reader.GetInt64(4))); + } + + return rows.ToArray(); + } + + /// The raw reads statement ( ranked by reads) over the window, as totals in page order. + private static async Task RunRawReadsAsync(NpgsqlConnection connection, CancellationToken ct) + { + var sql = TopRankings.Apply(DarlingDataReader.TopQueriesSql, TopRanking.Reads, hourly: false); + await using var command = new NpgsqlCommand(sql, connection); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = ServerId }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = WindowStart }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = WindowEnd }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = Top }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Array | NpgsqlDbType.Text, Value = DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = 0 }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = Top + 5 }); + var rows = new List(); + await using var reader = await command.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + if (reader.IsDBNull(44)) + { + continue; /* the candidate-count row */ + } + + rows.Add(new Total( + reader.GetString(0), reader.GetString(1), reader.GetInt64(6), reader.GetInt64(7), reader.GetInt64(8), + reader.IsDBNull(9) ? 0 : reader.GetInt64(9), reader.IsDBNull(11) ? 0 : reader.GetInt64(11), reader.IsDBNull(10) ? 0 : reader.GetInt64(10))); + } + + return rows.ToArray(); + } + + private static async Task PurgeRawBeforeAsync(NpgsqlConnection connection, DateTime before, CancellationToken ct) + { + await using var purge = new NpgsqlCommand("DELETE FROM collect.query_stats WHERE server_id = $1 AND collection_time < $2", connection); + purge.Parameters.AddWithValue(ServerId); + purge.Parameters.AddWithValue(before); + await purge.ExecuteNonQueryAsync(ct); + } + + private static async Task RefreshAsync(NpgsqlConnection connection, string view, DateTime from, CancellationToken ct) + { + await using var refresh = new NpgsqlCommand($"CALL refresh_continuous_aggregate('collect.{view}'::regclass, $1::timestamp, $2::timestamp)", connection); + refresh.Parameters.AddWithValue(from); + refresh.Parameters.AddWithValue(WindowEnd.AddHours(1)); + await refresh.ExecuteNonQueryAsync(ct); + } + + /// Mints a scratch TimescaleDB store with the continuous aggregates and this class's server registered, and runs the body on it. + private static async Task WithStoreAsync(Func body) + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live #5329 io route tests."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var timescaleEnabled = await TimescaleSupport.TryEnableAsync(connection, null, ct); + Assert.SkipWhen(!timescaleEnabled, "The live #5329 io route tests need TimescaleDB."); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + Assert.True(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, null, ct)); + + await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection)) + { + await stop.ExecuteNonQueryAsync(ct); + } + + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, ct); + + var bodySucceeded = false; + try + { + await body(scratch, connection, ct); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (cleanup, cleanupCt) => + { + await using var probe = new NpgsqlCommand( + "SELECT count(*) FROM pg_catalog.pg_stat_activity WHERE datname = pg_catalog.current_database() " + + "AND backend_type LIKE 'TimescaleDB Background Worker Scheduler%'", cleanup); + var schedulers = Convert.ToInt64(await probe.ExecuteScalarAsync(cleanupCt)); + Assert.Equal(0L, schedulers); + }); + } + } +} diff --git a/Darling/Darling.Tests/TopRankingDifferentialLiveTests.cs b/Darling/Darling.Tests/TopRankingDifferentialLiveTests.cs new file mode 100644 index 0000000000..0046371b8e --- /dev/null +++ b/Darling/Darling.Tests/TopRankingDifferentialLiveTests.cs @@ -0,0 +1,572 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Darling.Tests; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace PerformanceMonitor.Darling.Tests; + +/// +/// #5226 differential: the shipped TWO-PASS raw top-N reads against the ONE-PASS statements they replaced, on one seed, +/// row for row. The old statements are kept verbatim in (with only the tie order +/// and NULL placement the two-pass work changed on purpose); the new ones are read through the shipped reader, so the +/// comparison covers the statement text, the parameter binding and the row mapping, and it keeps covering whatever the +/// reader's SQL becomes. +/// +/// What the seed is built to catch. Ties on the ranking metric alone and on metric and CPU together (only the +/// group key orders them); groups whose reads are all NULL or partly NULL; NULL database_name, query_hash and +/// host_object_name key parts (a join on plain = drops a NULL-keyed winner); several snapshots per group (a +/// sum is not a max); a proc-hosted group whose statement fragments across four hashes (the roll-up variant); a zero- +/// execution group (the HAVING); WAITFOR shells that outrank everything (the over-fetch and the trim); parallel-only floors +/// (max_dop NULL, 1, 4, mixed); zero-interval rows (the honest-interval filter); rows just outside the window; a latest +/// text that changed between snapshots; and forty seeded random groups drawn from small value sets, so ties are the rule. +/// Each comparison is the full page, keys and totals and order. +/// +/// Each (statement, ranking, top, database filter, parallelism floor) is one comparison, and a failure names it and +/// the first row that differs. +/// +[Collection("live-postgres")] +public sealed class TopRankingDifferentialLiveTests +{ + private const string ServerName = "darling-top-ranking-differential-e2e"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + + private const string Db1 = "DiffDb1"; + private const string Db2 = "DiffDb2"; + + private static readonly TopRanking[] Rankings = [TopRanking.Cpu, TopRanking.Duration, TopRanking.Reads, TopRanking.Executions]; + + /// The page sizes: smaller than the WAITFOR crowd (the +5 over-fetch bites), mid-size, and larger than the seed. + private static readonly int[] Tops = [3, 12, 150]; + + // ---------------------------------------------------------------- the seed + + /// One query_stats snapshot. is relative to now; null takes the next unique in-window instant. + private sealed record QRow( + string? Db, string? Hash, string? Host, string? Text, long Exec, long Cpu, long Elapsed, long? Reads, + int? MaxDop = 1, int Interval = 3600, TimeSpan? At = null); + + /// One procedure_stats snapshot. + private sealed record PRow( + string? Db, string? Schema, string? Object, string? Type, long Exec, long Cpu, long Elapsed, long? Reads, + int Interval = 3600, TimeSpan? At = null); + + private const string WaitforText = "WAITFOR DELAY '00:00:30'"; + + private static List QuerySeed() + { + var rows = new List(); + + /* One winner per ranking, three snapshots each. */ + foreach (var (name, cpu, elapsed, reads, exec) in new[] + { + ("HWCPU", 9_000L, 1_000L, 100L, 10L), ("HWDUR", 100L, 8_000L, 50L, 5L), + ("HWREADS", 500L, 600L, 90_000L, 20L), ("HWEXEC", 300L, 400L, 200L, 5_000L), + }) + { + for (var i = 0; i < 3; i++) + { + rows.Add(new QRow(Db1, "0x" + name, null, "SELECT " + name, exec / 3, cpu / 3, elapsed / 3, reads / 3)); + } + } + + /* Exact ties on all four metrics: only the group key can order TIE1..TIE4 (and TIE5/6 in the other database). */ + foreach (var (db, name) in new[] { (Db1, "TIE1"), (Db1, "TIE2"), (Db1, "TIE3"), (Db1, "TIE4"), (Db2, "TIE5"), (Db2, "TIE6") }) + { + rows.Add(new QRow(db, "0x" + name, null, "SELECT " + name, 10, 400, 700, 55)); + rows.Add(new QRow(db, "0x" + name, null, "SELECT " + name, 10, 100, 300, 45)); + } + + /* Ties on the metric alone: same duration, reads and executions, different CPU (the CPU tie-break), and two with equal CPU too. */ + rows.Add(new QRow(Db1, "0xTMA", null, "SELECT TMA", 30, 900, 5_000, 777)); + rows.Add(new QRow(Db1, "0xTMB", null, "SELECT TMB", 30, 800, 5_000, 777)); + rows.Add(new QRow(Db1, "0xTMC", null, "SELECT TMC", 30, 800, 5_000, 777)); + rows.Add(new QRow(Db2, "0xTMD", null, "SELECT TMD", 30, 100, 5_000, 777)); + + /* Reads that are NULL: all snapshots, and some snapshots (a SUM skips the NULLs). They must rank last by reads, never first. */ + rows.Add(new QRow(Db1, "0xNULLREADS", null, "SELECT NULLREADS", 50, 9_999, 9_999, null)); + rows.Add(new QRow(Db1, "0xNULLREADS", null, "SELECT NULLREADS", 50, 9_999, 9_999, null)); + rows.Add(new QRow(Db1, "0xPARTNULL", null, "SELECT PARTNULL", 8, 400, 400, null)); + rows.Add(new QRow(Db1, "0xPARTNULL", null, "SELECT PARTNULL", 8, 400, 400, 25)); + rows.Add(new QRow(Db2, "0xNULLREADS2", null, "SELECT NULLREADS2", 1, 1, 1, null)); + + /* NULL key parts: a NULL database, a NULL hash, both. */ + rows.Add(new QRow(null, "0xNULLDB", null, "SELECT NULLDB", 20, 3_000, 2_000, 5_000)); + rows.Add(new QRow(null, "0xNULLDB", null, "SELECT NULLDB", 20, 3_000, 2_000, 5_000)); + rows.Add(new QRow(Db1, null, null, "SELECT NULLHASH", 20, 2_900, 2_100, 5_100)); + rows.Add(new QRow(Db1, null, null, "SELECT NULLHASH", 20, 2_900, 2_100, 5_100)); + rows.Add(new QRow(null, null, null, "SELECT NULLBOTH", 20, 2_800, 2_200, 5_200)); + rows.Add(new QRow(Db1, null, "dbo.usp_NullHashHosted", "SELECT NULLHASHHOST", 20, 2_700, 2_300, 5_300)); + + /* Host objects: one hash that is ad hoc (NULL host) and proc-hosted by two procedures, and one proc whose dynamic SQL + fragments across four hashes (four groups by hash, one group by host object). */ + rows.Add(new QRow(Db1, "0xHOSTH", null, "SELECT adhoc", 40, 1_500, 1_200, 800)); + rows.Add(new QRow(Db1, "0xHOSTH", "dbo.usp_HostA", "INSERT A EXEC", 40, 1_400, 1_300, 810)); + rows.Add(new QRow(Db1, "0xHOSTH", "dbo.usp_HostB", "INSERT B EXEC", 40, 1_300, 1_400, 820)); + foreach (var i in new[] { 1, 2, 3, 4 }) + { + rows.Add(new QRow(Db1, "0xROLL" + i, "dbo.usp_Roll", "SELECT roll " + i, 10 * i, 300 * i, 200 * i, 100 * i)); + rows.Add(new QRow(Db1, "0xROLL" + i, "dbo.usp_Roll", "SELECT roll " + i + " newer", 10 * i, 300 * i, 200 * i, 100 * i)); + } + + /* Zero executions: out by the HAVING with zero elapsed, in with elapsed time. */ + rows.Add(new QRow(Db1, "0xZEROEXEC", null, "SELECT ZEROEXEC", 0, 50_000, 0, 60_000)); + rows.Add(new QRow(Db1, "0xZEROEXECELAPSED", null, "SELECT ZEROEXECELAPSED", 0, 1_000, 4_000, 10)); + + /* WAITFOR shells that outrank every real group on every metric: seven of them, more than the +5 over-fetch. */ + for (var i = 1; i <= 7; i++) + { + rows.Add(new QRow(Db1, "0xWF" + i, null, WaitforText, 90_000 + i, 900_000 + i, 900_000 + i, 900_000 + i)); + rows.Add(new QRow(Db1, "0xWF" + i, null, WaitforText, 1, 1, 1, 1)); + } + + /* Parallelism floor: the lifetime max_dop of the group decides. */ + rows.Add(new QRow(Db1, "0xPARSERIAL", null, "SELECT serial", 25, 1_800, 1_100, 900, MaxDop: 1)); + rows.Add(new QRow(Db1, "0xPARWIDE", null, "SELECT wide", 25, 1_700, 1_000, 910, MaxDop: 4)); + rows.Add(new QRow(Db1, "0xPARNULL", null, "SELECT neverCaptured", 25, 1_600, 1_050, 920, MaxDop: null)); + rows.Add(new QRow(Db1, "0xPARMIX", null, "SELECT mixed", 12, 800, 500, 450, MaxDop: 1)); + rows.Add(new QRow(Db1, "0xPARMIX", null, "SELECT mixed", 13, 900, 550, 460, MaxDop: 8)); + + /* Zero-interval rows: dropped by the honest-interval filter, so only the second snapshot counts. */ + rows.Add(new QRow(Db1, "0xZINT", null, "SELECT zint", 70, 50_000, 50_000, 50_000, Interval: 0)); + rows.Add(new QRow(Db1, "0xZINT", null, "SELECT zint", 7, 70, 70, 70)); + rows.Add(new QRow(Db1, "0xZINTONLY", null, "SELECT zintonly", 70, 60_000, 60_000, 60_000, Interval: 0)); + + /* The window: a group wholly outside it, and one with an in-window snapshot beside huge ones before the start and after the end. */ + rows.Add(new QRow(Db1, "0xOUTSIDE", null, "SELECT outside", 99, 99_999, 99_999, 99_999, At: TimeSpan.FromHours(-30))); + rows.Add(new QRow(Db1, "0xSTRADDLE", null, "SELECT straddle old", 99, 99_999, 99_999, 99_999, At: TimeSpan.FromHours(-29))); + rows.Add(new QRow(Db1, "0xSTRADDLE", null, "SELECT straddle in", 3, 33, 33, 33)); + rows.Add(new QRow(Db1, "0xSTRADDLE", null, "SELECT straddle future", 99, 99_999, 99_999, 99_999, At: TimeSpan.FromHours(1))); + + /* Text: the newest non-NULL text wins, and a group with only NULL texts reads an empty one. */ + rows.Add(new QRow(Db1, "0xTEXTVAR", null, "SELECT old text", 6, 600, 600, 60)); + rows.Add(new QRow(Db1, "0xTEXTVAR", null, "SELECT newer text", 6, 600, 600, 60)); + rows.Add(new QRow(Db1, "0xTEXTVAR", null, null, 6, 600, 600, 60)); + rows.Add(new QRow(Db1, "0xNOTEXT", null, null, 6, 650, 650, 65)); + + /* Seeded random groups from small value sets, so ties on the metric and on CPU come up on their own. */ + var rng = new Random(5299); + for (var g = 0; g < 40; g++) + { + var db = rng.Next(4) == 0 ? Db2 : Db1; + var hash = "0xR" + rng.Next(1, 13).ToString("00", CultureInfo.InvariantCulture); + var host = rng.Next(5) switch { 0 => "dbo.usp_R1", 1 => "dbo.usp_R2", _ => null }; + var snapshots = rng.Next(1, 5); + for (var s = 0; s < snapshots; s++) + { + rows.Add(new QRow( + db, hash, host, + rng.Next(6) == 0 ? null : "SELECT " + hash + " v" + rng.Next(3), + new long[] { 0, 0, 5, 10, 10, 20 }[rng.Next(6)], + new long[] { 0, 100, 200, 200, 500, 1_000 }[rng.Next(6)], + new long[] { 0, 300, 300, 900 }[rng.Next(4)], + rng.Next(5) == 0 ? null : new long[] { 0, 10, 10, 50 }[rng.Next(4)], + MaxDop: new int?[] { null, 1, 1, 4 }[rng.Next(4)])); + } + } + + return rows; + } + + private static List ProcedureSeed() + { + var rows = new List(); + const string Proc = "SQL_STORED_PROCEDURE"; + + foreach (var (name, cpu, elapsed, reads, exec) in new[] + { + ("HWCPU", 9_000L, 1_000L, 100L, 10L), ("HWDUR", 100L, 8_000L, 50L, 5L), + ("HWREADS", 500L, 600L, 90_000L, 20L), ("HWEXEC", 300L, 400L, 200L, 5_000L), + }) + { + for (var i = 0; i < 3; i++) + { + rows.Add(new PRow(Db1, "dbo", "usp_" + name, Proc, exec / 3, cpu / 3, elapsed / 3, reads / 3)); + } + } + + foreach (var (db, name) in new[] { (Db1, "TIE1"), (Db1, "TIE2"), (Db1, "TIE3"), (Db1, "TIE4"), (Db2, "TIE5"), (Db2, "TIE6") }) + { + rows.Add(new PRow(db, "dbo", "usp_" + name, Proc, 10, 400, 700, 55)); + rows.Add(new PRow(db, "dbo", "usp_" + name, Proc, 10, 100, 300, 45)); + } + + /* The same name in two schemas and two object types: each is its own group. */ + rows.Add(new PRow(Db1, "dbo", "usp_Same", Proc, 10, 500, 500, 50)); + rows.Add(new PRow(Db1, "sales", "usp_Same", Proc, 10, 500, 500, 50)); + rows.Add(new PRow(Db1, "dbo", "usp_Same", "SQL_SCALAR_FUNCTION", 10, 500, 500, 50)); + + rows.Add(new PRow(Db1, "dbo", "usp_TMA", Proc, 30, 900, 5_000, 777)); + rows.Add(new PRow(Db1, "dbo", "usp_TMB", Proc, 30, 800, 5_000, 777)); + rows.Add(new PRow(Db1, "dbo", "usp_TMC", Proc, 30, 800, 5_000, 777)); + + rows.Add(new PRow(Db1, "dbo", "usp_NULLREADS", Proc, 50, 9_999, 9_999, null)); + rows.Add(new PRow(Db1, "dbo", "usp_NULLREADS", Proc, 50, 9_999, 9_999, null)); + rows.Add(new PRow(Db1, "dbo", "usp_PARTNULL", Proc, 8, 400, 400, null)); + rows.Add(new PRow(Db1, "dbo", "usp_PARTNULL", Proc, 8, 400, 400, 25)); + + /* NULL key parts: database, schema, object name and object type. */ + rows.Add(new PRow(null, "dbo", "usp_NULLDB", Proc, 20, 3_000, 2_000, 5_000)); + rows.Add(new PRow(Db1, null, "usp_NULLSCHEMA", Proc, 20, 2_900, 2_100, 5_100)); + rows.Add(new PRow(Db1, "dbo", null, Proc, 20, 2_800, 2_200, 5_200)); + rows.Add(new PRow(Db1, "dbo", "usp_NULLTYPE", null, 20, 2_700, 2_300, 5_300)); + rows.Add(new PRow(Db1, "dbo", "usp_NULLTYPE", null, 20, 2_700, 2_300, 5_300)); + rows.Add(new PRow(null, null, null, null, 20, 2_600, 2_400, 5_400)); + + rows.Add(new PRow(Db1, "dbo", "usp_ZEROEXEC", Proc, 0, 50_000, 0, 60_000)); + rows.Add(new PRow(Db1, "dbo", "usp_ZEROEXECELAPSED", Proc, 0, 1_000, 4_000, 10)); + + rows.Add(new PRow(Db1, "dbo", "usp_ZINT", Proc, 70, 50_000, 50_000, 50_000, Interval: 0)); + rows.Add(new PRow(Db1, "dbo", "usp_ZINT", Proc, 7, 70, 70, 70)); + rows.Add(new PRow(Db1, "dbo", "usp_ZINTONLY", Proc, 70, 60_000, 60_000, 60_000, Interval: 0)); + + rows.Add(new PRow(Db1, "dbo", "usp_OUTSIDE", Proc, 99, 99_999, 99_999, 99_999, At: TimeSpan.FromHours(-30))); + rows.Add(new PRow(Db1, "dbo", "usp_STRADDLE", Proc, 99, 99_999, 99_999, 99_999, At: TimeSpan.FromHours(-29))); + rows.Add(new PRow(Db1, "dbo", "usp_STRADDLE", Proc, 3, 33, 33, 33)); + rows.Add(new PRow(Db1, "dbo", "usp_STRADDLE", Proc, 99, 99_999, 99_999, 99_999, At: TimeSpan.FromHours(1))); + + var rng = new Random(5226); + for (var g = 0; g < 30; g++) + { + var db = rng.Next(4) == 0 ? Db2 : Db1; + var obj = "usp_R" + rng.Next(1, 11).ToString("00", CultureInfo.InvariantCulture); + var schema = rng.Next(3) == 0 ? "sales" : "dbo"; + var type = rng.Next(5) == 0 ? "SQL_SCALAR_FUNCTION" : Proc; + var snapshots = rng.Next(1, 5); + for (var s = 0; s < snapshots; s++) + { + rows.Add(new PRow( + db, schema, obj, type, + new long[] { 0, 0, 5, 10, 10, 20 }[rng.Next(6)], + new long[] { 0, 100, 200, 200, 500, 1_000 }[rng.Next(6)], + new long[] { 0, 300, 300, 900 }[rng.Next(4)], + rng.Next(5) == 0 ? null : new long[] { 0, 10, 10, 50 }[rng.Next(4)])); + } + } + + return rows; + } + + // ---------------------------------------------------------------- the differential + + /// + /// Every ranking, both query statements (per hash and rolled up by host object), every page size, the database filter + /// and the parallelism floor: the shipped read returns the same page as the one-pass statement, row for row. + /// + [Fact] + public Task Queries_EveryRankingAndShape_ReturnTheRowsTheOnePassStatementsDid() => + WithSharedStoreAsync(async (connection, postgres, now, ct) => + { + await PlantQueriesAsync(connection, ct, now); + var (start, end) = (now.AddHours(-24), now.AddMinutes(5)); + var mismatches = new List(); + var stats = new Stats(); + + foreach (var ranking in Rankings) + { + foreach (var rollUp in new[] { false, true }) + { + foreach (var top in Tops) + { + foreach (var db in new string?[] { null, Db1 }) + { + foreach (var minDop in new[] { 0, 2 }) + { + var label = $"queries {(rollUp ? "host-object roll-up" : "per hash")} by {TopRankings.WireName(ranking)}, top {top}, db {db ?? "(all)"}, min_max_dop {minDop}"; + var oldSql = rollUp ? TopRankingOnePassBaseline.QueriesByHostObject(ranking) : TopRankingOnePassBaseline.Queries(ranking); + var oldPage = await ReadOldQueriesAsync(postgres, oldSql, start, end, top, db, minDop, ct); + var result = await DarlingDataReader.GetTopQueriesByCpuRoutedAsync( + postgres, ServerId, start, end, top, db, rollUp, minDop, ranking, ct); + Assert.Equal(RetentionTier.Raw, result.Tier); + var newPage = result.Rows.Select(FormatQuery).ToList(); + Compare(label, oldPage, newPage, mismatches); + stats.Note(ranking, oldPage); + } + } + } + } + } + + AssertNoMismatches(mismatches, stats, expectedComparisons: Rankings.Length * 2 * Tops.Length * 2 * 2); + + /* The traps the seed plants really are on the pages being compared (a join on plain = would lose the NULL host group). */ + var rolled = await DarlingDataReader.GetTopQueriesByCpuAsync( + postgres, ServerId, start, end, top: 150, databaseName: null, rollUpByHostObject: true, ranking: TopRanking.Cpu, cancellationToken: ct); + Assert.Contains(rolled, r => r.HostObjectName is null); + Assert.Equal(4, Assert.Single(rolled, r => r.HostObjectName == "dbo.usp_Roll").DistinctQueryHashes); + var perHash = await DarlingDataReader.GetTopQueriesByCpuAsync( + postgres, ServerId, start, end, top: 150, databaseName: null, ranking: TopRanking.Reads, cancellationToken: ct); + Assert.Equal(4, perHash.Count(r => r.HostObjectName == "dbo.usp_Roll")); + Assert.Contains(perHash, r => r.DatabaseName == "" && r.QueryHash == "0xNULLDB"); + Assert.Contains(perHash, r => r.QueryHash == "" && r.DatabaseName == Db1); + var readsOrder = perHash.Select(r => r.QueryHash).ToList(); + Assert.True(readsOrder.IndexOf("0xNULLREADS") > readsOrder.IndexOf("0xPARTNULL"), + "a group whose reads are all NULL ranks after one that has some: " + string.Join(",", readsOrder)); + }); + + /// The same comparison for the procedures grid. + [Fact] + public Task Procedures_EveryRanking_ReturnTheRowsTheOnePassStatementDid() => + WithSharedStoreAsync(async (connection, postgres, now, ct) => + { + await PlantProceduresAsync(connection, ct, now); + var (start, end) = (now.AddHours(-24), now.AddMinutes(5)); + var mismatches = new List(); + var stats = new Stats(); + + foreach (var ranking in Rankings) + { + foreach (var top in Tops) + { + foreach (var db in new string?[] { null, Db1 }) + { + var label = $"procedures by {TopRankings.WireName(ranking)}, top {top}, db {db ?? "(all)"}"; + var oldPage = await ReadOldProceduresAsync(postgres, TopRankingOnePassBaseline.Procedures(ranking), start, end, top, db, ct); + var result = await DarlingDataReader.GetTopProceduresByCpuRoutedAsync( + postgres, ServerId, start, end, top, db, ranking, ct); + Assert.Equal(RetentionTier.Raw, result.Tier); + var newPage = result.Rows.Select(FormatProcedure).ToList(); + Compare(label, oldPage, newPage, mismatches); + stats.Note(ranking, oldPage); + } + } + } + + AssertNoMismatches(mismatches, stats, expectedComparisons: Rankings.Length * Tops.Length * 2); + + var perName = await DarlingDataReader.GetTopProceduresByCpuAsync( + postgres, ServerId, start, end, top: 150, databaseName: null, ranking: TopRanking.Reads, cancellationToken: ct); + Assert.Contains(perName, r => r.DatabaseName == "" && r.ObjectName == "usp_NULLDB"); + Assert.Contains(perName, r => r.ObjectName == "usp_NULLTYPE"); + Assert.Equal(3, perName.Count(r => r.ObjectName == "usp_Same")); + var readsOrder = perName.Select(r => r.ObjectName).ToList(); + Assert.True(readsOrder.IndexOf("usp_NULLREADS") > readsOrder.IndexOf("usp_PARTNULL"), + "a procedure whose reads are all NULL ranks after one that has some: " + string.Join(",", readsOrder)); + }); + + // ---------------------------------------------------------------- comparing + + /// What the comparisons covered, so a seed that stopped exercising ties or filled no page fails loudly. + private sealed class Stats + { + public int Comparisons; + public int EmptyPages; + public readonly Dictionary Rows = []; + + public void Note(TopRanking ranking, List page) + { + Comparisons++; + EmptyPages += page.Count == 0 ? 1 : 0; + Rows[ranking] = Rows.GetValueOrDefault(ranking) + page.Count; + } + } + + private static void Compare(string label, List oldPage, List newPage, List mismatches) + { + if (oldPage.SequenceEqual(newPage)) + { + return; + } + + var at = 0; + while (at < oldPage.Count && at < newPage.Count && oldPage[at] == newPage[at]) + { + at++; + } + + mismatches.Add($"{label}: {oldPage.Count} old rows, {newPage.Count} new rows, first difference at row {at}:\n" + + $" old: {(at < oldPage.Count ? oldPage[at] : "(none)")}\n new: {(at < newPage.Count ? newPage[at] : "(none)")}"); + } + + private static void AssertNoMismatches(List mismatches, Stats stats, int expectedComparisons) + { + Assert.True(mismatches.Count == 0, + $"{mismatches.Count} of {stats.Comparisons} comparisons differ from the one-pass statements. First:\n" + string.Join("\n", mismatches.Take(5))); + Assert.Equal(expectedComparisons, stats.Comparisons); + Assert.True(stats.EmptyPages * 4 < stats.Comparisons, $"{stats.EmptyPages} of {stats.Comparisons} compared pages were empty; the seed no longer fills them"); + foreach (var ranking in Rankings) + { + Assert.True(stats.Rows.GetValueOrDefault(ranking) > 50, $"{TopRankings.WireName(ranking)}: only {stats.Rows.GetValueOrDefault(ranking)} rows compared"); + } + } + + /// One page row as a comparable line: every column the reader maps from the statement's first block, in order. + private static string FormatQuery(DarlingDataReader.TopQueryRow r) => string.Join("|", new object?[] + { + r.DatabaseName, r.QueryHash, r.HostObjectName ?? "", r.QueryPlanHash, r.SqlHandle, r.PlanHandle, + r.TotalExecutions, r.TotalCpuUs, r.TotalElapsedUs, r.TotalLogicalReads, r.TotalLogicalWrites, r.TotalPhysicalReads, + r.TotalRows, r.TotalSpills, r.MinDop, r.MaxDop, r.MinCpuUs, r.MaxCpuUs, r.MinElapsedUs, r.MaxElapsedUs, + r.QueryText ?? "", r.DistinctTexts, r.DistinctQueryHashes, + }.Select(v => Convert.ToString(v, CultureInfo.InvariantCulture))); + + private static string FormatProcedure(DarlingDataReader.TopProcedureRow r) => string.Join("|", new object?[] + { + r.DatabaseName, r.SchemaName, r.ObjectName, r.ObjectType, r.SqlHandle, r.PlanHandle, + r.TotalExecutions, r.TotalCpuUs, r.TotalElapsedUs, r.TotalLogicalReads, r.TotalLogicalWrites, r.TotalPhysicalReads, + r.TotalSpills, r.MinCpuUs, r.MaxCpuUs, r.MinElapsedUs, r.MaxElapsedUs, + }.Select(v => Convert.ToString(v, CultureInfo.InvariantCulture))); + + /// Runs an old one-pass queries statement and maps its columns the way the reader does (NULL text and strings read empty, NULL numbers read zero). + private static async Task> ReadOldQueriesAsync( + NpgsqlDataSource postgres, string sql, DateTime start, DateTime end, int top, string? database, int minMaxDop, CancellationToken ct) + { + /* #5313: the shipped read refills past the WAITFOR trim (TopFill), so a page is full whenever the window has enough + real groups. The one-pass reference gets the same answer by over-fetching deep enough up front. */ + Assert.Contains("LIMIT $4 + 5", sql, StringComparison.Ordinal); + sql = sql.Replace("LIMIT $4 + 5", "LIMIT $4 + 200", StringComparison.Ordinal); + /* #5309 matches the latest-text lookup NULL-safe on every key column, so a group with a NULL database or hash gets + its text (the old strict equality gave it none). The reference takes the same rule: a stated, deliberate change. */ + sql = System.Text.RegularExpressions.Regex.Replace(sql, @"\b(query_hash|database_name) = r\.\1\b", "$1 IS NOT DISTINCT FROM r.$1"); + /* #5309 also reads the newest text INSIDE the window (the old lookup had no time bound, so a row after the window could + supply a group's text). Same stated change, same treatment. */ + sql = System.Text.RegularExpressions.Regex.Replace(sql, @"(SELECT query_text\s+FROM v_query_stats\s+WHERE server_id = \$1)", + "$1 AND collection_time >= $2 AND collection_time <= $3"); + await using var command = postgres.CreateCommand(sql); + AddCommonParameters(command, start, end, top, database); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = minMaxDop }); + await using var reader = await command.ExecuteReaderAsync(ct); + var page = new List(); + while (await reader.ReadAsync(ct)) + { + string Text(int i) => reader.IsDBNull(i) ? "" : reader.GetString(i); + long Long(int i) => reader.IsDBNull(i) ? 0 : reader.GetInt64(i); + int Int(int i) => reader.IsDBNull(i) ? 0 : Convert.ToInt32(reader.GetValue(i)); + page.Add(string.Join("|", new object?[] + { + Text(0), Text(1), reader.IsDBNull(2) ? "" : reader.GetString(2), Text(3), Text(4), Text(5), + Long(6), Long(7), Long(8), Long(9), Long(10), Long(11), Long(12), Long(13), Int(14), Int(15), + Long(16), Long(17), Long(18), Long(19), Text(20), Long(21), reader.IsDBNull(22) ? 1L : reader.GetInt64(22), + }.Select(v => Convert.ToString(v, CultureInfo.InvariantCulture)))); + } + + return page; + } + + private static async Task> ReadOldProceduresAsync( + NpgsqlDataSource postgres, string sql, DateTime start, DateTime end, int top, string? database, CancellationToken ct) + { + await using var command = postgres.CreateCommand(sql); + AddCommonParameters(command, start, end, top, database); + await using var reader = await command.ExecuteReaderAsync(ct); + var page = new List(); + while (await reader.ReadAsync(ct)) + { + string Text(int i) => reader.IsDBNull(i) ? "" : reader.GetString(i); + long Long(int i) => reader.IsDBNull(i) ? 0 : reader.GetInt64(i); + page.Add(string.Join("|", new object?[] + { + Text(0), Text(1), Text(2), Text(3), Text(4), Text(5), + Long(6), Long(7), Long(8), Long(9), Long(10), Long(11), Long(12), Long(13), Long(14), Long(15), Long(16), + }.Select(v => Convert.ToString(v, CultureInfo.InvariantCulture)))); + } + + return page; + } + + /// $1 server, $2 and $3 the window, $4 top, $5 the database filter: the order both old statements and the shipped reader bind. + private static void AddCommonParameters(NpgsqlCommand command, DateTime start, DateTime end, int top, string? database) + { + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = ServerId }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = start }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Timestamp, Value = end }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = top }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Text, Value = (object?)database ?? DBNull.Value }); + } + + // ---------------------------------------------------------------- planting and plumbing + + /// Plants the query seed. Every in-window row gets its own second, so a group's latest text is never a tie. + private static async Task PlantQueriesAsync(NpgsqlConnection connection, CancellationToken ct, DateTime now) + { + var tick = 0; + foreach (var row in QuerySeed()) + { + var at = DarlingMcpTestData.TruncateToSeconds(row.At is { } offset ? now + offset : now.AddHours(-3).AddSeconds(7 * tick++)); + byte[]? digest = row.Text is null ? null : SHA256.HashData(Encoding.UTF8.GetBytes(row.Text)); + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO query_stats + (collection_id, collection_time, server_id, server_name, database_name, query_hash, host_object_name, + query_plan_hash, sql_handle, plan_handle, query_text, query_text_digest, + delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, + delta_logical_writes, delta_physical_reads, min_worker_time, max_worker_time, + min_elapsed_time, max_elapsed_time, min_dop, max_dop, sample_interval_seconds) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25)", + CollectionIdGenerator.Next(), at, ServerId, ServerName, (object?)row.Db ?? DBNull.Value, (object?)row.Hash ?? DBNull.Value, + (object?)row.Host ?? DBNull.Value, "0xPH" + row.Hash, "0xSH" + row.Hash, "0xPLH" + row.Hash, + (object?)row.Text ?? DBNull.Value, (object?)digest ?? DBNull.Value, + row.Exec, row.Cpu, row.Elapsed, (object?)row.Reads ?? DBNull.Value, row.Cpu / 10, (object?)(row.Reads / 100) ?? DBNull.Value, + row.Cpu / 2, row.Cpu, row.Elapsed / 2, row.Elapsed, + (object?)row.MaxDop ?? DBNull.Value, (object?)row.MaxDop ?? DBNull.Value, row.Interval); + } + } + + private static async Task PlantProceduresAsync(NpgsqlConnection connection, CancellationToken ct, DateTime now) + { + var tick = 0; + foreach (var row in ProcedureSeed()) + { + var at = DarlingMcpTestData.TruncateToSeconds(row.At is { } offset ? now + offset : now.AddHours(-3).AddSeconds(7 * tick++)); + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO procedure_stats + (collection_id, collection_time, server_id, server_name, database_name, schema_name, object_name, object_type, + sql_handle, plan_handle, delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, + delta_logical_writes, delta_physical_reads, delta_spills, min_worker_time, max_worker_time, + min_elapsed_time, max_elapsed_time, sample_interval_seconds) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22)", + CollectionIdGenerator.Next(), at, ServerId, ServerName, (object?)row.Db ?? DBNull.Value, (object?)row.Schema ?? DBNull.Value, + (object?)row.Object ?? DBNull.Value, (object?)row.Type ?? DBNull.Value, "0xSH" + row.Object, "0xPLH" + row.Object, + row.Exec, row.Cpu, row.Elapsed, (object?)row.Reads ?? DBNull.Value, row.Cpu / 10, (object?)(row.Reads / 100) ?? DBNull.Value, + row.Exec / 5, row.Cpu / 2, row.Cpu, row.Elapsed / 2, row.Elapsed, row.Interval); + } + } + + /// Runs a body against the shared store with this class's server registered, and leaves nothing behind. + private static async Task WithSharedStoreAsync(Func body) + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), "Set DARLING_TEST_PG to run the live top-ranking differential tests."); + var ct = TestContext.Current.CancellationToken; + + using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await CleanupAsync(connection, ct); + + await using var postgres = NpgsqlDataSource.Create(connectionString!); + var succeeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + await body(connection, postgres, DarlingMcpTestData.Naive(DateTime.UtcNow), ct); + succeeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, succeeded, async (cleanup, cleanupCt) => + await CleanupAsync(cleanup, cleanupCt)); + } + } + + private static async Task CleanupAsync(NpgsqlConnection connection, CancellationToken ct) => + await DarlingMcpTestData.ExecAsync(connection, ct, + $"DELETE FROM query_stats WHERE server_id = {ServerId}; DELETE FROM procedure_stats WHERE server_id = {ServerId}; DELETE FROM servers WHERE server_id = {ServerId}"); +} diff --git a/Darling/Darling.Tests/TopRankingHourlyLiveTests.cs b/Darling/Darling.Tests/TopRankingHourlyLiveTests.cs new file mode 100644 index 0000000000..ea5b4e6e3f --- /dev/null +++ b/Darling/Darling.Tests/TopRankingHourlyLiveTests.cs @@ -0,0 +1,293 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Darling.Tests; +using Npgsql; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace PerformanceMonitor.Darling.Tests; + +/// +/// #5226 live pins for the ranking choice where it needs rollups: the hourly path answers CPU, duration and executions +/// from the rollup on a window raw no longer holds, reads stays on raw (the rollups carry no logical reads) and says +/// so, and a reads ranking over a window that starts before the raw floor carries the retention notice while one inside +/// the floor, and every other ranking, carries none. The raw path and the endpoint's refusal are in +/// , which runs on the shared store. +/// +/// #1776 own-store — each test mints a scratch database, because it materializes continuous aggregates the +/// shared fixture must never inherit, so this class is deliberately NOT in the live-postgres collection. +/// +/// One test per grid, not per choice. Every test builds a TimescaleDB store, which is most of its cost, so the +/// choices are looped inside one test per grid and each assertion names the choice it checks. The hourly tests reuse the +/// seed of ; the raw floor and the rollup floor are made to differ by planting the seed +/// in the first hours of an aged window, refreshing the hourly view, then deleting those raw rows and keeping one later +/// survivor, so the router sends the CPU, duration and executions reads to the rollup and the reads read finds only the +/// survivor in raw. The reads are read through a FRESH data source, because coverage is cached per data source. +/// +public sealed class TopRankingHourlyLiveTests +{ + private const int ServerId = -943851; + private const string ServerName = "a5226-top-ranking-hourly"; + + /// Fixed anchor, never wall-clock relative: the window ages past raw's floor once the seed's raw rows are deleted, + /// which is what drives the router, not calendar time (the same shape the #4231 routing tests use). + private static readonly DateTime WindowStart = new(2026, 1, 5, 0, 0, 0, DateTimeKind.Unspecified); + + private static readonly TimeSpan[] SeedOffsets = + [ + TimeSpan.FromHours(1), TimeSpan.FromMinutes(80), TimeSpan.FromHours(2), TimeSpan.FromMinutes(140), + ]; + + private static readonly DateTime SurvivorAt = WindowStart.AddHours(5); + + [Fact] + public Task Queries_OnTheHourlyPath_EachChoiceButReadsRanksItsOwnMetric_AndReadsStaysOnRaw() => + WithHourlyStoreAsync(async (scratch, connection, ct) => + { + var windowEnd = WindowStart.AddDays(1); + for (var i = 0; i < TopRankingLiveTests.Seeds.Length; i++) + { + var seed = TopRankingLiveTests.Seeds[i]; + await TopRankingLiveTests.PlantQueryAsync(connection, ct, "collect.query_stats", ServerId, ServerName, WindowStart + SeedOffsets[i], + seed.Name, seed.CpuUs, seed.ElapsedUs, seed.Reads, seed.Executions); + } + + await TopRankingLiveTests.PlantQueryAsync(connection, ct, "collect.query_stats", ServerId, ServerName, SurvivorAt, "QSURV", 1_000, 1_000, 7, 1); + await RefreshAsync(connection, TimescaleSupport.QueryStatsIntervalHourlyView, WindowStart, windowEnd.AddHours(1), ct); + await PurgeSeedAsync(connection, "collect.query_stats", ct); + + await using var hourlyDataSource = NpgsqlDataSource.Create(scratch.ConnectionString); + foreach (var (ranking, expectedOrder) in new[] + { + (TopRanking.Cpu, "0xQCPU,0xQREADS,0xQEXEC,0xQDUR,0xQSURV"), + (TopRanking.Duration, "0xQDUR,0xQCPU,0xQREADS,0xQEXEC,0xQSURV"), + (TopRanking.Executions, "0xQEXEC,0xQREADS,0xQCPU,0xQDUR,0xQSURV"), + }) + { + var label = "queries by " + TopRankings.WireName(ranking); + var result = await DarlingDataReader.GetTopQueriesByCpuRoutedAsync( + hourlyDataSource, ServerId, WindowStart, windowEnd, top: 10, databaseName: null, ranking: ranking, cancellationToken: ct); + Assert.True(result.Tier == RetentionTier.Hourly, $"{label}: expected the hourly tier but read {result.Tier}"); + TopRankingLiveTests.AssertOrder(result.Rows.Select(r => r.QueryHash), expectedOrder, label + " (hourly)"); + Assert.Null(result.RetentionNotice); + } + + /* Reads: the same window, which the router would send to the rollup, is read from raw, where only the survivor is. */ + var reads = await DarlingDataReader.GetTopQueriesByCpuRoutedAsync( + hourlyDataSource, ServerId, WindowStart, windowEnd, top: 10, databaseName: null, ranking: TopRanking.Reads, cancellationToken: ct); + Assert.Equal(RetentionTier.Raw, reads.Tier); + Assert.True(reads.RawForced); + Assert.Equal("0xQSURV", Assert.Single(reads.Rows).QueryHash); + Assert.NotNull(reads.RetentionNotice); + + /* The page sees it as the tier it read, the reason, and how far back raw reached. */ + var query = "?server=" + ServerName + "&hours=24&as_of=" + Uri.EscapeDataString(windowEnd.ToString("o")); + using (var readsDoc = JsonDocument.Parse(await TopRankingLiveTests.WebReadAsync(hourlyDataSource, "get_top_queries_by_cpu", query + "&order_by=reads"))) + { + var root = readsDoc.RootElement; + Assert.Equal("raw", root.GetProperty("tier_used").GetString()); + Assert.Contains("order_by=reads needs per-query logical reads", root.GetProperty("precision_note").GetString(), StringComparison.Ordinal); + Assert.True(root.GetProperty("window_truncated").GetBoolean()); + Assert.Equal(DateTime.SpecifyKind(DarlingMcpTestData.TruncateToSeconds(SurvivorAt), DateTimeKind.Utc).ToString("o"), root.GetProperty("effective_start").GetString()); + Assert.Equal(JsonValueKind.String, root.GetProperty("retention_notice").ValueKind); + } + + using var durationDoc = JsonDocument.Parse(await TopRankingLiveTests.WebReadAsync(hourlyDataSource, "get_top_queries_by_cpu", query + "&order_by=duration")); + Assert.Equal("hourly", durationDoc.RootElement.GetProperty("tier_used").GetString()); + Assert.Equal("0xQDUR", durationDoc.RootElement.GetProperty("queries")[0].GetProperty("query_hash").GetString()); + }); + + [Fact] + public Task Procedures_OnTheHourlyPath_EachChoiceButReadsRanksItsOwnMetric_AndReadsStaysOnRaw() => + WithHourlyStoreAsync(async (scratch, connection, ct) => + { + var windowEnd = WindowStart.AddDays(1); + for (var i = 0; i < TopRankingLiveTests.Seeds.Length; i++) + { + var seed = TopRankingLiveTests.Seeds[i]; + await TopRankingLiveTests.PlantProcedureAsync(connection, ct, "collect.procedure_stats", ServerId, ServerName, WindowStart + SeedOffsets[i], + seed.Name, seed.CpuUs, seed.ElapsedUs, seed.Reads, seed.Executions); + } + + await TopRankingLiveTests.PlantProcedureAsync(connection, ct, "collect.procedure_stats", ServerId, ServerName, SurvivorAt, "QSURV", 1_000, 1_000, 7, 1); + await RefreshAsync(connection, TimescaleSupport.ProcedureStatsIntervalHourlyView, WindowStart, windowEnd.AddHours(1), ct); + await PurgeSeedAsync(connection, "collect.procedure_stats", ct); + + await using var hourlyDataSource = NpgsqlDataSource.Create(scratch.ConnectionString); + foreach (var (ranking, expectedOrder) in new[] + { + (TopRanking.Cpu, "usp_QCPU,usp_QREADS,usp_QEXEC,usp_QDUR,usp_QSURV"), + (TopRanking.Duration, "usp_QDUR,usp_QCPU,usp_QREADS,usp_QEXEC,usp_QSURV"), + (TopRanking.Executions, "usp_QEXEC,usp_QREADS,usp_QCPU,usp_QDUR,usp_QSURV"), + }) + { + var label = "procedures by " + TopRankings.WireName(ranking); + var result = await DarlingDataReader.GetTopProceduresByCpuRoutedAsync( + hourlyDataSource, ServerId, WindowStart, windowEnd, top: 10, databaseName: null, ranking: ranking, cancellationToken: ct); + Assert.True(result.Tier == RetentionTier.Hourly, $"{label}: expected the hourly tier but read {result.Tier}"); + TopRankingLiveTests.AssertOrder(result.Rows.Select(r => r.ObjectName), expectedOrder, label + " (hourly)"); + Assert.Null(result.RetentionNotice); + } + + var reads = await DarlingDataReader.GetTopProceduresByCpuRoutedAsync( + hourlyDataSource, ServerId, WindowStart, windowEnd, top: 10, databaseName: null, ranking: TopRanking.Reads, cancellationToken: ct); + Assert.Equal(RetentionTier.Raw, reads.Tier); + Assert.True(reads.RawForced); + Assert.Equal("usp_QSURV", Assert.Single(reads.Rows).ObjectName); + Assert.NotNull(reads.RetentionNotice); + + var query = "?server=" + ServerName + "&hours=24&as_of=" + Uri.EscapeDataString(windowEnd.ToString("o")); + using (var readsDoc = JsonDocument.Parse(await TopRankingLiveTests.WebReadAsync(hourlyDataSource, "get_top_procedures_by_cpu", query + "&order_by=reads"))) + { + var root = readsDoc.RootElement; + Assert.Equal("raw", root.GetProperty("tier_used").GetString()); + Assert.Contains("order_by=reads needs per-procedure logical reads", root.GetProperty("precision_note").GetString(), StringComparison.Ordinal); + Assert.Equal(JsonValueKind.String, root.GetProperty("retention_notice").ValueKind); + } + + using var durationDoc = JsonDocument.Parse(await TopRankingLiveTests.WebReadAsync(hourlyDataSource, "get_top_procedures_by_cpu", query + "&order_by=duration")); + Assert.Equal("hourly", durationDoc.RootElement.GetProperty("tier_used").GetString()); + Assert.Equal("dbo.usp_QDUR", durationDoc.RootElement.GetProperty("procedures")[0].GetProperty("full_name").GetString()); + }); + + /// + /// The notice, both sides, through the web read dispatch. Raw holds only the last six hours, so a reads ranking over seven days + /// starts before the store's measured raw floor and says the list is partial, while one over three hours starts inside the + /// floor and says nothing, and no other ranking says anything over either window (they read the rollup, which can answer). + /// + [Fact] + public Task RetentionNotice_AReadsRankingStartingBeforeTheRawFloorCarriesIt_AndOneInsideTheFloorDoesNot() => + WithHourlyStoreAsync(async (scratch, connection, ct) => + { + var now = DarlingMcpTestData.Naive(DateTime.UtcNow); + /* Each group is planted twice: at now minus 6 h (the row the seven-day read reaches back to) and at now minus 1 h + (the row the three-hour read finds). #5226 F6: with only the six-hour rows the three-hour window held nothing, the + read answered an empty envelope with no retention_notice key at all, and a notice builder that fired on every + reads read would have passed the "inside the floor" arm. */ + foreach (var seed in TopRankingLiveTests.Seeds) + { + foreach (var hoursBack in new[] { 6, 1 }) + { + await TopRankingLiveTests.PlantQueryAsync(connection, ct, "collect.query_stats", ServerId, ServerName, now.AddHours(-hoursBack), + seed.Name, seed.CpuUs, seed.ElapsedUs, seed.Reads, seed.Executions); + await TopRankingLiveTests.PlantProcedureAsync(connection, ct, "collect.procedure_stats", ServerId, ServerName, now.AddHours(-hoursBack), + seed.Name, seed.CpuUs, seed.ElapsedUs, seed.Reads, seed.Executions); + } + } + + await using var postgres = NpgsqlDataSource.Create(scratch.ConnectionString); + foreach (var (read, rowsKey) in new[] { ("get_top_queries_by_cpu", "queries"), ("get_top_procedures_by_cpu", "procedures") }) + { + var before = NoticeOf(await TopRankingLiveTests.WebReadAsync(postgres, read, $"?server={ServerName}&hours=168&order_by=reads"), rowsKey, read); + Assert.True(before is not null, $"{read}: a reads ranking over seven days starts before raw's floor and must carry the notice"); + Assert.Contains("partial window", before, StringComparison.Ordinal); + Assert.Contains("raw tier", before, StringComparison.Ordinal); + + /* NoticeOf asserts the answer holds rows, so a null here is a data answer that carries no notice, not an empty envelope. */ + var inside = NoticeOf(await TopRankingLiveTests.WebReadAsync(postgres, read, $"?server={ServerName}&hours=3&order_by=reads"), rowsKey, read); + Assert.True(inside is null, $"{read}: a reads ranking that starts inside raw's floor carries no notice, but read: {inside}"); + + foreach (var other in new[] { "", "&order_by=cpu", "&order_by=duration", "&order_by=executions" }) + { + var notice = NoticeOf(await TopRankingLiveTests.WebReadAsync(postgres, read, $"?server={ServerName}&hours=168{other}"), rowsKey, read + other); + Assert.True(notice is null, $"{read}{other}: only the reads ranking carries the notice, but read: {notice}"); + } + } + }); + + // ---------------------------------------------------------------- plumbing + + /// + /// The notice a DATA answer carries, or null when it carries none. The payload must hold at least one row under + /// and its retention_notice must be present: a JSON null or a string. An empty envelope + /// (no rows, no retention_notice key) fails here instead of reading as "no notice", which is what let the + /// "carries no notice" arms pass without reading anything (#5226 F6). + /// + private static string? NoticeOf(string json, string rowsKey, string label) + { + using var doc = JsonDocument.Parse(json); + var root = doc.RootElement; + Assert.True( + root.TryGetProperty(rowsKey, out var rows) && rows.ValueKind == JsonValueKind.Array && rows.GetArrayLength() > 0, + $"{label}: expected a data answer with rows under '{rowsKey}', but read: {json[..Math.Min(json.Length, 300)]}"); + Assert.True(root.TryGetProperty("retention_notice", out var notice), $"{label}: a data answer carries a retention_notice key"); + Assert.True(notice.ValueKind is JsonValueKind.Null or JsonValueKind.String, $"{label}: retention_notice is null or a string"); + return notice.ValueKind == JsonValueKind.String ? notice.GetString() : null; + } + + /// Deletes the seed's raw rows (the first three hours of the window) and leaves the survivor. + private static async Task PurgeSeedAsync(NpgsqlConnection connection, string table, CancellationToken ct) + { + await using var purge = new NpgsqlCommand( + $"DELETE FROM {table} WHERE server_id = $1 AND collection_time >= $2 AND collection_time < $3", connection); + purge.Parameters.AddWithValue(ServerId); + purge.Parameters.AddWithValue(WindowStart); + purge.Parameters.AddWithValue(WindowStart.AddHours(3)); + await purge.ExecuteNonQueryAsync(ct); + } + + private static async Task RefreshAsync(NpgsqlConnection connection, string view, DateTime from, DateTime to, CancellationToken ct) + { + await using var refresh = new NpgsqlCommand($"CALL refresh_continuous_aggregate('collect.{view}'::regclass, $1::timestamp, $2::timestamp)", connection); + refresh.Parameters.AddWithValue(from); + refresh.Parameters.AddWithValue(to); + await refresh.ExecuteNonQueryAsync(ct); + } + + /// Mints a scratch TimescaleDB store with the continuous aggregates and this class's server registered, and runs the body on it. + private static async Task WithHourlyStoreAsync(Func body) + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live #5226 hourly ranking tests."); + var ct = TestContext.Current.CancellationToken; + + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + + var timescaleEnabled = await TimescaleSupport.TryEnableAsync(connection, null, ct); + Assert.SkipWhen(!timescaleEnabled, "The live #5226 hourly ranking tests need TimescaleDB."); + await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct); + Assert.True(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, null, ct)); + + await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection)) + { + await stop.ExecuteNonQueryAsync(ct); + } + + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, ct); + + var bodySucceeded = false; + try + { + await body(scratch, connection, ct); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (cleanup, cleanupCt) => + { + await using var probe = new NpgsqlCommand( + "SELECT count(*) FROM pg_catalog.pg_stat_activity WHERE datname = pg_catalog.current_database() " + + "AND backend_type LIKE 'TimescaleDB Background Worker Scheduler%'", cleanup); + var schedulers = Convert.ToInt64(await probe.ExecuteScalarAsync(cleanupCt)); + Assert.Equal(0L, schedulers); + }); + } + } +} diff --git a/Darling/Darling.Tests/TopRankingLiveTests.cs b/Darling/Darling.Tests/TopRankingLiveTests.cs new file mode 100644 index 0000000000..458102754c --- /dev/null +++ b/Darling/Darling.Tests/TopRankingLiveTests.cs @@ -0,0 +1,394 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Darling.Tests; +using Microsoft.AspNetCore.Http; +using Microsoft.Extensions.DependencyInjection; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace PerformanceMonitor.Darling.Tests; + +/// +/// #5226 live pins for the ranking choice the web's Top Queries and Top Procedures offer (CPU, duration, reads, +/// executions), on the raw tier of the shared PostgreSQL store, where a store with no rollups always answers raw. +/// The readers are called directly and through the web read dispatch, so a swap that lands in the SQL and a +/// order_by that never reaches it both fail here. The hourly path, the reads route and the retention notice +/// need rollups and live in , on a store of their own. +/// +/// Why the seed is shaped this way. Each of the four groups wins exactly one ranking, so a swap that +/// does nothing (every choice answering CPU) and a swap that sorts on the wrong column both change a first row. +/// The delta-versus-cumulative seed gives one group a huge lifetime total_logical_reads and a small window +/// delta and another the reverse: a reads ranking that sums the cumulative column puts the wrong one first. +/// +[Collection("live-postgres")] +public sealed class TopRankingLiveTests +{ + private const string ServerName = "darling-top-ranking-raw-e2e"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + + /// The database name every planted row carries. + internal const string Db = "TopRankDb"; + + /// One planted group: the four metrics the rankings sort on (microseconds for CPU and elapsed time). + internal readonly record struct Seed(string Name, long CpuUs, long ElapsedUs, long Reads, long Executions); + + /// + /// The seed that separates the four rankings. By CPU the first row is QCPU, by duration QDUR, by reads QREADS and + /// by executions QEXEC. QDUR is the case the issue is about: eight seconds of elapsed time on a tenth of a second + /// of CPU (blocked, or waiting on I/O), which a CPU-only list never shows near the top. + /// + internal static readonly Seed[] Seeds = + [ + new("QCPU", 9_000_000, 1_000_000, 100, 10), + new("QDUR", 100_000, 8_000_000, 50, 5), + new("QREADS", 500_000, 600_000, 900_000, 20), + new("QEXEC", 300_000, 400_000, 200, 5_000), + ]; + + /// A procedure group's name for a seed name. + internal static string ProcName(string name) => "usp_" + name; + + /// A query group's hash for a seed name. + internal static string QueryHash(string name) => "0x" + name; + + // ---------------------------------------------------------------- the choices on the raw path + + [Theory] + [InlineData(TopRanking.Cpu, "0xQCPU,0xQREADS,0xQEXEC,0xQDUR")] + [InlineData(TopRanking.Duration, "0xQDUR,0xQCPU,0xQREADS,0xQEXEC")] + [InlineData(TopRanking.Reads, "0xQREADS,0xQEXEC,0xQCPU,0xQDUR")] + [InlineData(TopRanking.Executions, "0xQEXEC,0xQREADS,0xQCPU,0xQDUR")] + public Task Queries_OnTheRawPath_EachChoiceRanksItsOwnMetric(TopRanking ranking, string expectedOrder) => + WithSharedStoreAsync(async (connection, postgres, now, ct) => + { + await PlantSeedsAsync(connection, ct, now); + var result = await DarlingDataReader.GetTopQueriesByCpuRoutedAsync( + postgres, ServerId, now.AddHours(-24), now.AddMinutes(5), top: 10, databaseName: null, ranking: ranking, cancellationToken: ct); + Assert.Equal(RetentionTier.Raw, result.Tier); + AssertOrder(result.Rows.Select(r => r.QueryHash), expectedOrder, "queries by " + TopRankings.WireName(ranking)); + }); + + [Theory] + [InlineData(TopRanking.Cpu, "usp_QCPU,usp_QREADS,usp_QEXEC,usp_QDUR")] + [InlineData(TopRanking.Duration, "usp_QDUR,usp_QCPU,usp_QREADS,usp_QEXEC")] + [InlineData(TopRanking.Reads, "usp_QREADS,usp_QEXEC,usp_QCPU,usp_QDUR")] + [InlineData(TopRanking.Executions, "usp_QEXEC,usp_QREADS,usp_QCPU,usp_QDUR")] + public Task Procedures_OnTheRawPath_EachChoiceRanksItsOwnMetric(TopRanking ranking, string expectedOrder) => + WithSharedStoreAsync(async (connection, postgres, now, ct) => + { + await PlantSeedsAsync(connection, ct, now); + var result = await DarlingDataReader.GetTopProceduresByCpuRoutedAsync( + postgres, ServerId, now.AddHours(-24), now.AddMinutes(5), top: 10, databaseName: null, ranking: ranking, cancellationToken: ct); + Assert.Equal(RetentionTier.Raw, result.Tier); + AssertOrder(result.Rows.Select(r => r.ObjectName), expectedOrder, "procedures by " + TopRankings.WireName(ranking)); + }); + + // ---------------------------------------------------------------- the issue's own case + + /// + /// The case #5226 exists for: a query that ran long and used little CPU ranks first by duration and not by CPU. + /// This is the test shown RED by making TopRankings.Apply return its input unchanged. + /// + [Fact] + public Task Queries_ALongQueryWithLittleCpu_RanksFirstByDuration_AndNotByCpu() => + WithSharedStoreAsync(async (connection, postgres, now, ct) => + { + await PlantSeedsAsync(connection, ct, now); + var byCpu = await DarlingDataReader.GetTopQueriesByCpuAsync( + postgres, ServerId, now.AddHours(-24), now.AddMinutes(5), top: 10, databaseName: null, ranking: TopRanking.Cpu, cancellationToken: ct); + var byDuration = await DarlingDataReader.GetTopQueriesByCpuAsync( + postgres, ServerId, now.AddHours(-24), now.AddMinutes(5), top: 10, databaseName: null, ranking: TopRanking.Duration, cancellationToken: ct); + + Assert.Equal(QueryHash("QCPU"), byCpu[0].QueryHash); + Assert.Equal(QueryHash("QDUR"), byCpu[^1].QueryHash); + var first = byDuration[0]; + Assert.Equal(QueryHash("QDUR"), first.QueryHash); + Assert.Equal(8_000_000L, first.TotalElapsedUs); + Assert.Equal(100_000L, first.TotalCpuUs); + }); + + /// The same case on the procedures grid. + [Fact] + public Task Procedures_ALongProcedureWithLittleCpu_RanksFirstByDuration_AndNotByCpu() => + WithSharedStoreAsync(async (connection, postgres, now, ct) => + { + await PlantSeedsAsync(connection, ct, now); + var byCpu = await DarlingDataReader.GetTopProceduresByCpuAsync( + postgres, ServerId, now.AddHours(-24), now.AddMinutes(5), top: 10, databaseName: null, ranking: TopRanking.Cpu, cancellationToken: ct); + var byDuration = await DarlingDataReader.GetTopProceduresByCpuAsync( + postgres, ServerId, now.AddHours(-24), now.AddMinutes(5), top: 10, databaseName: null, ranking: TopRanking.Duration, cancellationToken: ct); + + Assert.Equal(ProcName("QCPU"), byCpu[0].ObjectName); + Assert.Equal(ProcName("QDUR"), byCpu[^1].ObjectName); + var first = byDuration[0]; + Assert.Equal(ProcName("QDUR"), first.ObjectName); + Assert.Equal(8_000_000L, first.TotalElapsedUs); + Assert.Equal(100_000L, first.TotalCpuUs); + }); + + // ---------------------------------------------------------------- delta versus cumulative + + /// + /// Reads ranks on the window's delta_logical_reads, never the lifetime total_logical_reads. QCUM has the + /// huge cumulative total and a small delta; QDELTA the reverse, spread over three snapshots. The ask is for ONE row (the statement over-fetches + /// five more than that, then re-ranks the page by the summed delta), so six fillers whose cumulative totals beat + /// QDELTA's fill that page when the inner sort reads the cumulative column: with it QDELTA never reaches the outer + /// re-rank at all. This is the test shown RED by pointing TopRankings.Apply's reads metric at total_logical_reads. + /// + [Fact] + public Task Queries_Reads_SumsTheWindowsDeltas_NotTheCumulativeLifetimeTotal() => + WithSharedStoreAsync(async (connection, postgres, now, ct) => + { + await PlantCumulativeSeedAsync(connection, ct, now); + var rows = await DarlingDataReader.GetTopQueriesByCpuAsync( + postgres, ServerId, now.AddHours(-24), now.AddMinutes(5), top: 1, databaseName: null, ranking: TopRanking.Reads, cancellationToken: ct); + + var first = Assert.Single(rows); + Assert.Equal(QueryHash("QDELTA"), first.QueryHash); + Assert.Equal(5_100L, first.TotalLogicalReads); + }); + + /// The same pin on the procedures grid, whose single-level statement sorts and caps in one place. + [Fact] + public Task Procedures_Reads_SumsTheWindowsDeltas_NotTheCumulativeLifetimeTotal() => + WithSharedStoreAsync(async (connection, postgres, now, ct) => + { + await PlantCumulativeSeedAsync(connection, ct, now); + var rows = await DarlingDataReader.GetTopProceduresByCpuAsync( + postgres, ServerId, now.AddHours(-24), now.AddMinutes(5), top: 1, databaseName: null, ranking: TopRanking.Reads, cancellationToken: ct); + + var first = Assert.Single(rows); + Assert.Equal(ProcName("QDELTA"), first.ObjectName); + Assert.Equal(5_100L, first.TotalLogicalReads); + }); + + // ---------------------------------------------------------------- the web read dispatch + + /// + /// The page's order_by reaches the ranking through the web read dispatch: the served list's first row follows + /// the choice, and a request that names none is the CPU list it always was. The readers above cannot see a + /// dispatch that stops passing the choice down. + /// + [Fact] + public Task Endpoint_OrderBy_RanksTheServedList_AndAnAbsentValueIsCpu() => + WithSharedStoreAsync(async (connection, postgres, now, ct) => + { + await PlantSeedsAsync(connection, ct, now); + foreach (var (orderBy, firstQuery, firstProcedure) in new[] + { + ("", "0xQCPU", "dbo.usp_QCPU"), + ("cpu", "0xQCPU", "dbo.usp_QCPU"), + ("duration", "0xQDUR", "dbo.usp_QDUR"), + ("reads", "0xQREADS", "dbo.usp_QREADS"), + ("executions", "0xQEXEC", "dbo.usp_QEXEC"), + }) + { + var query = "?server=" + ServerName + "&hours=24" + (orderBy.Length == 0 ? "" : "&order_by=" + orderBy); + + using var queries = JsonDocument.Parse(await WebReadAsync(postgres, "get_top_queries_by_cpu", query)); + Assert.Equal(firstQuery, queries.RootElement.GetProperty("queries")[0].GetProperty("query_hash").GetString()); + + using var procedures = JsonDocument.Parse(await WebReadAsync(postgres, "get_top_procedures_by_cpu", query)); + Assert.Equal(firstProcedure, procedures.RootElement.GetProperty("procedures")[0].GetProperty("full_name").GetString()); + } + }); + + /// + /// An order_by the whitelist does not hold is refused, never read as CPU: the read dispatch returns the + /// invalid envelope, which calls a refusal and + /// answers with a 400. The four spellings, in any case, are served. + /// The statement text never carries the request value, so the table the hostile spelling names is still there. + /// + [Fact] + public Task Endpoint_AnUnknownOrderBy_IsRefused_As400_OnBothReads() => + WithSharedStoreAsync(async (connection, postgres, now, ct) => + { + await PlantSeedsAsync(connection, ct, now); + foreach (var read in new[] { "get_top_queries_by_cpu", "get_top_procedures_by_cpu" }) + { + foreach (var bad in new[] { "bogus", "cpu; DROP TABLE query_stats", "total_cpu_us", "worker_time" }) + { + var body = await WebReadAsync(postgres, read, "?server=" + ServerName + "&hours=24&order_by=" + Uri.EscapeDataString(bad)); + Assert.Equal(DarlingWebEndpoints.ToolResponseKind.Refusal, DarlingWebEndpoints.ClassifyToolResponse(body)); + Assert.Equal(StatusCodes.Status400BadRequest, HttpStatusOf(read, body)); + Assert.Contains("order_by must be cpu, duration, reads or executions", body, StringComparison.Ordinal); + } + + foreach (var good in new[] { "cpu", "Duration", "READS", "executions" }) + { + var body = await WebReadAsync(postgres, read, "?server=" + ServerName + "&hours=24&order_by=" + good); + Assert.Equal(DarlingWebEndpoints.ToolResponseKind.JsonPassthrough, DarlingWebEndpoints.ClassifyToolResponse(body)); + Assert.Equal(StatusCodes.Status200OK, HttpStatusOf(read, body)); + } + } + + await using var count = new NpgsqlCommand("SELECT count(*) FROM query_stats WHERE server_id = $1", connection); + count.Parameters.AddWithValue(ServerId); + Assert.Equal((long)Seeds.Length * SnapshotsPerGroup, Convert.ToInt64(await count.ExecuteScalarAsync(ct))); + }); + + // ---------------------------------------------------------------- planting and plumbing + + /// + /// Plants the four-group seed on both grids, two hours back, with the interval an honest collection carries. Each group + /// lands as THREE snapshots a minute apart whose halves, quarters and remainder add up to the seed's totals (#5226 F7): + /// with one row per group SUM, MAX, AVG and "the last value" all answer the same, so a ranking that stopped summing would + /// still pass. is how many rows a group holds. + /// + private static async Task PlantSeedsAsync(NpgsqlConnection connection, CancellationToken ct, DateTime now) + { + foreach (var seed in Seeds) + { + for (var i = 0; i < SnapshotsPerGroup; i++) + { + var at = now.AddHours(-2).AddMinutes(i); + await PlantQueryAsync(connection, ct, "query_stats", ServerId, ServerName, at, seed.Name, + Part(seed.CpuUs, i), Part(seed.ElapsedUs, i), Part(seed.Reads, i), Part(seed.Executions, i)); + await PlantProcedureAsync(connection, ct, "procedure_stats", ServerId, ServerName, at, seed.Name, + Part(seed.CpuUs, i), Part(seed.ElapsedUs, i), Part(seed.Reads, i), Part(seed.Executions, i)); + } + } + } + + /// How many snapshots plants per group. + internal const int SnapshotsPerGroup = 3; + + /// Snapshot of a total split in halves, quarters and the remainder; the three parts add up to it. + internal static long Part(long total, int index) => index switch + { + 0 => total / 2, + 1 => total / 4, + _ => total - total / 2 - total / 4, + }; + + /// + /// QCUM: a delta of 10 against a lifetime total of fifty billion. QDELTA: three snapshots of 1,700 (a window sum of + /// 5,100) against a lifetime total of 1,000. QMAXDECOY: ONE snapshot of 4,000, so only a SUM puts QDELTA first (its MAX, + /// its AVG and its last value are all 1,700, under the decoy; #5226 F7). Six fillers sit between them on the cumulative + /// column (more than QDELTA's total, a window delta under 110), so a read that ranks on the lifetime total fills the + /// over-fetched page without QDELTA. + /// + private static async Task PlantCumulativeSeedAsync(NpgsqlConnection connection, CancellationToken ct, DateTime now) + { + var at = now.AddHours(-2); + await PlantQueryAsync(connection, ct, "query_stats", ServerId, ServerName, at, "QCUM", 10_000, 10_000, 10, 5, cumulativeReads: 50_000_000_000L); + await PlantProcedureAsync(connection, ct, "procedure_stats", ServerId, ServerName, at, "QCUM", 10_000, 10_000, 10, 5, cumulativeReads: 50_000_000_000L); + for (var i = 0; i < 3; i++) + { + await PlantQueryAsync(connection, ct, "query_stats", ServerId, ServerName, at.AddMinutes(i), "QDELTA", 10_000, 10_000, 1_700, 5, cumulativeReads: 1_000L); + await PlantProcedureAsync(connection, ct, "procedure_stats", ServerId, ServerName, at.AddMinutes(i), "QDELTA", 10_000, 10_000, 1_700, 5, cumulativeReads: 1_000L); + } + + await PlantQueryAsync(connection, ct, "query_stats", ServerId, ServerName, at, "QMAXDECOY", 10_000, 10_000, 4_000, 5, cumulativeReads: 500L); + await PlantProcedureAsync(connection, ct, "procedure_stats", ServerId, ServerName, at, "QMAXDECOY", 10_000, 10_000, 4_000, 5, cumulativeReads: 500L); + for (var i = 1; i <= 6; i++) + { + await PlantQueryAsync(connection, ct, "query_stats", ServerId, ServerName, at, "QFILL" + i, 10_000, 10_000, 100 + i, 5, cumulativeReads: 1_000_000L + i); + await PlantProcedureAsync(connection, ct, "procedure_stats", ServerId, ServerName, at, "QFILL" + i, 10_000, 10_000, 100 + i, 5, cumulativeReads: 1_000_000L + i); + } + } + + /// One query_stats row. is the store's spelling (shared: bare; scratch: collect.-qualified). + internal static Task PlantQueryAsync( + NpgsqlConnection connection, CancellationToken ct, string table, int serverId, string serverName, DateTime at, + string name, long cpuUs, long elapsedUs, long reads, long executions, long? cumulativeReads = null) => + DarlingMcpTestData.ExecAsync(connection, ct, + $@"INSERT INTO {table} + (collection_id, collection_time, server_id, server_name, database_name, query_hash, sql_handle, + delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, total_logical_reads, + sample_interval_seconds) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)", + CollectionIdGenerator.Next(), DarlingMcpTestData.TruncateToSeconds(at), serverId, serverName, Db, + QueryHash(name), "0xSQLH" + name, executions, cpuUs, elapsedUs, reads, (object?)cumulativeReads ?? DBNull.Value, 3600); + + /// One procedure_stats row, in schema dbo; as for . + internal static Task PlantProcedureAsync( + NpgsqlConnection connection, CancellationToken ct, string table, int serverId, string serverName, DateTime at, + string name, long cpuUs, long elapsedUs, long reads, long executions, long? cumulativeReads = null) => + DarlingMcpTestData.ExecAsync(connection, ct, + $@"INSERT INTO {table} + (collection_id, collection_time, server_id, server_name, database_name, schema_name, object_name, sql_handle, + delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, total_logical_reads, + sample_interval_seconds) + VALUES ($1, $2, $3, $4, $5, 'dbo', $6, $7, $8, $9, $10, $11, $12, $13)", + CollectionIdGenerator.Next(), DarlingMcpTestData.TruncateToSeconds(at), serverId, serverName, Db, + ProcName(name), "0xSQLH" + name, executions, cpuUs, elapsedUs, reads, (object?)cumulativeReads ?? DBNull.Value, 3600); + + /// Fails with the order actually read, so a RED run shows what the swap did to it. + internal static void AssertOrder(IEnumerable actual, string expectedOrder, string label) + { + var got = string.Join(",", actual); + Assert.True(got == expectedOrder, $"{label}: expected the order {expectedOrder} but read {got}"); + } + + /// A read through the web dispatch, the way the page's request reaches it. + internal static async Task WebReadAsync(NpgsqlDataSource postgres, string read, string query) + { + var context = new DefaultHttpContext(); + context.Request.QueryString = new QueryString(query); + return await DarlingWebEndpoints.BuildReadDispatch()[read](context, postgres, null!); + } + + /// The HTTP status the web answers a read's returned string with. + internal static int HttpStatusOf(string read, string toolResult) + { + var services = new ServiceCollection(); + services.AddOptions(); + services.AddLogging(); + using var provider = services.BuildServiceProvider(); + var context = new DefaultHttpContext { RequestServices = provider }; + context.Response.Body = new MemoryStream(); + DarlingWebEndpoints.ToHttpResult(toolResult, "/api/read/" + read, new CapturingTestLogger(), 1) + .ExecuteAsync(context).GetAwaiter().GetResult(); + return context.Response.StatusCode; + } + + /// Runs a body against the shared store with this class's server registered, and leaves nothing behind. + private static async Task WithSharedStoreAsync(Func body) + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), "Set DARLING_TEST_PG to run the live top-ranking tests."); + var ct = TestContext.Current.CancellationToken; + + using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await CleanupAsync(connection, ct); + + await using var postgres = NpgsqlDataSource.Create(connectionString!); + var succeeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + await body(connection, postgres, DarlingMcpTestData.Naive(DateTime.UtcNow), ct); + succeeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, succeeded, async (cleanup, cleanupCt) => + await CleanupAsync(cleanup, cleanupCt)); + } + } + + private static async Task CleanupAsync(NpgsqlConnection connection, CancellationToken ct) => + await DarlingMcpTestData.ExecAsync(connection, ct, + $"DELETE FROM query_stats WHERE server_id = {ServerId}; DELETE FROM procedure_stats WHERE server_id = {ServerId}; DELETE FROM servers WHERE server_id = {ServerId}"); +} diff --git a/Darling/Darling.Tests/TopRankingNoticeBehaviourTests.cs b/Darling/Darling.Tests/TopRankingNoticeBehaviourTests.cs new file mode 100644 index 0000000000..046603a78e --- /dev/null +++ b/Darling/Darling.Tests/TopRankingNoticeBehaviourTests.cs @@ -0,0 +1,152 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.ComponentModel; +using System.Diagnostics; +using System.Linq; +using System.Text.Json; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// #5299 (F9): a Reads ranking whose window reaches past what raw keeps comes back with two sentences that say the same thing, the window +/// floor's truncation_note (this server's raw rows start after the window does) and the reads ranking's retention_notice +/// (the raw route's partial-window notice, with the store's measured reach). The page draws ONE notice, the retention notice, on each of the +/// four Top Queries and Top Procedures cards (the CPU tab's two grids, the Queries tab's composite and its procedures grid), as text. The +/// shipped page runs under Node (top-ranking-notice-harness.mjs); Node is skipped when it is not installed. +/// +public sealed class TopRankingNoticeBehaviourTests +{ + /// The window-floor sentence get_top_queries_by_cpu writes for a raw-tier answer (DarlingMcpDataTools, the queries tool's truncation_note). + private const string FloorNote = + "The window reaches further back than this server's raw query_stats retains (or this server has been monitored for less time than that), so the older part of it was not read."; + + private static readonly string[] Cards = ["cpuQueries", "cpuProcedures", "queriesQueries", "queriesProcedures"]; + + /// The retention notice a store with rollups gives a seven-day window over raw, written by the server's own builder, not copied. + private static string RetentionNotice() + { + var now = new DateTime(2026, 7, 23, 12, 0, 0, DateTimeKind.Utc); + return ComposeStoreAvailability.BuildRetentionNotice( + "query_stats", ComposeRoute.Raw, now.AddDays(-7), now, RollupAvailability.All, RollupCoverage.Unknown)!; + } + + private static JsonElement Run(string scenario) + { + var psi = new ProcessStartInfo("node") { RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false }; + psi.Environment["HARNESS_INPUT"] = JsonSerializer.Serialize(new { floor = FloorNote, retention = RetentionNotice() }); + psi.ArgumentList.Add(PathTo("Darling", "Darling.Tests", "top-ranking-notice-harness.mjs")); + psi.ArgumentList.Add(PathTo("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js")); + psi.ArgumentList.Add(scenario); + + Process proc; + try + { + proc = Process.Start(psi)!; + } + catch (Win32Exception) + { + Assert.Skip("Node is not installed, so the shipped page script cannot be run."); + return default; + } + + using (proc) + { + var error = proc.StandardError.ReadToEndAsync(); + var output = proc.StandardOutput.ReadToEnd().Trim(); + if (!proc.WaitForExit(30000)) + { + proc.Kill(entireProcessTree: true); + Assert.Fail("the Top Queries and Top Procedures notice harness did not finish in 30 s for scenario " + scenario); + } + + Assert.True(proc.ExitCode == 0, "the notice harness failed for scenario " + scenario + ": " + error.Result); + using var doc = JsonDocument.Parse(output.Split('\n').First(l => l.StartsWith('{'))); + return doc.RootElement.Clone(); + } + } + + private static string[] NoticesOf(JsonElement run, string card) + { + var drawn = run.GetProperty("cards").GetProperty(card); + Assert.True(drawn.GetProperty("found").GetBoolean(), card + " must be drawn"); + return drawn.GetProperty("notices").EnumerateArray().Select(e => e.GetString()!).ToArray(); + } + + /// + /// The case the review found: the window starts before raw's reach, so the answer carries both sentences. Each card draws one, the + /// retention notice in the server's own words, and not the floor sentence beside it. The cards are on the Reads ranking, and the + /// page asked for it (order_by=reads) on both reads. + /// + [Fact] + public void AReadsPagePastRaw_DrawsOneNotice_OnEveryRankedCard() + { + var run = Run("both"); + foreach (var card in Cards) + { + Assert.Equal([RetentionNotice()], NoticesOf(run, card)); + Assert.EndsWith("by Reads", run.GetProperty("cards").GetProperty(card).GetProperty("title").GetString()); + } + + var asks = run.GetProperty("asks").EnumerateArray().Select(e => e.GetString()!).ToList(); + Assert.True(asks.Count(a => a == "get_top_queries_by_cpu:reads") >= 2, "both Top Queries cards ask for the reads ranking: " + string.Join(", ", asks)); + Assert.True(asks.Count(a => a == "get_top_procedures_by_cpu:reads") >= 2, "both Top Procedures cards ask for the reads ranking: " + string.Join(", ", asks)); + } + + /// An answer with only the window floor's note (a store with no rollups, or a server monitored for less time than the window) still draws that one note. + [Fact] + public void AFloorNoteAlone_IsDrawnOnce() + { + var run = Run("floorOnly"); + foreach (var card in Cards) + { + Assert.Equal([FloorNote], NoticesOf(run, card)); + } + } + + /// An answer with only the retention notice is drawn once, so the reads ranking is never silently partial. + [Fact] + public void ARetentionNoticeAlone_IsDrawnOnce() + { + var run = Run("retentionOnly"); + foreach (var card in Cards) + { + Assert.Equal([RetentionNotice()], NoticesOf(run, card)); + } + } + + /// An answer that carries neither draws no notice line. + [Fact] + public void AnAnswerWithNeither_DrawsNoNotice() + { + var run = Run("neither"); + foreach (var card in Cards) + { + Assert.Empty(NoticesOf(run, card)); + } + } + + /// The notice is a server value, so it is drawn as text: markup in it stays literal characters inside one text node. + [Fact] + public void TheNotice_IsDrawnAsText_NeverMarkup() + { + var run = Run("markup"); + foreach (var card in Cards) + { + var notices = NoticesOf(run, card); + Assert.Single(notices); + Assert.Contains("bold", notices[0], StringComparison.Ordinal); + Assert.True(run.GetProperty("cards").GetProperty(card).GetProperty("textOnly").GetBoolean(), card + ": the notice must hold a text node and no element"); + } + } +} diff --git a/Darling/Darling.Tests/TopRankingOnePassBaseline.cs b/Darling/Darling.Tests/TopRankingOnePassBaseline.cs new file mode 100644 index 0000000000..bef691cf45 --- /dev/null +++ b/Darling/Darling.Tests/TopRankingOnePassBaseline.cs @@ -0,0 +1,405 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using PerformanceMonitor.Darling.Service.Mcp; + +namespace PerformanceMonitor.Darling.Tests; + +/// +/// The ONE-PASS raw top-N statements as they stood before #5226's two-pass restructure (the reader at commit +/// d0a486829), kept as the oracle compares the shipped reader against. +/// The statement bodies are copied verbatim, comments included, with exactly one edit: the interpolated +/// TimescaleSupport.IntervalHonestSourceFilter is written out as its value, so a later change to that constant +/// cannot move the oracle. They are NOT the shipped statements and must not be edited to track them. +/// +/// The one deliberate difference. The old statements ranked with a bare SUM(x) DESC, which sorts NULL +/// first in PostgreSQL and leaves ties in plan order. The two-pass work changed that on purpose: every deciding +/// ORDER BY is now DESC NULLS LAST on the ranking metric and then on CPU, ending in the group key. +/// applies that same ordering to the old text (and expands the ranking the way the old +/// ranking swap did), so the comparison isolates the restructure and nothing else: a different set of rows, a +/// dropped group or a different totals row is a failure, and so is a different tie order. +/// +internal static class TopRankingOnePassBaseline +{ + /// The metric the ranking sorts on: the raw-table sum and the page-level output column. + private static (string Sum, string Output) Metric(TopRanking ranking) => ranking switch + { + TopRanking.Duration => ("SUM(delta_elapsed_time)", "r.total_elapsed_us"), + TopRanking.Reads => ("SUM(delta_logical_reads)", "r.total_reads"), + TopRanking.Executions => ("SUM(delta_execution_count)", "r.total_executions"), + _ => ("SUM(delta_worker_time)", "r.total_cpu_us"), + }; + + /// + /// The old statement for a ranking, with the old inner ORDER BY and the old outer ORDER BY (the procedures statement + /// has no outer one) rewritten to the total order described on the class. and + /// are the group key columns that end each ORDER BY. + /// + public static string Expand(string oldSql, TopRanking ranking, string innerKey, string? outerKey) + { + var (sum, output) = Metric(ranking); + var sql = ReplaceOnce(oldSql, "ORDER BY SUM(delta_worker_time) DESC", + $"ORDER BY {sum} DESC NULLS LAST, SUM(delta_worker_time) DESC NULLS LAST, {innerKey}"); + if (outerKey is not null) + { + sql = ReplaceOnce(sql, "ORDER BY r.total_cpu_us DESC", + $"ORDER BY {output} DESC NULLS LAST, r.total_cpu_us DESC NULLS LAST, {outerKey}"); + } + + return sql; + } + + private static string ReplaceOnce(string sql, string find, string replacement) + { + var at = sql.IndexOf(find, StringComparison.Ordinal); + if (at < 0 || sql.IndexOf(find, at + find.Length, StringComparison.Ordinal) >= 0) + { + throw new InvalidOperationException($"Expected exactly one '{find}' in the one-pass baseline text."); + } + + return string.Concat(sql.AsSpan(0, at), replacement, sql.AsSpan(at + find.Length)); + } + + /// The old TopQueriesSql, the per-hash grouping, for a ranking. + public static string Queries(TopRanking ranking) => Expand(OneQueriesSql, ranking, + "database_name, query_hash, host_object_name", "r.database_name, r.query_hash, r.host_object_name"); + + /// The old TopQueriesByHostObjectSql, the host-object roll-up, for a ranking. + public static string QueriesByHostObject(TopRanking ranking) => Expand(OneHostObjectSql, ranking, + "database_name, host_object_name, CASE WHEN host_object_name IS NULL THEN query_hash END", + "r.database_name, r.host_object_name, r.query_hash"); + + /// The old TopProceduresSql for a ranking. + public static string Procedures(TopRanking ranking) => Expand(OneProceduresSql, ranking, + "database_name, schema_name, object_name, object_type", null); + + /// Old TopQueriesSql at d0a486829, the CPU-ranked text. + private const string OneQueriesSql = """ + WITH ranked AS ( + SELECT + database_name, + query_hash, + host_object_name, + CAST(SUM(delta_execution_count) AS bigint) AS total_executions, + CAST(SUM(delta_worker_time) AS bigint) AS total_cpu_us, + CAST(SUM(delta_elapsed_time) AS bigint) AS total_elapsed_us, + CAST(SUM(delta_logical_reads) AS bigint) AS total_reads, + CAST(SUM(delta_logical_writes) AS bigint) AS total_writes, + CAST(SUM(delta_physical_reads) AS bigint) AS total_physical_reads, + CAST(SUM(delta_rows) AS bigint) AS total_rows, + CAST(SUM(delta_spills) AS bigint) AS total_spills, + MIN(min_dop) AS min_dop, + MAX(max_dop) AS max_dop, + MIN(min_worker_time) AS min_worker_time, + MAX(max_worker_time) AS max_worker_time, + MIN(min_elapsed_time) AS min_elapsed_time, + MAX(max_elapsed_time) AS max_elapsed_time, + MAX(query_plan_hash) AS query_plan_hash, + MAX(sql_handle) AS sql_handle, + MAX(plan_handle) AS plan_handle, + MAX(last_execution_time) AS last_execution_time, + MAX(creation_time) AS creation_time, + MIN(min_physical_reads) AS min_physical_reads, + MAX(max_physical_reads) AS max_physical_reads, + MIN(min_rows) AS min_rows, + MAX(max_rows) AS max_rows, + MIN(min_grant_kb) AS min_grant_kb, + MAX(max_grant_kb) AS max_grant_kb, + MIN(min_used_grant_kb) AS min_used_grant_kb, + MAX(max_used_grant_kb) AS max_used_grant_kb, + MIN(min_ideal_grant_kb) AS min_ideal_grant_kb, + MAX(max_ideal_grant_kb) AS max_ideal_grant_kb, + MIN(min_spills) AS min_spills, + MAX(max_spills) AS max_spills, + MIN(min_reserved_threads) AS min_reserved_threads, + MAX(max_reserved_threads) AS max_reserved_threads, + MIN(min_used_threads) AS min_used_threads, + MAX(max_used_threads) AS max_used_threads, + MAX(total_clr_time) AS total_clr_time, + MAX(plan_generation_num) AS plan_generation_num, + MAX(CAST(delta_worker_time AS double precision) / NULLIF(sample_interval_seconds, 0) / 1000.0) AS worker_time_per_second, + /* #2012: how many DISTINCT statement texts this hash group merged. query_hash is a + SHAPE hash — INSERT...EXEC statements naming DIFFERENT callee procs share one + (reproduced live), and ad-hoc literal variants collapse too — so a group with + distinct_texts > 1 is a BLEND whose representative text below is one member, not + the statement. Counted over the #1767 content digest already on every row (~free); + COUNT(DISTINCT) skips NULLs, so 0 means only pre-dimension legacy rows, which age + out with raw retention. */ + COUNT(DISTINCT query_text_digest) AS distinct_texts + FROM query_stats + WHERE server_id = $1 + AND collection_time >= $2 + AND collection_time <= $3 + AND ($5::text IS NULL OR database_name = $5) + /* #4394: excludes zero-interval rows (sample_interval_seconds = 0) through + TimescaleSupport.IntervalHonestSourceFilter, the same filter the hourly successors + bake into their CREATE, so a raw-served and an hourly-served read of the same window + agree by construction. The collector writes a zero-interval row with zero deltas + (CollectorDeltaCalculator's first-sighting, reset and gap cases), so this changes no + total in practice. It keeps the two tiers from disagreeing if that ever stops holding. */ + AND sample_interval_seconds IS DISTINCT FROM 0 + /* #2012 stage 2: host_object_name splits INSERT...EXEC callers that share a query_hash + (each proc-hosted statement groups under its own host object), while ad-hoc rows carry + NULL and keep collapsing into one group per hash exactly as before. */ + GROUP BY database_name, query_hash, host_object_name + HAVING (SUM(delta_execution_count) > 0 OR SUM(delta_elapsed_time) > 0) + /* #3541 A13: the parallelism filter is part of the QUERY, applied to the grouped population + BEFORE the CPU ranking and the cap. It used to run in C# over the returned top-N page, so + parallel_only=true on a box whose twenty hottest plans were serial answered an empty page while + the window held parallel plans further down — and the engine's own CXPACKET advice sends agents + to exactly that call. $6 is the group's lifetime max_dop floor: 0 admits every group (the + unfiltered read, byte-identical in result to before), 2 is parallel_only, min_dop is itself. The + COALESCE keeps a group whose max_dop was never captured (NULL) out of a filtered page, which is + what the C# arm did too (null read as 0, and 0 > 1 is false). */ + AND COALESCE(MAX(max_dop), 0) >= $6 + ORDER BY SUM(delta_worker_time) DESC + LIMIT $4 + 5 + ) + SELECT + r.database_name, + r.query_hash, + r.host_object_name, + r.query_plan_hash, + r.sql_handle, + r.plan_handle, + r.total_executions, + r.total_cpu_us, + r.total_elapsed_us, + r.total_reads, + r.total_writes, + r.total_physical_reads, + r.total_rows, + r.total_spills, + r.min_dop, + r.max_dop, + r.min_worker_time, + r.max_worker_time, + r.min_elapsed_time, + r.max_elapsed_time, + t.query_text, + r.distinct_texts, + CAST(1 AS bigint) AS distinct_query_hashes, + r.last_execution_time, + r.creation_time, + r.min_physical_reads, + r.max_physical_reads, + r.min_rows, + r.max_rows, + r.min_grant_kb, + r.max_grant_kb, + r.min_used_grant_kb, + r.max_used_grant_kb, + r.min_ideal_grant_kb, + r.max_ideal_grant_kb, + r.min_spills, + r.max_spills, + r.min_reserved_threads, + r.max_reserved_threads, + r.min_used_threads, + r.max_used_threads, + r.total_clr_time, + r.plan_generation_num, + r.worker_time_per_second + FROM ranked AS r + LEFT JOIN LATERAL ( + SELECT query_text + FROM v_query_stats + WHERE server_id = $1 + AND query_hash = r.query_hash + AND database_name = r.database_name + /* #2012 stage 2: the representative text must come from THIS group's own rows — before + this, the lookup could serve one caller's text for another caller's stats, which is + the exact mis-attribution the issue documents from live triage. */ + AND host_object_name IS NOT DISTINCT FROM r.host_object_name + AND query_text IS NOT NULL + ORDER BY collection_time DESC + LIMIT 1 + ) AS t ON TRUE + WHERE t.query_text IS NULL OR t.query_text NOT LIKE 'WAITFOR%' + ORDER BY r.total_cpu_us DESC + LIMIT $4 + """; + + /// Old TopQueriesByHostObjectSql at d0a486829, the CPU-ranked text. + private const string OneHostObjectSql = """ + WITH ranked AS ( + SELECT + database_name, + MAX(query_hash) AS query_hash, + host_object_name, + CAST(SUM(delta_execution_count) AS bigint) AS total_executions, + CAST(SUM(delta_worker_time) AS bigint) AS total_cpu_us, + CAST(SUM(delta_elapsed_time) AS bigint) AS total_elapsed_us, + CAST(SUM(delta_logical_reads) AS bigint) AS total_reads, + CAST(SUM(delta_logical_writes) AS bigint) AS total_writes, + CAST(SUM(delta_physical_reads) AS bigint) AS total_physical_reads, + CAST(SUM(delta_rows) AS bigint) AS total_rows, + CAST(SUM(delta_spills) AS bigint) AS total_spills, + MIN(min_dop) AS min_dop, + MAX(max_dop) AS max_dop, + MIN(min_worker_time) AS min_worker_time, + MAX(max_worker_time) AS max_worker_time, + MIN(min_elapsed_time) AS min_elapsed_time, + MAX(max_elapsed_time) AS max_elapsed_time, + MAX(query_plan_hash) AS query_plan_hash, + MAX(sql_handle) AS sql_handle, + MAX(plan_handle) AS plan_handle, + MAX(last_execution_time) AS last_execution_time, + MAX(creation_time) AS creation_time, + MIN(min_physical_reads) AS min_physical_reads, + MAX(max_physical_reads) AS max_physical_reads, + MIN(min_rows) AS min_rows, + MAX(max_rows) AS max_rows, + MIN(min_grant_kb) AS min_grant_kb, + MAX(max_grant_kb) AS max_grant_kb, + MIN(min_used_grant_kb) AS min_used_grant_kb, + MAX(max_used_grant_kb) AS max_used_grant_kb, + MIN(min_ideal_grant_kb) AS min_ideal_grant_kb, + MAX(max_ideal_grant_kb) AS max_ideal_grant_kb, + MIN(min_spills) AS min_spills, + MAX(max_spills) AS max_spills, + MIN(min_reserved_threads) AS min_reserved_threads, + MAX(max_reserved_threads) AS max_reserved_threads, + MIN(min_used_threads) AS min_used_threads, + MAX(max_used_threads) AS max_used_threads, + MAX(total_clr_time) AS total_clr_time, + MAX(plan_generation_num) AS plan_generation_num, + MAX(CAST(delta_worker_time AS double precision) / NULLIF(sample_interval_seconds, 0) / 1000.0) AS worker_time_per_second, + COUNT(DISTINCT query_text_digest) AS distinct_texts, + /* #2235: the fragment count IS the finding — 21 here is why a per-hash ranking missed it. */ + COUNT(DISTINCT query_hash) AS distinct_query_hashes + FROM query_stats + WHERE server_id = $1 + AND collection_time >= $2 + AND collection_time <= $3 + AND ($5::text IS NULL OR database_name = $5) + /* #4394: same first-collection exclusion as TopQueriesSql — see its note. */ + AND sample_interval_seconds IS DISTINCT FROM 0 + /* #2235: proc-hosted rows collapse to one row per (database, host object) — every literal + fragment of one statement lands together. Ad-hoc rows (host_object_name NULL) fall to the + CASE and stay keyed on their OWN query_hash, so they group exactly as the default read does; + without that arm every unrelated ad-hoc statement in a database would pool into one row. */ + GROUP BY database_name, host_object_name, + CASE WHEN host_object_name IS NULL THEN query_hash END + HAVING (SUM(delta_execution_count) > 0 OR SUM(delta_elapsed_time) > 0) + /* #3541 A13: same in-query parallelism floor as TopQueriesSql — see its note. Under the rollup + the group's max_dop is the max across every fragment, so a procedure whose dynamic SQL went + parallel in ANY fragment passes parallel_only, which is the question being asked. */ + AND COALESCE(MAX(max_dop), 0) >= $6 + ORDER BY SUM(delta_worker_time) DESC + LIMIT $4 + 5 + ) + SELECT + r.database_name, + r.query_hash, + r.host_object_name, + r.query_plan_hash, + r.sql_handle, + r.plan_handle, + r.total_executions, + r.total_cpu_us, + r.total_elapsed_us, + r.total_reads, + r.total_writes, + r.total_physical_reads, + r.total_rows, + r.total_spills, + r.min_dop, + r.max_dop, + r.min_worker_time, + r.max_worker_time, + r.min_elapsed_time, + r.max_elapsed_time, + t.query_text, + r.distinct_texts, + r.distinct_query_hashes, + r.last_execution_time, + r.creation_time, + r.min_physical_reads, + r.max_physical_reads, + r.min_rows, + r.max_rows, + r.min_grant_kb, + r.max_grant_kb, + r.min_used_grant_kb, + r.max_used_grant_kb, + r.min_ideal_grant_kb, + r.max_ideal_grant_kb, + r.min_spills, + r.max_spills, + r.min_reserved_threads, + r.max_reserved_threads, + r.min_used_threads, + r.max_used_threads, + r.total_clr_time, + r.plan_generation_num, + r.worker_time_per_second + FROM ranked AS r + LEFT JOIN LATERAL ( + SELECT query_text + FROM v_query_stats + WHERE server_id = $1 + AND database_name = r.database_name + /* Mirrors the grouping: for a rolled-up proc any of its fragments' texts is a valid + representative, but an ad-hoc row must still match its own hash or the text could come from + an unrelated statement. */ + AND host_object_name IS NOT DISTINCT FROM r.host_object_name + AND (r.host_object_name IS NOT NULL OR query_hash = r.query_hash) + AND query_text IS NOT NULL + ORDER BY collection_time DESC + LIMIT 1 + ) AS t ON TRUE + WHERE t.query_text IS NULL OR t.query_text NOT LIKE 'WAITFOR%' + ORDER BY r.total_cpu_us DESC + LIMIT $4 + """; + + /// Old TopProceduresSql at d0a486829, the CPU-ranked text. + private const string OneProceduresSql = """ + SELECT + database_name, + schema_name, + object_name, + object_type, + MAX(sql_handle) AS sql_handle, + MAX(plan_handle) AS plan_handle, + CAST(SUM(delta_execution_count) AS bigint) AS total_executions, + CAST(SUM(delta_worker_time) AS bigint) AS total_cpu_us, + CAST(SUM(delta_elapsed_time) AS bigint) AS total_elapsed_us, + CAST(SUM(delta_logical_reads) AS bigint) AS total_reads, + CAST(SUM(delta_logical_writes) AS bigint) AS total_writes, + CAST(SUM(delta_physical_reads) AS bigint) AS total_physical_reads, + CAST(SUM(delta_spills) AS bigint) AS total_spills, + MIN(min_worker_time) AS min_worker_time, + MAX(max_worker_time) AS max_worker_time, + MIN(min_elapsed_time) AS min_elapsed_time, + MAX(max_elapsed_time) AS max_elapsed_time, + MAX(last_execution_time) AS last_execution_time, + MAX(cached_time) AS cached_time, + MIN(min_logical_reads) AS min_logical_reads, + MAX(max_logical_reads) AS max_logical_reads, + MIN(min_physical_reads) AS min_physical_reads, + MAX(max_physical_reads) AS max_physical_reads, + MIN(min_logical_writes) AS min_logical_writes, + MAX(max_logical_writes) AS max_logical_writes, + MIN(min_spills) AS min_spills, + MAX(max_spills) AS max_spills + FROM procedure_stats + WHERE server_id = $1 + AND collection_time >= $2 + AND collection_time <= $3 + AND ($5::text IS NULL OR database_name = $5) + /* #4394: same first-collection exclusion as TopQueriesSql — see its note. */ + AND sample_interval_seconds IS DISTINCT FROM 0 + GROUP BY database_name, schema_name, object_name, object_type + HAVING SUM(delta_execution_count) > 0 OR SUM(delta_elapsed_time) > 0 + ORDER BY SUM(delta_worker_time) DESC + LIMIT $4 + """; +} diff --git a/Darling/Darling.Tests/TopRankingPagePinTests.cs b/Darling/Darling.Tests/TopRankingPagePinTests.cs new file mode 100644 index 0000000000..6a5ec3c821 --- /dev/null +++ b/Darling/Darling.Tests/TopRankingPagePinTests.cs @@ -0,0 +1,104 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Text.RegularExpressions; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// #5226 source pins for the page half of the ranking choice (no database): the selector on the Top Queries and Top +/// Procedures cards, where its choice lives, what it sends, and that the card draws the reads ranking's retention +/// notice. The server half is pinned by TopRankingTests and the live classes. +/// +[Trait("Stage", "Guard")] +public sealed class TopRankingPagePinTests +{ + private static string Tab() => + ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "pages", "server-tabs.js") + .ReplaceLineEndings("\n"); + + private static string Panels() => + ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "panels.js") + .ReplaceLineEndings("\n"); + + /// The page offers exactly the server's whitelist, in the server's order, and the choice lives at module scope. + [Fact] + public void TheSelectorOffersTheServersWhitelist_AndTheChoiceLivesAtModuleScope() + { + var js = Tab(); + var list = Regex.Match(js, @"const TOP_RANKINGS = \[(?[^\]]*)\];"); + Assert.True(list.Success, "TOP_RANKINGS must declare the choices."); + var values = Regex.Matches(list.Groups["body"].Value, @"value: ""([a-z]+)"", label: ""([A-Za-z]+)"""); + Assert.Equal( + new[] { "cpu=CPU", "duration=Duration", "reads=Reads", "executions=Executions" }, + values.Select(m => m.Groups[1].Value + "=" + m.Groups[2].Value).ToArray()); + + /* Column zero of the file: a module-scope binding, so the 60 s rebuild of a tab (a new card each time) keeps the pick. */ + Assert.Matches(@"(?m)^const topRankingPick = \{ queries: ""cpu"", procedures: ""cpu"" \};", js); + /* A pick stores the choice and draws the card again, so the title follows it. */ + Assert.Contains("topRankingPick[kind] = value;\n draw();", js, StringComparison.Ordinal); + Assert.Contains("\"Top Queries by \" + topRankingLabel(ranking)", js, StringComparison.Ordinal); + Assert.Contains("\"Top Procedures by \" + topRankingLabel(ranking)", js, StringComparison.Ordinal); + } + + /// The default read is unchanged: order_by is sent only for a pick that is not CPU. + [Fact] + public void OrderByIsSentOnlyForANonDefaultPick() + { + var js = Tab(); + Assert.Contains( + "return ranking === \"cpu\" ? params : { ...params, order_by: ranking };", + js, StringComparison.Ordinal); + /* Every read of the two lists goes through it, so no call site can send order_by (or forget to) on its own. */ + Assert.DoesNotContain("order_by:", js.Replace("{ ...params, order_by: ranking }", ""), StringComparison.Ordinal); + } + + /// + /// A reads ranking over a window past what raw keeps carries the raw route's retention notice (retention_notice), and the page + /// draws it ONCE (#5299): readTool puts it in the window note's place (util.js), so each card names truncation_note as its one + /// note and holds no second key or strip for it. What each card draws is run by TopRankingNoticeBehaviourTests. + /// + [Fact] + public void TheRetentionNotice_RidesTheWindowNote_SoNoCardDrawsItTwice() + { + var js = Tab(); + Assert.DoesNotContain("RANKING_NOTE_KEYS", js, StringComparison.Ordinal); + /* The three plain grids (queries on the CPU tab, procedures on both tabs) name the window note and no further note key... */ + Assert.Equal(3, Regex.Matches(js, @"""truncation_note"",\s*null,\s*TOP_(QUERY|PROC)_GROUPS,\s*picker").Count); + /* ...and the Queries tab's composite draws no strip of its own for the retention notice. */ + Assert.DoesNotContain("noticeStrip(res.data.retention_notice)", js, StringComparison.Ordinal); + var util = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "util.js").ReplaceLineEndings("\n"); + Assert.Contains("const note = retentionNoticeOf(res.data) ?? windowNoteText(res.data);", util, StringComparison.Ordinal); + } + + /// The selector sits under the card's title: panels.js draws an optional control before the body. + [Fact] + public void TheCardDrawsItsControlUnderTheTitle_BeforeTheBody() + { + Assert.Matches(@"desc\.control \|\| null,\s*body,", Panels()); + Assert.Contains("control,\n body,", Tab(), StringComparison.Ordinal); + } + + /// #5329: a Reads ranking reads raw OR the io hourly rollup, so the card holds no fixed sentence about the source: the one it + /// had ("ranked from the raw collections because the per-query hourly rollups keep no reads") is false on the io route. The window note + /// (truncation_note, with retention_notice riding in its place) is what tells a reader the answer is partial. + [Fact] + public void AReadsRanking_CarriesNoFixedSentenceAboutItsSource_SoItCannotBeFalseOnTheIoRoute() + { + var js = Tab(); + Assert.DoesNotContain("READS_RAW_SENTENCE", js, StringComparison.Ordinal); + Assert.DoesNotContain("ranked from the raw collections", js, StringComparison.Ordinal); + Assert.DoesNotContain("keep no reads", js, StringComparison.Ordinal); + /* The picker row holds the selector alone, whatever is picked. */ + Assert.DoesNotContain("ranking === \"reads\"", js, StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/TopRankingTests.cs b/Darling/Darling.Tests/TopRankingTests.cs new file mode 100644 index 0000000000..df87ba38cf --- /dev/null +++ b/Darling/Darling.Tests/TopRankingTests.cs @@ -0,0 +1,239 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Service.Mcp; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// #5226 pins that need no database: the order choice is a whitelist (a request value is parsed to an enum, never +/// spliced), each choice swaps the one rank-metric anchor in every one of the five top-N statements, the hourly rollup answers +/// every choice but reads, the web catalog and dispatch carry order_by for the two reads, and the MCP tools +/// keep their parameters (no order_by) and so keep ranking by CPU. +/// +public sealed class TopRankingTests +{ + [Theory] + [InlineData(null, TopRanking.Cpu)] + [InlineData("", TopRanking.Cpu)] + [InlineData(" ", TopRanking.Cpu)] + [InlineData("cpu", TopRanking.Cpu)] + [InlineData("CPU", TopRanking.Cpu)] + [InlineData("duration", TopRanking.Duration)] + [InlineData(" Duration ", TopRanking.Duration)] + [InlineData("reads", TopRanking.Reads)] + [InlineData("executions", TopRanking.Executions)] + public void TryParse_AcceptsTheFourSpellings_AndAbsentMeansCpu(string? value, TopRanking expected) + { + Assert.True(TopRankings.TryParse(value, out var ranking)); + Assert.Equal(expected, ranking); + } + + [Theory] + [InlineData("elapsed")] + [InlineData("cpu; DROP TABLE query_stats")] + [InlineData("delta_worker_time")] + [InlineData("1")] + [InlineData("duration,reads")] + public void TryParse_RefusesAnythingElse_NeverCarriesItAlong(string value) + { + Assert.False(TopRankings.TryParse(value, out var ranking)); + Assert.Equal(TopRanking.Cpu, ranking); + } + + [Fact] + public void WireName_RoundTripsEveryChoice_AndTheChoiceListMatchesTheEnum() + { + foreach (var ranking in Enum.GetValues()) + { + Assert.True(TopRankings.TryParse(TopRankings.WireName(ranking), out var back)); + Assert.Equal(ranking, back); + } + + Assert.Equal( + new[] { "cpu", "duration", "reads", "executions" }, + TopRankings.Choices.Select(c => c.Value).ToArray()); + } + + /// + /// The CPU choice is the default read. Its statement is the const with the anchor replaced by the worker-time sum + /// and nothing else changed (the const is never run unexpanded, so even the default goes through Apply). + /// + [Fact] + public void Apply_Cpu_ExpandsEveryConstToTheWorkerTimeSum_AndNothingElseChanges() + { + foreach (var raw in new[] { DarlingDataReader.TopQueriesSql, DarlingDataReader.TopQueriesByHostObjectSql, DarlingDataReader.TopProceduresSql }) + { + Assert.Equal(raw.Replace("$RANK$", "SUM(delta_worker_time)", StringComparison.Ordinal), TopRankings.Apply(raw, TopRanking.Cpu, hourly: false)); + } + + foreach (var hourly in new[] { DarlingDataReader.TopQueriesHourlySql, DarlingDataReader.TopProceduresHourlySql }) + { + Assert.Equal(hourly.Replace("$RANK$", "SUM(worker_time_sum)", StringComparison.Ordinal), TopRankings.Apply(hourly, TopRanking.Cpu, hourly: true)); + } + } + + /// + /// Each choice swaps ONE thing in each raw statement, the ranking pass's metric. The ORDER BYs, the caps and the + /// filters are the const's own text, so they are the same for every choice. + /// + [Theory] + [InlineData(TopRanking.Duration, "SUM(delta_elapsed_time)")] + [InlineData(TopRanking.Reads, "SUM(delta_logical_reads)")] + [InlineData(TopRanking.Executions, "SUM(delta_execution_count)")] + public void Apply_Raw_SwapsTheRankMetricInEveryRawStatement(TopRanking ranking, string metric) + { + foreach (var cpuSql in new[] { DarlingDataReader.TopQueriesSql, DarlingDataReader.TopQueriesByHostObjectSql, DarlingDataReader.TopProceduresSql }) + { + var sql = TopRankings.Apply(cpuSql, ranking, hourly: false); + Assert.Contains(metric + " AS rank_metric", sql, StringComparison.Ordinal); + Assert.Contains("SUM(delta_worker_time) AS rank_cpu", sql, StringComparison.Ordinal); + Assert.DoesNotContain("$RANK$", sql, StringComparison.Ordinal); + Assert.Equal(cpuSql.Replace("$RANK$", metric, StringComparison.Ordinal), sql); + } + + Assert.Contains("LIMIT $7", TopRankings.Apply(DarlingDataReader.TopQueriesSql, ranking, hourly: false), StringComparison.Ordinal); + Assert.Contains("NOT LIKE 'WAITFOR%'", TopRankings.Apply(DarlingDataReader.TopQueriesByHostObjectSql, ranking, hourly: false), StringComparison.Ordinal); + } + + [Theory] + [InlineData(TopRanking.Duration, "SUM(elapsed_time_sum)")] + [InlineData(TopRanking.Executions, "SUM(execution_count_sum)")] + public void Apply_Hourly_SwapsTheRankMetricInBothRollupStatements(TopRanking ranking, string metric) + { + foreach (var cpuSql in new[] { DarlingDataReader.TopQueriesHourlySql, DarlingDataReader.TopProceduresHourlySql }) + { + var sql = TopRankings.Apply(cpuSql, ranking, hourly: true); + Assert.Contains(metric + " AS rank_metric", sql, StringComparison.Ordinal); + Assert.Contains("SUM(worker_time_sum) AS rank_cpu", sql, StringComparison.Ordinal); + Assert.Contains(DarlingDataReader.TopQueriesHourlyFromPlaceholder, sql, StringComparison.Ordinal); + Assert.DoesNotContain("$RANK$", sql, StringComparison.Ordinal); + } + } + + /// + /// #5329: query_stats_io_hourly carries the reads sums, so on the io shape EVERY ranking has a statement, + /// reads included; the three sums are projected there and as typed NULLs on the stitched shape. + /// + [Fact] + public void OnTheIoShape_ReadsHasAStatement_RankedByTheLogicalReadsSum() + { + foreach (var ranking in Enum.GetValues()) + { + Assert.True(TopRankings.HourlyCarries(ranking, io: true)); + } + + var sql = TopRankings.Apply(DarlingDataReader.TopQueriesHourlySql, TopRanking.Reads, hourly: true, io: true); + Assert.Contains("SUM(logical_reads_sum) AS rank_metric", sql, StringComparison.Ordinal); + Assert.Contains("SUM(worker_time_sum) AS rank_cpu", sql, StringComparison.Ordinal); + Assert.DoesNotContain(TopRankings.RankAnchor, sql, StringComparison.Ordinal); + Assert.Throws(() => TopRankings.Apply(DarlingDataReader.TopQueriesHourlySql, TopRanking.Reads, hourly: true, io: false)); + Assert.Contains("SUM(physical_reads_sum)", TopRankings.HourlyIoSums, StringComparison.Ordinal); + Assert.Contains("SUM(logical_writes_sum)", TopRankings.HourlyIoSums, StringComparison.Ordinal); + Assert.DoesNotContain("SUM(", TopRankings.HourlyNoIoSums, StringComparison.Ordinal); + Assert.Contains(DarlingDataReader.TopQueriesHourlyIoSumsPlaceholder, DarlingDataReader.TopQueriesHourlySql, StringComparison.Ordinal); + } + + /// + /// #5329 route table over the io floor and raw's oldest row, window start 10 days back. io absent and io empty both + /// answer a null floor. Covering io (floor at or before the start) takes EVERY ranking. Otherwise a reads ranking + /// reads raw when raw reaches the start, else whichever reaches further back (a tie goes to raw), and any other + /// ranking keeps the stitched route. + /// + [Theory] + [InlineData(TopRanking.Reads, null, 4, HourlyRoute.Raw)] // io absent or empty, raw does not reach: raw + [InlineData(TopRanking.Reads, null, null, HourlyRoute.Raw)] // nothing anywhere + [InlineData(TopRanking.Reads, null, 12, HourlyRoute.Raw)] // io absent, raw reaches the start + [InlineData(TopRanking.Reads, 20, 4, HourlyRoute.Io)] // io covers (floor 20 days back) + [InlineData(TopRanking.Reads, 10, 4, HourlyRoute.Io)] // io floor exactly at the start covers + [InlineData(TopRanking.Reads, 20, 12, HourlyRoute.Io)] // io covers though raw does too + [InlineData(TopRanking.Reads, 6, 4, HourlyRoute.Io)] // io partial but deeper than raw + [InlineData(TopRanking.Reads, 4, 6, HourlyRoute.Raw)] // io partial, raw is deeper + [InlineData(TopRanking.Reads, 5, 5, HourlyRoute.Raw)] // a tie goes to raw + [InlineData(TopRanking.Reads, 5, 12, HourlyRoute.Raw)] // io partial, raw reaches the start + [InlineData(TopRanking.Reads, 5, null, HourlyRoute.Io)] // io partial, raw empty + [InlineData(TopRanking.Cpu, null, 4, HourlyRoute.Stitched)] + [InlineData(TopRanking.Cpu, 5, 4, HourlyRoute.Stitched)] // io partial: today's route + [InlineData(TopRanking.Cpu, 20, 4, HourlyRoute.Io)] + [InlineData(TopRanking.Duration, 20, 4, HourlyRoute.Io)] + [InlineData(TopRanking.Executions, 20, null, HourlyRoute.Io)] + [InlineData(TopRanking.Duration, 5, 12, HourlyRoute.Stitched)] + [InlineData(TopRanking.Executions, null, null, HourlyRoute.Stitched)] + public void ChooseHourlyRoute_FollowsTheIoFloorAndRawsReach(TopRanking ranking, int? ioDaysBack, int? rawDaysBack, HourlyRoute expected) + { + var now = new DateTime(2026, 6, 1, 0, 0, 0, DateTimeKind.Unspecified); + DateTime? Back(int? days) => days is null ? null : now.AddDays(-days.Value); + Assert.Equal(expected, TopRankings.ChooseHourlyRoute(ranking, now.AddDays(-10), Back(ioDaysBack), Back(rawDaysBack))); + } + + /// + /// The rollups keep worker time, elapsed time and execution counts and no logical reads, so a reads ranking has + /// no hourly statement: the reader routes it to raw, and asking Apply for one is a programming error. + /// + [Fact] + public void Reads_HasNoHourlyStatement_BecauseTheRollupHasNoReadsColumn() + { + Assert.False(TopRankings.HourlyCarries(TopRanking.Reads)); + Assert.True(TopRankings.HourlyCarries(TopRanking.Cpu)); + Assert.True(TopRankings.HourlyCarries(TopRanking.Duration)); + Assert.True(TopRankings.HourlyCarries(TopRanking.Executions)); + Assert.Throws(() => TopRankings.Apply(DarlingDataReader.TopQueriesHourlySql, TopRanking.Reads, hourly: true)); + + var js = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "TimescaleSupport.cs"); + var at = js.IndexOf("[TimescaleSupport.QueryStatsHourlyView] = new[]", StringComparison.Ordinal); + Assert.True(at > 0, "the stitch column list moved — remap this test"); + var list = js[at..js.IndexOf("[TimescaleSupport.QueryStatsDbHourlyView]", at, StringComparison.Ordinal)]; + Assert.DoesNotContain("logical_reads_sum", list, StringComparison.Ordinal); + } + + /// + /// A swap that did not land would silently rank by CPU; Apply refuses a statement whose anchor is absent or doubled + /// (the full per-const check is in ). + /// + [Fact] + public void Apply_ThrowsWhenTheStatementLacksItsAnchor_OrCarriesItTwice() + { + foreach (var ranking in new[] { TopRanking.Cpu, TopRanking.Duration }) + { + Assert.Throws(() => TopRankings.Apply("SELECT 1", ranking, hourly: false)); + Assert.Throws(() => TopRankings.Apply("SELECT $RANK$, $RANK$", ranking, hourly: false)); + } + } + + [Theory] + [InlineData("get_top_queries_by_cpu")] + [InlineData("get_top_procedures_by_cpu")] + public void TheWebCatalog_AdvertisesOrderBy_WithCpuAsTheDefault(string read) + { + var p = Assert.Single(DarlingWebEndpoints.CatalogDescriptors[read].Params, x => x.Name == "order_by"); + Assert.Equal("text", p.Type); + Assert.False(p.Required); + Assert.Equal("cpu", p.Default); + Assert.True(DarlingWebEndpoints.BuildReadDispatch().ContainsKey(read)); + } + + [Fact] + public void TheWebDispatch_PassesOrderBy_ToTheRankedSiblings_AndTheMcpToolsKeepTheirParameters() + { + var web = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Service", "DarlingWebEndpoints.cs"); + Assert.Matches(@"GetTopQueriesRanked\([^\n]*order_by: Str\(c, ""order_by""\)", web); + Assert.Matches(@"GetTopProceduresRanked\([^\n]*order_by: Str\(c, ""order_by""\)", web); + + /* The MCP surface does not change (#5226): neither tool has an order parameter, so both still rank by CPU. */ + foreach (var name in new[] { nameof(DarlingMcpDataTools.GetTopQueriesByCpu), nameof(DarlingMcpDataTools.GetTopProceduresByCpu) }) + { + var method = typeof(DarlingMcpDataTools).GetMethod(name)!; + Assert.DoesNotContain(method.GetParameters(), p => p.Name == "order_by"); + } + } +} diff --git a/Darling/Darling.Tests/TopRankingTwoPassTests.cs b/Darling/Darling.Tests/TopRankingTwoPassTests.cs new file mode 100644 index 0000000000..010a984ebc --- /dev/null +++ b/Darling/Darling.Tests/TopRankingTwoPassTests.cs @@ -0,0 +1,204 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.RegularExpressions; +using PerformanceMonitor.Darling.Service.Mcp; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5226 pins for the raw ranking statements' two-pass shape (check 3 of the review: identical results only under +/// four conditions), the stable-final-key and NULLS LAST rules on every deciding ORDER BY (F2, F8), and the single +/// rank anchor TopRankings.Apply swaps (F11). They read the consts' text only; the row-for-row proof that the +/// two passes answer what the single pass did is the live differential test. The anchor is spelled out here, not +/// read from TopRankings, so a rename of it fails these pins instead of following the code. +/// +public sealed class TopRankingTwoPassTests +{ + private const string Anchor = "$RANK$"; + + private static string Sql(string name) => name switch + { + "TopQueriesSql" => DarlingDataReader.TopQueriesSql, + "TopQueriesByHostObjectSql" => DarlingDataReader.TopQueriesByHostObjectSql, + "TopProceduresSql" => DarlingDataReader.TopProceduresSql, + "TopQueriesHourlySql" => DarlingDataReader.TopQueriesHourlySql, + "TopProceduresHourlySql" => DarlingDataReader.TopProceduresHourlySql, + _ => throw new ArgumentOutOfRangeException(nameof(name)), + }; + + private static bool IsHourly(string name) => name.Contains("Hourly", StringComparison.Ordinal); + + private static IEnumerable Rankings(string name) => + Enum.GetValues().Where(r => !IsHourly(name) || TopRankings.HourlyCarries(r)); + + public static IEnumerable AllFive() => + new[] { "TopQueriesSql", "TopQueriesByHostObjectSql", "TopProceduresSql", "TopQueriesHourlySql", "TopProceduresHourlySql" } + .Select(n => new object[] { n }); + + public static IEnumerable RawThree() => + new[] { "TopQueriesSql", "TopQueriesByHostObjectSql", "TopProceduresSql" }.Select(n => new object[] { n }); + + // ------------------------------------------------------------------ F11: the anchor + + [Theory] + [MemberData(nameof(AllFive))] + public void EveryConst_CarriesTheRankAnchorExactlyOnce_InItsRankingSelectList(string name) + { + var sql = Sql(name); + Assert.Single(Regex.Matches(sql, Regex.Escape(Anchor))); + Assert.Contains(Anchor + " AS rank_metric", sql, StringComparison.Ordinal); + Assert.Single(Regex.Matches(sql, @"SUM\(\w+\) AS rank_cpu")); + } + + [Theory] + [MemberData(nameof(AllFive))] + public void Apply_ThrowsWhenAnyAnchorIsMissing_OrAppearsTwice(string name) + { + var sql = Sql(name); + foreach (var ranking in Rankings(name)) + { + var hourly = IsHourly(name); + var missing = sql.Replace(Anchor, "SUM(x)", StringComparison.Ordinal); + var twice = sql + "\n" + Anchor; + Assert.Throws(() => TopRankings.Apply(missing, ranking, hourly)); + Assert.Throws(() => TopRankings.Apply(twice, ranking, hourly)); + Assert.DoesNotContain(Anchor, TopRankings.Apply(sql, ranking, hourly), StringComparison.Ordinal); + } + } + + // ------------------------------------------------------------------ F2 + F8: NULLS LAST and a final key + + private static readonly Dictionary KeyTails = new() + { + ["TopQueriesSql"] = new[] { "win_database_name, win_query_hash, win_host_object_name", "r.database_name, r.query_hash, r.host_object_name" }, + ["TopQueriesByHostObjectSql"] = new[] { "win_database_name, win_host_object_name, win_group_hash", "r.database_name, r.host_object_name, r.query_hash" }, + ["TopProceduresSql"] = new[] { "win_database_name, win_schema_name, win_object_name, win_object_type", "database_name, schema_name, object_name, object_type" }, + ["TopQueriesHourlySql"] = new[] { "database_name, query_hash", "r.database_name, r.query_hash" }, + ["TopProceduresHourlySql"] = new[] { "database_name, schema_name, object_name", "r.database_name, r.schema_name, r.object_name" }, + }; + + /// Every ORDER BY that decides which groups survive or in what order (the latest-text lookup's is not one). + private static List RankingOrderBys(string sql) => + Regex.Matches(sql, @"(?m)^\s*ORDER BY ([^\r\n]+)").Select(m => m.Groups[1].Value.Trim()) + .Where(o => !o.StartsWith("collection_time", StringComparison.Ordinal) && !o.StartsWith("q.", StringComparison.Ordinal) + && !o.StartsWith("p.page_ord", StringComparison.Ordinal)).ToList(); /* #5313: the count-row join re-sorts the page by its position */ + + [Theory] + [MemberData(nameof(AllFive))] + public void EveryDecidingOrderBy_SortsDescSumsNullsLast_AndEndsInTheUniqueGroupKey(string name) + { + foreach (var ranking in Rankings(name)) + { + var orders = RankingOrderBys(TopRankings.Apply(Sql(name), ranking, IsHourly(name))); + Assert.Equal(2, orders.Count); + for (var i = 0; i < orders.Count; i++) + { + var o = orders[i]; + Assert.Matches(@"^(MAX\()?([rw]\.)?rank_metric(\))? DESC NULLS LAST, (MAX\()?([rw]\.)?rank_cpu(\))? DESC NULLS LAST, ", o); + Assert.EndsWith(", " + KeyTails[name][i], o, StringComparison.Ordinal); + foreach (var item in o.Split(", ")) + { + Assert.True(!item.EndsWith(" DESC", StringComparison.Ordinal), $"{name}: '{item}' sorts DESC without NULLS LAST"); + } + } + } + } + + // ------------------------------------------------------------------ check 3: the four conditions of the two passes + + [Theory] + [MemberData(nameof(RawThree))] + public void Raw_Pass1RanksNarrow_AndRepeatsTheHavingTheFloorAndTheKey(string name) + { + var sql = Sql(name); + var p1 = sql[sql.IndexOf("winners AS MATERIALIZED (", StringComparison.Ordinal)..]; + var limitAt = p1.IndexOf(name == "TopProceduresSql" ? "LIMIT $4" : "LIMIT $7", StringComparison.Ordinal); + p1 = p1[..p1.IndexOfAny(new[] { '\r', '\n' }, limitAt)]; /* through the end of pass 1's LIMIT line */ + Assert.DoesNotContain("COUNT(DISTINCT", p1, StringComparison.Ordinal); + Assert.DoesNotContain("MAX(plan_handle)", p1, StringComparison.Ordinal); + Assert.DoesNotContain("MIN(min_", p1, StringComparison.Ordinal); + if (name == "TopProceduresSql") + { + Assert.Contains("GROUP BY database_name, schema_name, object_name, object_type", p1, StringComparison.Ordinal); + Assert.Contains("HAVING SUM(delta_execution_count) > 0 OR SUM(delta_elapsed_time) > 0", p1, StringComparison.Ordinal); + Assert.EndsWith("LIMIT $4", p1, StringComparison.Ordinal); + Assert.DoesNotContain("+ 5", sql, StringComparison.Ordinal); /* procedures has no over-fetch */ + return; + } + + Assert.Contains("HAVING (SUM(delta_execution_count) > 0 OR SUM(delta_elapsed_time) > 0)", p1, StringComparison.Ordinal); + Assert.Contains("AND COALESCE(MAX(max_dop), 0) >= $6", p1, StringComparison.Ordinal); + Assert.Contains(name == "TopQueriesSql" + ? "GROUP BY database_name, query_hash, host_object_name" + : "CASE WHEN host_object_name IS NULL THEN query_hash END", p1, StringComparison.Ordinal); + Assert.EndsWith("LIMIT $7", p1.TrimEnd(), StringComparison.Ordinal); /* the candidate limit (#5313) stays in pass 1, and only there */ + Assert.Single(Regex.Matches(sql, Regex.Escape("LIMIT $7"))); + Assert.DoesNotContain("+ 5", sql, StringComparison.Ordinal); /* no fixed over-fetch: TopFill picks the candidate limit */ + } + + [Theory] + [MemberData(nameof(RawThree))] + public void Raw_Pass2RescansTheSameWindow_AndMatchesTheKeyNullSafely(string name) + { + var sql = Sql(name); + foreach (var filter in new[] + { + "server_id = $1", "collection_time >= $2", "collection_time <= $3", + "$5::text[] IS NULL OR database_name = ANY($5)", "{TimescaleSupport.IntervalHonestSourceFilter}", + }) + { + var text = filter.Replace("{TimescaleSupport.IntervalHonestSourceFilter}", "sample_interval_seconds IS DISTINCT FROM 0", StringComparison.Ordinal); + /* once per pass; the queries' one latest-text lookup (#5309) scopes by server and reads the window too */ + var lookup = name != "TopProceduresSql" && (filter == "server_id = $1" || filter.StartsWith("collection_time", StringComparison.Ordinal)); + Assert.Equal(lookup ? 3 : 2, Regex.Matches(sql, Regex.Escape(text)).Count); + } + + var p2 = sql[sql.IndexOf("JOIN winners AS w", StringComparison.Ordinal)..]; + var keys = name switch + { + "TopQueriesSql" => new[] { "database_name", "query_hash", "host_object_name" }, + "TopQueriesByHostObjectSql" => new[] { "database_name", "host_object_name", "group_hash" }, + _ => new[] { "database_name", "schema_name", "object_name", "object_type" }, + }; + foreach (var key in keys) + { + Assert.Contains("IS NOT DISTINCT FROM w.win_" + key, p2, StringComparison.Ordinal); + } + + Assert.DoesNotMatch(@"=\s*w\.win_", p2); /* a plain equality drops the NULL-keyed groups */ + Assert.DoesNotMatch(@"\bIN \(SELECT", sql); + } + + [Theory] + [MemberData(nameof(RawThree))] + public void Raw_TextLookupWaitforTrimAndFinalLimit_RunAfterPass2(string name) + { + var sql = Sql(name); + var pass2 = sql.IndexOf("JOIN winners AS w", StringComparison.Ordinal); + Assert.True(sql.IndexOf("winners AS MATERIALIZED", StringComparison.Ordinal) < pass2); + if (name == "TopProceduresSql") + { + Assert.DoesNotContain("LATERAL", sql, StringComparison.Ordinal); + return; + } + + /* #5309: the text is ONE lookup over the winners (the latest_text CTE), joined to the ranked rows once. */ + Assert.DoesNotContain("LATERAL", sql, StringComparison.Ordinal); + var lookup = sql.IndexOf("latest_text AS MATERIALIZED (", StringComparison.Ordinal); + var lookupJoin = sql.IndexOf("LEFT JOIN latest_text AS t", StringComparison.Ordinal); + var waitfor = sql.IndexOf("NOT LIKE 'WAITFOR%'", StringComparison.Ordinal); + var finalLimit = sql.LastIndexOf("LIMIT $4", StringComparison.Ordinal); + Assert.True(pass2 < lookup && lookup < lookupJoin && lookupJoin < waitfor && waitfor < finalLimit, "text lookup, WAITFOR trim and LIMIT $4 follow pass 2"); + Assert.Single(Regex.Matches(sql, @"LIMIT \$4\s*$", RegexOptions.Multiline)); + } +} diff --git a/Darling/Darling.Tests/TopTextLookupFillTests.cs b/Darling/Darling.Tests/TopTextLookupFillTests.cs new file mode 100644 index 0000000000..277163cea1 --- /dev/null +++ b/Darling/Darling.Tests/TopTextLookupFillTests.cs @@ -0,0 +1,402 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.RegularExpressions; +using System.Threading.Tasks; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5309 and #5313 (inside #5299): every Top Queries statement resolves the latest text of all its ranked groups with +/// ONE lookup (no per-row ORDER BY collection_time DESC LIMIT 1, which scanned the raw table once per row on a +/// store without TimescaleDB), and fills the page past the WAITFOR trim through 's bounded +/// rounds instead of a fixed over-fetch of five. These are SQL-shape pins plus a unit test of the fill loop; the +/// plans and timings are measured on a live store, not here. +/// +public sealed class TopTextLookupFillTests +{ + private static string Sql(string name) => name switch + { + "TopQueriesSql" => DarlingDataReader.TopQueriesSql, + "TopQueriesByHostObjectSql" => DarlingDataReader.TopQueriesByHostObjectSql, + "TopQueriesHourlySql" => DarlingDataReader.TopQueriesHourlySql, + "ViewerTopQueriesSql" => ViewerDataService.TopQueriesSql, + _ => throw new ArgumentOutOfRangeException(nameof(name)), + }; + + public static IEnumerable Statements() => + new[] { "TopQueriesSql", "TopQueriesByHostObjectSql", "TopQueriesHourlySql", "ViewerTopQueriesSql" } + .Select(n => new object[] { n }); + + // ------------------------------------------------------------------ #5309: one lookup + + /// The raw statements (and the Viewer's) read the text in ONE pass over the winners. The hourly statement is + /// different on purpose and has its own test below. + public static IEnumerable RawStatements() => + new[] { "TopQueriesSql", "TopQueriesByHostObjectSql", "ViewerTopQueriesSql" }.Select(n => new object[] { n }); + + [Theory] + [MemberData(nameof(RawStatements))] + public void TheLatestTextIsOneLookupForAllRankedGroups_NotOnePerRow(string name) + { + var sql = Sql(name); + /* Materialized: the join to the lookup is NULL-safe, which PostgreSQL cannot hash, so a plain CTE is inlined into a + nested loop and re-run per ranked row (#5309's per-row re-run in a new shape; TopTextLookupPlanLiveTests reads it + from the plan). */ + Assert.Contains("latest_text AS MATERIALIZED (", sql, StringComparison.Ordinal); + Assert.DoesNotContain("LATERAL", sql, StringComparison.Ordinal); + Assert.DoesNotContain("LIMIT 1", sql, StringComparison.Ordinal); /* the per-row text lookup's shape */ + Assert.DoesNotMatch(@"(FROM|JOIN) v_query_stats", sql); + /* Each text lookup picks the newest row per key in one pass: DISTINCT ON over (key, collection_time DESC). */ + Assert.Matches(@"SELECT DISTINCT ON \(q\.database_name, q\.(query_hash|host_object_name)", sql); + Assert.Matches(@"ORDER BY q\.database_name, q\.(query_hash|host_object_name)[^\r\n]*q\.collection_time DESC", sql); + /* The newest row's text resolves through the dimension for that one row, as v_query_stats' COALESCE does. */ + Assert.Contains("COALESCE(l.query_text, d.query_text) AS query_text", sql, StringComparison.Ordinal); + Assert.Contains("LEFT JOIN query_text_dim AS d ON d.digest = l.query_text_digest", sql, StringComparison.Ordinal); + /* The lookup joins the winners (or ranked) CTE, which is read, not repeated. */ + Assert.Matches(@"JOIN (winners|ranked) AS (w|rk)\b[\s\S]*COALESCE\(q\.", sql); + } + + [Theory] + [MemberData(nameof(Statements))] + public void TheLookupMatchesTheKeyNullSafely_AndOnlyTheOuterJoinReadsTheResult(string name) + { + var sql = Sql(name); + Assert.Contains("q.database_name IS NOT DISTINCT FROM", sql, StringComparison.Ordinal); + /* The one join to the ranked rows is NULL-safe on every key column; no plain "t.x = r.x" equality that would + drop a NULL-keyed group's text. */ + Assert.DoesNotMatch(@"\bt\.\w+\s*=\s*r\.", sql); + Assert.Single(Regex.Matches(sql, @"LEFT JOIN latest_text AS t|LEFT JOIN latest_in_window AS w")); + } + + [Theory] + [InlineData("TopQueriesSql")] + [InlineData("TopQueriesByHostObjectSql")] + [InlineData("ViewerTopQueriesSql")] + public void TheRawLookupIsBoundedToTheWindow(string name) + { + var sql = Sql(name); + var lookup = sql[sql.IndexOf("latest_text AS MATERIALIZED (", StringComparison.Ordinal)..]; + var end = lookup.IndexOf("LEFT JOIN query_text_dim AS d ON d.digest = l.query_text_digest", StringComparison.Ordinal); + lookup = lookup[..end]; + Assert.Contains("q.collection_time >= $2", lookup, StringComparison.Ordinal); + Assert.Contains("q.collection_time <= $3", lookup, StringComparison.Ordinal); + Assert.Contains("q.server_id = $1", lookup, StringComparison.Ordinal); + } + + [Fact] + public void TheHourlyWindowLookup_ProbesTheIndexOncePerRankedKey_NotTheWholeWindow() + { + var sql = DarlingDataReader.TopQueriesHourlySql; + var window = sql.IndexOf("latest_in_window AS (", StringComparison.Ordinal); + var lookup = sql[window..sql.IndexOf("missing AS MATERIALIZED (", StringComparison.Ordinal)]; + /* One probe per ranked key: a lateral over ranked, each arm an ordered LIMIT 1 on the indexed hash equality + (idx_query_stats_server_hash_time: server_id, query_hash, collection_time DESC). The old form read every raw row + of the window and hash joined it to the ranked keys (1,439,400 rows at 168 hours to find 25 texts). */ + Assert.Contains("FROM ranked AS rk", lookup, StringComparison.Ordinal); + Assert.Contains("CROSS JOIN LATERAL", lookup, StringComparison.Ordinal); + Assert.DoesNotContain("DISTINCT ON", lookup, StringComparison.Ordinal); + Assert.DoesNotContain("JOIN ranked", lookup, StringComparison.Ordinal); + Assert.Contains("q.query_hash = rk.query_hash", lookup, StringComparison.Ordinal); + Assert.Equal(2, Regex.Matches(lookup, @"ORDER BY q\.collection_time DESC, q\.collection_id DESC\s+LIMIT 1").Count); /* N3: a tie on the time breaks on the row collected last */ + /* NULL-safe on every key column: the database by IS NOT DISTINCT FROM, the hash by its own NULL arm. */ + Assert.Contains("q.database_name IS NOT DISTINCT FROM rk.database_name", lookup, StringComparison.Ordinal); + Assert.Contains("q.query_hash IS NULL", lookup, StringComparison.Ordinal); + Assert.Contains("rk.query_hash IS NULL", lookup, StringComparison.Ordinal); + /* One text lookup per statement still: the window pass, then the fallback for the keys it found nothing for. */ + Assert.Single(Regex.Matches(sql, @"LEFT JOIN latest_in_window AS w")); + Assert.Single(Regex.Matches(sql, @"FROM latest_in_window")); + } + + [Fact] + public void TheHourlyLookup_ReadsTheWindowFirst_ThenOnlyTheKeysTheWindowHadNoTextFor() + { + var sql = DarlingDataReader.TopQueriesHourlySql; + var window = sql.IndexOf("latest_in_window AS (", StringComparison.Ordinal); + var missing = sql.IndexOf("missing AS MATERIALIZED (", StringComparison.Ordinal); + var any = sql.IndexOf("latest_any AS MATERIALIZED (", StringComparison.Ordinal); + var final = sql.IndexOf("COALESCE(w.query_text, a.query_text) AS query_text", StringComparison.Ordinal); + Assert.True(window > 0 && window < missing && missing < any && any < final); + var windowPass = sql[window..missing]; + Assert.Contains("q.collection_time >= $2", windowPass, StringComparison.Ordinal); + Assert.Contains("q.collection_time < $3", windowPass, StringComparison.Ordinal); + /* The fallback has no window bound (the rollup outlives raw), but it joins only the missing keys. */ + var anyPass = sql[any..final]; + Assert.DoesNotContain("collection_time >=", anyPass, StringComparison.Ordinal); + /* #5299 round 2 (N1): a per-key lateral probe of the hash index (two arms, so a NULL-hash key still finds its text), + not one DISTINCT ON over every raw row of the server joined to the missing keys. */ + Assert.Contains("FROM missing AS m", anyPass, StringComparison.Ordinal); + Assert.Contains("CROSS JOIN LATERAL", anyPass, StringComparison.Ordinal); + Assert.DoesNotContain("DISTINCT ON", anyPass, StringComparison.Ordinal); + Assert.Equal(3, Regex.Matches(anyPass, @"LIMIT 1\s*$", RegexOptions.Multiline).Count); + Assert.Contains("NOT EXISTS", sql[missing..any], StringComparison.Ordinal); + } + + // ------------------------------------------------------------------ #5299 round 2: N3 and N6, a total order + + [Fact] + public void EveryNewestTextPick_BreaksATieOnTheTimeOnTheRowCollectedLast() + { + /* Two rows of one key at one collection_time (one shape hash, two plans, one collection) must give the same text every + run, because the text also decides the WAITFOR trim. collection_id is unique per row, so it is a total tie-break. */ + foreach (var (name, sql) in new[] + { + ("TopQueriesSql", DarlingDataReader.TopQueriesSql), + ("TopQueriesByHostObjectSql", DarlingDataReader.TopQueriesByHostObjectSql), + ("TopQueriesHourlySql", DarlingDataReader.TopQueriesHourlySql), + ("QueryStoreTopSql", DarlingDataReader.QueryStoreTopSql), + ("QueryStoreTopTableSql", DarlingDataReader.QueryStoreTopTableSql), + ("QueryStoreTopDailyTableSql", DarlingDataReader.QueryStoreTopDailyTableSql), + ("ViewerTopQueriesSql", ViewerDataService.TopQueriesSql), + ("ViewerQueryStoreTopSql", ViewerDataService.QueryStoreTopSql), + }) + { + var picks = Regex.Matches(sql, @"ORDER BY[^\n]*\b[qs]\.collection_time DESC[^\n]*").Select(m => m.Value).ToList(); + Assert.NotEmpty(picks); + Assert.All(picks, pick => Assert.True(pick.Contains("collection_id DESC", StringComparison.Ordinal), + $"{name}: the newest-text pick '{pick.Trim()}' has no tie-break after collection_time")); + } + } + + [Fact] + public void TheQueryStoreAndViewerTopStatements_RankOnATotalOrder() + { + /* Ranking value, then the whole group key: an exact tie at the candidate boundary must not pick different members in a + refill round. The page order uses the same columns the ranking did. */ + const string queryStoreKey = "database_name, query_id, plan_id, query_hash, execution_type_desc, replica_role"; + foreach (var sql in new[] { DarlingDataReader.QueryStoreTopSql, DarlingDataReader.QueryStoreTopTableSql, DarlingDataReader.QueryStoreTopDailyTableSql, ViewerDataService.QueryStoreTopSql }) + { + Assert.Contains("DESC, " + queryStoreKey + "\n", sql.ReplaceLineEndings("\n"), StringComparison.Ordinal); + Assert.Contains("DESC, r.database_name, r.query_id, r.plan_id, r.query_hash, r.execution_type_desc, r.replica_role) AS page_ord", sql, StringComparison.Ordinal); + } + + var viewer = ViewerDataService.TopQueriesSql.ReplaceLineEndings("\n"); + Assert.Contains("ORDER BY SUM(delta_elapsed_time) DESC, database_name, query_hash, host_object_name\n", viewer, StringComparison.Ordinal); + Assert.Contains("ROW_NUMBER() OVER (ORDER BY r.total_elapsed_us DESC, r.database_name, r.query_hash, r.host_object_name) AS page_ord", viewer, StringComparison.Ordinal); + Assert.Contains("ORDER BY r.total_elapsed_us DESC, r.database_name, r.query_hash, r.host_object_name\n", viewer, StringComparison.Ordinal); + } + + [Fact] + public void TheHostObjectLookup_KeysOnTheGroupingKey_SoAProcGroupTakesAnyFragmentsText() + { + var sql = DarlingDataReader.TopQueriesByHostObjectSql; + Assert.Contains("SELECT DISTINCT ON (q.database_name, q.host_object_name, CASE WHEN q.host_object_name IS NULL THEN q.query_hash END)", sql, StringComparison.Ordinal); + Assert.Contains("AND t.group_hash IS NOT DISTINCT FROM CASE WHEN r.host_object_name IS NULL THEN r.query_hash END", sql, StringComparison.Ordinal); + } + + // ------------------------------------------------------------------ #5313: the bounded fill + + [Theory] + [MemberData(nameof(Statements))] + public void TheCandidateLimitIsAParameter_NotAFixedFive_AndTheCountRidesLast(string name) + { + var sql = Sql(name); + Assert.DoesNotContain("+ 5", sql, StringComparison.Ordinal); + var limit = name is "TopQueriesHourlySql" or "ViewerTopQueriesSql" ? "LIMIT $6" : "LIMIT $7"; + Assert.Single(Regex.Matches(sql, Regex.Escape(limit) + @"\s*$", RegexOptions.Multiline)); + Assert.Single(Regex.Matches(sql, @"LIMIT \$4\s*$", RegexOptions.Multiline)); /* the final cap stays top */ + /* The count rides on a row of its own, joined to the page, so a round whose candidates were ALL trimmed still + reports it (an empty page used to read as exhausted). page_ord is the page's last column and is NULL on that row. */ + Assert.Matches(@"FROM \(SELECT COUNT\(\*\) AS candidate_count FROM (winners|ranked)\) AS c\s+LEFT JOIN page AS p ON TRUE", sql); + Assert.Single(Regex.Matches(sql, @"SELECT p\.\*, c\.candidate_count")); + Assert.Single(Regex.Matches(sql, @"AS page_ord")); + Assert.Matches(@"ORDER BY p\.page_ord\s*$", sql); + } + + [Fact] + public void TheCallersRunTheStatementsThroughTopFill() + { + var reader = System.IO.File.ReadAllText(FindSource("PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs")); + Assert.Equal(4, Regex.Matches(reader, @"TopFill\.RunAsync\(top,").Count); /* raw and hourly Top Queries, then the Query Store raw read and its table read (daily shares the table call site) */ + var viewer = System.IO.File.ReadAllText(FindSource("PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStats.cs")); + Assert.Single(Regex.Matches(viewer, @"TopFill\.RunAsync\(top,")); + var viewerQs = System.IO.File.ReadAllText(FindSource("PerformanceMonitor.Darling.Viewer/ViewerDataService.QueryStore.cs")); + Assert.Equal(2, Regex.Matches(viewerQs, @"TopFill\.RunAsync\(top,").Count); /* raw and interval table */ + } + + private static string FindSource(string relative) + { + var dir = new System.IO.DirectoryInfo(AppContext.BaseDirectory); + while (dir is not null) + { + var path = System.IO.Path.Combine(dir.FullName, "Darling", relative); + if (System.IO.File.Exists(path)) + { + return path; + } + + path = System.IO.Path.Combine(dir.FullName, relative); + if (System.IO.File.Exists(path)) + { + return path; + } + + dir = dir.Parent; + } + + throw new System.IO.FileNotFoundException(relative); + } + + [Fact] + public void FirstRound_IsTopPlusFive_TheOldOverFetch() + { + Assert.Equal(30, TopFill.FirstCandidates(25)); + Assert.Equal(int.MaxValue, TopFill.FirstCandidates(int.MaxValue)); + } + + [Theory] + [InlineData(25, 1, 30, 25, 30)] /* a full page stops */ + [InlineData(25, 1, 30, 20, 12)] /* short, but the candidates ran out (12 < 30): nothing more to fetch */ + [InlineData(25, 1, 30, 0, 0)] /* the round found no candidates at all: exhausted */ + [InlineData(25, 3, 225, 10, 225)] /* the round bound */ + public void NoRefill_WhenFull_Exhausted_OrAtTheBound(int top, int round, int candidates, int returned, int candidateCount) + { + Assert.Null(TopFill.NextCandidates(top, round, candidates, returned, candidateCount)); + } + + [Fact] + public void Refill_GrowsFourfold_CappedAtTopPlus200() + { + Assert.Equal(120, TopFill.NextCandidates(25, 1, 30, 20, 30)); + Assert.Equal(225, TopFill.NextCandidates(25, 2, 120, 20, 120)); /* 480 would pass top + 200 */ + Assert.Null(TopFill.NextCandidates(25, 2, 225, 20, 225)); /* already at the cap */ + } + + [Fact] + public async Task Run_AnEmptyRoundThatSawAFullCandidateLimit_AsksAgain_UpToTheBound() + { + /* #5313: every candidate is a shell, so each round trims to nothing and returns no page row, but it still reports + that it considered its whole candidate limit. The fill must go on: 30, 120, 225 candidates and then stop. */ + var asked = new List(); + var rows = await TopFill.RunAsync(25, c => + { + asked.Add(c); + return Task.FromResult((new List(), c)); + }); + Assert.Equal(new[] { 30, 120, 225 }, asked); /* at most three rounds, never more than top + 200 candidates */ + Assert.Empty(rows); + } + + [Fact] + public async Task Run_AnIdleWindow_CostsOneRound() + { + /* No candidates at all: the count row says 0, which is under the first limit, so there is nothing to refill. */ + var asked = new List(); + var rows = await TopFill.RunAsync(25, c => + { + asked.Add(c); + return Task.FromResult((new List(), 0)); + }); + Assert.Equal(new[] { 30 }, asked); + Assert.Empty(rows); + } + + [Fact] + public async Task Run_AnEmptyFirstRound_ThenRealRows_FillsThePage() + { + var asked = new List(); + var rows = await TopFill.RunAsync(3, c => + { + asked.Add(c); + return Task.FromResult(c < 30 ? (new List(), c) : (new List { 1, 2, 3 }, c)); + }); + Assert.Equal(new[] { 8, 32 }, asked); + Assert.Equal(3, rows.Count); + } + + // ------------------------------------------------------------------ the Query Store twin (#5313) + + public static IEnumerable QueryStoreStatements() => + new[] { "McpRaw", "McpTable", "McpDaily", "ViewerRaw", "ViewerTable" }.Select(n => new object[] { n }); + + private static (string Sql, string Limit) QueryStoreSql(string name) => name switch + { + "McpRaw" => (DarlingDataReader.QueryStoreTopSql, "LIMIT $8"), + "McpTable" => (DarlingDataReader.QueryStoreTopTableSql, "LIMIT $8"), + "McpDaily" => (DarlingDataReader.QueryStoreTopDailyTableSql, "LIMIT $10"), + "ViewerRaw" => (ViewerDataService.QueryStoreTopSql, "LIMIT $6"), + "ViewerTable" => (ViewerDataService.QueryStoreTopTableSql, "LIMIT $6"), + _ => throw new ArgumentOutOfRangeException(nameof(name)), + }; + + [Theory] + [MemberData(nameof(QueryStoreStatements))] + public void TheQueryStoreReads_TakeTheCandidateLimitAsAParameter_AndReportTheCountOnTheirOwnRow(string name) + { + var (sql, limit) = QueryStoreSql(name); + Assert.DoesNotContain("+ 5", sql, StringComparison.Ordinal); + Assert.Single(Regex.Matches(sql, Regex.Escape(limit) + @"\s*$", RegexOptions.Multiline)); + Assert.Single(Regex.Matches(sql, @"LIMIT \$4\s*$", RegexOptions.Multiline)); /* the final cap stays top */ + Assert.Matches(@"FROM \(SELECT COUNT\(\*\) AS candidate_count FROM ranked\) AS c\s+LEFT JOIN page AS p ON TRUE", sql); + Assert.Single(Regex.Matches(sql, @"SELECT p\.\*, c\.candidate_count")); + Assert.Single(Regex.Matches(sql, @"AS page_ord")); + Assert.Matches(@"ORDER BY p\.page_ord\s*$", sql); + } + + [Fact] + public async Task Run_StopsAtTheFirstFullPage_AfterOneRound() + { + var asked = new List(); + var rows = await TopFill.RunAsync(25, c => + { + asked.Add(c); + return Task.FromResult((Enumerable.Range(0, 25).ToList(), c)); + }); + Assert.Equal(new[] { 30 }, asked); + Assert.Equal(25, rows.Count); + } + + [Fact] + public async Task Run_FillsThePageWhenMoreThanFiveShellsAreTrimmed() + { + /* 40 WAITFOR shells lead the ranking: the page can only fill once the candidate limit passes 40 + 25. */ + var asked = new List(); + var rows = await TopFill.RunAsync(25, c => + { + asked.Add(c); + var survivors = Math.Max(0, Math.Min(c, 1000) - 40); + return Task.FromResult((Enumerable.Range(0, Math.Min(25, survivors)).ToList(), Math.Min(c, 1000))); + }); + Assert.Equal(new[] { 30, 120 }, asked); + Assert.Equal(25, rows.Count); + } + + [Fact] + public async Task Run_NeverRunsMoreThanThreeRounds_AndNeverPassesTheCandidateCap() + { + var asked = new List(); + var rows = await TopFill.RunAsync(25, c => + { + asked.Add(c); + return Task.FromResult((new List(), c)); /* every candidate trimmed, always more behind them */ + }); + Assert.Equal(new[] { 30, 120, 225 }, asked); + Assert.Empty(rows); + Assert.All(asked, c => Assert.True(c <= 25 + TopFill.MaxExtraCandidates)); + } + + [Fact] + public async Task Run_StopsWhenTheCandidatesRunOut_WithAShortPage() + { + var asked = new List(); + var rows = await TopFill.RunAsync(25, c => + { + asked.Add(c); + return Task.FromResult((Enumerable.Range(0, 3).ToList(), 12)); /* only 12 candidates exist, 9 of them shells */ + }); + Assert.Equal(new[] { 30 }, asked); + Assert.Equal(3, rows.Count); + } +} diff --git a/Darling/Darling.Tests/TopTextLookupPlanLiveTests.cs b/Darling/Darling.Tests/TopTextLookupPlanLiveTests.cs new file mode 100644 index 0000000000..6c716c15fd --- /dev/null +++ b/Darling/Darling.Tests/TopTextLookupPlanLiveTests.cs @@ -0,0 +1,358 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Darling.Tests; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace PerformanceMonitor.Darling.Tests; + +/* #1776 own-store: every fact mints its own scratch database through ScratchPostgres (#4650: EXPLAIN reads the chunk list, + and a shared store carries other classes' leftovers) and touches nothing on the shared one, so it is not serialized + against the live-postgres collection. */ + +/// +/// #5309 and #5313 (inside #5299): the PLAN of the text lookup, read from EXPLAIN (ANALYZE, FORMAT JSON) on a small seed. +/// Two defects the large-store timing found, each pinned here by what the plan does, not by what the text says. +/// +/// The raw lookup runs once. latest_text is joined to the ranked rows NULL-safe +/// (IS NOT DISTINCT FROM), which PostgreSQL cannot hash, so a plain CTE is inlined into a nested loop and re-run for +/// every ranked row (a 120-candidate round scanned the 20,000-row text dimension 10,000 times, 9,975 ms against 805 ms +/// materialized). With WAITFOR shells planted above the real groups the round has to read past them, which is the case +/// that made it expensive. The seed here is 125 candidates over unanalyzed tables, so the planner estimates one row and +/// takes the nested loop on the plain CTE; the test asserts the lookup is its own CTE node that ran one time. +/// +/// The hourly lookup reads the ranked keys, not the window. latest_in_window read every raw row of the +/// window and joined it to the ranked keys (1,439,400 rows at 168 hours to find 25 texts). It now probes +/// idx_query_stats_server_hash_time once per ranked key. The test asserts the rows the lookup reads from raw are +/// bounded by the ranked keys, not by the window's row count. +/// +public sealed class TopTextLookupPlanLiveTests +{ + private const string ServerName = "a5299-text-lookup-plan"; + private const int Shells = 120; + private const int Real = 30; + private const int RowsPerGroup = 12; + private const int Candidates = 125; + private const int Top = 5; + + [Fact] + public Task TheRawLookup_RunsOnce_NotOncePerRankedRow() => + WithSeedAsync(async (connection, serverId, now, ct) => + { + var start = now.AddHours(-24); + var end = now.AddMinutes(5); + + foreach (var name in new[] { "TopQueriesSql", "TopQueriesByHostObjectSql" }) + { + var sql = TopRankings.Apply(name == "TopQueriesSql" ? DarlingDataReader.TopQueriesSql : DarlingDataReader.TopQueriesByHostObjectSql, + TopRanking.Cpu, hourly: false); + var plan = await ExplainAsync(connection, sql, ct, serverId, start, end, + P(NpgsqlDbType.Integer, Top), P(NpgsqlDbType.Array | NpgsqlDbType.Text, null), P(NpgsqlDbType.Integer, 0), P(NpgsqlDbType.Integer, Candidates)); + AssertLookupRanOnce(name, plan); + } + + var viewerPlan = await ExplainAsync(connection, ViewerDataService.TopQueriesSql, ct, serverId, start, end, + P(NpgsqlDbType.Integer, Top), P(NpgsqlDbType.Array | NpgsqlDbType.Text, null), P(NpgsqlDbType.Integer, Candidates)); + AssertLookupRanOnce("ViewerTopQueriesSql", viewerPlan); + }); + + [Fact] + public Task TheHourlyLookup_ReadsTheRankedKeys_NotTheWholeWindow() => + WithSeedAsync(async (connection, serverId, now, ct) => + { + /* The index the lookup probes is created by the service at start (PgTableTuning.ApplyAsync), not by a migration, so a + test store may lack it: make sure it is there, and give the planner statistics, as a store that has run a while has. + (Without the statistics the unanalyzed seed makes the planner walk the time index and filter on the hash.) */ + const string indexName = "idx_query_stats_server_hash_time"; + await DarlingMcpTestData.ExecAsync(connection, ct, + "CREATE INDEX IF NOT EXISTS " + indexName + " ON collect.query_stats (server_id, query_hash, collection_time DESC)"); + /* The database is this test's own scratch store, so the index goes with it: no teardown to run. */ + await DarlingMcpTestData.ExecAsync(connection, ct, "ANALYZE collect.query_stats"); + await AssertHourlyLookupAsync(connection, serverId, now, ct); + }); + + private static async Task AssertHourlyLookupAsync(NpgsqlConnection connection, int serverId, DateTime now, CancellationToken ct) + { + /* The rollup is stood in for by a plain relation over the raw rows, so the test reads the TEXT lookup's plan + and not the router's: ranked reads it, the lookup reads query_stats. */ + const string standIn = "(SELECT server_id, collection_time AS bucket, database_name, query_hash, sql_handle, " + + "delta_execution_count AS execution_count_sum, delta_worker_time AS worker_time_sum, " + + "delta_elapsed_time AS elapsed_time_sum FROM collect.query_stats) AS f"; + var sql = TopRankings.Apply(DarlingDataReader.TopQueriesHourlySql, TopRanking.Cpu, hourly: true) + .Replace("$FROM$", standIn, StringComparison.Ordinal) + .Replace("$IOSUMS$", TopRankings.HourlyNoIoSums, StringComparison.Ordinal) + .Replace("$CEIL$", "", StringComparison.Ordinal); + var plan = await ExplainAsync(connection, sql, ct, serverId, now.AddHours(-24), now.AddMinutes(5), + P(NpgsqlDbType.Integer, Top), P(NpgsqlDbType.Array | NpgsqlDbType.Text, null), P(NpgsqlDbType.Integer, Candidates)); + + var lookup = Nodes(plan.Root, null).Where(n => n.Cte == "latest_in_window" && n.Relation == "query_stats").ToList(); + Assert.NotEmpty(lookup); + /* Rows the lookup touched in raw: what each scan returned plus what its filter threw away, over every loop. */ + var read = lookup.Sum(n => (n.Rows + n.Removed) * n.Loops); + var windowRows = (long)(Shells + Real) * RowsPerGroup; + var rankedKeys = Candidates; + Assert.True(read <= rankedKeys * 4, + $"the hourly text lookup read {read} raw rows [{string.Join("; ", lookup.Select(n => $"{n.NodeType} loops={n.Loops} rows={n.Rows} removed={n.Removed}"))}] for {rankedKeys} ranked keys over a window of {windowRows}; it must read about one per key, not the window"); + } + + [Fact] + public Task TheHourlyLookup_StillFindsTheText_ForANullKey_ATextlessNewestRow_AndAKeyWithNoRowInTheWindow() => + WithSeedAsync(async (connection, serverId, now, ct) => + { + async Task Plant(string? database, string? hash, DateTime at, string? text) => + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO query_stats + (collection_id, collection_time, server_id, server_name, database_name, query_hash, query_plan_hash, sql_handle, plan_handle, + query_text, delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, + delta_logical_writes, delta_physical_reads, min_worker_time, max_worker_time, min_elapsed_time, max_elapsed_time, + min_dop, max_dop, sample_interval_seconds) + VALUES ($1, $2, $3, $4, $5, $6, '0xP', '0xS', '0xL', $7, 10, $8, 100, 100, 0, 0, 1, 100, 1, 100, 1, 1, 3600)", + CollectionIdGenerator.Next(), at, serverId, "a5299-hourly-keys", database, hash, text, 1_000_000L - (long)(now - at).TotalMinutes); + + /* The newest row in the window has no text: the lookup takes the newest row that does, inside the window. */ + await Plant("D1", "0xH1", now.AddHours(-3), "older text in the window"); + await Plant("D1", "0xH1", now.AddHours(-1), null); + /* A group with a NULL database and a NULL hash still gets its text (strict equality would have lost it). */ + await Plant(null, null, now.AddHours(-2), "text of the null-keyed group"); + /* A group the window holds no text for: the rollup outlives raw, so the fallback finds the newest text raw has at all. */ + await Plant("D3", "0xH3", now.AddHours(-2), null); + await Plant("D3", "0xH3", now.AddDays(-10), "text outside the window"); + + var standIn = "(SELECT server_id, collection_time AS bucket, database_name, query_hash, sql_handle, " + + "delta_execution_count AS execution_count_sum, delta_worker_time AS worker_time_sum, " + + "delta_elapsed_time AS elapsed_time_sum FROM collect.query_stats WHERE collection_time >= '" + + now.AddDays(-2).ToString("yyyy-MM-dd HH:mm:ss", CultureInfo.InvariantCulture) + "') AS f"; + /* The stand-in keeps the window rows only, so the fallback's ten-day-old row is read by latest_any from raw, as in life. */ + var sql = TopRankings.Apply(DarlingDataReader.TopQueriesHourlySql, TopRanking.Cpu, hourly: true) + .Replace("$FROM$", standIn, StringComparison.Ordinal) + .Replace("$IOSUMS$", TopRankings.HourlyNoIoSums, StringComparison.Ordinal) + .Replace("$CEIL$", "", StringComparison.Ordinal); + await using var command = new NpgsqlCommand(sql, connection); + command.Parameters.Add(P(NpgsqlDbType.Integer, serverId)); + command.Parameters.Add(P(NpgsqlDbType.Timestamp, DateTime.SpecifyKind(now.AddHours(-24), DateTimeKind.Unspecified))); + command.Parameters.Add(P(NpgsqlDbType.Timestamp, DateTime.SpecifyKind(now.AddMinutes(5), DateTimeKind.Unspecified))); + command.Parameters.Add(P(NpgsqlDbType.Integer, 10)); + command.Parameters.Add(P(NpgsqlDbType.Array | NpgsqlDbType.Text, null)); + command.Parameters.Add(P(NpgsqlDbType.Integer, 10)); + var texts = new Dictionary(); + await using (var reader = await command.ExecuteReaderAsync(ct)) + { + while (await reader.ReadAsync(ct)) + { + if (!reader.IsDBNull(10)) + { + texts[(reader.IsDBNull(0) ? "(null)" : reader.GetString(0)) + "/" + (reader.IsDBNull(1) ? "(null)" : reader.GetString(1))] = + reader.IsDBNull(6) ? null : reader.GetString(6); + } + } + } + + Assert.Equal(3, texts.Count); + Assert.Equal("older text in the window", texts["D1/0xH1"]); + Assert.Equal("text of the null-keyed group", texts["(null)/(null)"]); + Assert.Equal("text outside the window", texts["D3/0xH3"]); + }, "a5299-hourly-keys", bulk: false); + + [Fact] + public Task TheHourlyFallback_ProbesEachMissingKey_NotTheWholeRawTable() => + WithSeedAsync(async (connection, serverId, now, ct) => + { + /* #5299 round 2, N1: a ranked key whose only text lies outside the window (a window past raw retention) lands in + latest_any. It used to scan every raw row of the server and join them to the missing keys; it must probe + idx_query_stats_server_hash_time once per missing key instead. 125 keys each have one textless row in the + window and 20 rows with a text ten days back, and 5,000 rows of other keys sit beside them, so a whole-table + read is about 7,600 rows and a per-key probe is a handful per key. */ + const string indexName = "idx_query_stats_server_hash_time"; + await DarlingMcpTestData.ExecAsync(connection, ct, + "CREATE INDEX IF NOT EXISTS " + indexName + " ON collect.query_stats (server_id, query_hash, collection_time DESC)"); + /* The database is this test's own scratch store, so the index goes with it: no teardown to run. */ + const string columns = @"(collection_id, collection_time, server_id, server_name, database_name, query_hash, query_plan_hash, sql_handle, plan_handle, + query_text, delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, + delta_logical_writes, delta_physical_reads, min_worker_time, max_worker_time, min_elapsed_time, max_elapsed_time, + min_dop, max_dop, sample_interval_seconds)"; + await DarlingMcpTestData.ExecAsync(connection, ct, + "INSERT INTO query_stats " + columns + @" + SELECT $1 + g, $2, $3, $4, 'PlanDb', '0xK' || g, '0xP', '0xS', '0xL', NULL, 10, 1000000 - g, 100, 100, 0, 0, 1, 100, 1, 100, 1, 1, 3600 + FROM generate_series(1, $5) AS g", + CollectionIdGenerator.Next(), now.AddHours(-1), serverId, ServerName, Candidates); + await DarlingMcpTestData.ExecAsync(connection, ct, + "INSERT INTO query_stats " + columns + @" + SELECT $1 + g * 100 + r, $2 - (r * interval '1 minute'), $3, $4, 'PlanDb', '0xK' || g, '0xP', '0xS', '0xL', 'a5299-plan old text ' || g, 10, 100, 100, 100, 0, 0, 1, 100, 1, 100, 1, 1, 3600 + FROM generate_series(1, $5) AS g CROSS JOIN generate_series(1, 20) AS r", + CollectionIdGenerator.Next() + 1_000_000, now.AddDays(-10), serverId, ServerName, Candidates); + await DarlingMcpTestData.ExecAsync(connection, ct, + "INSERT INTO query_stats " + columns + @" + SELECT $1 + g, $2 - (g * interval '1 second'), $3, $4, 'PlanDb', '0xF' || g, '0xP', '0xS', '0xL', 'a5299-plan filler ' || g, 10, 100, 100, 100, 0, 0, 1, 100, 1, 100, 1, 1, 3600 + FROM generate_series(1, 5000) AS g", + CollectionIdGenerator.Next() + 2_000_000, now.AddDays(-9), serverId, ServerName); + await DarlingMcpTestData.ExecAsync(connection, ct, "ANALYZE collect.query_stats"); + + /* The stand-in keeps the window rows only (the rollup outlives raw), so the ranked keys are the 125 textless + keys and every text has to come from the fallback. */ + var standIn = "(SELECT server_id, collection_time AS bucket, database_name, query_hash, sql_handle, " + + "delta_execution_count AS execution_count_sum, delta_worker_time AS worker_time_sum, " + + "delta_elapsed_time AS elapsed_time_sum FROM collect.query_stats WHERE collection_time >= '" + + now.AddDays(-2).ToString("yyyy-MM-dd HH:mm:ss", CultureInfo.InvariantCulture) + "') AS f"; + var sql = TopRankings.Apply(DarlingDataReader.TopQueriesHourlySql, TopRanking.Cpu, hourly: true) + .Replace("$FROM$", standIn, StringComparison.Ordinal) + .Replace("$IOSUMS$", TopRankings.HourlyNoIoSums, StringComparison.Ordinal) + .Replace("$CEIL$", "", StringComparison.Ordinal); + var plan = await ExplainAsync(connection, sql, ct, serverId, now.AddHours(-24), now.AddMinutes(5), + P(NpgsqlDbType.Integer, Top), P(NpgsqlDbType.Array | NpgsqlDbType.Text, null), P(NpgsqlDbType.Integer, Candidates)); + + /* Every raw read that is neither the ranking nor the in-window lookup is the fallback (the old shape was inlined into the join, so no CTE name). */ + var lookup = Nodes(plan.Root, null).Where(n => n.Relation == "query_stats" && n.Cte != "ranked" && n.Cte != "latest_in_window").ToList(); + Assert.NotEmpty(lookup); + var read = lookup.Sum(n => (n.Rows + n.Removed) * n.Loops); + Assert.True(read <= Candidates * 4, + $"the hourly fallback read {read} raw rows [{string.Join("; ", lookup.Select(n => $"{n.NodeType} loops={n.Loops} rows={n.Rows} removed={n.Removed}"))}] for {Candidates} missing keys; " + + "it must probe the index about once per key, not read the raw table"); + }, "a5299-hourly-fallback", bulk: false); + + // ------------------------------------------------------------------ plan reading + + private static void AssertLookupRanOnce(string name, Plan plan) + { + var all = Nodes(plan.Root, null).ToList(); + /* The defect, as the plan shows it: a sequential scan of the text dimension that ran once per ranked row. */ + var rescans = all.Where(n => n.Relation == "query_text_dim" && n.NodeType == "Seq Scan" && n.Loops > 1).ToList(); + var lookup = all.Where(n => n.Cte == "latest_text").ToList(); + Assert.True(rescans.Count == 0 && lookup.Count > 0, + $"{name}: latest_text is not its own CTE node (found {lookup.Count} nodes), so PostgreSQL inlined it into the join and re-runs it per ranked row; " + + $"query_text_dim was scanned {string.Join(", ", rescans.Select(n => n.Loops + " times"))} (add MATERIALIZED)"); + var root = lookup.OrderBy(n => n.Depth).First(); + Assert.True(root.Loops == 1, $"{name}: the text lookup ran {root.Loops} times; it must run once"); + } + + private sealed record Plan(JsonElement Root, double Ms); + + private sealed record PlanNode(string NodeType, string? Relation, string? Cte, long Loops, long Rows, long Removed, int Depth); + + /// Walks the plan. Cte is the CTE a node sits under ("CTE name" in Subplan Name), or null. + private static IEnumerable Nodes(JsonElement node, string? cte, int depth = 0) + { + if (node.TryGetProperty("Subplan Name", out var sub) && sub.GetString() is { } s && s.StartsWith("CTE ", StringComparison.Ordinal)) + { + cte = s[4..]; + } + + long Get(string p) => node.TryGetProperty(p, out var v) && v.ValueKind == JsonValueKind.Number ? (long)v.GetDouble() : 0; + yield return new PlanNode( + node.GetProperty("Node Type").GetString()!, + node.TryGetProperty("Relation Name", out var r) ? r.GetString() : null, + cte, Get("Actual Loops"), Get("Actual Rows"), + Get("Rows Removed by Filter") + Get("Rows Removed by Index Recheck"), depth); + + if (node.TryGetProperty("Plans", out var plans)) + { + foreach (var child in plans.EnumerateArray()) + { + foreach (var n in Nodes(child, cte, depth + 1)) + { + yield return n; + } + } + } + } + + private static NpgsqlParameter P(NpgsqlDbType type, object? value) => new() { NpgsqlDbType = type, Value = value ?? DBNull.Value }; + + private static async Task ExplainAsync(NpgsqlConnection connection, string sql, CancellationToken ct, + int serverId, DateTime start, DateTime end, params NpgsqlParameter[] rest) + { + await using var command = new NpgsqlCommand("EXPLAIN (ANALYZE, FORMAT JSON) " + sql, connection) { CommandTimeout = 120 }; + command.Parameters.Add(P(NpgsqlDbType.Integer, serverId)); + command.Parameters.Add(P(NpgsqlDbType.Timestamp, DateTime.SpecifyKind(start, DateTimeKind.Unspecified))); + command.Parameters.Add(P(NpgsqlDbType.Timestamp, DateTime.SpecifyKind(end, DateTimeKind.Unspecified))); + foreach (var p in rest) + { + command.Parameters.Add(p); + } + + var json = (string)(await command.ExecuteScalarAsync(ct))!; + var doc = JsonDocument.Parse(json).RootElement[0]; + return new Plan(doc.GetProperty("Plan").Clone(), doc.GetProperty("Execution Time").GetDouble()); + } + + // ------------------------------------------------------------------ the seed + + /// + /// 120 WAITFOR shells that outrank 30 real groups, twelve rows each, texts only in the dimension (the modern shape: + /// the stat row keeps a digest, not the text). The tables are NOT analyzed, which is what makes the planner estimate + /// one row for the lookup on a store the collectors have just filled. + /// + private static async Task WithSeedAsync(Func body, string serverName = ServerName, bool bulk = true) + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to run the live plan tests (each mints its own scratch database)."); + var ct = TestContext.Current.CancellationToken; + /* #4650: a scratch database, so no other class's leftover chunks shape the plan under test. */ + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var connectionString = scratch.ConnectionString; + var serverId = ServerIdHelper.GetDeterministicHashCode(serverName); + var cleanup = string.Format(CultureInfo.InvariantCulture, + "DELETE FROM query_stats WHERE server_id = {0}; DELETE FROM query_text_dim WHERE query_text LIKE '%a5299-plan%'; DELETE FROM servers WHERE server_id = {0}", serverId); + + using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DarlingMcpTestData.ExecAsync(connection, ct, cleanup); + var succeeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, serverId, serverName, ct); + var now = DarlingMcpTestData.Naive(DateTime.SpecifyKind(LiveClock.Now(), DateTimeKind.Utc)); + if (bulk) + { + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO query_text_dim (digest, query_text, last_seen) + SELECT sha256(convert_to(t.text, 'UTF8')), t.text, now() AT TIME ZONE 'UTC' + FROM (SELECT CASE WHEN g <= $1 THEN 'WAITFOR DELAY ''00:00:30'' /* a5299-plan ' || g || ' */' + ELSE 'SELECT real /* a5299-plan ' || g || ' */' END AS text + FROM generate_series(1, $1 + $2) AS g) AS t + ON CONFLICT (digest) DO NOTHING", + Shells, Real); + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO query_stats + (collection_id, collection_time, server_id, server_name, database_name, query_hash, query_plan_hash, sql_handle, plan_handle, + query_text, query_text_digest, delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, + delta_logical_writes, delta_physical_reads, min_worker_time, max_worker_time, min_elapsed_time, max_elapsed_time, + min_dop, max_dop, sample_interval_seconds) + SELECT $1 + g * 100 + r, $2 - (r * interval '1 hour'), $3, $4, 'PlanDb', '0xH' || g, '0xP' || g, '0xS' || g, '0xL' || g, + NULL, + sha256(convert_to(CASE WHEN g <= $5 THEN 'WAITFOR DELAY ''00:00:30'' /* a5299-plan ' || g || ' */' + ELSE 'SELECT real /* a5299-plan ' || g || ' */' END, 'UTF8')), + 10, (CASE WHEN g <= $5 THEN 2000000 ELSE 20000 END) - g * 100, 100, 100, 0, 0, 1, 100, 1, 100, 1, 1, 3600 + FROM generate_series(1, $5 + $6) AS g CROSS JOIN generate_series(1, $7) AS r", + CollectionIdGenerator.Next(), now.AddMinutes(-30), serverId, serverName, Shells, Real, RowsPerGroup); + } + + await body(connection, serverId, now, ct); + succeeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, succeeded, async (cleanupConnection, cleanupCt) => + await DarlingMcpTestData.ExecAsync(cleanupConnection, cleanupCt, cleanup)); + } + } +} diff --git a/Darling/Darling.Tests/TopTextTieBreakLiveTests.cs b/Darling/Darling.Tests/TopTextTieBreakLiveTests.cs new file mode 100644 index 0000000000..39423fecf1 --- /dev/null +++ b/Darling/Darling.Tests/TopTextTieBreakLiveTests.cs @@ -0,0 +1,222 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using Darling.Tests; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace PerformanceMonitor.Darling.Tests; + +/// +/// #5299 round 2 (N3 and N6): a newest-text pick and a top statement are deterministic when two rows tie. Two rows of one key at the +/// same collection_time with different texts (one shape hash, two plans, one collection) used to give an arbitrary text, which +/// also decides the WAITFOR trim; two groups with an exactly equal ranking value used to come out in an arbitrary order, so a refill +/// round could pick different members. The text now breaks the tie on collection_id DESC (the row collected last, unique per +/// row) and the ranking on the group key. Each case seeds the tie so the arbitrary answer is the WRONG one (the lower id is inserted +/// first, and a heap scan returns it first), reads the statement three times, and asserts the same answer each time. +/// +[Collection("live-postgres")] +public sealed class TopTextTieBreakLiveTests +{ + private const string TextServer = "a5299-tie-text"; + private const string StoreServer = "a5299-tie-store"; + + private static NpgsqlParameter P(NpgsqlDbType type, object? value) => new() { NpgsqlDbType = type, Value = value ?? DBNull.Value }; + + private const string StatsColumns = @"(collection_id, collection_time, server_id, server_name, database_name, query_hash, query_plan_hash, sql_handle, plan_handle, + query_text, delta_execution_count, delta_worker_time, delta_elapsed_time, delta_logical_reads, + delta_logical_writes, delta_physical_reads, min_worker_time, max_worker_time, min_elapsed_time, max_elapsed_time, + min_dop, max_dop, sample_interval_seconds)"; + + private static async Task PlantStatAsync(NpgsqlConnection connection, int serverId, string serverName, string hash, DateTime at, string? text, long cpu, CancellationToken ct) => + await DarlingMcpTestData.ExecAsync(connection, ct, + "INSERT INTO query_stats " + StatsColumns + " VALUES ($1, $2, $3, $4, 'D1', $5, '0xP', '0xS', '0xL', $6, 10, $7, 100, 100, 0, 0, 1, 100, 1, 100, 1, 1, 3600)", + CollectionIdGenerator.Next(), at, serverId, serverName, hash, text, cpu); + + private static async Task PlantStoreAsync(NpgsqlConnection connection, int serverId, string serverName, long queryId, DateTime at, string text, CancellationToken ct, long? collectionId = null) => + await DarlingMcpTestData.ExecAsync(connection, ct, + @"INSERT INTO query_store_stats (collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, execution_type_desc, query_hash, query_plan_hash, query_text, execution_count, avg_duration_us, avg_cpu_time_us, avg_logical_io_reads, avg_logical_io_writes, avg_physical_io_reads, avg_rowcount, min_dop, max_dop, last_execution_time) + VALUES ($1,$2,$3,$4,'D1',$5,7,'Regular','0xQH','0xPLAN',$6,10,5000,4000,800,0,40,250,1,4,$2)", + collectionId ?? CollectionIdGenerator.Next(), at, serverId, serverName, queryId, text); + + /// Runs the statement and returns, per page row, the group key text and the representative text. + private static async Task> ReadPageAsync(NpgsqlConnection connection, string sql, string keyColumn, CancellationToken ct, + int serverId, DateTime start, DateTime end, params NpgsqlParameter[] rest) + { + await using var command = new NpgsqlCommand(sql, connection) { CommandTimeout = 60 }; + command.Parameters.Add(P(NpgsqlDbType.Integer, serverId)); + command.Parameters.Add(P(NpgsqlDbType.Timestamp, DateTime.SpecifyKind(start, DateTimeKind.Unspecified))); + command.Parameters.Add(P(NpgsqlDbType.Timestamp, DateTime.SpecifyKind(end, DateTimeKind.Unspecified))); + foreach (var p in rest) + { + command.Parameters.Add(p); + } + + var rows = new List<(string?, string?)>(); + await using var reader = await command.ExecuteReaderAsync(ct); + var page = reader.GetOrdinal("page_ord"); + var key = reader.GetOrdinal(keyColumn); + var text = reader.GetOrdinal("query_text"); + while (await reader.ReadAsync(ct)) + { + if (!reader.IsDBNull(page)) + { + rows.Add((reader.IsDBNull(key) ? null : Convert.ToString(reader.GetValue(key), CultureInfo.InvariantCulture), + reader.IsDBNull(text) ? null : reader.GetString(text))); + } + } + + return rows; + } + + private static async Task WithServerAsync(string serverName, Func body) + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), "Set DARLING_TEST_PG to run the live tie-break tests."); + var ct = TestContext.Current.CancellationToken; + var serverId = ServerIdHelper.GetDeterministicHashCode(serverName); + var cleanup = string.Format(CultureInfo.InvariantCulture, + "DELETE FROM query_stats WHERE server_id = {0}; DELETE FROM query_store_stats WHERE server_id = {0}; DELETE FROM servers WHERE server_id = {0}", serverId); + + using var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DarlingMcpTestData.ExecAsync(connection, ct, cleanup); + var succeeded = false; + try + { + await DarlingMcpTestData.RegisterServerAsync(connection, serverId, serverName, ct); + await body(connection, serverId, DarlingMcpTestData.Naive(DateTime.UtcNow), ct); + succeeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(connectionString!, succeeded, async (cleanupConnection, cleanupCt) => + await DarlingMcpTestData.ExecAsync(cleanupConnection, cleanupCt, cleanup)); + } + } + + [Fact] + public Task TheRawAndViewerStatements_PickTheTextOfTheRowCollectedLast_WhenTwoRowsShareATime() => + WithServerAsync(TextServer, async (connection, serverId, now, ct) => + { + var tie = now.AddHours(-2); + await PlantStatAsync(connection, serverId, TextServer, "0xT1", tie, "tie text A (lower collection id)", 5000, ct); + await PlantStatAsync(connection, serverId, TextServer, "0xT1", tie, "tie text B (higher collection id)", 5000, ct); + + var start = now.AddHours(-24); + var end = now.AddMinutes(5); + for (var run = 0; run < 3; run++) + { + foreach (var sql in new[] { DarlingDataReader.TopQueriesSql, DarlingDataReader.TopQueriesByHostObjectSql }) + { + var page = await ReadPageAsync(connection, TopRankings.Apply(sql, TopRanking.Cpu, hourly: false), "query_hash", ct, serverId, start, end, + P(NpgsqlDbType.Integer, 5), P(NpgsqlDbType.Array | NpgsqlDbType.Text, null), P(NpgsqlDbType.Integer, 0), P(NpgsqlDbType.Integer, 10)); + Assert.Equal("tie text B (higher collection id)", Assert.Single(page).Text); + } + + var viewer = await ReadPageAsync(connection, ViewerDataService.TopQueriesSql, "query_hash", ct, serverId, start, end, + P(NpgsqlDbType.Integer, 5), P(NpgsqlDbType.Array | NpgsqlDbType.Text, null), P(NpgsqlDbType.Integer, 10)); + Assert.Equal("tie text B (higher collection id)", Assert.Single(viewer).Text); + } + }); + + [Fact] + public Task TheHourlyStatement_PicksTheTextOfTheRowCollectedLast_InTheWindowAndInTheFallback() => + WithServerAsync(TextServer, async (connection, serverId, now, ct) => + { + /* In the window: two rows at one time. In the fallback: the key's window row has no text, and two rows ten days back share a time. */ + await PlantStatAsync(connection, serverId, TextServer, "0xT1", now.AddHours(-2), "window text A", 5000, ct); + await PlantStatAsync(connection, serverId, TextServer, "0xT1", now.AddHours(-2), "window text B", 5000, ct); + await PlantStatAsync(connection, serverId, TextServer, "0xT2", now.AddHours(-3), null, 4000, ct); + await PlantStatAsync(connection, serverId, TextServer, "0xT2", now.AddDays(-10), "old text A", 4000, ct); + await PlantStatAsync(connection, serverId, TextServer, "0xT2", now.AddDays(-10), "old text B", 4000, ct); + + var standIn = "(SELECT server_id, collection_time AS bucket, database_name, query_hash, sql_handle, " + + "delta_execution_count AS execution_count_sum, delta_worker_time AS worker_time_sum, " + + "delta_elapsed_time AS elapsed_time_sum FROM collect.query_stats WHERE collection_time >= '" + + now.AddDays(-2).ToString("yyyy-MM-dd HH:mm:ss", CultureInfo.InvariantCulture) + "') AS f"; + var sql = TopRankings.Apply(DarlingDataReader.TopQueriesHourlySql, TopRanking.Cpu, hourly: true) + .Replace("$FROM$", standIn, StringComparison.Ordinal) + .Replace("$IOSUMS$", TopRankings.HourlyNoIoSums, StringComparison.Ordinal) + .Replace("$CEIL$", "", StringComparison.Ordinal); + for (var run = 0; run < 3; run++) + { + var page = await ReadPageAsync(connection, sql, "query_hash", ct, serverId, now.AddHours(-24), now.AddMinutes(5), + P(NpgsqlDbType.Integer, 5), P(NpgsqlDbType.Array | NpgsqlDbType.Text, null), P(NpgsqlDbType.Integer, 10)); + Assert.Equal(2, page.Count); + Assert.Equal("window text B", page.Find(r => r.Key == "0xT1").Text); + Assert.Equal("old text B", page.Find(r => r.Key == "0xT2").Text); + } + }); + + [Fact] + public Task TheQueryStoreStatements_PickTheTextOfTheRowCollectedLast_AndTheLowestKeyOnATie() => + WithServerAsync(StoreServer, async (connection, serverId, now, ct) => + { + var tie = now.AddHours(-2); + /* Query 20 is inserted before query 10, and both rank equally: the page of one must be query 10 on every read. */ + await PlantStoreAsync(connection, serverId, StoreServer, 20, tie, "store text 20", ct); + await PlantStoreAsync(connection, serverId, StoreServer, 10, tie, "store text 10 A (lower collection id)", ct); + await PlantStoreAsync(connection, serverId, StoreServer, 10, tie, "store text 10 B (higher collection id)", ct); + /* Query 11's rows go in the other way round (the higher collection id first), so neither insertion order can be the answer. */ + var baseId = CollectionIdGenerator.Next() + 1_000_000; + await PlantStoreAsync(connection, serverId, StoreServer, 11, tie, "store text 11 B (higher collection id)", ct, baseId + 2); + await PlantStoreAsync(connection, serverId, StoreServer, 11, tie, "store text 11 A (lower collection id)", ct, baseId + 1); + + var start = now.AddHours(-24); + var end = now.AddMinutes(5); + for (var run = 0; run < 3; run++) + { + var page = await ReadPageAsync(connection, DarlingDataReader.QueryStoreTopSql, "query_id", ct, serverId, start, end, + P(NpgsqlDbType.Integer, 1), P(NpgsqlDbType.Array | NpgsqlDbType.Text, null), P(NpgsqlDbType.Text, null), P(NpgsqlDbType.Text, null), P(NpgsqlDbType.Integer, 1)); + Assert.Equal("10", Assert.Single(page).Key); + var viewerOne = await ReadPageAsync(connection, ViewerDataService.QueryStoreTopSql, "query_id", ct, serverId, start, end, + P(NpgsqlDbType.Integer, 1), P(NpgsqlDbType.Array | NpgsqlDbType.Text, null), P(NpgsqlDbType.Integer, 1)); + Assert.Equal("10", Assert.Single(viewerOne).Key); + + var all = await ReadPageAsync(connection, DarlingDataReader.QueryStoreTopSql, "query_id", ct, serverId, start, end, + P(NpgsqlDbType.Integer, 3), P(NpgsqlDbType.Array | NpgsqlDbType.Text, null), P(NpgsqlDbType.Text, null), P(NpgsqlDbType.Text, null), P(NpgsqlDbType.Integer, 3)); + var viewerAll = await ReadPageAsync(connection, ViewerDataService.QueryStoreTopSql, "query_id", ct, serverId, start, end, + P(NpgsqlDbType.Integer, 3), P(NpgsqlDbType.Array | NpgsqlDbType.Text, null), P(NpgsqlDbType.Integer, 3)); + foreach (var rows in new[] { all, viewerAll }) + { + Assert.Equal(new string?[] { "10", "11", "20" }, rows.ConvertAll(r => r.Key).ToArray()); + Assert.Equal("store text 10 B (higher collection id)", rows[0].Text); + Assert.Equal("store text 11 B (higher collection id)", rows[1].Text); + } + } + }); + + [Fact] + public Task TheViewerTopQueries_BreaksAnExactTieOnTheGroupKey() => + WithServerAsync(TextServer, async (connection, serverId, now, ct) => + { + /* Two groups with equal elapsed time; the higher hash is inserted first. One candidate: the lowest key must win every time. */ + await PlantStatAsync(connection, serverId, TextServer, "0xT9", now.AddHours(-2), "viewer text 9", 5000, ct); + await PlantStatAsync(connection, serverId, TextServer, "0xT3", now.AddHours(-2), "viewer text 3", 5000, ct); + + for (var run = 0; run < 3; run++) + { + var page = await ReadPageAsync(connection, ViewerDataService.TopQueriesSql, "query_hash", ct, serverId, now.AddHours(-24), now.AddMinutes(5), + P(NpgsqlDbType.Integer, 1), P(NpgsqlDbType.Array | NpgsqlDbType.Text, null), P(NpgsqlDbType.Integer, 1)); + Assert.Equal(("0xT3", "viewer text 3"), Assert.Single(page)); + } + }); +} diff --git a/Darling/Darling.Tests/TrendPayloadBudgetLiveTests.cs b/Darling/Darling.Tests/TrendPayloadBudgetLiveTests.cs index 0a3d3f81c6..fcc0073506 100644 --- a/Darling/Darling.Tests/TrendPayloadBudgetLiveTests.cs +++ b/Darling/Darling.Tests/TrendPayloadBudgetLiveTests.cs @@ -225,12 +225,18 @@ private static void Measure(List measured, string tool, int hours, strin /// database's pg_stat_database — counters cumulative where the collector stores them cumulative. And, over /// the last four hours only, 300 tenant databases with a data and a log file each: the 600-file server. /// - private static async Task SeedAsync(NpgsqlConnection connection, DateTime end, CancellationToken ct) + internal static async Task SeedAsync( + NpgsqlConnection connection, DateTime end, CancellationToken ct, int? serverId = null, string? serverName = null) { + /* #5425: TrendStaleStatsLiveTests seeds this same shape for a second server, a month earlier. */ + var id = serverId ?? ServerId; + var name = serverName ?? ServerName; + Task Plant(string sql, long idOffset, DateTime first, int minutes) => PlantAsync(connection, ct, sql, idOffset, first, minutes, id, name); + var weekStart = end.AddMinutes(-WeekMinutes); var rawStart = end.AddMinutes(-RawMinutes); - await PlantAsync(connection, ct, @" + await Plant(@" INSERT INTO file_io_stats (collection_id, collection_time, server_id, server_name, database_name, file_name, file_type, physical_name, size_mb, delta_reads, delta_writes, delta_read_bytes, delta_write_bytes, @@ -255,7 +261,7 @@ CROSS JOIN (VALUES (11, 'msdb', 'MSDBData.mdf', 'ROWS') ) AS f(i, db, file, kind)", 1_000_000L, weekStart, WeekMinutes); - await PlantAsync(connection, ct, @" + await Plant(@" INSERT INTO file_io_stats (collection_id, collection_time, server_id, server_name, database_name, file_name, file_type, physical_name, size_mb, delta_reads, delta_writes, delta_read_bytes, delta_write_bytes, @@ -269,7 +275,7 @@ FROM generate_series(0, $5) AS n CROSS JOIN generate_series(0, 299) AS d(i) CROSS JOIN (VALUES (0, '.mdf', 'ROWS'), (1, '_log.ldf', 'LOG')) AS k(j, suffix, kind)", 6_000_000L, end.AddHours(-4), 240); - await PlantAsync(connection, ct, @" + await Plant(@" INSERT INTO wait_stats (collection_id, collection_time, server_id, server_name, wait_type, delta_wait_time_ms, delta_signal_wait_time_ms, delta_waiting_tasks, sample_interval_seconds) @@ -280,7 +286,7 @@ CROSS JOIN (VALUES (0, 'LCK_M_S'), (1, 'LCK_M_U'), (2, 'LCK_M_X'), (3, 'LCK_M_IS (6, 'LCK_M_SIU'), (7, 'LCK_M_SIX'), (8, 'LCK_M_UIX'), (9, 'LCK_M_BU'), (10, 'LCK_M_RS_S'), (11, 'LCK_M_RIn_NL') ) AS w(i, wait_type)", 2_000_000L, weekStart, WeekMinutes); - await PlantAsync(connection, ct, @" + await Plant(@" INSERT INTO query_stats (collection_id, collection_time, server_id, server_name, database_name, query_hash, query_plan_hash, sql_handle, plan_handle, query_text, execution_count, total_worker_time, total_elapsed_time, @@ -291,7 +297,7 @@ INSERT INTO query_stats FROM generate_series(0, $5) AS n CROSS JOIN (VALUES (0, '0x1C8F0E7D55A2B391'), (1, '0x2D90F18E66B3C4A2'), (2, '0x3EA1029F77C4D5B3')) AS q(i, hash)", 3_000_000L, rawStart, RawMinutes); - await PlantAsync(connection, ct, @" + await Plant(@" WITH s AS ( SELECT n, o.i, o.object_type, 1100 + (n % 13) * 57 AS r FROM generate_series(0, $5) AS n @@ -306,7 +312,7 @@ INSERT INTO pg_io_stats FROM s WINDOW w AS (PARTITION BY i ORDER BY n)", 4_000_000L, weekStart, WeekMinutes); - await PlantAsync(connection, ct, @" + await Plant(@" WITH s AS ( SELECT n, 6000 + (n % 11) * 217 AS commits, @@ -329,7 +335,7 @@ FROM s /* #3960: the rest of the trend family, on the same week. One series each is enough for a payload size and cap census -- get_wait_trend's own series count comes from wait_stats' LCK_M_S rows above. */ - await PlantAsync(connection, ct, @" + await Plant(@" INSERT INTO cpu_utilization_stats (collection_id, collection_time, server_id, server_name, sample_time, sqlserver_cpu_utilization, other_process_cpu_utilization) @@ -337,7 +343,7 @@ INSERT INTO cpu_utilization_stats 40 + (n % 30), 10 + (n % 5) FROM generate_series(0, $5) AS n", 7_000_000L, weekStart, WeekMinutes); - await PlantAsync(connection, ct, @" + await Plant(@" INSERT INTO tempdb_stats (collection_id, collection_time, server_id, server_name, user_object_reserved_mb, internal_object_reserved_mb, version_store_reserved_mb, @@ -347,7 +353,7 @@ INSERT INTO tempdb_stats 175 + (n % 80), 825 - (n % 80), 5 + (n % 10), 55 + (n % 3), 12 + (n % 5) FROM generate_series(0, $5) AS n", 8_000_000L, weekStart, WeekMinutes); - await PlantAsync(connection, ct, @" + await Plant(@" INSERT INTO memory_stats (collection_id, collection_time, server_id, server_name, total_server_memory_mb, target_server_memory_mb, buffer_pool_mb, plan_cache_mb) @@ -355,13 +361,13 @@ INSERT INTO memory_stats 40000 + (n % 500), 49152, 35000 + (n % 400), 5000 + (n % 100) FROM generate_series(0, $5) AS n", 9_000_000L, weekStart, WeekMinutes); - await PlantAsync(connection, ct, @" + await Plant(@" INSERT INTO memory_grant_stats (collection_id, collection_time, server_id, server_name, resource_semaphore_id, pool_id, granted_memory_mb) SELECT $1 + n, $2 + n * interval '1 minute', $3, $4, 0, 2, 50 + (n % 30) FROM generate_series(0, $5) AS n", 10_000_000L, weekStart, WeekMinutes); - await PlantAsync(connection, ct, @" + await Plant(@" INSERT INTO perfmon_stats (collection_id, collection_time, server_id, server_name, object_name, counter_name, instance_name, cntr_value, delta_cntr_value, sample_interval_seconds) @@ -369,7 +375,7 @@ INSERT INTO perfmon_stats 5000000 + n * 900, 900 + (n % 50) * 3, 60 FROM generate_series(0, $5) AS n", 11_000_000L, weekStart, WeekMinutes); - await PlantAsync(connection, ct, $@" + await Plant($@" INSERT INTO pg_statement_stats (collection_id, collection_time, server_id, server_name, queryid, delta_calls, delta_total_exec_time_ms, sample_interval_seconds) @@ -380,7 +386,7 @@ INSERT INTO pg_statement_stats /* #4193: get_pg_cpu_utilization's own table, one row/minute like the rest of the family, with the V136 host-memory columns populated (not left NULL) since those are what doubled the pre-bucketing row width the issue measured. */ - await PlantAsync(connection, ct, @" + await Plant(@" INSERT INTO pg_cpu_utilization (collection_id, collection_time, server_id, server_name, sample_time, cpu_percent, acu_utilization_percent, serverless_capacity_acu, max_configured_acu, @@ -393,13 +399,14 @@ INSERT INTO pg_cpu_utilization /// One generate_series plant: $1 an id base past anything the generator has handed out, $2 the first /// collection, $3/$4 the server, $5 the last minute's index. - private static async Task PlantAsync(NpgsqlConnection connection, CancellationToken ct, string sql, long idOffset, DateTime first, int minutes) + private static async Task PlantAsync( + NpgsqlConnection connection, CancellationToken ct, string sql, long idOffset, DateTime first, int minutes, int serverId, string serverName) { using var plant = new NpgsqlCommand(sql, connection) { CommandTimeout = 300 }; plant.Parameters.AddWithValue(CollectionIdGenerator.Next() + idOffset * 100); plant.Parameters.AddWithValue(DarlingMcpTestData.Naive(first)); - plant.Parameters.AddWithValue(ServerId); - plant.Parameters.AddWithValue(ServerName); + plant.Parameters.AddWithValue(serverId); + plant.Parameters.AddWithValue(serverName); plant.Parameters.AddWithValue(minutes); await plant.ExecuteNonQueryAsync(ct); } diff --git a/Darling/Darling.Tests/TrendStaleStatsLiveTests.cs b/Darling/Darling.Tests/TrendStaleStatsLiveTests.cs new file mode 100644 index 0000000000..72199e9e17 --- /dev/null +++ b/Darling/Darling.Tests/TrendStaleStatsLiveTests.cs @@ -0,0 +1,183 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5425: every trend read covers, run against the planner state that +/// made get_pg_io_trend answer "Exception while reading from stream" (its 30 s read deadline). +/// +/// One store holds many servers, so a table's statistics routinely describe other rows: an autoanalyze that +/// ran while the table held another server's data, or none. The planner then estimates one row for this server's +/// window, picks a nested loop for any join, and re-runs whatever sits on the inner side once per outer row. +/// Two reads had such a join over a windowed pass: get_pg_io_trend (the interval to the previous +/// collection; 64.7 s over 72 hours, 60 ms after the fix) and get_file_io_trend (every file I/O row joined +/// to the ranking of the series; 84.6 s over a week of the 609-series census shape, 2 s after the fix). Every +/// other read is a single pass over its table and measured under 1.5 s on the same state. +/// +/// This fixture builds that state on purpose, for all twelve tables the census seeds: statistics taken over +/// another server's rows from well before the window, then those rows deleted, then the real server's rows +/// planted. Autovacuum is switched off on the tables for the duration (best effort: a hypertable's chunks keep +/// their own setting), so an autoanalyze cannot heal the state between planting and reading and turn this into a +/// test of nothing. The reads go through the tools, with the real 30 s deadline: a read that goes quadratic is +/// cancelled at the deadline and the tool answers an error envelope, which fails the status check below. +/// +[Collection("live-postgres")] +public sealed class TrendStaleStatsLiveTests +{ + private const string ServerName = "trend-stale-stats"; + private const string OtherServerName = "trend-stale-stats-other"; + private static readonly int ServerId = ServerIdHelper.GetDeterministicHashCode(ServerName); + private static readonly int OtherServerId = ServerIdHelper.GetDeterministicHashCode(OtherServerName); + + /// The same shape TrendPayloadBudgetLiveTests.SeedAsync plants. + private const string PgQueryIdText = "987654321"; + + private static readonly string[] Tables = + { + "file_io_stats", "wait_stats", "query_stats", "pg_io_stats", "pg_database_stats", + "cpu_utilization_stats", "tempdb_stats", "memory_stats", "memory_grant_stats", "perfmon_stats", "pg_statement_stats", + "pg_cpu_utilization", + }; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + [Fact] + public async Task EveryTrendRead_AnswersWithinTheDeadline_WhenThePlannersRowEstimatesAreStale() + { + var cs = ConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(cs), "Set DARLING_TEST_PG to a Postgres connection string to run the live stale-statistics trend reads."); + + var ct = TestContext.Current.CancellationToken; + using var connection = new NpgsqlConnection(cs); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await DeleteRowsAsync(connection, ct); + await using var postgres = NpgsqlDataSource.Create(cs!); + + var bodySucceeded = false; + try + { + await SetAutovacuumAsync(connection, "autovacuum_enabled = false", ct); + + var now = DateTime.UtcNow; + var end = new DateTime(now.Year, now.Month, now.Day, now.Hour, now.Minute, 0).AddMinutes(-1); + + /* Statistics over ANOTHER server's rows, from well before this window, then those rows gone: the + planner's picture of every table is now "nothing near this server_id or these times". */ + await TrendPayloadBudgetLiveTests.SeedAsync(connection, end.AddDays(-40), ct, OtherServerId, OtherServerName); + foreach (var table in Tables) + { + using var analyze = new NpgsqlCommand($"ANALYZE {table}", connection) { CommandTimeout = 300 }; + await analyze.ExecuteNonQueryAsync(ct); + } + + await DeleteRowsAsync(connection, ct); + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); + await TrendPayloadBudgetLiveTests.SeedAsync(connection, end, ct, ServerId, ServerName); + + var output = TestContext.Current.TestOutputHelper; + var failures = new List(); + foreach (var hours in new[] { 72, 168 }) + { + var reads = new (string Tool, string PointsKey, Func> Run)[] + { + ("get_file_io_trend", "trend", () => DarlingMcpTrendTools.GetFileIoTrend(postgres, ServerName, hours)), + ("get_lock_wait_trend", "trend", () => DarlingMcpBlockingTools.GetLockWaitTrend(postgres, ServerName, hours)), + ("get_pg_io_trend", "points", () => DarlingMcpPgTrendTools.GetPgIoTrend(postgres, ServerName, hours_back: hours)), + ("get_pg_database_trend", "points", () => DarlingMcpPgTrendTools.GetPgDatabaseTrend(postgres, ServerName, hours_back: hours)), + ("get_pg_cpu_utilization", "samples", () => DarlingMcpPgCpuUtilizationTools.GetPgCpuUtilization(postgres, ServerName, hours_back: hours)), + ("get_wait_trend", "trend", () => DarlingMcpDataTools.GetWaitTrend(postgres, "LCK_M_S", ServerName, hours)), + ("get_cpu_utilization", "samples", () => DarlingMcpDataTools.GetCpuUtilization(postgres, ServerName, hours)), + ("get_tempdb_trend", "trend", () => DarlingMcpDataTools.GetTempDbTrend(postgres, ServerName, hours)), + ("get_memory_trend", "trend", () => DarlingMcpTrendTools.GetMemoryTrend(postgres, ServerName, hours)), + ("get_perfmon_trend", "trend", () => DarlingMcpTrendTools.GetPerfmonTrend(postgres, "Batch Requests/sec", ServerName, hours)), + ("get_pg_query_duration_trend", "points", () => DarlingMcpPgTrendTools.GetPgQueryDurationTrend(postgres, ServerName, PgQueryIdText, hours)), + /* The raw query-stats tier holds four days: get_query_duration_trend is rostered to 72 h. */ + ("get_query_duration_trend", "trend", () => DarlingMcpTrendTools.GetQueryDurationTrend(postgres, ServerName, hours_back: Math.Min(hours, 72))), + }; + + foreach (var (tool, pointsKey, run) in reads) + { + var clock = Stopwatch.StartNew(); + var answer = await run(); + clock.Stop(); + output?.WriteLine($"{tool} {hours}h: {clock.ElapsedMilliseconds} ms"); + + var root = JsonDocument.Parse(answer).RootElement; + var status = root.TryGetProperty("status", out var s) ? s.GetString() : null; + if (status is not (null or "io_trend" or "database_trend" or "query_duration_trend")) + { + failures.Add($"{tool} over {hours}h answered {status} after {clock.ElapsedMilliseconds} ms: {answer[..Math.Min(answer.Length, 300)]}"); + } + else if (root.GetProperty(pointsKey).GetArrayLength() == 0) + { + failures.Add($"{tool} over {hours}h returned no points"); + } + } + } + + Assert.True(failures.Count == 0, string.Join(Environment.NewLine, failures)); + + /* #5425 first pass: the interval join of get_pg_io_trend, at the width and window it failed at. + 72 hours at 30-minute buckets is 144 buckets; the first snapshot has nothing to difference against + and the last bucket may be partial. */ + var points = await DarlingPgTrendReader.GetIoTrendAsync( + postgres, ServerId, "client backend", "normal", end.AddHours(-72), end, 30, ct); + Assert.InRange(points.Count, 140, 146); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(cs!, bodySucceeded, async (cleanup, token) => + { + await DeleteRowsAsync(cleanup, token); + await SetAutovacuumAsync(cleanup, null, token); + }); + } + } + + /// Turns autovacuum off (or, with null, back to the table's default) on every table the + /// census seeds. + private static async Task SetAutovacuumAsync(NpgsqlConnection connection, string? setting, CancellationToken ct) + { + foreach (var table in Tables) + { + var sql = setting is null ? $"ALTER TABLE {table} RESET (autovacuum_enabled)" : $"ALTER TABLE {table} SET ({setting})"; + using var alter = new NpgsqlCommand(sql, connection) { CommandTimeout = 60 }; + await alter.ExecuteNonQueryAsync(ct); + } + } + + private static async Task DeleteRowsAsync(NpgsqlConnection connection, CancellationToken ct) + { + foreach (var id in new[] { ServerId, OtherServerId }) + { + foreach (var table in Tables.Append("servers")) + { + using var delete = new NpgsqlCommand($"DELETE FROM {table} WHERE server_id = $1", connection) { CommandTimeout = 300 }; + delete.Parameters.AddWithValue(id); + await delete.ExecuteNonQueryAsync(ct); + } + } + } +} diff --git a/Darling/Darling.Tests/TsqlConventionGuardTests.cs b/Darling/Darling.Tests/TsqlConventionGuardTests.cs index 3c5d132d26..186aaa3b87 100644 --- a/Darling/Darling.Tests/TsqlConventionGuardTests.cs +++ b/Darling/Darling.Tests/TsqlConventionGuardTests.cs @@ -94,6 +94,8 @@ namespace Darling.Tests; /// which is how the gap was found rather than assumed. The injected path has to EXIST — that guard drops /// references which do not resolve on disk, so a made-up filename is a red-proof that proves nothing. /// +[Trait("Stage", "Guard")] +[Trait("Reads", "Lite")] public sealed class TsqlConventionGuardTests { /* Rule identifiers. Covered ones are what Findings can emit; uncovered ones exist so the disposition map @@ -1457,6 +1459,19 @@ of that kind here. */ "Darling/PerformanceMonitor.Darling.Service/Hosting/DarlingWebFailureLog.cs IsStatementTimeout", "Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs ToSettings", "Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs IsConfigured", + /* #5452: two expression-bodied members that open with a property pattern (`is { IsConfiguration: true } found`, + `role is { } key`) before a ternary, the shape of ListLongQueryTraceDatabasesAsync below. The walk's brace match + closes each range at the pattern's own closing brace, stranding the ternary's arms (`found : null`, + `key.CredentialScope : string.Empty`). Neither arm holds a string literal, so no census reads a site of that + kind here; the stranded text is not T-SQL and not a tempdb label. */ + "Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs AwsRoleConfigurationFault", + "Darling/PerformanceMonitor.Darling.Service/Targets/AwsRoleKey.cs ScopeOf", + /* The long-query trace's database listing: an expression-bodied member whose body opens with a property + pattern (`LongQueryTraceListOverrideForTests is { } listOverride`) before its ternary. The walk's brace + match closes the range at the pattern's own closing brace, stranding the ternary's arms, which are calls + to the test hook and to GetAzureDatabaseListAsync. They hold no string literal, so no census reads a site + of that kind here. The Lite twin (RemoteCollectorService.LongQueryCompletions.cs, below) has the same shape. */ + "Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs ListLongQueryTraceDatabasesAsync", /* #4253: an expression-bodied member whose body opens with a property pattern (`marker is { State: ... }`) before the rest of the expression. The walk's brace match closes the range at the pattern's own closing brace, stranding the trailing `&& !cancellationToken. @@ -1467,6 +1482,12 @@ census reads a site of that kind here. */ /* #4214: an expression-bodied one-liner, same shape as the pairs elsewhere in this list — it strands only its own ternary, not T-SQL or a tempdb label, so no census reads a site of that kind here. */ "Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs ComputeEffectiveMemoryLimitBytes", + /* #5329: ARRIVED, the walk now stops short here. IoHourlyCoversWindow is an expression-bodied member + whose body opens with a property pattern (`coverage.FloorOf(ioView) is { } floor && floor <= startUtc`), + the same shape as the property-pattern members above. The brace match closes the range at the + pattern's own `{ }`, stranding `floor <= startUtc`. That is a comparison of two locals: no string + literal, no T-SQL and no tempdb label, so no census reads a site of that kind there. */ + "Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.RollupAvailability.cs IoHourlyCoversWindow", "Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs OutputFinding", "Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStallProbeReader.cs TriggerMbPerSecond", "Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingStallProbeReader.cs TerminalSilenceMs", @@ -1475,6 +1496,12 @@ census reads a site of that kind here. */ Neither is T-SQL and neither is a tempdb label, so no census reads a site of that kind here. */ "Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreMetricsTools.cs Stamp", "Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpStoreMetricsTools.cs Window", + /* #4938: get_collection_health's compact row is an expression-bodied `runTime is null ? new { … } : new { … }`, + so the walk stops at the first anonymous object's closing brace and strands the second shape. What falls + outside the range is that shape's own KEY NAMES and the "o" round-trip format on last_success: not T-SQL, not + a tempdb label, and read by the payload censuses from the serialized payload and the tool's description rather + than from a member-scoped literal sweep. The Lite twin (McpHealthTools.cs, below) has the same shape. */ + "Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpDataTools.cs CompactCollectionHealthRow", /* #3691 line 70: audit_config's PostgreSQL projection row serializes through an expression-bodied `ObjectName is null ? new { … } : new { … }` — the two anonymous-object initializers are where the walk stops, the same shape as the property-pattern derivations below. What falls outside the range @@ -1496,7 +1523,6 @@ Lite twin (LocalDataService.FinOps.IndexObjects.cs, below) derives the same four "Darling/PerformanceMonitor.Darling.Service/Targets/SqlServerTargetProvider.cs WithDatabase", "Darling/PerformanceMonitor.Darling.Viewer/MainWindow.ServerManagement.cs SelectedTabCollectorScope", "Darling/PerformanceMonitor.Darling.Viewer/ManageServersWindow.xaml.cs LastCollectedDisplay", - "Darling/PerformanceMonitor.Darling.Viewer/RecommendationsViewModel.cs HasStructuredFixAction", "Darling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml.cs BuildViewerPreferences", "Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Blocking.cs EventTimeLocal", "Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Deadlock.cs DeadlockTimeLocal", @@ -1507,6 +1533,9 @@ Lite twin (LocalDataService.FinOps.IndexObjects.cs, below) derives the same four "Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.PlanCorrection.cs Local", "Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.SystemEvents.cs Local", "Darling/PerformanceMonitor.Darling.Viewer/ViewerPostgresDisplay.cs Timestamp", + /* #4938: the Lite twin of DarlingMcpDataTools' CompactCollectionHealthRow above: the same + `runTime is null ? new { … } : new { … }` shape, stranding the same key names and "o" format. */ + "Lite/Mcp/McpHealthTools.cs CompactCollectionHealthRow", "Lite/Services/LocalDataService.CollectionHealth.cs OutputFinding", /* #4917: an expression-bodied property whose body opens with a property pattern (`Newest is { } newest`) before the rest of the expression. The walk's brace match closes the @@ -1514,12 +1543,20 @@ Lite twin (LocalDataService.FinOps.IndexObjects.cs, below) derives the same four newest > TimeSpan.FromDays(ArchiveService.HotDataDays)`, an age comparison on the newest snapshot time, not T-SQL and not a tempdb label, so no census reads a site of that kind here. */ "Lite/Services/LocalDataService.DatabaseStates.cs IsStale", + /* #5562: an expression-bodied method whose body opens with a property pattern (`untilUtc is { } end ? … : …`), the + same shape as IsStale above. The brace match closes the range at the pattern's own `}`, stranding the two calls to + ReadAlertHistoryAsync. They hold no string literal and no T-SQL (the SQL is in ReadAlertHistoryAsync, which the scan + reads whole), so no census reads a site of that kind outside the range. */ + "Lite/Services/LocalDataService.AlertHistory.cs GetAlertHistoryWindowAsync", /* #3541 A12: the Lite twin of the four DarlingObjectStatsReader growth derivations above — the same `is { } b ? … : null` shape, the same absence of any string literal. */ "Lite/Services/LocalDataService.FinOps.IndexObjects.cs DailyGrowthRateMb", "Lite/Services/LocalDataService.FinOps.IndexObjects.cs Growth30dMb", "Lite/Services/LocalDataService.FinOps.IndexObjects.cs Growth7dMb", "Lite/Services/LocalDataService.FinOps.IndexObjects.cs GrowthOverAvailableHistoryMb", + /* The Lite twin of DarlingCollectorRunner's ListLongQueryTraceDatabasesAsync above: the same + `is { } listOverride ? … : …` shape, and the same absence of any string literal. */ + "Lite/Services/RemoteCollectorService.LongQueryCompletions.cs ListLongQueryTraceDatabasesAsync", /* #4766: the history rows' expression-bodied formatter, `naiveUtc is not { } instant ? "" : ...`. The property pattern's braces are where the walk stops. What it strands is the "yyyy-MM-dd HH:mm:ss" grid format and nothing else: not T-SQL and not a tempdb label, so no census reads a site of that kind here. */ @@ -1532,6 +1569,10 @@ Lite twin (LocalDataService.FinOps.IndexObjects.cs, below) derives the same four on dev's 291e15f8 run); the buffer drill-down's `Round` LEFT on the same run — #3799 reshaped it into a block the walk reads whole — so its #3804 line comes off here. */ "Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Plans.cs Ratio", + /* #5558: the secondary-replica rule's expression-bodied `IsFresh` (`snapshotUtc is { } at && at <= windowEndUtc && ...`), + the same expression-shaped member as the lines above. What its range strands is the pattern variable and the + comparison, no T-SQL, no tempdb label and no SQL literal a census reads. ARRIVED with #5558. */ + "PerformanceMonitor.Common/AgReplicaScope.cs IsFresh", ]; /* ───────────────────────── the resolver, pinned on arranged source ───────────────────────── */ diff --git a/Darling/Darling.Tests/TuningDropLockRecoveryLiveTests.cs b/Darling/Darling.Tests/TuningDropLockRecoveryLiveTests.cs index 809a6606a0..afee16b890 100644 --- a/Darling/Darling.Tests/TuningDropLockRecoveryLiveTests.cs +++ b/Darling/Darling.Tests/TuningDropLockRecoveryLiveTests.cs @@ -48,6 +48,9 @@ public async Task BlockedDrop_GivesUpAtItsTimeout_AndTheSweepRecoversForLaterSta await body.OpenAsync(ct); await PgMigrations.MigrateAsync(body, ct); + /* A start builds at most one large index, so tune the store fully first; the sweep below then has only the blocked drop to fail. */ + await TuningStartPasses.ConvergeAsync(body, ct); + /* The pre-#4247 shape of one of the five dropped indexes, recreated by hand: PgTableTuning no longer creates it, so a store still carrying it (a field box from before this PR) is exactly the case the guarded drop exists for. */ diff --git a/Darling/Darling.Tests/TuningHourlyCreateLiveTests.cs b/Darling/Darling.Tests/TuningHourlyCreateLiveTests.cs new file mode 100644 index 0000000000..3b157ea480 --- /dev/null +++ b/Darling/Darling.Tests/TuningHourlyCreateLiveTests.cs @@ -0,0 +1,199 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Diagnostics; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// The hourly tuning pass runs beside live collection, so a converged store must issue no CREATE INDEX at all +/// (IF NOT EXISTS takes its table locks before it looks), a missing index must still be built, and a +/// CREATE that is issued must be bounded by a lock_timeout. Own scratch database per test. +/// +/* #1776 own-store: each live fact mints its own scratch database through ScratchPostgres. */ +[Collection("timing")] +public sealed class TuningHourlyCreateLiveTests +{ + private static readonly string[] TunedIndexes = + { + "idx_query_stats_server_hash_time", + "idx_query_store_stats_server_db_query_plan_time", + PgTableTuning.ForcePlanFailuresIndexName, + PgTableTuning.LegacyRowIndexName, + PgTableTuning.QueryStatsRestartRowIndexName, + PgTableTuning.ProcedureStatsRestartRowIndexName, + "idx_store_metrics_kind_name_time", + }; + + /// + /// Deterministic method: a second connection holds ROW EXCLUSIVE (what a writer holds) on each tuned table, + /// which conflicts with the ShareLock a CREATE INDEX takes, even an IF NOT EXISTS one that would + /// no-op, but not with the lock the pass's ALTER TABLE statements take. The pass's session caps every wait + /// at 2 s, so any CREATE that reaches a locked table fails with 55P03 and is not counted as applied: a + /// pass that issues none counts every statement. + /// + [Fact] + public async Task AConvergedStore_IssuesNoCreateIndex_EvenWhileWritersHoldTheTables() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the hourly tuning live test."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var body = new NpgsqlConnection(scratch.ConnectionString); + await body.OpenAsync(ct); + await PgMigrations.MigrateAsync(body, ct); + await TuningStartPasses.ConvergeAsync(body, ct); + foreach (var name in TunedIndexes) + { + Assert.True(await IndexExistsAsync(body, name, ct), name + " should exist after the first pass"); + } + + await using var holder = new NpgsqlConnection(scratch.ConnectionString); + await holder.OpenAsync(ct); + await Exec(holder, "BEGIN", ct); + try + { + await Exec(holder, "LOCK TABLE collect.query_stats, collect.procedure_stats, collect.query_store_stats, collect.store_metrics IN ROW EXCLUSIVE MODE", ct); + await Exec(body, "SET lock_timeout = '2s'", ct); + + var applied = await PgTableTuning.ApplyAsync(body, NullLogger.Instance, ct); + + Assert.Equal(PgTableTuning.Statements.Count, applied); + } + finally + { + await Exec(holder, "ROLLBACK", CancellationToken.None); + } + } + + [Fact] + public async Task AMissingIndex_IsStillBuilt() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the hourly tuning live test."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var body = new NpgsqlConnection(scratch.ConnectionString); + await body.OpenAsync(ct); + await PgMigrations.MigrateAsync(body, ct); + await TuningStartPasses.ConvergeAsync(body, ct); + + await Exec(body, "DROP INDEX collect.idx_store_metrics_kind_name_time", ct); + Assert.False(await IndexExistsAsync(body, "idx_store_metrics_kind_name_time", ct)); + + await PgTableTuning.ApplyAsync(body, NullLogger.Instance, ct); + Assert.True(await IndexExistsAsync(body, "idx_store_metrics_kind_name_time", ct)); + } + + [Fact] + public async Task OnTheHourlyPass_AMissingIndexIsNotBuilt_AndOneWarningNamesIt() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the hourly tuning live test."); + + var ct = TestContext.Current.CancellationToken; + PgTableTuning.ResetHourlyMissingWarnings(); + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var body = new NpgsqlConnection(scratch.ConnectionString); + await body.OpenAsync(ct); + await PgMigrations.MigrateAsync(body, ct); + await TuningStartPasses.ConvergeAsync(body, ct); + + await Exec(body, "DROP INDEX collect.idx_store_metrics_kind_name_time", ct); + var logger = new CapturingTestLogger(); + + var first = await PgTableTuning.ApplyAsync(body, logger, hourly: true, ct); + var second = await PgTableTuning.ApplyAsync(body, logger, hourly: true, ct); + + Assert.False(await IndexExistsAsync(body, "idx_store_metrics_kind_name_time", ct), + "the hourly pass must not build a missing index"); + Assert.Equal(PgTableTuning.Statements.Count - 1, first); + Assert.Equal(PgTableTuning.Statements.Count - 1, second); + var warnings = logger.Lines.Where(l => l.StartsWith("Warning:", StringComparison.Ordinal) + && l.Contains("idx_store_metrics_kind_name_time", StringComparison.Ordinal) + && l.Contains("collect.store_metrics", StringComparison.Ordinal)).ToList(); + Assert.True(warnings.Count == 1, "expected exactly one warning naming the index and its table: " + logger.Joined); + Assert.Equal(1, logger.CountAtLevel(LogLevel.Warning)); + } + + [Fact] + public async Task AStartPathCreate_ThatFindsTheTableBusy_GivesUpAndTheSweepRecovers() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the lock-busy create live test."); + + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var body = new NpgsqlConnection(scratch.ConnectionString); + await body.OpenAsync(ct); + await PgMigrations.MigrateAsync(body, ct); + await TuningStartPasses.ConvergeAsync(body, ct); + await Exec(body, "DROP INDEX collect.idx_query_stats_server_hash_time", ct); + + await using var holder = new NpgsqlConnection(scratch.ConnectionString); + await holder.OpenAsync(ct); + await Exec(holder, "BEGIN", ct); + try + { + await Exec(holder, "LOCK TABLE collect.query_stats IN ROW EXCLUSIVE MODE", ct); + var logger = new CapturingTestLogger(); + var clock = Stopwatch.StartNew(); + var applied = await PgTableTuning.ApplyAsync(body, logger, ct); + clock.Stop(); + + Assert.True(clock.Elapsed < TimeSpan.FromSeconds(10), $"the busy create took {clock.Elapsed.TotalSeconds:F1}s"); + Assert.Equal(PgTableTuning.Statements.Count - 1, applied); + await using var observer = new NpgsqlConnection(scratch.ConnectionString); + await observer.OpenAsync(ct); + Assert.False(await IndexExistsAsync(observer, "idx_query_stats_server_hash_time", ct)); + Assert.Single(logger.Lines, l => l.StartsWith("Information:", StringComparison.Ordinal) + && l.Contains("idx_query_stats_server_hash_time", StringComparison.Ordinal) + && l.Contains("table was busy", StringComparison.Ordinal)); + } + finally + { + await Exec(holder, "ROLLBACK", CancellationToken.None); + } + } + + [Fact] + public void ACreateThatIsStillIssued_RunsUnderASetLocalLockTimeout() + { + var src = RepoFile.ReadRepoFileLf("Darling", "PerformanceMonitor.Darling.Storage", "PgTableTuning.cs"); + Assert.Contains("GuardedCreate(statement)", src, StringComparison.Ordinal); + Assert.Contains("\"BEGIN; SET LOCAL lock_timeout = '\" + CreateLockTimeoutSeconds + \"'; \" + statement + \"; COMMIT;\"", src, StringComparison.Ordinal); + } + + private static async Task Exec(NpgsqlConnection c, string sql, CancellationToken ct) + { + await using var cmd = new NpgsqlCommand(sql, c); + await cmd.ExecuteNonQueryAsync(ct); + } + + private static async Task IndexExistsAsync(NpgsqlConnection c, string name, CancellationToken ct) + { + await using var cmd = new NpgsqlCommand("SELECT 1 FROM pg_indexes WHERE schemaname = 'collect' AND indexname = $1", c); + cmd.Parameters.AddWithValue(name); + return await cmd.ExecuteScalarAsync(ct) is not null; + } +} diff --git a/Darling/Darling.Tests/TuningIndexReaderPairingTests.cs b/Darling/Darling.Tests/TuningIndexReaderPairingTests.cs index 654ffeeff6..c2c88f09a8 100644 --- a/Darling/Darling.Tests/TuningIndexReaderPairingTests.cs +++ b/Darling/Darling.Tests/TuningIndexReaderPairingTests.cs @@ -6,6 +6,7 @@ * Licensed under the MIT License. See LICENSE file in the project root for full license information. */ +using System; using System.Collections.Generic; using System.Linq; using System.Text.RegularExpressions; @@ -34,8 +35,11 @@ public sealed class TuningIndexReaderPairingTests new Dictionary { ["idx_query_stats_server_hash_time"] = ("server_id", DarlingStoredPlanReader.QueryStatsPlanXmlByHashSql), - ["idx_query_store_stats_server_db_query_plan_time"] = ("server_id", DarlingStoredPlanReader.QueryStorePlanTextSql), + ["idx_query_store_stats_server_db_query_plan_time"] = ("server_id", DarlingStoredPlanReader.QueryStorePlanCandidatesSql), ["idx_query_store_stats_server_time_forcing"] = ("server_id", DarlingAlertReadAdapter.ForcePlanFailuresSql), + [PgTableTuning.LegacyRowIndexName] = ("server_id", QueryStoreIntervalWide.HasLegacyRowSql), + [PgTableTuning.QueryStatsRestartRowIndexName] = ("collection_time", IntervalRollupRestartRows.QueryStatsRestartRowsSql), + [PgTableTuning.ProcedureStatsRestartRowIndexName] = ("collection_time", IntervalRollupRestartRows.ProcedureStatsRestartRowsSql), ["idx_store_metrics_kind_name_time"] = ("object_kind", DarlingStoreMetricsReader.StoreMetricsLatestSql), }; @@ -58,14 +62,40 @@ public sealed class TuningIndexReaderPairingTests } } + /// The index is partial, so the read must carry the same predicate or the planner cannot use it. + [Fact] + public void TheLegacyRowRead_CarriesThePartialIndexPredicate_BesideItsServerAndTimeBounds() + { + var sql = QueryStoreIntervalWide.HasLegacyRowSql; + Assert.Contains("interval_start_time_utc IS NULL", sql, StringComparison.Ordinal); + Assert.Contains("server_id = $1", sql, StringComparison.Ordinal); + Assert.Contains("collection_time >= $2", sql, StringComparison.Ordinal); + Assert.Contains("collection_time <= $3", sql, StringComparison.Ordinal); + } + + /// Both restart-row indexes are partial, so each reader must carry the predicate and the half-open time bounds. + [Fact] + public void TheRestartRowReads_CarryThePartialIndexPredicate_BesideTheirTimeBounds() + { + foreach (var sql in new[] { IntervalRollupRestartRows.QueryStatsRestartRowsSql, IntervalRollupRestartRows.ProcedureStatsRestartRowsSql }) + { + Assert.Contains("sample_interval_seconds = 0", sql, StringComparison.Ordinal); + Assert.Contains("collection_time >= $1", sql, StringComparison.Ordinal); + Assert.Contains("collection_time < $2", sql, StringComparison.Ordinal); + } + + Assert.Contains("FROM collect.query_stats", IntervalRollupRestartRows.QueryStatsRestartRowsSql, StringComparison.Ordinal); + Assert.Contains("FROM collect.procedure_stats", IntervalRollupRestartRows.ProcedureStatsRestartRowsSql, StringComparison.Ordinal); + } + [Fact] public void EveryCreatedIndex_HasANamedReader_ThatReferencesItsLeadingColumn() { var created = CreatedIndexes().ToList(); - /* Sanity: this must actually see the four indexes the list ships today (#4247 dropped the other - five), or the regex/list drifted and the pairing check below would be vacuously true. */ - Assert.Equal(4, created.Count); + /* Sanity: this must actually see the seven indexes the list ships today (#4247 dropped five others), + or the regex/list drifted and the pairing check below would be vacuously true. */ + Assert.Equal(7, created.Count); foreach (var (indexName, leadingColumn) in created) { diff --git a/Darling/Darling.Tests/TuningReaderBudgetLiveTests.cs b/Darling/Darling.Tests/TuningReaderBudgetLiveTests.cs index 3a0e3fb23b..ff84e94d2f 100644 --- a/Darling/Darling.Tests/TuningReaderBudgetLiveTests.cs +++ b/Darling/Darling.Tests/TuningReaderBudgetLiveTests.cs @@ -103,6 +103,7 @@ public async Task NamedReaders_StayUnderHalfTheirDeadline_OnceTheFiveIndexesAreD await connection.OpenAsync(ct); await PgMigrations.MigrateAsync(connection, ct); await DeleteRowsAsync(connection, ct); + await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct); var bodySucceeded = false; try @@ -122,7 +123,7 @@ public async Task NamedReaders_StayUnderHalfTheirDeadline_OnceTheFiveIndexesAreD $"{indexName} should be gone after PgTableTuning.ApplyAsync — #4247 dropped it."); } - var end = new DateTime(DateTime.UtcNow.Ticks - (DateTime.UtcNow.Ticks % TimeSpan.TicksPerSecond), DateTimeKind.Utc); + var end = new DateTime(DateTime.UtcNow.Ticks / TimeSpan.TicksPerSecond * TimeSpan.TicksPerSecond, DateTimeKind.Utc); var currentStart = end.AddHours(-1); var baselineStart = currentStart.AddHours(-1); var collectionTime = currentStart.AddMinutes(10); @@ -306,7 +307,8 @@ private static async Task DeleteRowsAsync(NpgsqlConnection connection, Cancellat await using var cleanup = new NpgsqlCommand( $"DELETE FROM query_stats WHERE server_id = {ServerId}; " + $"DELETE FROM procedure_stats WHERE server_id = {ServerId}; " + - $"DELETE FROM query_store_stats WHERE server_id = {ServerId};", connection); + $"DELETE FROM query_store_stats WHERE server_id = {ServerId}; " + + $"DELETE FROM servers WHERE server_id = {ServerId};", connection); await cleanup.ExecuteNonQueryAsync(ct); } } diff --git a/Darling/Darling.Tests/TuningStartPasses.cs b/Darling/Darling.Tests/TuningStartPasses.cs new file mode 100644 index 0000000000..b73e8f736b --- /dev/null +++ b/Darling/Darling.Tests/TuningStartPasses.cs @@ -0,0 +1,31 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using PerformanceMonitor.Darling.Storage; + +namespace Darling.Tests; + +/// +/// A start builds at most one of the large start-path indexes, so a store with several of them missing needs +/// several starts to be fully tuned. A test that needs a fully tuned store runs the start pass until it stops building. +/// +internal static class TuningStartPasses +{ + /// Runs the start-path pass once per large start-path index, which is enough to build every missing one. + internal static async Task ConvergeAsync(NpgsqlConnection connection, CancellationToken ct) + { + for (var pass = 0; pass < 4; pass++) + { + await PgTableTuning.ApplyAsync(connection, NullLogger.Instance, ct); + } + } +} diff --git a/Darling/Darling.Tests/TuningStartPathOneBuildTests.cs b/Darling/Darling.Tests/TuningStartPathOneBuildTests.cs new file mode 100644 index 0000000000..e63856bbf2 --- /dev/null +++ b/Darling/Darling.Tests/TuningStartPathOneBuildTests.cs @@ -0,0 +1,150 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Npgsql; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// A service start builds at most one of the large start-path indexes: once one build has been issued, the other +/// missing ones wait for the next start and the log names each. Also pins how a start-path build that runs out +/// of time is told apart from any other failure, and the text of the warning it gets. Own scratch database per live test. +/// +/* #1776 own-store: the live fact mints its own scratch database through ScratchPostgres. */ +public sealed class TuningStartPathOneBuildTests +{ + private static readonly string[] Deferred = + { + PgTableTuning.LegacyRowIndexName, + PgTableTuning.QueryStatsRestartRowIndexName, + PgTableTuning.ProcedureStatsRestartRowIndexName, + }; + + [Fact] + public async Task AStartBuildsOneMissingIndex_DefersTheRest_AndTheNextStartsBuildThem() + { + var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the one-build-per-start live test."); + + var ct = TestContext.Current.CancellationToken; + PgTableTuning.ResetHourlyMissingWarnings(); + await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + await using var body = new NpgsqlConnection(scratch.ConnectionString); + await body.OpenAsync(ct); + await PgMigrations.MigrateAsync(body, ct); + await TuningStartPasses.ConvergeAsync(body, ct); + foreach (var name in Deferred) + { + Assert.True(await IndexExistsAsync(body, name, ct), name + " should exist after the start passes"); + } + + /* The hourly pass never builds any of them, however many are missing. */ + await Exec(body, "DROP INDEX collect." + PgTableTuning.QueryStatsRestartRowIndexName, ct); + await Exec(body, "DROP INDEX collect." + PgTableTuning.ProcedureStatsRestartRowIndexName, ct); + await PgTableTuning.ApplyAsync(body, new CapturingTestLogger(), hourly: true, ct); + Assert.False(await IndexExistsAsync(body, PgTableTuning.QueryStatsRestartRowIndexName, ct)); + Assert.False(await IndexExistsAsync(body, PgTableTuning.ProcedureStatsRestartRowIndexName, ct)); + + /* Start 1: builds exactly the first missing one in list order and defers the other, by name. */ + var start1 = new CapturingTestLogger(); + await PgTableTuning.ApplyAsync(body, start1, ct); + Assert.True(await IndexExistsAsync(body, PgTableTuning.QueryStatsRestartRowIndexName, ct), start1.Joined); + Assert.False(await IndexExistsAsync(body, PgTableTuning.ProcedureStatsRestartRowIndexName, ct), start1.Joined); + Assert.Single(start1.Lines, l => l.StartsWith("Information:", StringComparison.Ordinal) + && l.Contains(PgTableTuning.ProcedureStatsRestartRowIndexName, StringComparison.Ordinal) + && l.Contains("built at the next start", StringComparison.Ordinal)); + Assert.DoesNotContain(start1.Lines, l => l.Contains(PgTableTuning.QueryStatsRestartRowIndexName, StringComparison.Ordinal) + && l.Contains("built at the next start", StringComparison.Ordinal)); + + /* Start 2: builds the other. */ + var start2 = new CapturingTestLogger(); + await PgTableTuning.ApplyAsync(body, start2, ct); + Assert.True(await IndexExistsAsync(body, PgTableTuning.ProcedureStatsRestartRowIndexName, ct), start2.Joined); + Assert.DoesNotContain(start2.Lines, l => l.Contains("built at the next start", StringComparison.Ordinal)); + + /* Start 3: nothing is missing, nothing is built or deferred, and no index OID changes. */ + var oids = await OidsAsync(body, ct); + var start3 = new CapturingTestLogger(); + await PgTableTuning.ApplyAsync(body, start3, ct); + Assert.Equal(oids, await OidsAsync(body, ct)); + Assert.DoesNotContain(start3.Lines, l => l.Contains("built at the next start", StringComparison.Ordinal)); + Assert.Equal(0, start3.CountAtLevel(LogLevel.Warning)); + } + + /// The first-missing index in list order is the earliest start-path index, so the legacy-row index goes before the restart-row ones. + [Fact] + public void TheLegacyRowIndex_IsListedBeforeTheRestartRowIndexes_AndAllThreeShareTheBudget() + { + var order = PgTableTuning.Statements.Select((s, i) => (s, i)).ToList(); + int At(string name) => order.First(o => o.s.Contains(name + " ON", StringComparison.Ordinal)).i; + Assert.True(At(PgTableTuning.LegacyRowIndexName) < At(PgTableTuning.QueryStatsRestartRowIndexName)); + Assert.True(At(PgTableTuning.QueryStatsRestartRowIndexName) < At(PgTableTuning.ProcedureStatsRestartRowIndexName)); + foreach (var name in Deferred) + { + Assert.True(PgTableTuning.IsOneBuildPerStart(name), name); + } + } + + [Fact] + public void ABuildThatRanOutOfTime_IsToldApartFromAnyOtherFailure() + { + Assert.True(PgTableTuning.IsBuildTimeout(new NpgsqlException("Exception while reading from stream", new TimeoutException("Timeout during reading attempt")))); + Assert.True(PgTableTuning.IsBuildTimeout(new PostgresException("canceling statement due to statement timeout", "ERROR", "ERROR", PostgresErrorCodes.QueryCanceled))); + Assert.False(PgTableTuning.IsBuildTimeout(new PostgresException("relation does not exist", "ERROR", "ERROR", PostgresErrorCodes.UndefinedTable))); + Assert.False(PgTableTuning.IsBuildTimeout(new NpgsqlException("connection reset", new System.IO.IOException("reset")))); + Assert.False(PgTableTuning.IsBuildTimeout(new InvalidOperationException("boom"))); + } + + [Fact] + public void TheTimeoutWarning_NamesTheIndexTheTableTheElapsedTimeAndTheRetry() + { + var message = PgTableTuning.BuildTimeoutMessage("idx_demo", "collect.query_stats", 301.4); + Assert.Equal( + "Index collect.idx_demo on collect.query_stats was not built: the build ran out of time after 301 s (limit 300 s) and was rolled back; it is retried at the next start, and each try pauses collection at start for up to 300 s", + message); + } + + [Fact] + public void TheStartPathCatch_UsesTheTimeoutWarning_OnlyForABuildThatTimedOut() + { + var src = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Storage", "PgTableTuning.cs").ReplaceLineEndings("\n"); + Assert.Contains("statement.StartsWith(CreateIndexPrefix, StringComparison.Ordinal) && IsBuildTimeout(ex)", src, StringComparison.Ordinal); + Assert.Contains("BuildTimeoutMessage(", src, StringComparison.Ordinal); + Assert.Contains("Composer performance-tuning statement failed", src, StringComparison.Ordinal); + } + + private static async Task OidsAsync(NpgsqlConnection c, CancellationToken ct) + { + await using var cmd = new NpgsqlCommand( + "SELECT string_agg(c.relname || ':' || c.oid::text, ',' ORDER BY c.relname) FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace " + + "WHERE n.nspname = 'collect' AND c.relname = ANY($1)", c); + cmd.Parameters.AddWithValue(Deferred); + return (string)(await cmd.ExecuteScalarAsync(ct))!; + } + + private static async Task Exec(NpgsqlConnection c, string sql, CancellationToken ct) + { + await using var cmd = new NpgsqlCommand(sql, c); + await cmd.ExecuteNonQueryAsync(ct); + } + + private static async Task IndexExistsAsync(NpgsqlConnection c, string name, CancellationToken ct) + { + await using var cmd = new NpgsqlCommand("SELECT 1 FROM pg_indexes WHERE schemaname = 'collect' AND indexname = $1", c); + cmd.Parameters.AddWithValue(name); + return await cmd.ExecuteScalarAsync(ct) is not null; + } +} diff --git a/Darling/Darling.Tests/UnpooledDropConnectionCensusTests.cs b/Darling/Darling.Tests/UnpooledDropConnectionCensusTests.cs new file mode 100644 index 0000000000..5ee56fd705 --- /dev/null +++ b/Darling/Darling.Tests/UnpooledDropConnectionCensusTests.cs @@ -0,0 +1,709 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text.RegularExpressions; +using Xunit; + +namespace Darling.Tests; + +/// +/// A test that runs DROP DATABASE does it on an unpooled connection, and quiesces the database's TimescaleDB +/// jobs first (#5549). +/// +/// The defect this closes. A PostgreSQL client backend crashed with 0xC0000005 in a TimescaleDB +/// run_job, after the same backend had run a scratch DROP DATABASE ... WITH (FORCE). The admin connection +/// string is the caller's DARLING_TEST_PG, so a pooled admin connection went back into the pool every live test +/// draws from, and the next test took the backend that had just run the drop. The ruling: a database drop and a job +/// run never share a backend. ScratchPostgres opens its own create, drop and sweep connections unpooled +/// (); this census holds every other test file to the same +/// rule. +/// +/// What this pins. For every PostgreSQL DROP DATABASE string literal in the test project (comments +/// are blanked first): if the enclosing method opens a connection itself (new NpgsqlConnection( or +/// LiveStoreCleanup.RunAsync(), that method must also hold the evidence of an unpooled connection +/// (UnpooledAdminConnectionString(, ExitDrainConnectionString( or Pooling = false). A method that +/// is handed its connection (a drop helper that takes the cleanup connection as a parameter) is held to the weaker +/// file-level rule: the file must hold that evidence somewhere. The file-level rule cannot see which connection the +/// caller passes, so a drop helper's callers are the thing a reviewer still checks; the method-level rule is what stops +/// the common shape, a new drop with its own pooled admin connection beside it. A file that is not about PostgreSQL +/// drops at all is named in with its reason. +/// +/// The second rule: quiesce before the drop. The same ruling says TimescaleDB jobs are quiesced before every +/// in-test drop, so a job worker is never running in a database while it is dropped. Every PostgreSQL DROP DATABASE +/// site (the same sites, under the same ) must sit in a member that calls +/// QuiesceTimescaleJobsAsync BEFORE the drop, plain or WITH (FORCE) (the older census in +/// sees only the FORCE form). A method that cannot be dropping a database +/// with jobs, because the test never created the extension in it and never ran a migration there, is named in +/// as File.cs::Method (or File.cs for the whole file) with its reason. +/// +/// How a drop's member is found. The census is a source scan with no parser package. A small brace walk over +/// the file (comments and string literals blanked, so a brace or a call inside text is never read) finds the innermost +/// method, constructor, local function, property or accessor body around the drop, at any indentation and with or without +/// an access modifier; for an expression-bodied member or a field it is the declaration up to its own ;. A region +/// never spans two members, so a quiesce in a neighbour cannot excuse a drop. The quiesce counts only as a real call +/// (not inside a string or a comment) and only when it comes before the drop in that member. +/// +[Trait("Stage", "Guard")] +public sealed class UnpooledDropConnectionCensusTests +{ + /// The files that contain the text DROP DATABASE in a string but never drop a PostgreSQL database on a pooled connection, each with why. + internal static readonly IReadOnlyDictionary Exceptions = new Dictionary(StringComparer.Ordinal) + { + ["DarlingCollectorRunnerTests.cs"] = "T-SQL against SQL Server (DROP DATABASE [name]); SQL Server has no backend pool shared with a TimescaleDB job.", + ["ProcedureStatsDeferredPlanFetchLiveTests.cs"] = "T-SQL against SQL Server (SINGLE_USER, then DROP DATABASE [name]).", + ["QueryStatsDeferredPlanFetchLiveTests.cs"] = "T-SQL against SQL Server (SINGLE_USER, then DROP DATABASE [name]).", + ["DropXeSessionsLiveTests.cs"] = "A SQL-injection session-name fixture: the text is data a validator must refuse, never run as a statement.", + ["DropXeSessionsPerInstallTests.cs"] = "A SQL-injection session-name fixture: the text is data a validator must refuse, never run as a statement.", + ["DropXeSessionsReadOnlyIntentTests.cs"] = "A SQL-injection session-name fixture: the text is data a validator must refuse, never run as a statement.", + ["DropXeSessionsVerbTests.cs"] = "A SQL-injection session-name fixture and a list of forbidden verbs: text a validator must refuse, never run as a statement.", + ["UnpooledDropConnectionCensusTests.cs"] = "This census: its own samples hold the drop text on purpose.", + ["ScratchPostgresQuiesceTests.cs"] = "The quiesce census: its samples hold the drop text on purpose, and the live test there drops through ScratchPostgres.", + }; + + /// + /// The PostgreSQL drop sites that skip QuiesceTimescaleJobsAsync, each as File.cs::Method (or File.cs + /// for the whole file) with why the dropped database cannot hold a TimescaleDB job. + /// + internal static readonly IReadOnlyDictionary QuiesceExceptions = new Dictionary(StringComparer.Ordinal) + { + ["ComposeStoreRolesLiveTests.cs::TheComposeStore_ProvisionsItsOwnRoles_AndTheRealHostsConnectAsViewerAndMcp_Gated"] = + "operator_app_3914 is a bare CREATE DATABASE from template1, made to stand for an operator's own database; no extension or migration ever runs in it.", + ["PgReadBinaryFileCapabilityLiveTests.cs::IsGrantedAsync_OnAWin1252Database_GrantsTheRoute_AndCachesItsEncoding"] = + "pm_test_enc1252 is a bare WIN1252 database made from template0, so it never holds the TimescaleDB extension.", + }; + + /// Matches case-insensitively: T-SQL and PostgreSQL both accept drop database. + private static readonly Regex DropStatement = new("DROP DATABASE", RegexOptions.Compiled | RegexOptions.IgnoreCase); + + /// A call to the quiesce helper, matched in code only (comments and string literals are blanked first). + private static readonly Regex QuiesceCall = new(@"\bQuiesceTimescaleJobsAsync\s*\(", RegexOptions.Compiled); + + /// + /// The member's name: the identifier right before the first ( of its declaration, after attribute lists are + /// removed, with an optional generic argument list between (Foo<T>(). + /// + private static readonly Regex MethodName = new(@"(\w+)\s*(?:<[^()]*>)?\s*\(", RegexOptions.Compiled); + + private static readonly Regex AttributeList = new(@"\[[^\]]*\]", RegexOptions.Compiled); + + private static readonly Regex Evidence = new( + @"UnpooledAdminConnectionString\(|ExitDrainConnectionString\(|Pooling\s*=\s*false", RegexOptions.Compiled); + + private static readonly Regex OwnConnection = new(@"new NpgsqlConnection\(|LiveStoreCleanup\.RunAsync\(", RegexOptions.Compiled); + + /// A type header: modifiers, then the keyword. A where T : class constraint on a method is not one. + private static readonly Regex TypeWord = new(@"^(?:\w+\s+)*?(?:class|struct|interface|record|enum|namespace|union)\b", RegexOptions.Compiled); + + private static readonly Regex NewOrDelegateWord = new(@"\b(?:new|delegate)\b", RegexOptions.Compiled); + + /// First words that open a block which is not a member: control flow and the like. + private static readonly HashSet ControlWords = new(StringComparer.Ordinal) + { + "if", "else", "for", "foreach", "while", "do", "switch", "using", "lock", "try", "catch", "finally", "fixed", + "unsafe", "checked", "unchecked", "case", "default", + }; + + /// First words that start a statement which a { continues (return x switch { ... }): not a member. + private static readonly HashSet StatementWords = new(StringComparer.Ordinal) { "return", "yield", "throw", "var" }; + + /// What a { ... } block is, which decides where a member (the region a drop is judged in) starts and ends. + private enum FrameKind + { + /// The file itself. + Root, + + /// A namespace, class, struct, interface, record or enum body. + Type, + + /// A method, constructor, local function, property or accessor body, at any indentation. + Member, + + /// A control-flow block (if, try, using, a bare block): the statement ends when it closes. + Control, + + /// A lambda, an initializer, an interpolation hole: part of the statement around it. + Other, + } + + private sealed class Frame + { + public Frame(FrameKind kind, int headerStart, int open, Frame? parent) + { + Kind = kind; + HeaderStart = headerStart; + Open = open; + StatementStart = open + 1; + Parent = parent; + } + + public FrameKind Kind { get; } + + public int HeaderStart { get; } + + public int Open { get; } + + public int Close { get; set; } = -1; + + /// Where the statement being read at this block's own level started: just after the last ;, or the last closed member, type or control block. + public int StatementStart { get; set; } + + /// Open parentheses not yet closed at this block's own level; a { inside them is a lambda or an initializer. + public int Parens { get; set; } + + public Frame? Parent { get; } + } + + /// + /// One DROP DATABASE line: where it is, the member it sits in (, comments blanked, + /// whole member), the member's name, and whether a real call to the quiesce helper comes before the drop in that member. + /// + internal sealed record DropSite(int Line, string MemberName, string MemberText, bool QuiescedBefore); + + /// + /// Every DROP DATABASE line in (case-insensitive, comments blanked first) with its + /// member. The member is the innermost method, constructor, local function, property or accessor body around the + /// line, at any indentation and with or without an access modifier; for an expression-bodied member or a field it + /// is the declaration up to its terminating ;. It is never wider than that, so a quiesce in a neighbouring + /// member can never excuse a drop. The quiesce call is looked for in code only (comments and string literals are + /// blanked), and only before the drop. + /// + internal static List FindDropSites(string source) + { + var blanked = ScratchPostgresQuiesceCensusTests.BlankComments(source); + var code = CSharpSourceWalker.StripCommentsAndStrings(source); + + var drops = new List<(int Offset, int Line)>(); + var lineStart = 0; + for (var line = 1; lineStart <= blanked.Length; line++) + { + var newline = blanked.IndexOf('\n', lineStart); + var lineEnd = newline < 0 ? blanked.Length : newline; + var found = DropStatement.Match(blanked, lineStart, lineEnd - lineStart); + if (found.Success) + { + drops.Add((found.Index, line)); + } + + lineStart = lineEnd + 1; + } + + var root = new Frame(FrameKind.Root, 0, -1, null); + var top = root; + var snapshots = new List<(Frame Inner, int ContainerStatementStart, int Offset, int Line)>(); + var next = 0; + for (var i = 0; i < code.Length; i++) + { + if (next < drops.Count && drops[next].Offset == i) + { + var container = top; + while (container.Kind is not (FrameKind.Type or FrameKind.Root)) + { + container = container.Parent!; + } + + snapshots.Add((top, container.StatementStart, i, drops[next].Line)); + next++; + } + + switch (code[i]) + { + case '(': + top.Parens++; + break; + case ')': + if (top.Parens > 0) + { + top.Parens--; + } + + break; + case ';': + if (top.Parens == 0) + { + top.StatementStart = i + 1; + } + + break; + case '{': + var header = code.Substring(top.StatementStart, i - top.StatementStart); + var kind = top.Parens > 0 ? FrameKind.Other : Classify(header); + var firstCode = top.StatementStart; + while (firstCode < i && char.IsWhiteSpace(code[firstCode])) + { + firstCode++; + } + + top = new Frame(kind, firstCode, i, top); + break; + case '}': + if (top.Parent is not null) + { + top.Close = i; + var closed = top; + top = top.Parent; + if (closed.Kind != FrameKind.Other) + { + top.StatementStart = i + 1; + } + } + + break; + } + } + + var sites = new List(); + foreach (var (inner, containerStatementStart, offset, line) in snapshots) + { + var member = inner; + while (member is not null && member.Kind != FrameKind.Member) + { + member = member.Parent; + } + + int start; + int end; + if (member is not null) + { + start = member.HeaderStart; + end = member.Close >= 0 ? member.Close : code.Length - 1; + } + else + { + /* An expression-bodied member or a field: the declaration, up to the first ; that is not inside a nested block. */ + start = containerStatementStart; + end = code.Length - 1; + var depth = 0; + for (var j = offset; j < code.Length; j++) + { + if (code[j] == '{') + { + depth++; + } + else if (code[j] == '}') + { + if (depth == 0) + { + end = j; + break; + } + + depth--; + } + else if (code[j] == ';' && depth == 0) + { + end = j; + break; + } + } + } + + start = Math.Min(start, offset); + end = Math.Max(end, offset); + var region = code.Substring(start, end - start + 1); + var declared = MethodName.Match(AttributeList.Replace(region, " ")); + sites.Add(new DropSite( + line, + declared.Success ? declared.Groups[1].Value : string.Empty, + blanked.Substring(start, end - start + 1), + QuiesceCall.IsMatch(code.Substring(start, offset - start)))); + } + + return sites; + } + + /// What the block opened by the { that follows is. + private static FrameKind Classify(string header) + { + var text = StripParentheses(AttributeList.Replace(header, " ")).Trim(); + if (text.StartsWith("await ", StringComparison.Ordinal)) + { + text = text[6..].TrimStart(); + } + + var first = Regex.Match(text, @"^\w+"); + if (text.Length == 0 || (first.Success && ControlWords.Contains(first.Value))) + { + return FrameKind.Control; + } + + if ((first.Success && StatementWords.Contains(first.Value)) + || text.Contains('=', StringComparison.Ordinal) + || NewOrDelegateWord.IsMatch(text)) + { + return FrameKind.Other; + } + + return TypeWord.IsMatch(text) ? FrameKind.Type : FrameKind.Member; + } + + /// The text with every parenthesised run (nested ones too) removed, parentheses included. + private static string StripParentheses(string text) + { + var sb = new System.Text.StringBuilder(text.Length); + var depth = 0; + foreach (var c in text) + { + if (c == '(') + { + depth++; + } + else if (c == ')') + { + if (depth > 0) + { + depth--; + } + } + else if (depth == 0) + { + sb.Append(c); + } + } + + return sb.ToString(); + } + + /// + /// Every drop site that breaks the rule, as file:line, and how many drop sites were looked at. + /// + internal static (int Sites, List Pooled) Scan(IEnumerable<(string Name, string Source)> files) + { + var sites = 0; + var pooled = new List(); + foreach (var (name, source) in files) + { + if (Exceptions.ContainsKey(Path.GetFileName(name))) + { + continue; + } + + var blanked = ScratchPostgresQuiesceCensusTests.BlankComments(source); + foreach (var site in FindDropSites(source)) + { + sites++; + var ok = OwnConnection.IsMatch(site.MemberText) ? Evidence.IsMatch(site.MemberText) : Evidence.IsMatch(blanked); + if (!ok) + { + pooled.Add($"{name}:{site.Line}"); + } + } + } + + return (sites, pooled); + } + + /// + /// Every PostgreSQL drop site that has no call to QuiesceTimescaleJobsAsync before it in its own member and is + /// not in , as file:line, and how many drop sites were looked at. + /// + internal static (int Sites, List Unquiesced) ScanForQuiesce(IEnumerable<(string Name, string Source)> files) + { + var sites = 0; + var unquiesced = new List(); + foreach (var (name, source) in files) + { + var file = Path.GetFileName(name); + if (Exceptions.ContainsKey(file)) + { + continue; + } + + foreach (var site in FindDropSites(source)) + { + sites++; + if (site.QuiescedBefore + || QuiesceExceptions.ContainsKey(file) + || (site.MemberName.Length > 0 && QuiesceExceptions.ContainsKey($"{file}::{site.MemberName}"))) + { + continue; + } + + unquiesced.Add($"{name}:{site.Line}"); + } + } + + return (sites, unquiesced); + } + + [Fact] + public void EveryPostgresDropInTheTestProject_RunsOnAnUnpooledConnection() + { + var directory = FindTestProjectDirectory(); + Assert.True(directory is not null, "could not find Darling/Darling.Tests from the test binary."); + + var files = Directory.EnumerateFiles(directory!, "*.cs", SearchOption.AllDirectories) + .Select(path => (Name: Path.GetRelativePath(directory!, path).Replace('\\', '/'), Path: path)) + .Where(found => !found.Name.Split('/').Any(segment => segment is "bin" or "obj")) + .Select(found => (found.Name, File.ReadAllText(found.Path))) + .ToList(); + + var (sites, pooled) = Scan(files); + + Assert.True(sites >= 9, $"the scan found {sites} PostgreSQL DROP DATABASE site(s); the project has at least 9, so the match found nothing."); + Assert.True(pooled.Count == 0, + "These DROP DATABASE statements run where no unpooled connection is in sight (#5549). A pooled connection returns the " + + "backend that ran the drop to the pool every live test draws from, and a TimescaleDB job run on it crashed. Open the " + + "connection with ScratchPostgres.UnpooledAdminConnectionString(...), or add the file to Exceptions with its reason: " + + string.Join(", ", pooled)); + } + + [Fact] + public void EveryPostgresDropInTheTestProject_QuiescesTheTimescaleJobsFirst() + { + var directory = FindTestProjectDirectory(); + Assert.True(directory is not null, "could not find Darling/Darling.Tests from the test binary."); + + var files = Directory.EnumerateFiles(directory!, "*.cs", SearchOption.AllDirectories) + .Select(path => (Name: Path.GetRelativePath(directory!, path).Replace('\\', '/'), Path: path)) + .Where(found => !found.Name.Split('/').Any(segment => segment is "bin" or "obj")) + .Select(found => (found.Name, File.ReadAllText(found.Path))) + .ToList(); + + var (sites, unquiesced) = ScanForQuiesce(files); + + Assert.True(sites >= 9, $"the scan found {sites} PostgreSQL DROP DATABASE site(s); the project has at least 9, so the match found nothing."); + Assert.True(unquiesced.Count == 0, + "These DROP DATABASE statements are not in a method that calls ScratchPostgres.QuiesceTimescaleJobsAsync (#5549). A drop " + + "while a TimescaleDB job worker runs in the database kills the worker, and a job run next to a drop crashed a backend. " + + "Call the helper before the drop, or add File.cs::Method to QuiesceExceptions with the reason the database holds no job: " + + string.Join(", ", unquiesced)); + } + + [Fact] + public void TheQuiesceExceptions_NameRealFiles_AndEachOneStillHoldsTheDropText() + { + var directory = FindTestProjectDirectory(); + Assert.True(directory is not null, "could not find Darling/Darling.Tests from the test binary."); + + foreach (var (key, reason) in QuiesceExceptions) + { + Assert.False(string.IsNullOrWhiteSpace(reason), $"{key} is allowed without a reason."); + var parts = key.Split("::"); + var path = Path.Combine(directory!, parts[0]); + Assert.True(File.Exists(path), $"{parts[0]} is in QuiesceExceptions but is not in the test project; remove the entry."); + Assert.False(Exceptions.ContainsKey(parts[0]), $"{parts[0]} is already skipped by Exceptions; remove the entry from QuiesceExceptions."); + var text = File.ReadAllText(path); + Assert.True(text.Contains("DROP DATABASE", StringComparison.OrdinalIgnoreCase), + $"{parts[0]} no longer holds DROP DATABASE; remove {key} from QuiesceExceptions."); + if (parts.Length > 1) + { + Assert.True(Regex.IsMatch(text, @"\b" + Regex.Escape(parts[1]) + @"\s*(?:<[^()]*>)?\s*\("), + $"{parts[0]} no longer declares {parts[1]}; remove {key} from QuiesceExceptions."); + Assert.True(MethodStillSkipsQuiesce(parts[1], text), + $"{parts[1]} in {parts[0]} no longer holds a DROP DATABASE that skips the quiesce; remove {key} from QuiesceExceptions."); + } + } + } + + /// True when in holds a drop with no quiesce call before it, so its allow-list entry is still needed. + private static bool MethodStillSkipsQuiesce(string method, string source) => + FindDropSites(source).Any(site => site.MemberName == method && !site.QuiescedBefore); + + [Fact] + public void ADropWithoutTheQuiesce_IsReported_AndOneWithItOrAnAllowListedOneIsNot() + { + const string quiesced = + "public class A\n{\n public async Task Go()\n {\n" + + " await ScratchPostgres.QuiesceTimescaleJobsAsync(cs, n);\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n}\", admin);\n }\n}\n"; + const string plainBare = + "public class A\n{\n public async Task Go()\n {\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n}\", admin);\n }\n}\n"; + const string forceBare = + "public class A\n{\n public async Task Go()\n {\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n} WITH (FORCE)\", admin);\n }\n}\n"; + /* A quiesce in another method of the file does not excuse this one. */ + const string quiesceElsewhere = + "public class A\n{\n public async Task Go()\n {\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n}\", admin);\n }\n\n" + + " public async Task Other()\n {\n await ScratchPostgres.QuiesceTimescaleJobsAsync(cs, n);\n }\n}\n"; + const string commentOnlyQuiesce = + "public class A\n{\n public async Task Go()\n {\n" + + " // QuiesceTimescaleJobsAsync(cs, n) would go here\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n}\", admin);\n }\n}\n"; + const string commentOnly = "public class A\n{\n // DROP DATABASE x\n /// DROP DATABASE y\n}\n"; + var allowListed = QuiesceExceptions.Keys.First(); + var allowListedFile = allowListed.Split("::")[0]; + var allowListedMethod = allowListed.Split("::")[1]; + var allowListedSource = + "public class A\n{\n public async Task " + allowListedMethod + "()\n {\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n}\", admin);\n }\n}\n"; + + Assert.Empty(ScanForQuiesce([("Q.cs", quiesced)]).Unquiesced); + Assert.Equal(["P.cs:5"], ScanForQuiesce([("P.cs", plainBare)]).Unquiesced); + Assert.Equal(["F.cs:5"], ScanForQuiesce([("F.cs", forceBare)]).Unquiesced); + Assert.Equal(["E.cs:5"], ScanForQuiesce([("E.cs", quiesceElsewhere)]).Unquiesced); + Assert.Equal(["M.cs:6"], ScanForQuiesce([("M.cs", commentOnlyQuiesce)]).Unquiesced); + Assert.Equal(0, ScanForQuiesce([("C.cs", commentOnly)]).Sites); + /* The allow-list names a file and a method: that method in that file is excused, the same method in another file is not. */ + Assert.Empty(ScanForQuiesce([(allowListedFile, allowListedSource)]).Unquiesced); + Assert.Equal(["Other.cs:5"], ScanForQuiesce([("Other.cs", allowListedSource)]).Unquiesced); + /* A SQL Server or fixture file is outside the rule altogether (Exceptions). */ + Assert.Equal(0, ScanForQuiesce([("DarlingCollectorRunnerTests.cs", plainBare)]).Sites); + } + + /// + /// The ways the quiesce rule once passed a drop it should have failed: a lowercase drop database, a quiesce call + /// that is only text (a string literal), a quiesce that comes after the drop, a member with no access modifier, a + /// local function at a deeper indent, and an expression-bodied neighbour that stretched a region over two members. + /// Each sample is a whole source file; the line number in each expectation is the drop line. + /// + [Fact] + public void TheQuiesceRule_CannotBePassedBy_ALowercaseDrop_AStringCall_ALateCall_OrANeighbouringMember() + { + const string lowercase = + "public class A\n{\n public async Task Go()\n {\n" + + " await using var drop = new NpgsqlCommand($\"drop database if exists {n}\", admin);\n }\n}\n"; + /* The string also holds a } so a scan that reads literals as code would close the member early. */ + const string quiesceInString = + "public class A\n{\n public async Task Go()\n {\n" + + " var s = \"await ScratchPostgres.QuiesceTimescaleJobsAsync(cs, n); }\";\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n}\", admin);\n }\n}\n"; + const string quiesceAfterTheDrop = + "public class A\n{\n public async Task Go()\n {\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n}\", admin);\n" + + " await ScratchPostgres.QuiesceTimescaleJobsAsync(cs, n);\n }\n}\n"; + /* The member above quiesces; the one with no access modifier does not, and one member must not excuse the other. */ + const string noModifier = + "public class A\n{\n public async Task Other()\n {\n" + + " await ScratchPostgres.QuiesceTimescaleJobsAsync(cs, n);\n }\n\n" + + " Task Go()\n {\n return Run($\"DROP DATABASE IF EXISTS {n}\");\n }\n}\n"; + /* The enclosing method quiesces, but the local function that drops does not. */ + const string localFunctionWithout = + "public class A\n{\n public async Task Go()\n {\n" + + " await ScratchPostgres.QuiesceTimescaleJobsAsync(cs, n);\n" + + " async Task Drop()\n {\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n}\", admin);\n }\n" + + " await Drop();\n }\n}\n"; + const string localFunctionWith = + "public class A\n{\n public async Task Go()\n {\n" + + " async Task Drop()\n {\n" + + " await ScratchPostgres.QuiesceTimescaleJobsAsync(cs, n);\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n}\", admin);\n }\n" + + " await Drop();\n }\n}\n"; + /* The expression-bodied drop ends at its own semicolon, not at the next member's closing brace. */ + const string expressionBodiedNeighbour = + "public class A\n{\n public Task Drop() => admin.RunAsync(\"DROP DATABASE IF EXISTS x\");\n\n" + + " public async Task Other()\n {\n await ScratchPostgres.QuiesceTimescaleJobsAsync(cs, n);\n }\n}\n"; + const string expressionBodiedQuiesced = + "public class A\n{\n public Task Drop() => ScratchPostgres.QuiesceTimescaleJobsAsync(cs, n)" + + ".ContinueWith(_ => admin.RunAsync(\"DROP DATABASE IF EXISTS x\"));\n}\n"; + /* The shape every LiveStoreCleanup drop has: the quiesce and the drop sit in one lambda inside the member. */ + const string insideLambda = + "public class A\n{\n public async Task Go()\n {\n" + + " await LiveStoreCleanup.RunAsync(cs, ok, async (cleanup, ct) =>\n {\n" + + " await ScratchPostgres.QuiesceTimescaleJobsAsync(cs, n);\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n}\", cleanup);\n });\n }\n}\n"; + + Assert.Equal(["L.cs:5"], ScanForQuiesce([("L.cs", lowercase)]).Unquiesced); + Assert.Equal(["L.cs:5"], Scan([("L.cs", lowercase)]).Pooled); + Assert.Equal(["S.cs:6"], ScanForQuiesce([("S.cs", quiesceInString)]).Unquiesced); + Assert.Equal(["T.cs:5"], ScanForQuiesce([("T.cs", quiesceAfterTheDrop)]).Unquiesced); + Assert.Equal(["N.cs:10"], ScanForQuiesce([("N.cs", noModifier)]).Unquiesced); + Assert.Equal(["D.cs:8"], ScanForQuiesce([("D.cs", localFunctionWithout)]).Unquiesced); + Assert.Empty(ScanForQuiesce([("W.cs", localFunctionWith)]).Unquiesced); + Assert.Equal(["X.cs:3"], ScanForQuiesce([("X.cs", expressionBodiedNeighbour)]).Unquiesced); + Assert.Empty(ScanForQuiesce([("Y.cs", expressionBodiedQuiesced)]).Unquiesced); + Assert.Empty(ScanForQuiesce([("Z.cs", insideLambda)]).Unquiesced); + } + + /// A generic method's name is found (Name<T>(), so an allow-list entry for it matches and its still-needed check works. + [Fact] + public void AGenericMethod_OnTheAllowList_IsExcused_AndFoundByTheStaleEntryCheck() + { + var allowListed = QuiesceExceptions.Keys.First(); + var allowListedFile = allowListed.Split("::")[0]; + var allowListedMethod = allowListed.Split("::")[1]; + var generic = + "public class A\n{\n public async Task " + allowListedMethod + "() where T : class\n {\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n}\", admin);\n }\n}\n"; + + Assert.Empty(ScanForQuiesce([(allowListedFile, generic)]).Unquiesced); + Assert.Equal(["Other.cs:5"], ScanForQuiesce([("Other.cs", generic)]).Unquiesced); + Assert.True(MethodStillSkipsQuiesce(allowListedMethod, generic)); + Assert.False(MethodStillSkipsQuiesce("SomeOtherMethod", generic)); + } + + [Fact] + public void TheExceptions_NameRealFiles_AndEachOneStillHoldsTheDropText() + { + var directory = FindTestProjectDirectory(); + Assert.True(directory is not null, "could not find Darling/Darling.Tests from the test binary."); + + foreach (var (file, reason) in Exceptions) + { + Assert.False(string.IsNullOrWhiteSpace(reason), $"{file} is allowed without a reason."); + var path = Path.Combine(directory!, file); + Assert.True(File.Exists(path), $"{file} is in Exceptions but is not in the test project; remove the entry."); + Assert.True(File.ReadAllText(path).Contains("DROP DATABASE", StringComparison.OrdinalIgnoreCase), + $"{file} no longer holds DROP DATABASE; remove it from Exceptions."); + } + } + + [Fact] + public void APooledDrop_IsReported_AndAnUnpooledOrHandedInOneIsNot() + { + const string pooledOwn = + "public class A\n{\n public async Task Go()\n {\n" + + " await using var admin = new NpgsqlConnection(cs);\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n}\", admin);\n }\n}\n"; + const string unpooledOwn = + "public class A\n{\n public async Task Go()\n {\n" + + " await using var admin = new NpgsqlConnection(ScratchPostgres.UnpooledAdminConnectionString(cs));\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n}\", admin);\n }\n}\n"; + const string explicitOff = + "public class A\n{\n public async Task Go()\n {\n" + + " var b = new NpgsqlConnectionStringBuilder(cs) { Pooling = false };\n" + + " await using var admin = new NpgsqlConnection(b.ConnectionString);\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n}\", admin);\n }\n}\n"; + const string unpooledElsewhereInFile = + "public class A\n{\n public async Task Go()\n {\n" + + " await using var admin = new NpgsqlConnection(cs);\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n}\", admin);\n }\n\n" + + " public string Other() => ScratchPostgres.UnpooledAdminConnectionString(cs);\n}\n"; + const string handedIn = + "public class A\n{\n private static async Task Drop(NpgsqlConnection c)\n {\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n}\", c);\n }\n\n" + + " public string Other() => ScratchPostgres.UnpooledAdminConnectionString(cs);\n}\n"; + const string handedInNoEvidence = + "public class A\n{\n private static async Task Drop(NpgsqlConnection c)\n {\n" + + " await using var drop = new NpgsqlCommand($\"DROP DATABASE IF EXISTS {n}\", c);\n }\n}\n"; + const string commentOnly = "public class A\n{\n // DROP DATABASE x\n /// DROP DATABASE y\n}\n"; + + Assert.Equal(["P.cs:6"], Scan([("P.cs", pooledOwn)]).Pooled); + Assert.Empty(Scan([("U.cs", unpooledOwn)]).Pooled); + Assert.Empty(Scan([("E.cs", explicitOff)]).Pooled); + /* An unrelated unpooled helper elsewhere in the file does not excuse a drop that opens its own pooled connection. */ + Assert.Equal(["F.cs:6"], Scan([("F.cs", unpooledElsewhereInFile)]).Pooled); + Assert.Empty(Scan([("H.cs", handedIn)]).Pooled); + Assert.Equal(["N.cs:5"], Scan([("N.cs", handedInNoEvidence)]).Pooled); + Assert.Equal(0, Scan([("C.cs", commentOnly)]).Sites); + Assert.Equal(0, Scan([("DarlingCollectorRunnerTests.cs", pooledOwn)]).Sites); + } + + private static string? FindTestProjectDirectory() + { + var directory = new DirectoryInfo(AppContext.BaseDirectory); + for (var i = 0; i < 10 && directory is not null; i++) + { + if (File.Exists(Path.Combine(directory.FullName, "PerformanceMonitor.sln"))) + { + var source = Path.Combine(directory.FullName, "Darling", "Darling.Tests"); + return Directory.Exists(source) ? source : null; + } + + directory = directory.Parent; + } + + return null; + } +} diff --git a/Darling/Darling.Tests/ValidateConfigOldFormatPasswordLiveTests.cs b/Darling/Darling.Tests/ValidateConfigOldFormatPasswordLiveTests.cs new file mode 100644 index 0000000000..74973ce77e --- /dev/null +++ b/Darling/Darling.Tests/ValidateConfigOldFormatPasswordLiveTests.cs @@ -0,0 +1,168 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.IO; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: each fact mints its own scratch database through ScratchPostgres and never touches another one. + The role it connects as is created for the fact and dropped in its cleanup. */ + +/// +/// --validate-config against a store (#5366): a connection that owns the pin table reads the pins, so a pinned +/// old-format saved password opens and is tested; a connection that is not the owner cannot read them, so the verb says +/// it could not check old-format passwords and leaves those servers out, and never says one needs to be entered again. +/// +public sealed class ValidateConfigOldFormatPasswordLiveTests +{ + private const string Plain = "p@ss-not-real"; + private const int ServerId = 5366101; + + /// An old-format value is any text that is neither a reference nor sealed; it is not opened until a server's + /// password is resolved, and the registry read does not do that. + private const string OldFormatText = "AAAAAAAAAAAAAAAAAAAAAAAA"; + + private static async Task ExecAsync(NpgsqlDataSource source, string sql, CancellationToken ct, params (string Name, NpgsqlDbType Type, object Value)[] parameters) + { + await using var command = source.CreateCommand(sql); + foreach (var (name, type, value) in parameters) + { + command.Parameters.Add(new NpgsqlParameter(name, type) { Value = value }); + } + + await command.ExecuteNonQueryAsync(ct); + } + + /* The server is on this machine at a closed port, so the probe the verb runs on it fails at once. */ + private static Task InsertServerAsync(NpgsqlDataSource owner, string stored, CancellationToken ct) => ExecAsync( + owner, + "INSERT INTO config_monitored_servers (server_id, name, host, port, auth, username, encrypted_password) " + + $"VALUES ({ServerId}, 'alpha', '127.0.0.1', 1, 'sql', 'monitor', @stored)", + ct, + ("stored", NpgsqlDbType.Text, stored)); + + private static string WriteConfig(DirectoryInfo root, string connectionString) + { + var path = Path.Combine(root.FullName, "darling.json"); + File.WriteAllText(path, "{ \"postgres\": { \"connectionString\": " + JsonSerializer.Serialize(connectionString) + + " }, \"servers\": [ { \"name\": \"file-only\", \"host\": \"file-only.example.test\" } ] }"); + return path; + } + + private static async Task<(int Exit, string Output, string Error)> RunVerbAsync(string configPath, CancellationToken ct) + { + var output = new StringWriter(); + var error = new StringWriter(); + var exit = await DarlingCliCommands.ValidateConfigAsync(configPath, output, error, ct); + return (exit, output.ToString(), error.ToString()); + } + + [Fact] + public async Task AsTheStoreOwner_APinnedOldFormatPasswordIsOpenedAndTested_AndNeverReportedAsNeedingToBeEnteredAgain() + { + Assert.SkipUnless(OperatingSystem.IsWindows(), "Old-format values are Windows DPAPI values."); + var ct = TestContext.Current.CancellationToken; + var stored = DarlingSecrets.Protect(Plain); + var (scratch, owner, ownerString) = await ServerAddViewerRoleLiveTests.OpenAsync(ct); + await using var scratchHolder = scratch; + await using var ownerHolder = owner; + var root = Directory.CreateTempSubdirectory("darling-validate-pins-"); + var ok = false; + try + { + await InsertServerAsync(owner, stored, ct); + var config = WriteConfig(root, ownerString); + + /* Without a pin the value cannot be opened here, and the verb says so: the control for the run below. */ + var unpinned = await RunVerbAsync(config, ct); + Assert.Contains("Validating connectivity to 1 server(s)", unpinned.Output, StringComparison.Ordinal); + Assert.Contains("is in the old format", unpinned.Output, StringComparison.Ordinal); + + /* The pin the upgrade writes: the value's hash and the hash of the connection as the row holds it. */ + await using (var connection = await owner.OpenConnectionAsync(ct)) + { + var row = (await StoreConfigProvider.ReadMonitoredServersAsync(connection, new DarlingConfig(), ct))[0]; + await ExecAsync( + owner, + "INSERT INTO config.legacy_secret_pin (server_id, slot, value_sha256, binding_sha256) VALUES (@id, 'server', @value, @binding)", + ct, + ("id", NpgsqlDbType.Integer, ServerId), + ("value", NpgsqlDbType.Bytea, SHA256.HashData(Encoding.UTF8.GetBytes(stored))), + ("binding", NpgsqlDbType.Bytea, row.SecretBinding.LegacyPinHash())); + } + + var pinned = await RunVerbAsync(config, ct); + Assert.Contains("Validating connectivity to 1 server(s)", pinned.Output, StringComparison.Ordinal); + Assert.DoesNotContain("could not be checked", pinned.Output, StringComparison.Ordinal); + Assert.DoesNotContain("old format", pinned.Output, StringComparison.Ordinal); + Assert.DoesNotContain("entered again", pinned.Output + pinned.Error, StringComparison.Ordinal); + Assert.Contains("[FAIL] alpha", pinned.Output, StringComparison.Ordinal); + ok = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(ok, () => + { + root.Delete(true); + return Task.CompletedTask; + }); + } + } + + [Fact] + public async Task AsARoleThatIsNotTheStoreOwner_TheVerbSaysItCouldNotCheckOldFormatPasswords_AndNeverSaysOneNeedsToBeEnteredAgain() + { + var ct = TestContext.Current.CancellationToken; + var (scratch, owner, ownerString) = await ServerAddViewerRoleLiveTests.OpenAsync(ct); + await using var scratchHolder = scratch; + await using var ownerHolder = owner; + var role = "val_cfg_" + Guid.NewGuid().ToString("N")[..8]; + var root = Directory.CreateTempSubdirectory("darling-validate-pins-"); + var ok = false; + try + { + /* A login that can read the registry (the service's other roles can) but owns nothing in the store. */ + await ExecAsync(owner, $"CREATE ROLE {role} LOGIN NOSUPERUSER PASSWORD '{ServerAddViewerRoleLiveTests.RolePassword}'", ct); + await ExecAsync(owner, $"GRANT USAGE ON SCHEMA config TO {role}", ct); + await ExecAsync(owner, $"GRANT SELECT ON config.config_monitored_servers TO {role}", ct); + await InsertServerAsync(owner, OldFormatText, ct); + var viewerString = new NpgsqlConnectionStringBuilder(ownerString) + { + Username = role, + Password = ServerAddViewerRoleLiveTests.RolePassword, + }.ConnectionString; + + var result = await RunVerbAsync(WriteConfig(root, viewerString), ct); + + Assert.Contains(DarlingCliCommands.OldFormatPasswordsNotCheckedText(1), result.Output, StringComparison.Ordinal); + Assert.Contains("Validating connectivity to 0 server(s)", result.Output, StringComparison.Ordinal); + Assert.DoesNotContain("entered again", result.Output + result.Error, StringComparison.Ordinal); + Assert.DoesNotContain("[FAIL] alpha", result.Output, StringComparison.Ordinal); + ok = true; + } + finally + { + await LiveStoreCleanup.RunOwnedAsync(ok, async () => + { + root.Delete(true); + await ExecAsync(owner, $"DROP OWNED BY {role}; DROP ROLE IF EXISTS {role};", CancellationToken.None); + }); + } + } +} diff --git a/Darling/Darling.Tests/ViewSavedServerScopeTests.cs b/Darling/Darling.Tests/ViewSavedServerScopeTests.cs new file mode 100644 index 0000000000..3f92bd90f0 --- /dev/null +++ b/Darling/Darling.Tests/ViewSavedServerScopeTests.cs @@ -0,0 +1,180 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System.ComponentModel; +using System.Diagnostics; +using System.Linq; +using System.Text.Json; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// W4 of the morning walk: a custom view saved with a server (its server-dimension variable's default) opened on +/// "All servers (fleet)" and read the whole fleet, because the scope control always seeded "All" and the run body's +/// explicit server "All" beat the variable on the service side. The shipped views.js seeding is run under Node +/// (view-scope-harness.mjs); Node is skipped when it is not installed. +/// +public sealed class ViewSavedServerScopeTests +{ + private static bool TryRun(out JsonElement result) + { + result = default; + var psi = new ProcessStartInfo("node") { RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false }; + psi.ArgumentList.Add(PathTo("Darling", "Darling.Tests", "view-scope-harness.mjs")); + psi.ArgumentList.Add(PathTo("Darling", "PerformanceMonitor.Darling.Service", "wwwroot", "js", "pages", "views.js")); + + Process proc; + try + { + proc = Process.Start(psi)!; + } + catch (Win32Exception) + { + Assert.Skip("Node is not installed, so the shipped view-scope code cannot be run."); + return false; + } + + using (proc) + { + var error = proc.StandardError.ReadToEndAsync(); + var output = proc.StandardOutput.ReadToEnd().Trim(); + if (!proc.WaitForExit(20000)) + { + proc.Kill(entireProcessTree: true); + Assert.Fail("the view-scope harness did not finish in 20 s"); + } + + Assert.True(proc.ExitCode == 0, "the view-scope harness failed: " + error.Result); + using var doc = JsonDocument.Parse(output); + result = doc.RootElement.Clone(); + return true; + } + } + + private static string[] Names(JsonElement e) => e.EnumerateArray().Select(x => x.GetString()!).ToArray(); + + [Fact] + public void AViewSavedWithAServer_OpensOnThatServer() + { + if (!TryRun(out var r)) return; + Assert.Equal(new[] { "sql2025" }, Names(r.GetProperty("byName"))); + Assert.Equal(new[] { "sql2022" }, Names(r.GetProperty("byDisplayName"))); + Assert.Equal(new[] { "sql2025", "sql2022" }, Names(r.GetProperty("list"))); + } + + [Theory] + [InlineData("none")] + [InlineData("all")] + [InlineData("blank")] + [InlineData("reference")] + [InlineData("unknown")] + [InlineData("twoServerVariables")] + [InlineData("otherDimension")] + public void AViewWithNoUsableSavedServer_StaysOnTheWholeFleet(string scenario) + { + if (!TryRun(out var r)) return; + Assert.Equal("All", r.GetProperty(scenario).GetString()); + } + + // ---- W4b: a view with no server variable whose every panel filters to one server (custom view 3's shape) ---- + + [Fact] + public void AViewWhoseEveryPanelFiltersToOneServer_OpensOnThatServer() + { + if (!TryRun(out var r)) return; + Assert.Equal("sql2025", r.GetProperty("panelServerOfWholeView").GetString()); + Assert.Equal(new[] { "sql2025" }, Names(r.GetProperty("panelSeed"))); + // an explicit server variable still decides first; no panel server leaves the fleet + Assert.Equal(new[] { "sql2022" }, Names(r.GetProperty("panelSeedVariableWins"))); + Assert.Equal("All", r.GetProperty("panelSeedNone").GetString()); + Assert.Contains("every panel is set to SQL2025", r.GetProperty("noteOnServer").GetString()); + } + + [Theory] + [InlineData("panelServerMixed")] + [InlineData("panelServerOnePanelUnfiltered")] + [InlineData("panelServerTwoServerFilters")] + [InlineData("panelServerInOp")] + [InlineData("panelServerNotInFleet")] + [InlineData("panelServerNoComposed")] + public void APanelSetThatIsNotOneServerEverywhere_HasNoPanelServer(string scenario) + { + if (!TryRun(out var r)) return; + Assert.Equal(JsonValueKind.Null, r.GetProperty(scenario).ValueKind); + } + + [Fact] + public void ChoosingAnotherServer_ReplacesThePanelsServerFilter_InsteadOfEmptyingThem() + { + if (!TryRun(out var r)) return; + // still on the panels' server: the panel runs as saved (both filters kept) + Assert.Equal(2, r.GetProperty("underSame").GetInt32()); + // another server, or All servers: only the server filter goes, the other filters stay + Assert.Equal(new[] { "database" }, Names(r.GetProperty("underOther"))); + Assert.Equal(0, r.GetProperty("underAll").GetInt32()); + // a panel that is not composed passes through, and read panels do not stop a view from having a panel server + Assert.Equal("get_x", r.GetProperty("underReadPanel").GetProperty("read").GetString()); + Assert.Equal("sql2025", r.GetProperty("panelServerReadPanelsIgnored").GetString()); + Assert.Contains("Showing your pick instead of SQL2025", r.GetProperty("noteOnOther").GetString()); + } + + // ---- review round (L6, L7): an explicit server variable decides first; "your pick" only when the reader picked; one eq only ---- + + [Theory] + [InlineData("panelSeedVariableAll")] + [InlineData("panelSeedVariableReference")] + [InlineData("panelSeedVariableUnknown")] + public void AnExplicitServerVariable_AllIncluded_DecidesBeforeThePanelServerSeeds(string scenario) + { + if (!TryRun(out var r)) return; + Assert.Equal("All", r.GetProperty(scenario).GetString()); + // a variable of another dimension is not a server variable: the panels' server still seeds + Assert.Equal(new[] { "sql2025" }, Names(r.GetProperty("panelSeedOtherDimensionVariable"))); + } + + [Fact] + public void TheNoteSaysYourPick_OnlyWhenTheReaderPicked_AndNamesTheViewsOwnSettingOtherwise() + { + if (!TryRun(out var r)) return; + Assert.Contains("Showing your pick instead of SQL2025", r.GetProperty("noteOnOther").GetString()); + var variableNote = r.GetProperty("noteOnOtherNotPicked").GetString()!; + Assert.DoesNotContain("your pick", variableNote, System.StringComparison.Ordinal); + Assert.Contains("this view's own setting", variableNote, System.StringComparison.Ordinal); + Assert.Contains("SQL2025", variableNote, System.StringComparison.Ordinal); + + var flags = r.GetProperty("pickedFlags"); + Assert.False(flags.GetProperty("fresh").GetBoolean()); + Assert.False(flags.GetProperty("variable").GetBoolean()); + Assert.True(flags.GetProperty("remembered").GetBoolean()); + Assert.False(flags.GetProperty("rememberedHoursOnly").GetBoolean()); + } + + [Theory] + [InlineData("noteOnAllNotPicked")] + [InlineData("noteOnBlankNotPicked")] + public void WhenTheViewsOwnServerSettingIsAllOrBlank_TheNoteSaysAllServers_NotAServer(string scenario) + { + if (!TryRun(out var r)) return; + Assert.Equal( + "Showing all servers, as this view's own setting says, instead of SQL2025, the server this view's panels are set to.", + r.GetProperty(scenario).GetString()); + } + + [Theory] + [InlineData("panelServerEqPlusIn")] + [InlineData("panelServerEqPlusNeq")] + public void APanelWithAnyServerFilterBesidesTheOneEq_HasNoPanelServer(string scenario) + { + if (!TryRun(out var r)) return; + Assert.Equal(JsonValueKind.Null, r.GetProperty(scenario).ValueKind); + // a filter on another dimension beside the eq is fine + Assert.Equal("sql2025", r.GetProperty("panelServerEqPlusOtherDimension").GetString()); + } +} diff --git a/Darling/Darling.Tests/Viewer4627EdgeCallerTests.cs b/Darling/Darling.Tests/Viewer4627EdgeCallerTests.cs index 4850b0572f..3ae356a251 100644 --- a/Darling/Darling.Tests/Viewer4627EdgeCallerTests.cs +++ b/Darling/Darling.Tests/Viewer4627EdgeCallerTests.cs @@ -34,6 +34,7 @@ namespace Darling.Tests; /// nesting level only, so a nested call's own commas are not miscounted; a self-check below feeds it both /// shapes. /// +[Trait("Reads", "Lite")] public sealed class Viewer4627EdgeCallerTests { /// The two files that turn a plan edge into a brush. diff --git a/Darling/Darling.Tests/Viewer4629Tests.cs b/Darling/Darling.Tests/Viewer4629Tests.cs index 59a6ad85d5..02ffd99cb8 100644 --- a/Darling/Darling.Tests/Viewer4629Tests.cs +++ b/Darling/Darling.Tests/Viewer4629Tests.cs @@ -52,6 +52,7 @@ namespace Darling.Tests; /// assertion rather than left to that coincidence. No per-theme value changed for #4635: the existing /// #4629 hex already clears this background too. /// +[Trait("Reads", "Lite")] public class Viewer4629Tests { // The node card background (PlanViewerControl.Rendering.cs BuildNode: FindResource("BackgroundLightBrush")) — diff --git a/Darling/Darling.Tests/ViewerAddServerIdCollisionTests.cs b/Darling/Darling.Tests/ViewerAddServerIdCollisionTests.cs index bb0d974bbf..441b035d52 100644 --- a/Darling/Darling.Tests/ViewerAddServerIdCollisionTests.cs +++ b/Darling/Darling.Tests/ViewerAddServerIdCollisionTests.cs @@ -257,10 +257,17 @@ public void TheAddMethod_InsertsIfAbsent_ReadsTheHolderWithoutTheSecret_AndClass Assert.True(start >= 0 && end > start, "the Add method and the classifier must both be there, in that order"); var body = source[start..end]; - Assert.Contains("InsertMonitoredServerIfAbsentAsync(row, cancellationToken)", body, StringComparison.Ordinal); - Assert.Contains("CreateCommand(MonitoredServerByIdNoSecretSql)", body, StringComparison.Ordinal); + /* #5240: the insert-if-absent and the holder read now run on the transaction that holds the identity lock, + so they are built on that connection (new NpgsqlCommand(sql, connection, transaction)) instead of through + the data source; the statements are the same ones, and the lock is taken before the insert. */ + Assert.Contains("new NpgsqlCommand(MonitoredServerInsertIfAbsentSql, connection, transaction)", body, StringComparison.Ordinal); + Assert.Contains("new NpgsqlCommand(MonitoredServerByIdNoSecretSql, connection, transaction)", body, StringComparison.Ordinal); Assert.Contains("ClassifyAddAgainstOccupant(row, occupant)", body, StringComparison.Ordinal); Assert.DoesNotContain("Upsert", body, StringComparison.Ordinal); + Assert.True( + body.IndexOf("TakeIdentityLockAndFindClaimantAsync(", StringComparison.Ordinal) is var lockAt and >= 0 + && lockAt < body.IndexOf("MonitoredServerInsertIfAbsentSql", StringComparison.Ordinal), + "the add must take the identity lock and re-read the addresses before it inserts"); } private static int CountOf(string source, string needle) diff --git a/Darling/Darling.Tests/ViewerAlertClockCacheTests.cs b/Darling/Darling.Tests/ViewerAlertClockCacheTests.cs index 33e1cb6762..453d3121a0 100644 --- a/Darling/Darling.Tests/ViewerAlertClockCacheTests.cs +++ b/Darling/Darling.Tests/ViewerAlertClockCacheTests.cs @@ -142,7 +142,7 @@ public void TheAlertHistoryReadTakesItsClocksFromTheCache_NotFromTheStoreEveryCa var alertHistory = ViewerCode("ViewerDataService.AlertHistory.cs"); var method = Regex.Match( alertHistory, - @"public async Task> GetAlertHistoryAsync\(.*?\n \}\n", + @"public async Task> GetAlertHistoryWindowAsync\(.*?\n \}\n", RegexOptions.Singleline); Assert.True(method.Success); diff --git a/Darling/Darling.Tests/ViewerApplyToAllClockTests.cs b/Darling/Darling.Tests/ViewerApplyToAllClockTests.cs index f2e8583f5e..806c1d039d 100644 --- a/Darling/Darling.Tests/ViewerApplyToAllClockTests.cs +++ b/Darling/Darling.Tests/ViewerApplyToAllClockTests.cs @@ -61,7 +61,7 @@ public void ThePlaceThatBroadcastsTheRange_SkipsOnlyTheSource_AndASelectedTabRel var selection = Member("MainWindow.xaml.cs", "MainTabs_SelectionChanged"); var load = Member("MainWindow.xaml.cs", "LoadVisibleTabAsync"); - Assert.Contains("ApplyExternalTimeRange(index, customFromUtc, customToUtc)", broadcast); + Assert.Contains("ApplyExternalTimeRange(range)", broadcast); Assert.Contains("!ReferenceEquals(serverTab, source)", broadcast); Assert.Contains("await RefreshVisibleAsync()", selection); Assert.Matches(@"case\s+TabItem\s*\{\s*Content:\s*ViewerServerTab\s+serverTab\s*\}\s*:\s*await\s+serverTab\.RefreshActiveInnerTabAsync\(\)", load); diff --git a/Darling/Darling.Tests/ViewerAwsRoleFieldsTests.cs b/Darling/Darling.Tests/ViewerAwsRoleFieldsTests.cs new file mode 100644 index 0000000000..0315915881 --- /dev/null +++ b/Darling/Darling.Tests/ViewerAwsRoleFieldsTests.cs @@ -0,0 +1,229 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Globalization; +using System.Linq; +using System.Text.RegularExpressions; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// The desktop viewer's per-server AWS role fields (#5452). The writes name aws_role_arn and aws_external_id and +/// never the generated aws_external_id_set; no read selects the external ID, only whether one is set, and an edit +/// keeps the stored ID unless a new one is typed or the clear box is ticked; and every read's column count agrees with the +/// ordinals its reader uses, which a live test would only catch on a store with the V169 columns. +/// +public sealed class ViewerAwsRoleFieldsTests +{ + private static readonly Regex ExternalIdColumn = new(@"\baws_external_id\b", RegexOptions.CultureInvariant); + + /// The comma-separated column list of a SELECT, between SELECT and FROM. + private static string[] SelectedColumns(string sql) + { + var select = sql.IndexOf("SELECT", StringComparison.Ordinal) + "SELECT".Length; + var from = sql.IndexOf("FROM config_monitored_servers", StringComparison.Ordinal); + return sql[select..from].Split(',', StringSplitOptions.TrimEntries | StringSplitOptions.RemoveEmptyEntries); + } + + private static int HighestOrdinal(string source, string methodName) + { + var start = source.IndexOf(methodName + "(NpgsqlDataReader reader)", StringComparison.Ordinal); + Assert.True(start >= 0, methodName); + var end = source.IndexOf("};", start, StringComparison.Ordinal); + return Regex.Matches(source[start..end], @"reader\.\w+(?:<[^>]+>)?\((\d+)\)") + .Select(m => int.Parse(m.Groups[1].Value, CultureInfo.InvariantCulture)).Max(); + } + + [Fact] + public void TheWrites_NameTheRoleAndTheExternalId_AndNeverTheGeneratedFlag() + { + foreach (var sql in new[] { ViewerDataService.MonitoredServerUpsertSql, ViewerDataService.MonitoredServerInsertIfAbsentSql }) + { + Assert.Contains("aws_role_arn", sql, StringComparison.Ordinal); + Assert.Matches(ExternalIdColumn, sql); + Assert.DoesNotContain("aws_external_id_set", sql, StringComparison.Ordinal); + + /* One value per column: the column list and the VALUES list agree (the two server-side timestamps are columns too). */ + var columnsStart = sql.IndexOf('(', StringComparison.Ordinal) + 1; + var columns = sql[columnsStart..sql.IndexOf(')', StringComparison.Ordinal)].Split(',').Length; + var valuesStart = sql.IndexOf("VALUES (", StringComparison.Ordinal) + "VALUES (".Length; + var values = sql[valuesStart..sql.IndexOf("ON CONFLICT", StringComparison.Ordinal)]; + Assert.Equal(columns, Regex.Matches(values, @"\$\d+|\(now\(\) AT TIME ZONE 'UTC'\)").Count); + Assert.Contains("$20", sql, StringComparison.Ordinal); + } + + Assert.DoesNotContain("$21", ViewerDataService.MonitoredServerInsertIfAbsentSql, StringComparison.Ordinal); + Assert.Contains("aws_role_arn = EXCLUDED.aws_role_arn", ViewerDataService.MonitoredServerUpsertSql, StringComparison.Ordinal); + } + + [Fact] + public void TheUpsert_KeepsTheStoredExternalId_UnlessOneWasTypedOrTheBoxWasCleared() + { + var sql = ViewerDataService.MonitoredServerUpsertSql; + Assert.DoesNotContain("aws_external_id = EXCLUDED.aws_external_id,", sql, StringComparison.Ordinal); + Assert.Contains("WHEN $21 THEN EXCLUDED.aws_external_id", sql, StringComparison.Ordinal); + Assert.Contains("ELSE config_monitored_servers.aws_external_id END", sql, StringComparison.Ordinal); + Assert.Contains("WHEN EXCLUDED.aws_role_arn IS NULL THEN NULL", sql, StringComparison.Ordinal); + + /* A role that changes with nothing typed clears the stored ID; the arm sits after the typed one and before the keep. */ + const string changed = "WHEN EXCLUDED.aws_role_arn IS DISTINCT FROM config_monitored_servers.aws_role_arn THEN NULL"; + var typed = sql.IndexOf("WHEN $21 THEN EXCLUDED.aws_external_id", StringComparison.Ordinal); + var kept = sql.IndexOf("ELSE config_monitored_servers.aws_external_id END", StringComparison.Ordinal); + var arm = sql.IndexOf(changed, StringComparison.Ordinal); + Assert.True(arm > typed && arm < kept, "the role-changed arm follows the typed arm and precedes the keep"); + } + + [Fact] + public void NoRead_SelectsTheExternalId_EachCarriesTheRoleAndTheFlag() + { + foreach (var sql in new[] + { + ViewerDataService.MonitoredServerByIdSql, + ViewerDataService.MonitoredServersSelectSql, + ViewerDataService.MonitoredServerByIdNoSecretSql, + ViewerDataService.MonitoredServerByAddressSql, + }) + { + Assert.DoesNotMatch(ExternalIdColumn, sql); + Assert.Contains("aws_role_arn", sql, StringComparison.Ordinal); + Assert.Contains("aws_external_id_set", sql, StringComparison.Ordinal); + } + } + + [Fact] + public void EveryRead_HasAsManyColumnsAsItsReaderUses() + { + var source = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.MonitoredServers.cs"); + + Assert.Equal(HighestOrdinal(source, "ReadMonitoredServerRow") + 1, SelectedColumns(ViewerDataService.MonitoredServerByIdSql).Length); + + var noSecret = HighestOrdinal(source, "ReadMonitoredServerRowNoSecret") + 1; + Assert.Equal(noSecret, SelectedColumns(ViewerDataService.MonitoredServersSelectSql).Length); + Assert.Equal(noSecret, SelectedColumns(ViewerDataService.MonitoredServerByIdNoSecretSql).Length); + Assert.Equal(noSecret, SelectedColumns(ViewerDataService.MonitoredServerByAddressSql).Length); + } + + [Fact] + public void TheRowDefaults_AreNoRole_SoEveryExistingWriterIsUnchanged() + { + var row = new MonitoredServerRow(); + Assert.Null(row.AwsRoleArn); + Assert.Null(row.AwsExternalId); + Assert.False(row.AwsExternalIdSent); + Assert.False(row.AwsExternalIdSet); + } + + [Fact] + public void TheBulkDialog_TakesNoRole() + { + var source = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "AddMultipleServersDialog.xaml.cs"); + Assert.DoesNotContain("Aws", source, StringComparison.Ordinal); + } + + private const string Role = "arn:aws:iam::123456789012:role/darling-monitor"; + private const string OtherRole = "arn:aws:iam::123456789012:role/darling-other"; + + [Fact] + public void Resolve_BlankBoxes_AreNotSet_AndOnAnEditClearTheStoredRole() + { + Assert.Equal((null, null, false, null), AddServerDialog.ResolveAwsRole(" ", "", false, null, false)); + Assert.Equal((null, null, false, null), AddServerDialog.ResolveAwsRole("", "", false, Role, false)); + } + + [Fact] + public void Resolve_TheSharedChecksApply_WithTheirFixedSentences() + { + Assert.Equal(AwsRoleSettings.InvalidRoleMessage, AddServerDialog.ResolveAwsRole("not-an-arn", "", false, null, false).Error); + Assert.Equal(AwsRoleSettings.InvalidExternalIdMessage, AddServerDialog.ResolveAwsRole(Role, "has space", false, null, false).Error); + Assert.Equal(AwsRoleSettings.ExternalIdNeedsRoleMessage, AddServerDialog.ResolveAwsRole("", "ext-1234", false, null, false).Error); + Assert.Equal((Role, "ext-1234", true, null), AddServerDialog.ResolveAwsRole($" {Role} ", " ext-1234 ", false, null, false)); + } + + [Fact] + public void Resolve_AnEditWithTheBoxUntouched_KeepsTheStoredExternalId() + { + /* Nothing typed and the clear box not ticked: nothing is sent, so the upsert keeps the stored ID. */ + Assert.Equal((Role, null, false, null), AddServerDialog.ResolveAwsRole(Role, "", false, Role, true)); + Assert.Equal((Role, null, false, null), AddServerDialog.ResolveAwsRole(Role, " ", false, Role, true)); + } + + [Fact] + public void Resolve_ATypedExternalId_Replaces_AndTheClearBoxRemoves() + { + Assert.Equal((Role, "ext-5678", true, null), AddServerDialog.ResolveAwsRole(Role, "ext-5678", false, Role, true)); + Assert.Equal((Role, null, true, null), AddServerDialog.ResolveAwsRole(Role, "", true, Role, true)); + Assert.Equal(AddServerDialog.AwsExternalIdTypedAndClearedMessage, AddServerDialog.ResolveAwsRole(Role, "ext-5678", true, Role, true).Error); + } + + [Fact] + public void Resolve_ANewRole_NeedsTheStoredExternalIdChangedOrCleared() + { + /* The box is untouched and one is stored: the store's trigger would refuse it (PW004), so the dialog says so first. */ + Assert.Equal(AwsRoleSettings.RoleChangeNeedsExternalIdMessage, + AddServerDialog.ResolveAwsRole(OtherRole, "", false, Role, true).Error); + + /* A new ID, or a cleared one, goes through; an unchanged role never needs it. */ + Assert.Null(AddServerDialog.ResolveAwsRole(OtherRole, "ext-5678", false, Role, true).Error); + Assert.Null(AddServerDialog.ResolveAwsRole(OtherRole, "", true, Role, true).Error); + Assert.Null(AddServerDialog.ResolveAwsRole(Role, "", false, Role, true).Error); + + /* A target with no stored ID takes a new role freely. */ + Assert.Null(AddServerDialog.ResolveAwsRole(OtherRole, "", false, Role, false).Error); + } + + [Fact] + public void TheDialog_ShowsWhetherAnExternalIdIsSet_AndOffersAClearBox_AndNeverFillsTheBox() + { + var xaml = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "AddServerDialog.xaml"); + Assert.Contains("x:Name=\"AwsClearExternalIdBox\" Content=\"Clear external ID\"", xaml, StringComparison.Ordinal); + + var code = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "AddServerDialog.xaml.cs"); + Assert.Contains("AwsExternalIdState.Text = existing.AwsExternalIdSet ? AwsExternalIdSetNote : AwsExternalIdNoneNote;", code, StringComparison.Ordinal); + Assert.DoesNotContain("AwsExternalIdBox.Password =", code, StringComparison.Ordinal); + Assert.Contains("Leave the box blank to keep it", AddServerDialog.AwsExternalIdSetNote, StringComparison.Ordinal); + } + + [Fact] + public void TheSaveNote_SaysTheServiceUsesTheRoleOnlyWhenItIsAllowed() + { + Assert.Equal( + "The service uses this AWS role only if it is listed in allowedAwsRoles in darling.json, or a server in darling.json uses it. " + + "A role saved here does not take effect until then.", + AddServerDialog.AwsRoleSaveNote); + } + + [Fact] + public void TheDialog_PutsTheBoxesInThePostgresPanel_AndBindsTheNoteFromTheConstant() + { + var xaml = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "AddServerDialog.xaml"); + var panel = xaml.IndexOf("x:Name=\"PostgresOptionsPanel\"", StringComparison.Ordinal); + var panelEnd = xaml.IndexOf("", "", "", RegexOptions.Singleline); + + private static IEnumerable ViewerXamlFiles() => + Directory.EnumerateFiles(ViewerDir(), "*.xaml", SearchOption.AllDirectories) + .Where(f => !f.Contains(Path.DirectorySeparatorChar + "obj" + Path.DirectorySeparatorChar, StringComparison.Ordinal) + && !f.Contains(Path.DirectorySeparatorChar + "bin" + Path.DirectorySeparatorChar, StringComparison.Ordinal)); + + [Fact] + public void EveryColumnHeaderMadeOfAPanel_CarriesAVisibleTitle_SoItHasAScreenReaderName() + { + var offenders = new List(); + var checkedHeaders = 0; + + foreach (var file in ViewerXamlFiles()) + { + foreach (Match header in s_columnHeader.Matches(StripComments(File.ReadAllText(file)))) + { + var body = header.Groups["body"].Value; + + if (!body.Contains("]*\\bText=\"[^\"]+\"")) + { + var flat = Regex.Replace(body.Trim(), @"\s+", " "); + offenders.Add(Path.GetFileName(file) + ": " + flat.Substring(0, Math.Min(120, flat.Length))); + } + } + } + + Assert.True(checkedHeaders > 50, "the census found only " + checkedHeaders + " panel headers; the pattern no longer matches the XAML"); + Assert.True(offenders.Count == 0, "column headers with no visible title to name them from:\n" + string.Join("\n", offenders)); + } + + [Fact] + public void EveryFilterButtonSitsInAColumnHeader_WhereTheNameIsGivenToIt() + { + var offenders = new List(); + + foreach (var file in ViewerXamlFiles().Where(f => !f.Contains(Path.DirectorySeparatorChar + "Themes" + Path.DirectorySeparatorChar, StringComparison.Ordinal))) + { + var text = StripComments(File.ReadAllText(file)); + var inHeaders = s_columnHeader.Matches(text).Sum(m => Regex.Matches(m.Groups["body"].Value, "ColumnFilterButtonStyle").Count); + var all = Regex.Matches(text, "ColumnFilterButtonStyle").Count; + + if (all != inHeaders) + { + offenders.Add(Path.GetFileName(file) + ": " + (all - inHeaders) + " filter button(s) outside a column header"); + } + } + + Assert.True(offenders.Count == 0, string.Join("\n", offenders)); + } + + [Theory] + [InlineData("FinOpsTab.xaml", "ServerSelector", "PickerLabel")] + [InlineData("MainWindow.xaml", "RecommendationsServerSelector", "DisplayName")] + public void TheServerBox_HasAName_AndItsItemsAreNamedByTheServer(string file, string boxName, string member) + { + var text = File.ReadAllText(Path.Combine(ViewerDir(), file)); + var start = text.IndexOf("= 0); + var box = text.Substring(start, text.IndexOf("", start, StringComparison.Ordinal) - start); + + Assert.Contains("AutomationProperties.Name=\"Server\"", box, StringComparison.Ordinal); + Assert.Contains("", box, StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/ViewerScreenReaderRowNamesTests.cs b/Darling/Darling.Tests/ViewerScreenReaderRowNamesTests.cs new file mode 100644 index 0000000000..8421dc80e8 --- /dev/null +++ b/Darling/Darling.Tests/ViewerScreenReaderRowNamesTests.cs @@ -0,0 +1,311 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Windows; +using System.Windows.Automation; +using System.Windows.Automation.Peers; +using System.Windows.Controls; +using System.Windows.Data; +using System.Windows.Threading; +using PerformanceMonitor.Ui; +using Xunit; + +namespace Darling.Tests; + +/// +/// What UI Automation reads for a grid ROW (it was the data item's type name, "...ViewerJobHistoryRow"), for a templated cell with +/// nothing in it, and what the per-event handlers cost. Lite's twin is ScreenReaderRowNamesTests in Lite.Tests. +/// +public sealed class ViewerScreenReaderRowNamesTests +{ + private static void Settle(Window window) + { + window.Show(); + var frame = new DispatcherFrame(); + window.Dispatcher.BeginInvoke(DispatcherPriority.ApplicationIdle, new Action(() => frame.Continue = false)); + Dispatcher.PushFrame(frame); + } + + private static void Walk(AutomationPeer peer, List into) + { + into.Add(peer); + foreach (var child in peer.GetChildren() ?? new List()) + { + Walk(child, into); + } + } + + private static IEnumerable Descendants(DependencyObject root) + { + for (var i = 0; i < System.Windows.Media.VisualTreeHelper.GetChildrenCount(root); i++) + { + var child = System.Windows.Media.VisualTreeHelper.GetChild(root, i); + yield return child; + foreach (var deeper in Descendants(child)) + { + yield return deeper; + } + } + } + + /// No ToString override: a screen reader reads the type name, as it did for the job history row. + public sealed class JobRow + { + public string Time { get; set; } = ""; + public string Server { get; set; } = ""; + public string Retries { get; set; } = ""; + } + + private static DataTemplate TextTemplate(string path) + { + var template = new DataTemplate(); + var panel = new FrameworkElementFactory(typeof(StackPanel)); + var text = new FrameworkElementFactory(typeof(TextBlock)); + text.SetBinding(TextBlock.TextProperty, new Binding(path)); + panel.AppendChild(text); + template.VisualTree = panel; + return template; + } + + private static DataGrid NewGrid(IEnumerable columns, IEnumerable rows, bool recycling = false) + { + var grid = new DataGrid { AutoGenerateColumns = false, HeadersVisibility = DataGridHeadersVisibility.Column }; + foreach (var column in columns) + { + grid.Columns.Add(column); + } + + if (recycling) + { + VirtualizingPanel.SetVirtualizationMode(grid, VirtualizationMode.Recycling); + } + + grid.ItemsSource = rows.ToList(); + return grid; + } + + private static DataGridTextColumn TextColumn(string header, string path) => + new() { Header = header, Binding = new Binding(path) }; + + [Fact] + public void AGridRow_IsNamedFromItsFirstTwoVisibleColumns_NotFromItsDataTypeName() + { + var names = StaTestThread.Run(() => + { + AccessibleNames.Register(); + var grid = NewGrid(new DataGridColumn[] { TextColumn("Time", "Time"), TextColumn("Server", "Server"), TextColumn("Retries", "Retries") }, + new[] { new JobRow { Time = "06:01:12", Server = "example-sql-01" } }); + var window = new Window { Content = grid, Width = 500, Height = 300 }; + Settle(window); + var peers = new List(); + Walk(UIElementAutomationPeer.CreatePeerForElement(grid), peers); + window.Close(); + return peers.Select(p => (Type: p.GetType().Name, Name: p.GetName())).ToList(); + }); + + Assert.Contains(names, n => n.Type == "DataGridItemAutomationPeer" && n.Name == "Time 06:01:12, Server example-sql-01"); + Assert.DoesNotContain(names, n => n.Name.Contains("JobRow", StringComparison.Ordinal)); + } + + [Fact] + public void AGridRow_SkipsHiddenColumnsAndReadsTemplatedColumns_AndFollowsTheDisplayOrder() + { + var name = StaTestThread.Run(() => + { + AccessibleNames.Register(); + var grid = NewGrid(new DataGridColumn[] + { + new DataGridTemplateColumn { Header = "Retries", CellTemplate = TextTemplate("Retries") }, + new DataGridTextColumn { Header = "Hidden", Binding = new Binding("Time"), Visibility = Visibility.Collapsed }, + new DataGridTemplateColumn { Header = "Server", CellTemplate = TextTemplate("Server") }, + TextColumn("Time", "Time"), + }, + new[] { new JobRow { Time = "06:01:12", Server = "example-sql-01", Retries = "" } }); + var window = new Window { Content = grid, Width = 500, Height = 300 }; + Settle(window); + var row = Descendants(grid).OfType().First(); + var result = AutomationProperties.GetName(row); + window.Close(); + return result; + }); + + /* Retries is empty, so the first two columns WITH text are Server and Time; the hidden column is never read. */ + Assert.Equal("Server example-sql-01, Time 06:01:12", name); + } + + [Fact] + public void ARowReusedForAnotherItem_IsRenamed_AndWatchersFollowTheRealizedCells_NotTheRowCount() + { + var (mismatches, checkedRows, atTop, peak, realized) = StaTestThread.Run(() => + { + AccessibleNames.Register(); + var rows = Enumerable.Range(0, 3000).Select(i => new JobRow { Time = "T" + i, Server = "S" + i, Retries = i % 2 == 0 ? "" : "1" }).ToList(); + var grid = NewGrid(new DataGridColumn[] { TextColumn("Time", "Time"), TextColumn("Server", "Server"), TextColumn("Retries", "Retries") }, rows, recycling: true); + var window = new Window { Content = grid, Width = 500, Height = 300 }; + Settle(window); + var scroll = Descendants(grid).OfType().First(); + var top = AccessibleNames.WatcherCount; + var max = top; + var bad = 0; + var seen = 0; + + for (var offset = 0; offset <= rows.Count; offset += 37) + { + scroll.ScrollToVerticalOffset(offset); + Settle(window); + max = Math.Max(max, AccessibleNames.WatcherCount); + + foreach (var row in Descendants(grid).OfType()) + { + if (row.Item is not JobRow item) + { + continue; + } + + seen++; + + if (AutomationProperties.GetName(row) != "Time " + item.Time + ", Server " + item.Server) + { + bad++; + } + } + } + + var realizedCells = Descendants(grid).OfType().Count(); + window.Close(); + return (bad, seen, top, max, realizedCells); + }); + + Assert.True(checkedRows > 1000, "the scroll visited only " + checkedRows + " rows"); + Assert.Equal(0, mismatches); + /* A few watchers per realized cell and row, never one per data row. */ + Assert.True(peak <= atTop * 2, "watchers grew from " + atTop + " to " + peak + " while scrolling 3000 rows (" + realized + " cells realized at the end)"); + Assert.True(peak < 600, "watchers peaked at " + peak); + } + + [Fact] + public void ATemplatedCellWithNothingVisible_IsNamedBlank_AndKeepsItsRealContent() + { + var (blank, filled, withBox, withText) = StaTestThread.Run(() => + { + AccessibleNames.Register(); + var box = new DataTemplate { VisualTree = new FrameworkElementFactory(typeof(CheckBox)) }; + var grid = NewGrid(new DataGridColumn[] + { + TextColumn("Time", "Time"), + new DataGridTemplateColumn { Header = "Retries", CellTemplate = TextTemplate("Retries") }, + new DataGridTemplateColumn { Header = "Done", CellTemplate = box }, + }, + new[] { new JobRow { Time = "06:01:12" } }); + var window = new Window { Content = grid, Width = 500, Height = 300 }; + Settle(window); + var cells = Descendants(grid).OfType().ToList(); + var retries = cells.First(c => c.Column.Header as string == "Retries"); + var done = cells.First(c => c.Column.Header as string == "Done"); + var time = cells.First(c => c.Column.Header as string == "Time"); + var first = AutomationProperties.GetName(retries); + var boxName = AutomationProperties.GetName(done); + var timeName = AutomationProperties.GetName(time); + var text = Descendants(retries).OfType().First(); + text.Text = "3"; + var second = AutomationProperties.GetName(retries); + window.Close(); + return (first, second, boxName, timeName); + }); + + Assert.Equal("Retries: blank", blank); + Assert.Equal("", filled); + Assert.Equal("", withBox); + Assert.Equal("", withText); + } + + [Fact] + public void AfterTheFirstLook_TheSizeChangedHandlers_DoNoAllocationAndAddNoWatchers() + { + var (bytes, before, after) = StaTestThread.Run(() => + { + AccessibleNames.Register(); + var tabs = new TabControl(); + tabs.Items.Add(new TabItem { Header = new StackPanel { Children = { new TextBlock { Text = "example-sql-01" } } }, Content = new TextBlock() }); + tabs.Items.Add(new TabItem { Header = new StackPanel(), Content = new TextBlock() }); + var panelHeader = new StackPanel { Orientation = Orientation.Horizontal }; + panelHeader.Children.Add(new TextBlock { Text = "Time" }); + var grid = NewGrid(new DataGridColumn[] + { + new DataGridTextColumn { Header = panelHeader, Binding = new Binding("Time") }, + new DataGridTextColumn { Header = new StackPanel(), Binding = new Binding("Retries") }, + TextColumn("Server", "Server"), + }, + new[] { new JobRow { Time = "06:01:12", Server = "example-sql-01", Retries = "" } }); + var host = new StackPanel(); + host.Children.Add(tabs); + host.Children.Add(grid); + var window = new Window { Content = host, Width = 500, Height = 400 }; + Settle(window); + + var elements = Descendants(host).Where(d => d is TabItem or System.Windows.Controls.Primitives.DataGridColumnHeader or DataGridRow or DataGridCell).Cast().ToList(); + Assert.True(elements.Count >= 9, "found " + elements.Count + " elements"); + var args = new RoutedEventArgs(FrameworkElement.SizeChangedEvent); + RoutedEventHandler[] handlers = + { + AccessibleNames.OnCellSized, AccessibleNames.OnRowSized, AccessibleNames.OnColumnHeaderSized, AccessibleNames.OnTabItemSized, + }; + + foreach (var element in elements) + { + foreach (var handler in handlers) + { + handler(element, args); + } + } + + var watchersBefore = AccessibleNames.WatcherCount; + + /* A handler that allocates per call allocates in EVERY window below; a one-off cost the runtime pays on this thread + (the JIT moving a handler to its optimised code, a type or static first reached on the CI machine) lands in one + window at most. CI saw 23704 bytes once over 640,000 calls, which is the second kind, so the proof is the + smallest window: a real per-call allocation can never make it zero. */ + long Window(int passes) + { + var allocatedBefore = GC.GetAllocatedBytesForCurrentThread(); + + for (var i = 0; i < passes; i++) + { + foreach (var element in elements) + { + foreach (var handler in handlers) + { + handler(element, args); + } + } + } + + return GC.GetAllocatedBytesForCurrentThread() - allocatedBefore; + } + + Window(500); + var allocated = long.MaxValue; + + for (var w = 0; w < 5; w++) + { + allocated = Math.Min(allocated, Window(400)); + } + + var watchersAfter = AccessibleNames.WatcherCount; + window.Close(); + return (allocated, watchersBefore, watchersAfter); + }); + + Assert.Equal(0, bytes); + Assert.Equal(before, after); + } +} diff --git a/Darling/Darling.Tests/ViewerServerInventoryEnabledTests.cs b/Darling/Darling.Tests/ViewerServerInventoryEnabledTests.cs index 7d17b23775..9ff81606a9 100644 --- a/Darling/Darling.Tests/ViewerServerInventoryEnabledTests.cs +++ b/Darling/Darling.Tests/ViewerServerInventoryEnabledTests.cs @@ -31,7 +31,7 @@ namespace Darling.Tests; public class ViewerServerInventoryEnabledTests { private static string InventorySource => ReadRepoFile(Path.Combine( - "Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.FinOps.Inventory.cs")); + "Darling", "PerformanceMonitor.Darling.Storage", "FinOps", "DarlingFinOpsInventoryReader.cs")); /// /// The projection carries is_enabled, and it comes from the REGISTRY rather than from the properties @@ -66,12 +66,12 @@ public void TheInventoryQuery_SortsLiveServersFirst() [Fact] public void TheReader_ReadsIsEnabledAt17_MonthlyCostAt18_AndLastCollectedAt19() { - Assert.Contains("IsEnabled = reader.IsDBNull(17)", InventorySource, StringComparison.Ordinal); - Assert.Contains("MonthlyCost = reader.IsDBNull(18)", InventorySource, StringComparison.Ordinal); - Assert.Contains("LastCollected = reader.IsDBNull(19)", InventorySource, StringComparison.Ordinal); + Assert.Contains("IsEnabled: reader.IsDBNull(17)", InventorySource, StringComparison.Ordinal); + Assert.Contains("MonthlyCost: reader.IsDBNull(18)", InventorySource, StringComparison.Ordinal); + Assert.Contains("LastCollectedUtc: reader.IsDBNull(19)", InventorySource, StringComparison.Ordinal); /* And nothing still reads the pre-shift position for the cost. */ - Assert.DoesNotContain("MonthlyCost = reader.IsDBNull(17)", InventorySource, StringComparison.Ordinal); + Assert.DoesNotContain("MonthlyCost: reader.IsDBNull(17)", InventorySource, StringComparison.Ordinal); } /// diff --git a/Darling/Darling.Tests/ViewerServerManagementTests.cs b/Darling/Darling.Tests/ViewerServerManagementTests.cs index e13edc1e1b..93929194fb 100644 --- a/Darling/Darling.Tests/ViewerServerManagementTests.cs +++ b/Darling/Darling.Tests/ViewerServerManagementTests.cs @@ -12,7 +12,9 @@ using System.Text.Json; using PerformanceMonitor.Collectors; using PerformanceMonitor.Common; +using System.Linq; using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; using PerformanceMonitor.Darling.Viewer; using Xunit; @@ -31,6 +33,7 @@ public sealed class ViewerMonitoredServerSqlTests "server_id", "name", "host", "database", "auth", "username", "encrypted_password", "encrypt_mode", "trust_server_certificate", "read_only_intent", "multi_subnet_failover", "excluded_databases", "monthly_cost_usd", "capture_plans", "is_enabled", "alert_delivery_mode_override", "engine", "port", + "aws_role_arn", "aws_external_id", }; [Fact] @@ -44,9 +47,10 @@ public void UpsertSql_WritesEveryStoreColumn_AsBoundParameters_OnConflictUpdate( Assert.Contains(column, sql, StringComparison.Ordinal); } - /* Every value is a $N parameter (18 columns incl. the V18 alert_delivery_mode_override and the #3499 - engine + port — the V70 columns the MCP tool wrote and the viewer silently defaulted), never inlined. */ - for (var i = 1; i <= 18; i++) + /* Every value is a $N parameter (20 columns incl. the V18 alert_delivery_mode_override, the #3499 + engine + port — the V70 columns the MCP tool wrote and the viewer silently defaulted — and the #5452 + aws_role_arn + aws_external_id), never inlined. */ + for (var i = 1; i <= 20; i++) { Assert.Contains("$" + i.ToString(System.Globalization.CultureInfo.InvariantCulture), sql, StringComparison.Ordinal); } @@ -379,40 +383,6 @@ public void RequiresSecret_ForSqlAndServicePrincipal_NotWindowsOrManagedIdentity } } -/// -/// The DPAPI blob the viewer writes into encrypted_password must be readable by the SERVICE, and vice -/// versa — a cross-project round trip against proving the entropy + scope match -/// (Windows-only; DPAPI is unavailable elsewhere). -/// -public sealed class ViewerServerSecretTests -{ - [Fact] - public void ViewerProtect_IsReadableByTheService() - { - Assert.SkipUnless(OperatingSystem.IsWindows(), "DPAPI requires Windows."); - var blob = ViewerServerSecret.Protect("s3cr3t!"); - Assert.Equal("s3cr3t!", DarlingSecrets.Unprotect(blob)); - } - - [Fact] - public void ServiceProtect_IsReadableByTheViewer() - { - Assert.SkipUnless(OperatingSystem.IsWindows(), "DPAPI requires Windows."); - var blob = DarlingSecrets.Protect("another-pw"); - Assert.Equal("another-pw", ViewerServerSecret.TryUnprotect(blob)); - } - - [Fact] - public void TryUnprotect_ReturnsNullOnGarbage_NeverThrows() - { - Assert.SkipUnless(OperatingSystem.IsWindows(), "DPAPI requires Windows."); - Assert.Null(ViewerServerSecret.TryUnprotect(null)); - Assert.Null(ViewerServerSecret.TryUnprotect("")); - Assert.Null(ViewerServerSecret.TryUnprotect("not-base64!!")); - Assert.Null(ViewerServerSecret.TryUnprotect(Convert.ToBase64String(new byte[] { 1, 2, 3, 4 }))); - } -} - /// /// The one-time viewer-servers.json → config migrate-in: the per-entry projection (auth mapping + secret /// resolution + the shared server_id) and the once-marker guard. Uses an in-memory secret fake + temp files, @@ -448,15 +418,32 @@ public void Projection_SqlEntry_SealsTheSecretAsAServiceReadableBlob() var entry = new ViewerServerEntry { ServerName = "SQL2019", DisplayName = "Dev", AuthenticationType = AuthenticationTypes.SqlServer }; fixture.ServerStore.AddServer(entry, "monitor", "p@ss"); - var (row, reason) = fixture.Migration.TryProjectEntry(fixture.ServerStore.GetByServerName("SQL2019")!); + using var key = PasswordPrivateKey.Generate(); + var (row, reason) = fixture.Migration.TryProjectEntry( + fixture.ServerStore.GetByServerName("SQL2019")!, new ViewerPasswordSealer(key.PublicKey)); Assert.Null(reason); Assert.NotNull(row); Assert.Equal("sql", row!.Auth); Assert.Equal("monitor", row.Username); Assert.NotNull(row.EncryptedPassword); - /* The service must be able to read what the migrate wrote. */ - Assert.Equal("p@ss", DarlingSecrets.Unprotect(row.EncryptedPassword!)); + /* #5366: the migrate seals; the service opens it with its key, for this row's settings, and no DPAPI blob is written. */ + Assert.StartsWith(PasswordSeal.V1Prefix, row.EncryptedPassword, StringComparison.Ordinal); + Assert.Equal("p@ss", PasswordSeal.Open(row.EncryptedPassword!, key, ViewerPasswordSealer.BindingOf(row))); + } + + [Fact] + public void Projection_SqlEntry_WithoutAKey_IsLeftForALaterRun() + { + using var fixture = new Fixture(); + fixture.ServerStore.AddServer( + new ViewerServerEntry { ServerName = "SQL2019", DisplayName = "Dev", AuthenticationType = AuthenticationTypes.SqlServer }, + "monitor", "p@ss"); + + var (row, reason) = fixture.Migration.TryProjectEntry(fixture.ServerStore.GetByServerName("SQL2019")!); + + Assert.Null(row); + Assert.Equal(ViewerServerMigration.NoPasswordKeyReason, reason); } [Fact] @@ -510,14 +497,16 @@ public void Projection_ServicePrincipal_SkipsWithoutSecret_ProjectsWithOne() withSecret.ServerStore.AddServer( new ViewerServerEntry { ServerName = "azure", DisplayName = "Azure", AuthenticationType = AuthenticationTypes.ServicePrincipal }, "app-client-id", "the-client-secret"); - var (row, reason) = withSecret.Migration.TryProjectEntry(withSecret.ServerStore.GetAllServers()[0]); + using var key = PasswordPrivateKey.Generate(); + var (row, reason) = withSecret.Migration.TryProjectEntry( + withSecret.ServerStore.GetAllServers()[0], new ViewerPasswordSealer(key.PublicKey)); Assert.Null(reason); Assert.NotNull(row); Assert.Equal("serviceprincipal", row!.Auth); Assert.Equal("app-client-id", row.Username); Assert.NotNull(row.EncryptedPassword); - /* The service must be able to read the client secret the migrate wrote. */ - Assert.Equal("the-client-secret", DarlingSecrets.Unprotect(row.EncryptedPassword!)); + /* The service opens the client secret the migrate wrote, for this row's settings. */ + Assert.Equal("the-client-secret", PasswordSeal.Open(row.EncryptedPassword!, key, ViewerPasswordSealer.BindingOf(row))); } [Fact] @@ -729,13 +718,27 @@ public void BuildMonitoredServerRow_WindowsAuth_MapsToIntegrated_NoSecret() [Fact] public void BuildMonitoredServerRow_SqlAuth_MapsToSql_CarriesBlobAndUsername() { + using var key = PasswordPrivateKey.Generate(); var (row, _) = AddMultipleServersDialog.BuildMonitoredServerRow( Line("SQL2022"), - new BulkSharedSettings { AuthType = AuthenticationTypes.SqlServer, Username = "monitor", EncryptedPassword = "DPAPI-BLOB==" }); + new BulkSharedSettings { AuthType = AuthenticationTypes.SqlServer, Username = "monitor", Secret = "p@ss-not-real" }, + new ViewerPasswordSealer(key.PublicKey)); Assert.Equal("sql", row!.Auth); Assert.Equal("monitor", row.Username); - Assert.Equal("DPAPI-BLOB==", row.EncryptedPassword); + Assert.StartsWith(PasswordSeal.V1Prefix, row.EncryptedPassword, StringComparison.Ordinal); + Assert.Equal("p@ss-not-real", PasswordSeal.Open(row.EncryptedPassword!, key, ViewerPasswordSealer.BindingOf(row))); + } + + [Fact] + public void BuildMonitoredServerRow_APasswordWithoutAKey_IsRefusedWithTheKeySentence() + { + var (row, error) = AddMultipleServersDialog.BuildMonitoredServerRow( + Line("SQL2022"), + new BulkSharedSettings { AuthType = AuthenticationTypes.SqlServer, Username = "monitor", Secret = "p@ss-not-real" }); + + Assert.Null(row); + Assert.Equal(ViewerPasswordKey.NoKeyText, error); } [Theory] @@ -764,7 +767,8 @@ public void BuildMonitoredServerRow_NonInteractiveEntra_MapsThrough(string authT // client id in username, the client secret in the DPAPI blob for SP; MI carries neither). var (row, error) = AddMultipleServersDialog.BuildMonitoredServerRow( Line("azure.database.windows.net"), - new BulkSharedSettings { AuthType = authType, Username = "app-id", EncryptedPassword = "DPAPI-BLOB==" }); + new BulkSharedSettings { AuthType = authType, Username = "app-id", Secret = "p@ss-not-real" }, + new ViewerPasswordSealer(PasswordPrivateKey.Generate().PublicKey)); Assert.Null(error); Assert.NotNull(row); @@ -808,16 +812,31 @@ public void BuildMonitoredServerRow_CarriesEncryptTrustDatabase_AndDisplayFallba } [Fact] - public void SharedBlob_IsReusedAcrossRows_ProtectNotCalledPerRow() + public void MultiAddOfThreeRows_GivesThreeValues_EachOpeningOnlyWithItsOwnBinding() { - // The shared settings carry ONE pre-Protected blob (Protect is called once at resolve time); the mapping - // helper reuses it verbatim on every row rather than re-encrypting per server. - var shared = new BulkSharedSettings { AuthType = AuthenticationTypes.SqlServer, Username = "u", EncryptedPassword = "ONE-BLOB==" }; - var (a, _) = AddMultipleServersDialog.BuildMonitoredServerRow(Line("SQL2022"), shared); - var (b, _) = AddMultipleServersDialog.BuildMonitoredServerRow(Line("SQL2019"), shared); + // #5366: the one shared password is sealed once per row, for that row's own connection settings. Three rows give + // three different values, and each opens only with its own binding. + using var key = PasswordPrivateKey.Generate(); + var sealer = new ViewerPasswordSealer(key.PublicKey); + var shared = new BulkSharedSettings { AuthType = AuthenticationTypes.SqlServer, Username = "u", Secret = "p@ss-not-real" }; + var rows = new[] { "alpha-sql", "beta-sql", "gamma-sql" } + .Select(host => AddMultipleServersDialog.BuildMonitoredServerRow(Line(host), shared, sealer).Row!) + .ToArray(); - Assert.Equal("ONE-BLOB==", a!.EncryptedPassword); - Assert.Same(a.EncryptedPassword, b!.EncryptedPassword); + Assert.Equal(3, rows.Select(r => r.EncryptedPassword).Distinct().Count()); + for (var i = 0; i < rows.Length; i++) + { + Assert.Equal("p@ss-not-real", PasswordSeal.Open(rows[i].EncryptedPassword!, key, ViewerPasswordSealer.BindingOf(rows[i]))); + for (var j = 0; j < rows.Length; j++) + { + if (i != j) + { + var other = ViewerPasswordSealer.BindingOf(rows[j]); + var ex = Assert.Throws(() => PasswordSeal.Open(rows[i].EncryptedPassword!, key, other)); + Assert.Equal(PasswordSealFailure.BindingOrTamper, ex.Kind); + } + } + } } [Fact] @@ -825,7 +844,8 @@ public void BuildTestConnectServer_ProjectsTheSameFields() { var (test, error) = AddMultipleServersDialog.BuildTestConnectServer( Line("SQL2022", "Prod", "tpcc"), - new BulkSharedSettings { AuthType = AuthenticationTypes.SqlServer, Username = "monitor", EncryptedPassword = "BLOB==", EncryptMode = "Strict", TrustServerCertificate = true }); + new BulkSharedSettings { AuthType = AuthenticationTypes.SqlServer, Username = "monitor", Secret = "p@ss-not-real", EncryptMode = "Strict", TrustServerCertificate = true }, + new ViewerPasswordSealer(PasswordPrivateKey.Generate().PublicKey)); Assert.Null(error); Assert.NotNull(test); @@ -834,7 +854,7 @@ public void BuildTestConnectServer_ProjectsTheSameFields() Assert.Equal("tpcc", test.Database); Assert.Equal("sql", test.Auth); Assert.Equal("monitor", test.Username); - Assert.Equal("BLOB==", test.EncryptedPassword); + Assert.StartsWith(PasswordSeal.V1Prefix, test.EncryptedPassword, StringComparison.Ordinal); Assert.Equal("Strict", test.EncryptMode); Assert.True(test.TrustServerCertificate); } @@ -852,7 +872,8 @@ public void BuildTestConnectServer_InteractiveEntra_IsRejected_ServicePrincipalM // ...while a service principal builds the probe (Auth mapped to serviceprincipal). var (test, error) = AddMultipleServersDialog.BuildTestConnectServer( Line("azure"), - new BulkSharedSettings { AuthType = AuthenticationTypes.ServicePrincipal, Username = "app-id", EncryptedPassword = "DPAPI-BLOB==" }); + new BulkSharedSettings { AuthType = AuthenticationTypes.ServicePrincipal, Username = "app-id", Secret = "p@ss-not-real" }, + new ViewerPasswordSealer(PasswordPrivateKey.Generate().PublicKey)); Assert.Null(error); Assert.NotNull(test); Assert.Equal("serviceprincipal", test!.Auth); diff --git a/Darling/Darling.Tests/ViewerServerSealLiveTests.cs b/Darling/Darling.Tests/ViewerServerSealLiveTests.cs new file mode 100644 index 0000000000..a0c050b9d4 --- /dev/null +++ b/Darling/Darling.Tests/ViewerServerSealLiveTests.cs @@ -0,0 +1,181 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/* #1776 own-store: each fact mints its own scratch database through ScratchPostgres and never touches another + test's rows, so it is deliberately NOT [Collection("live-postgres")]. */ + +/// +/// The viewer seals a server password for its connection settings and saves it (#5366), against a real store: the +/// value in the raw row, read back through , opens with the key +/// and the stored settings, and a change of how the server is reached with the stored value kept is refused. +/// +public sealed class ViewerServerSealLiveTests +{ + private const string Password = "p@ss-not-real"; + + private static string? BaseConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static MonitoredServerRow Row(bool trust) => new() + { + ServerId = ViewerDataService.ComputeServerId("alpha-live", null, false), + Name = "Alpha live", + Host = "alpha-live", + Auth = "sql", + Username = "monitor", + EncryptMode = "Mandatory", + TrustServerCertificate = trust, + }; + + private static async Task MigrateStoreAsync(ScratchPostgres scratch, CancellationToken ct) + { + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + } + + private static async Task PublishAsync(ScratchPostgres scratch, PasswordPrivateKey key, string serviceState, string? note, CancellationToken ct) + { + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await using (var keyCommand = new NpgsqlCommand( + "INSERT INTO config.password_key (key_id, public_key, algorithm, state) VALUES ($1, $2, $3, 'current')", connection)) + { + keyCommand.Parameters.Add(new NpgsqlParameter { Value = key.PublicKey.KeyId }); + keyCommand.Parameters.Add(new NpgsqlParameter { Value = key.PublicKey.Spki }); + keyCommand.Parameters.Add(new NpgsqlParameter { Value = PasswordSeal.Algorithm }); + await keyCommand.ExecuteNonQueryAsync(ct); + } + + await using var stateCommand = new NpgsqlCommand( + "INSERT INTO config.password_key_service (service_host, key_id, state, note, updated_at) " + + "VALUES ('alpha-host', $1, $2, $3, now() AT TIME ZONE 'UTC')", connection); + stateCommand.Parameters.Add(new NpgsqlParameter { Value = key.PublicKey.KeyId }); + stateCommand.Parameters.Add(new NpgsqlParameter { Value = serviceState }); + stateCommand.Parameters.Add(new NpgsqlParameter { Value = (object?)note ?? DBNull.Value }); + await stateCommand.ExecuteNonQueryAsync(ct); + } + + private static async Task<(ServerConnectionIdentity Identity, string Stored)> ReadRawRowAsync( + ScratchPostgres scratch, int serverId, CancellationToken ct) + { + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await using var command = new NpgsqlCommand( + "SELECT " + ServerConnectionIdentity.StoredColumns + ", encrypted_password FROM config.config_monitored_servers WHERE server_id = $1", + connection); + command.Parameters.Add(new NpgsqlParameter { Value = serverId }); + await using var reader = await command.ExecuteReaderAsync(ct); + Assert.True(await reader.ReadAsync(ct)); + var identity = ServerConnectionIdentity.FromStoredColumns( + reader.IsDBNull(0) ? null : reader.GetString(0), reader.IsDBNull(1) ? null : reader.GetInt32(1), + reader.IsDBNull(2) ? null : reader.GetString(2), reader.IsDBNull(3) ? null : reader.GetString(3), + !reader.IsDBNull(4) && reader.GetBoolean(4), reader.IsDBNull(5) ? null : reader.GetString(5), + reader.IsDBNull(6) ? null : reader.GetString(6), reader.IsDBNull(7) ? null : reader.GetString(7), + !reader.IsDBNull(8) && reader.GetBoolean(8), !reader.IsDBNull(9) && reader.GetBoolean(9)); + return (identity, reader.GetString(10)); + } + + [Fact] + public async Task AddAndEdit_SealForTheStoredSettings_AndAMovedReachKeepsNoStoredValue() + { + var baseConnectionString = BaseConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to run the live seal round trip."); + + var ct = TestContext.Current.CancellationToken; + var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var directory = Directory.CreateTempSubdirectory("darling-seal-live-").FullName; + using var key = PasswordPrivateKey.Generate(); + var bodySucceeded = false; + try + { + await MigrateStoreAsync(scratch, ct); + await PublishAsync(scratch, key, "ok", null, ct); + await using var viewer = new ViewerDataService(scratch.ConnectionString); + + var (published, state) = await viewer.GetPublishedPasswordKeyAsync(ct); + var decision = ViewerPasswordKey.Evaluate(published, state, viewer.StoreConnectionIdentity, new ViewerPasswordKeyPins(Path.Combine(directory, "pins.json"))); + Assert.NotNull(decision.Key); + var sealer = new ViewerPasswordSealer(decision.Key!); + + /* Add: the value in the raw row opens with the key and the settings the row stores. */ + var row = Row(trust: false); + row.EncryptedPassword = sealer.Seal(Password, row); + Assert.Equal(MonitoredServerAddOutcome.Added, (await viewer.AddMonitoredServerAsync(row, ct)).Outcome); + var (identity, stored) = await ReadRawRowAsync(scratch, row.ServerId, ct); + Assert.Equal(Password, PasswordSeal.Open(stored, key, PasswordBinding.ForServer(identity))); + + /* Edit that changes only the trust setting and keeps the stored value: refused, nothing written. */ + var kept = Row(trust: true); + kept.EncryptedPassword = stored; + await Assert.ThrowsAsync(() => viewer.UpsertMonitoredServerAsync(kept, ct)); + Assert.Equal(stored, (await ReadRawRowAsync(scratch, row.ServerId, ct)).Stored); + + /* The same edit with the password typed again seals for the new settings. */ + var moved = Row(trust: true); + moved.EncryptedPassword = sealer.Seal(Password, moved); + await viewer.UpsertMonitoredServerAsync(moved, ct); + var (movedIdentity, movedStored) = await ReadRawRowAsync(scratch, row.ServerId, ct); + Assert.True(movedIdentity.TrustServerCertificate); + Assert.Equal(Password, PasswordSeal.Open(movedStored, key, PasswordBinding.ForServer(movedIdentity))); + Assert.Throws(() => PasswordSeal.Open(movedStored, key, PasswordBinding.ForServer(identity))); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => await Task.CompletedTask); + await scratch.DisposeAsync(); + try { Directory.Delete(directory, recursive: true); } catch { /* best-effort */ } + } + } + + [Fact] + public async Task AServiceStateThatIsNotOk_RefusesWithItsNote_OnARealStore() + { + var baseConnectionString = BaseConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to run the live seal round trip."); + + var ct = TestContext.Current.CancellationToken; + var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var directory = Directory.CreateTempSubdirectory("darling-seal-live-").FullName; + using var key = PasswordPrivateKey.Generate(); + var bodySucceeded = false; + try + { + await MigrateStoreAsync(scratch, ct); + await PublishAsync(scratch, key, "missing", "The key file was not found.", ct); + await using var viewer = new ViewerDataService(scratch.ConnectionString); + + var (published, state) = await viewer.GetPublishedPasswordKeyAsync(ct); + var decision = ViewerPasswordKey.Evaluate(published, state, viewer.StoreConnectionIdentity, new ViewerPasswordKeyPins(Path.Combine(directory, "pins.json"))); + + Assert.Null(decision.Key); + Assert.Contains("The key file was not found.", decision.Refusal, StringComparison.Ordinal); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => await Task.CompletedTask); + await scratch.DisposeAsync(); + try { Directory.Delete(directory, recursive: true); } catch { /* best-effort */ } + } + } +} diff --git a/Darling/Darling.Tests/ViewerServerSetSyncTests.cs b/Darling/Darling.Tests/ViewerServerSetSyncTests.cs index 21bbc431fa..8b66c1800e 100644 --- a/Darling/Darling.Tests/ViewerServerSetSyncTests.cs +++ b/Darling/Darling.Tests/ViewerServerSetSyncTests.cs @@ -33,6 +33,12 @@ namespace Darling.Tests; /// local add or remove ends with, a server removed elsewhere gets the local remove's own cleanup, and every /// surface that lists servers reads the fleet that reload rebuilds. The behavior tests drive the comparer /// that decides whether to reload at all, against a real FleetView, and count the reloads. +/// +/// The pickers keep their servers. The reload also rebuilds the Recommendations and FinOps tabs' +/// own server pickers, and the favorites re-sort rebuilds the sidebar. Neither is the user choosing a server, +/// so each picker keeps its server while that server exists. Only a picker whose server is gone falls back +/// to the sidebar's server, as on a fresh load. Every reload runs this path: a local add, edit or remove, and +/// the sync above, all through LoadServersAsync(preserveSelection: true). /// public sealed class ViewerServerSetSyncTests { @@ -164,7 +170,11 @@ public void EverySurfaceThatListsServers_ReadsTheFleetTheReloadRebuilds() Assert.Matches(@"\bservers\s*=\s*_fleet\s*\.\s*All\b", overview); Assert.Matches(@"\blist\s*=\s*servers\s*\.\s*ToList\s*\(", overview); - Assert.Matches(@"totalServerCount\s*:\s*list\s*\.\s*Count\b", overview); + /* The roll-up goes through OverviewCardView.BuildRollup (the Overview search narrows it, D9): its registered + count is still list.Count, and the count a search-box change rebuilds the roll-up with (the + _overviewRegisteredCount field) is stored from that same list.Count. */ + Assert.Matches(new Regex(@"\bBuildRollup\s*\([^;]*,\s*list\s*\.\s*Count\s*,", RegexOptions.Singleline), overview); + Assert.Matches(@"\b_overviewRegisteredCount\s*=\s*list\s*\.\s*Count\b", overview); /* The database state editor lists the servers Settings was handed, and Settings is handed the fleet when it opens, so an editor opened after the reload lists the reloaded set. */ @@ -178,6 +188,140 @@ public void EverySurfaceThatListsServers_ReadsTheFleetTheReloadRebuilds() Assert.Matches(@"new\s+DatabaseStateOverridesWindow\s*\(\s*_dataService\s*,\s*_servers\s*\)", settings); } + [Fact] + public void Reload_RebuildsTheRecommendationsPicker_KeepingItsServerById() + { + var load = MethodBody("MainWindow.xaml.cs", "LoadServersAsync"); + + /* The old rebuild went back to the first server by index, whatever the picker showed. */ + Assert.DoesNotMatch(@"RecommendationsServerSelector\s*\.\s*SelectedIndex\s*=", load); + + var previous = Regex.Match(load, @"RecommendationsServerSelector\s*\.\s*SelectedItem\s+as\s+DarlingServer\b"); + var rebuild = Regex.Match(load, @"RecommendationsServerSelector\s*\.\s*ItemsSource\s*="); + + Assert.True( + previous.Success && rebuild.Success && previous.Index < rebuild.Index, + "LoadServersAsync must read the picker's server before it rebuilds the picker's list"); + Assert.Matches( + @"RecommendationsServerSelector\s*\.\s*SelectedItem\s*=\s*ViewerServerSetSync\s*\.\s*PickerSelectionAfterReload\s*\(", + load); + } + + [Fact] + public void Reload_AndTheFavoritesResort_RestoreTheSidebar_WithoutMovingThePickers() + { + foreach (var (file, method) in new[] + { + ("MainWindow.xaml.cs", "LoadServersAsync"), + ("MainWindow.ServerManagement.cs", "ReapplyFavoritesToServerList"), + }) + { + var body = MethodBody(file, method); + + Assert.Matches(@"\bRestoreSidebarSelection\s*\(", body); + Assert.DoesNotMatch(@"ServerList\s*\.\s*SelectedItem\s*=", body); + } + + /* An unguarded restore runs ServerList_SelectionChanged, and that moves both pickers to the + sidebar's server: SyncAggregateServerSelectors. */ + var restore = MethodBody("MainWindow.xaml.cs", "RestoreSidebarSelection"); + var guard = Regex.Match(restore, @"_suppressSidebarSelection\s*=\s*true\b"); + var select = Regex.Match(restore, @"ServerList\s*\.\s*SelectedItem\s*="); + + Assert.True( + guard.Success && select.Success && guard.Index < select.Index, + "RestoreSidebarSelection must set the sidebar's selection under _suppressSidebarSelection"); + Assert.DoesNotMatch(@"\bSyncAggregateServerSelectors\s*\(", restore); + + /* The guard also skips the handler's reload of the visible tab, so the restore runs it. */ + Assert.Matches(@"\bRefreshVisibleAsync\s*\(", restore); + } + + [Fact] + public void FinOpsPicker_KeepsItsServer_AndFallsBackToTheSidebarsServer_AtBothOfItsRebuilds() + { + var setServers = MethodBody("FinOpsTab.xaml.cs", "SetServers"); + + Assert.Matches( + @"ServerSelector\s*\.\s*SelectedItem\s*=\s*FinOpsServerChoice\s*\.\s*Selection\s*\(", + setServers); + + /* The reload hands it the list, and so does the FinOps tab when it opens. Both pass the sidebar's + server, for the fallback. */ + var shell = Stripped("MainWindow.xaml.cs"); + var calls = Regex.Matches(shell, @"FinOpsContent\s*\.\s*SetServers\s*\("); + + Assert.Equal(2, calls.Count); + foreach (Match call in calls) + { + var open = call.Index + call.Length - 1; + var arguments = shell[open..(ClosingParen(shell, open) + 1)]; + + Assert.Matches(@",\s*[^,]*ServerId\b", arguments); + } + } + + [Fact] + public void Reload_ReadsTheServersItKeeps_AfterItsAwait() + { + var load = MethodBody("MainWindow.xaml.cs", "LoadServersAsync"); + + /* A change made while the server list loads is the user's latest choice. Read before the await, the + reload put back the server that the sidebar or the picker showed when the load began. */ + var loaded = Regex.Match(load, @"\bawait\s+_dataService\s*\.\s*GetManagedServersAsync\s*\("); + var rebind = Regex.Match(load, @"ServerList\s*\.\s*ItemsSource\s*="); + + Assert.True(loaded.Success && rebind.Success, "LoadServersAsync must await the server list, then rebind the sidebar"); + + foreach (var (what, pattern) in new[] + { + ("the sidebar's server", @"ServerList\s*\.\s*SelectedItem\s+as\s+FleetServerRow\b"), + ("the Recommendations picker's server", @"RecommendationsServerSelector\s*\.\s*SelectedItem\s+as\s+DarlingServer\b"), + }) + { + var read = Regex.Match(load, pattern); + + Assert.True( + read.Success && loaded.Index < read.Index && read.Index < rebind.Index, + $"LoadServersAsync must read {what} after its await and before it rebinds the sidebar"); + } + } + + [Fact] + public void EverySidebarSelectionWrite_RunsUnderTheSuppressionGuard() + { + /* An unguarded write runs ServerList_SelectionChanged, and that moves both pickers to the sidebar's + server. Each write must follow a "_suppressSidebarSelection = true" with no "= false" between them, as + RestoreSidebarSelection does. */ + var viewer = Path.GetDirectoryName(ViewerPath("MainWindow.xaml.cs"))!; + var writes = 0; + var unguarded = new List(); + + foreach (var path in Directory.EnumerateFiles(viewer, "*.cs", SearchOption.AllDirectories) + .Where(path => !path.Contains($"{Path.DirectorySeparatorChar}bin{Path.DirectorySeparatorChar}", StringComparison.Ordinal) + && !path.Contains($"{Path.DirectorySeparatorChar}obj{Path.DirectorySeparatorChar}", StringComparison.Ordinal))) + { + var code = CSharpSourceWalker.StripCommentsAndStrings(File.ReadAllText(path)); + var toggles = Regex.Matches(code, @"\b_suppressSidebarSelection\s*=\s*(?true|false)\b"); + + foreach (Match write in Regex.Matches(code, @"\bServerList\s*\.\s*Selected(?:Item|Index|Value)\s*=(?!=)")) + { + writes++; + + var last = toggles.LastOrDefault(toggle => toggle.Index < write.Index); + + if (last is null || last.Groups["on"].Value != "true") + { + var line = code.AsSpan(0, write.Index).Count('\n') + 1; + unguarded.Add($"{Path.GetFileName(path)}:{line}"); + } + } + } + + Assert.True(writes > 0, "the scan found no sidebar selection write, so its pattern no longer matches the viewer"); + Assert.True(unguarded.Count == 0, "a sidebar selection write outside the guard: " + string.Join(", ", unguarded)); + } + // ── Behavior: the comparer against a real FleetView, counting the reloads ────────────────────── [Fact] @@ -275,6 +419,48 @@ public async Task OneServerSwappedForAnother_SameCount_StillReloads() Assert.Equal(1, tick.SelectedId); } + // ── Behavior: the server each picker shows after a rebuild ────────────────────────────────────── + + [Fact] + public void Picker_KeepsItsServer_WhileItExists_WhereverTheSidebarIs() + { + var servers = new[] { Server(1, "SQL01"), Server(2, "SQL02"), Server(3, "SQL03") }; + + var picked = ViewerServerSetSync.PickerSelectionAfterReload(servers, previousServerId: 2, sidebarServerId: 1); + + Assert.Equal(2, picked?.ServerId); + } + + [Fact] + public void Picker_WhoseServerWasRemoved_FollowsTheSidebar_AsOnAFreshLoad() + { + var servers = new[] { Server(1, "SQL01"), Server(3, "SQL03") }; + + var picked = ViewerServerSetSync.PickerSelectionAfterReload(servers, previousServerId: 2, sidebarServerId: 3); + + Assert.Equal(3, picked?.ServerId); + } + + [Fact] + public void FreshLoad_StartsThePicker_OnTheSidebarsServer() + { + var servers = new[] { Server(1, "SQL01"), Server(2, "SQL02") }; + + var picked = ViewerServerSetSync.PickerSelectionAfterReload(servers, previousServerId: null, sidebarServerId: 2); + + Assert.Equal(2, picked?.ServerId); + } + + [Fact] + public void NothingToKeepOrFollow_TakesTheFirstServer_AndAnEmptyListSelectsNothing() + { + var servers = new[] { Server(1, "SQL01"), Server(3, "SQL03") }; + + Assert.Equal(1, ViewerServerSetSync.PickerSelectionAfterReload(servers, previousServerId: 2, sidebarServerId: null)?.ServerId); + Assert.Equal(1, ViewerServerSetSync.PickerSelectionAfterReload(servers, previousServerId: 2, sidebarServerId: 9)?.ServerId); + Assert.Null(ViewerServerSetSync.PickerSelectionAfterReload(Array.Empty(), previousServerId: 2, sidebarServerId: 1)); + } + private static DarlingServer Server(int id, string name) => new(id, name, name, isEnabled: true, sqlMajorVersion: 16); diff --git a/Darling/Darling.Tests/ViewerServerTabCapabilityPinTests.cs b/Darling/Darling.Tests/ViewerServerTabCapabilityPinTests.cs index 140e6f7abc..155e3f7276 100644 --- a/Darling/Darling.Tests/ViewerServerTabCapabilityPinTests.cs +++ b/Darling/Darling.Tests/ViewerServerTabCapabilityPinTests.cs @@ -32,6 +32,7 @@ namespace Darling.Tests; /// PM_PIN_REGEN=1. Text-scans SOURCE (XAML + .cs) located from this file's compile-time path — NO WPF /// / Npgsql / assembly load, like the other Darling parity pins (see ThemeCompletenessTests). /// +[Trait("Stage", "Guard")] public sealed class ViewerServerTabCapabilityPinTests { [Fact] diff --git a/Darling/Darling.Tests/ViewerSettingsFileGuardTests.cs b/Darling/Darling.Tests/ViewerSettingsFileGuardTests.cs index 32b73b2237..5733c4a130 100644 --- a/Darling/Darling.Tests/ViewerSettingsFileGuardTests.cs +++ b/Darling/Darling.Tests/ViewerSettingsFileGuardTests.cs @@ -34,6 +34,7 @@ namespace Darling.Tests; /// So the pins below come in pairs — what the store now SAYS about each of the three states, and what /// it leaves on disk. The second is the one that turns an annoyance into data loss. /// +[Trait("Stage", "Guard")] public sealed class ViewerSettingsFileGuardTests : IDisposable { /// A viewer-settings.json a user would recognize: real keys, one trailing comma. diff --git a/Darling/Darling.Tests/ViewerShortWindowDataStartTests.cs b/Darling/Darling.Tests/ViewerShortWindowDataStartTests.cs new file mode 100644 index 0000000000..cad0bbd0be --- /dev/null +++ b/Darling/Darling.Tests/ViewerShortWindowDataStartTests.cs @@ -0,0 +1,249 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text.RegularExpressions; +using System.Threading; +using System.Threading.Tasks; +using System.Windows; +using System.Windows.Controls; +using Npgsql; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using PerformanceMonitor.Ui; +using Xunit; + +namespace Darling.Tests; + +/// +/// The "Showing since" notice on a range of an hour or less (#4966). The 90-minute slack belongs to the coverage probe alone, where it +/// absorbs a first collection that lands a little after the window starts: a window no longer than the slack can never get a coverage +/// note, so its probe starts no query. A read that filled its row cap dropped rows for certain, so its verdict has no slack: on a short +/// range the banner shows whenever the oldest row shown is later than the window's start. +/// +/* The banner's time text reads the process-wide display mode, which these tests set (restored in Dispose); one shared + collection serializes it with every other class that flips the viewer's time statics. */ +[Collection("viewer-time-statics")] +public sealed class ViewerShortWindowDataStartTests : IDisposable +{ + private readonly TimeDisplayMode _savedMode = ViewerTimeHelper.CurrentDisplayMode; + + public void Dispose() => ViewerTimeHelper.CurrentDisplayMode = _savedMode; + + private static readonly DateTime RangeStart = new(2026, 9, 1, 0, 0, 0, DateTimeKind.Utc); + + private const int Cap = ViewerDataService.PlanCorrectionsRowCap; + + /* A store nothing listens on: a probe that starts a query against it fails, one that does not returns at once. */ + private const string UnreachableStore = "Host=127.0.0.1;Port=1;Username=nobody;Database=nothing;Timeout=3;Pooling=false"; + + private static DateTime At(int minutes) => RangeStart.AddMinutes(minutes); + + /* A page of `count` rows 10 seconds apart from `oldest`: 200 of them span 33 minutes, so a page that starts inside the first + half hour of a one-hour range stays inside it. */ + private static IEnumerable Page(DateTime oldest, int count) => Enumerable.Range(0, count).Select(i => oldest.AddSeconds(10 * i)); + + // ── A capped verdict gets no slack ── + + /* A capped read on a one-hour range whose oldest row came 20 minutes in: the grid reaches back no further, so the banner names that + row. Before, the slack hid it, and on a range this short no row could ever sit past it. */ + [Fact] + public void ACappedRead_OnAOneHourRange_NamesItsOldestRow_WhenItIsLaterThanTheStart() + { + Assert.Equal("Showing since 2026-09-01 00:20:00", BannerFor(coverageStartUtc: null, Page(At(20), Cap), Cap)); + /* The same rows with a coverage answer that reaches the range start: the cap is what names the row. */ + Assert.Equal("Showing since 2026-09-01 00:20:00", BannerFor(At(-600), Page(At(20), Cap), Cap)); + } + + /* A capped read whose oldest row is at the window's start reaches the start: no banner. */ + [Fact] + public void ACappedRead_WhoseOldestRowIsAtTheStart_ShowsNoBanner() + { + Assert.Null(BannerFor(coverageStartUtc: null, Page(At(0), Cap), Cap)); + Assert.Null(BannerFor(At(-600), Page(At(0), Cap), Cap)); + } + + /* The note names its time to the second. A capped read has no slack, so a read whose oldest row came 42 seconds into the range's + first minute shows a banner, and a note in whole minutes would print the range's own start minute for it (10:15 for a range from + 10:15:00), which reads as if the grid started at the range's start. */ + [Fact] + public void ACappedRead_WhoseOldestRowIsSecondsPastTheStart_NamesTheSecond_NotTheStartsMinute() + { + var rangeStart = new DateTime(2026, 9, 1, 10, 15, 0, DateTimeKind.Utc); + var oldest = rangeStart.AddSeconds(42); + + Assert.Equal("Showing since 2026-09-01 10:15:42", BannerFor(coverageStartUtc: null, Page(oldest, Cap), Cap, rangeStart)); + /* Under a coverage answer that reaches the start, the cap still names the row, to the second. */ + Assert.Equal("Showing since 2026-09-01 10:15:42", BannerFor(rangeStart.AddHours(-10), Page(oldest, Cap), Cap, rangeStart)); + } + + /* A read under its cap keeps its verdict, and the slack stays on the coverage probe: a first collection 85 minutes in is + absorbed, one 95 minutes in is named. */ + [Fact] + public void TheSlack_StaysOnTheCoverageProbe() + { + Assert.Null(BannerFor(At(85), Page(At(86), 3), Cap)); + Assert.Equal("Showing since 2026-09-01 01:35:00", BannerFor(At(95), Page(At(96), 3), Cap)); + /* A page one row under the cap is not capped. */ + Assert.Null(BannerFor(coverageStartUtc: null, Page(At(20), Cap - 1), Cap)); + } + + // ── A window no longer than the slack starts no probe ── + + /* Every window of an hour up to the slack itself returns at once, with no query against a store nothing listens on; one minute over + the slack starts the query, which fails there. */ + [Theory] + [InlineData(60)] + [InlineData(90)] + public async Task AWindowNoLongerThanTheSlack_StartsNoProbe_AgainstNoStore(int minutes) + { + var ct = TestContext.Current.CancellationToken; + await using var source = NpgsqlDataSource.Create(UnreachableStore); + + var answer = await DataWindowFloor.GetForServerAsync( + source, DataWindowFloor.Source.ForCollectorTable("plan_correction"), 1, RangeStart, RangeStart.AddMinutes(minutes), 3, ct); + + Assert.Null(answer); + } + + [Fact] + public async Task AWindowOneMinuteOverTheSlack_StartsItsProbe_AgainstNoStore() + { + var ct = TestContext.Current.CancellationToken; + await using var source = NpgsqlDataSource.Create(UnreachableStore); + + await Assert.ThrowsAnyAsync(() => DataWindowFloor.GetForServerAsync( + source, DataWindowFloor.Source.ForCollectorTable("plan_correction"), 1, RangeStart, RangeStart.AddMinutes(91), 3, ct)); + } + + /* The tab's own probes go through the shared form, so each skips a one-hour window; the Regressions probe asks about the baseline + window, 7 days before the range, so a one-hour range still starts its probe. */ + [Fact] + public async Task TheTabsProbes_SkipAOneHourWindow_AgainstNoStore() + { + var ct = TestContext.Current.CancellationToken; + await using var viewer = new ViewerDataService(UnreachableStore); + var end = RangeStart.AddHours(1); + + Assert.Null(await viewer.GetPlanCorrectionsDataStartAsync(1, RangeStart, end, ct)); + Assert.Null(await viewer.GetQueryHeatmapDataStartAsync(1, RangeStart, end, ct)); + Assert.Null(await viewer.GetBlockedProcessReportsDataStartAsync(1, RangeStart, end, ct)); + Assert.Null(await viewer.GetDeadlocksDataStartAsync(1, RangeStart, end, ct)); + Assert.Null(await viewer.GetLongQueriesDataStartAsync(1, RangeStart, end, ct)); + Assert.Null(await viewer.GetSystemHealthEventsDataStartAsync(1, RangeStart, end, ct)); + await Assert.ThrowsAnyAsync(() => viewer.GetQueryStoreRegressionsDataStartAsync(1, RangeStart, end, ct)); + } + + // ── The Queries tab's three floor probes ── + + /* One of the Queries tab's three raw-table floor probes, by the table it reads, asked through the viewer's own method. */ + private static Task QueriesTabFloorAsync(ViewerDataService viewer, string table, DateTime startUtc, DateTime endUtc, CancellationToken ct) => table switch + { + "query_stats" => viewer.GetQueryStatsWindowFloorAsync(1, startUtc, endUtc, ct), + "procedure_stats" => viewer.GetProcedureStatsWindowFloorAsync(1, startUtc, endUtc, ct), + "query_store_stats" => viewer.GetQueryStoreWindowFloorAsync(1, startUtc, endUtc, ct), + _ => throw new ArgumentOutOfRangeException(nameof(table), table, "not one of the Queries tab's floor probes"), + }; + + /* The same three reads as Darling's MCP tools make them, through DarlingDataReader and the shared probe. */ + private static Task McpFloorAsync(NpgsqlDataSource source, string table, DateTime startUtc, DateTime endUtc, CancellationToken ct) => table switch + { + "query_stats" => DarlingDataReader.GetQueryStatsWindowFloorAsync(source, 1, startUtc, endUtc, ct), + "procedure_stats" => DarlingDataReader.GetProcedureStatsWindowFloorAsync(source, 1, startUtc, endUtc, ct), + "query_store_stats" => DarlingDataReader.GetQueryStoreWindowFloorAsync(source, 1, startUtc, endUtc, ct), + _ => throw new ArgumentOutOfRangeException(nameof(table), table, "not one of the MCP floor reads"), + }; + + /* The Query Stats, Procedure Stats and Query Store probes answer a window of an hour up to the slack itself at once, with no + query against a store nothing listens on: the answer sits at or before the window's end, so no coverage note can show for it. */ + [Theory] + [InlineData("query_stats", 60)] + [InlineData("query_stats", 90)] + [InlineData("procedure_stats", 60)] + [InlineData("procedure_stats", 90)] + [InlineData("query_store_stats", 60)] + [InlineData("query_store_stats", 90)] + public async Task AQueriesTabFloorProbe_OverAWindowNoLongerThanTheSlack_StartsNoQuery_AgainstNoStore(string table, int minutes) + { + var ct = TestContext.Current.CancellationToken; + await using var viewer = new ViewerDataService(UnreachableStore); + + Assert.Null(await QueriesTabFloorAsync(viewer, table, RangeStart, RangeStart.AddMinutes(minutes), ct)); + } + + /* One minute over the slack starts the query, which fails against that store: the skip stops at the slack. */ + [Theory] + [InlineData("query_stats")] + [InlineData("procedure_stats")] + [InlineData("query_store_stats")] + public async Task AQueriesTabFloorProbe_OneMinuteOverTheSlack_StartsItsQuery_AgainstNoStore(string table) + { + var ct = TestContext.Current.CancellationToken; + await using var viewer = new ViewerDataService(UnreachableStore); + + await Assert.ThrowsAnyAsync(() => QueriesTabFloorAsync(viewer, table, RangeStart, RangeStart.AddMinutes(91), ct)); + } + + /* Darling's MCP tools read the same shared probe's null as "nothing was read", so the skip is the viewer's alone: an MCP floor + read over a short window still queries the store, and fails against one nothing listens on. */ + [Theory] + [InlineData("query_stats", 60)] + [InlineData("query_stats", 90)] + [InlineData("procedure_stats", 60)] + [InlineData("procedure_stats", 90)] + [InlineData("query_store_stats", 60)] + [InlineData("query_store_stats", 90)] + public async Task AnMcpFloorRead_OverAWindowNoLongerThanTheSlack_StillQueriesTheStore_AgainstNoStore(string table, int minutes) + { + var ct = TestContext.Current.CancellationToken; + await using var source = NpgsqlDataSource.Create(UnreachableStore); + + await Assert.ThrowsAnyAsync(() => McpFloorAsync(source, table, RangeStart, RangeStart.AddMinutes(minutes), ct)); + } + + /* Every floor read the Queries tab makes, three loads and three slicer re-reads of a sub-range, goes through those viewer + methods, so each carries the skip; none reaches the shared probe on its own. */ + [Fact] + public void EveryQueriesTabFloorRead_GoesThroughTheViewersMethod_NeverTheSharedProbe() + { + var tabs = string.Join("\n", Directory.GetFiles(RepoFile.PathTo("Darling", "PerformanceMonitor.Darling.Viewer"), "ViewerServerTab*.cs").Select(File.ReadAllText)); + + foreach (var method in new[] { "GetQueryStatsWindowFloorAsync", "GetProcedureStatsWindowFloorAsync", "GetQueryStoreWindowFloorAsync" }) + { + Assert.Equal(2, Regex.Matches(tabs, $@"_dataService\.{method}\(").Count); + } + + Assert.DoesNotContain("RawWindowFloor.GetAsync(", tabs, StringComparison.Ordinal); + } + + // ── The banner, read off the real control ── + + /* The banner raised for the probe's answer and the rows the grid shows on the range that starts at startUtc (RangeStart when + omitted), read off the control; null when it is hidden. Seeded visible, so a no-op cannot pass as a hidden banner. */ + private static string? BannerFor(DateTime? coverageStartUtc, IEnumerable shownTimes, int? rowCap, DateTime? startUtc = null) + { + string? text = null; + StaTestThread.Run(() => + { + ViewerTimeHelper.CurrentDisplayMode = TimeDisplayMode.UTC; + var banner = new TextBlock { Visibility = Visibility.Visible, Text = "stale" }; + + ViewerServerTab.ShowEventDataStartAsync( + banner, Task.FromResult(coverageStartUtc), "Short Window", startUtc ?? RangeStart, shownTimes.Select(t => (DateTime?)t), rowCap).GetAwaiter().GetResult(); + + text = banner.Visibility == Visibility.Visible ? banner.Text : null; + }); + return text; + } + + /* WPF objects require STA; same shape as ViewerPlanCorrectionsDataStartTests. */ +} diff --git a/Darling/Darling.Tests/ViewerSlicerRangeLabelTests.cs b/Darling/Darling.Tests/ViewerSlicerRangeLabelTests.cs index 9e9af1dca0..3638b17762 100644 --- a/Darling/Darling.Tests/ViewerSlicerRangeLabelTests.cs +++ b/Darling/Darling.Tests/ViewerSlicerRangeLabelTests.cs @@ -30,6 +30,7 @@ namespace Darling.Tests; /// must be Lite's call (same method, same instants, same format, so the same culture), and the text that call gives for /// the pair is checked on US Eastern (autumn change 2026-11-01 at 06:00 UTC). /// +[Trait("Reads", "Lite")] public sealed class ViewerSlicerRangeLabelTests { private static readonly ServerClock Eastern = ServerClock.Resolve("Eastern Standard Time", -300); diff --git a/Darling/Darling.Tests/ViewerSmtpSealTests.cs b/Darling/Darling.Tests/ViewerSmtpSealTests.cs new file mode 100644 index 0000000000..9b01c9da99 --- /dev/null +++ b/Darling/Darling.Tests/ViewerSmtpSealTests.cs @@ -0,0 +1,314 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.IO; +using System.Runtime.CompilerServices; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Collectors; +using PerformanceMonitor.Common; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// The Viewer seals the SMTP password (#5366): it is sealed for the host, port, SSL flag and user name it is stored with, +/// a blank password box keeps the saved value only while those four stay the same, and the service opens what the Viewer +/// stored. The sentences never carry a password. +/// +public sealed class ViewerSmtpSealTests : IDisposable +{ + private const string Plain = "p@ss-not-real"; + + private readonly PasswordPrivateKey _key = PasswordPrivateKey.Generate(); + + public void Dispose() => _key.Dispose(); + + private static NotificationRow Row( + string host = "mail.example.test", int port = 587, bool ssl = true, string? username = "mailer", string? blob = "sealed:v1:stored") => new() + { + SmtpHost = host, + SmtpPort = port, + SmtpUseSsl = ssl, + SmtpUsername = username, + SmtpEncryptedPassword = blob, + }; + + /* ═══════════ the re-entry rule ═══════════ */ + + [Fact] + public void ABlankPassword_WithAChangedHost_IsRefused_WithTheReenterSentence() + { + var action = ViewerSmtpSeal.Decide(Row(), null, Row(host: "other-mail.example.test"), ""); + + Assert.Equal(ViewerSmtpPasswordAction.Refuse, action); + Assert.Equal("Changing how mail is sent needs the SMTP password again.", ViewerSmtpSeal.ReenterText); + } + + [Theory] + [InlineData("port")] + [InlineData("ssl")] + [InlineData("username")] + [InlineData("username-cleared")] + public void ABlankPassword_WithAnyOtherChangedBoundValue_IsRefused(string changed) + { + var edited = changed switch + { + "port" => Row(port: 465), + "ssl" => Row(ssl: false), + "username" => Row(username: "someone-else"), + _ => Row(username: null), + }; + + Assert.Equal(ViewerSmtpPasswordAction.Refuse, ViewerSmtpSeal.Decide(Row(), null, edited, "")); + Assert.Equal(ViewerSmtpPasswordAction.Refuse, ViewerSmtpSeal.Decide(Row(), null, edited, null)); + } + + [Fact] + public void ABlankPassword_WithNothingChanged_KeepsTheSavedValue_AndAnEmptyUsernameIsTheSameAsNone() + { + Assert.Equal(ViewerSmtpPasswordAction.Keep, ViewerSmtpSeal.Decide(Row(), null, Row(), "")); + Assert.Equal(ViewerSmtpPasswordAction.Keep, ViewerSmtpSeal.Decide(Row(username: null), null, Row(username: ""), "")); + } + + [Fact] + public void ABlankPassword_WithAChangedHost_IsFine_WhenNoPasswordIsSaved() + { + Assert.Equal(ViewerSmtpPasswordAction.Keep, ViewerSmtpSeal.Decide(Row(blob: null), null, Row(blob: null, host: "other-mail.example.test"), "")); + Assert.Equal(ViewerSmtpPasswordAction.Keep, ViewerSmtpSeal.Decide(Row(blob: ""), null, Row(blob: "", port: 25), "")); + } + + [Fact] + public void ATypedPassword_IsSealed_WhateverChanged() + { + Assert.Equal(ViewerSmtpPasswordAction.Seal, ViewerSmtpSeal.Decide(Row(), null, Row(), Plain)); + Assert.Equal(ViewerSmtpPasswordAction.Seal, ViewerSmtpSeal.Decide(Row(), null, Row(host: "other-mail.example.test"), Plain)); + Assert.Equal(ViewerSmtpPasswordAction.Seal, ViewerSmtpSeal.Decide(Row(blob: null), null, Row(blob: null), Plain)); + } + + [Fact] + public void AValueThatThisMachineCouldRead_AndIsLeftAlone_IsNotSealedAgain_UnlessTheSettingsChanged() + { + Assert.Equal(ViewerSmtpPasswordAction.Keep, ViewerSmtpSeal.Decide(Row(blob: "old-format"), Plain, Row(blob: "old-format"), Plain)); + Assert.Equal(ViewerSmtpPasswordAction.Seal, ViewerSmtpSeal.Decide(Row(blob: "old-format"), Plain, Row(blob: "old-format", port: 25), Plain)); + Assert.Equal(ViewerSmtpPasswordAction.Seal, ViewerSmtpSeal.Decide(Row(blob: "old-format"), Plain, Row(blob: "old-format"), Plain + "x")); + } + + /* ═══════════ the seal ═══════════ */ + + [Fact] + public void ASealedSmtpPassword_OpensForItsOwnFourValues_AndNotAfterAnyOfThemChanges() + { + var sealer = new ViewerPasswordSealer(_key.PublicKey); + + var sealedText = sealer.SealSmtp(Plain, "mail.example.test", 587, true, "mailer"); + + Assert.StartsWith("sealed:v1:", sealedText, StringComparison.Ordinal); + Assert.DoesNotContain(Plain, sealedText, StringComparison.Ordinal); + Assert.Equal(Plain, PasswordSeal.Open(sealedText, _key, PasswordBinding.ForSmtp("mail.example.test", 587, true, "mailer"))); + foreach (var changed in new[] + { + PasswordBinding.ForSmtp("other-mail.example.test", 587, true, "mailer"), + PasswordBinding.ForSmtp("mail.example.test", 25, true, "mailer"), + PasswordBinding.ForSmtp("mail.example.test", 587, false, "mailer"), + PasswordBinding.ForSmtp("mail.example.test", 587, true, "someone-else"), + }) + { + Assert.Throws(() => PasswordSeal.Open(sealedText, _key, changed)); + } + } + + [Fact] + public void ASealedSmtpPassword_OpensWithTheBindingTheServiceBuildsFromTheStoredRow() + { + var sealer = new ViewerPasswordSealer(_key.PublicKey); + var row = Row(username: null, blob: null); + row.SmtpEncryptedPassword = sealer.SealSmtp(Plain, row.SmtpHost, row.SmtpPort, row.SmtpUseSsl, row.SmtpUsername); + + /* The service builds its SmtpConfig from the stored columns; an unset user name reads back as empty text. */ + var smtp = new SmtpConfig + { + Host = row.SmtpHost, Port = row.SmtpPort, UseSsl = row.SmtpUseSsl, Username = "", EncryptedPassword = row.SmtpEncryptedPassword, + }; + + Assert.Equal(Plain, DarlingSecrets.ResolveSmtpPassword(smtp, DarlingPasswordKey.FromPrivateKey(_key), LegacyDpapi.Current)); + } + + [Fact] + public void ALoneSurrogate_InThePassword_OrABoundField_IsRefused_WithASentenceThatNamesNoValue() + { + var sealer = new ViewerPasswordSealer(_key.PublicKey); + const string bad = "abcd\ud800"; + + var inPassword = Assert.Throws(() => sealer.SealSmtp(bad, "mail.example.test", 587, true, "mailer")); + var inHost = Assert.Throws(() => sealer.SealSmtp(Plain, bad, 587, true, "mailer")); + var inUser = Assert.Throws(() => sealer.SealSmtp(Plain, "mail.example.test", 587, true, bad)); + + Assert.Equal(ViewerPasswordSealer.PasswordCharactersText, inPassword.Message); + Assert.Equal("The mail server contains characters that cannot be stored.", inHost.Message); + Assert.Equal("The SMTP user name contains characters that cannot be stored.", inUser.Message); + foreach (var message in new[] { inPassword.Message, inHost.Message, inUser.Message }) + { + Assert.DoesNotContain("abcd", message, StringComparison.Ordinal); + } + + Assert.False(ViewerSmtpSeal.IsValidText(bad)); + Assert.True(ViewerSmtpSeal.IsValidText("a\U0001F600b")); + Assert.True(ViewerSmtpSeal.IsValidText(null)); + } + + [Fact] + public void APasswordLongerThanASealedValueCanHold_IsRefused_WithTheTooLongSentence() + { + var sealer = new ViewerPasswordSealer(_key.PublicKey); + + var ex = Assert.Throws( + () => sealer.SealSmtp(new string('x', PasswordSeal.MaxPlaintextBytes + 1), "mail.example.test", 587, true, "mailer")); + + Assert.Equal(ViewerPasswordSealer.PasswordTooLongText, ex.Message); + } + + /* ═══════════ the migration ═══════════ */ + + [Fact] + public async Task TheMigration_WithNoPasswordToMove_NeedsNoKey_AndLeavesTheRowWithoutOne() + { + var row = Row(); + + var outcome = await ViewerControlPlaneMigration.SealSmtpPasswordAsync(null!, row, "", TestContext.Current.CancellationToken); + + Assert.Equal(ViewerControlPlaneMigration.SmtpSealOutcome.Done, outcome); + Assert.Null(row.SmtpEncryptedPassword); + } + + /* ═══════════ the source: no old-format seal is left for SMTP ═══════════ */ + + [Fact] + public void NoViewerSmtpPath_StillCallsTheOldFormatSeal() + { + var viewer = Path.Combine(RepoRoot(), "Darling", "PerformanceMonitor.Darling.Viewer"); + foreach (var file in new[] { "SettingsWindow.xaml.cs", "ViewerControlPlaneMigration.cs" }) + { + var text = File.ReadAllText(Path.Combine(viewer, file)); + Assert.DoesNotContain("ViewerServerSecret.", text, StringComparison.Ordinal); + } + } + + [Fact] + public void TheSettingsWindow_SealsThroughTheKeyRead_AndThePathsWithNoWindowPassNoOwner() + { + var viewer = Path.Combine(RepoRoot(), "Darling", "PerformanceMonitor.Darling.Viewer"); + var settings = File.ReadAllText(Path.Combine(viewer, "SettingsWindow.xaml.cs")); + var migration = File.ReadAllText(Path.Combine(viewer, "ViewerControlPlaneMigration.cs")); + + Assert.Contains("ViewerPasswordKey.GetSealKeyAsync(_dataService!, this)", settings, StringComparison.Ordinal); + Assert.Contains("SealSmtp(", settings, StringComparison.Ordinal); + Assert.Contains("ViewerPasswordKey.GetSealKeyAsync(dataService, null, cancellationToken)", migration, StringComparison.Ordinal); + Assert.Contains("SealSmtp(", migration, StringComparison.Ordinal); + } + + private static string RepoRoot([CallerFilePath] string thisFile = "") + => Path.GetFullPath(Path.Combine(Path.GetDirectoryName(thisFile)!, "..", "..")); +} + +/* #1776 own-store: each fact mints its own scratch database through ScratchPostgres and never touches another + test's rows, so it is deliberately NOT [Collection("live-postgres")]. */ + +/// +/// #5366 against a real store: a sealed SMTP password the Viewer saves reads back exactly as written and is opened by the +/// service for the stored host, port, SSL flag and user name; a value sealed for one host does not open once the stored host +/// is changed underneath it. +/// +public sealed class ViewerSmtpSealLiveTests +{ + private const string Plain = "p@ss-not-real"; + + private static string? BaseConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + private static SmtpConfig ServiceReadOf(NotificationRow stored) => new() + { + Host = stored.SmtpHost, + Port = stored.SmtpPort, + UseSsl = stored.SmtpUseSsl, + Username = stored.SmtpUsername ?? "", + EncryptedPassword = stored.SmtpEncryptedPassword!, + }; + + [Fact] + public async Task ASavedSealedSmtpPassword_IsOpenedByTheServiceRead_AndNotAfterTheStoredHostChanges() + { + var baseConnectionString = BaseConnectionString; + Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString), "Set DARLING_TEST_PG to run the live SMTP seal round trip."); + + var ct = TestContext.Current.CancellationToken; + var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct); + var bodySucceeded = false; + try + { + await using (var connection = new NpgsqlConnection(scratch.ConnectionString)) + { + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + } + + await using var viewer = new ViewerDataService(scratch.ConnectionString); + using var key = PasswordPrivateKey.Generate(); + var ring = DarlingPasswordKey.FromPrivateKey(key); + + /* What the Settings window writes: the row's own four values, sealed. */ + var row = new NotificationRow + { + SmtpHost = "mail.example.test", SmtpPort = 465, SmtpUseSsl = false, SmtpUsername = "mailer", + SmtpFromAddress = "alerts@example.test", SmtpRecipients = "dba@example.test", + }; + row.SmtpEncryptedPassword = new ViewerPasswordSealer(key.PublicKey) + .SealSmtp(Plain, row.SmtpHost, row.SmtpPort, row.SmtpUseSsl, row.SmtpUsername); + await viewer.UpsertNotificationAsync(row, ct); + + var stored = await viewer.GetNotificationAsync(ct); + Assert.NotNull(stored); + Assert.DoesNotContain(Plain, stored!.SmtpEncryptedPassword, StringComparison.Ordinal); + Assert.Equal(Plain, DarlingSecrets.ResolveSmtpPassword(ServiceReadOf(stored), ring, LegacyDpapi.Current)); + + /* A direct store UPDATE of the host: the value was sealed for the old one. */ + await using (var connection = new NpgsqlConnection(scratch.ConnectionString)) + { + await connection.OpenAsync(ct); + await using var update = new NpgsqlCommand("UPDATE config_notification SET smtp_host = 'other-mail.example.test' WHERE id = 1", connection); + Assert.Equal(1, await update.ExecuteNonQueryAsync(ct)); + } + + var changed = await viewer.GetNotificationAsync(ct); + Assert.Equal("other-mail.example.test", changed!.SmtpHost); + var ex = Assert.Throws( + () => DarlingSecrets.ResolveSmtpPassword(ServiceReadOf(changed), ring, LegacyDpapi.Current)); + Assert.DoesNotContain(Plain, ex.Message, StringComparison.Ordinal); + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => await Task.CompletedTask); + await scratch.DisposeAsync(); + } + } + + [Fact] + public void TheSettingsWindow_NeverPutsASavedSmtpPasswordBackInTheBox() + { + var source = RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "SettingsWindow.xaml.cs"); + + Assert.DoesNotContain("TryUnprotect(", source, StringComparison.Ordinal); + Assert.DoesNotContain("_loadedSmtpPlain", source, StringComparison.Ordinal); + Assert.Contains("SmtpPasswordBox.Password = \"\";", source, StringComparison.Ordinal); + } +} diff --git a/Darling/Darling.Tests/ViewerSnapshotCollectedColumnTests.cs b/Darling/Darling.Tests/ViewerSnapshotCollectedColumnTests.cs new file mode 100644 index 0000000000..141585b18e --- /dev/null +++ b/Darling/Darling.Tests/ViewerSnapshotCollectedColumnTests.cs @@ -0,0 +1,85 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Globalization; +using System.Linq; +using System.Text.RegularExpressions; +using PerformanceMonitor.Darling.Viewer; +using PerformanceMonitor.Ui; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// The CPU Scheduler, Latch, Spinlock and Running Jobs grids end in a "Collected" column that says when their snapshot +/// was collected (#4966): the display-zone time to the second, sorted on the stored DateTime, as the viewer's other Collected columns. +[Collection("viewer-time-statics")] +public sealed class ViewerSnapshotCollectedColumnTests +{ + private static string Xaml() => + ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerServerTab.xaml").ReplaceLineEndings("\n"); + + private static string ViewerFile(string file) => + ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", file).ReplaceLineEndings("\n"); + + [Theory] + [InlineData("CpuSchedulerGrid")] + [InlineData("LatchStatsGrid")] + [InlineData("SpinlockStatsGrid")] + [InlineData("RunningJobsGrid")] + public void TheGrid_EndsInACollectedColumn_BoundToTheDisplayTime_SortedOnTheStoredDateTime(string grid) + { + var xaml = Xaml(); + var start = Regex.Match(xaml, @"]*?x:Name=""" + grid + @"""").Index; + var end = xaml.IndexOf("", start, StringComparison.Ordinal); + var last = Regex.Matches(xaml[start..end], @"""])*>").Last().Value; + Assert.Contains("Header=\"Collected\"", last, StringComparison.Ordinal); + Assert.Contains("Binding=\"{Binding CollectionTimeLocal}\"", last, StringComparison.Ordinal); + Assert.Contains("SortMemberPath=\"CollectionTime\"", last, StringComparison.Ordinal); + } + + [Fact] + public void TheOldSnapshotLabel_IsGone() + { + Assert.DoesNotContain("SnapshotTime\"", Xaml(), StringComparison.Ordinal); + Assert.DoesNotContain("Snapshot at", Xaml(), StringComparison.Ordinal); + } + + [Fact] + public void TheCpuSchedulerLoad_WrapsItsRowsWithTheSnapshotsCollectionTime() + { + var source = ViewerFile("ViewerServerTab.CpuScheduler.cs"); + Assert.Contains("CpuSchedulerGrid.ItemsSource = CpuSchedulerGridRow.Build(snapshotTask.Result);", source, StringComparison.Ordinal); + } + + [Fact] + public void TheDisplayText_IsTheDisplayZonesWallTime_ToTheSecond() + { + var saved = CultureInfo.CurrentCulture; + var savedMode = ViewerTimeHelper.CurrentDisplayMode; + try + { + CultureInfo.CurrentCulture = CultureInfo.InvariantCulture; + ViewerTimeHelper.CurrentDisplayMode = TimeDisplayMode.UTC; + var at = new DateTime(2026, 10, 3, 14, 5, 9, DateTimeKind.Unspecified); + // A literal, so the format string itself is pinned. + var expected = "2026-10-03 14:05:09"; + Assert.Equal(expected, HistoryTime.CollectionLocal(at)); + Assert.Equal(expected, new RunningJobRow { CollectionTime = at }.CollectionTimeLocal); + Assert.Equal(expected, new LatchStatsSnapshotRow("L", 0, 0, 0, null, null, null, at).CollectionTimeLocal); + Assert.Equal(expected, new SpinlockStatsSnapshotRow("S", 0, 0, 0, 0, 0, null, null, null, at).CollectionTimeLocal); + Assert.Equal(expected, new CpuSchedulerGridRow { CollectionTime = at }.CollectionTimeLocal); + } + finally + { + ViewerTimeHelper.CurrentDisplayMode = savedMode; + CultureInfo.CurrentCulture = saved; + } + } +} diff --git a/Darling/Darling.Tests/ViewerSnapshotStripCollectedTests.cs b/Darling/Darling.Tests/ViewerSnapshotStripCollectedTests.cs new file mode 100644 index 0000000000..8ab286d9d3 --- /dev/null +++ b/Darling/Darling.Tests/ViewerSnapshotStripCollectedTests.cs @@ -0,0 +1,169 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Globalization; +using System.Text.RegularExpressions; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using PerformanceMonitor.Ui; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// The desktop viewer's Memory, Plan Cache and Session Stats summary strips draw the newest snapshot, so each says when that +/// snapshot was collected (#4966): in the display zone, to the second, or the strip's own empty marker. Lite words it the same way. +/// +[Collection("viewer-time-statics")] +public sealed class ViewerSnapshotStripCollectedTests : IDisposable +{ + private readonly TimeDisplayMode _savedMode = ViewerTimeHelper.CurrentDisplayMode; + private readonly CultureInfo _savedCulture = CultureInfo.CurrentCulture; + + public ViewerSnapshotStripCollectedTests() => CultureInfo.CurrentCulture = CultureInfo.InvariantCulture; + + public void Dispose() + { + ViewerTimeHelper.CurrentDisplayMode = _savedMode; + CultureInfo.CurrentCulture = _savedCulture; + } + + private static string ViewerFile(string file) => ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", file).ReplaceLineEndings("\n"); + + private static string MethodBody(string source, string signaturePattern) + { + var match = Regex.Match(source, signaturePattern + @".*?\n \}\n", RegexOptions.Singleline); + Assert.True(match.Success, $"{signaturePattern} was not found"); + return match.Value; + } + + [Fact] + public void TheText_IsTheCollectionTimeInTheDisplayZone_ToTheSecond() + { + ViewerTimeHelper.CurrentDisplayMode = TimeDisplayMode.UTC; + var utc = new DateTime(2026, 10, 3, 14, 5, 9, 700, DateTimeKind.Unspecified); + + Assert.Equal("2026-10-03 14:05:09", HistoryTime.SnapshotCollected(utc, "--")); + Assert.Equal(HistoryTime.CollectionLocal(utc), HistoryTime.SnapshotCollected(utc, "N/A")); + } + + [Fact] + public void TheText_FollowsTheDisplayMode_AndTheEmptyMarkerIsTheStripsOwn() + { + ViewerTimeHelper.CurrentDisplayMode = TimeDisplayMode.LocalTime; + var utc = new DateTime(2026, 7, 1, 12, 0, 0, DateTimeKind.Unspecified); + var expected = TimeZoneInfo.ConvertTimeFromUtc(utc, TimeZoneInfo.Local).ToString("yyyy-MM-dd HH:mm:ss", CultureInfo.InvariantCulture); + + Assert.Equal(expected, HistoryTime.SnapshotCollected(utc, "--")); + Assert.Equal("--", HistoryTime.SnapshotCollected(null, "--")); + Assert.Equal("N/A", HistoryTime.SnapshotCollected(null, "N/A")); + } + + [Fact] + public void EachStrip_DeclaresItsCollectedElement_WithTheStripsEmptyText() + { + var xaml = ViewerFile("ViewerServerTab.xaml"); + + Assert.Matches(@"\s*]*ForegroundDimBrush[^>]*/>\s*]*ForegroundDimBrush[^>]*/>\s*The latch and spinlock snapshot reads against a real store return the collection time of the snapshot they rendered (#4966). +/* #1776 own-store: deliberately NOT [Collection("live-postgres")]. Every test here reaches DARLING_TEST_PG only to CREATE + and DROP its own scratch database. */ +[Collection("viewer-time-statics")] +public sealed class ViewerSnapshotTimeLiveTests +{ + private const int ServerId = -496660; + + [Fact] + public async Task TheLatchAndSpinlockSnapshots_CarryTheNewestCollectionTime_AgainstDevPostgres() + { + var ct = TestContext.Current.CancellationToken; + await using var scratch = await QueryGridSeed.OpenScratchAsync(ct); + var end = QueryGridSeed.NowToTheMinute(); + var older = end.AddHours(-3); + var newest = end.AddHours(-1); + await using (var connection = new NpgsqlConnection(scratch.ConnectionString)) + { + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + await EventGridSeed.AddServerAsync(connection, ServerId, "snapshot-time", end.AddDays(-2), "latch_stats", end, clockOffsetMinutes: null, ct); + long id = 1; + foreach (var at in new[] { older, newest }) + { + await ExecAsync(connection, $"INSERT INTO latch_stats (collection_id, collection_time, server_id, server_name, latch_class, waiting_requests_count, wait_time_ms, max_wait_time_ms, delta_waiting_requests_count, delta_wait_time_ms, delta_max_wait_time_ms, sample_interval_seconds) VALUES ({id++}, @t, {ServerId}, 'snapshot-time', 'BUFFER', 10, 20, 5, 1, 2, 1, 60)", at, ct); + await ExecAsync(connection, $"INSERT INTO spinlock_stats (collection_id, collection_time, server_id, server_name, spinlock_name, collisions, spins, spins_per_collision, sleep_time, backoffs, delta_collisions, delta_spins, delta_sleep_time, delta_backoffs, sample_interval_seconds) VALUES ({id++}, @t, {ServerId}, 'snapshot-time', 'LOCK_HASH', 10, 20, 2, 0, 0, 1, 2, 0, 0, 60)", at, ct); + } + } + + var viewer = new ViewerDataService(scratch.ConnectionString); + var latch = await viewer.GetLatchStatsSnapshotAsync(ServerId, end.AddDays(-7), end, ct); + var spinlock = await viewer.GetSpinlockStatsSnapshotAsync(ServerId, end.AddDays(-7), end, ct); + + Assert.Single(latch); + Assert.Equal(newest, latch[0].CollectionTime); + Assert.Single(spinlock); + Assert.Equal(newest, spinlock[0].CollectionTime); + Assert.Empty(await viewer.GetLatchStatsSnapshotAsync(ServerId + 1, end.AddDays(-7), end, ct)); + } + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, DateTime at, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + command.Parameters.AddWithValue("t", DateTime.SpecifyKind(at, DateTimeKind.Unspecified)); + await command.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/ViewerStatsRowBlankCellTests.cs b/Darling/Darling.Tests/ViewerStatsRowBlankCellTests.cs new file mode 100644 index 0000000000..7e84467536 --- /dev/null +++ b/Darling/Darling.Tests/ViewerStatsRowBlankCellTests.cs @@ -0,0 +1,151 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.ComponentModel; +using System.Linq; +using System.Text.RegularExpressions; +using System.Threading; +using System.Windows.Controls; +using System.Windows.Data; +using PerformanceMonitor.Darling.Viewer; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// #5329: the Top Queries and Top Procedures grid rows carry NULL, not 0, for the columns the hourly rollups keep no +/// copy of, and a null cell shows blank. A page read from the hourly rollup holds every such column null at once, so +/// the order within a page never mixes blank with numbers; the pins below still cover what a null does when a sort +/// meets it. The live half (the hourly read returns null, the raw read returns values) is +/// . +/// +public sealed class ViewerStatsRowBlankCellTests +{ + /// The Top Queries row's columns the hourly rollup lacks (the grid's XAML binds each one by this name). + private static readonly string[] QueryBlankColumns = + { + "TotalLogicalReads", "AvgReads", "TotalLogicalWrites", "TotalPhysicalReads", "TotalRows", "TotalSpills", + "MinPhysicalReads", "MaxPhysicalReads", "MinRows", "MaxRows", "MinSpills", "MaxSpills", "MinDop", "MaxDop", + "MinGrantKb", "MaxGrantKb", "MinUsedGrantKb", "MaxUsedGrantKb", "MinIdealGrantKb", "MaxIdealGrantKb", + "MinReservedThreads", "MaxReservedThreads", "MinUsedThreads", "MaxUsedThreads", "TotalClrMs", + "PlanGenerationNum", "WorkerTimePerSecond", + /* #5329: the rollup keeps min/max of per-collection deltas, not per-execution extremes. */ + "MinCpuMs", "MaxCpuMs", "MinElapsedMs", "MaxElapsedMs", + }; + + private static readonly string[] ProcedureBlankColumns = + { + "TotalLogicalReads", "AvgReads", "TotalLogicalWrites", "TotalPhysicalReads", "TotalSpills", "AvgSpills", + "MinLogicalReads", "MaxLogicalReads", "MinPhysicalReads", "MaxPhysicalReads", "MinLogicalWrites", "MaxLogicalWrites", + "MinSpills", "MaxSpills", + "MinCpuMs", "MaxCpuMs", "MinElapsedMs", "MaxElapsedMs", + }; + + [Fact] + public void AFreshRow_HoldsNullForEveryColumnTheRollupLacks_NotZero() + { + foreach (var name in QueryBlankColumns) + { + Assert.Null(typeof(ViewerQueryStatsRow).GetProperty(name)!.GetValue(new ViewerQueryStatsRow())); + } + + foreach (var name in ProcedureBlankColumns) + { + Assert.Null(typeof(ViewerProcedureStatsRow).GetProperty(name)!.GetValue(new ViewerProcedureStatsRow())); + } + } + + /// A real zero from the raw route stays a zero: blank means "not kept", 0 means "kept and was 0". + [Fact] + public void ARawRowWithARealZero_StaysZero_AndItsAverageIsStillComputed() + { + var q = new ViewerQueryStatsRow { TotalExecutions = 4, TotalLogicalReads = 0, TotalSpills = 0 }; + Assert.Equal(0L, q.TotalLogicalReads); + Assert.Equal(0.0, q.AvgReads); + var p = new ViewerProcedureStatsRow { TotalExecutions = 4, TotalLogicalReads = 400 }; + Assert.Equal(100.0, p.AvgReads); + } + + /// The grid's cell is a TextBlock bound as {Binding X, StringFormat=N0}: a null shows empty, a number + /// shows formatted, and a real zero shows 0. + [Fact] + public void ANullCell_ShowsBlank_ANumberShowsFormatted_AZeroShowsZero() + { + StaTestThread.Run(() => + { + Assert.Equal("", CellText(new ViewerQueryStatsRow(), "TotalLogicalReads")); + Assert.Equal("", CellText(new ViewerQueryStatsRow(), "AvgReads")); + Assert.Equal("", CellText(new ViewerProcedureStatsRow(), "TotalSpills")); + Assert.Equal("", CellText(new ViewerProcedureStatsRow(), "AvgSpills")); + Assert.Equal(1234.ToString("N0"), CellText(new ViewerQueryStatsRow { TotalLogicalReads = 1234 }, "TotalLogicalReads")); + Assert.Equal(0.ToString("N0"), CellText(new ViewerQueryStatsRow { TotalLogicalReads = 0 }, "TotalLogicalReads")); + }); + } + + /// A descending sort (the click that ranks by a column) puts the blank rows after every number, so a null + /// is never ranked as if it were the biggest value; and a page that is blank throughout keeps the order the read gave it. + [Fact] + public void ADescendingSort_PutsBlankRowsLast_AndAllBlankKeepsTheReadsOrder() + { + StaTestThread.Run(() => + { + var mixed = new List + { + new() { QueryHash = "blank-1" }, + new() { QueryHash = "big", TotalLogicalReads = 900 }, + new() { QueryHash = "blank-2" }, + new() { QueryHash = "small", TotalLogicalReads = 5 }, + }; + var view = new ListCollectionView(mixed); + view.SortDescriptions.Add(new SortDescription("TotalLogicalReads", ListSortDirection.Descending)); + Assert.Equal(new[] { "big", "small", "blank-1", "blank-2" }, view.Cast().Select(r => r.QueryHash).ToArray()); + + var allBlank = new List + { + new() { QueryHash = "a" }, new() { QueryHash = "b" }, new() { QueryHash = "c" }, + }; + foreach (var direction in new[] { ListSortDirection.Descending, ListSortDirection.Ascending }) + { + var blankView = new ListCollectionView(allBlank); + blankView.SortDescriptions.Add(new SortDescription("TotalLogicalReads", direction)); + Assert.Equal(new[] { "a", "b", "c" }, blankView.Cast().Select(r => r.QueryHash).ToArray()); + } + }); + } + + /// A binding that names a fallback or a null value (TargetNullValue=0) would turn a blank back into a + /// false 0, so none of the blank-able columns may carry one, on either grid. + [Fact] + public void NoBlankableColumnBinding_NamesATargetNullValueOrAFallback() + { + var xaml = ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerServerTab.xaml"); + var checkedBindings = 0; + foreach (var name in QueryBlankColumns.Concat(ProcedureBlankColumns).Distinct()) + { + foreach (Match m in Regex.Matches(xaml, @"\{Binding " + name + @"[,}][^{}]*\}")) + { + checkedBindings++; + Assert.DoesNotContain("TargetNullValue", m.Value, StringComparison.Ordinal); + Assert.DoesNotContain("FallbackValue", m.Value, StringComparison.Ordinal); + } + } + + Assert.True(checkedBindings >= 20, $"the scan must find the grids' bindings, found {checkedBindings}"); + } + + private static string CellText(object row, string path) + { + var cell = new TextBlock { DataContext = row }; + cell.SetBinding(TextBlock.TextProperty, new Binding(path) { StringFormat = "N0" }); + return cell.Text; + } + +} diff --git a/Darling/Darling.Tests/ViewerStoreLoginRawReadLiveTests.cs b/Darling/Darling.Tests/ViewerStoreLoginRawReadLiveTests.cs new file mode 100644 index 0000000000..7e2ad2ee98 --- /dev/null +++ b/Darling/Darling.Tests/ViewerStoreLoginRawReadLiveTests.cs @@ -0,0 +1,96 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Text.RegularExpressions; +using System.Threading.Tasks; +using Npgsql; +using PerformanceMonitor.Darling.Service; +using PerformanceMonitor.Darling.Storage; +using Xunit; + +namespace Darling.Tests; + +/// +/// #4348: checks the store login grants of the Darling desktop viewer. The viewer's grants are lifted from the shipped +/// provisioning SQL (), put on a disposable role in a scratch store, and +/// the role then SELECTs the three tables that hold statement text and plans. The test fails if any of those reads is refused. +/// +/// #1776 own-store: mints its own scratch database (), never the shared +/// live-postgres collection. The role is cluster-wide, so it carries a per-run suffix and is dropped in cleanup. +/// +public sealed class ViewerStoreLoginRawReadLiveTests +{ + private static readonly string Role = "ssf_viewer_" + Guid.NewGuid().ToString("N")[..8]; + private const string RolePassword = "SsfViewerTestPw0123456789abcdef01"; + + /// The tables a plan or statement read goes to (plan section 6.6). + private static readonly string[] Tables = { "query_text_dim", "query_plan_dim", "query_store_text" }; + + [Fact] + public async Task TheViewersShippedGrants_LetItSelectTheStatementAndPlanTables() + { + var connectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + Assert.SkipWhen(string.IsNullOrEmpty(connectionString), + "Set DARLING_TEST_PG to a Postgres connection string to run the viewer store-login grants live test."); + var ct = TestContext.Current.CancellationToken; + + var managed = DarlingManagedRoles.BuildProvisioningSql( + ProvisioningTestSecrets.Admin, ProvisioningTestSecrets.Viewer, ProvisioningTestSecrets.Mcp, 15); + + /* The shipped statements that give viewer its read of the collect schema, lifted verbatim and retargeted. */ + var usage = Regex.Match(managed, @"GRANT USAGE ON SCHEMA [^;]*? TO admin, viewer;"); + var select = Regex.Match(managed, @"GRANT SELECT ON ALL TABLES IN SCHEMA collect TO admin, viewer;"); + Assert.True(usage.Success, "the managed provisioning SQL no longer grants viewer USAGE on the schemas"); + Assert.True(select.Success, "the managed provisioning SQL no longer grants viewer SELECT on the collect tables"); + + await using var scratch = await ScratchPostgres.CreateAsync(connectionString!, ct); + await using var owner = new NpgsqlConnection(scratch.ConnectionString); + await owner.OpenAsync(ct); + await PgMigrations.MigrateAsync(owner, ct); + + var bodySucceeded = false; + try + { + await ExecAsync(owner, $"CREATE ROLE {Role} LOGIN NOSUPERUSER PASSWORD '{RolePassword}'", ct); + await ExecAsync(owner, usage.Value.Replace("TO admin, viewer;", "TO " + Role + ";", StringComparison.Ordinal), ct); + await ExecAsync(owner, select.Value.Replace("TO admin, viewer;", "TO " + Role + ";", StringComparison.Ordinal), ct); + + var roleString = new NpgsqlConnectionStringBuilder(scratch.ConnectionString) + { + Username = Role, + Password = RolePassword, + SearchPath = "collect,config,public", + Pooling = false, + }.ConnectionString; + await using var viewer = new NpgsqlConnection(roleString); + await viewer.OpenAsync(ct); + + foreach (var table in Tables) + { + await using var command = new NpgsqlCommand($"SELECT count(*) FROM collect.{table}", viewer); + var count = await command.ExecuteScalarAsync(ct); + Assert.NotNull(count); + } + + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (cleanup, cleanupCt) => + await new LiveCleanupBatch(cleanup).DropRolesAsync( + $"DROP OWNED BY {Role}; DROP ROLE IF EXISTS {Role};", new[] { Role }, cleanupCt)); + } + } + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, System.Threading.CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(ct); + } +} diff --git a/Darling/Darling.Tests/ViewerSystemEventsDataStartTests.cs b/Darling/Darling.Tests/ViewerSystemEventsDataStartTests.cs new file mode 100644 index 0000000000..ed0b67efbc --- /dev/null +++ b/Darling/Darling.Tests/ViewerSystemEventsDataStartTests.cs @@ -0,0 +1,237 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Data; +using System.IO; +using System.Linq; +using System.Text.RegularExpressions; +using System.Threading.Tasks; +using PerformanceMonitor.Analysis.Baselines; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; +using static Darling.Tests.RepoFile; + +namespace Darling.Tests; + +/// +/// The System Events grids and the Default Trace grid say where their data starts (#4966): one banner above the sub-tab +/// control covers the nine, the eight system_health grids on one probe and Default Trace on its own. The notice names the +/// earlier of the collector's coverage start and the earliest event a grid shows, and the Default Trace events, stored on +/// the server's local clock, are compared in UTC after the read's own per-row conversion. +/// +public sealed class ViewerSystemEventsDataStartTests +{ + private static readonly DateTime RangeStart = new(2026, 9, 1, 0, 0, 0, DateTimeKind.Unspecified); + + private static string ViewerFile(string file) => ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", file); + + private static string MethodBody(string source, string signaturePattern) + { + var match = Regex.Match(source, signaturePattern + @".*?\r?\n \}\r?\n", RegexOptions.Singleline); + Assert.True(match.Success, $"{signaturePattern} was not found"); + return match.Value; + } + + private static int Matches(string source, string pattern) => Regex.Matches(source, pattern).Count; + + private static string TabSources() => + string.Join("\n", Directory.GetFiles(PathTo("Darling", "PerformanceMonitor.Darling.Viewer"), "ViewerServerTab*.cs").Select(File.ReadAllText)); + + // ── Every read path asks the probe ── + + [Theory] + [InlineData("LoadSchedulerIssuesAsync")] + [InlineData("LoadSevereErrorsAsync")] + [InlineData("LoadMemoryConditionsAsync")] + [InlineData("LoadMemoryBrokerAsync")] + [InlineData("LoadMemoryNodeOomAsync")] + [InlineData("LoadSignificantWaitsAsync")] + [InlineData("LoadCpuTasksAsync")] + [InlineData("LoadIoIssuesAsync")] + public void EachSystemHealthGrid_AsksAboutTheWindowItDraws_AndRaisesTheOneBanner(string load) + { + var body = MethodBody(ViewerFile("ViewerServerTab.SystemEvents.cs"), @"private async Task " + load + @"\("); + + Assert.Equal(1, Matches(body, @"var dataStartTask = _dataService\.GetSystemHealthEventsDataStartAsync\(_server\.ServerId,\s*startUtc,\s*endUtc\);")); + Assert.Equal(1, Matches(body, + @"await ShowEventDataStartAsync\(SystemEventsTruncationBanner,\s*dataStartTask,\s*""System Events"",\s*startUtc,\s*data\.Select\(r => r\.EventTime\)\);")); + } + + [Fact] + public void DefaultTrace_AsksAboutTheWindowItDraws_AndRaisesTheOneBanner_OverTheEventsInUtc() + { + var body = MethodBody(ViewerFile("ViewerServerTab.SystemEvents.cs"), @"private async Task LoadDefaultTraceEventsAsync\("); + + Assert.Equal(1, Matches(body, @"var dataStartTask = _dataService\.GetDefaultTraceDataStartAsync\(_server\.ServerId,\s*startUtc,\s*endUtc\);")); + Assert.Equal(1, Matches(body, + @"await ShowEventDataStartAsync\(SystemEventsTruncationBanner,\s*dataStartTask,\s*""Default Trace"",\s*startUtc,\s*data\.Select\(r => r\.EventTimeUtc\)\);")); + } + + /* A census over every server-tab file: the nine reads are paired with the nine banner calls, so a tenth read added without + one fails here instead of shipping with a stale banner. */ + [Fact] + public void EveryTabRead_OfTheNineGrids_HasItsProbe_AndNoneIsAwaitedBare() + { + var tabs = TabSources(); + + Assert.Equal(8, Matches(tabs, @"_dataService\.GetSystemHealthEventsDataStartAsync\(")); + Assert.Equal(1, Matches(tabs, @"_dataService\.GetDefaultTraceDataStartAsync\(")); + Assert.Equal(9, Matches(tabs, @"ShowEventDataStartAsync\(SystemEventsTruncationBanner,")); + Assert.Equal(1, Matches(tabs, @"_dataService\.GetDefaultTraceEventsAsync\(")); + Assert.DoesNotContain("await dataStartTask", ViewerFile("ViewerServerTab.SystemEvents.cs"), StringComparison.Ordinal); + } + + /* The Corruption and Contention sub-tabs are charts on a time axis that already shows the empty span, so they take the + banner down instead of leaving the last grid's notice over them. */ + [Fact] + public void TheChartSubTabs_TakeTheBannerDown() + { + var body = MethodBody(ViewerFile("ViewerServerTab.SystemEvents.cs"), @"private async Task LoadSystemEventsAsync\("); + + Assert.Equal(1, Matches(body, + @"case SystemEventsContentionSubTabIndex:\s*default:\s*(?:/\*.*?\*/\s*)?SystemEventsTruncationBanner\.Visibility = Visibility\.Collapsed;\s*await LoadSystemHealthChartsAsync\(\);")); + } + + [Fact] + public void TheBanner_SitsAboveTheSubTabControl_InTheExistingStyle() + { + var xaml = ViewerFile("ViewerServerTab.xaml"); + + var banner = Regex.Match(xaml, @"]*x:Name=""SystemEventsTruncationBanner""[^>]*/>"); + Assert.True(banner.Success, "SystemEventsTruncationBanner is not declared in ViewerServerTab.xaml"); + Assert.Contains("Visibility=\"Collapsed\"", banner.Value, StringComparison.Ordinal); + Assert.Contains("Background=\"#22FFAA00\"", banner.Value, StringComparison.Ordinal); + Assert.True(banner.Index < xaml.IndexOf("x:Name=\"SystemEventsSubTabControl\"", StringComparison.Ordinal), "the banner must sit above the sub-tab control"); + } + + // ── The probes ── + + [Fact] + public void TheProbes_GoThroughTheSharedFloor_OverTheCollectorTablesTheGridsReadFrom_OnTheCollectorsUtcClock() + { + var source = ViewerFile("ViewerDataService.SystemEvents.cs"); + Assert.Contains("DataWindowFloor.Source.ForCollectorTable(\"system_health_events\")", source, StringComparison.Ordinal); + Assert.Contains("DataWindowFloor.Source.ForCollectorTable(\"default_trace_events\")", source, StringComparison.Ordinal); + + /* The probe reads collection_time (UTC), never the Default Trace's server-local event_time, so the coverage needs no clock. */ + Assert.Equal("collection_time", DataWindowFloor.Source.ForCollectorTable("system_health_events").TimeColumn); + Assert.Equal("collection_time", DataWindowFloor.Source.ForCollectorTable("default_trace_events").TimeColumn); + } + + /* Neither read keeps a newest-first page, so no row cap applies: a LIMIT added later has to be wired to the cap rule. */ + [Fact] + public void TheReads_CarryNoRowCap() + { + Assert.DoesNotMatch(@"LIMIT\s+\d{2,}", ViewerDataService.SystemHealthEventsByTypeSql); + Assert.DoesNotMatch(@"LIMIT\s+\d{2,}", ViewerDataService.DefaultTraceEventsByWindowSql); + } + + // ── Default Trace: the server's clock, compared in UTC ── + + private const string EasternWindowsId = "Eastern Standard Time"; + + private static DateTime Naive(int year, int month, int day, int hour, int minute = 0) => + new(year, month, day, hour, minute, 0, DateTimeKind.Unspecified); + + /// The notice the tab names for Default Trace events stored at on the server's own clock. + private static async Task NoticeAsync(ServerClock clock, DateTime startUtc, DateTime endUtc, DateTime coverageStartUtc, params DateTime[] storedLocal) + { + var table = new DataTable(); + table.Columns.Add("event_time_local", typeof(DateTime)); + table.Columns.Add("event_name", typeof(string)); + foreach (var name in new[] { "database_name", "object_name", "login_name", "host_name", "application_name" }) + { + table.Columns.Add(name, typeof(string)); + } + + table.Columns.Add("spid", typeof(int)); + table.Columns.Add("duration_us", typeof(long)); + table.Columns.Add("integer_data", typeof(long)); + table.Columns.Add("severity", typeof(int)); + table.Columns.Add("error_number", typeof(int)); + table.Columns.Add("text_data", typeof(string)); + foreach (var local in storedLocal) + { + table.Rows.Add(local, "Data File Auto Grow", "db", DBNull.Value, DBNull.Value, DBNull.Value, DBNull.Value, 55, 1_500_000L, 256L, DBNull.Value, DBNull.Value, DBNull.Value); + } + + using var reader = table.CreateDataReader(); + var rows = await ViewerDataService.ReadDefaultTraceEventsAsync(reader, clock, startUtc, endUtc, TestContext.Current.CancellationToken); + Assert.NotEmpty(rows); + return ViewerEventDataStart.Of(coverageStartUtc, ViewerEventDataStart.EarliestOf(rows.Select(r => r.EventTimeUtc))); + } + + /* A clock 5 hours behind UTC: the server was added on Sep 5 (coverage), and its first collection stored trace history from + Sep 2 at 00:00 on the server's own clock. The notice is that moment in UTC, 05:00, not the stored 00:00. */ + [Fact] + public async Task WithAClock5HoursBehindUtc_HistoryBeforeTheCoverage_GivesANoticeAtItsStartInUtc() + { + var notice = await NoticeAsync(ServerClock.Resolve(null, -300), RangeStart, RangeStart.AddDays(9), Naive(2026, 9, 5, 0), + Naive(2026, 9, 2, 0), Naive(2026, 9, 6, 10)); + + Assert.Equal(Naive(2026, 9, 2, 5), notice); + Assert.True(RawWindowFloor.IsTruncated(notice, RangeStart)); + } + + /* History that reaches the range start, on the same clock: the oldest stored event is Aug 31 at 20:00 local, 01:00 UTC on Sep 1. + No notice, though the coverage starts days later. */ + [Fact] + public async Task WithAClock5HoursBehindUtc_HistoryThatReachesTheRangeStart_GivesNoNotice() + { + var notice = await NoticeAsync(ServerClock.Resolve(null, -300), RangeStart, RangeStart.AddDays(9), Naive(2026, 9, 5, 0), + Naive(2026, 8, 31, 20), Naive(2026, 9, 6, 10)); + + Assert.Equal(Naive(2026, 9, 1, 1), notice); + Assert.False(RawWindowFloor.IsTruncated(notice, RangeStart)); + } + + /* A quiet start: the store covered the range from before it, and the first event came 5 hours in (stored 00:00 local on Sep 1, + which is 05:00 UTC). No notice. */ + [Fact] + public async Task WithAClock5HoursBehindUtc_AQuietStart_GivesNoNotice() + { + var notice = await NoticeAsync(ServerClock.Resolve(null, -300), RangeStart, RangeStart.AddDays(9), RangeStart.AddDays(-20), + Naive(2026, 9, 1, 0), Naive(2026, 9, 6, 10)); + + Assert.False(RawWindowFloor.IsTruncated(notice, RangeStart)); + } + + /* A day that crosses the spring change (Mar 8, 2026, 02:00 local): the event stored at 01:30 is standard time, 06:30 UTC, and the + one at 03:30 is daylight time. The snapshot's offset says -240; a single subtracted offset would put the first event at 05:30. */ + [Fact] + public async Task ADayThatCrossesTheSpringChange_UsesThatDatesOffset() + { + var notice = await NoticeAsync(ServerClock.Resolve(EasternWindowsId, -240), Naive(2026, 3, 1, 0), Naive(2026, 3, 20, 0), Naive(2026, 3, 9, 0), + Naive(2026, 3, 8, 1, 30), Naive(2026, 3, 8, 3, 30)); + + Assert.Equal(Naive(2026, 3, 8, 6, 30), notice); + } + + /* Rows that start inside the range: the first event came after the coverage began (Sep 6 at 10:00 local, 15:00 UTC, against a + coverage on Sep 5), so the notice reads back the coverage start. */ + [Fact] + public async Task WithAClock5HoursBehindUtc_RowsThatStartInsideTheRange_GiveANoticeAtTheCoverageStart() + { + var notice = await NoticeAsync(ServerClock.Resolve(null, -300), RangeStart, RangeStart.AddDays(9), Naive(2026, 9, 5, 0), Naive(2026, 9, 6, 10)); + + Assert.Equal(Naive(2026, 9, 5, 0), notice); + Assert.True(RawWindowFloor.IsTruncated(notice, RangeStart)); + } + + /* The fall change (Nov 1, 2026, 02:00 local): the event at 00:30 is daylight time, 04:30 UTC. The snapshot says -300. */ + [Fact] + public async Task ADayThatCrossesTheFallChange_UsesThatDatesOffset() + { + var notice = await NoticeAsync(ServerClock.Resolve(EasternWindowsId, -300), Naive(2026, 10, 25, 0), Naive(2026, 11, 10, 0), Naive(2026, 11, 2, 0), + Naive(2026, 11, 1, 0, 30), Naive(2026, 11, 1, 3, 30)); + + Assert.Equal(Naive(2026, 11, 1, 4, 30), notice); + } +} diff --git a/Darling/Darling.Tests/ViewerSystemEventsTests.cs b/Darling/Darling.Tests/ViewerSystemEventsTests.cs index 2da6841904..11464eb121 100644 --- a/Darling/Darling.Tests/ViewerSystemEventsTests.cs +++ b/Darling/Darling.Tests/ViewerSystemEventsTests.cs @@ -50,20 +50,8 @@ public void SystemHealthEventsByTypeSql_ReadsViewByServerTypeAndEventTimeWindow( Assert.Contains("ORDER BY event_time DESC", sql, StringComparison.Ordinal); } - [Fact] - public void DatabaseNameMapSql_LatestNamePerId_FromSizeStatsView() - { - var sql = ViewerDataService.DatabaseNameMapSql; - - Assert.Contains("DISTINCT ON (database_id)", sql, StringComparison.Ordinal); - Assert.Contains("FROM v_database_size_stats", sql, StringComparison.Ordinal); - Assert.Contains("WHERE server_id = $1", sql, StringComparison.Ordinal); - Assert.Contains("ORDER BY database_id, collection_time DESC", sql, StringComparison.Ordinal); - } - [Theory] [InlineData(nameof(ViewerDataService.SystemHealthEventsByTypeSql))] - [InlineData(nameof(ViewerDataService.DatabaseNameMapSql))] public void SystemEventsReads_PgDialect_PositionalParams_NoTSqlNamedParams(string sqlName) { var sql = (string)typeof(ViewerDataService).GetField(sqlName)!.GetValue(null)!; @@ -383,31 +371,7 @@ public void ShredThenFilter_KeepsOnlySignificantRows_AcrossAllCategories() Assert.Single(result.SystemHealth); } - // ── Severe-Errors database_id resolution ── - - [Fact] - public void ResolveDatabaseName_MappedId_ReturnsName() - { - var map = new Dictionary { [5] = "AdventureWorks", [1] = "master" }; - Assert.Equal("AdventureWorks", ViewerDataService.ResolveDatabaseName(5, map)); - Assert.Equal("master", ViewerDataService.ResolveDatabaseName(1, map)); - } - - [Fact] - public void ResolveDatabaseName_NoContext_ZeroOrNull_IsBlank() - { - var map = new Dictionary { [5] = "AdventureWorks" }; - Assert.Equal(string.Empty, ViewerDataService.ResolveDatabaseName(0, map)); - Assert.Equal(string.Empty, ViewerDataService.ResolveDatabaseName(null, map)); - } - - [Fact] - public void ResolveDatabaseName_RealButUnmappedId_SurfacesTheRawId_NotBlank() - { - var map = new Dictionary { [5] = "AdventureWorks" }; - // A database dropped before the latest size-stats snapshot: surface the id rather than silently blank. - Assert.Equal("database_id 7", ViewerDataService.ResolveDatabaseName(7, map)); - } + // Severe-Errors database_id resolution (the name the id carried at the error's time) is pinned in DatabaseNameHistoryTests (#5373). // ── Row projection: machine-local event time + resolved DB passthrough ── diff --git a/Darling/Darling.Tests/ViewerTextLookupWindowBoundLiveTests.cs b/Darling/Darling.Tests/ViewerTextLookupWindowBoundLiveTests.cs new file mode 100644 index 0000000000..757b6c3f99 --- /dev/null +++ b/Darling/Darling.Tests/ViewerTextLookupWindowBoundLiveTests.cs @@ -0,0 +1,1070 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using NpgsqlTypes; +using PerformanceMonitor.Darling.Service.Mcp; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using Xunit; + +namespace Darling.Tests; + +/// +/// #5420: the viewer's procedure comparison and the Query Store top reads look text up with a time bound. +/// +/// The comparison used to pick each procedure's statement with a per-procedure lookup that had no time bound (no index +/// covers sql_handle, so a procedure with no text walked every retained chunk), and joined the period rows to the top +/// procedures with IS NOT DISTINCT FROM, which cannot hash and ran as a nested loop that threw away millions of rows. +/// The Query Store top reads' inline-text fallback (query_store_stats, one lookup per candidate row with no text +/// row) had the same unbounded shape, in the viewer and in Darling's MCP reader, which keep one tail text each. +/// +/// Every fact runs on a scratch store with the collector tables converted to hypertables (1-day chunks) over 12 days, so +/// "reads only the window's chunks" is something the plan can show. The one documented change is pinned by name: a procedure +/// or query whose only text is older than the window shows none. +/// +/* #1776 own-store: deliberately NOT [Collection("live-postgres")]. Every live fact here reaches DARLING_TEST_PG only to + CREATE and DROP its own database through ScratchPostgres and then works entirely inside it, so it cannot race live + collection. */ +[Trait("Cost", "Slow")] +public sealed class ViewerTextLookupWindowBoundLiveTests +{ + private const string SkipText = "Set DARLING_TEST_PG to a Postgres connection string to run the #5420 text-lookup window-bound live pins (each mints its own scratch database)."; + private const int ServerId = 5420; + + /// The window's end. History runs 12 days back from here, so the 1-day chunks number 13. + private static readonly DateTime End = new(2026, 2, 20, 12, 0, 0, DateTimeKind.Unspecified); + + /// Comparison windows: current = the last 24 h, baseline = the 24 h before it. The text lookup's window is + /// baseline start to current end: 48 h, which touches at most 3 one-day chunks. + private static readonly DateTime CurrentStart = End.AddHours(-24); + private static readonly DateTime BaselineStart = End.AddHours(-48); + private const int WindowChunkCeiling = 3; + + private static string? ConnectionString => Environment.GetEnvironmentVariable("DARLING_TEST_PG"); + + /// + /// as it stood before #5420, verbatim: the oracle the new + /// statement is compared with. Do not edit it to follow the live statement. + /// + private const string OldProcedureStatsComparisonSql = """ + WITH top_current AS ( + SELECT database_name, schema_name, object_name + FROM procedure_stats + WHERE server_id = $1 + AND collection_time >= $2 AND collection_time <= $3 + AND ($6::text[] IS NULL OR database_name = ANY($6)) + AND delta_execution_count > 0 + GROUP BY database_name, schema_name, object_name + ORDER BY SUM(delta_execution_count) DESC + LIMIT 100 + ), + top_baseline AS ( + SELECT database_name, schema_name, object_name + FROM procedure_stats + WHERE server_id = $1 + AND collection_time >= $4 AND collection_time <= $5 + AND ($6::text[] IS NULL OR database_name = ANY($6)) + AND delta_execution_count > 0 + GROUP BY database_name, schema_name, object_name + ORDER BY SUM(delta_execution_count) DESC + LIMIT 100 + ), + top_procs AS ( + SELECT DISTINCT database_name, schema_name, object_name + FROM ( + SELECT * FROM top_current + UNION ALL + SELECT * FROM top_baseline + ) AS combined + ), + current_period AS ( + SELECT tp.database_name, tp.schema_name, tp.object_name, + SUM(ps.delta_execution_count) AS exec_count, + SUM(ps.delta_elapsed_time)::double precision / NULLIF(SUM(ps.delta_execution_count), 0) / 1000.0 AS avg_duration_ms, + SUM(ps.delta_worker_time)::double precision / NULLIF(SUM(ps.delta_execution_count), 0) / 1000.0 AS avg_cpu_ms, + SUM(ps.delta_physical_reads)::double precision / NULLIF(SUM(ps.delta_execution_count), 0) AS avg_reads, + MAX(ps.sql_handle) AS sql_handle + FROM top_procs tp + INNER JOIN procedure_stats ps + ON ps.database_name IS NOT DISTINCT FROM tp.database_name + AND ps.schema_name IS NOT DISTINCT FROM tp.schema_name + AND ps.object_name IS NOT DISTINCT FROM tp.object_name + WHERE ps.server_id = $1 + AND ps.collection_time >= $2 AND ps.collection_time <= $3 + AND ps.delta_execution_count > 0 + GROUP BY tp.database_name, tp.schema_name, tp.object_name + ), + baseline_period AS ( + SELECT tp.database_name, tp.schema_name, tp.object_name, + SUM(ps.delta_execution_count) AS exec_count, + SUM(ps.delta_elapsed_time)::double precision / NULLIF(SUM(ps.delta_execution_count), 0) / 1000.0 AS avg_duration_ms, + SUM(ps.delta_worker_time)::double precision / NULLIF(SUM(ps.delta_execution_count), 0) / 1000.0 AS avg_cpu_ms, + SUM(ps.delta_physical_reads)::double precision / NULLIF(SUM(ps.delta_execution_count), 0) AS avg_reads, + MAX(ps.sql_handle) AS sql_handle + FROM top_procs tp + INNER JOIN procedure_stats ps + ON ps.database_name IS NOT DISTINCT FROM tp.database_name + AND ps.schema_name IS NOT DISTINCT FROM tp.schema_name + AND ps.object_name IS NOT DISTINCT FROM tp.object_name + WHERE ps.server_id = $1 + AND ps.collection_time >= $4 AND ps.collection_time <= $5 + AND ps.delta_execution_count > 0 + GROUP BY tp.database_name, tp.schema_name, tp.object_name + ) + SELECT COALESCE(c.database_name, b.database_name) AS database_name, + COALESCE(c.schema_name, b.schema_name) AS schema_name, + COALESCE(c.object_name, b.object_name) AS object_name, + c.exec_count, c.avg_duration_ms, c.avg_cpu_ms, c.avg_reads, + b.exec_count AS baseline_exec_count, + b.avg_duration_ms AS baseline_avg_duration_ms, + b.avg_cpu_ms AS baseline_avg_cpu_ms, + b.avg_reads AS baseline_avg_reads, + t.query_text + FROM current_period c + FULL OUTER JOIN baseline_period b + ON COALESCE(c.database_name, '') = COALESCE(b.database_name, '') + AND COALESCE(c.schema_name, '') = COALESCE(b.schema_name, '') + AND COALESCE(c.object_name, '') = COALESCE(b.object_name, '') + /* #1981: a REPRESENTATIVE statement of the procedure via the same normalized sql_handle + join #1568's module attribution relies on (both stores persist the identical + CONVERT(varchar(130), ..., 1) text). procedure_stats captures no text of its own, so + this is the latest captured statement from inside the module — parity with the other + two comparison grids, labeled a statement rather than the definition. v_query_stats + resolves the #1767 payload dimension. */ + LEFT JOIN LATERAL ( + SELECT qs.query_text + FROM v_query_stats qs + WHERE qs.server_id = $1 + AND qs.sql_handle = COALESCE(c.sql_handle, b.sql_handle) + AND qs.query_text IS NOT NULL + ORDER BY qs.collection_time DESC + LIMIT 1 + ) t ON TRUE + """; + + /* ───────────────────────────── seeds ───────────────────────────── */ + + /// + /// as it stood before the #5420 review round, verbatim: the oracle the + /// derived-table statement is compared with. Do not edit it to follow the live statement. + /// + private const string OldQueryStatsComparisonSql = """ + WITH top_current AS ( + SELECT query_hash, database_name + FROM query_stats + WHERE server_id = $1 + AND collection_time >= $2 AND collection_time <= $3 + AND ($6::text[] IS NULL OR database_name = ANY($6)) + AND delta_execution_count > 0 + GROUP BY query_hash, database_name + ORDER BY SUM(delta_execution_count) DESC + LIMIT 100 + ), + top_baseline AS ( + SELECT query_hash, database_name + FROM query_stats + WHERE server_id = $1 + AND collection_time >= $4 AND collection_time <= $5 + AND ($6::text[] IS NULL OR database_name = ANY($6)) + AND delta_execution_count > 0 + GROUP BY query_hash, database_name + ORDER BY SUM(delta_execution_count) DESC + LIMIT 100 + ), + top_hashes AS ( + SELECT DISTINCT query_hash, database_name + FROM ( + SELECT * FROM top_current + UNION ALL + SELECT * FROM top_baseline + ) AS combined + ), + current_period AS ( + SELECT th.database_name, th.query_hash, + SUM(qs.delta_execution_count) AS exec_count, + SUM(qs.delta_elapsed_time)::double precision / NULLIF(SUM(qs.delta_execution_count), 0) / 1000.0 AS avg_duration_ms, + SUM(qs.delta_worker_time)::double precision / NULLIF(SUM(qs.delta_execution_count), 0) / 1000.0 AS avg_cpu_ms, + SUM(qs.delta_physical_reads)::double precision / NULLIF(SUM(qs.delta_execution_count), 0) AS avg_reads, + MAX(qs.query_text) AS query_text + FROM top_hashes th + INNER JOIN v_query_stats qs + ON qs.query_hash IS NOT DISTINCT FROM th.query_hash + AND qs.database_name IS NOT DISTINCT FROM th.database_name + WHERE qs.server_id = $1 + AND qs.collection_time >= $2 AND qs.collection_time <= $3 + AND qs.delta_execution_count > 0 + GROUP BY th.database_name, th.query_hash + ), + baseline_period AS ( + SELECT th.database_name, th.query_hash, + SUM(qs.delta_execution_count) AS exec_count, + SUM(qs.delta_elapsed_time)::double precision / NULLIF(SUM(qs.delta_execution_count), 0) / 1000.0 AS avg_duration_ms, + SUM(qs.delta_worker_time)::double precision / NULLIF(SUM(qs.delta_execution_count), 0) / 1000.0 AS avg_cpu_ms, + SUM(qs.delta_physical_reads)::double precision / NULLIF(SUM(qs.delta_execution_count), 0) AS avg_reads, + MAX(qs.query_text) AS query_text + FROM top_hashes th + INNER JOIN v_query_stats qs + ON qs.query_hash IS NOT DISTINCT FROM th.query_hash + AND qs.database_name IS NOT DISTINCT FROM th.database_name + WHERE qs.server_id = $1 + AND qs.collection_time >= $4 AND qs.collection_time <= $5 + AND qs.delta_execution_count > 0 + GROUP BY th.database_name, th.query_hash + ) + SELECT COALESCE(c.database_name, b.database_name) AS database_name, + COALESCE(c.query_hash, b.query_hash) AS query_hash, + COALESCE(c.query_text, b.query_text) AS query_text, + c.exec_count, c.avg_duration_ms, c.avg_cpu_ms, c.avg_reads, + b.exec_count AS baseline_exec_count, + b.avg_duration_ms AS baseline_avg_duration_ms, + b.avg_cpu_ms AS baseline_avg_cpu_ms, + b.avg_reads AS baseline_avg_reads + FROM current_period c + FULL OUTER JOIN baseline_period b + ON COALESCE(c.database_name, '') = COALESCE(b.database_name, '') + AND COALESCE(c.query_hash, '') = COALESCE(b.query_hash, '') + """; + + /// + /// as it stood before the #5420 review round, verbatim (oracle). + /// + private const string OldQueryStoreComparisonSql = """ + WITH deduped_current AS ( + /* LOAD-BEARING (correctness, not just perf) — #1841. The rows are CUMULATIVE per-interval + snapshots and the collector re-fetches the OPEN interval every cycle, so the SAME interval + (same first_execution_time) is stored repeatedly with a growing execution_count. Keep the + LATEST snapshot per interval before aggregating. + + The execution-count weighting below does NOT rescue this on its own: the repeated snapshots + of one interval carry DIFFERENT (growing) weights AND different avg_* values, so an open + interval is weighted by the triangular sum of its own growth. That bias is stronger in the + recent window than in the baseline window (recent windows hold more still-open intervals), + which skews the very delta this comparison exists to compute. One deduped CTE per window, + so both arms are treated identically. */ + SELECT + database_name, + /* #2150: query_id is projected (it was already a partition key) so the period CTEs can + resolve text from collect.query_store_text, which is keyed on it. */ + query_id, + query_hash, + query_text, + execution_count, + avg_duration_us, + avg_cpu_time_us, + avg_logical_io_reads, + ROW_NUMBER() OVER + ( + PARTITION BY database_name, query_id, plan_id, runtime_stats_interval_id, first_execution_time, execution_type_desc, replica_role + ORDER BY collection_time DESC, execution_count DESC + ) AS rn + FROM query_store_stats + WHERE server_id = $1 + AND collection_time >= $2 AND collection_time <= $3 + AND ($6::text[] IS NULL OR database_name = ANY($6)) + ), + deduped_baseline AS ( + SELECT + database_name, + /* #2150: query_id is projected (it was already a partition key) so the period CTEs can + resolve text from collect.query_store_text, which is keyed on it. */ + query_id, + query_hash, + query_text, + execution_count, + avg_duration_us, + avg_cpu_time_us, + avg_logical_io_reads, + ROW_NUMBER() OVER + ( + PARTITION BY database_name, query_id, plan_id, runtime_stats_interval_id, first_execution_time, execution_type_desc, replica_role + ORDER BY collection_time DESC, execution_count DESC + ) AS rn + FROM query_store_stats + WHERE server_id = $1 + AND collection_time >= $4 AND collection_time <= $5 + AND ($6::text[] IS NULL OR database_name = ANY($6)) + ), + top_current AS ( + SELECT database_name, query_hash + FROM deduped_current + WHERE rn = 1 + AND execution_count > 0 + GROUP BY database_name, query_hash + ORDER BY SUM(execution_count) DESC + LIMIT 100 + ), + top_baseline AS ( + SELECT database_name, query_hash + FROM deduped_baseline + WHERE rn = 1 + AND execution_count > 0 + GROUP BY database_name, query_hash + ORDER BY SUM(execution_count) DESC + LIMIT 100 + ), + top_hashes AS ( + SELECT DISTINCT database_name, query_hash + FROM ( + SELECT * FROM top_current + UNION ALL + SELECT * FROM top_baseline + ) AS combined + ), + current_period AS ( + SELECT th.database_name, th.query_hash, + SUM(qs.execution_count) AS exec_count, + SUM(qs.execution_count * qs.avg_duration_us::double precision) / NULLIF(SUM(qs.execution_count), 0) / 1000.0 AS avg_duration_ms, + SUM(qs.execution_count * qs.avg_cpu_time_us::double precision) / NULLIF(SUM(qs.execution_count), 0) / 1000.0 AS avg_cpu_ms, + SUM(qs.execution_count * qs.avg_logical_io_reads::double precision) / NULLIF(SUM(qs.execution_count), 0) AS avg_reads, + /* #2150: this comparison groups by query_hash, but text is stored per query_id, so the + side table is joined on the finer key and MAX still picks one member's text for the + group — the same arbitrary-but-deterministic choice MAX(qs.query_text) made before. + The join cannot fan out (query_store_text is one row per server/database/query_id, by + primary key), so the execution-count SUMs above are unaffected. The COALESCE keeps + pre-cutover rows, whose text is still inline, reading exactly as they used to. */ + MAX(COALESCE(x.query_sql_text, qs.query_text)) AS query_text + FROM top_hashes th + INNER JOIN deduped_current qs + ON qs.query_hash IS NOT DISTINCT FROM th.query_hash + AND qs.database_name IS NOT DISTINCT FROM th.database_name + LEFT JOIN query_store_text AS x + ON x.server_id = $1 + AND x.database_name = qs.database_name + AND x.query_id = qs.query_id + WHERE qs.rn = 1 + AND qs.execution_count > 0 + GROUP BY th.database_name, th.query_hash + ), + baseline_period AS ( + SELECT th.database_name, th.query_hash, + SUM(qs.execution_count) AS exec_count, + SUM(qs.execution_count * qs.avg_duration_us::double precision) / NULLIF(SUM(qs.execution_count), 0) / 1000.0 AS avg_duration_ms, + SUM(qs.execution_count * qs.avg_cpu_time_us::double precision) / NULLIF(SUM(qs.execution_count), 0) / 1000.0 AS avg_cpu_ms, + SUM(qs.execution_count * qs.avg_logical_io_reads::double precision) / NULLIF(SUM(qs.execution_count), 0) AS avg_reads, + /* #2150 — same resolution as current_period above. Both arms need it because the final + projection takes COALESCE(c.query_text, b.query_text): converting only one arm would + leave a GONE row (present in baseline only) with no text to fall back to. */ + MAX(COALESCE(x.query_sql_text, qs.query_text)) AS query_text + FROM top_hashes th + INNER JOIN deduped_baseline qs + ON qs.query_hash IS NOT DISTINCT FROM th.query_hash + AND qs.database_name IS NOT DISTINCT FROM th.database_name + LEFT JOIN query_store_text AS x + ON x.server_id = $1 + AND x.database_name = qs.database_name + AND x.query_id = qs.query_id + WHERE qs.rn = 1 + AND qs.execution_count > 0 + GROUP BY th.database_name, th.query_hash + ) + SELECT COALESCE(c.database_name, b.database_name) AS database_name, + COALESCE(c.query_hash, b.query_hash) AS query_hash, + COALESCE(c.query_text, b.query_text) AS query_text, + c.exec_count, c.avg_duration_ms, c.avg_cpu_ms, c.avg_reads, + b.exec_count AS baseline_exec_count, + b.avg_duration_ms AS baseline_avg_duration_ms, + b.avg_cpu_ms AS baseline_avg_cpu_ms, + b.avg_reads AS baseline_avg_reads + FROM current_period c + FULL OUTER JOIN baseline_period b + ON COALESCE(c.database_name, '') = COALESCE(b.database_name, '') + AND COALESCE(c.query_hash, '') = COALESCE(b.query_hash, '') + """; + + /// + /// query_stats for the Top Queries comparison twin: 130 hashes, one row every 2 hours for 12 days, with every delta + /// column set so the averages are real numbers. NULL parts: the database of every 25th, the hash of every 40th and of 125 + /// (so 125 is NULL on both keys, and 40, 80 and 120 merge into one NULL-hash group). Empty-string keys beside them, each against a NULL it must stay apart from: 48 + /// is the one empty hash, in the database that the heavy NULL-hash group (40, 80, 120) sits in, and 127 is the one empty + /// database while 51 is a NULL database on its hash. 48 and 51 carry one execution per row, so they never make the top 100 and a + /// join that merged a NULL key with an empty one would fan the heavy row out; 100-109 only before the current window + /// (GONE) and 110-119 only inside it (NEW). 100 hashes cannot hold all of them, so the top-100 cut trims each period. + /// + private const string QueryStatsTwinSeedSql = @" +INSERT INTO collect.query_stats +(collection_id, collection_time, server_id, server_name, database_name, query_hash, sql_handle, query_text, + delta_execution_count, delta_worker_time, delta_elapsed_time, delta_physical_reads) +SELECT 3000000 + row_number() OVER ()::bigint, g.t, 5420, 'srv', + CASE WHEN p % 25 = 0 OR p = 51 THEN NULL WHEN p = 127 THEN '' ELSE 'db' || (p % 4) END, + CASE WHEN p % 40 = 0 OR p = 125 THEN NULL WHEN p = 48 THEN '' WHEN p = 51 THEN '0xQ127' ELSE '0xQ' || p END, + '0xT' || p, 'SELECT ' || p || ' /* ' || to_char(g.t, 'YYYY-MM-DD HH24:MI') || ' */', + CASE WHEN p IN (48, 51) THEN 1 + WHEN (p + extract(epoch FROM g.t)::bigint / 7200) % 7 = 0 THEN 0 + ELSE (CASE WHEN p <= 10 OR p % 25 = 0 OR p % 40 = 0 OR p >= 100 THEN 1000 ELSE 1 END) + * (1 + (p * 31 + extract(epoch FROM g.t)::bigint / 7200) % 50) END, + 1000 + (p * 7919) % 90000 + (extract(epoch FROM g.t)::bigint / 7200) % 13, + 2000 + (p * 104729) % 90000 + (extract(epoch FROM g.t)::bigint / 7200) % 17, + (p * 3 + extract(epoch FROM g.t)::bigint / 7200) % 97 +FROM generate_series(1, 130) AS p +CROSS JOIN generate_series(TIMESTAMP '2026-02-08 12:00:00', TIMESTAMP '2026-02-20 12:00:00', INTERVAL '2 hours') AS g(t) +WHERE NOT (p BETWEEN 100 AND 109 AND g.t >= TIMESTAMP '2026-02-19 12:00:00') +AND NOT (p BETWEEN 110 AND 119 AND g.t <= TIMESTAMP '2026-02-19 12:00:00');"; + + /// + /// query_store_stats for the Query Store comparison twin: 130 queries, one snapshot an hour with two snapshots per + /// 2-hour interval (so the dedupe keeps the later one), the same NULL, empty-string (47: the one empty hash, in the database the NULL-hash group 80 sits in; 127: the one + /// empty database, 51 is a NULL database on its hash; 47 and 51 never make the top 100) and GONE/NEW shapes as the Top Queries seed, and a + /// query_store_text row for every query whose inline text is NULL (q divisible by 3, outside the NULL-database ones). + /// + private const string QueryStoreTwinSeedSql = @" +INSERT INTO collect.query_store_text (server_id, database_name, query_id, query_sql_text, last_seen) +SELECT 5420, 'db' || (q % 3), q, 'SELECT ' || q || ' /* side table */', TIMESTAMP '2026-02-20 12:00:00' +FROM generate_series(1, 130) AS q +WHERE q % 3 = 0 AND q % 25 <> 0; +INSERT INTO collect.query_store_stats +(collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, execution_type_desc, first_execution_time, + last_execution_time, query_text, query_hash, execution_count, avg_duration_us, avg_cpu_time_us, avg_logical_io_reads, + avg_logical_io_writes, avg_physical_io_reads, avg_rowcount, query_plan_hash, runtime_stats_interval_id) +SELECT 4000000 + row_number() OVER ()::bigint, g.t, 5420, 'srv', + CASE WHEN q % 25 = 0 OR q = 51 THEN NULL WHEN q = 127 THEN '' ELSE 'db' || (q % 3) END, q, q, 'Regular', + date_bin(INTERVAL '2 hours', g.t, TIMESTAMP '2026-02-08 12:00:00'), g.t, + CASE WHEN q % 3 = 0 THEN NULL ELSE 'SELECT ' || q || ' /* inline */' END, + CASE WHEN q % 40 = 0 OR q = 125 THEN NULL WHEN q = 47 THEN '' WHEN q = 51 THEN 'h127' ELSE 'h' || q END, + CASE WHEN q IN (47, 51) THEN 1 + WHEN (q + extract(epoch FROM g.t)::bigint / 3600) % 7 = 0 THEN 0 + ELSE (CASE WHEN q <= 10 OR q % 25 = 0 OR q % 40 = 0 OR q >= 100 THEN 1000 ELSE 1 END) + * (1 + (q * 31 + extract(epoch FROM g.t)::bigint / 3600) % 50) END, + 1000 + (q * 7919) % 90000 + (extract(epoch FROM g.t)::bigint / 3600) % 13, + 2000 + (q * 104729) % 90000 + (extract(epoch FROM g.t)::bigint / 3600) % 17, + (q * 3 + extract(epoch FROM g.t)::bigint / 3600) % 97, 5, 5, 5, 'ph' || q, + q * 100000 + (extract(epoch FROM g.t)::bigint / 7200) +FROM generate_series(1, 130) AS q +CROSS JOIN generate_series(TIMESTAMP '2026-02-08 12:00:00', TIMESTAMP '2026-02-20 12:00:00', INTERVAL '1 hour') AS g(t) +WHERE NOT (q BETWEEN 100 AND 109 AND g.t >= TIMESTAMP '2026-02-19 12:00:00') +AND NOT (q BETWEEN 110 AND 119 AND g.t <= TIMESTAMP '2026-02-19 12:00:00');"; + + /// + /// Two procedures whose statements share one instant: each handle has two query_stats rows at 11:00 on the last day + /// with different collection ids and texts. The first procedure's rows go in lower id first, the second's higher id first, + /// so no heap order picks the higher id for both. + /// + private const string TieSeedSql = @" +INSERT INTO collect.procedure_stats +(collection_id, collection_time, server_id, server_name, database_name, schema_name, object_name, sql_handle, + delta_execution_count, delta_worker_time, delta_elapsed_time, delta_physical_reads) +VALUES (9000001, TIMESTAMP '2026-02-20 11:00:00', 5420, 'srv', 'db1', 'dbo', 'proc_tie_a', '0xHTIEA', 500000, 1, 1, 1), + (9000002, TIMESTAMP '2026-02-20 11:00:00', 5420, 'srv', 'db1', 'dbo', 'proc_tie_b', '0xHTIEB', 500000, 1, 1, 1); +INSERT INTO collect.query_stats +(collection_id, collection_time, server_id, server_name, database_name, query_hash, sql_handle, query_text, delta_execution_count) +VALUES (9100001, TIMESTAMP '2026-02-20 11:00:00', 5420, 'srv', 'db1', '0xQTIEA1', '0xHTIEA', 'tie A: the lower collection id', 1), + (9100002, TIMESTAMP '2026-02-20 11:00:00', 5420, 'srv', 'db1', '0xQTIEA2', '0xHTIEA', 'tie A: the higher collection id', 1), + (9100004, TIMESTAMP '2026-02-20 11:00:00', 5420, 'srv', 'db1', '0xQTIEB2', '0xHTIEB', 'tie B: the higher collection id', 1), + (9100003, TIMESTAMP '2026-02-20 11:00:00', 5420, 'srv', 'db1', '0xQTIEB1', '0xHTIEB', 'tie B: the lower collection id', 1);"; + + /// + /// 202 procedures, one row per hour for 12 days. Keys with NULL parts (database NULL for every 50th, schema for every + /// 40th, object for every 60th), two keys that differ only by an empty-string versus NULL object (201, 202), procedures + /// 150-159 present only before the current window (GONE) and 160-169 only inside it (NEW). The ones the facts care about + /// carry large counts so they rank inside each period's top 100. + /// + private const string ProcedureSeedSql = @" +INSERT INTO collect.procedure_stats +(collection_id, collection_time, server_id, server_name, database_name, schema_name, object_name, sql_handle, + delta_execution_count, delta_worker_time, delta_elapsed_time, delta_physical_reads) +SELECT row_number() OVER ()::bigint, g.t, 5420, 'srv', + CASE WHEN p % 50 = 0 THEN NULL ELSE 'db' || (p % 4) END, + CASE WHEN p % 40 = 0 THEN NULL ELSE 'dbo' END, + CASE WHEN p = 201 THEN '' WHEN p = 202 OR p % 60 = 0 THEN NULL ELSE 'proc_' || p END, + '0xH' || p, + CASE WHEN (p + extract(epoch FROM g.t)::bigint / 3600) % 7 = 0 THEN 0 + ELSE (CASE WHEN p <= 10 OR p % 50 = 0 OR p % 40 = 0 OR p % 60 = 0 OR p >= 150 THEN 1000 ELSE 1 END) + * (1 + (p * 31 + extract(epoch FROM g.t)::bigint / 3600) % 50) END, + 1000 + (p * 7919) % 90000 + (extract(epoch FROM g.t)::bigint / 3600) % 13, + 2000 + (p * 104729) % 90000 + (extract(epoch FROM g.t)::bigint / 3600) % 17, + (p * 3 + extract(epoch FROM g.t)::bigint / 3600) % 97 +FROM generate_series(1, 202) AS p +CROSS JOIN generate_series(TIMESTAMP '2026-02-08 12:00:00', TIMESTAMP '2026-02-20 12:00:00', INTERVAL '1 hour') AS g(t) +WHERE NOT (p BETWEEN 150 AND 159 AND g.t >= TIMESTAMP '2026-02-19 12:00:00') +AND NOT (p BETWEEN 160 AND 169 AND g.t <= TIMESTAMP '2026-02-19 12:00:00');"; + + /// + /// query_stats, one row per handle every 2 hours, the text different on every row so the newest pick matters, and + /// never two rows of one handle on one instant (a tie on the time is arbitrary in both statements). Handles 1-5 carry text + /// only older than 3 days (before the 48 h window) and NULL-text rows inside it; 6-10 carry no inline text but a digest that + /// resolves through query_text_dim (the #1767 payload dimension); 11-160 carry text throughout; 161-180 only NULL + /// text; 181+ have no row at all. + /// + private const string QueryStatsSeedSql = @" +INSERT INTO collect.query_text_dim (digest, query_text, last_seen) +SELECT decode(md5('dim' || p), 'hex'), 'SELECT ' || p || ' /* from the payload dimension */', TIMESTAMP '2026-02-20 12:00:00' +FROM generate_series(6, 10) AS p; +INSERT INTO collect.query_stats +(collection_id, collection_time, server_id, server_name, database_name, query_hash, sql_handle, query_text, query_text_digest, delta_execution_count) +SELECT 1000000 + row_number() OVER ()::bigint, g.t, 5420, 'srv', 'db' || (p % 4), '0xQ' || p, '0xH' || p, + CASE WHEN p <= 5 THEN CASE WHEN g.t < TIMESTAMP '2026-02-17 12:00:00' THEN 'SELECT ' || p || ' /* only before the window */' END + WHEN p BETWEEN 6 AND 10 THEN NULL + WHEN p <= 160 THEN 'SELECT ' || p || ' /* ' || to_char(g.t, 'YYYY-MM-DD HH24:MI') || ' */' + ELSE NULL END, + CASE WHEN p BETWEEN 6 AND 10 THEN decode(md5('dim' || p), 'hex') END, + 1 +FROM generate_series(1, 180) AS p +CROSS JOIN generate_series(TIMESTAMP '2026-02-08 12:00:00', TIMESTAMP '2026-02-20 12:00:00', INTERVAL '2 hours') AS g(t);"; + + /// + /// query_store_stats: 60 queries, one raw row per hour for 12 days, each its own interval so none dedupes. 1-30 carry + /// text on every row (different each time); 31-40 only older than the 48 h window; 41-60 none anywhere. No + /// query_store_text row exists, so every candidate takes the inline-text fallback. + /// + private const string QueryStoreSeedSql = @" +INSERT INTO collect.query_store_stats +(collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, execution_type_desc, first_execution_time, + last_execution_time, query_text, query_hash, execution_count, avg_duration_us, avg_cpu_time_us, avg_logical_io_reads, + avg_logical_io_writes, avg_physical_io_reads, avg_rowcount, query_plan_hash, runtime_stats_interval_id) +SELECT 2000000 + row_number() OVER ()::bigint, g.t, 5420, 'srv', 'db' || (q % 3), q, q, 'Regular', g.t - INTERVAL '10 minutes', + g.t, + CASE WHEN q <= 30 THEN 'SELECT ' || q || ' /* ' || to_char(g.t, 'YYYY-MM-DD HH24:MI') || ' */' + WHEN q <= 40 THEN CASE WHEN g.t < TIMESTAMP '2026-02-18 12:00:00' THEN 'SELECT ' || q || ' /* only before the window */' END + ELSE NULL END, + 'h' || q, 10 + q, 1000 + q * 37, 500 + q, 5, 5, 5, 5, 'ph' || q, + q * 100000 + (extract(epoch FROM g.t)::bigint / 3600) +FROM generate_series(1, 60) AS q +CROSS JOIN generate_series(TIMESTAMP '2026-02-08 12:00:00', TIMESTAMP '2026-02-20 12:00:00', INTERVAL '1 hour') AS g(t);"; + + private static async Task ExecAsync(NpgsqlConnection connection, string sql, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection); + command.CommandTimeout = 120; + await command.ExecuteNonQueryAsync(ct); + } + + /// + /// Six groups per twin, each pair a NULL key beside the empty-string key it must stay apart from, all heavy enough to make the + /// top 100 in BOTH windows (one row each at 06:00 of the baseline day and of the current day; the counts are 100 to 600 in + /// the current window and a tenth of that in the baseline). The outer FULL JOIN that merged a NULL key with an empty one + /// would fan a row out here (#5420). + /// + private const string QueryStatsNullEmptySeedSql = @" +INSERT INTO collect.query_stats +(collection_id, collection_time, server_id, server_name, database_name, query_hash, sql_handle, query_text, delta_execution_count, + delta_worker_time, delta_elapsed_time, delta_physical_reads) +SELECT 7000000 + row_number() OVER ()::bigint, g.t, 5420, 'srv', v.db, v.h, '0xN' || v.n, 'SELECT ' || v.n, + CASE WHEN g.cur THEN v.cur_exec ELSE v.cur_exec / 10 END, 1000, 2000, 3 +FROM (VALUES (1, 'dbA'::text, NULL::text, 100), (2, 'dbA', '', 200), (3, NULL, 'hC', 300), (4, '', 'hC', 400), + (5, NULL, NULL, 500), (6, '', '', 600)) AS v(n, db, h, cur_exec) +CROSS JOIN (VALUES (TIMESTAMP '2026-02-20 06:00:00', true), (TIMESTAMP '2026-02-19 06:00:00', false)) AS g(t, cur);"; + + /// The same six NULL-versus-empty groups as , in query_store_stats. + private const string QueryStoreNullEmptySeedSql = @" +INSERT INTO collect.query_store_stats +(collection_id, collection_time, server_id, server_name, database_name, query_id, plan_id, execution_type_desc, first_execution_time, + last_execution_time, query_text, query_hash, execution_count, avg_duration_us, avg_cpu_time_us, avg_logical_io_reads, + avg_logical_io_writes, avg_physical_io_reads, avg_rowcount, query_plan_hash, runtime_stats_interval_id) +SELECT 7000000 + row_number() OVER ()::bigint, g.t, 5420, 'srv', v.db, v.n, v.n, 'Regular', g.t, g.t, 'SELECT ' || v.n, v.h, + CASE WHEN g.cur THEN v.cur_exec ELSE v.cur_exec / 10 END, 1000, 2000, 3, 5, 5, 5, 'ph' || v.n, + v.n * 100000 + g.cur::int +FROM (VALUES (1, 'dbA'::text, NULL::text, 100), (2, 'dbA', '', 200), (3, NULL, 'hC', 300), (4, '', 'hC', 400), + (5, NULL, NULL, 500), (6, '', '', 600)) AS v(n, db, h, cur_exec) +CROSS JOIN (VALUES (TIMESTAMP '2026-02-20 06:00:00', true), (TIMESTAMP '2026-02-19 06:00:00', false)) AS g(t, cur);"; + + /// Six procedures in procedure_stats, a NULL part beside the empty-string one in each of the three key columns. + private const string ProcedureNullEmptySeedSql = @" +INSERT INTO collect.procedure_stats +(collection_id, collection_time, server_id, server_name, database_name, schema_name, object_name, sql_handle, + delta_execution_count, delta_worker_time, delta_elapsed_time, delta_physical_reads) +SELECT 7000000 + row_number() OVER ()::bigint, g.t, 5420, 'srv', v.db, v.sch, v.obj, '0xN' || v.n, + CASE WHEN g.cur THEN v.cur_exec ELSE v.cur_exec / 10 END, 1000, 2000, 3 +FROM (VALUES (1, 'dbA'::text, 'dbo'::text, NULL::text, 100), (2, 'dbA', 'dbo', '', 200), (3, NULL, 'dbo', 'pX', 300), + (4, '', 'dbo', 'pX', 400), (5, 'dbA', NULL, 'pX', 500), (6, 'dbA', '', 'pX', 600)) AS v(n, db, sch, obj, cur_exec) +CROSS JOIN (VALUES (TIMESTAMP '2026-02-20 06:00:00', true), (TIMESTAMP '2026-02-19 06:00:00', false)) AS g(t, cur);"; + + /// Migrates a scratch store, converts the collector tables to hypertables BEFORE any row lands (so the + /// seed creates real 1-day chunks), runs the seeds and hands the body a connection. + private static async Task RunLiveAsync(string[] seeds, Func body) + { + Assert.SkipWhen(string.IsNullOrEmpty(ConnectionString), SkipText); + var ct = TestContext.Current.CancellationToken; + await using var scratch = await ScratchPostgres.CreateAsync(ConnectionString!, ct); + await using var connection = new NpgsqlConnection(scratch.ConnectionString); + await connection.OpenAsync(ct); + await PgMigrations.MigrateAsync(connection, ct); + Assert.True(await LiveTimescaleProbe.TryEnableAsync(scratch.ConnectionString, ct), "the scratch store needs TimescaleDB for the chunked shape"); + await TimescaleSupport.ConvertToHypertablesAsync(connection, NullLogger.Instance, ct); + await ExecAsync(connection, "SELECT _timescaledb_functions.stop_background_workers()", ct); /* no policy job reshapes the chunks under a plan assertion */ + var bodySucceeded = false; + try + { + foreach (var seed in seeds) + { + await ExecAsync(connection, seed, ct); + } + + await ExecAsync(connection, "ANALYZE collect.procedure_stats; ANALYZE collect.query_stats; ANALYZE collect.query_store_stats;", ct); + await body(connection, ct); + bodySucceeded = true; + } + finally + { + await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (_, _) => { }); + } + } + + /* ───────────────────────────── plan helpers ───────────────────────────── */ + + private static NpgsqlParameter Timestamp(DateTime value) => + new NpgsqlParameter { TypedValue = value, NpgsqlDbType = NpgsqlDbType.Timestamp }; + + private static void AddComparisonParameters(NpgsqlCommand command) => + BindComparison(command, BaselineStart, CurrentStart, DatabaseFilter.All); + + /// The comparison's six parameters: the last 24 h as the current window, to + /// as the baseline, and as the database filter. + private static void BindComparison(NpgsqlCommand command, DateTime baselineStart, DateTime baselineEnd, DatabaseFilter filter) + { + command.Parameters.Add(new NpgsqlParameter { TypedValue = ServerId }); + command.Parameters.Add(Timestamp(CurrentStart)); + command.Parameters.Add(Timestamp(End)); + command.Parameters.Add(Timestamp(baselineStart)); + command.Parameters.Add(Timestamp(baselineEnd)); + command.Parameters.Add(filter.Parameter()); + } + + /// EXPLAIN (ANALYZE, FORMAT JSON) of with the parameters adds. + private static async Task ExplainAsync(NpgsqlConnection connection, string sql, Action bind, CancellationToken ct) + { + await using var command = new NpgsqlCommand("EXPLAIN (ANALYZE, FORMAT JSON) " + sql, connection); + command.CommandTimeout = 120; + bind(command); + var json = (string)(await command.ExecuteScalarAsync(ct))!; + return JsonDocument.Parse(json).RootElement[0].GetProperty("Plan"); + } + + private static IEnumerable Nodes(JsonElement node) + { + yield return node; + if (node.TryGetProperty("Plans", out var children)) + { + foreach (var child in children.EnumerateArray()) + { + foreach (var descendant in Nodes(child)) + { + yield return descendant; + } + } + } + } + + private static string Text(JsonElement node, string property) => + node.TryGetProperty(property, out var value) && value.ValueKind == JsonValueKind.String ? value.GetString()! : ""; + + private static async Task> ChunkNamesAsync(NpgsqlConnection connection, string table, CancellationToken ct) + { + await using var command = new NpgsqlCommand("SELECT chunk_name FROM timescaledb_information.chunks WHERE hypertable_schema = 'collect' AND hypertable_name = @t", connection); + command.Parameters.AddWithValue("t", table); + var names = new HashSet(StringComparer.Ordinal); + await using var reader = await command.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + names.Add(reader.GetString(0)); + } + + return names; + } + + /// The distinct chunks of the plan actually read (a node that never ran has 0 loops). + private static async Task<(int Scanned, int Total)> ChunksReadAsync(NpgsqlConnection connection, JsonElement plan, string table, CancellationToken ct) + { + var chunks = await ChunkNamesAsync(connection, table, ct); + var read = Nodes(plan) + .Where(n => chunks.Contains(Text(n, "Relation Name")) && n.TryGetProperty("Actual Loops", out var loops) && loops.GetDouble() > 0) + .Select(n => Text(n, "Relation Name")) + .ToHashSet(StringComparer.Ordinal); + return (read.Count, chunks.Count); + } + + /* ───────────────────────────── the procedure comparison ───────────────────────────── */ + + private static Task> RunComparisonAsync(NpgsqlConnection connection, string sql, CancellationToken ct) => + RunKeyedAsync(connection, sql, AddComparisonParameters, 3, ct); + + /// Every row of as text cells (doubles by their exact bits), keyed on the first + /// columns. + private static async Task> RunKeyedAsync( + NpgsqlConnection connection, string sql, Action bind, int keyColumns, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection) { CommandTimeout = 120 }; + bind(command); + var rows = new SortedDictionary(StringComparer.Ordinal); + await using var reader = await command.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + var cells = new string[reader.FieldCount]; + for (var i = 0; i < cells.Length; i++) + { + cells[i] = reader.IsDBNull(i) ? "" : reader.GetValue(i) is double d + ? BitConverter.DoubleToInt64Bits(d).ToString("x", CultureInfo.InvariantCulture) + : Convert.ToString(reader.GetValue(i), CultureInfo.InvariantCulture)!; + } + + rows.Add(string.Join('|', cells.Take(keyColumns)), cells); + } + + return rows; + } + + [Fact] + public async Task TheComparison_ReadsTheTextFromTheWindowsChunksOnly_AndItsPeriodJoinsHaveNoNullSafeJoinFilter() + { + await RunLiveAsync(new[] { ProcedureSeedSql, QueryStatsSeedSql }, async (connection, ct) => + { + var plan = await ExplainAsync(connection, ViewerDataService.ProcedureStatsComparisonSql, AddComparisonParameters, ct); + + var (scanned, total) = await ChunksReadAsync(connection, plan, "query_stats", ct); + Assert.True(total >= 12, $"the seed must span many chunks (found {total}) or this proves nothing."); + Assert.True(scanned is > 0 and <= WindowChunkCeiling, + $"the text lookup read {scanned} of {total} query_stats chunks; a 48 h window touches at most {WindowChunkCeiling}."); + + /* IS NOT DISTINCT FROM is not hashable: the old period joins were nested loops with that as a join filter + over every window row. A hash or merge join on plain equalities has no such filter. */ + Assert.DoesNotContain(Nodes(plan), n => Text(n, "Join Filter").Contains("DISTINCT FROM", StringComparison.Ordinal)); + var keyedJoins = Nodes(plan).Count(n => Text(n, "Node Type") is "Hash Join" or "Merge Join" + && (Text(n, "Hash Cond") + Text(n, "Merge Cond")).Contains("object_name", StringComparison.Ordinal)); + Assert.True(keyedJoins >= 2, $"expected both period joins as hash or merge joins on the keys, found {keyedJoins}."); + }); + } + + [Fact] + public async Task TheOldStatement_ReadsEveryChunk_WhichIsWhatTheBoundRemoves() + { + /* The oracle's own shape, so the shape test above cannot pass vacuously: the same seed under the pre-#5420 statement. */ + await RunLiveAsync(new[] { ProcedureSeedSql, QueryStatsSeedSql }, async (connection, ct) => + { + var plan = await ExplainAsync(connection, OldProcedureStatsComparisonSql, AddComparisonParameters, ct); + var (scanned, total) = await ChunksReadAsync(connection, plan, "query_stats", ct); + Assert.True(scanned > WindowChunkCeiling, $"the old lookup read only {scanned} of {total} chunks; the seed must make it walk the history."); + Assert.Contains(Nodes(plan), n => Text(n, "Join Filter").Contains("DISTINCT FROM", StringComparison.Ordinal)); + }); + } + + [Fact] + public async Task TheComparison_AnswersTheOldStatementsAnswer_OnEveryColumn_ExceptTheDocumentedTextChange() + { + await RunLiveAsync(new[] { ProcedureSeedSql, QueryStatsSeedSql }, async (connection, ct) => + { + var old = await RunComparisonAsync(connection, OldProcedureStatsComparisonSql, ct); + var current = await RunComparisonAsync(connection, ViewerDataService.ProcedureStatsComparisonSql, ct); + + /* the seed has to hold what the facts claim, or equality proves nothing. */ + Assert.True(old.Count >= 100, $"the old statement returned {old.Count} rows."); + Assert.Contains(old.Keys, k => k.StartsWith("|", StringComparison.Ordinal)); /* a NULL database */ + Assert.Contains(old.Keys, k => k.Contains("||", StringComparison.Ordinal)); /* a NULL schema */ + Assert.Contains(old.Keys, k => k.EndsWith("|", StringComparison.Ordinal)); /* a NULL object */ + Assert.Contains(old.Keys, k => k.EndsWith("|dbo|", StringComparison.Ordinal)); /* an empty-string object */ + Assert.Contains(old.Values, r => r[3] == "" && r[7] != ""); /* GONE */ + Assert.Contains(old.Values, r => r[3] != "" && r[7] == ""); /* NEW */ + + Assert.Equal(old.Keys, current.Keys); + + var changed = new List(); + foreach (var (key, oldRow) in old) + { + var newRow = current[key]; + for (var i = 0; i < 11; i++) + { + Assert.True(oldRow[i] == newRow[i], $"{key}: column {i} was {oldRow[i]}, is now {newRow[i]}."); + } + + if (oldRow[11] != newRow[11]) + { + changed.Add(key); + Assert.Equal("", newRow[11]); /* only ever text -> none */ + Assert.Contains("only before the window", oldRow[11], StringComparison.Ordinal); + } + } + + /* procedures 1-5 are the only ones whose sole text predates the window; every other text (the dim-resolved + 6-10 included) is the old statement's, character for character. */ + Assert.Equal(5, changed.Count); + Assert.Contains(current.Values, r => r[11].Contains("payload dimension", StringComparison.Ordinal)); + }); + } + + /* ───────────────────────────── the Query Store top fallback ───────────────────────────── */ + + private static void AddViewerQueryStoreParameters(NpgsqlCommand command) + { + command.Parameters.Add(new NpgsqlParameter { TypedValue = ServerId }); + command.Parameters.Add(Timestamp(BaselineStart)); + command.Parameters.Add(Timestamp(End)); + command.Parameters.Add(new NpgsqlParameter { TypedValue = 100 }); + command.Parameters.Add(DatabaseFilter.All.Parameter()); + command.Parameters.Add(new NpgsqlParameter { TypedValue = 100 }); + } + + private static void AddMcpQueryStoreParameters(NpgsqlCommand command) + { + command.Parameters.Add(new NpgsqlParameter { TypedValue = ServerId }); + command.Parameters.Add(Timestamp(BaselineStart)); + command.Parameters.Add(Timestamp(End)); + command.Parameters.Add(new NpgsqlParameter { TypedValue = 100 }); + command.Parameters.Add(DatabaseFilter.All.Parameter()); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Text, Value = DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Text, Value = DBNull.Value }); + command.Parameters.Add(new NpgsqlParameter { TypedValue = 100 }); + } + + private static async Task> QueryStoreTextsAsync(NpgsqlConnection connection, string sql, Action bind, CancellationToken ct) + { + await using var command = new NpgsqlCommand(sql, connection) { CommandTimeout = 120 }; + bind(command); + var texts = new Dictionary(); + await using var reader = await command.ExecuteReaderAsync(ct); + var query = reader.GetOrdinal("query_id"); + var text = reader.GetOrdinal("query_text"); + while (await reader.ReadAsync(ct)) + { + if (reader.IsDBNull(query)) + { + continue; /* the candidate-count row */ + } + + texts[reader.GetInt64(query)] = reader.IsDBNull(text) ? null : reader.GetString(text); + } + + return texts; + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public async Task TheQueryStoreTopFallback_ReadsOnlyTheWindowsChunks_InTheViewerAndTheMcpCopy(bool viewer) + { + var (sql, bind) = viewer + ? (ViewerDataService.QueryStoreTopSql, (Action)AddViewerQueryStoreParameters) + : (DarlingDataReader.QueryStoreTopSql, AddMcpQueryStoreParameters); + await RunLiveAsync(new[] { QueryStoreSeedSql }, async (connection, ct) => + { + var plan = await ExplainAsync(connection, sql, bind, ct); + var (scanned, total) = await ChunksReadAsync(connection, plan, "query_store_stats", ct); + Assert.True(total >= 12, $"the seed must span many chunks (found {total}) or this proves nothing."); + Assert.True(scanned is > 0 and <= WindowChunkCeiling, + $"the read took {scanned} of {total} query_store_stats chunks; a 48 h window touches at most {WindowChunkCeiling}, text fallback included."); + }); + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public async Task TheQueryStoreTopFallback_KeepsTextFromInsideTheWindow_AndShowsNoneForTextOnlyBeforeIt(bool viewer) + { + var (sql, bind) = viewer + ? (ViewerDataService.QueryStoreTopSql, (Action)AddViewerQueryStoreParameters) + : (DarlingDataReader.QueryStoreTopSql, AddMcpQueryStoreParameters); + await RunLiveAsync(new[] { QueryStoreSeedSql }, async (connection, ct) => + { + var texts = await QueryStoreTextsAsync(connection, sql, bind, ct); + Assert.Equal(60, texts.Count); + for (var q = 1; q <= 30; q++) + { + /* the newest in-window row's text: collection_time DESC, and the window's last row is End itself. */ + Assert.Equal($"SELECT {q} /* 2026-02-20 12:00 */", texts[q]); + } + + for (var q = 31; q <= 60; q++) + { + Assert.Null(texts[q]); + } + }); + } + + [Fact] + public void EveryQueryStoreTopStatement_CarriesTheWindowBoundOnItsInlineTextFallback() + { + /* The viewer's raw and table reads share one tail; the MCP reader's raw, table and daily reads share another. Both + carry the bound on the fallback's own scan. */ + foreach (var sql in new[] + { + ViewerDataService.QueryStoreTopSql, ViewerDataService.QueryStoreTopTableSql, + DarlingDataReader.QueryStoreTopSql, DarlingDataReader.QueryStoreTopTableSql, DarlingDataReader.QueryStoreTopDailyTableSql, + }) + { + var normalized = sql.ReplaceLineEndings("\n"); + var fallback = normalized[normalized.IndexOf("FROM query_store_stats AS s", StringComparison.Ordinal)..]; + fallback = fallback[..fallback.IndexOf("LIMIT 1", StringComparison.Ordinal)]; + Assert.Contains("s.collection_time >= $2", fallback, StringComparison.Ordinal); + Assert.Contains("s.collection_time <= $3", fallback, StringComparison.Ordinal); + } + } + + /* ───────────────────────────── review round 1 (#5420): the gap, the tie, the twins, the filter ───────────────────────────── */ + + /// Two 24 h windows that each touch at most 2 one-day chunks. + private const int TwoWindowChunkCeiling = 4; + + [Fact] + public async Task TheComparison_WithABaselineAWeekBack_ReadsTheTwoWindowsChunksAndNotTheGapBetweenThem() + { + await RunLiveAsync(new[] { ProcedureSeedSql, QueryStatsSeedSql }, async (connection, ct) => + { + var baselineStart = End.AddDays(-8); + var baselineEnd = End.AddDays(-7); + void Bind(NpgsqlCommand c) => BindComparison(c, baselineStart, baselineEnd, DatabaseFilter.All); + + /* the baseline must hold procedures, or a read that skips it proves nothing. */ + var rows = await RunKeyedAsync(connection, ViewerDataService.ProcedureStatsComparisonSql, Bind, 3, ct); + Assert.Contains(rows.Values, r => r[7] != "" && r[3] != ""); + + var plan = await ExplainAsync(connection, ViewerDataService.ProcedureStatsComparisonSql, Bind, ct); + var (scanned, total) = await ChunksReadAsync(connection, plan, "query_stats", ct); + Assert.True(total >= 12, $"the seed must span many chunks (found {total}) or this proves nothing."); + Assert.True(scanned is > 0 and <= TwoWindowChunkCeiling, + $"the text lookup read {scanned} of {total} query_stats chunks; two 24 h windows touch at most {TwoWindowChunkCeiling}, and the span between them 9."); + }); + } + + [Fact] + public async Task WhenOneProceduresStatementsShareAnInstant_TheRowCollectedLastSuppliesTheText() + { + await RunLiveAsync(new[] { ProcedureSeedSql, QueryStatsSeedSql, TieSeedSql }, async (connection, ct) => + { + var rows = await RunComparisonAsync(connection, ViewerDataService.ProcedureStatsComparisonSql, ct); + Assert.Equal("tie A: the higher collection id", rows["db1|dbo|proc_tie_a"][11]); + Assert.Equal("tie B: the higher collection id", rows["db1|dbo|proc_tie_b"][11]); + }); + } + + [Fact] + public async Task TheComparison_UnderADatabaseFilter_AnswersTheOldStatementsAnswer_ExceptTheDocumentedTextChange() + { + await RunLiveAsync(new[] { ProcedureSeedSql, QueryStatsSeedSql }, async (connection, ct) => + { + var filter = DatabaseFilter.One("db1"); + void Bind(NpgsqlCommand c) => BindComparison(c, BaselineStart, CurrentStart, filter); + var old = await RunKeyedAsync(connection, OldProcedureStatsComparisonSql, Bind, 3, ct); + var current = await RunKeyedAsync(connection, ViewerDataService.ProcedureStatsComparisonSql, Bind, 3, ct); + + Assert.True(old.Count >= 20, $"the filtered old statement returned {old.Count} rows."); + Assert.All(old.Keys, k => Assert.StartsWith("db1|", k, StringComparison.Ordinal)); + Assert.Equal(old.Keys, current.Keys); + foreach (var (key, oldRow) in old) + { + for (var i = 0; i < 11; i++) + { + Assert.True(oldRow[i] == current[key][i], $"{key}: column {i} was {oldRow[i]}, is now {current[key][i]}."); + } + + if (oldRow[11] != current[key][11]) + { + Assert.Equal("", current[key][11]); /* only ever text -> none */ + Assert.Contains("only before the window", oldRow[11], StringComparison.Ordinal); + } + } + }); + } + + /// Hash or merge joins (any join type) whose condition names . + private static int KeyedJoins(JsonElement plan, string key) => + Nodes(plan).Count(n => Text(n, "Node Type") is "Hash Join" or "Merge Join" + && (Text(n, "Hash Cond") + Text(n, "Merge Cond")).Contains(key, StringComparison.Ordinal)); + + [Theory] + [InlineData(true)] + [InlineData(false)] + public async Task TheTwinComparison_JoinsItsPeriodsOnPlainEqualities_NotANullSafeJoinFilter(bool queryStats) + { + await RunLiveAsync(new[] { queryStats ? QueryStatsTwinSeedSql : QueryStoreTwinSeedSql }, async (connection, ct) => + { + var sql = queryStats ? ViewerDataService.QueryStatsComparisonSql : ViewerDataService.QueryStoreComparisonSql; + var plan = await ExplainAsync(connection, sql, AddComparisonParameters, ct); + Assert.DoesNotContain(Nodes(plan), n => Text(n, "Join Filter").Contains("DISTINCT FROM", StringComparison.Ordinal)); + var keyedJoins = KeyedJoins(plan, "query_hash"); + Assert.True(keyedJoins >= 2, $"expected both period joins as hash or merge joins on query_hash, found {keyedJoins}."); + + /* the oracle's own shape, so the assertions above cannot pass vacuously on this seed. */ + var oldPlan = await ExplainAsync(connection, queryStats ? OldQueryStatsComparisonSql : OldQueryStoreComparisonSql, AddComparisonParameters, ct); + Assert.Contains(Nodes(oldPlan), n => Text(n, "Join Filter").Contains("DISTINCT FROM", StringComparison.Ordinal)); + }); + } + + [Theory] + [InlineData(true, false)] + [InlineData(true, true)] + [InlineData(false, false)] + [InlineData(false, true)] + public async Task TheTwinComparison_AnswersTheOldStatementsAnswer_OnEveryColumn_NullKeysIncluded(bool queryStats, bool filtered) + { + await RunLiveAsync(new[] { queryStats ? QueryStatsTwinSeedSql : QueryStoreTwinSeedSql }, async (connection, ct) => + { + var filter = filtered ? DatabaseFilter.One("db1") : DatabaseFilter.All; + void Bind(NpgsqlCommand c) => BindComparison(c, BaselineStart, CurrentStart, filter); + var old = await RunKeyedAsync(connection, queryStats ? OldQueryStatsComparisonSql : OldQueryStoreComparisonSql, Bind, 2, ct); + var current = await RunKeyedAsync(connection, queryStats ? ViewerDataService.QueryStatsComparisonSql : ViewerDataService.QueryStoreComparisonSql, Bind, 2, ct); + + /* the seed has to hold what the facts claim, or equality proves nothing. */ + Assert.True(old.Count >= (filtered ? 20 : 100), $"the old statement returned {old.Count} rows."); + Assert.Contains(old.Values, r => r[3] == "" && r[7] != ""); /* GONE */ + Assert.Contains(old.Values, r => r[3] != "" && r[7] == ""); /* NEW */ + if (!filtered) + { + Assert.Contains(old.Keys, k => k.StartsWith("|", StringComparison.Ordinal)); /* a NULL database */ + Assert.Contains(old.Keys, k => k.EndsWith("|", StringComparison.Ordinal)); /* a NULL hash */ + Assert.Contains(old.Keys, k => k == "|"); /* both NULL */ + /* an empty key and a NULL one that must stay apart, both in the seed's tables (so the lookup has the temptation) and one + of each pair in the answer: the heavy NULL-hash group and the heavy empty database are compared, the light empty + hash and the light NULL database are trimmed, and a join that merged them would fan a heavy row out. */ + var (table, emptyHashDb, emptyDbHash) = queryStats ? ("collect.query_stats", "db0", "0xQ127") : ("collect.query_store_stats", "db2", "h127"); + Assert.Contains(old.Keys, k => k == $"{emptyHashDb}|"); + Assert.Contains(old.Keys, k => k == $"|{emptyDbHash}"); + Assert.DoesNotContain(old.Keys, k => k == $"{emptyHashDb}|"); + Assert.DoesNotContain(old.Keys, k => k == $"|{emptyDbHash}"); + await using var seeded = new NpgsqlCommand( + $"SELECT count(*) FROM {table} WHERE (database_name = '{emptyHashDb}' AND query_hash = '') OR (database_name IS NULL AND query_hash = '{emptyDbHash}')", connection); + Assert.True((long)(await seeded.ExecuteScalarAsync(ct))! > 0, "the empty-versus-NULL seed rows are missing."); + } + + Assert.Equal(old.Keys, current.Keys); + foreach (var (key, oldRow) in old) + { + for (var i = 0; i < oldRow.Length; i++) + { + Assert.True(oldRow[i] == current[key][i], $"{key}: column {i} was {oldRow[i]}, is now {current[key][i]}."); + } + } + }); + } + + [Theory] + [InlineData(0)] + [InlineData(1)] + [InlineData(2)] + public async Task TheComparison_KeepsANullKeyAndItsEmptyKeyPartnerApart_WhenBothMakeTheTopHundredInBothWindows(int twin) + { + var (seed, sql, keyColumns, expected) = twin switch + { + 0 => (QueryStatsNullEmptySeedSql, ViewerDataService.QueryStatsComparisonSql, 2, new[] { "dbA|", "dbA|", "|hC", "|hC", "|", "|" }), + 1 => (QueryStoreNullEmptySeedSql, ViewerDataService.QueryStoreComparisonSql, 2, new[] { "dbA|", "dbA|", "|hC", "|hC", "|", "|" }), + _ => (ProcedureNullEmptySeedSql, ViewerDataService.ProcedureStatsComparisonSql, 3, new[] { "dbA|dbo|", "dbA|dbo|", "|dbo|pX", "|dbo|pX", "dbA||pX", "dbA||pX" }), + }; + await RunLiveAsync(new[] { seed }, async (connection, ct) => + { + /* the outer join is still a hash or merge FULL JOIN with both halves (Postgres refuses a FULL JOIN it cannot hash or merge). */ + var plan = await ExplainAsync(connection, sql, AddComparisonParameters, ct); + Assert.Contains(Nodes(plan), n => Text(n, "Node Type") is "Hash Join" or "Merge Join" && Text(n, "Join Type") == "Full"); + + /* RunKeyedAsync adds each row under its key, so a fanned-out row (the same key twice) fails right there. */ + var rows = await RunKeyedAsync(connection, sql, AddComparisonParameters, keyColumns, ct); + Assert.Equal(expected.OrderBy(k => k, StringComparer.Ordinal), rows.Keys); + for (var i = 0; i < expected.Length; i++) + { + /* each group is paired with its OWN baseline: the counts are 100 to 600 now and a tenth of that before, in key order of the seed. */ + var current = (100 * (i + 1)).ToString(CultureInfo.InvariantCulture); + var baseline = (10 * (i + 1)).ToString(CultureInfo.InvariantCulture); + Assert.Equal(current, rows[expected[i]][3]); + Assert.Equal(baseline, rows[expected[i]][7]); + } + }); + } +} diff --git a/Darling/Darling.Tests/ViewerTimeRangeGapFixTests.cs b/Darling/Darling.Tests/ViewerTimeRangeGapFixTests.cs new file mode 100644 index 0000000000..b158a21e79 --- /dev/null +++ b/Darling/Darling.Tests/ViewerTimeRangeGapFixTests.cs @@ -0,0 +1,353 @@ +/* + * Copyright (c) 2026 Erik Darling, Darling Data LLC + * + * This file is part of the SQL Server Performance Monitor. + * + * Licensed under the MIT License. See LICENSE file in the project root for full license information. + */ + +using System; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Text.RegularExpressions; +using System.Threading; +using System.Threading.Tasks; +using System.Windows.Controls; +using Npgsql; +using PerformanceMonitor.Darling.Storage; +using PerformanceMonitor.Darling.Viewer; +using PerformanceMonitor.Ui; +using Xunit; + +/* Darling.Tests only: a Darling file Lite.Tests compiles must be named in build.yml's Lite path filter. */ +namespace Darling.Tests; + +/// +/// #5562, the four gaps the first Viewer pass left. R5: a FinOps picker offers only a range its "hours back from now" read can +/// honor. R6: Job History's finished range is bounded in the SQL, before the row cap. R7: every "Showing since" banner site feeds +/// the picker's data-start note. R8: Alert History's longest choice is the span its table keeps. +/// +[Collection("live-postgres")] +public sealed class ViewerTimeRangeGapFixTests +{ + private static string ViewerFile(string file) => RepoFile.ReadRepoFile("Darling", "PerformanceMonitor.Darling.Viewer", file); + + private static string Code(string file) => ViewerTypedRangeTests.StripComments(ViewerFile(file)); + + private static TimeRangeSpec Spec(string id) + { + Assert.True(TimeRangeSpec.TryFromId(id, out var spec), id); + return spec!; + } + + // ── R5: FinOps pickers are rolling-only, in whole hours (the heatmap in whole days) ── + + [Theory] + [InlineData("1h", true)] + [InlineData("4h", true)] + [InlineData("1d", true)] + [InlineData("1w", true)] + [InlineData("1mo", true)] + [InlineData("90m", false)] + [InlineData("30m", false)] + [InlineData("5m", false)] + [InlineData("today", false)] + [InlineData("previous-week", false)] + [InlineData("since:2026-10-01T04:00:00Z", false)] + [InlineData("fixed:2026-10-01T04:00:00Z/2026-10-02T04:00:00Z", false)] + public void TheHourRule_AdmitsOnlyAWholeNumberOfHoursBackFromNow(string id, bool admitted) + { + var spec = id == "90m" ? TimeRangeSpec.Relative(TimeSpan.FromMinutes(90)) : Spec(id); + Assert.Equal(admitted, RollingUnitRule.Admits(spec, RollingUnitRule.Hour)); + Assert.Equal(admitted, RollingUnitRule.Refusal(spec, RollingUnitRule.Hour) is null); + } + + [Theory] + [InlineData("1d", true)] + [InlineData("2d", true)] + [InlineData("1mo", true)] + [InlineData("1h", false)] + [InlineData("4h", false)] + public void TheDayRule_AdmitsOnlyAWholeNumberOfDaysBackFromNow(string id, bool admitted) + { + var spec = Spec(id); + Assert.Equal(admitted, RollingUnitRule.Admits(spec, RollingUnitRule.Day)); + Assert.False(RollingUnitRule.Admits(TimeRangeSpec.Relative(TimeSpan.FromHours(36)), RollingUnitRule.Day)); + Assert.False(RollingUnitRule.Admits(TimeRangeSpec.ForPeriod(CalendarPeriod.MonthToDate), RollingUnitRule.Day)); + } + + [Fact] + public void TheRefusal_SaysWhatTheReadTakes() + { + Assert.Equal("The shortest range for this read is 1 hour.", RollingUnitRule.Refusal(TimeRangeSpec.Relative(TimeSpan.FromMinutes(30)), RollingUnitRule.Hour)); + Assert.Equal("This read takes whole hours; try 2h.", RollingUnitRule.Refusal(TimeRangeSpec.Relative(TimeSpan.FromMinutes(90)), RollingUnitRule.Hour)); + Assert.Contains("length back from now", RollingUnitRule.Refusal(TimeRangeSpec.ForPeriod(CalendarPeriod.Today), RollingUnitRule.Hour), StringComparison.Ordinal); + Assert.Equal("The shortest range for this read is 1 day.", RollingUnitRule.Refusal(TimeRangeSpec.Relative(TimeSpan.FromHours(4)), RollingUnitRule.Day)); + } + + [Fact] + public void ARollingOnlyPicker_RefusesWhatItsReadCannotHonor_AndHidesTheCalendar() + { + StaTestThread.Run(() => + { + var now = new DateTime(2026, 10, 8, 11, 1, 0, DateTimeKind.Unspecified); + var picker = new TimeRangePicker { ZoneProvider = () => TimeZoneInfo.Utc, NowProvider = () => now, Compact = true }; + picker.RollingUnit = RollingUnitRule.Hour; + var raised = 0; + picker.RangeChanged += (_, _) => raised++; + + Assert.Equal(System.Windows.Visibility.Collapsed, picker.PeriodPanel.Visibility); + Assert.Equal(System.Windows.Visibility.Collapsed, picker.PickCalendar.Visibility); + Assert.Equal(System.Windows.Visibility.Collapsed, picker.CalendarHeader.Visibility); + + /* Presets under an hour are greyed; the others stay. */ + var enabled = picker.RollingPanel.Children.OfType