Repository navigation
Tests stop sharing WPF threads, Postgres roles and runner CPU - #5604
Merged
Merged
Conversation
A shared background STA thread runs every WPF test body, one at a time, behind a caller-held gate, and drains the dispatcher after each body. A hung body fails its own test by name and the later ones fail fast. The 54 per-file STA helpers and WpfStaGate are gone, and StaThreadCensusTests (both test projects) keeps them from coming back.
…re (#5602, #5600) Classes that time their own work join the one non-parallel timing collection (it replaces the redactor and warm-up collections). The cloud identity probe test stamps the probe's return where it completes. TimingTestCensusTests in both test projects fails a class that starts a clock outside the collection unless it is on an allow list with its reason.
…run alone, and a census keeps it that way (#5602) Roles, grants and ALTER SYSTEM are cluster-wide, so scratch databases do not isolate them. Classes that provision or alter the fixed admin/viewer/mcp roles, or change cluster settings, move into one pg-cluster-roles collection that runs with DisableParallelization. The V147 re-assertion fact now drops the viewer/mcp roles it made (they made later classes skip). StoreStatementStatsLiveTests names its roles and database per run. PgClusterObjectCensusTests fails when a live class does otherwise.
…e timing collection (#5602)
5 of 6 tasks
… copy phase drops its upper clock bound; sample-text allow entries pin their statements (#5602)
erikdarlingdata
marked this pull request as ready for review
October 9, 2026 00:40
erikdarlingdata
enabled auto-merge (squash)
October 9, 2026 00:55
This was referenced Oct 9, 2026
4 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5602
Fixes #5600
Test classes were sharing WPF threads, Postgres cluster objects and runner CPU, and that sharing produced a new flake every day. Each lane owns one section below.
CHANGELOG
SECTION: None
ENTRY: - Test isolation ([#5602]) - tests only, with no product change.
WPF (lane a)
Every WPF test body now runs on ONE shared background STA thread per test process, instead of each of 54 test files starting its own thread behind
WpfStaGate.What changed
Darling/Darling.Tests/StaTestThread.cs(linked into Lite.Tests; the filter inbuild.ymlandLite.Tests.csprojfollow):StaTestThread.Run(Action),Run<T>(Func<T>)andRun(Func<Task>). The thread starts lazily and runsDispatcher.Run().PushFrame) can never let a second one in (Lite test flake: ScreenReaderNamesFollowUpTests hits a WPF dictionary from two threads #5559). The gate replacesWpfStaGate.ExceptionDispatchInfo)..GetAwaiter().GetResult()of a task whose awaits capture the dispatcher's context would wait for the thread it is blocking. Async bodies (Run(Func<Task>)) keep the dispatcher's context.OnStaThreadcopies,WpfStaGate.csand bothWpfStaGateCensusTestsare removed. A few files keep a small wrapper because they set something per body:ViewerCollectorDurationHoverTests.InEnglish(sets and restores the thread's culture, which the shared thread would otherwise keep),ViewerCollectionHealthLoadTests.OnTabDispatcher(display mode UTC, async body),StandalonePlanViewerPlusTabTests.OnRig(builds the rig on the thread; the oldInvokeShutdownin itsfinallyis gone, a shared dispatcher must not be shut down). The two live files that ran their ownDispatcher.Run()or thread (ViewerCollectionLogDataStartLiveTests,ViewerQueryGridsDataStartLiveTests) callStaTestThread.Rundirectly.StaThreadCensusTests(both projects,Stage=Guard) replacesWpfStaGateCensusTests. It reads code only (throughCSharpSourceWalker) and fails, naming the file and the rule, when a test file outside the allow list callsSetApartmentState(ApartmentState.STA), defines anOnStaThread-style helper, or callsDispatcher.Run(). It also needs at least 20 files that use the host (anti-vacuity), and every allow-list entry needs a reason.StaTestThreadTests(Darling.Tests, in thetimingcollection because it judges 200 to 300 ms limits): bodies run in order on one background STA thread, an exception keeps its stack and the thread survives it, a pumping body keeps a second caller out (3 callers x 3 bodies, never more than 1 inside), the hang guard names the test and later calls fail at once (on a host of its own, so the shared thread is never left stuck), the drain runs queued work and closes a window left open, an async body resumes on the shared thread, an exception from queued work reaches the test,EnterExclusivekeeps a body out.Allow list
StaTestThread.cs, its own testsStaTestThreadTests.cs, the census itself, andWpfExitSwitchTests.cs(Test processes no longer pay 0.3 s per WPF STA thread at exit, which failed green Darling runs (#5596) #5598: it needs an STA thread that has ENDED, which the shared thread never does). That file is not on this branch yet; it must takeStaTestThread.EnterExclusive()around its own thread (the census checks that once the file exists).WpfExitSwitchTests.csstill callsWpfStaGate.Enter(). Change that one call toStaTestThread.EnterExclusive(); nothing else needs touching.RED-plant. A planted file with an own-thread helper, an
ApartmentState.STAstart and aDispatcher.Run()fails the census at my head in both projects, once per rule, naming the file. On the origin/dev tree (scanned with a Python port of the same rules, comment lines dropped, not by building dev) the census would flag Darling.Tests: 28 files starting a thread, 26 own helpers, 2Dispatcher.Run(); Lite.Tests: 26, 26, 0.Tests (Total / Failed, at the pushed head; live classes skip with no Postgres rig)
StaTestThreadTests, bothStaThreadCensusTests,TimingTestCensusTests: 16 / 0 (Darling.Tests).*Census*+*Ratchet*+CommentFilterAdoptionTests+DarlingReadsTraitGuardTests+CrossAppGuardCiGateTests: Darling.Tests 518 / 0 (1 skipped), Lite.Tests*Census*+*Ratchet*67 / 0.-trait Stage=Guard -failSkips: Darling.Tests 1438 / 0, Lite.Tests 450 / 0.ViewerCollectionLogDataStartLiveTestsdrill and banner paths,ViewerQueryGridsDataStartLiveTests.ReadBanner,ViewerBlockedProcessReportsCoverageLiveTests,ViewerJobHistoryDataStartLiveTests,OverviewBlockingLaneDataStartLiveTests): they skip here, CI runs them.Check on CI: the async drill path in
ViewerCollectionLogDataStartLiveTestsis the one conversion that changed an STA shape (it ranDispatcher.Run()on its own thread; it now awaits on the shared dispatcher).Postgres (lane b)
What changed
pg-cluster-roles, declared withDisableParallelization(Darling.Tests/PgClusterRolesCollection.cs). xunit runs it alone, after the parallel classes. It has no collection fixture: its members build their own scratch database or change only cluster settings, and a secondLivePostgresStoreFixturewould run its residue check while parallel collections still write the store.ScratchPostgresisolates tables, not the cluster's catalogs. The product provisions FIXED role names (admin,viewer,mcp), so any test that runs its provisioning batch, or creates, drops or alters those roles, shares them with every other such test whatever database it aims at.pg-cluster-roles:PasswordKeyTablesLiveTests,DarlingSecuritySplitLiveTests,ProvisioningBatchLockedTransactionLiveTests(the second class inProvisioningSerializationTests.cs),StoreServerPasswordRulesLiveTests,GetReadLatencyLiveTests(it had no collection at all),ComposeStatementTimeoutV147MigrationLiveTests(no collection either).DarlingStoreHostProfileLiveTests(ALTER SYSTEM SET max_wal_size),PgServerConfigPendingRestartLiveTests(ALTER SYSTEMonshared_buffers),PgSettingRedactionLivePostgresTests(ALTER SYSTEM SET primary_conninfo).viewerandmcpand never dropped them. The classes that provision the managed roles skip themselves when those names already exist, so after V147 ran they quietly skipped (24 role-exists skips on the first run on a fresh rig, none after the fix). The fact now records which of the two it made and drops only those, throughLiveStoreCleanup.RunAsync.StoreStatementStatsLiveTestscreated rolespgss_reader,pgss_outsider,pgss_plainownerand the databasepgss_plainunder fixed names. They now carry a per-run suffix.aws_role_,srv_add_,tag_core_,dismiss_only_,host_mcp_,sec_admin_and the like, Darling.Tests: a full local run leaves state that fails later runs #4981), so they need nothing.LivePostgresCollectionHygieneTestsnow accepts[Collection("pg-cluster-roles")]as a recorded decision (it runs alone, which is stronger thanlive-postgres).PgClusterObjectCensusTests(Darling.Tests; Lite.Tests has no Postgres tests). It reads each class that reaches theDARLING_TEST_PGcluster, strips comments, and fails on CREATE/ALTER/DROP ROLE with a literal name, DROP OWNED BY, GRANT ... TO / REVOKE ... FROM a literal role,ALTER SYSTEM SET/RESET,ALTER DATABASE x SET,ON DATABASE xwith a literal name, the provisioning entry points, and theDarlingManagedPostgres.*RoleNameconstants, unless the class is inpg-cluster-rolesor on the allow list. A stale allow-list entry fails too. A line withAssert.orRegexis skipped (text the test only checks).Audit (every test that creates, alters, grants, revokes or drops a role, membership, database grant, role setting or cluster setting)
ComposeStoreRolesLiveTests,ManagedConfVerdictsLiveTests,ScramVerifierLiveTests,DarlingManagedPostgresTests,DarlingStoreUpgradeTests. The census does not reach them.AdminServersRouteLiveTests,AwsPerServerRoleLiveTests,CustomAlertResolveGrantLiveTests,DarlingMcpStoreHostToolsLiveTests,DarlingPasswordKeyStoreLiveTests,DiagnosticsBundleLeakLiveTests,PgReadBinaryFileCapabilityLiveTests,StoreInstallIdRefusedClusterLiveTests,ValidateConfigOldFormatPasswordLiveTests,ServerEditViewerRoleLiveTests.PgClusterObjectCensusTests.AllowList, each entry has its reason):MigrationUpgradeLadderLiveTests(CREATE ROLE darling only if missing; on a CI cluster it exists),AlertHistoryDismissTests,ViewerStoreLoginRawReadLiveTests,ServerAddViewerRoleLiveTests,ServerEditLiveTests,ServerTagMcpRoleLiveTests,ServerTagViewerRoleLiveTests,ComposeTempFileLimitLiveTests(shipped GRANT or role text retargeted at a per-run role, never the fixed roles),PgStatementTextScrubLiveTests,StatementAnalysisReadCutLiveTests,StoreLogRemaskLiveTests,StoreStatementStatsLiveTests(sample statement text fed to a scrubber as data).RED-plants
DARLING_TEST_PGand holdsCREATE ROLE planted_literal LOGINoutside the collection fails the census with the file, line, class and rule. Removed afterwards.StoreStatementStatsLiveTests.Test plan
darling), 12 classes in one run: Total 89, Failed 0, Skipped 1 (the compressed-hypertable fact skips because the test database had no TimescaleDB extension; the live collection's fixture normally enables it). Roles left on the rig afterwards: none.StoreStatementStatsLiveTests+StoreStatementStatsTests: Total 41, Failed 0.*Census*,*Ratchet*and*Hygiene*classes: Total 599, Failed 0, Skipped 1 (existing).-trait Stage=Guard -failSkips: Total 1441, Failed 0, Skipped 0.Handoff notes for the seat
DarlingSecuritySplitLiveTestshas 10 tests but only one uses the fixed roles; splitting it would shave a little.GuardStageScanneronly knowslive-postgresas a live collection. Every moved class also readsDARLING_TEST_PG, which it already catches, so nothing is lost today.Timing (lane c)
What changed
timing, declared withDisableParallelization(Darling.Tests/TimingCollection.cs,Lite.Tests/TimingCollection.cs). xunit runs it alone, after the parallel classes. It replacespg-setting-redactor-timingandstatement-filter-warm-up: both old definitions are deleted and their classes (PgSettingRedactorTests,StatementFilterWarmUpWaitTests) joinedtiming. The warm-up class stages a process-wide fake warm-up; running alone intimingkeeps it apart from every other class, which is all its own collection did.timing(each judges a real upper time bound and owns its store or needs none):DarlingCloudIdentityProbeTests,PgLogEventsPipelineTests,PgSensitiveStatementFilterLiveTests,SensitiveStatementsParityLiveTests,SensitiveStatementAutoParamTests,StatementFilterAlertTests,StatementFilterWebTests,WebhookPostTimeoutTests,ScratchPostgresExitDrainTests,ScratchPostgresQuiesceLiveTests,TuningHourlyCreateLiveTests,RawRetentionPurgeJobLiveTests,BoundedCompressionDdlTests,QueryStoreIntervalBrinLiveTests.timing:CollectionHealthReadBehaviourTests,StatusBarSizeReadLockTests,DetachedCollectorGateTests,QueryStoreServerGateTests.DarlingCloudIdentityProbeTeststakes the return time from a synchronous continuation on the probe task, so the hop back into the test is no longer counted, and the class runs intiming. The stall-watcher thread from The cloud probe budget test stops counting machine stalls against the probe (#5600) #5601 is not taken: the bar stays 1000 ms with no allowance. The cloud probe budget test stops counting machine stalls against the probe (#5600) #5601 is to be closed in favour of this.TimingTestCensusTestsin each project, on a shared scanner (Darling.Tests/TimingTestScanner.cs, linked into Lite.Tests; the path is in build.yml'slite_linked_shard). It fails when a class starts a clock (Stopwatch.StartNew/GetTimestamp/GetElapsedTime,new Stopwatch,DateTime.UtcNow - started,Environment.TickCount; comments and string literals are blanked first, so source pins that quote the calls do not count) outside[Collection("timing")]and off the allow list. The rule is wider than "asserts on the time" on purpose: a diagnostic has to say it is one. Allow-list entries are per file with a reason, a stale entry fails (the list only shrinks), and the census checks the collection isDisableParallelizationand that the scan still sees at least 15 (Darling) / 2 (Lite) members.SensitiveStatementOutputFilterTests(best of three, throughTimingClaim) and the sensitive-statement perf tests already do. I added no new repeat to the wait-based budget tests (the probe, the webhook post, the drains): they wait real budgets of 200 ms to several seconds, so three runs would add minutes, and running alone is what removes the competition.Audit: every test file that starts a clock
Moved: the 18 classes above. Allow list (reasons are in
TimingTestCensusTests.s_allowed, 31 Darling files and 6 Lite files). In brief:TimingClaim.AtMost, which asserts only withDARLING_TIMING_TESTS=1: SensitiveStatementOutputFilterTests, SensitiveStatementsTests.CollectionResetGate(DisableParallelization for process-wide state): DuckDbSentinelConnection, PostCollectionCheckpointBudget, CollectionResetGate.LivePostgresCollectionHygieneTeststies them to the shared migrated store and a class has one collection): MigrationLockWaitContention (750 ms against a 1 s poll sleep), CollectionLogSegmentByLive (30 s), PgWaitSamplerLive, StatementFilterBlockingReadsLive (10 s), StatementFilterPlanReadsLive (15 s), StoreCopyPhaseLivePostgres, TuningReaderBudgetLive (half the command deadline).Left for a successor
The live-postgres members above still run in parallel with non-live classes. Their bounds are wide (10 s and up, or a server-side timeout with headroom) except
MigrationLockWaitContentionTests(750 ms) andStoreCopyPhaseLivePostgresTests. Moving them needs either a copy of the shared-store fixture intotimingor assertions on the work (the lock was taken on the first attempt, the deadline fired on its own clock). I did not do either here.Numbers (item 5)
BEFORE, from the last three green dev Build runs:
AFTER: to come from this PR's own run (not measured here; the laptop does not run the suite). The
timingclasses now run in one serial phase at the end of each shard, so a shard's wall time can rise by the sum of those classes' times (a few minutes at most) while the per-class variance falls.Tests run (laptop, build 0 warnings in both projects)
Darling.Tests: TimingTestCensusTests 5/0 (Total/Failed), DarlingCloudIdentityProbe 20/0, PgSettingRedactorTests 139/0, StatementFilterWarmUpWaitTests 18/0, WebhookPostTimeoutTests 7/0, StatementFilterAlertTests 19/0, SensitiveStatementAutoParamTests 44/0, StatementFilterWebTests 14/0, PgLogEventsPipelineTests 94/0, BoundedCompressionDdlTests 5/0 (2 skipped, no store), ScratchPostgresExitDrainTests 9/0 (2 skipped), all
*Census*490/0, all*Ratchet*15/0, LivePostgresCollectionHygiene 11/0,-trait Stage=Guard -failSkips1437/0.Lite.Tests: TimingTestCensusTests 3/0, CollectionHealthReadBehaviourTests 26/0, StatusBarSizeReadLockTests 1/0, DetachedCollectorGateTests 7/0, QueryStoreServerGateTests 13/0, GuardStageCensusTests 9/0, all
*Census*66/0, CrossAppGuardCiGate 19/0,-trait Stage=Guard -failSkips449/0.Not run: the live Postgres classes that got only an attribute (no rig, by the brief); the full suites (CI decides).
RED plants
Stopwatch.StartNew()added to each project. Darling's census failed namingZzTimingPlantTests.cs(5 run, 1 failed); Lite's failed the same way (3 run, 1 failed). Both plants removed.DarlingCloudIdentityProbeTests,PgLogEventsPipelineTests,WebhookPostTimeoutTestsand the rest of the unmoved classes; Lite's failed namingCollectionHealthReadBehaviourTests,DetachedCollectorGateTestsandStatusBarSizeReadLockTests.Merge (lane d)
dev was merged into this branch twice (the second time after the time range picker change landed). Branch head after this lane: see the PR.
What the censuses caught from dev, and what I did
WpfExitSwitchTests(from Test processes no longer pay 0.3 s per WPF STA thread at exit, which failed green Darling runs (#5596) #5598) still called the removedWpfStaGate.Enter(): nowStaTestThread.EnterExclusive(). The timing census then caught it (it starts a clock): it now carries[Collection("timing")], since it judges the fastest of three exit times. ItsLite.Testslink andbuild.ymlfilter entry were merged keeping both sides (the oldWpfStaGate.csentries are gone, the newStaTestThread.cs,TimingTestScanner.csandWpfExitSwitchTests.csentries stay).TimeRangePickerControlTestsandViewerTimeRangeGapFixTests, each with its ownOnStaThreadhelper that tookWpfStaGate. That broke the build, so both now callStaTestThread.Run(...)and the helpers are deleted, as lane a did elsewhere. The STA census passes with no allow-list entry.PgClusterObjectCensusTests, the Lite STA census and the Lite timing census caught nothing else from dev.Live classes run on a throwaway server (TimescaleDB and pg_stat_statements loaded, UTC, a
darlingdatabase present)Total/Failed: ViewerCollectionLogDataStartLive 9/0 (the riskiest conversion: its drill path now awaits on the shared dispatcher), ViewerPlanCorrectionsDataStartLive 5/0 (the class in the
ViewerQueryGridsDataStartLiveTestsfile), ViewerBlockedProcessReportsCoverageLive 17/0, ViewerJobHistoryDataStartLive 5/0, LiteOverviewBlockingLaneDataStartLiveTests12/0, PgLogEventsPipeline 94/0, PgSensitiveStatementFilterLive 2/0, SensitiveStatementsParityLive 5/0, ScratchPostgresExitDrain 9/0, ScratchPostgresQuiesceLive 4/0, TuningHourlyCreateLive 5/0, RawRetentionPurgeJobLive 2/0, BoundedCompressionDdl 5/0, QueryStoreIntervalBrinLive 31/0, DarlingCloudIdentityProbe 20/0. No failure, so nothing from this branch to fix.The two tight live-postgres timing checks
MigrationLockWaitContentionTests: the uncontended-acquire fact (750 ms against the one-second poll sleep) cannot drop its clock. The bug it guards is a poll sleep moved above the first attempt, which keeps the attempt count at 1, so no count of work tells it apart. It moved to a new class,MigrationLockUncontendedAcquireTimingTests, intiming, on its own scratch database (the migration lock is per database, so nothing there to contend with). The bar is unchanged. The three contention facts stay inlive-postgres, and their clock is gone, so that allow-list entry is deleted.StoreCopyPhaseLivePostgresTests.TheStartPhaseIsBounded...: the upper bound (the connection's command timeout) is dropped, because the exact fault type and exact message assertions already tell the sweep deadline from the connection's own timeout (the latter is anNpgsqlException). The lower bound stays (deadline less a second; load can only lengthen a wait). Its allow-list entry now says lower bound only.TimeoutException, gotNpgsqlException). The old upper bound caught only the third, and the type check still does.live-postgres: CollectionLogSegmentByLive, PgWaitSamplerLive, StatementFilterBlockingReadsLive, StatementFilterPlanReadsLive, TuningReaderBudgetLive. The shared reason in the timing census now says why: a second copy of the shared-store fixture intimingwould read the process-wide cleanup ledger again and report every live class's residue twice, and their bounds are 10 s and up against milliseconds of work.Cluster-object census
PgStatementTextScrubLiveTests,StatementAnalysisReadCutLiveTests,StoreLogRemaskLiveTests,StoreStatementStatsLiveTests) now pin the exact matching code lines (PinnedSampleText; 1, 1, 1 and 4 lines). A new matching line in one of those classes fails ("NOT PINNED"), and a pinned line that stops matching fails (EveryPinnedSampleStatement_StillMatchesInItsClass). RED plant: aDROP ROLEliteral added toPgStatementTextScrubLiveTestsfailed the census naming the file and line; reverted.DarlingSecuritySplitLiveTestsis NOT split. Only one fact uses the fixedadmin/viewer/mcproles, but the other facts runALTER DEFAULT PRIVILEGES FOR ROLE <owner> IN SCHEMA collect ...and grant and drop roles on the shared store. The default-privileges row is shared by every test run by that owner, so moving the class out ofpg-cluster-roleswould put those writes back in parallel with other classes, which is the kind of overlap that produced "tuple concurrently updated" (Darling test flake: viewer-role grant test fails with 'tuple concurrently updated' #5560). Leaving the whole class in the collection keeps them alone.Tests (laptop; the full suites were not run, CI decides). 0 build warnings in both projects.
Darling.Tests: all
*Census*495/0 (1 skipped, as before), all*Ratchet*15/0, all*Hygiene*90/0, StaThreadCensus 3/0, TimingTestCensus 5/0, PgClusterObjectCensus 4/0, WpfExitSwitch 5/0, TimeRangePickerControl 8/0, ViewerTimeRangeGapFix 25/0 (1 skipped), MigrationLock 4/0, StoreCopyPhase 15/0, LivePostgresCollectionHygiene 11/0,-trait Stage=Guard -failSkips1444/0.Lite.Tests: all
*Census*67/0, CrossAppGuardCiGate 19/0, StaThreadCensus 3/0, TimingTestCensus 3/0, WpfExitSwitch 3/0,-trait Stage=Guard -failSkips450/0.The throwaway server was stopped and its directory deleted; only the bootstrap role existed, no test role or scratch database was left.
Not done:
AggregateJobQuiesceLiveTests(#5603, another seat).