Skip to content

Tests share WPF threads, Postgres roles and runner CPU by design: give each test its own, and guard it #5602

Closed

Description

@erikdarlingdata

Claude posting for Erik Darling.

What happens

The Darling.Tests and Lite.Tests suites fail CI for reasons that have nothing to do with the change. Sometimes no test fails at all. Each failure blocks every merge until a lane fixes it. Today there were four:

Issue Shared thing What collided
#5559 WPF state Two test classes used a WPF dictionary from two threads at the same time.
#5560 Postgres cluster Two tests changed the same grants at the same time, and Postgres refused one with "tuple concurrently updated".
#5561 Runner CPU A timing test on a 27 MB plan went over its limit on a busy runner.
#5596 WPF state All tests passed, but the process took longer than 11 seconds to exit. Each STA test thread left a WPF exit handler that waits 300 ms.

The fixes (#5567, #5566, #5563, #5598) each remove one collision. The design that causes the collisions stays, so the next test that uses one of these shared things can fail the same way.

Cause

The tests share three things by design.

  • WPF threads. 55 test files call SetApartmentState(ApartmentState.STA) and start their own thread. Several files carry their own private copy of an OnStaThread helper. WpfStaGate (Lite and Darling WPF tests take one gate so parallel STA classes stop racing a shared WPF cache (#5559) #5567) now makes these tests take turns, but each call still starts a new thread. So the WPF state per thread and the exit handlers per thread grow with the number of tests.
  • The Postgres cluster. ScratchPostgres gives a test its own database. Roles and the grants on a database are stored for the whole cluster. So tests that run at the same time can still change the same rows.
  • Runner CPU. About 45 test files compare an elapsed time with a limit, and they run in parallel with other tests. This count comes from a grep, not a full audit.

No open issue covers this.

What to fix

  1. Run every WPF test on one STA thread for each test process. The test host starts that thread with a running Dispatcher at the first use. One shared helper runs a test body on it. Remove the private OnStaThread copies. Remove WpfStaGate when nothing uses it. Some tests must have their own thread, for example a test of dispatcher shutdown. Put each of them on an allow list with the reason.
  2. Make each test change only Postgres objects that no other test can see. Audit every test that creates or changes a role or a grant. Give each such object a name that includes the scratch database name. If product code changes an object that every test shares, put those tests in one collection that runs alone.
  3. Stop timing tests from competing for CPU. Some tests can assert on the work done: rows read, plan nodes walked, or calls made. Change those tests to do that. Put each test that must check an elapsed time in one collection with DisableParallelization = true. Judge the fastest of three runs, as The 27 MB plan walk and 20 MB cut-walk timing pins judge the fastest of three runs (#5561) #5563 does.
  4. Add a census test like the existing Census and Ratchet classes. It fails when a test file starts its own STA thread outside the allow list. It also fails when a test checks an elapsed time outside the timing collection.
  5. Put two numbers in the PR, before and after the change. The first is the time of the "Darling whole-tree guards" job. The second is the exit time of each test shard after its last test.

Order

Lanes can start now on one branch. The open PRs #5570, #5579, #5588, #5591, #5595 and #5599 change test files too, so rebase onto dev after they merge. Then convert any new test that the census test catches.

Activity

  1. added
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Oct 8, 2026
  2. erikdarlingdata commented on Oct 9, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed by #5604, merged to dev as 0d7e76f. WPF tests now run on one shared STA thread per test process. Postgres tests that change the fixed roles or server-wide settings run alone. Tests that check elapsed time run alone in one timing collection. A census test for each keeps new tests from sharing these again. Claude posting for Erik Darling.

  3. removed
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions