[rafalmaciag]

Card microplumberd-testing-store-port-race.

The defect

Two test processes on one host (two engineers' suites) clobber each other's EventStoreServers:

• Every process's PortSearcher starts at 2701, and a free-port probe only sees a port as taken once its container has started, seconds later. So both processes chose the same port.
• Containers are named eventstore-mem-<port> and were found by name. So the second process:
  • restarted the first's in-memory store (the reuse branch), wiping it, and then both shared it; or
  • hit Docker's name Conflict, whose catch → Cleanup removed the first's container by name ("connection refused"; Docker 409 "removal … already in progress" when both cleaned up, observed 2026-10-18 13:52Z by dev-clean); or
  • removed it on dispose.

A name over a shared namespace is a shared-resource operation.

Reproduction (red on master first)

MicroPlumberd.Testing.StoreHolder is a tiny console, not packable. Two copies start their stores at one barrier instant. For 20 s each asserts its own container (by id) runs and its own marker event stays readable. StorePortRaceTests runs the pair and asserts both survived, on unique ports and in unique containers.

	Result
master (the fix files restored from base)	RED 7/10. Every red had both processes on 2701: 4× the loser's create Conflict; 3× the loser's Cleanup removed the winner's container by name and both lost (the winner's /health/live refused).
with the fix	GREEN 10/10

The fix

• PortLock: a lock file per port in a machine-wide directory (/tmp/microplumberd-testing-ports; CommonApplicationData on Windows), opened FileShare.None (an advisory flock on Unix). It is held for the store's lifetime and released by the OS if the process dies; another process skips a held port.
• By id, never by name: the container id is captured at create, and Cleanup, Stop and Restart act on that id only.
• Another live process's container with the same name is refused at start, named, and neither restarted nor removed. A dead owner's is reaped, as the owner-label reaper already did.
• A failed start's own failure is thrown: a cleanup that also failed used to replace it (the 409 masked the first failure).
• The static readiness and removal seams are given the server, so a test touches only its own store. The existing readiness test was counting every parallel class's stores through the global seam.

Rows and mutants

Row	Mutant → result
two-process race	the port lock non-exclusive (FileShare.ReadWrite) → RED 4/10 (each a create Conflict; with by-id removal and the refusal, the loser fails alone and the winner survives)
a planted container with this server's name and a live foreign owner (pid 1 + its real start time): start refused, named, left present and stopped	the reuse branch without the refusal → red
dispose never removes a container this server did not create	by-name Cleanup → red
a start failure whose cleanup also fails throws the start failure	the masking catch → red

The by-name removal cannot be reached by the race row, because the lock prevents the clash. That is why it has its own rows.

A first run of the lock mutant was invalid: my edit broke the build, and the runs used the stale fixed binary. It was discarded and re-run with a build gate; the numbers above are from the valid run.

Packaging

StoreHolder evaluates IsPackable=false. The packable project set, evaluated on base and on this branch, is identical (27 = 27).

Counts

Full MicroPlumberd.Tests at 991d715: 196/202 (16 m 58 s). The 6 failures are exactly the long-standing set:

• GrpcApiTests ×3;
• ProcessManagerSourceGenerationTests.CommandTypes;
• SubscriptionRunnerStateTests ×2.

No test-host crash; no containers left behind. Another repository's image smokes were running beside it as docker load.

Follow-up (rev-reach 9de418b9d, PR two):

• lock-dir permissions for a second user on a shared box (1777/0666, and UnauthorizedAccessException treated as "next port");
• the cleanup failure surfaced in ToString (an AggregateException), not in Data;
• the race row asserts it got two results before its "only/unique" claims.

🤖 Generated with Claude Code