Eliminate macOS cold-refresh 30-second tail latency

[karthiknadig]

Summary

PET's macOS performance baseline has a deterministic cold-refresh tail near 30 seconds that is hidden by healthy P50 values.

Evidence

Across 11 successful main-branch performance baselines from July 24 through August 6, 2026:

• macOS full-refresh P50 median: 137 ms
• macOS full-refresh P95 median: 30,291 ms
• Latest baseline (8bd5fb4, run 31071711968): P50 142 ms, P95 30,291 ms, mean 8,642 ms
• The latest raw iterations were:
  • 30,370 ms (9 environments)
  • 30,195 ms (9 environments)
  • 24,927 ms (10 environments)
  • then 139, 131, 110, 144, 105, 128, and 174 ms (10 environments)

The same cache directory is reused while each iteration starts a fresh PET server. The first three processes are slow and initially miss one environment, after which the disk cache is warm. Environment inventory is otherwise stable (10 environments, 1 manager), so this is not explained by changing runner inventory.

Investigation requirements

• Attribute each slow iteration to a refresh phase and locator using existing RefreshProgress/RefreshPerformance data or equivalent privacy-safe instrumentation.
• Determine whether the wait is interpreter spawning, macOS locator probing, cache population, subprocess/pipe behavior, or another blocking path.
• Verify every file-based detection path before retaining any spawn fallback.
• Measure exact spawn/read operation counts in regression tests.

Acceptance criteria

• Root cause is demonstrated with phase/locator and operation-count evidence.
• A cold PET process discovers the complete expected environment inventory on its first refresh.
• macOS cold-refresh tail no longer approaches the 15/30-second timeout boundaries.
• A realistic macOS CI P95 threshold is established and enforced by the snapshot gate.
• Median performance does not regress on macOS, Windows, or Linux.
• Fix includes behavior tests and performance/operation-count tests.
• No extra Python or manager process spawning is introduced.

Related

• Add end-to-end performance tests for JSONRPC server across platforms #313
• Quality snapshot enforcement issue filed from the same audit.

[karthiknadig]

Diagnostic run 31423551324 on head f9485d0 established the root cause with phase, locator, and operation-count evidence:

• First three iterations each emitted exactly two interpreter-probe timeouts, both classified as usrBinPython3.
• Iterations: ~30.3s, ~30.2s, ~30.3s; each found only 9 environments.
• Path P95: 30,308 ms.
• MacCommandLineTools P95: 15,185 ms.
• Later iterations had no probe timeouts and completed in 182-294ms with 10 environments.

The two /usr/bin/python3 calls share the same interpreter cache entry. MacCommandLineTools::try_from() starts a 15-second spawn to enrich the alias while PATH discovery asks to resolve the same shim. The cache-entry mutex serializes them: PATH waits for the locator probe, then performs its own 15-second probe after the first timeout, producing the ~30-second phase.

The product fix should therefore:

1. Resolve Apple /usr/bin/python* shims from the active developer directory using filesystem state (DEVELOPER_DIR / /var/db/xcode_select_link / standard fallbacks).
2. Let MacXCode or MacCommandLineTools identify the selected shim without spawning it.
3. Skip spawning unresolved Apple system shims; a non-selected or unavailable shim is not a usable Python environment.
4. Remove the duplicate /usr/bin/python3 alias-enrichment spawn from the macOS locators.
5. Add operation-count tests proving zero interpreter spawns for these paths, plus cross-platform behavior tests.

[karthiknadig]

First post-fix validation run 31425828087 on b923035 passed all platform gates.

macOS results:

• 0 interpreter probe timeouts (previously 2 per cold iteration).
• Iteration 1 found all 10 environments in 2,436ms (previously 9 environments in ~30s).
• Full refresh P95: 1,437ms (previously ~30,308ms).
• Path P95: 1,436ms (previously ~30,308ms).
• MacCommandLineTools P95: 137ms (previously ~15,185ms).
• All ten iterations found the complete inventory.

Linux and Windows also passed, retained identical environment inventories, and had no interpreter probe timeouts. A second identical run is in progress to distinguish normal macOS median variance from code drift before opening the PR.