Skip to content

Tests flake under parallel load: hard 5s timeouts lose races against CPU contention #53

Description

@axisrow

Pre-existing flake on fork/main — not introduced by any open PR.

Symptom

When several full npm test runs execute concurrently, unrelated tests fail at almost exactly 5000 ms:

  • withAppServer falls back to a direct app-server on broker/busy after a successful handshake (#32) — 5003 ms
  • broker CLI: exits with code 1 when app-server child dies (ACK-based) — 5006 ms

The ~5000 ms figure and the fact that which test fails varies per run point at a hard 5-second timeout being exceeded under CPU contention, not at a logic defect.

How it was isolated

Ran the same tests in three states on macOS (Node v25.9.0, codex-cli 0.146.0), using separate git worktrees:

State Isolated run Under parallel load
fork/main (baseline, no PRs) 5/5 green fails
PR #50 branch 190/190 green fails
PR #52 branch 193/193 green, 4 consecutive clean runs fails

Reproduction of the load condition: three full npm test runs started simultaneously, one per worktree. All three failed — including the untouched baseline — each on a different test, each at ~5000 ms.

A narrower load probe (6 parallel runs of a single test) did not reproduce it; the failure needs the full suite running concurrently, i.e. many app-servers/brokers competing at once.

Conclusion

The flake belongs to the test suite itself, not to any PR under review. Serialized runs are green everywhere.

Possible directions

  • Make the 5s timeouts adaptive, or raise them when the suite detects contention.
  • Gate the broker/app-server-heavy tests behind a concurrency limiter so they do not race each other.
  • Alternatively, mark the suite as serial-only and document that concurrent npm test runs are unsupported.

Filed as a follow-up; no change proposed here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions