Skip to content

flake: teammate-mcp.test.ts times out at 5000ms and blocks releases — proven fail+pass at the same commit #373

Description

@khaliqgant

Filed from the 2026-08-25 release of 0.1.75, which carried a production
dispatch fix. This test blocked it.

The failure

FAIL src/cli/teammate-mcp.test.ts > Factory teammate MCP
     > exposes discover and bounded ask to the spawned worker
       through its injected MCP server
Error: Test timed out in 5000ms.

Test Files  1 failed | 109 passed | 1 skipped (111)
     Tests  1 failed | 2254 passed | 1 skipped (2256)

Publish run 32821387417. Because Test is step 7, every later step —
Resolve version, Create release tag, Publish, Verify final release state — was skipped. One test held back the whole release.

It is a flake, and here is the proof rather than an assertion

Same commit 952d450e, two runs:

run Test step
32821387417 failure — timed out in 5000ms
32821900947 success

A fail and a pass at the same commit is the pair required before calling
anything a flake; "it passed on rerun" alone would not be. The identical tree
had also already passed CI twice — on main at 423e293b and on
release/v0.1.75 at 763710b2.

Why it matters more than a normal flake

It sits on the release critical path. publish.yml runs the suite itself,
so this test can block shipping a fix even when the change under test is
unrelated — which is exactly what happened, during an active dispatch outage.

Origin

src/cli/teammate-mcp.test.ts arrived in 92a6c24#178, "Who can I ask?".
It is not covered by the existing flake issues #342 (shared timing substrate
in factory.test.ts), #328 or #353.

Suspected shape, not asserted

A 5000ms default testTimeout on a test that stands up an injected MCP server
and exercises a spawned-worker round trip. Under a loaded CI runner, server
start plus round trip can plausibly exceed 5s without anything being wrong. That
is a hypothesis from the failure mode and the test's name — I have not profiled
it, and it should be measured rather than assumed. In particular, do NOT simply
raise the timeout until it stops failing: that hides a real race just as
effectively as it fixes a slow setup.

What done looks like

  1. Establish which it is — slow setup, or a genuine race in the injected MCP
    server / spawned-worker handshake. Measure the actual distribution rather
    than inferring from one timeout.
  2. If slow setup: an explicit, justified timeout for this test with the measured
    headroom stated, not a global bump.
  3. If a race: fix the race. A raised timeout would convert a reproducible
    failure into a rare one, which is worse.
  4. Either way, the test must be deterministic enough that a release is never
    blocked by it again. size the fixture to the mechanism's boundary applies:
    name the boundary first.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions