Skip to content

test(episode): give the episode tests' wait a budget that covers a cold start - #745

Merged
andrewpark-scaleai merged 1 commit into
mainfrom
andrew-park/fix-flaky-turn-deadline-test
Nov 18, 2026
Merged

andrewpark-scaleai merged 1 commit into
mainfrom
andrew-park/fix-flaky-turn-deadline-test

Conversation

@andrewpark-scaleai

@andrewpark-scaleai andrewpark-scaleai commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Why

test_a_turn_not_taken_by_its_deadline_fails_the_run_and_is_never_handed_out fails every time it runs alone, on main included, with a TimeoutError from the test's own wait. It passes in CI only because an earlier test in the same worker usually pays a one-time cost first.

The first episode in a process looks up request headers for its in-process agent's MCP URL. That lookup imports every registered sandbox provider's SDK, which took 5 to 10 s on a dev box (Daytona's alone was 5.7 s under a profiler). The test's _until helper allows 5 s, so the import used up the whole budget before the run's 1 s turn deadline started.

What

_until in tst/unit/task/episode/run_test.py allows 30 s instead of 5 s. It polls every 10 ms and returns as soon as its condition holds, so a passing test finishes at the same speed. Only a test that is really broken waits the full 30 s.

Design decisions

  • Fix the wait budget, not the import. Importing every provider to build headers for one URL is slow, but it's a one-time cost per process, and changing how providers load is a larger change than a flaky test justifies. It can follow separately.
  • No warm-up fixture. Importing the providers in a fixture would hide the cold start from this file only. Any other episode test that runs first in a process would still hit it.

Test

uv run --all-extras pytest -p no:cacheprovider \
  "tst/unit/task/episode/run_test.py::test_a_turn_not_taken_by_its_deadline_fails_the_run_and_is_never_handed_out"

Before: 3 of 3 runs failed with TimeoutError in _until. After: 3 of 3 passed, in about 7 s each.

uv run --all-extras pytest -p no:cacheprovider tst/unit/task/episode
109 passed

RetriggerView in GreptileConfidence Score: 5/5

This PR appears safe to merge.

Summary

Raises the episode tests’ _until timeout from 5 to 30 seconds to cover cold-start imports.

  • Episode tests wait longer for slow first-start setup.

Reviews (1) · Last reviewed commit: "test(episode): give _until a budget that..." · Reviewed by Greptile

The first episode in a process imports every sandbox provider's SDK
before its first turn opens, which takes several seconds. Run on its
own, test_a_turn_not_taken_by_its_deadline_fails_the_run_and_is_never_handed_out
spent its whole 5 s wait on that import and timed out before the 1 s
turn deadline began. The wait returns as soon as its condition holds,
so a 30 s budget costs a passing test nothing.
@andrewpark-scaleai
andrewpark-scaleai requested a review from a team as a code owner November 17, 2026 19:34
@andrewpark-scaleai
andrewpark-scaleai merged commit b303058 into main Nov 18, 2026
35 checks passed
@andrewpark-scaleai
andrewpark-scaleai deleted the andrew-park/fix-flaky-turn-deadline-test branch November 18, 2026 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants