Skip to content

fix(test): preserve captured output after lane timeouts - #3924

Closed
luvs01 wants to merge 1 commit into
lidge-jun:devfrom
luvs01:agent/test-runner-timeout-output-20260908
Closed

fix(test): preserve captured output after lane timeouts#3924
luvs01 wants to merge 1 commit into
lidge-jun:devfrom
luvs01:agent/test-runner-timeout-output-20260908

Conversation

@luvs01

@luvs01 luvs01 commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Summary

When a captured test lane exceeded its timeout, the runner returned exit code 124 before forwarding any stdout or stderr. A long test:changed run therefore lost the diagnostics needed to explain its failure.

Collect output as it arrives and forward the retained stdout and stderr before returning the timeout result. After process exit, allow one second for the pipes to drain; a descendant holding a pipe open cannot cause an unlimited wait. Report incomplete capture and return a failure when the direct child exited successfully but its captured output is incomplete. Preserve the existing signal escalation, interruption exit codes, and inherited-output mode.

The contributor guide documents the timeout and incomplete-output behavior. Scope is limited to the runner, its regression tests, and that guide.

Verification

Head e24163231edeaa09a30a99ca1746e3b573af78ae, based on dev aeefb3ab5433c69f7621695413ec1f77a05ed54f. Tree: 8c7c2aae804be1eee69049751534753b6e688f12.

  • bun test tests/ci-workflows/test-runner.test.ts --timeout 60000: 52 passed with the project-pinned Bun 1.4.0 on Windows. The subsequent rebase changes only README and sponsor content; the tested implementation, tests, dependencies, and documentation build inputs are identical.
  • Eight regressions cover successful/failed/timed-out child output, UTF-8 split across chunks, read failure, nonclosing pipes, pending/rejected cancellation, and an exited child with incomplete capture. Real child fixtures verify exit codes and that stdout/stderr markers are emitted once.
  • Restoring the early timeout return made the real timeout regression fail with empty output. Removing the drain limit made both open-pipe regressions fail. The production source was restored and its SHA-256 verified after each ablation.
  • bun run typecheck, bun run privacy:scan, and git diff --check: passed.
  • Documentation build: 425 pages passed. Package assets were prepared before the full local checks.
  • Independent read-only security/behavior review found no blocking issue; the additional incomplete-capture exit-policy regression also passed.

The full local bun run prepush check reached the parallel lane's 900-second limit (exit 124) before a complete suite summary. Its runner and directly owned workers were confirmed absent afterward. This incomplete run is not a pass, and the identical timed-out run has not been repeated. Full cross-platform CI passed all 26 jobs on this exact head, including Linux shards, Windows shards, macOS lanes/control, gates, and platform checks. Contributor readiness uses that full CI and the focused local checks above; the incomplete local full run remains recorded and is not treated as a pass. CodeRabbit completed with no actionable findings or unresolved review threads. Maintainer approval and upstream-required CI remain separate merge requirements.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed.
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults.

Review readiness checklist

  • All CI tests are green on my local testing.
  • I pushed my PR to the latest dev commit.
  • I resolved all correct Codex and CodeRabbit findings.
  • My PR is ready for review.

Summary by CodeRabbit

  • Bug Fixes

    • Test runs now reliably preserve and display captured standard output and error output, including output containing split or non-ASCII characters.
    • Prevented test runs from hanging when child processes keep output streams open.
    • Incomplete output capture is now reported and cannot be mistaken for a successful test run.
    • Timed-out test lanes now display captured output and exit with the expected timeout status.
  • Documentation

    • Updated contributor documentation to explain test-runner timeout and output-capture behavior.

@coderabbitai

coderabbitai Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Team

Run ID: 1deaf82a-39cb-49cd-8274-4323855c9df1

📥 Commits

Reviewing files that changed from the base of the PR and between 17d2a17 and e241632.

📒 Files selected for processing (3)
  • docs-site/src/content/docs/contributing.md
  • scripts/test.ts
  • tests/ci-workflows/test-runner.test.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review.


📝 Walkthrough

Walkthrough

The test runner now captures stdout and stderr with bounded draining, preserves partial output, forwards output through injectable writers, and marks incomplete captures as failures. Tests cover stream, timeout, cancellation, and lane-result behavior. The contributing guide documents these semantics.

Changes

Test output capture

Layer / File(s) Summary
Streaming capture primitive
scripts/test.ts:397-458
Adds captureTestOutput with concurrent stdout/stderr reads, UTF-8-safe accumulation, cancellation handling, bounded draining, and completion tracking.
Lane execution integration
scripts/test.ts:460-468, scripts/test.ts:488, scripts/test.ts:500, scripts/test.ts:509-521
Exports runTestLane, accepts injectable writers, uses the capture helper, forwards output after process termination, preserves timeout output, and converts incomplete successful captures to exit code 1.
Capture validation and documentation
tests/ci-workflows/test-runner.test.ts:1-245, docs-site/src/content/docs/contributing.md:45-50
Tests stream reconstruction, cancellation deadlines, read errors, incomplete capture, and pass/fail/timeout lanes. Documents timeout output, one-second draining, and incomplete capture behavior.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: ⚪ Minimal · up to e2416

The test runner now retains and prints partial stdout/stderr for timed-out lanes, bounds pipe draining, and treats incomplete successful captures as failures. No concrete current-head merge-blocking risk remains.

Sequence Diagram(s)

sequenceDiagram
  participant runTestLane
  participant Bun.spawn
  participant captureTestOutput
  participant stdoutWriter
  participant stderrWriter
  runTestLane->>Bun.spawn: start test lane
  Bun.spawn-->>captureTestOutput: provide stdout and stderr streams
  captureTestOutput-->>runTestLane: return output and complete status
  runTestLane->>stdoutWriter: emit captured stdout
  runTestLane->>stderrWriter: emit captured stderr
  runTestLane-->>runTestLane: return exit code 0, 1, or 124
Loading

Suggested reviewers: lidge-j

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 2 files. (1 skipped: 1… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the primary change: preserving captured test output when a test lane times out.
Full details: Docstring Coverage

Explanation

Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 2 files. (1 skipped: 1 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Deterministic PR hygiene checks passed.

@github-actions github-actions Bot added the bug Something isn't working label Sep 7, 2026
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

✅ READY

  • all PR quality gates passed; the review readiness checklist is complete.

Review readiness checklist

  • ✅ All CI tests are green on my local testing.
  • ✅ I pushed my PR to the latest dev commit.
  • ✅ I resolved all correct Codex and CodeRabbit findings.
  • ✅ My PR is ready for review.

4/4 boxes ticked.

This pull request has been marked Ready for Review.
The review-ready label marks this PR as ready; review automation runs independently.
Maintainers notified: @lidge-jun @Ingwannu

@luvs01

luvs01 commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@lidge-jun

Copy link
Copy Markdown
Owner

리뷰 · 우선순위 58 / 80

이 PR은 scripts/test.ts의 테스트 레인 실행기에서, 캡처 모드로 돌린 자식 프로세스가 시간 제한에 걸렸을 때 stdout/stderr를 통째로 버리던 구멍을 막습니다. 지금 dev(HEAD 09f669a75, package 2.48.0 open-dev after #3925)의 runTestLane은 캡처 모드에서 new Response(child.stdout).text()로 끝까지 읽고, 타임아웃이면 바로 exitCode: 124와 빈 output을 돌려줍니다. 그래서 test:changed처럼 긴 레인이 124로 끊기면 “왜 죽었는지”를 보여주는 로그가 사라집니다. 이 변경은 읽는 동안 계속 모으는 captureTestOutput을 두고, 프로세스 종료(또는 타임아웃 종료) 뒤에 파이프를 최대 1초만 더 비운 다음, 모아 둔 stdout/stderr를 먼저 찍고 나서 124를 돌려줍니다. 자손이 파이프를 붙잡고 있어도 무한 대기는 하지 않습니다. 직접 자식이 0으로 끝났는데 캡처가 불완전하면 실패(1)로 바꿔서, “성공처럼 보이지만 로그가 잘린 경우”를 성공으로 치지 않게 합니다. 시그널 중단 코드(130/143)와 inherit 출력 모드는 그대로 둡니다. docs-site contributing 안내에도 같은 동작을 짧게 적어 두었고, tests/ci-workflows/test-runner.test.ts에 UTF-8 조각 합치기·열린 파이프 취소·읽기 실패·불완전 캡처·실제 자식 fixture(성공/실패/타임아웃) 회귀가 들어 있습니다. 지금 dev 방향(릴리스 트레인·스폰서·OrcaRouter 등)과는 겹치지 않는 CI/DX 버그 픽스라서 types/config 분할에 무효화되지도 않습니다.

라인 scripts/test.ts captureTestOutput - 타임아웃 전에 모은 출력을 남기고, finish(1000)로 배수(drain)를 1초로 묶어 자손 파이프에 막히지 않게 한 핵심 경로다.
라인 scripts/test.ts runTestLane - 예전에는 타임아웃 분기에서 return { exitCode: 124, output: "" }로 바로 나갔는데, 지금은 종료 처리 뒤에도 캡처 결과를 찍은 다음 124를 반환한다.
라인 scripts/test.ts incomplete capture - 캡처가 불완전하고 자식 exit가 0이면 exitCode를 1로 올린다. 성공으로 위장한 잘린 로그를 막는다.
경로 tests/ci-workflows/test-runner.test.ts - 스트림 단위·실제 spawn fixture로 타임아웃/실패/성공 출력이 한 번씩만 나오는지까지 검증한다. 범위는 러너·테스트·가이드로 좁다.
경로 PR draft / 작성자 검증 - 작성자가 로컬 prepush가 900초 레인 한도에 걸려 124로 끊겼고 Cross-platform CI도 이 exact head에서 아직 미완료라고 스스로 적었다. draft 체크리스트도 “ready for review”가 비어 있다.

메인테이너의 판단이 필요한 지점

  • draft를 유지한 채 Cross-platform CI가 이 head에서 초록이 될 때까지 기다릴지, 아니면 러너 단위 테스트만 초록이면 ready로 올려 merge 후보에 넣을지
  • incomplete-capture 때 exit 0→1 정책이 test:changed 집계/선택 실패 메시지와 맞는지(작성자 의도대로면 유지)
  • contributing 문구를 영문 docs-site에만 둘지, 기여자들이 자주 보는 다른 가이드에도 한 줄 더 둘지

너의 추천
draft 유지. Cross-platform CI가 이 head에서 통과하면 ready로 올리고 merge. 로컬 prepush가 124로 끊긴 기록만으로는 막지 말고, 러너 회귀(tests/ci-workflows/test-runner.test.ts)와 CI 결과가 맞으면 랜딩해도 된다. types/config 분할과 무관하니 close-don't-rebase 대상이 아니다.

이 댓글은 grok-bot이 작성했습니다

@github-actions
github-actions Bot marked this pull request as ready for review September 7, 2026 16:19
lidge-jun added a commit that referenced this pull request Sep 7, 2026
…or stream-health watchdog (#3940)

Preserve captured stdout and stderr when a test lane times out. Bound
post-exit pipe draining to one second and report incomplete capture as a
failure when the child otherwise exited successfully.

Scale the Cursor stream-health fixture deadlines from one load-adjusted
budget and verify meaningful progress continues for the required interval
after the client receives its first text.

Carries #3924 and #3930 without changes to their files. The branch commits
retain the original author and cherry-pick provenance from:
e241632
141077f

Co-authored-by: luvs01 <27862058+luvs01@users.noreply.github.com>
@lidge-jun

Copy link
Copy Markdown
Owner

Landed on dev as 221617b80756f1be13a5db942cd80a3d8f79ab01, through #3940.

Your commit e24163231edeaa09a30a99ca1746e3b573af78ae was carried unchanged with git cherry-pick -x, so the branch commit kept your authorship and its provenance line. I verified the landed files byte-for-byte against your head: scripts/test.ts, tests/ci-workflows/test-runner.test.ts and docs-site/src/content/docs/contributing.md on dev are identical to this pull request's version. The squash commit carries Co-authored-by: luvs01 <27862058+luvs01@users.noreply.github.com>, which is what GitHub reads for contribution credit.

It landed as the lower layer of a two-layer stack together with #3930, so one CI run verified both changes on the cumulative tree: run 34152136978, all 16 applicable jobs green. Your own contributor run on e2416323 had already passed all 26 jobs.

Closing as superseded by that verified landing. Thank you — losing the output that explains a timeout is exactly the kind of failure that wastes an afternoon, and the incomplete-capture rule is the part I expect to pay for itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working review-ready

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants