Skip to content

ci: raise the reviewer turn cap to 120 - #11

Merged
bawoodruff merged 1 commit into
mainfrom
ci/reviewer-turn-cap
Aug 29, 2026
Merged

bawoodruff merged 1 commit into
mainfrom
ci/reviewer-turn-cap

Conversation

@bawoodruff

Copy link
Copy Markdown
Contributor

#4 and #7 got no review at all on their first run — the reviewer exhausted --max-turns 40 and exited with an empty result.

Four real runs

PR lines turns outcome
#8 584 24 approved
#5 2,704 35 posted a 5,599-char review
#4 848 41 ❌ hit the cap
#7 989 41 ❌ hit the cap

Turn count tracks how far the reviewer wanders, not diff size. #5 is three times larger than #4 and used fewer turns. So sizing the cap to the diff is the wrong model.

Both failures stopped exactly at the cap, not near it — the real ceiling is unknown, and a snug cap keeps guillotining runs that were about to finish.

Why 120

The cap is a runaway guard, not a budget:

  • Reviews run on a subscription OAuth token, so the reported total_cost_usd is notional rather than billed.
  • The binding limit is the job's timeout-minutes: 30. At the observed ~6s/turn, 120 turns ≈ 12 minutes — a genuine runaway still gets caught by the wall clock.

This matters more than a truncated review would, because exceeding the cap fails as a non-zero exit with an empty result — the run produces nothing at all, rather than a short review. That's why #4 and #7 are currently unreviewed.

num_turns is logged on every run, so if reviews start landing near 120 the signal is visible rather than surfacing as unexplained silence.

After this merges

#4 and #7 need re-triggering to get their first review.

🤖 Generated with Claude Code

Four real runs since the reviewer went live show 40 is too snug, and that turn
count tracks how far the reviewer wanders rather than how big the diff is:

  PR   lines  turns  outcome
  #8     584     24  approved
  #5   2,704     35  posted a review     <- largest diff, fewest turns
  #4     848     41  hit the 40 cap
  #7     989     41  hit the 40 cap

The largest PR by a factor of three used the fewest turns, so sizing the cap
to the diff is the wrong model. Both failures also stopped exactly AT the cap
rather than near it, so the real ceiling is unknown — a snug cap will keep
catching runs that were about to finish.

Treat the cap as a runaway guard rather than a budget. Reviews run on a
subscription OAuth token, so the reported total_cost_usd is notional rather
than billed, and the binding limit is the job's timeout-minutes. At the
observed ~6s/turn, 120 turns is roughly 12 minutes against a 30-minute wall
clock, so the timeout still catches a genuine runaway.

This matters more than a short review would, because exceeding the cap fails
as a non-zero exit with an empty result, not as a truncated review — the run
produces nothing at all. #4 and #7 currently have no review for that reason.
num_turns is logged on every run, so if reviews start landing near 120 the
signal is there rather than showing up as an unexplained silence.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: comment

Review Summary

  • Files reviewed: .github/workflows/claude-pr-review.yml (diff hunk + surrounding job definition, lines ~114-135 and ~505-570)
  • Overall verdict: COMMENT

Major Issues (should fix)

  • .github/workflows/claude-pr-review.yml:530 — The new comment justifies 120 turns as "~12 minutes" using an "observed ~6s/turn" rate, but the unmodified comment at .github/workflows/claude-pr-review.yml:123 (still in the same file) documents "a healthy 24-turn review took ~5.8 min" — that's ~14.5s/turn, over 2x the rate this diff relies on. At the documented rate, 120 turns ≈ 29 minutes, which nearly exhausts the 30-minute timeout-minutes budget (line 126) rather than leaving the "well inside" headroom the new comment claims. If the wall clock kills the step instead of the turn cap, you get the exact same failure mode this PR sets out to fix (continue-on-error swallows it, but the run still produces no review/diagnosis) — just via a different trigger. Fix: reconcile the per-turn estimate with the existing ~5.8min/24turns data point (or pull real numbers from the four cited runs' logs), and either raise timeout-minutes to give genuine headroom for 120 turns at the higher observed rate, or pick a lower turn cap that's actually bounded by 30 minutes with margin.

Minor Issues (optional)

  • .github/workflows/claude-pr-review.yml:122-125 — This job-level comment justifying timeout-minutes: 30 still says "--max-turns was raised to 40" and computes headroom off a 40-turn cap ("40 turns lands near 10"). This diff bumps the step's cap to 120 but leaves this comment describing the old value, so the file now contains two comments that disagree about the current --max-turns. Fix: update this comment to reference 120 and its own recomputed timing, or replace the duplicated rationale with a pointer to the step-level comment (line 516) as the single source of truth.

Security Findings

  • None identified in this diff (no permissions, secrets, or tool-allowlist changes).

Positive Observations

  • The step-level comment (line 516) is thorough and data-driven, citing four real PR runs with clear per-PR outcomes.
  • The error message at line 568 was correctly kept in sync with the new cap value.
  • continue-on-error: true correctly bounds the blast radius of a runaway/timeout to "advisory failure," not a red build.

@bawoodruff
bawoodruff merged commit 0f13260 into main Aug 29, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant