ci: raise the reviewer turn cap to 120 - #11
Merged
Merged
Conversation
Four real runs since the reviewer went live show 40 is too snug, and that turn count tracks how far the reviewer wanders rather than how big the diff is: PR lines turns outcome #8 584 24 approved #5 2,704 35 posted a review <- largest diff, fewest turns #4 848 41 hit the 40 cap #7 989 41 hit the 40 cap The largest PR by a factor of three used the fewest turns, so sizing the cap to the diff is the wrong model. Both failures also stopped exactly AT the cap rather than near it, so the real ceiling is unknown — a snug cap will keep catching runs that were about to finish. Treat the cap as a runaway guard rather than a budget. Reviews run on a subscription OAuth token, so the reported total_cost_usd is notional rather than billed, and the binding limit is the job's timeout-minutes. At the observed ~6s/turn, 120 turns is roughly 12 minutes against a 30-minute wall clock, so the timeout still catches a genuine runaway. This matters more than a short review would, because exceeding the cap fails as a non-zero exit with an empty result, not as a truncated review — the run produces nothing at all. #4 and #7 currently have no review for that reason. num_turns is logged on every run, so if reviews start landing near 120 the signal is there rather than showing up as an unexplained silence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
VERDICT: comment
Review Summary
- Files reviewed:
.github/workflows/claude-pr-review.yml(diff hunk + surrounding job definition, lines ~114-135 and ~505-570) - Overall verdict: COMMENT
Major Issues (should fix)
.github/workflows/claude-pr-review.yml:530— The new comment justifies 120 turns as "~12 minutes" using an "observed ~6s/turn" rate, but the unmodified comment at.github/workflows/claude-pr-review.yml:123(still in the same file) documents "a healthy 24-turn review took ~5.8 min" — that's ~14.5s/turn, over 2x the rate this diff relies on. At the documented rate, 120 turns ≈ 29 minutes, which nearly exhausts the 30-minutetimeout-minutesbudget (line 126) rather than leaving the "well inside" headroom the new comment claims. If the wall clock kills the step instead of the turn cap, you get the exact same failure mode this PR sets out to fix (continue-on-errorswallows it, but the run still produces no review/diagnosis) — just via a different trigger. Fix: reconcile the per-turn estimate with the existing~5.8min/24turnsdata point (or pull real numbers from the four cited runs' logs), and either raisetimeout-minutesto give genuine headroom for 120 turns at the higher observed rate, or pick a lower turn cap that's actually bounded by 30 minutes with margin.
Minor Issues (optional)
.github/workflows/claude-pr-review.yml:122-125— This job-level comment justifyingtimeout-minutes: 30still says "--max-turnswas raised to 40" and computes headroom off a 40-turn cap ("40 turns lands near 10"). This diff bumps the step's cap to 120 but leaves this comment describing the old value, so the file now contains two comments that disagree about the current--max-turns. Fix: update this comment to reference 120 and its own recomputed timing, or replace the duplicated rationale with a pointer to the step-level comment (line 516) as the single source of truth.
Security Findings
- None identified in this diff (no permissions, secrets, or tool-allowlist changes).
Positive Observations
- The step-level comment (line 516) is thorough and data-driven, citing four real PR runs with clear per-PR outcomes.
- The error message at line 568 was correctly kept in sync with the new cap value.
continue-on-error: truecorrectly bounds the blast radius of a runaway/timeout to "advisory failure," not a red build.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#4 and #7 got no review at all on their first run — the reviewer exhausted
--max-turns 40and exited with an empty result.Four real runs
Turn count tracks how far the reviewer wanders, not diff size. #5 is three times larger than #4 and used fewer turns. So sizing the cap to the diff is the wrong model.
Both failures stopped exactly at the cap, not near it — the real ceiling is unknown, and a snug cap keeps guillotining runs that were about to finish.
Why 120
The cap is a runaway guard, not a budget:
total_cost_usdis notional rather than billed.timeout-minutes: 30. At the observed ~6s/turn, 120 turns ≈ 12 minutes — a genuine runaway still gets caught by the wall clock.This matters more than a truncated review would, because exceeding the cap fails as a non-zero exit with an empty result — the run produces nothing at all, rather than a short review. That's why #4 and #7 are currently unreviewed.
num_turnsis logged on every run, so if reviews start landing near 120 the signal is visible rather than surfacing as unexplained silence.After this merges
#4 and #7 need re-triggering to get their first review.
🤖 Generated with Claude Code