Repository navigation
Keep requested reviewbot reviews from reading the pull request id - #63
josephschorr wants to merge 1 commit into
Conversation
A review asked for in Slack is not bound to its pull request, so reading the pull request's node id is refused. The prompt now keeps that read out of requested mode, stops reporting unrelated refusals as allowlist refusals, and reviews several requested pull requests one at a time.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
| allowlist, that is by design: relay the refusal to the thread and stop — | ||
| do not retry or try another spelling of the URL. | ||
| allowlist — the refusal itself says "not in reviewbot's allowlist" — | ||
| that is by design: relay the refusal to the thread and stop — do not |
There was a problem hiding this comment.
[minor] The allowlist "relay and stop" rule is unscoped and conflicts with the new multi-PR rule
Problem: this sentence says to relay the refusal "and stop". Lines 568-570 say "A refusal or failure on one does not stop the rest." For a request naming two PRs where the second repo is not allowlisted, the model may end the whole turn instead of continuing to the other PR. The multi-PR paragraph also never says to report each failed PR in the thread. The "If you cannot review" section (line 510) is written around conclude_trigger_status, which this mode does not have, so a clone failure on one PR can end silently.
Fix: in the Requested review section, scope the sentence to the PR ("relay the refusal to the thread and stop reviewing THAT pull request"). In the multi-PR paragraph (lines 568-570), add that every PR that is refused or fails must get its own thread message stating the actual reason, and that the turn does not end until each PR has either a delivered review or such a message.
Test: none can prove a prompt claim. Re-run the multi-PR request by hand against a session with one allowlisted and one non-allowlisted repo and confirm each PR gets its own message. Validate the YAML with oap agent lint examples/reviewbot or equivalent.
Constraints: CLAUDE.md "Never silently drop errors" (surface a user-visible message for each failure).
| allowlist refusal; never describe it as one. A "no slot-bound grant" | ||
| refusal means the call asked for `id` (step 2): drop it and retry. | ||
|
|
||
| Several pull requests in one request: review them one at a time, each |
There was a problem hiding this comment.
[minor] Multi-PR flow does not give each PR its own artifact
Problem: "each exactly as above" pulls in step 4 (line 428-429), which says "On a later round pass revises with the previous revision". That text is meant for a later push on the same PR. In one session reviewing PR A then PR B, the model can pass A's revision as revises for B, so one artifact carries both reviews' history and A's head-SHA tag. Step 2's clone dir (/workspace/<dir>, line 352) is also unspecified per PR.
Fix: add one sentence to the paragraph at lines 568-570: each pull request gets its own fresh artifact_prepare with no revises (revising applies only to a later push on the SAME pull request), and its own /workspace directory named after owner, repo and number.
Test: none can prove a prompt claim. Check by hand with a two-PR request and confirm two distinct artifacts, each attached by its own respond_to_user. The artifact-delivered requirement is already per render (pkg/agent/completion/kinds/artifactdelivery/artifactdelivery.go:81), so no code change is needed.
| that is by design: relay the refusal to the thread and stop — do not | ||
| retry or try another spelling of the URL. Any OTHER refusal is not an | ||
| allowlist refusal; never describe it as one. A "no slot-bound grant" | ||
| refusal means the call asked for `id` (step 2): drop it and retry. |
There was a problem hiding this comment.
[minor] "Drop it and retry" has no bound and no outcome for a failed retry
Problem: the instruction does not say how many retries are allowed or what to report if the retry is also refused. It sits next to "do not retry" for the allowlist case, so the retry rule is easy to misapply.
Verify: confirm that a refusal whose text contains "no slot-bound grant" can only come from a call that asked for id (relwrites.go:164, block requireSlotBound). If so:
Fix: reword to "retry ONCE without id; if that call is refused too, relay the refusal text to the thread verbatim and stop reviewing that pull request".
Test: none can prove a prompt claim; check by hand.
| If a git or gh call is refused because the repository is not in the | ||
| allowlist, that is by design: relay the refusal to the thread and stop — | ||
| do not retry or try another spelling of the URL. | ||
| allowlist — the refusal itself says "not in reviewbot's allowlist" — |
There was a problem hiding this comment.
[minor] Prompt matches refusal text by hand; nothing records the dependency
Problem: this paragraph keys on the literal phrase "not in reviewbot's allowlist" (examples/reviewbot/manifests/toolspecs.yaml:83 and :189) and on "no slot-bound grant" (pkg/authz/relwrites/relwrites.go:164). Rewording either message silently breaks the discriminator, and no test notices.
Fix: add a YAML comment next to both message: lines in toolspecs.yaml (83, 189) saying agentclass.yaml's Requested review section matches the phrase "not in reviewbot's allowlist", so the message must keep it. Put the comment in toolspecs.yaml, not inside this block scalar, where it would become prompt text.
Test: do not add a test that reads examples/.
Constraints: CLAUDE.md "Tests MUST NOT reference examples/" rules out a test that pins the prompt text.
Reviews requested in Slack were failing for every pull request: the agent read the pull request's id, which a requested session isn't permitted to record, and then told the user the repository wasn't on the allowlist.