Skip to content

Keep requested reviewbot reviews from reading the pull request id - #63

Open
josephschorr wants to merge 1 commit into
mainfrom
fix/reviewbot-requested-no-pr-id
Open

josephschorr wants to merge 1 commit into
mainfrom
fix/reviewbot-requested-no-pr-id

Conversation

@josephschorr

Copy link
Copy Markdown
Member

Reviews requested in Slack were failing for every pull request: the agent read the pull request's id, which a requested session isn't permitted to record, and then told the user the repository wasn't on the allowlist.

  • Requested reviews now read only the fields they need, without the id.
  • Only a genuine allowlist refusal is reported as one; the id refusal is retried without it.
  • Several pull requests in one request are reviewed one at a time, and a failure on one no longer stops the rest.

A review asked for in Slack is not bound to its pull request, so reading
the pull request's node id is refused. The prompt now keeps that read out
of requested mode, stops reporting unrelated refusals as allowlist
refusals, and reviews several requested pull requests one at a time.
@vercel

vercel Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
openagentprimitives Ignored Ignored Nov 10, 2026 4:37pm UTC

Request Review

allowlist, that is by design: relay the refusal to the thread and stop —
do not retry or try another spelling of the URL.
allowlist — the refusal itself says "not in reviewbot's allowlist" —
that is by design: relay the refusal to the thread and stop — do not

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] The allowlist "relay and stop" rule is unscoped and conflicts with the new multi-PR rule

Problem: this sentence says to relay the refusal "and stop". Lines 568-570 say "A refusal or failure on one does not stop the rest." For a request naming two PRs where the second repo is not allowlisted, the model may end the whole turn instead of continuing to the other PR. The multi-PR paragraph also never says to report each failed PR in the thread. The "If you cannot review" section (line 510) is written around conclude_trigger_status, which this mode does not have, so a clone failure on one PR can end silently.
Fix: in the Requested review section, scope the sentence to the PR ("relay the refusal to the thread and stop reviewing THAT pull request"). In the multi-PR paragraph (lines 568-570), add that every PR that is refused or fails must get its own thread message stating the actual reason, and that the turn does not end until each PR has either a delivered review or such a message.
Test: none can prove a prompt claim. Re-run the multi-PR request by hand against a session with one allowlisted and one non-allowlisted repo and confirm each PR gets its own message. Validate the YAML with oap agent lint examples/reviewbot or equivalent.
Constraints: CLAUDE.md "Never silently drop errors" (surface a user-visible message for each failure).

allowlist refusal; never describe it as one. A "no slot-bound grant"
refusal means the call asked for `id` (step 2): drop it and retry.

Several pull requests in one request: review them one at a time, each

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] Multi-PR flow does not give each PR its own artifact

Problem: "each exactly as above" pulls in step 4 (line 428-429), which says "On a later round pass revises with the previous revision". That text is meant for a later push on the same PR. In one session reviewing PR A then PR B, the model can pass A's revision as revises for B, so one artifact carries both reviews' history and A's head-SHA tag. Step 2's clone dir (/workspace/<dir>, line 352) is also unspecified per PR.
Fix: add one sentence to the paragraph at lines 568-570: each pull request gets its own fresh artifact_prepare with no revises (revising applies only to a later push on the SAME pull request), and its own /workspace directory named after owner, repo and number.
Test: none can prove a prompt claim. Check by hand with a two-PR request and confirm two distinct artifacts, each attached by its own respond_to_user. The artifact-delivered requirement is already per render (pkg/agent/completion/kinds/artifactdelivery/artifactdelivery.go:81), so no code change is needed.

that is by design: relay the refusal to the thread and stop — do not
retry or try another spelling of the URL. Any OTHER refusal is not an
allowlist refusal; never describe it as one. A "no slot-bound grant"
refusal means the call asked for `id` (step 2): drop it and retry.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] "Drop it and retry" has no bound and no outcome for a failed retry

Problem: the instruction does not say how many retries are allowed or what to report if the retry is also refused. It sits next to "do not retry" for the allowlist case, so the retry rule is easy to misapply.
Verify: confirm that a refusal whose text contains "no slot-bound grant" can only come from a call that asked for id (relwrites.go:164, block requireSlotBound). If so:
Fix: reword to "retry ONCE without id; if that call is refused too, relay the refusal text to the thread verbatim and stop reviewing that pull request".
Test: none can prove a prompt claim; check by hand.

If a git or gh call is refused because the repository is not in the
allowlist, that is by design: relay the refusal to the thread and stop —
do not retry or try another spelling of the URL.
allowlist — the refusal itself says "not in reviewbot's allowlist" —

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] Prompt matches refusal text by hand; nothing records the dependency

Problem: this paragraph keys on the literal phrase "not in reviewbot's allowlist" (examples/reviewbot/manifests/toolspecs.yaml:83 and :189) and on "no slot-bound grant" (pkg/authz/relwrites/relwrites.go:164). Rewording either message silently breaks the discriminator, and no test notices.
Fix: add a YAML comment next to both message: lines in toolspecs.yaml (83, 189) saying agentclass.yaml's Requested review section matches the phrase "not in reviewbot's allowlist", so the message must keep it. Put the comment in toolspecs.yaml, not inside this block scalar, where it would become prompt text.
Test: do not add a test that reads examples/.
Constraints: CLAUDE.md "Tests MUST NOT reference examples/" rules out a test that pins the prompt text.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants