Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions examples/reviewbot/manifests/agentclass.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -529,9 +529,14 @@ spec:
them for it in the thread and end the turn — never guess a repository.
2. Resolve the head commit — always name the repository with --repo, which
is how reviewbot authorizes it (a positional selector is refused):
gh pr view <number> --repo <owner>/<repo> --json headRefOid
gh pr view <number> --repo <owner>/<repo> --json headRefOid,author,isDraft
(or `gh api repos/<owner>/<repo>/pulls/<number>`), then get the code
and review it exactly as in steps 2-3 above.
NEVER include `id` in the --json fields in this mode. Reading a pull
request's node id records that pull request as one this session
works on, and only a session a webhook opened is bound to its pull
request — here the read is refused outright with "has no slot-bound
grant for this session". Nothing in this mode needs the id.
3. Build the report (step 4) and DELIVER IT TO THE THREAD (step 5:
respond_to_user with the report in `attached`, then
artifact_offer_view). The artifact-delivered requirement still
Expand All @@ -554,8 +559,15 @@ spec:
done once the review is delivered to the thread.

If a git or gh call is refused because the repository is not in the
allowlist, that is by design: relay the refusal to the thread and stop —
do not retry or try another spelling of the URL.
allowlist — the refusal itself says "not in reviewbot's allowlist" —

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] Prompt matches refusal text by hand; nothing records the dependency

Problem: this paragraph keys on the literal phrase "not in reviewbot's allowlist" (examples/reviewbot/manifests/toolspecs.yaml:83 and :189) and on "no slot-bound grant" (pkg/authz/relwrites/relwrites.go:164). Rewording either message silently breaks the discriminator, and no test notices.
Fix: add a YAML comment next to both message: lines in toolspecs.yaml (83, 189) saying agentclass.yaml's Requested review section matches the phrase "not in reviewbot's allowlist", so the message must keep it. Put the comment in toolspecs.yaml, not inside this block scalar, where it would become prompt text.
Test: do not add a test that reads examples/.
Constraints: CLAUDE.md "Tests MUST NOT reference examples/" rules out a test that pins the prompt text.

that is by design: relay the refusal to the thread and stop — do not

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] The allowlist "relay and stop" rule is unscoped and conflicts with the new multi-PR rule

Problem: this sentence says to relay the refusal "and stop". Lines 568-570 say "A refusal or failure on one does not stop the rest." For a request naming two PRs where the second repo is not allowlisted, the model may end the whole turn instead of continuing to the other PR. The multi-PR paragraph also never says to report each failed PR in the thread. The "If you cannot review" section (line 510) is written around conclude_trigger_status, which this mode does not have, so a clone failure on one PR can end silently.
Fix: in the Requested review section, scope the sentence to the PR ("relay the refusal to the thread and stop reviewing THAT pull request"). In the multi-PR paragraph (lines 568-570), add that every PR that is refused or fails must get its own thread message stating the actual reason, and that the turn does not end until each PR has either a delivered review or such a message.
Test: none can prove a prompt claim. Re-run the multi-PR request by hand against a session with one allowlisted and one non-allowlisted repo and confirm each PR gets its own message. Validate the YAML with oap agent lint examples/reviewbot or equivalent.
Constraints: CLAUDE.md "Never silently drop errors" (surface a user-visible message for each failure).

retry or try another spelling of the URL. Any OTHER refusal is not an
allowlist refusal; never describe it as one. A "no slot-bound grant"
refusal means the call asked for `id` (step 2): drop it and retry.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] "Drop it and retry" has no bound and no outcome for a failed retry

Problem: the instruction does not say how many retries are allowed or what to report if the retry is also refused. It sits next to "do not retry" for the allowlist case, so the retry rule is easy to misapply.
Verify: confirm that a refusal whose text contains "no slot-bound grant" can only come from a call that asked for id (relwrites.go:164, block requireSlotBound). If so:
Fix: reword to "retry ONCE without id; if that call is refused too, relay the refusal text to the thread verbatim and stop reviewing that pull request".
Test: none can prove a prompt claim; check by hand.


Several pull requests in one request: review them one at a time, each

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] Multi-PR flow does not give each PR its own artifact

Problem: "each exactly as above" pulls in step 4 (line 428-429), which says "On a later round pass revises with the previous revision". That text is meant for a later push on the same PR. In one session reviewing PR A then PR B, the model can pass A's revision as revises for B, so one artifact carries both reviews' history and A's head-SHA tag. Step 2's clone dir (/workspace/<dir>, line 352) is also unspecified per PR.
Fix: add one sentence to the paragraph at lines 568-570: each pull request gets its own fresh artifact_prepare with no revises (revising applies only to a later push on the SAME pull request), and its own /workspace directory named after owner, repo and number.
Test: none can prove a prompt claim. Check by hand with a two-PR request and confirm two distinct artifacts, each attached by its own respond_to_user. The artifact-delivered requirement is already per render (pkg/agent/completion/kinds/artifactdelivery/artifactdelivery.go:81), so no code change is needed.

exactly as above, and deliver each review as its own message. A refusal
or failure on one does not stop the rest.

## Pausing and resuming a pull request
Someone may ask you to stop reviewing a particular pull request — "stop
Expand Down
Loading