Skip to content

Let agents watch GitHub pull requests for CI and feedback - #1322

Closed
baxen wants to merge 1 commit into
mainfrom
mbot/github-pr-watch
Closed

baxen wants to merge 1 commit into
mainfrom
mbot/github-pr-watch

Conversation

@baxen

@baxen baxen commented Oct 18, 2026

Copy link
Copy Markdown
Collaborator

Draft for the agent "outer loop": an agent watches a pull request, gets woken when CI finishes or feedback arrives, fixes, pushes, and keeps waiting. No model calls while waiting.

How it works

  • The bundled GitHub plugin adds an agent tool, watch_pr { url, stop? }, through ctx.agents.addTool. The tool call already knows the agent and the conversation (channel / thread / Solo), and the watch remembers both.
  • While the desktop app is open, the plugin reads each watched PR once a minute. It uses one gh api graphql --input - call through a new manifest host command, so it relies on the user's existing gh login and needs no token handling. Plain code decides when to wake the agent:
    • CI settled on the head commit: every check done, all of them on the first page, and the rollup not pending or expected. This is reported once per sha:result, so a failed head that turns green after a rerun is reported again.
    • New feedback: reviews, PR comments and inline comments from anyone except the gh account (which agents also push and comment as). It is batched until there has been 90s of quiet, so one review is one wake.
  • The wake is one message from the owner in that conversation that mentions the agent. Every harness (Claude, Codex, Goose) already handles that as a normal mention: owner trust check, session routing, thread context. There is no new host API and no harness changes.
  • Prompt-injection guard: the wake is signed by the owner, and agents trust the owner, so it never includes text written by anyone else. It lists only failed check names, comment authors, path:line and links. The agent reads the feedback itself with gh.
  • Persistence and lifetime: watches are saved in localStorage on the device. After a restart they are read at once, so nothing is lost while the app is closed, only delayed. A watch ends on merge or close, after 20 wakes, after 30 failed polls in a row (the agent is told), or on stop: true.

Why not a ctx.agents.wake host API

Each harness gets its owner trust check, the session to resume and the conversation context from the signed message that triggers a turn. A wake without a message would need a parallel path in all three harnesses. An owner-signed message reuses the proven path and also shows the loop's progress in the thread. The send is one function (wake in watcher.ts), so it can switch to a host API later.

Validation

  • vitest covers the github plugin, features/agents and plugins: 58 files, 753 tests passing. The new tests cover settling, once per sha:result, rerun-to-green, quiet batching, skipping the viewer's own comments, large seen sets, routing to a thread or the top level, a restart reading at once, unsent news being kept, the failure limit, stop, merge, and moving the watch to another conversation.
  • The plugin manager's Rust tests pass. The bundled manifest with the new host.commands entry validates.
  • The real query and parser were run against block/buzz-app Name Solo chats with goose's and Codex's own titles #1320, fix(composer): keep the caret after the first typed character #1316 and Add Mark as Read and archive/delete to Solo chats, with chat wording #1311. Responses were 4–12 KB, about 0.7s each.
  • An independent agent review found that comment bodies were quoted inside the owner's message (an injection risk). That is fixed, along with response size, the failure limit, isolating one watch's errors from the others, rerun semantics, seen trimming and URL case.

Not yet done: a native app run (bundled host commands need a native rebuild), and a real agent loop on a live PR.

Known limits

  • Your own GitHub comments on the PR don't wake the agent, because agents use your gh account.
  • Only the last 50 review threads are read.
  • A workflow that registers after all the others have finished could be reported as passed early.
  • Adding the tool replaces each agent's session once (addedKey); for Codex, that means a new thread.

🤖 Generated with Claude Code

The GitHub plugin adds a watch_pr agent tool. While the app is open it
reads each watched pull request through gh once a minute, and code
decides when to wake the agent: CI finished on the head commit, or new
feedback from someone other than the gh account after a quiet period.
The wake is a message from the owner in the conversation where the
agent asked, mentioning it, so every agent type handles it unchanged.

Co-authored-by: mbot <3754f8729004d95654c46dbab3129e4ab9ef05cc2534e2a3fbfc155983bd637b@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Bradley Axen <baxen@squareup.com>
baxen added a commit that referenced this pull request Oct 19, 2026
Both GitHub host entries pin --hostname github.com.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Bradley Axen <baxen@squareup.com>
@baxen

baxen commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

🤖 Folded into #1325, which now carries the whole GitHub integration.

@baxen baxen closed this Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant