Skip to content

feat: advisory auto-review for showcase contribution PRs - #98

Merged
celesteanglm merged 6 commits into
mainfrom
feat/showcase-pr-auto-review
Aug 3, 2026
Merged

feat: advisory auto-review for showcase contribution PRs#98
celesteanglm merged 6 commits into
mainfrom
feat/showcase-pr-auto-review

Conversation

@celesteanglm

Copy link
Copy Markdown
Collaborator

Why

Showcase PRs wait for a maintainer before getting any feedback, and CI only covers the mechanical half (validate-showcase.mjs). This adds a first pass so contributors hear back immediately and reviewers start from a draft instead of a blank page.

It reuses the existing review-showcase-pr rubric, so the bot applies the same criteria a maintainer would — including the recent tone work, so the output reads like a teammate rather than a linter.

Advisory only

Posts a COMMENT. It cannot approve, request changes, or merge, and a human still reviews every PR. The comment says as much and invites pushback.

How it triggers

pull_request_target on opened / synchronize / reopened, filtered to showcase/**. No label or comment gate — every showcase PR gets a review. Two filters decide whether it actually speaks:

  1. structural — the PR must touch showcase/;
  2. the model's own is_showcase_contribution call, which skips e.g. a maintainer editing showcase tooling.

Pipeline

Deterministic work runs first, so the model reasons over real validator output and real HTTP statuses instead of guessing.

Stage File Does
Gather gather.mjs PR metadata, head-commit file contents, validator run, URL checks, secret scan
Judge gemini.mjs evidence + rubric → structured JSON (gemini-2.5-pro)
Post post.mjs upserts one comment; inline suggestion blocks on the first pass
Orchestrate run.mjs wiring, short-circuits when there's nothing to review

Security

pull_request_target is needed for fork PRs to get a review at all (a fork PR on plain pull_request gets a read-only token and no secrets). Because that trigger carries secrets and write access:

  • checks out the base commit, never head.sha;
  • no PR file is ever executed — fetched as text, read only;
  • contributor content enters the prompt as explicitly-delimited untrusted data and the model gets no tools, so the worst case for an injection attempt is a badly worded comment;
  • the validator runs in a temp copy, so local dry runs can't dirty a working tree.

Running from the base branch also means a PR can't change how its own review behaves.

Correctness fixes found testing against real PRs (#97, #96, #69)

  • Refuse to run without a head sha, and fail when no text file is readable — both previously degraded to a confident "validator passed" having read nothing.
  • Copy the repo-root skills/ tree that some manifests reference via skills[].sourcePath (acp-earner-loop does), which otherwise produced bogus must contain SKILL.md failures. Text files only: showcase/ costs ~340ms instead of 173MB.
  • Resolve github.com blob/tree and raw.githubusercontent /main/ 404s against the PR branch, separating the documented until-merge convention from an actually stale href.

Verified on those three PRs: files read, validator green, 15/7/6 URLs classified with zero false "broken".

Before this can run

  1. Add secrets:
    • GEMINI_API_KEY
    • DEVREL_SKILL_TOKEN — fine-grained read-only PAT, Contents: Read on game-by-virtuals/devrel (the rubric is private). Token expiry is the likeliest future breakage.
  2. Merge — pull_request_target won't run for the PR that adds it.
  3. Dry-run against a real PR first (command in scripts/showcase-review/README.md).

Not covered by local testing

The Gemini call and the comment post — no API key available locally. Everything up to and including the prompt payload is verified.

🤖 Generated with Claude Code

celesteanglm and others added 6 commits August 3, 2026 02:10
Showcase PRs currently wait for a maintainer before getting any feedback,
and CI only covers the mechanical half (validate-showcase.mjs). This adds a
first pass so contributors hear back immediately and reviewers start from a
draft instead of a blank page.

Pipeline, deterministic work first so the model reasons over real output
rather than guessing:

  gather.mjs  PR metadata, head-commit file contents, validator run,
              URL checks, secret scan
  gemini.mjs  evidence + the review rubric -> structured JSON (gemini-2.5-pro)
  post.mjs    upserts one comment; inline suggestion blocks on the first pass
  run.mjs     orchestration, short-circuits when there is nothing to review

Advisory only: it posts a COMMENT and can never approve, request changes, or
merge. No label or comment gate -- every PR touching showcase/** is reviewed,
with the model's is_showcase_contribution call filtering out PRs that only
touch showcase tooling.

The review rubric lives in the private game-by-virtuals/devrel repo, so it is
fetched at runtime with a read-only PAT (DEVREL_SKILL_TOKEN). A missing or
expired token fails the run loudly rather than reviewing against no rubric.

Security: pull_request_target is required so fork PRs get a review at all, so
the workflow checks out the base commit only and never executes a file from
the PR head. Contributor files are fetched as text, enter the prompt as
explicitly-delimited untrusted data, and the model is given no tools -- the
worst case for an injection attempt is a badly worded comment. Running from
the base branch also means a PR cannot change how its own review behaves.

Notable correctness fixes found while testing against real PRs (97, 96, 69):

- refuse to run without a head sha, and fail when no text file is readable;
  both previously degraded to a confident "validator passed" having read
  nothing
- run the validator in a temp copy so a local dry run cannot dirty the
  working tree
- copy the repo-root skills/ tree that some manifests reference via
  skills[].sourcePath, which otherwise produced bogus "must contain SKILL.md"
  failures; text files only, so showcase/ costs 340ms instead of 173MB
- resolve github.com blob/tree and raw.githubusercontent /main/ 404s against
  the PR branch, separating the documented until-merge convention from an
  actually stale href

The Gemini call and comment post are not covered by local testing (no API key
available); everything up to the prompt payload is verified.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The event trigger can only reach PRs opened after it ships, so the showcase PRs
already open get no review. Adds a workflow_dispatch path that reviews an
explicit list of PRs or every open one.

- pr_numbers input takes "all" or a comma-separated list; PR_NUMBERS does the
  same locally, and PR_NUMBER still works for the event-driven single-PR run
- dry_run input defaults to TRUE. A backfill posts to several PRs at once and
  that is hard to walk back, so posting is opt-in: read the job log first
- head repo and sha now resolve from the PR itself when no event payload is
  present, which also shortens the local dry-run command to a PR number
- PRs run sequentially and failures are isolated per PR, so one deleted fork or
  force-push no longer abandons the rest of the batch; the job fails only when
  nothing succeeded

Two fixes found while testing the batch path against the seven currently open
PRs:

- coverage limits are now stated in the prompt. PR #95 changes 404 files, of
  which 60 are read, and 85 URLs, of which 40 are checked — the model was being
  asked to review that with no idea its view was partial, and would have
  reported as if it had seen everything
- Gemini API errors are summarized to their message line instead of dumping the
  full nested JSON blob for every PR in a batch

Verified against the open PRs: 7 enumerated, the non-showcase one skipped
before spending a model call, coverage warnings present for #95 and absent for
#96, and a non-zero exit when every PR fails.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
workflow_dispatch is unavailable until the workflow reaches the default branch,
so this push-triggered job verifies DEVREL_SKILL_TOKEN and GEMINI_API_KEY work
in CI while the feature is still on a branch. DRY_RUN=1, so nothing is posted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The first live dry run against PR 97 reported three should-fix findings, emitted
zero inline suggestions, and wrote an overview saying "I've left them inline".
Nothing would have reached the contributor: blockers/should_fix/minor are a
maintainer summary that is never rendered into the comment, so all three
findings existed only in the JSON.

- tell the model those arrays are not shown to the author, and not to reference
  inline comments unless it actually emitted some
- add a deterministic net rather than relying on prompt wording: any finding not
  already present in the overview is appended as "Also flagged". Word-overlap
  check avoids repeating a finding the overview reworded; when inline
  suggestions do exist, only blockers escalate
- dry runs now print the real assembled body via preview_body, so the log shows
  what would actually be posted instead of the model's raw prose

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both suggestions from the first live dry run would have damaged the file if the
author clicked Apply. The model was anchoring to estimated line numbers: one
targeted line 10 (builder.name) while describing a topics edit, which parses as
valid JSON but silently deletes builder.name; the other targeted line 23 inside
visual with a full visual block, producing invalid JSON. It also asserted the
singular topic field was deprecated, which the validator contradicts.

Cause: file contents went into the prompt without line numbers, so the anchor
was guesswork.

- number every line in the prompt listing, and tell the model to read the anchor
  off the listing rather than estimate it, matching indentation and trailing
  comma so the file still parses
- forbid claiming a field is deprecated/required unless the rubric or validator
  output says so; note that topic and topics are both required today
- add screen_suggestions as a hard gate before anything is posted. JSON.parse
  alone was not enough (the builder.name case parses fine), so for
  showcase/<slug>/showcase.json it applies the patch and re-runs the real
  validator, rejecting any suggestion that would fail it. Also rejects unknown
  paths, out-of-range lines, and no-op suggestions
- dry runs list dropped suggestions with the reason, so a rejection is visible
  rather than silent

Verified against PR 97's real file set: both original bad suggestions rejected,
correctly-anchored topics and posterUrl edits kept, no false positives.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
It existed only to verify the secrets and the model leg while the feature was
still on a branch, since workflow_dispatch is unavailable until a workflow
reaches the default branch. All three runs passed, so it has served its purpose.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@celesteanglm
celesteanglm merged commit e52439f into main Aug 3, 2026
1 check passed
@celesteanglm
celesteanglm deleted the feat/showcase-pr-auto-review branch August 3, 2026 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants