feat: advisory auto-review for showcase contribution PRs - #98
Merged
Conversation
Showcase PRs currently wait for a maintainer before getting any feedback,
and CI only covers the mechanical half (validate-showcase.mjs). This adds a
first pass so contributors hear back immediately and reviewers start from a
draft instead of a blank page.
Pipeline, deterministic work first so the model reasons over real output
rather than guessing:
gather.mjs PR metadata, head-commit file contents, validator run,
URL checks, secret scan
gemini.mjs evidence + the review rubric -> structured JSON (gemini-2.5-pro)
post.mjs upserts one comment; inline suggestion blocks on the first pass
run.mjs orchestration, short-circuits when there is nothing to review
Advisory only: it posts a COMMENT and can never approve, request changes, or
merge. No label or comment gate -- every PR touching showcase/** is reviewed,
with the model's is_showcase_contribution call filtering out PRs that only
touch showcase tooling.
The review rubric lives in the private game-by-virtuals/devrel repo, so it is
fetched at runtime with a read-only PAT (DEVREL_SKILL_TOKEN). A missing or
expired token fails the run loudly rather than reviewing against no rubric.
Security: pull_request_target is required so fork PRs get a review at all, so
the workflow checks out the base commit only and never executes a file from
the PR head. Contributor files are fetched as text, enter the prompt as
explicitly-delimited untrusted data, and the model is given no tools -- the
worst case for an injection attempt is a badly worded comment. Running from
the base branch also means a PR cannot change how its own review behaves.
Notable correctness fixes found while testing against real PRs (97, 96, 69):
- refuse to run without a head sha, and fail when no text file is readable;
both previously degraded to a confident "validator passed" having read
nothing
- run the validator in a temp copy so a local dry run cannot dirty the
working tree
- copy the repo-root skills/ tree that some manifests reference via
skills[].sourcePath, which otherwise produced bogus "must contain SKILL.md"
failures; text files only, so showcase/ costs 340ms instead of 173MB
- resolve github.com blob/tree and raw.githubusercontent /main/ 404s against
the PR branch, separating the documented until-merge convention from an
actually stale href
The Gemini call and comment post are not covered by local testing (no API key
available); everything up to the prompt payload is verified.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The event trigger can only reach PRs opened after it ships, so the showcase PRs already open get no review. Adds a workflow_dispatch path that reviews an explicit list of PRs or every open one. - pr_numbers input takes "all" or a comma-separated list; PR_NUMBERS does the same locally, and PR_NUMBER still works for the event-driven single-PR run - dry_run input defaults to TRUE. A backfill posts to several PRs at once and that is hard to walk back, so posting is opt-in: read the job log first - head repo and sha now resolve from the PR itself when no event payload is present, which also shortens the local dry-run command to a PR number - PRs run sequentially and failures are isolated per PR, so one deleted fork or force-push no longer abandons the rest of the batch; the job fails only when nothing succeeded Two fixes found while testing the batch path against the seven currently open PRs: - coverage limits are now stated in the prompt. PR #95 changes 404 files, of which 60 are read, and 85 URLs, of which 40 are checked — the model was being asked to review that with no idea its view was partial, and would have reported as if it had seen everything - Gemini API errors are summarized to their message line instead of dumping the full nested JSON blob for every PR in a batch Verified against the open PRs: 7 enumerated, the non-showcase one skipped before spending a model call, coverage warnings present for #95 and absent for #96, and a non-zero exit when every PR fails. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
workflow_dispatch is unavailable until the workflow reaches the default branch, so this push-triggered job verifies DEVREL_SKILL_TOKEN and GEMINI_API_KEY work in CI while the feature is still on a branch. DRY_RUN=1, so nothing is posted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The first live dry run against PR 97 reported three should-fix findings, emitted zero inline suggestions, and wrote an overview saying "I've left them inline". Nothing would have reached the contributor: blockers/should_fix/minor are a maintainer summary that is never rendered into the comment, so all three findings existed only in the JSON. - tell the model those arrays are not shown to the author, and not to reference inline comments unless it actually emitted some - add a deterministic net rather than relying on prompt wording: any finding not already present in the overview is appended as "Also flagged". Word-overlap check avoids repeating a finding the overview reworded; when inline suggestions do exist, only blockers escalate - dry runs now print the real assembled body via preview_body, so the log shows what would actually be posted instead of the model's raw prose Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both suggestions from the first live dry run would have damaged the file if the author clicked Apply. The model was anchoring to estimated line numbers: one targeted line 10 (builder.name) while describing a topics edit, which parses as valid JSON but silently deletes builder.name; the other targeted line 23 inside visual with a full visual block, producing invalid JSON. It also asserted the singular topic field was deprecated, which the validator contradicts. Cause: file contents went into the prompt without line numbers, so the anchor was guesswork. - number every line in the prompt listing, and tell the model to read the anchor off the listing rather than estimate it, matching indentation and trailing comma so the file still parses - forbid claiming a field is deprecated/required unless the rubric or validator output says so; note that topic and topics are both required today - add screen_suggestions as a hard gate before anything is posted. JSON.parse alone was not enough (the builder.name case parses fine), so for showcase/<slug>/showcase.json it applies the patch and re-runs the real validator, rejecting any suggestion that would fail it. Also rejects unknown paths, out-of-range lines, and no-op suggestions - dry runs list dropped suggestions with the reason, so a rejection is visible rather than silent Verified against PR 97's real file set: both original bad suggestions rejected, correctly-anchored topics and posterUrl edits kept, no false positives. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
It existed only to verify the secrets and the model leg while the feature was still on a branch, since workflow_dispatch is unavailable until a workflow reaches the default branch. All three runs passed, so it has served its purpose. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
LauJoeYing
approved these changes
Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Showcase PRs wait for a maintainer before getting any feedback, and CI only covers the mechanical half (
validate-showcase.mjs). This adds a first pass so contributors hear back immediately and reviewers start from a draft instead of a blank page.It reuses the existing
review-showcase-prrubric, so the bot applies the same criteria a maintainer would — including the recent tone work, so the output reads like a teammate rather than a linter.Advisory only
Posts a COMMENT. It cannot approve, request changes, or merge, and a human still reviews every PR. The comment says as much and invites pushback.
How it triggers
pull_request_targetonopened/synchronize/reopened, filtered toshowcase/**. No label or comment gate — every showcase PR gets a review. Two filters decide whether it actually speaks:showcase/;is_showcase_contributioncall, which skips e.g. a maintainer editing showcase tooling.Pipeline
Deterministic work runs first, so the model reasons over real validator output and real HTTP statuses instead of guessing.
gather.mjsgemini.mjsgemini-2.5-pro)post.mjssuggestionblocks on the first passrun.mjsSecurity
pull_request_targetis needed for fork PRs to get a review at all (a fork PR on plainpull_requestgets a read-only token and no secrets). Because that trigger carries secrets and write access:head.sha;Running from the base branch also means a PR can't change how its own review behaves.
Correctness fixes found testing against real PRs (#97, #96, #69)
skills/tree that some manifests reference viaskills[].sourcePath(acp-earner-loopdoes), which otherwise produced bogusmust contain SKILL.mdfailures. Text files only:showcase/costs ~340ms instead of 173MB.github.comblob/tree andraw.githubusercontent/main/404s against the PR branch, separating the documented until-merge convention from an actually stale href.Verified on those three PRs: files read, validator green, 15/7/6 URLs classified with zero false "broken".
Before this can run
GEMINI_API_KEYDEVREL_SKILL_TOKEN— fine-grained read-only PAT, Contents: Read ongame-by-virtuals/devrel(the rubric is private). Token expiry is the likeliest future breakage.pull_request_targetwon't run for the PR that adds it.scripts/showcase-review/README.md).Not covered by local testing
The Gemini call and the comment post — no API key available locally. Everything up to and including the prompt payload is verified.
🤖 Generated with Claude Code