Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions .github/workflows/pr-risk.yml
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,11 @@ name: PR Risk Grade (reusable)
# uses: Comfy-Org/github-workflows/.github/workflows/pr-risk.yml@<sha> # v1
# with:
# workflows_ref: <same sha>
# # OFF BY DEFAULT. Enrolling and switching on are two decisions: land the caller, get
# # it reviewed, then start grading. Either pin it on here, or leave this out and set
# # the repo variable RISK_CONFIG to {"enabled": true} — the variable outranks this
# # input in BOTH directions, so {"enabled": false} is also a kill switch needing no PR.
# enabled: true
# # Both are empty on a `pull_request` run (`inputs` is empty there), which is exactly
# # the no-input event path — so ONE caller shape serves both event and dispatch and
# # there is nothing to keep in sync between two jobs.
Expand Down Expand Up @@ -253,12 +258,69 @@ on:
is not a chain.
type: string
required: true
enabled:
description: >-
Whether to grade at all. DEFAULTS TO FALSE: enrolling this workflow and
switching it on are two separate decisions, so a repo can land the
caller, get it reviewed, and start grading later — or stop grading
without reverting anything. When it resolves false the `grade` job does
not run: nothing is read, nothing is labelled, and no existing `risk:*`
label is touched (a disabled run leaves whatever the last enabled one
left, it does not clean up).


`vars.RISK_CONFIG` on the CALLING repo OVERRIDES this, in both
directions — `{"enabled": true}` switches a caller on with no `with:`
change, and `{"enabled": false}` is a kill switch that needs no PR at
all. That works because the `vars` context inside a reusable resolves
against the caller's repository (the same mechanism groom.yml uses for
`vars.GROOM_CONFIG`). The variable is the OPERATIONAL lever; this input
is the REVIEWED default the variable falls back to when it is absent,
empty, not an object, or carries no boolean `enabled` — a malformed
variable must never be the reason a repo silently stops grading, so it
degrades to this value and says so in an annotation.
type: boolean
required: false
default: false

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — Defaulting enabled to false silently disables grading for every already-enrolled caller the next time it moves its SHA pin, and a disabled run deliberately leaves the previous risk:* label in place — so those PRs keep showing a stale grade indistinguishable from a current one. There is no bump-pr-risk-callers fleet, so pins move by hand with no coordinated migration; this is a behavior break riding the in-place v1 tag. Raised by 2 of 8 reviewers (claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case).


permissions:
contents: read

jobs:
# Resolve enablement before anything is read. Its own job rather than a first step of `grade`
# so a disabled repo pays one bare runner instead of booting the grading job to immediately
# no-op, and so the decision — and the reason for it — is visible in the run graph rather than
# buried in a step log. It holds NO token scopes: it reads only its inputs and the caller's
# variable, and must never be the thing that touches a PR.
gate:
name: Resolve enablement
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The gate job declares permissions: {} but immediately runs actions/checkout against Comfy-Org/github-workflows, which presents the zero-scope GITHUB_TOKEN rather than falling back to anonymous read. If that clone is refused the gate job fails, enabled is empty, and the needs.gate.outputs.enabled == 'true' condition skips grading in every consumer — the sibling gate in groom.yml and the grade job here both declare contents: read, so match that proven pattern. Raised by 3 of 8 reviewers (gpt-5.6-sol-max edge-case, claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case).

outputs:
enabled: ${{ steps.resolve.outputs.enabled }}
steps:
# The tool checkout is the same pinned-ref load the grade job does: the resolver is this
# repo's code at `workflows_ref`, never the graded PR's.
- name: Load pr-risk tool
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: Comfy-Org/github-workflows
ref: ${{ inputs.workflows_ref }}
path: _pr_risk_tool
persist-credentials: false
- name: Resolve enabled
id: resolve
env:
INPUT_ENABLED: ${{ inputs.enabled }}
# Empty when the caller repo has no such variable — which is the common case and is
# NOT an error; it just means the reviewed input decides.
RISK_CONFIG: ${{ vars.RISK_CONFIG }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Lowvars.RISK_CONFIG is an out-of-band disable for a compliance-adjacent check that leaves no trace in git history, and because vars also resolves organization-level variables, one org variable of {"enabled": false} turns grading off across every enrolled repo at once. The disabled path deliberately emits no ::warning::, so the only evidence is a step summary nobody opens on a green run — emit an annotation whenever the variable is what turned grading off. Raised by 1 of 8 reviewers (claude-opus-5-thinking-max adversarial).

run: bash _pr_risk_tool/scripts/pr-risk/resolve-enabled.sh

grade:
needs: gate
if: needs.gate.outputs.enabled == 'true'
name: Grade PR risk
runs-on: ubuntu-latest
timeout-minutes: 30
Expand Down
8 changes: 7 additions & 1 deletion .github/workflows/test-pr-risk.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ jobs:
persist-credentials: false

- name: shellcheck
run: shellcheck -x grade-pr-risk.sh apply-risk-label.sh grade-targets.sh tests/test_grade_pr_risk.sh tests/test_apply_risk_label.sh tests/test_grade_targets.sh
run: shellcheck -x grade-pr-risk.sh apply-risk-label.sh grade-targets.sh resolve-enabled.sh tests/test_grade_pr_risk.sh tests/test_apply_risk_label.sh tests/test_grade_targets.sh tests/test_resolve_enabled.sh

- name: default map + registry parse and validate
# The shipped defaults must pass the grader's own structural validation:
Expand All @@ -68,3 +68,9 @@ jobs:
# contract, the settle poll, per-target failure isolation. Hermetic — `gh` is stubbed and
# every call it receives is logged, so the suite asserts on which requests were made.
run: bash tests/test_grade_targets.sh

- name: enablement suite
# The `enabled` switch: that `vars.RISK_CONFIG` outranks the reviewed input in BOTH
# directions, and that a malformed variable degrades to the reviewed value rather than
# to off — a typo must never be indistinguishable from a deliberate shutdown.
run: bash tests/test_resolve_enabled.sh
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ This repo is **public** so any repo — public or private, inside or outside the
| [`assign-reviewers.yml`](.github/workflows/assign-reviewers.yml) | Auto-requests expertise-aware, load-balanced PR reviewers with new-folk randomization. Matches changed paths against a caller-repo `.github/reviewers.yml` (path-glob → reviewers, plus a `default_pool`), drops the author + `vars.REVIEWER_EXCLUDE`, ranks candidates by open review load (steering off anyone at/over `vars.REVIEWER_LOAD_CAP`), and may swap a slot for a `vars.REVIEWER_GROWTH_POOL` member. Requests go through the CLOUD_CODE_BOT app token so they work on fork PRs. Requires `vars.APP_ID` + `CLOUD_CODE_BOT_PRIVATE_KEY`. |
| [`assign-prs-to-author.yml`](.github/workflows/assign-prs-to-author.yml) | Housekeeping — assigns every open PR with no assignees to its author (bot-authored PRs skipped by default). Run on a schedule from a thin caller; useful when a team tracks PR ownership via assignees. The calling job needs `pull-requests: write` and `issues: write`. |
| [`pr-size.yml`](.github/workflows/pr-size.yml) | PR-size cap — fails (or, in `mode: warn`, only reports) when a PR's net diff exceeds `max_lines` non-generated changed lines, keeping diffs reviewable. Excludes dependency lockfiles, `linguist-generated` files (read from the base ref, so a PR can't exempt itself), Go generated-code markers, and per-repo `extra_lockfiles` / `extra_generated_globs`. A `bypass_label` (default `oversized-ok`) waves through a legitimately large change; a sticky bot comment explains overages when `bot_app_id` + `BOT_APP_PRIVATE_KEY` are supplied (degrades to status + step summary without them). Counting logic + tests live in [`scripts/check-pr-size/`](scripts/check-pr-size). |
| [`pr-risk.yml`](.github/workflows/pr-risk.yml) | **Advisory PR risk grading (shadow check)** — grades every PR into a tier `R0` (safest) .. `R3` (riskiest) and syncs one label (`risk:R0`..`risk:R3`, or `risk:ungraded` when an input was unreadable). The label is the entire product: nothing is gated, routed, commented, or merged. Deterministic (`gh` + `jq`, no LLM): `grade = worst(path_floor, provenance, reversibility)` — path-glob map, what-process-produced-the-diff (registered runbooks with identity + diff-shape assertions; forks are R3 with no exceptions), and revertability (persistent-state mutation, deletions under sensitive classes, did green checks cover the lines). Grader + generic defaults live in [`scripts/pr-risk/`](scripts/pr-risk); a consumer sharpens them with `.github/risk.json` / `.github/risk-runbooks.json`, read from the PR's **base ref** so a PR can't edit the rules that judge it. The job excludes its own run from the check rollup and waits (`wait_for_checks_minutes`) for the rest to settle before labeling. Labels ride the plain `GITHUB_TOKEN` (cannot fire `labeled` triggers — no cascade risk); disagreement is recorded with a human-owned `risk-dispute` label. Label text is remappable via `label_map`. `workflows_ref` is **required** — pin it to the same full commit SHA as `uses:`, so the grader cannot be loaded from a floating ref after the caller was reviewed. Enroll it as its own workflow rather than a job inside an existing CI workflow (the rollup exclusion is per-run). The calling job needs `contents: read` + `issues: write` + `pull-requests: write` + `checks: read` + `actions: read` + `statuses: read`; GitHub rejects a shorter grant at startup (a reusable workflow can only narrow the caller's token, never elevate it), so a caller enrolled from an older copy of this row fails before any step runs. Both writes are the ONE label: repo-side label creation on first use maps to `issues`, and labeling a PR maps to `pull-requests` (the labels endpoint is dual-mapped by what the "issue" is, so `issues: write` alone 403s on a PR). `actions: read` is for the rollup's `CheckRun -> checkSuite -> workflowRun` self-exclusion hop. No secrets. |
| [`pr-risk.yml`](.github/workflows/pr-risk.yml) | **Advisory PR risk grading (shadow check)** — **off by default** (`enabled: false`); switch it on with `enabled: true` or by setting the caller repo's `RISK_CONFIG` variable to `{"enabled": true}`, which outranks the input in both directions so `{"enabled": false}` is a no-PR kill switch. Grades every PR into a tier `R0` (safest) .. `R3` (riskiest) and syncs one label (`risk:R0`..`risk:R3`, or `risk:ungraded` when an input was unreadable). The label is the entire product: nothing is gated, routed, commented, or merged. Deterministic (`gh` + `jq`, no LLM): `grade = worst(path_floor, provenance, reversibility)` — path-glob map, what-process-produced-the-diff (registered runbooks with identity + diff-shape assertions; forks are R3 with no exceptions), and revertability (persistent-state mutation, deletions under sensitive classes, did green checks cover the lines). Grader + generic defaults live in [`scripts/pr-risk/`](scripts/pr-risk); a consumer sharpens them with `.github/risk.json` / `.github/risk-runbooks.json`, read from the PR's **base ref** so a PR can't edit the rules that judge it. The job excludes its own run from the check rollup and waits (`wait_for_checks_minutes`) for the rest to settle before labeling. Labels ride the plain `GITHUB_TOKEN` (cannot fire `labeled` triggers — no cascade risk); disagreement is recorded with a human-owned `risk-dispute` label. Label text is remappable via `label_map`. `workflows_ref` is **required** — pin it to the same full commit SHA as `uses:`, so the grader cannot be loaded from a floating ref after the caller was reviewed. Enroll it as its own workflow rather than a job inside an existing CI workflow (the rollup exclusion is per-run). The calling job needs `contents: read` + `issues: write` + `pull-requests: write` + `checks: read` + `actions: read` + `statuses: read`; GitHub rejects a shorter grant at startup (a reusable workflow can only narrow the caller's token, never elevate it), so a caller enrolled from an older copy of this row fails before any step runs. Both writes are the ONE label: repo-side label creation on first use maps to `issues`, and labeling a PR maps to `pull-requests` (the labels endpoint is dual-mapped by what the "issue" is, so `issues: write` alone 403s on a PR). `actions: read` is for the rollup's `CheckRun -> checkSuite -> workflowRun` self-exclusion hop. No secrets. |
| [`stale.yml`](.github/workflows/stale.yml) | Stale-PR sweeper (`actions/stale`) plus a Slack digest of what it touched. PRs inactive for N days are labeled `stale`; still-inactive PRs are closed. The digest header names the source repo so batches from different repos posted to the same channel are unambiguous. Thresholds, messages, exempt labels, and the Slack channel are inputs; the caller owns the schedule + dry-run toggle. The calling job needs `pull-requests: write` and `issues: write`. Optional `SLACK_BOT_TOKEN`. |
| [`groom.yml`](.github/workflows/groom.yml) | Scheduled/dispatch org-wide **code-cleanup sweep** (finds only — no commits, no PRs, never merges). A read-only FINDER agent scans a clean default-branch checkout (whole-repo, not a diff) for high-value refactors; an INDEPENDENT VERIFIER agent (fresh session) re-checks each as CONFIRM/DOWNGRADE/REJECT with a stable dedup signature; survivors are deduped against a durable GitHub-issue-state ledger and filed as `groom`-labeled GitHub issues (security-adjacent ones get `groom-security` — investigate, don't auto-implement). Mirrors the cursor-review topology: briefs + ledger live in [`.github/groom/`](.github/groom) as the single source of truth. The finder/verifier/builder agent jobs invoke the Claude CLI directly and mint no GitHub token, so they need nothing beyond `contents: read`; filing runs in a separate job as the bot you configure via `bot_app_id` (Comfy: cloud-code-bot). `dry_run` reports what it would file without opening issues. Runs on a **daily base cron** with a runtime cadence gate: set repo Actions variable `GROOM_INTERVAL_DAYS` (default 7 = weekly) to retune how often a real run happens — weekly → every-3-days → daily — with no workflow-file edit; a tick within the interval no-ops before the finder (`workflow_dispatch` bypasses the interval gate, but the volume gate — when the caller leaves it on — still applies). The calling job must grant `contents: read` + `issues: write` + `pull-requests: read` + `actions: read` — the first three are declared by the `file` / `build_select` jobs (needed even with `bot_app_id` set), and the interval gate needs `actions: read` (reads run history for the last real run); GitHub rejects a shorter grant at startup. Requires `ANTHROPIC_API_KEY` (+ `BOT_APP_PRIVATE_KEY` when `bot_app_id` is set). **Opt-in auto-builder** (`builder: true`, BE-4003): the top `max_prs` (default 5) CONFIRMED, non-security findings become **review-gated PRs** (full CI + cursor-review, **never auto-merged**) instead of issues; a credential-free `build` job emits only a patch artifact and a separate `build_pr` job opens the PR as the bot, preserving the security boundary. The ledger's PR-state (open/merged/closed) stops a built finding being re-proposed. Requires `bot_app_id`. `max_prs` is typed **`string`**, not `number`, so a caller can forward its own `workflow_dispatch` input straight through (`max_prs: ${{ github.event.inputs.max_prs \|\| '1' }}`) and let an operator raise the ceiling for one manual run — no `fromJSON()` cast in the caller, and the parse/clamp (empty → default, non-numeric → 0 PRs + warning, never a failed run) happens once inside the reusable. |
| [`agents-md-integrity.yml`](.github/workflows/agents-md-integrity.yml) | Enforces the Comfy `AGENTS.md` standard on the caller repo: a top-level `AGENTS.md` must exist and stay under a hard line ceiling (`max_lines`, default 200; warns over `warn_lines`, default 150), a `CLAUDE.md` (if present) must be a thin `@AGENTS.md` shim rather than a divergent copy, no legacy `.cursorrules` (gated `forbid_cursorrules`), every nested monorepo `AGENTS.md` needs a sibling `@AGENTS.md` shim and to be under the ceiling (gated `check_nested`), and `AGENTS.md` should have a CODEOWNERS DRI (`require_codeowners`, warn-only by default). Fails with a non-zero exit + GitHub annotations so it wires in as a required status check. The checker lives in [`.github/agents-md-integrity/`](.github/agents-md-integrity) (pin `workflows_ref` to the same ref as `uses:`); no secrets required. |
Expand Down
75 changes: 75 additions & 0 deletions scripts/pr-risk/resolve-enabled.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
#!/usr/bin/env bash
# resolve-enabled.sh — decide whether pr-risk grades this run, and say who decided.
#
# Two layers, and the VARIABLE outranks the INPUT in both directions:
#
# vars.RISK_CONFIG `{"enabled": true}` switches a caller on with no `with:` change;
# (operational) `{"enabled": false}` is a kill switch that needs no PR at all. The `vars`
# context inside a reusable resolves against the CALLING repository, which
# is what makes this reachable from here (same mechanism groom.yml uses for
# vars.GROOM_CONFIG).
#
# inputs.enabled the reviewed default, pinned in the caller's workflow file. Used whenever
# (reviewed) the variable does not say — absent, blank, not an object, or carrying an
# `enabled` that is not a boolean.
#
# DEGRADE TOWARD THE REVIEWED VALUE, NEVER TOWARD OFF. A malformed variable must not be the
# reason a repo silently stops grading, so a value this cannot read is announced and discarded
# rather than treated as `false`. "The variable is broken" and "the operator switched it off"
# are different states and must not look alike.
#
# A well-formed object with no `enabled` key is silent, not a warning: `enabled` is the only key
# today, and warning on every run would train operators to ignore the annotation that flags a
# real typo — and would start lying the moment a second key lands here.
#
# Env in: INPUT_ENABLED (the reusable's `enabled` input), RISK_CONFIG (the caller's variable).
# Env out: GITHUB_OUTPUT gets `enabled=true|false`; GITHUB_STEP_SUMMARY explains a disabled run.
# Both are optional so the script is runnable — and testable — outside Actions.
set -uo pipefail

warn() { printf '::warning::%s\n' "$1" >&2; }

reviewed="${INPUT_ENABLED:-false}"
# Anything that is not exactly `true` is off. The input arrives as a workflow-call boolean, so
# this is belt-and-braces against a caller forwarding a string expression into it.
[ "$reviewed" = true ] || reviewed=false

enabled="$reviewed"
decided_by="the caller's reviewed \`enabled:\` input"

raw="${RISK_CONFIG:-}"
if [ -n "${raw//[[:space:]]/}" ]; then
if ! jq -e 'type == "object"' >/dev/null 2>&1 <<<"$raw"; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium! jq -e 'type == "object"' with stderr discarded treats every non-zero exit as "not a JSON object" — including exit 127 when jq is missing from the runner, OOM, or any jq-internal error. A perfectly valid {"enabled": true} is then discarded, the kill switch stops working, and the annotation blames the operator's variable for a tool failure. Distinguish jq exit 1 (predicate false) from exit >1 (jq itself failed) and fail loudly on the latter. Raised by 2 of 8 reviewers (claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case).

warn "vars.RISK_CONFIG is set but is not a JSON object — ignoring it and using the reviewed input (enabled=${reviewed}). Expected {\"enabled\": true}."
elif jq -e 'has("enabled")' >/dev/null 2>&1 <<<"$raw"; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Lowjq -e exit status reflects only the truthiness of the LAST output value, so a multi-document RISK_CONFIG like {"enabled": true} {} passes the type == "object" test but makes has("enabled") exit 1, dropping into the deliberately-silent no-key path — the operator's enable is discarded with no warning at all. Assert a single document, e.g. jq -se 'length == 1 and (.[0] | type == "object")'. Raised by 1 of 8 reviewers (claude-opus-5-thinking-max adversarial).

# Read the value as TEXT and test the text. `jq -e` is not usable here: its exit code
# reflects the TRUTHINESS of the output, so a perfectly valid `{"enabled": false}` exits 1
# and would be mistaken for a parse failure — silently disarming the kill switch, which is
# the half of this lever that has to work when something is going wrong.
val="$(jq -r '.enabled | if type == "boolean" then tostring else "" end' <<<"$raw" 2>/dev/null)"
if [ "$val" = true ] || [ "$val" = false ]; then
enabled="$val"
# shellcheck disable=SC2016 # markdown backticks for the log line, not a substitution
decided_by='`vars.RISK_CONFIG`'
else
warn "vars.RISK_CONFIG has an \`enabled\` key but it is not a boolean — ignoring it and using the reviewed input (enabled=${reviewed}). Use true/false, not \"true\"."
fi
fi
fi

[ -z "${GITHUB_OUTPUT:-}" ] || echo "enabled=$enabled" >> "$GITHUB_OUTPUT"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The append to $GITHUB_OUTPUT is unchecked and the script runs under set -uo pipefail without -e, so a failed write leaves the step green with an empty enabled output; grade then skips as though the repo were disabled — the exact "degrade toward off" the file header forbids. Check the write and fail the step (or treat a missing output as an error in the workflow). Raised by 4 of 8 reviewers (gpt-5.6-sol-max edge-case, kimi-k2.7-code edge-case, claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case).

echo "pr-risk enabled=$enabled (decided by ${decided_by})"

if [ "$enabled" != true ] && [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
# shellcheck disable=SC2016 # the body is literal markdown; backticks are code spans
{
echo "## PR risk grading is DISABLED for this repository"
echo
echo "No grade was computed, and no \`risk:*\` label was added, changed, or removed —"
echo "any label already on this PR is left exactly as the last enabled run left it."

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit — The summary says "any label already on this PR is left exactly as the last enabled run left it", but the gate also runs on the pr_numbers batch-dispatch path where there are N target PRs (and on a dispatch with no target, none at all). Reword so it does not assume a single event-scoped pull request. Raised by 1 of 8 reviewers (claude-opus-5-thinking-max edge-case).

echo
echo "Decided by ${decided_by}. To switch grading on, either set the repository variable"
echo '`RISK_CONFIG` to `{"enabled": true}` (takes effect on the next run, no PR needed), or'
echo 'pass `enabled: true` in the caller'"'"'s `with:` block.'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The disabled-run summary always offers enabled: true in the caller's with: block as a remedy, but when decided_by is vars.RISK_CONFIG the variable outranks the input and that advice provably will not work. Branch the remediation text on decided_by and, for the kill-switch case, say the variable must be changed or removed first. Raised by 1 of 8 reviewers (claude-opus-5-thinking-max edge-case).

} >> "$GITHUB_STEP_SUMMARY"
fi
Loading
Loading