feat(pr-risk): advisory PR risk grader + label-only shadow check (reusable)

.github/workflows/test-pr-risk.yml

@@ -0,0 +1,60 @@
+# CONTRACT: this is one of THIS repo's own path-filtered script tests, NOT a reusable
+# workflow. It is not callable and has no caller pattern:
+#   TRIGGERS — `pull_request` and `push` to main, both filtered to scripts/pr-risk/**
+#              and this file. A change elsewhere in the repo does not run it.
+#   INPUTS   — none (no `workflow_call`, no `workflow_dispatch`).
+#   SECRETS  — none. Read-only `contents: read`; the suites are hermetic and make no
+#              network call (the live-PR path stubs `gh` on PATH with a fixture).
+# The reusable workflow these scripts back is .github/workflows/pr-risk.yml — that is
+# the file with the inputs/secrets/caller-pattern header.
+
+name: Test pr-risk scripts
+
+# Runs the hermetic test suites (+ shellcheck) for the grader and label-sync
+# scripts behind pr-risk.yml. The grader stamps an advisory risk tier on every
+# consumer repo's PRs, so a grading regression silently mislabels PRs org-wide
+# — cheap to guard with a unit run on change. No network: the suites feed the
+# grader synthetic records and stub `gh` for the live-PR path.
+
+on:
+  pull_request:
+    paths:
+      - 'scripts/pr-risk/**'
+      - '.github/workflows/test-pr-risk.yml'
+  push:
+    branches: [main]
+    paths:
+      - 'scripts/pr-risk/**'
+      - '.github/workflows/test-pr-risk.yml'
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    name: shellcheck + suites
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: scripts/pr-risk
+    steps:
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
+        with:
+          persist-credentials: false
+
+      - name: shellcheck
+        run: shellcheck -x grade-pr-risk.sh apply-risk-label.sh tests/test_grade_pr_risk.sh tests/test_apply_risk_label.sh
+
+      - name: default map + registry parse and validate
+        # The shipped defaults must pass the grader's own structural validation:
+        # a malformed default would fail every consumer at once.
+        run: |
+          bash -c 'source ./grade-pr-risk.sh; read_map risk-map.v0.json map >/dev/null && read_map runbook-registry.v0.json runbooks >/dev/null'
+          echo "defaults validate"
+
+      - name: grader suite
+        run: bash tests/test_grade_pr_risk.sh
+
+      - name: label suite
+        run: bash tests/test_apply_risk_label.sh

README.md

@@ -14,6 +14,7 @@ This repo is **public** so any repo — public or private, inside or outside the
 | [`assign-reviewers.yml`](.github/workflows/assign-reviewers.yml) | Auto-requests expertise-aware, load-balanced PR reviewers with new-folk randomization. Matches changed paths against a caller-repo `.github/reviewers.yml` (path-glob → reviewers, plus a `default_pool`), drops the author + `vars.REVIEWER_EXCLUDE`, ranks candidates by open review load (steering off anyone at/over `vars.REVIEWER_LOAD_CAP`), and may swap a slot for a `vars.REVIEWER_GROWTH_POOL` member. Requests go through the CLOUD_CODE_BOT app token so they work on fork PRs. Requires `vars.APP_ID` + `CLOUD_CODE_BOT_PRIVATE_KEY`. |
 | [`assign-prs-to-author.yml`](.github/workflows/assign-prs-to-author.yml) | Housekeeping — assigns every open PR with no assignees to its author (bot-authored PRs skipped by default). Run on a schedule from a thin caller; useful when a team tracks PR ownership via assignees. The calling job needs `pull-requests: write` and `issues: write`. |
 | [`pr-size.yml`](.github/workflows/pr-size.yml) | PR-size cap — fails (or, in `mode: warn`, only reports) when a PR's net diff exceeds `max_lines` non-generated changed lines, keeping diffs reviewable. Excludes dependency lockfiles, `linguist-generated` files (read from the base ref, so a PR can't exempt itself), Go generated-code markers, and per-repo `extra_lockfiles` / `extra_generated_globs`. A `bypass_label` (default `oversized-ok`) waves through a legitimately large change; a sticky bot comment explains overages when `bot_app_id` + `BOT_APP_PRIVATE_KEY` are supplied (degrades to status + step summary without them). Counting logic + tests live in [`scripts/check-pr-size/`](scripts/check-pr-size). |
+| [`pr-risk.yml`](.github/workflows/pr-risk.yml) | **Advisory PR risk grading (shadow check)** — grades every PR into a tier `R0` (safest) .. `R3` (riskiest) and syncs one label (`risk:R0`..`risk:R3`, or `risk:ungraded` when an input was unreadable). The label is the entire product: nothing is gated, routed, commented, or merged. Deterministic (`gh` + `jq`, no LLM): `grade = worst(path_floor, provenance, reversibility)` — path-glob map, what-process-produced-the-diff (registered runbooks with identity + diff-shape assertions; forks are R3 with no exceptions), and revertability (persistent-state mutation, deletions under sensitive classes, did green checks cover the lines). Grader + generic defaults live in [`scripts/pr-risk/`](scripts/pr-risk); a consumer sharpens them with `.github/risk.json` / `.github/risk-runbooks.json`, read from the PR's **base ref** so a PR can't edit the rules that judge it. The job excludes its own run from the check rollup and waits (`wait_for_checks_minutes`) for the rest to settle before labeling. Labels ride the plain `GITHUB_TOKEN` (cannot fire `labeled` triggers — no cascade risk); disagreement is recorded with a human-owned `risk-dispute` label. Label text is remappable via `label_map`. `workflows_ref` is **required** — pin it to the same full commit SHA as `uses:`, so the grader cannot be loaded from a floating ref after the caller was reviewed. Enroll it as its own workflow rather than a job inside an existing CI workflow (the rollup exclusion is per-run). The calling job needs `issues: write` + `pull-requests: read` + `checks: read` + `statuses: read`; no secrets. |
 | [`stale.yml`](.github/workflows/stale.yml) | Stale-PR sweeper (`actions/stale`) plus a Slack digest of what it touched. PRs inactive for N days are labeled `stale`; still-inactive PRs are closed. The digest header names the source repo so batches from different repos posted to the same channel are unambiguous. Thresholds, messages, exempt labels, and the Slack channel are inputs; the caller owns the schedule + dry-run toggle. The calling job needs `pull-requests: write` and `issues: write`. Optional `SLACK_BOT_TOKEN`. |
 | [`groom.yml`](.github/workflows/groom.yml) | Scheduled/dispatch org-wide **code-cleanup sweep** (finds only — no commits, no PRs, never merges). A read-only FINDER agent scans a clean default-branch checkout (whole-repo, not a diff) for high-value refactors; an INDEPENDENT VERIFIER agent (fresh session) re-checks each as CONFIRM/DOWNGRADE/REJECT with a stable dedup signature; survivors are deduped against a durable GitHub-issue-state ledger and filed as `groom`-labeled GitHub issues (security-adjacent ones get `groom-security` — investigate, don't auto-implement). Mirrors the cursor-review topology: briefs + ledger live in [`.github/groom/`](.github/groom) as the single source of truth. The finder/verifier/builder agent jobs invoke the Claude CLI directly and mint no GitHub token, so they need nothing beyond `contents: read`; filing runs in a separate job as the bot you configure via `bot_app_id` (Comfy: cloud-code-bot). `dry_run` reports what it would file without opening issues. Runs on a **daily base cron** with a runtime cadence gate: set repo Actions variable `GROOM_INTERVAL_DAYS` (default 7 = weekly) to retune how often a real run happens — weekly → every-3-days → daily — with no workflow-file edit; a tick within the interval no-ops before the finder (`workflow_dispatch` bypasses the interval gate, but the volume gate — when the caller leaves it on — still applies). The calling job must grant `contents: read` + `issues: write` + `pull-requests: read` + `actions: read` — the first three are declared by the `file` / `build_select` jobs (needed even with `bot_app_id` set), and the interval gate needs `actions: read` (reads run history for the last real run); GitHub rejects a shorter grant at startup. Requires `ANTHROPIC_API_KEY` (+ `BOT_APP_PRIVATE_KEY` when `bot_app_id` is set). **Opt-in auto-builder** (`builder: true`, BE-4003): the top `max_prs` (default 5) CONFIRMED, non-security findings become **review-gated PRs** (full CI + cursor-review, **never auto-merged**) instead of issues; a credential-free `build` job emits only a patch artifact and a separate `build_pr` job opens the PR as the bot, preserving the security boundary. The ledger's PR-state (open/merged/closed) stops a built finding being re-proposed. Requires `bot_app_id`. `max_prs` is typed **`string`**, not `number`, so a caller can forward its own `workflow_dispatch` input straight through (`max_prs: ${{ github.event.inputs.max_prs \|\| '1' }}`) and let an operator raise the ceiling for one manual run — no `fromJSON()` cast in the caller, and the parse/clamp (empty → default, non-numeric → 0 PRs + warning, never a failed run) happens once inside the reusable. |
 | [`agents-md-integrity.yml`](.github/workflows/agents-md-integrity.yml) | Enforces the Comfy `AGENTS.md` standard on the caller repo: a top-level `AGENTS.md` must exist and stay under a hard line ceiling (`max_lines`, default 200; warns over `warn_lines`, default 150), a `CLAUDE.md` (if present) must be a thin `@AGENTS.md` shim rather than a divergent copy, no legacy `.cursorrules` (gated `forbid_cursorrules`), every nested monorepo `AGENTS.md` needs a sibling `@AGENTS.md` shim and to be under the ceiling (gated `check_nested`), and `AGENTS.md` should have a CODEOWNERS DRI (`require_codeowners`, warn-only by default). Fails with a non-zero exit + GitHub annotations so it wires in as a required status check. The checker lives in [`.github/agents-md-integrity/`](.github/agents-md-integrity) (pin `workflows_ref` to the same ref as `uses:`); no secrets required. |

scripts/pr-risk/README.md

@@ -0,0 +1,130 @@
+# pr-risk — advisory PR risk grading (the shadow check)
+
+The scripts behind [`pr-risk.yml`](../../.github/workflows/pr-risk.yml). Every PR
+event is graded into a tier and gets ONE label:
+
+| tier | label (default) | meaning | eventual routing (later phases — nothing routes today) |
+|---|---|---|---|
+| R0 | `risk:R0` | inert — docs, tests, provably-shaped runbook output | auto-merge candidate |
+| R1 | `risk:R1` | contained — bounded, covered, revertable in one click | rubber-stamp |
+| R2 | `risk:R2` | standard — ordinary product code | normal review |
+| R3 | `risk:R3` | elevated — auth, billing, migrations, IaC, CI, deps, secrets | owner + e2e |
+| — | `risk:ungraded` | an input could not be read; deliberately NOT a tier | human review |
+
+**The label is the entire product.** Nothing is gated, blocked, routed, commented
+on, or merged. Humans glance at the label and either agree or disagree.
+Disagree by adding the `risk-dispute` label (never touched by the grader) plus a
+comment saying why — disputes are the pilot's calibration data.
+
+## How a grade is computed
+
+`grade = worst(path_floor, provenance, reversibility)` — three deterministic
+axes, worst wins, so no axis can move a PR into a safer lane than another axis
+put it. No LLM anywhere; the whole thing is `gh` + `jq` over the PR's API
+record.
+
+1. **Path floor** — [`risk-map.v0.json`](risk-map.v0.json): versioned path-glob
+   rules. The floor is the worst tier over every rule any changed path matches,
+   so a docs file can never cancel a migration in the same PR. Matching covers
+   every path the diff touches, **destination and origin** — a renamed file is
+   graded under its previous path too, so `git mv auth/x.go misc/x.go` cannot
+   walk a file out of the rule that guards it. In globs `**` crosses `/` and `*`
+   does not, and matching is whole-path anchored: a rule without a leading `**/`
+   matches root-level files ONLY.
+2. **Provenance** — what PROCESS produced the diff: `runbook` (a registered
+   producer in [`runbook-registry.v0.json`](runbook-registry.v0.json) whose
+   identity AND diff shape both assert), `agent-supervised`, `human`, or
+   `external` (fork / first-time contributor — R3, no exceptions, even when a
+   runbook shape matches). Identity is the server-attributed author login,
+   never the forgeable commit author string.
+3. **Reversibility** — mutates persistent state or deletes data → R3; **removes**
+   a file under a sensitive class → R3 (a delete, or a rename out of that class);
+   no green check rollup → R2; green but no test file touched → R1; green with
+   tests touched → R0. "Green" means at least one check actually CONCLUDED
+   success: a rollup of nothing but skipped/neutral answers "did tests covering
+   these lines run?" with nothing, so it cannot drop the axis below R2. What
+   counts as a test file is `reversibility.test_path_patterns` in the map (omit
+   the key and the grader falls back to a built-in regex that only knows the
+   Go/TS shapes).
+
+Anything unreadable grades `unknown` (labeled `risk:ungraded`), never a
+confident tier, and never "the axes that did resolve" — a PR whose file list we
+could not read is exactly the PR that might touch auth.
+
+Two CI-specific mechanics worth knowing:
+
+- **The grading run excludes itself from the check rollup it reads** (its own
+  check is always in-flight at grade time), and the job re-polls until the rest
+  of the rollup settles or `wait_for_checks_minutes` runs out — otherwise every
+  live grade would floor at R2 as an artifact of the measurement. Exclusion is
+  keyed on `github.run_id` (`--self-run-id`), and a **FAILING check is never
+  excluded**: self-exclusion may only ever hide our own pending run, never a red
+  one. Enroll pr-risk as its **own workflow** rather than a job inside an
+  existing CI workflow — a job sharing a run with the rest of CI excludes its
+  siblings too, and lands on the honest R2 floor instead of a full rollup.
+- **The label is applied with the plain `GITHUB_TOKEN`**, which cannot fire
+  `labeled` triggers — the shadow check is structurally unable to start a
+  workflow cascade. Later phases that WANT label-triggered routing switch to an
+  app token deliberately.
+
+## Per-repo overrides (read from the base ref)
+
+The shipped map and registry are deliberately generic. A consumer repo sharpens
+them by committing:
+
+- `.github/risk.json` — the repo's own path→tier map (same schema as
+  [`risk-map.v0.json`](risk-map.v0.json))
+- `.github/risk-runbooks.json` — the repo's own producer registry (same schema
+  as [`runbook-registry.v0.json`](runbook-registry.v0.json))
+
+Both are read from the PR's **base ref**, so a PR cannot edit the rules that
+judge it (editing them — or the grader — at all is R3 by the map's own first
+rule). A genuine 404 falls back to the shipped defaults; a present-but-invalid
+file fails the run loudly rather than silently grading generic, and so does any
+non-404 read failure (a 403 rate-limit or 5xx must not quietly demote the PR to
+the generic map, which would be a lower tier computed from an input nobody read).
+
+A map must MAP every provenance class (`runbook`, `agent-supervised`, `human`,
+`external`). Omitting one is refused at load time rather than filled in with a
+tier nobody chose — that is how a map that forgot `external` used to grade a fork
+the same as a teammate.
+
+Every graded record carries `map_version` + `registry_version`, so grades made
+under different maps stay comparable and a map revision can be replayed against
+accumulated records.
+
+## Relabeling (R0–R3 vs R1–R4 and friends)
+
+Tier SEMANTICS are fixed (R0 safest .. R3 riskiest, `unknown` separate)
+everywhere records are stored. The label TEXT is the caller's, via `label_map`:
+
+```yaml
+with:
+  label_map: "R0=risk:R1,R1=risk:R2,R2=risk:R3,R3=risk:R4,unknown=risk:ungraded"
+```
+
+Labels are created on first use, color-coded green → red (gray for ungraded).
+
+## Files
+
+- `grade-pr-risk.sh` — the grader. `--pr N --repo o/r` grades a live PR;
+  `--stdin` grades synthetic records (the no-network test surface). Extracted
+  from the fleet's offline corpus grader (BE-5507); the identity jq is inlined
+  from its collector (BE-5030) — keep the two in sync when either changes. The
+  changed-file list comes from REST `pulls/{n}/files`, not the GraphQL `files`
+  connection: GraphQL has no previous-path field (so renames are invisible) and
+  its connection capped the list at 100, which put exactly the PRs a risk grade
+  helps most in the ungraded lane. A read that comes back short of `changedFiles`
+  is still `unknown`.
+- `apply-risk-label.sh` — the one write. Owns exactly the five mapped labels:
+  removes stale ones, applies the computed one, touches nothing else.
+- `risk-map.v0.json` / `runbook-registry.v0.json` — the generic defaults.
+- `tests/` — hermetic suites (synthetic records + a stubbed `gh`); run via
+  [`test-pr-risk.yml`](../../.github/workflows/test-pr-risk.yml).
+
+## What is deliberately NOT here
+
+No auto-merge, no routing, no required check, no PR comment, no LLM judgement,
+no linked-ticket requirement. Those are later rungs of the ladder and each one
+is its own explicit switch — this workflow exists to accumulate the
+agree/disagree evidence that decides whether any of them turn on.