-
Notifications
You must be signed in to change notification settings - Fork 0
feat(pr-risk): gate grading behind an enabled switch, off by default
#115
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -142,6 +142,11 @@ name: PR Risk Grade (reusable) | |
| # uses: Comfy-Org/github-workflows/.github/workflows/pr-risk.yml@<sha> # v1 | ||
| # with: | ||
| # workflows_ref: <same sha> | ||
| # # OFF BY DEFAULT. Enrolling and switching on are two decisions: land the caller, get | ||
| # # it reviewed, then start grading. Either pin it on here, or leave this out and set | ||
| # # the repo variable RISK_CONFIG to {"enabled": true} — the variable outranks this | ||
| # # input in BOTH directions, so {"enabled": false} is also a kill switch needing no PR. | ||
| # enabled: true | ||
| # # Both are empty on a `pull_request` run (`inputs` is empty there), which is exactly | ||
| # # the no-input event path — so ONE caller shape serves both event and dispatch and | ||
| # # there is nothing to keep in sync between two jobs. | ||
|
|
@@ -253,12 +258,69 @@ on: | |
| is not a chain. | ||
| type: string | ||
| required: true | ||
| enabled: | ||
| description: >- | ||
| Whether to grade at all. DEFAULTS TO FALSE: enrolling this workflow and | ||
| switching it on are two separate decisions, so a repo can land the | ||
| caller, get it reviewed, and start grading later — or stop grading | ||
| without reverting anything. When it resolves false the `grade` job does | ||
| not run: nothing is read, nothing is labelled, and no existing `risk:*` | ||
| label is touched (a disabled run leaves whatever the last enabled one | ||
| left, it does not clean up). | ||
|
|
||
|
|
||
| `vars.RISK_CONFIG` on the CALLING repo OVERRIDES this, in both | ||
| directions — `{"enabled": true}` switches a caller on with no `with:` | ||
| change, and `{"enabled": false}` is a kill switch that needs no PR at | ||
| all. That works because the `vars` context inside a reusable resolves | ||
| against the caller's repository (the same mechanism groom.yml uses for | ||
| `vars.GROOM_CONFIG`). The variable is the OPERATIONAL lever; this input | ||
| is the REVIEWED default the variable falls back to when it is absent, | ||
| empty, not an object, or carries no boolean `enabled` — a malformed | ||
| variable must never be the reason a repo silently stops grading, so it | ||
| degrades to this value and says so in an annotation. | ||
| type: boolean | ||
| required: false | ||
| default: false | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| # Resolve enablement before anything is read. Its own job rather than a first step of `grade` | ||
| # so a disabled repo pays one bare runner instead of booting the grading job to immediately | ||
| # no-op, and so the decision — and the reason for it — is visible in the run graph rather than | ||
| # buried in a step log. It holds NO token scopes: it reads only its inputs and the caller's | ||
| # variable, and must never be the thing that touches a PR. | ||
| gate: | ||
| name: Resolve enablement | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 5 | ||
| permissions: {} | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟠 High — The |
||
| outputs: | ||
| enabled: ${{ steps.resolve.outputs.enabled }} | ||
| steps: | ||
| # The tool checkout is the same pinned-ref load the grade job does: the resolver is this | ||
| # repo's code at `workflows_ref`, never the graded PR's. | ||
| - name: Load pr-risk tool | ||
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
| with: | ||
| repository: Comfy-Org/github-workflows | ||
| ref: ${{ inputs.workflows_ref }} | ||
| path: _pr_risk_tool | ||
| persist-credentials: false | ||
| - name: Resolve enabled | ||
| id: resolve | ||
| env: | ||
| INPUT_ENABLED: ${{ inputs.enabled }} | ||
| # Empty when the caller repo has no such variable — which is the common case and is | ||
| # NOT an error; it just means the reviewed input decides. | ||
| RISK_CONFIG: ${{ vars.RISK_CONFIG }} | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟢 Low — |
||
| run: bash _pr_risk_tool/scripts/pr-risk/resolve-enabled.sh | ||
|
|
||
| grade: | ||
| needs: gate | ||
| if: needs.gate.outputs.enabled == 'true' | ||
| name: Grade PR risk | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 30 | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| #!/usr/bin/env bash | ||
| # resolve-enabled.sh — decide whether pr-risk grades this run, and say who decided. | ||
| # | ||
| # Two layers, and the VARIABLE outranks the INPUT in both directions: | ||
| # | ||
| # vars.RISK_CONFIG `{"enabled": true}` switches a caller on with no `with:` change; | ||
| # (operational) `{"enabled": false}` is a kill switch that needs no PR at all. The `vars` | ||
| # context inside a reusable resolves against the CALLING repository, which | ||
| # is what makes this reachable from here (same mechanism groom.yml uses for | ||
| # vars.GROOM_CONFIG). | ||
| # | ||
| # inputs.enabled the reviewed default, pinned in the caller's workflow file. Used whenever | ||
| # (reviewed) the variable does not say — absent, blank, not an object, or carrying an | ||
| # `enabled` that is not a boolean. | ||
| # | ||
| # DEGRADE TOWARD THE REVIEWED VALUE, NEVER TOWARD OFF. A malformed variable must not be the | ||
| # reason a repo silently stops grading, so a value this cannot read is announced and discarded | ||
| # rather than treated as `false`. "The variable is broken" and "the operator switched it off" | ||
| # are different states and must not look alike. | ||
| # | ||
| # A well-formed object with no `enabled` key is silent, not a warning: `enabled` is the only key | ||
| # today, and warning on every run would train operators to ignore the annotation that flags a | ||
| # real typo — and would start lying the moment a second key lands here. | ||
| # | ||
| # Env in: INPUT_ENABLED (the reusable's `enabled` input), RISK_CONFIG (the caller's variable). | ||
| # Env out: GITHUB_OUTPUT gets `enabled=true|false`; GITHUB_STEP_SUMMARY explains a disabled run. | ||
| # Both are optional so the script is runnable — and testable — outside Actions. | ||
| set -uo pipefail | ||
|
|
||
| warn() { printf '::warning::%s\n' "$1" >&2; } | ||
|
|
||
| reviewed="${INPUT_ENABLED:-false}" | ||
| # Anything that is not exactly `true` is off. The input arrives as a workflow-call boolean, so | ||
| # this is belt-and-braces against a caller forwarding a string expression into it. | ||
| [ "$reviewed" = true ] || reviewed=false | ||
|
|
||
| enabled="$reviewed" | ||
| decided_by="the caller's reviewed \`enabled:\` input" | ||
|
|
||
| raw="${RISK_CONFIG:-}" | ||
| if [ -n "${raw//[[:space:]]/}" ]; then | ||
| if ! jq -e 'type == "object"' >/dev/null 2>&1 <<<"$raw"; then | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium — |
||
| warn "vars.RISK_CONFIG is set but is not a JSON object — ignoring it and using the reviewed input (enabled=${reviewed}). Expected {\"enabled\": true}." | ||
| elif jq -e 'has("enabled")' >/dev/null 2>&1 <<<"$raw"; then | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟢 Low — |
||
| # Read the value as TEXT and test the text. `jq -e` is not usable here: its exit code | ||
| # reflects the TRUTHINESS of the output, so a perfectly valid `{"enabled": false}` exits 1 | ||
| # and would be mistaken for a parse failure — silently disarming the kill switch, which is | ||
| # the half of this lever that has to work when something is going wrong. | ||
| val="$(jq -r '.enabled | if type == "boolean" then tostring else "" end' <<<"$raw" 2>/dev/null)" | ||
| if [ "$val" = true ] || [ "$val" = false ]; then | ||
| enabled="$val" | ||
| # shellcheck disable=SC2016 # markdown backticks for the log line, not a substitution | ||
| decided_by='`vars.RISK_CONFIG`' | ||
| else | ||
| warn "vars.RISK_CONFIG has an \`enabled\` key but it is not a boolean — ignoring it and using the reviewed input (enabled=${reviewed}). Use true/false, not \"true\"." | ||
| fi | ||
| fi | ||
| fi | ||
|
|
||
| [ -z "${GITHUB_OUTPUT:-}" ] || echo "enabled=$enabled" >> "$GITHUB_OUTPUT" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium — The append to |
||
| echo "pr-risk enabled=$enabled (decided by ${decided_by})" | ||
|
|
||
| if [ "$enabled" != true ] && [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then | ||
| # shellcheck disable=SC2016 # the body is literal markdown; backticks are code spans | ||
| { | ||
| echo "## PR risk grading is DISABLED for this repository" | ||
| echo | ||
| echo "No grade was computed, and no \`risk:*\` label was added, changed, or removed —" | ||
| echo "any label already on this PR is left exactly as the last enabled run left it." | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ⚪ Nit — The summary says "any label already on this PR is left exactly as the last enabled run left it", but the gate also runs on the |
||
| echo | ||
| echo "Decided by ${decided_by}. To switch grading on, either set the repository variable" | ||
| echo '`RISK_CONFIG` to `{"enabled": true}` (takes effect on the next run, no PR needed), or' | ||
| echo 'pass `enabled: true` in the caller'"'"'s `with:` block.' | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium — The disabled-run summary always offers |
||
| } >> "$GITHUB_STEP_SUMMARY" | ||
| fi | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 Medium — Defaulting
enabledtofalsesilently disables grading for every already-enrolled caller the next time it moves its SHA pin, and a disabled run deliberately leaves the previousrisk:*label in place — so those PRs keep showing a stale grade indistinguishable from a current one. There is nobump-pr-risk-callersfleet, so pins move by hand with no coordinated migration; this is a behavior break riding the in-placev1tag. Raised by 2 of 8 reviewers (claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case).