Skip to content

[#2863] Added opt-in for visual regression on non-PR deployments. #2864

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AlexSkrypnyk merged 10 commits into from
Jul 29, 2026
Merged

[#2863] Added opt-in for visual regression on non-PR deployments. #2864

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
5757c4b
[#2863] Added opt-in for visual regression on non-PR deployments.
AlexSkrypnyk Jul 29, 2026
0aeea61
Updated snapshots.
AlexSkrypnyk Jul 29, 2026
28addc8
[#2863] Trimmed all surrounding whitespace from branch patterns.
AlexSkrypnyk Jul 29, 2026
dd8b8c6
[#2863] Documented where runs without a pull request report their res…
AlexSkrypnyk Jul 29, 2026
2c1b11f
Updated snapshots.
AlexSkrypnyk Jul 29, 2026
5c04b62
[#2863] Placed both gate decisions at their evaluation point in the f…
AlexSkrypnyk Jul 29, 2026
5c4118f
[#2863] Extracted the gate decision into named 'proceed' and 'skip' h…
AlexSkrypnyk Jul 29, 2026
7c08efd
Updated snapshots.
AlexSkrypnyk Jul 29, 2026
02b37d3
[#2863] Renamed the branch gate variables and compacted the gate step.
AlexSkrypnyk Jul 29, 2026
eea9651
Updated snapshots.
AlexSkrypnyk Jul 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 47 additions & 50 deletions .github/workflows/test-vr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# GitHub Actions visual regression testing workflow.
#
# Runs a Diffy visual regression comparison after a deployment lands.
# Triggered automatically by 'notify-diffy' (via repository_dispatch)
# when the deployed PR has the configured label, or manually via
# 'workflow_dispatch' against any URL.
# Triggered automatically by 'notify-diffy' (via repository_dispatch) when
# the deployed PR has the configured label or the deployed branch is opted
# in, or manually via 'workflow_dispatch' against any URL.
name: Test - Visual regression

on:
Expand Down Expand Up @@ -43,10 +43,12 @@ env:
DIFFY_CLI_VERSION: ${{ vars.VR_DIFFY_CLI_VERSION || '0.1.53' }}
DIFFY_MAX_WAIT: ${{ vars.VR_DIFFY_MAX_WAIT || '2700' }}
DIFFY_PR_LABEL: ${{ vars.VR_DIFFY_PR_LABEL || 'VR' }}
DIFFY_AUTO_BRANCHES: ${{ vars.VR_DIFFY_AUTO_BRANCHES || 'deps/*' }}
DIFFY_PR_SKIP_BRANCHES: ${{ vars.VR_DIFFY_PR_SKIP_BRANCHES || 'deps/*' }}
DIFFY_BRANCHES: ${{ vars.VR_DIFFY_BRANCHES }}
DIFFY_POLL_INTERVAL: ${{ vars.VR_DIFFY_POLL_INTERVAL || '30' }}
SOURCE_ENV: ${{ github.event.client_payload.source_env || inputs.source_env || 'production' }}
TARGET_URL: ${{ github.event.client_payload.target_url || inputs.target_url }}
BRANCH: ${{ github.event.client_payload.branch }}
LABEL: ${{ github.event.client_payload.label || inputs.label || 'manual' }}

jobs:
Expand Down Expand Up @@ -74,18 +76,18 @@ jobs:

# Hosting providers expose PR environments via URLs that contain
# a 'pr-<number>' segment (e.g. 'app.pr-123.example.lagoon.cloud').
# Extract the number from the URL; if no match, this is not a PR
# deployment and visual regression should not run.
# Extract the number from the URL; a URL without one is not a PR
# deployment and is gated on the deployed branch instead.
pr_number="$(printf '%s' "${TARGET_URL}" | sed -n 's|.*pr-\([0-9]\{1,\}\).*|\1|p')"

if [ -n "${pr_number}" ]; then
echo "PR #${pr_number} resolved from target URL ${TARGET_URL}."
else
echo "::notice::No PR pattern found in target URL ${TARGET_URL}. Visual regression will not run."
echo "No PR pattern found in target URL ${TARGET_URL}."
fi
echo "pr_number=${pr_number}" >> "$GITHUB_OUTPUT"

- name: Gate run on PR label (or skip for non-PR contexts on dispatch)
- name: Gate run on PR label or opted-in branch
id: gate
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Expand All @@ -95,55 +97,50 @@ jobs:
run: |
set -euo pipefail

# Manual workflow_dispatch always proceeds - the operator
# explicitly asked for a comparison.
if [ "${GITHUB_EVENT_NAME}" = "workflow_dispatch" ]; then
echo "Manual dispatch, proceeding."
echo "skipped=false" >> "$GITHUB_OUTPUT"
exit 0
fi
# Matches a git ref against a comma-separated glob list. Splitting
# on whitespace as well as commas absorbs padding around each
# pattern.
ref_matches() {
local ref="${1}" pattern patterns
[ -n "${ref}" ] || return 1
[ -n "${2}" ] || return 1
IFS=$', \t' read -ra patterns <<<"${2}"
for pattern in "${patterns[@]}"; do
# shellcheck disable=SC2254
case "${ref}" in ${pattern}) return 0 ;; esac
done
return 1
}

# repository_dispatch with no PR resolved - nothing to gate on.
# Record the gate decision and end the step.
proceed() { echo "${1}"; echo "skipped=false" >> "$GITHUB_OUTPUT"; exit 0; }
skip() { echo "::notice::${1}"; echo "skipped=true" >> "$GITHUB_OUTPUT"; exit 0; }

# The operator explicitly asked for a comparison.
[ "${GITHUB_EVENT_NAME}" = "workflow_dispatch" ] && proceed "Manual dispatch."

# No PR resolved - the deployment targets a permanent environment
# such as a release branch, 'develop', 'dev' or 'stage'. Those run
# only when the deployed branch is explicitly opted in.
if [ -z "${PR_NUMBER}" ]; then
echo "::notice::No PR associated with this deployment, skipping visual regression."
echo "skipped=true" >> "$GITHUB_OUTPUT"
exit 0
ref_matches "${BRANCH}" "${DIFFY_BRANCHES}" && proceed "Deployed branch '${BRANCH}' matches VR_DIFFY_BRANCHES."
skip "No PR associated with this deployment and branch '${BRANCH}' does not match VR_DIFFY_BRANCHES, skipping visual regression."
fi

# PR resolved - fetch labels and head branch in one call.
pr_data="$(gh pr view "${PR_NUMBER}" --repo "${GITHUB_REPOSITORY}" --json labels,headRefName)"
# A failed lookup means the 'pr-<number>' segment in the URL did
# not come from a pull request in this repository.
pr_data="$(gh pr view "${PR_NUMBER}" --repo "${GITHUB_REPOSITORY}" --json labels,headRefName)" || skip "PR #${PR_NUMBER} could not be read, skipping visual regression."

head_ref="$(printf '%s' "${pr_data}" | jq -r '.headRefName')"
labels_lower="$(printf '%s' "${pr_data}" | jq -r '.labels[].name' | tr '[:upper:]' '[:lower:]')"
labels="$(printf '%s' "${pr_data}" | jq -r '.labels[].name')"

# Auto-run bypass: PRs from configured branches (e.g. Renovate's
# `deps/*`) skip the label check.
if [ -n "${DIFFY_AUTO_BRANCHES}" ]; then
IFS=',' read -ra patterns <<<"${DIFFY_AUTO_BRANCHES}"
for pattern in "${patterns[@]}"; do
# Trim surrounding whitespace.
pattern="${pattern# }"
pattern="${pattern% }"
[ -z "${pattern}" ] && continue
# shellcheck disable=SC2254
case "${head_ref}" in
${pattern})
echo "PR #${PR_NUMBER} head branch '${head_ref}' matches auto-run pattern '${pattern}', proceeding."
echo "skipped=false" >> "$GITHUB_OUTPUT"
exit 0
;;
esac
done
fi
# PRs from configured branches (e.g. Renovate's `deps/*`) skip the
# label check.
ref_matches "${head_ref}" "${DIFFY_PR_SKIP_BRANCHES}" && proceed "PR #${PR_NUMBER} head branch '${head_ref}' matches VR_DIFFY_PR_SKIP_BRANCHES."

# Label check (case-insensitive).
needle="$(printf '%s' "${DIFFY_PR_LABEL}" | tr '[:upper:]' '[:lower:]')"
if echo "${labels_lower}" | grep -qx "${needle}"; then
echo "PR #${PR_NUMBER} has the '${DIFFY_PR_LABEL}' label, proceeding."
echo "skipped=false" >> "$GITHUB_OUTPUT"
else
echo "::notice::PR #${PR_NUMBER} does not have the '${DIFFY_PR_LABEL}' label and head branch '${head_ref}' does not match VR_DIFFY_AUTO_BRANCHES, skipping visual regression."
echo "skipped=true" >> "$GITHUB_OUTPUT"
fi
echo "${labels}" | grep -qixF "${DIFFY_PR_LABEL}" && proceed "PR #${PR_NUMBER} has the '${DIFFY_PR_LABEL}' label."

skip "PR #${PR_NUMBER} does not have the '${DIFFY_PR_LABEL}' label and head branch '${head_ref}' does not match VR_DIFFY_PR_SKIP_BRANCHES, skipping visual regression."

- name: Validate target URL
if: steps.gate.outputs.skipped != 'true'
Expand Down
98 changes: 78 additions & 20 deletions .vortex/docs/content/development/visual-regression.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ sidebar_label: Visual regression
**Vortex** ships an optional visual regression workflow powered by
[Diffy](../tools/diffy.mdx). It compares the just-deployed environment
against a baseline (typically `production`) and posts the result back to
the related pull request.
the related pull request, or to the workflow run summary when the
deployment has no pull request.

## Account setup

Expand Down Expand Up @@ -74,7 +75,8 @@ the site. In Cloudflare, add an IP Access Rule for that address with the
| `VR_DIFFY_MAX_WAIT` | `2700` | Maximum seconds to wait for a comparison to complete (45 minutes) |
| `VR_DIFFY_POLL_INTERVAL` | `30` | Polling interval in seconds |
| `VR_DIFFY_PR_LABEL` | `VR` | PR label that opts a deployment into visual regression. Case-insensitive |
| `VR_DIFFY_AUTO_BRANCHES` | `deps/*` | Comma-separated glob list of PR head branches that bypass the label gate (matches Renovate's `branchPrefix`). Set to empty to require the label on every PR. |
| `VR_DIFFY_PR_SKIP_BRANCHES` | `deps/*` | Comma-separated glob list of PR head branches that skip the label gate (matches Renovate's `branchPrefix`). Set to empty to require the label on every PR. |
| `VR_DIFFY_BRANCHES` | (empty) | Comma-separated glob list of deployed branches that run visual regression when the deployment has no pull request (for example `release/*`). Empty means such deployments never run. |
| `VR_DIFFY_SOURCE_ENV` | `production` | Default Diffy source environment for comparisons |

Add these under *Settings > Secrets and variables > Actions > Repository
Expand Down Expand Up @@ -106,7 +108,7 @@ GitHub branch protection rules.
```text
┌─ GitHub ────────────────────────────────────┐
│ │
│ PR opened with `VR` label │
│ PR opened or branch pushed │
│ │ │
│ ▼ │
│ build-test-deploy.yml │
Expand Down Expand Up @@ -140,11 +142,20 @@ GitHub branch protection rules.
│ ▼ │
│ vr-compare: parse PR from URL, gate │
│ │ │
│ ┌────────┴────────┐ │
│ ▼ ▼ │
│ PR deployment no PR resolved │
│ │ │ │
│ ▼ ▼ │
│ VR label or branch in │
│ PR skip list branch list │
│ │ │ │
│ └────────┬────────┘ │
│ ▼ │
│ Report in workflow run summary │
│ │ │
│ ▼ │
│ vr-report │
│ vr-report (PR only) │
│ │ │
│ ▼ │
│ PR comment + Diffy report link │
Expand All @@ -158,8 +169,9 @@ commit SHA**. The workflow itself extracts the PR number from the
deployed environment URL by matching the `pr-<number>` segment (e.g.
`https://pr-123.example.com/` resolves to PR #123) and verifies the
`VR` label is present (case-insensitive). If the target URL has no
`pr-<number>` segment, the deployment is not a PR environment and the
run is skipped.
`pr-<number>` segment, the deployment is not a PR environment and the run
is skipped unless the deployed branch is opted in - see [Release and other
non-PR deployments](#release-and-other-non-pr-deployments).

This means the PR lookup works uniformly across hosting providers - the
host only needs to expose the deployed environment URL, which all of
Expand Down Expand Up @@ -218,8 +230,8 @@ be the sole gate.
PRs raised by Renovate (or any other automated dependency-update bot)
typically do not carry the `VR` label - they carry their own bot label
(e.g. `Dependencies`). To still run visual regression on them, the
workflow consults `VR_DIFFY_AUTO_BRANCHES`: a comma-separated glob list
of PR head branches that bypass the `VR` label gate.
workflow consults `VR_DIFFY_PR_SKIP_BRANCHES`: a comma-separated glob
list of PR head branches that skip the `VR` label gate.

The default value is `deps/*`, matching **Vortex**'s Renovate `branchPrefix`
configuration. Other common values:
Expand All @@ -231,13 +243,12 @@ configuration. Other common values:
| Dependabot | `dependabot/` | `dependabot/*` |

Multiple patterns can be combined with commas:
`VR_DIFFY_AUTO_BRANCHES=deps/*,dependabot/*`. Set the variable to empty
to disable the bypass entirely (every PR, including bot PRs, then needs
the label).
`VR_DIFFY_PR_SKIP_BRANCHES=deps/*,dependabot/*`. Set the variable to
empty to require the label on every PR, including bot PRs.

### Default behavior out of the box

With `VR_DIFFY_AUTO_BRANCHES=deps/*` (default) and
With `VR_DIFFY_PR_SKIP_BRANCHES=deps/*` (default) and
`VR_DIFFY_PR_LABEL=VR` (default):

| PR head branch | Has `VR` label? | Runs? |
Expand All @@ -248,8 +259,47 @@ With `VR_DIFFY_AUTO_BRANCHES=deps/*` (default) and
| `deps/drupal-core-11.2` | yes | yes (matches `deps/*`, label irrelevant) |

Consumers using Dependabot just append:
`VR_DIFFY_AUTO_BRANCHES=deps/*,dependabot/*`. Consumers who want the
label as the only gate (no auto-bypass) set the variable to empty.
`VR_DIFFY_PR_SKIP_BRANCHES=deps/*,dependabot/*`. Consumers who want the
label as the only gate set the variable to empty.

## Release and other non-PR deployments

A deployment that is not a pull request environment - a release branch, an
integration branch such as `develop`, or a permanent `dev`/`stage`
environment - has no pull request to carry the `VR` label, so it is skipped
by default.

Set `VR_DIFFY_BRANCHES` to a comma-separated glob list of deployed branch
names to compare them automatically:

```ini
VR_DIFFY_BRANCHES=release/*
```

| Deployment | `VR_DIFFY_BRANCHES` | Runs? |
|---|---|---|
| PR environment (`pr-123` in the URL) | any value | gated by the `VR` label and `VR_DIFFY_PR_SKIP_BRANCHES` as usual |
| Branch `release/1.2.3` | (empty) | no |
| Branch `release/1.2.3` | `release/*` | **yes** |
| Branch `develop` | `release/*` | no |
| Branch `develop` | `release/*,develop` | **yes** |
| Any branch | `*` | **yes** (every non-PR deployment) |

The deployed branch travels in the dispatch payload, so no GitHub API call
is needed to resolve it.

:::note

Patterns match the **git branch name**, not the hostname. Hosting providers
sanitize branch names for URLs - Lagoon deploys `release/26.7.5` to a host
containing `release-26-7-5` - but the gate compares against the original
`release/26.7.5`.

:::

There is no pull request to comment on for these runs, so the result
appears in the workflow run summary and in the Diffy UI instead of as a PR
comment.

## How the PR is resolved

Expand All @@ -272,7 +322,8 @@ This means:

If the target URL has no `pr-<number>` segment (for example, a deploy to
a named environment like `dev`/`test`/`prod`), the workflow treats it as
"not a PR deployment" and exits without running.
"not a PR deployment" and runs only when the deployed branch matches
`VR_DIFFY_BRANCHES`.

## Missed-window behavior

Expand All @@ -288,8 +339,9 @@ entry point.

`vr-compare`:

1. Resolves the PR number from the target URL and gates on the label
(or auto-branch pattern for dependency PRs).
1. Resolves the PR number from the target URL and gates on the label (or
the auto-branch pattern for dependency PRs, or the deployed-branch
opt-in for non-PR deployments).
2. Installs the pinned Diffy CLI.
3. Calls `diffy project:compare` with the target URL and a label.
4. Polls the comparison status, printing progress to the job log every
Expand All @@ -303,7 +355,8 @@ entry point.

1. Posts the same report as a sticky comment on the PR, with a link to
the Diffy report. Re-deploys edit the same comment rather than
stacking.
stacking. The job runs only when a PR was resolved - manual runs and
opted-in branch deployments report through the run summary alone.

## Making it blocking

Expand All @@ -323,10 +376,15 @@ be posted, which is not a merge-blocker.
## Costs and quotas

Diffy bills per screenshot set. **Vortex**'s default gating (`VR` label on
PRs only, plus the `deps/*` auto-branch list) keeps the run rate low.
Adjust `VR_DIFFY_PR_LABEL`, `VR_DIFFY_AUTO_BRANCHES`, and
PRs only, plus the `deps/*` auto-branch list, with non-PR deployments off
entirely) keeps the run rate low. Adjust `VR_DIFFY_PR_LABEL`,
`VR_DIFFY_PR_SKIP_BRANCHES`, `VR_DIFFY_BRANCHES`, and
`VORTEX_NOTIFY_DIFFY_BRANCHES` to suit the team's review rhythm.

Scope `VR_DIFFY_BRANCHES` as narrowly as the workflow allows: `*`
compares on every deployment to every permanent environment, which on a
busy integration branch consumes quota quickly.

## Disabling

Remove `diffy` from `VORTEX_NOTIFY_CHANNELS` and delete the
Expand Down
Loading