Skip to content

Added the visual regression report to the GitHub Actions run summary. #2865

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AlexSkrypnyk merged 2 commits into from
Jul 29, 2026
Merged

Added the visual regression report to the GitHub Actions run summary. #2865

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a59f1c3
Added the visual regression report to the workflow run summary.
AlexSkrypnyk Jul 29, 2026
79b683e
Updated snapshots.
AlexSkrypnyk Jul 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 36 additions & 10 deletions .github/workflows/test-vr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ jobs:
shared_url: ${{ steps.result.outputs.shared_url }}
pages_changed: ${{ steps.result.outputs.pages_changed }}
pages_total: ${{ steps.result.outputs.pages_total }}
summary: ${{ steps.summary.outputs.summary }}

steps:
- name: Resolve PR number from target URL
Expand Down Expand Up @@ -253,7 +254,7 @@ jobs:
result_json="$(/tmp/diffy.phar diff:get-result "${diff_id}" --format=json)"

shared_url="$(printf '%s' "${result_json}" | jq -r '.diffSharedUrl // empty')"
pages_changed="$(printf '%s' "${result_json}" | jq -r '[.diffs | to_entries[] | select(.value | to_entries | map(.value.percentageChanges // 0) | add > 0)] | length')"
pages_changed="$(printf '%s' "${result_json}" | jq -r '[.diffs[] | select([.[].percentageChanges | tonumber? // 0] | add > 0)] | length')"
pages_total="$(printf '%s' "${result_json}" | jq -r '.diffs | length')"

{
Expand All @@ -268,6 +269,39 @@ jobs:
env:
STEPS_COMPARE_OUTPUTS_DIFF_ID: ${{ steps.compare.outputs.diff_id }}

- name: Write run summary
id: summary
if: steps.gate.outputs.skipped != 'true'
run: |
set -euo pipefail

summary="$(printf '%s\n' \
"### Visual regression report" \
"" \
"**${PAGES_CHANGED} of ${PAGES_TOTAL}** pages changed, **${CHANGES_PERCENT}%** overall difference." \
"" \
"| Setting | Value |" \
"| --- | --- |" \
"| Source environment | \`${SOURCE_ENV}\` |" \
"| Target environment | \`${TARGET_URL}\` |" \
"| Diff ID | \`${DIFF_ID}\` |" \
"" \
"[View full Diffy report](${SHARED_URL})")"

printf '%s\n' "${summary}" >> "$GITHUB_STEP_SUMMARY"

{
echo "summary<<VR_SUMMARY_EOF"
printf '%s\n' "${summary}"
echo "VR_SUMMARY_EOF"
} >> "$GITHUB_OUTPUT"
env:
CHANGES_PERCENT: ${{ steps.result.outputs.changes_percent }}
SHARED_URL: ${{ steps.result.outputs.shared_url }}
PAGES_CHANGED: ${{ steps.result.outputs.pages_changed }}
PAGES_TOTAL: ${{ steps.result.outputs.pages_total }}
DIFF_ID: ${{ steps.compare.outputs.diff_id }}

vr-report:
runs-on: ubuntu-latest
needs: vr-compare
Expand All @@ -282,13 +316,5 @@ jobs:
with:
number: ${{ needs.vr-compare.outputs.pr_number }}
header: vr-diffy
message: |
### Visual regression report

- **Pages changed**: ${{ needs.vr-compare.outputs.pages_changed }} of ${{ needs.vr-compare.outputs.pages_total }}
- **Overall difference**: ${{ needs.vr-compare.outputs.changes_percent }}%
- **Target environment**: ${{ env.TARGET_URL }}
- **Source environment**: ${{ env.SOURCE_ENV }}

[View full Diffy report](${{ needs.vr-compare.outputs.shared_url }})
message: ${{ needs.vr-compare.outputs.summary }}
hide_and_recreate: true
16 changes: 11 additions & 5 deletions .vortex/docs/content/development/visual-regression.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,9 @@ GitHub branch protection rules.
│ vr-compare: parse PR from URL, gate │
│ │ │
│ ▼ │
│ Report in workflow run summary │
│ │ │
│ ▼ │
│ vr-report │
│ │ │
│ ▼ │
Expand Down Expand Up @@ -179,7 +182,7 @@ them do.
│ vr-compare job │
│ │ │
│ ▼ │
│ vr-report job │
│ Report in workflow run summary │
│ │
└────────────────────┬────────────────────────┘
│
Expand All @@ -192,8 +195,8 @@ them do.
```

Use this entry point for ad-hoc comparisons against a known environment
URL. No PR is involved, so the result is visible only in the Diffy UI
(and in the workflow run log).
URL. No PR is involved, so there is no comment to post - the result is
shown in the workflow run summary and in the Diffy UI.

## Limiting which branches dispatch

Expand Down Expand Up @@ -293,11 +296,14 @@ entry point.
`VR_DIFFY_POLL_INTERVAL` seconds.
5. Fetches the diff result and exposes the diff ID, PR number, change
percentage, page counts, and shared report URL as job outputs.
6. Renders the report into the workflow run summary, so the result is
readable in the GitHub Actions UI without opening the job log.

`vr-report`:

1. Posts a sticky comment on the PR with the summary and a link to the
Diffy report. Re-deploys edit the same comment rather than stacking.
1. Posts the same report as a sticky comment on the PR, with a link to
the Diffy report. Re-deploys edit the same comment rather than
stacking.

## Making it blocking

Expand Down
46 changes: 36 additions & 10 deletions ...er/tests/Fixtures/handler_process/visual_regression_enabled/.github/workflows/test-vr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ jobs:
shared_url: ${{ steps.result.outputs.shared_url }}
pages_changed: ${{ steps.result.outputs.pages_changed }}
pages_total: ${{ steps.result.outputs.pages_total }}
summary: ${{ steps.summary.outputs.summary }}

steps:
- name: Resolve PR number from target URL
Expand Down Expand Up @@ -253,7 +254,7 @@ jobs:
result_json="$(/tmp/diffy.phar diff:get-result "${diff_id}" --format=json)"

shared_url="$(printf '%s' "${result_json}" | jq -r '.diffSharedUrl // empty')"
pages_changed="$(printf '%s' "${result_json}" | jq -r '[.diffs | to_entries[] | select(.value | to_entries | map(.value.percentageChanges // 0) | add > 0)] | length')"
pages_changed="$(printf '%s' "${result_json}" | jq -r '[.diffs[] | select([.[].percentageChanges | tonumber? // 0] | add > 0)] | length')"
pages_total="$(printf '%s' "${result_json}" | jq -r '.diffs | length')"

{
Expand All @@ -268,6 +269,39 @@ jobs:
env:
STEPS_COMPARE_OUTPUTS_DIFF_ID: ${{ steps.compare.outputs.diff_id }}

- name: Write run summary
id: summary
if: steps.gate.outputs.skipped != 'true'
run: |
set -euo pipefail

summary="$(printf '%s\n' \
"### Visual regression report" \
"" \
"**${PAGES_CHANGED} of ${PAGES_TOTAL}** pages changed, **${CHANGES_PERCENT}%** overall difference." \
"" \
"| Setting | Value |" \
"| --- | --- |" \
"| Source environment | \`${SOURCE_ENV}\` |" \
"| Target environment | \`${TARGET_URL}\` |" \
"| Diff ID | \`${DIFF_ID}\` |" \
"" \
"[View full Diffy report](${SHARED_URL})")"

printf '%s\n' "${summary}" >> "$GITHUB_STEP_SUMMARY"

{
echo "summary<<VR_SUMMARY_EOF"
printf '%s\n' "${summary}"
echo "VR_SUMMARY_EOF"
} >> "$GITHUB_OUTPUT"
env:
CHANGES_PERCENT: ${{ steps.result.outputs.changes_percent }}
SHARED_URL: ${{ steps.result.outputs.shared_url }}
PAGES_CHANGED: ${{ steps.result.outputs.pages_changed }}
PAGES_TOTAL: ${{ steps.result.outputs.pages_total }}
DIFF_ID: ${{ steps.compare.outputs.diff_id }}

vr-report:
runs-on: ubuntu-latest
needs: vr-compare
Expand All @@ -282,13 +316,5 @@ jobs:
with:
number: ${{ needs.vr-compare.outputs.pr_number }}
header: vr-diffy
message: |
### Visual regression report

- **Pages changed**: ${{ needs.vr-compare.outputs.pages_changed }} of ${{ needs.vr-compare.outputs.pages_total }}
- **Overall difference**: ${{ needs.vr-compare.outputs.changes_percent }}%
- **Target environment**: ${{ env.TARGET_URL }}
- **Source environment**: ${{ env.SOURCE_ENV }}

[View full Diffy report](${{ needs.vr-compare.outputs.shared_url }})
message: ${{ needs.vr-compare.outputs.summary }}
hide_and_recreate: true