Skip to content

Added the visual regression report to the GitHub Actions run summary. - #2865

Merged
AlexSkrypnyk merged 2 commits into
mainfrom
feature/vr-run-summary
Jul 29, 2026
Merged

Added the visual regression report to the GitHub Actions run summary.#2865
AlexSkrypnyk merged 2 commits into
mainfrom
feature/vr-run-summary

Conversation

@AlexSkrypnyk

@AlexSkrypnyk AlexSkrypnyk commented Jul 29, 2026

Copy link
Copy Markdown
Member

Summary

The visual regression result was readable only in the vr-compare job log and, for PR runs, in a sticky PR comment that vr-report assembled itself from four separate job outputs. This adds a Write run summary step to vr-compare that renders the report once into $GITHUB_STEP_SUMMARY, so the result is visible directly in the GitHub Actions run UI - including on manual dispatches, where no PR comment is posted at all. The same rendered markdown is exposed as a new summary job output and reused verbatim by vr-report, so the run summary and the PR comment can no longer drift apart. The change also fixes a jq bug that made the report claim every page had changed regardless of its actual diff percentage.

Changes

  • Added a Write run summary step to the vr-compare job. It renders a heading, a one-line X of Y pages changed, Z% overall difference statement, a settings table (source environment, target environment, diff ID), and a link to the Diffy report into $GITHUB_STEP_SUMMARY, and writes the same markdown to a summary step output. Every interpolated value is passed through the step's env block rather than inlined into the script.
  • Exposed summary: ${{ steps.summary.outputs.summary }} as a vr-compare job output.
  • Reduced the vr-report sticky comment to message: ${{ needs.vr-compare.outputs.summary }}, replacing the hand-assembled bullet list that read pages_changed, pages_total, changes_percent, and shared_url individually and additionally depended on workflow-level env resolving in the second job.
  • Fixed the pages_changed expression in the Fetch comparison result step. Diffy returns percentageChanges as a JSON string, so the previous map(.value.percentageChanges // 0) | add > 0 concatenated the string values ("0" + "0" becomes "00") and, because jq sorts every string above every number, the comparison was true for every page - the report always claimed all pages had changed. The new [.[].percentageChanges | tonumber? // 0] | add > 0 coerces each value to a number before summing. Checked against a sample Diffy payload: the old expression reported 3 of 3 pages changed where the new one correctly reports 1 of 3, and the new form tolerates numeric values, absent percentageChanges keys, and empty page objects without erroring.
  • Updated .vortex/docs/content/development/visual-regression.mdx to match: the vr-compare step list gained the run-summary step, the vr-report entry now states it posts the same report, both ASCII flow diagrams show where the run summary is produced, and the manual-dispatch note no longer claims the result is visible "only in the Diffy UI (and in the workflow run log)".
  • Regenerated .vortex/installer/tests/Fixtures/handler_process/visual_regression_enabled/.github/workflows/test-vr.yml with ahoy update-snapshots.

Screenshots

N/A - the change covers a CI workflow and a documentation page.

Before / After

BEFORE

┌──────────────────────────────────────────────────────────────┐
│ vr-compare job                                               │
│   parse result: buggy jq marks every page as changed         │
│   outputs: pages_changed, pages_total, changes_percent,      │
│            shared_url                                        │
└────────────────────────────┬─────────────────────────────────┘
                             │  job log only, run summary empty
                             ▼
┌──────────────────────────────────────────────────────────────┐
│ vr-report job                                                │
│   rebuilds its own bullet list from the four outputs         │
│   above, plus workflow-level env for the environments        │
│   posts the sticky PR comment                                │
└──────────────────────────────────────────────────────────────┘

Manual dispatch: no PR, so no comment - the result never leaves
the job log and the Diffy UI.


AFTER

┌──────────────────────────────────────────────────────────────┐
│ vr-compare job                                               │
│   parse result: jq coerces to number before comparing        │
│   Write run summary renders the report once into             │
│   the run summary                                            │
│   output: summary (one rendered markdown blob)               │
└────────────────────────────┬─────────────────────────────────┘
                             │
               ┌─────────────┴─────────────┐
               ▼                           ▼
          run summary                 PR comment

Both are produced from the same rendered markdown, so they cannot
disagree, and the run summary is written whether or not the run
resolved a PR.

Also fixed 'pages_changed' counting every page as changed, as Diffy returns 'percentageChanges' as a string and the comparison ran against the concatenated value.
@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 6 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: b1ff3eab-680b-4f48-bb90-7a32e0748eb0

📥 Commits

Reviewing files that changed from the base of the PR and between ece8160 and 79b683e.

⛔ Files ignored due to path filters (1)
  • .vortex/installer/tests/Fixtures/handler_process/visual_regression_enabled/.github/workflows/test-vr.yml is excluded by !.vortex/installer/tests/Fixtures/**
📒 Files selected for processing (2)
  • .github/workflows/test-vr.yml
  • .vortex/docs/content/development/visual-regression.mdx

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

Code coverage (threshold: 90%)

  Classes: 100.00% (1/1)
  Methods: 100.00% (2/2)
  Lines:   98.55% (204/207)
Per-class coverage
Drupal\ys_demo\Plugin\Block\CounterBlock
  Methods: 100.00% ( 2/ 2)   Lines: 100.00% ( 10/ 10)

@AlexSkrypnyk

This comment has been minimized.

2 similar comments
@AlexSkrypnyk

This comment has been minimized.

@AlexSkrypnyk

Copy link
Copy Markdown
Member Author

Code coverage (threshold: 90%)

  Classes: 100.00% (1/1)
  Methods: 100.00% (2/2)
  Lines:   98.55% (204/207)
Per-class coverage
Drupal\ys_demo\Plugin\Block\CounterBlock
  Methods: 100.00% ( 2/ 2)   Lines: 100.00% ( 10/ 10)

@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.38%. Comparing base (ece8160) to head (79b683e).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2865      +/-   ##
==========================================
- Coverage   86.81%   86.38%   -0.43%     
==========================================
  Files         100       93       -7     
  Lines        4846     4687     -159     
  Branches       47        3      -44     
==========================================
- Hits         4207     4049     -158     
+ Misses        639      638       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown

📖 Documentation preview for this pull request has been deployed to Netlify:

https://6a69bac2439d02e7c9adcf1a--vortex-docs.netlify.app

This preview is rebuilt on every commit and is not the production documentation site.

@AlexSkrypnyk AlexSkrypnyk added the Needs review Pull request needs a review from assigned developers label Jul 29, 2026
@AlexSkrypnyk
AlexSkrypnyk merged commit ab2ae94 into main Jul 29, 2026
34 checks passed
@AlexSkrypnyk
AlexSkrypnyk deleted the feature/vr-run-summary branch July 29, 2026 09:04
@github-project-automation github-project-automation Bot moved this from BACKLOG to Release queue in Vortex 1.x Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Needs review Pull request needs a review from assigned developers

Projects

Status: Release queue

Development

Successfully merging this pull request may close these issues.

1 participant