Skip to content

feat: add browser screenshot baseline support - #157

Draft
joshblack with Copilot wants to merge 4 commits into
mainfrom
copilot/add-screenshot-tests-to-vitest
Draft

feat: add browser screenshot baseline support#157
joshblack with Copilot wants to merge 4 commits into
mainfrom
copilot/add-screenshot-tests-to-vitest

Conversation

Copilot AI commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Scenarios can now use Vitest browser screenshot assertions without exposing expected baselines to the agent. Screenshots are generated and evaluated in the same Linux/Chromium environment.

Changes

  • Baseline isolation

    • Stores baselines under each scenario’s __screenshots__ directory.
    • Copies baselines into the workspace only after agent execution.
  • Deterministic screenshots

    • Pins Vitest, Playwright, and Chromium tooling.
    • Uses stable filenames independent of the host platform.
  • Regeneration

    • Adds a scenario-local command that regenerates baselines in the evaluation container:
pnpm exec agent-eval --update-screenshots .
  • Documentation
    • Documents screenshot assertions, storage, isolation, and regeneration.

Copilot AI and others added 3 commits August 22, 2026 14:14
Co-authored-by: joshblack <3901764+joshblack@users.noreply.github.com>
Co-authored-by: joshblack <3901764+joshblack@users.noreply.github.com>
Co-authored-by: joshblack <3901764+joshblack@users.noreply.github.com>
@joshblack

Copy link
Copy Markdown
Member

@copilot the way it should probably work for update snapshots is that it should run the scenario exactly as it is within the container since folks likely will change the scenario, take a screenshot, and then only commit the screenshot results as that's the ideal.

Co-authored-by: joshblack <3901764+joshblack@users.noreply.github.com>

Copilot AI commented Aug 22, 2026

Copy link
Copy Markdown
Contributor Author

@copilot the way it should probably work for update snapshots is that it should run the scenario exactly as it is within the container since folks likely will change the scenario, take a screenshot, a...

Addressed in a075f55: --update-screenshots now copies the authored scenario into the container as-is, excluding only generated/cache directories and existing baselines, before regenerating screenshots.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants