Add manifest-driven prep script for hosted data-heavy e2e fixtures by gcsecsey · Pull Request #4152 · Automattic/studio

gcsecsey · 2026-07-09T14:38:36Z

Related issues

Related to STU-1866
Stacked on Move backup fixtures to a shared root-level test-fixtures/ directory #4151

How AI was used in this PR

Claude Code planned and implemented the change under my direction, and verified the script’s behaviors (download, idempotence, corruption self-heal, offline modes) against a local HTTP server and then against the live fixtures site. I reviewed the result.

Proposed Changes

The minimal backup fixtures live in the repo (#4057 / #4151) and deliberately stay there. They’re small, and keeping them in git means per-PR CI doesn't need network access. This PR is for the other tier: realistic, data-heavy backups that are too large to commit. These are currently fetched manually from a shared Drive folder.

Data-heavy artifacts now live in the media library of studioe2efixtures.wordpress.com, a dedicated site, and are fetched on demand:

test-fixtures/manifest.json pins each hosted artifact to a URL, byte size, and SHA-256 hashes. Adding a new fixture is a manifest-only PR.
npm run e2e:fixtures downloads whatever the manifest declares into gitignored test-fixtures/downloads/, verifying every hash. Runs are idempotent, corrupted files self-heal, and while a failed download only warns locally (affected tests skip), CI runs with --require and fails loudly.
WP.com media uploads accept .zip but reject .tar.gz/.wpress/.sql, so hosted artifacts are zip wrapped. The manifest records both the wrapper and inner hashes and the script verifies both.
Playwright runs the prep automatically via globalSetup. Buildkite runs it explicitly and caches downloads/ keyed on the manifest hash.
I added a small (327 B) smoke-test artifact hosted on the site, so this mechanism is exercised on this PR's CI run.

See it working

From this PR’s own CI (E2E Tests on mac-arm64, build 19149):

--- :inbox_tray: Prepare e2e fixtures
No cache entry found for 'studio-e2e-fixtures-b6165840e86d…'
> tsx ./scripts/prepare-e2e-fixtures.ts --require
pipeline-test.sql: downloading https://studioe2efixtures.wordpress.com/wp-content/uploads/2026/07/studio-fixture-pipeline-test-2026-07.zip
pipeline-test.sql: ready.
No existing cache entry for studio-e2e-fixtures-b6165840e86d… – storing in cache

--- :playwright: Run End To End Tests
pipeline-test.sql: already downloaded and verified — skipping.

Testing Instructions

Run npm run e2e:fixtures
Check that it downloads the smoke-test artifact from the live fixtures site, verifies both hashes, and unwraps it into test-fixtures/downloads/pipeline-test.sql.
Run it again
Check that it prints "already downloaded and verified — skipping"
Corrupt the downloaded file (echo x >> test-fixtures/downloads/pipeline-test.sql) and run again to see it re-download.
With a wrong hash in the manifest, the run reports the mismatch (and -- --require exits 1).

Pre-merge Checklist

Have you checked for TypeScript, React or other console errors?

…nd CLI tests

…e fixtures

…t fixture

…d-fixture-prep-script

…@studio/common alias

gavande1

Thanks @gcsecsey for addressing the comments. Some tests are failing, but approving to unblock you.

gcsecsey · 2026-07-14T10:34:52Z

Thanks @gcsecsey for addressing the comments. Some tests are failing, but approving to unblock you.

Thanks @gavande1, I've looked into the failing Performance metrics job, and it seems to have inherited the fixture download step from this PR. I'll push a fix for this.

wpmobilebot · 2026-07-14T10:51:23Z

📊 Performance Test Results

Comparing 4e0ea7f vs trunk

app-size

Metric	trunk	`4e0ea7f`	Diff	Change
App Size (Mac)	1350.36 MB	1350.34 MB	0.02 MB	⚪ 0.0%

site-editor

Metric	trunk	`4e0ea7f`	Diff	Change
load	1082 ms	1047 ms	35 ms	⚪ 0.0%

site-startup

Metric	trunk	`4e0ea7f`	Diff	Change
siteCreation	7022 ms	6993 ms	29 ms	⚪ 0.0%
siteStartup	2360 ms	2365 ms	+5 ms	⚪ 0.0%

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

## Related issues - Related to [STU-1866](https://linear.app/a8c/issue/STU-1866/shared-artifact-storage-for-import-data-heavy-fixtures) (completes its last acceptance criterion: import tests updated) and [STU-1870](https://linear.app/a8c/issue/STU-1870/migrate-import-export-tests) - Stacked on #4152 ## How AI was used in this PR Claude Code seeded the source site's content, pre-flighted the artifact through real CLI imports (which surfaced a platform drop-in bug — Linear issue incoming), and implemented the rewire under my direction; I reviewed the result. ## Proposed Changes `import.test.ts`, the release-time test that imports a WordPress.com Jetpack backup, was skipped in CI since it was written: its archive had to be fetched by hand from a shared Drive folder, and the files there went stale. The minimal fixture in `import-formats.test.ts` covers the format's shape, this test's distinct job is catching drift in what customer backups actually contain, so it needs a real, current artifact. - The test consumes `test-fixtures/downloads/coolbeans-jetpack-backup-2026-07.tar.gz`, fetched by the #4152 pipeline from the fixtures site. Locally it still skips with an actionable message when the file is absent; in CI the prep step hard-fails first, so the silent-skip era ends. - The backup is a genuine Jetpack Cloud export (149 MB, 51 per-table SQL files incl. WooCommerce tables, full uploads tree, wpcomsh mu-plugins, `__wp__` core dir) of **Cool Beans** (gcsecseywoa11.wpcomstaging.com), a purpose-built public test site whose content is self-describing: a hero post stating the site's purpose, encoding-test posts (ASCII and UTF-8/emoji), a long-form truncation canary, media/taxonomy/comment-thread tests, and nested pages. - Assertions verify the imported site really serves the backup: blog name + tagline in the page title, the custom theme's landing page rendering, and the hero post resolving at its pretty permalink (which also proves the permalink structure imported). - The stale `apps/studio/e2e/imports/` directory, its P2-pointing readme, and its `.gitignore` entries are removed. ## Verified end to end - `npm run e2e:fixtures` downloads the wrapper from the live fixtures site, verifies both SHA-256 hashes, unwraps; a second run verified-skips both fixtures. - Pre-flight of this exact artifact through the real CLI: import exits 0, `wp option get blogname` → `Cool Beans`, started site serves HTTP 200 with the full title, theme landing renders, hero permalink returns 200 with the checklist text — i.e., every assertion in this PR's test, validated against the shipped artifact. ## Testing Instructions 1. `npm run e2e:fixtures` — downloads + verifies both hosted fixtures (~150 MB). 2. This PR's e2e CI runs `import.test.ts` for real for the first time — the E2E jobs are the authoritative proof (note: they now download the artifact: mac/windows cache it keyed on the manifest, Linux fetches per run). ## Pre-merge Checklist - [ ] Have you checked for TypeScript, React or other console errors?

gcsecsey added 2 commits July 9, 2026 14:53

Move backup fixtures to root-level test-fixtures/ shared by desktop a…

712b83c

…nd CLI tests

Add manifest-driven prep script and hosting runbook for data-heavy e2…

4f36800

…e fixtures

github-actions Bot assigned gcsecsey Jul 9, 2026

Register the fixtures site in the runbook and seed a hosted smoke-tes…

d74d467

…t fixture

gcsecsey requested review from a team and gavande1 July 10, 2026 12:34

gcsecsey marked this pull request as ready for review July 10, 2026 12:34

gcsecsey requested a review from a team as a code owner July 10, 2026 12:34

Base automatically changed from gcsecsey/stu-1866-move-backup-fixtures to trunk July 10, 2026 12:35

gavande1 reviewed Jul 13, 2026

View reviewed changes

Comment thread scripts/lib/prepare-e2e-fixtures.ts Outdated

gavande1 reviewed Jul 13, 2026

View reviewed changes

Comment thread apps/studio/e2e/global-setup.ts Outdated

gavande1 reviewed Jul 13, 2026

View reviewed changes

Comment thread apps/studio/e2e/import-formats.test.ts Outdated

gavande1 reviewed Jul 13, 2026

View reviewed changes

Comment thread scripts/lib/prepare-e2e-fixtures.ts Outdated

gcsecsey and others added 6 commits July 13, 2026 11:48

Merge remote-tracking branch 'origin/trunk' into gcsecsey/stu-1866-ad…

1d514e3

…d-fixture-prep-script

Move e2e fixture prep lib into packages/common and import it via the …

f7d8e1a

…@studio/common alias

Resolve e2e fixture paths from a shared repo-root constant

271318c

Keep the fixture zip-wrap rationale only in the fixtures readme

6e27710

Merge branch 'trunk' into gcsecsey/stu-1866-add-fixture-prep-script

b9f03d5

Merge branch 'trunk' into gcsecsey/stu-1866-add-fixture-prep-script

04c852e

gcsecsey mentioned this pull request Jul 13, 2026

Rewire import e2e test to a hosted Jetpack backup #4186

Merged

1 task

Merge branch 'trunk' into gcsecsey/stu-1866-add-fixture-prep-script

50a1e6c

gavande1 approved these changes Jul 14, 2026

View reviewed changes

Merge branch 'trunk' into gcsecsey/stu-1866-add-fixture-prep-script

3415da4

Skip the e2e fixture globalSetup in the performance metrics config

4e0ea7f

gcsecsey enabled auto-merge (squash) July 14, 2026 10:40

gcsecsey merged commit f6124f8 into trunk Jul 14, 2026
11 checks passed

gcsecsey deleted the gcsecsey/stu-1866-add-fixture-prep-script branch July 14, 2026 11:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add manifest-driven prep script for hosted data-heavy e2e fixtures#4152

Add manifest-driven prep script for hosted data-heavy e2e fixtures#4152
gcsecsey merged 12 commits into
trunkfrom
gcsecsey/stu-1866-add-fixture-prep-script

gcsecsey commented Jul 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gavande1 left a comment

Uh oh!

gcsecsey commented Jul 14, 2026

Uh oh!

wpmobilebot commented Jul 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

gcsecsey commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related issues

How AI was used in this PR

Proposed Changes

See it working

Testing Instructions

Pre-merge Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gavande1 left a comment

Choose a reason for hiding this comment

Uh oh!

gcsecsey commented Jul 14, 2026

Uh oh!

wpmobilebot commented Jul 14, 2026

📊 Performance Test Results

app-size

site-editor

site-startup

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gcsecsey commented Jul 9, 2026 •

edited

Loading