The user journeys take longer in CI than what they check needs, and one engine shard holds up every run #201

The problem

Since #200, each browser's journeys run in two shards, and a pull request and the merge queue wait for the slowest one. In the run that tested #200, the four shards took 4.3, 3.9, 2.9 and 4.8 minutes of tests, each plus about a minute of setup. That is about 20 job-minutes, and the critical path is journeys (engines, 2/2) at about 5.8 minutes. Much of that time checks nothing that a faster run does not already check:

• Every step costs more on WebKit and the phone. App.step takes a screenshot and runs axe twice, once per theme, after every step. The same steps take 5–8 s longer each on WebKit and the phone than on Chromium: study-reload takes 37 s in Chromium and 2.0 min on the phone; guest takes 41 s and 2.1 min. Nothing measures yet how much of that is axe.
• The engine shards are uneven. Playwright splits the engine projects by test count, in project order. So engines, 2/2 gets every slow WebKit and phone journey, about 505 worker-seconds, while engines, 1/2 gets about 276.
• The engines repeat app logic that doesn't differ between engines. Only what differs most between engines needs to run in them (docs/testing.md#browsers): the login's redirects, a guest's storage, a session restored after a reload, and a phone's layout. But study-reload also checks the statistics and each card's history (steps 09–10), and try-link imports and studies the deck, which guest already covers in every engine. @smoke logs in and out, which study-reload does too.
• Chromium's shards don't balance either. fullyParallel: false keeps pod-safety's six tests (about 100 s) together in one shard. studio ("manage an instance's decks", 2.4 min) is one serial test that a shard can't split.

Your proposal

Make the journeys spend CI time only on what each run checks, without losing a real login, a real pod round trip, or what differs between engines. In order:

1. Measure first. Wrap the axe check in App.step in a test.step("accessibility") of its own, so one CI run shows how long axe takes per step and per engine. The estimates below rest on that number.
2. Split the engine jobs by project, not by --shard. Use one job for each of firefox, webkit and phone, or one for phone and one for firefox and webkit. Each job then holds one engine's journeys instead of whatever the shard order gives it. This needs new job names, blob names and docs. Estimate: about 1.5 min off the critical path with three jobs, for one more minute of setup.
3. Check accessibility with axe on Chromium and the phone only. The phone still checks WebKit. Desktop Firefox and WebKit skip axe, and docs/accessibility.md says so. Estimate: 30–55 s off the critical path, to be confirmed by step 1. What is lost: Firefox's own contrast calculation.
4. Drop @smoke from the engines (ENGINE_JOURNEYS in playwright.config.ts), and have study-reload log out at its end, as smoke does. Each engine still logs in and out once.
5. Run some steps on Chromium only. Add an option to App.step that names the projects a step runs in, and skips it elsewhere through test.step.skip, so the report shows the skip. Use it for:
   • study-reload steps 09–10 (statistics and card history); the phone may keep one card if its layout should be checked there;
   • try-link steps 04–05 (import and study), which guest covers in every engine.
6. Let Chromium's journeys run file by file no longer: set fullyParallel: true, so pod-safety's tests spread across shards and workers. studio stays serial, as it says.
7. Split studio's long journey at its course steps into a journey file of its own, so no single test takes 2.4 min. This is the most work here, and is worth it only once Chromium is the critical path.

Estimated result:

The axe savings in this table are estimates; step 1 measures them. Each step is a commit of its own, checked against the next CI run's timings.

Alternatives

• More shards. This shortens the wait, but each shard pays about a minute of setup again, and the slow WebKit and phone journeys still land together.
• More workers per job (workers: 2 today, on 4-vCPU runners). This is free in job-minutes, but more browsers sharing one runner with the Solid server is the timing pressure that makes journeys flaky (The same code is tested up to four times on its way to the site, and any one flaky run blocks the release or the deploy #199).
• Seed pods directly instead of through the app, to skip the setup steps. Rejected: a seed would have to repeat what the app writes when it creates an instance, such as the preferences that pick FSRS-6, and would let journeys quietly study under SM-2 if it drifted.
• Log in through the identity provider's API instead of its pages. Little time saved, and the journeys would no longer check the real login, which is what the engines are there for.
• Check each screen with axe once per worker. This saves 15–30 s per Chromium shard, but screens that look alike yet hold different text (long card text, right-to-left text) would go unchecked. Left out for now.

Data

• This changes the vocabulary (ns/vocab/), the shapes (ns/shapes/) or the data the app writes to a pod.

Would you like to make it?

Yes, I would like to open a pull request