Skip to content

test(journeys): spend the journeys' CI time only on what each run checks - #202

Merged
ixuz merged 7 commits into
mainfrom
journey-time
Oct 20, 2026
Merged

ixuz merged 7 commits into
mainfrom
journey-time

Conversation

@ixuz

@ixuz ixuz commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

What and why

Since #200, a pull request and the merge queue wait for the slowest journeys job. In the run that tested #200, that was journeys (engines, 2/2) at about 5.8 minutes, and the four jobs took about 20 job-minutes between them. Much of that time checked nothing a faster run did not already check. This follows the issue's proposal, one commit per step:

  1. Time the accessibility check apart. Each step's axe check now runs in a step of its own, "Accessibility", so the report times it. Locally it takes 35–67% of every journey's time, Chromium included: more than the issue's estimate.
  2. One job per engine. The journeys now run in five jobs side by side: chromium-1 and chromium-2 (Chromium in two shards), firefox, webkit and phone. The slow WebKit and phone journeys no longer land in one job. WebKit's and the phone's browsers share a cache. journeys-report and the warm jobs follow the new names.
  3. Axe in Chromium and on the phone only. Desktop Firefox and WebKit leave it out (metadata: { accessibility: false } on their projects). The phone still checks WebKit's rendering, and every screen is checked in Chromium. docs/accessibility.md says what goes unchecked: what only Firefox computes, such as a mixed color's contrast. Locally, Firefox's reload journey went from 31.7 to 17.5 seconds.
  4. No smoke journey in the engines. The reload journey now logs out at its end, so each engine still logs in and out once. The engines run 9 journeys instead of 12; Chromium still runs @smoke.
  5. The app's logic in Chromium only. App.step(title, body, { in: ["chromium"] }) runs a step only in the projects named, and the report shows it skipped in the others (test.step.skip). It is used for the reload journey's statistics and card history (steps 09–10) and the shared link's import, study and unknown link (steps 04–05). The guest journey still imports and studies in every engine.
  6. Chromium runs fully parallel. Each journey is a unit the shards and workers share out, so pod-safety's six journeys, about 100 seconds, are spread over both shards instead of ending one.
  7. The Studio journey, split. At 2.4 minutes it was the longest test, and one test cannot be split by a shard or a worker. Writing, checking, test-playing and publishing a course now make a journey of their own, in studio-course.journey.ts, with its own user, instance and studied deck. It is followed, in turn, by the journey that adds the published course from its link. The Studio journey keeps the decks, the workbench, the drafts and the graph. Locally the two take 64 and 28 seconds. docs/user-guide.md and docs/statistics.md point to the new steps.

Every Chromium and Firefox journey passes locally. WebKit and the phone could not run on the machine this was made on (missing system libraries), so this pull request's CI run is the first to run them with these changes, including the reload journey's new logout and the skipped steps. Its timings will show the saving per job.

Not done here: axe also takes about 44% of Chromium's journey time. Checking each screen once instead of at every step, an alternative the issue set aside, may be worth an issue of its own now that it is measured.

Closes: #201

Checklist

  • One issue. This pull request addresses one issue, linked above.
  • One step. This pull request does not finish its issue, so it says Refs:, not Closes:. Leave this unchecked otherwise.
  • Done, as AGENTS.md defines it: npm run check passes (lint, Markdown and spelling included), npm run lint:reuse too if a file was added, and npm run crosscheck too if ns/ or decks/ changed. (npm run lint:workflows passes too; crosscheck n/a: ns/ and decks/ unchanged)
  • Tests. New and changed code has its tests, and every package stays at 100% coverage (docs/testing.md). (The changes are to the journeys themselves, which run in Chromium and Firefox locally and in every engine in CI.)
  • Docs. The docs say what changed, and a new page in docs/ is listed in docs/docs.ttl. (no new page)
  • Languages. A new or changed message is in every locale: en, sv, ko, de, es and fr (packages/ui/src/i18n/). (n/a: no message changed)
  • Vocabulary and shapes. Any change to ns/ was agreed in an issue first; no published ns/shapes/*/vN.ttl or decks/*/vN.ttl was edited; generated files were regenerated, not edited. (n/a: nothing in ns/ or decks/ changed)
  • Breaking? Nothing in the public API breaks, or a commit says so (! and a BREAKING CHANGE: footer). npm run check:breaking sees a line of api/ that goes, but not a feature removed, nor a reader that reads less than before.
  • Pod data. Any change to what the app writes to a pod comes with its migration (docs/migrations.md). (n/a: the app is unchanged)
  • Policy. Any change to an external rule the code follows (SM-2, the plural rules, a specification) is cited. (n/a)
  • Licenses. New content is MIT, or a deck states its own license; a new file has its SPDX header (npm run lint:reuse), or a new deck its line in REUSE.toml; third-party content is credited.
  • Architecture. Imports follow docs/boundaries.md and the layers of docs/architecture.md.
  • Run. I ran the app (npm run dev) and tried the change, and considered the user journeys. (The change is to the journeys: every Chromium and Firefox journey was run against the built app; WebKit and the phone run in CI)
  • Authorship. I read and understand every line of this pull request, and answer for it, whatever tools helped write it (CONTRIBUTING.md).
  • Commits. Each commit is one logical change, in Conventional Commits form (type(scope): subject, ! and a BREAKING CHANGE: footer if it breaks something), with an imperative subject and a body that explains why (CONTRIBUTING.md).

ixuz added 7 commits October 20, 2026 15:22
Every journey step ends with axe, twice, once per theme, and the same
steps take 5 to 8 seconds longer on WebKit and the phone than in
Chromium. Nothing said how much of that was axe, which decides whether
leaving it out of some browsers is worth what it checks. The check now
runs in a step of its own, "Accessibility", so the report times it.

Refs: #201
Playwright shards the engine projects by test count, in project order,
so one shard got every WebKit and phone journey, the slowest, about 505
worker-seconds against 276, and every run waited for it. Firefox,
WebKit and the phone now each have a job of their own, beside
Chromium's two shards, so the slowest journeys never land together.
Five jobs, one more than before, each paying about a minute of setup.

WebKit's and the phone's browsers share a cache, as they install the
same ones.

Refs: #201
Timed apart, axe takes 35 to 47% of a journey's time: 9 to 15 seconds
of the 25 to 33 that the guest and reload journeys take, in Chromium
and Firefox. Desktop Firefox and WebKit ran it on the same few screens
that Chromium checks on every journey, and the phone checks WebKit's
rendering. Their projects now leave it out (metadata accessibility:
false), and docs/accessibility.md says what that leaves unchecked:
what only Firefox computes, such as a mixed color's contrast.

Refs: #201
…oke run per engine

Firefox, WebKit and the phone each ran the smoke journey, which logs in
and out, beside the reload journey, which logs in too. The reload
journey now also logs out, so each engine still logs in and out once,
and the engines no longer run @smoke: three journeys fewer a run, each
with its own account, login and instance. Chromium still runs it.

Refs: #201
…ys every engine runs

The reload journey checked the statistics and each card's history, and
the shared link's journey imported and studied the deck and followed a
link to no deck, in every engine. That is the app's logic, the same in
each, and Chromium checks it; the guest journey imports and studies in
every engine too. App.step now takes the projects a step runs in, and
skips it in the others through test.step.skip, so the report shows it
skipped. In Firefox, the reload journey went from 31.7 to 14.4 seconds
with axe left out too, and the shared link's from 11 to 5.

Refs: #201
With fullyParallel off, a shard and a worker took a journey file whole,
so pod-safety's six journeys, about 100 seconds, ran one after another
at the end of one Chromium shard. Chromium's project now runs its
journeys fully in parallel, each a unit the shards and workers share out, and
pod-safety's are spread over both shards. The studio file, whose second
journey needs what the first publishes, stays serial, as it says.

Refs: #201
The Studio journey took 2.4 minutes in CI, the longest test by far,
and one test is the one thing a shard or a worker cannot split. Its
course steps, writing a course, checking its release, test-playing it
in the trial and publishing it, now make a journey of their own in
studio-course.journey.ts, with its own user, instance and studied deck,
followed by the journey that adds the published course from its link.
The Studio journey keeps the decks, the workbench, the drafts of a deck
and a library deck's next version, and the graph; locally it went from
the whole to 64 seconds, and the course journey takes 28.

Closes: #201
@ixuz
ixuz added this pull request to the merge queue Oct 20, 2026
Merged via the queue into main with commit 931d167 Oct 20, 2026
31 checks passed
@ixuz
ixuz deleted the journey-time branch October 20, 2026 13:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The user journeys take longer in CI than what they check needs, and one engine shard holds up every run

1 participant