Repository navigation
test(journeys): spend the journeys' CI time only on what each run checks - #202
Merged
Merged
Conversation
Every journey step ends with axe, twice, once per theme, and the same steps take 5 to 8 seconds longer on WebKit and the phone than in Chromium. Nothing said how much of that was axe, which decides whether leaving it out of some browsers is worth what it checks. The check now runs in a step of its own, "Accessibility", so the report times it. Refs: #201
Playwright shards the engine projects by test count, in project order, so one shard got every WebKit and phone journey, the slowest, about 505 worker-seconds against 276, and every run waited for it. Firefox, WebKit and the phone now each have a job of their own, beside Chromium's two shards, so the slowest journeys never land together. Five jobs, one more than before, each paying about a minute of setup. WebKit's and the phone's browsers share a cache, as they install the same ones. Refs: #201
Timed apart, axe takes 35 to 47% of a journey's time: 9 to 15 seconds of the 25 to 33 that the guest and reload journeys take, in Chromium and Firefox. Desktop Firefox and WebKit ran it on the same few screens that Chromium checks on every journey, and the phone checks WebKit's rendering. Their projects now leave it out (metadata accessibility: false), and docs/accessibility.md says what that leaves unchecked: what only Firefox computes, such as a mixed color's contrast. Refs: #201
…oke run per engine Firefox, WebKit and the phone each ran the smoke journey, which logs in and out, beside the reload journey, which logs in too. The reload journey now also logs out, so each engine still logs in and out once, and the engines no longer run @smoke: three journeys fewer a run, each with its own account, login and instance. Chromium still runs it. Refs: #201
…ys every engine runs The reload journey checked the statistics and each card's history, and the shared link's journey imported and studied the deck and followed a link to no deck, in every engine. That is the app's logic, the same in each, and Chromium checks it; the guest journey imports and studies in every engine too. App.step now takes the projects a step runs in, and skips it in the others through test.step.skip, so the report shows it skipped. In Firefox, the reload journey went from 31.7 to 14.4 seconds with axe left out too, and the shared link's from 11 to 5. Refs: #201
With fullyParallel off, a shard and a worker took a journey file whole, so pod-safety's six journeys, about 100 seconds, ran one after another at the end of one Chromium shard. Chromium's project now runs its journeys fully in parallel, each a unit the shards and workers share out, and pod-safety's are spread over both shards. The studio file, whose second journey needs what the first publishes, stays serial, as it says. Refs: #201
The Studio journey took 2.4 minutes in CI, the longest test by far, and one test is the one thing a shard or a worker cannot split. Its course steps, writing a course, checking its release, test-playing it in the trial and publishing it, now make a journey of their own in studio-course.journey.ts, with its own user, instance and studied deck, followed by the journey that adds the published course from its link. The Studio journey keeps the decks, the workbench, the drafts of a deck and a library deck's next version, and the graph; locally it went from the whole to 64 seconds, and the course journey takes 28. Closes: #201
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Since #200, a pull request and the merge queue wait for the slowest journeys job. In the run that tested #200, that was
journeys (engines, 2/2)at about 5.8 minutes, and the four jobs took about 20 job-minutes between them. Much of that time checked nothing a faster run did not already check. This follows the issue's proposal, one commit per step:chromium-1andchromium-2(Chromium in two shards),firefox,webkitandphone. The slow WebKit and phone journeys no longer land in one job. WebKit's and the phone's browsers share a cache.journeys-reportand thewarmjobs follow the new names.metadata: { accessibility: false }on their projects). The phone still checks WebKit's rendering, and every screen is checked in Chromium.docs/accessibility.mdsays what goes unchecked: what only Firefox computes, such as a mixed color's contrast. Locally, Firefox's reload journey went from 31.7 to 17.5 seconds.@smoke.App.step(title, body, { in: ["chromium"] })runs a step only in the projects named, and the report shows it skipped in the others (test.step.skip). It is used for the reload journey's statistics and card history (steps 09–10) and the shared link's import, study and unknown link (steps 04–05). The guest journey still imports and studies in every engine.pod-safety's six journeys, about 100 seconds, are spread over both shards instead of ending one.studio-course.journey.ts, with its own user, instance and studied deck. It is followed, in turn, by the journey that adds the published course from its link. The Studio journey keeps the decks, the workbench, the drafts and the graph. Locally the two take 64 and 28 seconds.docs/user-guide.mdanddocs/statistics.mdpoint to the new steps.Every Chromium and Firefox journey passes locally. WebKit and the phone could not run on the machine this was made on (missing system libraries), so this pull request's CI run is the first to run them with these changes, including the reload journey's new logout and the skipped steps. Its timings will show the saving per job.
Not done here: axe also takes about 44% of Chromium's journey time. Checking each screen once instead of at every step, an alternative the issue set aside, may be worth an issue of its own now that it is measured.
Closes: #201
Checklist
Refs:, notCloses:. Leave this unchecked otherwise.npm run checkpasses (lint, Markdown and spelling included),npm run lint:reusetoo if a file was added, andnpm run crosschecktoo ifns/ordecks/changed. (npm run lint:workflowspasses too; crosscheck n/a:ns/anddecks/unchanged)docs/is listed in docs/docs.ttl. (no new page)packages/ui/src/i18n/). (n/a: no message changed)ns/was agreed in an issue first; no publishedns/shapes/*/vN.ttlordecks/*/vN.ttlwas edited; generated files were regenerated, not edited. (n/a: nothing inns/ordecks/changed)!and aBREAKING CHANGE:footer).npm run check:breakingsees a line ofapi/that goes, but not a feature removed, nor a reader that reads less than before.npm run lint:reuse), or a new deck its line inREUSE.toml; third-party content is credited.npm run dev) and tried the change, and considered the user journeys. (The change is to the journeys: every Chromium and Firefox journey was run against the built app; WebKit and the phone run in CI)type(scope): subject,!and aBREAKING CHANGE:footer if it breaks something), with an imperative subject and a body that explains why (CONTRIBUTING.md).