Skip to content

ci: run timing-dependent jobs only where a tree is first tested - #200

Merged
ixuz merged 4 commits into
mainfrom
ci-gates
Oct 20, 2026
Merged

ixuz merged 4 commits into
mainfrom
ci-gates

Conversation

@ixuz

@ixuz ixuz commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What and why

A change met CI up to four times on its way to the site: its pull request, the merge queue, the push to main, and the release commit. Every run had the same say, and every run ran the slowest jobs again: the timing-dependent ones (pod, contract and the user journeys). In recent runs they took about 30 of each run's 36–41 job-minutes. 4.1.1 stayed released but not deployed until the engines' journeys, flaky on the release commit, had been re-run by hand three times. Its code had already passed in the merge queue and on main.

This makes each stage's job explicit, as the issue proposes (written down in docs/releasing.md#what-each-stage-checks):

  • A new gate job (npm run ci:gate, scripts/ciGate.ts) decides what each run runs. Every job that runs blocks.
    • Every job runs in a pull request, in the merge queue, on a run by hand, and on a push to main that no merge_group run passed, such as an advisory's merge through the ruleset bypass. So a release built on such a commit waits for the full run.
    • The timing-dependent jobs are skipped on a push to main that the queue passed, and on a release commit whose parent passed CI on main. These are pod, contract, and the journeys in every browser. The same code already passed them. ci-ok accepts a skipped pod, contract or journeys then, and only then.
  • The gate fails a release commit that changes anything besides the files release.config.mjs commits, or that has more than one parent.
  • The /journeys/ report keeps every browser. The release commit's run takes the journeys' blobs from the run that ran them on its code, which the gate names (tested-by). That is usually the parent's merge queue run, which now keeps traces and blobs for every journeys job. If those blobs have expired (GitHub keeps them 7 days, so a release made late by hand), the gate runs every job instead, and the deploy is never left stuck.
  • The journeys are split into shards: each browser's journeys run in two shards side by side (journeys (chromium, 1/2) to journeys (engines, 2/2)). Pull requests and the queue wait about half as long for them. Each shard repeats about a minute of setup.
  • New warm jobs keep main's caches filled. The skipped jobs used to save each server's images, the journeys' server image and the journeys' browsers. warm looks them up, and warm-images and warm-browsers make and save only what is missing. Like caches, they never fail the run.
  • Interop skips a release commit, which only changes the version's files. Its run has a concurrency group of its own, so it no longer cancels the Interop run on the code it releases.
  • Each job is labeled deterministic or timing-dependent, in ci.yml and in docs/testing.md. docs/testing.md also adds the rule to count requests rather than time them.

The timing-dependent jobs now run twice per merged pull request (in its pull request and in the queue), not four times. A push to main the queue passed now takes about 4 minutes instead of about 9.

Some of this can only be confirmed on GitHub: the run passing with skipped jobs, downloading the journeys' blobs from another run, and the cache lookup. Please watch the first push to main and the first release after merging. On that first push, every journeys job should be skipped and journeys-report should pull all four blobs from the queue run. No repository settings need to change.

Closes: #199

Checklist

  • One issue. This pull request addresses one issue, linked above.
  • One step. This pull request does not finish its issue, so it says Refs:, not Closes:. Leave this unchecked otherwise.
  • Done, as AGENTS.md defines it: npm run check passes (lint, Markdown and spelling included), npm run lint:reuse too if a file was added, and npm run crosscheck too if ns/ or decks/ changed. (npm run lint:workflows passes too; crosscheck n/a: ns/ and decks/ unchanged)
  • Tests. New and changed code has its tests, and every package stays at 100% coverage (docs/testing.md).
  • Docs. The docs say what changed, and a new page in docs/ is listed in docs/docs.ttl. (no new page)
  • Languages. A new or changed message is in every locale: en, sv, ko, de, es and fr (packages/ui/src/i18n/). (n/a: no message changed)
  • Vocabulary and shapes. Any change to ns/ was agreed in an issue first; no published ns/shapes/*/vN.ttl or decks/*/vN.ttl was edited; generated files were regenerated, not edited. (n/a: nothing in ns/ or decks/ changed)
  • Breaking? Nothing in the public API breaks, or a commit says so (! and a BREAKING CHANGE: footer). npm run check:breaking sees a line of api/ that goes, but not a feature removed, nor a reader that reads less than before.
  • Pod data. Any change to what the app writes to a pod comes with its migration (docs/migrations.md). (n/a: the app is unchanged)
  • Policy. Any change to an external rule the code follows (SM-2, the plural rules, a specification) is cited. (n/a)
  • Licenses. New content is MIT, or a deck states its own license; a new file has its SPDX header (npm run lint:reuse), or a new deck its line in REUSE.toml; third-party content is credited.
  • Architecture. Imports follow docs/boundaries.md and the layers of docs/architecture.md.
  • Run. I ran the app (npm run dev) and tried the change, and considered the user journeys. (n/a: CI only; the journeys' report on main is covered above)
  • Authorship. I read and understand every line of this pull request, and answer for it, whatever tools helped write it (CONTRIBUTING.md).
  • Commits. Each commit is one logical change, in Conventional Commits form (type(scope): subject, ! and a BREAKING CHANGE: footer if it breaks something), with an imperative subject and a body that explains why (CONTRIBUTING.md).

ixuz added 2 commits October 20, 2026 14:17
A release commit changes only the version's files, but it matches
Interop's paths (package.json, package-lock.json), so Interop ran the
advisory servers' suites again on code it had just run them on, and,
sharing one concurrency group on main, canceled that run if it was
still going. Its job is now skipped, and its run has a group of its
own, so the run on the code it releases always finishes.

Refs: #199
A change met CI up to four times on its way to the site, and every run
had the same say: 4.1.1 stayed released but undeployed until the
engines' journeys, flaky on the release commit, were re-run by hand
three times, although its code had passed in the merge queue and on
main already. Each of those runs also spent its slowest jobs, the
timing-dependent ones, on code they had already passed.

A new gate job (npm run ci:gate) decides what each run runs. A pull
request, the merge queue, a run by hand and a push to main that no
merge_group run passed (an advisory's merge through the ruleset
bypass, say) run every job, and every job blocks. A push the queue
passed, and a release commit whose parent CI passed on main, skip the
timing-dependent jobs (pod, contract, the journeys in Firefox, WebKit
and on a phone) instead of running them again; ci-ok lets them be
skipped then and only then. The gate fails a release commit that
changes anything but the files release.config.mjs commits.

The report the site publishes at /journeys/ keeps every browser's
journeys, as the release commit's run takes the engines' from the run
that ran them on its code, which the gate names, and the queue's
engines keep their traces and blob for it. When that blob has expired,
the gate runs every job instead. The caches the skipped jobs saved on
main, each server's images and the engines' browsers, are looked up by
a warm job and only the missing ones made and saved.

Each job in ci.yml is labeled deterministic or timing-dependent, as in
docs/testing.md, which also says to count requests rather than time
them; docs/releasing.md says what each stage runs.

Closes: #199
@ixuz ixuz added enhancement New feature or request ci/cd Countinuous integration and deployment labels Oct 20, 2026
ixuz added 2 commits October 20, 2026 14:48
The Chromium journeys were kept on every push to main and on the
release commit, as the deterministic jobs were, although they wait on a
real server behind TLS as the other engines' do, and although that tree
had just passed them in the merge queue. They are the run's longest
job, about 8 minutes, spent twice more on main for every merged pull
request.

Both journeys jobs are now timing-dependent, and skipped where the gate
skips the others. The merge queue keeps both jobs' traces and blobs, and
main's report takes both from the run the gate names, which must still
keep them. The warm jobs also refill the journeys' server image and Chromium's
browsers, which the Chromium job saved on main. A skipped
warm-images job no longer shows its name's unexpanded matrix.

Refs: #199
The journeys are the run's longest jobs, about 8.5 minutes each, and a
pull request and the merge queue wait for them. Each browser's journeys
now run in two shards side by side (npx playwright test --shard), so
the run waits for about half as long. Each shard pays the job's setup
again, about a minute, for the shorter wait.

Each shard keeps a blob of its own, which journeys-report merges and
requires, and the gate checks that the run it takes them from keeps its
journeys-blob-* artifacts, whatever the number of shards.

Refs: #199
@ixuz
ixuz added this pull request to the merge queue Oct 20, 2026
Merged via the queue into main with commit f6c54f3 Oct 20, 2026
30 checks passed
@ixuz
ixuz deleted the ci-gates branch October 20, 2026 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cd Countinuous integration and deployment enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The same code is tested up to four times on its way to the site, and any one flaky run blocks the release or the deploy

1 participant