ci: run timing-dependent jobs only where a tree is first tested

.github/workflows/interop.yml

@@ -9,7 +9,11 @@ name: Interop
 # job per server, as CI runs the blocking ones. A job fails only on a test
 # failing that its servers/<id>/expected-failures.json does not list, or
 # when no test ran; the job's summary says what passed, what failed as
-# expected and why, and what passes though listed.
+# expected and why, and what passes though listed. A release commit,
+# whose push its paths match (package.json, package-lock.json), runs
+# nothing: it changes only the version's files, so its code is the
+# commit's before it, which Interop already ran on
+# (docs/releasing.md).
 
 on:
   push:
@@ -31,13 +35,21 @@ on:
 permissions:
   contents: read
 
+# A newer push's run cancels the one still going. A release commit's run,
+# which runs nothing, has a group of its own, so it never cancels the run
+# on the code it releases.
 concurrency:
-  group: interop-${{ github.ref }}
+  group: >-
+    interop-${{ github.ref }}${{
+    startsWith(github.event.head_commit.message, 'chore(release): ') &&
+    format('-{0}', github.sha) || '' }}
   cancel-in-progress: true
 
 jobs:
   pod:
     name: pod (${{ matrix.server }})
+    if: >-
+      !startsWith(github.event.head_commit.message, 'chore(release): ')
     runs-on: ubuntu-latest
     timeout-minutes: 15
     strategy:

docs/deployment.md

@@ -18,7 +18,7 @@ is made:
 flowchart LR
     accTitle: When a version is deployed
     accDescr: A push to main, or a run by hand, deploys only the newest version, from its release commit and with its GitHub release, after the build, each release's docs and the guard.
-    push["push to main"] --> ci["CI: npm run check,<br/>build, pySHACL, pods,<br/>journeys, docs"] -->|success| tag{"deploy.yml:<br/>a version tag<br/>names the commit?"}
+    push["push to main"] --> ci["CI: npm run check,<br/>build, pySHACL, docs"] -->|success| tag{"deploy.yml:<br/>a version tag<br/>names the commit?"}
     tag -->|no| skip["not deployed"]
     tag -->|yes| release{"the newest version,<br/>its release commit,<br/>its GitHub release?"}
     release -->|no| skip
@@ -34,8 +34,10 @@ repository:
 - **After CI on a release.** A release ([releasing.md](releasing.md))
   follows a green CI run on `main`, in a workflow of its own. It pushes
   a commit to `main` that sets the new version,
-  `chore(release): X.Y.Z`, and tags it `vX.Y.Z`. CI runs on that commit
-  as on any push, and once it passes, the deploy builds it, so the
+  `chore(release): X.Y.Z`, and tags it `vX.Y.Z`. CI runs on that commit,
+  its deterministic jobs only, as the code is its parent's
+  ([releasing.md](releasing.md#what-each-stage-checks)), and once it
+  passes, the deploy builds it, so the
   site's footer shows the new version. The deploy looks for a version
   tag that names the commit CI checked, and deploys nothing when there
   is none: any other push to `main`, a merged pull request or a

docs/glossary.md

@@ -221,7 +221,8 @@ it most. The specifications' versions and status are in
   every pull request and every push to `main`. A change is merged only
   when they pass. [testing.md](testing.md#continuous-integration).
 - **`ci-ok`**: the one CI check a merge to `main` waits for, which
-  passes only when every other job passed.
+  passes only when every other job passed, or was skipped as CI's gate
+  decides ([releasing.md](releasing.md#what-each-stage-checks)).
   [testing.md](testing.md#continuous-integration).
 - **commitlint**: the tool that checks that a commit message follows
   Conventional Commits. CI runs it on a pull request's commits.

docs/releasing.md

@@ -206,6 +206,78 @@ checks it). At the end of the file, a section written by hand sums up
 what the first version, 1.0.0, contains, which its generated entry
 lists only in part.
 
+## What each stage checks
+
+A change to `main` meets CI up to four times on its way to the site,
+and the stages do not all have the same say. Each job of
+[ci.yml](../.github/workflows/ci.yml) is deterministic or
+timing-dependent ([testing.md](testing.md#deterministic-and-timing-dependent-jobs)),
+and CI's `gate` job (`npm run ci:gate`,
+[ciGate.ts](../scripts/ciGate.ts)) decides, for each run, whether the
+timing-dependent jobs run. Every job that runs blocks; the
+timing-dependent ones are skipped, not run to no purpose, where the same
+code already passed them, as they are the slowest jobs and the ones a
+busy runner fails.
+
+| Stage | What runs, and blocks |
+|---|---|
+| A pull request | every job |
+| The merge queue (`merge_group`) | every job: this is where a tree is tested at full strength before it lands, and where a flaky check is fixed, not retried |
+| A push to `main` the merge queue passed | the deterministic jobs; the same commit passed every job in the queue |
+| A push to `main` that skipped the queue | every job, as in the queue |
+| A release commit | the deterministic jobs, and that the commit changes nothing but the files a release writes, when CI passed its parent on `main`; every job when it did not |
+| A run by hand | every job |
+
+So a change's timing-dependent jobs run in its pull request, once for
+each push to it, and once more in the merge queue, and never again for
+the same code. Interop ([testing.md](testing.md)), which runs the
+advisory servers' suites after a push to `main`, likewise skips a
+release commit.
+
+**Test a tree once, at full strength, before it lands.** The merge
+queue builds exactly the commit `main` moves to
+([testing.md](testing.md#the-merge-queue)), so the push run on `main`
+that follows checks a tree CI already passed in full. The gate asks
+GitHub's API whether a `merge_group` run passed that commit; when one
+did, only the deterministic jobs run.
+
+**Know when a change skipped the queue.** A change can reach `main`
+without the queue: a merge through a ruleset bypass, such as one from a
+security advisory's private fork, where Actions do not run. Then the
+push run is the first CI the change gets, no `merge_group` run passed
+its commit, and every job runs. The release follows that run only
+once it passes ([below](#what-a-release-does)), so a release built on
+such a commit waits for the full run.
+
+**Deploy on the tested tree.** A release commit changes only the
+version's files: the five that `release.config.mjs` lists (`assets`):
+`CHANGELOG.md`, `CITATION.cff`, `package.json`, `package-lock.json`
+and `publiccode.yml`. Its code is its parent's, which CI passed on
+`main` before the release was made. So its run runs the deterministic
+jobs, the build among them, and the gate fails it if the commit changes
+any other file, or has more than one parent. The timing-dependent jobs,
+the user journeys in every browser among them, are skipped, and cannot
+hold up the deploy. Should CI not have passed its parent on `main` (a release
+commit pushed by hand), every job runs.
+
+The report of the journeys the site publishes at `/journeys/`
+([deployment.md](deployment.md)) still has every browser's: the release
+commit's run takes the journeys from the run that ran them on its code,
+which the gate names (`tested-by`). That is its parent's merge queue
+run, which keeps the journeys' traces and reports for this, or, for a
+parent that skipped the queue, the parent's own push run. GitHub keeps
+that run's reports for 7 days; once one has expired, as for a release
+made late by hand, the gate runs every job instead.
+
+What a skipped job would have saved on `main`, the caches every pull
+request reads, CI's `warm` jobs save instead: `warm` looks up each
+server's images, the journeys' server's, and the journeys' browsers, and `warm-images` and
+`warm-browsers` pull, build or download only what is missing, and save
+it ([testing.md](testing.md#continuous-integration)).
+
+A `workflow_dispatch` run of the deploy runs `npm run check` itself,
+as before, and needs no CI run.
+
 ## What a release does
 
 [release.yml](../.github/workflows/release.yml) runs when CI has passed
@@ -421,9 +493,14 @@ next push to `main` that passes CI also tries again.
   It deploys nothing, and fails, if the tag names any commit but
   `chore(release): X.Y.Z`, or if the GitHub release is missing.
 
-- **CI failed on the release commit** (a flaky test, say): the version
-  is released but not deployed. Re-run CI's failed jobs on that commit;
-  once it passes, the deploy follows as usual. Or run the deploy by
+- **CI failed on the release commit**: the version is released but not
+  deployed. No timing-dependent job runs there
+  ([What each stage checks](#what-each-stage-checks)), so the failure is
+  a deterministic job's, or the `gate`'s, which refuses a release commit
+  that changes more than the version's files. A deterministic job that
+  failed on code its parent passed with is flaky: re-run CI's failed
+  jobs on that commit, and fix the flake in a pull request of its own;
+  once the run passes, the deploy follows as usual. Or run the deploy by
   hand, which checks the version itself. Either deploys the version
   only while it is the newest: once a newer one is released, the deploy
   after a re-run deploys nothing, and says so, and the site stays on

docs/standard-for-public-code.md

@@ -651,7 +651,9 @@ What holds today, and what does not, criterion by criterion.
   tests, which CI runs before the cross-check ([testing.md](testing.md)).
 - MUST, met — contributions pass all tests before they are merged.
   CI runs on every pull request, in the merge queue and on every push to
-  `main`, and `ci-ok` passes only when all its jobs did: `build`,
+  `main`, and `ci-ok` passes only when all its jobs did (on `main`,
+  the timing-dependent ones that the merge queue already passed are
+  skipped, [releasing.md](releasing.md#what-each-stage-checks)): `build`,
   `library` (the deck library and the public API compared with the
   change's base), `crosscheck`, `pod`, `contract`, `journeys` (every
   journey in Chromium, and a set of them in Firefox, WebKit and on a