From 0091e6710b1cbfc557bc81111dde3d712efe18e3 Mon Sep 17 00:00:00 2001 From: Chris Shuttlesworth Date: Wed, 19 Aug 2026 12:05:59 -0400 Subject: [PATCH 1/2] =?UTF-8?q?feat:=20playwright-test=20=E2=80=94=20the?= =?UTF-8?q?=20estate=20browser=20testing=20convention=20(ADR=200017)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extracts nmon's proven Playwright CI pattern (nmon#76/#78/#79) into a reusable workflow, per Homelab-Skills ADR 0017 and nmon#125: the estate Playwright pin lives in playwright-test.yml's playwright-image default, consumers run their suites in that container on their own ARC runner, and a lockstep job holds every PR red until the caller's npm pin, the image tag, and the ghcr mirror's content agree. mirror-playwright moves here from nmon with the pin it serves; being operated from a repo that is not the package's source also makes a one-time Actions-access grant possible where nmon's GITHUB_TOKEN was denied. renovate.json points version discovery for the pin back at MCR (the mirror would never propose an upgrade). Co-Authored-By: Claude Fable 5 --- .github/workflows/mirror-playwright.yml | 69 +++++++ .github/workflows/playwright-test.yml | 229 ++++++++++++++++++++++++ README.md | 62 ++++++- renovate.json | 14 ++ 4 files changed, 372 insertions(+), 2 deletions(-) create mode 100644 .github/workflows/mirror-playwright.yml create mode 100644 .github/workflows/playwright-test.yml create mode 100644 renovate.json diff --git a/.github/workflows/mirror-playwright.yml b/.github/workflows/mirror-playwright.yml new file mode 100644 index 0000000..e8c672d --- /dev/null +++ b/.github/workflows/mirror-playwright.yml @@ -0,0 +1,69 @@ +# Keep ghcr.io//playwright in step with the upstream image that +# playwright-test.yml pins. Moved here from nmon (nmon#125) when the pin moved: +# the tag's source of truth is this repo's playwright-test.yml, so the mirror +# that serves it is operated from here too. +# +# WHY A MIRROR AT ALL: mcr.microsoft.com measures ~2.2 MB/s to this site +# against ghcr.io's 192 MB/s. That pull was 391s of a ~640s test job, and +# cdn.playwright.dev — same slow path — blew playwright's 30s download timeout +# under CI contention and failed builds outright. See nmon#76. +# +# MANUAL ONLY — there is deliberately no schedule. In nmon, the job's +# GITHUB_TOKEN could not push to the ghcr package: being the package's +# Repository source (via the org.opencontainers.image.source label) confers +# READ only, and the Actions-access panel will not offer the source repo. +# Proven, not assumed — a real run pulled for 7m21s and then died on +# `DENIED: permission_denied: write_package`. THIS repo is not the package's +# source, so the panel CAN grant it Actions access (package settings → +# Manage Actions access → add cshuttle/workflows, role Write) — until a human +# does that one-time step, a bump is mirrored by hand: +# crane pull --platform linux/amd64 mcr.microsoft.com/playwright: /tmp/pw.tar +# crane push /tmp/pw.tar ghcr.io/cshuttle/playwright: +# Nothing silently rots either way: playwright-test.yml's lockstep job fails +# any PR whose tag is not mirrored, so the reminder arrives before the merge +# rather than in a scheduled run nobody watches. +name: mirror-playwright +on: + workflow_dispatch: + inputs: + tag: + description: "Playwright image tag (blank = whatever playwright-test.yml pins)" + required: false + +jobs: + # One source of truth for the tag: playwright-test.yml's playwright-image + # default. Passing a tag by hand is for backfilling an older one, not normal + # operation — otherwise the mirror and its consumer drift and the failure is + # an ImagePullBackOff, which reads like a broken registry rather than a + # stale pin. + resolve: + name: resolve tag + runs-on: arc-workflows + outputs: + tag: ${{ steps.tag.outputs.tag }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - id: tag + run: | + if [ -n "${{ inputs.tag }}" ]; then + TAG="${{ inputs.tag }}" + else + TAG=$(grep -oP 'default:\s*\S*playwright:\K[^\s"'"'"']+' .github/workflows/playwright-test.yml | head -1) + fi + [ -n "$TAG" ] || { echo "could not resolve a playwright tag from playwright-test.yml"; exit 1; } + echo "tag=$TAG" >> "$GITHUB_OUTPUT" + echo "mirroring playwright:$TAG" + + mirror: + needs: resolve + permissions: + contents: read + packages: write + # Local ref, not @tag: the mirror mechanics and the pin they serve should + # move together in one commit here. + uses: ./.github/workflows/mirror-image.yml + with: + source: mcr.microsoft.com/playwright + destination: ghcr.io/${{ github.repository_owner }}/playwright + tag: ${{ needs.resolve.outputs.tag }} + runner: arc-workflows diff --git a/.github/workflows/playwright-test.yml b/.github/workflows/playwright-test.yml new file mode 100644 index 0000000..181a7b2 --- /dev/null +++ b/.github/workflows/playwright-test.yml @@ -0,0 +1,229 @@ +# The estate's Playwright CI (Homelab-Skills ADR 0017, extracted from nmon — +# nmon#76/#78/#79 and nmon#125). One estate-wide Playwright pin lives HERE; +# consumers call this workflow and pin @playwright/test to the same version. +# +# Two jobs, deliberately: +# lockstep — asserts the version chain has not drifted BEFORE the tests run: +# the caller's npm pin, this workflow's image tag, and the ghcr +# mirror's content must all agree. Each breaks silently and late +# on its own (see the job comment). +# test — the caller's suites inside the pinned browser container. +name: playwright-test + +on: + workflow_call: + inputs: + runner: + description: >- + The caller's ARC runner label (arc-). The test job is a + `container:` job, so the scale set must run containerMode: dind and + carry an ephemeral-storage request for the image (see arc-nmon in + cshuttle/main, arc-runners-appset.yaml) — a bare runner has no + Docker daemon and the job queues forever or dies at image pull. + required: true + type: string + unit-command: + description: >- + Optional non-browser suite run in the same container before the e2e + suite; empty = skipped. When set, BOTH suites run even if this one + fails — a red unit test and a red e2e are different diagnoses, and + stopping at the first hides the second. + required: false + type: string + default: "" + e2e-command: + description: The Playwright suite command. + required: false + type: string + default: npm run test:e2e + playwright-image: + description: >- + The estate Playwright pin (ADR 0017). Override only to debug this + workflow itself — a consumer passing its own image has left the + estate convention, and lockstep will hold it red against its own + package.json anyway. + required: false + type: string + # The MIRROR, not mcr.microsoft.com — mcr measures ~2.2 MB/s to this + # site against ghcr's 192 MB/s; the MCR pull was 391s of a ~640s job + # (nmon#76). ghcr.io/cshuttle/playwright is an amd64 copy of the same + # digest, pushed by this repo's mirror-playwright.yml. + # + # The renovate: annotation is LOAD-BEARING, not decoration. Mirroring + # cut us off from upstream discovery — Renovate would otherwise watch + # a registry we control that only ever contains what we push, and so + # would never propose an upgrade. This points version lookup back at + # MCR (this repo's renovate.json custom manager reads this exact + # comment+default pair) while the pull still comes from the mirror — + # which also means a bump is only proposed once a matching image + # REALLY exists (npm publishes ahead of the image; that mismatch is + # what broke CI in nmon#76). + # renovate: datasource=docker depName=mcr.microsoft.com/playwright + default: ghcr.io/cshuttle/playwright:v1.62.1-noble + +jobs: + # Assert the playwright version chain has not drifted. Three things must + # agree, and each breaks silently and late on its own: + # - caller npm pin ahead of the image -> `playwright install` downloads + # from a Microsoft CDN this site cannot reliably reach; on 2026-07-26 it + # timed out and failed CI outright (nmon#76). + # - tag not mirrored -> ImagePullBackOff that reads like a broken registry + # rather than a stale pin. + # - mirror != upstream -> we quietly test against something that is not + # the image we think it is. + # Deliberately cheap: manifest HEADs and GETs only, no image pull, no crane, + # no container. It runs in seconds so it can gate every PR without adding to + # the queue it exists to protect. + lockstep: + name: lockstep + runs-on: ${{ inputs.runner }} + env: + PLAYWRIGHT_IMAGE: ${{ inputs.playwright-image }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9 + with: + script: | + const fs = require('fs'); + const fail = []; + + // Must match the depName in the renovate: annotation on this + // workflow's playwright-image default — they live a page apart in + // the same file, which is as close as a job can get to reading + // its own workflow source. + const upstream = 'mcr.microsoft.com/playwright'; + + // ---- what the caller and this workflow say -------------------- + const pkg = JSON.parse(fs.readFileSync('package.json', 'utf8')); + const lock = JSON.parse(fs.readFileSync('package-lock.json', 'utf8')); + + const npmVer = (pkg.devDependencies || {})['@playwright/test']; + const lockVer = lock.packages['node_modules/@playwright/test']?.version; + if (!npmVer) { core.setFailed('no @playwright/test in devDependencies — the convention pins it exactly (see cshuttle/workflows README)'); return; } + + const img = process.env.PLAYWRIGHT_IMAGE.match(/^(\S+?):(\S+)$/); + if (!img) { core.setFailed(`could not parse image ref "${process.env.PLAYWRIGHT_IMAGE}"`); return; } + const [, mirrorRepo, tag] = img; + // v1.62.1-noble -> 1.62.1 + const tagVer = tag.replace(/^v/, '').split('-')[0]; + + core.info(`npm ${npmVer} | lock ${lockVer} | image ${mirrorRepo}:${tag} | upstream ${upstream}`); + + // ---- 1. the npm pin is exact and matches the lock ------------- + if (!/^\d+\.\d+\.\d+$/.test(npmVer)) + fail.push(`@playwright/test must be pinned exactly, found "${npmVer}". A range lets npm drift ahead of the image.`); + if (npmVer !== lockVer) + fail.push(`package.json (${npmVer}) and package-lock.json (${lockVer}) disagree`); + + // ---- 2. the npm pin matches the image tag --------------------- + if (npmVer !== tagVer) + fail.push(`@playwright/test ${npmVer} != image tag ${tag} (${tagVer}). ` + + `They must move together, or the browser is downloaded at test time.`); + + // ---- registry helpers ---------------------------------------- + const ACCEPT = [ + 'application/vnd.oci.image.index.v1+json', + 'application/vnd.docker.distribution.manifest.list.v2+json', + 'application/vnd.oci.image.manifest.v1+json', + 'application/vnd.docker.distribution.manifest.v2+json', + ].join(','); + + async function ghcrToken(repoPath) { + const r = await fetch(`https://ghcr.io/token?scope=repository:${repoPath}:pull&service=ghcr.io`); + if (!r.ok) throw new Error(`ghcr token: ${r.status}`); + return (await r.json()).token; + } + async function manifest(host, repoPath, ref, token) { + const h = { Accept: ACCEPT }; + if (token) h.Authorization = `Bearer ${token}`; + const r = await fetch(`https://${host}/v2/${repoPath}/manifests/${ref}`, { headers: h }); + if (!r.ok) return { status: r.status }; + return { status: r.status, body: await r.json() }; + } + // An index has to be resolved to the amd64 child; the mirror is + // single-arch already, so this is a no-op there. + async function amd64Layers(host, repoPath, ref, token) { + let m = await manifest(host, repoPath, ref, token); + if (m.status !== 200) return { status: m.status }; + if (m.body.manifests) { + const child = m.body.manifests.find( + (x) => x.platform?.architecture === 'amd64' && x.platform?.os === 'linux'); + if (!child) return { status: 'no amd64 in index' }; + m = await manifest(host, repoPath, child.digest, token); + if (m.status !== 200) return { status: m.status }; + } + return { status: 200, layers: m.body.layers.map((l) => l.digest) }; + } + + // ---- 3. the mirror has the tag, with upstream's layers --------- + const [mHost, ...mRest] = mirrorRepo.split('/'); + const mPath = mRest.join('/'); + const [uHost, ...uRest] = upstream.split('/'); + const uPath = uRest.join('/'); + + const mine = await amd64Layers(mHost, mPath, tag, await ghcrToken(mPath)); + if (mine.status !== 200) { + fail.push(`mirror ${mirrorRepo}:${tag} not pullable (${mine.status}). ` + + `Run cshuttle/workflows' mirror-playwright workflow (or crane by hand) before merging a bump.`); + } else { + const theirs = await amd64Layers(uHost, uPath, tag, null); + if (theirs.status !== 200) { + // Upstream being unreachable is not the estate drifting. + core.warning(`could not read ${upstream}:${tag} (${theirs.status}) — skipped the content comparison`); + } else { + // Compare LAYERS, not the manifest digest: the mirror carries + // extra OCI labels, so its manifest and config digests will + // never equal upstream's even when the content is identical. + const a = JSON.stringify(mine.layers); + const b = JSON.stringify(theirs.layers); + if (a !== b) + fail.push(`mirror content differs from ${upstream}:${tag}\n mirror: ${a}\n upstream: ${b}`); + else + core.info(`mirror matches upstream: ${mine.layers.length} identical layers`); + } + } + + if (fail.length) core.setFailed('lockstep drift:\n- ' + fail.join('\n- ')); + else core.notice(`lockstep OK — playwright ${npmVer} across npm, image tag and mirror`); + + test: + needs: lockstep + runs-on: ${{ inputs.runner }} + # The image is pinned for its OS libraries (Chromium's system deps — the + # bare runner has neither root nor apt, so they cannot be installed + # there); the BROWSER ships inside it at PLAYWRIGHT_BROWSERS_PATH. The + # PUBLIC package needs no credentials and no packages: scope. If it is + # ever made private again, both come back — and the failure will be a 403 + # on the manifest HEAD, which reads like the image is missing rather than + # like a permissions problem. That misdirection cost real time (nmon#76). + container: ${{ inputs.playwright-image }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - run: npm ci + # A NO-OP by design: lockstep has already proven @playwright/test + # matches the image, so the browser is present and this exits in about + # a second without touching the network. Left in place so that if the + # chain ever drifts past lockstep, the failure is a slow install rather + # than a confusing "executable doesn't exist" at test time. + - run: npx playwright install chromium + - if: inputs.unit-command != '' + run: ${{ inputs.unit-command }} + id: unit + continue-on-error: true + - run: ${{ inputs.e2e-command }} + id: e2e + continue-on-error: true + - name: fail if either suite failed + if: steps.unit.outcome == 'failure' || steps.e2e.outcome == 'failure' + run: | + echo "unit: ${{ steps.unit.outcome }}" + echo "e2e: ${{ steps.e2e.outcome }}" + exit 1 + - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + if: failure() + with: + name: playwright-report + path: | + playwright-report/ + test-results/ + retention-days: 7 diff --git a/README.md b/README.md index 9393078..50d48da 100644 --- a/README.md +++ b/README.md @@ -212,8 +212,66 @@ Notes: need for `credentials:` on the consumer's `container:`. - The consumer must keep its own pin (image tag, and any client library version that must match it) in step — this workflow mirrors, it does not - reconcile. See `cshuttle/nmon` `.github/workflows/lockstep.yml` for one way - to enforce that. + reconcile. See the `lockstep` job in `playwright-test.yml` (below) for one + way to enforce that. + +### `playwright-test.yml` + +The estate's browser testing convention (Homelab-Skills ADR 0017): one +Playwright pin for the whole estate lives in this workflow's +`playwright-image` default, and consumers run their suites inside that +container on their own ARC runner. Extracted from `cshuttle/nmon` +(nmon#76/#78/#79/#125), which proved the pattern: the pinned ghcr mirror +avoids Microsoft's badly-peered registry, and a `lockstep` job holds every PR +red until the caller's `@playwright/test` pin, this workflow's image tag, and +the mirror's actual content agree — each of those drifts silently and late on +its own. + +```yaml +# .github/workflows/test.yml in the consuming repo +name: test +on: + pull_request: + schedule: + - cron: "23 5 * * 1" # weekly: catches the mirror being pruned or MCR re-tagging +permissions: + contents: read +jobs: + test: + uses: cshuttle/workflows/.github/workflows/playwright-test.yml@v1.7.0 + with: + runner: arc- + unit-command: npm run test:unit # optional; omit if e2e is the only suite +``` + +Adopting it in a repo means, one time: + +- **An ARC runner with dind.** The test job is a `container:` job; the repo's + scale set needs `containerMode: dind` and an ephemeral-storage request (see + `arc-nmon` in `cshuttle/main`, `arc-runners-appset.yaml`). +- **Pin `@playwright/test` exactly** in `devDependencies`, to the version the + image tag carries (`v1.62.1-noble` → `1.62.1`). Ranges fail lockstep: a + range lets npm drift ahead of the image and reintroduces a browser download + from a CDN this site cannot reliably reach. +- **Disable Renovate's playwright npm bumps** in the repo's `renovate.json` — + the image leads and npm follows (npm publishes ahead of the image; an + automatic npm bump breaks CI, nmon#76). When Renovate bumps this repo's + image pin and the consumer's `uses:` tag, update `package.json` in that same + PR; lockstep holds it red until they agree. + + ```json + { + "description": "The estate Playwright pin leads and npm follows (cshuttle/workflows playwright-test.yml; ADR 0017). Bump @playwright/test by hand in the same PR as the uses: tag bump — lockstep holds it red until they agree.", + "matchPackageNames": ["@playwright/test", "playwright", "playwright-core"], + "enabled": false + } + ``` + +On failure the caller gets a `playwright-report` artifact (report + traces, +7 days). Bumping the estate pin: Renovate proposes the image bump here once +MCR really has the tag; mirror it (`mirror-playwright.yml`, or crane by hand — +see that workflow's header), merge, cut a release, and Renovate walks the +consumers' `uses:` tags forward. ### `release-image.yml` diff --git a/renovate.json b/renovate.json new file mode 100644 index 0000000..2ffa29a --- /dev/null +++ b/renovate.json @@ -0,0 +1,14 @@ +{ + "$schema": "https://docs.renovatebot.com/renovate-schema.json", + "extends": ["github>cshuttle/renovate-config"], + "customManagers": [ + { + "customType": "regex", + "description": "Track the estate Playwright pin (playwright-test.yml's playwright-image default) against MCR even though consumers pull it from our ghcr mirror. Mirroring cut off upstream discovery: a registry we control only ever contains what we push, so nothing would ever propose an upgrade. Looking up MCR instead also means a bump is proposed ONLY once a matching image really exists — npm publishes ahead of the image, and that mismatch is what broke CI in nmon#76. The regex reads the `# renovate:` annotation plus the `default:` line under it; playwright-test.yml's lockstep job holds every consumer red until its @playwright/test pin follows.", + "managerFilePatterns": ["/^\\.github/workflows/playwright-test\\.yml$/"], + "matchStrings": [ + "# renovate: datasource=(?\\S+) depName=(?\\S+)\\s*\\n\\s*default:\\s*\\S+?:(?\\S+)" + ] + } + ] +} From 4a0751dc30edba8c3c86d74546cde90dfcafeecb Mon Sep 17 00:00:00 2001 From: Chris Shuttlesworth Date: Wed, 19 Aug 2026 12:12:09 -0400 Subject: [PATCH 2/2] ci(selftest): assert the playwright annotation and lockstep upstream agree MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review finding on #58: the reusable lockstep job hardcodes its upstream registry (it checks out the caller, so it cannot parse its own annotation the way nmon's old lockstep did) — this selftest job is what keeps the two lines from rotting apart silently. Co-Authored-By: Claude Fable 5 --- .github/workflows/selftest.yml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/.github/workflows/selftest.yml b/.github/workflows/selftest.yml index f4dbc21..65c7470 100644 --- a/.github/workflows/selftest.yml +++ b/.github/workflows/selftest.yml @@ -64,6 +64,23 @@ jobs: # bare runner image needs no extra package (no PyYAML here on purpose). run: python3 tests/test_komodo_pin.py + playwright-pin: + runs-on: arc-workflows + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - name: renovate annotation and lockstep upstream agree + # playwright-test.yml's lockstep job hardcodes its upstream registry: + # a reusable workflow checks out the CALLER's repo, so unlike nmon's + # old lockstep it cannot parse its own renovate: annotation at run + # time. This is the check that keeps the two lines — a page apart in + # the same file — from rotting apart silently. + run: | + wf=.github/workflows/playwright-test.yml + ann=$(grep -oP '#\s*renovate:\s*datasource=docker\s+depName=\K\S+' "$wf") + const=$(grep -oP "const upstream = '\K[^']+" "$wf") + echo "annotation: ${ann:-} | lockstep const: ${const:-}" + [ -n "$ann" ] && [ "$ann" = "$const" ] || { echo "renovate annotation and lockstep upstream disagree in $wf"; exit 1; } + shellcheck: runs-on: arc-workflows steps: