From 16caea65a5b93633d6fb3e01e1c98a98ec49d040 Mon Sep 17 00:00:00 2001 From: Chris Shuttlesworth Date: Wed, 19 Aug 2026 14:17:46 -0400 Subject: [PATCH 1/2] =?UTF-8?q?docs:=20the=20consumer=20pattern=20?= =?UTF-8?q?=E2=80=94=20the=20in-repo=20half=20of=20the=20browser=20testing?= =?UTF-8?q?=20convention?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The reusable workflow's README covers CI wiring and the pin chain; nothing covered what a consuming repo looks like on the inside, so each adopter would reverse-engineer nmon and diverge. docs/playwright-consumer.md is that half: the root-manifest contract lockstep assumes, the exact-pin + renovate-rule npm conventions, the config baseline (derived port, vite preview of BUILT assets, report paths matching the artifact glob), the test-layout split, the starter smoke (a selector only the booted app produces, plus no page errors — never a bare HTTP 200), and the subdirectory-frontend shape. nmon stays the living reference throughout. Closes #61 Co-Authored-By: Claude Fable 5 --- README.md | 5 +- docs/playwright-consumer.md | 203 ++++++++++++++++++++++++++++++++++++ 2 files changed, 207 insertions(+), 1 deletion(-) create mode 100644 docs/playwright-consumer.md diff --git a/README.md b/README.md index 50d48da..0907097 100644 --- a/README.md +++ b/README.md @@ -244,7 +244,10 @@ jobs: unit-command: npm run test:unit # optional; omit if e2e is the only suite ``` -Adopting it in a repo means, one time: +This section is the CI half only. What the repo looks like on the **inside** — +the Playwright config baseline, how the app-under-test boots, test layout, and +the starter smoke — is [docs/playwright-consumer.md](docs/playwright-consumer.md), +the consumer half of the convention. Adopting it in a repo means, one time: - **An ARC runner with dind.** The test job is a `container:` job; the repo's scale set needs `containerMode: dind` and an ephemeral-storage request (see diff --git a/docs/playwright-consumer.md b/docs/playwright-consumer.md new file mode 100644 index 0000000..ad0b10d --- /dev/null +++ b/docs/playwright-consumer.md @@ -0,0 +1,203 @@ +# Playwright in a consumer repo — the in-repo half of the browser testing convention + +The estate's browser testing convention (Homelab-Skills ADR 0017) has two +halves. The CI half — calling `playwright-test.yml`, the estate image pin and +lockstep, the ARC runner prerequisites — lives in the +[README](../README.md#playwright-testyml) and is deliberately not repeated +here. This document is the other half: what the inside of a consuming repo +looks like, so adopting browser testing is a copy from here rather than a +reverse-engineering of a sibling. + +**`cshuttle/nmon` is the living reference.** Where this document and nmon +disagree, nmon has probably grown a lesson this document has not caught up +with — read its `playwright.config.js` header comments before assuming the +divergence is drift. This document exists so consumers make the same choices; +nmon exists to prove they work. + +## What the reusable workflow assumes about your repo + +`playwright-test.yml` checks out your repo and, at its **root**: + +- reads `package.json` and `package-lock.json` (the `lockstep` job — so the + Playwright pin must live in the **root** manifest, even when the app itself + lives in a subdirectory; see the last section), +- runs `npm ci`, then your `e2e-command` (default `npm run test:e2e`), +- on failure, uploads `playwright-report/` and `test-results/` as the + `playwright-report` artifact. + +Everything below exists to satisfy that contract. + +## npm conventions — the pin, the lockfile, the Renovate rule + +**Pin `@playwright/test` exactly**, to the version the estate image tag +carries (`ghcr.io/cshuttle/playwright:v1.62.1-noble` → `1.62.1`): + +```json +"devDependencies": { + "@playwright/test": "1.62.1" +} +``` + +No `^`, no `~`. A range lets npm drift ahead of the image, and then +`playwright install` reaches for `cdn.playwright.dev` — a Microsoft CDN this +site measures at 2.2 MB/s, which timed out and failed CI outright on +2026-07-26 (nmon#76). The lockfile must agree with the manifest (run +`npm install` after editing, commit both); lockstep fails on a range, on a +manifest/lockfile mismatch, and on a pin that differs from the image tag. + +**Disable Renovate's npm-side Playwright bumps** in the repo's +`renovate.json` — the image leads and npm follows, because npm publishes +ahead of the image: + +```json +{ + "description": "The estate Playwright pin leads and npm follows (cshuttle/workflows playwright-test.yml; ADR 0017). Bump @playwright/test by hand in the same PR as the uses: tag bump — lockstep holds it red until they agree.", + "matchPackageNames": ["@playwright/test", "playwright", "playwright-core"], + "enabled": false +} +``` + +When Renovate walks your `uses:` tag forward after an estate pin bump, update +`package.json` and the lockfile in that same PR. + +On CWS, run `npx playwright install chromium` once per machine for local and +agent loops (ADR 0017 — browsers co-located, never a remote endpoint). In CI +the browser ships inside the image; the workflow's `playwright install` step +is a deliberate no-op. + +## Playwright config baseline + +```ts +// playwright.config.ts +import { defineConfig, devices } from "@playwright/test"; +import { createHash } from "node:crypto"; +import { fileURLToPath } from "node:url"; +import path from "node:path"; + +// The port is DERIVED FROM THE CHECKOUT PATH, not fixed (nmon#47): two +// checkouts of one repo — git worktrees, which agent workflows create +// routinely — sharing a fixed port plus `reuseExistingServer` silently run +// one tree's suite against the OTHER tree's server. Hashing the root gives +// every worktree its own port; TEST_PORT still wins. +const ROOT = path.dirname(fileURLToPath(import.meta.url)); +const SLOT = + parseInt(createHash("sha1").update(ROOT).digest("hex").slice(0, 4), 16) % 900; +const PORT = Number(process.env.TEST_PORT || 4300 + SLOT); + +export default defineConfig({ + testDir: "./tests/e2e", + testMatch: "**/*.spec.ts", + forbidOnly: !!process.env.CI, + retries: 0, + // `html` writes playwright-report/, matching the workflow's failure + // artifact glob; screenshots land in test-results/ (the glob's other half). + reporter: process.env.CI ? [["line"], ["html", { open: "never" }]] : "list", + use: { + baseURL: `http://127.0.0.1:${PORT}`, + screenshot: "only-on-failure", + }, + projects: [{ name: "chromium", use: { ...devices["Desktop Chrome"] } }], + webServer: { + command: `vite preview --port ${PORT} --strictPort`, + url: `http://127.0.0.1:${PORT}/`, + reuseExistingServer: !process.env.CI, + }, +}); +``` + +The `webServer` boots **built assets** via `vite preview`, so the script pair +is: + +```json +"scripts": { + "test:e2e": "npm run build && playwright test" +} +``` + +Testing `vite dev` instead would pass a production bundle that is broken — +the dev server transforms modules on the fly and masks build-only failures. +(nmon differs here by design: its app is unbundled, so its `webServer` runs +the same `server.js` production serves. Same principle — test what ships.) + +Do not move the report paths: the workflow uploads `playwright-report/` and +`test-results/` on failure, and output written anywhere else silently +vanishes from the artifact. + +## Test directory layout + +``` +tests/e2e/*.spec.ts Playwright suites (testDir above) +tests/… everything else — unit suites, other runners +``` + +Keep the Playwright glob disjoint from any other runner's. Playwright's +default `testMatch` also collects `**/*.test.{js,ts}`, which sweeps up +`node --test` and vitest files that have no `page` fixture and fail instantly +under the Playwright runner (nmon splits `*.spec.js` / `*.test.js` for +exactly this reason). The explicit `testDir` + `testMatch` pair above makes +the split structural. + +## The starter smoke + +The first test in every adopting repo is a shell-render smoke: + +```ts +// tests/e2e/smoke.spec.ts +import { expect, test } from "@playwright/test"; + +test("the app shell renders", async ({ page }) => { + const errors: Error[] = []; + page.on("pageerror", (e) => errors.push(e)); + + await page.goto("/"); + + // A selector only the BOOTED APP produces — the static mount point + // (`
`) is served even when the bundle throws at import + // time, so asserting it proves nothing. Assert a child the module graph + // must execute to create; prefer the most distinctive stable element the + // app owns (a brand block, an app bar) over a bare `#root > *`. + await expect(page.locator("#root > *").first()).toBeVisible(); + + // No uncaught exceptions. THIS is the assertion that catches a + // white-screen bundle error — a bare HTTP 200 (or a title check) passes + // while the page renders nothing. Handled fetch failures (an absent + // backend logging to console) do not trip it; only real page errors do. + expect(errors, errors.map(String).join("\n")).toEqual([]); +}); +``` + +Never assert a bare HTTP 200. Every failure mode worth catching — a manifest +that didn't copy, a `const` hoisted above its definition, a module that +throws at import — serves its files with a 200 and renders a white screen. +The selector proves the app executed; the `pageerror` listener proves it +executed cleanly. + +The smoke needs no backend. A frontend whose API is absent should still +execute its module graph and render *something it owns* (a shell, an error +state); assert that. Route-mocking the backend (nmon mocks every Netdata +call) is the next step when the suite grows past the smoke, not a smoke +prerequisite. + +## When the app lives in a subdirectory + +Lockstep reads the **root** `package.json`, and the workflow's `npm ci` runs +at the root — so a repo whose frontend lives in a subdirectory (Topology: +`frontend/`) keeps a root-level e2e harness: + +- root `package.json` + lockfile carrying only `@playwright/test` (the exact + pin) and the e2e script; `playwright.config.ts` and `tests/e2e/` at the + root beside it, +- the e2e script reaches into the app directory to install, build and + preview: + +```json +"scripts": { + "test:e2e": "npm --prefix frontend ci && npm --prefix frontend run build && playwright test" +} +``` + +with the config's `webServer.command` set to +`npm --prefix frontend run preview -- --port … --strictPort`. The app's own +`package.json` stays untouched apart from a `preview` script; do not convert +the repo to npm workspaces just for this — the existing CI's `working-directory` +assumptions would all move. From 8b61d64adeaec2c2b4cf06f5cf72785ec2c6b87b Mon Sep 17 00:00:00 2001 From: Chris Shuttlesworth Date: Wed, 19 Aug 2026 14:58:53 -0400 Subject: [PATCH 2/2] docs(playwright-consumer): --host 127.0.0.1 is load-bearing in the container MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Surfaced by the Atlas/Topology seeds: vite preview's default localhost bind inside the job container is IPv6-only (::1) while the webServer probe dials 127.0.0.1, so the suite times out on a server that is up — and passes on a dev host, which binds both families. Co-Authored-By: Claude Fable 5 --- docs/playwright-consumer.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/playwright-consumer.md b/docs/playwright-consumer.md index ad0b10d..7bd4f48 100644 --- a/docs/playwright-consumer.md +++ b/docs/playwright-consumer.md @@ -98,7 +98,12 @@ export default defineConfig({ }, projects: [{ name: "chromium", use: { ...devices["Desktop Chrome"] } }], webServer: { - command: `vite preview --port ${PORT} --strictPort`, + // --host 127.0.0.1 is LOAD-BEARING in CI: inside the test container + // vite's default localhost bind is IPv6-only (::1), and the url probe + // below dials 127.0.0.1 — without the flag the suite times out waiting + // for a server that is actually up (it passes on a dev host, where + // localhost binds both families, which is what makes it a trap). + command: `vite preview --port ${PORT} --strictPort --host 127.0.0.1`, url: `http://127.0.0.1:${PORT}/`, reuseExistingServer: !process.env.CI, }, @@ -197,7 +202,8 @@ at the root — so a repo whose frontend lives in a subdirectory (Topology: ``` with the config's `webServer.command` set to -`npm --prefix frontend run preview -- --port … --strictPort`. The app's own +`npm --prefix frontend run preview -- --port … --strictPort --host 127.0.0.1`. +The app's own `package.json` stays untouched apart from a `preview` script; do not convert the repo to npm workspaces just for this — the existing CI's `working-directory` assumptions would all move.