Skip to content

Replace pa11y-ci with axe-core audit reusing Playwright - #242

Merged
shouze merged 6 commits into
mainfrom
shouze-lightweight-a11y
Oct 28, 2026
Merged

shouze merged 6 commits into
mainfrom
shouze-lightweight-a11y

Conversation

@shouze

@shouze shouze commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Closes #241.

Replaces pa11y-ci (Pa11y + Puppeteer + HTML_CodeSniffer) with the official @axe-core/playwright adapter, which reuses the @playwright/test dependency we already have and the runner's system Chrome. No extra browser download.

  • scripts/a11y.ts: audits every sitemap URL with one shared browser and 2 workers. Uses explicit WCAG 2.0/2.1/2.2 A/AA tags, with no fixed 1.5 s wait and no blanket ignores. Violations, HTTP errors, redirects, timeouts and empty or invalid sitemaps all exit with a nonzero status.
  • scripts/a11y-utils.ts (+ unit tests) and scripts/a11y.pw.ts: these harness regression tests cover pass, a deliberate violation (image-alt), an empty sitemap, malformed XML, a missing <loc>, an out-of-base URL, an HTTP 404 and a sitemap 500. Reports are kept on failure.
  • Workflow: produces the JSON + Markdown artifacts, a job summary and a sticky PR comment (same-repo PRs only, so forks are safe). It triggers on changes to the scripts, manifest, lockfile and the shared preview action.
  • Theme fixes for real axe findings: the DocSearch accessible name now matches its visible label (including on mobile), and labelled Mermaid containers get role="group".
  • Docs: no product-docs page (the audit only covers the docs site; benchmark in Replace pa11y-ci with a lightweight axe-core audit reusing Playwright #241, manual-review checklist in CONTRIBUTING), updates to README, CONTRIBUTING, AGENTS.md and the skills/instructions (documentation, github-actions, feature, release). There are no repo-local custom agents to update.

Measurements (same macOS machine, same built site, system Chrome)

pa11y-ci (main) axe-core (this PR)
node_modules (bun install --ignore-scripts) 610,532 KB 575,496 KB (−34 MB)
bun.lock package entries 858 788 (−70)
Full sitemap (37 pages) 293.5 s 116.3 s (−60 %)
Result 37/37 pass 37/37 pass, 0 violations

These are local measurements only; hosted CI timings will appear on this PR's run. The other lockfile + lines are only Lighthouse's existing Puppeteer deps being re-hoisted, with the same versions. Note that axe and HTML_CodeSniffer have different rule sets, and a clean automated audit does not mean WCAG certification.

How did you verify your code works?

  • bun test: 1039 pass, 0 fail
  • bun run lint: 0 errors (no new warnings)
  • bun run format:check: pass
  • bun run knip: clean
  • bun run docs:build:a11y: success
  • bun run docs:a11y: 37/37 pages, 0 violations
  • bun run docs:test:a11y: 10/10 pass
  • Baseline pa11y-ci --config .pa11yci.json --sitemap … on main: 37/37 pass, 293.5 s

Copilot AI balanced review requested due to automatic review settings October 28, 2026 14:05
Closes #241

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@shouze
shouze force-pushed the shouze-lightweight-a11y branch from b0eb6f3 to 885f642 Compare October 28, 2026 14:07
@github-actions

Copy link
Copy Markdown
Contributor

Coverage after merging shouze-lightweight-a11y into main will be

96.59%

Coverage Report
FileStmtsBranchesFuncsLinesUncovered Lines
scripts
   a11y-utils.ts100%100%100%100%
src
   aggregate.ts100%100%100%100%
   api-utils.ts92.80%100%91.67%92.92%101–103, 165, 86–87, 92–93
   api.ts94.96%100%100%94.42%340–343, 405, 422, 53, 64–67, 69
   cache.ts94.37%100%100%93.94%139–141, 39
   completions.ts99.44%100%100%99.39%268
   gh-cli.ts100%100%100%100%
   group.ts98.37%100%98.33%98.37%1105, 263–265, 525, 531, 607, 865, 980
   output.ts99.31%100%95.83%99.62%89
   regex.ts99.37%100%100%99.32%360
   render.ts90.35%100%88.24%90.43%175, 177–178, 201–206, 208–210, 213, 234–235, 427–428, 528–532, 562–569, 571–579, 582–584
   scroll-cooldown.ts100%100%100%100%
   style.ts100%100%100%100%
   upgrade.ts86.73%100%92.31%86.36%128, 131, 133–134, 153–154, 167–169, 188–193, 195, 198–202, 209, 214, 250–253
src/render
   filter-match.ts97.22%100%92.31%100%
   filter.ts100%100%100%100%
   highlight.ts96.53%100%90.40%99.28%284–285
   layout-constants.ts100%100%100%100%
   mouse-hit.ts100%100%100%100%
   mouse.ts100%100%100%100%
   rows.ts99.34%100%100%99.29%239
   selection.ts100%100%100%100%
   summary.ts100%100%100%100%
   team-pick.ts100%100%100%100%
   terminal.ts100%100%100%100%

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

🔦 Lighthouse Report

Page ⚡ Perf ♿ A11y 🛡️ BP 🔍 SEO Report
/github-code-search/ 🟢 100 (≥96) 🟢 100 (≥99) 🟢 100 (≥99) 🟢 100 (≥99) 🔗 view
/github-code-search/getting-started/ 🟢 99 (≥96) 🟢 100 (≥99) 🟢 100 (≥99) 🟢 100 (≥99) 🔗 view

Thresholds: Perf ≥ 96 · A11y ≥ 99 · BP ≥ 99 · SEO ≥ 99
commit daf5240 · full workflow run

This comment was marked as resolved.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Accessibility audit — axe-core / WCAG 2.2 AA

Pages scanned: 36.

Page Violations Incomplete rules Status
/github-code-search/ 0 2 PASS
/github-code-search/architecture/components.html 0 1 PASS
/github-code-search/architecture/containers.html 0 1 PASS
/github-code-search/architecture/overview.html 0 1 PASS
/github-code-search/blog/ 0 1 PASS
/github-code-search/blog/release-v1-0-0.html 0 1 PASS
/github-code-search/blog/release-v1-10-0.html 0 1 PASS
/github-code-search/blog/release-v1-11-0.html 0 1 PASS
/github-code-search/blog/release-v1-12-0.html 0 1 PASS
/github-code-search/blog/release-v1-3-0.html 0 1 PASS
/github-code-search/blog/release-v1-4-0.html 0 1 PASS
/github-code-search/blog/release-v1-5-0.html 0 1 PASS
/github-code-search/blog/release-v1-6-0.html 0 1 PASS
/github-code-search/blog/release-v1-6-1.html 0 1 PASS
/github-code-search/blog/release-v1-7-0.html 0 1 PASS
/github-code-search/blog/release-v1-8-0.html 0 1 PASS
/github-code-search/blog/release-v1-8-1.html 0 1 PASS
/github-code-search/blog/release-v1-8-2.html 0 1 PASS
/github-code-search/blog/release-v1-8-3.html 0 1 PASS
/github-code-search/blog/release-v1-9-0.html 0 1 PASS
/github-code-search/blog/release-v2-0-0.html 0 1 PASS
/github-code-search/getting-started/ 0 1 PASS
/github-code-search/getting-started/first-search.html 0 1 PASS
/github-code-search/getting-started/installation.html 0 1 PASS
/github-code-search/reference/cli-options.html 0 1 PASS
/github-code-search/reference/environment.html 0 1 PASS
/github-code-search/reference/github-api-limits.html 0 1 PASS
/github-code-search/reference/keyboard-shortcuts.html 0 1 PASS
/github-code-search/usage/ 0 1 PASS
/github-code-search/usage/filtering.html 0 1 PASS
/github-code-search/usage/interactive-mode.html 0 1 PASS
/github-code-search/usage/non-interactive-mode.html 0 1 PASS
/github-code-search/usage/output-formats.html 0 1 PASS
/github-code-search/usage/search-syntax.html 0 1 PASS
/github-code-search/usage/team-grouping.html 0 1 PASS
/github-code-search/usage/upgrade.html 0 1 PASS

Incomplete results require manual review; zero violations is not WCAG certification.
Full rule details, affected nodes and operational errors are in a11y-report.json.

@shouze

This comment was marked as outdated.

vitepress preview Brotli-compresses the 4 MB Mermaid chunk on every
request (~5 s per page). Request identity encoding in the audit context
and scale workers with available CPUs (up to four).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

Coverage after merging shouze-lightweight-a11y into main will be

96.59%

Coverage Report
FileStmtsBranchesFuncsLinesUncovered Lines
scripts
   a11y-utils.ts100%100%100%100%
src
   aggregate.ts100%100%100%100%
   api-utils.ts92.80%100%91.67%92.92%101–103, 165, 86–87, 92–93
   api.ts94.96%100%100%94.42%340–343, 405, 422, 53, 64–67, 69
   cache.ts94.37%100%100%93.94%139–141, 39
   completions.ts99.44%100%100%99.39%268
   gh-cli.ts100%100%100%100%
   group.ts98.37%100%98.33%98.37%1105, 263–265, 525, 531, 607, 865, 980
   output.ts99.31%100%95.83%99.62%89
   regex.ts99.37%100%100%99.32%360
   render.ts90.35%100%88.24%90.43%175, 177–178, 201–206, 208–210, 213, 234–235, 427–428, 528–532, 562–569, 571–579, 582–584
   scroll-cooldown.ts100%100%100%100%
   style.ts100%100%100%100%
   upgrade.ts86.73%100%92.31%86.36%128, 131, 133–134, 153–154, 167–169, 188–193, 195, 198–202, 209, 214, 250–253
src/render
   filter-match.ts97.22%100%92.31%100%
   filter.ts100%100%100%100%
   highlight.ts96.53%100%90.40%99.28%284–285
   layout-constants.ts100%100%100%100%
   mouse-hit.ts100%100%100%100%
   mouse.ts100%100%100%100%
   rows.ts99.34%100%100%99.29%239
   selection.ts100%100%100%100%
   summary.ts100%100%100%100%
   team-pick.ts100%100%100%100%
   terminal.ts100%100%100%100%

@shouze

This comment was marked as outdated.

Keep the Vue runtime and Vite helpers in the framework chunk so pages
without diagrams no longer download the 4 MB Mermaid chunk, import the
Mermaid renderer only when a diagram is present, reuse one browser
context per audit worker, and run the fixture harness in a parallel job.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

Coverage after merging shouze-lightweight-a11y into main will be

96.59%

Coverage Report
FileStmtsBranchesFuncsLinesUncovered Lines
scripts
   a11y-utils.ts100%100%100%100%
src
   aggregate.ts100%100%100%100%
   api-utils.ts92.80%100%91.67%92.92%101–103, 165, 86–87, 92–93
   api.ts94.96%100%100%94.42%340–343, 405, 422, 53, 64–67, 69
   cache.ts94.37%100%100%93.94%139–141, 39
   completions.ts99.44%100%100%99.39%268
   gh-cli.ts100%100%100%100%
   group.ts98.37%100%98.33%98.37%1105, 263–265, 525, 531, 607, 865, 980
   output.ts99.31%100%95.83%99.62%89
   regex.ts99.37%100%100%99.32%360
   render.ts90.35%100%88.24%90.43%175, 177–178, 201–206, 208–210, 213, 234–235, 427–428, 528–532, 562–569, 571–579, 582–584
   scroll-cooldown.ts100%100%100%100%
   style.ts100%100%100%100%
   upgrade.ts86.73%100%92.31%86.36%128, 131, 133–134, 153–154, 167–169, 188–193, 195, 198–202, 209, 214, 250–253
src/render
   filter-match.ts97.22%100%92.31%100%
   filter.ts100%100%100%100%
   highlight.ts96.53%100%90.40%99.28%284–285
   layout-constants.ts100%100%100%100%
   mouse-hit.ts100%100%100%100%
   mouse.ts100%100%100%100%
   rows.ts99.34%100%100%99.29%239
   selection.ts100%100%100%100%
   summary.ts100%100%100%100%
   team-pick.ts100%100%100%100%
   terminal.ts100%100%100%100%

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

Coverage after merging shouze-lightweight-a11y into main will be

96.59%

Coverage Report
FileStmtsBranchesFuncsLinesUncovered Lines
scripts
   a11y-utils.ts100%100%100%100%
src
   aggregate.ts100%100%100%100%
   api-utils.ts92.80%100%91.67%92.92%101–103, 165, 86–87, 92–93
   api.ts94.96%100%100%94.42%340–343, 405, 422, 53, 64–67, 69
   cache.ts94.37%100%100%93.94%139–141, 39
   completions.ts99.44%100%100%99.39%268
   gh-cli.ts100%100%100%100%
   group.ts98.37%100%98.33%98.37%1105, 263–265, 525, 531, 607, 865, 980
   output.ts99.31%100%95.83%99.62%89
   regex.ts99.37%100%100%99.32%360
   render.ts90.35%100%88.24%90.43%175, 177–178, 201–206, 208–210, 213, 234–235, 427–428, 528–532, 562–569, 571–579, 582–584
   scroll-cooldown.ts100%100%100%100%
   style.ts100%100%100%100%
   upgrade.ts86.73%100%92.31%86.36%128, 131, 133–134, 153–154, 167–169, 188–193, 195, 198–202, 209, 214, 250–253
src/render
   filter-match.ts97.22%100%92.31%100%
   filter.ts100%100%100%100%
   highlight.ts96.53%100%90.40%99.28%284–285
   layout-constants.ts100%100%100%100%
   mouse-hit.ts100%100%100%100%
   mouse.ts100%100%100%100%
   rows.ts99.34%100%100%99.29%239
   selection.ts100%100%100%100%
   summary.ts100%100%100%100%
   team-pick.ts100%100%100%100%
   terminal.ts100%100%100%100%

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@shouze

shouze commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

⚡ Second optimisation pass: audit job 108 s → 45 s (×2.4)

Step (CI, ubuntu-latest) Before After
Build docs 28 s 17–23 s ¹
Start preview 5 s 1 s
Harness + theme tests 20 s (serial) parallel job / alongside the audit
axe audit (37 pages) 41 s 17 s
WCAG audit job 108 s 45 s
Full workflow (wall time) ~113 s 50 s

Local audit: 14 s → 8 s.

Root causes found and fixed

  1. Every page downloaded ~5.4 MB of JS, Mermaid included. The custom manualChunks replaced VitePress's chunking, so Rollup put the Vue runtime inside the Mermaid chunk, and the theme imported the renderer statically. Fix: Vue + Vite helpers → framework chunk, renderer loaded with import() only when the page has a diagram. Pages without diagrams now modulepreload ~200 kB (a side benefit for real users too).
  2. Fresh browser context per page → no HTTP / V8 code cache. Each worker now reuses one context (recreated after an error), and axe collects only violations + incomplete.
  3. Pitfall avoided: AxeBuilder.options() replaces the runOnly set by withTags(). It is now called first, otherwise non-WCAG best-practice rules came back.
  4. Mermaid readiness adapted to lazy loading: wait until every visible diagram is rendered (off-screen ones stay as source, as before).
  5. CI:
    • The fixture tests (no build needed) run in a parallel harness job.
    • The theme tests run in the background alongside the audit, and their exit code is checked afterwards.
    • The preview action redirected its output: the background process kept stdout open, and the runner waited 5 s.
  6. Playwright: Accept-Encoding: identity across the whole config (on-the-fly Brotli compression from vitepress preview), which also benefits the responsive tests.

¹ The build stays dominated by single-threaded Rollup bundling of Mermaid. Minification and reportCompressedSize were measured and are negligible.

Verified locally:

  • Lazy loading on direct load and after client-side navigation to and from a diagram page.
  • Re-render when the colour scheme changes.
  • bun test, lint, format:check, knip, the 30 Playwright tests, and the audit with 0 violations.

Aikido MCP is not installed in my environment, so I could not run the scan.

@github-actions

Copy link
Copy Markdown
Contributor

Coverage after merging shouze-lightweight-a11y into main will be

96.59%

Coverage Report
FileStmtsBranchesFuncsLinesUncovered Lines
scripts
   a11y-utils.ts100%100%100%100%
src
   aggregate.ts100%100%100%100%
   api-utils.ts92.80%100%91.67%92.92%101–103, 165, 86–87, 92–93
   api.ts94.96%100%100%94.42%340–343, 405, 422, 53, 64–67, 69
   cache.ts94.37%100%100%93.94%139–141, 39
   completions.ts99.44%100%100%99.39%268
   gh-cli.ts100%100%100%100%
   group.ts98.37%100%98.33%98.37%1105, 263–265, 525, 531, 607, 865, 980
   output.ts99.31%100%95.83%99.62%89
   regex.ts99.37%100%100%99.32%360
   render.ts90.35%100%88.24%90.43%175, 177–178, 201–206, 208–210, 213, 234–235, 427–428, 528–532, 562–569, 571–579, 582–584
   scroll-cooldown.ts100%100%100%100%
   style.ts100%100%100%100%
   upgrade.ts86.73%100%92.31%86.36%128, 131, 133–134, 153–154, 167–169, 188–193, 195, 198–202, 209, 214, 250–253
src/render
   filter-match.ts97.22%100%92.31%100%
   filter.ts100%100%100%100%
   highlight.ts96.53%100%90.40%99.28%284–285
   layout-constants.ts100%100%100%100%
   mouse-hit.ts100%100%100%100%
   mouse.ts100%100%100%100%
   rows.ts99.34%100%100%99.29%239
   selection.ts100%100%100%100%
   summary.ts100%100%100%100%
   team-pick.ts100%100%100%100%
   terminal.ts100%100%100%100%

This comment was marked as resolved.

Reject redirected sitemaps, skip the sticky comment on Dependabot runs,
wait for the destination heading in the Mermaid navigation test, and
remove the reference page: the audit covers the docs site, not the CLI.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

Coverage after merging shouze-lightweight-a11y into main will be

96.59%

Coverage Report
FileStmtsBranchesFuncsLinesUncovered Lines
scripts
   a11y-utils.ts100%100%100%100%
src
   aggregate.ts100%100%100%100%
   api-utils.ts92.80%100%91.67%92.92%101–103, 165, 86–87, 92–93
   api.ts94.96%100%100%94.42%340–343, 405, 422, 53, 64–67, 69
   cache.ts94.37%100%100%93.94%139–141, 39
   completions.ts99.44%100%100%99.39%268
   gh-cli.ts100%100%100%100%
   group.ts98.37%100%98.33%98.37%1105, 263–265, 525, 531, 607, 865, 980
   output.ts99.31%100%95.83%99.62%89
   regex.ts99.37%100%100%99.32%360
   render.ts90.35%100%88.24%90.43%175, 177–178, 201–206, 208–210, 213, 234–235, 427–428, 528–532, 562–569, 571–579, 582–584
   scroll-cooldown.ts100%100%100%100%
   style.ts100%100%100%100%
   upgrade.ts86.73%100%92.31%86.36%128, 131, 133–134, 153–154, 167–169, 188–193, 195, 198–202, 209, 214, 250–253
src/render
   filter-match.ts97.22%100%92.31%100%
   filter.ts100%100%100%100%
   highlight.ts96.53%100%90.40%99.28%284–285
   layout-constants.ts100%100%100%100%
   mouse-hit.ts100%100%100%100%
   mouse.ts100%100%100%100%
   rows.ts99.34%100%100%99.29%239
   selection.ts100%100%100%100%
   summary.ts100%100%100%100%
   team-pick.ts100%100%100%100%
   terminal.ts100%100%100%100%

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The required Aikido security scan could not be completed because its MCP server was unavailable, and browser-based validation was not rerun.

No open issues 🎉

3 resolved since last review

Review effort: Balanced

@shouze
shouze merged commit 8e0aa00 into main Oct 28, 2026
10 checks passed
@shouze
shouze deleted the shouze-lightweight-a11y branch October 28, 2026 20:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace pa11y-ci with a lightweight axe-core audit reusing Playwright

2 participants