Conversation
…change The suite drives a real VoiceOver instance one test at a time, so it cannot sit in the path of every PR the way the headless suites do. It is wired as a job condition rather than a `paths:` filter on the workflow: GitHub leaves the checks of a path-filtered workflow Pending forever, which blocks any PR required to pass it, whereas a skipped job reports success and does not. The workflow therefore always runs and the macOS job decides for itself. `should-run-voiceover.sh` answers a narrower question than `should-run-ci.sh`: not "could this affect @editorjs/editorjs" - true of almost every change here - but "could this alter what a screen reader announces". It watches the UI and paragraph sources, the bundle, the suite itself, the lockfile (@editorjs/ui-kit owns the popover roles), and its own CI wiring. It fails open on anything it cannot determine, for the same reason its sibling does. Runs on a standard macos-15 runner, which is free on a public repository, and cancels a superseded run so a stale push does not hold one of the five macOS job slots shared across the account. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Unit Tests
Mutation Tests
|
`npx @guidepup/setup` runs the package's `guidepup` binary, which needs a subcommand - without one it prints its usage and exits 1, which is what failed the job. The correct call is `setup`, plus the `--ci` flag the README calls for in CI so the command does not wait on the manual steps a local machine needs. The change detection goes away with it. Deciding what counts as announcement- relevant turned out to be guesswork that had already missed the inline tools, and it is moot once the suite moves off the PR path onto a schedule. Runs on every PR for now, only so the job can be proven green before that move. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`setup` and `install` are separate commands and both are required. `setup` configures the OS; `install` puts the preferences disk image into the project. With only the first, every test failed in VoiceOver.start() with "Failed to mount Guidepup preferences" out of resolveDmgPath - the error names both commands, and only one of them had been run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The baseline assertion guards the real one, but it fails on an *empty* match list - and the matches alone say nothing in that case. On CI it failed with "Expected: > 0, Received: 0" and no indication whether the cursor never reached the menu or VoiceOver worded the item differently than /menu item/ expects. `collectReachable` now carries the whole sweep alongside the matches so the message can distinguish the two. Retries follow playwright.config.ts, which already uses `isCI ? 2 : 0`. A real screen reader is driven by real keystrokes whose timing shifts under a loaded runner, and resetCursor's own comment records Case 18 as having been non-deterministic before. This does not hide a regression: a case that fails three times running is not a timing artefact, and the message now says what was announced each time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The sweep never reached a menu item because it started from the top of the web content. VO+Right moves between siblings rather than into them, and the block actions toolbar is the last sibling, so `next()` returned that same item for all twenty-five stops: ["hello world paragraph...", "block actions toolbar" x24]. The menu was open and correct the whole time - the page snapshot in the failure artifact shows menuitem "Text" present. Walking to the button that opens the toolbox first puts the cursor inside that container, which is where Case 4 reaches menu items from. Worth noting the baseline assertion earned its keep exactly as its comment said it would: without it, "no menu item is reachable after filtering" would have passed because no menu item was reachable before filtering either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…open Three CI runs agree on the constraint: once the toolbox menu is open, VoiceOver will not enter the block actions toolbar from the web content root. A forward walk from there reports "block actions toolbar" at all twenty-five stops and never descends, and the previous attempt showed even the button that opened the menu is unreachable that way - "did not reach an item matching /add block/i within 40 next steps". Case 4 reaches the items only because it never leaves the container. So neither sweep resets now. The first runs from where `act()` left the cursor. The second walks backwards to the search field, which re-anchors without going to the root and doubles as the navigation command VoiceOver needs before it will see the `fill()`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…reen The investigation notes were useful while the cause was unknown and are noise now that it is settled. What is left is one line per non-obvious decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Eight minutes driving a real screen reader is too slow to sit in front of every PR push. The workflow still runs on pull_request for the rest of its jobs; the VoiceOver job skips there, and a skipped job reports success, so it never blocks a pull request. Inert until a merge queue is enabled on main - `merge_group` does not fire without one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…gate # Conflicts: # packages/editorjs/e2e/tests/voiceover.spec.ts
gohabereg
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #187. Runs the 18-case VoiceOver suite in CI, on the merge queue only.
Why the merge queue and not every PR
The suite drives a real VoiceOver instance one test at a time and takes ~8 minutes, which is too
slow to sit in front of every push. It is wired as a job condition, not a
paths:filter onthe workflow, and the distinction matters:
paths:filter leaves its checks Pending, which blocks a PRrequired to pass it. GitHub's own advice is "avoid requiring workflows that can be skipped".
if:reports success and does not block.So
editorjs.ymlstill runs onpull_requestfor the rest of its jobs, and only the macOS jobcarries
if: github.event_name == 'merge_group'.An earlier revision of this PR tried to narrow it with a change-detection script instead. That is
removed: deciding what counts as "announcement-relevant" turned out to be guesswork that had
already missed the inline tools, and it is moot once the suite is off the PR path.
Getting it green
Three rounds, each a real failure rather than a flake:
npx @guidepup/setupneeds a subcommand. Without one it prints usage and exits 1, so everylater step was skipped. It is
setup --ci— the flag keeps it from waiting on prompts for themanual steps a local machine would need.
setupandinstallare both required.setupconfigures the OS;installputs thepreferences disk image into the project. With only the first, all 17 cases failed in
VoiceOver.start()with "Failed to mount Guidepup preferences".the toolbox menu is open VoiceOver will not enter the block actions toolbar from there — a
forward walk reports
"block actions toolbar"at all 25 stops, and even the button that openedthe menu becomes unreachable. Both sweeps now anchor on that button, which is the position
Case 4 reaches the items from.
Diagnosing (3) needed two throwaway diagnostics that are kept, because both gaps would recur:
collectReachablereturns the whole sweep alongside the matches, andwalkTo's error carries theroute it took. The first failure said only
Expected: > 0, Received: 0; the second said only"did not reach it" — neither distinguished "swept past" from "stalled against a wall".
retries: isCI ? 2 : 0matchesplaywright.config.ts. It earned itself immediately: on one runCase 7 failed and passed on retry (a genuine timing flake) while Case 18 failed all three, which
is what separated the two.
Verification
Suite green locally: 18/18, ~6–8 minutes.
CI reached the same point — 16 passed, 1 flaky, 1 failed — before the Case 18 fix, which was
diagnosed from the CI announcement logs and confirmed locally.
Before this gates anything
Two repository settings, neither of which I can make:
main.required_merge_queueisnulltoday, somerge_groupnever fires and this job is inert until it is turned on.
VoiceOverto the required checks. Running in the queue does not gate on its own.Worth considering first:
package-check / e2e-testsis also not required. The headlessaria.spec.ts/axe.spec.tssuite runs on every PR, takes seconds, is deterministic, and coversthe structural half of the same surface — the cheaper gate to require.
🤖 Generated with Claude Code