test(e2e): add desktop verification lanes by wibus-wee · Pull Request #372 · LodyAI/Lody

wibus-wee · 2026-09-04T03:13:20Z

Problem / pressure

Lody Desktop had unit and renderer coverage but no deterministic application-level suite spanning the built Electron main process, preload bridge, renderer, IPC, bundled CLI, and child-process cleanup. Resource exploration, delivery evidence, and deterministic regression also have different reliability requirements and should not share one blocking lane.

The repository has no model API key. Journey growth must therefore use a maintainer's existing local Codex login, keep credentials off GitHub, and never execute generated code before human review.

Summary

Add an isolated Playwright Electron + Cucumber harness with synthetic ACP and Git fixtures, stable scenario IDs, Page Objects, and per-scenario evidence.
Add P0 smoke and P0/P1 full regression suites, immutable acceptance rounds, and a non-blocking Scout that measures post-cleanup resource trends.
Add a validated journey registry and generated coverage matrix with two-way checks against executable scenarios.
Add a local Journey Foundry that selects one evidence-backed gap, produces a checksummed review bundle, and validates an approved candidate in a disposable worktree.
Require an assertion counterfactual, exact source restoration, three fresh focused runs, and the full desktop suite before applying a frozen journey patch.
Record every Daily scenario at 640x360, delete passing recordings, and retain one failure.webm for every failed scenario.
Add a trusted default-branch reconciler that validates the untrusted evidence artifact and uploads every retained WebM into one durable failure Issue as independently retryable inline video comments. A successful Daily closes the Issue.

Before / after

Before	After
Electron behavior stopped at Node tests and renderer stories.	P0/P1 journeys launch the built OSS Electron app and bundled CLI.
Resource monitoring exposed snapshots only.	Scout repeats Session, Review, and Work cleanup cycles and classifies post-cleanup slopes.
Coverage was a hand-maintained Markdown snapshot.	A validated registry generates the matrix and maps intent to executable scenarios in both directions.
Journey growth depended on memory or a repository model secret.	A maintainer command selects and authors one risk-ranked gap using local Codex authentication.
Daily artifacts required downloading and unpacking before diagnosis.	Every failed journey is attached directly as a playable WebM; multiple failures produce multiple videos.
A multi-file upload could become ambiguous after partial success.	Each scenario video has its own marker and retry boundary, so only incomplete uploads are retried.

Trust boundaries

Pull-request journeys use synthetic fixtures and never receive product or model credentials.
Daily execution is read-only. The workflow_run reconciler checks out the trusted default branch and never executes artifact content.
Artifact paths, file types, size limits, symlinks, run identity, and issue/comment ownership are validated before upload.
The reconciler pins GitHub CLI v2.100.0 and verifies the official Linux archive checksum before using gh issue comment --attach.
Journey authoring stops at a human-readable bundle. Validation needs explicit approval and runs with a temporary home and scrubbed environment.

Test plan

pnpm --filter @lody/e2e check (suite contract plus support, Scout, reporter, and policy tests)
pnpm --filter @lody/e2e smoke (3 scenarios, 18 steps passed)
pnpm e2e:full (4 scenarios, 25 steps passed)
pnpm e2e:acceptance -- --subject desktop-lifecycle
pnpm e2e:scout and pnpm e2e:scout:ablation
Daily reporter/policy suite: 11 tests, including two and 51 failed-video cases, deduplication, missing/oversized files, final and parent symlinks, bot ownership, and incomplete-upload retry
Real recording ablation: a forced onboarding assertion failure retained a valid WebM; restoring the assertion made the journey pass and removed the recording
actionlint for Daily, Daily reconciler, and PR smoke workflows
pnpm check:public-boundary, targeted Oxlint and Prettier, and git diff --check

Review focus

Electron/video lifecycle in e2e/src/support/electron-harness.ts and e2e/src/support/hooks.ts
Artifact validation and one-video retry semantics in .github/scripts/e2e-daily-failure.mjs
Bot-owned Issue/comment policy in .github/scripts/e2e-daily-policy.mjs
Trusted/untrusted workflow boundary in .github/workflows/e2e-daily-reconcile.yml
Local-author review and disposable-validation boundary

Residual scope

CI currently executes desktop journeys on macOS. Windows pipe behavior is unit-tested, but a Windows Electron lane remains follow-up work.
Scout findings are informational until a leak is reproducible and converted into a narrow deterministic regression.
The first real GitHub-hosted attachment is exercised after this workflow reaches the default branch and a Daily journey fails; the pinned CLI capability, reporting contract, retry policy, and real WebM production are covered independently in this PR.

Model: gpt-5.6-sol

chatgpt-codex-connector · 2026-09-04T03:13:35Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-05T09:06:54.075452Z	`e031254`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e9fa5f7864

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Model: gpt-5.6-sol

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d944fe65e5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

wibus-wee · 2026-09-05T06:20:36Z

@copilot resolve the merge conflicts in this pull request

# Conflicts: # pnpm-lock.yaml Co-authored-by: wibus-wee <62133302+wibus-wee@users.noreply.github.com>

Co-authored-by: wibus-wee <62133302+wibus-wee@users.noreply.github.com>

Copilot · 2026-09-05T06:30:25Z

@copilot resolve the merge conflicts in this pull request

Resolved in 2301a6f. The branch now includes a two-parent merge from main, the pnpm-lock.yaml conflict is resolved, and I fixed the follow-up regressions that surfaced during validation.

Model: gpt-5.6-sol

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 96d5e05769

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Model: gpt-5.6-sol

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d719fce118

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Model: gpt-5.6-sol

test(e2e): add desktop verification lanes

e9fa5f7

Model: gpt-5.6-sol

github-actions Bot added scope: desktop scope: shared scope: components scope: repository labels Sep 4, 2026

chatgpt-codex-connector Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread e2e/src/support/pages/review-page.ts Outdated

wibus-wee added 3 commits September 4, 2026 11:22

fix(ci): create desktop E2E artifact directory

6f1199b

Model: gpt-5.6-sol

test(e2e): add automated journey foundry

1aaee33

Model: gpt-5.6-sol

test(e2e): synchronize held ACP prompt

aac24c4

Model: gpt-5.6-sol

wibus-wee marked this pull request as draft September 4, 2026 06:16

wibus-wee added 2 commits September 4, 2026 14:43

fix(e2e): move journey authoring to maintainer machines

0595da1

Model: gpt-5.6-sol

fix(e2e): attach Daily failure recordings

d944fe6

Model: gpt-5.6-sol

wibus-wee marked this pull request as ready for review September 4, 2026 16:11

chatgpt-codex-connector Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread .github/workflows/e2e-daily-reconcile.yml Outdated

Copilot started work on behalf of wibus-wee September 5, 2026 06:20 View session

Copilot AI and others added 5 commits September 5, 2026 06:23

Merge remote-tracking branch 'origin/main' into feat/electron-e2e-scout

7b178e6

# Conflicts: # pnpm-lock.yaml Co-authored-by: wibus-wee <62133302+wibus-wee@users.noreply.github.com>

fix(components): preserve review prompt idle timer\n\nModel: GPT-5

85d83db

Co-authored-by: wibus-wee <62133302+wibus-wee@users.noreply.github.com>

fix(components): clean markdown and billing state\n\nModel: GPT-5

5043c93

Co-authored-by: wibus-wee <62133302+wibus-wee@users.noreply.github.com>

fix(components): clear stale billing overview cache\n\nModel: GPT-5

3a51629

Co-authored-by: wibus-wee <62133302+wibus-wee@users.noreply.github.com>

fix(components): preserve cache and markdown attributes\n\nModel: GPT-5

2301a6f

Co-authored-by: wibus-wee <62133302+wibus-wee@users.noreply.github.com>

Copilot finished work on behalf of wibus-wee September 5, 2026 06:30

fix(e2e): preserve full-suite failure state

904e94e

Model: gpt-5.6-sol

wibus-wee added the e2e-full Run the full desktop E2E suite label Sep 5, 2026

wibus-wee added 3 commits September 5, 2026 15:51

fix(e2e): report hosted regression failures

94393f1

Model: gpt-5.6-sol

fix(e2e): bound recorded scenario teardown

ffb854c

Model: gpt-5.6-sol

fix(e2e): clean up recorded launch failures

96d5e05

Model: gpt-5.6-sol

chatgpt-codex-connector Bot reviewed Sep 5, 2026

View reviewed changes

Comment thread .github/workflows/e2e-scout-reconcile.yml

fix(e2e): render failure video from traces

d719fce

Model: gpt-5.6-sol

chatgpt-codex-connector Bot reviewed Sep 5, 2026

View reviewed changes

Comment thread e2e/src/support/hooks.ts

fix(e2e): preserve trusted failure evidence

e031254

Model: gpt-5.6-sol

wibus-wee merged commit 2407786 into main Sep 5, 2026
6 checks passed

wibus-wee deleted the feat/electron-e2e-scout branch September 5, 2026 11:16

Navigation Menu

Uh oh!

test(e2e): add desktop verification lanes - #372

test(e2e): add desktop verification lanes#372
wibus-wee merged 17 commits into
mainfrom
feat/electron-e2e-scout

wibus-wee commented Sep 4, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Sep 4, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

wibus-wee commented Sep 5, 2026

Uh oh!

Copilot AI commented Sep 5, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

wibus-wee commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem / pressure

Summary

Before / after

Trust boundaries

Test plan

Review focus

Residual scope

Uh oh!

chatgpt-codex-connector Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

wibus-wee commented Sep 5, 2026

Uh oh!

Copilot AI commented Sep 5, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wibus-wee commented Sep 4, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 4, 2026 •

edited

Loading