Skip to content

e2e tests: collect unstable tests statistics#34227

Merged
EugeniyKiyashko merged 41 commits into
DevExpress:mainfrom
EugeniyKiyashko:26_1_run_unstable_with_statistics_
Jul 14, 2026
Merged

e2e tests: collect unstable tests statistics#34227
EugeniyKiyashko merged 41 commits into
DevExpress:mainfrom
EugeniyKiyashko:26_1_run_unstable_with_statistics_

Conversation

@EugeniyKiyashko

Copy link
Copy Markdown
Contributor

No description provided.

@EugeniyKiyashko EugeniyKiyashko self-assigned this Jul 7, 2026
Copilot AI review requested due to automatic review settings July 7, 2026 18:00
@EugeniyKiyashko EugeniyKiyashko requested review from a team as code owners July 7, 2026 18:00
@EugeniyKiyashko EugeniyKiyashko changed the title 26 1 run unstable with statistics e2e tests: collect unstable tests statistics Jul 7, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds infrastructure to identify, forward, and rerun “unstable” (flaky) TestCafe e2e tests in CI, collecting JSON reports to drive a dedicated “unstable” rerun job and reduce noise in regular test matrix jobs.

Changes:

  • Mark specific e2e tests as unstable via test.meta({ unstable: true }) (and add an “only unstable” execution mode).
  • Extend the TestCafe e2e runner to support: filtering by a report file, configurable retry attempts, and writing JSON reports for failures/unstable-forwarding.
  • Update GitHub workflows to upload unstable reports from regular jobs and add a dedicated testcafe-unstable job that reruns forwarded failures + all meta-unstable tests.

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
e2e/testcafe-devextreme/tests/scheduler/common/dragAndDrop/appointmentCollector.ts Marks a scheduler drag-n-drop test as unstable.
e2e/testcafe-devextreme/tests/editors/dateRangeBox/keyboard.ts Removes disablePageReloads for the fixture; marks one test as unstable.
e2e/testcafe-devextreme/tests/editors/dateRangeBox/focus.ts Marks one focus-related test as unstable.
e2e/testcafe-devextreme/tests/dataGrid/common/selection.ts Marks a selection regression test as unstable.
e2e/testcafe-devextreme/tests/dataGrid/common/keyboardNavigation/keyboardNavigation.functional.ts Marks one parameterized test variant as unstable via meta.
e2e/testcafe-devextreme/tests/common/treeList/stickyColumns/stickyColumns.ts Narrows theme coverage for a screenshot test (genericLight only).
e2e/testcafe-devextreme/runner.ts Adds report writing, filtering by tests file, onlyUnstable, and configurable retry attempts; forwards single failures as unstable.
apps/demos/utils/visual-tests/testcafe-runner.ts Hard-disables TestCafe quarantine mode (env-driven toggle removed).
apps/demos/project.json Removes TCQUARANTINE from the demos test env passthrough list.
.github/workflows/wrapper_tests.yml Adjusts nx cache restore keys to prefer workflow build caches.
.github/workflows/visual-tests-demos.yml Increases timeouts; removes TCQUARANTINE env usage in demos visual tests jobs.
.github/workflows/testcafe_tests.yml Adds unstable-report upload, adds testcafe-unstable job to rerun forwarded + meta-unstable tests, and updates downstream job dependencies.

Comment thread e2e/testcafe-devextreme/runner.ts Outdated
Comment thread apps/demos/utils/visual-tests/testcafe-runner.ts
@EugeniyKiyashko EugeniyKiyashko marked this pull request as draft July 7, 2026 21:04
@EugeniyKiyashko EugeniyKiyashko force-pushed the 26_1_run_unstable_with_statistics_ branch from f8e26ec to 359677d Compare July 8, 2026 06:05
@EugeniyKiyashko EugeniyKiyashko changed the base branch from 26_1 to main July 13, 2026 11:41
@EugeniyKiyashko EugeniyKiyashko marked this pull request as ready for review July 14, 2026 12:11
Copilot AI review requested due to automatic review settings July 14, 2026 12:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Comment thread e2e/testcafe-devextreme/runner.ts
Comment thread e2e/testcafe-devextreme/runner.ts
Comment thread .github/workflows/testcafe_tests.yml
@EugeniyKiyashko EugeniyKiyashko added this pull request to the merge queue Jul 14, 2026
Merged via the queue into DevExpress:main with commit 6fe6d5b Jul 14, 2026
172 of 174 checks passed
@EugeniyKiyashko EugeniyKiyashko deleted the 26_1_run_unstable_with_statistics_ branch July 14, 2026 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants