Skip to content

feat(run): --repeat <n> to run the selected tests n times and report flaky ones - #42

Open
YodHeVauHe wants to merge 1 commit into
BRIKEV:mainfrom
YodHeVauHe:cursor/repeat-flaky-tests-57ff
Open

YodHeVauHe wants to merge 1 commit into
BRIKEV:mainfrom
YodHeVauHe:cursor/repeat-flaky-tests-57ff

Conversation

@YodHeVauHe

Copy link
Copy Markdown

Fixes #41

Description

--repeat <n> runs the selected tests n times in one browser and reports which ones are flaky. The page reloads between rounds (one page.goto and a wait for #twd-sidebar-root), so a round does not inherit the previous round's page state, and Chrome is not relaunched. Test ids are random per load, so rounds are joined by registration index.

The run-complete block counts every round and lists a test that both passed and failed under Flaky, with (passes/runs passed) and the last error. A test that failed every round stays under Failed tests with the same count. The run exits 1 if any round of any test failed.

Decisions for this first version:

  • --repeat accepts a positive integer in either --repeat 5 or --repeat=5 form and refuses anything else.
  • Retries are forced off (retryCount 1) for the run, so a pass on a second attempt cannot hide a flake. 0 is not used, because that crashes the in-page onFail.
  • maxFailures counts distinct tests that have failed at least once, not each failed round.
  • --record (including record.enabled in config) and --shard are refused before launch.
  • Coverage collection and contract validation are skipped.
  • run.json stays on schema v3. It adds repeat on the run header and runs, passes, and skips on each test. outcome is failed when any round failed. The HTML report and summary.md list flaky tests under Needs attention.

Type of Change

  • New feature (non-breaking change which adds functionality)
  • Documentation update
  • Test addition or update

Testing

  • I have added tests that prove my fix is effective or that my feature works
  • All existing tests pass
  • New and existing unit tests pass locally

npx vitest run — 35 files, 796 tests passed.

There is no lint script in package.json.

Covered: flag parsing (both forms, rejection, suggestion), round aggregation across reloads, the exit code, maxFailures counting tests rather than rounds, refusal of --record and --shard, skipped coverage and contracts, the run-complete block, and the run.json / HTML / markdown fields.

Documentation

  • I have updated the documentation accordingly

README documents the flag. docs/cpu-throttling.md uses --repeat instead of the shell loop.

Run the selected tests n times in one browser, reloading the page
between rounds, and list each test that failed a round with its pass
count. A flaky result still fails the run.

Co-authored-by: ZaSourceCode <YodHeVauHe@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(run): --repeat <n> to run the selected tests n times and report flaky ones

2 participants