Repository navigation
feat(run): --repeat <n> to run the selected tests n times and report flaky ones - #42
Open
YodHeVauHe wants to merge 1 commit into
Open
YodHeVauHe wants to merge 1 commit into
YodHeVauHe wants to merge 1 commit into
Conversation
Run the selected tests n times in one browser, reloading the page between rounds, and list each test that failed a round with its pass count. A flaky result still fails the run. Co-authored-by: ZaSourceCode <YodHeVauHe@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #41
Description
--repeat <n>runs the selected tests n times in one browser and reports which ones are flaky. The page reloads between rounds (onepage.gotoand a wait for#twd-sidebar-root), so a round does not inherit the previous round's page state, and Chrome is not relaunched. Test ids are random per load, so rounds are joined by registration index.The run-complete block counts every round and lists a test that both passed and failed under
Flaky, with(passes/runs passed)and the last error. A test that failed every round stays underFailed testswith the same count. The run exits 1 if any round of any test failed.Decisions for this first version:
--repeataccepts a positive integer in either--repeat 5or--repeat=5form and refuses anything else.retryCount1) for the run, so a pass on a second attempt cannot hide a flake.0is not used, because that crashes the in-pageonFail.maxFailurescounts distinct tests that have failed at least once, not each failed round.--record(includingrecord.enabledin config) and--shardare refused before launch.run.jsonstays on schema v3. It addsrepeaton the run header andruns,passes, andskipson each test.outcomeisfailedwhen any round failed. The HTML report andsummary.mdlist flaky tests under Needs attention.Type of Change
Testing
npx vitest run— 35 files, 796 tests passed.There is no lint script in
package.json.Covered: flag parsing (both forms, rejection, suggestion), round aggregation across reloads, the exit code,
maxFailurescounting tests rather than rounds, refusal of--recordand--shard, skipped coverage and contracts, the run-complete block, and therun.json/ HTML / markdown fields.Documentation
README documents the flag.
docs/cpu-throttling.mduses--repeatinstead of the shell loop.