Add flaky-test detector (flaky) + FLAKY_TESTS.md#102
Merged
Conversation
Add a report-only `flaky` root script that detects flaky tests in two dimensions: it runs the full Stuff-iOS-Tests suite N times (default 10), then tight-loops (in isolation, fresh process per iteration) any test that failed at least once M times (default 50). A test is reported only when it both passed and failed across those runs (non-deterministic), so consistently passing/failing tests are excluded. Findings and flake counts are written to FLAKY_TESTS.md at the repo root. Like ./profile it never fails on flaky/failing tests (only a build failure errors) and runs tuist generate --no-open so it never opens Xcode. Documented in AGENTS.md and README.md; CLAUDE.md regenerated via ./sync-agents.
- Count suspects with awk|wc instead of `grep -c`, which printed 0 and exited non-zero on no match and tripped the fallback into a bogus two-line count (the harmless "integer expression expected" warning at 0 suspects). - Seed FLAKY_TESTS.md from a real run: 1248 tests across 10 suite runs, no flaky tests detected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a report-only `flaky` root script that hunts for flaky tests along two dimensions, and a root `FLAKY_TESTS.md` that records the findings. Intended to be driven by a weekly Cursor automation (set up separately) that runs the script to detect/update — and optionally fix — flaky tests.
How it works
Ntimes (default 10) via `test-without-building`; parse each `.xcresult` for per-test pass/fail. Any test that fails in ≥1 run is a "suspect".Flags: `--suite-runs`, `--iterations`, `--device`/`--os`, `--scheme`, `--relaunch`, `--no-update`, `--top`, `-h`.
FLAKY_TESTS.md
Written/updated by the script (unless `--no-update`): a do-not-hand-edit header with run params + UTC timestamp, a table sorted by flake rate (test id, bundle, suite fails `k/N`, tight-loop fails `j/M`, flake rate), and an explicit no-flakes state.
Seed run
Ran the script end to end: 1248 distinct tests across 10 suite runs, 0 suspects → no flaky tests detected. `FLAKY_TESTS.md` is seeded with that clean result.
Docs
Documented `flaky` in `AGENTS.md` (Root dev scripts) and `README.md` (scripts list + prose); `CLAUDE.md` regenerated via `./sync-agents`. No `Project.swift`/`Package.swift` changes — it targets the existing `Stuff-iOS-Tests` scheme and per-test `-only-testing`.
Notes