Skip to content

Add the non-regression suite on pinned repositories to CI - #66

Merged
SizzleUnrlsd merged 3 commits into
mainfrom
feat/regression-suite
Sep 4, 2026
Merged

SizzleUnrlsd merged 3 commits into
mainfrom
feat/regression-suite

Conversation

@SizzleUnrlsd

@SizzleUnrlsd SizzleUnrlsd commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Second deliverable of the product track: a non-regression suite on real repositories, run in continuous integration.

  • Acceptance tests first (test: define the non-regression suite on real repositories), the suite follows.
  • tests/regression/repositories.toml pins 12 public repositories by commit (the corpus of the audit campaign: Flask, Django, FastAPI, CLI tools, IoT and desktop code), plus the intentionally vulnerable Flask fixture kept in the tree under tests/regression/fixtures/vulnerable-flask/.
  • tests/test_regression.py checks out each repository at its commit (shallow fetch on first use, cached under .regression/ or CORETRACE_REGRESSION_ROOT), analyses it with the bundled plugins and compares findings (file, line, rule, function) and coverage with the snapshot in tests/regression/expected/<name>.json. CORETRACE_REGRESSION_UPDATE=1 records an intended change.
  • The snapshots cover 13 repositories, 120 files, 532 functions and 62 findings, all at full function coverage; the only file not analysed is a genuine syntax error upstream in yt-fts.
  • The repository tests carry the regression marker, deselected by default so the unit suite stays fast; a quick unmarked test keeps the manifest and the snapshots consistent.
  • CI gains a regression job (pytest -m regression, checkouts cached by manifest hash). The whole suite runs in about 25 seconds with cold clones, 6 seconds warm.
  • README: how to run and update the suite.
  • Three repositories of the audit corpus are private (Pylinkit, tag-collector-server, VideoSubtitles) and cannot be cloned by the CI job, so they stay out of the suite.

@SizzleUnrlsd SizzleUnrlsd self-assigned this Sep 4, 2026
@SizzleUnrlsd
SizzleUnrlsd merged commit 3a96c31 into main Sep 4, 2026
10 checks passed
@SizzleUnrlsd
SizzleUnrlsd deleted the feat/regression-suite branch September 4, 2026 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant