Repository navigation
acc: Add -regr-main and -regr-release to compare changed tests against main and latest release - #7105
Merged
Merged
Conversation
Collaborator
Integration test reportCommit: b064dbd
Top 6 slowest tests (at least 2 minutes):
|
shreyas-goenka
approved these changes
Oct 19, 2026
…t main and the latest release Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…date" This reverts commit 437c0e9.
This reverts commit e9d29da.
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…nymore Co-authored-by: Isaac <no-reply@databricks.com>
denik
force-pushed
the
denik/regression-test-report2
branch
from
October 19, 2026 15:26
a7f0ef9 to
b064dbd
Compare
denik
enabled auto-merge
October 19, 2026 15:26
arushiagarwal-db
pushed a commit
to arushiagarwal-db/cli
that referenced
this pull request
Oct 19, 2026
…atabricks#7108) Unauthenticated GitHub API requests from shared CI runner IPs get a 403 (rate limit), which fails `-useversion latest`. Send `GITHUB_TOKEN` when it is set. Needed by databricks#7105. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
This was referenced Oct 20, 2026
github-merge-queue Bot
pushed a commit
that referenced
this pull request
Oct 20, 2026
Follow-up to #7105. The comments on #6989 and #7109 were 34-44 KB, close to the 60 KB cap, mostly because a new test fails the same way on main and on the latest release and the same diff was printed twice. - A failing run whose output equals an earlier column's (ignoring the random temp dir) now says "same output as <column>". - When the comment is over the cap, whole failure details are dropped (with a count) instead of cutting one in the middle and leaving its markup open. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
github-merge-queue Bot
pushed a commit
that referenced
this pull request
Oct 20, 2026
…7124) Follow-up to #7105. The tests are selected from changes under `acceptance/`, but the job set up the whole build environment and compiled the test binary (~200s) just to find nothing to compare. A lightweight `changes` job on a hosted runner now checks the diff first and skips the job; it also deletes the comment of an earlier commit, which the skipped job can no longer do. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Acceptance runner gets
-regr-mainand-regr-release: they run the tests changed on the branch with the branch CLI, a CLI built from the merge base with main, and the latest release, print a short summary and write a full markdown report. Taskfile:regr,regr-main,regr-release.New
regression-reportworkflow runs this on PRs and posts or updates a single comment with the tested commit and a link to the run. No comment when no tests were changed.Why
I've been using a Python script that did the same for a few months now and find it a good tool to validate agent work.
For PRs that claim to fix issues, it's good to see that the tests are attached are actual regression tests (as well as what output they get on main). For issues that are broken on main but not released yet, this also gives additional assurance.
Some examples on how it may look on real PRs:
#6958 (comment)
#6690 (comment)