Skip to content

Run pipelines test against the pipeline's testing.sources - #7429

Open
tharunbhuk-db wants to merge 10 commits into
tharun/pipelines-test-b4-testing-deployfrom
tharun/pipelines-test-j1-testing-sources
Open

Run pipelines test against the pipeline's testing.sources#7429
tharunbhuk-db wants to merge 10 commits into
tharun/pipelines-test-b4-testing-deployfrom
tharun/pipelines-test-j1-testing-sources

Conversation

@tharunbhuk-db

@tharunbhuk-db tharunbhuk-db commented Nov 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🥞 Stacked PR

Review only the last commit (Run pipelines test against the pipeline's testing.sources). The merge commit before it brings in the pipelines test command stack (#7297 → #7434), so CI can run. When that stack merges, this PR moves to main and the merge goes away.

  • The pipelines test acceptance bundles declare testing, as the service requires testing.sources.
  • When Start Update fails with PIPELINE_TEST_NO_TEST_SOURCES and the bundle sets testing, the error says to deploy first.
  • A live test deploys a bundle with testing and a local test dependency after -e, and runs its tests. Locally it uses stubs; on a real workspace it runs only with TEST_PIPELINES_TESTING_ENABLED.
  • I'm not sure of the service's error shape for PIPELINE_TEST_NO_TEST_SOURCES, so the hint matches the class in the error code or the message.

Tests: TestWithDeployHint; acceptance pipelines/test/no-test-sources and pipelines/test/live.

This pull request and its description were written by Isaac.

This was referenced Nov 10, 2026
Open
Draft
Open
Open
Open
Open
Open
Draft
@github-actions

github-actions Bot commented Nov 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Commit: bbb4adb4a | Workflow run

Regression test report

Selected 20 changed tests (limit=20, 0 not selected)

24 tests: regression=24 unreleased=0 coverage=0 failing=0
test branch main (688c228) latest (v1.25.0)
TestAccept/bundle/environments/editable-folder/DMS= ✅ ❌ ❌
TestAccept/bundle/environments/editable-folder/DMS=true ✅ ❌ ❌
TestAccept/bundle/refschema/DMS= ✅ ❌ ❌
TestAccept/bundle/refschema/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/pipelines/testing/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/pipelines/testing/DMS=true ✅ ❌ ❌
TestAccept/pipelines/databricks-cli-help ✅ ❌ ❌
TestAccept/pipelines/test/arguments ✅ ❌ ❌
TestAccept/pipelines/test/immutable ✅ ❌ ❌
TestAccept/pipelines/test/invalid-project ✅ ❌ ❌
TestAccept/pipelines/test/live ✅ ❌ ❌
TestAccept/pipelines/test/no-test-sources ✅ ❌ ❌
TestAccept/pipelines/test/pipeline-id ✅ ❌ ❌
TestAccept/pipelines/test/results/failed ✅ ❌ ❌
TestAccept/pipelines/test/results/incomplete ✅ ❌ ❌
TestAccept/pipelines/test/results/passed ✅ ❌ ❌
TestAccept/pipelines/test/run-control/canceled ✅ ❌ ❌
TestAccept/pipelines/test/run-control/not-test-update ✅ ❌ ❌
TestAccept/pipelines/test/run-control/resume ✅ ❌ ❌
TestAccept/pipelines/test/run-control/timeout ✅ ❌ ❌
TestAccept/pipelines/test/selection ✅ ❌ ❌
TestAccept/pipelines/test/submit-error ✅ ❌ ❌
TestAccept/bundle/environments/editable-folder/DMS=: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n=== With pipeline testing, a folder after -e maps to its deployed folder\n\n>>> [CLI] bundle validate -o json\n{\n  \"job\": [\n    \"-e /Workspace/Users/[USERNAME]/.bundle/editable-folder/default/files/project\"\n  ],\n  \"pipeline\": [\n    \"-e /Workspace/Users/[USERNAME]/.bundle/editable-folder/default/files/project[test]\"\n  ]\n}\n\n=== Without pipeline testing, a folder after -e is an error, as before\nError: unable to determine if [TEST_TMP_DIR]/project is not a notebook\n"
        	            	actual  : "\n=== With pipeline testing, a folder after -e maps to its deployed folder\n\n>>> [CLI] bundle validate -o json\nError: unable to determine if [TEST_TMP_DIR]/project is not a notebook: read [TEST_TMP_DIR]/project: is a directory\n\n{\n  \"job\": [\n    \"-e project\"\n  ],\n  \"pipeline\": [\n    \"-e project[test]\"\n  ]\n}\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,8 +4,10 @@
        	            	 >>> [CLI] bundle validate -o json
        	            	+Error: unable to determine if [TEST_TMP_DIR]/project is not a notebook: read [TEST_TMP_DIR]/project: is a directory
        	            	+
        	            	 {
        	            	   "job": [
        	            	-    "-e /Workspace/Users/[USERNAME]/.bundle/editable-folder/default/files/project"
        	            	+    "-e project"
        	            	   ],
        	            	   "pipeline": [
        	            	-    "-e /Workspace/Users/[USERNAME]/.bundle/editable-folder/default/files/project[test]"
        	            	+    "-e project[test]"
        	            	   ]
        	            	@@ -13,4 +15,3 @@
        	            	 
        	            	-=== Without pipeline testing, a folder after -e is an error, as before
        	            	-Error: unable to determine if [TEST_TMP_DIR]/project is not a notebook
        	            	+Exit code: 1
        	            	 
        	Test:       	TestAccept/bundle/environments/editable-folder/DMS=
        	Messages:   	bundle/environments/editable-folder/output.txt vs /tmp/TestAcceptbundleenvironmentseditable-folderDMS=1545174144/002/output.txt

latest (v1.25.0): same output as main (688c228)

TestAccept/bundle/environments/editable-folder/DMS=true: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n=== With pipeline testing, a folder after -e maps to its deployed folder\n\n>>> [CLI] bundle validate -o json\n{\n  \"job\": [\n    \"-e /Workspace/Users/[USERNAME]/.bundle/editable-folder/default/files/project\"\n  ],\n  \"pipeline\": [\n    \"-e /Workspace/Users/[USERNAME]/.bundle/editable-folder/default/files/project[test]\"\n  ]\n}\n\n=== Without pipeline testing, a folder after -e is an error, as before\nError: unable to determine if [TEST_TMP_DIR]/project is not a notebook\n"
        	            	actual  : "\n=== With pipeline testing, a folder after -e maps to its deployed folder\n\n>>> [CLI] bundle validate -o json\nError: unable to determine if [TEST_TMP_DIR]/project is not a notebook: read [TEST_TMP_DIR]/project: is a directory\n\n{\n  \"job\": [\n    \"-e project\"\n  ],\n  \"pipeline\": [\n    \"-e project[test]\"\n  ]\n}\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,8 +4,10 @@
        	            	 >>> [CLI] bundle validate -o json
        	            	+Error: unable to determine if [TEST_TMP_DIR]/project is not a notebook: read [TEST_TMP_DIR]/project: is a directory
        	            	+
        	            	 {
        	            	   "job": [
        	            	-    "-e /Workspace/Users/[USERNAME]/.bundle/editable-folder/default/files/project"
        	            	+    "-e project"
        	            	   ],
        	            	   "pipeline": [
        	            	-    "-e /Workspace/Users/[USERNAME]/.bundle/editable-folder/default/files/project[test]"
        	            	+    "-e project[test]"
        	            	   ]
        	            	@@ -13,4 +15,3 @@
        	            	 
        	            	-=== Without pipeline testing, a folder after -e is an error, as before
        	            	-Error: unable to determine if [TEST_TMP_DIR]/project is not a notebook
        	            	+Exit code: 1
        	            	 
        	Test:       	TestAccept/bundle/environments/editable-folder/DMS=true
        	Messages:   	bundle/environments/editable-folder/output.txt vs /tmp/TestAcceptbundleenvironmentseditable-folderDMS=true1468573854/002/output.txt

latest (v1.25.0): same output as main (688c228)

TestAccept/bundle/refschema/DMS=: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- bundle/refschema/output/pipelines.txt
        +++ /tmp/TestAcceptbundlerefschemaDMS=1237845323/001/output/pipelines.txt
        @@ -867,14 +867,6 @@
         resources.pipelines.*.tags	map[string]string	ALL
         resources.pipelines.*.tags.*	string	ALL
         resources.pipelines.*.target	string	ALL
        -resources.pipelines.*.testing	*resources.PipelineTesting	ALL
        -resources.pipelines.*.testing.environment	*resources.PipelineTestEnvironment	ALL
        -resources.pipelines.*.testing.environment.dependencies	[]string	ALL
        -resources.pipelines.*.testing.environment.dependencies[*]	string	ALL
        -resources.pipelines.*.testing.sources	[]resources.PipelineTestSource	ALL
        -resources.pipelines.*.testing.sources[*]	resources.PipelineTestSource	ALL
        -resources.pipelines.*.testing.sources[*].glob	*pipelines.PathPattern	ALL
        -resources.pipelines.*.testing.sources[*].glob.include	string	ALL
         resources.pipelines.*.trigger	*pipelines.PipelineTrigger	ALL
         resources.pipelines.*.trigger.cron	*pipelines.CronTrigger	ALL
         resources.pipelines.*.trigger.cron.quartz_cron_schedule	string	ALL

latest (v1.25.0): same output as main (688c228)

TestAccept/bundle/refschema/DMS=true: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- bundle/refschema/output/pipelines.txt
        +++ /tmp/TestAcceptbundlerefschemaDMS=true3533062040/001/output/pipelines.txt
        @@ -867,14 +867,6 @@
         resources.pipelines.*.tags	map[string]string	ALL
         resources.pipelines.*.tags.*	string	ALL
         resources.pipelines.*.target	string	ALL
        -resources.pipelines.*.testing	*resources.PipelineTesting	ALL
        -resources.pipelines.*.testing.environment	*resources.PipelineTestEnvironment	ALL
        -resources.pipelines.*.testing.environment.dependencies	[]string	ALL
        -resources.pipelines.*.testing.environment.dependencies[*]	string	ALL
        -resources.pipelines.*.testing.sources	[]resources.PipelineTestSource	ALL
        -resources.pipelines.*.testing.sources[*]	resources.PipelineTestSource	ALL
        -resources.pipelines.*.testing.sources[*].glob	*pipelines.PathPattern	ALL
        -resources.pipelines.*.testing.sources[*].glob.include	string	ALL
         resources.pipelines.*.trigger	*pipelines.PipelineTrigger	ALL
         resources.pipelines.*.trigger.cron	*pipelines.CronTrigger	ALL
         resources.pipelines.*.trigger.cron.quartz_cron_schedule	string	ALL
        
FAIL

latest (v1.25.0): same output as main (688c228)

TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS=: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> musterr [CLI] bundle deploy\nError: Duplicate local library names: test_helpers-0.1-py3-none-any.whl\n  at resources.pipelines.orders.environment.dependencies[0]\n     resources.pipelines.orders.testing.environment.dependencies[0]\n  in databricks.yml:13:13\n     databricks.yml:21:15\n\nLocal library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl\n\n"
        	            	actual  : "\n>>> musterr [CLI] bundle deploy\nWarning: unknown field: testing\n  at resources.pipelines.orders\n  in databricks.yml:14:7\n\nUploading pipeline/test_helpers-0.1-py3-none-any.whl...\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-same-name-libraries/default/files...\nCreated pipelines.orders\nFiles: 5 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\n\nUnexpected success\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,10 +2,15 @@
        	            	 >>> musterr [CLI] bundle deploy
        	            	-Error: Duplicate local library names: test_helpers-0.1-py3-none-any.whl
        	            	-  at resources.pipelines.orders.environment.dependencies[0]
        	            	-     resources.pipelines.orders.testing.environment.dependencies[0]
        	            	-  in databricks.yml:13:13
        	            	-     databricks.yml:21:15
        	            	+Warning: unknown field: testing
        	            	+  at resources.pipelines.orders
        	            	+  in databricks.yml:14:7
        	            	 
        	            	-Local library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl
        	            	+Uploading pipeline/test_helpers-0.1-py3-none-any.whl...
        	            	+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-same-name-libraries/default/files...
        	            	+Created pipelines.orders
        	            	+Files: 5 uploaded, 0 deleted
        	            	+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        	            	 
        	            	+Unexpected success
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS=
        	Messages:   	bundle/resources/pipelines/testing-same-name-libraries/output.txt vs /tmp/TestAcceptbundleresourcespipelinestesting-same-name-librarie2097585946/002/output.txt
    acceptance_test.go:1135: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json
        .databricks/bundle/default/sync-snapshots/b407a45e1eee3126.json

latest (v1.25.0):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> musterr [CLI] bundle deploy\nError: Duplicate local library names: test_helpers-0.1-py3-none-any.whl\n  at resources.pipelines.orders.environment.dependencies[0]\n     resources.pipelines.orders.testing.environment.dependencies[0]\n  in databricks.yml:13:13\n     databricks.yml:21:15\n\nLocal library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl\n\n"
        	            	actual  : "\n>>> musterr [CLI] bundle deploy\nWarning: unknown field: testing\n  at resources.pipelines.orders\n  in databricks.yml:14:7\n\nUploading pipeline/test_helpers-0.1-py3-none-any.whl...\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-same-name-libraries/default/files...\nCreated pipelines.orders\nFiles: 5 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\n\nUnexpected success\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,10 +2,15 @@
        	            	 >>> musterr [CLI] bundle deploy
        	            	-Error: Duplicate local library names: test_helpers-0.1-py3-none-any.whl
        	            	-  at resources.pipelines.orders.environment.dependencies[0]
        	            	-     resources.pipelines.orders.testing.environment.dependencies[0]
        	            	-  in databricks.yml:13:13
        	            	-     databricks.yml:21:15
        	            	+Warning: unknown field: testing
        	            	+  at resources.pipelines.orders
        	            	+  in databricks.yml:14:7
        	            	 
        	            	-Local library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl
        	            	+Uploading pipeline/test_helpers-0.1-py3-none-any.whl...
        	            	+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-same-name-libraries/default/files...
        	            	+Created pipelines.orders
        	            	+Files: 5 uploaded, 0 deleted
        	            	+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        	            	 
        	            	+Unexpected success
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS=
        	Messages:   	bundle/resources/pipelines/testing-same-name-libraries/output.txt vs /tmp/TestAcceptbundleresourcespipelinestesting-same-name-librarie2399655598/002/output.txt
    acceptance_test.go:1135: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json
        .databricks/bundle/default/sync-snapshots/e4c2e14012dbfaea.json
TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS=true: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> musterr [CLI] bundle deploy\nError: Duplicate local library names: test_helpers-0.1-py3-none-any.whl\n  at resources.pipelines.orders.environment.dependencies[0]\n     resources.pipelines.orders.testing.environment.dependencies[0]\n  in databricks.yml:13:13\n     databricks.yml:21:15\n\nLocal library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl\n\n"
        	            	actual  : "\n>>> musterr [CLI] bundle deploy\nWarning: unknown field: testing\n  at resources.pipelines.orders\n  in databricks.yml:14:7\n\nUploading pipeline/test_helpers-0.1-py3-none-any.whl...\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-same-name-libraries/default/files...\nCreated pipelines.orders\nFiles: 5 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\n\nUnexpected success\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,10 +2,15 @@
        	            	 >>> musterr [CLI] bundle deploy
        	            	-Error: Duplicate local library names: test_helpers-0.1-py3-none-any.whl
        	            	-  at resources.pipelines.orders.environment.dependencies[0]
        	            	-     resources.pipelines.orders.testing.environment.dependencies[0]
        	            	-  in databricks.yml:13:13
        	            	-     databricks.yml:21:15
        	            	+Warning: unknown field: testing
        	            	+  at resources.pipelines.orders
        	            	+  in databricks.yml:14:7
        	            	 
        	            	-Local library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl
        	            	+Uploading pipeline/test_helpers-0.1-py3-none-any.whl...
        	            	+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-same-name-libraries/default/files...
        	            	+Created pipelines.orders
        	            	+Files: 5 uploaded, 0 deleted
        	            	+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        	            	 
        	            	+Unexpected success
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS=true
        	Messages:   	bundle/resources/pipelines/testing-same-name-libraries/output.txt vs /tmp/TestAcceptbundleresourcespipelinestesting-same-name-librarie3629588412/002/output.txt
    acceptance_test.go:1135: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json
        .databricks/bundle/default/sync-snapshots/92144f04b0361e60.json

latest (v1.25.0):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> musterr [CLI] bundle deploy\nError: Duplicate local library names: test_helpers-0.1-py3-none-any.whl\n  at resources.pipelines.orders.environment.dependencies[0]\n     resources.pipelines.orders.testing.environment.dependencies[0]\n  in databricks.yml:13:13\n     databricks.yml:21:15\n\nLocal library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl\n\n"
        	            	actual  : "\n>>> musterr [CLI] bundle deploy\nWarning: unknown field: testing\n  at resources.pipelines.orders\n  in databricks.yml:14:7\n\nUploading pipeline/test_helpers-0.1-py3-none-any.whl...\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-same-name-libraries/default/files...\nCreated pipelines.orders\nFiles: 5 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\n\nUnexpected success\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,10 +2,15 @@
        	            	 >>> musterr [CLI] bundle deploy
        	            	-Error: Duplicate local library names: test_helpers-0.1-py3-none-any.whl
        	            	-  at resources.pipelines.orders.environment.dependencies[0]
        	            	-     resources.pipelines.orders.testing.environment.dependencies[0]
        	            	-  in databricks.yml:13:13
        	            	-     databricks.yml:21:15
        	            	+Warning: unknown field: testing
        	            	+  at resources.pipelines.orders
        	            	+  in databricks.yml:14:7
        	            	 
        	            	-Local library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl
        	            	+Uploading pipeline/test_helpers-0.1-py3-none-any.whl...
        	            	+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-same-name-libraries/default/files...
        	            	+Created pipelines.orders
        	            	+Files: 5 uploaded, 0 deleted
        	            	+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        	            	 
        	            	+Unexpected success
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS=true
        	Messages:   	bundle/resources/pipelines/testing-same-name-libraries/output.txt vs /tmp/TestAcceptbundleresourcespipelinestesting-same-name-librarie1351514248/002/output.txt
    acceptance_test.go:1135: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json
        .databricks/bundle/default/sync-snapshots/154f1b5e3c34a2ea.json
TestAccept/bundle/resources/pipelines/testing/DMS=: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- bundle/resources/pipelines/testing/output.txt
        +++ /tmp/TestAcceptbundleresourcespipelinestestingDMS=3923220020/002/output.txt
        @@ -2,29 +2,18 @@
         === Relative test paths resolve to the deployed workspace paths
         
         >>> [CLI] bundle validate -o json
        -{
        -  "environment": {
        -    "dependencies": [
        -      "-e /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files[test]",
        -      "-r /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/requirements.txt",
        -      "./dist/*.whl"
        -    ]
        -  },
        -  "sources": [
        -    {
        -      "glob": {
        -        "include": "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/**"
        -      }
        -    }
        -  ]
        -}
        +Warning: unknown field: testing
        +  at resources.pipelines.orders
        +  in databricks.yml:11:7
         
        +null
        +
         === Without pipeline testing, the property is an error
         
         >>> musterr env -u DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING [CLI] bundle validate
        -Error: the testing property of pipelines is experimental. Set DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING=true to enable it
        -  at resources.pipelines.orders.testing
        -  in databricks.yml:12:9
        +Warning: unknown field: testing
        +  at resources.pipelines.orders
        +  in databricks.yml:11:7
         
         Name: pipeline-testing
         Target: default
        @@ -32,116 +17,8 @@
           User: [USERNAME]
           Path: /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default
         
        -Found 1 error
        +Found 1 warning
         
        -=== Deploy sends testing, with the local wheel uploaded
        +Unexpected success
         
        ->>> [CLI] bundle deploy
        -Uploading dist/test_helpers-0.1-py3-none-any.whl...
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
        -Created pipelines.orders
        -Files: 6 uploaded, 0 deleted
        -Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py //api/2.0/pipelines --nostamp
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines",
        -  "testing": {
        -    "sources": [
        -      {
        -        "glob": {
        -          "include": "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/**"
        -        }
        -      }
        -    ],
        -    "environment": {
        -      "dependencies": [
        -        "-e /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files[test]",
        -        "-r /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/requirements.txt",
        -        "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/artifacts/.internal/test_helpers-0.1-py3-none-any.whl"
        -      ]
        -    }
        -  }
        -}
        -
        -=== No changes after the deploy
        -
        ->>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
        -
        -=== A remote edit that drops testing shows in the plan
        -
        ->>> [CLI] bundle plan -o json
        -"update"
        -
        ->>> [CLI] bundle deploy
        -Uploading dist/test_helpers-0.1-py3-none-any.whl...
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
        -Updated pipelines.orders
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        -
        ->>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
        -
        -=== Changing testing updates the pipeline
        -
        ->>> update_file.py databricks.yml ./tests/** ./tests/unit/**
        -
        ->>> [CLI] bundle plan
        -update pipelines.orders
        -
        -Plan: 0 to add, 1 to change, 0 to delete, 0 unchanged
        -
        ->>> [CLI] bundle deploy
        -Uploading dist/test_helpers-0.1-py3-none-any.whl...
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
        -Updated pipelines.orders
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py //api/2.0/pipelines --nostamp
        -{
        -  "method": "PUT",
        -  "path": "/api/2.0/pipelines/[ORDERS_ID]",
        -  "sources": [
        -    {
        -      "glob": {
        -        "include": "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/unit/**"
        -      }
        -    }
        -  ]
        -}
        -
        -=== Removing testing sends an update without it
        -
        ->>> [CLI] bundle plan
        -update pipelines.orders
        -
        -Plan: 0 to add, 1 to change, 0 to delete, 0 unchanged
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
        -Updated pipelines.orders
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py //api/2.0/pipelines --nostamp
        -{
        -  "method": "PUT",
        -  "path": "/api/2.0/pipelines/[ORDERS_ID]",
        -  "testing": null
        -}
        -
        ->>> [CLI] bundle destroy --auto-approve
        -The following resources will be deleted:
        -  delete resources.pipelines.orders
        -
        -This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
        -Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
        -  delete resources.pipelines.orders
        -
        -All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default
        -
        -Destroy: 1 deleted
        +Exit code: 1

latest (v1.25.0): same output as main (688c228)

TestAccept/bundle/resources/pipelines/testing/DMS=true: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- bundle/resources/pipelines/testing/output.txt
        +++ /tmp/TestAcceptbundleresourcespipelinestestingDMS=true3395941117/002/output.txt
        @@ -2,29 +2,18 @@
         === Relative test paths resolve to the deployed workspace paths
         
         >>> [CLI] bundle validate -o json
        -{
        -  "environment": {
        -    "dependencies": [
        -      "-e /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files[test]",
        -      "-r /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/requirements.txt",
        -      "./dist/*.whl"
        -    ]
        -  },
        -  "sources": [
        -    {
        -      "glob": {
        -        "include": "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/**"
        -      }
        -    }
        -  ]
        -}
        +Warning: unknown field: testing
        +  at resources.pipelines.orders
        +  in databricks.yml:11:7
         
        +null
        +
         === Without pipeline testing, the property is an error
         
         >>> musterr env -u DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING [CLI] bundle validate
        -Error: the testing property of pipelines is experimental. Set DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING=true to enable it
        -  at resources.pipelines.orders.testing
        -  in databricks.yml:12:9
        +Warning: unknown field: testing
        +  at resources.pipelines.orders
        +  in databricks.yml:11:7
         
         Name: pipeline-testing
         Target: default
        @@ -32,116 +17,8 @@
           User: [USERNAME]
           Path: /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default
         
        -Found 1 error
        +Found 1 warning
         
        -=== Deploy sends testing, with the local wheel uploaded
        +Unexpected success
         
        ->>> [CLI] bundle deploy
        -Uploading dist/test_helpers-0.1-py3-none-any.whl...
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
        -Created pipelines.orders
        -Files: 6 uploaded, 0 deleted
        -Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py //api/2.0/pipelines --nostamp
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines",
        -  "testing": {
        -    "sources": [
        -      {
        -        "glob": {
        -          "include": "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/**"
        -        }
        -      }
        -    ],
        -    "environment": {
        -      "dependencies": [
        -        "-e /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files[test]",
        -        "-r /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/requirements.txt",
        -        "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/artifacts/.internal/test_helpers-0.1-py3-none-any.whl"
        -      ]
        -    }
        -  }
        -}
        -
        -=== No changes after the deploy
        -
        ->>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
        -
        -=== A remote edit that drops testing shows in the plan
        -
        ->>> [CLI] bundle plan -o json
        -"update"
        -
        ->>> [CLI] bundle deploy
        -Uploading dist/test_helpers-0.1-py3-none-any.whl...
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
        -Updated pipelines.orders
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        -
        ->>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
        -
        -=== Changing testing updates the pipeline
        -
        ->>> update_file.py databricks.yml ./tests/** ./tests/unit/**
        -
        ->>> [CLI] bundle plan
        -update pipelines.orders
        -
        -Plan: 0 to add, 1 to change, 0 to delete, 0 unchanged
        -
        ->>> [CLI] bundle deploy
        -Uploading dist/test_helpers-0.1-py3-none-any.whl...
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
        -Updated pipelines.orders
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py //api/2.0/pipelines --nostamp
        -{
        -  "method": "PUT",
        -  "path": "/api/2.0/pipelines/[ORDERS_ID]",
        -  "sources": [
        -    {
        -      "glob": {
        -        "include": "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/unit/**"
        -      }
        -    }
        -  ]
        -}
        -
        -=== Removing testing sends an update without it
        -
        ->>> [CLI] bundle plan
        -update pipelines.orders
        -
        -Plan: 0 to add, 1 to change, 0 to delete, 0 unchanged
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
        -Updated pipelines.orders
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py //api/2.0/pipelines --nostamp
        -{
        -  "method": "PUT",
        -  "path": "/api/2.0/pipelines/[ORDERS_ID]",
        -  "testing": null
        -}
        -
        ->>> [CLI] bundle destroy --auto-approve
        -The following resources will be deleted:
        -  delete resources.pipelines.orders
        -
        -This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
        -Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
        -  delete resources.pipelines.orders
        -
        -All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default
        -
        -Destroy: 1 deleted
        +Exit code: 1

latest (v1.25.0): same output as main (688c228)

TestAccept/pipelines/databricks-cli-help: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- pipelines/databricks-cli-help/output.txt
        +++ /tmp/TestAcceptpipelinesdatabricks-cli-help193421886/002/output.txt
        @@ -30,7 +30,6 @@
           run                   Run a pipeline
           show                  Preview a table's columns and sample rows
           stop                  Stop a pipeline
        -  test                  *Preview* Run the tests of a pipeline
         
         Management Commands
           apply-environment     *Public Preview* Apply the latest environment to the pipeline.

latest (v1.25.0): same output as main (688c228)

TestAccept/pipelines/test/arguments: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- pipelines/test/arguments/output.txt
        +++ /tmp/TestAcceptpipelinestestarguments18668422/002/output.txt
        @@ -1,116 +1,906 @@
         
         >>> [CLI] pipelines test --help
        -Run the tests of a pipeline in a test-only update.
        +The Lakeflow Spark Declarative Pipelines API allows you to create, edit,
        +  delete, start, and view details about pipelines.
         
        -This command is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it.
        +  Spark Declarative Pipelines is a framework for building reliable,
        +  maintainable, and testable data processing pipelines. You define the
        +  transformations to perform on your data, and Spark Declarative Pipelines
        +  manages task orchestration, cluster management, monitoring, data quality, and
        +  error handling.
         
        -KEY is the unique name of the pipeline, as defined in its YAML file.
        -It is required when test selections are given. If there is only one pipeline
        -in the project and no arguments are given, it is selected automatically.
        -Outside a project, identify the pipeline with --pipeline-id; all positional
        -arguments are then test selections.
        +  Instead of defining your data pipelines using a series of separate Apache
        +  Spark tasks, Spark Declarative Pipelines manages how your data is transformed
        +  based on a target schema you define for each processing step. You can also
        +  enforce data quality with Spark Declarative Pipelines expectations.
        +  Expectations allow you to define expected data quality and specify how to
        +  handle records that fail those expectations.
         
        -TEST_SELECTION is a test directory, file, or pytest test ID, for example
        -tests/test_orders.py::test_backfill. In a project, a relative path is a local
        -path, relative to the working directory, and is converted to its deployed
        -workspace path. With --pipeline-id, a relative path is relative to the
        -pipeline's root_path. Paths that start with / are workspace paths. Without a
        -selection, all of the pipeline's tests run.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Each --pytest-arg passes one argument to pytest, in the order given. Only -k
        -and -m are supported, each followed by its value in the next --pytest-arg, for
        -example:
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        -  databricks pipelines test my_pipeline --pytest-arg=-k --pytest-arg='orders and not slow'
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        -The command waits for the results. It exits with code 0 only if at least one
        -test was collected and none failed or errored, so a run whose tests were all
        -skipped succeeds; otherwise it exits with code 1. --no-wait returns after the
        -update starts.
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        ---timeout and Ctrl+C stop waiting but not the update; pass its ID to --resume
        -to read the results later. To stop the update, run 'pipelines stop'.
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
        ---junit-xml writes one entry per test with its outcome and duration. An
        -incomplete run, a run without tests, or a timeout adds a failing entry that
        -names the cause. The report replaces the file on every exit except a flag that
        -cannot be parsed. With workspace.sync_mode immutable, test names keep the path
        -of the deployed snapshot, which changes with each deployment.
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +=== The command is experimental and off by default
        +>>> musterr [CLI] pipelines test --pipeline-id 1234
        +Error: unknown flag: --pipeline-id
        +
         Usage:
        -  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Pipeline Test Flags:
        -      --junit-xml string         Write the results as a JUnit XML report to this file, replacing it.
        -      --no-wait                  Don't wait for the tests to complete.
        -      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        -      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        -      --resume string            Read the results of an existing test update instead of starting one.
        -      --timeout duration         Stop waiting for results after this duration, for example 30m. The update keeps running.
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
         Flags:
        -  -h, --help   help for test
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
         Global Flags:
               --debug            enable debug logging
           -o, --output type      output type: text or json (default text)
           -p, --profile string   ~/.databrickscfg profile
           -t, --target string    bundle target to use (if applicable)
        -      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
         
        -=== The command is experimental and off by default
        ->>> musterr [CLI] pipelines test --pipeline-id 1234
        -Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
        +Use "databricks pipelines [command] --help" for more information about a command.
         
        +
         === With the command off, an older report is still replaced
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --junit-xml out.gate-off.xml
        -Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
        +Error: unknown flag: --pipeline-id
         
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
        +
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
        +
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
... (818 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)

latest (v1.25.0): same output as main (688c228)

TestAccept/pipelines/test/immutable: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- pipelines/test/immutable/output.txt
        +++ /tmp/TestAcceptpipelinestestimmutable4000953062/002/output.txt
        @@ -1,53 +1,119 @@
         
         >>> [CLI] pipelines deploy
        +Warning: unknown field: testing
        +  at resources.pipelines.my_pipeline
        +  in databricks.yml:14:7
        +
         Created internal_immutable_snapshots.immutable
         Created pipelines.my_pipeline
         Files: 0 uploaded, 0 deleted
         Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID][0]?w=[WORKSPACE_ID]
        +View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]
         
         === Local test paths are not supported with workspace.sync_mode immutable
         >>> musterr [CLI] pipelines test my_pipeline tests/test_orders.py::test_backfill
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown command "test" for "databricks pipelines"
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Error: cannot resolve test selection "tests/test_orders.py::test_backfill": local test paths are not supported with workspace.sync_mode immutable; pass an absolute workspace path
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +
         === Absolute workspace paths and running all tests still work
         >>> MSYS_NO_PATHCONV=1 [CLI] pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown flag: --no-wait
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][1]
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        ->>> [CLI] pipelines test my_pipeline --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][2]
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill"
        -      ]
        -    }
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true
        -  }
        -}
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +
        +Exit code: 1

latest (v1.25.0): same output as main (688c228)

13 more failure details omitted, see the full report in the run.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 75f2713

Run: 42734876408

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 271 31 3:59
✅​ aws-windows-latest-4core-16gb 273 29 3:07
✅​ azure linux-2core-8gb 270 31 3:49
✅​ azure-windows-latest-4core-16gb 272 29 3:10
✅​ gcp linux-2core-8gb 271 31 4:16
✅​ gcp-windows-latest-4core-16gb 273 29 3:20
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:15 gcp-windows-latest-4core-16gb TestAccept
3:07 azure-windows-latest-4core-16gb TestAccept
3:04 aws-windows-latest-4core-16gb TestAccept

@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from ed5750c to 75f2713 Compare November 10, 2026 01:16
@tharunbhuk-db
tharunbhuk-db changed the base branch from tharun/pipelines-test-j1-base to tharun/pipelines-test-b3-dependency-checks November 10, 2026 01:16
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from 75f2713 to ccdd807 Compare November 10, 2026 10:07
@tharunbhuk-db
tharunbhuk-db changed the base branch from tharun/pipelines-test-b3-dependency-checks to tharun/pipelines-test-b4-testing-deploy November 10, 2026 10:07
This was referenced Nov 10, 2026
Open
Open
Open
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-b4-testing-deploy branch from 1fd5e11 to 3af4a58 Compare November 10, 2026 10:46
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch 2 times, most recently from 2e9b82e to a149708 Compare November 10, 2026 11:11
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-b4-testing-deploy branch from 3af4a58 to 7498e1b Compare November 10, 2026 16:49
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from a149708 to 490849a Compare November 10, 2026 16:49
tharunbhuk-db and others added 8 commits November 10, 2026 17:09
@tharunbhuk-db Isaac
Add the databricks pipelines test command
9be85fd 
The command starts a test-only update of the pipeline named by --pipeline-id
and prints its IDs. Selections are workspace paths or paths relative to the
pipeline's root_path, and are sent as typed so that the service checks them.
--pytest-arg accepts only -k and -m. The command is experimental and runs only
with DATABRICKS_CLI_PIPELINES_TEST=true.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db Isaac
Run a bundle pipeline's tests by its KEY
7d64487 
In a bundle, the first argument is the pipeline KEY, and relative test
selections are local paths that map to their deployed workspace paths. When the
first argument is not a pipeline KEY, the error names the bundle's pipelines and
shows the working command.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db Isaac
Read the test events of a pipeline update
53b3c58 
readTestEvents reads every page of an update's events and keeps the test fields
that the Go SDK event type does not have. It sends the SDK's
ListPipelineEventsRequest and uses the filter of buildPipelineEventFilter.
fetchAllPipelineEvents, which pipelines logs and run use, is not changed.
testRun collects one result per test case and reports whether the results are
complete. The command does not use them yet.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db Isaac
Print test progress and results
42a9409 
Each finished case prints "<i> of <n> <STATUS> <test ID>", padded with dots to
80 columns, then "[<STATUS> in <s>s]", so the statuses line up while each case
prints as it ends. After the run, the details of every failed or errored case
follow, then the reason for an incomplete run, and the run ends with
"Done. PASSED=… FAILED=… SKIPPED=… ERROR=… TOTAL=…". The command does not use
the printer yet.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db Isaac
Wait for and print the results of databricks pipelines test
d0d7138 
The command waits for the test-only update, prints each test case as it
finishes, then the failure details and a summary line. It exits with code 0
only for a complete run where at least one test ran and none failed or errored,
and a completed update that reports a fatal error does not pass. Test IDs are
shown relative to the project folder. With the no-wait flag, the command returns
after the update starts.

The wait takes the steps of the bundle pipeline runner's wait loop. The
acceptance tests stub the service per case, as the other pipelines tests do.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db Isaac
Add --resume and --timeout to databricks pipelines test
66654a5 
--timeout and Ctrl+C end only the local wait and print the command that reads
the results later; the update keeps running. That command keeps the KEY or
--pipeline-id, --target, --profile and every --var. --resume <update-id> reads
the results of an existing test update instead of starting one, and rejects an
update that the service reports is not a test update.

The signal handler is installed before the update starts, but only the wait
listens to it, so a Ctrl+C during Start Update still prints the update ID. The
first signal releases the handler, so a second one ends the process.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db Isaac
Build a JUnit XML report from the results of a test run
c7628c9 
The report has one testcase per collected test case, named as pytest's JUnit
report names it, with its outcome and duration. A case without a final outcome
is an error, and a run problem, such as an incomplete run, adds a failing entry.
A bracket in a folder name stays part of the path, and the file is replaced
atomically so that a reader never sees a partly written report. The command
does not write the report yet.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db Isaac
Write a JUnit XML report from databricks pipelines test
8b690b3 
--junit-xml writes one testcase per collected test case, with its outcome and
duration; an incomplete run, a run without tests, or a timeout adds a failing
entry that names the cause. The report replaces the file on every exit after
the flags parse, including the disabled-command error and project errors, so
an older report never survives; a flag that cannot be parsed ends the command
before it runs. --junit-xml cannot be combined with the no-wait flag, and the
printed resume command keeps --junit-xml so that it replaces a partial report.
The report line prints before the summary, so the summary stays the last line.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from 490849a to 6c31046 Compare November 10, 2026 17:14
tharunbhuk-db and others added 2 commits November 10, 2026 17:22
@tharunbhuk-db Isaac
Merge the pipelines test command stack (#7297 to #7434) until it merges
6bb1f2b 
Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db Isaac
Run pipelines test against the pipeline's testing.sources
bbb4adb 
The pipelines test acceptance bundles declare testing, as the service requires
testing.sources for a test-only update.

When Start Update fails with PIPELINE_TEST_NO_TEST_SOURCES and the bundle sets
testing for the pipeline, the error says to deploy first: the bundle changed
after its last deploy.

Add a live acceptance test that deploys a bundle with testing, including a local
test dependency after -e, and runs its tests. Locally it uses stubs for the
test run. Against a real workspace it runs only when
TEST_PIPELINES_TESTING_ENABLED is set.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-b4-testing-deploy branch from 7498e1b to e79333b Compare November 10, 2026 17:22
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from 6c31046 to bbb4adb Compare November 10, 2026 17:22
@tharunbhuk-db
tharunbhuk-db marked this pull request as ready for review November 10, 2026 18:11
@tharunbhuk-db
tharunbhuk-db requested review from a team as code owners November 10, 2026 18:11
@tharunbhuk-db
tharunbhuk-db requested review from andrewnester and removed request for a team and andrewnester November 10, 2026 18:11

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pranshupand-db pranshupand-db Awaiting requested review from pranshupand-db

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tharunbhuk-db @eng-dev-ecosystem-bot