[shreyas-goenka]

Remove the dependence on DMS for bundle generate. The command should purely rely on resources.json to generate configuration.

[github-actions]

Commit: 6e742798f | Workflow run

Regression test report

Selected 4 changed tests (limit=20, 0 not selected)

7 tests: regression=4 unreleased=0 coverage=3 failing=0

test	branch	main (5e52c4c)	latest (v1.22.1)
TestAccept/bundle/dms/bind-unbind/DMS=true	✅	❌	❌
TestAccept/bundle/generate/auto-bind/DMS=	✅	✅	✅
TestAccept/bundle/generate/auto-bind/DMS=true	✅	❌	❌
TestAccept/bundle/generate/dashboard-inplace/DMS=	✅	✅	✅
TestAccept/bundle/generate/dashboard-inplace/DMS=true	✅	❌	❌
TestAccept/bundle/generate/genie_space_inplace/DMS=	✅	✅	✅
TestAccept/bundle/generate/genie_space_inplace/DMS=true	✅	❌	❌

TestAccept/bundle/dms/bind-unbind/DMS=true: branch ✅ | main (5e52c4c) ❌ | latest (v1.22.1) ❌

main (5e52c4c):

acceptance_test.go:1341: Diff:
        --- bundle/dms/bind-unbind/output.txt
        +++ /tmp/TestAcceptbundledmsbind-unbindDMS=true3944315227/002/output.txt
        @@ -1,63 +1,7 @@
         
         === Bind an existing job with deployment history enabled
         >>> [CLI] bundle deployment bind foo [JOB_ID] --auto-approve
        -Successfully bound job with an id '[JOB_ID]'
        -Run 'bundle deploy' to deploy changes to your workspace
        +Error: bind is not supported for a bundle target that records deployment history
         
        -=== Deploy keeps the bound job ID and enables shadow history
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-bind-unbind-[UNIQUE_NAME]/default/files...
        -Updated jobs.foo
        -Files: 2 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
         
        ->>> jq {features, resource_id: .state["resources.jobs.foo"].__id__} .databricks/bundle/default/resources.json
        -{
        -  "features": {
        -    "deployment_history_shadow": {}
        -  },
        -  "resource_id": "[JOB_ID]"
        -}
        -
        -=== Unbind preserves the shadow history marker and leaves the job intact
        ->>> [CLI] bundle deployment unbind foo
        -
        ->>> jq {features, state} .databricks/bundle/default/resources.json
        -{
        -  "features": {
        -    "deployment_history_shadow": {}
        -  },
        -  "state": {}
        -}
        -
        ->>> [CLI] jobs get [JOB_ID]
        -{
        -  "job_id": [JOB_ID]
        -}
        -
        -=== Rebind preserves the existing shadow history marker
        ->>> [CLI] bundle deployment bind foo [JOB_ID] --auto-approve
        -Successfully bound job with an id '[JOB_ID]'
        -Run 'bundle deploy' to deploy changes to your workspace
        -
        ->>> jq {features, resource_id: .state["resources.jobs.foo"].__id__} .databricks/bundle/default/resources.json
        -{
        -  "features": {
        -    "deployment_history_shadow": {}
        -  },
        -  "resource_id": "[JOB_ID]"
        -}
        -
        -=== A subsequent deploy continues managing the same job
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-bind-unbind-[UNIQUE_NAME]/default/files...
        -Files: 0 uploaded, 0 deleted
        -Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
        -
        ->>> jq {features, resource_id: .state["resources.jobs.foo"].__id__} .databricks/bundle/default/resources.json
        -{
        -  "features": {
        -    "deployment_history_shadow": {}
        -  },
        -  "resource_id": "[JOB_ID]"
        -}
        +Exit code: 1

latest (v1.22.1): same output as main (5e52c4c)

TestAccept/bundle/generate/auto-bind/DMS=true: branch ✅ | main (5e52c4c) ❌ | latest (v1.22.1) ❌

main (5e52c4c):

acceptance_test.go:1341: Diff:
        --- bundle/generate/auto-bind/output.txt
        +++ /tmp/TestAcceptbundlegenerateauto-bindDMS=true2565421622/002/output.txt
        @@ -10,37 +10,12 @@
         >>> [CLI] bundle generate job --key test_job --existing-job-id [JOB_ID] --config-dir resources --source-dir src --bind
         File successfully saved to src/test.py
         Job configuration successfully saved to resources/test_job.job.yml
        -Successfully bound job with an id '[JOB_ID]'
        +Error: bind is not supported for a bundle target that records deployment history
         
        ->>> ls src/
        -test.py
         
        ->>> cat resources/test_job.job.yml
        -      name: auto-bind-job-[UNIQUE_NAME]
        -
        -=== Deploy the bound job:
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/auto-bind-test-[UNIQUE_NAME]/files...
        -Updated jobs.test_job
        -Files: 5 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        -
        -=== Destroy the bundle:
        ->>> [CLI] bundle destroy --auto-approve
        -The following resources will be deleted:
        -  delete resources.jobs.test_job
        -
        -All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/auto-bind-test-[UNIQUE_NAME]
        -
        -Destroy: 1 deleted
        -
        -=== Check that job is bound and does not exist after bundle is destroyed:
        ->>> errcode [CLI] jobs get [JOB_ID] --output json
        -Error: Job [JOB_ID] does not exist.
        -
        -Exit code: 1
        -
         === Delete the tmp folder:
         >>> [CLI] workspace delete /Workspace/Users/[USERNAME]/python-[UNIQUE_NAME]/test
         
         >>> [CLI] workspace delete /Workspace/Users/[USERNAME]/python-[UNIQUE_NAME]
        +
        +Exit code: 1

latest (v1.22.1):

acceptance_test.go:1341: Diff:
        --- bundle/generate/auto-bind/output.txt
        +++ /tmp/TestAcceptbundlegenerateauto-bindDMS=true2223194572/002/output.txt
        @@ -10,37 +10,12 @@
         >>> [CLI] bundle generate job --key test_job --existing-job-id [JOB_ID] --config-dir resources --source-dir src --bind
         File successfully saved to src/test.py
         Job configuration successfully saved to resources/test_job.job.yml
        -Successfully bound job with an id '[JOB_ID]'
        +Error: bind is not supported for a bundle target that records deployment history
         
        ->>> ls src/
        -test.py
         
        ->>> cat resources/test_job.job.yml
        -      name: auto-bind-job-[UNIQUE_NAME]
        -
        -=== Deploy the bound job:
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/auto-bind-test-[UNIQUE_NAME]/files...
        -Updated jobs.test_job
        -Files: 5 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        -
        -=== Destroy the bundle:
        ->>> [CLI] bundle destroy --auto-approve
        -The following resources will be deleted:
        -  delete resources.jobs.test_job
        -
        -All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/auto-bind-test-[UNIQUE_NAME]
        -
        -Destroy: 1 deleted
        -
        -=== Check that job is bound and does not exist after bundle is destroyed:
        ->>> errcode [CLI] jobs get [JOB_ID] --output json
        -Error: Job [JOB_ID] does not exist.
        -
        -Exit code: 1
        -
         === Delete the tmp folder:
         >>> [CLI] workspace delete /Workspace/Users/[USERNAME]/python-[UNIQUE_NAME]/test
         
         >>> [CLI] workspace delete /Workspace/Users/[USERNAME]/python-[UNIQUE_NAME]
        +
        +Exit code: 1
        
FAIL

TestAccept/bundle/generate/dashboard-inplace/DMS=true: branch ✅ | main (5e52c4c) ❌ | latest (v1.22.1) ❌

main (5e52c4c):

acceptance_test.go:1341: Diff:
        --- bundle/generate/dashboard-inplace/output.txt
        +++ /tmp/TestAcceptbundlegeneratedashboard-inplaceDMS=true2578388193/002/output.txt
        @@ -28,6 +28,7 @@
         
         === update the dashboard file using bundle generate
         >>> [CLI] bundle generate dashboard --resource test_dashboard --force
        +Warn: Could not read deployment history: Fault injected by test.
         Writing dashboard to dash.lvdash.json
         
         >>> cat dash.lvdash.json
        @@ -36,3 +37,4 @@
         }
         
         >>> print_requests.py --dms //api/2.0/bundle --get --oneline
        +{"method": "GET", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]"}

latest (v1.22.1):

acceptance_test.go:1341: Diff:
        --- bundle/generate/dashboard-inplace/output.txt
        +++ /tmp/TestAcceptbundlegeneratedashboard-inplaceDMS=true634972360/002/output.txt
        @@ -28,11 +28,12 @@
         
         === update the dashboard file using bundle generate
         >>> [CLI] bundle generate dashboard --resource test_dashboard --force
        -Writing dashboard to dash.lvdash.json
        +Error: Fault injected by test. (403 PERMISSION_DENIED)
         
        ->>> cat dash.lvdash.json
        -{
        -  "a": "b"
        -}
        +Endpoint: GET [DATABRICKS_URL]/api/2.0/bundle/deployments/[DEPLOYMENT_ID]?
        +HTTP Status: 403 Forbidden
        +API error_code: PERMISSION_DENIED
        +API message: Fault injected by test.
         
        ->>> print_requests.py --dms //api/2.0/bundle --get --oneline
        +
        +Exit code: 1

TestAccept/bundle/generate/genie_space_inplace/DMS=true: branch ✅ | main (5e52c4c) ❌ | latest (v1.22.1) ❌

main (5e52c4c):

acceptance_test.go:1341: Diff:
        --- bundle/generate/genie_space_inplace/output.txt
        +++ /tmp/TestAcceptbundlegenerategenie_space_inplaceDMS=true1415542897/002/output.txt
        @@ -24,6 +24,7 @@
         
         === update the genie space file using bundle generate
         >>> [CLI] bundle generate genie-space --resource test_genie_space --force
        +Warn: Could not read deployment history: Fault injected by test.
         Writing genie space to space.geniespace.json
         
         >>> cat space.geniespace.json
        @@ -32,3 +33,4 @@
         }
         
         >>> print_requests.py --dms //api/2.0/bundle --get --oneline
        +{"method": "GET", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]"}
        
FAIL

latest (v1.22.1):

acceptance_test.go:1341: Diff:
        --- bundle/generate/genie_space_inplace/output.txt
        +++ /tmp/TestAcceptbundlegenerategenie_space_inplaceDMS=true3240179868/002/output.txt
        @@ -24,11 +24,12 @@
         
         === update the genie space file using bundle generate
         >>> [CLI] bundle generate genie-space --resource test_genie_space --force
        -Writing genie space to space.geniespace.json
        +Error: Fault injected by test. (403 PERMISSION_DENIED)
         
        ->>> cat space.geniespace.json
        -{
        -  "a": "b"
        -}
        +Endpoint: GET [DATABRICKS_URL]/api/2.0/bundle/deployments/[DEPLOYMENT_ID]?
        +HTTP Status: 403 Forbidden
        +API error_code: PERMISSION_DENIED
        +API message: Fault injected by test.
         
        ->>> print_requests.py --dms //api/2.0/bundle --get --oneline
        +
        +Exit code: 1

[eng-dev-ecosystem-bot]

Integration test report

Commit: 6e74279

Run: 40611767407

	Env	✅​pass	🙈​skip	Time
✅​	aws linux-2core-8gb	284	18	5:43
✅​	aws-windows-latest-4core-16gb	286	16	3:43
✅​	azure linux-2core-8gb	283	18	5:43
✅​	azure-windows-latest-4core-16gb	285	16	3:57
✅​	gcp linux-2core-8gb	284	18	5:47
✅​	gcp-windows-latest-4core-16gb	286	16	3:55

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
4:09	aws linux-2core-8gb	TestAccept
3:56	azure linux-2core-8gb	TestAccept
3:54	gcp linux-2core-8gb	TestAccept
3:24	gcp-windows-latest-4core-16gb	TestAccept
3:23	azure-windows-latest-4core-16gb	TestAccept
3:12	aws-windows-latest-4core-16gb	TestAccept