Record bundle behavior with faults for every DMS API by shreyas-goenka · Pull Request #7231 · databricks/cli

shreyas-goenka · 2026-10-28T14:38:28Z

Changes

Procedurally generate and run tests for all DMS APIs to ensure that the deployment history feature is truly best effort and DMS going down will not cause deployments to start failing.

This tests also ensure we display adequate warnings when the errors actually happen.

github-actions · 2026-10-28T14:45:03Z

Commit: 260fb4e3a | Workflow run

Regression test report

Selected 15 changed tests (limit=20, 0 not selected)

15 tests: regression=0 unreleased=15 coverage=0 failing=0

test	branch	main (`e9687c1`)	latest (v1.22.1)
TestAccept/bundle/dms_fault/complete_version/DMS=true	✅	✅	❌
TestAccept/bundle/dms_fault/create_deployment/DMS=true	✅	✅	❌
TestAccept/bundle/dms_fault/create_version/DMS=true	✅	✅	❌
TestAccept/bundle/dms_fault/delete_deployment/DMS=true	✅	✅	❌
TestAccept/bundle/dms_fault/get_deployment/DMS=true	✅	✅	❌
TestAccept/bundle/dms_fault/get_operation/DMS=true	✅	✅	❌
TestAccept/bundle/dms_fault/get_resource/DMS=true	✅	✅	❌
TestAccept/bundle/dms_fault/get_version/DMS=true	✅	✅	❌
TestAccept/bundle/dms_fault/heartbeat/DMS=true	✅	✅	❌
TestAccept/bundle/dms_fault/list_deployments/DMS=true	✅	✅	❌
TestAccept/bundle/dms_fault/list_operations/DMS=true	✅	✅	❌
TestAccept/bundle/dms_fault/list_resources/DMS=true	✅	✅	❌
TestAccept/bundle/dms_fault/list_versions/DMS=true	✅	✅	❌
TestAccept/bundle/dms_fault/update_deployment/DMS=true	✅	✅	❌
TestAccept/bundle/dms_fault/update_operation/DMS=true	✅	✅	❌

TestAccept/bundle/dms_fault/complete_version/DMS=true: branch ✅ | main (e9687c1) ✅ | latest (v1.22.1) ❌

latest (v1.22.1):

acceptance_test.go:1341: Diff:
        --- bundle/dms_fault/complete_version/output.txt
        +++ /tmp/TestAcceptbundledms_faultcomplete_versionDMS=true3824866760/002/output.txt
        @@ -18,131 +18,35 @@
           Updated pipelines.bar
           Files: 3 uploaded, 0 deleted
           Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
        -  Warn: Failed to complete deployment history version: Fault injected by test.
        +  Error: Fault injected by test. (500 INJECTED)
         
        ->>> bundle summary
        -  Name: updated-dms-fault-complete_version-commands-[UNIQUE_NAME]
        -  Target: default
        -  Workspace:
        -    User: [USERNAME]
        -    Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]
        -  Resources:
        -    Jobs:
        -      foo:
        -        Name: updated-dms-fault-complete_version-commands-[UNIQUE_NAME]
        -        URL:  [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID]
        -    Pipelines:
        -      bar:
        -        Name: updated-dms-fault-complete_version-commands-[UNIQUE_NAME]
        -        URL:  [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID]
        +  Endpoint: POST [DATABRICKS_URL]/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/complete
        +  HTTP Status: 500 Internal Server Error
        +  API error_code: INJECTED
        +  API message: Fault injected by test.
         
        ->>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force
        -  Job configuration successfully saved to exported/foo.job.yml
        -  Warning: Generated configuration is not included in the bundle
        +Traceback (most recent call last):
        +  File "[TESTROOT]/bundle/dms_fault/complete_version/../run.py", line 248, in <module>
        +    main()
        +    ~~~~^^
        +  File "[TESTROOT]/bundle/dms_fault/complete_version/../run.py", line 194, in main
        +    run(cwd, "bundle", "deploy", "--auto-approve")
        +    ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        +  File "[TESTROOT]/bundle/dms_fault/complete_version/../run.py", line 64, in run
        +    assert result.returncode == expected, output
        +           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        +AssertionError: Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]/files...
        +Updated pipelines.bar
        +Updated jobs.foo
        +Files: 3 uploaded, 0 deleted
        +Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
        +Error: Fault injected by test. (500 INJECTED)
         
        -  The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml,
        -  so it will not be deployed. Add a matching entry to the 'include' section, for example:
        +Endpoint: POST [DATABRICKS_URL]/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/complete
        +HTTP Status: 500 Internal Server Error
        +API error_code: INJECTED
        +API message: Fault injected by test.
         
        -  include:
        -    - exported/*.yml
         
         
        ->>> bundle deployment unbind foo
        -
        ->>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind
        -  Job configuration successfully saved to resources/foo.job.yml
        -  Successfully bound job with an id '[FOO_ID]'
        -
        ->>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force
        -  File successfully saved to exported-src/pipeline.sql
        -  Pipeline configuration successfully saved to exported/bar.pipeline.yml
        -  Warning: Generated configuration is not included in the bundle
        -
        -  The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml,
        -  so it will not be deployed. Add a matching entry to the 'include' section, for example:
        -
        -  include:
        -    - exported/*.yml
        -
        -
        ->>> bundle deployment unbind bar
        -
        ->>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind
        -  File successfully saved to src/pipeline.sql
        -  Pipeline configuration successfully saved to resources/bar.pipeline.yml
        -  Successfully bound pipeline with an id '[BAR_ID]'
        -
        ->>> bundle validate
        -  Name: updated-dms-fault-complete_version-commands-[UNIQUE_NAME]
        -  Target: default
        -  Workspace:
        -    User: [USERNAME]
        -    Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]
        -
        -  Validation OK!
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]/files...
        -  Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=9
        -  Updated pipelines.bar
        -  Files: 6 uploaded, 0 deleted
        -  Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
        -  Warn: Failed to complete deployment history version: Fault injected by test.
        -
        -=== deployment_history: false
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]/files...
        -  Updated jobs.foo
        -  Updated pipelines.bar
        -  Files: 2 uploaded, 0 deleted
        -  Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
        -
        -=== deployment_history: true
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]/files...
        -  Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=11
        -  Updated jobs.foo
        -  Updated pipelines.bar
        -  Files: 2 uploaded, 0 deleted
        -  Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
        -  Warn: Failed to complete deployment history version: Fault injected by test.
        -
        ->>> bundle destroy --auto-approve
        -  The following resources will be deleted:
        -    delete resources.jobs.foo
        -    delete resources.pipelines.bar
        -
        -  This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
        -  Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
        -    delete resources.pipelines.bar
        -
        -  All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]
        -
        -  Warn: Failed to complete deployment history version: Fault injected by test.
        -  Destroy: 2 deleted
        -
        -=== deployment_history: false
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]/files...
        -  Created jobs.foo
        -  Created pipelines.bar
        -  Files: 8 uploaded, 0 deleted
        -  Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> bundle destroy --auto-approve
        -  The following resources will be deleted:
        -    delete resources.jobs.foo
        -    delete resources.pipelines.bar
        -
        -  This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
        -  Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
        -    delete resources.pipelines.bar
        -
        -  All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]
        -
        -  Destroy: 2 deleted
        -
        -DMS API called: yes
        +Exit code: 1
        
    acceptance_test.go:1119: 
        LOG.commands: 
        LOG.commands: commands: ('bundle', 'deploy', '--auto-approve')
        LOG.commands: exit=0
        LOG.commands: Uploading bundle files to /Workspace/Users/tester@databricks.com/.bundle/dms-fault-complete_version-commands-k6hrayo4kbajdcjm5epp32zfcy/files...
        LOG.commands: Current Deployment Version: http://127.0.0.1:44021/deployments/1037?version=1
        LOG.commands: Created pipelines.bar
        LOG.commands: Created jobs.foo
        LOG.commands: Files: 4 uploaded, 0 deleted
        LOG.commands: Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        LOG.commands: 
        LOG.commands: commands: ('workspace', 'get-status', '/Workspace/Users/tester@databricks.com/.bundle/dms-fault-complete_version-commands-k6hrayo4kbajdcjm5epp32zfcy/state/resources.deployment.json', '-o', 'json')
        LOG.commands: exit=0
        LOG.commands: {
        LOG.commands:   "object_id": 1037,
        LOG.commands:   "object_type": "BUNDLE_DEPLOYMENT",
        LOG.commands:   "path": "/Workspace/Users/tester@databricks.com/.bundle/dms-fault-complete_version-commands-k6hrayo4kbajdcjm5epp32zfcy/state/resources.deployment.json"
        LOG.commands: }
        LOG.commands: 
        LOG.commands: commands: ('bundle', 'deploy', '--auto-approve')
        LOG.commands: exit=1
        LOG.commands: Uploading bundle files to /Workspace/Users/tester@databricks.com/.bundle/dms-fault-complete_version-commands-k6hrayo4kbajdcjm5epp32zfcy/files...
        LOG.commands: Current Deployment Version: http://127.0.0.1:44021/deployments/1037?version=2
        LOG.commands: Updated pipelines.bar
        LOG.commands: Updated jobs.foo
        LOG.commands: Files: 3 uploaded, 0 deleted
        LOG.commands: Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
        LOG.commands: Error: Fault injected by test. (500 INJECTED)
        LOG.commands: 
        LOG.commands: Endpoint: POST http://127.0.0.1:44021/api/2.0/bundle/deployments/1037/versions/2/complete
        LOG.commands: HTTP Status: 500 Internal Server Error
        LOG.commands: API error_code: INJECTED
        LOG.commands: API message: Fault injected by test.

TestAccept/bundle/dms_fault/create_deployment/DMS=true: branch ✅ | main (e9687c1) ✅ | latest (v1.22.1) ❌

latest (v1.22.1):

acceptance_test.go:1341: Diff:
        --- bundle/dms_fault/create_deployment/output.txt
        +++ /tmp/TestAcceptbundledms_faultcreate_deploymentDMS=true3542670555/002/output.txt
        @@ -5,131 +5,33 @@
         
         >>> bundle deploy --auto-approve
           Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]/files...
        -  Warn: Could not enable deployment history for this deploy: failed to create deployment: Fault injected by test.
        -  Created jobs.foo
        -  Created pipelines.bar
        -  Files: 4 uploaded, 0 deleted
        -  Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        +  Error: failed to create deployment: Fault injected by test. (500 INJECTED)
         
        ->>> bundle summary
        -  Name: updated-dms-fault-create_deployment-commands-[UNIQUE_NAME]
        -  Target: default
        -  Workspace:
        -    User: [USERNAME]
        -    Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]
        -  Resources:
        -    Jobs:
        -      foo:
        -        Name: updated-dms-fault-create_deployment-commands-[UNIQUE_NAME]
        -        URL:  [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID]
        -    Pipelines:
        -      bar:
        -        Name: updated-dms-fault-create_deployment-commands-[UNIQUE_NAME]
        -        URL:  [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID]
        +  Endpoint: POST [DATABRICKS_URL]/api/2.0/bundle/deployments
        +  HTTP Status: 500 Internal Server Error
        +  API error_code: INJECTED
        +  API message: Fault injected by test.
         
        ->>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force
        -  Job configuration successfully saved to exported/foo.job.yml
        -  Warning: Generated configuration is not included in the bundle
        +  Files: 4 uploaded, 0 deleted
        +Traceback (most recent call last):
        +  File "[TESTROOT]/bundle/dms_fault/create_deployment/../run.py", line 248, in <module>
        +    main()
        +    ~~~~^^
        +  File "[TESTROOT]/bundle/dms_fault/create_deployment/../run.py", line 194, in main
        +    run(cwd, "bundle", "deploy", "--auto-approve")
        +    ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        +  File "[TESTROOT]/bundle/dms_fault/create_deployment/../run.py", line 64, in run
        +    assert result.returncode == expected, output
        +           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        +AssertionError: Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]/files...
        +Error: failed to create deployment: Fault injected by test. (500 INJECTED)
         
        -  The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml,
        -  so it will not be deployed. Add a matching entry to the 'include' section, for example:
        +Endpoint: POST [DATABRICKS_URL]/api/2.0/bundle/deployments
        +HTTP Status: 500 Internal Server Error
        +API error_code: INJECTED
        +API message: Fault injected by test.
         
        -  include:
        -    - exported/*.yml
        +Files: 4 uploaded, 0 deleted
         
         
        ->>> bundle deployment unbind foo
        -
        ->>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind
        -  Job configuration successfully saved to resources/foo.job.yml
        -  Successfully bound job with an id '[FOO_ID]'
        -
        ->>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force
        -  File successfully saved to exported-src/pipeline.sql
        -  Pipeline configuration successfully saved to exported/bar.pipeline.yml
        -  Warning: Generated configuration is not included in the bundle
        -
        -  The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml,
        -  so it will not be deployed. Add a matching entry to the 'include' section, for example:
        -
        -  include:
        -    - exported/*.yml
        -
        -
        ->>> bundle deployment unbind bar
        -
        ->>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind
        -  File successfully saved to src/pipeline.sql
        -  Pipeline configuration successfully saved to resources/bar.pipeline.yml
        -  Successfully bound pipeline with an id '[BAR_ID]'
        -
        ->>> bundle validate
        -  Name: updated-dms-fault-create_deployment-commands-[UNIQUE_NAME]
        -  Target: default
        -  Workspace:
        -    User: [USERNAME]
        -    Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]
        -
        -  Validation OK!
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]/files...
        -  Warn: Could not enable deployment history for this deploy: failed to create deployment: Fault injected by test.
        -  Updated pipelines.bar
        -  Files: 6 uploaded, 0 deleted
        -  Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
        -
        -=== deployment_history: false
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]/files...
        -  Updated jobs.foo
        -  Files: 2 uploaded, 0 deleted
        -  Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
        -
        -=== deployment_history: true
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]/files...
        -  Warn: Could not enable deployment history for this deploy: failed to create deployment: Fault injected by test.
        -  Updated jobs.foo
        -  Files: 2 uploaded, 0 deleted
        -  Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
        -
        ->>> bundle destroy --auto-approve
        -  The following resources will be deleted:
        -    delete resources.jobs.foo
        -    delete resources.pipelines.bar
        -
        -  This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
        -  Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
        -    delete resources.pipelines.bar
        -
        -  All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]
        -
        -  Warn: Could not record deployment history for this destroy: failed to create deployment: Fault injected by test.
        -  Destroy: 2 deleted
        -
        -=== deployment_history: false
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]/files...
        -  Created jobs.foo
        -  Created pipelines.bar
        -  Files: 8 uploaded, 0 deleted
        -  Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> bundle destroy --auto-approve
        -  The following resources will be deleted:
        -    delete resources.jobs.foo
        -    delete resources.pipelines.bar
        -
        -  This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
        -  Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
        -    delete resources.pipelines.bar
        -
        -  All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]
        -
        -  Destroy: 2 deleted
        -
        -DMS API called: yes
        +Exit code: 1
        
    acceptance_test.go:1119: 
        LOG.commands: 
        LOG.commands: commands: ('bundle', 'deploy', '--auto-approve')
        LOG.commands: exit=1
        LOG.commands: Uploading bundle files to /Workspace/Users/tester@databricks.com/.bundle/dms-fault-create_deployment-commands-6mg3usvluzepljstrg5hyay7jm/files...
        LOG.commands: Error: failed to create deployment: Fault injected by test. (500 INJECTED)
        LOG.commands: 
        LOG.commands: Endpoint: POST http://127.0.0.1:32853/api/2.0/bundle/deployments
        LOG.commands: HTTP Status: 500 Internal Server Error
        LOG.commands: API error_code: INJECTED
        LOG.commands: API message: Fault injected by test.
        LOG.commands: 
        LOG.commands: Files: 4 uploaded, 0 deleted
FAIL

TestAccept/bundle/dms_fault/create_version/DMS=true: branch ✅ | main (e9687c1) ✅ | latest (v1.22.1) ❌

latest (v1.22.1):

acceptance_test.go:1341: Diff:
        --- bundle/dms_fault/create_version/output.txt
        +++ /tmp/TestAcceptbundledms_faultcreate_versionDMS=true3152750158/002/output.txt
        @@ -13,133 +13,33 @@
         
         >>> bundle deploy --auto-approve
           Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME]/files...
        -  Warn: Could not enable deployment history for this deploy: failed to create deployment version: Fault injected by test.
        -  Updated jobs.foo
        -  Updated pipelines.bar
        -  Files: 3 uploaded, 0 deleted
        -  Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
        +  Error: failed to create deployment version: Fault injected by test. (500 INJECTED)
         
        ->>> bundle summary
        -  Name: updated-dms-fault-create_version-commands-[UNIQUE_NAME]
        -  Target: default
        -  Workspace:
        -    User: [USERNAME]
        -    Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME]
        -  Resources:
        -    Jobs:
        -      foo:
        -        Name: updated-dms-fault-create_version-commands-[UNIQUE_NAME]
        -        URL:  [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID]
        -    Pipelines:
        -      bar:
        -        Name: updated-dms-fault-create_version-commands-[UNIQUE_NAME]
        -        URL:  [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID]
        +  Endpoint: POST [DATABRICKS_URL]/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions?version_id=2
        +  HTTP Status: 500 Internal Server Error
        +  API error_code: INJECTED
        +  API message: Fault injected by test.
         
        ->>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force
        -  Job configuration successfully saved to exported/foo.job.yml
        -  Warning: Generated configuration is not included in the bundle
        +  Files: 3 uploaded, 0 deleted
        +Traceback (most recent call last):
        +  File "[TESTROOT]/bundle/dms_fault/create_version/../run.py", line 248, in <module>
        +    main()
        +    ~~~~^^
        +  File "[TESTROOT]/bundle/dms_fault/create_version/../run.py", line 194, in main
        +    run(cwd, "bundle", "deploy", "--auto-approve")
        +    ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        +  File "[TESTROOT]/bundle/dms_fault/create_version/../run.py", line 64, in run
        +    assert result.returncode == expected, output
        +           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        +AssertionError: Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME]/files...
        +Error: failed to create deployment version: Fault injected by test. (500 INJECTED)
         
        -  The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml,
        -  so it will not be deployed. Add a matching entry to the 'include' section, for example:
        +Endpoint: POST [DATABRICKS_URL]/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions?version_id=2
        +HTTP Status: 500 Internal Server Error
        +API error_code: INJECTED
        +API message: Fault injected by test.
         
        -  include:
        -    - exported/*.yml
        +Files: 3 uploaded, 0 deleted
         
         
        ->>> bundle deployment unbind foo
        -
        ->>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind
        -  Job configuration successfully saved to resources/foo.job.yml
        -  Successfully bound job with an id '[FOO_ID]'
        -
        ->>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force
        -  File successfully saved to exported-src/pipeline.sql
        -  Pipeline configuration successfully saved to exported/bar.pipeline.yml
        -  Warning: Generated configuration is not included in the bundle
        -
        -  The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml,
        -  so it will not be deployed. Add a matching entry to the 'include' section, for example:
        -
        -  include:
        -    - exported/*.yml
        -
        -
        ->>> bundle deployment unbind bar
        -
        ->>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind
        -  File successfully saved to src/pipeline.sql
        -  Pipeline configuration successfully saved to resources/bar.pipeline.yml
        -  Successfully bound pipeline with an id '[BAR_ID]'
        -
        ->>> bundle validate
        -  Name: updated-dms-fault-create_version-commands-[UNIQUE_NAME]
        -  Target: default
        -  Workspace:
        -    User: [USERNAME]
        -    Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME]
        -
        -  Validation OK!
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME]/files...
        -  Warn: Could not enable deployment history for this deploy: failed to create deployment version: Fault injected by test.
        -  Updated pipelines.bar
        -  Files: 6 uploaded, 0 deleted
        -  Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
        -
        -=== deployment_history: false
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME]/files...
        -  Updated jobs.foo
        -  Updated pipelines.bar
        -  Files: 2 uploaded, 0 deleted
        -  Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
        -
        -=== deployment_history: true
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME]/files...
        -  Warn: Could not enable deployment history for this deploy: failed to create deployment version: Fault injected by test.
        -  Updated jobs.foo
        -  Updated pipelines.bar
        -  Files: 2 uploaded, 0 deleted
        -  Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
        -
        ->>> bundle destroy --auto-approve
        -  The following resources will be deleted:
        -    delete resources.jobs.foo
        -    delete resources.pipelines.bar
        -
        -  This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
        -  Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
        -    delete resources.pipelines.bar
        -
        -  All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME]
        -
        -  Warn: Could not record deployment history for this destroy: failed to create deployment version: Fault injected by test.
        -  Destroy: 2 deleted
        -
        -=== deployment_history: false
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME]/files...
        -  Created jobs.foo
        -  Created pipelines.bar
        -  Files: 8 uploaded, 0 deleted
        -  Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> bundle destroy --auto-approve
        -  The following resources will be deleted:
        -    delete resources.jobs.foo
        -    delete resources.pipelines.bar
        -
        -  This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
        -  Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
        -    delete resources.pipelines.bar
        -
        -  All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_version-commands-[UNIQUE_NAME]
        -
        -  Destroy: 2 deleted
        -
        -DMS API called: yes
        +Exit code: 1
        
    acceptance_test.go:1119: 
        LOG.commands: 
        LOG.commands: commands: ('bundle', 'deploy', '--auto-approve')
        LOG.commands: exit=0
        LOG.commands: Uploading bundle files to /Workspace/Users/tester@databricks.com/.bundle/dms-fault-create_version-commands-d57lyaajorbqxbxei6aqwlcroq/files...
        LOG.commands: Current Deployment Version: http://127.0.0.1:43373/deployments/1381?version=1
        LOG.commands: Created jobs.foo
        LOG.commands: Created pipelines.bar
        LOG.commands: Files: 4 uploaded, 0 deleted
        LOG.commands: Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        LOG.commands: 
        LOG.commands: commands: ('workspace', 'get-status', '/Workspace/Users/tester@databricks.com/.bundle/dms-fault-create_version-commands-d57lyaajorbqxbxei6aqwlcroq/state/resources.deployment.json', '-o', 'json')
        LOG.commands: exit=0
        LOG.commands: {
        LOG.commands:   "object_id": 1381,
        LOG.commands:   "object_type": "BUNDLE_DEPLOYMENT",
        LOG.commands:   "path": "/Workspace/Users/tester@databricks.com/.bundle/dms-fault-create_version-commands-d57lyaajorbqxbxei6aqwlcroq/state/resources.deployment.json"
        LOG.commands: }
        LOG.commands: 
        LOG.commands: commands: ('bundle', 'deploy', '--auto-approve')
        LOG.commands: exit=1
        LOG.commands: Uploading bundle files to /Workspace/Users/tester@databricks.com/.bundle/dms-fault-create_version-commands-d57lyaajorbqxbxei6aqwlcroq/files...
        LOG.commands: Error: failed to create deployment version: Fault injected by test. (500 INJECTED)
        LOG.commands: 
        LOG.commands: Endpoint: POST http://127.0.0.1:43373/api/2.0/bundle/deployments/1381/versions?version_id=2
        LOG.commands: HTTP Status: 500 Internal Server Error
        LOG.commands: API error_code: INJECTED
        LOG.commands: API message: Fault injected by test.
        LOG.commands: 
        LOG.commands: Files: 3 uploaded, 0 deleted

TestAccept/bundle/dms_fault/delete_deployment/DMS=true: branch ✅ | main (e9687c1) ✅ | latest (v1.22.1) ❌

latest (v1.22.1):

acceptance_test.go:1341: Diff:
        --- bundle/dms_fault/delete_deployment/output.txt
        +++ /tmp/TestAcceptbundledms_faultdelete_deploymentDMS=true513265723/002/output.txt
        @@ -18,130 +18,15 @@
           Updated pipelines.bar
           Files: 3 uploaded, 0 deleted
           Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
        +Traceback (most recent call last):
        +  File "[TESTROOT]/bundle/dms_fault/delete_deployment/../run.py", line 248, in <module>
        +    main()
        +    ~~~~^^
        +  File "[TESTROOT]/bundle/dms_fault/delete_deployment/../run.py", line 195, in main
        +    current = ids(cwd)
        +  File "[TESTROOT]/bundle/dms_fault/delete_deployment/../run.py", line 91, in ids
        +    assert ("deployment_history_shadow" in features) == history, data
        +           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        +AssertionError: {'state_version': 3, 'cli_version': '[CLI_VERSION]', 'lineage': '[UUID]', 'serial': 2, 'features': {'deployment_history': {}}, 'state': {}}
         
        ->>> bundle summary
        -  Name: updated-dms-fault-delete_deployment-commands-[UNIQUE_NAME]
        -  Target: default
        -  Workspace:
        -    User: [USERNAME]
        -    Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME]
        -  Resources:
        -    Jobs:
        -      foo:
        -        Name: updated-dms-fault-delete_deployment-commands-[UNIQUE_NAME]
        -        URL:  [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID]
        -    Pipelines:
        -      bar:
        -        Name: updated-dms-fault-delete_deployment-commands-[UNIQUE_NAME]
        -        URL:  [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID]
        -
        ->>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force
        -  Job configuration successfully saved to exported/foo.job.yml
        -  Warning: Generated configuration is not included in the bundle
        -
        -  The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml,
        -  so it will not be deployed. Add a matching entry to the 'include' section, for example:
        -
        -  include:
        -    - exported/*.yml
        -
        -
        ->>> bundle deployment unbind foo
        -
        ->>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind
        -  Job configuration successfully saved to resources/foo.job.yml
        -  Successfully bound job with an id '[FOO_ID]'
        -
        ->>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force
        -  File successfully saved to exported-src/pipeline.sql
        -  Pipeline configuration successfully saved to exported/bar.pipeline.yml
        -  Warning: Generated configuration is not included in the bundle
        -
        -  The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml,
        -  so it will not be deployed. Add a matching entry to the 'include' section, for example:
        -
        -  include:
        -    - exported/*.yml
        -
        -
        ->>> bundle deployment unbind bar
        -
        ->>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind
        -  File successfully saved to src/pipeline.sql
        -  Pipeline configuration successfully saved to resources/bar.pipeline.yml
        -  Successfully bound pipeline with an id '[BAR_ID]'
        -
        ->>> bundle validate
        -  Name: updated-dms-fault-delete_deployment-commands-[UNIQUE_NAME]
        -  Target: default
        -  Workspace:
        -    User: [USERNAME]
        -    Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME]
        -
        -  Validation OK!
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME]/files...
        -  Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=9
        -  Updated pipelines.bar
        -  Files: 6 uploaded, 0 deleted
        -  Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
        -
        -=== deployment_history: false
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME]/files...
        -  Updated jobs.foo
        -  Updated pipelines.bar
        -  Files: 2 uploaded, 0 deleted
        -  Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
        -
        -=== deployment_history: true
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME]/files...
        -  Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=11
        -  Updated jobs.foo
        -  Updated pipelines.bar
        -  Files: 2 uploaded, 0 deleted
        -  Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
        -
        ->>> bundle destroy --auto-approve
        -  The following resources will be deleted:
        -    delete resources.jobs.foo
        -    delete resources.pipelines.bar
        -
        -  This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
        -  Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
        -    delete resources.pipelines.bar
        -
        -  All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME]
        -
        -  Warn: retrying after 500 Internal Server Error from DELETE /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
        -  Warn: retrying after 500 Internal Server Error from DELETE /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
        -  Warn: Could not delete deployment history: Fault injected by test.
        -  Destroy: 2 deleted
        -
        -=== deployment_history: false
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME]/files...
        -  Created jobs.foo
        -  Created pipelines.bar
        -  Files: 8 uploaded, 0 deleted
        -  Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> bundle destroy --auto-approve
        -  The following resources will be deleted:
        -    delete resources.jobs.foo
        -    delete resources.pipelines.bar
        -
        -  This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
        -  Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
        -    delete resources.pipelines.bar
        -
        -  All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-delete_deployment-commands-[UNIQUE_NAME]
        -
        -  Destroy: 2 deleted
        -
        -DMS API called: yes
        +Exit code: 1
        
    acceptance_test.go:1119: 
        LOG.commands: 
        LOG.commands: commands: ('bundle', 'deploy', '--auto-approve')
        LOG.commands: exit=0
        LOG.commands: Uploading bundle files to /Workspace/Users/tester@databricks.com/.bundle/dms-fault-delete_deployment-commands-thsrjetlvzdbpfxxq654ni5kru/files...
        LOG.commands: Current Deployment Version: http://127.0.0.1:39097/deployments/1397?version=1
        LOG.commands: Created jobs.foo
        LOG.commands: Created pipelines.bar
        LOG.commands: Files: 4 uploaded, 0 deleted
        LOG.commands: Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        LOG.commands: 
        LOG.commands: commands: ('workspace', 'get-status', '/Workspace/Users/tester@databricks.com/.bundle/dms-fault-delete_deployment-commands-thsrjetlvzdbpfxxq654ni5kru/state/resources.deployment.json', '-o', 'json')
        LOG.commands: exit=0
        LOG.commands: {
        LOG.commands:   "object_id": 1397,
        LOG.commands:   "object_type": "BUNDLE_DEPLOYMENT",
        LOG.commands:   "path": "/Workspace/Users/tester@databricks.com/.bundle/dms-fault-delete_deployment-commands-thsrjetlvzdbpfxxq654ni5kru/state/resources.deployment.json"
        LOG.commands: }
        LOG.commands: 
        LOG.commands: commands: ('bundle', 'deploy', '--auto-approve')
        LOG.commands: exit=0
        LOG.commands: Uploading bundle files to /Workspace/Users/tester@databricks.com/.bundle/dms-fault-delete_deployment-commands-thsrjetlvzdbpfxxq654ni5kru/files...
        LOG.commands: Current Deployment Version: http://127.0.0.1:39097/deployments/1397?version=2
        LOG.commands: Updated pipelines.bar
        LOG.commands: Updated jobs.foo
        LOG.commands: Files: 3 uploaded, 0 deleted
        LOG.commands: Resources: 0 created, 2 changed, 0 deleted, 0 unchanged

TestAccept/bundle/dms_fault/get_deployment/DMS=true: branch ✅ | main (e9687c1) ✅ | latest (v1.22.1) ❌

latest (v1.22.1):

acceptance_test.go:1287: Suppressed from output.txt:
        Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
    acceptance_test.go:1341: Diff:
        --- bundle/dms_fault/get_deployment/output.txt
        +++ /tmp/TestAcceptbundledms_faultget_deploymentDMS=true3693353463/002/output.txt
        @@ -14,146 +14,31 @@
         >>> bundle deploy --auto-approve
           Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
           Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
        -  Warn: Could not read deployment history: Fault injected by test.
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]/files...
        -  Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=2
        -  Updated jobs.foo
        -  Updated pipelines.bar
        -  Files: 3 uploaded, 0 deleted
        -  Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
        +  Error: Fault injected by test. (500 INJECTED)
         
        ->>> bundle summary
        -  Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
        -  Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
        -  Warn: Could not read deployment history: Fault injected by test.
        -  Name: updated-dms-fault-get_deployment-commands-[UNIQUE_NAME]
        -  Target: default
        -  Workspace:
        -    User: [USERNAME]
        -    Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]
        -  Resources:
        -    Jobs:
        -      foo:
        -        Name: updated-dms-fault-get_deployment-commands-[UNIQUE_NAME]
        -        URL:  [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID]
        -    Pipelines:
        -      bar:
        -        Name: updated-dms-fault-get_deployment-commands-[UNIQUE_NAME]
        -        URL:  [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID]
        +  Endpoint: GET [DATABRICKS_URL]/api/2.0/bundle/deployments/[DEPLOYMENT_ID]?
        +  HTTP Status: 500 Internal Server Error
        +  API error_code: INJECTED
        +  API message: Fault injected by test.
         
        ->>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force
        -  Job configuration successfully saved to exported/foo.job.yml
        -  Warning: Generated configuration is not included in the bundle
        +Traceback (most recent call last):
        +  File "[TESTROOT]/bundle/dms_fault/get_deployment/../run.py", line 248, in <module>
        +    main()
        +    ~~~~^^
        +  File "[TESTROOT]/bundle/dms_fault/get_deployment/../run.py", line 194, in main
        +    run(cwd, "bundle", "deploy", "--auto-approve")
        +    ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        +  File "[TESTROOT]/bundle/dms_fault/get_deployment/../run.py", line 64, in run
        +    assert result.returncode == expected, output
        +           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        +AssertionError: Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
        +Error: Fault injected by test. (500 INJECTED)
         
        -  The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml,
        -  so it will not be deployed. Add a matching entry to the 'include' section, for example:
        +Endpoint: GET [DATABRICKS_URL]/api/2.0/bundle/deployments/[DEPLOYMENT_ID]?
        +HTTP Status: 500 Internal Server Error
        +API error_code: INJECTED
        +API message: Fault injected by test.
         
        -  include:
        -    - exported/*.yml
         
         
        ->>> bundle deployment unbind foo
        -
        ->>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind
        -  Job configuration successfully saved to resources/foo.job.yml
        -  Successfully bound job with an id '[FOO_ID]'
        -
        ->>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force
        -  File successfully saved to exported-src/pipeline.sql
        -  Pipeline configuration successfully saved to exported/bar.pipeline.yml
        -  Warning: Generated configuration is not included in the bundle
        -
        -  The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml,
        -  so it will not be deployed. Add a matching entry to the 'include' section, for example:
        -
        -  include:
        -    - exported/*.yml
        -
        -
        ->>> bundle deployment unbind bar
        -
        ->>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind
        -  File successfully saved to src/pipeline.sql
        -  Pipeline configuration successfully saved to resources/bar.pipeline.yml
        -  Successfully bound pipeline with an id '[BAR_ID]'
        -
        ->>> bundle validate
        -  Name: updated-dms-fault-get_deployment-commands-[UNIQUE_NAME]
        -  Target: default
        -  Workspace:
        -    User: [USERNAME]
        -    Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]
        -
        -  Validation OK!
        -
        ->>> bundle deploy --auto-approve
        -  Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
        -  Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
        -  Warn: Could not read deployment history: Fault injected by test.
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]/files...
        -  Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=9
        -  Updated jobs.foo
        -  Updated pipelines.bar
        -  Files: 6 uploaded, 0 deleted
        -  Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
        -
        -=== deployment_history: false
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]/files...
        -  Updated jobs.foo
        -  Updated pipelines.bar
        -  Files: 2 uploaded, 0 deleted
        -  Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
        -
        -=== deployment_history: true
        -
        ->>> bundle deploy --auto-approve
        -  Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
        -  Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
        -  Warn: Could not read deployment history: Fault injected by test.
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]/files...
        -  Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=11
        -  Updated jobs.foo
        -  Files: 2 uploaded, 0 deleted
        -  Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
        -
        ->>> bundle destroy --auto-approve
        -  Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
        -  Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]
        -  Warn: Could not read deployment history: Fault injected by test.
        -  The following resources will be deleted:
        -    delete resources.jobs.foo
        -    delete resources.pipelines.bar
        -
        -  This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
        -  Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
        -    delete resources.pipelines.bar
        -
        -  All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]
        -
        -  Destroy: 2 deleted
        -
        -=== deployment_history: false
        -
        ->>> bundle deploy --auto-approve
        -  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]/files...
        -  Created jobs.foo
        -  Created pipelines.bar
        -  Files: 8 uploaded, 0 deleted
        -  Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> bundle destroy --auto-approve
        -  The following resources will be deleted:
        -    delete resources.jobs.foo
        -    delete resources.pipelines.bar
        -
        -  This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
        -  Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
        -    delete resources.pipelines.bar
        -
        -  All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-get_deployment-commands-[UNIQUE_NAME]
        -
        -  Destroy: 2 deleted
        -
        -DMS API called: yes
        +Exit code: 1
        
    acceptance_test.go:1119: 
        LOG.commands: 
        LOG.commands: commands: ('bundle', 'deploy', '--auto-approve')
        LOG.commands: exit=0
        LOG.commands: Uploading bundle files to /Workspace/Users/tester@databricks.com/.bundle/dms-fault-get_deployment-commands-gr2zfn2lincmppemyng7u6slj4/files...
        LOG.commands: Current Deployment Version: http://127.0.0.1:37913/deployments/1343?version=1
        LOG.commands: Created jobs.foo
        LOG.commands: Created pipelines.bar
        LOG.commands: Files: 4 uploaded, 0 deleted
        LOG.commands: Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        LOG.commands: 
        LOG.commands: commands: ('workspace', 'get-status', '/Workspace/Users/tester@databricks.com/.bundle/dms-fault-get_deployment-commands-gr2zfn2lincmppemyng7u6slj4/state/resources.deployment.json', '-o', 'json')
        LOG.commands: exit=0
        LOG.commands: {
        LOG.commands:   "object_id": 1343,
        LOG.commands:   "object_type": "BUNDLE_DEPLOYMENT",
        LOG.commands:   "path": "/Workspace/Users/tester@databricks.com/.bundle/dms-fault-get_deployment-commands-gr2zfn2lincmppemyng7u6slj4/state/resources.deployment.json"
        LOG.commands: }
        LOG.commands: 
        LOG.commands: commands: ('bundle', 'deploy', '--auto-approve')
        LOG.commands: exit=1
        LOG.commands: Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/1343
        LOG.commands: Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/1343
        LOG.commands: Error: Fault injected by test. (500 INJECTED)
        LOG.commands: 
... (4 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)

10 more failure details omitted, see the full report in the run.

eng-dev-ecosystem-bot · 2026-10-28T15:03:37Z

Integration test report

Commit: 260fb4e

Run: 40782891785

	Env	❌FAIL	✅pass	🙈skip	Time
❌	aws linux-2core-8gb	1	280	30	5:10
❌	aws-windows-latest-4core-16gb	1	282	28	3:34
❌	azure linux-2core-8gb	1	279	30	5:04
❌	azure-windows-latest-4core-16gb	1	281	28	3:40
❌	gcp linux-2core-8gb	1	280	30	5:06
❌	gcp-windows-latest-4core-16gb	1	282	28	3:39

	Test Name	aws linux-2core-8gb	aws-windows-latest-4core-16gb	azure linux-2core-8gb	azure-windows-latest-4core-16gb	gcp linux-2core-8gb	gcp-windows-latest-4core-16gb
❌	TestSetupLocalServerlessProvision	❌F	❌F	❌F	❌F	❌F	❌F

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
3:42	aws linux-2core-8gb	TestAccept
3:42	gcp linux-2core-8gb	TestAccept
3:39	azure linux-2core-8gb	TestAccept
3:17	gcp-windows-latest-4core-16gb	TestAccept
3:16	azure-windows-latest-4core-16gb	TestAccept
3:11	aws-windows-latest-4core-16gb	TestAccept

## Changes Support `*` within a path segment and `**` across path segments in `fault.py` patterns. Both match zero or more characters; all other characters are literal. Compile each pattern once when its rule is registered.. ## Why useful for tests being added in #7231.

shreyas-goenka · 2026-10-28T22:38:34Z

+=== commands: get_deployment fault enabled
+
+>>> bundle deploy --auto-approve
+  Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]


these repeated warnings can be made prettier, we can iterate on them separately. It is good enough for now tbh.

shreyas-goenka · 2026-10-28T22:39:48Z

+
+  Destroy: 2 deleted
+
+DMS API called: no


this is an important signal, shows whether the API is called or not (should we thus expect a warning in the output or not).

ilyakuz-db

This approach is fine, but maybe it will be simpler instead of making a suite for each API, do suite for each CLI command, and force all DMS APIs to return 500 there (and maybe also add some other Go error to ensure that any bugs won't break the direct engine flow)

@denik WDYT?

ilyakuz-db · 2026-10-29T10:31:14Z

@@ -0,0 +1,148 @@
+


Should we add init / config-remote-sync? I assume they don't interact directly in DMS, but is it possible through some shared code?

shreyas-goenka · 2026-10-29T10:41:23Z

@ilyakuz-db if we make all APIs return 500, that'll not exercise all code paths since later flows can depend upon earlier ones succeeding. This is the most robust way to ensure nothing can break for customers.

shreyas-goenka changed the title ~~Test bundle commands with individual DMS API failures~~ Record bundle behavior with faults for every DMS API Oct 28, 2026

shreyas-goenka mentioned this pull request Oct 28, 2026

Support segment and recursive wildcards in fault patterns #7238

Merged

shreyas-goenka force-pushed the shreyas-goenka_data/shreyas/test-bundle-dms-fail-open branch from 5dbdc5d to 9b95f43 Compare October 28, 2026 16:06

shreyas-goenka changed the base branch from shreyas-goenka_data/shreyas/generate-omit-deployment-metadata to shreyas-goenka_data/shreyas/test-fault-glob-segments October 28, 2026 16:06

shreyas-goenka added this pull request to stack #7239 October 28, 2026 16:07

shreyas-goenka force-pushed the shreyas-goenka_data/shreyas/test-bundle-dms-fail-open branch from 9b95f43 to c769fef Compare October 28, 2026 16:15

Base automatically changed from shreyas-goenka_data/shreyas/test-fault-glob-segments to shreyas-goenka_data/shreyas/generate-omit-deployment-metadata October 28, 2026 16:50

shreyas-goenka force-pushed the shreyas-goenka_data/shreyas/test-bundle-dms-fail-open branch from c769fef to 15d2a7e Compare October 28, 2026 16:50

shreyas-goenka mentioned this pull request Oct 28, 2026

Support segment and recursive wildcards in fault patterns #7241

Merged

shreyas-goenka removed this pull request from stack #7239 October 28, 2026 16:56

shreyas-goenka force-pushed the shreyas-goenka_data/shreyas/test-bundle-dms-fail-open branch from 15d2a7e to e1c4557 Compare October 28, 2026 16:56

shreyas-goenka changed the base branch from shreyas-goenka_data/shreyas/generate-omit-deployment-metadata to shreyas-goenka_data/shreyas/test-fault-glob-main October 28, 2026 16:56

shreyas-goenka added this pull request to stack #7242 October 28, 2026 16:56

github-actions Bot added the DABs DABs related issues label Oct 28, 2026

Base automatically changed from shreyas-goenka_data/shreyas/test-fault-glob-main to main October 28, 2026 17:34

shreyas-goenka added 6 commits October 28, 2026 18:34

Test bundle commands with individual DMS API failures

076e085

Record bundle output for faults across all DMS APIs

3798de3

Simplify and document the DMS fault test workflow

0285da7

Support middle wildcards in fault injection patterns

532c5e3

Document DMS fault test helpers with short comments

8aeceed

Test deployment history toggles under DMS API faults

1e09ad4

shreyas-goenka force-pushed the shreyas-goenka_data/shreyas/test-bundle-dms-fail-open branch from 9e2109d to 1e09ad4 Compare October 28, 2026 17:34

shreyas-goenka added 2 commits October 28, 2026 22:30

Keep GetDeployment faults active across deployment IDs

d8da8e1

Clarify the DMS fault bundle setup comment

ef5b0ec

shreyas-goenka commented Oct 28, 2026

View reviewed changes

shreyas-goenka requested a review from denik October 28, 2026 22:41

Make DMS fault runner safe to import for doctests

260fb4e

shreyas-goenka requested a review from ilyakuz-db October 28, 2026 22:53

shreyas-goenka marked this pull request as ready for review October 28, 2026 22:53

shreyas-goenka requested review from a team as code owners October 28, 2026 22:53

ilyakuz-db approved these changes Oct 29, 2026

View reviewed changes

Navigation Menu

Repository navigation

Record bundle behavior with faults for every DMS API - #7231

Record bundle behavior with faults for every DMS API#7231
shreyas-goenka wants to merge 9 commits into
mainfrom
shreyas-goenka_data/shreyas/test-bundle-dms-fail-open

shreyas-goenka commented Oct 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 28, 2026 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Oct 28, 2026 •

edited

Loading

Uh oh!

shreyas-goenka Oct 28, 2026

Uh oh!

shreyas-goenka Oct 28, 2026

Uh oh!

ilyakuz-db left a comment

Uh oh!

ilyakuz-db Oct 29, 2026

Uh oh!

shreyas-goenka commented Oct 29, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

shreyas-goenka commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

github-actions Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression test report

Uh oh!

eng-dev-ecosystem-bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

shreyas-goenka Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

ilyakuz-db left a comment

Choose a reason for hiding this comment

Uh oh!

ilyakuz-db Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka commented Oct 29, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shreyas-goenka commented Oct 28, 2026 •

edited

Loading

github-actions Bot commented Oct 28, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented Oct 28, 2026 •

edited

Loading