Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
e91d35f
Rename schemas in place in the direct engine
denik Nov 4, 2026
6503db4
Link PR in changelog fragment
denik Nov 4, 2026
f1bea37
testserver: keep volume create lenient
denik Nov 4, 2026
69b7824
Drop engine mention from changelog entry
denik Nov 4, 2026
44b1ce6
acc: drop text plan where the JSON plan already shows it
denik Nov 4, 2026
278754b
acc: show only the plan field each rename test is about
denik Nov 4, 2026
0161163
acc: point migrate recreate comments at schema-catalog-change
denik Nov 5, 2026
a81a415
acc: show the whole plan in the rename tests
denik Nov 5, 2026
436f3ff
direct: don't send an empty update to a child moved by its parent's r…
denik Nov 5, 2026
a4955f9
acc: update, rename and recreate a volume after its schema is renamed
denik Nov 5, 2026
26d60e8
acc: sort requests of the parallel schema creates
denik Nov 5, 2026
889bb81
acc: a job reading a volume's full_name and id follows its schema's r…
denik Nov 5, 2026
7ee5862
acc: drop cloud-specific fields from the rename plans
denik Nov 5, 2026
574d766
acc: show preserved schema and volume ids through replacements
denik Nov 5, 2026
bc0dff1
direct: build the re-keyed child's ID from config instead of reading it
denik Nov 5, 2026
d80bb6b
direct: follow a parent rename that an interrupted deploy already app…
denik Nov 5, 2026
193749d
direct: find a moved schema or volume by its stable ID instead of ren…
denik Nov 6, 2026
a603fd0
acc: out-of-band renames and recreates of volumes, schemas and catalogs
denik Nov 6, 2026
f33ab00
acc: record stable_id in the user_agent DMS request golden
denik Nov 6, 2026
63444e7
acc: normalize a killed CLI on Windows in the interrupted rename tests
denik Nov 6, 2026
fcf5088
direct: record volume_id and schema_id in the resource state instead …
denik Nov 6, 2026
5ef55bd
Regenerate apitypes and volume spec rules for the new state types
denik Nov 9, 2026
7bab424
direct: ignore volume_id and schema_id changes instead of dropping them
denik Nov 9, 2026
0f6b721
acc: show the skipped volume_id and schema_id changes in plans
denik Nov 9, 2026
4a2dcc1
acc: a volume follows its catalog and schema renamed in one deploy
denik Nov 9, 2026
f3217a9
acc: keep the captured migrate fixtures; let the replay create the ca…
denik Nov 9, 2026
7e8b575
direct: classify a parent-renamed ID field as update_id directly
denik Nov 9, 2026
6379094
Point comments at followsParentRename
denik Nov 9, 2026
b1b6b03
direct: apply re-key-only update_id in the framework and record inter…
denik Nov 9, 2026
19973af
acc: renames with another field changed, and case-only renames
denik Nov 9, 2026
3389391
direct: match the internal ID field structurally, ahead of the skip c…
denik Nov 9, 2026
c23c64f
acc: regenerate for state-only re-keys and recorded schema_id
denik Nov 9, 2026
55e08dd
deployplan: name the reason at the call site with Changes.HasOnlySing…
denik Nov 9, 2026
37e8eb3
acc: mark the catalog out-of-band rename tests with Badness (pre-exis…
denik Nov 9, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .nextchanges/bundles/schema-rename.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* Rename a schema in place when its `name` changes, instead of deleting and recreating it. Schemas and volumes inside a renamed catalog or schema now follow it to the new name; recreating them failed because they already exist there. ([#7343](https://github.com/databricks/cli/pull/7343))
5 changes: 3 additions & 2 deletions acceptance/bundle/dms/declined-deploy/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
"error_message": "",
"resource_id": "main.dms_declined_deploy_schema_[UNIQUE_NAME]",
"sequence_id": 0,
"state": "{\"state\":{\"catalog_name\":\"main\",\"name\":\"dms_declined_deploy_schema_[UNIQUE_NAME]\"}}",
"state": "{\"state\":{\"catalog_name\":\"main\",\"name\":\"dms_declined_deploy_schema_[UNIQUE_NAME]\",\"schema_id\":\"[UUID]\"}}",
"status": "OPERATION_STATUS_SUCCEEDED"
}
}
Expand Down Expand Up @@ -98,7 +98,8 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"__id__": "main.dms_declined_deploy_schema_[UNIQUE_NAME]",
"state": {
"catalog_name": "main",
"name": "dms_declined_deploy_schema_[UNIQUE_NAME]"
"name": "dms_declined_deploy_schema_[UNIQUE_NAME]",
"schema_id": "[UUID]"
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
"error_message": "",
"resource_id": "main.dms_metadata_schema_[UNIQUE_NAME]",
"sequence_id": 0,
"state": "{\"state\":{\"catalog_name\":\"main\",\"name\":\"dms_metadata_schema_[UNIQUE_NAME]\"}}",
"state": "{\"state\":{\"catalog_name\":\"main\",\"name\":\"dms_metadata_schema_[UNIQUE_NAME]\",\"schema_id\":\"[UUID]\"}}",
"status": "OPERATION_STATUS_SUCCEEDED"
}
}
Expand Down
5 changes: 3 additions & 2 deletions acceptance/bundle/dms/empty-grants/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
"error_message": "",
"resource_id": "main.dms_empty_grants_[UNIQUE_NAME]",
"sequence_id": 0,
"state": "{\"state\":{\"catalog_name\":\"main\",\"name\":\"dms_empty_grants_[UNIQUE_NAME]\"}}",
"state": "{\"state\":{\"catalog_name\":\"main\",\"name\":\"dms_empty_grants_[UNIQUE_NAME]\",\"schema_id\":\"[UUID]\"}}",
"status": "OPERATION_STATUS_SUCCEEDED"
}
}
Expand Down Expand Up @@ -170,7 +170,8 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"__id__": "main.dms_empty_grants_[UNIQUE_NAME]",
"state": {
"catalog_name": "main",
"name": "dms_empty_grants_[UNIQUE_NAME]"
"name": "dms_empty_grants_[UNIQUE_NAME]",
"schema_id": "[UUID]"
}
}
}
Expand Down
5 changes: 3 additions & 2 deletions acceptance/bundle/dms/failed-recreate/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
"error_message": "",
"resource_id": "main.dms_failed_recreate_schema_[UNIQUE_NAME]",
"sequence_id": 0,
"state": "{\"state\":{\"catalog_name\":\"main\",\"comment\":\"v1\",\"name\":\"dms_failed_recreate_schema_[UNIQUE_NAME]\"}}",
"state": "{\"state\":{\"catalog_name\":\"main\",\"comment\":\"v1\",\"name\":\"dms_failed_recreate_schema_[UNIQUE_NAME]\",\"schema_id\":\"[UUID]\"}}",
"status": "OPERATION_STATUS_SUCCEEDED"
}
}
Expand Down Expand Up @@ -100,7 +100,8 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"state": {
"catalog_name": "main",
"comment": "v1",
"name": "dms_failed_recreate_schema_[UNIQUE_NAME]"
"name": "dms_failed_recreate_schema_[UNIQUE_NAME]",
"schema_id": "[UUID]"
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,8 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"state": {
"catalog_name": "main",
"comment": "v1",
"name": "dms_upload_fails_once_schema_[UNIQUE_NAME]"
"name": "dms_upload_fails_once_schema_[UNIQUE_NAME]",
"schema_id": "[UUID]"
}
}
}
Expand Down Expand Up @@ -106,7 +107,8 @@ Resources: 1 created, 0 changed, 1 deleted, 0 unchanged
"state": {
"catalog_name": "dms_other_[UNIQUE_NAME]",
"comment": "v1",
"name": "dms_upload_fails_once_schema_[UNIQUE_NAME]"
"name": "dms_upload_fails_once_schema_[UNIQUE_NAME]",
"schema_id": "[UUID]"
}
}
}
Expand Down Expand Up @@ -153,7 +155,7 @@ Resources: 1 created, 0 changed, 1 deleted, 0 unchanged
"error_message": "",
"resource_id": "dms_other_[UNIQUE_NAME].dms_upload_fails_once_schema_[UNIQUE_NAME]",
"sequence_id": 0,
"state": "{\"state\":{\"catalog_name\":\"dms_other_[UNIQUE_NAME]\",\"comment\":\"v1\",\"name\":\"dms_upload_fails_once_schema_[UNIQUE_NAME]\"}}",
"state": "{\"state\":{\"catalog_name\":\"dms_other_[UNIQUE_NAME]\",\"comment\":\"v1\",\"name\":\"dms_upload_fails_once_schema_[UNIQUE_NAME]\",\"schema_id\":\"[UUID]\"}}",
"status": "OPERATION_STATUS_SUCCEEDED"
}
}
Expand Down
10 changes: 6 additions & 4 deletions acceptance/bundle/dms/successful-recreate/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
"error_message": "",
"resource_id": "main.dms_successful_recreate_schema_[UNIQUE_NAME]",
"sequence_id": 0,
"state": "{\"state\":{\"catalog_name\":\"main\",\"comment\":\"v1\",\"name\":\"dms_successful_recreate_schema_[UNIQUE_NAME]\"}}",
"state": "{\"state\":{\"catalog_name\":\"main\",\"comment\":\"v1\",\"name\":\"dms_successful_recreate_schema_[UNIQUE_NAME]\",\"schema_id\":\"[UUID]\"}}",
"status": "OPERATION_STATUS_SUCCEEDED"
}
}
Expand Down Expand Up @@ -100,7 +100,8 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"state": {
"catalog_name": "main",
"comment": "v1",
"name": "dms_successful_recreate_schema_[UNIQUE_NAME]"
"name": "dms_successful_recreate_schema_[UNIQUE_NAME]",
"schema_id": "[UUID]"
}
}
}
Expand Down Expand Up @@ -159,7 +160,7 @@ Resources: 1 created, 0 changed, 1 deleted, 0 unchanged
"error_message": "",
"resource_id": "dms_other_[UNIQUE_NAME].dms_successful_recreate_schema_[UNIQUE_NAME]",
"sequence_id": 1,
"state": "{\"state\":{\"catalog_name\":\"dms_other_[UNIQUE_NAME]\",\"comment\":\"v1\",\"name\":\"dms_successful_recreate_schema_[UNIQUE_NAME]\"}}",
"state": "{\"state\":{\"catalog_name\":\"dms_other_[UNIQUE_NAME]\",\"comment\":\"v1\",\"name\":\"dms_successful_recreate_schema_[UNIQUE_NAME]\",\"schema_id\":\"[UUID]\"}}",
"status": "OPERATION_STATUS_SUCCEEDED"
}
}
Expand Down Expand Up @@ -212,7 +213,8 @@ Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
"state": {
"catalog_name": "dms_other_[UNIQUE_NAME]",
"comment": "v1",
"name": "dms_successful_recreate_schema_[UNIQUE_NAME]"
"name": "dms_successful_recreate_schema_[UNIQUE_NAME]",
"schema_id": "[UUID]"
}
}
}
Expand Down
4 changes: 2 additions & 2 deletions acceptance/bundle/lifecycle/prevent-destroy/script
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ update_file.py resources/pipeline.yml ' catalog: main' ' storage: "dbf
trace errcode $CLI bundle plan >>out.txt 2>&1
trace musterr $CLI bundle deploy -qq >>out.txt 2>&1

# Changing the schema name, deploy must fail because schema will be recreated
update_file.py resources/schema.yml 'name: test-schema' 'name: test-schema-new'
# Changing the schema catalog, deploy must fail because schema will be recreated
update_file.py resources/schema.yml 'catalog_name: "test-catalog"' 'catalog_name: "test-catalog-new"'
trace errcode $CLI bundle plan >>out.txt 2>&1
trace musterr $CLI bundle deploy -qq >>out.txt 2>&1

Expand Down
35 changes: 6 additions & 29 deletions acceptance/bundle/migrate/auto/schema-rename/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,38 +2,15 @@
=== Replay the prior terraform deployment (captured from a real terraform run)
>>> replay_tfstate.py terraform-requests.json terraform.tfstate

=== Rename the schema (an immutable id field). Without --auto-approve the recreate is refused
>>> DATABRICKS_BUNDLE_ENGINE=direct errcode [CLI] bundle deploy
Warn: migration to direct: schemas.my_schema: name differs between config ("myschema-[UNIQUE_NAME]-renamed") and terraform state ("myschema-[UNIQUE_NAME]"); this resource will be recreated.
Notice: migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/files...

This action will result in the deletion or recreation of the following UC schemas. Any underlying data may be lost:
recreate resources.schemas.my_schema
Warn: Migration not committed, keeping Terraform state
Error: the deployment requires destructive actions, but the current console does not support prompting.
Deleting data assets such as schemas, pipelines, or volumes may cause permanent data loss and should be carefully reviewed.
To proceed, use --auto-approve after reviewing the plan above.

Files: 2 uploaded, 0 deleted

Exit code: 1

>>> print_migration_telemetry
direct_migrate_recreate_planned true

=== With --auto-approve the recreate is applied on direct under the new name
>>> DATABRICKS_BUNDLE_ENGINE=direct [CLI] bundle deploy --auto-approve
Warn: migration to direct: schemas.my_schema: name differs between config ("myschema-[UNIQUE_NAME]-renamed") and terraform state ("myschema-[UNIQUE_NAME]"); this resource will be recreated.
=== Rename the schema (an updatable id field) and deploy on direct
>>> DATABRICKS_BUNDLE_ENGINE=direct [CLI] bundle deploy
Warn: migration to direct: schemas.my_schema: name differs between config ("myschema-[UNIQUE_NAME]-renamed") and terraform state ("myschema-[UNIQUE_NAME]"); this resource will be renamed.
Notice: migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/files...

This action will result in the deletion or recreation of the following UC schemas. Any underlying data may be lost:
recreate resources.schemas.my_schema
Migrated 1 resource to direct deployment engine.
Recreated schemas.my_schema
Files: 0 uploaded, 0 deleted
Resources: 1 created, 0 changed, 1 deleted, 0 unchanged
Updated schemas.my_schema
Files: 2 uploaded, 0 deleted
Resources: 0 created, 1 changed, 0 deleted, 0 unchanged

=== Redeploy on direct: the schema now matches config, so the plan is a clean no-op
>>> DATABRICKS_BUNDLE_ENGINE=direct [CLI] bundle deploy
Expand Down
15 changes: 5 additions & 10 deletions acceptance/bundle/migrate/auto/schema-rename/script
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,12 @@ title "Replay the prior terraform deployment (captured from a real terraform run
trace replay_tfstate.py terraform-requests.json terraform.tfstate
rm -f "$OUT_REQUESTS"

title "Rename the schema (an immutable id field). Without --auto-approve the recreate is refused"
# name is a provided id field, so a rename recreates. reconcileIDFields seeds the deployed
# name into the converted state, so the migration's plan sees a real recreate. Schemas are in
# the destructive-approval group, so without --auto-approve the deploy refuses the recreate
# (exit 1) and the migration is not committed - the run stays on terraform.
title "Rename the schema (an updatable id field) and deploy on direct"
# name is an updatable id field: the schema is renamed in place (UpdateWithID), so the deploy
# needs no --auto-approve. reconcileIDFields seeds the deployed name into the converted state,
# so the migration's plan sees a rename rather than snapshotting the new name as already applied.
update_file.py databricks.yml "myschema-$UNIQUE_NAME" "myschema-$UNIQUE_NAME-renamed"
trace DATABRICKS_BUNDLE_ENGINE=direct errcode $CLI bundle deploy
trace print_migration_telemetry

title "With --auto-approve the recreate is applied on direct under the new name"
trace DATABRICKS_BUNDLE_ENGINE=direct $CLI bundle deploy --auto-approve
trace DATABRICKS_BUNDLE_ENGINE=direct $CLI bundle deploy

title "Redeploy on direct: the schema now matches config, so the plan is a clean no-op"
trace DATABRICKS_BUNDLE_ENGINE=direct $CLI bundle deploy
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ export DATABRICKS_BUNDLE_ENGINE=

# Testserver-only: an explicit storage_root needs a registered external location on a real
# workspace, so this is not Cloud-enabled. The recreate/refusal class it covers is verified
# on cloud by schema-rename and pipeline-storage-change.
# on cloud by schema-catalog-change and pipeline-storage-change.

title "Replay the prior terraform deployment (captured from a real terraform run)"
trace replay_tfstate.py terraform-requests.json terraform.tfstate
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/migrate/auto/volume-rename/script
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ title "Replay the prior terraform deployment (captured from a real terraform run
trace replay_tfstate.py terraform-requests.json terraform.tfstate

title "Rename the volume (an updatable id field) and deploy on direct"
# Unlike a schema (whose name recreates), a volume name is an updatable id field: it is
# A volume name is an updatable id field: it is
# renamed in place (UpdateWithID). The migration seeds the deployed name into the converted
# state, so its plan sees a rename and the migration succeeds; the deploy then renames the
# volume rather than the migration snapshotting the new name as already applied. Verifies the
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/migrate/auto/volume-type-change/script
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ export DATABRICKS_BUNDLE_ENGINE=

# Testserver-only: changing volume_type to EXTERNAL needs a storage_location backed by a
# registered external location on a real workspace, so this is not Cloud-enabled. The
# recreate class it covers is verified on cloud by schema-rename / registered-model-rename.
# recreate class it covers is verified on cloud by schema-catalog-change / registered-model-rename.

# volume_type is both a provided id field and a recreate_on_changes field. reconcileIDFields
# records the deployed value, so the migration's plan sees a real change and recreates. Volumes
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,11 @@
"action": "skip",
"reason": "immutable_backend_value",
"remote": "s3://[METASTORE_NAME]/metastore/[UUID]/volumes/[UUID]"
},
"volume_id": {
"action": "skip",
"reason": "internal_id",
"remote": "[UUID]"
}
}
}
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/refschema/output/schemas.txt
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ resources.schemas.*.name string ALL
resources.schemas.*.owner string REMOTE
resources.schemas.*.properties map[string]string ALL
resources.schemas.*.properties.* string ALL
resources.schemas.*.schema_id string REMOTE
resources.schemas.*.schema_id string REMOTE STATE
resources.schemas.*.storage_location string REMOTE
resources.schemas.*.storage_root string ALL
resources.schemas.*.updated_at int64 REMOTE
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/refschema/output/volumes.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ resources.volumes.*.storage_location string ALL
resources.volumes.*.updated_at int64 REMOTE
resources.volumes.*.updated_by string REMOTE
resources.volumes.*.url string INPUT
resources.volumes.*.volume_id string REMOTE
resources.volumes.*.volume_id string REMOTE STATE
resources.volumes.*.volume_path string INPUT
resources.volumes.*.volume_type catalog.VolumeType ALL
resources.volumes.*.grants.full_name string ALL
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
Error: cannot plan resources.volumes.foo: cannot resolve "${resources.volumes.bar.non_existent}": schema mismatch: non_existent: field "non_existent" not found in catalog.CreateVolumeRequestContent; non_existent: field "non_existent" not found in catalog.VolumeInfo
Error: cannot plan resources.volumes.foo: cannot resolve "${resources.volumes.bar.non_existent}": schema mismatch: non_existent: field "non_existent" not found in dresources.VolumeState; non_existent: field "non_existent" not found in catalog.VolumeInfo

Error: planning failed

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Error: cannot plan resources.volumes.foo: cannot resolve "${resources.volumes.bar.non_existent}": schema mismatch: non_existent: field "non_existent" not found in catalog.CreateVolumeRequestContent; non_existent: field "non_existent" not found in catalog.VolumeInfo
Error: cannot plan resources.volumes.foo: cannot resolve "${resources.volumes.bar.non_existent}": schema mismatch: non_existent: field "non_existent" not found in dresources.VolumeState; non_existent: field "non_existent" not found in catalog.VolumeInfo

Error: planning failed

16 changes: 16 additions & 0 deletions acceptance/bundle/resources/catalogs/rename/databricks.yml.tmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
bundle:
name: test-bundle-$UNIQUE_NAME

resources:
catalogs:
catalog1:
name: test_catalog_$UNIQUE_NAME
schemas:
schema1:
catalog_name: ${resources.catalogs.catalog1.name}
name: myschema
volumes:
volume1:
catalog_name: ${resources.catalogs.catalog1.name}
schema_name: ${resources.schemas.schema1.name}
name: myvolume
2 changes: 2 additions & 0 deletions acceptance/bundle/resources/catalogs/rename/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Cloud = true
EnvMatrix.DMS = ["", "true"]
Loading
Loading