Repository navigation
Support DMS migration during bundle deploy #6989
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,38 +1,39 @@ | ||
| envsubst < databricks.yml.tmpl > databricks.yml | ||
|
|
||
| title "1. Deploy without history: existing state tracks one job" | ||
| trace $CLI bundle deploy | ||
| dms_plan | ||
| trace print_requests.py --dms //api/2.0/bundle --oneline | ||
| trace jq '{state_version, features}' .databricks/bundle/default/resources.json | ||
|
|
||
| title "2. Enable history on the existing deployment\nExpect: migration succeeds without recreating the job." | ||
| trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true $CLI bundle deploy | ||
| deployment_id=$(MSYS_NO_PATHCONV=1 $CLI workspace get-status "/Workspace/Users/${CURRENT_USER_NAME}/.bundle/dms-existing-state-${UNIQUE_NAME}/default/state/resources.deployment.json" -o json | python3 -c 'import sys, json; print(json.load(sys.stdin)["object_id"])') | ||
| add_repl "$deployment_id" DEPLOYMENT_ID | ||
|
|
||
| title "3. Remove local state and deploy again\nExpect: remote DMS state is loaded; no migration or resource creation." | ||
| rm -rf .databricks | ||
| trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true $CLI bundle deploy | ||
|
|
||
| title "4. A leftover local WAL is refused: DMS deployments do not use WALs" | ||
| mkdir -p .databricks/bundle/default | ||
| printf '{"lineage":"stale","serial":99,"state_version":2,"features":{"deployment_history":{}}}\n' > .databricks/bundle/default/resources.json.wal | ||
| musterr trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true $CLI bundle deploy | ||
|
|
||
| title "5. Destroy follows recorded state; redeploy starts a new history" | ||
| # The rejected WAL remains on disk. Remove it before destroying the recorded deployment. | ||
| rm -f .databricks/bundle/default/resources.json.wal | ||
| # Turn history off to check that destroy still follows the recorded backend. | ||
| trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=false $CLI bundle destroy --auto-approve | ||
| title "Fresh history: version 1 creates the job without migration" | ||
| rm -f "$OUT_REQUESTS" | ||
| trace DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true $CLI bundle deploy | ||
| trace print_requests.py --dms //api/2.0/bundle --oneline | ||
| trace find.py 'resources[.]json[.]wal' --expect 0 | ||
| trace jq '{state_version, features}' .databricks/bundle/default/resources.json | ||
| DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true replace_ids.py | ||
| deployment_id=$(MSYS_NO_PATHCONV=1 $CLI workspace get-status "/Workspace/Users/${CURRENT_USER_NAME}/.bundle/dms-existing-state-${UNIQUE_NAME}/default/state/resources.deployment.json" -o json | python3 -c 'import sys, json; print(json.load(sys.stdin)["object_id"])') | ||
| add_repl "$deployment_id" DEPLOYMENT_ID | ||
|
|
||
| # Drop telemetry and state probes recorded after the fresh-deployment assertion above. | ||
| rm -f "$OUT_REQUESTS" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| bundle: | ||
| name: dms-migration-before-deploy-$UNIQUE_NAME | ||
|
|
||
| experimental: | ||
| deployment_history: false | ||
|
|
||
| sync: | ||
| exclude: | ||
| - LOG.* | ||
|
|
||
| resources: | ||
| jobs: | ||
| job: | ||
| name: migration-before-deploy | ||
| schemas: | ||
| schema: | ||
| name: dms_migration_before_deploy_$UNIQUE_NAME | ||
| catalog_name: main |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| Cloud = false | ||
| EnvMatrix.DMS = ["true"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
|
|
||
| === Setup (LOG.setup): create a job and schema without history | ||
| Next deploy enables history and recreates the schema in another catalog, requiring approval. | ||
| === 1. Fail deployment planning | ||
| Expect: migration version 2 stays complete; no deployment version is created. | ||
| >>> fault.py GET /api/2.2/jobs/get* 403 0 1 | ||
|
|
||
| >>> musterr [CLI] bundle deploy -qq | ||
| Enabling deployment history... | ||
| Deployment history enabled: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=2 | ||
| Error: cannot plan resources.jobs.job: reading id="[JOB_ID]": Fault injected by test. (403 INJECTED) | ||
|
|
||
| Endpoint: GET [DATABRICKS_URL]/api/2.2/jobs/get?job_id=[JOB_ID] | ||
| HTTP Status: 403 Forbidden | ||
| API error_code: INJECTED | ||
| API message: Fault injected by test. | ||
|
|
||
| Error: planning failed | ||
|
|
||
|
|
||
| === Request order: complete migration -> upload marker -> read DMS -> plan jobs | ||
| >>> print_requests.py --dms --get //api/2.0/bundle //jobs/get //workspace/import //workspace-files/import-file --oneline | ||
| {"method":"POST","path":"/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/2/complete"} | ||
| {"method":"POST","path":"/Workspace/Users/[USERNAME]/.bundle/dms-migration-before-deploy-[UNIQUE_NAME]/default/state/resources.json"} | ||
| {"method":"GET","path":"/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/resources"} | ||
| {"method":"GET","path":"/api/2.2/jobs/get"} | ||
|
|
||
| === Remote state: version 2 marker; original resources preserved in the backup | ||
| >>> MSYS_NO_PATHCONV=1 [CLI] workspace export /Workspace/Users/[USERNAME]/.bundle/dms-migration-before-deploy-[UNIQUE_NAME]/default/state/resources.json --format AUTO --file .databricks/resources.marker.json | ||
|
|
||
| >>> jq -c {serial, features, state} .databricks/resources.marker.json | ||
| {"serial":2,"features":{"deployment_history":{}},"state":{}} | ||
|
|
||
| >>> MSYS_NO_PATHCONV=1 [CLI] workspace export /Workspace/Users/[USERNAME]/.bundle/dms-migration-before-deploy-[UNIQUE_NAME]/default/state/resources.json.backup --format AUTO --file .databricks/resources.backup.json | ||
|
|
||
| >>> cmp .databricks/resources.before-migration.json .databricks/resources.backup.json | ||
|
|
||
| === Migration result: only completed version 2, with INITIAL_REGISTER operations | ||
| >>> MSYS_NO_PATHCONV=1 [CLI] bundle-deployments list-versions deployments/[DEPLOYMENT_ID] -o json | ||
| [{"version_id":"2","status":"VERSION_STATUS_COMPLETED","completion_reason":"VERSION_COMPLETE_SUCCESS"}] | ||
|
|
||
| >>> MSYS_NO_PATHCONV=1 [CLI] bundle-deployments list-operations deployments/[DEPLOYMENT_ID]/versions/2 -o json | ||
| [{"resource_key":"jobs.job","action_type":"OPERATION_ACTION_TYPE_INITIAL_REGISTER","status":"OPERATION_STATUS_SUCCEEDED"},{"resource_key":"schemas.schema","action_type":"OPERATION_ACTION_TYPE_INITIAL_REGISTER","status":"OPERATION_STATUS_SUCCEEDED"}] | ||
|
|
||
| === 2. Retry without approval | ||
| Expect: deployment declined; migration stays at version 2 and is not repeated. | ||
| >>> musterr [CLI] bundle deploy -qq | ||
|
|
||
| This action will result in the deletion or recreation of the following UC schemas. Any underlying data may be lost: | ||
| recreate resources.schemas.schema | ||
| Error: the deployment requires destructive actions, but the current console does not support prompting. | ||
| Deleting data assets such as schemas, pipelines, or volumes may cause permanent data loss and should be carefully reviewed. | ||
| To proceed, use --auto-approve after reviewing the plan above. | ||
|
|
||
|
|
||
| === Local and remote markers must stay byte-for-byte unchanged | ||
| >>> cmp .databricks/resources.marker.json .databricks/bundle/default/resources.json | ||
|
|
||
| >>> MSYS_NO_PATHCONV=1 [CLI] workspace export /Workspace/Users/[USERNAME]/.bundle/dms-migration-before-deploy-[UNIQUE_NAME]/default/state/resources.json --format AUTO --file .databricks/resources.after-decline.json | ||
|
|
||
| >>> cmp .databricks/resources.marker.json .databricks/resources.after-decline.json | ||
|
|
||
| === No DMS writes and no extra version on the declined retry | ||
| >>> print_requests.py --dms //api/2.0/bundle | ||
|
|
||
| >>> MSYS_NO_PATHCONV=1 [CLI] bundle-deployments list-versions deployments/[DEPLOYMENT_ID] -o json | ||
| [{"version_id":"2","status":"VERSION_STATUS_COMPLETED","completion_reason":"VERSION_COMPLETE_SUCCESS"}] | ||
|
|
||
| === 3. Approve deployment | ||
| Expect: ordinary deployment version 3; version 2 remains the only migration. | ||
| >>> [CLI] bundle deploy --auto-approve -qq | ||
|
|
||
| This action will result in the deletion or recreation of the following UC schemas. Any underlying data may be lost: | ||
| recreate resources.schemas.schema | ||
|
|
||
| >>> [CLI] bundle summary -o json | ||
| {"latest_version_id":3} | ||
|
|
||
| >>> MSYS_NO_PATHCONV=1 [CLI] bundle-deployments list-versions deployments/[DEPLOYMENT_ID] -o json | ||
| [{"version_id":"2","status":"VERSION_STATUS_COMPLETED","completion_reason":"VERSION_COMPLETE_SUCCESS"},{"version_id":"3","status":"VERSION_STATUS_COMPLETED","completion_reason":"VERSION_COMPLETE_SUCCESS"}] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| envsubst < databricks.yml.tmpl > databricks.yml | ||
| state_path="/Workspace/Users/${CURRENT_USER_NAME}/.bundle/dms-migration-before-deploy-${UNIQUE_NAME}/default/state" | ||
| other_catalog="dms_other_${UNIQUE_NAME}" | ||
|
|
||
| title "Setup (LOG.setup): create a job and schema without history\nNext deploy enables history and recreates the schema in another catalog, requiring approval." | ||
| { | ||
|
Contributor
There was a problem hiding this comment. Why do we need to group it? cannot we record the command + output with trace? This makes output easier to read. As far as I can see the commands below have no or very little output. |
||
| $CLI bundle deploy -qq | ||
| DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=false replace_ids.py | ||
|
Contributor
There was a problem hiding this comment. Is it significant to have env var here that matches config? |
||
| cp .databricks/bundle/default/resources.json .databricks/resources.before-migration.json | ||
| $CLI catalogs create "$other_catalog" | ||
| update_file.py databricks.yml 'deployment_history: false' 'deployment_history: true' | ||
| update_file.py databricks.yml 'catalog_name: main' "catalog_name: ${other_catalog}" | ||
| } &> LOG.setup | ||
| trap '$CLI bundle destroy --auto-approve >> LOG.cleanup 2>&1; $CLI catalogs delete "$other_catalog" --force >> LOG.cleanup 2>&1; rm -f "$OUT_REQUESTS"' EXIT | ||
| rm -f "$OUT_REQUESTS" | ||
|
|
||
| title "1. Fail deployment planning\nExpect: migration version 2 stays complete; no deployment version is created." | ||
| trace fault.py 'GET /api/2.2/jobs/get*' 403 0 1 | ||
| trace musterr $CLI bundle deploy -qq | ||
| deployment_id=$(MSYS_NO_PATHCONV=1 $CLI workspace get-status "$state_path/resources.deployment.json" -o json | jq -r '.object_id') | ||
| add_repl "$deployment_id" DEPLOYMENT_ID | ||
| title "Request order: complete migration -> upload marker -> read DMS -> plan jobs" | ||
| trace print_requests.py --dms --get //api/2.0/bundle //jobs/get //workspace/import //workspace-files/import-file --oneline | | ||
| MSYS_NO_PATHCONV=1 jq -c --arg marker "$state_path/resources.json" ' | ||
| (.body.multipart_form.path // .body.path // (.path | ltrimstr("/api/2.0/workspace-files/import-file"))) as $destination | | ||
| select((.path | endswith("/complete")) or (.path | endswith("/resources")) or .path == "/api/2.2/jobs/get" or $destination == $marker) | | ||
| {method, path: (if $destination == $marker then $destination else .path end)}' | ||
| title "Remote state: version 2 marker; original resources preserved in the backup" | ||
| trace MSYS_NO_PATHCONV=1 $CLI workspace export "$state_path/resources.json" --format AUTO --file .databricks/resources.marker.json | ||
| trace jq -c '{serial, features, state}' .databricks/resources.marker.json | ||
| trace MSYS_NO_PATHCONV=1 $CLI workspace export "$state_path/resources.json.backup" --format AUTO --file .databricks/resources.backup.json | ||
| trace cmp .databricks/resources.before-migration.json .databricks/resources.backup.json | ||
| title "Migration result: only completed version 2, with INITIAL_REGISTER operations" | ||
| trace MSYS_NO_PATHCONV=1 $CLI bundle-deployments list-versions "deployments/${deployment_id}" -o json | jq -c 'map({version_id, status, completion_reason})' | ||
| trace MSYS_NO_PATHCONV=1 $CLI bundle-deployments list-operations "deployments/${deployment_id}/versions/2" -o json | jq -c 'sort_by(.resource_key) | map({resource_key, action_type, status})' | ||
|
|
||
| title "2. Retry without approval\nExpect: deployment declined; migration stays at version 2 and is not repeated." | ||
| trace musterr $CLI bundle deploy -qq | ||
| title "Local and remote markers must stay byte-for-byte unchanged" | ||
| trace cmp .databricks/resources.marker.json .databricks/bundle/default/resources.json | ||
| trace MSYS_NO_PATHCONV=1 $CLI workspace export "$state_path/resources.json" --format AUTO --file .databricks/resources.after-decline.json | ||
| trace cmp .databricks/resources.marker.json .databricks/resources.after-decline.json | ||
| title "No DMS writes and no extra version on the declined retry" | ||
| trace print_requests.py --dms //api/2.0/bundle | ||
| trace MSYS_NO_PATHCONV=1 $CLI bundle-deployments list-versions "deployments/${deployment_id}" -o json | jq -c 'map({version_id, status, completion_reason})' | ||
|
|
||
| title "3. Approve deployment\nExpect: ordinary deployment version 3; version 2 remains the only migration." | ||
| trace $CLI bundle deploy --auto-approve -qq | ||
| trace $CLI bundle summary -o json | jq -c '.bundle.deployment | {latest_version_id}' | ||
| trace MSYS_NO_PATHCONV=1 $CLI bundle-deployments list-versions "deployments/${deployment_id}" -o json | jq -c 'sort_by(.version_id | tonumber) | map({version_id, status, completion_reason})' | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| # Fault injection exercises a real planner failure against the local testserver. | ||
| Cloud = false |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| bundle: | ||
| name: dms-migration-$UNIQUE_NAME | ||
|
|
||
| experimental: | ||
| deployment_history: false | ||
|
|
||
| resources: | ||
| jobs: | ||
| one: | ||
| name: one | ||
| two: | ||
| name: two | ||
| tasks: | ||
| - task_key: run_one | ||
| run_job_task: | ||
| job_id: ${resources.jobs.one.id} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| Cloud = true | ||
| EnvMatrix.DMS = [""] |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
general observation, this test is rewritten so much it is practically a different test. Maybe worth adding this new test separately and record output changes for the original test (with minimal script changes).