Skip to content

Drop permission and grant entries that reference undeployed resources on migration - #7229

Draft
denik wants to merge 3 commits into
mainfrom
denik/migrate-permission-undeployed-ref
Draft

Drop permission and grant entries that reference undeployed resources on migration#7229
denik wants to merge 3 commits into
mainfrom
denik/migrate-permission-undeployed-ref

Conversation

@denik

@denik denik commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Changes

Follow-up to #7157. When migrating terraform state, a permission or grant entry that references a resource not yet deployed (e.g. service_principal_name: ${resources.apps.new_app.service_principal_client_id}) had its principal cleared, leaving an entry without a key in the migrated state. The entry was never deployed, so it is now dropped and the first direct plan adds it.

Tests

Unit tests in bundle/migrate/build_state_test.go and acceptance test bundle/migrate/auto/new-resource-ref-permissions.

This pull request and its description were written by Isaac.

@github-actions github-actions Bot added the DABs DABs related issues label Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Commit: 392be39cf | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0

test branch main (e9e2d63) latest (v1.23.0)
TestAccept/bundle/migrate/auto/new-resource-ref-permissions/DMS= ✅ ❌ ❌
TestAccept/bundle/migrate/auto/new-resource-ref-permissions/DMS=: branch ✅ | main (e9e2d63) ❌ | latest (v1.23.0) ❌

main (e9e2d63):

acceptance_test.go:1341: Diff:
        --- bundle/migrate/auto/new-resource-ref-permissions/output.txt
        +++ /tmp/TestAcceptbundlemigrateautonew-resource-ref-permissionsDMS=3293736245/002/output.txt
        @@ -5,235 +5,10 @@
         === Add a new app and a job permission that references its service principal
         === Plan shows the new app and the added permission
         >>> [CLI] bundle plan -o json
        -Notice: migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
        -{
        -  "plan_version": [PLAN_VERSION],
        -  "cli_version": "[CLI_VERSION]",
        -  "lineage": "[UUID]",
        -  "serial": 7,
        -  "plan": {
        -    "resources.apps.new_app": {
        -      "action": "create",
        -      "new_state": {
        -        "value": {
        -          "description": "",
        -          "name": "new-app",
        -          "source_code_path": "/Workspace/Users/[USERNAME]/.bundle/migrate-basic-test/default/files"
        -        }
        -      }
        -    },
        -    "resources.jobs.test_job": {
        -      "action": "skip",
        -      "remote_state": {
        -        "created_time": [UNIX_TIME_MILLIS][0],
        -        "creator_user_name": "[USERNAME]",
        -        "deployment": {
        -          "kind": "BUNDLE",
        -          "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/migrate-basic-test/default/state/metadata.json"
        -        },
        -        "edit_mode": "UI_LOCKED",
        -        "email_notifications": {},
        -        "format": "MULTI_TASK",
        -        "job_id": [TEST_JOB_ID],
        -        "max_concurrent_runs": 1,
        -        "name": "Test Migration Job",
        -        "queue": {
        -          "enabled": true
        -        },
        -        "run_as_user_name": "[USERNAME]",
        -        "tasks": [
        -          {
        -            "email_notifications": {},
        -            "notebook_task": {
        -              "notebook_path": "/Workspace/Users/[USERNAME]/.bundle/migrate-basic-test/default/files/notebook",
        -              "source": "WORKSPACE"
        -            },
        -            "run_if": "ALL_SUCCESS",
        -            "task_key": "main",
        -            "timeout_seconds": 0
        -          }
        -        ],
        -        "timeout_seconds": 0,
        -        "webhook_notifications": {}
        -      },
        -      "changes": {
        -        "email_notifications": {
        -          "action": "skip",
        -          "reason": "empty",
        -          "remote": {}
        -        },
        -        "tasks[task_key='main'].email_notifications": {
        -          "action": "skip",
        -          "reason": "empty",
        -          "remote": {}
        -        },
        -        "tasks[task_key='main'].notebook_task.source": {
        -          "action": "skip",
        -          "reason": "backend_default",
        -          "remote": "WORKSPACE"
        -        },
        -        "tasks[task_key='main'].run_if": {
        -          "action": "skip",
        -          "reason": "backend_default",
        -          "remote": "ALL_SUCCESS"
        -        },
        -        "tasks[task_key='main'].timeout_seconds": {
        -          "action": "skip",
        -          "reason": "empty",
        -          "remote": 0
        -        },
        -        "timeout_seconds": {
        -          "action": "skip",
        -          "reason": "empty",
        -          "remote": 0
        -        },
        -        "webhook_notifications": {
        -          "action": "skip",
        -          "reason": "empty",
        -          "remote": {}
        -        }
        -      }
        -    },
        -    "resources.jobs.test_job.permissions": {
        -      "depends_on": [
        -        {
        -          "node": "resources.apps.new_app",
        -          "label": "${resources.apps.new_app.service_principal_client_id}"
        -        },
        -        {
        -          "node": "resources.jobs.test_job",
        -          "label": "${resources.jobs.test_job.id}"
        -        }
        -      ],
        -      "action": "update",
        -      "new_state": {
        -        "value": {
        -          "object_id": "/jobs/[TEST_JOB_ID]",
        -          "__embed__": [
        -            {
        -              "level": "CAN_MANAGE",
        -              "service_principal_name": "${resources.apps.new_app.service_principal_client_id}"
        -            },
        -            {
        -              "level": "CAN_VIEW",
        -              "user_name": "viewer@databricks.com"
        -            },
        -            {
        -              "level": "IS_OWNER",
        -              "user_name": "[USERNAME]"
        -            }
        -          ]
        -        },
        -        "vars": {
        -          "[0].service_principal_name": "${resources.apps.new_app.service_principal_client_id}"
        -        }
        -      },
        -      "remote_state": {
        -        "object_id": "/jobs/[TEST_JOB_ID]",
        -        "__embed__": [
        -          {
        -            "level": "CAN_VIEW",
        -            "user_name": "viewer@databricks.com"
        -          },
        -          {
        -            "level": "IS_OWNER",
        -            "user_name": "[USERNAME]"
        -          }
        -        ]
        -      },
        -      "changes": {
        -        "[service_principal_name='${resources.apps.new_app.service_principal_client_id}']": {
        -          "action": "update",
        -          "new": {
        -            "level": "CAN_MANAGE",
        -            "service_principal_name": "${resources.apps.new_app.service_principal_client_id}"
        -          }
        -        }
        -      }
        -    },
        -    "resources.pipelines.test_pipeline": {
        -      "action": "skip",
        -      "remote_state": {
        -        "channel": "CURRENT",
        -        "creator_user_name": "[USERNAME]",
        -        "deployment": {
        -          "kind": "BUNDLE",
        -          "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/migrate-basic-test/default/state/metadata.json"
        -        },
        -        "edition": "ADVANCED",
        -        "effective_publishing_mode": "DEFAULT_PUBLISHING_MODE",
        -        "id": "[TEST_PIPELINE_ID]",
        -        "last_modified": [UNIX_TIME_MILLIS][1],
        -        "libraries": [
        -          {
        -            "notebook": {
        -              "path": "/Workspace/Users/[USERNAME]/.bundle/migrate-basic-test/default/files/pipeline"
        -            }
        -          }
        -        ],
        -        "name": "Test Migration Pipeline",
        -        "pipeline_id": "[TEST_PIPELINE_ID]",
        -        "run_as_user_name": "[USERNAME]",
        -        "state": "IDLE",
        -        "storage": "dbfs:[PIPELINE_TEST_PIPELINE_ID]"
        -      },
        -      "changes": {
        -        "storage": {
        -          "action": "skip",
        -          "reason": "immutable_backend_value",
        -          "remote": "dbfs:[PIPELINE_TEST_PIPELINE_ID]"
        -        }
        -      }
        -    },
        -    "resources.pipelines.test_pipeline.permissions": {
        -      "depends_on": [
        -        {
        -          "node": "resources.pipelines.test_pipeline",
        -          "label": "${resources.pipelines.test_pipeline.id}"
        -        }
        -      ],
        -      "action": "skip",
        -      "remote_state": {
        -        "object_id": "[PIPELINE_TEST_PIPELINE_ID]",
        -        "__embed__": [
        -          {
        -            "level": "CAN_MANAGE",
... (45 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)

latest (v1.23.0): same output as main (e9e2d63)

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 392be39

Run: 41047131522

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 281 16 5:15
✅​ aws-windows-latest-4core-16gb 283 14 5:35
🔄​ azure linux-2core-8gb 2 278 16 5:14
✅​ azure-windows-latest-4core-16gb 282 14 6:32
✅​ gcp linux-2core-8gb 281 16 5:28
✅​ gcp-windows-latest-4core-16gb 283 14 4:02
Test Name azure linux-2core-8gb
🔄​ TestFsCpFileToFile 🔄​f
🔄​ TestFsCpFileToFile/uc-volumes_to_local 🔄​f
Top 6 slowest tests (at least 2 minutes):
duration env testname
6:25 azure-windows-latest-4core-16gb TestAccept
5:27 aws-windows-latest-4core-16gb TestAccept
3:57 gcp-windows-latest-4core-16gb TestAccept
3:48 azure linux-2core-8gb TestAccept
3:45 gcp linux-2core-8gb TestAccept
3:45 aws linux-2core-8gb TestAccept

denik and others added 3 commits October 30, 2026 12:13
@denik Isaac
Drop permission and grant entries that reference undeployed resources…
f29a666 
… on migration

Clearing the principal of an entry left an entry without a key in the
migrated state. The entry was not deployed, so drop it and let the first
direct plan add it.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik Isaac
Add changelog fragment
1d1c7e6 
Co-authored-by: Isaac <no-reply@databricks.com>
@denik Isaac
acc: fix yamlfmt of the commented permission in new-resource-ref-perm…
392be39 
…issions

Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik force-pushed the denik/migrate-permission-undeployed-ref branch from fafd106 to 392be39 Compare October 30, 2026 11:17

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

DABs DABs related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@denik @eng-dev-ecosystem-bot