acc: read recorded ids and state from the deployment metadata service - #6503
Closed
shreyas-goenka wants to merge 1 commit into
Closed
acc: read recorded ids and state from the deployment metadata service#6503shreyas-goenka wants to merge 1 commit into
shreyas-goenka wants to merge 1 commit into
Conversation
WIP. The id helpers read the direct-engine state file, which is the wrong source while a bundle records deployment history: the service owns the resource set. Add dms_resources.get_resources, which resolves the deployment from the workspace node the service registers and lists its resources, and have read_id, replace_ids and read_state use it when recording is on. Not ready: resolving the remote state path runs "bundle validate", which fails for tests that deploy with --var or other flags the helper cannot know (see bundle/resources/secrets/*). And a hollow state file - the reason for reading from the service at all - breaks "bundle generate", bind and configsync, which open the state with no DMS client and need the resource ids from it. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Stacked on #6475 (do not merge before it).
read_id.py,replace_ids.pyandread_state.pyread the direct-engine state file; while a bundle records deployment history the service owns the resource set, so they now ask it instead. Newacceptance/bin/dms_resources.pyresolves the deployment from the workspace node the service registers under the state path (the node's object id is the deployment id, perlibs/dms/resolve.go) and lists its resources, which carry bothresource_idand the recorded state.Why
Groundwork for making
resources.jsona hollow marker while recording: the file would carry only its header, and ids/state would come from the service.Tests
WIP — pushed so it can be tried out, not ready to merge.
bundle/resources/schemas/updatepasses under both DMS variants, reading state and id from the service. Three known blockers:bundle validate, which fails for tests that deploy with--varor other flags the helper cannot know —bundle/resources/secrets/*abort.bundle generate(and by inspection bind and configsync): they open the state with no DMS client and read__id__from it, so generate callsGET /api/2.0/lakeview/dashboards/with an empty id.With the state file left intact (as in #6475), the suite is green apart from
bundle/resources/{secrets/*,pipelines/update,apps/job_permissions}plus the pre-existing offlinetemplates/lakeflow-integrations.This PR was written by Claude Code.