Skip to content

bundle: add DATABRICKS_BUNDLE_MANAGED_STATE gate + DMS-backed deployment lock#5331

Draft
shreyas-goenka wants to merge 1 commit into
shreyas-goenka/bundle-managed-state-gatefrom
shreyas-goenka/bundle-dms-lock-impl
Draft

bundle: add DATABRICKS_BUNDLE_MANAGED_STATE gate + DMS-backed deployment lock#5331
shreyas-goenka wants to merge 1 commit into
shreyas-goenka/bundle-managed-state-gatefrom
shreyas-goenka/bundle-dms-lock-impl

Conversation

@shreyas-goenka
Copy link
Copy Markdown
Contributor

Summary

Step 3 of the DMS split (see PR #4856 for the full kitchen-sink change).

  • Adds the DATABRICKS_BUNDLE_MANAGED_STATE environment variable
    (bundle/env/deployment_metadata.go). The value is parsed with the usual
    boolean tolerance — true/false, 1/0, yes/no, on/off
    (case-insensitive). Default behavior (var unset) is unchanged.
  • Adds a DMS-backed DeploymentLock implementation that calls the bundle
    service through the SDK client. Acquire = CreateDeployment (for fresh
    deployments) / GetDeployment + CreateVersion, plus a background
    heartbeat goroutine that renews the lock lease. Release =
    CompleteVersion, plus DeleteDeployment on successful destroy.
  • Updates lock.NewDeploymentLock to branch on env.IsManagedState(ctx)
    and return the DMS impl, falling back to the existing workspace
    filesystem lock otherwise. The factory now takes ctx so it can read
    the env var — phases call sites updated to pass it.
  • Adds a fake DMS server in libs/testserver (modeled on the existing
    apps.go pattern) and registers the routes in handlers.go.
  • Adds the acceptance/bundle/dms/release-lock-error acceptance test
    that exercises a CompleteVersion failure end-to-end.

Dependencies

Notes vs the kitchen-sink branch (shreyas-goenka/deployment-metadata-service)

  • The kitchen-sink branch is built on libs/tmpdms (hand-rolled DMS types
    written before the SDK had the bundle service). This PR uses the real
    SDK package instead. The on-the-wire schema matches — request/response
    bodies and URL shapes are identical — but Go type names differ
    (tmpdms.CreateDeploymentRequest.DeploymentID → SDK
    bundle.CreateDeploymentRequest.DeploymentId; enum names switched from
    tmpdms.VersionTypeDeploy to SDK
    bundle.VersionTypeVersionTypeDeploy, etc.).
  • The kitchen-sink branch's deployment_metadata_service.go reads/writes
    bundle/statemgmt.ManagedServiceJSON / ManagedServiceFileName. Those
    live in statemgmt/managed_service_json.go in the kitchen-sink branch
    — that file is step 4 of the split. To keep this PR scoped to the lock,
    the deployment-ID persistence (struct + filename) is kept as
    package-private inside bundle/deploy/lock and will be promoted to
    statemgmt when step 4 lands and state_pull.go/state_push.go also
    need it.
  • async_reporter.go from the kitchen-sink branch is intentionally not
    included here. It exists to feed operation events into the DMS, which
    requires the direct.OperationReporter hook on
    DeploymentBundle that lives in step 5. The lock works end-to-end
    without it; the heartbeat goroutine that the original task description
    conflated with async_reporter.go lives in
    deployment_metadata_service.go (startHeartbeat).
  • Today the DMS path is gated solely by
    env.IsManagedState. The kitchen-sink branch uses
    statemgmt.IsDmsActive, which also factors in server-side opt-in. That
    predicate is part of step 4; promoting the gate to it is an open item
    for the next PR.

Test plan

  • go build ./... clean.
  • go vet ./... clean (ignoring pre-existing JSON tag warnings).
  • go test ./bundle/... ./libs/testserver/... ./cmd/... — all green.
  • go test ./acceptance -run "TestAccept/bundle/dms/release-lock-error"
    passes without -update.
  • ./task lint clean.
  • Manual verification that IsManagedState returns the right bool for
    true, false, TRUE, yes, 1, 0, no, off, garbage, and unset.
  • Existing filesystem-lock-using acceptance tests
    (acceptance/bundle/deploy/empty-bundle, bundle/deploy/files,
    bundle/destroy) still pass — the default DATABRICKS_BUNDLE_MANAGED_STATE path is the workspace filesystem lock, unchanged from PR bundle: extract DeploymentLock interface + workspace filesystem impl #5314.

This pull request and its description were written by Isaac.

…ent lock

Wires up a new env-var gate (DATABRICKS_BUNDLE_MANAGED_STATE) that switches
the deployment lock from the workspace filesystem to the bundle deployment
metadata service (DMS). The env var accepts the usual boolean spellings
(true/false, 1/0, yes/no, on/off); the historical filesystem lock is the
default.

The DMS-backed lock uses the SDK's databricks-sdk-go/service/bundle client
(merged via #5311, available since v0.135.0) — no hand-rolled DMS client.
Acquire calls CreateDeployment / CreateVersion and starts a background
heartbeat goroutine; Release stops the heartbeat and calls CompleteVersion
(plus DeleteDeployment on successful destroy). Bind and Unbind are not yet
supported under DMS and return an error at lock construction.

Deployment ID persistence lives inside the lock package for now
(managed_service.json in the workspace state dir). In step 4 of the DMS
split it will move to bundle/statemgmt so it can be shared with the
state-from-DMS path.

Co-authored-by: Isaac
@shreyas-goenka shreyas-goenka changed the base branch from shreyas-goenka/bundle-lock-abstraction to shreyas-goenka/bundle-managed-state-gate May 26, 2026 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant