Repository navigation
Detect notebooks with the workspace's first-line rule - #7290
Merged
Merged
Conversation
The CLI classifies a local source file as a notebook only to predict the remote name the workspace will give it (sync snapshot, translate_paths, deployment state, workspace import-dir). It required the first line to equal the language-specific header exactly, while the import-file endpoint only requires the first line of the leading 200 bytes to contain "Databricks notebook source" (NotebookUtils.checkSourceFileNotebook). A BOM-prefixed SQL notebook therefore landed as notebook "bom" while the sync snapshot tracked "bom.sql"; removing it locally deleted a path that did not exist and left the notebook behind. detectSource now applies the server's rule. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 1 changed tests (limit=20, 0 not selected) 2 tests: regression=2 unreleased=0 coverage=0 failing=0
TestAccept/bundle/sync/replace-node-type/DMS=: branch ✅ | main (050b89d) ❌ | latest (v1.23.0) ❌main (050b89d): latest (v1.23.0): same output as main (050b89d) TestAccept/bundle/sync/replace-node-type/DMS=true: branch ✅ | main (050b89d) ❌ | latest (v1.23.0) ❌main (050b89d): latest (v1.23.0): same output as main (050b89d) |
Collaborator
Integration test reportCommit: d8b0d65
Top 6 slowest tests (at least 2 minutes):
|
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Resolves the replace-node-type conflict with #7247: keep the fixed deletion output and drop the Badness entry, since this branch fixes the bug it recorded. Co-authored-by: Isaac <no-reply@databricks.com>
pietern
approved these changes
Nov 3, 2026
Collaborator
Integration test reportCommit: 4b902e8
485 interesting tests: 339 MISS, 130 FAIL, 16 flaky
Top 50 slowest tests (at least 2 minutes):
|
deco-sdk-tagging Bot
added a commit
that referenced
this pull request
Nov 4, 2026
## Release v1.24.0 ### Notable Changes * A variable used as the value of a secret is now sensitive: referencing it from any other field, such as a job parameter or a secret's `comment`, is an error because the value would be used in plain text. Mark other secret-holding variables with `sensitive: true` to get the same protection. ([#7288](#7288)) ### CLI * Fix `databricks auth login` and `databricks configure` overwriting the parent profile's host and token when saving a profile with a dotted name such as `team.prod` next to an existing `[team]`. ([#7266](#7266)) * `databricks auth login` and `databricks auth token` now reject the name of an existing profile after "Create a new profile" instead of logging in to that profile. ([#7196](#7196)) * `databricks auth login` and `databricks auth token` now reject a host that is not a URL with a hostname, such as `/`, `?o=123` or `https://`, instead of falling back to `DATABRICKS_HOST` or failing later. An empty host after "Enter a host URL manually" in the login profile picker no longer starts discovery login. ([#7195](#7195)) * Prompts that validate their input, such as the `databricks configure` host prompt, no longer open with a red cross before anything is typed. ([#7314](#7314)) * `databricks ssh connect` now documents `--environment-version` for choosing the serverless environment version, starts a separate default-named server for each version, and rejects the flag together with `--cluster`. Default connection names also include the environment version in their hash, so the default name when `--environment-version` is not specified changes with this release. Configuration associated with the old connection name (secret scope, SSH keys and known hosts in your local `.databricks` folder) will no longer be used and can be safely removed. ([#7310](#7310)) ### AI Runtime * Show workspace-supported accelerator types in `databricks air run -h config.compute.accelerator_type`. ([#7275](#7275)) * `air list` now reports `No active runs found.` for an empty active-only result. ([#7276](#7276)) * Stop printing a retry notice on every `air run --watch`; retries are still reported when they happen. ([#7270](#7270)) ### Bundles * Fix `bundle plan` reporting an update to an app's `config.env` values on every run, which also redeployed the app on every `bundle deploy`. The Apps API returns environment variable names only, so `value` and `value_from` are no longer compared against the remote deployment. ([#7326](#7326)) * Add the `data_quality_monitors` resource for data quality monitors on Unity Catalog schemas and tables (direct engine only). ([#7094](#7094)) * `bundle deploy` now asks for confirmation before recreating or deleting a Feature View, and `bundle deploy` and `bundle destroy` warn that FeatureSpecs using a Feature View are not checked. ([#7164](#7164)) * Accept grant `privileges` values that the SDK enum does not list yet: `bundle validate` no longer warns (or errors under `--strict`) on them, and the PyDABs `Privilege` enum accepts unknown strings instead of raising. The JSON schema and PyDABs still advertise the known values for completion. ([#7315](#7315)) * Add the `materialization_sets` resource, which materializes Feature Store Feature Views to offline and online stores. ([#7164](#7164)) * Detect notebooks with the same rule as the workspace: a `.py`, `.sql`, `.scala` or `.r` file is a notebook when its first line contains `Databricks notebook source`, for example after a byte order mark. Previously such a file was tracked as a plain file, so removing it locally left the remote notebook behind. ([#7290](#7290)) * Add `sensitive: true` for bundle variables. A variable used as the value of a secret is sensitive by default. `bundle validate -o json` redacts only sensitive variables instead of every string variable, and a sensitive variable can only be referenced from a sensitive resource field (such as a secret's `value`) or from another sensitive variable. ([#7288](#7288)) ### Dependency Updates * Bump dependencies with known vulnerabilities. ([#7305](#7305)) * Bump dependencies with known vulnerabilities. ([#7327](#7327)) * Update the Go SDK to [v0.191.1](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.191.1). ([#7277](#7277)) * Update the Go SDK to [v0.193.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.193.0). ([#7303](#7303))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
libs/notebooknow classifies a source file the way the workspace'simport-fileendpoint does (NotebookUtils.checkSourceFileNotebookin universe): a.py,.sql,.scalaor.rfile is a notebook when the first line of its leading 200 bytes containsDatabricks notebook source. Previously the CLI required the first line to equal the language-specific header exactly, so a leading byte order mark, or a different comment prefix, made the CLI treat a file the workspace imports as a notebook as a plain file.The CLI only uses this classification to predict the remote name the workspace will give a file: in the sync snapshot,
translate_paths, deployment state andworkspace import-dir. When the prediction disagrees with the server, the snapshot tracksbom.sqlwhile the workspace stored notebookbom. Removing the file locally then deletes a path that doesn't exist, which sync tolerates, and the notebook is orphaned. This is the badness documented in #7247.Why
Since #7218 the server classifies uploads again, so the CLI's local rule has to match the server's for the snapshot to address the right remote object.
Tests
bundle/sync/replace-node-typenow also removes both local files and syncs again; bothbomandplain.sqlare deleted. Passes locally and onaws-cliin both DMS variants. On the real workspace the old code'sbom.sqldelete 404'd and leftbombehind. Add acceptance coverage for BOM notebook deletion badness #7247 added this step with aBadnessentry recording the orphaned notebook; this PR removes that entry and the "BUG" section title now that the step passes.TestDetectBytesgains cases for the marker mid-line, another language's comment prefix, the marker on the second line, and the marker past the 200-byte buffer.libs/notebook:libs/sync,libs/fileset,libs/filer,bundle/config/mutator,bundle/deploy,bundle/generate,cmd/workspace/workspace.templates/default-python/*,templates/pydabs/deploy-classic,artifacts/issue_3109,migrate/command/default-python), all becauseuvcan't resolvehatchlingthrough the internal proxy in my environment. CI covers them.This pull request and its description were written by Isaac.