Repository navigation
Conversation
denik
force-pushed
the
denik/bundle-bitmap
branch
from
July 30, 2026 10:11
221d417 to
39faace
Compare
Collaborator
✅ Integration test reportCommit: e070c71
Top 3 slowest tests (at least 2 minutes):
|
denik
force-pushed
the
denik/bundle-bitmap
branch
2 times, most recently
from
August 3, 2026 10:01
eb6b1f0 to
29c5b44
Compare
denik
force-pushed
the
denik/bundle-bitmap
branch
2 times, most recently
from
August 17, 2026 11:52
45b8bfe to
9ad0bbb
Compare
Contributor
|
This PR has not received an update in a while. If you want to keep this PR open, please leave a comment below or push a new commit and auto-close will be canceled. |
denik
force-pushed
the
denik/bundle-bitmap
branch
from
November 3, 2026 16:34
9ad0bbb to
96a6df5
Compare
denik
force-pushed
the
denik/bundle-bitmap
branch
from
November 3, 2026 16:42
96a6df5 to
f972c44
Compare
Contributor
|
Commit: Regression test reportSelected 10 changed tests (limit=20, 0 not selected) 79 tests: regression=78 unreleased=0 coverage=1 failing=0
TestAccept/bundle/bitmap/full: branch ✅ | main (df79350) ❌ | latest (v1.25.0) ❌main (df79350): latest (v1.25.0): TestAccept/bundle/bitmap/presence/DMS=: branch ✅ | main (df79350) ❌ | latest (v1.25.0) ❌main (df79350): latest (v1.25.0): same output as main (df79350) TestAccept/bundle/bitmap/presence/DMS=true: branch ✅ | main (df79350) ❌ | latest (v1.25.0) ❌main (df79350): latest (v1.25.0): same output as main (df79350) 75 more failure details omitted, see the full report in the run. |
denik
force-pushed
the
denik/bundle-bitmap
branch
from
November 4, 2026 10:08
f972c44 to
1cc9186
Compare
denik
force-pushed
the
denik/bundle-bitmap
branch
3 times, most recently
from
November 5, 2026 13:52
5e60941 to
445b523
Compare
denik
force-pushed
the
denik/bundle-bitmap
branch
from
November 9, 2026 15:40
108e307 to
6d41413
Compare
Co-authored-by: Isaac
Encode presence of every bundle config field as one bit. The schema is derived by reflecting over config.Root, recorded in bundle/bitmap/schema.txt, and embedded into the binary. Append-only so bit positions stay stable. Adds hidden `databricks bundle bitmap` group: schema, update-schema [--validate], bitmap-text, bitmap. Wires generate-bitmap-schema into the Taskfile and runs the encoder over all no_drift invariant configs. Co-authored-by: Isaac
Kept in PR history for context; not landed. Co-authored-by: Isaac
- Every bool/*bool field now gets a "#true" value bit in addition to its presence bit, so a *bool encodes unset/false/true as 00/10/11. - decode_bitmap.py decodes the base64 bitmap against the schema; the dedicated acceptance test asserts it round-trips to bitmap-text. - Record the encoded bitmap and set-bit list for the default-python template. Co-authored-by: Isaac
Replace the string-keyed BoolValues map with a typed bundle.Telemetry struct whose fields feed both the deploy telemetry wire (via BoolValues()) and the bundle bitmap (namespaced under telemetry.*). Value-bearing flags whose false is meaningful are stored as True/False pairs so the bitmap can represent both outcomes; BoolValues() collapses each pair back to its single wire key. libs/cache gets a concrete Metrics struct (embedded into Telemetry) instead of the old interface. The wire is unchanged except that redundant false entries of one-hot groups (state_path_*, dms_undeclared_*) are no longer emitted; their true member is preserved. Co-authored-by: Isaac
Co-authored-by: Isaac
decode_bitmap.py was committed without the executable bit, so the round-trip step failed with "permission denied" (exit code differed across OSes, breaking the golden). Mark it executable. bitmap-text and bitmap are separate invocations sharing a cache dir, so the second recorded a cache hit instead of a miss and the round-trip check diverged. Give each its own cache dir so both observe a cold miss. Co-authored-by: Isaac
On a CRLF checkout (Windows) the embedded schema.txt lines kept a trailing \r, so they never matched the clean paths from WalkType. Merge's dedup then failed and the schema doubled, breaking bundle bitmap schema/bitmap-text on Windows. splitLines now normalizes CRLF. Co-authored-by: Isaac
The schema had telemetry fields appended at the tail (from append-only merges across a rebase). Since the PR has not landed, bit positions need not be stable yet, so regenerate from scratch to put telemetry fields in natural walk order. generate-bitmap-schema now also regenerates the bitmap acceptance goldens (out.schema.txt, out.bitmap-text.txt, encoded bitmap), which embed the schema and depend on bit positions; otherwise they go stale when the schema changes. Co-authored-by: Isaac
Existing telemetry (bundle/metrics AddBoolValue/SetBoolValue and the wire) is unchanged. Add a Telemetry struct on Bundle with a field per bool flag; each metrics call site additionally sets the matching field so the same signal reaches the bitmap. Value-bearing flags use True/False pairs so the presence-only bitmap can distinguish measured-false from not-measured. Fields have no json tags, so they are invisible to the config schema and walked into the bitmap under their Go field names (telemetry.*). Verified the telemetry and resource_deps goldens are byte-identical to main: the change is strictly additive to the existing wire. Co-authored-by: Isaac
The four dms_undeclared_* flags are independent booleans recorded with their real value on every deploy, so a measured false is meaningful (it is the common "no undeclared writers" case). As single presence bits a false collapsed to 0, indistinguishable from "not measured", losing the signal. Store each as a True/False pair like the other value-bearing flags. The telemetry wire (BoolValues) is unchanged. Co-authored-by: Isaac
The dms_undeclared_* flags are always co-emitted with the one-hot DMSCompat verdict, so a 0 bit is unambiguously "measured false" rather than "not measured" — a True/False pair adds no information. A pair is only warranted for a lone boolean with no co-located signal that the measurement ran (e.g. run_as_set). Adds a comment recording why. Co-authored-by: Isaac
The telemetry struct's fields have no json tags, so structwalk.Walk does not honor omitempty and visits every field regardless of value. Bits() ignored the visited value and set the bit on mere presence, so every telemetry bit was 1 on every deploy (the acceptance golden showed all mutually-exclusive one-hot members set at once). Gate the telemetry walk on the visited bool value so a field maps to a bit only when true. Also drop blank lines in splitLines so a stray one can never shift bit indices in the append-only schema, and make the test stand-in tagless to match the real struct (the tagged stand-in masked the bug). Co-authored-by: Isaac
Rebase onto main picked up new config fields (resources.secrets.*) and removed the acceptance Local option, which now fails as an undecoded key. Co-authored-by: Isaac
Main added resources.pipelines.*.cascade_on_destroy, which shifts the schema and the encoded bitmaps. Co-authored-by: Isaac
Main moved the select/source-linked metrics into mark_select_used.go and apply_sync_mode.go, dropped the dry-run migration metrics, and added plan-error, recreate-planned, via-default, deployment-history, immutable-snapshot, and ai_runtime_task metrics. Mirror them into Telemetry, regenerate the (still unreleased) schema from scratch, and drop the engine matrix from the bitmap test now that direct is the only engine. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
… tests - Compute bits from bundle.UserConfig, a snapshot of the config taken at the start of Initialize (plus resources added by Python before their defaults are filled in), so the bitmap records what users configure rather than CLI defaults (workspace paths, current_user, job format/queue, ...). - Walk the dyn.Value instead of the typed struct: interface-typed fields (variables.*.default) and empty containers were never marked, and zero non-omitempty fields always were. - Replace True/False field pairs and SetPaired with a BoolPair type; per-artifact flags call Set per artifact, so mixed artifacts set both bits. - bundle bitmap now fails on an invalid bundle instead of printing a bitmap. - Goldens record only set bits; the schema and encoded blob are scratch files checked by the round trip, so an added config field touches only schema.txt. test-acc/test-update no longer regenerate the schema and goldens first. - Decode: overflow-safe size check; tests for header errors, byte boundaries, non-zero context; Python decoder checks the context; unique-schema test. - Acceptance coverage for interface defaults, empty containers, explicit zero, dotted map keys, target overrides, run_as, escapes, mixed artifacts, an invalid bundle, and a Python-defined resource. Co-authored-by: Isaac <no-reply@databricks.com>
Split acceptance/bundle/bitmap into presence/ (set bits, round trip, invalid bundle) and full/, which records the complete bitmap-text so a PR adding a config field shows where it landed. generate-bitmap-schema refreshes full/. Co-authored-by: Isaac <no-reply@databricks.com>
- bundle.Telemetry is now the single source of the deploy event's bool_values: fields are bool (reported when true), *bool (a measured value) or []bool (one value per item), keyed by a `metric` tag. Call sites set one field instead of a Metrics call plus a mirror. The wire has the same entries in field order; has_<task_type> and local.cache.* stay in Metrics.BoolValues. - The bitmap records each bool field's value: the schema has "=true"/"=false" entries after every bool field (config and telemetry). - bitmap-text and the new `bundle bitmap decode` (stdin) print one line per set field: "T"/"F" for a bool value, "1" otherwise. - acceptance/bundle/bitmap/full records the whole schema; presence covers bool variants (SDK omitempty false, non-omitempty true/false, *bool false, an unresolved reference) and pins the known gap: an explicit false on an omitempty bool of a CLI-defined struct is dropped by the loader. Co-authored-by: Isaac <no-reply@databricks.com>
databricks.bundles._bitmap decodes an encoded bundle bitmap without the CLI: it embeds the schema, raw-DEFLATE compressed and base64 encoded, and decompresses it on import. `python -m databricks.bundles._bitmap` reads a bitmap on stdin and prints the same lines as `bundle bitmap decode`. bundle/bitmap/genpython writes the embedded schema; compressing in Go keeps the generated file identical regardless of the Python's zlib build. It runs as part of `task generate-bitmap-schema`. The presence test and every no_drift invariant config check that the Python decoder's output matches the CLI's. The tests pass --reinstall-package because the wheel's version does not change with the code and uv would otherwise reuse a cached install. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
pydabs-codegen deletes every databricks/bundles package except core before regenerating, which removed _bitmap (and then ruff re-sorted the test's imports). Exclude it there, and skip private packages in the generated docs. Co-authored-by: Isaac <no-reply@databricks.com>
Configs with warnings (e.g. the deprecated pipeline development property) leaked them into output.txt once stdout and stderr were split. Log stderr and keep the panic check on it. Co-authored-by: Isaac <no-reply@databricks.com>
Runs over every invariant config without deploying: bundle validate (text and JSON) must succeed, and both the CLI and Python decoders must reproduce the bitmap-text bits from the encoded bitmap. Replaces the bitmap block in no_drift, which only compared the two decoders and ran on cloud too. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
bitmap-text is folded into "bundle bitmap bitmap -o text"; the default stays the compressed form. decode takes the same flag and defaults to text. The text form starts with a "size=N context=C" header, and -o json renders size, context and trees of the present fields and of the bool fields seen true and false. The Python decoder prints the same header. The presence test pins the JSON form and checks that decode reproduces it and that re-encoding is byte-identical; validate_bitmap round-trips JSON on every invariant config. Co-authored-by: Isaac <no-reply@databricks.com>
Like acceptance/bundle/refschema: resources.<type>.* goes to output/<type>.txt and everything else to output/root.txt. Each line carries its bit, so a file still shows where its fields sit in the schema. Co-authored-by: Isaac <no-reply@databricks.com>
…nother Co-authored-by: Isaac <no-reply@databricks.com>
app_lifecycle_started, cluster_lifecycle_started and sql_warehouse_lifecycle_started took the value of whichever resource came first in map order, so with one app started and another stopped the reported value changed from run to run. They now report true if any resource of the type sets started: true and false if any sets false, each value once; a bundle whose resources agree still sends a single entry. The fields become []bool, which changes their bitmap entries; the schema is not released yet, so it is regenerated from scratch. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
The per-artifact and per-resource flags were []bool fields grown with append. They are now BoolSet, which records the values a measurement took: call sites read b.Telemetry.ArtifactFilesIsSet.SetBool(x), bool_values gets each value taken once (true before false), and the bitmap gives the field a presence bit plus its =true/=false bits, the same entries as any bool. The lifecycle.started loops call SetBool per resource, which replaces the startedValues helper. Telemetry bits are now set directly instead of through a dyn conversion. The schema is not released yet, so it is regenerated. Co-authored-by: Isaac <no-reply@databricks.com>
Initialize now marks the loaded config into a bitmap.Builder kept on the
bundle (replacing the UserConfig snapshot), and the Python mutator marks the
resources it adds. Fields holding a pure ${var.*} reference are kept in a side
set; at the end of Initialize each reference is evaluated against the
variables' final values and its value is marked in place. So a bool set
through a variable records its value (strings from --var or BUNDLE_VAR_* are
coerced like the loader does), and a complex variable records the fields
inside it. Bits are only ever added: an unresolved reference keeps its
presence bit.
bundle.target and bundle.environment are pruned: target selection sets them
on every bundle. The schema is unreleased, so it is regenerated.
The bitmap tests that need bundle.Telemetry move to an external test package,
since bundle now imports bitmap.
Co-authored-by: Isaac <no-reply@databricks.com>
use_legacy_run_as is now a *bool, so read it with config.IsExplicitlyEnabled. An explicit false on such a field now survives loading, so the presence fixture records F experimental.use_legacy_run_as; the comment about the gap is updated. Co-authored-by: Isaac <no-reply@databricks.com>
denik
force-pushed
the
denik/bundle-bitmap
branch
from
November 9, 2026 15:53
6d41413 to
e070c71
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Today each tracked DABs feature needs custom code (a key constant plus a SetBoolValue call), so only hand-picked features are tracked. The bundle bitmap encodes the presence of every bundle config field as one bit, derived by reflecting over config.Root, so all current and future fields are covered for free (http://go/dabs-telemetry).
Stacked on #7412, which keeps an explicit
falseon optional config bools so the bitmap can record it.Changes
databricks bundle bitmapgroup:schema,bitmap -o compressed|text|json(compressed is DEFLATE + base64, the default),decode(stdin, same-o, defaults to text) andupdate-schema [--validate]. The schema lives in bundle/bitmap/schema.txt, is embedded into the binary, and is append-only so bit positions stay stable;task generateappends new fields. Bool fields also record their value.${var.*}(so a bool set through a variable records its value and a complex variable records the fields inside it).bundle.target/bundle.environmentare left out: target selection sets them on every bundle.bundle.Telemetrystruct (metrictags): one assignment per call site instead of a key constant plus SetBoolValue. Flags measured per artifact or per resource are aBoolSet(SetBool(v)) and report each value taken once:*_lifecycle_startedis true if any resource is started and false if any is stopped (it used to take a random resource's value), andartifact_*_is_setno longer repeats a value once per artifact. Otherwisebool_valueskeeps the same entries, now in field order. The bitmap is not sent yet.databricks.bundles._bitmapin databricks-bundles: a self-contained decoder with the schema embedded.Tests
Go and Python unit tests for presence, bool variants and the wire format. acceptance/bundle/bitmap/presence pins the set bits of a fixture and checks that the CLI (text, JSON) and Python decoders reproduce it; bitmap/full records the whole schema with each field's bit, split per resource type; python/telemetry covers a Python-defined resource; invariant/validate_bitmap (local-only) runs validate on every invariant config and checks that both decoders reproduce the bits from the encoded bitmap.
This pull request and its description were written by Isaac.