Skip to content

Add bundle bitmap for per-field telemetry #6104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
denik wants to merge 35 commits into from
Draft

Add bundle bitmap for per-field telemetry #6104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
2e8eda7
Add PLAN.md for bundle bitmap
denik Jul 29, 2026
b24805c
Add bundle bitmap for per-field telemetry
denik Jul 29, 2026
a41e82e
Remove PLAN.md
denik Jul 29, 2026
d8b292a
Add bool value bits, decode round-trip, and template recording
denik Jul 29, 2026
742fae8
Move boolean telemetry into a typed Telemetry struct
denik Jul 30, 2026
b2309e7
Regenerate goldens after rebase; ignore .databricks in bitmap test
denik Jul 30, 2026
fa17431
Fix bitmap test: executable decode script and deterministic cache
denik Jul 30, 2026
cf049f7
Strip CR when parsing embedded bitmap schema
denik Jul 30, 2026
39c3a3a
Regenerate bitmap schema from scratch; refresh goldens in generate task
denik Jul 30, 2026
1b149fa
Make Telemetry additive: mirror bool metrics into a struct on Bundle
denik Jul 31, 2026
0ae0095
Record DMS undeclared-writer flags as True/False pairs in bitmap
denik Jul 31, 2026
1424d6e
Revert DMS flags to single bits: false is not lost here
denik Aug 3, 2026
bc50888
Fix bitmap encoding every telemetry bit as set
denik Aug 3, 2026
d3de01e
Regenerate bitmap schema and drop removed Local test option
denik Aug 11, 2026
49ba379
Regenerate bitmap schema and goldens after rebase
denik Aug 17, 2026
51bcef3
Port bitmap telemetry mirror to current main
denik Nov 3, 2026
68b0ea4
Regenerate bitmap schema after rebase (data_quality_monitors)
denik Nov 3, 2026
38078f1
Bitmap: record user-authored config; fix presence detection; simplify…
denik Nov 4, 2026
1942f38
Add a dedicated bitmap test that records every schema line
denik Nov 4, 2026
0faf400
Make Telemetry the source of bool_values; encode bool values; add decode
denik Nov 4, 2026
34167dd
Add a self-contained bitmap decoder to databricks-bundles
denik Nov 4, 2026
8b2ccf3
Regenerate bitmap schema after rebase (variables.*.sensitive)
denik Nov 4, 2026
9e2815a
Regenerate bitmap schema after rebase (materialization_sets)
denik Nov 4, 2026
b23d01d
Keep the bitmap decoder out of pydabs codegen cleanup and docs
denik Nov 4, 2026
9e3dbc1
no_drift: log bitmap stderr instead of diffing it
denik Nov 4, 2026
6b68929
Add a local-only invariant test for validate and the bitmap round trip
denik Nov 4, 2026
8c832e1
Update selftest/selection for the new validate_bitmap invariant test
denik Nov 4, 2026
953759c
bitmap: one bitmap command with -o compressed|text|json
denik Nov 5, 2026
110837c
bitmap/full: split the schema golden per resource type
denik Nov 5, 2026
2865e36
bitmap/presence: lifecycle.started true on one cluster and false on a…
denik Nov 5, 2026
1235468
Report each lifecycle.started value seen instead of a random resource's
denik Nov 5, 2026
b87e62b
Regenerate bitmap schema after rebase (postgres_data_apis)
denik Nov 5, 2026
448c386
Telemetry: BoolSet.SetBool instead of appending to []bool
denik Nov 6, 2026
c00eb87
bitmap: compute bits early and fill in variable references later
denik Nov 9, 2026
e070c71
Adapt to optional config bools being *bool
denik Nov 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/rules/telemetry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ The deploy payload is declared twice: the proto in the universe repo (`proto/log

Other things worth knowing:

- A one-off boolean needs no proto change: add a key constant to `bundle/metrics/metrics.go` (snake_case, matching the constant name) and call `b.Metrics.SetBoolValue(...)`. It lands in `experimental.bool_values`.
- A one-off boolean needs no proto change: add a field to `bundle.Telemetry` (`bundle/telemetry.go`) with a snake_case `metric:"..."` tag and set it. Pick the type by what is measured: `bool` for a flag reported only when true, `*bool` for a measured value (nil when not measured), `BoolSet` for a measurement taken several times (e.g. once per artifact; record each with `SetBool`, and each value taken is reported once). It lands in `experimental.bool_values` and in the bundle bitmap; run `./task generate-bitmap-schema` to append it to the bitmap schema.
- Metrics that may be removed later belong in `BundleDeployExperimental`, which carries no compatibility guarantees.
- Positional encodings such as the `upload_file_sizes` histogram are frozen once the metric has adoption: changing a bucket bound silently changes the meaning of every previously reported entry.
- Per-resource state sizes come from the direct engine's state in `resources.json`.
Expand Down
29 changes: 29 additions & 0 deletions Taskfile.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -802,6 +802,7 @@ tasks:
# Refreshes acceptance/bundle/refschema/output/*.txt, which feeds
# generate-direct-apitypes and generate-direct-resources below.
- task: generate-refschema
- task: generate-bitmap-schema
- task: generate-schema
- task: generate-validation
- task: generate-direct
Expand All @@ -820,6 +821,7 @@ tasks:
cmds:
- task: generate-cligen
- task: generate-refschema
- task: generate-bitmap-schema
- task: generate-schema
- task: generate-validation
- task: generate-direct
Expand Down Expand Up @@ -989,6 +991,33 @@ tasks:
cmds:
- go test ./acceptance -run TestAccept/bundle/refschema -update

# Appends any newly added bundle config fields to bundle/bitmap/schema.txt, the
# embedded bitmap schema (see cmd/bundle/bitmap). The schema is append-only, so
# update-schema reads the currently embedded schema and only appends. Output is
# written to a temp file first so the redirect does not truncate schema.txt
# before `go run` embeds it at compile time. Also refreshes the schema embedded
# in the databricks-bundles decoder (python/databricks/bundles/_bitmap) and the
# listing in acceptance/bundle/bitmap/full, which tracks every schema line.
generate-bitmap-schema:
desc: Regenerate bundle/bitmap/schema.txt and its acceptance listing
sources:
- bundle/**/*.go
- exclude: bundle/**/*_test.go
- acceptance/bundle/bitmap/full/script
- acceptance/bundle/bitmap/full/split.py
- go.mod
- go.sum
- "{{.EMBED_SOURCES}}"
generates:
- bundle/bitmap/schema.txt
- acceptance/bundle/bitmap/full/output/*.txt
- python/databricks/bundles/_bitmap/_schema.py
cmds:
- go run . bundle bitmap update-schema > bundle/bitmap/schema.txt.tmp
- mv bundle/bitmap/schema.txt.tmp bundle/bitmap/schema.txt
- go run ./bundle/bitmap/genpython python/databricks/bundles/_bitmap/_schema.py
- go test ./acceptance -run TestAccept/bundle/bitmap/full -update

# Upstream field documentation comes from the checked-in .codegen/cli.json;
# bundle/internal/schema/annotations.yml carries the CLI-owned docs and
# overrides and is rewritten in place (synced with the config structure).
Expand Down
2 changes: 2 additions & 0 deletions acceptance/bundle/bitmap/full/out.test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Cloud = false
EnvMatrix.DMS = []
Empty file added acceptance/bundle/bitmap/full/output.txt
View file
Open in desktop
Empty file.
75 changes: 75 additions & 0 deletions acceptance/bundle/bitmap/full/output/alerts.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
resources.alerts 4169
resources.alerts.* 4170
resources.alerts.*.url 4171
resources.alerts.*.modified_status 4172
resources.alerts.*.lifecycle 4173
resources.alerts.*.lifecycle.prevent_destroy 4174
resources.alerts.*.lifecycle.prevent_destroy=true 4175
resources.alerts.*.lifecycle.prevent_destroy=false 4176
resources.alerts.*.lifecycle.depends_on 4177
resources.alerts.*.lifecycle.depends_on[*] 4178
resources.alerts.*.lifecycle.depends_on[*].resource 4179
resources.alerts.*.create_time 4180
resources.alerts.*.custom_description 4181
resources.alerts.*.custom_summary 4182
resources.alerts.*.custom_template_format 4183
resources.alerts.*.display_name 4184
resources.alerts.*.effective_run_as 4185
resources.alerts.*.effective_run_as.service_principal_name 4186
resources.alerts.*.effective_run_as.user_name 4187
resources.alerts.*.evaluation 4188
resources.alerts.*.evaluation.comparison_operator 4189
resources.alerts.*.evaluation.empty_result_state 4190
resources.alerts.*.evaluation.last_evaluated_at 4191
resources.alerts.*.evaluation.notification 4192
resources.alerts.*.evaluation.notification.notify_on_ok 4193
resources.alerts.*.evaluation.notification.notify_on_ok=true 4194
resources.alerts.*.evaluation.notification.notify_on_ok=false 4195
resources.alerts.*.evaluation.notification.retrigger_seconds 4196
resources.alerts.*.evaluation.notification.subscriptions 4197
resources.alerts.*.evaluation.notification.subscriptions[*] 4198
resources.alerts.*.evaluation.notification.subscriptions[*].destination_id 4199
resources.alerts.*.evaluation.notification.subscriptions[*].user_email 4200
resources.alerts.*.evaluation.source 4201
resources.alerts.*.evaluation.source.aggregation 4202
resources.alerts.*.evaluation.source.display 4203
resources.alerts.*.evaluation.source.name 4204
resources.alerts.*.evaluation.state 4205
resources.alerts.*.evaluation.threshold 4206
resources.alerts.*.evaluation.threshold.column 4207
resources.alerts.*.evaluation.threshold.column.aggregation 4208
resources.alerts.*.evaluation.threshold.column.display 4209
resources.alerts.*.evaluation.threshold.column.name 4210
resources.alerts.*.evaluation.threshold.value 4211
resources.alerts.*.evaluation.threshold.value.bool_value 4212
resources.alerts.*.evaluation.threshold.value.bool_value=true 4213
resources.alerts.*.evaluation.threshold.value.bool_value=false 4214
resources.alerts.*.evaluation.threshold.value.double_value 4215
resources.alerts.*.evaluation.threshold.value.string_value 4216
resources.alerts.*.id 4217
resources.alerts.*.lifecycle_state 4218
resources.alerts.*.owner_user_name 4219
resources.alerts.*.parameters 4220
resources.alerts.*.parameters[*] 4221
resources.alerts.*.parameters[*].name 4222
resources.alerts.*.parameters[*].type 4223
resources.alerts.*.parameters[*].value 4224
resources.alerts.*.parent_path 4225
resources.alerts.*.query_text 4226
resources.alerts.*.run_as 4227
resources.alerts.*.run_as.service_principal_name 4228
resources.alerts.*.run_as.user_name 4229
resources.alerts.*.run_as_user_name 4230
resources.alerts.*.schedule 4231
resources.alerts.*.schedule.pause_status 4232
resources.alerts.*.schedule.quartz_cron_schedule 4233
resources.alerts.*.schedule.timezone_id 4234
resources.alerts.*.update_time 4235
resources.alerts.*.warehouse_id 4236
resources.alerts.*.permissions 4237
resources.alerts.*.permissions[*] 4238
resources.alerts.*.permissions[*].level 4239
resources.alerts.*.permissions[*].user_name 4240
resources.alerts.*.permissions[*].service_principal_name 4241
resources.alerts.*.permissions[*].group_name 4242
resources.alerts.*.file_path 4243
215 changes: 215 additions & 0 deletions acceptance/bundle/bitmap/full/output/apps.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
resources.apps 3931
resources.apps.* 3932
resources.apps.*.url 3933
resources.apps.*.modified_status 3934
resources.apps.*.lifecycle 3935
resources.apps.*.lifecycle.prevent_destroy 3936
resources.apps.*.lifecycle.prevent_destroy=true 3937
resources.apps.*.lifecycle.prevent_destroy=false 3938
resources.apps.*.lifecycle.depends_on 3939
resources.apps.*.lifecycle.depends_on[*] 3940
resources.apps.*.lifecycle.depends_on[*].resource 3941
resources.apps.*.active_deployment 3942
resources.apps.*.active_deployment.command 3943
resources.apps.*.active_deployment.command[*] 3944
resources.apps.*.active_deployment.create_time 3945
resources.apps.*.active_deployment.creator 3946
resources.apps.*.active_deployment.deployment_artifacts 3947
resources.apps.*.active_deployment.deployment_artifacts.source_code_path 3948
resources.apps.*.active_deployment.deployment_id 3949
resources.apps.*.active_deployment.env_vars 3950
resources.apps.*.active_deployment.env_vars[*] 3951
resources.apps.*.active_deployment.env_vars[*].name 3952
resources.apps.*.active_deployment.env_vars[*].value 3953
resources.apps.*.active_deployment.env_vars[*].value_from 3954
resources.apps.*.active_deployment.git_source 3955
resources.apps.*.active_deployment.git_source.branch 3956
resources.apps.*.active_deployment.git_source.commit 3957
resources.apps.*.active_deployment.git_source.git_repository 3958
resources.apps.*.active_deployment.git_source.git_repository.auto_deploy 3959
resources.apps.*.active_deployment.git_source.git_repository.auto_deploy=true 3960
resources.apps.*.active_deployment.git_source.git_repository.auto_deploy=false 3961
resources.apps.*.active_deployment.git_source.git_repository.caller_credential_id 3962
resources.apps.*.active_deployment.git_source.git_repository.provider 3963
resources.apps.*.active_deployment.git_source.git_repository.url 3964
resources.apps.*.active_deployment.git_source.resolved_commit 3965
resources.apps.*.active_deployment.git_source.source_code_path 3966
resources.apps.*.active_deployment.git_source.tag 3967
resources.apps.*.active_deployment.health_check 3968
resources.apps.*.active_deployment.health_check.path 3969
resources.apps.*.active_deployment.health_check.timeout 3970
resources.apps.*.active_deployment.mode 3971
resources.apps.*.active_deployment.source_code_path 3972
resources.apps.*.active_deployment.status 3973
resources.apps.*.active_deployment.status.message 3974
resources.apps.*.active_deployment.status.state 3975
resources.apps.*.active_deployment.update_time 3976
resources.apps.*.app_status 3977
resources.apps.*.app_status.message 3978
resources.apps.*.app_status.running_instances 3979
resources.apps.*.app_status.state 3980
resources.apps.*.budget_policy_id 3981
resources.apps.*.compute_max_instances 3982
resources.apps.*.compute_min_instances 3983
resources.apps.*.compute_size 3984
resources.apps.*.compute_status 3985
resources.apps.*.compute_status.active_instances 3986
resources.apps.*.compute_status.message 3987
resources.apps.*.compute_status.state 3988
resources.apps.*.create_time 3989
resources.apps.*.creator 3990
resources.apps.*.default_git_source 3991
resources.apps.*.default_git_source.branch 3992
resources.apps.*.default_git_source.commit 3993
resources.apps.*.default_git_source.git_repository 3994
resources.apps.*.default_git_source.git_repository.auto_deploy 3995
resources.apps.*.default_git_source.git_repository.auto_deploy=true 3996
resources.apps.*.default_git_source.git_repository.auto_deploy=false 3997
resources.apps.*.default_git_source.git_repository.caller_credential_id 3998
resources.apps.*.default_git_source.git_repository.provider 3999
resources.apps.*.default_git_source.git_repository.url 4000
resources.apps.*.default_git_source.resolved_commit 4001
resources.apps.*.default_git_source.source_code_path 4002
resources.apps.*.default_git_source.tag 4003
resources.apps.*.default_source_code_path 4004
resources.apps.*.description 4005
resources.apps.*.effective_budget_policy_id 4006
resources.apps.*.effective_usage_policy_id 4007
resources.apps.*.effective_user_api_scopes 4008
resources.apps.*.effective_user_api_scopes[*] 4009
resources.apps.*.forward_user_access_token 4010
resources.apps.*.forward_user_access_token=true 4011
resources.apps.*.forward_user_access_token=false 4012
resources.apps.*.git_repository 4013
resources.apps.*.git_repository.auto_deploy 4014
resources.apps.*.git_repository.auto_deploy=true 4015
resources.apps.*.git_repository.auto_deploy=false 4016
resources.apps.*.git_repository.caller_credential_id 4017
resources.apps.*.git_repository.provider 4018
resources.apps.*.git_repository.url 4019
resources.apps.*.git_source 4020
resources.apps.*.git_source.branch 4021
resources.apps.*.git_source.commit 4022
resources.apps.*.git_source.git_repository 4023
resources.apps.*.git_source.git_repository.auto_deploy 4024
resources.apps.*.git_source.git_repository.auto_deploy=true 4025
resources.apps.*.git_source.git_repository.auto_deploy=false 4026
resources.apps.*.git_source.git_repository.caller_credential_id 4027
resources.apps.*.git_source.git_repository.provider 4028
resources.apps.*.git_source.git_repository.url 4029
resources.apps.*.git_source.resolved_commit 4030
resources.apps.*.git_source.source_code_path 4031
resources.apps.*.git_source.tag 4032
resources.apps.*.id 4033
resources.apps.*.name 4034
resources.apps.*.oauth2_app_client_id 4035
resources.apps.*.oauth2_app_integration_id 4036
resources.apps.*.pending_deployment 4037
resources.apps.*.pending_deployment.command 4038
resources.apps.*.pending_deployment.command[*] 4039
resources.apps.*.pending_deployment.create_time 4040
resources.apps.*.pending_deployment.creator 4041
resources.apps.*.pending_deployment.deployment_artifacts 4042
resources.apps.*.pending_deployment.deployment_artifacts.source_code_path 4043
resources.apps.*.pending_deployment.deployment_id 4044
resources.apps.*.pending_deployment.env_vars 4045
resources.apps.*.pending_deployment.env_vars[*] 4046
resources.apps.*.pending_deployment.env_vars[*].name 4047
resources.apps.*.pending_deployment.env_vars[*].value 4048
resources.apps.*.pending_deployment.env_vars[*].value_from 4049
resources.apps.*.pending_deployment.git_source 4050
resources.apps.*.pending_deployment.git_source.branch 4051
resources.apps.*.pending_deployment.git_source.commit 4052
resources.apps.*.pending_deployment.git_source.git_repository 4053
resources.apps.*.pending_deployment.git_source.git_repository.auto_deploy 4054
resources.apps.*.pending_deployment.git_source.git_repository.auto_deploy=true 4055
resources.apps.*.pending_deployment.git_source.git_repository.auto_deploy=false 4056
resources.apps.*.pending_deployment.git_source.git_repository.caller_credential_id 4057
resources.apps.*.pending_deployment.git_source.git_repository.provider 4058
resources.apps.*.pending_deployment.git_source.git_repository.url 4059
resources.apps.*.pending_deployment.git_source.resolved_commit 4060
resources.apps.*.pending_deployment.git_source.source_code_path 4061
resources.apps.*.pending_deployment.git_source.tag 4062
resources.apps.*.pending_deployment.health_check 4063
resources.apps.*.pending_deployment.health_check.path 4064
resources.apps.*.pending_deployment.health_check.timeout 4065
resources.apps.*.pending_deployment.mode 4066
resources.apps.*.pending_deployment.source_code_path 4067
resources.apps.*.pending_deployment.status 4068
resources.apps.*.pending_deployment.status.message 4069
resources.apps.*.pending_deployment.status.state 4070
resources.apps.*.pending_deployment.update_time 4071
resources.apps.*.resources 4072
resources.apps.*.resources[*] 4073
resources.apps.*.resources[*].app 4074
resources.apps.*.resources[*].app.name 4075
resources.apps.*.resources[*].app.permission 4076
resources.apps.*.resources[*].database 4077
resources.apps.*.resources[*].database.database_name 4078
resources.apps.*.resources[*].database.instance_name 4079
resources.apps.*.resources[*].database.permission 4080
resources.apps.*.resources[*].description 4081
resources.apps.*.resources[*].experiment 4082
resources.apps.*.resources[*].experiment.experiment_id 4083
resources.apps.*.resources[*].experiment.permission 4084
resources.apps.*.resources[*].genie_space 4085
resources.apps.*.resources[*].genie_space.name 4086
resources.apps.*.resources[*].genie_space.permission 4087
resources.apps.*.resources[*].genie_space.space_id 4088
resources.apps.*.resources[*].job 4089
resources.apps.*.resources[*].job.id 4090
resources.apps.*.resources[*].job.permission 4091
resources.apps.*.resources[*].name 4092
resources.apps.*.resources[*].postgres 4093
resources.apps.*.resources[*].postgres.branch 4094
resources.apps.*.resources[*].postgres.database 4095
resources.apps.*.resources[*].postgres.permission 4096
resources.apps.*.resources[*].secret 4097
resources.apps.*.resources[*].secret.key 4098
resources.apps.*.resources[*].secret.permission 4099
resources.apps.*.resources[*].secret.scope 4100
resources.apps.*.resources[*].serving_endpoint 4101
resources.apps.*.resources[*].serving_endpoint.name 4102
resources.apps.*.resources[*].serving_endpoint.permission 4103
resources.apps.*.resources[*].sql_warehouse 4104
resources.apps.*.resources[*].sql_warehouse.id 4105
resources.apps.*.resources[*].sql_warehouse.permission 4106
resources.apps.*.resources[*].uc_securable 4107
resources.apps.*.resources[*].uc_securable.permission 4108
resources.apps.*.resources[*].uc_securable.securable_full_name 4109
resources.apps.*.resources[*].uc_securable.securable_kind 4110
resources.apps.*.resources[*].uc_securable.securable_type 4111
resources.apps.*.service_principal_client_id 4112
resources.apps.*.service_principal_id 4113
resources.apps.*.service_principal_name 4114
resources.apps.*.source_code_path 4115
resources.apps.*.space 4116
resources.apps.*.telemetry_export_destinations 4117
resources.apps.*.telemetry_export_destinations[*] 4118
resources.apps.*.telemetry_export_destinations[*].unity_catalog 4119
resources.apps.*.telemetry_export_destinations[*].unity_catalog.logs_table 4120
resources.apps.*.telemetry_export_destinations[*].unity_catalog.metrics_table 4121
resources.apps.*.telemetry_export_destinations[*].unity_catalog.traces_table 4122
resources.apps.*.thumbnail_url 4123
resources.apps.*.update_time 4124
resources.apps.*.updater 4125
resources.apps.*.usage_policy_id 4126
resources.apps.*.user_api_scopes 4127
resources.apps.*.user_api_scopes[*] 4128
resources.apps.*.lifecycle.started 4129
resources.apps.*.lifecycle.started=true 4130
resources.apps.*.lifecycle.started=false 4131
resources.apps.*.config 4132
resources.apps.*.config.command 4133
resources.apps.*.config.command[*] 4134
resources.apps.*.config.env 4135
resources.apps.*.config.env[*] 4136
resources.apps.*.config.env[*].name 4137
resources.apps.*.config.env[*].value 4138
resources.apps.*.config.env[*].value_from 4139
resources.apps.*.permissions 4140
resources.apps.*.permissions[*] 4141
resources.apps.*.permissions[*].level 4142
resources.apps.*.permissions[*].user_name 4143
resources.apps.*.permissions[*].service_principal_name 4144
resources.apps.*.permissions[*].group_name 4145
Loading
Loading