Package AI Runtime code_source via a tgz artifact, not a sync overlay - #6494
Package AI Runtime code_source via a tgz artifact, not a sync overlay#6494vinchenzo-db wants to merge 2 commits into
Conversation
A local-directory code_source_path is now turned into a `tgz` artifact and built and uploaded through the standard artifact path, instead of the aicode mutator splicing a content-addressed tarball onto the sync root. The tgz artifact is the single packing mechanism. - aicode.PackageCodeSource now synthesizes a `tgz` artifact per local-dir code_source_path (path = the dir's parent, include = its basename, so archive entries nest under the basename for the /databricks/code_source/<dir> layout) and rewrites code_source_path to the built tarball. It runs before artifacts.Prepare (initialize) instead of in the build phase. - Remove the sync-root overlay, the content-addressed packer (buildCodeSnapshot), bundle.HasAiRuntimeCodeSnapshot, bundle.AiCodeSnapshotDir, and the validateSnapshotDir guards that only existed for the overlay. - Behavior change: only .gitignore filters the packaged files now; the bundle-wide sync.include/exclude no longer apply to a code artifact. - Keep the git_source / immutable-folder / source-linked / for_each rejections. Co-authored-by: Isaac <no-reply@databricks.com>
Approval status: pending
|
ben-hansen-db
left a comment
ben-hansen-db
left a comment
There was a problem hiding this comment.
I think PR description could be a bit more clear. We are basically reverting the other PR right? That's the the majority of code changes are?
| }), nil | ||
| // artifactKey is a stable, unique artifact name for a code directory (relative to the | ||
| // bundle). Two tasks pointing at the same directory collapse to one artifact. | ||
| func artifactKey(relDir string) string { |
There was a problem hiding this comment.
from claude:
Issue: artifactKey maps every non-alphanumeric to _, so ./a/b and ./a_b both produce air_code_source_a_b. artifacts[key] (:85) + maps.Copy (:118) collapse them to one tarball; both tasks' code_source_path point at it, so one task silently ships the other's code.
Fix: Add a hash disambiguator to the key, or error on a collision across distinct relDirs
Yeah that's the idea + using new code uploader + keeping some of the old guardrails |
artifactKey sanitizes non-alphanumerics to '_', so distinct code_source
directories ("a/b" and "a_b") could collide on one key and collapse into a
single tarball — silently shipping one task's code for another. Detect the
collision across distinct directories and error instead.
Co-authored-by: Isaac <no-reply@databricks.com>
2 participants
#6110 <- basically reverting this pr
A local-directory code_source_path is now turned into a
tgzartifact and built and uploaded through the standard artifact path, instead of the aicode mutator splicing a content-addressed tarball onto the sync root. The tgz artifact is the single packing mechanism.tgzartifact per local-dir code_source_path (path = the dir's parent, include = its basename, so archive entries nest under the basename for the /databricks/code_source/ layout) and rewrites code_source_path to the built tarball. It runs before artifacts.Prepare (initialize) instead of in the build phase.Why
#6428 We recently merged a DABs native uploader, which supercedes this mutator workaround
Tests
Unit tests
E2E test:
Setup:
Show deploy works:
Investigate what is in tar:
bundle summary:
bundle run:
Run details:
