fix(gpu): tear down prebaked GPU driver on version mismatch, not just kind mismatch#8945
Open
ganeshkumarashok wants to merge 1 commit into
Open
fix(gpu): tear down prebaked GPU driver on version mismatch, not just kind mismatch#8945ganeshkumarashok wants to merge 1 commit into
ganeshkumarashok wants to merge 1 commit into
Conversation
… kind mismatch Generalizes the managed-GPU prebake teardown from grid-only kind mismatch (cleanUpGridNodeCudaPrebake, #8919) to any prebake/node driver mismatch (cleanUpMismatchedPrebakedGPUDriver): 1. driver-KIND mismatch (cuda prebake on a GRID node) -- unchanged behavior. 2. driver-VERSION mismatch, same kind -- NEW. If the baked cuda(-lts) prebake's driver version differs from the version this node installs (GPU_DV), the stale prebaked module + /usr/bin/lib64 libs would collide with the fresh install -> "Failed to initialize NVML: Driver/library version mismatch". Now torn down before install. The dkms-marker previously recorded only driver_kind=, so CSE could not detect same-kind version drift (the gap #8915 flagged and deferred). This adds driver_version= to the marker at VHD-build time (install-dependencies.sh, from NVIDIA_DRIVER_IMAGE_TAG with the image-SHA suffix stripped to match CSE's GPU_DV) and compares it at node boot. Backward compatible (VHDs live ~6 months): markers written before this change have no driver_version= line; when absent, CSE cannot compare versions and falls back to kind-only behavior -- identical to the prior cleanUpGridNodeCudaPrebake. No teardown is ever triggered by a missing version field. Adds normalizeGPUDriverKind helper (cuda-lts->cuda, grid-v20->grid) and shellspec coverage for kind mismatch, version mismatch, version match no-op, missing-version fallback, unknown-GPU_DV no-op, and the existing kind/OS/marker-absence no-ops. make shellspec: 142 examples, 0 failures. make generate: no testdata drift (cse_config.sh is VHD-build-uploaded, not embedded in CSE-gen snapshots).
ganeshkumarashok
requested review from
AbelHu,
Devinwong,
SriHarsha001,
awesomenix,
calvin197,
cameronmeissner,
djsly,
karenychen,
lilypan26,
mxj220,
pdamianov-dev,
phealy,
r2k1,
runzhen,
sulixu,
surajssd,
timmy-wright,
titilambert,
xuexu6666 and
zachary-bailey
as code owners
July 15, 2026 00:10
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens the managed GPU driver install path on Ubuntu by tearing down a pre-baked NVIDIA driver when it doesn’t match what the node is about to install, extending the prior GRID-vs-CUDA mismatch handling to also cover same-kind version drift.
Changes:
- Add
driver_version=to the VHD prebake DKMS marker so node-boot CSE can detect same-kind version mismatches. - Replace the GRID-only teardown logic with
cleanUpMismatchedPrebakedGPUDriver, which tears down on driver kind mismatch or (when both sides are known) driver version mismatch. - Extend ShellSpec coverage for kind mismatch, version mismatch, and backward-compatible fallback behaviors.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
vhdbuilder/packer/install-dependencies.sh |
Appends the prebaked driver version to the DKMS marker during CUDA prebake VHD build. |
parts/linux/cloud-init/artifacts/cse_config.sh |
Introduces cleanUpMismatchedPrebakedGPUDriver + kind normalization and wires it into ensureGPUDrivers before managed install. |
spec/parts/linux/cloud-init/artifacts/cse_config_spec.sh |
Adds ShellSpec cases covering kind mismatch, version mismatch, and missing-version fallback behavior. |
Comment on lines
+1368
to
1377
| if [ -n "${node_kind}" ] && [ "${m_kind}" != "${node_kind}" ]; then | ||
| reason=driver_kind_mismatch | ||
| elif [ -n "${m_version}" ] && [ -n "${GPU_DV:-}" ] && [ "${m_version}" != "${GPU_DV}" ]; then | ||
| # Same kind but the baked driver version differs from what this node installs (GPU_DV is the | ||
| # AgentBaker-requested driver version, cse_helpers.sh). Only compared when both are known. | ||
| reason=driver_version_mismatch | ||
| else | ||
| # Match (or not enough info to declare a mismatch): keep the prebake for the skip-build path. | ||
| return 0 | ||
| fi |
Contributor
|
AgentBaker Linux gate detective RCA for failed run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=172262629
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Generalizes the managed-GPU VHD-prebake teardown from grid-only driver-kind mismatch (
cleanUpGridNodeCudaPrebake, #8919) to any prebake/node driver mismatch (cleanUpMismatchedPrebakedGPUDriver), closing the CUDA-vs-CUDA version-drift gap that #8915 explicitly deferred ("CUDA-vs-CUDA version drift still needs separate marker/version comparison work").Two mismatch classes are now handled:
/usr/bin/lib64userspace libs collide with the fresh install →Failed to initialize NVML: Driver/library version mismatch. Now torn down before install.Why the version case needed marker changes
The
dkms-markerpreviously recorded onlydriver_kind=(cuda|grid), so CSE could see family mismatches but not same-family version drift. This PR:install-dependencies.sh): appendsdriver_version=to the marker, derived fromNVIDIA_DRIVER_IMAGE_TAGwith the trailing image-SHA stripped so it matches CSE'sGPU_DV(GPUDriverVersion). Appended after the container-writtendriver_kind=line, not overwriting it.cse_config.sh):cleanUpMismatchedPrebakedGPUDrivercomparesdriver_version=againstGPU_DVand tears down on mismatch.Backward compatibility (VHDs live ~6 months)
Markers written before this change have no
driver_version=line. When the marker version is absent (orGPU_DVis unset), CSE cannot compare versions and falls back to kind-only behavior — byte-for-byte identical to the priorcleanUpGridNodeCudaPrebake. No teardown is ever triggered by a missing version field, so old VHDs booting under a new CSE behave exactly as today. New teardown only fires when both sides are known and differ.Scope note (honesty on exposure)
The concrete same-kind drift case in today's fleet (NCv1/K80 R470 vs baked cuda-lts R580) has zero AKS exposure — NCv1 is retired and runs on no AKS Ubuntu VHD (verified against fleet telemetry). This PR is forward-looking hardening: the moment a second cuda driver version reaches the shared
aks-gpu-cuda-ltsprebake image (e.g. a driver bump under VHD/CSE version skew where the baked version lags the requested one), same-kind drift becomes real, and the kind-only marker could not detect it. The change is safe today (kind behavior preserved) and correct when that day comes.Tests
make shellspec(docker/bash): 142 examples, 0 failures. New coverage: kind mismatch, grid-v20→grid, legacy-marker(no kind), version mismatch, version match no-op, missing-version fallback, unknown-GPU_DVno-op, kind-match no-op, grid-match no-op, marker-absent no-op, non-Ubuntu no-op; plusnormalizeGPUDriverKindunit cases.make generate: no testdata drift (cse_config.shis VHD-build-uploaded, not embedded in the CSE-gen snapshots). (Note: fullmake generatealso tripsvalidate-shellon pre-existing SC3014 warnings in untouched filesinit-aks-custom-cloud.sh/disk_queue.shunder local shellcheck 0.11.0 — reproduces on pristinemain, unrelated to this PR.)Risk
🟡 Low-Medium. Managed-GPU install path only (never runs on
--gpu-driver None/skip nodes). The teardown decision is strictly more precise than before, guarded so a missing/unknown version can never cause a false teardown.install-dependencies.shchange runs only underNVIDIA_CUDA_PREBAKE(the shared 2204/2404 gen2 build).Related: #8919 (kind teardown, on main), #8915 (closed; deferred this version work), #8933 (rmmod of resident module — complementary).