feat: on arm64 (Grace-Blackwell GB200/GB300) skip fabric manager and install nvidia-imex#170
Open
xuexu6666 wants to merge 2 commits into
Open
feat: on arm64 (Grace-Blackwell GB200/GB300) skip fabric manager and install nvidia-imex#170xuexu6666 wants to merge 2 commits into
xuexu6666 wants to merge 2 commits into
Conversation
arm64 cuda nodes are the Grace-Blackwell SKUs (GB200/GB300), which have no node-local fabric manager -- the NVLink fabric is managed out-of-band via IMEX. The driver container should neither install nor bundle fabricmanager there. Today device_init() runs the fabricmanager package installer for every cuda node. On arm64 it computes NVIDIA_FM_ARCH from $ARCH (uname -m = "aarch64") and then only remaps to "sbsa" when the value equals "arm64" -- which never matches -- so it looks for /opt/gpu/fabricmanager-linux-aarch64-<ver>/... which does not exist (the tarball download.sh bundles is named "sbsa"). The installer exits 127; installGPUDriverImage retries the whole container until CSE hits its ~780s timeout and exits 84, so ensureKubelet never runs and the node never joins the cluster. Surfaced by the GB300 managed-driver (--gpu-driver Install) e2e. Fix: gate fabric manager to x86_64 only. - install.sh: run fm_run_package_installer.sh only when ARCH == x86_64; drop the dead arm64->sbsa remap. - download.sh: download/bundle the fabricmanager tarball only when TARGETARCH == amd64, so the arm64 image no longer ships an unused tarball.
xuexu6666
force-pushed
the
xuxue/skip-fabric-manager-arm64-gb
branch
from
July 17, 2026 02:38
9aa3972 to
0865ff8
Compare
xuexu6666
changed the title
GB uses IMEX -- not a node-local fabric manager -- to coordinate cross-node NVLink (MNNVL / ComputeDomains). The nvidia-imex daemon ships only as a separate deb in the CUDA repo (not in the driver .run or the fabric-manager redist), so: - download.sh bundles the version-matched nvidia-imex arm64 deb into the image, resolving the exact filename from the repo Packages index (the revision suffix varies by version). Skipped on x86 (which uses fabric manager instead). - install.sh installs it at node boot via dpkg -i --force-depends (the deb declares nvidia-modprobe, provided by the runfile driver rather than a deb), and leaves the node-wide nvidia-imex.service OFF -- IMEX is orchestrated per ComputeDomain by the NVIDIA DRA driver, not run cluster-wide from the host. The host binary must be present so the DRA driver can CDI-inject it into its per-workload IMEX daemon pods. Pairs with the fabric-manager arm64 skip: on GB, skip FM and install IMEX in the same code path where x86 installs FM.
xuexu6666
force-pushed
the
xuxue/skip-fabric-manager-arm64-gb
branch
from
July 18, 2026 01:03
ff3834c to
f3a3752
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On arm64 Grace-Blackwell (GB200/GB300), cross-node NVLink is coordinated by IMEX, not a node-local fabric manager. This PR makes the aks-gpu image do the right thing on GB, in the same code path where x86 installs fabric manager:
1. Skip fabric manager on arm64
The FM installer looked for
fabricmanager-linux-aarch64-…(fromuname -m), which doesn't exist (the redist archive issbsa) →exit 127→ CSE retries the whole container → ~780s timeout →exit 84→ensureKubeletnever runs → the GB node never joins the cluster. GB has no node-local fabric manager, so FM is now skipped on arm64 (download + install).2. Install nvidia-imex on arm64
nvidia-imex(the cross-node NVLink / ComputeDomains coordinator) is not in the driver.runor the fabric-manager redist — it ships only as a separate deb in the CUDA repo. So:download.shbundles the version-matchednvidia-imexarm64 deb into the image, resolving the exact filename from the repoPackagesindex (the deb revision suffix, e.g.-1ubuntu1, varies by version), OS path adapted viaVERSION_ID.install.shinstalls it at node boot viadpkg -i --force-depends(the deb declaresnvidia-modprobe, which the runfile driver provides on-disk but not as a deb) and leaves the node-widenvidia-imex.serviceOFF — IMEX is orchestrated per-ComputeDomain by the NVIDIA DRA driver, which CDI-injects the host binary into its daemon pods.Validation
Proven end-to-end on a managed-driver GB300 cluster: with
nvidia-imexinstalled, the DRA ComputeDomain reaches Ready and cross-node MNNVL NCCL all_reduce runsvia P2P/MNNVLat ~698 GB/s (peak). x86 behavior is unchanged.