fix(core): make cuda.core.system actually fall back when NVML is unimportable - #2574
Open
LeSingh1 wants to merge 1 commit into
Open
fix(core): make cuda.core.system actually fall back when NVML is unimportable#2574LeSingh1 wants to merge 1 commit into
LeSingh1 wants to merge 1 commit into
Conversation
…portable
The file's own header states the contract: use NVML exclusively, or when
`cuda.bindings.nvml` is not available fall back to non-NVML methods. The code
does neither:
if CUDA_BINDINGS_NVML_IS_COMPATIBLE:
try:
from cuda.bindings import nvml
except ImportError:
CUDA_BINDINGS_NVML_IS_COMPATIBLE = False
from cuda.core.system._nvml_context import initialize
else:
from cuda.core._utils.cuda_utils import driver, handle_return, runtime
The `except` clears the flag, but the `else` belongs to the outer `if`,
which has already been evaluated -- so the fallback names are never bound on
the path that clears the flag. Every consumer keys off the now-False flag and
reaches for exactly those names:
get_user_mode_driver_version() -> handle_return(driver.cuDriverGetVersion())
get_num_devices() -> handle_return(runtime.cudaGetDeviceCount())
both of which would raise `NameError`. In practice the module never gets that
far: `_nvml_context` is imported unconditionally on the next line and
`_nvml_context.pyx` starts with `from cuda.bindings import nvml`, so the
ImportError the try/except exists to absorb is re-raised out of
`import cuda.core.system` a statement later.
Move the `_nvml_context` import inside the same `try` and turn the `else`
into a second `if not ...` so a cleared flag selects the fallback.
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
cuda_core/cuda/core/system/_system.pyxopens with the contract it is meant to satisfy:The code that is supposed to implement that does neither:
The
exceptclearsCUDA_BINDINGS_NVML_IS_COMPATIBLE, but theelsehangs off the outerif, which has already been evaluated. So on the one path that clears the flag, the non-NVML names are never bound — while every consumer in the file keys off the now-Falseflag and reaches for exactly those names:get_user_mode_driver_version()handle_return(driver.cuDriverGetVersion())get_num_devices()handle_return(runtime.cudaGetDeviceCount())Both would raise
NameError. In practice the module never reaches them:from cuda.core.system._nvml_context import initializeon the next line runs unconditionally, and_nvml_context.pyx:7is itselffrom cuda.bindings import nvml— so the veryImportErrorthetry/exceptexists to absorb is re-raised one statement later andimport cuda.core.systemfails outright.cuda.core.system.__init__is built around the flag being usable in this state (elif CUDA_BINDINGS_NVML_IS_COMPATIBLE:gates the NVML-only submodules), andcuda.core._devicereads it at_device.pyx:1053, so the degraded mode is a supported configuration — it just cannot be reached.Fix
Move the
_nvml_contextimport into the sametry(it depends onnvml, so it belongs there), and replace theelsewith a secondif not CUDA_BINDINGS_NVML_IS_COMPATIBLE:so a flag cleared by the failed import selects the fallback.Tests
The branch is import-time, so it can only be exercised in a fresh interpreter.
test_system_falls_back_when_nvml_is_unimportableruns a subprocess that installs asys.meta_pathfinder raisingImportErrorforcuda.bindings.nvml, then importscuda.core.systemand asserts:CUDA_BINDINGS_NVML_IS_COMPATIBLE is False,driver,handle_return,runtimeare bound oncuda.core.system._system.It does not call into the driver, so it needs no GPU.
What I ran
Environment: macOS, no CUDA driver and no CUDA toolkit, so
cuda.corecannot be built or imported here.cuda_core/tests/— they need a builtcuda.core.repro_nvml, unimportable;repro_nvml_context, which imports it), keeping the control flow verbatim:sys.meta_pathblocker used by the new test, standalone against a stdlib submodule, to confirm it turns bothfrom X import Yandimport X.YintoImportError.python -m py_compileon the changed test file andcompile()on the embedded subprocess script (both parse), plusruff check/ruff format --check.ruff checkreports the same single pre-existingI001on this file as it does onmain; no new findings.CUDA_BINDINGS_NVML_IS_COMPATIBLEin the tree (_system.pyx,system/__init__.py,_device.pyx:1053, and four test modules) — none of them depend on theelseform.