ci: fix cupy numba PTX version mismatch on self-hosted runner#753
Open
hongyx11 wants to merge 1 commit into
Open
ci: fix cupy numba PTX version mismatch on self-hosted runner#753hongyx11 wants to merge 1 commit into
hongyx11 wants to merge 1 commit into
Conversation
numba compiled @cuda.jit kernels with the runner's system CUDA 12.9 libNVVM (via CUDA_HOME), emitting PTX .version 8.8, which the 12.8 GPU driver cannot JIT-link (CUDA_ERROR_UNSUPPORTED_PTX_VERSION). This broke 19 tests in test_fourierradon/test_kirchhoff/test_nonstatconvolve after the cupy CI switched to uv sync. Point numba at the pip-installed CUDA 12.8 toolkit instead: - pin nvidia-cuda-nvcc-cu12<12.9 in the gpu-cu12 extra (relock) - export CUDA_HOME to the wheel's cuda_nvcc dir in the test step - add a versioned libnvvm.so.4 symlink, since numba's find_lib only matches libnvvm.so.N but the wheel ships a bare libnvvm.so Full pytests/ suite: 2156 passed, 0 failed (was 19 failed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 0 |
| Duplication | 0 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The CuPy CI (
buildcupy.yaml) had 19 failing tests intest_fourierradon,test_kirchhoff, andtest_nonstatconvolve, all with:Root cause
numba compiles
@cuda.jitkernels using thelibNVVMit finds viaCUDA_HOME. On the self-hosted runner that pointed at a system CUDA 12.9 toolkit, whose NVVM emits PTX.version 8.8. The GPU driver is 12.8, and its in-driver JIT linker (cuLinkAddData) only accepts PTX ≤8.7→ the kernels fail to link.This regressed when the CuPy CI switched to
uv sync(commit 7697d6e); the previous pip flow never pulled a 12.9 toolchain. cupy itself is unaffected — only numba kernels.Fix
Force numba to use the pip-installed CUDA 12.8 toolkit (matches the 12.8 driver, emits PTX 8.7):
pyproject.toml: pinnvidia-cuda-nvcc-cu12<12.9in thegpu-cu12extra (+ relockuv.lock→ 12.8.93).buildcupy.yamltest step:export CUDA_HOMEto the wheel'scuda_nvccdir.buildcupy.yamlinstall step: add a versionedlibnvvm.so.4symlink — numba'sfind_libregex only matcheslibnvvm.so.N, but the wheel ships a barelibnvvm.so.Verification
Ran the exact workflow steps on an L40S GPU node:
19 failed, 2139 passed, 881 skipped2156 passed, 0 failed, 881 skippedOffline check confirms the wheel NVVM emits
.version 8.7(was8.8).🤖 Generated with Claude Code