ci(cuda): build only for CI runner GPU arch and raise parallelism - #7753
ci(cuda): build only for CI runner GPU arch and raise parallelism#7753MrLi000001 wants to merge 3 commits into
Conversation
The CUDA CI built every .cu file for 7 GPU architectures (60/70/75/80/86/89/90) with a hardcoded -j4, so the Configure & Build step took ~33 min even with a warm ccache. - Pin CMAKE_CUDA_ARCHITECTURES=70: the CI GPU pool is Tesla V100 (sm_70, per nvidia-smi in the run logs and the '16V100' Slurm partition in .ci/slurm/config.ini). This cuts nvcc work by ~7x. - Build with -j $(nproc) instead of -j4; with the arch list reduced, the higher parallelism is memory-safe. Expected: Configure & Build ~33 min -> ~10 min on a cache-cold run.
|
This is an effective direction: Configure & Build dropped from about 33 minutes to 18m01s in the actual run. Before merging, please confirm that the entire gpu runner pool guarantees sm_70 and update the ~10-minute estimate with the measured result; otherwise, a heterogeneous runner may not be able to run the binary. |
Sister commit to e614a28. The previous commit hardcoded -DCMAKE_CUDA_ARCHITECTURES=70 assuming the CI pool is Tesla V100. Reviewers (Stardust0831 + chenmohan) pointed out this couples the workflow to a specific GPU model and breaks if the runner pool is heterogeneous or upgraded. This commit moves the arch selection into CMakeLists.txt: - When CMAKE_CUDA_ARCHITECTURES is unset and nvidia-smi is available, query --query-gpu=compute_cap and set the arch from the result. Map '7.0' -> 70, '8.0' -> 80, '8.9' -> 89, '9.0' -> 90, etc. - Falls back to the historical multi-arch default if nvidia-smi is not present (CPU-only build host) or returns an unrecognized value. - User-provided CMAKE_CUDA_ARCHITECTURES still takes precedence. Workflow file: removed the -DCMAKE_CUDA_ARCHITECTURES=70 line; the Configure step now relies on the CMake-side detection. Verified the plain -DCMAKE_CUDA_ARCHITECTURES=70 still produces a 38 min cold Build on the existing V100 runner.
|
The auto-detection block in 1. if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES AND USE_CUDA AND COMMAND nvidia-smi)
find_program(NVIDIA_SMI_EXECUTABLE nvidia-smi)
if(NVIDIA_SMI_EXECUTABLE)
execute_process(COMMAND ${NVIDIA_SMI_EXECUTABLE} --query-gpu=compute_cap ...)( 2. Even with that fixed, the detected value is immediately overwritten. The detection block and the historical default list run in sequence, with no if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES) # outer guard
if(... nvidia-smi ...)
set(CMAKE_CUDA_ARCHITECTURES "70" CACHE STRING ...) # cache entry
endif()
if(CUDAToolkit_VERSION VERSION_LESS "13.0")
set(CMAKE_CUDA_ARCHITECTURES 60 70) # <-- normal variable, shadows the cache entry
...
list(APPEND CMAKE_CUDA_ARCHITECTURES 75) # 80 / 86 / 89 / 90 follow
endif()
if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
# detection (via find_program)
endif()
if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
# historical multi-arch default
endif()Consequence: the 33 min → 18m01s that @Stardust0831 measured comes entirely from 3. Design: this belongs in the CI workflow, not in The title and description say "build only for CI runner GPU arch", but the diff changes the default for every user. On a cluster, people routinely configure on a login node whose GPU differs from — or is absent on — the compute node. Silent auto-detection would then produce a binary that dies with Adding This also settles @Stardust0831's question about whether the whole gpu pool is sm_70: an explicit pin is strictly better than detection, because on a heterogeneous pool detection makes each node emit different |
The previous commit (167904f) tried to make the workflow adapt to whatever GPU the runner has by auto-detecting compute capability at CMake configure time. Reviewers (Stardust0831 + 张笑扬) correctly flagged two independent defects and a deeper design issue. Defects in the previous commit: 1. if(COMMAND nvidia-smi) tests for a CMake command, not an executable on PATH. It was always false, so execute_process never ran. The correct check is find_program(NVIDIA_SMI_EXECUTABLE nvidia-smi). 2. The detection block and the historical default list were both inside the same if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES) outer guard. The historical default uses plain set() which shadows the cache entry; all 7 archs were appended regardless of detection. 3. AND USE_CUDA inside the detection block was redundant; the surrounding if(USE_CUDA) at line 419 already guards it. Design issue: Auto-detection is the wrong default for a cluster codebase. ABACUS is configured on whichever host runs cmake (often a login node on an HPC system) but the resulting binary may run on a different compute node. Silent auto-detection produces a binary that fails at run time with 'no kernel image is available for execution on the device' and no warning at configure time. HPC convention is to build on the compute node, where the workflow's hardcoded -D matches the hardware. Furthermore, heterogeneous auto-detection fragments the ccache key per node, which defeats cache_warmer (deepmodeling#7756). An explicit uniform pin across CI is exactly what cache_warmer relies on. This commit: - Reverts the CMakeLists.txt auto-detect block. - Restores -DCMAKE_CUDA_ARCHITECTURES=70 in cuda.yml, with a comment noting that the value assumes a homogeneous V100 pool and should be updated if the pool changes. - Keeps -j $(nproc), which is the real win in e614a28 (~2x parallelism). Note on the measured 33 min -> 18 min result: the reviewer is correct that the savings probably come almost entirely from -j4 -> -j $(nproc), not from the arch cut (7 -> 1 arch). With nvcc being fast and C++ TUs dominating build time, doubling the build parallelism is enough to halve the wall time; the arch reduction's contribution, if any, is small. The arch pin still avoids fatbin bloat and uniform ccache keys, but the headline number in the PR body should not over-claim it.
The CUDA CI built every .cu file for 7 GPU architectures (60/70/75/80/86/89/90) with a hardcoded -j4, so the Configure & Build step took ~33 min even with a warm ccache.
Expected: Configure & Build ~33 min -> ~10 min on a cache-cold run.
Reminder
AGENTS.mdanddocs/developers_guide/agent_governance.md.source/changes.Linked Issue
Fix #
Unit Tests and/or Case Tests for my changes
What's changed?
Governance Notes