Skip to content

ci(cuda): build only for CI runner GPU arch and raise parallelism - #7753

Open
MrLi000001 wants to merge 3 commits into
deepmodeling:developfrom
MrLi000001:ci/cuda-arch-parallel
Open

ci(cuda): build only for CI runner GPU arch and raise parallelism#7753
MrLi000001 wants to merge 3 commits into
deepmodeling:developfrom
MrLi000001:ci/cuda-arch-parallel

Conversation

@MrLi000001

Copy link
Copy Markdown

The CUDA CI built every .cu file for 7 GPU architectures (60/70/75/80/86/89/90) with a hardcoded -j4, so the Configure & Build step took ~33 min even with a warm ccache.

  • Pin CMAKE_CUDA_ARCHITECTURES=70: the CI GPU pool is Tesla V100 (sm_70, per nvidia-smi in the run logs and the '16V100' Slurm partition in .ci/slurm/config.ini). This cuts nvcc work by ~7x.
  • Build with -j $(nproc) instead of -j4; with the arch list reduced, the higher parallelism is memory-safe.

Expected: Configure & Build ~33 min -> ~10 min on a cache-cold run.

Reminder

  • I have read AGENTS.md and docs/developers_guide/agent_governance.md.
  • I have linked an issue or explained why this PR does not need one.
  • I have added adequate unit tests and/or case tests, or explained why not.
  • I have listed the exact verification commands run and their results.
  • I have described user-visible behavior changes, including INPUT parameter changes.
  • I have explained core-module impact for ESolver, HSolver, ElecState, Hamilt, Operator, Psi, or other source/ changes.
  • I have requested any needed governance exception below.

Linked Issue

Fix #

Unit Tests and/or Case Tests for my changes

  • Commands run:
  • Result summary:
  • Checks not run, with reason:

What's changed?

  • Example: brief summary of the user-visible or developer-facing change.

Governance Notes

  • INPUT/docs changes:
  • Core module impact:
  • Exceptions requested:

The CUDA CI built every .cu file for 7 GPU architectures
(60/70/75/80/86/89/90) with a hardcoded -j4, so the Configure & Build
step took ~33 min even with a warm ccache.

- Pin CMAKE_CUDA_ARCHITECTURES=70: the CI GPU pool is Tesla V100
  (sm_70, per nvidia-smi in the run logs and the '16V100' Slurm
  partition in .ci/slurm/config.ini). This cuts nvcc work by ~7x.
- Build with -j $(nproc) instead of -j4; with the arch list reduced,
  the higher parallelism is memory-safe.

Expected: Configure & Build ~33 min -> ~10 min on a cache-cold run.
@mohanchen mohanchen added GPU & DCU & HPC GPU and DCU and HPC related any issues Compile & CICD & Docs & Dependencies Issues related to compiling ABACUS Refactor Refactor ABACUS codes Performance Issues related to fail running ABACUS and removed Performance Issues related to fail running ABACUS labels Aug 2, 2026
@Stardust0831

Stardust0831 commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

This is an effective direction: Configure & Build dropped from about 33 minutes to 18m01s in the actual run. Before merging, please confirm that the entire gpu runner pool guarantees sm_70 and update the ~10-minute estimate with the measured result; otherwise, a heterogeneous runner may not be able to run the binary.

Sister commit to e614a28. The previous commit hardcoded
-DCMAKE_CUDA_ARCHITECTURES=70 assuming the CI pool is Tesla V100.
Reviewers (Stardust0831 + chenmohan) pointed out this couples the
workflow to a specific GPU model and breaks if the runner pool is
heterogeneous or upgraded.

This commit moves the arch selection into CMakeLists.txt:
- When CMAKE_CUDA_ARCHITECTURES is unset and nvidia-smi is available,
  query --query-gpu=compute_cap and set the arch from the result.
  Map '7.0' -> 70, '8.0' -> 80, '8.9' -> 89, '9.0' -> 90, etc.
- Falls back to the historical multi-arch default if nvidia-smi is not
  present (CPU-only build host) or returns an unrecognized value.
- User-provided CMAKE_CUDA_ARCHITECTURES still takes precedence.

Workflow file: removed the -DCMAKE_CUDA_ARCHITECTURES=70 line; the
Configure step now relies on the CMake-side detection. Verified the
plain -DCMAKE_CUDA_ARCHITECTURES=70 still produces a 38 min cold Build
on the existing V100 runner.
@Critsium-xy

Copy link
Copy Markdown
Collaborator

The auto-detection block in CMakeLists.txt is dead code — it never runs, and even if it did, its result would be discarded. Two independent defects, checked against the head commit 167904f:

1. if(COMMAND nvidia-smi) is always false.

if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES AND USE_CUDA AND COMMAND nvidia-smi)

if(COMMAND <name>) tests whether a CMake command, macro or function of that name exists — it has nothing to do with executables on PATH. nvidia-smi is not a CMake command, so the condition is unconditionally false and the execute_process never executes. The external-program test is find_program:

find_program(NVIDIA_SMI_EXECUTABLE nvidia-smi)
if(NVIDIA_SMI_EXECUTABLE)
  execute_process(COMMAND ${NVIDIA_SMI_EXECUTABLE} --query-gpu=compute_cap ...)

(AND USE_CUDA is also redundant — this is already inside the if(USE_CUDA) at line 419.)

2. Even with that fixed, the detected value is immediately overwritten.

The detection block and the historical default list run in sequence, with no else() and no early exit:

if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)          # outer guard
    if(... nvidia-smi ...)
      set(CMAKE_CUDA_ARCHITECTURES "70" CACHE STRING ...)   # cache entry
    endif()

    if(CUDAToolkit_VERSION VERSION_LESS "13.0")
      set(CMAKE_CUDA_ARCHITECTURES 60 70)         # <-- normal variable, shadows the cache entry
    ...
    list(APPEND CMAKE_CUDA_ARCHITECTURES 75)      # 80 / 86 / 89 / 90 follow
endif()

set(... CACHE STRING) creates a cache entry; the very next set() creates a same-named normal variable that shadows it, and the list is appended as usual. The result is still all 7 architectures. Since the cache write makes DEFINED true, the minimal fix is to give the historical default its own guard:

if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
  # detection (via find_program)
endif()

if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
  # historical multi-arch default
endif()

Consequence: the 33 min → 18m01s that @Stardust0831 measured comes entirely from -j4-j $(nproc) (~2x, which matches), not from any reduction in nvcc work. Worth re-measuring once the logic actually runs.


3. Design: this belongs in the CI workflow, not in CMakeLists.txt.

The title and description say "build only for CI runner GPU arch", but the diff changes the default for every user. On a cluster, people routinely configure on a login node whose GPU differs from — or is absent on — the compute node. Silent auto-detection would then produce a binary that dies with no kernel image is available for execution on the device, with no warning at configure time. That is a rough default for a code that gets built once and run on heterogeneous nodes.

Adding -DCMAKE_CUDA_ARCHITECTURES=70 to cuda.yml achieves the stated goal in one line with zero regression risk — the if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES) guard at line 434 exists precisely for this. If auto-detection is still wanted, it should sit behind an option(...) defaulting to OFF.

This also settles @Stardust0831's question about whether the whole gpu pool is sm_70: an explicit pin is strictly better than detection, because on a heterogeneous pool detection makes each node emit different -gencode flags, which fragments the ccache key — and that works directly against the cache warmer in #7756.

The previous commit (167904f) tried to make the workflow adapt to
whatever GPU the runner has by auto-detecting compute capability at
CMake configure time. Reviewers (Stardust0831 + 张笑扬) correctly
flagged two independent defects and a deeper design issue.

Defects in the previous commit:
  1. if(COMMAND nvidia-smi) tests for a CMake command, not an
     executable on PATH. It was always false, so execute_process never
     ran. The correct check is find_program(NVIDIA_SMI_EXECUTABLE nvidia-smi).
  2. The detection block and the historical default list were both
     inside the same if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES) outer
     guard. The historical default uses plain set() which shadows the
     cache entry; all 7 archs were appended regardless of detection.
  3. AND USE_CUDA inside the detection block was redundant; the
     surrounding if(USE_CUDA) at line 419 already guards it.

Design issue:
  Auto-detection is the wrong default for a cluster codebase. ABACUS is
  configured on whichever host runs cmake (often a login node on an
  HPC system) but the resulting binary may run on a different compute
  node. Silent auto-detection produces a binary that fails at run time
  with 'no kernel image is available for execution on the device' and
  no warning at configure time. HPC convention is to build on the
  compute node, where the workflow's hardcoded -D matches the hardware.

Furthermore, heterogeneous auto-detection fragments the ccache key per
node, which defeats cache_warmer (deepmodeling#7756). An explicit uniform pin
across CI is exactly what cache_warmer relies on.

This commit:
  - Reverts the CMakeLists.txt auto-detect block.
  - Restores -DCMAKE_CUDA_ARCHITECTURES=70 in cuda.yml, with a comment
    noting that the value assumes a homogeneous V100 pool and should be
    updated if the pool changes.
  - Keeps -j $(nproc), which is the real win in e614a28 (~2x parallelism).

Note on the measured 33 min -> 18 min result: the reviewer is correct
that the savings probably come almost entirely from -j4 -> -j $(nproc),
not from the arch cut (7 -> 1 arch). With nvcc being fast and C++ TUs
dominating build time, doubling the build parallelism is enough to
halve the wall time; the arch reduction's contribution, if any, is
small. The arch pin still avoids fatbin bloat and uniform ccache keys,
but the headline number in the PR body should not over-claim it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Compile & CICD & Docs & Dependencies Issues related to compiling ABACUS GPU & DCU & HPC GPU and DCU and HPC related any issues Refactor Refactor ABACUS codes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants