ci: make ccache effective and observable, shallow checkout, add cache… - #7756
ci: make ccache effective and observable, shallow checkout, add cache…#7756MrLi000001 wants to merge 1 commit into
Conversation
… warmer Measured on the self-hosted runners, the Build step of the same workflow ranges from 2.7 min (warm ccache) to 37.5 min (stale ccache), and a full-history checkout took ~12 min on cache-cold runners. - Set ccache --max-size=30G (default ~10G is tight for -O3 -g builds) and print ccache -s before/after the build so hit rates are visible in every run log (if: always()). - fetch-depth: 0 -> 1; CMake only runs 'git log -1' for commit info. - Add cache_warmer.yml: on every push to develop, build (without tests) on both the X64 and gpu pools so /tmp/ccache stays primed with the latest develop. This prevents the first PR runs after a big refactor merge from all paying a full rebuild.
|
The shallow checkout and ccache statistics are useful, but I suggest splitting out the cache warmer. It is not yet clear whether /tmp/ccache persists across jobs and nodes in the dynamic runner pool, so one warmer may only warm a single node while consuming an additional GPU. It also builds the default seven CUDA architectures, while #7753 uses sm_70, so those CUDA cache entries will not be effectively reused. |
|
A few concerns on the 1. The configure flags are copy-pasted and already out of sync. The header comment says to keep them aligned with 2. 3. Neither warmer job sets Given 2 and 3, I'd suggest splitting as @Stardust0831 proposed: land the |
The previous commit (167904f) tried to make the workflow adapt to whatever GPU the runner has by auto-detecting compute capability at CMake configure time. Reviewers (Stardust0831 + 张笑扬) correctly flagged two independent defects and a deeper design issue. Defects in the previous commit: 1. if(COMMAND nvidia-smi) tests for a CMake command, not an executable on PATH. It was always false, so execute_process never ran. The correct check is find_program(NVIDIA_SMI_EXECUTABLE nvidia-smi). 2. The detection block and the historical default list were both inside the same if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES) outer guard. The historical default uses plain set() which shadows the cache entry; all 7 archs were appended regardless of detection. 3. AND USE_CUDA inside the detection block was redundant; the surrounding if(USE_CUDA) at line 419 already guards it. Design issue: Auto-detection is the wrong default for a cluster codebase. ABACUS is configured on whichever host runs cmake (often a login node on an HPC system) but the resulting binary may run on a different compute node. Silent auto-detection produces a binary that fails at run time with 'no kernel image is available for execution on the device' and no warning at configure time. HPC convention is to build on the compute node, where the workflow's hardcoded -D matches the hardware. Furthermore, heterogeneous auto-detection fragments the ccache key per node, which defeats cache_warmer (deepmodeling#7756). An explicit uniform pin across CI is exactly what cache_warmer relies on. This commit: - Reverts the CMakeLists.txt auto-detect block. - Restores -DCMAKE_CUDA_ARCHITECTURES=70 in cuda.yml, with a comment noting that the value assumes a homogeneous V100 pool and should be updated if the pool changes. - Keeps -j $(nproc), which is the real win in e614a28 (~2x parallelism). Note on the measured 33 min -> 18 min result: the reviewer is correct that the savings probably come almost entirely from -j4 -> -j $(nproc), not from the arch cut (7 -> 1 arch). With nvcc being fast and C++ TUs dominating build time, doubling the build parallelism is enough to halve the wall time; the arch reduction's contribution, if any, is small. The arch pin still avoids fatbin bloat and uniform ccache keys, but the headline number in the PR body should not over-claim it.
… warmer
Measured on the self-hosted runners, the Build step of the same workflow ranges from 2.7 min (warm ccache) to 37.5 min (stale ccache), and a full-history checkout took ~12 min on cache-cold runners.
Reminder
AGENTS.mdanddocs/developers_guide/agent_governance.md.source/changes.Linked Issue
Fix #
Unit Tests and/or Case Tests for my changes
What's changed?
Governance Notes