Skip to content

Add GPU-aware MPI halo exchange for MPMesh #90

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Shahrear2000 wants to merge 12 commits into
base: dn/Velocity_solver_improvement
Choose a base branch
from
Open

Add GPU-aware MPI halo exchange for MPMesh #90

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
7ba0b24
Add CUDA-aware MPI halo exchange
Sep 25, 2026
ce6cfc9
Add GPU-aware halo exchange, unit test and cleanup
Sep 25, 2026
7760bae
Clean up unrelated formatting changes
Sep 25, 2026
b993971
Clean up remaining unrelated formatting changes
Sep 25, 2026
e509612
Clean up remaining formatting changes
Sep 25, 2026
b096020
Clean up
Sep 25, 2026
7a46fd5
Make GPU-aware MPI a CMake option, on by default for Cray GPU builds
Shahrear2000 Sep 25, 2026
1d95e6a
CI: launch MPI tests with MPICH's mpiexec
Shahrear2000 Sep 25, 2026
95c1885
CI: restrict UCX transports so MPICH initializes on GitHub runners
Shahrear2000 Sep 25, 2026
78452b8
Revert "CI: restrict UCX transports so MPICH initializes on GitHub ru…
Shahrear2000 Sep 25, 2026
29031aa
Revert "CI: launch MPI tests with MPICH's mpiexec"
Shahrear2000 Sep 25, 2026
fdb7efd
Skip GPU-aware halo exchange test when not run with 4 MPI ranks
Shahrear2000 Sep 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions src/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,44 @@ set(SOURCES
)

add_library(polyMPO-core ${SOURCES})

# Default ON only for Cray GPU builds (Kokkos CUDA + Cray MPICH GTL available).
# Can still be overridden with -DpolyMPO_ENABLE_GPU_AWARE_MPI=ON/OFF.
set(polyMPO_GPU_AWARE_MPI_DEFAULT OFF)
if(Kokkos_ENABLE_CUDA
AND DEFINED ENV{PE_MPICH_GTL_DIR_nvidia80}
AND DEFINED ENV{PE_MPICH_GTL_LIBS_nvidia80})
set(polyMPO_GPU_AWARE_MPI_DEFAULT ON)
endif()

option(polyMPO_ENABLE_GPU_AWARE_MPI
"Use GPU-aware (CUDA-aware) MPI for halo exchange"
${polyMPO_GPU_AWARE_MPI_DEFAULT})
message(STATUS "polyMPO_ENABLE_GPU_AWARE_MPI: ${polyMPO_ENABLE_GPU_AWARE_MPI}")

if(polyMPO_ENABLE_GPU_AWARE_MPI)
if(NOT Kokkos_ENABLE_CUDA)
message(FATAL_ERROR
"polyMPO_ENABLE_GPU_AWARE_MPI=ON requires Kokkos built with CUDA.")
endif()

if(DEFINED ENV{PE_MPICH_GTL_DIR_nvidia80}
AND DEFINED ENV{PE_MPICH_GTL_LIBS_nvidia80})
target_link_options(polyMPO-core PUBLIC
"$ENV{PE_MPICH_GTL_DIR_nvidia80}"
"$ENV{PE_MPICH_GTL_LIBS_nvidia80}"
)
message(STATUS "polyMPO: enabling Cray MPICH CUDA GTL linkage")
else()
message(FATAL_ERROR
"Cray MPICH CUDA GTL environment variables are unavailable. "
"Load cudatoolkit and craype-accel-nvidia80, or set "
"CRAY_ACCEL_TARGET=nvidia80 before configuring.")
endif()

target_compile_definitions(polyMPO-core PUBLIC GPU_AWARE_MPI)
endif()

set_property(TARGET polyMPO-core PROPERTY CXX_STANDARD "17")
set_property(TARGET polyMPO-core PROPERTY CXX_STANDARD_REQUIRED ON)
set_property(TARGET polyMPO-core PROPERTY CXX_EXTENSIONS OFF)
Expand Down
4 changes: 2 additions & 2 deletions src/pmpo_MPMesh.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,12 +166,12 @@ void MPMesh::calculateStressDivergence(){
};
p_MPs->parallel_for(stress_div, " stress_div_assembly");
Kokkos::fence();
pumipic::RecordTime("Stress_Divergence_Reconstruction" + std::to_string(self), timer.seconds());
pumipic::RecordTime("Stress_Divergence_Reconstruction" + std::to_string(self), timer.seconds());

timer.reset();
if(numProcsTot>1){
//Takes contribution of halo vertices and adds it in owner procs
communicate_and_take_halo_contributions1(stress_divUV, numVertices, 2, 0, 0);
communicate_and_take_halo_contributions_gpu_aware(stress_divUV, numVertices, 2, 0, 0);
//Transfer the correct values at owned vertices to halo vertices
//communicate_and_take_halo_contributions(stress_divUV, numVertices, 2, 1, 1);
}
Expand Down
Loading
Loading