Skip to content

Add GPU-aware MPI halo exchange for MPMesh - #90

Open
Shahrear2000 wants to merge 12 commits into
dn/Velocity_solver_improvementfrom
sj/gpu-aware-mpi-halo-exchange
Open

Shahrear2000 wants to merge 12 commits into
dn/Velocity_solver_improvementfrom
sj/gpu-aware-mpi-halo-exchange

Conversation

@Shahrear2000

Copy link
Copy Markdown

Replaces #88, with the same changes opened from the polyMPO branch sj/gpu-aware-mpi-halo-exchange.

Main changes:

  • Adds a GPU-aware MPI halo-exchange path in MPMesh, compiled when GPU_AWARE_MPI is defined.
  • Keeps CPU-GPU staged (host copy) path as fallback.
  • MPI buffers are allocated with plain cudaMalloc rather than Kokkos, because Kokkos' async memory pools are rejected by the CUDA IPC calls that Cray MPICH/GTL uses.

Build option:

  • New CMake option polyMPO_ENABLE_GPU_AWARE_MPI. It is ON by default only when Kokkos has CUDA and Cray MPICH GTL is available; otherwise OFF.
  • Override with -DpolyMPO_ENABLE_GPU_AWARE_MPI=ON/OFF.
  • When OFF (for example CPU-only builds and GitHub CI), no CUDA or Cray code is compiled and the CPU-staged path is used.

Runtime notes:

  • The GPU-aware path is used only when MPICH_GPU_SUPPORT_ENABLED=1; otherwise it falls back to the CPU-staged path.
  • The selected path is printed once on rank 0 (polyMPO halo exchange: ...).

Testing:

  • New testGPUAwareHaloExchange (4 ranks, owner → halo assignment, exact values checked).
  • Passes on Perlmutter with 4 ranks in both modes: MPICH_GPU_SUPPORT_ENABLED=1 (GPU-aware) and =0 (CPU-staged).
  • In GitHub CI the test is skipped: mpiexec -n 4 on the runner starts four independent single-rank processes, so the test exits with CTest's skip code (77).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant