Add CUDA-aware MPI halo exchange for MPMesh by Shahrear2000 · Pull Request #88 · SCOREC/polyMPO

Shahrear2000 · 2026-06-29T14:50:42Z

Main changes:

Adds a GPU-aware MPI halo-exchange path in MPMesh, compiled when GPU_AWARE_MPI is defined.
Keeps CPU-GPU staged (host copy) path as fallback.
MPI buffers are allocated with plain cudaMalloc rather than Kokkos, because Kokkos' async memory pools are rejected by the CUDA IPC calls that Cray MPICH/GTL uses.

Build option:

New CMake option polyMPO_ENABLE_GPU_AWARE_MPI. It is ON by default only when Kokkos has CUDA and Cray MPICH GTL is available; otherwise OFF.
Override with -DpolyMPO_ENABLE_GPU_AWARE_MPI=ON/OFF.
When OFF (for example CPU-only builds and GitHub CI), no CUDA or Cray code is compiled and the CPU-staged path is used.

Runtime notes:

The GPU-aware path is used only when MPICH_GPU_SUPPORT_ENABLED=1; otherwise it falls back to the CPU-staged path.
The selected path is printed once on rank 0 (polyMPO halo exchange: ...).

Testing:

New testGPUAwareHaloExchange (4 ranks, owner → halo assignment, exact values checked).
Passes on Perlmutter with 4 ranks in both modes: MPICH_GPU_SUPPORT_ENABLED=1 (GPU-aware) and =0 (CPU-staged).
In GitHub CI the test is skipped: mpiexec -n 4 on the runner starts four independent single-rank processes, so the test exits with CTest's skip code (77).

cwsmith

I didn't see anything that was CUDA specific so I'd suggest replacing 'CUDA' in the function/structure/etc. names with 'GPU' for portability to AMD and Intel GPUs.

It looked like one of the files was mostly formatting changes. Including them in the PR makes it hard to find the critical changes.

I also complained about the function names having a suffix with a number and another with 'improved'.

There was also no test case added for this functionality. Any unit test that exercises this code would be a significant improvement.

Note, I didn't read the gpu mpi exchange code in detail. If feedback at that level is needed I can take a look.

cwsmith · 2026-06-30T14:25:52Z

 )

 add_library(polyMPO-core ${SOURCES})
+target_compile_definitions(polyMPO-core PUBLIC CUDA_AWARE_MPI)


Naming this 'GPU_AWARE_MPI' would be more portable (i.e., to AMD and Intel GPUs).

cwsmith · 2026-06-30T14:26:45Z

  int numVertices = p_mesh->getNumVertices();
  auto vtxFieldVel = p_mesh->getMeshField<polyMPO::MeshF_Vel>();
-  mpMesh->communicate_and_take_halo_contributions1(vtxFieldVel, numVertices, 2, 1, 1); 
+  mpMesh->communicate_and_take_halo_contributions1_improved(vtxFieldVel, numVertices, 2, 1, 1); 


I suggest removing the '1' and replacing 'improved' with something meaningful.

cwsmith · 2026-06-30T14:27:18Z

    void startCommunication();

-    void communicate_and_take_halo_contributions(const Kokkos::View<double**>& meshField, int nEntities, int numEntries, int mode, int op);
+    void communicate_and_take_halo_contributions(


Is this API still in use?

cwsmith · 2026-06-30T14:28:55Z

-      //communicateFields1(fieldData1, nEntities, numEntries, mode, recvIDVec, recvDataVec);
-      communicateFields1(reconVals_host, nEntities, numEntries, mode, recvIDVec, recvDataVec);
+
+      communicateFields1(


In general, adding a number to an API (i.e., the 1 at the end) is just going to create problems in the long term for anyone but the person who wrote them.

cwsmith · 2026-06-30T14:30:14Z

+        const ViewType& fieldData,
+        const int numEntities,
+        const int numEntries,
+        int mode,


these formatting only changes should ideally not be part of this PR

cwsmith · 2026-06-30T14:35:19Z

+          }
+        }
+
+        Kokkos::fence();


is this required?

cwsmith · 2026-06-30T14:36:16Z

+                const int vertex = recvIDGPU(i);
+
+                for(int k = 0; k < numEntries; k++){
+#ifdef POLYMPO_ASSUME_UNIQUE_HALO_CONTRIBS


describing this flag in the docstring for the function would be a good idea

…nners" This reverts commit 95c1885.

This reverts commit 1d95e6a.

Shahrear2000 · 2026-09-26T00:20:14Z

Superseded by #90, which contains the same changes opened from the polyMPO branch sj/gpu-aware-mpi-halo-exchange. Closing this PR.

Shahrear2000 requested review from Angelyr, cwsmith, dhyan1272 and onkarsahni as code owners June 29, 2026 14:50

cwsmith reviewed Jun 30, 2026

View reviewed changes

Shahrear2000 force-pushed the pmpo_MPMesh_improved branch from c3622fd to fdb365e Compare July 13, 2026 22:58

Shahrear2000 force-pushed the pmpo_MPMesh_improved branch 3 times, most recently from 37b4533 to 4d6dcda Compare August 4, 2026 05:22

Shahrear2000 force-pushed the pmpo_MPMesh_improved branch from cb5fcea to eacbd80 Compare September 15, 2026 01:50

Shahrear Jahan Santho added 2 commits September 24, 2026 18:31

Add CUDA-aware MPI halo exchange

7ba0b24

Add GPU-aware halo exchange, unit test and cleanup

ce6cfc9

Shahrear2000 force-pushed the pmpo_MPMesh_improved branch from eacbd80 to ce6cfc9 Compare September 25, 2026 03:38

Shahrear Jahan Santho and others added 10 commits September 24, 2026 22:27

Clean up unrelated formatting changes

7760bae

Clean up remaining unrelated formatting changes

b993971

Clean up remaining formatting changes

e509612

Clean up

b096020

Make GPU-aware MPI a CMake option, on by default for Cray GPU builds

7a46fd5

CI: launch MPI tests with MPICH's mpiexec

1d95e6a

CI: restrict UCX transports so MPICH initializes on GitHub runners

95c1885

Revert "CI: restrict UCX transports so MPICH initializes on GitHub ru…

78452b8

…nners" This reverts commit 95c1885.

Revert "CI: launch MPI tests with MPICH's mpiexec"

29031aa

This reverts commit 1d95e6a.

Skip GPU-aware halo exchange test when not run with 4 MPI ranks

fdb7efd

Shahrear2000 mentioned this pull request Sep 26, 2026

Add GPU-aware MPI halo exchange for MPMesh #90

Open

Shahrear2000 closed this Sep 26, 2026

Navigation Menu

Repository navigation

Add CUDA-aware MPI halo exchange for MPMesh - #88

Add CUDA-aware MPI halo exchange for MPMesh#88
Shahrear2000 wants to merge 12 commits into
SCOREC:dn/Velocity_solver_improvementfrom
Shahrear2000:pmpo_MPMesh_improved

Shahrear2000 commented Jun 29, 2026 •

edited

Loading

Uh oh!

cwsmith left a comment

Uh oh!

cwsmith Jun 30, 2026

Uh oh!

cwsmith Jun 30, 2026

Uh oh!

cwsmith Jun 30, 2026

Uh oh!

cwsmith Jun 30, 2026

Uh oh!

cwsmith Jun 30, 2026

Uh oh!

cwsmith Jun 30, 2026

Uh oh!

cwsmith Jun 30, 2026

Uh oh!

Shahrear2000 commented Sep 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Shahrear2000 commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cwsmith left a comment

Choose a reason for hiding this comment

Uh oh!

cwsmith Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

cwsmith Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

cwsmith Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

cwsmith Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

cwsmith Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

cwsmith Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

cwsmith Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Shahrear2000 commented Sep 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Shahrear2000 commented Jun 29, 2026 •

edited

Loading