Run the hybrid solver on nano-mpi: ranks as threads, no mpirun - #114
Run the hybrid solver on nano-mpi: ranks as threads, no mpirun#114danielepanozzo wants to merge 5 commits into
Conversation
The hybrid solver needed MPI, which meant the whole application had to be launched under mpirun. That is fine for a driver but not for a library: anything embedding polysolve inherited the requirement. hypre can now build its MPI surface on threads of one process (HYPRE_ENABLE_THREAD_MPI), so the ranks become threads and mpirun goes away. The resulting binary links no libmpi at all. What this needed: - tmpi_mpi_compat.hpp: the few MPI facilities hypre's backend does not already provide. hypre maps MPI_Bcast/Allreduce/Scatterv/... onto its own surface already; missing were the MPI-3 shared-memory windows (with threads, "shared memory" is just a pointer broadcast), MPI_IN_PLACE, and MPI_Init/Initialized/Finalized. - The ranks are started by the solver rather than by mpirun. The first hybrid solver constructed calls hypre_tmpi_team_start(), the calling thread becomes rank 0 and keeps driving, and the workers sit in the existing command loop. They now return from their thread function instead of std::exit(0), and the last solver destroyed shuts the team down so the rest of the process sees a one-rank world again. - is_running_worker_loop and worker_registry are thread_local. They were per-rank only because each rank used to be a process; as threads they were shared and the workers raced on them. MPI remains available: POLYSOLVE_WITH_MPI still selects a rank-parallel hybrid solver, it just no longer implies OpenMPI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## hybrid-solver #114 +/- ##
==============================================
Coverage 80.07% 80.07%
==============================================
Files 52 52
Lines 2138 2138
Branches 284 284
==============================================
Hits 1712 1712
Misses 426 426
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The Catch2 tests cannot time the multi-rank path: they drive the solver from rank 0 while the other ranks sit in the worker loop, and under mpirun at more than one rank the suite is torn down when a worker leaves the loop. Timing that measures the teardown, not the solve. tests/bench_spmd.cpp is the same source built into both an OpenMPI and a thread-MPI build, so the two backends are compared on one driver rather than on two that happen to look similar. It builds a 7-point Laplacian of a given grid size, solves it, and prints setup/solve/total plus the relative residual so a run that converged differently cannot be mistaken for a run that was merely faster. scripts/bench_hybrid.sh sweeps rank counts on both backends. It takes an flock so two sweeps cannot halve each other's scores, refuses to start against a loaded machine rather than quietly reporting contended numbers, and interleaves the backends inside each repetition so drift hits both equally. scripts/bench_report.py renders the CSV as Markdown: best-of-N rather than mean, the worst run-to-run spread so the reader can judge whether a gap is real, and a check that both backends reached the same residual at each rank count. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Benchmark: OpenMPI processes vs thread-MPI threadsThreadripper PRO 3995WX, 64 cores, single NUMA node, idle. 7-point Laplacian, 120³ = 1.73M unknowns, Grid 120^3, best of 3 runs; worst run-to-run spread 10.7%.
setup vs solve
Reading thisCorrectness first: both backends reach an identical relative residual at every rank count. The residual changes with the rank count — the partitioning changes, so AMG builds a different hierarchy — but never between backends at the same count. Threads win at 1–2 ranks, OpenMPI wins from 4 up, ending 27% ahead at 64. The loss is not where I expected: look at the setup/solve split. thread-MPI's solve is competitive and actually faster at 16 and 32 ranks (0.50s vs 0.54s). The entire deficit is in setup, which plateaus around 0.51s for threads while OpenMPI keeps falling to 0.28s. AMG setup is the communication-heavy phase, so this points at the thread-MPI collective/matching path rather than anything about the solve. Caveat on precision: worst run-to-run spread is 10.7%, which is larger than I would like. The 1–2 rank and 32–64 rank conclusions are safely outside that; the 4-rank crossover (0.96x) is inside it and should be read as "roughly equal", not as a win for either. What this does and does not justifyIf you need Still to comeGPU numbers. |
GPU resultsSame driver, same 120³ problem, RTX 3080 Ti. Best of 3, load 2.1–2.2, spread under 2%.
One GPU is 2.5× faster than all 64 CPU cores at their best, and 18× faster than a single rank. The two builds are 1.2% apart, which is the point of the table. A correction worth recordingMy first GPU run reported times identical to CPU. That was not the GPU being slow: Note also that Summary across both comments
|
| find_package(MPI QUIET) | ||
| if (NOT MPI_CXX_FOUND) | ||
| message(WARNING "POLYSOLVE_WITH_MPI was requested but MPI was not found, proceeding without MPI dependent solvers.") | ||
| if (FALSE) |
There was a problem hiding this comment.
Pull request overview
This PR updates the hybrid linear solver’s “rank” acquisition to use hypre’s threads-as-ranks backend so embedding applications no longer need to launch under mpirun, while keeping POLYSOLVE_WITH_MPI=ON as the selector for rank-parallel hybrid solving.
Changes:
- Introduces a thread-MPI compatibility shim (
tmpi_mpi_compat.hpp) to cover missing MPI facilities (shared windows,MPI_IN_PLACE, init-state calls). - Refactors
CPUHybridSolverto start/join hypre thread ranks internally, and makes worker-loop state/registrythread_localto avoid cross-thread races. - Adds an SPMD benchmark driver plus scripts to sweep rank counts and report results.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| CMakeLists.txt | Adjusts MPI discovery behavior when POLYSOLVE_WITH_MPI is enabled. |
| cmake/recipes/hypre.cmake | Switches hypre recipe to thread-MPI backend (fork/branch) and toggles hypre MPI options. |
| src/polysolve/linear/tmpi_mpi_compat.hpp | New MPI-compat layer for hypre thread-MPI builds (windows, in-place reductions, init-state). |
| src/polysolve/linear/CPUHybridSolver.hpp | Makes worker state thread_local; adds rank-team lifetime management helpers. |
| src/polysolve/linear/CPUHybridSolver.cpp | Starts/stops thread ranks via hypre tmpi team; worker threads return instead of exiting. |
| tests/CMakeLists.txt | Adds bench_spmd executable for benchmarking. |
| tests/bench_spmd.cpp | New SPMD benchmark program that drives the hybrid solver from all ranks. |
| scripts/bench_hybrid.sh | New sweep script to benchmark OpenMPI vs thread-MPI backends. |
| scripts/bench_report.py | New CSV→Markdown report generator for benchmark output. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| if(POLYSOLVE_WITH_MPI) | ||
| # hypre's thread-MPI backend supplies the ranks, so no MPI installation is | ||
| # needed. Keep looking for one only to satisfy anything that still wants it. | ||
| find_package(MPI QUIET) | ||
| if (NOT MPI_CXX_FOUND) | ||
| message(WARNING "POLYSOLVE_WITH_MPI was requested but MPI was not found, proceeding without MPI dependent solvers.") | ||
| if (FALSE) |
| include(CPM) | ||
| CPMAddPackage( | ||
| NAME hypre | ||
| GITHUB_REPOSITORY hypre-space/hypre | ||
| GIT_TAG 7e247a231ebdeb44b06c7c9d3b5bee3bac21123f | ||
| GITHUB_REPOSITORY danielepanozzo/hypre | ||
| GIT_TAG thread-mpi-backend | ||
| SOURCE_SUBDIR src | ||
| ) |
| Eigen::setNbThreads(1); | ||
| HYPRE_SetMemoryLocation(HYPRE_MEMORY_HOST); | ||
| HYPRE_SetExecutionPolicy(HYPRE_EXEC_HOST); | ||
| spdlog::set_level(spdlog::level::off); |
| if [ "$backend" = openmpi ]; then | ||
| line=$(mpirun -quiet --oversubscribe -np "$n" "$MPI_BUILD/tests/bench_spmd" "$GRID" 2>/dev/null | tail -1) | ||
| else | ||
| line=$(HYPRE_TMPI_NUM_THREADS=$n "$TMPI_BUILD/tests/bench_spmd" "$GRID" 2>/dev/null | tail -1) | ||
| fi |
| runs = defaultdict(list) | ||
| grids = set() | ||
| res = defaultdict(set) # residual per rank count, not overall | ||
| with open(path) as fh: |
| if (sendbuf == MPI_IN_PLACE) | ||
| { | ||
| const size_t nb = polysolve::tmpi_compat::datatype_size(dt) * (size_t) count; | ||
| void *tmp = std::malloc(nb ? nb : 1); | ||
| std::memcpy(tmp, recvbuf, nb); | ||
| const HYPRE_Int rc = hypre_MPI_Allreduce(tmp, recvbuf, count, dt, op, comm); | ||
| std::free(tmp); | ||
| return rc; | ||
| } |
The threads-as-ranks backend this branch was written against lived inside hypre, behind a HYPRE_ENABLE_THREAD_MPI build mode, and needed a 183-line compatibility shim here to fill the gaps. It is now a library of its own: https://github.com/danielepanozzo/nano-mpi which changes what polysolve has to do. nano-mpi installs a header named mpi.h, so hypre is built as an ordinary MPI build -- HYPRE_ENABLE_MPI=ON, find_package(MPI) as it always did -- and the shim deletes entirely: * MPI_IN_PLACE, MPI_Init/Initialized/Finalized, and const-correct collectives are all native now. * MPI-3 shared-memory windows are native too. They were the last thing keeping the shim alive, and they are the one part of MPI's one-sided chapter that is trivially true when ranks are threads: the window is a real allocation plus everyone's offset into it. nano-mpi implements the layout MPI actually promises rather than the shim's simplification, so MPI_Win_shared_query answers correctly for every rank, not just rank 0. hypre_tmpi_team_start/join -> nanompi_team_start/join HYPRE_TMPI_NUM_THREADS -> NANOMPI_NUM_RANKS The one piece of machinery this adds is cmake/nanompi-as-mpi/FindMPI.cmake. hypre calls find_package(MPI REQUIRED) internally and would otherwise find a system Open MPI -- whose ranks are processes, needing a launcher polysolve has no way to invoke. The shim is on CMAKE_MODULE_PATH only when POLYSOLVE_WITH_MPI is on, and it points MPI at nano-mpi. It gives MPI::MPI_C the header path and nothing else, deliberately: hypre puts that target in CMAKE_REQUIRED_LIBRARIES and runs check_c_source_compiles, and try_compile() exports it into a scratch project where a link interface naming nanompi::nanompi is a hard error. Whoever links MPI links nanompi::nanompi explicitly instead. The probe in question is for MPI_Comm_f2c, which nano-mpi does not declare -- it has no Fortran bindings and cannot, since Fortran SAVE storage is per-process by language rule -- so it correctly comes out false. Verified on macOS/arm64, Release: unit_tests: All tests passed (1521 assertions in 23 test cases) otool -L bench_spmd | grep -i mpi -> nothing bench_spmd 40 CPUHybrid, 64k unknowns: 1 rank setup 0.040 solve 0.098 relres 3.02e-11 2 ranks setup 0.028 solve 0.057 relres 7.57e-11 4 ranks setup 0.014 solve 0.041 relres 1.97e-11 nano-mpi is pinned to v0.1.0. hypre still points at the fork branch, because the re-entrancy fix it needs (nine globals made thread-local) is still an open PR upstream; that pointer moves when it lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
danielepanozzo
changed the title
Every existing job configures with POLYSOLVE_WITH_MPI=OFF, which leaves the
hybrid solver out of the build entirely. A green CI on this branch therefore
said nothing about the thing the branch is for.
This job turns MPI on. Ranks are threads of the test process, so there is
nothing to install and nothing to launch -- the Linux dependency list is the
same as the others minus the mpi package.
Beyond building and running ctest it checks two things the port could plausibly
get wrong and still pass tests:
* that no MPI runtime is linked into the binary, since the whole point is
that ranks are threads and not processes;
* that the solver converges at 1, 2 and 4 ranks, since a rank count that
silently produces a wrong decomposition would otherwise look like a pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Linux CI job added in the previous commit found what the macOS build did not: AMGCL's MPI headers need MPI_CXX_DOUBLE_COMPLEX and MPI_CXX_FLOAT_COMPLEX (amgcl/mpi/util.hpp maps std::complex<T> onto them unconditionally), MPI_Exscan (mpi/partition/util.hpp) and MPI_Ialltoall (mpi/coarsening/pmis.hpp). v0.1.1 has all four, plus the rest of the nonblocking collectives and a Windows port. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Runs the hybrid solver's MPI ranks as threads of the calling process, so polysolve no longer needs its caller to launch under
mpirunto get hypre's fast, domain-decomposed algorithms.Stacked on #112.
How
The MPI is nano-mpi (v0.1.0), a small implementation in which every rank is a thread: no launcher, no daemon,
MPI_Sendbetween ranks is amemcpy. hypre is built as an ordinary MPI build against it —HYPRE_ENABLE_MPI=ON,find_package(MPI)as always — because nano-mpi installs a header namedmpi.h.CPUHybridSolverkeeps its SPMD shape. It brings a rank team up on first use and tears it down when the last hybrid solver is destroyed, so anything else in the process that uses hypre — the plainHypreSolver, say — still sees a one-rank world:Results
macOS/arm64, Release, this branch:
bench_spmd 40 CPUHybrid, 64k unknowns:Earlier measurements on a 64-core Threadripper PRO 3995WX (120³ = 1.73M unknowns, idle machine, best of 3, worst spread 10.7%) are in the comments below, comparing this against a real OpenMPI build of the same driver. Short version: threads win at 1–2 ranks, OpenMPI pulls ahead from 4 up, and the gap is entirely in setup — the solve phase is actually faster with threads at 16 and 32 ranks. Identical residuals at every rank count. On GPU,
GPUHybridis within 1.2% between the two builds, confirming the port does not touch the GPU path.What this PR now contains
The branch was originally written against a threads-as-ranks backend vendored inside hypre, behind a
HYPRE_ENABLE_THREAD_MPIbuild mode, plus a 183-line compatibility shim here (tmpi_mpi_compat.hpp) supplying MPI-3 shared-memory windows,MPI_IN_PLACEand the init calls that backend lacked. That backend is now a project of its own, and the shim deletes entirely — all of it is native in nano-mpi, including the windows, which are the one part of MPI's one-sided chapter that is trivially true when ranks are threads.Also here:
tests/bench_spmd.cpp, a standalone SPMD driver that builds a 7-point Laplacian and times setup and solve separately, plusscripts/bench_hybrid.sh(rank sweep across both backends, interleaved,flocked, waits for an idle machine) andscripts/bench_report.py(renders the CSV, best-of-N with the spread reported, and checks that the two backends agree on the residual at each rank count — it legitimately varies between rank counts, because the partitioning changes the AMG hierarchy).The one piece of machinery
cmake/nanompi-as-mpi/FindMPI.cmake. hypre callsfind_package(MPI REQUIRED)internally and would otherwise find a system Open MPI, whose ranks are processes and which then wants a launcher polysolve cannot invoke. The shim is onCMAKE_MODULE_PATHonly whenPOLYSOLVE_WITH_MPIis on, and points MPI at nano-mpi.It gives
MPI::MPI_Cthe header path and nothing else, deliberately: hypre puts that target inCMAKE_REQUIRED_LIBRARIESand runscheck_c_source_compiles, andtry_compile()exports it into a scratch project where a link interface namingnanompi::nanompiis a hard error. Whoever links MPI linksnanompi::nanompiexplicitly instead. The probe in question is forMPI_Comm_f2c, which nano-mpi does not declare — it has no Fortran bindings and cannot, since FortranSAVEstorage is per-process by language rule — so it correctly comes out false, which hypre already handles.Caveats
cmake/recipes/hypre.cmakestill points atdanielepanozzo/hypre@thread-mpi-backend. hypre needs nine process-wide globals made thread-local — a re-entrancy fix that is worth having on its own terms and is an open PR (danielepanozzo/hypre#1). That pointer moves to upstream hypre when it lands, and this should not merge before then.SCOPE.md§5 covers it.exit()takes the whole process with it.🤖 Generated with Claude Code