perf(mpi): remove the serial O(R·S²) protocol floor from HybridComm

.gitignore

@@ -176,3 +176,5 @@ benches/results/**
 
 # devcontainer files
 .devcontainer/devcontainer-lock.json
+# local HPC job scripts, tools and measurement logs (see hpc/deucalion/README.md)
+/hpc/

AGENTS.md

@@ -52,7 +52,12 @@ Key files:
   agree, or dispatch routes at a template the bindings never instantiated.
 - `CMakePresets.json`: the single source of truth for the supported C++ unit-test build/run entry
   points. The presets adopt the scikit-build-core trees generated by `uv sync`; regenerate the tree
-  with `uv sync`, then use the matching `skbuild-*` preset to build or run CTest.
+  with `uv sync`, then use the matching `skbuild-*` preset to build or run CTest. `skbuild-tsan`
+  adopts the `monoprop_ENABLE_TSAN=ON` (ThreadSanitizer) tree — see
+  `docs/content/docs/building.mdx`. That option is opt-in, composes with any build type, and is the
+  only machine check on the hand-written memory orderings in
+  `cpp/monoprop/detail/mpi/PartitionBarrier.h`; `cpp/tests/tsan.supp` suppresses third-party
+  reports only, never anything under `cpp/monoprop/`.
 - `benches/report.py` and `benches/bmf.py`: the two renderers of a benchmark run's artifacts —
   `REPORT.md` for humans, Bencher Metric Format JSON for the `bench_main.yml` continuous-benchmarking
   workflow. Benchmark names are Bencher's history key, so renaming or moving a `bench_*` test
@@ -73,6 +78,21 @@ Key files:
   for the mutating/collecting paths, which run on the partitions' own pinned masters; `sum_partitions_`,
   `fold_partitions_`, `first_partition_` for reads off quiescent partitions) rather than hand-rolling a
   `run_on_all` loop — the declarations record which helper is legal where.
+- **Placement must not divide an already-divided machine.** `enumerate_physical_cores` reports only cores
+  inside the calling thread's affinity mask, so when a launcher has given each co-located rank its own
+  disjoint slice (`srun --cpu-bind=cores`), the slice *is* the rank's share. Passing the node-wide
+  ranks-per-node through as `group_count` then asks for `group_count × n` cores out of a list that only
+  held `n`, `placement_order` correctly refuses, and every rank silently runs unpinned — which also costs
+  the two-level barrier its domains, because `cpuset_domains` derives them from the placement. Measured on
+  Deucalion at 8 ranks × 16 partitions: 437 µs/sync against 15.5 µs/sync placed. `PartitionGroup` therefore
+  allgathers the masks over its node-local communicator and `classify_node_mask` **measures** disjointness;
+  a `NodeMask::PerRank` result collapses `group_count` to 1. Mask *width* cannot substitute for this — "8
+  ranks holding 16 cores each" and "8 ranks sharing one 16-core mask" both leave a rank seeing 16 of 128,
+  and they need opposite placement. Collapse in the wrong direction and every co-located rank pins to the
+  *same* cores, so `Shared` is the default and the safe error. This regressed once already, when topology
+  discovery was rewritten onto hwloc, because the guard lives in the placement policy rather than in
+  discovery: any rework of that layer must re-check it. `cpu_topology_policy_per_rank_slice_starves_without_collapse`
+  pins the mechanism without needing live hardware.
 
 
 ### Environment Management
@@ -110,6 +130,29 @@ mp = MajoranaPropagator(operator, initial_state, cutoff=4)
 - Fixture msgpack schema is documented in `tests/data/README.md`
 - Tests validate against exact solutions for small systems
 - Heavy use of `@parametrize_with_cases` decorators
+- **pytest's default capture is fd-level**, so it hides C++ diagnostics: it replaces fd 2 for each
+  test and discards the buffer when the test passes, while the engine writes straight to fd 2 (the
+  `COMMPROF` line from `monoprop_COMM_PROFILE` comes out of a transport destructor). Reading those
+  under pytest requires `-s`; the `just bench` recipes pass it for this reason.
+- **Assert the count of what an instrument should emit, at the point of collection.** "The two arms
+  measured the same" and "the instrument never fired" are otherwise the same observation, and that
+  reads as a result rather than a failure. For the same reason, never diagnose by comparing two
+  zeros — get a positive control that is known to emit before concluding anything from silence.
+- **A slow CTest run on an MPI build is `MPI_Init`, not slow tests.** CTest runs each Boost case as
+  its own process, so each pays a full `MPI_Init`, which initialises every fabric device present
+  whether or not the process will send a message — 8.8 s against 0.2 s of user CPU on a login node
+  with 8 HCAs. The tell is wall time with no CPU behind it. `monoprop_ENABLE_MPI` builds default to
+  `monoprop_TEST_EXCLUDE_MPI_FABRIC=ON`, which skips fabric init for the per-case tests (8.8 s → 1.9 s;
+  suite 34 min → 5.4).
+- **That exclusion is scoped to the `serial` variants and must stay that way.** A per-case launch is
+  one process — world size 1, `*_World` cases skip themselves, everything else is `MPI_COMM_SELF` — so
+  no transport is used and the fabric can only cost startup time. The multi-rank variants exchange
+  real messages: `OMPI_MCA_pml=^ucx` makes a 2-rank run of the suite hang indefinitely on a case that
+  otherwise passes in 29 ms, on `main` as well, so it is component selection rather than engine code.
+  `discover_tests`' `SERIAL_ENVIRONMENT` argument exists for exactly this split.
+- Use exclusions (`^…`), never a positive component list — naming a component that must exist breaks
+  on the next machine, since `vader` became `sm` in Open MPI 5 and `OMPI_MCA_btl=self,vader` there
+  silently reduces to `self` alone.
 
 ## Key Dependencies & Integration
 
@@ -149,5 +192,15 @@ When changing behavior, APIs, build/test workflows, paths, or developer conventi
 - Check `build/*/compile_commands.json` for compilation flags
 - Use `rm -rf build` to clear environment-specific builds
 - Verify `monoprop_MAX_NUM_MODES` matches your use case (default: 250)
+- **`uv sync` does not relink the C++ test binary.** It builds in a temporary directory and installs
+  only the wheel, and `bin/monoprop_unit_tests.x` is not a wheel target — so an edited test can leave
+  a stale binary that passes, or that reports `no test cases matching filter` for a case you just
+  wrote. Compare the binary's mtime against the source's before trusting either outcome.
+- **The editable tree cannot be reconfigured in place**: its cache pins the build-isolation
+  interpreter scikit-build-core created and deleted, so `cmake --build build/editable/<type>` fails
+  regenerating `build.ninja`, and the `skbuild-*` presets inherit that because they only adopt the
+  tree. To iterate on C++ tests, configure a standalone tree with an explicit `-Dnanobind_DIR=…`
+  (nanobind is a build-isolation-only dependency, absent from `.venv`) — recipe in
+  `docs/content/docs/building.mdx`.
 
 This is a sophisticated scientific computing project requiring careful attention to template instantiation, build system configuration, and the C++/Python boundary.

CMakeLists.txt

@@ -98,6 +98,7 @@ message(
 message(STATUS "   User-appended          : ${EXTRA_CXXFLAGS}")
 
 message(STATUS "   MPI parallelization    : ${monoprop_ENABLE_MPI}")
+message(STATUS "   ThreadSanitizer        : ${monoprop_ENABLE_TSAN}")
 message(STATUS "   Wide term index        : ${monoprop_WIDE_TERM_INDEX}")
 message(STATUS "   Max simulable modes    : ${monoprop_MAX_NUM_MODES}")
 message(STATUS "   C++ unit tests         : ${monoprop_ENABLE_CXX_UNIT_TESTS}")

CMakePresets.json

@@ -14,6 +14,13 @@
             "description": "Adopt the Debug build tree generated by 'uv sync --config-settings=cmake.build-type=Debug' (the 'Install in Debug mode' task). Do NOT reconfigure this preset from CMake Tools; regenerate the tree with uv/scikit-build-core, then use this preset only to build and run the C++ CTest suite.",
             "binaryDir": "${sourceDir}/build/editable/Debug",
             "generator": "Ninja"
+        },
+        {
+            "name": "skbuild-tsan",
+            "displayName": "scikit-build-core (ThreadSanitizer, adopt existing tree)",
+            "description": "Adopt the ThreadSanitizer tree generated by 'uv sync --all-extras --reinstall-package monoprop --no-cache --config-settings-package=\"monoprop:cmake.build-type=RelWithDebInfo\" --config-settings-package=\"monoprop:cmake.define.monoprop_ENABLE_TSAN=ON\"'. RelWithDebInfo, not Debug: the partition threading layer is only worth auditing at the optimization level it ships with, and -g is added by monoprop_ENABLE_TSAN regardless. Do NOT reconfigure this preset from CMake Tools; regenerate the tree with uv/scikit-build-core, then use this preset only to build and run the C++ CTest suite.",
+            "binaryDir": "${sourceDir}/build/editable/RelWithDebInfo",
+            "generator": "Ninja"
         }
     ],
     "buildPresets": [
@@ -36,6 +43,16 @@
                 "monoprop_unit_tests.x"
             ],
             "jobs": 4
+        },
+        {
+            "name": "skbuild-tsan",
+            "displayName": "scikit-build-core (ThreadSanitizer) - Build C++ unit tests",
+            "description": "Build the C++ unit test executable in the ThreadSanitizer tree. Requires the tree to already exist (see the skbuild-tsan configure preset).",
+            "configurePreset": "skbuild-tsan",
+            "targets": [
+                "monoprop_unit_tests.x"
+            ],
+            "jobs": 4
         }
     ],
     "testPresets": [
@@ -58,6 +75,16 @@
                 "outputOnFailure": true
             },
             "inheritConfigureEnvironment": true
+        },
+        {
+            "name": "skbuild-tsan",
+            "displayName": "scikit-build-core (ThreadSanitizer) - Run C++ unit tests",
+            "description": "Run the C++ CTest suite under ThreadSanitizer. cpp/tests/CMakeLists.txt attaches TSAN_OPTIONS (including cpp/tests/tsan.supp) to every discovered test, so no extra environment is needed. Expect a large slowdown; a race makes the owning test exit non-zero.",
+            "configurePreset": "skbuild-tsan",
+            "output": {
+                "outputOnFailure": true
+            },
+            "inheritConfigureEnvironment": true
         }
     ]
 }

README.md

@@ -107,6 +107,10 @@ uv sync --all-extras -v
 ctest --test-dir build/editable/Release
 ```
 
+Note that `uv sync` builds in a temporary directory and installs only the wheel, so
+it does **not** relink the test binary: after editing a C++ test, compare the
+binary's mtime against the source's rather than trusting the result.
+
 Full instructions — prerequisites, MPI options, and running the example
 executable — are in the [building guide](https://docs.monoprop.algorithmiq.tech/building).
 In particular, from-source builds require `hwloc` and `pkg-config` so CMake can
@@ -120,6 +124,16 @@ just test-mpi                          # Python + C++ tests under MPI
 just test-wide                         # Python + C++ unit tests with a 64-bit TermIndex
 ```
 
+On an MPI build, CTest runs each case as its own process and so pays a full
+`MPI_Init` per case — which initialises every fabric device present even though a
+single-process test never sends a message. `monoprop_TEST_EXCLUDE_MPI_FABRIC` (on by
+default) skips that for the per-case tests only, leaving the multi-rank variants on the
+full component set; see the [testing guide](https://docs.monoprop.algorithmiq.tech/testing).
+
+To audit the partition threading layer for data races, build with
+`cmake.define.monoprop_ENABLE_TSAN=ON` and run CTest against that tree; see the
+[building guide](https://docs.monoprop.algorithmiq.tech/building).
+
 See the [testing guide](https://docs.monoprop.algorithmiq.tech/testing)
 for the with/without-MPI details and the rank matrix.
 

cmake/compiler_flags/CXXFlags.cmake

@@ -24,9 +24,14 @@
 #   list, *e.g.* to override previous compiler flags without touching the CMake
 #   scripts.  Default is empty.
 #
+# Opt-in sanitizer flags are appended to ``CMAKE_CXX_FLAGS`` by
+# ``Sanitizers.cmake``; see that file for why they do not go into
+# ``monoprop_CXX_FLAGS``.
+#
 # Variables used::
 #
 #   monoprop_ENABLE_ARCH_FLAGS
+#   monoprop_ENABLE_TSAN
 #   EXTRA_CXXFLAGS
 #
 # Variables modified::
@@ -178,3 +183,5 @@ set(monoprop_CXX_FLAGS "")
 include(${CMAKE_CURRENT_LIST_DIR}/GNU.CXX.cmake)
 include(${CMAKE_CURRENT_LIST_DIR}/Intel.CXX.cmake)
 include(${CMAKE_CURRENT_LIST_DIR}/Clang.CXX.cmake)
+# Last, so the sanitizer flags cannot be clobbered by a per-compiler `set()`.
+include(${CMAKE_CURRENT_LIST_DIR}/Sanitizers.cmake)

cmake/compiler_flags/Sanitizers.cmake

@@ -0,0 +1,54 @@
+#.rst:
+#
+# Opt-in sanitizer configuration.
+#
+# Sanitizers are wired in here rather than as a build type because they are
+# orthogonal to optimization level: the partition threading layer is only worth
+# auditing at the optimization level it ships with, so
+# ``monoprop_ENABLE_TSAN=ON`` composes with any ``CMAKE_BUILD_TYPE`` instead of
+# replacing it.
+#
+# The flags are appended to ``CMAKE_CXX_FLAGS`` (not to ``monoprop_CXX_FLAGS``,
+# which reaches ``target_compile_options`` only) because ``-fsanitize=thread``
+# must appear on the *link* line as well as every compile line; CMake passes
+# ``CMAKE_CXX_FLAGS`` to both. The explicit linker-flag appends below cover
+# link steps that a toolchain file may drive without ``CMAKE_CXX_FLAGS``.
+#
+# Variables used::
+#
+#   monoprop_ENABLE_TSAN
+#
+# Variables modified::
+#
+#   CMAKE_CXX_FLAGS
+#   CMAKE_EXE_LINKER_FLAGS
+#   CMAKE_SHARED_LINKER_FLAGS
+#   CMAKE_MODULE_LINKER_FLAGS
+
+option(
+  monoprop_ENABLE_TSAN
+  "Build with ThreadSanitizer (-fsanitize=thread); opt-in, never on by default"
+  OFF
+)
+
+if(monoprop_ENABLE_TSAN)
+  if(NOT CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang")
+    message(
+      FATAL_ERROR
+      "monoprop_ENABLE_TSAN requires GCC or Clang. Detected: ${CMAKE_CXX_COMPILER_ID}"
+    )
+  endif()
+
+  # -g is forced even in Release: without line tables a TSan report names only
+  # addresses, and a report you cannot attribute to a memory ordering is not an
+  # audit. -fno-omit-frame-pointer is already in monoprop_CXX_FLAGS, but repeat
+  # it here so the setting survives an EXTRA_CXXFLAGS override of that list.
+  set(
+    monoprop_TSAN_FLAGS
+    "-fsanitize=thread -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g"
+  )
+  string(APPEND CMAKE_CXX_FLAGS " ${monoprop_TSAN_FLAGS}")
+  string(APPEND CMAKE_EXE_LINKER_FLAGS " -fsanitize=thread")
+  string(APPEND CMAKE_SHARED_LINKER_FLAGS " -fsanitize=thread")
+  string(APPEND CMAKE_MODULE_LINKER_FLAGS " -fsanitize=thread")
+endif()

cpp/monoprop/detail/EnvConfig.h

@@ -24,6 +24,14 @@
 //   monoprop_NUM_THREADS    positive int (1..1e6), else ignored                → num_threads
 //   monoprop_PARTITION_PINNING  bool, default ON; 0/false disables per-core pinning → partition_pinning
 //   monoprop_PARTITIONS         int N | "auto" | "off"; parsed where it is used (resolve_partition_count_)
+//   monoprop_COMM_PROFILE       bool, default OFF; per-partition collective accounting to stderr → comm_profile
+//   monoprop_SPIN_BUDGET_US     positive int us, default kDefaultSpinBudgetUs; barrier on-core spin before
+//                               yielding. Exists to be swept against a real workload → spin_budget_us
+//   monoprop_BARRIER_GROUPING   bool, default ON; 0/false forces the flat barrier while LEAVING PINNING ON
+//                               → barrier_grouping. Without this the two-level barrier cannot be measured:
+//                               its domains come from the cpusets, so turning pinning off to get a flat
+//                               barrier also unpins, and every before/after confounds the two. Exists so
+//                               "grouped vs flat, both pinned" is a run rather than an argument.
 
 namespace monoprop::config {
 
@@ -57,6 +65,9 @@ inline auto parse_positive_int(const char *text) -> std::optional<int> {
 struct Settings {
     std::optional<int> num_threads;
     bool partition_pinning = true;
+    bool comm_profile = false;
+    std::optional<int> spin_budget_us; // nullopt ⇒ the barrier's own default
+    bool barrier_grouping = true;
 };
 
 // Parse the environment once; the Settings are cached and shared across TUs.
@@ -65,6 +76,9 @@ inline auto get() -> const Settings & {
         Settings s;
         s.num_threads = detail::parse_positive_int(std::getenv("monoprop_NUM_THREADS"));
         s.partition_pinning = detail::parse_flag(std::getenv("monoprop_PARTITION_PINNING"), true);
+        s.comm_profile = detail::parse_flag(std::getenv("monoprop_COMM_PROFILE"), false);
+        s.spin_budget_us = detail::parse_positive_int(std::getenv("monoprop_SPIN_BUDGET_US"));
+        s.barrier_grouping = detail::parse_flag(std::getenv("monoprop_BARRIER_GROUPING"), true);
         return s;
     }();
     return settings;

cpp/monoprop/detail/evolution/layer_build/Engine.h

@@ -382,7 +382,16 @@ struct LayerBuildEngine {
         auto resp = resolve_incoming<NumModes>(inc_q, local_op, R, is_leader_pass, matched, combined_size, sink);
         std::vector<int> resp_recv = response_recv_counts();
         std::vector<std::vector<typename Sink::Response>> inc_r;
-        mpi::begin_alltoallv(resp, comm, /*skip_self=*/false, &resp_recv).wait_into(inc_r);
+        // The answers travel the query exchange's legs backwards, one per query, so the hybrid transport
+        // reuses that exchange's offset tables; nothing may collectively intervene between the two calls.
+        // Sink::kStride is the query leg's words per query, the ratio between the two legs' counts.
+        mpi::begin_alltoallv(resp,
+                             comm,
+                             /*skip_self=*/false,
+                             &resp_recv,
+                             /*reverse_of_previous=*/true,
+                             /*forward_stride=*/static_cast<int>(Sink::kStride))
+            .wait_into(inc_r);
         process_responses<NumModes>(inc_r, src_idx_r, queries_r, R, my_rank, sink);
     }
 

cpp/monoprop/detail/mpi/CMakeLists.txt

@@ -6,6 +6,7 @@ target_sources(
       FILES
         "CheckedCount.h"
         "Comm.h"
+        "CommProfile.h"
         "CpuRelax.h"
         "Exchange.h"
         "HybridComm.h"