Continuous benchmarking with google benchmark and bencher.dev

.gitignore

@@ -26,3 +26,6 @@ build*/
 
 # lsp info
 compile_commands.json
+
+# local agent scratch files (may contain secrets)
+.scratch/**/*.env

.scratch/continuous-benchmarking/issues/01-cmake-google-benchmark.md

@@ -0,0 +1,12 @@
+# 01: Add `IPPL_ENABLE_BENCHMARK` CMake option and fetch Google Benchmark
+
+**What to build:** A CMake-controlled path that, when enabled, fetches Google Benchmark v1.9.4 and makes it available to targets. A minimal smoke benchmark executable proves the whole path works: configure with `IPPL_ENABLE_BENCHMARK=ON`, build, run, and see Google Benchmark JSON output.
+
+**Blocked by:** None (can start immediately)
+
+**Status:** resolved
+
+- [ ] `option(IPPL_ENABLE_BENCHMARK ... OFF)` exists in root `CMakeLists.txt`.
+- [ ] Google Benchmark v1.9.4 is fetched via `FetchContent` when the option is ON.
+- [ ] A minimal `BenchmarkSmoke` executable builds and runs under `ctest`/manually.
+- [ ] When the option is OFF, no benchmark code is compiled or linked.

.scratch/continuous-benchmarking/issues/02-add-ippl-benchmark-helper.md

@@ -0,0 +1,12 @@
+# 02: Implement `add_ippl_benchmark()` CMake helper
+
+**What to build:** A reusable CMake function that creates a benchmark target, accepts `REPORT_TIMERS`/`IGNORE_TIMERS`, and generates a compile-time timer-whitelist header. The smoke benchmark from ticket 01 is converted to use this helper.
+
+**Blocked by:** 01
+
+**Status:** resolved
+
+- [ ] `cmake/AddIpplBenchmark.cmake` defines `add_ippl_benchmark(<name> SOURCES ... [REPORT_TIMERS ...] [IGNORE_TIMERS ...])`.
+- [ ] The function generates `<name>_timers.h` in the build tree containing a `constexpr std::array<std::string_view, N> reportedTimers`.
+- [ ] Supplying both `REPORT_TIMERS` and `IGNORE_TIMERS` is a CMake configure error.
+- [ ] The smoke benchmark uses the helper and compiles with a whitelist.

.scratch/continuous-benchmarking/issues/03-bmf-json-writer.md

@@ -0,0 +1,12 @@
+# 03: Implement BMF JSON writer for `IpplTimings`
+
+**What to build:** A small utility that reads the existing `IpplTimings` measurements, filters them against the generated whitelist, and writes a BMF JSON file. The smoke benchmark is updated to emit BMF JSON alongside Google Benchmark JSON.
+
+**Blocked by:** 02
+
+**Status:** resolved
+
+- [ ] A header-only (or small library) BMF writer computes `latency` (mean/min/max) and `count` per timer.
+- [ ] Only whitelisted timer names appear in the BMF output.
+- [ ] The smoke benchmark produces a valid BMF JSON file.
+- [ ] No changes to `IpplTimings` internals.

.scratch/continuous-benchmarking/issues/04-landau-damping-bench.md

@@ -0,0 +1,13 @@
+# 04: Implement `LandauDampingBench`
+
+**What to build:** The first real benchmark executable for the Alpine module, instrumenting `LandauDampingManager`. It uses `add_ippl_benchmark()` and the BMF writer, runs `pre_run()` once as warmup, then benchmarks `advance()` per Google Benchmark iteration.
+
+**Blocked by:** 03
+
+**Status:** resolved
+
+- [ ] `LandauDampingBench` builds with `IPPL_ENABLE_BENCHMARK=ON`.
+- [ ] Running it produces Google Benchmark JSON and BMF JSON.
+- [ ] Only rank 0 emits JSON artifacts.
+- [ ] `IpplTimings::resetAllTimers()` is called before the measured loop.
+- [ ] The benchmark still prints `IpplTimings` output for local debugging.

.scratch/continuous-benchmarking/issues/05-ci-bencher-integration.md

@@ -0,0 +1,12 @@
+# 05: Add CI jobs and bencher.dev submission
+
+**What to build:** GitLab CI jobs on Eiger OpenMP, Daint GH200 CUDA, and MI300 ROCm that build with `IPPL_ENABLE_BENCHMARK=ON`, run `LandauDampingBench`, and submit both Google Benchmark JSON and BMF JSON to bencher.dev. Jobs remain non-blocking.
+
+**Blocked by:** 01, 04
+
+**Status:** resolved
+
+- [x] One benchmark job exists per platform under `ci/cscs/`.
+- [x] Each job runs `bencher run --adapter cpp_google` and `bencher run --adapter json --file <bmf>`.
+- [x] Jobs are gated by `IPPL_RUN_BENCHMARKS` and do not fail the pipeline on alerts.
+- [x] Bencher testbed names match the spec (`eiger-openmp-1node`, `daint-gh200-1node`, `mi300-rocm-1node`).

.scratch/continuous-benchmarking/spec.md

@@ -0,0 +1,78 @@
+# Continuous Benchmarking for IPPL/OpalX
+
+Status: ready-for-agent
+
+## Problem Statement
+
+IPPL/OpalX has no systematic, automated way to detect performance regressions in its solver and particle kernels before code merges. Kernel runtimes are currently observed ad-hoc through `IpplTimings` console output and CSV dumps. Without a continuous benchmarking pipeline, regressions are only noticed late, are hard to attribute to a specific change, and require manual reproduction across the three CSCS target platforms (Eiger OpenMP, Daint GH200 CUDA, MI300 ROCm).
+
+## Solution
+
+Introduce an optional `IPPL_ENABLE_BENCHMARK` CMake path that fetches Google Benchmark and builds dedicated benchmark executables alongside existing demos/tests. The first benchmark will instrument `LandauDampingManager` via the existing `IpplTimings` framework, emit both Google Benchmark JSON and a BMF JSON file, and submit both to bencher.dev from per-platform GitLab CI jobs. The pipeline starts non-blocking and flips to blocking once thresholds are trustworthy.
+
+## User Stories
+
+1. As an IPPL developer, I want benchmark executables to live next to the demos they instrument, so that I can reuse the existing manager classes without modifying the demo binary.
+2. As a CI maintainer, I want a single CMake option (`IPPL_ENABLE_BENCHMARK`) to enable the benchmarking build, so that benchmark jobs are opt-in and do not affect normal builds.
+3. As a performance engineer, I want Google Benchmark fetched automatically via CMake `FetchContent`, so that no manual dependency installation is required on any platform.
+4. As a developer adding a new benchmark, I want an `add_ippl_benchmark()` CMake helper that accepts `REPORT_TIMERS` and `IGNORE_TIMERS`, so that the set of `IpplTimings` phases submitted to bencher.dev is version-controlled and visible next to the target.
+5. As a benchmark author, I want the helper to generate a compile-time timer whitelist header, so that the binary only emits the timers I care about with no runtime file I/O or parsing.
+6. As a CI runner, I want the benchmark executable to write Google Benchmark JSON and a BMF JSON file, so that bencher.dev can ingest headline latency plus per-phase breakdowns.
+7. As a platform maintainer, I want one GitLab CI job per CSCS platform (OpenMP, CUDA, ROCm), so that regressions are caught on every target architecture.
+8. As a project lead, I want the CI jobs to be non-blocking initially, so that we can seed baselines and tune statistical thresholds before making them merge-blocking.
+9. As a reviewer, I want benchmark results to appear on PRs via bencher.dev, so that performance impact is visible alongside code changes.
+10. As a developer debugging a regression, I want the benchmark to still print `IpplTimings` output and CSV dumps, so that I can investigate locally with familiar tooling.
+11. As a maintainer of a non-CSCS fork, I want the benchmarking code to be self-contained in IPPL and not require bencher.dev credentials to build or run, so that the feature is portable.
+12. As a future benchmark author, I want the first benchmark (`LandauDampingBench`) to be a clear template, so that adding `PenningTrapBench` or `BumponTailInstabilityBench` is mostly copy-paste.
+13. As a CI operator, I want only rank 0 to emit JSON output, so that multi-rank runs do not produce corrupt or duplicate benchmark artifacts.
+14. As a performance engineer, I want benchmark iterations to call `IpplTimings::resetAllTimers()` before the measured loop, so that cumulative timer state does not leak across Google Benchmark iterations.
+15. As a project lead, I want the bencher.dev project name and testbed names documented in the spec, so that CI configuration is reproducible.
+
+## Implementation Decisions
+
+- **CMake option**: Add `option(IPPL_ENABLE_BENCHMARK "Enable Google Benchmark-based benchmarks" OFF)` in the root `CMakeLists.txt`.
+- **Dependency fetching**: Google Benchmark v1.9.4 is fetched via `FetchContent_Declare` in `cmake/Dependencies.cmake`, mirroring the existing GTest fetch. Benchmark tests and installation are disabled.
+- **Benchmark helper**: A new CMake module `cmake/AddIpplBenchmark.cmake` defines `add_ippl_benchmark(<name> SOURCES ... [REPORT_TIMERS ...] [IGNORE_TIMERS ...])`.
+  - If neither `REPORT_TIMERS` nor `IGNORE_TIMERS` is given, all `IpplTimings` phases are reportable.
+  - `REPORT_TIMERS` is a whitelist; `IGNORE_TIMERS` is a blacklist; supplying both is an error.
+  - The helper generates a header `<name>_timers.h` in the build tree containing a `constexpr std::array<std::string_view, N> reportedTimers`.
+- **BMF writer**: A small header-only utility (e.g. `src/Utility/BenchmarkMetrics.h`) collects timer statistics from `IpplTimings` and writes BMF JSON. It filters emitted timers against the generated whitelist.
+- **First benchmark**: `demos/alpine/LandauDampingBench.cpp` builds a `LandauDampingManager<>` in `SetUp()`, runs `pre_run()` once as warmup, resets timers, then benchmarks `manager->advance()` per Google Benchmark iteration. `TearDown()` prints timers and writes BMF JSON.
+- **JSON emission**: Only rank 0 uses the real `benchmark::BenchmarkReporter`; other ranks use `benchmark::NullReporter`.
+- **CI integration**: Extend `ci/cscs/` with a benchmark stage/job per platform. Jobs run only when `IPPL_RUN_BENCHMARKS` is `"true"` and execute two `bencher run` submissions:
+  1. `--adapter cpp_google` against the Google Benchmark JSON.
+  2. `--adapter json --file timings.bmf.json` for per-phase metrics.
+  - The variable defaults to `"false"` in `ci/cscs/common.yml` so normal build/test pipelines are unaffected.
+  - Set it to `"true"` via the GitLab pipeline trigger variables or the CSCS CI admin console to enable benchmarks.
+- **Bencher project**: Target project is `ippl/OpalX` (name TBD with team); testbeds are `eiger-openmp-1node`, `daint-gh200-1node`, `mi300-rocm-1node`.
+- **CI blocking policy**: Jobs set `IPPL_BENCH_ERROR_ON_ALERT=false` (or equivalent non-blocking configuration) until baselines are seeded and thresholds tuned.
+- **Warmup semantics**: `SetUp()` calls `pre_run()` once and `IpplTimings::resetAllTimers()` so that initialization and solver warmup are not included in the measured `advance()` loop.
+- **Benchmark arguments**: Problem size defaults are configurable via Google Benchmark `->Args(...)` or CLI args; the CI default is left as a spec decision (initially `16 16 16` with `10000000` particles and a small number of steps to keep iteration time reasonable).
+
+## Testing Decisions
+
+- **Seam 1 — CMake helper**: Verify that `add_ippl_benchmark(LandauDampingBench ... REPORT_TIMERS solve pushVelocity ...)` configures successfully and that the generated `<name>_timers.h` contains exactly the requested timer names. This is tested by a configure-and-build check, not a runtime unit test.
+- **Seam 2 — BMF writer**: If the BMF writer is extracted into a small pure function (string list + timer statistics → JSON string), it can be covered by a lightweight unit test that asserts correct JSON shape and whitelist filtering. Otherwise it is tested indirectly via the benchmark executable.
+- **Seam 3 — First benchmark executable**: Run `LandauDampingBench` locally (single-rank and multi-rank) and assert that:
+  - It exits 0.
+  - Google Benchmark JSON is produced and contains the expected benchmark name.
+  - BMF JSON is produced and contains only whitelisted timer names.
+  - Only rank 0 writes JSON artifacts.
+- **Seam 4 — CI integration**: A dry-run on one platform (e.g. OpenMP) confirms that `bencher run` is invoked with the correct adapters and that the job is non-blocking.
+- **Regression policy**: No runtime assertions on performance numbers are added to the test suite; bencher.dev owns regression detection.
+
+## Out of Scope
+
+- Modifying existing demo/test binaries to act as benchmarks (the plan explicitly creates separate benchmark executables).
+- Large-scale (>1 node) benchmarks for this first effort; the testbeds remain single-node.
+- OpalX-specific benchmark integration or regression-test coupling; this effort is scoped to IPPL demos only.
+- Bencher.dev dashboard configuration beyond project/testbed naming; the skill assumes the project exists or will be created separately.
+- Runtime JSON config for the timer whitelist; the generated-header approach is chosen for simplicity.
+- Changing `IpplTimings` internals; only the existing public interface is consumed.
+
+## Further Notes
+
+- The generated timer whitelist header should be placed in the build tree next to the benchmark binary (e.g. `${CMAKE_CURRENT_BINARY_DIR}/<name>_timers.h`) and included privately by the benchmark source.
+- Keep benchmark binaries out of the default `IPPL_ENABLE_TESTS` path so that `ctest` does not try to run Google Benchmark executables as Catch/GTest-style tests.
+- The LandauDamping integration test in `demos/alpine/CMakeLists.txt` already passes arguments `"16" "16" "16" "10000000" "25" "FFT" "0.01" "LeapFrog" "--overallocate" "2.0" "--info" "10"`; these are a reasonable starting point for the benchmark, though step count and overallocate factor may be tuned for stable iteration time.
+- When `IPPL_ENABLE_BENCHMARK` is OFF, no Google Benchmark code or headers should be compiled or linked.

CMakeLists.txt

@@ -58,6 +58,7 @@ option(IPPL_MARK_FAILING_TESTS
        "Prefix names of tests that are known to fail with 'known_fail' for filtering with ctest"
        OFF)
 option(IPPL_ENABLE_SCRIPTS "Generate job script templates for some benchmarks/tests" OFF)
+option(IPPL_ENABLE_BENCHMARK "Enable Google Benchmark-based benchmarks" OFF)
 
 
 option(IPPL_ENABLE_CATALYST "Enable ParaView Catalyst" OFF)
@@ -142,3 +143,7 @@ add_subdirectory(demos)
 if(IPPL_ENABLE_SCRIPTS)
   add_subdirectory(scripts)
 endif()
+
+if(IPPL_ENABLE_BENCHMARK)
+  add_subdirectory(benchmarks)
+endif()

benchmarks/CMakeLists.txt

@@ -0,0 +1,10 @@
+# ------------------------------------------------------------------------------
+# benchmarks/CMakeLists.txt
+#
+# Optional benchmark executables built when IPPL_ENABLE_BENCHMARK is ON.
+# ------------------------------------------------------------------------------
+message(STATUS "Configuring benchmarks/")
+
+include(AddIpplBenchmark)
+
+add_ippl_benchmark(BenchmarkSmoke SOURCES SmokeBench.cpp REPORT_TIMERS smoke)

benchmarks/SmokeBench.cpp

@@ -0,0 +1,58 @@
+// ------------------------------------------------------------------------------
+// Smoke benchmark for IPPL's Google Benchmark integration.
+//
+// This is a minimal executable used to verify that the CMake FetchContent setup,
+// compile flags, benchmark::benchmark linking, add_ippl_benchmark() timer
+// whitelist generation, and BenchmarkMetrics BMF writing all work correctly.
+// ------------------------------------------------------------------------------
+
+#include "Ippl.h"
+
+#include <benchmark/benchmark.h>
+
+#include "Utility/BenchmarkMetrics.h"
+#include "Utility/IpplTimings.h"
+
+#include "BenchmarkSmoke_timers.h"
+
+class SmokeFixture : public benchmark::Fixture {
+public:
+    void SetUp(const ::benchmark::State&) override { IpplTimings::resetAllTimers(); }
+
+    void TearDown(const ::benchmark::State&) override {
+        ippl::benchmark::writeBMF("BenchmarkSmoke.bmf.json", reportedTimers);
+    }
+};
+
+BENCHMARK_DEFINE_F(SmokeFixture, BM_Smoke)(benchmark::State& state) {
+    static IpplTimings::TimerRef smokeTimer = IpplTimings::getTimer("smoke");
+
+    for (auto _ : state) {
+        IpplTimings::startTimer(smokeTimer);
+
+        int sum = 0;
+        for (int i = 0; i < 100; ++i) {
+            sum += i;
+        }
+        benchmark::DoNotOptimize(sum);
+
+        IpplTimings::stopTimer(smokeTimer);
+    }
+}
+
+BENCHMARK_REGISTER_F(SmokeFixture, BM_Smoke);
+
+int main(int argc, char** argv) {
+    ippl::initialize(argc, argv);
+
+    ::benchmark::Initialize(&argc, argv);
+    if (::benchmark::ReportUnrecognizedArguments(argc, argv)) {
+        ippl::finalize();
+        return 1;
+    }
+    ::benchmark::RunSpecifiedBenchmarks();
+    ::benchmark::Shutdown();
+
+    ippl::finalize();
+    return 0;
+}

ci/cscs/benchmark/benchmark-cuda.yml

@@ -0,0 +1,44 @@
+# -----------------------------------------------------------------------------
+
+# CUDA benchmark on Daint GH200
+# -----------------------------------------------------------------------------
+ippl-benchmark-cuda:
+  extends: .uenv-runner-daint-gh200
+  stage: benchmark
+  image: "/capstor/store/cscs/cscs/public/uenvs/opal-x-gh200-mpich-gcc-2026-08-04-cuda-13.1.1.squashfs"
+  needs: ["ippl-build-cuda13-sm90-release", "ippl-test-cuda13-sm90-release-4-ranks"]
+  variables:
+    SLURM_JOB_NUM_NODES: 1
+    SLURM_TIMELIMIT: "0:15:00"
+    BENCHMARK_MIN_TIME: "1.0s"
+    BENCHER_PROJECT: "ippl-opalx"
+    CUDA_UENV: "/capstor/store/cscs/cscs/public/uenvs/opal-x-gh200-mpich-gcc-2026-08-04-cuda-13.1.1.squashfs"
+    WITH_UENV_VIEW: "default"
+    WRAPPER: "/user-environment/wrapper-mpi.sh"
+    SCRATCH: "/capstor/scratch/cscs/biddisco"
+    SRUN_FLAGS: "--uenv=${CUDA_UENV} --view=${WITH_UENV_VIEW} --repo=$SCRATCH/.uenv-images-ci-daint ${WRAPPER}"
+    BUILD_DIR: "build-$CI_COMMIT_SHORT_SHA-release"
+    SLURM_NTASKS: 4
+    SLURM_CPUS_PER_TASK: 32
+    BENCHER_TESTBED: "daint-gh200-1node"
+  before_script:
+    - echo "CI_PROJECT_URL=$CI_PROJECT_URL"
+    - echo "CI_COMMIT_SHORT_SHA=$CI_COMMIT_SHORT_SHA"
+    - echo "IPPL_RUN_BENCHMARKS=$IPPL_RUN_BENCHMARKS"
+    - export BUILD_PATH=$(pwd)/$BUILD_DIR
+    - pwd
+    - env | sort | grep CI
+  script:
+    - ./ci/scripts/install_bencher.sh
+    - export PATH=$HOME/.cargo/bin:$PATH
+    - ./ci/scripts/run_benchmark.sh
+  artifacts:
+    paths:
+      - LandauDampingBench.json
+      - LandauDampingBench.bmf.json
+    expire_in: 1 week
+  allow_failure: true
+  rules:
+    - if: $IPPL_RUN_BENCHMARKS == "true"
+      when: always
+    - when: never

ci/cscs/benchmark/benchmark-openmp.yml

@@ -0,0 +1,42 @@
+# -----------------------------------------------------------------------------
+# OpenMP benchmark on Eiger
+# -----------------------------------------------------------------------------
+ippl-benchmark-openmp:
+  extends: .uenv-runner-eiger-zen2
+  stage: benchmark
+  image: "prgenv-gnu/25.6:v2"
+  needs: ["ippl-build-openmp-release", "ippl-test-openmp-release-4-ranks"]
+  variables:
+    SLURM_JOB_NUM_NODES: 1
+    SLURM_TIMELIMIT: "0:15:00"
+    BENCHMARK_MIN_TIME: "1.0s"
+    BENCHER_PROJECT: "ippl-opalx"
+    EIGER_UENV: "prgenv-gnu/25.6:v2"
+    WITH_UENV_VIEW: "default"
+    SCRATCH: "/capstor/scratch/cscs/biddisco"
+    SRUN_FLAGS: "--uenv=${EIGER_UENV} --view=${WITH_UENV_VIEW} --repo=$SCRATCH/.uenv-images-ci-eiger"
+    BUILD_DIR: "build-$CI_COMMIT_SHORT_SHA-release"
+    SLURM_NTASKS: 4
+    SLURM_CPUS_PER_TASK: 32
+    BENCHER_TESTBED: "eiger-openmp-1node"
+  before_script:
+    - echo "CI_PROJECT_URL=$CI_PROJECT_URL"
+    - echo "CI_COMMIT_SHORT_SHA=$CI_COMMIT_SHORT_SHA"
+    - echo "IPPL_RUN_BENCHMARKS=$IPPL_RUN_BENCHMARKS"
+    - export BUILD_PATH=$(pwd)/$BUILD_DIR
+    - pwd
+    - env | sort | grep CI
+  script:
+    - ./ci/scripts/install_bencher.sh
+    - export PATH=$HOME/.cargo/bin:$PATH
+    - ./ci/scripts/run_benchmark.sh
+  artifacts:
+    paths:
+      - LandauDampingBench.json
+      - LandauDampingBench.bmf.json
+    expire_in: 1 week
+  allow_failure: true
+  rules:
+    - if: $IPPL_RUN_BENCHMARKS == "true"
+      when: always
+    - when: never