Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,11 @@ The root `pyproject.toml` declares a `[tool.uv.workspace]`, so one `uv.lock` cov
wheels, so it is a standalone uv project with its own `uv.lock`. Run it with
`cd packages/bench-third-party && uv sync`, never from the root environment.
- `benches/` — monoprop's own benchmark suite (`conftest.py`, `bench_*.py`, `results/`). It stays in
the repository and imports the tools package. Benchmark names are Bencher's history key, so they
must not move with a library release; that is why the suite is not in `monoprop-bench-tools`.
the repository and imports the tools package. `bench_bindings.py` contains serial-only boundary
microbenchmarks; run it on a fixed host and Release build. Artifacts record Python, compiled
nanobind frontend, and runtime backend versions. Benchmark names are Bencher's history key, so
they must not move with a library release; that is why the suite is not in
`monoprop-bench-tools`.

Dependency groups follow from that split: `test` is monoprop's own suite only (cibuildwheel installs
it against a built wheel, so it must not reference a workspace member), `workspace-test` adds
Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,9 @@ and against [`MajoranaPropagation.jl`](https://github.com/SparqleSim/MajoranaPro
Head to our [benchmarks page](https://docs.monoprop.algorithmiq.tech/benchmarks) for more details.

Every commit on `main` also runs the internal benchmark suite, tracked over time
with [Bencher](https://bencher.dev/) to catch performance regressions.
with [Bencher](https://bencher.dev/) to catch propagation and Python/C++ binding
regressions. Reports record the Python, nanobind frontend, and runtime backend
versions for reproducible comparisons.

📖 **Full documentation:** <https://docs.monoprop.algorithmiq.tech>

Expand Down
15 changes: 13 additions & 2 deletions benches/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,27 @@ The `monoprop` repository includes a pytest suite measuring the **time** and
The suite is separated from the test suite and can be run with `just bench`.
See below for more detailed instructions.

See the the [Benchmarks](../docs/content/docs/benchmarks.mdx) section of the
See the [Benchmarks](../docs/content/docs/benchmarks.mdx) section of the
documentation for detailed instructions.

## What lives where

This directory holds only monoprop's own benchmarks — `conftest.py` (the fixtures
and the results schema), `bench_random.py`, `bench_models.py`, and `results/`.
and the results schema), `bench_random.py`, `bench_models.py`, `bench_bindings.py`,
and `results/`. The binding microbenchmarks are serial-only and isolate Python/C++
call and return-conversion costs from propagation work. Run them directly with:

```bash
uv run pytest benches/bench_bindings.py --benchmark-only
```

Benchmark names are the key [Bencher](https://bencher.dev/) stores history under,
so they stay here rather than moving with a library release.

Each run records its Python version, the nanobind version compiled into the
extension, and the installed `nanobind-backend` version. Keep the host, Release
build, and benchmark command fixed when comparing binding revisions.

Everything reusable is in the `monoprop-bench-tools` package
([`../packages/monoprop-bench-tools`](../packages/monoprop-bench-tools)): the
memory instrumentation, the model builders, and the two renderers that turn a
Expand Down
117 changes: 117 additions & 0 deletions benches/bench_bindings.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# Copyright 2026 Algorithmiq
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Microbenchmarks isolating Python/C++ binding overhead."""

from __future__ import annotations

from itertools import combinations, islice

import pytest

import monoprop
from monoprop import MajoranaPropagator
from monoprop.fermi import MajoranaOperator

BOUNDARY_ROUNDS = 20
CHEAP_CALL_ITERATIONS = 10_000
FUNCTIONAL_ITERATIONS = 1_000
LARGE_TERM_COUNT = 1_024
NUM_MODES = 32
OUTPUT_ATOL = 1e-300


@pytest.fixture(scope="module", autouse=True)
def _serial_only(bench_comm):
if bench_comm is not None and bench_comm.Get_size() > 1:
pytest.skip("binding microbenchmarks are serial-only")


@pytest.fixture(scope="module")
def binding_propagators():
"""Build fixed small and large propagators outside timed regions."""
small = MajoranaPropagator(
MajoranaOperator({(0, 1, 2, 3): 1.0}, NUM_MODES),
[],
cutoff=2 * NUM_MODES,
)
large_terms = dict.fromkeys(
islice(combinations(range(2 * NUM_MODES), 4), LARGE_TERM_COUNT), 1.0
)
large = MajoranaPropagator(
MajoranaOperator(large_terms, NUM_MODES),
[],
cutoff=2 * NUM_MODES,
)
return {"small": small, "large": large}


def test_binding_is_antihermitian_positional(benchmark):
"""Benchmark positional free-function dispatch."""
result = benchmark.pedantic(
monoprop.is_antihermitian,
args=([0, 1],),
rounds=BOUNDARY_ROUNDS,
iterations=CHEAP_CALL_ITERATIONS,
)
assert result is True


def test_binding_is_antihermitian_keyword(benchmark):
"""Benchmark keyword free-function dispatch."""
result = benchmark.pedantic(
monoprop.is_antihermitian,
kwargs={"indices": [0, 1]},
rounds=BOUNDARY_ROUNDS,
iterations=CHEAP_CALL_ITERATIONS,
)
assert result is True


def test_binding_size(benchmark, binding_propagators):
"""Benchmark a bound method with a scalar return value."""
simulator = binding_propagators["small"]._simulator
result = benchmark.pedantic(
simulator.size,
rounds=BOUNDARY_ROUNDS,
iterations=CHEAP_CALL_ITERATIONS,
)
assert result == 1


def test_binding_expectation_value_functional(benchmark, binding_propagators):
"""Benchmark calls through an already-created functional."""
functional = binding_propagators["small"].expectation_value_functional()
result = benchmark.pedantic(
functional,
args=([],),
rounds=BOUNDARY_ROUNDS,
iterations=FUNCTIONAL_ITERATIONS,
)
assert isinstance(result, float)


@pytest.mark.parametrize("operator_size", ["small", "large"])
def test_binding_evolved_operator(benchmark, binding_propagators, operator_size):
"""Benchmark raw evolved-operator dictionary conversion."""
simulator = binding_propagators[operator_size]._simulator
iterations = 100 if operator_size == "small" else 1
expected_terms = 1 if operator_size == "small" else LARGE_TERM_COUNT
result = benchmark.pedantic(
simulator.evolved_operator,
args=([], OUTPUT_ATOL),
rounds=BOUNDARY_ROUNDS,
iterations=iterations,
)
assert len(result) == expected_terms
10 changes: 10 additions & 0 deletions benches/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,10 @@
import hashlib
import json
import os
import platform
import socket
from dataclasses import asdict, fields
from importlib.metadata import PackageNotFoundError, version
from pathlib import Path
from typing import TYPE_CHECKING, Any

Expand Down Expand Up @@ -191,6 +193,11 @@ def _core_md5() -> str:

def _meta() -> dict[str, Any]:
"""Return this run's configuration metadata for the report."""
try:
nanobind_backend_version = version("nanobind-backend")
except PackageNotFoundError:
nanobind_backend_version = "not installed"

return {
"label": os.environ.get("monoprop_BENCH_LABEL", "?"), # noqa: SIM112
"ranks": _size(),
Expand All @@ -202,6 +209,9 @@ def _meta() -> dict[str, Any]:
"monoprop_core_md5": _core_md5(),
"monoprop_variant": monoprop.__variant__,
"monoprop_compiler_flags": monoprop.__compiler_flags__,
"python_version": platform.python_version(),
"nanobind_version": monoprop.__nanobind_version__,
"nanobind_backend_version": nanobind_backend_version,
"monoprop_max_num_modes": monoprop.MAX_NUM_MODES,
"malloc_arena_max": os.environ.get("MALLOC_ARENA_MAX", "default"),
"omp_num_threads": os.environ.get("OMP_NUM_THREADS", "default"),
Expand Down
19 changes: 18 additions & 1 deletion docs/content/docs/benchmarks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -446,11 +446,28 @@ aborts if MPI is missing; extra args are forwarded to `mpiexec` (as root, add
Override any config field via `--<model>-<field>` (e.g. `--pauli-num-layers 30`);
`just bench --help` lists all.

**Bindings** (`bench_bindings.py`; fixed, serial-only) — low-work calls that
isolate Python/C++ boundary costs: positional and keyword dispatch, a bound
method, an already-created expectation-value functional, and raw evolved-operator
conversion for one and 1,024 terms. These are microbenchmarks of binding overhead,
not propagation throughput. Run only this group with:

```bash
uv run pytest benches/bench_bindings.py --benchmark-only
```

For an upgrade comparison, use Release builds on one pinned host and keep the
interpreter and command fixed. Compare the old dependency, the new dependency
before wrapper edits, and the new dependency after wrapper edits to separate
automatic nanobind improvements from source-level changes.

### Output

Each run writes `results/time-<label>.json` (pytest-benchmark) and
`<label>.json` (everything else); `monoprop-bench-report` merges all labels into
`results/REPORT.md`.
`results/REPORT.md`. The configuration table records the Python version, the
nanobind frontend version compiled into the extension, and the installed
`nanobind-backend` version. Older artifacts without these fields render `—`.

### Continuous benchmarking

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -198,10 +198,22 @@ def _config_table(labels: list[str], results: dict[str, dict]) -> list[str]:
metas = {lbl: results.get(lbl, {}).get("meta", {}) for lbl in labels}
if not any(metas.values()):
return []
headers = ["Label", "Ranks", "monoprop threads", "CPUs (logical/physical)", "Host"]
headers = [
"Label",
"Python",
"nanobind",
"Backend",
"Ranks",
"monoprop threads",
"CPUs (logical/physical)",
"Host",
]
rows = [
[
label,
str(metas[label].get("python_version", "—")),
str(metas[label].get("nanobind_version", "—")),
str(metas[label].get("nanobind_backend_version", "—")),
str(metas[label].get("ranks", "—")),
str(metas[label].get("monoprop_threads", "default")),
_fmt_cpus(metas[label]),
Expand Down
16 changes: 16 additions & 0 deletions packages/monoprop-bench-tools/tests/test_report.py
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,22 @@ def test_build_report_includes_hyperparameters(tmp_path: Path) -> None:
assert md.index("## Hyperparameters") < md.index("## Heisenberg")


def test_build_report_includes_runtime_provenance(tmp_path: Path) -> None:
_write_timings(tmp_path)
_write_results(
tmp_path,
meta={
"python_version": "3.13.2",
"nanobind_version": "3.2.0",
"nanobind_backend_version": "2.4.0",
},
)
md = _collapse(report.build_report(tmp_path))

assert "| Python | nanobind | Backend |" in md
assert "| np1 | 3.13.2 | 3.2.0 | 2.4.0 |" in md


def test_fmt_config_formats_floats_compactly() -> None:
assert report._fmt_config(1e-5) == "1e-05"
assert report._fmt_config(1.0) == "1"
Expand Down
17 changes: 10 additions & 7 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
[build-system]
requires = [
"nanobind>=2.13.0,<3",
"mpi4py>=4.1.0",
"nanobind-backend>=1.0.0.dev3",
"nanobind==3.0.0.dev3",
"scikit-build-core>=1.0.3,<2",
"setuptools-scm>=8",
"mpi4py>=4.1.0",
]
build-backend = "scikit_build_core.build"

Expand Down Expand Up @@ -37,7 +38,7 @@ classifiers = [
"Typing :: Typed",
]
dynamic = ["version"]
dependencies = ["numpy>=2", "msgpack>=1.0.0"]
dependencies = ["msgpack>=1.0.0", "nanobind-backend>=1.0.0.dev3", "numpy>=2"]


[project.urls]
Expand Down Expand Up @@ -113,8 +114,8 @@ provider = "scikit_build_core.metadata.setuptools_scm"
minimum-version = "build-system.requires"
# setuptools-style build caching in a local directory
build-dir = "build/{state}/{build_type}"
# build stable ABI wheels for CPython 3.12+
wheel.py-api = "cp312"
# build stable ABI wheels for CPython 3.11
wheel.py-api = "cp311"
logging.level = "INFO"
# read CMake version from CMakeLists.txt
cmake.version = "CMakeLists.txt"
Expand Down Expand Up @@ -325,7 +326,9 @@ flake8-annotations.allow-star-arg-any = true
"packages/monoprop-bench-tools/src/**" = ["ANN401", "T201"]
# cupy comes from an optional, GPU-only extra, so it is imported inside the
# functions that need it (PLC0415).
"packages/monoprop-bench-tools/src/monoprop_bench_tools/memory/gpu.py" = ["PLC0415"]
"packages/monoprop-bench-tools/src/monoprop_bench_tools/memory/gpu.py" = [
"PLC0415",
]

[tool.ruff.lint.pycodestyle]
# maximum line length to allow for line-length violations within documentation,
Expand Down Expand Up @@ -365,9 +368,9 @@ reportMatchNotExhaustive = "information"


[tool.cibuildwheel]
build = "cp3??-*"
build-frontend = "uv"
build-verbosity = 1
build = ["cp311-*", "cp312-*", "cp313-*", "cp314-*"]
skip = ["*-musllinux_*"]
test-command = "python -m pytest {package}/tests"
test-groups = ["test"]
Expand Down
2 changes: 2 additions & 0 deletions src/monoprop/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
MAX_NUM_MODES,
__build_type__,
__compiler_flags__,
__nanobind_version__,
__variant__,
antihermitian_generator_correction,
has_mpi,
Expand Down Expand Up @@ -58,6 +59,7 @@
"PauliPropagator",
"__build_type__",
"__compiler_flags__",
"__nanobind_version__",
"__variant__",
"__version__",
"antihermitian_generator_correction",
Expand Down
7 changes: 4 additions & 3 deletions src/monoprop/bindings/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -104,9 +104,10 @@ configure_file(
)

nanobind_add_module(_core
STABLE_ABI # Perform a stable ABI build
NB_STATIC # Compile the core nanobind library as a static library
NOMINSIZE # Don’t perform optimizations to minimize binary size
NB_SUPPRESS_WARNINGS # suppress warnings from nanobind and Python headers
NOMINSIZE # don’t perform optimizations to minimize binary size
LTO # perform link-time optimization
BACKEND_MODULE nanobind_backend
${CMAKE_CURRENT_BINARY_DIR}/bindings.cpp
${_bind_cpps} # List of generated binders
)
Expand Down
6 changes: 3 additions & 3 deletions src/monoprop/bindings/binder.h
Original file line number Diff line number Diff line change
Expand Up @@ -202,11 +202,11 @@ auto bind_monomial_propagator(nb::module_ &mod) -> void {
[](MonomialPropagator<NumModes> &self, const VecD &parameters, double atol) -> nb::dict {
nb::dict py_result;
for (const auto &[indices, coeff] : self.evolved_operator_terms(parameters, atol)) {
nb::list key;
nb::tuple_builder key(indices.size());
for (const auto &i : indices) {
key.append(i);
key.put(i);
}
py_result[nb::tuple(key)] = coeff;
py_result[key.commit()] = coeff;
}

if (!self.schrodinger() && std::abs(self.core_term()) >= atol) {
Expand Down
1 change: 1 addition & 0 deletions src/monoprop/bindings/bindings.cpp.in
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@ NB_MODULE(_core, m) {
// clang-format on
m.attr("__build_type__") = std::string(build_type());
m.attr("__compiler_flags__") = compiler_flags();
m.attr("__nanobind_version__") = std::string("@nanobind_VERSION@");
m.attr("__variant__") = std::string(variant());

#ifdef monoprop_ENABLE_MPI
Expand Down
Loading
Loading