refactor(bench): 📦 release the benchmark harness as monoprop-bench-tools

[Panadestein]

🤖 AI text below 🤖

Turns the repository into a uv workspace and splits the benchmark harness in two: the reusable half becomes a package we can publish, the benchmarks stay here.

monoprop/
├── pyproject.toml            # the library (unchanged in scope)
├── uv.lock                   # one lock for the whole workspace
├── src/monoprop/
├── benches/                  # monoprop's own benchmark suite — stays put
└── packages/
    ├── monoprop-bench-tools/ # → PyPI
    └── bench-third-party/    # → GitHub only, standalone uv project

What moved

Was	Is
benches/_memory_cpu.py	monoprop_bench_tools.memory.cpu
benches/_memory_gpu.py	monoprop_bench_tools.memory.gpu
benches/_builders.py	monoprop_bench_tools.models
benches/report.py	monoprop_bench_tools.report → monoprop-bench-report
benches/bmf.py	monoprop_bench_tools.bmf → monoprop-bench-bmf
tests/test_bench_*.py	packages/monoprop-bench-tools/tests/
benches/third_party/	packages/bench-third-party/

All moves are git mv, so git log --follow still works.

Design decisions worth reviewing

benches/ does not move. Bencher keys its metric history on the benchmark name, so moving or renaming a bench_* test orphans its tracked series and resets the thresholds' sample windows. The suite therefore cannot version with a library release, which is precisely the line between "harness" and "benchmarks". monoprop-bench-tools contains no benchmarks.

The bencher.dev tracking stays in the repo. Only the renderer (bmf.py) is tooling and moved. bench_main.yml carries BENCHER_API_KEY, vars.BENCHER_PROJECT, the --branch/--testbed identity and the four threshold blocks — that is this repository's CI identity, and a published package cannot own a secret. The workflow itself is unchanged; just bench-bmf now shells out to the console script.

Three dependency groups instead of two.

• test — monoprop's suite only, deliberately free of workspace members. cibuildwheel installs it against a built wheel via test-groups = ["test"], where a monoprop-bench-tools requirement would resolve from PyPI (nonexistent) and drag monoprop back in from source.
• workspace-test — test + the member. What uv run pytest at the root needs.
• bench — workspace-test + pytest-benchmark.

bench-third-party is excluded from the workspace. It pins a narrower requires-python, a git dependency and linux-x86_64-only CUDA wheels; folding it in would drag all of that into the root lock. It stays a standalone uv project with its own lockfile (cd packages/bench-third-party && uv sync). It now depends on monoprop-bench-tools properly instead of poking benches/ onto sys.path.

No gpu extra on the tools package. Adding one put cupy-cuda12x + cuda-pathfinder into the root lock. CuPy is installed separately; memory.gpu already degrades to Method.UNAVAILABLE without it.

Publishing

deploy.yml gains build-bench-tools (uv build --package monoprop-bench-tools — pure Python, no cibuildwheel matrix) and a separate upload_pypi_bench_tools job. Separate because it is a separate PyPI project, and because a failure there must not hold back the library's own upload. hatch-vcs reads the same git tags via raw-options = { root = "../.." }, so the two distributions share a version line; verified that the wheel still builds from the sdist, where there is no git.

Important

Before the next release someone has to register a trusted publisher for monoprop-bench-tools on PyPI (this workflow file, the pypi environment), or upload_pypi_bench_tools will fail on its first run.

Validation

• uv lock --check, uv sync --all-groups --all-extras
• pytest -m "not mpi" → 617 passed (both suites)
• just bench-smoke and just bench-bmf smoke end to end through the console scripts
• Both distributions build; packages/bench-third-party/uv.lock re-resolved
• prek run --all-files clean

Deliberate non-changes

• packages/** is excluded from Sonar scope, preserving current behaviour. Bringing it in is a separate call.
• benches/results/*.json is untouched, so Bencher continuity holds.

One seam this creates

benches/conftest.py writes the results artifact schema and monoprop_bench_tools.report/.bmf read it — now across a package boundary. Called out in both docstrings and in AGENTS.md: a change to the recorded sections has to land on both sides.

Docs updated in the same change: AGENTS.md (new "Workspace layout" section), README.md, docs/content/docs/benchmarks.mdx, docs/content/docs/testing.mdx, benches/README.md.

[github-actions]

Docs preview: https://pr-227.monoprop-docs.pages.dev

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.70%. Comparing base (275c2a5) to head (a9313ef).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #227   +/-   ##
=======================================
  Coverage   97.70%   97.70%           
=======================================
  Files          14       14           
  Lines         742      742           
  Branches       98       98           
=======================================
  Hits          725      725           
  Misses         12       12           
  Partials        5        5           

Flag	Coverage Δ
cpp	97.70% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[diagonal-hamiltonian]

Good idea 👍

[Copilot]

Pull request overview

This PR restructures the repository into a uv workspace and extracts the reusable benchmark harness into a separately publishable Python distribution (monoprop-bench-tools), while keeping benches/ in-repo to preserve Bencher metric history continuity.

Changes:

• Introduces a uv workspace setup and dependency-group split (test vs workspace-test vs bench) to keep wheel-testing isolated from workspace members.
• Creates the new packages/monoprop-bench-tools distribution containing memory tooling, model builders, and artifact renderers with console scripts.
• Moves third-party comparison scripts into a standalone packages/bench-third-party project with its own lockfile and updated imports.

Reviewed changes

Copilot reviewed 35 out of 57 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
uv.lock	Adds workspace manifest members and a workspace-test group; locks the new monoprop-bench-tools editable member.
sonar-project.properties	Excludes packages/** from Sonar scope to preserve previous analysis behavior.
README.md	Documents workspace install flow and explains the new repository layout.
pyproject.toml	Defines [tool.uv.workspace], introduces workspace-test, updates pytest collection paths, and adjusts ruff per-file ignores for new package layout.
packages/monoprop-bench-tools/tests/test_report.py	Updates imports to reference monoprop_bench_tools.report.
packages/monoprop-bench-tools/tests/test_models.py	Updates imports to reference monoprop_bench_tools.models.
packages/monoprop-bench-tools/tests/test_memory.py	Updates imports to reference monoprop_bench_tools.memory.cpu.
packages/monoprop-bench-tools/tests/test_bmf.py	Updates imports to reference monoprop_bench_tools.bmf.
packages/monoprop-bench-tools/tests/conftest.py	Adds isolated pytest config/fixtures so the tools package can test standalone.
packages/monoprop-bench-tools/src/monoprop_bench_tools/report.py	Converts report renderer into a console-script entrypoint (monoprop-bench-report) and tightens CLI usage.
packages/monoprop-bench-tools/src/monoprop_bench_tools/models.py	Introduces reusable benchmark model builders and registry (MODELS).
packages/monoprop-bench-tools/src/monoprop_bench_tools/memory/gpu.py	Moves GPU peak-memory measurement into the publishable tooling package and updates module references/docstrings.
packages/monoprop-bench-tools/src/monoprop_bench_tools/memory/cpu.py	Moves CPU peak-memory and PSS sampling utilities into the publishable tooling package and expands docstrings.
packages/monoprop-bench-tools/src/monoprop_bench_tools/memory/init.py	Adds package init describing CPU/GPU backends and import expectations.
packages/monoprop-bench-tools/src/monoprop_bench_tools/bmf.py	Converts BMF exporter into a console-script entrypoint (monoprop-bench-bmf) and updates usage/errors.
packages/monoprop-bench-tools/src/monoprop_bench_tools/init.py	Adds package top-level docstring and version exports.
packages/monoprop-bench-tools/README.md	Adds end-user README for the new PyPI-distributed tooling package.
packages/monoprop-bench-tools/pyproject.toml	Defines the new monoprop-bench-tools project metadata and console scripts; configures hatch-vcs versioning from repo tags.
packages/monoprop-bench-tools/LICENSE	Adds Apache-2.0 license file for the new distribution.
packages/bench-third-party/uv.lock	Re-resolves the standalone third-party project lockfile and adds an editable dependency on monoprop-bench-tools.
packages/bench-third-party/README.md	Updates documentation to refer to monoprop_bench_tools.memory.gpu and new invocation patterns.
packages/bench-third-party/pyproject.toml	Defines the standalone monoprop-bench-third-party project and consumes both monoprop packages from the checkout.
packages/bench-third-party/pauli_prop/settings.json	Adds committed settings for the third-party Pauli benchmark configuration.
packages/bench-third-party/pauli_prop/scaling_results.jsonl	Adds committed scaling results dataset used by plotting scripts.
packages/bench-third-party/pauli_prop/run_scaling.py	Adds sweep driver that spawns per-(backend,size) subprocesses with timeouts and captures failures.
packages/bench-third-party/pauli_prop/run_scaling.jl	Adds Julia scaling runner matching the Python record schema.
packages/bench-third-party/pauli_prop/run_one.py	Adds single-backend runner used as the subprocess work unit for scaling sweeps.
packages/bench-third-party/pauli_prop/run_model.py	Adds fixed-size multi-backend runner emitting per-step curves.
packages/bench-third-party/pauli_prop/run_model.jl	Adds Julia fixed-size runner merging results into the shared JSON file.
packages/bench-third-party/pauli_prop/results.json	Adds committed fixed-size results dataset used by plotting scripts.
packages/bench-third-party/pauli_prop/plot_speedup.py	Adds speed-up bar chart generator (monoprop baseline vs other backends).
packages/bench-third-party/pauli_prop/plot_scaling.py	Adds scaling-curve plotting and sidecar table/provenance generation.
packages/bench-third-party/pauli_prop/plot_results.py	Adds fixed-size per-step plotting for runtime/memory curves.
packages/bench-third-party/pauli_prop/model.py	Adds shared benchmark model definition (TFIM) used across backends.
packages/bench-third-party/pauli_prop/backends.py	Switches memory instrumentation imports to monoprop-bench-tools rather than sys.path hacks; updates metric docs.
packages/bench-third-party/majorana_prop/run_benchmarks.sh	Adds orchestration script to run monoprop + Julia benchmarks back-to-back.
packages/bench-third-party/majorana_prop/results.json	Adds committed results dataset for the majorana comparison benchmark.
packages/bench-third-party/majorana_prop/README.md	Adds documentation for the majorana comparison benchmark workflow.
packages/bench-third-party/majorana_prop/Project.toml	Adds Julia project deps for the majorana benchmark.
packages/bench-third-party/majorana_prop/plot_results.py	Adds plotting for majorana benchmark results, including a runtime-only figure.
packages/bench-third-party/majorana_prop/monoprop_hubbard1d_benchmark.py	Updates benchmark script to import memory instrumentation from monoprop-bench-tools.
packages/bench-third-party/majorana_prop/julia_hubbard1d_benchmark.jl	Adds Julia benchmark implementation and merges results/provenance into shared JSON.
packages/bench-third-party/bench_common.jl	Updates comment references to match the new monoprop_bench_tools.memory.cpu source of truth.
benches/third_party/pyproject.toml	Removes the previous third-party benchmark project definition (now replaced by packages/bench-third-party).
benches/README.md	Documents the split between benches/ (benchmarks) and monoprop-bench-tools (reusable harness) plus the standalone third-party project.
benches/conftest.py	Updates imports to consume builders/memory tooling from monoprop-bench-tools and clarifies the artifact schema boundary.
benches/bench_random.py	Updates to import barriered from the new tools package.
benches/bench_models.py	Updates to import model registry/build helpers from the new tools package and updates doc references.
AGENTS.md	Documents the new workspace layout and the cross-package schema seam between bench writer and renderers.
justfile	Updates test groups to workspace-test and switches report/BMF invocations to the new console scripts.
docs/content/docs/testing.mdx	Documents that uv run pytest now collects both root and tools-member test suites and how to install them.
docs/content/docs/benchmarks.mdx	Updates paths and commands to match the new packages/bench-third-party location and console scripts.
.gitignore	Ignores hatch-vcs/setuptools-scm generated version files for workspace members and adds logs/.
.github/workflows/test.yml	Installs workspace-test group in CI so both test suites can be collected/imported.
.github/workflows/qa-analysis.yml	Installs workspace-test group for QA analysis runs.
.github/workflows/deploy.yml	Adds build+upload jobs for monoprop-bench-tools and wires PR-change detection for its pyproject.toml.
.github/ISSUE_TEMPLATE/performance_issue.yml	Updates suggested repro steps to the new uv sync + just bench-smoke flow.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[robertodr]

Overall LGTM! Thank you @Panadestein I think we'll gain very valuable experience with uv workspaces. I have some comments on the extras for the other workspace packages.

[robertodr]

Thank you!

[Copilot]

Pull request overview

Copilot reviewed 35 out of 57 changed files in this pull request and generated no new comments.

Suppressed comments (1)

packages/monoprop-bench-tools/pyproject.toml:40

• The PR description says there is no GPU extra on monoprop-bench-tools to avoid pulling CuPy (and cuda-pathfinder) into the root lock, but the package now defines gpu-cuda12 / gpu-cuda13 extras (and uv.lock includes cupy-cuda12x, cupy-cuda13x, cuda-pathfinder). Please align the implementation with the stated design decision: either remove these extras (and related docs) or update the PR description/rationale to reflect that the extras (and lock growth) are intentional.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud