Python package generation script - #121
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #121 +/- ##
===========================================
+ Coverage 99.73% 99.75% +0.02%
===========================================
Files 29 31 +2
Lines 2258 2469 +211
Branches 490 532 +42
===========================================
+ Hits 2252 2463 +211
Misses 5 5
Partials 1 1
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Add build_python_model (cppwg/utils/python_model.py) and call it from CppWrapperGenerator.generate() via write_python_model(), writing cppwg_model.json to the wrapper root once the info tree is final. The model describes each module's compiled extension name, its classes (base py-name, templated flag, and per-instantiation arg-list -> py_name), enums and free functions - enough for a separate step to generate the Python package layer without re-parsing the source. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A standalone script that reads cppwg_model.json plus a small YAML layout manifest and writes a per-subpackage _generated.py: the compiled-extension import and the TemplateClass subscript stubs (Point[2] -> Point_2). It never touches the hand-written __init__.py, which does `from ._generated import *` and keeps the bespoke pieces (TemplateMethod attachments, package init, curation). Output is black-formatted so it is stable under a git-diff / black --check reproducibility gate. Two layouts are supported: module-per-subpackage (each cppwg module owns its own extension, `from .<module> import *`) and shared-module split (one extension divided into subpackages by an explicit membership manifest, used where a single compiled module backs several Python subpackages). An opt-in diagonal_shorthand flag adds a single-arg alias for all-equal instantiations (Element[2] == Element[2, 2]). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Apply cppwg_initgen.py to both examples. Each subpackage's mechanical __init__.py body is replaced with `from ._generated import *`, and the generated _generated.py (compiled import + TemplateClass stubs) is added alongside; the shapes primitives __init__ keeps its bespoke UnitSquare.GetAreaIn TemplateMethod attachment. A py_layout.yaml manifest and the emitted cppwg_model.json are committed for each (cells uses a module_dirs override to place its single `all` module at the package root). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Switch cppwg_model from JSON to YAML: it is human-readable and reuses the config format cppwg already depends on, so no new dependency and one fewer format in the tree. write_python_model() now uses yaml.safe_dump and cppwg_initgen.py reads it with yaml.safe_load; the model dict and the generated _generated.py are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Opt-in manifest flag `flatten_to_root: true` makes the generator also write a top-level _generated.py that re-exports every subpackage's class, enum and free function into the package root, so `package.ClassName` works in addition to `package.subpackage.ClassName` (PyChaste issue #73). It re-exports the base (stub) names, not the concrete instantiations, and warns if a name is exported by more than one subpackage. Only the shared-module-split layout uses it; shapes/cells leave it off. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A template argument that is itself a wrapped templated type - cells' MeshFactory<PottsMesh<2>> - is now keyed in the generated stub by the Python concrete class name (PottsMesh_2) instead of the raw C++ type string (PottsMesh<2>). So the natural MeshFactory[PottsMesh[2]] resolves: PottsMesh[2] is the PottsMesh_2 class and _normalize_key keys it by __name__. The split-argument MeshFactory[PottsMesh, 2] form is dropped. _build_cxx_index maps each wrapped instantiation's C++ type string to its py_name; only cells' MeshFactory has such an argument, so shapes and pychaste are unchanged. Adds a cells test exercising MeshFactory[PottsMesh[2]]. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1fcb0a7 to
9b427c8
Compare
The "_" joining a templated class's base name to its template arguments (and successive/nested args) was hardcoded in class_info.update_py_names. Hoist it into a single CPPWG_TEMPLATE_ARG_SEPARATOR constant so a project can widen it (e.g. "__") to avoid Python-name clashes. Defaults to "_", so generated names are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The package-model docstring still called the model "JSON-serialisable" even though cppwg now writes it as cppwg_model.yaml; "serialisable (plain-dict)" describes the shape without implying a format. Also reword the BASE_INFO_OPTIONS comment about options becoming unreachable from the YAML. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Unify the package-layer generation vocabulary and fix a misnomer: - python_model -> package_model: the module (cppwg/utils/package_model.py), build_python_model/write_python_model -> build_package_model/write_package_model, CPPWG_PYTHON_MODEL_FILENAME -> CPPWG_PACKAGE_MODEL_FILENAME, and the emitted file cppwg_model.yaml -> cppwg_package_model.yaml. "python" said nothing; "package" names what is modelled. - py_layout.yaml -> package_layout.yaml, pairing with the existing package_info.yaml config; the initgen CLI flag --manifest -> --layout and the `manifest` variable -> `layout`. - cppwg_initgen.py -> cppwg_genpackage.py: the script generates _generated.py and never touches the hand-written __init__.py, so "initgen" was misleading. Behaviour is unchanged; re-running the generator on the shapes/cells examples reproduces identical output (only the "Generated by ..." header names the new script). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the package-layer generator into the installed package as cppwg/genpackage.py and dispatch to it from cppwg.__main__ when the first argument is "genpackage". The subcommand is intercepted before argument parsing, so a normal `cppwg SOURCE_ROOT ...` run is entirely unaffected. Because tools/ is not part of the installed package, the generator was previously unavailable to pip-installed users; it now ships and runs wherever cppwg is installed (`cppwg genpackage --model ... --layout ...`). tools/cppwg_genpackage.py remains as a thin launcher for the old direct-path invocation. Now that the code always lives in-package, the standalone write_file_if_changed fallback (and its no-cover pragma) is dropped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The cppwg_package_model.yaml files carried no header, unlike the generated wrapper sources (and the _generated.py files). Prepend a two-line YAML-comment banner (CPPWG_PACKAGE_MODEL_HEADER) so the committed model reads clearly as a generated artifact. The banner is ignored on load, so the model still parses. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a "Python packages" doc page covering the `cppwg genpackage` subcommand: the emitted package model, the two layouts (one subpackage per module with `module_dirs`, and the shared-extension split with `subpackages`), the `TemplateClass` subscript stubs, the `diagonal_shorthand`/`flatten_to_root` options, and the hand-written `__init__.py` bits. Slot it into the toctree after custom-generators. Also lowercase the "CppWG" branding to "cppwg" in the index/basics/configuration pages for consistency with the command name. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Moving the package-layer generator into the cppwg package put it under coverage measurement for the first time, exposing that main() (and a couple of branches) were untested. Add tests exercising main() end-to-end for both dispatch paths, the "unchanged" second-run branch, and the flatten skip for a subpackage that exports nothing. genpackage.py 88% -> 100%. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds config-driven generation of Python package layers to prevent wrapper/package drift.
Changes:
- Emits a YAML package model during wrapper generation.
- Adds
cppwg genpackagefor generating imports and template aliases. - Updates examples, tests, and documentation.
Reviewed changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
tools/cppwg_genpackage.py |
Adds standalone launcher. |
tests/test_package_model.py |
Tests package-model construction. |
tests/test_genpackage.py |
Tests package generation and CLI. |
examples/shapes/wrapper/package_layout.yaml |
Defines shapes package layout. |
examples/shapes/wrapper/cppwg_package_model.yaml |
Adds generated shapes model. |
examples/shapes/src/py/pyshapes/primitives/_generated.py |
Generates primitive exports and aliases. |
examples/shapes/src/py/pyshapes/primitives/__init__.py |
Imports generated primitives. |
examples/shapes/src/py/pyshapes/math_funcs/_generated.py |
Generates function exports. |
examples/shapes/src/py/pyshapes/math_funcs/__init__.py |
Imports generated functions. |
examples/shapes/src/py/pyshapes/geometry/_generated.py |
Generates geometry aliases. |
examples/shapes/src/py/pyshapes/geometry/__init__.py |
Imports generated geometry. |
examples/shapes/src/py/pyshapes/composites/_generated.py |
Generates composite exports. |
examples/shapes/src/py/pyshapes/composites/__init__.py |
Imports generated composites. |
examples/cells/tests/test_cells.py |
Tests nested template subscripting. |
examples/cells/src/py/pycells/_generated.py |
Generates cell exports and aliases. |
examples/cells/src/py/pycells/__init__.py |
Imports generated cell API. |
examples/cells/dynamic/wrappers/cppwg_package_model.yaml |
Adds generated cells model. |
examples/cells/dynamic/package_layout.yaml |
Defines flat cells layout. |
doc/python-packages.md |
Documents package generation. |
doc/index.md |
Adds documentation navigation. |
doc/configuration.md |
Updates project naming. |
doc/basics.md |
Updates project naming. |
cppwg/utils/package_model.py |
Builds serializable package models. |
cppwg/utils/constants.py |
Adds model and template constants. |
cppwg/info/class_info.py |
Centralizes template-name separators. |
cppwg/info/base_info.py |
Revises option commentary. |
cppwg/genpackage.py |
Implements package-layer generation. |
cppwg/generators.py |
Writes package models. |
cppwg/__main__.py |
Dispatches the new subcommand. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- _key_repr renders each template argument with json.dumps, so a value with
special characters produces a valid, escaped Python string literal.
- An empty (or all-unknown) subpackage now imports the extension module
instead of emitting `from ... import ()`, which is a SyntaxError.
- Drop the optional-black formatting pass: the generator emits its final
layout directly, the way cppwg's C++ templates do, so output no longer
depends on whether black happens to be installed.
- Fix a typo in the BASE_INFO_OPTIONS comment ("became" -> "become").
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_build_cxx_index reconstructed a class's C++ type as base<args> using the Python (overridden) name, so a class with a name_override used as a nested template argument (e.g. Factory[NewName[2]]) failed to resolve and raised KeyError. Carry each instantiation's real C++ type name (cxx_name) in the package model and key on it, falling back to the reconstruction for older models. +2 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The package model is a machine-generated, git-diff-checked interchange artifact consumed by `cppwg genpackage`, so a deterministic serialization matters more than YAML's readability (the hand-authored *.yaml configs stay YAML, and YAML's implicit typing is a footgun for machine data). Write cppwg_package_model.json via json.dumps(sort_keys); the do-not-edit banner becomes a `_comment` field (JSON has no comments), which the generator ignores. genpackage reads it with json.load. Regenerated shapes/cells models. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When a module or subpackage is removed from the config, its previously generated _generated.py is left behind and can keep exporting removed bindings. After generating, walk the package root and warn about any _generated.py that carries the do-not-edit banner but was not written this run. cppwg never deletes files (it does not own the hand-written __init__.py), so it flags the orphan for the user to remove the containing directory rather than silently leaving stale bindings. +1 test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cppwg/genpackage.py is a library module, entered via the `cppwg genpackage` subcommand (__main__ dispatches to cppwg.genpackage.main) and via the tools/cppwg_genpackage.py launcher (which has its own guard). Its `if __name__ == "__main__"` block therefore only enabled the undocumented `python -m cppwg.genpackage` path, so remove it as dead code. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The shapes/cells flags run `coverage xml` from inside the example directories, where the repo's [tool.coverage.report] exclude_also is not found - so excluded lines (e.g. an `if __name__` guard) in cppwg modules those flags never import were reported as executable-but-missed. Merged with the unit flag (which does exclude them), codecov surfaced them as uncovered patch lines and failed the 100%-patch gate. Point both `coverage xml` steps at the repo pyproject.toml so the exclusions apply consistently across every flag. Verified locally: an `if __name__` line in an unimported module is reported as missed without --rcfile and excluded with it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…bcommand tools/cppwg_genpackage.py became redundant once the generator was exposed as the `cppwg genpackage` subcommand: it only enabled `python tools/cppwg_genpackage.py` (which the subcommand supersedes), it is not part of the installed package, and the generated-file banner pointed at it rather than the canonical command. Remove it and retarget the banner, the orphan-detection marker, and every doc/comment reference to `cppwg genpackage`. Regenerated the shapes/cells _generated.py with the new banner. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The package model carried each templated instantiation's C++ type name as `cxx_name`, and genpackage used `cxx`-prefixed helpers (`_build_cxx_index`, `cxx_to_pyname`) - but the rest of cppwg uses `cpp` throughout (class_info.cpp_names, cpp_name, *.cppwg.cpp). The value comes straight from cpp_names, so rename `cxx_*` to `cpp_*` everywhere: the model field, the genpackage helpers, the tests, and the regenerated example models. No output change - _generated.py is unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename internal helpers for readability, with no behaviour change (generated output is byte-identical): - _stub_source -> _render_template_stub and _key_repr -> _render_key, joining render_generated_module in a consistent "render" family. - _build_cpp_index -> _build_cpp_to_pyname, matching the cpp_to_pyname map it returns and dropping the vague "index". - _concrete_names -> _concrete_py_names (they are py_names, now sitting next to cpp_names in the model). Also add the missing _write_generated docstring. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 30 out of 30 changed files in this pull request and generated 1 comment.
Suppressed comments (2)
cppwg/genpackage.py:95
- This still omits untemplated classes from the C++-to-Python index. If
OldCellis wrapped asNewCelland used byFactory<OldCell>, the generated key remains("OldCell",)whileFactory[NewCell]normalizes to("NewCell",), causingKeyError. Carry the real C++ name for untemplated classes too and index every instantiation, not only those with template arguments.
if inst["args"]:
cpp = inst.get("cpp_name") or (
f'{class_info["base"]}<{",".join(inst["args"])}>'
)
index[cpp] = inst["py_name"]
cppwg/genpackage.py:163
- Importing the helper as public
TemplateClassmakesfrom ._generated import *expose it from every generated subpackage. It can also overwrite a legitimately wrapped class namedTemplateClass, making that binding inaccessible whenever any templated stub is present. Import the helper under a private alias and render stubs against that alias (or define an explicit__all__that excludes the helper).
lines = [GENERATED_HEADER.rstrip("\n"), "", compiled_import]
if templated_classes:
lines.append(f"from {package}._syntax import TemplateClass")
Hoist the free-function type-based exclusion rule into cppwg.info.exclusions.free_function_is_excluded, mirroring the existing method/constructor/variable predicates, so the writer and the package model apply one rule and cannot drift. Previously the model recorded functions the writer dropped for an excluded arg/return type, so a shared-module layout would emit an import of a symbol the extension never bound. Also honor the per-function config `excluded` flag in the free-function writer's exclude() (as the enum writer does with enum_info.excluded), so a config-excluded function emits no binding and the writer agrees with the model. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 33 out of 33 changed files in this pull request and generated no new comments.
Suppressed comments (3)
cppwg/genpackage.py:190
- This import binds the top-level package name in
_generated.py. Since the sibling__init__.pystar-imports this file, an empty subpackage unexpectedly exportspkgdespite owning no symbols. Alias the extension to a private name so the validity-only import does not leak into the package API.
return f"import {package}.{compiled_module} # noqa: F401"
cppwg/genpackage.py:261
- The shared-module split only knows the enum type name, but an enum with
should_export_values()also creates module-level enumerator bindings. For example,ShapeKindexposesCIRCLE(examples/shapes/src/py/tests/test_classes.py:89-95); assigningShapeKindto a split subpackage imports onlyShapeKind, sosubpackage.CIRCLEdisappears even though it is wrapped. Include exported enumerator names in the package model and import/export them with their owning enum.
for name in module["enums"] + module["free_functions"]:
other_names.add(name)
cppwg/genpackage.py:95
- Un-templated classes are excluded from this index, so a renamed class used as a template argument still cannot resolve. For example, if C++
OldNameis exposed asNewName, the model rendersFactory<OldName>with key("OldName",), whileFactory[NewName]normalizes to("NewName",)and raisesKeyError. Carry the untemplated instantiation's C++ name in the model and index it here asOldName -> NewName.
if inst["args"]:
cpp = inst.get("cpp_name") or (
f'{class_info["base"]}<{",".join(inst["args"])}>'
)
index[cpp] = inst["py_name"]
Three fixes from the PR review of the genpackage package-layer generator: - Empty subpackage: `import package.extension` bound the top-level `package` name, which the sibling __init__'s `from ._generated import *` leaked into the package API. Alias it to a private `_extension` so nothing leaks. - Renamed untemplated class as a template argument: the model now carries cpp_name for untemplated instantiations too, and _build_cpp_to_pyname indexes them, so an OldName-exposed-as-NewName class used as `Factory<OldName>` resolves to `Factory[NewName]` instead of raising KeyError. - Exported enum values under the shared-module split: a value-exporting enum also binds its enumerators at module scope (e.g. ShapeKind exports CIRCLE). The model records these per enum (enum_exports) and the split imports and re-exports them alongside the enum, so subpackage.CIRCLE keeps working. Regenerated the shapes and cells package models (additive cpp_name/enum_exports fields only; wrappers and _generated.py unchanged). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes #102