Skip to content

Python package generation script - #121

Merged
kwabenantim merged 24 commits into
developfrom
102-initgen-script
Aug 12, 2026
Merged

Python package generation script#121
kwabenantim merged 24 commits into
developfrom
102-initgen-script

Conversation

@kwabenantim

Copy link
Copy Markdown
Member

Fixes #102

@codecov

codecov Bot commented Aug 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.75%. Comparing base (d7de4ca) to head (8694917).

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #121      +/-   ##
===========================================
+ Coverage    99.73%   99.75%   +0.02%     
===========================================
  Files           29       31       +2     
  Lines         2258     2469     +211     
  Branches       490      532      +42     
===========================================
+ Hits          2252     2463     +211     
  Misses           5        5              
  Partials         1        1              
Flag Coverage Δ
cells 71.26% <18.77%> (-4.96%) ⬇️
shapes 71.28% <23.58%> (-4.94%) ⬇️
unit 93.14% <97.37%> (+0.37%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
cppwg/__main__.py 100.00% <100.00%> (ø)
cppwg/generators.py 99.45% <100.00%> (+0.02%) ⬆️
cppwg/genpackage.py 100.00% <100.00%> (ø)
cppwg/info/base_info.py 100.00% <ø> (ø)
cppwg/info/class_info.py 100.00% <100.00%> (ø)
cppwg/info/exclusions.py 100.00% <100.00%> (ø)
cppwg/utils/constants.py 100.00% <100.00%> (ø)
cppwg/utils/package_model.py 100.00% <100.00%> (ø)
cppwg/writers/free_function_writer.py 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

kwabenantim and others added 6 commits August 11, 2026 21:33
Add build_python_model (cppwg/utils/python_model.py) and call it from
CppWrapperGenerator.generate() via write_python_model(), writing
cppwg_model.json to the wrapper root once the info tree is final. The model
describes each module's compiled extension name, its classes (base py-name,
templated flag, and per-instantiation arg-list -> py_name), enums and free
functions - enough for a separate step to generate the Python package layer
without re-parsing the source.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A standalone script that reads cppwg_model.json plus a small YAML layout
manifest and writes a per-subpackage _generated.py: the compiled-extension
import and the TemplateClass subscript stubs (Point[2] -> Point_2). It never
touches the hand-written __init__.py, which does `from ._generated import *`
and keeps the bespoke pieces (TemplateMethod attachments, package init,
curation). Output is black-formatted so it is stable under a git-diff /
black --check reproducibility gate.

Two layouts are supported: module-per-subpackage (each cppwg module owns its
own extension, `from .<module> import *`) and shared-module split (one
extension divided into subpackages by an explicit membership manifest, used
where a single compiled module backs several Python subpackages). An opt-in
diagonal_shorthand flag adds a single-arg alias for all-equal instantiations
(Element[2] == Element[2, 2]).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Apply cppwg_initgen.py to both examples. Each subpackage's mechanical
__init__.py body is replaced with `from ._generated import *`, and the
generated _generated.py (compiled import + TemplateClass stubs) is added
alongside; the shapes primitives __init__ keeps its bespoke
UnitSquare.GetAreaIn TemplateMethod attachment. A py_layout.yaml manifest and
the emitted cppwg_model.json are committed for each (cells uses a module_dirs
override to place its single `all` module at the package root).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Switch cppwg_model from JSON to YAML: it is human-readable and reuses the
config format cppwg already depends on, so no new dependency and one fewer
format in the tree. write_python_model() now uses yaml.safe_dump and
cppwg_initgen.py reads it with yaml.safe_load; the model dict and the
generated _generated.py are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Opt-in manifest flag `flatten_to_root: true` makes the generator also write a
top-level _generated.py that re-exports every subpackage's class, enum and free
function into the package root, so `package.ClassName` works in addition to
`package.subpackage.ClassName` (PyChaste issue #73). It re-exports the base
(stub) names, not the concrete instantiations, and warns if a name is exported
by more than one subpackage. Only the shared-module-split layout uses it;
shapes/cells leave it off.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A template argument that is itself a wrapped templated type - cells'
MeshFactory<PottsMesh<2>> - is now keyed in the generated stub by the Python
concrete class name (PottsMesh_2) instead of the raw C++ type string
(PottsMesh<2>). So the natural MeshFactory[PottsMesh[2]] resolves: PottsMesh[2]
is the PottsMesh_2 class and _normalize_key keys it by __name__. The
split-argument MeshFactory[PottsMesh, 2] form is dropped.

_build_cxx_index maps each wrapped instantiation's C++ type string to its
py_name; only cells' MeshFactory has such an argument, so shapes and pychaste
are unchanged. Adds a cells test exercising MeshFactory[PottsMesh[2]].

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kwabenantim and others added 7 commits August 12, 2026 11:41
The "_" joining a templated class's base name to its template arguments
(and successive/nested args) was hardcoded in class_info.update_py_names.
Hoist it into a single CPPWG_TEMPLATE_ARG_SEPARATOR constant so a project
can widen it (e.g. "__") to avoid Python-name clashes. Defaults to "_", so
generated names are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The package-model docstring still called the model "JSON-serialisable"
even though cppwg now writes it as cppwg_model.yaml; "serialisable
(plain-dict)" describes the shape without implying a format. Also reword
the BASE_INFO_OPTIONS comment about options becoming unreachable from the
YAML.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Unify the package-layer generation vocabulary and fix a misnomer:

- python_model -> package_model: the module (cppwg/utils/package_model.py),
  build_python_model/write_python_model -> build_package_model/write_package_model,
  CPPWG_PYTHON_MODEL_FILENAME -> CPPWG_PACKAGE_MODEL_FILENAME, and the emitted
  file cppwg_model.yaml -> cppwg_package_model.yaml. "python" said nothing;
  "package" names what is modelled.
- py_layout.yaml -> package_layout.yaml, pairing with the existing
  package_info.yaml config; the initgen CLI flag --manifest -> --layout and the
  `manifest` variable -> `layout`.
- cppwg_initgen.py -> cppwg_genpackage.py: the script generates _generated.py
  and never touches the hand-written __init__.py, so "initgen" was misleading.

Behaviour is unchanged; re-running the generator on the shapes/cells examples
reproduces identical output (only the "Generated by ..." header names the new
script).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the package-layer generator into the installed package as
cppwg/genpackage.py and dispatch to it from cppwg.__main__ when the first
argument is "genpackage". The subcommand is intercepted before argument
parsing, so a normal `cppwg SOURCE_ROOT ...` run is entirely unaffected.

Because tools/ is not part of the installed package, the generator was
previously unavailable to pip-installed users; it now ships and runs wherever
cppwg is installed (`cppwg genpackage --model ... --layout ...`).
tools/cppwg_genpackage.py remains as a thin launcher for the old direct-path
invocation. Now that the code always lives in-package, the standalone
write_file_if_changed fallback (and its no-cover pragma) is dropped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The cppwg_package_model.yaml files carried no header, unlike the generated
wrapper sources (and the _generated.py files). Prepend a two-line YAML-comment
banner (CPPWG_PACKAGE_MODEL_HEADER) so the committed model reads clearly as a
generated artifact. The banner is ignored on load, so the model still parses.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a "Python packages" doc page covering the `cppwg genpackage` subcommand:
the emitted package model, the two layouts (one subpackage per module with
`module_dirs`, and the shared-extension split with `subpackages`), the
`TemplateClass` subscript stubs, the `diagonal_shorthand`/`flatten_to_root`
options, and the hand-written `__init__.py` bits. Slot it into the toctree
after custom-generators.

Also lowercase the "CppWG" branding to "cppwg" in the index/basics/configuration
pages for consistency with the command name.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Moving the package-layer generator into the cppwg package put it under
coverage measurement for the first time, exposing that main() (and a couple
of branches) were untested. Add tests exercising main() end-to-end for both
dispatch paths, the "unchanged" second-run branch, and the flatten skip for a
subpackage that exports nothing. genpackage.py 88% -> 100%.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kwabenantim
kwabenantim marked this pull request as ready for review August 12, 2026 13:42
@kwabenantim
kwabenantim requested a balanced review from Copilot August 12, 2026 13:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds config-driven generation of Python package layers to prevent wrapper/package drift.

Changes:

  • Emits a YAML package model during wrapper generation.
  • Adds cppwg genpackage for generating imports and template aliases.
  • Updates examples, tests, and documentation.

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
tools/cppwg_genpackage.py Adds standalone launcher.
tests/test_package_model.py Tests package-model construction.
tests/test_genpackage.py Tests package generation and CLI.
examples/shapes/wrapper/package_layout.yaml Defines shapes package layout.
examples/shapes/wrapper/cppwg_package_model.yaml Adds generated shapes model.
examples/shapes/src/py/pyshapes/primitives/_generated.py Generates primitive exports and aliases.
examples/shapes/src/py/pyshapes/primitives/__init__.py Imports generated primitives.
examples/shapes/src/py/pyshapes/math_funcs/_generated.py Generates function exports.
examples/shapes/src/py/pyshapes/math_funcs/__init__.py Imports generated functions.
examples/shapes/src/py/pyshapes/geometry/_generated.py Generates geometry aliases.
examples/shapes/src/py/pyshapes/geometry/__init__.py Imports generated geometry.
examples/shapes/src/py/pyshapes/composites/_generated.py Generates composite exports.
examples/shapes/src/py/pyshapes/composites/__init__.py Imports generated composites.
examples/cells/tests/test_cells.py Tests nested template subscripting.
examples/cells/src/py/pycells/_generated.py Generates cell exports and aliases.
examples/cells/src/py/pycells/__init__.py Imports generated cell API.
examples/cells/dynamic/wrappers/cppwg_package_model.yaml Adds generated cells model.
examples/cells/dynamic/package_layout.yaml Defines flat cells layout.
doc/python-packages.md Documents package generation.
doc/index.md Adds documentation navigation.
doc/configuration.md Updates project naming.
doc/basics.md Updates project naming.
cppwg/utils/package_model.py Builds serializable package models.
cppwg/utils/constants.py Adds model and template constants.
cppwg/info/class_info.py Centralizes template-name separators.
cppwg/info/base_info.py Revises option commentary.
cppwg/genpackage.py Implements package-layer generation.
cppwg/generators.py Writes package models.
cppwg/__main__.py Dispatches the new subcommand.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread cppwg/genpackage.py Outdated
Comment thread cppwg/genpackage.py Outdated
Comment thread cppwg/genpackage.py Outdated
Comment thread cppwg/genpackage.py
Comment thread cppwg/genpackage.py
Comment thread cppwg/info/base_info.py Outdated
kwabenantim and others added 9 commits August 12, 2026 16:15
- _key_repr renders each template argument with json.dumps, so a value with
  special characters produces a valid, escaped Python string literal.
- An empty (or all-unknown) subpackage now imports the extension module
  instead of emitting `from ... import ()`, which is a SyntaxError.
- Drop the optional-black formatting pass: the generator emits its final
  layout directly, the way cppwg's C++ templates do, so output no longer
  depends on whether black happens to be installed.
- Fix a typo in the BASE_INFO_OPTIONS comment ("became" -> "become").

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_build_cxx_index reconstructed a class's C++ type as base<args> using the
Python (overridden) name, so a class with a name_override used as a nested
template argument (e.g. Factory[NewName[2]]) failed to resolve and raised
KeyError. Carry each instantiation's real C++ type name (cxx_name) in the
package model and key on it, falling back to the reconstruction for older
models. +2 tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The package model is a machine-generated, git-diff-checked interchange
artifact consumed by `cppwg genpackage`, so a deterministic serialization
matters more than YAML's readability (the hand-authored *.yaml configs stay
YAML, and YAML's implicit typing is a footgun for machine data). Write
cppwg_package_model.json via json.dumps(sort_keys); the do-not-edit banner
becomes a `_comment` field (JSON has no comments), which the generator
ignores. genpackage reads it with json.load. Regenerated shapes/cells models.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When a module or subpackage is removed from the config, its previously
generated _generated.py is left behind and can keep exporting removed
bindings. After generating, walk the package root and warn about any
_generated.py that carries the do-not-edit banner but was not written this
run. cppwg never deletes files (it does not own the hand-written __init__.py),
so it flags the orphan for the user to remove the containing directory rather
than silently leaving stale bindings. +1 test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cppwg/genpackage.py is a library module, entered via the `cppwg genpackage`
subcommand (__main__ dispatches to cppwg.genpackage.main) and via the
tools/cppwg_genpackage.py launcher (which has its own guard). Its
`if __name__ == "__main__"` block therefore only enabled the undocumented
`python -m cppwg.genpackage` path, so remove it as dead code.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The shapes/cells flags run `coverage xml` from inside the example directories,
where the repo's [tool.coverage.report] exclude_also is not found - so excluded
lines (e.g. an `if __name__` guard) in cppwg modules those flags never import
were reported as executable-but-missed. Merged with the unit flag (which does
exclude them), codecov surfaced them as uncovered patch lines and failed the
100%-patch gate. Point both `coverage xml` steps at the repo pyproject.toml so
the exclusions apply consistently across every flag.

Verified locally: an `if __name__` line in an unimported module is reported as
missed without --rcfile and excluded with it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…bcommand

tools/cppwg_genpackage.py became redundant once the generator was exposed as
the `cppwg genpackage` subcommand: it only enabled `python
tools/cppwg_genpackage.py` (which the subcommand supersedes), it is not part of
the installed package, and the generated-file banner pointed at it rather than
the canonical command. Remove it and retarget the banner, the orphan-detection
marker, and every doc/comment reference to `cppwg genpackage`. Regenerated the
shapes/cells _generated.py with the new banner.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The package model carried each templated instantiation's C++ type name as
`cxx_name`, and genpackage used `cxx`-prefixed helpers (`_build_cxx_index`,
`cxx_to_pyname`) - but the rest of cppwg uses `cpp` throughout
(class_info.cpp_names, cpp_name, *.cppwg.cpp). The value comes straight from
cpp_names, so rename `cxx_*` to `cpp_*` everywhere: the model field, the
genpackage helpers, the tests, and the regenerated example models. No output
change - _generated.py is unaffected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename internal helpers for readability, with no behaviour change (generated
output is byte-identical):

- _stub_source -> _render_template_stub and _key_repr -> _render_key, joining
  render_generated_module in a consistent "render" family.
- _build_cpp_index -> _build_cpp_to_pyname, matching the cpp_to_pyname map it
  returns and dropping the vague "index".
- _concrete_names -> _concrete_py_names (they are py_names, now sitting next to
  cpp_names in the model).

Also add the missing _write_generated docstring.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

cppwg/genpackage.py:95

  • This still omits untemplated classes from the C++-to-Python index. If OldCell is wrapped as NewCell and used by Factory<OldCell>, the generated key remains ("OldCell",) while Factory[NewCell] normalizes to ("NewCell",), causing KeyError. Carry the real C++ name for untemplated classes too and index every instantiation, not only those with template arguments.
                if inst["args"]:
                    cpp = inst.get("cpp_name") or (
                        f'{class_info["base"]}<{",".join(inst["args"])}>'
                    )
                    index[cpp] = inst["py_name"]

cppwg/genpackage.py:163

  • Importing the helper as public TemplateClass makes from ._generated import * expose it from every generated subpackage. It can also overwrite a legitimately wrapped class named TemplateClass, making that binding inaccessible whenever any templated stub is present. Import the helper under a private alias and render stubs against that alias (or define an explicit __all__ that excludes the helper).
    lines = [GENERATED_HEADER.rstrip("\n"), "", compiled_import]
    if templated_classes:
        lines.append(f"from {package}._syntax import TemplateClass")

Comment thread cppwg/utils/package_model.py
Hoist the free-function type-based exclusion rule into
cppwg.info.exclusions.free_function_is_excluded, mirroring the existing
method/constructor/variable predicates, so the writer and the package model
apply one rule and cannot drift. Previously the model recorded functions the
writer dropped for an excluded arg/return type, so a shared-module layout would
emit an import of a symbol the extension never bound.

Also honor the per-function config `excluded` flag in the free-function
writer's exclude() (as the enum writer does with enum_info.excluded), so a
config-excluded function emits no binding and the writer agrees with the model.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated no new comments.

Suppressed comments (3)

cppwg/genpackage.py:190

  • This import binds the top-level package name in _generated.py. Since the sibling __init__.py star-imports this file, an empty subpackage unexpectedly exports pkg despite owning no symbols. Alias the extension to a private name so the validity-only import does not leak into the package API.
        return f"import {package}.{compiled_module}  # noqa: F401"

cppwg/genpackage.py:261

  • The shared-module split only knows the enum type name, but an enum with should_export_values() also creates module-level enumerator bindings. For example, ShapeKind exposes CIRCLE (examples/shapes/src/py/tests/test_classes.py:89-95); assigning ShapeKind to a split subpackage imports only ShapeKind, so subpackage.CIRCLE disappears even though it is wrapped. Include exported enumerator names in the package model and import/export them with their owning enum.
        for name in module["enums"] + module["free_functions"]:
            other_names.add(name)

cppwg/genpackage.py:95

  • Un-templated classes are excluded from this index, so a renamed class used as a template argument still cannot resolve. For example, if C++ OldName is exposed as NewName, the model renders Factory<OldName> with key ("OldName",), while Factory[NewName] normalizes to ("NewName",) and raises KeyError. Carry the untemplated instantiation's C++ name in the model and index it here as OldName -> NewName.
                if inst["args"]:
                    cpp = inst.get("cpp_name") or (
                        f'{class_info["base"]}<{",".join(inst["args"])}>'
                    )
                    index[cpp] = inst["py_name"]

Three fixes from the PR review of the genpackage package-layer generator:

- Empty subpackage: `import package.extension` bound the top-level `package`
  name, which the sibling __init__'s `from ._generated import *` leaked into the
  package API. Alias it to a private `_extension` so nothing leaks.

- Renamed untemplated class as a template argument: the model now carries
  cpp_name for untemplated instantiations too, and _build_cpp_to_pyname indexes
  them, so an OldName-exposed-as-NewName class used as `Factory<OldName>`
  resolves to `Factory[NewName]` instead of raising KeyError.

- Exported enum values under the shared-module split: a value-exporting enum
  also binds its enumerators at module scope (e.g. ShapeKind exports CIRCLE).
  The model records these per enum (enum_exports) and the split imports and
  re-exports them alongside the enum, so subpackage.CIRCLE keeps working.

Regenerated the shapes and cells package models (additive cpp_name/enum_exports
fields only; wrappers and _generated.py unchanged).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kwabenantim
kwabenantim merged commit 76bd4b6 into develop Aug 12, 2026
17 checks passed
@kwabenantim
kwabenantim deleted the 102-initgen-script branch August 12, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generate the Python package from the wrapper config

2 participants