diff --git a/.github/workflows/news.yaml b/.github/workflows/news.yaml index f194282..0ff5b91 100644 --- a/.github/workflows/news.yaml +++ b/.github/workflows/news.yaml @@ -28,3 +28,11 @@ jobs: venv: ${{ runner.temp }}/release-tools - name: 🔍 Validate changelog fragments uses: tox-dev/github-workflows/validate-news@0c4cffd63033d38196319524d101caae1eef925b # 6.7.2 + with: + exclude-newer-package: | + nab=2026-11-05T05:51:00Z + nab-index=2026-11-05T05:51:00Z + nab-markersets=2026-11-05T05:51:00Z + nab-project=2026-11-05T05:51:00Z + nab-provider=2026-11-05T05:51:00Z + nab-resolver=2026-11-05T05:51:00Z diff --git a/README.md b/README.md index 3d316ce..8f658ea 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,8 @@ + + + python-sbom + + # python-sbom [![PyPI](https://img.shields.io/pypi/v/python-sbom?style=flat-square)](https://pypi.org/project/python-sbom/) @@ -7,45 +12,60 @@ [![Downloads](https://static.pepy.tech/badge/python-sbom/month)](https://pepy.tech/project/python-sbom) [![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/tox-dev/python-sbom/badge)](https://scorecard.dev/viewer/?uri=github.com/tox-dev/python-sbom) -SBOMs for Python distributions. python-sbom writes a [CycloneDX](https://cyclonedx.org/) SBOM into every wheel you -build, where [PEP 770](https://peps.python.org/pep-0770/) tells installers to keep it. The SBOM records your project, -the runtime dependencies it declares, any wheels it bundles, and the build toolchain, and a rebuild with the same inputs -reproduces it byte for byte. +Software bills of materials (SBOMs) for Python distributions. python-sbom writes a [CycloneDX](https://cyclonedx.org/) +SBOM into every [wheel](https://packaging.python.org/en/latest/specifications/binary-distribution-format/) you build, +where [PEP 770](https://peps.python.org/pep-0770/) (a Python Enhancement Proposal) tells installers to keep it. The SBOM +records your project, the runtime dependencies it declares, any wheels it bundles, and the build toolchain, and a +rebuild with the same inputs reproduces it byte for byte. -## Install +## Use -A hatchling project names `python-sbom[hatch]` in its build requirements, as shown below. To embed an SBOM into a wheel -another backend built, install the command line into the environment that built the wheel, since the SBOM lists that -environment's packages as the build tools (or point `--build-env` at it): +python-sbom works with every [build backend](https://peps.python.org/pep-0517/). Pick the integration for yours: -```console -python -m pip install python-sbom -``` +| Backend | Add to `[build-system] requires` | Turn it on | +| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | --------------------------------------------------------------------------------- | +| [hatchling](https://hatch.pypa.io/latest/) | `python-sbom[hatch]` | `[tool.hatch.build.hooks.python-sbom]` | +| [pdm-backend](https://backend.pdm-project.org/) | `python-sbom[pdm]` | `[tool.python-sbom]` | +| [setuptools](https://setuptools.pypa.io/) | `python-sbom[setuptools]` | `[tool.python-sbom]` | +| any other, such as [meson-python](https://mesonbuild.com/meson-python/), [maturin](https://www.maturin.rs/), [scikit-build-core](https://scikit-build-core.readthedocs.io/) or [flit-core](https://flit.pypa.io/) | `python-sbom` | `build-backend = "python_sbom.backend"`, and your backend in `[tool.python-sbom]` | -To convert and validate SBOMs in CI, a separate tool environment is fine: +With hatchling: -```console -pipx install 'python-sbom[validate]' -``` +```toml +[build-system] +build-backend = "hatchling.build" +requires = ["hatchling>=1.28", "python-sbom[hatch]>=0.2"] -## Use +[tool.hatch.build.hooks.python-sbom] +``` -With [hatchling](https://hatch.pypa.io/latest/), add the hook to the build requirements and enable it: +With meson-python, or any backend without a plugin, python-sbom wraps it: ```toml [build-system] -build-backend = "hatchling.build" -requires = ["hatchling>=1.28", "python-sbom[hatch]>=0.1"] +build-backend = "python_sbom.backend" +requires = ["meson-python>=0.18", "python-sbom>=0.2"] -[tool.hatch.build.hooks.python-sbom] +[tool.python-sbom] +build-backend = "mesonpy" ``` -With another build backend, such as meson-python, embed the SBOM after the build: +For a wheel built outside the build interface, such as with [`maturin build`](https://www.maturin.rs/), or renamed after +it by [`auditwheel repair`](https://github.com/pypa/auditwheel), embed the SBOM afterwards: ```console python-sbom embed dist/*.whl ``` -Describe another zip archive the project publishes, such as a zipapp, with `python-sbom archive`. The -[documentation](https://python-sbom.readthedocs.io) covers the options, adding project knowledge through a content -provider, rendering an SBOM as SPDX and validating it. +To convert and validate SBOMs in continuous integration (CI), install the command line in a tool environment of its own +with [pipx](https://pipx.pypa.io/) or [uv](https://docs.astral.sh/uv/): + +```console +pipx install 'python-sbom[validate]' +python-sbom validate dist/ +``` + +The [documentation](https://python-sbom.readthedocs.io) starts with a tutorial, then has a guide per backend, the +options and commands, and how the SBOM is built. It also covers describing a +[zipapp](https://docs.python.org/3/library/zipapp.html), adding project knowledge through a content provider, and +rendering an SBOM as [SPDX](https://spdx.dev/) (Software Package Data Exchange). diff --git a/docs/_static/favicon.svg b/docs/_static/favicon.svg new file mode 100644 index 0000000..2e78c04 --- /dev/null +++ b/docs/_static/favicon.svg @@ -0,0 +1,17 @@ + + + + + + + + + + + + + + + + + diff --git a/docs/_static/logo-dark.svg b/docs/_static/logo-dark.svg new file mode 100644 index 0000000..92298a5 --- /dev/null +++ b/docs/_static/logo-dark.svg @@ -0,0 +1,20 @@ + + + + + + + + + + + + + + + + + + + + diff --git a/docs/_static/logo-horizontal-dark.svg b/docs/_static/logo-horizontal-dark.svg new file mode 100644 index 0000000..7526ce3 --- /dev/null +++ b/docs/_static/logo-horizontal-dark.svg @@ -0,0 +1,20 @@ + + + + + + + + + + + + + + + + + + + +python-sbom diff --git a/docs/_static/logo-horizontal-light.svg b/docs/_static/logo-horizontal-light.svg new file mode 100644 index 0000000..752f250 --- /dev/null +++ b/docs/_static/logo-horizontal-light.svg @@ -0,0 +1,20 @@ + + + + + + + + + + + + + + + + + + + +python-sbom diff --git a/docs/_static/logo-light.svg b/docs/_static/logo-light.svg new file mode 100644 index 0000000..3449e77 --- /dev/null +++ b/docs/_static/logo-light.svg @@ -0,0 +1,20 @@ + + + + + + + + + + + + + + + + + + + + diff --git a/docs/_static/python-sbom.css b/docs/_static/python-sbom.css new file mode 100644 index 0000000..79da34f --- /dev/null +++ b/docs/_static/python-sbom.css @@ -0,0 +1,193 @@ +:root { + --sbom-accent: #2b6599; + --sbom-accent-contrast: #ffffff; + --sbom-radius: 0.6rem; +} + +body[data-theme="dark"] { + --sbom-accent: #f0c94a; + --sbom-accent-contrast: #062d59; +} + +@media (prefers-color-scheme: dark) { + body:not([data-theme="light"]) { + --sbom-accent: #f0c94a; + --sbom-accent-contrast: #062d59; + } +} + +/* Furo fixes the reading column and gives each side drawer half of what is left, which leaves empty bands beside the + page on a wide window. Instead the page fills the window up to 120rem, and past that the drawers absorb the extra + width in equal halves, so the layout stays centered while their backgrounds still reach the window's edges. Below + 63em and 82em Furo turns the drawers into slide-outs whose width it sets itself, so these rules stop there. */ +@media (min-width: 63em) { + .sidebar-drawer { + min-width: 0; + width: calc(max(0px, (100vw - 120rem) / 2) + 16rem); + } + + .sidebar-container { + width: 16rem; + } + + .content { + flex: 1 1 0; + width: auto; + } +} + +@media (min-width: 82em) { + .toc-drawer { + width: calc(max(0px, (100vw - 120rem) / 2) + 15em); + } +} + +/* both are flex items, which by default never shrink below their widest child, so one long line in a code block would + widen the whole page; letting them shrink keeps that line scrolling inside its own block */ +.main, +.content { + min-width: 0; +} + +/* the logo is a square icon, which at the sidebar's full width would push the navigation below the fold */ +.sidebar-logo { + max-width: 7rem; +} + +h1, +h2, +h3 { + letter-spacing: -0.015em; +} + +article h1 { + font-weight: 750; + font-size: clamp(1.7rem, 4.5vw, 2.5rem); + overflow-wrap: anywhere; +} + +article h2 { + margin-top: 2.2rem; + padding-bottom: 0.3rem; + border-bottom: 1px solid var(--color-background-border); +} + +/* the hero carries the landing page's headline; keep the page title for screen readers and the browser tab */ +section#python-sbom > h1 { + position: absolute; + width: 1px; + height: 1px; + overflow: hidden; + clip-path: inset(50%); +} + +/* line sphinx-design grids up with the text column instead of indenting them by the gutter */ +article .sd-container-fluid { + padding-left: 0; + padding-right: 0; +} + +article .sd-container-fluid > .sd-row { + margin-left: calc(-0.5 * var(--sd-gutter-x)); + margin-right: calc(-0.5 * var(--sd-gutter-x)); +} + +/* console output lines stay readable; the theme's own code colors would otherwise gray them out */ +.highlight .go { + color: var(--color-foreground-secondary) !important; +} + +.highlight, +div[class^="highlight-"] pre { + border-radius: var(--sbom-radius); +} + +/* Center each diagram and let a wide one scroll on its own. */ +.mermaid { + text-align: center; + overflow-x: auto; +} + +/* landing page */ + +/* the page's view, edit and theme icons sit in the top-right corner; start the hero below them */ +.hero { + margin: 3.6rem 0 2rem; + padding: 2.4rem 2rem; + border-radius: calc(var(--sbom-radius) * 2); + background: + radial-gradient(circle at 100% 0%, rgba(55, 118, 171, 0.22), transparent 55%), + radial-gradient(circle at 0% 100%, rgba(255, 212, 59, 0.16), transparent 50%), + var(--color-background-secondary); + border: 1px solid var(--color-background-border); +} + +.hero-logo { + float: right; + width: clamp(4.5rem, 14vw, 8.5rem); + margin: 0 0 0.5rem 1.2rem; +} + +.hero-title { + margin: 0; + font-size: clamp(2rem, 5vw, 3rem); + font-weight: 800; + line-height: 1.1; + letter-spacing: -0.03em; +} + +.hero-title span { + color: var(--sbom-accent); +} + +.hero-tagline { + margin: 0.9rem 0 1.4rem; + max-width: 40em; + font-size: 1.15rem; + color: var(--color-foreground-secondary); +} + +.hero-actions { + display: flex; + flex-wrap: wrap; + gap: 0.6rem; +} + +.hero-actions a { + padding: 0.55rem 1.1rem; + border-radius: 999px; + font-weight: 600; + text-decoration: none; + border: 1px solid var(--sbom-accent); +} + +.hero-actions a.primary { + background: var(--sbom-accent); + color: var(--sbom-accent-contrast); +} + +.hero-actions a.secondary { + color: var(--sbom-accent); +} + +.hero-actions a:hover { + filter: brightness(1.1); +} + +.sd-card.feature { + border-radius: var(--sbom-radius); + border-color: var(--color-background-border); + background: var(--color-background-secondary); + transition: + transform 0.15s ease, + border-color 0.15s ease; +} + +.sd-card.feature:hover { + transform: translateY(-2px); + border-color: var(--sbom-accent); +} + +.sd-card.feature .sd-card-title { + font-size: 1.05rem; +} diff --git a/docs/changelog.d/+all-backends.feature.rst b/docs/changelog.d/+all-backends.feature.rst new file mode 100644 index 0000000..e237214 --- /dev/null +++ b/docs/changelog.d/+all-backends.feature.rst @@ -0,0 +1,3 @@ +Write the SBOM with any build backend: a pdm-backend hook and a setuptools plugin act for every project with a +``[tool.python-sbom]`` table, and ``python_sbom.backend`` wraps any other backend, such as meson-python or maturin, +named by the new ``build-backend`` option. Each records the source revision in the sdist, as the hatchling hook does. diff --git a/docs/cli.rst b/docs/cli.rst deleted file mode 100644 index 8d3ba9a..0000000 --- a/docs/cli.rst +++ /dev/null @@ -1,5 +0,0 @@ -.. sphinx_argparse_cli:: - :module: python_sbom._cli._parser - :func: build_parser - :title: Command line interface - :group_title_prefix: diff --git a/docs/concepts/architecture.rst b/docs/concepts/architecture.rst new file mode 100644 index 0000000..213f1dc --- /dev/null +++ b/docs/concepts/architecture.rst @@ -0,0 +1,36 @@ +############## + Architecture +############## + +Only the entry points know a build system or the command line: the hatchling_ and pdm-backend_ hooks, the setuptools_ +plugin, the :term:`wrapper backend`, and the ``embed`` and ``archive`` commands. Each gathers the options, the build +scripts and the build facts (time, revision, toolchain) and hands them, with a :class:`~python_sbom.ContentProvider`, to +the pipeline for its artifact. The pipeline opens the artifact once, reads it, gives the provider a +:class:`~python_sbom.ProviderContext` that reads from the same open artifact, and maps what it read to components. It +then adds what every artifact of the project shares: :term:`supplier`, source links, revision, tools, dependencies and +:term:`formulation`. + +The diagram follows one build, from the entry point to the document written into the artifact: + +.. mermaid:: + + flowchart LR + entry["Entry point
build hook, plugin, wrapper backend,
embed or archive command"] + collect["Collect
options, revision,
build tools, artifact"] + provider["Content provider
what the project adds"] + model["Model
the CycloneDX document"] + write["Write
into the wheel or archive"] + entry --> collect --> model --> write + provider --> model + classDef io fill:#cfe4ff,stroke:#1f6feb,color:#0b1f3a; + classDef core fill:#d8f3dc,stroke:#2d6a4f,color:#081c15; + class entry,write io; + class collect,provider,model core; + +Every step raises :class:`~python_sbom.SbomError`, which the command line reports without a traceback. Each integration +imports its own :term:`build backend` and nothing else does, so the provider :term:`API` and the command line work +without any backend installed. + +The one extension point is the :class:`~python_sbom.ContentProvider`, which every entry point loads from the +``content-provider`` option. A hatchling custom hook can override +:meth:`~python_sbom.hatch.SbomBuildHook.content_provider` to build the provider itself. diff --git a/docs/concepts/backends.rst b/docs/concepts/backends.rst new file mode 100644 index 0000000..10921d1 --- /dev/null +++ b/docs/concepts/backends.rst @@ -0,0 +1,66 @@ +####################################### + Why each backend gets its integration +####################################### + +The :term:`SBOM` describes the finished :term:`wheel`: its file name, its files with their hashes, and the environment +that built it. So it has to be written after the backend has written the wheel, inside the build, and every integration +is shaped by where a backend lets code run at that moment. + +**************************************************** + Plugins where a backend has a hook after the build +**************************************************** + +Three backends let a plugin run once the artifact exists: + +- hatchling_ calls a :term:`build hook`'s ``finalize`` with the wheel's path; +- `pdm-backend `_ calls ``pdm_build_finalize`` with the artifact's path; +- `setuptools `_ lets a plugin replace the commands it + runs, so python-sbom wraps ``bdist_wheel`` and ``sdist``. + +A plugin keeps the project's ``build-backend`` as it is and sees the artifact the backend built. hatchling enables a +hook by its table. pdm-backend_ and setuptools_ run every plugin installed in the build environment, so python-sbom acts +only for a project with a ``[tool.python-sbom]`` table, and a package that merely has python-sbom installed nearby +builds as before. + +************************************ + A wrapper where a backend has none +************************************ + +meson-python_, maturin_, flit-core_ and uv_build_ have no plugin interface. poetry-core_'s build script runs before +packaging, and scikit-build-core_ has none that runs after the build either; its own documentation recommends `wrapping +the backend `_. Static SBOM options, +such as hatchling's ``sbom-files`` or maturin's include list, add a file while the wheel is still being written, too +early for a document that lists the wheel's own hashes. + +Every backend shares one interface, though: the hooks of :term:`PEP 517` and :term:`PEP 660`. ``python_sbom.backend`` +implements it by handing each call to the backend the project names, then writing the SBOM into the wheel it returned. +It needs no code per backend, so a backend released later works with it too. It runs in the isolated environment the +:term:`frontend ` made for the build, so the SBOM lists the build tools that ran. + +The wrapper exposes an optional hook only when the wrapped backend has it. A frontend checks for an optional hook, such +as ``prepare_metadata_for_build_wheel``, and falls back to its own default when it is missing; a wrapper that defined +every hook would change what the frontend does. + +*********** + The sdist +*********** + +A wheel built from an :term:`sdist`, which is what ``python -m build`` and ``uv build`` do by default, is built outside +the git_ checkout, so it cannot ask git for its commit. Each integration therefore adds ``.python-sbom-revision``, +holding the commit, to the sdist of a clean checkout. The plugins add it while the backend writes the sdist; the wrapper +adds it to the finished archive, keeping the backend's own times so a reproducible sdist stays reproducible. + +********************************* + What still runs after the build +********************************* + +Two things sit outside every backend: + +- Tools that build without the build interface, such as ``maturin build`` in maturin-action_, never call a plugin or the + wrapper. +- ``auditwheel repair`` and ``delocate`` run after the build and rename the wheel, so a document written during the + build names a file that is never published. + +``python-sbom embed`` covers both: it describes a finished wheel from the outside. It cannot see the build environment, +so it records the one it runs in, or the one ``--build-env`` names; :doc:`/how-to/built-elsewhere` shows how to make +that the real one. diff --git a/docs/concepts/document.rst b/docs/concepts/document.rst new file mode 100644 index 0000000..aefba47 --- /dev/null +++ b/docs/concepts/document.rst @@ -0,0 +1,65 @@ +####################### + What the SBOM records +####################### + +The document's root :term:`component` is the project, described from the :term:`wheel`'s ``METADATA``: name, version, +summary, license expression, maintainers, project :term:`URLs `, classifiers and keywords. Links to the +:term:`PyPI` release, the wheel's attestation and, with a repository, the license file follow. A build from a git_ +checkout adds the commit and a link to that exact revision. Links to the license file and build scripts point at that +commit when the checkout is clean, and at the default branch otherwise. + +:term:`CycloneDX` requires each URL to be an :term:`IRI` reference. A project URL or home page that is not one gets +percent-encoded, or left out when encoding cannot mend it, as with an scp-style ``git@host:path`` address; either way a +``python:project-url`` or ``python:home-page`` property keeps the text as written. + +The components below the root are the wheels the project bundles and the runtime dependencies it declares. A bundled +wheel carries its :term:`SHA-256`, its license, a component per file from its ``RECORD``, and the packages it vendors. A +declared dependency carries no version, because the installer picks it at install time; the document marks the root's +dependencies incomplete for that reason. An extra's dependencies are left out. + +The :term:`formulation` records how the wheel was built: the interpreter's version and implementation, every +distribution of the build environment with its dependencies, the files and hashes of each pure-Python one, and +``SOURCE_DATE_EPOCH`` when it set the build time. + +Property names start with the project's normalized name, such as ``virtualenv:bundled-wheel``, or with ``python:`` for +values copied from core metadata. + +.. _reproducibility: + +***************** + Reproducibility +***************** + +The document holds nothing about the build machine. Its timestamp comes from ``SOURCE_DATE_EPOCH``. When that is unset, +a reproducible hatchling_ build uses hatchling's fixed 2020-02-02 and a pdm-backend_ build pdm-backend's fixed +2016-01-01, the times they stamp on the wheel's entries, and any other build uses the current time. Its serial number +derives from the content, so identical documents share one and any difference gets a new one. + +The build tools are listed down to their files, so a rebuild can check it installed the same bytes. Only what is the +same on every machine goes in: platform wheels appear without their files, because those differ per operating system and +architecture, and editable installs without theirs, because they point at a checkout. Installer metadata, console-script +launchers and uv_'s cache files are left out too, since they name the :term:`frontend `, the +environment's path or a time rather than the distribution. + +Hashing each tool's ``RECORD`` instead would be smaller, but that file lists the installer metadata, so two frontends +installing the same tools would give two hashes. The cost is size: the file list grows with the build environment, and a +large build tool such as setuptools_ adds hundreds of entries. + +So a rebuild reproduces the wheel byte for byte on any operating system when these match the original build: the source +tree, as a checkout of the same commit; ``SOURCE_DATE_EPOCH``; the Python version; and the versions of the :term:`build +backend` and its dependencies. The :term:`SBOM` records the last three, so it says what to install before rebuilding. + +************* + Limitations +************* + +- The document describes what the project declares and bundles, not the full tree of runtime dependencies, which depends + on the environment the wheel is installed into. +- The links to the PyPI release and attestation assume the project publishes to PyPI with attestations. +- A wheel built from an :term:`sdist` of a dirty checkout names no commit, since no commit reproduces it. +- ``python-sbom embed`` and ``archive`` record the packages of the environment ``--build-env`` names, by default the one + they run in, and always the interpreter they run on. Unless that is the environment the artifact was built in, the + build tool record does not match the real build. +- The :term:`SPDX` rendering leaves out per-file hashes and the build record, which SPDX 2.3 cannot hold. +- A tool that renames the wheel after the build, such as ``auditwheel repair``, leaves the attestation link pointing at + the old file name until ``python-sbom embed`` runs on the renamed wheel; :doc:`/concepts/backends` says why. diff --git a/docs/concepts/glossary.rst b/docs/concepts/glossary.rst new file mode 100644 index 0000000..da0b74d --- /dev/null +++ b/docs/concepts/glossary.rst @@ -0,0 +1,137 @@ +########## + Glossary +########## + +A :term:`build frontend` asks a :term:`build backend` for a :term:`wheel`. python-sbom runs inside that build, through a +:term:`build hook` or the :term:`wrapper backend`, and writes an :term:`SBOM` into the wheel where :term:`PEP 770` tells +installers to look. The SBOM is a :term:`CycloneDX` document. It names the :term:`supplier`, lists each +:term:`component` the wheel holds or declares, and records in its :term:`formulation` the :term:`source revision` and +the build tools, the facts a :term:`reproducible build` needs. A :term:`content provider` adds what python-sbom cannot +read from the wheel. :term:`Validation ` checks the document before upload, and python-sbom renders it as +:term:`SPDX` for tools that read only that format. + +.. glossary:: + :sorted: + + API + Application programming interface. Here, the Python classes and functions python-sbom publishes for + :term:`content providers ` and build integrations, listed in :doc:`/reference/api`. + + archive + A zip file other than a wheel that holds distributions, unpacked or as wheels, such as a :term:`zipapp`. + python-sbom describes it the way it describes a :term:`wheel`. + + build backend + The tool that turns a project's source into a :term:`wheel` and an :term:`sdist`, such as `hatchling`_, + `setuptools`_ or `maturin`_. A :term:`build frontend` calls it through the interface :term:`PEP 517` defines. + + build frontend + The tool a user runs to build or install a project, such as `pip`_, `uv`_ or `build + `_. It installs the :term:`build backend` in an isolated environment and asks it for + the artifacts. + + build hook + A plugin a :term:`build backend` calls during its own build. python-sbom has one for `hatchling`_, + `pdm-backend`_ and `setuptools`_, and writes the :term:`SBOM` before the backend finishes the :term:`wheel`. + + bundled wheel + A wheel a project ships as a data file inside its own :term:`wheel`, as `virtualenv`_ ships pip. The + :term:`SBOM` lists it as a :term:`component` with its hashes, license, files and :term:`vendored packages + `. + + CI + Continuous integration: the automated jobs, such as `GitHub Actions `_ + workflows, that build, test and publish a project on each change. + + component + One entry in the :term:`SBOM`: the project itself, a file, a :term:`bundled wheel`, a declared dependency or a + build tool. + + content provider + A class a project writes to add components and properties python-sbom cannot read from the artifact. Every + integration loads the one the project's configuration names. + + CycloneDX + The `CycloneDX `_ SBOM standard, which python-sbom writes in version 1.6 as :term:`JSON`. + + formulation + The part of a :term:`CycloneDX` document that records how the artifact was made: the build workflow, the tools + it used, and the build time when the environment set it. + + IRI + Internationalized Resource Identifier, defined by `RFC 3987 `_: a + :term:`URL` that may hold any Unicode character. CycloneDX requires every link to be one. + + JSON + JavaScript Object Notation, the `text format `_ CycloneDX and SPDX documents use. + + PEP 517 + The `Python Enhancement Proposal `_ that defines the interface between a + :term:`build frontend` and a :term:`build backend`. + + PEP 660 + The `Python Enhancement Proposal `_ that adds editable installs to the + interface of :term:`PEP 517`. + + PEP 770 + The `Python Enhancement Proposal `_ that gives a :term:`wheel` a folder for + SBOMs in its metadata directory, so installers copy them into the environment where scanners look. + + PyPI + The `Python Package Index `_, where projects publish their wheels and sdists. + + reproducible build + A build that yields the same bytes from the same source and tools, as `Reproducible Builds + `_ defines it. The :term:`SBOM` records the :term:`source revision` and each + build tool with its files, so a third party can rebuild the :term:`wheel` and compare. + + SBOM + Software bill of materials: a document that lists what a piece of software contains and how it was built. + + sdist + Source distribution: the `archive of a project's source + `_ from which a :term:`build + backend` builds a :term:`wheel` later. python-sbom stores the :term:`source revision` in it, so a wheel built + from it still names its commit. + + SHA-1 + Secure Hash Algorithm 1, an older `hash function `_ that SPDX 2.3 + requires for every file. + + SHA-256 + The 256-bit variant of Secure Hash Algorithm 2, the `hash function `_ + a wheel's file list records for each file and the SBOM copies, so anyone can check the bytes. + + source revision + The commit a build ran from, and whether the checkout had uncommitted changes. + + SPDX + Software Package Data Exchange, another `SBOM standard `_. python-sbom renders the + :term:`CycloneDX` document as SPDX 2.3 on request. + + supplier + The organization that ships the project. The configuration names it, and the :term:`SBOM` records it as the + document's supplier and manufacturer. + + URL + Uniform Resource Locator: a web address. + + validation + The check of a finished :term:`SBOM` against the :term:`CycloneDX` schema, its own references and an optional + license policy, meant to run in :term:`CI` before upload. + + vendored package + A package copied into another package's source instead of installed as a dependency. The :term:`SBOM` lists it + under the :term:`bundled wheel` that carries it. + + wheel + The `built distribution `_ + an installer unpacks into an environment, and the file python-sbom writes the :term:`SBOM` into. + + wrapper backend + python-sbom's own :term:`build backend`, which passes each call to the backend the project names and adds the + :term:`SBOM` to the :term:`wheel` it gets back. It serves backends without a :term:`build hook`. + + zipapp + A Python application packed into one zip file that the interpreter runs directly, as the `zipapp + `_ module builds it. diff --git a/docs/concepts/index.rst b/docs/concepts/index.rst new file mode 100644 index 0000000..1d616eb --- /dev/null +++ b/docs/concepts/index.rst @@ -0,0 +1,15 @@ +############## + How it works +############## + +These pages define the terms python-sbom uses, explain why each :term:`build backend` gets the integration it does, what +the :term:`SBOM` records and why, and how one build flows from the entry point to the written document. The +:doc:`/how-to/index` show how to set python-sbom up, and the :doc:`/reference/index` lists the options. + +.. toctree:: + :maxdepth: 1 + + glossary + backends + document + architecture diff --git a/docs/conf.py b/docs/conf.py index f2de70c..b865851 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -7,11 +7,13 @@ from tomllib import loads from typing import TYPE_CHECKING, Final +from docutils import nodes + import python_sbom from python_sbom import __version__ if TYPE_CHECKING: - from docutils.nodes import Element, reference + from docutils.nodes import Element, document, reference from sphinx.addnodes import pending_xref from sphinx.application import Sphinx from sphinx.environment import BuildEnvironment @@ -40,6 +42,9 @@ def _inheritance_aliases() -> dict[str, str]: "sphinx.ext.inheritance_diagram", "sphinx_argparse_cli", "sphinx_autodoc_typehints", + "sphinx_copybutton", + "sphinx_design", + "sphinx_inline_tabs", "sphinxcontrib.mermaid", ] @@ -47,14 +52,60 @@ def _inheritance_aliases() -> dict[str, str]: source_suffix: Final[str] = ".rst" exclude_patterns: Final[list[str]] = ["changelog.d/*"] autosectionlabel_maxdepth: Final[int] = 2 +autosectionlabel_prefix_document: Final[bool] = True html_theme: Final[str] = "furo" html_title: Final[str] = "python-sbom: SBOMs for Python distributions" html_last_updated_fmt: Final[str] = _NOW.isoformat() html_static_path: Final[list[str]] = ["_static"] -html_css_files: Final[list[str]] = ["changelog-toc.css"] -pygments_style: Final[str] = "sphinx" -pygments_dark_style: Final[str] = "monokai" +html_favicon: Final[str] = "_static/favicon.svg" +html_theme_options: Final[dict[str, str | bool | dict[str, str]]] = { + "source_repository": "https://github.com/tox-dev/python-sbom", + "source_branch": "main", + "source_directory": "docs/", + "light_logo": "logo-light.svg", + "dark_logo": "logo-dark.svg", + "sidebar_hide_name": True, + "light_css_variables": { + "color-brand-primary": "#2b6599", + "color-brand-content": "#2b6599", + "color-brand-visited": "#2b6599", + "font-stack": "Inter, ui-sans-serif, system-ui, -apple-system, 'Segoe UI', sans-serif", + "font-stack--monospace": "'JetBrains Mono', ui-monospace, SFMono-Regular, Menlo, monospace", + # top-level entries take the text color, so only the branch holding the current page shows the brand color + "color-sidebar-link-text--top-level": "#1f2937", + # Furo's grays read washed out in the small sidebar and contents text; its dark palette sets both already + "color-foreground-secondary": "#3b3f46", + "color-foreground-muted": "#4a4f57", + # a near-black on off-white reads at 15:1 without the glare of pure black on white + "color-foreground-primary": "#1e2329", + "color-background-primary": "#fcfcfb", + "color-foreground-border": "#63676f", + }, + "dark_css_variables": { + # the logo's yellow, toned down a step so links and the current page do not glare on the navy + "color-brand-primary": "#f0c94a", + "color-brand-content": "#f0c94a", + "color-brand-visited": "#f0c94a", + "color-background-primary": "#0b1626", + "color-background-secondary": "#0f1e33", + "color-background-hover": "#14273f", + "color-background-border": "#1f3654", + "color-sidebar-link-text--top-level": "#d6dde8", + "color-foreground-border": "#8a96a8", + }, +} +html_css_files: Final[list[str]] = ["changelog-toc.css", "python-sbom.css"] +# copying a snippet takes the command alone, not the console prompt or its output +copybutton_exclude: Final[str] = ".linenos, .gp, .go" +# the browser renders each diagram, so it follows furo's light and dark toggle +mermaid_output_format: Final[str] = "raw" +mermaid_light_theme: Final[str] = "neutral" +# the default fixed 500px height pads a wide diagram with empty bands +mermaid_height: Final[str] = "auto" +# both pass WCAG AA (4.5:1) for every token color against their own background +pygments_style: Final[str] = "xcode" +pygments_dark_style: Final[str] = "github-dark" autoclass_content: Final[str] = "both" inheritance_alias: Final[dict[str, str]] = _inheritance_aliases() @@ -66,6 +117,35 @@ def _inheritance_aliases() -> dict[str, str]: typehints_document_rtype: Final[bool] = True typehints_defaults: Final[str] = "comma" +# every page links the first mention of these projects by name, as in `hatchling`_ +rst_epilog: Final[str] = """ +.. _hatchling: https://hatch.pypa.io/latest/ +.. _pdm-backend: https://backend.pdm-project.org/ +.. _setuptools-scm: https://setuptools-scm.readthedocs.io/ +.. _setuptools: https://setuptools.pypa.io/ +.. _meson-python: https://mesonbuild.com/meson-python/ +.. _Meson: https://mesonbuild.com/ +.. _ninja: https://ninja-build.org/ +.. _maturin-action: https://github.com/PyO3/maturin-action +.. _maturin: https://www.maturin.rs/ +.. _scikit-build-core: https://scikit-build-core.readthedocs.io/ +.. _flit-core: https://flit.pypa.io/ +.. _poetry-core: https://python-poetry.org/docs/pyproject/#poetry-and-pep-517 +.. _uv_build: https://docs.astral.sh/uv/concepts/build-backend/ +.. _uv: https://docs.astral.sh/uv/ +.. _pipx: https://pipx.pypa.io/ +.. _pip: https://pip.pypa.io/ +.. _virtualenv: https://virtualenv.pypa.io/ +.. _auditwheel: https://github.com/pypa/auditwheel +.. _delocate: https://github.com/matthew-brett/delocate +.. _cibuildwheel: https://cibuildwheel.pypa.io/ +.. _pyperf: https://pyperf.readthedocs.io/ +.. _py-spy: https://github.com/benfred/py-spy +.. _tox: https://tox.wiki/ +.. _git: https://git-scm.com/ +.. _GitHub: https://github.com/ +.. _actions/attest: https://github.com/actions/attest +""" intersphinx_mapping: Final[dict[str, tuple[str, None]]] = {"python": ("https://docs.python.org/3.11", None)} extlinks: Final[dict[str, tuple[str, str]]] = { "pr": ("https://github.com/tox-dev/python-sbom/pull/%s", "#%s"), @@ -77,6 +157,12 @@ def _inheritance_aliases() -> dict[str, str]: ("py:class", "hatchling.builders.hooks.plugin.interface.BuildHookInterface"), ("py:class", "hatchling.metadata.core.ProjectMetadata"), ("py:class", "hatchling.bridge.app.Application"), + # neither pdm-backend nor setuptools publishes these in a Sphinx inventory + ("py:class", "pdm.backend.hooks.base.Context"), + ("py:class", "Context"), + ("py:class", "setuptools.dist.Distribution"), + ("py:class", "Distribution"), + ("py:class", "ConfigSettings"), ] @@ -84,6 +170,7 @@ def setup(app: Sphinx) -> None: # annotations name the private modules that define a type, and pathlib.Path an undocumented private alias app.connect("missing-reference", _resolve_private_path) app.connect("source-read", _include_unreleased) + app.connect("doctree-read", _anchor_by_label) def _resolve_private_path( @@ -131,3 +218,12 @@ def _include_unreleased(app: Sphinx, docname: str, source: list[str]) -> None: ".. towncrier release notes start", f".. _release-unreleased:\n\nUnreleased\n----------\n\n{joined}\n.. towncrier release notes start", ) + + +def _anchor_by_label(app: Sphinx, doctree: document) -> None: # ruff: ignore[unused-function-argument] # the event's signature + # a heading links by its title's id unless that id is taken, so the changelog's repeated "Added" and "Fixed" would + # link by whichever id was left; a heading that has an explicit label, such as release-0-1-2-fixed, links by it + for section in doctree.findall(nodes.section): + names = [name for name in section["names"] if doctree.nametypes.get(name) and doctree.nameids.get(name)] + if explicit := [doctree.nameids[name] for name in names]: + section["ids"] = [*explicit, *(ident for ident in section["ids"] if ident not in explicit)] diff --git a/docs/contributing.rst b/docs/contributing.rst index 0f2c0f0..883b79f 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -4,54 +4,39 @@ Contributing How the code fits together -------------------------- -Each artifact has one pipeline in ``_generate``, and the two read alike. An adapter, the hatchling hook or a command of -the command line, gathers the options, the content provider, the build scripts and the build facts (time, source -revision and toolchain). The pipeline opens the artifact once, reads it into a tree (``_collect/_wheel.py`` or -``_collect/_archive.py``), maps the tree to an ``Artifact`` with a pure function (``_model/_wheel.py`` or -``_model/_archive.py``), and hands that to ``_model/_bom.py``, which adds what every artifact of the project shares. -:doc:`explanation` draws the packages and their imports. - -To record a new field, read it at the edge into ``_model/_inputs.py`` and write it in ``_model``; to add a subcommand, -write a ``run`` function in a new ``_cli/_.py`` and register its arguments and the -``"python_sbom._cli._:run"`` handler in ``_cli/_parser.py``, which loads the module only when the command runs. -Tests mirror the source: ``tests//test_.py``. - -What each layer guarantees --------------------------- - -A change is easiest to review against the guarantee of the layer it touches, and the test named here is the one that -fails when that guarantee breaks. Each test crosses a boundary on purpose: unit tests of one layer did not catch the -mismatches between layers these guard against. - -- ``_wheel`` reads an artifact through one open handle per build, checked once for clashing member names, and never - trusts a member's stated size. The bytes it hands on are the bytes the artifact ships. - ``tests/_wheel/test_reader.py``, and ``test_embed_rewrites_record_by_parsed_path`` for rewriting RECORD. -- ``_collect`` turns the project and the artifact into inputs. Source configuration only selects: a bundled wheel is - described from the artifact's copy, matched by digest, and a selected wheel the artifact does not ship is refused. A - provider module another project already imported in the same process is refused rather than reused. - ``test_bundled_wheel_not_shipped_is_refused``, ``test_provider_of_another_project_is_not_reused``. -- ``_model`` builds documents with pure functions. ``_model/_graph.py`` defines the sets every layer judges alike: - ``artifact_contents`` is what the artifact holds or declares wherever the document nests it, ``shipped_libraries`` is - what it carries bytes for, and ``reference_problems`` is the one check of repeated and unknown references that - generation refuses and validation reports. ``test_archive_bundled_wheel_judged_wherever_it_sits``. -- ``_validate`` reads hostile input: bounded size, depth and component count before the schema runs, then references, - the SPDX rendering and the license policy over ``artifact_contents``. - ``test_repeated_dependency_entry_cannot_hide_an_invalid_one``. -- ``_formats`` renders without losing identity: every SPDX id is unique because it ends in the full digest of its - ``bom-ref``. ``test_spdx_ids_of_refs_differing_in_dropped_characters_stay_distinct``. -- ``_generate`` orders reading, building and writing: an artifact stays open while its document is built, is closed - before it is rewritten, and is never overwritten by its own SBOM. ``test_archive_refuses_output_over_archive``, in - ``tests/_cli`` since the command line is where an output path reaches it. +:doc:`/concepts/architecture` traces one build from its entry point to the written document. Tests mirror the source: +each module has its tests at the same relative path under ``tests``, in a file named after it. + +What the code guarantees +------------------------ + +A change is easiest to review against the guarantee it touches. Each guarantee below has a test that fails when it +breaks, and most of those tests cross a boundary on purpose: unit tests of one part did not catch these mismatches. + +- Reading an artifact uses one open handle per build, checked once for clashing member names, and never trusts a + member's stated size, so the bytes described are the bytes the artifact ships. +- Source configuration only selects. A :term:`bundled wheel` is described from the artifact's copy, matched by digest, + and a selected :term:`wheel` the artifact does not ship is refused. Two projects whose :term:`content providers + ` share a module name, built in one process, each load their own copy. +- Generation and :term:`validation` judge the same sets: what the artifact holds or declares, wherever the document + nests it, and what it carries bytes for. Repeated and unknown references are one check, which generation refuses and + validation reports. +- Validation treats its input as hostile. It bounds size, nesting depth and :term:`component` count before the schema + runs, then checks references, the :term:`SPDX` rendering and the license policy. +- Every SPDX id ends in the full digest of its ``bom-ref``, so references that differ only in characters SPDX drops + still get distinct ids. +- An artifact stays open while its document is built and closes before it is rewritten, and the ``archive`` command + refuses an output path that would overwrite the :term:`archive` it describes. Updating the schemas -------------------- -``python-sbom validate`` checks documents against the official CycloneDX schema files, which ship byte for byte under -``src/python_sbom/_validate/schemas``. Each comes from a tag of `CycloneDX/specification +``python-sbom validate`` checks documents against the official :term:`CycloneDX` schema files, which ship byte for byte +under ``src/python_sbom/_validate/schemas``. Each comes from a tag of `CycloneDX/specification `_: ``bom-1.6.schema.json`` and the ``jsf-0.82.schema.json`` it references from ``1.6.2``, and ``spdx.schema.json`` and the Apache-2.0 ``LICENSE`` from ``1.7.2``, the newest, since the -cyclonedx.org URL every schema version references serves the newest license list. These commands fetch the shipped files -from their raw URLs; change the tag to take a newer release: +cyclonedx.org :term:`URL` every schema version references serves the newest license list. These commands fetch the +shipped files from their raw URLs; change the tag to take a newer release: .. code-block:: console @@ -62,16 +47,16 @@ from their raw URLs; change the tag to take a newer release: $ curl -fsSLO https://raw.githubusercontent.com/CycloneDX/specification/1.7.2/LICENSE $ sha256sum * -Put each file's tag and SHA-256 into the cases of ``test_schema_files_are_the_upstream_release`` in +Put each file's tag and :term:`SHA-256` into the cases of ``test_schema_files_are_the_upstream_release`` in ``tests/_validate/test_schemas.py``, which fails when a shipped file is not the release it names. ``.gitattributes`` -marks the directory ``-text`` and ``.pre-commit-config.yaml`` excludes it, so neither git nor the hooks rewrite the +marks the directory ``-text`` and ``.pre-commit-config.yaml`` excludes it, so neither git_ nor the hooks rewrite the files' line endings, whitespace or formatting. Benchmarks ---------- -``bench/`` times python-sbom's hot paths with `pyperf `_; CI does not run it. The -``bench`` tox environment passes its arguments to ``python -m bench``: +``bench/`` times python-sbom's hot paths with pyperf_; :term:`CI` does not run it. The ``bench`` tox_ environment passes +its arguments to ``python -m bench``: .. code-block:: console @@ -80,11 +65,11 @@ Benchmarks $ tox r -e bench -- compare ../baseline/.tox/bench-results/cpython3.15 .tox/bench-results/cpython3.15 ``prepare`` builds the fixtures, the only step that needs the network. ``run`` times each case in its own pyperf process -tree and writes one JSON file per case under ``.tox/bench-results//``; options after a second ``--`` reach -pyperf. ``profile`` runs each case once under cProfile, and under py-spy with ``--py-spy``. Both take case names to run -only those, and ``list`` prints the names. ``report`` renders result files as a Markdown table, and ``compare`` puts two -result directories side by side. The timed process imports the python-sbom its environment installed, so to compare two -revisions, run the cases from a checkout of each and compare their result directories. +tree and writes one :term:`JSON` file per case under ``.tox/bench-results//``; options after a second ``--`` +reach pyperf. ``profile`` runs each case once under cProfile, and under py-spy_ with ``--py-spy``. Both take case names +to run only those, and ``list`` prints the names. ``report`` renders result files as a Markdown table, and ``compare`` +puts two result directories side by side. The timed process imports the python-sbom its environment installed, so to +compare two revisions, run the cases from a checkout of each and compare their result directories. Changelog fragments ------------------- @@ -112,7 +97,7 @@ pull request. Supplied fragments are still checked when the label is present. Bo they include one. The release workflow links each fragment to its merged pull request and credits its human author. If several people -authored a change, put their `GitHub `_ handles in a ``by`` credit in the fragment; omit bots and -reviewers. The changelog uses linked names, while ``GitHub`` Release notes use ``@handle``. +authored a change, put their GitHub_ handles in a ``by`` credit in the fragment; omit bots and reviewers. The changelog +uses linked names, while ``GitHub`` Release notes use ``@handle``. Build the documentation with ``tox r -e docs``; ``tox r -e docs -- -b linkcheck`` checks its external links. diff --git a/docs/explanation.rst b/docs/explanation.rst deleted file mode 100644 index 3b15a12..0000000 --- a/docs/explanation.rst +++ /dev/null @@ -1,130 +0,0 @@ -How it works -============ - -Architecture ------------- - -Each concern has its own subpackage, and only the adapters know a build system or the command line. The hatchling hook -and the ``embed`` and ``archive`` commands gather the options, the build scripts and the build facts (time, revision, -toolchain), and hand them, with a :class:`~python_sbom.ContentProvider`, to the pipeline for their artifact. Each -pipeline opens its artifact once, reads it, gives the provider a :class:`~python_sbom.ProviderContext` that reads from -the same open artifact, and maps what it read to components; one function of the model then adds what every artifact -of the project shares: supplier, source links, revision, tools, dependencies and formulation. - -The diagram follows the imports between the subpackages: a solid arrow points at what a package imports when it loads, a -dotted arrow labeled *lazy* at what it imports only once a function needs it, and a dotted arrow labeled *types* at -what it imports only for type annotations. The command line loads each command's module, and what it imports, only when -that command runs. - -.. mermaid:: - - flowchart TB - subgraph adapters[Adapters] - hook[hatch
hatchling build hook] - cli[_cli
embed, archive, spdx, validate] - end - api[python_sbom
provider API, __main__] - generate[_generate
wheel and archive pipelines] - collect[_collect
settings, facts, revision, project tree, wheels, archives, providers] - validate[_validate
limits, schema, references, licenses] - formats[_formats
CycloneDX JSON, SPDX 2.3] - wheel[_wheel
artifact reader, zip reads and writes, RECORD, bundled wheel checks] - model[_model
inputs, types, identity, graph rules, document building] - errors[_errors
SbomError] - version[version] - hook --> collect - hook -. lazy .-> generate - hook -. types .-> model - cli --> generate - cli --> validate - cli --> formats - cli --> version - cli -. types .-> model - api --> cli - api --> model - generate --> collect - generate --> formats - generate --> wheel - generate --> model - collect --> wheel - collect --> model - validate --> formats - validate --> wheel - validate --> model - formats --> model - formats --> version - model --> errors - wheel --> model - classDef edge fill:#cfe4ff,stroke:#1f6feb,color:#0b1f3a; - classDef core fill:#d8f3dc,stroke:#2d6a4f,color:#081c15; - class hook,cli,collect,api edge; - class generate,validate,formats,wheel,model,errors,version core; - -Every package raises :class:`~python_sbom.SbomError`, which the command line reports without a traceback; the diagram -draws only the model's import of it, since the model imports nothing else of the package. Only the hook imports -hatchling, so the provider API and the command line work without it. - -The one extension point is the :class:`~python_sbom.ContentProvider`, which every entry point loads from the -``content-provider`` option. A hatchling custom hook can override -:meth:`~python_sbom.hatch.SbomBuildHook.content_provider` to build the provider itself. - -What the SBOM records ---------------------- - -The document's root component is the project, described from the wheel's ``METADATA``: name, version, summary, license -expression, maintainers, project URLs, classifiers and keywords. Links to the PyPI release, the wheel's attestation and, -with a repository, the license file follow. A build from a git checkout adds the commit and a link to that exact -revision. Links to the license file and build scripts point at that commit when the checkout is clean, and at the -default branch otherwise. - -CycloneDX requires each URL to be an IRI reference. A project URL or home page that is not one gets percent-encoded, -or left out when encoding cannot mend it, as with an scp-style ``git@host:path`` address; either way a -``python:project-url`` or ``python:home-page`` property keeps the text as written. - -The components below the root are the wheels the project bundles and the runtime dependencies it declares. A bundled -wheel carries its SHA-256, its license, a component per file from its ``RECORD``, and the packages it vendors. A -declared dependency carries no version, because the installer picks it at install time; the document marks the root's -dependencies incomplete for that reason. An extra's dependencies are left out. - -The formulation records how the wheel was built: the interpreter's version and implementation, every distribution of the -build environment with its dependencies, the files and hashes of each pure-Python one, and ``SOURCE_DATE_EPOCH`` when it -set the build time. - -Property names start with the project's normalized name, such as ``virtualenv:bundled-wheel``, or with ``python:`` for -values copied from core metadata. - -Reproducibility ---------------- - -The document holds nothing about the build machine. Its timestamp comes from ``SOURCE_DATE_EPOCH``. When that is unset, -a reproducible hatchling build uses hatchling's fixed 2020-02-02, the time it stamps on the wheel's entries, and any -other build uses the current time. Its serial number derives from the content, so identical documents share one and any -difference gets a new one. - -The build tools are listed down to their files, so a rebuild can check it installed the same bytes. Only what is the -same on every machine goes in: platform wheels appear without their files, because those differ per operating system and -architecture, and editable installs without theirs, because they point at a checkout. Installer metadata, console-script -launchers and uv's cache files are left out too, since they name the frontend, the environment's path or a time rather -than the distribution. - -Hashing each tool's ``RECORD`` instead would be smaller, but that file lists the installer metadata, so two frontends -installing the same tools would give two hashes. The cost is size: the file list grows with the build environment, and -a large build tool such as setuptools adds hundreds of entries. - -So a rebuild reproduces the wheel byte for byte on any operating system when these match the original build: the source -tree, as a checkout of the same commit; ``SOURCE_DATE_EPOCH``; the Python version; and the versions of the build backend -and its dependencies. The SBOM records the last three, so it says what to install before rebuilding. - -Limitations ------------ - -- The document describes what the project declares and bundles, not the full tree of runtime dependencies, which depends - on the environment the wheel is installed into. -- The links to the PyPI release and attestation assume the project publishes to PyPI with attestations. -- A wheel built from an sdist of a dirty checkout names no commit, since no commit reproduces it. -- ``python-sbom embed`` and ``archive`` record the packages of the environment ``--build-env`` names, by default the one - they run in, and always the interpreter they run on. Unless that is the environment the artifact was built in, the - build tool record does not match the real build. -- The SPDX rendering leaves out per-file hashes and the build record, which SPDX 2.3 cannot hold. -- A tool that renames the wheel after the build, such as ``auditwheel repair``, leaves the attestation link pointing at - the old file name until ``python-sbom embed`` runs on the renamed wheel. diff --git a/docs/extend.rst b/docs/extend.rst deleted file mode 100644 index 9ec9dcf..0000000 --- a/docs/extend.rst +++ /dev/null @@ -1,113 +0,0 @@ -Extend it -========= - -The options cover what every project shares. A project that knows more than its metadata says adds it through a content -provider, and the same provider serves the wheel's SBOM and the SBOM of any other archive the project publishes. - -Add content with a provider ---------------------------- - -A :class:`~python_sbom.ContentProvider` has two methods. Each receives a :class:`~python_sbom.ProviderContext`: the -project directory, its :class:`~python_sbom.Settings`, its :class:`~python_sbom.ProjectIdentity` with the prefix of its -properties, and :meth:`~python_sbom.ProviderContext.read`, which reads a file of the artifact being described, a member -of the built wheel or of the archive. Each returns a value whose fields default to empty, so a provider names only what -it adds. - -- :meth:`~python_sbom.ContentProvider.bundled_wheel` returns the properties and the supplier of one bundled wheel. The - properties follow ``:bundled-wheel`` and come before those from the wheel's metadata. A wheel's supplier comes - only from here, never from the wheel's own metadata. -- :meth:`~python_sbom.ContentProvider.contributions` returns extra top-level components, the dependencies between them, - root properties and build tools, such as the C libraries a native wheel links. They join the document before the - root's dependencies and the serial number derive from it. The build fails when a component or tool lacks ``type``, - ``bom-ref`` or ``name``, a ``bom-ref`` repeats, the ``formula:`` and ``workflow:`` refs of the build's formulation - included, a dependency names one that no component has, a ``bom-ref`` gets a second dependency entry (the root's - included, which python-sbom writes), or a root property uses the project's own prefix. - -Name the provider in the ``content-provider`` option, as ``module:attribute`` importable from the project root, and the -hook, ``python-sbom embed`` and ``python-sbom archive`` all load it. When the provider's class lives in the project, the -SBOM lists its module among the build tools, hashed, because it shapes the document a rebuild has to reproduce: - -.. code-block:: python - - from python_sbom import ContentProvider, Contributions - - - class Content(ContentProvider): - def contributions(self, context): - zlib = {'type': 'library', 'bom-ref': 'pkg:generic/zlib@1.3.1', 'name': 'zlib', 'version': '1.3.1'} - return Contributions(components=[zlib]) - -The context has no method that lists files. A provider that needs the wheel's file list reads its ``RECORD`` through -:meth:`~python_sbom.ProviderContext.read`, by the path of the project's ``.dist-info`` directory, such as -``seeder-1.0.dist-info/RECORD``; an archive holds that directory at its top level, and the path reads the same there. The -provider knows the project's name, and a hook subclass, shown below, can hand it hatchling's ``self.metadata.version``. -Any other member reads the same way, by its path inside the artifact. - -Worked example: a seeder ------------------------- - -A seeder, a project like `virtualenv `_, ships the wheels it installs into new -environments under ``src/seeder/embed``. A ``BUNDLE_SUPPORT`` table in that package's ``__init__.py``, after a -``BUNDLE_FOLDER`` assignment, decides which Python versions receive each wheel. The project's options name the supplier -and the wheels, and enable hatchling's custom hook, which loads ``hatch_build.py`` from the project root: - -.. literalinclude:: ../tests/_seeder/pyproject.toml - :language: toml - :start-at: [tool.hatch] - -The hook script subclasses :class:`~python_sbom.hatch.SbomBuildHook` and overrides -:meth:`~python_sbom.hatch.SbomBuildHook.content_provider`, so it builds the provider with the table's name, an argument -the ``content-provider`` option has no way to pass. ``get_build_hook`` tells hatchling which class to load, since the -script also imports the base class. The provider reads the table next to each wheel through the context, by name, -without importing the package, and names the project's supplier as the wheels' supplier, since the seeder bundles only -that supplier's wheels. The test suite builds this project: - -.. literalinclude:: ../tests/_seeder/hatch_build.py - :language: python - -The SBOM lists ``hatch_build.py`` among the build tools, hashed. The bundled pip wheel's component then carries these -properties first, in this order, followed by those from its own metadata: - -.. list-table:: - :header-rows: 1 - - * - Property - - Value - * - ``seeder:bundled-wheel`` - - ``seeder/embed/pip-1.0-py3-none-any.whl`` - * - ``seeder:seeded-for-python`` - - ``3.14`` - * - ``seeder:seeded-for-python`` - - ``3.15`` - -A bundled wheel is named by its path inside the artifact, so the wheel's SBOM and an archive's give it the same -``bom-ref``. - -A hook subclass serves only the hatchling build: ``python-sbom embed`` and ``python-sbom archive`` load the provider the -``content-provider`` option or their ``--content-provider`` names, and build a class without arguments. A project that -wants the same provider in every SBOM it publishes gives the provider a constructor without arguments, or a module-level -instance, and names it in the option instead of overriding the hook. - -Describe another archive ------------------------- - -A project may also publish a zipapp, such as virtualenv's ``virtualenv.pyz``, which holds virtualenv's own -``.dist-info`` at its top level, the wheels it seeds, and its dependencies unpacked under ``__virtualenv__/``. -``python-sbom archive`` describes such an archive the way the hook describes a wheel: the archive's own files, the -project's distribution, each wheel it bundles as a ``.whl`` member, and each one unpacked into a directory with a -``.dist-info``. The content provider the option names hears of every bundled wheel, and its context reads the archive's -files: - -.. code-block:: console - - $ python-sbom archive virtualenv.pyz --build-script tasks/make_zipapp.py --embed - wrote virtualenv.pyz.cdx.json - wrote virtualenv.pyz.cdx.json into virtualenv.pyz - -The document takes its options, source revision and toolchain the way the wheel's does, so both name the project alike. -Its build time comes from ``SOURCE_DATE_EPOCH``, or the current time without it; set ``SOURCE_DATE_EPOCH`` for both -builds to give them one time. ``--embed`` adds the document to the archive's top level, where ``python-sbom validate -virtualenv.pyz`` finds it. It goes in deflated at level 9 whatever the archive's other members use, since a zipapp may -store them all and the SBOM, megabytes of JSON, would then add its full size. The entry carries a fixed time and Unix -attributes and the other members keep their bytes, so adding it to one archive gives the same bytes on every system with -the same zlib. diff --git a/docs/how-to/archive.rst b/docs/how-to/archive.rst new file mode 100644 index 0000000..512dedb --- /dev/null +++ b/docs/how-to/archive.rst @@ -0,0 +1,24 @@ +###################################### + Describe a zipapp or another archive +###################################### + +A project may also publish a :term:`zipapp`, such as virtualenv_'s ``virtualenv.pyz``, which holds virtualenv's own +``.dist-info`` at its top level, the :term:`wheels ` it seeds, and its dependencies unpacked under +``__virtualenv__/``. ``python-sbom archive`` describes such an :term:`archive` the way the hook describes a wheel: the +archive's own files, the project's distribution, each wheel it bundles as a ``.whl`` member, and each one unpacked into +a directory with a ``.dist-info``. The :term:`content provider` the option names hears of every :term:`bundled wheel`, +and its context reads the archive's files: + +.. code-block:: console + + $ python-sbom archive virtualenv.pyz --build-script tasks/make_zipapp.py --embed + wrote virtualenv.pyz.cdx.json + wrote virtualenv.pyz.cdx.json into virtualenv.pyz + +The document takes its options, :term:`source revision` and toolchain the way the wheel's does, so both name the project +alike. Its build time comes from ``SOURCE_DATE_EPOCH``, or the current time without it; set ``SOURCE_DATE_EPOCH`` for +both builds to give them one time. ``--embed`` adds the document to the archive's top level, where ``python-sbom +validate virtualenv.pyz`` finds it. It goes in deflated at level 9 whatever the archive's other members use, since a +zipapp may store them all and the :term:`SBOM`, megabytes of :term:`JSON`, would then add its full size. The entry +carries a fixed time and Unix attributes and the other members keep their bytes, so adding it to one archive gives the +same bytes on every system with the same zlib. diff --git a/docs/how-to/built-elsewhere.rst b/docs/how-to/built-elsewhere.rst new file mode 100644 index 0000000..c246cda --- /dev/null +++ b/docs/how-to/built-elsewhere.rst @@ -0,0 +1,63 @@ +#################################################### + Describe a wheel built outside the build interface +#################################################### + +Some :term:`wheels ` never pass through a python-sbom plugin or the :term:`wrapper backend`: maturin-action_ +calls ``maturin build`` directly, and ``auditwheel repair`` or ``delocate`` rename a wheel after the build and may copy +libraries into it. ``python-sbom embed`` describes such a wheel after the fact. It writes the same document the plugins +write, adds its ``RECORD`` row, keeps every other entry of the wheel as it was, and replaces an :term:`SBOM` python-sbom +wrote before. + +*********************** + Embed after the build +*********************** + +Run it from the project root, so it finds ``[tool.python-sbom]``: + +.. code-block:: console + + $ maturin build --release --out dist + $ python-sbom embed dist/*.whl + wrote matex-1.0.dist-info/sboms/matex.cdx.json into matex-1.0-py3-none-manylinux_2_17_x86_64.whl + +***************************** + Record the real build tools +***************************** + +``embed`` lists the packages of the environment ``--build-env`` names, by default the one it runs in, and always the +interpreter it runs on. Unless that is the environment the wheel was built in, the record does not match the real build. +Either build without isolation in the environment ``embed`` runs in, installed from the same lock: + +.. code-block:: console + + $ python -m pip install build python-sbom meson-python meson ninja patchelf + $ python -m build --wheel --no-isolation + $ python-sbom embed dist/*.whl + +or install the build requirements into an environment of their own and point ``--build-env`` at its ``site-packages``: + +.. code-block:: console + + $ python -m venv build-env && build-env/bin/pip install maturin + $ python-sbom embed dist/*.whl --build-env build-env/lib/python3.15/site-packages + +**************** + After a repair +**************** + +Run ``embed`` on the repaired wheel, so the SBOM names the file you publish: + +.. code-block:: console + + $ auditwheel repair dist/*.whl --wheel-dir wheelhouse + $ python-sbom embed wheelhouse/*.whl + +auditwheel_'s own SBOM, ``auditwheel.cdx.json``, stays in the wheel next to python-sbom's. + +************************** + Name the source revision +************************** + +``embed`` reads the commit from the checkout it runs in. A wheel built from an :term:`sdist` elsewhere names its commit +through ``--commit`` or the ``PYTHON_SBOM_COMMIT`` environment variable; :doc:`/reference/configuration` lists the order +in which python-sbom looks for it. diff --git a/docs/how-to/content-provider.rst b/docs/how-to/content-provider.rst new file mode 100644 index 0000000..31e5c1d --- /dev/null +++ b/docs/how-to/content-provider.rst @@ -0,0 +1,99 @@ +############################# + Add what your project knows +############################# + +The options cover what every project shares. A project that knows more than its metadata says adds it through a content +provider, and the same provider serves the :term:`wheel`'s :term:`SBOM` and the SBOM of any other :term:`archive` the +project publishes, through every integration. + +****************** + Write a provider +****************** + +A :class:`~python_sbom.ContentProvider` has two methods. Each receives a :class:`~python_sbom.ProviderContext`: the +project directory, its :class:`~python_sbom.Settings`, its :class:`~python_sbom.ProjectIdentity` with the prefix of its +properties, and :meth:`~python_sbom.ProviderContext.read`, which reads a file of the artifact being described, a member +of the built wheel or of the archive. Each returns a value whose fields default to empty, so a provider names only what +it adds. + +- :meth:`~python_sbom.ContentProvider.bundled_wheel` returns the properties and the :term:`supplier` of one + :term:`bundled wheel`. The properties follow ``:bundled-wheel`` and come before those from the wheel's + metadata. A wheel's supplier comes only from here, never from the wheel's own metadata. +- :meth:`~python_sbom.ContentProvider.contributions` returns extra top-level :term:`components `, the + dependencies between them, root properties and build tools, such as the C libraries a native wheel links. They join + the document before the root's dependencies and the serial number derive from it. The build fails when a component or + tool lacks ``type``, ``bom-ref`` or ``name``, a ``bom-ref`` repeats, the ``formula:`` and ``workflow:`` refs of the + build's :term:`formulation` included, a dependency names one that no component has, a ``bom-ref`` gets a second + dependency entry (the root's included, which python-sbom writes), or a root property uses the project's own prefix. + +Name the provider in the ``content-provider`` option, as ``module:attribute`` importable from the project root, and +every plugin, the :term:`wrapper backend`, ``python-sbom embed`` and ``python-sbom archive`` load it. When the +provider's class lives in the project, the SBOM lists its module among the build tools, hashed, because it shapes the +document a rebuild has to reproduce: + +.. code-block:: python + + from python_sbom import ContentProvider, Contributions + + + class Content(ContentProvider): + def contributions(self, context): + zlib = { + 'type': 'library', + 'bom-ref': 'pkg:generic/zlib@1.3.1', + 'name': 'zlib', + 'version': '1.3.1', + } + return Contributions(components=[zlib]) + +The context has no method that lists files. A provider that needs the wheel's file list reads its ``RECORD`` through +:meth:`~python_sbom.ProviderContext.read`, by the path of the project's ``.dist-info`` directory, such as +``seeder-1.0.dist-info/RECORD``; an archive holds that directory at its top level, and the path reads the same there. +The provider knows the project's name, and a hook subclass, shown below, can hand it hatchling_'s +``self.metadata.version``. Any other member reads the same way, by its path inside the artifact. + +************************** + Worked example: a seeder +************************** + +A seeder, a project like `virtualenv `_, ships the wheels it installs into new +environments under ``src/seeder/embed``. A ``BUNDLE_SUPPORT`` table in that package's ``__init__.py``, after a +``BUNDLE_FOLDER`` assignment, decides which Python versions receive each wheel. The project's options name the supplier +and the wheels, and enable hatchling's custom hook, which loads ``hatch_build.py`` from the project root: + +.. literalinclude:: ../../tests/_seeder/pyproject.toml + :language: toml + :start-at: [tool.hatch] + +The hook script subclasses :class:`~python_sbom.hatch.SbomBuildHook` and overrides +:meth:`~python_sbom.hatch.SbomBuildHook.content_provider`, so it builds the provider with the table's name, an argument +the ``content-provider`` option has no way to pass. ``get_build_hook`` tells hatchling which class to load, since the +script also imports the base class. The provider reads the table next to each wheel through the context, by name, +without importing the package, and names the project's supplier as the wheels' supplier, since the seeder bundles only +that supplier's wheels. The test suite builds this project: + +.. literalinclude:: ../../tests/_seeder/hatch_build.py + :language: python + +The SBOM lists ``hatch_build.py`` among the build tools, hashed. The bundled pip_ wheel's component then carries these +properties first, in this order, followed by those from its own metadata: + +.. list-table:: + :header-rows: 1 + + - - Property + - Value + - - ``seeder:bundled-wheel`` + - ``seeder/embed/pip-1.0-py3-none-any.whl`` + - - ``seeder:seeded-for-python`` + - ``3.14`` + - - ``seeder:seeded-for-python`` + - ``3.15`` + +A bundled wheel is named by its path inside the artifact, so the wheel's SBOM and an archive's give it the same +``bom-ref``. + +A hook subclass serves only the hatchling build: the other integrations, ``python-sbom embed`` and ``python-sbom +archive`` load the provider the ``content-provider`` option or ``--content-provider`` names, and build a class without +arguments. A project that wants the same provider in every SBOM it publishes gives the provider a constructor without +arguments, or a module-level instance, and names it in the option instead of overriding the hook. diff --git a/docs/how-to/hatchling.rst b/docs/how-to/hatchling.rst new file mode 100644 index 0000000..bba3f24 --- /dev/null +++ b/docs/how-to/hatchling.rst @@ -0,0 +1,77 @@ +####################### + Use it with hatchling +####################### + +python-sbom ships a hatchling_ :term:`build hook`. Add it to the build requirements, with the ``hatch`` extra, and +enable it with an empty table: + +.. code-block:: toml + + [build-system] + build-backend = 'hatchling.build' + requires = ['hatchling>=1.28', 'python-sbom[hatch]>=0.2'] + + [tool.hatch.build.hooks.python-sbom] + + [tool.python-sbom] + supplier = { name = 'tox-dev', url = ['https://github.com/tox-dev'] } + +Every :term:`wheel` you build now carries the :term:`SBOM`. An editable wheel carries none, and an :term:`sdist` carries +the :term:`source revision` instead, so a wheel built from it still names its commit. + +Put python-sbom's options in ``[tool.python-sbom]``, the table every integration reads, never in the hook's table: the +hook's table holds only hatchling's own options, such as ``enable-by-default``, and a python-sbom option there fails the +build with a pointer to ``[tool.python-sbom]``. :doc:`/reference/configuration` lists the options. + +The hook writes the SBOM once hatchling has written the wheel, so the document describes the final file name and +metadata, whatever dependencies or tag other hooks changed. The order of the hook tables does not matter. + +*************************************************** + Projects that still build on Python 3.10 or older +*************************************************** + +python-sbom needs Python 3.11 or later. A project that still builds on older interpreters guards the requirement with a +marker. An enabled ``python-sbom`` hook table would then fail those builds, since hatchling refuses a hook it cannot +find, so enable the hook from a custom hook script that falls back to a hook doing nothing: + +.. code-block:: text + + [build-system] + requires = ['hatchling>=1.28', '''python-sbom[hatch]>=0.2; python_version>='3.11''''] + + [tool.hatch.build.hooks.custom] + path = 'hatch_build.py' + +.. code-block:: python + + # hatch_build.py + import sys + + if sys.version_info >= (3, 11): + from python_sbom.hatch import SbomBuildHook + else: # no python-sbom here, so these builds carry no SBOM + from hatchling.builders.hooks.plugin.interface import ( + BuildHookInterface as SbomBuildHook, + ) + + + def get_build_hook(): + return SbomBuildHook + +Wheels built on Python 3.11 or later carry the SBOM; publish those. + +******************** + From a custom hook +******************** + +A project with its own hatchling custom hook can subclass :class:`~python_sbom.hatch.SbomBuildHook` instead of enabling +the plugin, and override :meth:`~python_sbom.hatch.SbomBuildHook.content_provider` to hand python-sbom what the hook +knows. :doc:`/how-to/content-provider` walks through one. + +***************************************** + Release builds that install from a lock +***************************************** + +A release workflow that installs ``[build-system] requires`` from a lock with hashes, such as `tox-dev/github-workflows +`_ does, needs python-sbom and its dependencies (``installer``, +``packageurl-python``, ``packaging`` and ``zipremove``) in the locked group, or the build stops on a missing hash. diff --git a/docs/how-to/index.rst b/docs/how-to/index.rst new file mode 100644 index 0000000..11d90d8 --- /dev/null +++ b/docs/how-to/index.rst @@ -0,0 +1,20 @@ +############### + How-to guides +############### + +Each guide solves one task and assumes you know what python-sbom does, as the :doc:`/tutorials/index` show. The first +four set it up for a :term:`build backend`; the rest describe other artifacts, add what a project knows, and check the +result. The :doc:`/reference/index` lists every option and command, and :doc:`/concepts/index` explains the design. + +.. toctree:: + :maxdepth: 1 + + hatchling + pdm-backend + setuptools + other-backends + built-elsewhere + archive + content-provider + spdx + validate diff --git a/docs/how-to/other-backends.rst b/docs/how-to/other-backends.rst new file mode 100644 index 0000000..c479f08 --- /dev/null +++ b/docs/how-to/other-backends.rst @@ -0,0 +1,92 @@ +############################### + Use it with any other backend +############################### + +Backends without a plugin interface, such as meson-python_, maturin_, scikit-build-core_, flit-core_, poetry-core_ and +uv_build_, get the :term:`SBOM` through ``python_sbom.backend``, a :term:`build backend` that wraps yours. Make it the +project's backend, keep yours in the build requirements, and name yours in ``[tool.python-sbom]``: + +.. code-block:: toml + + [build-system] + build-backend = 'python_sbom.backend' + requires = ['meson-python>=0.18', 'meson>=1.6', 'ninja', 'python-sbom>=0.2'] + + [tool.python-sbom] + build-backend = 'mesonpy' + supplier = { name = 'tox-dev', url = ['https://github.com/tox-dev'] } + +``build-backend`` takes what ``[build-system] build-backend`` took before: a module, or ``module:object``. Every hook a +:term:`frontend ` calls goes to your backend unchanged, ``--config-settings`` included, and the optional +hooks exist only when your backend defines them, so frontends behave as before. Once your backend has written a +:term:`wheel`, the wrapper writes the SBOM into it; once it has written an :term:`sdist`, it adds the :term:`source +revision`. + +The wrapper runs inside the build's isolated environment, so the SBOM lists the tools that built the wheel, such as +meson-python, `meson `_, ninja_ and patchelf, with the versions the frontend installed. + +************** + Each backend +************** + +Use the module name from your current ``[build-system] build-backend``: + +.. list-table:: + :header-rows: 1 + + - - Backend + - ``[tool.python-sbom] build-backend`` + - Notes + - - `meson-python `_ + - ``mesonpy`` + - Its sdist step needs a git_ or Mercurial checkout, as without the wrapper. + - - maturin_ + - ``maturin`` + - maturin writes its own SBOM of the Rust crates next to python-sbom's; see below. + - - scikit-build-core_ + - ``scikit_build_core.build`` + - A project built with hatchling_ and scikit-build-core's hatch plugin uses :doc:`/how-to/hatchling` instead. + - - flit-core_ + - ``flit_core.buildapi`` + - + - - `poetry-core `_ + - ``poetry.core.masonry.api`` + - + - - uv_build_ + - ``uv_build`` + - ``uv build`` builds a uv_build project in its own process, skipping the build interface; with the wrapper as the + backend, it calls the build interface, so the wrapper runs. + - - pdm-backend_, setuptools_ + - ``pdm.backend``, ``setuptools.build_meta`` + - Use :doc:`/how-to/pdm-backend` or :doc:`/how-to/setuptools` instead; wrap them only when the plugin cannot act. + +maturin's own SBOM +================== + +`maturin writes a CycloneDX SBOM `_ of the Rust crates a wheel links, as +``.cyclonedx.json``. :term:`PEP 770` allows several SBOMs per wheel, so python-sbom's ``.cdx.json`` sits +next to it: maturin's lists the crates, python-sbom's the wheel's files, metadata, build tools and source revision. + +*********************************************** + Pitfall: builds that skip the build interface +*********************************************** + +The wrapper runs only when a frontend builds through the standard build interface: pip_, uv_, ``python -m build`` and +cibuildwheel_ do. Tools that call the backend's own command line do not: ``maturin build``, which maturin-action_ runs, +and ``meson`` called directly. Describe those wheels after the build, as :doc:`/how-to/built-elsewhere` shows. + +************************************* + Pitfall: a backend in your own tree +************************************* + +The wrapper cannot wrap a backend the project ships in its own tree. Such a backend loads through ``[build-system] +backend-path``, and a frontend loads only a backend found on that path, so it refuses ``python_sbom.backend``, which +lives in the build environment. Describe those wheels after the build, as :doc:`/how-to/built-elsewhere` shows. + +************************************************* + Pitfall: tools that rename the wheel afterwards +************************************************* + +``auditwheel repair`` and ``delocate`` rename a wheel to its final platform tag and may copy shared libraries into it, +so the SBOM the wrapper wrote names the old file and misses those libraries. Run ``python-sbom embed`` on the repaired +wheel, as :doc:`/how-to/built-elsewhere` shows. diff --git a/docs/how-to/pdm-backend.rst b/docs/how-to/pdm-backend.rst new file mode 100644 index 0000000..25f1b17 --- /dev/null +++ b/docs/how-to/pdm-backend.rst @@ -0,0 +1,27 @@ +######################### + Use it with pdm-backend +######################### + +python-sbom ships a pdm-backend_ :term:`build hook`, registered in pdm-backend's ``pdm.build.hook`` entry point group. +Add python-sbom to the build requirements, with the ``pdm`` extra, and add a ``[tool.python-sbom]`` table, empty or not: + +.. code-block:: toml + + [build-system] + build-backend = 'pdm.backend' + requires = ['pdm-backend>=2.4', 'python-sbom[pdm]>=0.2'] + + [tool.python-sbom] + supplier = { name = 'tox-dev', url = ['https://github.com/tox-dev'] } + +Every :term:`wheel` you build now carries the :term:`SBOM`, and every :term:`sdist` the :term:`source revision`. An +editable wheel carries none. + +pdm-backend loads every hook installed in the build environment, so the table is what turns this one on: a project +without ``[tool.python-sbom]`` builds as before even when python-sbom is installed. + +The hook runs in pdm-backend's ``pdm_build_finalize`` step, once the wheel is written, so the document describes the +final file name and metadata. When ``SOURCE_DATE_EPOCH`` is unset, pdm-backend dates every wheel entry 2016-01-01, and +the SBOM takes the same time, so a rebuild reproduces both. + +A project with its own ``pdm_build.py`` keeps it; both hooks run. diff --git a/docs/how-to/setuptools.rst b/docs/how-to/setuptools.rst new file mode 100644 index 0000000..680c00a --- /dev/null +++ b/docs/how-to/setuptools.rst @@ -0,0 +1,49 @@ +######################## + Use it with setuptools +######################## + +python-sbom ships a setuptools_ plugin, registered in setuptools' ``setuptools.finalize_distribution_options`` entry +point group, the one setuptools-scm_ uses. Add python-sbom to the build requirements, with the ``setuptools`` extra, and +add a ``[tool.python-sbom]`` table, empty or not: + +.. code-block:: toml + + [build-system] + build-backend = 'setuptools.build_meta' + requires = ['setuptools>=77', 'python-sbom[setuptools]>=0.2'] + + [tool.python-sbom] + supplier = { name = 'tox-dev', url = ['https://github.com/tox-dev'] } + +Every :term:`wheel` you build now carries the :term:`SBOM`, and every :term:`sdist` the :term:`source revision`. An +editable wheel carries none. + +setuptools calls the plugin for every build in an environment where python-sbom is installed, so the table is what turns +it on: a project without ``[tool.python-sbom]``, or without a ``pyproject.toml``, builds as before. + +The plugin wraps the ``bdist_wheel`` and ``sdist`` commands the project uses, including subclasses it passes to +``setup()`` as ``cmdclass``, and writes the SBOM once the wheel is written. + +********************************************* + Pitfall: commands set in ``pyproject.toml`` +********************************************* + +setuptools applies ``[tool.setuptools.cmdclass]`` after the plugin runs, so a ``bdist_wheel`` or ``sdist`` named there +would replace the plugin's and the wheel would get no SBOM. The plugin stops such a build with an error instead. Move +that command to ``setup.py``'s ``cmdclass``, or wrap the backend instead, as :doc:`/how-to/other-backends` shows with +``build-backend = 'setuptools.build_meta'``. + +*************************************** + Pitfall: build output in the checkout +*************************************** + +setuptools writes ``build/`` and ``.egg-info/`` into the project directory, and :term:`frontends ` +run ``egg_info`` before they ask for the wheel or sdist. python-sbom counts untracked files as changes, so unless git_ +ignores both, every build reads the checkout as modified: the wheel's SBOM names no exact commit, and the sdist carries +no source revision. Ignore them: + +.. code-block:: text + + build/ + dist/ + *.egg-info/ diff --git a/docs/how-to/spdx.rst b/docs/how-to/spdx.rst new file mode 100644 index 0000000..a43b0ac --- /dev/null +++ b/docs/how-to/spdx.rst @@ -0,0 +1,21 @@ +######################## + Render an SBOM as SPDX +######################## + +Some tools read only :term:`SPDX`. ``python-sbom spdx`` renders the :term:`CycloneDX` document as SPDX 2.3 :term:`JSON`. +It needs no extra: + +.. code-block:: console + + $ unzip -p dist/project-1.0-py3-none-any.whl '*/sboms/project.cdx.json' > project.cdx.json + $ python-sbom spdx project.cdx.json project.spdx.json + +The SPDX document keeps the packages, their hashes, licenses and purls, and the containment, dependency and build tool +relationships. Its namespace ends in the CycloneDX serial number, so both documents name the same build. + +It leaves out two things SPDX 2.3 cannot hold: the per-file hashes, because SPDX 2.3 wants a :term:`SHA-1` for every +file and wheel ``RECORD`` files hold :term:`SHA-256`, and the build record. Keep the CycloneDX document when you need +those. + +The command checks only the fields the rendering reads. To check the whole document, run ``python-sbom validate``, which +also validates the SPDX rendering; see :doc:`/how-to/validate`. diff --git a/docs/how-to/validate.rst b/docs/how-to/validate.rst new file mode 100644 index 0000000..e41b556 --- /dev/null +++ b/docs/how-to/validate.rst @@ -0,0 +1,47 @@ +################### + Check SBOMs in CI +################### + +``python-sbom validate`` checks every :term:`CycloneDX` :term:`SBOM` in each :term:`wheel`'s ``.dist-info/sboms`` and in +each :term:`archive` it is given. Install it with the ``validate`` extra, in a tool environment of its own; it says so +when the extra is missing: + +.. code-block:: console + + $ pipx install 'python-sbom[validate]' + $ python-sbom validate dist/ + +Run it after the build and before the upload, on the files you publish. It fails the job when a wheel has no SBOM, so it +also catches a build that lost the plugin. + +**************** + What it checks +**************** + +Every SBOM gets the CycloneDX 1.6 schema check. For the SBOM python-sbom wrote, it also requires the ``urn:uuid`` serial +number `actions/attest`_ needs to recognize the format, checks that every reference resolves, and validates the +:term:`SPDX` rendering. An SBOM another tool wrote, such as auditwheel_'s or maturin_'s, gets the schema check alone. + +************************************** + Limit the licenses of bundled wheels +************************************** + +Each ``--allow-license`` adds an SPDX id that :term:`bundled wheels ` may use. With any given, a bundled +wheel whose license the ids do not allow fails the check, and so does one that declares a license only by name: + +.. code-block:: console + + $ python-sbom validate dist/ --allow-license MIT --allow-license Apache-2.0 + +******************** + Check a large SBOM +******************** + +``validate`` refuses an SBOM with more than 100,000 :term:`components ` before it checks anything else, which +bounds what an untrusted SBOM can ask of a :term:`CI` job. Raise the cap for an SBOM you expect to be larger: + +.. code-block:: console + + $ python-sbom validate dist/ --max-components 500000 + +:doc:`/reference/configuration` lists the limits and why they sit where they do. diff --git a/docs/index.rst b/docs/index.rst index 70083ca..10863b6 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,43 +1,196 @@ -python-sbom -=========== +############# + python-sbom +############# -SBOMs for Python distributions. python-sbom writes a `CycloneDX `_ 1.6 SBOM into every wheel you -build, at ``-.dist-info/sboms/.cdx.json``. That is where `PEP 770 -`_ tells installers to keep SBOMs, so they land in the environment next to the -installed package, where scanners look for them. +.. raw:: html -Why a wheel needs its own SBOM ------------------------------- +
+ + +

Every wheel you ship, + with its SBOM inside.

+

A build plugin that writes a + software bill of materials (SBOM) in the + CycloneDX format into each + wheel your + build backend makes, where + PEP 770 (a Python Enhancement Proposal) tells installers and + scanners to look for it.

+ +
+ +python-sbom writes a :term:`CycloneDX` 1.6 :term:`SBOM` into every :term:`wheel` you build, at +``-.dist-info/sboms/.cdx.json``. That is where :term:`PEP 770` tells installers to keep SBOMs, so +they land in the environment next to the installed package, where scanners look for them. General-purpose scanners read the dependencies a package declares or the packages an environment has installed. They -cannot see a wheel your project ships as a data file, and they cannot say how the wheel was built. The SBOM this package -writes records both: the wheels you bundle, with their hashes, licenses and vendored packages, and the Python version -and packages of the build environment. A third party can use that record to rebuild the wheel and compare the bytes. +cannot see a wheel your project ships as a data file, and they cannot say how the wheel was built. The SBOM python-sbom +writes records both. The :doc:`changelog` lists the changes in each version. + +.. grid:: 1 1 2 2 + :gutter: 3 + + .. grid-item-card:: Every build backend + :class-card: feature + :link: /reference/backends + :link-type: doc + + A plugin for hatchling_, pdm-backend_ and setuptools_, and a :term:`wrapper backend` for meson-python_, + maturin_ and the rest. + + .. grid-item-card:: How the wheel was built + :class-card: feature + :link: reproducibility + :link-type: ref + + The source commit, the interpreter and every build tool down to its files, so a third party can rebuild the + wheel and compare the bytes. + + .. grid-item-card:: What the wheel bundles + :class-card: feature + :link: bundled-wheels + :link-type: ref + + Each wheel a project ships as data, with its hashes, license, files and the packages it vendors. + + .. grid-item-card:: Checked before upload + :class-card: feature + :link: /how-to/validate + :link-type: doc + + ``python-sbom validate`` checks every SBOM against the CycloneDX schema in :term:`CI`, and ``spdx`` renders it + as :term:`SPDX` 2.3. + +A build, the SBOM it wrote, and the check a CI job runs before publishing: + +.. code-block:: console + + $ uv build + Successfully built dist/greet-1.0.tar.gz + Successfully built dist/greet-1.0-py3-none-any.whl + $ unzip -Z1 dist/greet-1.0-py3-none-any.whl | grep sboms + greet-1.0.dist-info/sboms/greet.cdx.json + $ python-sbom validate dist/ --allow-license MIT + SBOM in greet-1.0-py3-none-any.whl is valid + +********* + Install +********* + +python-sbom runs inside your build, so it goes in ``[build-system] requires`` with the extra for your backend, and a +``[tool.python-sbom]`` table names you as the :term:`supplier`. It needs Python 3.11 or later. + +.. tab:: hatchling + + .. code-block:: toml + + [build-system] + build-backend = 'hatchling.build' + requires = ['hatchling>=1.28', 'python-sbom[hatch]>=0.2'] + + [tool.hatch.build.hooks.python-sbom] + + [tool.python-sbom] + supplier = { name = 'tox-dev', url = ['https://github.com/tox-dev'] } + +.. tab:: pdm-backend + + .. code-block:: toml + + [build-system] + build-backend = 'pdm.backend' + requires = ['pdm-backend>=2.4', 'python-sbom[pdm]>=0.2'] -Quick start ------------ + [tool.python-sbom] + supplier = { name = 'tox-dev', url = ['https://github.com/tox-dev'] } -For a project built with `hatchling `_, add the hook to the build requirements and enable -it in ``pyproject.toml``: +.. tab:: setuptools -.. code-block:: toml + .. code-block:: toml - [build-system] - build-backend = 'hatchling.build' - requires = ['hatchling>=1.28', 'python-sbom[hatch]>=0.1'] + [build-system] + build-backend = 'setuptools.build_meta' + requires = ['setuptools>=77', 'python-sbom[setuptools]>=0.2'] - [tool.hatch.build.hooks.python-sbom] + [tool.python-sbom] + supplier = { name = 'tox-dev', url = ['https://github.com/tox-dev'] } -Every wheel you build now carries the SBOM. For a project built with another backend, such as meson-python, run -``python-sbom embed dist/*.whl`` after the build; :doc:`usage` shows both. +.. tab:: any other backend + + .. code-block:: toml + + [build-system] + build-backend = 'python_sbom.backend' + requires = ['meson-python>=0.18', 'meson>=1.6', 'ninja', 'python-sbom>=0.2'] + + [tool.python-sbom] + build-backend = 'mesonpy' + supplier = { name = 'tox-dev', url = ['https://github.com/tox-dev'] } + +Every wheel you build from then on carries the SBOM. :doc:`/how-to/index` covers each backend in detail. + +The command line, which describes wheels built elsewhere and checks SBOMs, installs as a tool: + +.. tab:: uv + + .. code-block:: bash + + uv tool install 'python-sbom[validate]' + python-sbom --help + +.. tab:: pipx + + .. code-block:: bash + + pipx install 'python-sbom[validate]' + python-sbom --help + +****************** + Where to go next +****************** + +.. grid:: 1 1 2 2 + :gutter: 3 + + .. grid-item-card:: New to python-sbom? + :class-card: feature + :link: /tutorials/first-sbom + :link-type: doc + + The tutorial builds a first package with an SBOM, reads it, and checks it. + + .. grid-item-card:: Have a specific task? + :class-card: feature + :link: /how-to/index + :link-type: doc + + Each guide sets python-sbom up for one backend or handles one kind of artifact. + + .. grid-item-card:: Looking up an option? + :class-card: feature + :link: /reference/index + :link-type: doc + + The backends, the configuration, the command line and the Python :term:`API`. + + .. grid-item-card:: Want to know how it works? + :class-card: feature + :link: /concepts/index + :link-type: doc + + Why each backend gets its integration, what the SBOM records, and how the code is laid out. .. toctree:: - :maxdepth: 2 - - usage - extend - explanation - cli - py_api - changelog - contributing + :hidden: + + Home + tutorials/index + how-to/index + reference/index + concepts/index + changelog + contributing diff --git a/docs/py_api.rst b/docs/reference/api.rst similarity index 51% rename from docs/py_api.rst rename to docs/reference/api.rst index a9a4bea..5514eea 100644 --- a/docs/py_api.rst +++ b/docs/reference/api.rst @@ -1,23 +1,37 @@ -Python API -========== +############ + Python API +############ -The Python API is what a project needs to add its own knowledge to the SBOM: the hatchling build hook and the content -provider it calls, with the types those take and return. Everything else, from embedding to validation, runs through the -:doc:`cli`. The core is importable from ``python_sbom`` without hatchling; the hook lives in ``python_sbom.hatch``. +The Python :term:`API` is what a :term:`build backend` loads and what a project needs to add its own knowledge to the +:term:`SBOM`: the backend integrations and the content provider they call, with the types those take and return. +Everything else, from embedding to :term:`validation`, runs through the :doc:`/reference/cli`. The core is importable +from ``python_sbom`` without any build backend; each integration lives in a module of its own and imports only its +backend. .. py:data:: python_sbom.__version__ :type: str The installed version of python-sbom. -Hatchling build hook --------------------- +********************** + Backend integrations +********************** .. autoclass:: python_sbom.hatch.SbomBuildHook :members: content_provider -Content provider ----------------- +.. autoclass:: python_sbom.pdm.SbomBuildHook + :members: pdm_build_hook_enabled, pdm_build_initialize, pdm_build_update_files, pdm_build_finalize + +.. autofunction:: python_sbom.setuptools.finalize_distribution_options + +.. autofunction:: python_sbom.backend.build_wheel + +.. autofunction:: python_sbom.backend.build_sdist + +****************** + Content provider +****************** .. autoclass:: python_sbom.ContentProvider :members: bundled_wheel, contributions @@ -33,16 +47,18 @@ Content provider .. autoclass:: python_sbom.ProjectIdentity -Component types ---------------- +***************** + Component types +***************** -These :class:`~typing.TypedDict` types describe the JSON a provider builds. Each follows the CycloneDX 1.6 object of the -same name; the `CycloneDX 1.6 reference `_ describes every key. +These :class:`~typing.TypedDict` types describe the :term:`JSON` a provider builds. Each follows the :term:`CycloneDX` +1.6 object of the same name; the `CycloneDX 1.6 reference `_ describes +every key. .. py:class:: python_sbom.Component - A CycloneDX 1.6 component, a distribution, an application, the interpreter or a file, as a dictionary with the keys - the CycloneDX component object defines. python-sbom uses ``type``, ``bom-ref``, ``name``, ``version``, ``purl``, + A CycloneDX 1.6 :term:`component`, a distribution, an application, the interpreter or a file, as a dictionary with + the keys the CycloneDX component object defines. python-sbom uses ``type``, ``bom-ref``, ``name``, ``version``, ``purl``, ``supplier`` (an :class:`~python_sbom.OrganizationalEntity`), ``authors`` (:class:`~python_sbom.Contact`), ``description``, ``licenses`` (:class:`~python_sbom.License`), ``copyright``, ``hashes`` (:class:`~python_sbom.Hash`), ``externalReferences`` (:class:`~python_sbom.ExternalReference`), ``properties`` @@ -81,7 +97,8 @@ same name; the `CycloneDX 1.6 reference ` it builds. None writes an SBOM into an editable wheel, which carries no code. diff --git a/docs/reference/cli.rst b/docs/reference/cli.rst new file mode 100644 index 0000000..dfe02a2 --- /dev/null +++ b/docs/reference/cli.rst @@ -0,0 +1,9 @@ +######################## + Command line interface +######################## + +.. sphinx_argparse_cli:: + :module: python_sbom._cli._parser + :func: build_parser + :title: + :group_title_prefix: diff --git a/docs/reference/configuration.rst b/docs/reference/configuration.rst new file mode 100644 index 0000000..b5b6784 --- /dev/null +++ b/docs/reference/configuration.rst @@ -0,0 +1,130 @@ +############### + Configuration +############### + +********************************** + The ``[tool.python-sbom]`` table +********************************** + +The options live in a ``[tool.python-sbom]`` table of the project's ``pyproject.toml``, the one place every plugin, the +:term:`wrapper backend`, ``python-sbom embed`` and ``python-sbom archive`` read, so every :term:`SBOM` of a project +names it alike. For the pdm-backend_ and setuptools_ plugins, the table's presence, even empty, is what turns them on. +Each option is optional; an unknown key fails the build, so a typo cannot pass silently. + +``supplier`` + The organization that supplies the project: a table with a ``name`` string and an optional ``url`` string array. It + becomes the :term:`supplier` and manufacturer of the document and the supplier of the project. Left out, the + document names no supplier. + + .. code-block:: toml + + [tool.python-sbom] + supplier = { name = 'Python Packaging Authority', url = ['https://www.pypa.io'] } + +``repository`` + The repository :term:`URL`, used for the link to the :term:`source revision`, the license file and, on GitHub_, the + security policy and advisories. It defaults to the first ``Source``, ``Repository`` or ``GitHub`` URL in + ``[project.urls]``. + + .. code-block:: toml + + [tool.python-sbom] + repository = 'https://gitlab.com/example/project' + +``default-branch`` + The branch in links to files of the repository when the build's commit is unknown or the checkout has uncommitted + changes, ``main`` by default; otherwise the links name the commit. + + .. code-block:: toml + + [tool.python-sbom] + default-branch = 'trunk' + +``bundled-wheels`` + Glob patterns, relative to the project root, of :term:`wheels ` the project ships as data files. Each + matching wheel becomes a :term:`component` with its :term:`SHA-256`, license, files and the packages it vendors. + + .. code-block:: toml + + [tool.python-sbom] + bundled-wheels = ['src/project/wheels/*.whl'] + +``content-provider`` + A :class:`~python_sbom.ContentProvider`, as ``module:attribute`` importable from the project root, that adds what + the project knows beyond its metadata; :doc:`/how-to/content-provider` shows one. Loading it imports, and so runs, + that module. + + .. code-block:: toml + + [tool.python-sbom] + content-provider = 'sbom_content:Content' + +``build-backend`` + The :term:`build backend` ``python_sbom.backend`` wraps, as ``module`` or ``module:object``, the form + ``[build-system] build-backend`` takes. Only the wrapper reads it, and it fails the build when the project uses the + wrapper without it; :doc:`/how-to/other-backends` shows it. + + .. code-block:: toml + + [tool.python-sbom] + build-backend = 'mesonpy' + +.. _bundled-wheels: + +**************** + Bundled wheels +**************** + +A :term:`bundled wheel`'s files come from its ``RECORD``, and the build fails when a member is missing from it, a row +names a member the wheel lacks, or a digest differs from the member's content. Its :term:`vendored packages ` come from ``_vendor/vendor.txt``, pip_'s manifest, whose lines must each pin one exact version, and from any +``.dist-info/METADATA`` under ``_vendor/``. A bundled wheel gets a supplier only from the project's :term:`content +provider`, never from its own metadata. + +*********************** + Environment variables +*********************** + +``SOURCE_DATE_EPOCH`` + The document's timestamp, as a Unix time. Unset, a reproducible hatchling_ build uses hatchling's fixed 2020-02-02, + a pdm-backend build pdm-backend's fixed 2016-01-01, and any other build the current time. + +``PYTHON_SBOM_COMMIT`` + The source revision to record, a full commit hash, for a wheel built from an :term:`sdist` without + ``.python-sbom-revision``. It wins over every other source. + +.. _source-revision: + +********************* + The source revision +********************* + +The SBOM names the commit the wheel was built from. It takes the first source that knows it: + +1. the ``PYTHON_SBOM_COMMIT`` environment variable, or ``--commit`` on the command line; +2. git_, when the project's ``pyproject.toml`` is tracked by the checkout that holds it, so a project unpacked inside an + unrelated repository never borrows that repository's commit; +3. ``.python-sbom-revision``, which every integration writes into an sdist built from a clean checkout, so a wheel built + from the sdist, as ``python -m build`` and ``uv build`` do by default, keeps the commit; +4. the ``node`` line of an expanded ``.git_archival.txt``, for a ``git archive``. + +A commit must be a full hash of 40 or 64 hex characters. A checkout with changes, including untracked files that are not +ignored, gets a ``:vcs-dirty`` property, and the link to the revision says so; its sdist records no revision. +When git cannot read the checkout, the build warns and moves on to the next source. git runs with ``core.fsmonitor`` +off, so a repository's configuration runs no command. + +******************* + Validation limits +******************* + +Before anything else, ``python-sbom validate`` counts the components an SBOM lists, nested ones included, and fails an +SBOM that lists more than ``--max-components``, 100,000 by default, without checking it further. Every check takes time +in proportion to the count: 100,000 files take about a second, and 100,000 packages, each also checked in the +:term:`SPDX` rendering, about six. The cap bounds what an untrusted SBOM can ask of a :term:`CI` job while staying far +above real SBOMs: every file of the wheel, of each bundled wheel and of each build tool counts, and virtualenv_'s wheel +lists about 5,200. + +The cap also sets the size limit: 1 KiB per component plus 1 MiB, 103,448,576 bytes at the default, about twice the 54 +MB python-sbom writes for 100,000 components. ``validate`` refuses a larger SBOM before parsing it, and reads the SBOMs +of a wheel one at a time, so memory holds only one however many the wheel packs. It also refuses an SBOM nested more +than 128 levels deep; the SBOMs python-sbom and other tools write nest about a dozen levels. diff --git a/docs/reference/index.rst b/docs/reference/index.rst new file mode 100644 index 0000000..045d3eb --- /dev/null +++ b/docs/reference/index.rst @@ -0,0 +1,13 @@ +########### + Reference +########### + +The backends python-sbom supports, the options it reads, its command line and its Python :term:`API`. + +.. toctree:: + :maxdepth: 1 + + backends + configuration + cli + api diff --git a/docs/tutorials/first-sbom.rst b/docs/tutorials/first-sbom.rst new file mode 100644 index 0000000..286fc25 --- /dev/null +++ b/docs/tutorials/first-sbom.rst @@ -0,0 +1,137 @@ +################# + Your first SBOM +################# + +In this tutorial you build a small package with an :term:`SBOM` inside its :term:`wheel`, look at what the SBOM says, +and check it the way a :term:`CI` job would. It takes about ten minutes. You need Python 3.11 or later, git_, and uv_; +any other :term:`build frontend`, such as ``python -m build``, works the same. + +******************** + Create the project +******************** + +Make a directory with a package, a license and a ``pyproject.toml``. The project uses hatchling_, the backend +python-sbom has the most to say about; :doc:`/reference/backends` lists the others. + +.. code-block:: console + + $ mkdir greet && cd greet + $ mkdir -p src/greet && touch src/greet/__init__.py + $ printf 'Copyright (c) 2026 You\n\nMIT terms\n' > LICENSE + +.. code-block:: toml + + [build-system] + build-backend = 'hatchling.build' + requires = ['hatchling>=1.28', 'python-sbom[hatch]>=0.2'] + + [project] + name = 'greet' + version = '1.0' + description = 'says hello' + license = 'MIT' + license-files = ['LICENSE'] + requires-python = '>=3.11' + dependencies = ['packaging>=24'] + urls.Source = 'https://github.com/you/greet' + + [tool.hatch.build.hooks.python-sbom] + + [tool.python-sbom] + supplier = { name = 'You', url = ['https://github.com/you'] } + +The empty ``[tool.hatch.build.hooks.python-sbom]`` table turns the hook on. ``[tool.python-sbom]`` holds python-sbom's +own options; here it names you as the :term:`supplier`. + +Commit it, with the build output ignored, so the SBOM can name the exact :term:`source revision`. A checkout with +uncommitted or untracked files has no commit that reproduces it, and then the SBOM names none: + +.. code-block:: console + + $ printf 'dist/\n*.cdx.json\n' > .gitignore + $ git init --quiet && git add . && git commit --quiet --message first + +***************** + Build the wheel +***************** + +.. code-block:: console + + $ uv build + Successfully built dist/greet-1.0.tar.gz + Successfully built dist/greet-1.0-py3-none-any.whl + +List the wheel's ``.dist-info`` and you find the SBOM next to the metadata: + +.. code-block:: console + + $ unzip -Z1 dist/greet-1.0-py3-none-any.whl | grep dist-info + greet-1.0.dist-info/METADATA + greet-1.0.dist-info/WHEEL + greet-1.0.dist-info/licenses/LICENSE + greet-1.0.dist-info/sboms/greet.cdx.json + greet-1.0.dist-info/RECORD + +The :term:`sdist` holds a ``.python-sbom-revision`` file with your commit. uv built the wheel from that sdist, not from +your checkout, and the file is how the wheel still knows which commit it came from. + +*************** + Read the SBOM +*************** + +Print a few parts of the document: + +.. code-block:: console + + $ unzip -p dist/greet-1.0-py3-none-any.whl greet-1.0.dist-info/sboms/greet.cdx.json > greet.cdx.json + $ python - <<'EOF' + import json + import pprint + sbom = json.load(open('greet.cdx.json')) + root = sbom['metadata']['component'] + print(root['name'], root['version'], root['licenses']) + print([component['name'] for component in sbom['components']]) + print([reference['url'] for reference in root['externalReferences'] if reference['type'] == 'vcs']) + pprint.pp(sorted(tool['name'] for tool in sbom['metadata']['tools']['components']), compact=True, width=100) + EOF + +It prints: + +.. code-block:: text + + greet 1.0 [{'expression': 'MIT', 'acknowledgement': 'declared'}] + ['packaging'] + ['https://github.com/you/greet', 'https://github.com/you/greet/tree/'] + ['cpython', 'hatchling', 'installer', 'packageurl-python', 'packaging', 'pathspec', 'pluggy', + 'python-sbom', 'tomlkit', 'trove-classifiers', 'zipremove'] + +That is four things: + +- the project itself, with its license; +- ``packaging``, the runtime dependency the project declares, without a version, since the installer picks it later; +- a link to your repository at the exact commit you built; +- the build tools: the interpreter, hatchling, python-sbom and every other package of the build environment. + +:doc:`/concepts/document` says what each part is for. + +********** + Check it +********** + +Install the command line with its ``validate`` extra and check the wheel: + +.. code-block:: console + + $ uvx --from 'python-sbom[validate]' python-sbom validate dist/ --allow-license MIT + SBOM in greet-1.0-py3-none-any.whl is valid + +That is the check a CI job runs before publishing; :doc:`/how-to/validate` shows how to wire it in. + +************ + Next steps +************ + +- Your project uses another :term:`build backend`? Pick its guide: :doc:`/how-to/pdm-backend`, :doc:`/how-to/setuptools` + or :doc:`/how-to/other-backends`. +- Your project bundles wheels or knows more than its metadata says: :doc:`/how-to/content-provider`. +- Every option python-sbom reads: :doc:`/reference/configuration`. diff --git a/docs/tutorials/index.rst b/docs/tutorials/index.rst new file mode 100644 index 0000000..5f263fb --- /dev/null +++ b/docs/tutorials/index.rst @@ -0,0 +1,11 @@ +########### + Tutorials +########### + +Start here if python-sbom is new to you: the tutorial builds a package whose :term:`wheel` carries an :term:`SBOM`, +reads the SBOM, and checks it. + +.. toctree:: + :maxdepth: 1 + + first-sbom diff --git a/docs/usage.rst b/docs/usage.rst deleted file mode 100644 index 363a3b4..0000000 --- a/docs/usage.rst +++ /dev/null @@ -1,177 +0,0 @@ -Use it -====== - -With hatchling --------------- - -Add the hook to ``build-system.requires`` and enable it with an empty table, as the quick start on the :doc:`index` -shows. The ``hatch`` extra brings hatchling; the command line needs none. The hook writes the SBOM into wheels only: an -editable wheel carries none, and an sdist carries the source revision instead, so a wheel built from it still names its -commit. - -The hook writes the SBOM once hatchling has written the wheel, so the document describes the final file name and -metadata, whatever dependencies or tag other hooks changed. Hatchling runs the hooks in the order of their tables; see -its `build hook documentation `_. The order does not matter for -this hook. - -Options -~~~~~~~ - -The options live in a ``[tool.python-sbom]`` table, the one place the hook, ``python-sbom embed`` and ``python-sbom -archive`` all read, so every SBOM of a project names it alike. Each option is optional; an unknown key fails the build, -so a typo cannot pass silently. The hook's own table holds only hatchling's options, and a python-sbom option there -fails the build with a pointer to ``[tool.python-sbom]``. - -``supplier`` - The organization that supplies the project: a table with a ``name`` string and an optional ``url`` string array. It - becomes the supplier and manufacturer of the document and the supplier of the project. Left out, the document names - no supplier. - - .. code-block:: toml - - [tool.python-sbom] - supplier = { name = 'Python Packaging Authority', url = ['https://www.pypa.io'] } - -``repository`` - The repository URL, used for the link to the source revision, the license file and, on GitHub, the security policy - and advisories. It defaults to the first ``Source``, ``Repository`` or ``GitHub`` URL in ``[project.urls]``. - - .. code-block:: toml - - [tool.python-sbom] - repository = 'https://gitlab.com/example/project' - -``default-branch`` - The branch in links to files of the repository when the build's commit is unknown or the checkout has - uncommitted changes, ``main`` by default; otherwise the links name the commit. - - .. code-block:: toml - - [tool.python-sbom] - default-branch = 'trunk' - -``bundled-wheels`` - Glob patterns, relative to the project root, of wheels the project ships as data files. Each matching wheel becomes - a component with its SHA-256, license, files and the packages it vendors. - - .. code-block:: toml - - [tool.python-sbom] - bundled-wheels = ['src/project/wheels/*.whl'] - -``content-provider`` - A :class:`~python_sbom.ContentProvider`, as ``module:attribute`` importable from the project root, that adds what - the project knows beyond its metadata; :doc:`extend` shows one. Loading it imports, and so runs, that module. - - .. code-block:: toml - - [tool.python-sbom] - content-provider = 'sbom_content:Content' - -A bundled wheel's files come from its ``RECORD``, and the build fails when a member is missing from it, a row names a -member the wheel lacks, or a digest differs from the member's content. Its vendored packages come from -``_vendor/vendor.txt``, pip's manifest, whose lines must each pin one exact version, and from any -``.dist-info/METADATA`` under ``_vendor/``. A bundled wheel gets a supplier only from the project's content provider, -never from its own metadata. - -The source revision -~~~~~~~~~~~~~~~~~~~ - -The SBOM names the commit the wheel was built from. It takes the first source that knows it: - -#. the ``PYTHON_SBOM_COMMIT`` environment variable, or ``--commit`` on the command line; -#. git, when the project's ``pyproject.toml`` is tracked by the checkout that holds it, so a project unpacked inside an - unrelated repository never borrows that repository's commit; -#. ``.python-sbom-revision``, which the hook writes into every sdist built from a clean checkout, so a wheel built from - the sdist, as ``python -m build`` and ``uv build`` do by default, keeps the commit; -#. the ``node`` line of an expanded ``.git_archival.txt``, for a ``git archive``. - -A commit must be a full hash of 40 or 64 hex characters. A checkout with changes, including untracked files that are not -ignored, gets a ``:vcs-dirty`` property, and the link to the revision says so. When git cannot read the -checkout, the build warns and moves on to the next source. git runs with ``core.fsmonitor`` off, so a repository's -configuration runs no command. - -With another build backend --------------------------- - -A backend other than hatchling, such as `meson-python `_, cannot run the hook. -``python-sbom embed`` writes the same document into a wheel that backend built, adds the ``RECORD`` row, and keeps every -other entry of the archive as it was. It reads the options from ``[tool.python-sbom]``: - -.. code-block:: toml - - [tool.python-sbom] - supplier = { name = 'tox-dev', url = ['https://github.com/tox-dev'] } - -Build, then embed, from the project root: - -.. code-block:: console - - $ python -m build --wheel - $ python-sbom embed dist/*.whl - wrote turbohtml-1.0.dist-info/sboms/turbohtml.cdx.json into turbohtml-1.0-cp314-cp314-linux_x86_64.whl - -``--commit`` names the source revision when the wheel came from an sdist, and ``--content-provider`` overrides the -``content-provider`` option. - -The build tool record lists the packages of the environment named by ``--build-env``, by default the one ``embed`` runs -in; the interpreter it records is always the one running ``embed``. To record the real toolchain, build without -isolation in the environment ``embed`` runs in, from the same lock. On Linux, meson-python also needs ``patchelf`` -there: - -.. code-block:: console - - $ python -m pip install build python-sbom meson-python meson ninja patchelf - $ python -m build --wheel --no-isolation - $ python-sbom embed dist/*.whl - -``auditwheel repair`` renames a Linux wheel to its ``manylinux`` tag, so the attestation link in an SBOM written before -the repair names the old file. Run ``embed`` on the repaired wheel, after ``auditwheel``, to describe the file you -publish; auditwheel's own SBOM in the same directory stays untouched. - -Render it as SPDX ------------------ - -Some tools read only `SPDX `_. ``python-sbom spdx`` renders the CycloneDX document as SPDX 2.3 JSON: - -.. code-block:: console - - $ unzip -p dist/project-1.0-py3-none-any.whl '*/sboms/project.cdx.json' > project.cdx.json - $ python-sbom spdx project.cdx.json project.spdx.json - -The SPDX document keeps the packages, their hashes, licenses and purls, and the containment, dependency and build tool -relationships. It leaves out the per-file hashes, because SPDX 2.3 wants a SHA-1 for every file and wheel ``RECORD`` -files hold SHA-256, and the build record, which SPDX 2.3 has no field for. Its namespace ends in the CycloneDX serial -number, so both documents name the same build. - -Check it in CI --------------- - -``python-sbom validate`` needs the ``validate`` extra, and says so when it is missing. It checks every CycloneDX SBOM in -each wheel's ``.dist-info/sboms`` against the CycloneDX 1.6 schema. For the SBOM python-sbom wrote, it also requires the -``urn:uuid`` serial number ``actions/attest`` needs to recognize the format, checks that every reference resolves, and -validates the SPDX rendering; an SBOM another tool wrote, such as auditwheel's, gets the schema check alone: - -.. code-block:: console - - $ python -m pip install 'python-sbom[validate]' - $ python-sbom validate dist/ --allow-license MIT --allow-license Apache-2.0 - -Each ``--allow-license`` adds an SPDX id that bundled wheels may use. With any given, a bundled wheel whose license the -ids do not allow fails the check, and so does one that declares a license only by name. - -Before anything else, ``validate`` counts the components an SBOM lists, nested ones included, and fails an SBOM that -lists more than 100,000 without checking it further. Every check takes time in proportion to the count: 100,000 files -take about a second, and 100,000 packages, each also checked in the SPDX rendering, about six. The cap thus bounds what -an untrusted SBOM can ask of a CI job while staying far above real SBOMs: every file of the wheel, of each bundled wheel -and of each build tool counts, and virtualenv's wheel lists about 5,200. Raise the cap for an SBOM you expect to be -larger, or lower it to refuse more: - -.. code-block:: console - - $ python-sbom validate dist/ --max-components 500000 - -The cap also sets the size limit: 1 KiB per component plus 1 MiB, 103,448,576 bytes at the default, about twice the 54 -MB python-sbom writes for 100,000 components. ``validate`` refuses a larger SBOM before parsing it, and reads the SBOMs -of a wheel one at a time, so memory holds only one however many the wheel packs. It also refuses an SBOM nested more -than 128 levels deep; the SBOMs python-sbom and other tools write nest about a dozen levels. diff --git a/pylock.toml b/pylock.toml index 988d405..63b8aa8 100644 --- a/pylock.toml +++ b/pylock.toml @@ -5,6 +5,8 @@ environments = [ requires-python = ">=3.11" extras = [ "hatch", + "pdm", + "setuptools", "validate", ] dependency-groups = [ @@ -646,6 +648,31 @@ size = 163877 [packages.wheels.hashes] sha256 = "46ce42d27e5cdde09ee7197349663e97cb106679bd9e75db3a09679fbc7aa1a8" +[[packages]] +name = "flit-core" +version = "4.2.0" +marker = "\"bench\" in dependency_groups or \"dev\" in dependency_groups or \"test\" in dependency_groups or \"type\" in dependency_groups" +requires-python = ">=3.8" +index = "https://pypi.org/simple/" + +[packages.sdist] +name = "flit_core-4.2.0.tar.gz" +upload-time = 2026-10-27T21:07:48.907048+00:00 +url = "https://files.pythonhosted.org/packages/d3/8b/0370578a456986d6995344967a479dd5c9d4e064681550c2a9594bb58b4a/flit_core-4.2.0.tar.gz" +size = 59707 + +[packages.sdist.hashes] +sha256 = "84e4e4d8fdd45ba2613800a3ebc6834de26d47ec9746efc3719f94181137b460" + +[[packages.wheels]] +name = "flit_core-4.2.0-py3-none-any.whl" +upload-time = 2026-10-27T21:07:46.392085+00:00 +url = "https://files.pythonhosted.org/packages/d9/f0/e546c28b937f094b354ca29b9060c5f1dd5043a332b2905049a51420b7a2/flit_core-4.2.0-py3-none-any.whl" +size = 48025 + +[packages.wheels.hashes] +sha256 = "2bee22dc8ba89c2e35b4e1ad575e596aac945c85672dd8f3cad21b629036e044" + [[packages]] name = "furo" version = "2025.12.19" @@ -1454,6 +1481,31 @@ size = 57328 [packages.wheels.hashes] sha256 = "a00ce642f577bf7f473932318056212bc4f8bfdf53128c78bbd5af0b9b20b189" +[[packages]] +name = "pdm-backend" +version = "2.5.0" +marker = "\"bench\" in dependency_groups or \"dev\" in dependency_groups or \"docs\" in dependency_groups or \"test\" in dependency_groups or \"type\" in dependency_groups or \"pdm\" in extras" +requires-python = ">=3.9" +index = "https://pypi.org/simple/" + +[packages.sdist] +name = "pdm_backend-2.5.0.tar.gz" +upload-time = 2026-09-30T13:35:35.980912+00:00 +url = "https://files.pythonhosted.org/packages/fc/d5/a82f533ed51f91a2183faf67a1fcb3b759615ffeca95d05b3e7648bf84bf/pdm_backend-2.5.0.tar.gz" +size = 199083 + +[packages.sdist.hashes] +sha256 = "7953b994563d3151755e3364b9d0cfe817ed0eaecdf27c8f777f412d26bcd98a" + +[[packages.wheels]] +name = "pdm_backend-2.5.0-py3-none-any.whl" +upload-time = 2026-09-30T13:35:34.996150+00:00 +url = "https://files.pythonhosted.org/packages/5e/a6/c743c589188f9e45e1a7c8f3e2e725f9b5df5decd72779fbfc1695b45866/pdm_backend-2.5.0-py3-none-any.whl" +size = 199650 + +[packages.wheels.hashes] +sha256 = "5b8746b567a16a1fd506ce66ee28e89d46374526f52238aa1b96cc2c08245c63" + [[packages]] name = "platformdirs" version = "4.14.0" @@ -2220,7 +2272,7 @@ sha256 = "de78a3b8e0feda74cabc54aab2da702113e33ac9d9eb9d2389bcf1f58b7d9177" [[packages]] name = "setuptools" version = "84.2.1" -marker = "\"bench\" in dependency_groups or \"build\" in dependency_groups or \"dev\" in dependency_groups or \"type\" in dependency_groups" +marker = "\"bench\" in dependency_groups or \"build\" in dependency_groups or \"dev\" in dependency_groups or \"docs\" in dependency_groups or \"test\" in dependency_groups or \"type\" in dependency_groups or \"setuptools\" in extras" requires-python = ">=3.10" index = "https://pypi.org/simple/" @@ -2485,6 +2537,90 @@ size = 22496 [packages.wheels.hashes] sha256 = "eb09aedbabfb650607e9b4b68c9d240b90b1e1be221d6ad71d61c52e29f7932b" +[[packages]] +name = "sphinx-copybutton" +version = "0.5.2" +marker = "\"dev\" in dependency_groups or \"docs\" in dependency_groups or \"type\" in dependency_groups" +requires-python = ">=3.7" +dependencies = [ + { name = "sphinx" }, +] +index = "https://pypi.org/simple/" + +[packages.sdist] +name = "sphinx-copybutton-0.5.2.tar.gz" +upload-time = 2023-04-14T08:10:22.998763+00:00 +url = "https://files.pythonhosted.org/packages/fc/2b/a964715e7f5295f77509e59309959f4125122d648f86b4fe7d70ca1d882c/sphinx-copybutton-0.5.2.tar.gz" +size = 23039 + +[packages.sdist.hashes] +sha256 = "4cf17c82fb9646d1bc9ca92ac280813a3b605d8c421225fd9913154103ee1fbd" + +[[packages.wheels]] +name = "sphinx_copybutton-0.5.2-py3-none-any.whl" +upload-time = 2023-04-14T08:10:20.844249+00:00 +url = "https://files.pythonhosted.org/packages/9e/48/1ea60e74949eecb12cdd6ac43987f9fd331156388dcc2319b45e2ebb81bf/sphinx_copybutton-0.5.2-py3-none-any.whl" +size = 13343 + +[packages.wheels.hashes] +sha256 = "fb543fd386d917746c9a2c50360c7905b605726b9355cd26e9974857afeae06e" + +[[packages]] +name = "sphinx-design" +version = "0.7.0" +marker = "\"dev\" in dependency_groups or \"docs\" in dependency_groups or \"type\" in dependency_groups" +requires-python = ">=3.11" +dependencies = [ + { name = "sphinx" }, +] +index = "https://pypi.org/simple/" + +[packages.sdist] +name = "sphinx_design-0.7.0.tar.gz" +upload-time = 2026-01-19T13:12:53.297780+00:00 +url = "https://files.pythonhosted.org/packages/13/7b/804f311da4663a4aecc6cf7abd83443f3d4ded970826d0c958edc77d4527/sphinx_design-0.7.0.tar.gz" +size = 2203582 + +[packages.sdist.hashes] +sha256 = "d2a3f5b19c24b916adb52f97c5f00efab4009ca337812001109084a740ec9b7a" + +[[packages.wheels]] +name = "sphinx_design-0.7.0-py3-none-any.whl" +upload-time = 2026-01-19T13:12:51.077508+00:00 +url = "https://files.pythonhosted.org/packages/30/cf/45dd359f6ca0c3762ce0490f681da242f0530c49c81050c035c016bfdd3a/sphinx_design-0.7.0-py3-none-any.whl" +size = 2220350 + +[packages.wheels.hashes] +sha256 = "f82bf179951d58f55dca78ab3706aeafa496b741a91b1911d371441127d64282" + +[[packages]] +name = "sphinx-inline-tabs" +version = "2025.12.21.14" +marker = "\"dev\" in dependency_groups or \"docs\" in dependency_groups or \"type\" in dependency_groups" +requires-python = ">=3.8" +dependencies = [ + { name = "sphinx" }, +] +index = "https://pypi.org/simple/" + +[packages.sdist] +name = "sphinx_inline_tabs-2025.12.21.14.tar.gz" +upload-time = 2025-12-21T13:30:51.071093+00:00 +url = "https://files.pythonhosted.org/packages/76/6a/f39bde46a79b80a9983233d99b773bd24b468bdd9c1e87acb46ff69af441/sphinx_inline_tabs-2025.12.21.14.tar.gz" +size = 45572 + +[packages.sdist.hashes] +sha256 = "c71a75800326e613fb4e410eed92a0934214741326aca9897c18018b9f968cb6" + +[[packages.wheels]] +name = "sphinx_inline_tabs-2025.12.21.14-py3-none-any.whl" +upload-time = 2025-12-21T13:30:50.142535+00:00 +url = "https://files.pythonhosted.org/packages/02/2b/e64e7de34663cff1df029ba4f05a86124315bd9eba3d3b78e64904bea7e0/sphinx_inline_tabs-2025.12.21.14-py3-none-any.whl" +size = 7082 + +[packages.wheels.hashes] +sha256 = "e685c782b58d4e01490bcc4e2367cf7135ec28e7283a05e89095394e4ca6e81a" + [[packages]] name = "sphinxcontrib-applehelp" version = "2.0.0" @@ -3109,6 +3245,7 @@ command-line = [ "lock", "--all-groups", "--all-extras", + "--no-cache", ] input-path = "pyproject.toml" mode = "universal" @@ -3119,6 +3256,6 @@ platforms = [ [tool.nab.validation] version = 1 -all-inputs-sha256 = "ccce22a3e18b958cdef60a6925cea289e8979f89899df0a6c90910e746591cdd" -selection-inputs-sha256 = "cb44884cc59469f40513d64048847ea2d328504995bb7469d924665ec67c247a" -contents-sha256 = "44f1d0f11596b036b76fd9dc9245369b0132a90f5568d8506481c3a70d7d4cf9" +all-inputs-sha256 = "4dafd95d61798da174d4b26af09530beecfc187b4e057d2b1b5341bf64a2c284" +selection-inputs-sha256 = "33c6780a0dd69f7f771ac2b8d2e37670c4eb5032e8bfaf5ea560df206ba95955" +contents-sha256 = "6dea5954bbdb8926ab3752c0a47e66dc572d3176889d239108a220092b0a4bc1" diff --git a/pyproject.toml b/pyproject.toml index 41a1e41..a507532 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -16,8 +16,13 @@ description = "SBOMs for Python distributions: write PEP 770 CycloneDX SBOMs int keywords = [ "cyclonedx", "hatch", + "maturin", + "meson-python", + "pdm-backend", + "pep517", "pep770", "sbom", + "setuptools", "spdx", ] license = "MIT AND Apache-2.0" @@ -57,6 +62,12 @@ dependencies = [ optional-dependencies.hatch = [ "hatchling>=1.28", ] +optional-dependencies.pdm = [ + "pdm-backend>=2.4", +] +optional-dependencies.setuptools = [ + "setuptools>=77", +] optional-dependencies.validate = [ "jsonschema-rs>=0.60.3", "license-expression>=30", @@ -68,6 +79,10 @@ urls.Homepage = "https://github.com/tox-dev/python-sbom" urls.Source = "https://github.com/tox-dev/python-sbom" urls.Tracker = "https://github.com/tox-dev/python-sbom/issues" scripts.python-sbom = "python_sbom._cli._main:main" +entry-points."pdm.build.hook".python-sbom = "python_sbom.pdm:SbomBuildHook" +entry-points."setuptools.finalize_distribution_options".python-sbom = """\ + python_sbom.setuptools:finalize_distribution_options\ + """ entry-points.hatch.python-sbom = "python_sbom.hatch.hooks" [dependency-groups] @@ -79,12 +94,15 @@ dev = [ ] test = [ "covdefaults>=2.3", + "flit-core>=3.12", "hatchling>=1.28", "installer>=1.1", "jsonschema-rs>=0.60.3", + "pdm-backend>=2.4", "pytest>=9.1.1", "pytest-cov>=7.1", "pytest-mock>=3.16", + "setuptools>=77", "spdx-tools>=0.8.5", ] type = [ @@ -96,9 +114,14 @@ type = [ docs = [ "furo>=2025.12.19", "hatchling>=1.28", + "pdm-backend>=2.4", + "setuptools>=77", "sphinx>=9.0.4", "sphinx-argparse-cli>=1.23", "sphinx-autodoc-typehints>=3.17", + "sphinx-copybutton>=0.5.2", + "sphinx-design>=0.7", + "sphinx-inline-tabs>=2025.12.21.14", "sphinxcontrib-mermaid>=2.1.1", ] bench = [ @@ -135,6 +158,10 @@ release-prepare = [ version.source = "vcs" metadata.hooks.fancy-pypi-readme.content-type = "text/markdown" metadata.hooks.fancy-pypi-readme.fragments = [ { path = "README.md" } ] +# PyPI cannot resolve the README's relative image paths, so the logo comes from the default branch there +metadata.hooks.fancy-pypi-readme.substitutions = [ + { pattern = '"docs/_static/', replacement = '"https://raw.githubusercontent.com/tox-dev/python-sbom/main/docs/_static/' }, +] build.hooks.vcs.version-file = "src/python_sbom/version.py" # writes python-sbom's own SBOM with the hook it ships, after the vcs hook writes the version build.hooks.custom.path = "hatch_build.py" diff --git a/src/python_sbom/_cli/_spdx.py b/src/python_sbom/_cli/_spdx.py index ad6de8a..898ed22 100644 --- a/src/python_sbom/_cli/_spdx.py +++ b/src/python_sbom/_cli/_spdx.py @@ -3,9 +3,8 @@ from typing import TYPE_CHECKING, Protocol from python_sbom._errors import SbomError -from python_sbom._formats._json import dumps +from python_sbom._formats._json import dumps, is_document, load_json from python_sbom._formats._spdx import render_spdx -from python_sbom._validate._document import is_document, load_json from python_sbom._validate._limits import MAX_COMPONENTS, byte_limit if TYPE_CHECKING: diff --git a/src/python_sbom/_collect/_facts.py b/src/python_sbom/_collect/_facts.py index 89d6823..d921d9a 100644 --- a/src/python_sbom/_collect/_facts.py +++ b/src/python_sbom/_collect/_facts.py @@ -19,7 +19,7 @@ from python_sbom._errors import SbomError from python_sbom._model._inputs import BuildFacts, InstalledDistribution, Interpreter, RecordRow from python_sbom._model._metadata import parse_metadata -from python_sbom._wheel._record import record_rows +from python_sbom._model._record import record_rows if TYPE_CHECKING: from collections.abc import Container, Sequence diff --git a/src/python_sbom/_collect/_revision.py b/src/python_sbom/_collect/_revision.py index 5f80406..0dafdd2 100644 --- a/src/python_sbom/_collect/_revision.py +++ b/src/python_sbom/_collect/_revision.py @@ -85,9 +85,18 @@ def _commit_revision(commit: str, origin: str) -> Revision: return Revision(commit=commit, dirty=False) +def revision_record(revision: Revision | None) -> bytes | None: + # a dirty checkout has no commit that reproduces the sdist, so a wheel built from it names none; bytes, so the + # sdist holds the same LF line on every OS + if revision is None or revision.dirty: + return None + return f"node: {revision.commit}\n".encode() + + __all__ = [ "REVISION_FILE", "RevisionReader", "default_revision", "revision_reader", + "revision_record", ] diff --git a/src/python_sbom/_collect/_settings.py b/src/python_sbom/_collect/_settings.py index 751d47f..05c3676 100644 --- a/src/python_sbom/_collect/_settings.py +++ b/src/python_sbom/_collect/_settings.py @@ -32,7 +32,7 @@ def settings_from_pyproject(root: Path) -> Settings: def settings_from_table(table: Mapping[str, Toml], where: str) -> Settings: - known = {"bundled-wheels", "content-provider", "default-branch", "repository", "supplier"} + known = {"build-backend", "bundled-wheels", "content-provider", "default-branch", "repository", "supplier"} if unknown := sorted(table.keys() - known): msg = f"{where} has unknown option(s) {', '.join(unknown)}; known: {', '.join(sorted(known))}" raise SbomError(msg) @@ -48,12 +48,16 @@ def settings_from_table(table: Mapping[str, Toml], where: str) -> Settings: if not isinstance(content_provider := table.get("content-provider"), str | None): msg = f"`content-provider` in {where} must be a string, `module:attribute`" raise SbomError(msg) + if not isinstance(build_backend := table.get("build-backend"), str | None): + msg = f"`build-backend` in {where} must be a string, the backend to wrap, `module` or `module:object`" + raise SbomError(msg) return Settings( supplier=_supplier(table.get("supplier"), where), repository=None if repository is None else repository.rstrip("/"), default_branch=default_branch, bundled_wheels=tuple(patterns), content_provider=content_provider, + build_backend=build_backend, ) diff --git a/src/python_sbom/_formats/_json.py b/src/python_sbom/_formats/_json.py index 4ec244d..fd92b68 100644 --- a/src/python_sbom/_formats/_json.py +++ b/src/python_sbom/_formats/_json.py @@ -1,18 +1,61 @@ from __future__ import annotations import json -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, Any, TypeAlias + +from python_sbom._errors import SbomError if TYPE_CHECKING: + from typing import TypeIs # new in 3.13; annotations here are never evaluated + from python_sbom._model._types import Document from ._spdx import Spdx +# a JSON value, as json.loads hands it over +Json: TypeAlias = "str | int | float | bool | list[Json] | dict[str, Json] | None" + def dumps(document: Document | Spdx) -> bytes: return f"{json.dumps(document, indent=2)}\n".encode() +def load_json(raw: bytes) -> Any: # ruff: ignore[any-type] # json.loads' own type: any shape until checked + # why bytes are no JSON value, worded once for validate and for spdx, which renders a document without validating it + try: + return json.loads(raw.decode("utf-8")) + except UnicodeDecodeError as error: + msg = f"is not UTF-8: {error.reason} at byte {error.start}" + raise SbomError(msg) from error + except json.JSONDecodeError as error: + msg = f"is not JSON: {error}" + raise SbomError(msg) from error + except RecursionError as error: + # the JSON parser recurses once per level and gives up near the recursion limit, long before the depth check + # sees the document + msg = "nests too deep for the JSON parser to read" + raise SbomError(msg) from error + + +def is_document(value: object) -> TypeIs[Document]: + # what the SPDX rendering reads without a fallback; the schema leaves even these optional + return isinstance(value, dict) and all( + has_path(value, ("metadata", "component", key)) for key in ("bom-ref", "name") + ) + + +def has_path(value: Json, path: tuple[str, ...]) -> bool: + for key in path: + if not isinstance(value, dict) or key not in value: + return False + value = value[key] + return True + + __all__ = [ + "Json", "dumps", + "has_path", + "is_document", + "load_json", ] diff --git a/src/python_sbom/_generate/_archive.py b/src/python_sbom/_generate/_archive.py index 1b14a98..cd55593 100644 --- a/src/python_sbom/_generate/_archive.py +++ b/src/python_sbom/_generate/_archive.py @@ -11,9 +11,8 @@ from python_sbom._errors import SbomError from python_sbom._formats._json import dumps from python_sbom._model._archive import archive_artifact, bundled_paths -from python_sbom._model._bom import assemble +from python_sbom._model._bom import assemble, checked_contributions from python_sbom._model._content import ProviderContext -from python_sbom._model._document import checked_contributions from python_sbom._model._identity import project_identity from python_sbom._model._names import archive_sbom_file_name from python_sbom._wheel._reader import open_artifact diff --git a/src/python_sbom/_generate/_embed.py b/src/python_sbom/_generate/_embed.py index 4f1dcfc..b79a1f7 100644 --- a/src/python_sbom/_generate/_embed.py +++ b/src/python_sbom/_generate/_embed.py @@ -9,9 +9,8 @@ from python_sbom._collect._settings import settings_from_pyproject from python_sbom._collect._wheel import read_wheel from python_sbom._formats._json import dumps -from python_sbom._model._bom import assemble +from python_sbom._model._bom import assemble, checked_contributions from python_sbom._model._content import ProviderContext -from python_sbom._model._document import checked_contributions from python_sbom._model._identity import project_identity from python_sbom._model._wheel import wheel_artifact from python_sbom._wheel._reader import open_artifact @@ -22,7 +21,7 @@ from pathlib import Path from python_sbom._model._content import ContentProvider - from python_sbom._model._inputs import BuildFacts, Settings + from python_sbom._model._inputs import BuildFacts, Revision, Settings def embed_sbom( @@ -68,7 +67,17 @@ def embed_into( # ruff: ignore[too-many-arguments] # what the hook and the comm ) +def embed_built(wheel: Path, root: Path, revision: Revision | None, *, default_epoch: int | None = None) -> str: + # a backend plugin reads the revision before the build writes into the checkout, which could make it look dirty, + # and runs in the build's own environment, so the SBOM records the tools that built the wheel + settings = settings_from_pyproject(root) + content, scripts = resolve_provider(None, settings, root) + facts = collect_facts(root, lambda _root: revision, default_epoch=default_epoch) + return embed_into(wheel, root, settings=settings, content=content, build_scripts=scripts, facts=facts) + + __all__ = [ + "embed_built", "embed_into", "embed_sbom", ] diff --git a/src/python_sbom/_model/_bom.py b/src/python_sbom/_model/_bom.py index d335732..a2820b2 100644 --- a/src/python_sbom/_model/_bom.py +++ b/src/python_sbom/_model/_bom.py @@ -1,19 +1,23 @@ from __future__ import annotations +import json +import uuid +from datetime import UTC, datetime from pathlib import PurePosixPath from typing import TYPE_CHECKING, Final from urllib.parse import urlsplit -from ._components import script_tool -from ._document import check_refs, contributed_properties, formulation, timestamp, with_header +from python_sbom._errors import SbomError + +from ._graph import reference_problems, walk_components from ._names import SBOM_FILE, SBOM_REPOSITORY, sbom_property -from ._tools import tool_components +from ._tools import script_tool, tool_components if TYPE_CHECKING: - from ._content import Contributions + from ._content import ContentProvider, Contributions from ._identity import ProjectIdentity from ._inputs import Artifact, BuildFacts, Settings - from ._types import Body, Component, Document, ExternalReference, Metadata, Property + from ._types import Body, Component, Document, ExternalReference, Formula, Metadata, Property def assemble( @@ -89,6 +93,13 @@ def _provenance( return root +def contributed_properties(properties: list[Property], prefix: str) -> list[Property]: + if claimed := sorted(prop["name"] for prop in properties if prop["name"].startswith(f"{prefix}:")): + msg = f"contributed root properties {', '.join(claimed)} use the {prefix}: prefix python-sbom writes" + raise SbomError(msg) + return properties + + def _file_link(project: ProjectIdentity, settings: Settings, facts: BuildFacts, path: PurePosixPath) -> str | None: if project.repository is None: return None @@ -107,7 +118,7 @@ def _metadata( facts: BuildFacts, ) -> Metadata: metadata: Final[Metadata] = { - "timestamp": timestamp(facts), + "timestamp": datetime.fromtimestamp(facts.epoch, tz=UTC).isoformat(), "lifecycles": [{"phase": "build"}], "tools": {"components": tools}, "component": (root := artifact.root), @@ -132,6 +143,75 @@ def _metadata( return metadata +def formulation(uid: str, name: str, tools: list[Component], output_ref: str, facts: BuildFacts) -> Formula: + formula: Formula = { + "bom-ref": f"formula:{uid}", + "workflows": [ + { + "bom-ref": f"workflow:{uid}", + "uid": uid, + "name": name, + "taskTypes": ["build"], + "resourceReferences": [{"ref": tool["bom-ref"]} for tool in tools], + "outputs": [{"type": "artifact", "resource": {"ref": output_ref}}], + } + ], + } + if facts.epoch_from_environment: + formula["workflows"][0]["inputs"] = [ + {"environmentVars": [{"name": "SOURCE_DATE_EPOCH", "value": str(facts.epoch)}]} + ] + return formula + + +def check_refs(body: Body) -> None: + # a provider contributes refs of its own, so the document is checked before it is written rather than by validate + components: Final[list[Component]] = [ + *walk_components([ + body["metadata"]["component"], + *body["components"], + *body["metadata"]["tools"]["components"], + ]) + ] + # formulas and workflows share the bom-ref namespace, so a component cannot take theirs + formulas: Final[list[str]] = [ + ref + for formula in body.get("formulation", []) + for ref in (formula["bom-ref"], *(workflow["bom-ref"] for workflow in formula["workflows"])) + ] + if problems := reference_problems( + [*(component["bom-ref"] for component in components), *formulas], body["dependencies"] + ): + raise SbomError(problems[0]) + + +def with_header(body: Body, serial_namespace: uuid.UUID) -> Document: + return { + "$schema": "http://cyclonedx.org/schema/bom-1.6.schema.json", + "bomFormat": "CycloneDX", + "specVersion": "1.6", + "serialNumber": f"urn:uuid:{uuid.uuid5(serial_namespace, json.dumps(body, sort_keys=True))}", + "version": 1, + **body, + } + + +def checked_contributions(contributions: Contributions, provider: ContentProvider) -> Contributions: + # what a provider adds is checked for the fields everything else here reads, so a mistake names the provider rather + # than surfacing as a KeyError deep in the build + for component in walk_components([*contributions.components, *contributions.tools]): + if missing := [key for key in ("type", "bom-ref", "name") if key not in component]: + name = type(provider).__qualname__ + msg = f"content provider {name} added a component without {', '.join(missing)}: {component!r:.200}" + raise SbomError(msg) + return contributions + + __all__ = [ "assemble", + "check_refs", + "checked_contributions", + "contributed_properties", + "formulation", + "with_header", ] diff --git a/src/python_sbom/_model/_components.py b/src/python_sbom/_model/_components.py index 319b039..b02c0d9 100644 --- a/src/python_sbom/_model/_components.py +++ b/src/python_sbom/_model/_components.py @@ -10,12 +10,10 @@ from ._graph import walk_components from ._iri import iri_reference from ._licenses import licenses -from ._names import purl, pypi_url, reference_type, sbom_property, tool_ref +from ._names import purl, pypi_url, reference_type, sbom_property from ._types import Dependency if TYPE_CHECKING: - from pathlib import PurePosixPath - from ._content import BundledWheelInfo from ._inputs import BundledWheel, RecordRow from ._metadata import CoreMetadata @@ -159,24 +157,10 @@ def vendored_dependencies(components: list[Component]) -> list[Dependency]: return found -def script_tool(path: PurePosixPath, sha256: str, version: str, link: str | None) -> Component: - tool: Final[Component] = { - "type": "application", - "bom-ref": tool_ref(str(path)), - "name": path.name, - "version": version, - "hashes": [{"alg": "SHA-256", "content": sha256}], - } - if link is not None: - tool["externalReferences"] = [{"type": "vcs", "url": link}] - return tool - - __all__ = [ "bundled_wheel_component", "component_from_metadata", "file_component", "mark_bundled", - "script_tool", "vendored_dependencies", ] diff --git a/src/python_sbom/_model/_document.py b/src/python_sbom/_model/_document.py deleted file mode 100644 index 42e4d74..0000000 --- a/src/python_sbom/_model/_document.py +++ /dev/null @@ -1,100 +0,0 @@ -from __future__ import annotations - -import json -import uuid -from datetime import UTC, datetime -from typing import TYPE_CHECKING, Final - -from python_sbom._errors import SbomError - -from ._graph import reference_problems, walk_components - -if TYPE_CHECKING: - from ._content import ContentProvider, Contributions - from ._inputs import BuildFacts - from ._types import Body, Component, Document, Formula, Property - - -def check_refs(body: Body) -> None: - # a provider contributes refs of its own, so the document is checked before it is written rather than by validate - components: Final[list[Component]] = [ - *walk_components([ - body["metadata"]["component"], - *body["components"], - *body["metadata"]["tools"]["components"], - ]) - ] - # formulas and workflows share the bom-ref namespace, so a component cannot take theirs - formulas: Final[list[str]] = [ - ref - for formula in body.get("formulation", []) - for ref in (formula["bom-ref"], *(workflow["bom-ref"] for workflow in formula["workflows"])) - ] - if problems := reference_problems( - [*(component["bom-ref"] for component in components), *formulas], body["dependencies"] - ): - raise SbomError(problems[0]) - - -def with_header(body: Body, serial_namespace: uuid.UUID) -> Document: - return { - "$schema": "http://cyclonedx.org/schema/bom-1.6.schema.json", - "bomFormat": "CycloneDX", - "specVersion": "1.6", - "serialNumber": f"urn:uuid:{uuid.uuid5(serial_namespace, json.dumps(body, sort_keys=True))}", - "version": 1, - **body, - } - - -def checked_contributions(contributions: Contributions, provider: ContentProvider) -> Contributions: - # what a provider adds is checked for the fields everything else here reads, so a mistake names the provider rather - # than surfacing as a KeyError deep in the build - for component in walk_components([*contributions.components, *contributions.tools]): - if missing := [key for key in ("type", "bom-ref", "name") if key not in component]: - name = type(provider).__qualname__ - msg = f"content provider {name} added a component without {', '.join(missing)}: {component!r:.200}" - raise SbomError(msg) - return contributions - - -def contributed_properties(properties: list[Property], prefix: str) -> list[Property]: - if claimed := sorted(prop["name"] for prop in properties if prop["name"].startswith(f"{prefix}:")): - msg = f"contributed root properties {', '.join(claimed)} use the {prefix}: prefix python-sbom writes" - raise SbomError(msg) - return properties - - -def formulation(uid: str, name: str, tools: list[Component], output_ref: str, facts: BuildFacts) -> Formula: - formula: Formula = { - "bom-ref": f"formula:{uid}", - "workflows": [ - { - "bom-ref": f"workflow:{uid}", - "uid": uid, - "name": name, - "taskTypes": ["build"], - "resourceReferences": [{"ref": tool["bom-ref"]} for tool in tools], - "outputs": [{"type": "artifact", "resource": {"ref": output_ref}}], - } - ], - } - if facts.epoch_from_environment: - formula["workflows"][0]["inputs"] = [ - {"environmentVars": [{"name": "SOURCE_DATE_EPOCH", "value": str(facts.epoch)}]} - ] - return formula - - -def timestamp(facts: BuildFacts) -> str: - return datetime.fromtimestamp(facts.epoch, tz=UTC).isoformat() - - -__all__ = [ - "check_refs", - "checked_contributions", - "contributed_properties", - "formulation", - "timestamp", - "with_header", -] diff --git a/src/python_sbom/_model/_inputs.py b/src/python_sbom/_model/_inputs.py index e4928f4..3f9652c 100644 --- a/src/python_sbom/_model/_inputs.py +++ b/src/python_sbom/_model/_inputs.py @@ -60,6 +60,8 @@ class Settings: uncommitted changes :param bundled_wheels: glob patterns, relative to the project directory, of wheels the project ships as data files :param content_provider: ``module:attribute`` of the content provider, or ``None`` + :param build_backend: the build backend ``python_sbom.backend`` wraps, as ``module`` or ``module:object``, or + ``None`` """ @@ -68,6 +70,7 @@ class Settings: default_branch: str bundled_wheels: tuple[str, ...] content_provider: str | None + build_backend: str | None @dataclass(frozen=True, slots=True, kw_only=True) diff --git a/src/python_sbom/_wheel/_record.py b/src/python_sbom/_model/_record.py similarity index 97% rename from src/python_sbom/_wheel/_record.py rename to src/python_sbom/_model/_record.py index 25d9260..9f7de64 100644 --- a/src/python_sbom/_wheel/_record.py +++ b/src/python_sbom/_model/_record.py @@ -8,7 +8,8 @@ from installer.records import RecordEntry from python_sbom._errors import SbomError -from python_sbom._model._inputs import RecordRow + +from ._inputs import RecordRow if TYPE_CHECKING: from collections.abc import Iterable diff --git a/src/python_sbom/_model/_tools.py b/src/python_sbom/_model/_tools.py index 870d6a4..43eac69 100644 --- a/src/python_sbom/_model/_tools.py +++ b/src/python_sbom/_model/_tools.py @@ -8,6 +8,8 @@ from ._names import purl, tool_ref if TYPE_CHECKING: + from pathlib import PurePosixPath + from ._inputs import BuildFacts from ._types import Component, Dependency @@ -43,6 +45,20 @@ def tool_components(facts: BuildFacts) -> tuple[list[Component], list[Dependency return tools, dependencies +def script_tool(path: PurePosixPath, sha256: str, version: str, link: str | None) -> Component: + tool: Final[Component] = { + "type": "application", + "bom-ref": tool_ref(str(path)), + "name": path.name, + "version": version, + "hashes": [{"alg": "SHA-256", "content": sha256}], + } + if link is not None: + tool["externalReferences"] = [{"type": "vcs", "url": link}] + return tool + + __all__ = [ + "script_tool", "tool_components", ] diff --git a/src/python_sbom/_validate/_document.py b/src/python_sbom/_validate/_document.py index e07e329..2c041a7 100644 --- a/src/python_sbom/_validate/_document.py +++ b/src/python_sbom/_validate/_document.py @@ -1,21 +1,20 @@ from __future__ import annotations -import json import math import re from importlib.util import find_spec -from typing import TYPE_CHECKING, Any, Final +from typing import TYPE_CHECKING, Final from python_sbom._errors import MissingExtraError, SbomError +from python_sbom._formats._json import Json, has_path, is_document, load_json from python_sbom._formats._spdx import render_spdx from python_sbom._model._graph import artifact_contents, reference_problems, repeated, walk_components from python_sbom._model._names import SBOM_FILE -from ._limits import MAX_COMPONENTS, MAX_TEXT, Json, byte_limit +from ._limits import MAX_COMPONENTS, MAX_TEXT, byte_limit if TYPE_CHECKING: from collections.abc import Iterable - from typing import TypeIs # new in 3.13; annotations here are never evaluated from python_sbom._model._types import Component, Document @@ -80,7 +79,7 @@ def _problems(raw: bytes, allowed_licenses: frozenset[str] | None, max_component prop.get("name", "").endswith(f":{SBOM_FILE}") for prop in loaded.get("metadata", {}).get("properties", []) ): return [] - missing: Final[list[str]] = [".".join(path) for path in _REQUIRED if not _has(loaded, path)] + missing: Final[list[str]] = [".".join(path) for path in _REQUIRED if not has_path(loaded, path)] # is_document holds once nothing is missing; asking it narrows the loaded JSON to a document if missing or not is_document(loaded): return [f"lacks {', '.join(missing)}, which every document python-sbom writes carries"] @@ -102,23 +101,6 @@ def _problems(raw: bytes, allowed_licenses: frozenset[str] | None, max_component return problems -def load_json(raw: bytes) -> Any: # ruff: ignore[any-type] # json.loads' own type: any shape until checked - # why bytes are no JSON value, worded once for validate and for spdx, which renders a document without validating it - try: - return json.loads(raw.decode("utf-8")) - except UnicodeDecodeError as error: - msg = f"is not UTF-8: {error.reason} at byte {error.start}" - raise SbomError(msg) from error - except json.JSONDecodeError as error: - msg = f"is not JSON: {error}" - raise SbomError(msg) from error - except RecursionError as error: - # the JSON parser recurses once per level and gives up near the recursion limit, long before the depth check - # sees the document - msg = "nests too deep for the JSON parser to read" - raise SbomError(msg) from error - - def _census_problems(document: Json, max_components: int) -> list[str]: # counts the components and bom-refs and measures the depth in one linear pass before the schema vouches for any # shape; iterative, so deep nesting cannot exhaust the stack. Nested components count too: the schema checks every @@ -165,19 +147,6 @@ def _push_containers( return [] -def _has(value: Json, path: tuple[str, ...]) -> bool: - for key in path: - if not isinstance(value, dict) or key not in value: - return False - value = value[key] - return True - - -def is_document(value: object) -> TypeIs[Document]: - # what the SPDX rendering reads without a fallback; the schema leaves even these optional - return isinstance(value, dict) and all(_has(value, ("metadata", "component", key)) for key in ("bom-ref", "name")) - - def _reference_problems(document: Document) -> list[str]: root: Final[Component] = document["metadata"]["component"] tools: Final[list[Component]] = document["metadata"]["tools"]["components"] @@ -215,7 +184,5 @@ def _reference_problems(document: Document) -> list[str]: __all__ = [ - "is_document", - "load_json", "validate_document", ] diff --git a/src/python_sbom/_validate/_limits.py b/src/python_sbom/_validate/_limits.py index 4df8753..eb7ef3e 100644 --- a/src/python_sbom/_validate/_limits.py +++ b/src/python_sbom/_validate/_limits.py @@ -1,9 +1,6 @@ from __future__ import annotations -from typing import Final, TypeAlias - -# a JSON value, as json.loads hands it over -Json: TypeAlias = "str | int | float | bool | list[Json] | dict[str, Json] | None" +from typing import Final # every check takes time in proportion to the count, so the cap bounds the work an untrusted document can ask for, far # above the few thousand components a wheel with its build tools' files lists @@ -22,6 +19,5 @@ def byte_limit(max_components: int) -> int: __all__ = [ "MAX_COMPONENTS", "MAX_TEXT", - "Json", "byte_limit", ] diff --git a/src/python_sbom/_validate/_schemas.py b/src/python_sbom/_validate/_schemas.py index f74ff75..d0e5f7a 100644 --- a/src/python_sbom/_validate/_schemas.py +++ b/src/python_sbom/_validate/_schemas.py @@ -23,10 +23,9 @@ from jsonschema_rs import ValidationError from spdx_tools.spdx.model import Document, SpdxNoAssertion, SpdxNone + from python_sbom._formats._json import Json from python_sbom._formats._spdx import Spdx - from ._limits import Json - def schema_error(document: Json) -> str | None: detail: Final[ValidationError | None] = next(_VALIDATOR.iter_errors(document), None) diff --git a/src/python_sbom/_wheel/_bundled.py b/src/python_sbom/_wheel/_bundled.py index 06d005b..c0a22f0 100644 --- a/src/python_sbom/_wheel/_bundled.py +++ b/src/python_sbom/_wheel/_bundled.py @@ -11,8 +11,8 @@ from python_sbom._errors import SbomError from python_sbom._model._inputs import BundledWheel, RecordRow from python_sbom._model._metadata import CoreMetadata, parse_metadata +from python_sbom._model._record import record_rows -from ._record import record_rows from ._zip import open_archive, read_member, read_wheel_metadata, sha256_hex if TYPE_CHECKING: diff --git a/src/python_sbom/_wheel/_sdist.py b/src/python_sbom/_wheel/_sdist.py new file mode 100644 index 0000000..9532f37 --- /dev/null +++ b/src/python_sbom/_wheel/_sdist.py @@ -0,0 +1,54 @@ +from __future__ import annotations + +import gzip +import io +import os +import stat +import tarfile +import tempfile +from pathlib import Path +from typing import Final + +from python_sbom._errors import SbomError + + +def add_to_sdist(sdist: Path, name: str, content: bytes) -> None: + descriptor, temporary = tempfile.mkstemp(dir=sdist.parent, prefix=".python-sbom-", suffix=".tmp") + os.close(descriptor) + try: + # the backend wrote the sdist already, so the file joins it under the sdist's top directory; a backend that + # added the file itself, such as one running python-sbom's own plugin, keeps its copy + with tarfile.open(sdist, "r:gz") as source: + members: Final[list[tarfile.TarInfo]] = source.getmembers() + if not members or (top := members[0].name.split("/", 1)[0]) in {"", ".", ".."}: + msg = f"{sdist} has no top-level directory to add {name} to" + raise SbomError(msg) + if (target := f"{top}/{name}") in {member.name for member in members}: + return + # the sdist's own newest time, so the added entry keeps a reproducible build reproducible + added: Final[tarfile.TarInfo] = tarfile.TarInfo(target) + added.size = len(content) + added.mtime = max(member.mtime for member in members) + added.mode = stat.S_IFREG | 0o644 + # the gzip header keeps the time the backend gave it, bytes 4 to 8 of the header, and names no file, since + # the temporary file's random name would differ on every build + with sdist.open("rb") as existing: + header = existing.read(8) + with ( + Path(temporary).open("wb") as raw, + gzip.GzipFile(filename="", fileobj=raw, mode="wb", mtime=int.from_bytes(header[4:8], "little")) as gz, + tarfile.open(fileobj=gz, mode="w") as rewritten, + ): + for member in members: + rewritten.addfile(member, source.extractfile(member) if member.isfile() else None) + rewritten.addfile(added, io.BytesIO(content)) + # only once the sdist is closed, since Windows refuses to replace an open file + Path(temporary).chmod(stat.S_IMODE(sdist.stat().st_mode)) + Path(temporary).replace(sdist) + finally: + Path(temporary).unlink(missing_ok=True) + + +__all__ = [ + "add_to_sdist", +] diff --git a/src/python_sbom/backend.py b/src/python_sbom/backend.py new file mode 100644 index 0000000..d78a5bd --- /dev/null +++ b/src/python_sbom/backend.py @@ -0,0 +1,118 @@ +"""A build backend that wraps another one and writes the SBOM into every wheel it builds.""" + +from __future__ import annotations + +import importlib +from functools import cache, reduce +from pathlib import Path +from typing import TYPE_CHECKING, Final, Protocol, cast + +from ._collect._revision import REVISION_FILE, default_revision, revision_record +from ._collect._settings import settings_from_pyproject +from ._errors import SbomError + +if TYPE_CHECKING: + from collections.abc import Mapping + + #: the settings a frontend passes each hook, such as ``--config-settings`` of pip and build + ConfigSettings = Mapping[str, str | list[str]] | None + +# the hooks PEP 517 and PEP 660 let a backend leave out, which a frontend then replaces with its own default +_OPTIONAL_HOOKS: Final[frozenset[str]] = frozenset({ + "build_editable", + "get_requires_for_build_editable", + "get_requires_for_build_sdist", + "get_requires_for_build_wheel", + "prepare_metadata_for_build_editable", + "prepare_metadata_for_build_wheel", +}) + + +class _Hook(Protocol): + # every optional hook takes directories and the frontend's settings, and returns a file name or requirements + def __call__(self, *arguments: str | ConfigSettings) -> str | list[str]: ... + + +class _Backend(Protocol): + def build_wheel( + self, wheel_directory: str, config_settings: ConfigSettings = None, metadata_directory: str | None = None, / + ) -> str: ... + + def build_sdist(self, sdist_directory: str, config_settings: ConfigSettings = None, /) -> str: ... + + +def build_wheel( + wheel_directory: str, config_settings: ConfigSettings = None, metadata_directory: str | None = None +) -> str: + """ + Build a wheel with the wrapped backend, then write the SBOM into it. + + :param wheel_directory: where the wheel goes + :param config_settings: the frontend's settings, passed on unchanged + :param metadata_directory: the ``.dist-info`` an earlier ``prepare_metadata_for_build_wheel`` call wrote + :returns: the wheel's file name + + """ + from ._generate._embed import embed_built # ruff: ignore[import-outside-top-level] # sdist builds skip it + + root = Path.cwd() + # read before the backend writes into the checkout, which may not ignore its build directory + revision = default_revision(root) + name = _wrapped().build_wheel(wheel_directory, config_settings, metadata_directory) + embed_built(Path(wheel_directory) / name, root, revision) + return name + + +def _wrapped() -> _Backend: + return _load(Path.cwd()) + + +def build_sdist(sdist_directory: str, config_settings: ConfigSettings = None) -> str: + """ + Build an sdist with the wrapped backend, then record the source revision in it. + + :param sdist_directory: where the sdist goes + :param config_settings: the frontend's settings, passed on unchanged + :returns: the sdist's file name + + """ + from ._wheel._sdist import add_to_sdist # ruff: ignore[import-outside-top-level] # wheel builds skip it + + revision = default_revision(Path.cwd()) + name = _wrapped().build_sdist(sdist_directory, config_settings) + if (record := revision_record(revision)) is not None: + add_to_sdist(Path(sdist_directory) / name, REVISION_FILE, record) + return name + + +def __getattr__(name: str) -> _Hook: + # a frontend asks with hasattr whether an optional hook exists, so one the wrapped backend lacks must not exist here + if name in _OPTIONAL_HOOKS and hasattr(backend := _wrapped(), name): + return cast("_Hook", getattr(backend, name)) + msg = f"module {__name__!r} has no attribute {name!r}" + raise AttributeError(msg) + + +@cache +def _load(root: Path) -> _Backend: + # PEP 517 runs every hook from the project directory, which holds the pyproject.toml naming the backend + if (reference := settings_from_pyproject(root).build_backend) is None: + msg = ( + f"{root / 'pyproject.toml'} sets build-backend = 'python_sbom.backend' but names no backend to wrap; set" + " build-backend in [tool.python-sbom], such as 'mesonpy'" + ) + raise SbomError(msg) + module, _, attribute = reference.partition(":") + if module == __name__: + msg = f"[tool.python-sbom] build-backend names {reference}, which would wrap itself" + raise SbomError(msg) + loaded = importlib.import_module(module) + # PEP 517 requires every backend to define the two mandatory hooks + return cast("_Backend", reduce(getattr, attribute.split("."), loaded) if attribute else loaded) + + +__all__ = [ + "ConfigSettings", + "build_sdist", + "build_wheel", +] diff --git a/src/python_sbom/hatch/_hook.py b/src/python_sbom/hatch/_hook.py index 455412e..7bbb9d8 100644 --- a/src/python_sbom/hatch/_hook.py +++ b/src/python_sbom/hatch/_hook.py @@ -12,7 +12,7 @@ from hatchling.utils.constants import DEFAULT_BUILD_SCRIPT from python_sbom._collect._content import resolve_provider -from python_sbom._collect._revision import REVISION_FILE, default_revision +from python_sbom._collect._revision import REVISION_FILE, default_revision, revision_record from python_sbom._collect._settings import settings_from_table from python_sbom._errors import SbomError @@ -64,17 +64,15 @@ def initialize( return # once per build, before it writes the artifact into a checkout that may not ignore it; a value kept for the # process would name an older commit in the next build after a commit - self._revision = revision = default_revision(Path(self.root)) - # a dirty checkout has no commit that reproduces the sdist, so a wheel built from it names none - if self.target_name != "sdist" or revision is None or revision.dirty: + self._revision = default_revision(Path(self.root)) + if self.target_name != "sdist" or (record := revision_record(self._revision)) is None: return # outside the project, so the build leaves nothing behind in it directory = tempfile.mkdtemp() # hatchling reads the file only while it writes the sdist, and skips finalize when the build fails, so the # directory then goes with the hook, at the latest when the interpreter exits self._remove_revision_file = weakref.finalize(self, shutil.rmtree, directory) - # bytes, so the sdist holds the same LF line on every OS - (written := Path(directory) / REVISION_FILE).write_bytes(f"node: {revision.commit}\n".encode()) + (written := Path(directory) / REVISION_FILE).write_bytes(record) build_data["force_include"][str(written)] = REVISION_FILE def finalize( diff --git a/src/python_sbom/pdm.py b/src/python_sbom/pdm.py new file mode 100644 index 0000000..8908983 --- /dev/null +++ b/src/python_sbom/pdm.py @@ -0,0 +1,81 @@ +"""The pdm-backend build hook, which writes the SBOM into every wheel pdm-backend builds.""" + +from __future__ import annotations + +import calendar +import tempfile +from pathlib import Path +from typing import TYPE_CHECKING + +from ._collect._revision import REVISION_FILE, default_revision, revision_record + +if TYPE_CHECKING: + from pdm.backend.hooks import Context + + from ._model._inputs import Revision + + +class SbomBuildHook: + """Write the SBOM into a wheel pdm-backend built, and record the source revision in an sdist.""" + + _revision: Revision | None = None + _scratch: tempfile.TemporaryDirectory[str] | None = None + + def pdm_build_hook_enabled(self, context: Context) -> bool: # ruff: ignore[no-self-use] # pdm-backend calls it on the hook + """ + Act only on a project that asks for an SBOM. + + :param context: pdm-backend's build context + :returns: whether ``[tool.python-sbom]`` exists and the build is a wheel or an sdist + + """ + return context.target in {"sdist", "wheel"} and "python-sbom" in context.config.data.get("tool", {}) + + def pdm_build_initialize(self, context: Context) -> None: + """ + Read the source revision, before the build writes into the checkout. + + :param context: pdm-backend's build context + + """ + self._revision = default_revision(context.root) + + def pdm_build_update_files(self, context: Context, files: dict[str, Path]) -> None: + """ + Add the source revision to an sdist, so a wheel built from it still names its commit. + + :param context: pdm-backend's build context + :param files: the files the artifact will hold, by their path in it + + """ + if context.target != "sdist" or (record := revision_record(self._revision)) is None: + return + # outside pdm-backend's build directory, whose every file lands in the sdist, the .gitignore it writes there too + self._scratch = tempfile.TemporaryDirectory(prefix="python-sbom-") + (written := Path(self._scratch.name) / REVISION_FILE).write_bytes(record) + files[REVISION_FILE] = written + + def pdm_build_finalize(self, context: Context, artifact: Path) -> None: + """ + Write the SBOM into the wheel pdm-backend built. + + :param context: pdm-backend's build context + :param artifact: the built wheel or sdist + + """ + if self._scratch is not None: + self._scratch.cleanup() + if context.target != "wheel": + return + # what a wheel needs loads only when one is built, so an sdist build stays light + from pdm.backend.wheel import ZIPINFO_DATE_TIME # ruff: ignore[import-outside-top-level] # see above + + from ._generate._embed import embed_built # ruff: ignore[import-outside-top-level] # see above + + # pdm-backend dates every zip entry at a fixed time when SOURCE_DATE_EPOCH is unset, so the SBOM agrees + embed_built(Path(artifact), context.root, self._revision, default_epoch=calendar.timegm(ZIPINFO_DATE_TIME)) + + +__all__ = [ + "SbomBuildHook", +] diff --git a/src/python_sbom/setuptools.py b/src/python_sbom/setuptools.py new file mode 100644 index 0000000..3b2c3d2 --- /dev/null +++ b/src/python_sbom/setuptools.py @@ -0,0 +1,83 @@ +"""The setuptools plugin, which writes the SBOM into every wheel setuptools builds.""" + +from __future__ import annotations + +import tomllib +from pathlib import Path +from typing import TYPE_CHECKING + +from setuptools.command.bdist_wheel import bdist_wheel +from setuptools.command.sdist import sdist + +from ._collect._revision import REVISION_FILE, default_revision, revision_record +from ._errors import SbomError + +if TYPE_CHECKING: + from setuptools.dist import Distribution + + from ._model._inputs import Revision + + +def finalize_distribution_options(distribution: Distribution) -> None: + """ + Wrap the wheel and sdist commands of a project that asks for an SBOM. + + :param distribution: the distribution setuptools is about to build + :raises SbomError: when the project's pyproject.toml is not valid TOML or replaces the wheel or sdist command, or + the wheel command writes no wheel + + """ + # setuptools runs every build from the project directory + root = Path.cwd() + try: + loaded = tomllib.loads((root / "pyproject.toml").read_text(encoding="utf-8")) + except FileNotFoundError: # a setup.py-only project, which cannot ask + return + except tomllib.TOMLDecodeError as error: + msg = f"{root / 'pyproject.toml'} is not valid TOML: {error}" + raise SbomError(msg) from error + if "python-sbom" not in (tools := loaded.get("tool", {})): + return + # setuptools applies these after this plugin ran, so they would replace its commands and drop the SBOM unnoticed + if replaced := sorted({"bdist_wheel", "sdist"} & set(tools.get("setuptools", {}).get("cmdclass", {}))): + msg = ( + f"[tool.setuptools.cmdclass] replaces {', '.join(replaced)} after python-sbom wraps it, so the build would" + " carry no SBOM; pass the command to setup(cmdclass=...) in setup.py, or wrap setuptools.build_meta with" + " python_sbom.backend" + ) + raise SbomError(msg) + wheel_base = distribution.cmdclass.get("bdist_wheel", bdist_wheel) + + class SbomWheel(wheel_base): + def run(self) -> None: + from ._generate._embed import embed_built # ruff: ignore[import-outside-top-level] # wheels only + + # read before this command writes into build/; a checkout that does not ignore build/ or the egg-info + # directory an earlier step wrote reads as dirty + revision = default_revision(root) + super().run() + # bdist_wheel lists the wheel it wrote for the upload command; editable wheels never run this command + if not (wheels := [entry[2] for entry in self.distribution.dist_files if entry[0] == "bdist_wheel"]): + msg = f"{wheel_base.__qualname__} wrote no wheel to describe" + raise SbomError(msg) + embed_built(Path(wheels[-1]), root, revision) + + class SbomSdist(distribution.cmdclass.get("sdist", sdist)): + _revision: Revision | None = None + + def run(self) -> None: + # read before this command writes its egg-info, which a checkout may not ignore + self._revision = default_revision(root) + super().run() + + def make_release_tree(self, base_dir: str, files: list[str]) -> None: + super().make_release_tree(base_dir, files) + if (record := revision_record(self._revision)) is not None: + (Path(base_dir) / REVISION_FILE).write_bytes(record) + + distribution.cmdclass.update(bdist_wheel=SbomWheel, sdist=SbomSdist) + + +__all__ = [ + "finalize_distribution_options", +] diff --git a/tests/_collect/test_settings.py b/tests/_collect/test_settings.py index a069ab3..a1776d5 100644 --- a/tests/_collect/test_settings.py +++ b/tests/_collect/test_settings.py @@ -37,7 +37,8 @@ def test_table(tmp_path: Path) -> None: "default-branch = 'dev'\n" "bundled-wheels = ['src/*.whl']\n" "supplier = { name = 'Example Org' }\n" - "content-provider = 'sbom:Content'\n", + "content-provider = 'sbom:Content'\n" + "build-backend = 'mesonpy'\n", encoding="utf-8", ) assert settings_from_pyproject(tmp_path) == Settings( @@ -46,11 +47,14 @@ def test_table(tmp_path: Path) -> None: default_branch="dev", bundled_wheels=("src/*.whl",), content_provider="sbom:Content", + build_backend="mesonpy", ) def test_unknown_option() -> None: - with pytest.raises(SbomError, match=r"here has unknown option\(s\) bundled, suplier; known: bundled-wheels"): + with pytest.raises( + SbomError, match=r"here has unknown option\(s\) bundled, suplier; known: build-backend, bundled-wheels" + ): settings_from_table({"suplier": {}, "bundled": []}, "here") @@ -62,6 +66,7 @@ def test_unknown_option() -> None: pytest.param({"bundled-wheels": "*.whl"}, "`bundled-wheels` in here must be an array", id="bundled-not-list"), pytest.param({"bundled-wheels": [1]}, "`bundled-wheels` in here must be an array", id="bundled-not-strings"), pytest.param({"content-provider": 1}, "`content-provider` in here must be a string", id="content-provider"), + pytest.param({"build-backend": 1}, "`build-backend` in here must be a string", id="build-backend"), pytest.param({"supplier": "Example"}, "`supplier` in here must be a table", id="supplier-not-table"), pytest.param({"supplier": {"name": 1}}, "`supplier` in here must be a table", id="supplier-name"), pytest.param({"supplier": {"name": "a", "url": "x"}}, "`supplier` in here must be a table", id="supplier-url"), diff --git a/tests/_formats/test_json.py b/tests/_formats/test_json.py new file mode 100644 index 0000000..147fd8e --- /dev/null +++ b/tests/_formats/test_json.py @@ -0,0 +1,24 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING + +import pytest + +from python_sbom._formats._json import is_document + +if TYPE_CHECKING: + from python_sbom._formats._json import Json + + +@pytest.mark.parametrize( + ("value", "expected"), + [ + pytest.param({"metadata": {"component": {"bom-ref": "plain", "name": "plain"}}}, True, id="root-named"), + pytest.param({"metadata": {"component": {"name": "plain"}}}, False, id="root-without-bom-ref"), + pytest.param({"metadata": {"component": {"bom-ref": "plain"}}}, False, id="root-without-name"), + pytest.param({"metadata": {}}, False, id="no-root"), + pytest.param([], False, id="not-an-object"), + ], +) +def test_is_document(value: Json, expected: bool) -> None: + assert is_document(value) is expected diff --git a/tests/_model/test_archive.py b/tests/_model/test_archive.py new file mode 100644 index 0000000..cab4bbe --- /dev/null +++ b/tests/_model/test_archive.py @@ -0,0 +1,27 @@ +from __future__ import annotations + +from pathlib import PurePosixPath + +from python_sbom._model._archive import bundled_paths +from python_sbom._model._inputs import ArchiveTree + + +def test_bundled_paths_list_top_level_wheels_then_each_unpacked_wheel_with_its_own() -> None: + tree = ArchiveTree( + metadata={}, + owned={ + "": ["top.txt"], + "lib/b/": ["lib/b/inner.whl", "lib/b/code.py"], + "lib/a/": ["lib/a/code.py"], + }, + loose=["z.whl", "readme.txt", "a.whl"], + files={}, + bundled={}, + ) + assert bundled_paths(tree) == [ + PurePosixPath("z.whl"), + PurePosixPath("a.whl"), + PurePosixPath("lib/a"), + PurePosixPath("lib/b"), + PurePosixPath("lib/b/inner.whl"), + ] diff --git a/tests/_model/test_bom.py b/tests/_model/test_bom.py index 2b0b632..3ed30d0 100644 --- a/tests/_model/test_bom.py +++ b/tests/_model/test_bom.py @@ -4,20 +4,28 @@ import json import uuid from pathlib import PurePosixPath -from typing import TYPE_CHECKING, Final +from typing import TYPE_CHECKING, Final, cast import pytest from python_sbom import BundledWheelInfo, ContentProvider, Contributions, ProviderContext +from python_sbom._errors import SbomError +from python_sbom._model._bom import ( + check_refs, + checked_contributions, + contributed_properties, + formulation, + with_header, +) from python_sbom._model._inputs import BuildScript, BundledWheel, Revision from python_sbom._model._metadata import parse_metadata if TYPE_CHECKING: from conftest import Describe - from python_sbom import Component + from python_sbom import Component, Dependency, Property from python_sbom._model._inputs import BuildFacts, Settings - from python_sbom._model._types import OrganizationalEntity + from python_sbom._model._types import Body, Formula, OrganizationalEntity _INTERPRETER: Final[str] = "tool:pkg:generic/cpython@3.15.0" @@ -212,68 +220,6 @@ def test_revision_without_repository_records_commit_only(describe: Describe, fac assert [reference["type"] for reference in root["externalReferences"]] == ["distribution", "attestation"] -def test_declared_dependency(describe: Describe) -> None: - requirement = 'tomli>=2; python_version < "3.11"' - document = describe(f"Requires-Dist: {requirement}\n") - assert document["components"] == [ - { - "type": "library", - "bom-ref": f"requires-dist:{requirement}", - "name": "tomli", - "purl": "pkg:pypi/tomli", - "externalReferences": [{"type": "distribution", "url": "https://pypi.org/project/tomli/"}], - "properties": [ - {"name": "example:requires-dist", "value": requirement}, - {"name": "example:environment-marker", "value": 'python_version < "3.11"'}, - ], - } - ] - - -def test_repeated_requirement_is_one_component(describe: Describe) -> None: - document = describe("Requires-Dist: packaging>=24\nRequires-Dist: packaging>=24\n") - assert [component["bom-ref"] for component in document["components"]] == ["requires-dist:packaging>=24"] - - -def test_scope_names_bundled_directories(describe: Describe, settings: Settings) -> None: - document = describe(settings=dataclasses.replace(settings, bundled_wheels=("src/b/*.whl", "src/a/x/*.whl"))) - assert document["metadata"]["properties"][0]["value"] == ( - "Components are the wheels bundled as data files under src/a/x and src/b and the direct runtime dependencies" - " declared in the wheel's core metadata, which carry no version because the installer resolves them." - ) - - -def test_bundled_wheel_and_its_vendored_packages(describe: Describe) -> None: - wheel = BundledWheel( - path="src/pip-1.0-py3-none-any.whl", - sha256="ab" * 32, - metadata_name="pip-1.0.dist-info/METADATA", - metadata=parse_metadata("Name: pip\nVersion: 1.0\n"), - files=[], - vendored=[("pip/_vendor/vendor.txt", parse_metadata("Name: urllib3\nVersion: 1.26.4\n"))], - ) - - class Seeded(ContentProvider): - def __init__(self, name: str) -> None: - self._name = name - - def bundled_wheel(self, context: ProviderContext, wheel: PurePosixPath) -> BundledWheelInfo: - return BundledWheelInfo( - properties=[{"name": f"{context.identity.property_prefix}:{self._name}", "value": str(wheel)}] - ) - - document = describe(bundled=[wheel], content=Seeded("seeded")) - assert document["components"][0]["properties"][1] == { - "name": "example:seeded", - "value": "src/pip-1.0-py3-none-any.whl", - } - bundled_ref = "bundled:src/pip-1.0-py3-none-any.whl" - assert document["dependencies"][:2] == [ - {"ref": "pkg:pypi/example@1.0", "dependsOn": [bundled_ref]}, - {"ref": bundled_ref, "dependsOn": [f"{bundled_ref}#vendored/pkg:pypi/urllib3@1.26.4"]}, - ] - - @pytest.mark.parametrize( ("revision", "tree"), [ @@ -393,3 +339,127 @@ def test_bundled_wheel_supplier_is_the_projects_choice( content=content, ) assert document["components"][0].get("supplier") == expected + + +def test_document_serial_derives_from_body() -> None: + body: Body = { + "metadata": { + "timestamp": "2023-11-14T22:13:20+00:00", + "lifecycles": [{"phase": "build"}], + "tools": {"components": []}, + "component": {"type": "application", "bom-ref": "app", "name": "app"}, + }, + "components": [], + "dependencies": [], + } + namespace = uuid.uuid5(uuid.NAMESPACE_URL, "https://example.org/sboms") + assert with_header(body, namespace) == { + "$schema": "http://cyclonedx.org/schema/bom-1.6.schema.json", + "bomFormat": "CycloneDX", + "specVersion": "1.6", + "serialNumber": f"urn:uuid:{uuid.uuid5(namespace, json.dumps(body, sort_keys=True))}", + "version": 1, + **body, + } + + +def test_check_refs_refuses_repeated_refs() -> None: + body: Body = { + "metadata": { + "timestamp": "2023-11-14T22:13:20+00:00", + "lifecycles": [{"phase": "build"}], + "tools": {"components": [{"type": "application", "bom-ref": "app", "name": "tool"}]}, + "component": {"type": "application", "bom-ref": "app", "name": "app"}, + }, + "components": [], + "dependencies": [], + } + with pytest.raises(SbomError, match="bom-ref values repeat: app"): + check_refs(body) + + +def test_check_refs_refuses_unknown_dependency() -> None: + body: Body = { + "metadata": { + "timestamp": "2023-11-14T22:13:20+00:00", + "lifecycles": [{"phase": "build"}], + "tools": {"components": []}, + "component": {"type": "application", "bom-ref": "app", "name": "app"}, + }, + "components": [], + "dependencies": [{"ref": "app", "dependsOn": ["ghost"]}], + } + with pytest.raises(SbomError, match="dependencies name bom-refs no component has: ghost"): + check_refs(body) + + +def test_contributed_properties_keep_their_own_prefix() -> None: + properties: list[Property] = [{"name": "extra:key", "value": "1"}] + assert contributed_properties(properties, "example") == properties + + +def test_contributed_properties_refuse_the_project_prefix() -> None: + with pytest.raises(SbomError, match="example:vcs-commit use the example: prefix python-sbom writes"): + contributed_properties([{"name": "example:vcs-commit", "value": "0"}], "example") + + +def _partial(**fields: str | list[dict[str, str]]) -> Component: + # a component a provider got wrong, which the types alone would not let through + return cast("Component", fields) + + +def test_check_refs_refuses_a_second_dependency_entry_for_a_ref() -> None: + # as when a provider adds an entry for the root, whose entry python-sbom writes + body = _body([], [{"ref": "app", "dependsOn": []}, {"ref": "app", "dependsOn": []}], []) + with pytest.raises(SbomError, match="dependencies list more than one entry for: app"): + check_refs(body) + + +def _body(components: list[Component], dependencies: list[Dependency], formulation: list[Formula]) -> Body: + return { + "metadata": { + "timestamp": "2023-11-14T22:13:20+00:00", + "lifecycles": [{"phase": "build"}], + "tools": {"components": []}, + "component": {"type": "application", "bom-ref": "app", "name": "app"}, + }, + "components": components, + "dependencies": dependencies, + "formulation": formulation, + } + + +@pytest.mark.parametrize( + "ref", [pytest.param("formula:build", id="formula"), pytest.param("workflow:build", id="workflow")] +) +def test_check_refs_refuses_a_component_taking_a_formulation_ref(facts: BuildFacts, ref: str) -> None: + formula = formulation("build", "build the app", [], "app", facts) + body = _body([{"type": "library", "bom-ref": ref, "name": "clash"}], [], [formula]) + with pytest.raises(SbomError, match=f"bom-ref values repeat: {ref}"): + check_refs(body) + + +@pytest.mark.parametrize( + ("contributions", "missing"), + [ + pytest.param(Contributions(components=[_partial(type="library", name="zlib")]), "bom-ref", id="component"), + pytest.param( + Contributions( + components=[ + _partial(type="library", name="zlib", **{"bom-ref": "zlib", "components": [{"type": "file"}]}) + ] + ), + "bom-ref, name", + id="nested", + ), + pytest.param(Contributions(tools=[_partial(name="cmake", **{"bom-ref": "tool:cmake"})]), "type", id="tool"), + ], +) +def test_contributions_need_the_fields_the_build_reads(contributions: Contributions, missing: str) -> None: + with pytest.raises(SbomError, match=f"content provider ContentProvider added a component without {missing}: "): + checked_contributions(contributions, ContentProvider()) + + +def test_complete_contributions_pass() -> None: + contributions = Contributions(components=[{"type": "library", "bom-ref": "zlib", "name": "zlib"}]) + assert checked_contributions(contributions, ContentProvider()) is contributions diff --git a/tests/_model/test_components.py b/tests/_model/test_components.py index c09ed62..df37e9f 100644 --- a/tests/_model/test_components.py +++ b/tests/_model/test_components.py @@ -1,6 +1,5 @@ from __future__ import annotations -from pathlib import PurePosixPath from typing import TYPE_CHECKING, Final import pytest @@ -9,7 +8,6 @@ from python_sbom._model._components import ( bundled_wheel_component, component_from_metadata, - script_tool, vendored_dependencies, ) from python_sbom._model._inputs import BundledWheel, RecordRow @@ -310,22 +308,6 @@ def test_bundled_wheel_supplier_never_comes_from_the_wheel() -> None: assert "supplier" not in component -def test_script_tool_with_link() -> None: - tool = script_tool(PurePosixPath("build/hook.py"), "ef" * 32, "1.0", "https://example.com/hook.py") - assert tool == { - "type": "application", - "bom-ref": "tool:build/hook.py", - "name": "hook.py", - "version": "1.0", - "hashes": [{"alg": "SHA-256", "content": "ef" * 32}], - "externalReferences": [{"type": "vcs", "url": "https://example.com/hook.py"}], - } - - -def test_script_tool_without_link() -> None: - assert "externalReferences" not in script_tool(PurePosixPath("hook.py"), "ef" * 32, "1.0", None) - - def test_vendored_spellings_of_one_package_merge() -> None: wheel = BundledWheel( path=_WHEEL, diff --git a/tests/_model/test_document.py b/tests/_model/test_document.py deleted file mode 100644 index 260a087..0000000 --- a/tests/_model/test_document.py +++ /dev/null @@ -1,146 +0,0 @@ -from __future__ import annotations - -import json -import uuid -from typing import TYPE_CHECKING, cast - -import pytest - -from python_sbom import ContentProvider, Contributions -from python_sbom._errors import SbomError -from python_sbom._model._document import ( - check_refs, - checked_contributions, - contributed_properties, - formulation, - with_header, -) - -if TYPE_CHECKING: - from python_sbom import Component, Dependency, Property - from python_sbom._model._inputs import BuildFacts - from python_sbom._model._types import Body, Formula - - -def test_document_serial_derives_from_body() -> None: - body: Body = { - "metadata": { - "timestamp": "2023-11-14T22:13:20+00:00", - "lifecycles": [{"phase": "build"}], - "tools": {"components": []}, - "component": {"type": "application", "bom-ref": "app", "name": "app"}, - }, - "components": [], - "dependencies": [], - } - namespace = uuid.uuid5(uuid.NAMESPACE_URL, "https://example.org/sboms") - assert with_header(body, namespace) == { - "$schema": "http://cyclonedx.org/schema/bom-1.6.schema.json", - "bomFormat": "CycloneDX", - "specVersion": "1.6", - "serialNumber": f"urn:uuid:{uuid.uuid5(namespace, json.dumps(body, sort_keys=True))}", - "version": 1, - **body, - } - - -def test_check_refs_refuses_repeated_refs() -> None: - body: Body = { - "metadata": { - "timestamp": "2023-11-14T22:13:20+00:00", - "lifecycles": [{"phase": "build"}], - "tools": {"components": [{"type": "application", "bom-ref": "app", "name": "tool"}]}, - "component": {"type": "application", "bom-ref": "app", "name": "app"}, - }, - "components": [], - "dependencies": [], - } - with pytest.raises(SbomError, match="bom-ref values repeat: app"): - check_refs(body) - - -def test_check_refs_refuses_unknown_dependency() -> None: - body: Body = { - "metadata": { - "timestamp": "2023-11-14T22:13:20+00:00", - "lifecycles": [{"phase": "build"}], - "tools": {"components": []}, - "component": {"type": "application", "bom-ref": "app", "name": "app"}, - }, - "components": [], - "dependencies": [{"ref": "app", "dependsOn": ["ghost"]}], - } - with pytest.raises(SbomError, match="dependencies name bom-refs no component has: ghost"): - check_refs(body) - - -def test_contributed_properties_keep_their_own_prefix() -> None: - properties: list[Property] = [{"name": "extra:key", "value": "1"}] - assert contributed_properties(properties, "example") == properties - - -def test_contributed_properties_refuse_the_project_prefix() -> None: - with pytest.raises(SbomError, match="example:vcs-commit use the example: prefix python-sbom writes"): - contributed_properties([{"name": "example:vcs-commit", "value": "0"}], "example") - - -def _partial(**fields: str | list[dict[str, str]]) -> Component: - # a component a provider got wrong, which the types alone would not let through - return cast("Component", fields) - - -def test_check_refs_refuses_a_second_dependency_entry_for_a_ref() -> None: - # as when a provider adds an entry for the root, whose entry python-sbom writes - body = _body([], [{"ref": "app", "dependsOn": []}, {"ref": "app", "dependsOn": []}], []) - with pytest.raises(SbomError, match="dependencies list more than one entry for: app"): - check_refs(body) - - -def _body(components: list[Component], dependencies: list[Dependency], formulation: list[Formula]) -> Body: - return { - "metadata": { - "timestamp": "2023-11-14T22:13:20+00:00", - "lifecycles": [{"phase": "build"}], - "tools": {"components": []}, - "component": {"type": "application", "bom-ref": "app", "name": "app"}, - }, - "components": components, - "dependencies": dependencies, - "formulation": formulation, - } - - -@pytest.mark.parametrize( - "ref", [pytest.param("formula:build", id="formula"), pytest.param("workflow:build", id="workflow")] -) -def test_check_refs_refuses_a_component_taking_a_formulation_ref(facts: BuildFacts, ref: str) -> None: - formula = formulation("build", "build the app", [], "app", facts) - body = _body([{"type": "library", "bom-ref": ref, "name": "clash"}], [], [formula]) - with pytest.raises(SbomError, match=f"bom-ref values repeat: {ref}"): - check_refs(body) - - -@pytest.mark.parametrize( - ("contributions", "missing"), - [ - pytest.param(Contributions(components=[_partial(type="library", name="zlib")]), "bom-ref", id="component"), - pytest.param( - Contributions( - components=[ - _partial(type="library", name="zlib", **{"bom-ref": "zlib", "components": [{"type": "file"}]}) - ] - ), - "bom-ref, name", - id="nested", - ), - pytest.param(Contributions(tools=[_partial(name="cmake", **{"bom-ref": "tool:cmake"})]), "type", id="tool"), - ], -) -def test_contributions_need_the_fields_the_build_reads(contributions: Contributions, missing: str) -> None: - with pytest.raises(SbomError, match=f"content provider ContentProvider added a component without {missing}: "): - checked_contributions(contributions, ContentProvider()) - - -def test_complete_contributions_pass() -> None: - contributions = Contributions(components=[{"type": "library", "bom-ref": "zlib", "name": "zlib"}]) - assert checked_contributions(contributions, ContentProvider()) is contributions diff --git a/tests/_model/test_graph.py b/tests/_model/test_graph.py new file mode 100644 index 0000000..2ebcd7c --- /dev/null +++ b/tests/_model/test_graph.py @@ -0,0 +1,71 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING + +import pytest + +from python_sbom._model._graph import reference_problems, shipped_libraries, walk_components + +if TYPE_CHECKING: + from python_sbom import Component, Dependency + + +def test_walk_visits_each_component_before_its_children_in_order() -> None: + components: list[Component] = [ + { + "type": "library", + "bom-ref": "a", + "name": "a", + "components": [{"type": "file", "bom-ref": "a1", "name": "a1"}], + }, + {"type": "library", "bom-ref": "b", "name": "b"}, + ] + assert [component["bom-ref"] for component in walk_components(components)] == ["a", "a1", "b"] + + +def test_walk_survives_nesting_deeper_than_the_recursion_limit() -> None: + root: Component = {"type": "library", "bom-ref": "0", "name": "0"} + node = root + for depth in range(1, 5000): + child: Component = {"type": "library", "bom-ref": str(depth), "name": str(depth)} + node["components"] = [child] + node = child + assert sum(1 for _ in walk_components([root])) == 5000 + + +def test_shipped_libraries_skip_declared_requirements() -> None: + hashed: Component = {"type": "library", "bom-ref": "hashed", "name": "x", "hashes": []} + with_files: Component = { + "type": "library", + "bom-ref": "files", + "name": "y", + "components": [{"type": "file", "bom-ref": "f", "name": "f"}], + } + declared: Component = {"type": "library", "bom-ref": "declared", "name": "z"} + assert [component["bom-ref"] for component in shipped_libraries([hashed, with_files, declared])] == [ + "hashed", + "files", + ] + + +@pytest.mark.parametrize( + ("refs", "dependencies", "expected"), + [ + pytest.param(["a", "b"], [{"ref": "a", "dependsOn": ["b"]}], [], id="valid"), + pytest.param(["a", "a"], [], ["bom-ref values repeat: a"], id="repeated-ref"), + pytest.param( + ["a"], + [{"ref": "a"}, {"ref": "a"}], + ["dependencies list more than one entry for: a"], + id="repeated-entry", + ), + pytest.param( + ["a"], + [{"ref": "a", "dependsOn": ["c", "b"]}], + ["dependencies name bom-refs no component has: b, c"], + id="unknown", + ), + ], +) +def test_reference_problems(refs: list[str], dependencies: list[Dependency], expected: list[str]) -> None: + assert reference_problems(refs, dependencies) == expected diff --git a/tests/_model/test_names.py b/tests/_model/test_names.py new file mode 100644 index 0000000..5006ad4 --- /dev/null +++ b/tests/_model/test_names.py @@ -0,0 +1,39 @@ +from __future__ import annotations + +import pytest + +from python_sbom._model._names import purl, pypi_url, reference_type, sbom_file_name + + +@pytest.mark.parametrize( + ("label", "expected"), + [ + pytest.param("Bug Tracker", "issue-tracker", id="spaced"), + pytest.param("Release-Notes", "release-notes", id="dashed"), + pytest.param("Source_Code", "vcs", id="underscored"), + pytest.param("Funding", "other", id="unknown"), + ], +) +def test_reference_type_normalizes_labels(label: str, expected: str) -> None: + assert reference_type(label) == expected + + +def test_purl_normalizes_every_spelling_alike() -> None: + assert {purl(name, "1.0") for name in ("jaraco.context", "jaraco_context", "Jaraco-Context")} == { + "pkg:pypi/jaraco-context@1.0" + } + + +@pytest.mark.parametrize( + ("version", "expected"), + [ + pytest.param("1.0", "https://pypi.org/project/pip/1.0/", id="release"), + pytest.param(None, "https://pypi.org/project/pip/", id="project"), + ], +) +def test_pypi_url(version: str | None, expected: str) -> None: + assert pypi_url("pip", version) == expected + + +def test_sbom_file_name_uses_the_normalized_name() -> None: + assert sbom_file_name("My_Project") == "my-project.cdx.json" diff --git a/tests/_wheel/test_record.py b/tests/_model/test_record.py similarity index 96% rename from tests/_wheel/test_record.py rename to tests/_model/test_record.py index 9fb6fd5..4266c89 100644 --- a/tests/_wheel/test_record.py +++ b/tests/_model/test_record.py @@ -6,7 +6,7 @@ from python_sbom._errors import SbomError from python_sbom._model._inputs import RecordRow -from python_sbom._wheel._record import record_rows +from python_sbom._model._record import record_rows @pytest.mark.parametrize( diff --git a/tests/_model/test_tools.py b/tests/_model/test_tools.py index cc3e859..704704b 100644 --- a/tests/_model/test_tools.py +++ b/tests/_model/test_tools.py @@ -1,11 +1,12 @@ from __future__ import annotations import dataclasses +from pathlib import PurePosixPath from typing import TYPE_CHECKING from python_sbom._model._inputs import InstalledDistribution, RecordRow from python_sbom._model._metadata import parse_metadata -from python_sbom._model._tools import tool_components +from python_sbom._model._tools import script_tool, tool_components if TYPE_CHECKING: from python_sbom._model._inputs import BuildFacts @@ -64,3 +65,19 @@ def test_tool_bom_ref_and_purl_agree(facts: BuildFacts) -> None: ) (_, component), _ = tool_components(dataclasses.replace(facts, distributions=[tool])) assert (component["bom-ref"], component.get("purl")) == ("tool:pkg:pypi/boolean-py@5.0", "pkg:pypi/boolean-py@5.0") + + +def test_script_tool_with_link() -> None: + tool = script_tool(PurePosixPath("build/hook.py"), "ef" * 32, "1.0", "https://example.com/hook.py") + assert tool == { + "type": "application", + "bom-ref": "tool:build/hook.py", + "name": "hook.py", + "version": "1.0", + "hashes": [{"alg": "SHA-256", "content": "ef" * 32}], + "externalReferences": [{"type": "vcs", "url": "https://example.com/hook.py"}], + } + + +def test_script_tool_without_link() -> None: + assert "externalReferences" not in script_tool(PurePosixPath("hook.py"), "ef" * 32, "1.0", None) diff --git a/tests/_model/test_wheel.py b/tests/_model/test_wheel.py new file mode 100644 index 0000000..d8d999c --- /dev/null +++ b/tests/_model/test_wheel.py @@ -0,0 +1,77 @@ +from __future__ import annotations + +import dataclasses +from typing import TYPE_CHECKING + +from python_sbom import BundledWheelInfo, ContentProvider, ProviderContext +from python_sbom._model._inputs import BundledWheel +from python_sbom._model._metadata import parse_metadata + +if TYPE_CHECKING: + from pathlib import PurePosixPath + + from conftest import Describe + + from python_sbom._model._inputs import Settings + + +def test_declared_dependency(describe: Describe) -> None: + requirement = 'tomli>=2; python_version < "3.11"' + document = describe(f"Requires-Dist: {requirement}\n") + assert document["components"] == [ + { + "type": "library", + "bom-ref": f"requires-dist:{requirement}", + "name": "tomli", + "purl": "pkg:pypi/tomli", + "externalReferences": [{"type": "distribution", "url": "https://pypi.org/project/tomli/"}], + "properties": [ + {"name": "example:requires-dist", "value": requirement}, + {"name": "example:environment-marker", "value": 'python_version < "3.11"'}, + ], + } + ] + + +def test_repeated_requirement_is_one_component(describe: Describe) -> None: + document = describe("Requires-Dist: packaging>=24\nRequires-Dist: packaging>=24\n") + assert [component["bom-ref"] for component in document["components"]] == ["requires-dist:packaging>=24"] + + +def test_scope_names_bundled_directories(describe: Describe, settings: Settings) -> None: + document = describe(settings=dataclasses.replace(settings, bundled_wheels=("src/b/*.whl", "src/a/x/*.whl"))) + assert document["metadata"]["properties"][0]["value"] == ( + "Components are the wheels bundled as data files under src/a/x and src/b and the direct runtime dependencies" + " declared in the wheel's core metadata, which carry no version because the installer resolves them." + ) + + +def test_bundled_wheel_and_its_vendored_packages(describe: Describe) -> None: + wheel = BundledWheel( + path="src/pip-1.0-py3-none-any.whl", + sha256="ab" * 32, + metadata_name="pip-1.0.dist-info/METADATA", + metadata=parse_metadata("Name: pip\nVersion: 1.0\n"), + files=[], + vendored=[("pip/_vendor/vendor.txt", parse_metadata("Name: urllib3\nVersion: 1.26.4\n"))], + ) + + class Seeded(ContentProvider): + def __init__(self, name: str) -> None: + self._name = name + + def bundled_wheel(self, context: ProviderContext, wheel: PurePosixPath) -> BundledWheelInfo: + return BundledWheelInfo( + properties=[{"name": f"{context.identity.property_prefix}:{self._name}", "value": str(wheel)}] + ) + + document = describe(bundled=[wheel], content=Seeded("seeded")) + assert document["components"][0]["properties"][1] == { + "name": "example:seeded", + "value": "src/pip-1.0-py3-none-any.whl", + } + bundled_ref = "bundled:src/pip-1.0-py3-none-any.whl" + assert document["dependencies"][:2] == [ + {"ref": "pkg:pypi/example@1.0", "dependsOn": [bundled_ref]}, + {"ref": bundled_ref, "dependsOn": [f"{bundled_ref}#vendored/pkg:pypi/urllib3@1.26.4"]}, + ] diff --git a/tests/_validate/test_document.py b/tests/_validate/test_document.py index 55e1354..6e58f99 100644 --- a/tests/_validate/test_document.py +++ b/tests/_validate/test_document.py @@ -9,7 +9,7 @@ from python_sbom._formats._json import dumps from python_sbom._model._inputs import InstalledDistribution, RecordRow from python_sbom._model._metadata import parse_metadata -from python_sbom._validate._document import is_document, validate_document +from python_sbom._validate._document import validate_document if TYPE_CHECKING: from collections.abc import Callable @@ -18,9 +18,9 @@ from pytest_mock import MockerFixture from python_sbom import Component, License + from python_sbom._formats._json import Json from python_sbom._model._inputs import BuildFacts from python_sbom._model._types import Document - from python_sbom._validate._limits import Json # what auditwheel writes: CycloneDX 1.4 conventions, a legacy tools array, no timestamp _FOREIGN: Final[bytes] = json.dumps({ @@ -301,20 +301,6 @@ def test_unexpected_error_is_reported(describe: Describe, mocker: MockerFixture) assert validate_document(dumps(describe()), None) == [f"could not be checked, RuntimeError: {'x' * 200}"] -@pytest.mark.parametrize( - ("value", "expected"), - [ - pytest.param(json.loads(_FOREIGN), True, id="foreign"), - pytest.param({"metadata": {"component": {"name": "plain"}}}, False, id="root-without-bom-ref"), - pytest.param({"metadata": {"component": {"bom-ref": "plain"}}}, False, id="root-without-name"), - pytest.param({"metadata": {}}, False, id="no-root"), - pytest.param([], False, id="not-an-object"), - ], -) -def test_is_document(value: Json, expected: bool) -> None: - assert is_document(value) is expected - - def test_repeated_dependency_entry_cannot_hide_an_invalid_one(describe: Describe) -> None: # read as a mapping, the valid second entry for the root would replace the invalid first one, which the SPDX # rendering still turns into an edge diff --git a/tests/_validate/test_schemas.py b/tests/_validate/test_schemas.py index 02820e7..9cf6bfe 100644 --- a/tests/_validate/test_schemas.py +++ b/tests/_validate/test_schemas.py @@ -21,8 +21,8 @@ from conftest import Describe from python_sbom import Component + from python_sbom._formats._json import Json from python_sbom._formats._spdx import Spdx - from python_sbom._validate._limits import Json @pytest.mark.parametrize( diff --git a/tests/_wheel/test_sdist.py b/tests/_wheel/test_sdist.py new file mode 100644 index 0000000..c2ac83a --- /dev/null +++ b/tests/_wheel/test_sdist.py @@ -0,0 +1,67 @@ +from __future__ import annotations + +import io +import tarfile +from typing import TYPE_CHECKING + +import pytest + +from python_sbom._wheel._sdist import add_to_sdist + +if TYPE_CHECKING: + from pathlib import Path + from typing import IO + + +@pytest.fixture +def sdist(tmp_path: Path) -> Path: + with tarfile.open(path := tmp_path / "example-1.0.tar.gz", "w:gz") as archive: + directory = tarfile.TarInfo("example-1.0") + directory.type, directory.mtime = tarfile.DIRTYPE, 100 + archive.addfile(directory) + source = tarfile.TarInfo("example-1.0/PKG-INFO") + source.size, source.mtime = 4, 200 + archive.addfile(source, io.BytesIO(b"meta")) + return path + + +def test_adds_under_the_top_directory(sdist: Path) -> None: + add_to_sdist(sdist, ".python-sbom-revision", b"node: x\n") + with tarfile.open(sdist) as archive: + added = archive.getmember("example-1.0/.python-sbom-revision") + handle = archive.extractfile(added) + assert handle is not None + assert (archive.getnames(), handle.read(), added.mtime) == ( + ["example-1.0", "example-1.0/PKG-INFO", "example-1.0/.python-sbom-revision"], + b"node: x\n", + 200, + ) + + +def test_sdist_already_holding_the_file_is_kept(sdist: Path) -> None: + # a backend running python-sbom's own plugin under the wrapper records the revision itself + add_to_sdist(sdist, ".python-sbom-revision", b"node: x\n") + first = sdist.read_bytes() + add_to_sdist(sdist, ".python-sbom-revision", b"node: y\n") + assert sdist.read_bytes() == first + + +def test_rewrite_is_reproducible(sdist: Path, tmp_path: Path) -> None: + (again := tmp_path / "again" / sdist.name).parent.mkdir() + again.write_bytes(sdist.read_bytes()) + add_to_sdist(sdist, ".python-sbom-revision", b"node: x\n") + add_to_sdist(again, ".python-sbom-revision", b"node: x\n") + assert sdist.read_bytes() == again.read_bytes() + + +def test_failed_rewrite_keeps_the_sdist(sdist: Path, monkeypatch: pytest.MonkeyPatch) -> None: + original = sdist.read_bytes() + + def full_disk(_archive: tarfile.TarFile, _member: tarfile.TarInfo, _content: IO[bytes] | None = None) -> None: + msg = "disk full" + raise OSError(msg) + + monkeypatch.setattr(tarfile.TarFile, "addfile", full_disk) + with pytest.raises(OSError, match="disk full"): + add_to_sdist(sdist, ".python-sbom-revision", b"node: x\n") + assert ([path.name for path in sdist.parent.iterdir()], sdist.read_bytes()) == ([sdist.name], original) diff --git a/tests/conftest.py b/tests/conftest.py index 47c74b6..e04ffc9 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -11,8 +11,7 @@ import pytest from python_sbom import ContentProvider, ProviderContext -from python_sbom._model._bom import assemble -from python_sbom._model._document import checked_contributions +from python_sbom._model._bom import assemble, checked_contributions from python_sbom._model._identity import project_identity from python_sbom._model._inputs import BuildFacts, BuildScript, BundledWheel, Interpreter, Settings, WheelTree from python_sbom._model._metadata import parse_metadata @@ -139,7 +138,14 @@ def facts() -> BuildFacts: @pytest.fixture def settings() -> Settings: - return Settings(supplier=None, repository=None, default_branch="main", bundled_wheels=(), content_provider=None) + return Settings( + supplier=None, + repository=None, + default_branch="main", + bundled_wheels=(), + content_provider=None, + build_backend=None, + ) @pytest.fixture diff --git a/tests/test_backend.py b/tests/test_backend.py new file mode 100644 index 0000000..f94d7c0 --- /dev/null +++ b/tests/test_backend.py @@ -0,0 +1,186 @@ +from __future__ import annotations + +import json +import tarfile +import zipfile +from pathlib import Path +from typing import TYPE_CHECKING, Final + +import flit_core.buildapi +import pytest + +from python_sbom import backend +from python_sbom._errors import SbomError +from python_sbom._validate._artifact import validate + +if TYPE_CHECKING: + from collections.abc import Mapping + + from tests.conftest import Git + +# an in-tree backend that defines only the two hooks PEP 517 requires, and an sdist holding the revision file already +_MINIMAL: Final[str] = """\ +import flit_core.buildapi as flit + +build_wheel = flit.build_wheel + + +def build_sdist(sdist_directory, config_settings=None): + return flit.build_sdist(sdist_directory, config_settings) +""" + + +@pytest.fixture +def project(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: + return write_project(tmp_path / "project", "build-backend = 'flit_core.buildapi'\n", monkeypatch) + + +def write_project(root: Path, options: str, monkeypatch: pytest.MonkeyPatch) -> Path: + (root / "example").mkdir(parents=True, exist_ok=True) + (root / "example" / "__init__.py").write_text("", encoding="utf-8") + (root / "LICENSE").write_text("Copyright (c) 2026 Example\n\nMIT terms\n", encoding="utf-8") + project = """\ +[build-system] +build-backend = "python_sbom.backend" +requires = ["flit-core>=3.12", "python-sbom"] + +[project] +name = "example" +version = "1.0" +description = "an example" +license = "MIT" +license-files = ["LICENSE"] +urls.Source = "https://github.com/example/example" + +[tool.python-sbom] +""" + (root / "pyproject.toml").write_text(f"{project}{options}", encoding="utf-8") + # PEP 517 runs every hook from the project directory, where an in-tree backend is importable too + # a frontend creates the output directory before it calls the backend + (root / "dist").mkdir(exist_ok=True) + monkeypatch.chdir(root) + monkeypatch.syspath_prepend(str(root)) + return root + + +def test_wheel_carries_valid_sbom(project: Path) -> None: + wheel = project / "dist" / backend.build_wheel(str(project / "dist")) + assert validate(wheel, frozenset({"MIT"})) == [] + + +def test_sbom_records_the_build_environment(project: Path) -> None: + wheel = project / "dist" / backend.build_wheel(str(project / "dist")) + tools = json.loads(zipfile.ZipFile(wheel).read("example-1.0.dist-info/sboms/example.cdx.json"))["metadata"][ + "tools" + ]["components"] + assert "flit_core" in {tool["name"] for tool in tools} + + +@pytest.mark.usefixtures("project") +def test_optional_hook_comes_from_the_wrapped_backend() -> None: + assert backend.prepare_metadata_for_build_wheel is flit_core.buildapi.prepare_metadata_for_build_wheel + + +def test_optional_hook_the_backend_lacks_is_missing(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + (tmp_path / "project").mkdir() + (tmp_path / "project" / f"{tmp_path.name}.py").write_text(_MINIMAL, encoding="utf-8") + write_project(tmp_path / "project", f"build-backend = '{tmp_path.name}'\n", monkeypatch) + assert not hasattr(backend, "prepare_metadata_for_build_wheel") + + +@pytest.mark.usefixtures("project") +def test_unknown_attribute_is_missing() -> None: + with pytest.raises(AttributeError, match=r"module 'python_sbom.backend' has no attribute 'build_everything'"): + backend.build_everything # ruff: ignore[useless-expression] # the lookup is the test + + +def test_backend_object_reference(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + (tmp_path / "project").mkdir() + (tmp_path / "project" / f"{tmp_path.name}.py").write_text( + "import types\nimport flit_core.buildapi as flit\n\n" + "holder = types.SimpleNamespace(api=types.SimpleNamespace(build_wheel=flit.build_wheel))\n", + encoding="utf-8", + ) + root = write_project(tmp_path / "project", f"build-backend = '{tmp_path.name}:holder.api'\n", monkeypatch) + assert validate(root / "dist" / backend.build_wheel(str(root / "dist")), frozenset({"MIT"})) == [] + + +def test_missing_backend_option(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + write_project(tmp_path / "project", "", monkeypatch) + with pytest.raises(SbomError, match=r"names no backend to wrap; set build-backend in \[tool.python-sbom\]"): + backend.build_wheel(str(tmp_path / "dist")) + + +def test_refuses_to_wrap_itself(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + write_project(tmp_path / "project", "build-backend = 'python_sbom.backend'\n", monkeypatch) + with pytest.raises(SbomError, match=r"names python_sbom.backend, which would wrap itself"): + backend.build_sdist(str(tmp_path / "dist")) + + +def test_editable_wheel_has_no_sbom(project: Path) -> None: + # the module-level hook lookup types every optional hook alike, so the name is narrowed here + name = backend.build_editable(str(project / "dist")) + assert isinstance(name, str) + wheel = project / "dist" / name + assert not [name for name in zipfile.ZipFile(wheel).namelist() if "/sboms/" in name] + + +@pytest.fixture +def checkout(tmp_path: Path, git: Git, monkeypatch: pytest.MonkeyPatch) -> Path: + root = write_project(tmp_path / "checkout", "build-backend = 'flit_core.buildapi'\n", monkeypatch) + (root / ".gitignore").write_text("dist/\n", encoding="utf-8") + git("add", ".") + git("commit", "--quiet", "--message", "first") + return root + + +def test_sdist_records_clean_revision(checkout: Path, git: Git) -> None: + with tarfile.open(checkout / "dist" / backend.build_sdist(str(checkout / "dist"))) as archive: + handle = archive.extractfile("example-1.0/.python-sbom-revision") + assert handle is not None + assert handle.read() == f"node: {git('rev-parse', 'HEAD')}\n".encode() + + +def test_sdist_revision_keeps_the_backend_times(checkout: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("SOURCE_DATE_EPOCH", "1700000000") + sdist = checkout / "dist" / backend.build_sdist(str(checkout / "dist")) + members = sdist_members(sdist) + assert members[".python-sbom-revision"].mtime == max(member.mtime for member in members.values()) + # the gzip header's time, bytes 4 to 8, is the one the backend took from SOURCE_DATE_EPOCH + assert int.from_bytes(sdist.read_bytes()[4:8], "little") == 1700000000 + + +def sdist_members(sdist: Path) -> dict[str, tarfile.TarInfo]: + with tarfile.open(sdist) as archive: + return {member.name.split("/", 1)[-1]: member for member in archive.getmembers()} + + +def test_sdist_of_dirty_checkout_records_no_revision(checkout: Path) -> None: + (checkout / "LICENSE").write_text("changed\n", encoding="utf-8") + sdist = checkout / "dist" / backend.build_sdist(str(checkout / "dist")) + assert ".python-sbom-revision" not in sdist_members(sdist) + + +def test_sdist_without_top_directory(checkout: Path, monkeypatch: pytest.MonkeyPatch) -> None: + def empty_sdist(sdist_directory: str, config_settings: Mapping[str, str] | None = None) -> str: # ruff: ignore[unused-function-argument] # the hook's signature + with tarfile.open(Path(sdist_directory) / "example-1.0.tar.gz", "w:gz"): + pass + return "example-1.0.tar.gz" + + monkeypatch.setattr(flit_core.buildapi, "build_sdist", empty_sdist) + with pytest.raises(SbomError, match=r"has no top-level directory to add \.python-sbom-revision to"): + backend.build_sdist(str(checkout / "dist")) + assert sorted(path.name for path in (checkout / "dist").iterdir()) == ["example-1.0.tar.gz"] + + +def test_wheel_from_sdist_names_its_commit(checkout: Path, git: Git, tmp_path: Path) -> None: + with tarfile.open(checkout / "dist" / backend.build_sdist(str(checkout / "dist"))) as archive: + archive.extractall(tmp_path / "unpacked", filter="data") + unpacked = tmp_path / "unpacked" / "example-1.0" + with pytest.MonkeyPatch.context() as patch: + patch.chdir(unpacked) + (unpacked / "dist").mkdir() + wheel = unpacked / "dist" / backend.build_wheel(str(unpacked / "dist")) + sbom = zipfile.ZipFile(wheel).read("example-1.0.dist-info/sboms/example.cdx.json") + component = json.loads(sbom)["metadata"]["component"] + assert git("rev-parse", "HEAD") in json.dumps(component) diff --git a/tests/test_pdm.py b/tests/test_pdm.py new file mode 100644 index 0000000..f999a17 --- /dev/null +++ b/tests/test_pdm.py @@ -0,0 +1,121 @@ +from __future__ import annotations + +import json +import tarfile +import zipfile +from typing import TYPE_CHECKING, Final + +import pytest +from pdm.backend import build_editable, build_sdist, build_wheel + +from python_sbom._validate._artifact import validate + +if TYPE_CHECKING: + from pathlib import Path + + from tests.conftest import Git + +_SBOM: Final[str] = "example-1.0.dist-info/sboms/example.cdx.json" + + +@pytest.fixture +def project(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: + return write_project(tmp_path / "project", "\n[tool.python-sbom]\n", monkeypatch) + + +def write_project(root: Path, tables: str, monkeypatch: pytest.MonkeyPatch) -> Path: + (root / "src" / "example").mkdir(parents=True, exist_ok=True) + (root / "src" / "example" / "__init__.py").write_text("", encoding="utf-8") + (root / "LICENSE").write_text("Copyright (c) 2026 Example\n\nMIT terms\n", encoding="utf-8") + project = """\ +[build-system] +build-backend = "pdm.backend" +requires = ["pdm-backend", "python-sbom"] + +[project] +name = "example" +version = "1.0" +description = "an example" +license = "MIT" +license-files = ["LICENSE"] +urls.Source = "https://github.com/example/example" +""" + (root / "pyproject.toml").write_text(f"{project}{tables}", encoding="utf-8") + # a frontend creates the output directory before it calls the backend + (root / "dist").mkdir(exist_ok=True) + monkeypatch.chdir(root) + return root + + +def test_wheel_carries_valid_sbom(project: Path) -> None: + assert validate(project / "dist" / build_wheel(str(project / "dist")), frozenset({"MIT"})) == [] + + +def test_reproducible_build_uses_pdm_backend_time(project: Path) -> None: + wheel = project / "dist" / build_wheel(str(project / "dist")) + assert json.loads(zipfile.ZipFile(wheel).read(_SBOM))["metadata"]["timestamp"] == "2016-01-01T00:00:00+00:00" + + +def test_project_without_table_has_no_sbom(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + root = write_project(tmp_path / "project", "", monkeypatch) + assert sboms(root / "dist" / build_wheel(str(root / "dist"))) == [] + + +def sboms(wheel: Path) -> list[str]: + return [name for name in zipfile.ZipFile(wheel).namelist() if "/sboms/" in name] + + +def test_editable_wheel_has_no_sbom(project: Path) -> None: + assert sboms(project / "dist" / build_editable(str(project / "dist"))) == [] + + +@pytest.fixture +def checkout(tmp_path: Path, git: Git, monkeypatch: pytest.MonkeyPatch) -> Path: + root = write_project(tmp_path / "checkout", "\n[tool.python-sbom]\n", monkeypatch) + (root / ".gitignore").write_text("dist/\n", encoding="utf-8") + git("add", ".") + git("commit", "--quiet", "--message", "first") + return root + + +def test_sdist_records_clean_revision(checkout: Path, git: Git) -> None: + sdist = checkout / "dist" / build_sdist(str(checkout / "dist")) + assert revision_of(sdist) == f"node: {git('rev-parse', 'HEAD')}\n".encode() + + +def revision_of(sdist: Path) -> bytes | None: + with tarfile.open(sdist) as archive: + if "example-1.0/.python-sbom-revision" not in archive.getnames(): + return None + handle = archive.extractfile("example-1.0/.python-sbom-revision") + assert handle is not None + return handle.read() + + +def test_sdist_holds_only_project_files(checkout: Path) -> None: + with tarfile.open(checkout / "dist" / build_sdist(str(checkout / "dist"))) as archive: + assert sorted(archive.getnames()) == [ + "example-1.0/.python-sbom-revision", + "example-1.0/LICENSE", + "example-1.0/PKG-INFO", + "example-1.0/pyproject.toml", + "example-1.0/src/example/__init__.py", + ] + + +def test_sdist_of_dirty_checkout_records_no_revision(checkout: Path) -> None: + (checkout / "LICENSE").write_text("changed\n", encoding="utf-8") + assert revision_of(checkout / "dist" / build_sdist(str(checkout / "dist"))) is None + + +def test_wheel_from_sdist_names_its_commit(checkout: Path, git: Git, tmp_path: Path) -> None: + with tarfile.open(checkout / "dist" / build_sdist(str(checkout / "dist"))) as archive: + archive.extractall(tmp_path / "unpacked", filter="data") + wheel = checkout / "dist" / build_wheel(str(checkout / "dist")) + unpacked = tmp_path / "unpacked" / "example-1.0" + with pytest.MonkeyPatch.context() as patch: + patch.chdir(unpacked) + from_sdist = unpacked / "dist" / build_wheel(str(unpacked / "dist")) + component = json.loads(zipfile.ZipFile(from_sdist).read(_SBOM))["metadata"]["component"] + assert git("rev-parse", "HEAD") in json.dumps(component) + assert wheel.name == from_sdist.name diff --git a/tests/test_setuptools.py b/tests/test_setuptools.py new file mode 100644 index 0000000..cf26aee --- /dev/null +++ b/tests/test_setuptools.py @@ -0,0 +1,182 @@ +from __future__ import annotations + +import json +import tarfile +import zipfile +from typing import TYPE_CHECKING, Final + +import pytest +from setuptools import build_meta +from setuptools.dist import Distribution + +from python_sbom._errors import SbomError +from python_sbom._validate._artifact import validate +from python_sbom.setuptools import finalize_distribution_options + +if TYPE_CHECKING: + from pathlib import Path + + from tests.conftest import Git + +# setuptools-scm, which the dev environment installs through hatch-vcs, warns about its file finder in every setuptools +# build +pytestmark: Final[pytest.MarkDecorator] = pytest.mark.filterwarnings( + "ignore:The setuptools.file_finders entry point is deprecated:DeprecationWarning" +) + +# the project's own wheel command, which the plugin must keep +_SETUP: Final[str] = """\ +from pathlib import Path + +from setuptools import setup +from setuptools.command.bdist_wheel import bdist_wheel + + +class Wheel(bdist_wheel): + def run(self): + Path("custom-ran").write_text("yes") + super().run() + + +setup(cmdclass={"bdist_wheel": Wheel}) +""" + + +@pytest.fixture +def project(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: + return write_project(tmp_path / "project", "\n[tool.python-sbom]\n", monkeypatch) + + +def write_project(root: Path, tables: str, monkeypatch: pytest.MonkeyPatch) -> Path: + (root / "example").mkdir(parents=True, exist_ok=True) + (root / "example" / "__init__.py").write_text("", encoding="utf-8") + (root / "LICENSE").write_text("Copyright (c) 2026 Example\n\nMIT terms\n", encoding="utf-8") + project = """\ +[build-system] +build-backend = "setuptools.build_meta" +requires = ["setuptools>=77", "python-sbom"] + +[project] +name = "example" +version = "1.0" +description = "an example" +license = "MIT" +license-files = ["LICENSE"] +urls.Source = "https://github.com/example/example" +""" + (root / "pyproject.toml").write_text(f"{project}{tables}", encoding="utf-8") + # a frontend creates the output directory before it calls the backend + (root / "dist").mkdir(exist_ok=True) + monkeypatch.chdir(root) + return root + + +def test_wheel_carries_valid_sbom(project: Path) -> None: + assert validate(project / "dist" / build_meta.build_wheel(str(project / "dist")), frozenset({"MIT"})) == [] + + +def test_project_wheel_command_is_kept(project: Path) -> None: + (project / "setup.py").write_text(_SETUP, encoding="utf-8") + wheel = project / "dist" / build_meta.build_wheel(str(project / "dist")) + assert ((project / "custom-ran").read_text(encoding="utf-8"), sboms(wheel)) == ( + "yes", + ["example-1.0.dist-info/sboms/example.cdx.json"], + ) + + +def sboms(wheel: Path) -> list[str]: + return [name for name in zipfile.ZipFile(wheel).namelist() if "/sboms/" in name] + + +def test_wheel_command_that_writes_no_wheel(project: Path) -> None: + (project / "setup.py").write_text( + "from setuptools import setup\nfrom setuptools.command.bdist_wheel import bdist_wheel\n\n\n" + "class Wheel(bdist_wheel):\n def run(self):\n pass\n\n\nsetup(cmdclass={'bdist_wheel': Wheel})\n", + encoding="utf-8", + ) + with pytest.raises(SbomError, match=r"Wheel wrote no wheel to describe"): + build_meta.build_wheel(str(project / "dist")) + + +@pytest.mark.parametrize( + ("commands", "named"), + [ + pytest.param('bdist_wheel = "custom:Wheel"', "bdist_wheel", id="wheel"), + pytest.param('sdist = "custom:Sdist"\nbdist_wheel = "custom:Wheel"', "bdist_wheel, sdist", id="both"), + ], +) +def test_pyproject_replacing_commands( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, commands: str, named: str +) -> None: + write_project( + tmp_path / "project", f"\n[tool.python-sbom]\n\n[tool.setuptools.cmdclass]\n{commands}\n", monkeypatch + ) + with pytest.raises(SbomError, match=rf"^\[tool.setuptools.cmdclass\] replaces {named} after python-sbom wraps it"): + finalize_distribution_options(Distribution()) + + +def test_project_without_table_has_no_sbom(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + root = write_project(tmp_path / "project", "", monkeypatch) + assert sboms(root / "dist" / build_meta.build_wheel(str(root / "dist"))) == [] + + +# setuptools tells about every editable build with a warning, which the suite turns into an error +@pytest.mark.filterwarnings("ignore::setuptools.warnings.InformationOnly") +def test_editable_wheel_has_no_sbom(project: Path) -> None: + assert sboms(project / "dist" / build_meta.build_editable(str(project / "dist"))) == [] + + +def test_project_without_pyproject_is_left_alone(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.chdir(tmp_path) + distribution = Distribution() + commands = dict(distribution.cmdclass) + finalize_distribution_options(distribution) + assert distribution.cmdclass == commands + + +def test_invalid_pyproject(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + (tmp_path / "pyproject.toml").write_text("[tool.python-sbom\n", encoding="utf-8") + monkeypatch.chdir(tmp_path) + with pytest.raises(SbomError, match=r"pyproject.toml is not valid TOML"): + finalize_distribution_options(Distribution()) + + +@pytest.fixture +def checkout(tmp_path: Path, git: Git, monkeypatch: pytest.MonkeyPatch) -> Path: + root = write_project(tmp_path / "checkout", "\n[tool.python-sbom]\n", monkeypatch) + (root / ".gitignore").write_text("dist/\nbuild/\n*.egg-info/\n", encoding="utf-8") + git("add", ".") + git("commit", "--quiet", "--message", "first") + return root + + +def test_sdist_records_clean_revision(checkout: Path, git: Git) -> None: + sdist = checkout / "dist" / build_meta.build_sdist(str(checkout / "dist")) + assert revision_of(sdist) == f"node: {git('rev-parse', 'HEAD')}\n".encode() + + +def revision_of(sdist: Path) -> bytes | None: + with tarfile.open(sdist) as archive: + if "example-1.0/.python-sbom-revision" not in archive.getnames(): + return None + handle = archive.extractfile("example-1.0/.python-sbom-revision") + assert handle is not None + return handle.read() + + +def test_sdist_of_dirty_checkout_records_no_revision(checkout: Path) -> None: + (checkout / "LICENSE").write_text("changed\n", encoding="utf-8") + assert revision_of(checkout / "dist" / build_meta.build_sdist(str(checkout / "dist"))) is None + + +def test_wheel_from_sdist_names_its_commit(checkout: Path, git: Git, tmp_path: Path) -> None: + with tarfile.open(checkout / "dist" / build_meta.build_sdist(str(checkout / "dist"))) as archive: + archive.extractall(tmp_path / "unpacked", filter="data") + unpacked = tmp_path / "unpacked" / "example-1.0" + with pytest.MonkeyPatch.context() as patch: + patch.chdir(unpacked) + (unpacked / "dist").mkdir() + wheel = unpacked / "dist" / build_meta.build_wheel(str(unpacked / "dist")) + sbom = zipfile.ZipFile(wheel).read("example-1.0.dist-info/sboms/example.cdx.json") + component = json.loads(sbom)["metadata"]["component"] + assert git("rev-parse", "HEAD") in json.dumps(component) diff --git a/tox.toml b/tox.toml index d558966..cbe54a9 100644 --- a/tox.toml +++ b/tox.toml @@ -45,8 +45,8 @@ commands = [ # RE2 lacks Python 3.15 wheels: https://github.com/google/re2/issues/647 """\ import glob; import pathlib; import subprocess; subprocess.check_call(['uv', 'tool', 'run', '--python', '3.14', \ - '--from', 'proselint>=0.16', 'proselint', 'check', *(path for path in glob.glob('docs/*.rst') if \ - pathlib.Path(path).name != 'changelog.rst')])\ + '--from', 'proselint>=0.16', 'proselint', 'check', *(path for path in glob.glob('docs/**/*.rst', recursive=True) if \ + pathlib.Path(path).name not in {'changelog.rst', 'index.rst'} and 'changelog.d' not in pathlib.Path(path).parts)])\ """, ], [