rpm: Install BuildRequires from the spec

README.md

@@ -207,6 +207,7 @@ User guides:
   itself embeds in the executables
 - [Fetching project sources](docs/user/git.md): pinning an external project, or use a local checkout
   for development
+- [Building RPM packages](docs/user/rpm.md): building a package from a spec, its build dependencies
 - [Maintaining packages](docs/user/importer.md): importing and updating packages from upstream
   distributions, local modifications, branch curation
 - [Development boxes](docs/user/box.md): pinned interactive development environments

docs/design/architecture.md

@@ -604,7 +604,9 @@ Native package installation has three phases shared by buildroots and images:
 
 1. **Plan.** Run `PackageSystemInfo.plan` against each configured repository's materialized directory,
    effective priority, and solver cache. Weak dependencies are disabled. Existing lower layers are mounted
-   read-only so installed packages can satisfy an incremental request.
+   read-only so installed packages can satisfy an incremental request. A request lists its packages in the
+   analysis, or in JSON files that an earlier action writes, such as the BuildRequires query of an rpm
+   package. A request with no packages installs nothing and leaves its layer empty.
 2. **Select.** Use the resulting transaction to select package files from repository pools, verified after
    download if the repository declares signing keys, named with the package system's declared suffix so its
    installer finds them. A local repository is materialized by
@@ -713,8 +715,12 @@ and runtime package closures are still delegated to libdnf5 at buildroot-plan ti
 An `rpm_package` action:
 
 1. obtains the shared base root from its `BuildrootInfo`;
-2. resolves and installs its BuildRequires delta, preferring RPMs from `buildroot_packages` over upstream;
-3. overlays the base and delta, stages its spec/sources in Buck action scratch, and runs `rpmbuild -ba`,
+2. resolves and installs its BuildRequires delta, preferring RPMs from `buildroot_packages` over upstream. A
+   hand-written `rpm.package()` also installs the BuildRequires of the spec that it builds. Analysis cannot
+   read the contents of a spec, so an action stages the spec as the build does and runs `rpmbuild -bd` with
+   the build's options in the base root and the delta. It writes the BuildRequires that are missing, and a
+   second delta installs them. That delta is empty when the spec needs nothing more;
+3. overlays the base and the deltas, stages its spec/sources in Buck action scratch, and runs `rpmbuild -ba`,
    or `rpmbuild -bb --build-in-place` on a volatile overlay of a populated source tree. overlayfs shows the
    directory under a mount rather than the mounted tree, so each directory mount below the source tree,
    such as a `tine mount`, gets a volatile overlay of its own. The driver discovers directory mounts with
@@ -739,15 +745,21 @@ names none, and gets no per-release aliases either.
 #### Limitations
 
 - The imported self-host dependency graph is inferred from stored BuildRequires/Provides/file metadata; it
-  does not run RPM's dynamic BuildRequires protocol.
+  does not run RPM's dynamic BuildRequires protocol. The BuildRequires query runs `%generate_buildrequires`
+  only when the base root and the declared delta contain every static BuildRequires, and runs it once.
+- An imported package installs only its recorded BuildRequires, also for a spec from a checkout or a pin.
+  `rpm.branch()` derives `buildroot_packages` from them, and Buck cannot add a target dependency that an
+  action output discovers. A queried BuildRequires on a package of the branch would install it from
+  upstream.
 - Build cycles fall back to upstream RPMs for ordinary intra-SCC edges, so the package set is not a fully
   self-hosted fixed point.
 - Builds use `--nocheck`; package test policy is not implemented.
 - Debuginfo/debugsource outputs are not first-class declared sub-targets.
 - The bootstrap extractor supports the pinned v4/newc payload form, not RPM v6 metadata.
 
-Entry points: `package_system/rpm/{rules,catalog,generated}.bzl` and
-`package_system/rpm/{snapshot,plan,install,pkgdb,createrepo,build,extract,rpmfile}.py`.
+Entry points: `package_system/rpm/{rules,catalog,generated}.bzl`,
+`package_system/rpm/{snapshot,plan,install,pkgdb,createrepo,build,extract,rpmfile}.py` and
+`package_system/rpm/{rpmbuild,buildrequires}.py`.
 
 ### The alpm package system
 
@@ -1583,8 +1595,10 @@ future remote workers. Bazel's broader language-rule ecosystem mattered less tha
 for a package-heavy repository.
 
 Buck cannot add ordinary target dependencies discovered from an action output. Dynamic actions may select
-among declared inputs but cannot turn newly discovered BuildRequires into a new static graph. Therefore
-dependency discovery/import must produce committed or analysis-time lock data before normal builds.
+among declared inputs but cannot turn newly discovered BuildRequires into a new static graph. An action can
+still install BuildRequires that an earlier action discovers, as the BuildRequires query of a hand-written
+`rpm.package()` does. A BuildRequires on a package that the project builds needs committed or analysis-time
+lock data before normal builds.
 
 ### Commit repository snapshots and selectively freeze boxes
 

docs/user/importer.md

@@ -202,8 +202,8 @@ Buck ignores, and `%install` can copy them into the RPMs. See
 
 Three per-package maps in `_properties.json` can adapt the build to that checkout. `in_place_specs` selects
 an RPM spec by a path relative to the checkout root. The spec's directory also supplies its Source/Patch
-files and relative includes. `in_place_packaging` takes the spec and its Source/Patch files from a dist-git
-commit that the checkout pins; see [Taking the packaging from a pin in the
+files. rpm opens a relative `%include` from the checkout root. `in_place_packaging` takes the spec and its
+Source/Patch files from a dist-git commit that the checkout pins; see [Taking the packaging from a pin in the
 checkout](#taking-the-packaging-from-a-pin-in-the-checkout). A package cannot set both `in_place_specs` and
 `in_place_packaging`. `in_place_rpmbuild_options` adds CLI options after the common `rpmbuild_options` and
 any dev-mode defaults.

docs/user/rpm.md

@@ -0,0 +1,51 @@
+<!--
+SPDX-FileCopyrightText: Amutable GmbH <https://amutable.com/>
+SPDX-License-Identifier: MPL-2.0
+-->
+
+# Building RPM packages
+
+`rpm.package()` builds RPM packages from a spec and its Source and Patch files. The importer declares the
+packages that it brings in from a distribution; see [Maintaining packages](importer.md).
+
+## Building a package from a spec
+
+Put the spec and the files that it names in the directory of the `BUCK` file or below it, and declare the
+package:
+
+```starlark
+load("@tine//package_system/rpm:defs.bzl", "rpm")
+
+rpm.package(
+    name = "hello",
+    package = "hello",
+    spec = "hello/hello.spec",
+    srcs = ["hello/hello-2.12.3.tar.gz"],
+    buildroot = "catalog//:fedora.rawhide.buildroot",
+    release = "1",
+    source_date_epoch = 1754006400,
+    subpackages = ["hello"],
+)
+```
+
+`package` is the `Name:` of the spec, and `subpackages` lists the binary packages that the build produces.
+Each subpackage is a sub-target, such as `:hello[hello]`.
+
+## Build dependencies
+
+The build installs the BuildRequires of the spec over the shared base buildroot. `build_requires` lists
+packages that the build installs before it reads the spec, such as tools or RPM macros that rpm needs to
+parse the spec. Before the build, `rpmbuild -bd` checks the spec in that buildroot with the build's rpmbuild
+options, so the spec's conditionals select the same BuildRequires as in the build. The build then installs
+the BuildRequires that the buildroot lacks.
+
+`rpmbuild -bd` runs `%generate_buildrequires` only when the buildroot contains every static BuildRequires
+of the spec, and runs it once. For a spec with the script, list its static BuildRequires in
+`build_requires`. If the script reports more packages once its first ones are installed, list those as
+well. Otherwise the build fails with rpm's `Failed build dependencies:`.
+
+Neither the check nor the build looks for the Source and Patch files of an in-place spec. The build only
+fails if the spec reads a missing file, for example a Source that `%install` copies.
+
+A package of an imported branch skips the check. It installs the BuildRequires that the importer recorded
+in `build_requires`.

examples/package/rpm/BUCK

@@ -19,6 +19,29 @@ _CHECK_BOX = "catalog//:fedora.rawhide.box"
 
 git.checkout(name = "hello.source")
 
+# rpm.branch() derives buildroot_packages from the BuildRequires of every architecture. A package can then
+# have buildroot_packages and no build_requires on the architecture that it builds for.
+rpm.package(
+    name = "hello-buildroot-packages",
+    distro = _DISTRO,
+    package = "hello",
+    spec = "hello/hello.spec",
+    buildroot = _BUILDROOT,
+    buildroot_packages = [":hello-archive"],
+    release = "1",
+    source_date_epoch = 1754006400,
+    subpackages = ["hello"],
+)
+
+box.sh_test(
+    name = "hello-buildroot-packages-test",
+    distro = _DISTRO,
+    labels = ["image"],
+    box = _CHECK_BOX,
+    script = 'python3 "$@"',
+    args = ["$(source hello_archive_check.py)", "$(location :hello-buildroot-packages)"],
+)
+
 # The archive path runs %prep and emits both binary and source RPMs. It exists separately from the
 # source-tree fixture to check release and dev compression through real RPM headers.
 rpm.package(
@@ -41,14 +64,35 @@ box.sh_test(
     args = ["$(source hello_archive_check.py)", "$(location :hello-archive)"],
 )
 
+rpm.package(
+    name = "hello-parse-helper",
+    distro = _DISTRO,
+    package = "hello",
+    spec = "hello/hello.spec",
+    build_requires = ["jq"],
+    buildroot = _BUILDROOT,
+    release = "1",
+    rpmbuild_options = ["--define=version_override %(jq -nr '\"1\"')"],
+    source_date_epoch = 1754006400,
+    subpackages = ["hello"],
+)
+
+box.sh_test(
+    name = "hello-parse-helper-test",
+    distro = _DISTRO,
+    labels = ["image"],
+    box = _CHECK_BOX,
+    script = 'python3 "$@"',
+    args = ["$(source hello_archive_check.py)", "$(location :hello-parse-helper)"],
+)
+
 # The explicit override keeps this source-tree fixture in dev mode without depending on the
 # user's root config.
 rpm.package(
     name = "hello-dev",
     distro = _DISTRO,
     package = "hello",
     buildroot = _BUILDROOT,
-    build_requires = ["gcc"],
     dev = True,
     in_place_spec = "rpm/hello.spec",
     release = "1",
@@ -77,7 +121,6 @@ rpm.package(
     distro = _DISTRO,
     package = "hello",
     buildroot = _BUILDROOT,
-    build_requires = ["gcc"],
     in_place_spec = "rpm/hello.spec",
     release = "1",
     source_date_epoch = 1754006400,
@@ -105,7 +148,6 @@ rpm.package(
     distro = _DISTRO,
     package = "hello",
     buildroot = _BUILDROOT,
-    build_requires = ["gcc", "meson"],
     in_place_spec = "rpm/hello-meson.spec",
     release = "1",
     source_date_epoch = 1754006400,
@@ -131,7 +173,6 @@ rpm.package(
     distro = _DISTRO,
     package = "hello",
     buildroot = _BUILDROOT,
-    build_requires = ["gcc"],
     in_place_packaging = "rpm/packaging.json",
     in_place_packaging_distro = "fedora",
     release = "1",

examples/package/rpm/hello.source/rpm/hello.spec

@@ -7,6 +7,10 @@ Release: %autorelease
 Summary: Exercise dev-mode source-tree RPM package builds
 License: MPL-2.0
 BuildRequires: gcc
+# The base buildroot doesn't contain jq, and the BUCK file doesn't list it.
+%if "%(echo jq > generated-buildrequires && cat generated-buildrequires)" == "jq"
+BuildRequires: jq
+%endif
 Source1: packaging-marker
 
 %description
@@ -25,10 +29,12 @@ annobin=enabled
 printf '%s\n' "$lto" > %{_vpath_builddir}/lto
 printf '%s\n' "$annobin" > %{_vpath_builddir}/annobin
 %{__cc} %{build_cflags} %{build_ldflags} -o %{_vpath_builddir}/hello hello.c
+jq -n '"checkout"' > %{_vpath_builddir}/jq
 
 %install
 install -Dpm0755 %{_vpath_builddir}/hello %{buildroot}%{_bindir}/tine-package-hello
 install -Dpm0644 %{SOURCE1} %{buildroot}%{_datadir}/tine-package/spec-from-checkout
+install -Dpm0644 %{_vpath_builddir}/jq %{buildroot}%{_datadir}/tine-package/built-with-jq
 install -d %{buildroot}%{_datadir}/tine-package
 lto="$(cat %{_vpath_builddir}/lto)"
 annobin="$(cat %{_vpath_builddir}/annobin)"

examples/package/rpm/hello.source/rpm/packaging.json

@@ -2,6 +2,6 @@
   "fedora": {
     "url": "examples/package/rpm/hello-packaging.bundle",
     "branch": "rawhide",
-    "commit": "fcf600f407123dde5ec79f27e7d8526c9a4bae03"
+    "commit": "ff16cc891d9bb222407e3edc73566a2eff00a570"
   }
 }

examples/package/rpm/hello/hello.spec

@@ -2,22 +2,29 @@
 # SPDX-License-Identifier: MPL-2.0
 
 Name: hello
-Version: 1
+Version: %{?version_override}%{!?version_override:1}
 Release: %autorelease
 Summary: Exercise regular RPM package builds
 License: MPL-2.0
 BuildArch: noarch
+# The base buildroot doesn't contain jq.
+%if "%{getenv:HOME}" == "/build" && "%{getenv:SOURCE_DATE_EPOCH}" == "1754006400"
+BuildRequires: jq
+%endif
 
 %description
 An integration fixture for tine's regular RPM package build path.
 
 %prep
 
 %build
+jq -n '"archive"' > jq
 
 %install
 install -d %{buildroot}%{_datadir}/tine-package
 touch %{buildroot}%{_datadir}/tine-package/archive
+install -pm0644 jq %{buildroot}%{_datadir}/tine-package/built-with-jq
 
 %files
 %{_datadir}/tine-package/archive
+%{_datadir}/tine-package/built-with-jq

examples/package/rpm/hello_archive_check.py

@@ -16,4 +16,9 @@
     query = ["rpm", "-qp", "--queryformat", "%{PAYLOADCOMPRESSOR}", rpm]
     compressor = subprocess.run(query, check=True, stdout=subprocess.PIPE, text=True).stdout
     assert compressor in {"gzip", "bzip2", "xz", "lzma", "zstd"}, (rpm, compressor)
+[binary] = [rpm for rpm in rpms if rpm not in sources]
+files = subprocess.run(
+    ["rpm", "-qpl", binary], check=True, stdout=subprocess.PIPE, text=True
+).stdout.splitlines()
+assert "/usr/share/tine-package/built-with-jq" in files, files
 print(f"archive: {len(rpms)} packages with compressed payloads")

examples/package/rpm/hello_dev_check.py

@@ -26,6 +26,8 @@ def query(package: Path, *args: str) -> str:
 files = set(query(package, "-l").splitlines())
 markers = "/usr/share/tine-package"
 assert f"{markers}/spec-from-checkout" in files, files
+# The build installs jq only because the spec lists it in BuildRequires, and %build runs jq.
+assert f"{markers}/built-with-jq" in files, files
 assert f"{markers}/profile-lto-disabled" in files, files
 assert f"{markers}/profile-annobin-disabled" in files, files
 assert not list(rpms.glob("hello-debuginfo-*.rpm")), sorted(rpms.iterdir())

examples/package/rpm/hello_pinned_check.py

@@ -25,5 +25,7 @@ def query(package: Path, *args: str) -> str:
 files = set(query(package, "-l").splitlines())
 assert "/usr/share/tine-package/spec-from-pin" in files, files
 assert "/usr/share/tine-package/spec-from-checkout" not in files, files
+# The build installs jq only because the pinned spec lists it in BuildRequires, and %build runs jq.
+assert "/usr/share/tine-package/built-with-jq" in files, files
 assert query(subpackage, "--queryformat", "%{NAME}") == "hello"
 print(f"hello: pinned packaging, {len(files)} files")

examples/package/rpm/hello_tree_check.py

@@ -26,6 +26,8 @@ def query(package: Path, *args: str) -> str:
 markers = "/usr/share/tine-package"
 # The spec comes out of the checkout as in a dev build, but none of the dev-only savings apply.
 assert f"{markers}/spec-from-checkout" in files, files
+# The build installs jq only because the spec lists it in BuildRequires, and %build runs jq.
+assert f"{markers}/built-with-jq" in files, files
 assert f"{markers}/profile-lto-enabled" in files, files
 assert f"{markers}/profile-annobin-enabled" in files, files
 assert list(rpms.glob("hello-debuginfo-*.rpm")), sorted(rpms.iterdir())

package/build.bzl

@@ -7,7 +7,7 @@ load("//:specs.bzl", "specs")
 load("//box:defs.bzl", "box")
 load("//git:defs.bzl", "git")
 load("//project:defs.bzl", "project")
-load(":buildroot.bzl", "BuildrootInfo", "package_buildroot")
+load(":buildroot.bzl", "BuildrootInfo")
 load(":manager.bzl", "PackageManagerInfo")
 load(":repository.bzl", "LocalPackageInfo")
 load(":system.bzl", "PackageSystemInfo")
@@ -27,7 +27,7 @@ PACKAGE_BUILD_ATTRS = {
     "buildroot_packages": attrs.list(
         attrs.dep(providers = [LocalPackageInfo]),
         default = [],
-        doc = "our packages that the solve of build_requires prefers over upstream ones",
+        doc = "our packages that the solves of the BuildRequires prefer over upstream ones",
     ),
     "configured_dev": attrs.bool(
         doc = "whether project configuration selects this package for dev mode",
@@ -66,16 +66,15 @@ def package_build(
     owner: str,
     package_system: Dependency,
     base: BuildrootInfo,
-    build_requires: list[str],
-    buildroot_packages: list[Dependency],
+    stack: list[Artifact],
     configured_dev: bool,
     keep_build_dir: bool,
     packages: str,
     subpackages: list[str],
     category: str,
     spec: dict[str, typing.Any],
 ) -> list[Provider]:
-    """Install `build_requires` over `base`, and run the `build` driver of `package_system` on the result.
+    """Run the `build` driver of `package_system` on the layer stack `stack` that `base` starts.
 
     `spec` contains the fields of the driver spec that the calling rule owns. `packages` names the output
     directory of the packages. A dev build keeps its build directory between runs if `keep_build_dir` is
@@ -95,7 +94,6 @@ def package_build(
     reserved = [name for name in ("build", "buildroot") if name in subpackages]
     if reserved:
         fail("{}: subpackages {} have the names of built-in sub-targets".format(owner, reserved))
-    stack = package_buildroot(ctx, owner, base, build_requires, buildroot_packages)
 
     out = project.kept_dir(ctx.actions, packages)
     build_dir = project.kept_dir(ctx.actions, _PRIVATE + "/build") if configured_dev and keep_build_dir else None
@@ -118,7 +116,6 @@ def package_build(
                 spec
                 | {
                     "build_dir": build_dir.as_output() if build_dir != None else None,
-                    # The base root is the lowest layer, and the layer of build_requires is above it.
                     "lower": stack,
                     "out": out.as_output(),
                     "subpackages": subpackages,

package/builder.py

@@ -76,8 +76,8 @@ def run(spec: Spec, build: Callable[[Path, Path, Path], int]) -> int:
     return rc
 
 
-def buildroot(spec: Spec, topdir: Path) -> AbstractContextManager[Path]:
-    """Chroot into the buildroot of `spec`, with topdir at `CHROOT_TOPDIR`."""
+def buildroot(lower: list[str], topdir: Path) -> AbstractContextManager[Path]:
+    """Chroot into the buildroot with the layers `lower`, with topdir at `CHROOT_TOPDIR`."""
     return rootfs.rootfs(
-        "/buildroot", lowers=spec["lower"], binds=[(topdir, CHROOT_TOPDIR)], apivfs=True, chroot=True
+        "/buildroot", lowers=lower, binds=[(topdir, CHROOT_TOPDIR)], apivfs=True, chroot=True
     )