rpm: Install BuildRequires from the spec

README.md

@@ -207,6 +207,7 @@ User guides:
   itself embeds in the executables
 - [Fetching project sources](docs/user/git.md): pinning an external project, or use a local checkout
   for development
+- [Building RPM packages](docs/user/rpm.md): building a package from a spec, its build dependencies
 - [Maintaining packages](docs/user/importer.md): importing and updating packages from upstream
   distributions, local modifications, branch curation
 - [Development boxes](docs/user/box.md): pinned interactive development environments

docs/design/architecture.md

@@ -713,7 +713,12 @@ and runtime package closures are still delegated to libdnf5 at buildroot-plan ti
 An `rpm_package` action:
 
 1. obtains the shared base root from its `BuildrootInfo`;
-2. resolves and installs its BuildRequires delta, preferring RPMs from `buildroot_packages` over upstream;
+2. resolves and installs its BuildRequires delta, preferring RPMs from `buildroot_packages` over upstream. A
+2. resolves and installs its BuildRequires delta, preferring RPMs from `buildroot_packages` over upstream. A
+   hand-written `rpm.package()` also installs the BuildRequires of the spec that it builds. Analysis cannot
+   read the contents of a spec, so an action stages the spec as the build does and runs `rpmbuild -br` with
+   the build's options in the base root and the delta. A second delta installs the BuildRequires that it
+   writes. That delta is empty when the spec needs nothing more;
 3. overlays the base and delta, stages its spec/sources in Buck action scratch, and runs `rpmbuild -ba`,
    or `rpmbuild -bb --build-in-place` on a volatile overlay of a populated source tree. overlayfs shows the
    directory under a mount rather than the mounted tree, so each directory mount below the source tree,
@@ -739,15 +744,21 @@ names none, and gets no per-release aliases either.
 #### Limitations
 
 - The imported self-host dependency graph is inferred from stored BuildRequires/Provides/file metadata; it
-  does not run RPM's dynamic BuildRequires protocol.
+  does not run RPM's dynamic BuildRequires protocol. The BuildRequires query runs `%generate_buildrequires`
+  once. It misses what the script only reports after its own BuildRequires are installed.
+- An imported package installs only its recorded BuildRequires, also for a spec from a checkout or a pin.
+  `rpm.branch()` derives `buildroot_packages` from them, and Buck cannot add a target dependency that an
+  action output discovers. A queried BuildRequires on a package of the branch would install it from
+  upstream.
 - Build cycles fall back to upstream RPMs for ordinary intra-SCC edges, so the package set is not a fully
   self-hosted fixed point.
 - Builds use `--nocheck`; package test policy is not implemented.
 - Debuginfo/debugsource outputs are not first-class declared sub-targets.
 - The bootstrap extractor supports the pinned v4/newc payload form, not RPM v6 metadata.
 
-Entry points: `package_system/rpm/{rules,catalog,generated}.bzl` and
-`package_system/rpm/{snapshot,plan,install,pkgdb,createrepo,build,extract,rpmfile}.py`.
+Entry points: `package_system/rpm/{rules,catalog,generated}.bzl`,
+`package_system/rpm/{snapshot,plan,install,pkgdb,createrepo,build,extract,rpmfile}.py` and
+`package_system/rpm/{rpmbuild,buildrequires}.py`.
 
 ### The alpm package system
 

docs/user/importer.md

@@ -140,7 +140,8 @@ introduce a proper tool verb for that cleanup, but that can wait until the need
 
 A `packages/`_distro_`/`_branch_`/` directory may carry a `_properties.json` alongside the per-package
 `srcpkg.json` files. It holds hand-authored, branch-level build curation. The generated branch `BUCK`
-loads it and passes its properties to `rpm.branch()`. Never edit the `@generated` `BUCK` by hand.
+loads it and passes every property to `rpm.branch()`, so loading the branch fails on a property that
+`rpm.branch()` doesn't know. Never edit the `@generated` `BUCK` by hand.
 
 ### Buildroot-only packages
 
@@ -202,8 +203,8 @@ Buck ignores, and `%install` can copy them into the RPMs. See
 
 Three per-package maps in `_properties.json` can adapt the build to that checkout. `in_place_specs` selects
 an RPM spec by a path relative to the checkout root. The spec's directory also supplies its Source/Patch
-files and relative includes. `in_place_packaging` takes the spec and its Source/Patch files from a dist-git
-commit that the checkout pins; see [Taking the packaging from a pin in the
+files. rpm opens a relative `%include` from the checkout root. `in_place_packaging` takes the spec and its
+Source/Patch files from a dist-git commit that the checkout pins; see [Taking the packaging from a pin in the
 checkout](#taking-the-packaging-from-a-pin-in-the-checkout). A package cannot set both `in_place_specs` and
 `in_place_packaging`. `in_place_rpmbuild_options` adds CLI options after the common `rpmbuild_options` and
 any dev-mode defaults.
@@ -278,6 +279,6 @@ The buildroot still contains the imported BuildRequires. If the pinned spec requ
 imported spec does not, rpmbuild fails with `Failed build dependencies:` until the package is imported again
 at a commit with that BuildRequires.
 
-The branch's generated `BUCK` passes `in_place_packaging` to the build only if its `rpm.branch()` call has
-an `in_place_packaging` line and a `distro` line. `import`, and an `update` that takes new commits,
-regenerate the `BUCK` with both lines.
+The branch's generated `BUCK` must pass the branch's `distro` and the properties from `_properties.json`
+to `rpm.branch()`. If your `BUCK` doesn't forward `in_place_packaging`, run `import`, or an `update` that
+takes new commits, to regenerate it.

docs/user/rpm.md

@@ -0,0 +1,42 @@
+<!--
+SPDX-FileCopyrightText: Amutable GmbH <https://amutable.com/>
+SPDX-License-Identifier: MPL-2.0
+-->
+
+# Building RPM packages
+
+`rpm.package()` builds RPM packages from a spec and its Source and Patch files. The importer declares the
+packages that it brings in from a distribution; see [Maintaining packages](importer.md).
+
+## Building a package from a spec
+
+Put the spec and the files that it names in the directory of the `BUCK` file or below it, and declare the
+package:
+
+```starlark
+load("@tine//package_system/rpm:defs.bzl", "rpm")
+
+rpm.package(
+    name = "hello",
+    package = "hello",
+    spec = "hello/hello.spec",
+    srcs = ["hello/hello-2.12.3.tar.gz"],
+    buildroot = "catalog//:fedora.rawhide.buildroot",
+    release = "1",
+    source_date_epoch = 1754006400,
+    subpackages = ["hello"],
+)
+```
+
+`package` is the `Name:` of the spec, and `subpackages` lists the binary packages that the build produces.
+Each subpackage is a sub-target, such as `:hello[hello]`.
+
+## Build dependencies
+
+The build installs the BuildRequires of the spec over the shared base buildroot. `build_requires` lists
+packages that the build installs before it reads the spec, such as tools or RPM macros that the spec uses
+while parsing or in `%prep`. Before the build, `rpmbuild -br` reads the spec in that buildroot with the
+build's rpmbuild options, so the spec's conditionals select the same BuildRequires as in the build.
+
+`rpmbuild -br` runs `%generate_buildrequires` once. If the script needs packages of its own before it
+reports everything, list them in `build_requires`.

examples/package/rpm/BUCK

@@ -41,14 +41,35 @@ box.sh_test(
     args = ["$(source hello_archive_check.py)", "$(location :hello-archive)"],
 )
 
+rpm.package(
+    name = "hello-parse-helper",
+    distro = _DISTRO,
+    package = "hello",
+    spec = "hello/hello.spec",
+    build_requires = ["jq"],
+    buildroot = _BUILDROOT,
+    release = "1",
+    rpmbuild_options = ["--define=version_override %(jq -nr '\"1\"')"],
+    source_date_epoch = 1754006400,
+    subpackages = ["hello"],
+)
+
+box.sh_test(
+    name = "hello-parse-helper-test",
+    distro = _DISTRO,
+    labels = ["image"],
+    box = _CHECK_BOX,
+    script = 'python3 "$@"',
+    args = ["$(source hello_archive_check.py)", "$(location :hello-parse-helper)"],
+)
+
 # The explicit override keeps this source-tree fixture in dev mode without depending on the
 # user's root config.
 rpm.package(
     name = "hello-dev",
     distro = _DISTRO,
     package = "hello",
     buildroot = _BUILDROOT,
-    build_requires = ["gcc"],
     dev = True,
     in_place_spec = "rpm/hello.spec",
     release = "1",
@@ -77,7 +98,6 @@ rpm.package(
     distro = _DISTRO,
     package = "hello",
     buildroot = _BUILDROOT,
-    build_requires = ["gcc"],
     in_place_spec = "rpm/hello.spec",
     release = "1",
     source_date_epoch = 1754006400,
@@ -105,7 +125,6 @@ rpm.package(
     distro = _DISTRO,
     package = "hello",
     buildroot = _BUILDROOT,
-    build_requires = ["gcc", "meson"],
     in_place_spec = "rpm/hello-meson.spec",
     release = "1",
     source_date_epoch = 1754006400,
@@ -131,7 +150,6 @@ rpm.package(
     distro = _DISTRO,
     package = "hello",
     buildroot = _BUILDROOT,
-    build_requires = ["gcc"],
     in_place_packaging = "rpm/packaging.json",
     in_place_packaging_distro = "fedora",
     release = "1",

examples/package/rpm/hello.source/rpm/hello.spec

@@ -7,6 +7,10 @@ Release: %autorelease
 Summary: Exercise dev-mode source-tree RPM package builds
 License: MPL-2.0
 BuildRequires: gcc
+# The base buildroot doesn't contain jq, and the BUCK file doesn't list it.
+%if "%(echo jq > generated-buildrequires && cat generated-buildrequires)" == "jq"
+BuildRequires: jq
+%endif
 Source1: packaging-marker
 
 %description
@@ -25,10 +29,12 @@ annobin=enabled
 printf '%s\n' "$lto" > %{_vpath_builddir}/lto
 printf '%s\n' "$annobin" > %{_vpath_builddir}/annobin
 %{__cc} %{build_cflags} %{build_ldflags} -o %{_vpath_builddir}/hello hello.c
+jq -n '"checkout"' > %{_vpath_builddir}/jq
 
 %install
 install -Dpm0755 %{_vpath_builddir}/hello %{buildroot}%{_bindir}/tine-package-hello
 install -Dpm0644 %{SOURCE1} %{buildroot}%{_datadir}/tine-package/spec-from-checkout
+install -Dpm0644 %{_vpath_builddir}/jq %{buildroot}%{_datadir}/tine-package/built-with-jq
 install -d %{buildroot}%{_datadir}/tine-package
 lto="$(cat %{_vpath_builddir}/lto)"
 annobin="$(cat %{_vpath_builddir}/annobin)"

examples/package/rpm/hello.source/rpm/packaging.json

@@ -2,6 +2,6 @@
   "fedora": {
     "url": "examples/package/rpm/hello-packaging.bundle",
     "branch": "rawhide",
-    "commit": "fcf600f407123dde5ec79f27e7d8526c9a4bae03"
+    "commit": "ff16cc891d9bb222407e3edc73566a2eff00a570"
   }
 }

examples/package/rpm/hello/hello.spec

@@ -2,22 +2,29 @@
 # SPDX-License-Identifier: MPL-2.0
 
 Name: hello
-Version: 1
+Version: %{?version_override}%{!?version_override:1}
 Release: %autorelease
 Summary: Exercise regular RPM package builds
 License: MPL-2.0
 BuildArch: noarch
+# The base buildroot doesn't contain jq.
+%if "%{getenv:HOME}" == "/build" && "%{getenv:SOURCE_DATE_EPOCH}" == "1754006400"
+BuildRequires: jq
+%endif
 
 %description
 An integration fixture for tine's regular RPM package build path.
 
 %prep
 
 %build
+jq -n '"archive"' > jq
 
 %install
 install -d %{buildroot}%{_datadir}/tine-package
 touch %{buildroot}%{_datadir}/tine-package/archive
+install -pm0644 jq %{buildroot}%{_datadir}/tine-package/built-with-jq
 
 %files
 %{_datadir}/tine-package/archive
+%{_datadir}/tine-package/built-with-jq

examples/package/rpm/hello_archive_check.py

@@ -16,4 +16,9 @@
     query = ["rpm", "-qp", "--queryformat", "%{PAYLOADCOMPRESSOR}", rpm]
     compressor = subprocess.run(query, check=True, stdout=subprocess.PIPE, text=True).stdout
     assert compressor in {"gzip", "bzip2", "xz", "lzma", "zstd"}, (rpm, compressor)
+[binary] = [rpm for rpm in rpms if rpm not in sources]
+files = subprocess.run(
+    ["rpm", "-qpl", binary], check=True, stdout=subprocess.PIPE, text=True
+).stdout.splitlines()
+assert "/usr/share/tine-package/built-with-jq" in files, files
 print(f"archive: {len(rpms)} packages with compressed payloads")

examples/package/rpm/hello_dev_check.py

@@ -26,6 +26,8 @@ def query(package: Path, *args: str) -> str:
 files = set(query(package, "-l").splitlines())
 markers = "/usr/share/tine-package"
 assert f"{markers}/spec-from-checkout" in files, files
+# The build installs jq only because the spec lists it in BuildRequires, and %build runs jq.
+assert f"{markers}/built-with-jq" in files, files
 assert f"{markers}/profile-lto-disabled" in files, files
 assert f"{markers}/profile-annobin-disabled" in files, files
 assert not list(rpms.glob("hello-debuginfo-*.rpm")), sorted(rpms.iterdir())

examples/package/rpm/hello_pinned_check.py

@@ -25,5 +25,7 @@ def query(package: Path, *args: str) -> str:
 files = set(query(package, "-l").splitlines())
 assert "/usr/share/tine-package/spec-from-pin" in files, files
 assert "/usr/share/tine-package/spec-from-checkout" not in files, files
+# The build installs jq only because the pinned spec lists it in BuildRequires, and %build runs jq.
+assert "/usr/share/tine-package/built-with-jq" in files, files
 assert query(subpackage, "--queryformat", "%{NAME}") == "hello"
 print(f"hello: pinned packaging, {len(files)} files")

examples/package/rpm/hello_tree_check.py

@@ -26,6 +26,8 @@ def query(package: Path, *args: str) -> str:
 markers = "/usr/share/tine-package"
 # The spec comes out of the checkout as in a dev build, but none of the dev-only savings apply.
 assert f"{markers}/spec-from-checkout" in files, files
+# The build installs jq only because the spec lists it in BuildRequires, and %build runs jq.
+assert f"{markers}/built-with-jq" in files, files
 assert f"{markers}/profile-lto-enabled" in files, files
 assert f"{markers}/profile-annobin-enabled" in files, files
 assert list(rpms.glob("hello-debuginfo-*.rpm")), sorted(rpms.iterdir())

package/build.bzl

@@ -7,7 +7,7 @@ load("//:specs.bzl", "specs")
 load("//box:defs.bzl", "box")
 load("//git:defs.bzl", "git")
 load("//project:defs.bzl", "project")
-load(":buildroot.bzl", "BuildrootInfo", "package_buildroot")
+load(":buildroot.bzl", "BuildrootInfo")
 load(":manager.bzl", "PackageManagerInfo")
 load(":repository.bzl", "LocalPackageInfo")
 load(":system.bzl", "PackageSystemInfo")
@@ -66,16 +66,15 @@ def package_build(
     owner: str,
     package_system: Dependency,
     base: BuildrootInfo,
-    build_requires: list[str],
-    buildroot_packages: list[Dependency],
+    stack: list[Artifact],
     configured_dev: bool,
     keep_build_dir: bool,
     packages: str,
     subpackages: list[str],
     category: str,
     spec: dict[str, typing.Any],
 ) -> list[Provider]:
-    """Install `build_requires` over `base`, and run the `build` driver of `package_system` on the result.
+    """Run the `build` driver of `package_system` on the layer stack `stack` that `base` starts.
 
     `spec` contains the fields of the driver spec that the calling rule owns. `packages` names the output
     directory of the packages. A dev build keeps its build directory between runs if `keep_build_dir` is
@@ -95,7 +94,6 @@ def package_build(
     reserved = [name for name in ("build", "buildroot") if name in subpackages]
     if reserved:
         fail("{}: subpackages {} have the names of built-in sub-targets".format(owner, reserved))
-    stack = package_buildroot(ctx, owner, base, build_requires, buildroot_packages)
 
     out = project.kept_dir(ctx.actions, packages)
     build_dir = project.kept_dir(ctx.actions, _PRIVATE + "/build") if configured_dev and keep_build_dir else None
@@ -118,7 +116,6 @@ def package_build(
                 spec
                 | {
                     "build_dir": build_dir.as_output() if build_dir != None else None,
-                    # The base root is the lowest layer, and the layer of build_requires is above it.
                     "lower": stack,
                     "out": out.as_output(),
                     "subpackages": subpackages,

package/builder.py

@@ -76,8 +76,8 @@ def run(spec: Spec, build: Callable[[Path, Path, Path], int]) -> int:
     return rc
 
 
-def buildroot(spec: Spec, topdir: Path) -> AbstractContextManager[Path]:
-    """Chroot into the buildroot of `spec`, with topdir at `CHROOT_TOPDIR`."""
+def buildroot(lower: list[str], topdir: Path) -> AbstractContextManager[Path]:
+    """Chroot into the buildroot with the layers `lower`, with topdir at `CHROOT_TOPDIR`."""
     return rootfs.rootfs(
-        "/buildroot", lowers=spec["lower"], binds=[(topdir, CHROOT_TOPDIR)], apivfs=True, chroot=True
+        "/buildroot", lowers=lower, binds=[(topdir, CHROOT_TOPDIR)], apivfs=True, chroot=True
     )