Skip to content

build: Key the Secure Boot signing layers on the certificate - #2498

Open
cgwalters-bot wants to merge 1 commit into
bootc-dev:mainfrom
cgwalters-forge:bot/secureboot-cache-key
Open

cgwalters-bot wants to merge 1 commit into
bootc-dev:mainfrom
cgwalters-forge:bot/secureboot-cache-key

Conversation

@cgwalters-bot

@cgwalters-bot cgwalters-bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Build secrets aren't part of podman's layer cache key. So the steps that sign systemd-boot and the UKIs (sdboot-signed, sealed-uki, and sealed-upgrade-uki in tmt/tests/Dockerfile.upgrade) can be served from cache after the test keys change. That happens after target/ is wiped, or when two checkouts share container storage. The image then fails Secure Boot with "Access Denied", which looks like a real bug. I hit this while testing several branches on one machine. CI builds one checkout per runner, so it isn't affected.

The fix copies the public certificate (db.crt, from a secureboot build context) into each signing stage and prints its checksum when signing. Copied content is part of the cache key, so a different certificate reruns those steps. The private key is still only a build secret. A new signing stage needs the same COPY.

Tested on a 16-core RHEL 10 devspace, Fedora 44 base, sealed UKI with systemd-boot: just build plus just _build-upgrade-image twice with key A, then with a regenerated key B. Round 2 reused the cache (no signing step ran); round 3 reran all three signing steps. sbverify shows systemd-boot and both UKIs signed with A in rounds 1-2 and with B (not A) in round 3. just validate passes.

Maintainers: this repository requires DCO, and these commits have no Signed-off-by. To add yours, run git rebase --signoff <upstream>/main on this branch and force-push it.

Generated-by: https://github.com/cgwalters/#llms

@cgwalters cgwalters left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is kind of hacky. I get the problem but what if we copied in the public key into a layer and had a trivial use of that like printing it from our script? I bet that'd work

cgwalters-bot added a commit to cgwalters-bot/homegit that referenced this pull request Sep 25, 2026
A URL seen for the first time was only recorded, its comments and
reviews marked as seen. That is right for someone else's issue added to
the board with its history, but not for a PR the bot just opened: the
board cache can lag behind 'bot-pr promote' by a sweep or two, and
cgwalters' CHANGES_REQUESTED review on bootc-dev/bootc#2498, 12 minutes
after the promote, was swallowed that way.

So for an issue or PR the bot opened, first sight reports everything
since its creation, with the usual filters (not the bot's own). The
created_at comes with the REST read the sweep makes anyway, where the
board item's own creation time would cost GraphQL. Seeding the watch
state at promote time instead would race with a sweep's
read-modify-write of the same state, and only cover promote. The very
first sweep, with an empty state, still reports nothing.

Generated-by: AI
cgwalters-bot added a commit to cgwalters-bot/homegit that referenced this pull request Sep 25, 2026
…sweep

bot-watch only reports edges: whatever it sees once is marked seen, so
news it misses or reports into a lost output is gone for good, as his
review on bootc-dev/bootc#2498 was. For the bot's own open upstream PRs
that means lost work, so every sweep now also lists, in its own
"Outstanding reviews by cgwalters" section, his latest review
requesting changes or commenting, or comment, while it is newer than
both the head commit and the bot's last comment or review there. A push
or a reply clears it, as does his approval after it.

It reuses the reviews the sweep already reads; the head commit and the
comments since it are ETag'd REST reads that mostly answer 304. Fork
PRs stay 'bot-pr inbox' business. The coordinator treats the section as
P0.

Generated-by: AI
@cgwalters-bot
cgwalters-bot force-pushed the bot/secureboot-cache-key branch from 17fd598 to bb81555 Compare September 25, 2026 15:45
@cgwalters-bot

Copy link
Copy Markdown
Contributor Author

Done, that works: the build arg is gone (force-pushed bb81555, same single commit).

  • db.crt comes in via a secureboot build context and is COPY'd into sdboot-signed, sealed-uki and sealed-upgrade-uki; each signing script prints its sha256sum. Only the cert is copied; the private key stays a build secret.
  • Tested on a 16-core RHEL 10 devspace (Fedora 44, sealed UKI + systemd-boot): built twice with key A (second run fully cached, no signing step ran), then with a regenerated key B, where all three signing steps reran and sbverify shows systemd-boot and both UKIs signed with B and not A. just validate passes.

Generated-by: https://github.com/cgwalters/#llms

@cgwalters cgwalters left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feels like the real fix is --mount=type=secret,invalidate or something for podman but yeah looks sane

@cgwalters

Copy link
Copy Markdown
Collaborator

@cgwalters-bot you can add my signoff

@cgwalters-bot
cgwalters-bot force-pushed the bot/secureboot-cache-key branch from bb81555 to f27d0cb Compare September 25, 2026 19:35
@cgwalters-bot

Copy link
Copy Markdown
Contributor Author

Added your sign-off.

Generated-by: https://github.com/cgwalters/#llms

Build secrets aren't part of the layer cache key, so the steps that
sign systemd-boot and the UKIs can be served from cache after the test
keys change: after `target/` is wiped and the keys regenerated, or from
another checkout sharing the same container storage. The result is an
image signed with a key the VM firmware doesn't trust, which fails to
boot with "Access Denied" in a way that looks like a real Secure Boot
bug.

Copy the public certificate into each signing stage from a build
context, and print its checksum when signing. Copied content is part
of the cache key, so a different certificate rebuilds those steps.
Only the certificate is copied; the private key stays a secret.

Generated-by: AI
Signed-off-by: Colin Walters <walters@verbum.org>
@cgwalters-bot

Copy link
Copy Markdown
Contributor Author

Rebased onto main; 1 commit, no content change.

Generated-by: https://github.com/cgwalters/#llms

@cgwalters-bot
cgwalters-bot force-pushed the bot/secureboot-cache-key branch from f27d0cb to e2c44d9 Compare September 26, 2026 03:22

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants