Skip to content

Ship a prebuilt agent binary instead of the crate source - #8

Merged
jetm merged 10 commits into
mainfrom
reland-prebuilt-binaries
Nov 20, 2026
Merged

jetm merged 10 commits into
mainfrom
reland-prebuilt-binaries

Conversation

@jetm

@jetm jetm commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Problem

#5 was meant to make this change, and it reads as merged, but main never received it. Its base was the stacked branch release-matrix-by-feed, which had already landed, so it merged into an orphaned branch. Its merge commit 3ba5456 is diverged from main.

Main therefore still ships the crate source. The 0.1.0 and 0.1.1 packages are about 74 KB of Rust that cad-compile.sh compiles at the consumer's build. A package-sourced extension does not carry its own sdk.packages to the consumer, so a clean project fails inside ring's build script with an error that never names this extension.

Solution

The same change as #5, cherry-picked unchanged onto current main (d1ad06c, no conflicts). The release workflow builds static musl binaries for x86_64 and aarch64 and attaches them with a SHA256SUMS file to the release. cad-compile.sh fetches the binary for the version in avocado.yaml and checks it against the SHA256SUMS from the same release. The package no longer carries the crate, so a consumer needs no Rust toolchain. Version is 0.2.0.

Key changes

  • release.yml: a binaries job (arch matrix, cross build, static-link check), a release-assets job (SHA256SUMS, gh release create and upload), and a publish job that needs: release-assets for the 2026/next and 2024/next feeds
  • cad-compile.sh fetches and verifies; cad-install.sh installs
  • avocado.yaml: version 0.2.0, crate and SDK toolchain declarations removed
  • CHANGELOG.md

Reviewer notes

I verified on avocado 1.1.0 that avocado ext package against qemux86-64 exits 0 and produces avocado-ext-container-agent-dev-0.2.0-r0.noarch.rpm containing only the manifest, the three scripts and the two overlay units. bash -n and shellcheck are clean.

The fetch path has never executed. It pulls from the 0.2.0 release, which does not exist until this merges and the tag is pushed, so the tag is its first real run. publish waits on release-assets, so the assets exist before anything is published.

Feedback I'd most like: the trust model. The binary and its checksum come from the same release, so this catches corruption but not a compromised release.

Following the published guide end to end from a clean project does not
work. `avocado ext package` runs no build step, so the published RPM can
only carry what is in the repo - which made it a source package and
pushed the compile onto the consumer. A consumer has no Rust cross
toolchain in their SDK and no reason to, and the extension's own
sdk.packages do not travel with a package-sourced extension, so the
build died inside ring's build script looking for
x86_64-avocado-linux-gcc. Both `avocado install` and the extension's
sysroot creation had already reported success, so nothing pointed at the
extension.

Fetching a prebuilt artifact is what ext-microclaw does, and this is why
rather than convenience: the release path has nowhere to compile, so the
only question is who compiles and when. Doing it in the extension's own
CI puts it where the toolchain is already known-good.

The binaries are static musl rather than glibc. A gnu build pins the
runner's glibc version into the artifact and fails on any older device
at exec time; static musl has no libc coupling at all, so one artifact
per architecture serves every target of that architecture. Measured: 3.8
MB x86_64 static-pie, 3.3 MB aarch64 static, both built under `cross`,
ring included.

SHA256SUMS comes from the same release as the artifact, so it verifies
the transfer and not the provenance. That is proportionate for a
first-party artifact and avoids pinning a digest in the compile hook,
which would force every release through build, commit, then tag just to
record a hash the tag itself produces.

cad-install.sh now checks the staged binary's architecture against the
target. The packaged extension is noarch and nothing downstream
re-checks that the two agree, so a mismatch would install cleanly and
fail only when the device tried to exec it.

Triple resolution is gone from both hooks along with the compile. The
ambiguity guard it carried is no longer needed here, but the five
sibling extensions still resolve by first prefix match and are still
exposed on any target whose architecture matches the SDK host's.

Signed-off-by: Javier Tia <javier@peridio.com>
Comment thread cad-compile.sh Resolved
Comment thread avocado.yaml Resolved
Comment thread .github/workflows/release.yml Resolved
Comment thread .github/workflows/release.yml Resolved
Comment thread .github/workflows/release.yml Resolved
Comment thread cad-compile.sh Resolved
Comment thread cad-compile.sh Resolved
Comment thread avocado.yaml Resolved
jetm added 3 commits November 20, 2026 08:48
The README, agent/README.md and the check-provider.sh header still
described the crate being compiled by cad-compile.sh at package time, a
release matrix with one row per target, and cad-compile.sh as the place
the provider guard runs. None of that is true after the switch, and the
local build recipe did not say it only works for a released version.

The avocado.yaml comment also claimed the removed sdk.packages never
reached a consumer. avocado-cli's deep_merge_ext_section copies every key
of the extension block missing from the consumer's stub, sdk included, so
the removal stands only because nothing compiles any more.

Signed-off-by: Javier Tia <javier@peridio.com>
`avocado ext package` rewrites the extension's version line to the quoted
form `version: '0.2.0'` in the avocado.yaml a consumer receives, and the
CLI does not export AVOCADO_EXT_VERSION at build time. The version grep
required a digit straight after the whitespace, so on every consumer the
assignment exited 1 under pipefail before anything was fetched. It also
cut a prerelease tag like 0.2.0-rc.1 down to 0.2.0 and would have fetched
the stable assets.

cad-clean.sh still ran `cargo clean` inside agent/, which is no longer
packaged and which a consumer has no cargo for, so the clean hook failed
for everyone and never removed the staged download. Together with the
skip-if-present check in the compile hook, an interrupted fetch left a
partial tarball that every later build reused.

Accept the version bare or quoted and keep the prerelease suffix. Fetch to
a .part file and rename on success. Make the clean hook remove the staged
agent directory, deriving it from AVOCADO_SDK_PREFIX when only that is
set, which is what `avocado ext clean` passes, and refuse when neither
variable is set rather than remove a path built from an empty string.

Signed-off-by: Javier Tia <javier@peridio.com>
GITHUB_REF_NAME names both the release assets and the release itself, so
a workflow_dispatch from a branch created a public release and tag named
after the branch, and a tag that differed from avocado.yaml attached
assets under a version no consumer resolves. The tag-equals-version check
inside extension-release.yml runs after the release exists and skips
non-tag refs.

The ring-only provider guard was only ever invoked by the old source
build in cad-compile.sh, so nothing checked the shipped binaries. PR CI
only exercises the extension build, which fetches the released binary, so
a change under agent/ was never compiled before a release.

Add a guard job that runs only on tag refs and compares the tag with the
manifest version before anything is built. Run check-provider.sh against
each shipped musl triple ahead of `cross build`, and add a PR job that
runs the same guard and build so a compile error fails the PR.

Signed-off-by: Javier Tia <javier@peridio.com>
@jetm
jetm requested a review from lee-reinhardt November 20, 2026 14:58
@jetm
jetm dismissed lee-reinhardt’s stale review November 20, 2026 15:22

All eight threads on this review are answered and resolved, and the fixes are in 1ece6dc. Two of them, the release process and the PR CI check for an unreleased version, are decisions I still have to make and are not changed in this PR. The build check stays red until the 0.2.0 release exists. Lee, please re-review at 1ece6dc.

jetm added 2 commits November 20, 2026 09:42
The release-assets job tolerated an existing release and uploaded with
--clobber, so a re-run under the same tag silently replaced the
binaries and SHA256SUMS. A consumer that already fetched the old asset
then holds a checksum the release no longer matches, and nothing fails
until their next build.

The job now refuses when a release for the tag exists, creates the
release as a draft, uploads without --clobber and publishes as its last
step. Publishing inside the same job matters because publish runs
cad-compile.sh, which fetches the assets anonymously, and a draft
release's assets cannot be downloaded that way. The draft keeps the
release from appearing without its files while the upload runs.

Signed-off-by: Javier Tia <javier@peridio.com>
A version-bump PR always shows a red extension build check, because that
check downloads the new version's release assets and they exist only
once the release workflow has run on the tag. Without the order written
down, the red check reads as a blocker and the tag never gets pushed,
or a re-run is tried to fix assets that cannot be replaced.

The Releasing section now gives the merge, tag, release, publish order
and states that a release is created once per tag, so a rebuilt binary
needs a new version.

Signed-off-by: Javier Tia <javier@peridio.com>
Comment thread cad-compile.sh Resolved
Comment thread .github/workflows/release.yml Resolved
Comment thread README.md Resolved
Comment thread .github/workflows/release.yml Resolved
Comment thread cad-compile.sh Resolved
Comment thread cad-compile.sh Resolved
The tarball was verified only after its .part file was renamed, so a
cached tarball that went bad failed sha256sum on every later build, with
no hint that `avocado ext clean` fixes it. SHA256SUMS was fetched again
on every run with no retry, so a build host without GitHub access could
not rebuild even with the tarball already cached.

Each download is now checked against the sums before the rename. A
cached tarball that fails the check is removed and fetched once more;
a second mismatch fails and names the clean command. The sums are cached
as SHA256SUMS-<version> through the same .part path, and both curl calls
retry like mc-compile.sh. A tarball and SHA256SUMS placed in dist/ are
used instead of the release, which is the consumer half of carrying the
digests in the package; nothing produces that directory yet, so the
release stays the trust anchor and the comment now says so.

Signed-off-by: Javier Tia <javier@peridio.com>
Comment thread .github/workflows/release.yml Resolved
lee-reinhardt
lee-reinhardt previously approved these changes Nov 20, 2026
jetm added 3 commits November 20, 2026 10:53
The release-exists comment said a broken lookup never turns into a silent
skip. gh prints "not found" for a missing tag and for a repository it cannot
see alike, so a wrong repository would skip. The repository is
$GITHUB_REPOSITORY, the one the job runs in, so that case cannot arise; the
comment now says so instead of claiming every lookup failure fails.

Signed-off-by: Javier Tia <javier@peridio.com>
The release workflow published the extension without ever running
cad-compile.sh. extension-release.yml@v1 only packages and uploads (its
own comment: no sdk/ext install or build step), so a wrong asset name or
a missing tool in the SDK image would first fail on a consumer's build,
after the extension was in both feeds. The comments here said publish
ran the fetch. On PRs, every version bump left `build` red with a 404,
because the release it fetches is created only after the tag is pushed.
The binaries were built with whatever stable rustc the runner had and
without --locked.

A verify job now runs extension-test.yml@v1 against the published
release for qemux86-64 and imx93-frdm on both publish feeds, and publish
needs it. In test.yml, a release-exists job looks the version up and
build runs only when the release exists, so a version bump shows a
skipped check with the reason logged. A lookup error other than "not
found" (network, auth) fails the job. gh prints "not found" for a missing
tag and for a repository it cannot see alike, which cannot arise for
$GITHUB_REPOSITORY, the repository the job runs in. The toolchain is
pinned in agent/rust-toolchain.toml and both cross builds use --locked.
The README drops its claim about main's required checks, which was not
verified, and states what the PR build does and does not test.

Signed-off-by: Javier Tia <javier@peridio.com>
A cached SHA256SUMS-<version> was trusted until `ext clean`. A proxy that
answers a download with a 200 page, or a copy fetched before the release was
complete, left a file that did not list the asset, and every later build
failed with "the release is incomplete" without fetching again or naming
the clean command.

A cached file that does not list the asset is now fetched once more, and the
failure that remains names the clean command. A dist/ that holds only one of
the tarball and SHA256SUMS now prints a warning before the build falls back
to the release, since the staged binary the user meant to use is otherwise
ignored without a word. The comment on the verify-before-rename step no
longer says every file under the final name matched the sums, which a
tarball cached by an earlier version of this script did not.

Signed-off-by: Javier Tia <javier@peridio.com>
@jetm
jetm merged commit 1557aaf into main Nov 20, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants