Skip to content

Speed up ARM production release builds - #480

Merged
matteius merged 2 commits into
mainfrom
perf/native-arm-release-builds
Aug 4, 2026
Merged

Speed up ARM production release builds#480
matteius merged 2 commits into
mainfrom
perf/native-arm-release-builds

Conversation

@matteius

@matteius matteius commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • run the Docker linux/arm64 build on GitHub's native ubuntu-24.04-arm runner
  • cross-compile linux/arm/v7 on x86_64 with Debian armhf multiarch packages and a native LiteRT flatc host tool
  • leave QEMU only for ARMv7 runtime-image assembly, not C/C++/Go/web compilation
  • persist a separate GHCR BuildKit cache for each platform
  • reduce the build context with .dockerignore, omit recommended build-only packages, and skip compiling the unused test target in production images

Why

In release run 30839856661, the jobs took:

  • linux/amd64: 15m44s
  • linux/arm/v7: 3h42m48s
  • linux/arm64: 4h58m47s

The ARM jobs were compiling the complete LiteRT/XNNPACK dependency tree under QEMU. In the ARMv7 log, the final LightNVR build layer alone took about 2h55m. Docker documents compilation as a case where QEMU can be much slower than native execution and recommends native nodes or cross-compilation.

The existing Debian release matrix already demonstrates the native ARM64 runner path: its ARM64 jobs finish in roughly 10–13 minutes, while emulated ARMv7 jobs take about two hours.

Validation

  • cold ARMv7 builder: 9m12s
    • docker buildx build --platform linux/arm/v7 --target builder --output type=cacheonly .
    • verified /bin/lightnvr and /bin/go2rtc are 32-bit ARM EABI5 binaries
  • complete native amd64 runtime image: 13m03s
    • docker buildx build --platform linux/amd64 --output type=cacheonly .
  • trixie/armhf package path passed multiarch package installation and CMake dependency discovery; the redundant cold TensorFlow checkout was canceled after GitHub network transfer stalled
  • docker buildx build --check for amd64, arm64, and arm/v7
  • actionlint .github/workflows/docker-publish.yml
  • workflow YAML parse, bash -n scripts/build.sh, and git diff --check

The first release still has to populate the new registry caches, but ARMv7 compilation no longer runs under emulation. Subsequent releases can also reuse unchanged dependency layers.

Copilot AI review requested due to automatic review settings August 3, 2026 23:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reworks the production Docker release pipeline to significantly reduce ARM build times by switching ARM64 builds to a native ARM runner and cross-compiling ARMv7 on amd64 (leaving QEMU primarily for target runtime-layer assembly), while also improving layer/cache reuse and reducing Docker build context size.

Changes:

  • Updates the Docker publish workflow to use a platform→runner matrix (native ARM64 runner) and per-platform BuildKit registry caches.
  • Adjusts the Dockerfile to support ARMv7 cross-compilation via Debian armhf multiarch packages + an ARMv7 CMake toolchain, and disables production test builds.
  • Adds .dockerignore and documents the updated release build approach.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
scripts/build.sh Adds optional CMAKE_TOOLCHAIN_FILE forwarding into the CMake configure step.
docs/RELEASE_PROCESS.md Documents native ARM64 runner usage, ARMv7 cross-compilation, and per-platform cache behavior.
Dockerfile Implements cross-build logic for ARMv7, uses $BUILDPLATFORM for the builder stage, and adjusts build steps/env for multiarch.
cmake/toolchains/armv7-linux-gnueabihf.cmake Introduces an ARMv7 hard-float/NEON toolchain file for cross-compiling via Debian multiarch.
.gitignore Ensures the new toolchain .cmake files under cmake/toolchains/ aren’t ignored.
.github/workflows/docker-publish.yml Moves to a matrix with platform-specific runners/QEMU and adds per-platform registry cache import/export.
.dockerignore Shrinks Docker build context by excluding VCS/CI/dev artifacts and other large directories.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .dockerignore
Copilot AI review requested due to automatic review settings August 4, 2026 00:08
@matteius
matteius merged commit ca9c19d into main Aug 4, 2026
4 checks passed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 12 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

Dockerfile:216

  • This case "$TARGETARCH/$TARGETVARIANT" only matches when TARGETVARIANT is empty. If the builder provides a variant like arm64/v8, this will incorrectly fail with Unsupported target architecture while building go2rtc.
      amd64/) GOARCH=amd64; GOARM= ;; \
      arm64/) GOARCH=arm64; GOARM= ;; \
      arm/v7) GOARCH=arm; GOARM=7 ;; \
      *) echo "Unsupported target architecture: $TARGETARCH/$TARGETVARIANT"; exit 1 ;; \

Dockerfile:268

  • These patterns assume TARGETVARIANT is empty for amd64/arm64 (matching amd64/ and arm64/). If a non-empty variant is provided (e.g. arm64/v8), the build will fail as an unsupported architecture when selecting pkg-config paths.
        amd64/) PKG_CONFIG_ARCH_PATH="/usr/lib/x86_64-linux-gnu/pkgconfig"; TOOLCHAIN_FILE="" ;; \
        arm64/) PKG_CONFIG_ARCH_PATH="/usr/lib/aarch64-linux-gnu/pkgconfig"; TOOLCHAIN_FILE="" ;; \

Comment thread Dockerfile
Comment on lines +182 to +184
cmake --build /tmp/flatc-build -j"$(nproc)" && \
test -x /opt/host-tools/bin/flatc && \
/opt/host-tools/bin/flatc --version; \
Comment thread Dockerfile
Comment on lines +195 to +198
amd64/) LIB_DIR="/usr/lib/x86_64-linux-gnu"; MBEDTLS_PACKAGE=libmbedtls-dev ;; \
arm64/) LIB_DIR="/usr/lib/aarch64-linux-gnu"; MBEDTLS_PACKAGE=libmbedtls-dev ;; \
arm/v7) LIB_DIR="/usr/lib/arm-linux-gnueabihf"; MBEDTLS_PACKAGE=libmbedtls-dev:armhf ;; \
*) echo "Unsupported target architecture: $TARGETARCH/$TARGETVARIANT"; exit 1 ;; \
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants