Integrate out-of-tree DKMS modules into the kernel package at build time

[bjordiscollaku]

Tracking the work to ship Qualcomm's out-of-tree kernel modules inside the kernel .deb itself, building each module at kernel-package build time rather than on the device. The mechanism is module-agnostic; kgsl (msm_kgsl.ko, from qualcomm-linux/pkg-kgsl) is the reference module used to validate it.

Branch: feat/dkms-build-time-module-integration (off qcom/debian/latest). PR: #54.
Status: the mechanism is implemented as a standalone tool and validated end to end on the Ubuntu/resolute path. The Debian/Debusine path is blocked because the out-of-tree module packages are not yet published through the Debusine flow, so there is no Debusine-side kgsl-dkms to depend on. PR #54 proposes landing the mechanism and staging per-family module enablement.

Latest status of development: the bundling logic lives in a standalone, documented tool, debian/scripts/bundle-dkms-modules.sh, that debian/rules invokes during the kernel build. On the Ubuntu/resolute path it is validated end to end: CI run 28193902324 (kernel 7.1.0-rc7-qcom-next-20260618) built kgsl 1.0.4 against the same-run staged headers and bundled msm_kgsl.ko into the kernel package (stripped, 2,783,736 bytes), with its 1,271,360 bytes of debug routed to the -dbg package. The tool's full control flow and every failure path are covered by an 18 of 18 fixture harness. On-target load testing is the remaining gate.

---

Goal

1. One kernel .deb containing the kernel image plus every required out-of-tree module, each compiled against that exact kernel.
2. DKMS used only at build time. No DKMS, compiler, or headers on the device.
3. Adding a new module is a two-line change (manifest entry plus Build-Depends), with no build-logic changes.

Requirements

#	Requirement
R1	One kernel .deb: kernel image plus all out-of-tree modules built in the same run. No DKMS on the device.
R2	The kernel build declares Build-Depends on DKMS source packages. DKMS is a build-time tool, not a runtime mechanism.
R3	DKMS packages must not appear in Depends, Recommends, or Suggests of any binary package.
R4	Each module is compiled from source against the exact kernel produced in the same run. No cached .ko.
R5	A change to a DKMS package requires a full kernel rebuild. Multiple DKMS changes can be batched into one rebuild.
R6	Each module's CI stays independent and validates its own *-dkms package. The kernel CI only consumes it.
R7	The bundled module is handled like an in-tree module: stripped on the device, debug symbols in the -dbg package.

---

Approach

module *-dkms package  (module source + dkms.conf, no prebuilt module)
    published to an apt source:
        Ubuntu: Qualcomm Artifactory   (available now)
        Debian: Debusine workspace     (module pkgs not yet on Debusine)
    |
    |  installed via Build-Depends
    v
================ one dpkg-buildpackage run ================
  debian/rules (override_dh_auto_install):
    1. build kernel + headers, stage them in the build tree
    2. call debian/scripts/bundle-dkms-modules.sh with the staged paths
         the tool, for each manifest module:
           a. resolve source + tokens via dpkg -L and dkms.conf
           b. dkms build against the staged headers (private tree)
           c. judge by artifact; fail with diagnostics if none
           d. install into lib/modules/<KVER>/extra/ (stripped)
           e. route debug symbols to the -dbg package
           f. guard against duplicate and in-tree name collisions
==========================================================
    |
    v
  kernel image .deb  (kernel + in-tree modules + extra/*.ko)
  -dbg .deb          (debug symbols, including the bundled module's)
    |
    v
  on device: the postinst depmod indexes extra/, modprobe and autoload resolve

Reference module (kgsl): dkms.conf sets PACKAGE_NAME=kgsl, BUILT_MODULE_NAME=msm_kgsl, source installs to /usr/src/kgsl-<version>/, and it gates the build with BUILD_EXCLUSIVE_CONFIG=CONFIG_ARCH_QCOM and BUILD_EXCLUSIVE_KERNEL_MIN. The latest validated build used kgsl 1.0.4.

---

The tool: debian/scripts/bundle-dkms-modules.sh

The bundling logic is a single standalone tool with a documented CLI, so debian/rules stays a thin call, future changes touch one file, and a developer can re-run just the bundling step against already-staged trees.

Required:
  --kver KVER               kernel release (uname -r)
  --headers-dir DIR         absolute path to the staged kernel headers root
  --image-pkg-dir DIR       linux-image staging tree root
  --dbg-pkg-dir DIR         linux-image-dbg staging tree root
Optional:
  --modules-manifest FILE   default: debian/dkms-modules
  --arch ARCH               default: aarch64 (uname -m vocabulary for dkms)
  --objcopy PATH            default: aarch64-linux-gnu-objcopy, then objcopy
  -h, --help

Two invocation paths, same tool:

• CI path (debian/rules, override_dh_auto_install) calls it with explicit absolute paths after the kernel, headers, and debug trees are staged:

  $(CURDIR)/debian/scripts/bundle-dkms-modules.sh \
      --kver            "$BASE" \
      --headers-dir     "$(CURDIR)/debian/linux-headers-$BASE-qcom/usr/src/linux-headers-$BASE" \
      --image-pkg-dir   "$(CURDIR)/$PKG" \
      --dbg-pkg-dir     "$(CURDIR)/$DBG_PKG" \
      --arch            "$(DKMS_ARCH)" \
      --objcopy         "$(OBJCOPY)" \
      --modules-manifest "$(CURDIR)/debian/dkms-modules"
  

• Developer path: a developer who has already staged the trees can re-run just the bundling step directly, passing the same paths. --help documents the full CLI and prerequisites.

---

How the module package reaches the build

The kernel build only needs to install <name>-dkms as a Build-Depends. Each path sources it from a different apt repository, and neither needs changes in pkg-linux-qcom:

• Ubuntu (resolute, docker): the Qualcomm Artifactory apt source is already configured in the resolute build image, so dpkg-buildpackage installs kgsl-dkms directly. This is what makes the Ubuntu path workable today.
• Debian (trixie, Debusine): the dependency would come from the Debusine-managed apt repository, which the Debusine build environment reaches automatically. The out-of-tree module packages are not yet published through the Debusine flow, so there is no Debusine-side kgsl-dkms to install; this path is deferred until that integration exists.

---

Implementation (on the branch)

• debian/control.in: adds kgsl-dkms (>= 1.0.2), dkms to Build-Depends only (R2, R3).
• debian/dkms-modules: plain-text manifest, one module per line. Comments, blank lines, and CRLF endings are tolerated.
• debian/scripts/bundle-dkms-modules.sh: the standalone bundling tool. For each manifest module it:
  1. resolves the source and the dkms name/version tokens via dpkg -L <name>-dkms and the shipped dkms.conf (the package manager and dkms.conf are the authorities, no /usr/src globbing);
  2. creates a private DKMS tree (mktemp -d, removed via an EXIT trap) with a source symlink;
  3. dkms build against the staged headers (absolute --kernelsourcedir, --arch aarch64 in dkms's uname -m vocabulary);
  4. judges the outcome from artifacts (built .ko, dkms exit code, make.log) and fails with precise diagnostics, never a guessed cause;
  5. installs the .ko into /lib/modules/<KVER>/extra/, extracts its debug into the -dbg tree, then strips the shipped copy;
  6. guards against duplicate and in-tree module name collisions.
• debian/rules (override_dh_auto_install): invokes the tool with explicit absolute paths after the trees are staged. dh_installmodules (called in override_dh_installdeb) wires the postinst depmod that indexes extra/ on the device.

---

Key design decisions

Decision	Rationale
Build-time DKMS, not on-device	Single self-contained .deb; no toolchain, headers, or DKMS on the target; the module is pinned to the exact kernel (R1, R4).
DKMS as a build-time dependency only, never runtime	The module is baked in, so nothing is resolved at install time and the runtime dependency graph stays clean (R3).
A standalone, documented tool	The logic lives in debian/scripts/bundle-dkms-modules.sh, so debian/rules describes what happens, future changes touch one file, and the bundling step is independently re-runnable by a developer.
Build the module against this run's kernel headers	The module is compiled against the exact kernel and headers produced in the same run, so it matches and loads against this kernel. Those headers are staged earlier in the run, so the tool runs late in override_dh_auto_install (R4).
Index the module on the device with the postinst depmod	The module is added to extra/ after make modules_install has generated the module metadata, so the standard dh_installmodules postinst depmod re-indexes extra/ at install, where modprobe and autoload resolve.
Manifest-driven	A new module is two lines with no logic change; which modules to build is separated from how to build them.
The manifest is a presence contract	If a listed module produces no .ko for the target kernel, the build fails rather than silently shipping without it (C4). A module that does not apply to a kernel is expressed by an explicit manifest entry, never inferred from the module's own BUILD_EXCLUSIVE_* directives, so a misconfigured gate cannot silently drop a declared module.
Judge dkms outcomes by artifacts, not exit codes	dkms exit-code conventions vary across versions (a gate skip can exit 0). Success is a built .ko; failure diagnosis comes from dkms's own make.log (printed inline, since the private tree is ephemeral) and from the gates declared in the module's dkms.conf (C4).
Resolve module identity from dpkg and dkms.conf	dpkg -L names the one dkms.conf the package ships, and its PACKAGE_NAME/PACKAGE_VERSION are the tokens dkms itself keys on. No path guessing, and the manifest entry only needs to match the package name (C5).
Consume each module's published *-dkms package	The kernel build pulls in the module's own dkms.conf and source as a build dependency rather than vendoring it, so each module's CI stays independent and the kernel build just consumes it (R6).
Private --dkmstree plus manual source symlink	Lets dkms build run rootless under fakeroot, away from root-owned /var/lib/dkms (C1).
Handle the bundled .ko like an in-tree module	Debug to -dbg via objcopy --only-keep-debug, then strip the shipped copy via strip --strip-debug (the in-tree INSTALL_MOD_STRIP=1 behavior).

---

Challenges and findings

C1 to C3 were initial bring-up. C4 to C6 are hardening. C7 is resolved.

#	Problem	Resolution / decision
C1	dkms build cannot write the root-owned /var/lib/dkms tree under fakeroot	Private --dkmstree $(mktemp -d) plus a manually created source symlink; build runs rootless
C2	make fails because KERNEL_SRC is relative (DKMS runs make from inside the module source dir)	Absolute --kernelsourcedir
C3	.ko installed to the wrong path (find -exec {} expands to the full path)	while read loop plus basename
C4	dkms build exit codes do not reliably signal the outcome (a BUILD_EXCLUSIVE_* skip can exit 0, conventions vary across dkms versions), so a non-matching kernel or a failed build could silently ship without the module; and on a real compile failure dkms's make.log lived only in the ephemeral build environment	Judge by artifacts: a built .ko means success; a present make.log means a failed build and its tail is printed inline; an absent make.log means dkms attempted no build, and the BUILD_EXCLUSIVE gates from dkms.conf are printed with each BUILD_EXCLUSIVE_CONFIG evaluated against the staged kernel config. The build hard-fails in every no-module case
C5	Module resolution parsed the dpkg version string (cut -d- mishandles epochs and hyphenated upstream versions); globbing /usr/src/<name>-* was rejected as fragile against look-alike prefixes	Resolve via the package manager: dpkg -L <name>-dkms names the one shipped dkms.conf, and its PACKAGE_NAME/PACKAGE_VERSION are the exact tokens dkms keys on
C6	The dkms build architecture was a hardcoded literal, and dkms speaks uname -m vocabulary (aarch64) while kbuild uses arm64; a future BUILD_EXCLUSIVE_ARCH gate would mismatch the kbuild token	Single-source it as DKMS_ARCH (aarch64), documented next to the kbuild ARCH variable, and pass --arch $(DKMS_ARCH)
C7 (resolved)	Whether strip --strip-debug was operating correctly on the bundled module	Confirmed correct against two module builds. kgsl 1.0.2 carried no DWARF, so strip was a correct no-op there. kgsl 1.0.4 carries debug information, and the latest CI build shows the two-stage handling doing real work: 1,271,360 bytes of debug separated into the -dbg package and the shipped copy stripped to 2,783,736 bytes. The handling correctly separates whatever debug the module emits, in both cases

A post-extraction hardening pass also fixed a recipe syntax error introduced by the extraction (caught by CI and a make reproduction), made the tool fail closed when the manifest is missing, and hardened its CLI.

---

Validation

• CI: run 28193902324 (resolute, kernel 7.1.0-rc7-qcom-next-20260618). The Build kernel package step builds and bundles the module from its kgsl-dkms package. msm_kgsl.ko is bundled at /lib/modules/<KVER>/extra/ (stripped, 2,783,736 bytes) and its debug (1,271,360 bytes) is in the -dbg package. All four packages (image, headers, dbg, dbgsym) build, the Upload .deb packages to S3 step passes, and the artifacts are published to the S3 bucket under .../pkg/temp/pkg-linux-qcom/28193902324-1/. Every step in the run is green.
• Local (standalone tool): the tool was exercised through its CLI against shimmed leaf tools so its real control flow, resolution, outcome judgment, collision guards, and path layout run. 18 of 18 cases pass: happy path (install, debug split, strip), resolution failures (package missing, multiple dkms.conf, missing PACKAGE_NAME), outcome diagnosis (build-failure make.log tail; gated skip with config set, unset, and no gates; compressed output), collisions (in-tree, duplicate basename, duplicate manifest entry), and manifest handling (comment-only skip, missing-manifest fail-closed, default path).
• Not done: the package has not been installed on target hardware, and module loading (modprobe msm_kgsl, autoload) has not been exercised on a device. On-target testing is a separate, later step.

---

Open items

• On-target validation: install the .deb on a device and exercise module loading (modprobe msm_kgsl, autoload). The remaining gate before this is fully validated end to end.
• Debian / Debusine path: deferred until the out-of-tree module packages are published through the Debusine flow; validate once a Debusine-side kgsl-dkms is available. The build logic already supports both paths.
• Standalone tool: the bundling logic is extracted into debian/scripts/bundle-dkms-modules.sh with a documented CLI.
• Raise the PR: PR Integrate out-of-tree DKMS modules into the kernel at build time #54 into qcom/debian/latest, proposing to land the mechanism and stage per-family enablement.

---

How to add a new module

1. Append the module name to debian/dkms-modules.
2. Add dkms and <name>-dkms (>= x.y.z) to Build-Depends in debian/control.in.

Prerequisite: the module ships a standard dh-dkms package (source at /usr/src/<name>-<version>/) reachable from the build environment's apt sources (Artifactory for Ubuntu, the Debusine workspace for Debian). No change to debian/rules or the bundling tool is needed.

[phanendra-qc]

are these DKMS packages pulling into pkg-linux-kernel are version controlled?

[phanendra-qc]

Salendarsingh Gaud (@sgaud-quic) knaveen-qc

[phanendra-qc]

curious to know How blacklisting of drivers handled with packing DKMS debs onto kernel debian ? Bjordis Collaku (@bjordiscollaku)

[bjordiscollaku]

@phani-sudo Good question. Bundling the module into the kernel .deb is orthogonal to blacklisting. The module lands in /lib/modules//extra/ and is indexed by the standard depmod, so to the loader it is an ordinary module, and blacklisting acts by module name in the modprobe/kmod and kernel-cmdline layer regardless of how the .ko was packaged. So the usual controls (a blacklist or install line in modprobe.d, or modprobe.blacklist= / module_blacklist= on the kernel command line) work exactly as they would for any module.

This mechanism intentionally ships no blacklist of its own. Not because a package cannot carry one, but because a blacklist is board and policy specific while this kernel package is shared across boards. So blacklisting stays a decision external to this mechanism, made at the image level (for example a modprobe.d drop-in added at the image build stage), and nothing here changes that workflow.

[phanendra-qc]

We could see this approach of DKMS packing into kernel making downstream ko's are being loaded without handling of blacklisting upstream drivers.
In this case, do we know how do we differentiate Upstream vs downstream kernel?

Where as , Traditional DKMS Flow follows
• When a new linux-image-* package installs:
• Kernel postinst triggers dkms autoinstall -k $(uname -r)
• DKMS rebuilds all registered modules for the new kernel version
• Initramfs updates via update-initramfs include rebuilt modules

Blacklisting workflow:
• DKMS packages (e.g., kgsl-dkms) ship with /etc/modprobe.d/ snippets:

/etc/modprobe.d/kgsl-dkms.conf to blacklist upstream driver

• This is packaged as a conffile in the DKMS .deb, not the kernel package
• Takes effect at next boot (prevents in-kernel driver from loading)

[phanendra-qc]

Robie Basak (@basak-qcom) snb (@nivasnaik)

[bjordiscollaku]

We could see this approach of DKMS packing into kernel making downstream ko's are being loaded without handling of blacklisting upstream drivers. In this case, do we know how do we differentiate Upstream vs downstream kernel?

Where as , Traditional DKMS Flow follows • When a new linux-image-* package installs: • Kernel postinst triggers dkms autoinstall -k $(uname -r) • DKMS rebuilds all registered modules for the new kernel version • Initramfs updates via update-initramfs include rebuilt modules

Blacklisting workflow: • DKMS packages (e.g., kgsl-dkms) ship with /etc/modprobe.d/ snippets:

/etc/modprobe.d/kgsl-dkms.conf to blacklist upstream driver

• This is packaged as a conffile in the DKMS .deb, not the kernel package • Takes effect at next boot (prevents in-kernel driver from loading)

@phani-sudo Replied to this on the qcom-deb-images thread where you raised the same point, to keep it in one place: qualcomm-linux/qcom-deb-images#386 (comment)

[bjordiscollaku]

Build-time DKMS bundling (#54) is now validated end to end on both distro families: Ubuntu resolute (run 28193902324) and Debian trixie (run 29125019996), each bundling msm_kgsl.ko into the linux-image. The Debian blocker (#58) is resolved by qualcomm-linux/pkg-kgsl#62 (the kgsl-dkms source is now published to the Debusine workspace). Pending: merge of #54.