[OMEGA-407] Enable GPU passthrough support in Omega Docker image

[paul-v-snet]

Description

The reason the agent cannot access the GPU is that the Landlock filesystem policy does not include the required NVIDIA driver and device paths. As a result, the main agent process cannot access the GPU, and its child processes cannot access it either because they inherit the same Landlock restrictions.

To fix the issue, it is sufficient to add the required paths to policy.yaml and implement a mechanism for adding them dynamically. The set of required paths may differ depending on the system and configuration. At the same time, if a path specified in the policy does not exist on the current system, Landlock cannot apply the policy and returns an error.

Therefore, all potentially required NVIDIA paths were added to policy.yaml, but before applying the policy, the apply method collects only the NVIDIA driver and device paths that actually exist on the current system and applies the policy using that set.

This should support different GPU configurations, including multiple GPUs exposed to the Docker container, as well as different environments such as native Linux and WSL.

How Has This Been Tested?

• Reproduced on WSL2 with --device nvidia.com/gpu=all passed to the container. Before the fix, the agent's own nvidia-smi -L shell call returned Failed to initialize NVML: N/A, while a plain docker exec ... nvidia-smi -L on the same running container detected the GPU correctly.
• Verified the fix in live testing. After the fix, nvidia-smi -L invoked by the agent from inside the container worked correctly, the agent gained access to the GPUs and was able to use them successfully.

Checklist

• PR contains autogenerated code
• Self-review completed
• Test scenarios above are passed with the version of the code from PR

[paul-v-snet]

I've converted the PR to draft since it appears that the issue is not fixed by this PR on native Linux systems.
I've tested the fix on my system, which is WSL2, and it works well there, but I'm not able to test it on native Linux yet.

[kabirkbr]

Hi,
Hope this could be helpful -- just vibe-constructed a small report on how this is done on NuNet's side, inspired by some discussions in Mattermost. Let me know if relevant and if I can provide any more refrences or explanations.

-- thanks!

How NuNet does GPU passthrough to Docker containers on native Linux

For reference, since NuNet's Device Management Service (DMS) runs GPU workloads in Docker on native Linux in production.

Short version: for NVIDIA, NuNet writes no passthrough code of its own. It adds no /dev/nvidia* binds, no cgroup device rules and no library mounts. The host has the NVIDIA Container Toolkit installed and registered with Docker. Each container is then created with a single Docker DeviceRequest carrying the gpu capability, which is the API form of docker run --gpus. The toolkit does the rest.

1. Host prerequisite: driver + NVIDIA Container Toolkit, registered with Docker

For a hand-built host, the DMS README lists it as a manual step: README.md L457-L463 (driver, then Container Toolkit, then reboot).

NuNet GPU appliances automate the same step in cloud-init. This is what runs on a NuNet GPU host (Ubuntu 24.04); the provisioning input is not in a public repo, so it is quoted rather than linked:

packages:
  - nvidia-driver-550-server
runcmd:
  # ... docker-ce installed first ...
  - install -d -m 0755 /usr/share/keyrings
  - bash -lc 'curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg'
  - bash -lc 'arch=$(dpkg --print-architecture); echo "deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://nvidia.github.io/libnvidia-container/stable/deb/${arch} /" > /etc/apt/sources.list.d/nvidia-container-toolkit.list'
  - apt-get update
  - DEBIAN_FRONTEND=noninteractive apt-get install -y nvidia-container-toolkit
  - nvidia-ctk runtime configure --runtime=docker --cdi.enabled=true
  - systemctl restart docker

Details that matter:

• Docker must be restarted after nvidia-ctk runtime configure. The runtime registration only takes effect after the restart.
• The default runtime stays runc. docker info shows Runtimes: io.containerd.runc.v2 nvidia runc, Default Runtime: runc. The nvidia runtime is registered but not the default, and Docker hands gpu requests to it per container.
• --cdi.enabled=true produces "features": {"cdi": true} in /etc/docker/daemon.json.
• Nothing is pulled in as a package dependency. The DMS .deb depends only on docker.io | docker-ce, iproute2, libsystemd-dev (control L4). The toolkit is installed explicitly, as above.

2. The container request: one DeviceRequest

configureHostConfig, executor/docker/executor.go L731-L747. This is the entire NVIDIA implementation:

deviceIDs := make([]string, len(params.Resources.GPUs))
for i, gpu := range params.Resources.GPUs {
    deviceIDs[i] = fmt.Sprint(gpu.Index)
}
hostConfig = container.HostConfig{
    Resources: container.Resources{
        DeviceRequests: []container.DeviceRequest{
            {
                DeviceIDs:    deviceIDs,
                Capabilities: [][]string{{"gpu"}},
            },
        },
    },
}

Driver is left empty, so Docker routes the request to whichever registered runtime advertises gpu. This is equivalent to docker run --gpus '"device=0,1"'. Compose spells the same thing gpus: all, as in NuNet's appliance: telemetry-exporter/compose.yaml L14-L20.

What the toolkit then does (NVIDIA's code, per their docs): before the container starts, it adds the device nodes for the requested GPUs, the matching cgroup device-allow rules, and bind mounts of the host driver libraries and nvidia-smi. The cgroup rule is the part that is easy to miss. Device nodes that are present without it produce Failed to initialize NVML: Insufficient Permissions: the device is visible, but opening it is denied.

3. Contrast: AMD and Intel, where there is no toolkit and DMS does everything by hand

executor.go L748-L769 (AMD) and L776-L790 (Intel):

• binds /dev/kfd and/or /dev/dri;
• adds a DeviceMapping with CgroupPermissions: "rwm", which is the cgroup rule written out explicitly;
• for AMD, adds GroupAdd: ["video"] so that a non-root container user can open the device.

Reading the three vendors side by side shows every piece that NVIDIA's toolkit otherwise does for you.

4. Detection and a smoke test

• Detection goes through NVML, never by scanning /dev: lib/hardware/gpu/nvidia.go. It is built only for linux && (amd64 || 386) (L9), so ARM hosts report no GPUs.

• Smoke test: nunet gpu test, in cmd/gpu.go L78-L237.

  • It refuses to run if the toolkit is missing (L103-L113).
  • Note that it requests Driver: "nvidia", Count: -1 (L136-L149), which is not the same request the executor makes.

• WSL2 vs native Linux, as NuNet's appliance code tells them apart (apply.sh L53-L86):

  • on WSL2, the GPU is /dev/dxg, and nvidia-smi lives at /usr/lib/wsl/lib/nvidia-smi, off PATH. There is no /dev/nvidia0.
  • on native Linux, the markers are /proc/driver/nvidia/gpus, /dev/nvidiactl and /dev/nvidia0.

  So a fix verified on WSL2 exercised a different device path from the one used on native Linux.

• Not covered by NuNet's e2e tests. Their acceptance suite runs without GPUs. What is in production is verified on real hosts, not in CI.

About the links: GitLab is the source, GitHub is a mirror

NuNet develops on GitLab (gitlab.com/nunet/device-management-service, gitlab.com/nunet/appliance). The GitHub repos are read-only mirrors with identical paths (their repo descriptions say "Mirror of …"). All links above point to GitHub and are pinned to a commit, so they work without a GitLab account and will not drift. On 2026-09-14 both mirrors were current: GitHub main == GitLab main == the pinned SHA (9f75a0bf for DMS, 0192439e for the appliance).

---

Additional: where this may meet Omega (observations, not yet tested)

Read at PR head 8b810452. These are leads to check, not a diagnosis.

1. The launcher never requests a GPU, and it recreates the container on every run.

   • scripts/omega runs docker rm -f omega (L796).
   • It then runs docker run with no --gpus (L798-L823).

   A GPU flag added to a container by hand is lost at the next launch. The NuNet equivalent would be an opt-in --gpus on that command, with the host prerequisite from section 1.

2. Landlock is the right layer for the agent side. The policy is applied in-process when the loop starts (src/loop.metta L73), and the shell skill's children inherit it (src/skills.pl L8-L10). A docker exec shell does not inherit it, which is why it sees the GPU. Two things in policy.yaml may matter on native Linux:

   • only /sys/devices/system is readable from /sys. NVML may need more of /sys (PCI device entries) on native Linux than on WSL2's /dev/dxg path. This is a hypothesis.
   • /dev itself is not listed. With Landlock enforcing, a real ls /dev from the agent should fail rather than print a list. So a /dev listing reported back by the agent is not reliable evidence.

3. A reproduction that takes the LLM out of the loop. This runs the same policy in a docker exec shell and then calls the GPU directly (untested sketch):

   docker exec -u 65534:65534 -w /PeTTa/repos/Omega omega python3 -c '
   import importlib.util, os, sys
   sys.path.insert(0, "/PeTTa/repos/Omega")
   s = importlib.util.spec_from_file_location("policy", "/PeTTa/repos/Omega/profile/policy.py")
   m = importlib.util.module_from_spec(s); s.loader.exec_module(m)
   m.apply_security_policy("/PeTTa/repos/Omega/profile/policy.yaml")
   os.execvp("sh", ["sh", "-c", "ls /dev; nvidia-smi -L"])'

   If this fails and plain docker exec … nvidia-smi -L succeeds, the policy is the cause. You can then add paths (e.g. /sys) one at a time to find the missing one.

4. Further blockers for actually running CUDA/JAX experiments:

   • the image installs CPU-only torch (Dockerfile L56-L60);
   • /tmp is a 64 MB tmpfs (scripts/omega L804);
   • shell-skill commands time out after 5 s (src/skills.pl L10).

   Even with the device reachable, these would stop a long GPU job.

[vsbogd]

@paul-v-snet was you intention keeping minimal required permissions for /proc access? While I totally agree with such intention I would suggest do not introduce the new categories when existing category (read_write) doesn't impose know security risk. The original intention behind this configuration was to keep policy.yaml compatible with OpenShell format. In OpenShell I believe one should replace both device_access and read_write_files by read_write category. But if it is not flexible enough (I mean there is security risk in using them) then I would suggest moving to the most flexible one, to the something which allows describing policy in detail (for example see https://github.com/landlock-lsm/landlockconfig).

I would suggest adding IOCTL_DEV into READ_WRITE_FILE_ACCESS permission set and move devices under read_write.