test(driver-podman): no CI coverage for rootless Podman GPU/CDI device injection

[politerealism]

Summary

Podman's own CDI resolution (ExtractCDIDevices/isCDIDevice in pkg/specgen/generate/container_create.go) already handles rootless GPU device injection transparently — confirmed with @elezar. The podman driver's LinuxDevice { path } construction requires no rootless-specific branching; CDI-qualified device strings route identically regardless of rootless/rootful. This is no longer a driver code gap.

Updated Scope

The confirmed, real gap is CI/test coverage — there is no test or CI job exercising GPU device injection combined with rootless Podman:

• Every GPU-related test in driver.rs/container.rs constructs the driver with rootless: false.
• .github/workflows/e2e-gpu-test.yaml ("E2E Docker GPU") only covers Docker, no Podman variant.
• The README's rootless-adaptations section doesn't mention GPU/CDI.

Known Rootless-Specific Prerequisites (host/operator config, not driver code)

Rootless CDI GPU access has documented real-world friction (podman-container-tools/podman#17539), caused by host configuration rather than driver logic:

• NVIDIA Container Toolkit needs no-cgroups = true set for rootless mode.
• The CDI spec (e.g. /etc/cdi/nvidia.yaml) must be readable by the rootless user — /etc/cdi isn't rootless-writable, so it typically needs to live in a user-owned directory instead.

Any CI coverage added here should validate these prerequisites too, not just that CDI strings pass through the driver.

Acceptance Criteria

• A GPU-enabled sandbox is verified working on a real rootless Podman host with GPU hardware, or existing CI runners are confirmed capable of rootless Podman + GPU.
• CI/test coverage added so this doesn't silently regress to "untested" again.
• Prerequisite host configuration (toolkit no-cgroups, CDI spec location) documented for rootless deployments.

Explicitly Out of Scope

• Any change to container.rs's device-injection code path — confirmed not needed.
• Docker driver GPU issues (tracked separately).
• Extending CDI/GPU support to Kubernetes or VM drivers.

[elezar]

@politerealism with CDI there should be no rootless-specific changes needed. For rootless mode podman and the low-level runtime should be adapting the request for this environment.

You are correct that better CI coverage is required. The primary issue is access to the correct hardware. (with that said, we may be able to use the existing runners with rootless podman).

[politerealism]

@elezar Thanks for confirming! That makes sense and lines up with what I saw when tracing it through Podman's source. It looks like pkg/specgen/generate/container_create.go just routes CDI device strings through libpod.WithCDI regardless of whether it's rootless or rootful, so there's no need for us to add a separate code path. I'll drop the "fix the driver" angle from the ticket.

Just a heads-up though is rootless CDI can be a bit of a pain in practice (see podman-container-tools/podman#17539). It's a host/operator config issue rather than a driver issue, but it will definitely bite us in CI if we aren't careful.
There are two main prerequisites we'll need to watch out for:

1. The NVIDIA Container Toolkit needs no-cgroups = true set for rootless mode.
2. The CDI spec has to live somewhere the rootless user can actually read. Since they can't write to /etc/cdi, it usually needs to drop into a user-owned directory instead.

If we end up using the existing runners, we'll need to double check that both of those are set up correctly before we trust a green run IMO. I’ll go ahead and narrow the ticket down to just CI coverage, and I'll start looking into whether our current GPU runners can be configured for rootless Podman.

[elezar]

Let me start by clarifying some things first:

The NVIDIA Container Toolkit needs no-cgroups = true set for rootless mode.

This is not accurate when CDI is used (or should not be). The no-cgroups = true setting was for the nvidia-container-cli which uses a prestart hook to modify the container. This includes setting up cgroups by default and these need to be disabled. The last couple of versions of the NVIDIA Container Toolkit do not use this hook by default and should generate CDI specifications for available devices on installation (and boot). If there's really something missing here, we can work on fixing it in the toolkit and the CDI spec generation logic.

The CDI spec has to live somewhere the rootless user can actually read. Since they can't write to /etc/cdi, it usually needs to drop into a user-owned directory instead.

Let's be careful to separate generating the CDI spec (which requires write permisions) from consuming the spec. In the same way that the NVIDIA GPU Driver is a prerequisite to using compute on NVIDIA GPUs, so too is the NVIDIA Container Toolkit. This should be installed on the system. Once this is done, the CDI specs should be generated and available.

In order to run a container with access to a requested GPU, podman needs to be able to read the spec. The default spec folders that are checked are ["/var/run/cdi", "/etc/cdi"]. The NVIDIA Container Toolkit places the specs in /var/run/cdi by default as these are cleared on boot. (Although not applicable here, some vendors may decide to generate specs at /etc/cdi if they are static and not expected to change across reboots). This means that as long as rootless podman has read access to /var/run/cdi (and/or /etc/cdi) the CDI devices should be resolvable. If this is not the case by default, then we should probably look at updating rootless podman to do this. If a user has explicitly configured podman to read CDI specs from some other location, then it is their responsibility to ensure that the CDI specifications for the available devices have been generated there.

If I recall correctly, the GPU runners that we use have the NVIDIA Container Toolkit already installed and have CDI specifications generated. (The WSL runner is a bit of a special case, but let's keep that out of scope for now). This means that we should be able to implement a podman-specific test there that validates GPU access even if podman is running rootless. We could also consider some nested configuration where we run rootless podman in an outer container in order to test different distibutions. This will require some tooling such as generating CDI specifications in the container, but is realizable.

A final point: CDI is not NVIDIA-specific. This means that we COULD in theory implement our CDI support such that we don't require NVIDIA devices -- at leat for initial sanity checks. We could generate test-specific CDI specifications and request these devices when running a sandbox to ensure that injection works as expected on all supported platforms. That's a little out of scope for this issue, but should help to iron out issues that are not GPU-specific.