Skip to content
This repository was archived by the owner on Aug 26, 2026. It is now read-only.

feat(gtx-960m): leave the nvidia driver unloaded so the dGPU stops drawing idle power - #762

Merged
ExtraToast merged 1 commit into
mainfrom
fix/gtx-960m-drop-nvidia
Aug 18, 2026
Merged

feat(gtx-960m): leave the nvidia driver unloaded so the dGPU stops drawing idle power#762
ExtraToast merged 1 commit into
mainfrom
fix/gtx-960m-drop-nvidia

Conversation

@ExtraToast

@ExtraToast ExtraToast commented Aug 18, 2026

Copy link
Copy Markdown
Owner

Stacked on #763.

Problem

The GTX 960M on enschede-gtx-960m-1 has never powered down once — not in 29.5 days of uptime:

runtime_suspended_time = 0 ms
runtime_active_time    = 2552506816 ms   (29.5 days)
power/control = on      runtime_status = active

Meanwhile the card is doing nothing: P8, 44C, 0% utilisation, 3 MiB of 2048 MiB used, no processes.

Why it cannot be fixed with configuration

The driver states the limit directly:

Runtime D3 status:          Disabled by default
GPU Hardware Support:
 Video Memory Self Refresh: Not Supported
 Video Memory Off:          Not Supported

Per the NVIDIA driver documentation, runtime D3 requires a Turing or newer GPU, and NVreg_DynamicPowerManagement: 3 means fine-grained control on Ampere+ notebooks and disabled everywhere else. This GPU is Maxwell 2.0 (GM107).

So the card idles at full power for as long as any driver is bound to it, regardless of which processes hold /dev/nvidia*. Stopping wolf or the device plugin does not change this — that was the wrong diagnosis. There is no supported way to suspend a Maxwell dGPU under the proprietary driver.

Change

The driver is no longer loaded on this host:

  • profiles/gpu-nvidia.nix and modules/services/game-streaming.nix dropped from the host imports (the latter forced boot.kernelModules = [ "nvidia" ... ]).
  • nouveau and the four nvidia* modules blacklisted.
  • A udev rule enables PCI runtime PM on the NVIDIA display controller so the kernel can take the now-unbound device out of D0.

Inventory follows: the node's gpus entry and its nvidia / game-streaming capabilities are removed, host_native no longer lists game-streaming for it, and jellyfin loses temporary_gpu_model: gtx960m.

GPU work stays on enschede-t1000-1 (transcode) and enschede-rx7900xtx-1 (render-compute), which keep their capabilities. nvidia-device-plugin and dcgm-exporter select on capability-nvidia, so both simply stop scheduling here and continue on t1000.

Dead code

Once this host stopped importing it, game-streaming.nix had no importer at all — enschede-rx7900xtx-1 uses the self-contained game-streaming-amd.nix. The module is deleted. Its wolf app-catalog assertions move onto the AMD module, which is the host that actually serves game streaming, and a new test pins the no-GPU invariant for this host.

Consequence that needs a follow-up decision

wolfmanager selects personal-stack/capability-game-streaming. After this change only enschede-rx7900xtx-1 carries that label, and that node has been NotReady for 90 days — so wolfmanager will be unschedulable and the wolf public host will not serve until the AMD node is back. Options are to bring that node up or retire the wolf ingress; retiring a public service is out of scope here.

Validation

  • cd platform/tests && node --test — 58/58 pass (the gtx test is split into three: the no-GPU invariant, the AMD game-streaming host, and continued gpu-nvidia coverage for t1000).
  • All five render scripts run clean, no rendered-output drift.
  • kubectl kustomize platform/cluster/flux/clusters/production builds.
  • nix flake check was not run locally — nix is not installed on this workstation. Platform Validate runs it in CI and must be green before merge.
  • Whether the unbound card actually reaches a low-power state is verifiable only after the rebuild, by re-reading runtime_suspended_time on the host.

@ExtraToast ExtraToast added the enhancement New feature or request label Aug 18, 2026
@ExtraToast ExtraToast self-assigned this Aug 18, 2026
@ExtraToast
ExtraToast force-pushed the fix/pyroscope-to-frankfurt branch from a8e2bb0 to 0416f6e Compare August 18, 2026 20:35
@ExtraToast
ExtraToast force-pushed the fix/gtx-960m-drop-nvidia branch from 84853ad to 9138202 Compare August 18, 2026 20:36
@ExtraToast
ExtraToast changed the base branch from fix/pyroscope-to-frankfurt to main August 18, 2026 20:37
…awing idle power

The GTX 960M on enschede-gtx-960m-1 had never once powered down. The
driver reported:

    Runtime D3 status:  Disabled by default
    Video Memory Off:   Not Supported

and the PCI device confirmed it across the whole boot:

    runtime_suspended_time = 0 ms
    runtime_active_time    = 2552506816 ms   (29.5 days)

Runtime D3 requires a Turing or newer GPU, so on Maxwell the driver
never suspends the card no matter what holds it. The GPU sat at P8, 44C,
0% utilisation, 3 MiB of 2048 MiB used and no compute clients, drawing
idle power continuously and keeping the shared heatpipe warm on a
chassis whose CPU already idles at 53-55C.

With no supported way to suspend the card while a driver is bound, the
driver is no longer loaded on this host. The gpu-nvidia profile and the
game-streaming module are dropped from its imports, the nvidia and
nouveau modules are blacklisted, and a udev rule enables PCI runtime PM
on the display controller so the kernel can take the unbound device out
of D0.

Inventory follows: the node's gpus entry and its nvidia and
game-streaming capabilities are gone, host_native no longer lists
game-streaming for it, and jellyfin loses `temporary_gpu_model:
gtx960m` since the 960M can no longer transcode. GPU work belongs on
enschede-t1000-1 (transcode) and enschede-rx7900xtx-1
(render-compute), both of which keep their capabilities.

game-streaming.nix had no remaining importer once this host dropped it —
enschede-rx7900xtx-1 uses the self-contained game-streaming-amd.nix — so
the dead module is deleted. Its wolf app-catalog coverage moves onto the
AMD module, which is the host that actually serves game streaming, and a
new test pins the no-GPU invariant for this host.
@ExtraToast
ExtraToast force-pushed the fix/gtx-960m-drop-nvidia branch from 9138202 to c793d1c Compare August 18, 2026 20:38
@ExtraToast
ExtraToast merged commit 29e66d1 into main Aug 18, 2026
22 checks passed
@ExtraToast
ExtraToast deleted the fix/gtx-960m-drop-nvidia branch August 18, 2026 20:45
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant