NVIDIA Open GPU Kernel Modules Version
595.71.05
Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.
Operating System and Version
ubuntu24.04/22.04
Kernel Release
Linux epyc 6.8.0-134-generic #134-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun 26 18:43:11 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.
Hardware: GPU
8x5060ti+rtx pro 6000
Describe the bug
[ 14.852483] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND
[ 24.302465] NVRM: GPU6 _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 812751949 heartbeat 1899482 heartbeatWithOffsetMs 812745879 diff 6070 timeout 5200
[ 24.302484] NVRM: GPU6 _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 812751949 heartbeat 1899482 heartbeatWithOffsetMs 812745879 diff 6070 timeout 5200
[ 84.300451] NVRM: GPU6 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: f420db7018cc0 >= f420db7018cc0
[ 84.300463] NVRM: GPU6 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
[ 84.300476] NVRM: GPU6 kbusSendSysmembarSingle_GH100: - timeout error waiting for startToken = 0x3 cnt=24105122
[ 84.300480] NVRM: GPU6 kgmmuCheckMemSubsysError_IMPL: Memory Subsystem Error detected. kbusSendSysmembarSingle failed.
[ 84.300482] NVRM: GPU6 _kgspCheckIfMemsysError: Unexpected error when checking for memory subsystem error: 0x72
[ 84.300488] NVRM: GPU6 _kgspLogRpcTimeout: ********************************* GSP Timeout **********************************
To Reproduce
after reboot randomly pick 1~3 gpu timeout
Bug Incidence
Sometimes
nvidia-bug-report.log.gz
nvidia-bug-report.log.gz
More Info
[ 14.650646] NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64 595.71.05 Release Build (root@epyc) Fri Jul 3 11:27:00 AM UTC 2026
[ 14.852483] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND
[ 24.302465] NVRM: GPU6 _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 812751949 heartbeat 1899482 heartbeatWithOffsetMs 812745879 diff 6070 timeout 5200
[ 24.302484] NVRM: GPU6 _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 812751949 heartbeat 1899482 heartbeatWithOffsetMs 812745879 diff 6070 timeout 5200
[ 84.300451] NVRM: GPU6 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: f420db7018cc0 >= f420db7018cc0
[ 84.300463] NVRM: GPU6 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
[ 84.300476] NVRM: GPU6 kbusSendSysmembarSingle_GH100: - timeout error waiting for startToken = 0x3 cnt=24105122
[ 84.300480] NVRM: GPU6 kgmmuCheckMemSubsysError_IMPL: Memory Subsystem Error detected. kbusSendSysmembarSingle failed.
[ 84.300482] NVRM: GPU6 _kgspCheckIfMemsysError: Unexpected error when checking for memory subsystem error: 0x72
[ 84.300488] NVRM: GPU6 _kgspLogRpcTimeout: ********************************* GSP Timeout **********************************
[ 84.300490] NVRM: GPU6 _kgspLogRpcTimeout: Note: Please also check logs above.
[ 84.300579] NVRM: GPU6 nvAssertFailedNoLog: Assertion failed: !pKernelGsp->bPollingForRpcResponse @ kernel_gsp.c:2726
[ 84.300732] NVRM: GPU6 _issueRpcAndWait: rpcRecvPoll failed with status 0x00000040 for fn 76 sequence 1!
[ 84.300741] NVRM: GPU6 nvCheckOkFailedNoLog: Check failed: Generic Error: Invalid state [NV_ERR_INVALID_STATE] (0x00000040) returned from pRmApi->Control(pRmApi, pGpu->hInternalClient, pGpu->hInternalSubdevice, NV2080_CTRL_CMD_INTERNAL_LOG_OOB_XID, ¶ms, sizeof(params)) @ gpu.c:7442
[ 84.300751] NVRM: GPU at PCI:0000:a1:00: GPU-3c312090-433f-77b6-71fa-a4bb8dc90518
[ 84.300758] NVRM: Xid (PCI:0000:a1:00): 175, Timeout after 66s of waiting for RPC response from GPU6 GSP! Expected function 1 (SET_GUEST_SYSTEM_INFO) sequence 0 (0x0 0x0).
[ 84.701372] NVRM: GPU6 _kgspDumpRmState: Failed to get RM state, timeout
[ 84.701419] NVRM: GPU6 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110624, regvalue: 0xbadf5720, error code: Unknown SYS_PRI_ERROR_CODE
[ 84.701426] NVRM: GPU6 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x11062c, regvalue: 0xbadf5720, error code: Unknown SYS_PRI_ERROR_CODE
[ 84.701441] NVRM: GPU6 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x111404, regvalue: 0xbadf5720, error code: Unknown SYS_PRI_ERROR_CODE
[ 84.701447] NVRM: GPU6 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x111408, regvalue: 0xbadf5720, error code: Unknown SYS_PRI_ERROR_CODE
[ 84.701449] NVRM: GPU6 kflcnDumpTracepc_GA102: Trace buffer blocked, skipping.
[ 84.701451] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: riscvPc : 9300dbd6
[ 84.701453] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: riscvCpuctl : 00000180
[ 84.701455] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: riscvIrqmask : 810490d2
[ 84.701456] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: riscvIrqdest : 010800d0
[ 84.701458] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: riscvPrivErrStat : 00000000
[ 84.701459] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: riscvPrivErrInfo : badf2800
[ 84.701461] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: riscvPrivErrAddr : 0000000000000000
[ 84.701462] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: riscvHubErrStat : 00000000
[ 84.701463] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: falconMailbox : 0:00000000 1:00000000
[ 84.701465] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: falconIrqstat : 00009040
[ 84.701466] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: falconIrqmode : ffacfc24
[ 84.701468] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: fbifInstblk : 00000000
[ 84.701469] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: fbifCtl : badf5720
[ 84.701471] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: fbifThrottle : badf5720
[ 84.701472] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: fbifAchkBlk : 0:00000000 1:00000000
[ 84.701474] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: fbifAchkCtl : 0:00000000 1:00000000
[ 84.701475] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: fbifCg1 : 0000000f
[ 84.701480] NVRM: GPU6 kgspDumpMailbox_TU102: GSP: MAILBOX(0) = 0x001CFBDA
[ 84.701484] NVRM: GPU6 kgspDumpMailbox_TU102: GSP: MAILBOX(1) = 0x001CFBDA
[ 84.701488] NVRM: GPU6 kgspDumpMailbox_TU102: GSP: MAILBOX(2) = 0x00000000
[ 84.701492] NVRM: GPU6 kgspDumpMailbox_TU102: GSP: MAILBOX(3) = 0x00000000
[ 84.701497] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: Core is booted.
[ 84.701514] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: RSTAT0 0x0000000000000000
[ 84.701546] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: RSTAT3 0x0000000000000000
[ 84.701560] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: RSTAT4 0x0000000000000000
[ 84.701609] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: PC = 0xffffffff9300dbc2
[ 84.702009] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: ra:0xffffffff9300426a sp:0xffffffffa3003ff0 gp:0xffffffff930043f2 tp:0xffffffffa3013970
[ 84.702011] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: a0:0x0000000000000021 a1:0x00000000000f0001 a2:0x000000002cb99ce0 a3:0x00000007f760fcd0
[ 84.702013] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: a4:0x0000000000000004 a5:0x0000000000000a00 a6:0xffffffff60141044 a7:0x0000000000000001
[ 84.702015] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: s0:0xffffffff60141030 s1:0xffffffff60141030 s2:0x0000000000000000 s3:0x000000000000025d
[ 84.702017] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: s4:0x000000000000025f s5:0x0000000000000002 s6:0x00000007f058793c s7:0x0000000004166cb0
[ 84.702018] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: s8:0x00000007f7e01010 s9:0x0000000000000000 s10:0x0000000020229ef0 s11:0x00000007f05878b0
[ 84.702020] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: t0:0x0000000000000002 t1:0x0000000000000003 t2:0x0000000000000000 t3:0xffffffff6014103c
[ 84.702022] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: t4:0xffffffffa3008d58 t5:0xffffffff60141044 t6:0x000000000100cd50
[ 84.704284] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[100] = 0x0000000200000020
[ 84.704326] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[104] = 0x0000000000000220
[ 84.704344] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[105] = 0xffffffff93000114
[ 84.704367] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[106] = 0x0000000000000003
[ 84.704881] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[140] = 0xffffffffa3013970
[ 84.704898] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[141] = 0x000000000100cd4e
[ 84.704913] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[142] = 0x0000000000000002
[ 84.704928] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[143] = 0x0000000000000000
[ 84.704943] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[144] = 0x0000000000000220
[ 84.705480] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[180] = 0xf000000000000000
[ 84.714729] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5a0] = 0xffffffffffffffff
[ 84.714751] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5a1] = 0xffffffffffffffff
[ 84.714770] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5a2] = 0xffffffffffffffff
[ 84.714907] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5b1] = 0x0000000000000000
[ 84.715044] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5c0] = 0x0000000000000000
[ 84.715129] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5c9] = 0x0000000000800000
[ 84.715144] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5ca] = 0x000000000000005c
[ 84.715159] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5cb] = 0x000000000b8000c1
[ 84.715174] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5cc] = 0x94000010dddd0000
[ 84.715189] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5cd] = 0x0000000000000000
[ 84.715204] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5ce] = 0x0000000000010000
[ 84.715219] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5cf] = 0x00000000000c0003
[ 84.715234] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5d0] = 0x0000000000000001
[ 84.715249] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5d1] = 0x2000000114406000
[ 84.715264] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5d2] = 0x4f7eef55df2eefed
[ 84.715281] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5d3] = 0xfffffffffffffffd
[ 84.715296] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5d4] = 0x0000000000000000
[ 84.715312] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5d5] = 0x0000000000000000
[ 84.715335] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5d7] = 0x0000000000000505
[ 84.715365] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5d9] = 0x0000000000000011
[ 84.715380] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5da] = 0x0000000000000000
[ 84.715412] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5dd] = 0x0000000000000000
[ 84.715620] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5f4] = 0x0000000000000000
[ 84.715636] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5f5] = 0x0000000000000000
[ 84.715677] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5f9] = 0x0000000000000000
[ 84.720255] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[803] = 0x0000000000000001
[ 84.721547] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[895] = 0x0000000000000000
[ 84.721573] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[897] = 0x0000000000000005
[ 84.721596] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[899] = 0x0000000000000001
[ 84.721611] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[89a] = 0x0000000000000000
[ 84.721843] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[8b4] = 0x0000000000000000
[ 84.721858] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[8b5] = 0x0000000000000000
[ 84.729237] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[c00] = 0x00000021f9147b27
[ 84.729254] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[c01] = 0x000001c9bd8d7a40
[ 84.729269] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[c02] = 0x000000060326623e
[ 84.738205] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: [WARN] unwind retrieved zero values :(
[ 84.738288] NVRM: GPU6 GSP RPC buffer contains function 76 (GSP_RM_CONTROL) sequence 1 and data 0x0000000020800a56 0x000000000000005c.
[ 84.738302] NVRM: GPU6 RPC history (CPU -> GSP):
[ 84.738306] NVRM: entry function sequence data0 data1 ts_start ts_end duration actively_polling
[ 84.738309] NVRM: 0 76 GSP_RM_CONTROL 1 0x0000000020800a56 0x000000000000005c 0x000655d53f648ee2 0x0000000000000000 y
[ 84.738318] NVRM: -1 1 SET_GUEST_SYSTEM_INFO 0 0x0000000000000000 0x0000000000000000 0x000655d53b6f002a 0x0000000000000000
[ 84.738325] NVRM: -2 73 SET_REGISTRY 0 0x0000000000000000 0x0000000000000000 0x000655d53b47bfc0 0x000655d53b587765 1095ms
[ 84.738332] NVRM: -3 72 GSP_SET_SYSTEM_INFO 0 0x0000000000000000 0x0000000000000000 0x000655d53b47bfbe 0x000655d53b587765 1095ms
[ 84.738339] NVRM: GPU6 RPC event history (CPU <- GSP):
[ 84.738342] NVRM: entry function sequence data0 data1 ts_start ts_end duration during_incomplete_rpc
[ 84.738345] NVRM: 0 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000000 0x0000000000000000 0x000655d53b56deca 0x000655d53b56deca
[ 84.738366] NVRM: -1 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000001 0x0000000000000000 0x000655d53b56ce62 0x000655d53b56ce62
[ 84.738373] NVRM: -2 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000000 0x0000000000000000 0x000655d53b56cd20 0x000655d53b56cd20
[ 84.738379] NVRM: -3 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000001 0x0000000000000000 0x000655d53b56c990 0x000655d53b56c992 2us
[ 84.738385] NVRM: -4 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000000 0x0000000000000000 0x000655d53b555cee 0x000655d53b555cee
[ 84.738390] NVRM: -5 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000001 0x0000000000000000 0x000655d53b555c69 0x000655d53b555c6a 1us
[ 84.738396] NVRM: -6 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000000 0x0000000000000000 0x000655d53b554a70 0x000655d53b554a70
[ 84.738402] NVRM: -7 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000001 0x0000000000000000 0x000655d53b553abe 0x000655d53b553abe
[ 84.738410] CPU: 32 PID: 2570 Comm: nv_open_q Tainted: G OE 6.8.0-134-generic #134-Ubuntu
[ 84.738418] Hardware name: Giga Computing MZ73-LM1-000/MZ73-LM1-000, BIOS R24_F44 03/31/2026
[ 84.738422] Call Trace:
[ 84.738428]
[ 84.738436] dump_stack_lvl+0x76/0xa0
[ 84.738451] dump_stack+0x10/0x20
[ 84.738461] os_dump_stack+0xe/0x20 [nvidia]
[ 84.738711] _kgspRpcRecvPoll+0xd52/0x1260 [nvidia]
[ 84.739107] ? os_get_system_time+0x3e/0x90 [nvidia]
[ 84.739341] _issueRpcAndWait+0xe9/0x970 [nvidia]
[ 84.739651] ? srso_alias_return_thunk+0x5/0xfbef5
[ 84.739662] ? osGetCurrentThread+0x26/0x60 [nvidia]
[ 84.740033] ? _raw_spin_lock_irqsave+0xe/0x20
[ 84.740044] ? srso_alias_return_thunk+0x5/0xfbef5
[ 84.740050] ? os_mem_copy+0x10/0x20 [nvidia]
[ 84.740271] ? srso_alias_return_thunk+0x5/0xfbef5
[ 84.740281] RmRpcSetGuestSystemInfo+0x1e8/0x4c0 [nvidia]
[ 84.740557] kgspInitRm_IMPL+0x1ab2/0x23d0 [nvidia]
[ 84.740893] RmInitAdapter+0x1216/0x21c0 [nvidia]
[ 84.741252] ? _raw_spin_lock_irqsave+0xe/0x20
[ 84.741264] ? srso_alias_return_thunk+0x5/0xfbef5
[ 84.741271] ? os_release_mutex+0xe/0x20 [nvidia]
[ 84.741492] ? srso_alias_return_thunk+0x5/0xfbef5
[ 84.741501] rm_init_adapter+0x135/0x140 [nvidia]
[ 84.741801] nv_start_device+0x3f2/0x7c0 [nvidia]
[ 84.742016] nv_open_device+0xa3/0x2d0 [nvidia]
[ 84.742226] nvidia_open_deferred+0x39/0xf0 [nvidia]
[ 84.742436] _main_loop+0x7f/0x140 [nvidia]
[ 84.742662] ? __pfx__main_loop+0x10/0x10 [nvidia]
[ 84.742875] kthread+0xef/0x120
[ 84.742884] ? __pfx_kthread+0x10/0x10
[ 84.742890] ret_from_fork+0x44/0x70
[ 84.742899] ? __pfx_kthread+0x10/0x10
[ 84.742904] ret_from_fork_asm+0x1b/0x30
[ 84.742918]
[ 84.742952] NVRM: GPU6 _kgspLogRpcTimeout: ********************************************************************************
[ 84.742959] NVRM: GPU6 nvAssertFailedNoLog: Assertion failed: GPU marked for reset @ kernel_gsp.c:2611
[ 84.743137] NVRM: GPU6 _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 812812391 heartbeat 1899482 heartbeatWithOffsetMs 812745879 diff 66512 timeout 5200
[ 84.743143] NVRM: GPU6 _kgspRpcRecvPoll: GSP RM heartbeat timed out
[ 84.743155] NVRM: GPU6 _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 812812391 heartbeat 1899482 heartbeatWithOffsetMs 812745879 diff 66512 timeout 5200
[ 84.743158] NVRM: GPU6 _kgspRpcRecvPoll: LibOS heartbeat timed out
[ 84.743164] NVRM: GPU6 _issueRpcAndWait: rpcRecvPoll timedout for fn 1 sequence 0!
[ 84.743169] NVRM: GPU6 kgspInitRm_IMPL: SET_GUEST_SYSTEM_INFO failed: 0x65
[ 84.743248] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 84.744455] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x65:2168)
[ 84.744468] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 85.865742] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 85.865752] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 85.865785] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 85.866356] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 85.868437] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 85.944690] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 85.944699] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 85.944729] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 85.945287] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 85.947074] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 86.028582] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 86.028593] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 86.028624] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 86.029172] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 86.031624] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 86.123583] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 86.123591] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 86.123620] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 86.124174] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 86.126227] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 86.206253] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 86.206263] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 86.206294] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 86.206828] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 86.208604] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 86.335332] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64 595.71.05 Release Build (dvs-builder@U22-I3-G08-03-1) Fri Apr 24 06:28:59 UTC 2026
[ 86.678705] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 86.678714] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 86.678743] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 86.679290] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 86.681360] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 86.838195] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 86.838203] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 86.838233] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 86.838781] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 86.840507] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 86.940974] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 86.940984] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 86.941012] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 86.941624] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 86.943480] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 87.024746] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 87.024753] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 87.024782] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 87.025449] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 87.027560] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 87.123621] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 87.123630] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 87.123662] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 87.124226] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 87.126310] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 87.655948] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 87.655956] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 87.655987] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 87.656584] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 87.658318] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 87.778338] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 87.778346] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 87.778377] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 87.779033] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 87.781050] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 87.860920] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 87.860929] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 87.860958] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 87.861557] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 87.863592] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 87.986679] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 87.986686] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 87.986715] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 87.987300] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 87.989045] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 88.192892] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 88.192899] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 88.192929] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 88.193504] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 88.195291] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 88.582855] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 88.582864] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 88.582894] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 88.583480] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 88.585226] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 88.669319] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 88.669328] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 88.669358] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 88.669932] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 88.671698] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 88.764090] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 88.764099] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 88.764129] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 88.764791] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 88.766818] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 88.867725] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 88.867734] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 88.867763] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 88.868334] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 88.870169] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 89.047984] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 89.047992] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 89.048022] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 89.048609] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 89.050403] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 89.316441] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 89.316450] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 89.316482] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 89.317151] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 89.318849] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 89.625141] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 89.625148] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 89.625178] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 89.625764] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 89.627619] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 89.725580] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 89.725588] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 89.725620] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 89.726198] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 89.727963] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 90.133325] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 90.133333] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 90.133363] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 90.134028] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 90.135757] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 90.711608] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 90.711618] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 90.711647] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 90.712327] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 90.714142] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 90.803159] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 90.803168] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 90.803197] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 90.803861] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 90.805753] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 90.886147] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 90.886154] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 90.886183] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 90.886713] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 90.888472] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 91.917871] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 91.917879] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 91.917919] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 91.918536] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 91.920266] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 101.208397] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 101.208406] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 101.208439] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 101.209077] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 101.210899] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 101.287612] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 101.287620] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 101.287650] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 101.288303] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 101.290263] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 101.365561] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 101.365570] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 101.365599] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 101.366268] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 101.368230] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 101.462392] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 101.462399] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 101.462432] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 101.463159] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 101.464993] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 101.546280] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 101.546289] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 101.546320] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 101.546999] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 101.549095] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 101.638121] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 101.638130] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 101.638160] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 101.638866] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 101.640659] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 101.725763] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 101.725771] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 101.725799] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 101.726418] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 101.728155] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 102.114039] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 102.114048] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 102.114079] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 102.114787] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 102.116594] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 113.875551] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 113.875559] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 113.875593] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 113.876278] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 113.878092] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 114.074288] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 114.074296] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 114.074326] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 114.074941] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 114.077050] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 114.283299] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 114.283307] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 114.283337] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 114.283993] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 114.286097] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 114.487786] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 114.487795] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 114.487827] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 114.488563] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 114.490526] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 114.582636] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 114.582644] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 114.582673] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 114.583388] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 114.585422] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 114.857414] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 114.857423] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 114.857453] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 114.858149] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 114.859956] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 115.083711] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 115.083720] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 115.083750] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 115.084348] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 115.086406] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 115.288253] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 115.288261] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 115.288295] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 115.288895] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 115.290671] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 115.624488] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 115.624496] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 115.624526] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 115.625109] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 115.626867] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 116.266598] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 116.266606] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 116.266636] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 116.267339] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 116.269503] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 120.603426] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 120.603446] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 120.603478] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 120.604117] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 120.605852] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 120.918479] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 120.918487] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 120.918516] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 120.919160] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 120.921039] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 121.106800] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 121.106809] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 121.106839] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 121.107557] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 121.109737] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 121.301274] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 121.301283] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 121.301312] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 121.302100] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 121.304068] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 121.507344] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 121.507352] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 121.507381] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 121.508040] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 121.509876] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 122.499239] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 122.499248] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 122.499278] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 122.499947] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 122.501793] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 126.887325] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 126.887335] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 126.887367] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 126.888073] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 126.889856] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 127.498509] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 127.498519] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 127.498549] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 127.499308] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 127.501267] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 127.680416] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 127.680424] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 127.680455] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 127.681207] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 127.683067] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 127.849897] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 127.849906] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 127.849936] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 127.850566] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 127.852333] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 128.018247] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 128.018255] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 128.018285] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 128.019009] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 128.020880] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 128.951076] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 128.951084] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 128.951114] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 128.951763] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 128.953598] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 129.705256] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 129.705265] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 129.705304] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 129.705964] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 129.707712] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 129.900957] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 129.900966] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 129.900997] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 129.901655] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 129.903460] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 130.236465] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 130.236474] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 130.236506] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 130.237254] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 130.238980] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 130.596285] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 130.596294] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 130.596324] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 130.597060] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 130.598990] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 133.582394] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 133.582402] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 133.582432] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 133.583151] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 133.584925] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 134.221144] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 134.221152] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 134.221183] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 134.222108] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 134.225731] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 134.421131] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 134.421139] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 134.421167] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 134.421991] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 134.424896] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 134.609718] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 134.609726] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 134.609756] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 134.610561] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 134.614013] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 134.782010] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 134.782017] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 134.782050] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 134.782837] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 134.785317] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 135.786257] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 135.786264] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 135.786293] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 135.787116] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 135.789794] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 140.517939] NVRM: GPU6 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x118f54, regvalue: 0xbadcfda0, error code: Unknown SYS_PRI_ERROR_CODE
[ 140.519800] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 140.519804] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 140.519838] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 140.520582] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 140.522248] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 141.194636] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 141.194644] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 141.194682] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 141.195517] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 141.197984] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 141.387755] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 141.387763] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 141.387791] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 141.388581] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 141.391440] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 141.576506] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 141.576514] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 141.576542] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 141.577311] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 141.579605] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 141.752242] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 141.752250] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 141.752279] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 141.753047] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 141.755353] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 142.784260] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 142.784267] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 142.784296] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 142.785106] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 142.788151] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 147.692169] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 147.692176] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 147.692206] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 147.693019] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 147.694822] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 180.198598] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 180.198608] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 180.198676] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 180.199701] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 180.202460] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 195.123951] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 195.123967] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 195.124038] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 195.125769] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 195.132226] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 216.630913] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 216.630920] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 216.630949] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 216.631596] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 216.635106] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
(base) root@epyc:~#
NVIDIA Open GPU Kernel Modules Version
595.71.05
Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.
Operating System and Version
ubuntu24.04/22.04
Kernel Release
Linux epyc 6.8.0-134-generic #134-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun 26 18:43:11 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.
Hardware: GPU
8x5060ti+rtx pro 6000
Describe the bug
[ 14.852483] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND
[ 24.302465] NVRM: GPU6 _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 812751949 heartbeat 1899482 heartbeatWithOffsetMs 812745879 diff 6070 timeout 5200
[ 24.302484] NVRM: GPU6 _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 812751949 heartbeat 1899482 heartbeatWithOffsetMs 812745879 diff 6070 timeout 5200
[ 84.300451] NVRM: GPU6 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: f420db7018cc0 >= f420db7018cc0
[ 84.300463] NVRM: GPU6 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
[ 84.300476] NVRM: GPU6 kbusSendSysmembarSingle_GH100: - timeout error waiting for startToken = 0x3 cnt=24105122
[ 84.300480] NVRM: GPU6 kgmmuCheckMemSubsysError_IMPL: Memory Subsystem Error detected. kbusSendSysmembarSingle failed.
[ 84.300482] NVRM: GPU6 _kgspCheckIfMemsysError: Unexpected error when checking for memory subsystem error: 0x72
[ 84.300488] NVRM: GPU6 _kgspLogRpcTimeout: ********************************* GSP Timeout **********************************
To Reproduce
after reboot randomly pick 1~3 gpu timeout
Bug Incidence
Sometimes
nvidia-bug-report.log.gz
nvidia-bug-report.log.gz
More Info
[ 14.650646] NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64 595.71.05 Release Build (root@epyc) Fri Jul 3 11:27:00 AM UTC 2026
[ 14.852483] workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND
[ 24.302465] NVRM: GPU6 _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 812751949 heartbeat 1899482 heartbeatWithOffsetMs 812745879 diff 6070 timeout 5200
[ 24.302484] NVRM: GPU6 _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 812751949 heartbeat 1899482 heartbeatWithOffsetMs 812745879 diff 6070 timeout 5200
[ 84.300451] NVRM: GPU6 _threadNodeCheckTimeout: _threadNodeCheckTimeout: currentTime: f420db7018cc0 >= f420db7018cc0
[ 84.300463] NVRM: GPU6 _threadNodeCheckTimeout: _threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
[ 84.300476] NVRM: GPU6 kbusSendSysmembarSingle_GH100: - timeout error waiting for startToken = 0x3 cnt=24105122
[ 84.300480] NVRM: GPU6 kgmmuCheckMemSubsysError_IMPL: Memory Subsystem Error detected. kbusSendSysmembarSingle failed.
[ 84.300482] NVRM: GPU6 _kgspCheckIfMemsysError: Unexpected error when checking for memory subsystem error: 0x72
[ 84.300488] NVRM: GPU6 _kgspLogRpcTimeout: ********************************* GSP Timeout **********************************
[ 84.300490] NVRM: GPU6 _kgspLogRpcTimeout: Note: Please also check logs above.
[ 84.300579] NVRM: GPU6 nvAssertFailedNoLog: Assertion failed: !pKernelGsp->bPollingForRpcResponse @ kernel_gsp.c:2726
[ 84.300732] NVRM: GPU6 _issueRpcAndWait: rpcRecvPoll failed with status 0x00000040 for fn 76 sequence 1!
[ 84.300741] NVRM: GPU6 nvCheckOkFailedNoLog: Check failed: Generic Error: Invalid state [NV_ERR_INVALID_STATE] (0x00000040) returned from pRmApi->Control(pRmApi, pGpu->hInternalClient, pGpu->hInternalSubdevice, NV2080_CTRL_CMD_INTERNAL_LOG_OOB_XID, ¶ms, sizeof(params)) @ gpu.c:7442
[ 84.300751] NVRM: GPU at PCI:0000:a1:00: GPU-3c312090-433f-77b6-71fa-a4bb8dc90518
[ 84.300758] NVRM: Xid (PCI:0000:a1:00): 175, Timeout after 66s of waiting for RPC response from GPU6 GSP! Expected function 1 (SET_GUEST_SYSTEM_INFO) sequence 0 (0x0 0x0).
[ 84.701372] NVRM: GPU6 _kgspDumpRmState: Failed to get RM state, timeout
[ 84.701419] NVRM: GPU6 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110624, regvalue: 0xbadf5720, error code: Unknown SYS_PRI_ERROR_CODE
[ 84.701426] NVRM: GPU6 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x11062c, regvalue: 0xbadf5720, error code: Unknown SYS_PRI_ERROR_CODE
[ 84.701441] NVRM: GPU6 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x111404, regvalue: 0xbadf5720, error code: Unknown SYS_PRI_ERROR_CODE
[ 84.701447] NVRM: GPU6 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x111408, regvalue: 0xbadf5720, error code: Unknown SYS_PRI_ERROR_CODE
[ 84.701449] NVRM: GPU6 kflcnDumpTracepc_GA102: Trace buffer blocked, skipping.
[ 84.701451] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: riscvPc : 9300dbd6
[ 84.701453] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: riscvCpuctl : 00000180
[ 84.701455] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: riscvIrqmask : 810490d2
[ 84.701456] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: riscvIrqdest : 010800d0
[ 84.701458] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: riscvPrivErrStat : 00000000
[ 84.701459] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: riscvPrivErrInfo : badf2800
[ 84.701461] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: riscvPrivErrAddr : 0000000000000000
[ 84.701462] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: riscvHubErrStat : 00000000
[ 84.701463] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: falconMailbox : 0:00000000 1:00000000
[ 84.701465] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: falconIrqstat : 00009040
[ 84.701466] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: falconIrqmode : ffacfc24
[ 84.701468] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: fbifInstblk : 00000000
[ 84.701469] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: fbifCtl : badf5720
[ 84.701471] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: fbifThrottle : badf5720
[ 84.701472] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: fbifAchkBlk : 0:00000000 1:00000000
[ 84.701474] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: fbifAchkCtl : 0:00000000 1:00000000
[ 84.701475] NVRM: GPU6 kflcnCoreDumpNondestructive_IMPL: PRI: fbifCg1 : 0000000f
[ 84.701480] NVRM: GPU6 kgspDumpMailbox_TU102: GSP: MAILBOX(0) = 0x001CFBDA
[ 84.701484] NVRM: GPU6 kgspDumpMailbox_TU102: GSP: MAILBOX(1) = 0x001CFBDA
[ 84.701488] NVRM: GPU6 kgspDumpMailbox_TU102: GSP: MAILBOX(2) = 0x00000000
[ 84.701492] NVRM: GPU6 kgspDumpMailbox_TU102: GSP: MAILBOX(3) = 0x00000000
[ 84.701497] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: Core is booted.
[ 84.701514] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: RSTAT0 0x0000000000000000
[ 84.701546] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: RSTAT3 0x0000000000000000
[ 84.701560] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: RSTAT4 0x0000000000000000
[ 84.701609] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: PC = 0xffffffff9300dbc2
[ 84.702009] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: ra:0xffffffff9300426a sp:0xffffffffa3003ff0 gp:0xffffffff930043f2 tp:0xffffffffa3013970
[ 84.702011] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: a0:0x0000000000000021 a1:0x00000000000f0001 a2:0x000000002cb99ce0 a3:0x00000007f760fcd0
[ 84.702013] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: a4:0x0000000000000004 a5:0x0000000000000a00 a6:0xffffffff60141044 a7:0x0000000000000001
[ 84.702015] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: s0:0xffffffff60141030 s1:0xffffffff60141030 s2:0x0000000000000000 s3:0x000000000000025d
[ 84.702017] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: s4:0x000000000000025f s5:0x0000000000000002 s6:0x00000007f058793c s7:0x0000000004166cb0
[ 84.702018] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: s8:0x00000007f7e01010 s9:0x0000000000000000 s10:0x0000000020229ef0 s11:0x00000007f05878b0
[ 84.702020] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: t0:0x0000000000000002 t1:0x0000000000000003 t2:0x0000000000000000 t3:0xffffffff6014103c
[ 84.702022] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: t4:0xffffffffa3008d58 t5:0xffffffff60141044 t6:0x000000000100cd50
[ 84.704284] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[100] = 0x0000000200000020
[ 84.704326] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[104] = 0x0000000000000220
[ 84.704344] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[105] = 0xffffffff93000114
[ 84.704367] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[106] = 0x0000000000000003
[ 84.704881] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[140] = 0xffffffffa3013970
[ 84.704898] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[141] = 0x000000000100cd4e
[ 84.704913] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[142] = 0x0000000000000002
[ 84.704928] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[143] = 0x0000000000000000
[ 84.704943] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[144] = 0x0000000000000220
[ 84.705480] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[180] = 0xf000000000000000
[ 84.714729] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5a0] = 0xffffffffffffffff
[ 84.714751] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5a1] = 0xffffffffffffffff
[ 84.714770] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5a2] = 0xffffffffffffffff
[ 84.714907] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5b1] = 0x0000000000000000
[ 84.715044] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5c0] = 0x0000000000000000
[ 84.715129] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5c9] = 0x0000000000800000
[ 84.715144] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5ca] = 0x000000000000005c
[ 84.715159] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5cb] = 0x000000000b8000c1
[ 84.715174] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5cc] = 0x94000010dddd0000
[ 84.715189] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5cd] = 0x0000000000000000
[ 84.715204] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5ce] = 0x0000000000010000
[ 84.715219] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5cf] = 0x00000000000c0003
[ 84.715234] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5d0] = 0x0000000000000001
[ 84.715249] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5d1] = 0x2000000114406000
[ 84.715264] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5d2] = 0x4f7eef55df2eefed
[ 84.715281] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5d3] = 0xfffffffffffffffd
[ 84.715296] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5d4] = 0x0000000000000000
[ 84.715312] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5d5] = 0x0000000000000000
[ 84.715335] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5d7] = 0x0000000000000505
[ 84.715365] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5d9] = 0x0000000000000011
[ 84.715380] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5da] = 0x0000000000000000
[ 84.715412] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5dd] = 0x0000000000000000
[ 84.715620] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5f4] = 0x0000000000000000
[ 84.715636] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5f5] = 0x0000000000000000
[ 84.715677] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[5f9] = 0x0000000000000000
[ 84.720255] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[803] = 0x0000000000000001
[ 84.721547] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[895] = 0x0000000000000000
[ 84.721573] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[897] = 0x0000000000000005
[ 84.721596] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[899] = 0x0000000000000001
[ 84.721611] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[89a] = 0x0000000000000000
[ 84.721843] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[8b4] = 0x0000000000000000
[ 84.721858] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[8b5] = 0x0000000000000000
[ 84.729237] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[c00] = 0x00000021f9147b27
[ 84.729254] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[c01] = 0x000001c9bd8d7a40
[ 84.729269] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: csr[c02] = 0x000000060326623e
[ 84.738205] NVRM: GPU6 kflcnCoreDumpDestructive_IMPL: ICD: [WARN] unwind retrieved zero values :(
[ 84.738288] NVRM: GPU6 GSP RPC buffer contains function 76 (GSP_RM_CONTROL) sequence 1 and data 0x0000000020800a56 0x000000000000005c.
[ 84.738302] NVRM: GPU6 RPC history (CPU -> GSP):
[ 84.738306] NVRM: entry function sequence data0 data1 ts_start ts_end duration actively_polling
[ 84.738309] NVRM: 0 76 GSP_RM_CONTROL 1 0x0000000020800a56 0x000000000000005c 0x000655d53f648ee2 0x0000000000000000 y
[ 84.738318] NVRM: -1 1 SET_GUEST_SYSTEM_INFO 0 0x0000000000000000 0x0000000000000000 0x000655d53b6f002a 0x0000000000000000
[ 84.738325] NVRM: -2 73 SET_REGISTRY 0 0x0000000000000000 0x0000000000000000 0x000655d53b47bfc0 0x000655d53b587765 1095ms
[ 84.738332] NVRM: -3 72 GSP_SET_SYSTEM_INFO 0 0x0000000000000000 0x0000000000000000 0x000655d53b47bfbe 0x000655d53b587765 1095ms
[ 84.738339] NVRM: GPU6 RPC event history (CPU <- GSP):
[ 84.738342] NVRM: entry function sequence data0 data1 ts_start ts_end duration during_incomplete_rpc
[ 84.738345] NVRM: 0 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000000 0x0000000000000000 0x000655d53b56deca 0x000655d53b56deca
[ 84.738366] NVRM: -1 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000001 0x0000000000000000 0x000655d53b56ce62 0x000655d53b56ce62
[ 84.738373] NVRM: -2 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000000 0x0000000000000000 0x000655d53b56cd20 0x000655d53b56cd20
[ 84.738379] NVRM: -3 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000001 0x0000000000000000 0x000655d53b56c990 0x000655d53b56c992 2us
[ 84.738385] NVRM: -4 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000000 0x0000000000000000 0x000655d53b555cee 0x000655d53b555cee
[ 84.738390] NVRM: -5 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000001 0x0000000000000000 0x000655d53b555c69 0x000655d53b555c6a 1us
[ 84.738396] NVRM: -6 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000000 0x0000000000000000 0x000655d53b554a70 0x000655d53b554a70
[ 84.738402] NVRM: -7 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000001 0x0000000000000000 0x000655d53b553abe 0x000655d53b553abe
[ 84.738410] CPU: 32 PID: 2570 Comm: nv_open_q Tainted: G OE 6.8.0-134-generic #134-Ubuntu
[ 84.738418] Hardware name: Giga Computing MZ73-LM1-000/MZ73-LM1-000, BIOS R24_F44 03/31/2026
[ 84.738422] Call Trace:
[ 84.738428]
[ 84.738436] dump_stack_lvl+0x76/0xa0
[ 84.738451] dump_stack+0x10/0x20
[ 84.738461] os_dump_stack+0xe/0x20 [nvidia]
[ 84.738711] _kgspRpcRecvPoll+0xd52/0x1260 [nvidia]
[ 84.739107] ? os_get_system_time+0x3e/0x90 [nvidia]
[ 84.739341] _issueRpcAndWait+0xe9/0x970 [nvidia]
[ 84.739651] ? srso_alias_return_thunk+0x5/0xfbef5
[ 84.739662] ? osGetCurrentThread+0x26/0x60 [nvidia]
[ 84.740033] ? _raw_spin_lock_irqsave+0xe/0x20
[ 84.740044] ? srso_alias_return_thunk+0x5/0xfbef5
[ 84.740050] ? os_mem_copy+0x10/0x20 [nvidia]
[ 84.740271] ? srso_alias_return_thunk+0x5/0xfbef5
[ 84.740281] RmRpcSetGuestSystemInfo+0x1e8/0x4c0 [nvidia]
[ 84.740557] kgspInitRm_IMPL+0x1ab2/0x23d0 [nvidia]
[ 84.740893] RmInitAdapter+0x1216/0x21c0 [nvidia]
[ 84.741252] ? _raw_spin_lock_irqsave+0xe/0x20
[ 84.741264] ? srso_alias_return_thunk+0x5/0xfbef5
[ 84.741271] ? os_release_mutex+0xe/0x20 [nvidia]
[ 84.741492] ? srso_alias_return_thunk+0x5/0xfbef5
[ 84.741501] rm_init_adapter+0x135/0x140 [nvidia]
[ 84.741801] nv_start_device+0x3f2/0x7c0 [nvidia]
[ 84.742016] nv_open_device+0xa3/0x2d0 [nvidia]
[ 84.742226] nvidia_open_deferred+0x39/0xf0 [nvidia]
[ 84.742436] _main_loop+0x7f/0x140 [nvidia]
[ 84.742662] ? __pfx__main_loop+0x10/0x10 [nvidia]
[ 84.742875] kthread+0xef/0x120
[ 84.742884] ? __pfx_kthread+0x10/0x10
[ 84.742890] ret_from_fork+0x44/0x70
[ 84.742899] ? __pfx_kthread+0x10/0x10
[ 84.742904] ret_from_fork_asm+0x1b/0x30
[ 84.742918]
[ 84.742952] NVRM: GPU6 _kgspLogRpcTimeout: ********************************************************************************
[ 84.742959] NVRM: GPU6 nvAssertFailedNoLog: Assertion failed: GPU marked for reset @ kernel_gsp.c:2611
[ 84.743137] NVRM: GPU6 _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 812812391 heartbeat 1899482 heartbeatWithOffsetMs 812745879 diff 66512 timeout 5200
[ 84.743143] NVRM: GPU6 _kgspRpcRecvPoll: GSP RM heartbeat timed out
[ 84.743155] NVRM: GPU6 _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 812812391 heartbeat 1899482 heartbeatWithOffsetMs 812745879 diff 66512 timeout 5200
[ 84.743158] NVRM: GPU6 _kgspRpcRecvPoll: LibOS heartbeat timed out
[ 84.743164] NVRM: GPU6 _issueRpcAndWait: rpcRecvPoll timedout for fn 1 sequence 0!
[ 84.743169] NVRM: GPU6 kgspInitRm_IMPL: SET_GUEST_SYSTEM_INFO failed: 0x65
[ 84.743248] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 84.744455] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x65:2168)
[ 84.744468] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 85.865742] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 85.865752] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 85.865785] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 85.866356] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 85.868437] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 85.944690] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 85.944699] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 85.944729] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 85.945287] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 85.947074] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 86.028582] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 86.028593] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 86.028624] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 86.029172] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 86.031624] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 86.123583] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 86.123591] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 86.123620] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 86.124174] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 86.126227] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 86.206253] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 86.206263] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 86.206294] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 86.206828] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 86.208604] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 86.335332] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64 595.71.05 Release Build (dvs-builder@U22-I3-G08-03-1) Fri Apr 24 06:28:59 UTC 2026
[ 86.678705] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 86.678714] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 86.678743] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 86.679290] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 86.681360] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 86.838195] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 86.838203] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 86.838233] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 86.838781] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 86.840507] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 86.940974] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 86.940984] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 86.941012] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 86.941624] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 86.943480] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 87.024746] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 87.024753] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 87.024782] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 87.025449] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 87.027560] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 87.123621] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 87.123630] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 87.123662] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 87.124226] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 87.126310] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 87.655948] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 87.655956] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 87.655987] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 87.656584] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 87.658318] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 87.778338] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 87.778346] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 87.778377] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 87.779033] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 87.781050] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 87.860920] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 87.860929] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 87.860958] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 87.861557] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 87.863592] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 87.986679] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 87.986686] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 87.986715] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 87.987300] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 87.989045] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 88.192892] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 88.192899] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 88.192929] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 88.193504] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 88.195291] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 88.582855] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 88.582864] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 88.582894] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 88.583480] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 88.585226] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 88.669319] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 88.669328] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 88.669358] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 88.669932] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 88.671698] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 88.764090] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 88.764099] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 88.764129] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 88.764791] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 88.766818] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 88.867725] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 88.867734] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 88.867763] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 88.868334] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 88.870169] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 89.047984] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 89.047992] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 89.048022] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 89.048609] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 89.050403] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 89.316441] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 89.316450] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 89.316482] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 89.317151] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 89.318849] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 89.625141] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 89.625148] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 89.625178] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 89.625764] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 89.627619] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 89.725580] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 89.725588] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 89.725620] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 89.726198] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 89.727963] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 90.133325] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 90.133333] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 90.133363] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 90.134028] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 90.135757] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 90.711608] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 90.711618] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 90.711647] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 90.712327] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 90.714142] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 90.803159] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 90.803168] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 90.803197] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 90.803861] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 90.805753] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 90.886147] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 90.886154] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 90.886183] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 90.886713] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 90.888472] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 91.917871] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 91.917879] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 91.917919] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 91.918536] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 91.920266] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 101.208397] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 101.208406] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 101.208439] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 101.209077] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 101.210899] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 101.287612] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 101.287620] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 101.287650] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 101.288303] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 101.290263] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 101.365561] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 101.365570] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 101.365599] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 101.366268] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 101.368230] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 101.462392] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 101.462399] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 101.462432] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 101.463159] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 101.464993] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 101.546280] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 101.546289] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 101.546320] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 101.546999] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 101.549095] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 101.638121] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 101.638130] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 101.638160] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 101.638866] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 101.640659] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 101.725763] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 101.725771] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 101.725799] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 101.726418] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 101.728155] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 102.114039] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 102.114048] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 102.114079] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 102.114787] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 102.116594] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 113.875551] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 113.875559] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 113.875593] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 113.876278] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 113.878092] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 114.074288] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 114.074296] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 114.074326] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 114.074941] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 114.077050] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 114.283299] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 114.283307] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 114.283337] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 114.283993] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 114.286097] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 114.487786] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 114.487795] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 114.487827] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 114.488563] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 114.490526] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 114.582636] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 114.582644] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 114.582673] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 114.583388] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 114.585422] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 114.857414] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 114.857423] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 114.857453] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 114.858149] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 114.859956] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 115.083711] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 115.083720] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 115.083750] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 115.084348] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 115.086406] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 115.288253] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 115.288261] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 115.288295] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 115.288895] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 115.290671] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 115.624488] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 115.624496] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 115.624526] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 115.625109] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 115.626867] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 116.266598] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 116.266606] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 116.266636] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 116.267339] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 116.269503] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 120.603426] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 120.603446] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 120.603478] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 120.604117] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 120.605852] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 120.918479] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 120.918487] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 120.918516] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 120.919160] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 120.921039] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 121.106800] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 121.106809] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 121.106839] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 121.107557] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 121.109737] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 121.301274] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 121.301283] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 121.301312] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 121.302100] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 121.304068] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 121.507344] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 121.507352] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 121.507381] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 121.508040] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 121.509876] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 122.499239] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 122.499248] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 122.499278] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 122.499947] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 122.501793] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 126.887325] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 126.887335] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 126.887367] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 126.888073] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 126.889856] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 127.498509] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 127.498519] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 127.498549] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 127.499308] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 127.501267] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 127.680416] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 127.680424] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 127.680455] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 127.681207] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 127.683067] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 127.849897] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 127.849906] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 127.849936] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 127.850566] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 127.852333] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 128.018247] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 128.018255] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 128.018285] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 128.019009] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 128.020880] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 128.951076] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 128.951084] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 128.951114] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 128.951763] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 128.953598] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 129.705256] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 129.705265] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 129.705304] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 129.705964] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 129.707712] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 129.900957] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 129.900966] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 129.900997] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 129.901655] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 129.903460] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 130.236465] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 130.236474] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 130.236506] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 130.237254] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 130.238980] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 130.596285] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 130.596294] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 130.596324] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 130.597060] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 130.598990] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 133.582394] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 133.582402] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 133.582432] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 133.583151] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 133.584925] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 134.221144] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 134.221152] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 134.221183] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 134.222108] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 134.225731] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 134.421131] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 134.421139] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 134.421167] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 134.421991] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 134.424896] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 134.609718] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 134.609726] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 134.609756] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 134.610561] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 134.614013] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 134.782010] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 134.782017] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 134.782050] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 134.782837] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 134.785317] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 135.786257] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 135.786264] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 135.786293] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 135.787116] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 135.789794] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 140.517939] NVRM: GPU6 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x118f54, regvalue: 0xbadcfda0, error code: Unknown SYS_PRI_ERROR_CODE
[ 140.519800] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 140.519804] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 140.519838] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 140.520582] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 140.522248] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 141.194636] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 141.194644] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 141.194682] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 141.195517] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 141.197984] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 141.387755] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 141.387763] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 141.387791] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 141.388581] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 141.391440] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 141.576506] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 141.576514] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 141.576542] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 141.577311] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 141.579605] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 141.752242] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 141.752250] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 141.752279] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 141.753047] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 141.755353] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 142.784260] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 142.784267] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 142.784296] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 142.785106] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 142.788151] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 147.692169] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 147.692176] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 147.692206] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 147.693019] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 147.694822] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 180.198598] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 180.198608] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 180.198676] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 180.199701] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 180.202460] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 195.123951] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 195.123967] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 195.124038] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 195.125769] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 195.132226] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
[ 216.630913] NVRM: GPU6 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 216.630920] NVRM: GPU6 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[ 216.630949] NVRM: GPU6 RmInitAdapter: Cannot initialize GSP firmware RM
[ 216.631596] NVRM: GPU 0000:a1:00.0: RmInitAdapter failed! (0x62:0x40:2168)
[ 216.635106] NVRM: GPU 0000:a1:00.0: rm_init_adapter failed, device minor number 7
(base) root@epyc:~#