Xid errors

An Xid is an error code the NVIDIA kernel driver writes to the kernel log when a GPU faults. This page lists every Xid number reported on CMP 170HX cards, with the cause and the fix as the person who hit it reported them. NVIDIA's own definitions are in its Xid documentation; the meanings below are what people saw on these cards. For problems that do not print an Xid, start at Troubleshooting.

Reading the log first

Xids appear in sudo dmesg as lines like NVRM: Xid (PCI:0000:c1:00): 48, .... The PCI address names the card that faulted.
Source: @ichundes, 2026-10-07
On a multi-card host, prove which physical card is at fault before changing anything: swap two devices in CUDA_VISIBLE_DEVICES and check whether the fault follows the card or stays with the pipeline stage.
Source: repo github.com/arvkmr/vllm-170hx-glm5 by arvkmr on GitHub, posted by @reala7096, 2026-10-08
Every card keeps an InfoROM log of past Xid events with timestamps. blackbox.py from gpcprobe dumps it (the DEM record is a FIFO of Xid events), so you can read a card's history after a reboot or before you buy it.
Source: repo github.com/matstatman/gpcprobe by matstatman on GitHub, posted by @matstatman, 2026-10-20
Source: @matstatman, 2026-10-20
Keep ECC enabled while you tune. With ECC on, an unstable card crashes the engine with an Xid instead of silently producing garbage. See Memory errors and ECC.
Source: @ichundes, 2026-10-07
Several Xids leave the card unusable until power is removed. A warm reboot is often not enough; see Recovery.

Xid table

Xid	What the log said (as posted)	Reported cause	What fixed it	Cards	Status	Source
1	`Xid 1 GSP kernel panic / illegal instruction` at init; elsewhere "GSP task exception"	(a) An unlocked 8 GB card (v0.5 with ECC) ran 25 h clean, then the first CUDA context hit Xid 119 then Xid 154; after that every boot failed GSP init with Xid 1, on the stock driver, on a second host and after a full power-off. Both hosts used the same eGPU dock. (b) A card with bad HBM crashed its GSP under load with Xid 119, Xid 1 and Xid 154.	(a) Not resolved in the thread. Suggested: unpowered reset, check InfoROM row-remap records, test in a native PCIe slot; another member had the same symptom from a broken PCIe slot. (b) Full power-off of about 30 s; a warm reboot was not enough.	8 GB	Reported	Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09 Source: @mistyashhole, 2026-10-10 Source: @matstatman, 2026-10-10
13	`Xid 13, pid=..., name=VLLM::Worker, Graphics Exception: ... Class 0000c6c0`	(a) Mailbox P2P behind a PLX switch, in tensor parallel. (b) Follows Xid 48 and Xid 172 on a card with too much undervolt. (c) SM clock offset too high: +300 and +350 pass benchmarks, then fault under a long serving soak on weak SMs (GPC3/TPC6 on the reference card). Under a 200 W cap, +280 and +290 threw Xid 13 then Xid 43.	(a) Avoid PLX mailbox P2P for TP, or run PP or P2P off. (b) Reduce the voltage offset by 50. (c) Use +200 or +250; those were clean in the same soak.	both	Reported	Source: repo github.com/cachenetics/170tune by cachenetics on GitHub, posted by @rh3dai, 2026-10-04 Source: repo github.com/vektorprime/170tune by vektorprime on GitHub, posted by @rh3dai, 2026-10-04 Source: @ichundes, 2026-10-07 Source: @ob88594, 2026-10-07
26	none posted	Asked about, not seen: a used ex-mining card showed "software glitches" in its seller's miner. One member suggested an IRQ conflict with another device and asked whether it was Xid 26. No log was posted.	Unknown.	unknown	Theory	Source: @endsjeremy, 2026-10-19
31	MMU fault, `FAULT_INFO_TYPE_REGION_VIOLATION`	(a) An unstable undervolt. (b) An allocation past the usable top of the unlocked window (fault at 63.86 GiB). (c) cmpunlocker v0.2 handed out a protected 141 MB region at the top of memory; long prompts then crashed the second card. (d) The late-PMA path, which publishes the reserved WPR/GSP region: Xid 31, later Xid 154. (e) A host out-of-memory during an eager model load, which preceded a fatal Xid 154. (f) A late-PMA build that registers the GSP WPR2 region into the memory pool: Xid 31 wedges under allocation churn. (g) NVIDIA's UVM managed-memory test run across CMP cards: Xid 31, then Xid 154 on every GPU.	(a) Back off the undervolt. (b) Offload one fewer layer to that card; the card needed a full reboot (fix not confirmed applied). (c) Use cmpunlocker v0.5 or newer. (d) asm64-hooligan's fork keeps late-PMA off by default; on 40 GB cards the cause was traced to cmpunlocker's own late-pma.patch (PR #32). (e) Size the load before running it on every card; a VM reboot cleared it. (f) On any full-VRAM Xid 31, check first that the build leaves the top reserved region (WPR2) reserved; some forks clamp the pool at 62 GiB. (g) A cold power cycle.	both	Reported	Source: repo github.com/asm64-hooligan/cmpunlocker by asm64-hooligan on GitHub, posted by @rh3dai, 2026-10-04 Source: repo github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx by mannydelrio on GitHub, posted by @mrmannyatx, 2026-10-06 Source: repo github.com/ahnguyen17/ga100-fuse-re by ahnguyen17 on GitHub, posted by @exile_1ck, 2026-10-07 Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09 Source: @wakeupmbrc1, 2026-10-12 Source: @darkmaniac7, 2026-10-19 (lab notes) Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20 Source: repo github.com/ZombieLurker/cmpunlocker by ZombieLurker on GitHub Source: repo github.com/amoghmunikote/cmpunlocker by amoghmunikote on GitHub
43	Xid 43	(a) After Xid 13 at SM offset +280/+290 under a 200 W cap. (b) One card in a 4-card host during a compute wedge; the driver re-initialized the GPU and the card stayed on the bus.	(a) Lower the offset. (b) None needed: the card passed the full correctness gate afterwards with results identical to the other three cards. club-170hx treats this Xid as note-and-continue.	8 GB	Reported	Source: repo github.com/vektorprime/170tune by vektorprime on GitHub, posted by @rh3dai, 2026-10-04 Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
45	Xid 45	(a) Changing the HBM divider (NDIV) while a CUDA context is active: the GPU wedges with Xid 45 and Xid 119. (b) SIGKILL on a running CUDA verification kernel. (c) On a 10 GB card pushed to 80 GB, as part of a chain: CUDA error 719, then Xid 45, then Xid 154.	(a) Power cycle. Never change NDIV with a CUDA context open. (b) Run memory checkers in the foreground and stop them with Ctrl-C between kernel launches, not kill -9. (c) No fix; see Memory errors and ECC.	both	Reported	Source: repo github.com/cachenetics/170tune by cachenetics on GitHub, posted by @rh3dai, 2026-10-04 Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
48	`Xid 48, An uncorrectable ECC error has been detected on GPU in GPC0 GCC`	Too much undervolt (V/F offset) on one card. Logged together with Xid 172 and followed by Xid 13, which killed the vLLM worker. Both people who hit it had passed 170tune's stability and workload tests at that offset.	Lower the voltage offset by 50. A second member stopped once-or-twice-a-day gibberish by lowering one card from +250 to +150.	unknown	Proven	Source: @ichundes, 2026-10-07 Source: @cibernox, 2026-10-07
62	PMU halt	A card with bad HBM, under load. nvidia-smi then showed `[GPU requires reset]` or `[N/A]`.	Full power-off of about 30 s. The same card later proved to have a load-transient memory fault; see Memory errors and ECC.	8 GB	Reported	Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09
79	"GPU has fallen off the bus"	(a) One card at the stock 250 W limit fell off three times under sustained load; nvidia-smi showed up to 333 W against the 250 W limit during prefill. (b) Forcing 8 GT/s (Gen3) on a card whose Gen3 fuse is still set. (c) A Gen3 retrain on a card with a damaged PCIe controller. (d) A card that does not come back from the Gen2 retrain.	(a) `nvidia-smi -pl 180`, re-applied at every boot by a systemd unit. (b, c, d) Only a true cold power cycle brought the card back: function-level reset, secondary bus reset, runtime-power changes and remove/rescan all failed. 30 s with main power removed worked.	both	Reported	Source: repo github.com/asm64-hooligan/cmpunlocker by asm64-hooligan on GitHub, posted by @rh3dai, 2026-10-04 Source: repo github.com/Zanooda/deepseek-v4.1-flash-cmp170hx by Zanooda on GitHub, posted by @gnasherrr, 2026-10-17 Source: repo github.com/lixiangwuxian/170hx_tips by lixiangwuxian on GitHub, posted by @lxtend, 2026-10-19 Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
109	`Xid 109 CTX SWITCH TIMEOUT`	A 10 GB card unlocked to all 128 SMs, where every extra GPC was marked defective. A simple int8 workload ran at 250 W, then crashed with Xid 109; the miner did not compute at all (power stuck near 60 W).	Disable the defective-marked GPCs one at a time until the card is stable, then re-enable earlier ones to find the boundary. Disabling GPC1 fixed it.	10 GB	Reported	Source: @lxtend, 2026-10-17 Source: @wakeupmbrc1, 2026-10-17
119	GSP RPC timeout. Two forms: 60 s waiting on function 4097 `GSP_INIT_DONE`, and 6 s on function 103 `GSP_RM_ALLOC`	(a) 60 s form: GSP never finished booting, usually after a derailed unlock attempt; the next load then reports "WPR2 already up". (b) 6 s form: the card boots, then every nvidia-smi hangs and the Xid repeats about every 6 s. (c) The first CUDA context on a card that had run 25 h, followed by Xid 154 and "GPU reset required" with NaN temperatures (see Xid 1). (d) Changing NDIV under an active CUDA context. (e) A bad-HBM card under load.	Reset the card; the state does not recover in place. Try function-level reset first, then secondary bus reset or a cold power cycle. For (d), power cycle.	both	Reported	Source: repo github.com/cachenetics/170tune by cachenetics on GitHub, posted by @rh3dai, 2026-10-04 Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09 Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09 Source: @mistyashhole, 2026-10-10
154	"GPU recovery action changed", for example to OS Reboot	(a) After Xid 119 on the card in the Xid 1 row. (b) A bad-HBM card under load. (c) A host out-of-memory during a pipeline-parallel load, after Xid 31 on two cards. (d) A 10 GB card over-provisioned to 80 GB, after CUDA memory tests: one CUDA context per attempt. (e) The late-PMA path, after Xid 31. (f) The UVM managed-memory test across CMP cards, after Xid 31.	A module reload did not clear it. A reboot did; on a VM with passthrough, rebooting the guest was enough. For (b), full power-off of about 30 s. For (d), no fix; stay at 40 GB. For (f), a cold power cycle.	both	Reported	Source: repo github.com/asm64-hooligan/cmpunlocker by asm64-hooligan on GitHub, posted by @rh3dai, 2026-10-04 Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09 Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09 Source: @mistyashhole, 2026-10-10 Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
172	`Xid 172, GCC, Uncorrectable SRAM error in GPC 0`	Same event as Xid 48: too much undervolt.	Lower the voltage offset (see Xid 48).	unknown	Reported	Source: @ichundes, 2026-10-07

Errors that are not Xids

These strings come up in the same logs and are often mistaken for a dead card.

What you see	What it was	What to do	Source
`kgspExtractVbiosFromRom_TU102: did not find valid ROM signature`, `RmInitAdapter failed! (0x62:0x25:2119)`, ROM header reads `0xffff` instead of `0xaa55`, nvidia-smi says "No devices were found"	A 10 GB card that never initialized, even on the stock 610.57.04 driver. The VBIOS SPI flash is suspected.	Try reflashing the VBIOS with a hardware programmer; failing that, a new flash chip or a damaged trace. See Recovery.	Source: @youngju10000, 2026-10-08 Source: @ichundes, 2026-10-08
`unexpected WPR2 already up, cannot proceed with booting GSP`, then `RmInitAdapter failed! (0x62:0x40:2028)`	A previous boot or unlock attempt left the WPR2 region programmed. In a VM it appears on the second and later VM starts.	Function-level reset or a cold power cycle. For VMs, restore the GSP boot state before each VM start (the cmpunlocker passthrough service and the Proxmox guide do this).	Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09 Source: repo github.com/anoane/01-proxmox-dual-gpu-passthrough by anoane on GitHub, posted by @logy0, 2026-10-10
`RmInitAdapter` or SEC2 failures inside a Proxmox VM	The VM was booted with OVMF (UEFI) firmware. They look exactly like a failed unlock.	Use SeaBIOS for the VM.	Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09 Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
NVRM `RC watchdog`, then `NV_ERR_RESET_REQUIRED (0x62)`	The HBM controller hard-hung after the tCCD_L memory timing was set to 3. Writing the old value back did not recover it and `nvidia-smi -r` was not supported.	Reboot.	Source: repo github.com/asm64-hooligan/cmpunlocker by asm64-hooligan on GitHub, posted by @rh3dai, 2026-10-04
`N/12 FBPAs failed to lock` in dmesg	The HBM divider (NDIV) is set too high for the card.	Lower NDIV. Values below stock are the documented way to stabilise a card that fails at stock.	Source: repo github.com/asm64-hooligan/cmpunlocker by asm64-hooligan on GitHub, posted by @rh3dai, 2026-10-04
NVRM `_scrubWaitAndSave: Timed out`, CUDA fails in every container while nvidia-smi still lists the cards	Left over from a crashed GPU job.	Stop nvidia-persistenced, `rmmod nvidia_uvm nvidia_drm nvidia_modeset nvidia`, then `modprobe nvidia && modprobe nvidia_uvm` (about 3 min). Re-apply the power cap afterwards: a module reload resets it to 250 W.	Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
`cuInit` returns 999 in every framework while nvidia-smi looks healthy	About 32 zombie CUDA processes left by `kill -9` on live multi-GPU jobs.	Reboot the host; it cannot be fixed inside a container. Stop multi-GPU jobs with their own stop script.	Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09 Source: repo github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx by GNmac-lab on GitHub, posted by @greencandle9753, 2026-10-21
"No CUDA GPUs are available" with nvidia-smi healthy	Usually a container in a `--restart always` crash loop holding the card.	`docker rm -f` the container, confirm 0 MiB in use, relaunch.	Source: repo github.com/vektorprime/qwen38-flash-next-pp2 by vektorprime on GitHub, posted by @vektor_prime, 2026-10-07
`HW Power Brake Slowdown` active in `nvidia-smi -q`, card stuck near 88 W and 1140 MHz	The board asserts PWRBRK# on edge pin B30 (seen on an ASUS Pro WS WRX80E-SAGE).	Kapton tape over pin B30, or a riser that does not route B30.	Source: repo github.com/Zanooda/deepseek-v4.1-flash-cmp170hx by Zanooda on GitHub, posted by @gnasherrr, 2026-10-17
170tune preflight: `PLL PLM 0xBADF2010`, "FBPA privilege masks are closed"	A `0xBADF....` read is a PRI error: the register window is fenced by the unlock build in use.	See Driver and unlock failures.	Source: @rh3dai, 2026-10-04
`krcWatchdog: GPU is probably locked` on another GPU in the same VM, then every new GPU process hangs at exit in `uvm_va_space_destroy`; no Xid	nvidia-open 615.71.09 with cmpunlocker in a VM that also holds RTX PRO 6000 (GB202) cards: any real CUDA context wedges UVM. NVIDIA lists the symptom as a known 615.71.09 issue.	Only a VM reset clears it. Keep the CMP cards off a 615 driver shared with those cards; the lab runs the unlock ported to 580.178.04 instead.	Source: @darkmaniac7, 2026-10-21 (lab notes, darkmaniac7/cmp170hx-esxi) Source: @darkmaniac7, 2026-10-25 (lab notes, darkmaniac7/cmp170hx-esxi)
Dozens of `Booter failed with non-zero error code: 0x31` (or `0xffff`) lines at boot	The unlock's own retry rounds. A search for booter or GSP errors returned 888 lines on a healthy boot.	Nothing; judge the boot by the checks on Troubleshooting	Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09 Source: @darkmaniac7, 2026-10-27 (lab notes)
`failed to fulfil mount request: open /run/nvidia-persistenced/socket` from the NVIDIA container runtime	nvidia-persistenced is not running after a VM power cycle	Start nvidia-persistenced before any GPU container	Source: @darkmaniac7, 2026-10-21 (lab notes)
`kbusEnableStaticBar1Mapping_TU102: Failed to create the static bar1 mapping`, then `RmInitAdapter failed`	`RMForceStaticBar1=1` alone in an ESXi guest: the static map lands at a 512 MiB offset and a 63.6 GiB framebuffer no longer fits the 64 GiB BAR1	Remove the option. Static BAR1 P2P does not work inside the VM anyway; see ESXi	Source: @darkmaniac7, 2026-10-21 (lab notes, darkmaniac7/cmp170hx-esxi)
`NV_ERR_INVALID_REGISTRY_KEY` after adding BAR1 P2P keys	`RMForceStaticBar1` / `RMPcieP2PType` passed through `NVreg_RegistryDwords` on a 610.43.02 GSP: rejected, and its protected memory region stays up	A cold power-off; put those keys where your fork documents them	Source: @darkmaniac7, 2026-10-19 (lab notes) Source: repo github.com/LingzheZhao/cmpunlocker by LingzheZhao on GitHub
NVRM `Out of memory [NV_ERR_NO_MEMORY]` from `system_mem.c` while a large multi-GPU job boots	Host memory fragmentation (large allocations failing), not a GPU or container limit; the engine recovered	`echo 1 > /proc/sys/vm/compact_memory` before the first boot	Source: @darkmaniac7, 2026-10-19 (lab notes)
Processes in D state with wait channel `os_acquire_rwlock` for under a second while KV cache is allocated	Normal: large allocations queue on the driver-wide lock	Nothing. A real UVM wedge holds D states in `uvm_va_space_destroy` or `uvm_gpu_release` for minutes; see Recovery	Source: @darkmaniac7, 2026-10-19 (lab notes)
GSP booter error `0xffff` and the card no longer initializes, after installing an HBM clock tool lineage	The card's FBPA0 is fused off and the tool writes FBPA0 PLL registers	See the warning	Source: @darkmaniac7, 2026-10-27 (lab notes)

Related pages