https://cmp.wiki/w/index.php?action=history&feed=atom&title=Xid_errorsXid errors - Revision history2026-10-31T06:04:51ZRevision history for this page on the wikiMediaWiki 1.43.11https://cmp.wiki/w/index.php?title=Xid_errors&diff=309&oldid=prevImportBot: Every Xid number reported on CMP 170HX cards in the cmpunlocker Discord (2026-10-04 to 10-28) and in community repos (170tune, club-170hx, cmp170hx-testsuite, Consensus-Protocol, cmpunlocker forks), with cause and fix as reported; credits inline; sources: @ichundes, @reala7096, @matstatman, @boz._., @mistyashhole, @rh3dai, @ob88594, @endsjeremy, @mrmannyatx, @exile_1ck, @rrickk420, @wakeupmbrc1 (+9 more); GitHub Consensus-Protocol/cmp170hx, GNmac-lab/deepseek-v41-flash-1017-c2026-10-30T08:17:31Z<p>Every Xid number reported on CMP 170HX cards in the cmpunlocker Discord (2026-10-04 to 10-28) and in community repos (170tune, club-170hx, cmp170hx-testsuite, Consensus-Protocol, cmpunlocker forks), with cause and fix as reported; credits inline; sources: @ichundes, @reala7096, @matstatman, @boz._., @mistyashhole, @rh3dai, @ob88594, @endsjeremy, @mrmannyatx, @exile_1ck, @rrickk420, @wakeupmbrc1 (+9 more); GitHub Consensus-Protocol/cmp170hx, GNmac-lab/deepseek-v41-flash-1017-c</p>
<p><b>New page</b></p><div>An '''Xid''' is an error code the NVIDIA kernel driver writes to the kernel log when a GPU faults. This page lists every Xid number reported on CMP 170HX cards, with the cause and the fix as the person who hit it reported them. NVIDIA's own definitions are in its [https://docs.nvidia.com/deploy/xid-errors/index.html Xid documentation]; the meanings below are what people saw on these cards. For problems that do not print an Xid, start at [[Troubleshooting]].<br />
<br />
== Reading the log first ==<br />
* Xids appear in <code><nowiki>sudo dmesg</nowiki></code> as lines like <code><nowiki>NVRM: Xid (PCI:0000:c1:00): 48, ...</nowiki></code>. The PCI address names the card that faulted.<br />{{Source|handle=ichundes|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557350219111342183}}<br />
* On a multi-card host, prove which physical card is at fault before changing anything: swap two devices in <code><nowiki>CUDA_VISIBLE_DEVICES</nowiki></code> and check whether the fault follows the card or stays with the pipeline stage.<br />{{Source|handle=reala7096|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557544812972871813|repo=https://github.com/arvkmr/vllm-170hx-glm5}}<br />
* Every card keeps an InfoROM log of past Xid events with timestamps. <code><nowiki>blackbox.py</nowiki></code> from gpcprobe dumps it (the DEM record is a FIFO of Xid events), so you can read a card's history after a reboot or before you buy it.<br />{{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850|repo=https://github.com/matstatman/gpcprobe}}<br />{{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850}}<br />
* Keep ECC enabled while you tune. With ECC on, an unstable card crashes the engine with an Xid instead of silently producing garbage. See [[Memory errors and ECC]].<br />{{Source|handle=ichundes|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557350680342298647}}<br />
* Several Xids leave the card unusable until power is removed. A warm reboot is often not enough; see [[Recovery]].<br />
<br />
== Xid table ==<br />
{| class="wikitable sortable"<br />
! Xid !! What the log said (as posted) !! Reported cause !! What fixed it !! Cards !! Status !! Source<br />
|-<br />
| 1<br />
| <code><nowiki>Xid 1 GSP kernel panic / illegal instruction</nowiki></code> at init; elsewhere "GSP task exception"<br />
| (a) An unlocked 8 GB card (v0.5 with ECC) ran 25 h clean, then the first CUDA context hit Xid 119 then Xid 154; after that every boot failed GSP init with Xid 1, on the stock driver, on a second host and after a full power-off. Both hosts used the same eGPU dock. (b) A card with bad HBM crashed its GSP under load with Xid 119, Xid 1 and Xid 154.<br />
| (a) Not resolved in the thread. Suggested: unpowered reset, check InfoROM row-remap records, test in a native PCIe slot; another member had the same symptom from a broken PCIe slot. (b) Full power-off of about 30 s; a warm reboot was not enough.<br />
| 8 GB<br />
| {{Status|Reported}}<br />
| {{Source|handle=boz._.|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1558098091574562848|repo=https://github.com/dev-boz/cmp170hx-testsuite}}<br />{{Source|handle=mistyashhole|date=2026-10-10|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1558576686214283376}}<br />{{Source|handle=matstatman|date=2026-10-10|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1558608314454249503}}<br />
|-<br />
| 13<br />
| <code><nowiki>Xid 13, pid=..., name=VLLM::Worker, Graphics Exception: ... Class 0000c6c0</nowiki></code><br />
| (a) Mailbox P2P behind a PLX switch, in tensor parallel. (b) Follows Xid 48 and Xid 172 on a card with too much undervolt. (c) SM clock offset too high: +300 and +350 pass benchmarks, then fault under a long serving soak on weak SMs (GPC3/TPC6 on the reference card). Under a 200 W cap, +280 and +290 threw Xid 13 then Xid 43.<br />
| (a) Avoid PLX mailbox P2P for TP, or run PP or P2P off. (b) Reduce the voltage offset by 50. (c) Use +200 or +250; those were clean in the same soak.<br />
| both<br />
| {{Status|Reported}}<br />
| {{Source|handle=rh3dai|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1556287917775655053|repo=https://github.com/cachenetics/170tune}}<br />{{Source|handle=rh3dai|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1556287917775655053|repo=https://github.com/vektorprime/170tune}}<br />{{Source|handle=ichundes|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557350219111342183}}<br />{{Source|handle=ob88594|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557383845844615241}}<br />
|-<br />
| 26<br />
| none posted<br />
| Asked about, not seen: a used ex-mining card showed "software glitches" in its seller's miner. One member suggested an IRQ conflict with another device and asked whether it was Xid 26. No log was posted.<br />
| Unknown.<br />
| unknown<br />
| {{Status|Theory}}<br />
| {{Source|handle=endsjeremy|date=2026-10-19|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1561844340354064485}}<br />
|-<br />
| 31<br />
| MMU fault, <code><nowiki>FAULT_INFO_TYPE_REGION_VIOLATION</nowiki></code><br />
| (a) An unstable undervolt. (b) An allocation past the usable top of the unlocked window (fault at 63.86 GiB). (c) cmpunlocker v0.2 handed out a protected 141 MB region at the top of memory; long prompts then crashed the second card. (d) The late-PMA path, which publishes the reserved WPR/GSP region: Xid 31, later Xid 154. (e) A host out-of-memory during an eager model load, which preceded a fatal Xid 154. (f) A late-PMA build that registers the GSP WPR2 region into the memory pool: Xid 31 wedges under allocation churn. (g) NVIDIA's UVM managed-memory test run across CMP cards: Xid 31, then Xid 154 on every GPU.<br />
| (a) Back off the undervolt. (b) Offload one fewer layer to that card; the card needed a full reboot (fix not confirmed applied). (c) Use cmpunlocker v0.5 or newer. (d) asm64-hooligan's fork keeps late-PMA off by default; on 40 GB cards the cause was traced to cmpunlocker's own late-pma.patch (PR #32). (e) Size the load before running it on every card; a VM reboot cleared it. (f) On any full-VRAM Xid 31, check first that the build leaves the top reserved region (WPR2) reserved; some forks clamp the pool at 62 GiB. (g) A cold power cycle.<br />
| both<br />
| {{Status|Reported}}<br />
| {{Source|handle=rh3dai|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1556287917775655053|repo=https://github.com/asm64-hooligan/cmpunlocker}}<br />{{Source|handle=mrmannyatx|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557041927977836687|repo=https://github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx}}<br />{{Source|handle=exile_1ck|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557384897260294235|repo=https://github.com/ahnguyen17/ga100-fuse-re}}<br />{{Source|handle=rrickk420|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1558127650223030302|repo=https://github.com/Consensus-Protocol/cmp170hx}}<br />{{Source|handle=wakeupmbrc1|date=2026-10-12|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1559239856519913593}}<br />{{Source|handle=darkmaniac7|date=2026-10-19|note=lab notes}}<br />{{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}<br />{{Source|repo=https://github.com/ZombieLurker/cmpunlocker}}<br />{{Source|repo=https://github.com/amoghmunikote/cmpunlocker}}<br />
|-<br />
| 43<br />
| Xid 43<br />
| (a) After Xid 13 at SM offset +280/+290 under a 200 W cap. (b) One card in a 4-card host during a compute wedge; the driver re-initialized the GPU and the card stayed on the bus.<br />
| (a) Lower the offset. (b) None needed: the card passed the full correctness gate afterwards with results identical to the other three cards. club-170hx treats this Xid as note-and-continue.<br />
| 8 GB<br />
| {{Status|Reported}}<br />
| {{Source|handle=rh3dai|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1556287917775655053|repo=https://github.com/vektorprime/170tune}}<br />{{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}<br />
|-<br />
| 45<br />
| Xid 45<br />
| (a) Changing the HBM divider (NDIV) while a CUDA context is active: the GPU wedges with Xid 45 and Xid 119. (b) SIGKILL on a running CUDA verification kernel. (c) On a 10 GB card pushed to 80 GB, as part of a chain: CUDA error 719, then Xid 45, then Xid 154.<br />
| (a) Power cycle. Never change NDIV with a CUDA context open. (b) Run memory checkers in the foreground and stop them with Ctrl-C between kernel launches, not kill -9. (c) No fix; see [[Memory errors and ECC]].<br />
| both<br />
| {{Status|Reported}}<br />
| {{Source|handle=rh3dai|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1556287917775655053|repo=https://github.com/cachenetics/170tune}}<br />{{Source|handle=rrickk420|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1558127650223030302|repo=https://github.com/Consensus-Protocol/cmp170hx}}<br />
|-<br />
| 48<br />
| <code><nowiki>Xid 48, An uncorrectable ECC error has been detected on GPU in GPC0 GCC</nowiki></code><br />
| Too much undervolt (V/F offset) on one card. Logged together with Xid 172 and followed by Xid 13, which killed the vLLM worker. Both people who hit it had passed 170tune's stability and workload tests at that offset.<br />
| Lower the voltage offset by 50. A second member stopped once-or-twice-a-day gibberish by lowering one card from +250 to +150.<br />
| unknown<br />
| {{Status|Proven}}<br />
| {{Source|handle=ichundes|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557350219111342183}}<br />{{Source|handle=cibernox|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557385320465702933}}<br />
|-<br />
| 62<br />
| PMU halt<br />
| A card with bad HBM, under load. nvidia-smi then showed <code><nowiki>[GPU requires reset]</nowiki></code> or <code><nowiki>[N/A]</nowiki></code>.<br />
| Full power-off of about 30 s. The same card later proved to have a load-transient memory fault; see [[Memory errors and ECC]].<br />
| 8 GB<br />
| {{Status|Reported}}<br />
| {{Source|handle=boz._.|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1558098091574562848|repo=https://github.com/dev-boz/cmp170hx-testsuite}}<br />
|-<br />
| 79<br />
| "GPU has fallen off the bus"<br />
| (a) One card at the stock 250 W limit fell off three times under sustained load; nvidia-smi showed up to 333 W against the 250 W limit during prefill. (b) Forcing 8 GT/s (Gen3) on a card whose Gen3 fuse is still set. (c) A Gen3 retrain on a card with a damaged PCIe controller. (d) A card that does not come back from the Gen2 retrain.<br />
| (a) <code><nowiki>nvidia-smi -pl 180</nowiki></code>, re-applied at every boot by a systemd unit. (b, c, d) Only a true cold power cycle brought the card back: function-level reset, secondary bus reset, runtime-power changes and remove/rescan all failed. 30 s with main power removed worked.<br />
| both<br />
| {{Status|Reported}}<br />
| {{Source|handle=rh3dai|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1556287917775655053|repo=https://github.com/asm64-hooligan/cmpunlocker}}<br />{{Source|handle=gnasherrr|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560998024136753172|repo=https://github.com/Zanooda/deepseek-v4.1-flash-cmp170hx}}<br />{{Source|handle=lxtend|date=2026-10-19|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1561646019870064710|repo=https://github.com/lixiangwuxian/170hx_tips}}<br />{{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}<br />
|-<br />
| 109<br />
| <code><nowiki>Xid 109 CTX SWITCH TIMEOUT</nowiki></code><br />
| A 10 GB card unlocked to all 128 SMs, where every extra GPC was marked defective. A simple int8 workload ran at 250 W, then crashed with Xid 109; the miner did not compute at all (power stuck near 60 W).<br />
| Disable the defective-marked GPCs one at a time until the card is stable, then re-enable earlier ones to find the boundary. Disabling GPC1 fixed it.<br />
| 10 GB<br />
| {{Status|Reported}}<br />
| {{Source|handle=lxtend|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1561004454965153833}}<br />{{Source|handle=wakeupmbrc1|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1561015848297299999}}<br />
|-<br />
| 119<br />
| GSP RPC timeout. Two forms: 60 s waiting on function 4097 <code><nowiki>GSP_INIT_DONE</nowiki></code>, and 6 s on function 103 <code><nowiki>GSP_RM_ALLOC</nowiki></code><br />
| (a) 60 s form: GSP never finished booting, usually after a derailed unlock attempt; the next load then reports "WPR2 already up". (b) 6 s form: the card boots, then every nvidia-smi hangs and the Xid repeats about every 6 s. (c) The first CUDA context on a card that had run 25 h, followed by Xid 154 and "GPU reset required" with NaN temperatures (see Xid 1). (d) Changing NDIV under an active CUDA context. (e) A bad-HBM card under load.<br />
| Reset the card; the state does not recover in place. Try function-level reset first, then secondary bus reset or a cold power cycle. For (d), power cycle.<br />
| both<br />
| {{Status|Reported}}<br />
| {{Source|handle=rh3dai|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1556287917775655053|repo=https://github.com/cachenetics/170tune}}<br />{{Source|handle=boz._.|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1558098091574562848|repo=https://github.com/dev-boz/cmp170hx-testsuite}}<br />{{Source|handle=rrickk420|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1558127650223030302|repo=https://github.com/Consensus-Protocol/cmp170hx}}<br />{{Source|handle=mistyashhole|date=2026-10-10|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1558605202674942033}}<br />
|-<br />
| 154<br />
| "GPU recovery action changed", for example to OS Reboot<br />
| (a) After Xid 119 on the card in the Xid 1 row. (b) A bad-HBM card under load. (c) A host out-of-memory during a pipeline-parallel load, after Xid 31 on two cards. (d) A 10 GB card over-provisioned to 80 GB, after CUDA memory tests: one CUDA context per attempt. (e) The late-PMA path, after Xid 31. (f) The UVM managed-memory test across CMP cards, after Xid 31.<br />
| A module reload did not clear it. A reboot did; on a VM with passthrough, rebooting the guest was enough. For (b), full power-off of about 30 s. For (d), no fix; stay at 40 GB. For (f), a cold power cycle.<br />
| both<br />
| {{Status|Reported}}<br />
| {{Source|handle=rh3dai|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1556287917775655053|repo=https://github.com/asm64-hooligan/cmpunlocker}}<br />{{Source|handle=boz._.|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1558098091574562848|repo=https://github.com/dev-boz/cmp170hx-testsuite}}<br />{{Source|handle=rrickk420|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1558127650223030302|repo=https://github.com/Consensus-Protocol/cmp170hx}}<br />{{Source|handle=mistyashhole|date=2026-10-10|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1558576686214283376}}<br />{{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}<br />
|-<br />
| 172<br />
| <code><nowiki>Xid 172, GCC, Uncorrectable SRAM error in GPC 0</nowiki></code><br />
| Same event as Xid 48: too much undervolt.<br />
| Lower the voltage offset (see Xid 48).<br />
| unknown<br />
| {{Status|Reported}}<br />
| {{Source|handle=ichundes|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557350219111342183}}<br />
|}<br />
<br />
== Errors that are not Xids ==<br />
These strings come up in the same logs and are often mistaken for a dead card.<br />
{| class="wikitable"<br />
! What you see !! What it was !! What to do !! Source<br />
|-<br />
| <code><nowiki>kgspExtractVbiosFromRom_TU102: did not find valid ROM signature</nowiki></code>, <code><nowiki>RmInitAdapter failed! (0x62:0x25:2119)</nowiki></code>, ROM header reads <code><nowiki>0xffff</nowiki></code> instead of <code><nowiki>0xaa55</nowiki></code>, nvidia-smi says "No devices were found"<br />
| A 10 GB card that never initialized, even on the stock 610.57.04 driver. The VBIOS SPI flash is suspected.<br />
| Try reflashing the VBIOS with a hardware programmer; failing that, a new flash chip or a damaged trace. See [[Recovery]].<br />
| {{Source|handle=youngju10000|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557644692487671822}}<br />{{Source|handle=ichundes|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557646290827870238}}<br />
|-<br />
| <code><nowiki>unexpected WPR2 already up, cannot proceed with booting GSP</nowiki></code>, then <code><nowiki>RmInitAdapter failed! (0x62:0x40:2028)</nowiki></code><br />
| A previous boot or unlock attempt left the WPR2 region programmed. In a VM it appears on the second and later VM starts.<br />
| Function-level reset or a cold power cycle. For VMs, restore the GSP boot state before each VM start (the cmpunlocker passthrough service and the Proxmox guide do this).<br />
| {{Source|handle=rrickk420|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1558127650223030302|repo=https://github.com/Consensus-Protocol/cmp170hx}}<br />{{Source|handle=logy0|date=2026-10-10|link=https://discord.com/channels/1518634379604136178/1540599965007478854/1558505023187714068|repo=https://github.com/anoane/01-proxmox-dual-gpu-passthrough}}<br />
|-<br />
| <code><nowiki>RmInitAdapter</nowiki></code> or SEC2 failures inside a Proxmox VM<br />
| The VM was booted with OVMF (UEFI) firmware. They look exactly like a failed unlock.<br />
| Use SeaBIOS for the VM.<br />
| {{Source|handle=rrickk420|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1558127650223030302|repo=https://github.com/Consensus-Protocol/cmp170hx}}<br />{{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}<br />
|-<br />
| NVRM <code><nowiki>RC watchdog</nowiki></code>, then <code><nowiki>NV_ERR_RESET_REQUIRED (0x62)</nowiki></code><br />
| The HBM controller hard-hung after the tCCD_L memory timing was set to 3. Writing the old value back did not recover it and <code><nowiki>nvidia-smi -r</nowiki></code> was not supported.<br />
| Reboot.<br />
| {{Source|handle=rh3dai|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1556287917775655053|repo=https://github.com/asm64-hooligan/cmpunlocker}}<br />
|-<br />
| <code><nowiki>N/12 FBPAs failed to lock</nowiki></code> in dmesg<br />
| The HBM divider (NDIV) is set too high for the card.<br />
| Lower NDIV. Values below stock are the documented way to stabilise a card that fails at stock.<br />
| {{Source|handle=rh3dai|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1556287917775655053|repo=https://github.com/asm64-hooligan/cmpunlocker}}<br />
|-<br />
| NVRM <code><nowiki>_scrubWaitAndSave: Timed out</nowiki></code>, CUDA fails in every container while nvidia-smi still lists the cards<br />
| Left over from a crashed GPU job.<br />
| Stop nvidia-persistenced, <code><nowiki>rmmod nvidia_uvm nvidia_drm nvidia_modeset nvidia</nowiki></code>, then <code><nowiki>modprobe nvidia && modprobe nvidia_uvm</nowiki></code> (about 3 min). Re-apply the power cap afterwards: a module reload resets it to 250 W.<br />
| {{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}<br />
|-<br />
| <code><nowiki>cuInit</nowiki></code> returns 999 in every framework while nvidia-smi looks healthy<br />
| About 32 zombie CUDA processes left by <code><nowiki>kill -9</nowiki></code> on live multi-GPU jobs.<br />
| Reboot the host; it cannot be fixed inside a container. Stop multi-GPU jobs with their own stop script.<br />
| {{Source|handle=rrickk420|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1558127650223030302|repo=https://github.com/Consensus-Protocol/cmp170hx}}<br />{{Source|handle=greencandle9753|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562465339395280909|repo=https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx}}<br />
|-<br />
| "No CUDA GPUs are available" with nvidia-smi healthy<br />
| Usually a container in a <code><nowiki>--restart always</nowiki></code> crash loop holding the card.<br />
| <code><nowiki>docker rm -f</nowiki></code> the container, confirm 0 MiB in use, relaunch.<br />
| {{Source|handle=vektor_prime|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557443747892035674|repo=https://github.com/vektorprime/qwen38-flash-next-pp2}}<br />
|-<br />
| <code><nowiki>HW Power Brake Slowdown</nowiki></code> active in <code><nowiki>nvidia-smi -q</nowiki></code>, card stuck near 88 W and 1140 MHz<br />
| The board asserts PWRBRK# on edge pin B30 (seen on an ASUS Pro WS WRX80E-SAGE).<br />
| Kapton tape over pin B30, or a riser that does not route B30.<br />
| {{Source|handle=gnasherrr|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560998024136753172|repo=https://github.com/Zanooda/deepseek-v4.1-flash-cmp170hx}}<br />
|-<br />
| 170tune preflight: <code><nowiki>PLL PLM 0xBADF2010</nowiki></code>, "FBPA privilege masks are closed"<br />
| A <code><nowiki>0xBADF....</nowiki></code> read is a PRI error: the register window is fenced by the unlock build in use.<br />
| See [[Driver and unlock failures]].<br />
| {{Source|handle=rh3dai|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1556290671839551501}}<br />
|-<br />
| <code><nowiki>krcWatchdog: GPU is probably locked</nowiki></code> on another GPU in the same VM, then every new GPU process hangs at exit in <code><nowiki>uvm_va_space_destroy</nowiki></code>; no Xid<br />
| nvidia-open 615.71.09 with cmpunlocker in a VM that also holds RTX PRO 6000 (GB202) cards: any real CUDA context wedges UVM. NVIDIA lists the symptom as a known 615.71.09 issue.<br />
| Only a VM reset clears it. Keep the CMP cards off a 615 driver shared with those cards; the lab runs the unlock ported to 580.178.04 instead.<br />
| {{Source|handle=darkmaniac7|date=2026-10-21}} (lab notes, [https://github.com/darkmaniac7/cmp170hx-esxi darkmaniac7/cmp170hx-esxi])<br />{{Source|handle=darkmaniac7|date=2026-10-25}} (lab notes, [https://github.com/darkmaniac7/cmp170hx-esxi darkmaniac7/cmp170hx-esxi])<br />
|-<br />
| Dozens of <code><nowiki>Booter failed with non-zero error code: 0x31</nowiki></code> (or <code><nowiki>0xffff</nowiki></code>) lines at boot<br />
| The unlock's own retry rounds. A search for booter or GSP errors returned 888 lines on a healthy boot.<br />
| Nothing; judge the boot by the checks on [[Troubleshooting#Before you ask for help|Troubleshooting]]<br />
| {{Source|handle=rrickk420|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1558127650223030302|repo=https://github.com/Consensus-Protocol/cmp170hx}}<br />{{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}<br />
|-<br />
| <code><nowiki>failed to fulfil mount request: open /run/nvidia-persistenced/socket</nowiki></code> from the NVIDIA container runtime<br />
| nvidia-persistenced is not running after a VM power cycle<br />
| Start nvidia-persistenced before any GPU container<br />
| {{Source|handle=darkmaniac7|date=2026-10-21|note=lab notes}}<br />
|-<br />
| <code><nowiki>kbusEnableStaticBar1Mapping_TU102: Failed to create the static bar1 mapping</nowiki></code>, then <code><nowiki>RmInitAdapter failed</nowiki></code><br />
| <code><nowiki>RMForceStaticBar1=1</nowiki></code> alone in an ESXi guest: the static map lands at a 512 MiB offset and a 63.6 GiB framebuffer no longer fits the 64 GiB BAR1<br />
| Remove the option. Static BAR1 P2P does not work inside the VM anyway; see [[Card not detected or BAR allocation fails#ESXi|ESXi]]<br />
| {{Source|handle=darkmaniac7|date=2026-10-21}} (lab notes, [https://github.com/darkmaniac7/cmp170hx-esxi darkmaniac7/cmp170hx-esxi])<br />
|-<br />
| <code><nowiki>NV_ERR_INVALID_REGISTRY_KEY</nowiki></code> after adding BAR1 P2P keys<br />
| <code><nowiki>RMForceStaticBar1</nowiki></code> / <code><nowiki>RMPcieP2PType</nowiki></code> passed through <code><nowiki>NVreg_RegistryDwords</nowiki></code> on a 610.43.02 GSP: rejected, and its protected memory region stays up<br />
| A cold power-off; put those keys where your fork documents them<br />
| {{Source|handle=darkmaniac7|date=2026-10-19|note=lab notes}}<br />{{Source|repo=https://github.com/LingzheZhao/cmpunlocker}}<br />
|-<br />
| NVRM <code><nowiki>Out of memory [NV_ERR_NO_MEMORY]</nowiki></code> from <code><nowiki>system_mem.c</nowiki></code> while a large multi-GPU job boots<br />
| Host memory fragmentation (large allocations failing), not a GPU or container limit; the engine recovered<br />
| <code><nowiki>echo 1 > /proc/sys/vm/compact_memory</nowiki></code> before the first boot<br />
| {{Source|handle=darkmaniac7|date=2026-10-19|note=lab notes}}<br />
|-<br />
| Processes in D state with wait channel <code><nowiki>os_acquire_rwlock</nowiki></code> for under a second while KV cache is allocated<br />
| Normal: large allocations queue on the driver-wide lock<br />
| Nothing. A real UVM wedge holds D states in <code><nowiki>uvm_va_space_destroy</nowiki></code> or <code><nowiki>uvm_gpu_release</nowiki></code> for minutes; see [[Recovery#Driver wedge inside a VM|Recovery]]<br />
| {{Source|handle=darkmaniac7|date=2026-10-19|note=lab notes}}<br />
|-<br />
| GSP booter error <code><nowiki>0xffff</nowiki></code> and the card no longer initializes, after installing an HBM clock tool lineage<br />
| The card's FBPA0 is fused off and the tool writes FBPA0 PLL registers<br />
| See [[Driver and unlock failures#FBPA PLL tools stopped two of eight cards from booting|the warning]]<br />
| {{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}<br />
|}<br />
<br />
== Related pages ==<br />
* [[Troubleshooting]] (symptom index)<br />
* [[Instability under load]]<br />
* [[Memory errors and ECC]]<br />
* [[Recovery]]<br />
* [[Driver unlock]]<br />
<br />
{{ImportFooter}}<br />
[[Category:Troubleshooting]]<br />
[[Category:8 GB]]<br />
[[Category:10 GB]]</div>ImportBot