Recovery

How people got unresponsive cards and hosts working again, from the cheapest step to the most drastic, what cannot be undone, and what board-level repair has worked on dead cards. Error numbers are explained on Xid errors.

First: remove power

A warm reboot is not a reset: the board stays powered and some state survives it. When a card is wedged, shows "GPU requires reset", has fallen off the bus or stays at its stock memory size, shut down, switch the PSU off or unplug it, wait, then power on. The published procedure waits 60 seconds. Measured cases: 30 seconds with main power removed brought back a card after Xid 79, and about 30 seconds off cleared firmware crashes (Xid 119, 1, 154, 62) where a warm reboot did not. Pull the plug completely, not just the soft power switch.
Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
Source: @bigptoughneigh, 2026-10-10
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20

For lighter cases a plain sudo reboot is often enough. Waiting about 20 s with the host off lets the PCIe slots discharge, which is what makes it a cold cycle for these cards; the unlocker's README says AC off for about 10 s. Some cards also needed their power cable unplugged and reseated after a cold boot.
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
Source: @lxtend, 2026-10-13
Source: @darkmaniac7, 2026-10-27 (lab notes)

The reset ladder

Work down the list; each step is more disruptive than the last. From the Consensus-Protocol recovery notes (snapshot of 2026-08-22).
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09

#	Step	Command	Clears
1	Reload the driver	`modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia && modprobe nvidia`	Driver state only
2	Function-level reset (FLR)	`sudo rmmod nvidia_uvm nvidia`, then `echo 1 \| sudo tee /sys/bus/pci/devices/$BDF/reset`	Engines and the WPR2 region left by a failed boot
3	PCI remove and rescan	`echo 1 \| sudo tee /sys/bus/pci/devices/$BDF/remove`, `echo 1 \| sudo tee /sys/bus/pci/rescan`, then `sudo setpci -s <bdf> COMMAND=0x0546` to turn bus mastering back on	A card whose BAR0 reads `0xffffffff` (off the bus)
4	Secondary bus reset on the upstream bridge	(bridge-specific)	Everything FLR clears plus the always-on power domain
5	Full power-off cold boot	PSU off or unplugged, 60 s	Everything
6	Take the card out for a while	-	Last resort; used once in an unexplained case

What happened in the field:

After Xid 79 a card stayed unusable through FLR, secondary bus reset, runtime-power changes and remove/rescan; only a true cold power cycle (30 s off) brought it back. Repeated warm reboots wasted time. Reported
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
After a crashed job, CUDA failed in every container while nvidia-smi still listed the cards (_scrubWaitAndSave: Timed out). A module reload (step 1) fixed it in about 3 minutes. When the log showed Xid 154 or a UVM global fatal error, the reload did not help; rebooting the VM did, and the passthrough host did not need a reboot. Both steps reset the power cap to 250 W. Reported
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
PCIe hot-swap on a board that supports it brought back only one or two of eight cards. Reported
Source: @reala7096, 2026-10-13
For repeated cold boots during fuse or register work, drive the power remotely: IPMI, a smart plug with power-on-after-power-loss, or a wireless power button.
Source: @domrockt2, 2026-10-13
Source: @djpandaman, 2026-10-13

Problems a cold boot clears

Problem	What brought it back	Status	Source
Errors after enabling the 6th HBM stack on a 10 GB card	A cold boot, every time	Reported	Source: @djpandaman, 2026-10-09
A 10 GB card lost during a Gen3 link transition	Reboot or full power cycle. To leave Gen3 without rebooting, `cmp-pcie-speed 2` returns the link to Gen2.	Reported	Source: repo github.com/lixiangwuxian/170hx_tips by lixiangwuxian on GitHub, posted by @lxtend, 2026-10-19
A card that does not come back from the Gen2 retrain	A cold power cycle; nothing less	Reported	Source: repo github.com/asm64-hooligan/cmpunlocker by asm64-hooligan on GitHub, posted by @rh3dai, 2026-10-04
Card still at 8 GB or 10 GB after installing the unlock	A cold boot (an OS reboot is not enough), then the checks on Driver and unlock failures	Reported	Source: repo github.com/Morrowmake/cmpunlocker by Morrowmake on GitHub, posted by @trustinter, 2026-10-07
`unexpected WPR2 already up` after a failed attempt	FLR or a cold power cycle	Reported	Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
GPU wedged after changing NDIV with a CUDA context open (Xid 45, 119)	Power cycle	Reported	Source: repo github.com/cachenetics/170tune by cachenetics on GitHub, posted by @rh3dai, 2026-10-04
Hard crash at SM offset +450	Power cycle	Reported	Source: repo github.com/cachenetics/170tune by cachenetics on GitHub, posted by @rh3dai, 2026-10-04
Two cards would not boot (GSP booter error `0xffff`) after an FBPA PLL tool lineage was installed	Installing a driver lineage that writes no FBPA PLL registers (Morrowmake) and cold booting; no lasting damage	Proven	Source: @darkmaniac7, 2026-10-27 (lab notes)
`RmInitAdapter` 0x62 after a driver build added a register to the Booter's privilege-mask table	Restoring the previous modules and power-cycling the VM; no lasting damage	Proven	Source: @darkmaniac7, 2026-10-25 (lab notes, darkmaniac7/cmp170hx-esxi)
An 8 GB card stuck at Gen1 x8 after a Gen3 register attempt	A VM power cycle, which re-ran the Gen2 retrain	Proven	Source: @darkmaniac7, 2026-10-21 (lab notes, darkmaniac7/cmp170hx-esxi)
A card off the bus (BAR0 all ones) after the PR #37 Gen3 retrain; PCI remove/rescan hung	A chassis power cycle	Reported	Source: @darkmaniac7, 2026-10-14 (lab notes) Source: repo github.com/amoghmunikote/cmpunlocker by amoghmunikote on GitHub
Memory controller hung after setting tCCD_L to 3 (`NV_ERR_RESET_REQUIRED`); writing the old value back did not help	Reboot	Reported	Source: repo github.com/asm64-hooligan/cmpunlocker by asm64-hooligan on GitHub, posted by @rh3dai, 2026-10-04
Card overheated and fell off the bus (95 C)	Let it cool, put airflow on it, power cycle	Reported	Source: @analyticeth, 2026-10-07

One that did not clear: an unlocked 8 GB card that hit Xid 119 then Xid 154 failed GSP init (Xid 1) on every boot afterwards, on the stock driver, on a second host and after a full power-off. Suggestions were a fully unpowered reset, checking the InfoROM row-remap records with NVIDIA's MODS tool, and checking the slot, contacts and cable before uninstalling, cold booting, reinstalling and cold booting again. Both hosts used the same eGPU dock and a broken PCIe slot once caused the same symptom for someone else, so test a native slot first. Reported
Source: @mistyashhole, 2026-10-10
Source: @bigptoughneigh, 2026-10-10
Source: @_orrorin, 2026-10-10
Source: @matstatman, 2026-10-10

Driver wedge inside a VM

Symptoms: RmInitAdapter failed, Falcon halt or reset timeouts (kflcnWaitForHalt_TU102, kflcnReset_TU102), hung-task reports, processes in D state on uvm_* or os_acquire_rwlock*, nvidia-smi hanging.
Source: @darkmaniac7, 2026-10-19 (lab notes, darkmaniac7/cmp170hx-esxi)

Before running nvidia-smi, list D-state processes with their wait channel (ps -eo pid,stat,wchan:32,comm) and search dmesg for RmInitAdapter failed, Xid, Falcon timeouts, "GPU is probably locked", "fallen off the bus" and "hung task". Always run nvidia-smi under timeout 20.
Once wedged, do not run nvidia-smi, nvidia-bug-report or CUDA probes: every new caller blocks in D state on the driver lock, and timeout cannot kill a D-state process. Capture dmesg and the process list over SSH instead.
If a test module set or option file caused it, restore the production files on disk first (file operations still work during the wedge), or the next boot loads the same thing again.
A guest poweroff does not complete while GPU processes sit in D state: hard-reset the VM from the hypervisor.
If SSH is gone, add module_blacklist=nvidia,nvidia_drm,nvidia_modeset,nvidia_uvm,nvidia_peermem at the GRUB prompt, fix the files, then power cycle.
A wedge that survives a VM power cycle needs a cold cycle of the host. Never remove a passed-through card from the hypervisor side.

Afterwards, verify in this order: the wedge pre-check, driver identity and options, any BAR1 repair hook, then NVML. A D state on os_acquire_rwlock for under a second while KV cache is allocated is normal; the real wedge holds D states in uvm_va_space_destroy or uvm_gpu_release for minutes, with nvidia-smi stuck and GPU memory never released.
Source: @darkmaniac7, 2026-10-19 (lab notes, darkmaniac7/cmp170hx-esxi)
Source: @darkmaniac7, 2026-10-21 (lab notes, darkmaniac7/cmp170hx-esxi)

When the host will not boot or shut down

The driver re-wedges the card at every boot. Boot once with the driver blacklisted on the kernel command line (modprobe.blacklist=nvidia,nvidia_uvm,nvidia_drm,nvidia_modeset) or with the card disconnected, then clean up. One 10 GB card stuck in uninterruptible sleep survived about five cold reboots and stopped Ubuntu from shutting down; the cause was the autoloading patched driver, not the card.
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
The Gen2 boot service hangs. Boot with systemd.mask=gen2.service.
Source: repo github.com/amoghmunikote/cmpunlocker by amoghmunikote on GitHub, posted by @rh3dai, 2026-10-08
A 170tune profile breaks boot. 170tune persist disable or systemctl mask 170tune-persist.service; its bootcheck service also reverts to stock by itself if the previous boot's apply never checked in.
Source: repo github.com/cachenetics/170tune by cachenetics on GitHub, posted by @rh3dai, 2026-10-04
The asm64-hooligan fork's rebuild service holds boot until patched modules exist for the running kernel and gives up after 3 consecutive failures; logs are in /var/log/cmpunlocker/.
Source: repo github.com/asm64-hooligan/cmpunlocker by asm64-hooligan on GitHub, posted by @rh3dai, 2026-10-04
Unloading the driver takes the desktop down. rmmod kills the session when other NVIDIA cards drive the display and there is no iGPU; unbind only the CMP through sysfs, or work headless over SSH.
Source: @lycorascence, 2026-10-18
Source: @c133, 2026-10-18
Source: @intervitens, 2026-10-23
Warning ESXi boot disk once flagged for passthrough. On an ESXi 8 host used for CMP passthrough, the boot NVMe's controller had been marked for passthrough in a stale saved config. Each boot handed the boot disk to passthrough, the boot bank fell back to RAM, every config save was lost and the next boot reloaded the same stale config (maintenance mode on, SSH off, no datastores). Fix: esxcli hardware pci pcipassthru set -d <dev> -e false -a (the -a applies it now), write the corrected config into both boot banks, and add a boot guard that forces passthrough off for the boot disk. Never keep ESXi on a disk that was ever marked for passthrough.
Source: @darkmaniac7, 2026-10-27 (lab notes, darkmaniac7/cmp170hx-esxi)
A botched driver deploy. One developer had to reinstall the OS after each bad nvidia.ko deploy on bare metal; test driver patches in a VM.
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
Warning A coding agent with root. A box stopped answering SSH after a coding harness ran Qwen 3.8 27B with root; a reboot brought it back. Check SMART and keep backups. Turn on the harness sandbox and do not give a model unsupervised root unless it is one of the strongest available. A sandbox does not help when the job itself is driver work.
Source: @bagstah, 2026-10-04
Source: @matstatman, 2026-10-04
Source: @segmond, 2026-10-04
Warning Agents with network access. An agent given a wrong IP probed the network, guessed passwords and found the right host. Another kept trying to get at passwords in its VM, and its macros changed every text box on three slides and set German number formats on every cell, then saved the files. Isolate agents and keep backups before you let them run.
Source: @segmond, 2026-10-20
Source: @con3po7016, 2026-10-23

Getting back to a stock card

sudo ./remove.sh --yes, then a cold boot. Confirm with cat /proc/driver/nvidia/version, sudo dmesg | grep SEC2_DEBUG (prints nothing) and nvidia-smi (8192 or 10240 MiB).
Source: repo github.com/amoghmunikote/cmpunlocker by amoghmunikote on GitHub, posted by @rh3dai, 2026-10-08
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
If a reinstall misbehaves: uninstall, cold boot, reboot, install, cold boot.
Source: @_orrorin, 2026-10-10
The card runs on the stock Linux NVIDIA driver with no patch (reported with nvidia-driver-570 and CUDA 12.8 on Ubuntu 24.04), shown as "NVIDIA Graphics Device", compute capability 8.0. Getting it running stock is a quick way to prove the card survived.
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
The card's InfoROM black box records its history (power-on hours, power-cap history, past Xids); dump it with gpcprobe's blackbox.py.
Source: repo github.com/matstatman/gpcprobe by matstatman on GitHub, posted by @matstatman, 2026-10-20

What cannot be undone

Warning RIR burns. They permanently change fuse state; some rewrites completely brick the card and a wrong burn wastes slots. No rollback script restores a burn. Details on Driver and unlock failures.
Source: @djpandaman, 2026-10-10
Source: repo github.com/evanrodgers/cmp-170hx-rir-burn-util by evanrodgers on GitHub, posted by @hezz1891, 2026-10-26
Warning VBIOS flashing. The VBIOS is signed. A member recalled a post where a VBIOS TOCTOU test caused damage that restoring the original BIOS did not fix (hearsay). Recovery from a bad flash is physical only, with an SPI clip on the flash chip; only a full power-off resets the card's Falcon reliably (function-level reset and PCIe remove/rescan worked about one time in three), so inside a VM every flash attempt costs a host power cycle. A wrong-voltage SPI programmer can destroy the flash chip; see Power and cabling warnings.
Source: @youngju10000, 2026-10-08
Source: @darkmaniac7, 2026-10-25 (lab notes)
Has anyone permanently bricked a card? Contested
- The Consensus-Protocol notes (snapshot 2026-08-22) found no confirmed permanent brick from the register-only unlock, but record one unexplained case: a 10 GB card with stuck kernel threads that survived FLR, secondary bus reset, PCI detach and a full PSU-off cold boot, and came back only after the power strip was switched off and the card removed for a few hours. The same account mentions an agent writing protected fuse-area registers (needed a full power cycle) and one register write that hard-bricked a test bench.
  Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
- A report relayed from the Chinese community: 5 of 20 cards were bricked during 10 GB development.
  Source: @wakeupmbrc1, 2026-10-09
- A developer: a cold boot always fixed his own errors; trouble comes from doing unlocks one by one through RIR.
  Source: @djpandaman, 2026-10-09

Dead or damaged cards: board-level repair

VBIOS flash unreadable (10 GB card, dead on arrival)

One of five cards bought never initialized, even on the stock 610.57.04 driver (Ubuntu 26.04, kernel 7.0.0-38): lspci listed it at Gen1 x16, nvidia-smi said "No devices were found", dmesg showed kgspExtractVbiosFromRom_TU102: did not find valid ROM signature and RmInitAdapter failed! (0x62:0x25:2119), and reading the ROM through sysfs gave an I/O error with header 0xffff instead of 0xaa55. Advice: SPI flash chips can fail on their own; reflash the VBIOS with a hardware programmer, and if that fails, replace the flash chip or look for a damaged trace. A card whose GSP boot is dead from the start is a bad bet. Mind the programmer voltage; see Power and cabling warnings.

Reported
Source: @youngju10000, 2026-10-08
Source: @ichundes, 2026-10-08
Source: @ob88594, 2026-10-09

Dead 3.3 V regulator (A100/170HX-class board)

A card ran for an hour, then dropped off the bus for good. Cause: a dead GS7155NVTD 3.3 V LDO shorting the PS_5V_PGOOD net to 5 ohms, which kept the MP1475DJ 5 V converter from starting (hiccup mode). Fix: replace the MP1475DJ and the GS7155NVTD and jumper PS_5V_PGOOD over from U816; the GA100 came back on PCIe. Bench-test a freshly soldered GS7155NVTD before trusting it: an open feedback pin drives 5 V onto the 3.3 V rail and destroys the 3.3 V logic. Rework on this board needs hot air at 420 C for about 2 minutes.

Reported
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09

Component replacement on a dead 8 GB card

A post on r/LocalAIServers ("Finaly I fixed CMP and Got 144GB VRAM") describes a dead 8 GB card brought back by replacing failed components, with the list of parts in the post. One member's guess at the failed rail was PEXVDD.

Reported
Source: @boz._., 2026-10-09
Source: @lia_oc, 2026-10-09
Source: @_arcanum_, 2026-10-09

10 GB card that will not finish booting (bad GPC)

A non-working 10 GB card bought from a local GPU repair technician (history unknown) appears blocked by GPC6. On a card whose software-override gate is open, writing OPT_GPC_DISABLE_CP turns a GPC off; on this card the gate is fused shut. The only route is an RIR burn, which needs the privilege masks opened, which needs firmware boot to complete, which fails because of GPC6: a circular dependency. Unresolved. Questions to answer first on any dead card: does it enumerate, does a driver bind, and where in the boot does it die.

Reported
Source: @exile_1ck, 2026-10-11
Source: @djpandaman, 2026-10-11

Code 43 and "DQ memory training failed" (CMP card, model not stated)

The card intermittently failed with Code 43 and reflashing the BIOS stopped helping. Debris between a capacitor and a memory chip and leftover no-clean flux were cleaned off; nvmt falmon then nvmt dramtrain x trained the memory; finally the failing chip (channel C1) was reheated with a soldering station and flux and the card ran again. The fix reworks a discrete memory chip, so it applies to GDDR CMP cards; HBM on the 170HX is on the GPU package.

Reported
Source: @d3m3n7r, 2026-10-11
Source: @_orrorin, 2026-10-14

Replacing a fallen-off capacitor

A 100 uF 16 V polymer capacitor (about 5 mm square, two prongs) came off a card; the card works without it. Matching parts: the same family, Lelon OVZ101M1CTR-0606 (5.9 mm), or the in-stock Panasonic 16SVPC100M with the same ripple rating, tolerance and footprint; options start at 5.7 mm tall. Avoid EEE1CA101WP: it is a wet (liquid electrolyte) part with half the endurance. Do not substitute a ceramic such as TDK CKG57NX7R1E107M500JH: lower ESR brings noise and oscillation in a power phase, and ceramics handle DC bias differently. The pinned board document may list 10 uF, but the part's marking reads 100 uF.

Reported
Source: @globalapathy, 2026-10-21
Source: @lia_oc, 2026-10-21

Shipping damage

A card that took a hard hit in shipping and is no longer detected on PCIe may still be fixable if it cannot be returned: open it and measure it with a multimeter. Replacing 0402 capacitors is work an owner with an iron can do at home; if the core needs a reball, it goes out to a shop with the right equipment, because reballing a chip the size of the CMP 170HX's takes a lot of gear and experience.

Reported
Source: @xeax, 2026-10-18
Source: @xeax, 2026-10-18
Source: @_orrorin, 2026-10-18
Source: @globalapathy, 2026-10-18

Capacitor-mod link problems and their fixes are on Card not detected or BAR allocation fails.

Cleaning a dirty card

Ex-mining cards can arrive with rock-hard paste on the IHS and dark dust over the PCB. One owner used about 32 oz of isopropyl alcohol for 8 cards, rinsing three times per card with light brushing and filtering the alcohol for reuse; Kapton tape lifted the last layer of paste off the IHS. Isopropyl alone cleaned another card in seconds. An ArctiClean 2-step kit (citrus degreaser first, then the solvent) also works. Reported
Source: @_syn_thetic_, 2026-10-21
Source: @lia_oc, 2026-10-21
Source: @globalapathy, 2026-10-21
A filthy card is not necessarily a bad one: one visibly filthy card unlocked to 64 GB on the first try.
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
Insects inside the card: take it apart and clean it.
Source: @wakeupmbrc1, 2026-10-09
Repasting and pads: see Cooling and power.

Related pages