Card not detected or BAR allocation fails

This page covers a CMP 170HX that does not show up, shows up in the wrong state (missing from nvidia-smi, wrong PCIe link width or speed), or cannot get its large BAR1 window. Work from the lowest layer up: do not reinstall a driver when the host cannot see the card.
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20

For which card you have and what it unlocks to, see Which card do I have. For the unlock itself, see Driver unlock.

Step 1: find the layer where the card is missing

Check	Command	If it fails
The host sees the card	`lspci -nn \| grep -i 10de` lists `10de:20c2` (8 GB) or `10de:2082` (10 GB)	Hardware: slot, riser, power, card. Go to Card is not in lspci.
The driver bound and GSP booted	`nvidia-smi` lists the card	Driver, GSP or temperature. Go to In lspci but not in nvidia-smi.
The unlock is active	`nvidia-smi` shows about 65536 MiB (8 GB card) or 40960 MiB (10 GB card), not 8192 or 10240 MiB	See Driver and unlock failures.
The link trained as expected	`sudo lspci -vv -s <bdf> \| grep -E 'LnkCap\|LnkSta'`	Go to Link width and speed.

nvidia-smi only guesses PCIe width and speed from bandwidth; lspci shows the real link state.
Source: @djpandaman, 2026-10-21

Server checklist: still 8 GB, stuck at Gen1, or BAR1 stuck

Collect the facts before you change anything: dmidecode, uname -r, /proc/cmdline, /proc/driver/nvidia/version and /proc/driver/nvidia/params, every nvidia line in modprobe.d, whether the initramfs carries nvidia (lsinitramfs), nvidia-smi generation, width and memory, BAR1 usage, lspci -tv, and for each card the Region, LnkCap, LnkSta, LnkCtl2 and Resizable BAR lines plus every upstream bridge's "Prefetchable memory behind" and link lines. In dmesg, look for "NVRM: CMP", retrain, Xid, "BAR 1", "no space", "failed to assign" and the root bus windows. More than one bridge between the card and the root port means a PCIe switch, whose windows must grow too.

Case A: memory still 8 GB (no "NVRM: CMP" lines). Install the same unlocker commit and driver as a known-good box, with an explicit --profile=8gb or --profile=10gb. A kernel or nvidia package upgrade puts stock DKMS modules back and the cards drop to 8 GB and Gen1 with no error, so hold the kernel and all nvidia packages. If memory stays stock, do a real cold start (AC off for about 10 s), not a reboot.
Case B: 64 GB but Gen1. The registry options must appear in /proc/driver/nvidia/params; if nvidia loads from the initramfs, the options must be in it (update-initramfs -u). Only one NVreg_RegistryDwords line counts; a second file silently replaces the first. Check dmesg for the retrain lines. Every upstream bridge's LnkCap must allow 5 GT/s, and LnkCtl2's target speed must not be 2.5 GT/s (a BIOS per-slot link speed setting). LnkSta 2.5GT/s (downgraded) after a retrain points at the signal path: riser, cable or slot; move the card.
Case C: Gen2 but BAR1 at 64 or 256 MB. No "resize enabled" line: check NVreg_EnableResizableBar and set options nvidia NVreg_EnableResizableBar=1 if it is 0. A "final size" below 65536, or kernel "no space for" / "failed to assign" near BAR 1, means the bridge window cannot take 64 GB. Change one thing per reboot, in this order: enable "Memory Mapped I/O above 4GB" in the BIOS; give 64-bit MMIO the most room (on PowerEdge servers "Memory Mapped I/O Base"; a higher base usually leaves more room); add pci=realloc to the kernel command line (remove it if NICs or storage break); isolate to one card (two cards behind one bridge need two 64 GB-aligned blocks of 64 GB plus 32 MB in one window). A BIOS ReBAR option is harmless but does nothing by itself.

Never flash a VBIOS and never hand-write PCI config space on a live system. Verify with nvidia-smi (generation, width, memory), BAR1 total 65536 MiB, and the cmpunlocker repo's tools/p2p-content-check.py (must exit 0), with dmesg | grep Xid empty. To undo: ./remove.sh --yes and reboot, restore the BIOS values, drop pci=realloc.
Source: @darkmaniac7, 2026-10-29 (lab notes)

Card is not in lspci at all

Power the machine fully off first, then change one thing per shutdown: slot, riser, power lead, card. Label cables and risers before each swap.
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
One card stayed undetected after swapping PCIe slots and power cables, while two other cards worked in the same system. The suggested next step was a different system; no resolution was posted. Reported
Source: @globalapathy, 2026-10-14
Source: @68x0, 2026-10-14
Warning Shipping damage: a card that had proof of life and a working unlock before it shipped arrived in a box with a hole, its bracket bent about 30 degrees, and was not picked up on PCIe at all. Nothing inside looked broken. Board-level repair at this level (possibly a reball of the GA100) needs specialist gear.
Source: @globalapathy, 2026-10-18
Source: @_orrorin, 2026-10-18
A broken PCIe slot produced the same symptom as a failing card (GSP init failing). Test the card in another slot before blaming it. Reported
Source: @matstatman, 2026-10-10
On a Dell R740, one GPU power plug on the system board is suspected dead. No fix posted. Reported
Source: @chonsworth, 2026-10-11
A card that ran for an hour and then dropped off the bus for good was traced, on A100/170HX-class hardware, to a dead 3.3 V LDO (GS7155NVTD) shorting the PS_5V_PGOOD net, and repaired at component level. See Recovery.
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
The server will not POST with the cards fitted: in one documented case the GPUs were not the problem. Moving cards between slots renamed the network interface, so the box booted headless with no IP. Some boards also refuse to POST with a CMP as the only card, because it has no video output (an ASRock X370-I was reported). Plan for a display-capable GPU or iGPU.
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09

In lspci but not in nvidia-smi

Warning Overheating. The card has no fan of its own. Without airflow it overheats even at idle and drops out of nvidia-smi while lspci still lists it. One 10 GB card "clocked out" in under 5 minutes with no cooling. Another card reached 85 C at an idle draw of about 40 W and vanished from nvidia-smi. The card falls off the bus at 95 C. Put airflow on the card before you power it on; see Cooling and power.
Source: @analyticeth, 2026-10-07
Source: @programmerboi, 2026-10-16
Source: @civiloid, 2026-10-16
Source: @globalapathy, 2026-10-25
Driver 615.78.08. The unlock installed without errors, but the cards failed to train PCIe 2.0 at boot and appeared in lspci but not in nvidia-smi. Downgrading to 615.71.09 loaded the unlock on the first try. See Known bad combinations. Reported
Source: @drew_2, 2026-10-10
GSP fails to boot. A card that does not initialize even on the stock driver, with did not find valid ROM signature and a ROM header of 0xffff, has a VBIOS flash problem; see Recovery. A GSP boot that is dead from the start is a bad sign for the card. A card that fails GSP init on every boot after an Xid is covered in Xid errors (Xid 1). Reported
Source: @youngju10000, 2026-10-08
Source: @ob88594, 2026-10-09
After a failed unlock attempt the driver can refuse with unexpected WPR2 already up. A cold power cycle clears it.
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
Inside a VM, see VMs and passthrough.

Link width and speed

Stock, the card links at PCIe Gen1 x4. The unlock raises the speed to Gen2. Width beyond x4 needs the capacitor mod: 24 capacitors (0402, 220 nF, X7R) on lanes 4-15. Speed and width are separate: the cap mod changes width only. Gen3 is fused off on the 8 GB card; on the 10 GB card it needs a RIR burn (see Unlocks).
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
Source: repo github.com/amoghmunikote/170th-Street by amoghmunikote on GitHub, posted by @markshowa, 2026-10-18

What you see	What it was	Fix	Status	Source
x4 after the cap mod, then x8 after cleaning the caps	Flux residue or a short or open on one lane pair	Check with a multimeter for shorts between the PCIe signal lines; clean and recheck	Reported	Source: @bigfoot1144, 2026-10-13 Source: @globalapathy, 2026-10-13
x8 after the cap mod; continuity from each cap top to the edge connector looked fine, no shorts	Cold joints on some caps	Test each cap against the specific finger it should reach. Tilt the card under the scope to see solder under the cap ends, or tap both sides of each cap with the iron. There are 8 pairs including the original 4, so the fault can be in any of them. Reflowing the suspect caps gave x16. Also confirm the slot is electrically x16.	Reported	Source: @anon33anon, 2026-10-15 Source: @ghostinthedatacenter_87281, 2026-10-15 Source: @nestaa51, 2026-10-15 Source: @globalapathy, 2026-10-15 Source: @chonsworth, 2026-10-15 Source: @cmdrwaka324, 2026-10-15
x4 or x8 on one card of a pair, no mod involved	The card was not fully seated in the slot	A spacer (folded cardboard) pushing the card into the slot gave the full width	Reported	Source: @cibernox, 2026-10-06
One slot trains x8 on a multi-slot board	The board shares lanes between neighbouring slots and the neighbour is occupied. Expected, not a fault.	Check the board manual for lane sharing	Reported	Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
PEX88096 switch board links to the host at x8 only	A SAS cable in the wrong port	Re-cable in the documented order; the board then ran at full width with P2P	Reported	Source: @domrockt2, 2026-10-15
One 10 GB card stays at Gen2 while the others reach Gen3	That card's on-die PCIe controller is physically damaged and cannot run Gen3 (it drops off the bus at Gen3). The other three cards ran Gen3 together on the same board.	None; run that card at Gen2	Reported	Source: repo github.com/lixiangwuxian/170hx_tips by lixiangwuxian on GitHub, posted by @lxtend, 2026-10-19 Source: @lxtend, 2026-10-19
Damaged PCIe edge contacts on arrival (a card from Alifond)	Shipping or handling damage	The card still passed stress tests after repeated insertion; long-term effect unknown	Reported	Source: @moliminous, 2026-10-10

Judge the link by LnkSta as the host sees it, not by NVML or LnkCap: LnkCap can be forced to advertise 8 GT/s while LnkSta stays at 2.5 GT/s.
Source: @darkmaniac7, 2026-07-28 (lab notes)
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
Source: repo github.com/HalfVulpes/CMP170HX-GPU by HalfVulpes on GitHub
The card's Gen2 advertisement window is open only about 0.4 s during driver init. If a host-side retrain catches it, NVML can keep reporting gen 1; check setpci -s <bdf> CAP_EXP+12.w on the host (0x1042 = 5 GT/s).
Source: repo github.com/ahnguyen17/ga100-fuse-re by ahnguyen17 on GitHub, posted by @exile_1ck, 2026-10-07
Source: @darkmaniac7, 2026-10-18 (lab notes)
Inside a VM the guest's config space shows the hypervisor's virtual link (VMware reports 5 GT/s x32 whatever the card does); read the real link on the host.
Source: @darkmaniac7, 2026-10-21 (lab notes, darkmaniac7/cmp170hx-esxi)
Gen2 does not train everywhere. Some Intel platforms never reached it across branches and slots while an AMD host did with no kernel options; the deciding factor is when the retrain fires. Some PCB revisions lock the Gen2 path (FAILED to set OPT_GEN23 in dmesg): 64 GB and the SM unlock still work there, Gen2 does not. Reported
Source: @darkmaniac7, 2026-09-24 (lab notes)
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
Source: repo github.com/rjohny55/cmp170hx-unlocker by rjohny55 on GitHub
A bounded second pass (function-level reset, then a second driver init and retrain) took one card on a Xeon E5 v4 host from Gen1 x16 to Gen2 x16; eight cards on an EPYC host reached Gen2 x16 on the first pass. Reported
Source: @darkmaniac7, 2026-09-24 (lab notes)
Source: repo github.com/rjohny55/cmp170hx-unlocker by rjohny55 on GitHub
Continuity measured from the top of each capacitor to the edge connector can look fine while a capacitor is not actually touching its pad; the top pad cannot be probed without scraping the trace. One owner staked a suspect capacitor with more solder; result pending.
Source: @froggao, 2026-10-27
Source: @globalapathy, 2026-10-27

BAR1: the 64 GB window

Stock, Resizable BAR on this card is limited to 64 MiB. The unlock can raise BAR1 to the full 64 GB, which BAR1-based P2P needs. Plain inference without P2P does not need it, and mailbox P2P works inside the stock 64 MB BAR1 with no ReBAR and no Above 4G Decoding.
Source: repo github.com/asm64-hooligan/cmpunlocker by asm64-hooligan on GitHub, posted by @rh3dai, 2026-10-04
Source: repo github.com/amoghmunikote/cmpunlocker by amoghmunikote on GitHub, posted by @rh3dai, 2026-10-08
Source: repo github.com/amoghmunikote/170th-Street by amoghmunikote on GitHub, posted by @markshowa, 2026-10-18

The 64 GB VRAM unlock, the Gen2 retrain and serving all work with a 64 MiB BAR1; BAR1 size only matters for P2P and static BAR1 mapping. Proven
Source: @darkmaniac7, 2026-10-19 (lab notes, darkmaniac7/cmp170hx-esxi)

BIOS and kernel settings

Above 4G Decoding and Resizable BAR in the BIOS are the prerequisites for the 64 GB BAR1.
Source: @globalapathy, 2026-10-19
The BIOS must offer a 64-bit MMIO window large enough for every card. Budget 64 GiB per card plus a small BAR3 and bridge alignment: eight cards need roughly 513 GiB.
Source: @darkmaniac7, 2026-10-26 (lab notes)
A working 3-card host behind PLX switches used intel_iommu=on iommu=pt pci=realloc on the kernel command line, Above 4G on and "MMIO high" set to 1024G in the BIOS. Reported
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
Hosts with several cards behind PLX PEX8747/PEX8749 switches needed two host-kernel patches: one fixes a bridge-window sizing bug in pbus_size_mem() so a 64 GB-aligned BAR1 plus a 32 MB BAR3 per card is budgeted (without it some cards silently get no BAR1), the other programs BAR1 to 64 GB before enumeration. That setup also used pci=hpmmioprefsize=2T and a large BIOS MMIO-high window; eight 64 GB BAR1s span 512 GB per root port.
Source: @darkmaniac7, 2026-09-12 (lab notes)
Source: repo github.com/Schaka/170hx-journey by Schaka on GitHub
Source: repo github.com/bayley/cmpunlocker by bayley on GitHub
Warning Behind PCIe switches, turn ACS redirect off for P2P: list the switch ports in pci=disable_acs_redir=... and check that lspci -vv | grep ACSCtl shows ReqRedir- CmpltRedir-. With ACS redirect on, BAR1 P2P across root ports lost 100% of a 256 MiB device-to-device copy with no error. Same-switch bandwidth also rose from 1.20 to 1.68 GB/s with it off.
Source: @darkmaniac7, 2026-10-19 (lab notes)
Source: repo github.com/bayley/cmpunlocker by bayley on GitHub
Source: repo github.com/lesj0610/cmpunlocker by lesj0610 on GitHub
The cmpunlocker installer adds intel_iommu=on or amd_iommu=on plus iommu=pt to the kernel command line by default (opt out with --no-iommu). IOMMU must also be enabled in the BIOS (VT-d or AMD-Vi).
Source: repo github.com/amoghmunikote/cmpunlocker by amoghmunikote on GitHub, posted by @rh3dai, 2026-10-08
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09

Boards where BAR1 failed

Board or host	What happened	Fix or workaround	Status	Source
ASUS P9X79 WS	Unlock failed. An AI assistant's explanation: the BIOS allocates only a 2-16 GB window above 4 GB, so a 64 GB BAR request fails.	The owner's assistant changed "some Ubuntu parameters" to raise the MMIOH size and it worked. The exact parameters were not posted.	Reported	Source: @globalapathy, 2026-10-19 Source: @fdelfini, 2026-10-19
Minisforum MS-01 with a PLX 88096 switch, 4 cards	BAR1 stuck at 64 MB with ReBAR enabled. MMIO is 256 GB, too small for 4 cards; the BIOS tried to place the BAR1s in a 512 MB section. The switch firmware did not have hotplug enabled, which is needed to remap BAR addressing.	Not resolved. Another owner runs 8 cards on MS-01 hosts with mailbox P2P.	Reported	Source: @replacedriver47, 2026-10-10 Source: @con3po7016, 2026-10-22
ASUS WRX80E	BAR1 stuck at 64 MB.	None posted.	Reported	Source: @trustinter, 2026-10-10
ASUS WRX90	BAR1 cannot be assigned ("no space"). An experimental BAR1 fork did not help.	None found; an AI assistant suggested asking ASUS for a BIOS change.	Reported	Source: @ghostinthedatacenter_87281, 2026-10-07
Two cards behind one PLX switch	The kernel resource tree was corrupted on every boot.	A mutex around the per-card BAR1 resize (`rebar-serialize.patch`, same fix as upstream cmpunlocker PR #59).	Reported	Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20

One member warns buyers of new boards to avoid ASUS for this reason.
Source: @ghostinthedatacenter_87281, 2026-10-22

When BAR1 cannot be fixed

Contested

Mailbox P2P runs at the 64 MB BAR1, so it is the usual workaround on boards that cannot map 64 GB.
Source: @ob88594, 2026-10-05
On an MS-01 with a PLX switch it reported working but no transfers went through.
Source: @replacedriver47, 2026-10-11

Whichever P2P path you use, check the data. The fork's own README documents a failure mode where cuMemcpyPeer reported success while 33,423,360 of 33,554,432 bytes came back wrong, so always run a content check (the fork ships tools/p2p-content-check.py) instead of trusting nvidia-smi topo. More on P2P in Driver and unlock failures.
Source: repo github.com/Morrowmake/cmpunlocker by Morrowmake on GitHub, posted by @trustinter, 2026-10-07

bar1-resize-unlock.patch has caused trouble on some hosts: behind VFIO it led to BAR2/MMU initialisation failures (BAR1 left small), and on an 8-card server with two PEX8796 switches it hung the board and was removed (mailbox P2P runs without it). Reported
Source: @darkmaniac7, 2026-10-19 (lab notes)
Source: repo github.com/Johnnydaboy/cmpunlocker by Johnnydaboy on GitHub
Source: repo github.com/cebeuq/glm53-cmp170hx-recipe by cebeuq on GitHub

Compute hangs in a passthrough VM after the BAR1 resize patch

In a Debian 13 VM, the unlock showed 64 GB and verify.sh passed, but any compute job hung, nvidia-smi stopped answering and the VM could not reboot cleanly. Removing the unlock stopped the hangs but put the card back to 8 GB. Building cmpunlocker without bar1-resize-unlock.patch (drop the bar1_resize entry from common/constants.yaml and the patch from PATCH_ORDER in driver/build.sh) stopped the hangs and kept all 64 GB. That patch writes the BAR1 XVE registers on 0x20C2 and 0x2082 cards and turns on NVreg_EnableResizableBar. Full write-up on Discord.

Reported
Source: @vipic, 2026-10-18

VMs and passthrough

Unlock on the hypervisor host, then pass the card through. Unlocking inside the VM was unreliable for one owner with two cards; unlocking on the Proxmox host worked. A second owner does the same and passes the card into a container. Proven
Source: @bismuth67, 2026-10-08
Source: @spiralglide, 2026-10-08
Proxmox VMs need SeaBIOS, not OVMF. OVMF (UEFI) gives RmInitAdapter and SEC2 failures that look like a failed unlock.
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
The GSP boot state must be restored before each VM start, or the guest logs "WPR2 already up". The cmpunlocker passthrough service does this when a card binds to vfio-pci; the Proxmox guide adds a pre-start hookscript because the bind event only fires on a real driver change.
Source: repo github.com/amoghmunikote/cmpunlocker by amoghmunikote on GitHub, posted by @rh3dai, 2026-10-08
Source: repo github.com/anoane/01-proxmox-dual-gpu-passthrough by anoane on GitHub, posted by @logy0, 2026-10-10
Source: @logy0, 2026-10-10
If you do not pass cards to VMs, disable cmpunlocker-passthrough.service: it blocks GPU resets.
Source: repo github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx by mannydelrio on GitHub, posted by @mrmannyatx, 2026-10-06
VFIO passthrough of the card did not work on a CachyOS (Arch) host, while the card ran fine on the host itself next to an RTX 4080 Super. Reported
Source: @ybryany, 2026-10-14
Gen2 under passthrough has its own feature-request thread ("Gen 2 Unlock not stable as PCIE Pass through Proxmox"). As of 2026-07-24 the memory and compute unlock worked in a VM but the Gen2 link change did not, and one guest hung with RCU stalls on the experimental Gen2 patch. The current cmpunlocker installs a host-side Gen2 service and a passthrough path; check the current state before relying on old reports.
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
Source: @logy0, 2026-10-10
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
Gen2 in a guest depends on the method: it did not train inside a VFIO guest for some, while later runbooks retrain the physical root port from the Proxmox host, and under ESXi the driver's own GSP boot path brings the link to Gen2 inside the guest.
Source: @darkmaniac7, 2026-09-24 (lab notes)
Source: @darkmaniac7, 2026-10-19 (lab notes, darkmaniac7/cmp170hx-esxi)
Under QEMU/VFIO the upstream in-guest unlock failed: the Booter answered 0x15 (ACR_ERROR_FLCN_REG_ACCESS) instead of the expected 0x31, and retries did not help, so upstream unlocks on the host and the guest runs a stock driver. The failure is not universal: a Proxmox fork and the ESXi lab unlock inside the guest. Reported
Source: @darkmaniac7, 2026-10-19 (lab notes)
Source: repo github.com/amoghmunikote/cmpunlocker by amoghmunikote on GitHub
A dual-CMP in-guest unlock under Proxmox (Q35, OVMF, driver 610.57.04) needed three changes: serialise the GSP boots (parallel boots raced), drop the FEAT_OVR_ECC_PLM write (on the second card it turned the expected Booter 0x31 into a fatal 0x15), and keep bar1-resize-unlock.patch off. Reported
Source: @darkmaniac7, 2026-09-04 (lab notes)
Source: repo github.com/Johnnydaboy/cmpunlocker by Johnnydaboy on GitHub
On KVM/libvirt hosts that switch cards between host and guest, the documented wedges are host-side: rmmod nvidia stuck in D state, unbind or bind hanging while vLLM holds the device; a reboot was the only fix.
Source: @darkmaniac7, 2026-10-19 (lab notes)
Source: repo github.com/18gogogo/170hx-hosttovm-passthrough by 18gogogo on GitHub

ESXi

The whole cmpunlocker core runs inside an ESXi 8 guest with no host-side unlock, and all four cards report 65536 MiB. The lab runs the unlock ported to driver 580.178.04. Proven
Source: @darkmaniac7, 2026-10-21 (lab notes, darkmaniac7/cmp170hx-esxi)
ESXi ignores Resizable BAR for passthrough devices and gives each card a 64 MiB BAR1 (Broadcom KB 312208). A small UEFI program run before the ESXi loader gave each card a physical 64 GiB BAR1 that ESXi keeps; see BAR1 64 GB on ESXi. The VM then needs pciPassthru.use64bitMMIO = TRUE and pciPassthru.64bitMMIOSizeGB = 1024 (384 was not enough for one 64 GiB card plus two 128 GiB Blackwells).
Source: @darkmaniac7, 2026-10-19 (lab notes, darkmaniac7/cmp170hx-esxi)
Source: @darkmaniac7, 2026-10-21 (lab notes, darkmaniac7/cmp170hx-esxi)
Every host power-on re-enumerates the cards cold at 64 MiB, so a normal host reboot after a successful 64 GiB boot throws the allocation away. Reported
Source: @darkmaniac7, 2026-10-21 (lab notes, darkmaniac7/cmp170hx-esxi)
Every VM power-on resets the card's internal ReBAR override: sysfs still shows a 64 GiB resource, but NVML reports a 256 MiB BAR1. A modprobe install hook that re-applies the override through the BAR0 config mirror before nvidia.ko loads fixes it; guest config-space writes never reach the device under VMware, which is why the hook goes through the mirror. Proven
Source: @darkmaniac7, 2026-10-18 (lab notes, darkmaniac7/cmp170hx-esxi)
Source: @darkmaniac7, 2026-10-19 (lab notes, darkmaniac7/cmp170hx-esxi)
A cold VM boot leaves a driverless card with memory decode off, so mirror reads return 0xffffffff: enable the device through sysfs first (memory decode only). udev runs the hook once per NVIDIA function at the same time, so serialise it with a lock, and keep nvidia.ko out of the initramfs or the hook is bypassed.
Source: @darkmaniac7, 2026-10-18 (lab notes, darkmaniac7/cmp170hx-esxi)
Warning P2P between CMP cards inside the ESXi guest does not work. Stock reports P2P unsupported and the mailbox path moves zero bytes. Static BAR1 P2P either failed init (RMForceStaticBar1=1 alone) or, with RMPcieP2PType=1 added, wedged UVM on the first plain CUDA context, twice, each needing a hard VM reset. The same cards pass 12 of 12 pairs on bare metal, and two Blackwells in the same VM do P2P fine. No working CMP P2P inside any hypervisor has been reported. Serve with NCCL_P2P_DISABLE=1.
Source: @darkmaniac7, 2026-10-19 (lab notes, darkmaniac7/cmp170hx-esxi)
Source: @darkmaniac7, 2026-10-21 (lab notes, darkmaniac7/cmp170hx-esxi)
Do not hide the hypervisor from the guest (hypervisor.cpuid.v0 = FALSE) to get around driver checks: the driver's VMware detection reads exactly those CPUID leaves, and the change affects every GPU in the VM.
Source: @darkmaniac7, 2026-10-21 (lab notes, darkmaniac7/cmp170hx-esxi)
One nvidia.ko drives every NVIDIA GPU in the VM, so a CMP module swap or a CMP-triggered wedge takes the other GPUs down too. Load a rebuilt driver by power-cycling the VM; in-guest rmmod / modprobe has wedged.
Source: @darkmaniac7, 2026-10-21 (lab notes, darkmaniac7/cmp170hx-esxi)
Source: @darkmaniac7, 2026-10-25 (lab notes, darkmaniac7/cmp170hx-esxi)
Wedges inside the VM: see Recovery.
Under VFIO there is no P2P between cards.
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20

eGPU enclosures, risers and the BMC

OCuLink eGPU from a Dell OptiPlex 7040m micro PC (M.2 to OCuLink adapter, external frame, separate 1000 W PSU): it worked at about 50 tok/s and 95% uptime, but the micro PC swapped and restarted under LLM load, remote desktop crashed and the OCuLink link was a bit flaky. Use a host with 32 GB or more. Reported
Source: @globalapathy, 2026-10-18
A Razer TB3 enclosure on an X299 host did not work; the owner blames X299's Thunderbolt support. Reported
Source: @_orrorin, 2026-10-18
An eGPU dock over USB4 was in use when a card started failing GSP init on every boot (see Xid errors, Xid 1); a native slot test was pending. Reported
Source: @mistyashhole, 2026-10-10
Behind a generic passive riser the BMC sees the card as an add-in card, not a GPU, and cannot read its temperature, so the BMC fan curve cannot follow it. Cards slotted directly in the board were read fine. Passive risers bring "inexplicable gremlins"; buy a reputable brand. Reported
Source: @fafinet, 2026-10-21
Source: @endsjeremy, 2026-10-21

Related pages