Virtual machines and passthrough - Revision history

ImportBot: Import: VM and passthrough update from the older history (2026-07-30 to 2026-10-03): host or guest unlock (contested), the upstream passthrough route, Proxmox problems and LXC, Unraid, XCP-ng, libvirt, nightdays' ESXi R7515 recipe, ESXi MMIO sizing (contested), P2P in a VM (warning); lab notes (ESXi 8, 4 cards), anoane/01-proxmox-dual-gpu-passthrough, PixelML/club-170hx, amoghmunikote/cmpunlocker passthrough branch, @.nightdays, @prometasfw, @ikyo, @bismuth67, @spiralglide, @

2026-11-01T02:56:33Z

Import: VM and passthrough update from the older history (2026-07-30 to 2026-10-03): host or guest unlock (contested), the upstream passthrough route, Proxmox problems and LXC, Unraid, XCP-ng, libvirt, nightdays' ESXi R7515 recipe, ESXi MMIO sizing (contested), P2P in a VM (warning); lab notes (ESXi 8, 4 cards), anoane/01-proxmox-dual-gpu-passthrough, PixelML/club-170hx, amoghmunikote/cmpunlocker passthrough branch, @.nightdays, @prometasfw, @ikyo, @bismuth67, @spiralglide, @

Show changes

ImportBot: ESXi and Proxmox passthrough of CMP 170HX cards from @darkmaniac7 lab notes (ESXi 8, 4 cards), anoane/01-proxmox-dual-gpu-passthrough (posted by @logy0), PixelML/club-170hx, and Discord reports by benoit, SpiralGlide, ybryany, Froggo, ViPiC and others; gap-fill from #general 2026-10-25 (CMP plus V100), @zingy90210, @nathang6866

2026-10-30T10:02:44Z

ESXi and Proxmox passthrough of CMP 170HX cards from @darkmaniac7 lab notes (ESXi 8, 4 cards), anoane/01-proxmox-dual-gpu-passthrough (posted by @logy0), PixelML/club-170hx, and Discord reports by benoit, SpiralGlide, ybryany, Froggo, ViPiC and others; gap-fill from #general 2026-10-25 (CMP plus V100), @zingy90210, @nathang6866

← Older revision		Revision as of 10:02, 30 October 2026
Line 55:		Line 55:
	* GalyGames: the unlock works on his Proxmox system (Debian based), and he reproduced Plean's 2-card Qwen3.8 Flash Next recipe on Proxmox. {{Source\|handle=galygious\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557452515354607679}}<br />{{Source\|handle=galygious\|date=2026-10-10\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1558336191634612340}}		* GalyGames: the unlock works on his Proxmox system (Debian based), and he reproduced Plean's 2-card Qwen3.8 Flash Next recipe on Proxmox. {{Source\|handle=galygious\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557452515354607679}}<br />{{Source\|handle=galygious\|date=2026-10-10\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1558336191634612340}}
	* ElectroHobo ran 170HX cards and V100s in one Dell R730 under Proxmox while first testing, in separate VMs rather than one model across both. {{Source\|handle=nathang6866\|date=2026-10-25\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563727993896177735}} See [[Dell PowerEdge R730]].		* ElectroHobo ran 170HX cards and V100s in one Dell R730 under Proxmox while first testing, in separate VMs rather than one model across both. {{Source\|handle=nathang6866\|date=2026-10-25\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563727993896177735}} See [[Dell PowerEdge R730]].
			** He answered PJ, who had asked owners of both cards to try combining them now that 170HX prices are up. He thinks the only way to run one model across both is two VMs plus software that shards the model across machines. He also has four 32 GB SXM2 V100s (no four-card board yet) and five 170HX in a Supermicro server, and plans separate models on each group. Mixed-GPU vLLM: see [[Operating systems and kernels#Mixed GPUs in one host]]. {{Source\|handle=zingy90210\|date=2026-10-25\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563725299273957438}}<br />{{Source\|handle=nathang6866\|date=2026-10-25\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563727993896177735}}

	== ESXi ==		== ESXi ==

ImportBot: ESXi and Proxmox passthrough of CMP 170HX cards from @darkmaniac7 lab notes (ESXi 8, 4 cards), anoane/01-proxmox-dual-gpu-passthrough (posted by @logy0), PixelML/club-170hx, and Discord reports by benoit, SpiralGlide, ybryany, Froggo, ViPiC and others

2026-10-30T08:47:27Z

← Older revision		Revision as of 08:47, 30 October 2026
Line 49:		Line 49:
	* Froggo: "mine wouldnt work in proxmox". No details. {{Source\|handle=froggao\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561443221832208445}}		* Froggo: "mine wouldnt work in proxmox". No details. {{Source\|handle=froggao\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561443221832208445}}
	* ybryany could not get hardware VFIO passthrough of the CMP to work with a CachyOS (Arch) host; the card works on the host itself beside an RTX 4080 Super. {{Source\|handle=ybryany\|date=2026-10-14\|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1559748494875426837}}<br />{{Source\|handle=ybryany\|date=2026-10-14\|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1559748535690203160}}		* ybryany could not get hardware VFIO passthrough of the CMP to work with a CachyOS (Arch) host; the card works on the host itself beside an RTX 4080 Super. {{Source\|handle=ybryany\|date=2026-10-14\|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1559748494875426837}}<br />{{Source\|handle=ybryany\|date=2026-10-14\|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1559748535690203160}}
	* Compute hangs in a ~~passthrough VM after~~ the unlock~~: A longer~~ write-up ~~by @vipic exists~~; see [https://discord.com/channels/1518634379604136178/1518692084729511967/1561427071157272727 Discord].		* '''Compute hangs after the unlock: drop <code><nowiki>bar1-resize-unlock.patch</nowiki></code>.''' ViPiC (Debian 13.4 guest with PCI passthrough, hypervisor not named, nvidia-open 615.71.09, also tried 610.43.02): after the unlock <code><nowiki>nvidia-smi</nowiki></code> showed 64 GB and <code><nowiki>./verify.sh</nowiki></code> passed, but any compute workload hung the process, <code><nowiki>nvidia-smi</nowiki></code> stopped responding and the VM could not reboot cleanly; <code><nowiki>./remove.sh --yes</nowiki></code> stopped the hangs but returned the card to 8 GB. The fix was to build the unlock without <code><nowiki>bar1-resize-unlock.patch</nowiki></code> (remove the <code><nowiki>bar1_resize</nowiki></code> entry from <code><nowiki>common/constants.yaml</nowiki></code> and the patch from <code><nowiki>PATCH_ORDER</nowiki></code> in <code><nowiki>driver/build.sh</nowiki></code>); that patch writes the card's BAR1 override registers and enables <code><nowiki>NVreg_EnableResizableBar</nowiki></code>. Without it there were no hangs and all 64 GB stayed. It had worked two days earlier, before a bootloader repair, a kernel update and a reinstall of the unlock. This matches the ESXi finding that the BAR1 resize is a Linux-host path that does not work inside a VM. Summary of his write-up; see [https://discord.com/channels/1518634379604136178/1518692084729511967/1561427071157272727 Discord]. {{Source\|handle=vipic\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561427071157272727}}<br />{{Source\|handle=vipic\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561429038361223329}}<br />{{Source\|handle=vipic\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561436105767788685}}<br />{{Source\|handle=vipic\|date=2026-10-19\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561714793637744711}}

	=== Proxmox setups people run ===		=== Proxmox setups people run ===

ImportBot: ESXi and Proxmox passthrough of CMP 170HX cards from @darkmaniac7 lab notes (ESXi 8, 4 cards), anoane/01-proxmox-dual-gpu-passthrough (posted by @logy0), PixelML/club-170hx, and Discord reports by benoit, SpiralGlide, ybryany, Froggo, ViPiC and others

2026-10-30T08:43:03Z

← Older revision		Revision as of 08:43, 30 October 2026
Line 57:		Line 57:

	== ESXi ==		== ESXi ==
			Step-by-step for the 64 GB BAR1: [[BAR1 64 GB on ESXi]]. Code and write-up: [https://github.com/darkmaniac7/cmp170hx-esxi darkmaniac7/cmp170hx-esxi]. {{Source\|handle=darkmaniac7\|date=2026-10-21\|repo=https://github.com/darkmaniac7/cmp170hx-esxi\|note=lab notes}}

	Four 8 GB cards (<code><nowiki>10de:20c2</nowiki></code>, Hynix HBM, VBIOS 92.00.67.00.01) were passed through from VMware ESXi 8.0.3 into one Ubuntu 24.04 VM, next to other newer NVIDIA GPUs in the same guest, on a single-socket EPYC server with the slots bifurcated x8/x8 so every card sits behind its own root port. Measured in October 2026. {{Source\|handle=darkmaniac7\|date=2026-10-21\|note=lab notes}}		Four 8 GB cards (<code><nowiki>10de:20c2</nowiki></code>, Hynix HBM, VBIOS 92.00.67.00.01) were passed through from VMware ESXi 8.0.3 into one Ubuntu 24.04 VM, next to other newer NVIDIA GPUs in the same guest, on a single-socket EPYC server with the slots bifurcated x8/x8 so every card sits behind its own root port. Measured in October 2026. {{Source\|handle=darkmaniac7\|date=2026-10-21\|note=lab notes}}

Line 97:		Line 99:
	# programs the parent port's prefetchable window;		# programs the parent port's prefetchable window;
	# restores the parent port's COMMAND register;		# restores the parent port's COMMAND register;
	then starts the existing ESXi loader with its original load options. It adds about 3.5 to 4 minutes to a host boot and has to run on every host boot. Two traps: the CYA store alone already moves CTRL from <code><nowiki>0x801</nowiki></code> to <code><nowiki>0x1001</nowiki></code>, so do not expect "unchanged"; and the card's extended capability chain has a null capability (ID 0) at <code><nowiki>0x900</nowiki></code>, so a strict decoder never reaches the ReBAR block at <code><nowiki>0xbb0</nowiki></code>. ESXi's own view then reports BAR size 68719476736 and the guest <code><nowiki>resource1</nowiki></code> is 64 GiB. ~~Ask @darkmaniac7~~ on ~~Discord for~~ the ~~writer before improvising one~~. {{Source\|handle=darkmaniac7\|date=2026-10-21\|note=lab notes}}		then starts the existing ESXi loader with its original load options. It adds about 3.5 to 4 minutes to a host boot and has to run on every host boot. Two traps: the CYA store alone already moves CTRL from <code><nowiki>0x801</nowiki></code> to <code><nowiki>0x1001</nowiki></code>, so do not expect "unchanged"; and the card's extended capability chain has a null capability (ID 0) at <code><nowiki>0x900</nowiki></code>, so a strict decoder never reaches the ReBAR block at <code><nowiki>0xbb0</nowiki></code>. ESXi's own view then reports BAR size 68719476736 and the guest <code><nowiki>resource1</nowiki></code> is 64 GiB. The full procedure, build notes and risks are on [[BAR1 64 GB on ESXi]]; the code is in [https://github.com/darkmaniac7/cmp170hx-esxi darkmaniac7/cmp170hx-esxi]. {{Source\|handle=darkmaniac7\|date=2026-10-21\|repo=https://github.com/darkmaniac7/cmp170hx-esxi\|note=lab notes}}

	=== VM settings ===		=== VM settings ===

ImportBot: ESXi and Proxmox passthrough of CMP 170HX cards from @darkmaniac7 lab notes (ESXi 8, 4 cards), anoane/01-proxmox-dual-gpu-passthrough (posted by @logy0), PixelML/club-170hx, and Discord reports by benoit, SpiralGlide, ybryany, Froggo, ViPiC and others

2026-10-30T08:05:20Z

New page

{{Status|Reported}} {{Cards|both}}

Running CMP 170HX cards inside a virtual machine works, with more moving parts than bare metal. The short version:
* '''Unlock where the NVIDIA driver runs.''' Under Proxmox/QEMU most people unlock on the host and pass the unlocked card in. Under ESXi the unlock runs inside the guest.
* '''64 GB VRAM works in a VM.''' It does not need a big BAR1.
* '''Gen2 works in a VM''', but how it is done differs per hypervisor, and the Gen2 path has caused guest hangs under Proxmox for some people.
* '''64 GB BAR1 in a VM is hard''' (ESXi needs a pre-boot step on every host boot).
* '''P2P between CMPs inside a VM has not been shown to work''' by anyone in these sources.
* '''Bare metal is much simpler.''' {{Source|handle=darkmaniac7|date=2026-10-29|note=lab notes}}

See also [[Server BIOS setup]] (case D), [[Operating systems and kernels]], [[Gen2 link retrain]], [[Resizable BAR and BAR1 64 GB]], [[P2P]].

== Where to run the unlock ==
Asked whether the unlock must happen on the hypervisor, benoit tried both on Proxmox: unlocking fully inside the VM gave him trouble unlocking both cards, while unlocking on the Proxmox host worked and is what he kept. {{Source|handle=jack0x01|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557804623996387329}}<br />{{Source|handle=bismuth67|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557833870626529311}} SpiralGlide does the same: unlock on the host and pass the card into a container that runs Docker containers. {{Status|Proven}} {{Source|handle=spiralglide|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557834451499880519}}

The PixelML club writes the same rule from its VFIO baseline: under VFIO the unlock can be applied host-side so the guest runs the stock driver; their second host showed Gen2 x16 and 64 GB in the guest that way. {{Source|handle=djpandaman|date=2026-10-24|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563629622673936384|repo=https://github.com/PixelML/club-170hx}} (docs/INSTALLATION.md)

The cmpunlocker upstream "Virtual Passthrough" route also unlocks on the host, because under QEMU/VFIO the in-guest Booter path returns <code><nowiki>0x15</nowiki></code> (ACR_ERROR_FLCN_REG_ACCESS) and the guest ends up on a stock driver. Under ESXi that error did not appear and the whole v0.5 core ran inside the guest. {{Source|handle=darkmaniac7|date=2026-10-21|note=lab notes}}

aidoc confirmed his register probe script can be run from inside a VM. {{Source|handle=sekkit_75019|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1556290013497528422}}

== Proxmox (QEMU/KVM, VFIO) ==

=== A working Proxmox host recipe ===
The most complete public Proxmox write-up is the anoane repo (MIT license), linked by Logy as the guide he followed: {{Source|handle=logy0|date=2026-10-10|link=https://discord.com/channels/1518634379604136178/1540599965007478854/1558505023187714068|repo=https://github.com/anoane/01-proxmox-dual-gpu-passthrough}}. It passes an RTX PRO 6000 Blackwell and an 8 GB CMP 170HX unlocked to 64 GB into one Ubuntu 24.04 guest. The author calls it a point-in-time transcription of one host, not a turnkey script. Key points from the repo (commit <code><nowiki>865a608</nowiki></code>, 2026-09-22):
* Host: Proxmox VE 9.2.x, kernel 7.0.x-pve, '''Secure Boot disabled''' (<code><nowiki>mokutil --sb-state</nowiki></code> must say disabled), AMD Ryzen 9 9950X3D. cmpunlocker pinned at commit <code><nowiki>76f0954</nowiki></code> on the host with driver 610.43.02. The guest runs Ubuntu 24.04.4, kernel 6.8.0-139-generic, the '''stock''' NVIDIA driver 610.43.02; the unlock lives on the host. The guest sees the CMP as "NVIDIA Graphics Device" with 65,536 MiB, so match it by PCI id <code><nowiki>10de:20c2</nowiki></code>, not by name.
* Host kernel command line: <code><nowiki>quiet amd_iommu=on iommu=pt</nowiki></code>; modules <code><nowiki>vfio vfio_iommu_type1 vfio_pci vfio_virqfd</nowiki></code>.
* <code><nowiki>options vfio-pci disable_idle_d3=1</nowiki></code>: if vfio idles the CMP into D3, BAR0 reads all ones and the card disappears.
* The CMP is armed and bound by cmpunlocker's boot service, not by the <code><nowiki>vfio-pci ids=</nowiki></code> list.
* VM: <code><nowiki>machine q35</nowiki></code> (required), <code><nowiki>bios seabios</nowiki></code>, <code><nowiki>cpu host</nowiki></code>, <code><nowiki>balloon 0</nowiki></code>, CMP passed as <code><nowiki>hostpci1: 0000:06:00.0,pcie=1,rombar=0</nowiki></code>. The CMP shared an IOMMU group with a PCIe bridge, so the whole group's function set is passed.
* A pre-start hookscript re-runs cmpunlocker's <code><nowiki>gsp-restore</nowiki></code> for cards on vfio-pci before every VM start, because the vfio bind event only fires on a real driver change and later VM starts would otherwise skip the restore.
* A VM that is killed rather than shut down leaves the GSP boot state latched and the next VM gets no GPU; only <code><nowiki>./tools/passthrough.sh restore <bdf></nowiki></code> clears it. <code><nowiki>cmp_no_bus_reset.ko</nowiki></code> blocks the bus reset vfio does at bind, which would otherwise erase the unlock.
* PCIe: the card trains Gen1 at boot. A host-side <code><nowiki>gen2-hammer</nowiki></code> script walks the upstream bridge and retrains with <code><nowiki>setpci</nowiki></code> until LnkSta reads Gen2, run by a <code><nowiki>gen2.service</nowiki></code> before <code><nowiki>basic.target</nowiki></code>. The card was confirmed Gen2 x16 in a PCIe 5.0 x16 slot; in the documented build it sits in a PCIe 3.0 x1 slot and measured about 0.38 GB/s.
From the repo README and vm/ai-vm.conf: {{Source|handle=logy0|date=2026-10-10|link=https://discord.com/channels/1518634379604136178/1540599965007478854/1558505023187714068|repo=https://github.com/anoane/01-proxmox-dual-gpu-passthrough}}

=== PixelML club VFIO baseline ===
The club's earlier VFIO baseline (docs/INSTALLATION.md): Ubuntu Server 22.04 guest, Linux 6.8, '''Q35 + SeaBIOS''' (OVMF/UEFI repeatedly gave <code><nowiki>RmInitAdapter</nowiki></code> and SEC2 failures; the same cards initialized under SeaBIOS), no <code><nowiki>x-vga=1</nowiki></code> on a compute-only card, NVIDIA 610.43.03 open module, cmpunlocker <code><nowiki>7019bc2</nowiki></code> with <code><nowiki>--profile=8gb</nowiki></code>. The experimental Gen2 patch was '''excluded''' because it caused guest and QEMU hangs and RCU stalls. Trap: a stock DKMS copy under <code><nowiki>updates/dkms</nowiki></code> shadowed the patched module and the card showed 8 GiB; fix with <code><nowiki>dkms remove</nowiki></code>, <code><nowiki>depmod -a</nowiki></code>, <code><nowiki>update-initramfs -u</nowiki></code>, confirm <code><nowiki>modinfo -n nvidia</nowiki></code> points at the cmpunlocker module, then a full guest shutdown. Under VFIO they had no P2P. The club later moved to bare-metal Proxmox VE 9.2 (kernel 7.0.2-6-pve) serving from a GPU LXC that shares the host driver. {{Source|handle=djpandaman|date=2026-10-24|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563629622673936384|repo=https://github.com/PixelML/club-170hx}}

=== Gen2 stability under Proxmox passthrough ===
A feature-request thread on the Discord is titled "Gen 2 Unlock not stable as PCIE Pass through Proxmox". Its only messages are Logy's: he runs the cards "in a vm : proxmox & qemu" and points anyone still having passthrough trouble to the anoane guide above. {{Source|handle=logy0|date=2026-10-10|link=https://discord.com/channels/1518634379604136178/1540599965007478854/1558454719297486903}}<br />{{Source|handle=logy0|date=2026-10-10|link=https://discord.com/channels/1518634379604136178/1540599965007478854/1558505023187714068}} Whether Gen2 is stable for him after following it is not stated. {{Status|Reported}}

What else is known:
* The PixelML club excluded the experimental in-guest Gen2 patch after guest hangs and RCU stalls (above).
* The anoane host retrains Gen2 from the host side (<code><nowiki>gen2-hammer</nowiki></code>) and reports Gen2 working.
* In a register review of a Gen3 patch, ropuls noted that in an earlier sweep the value <code><nowiki>0xA</nowiki></code> in <code><nowiki>XVE_FUSE_OVERRIDE</nowiki></code> eventually wedged the function under VFIO while <code><nowiki>0x6</nowiki></code> (the working Gen2 path) was stable. {{Source|handle=ropuls|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1557885155501277256}}
* Under ESXi, Gen2 works from inside the guest (below).

=== Problems reported under Proxmox ===
* Froggo: "mine wouldnt work in proxmox". No details. {{Source|handle=froggao|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561443221832208445}}
* ybryany could not get hardware VFIO passthrough of the CMP to work with a CachyOS (Arch) host; the card works on the host itself beside an RTX 4080 Super. {{Source|handle=ybryany|date=2026-10-14|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1559748494875426837}}<br />{{Source|handle=ybryany|date=2026-10-14|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1559748535690203160}}
* Compute hangs in a passthrough VM after the unlock: A longer write-up by @vipic exists; see [https://discord.com/channels/1518634379604136178/1518692084729511967/1561427071157272727 Discord].

=== Proxmox setups people run ===
* ikyo: six cards, one machine with four and another with two under Proxmox. {{Source|handle=ikyo|date=2026-10-26|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1564316034360672329}}
* GalyGames: the unlock works on his Proxmox system (Debian based), and he reproduced Plean's 2-card Qwen3.8 Flash Next recipe on Proxmox. {{Source|handle=galygious|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557452515354607679}}<br />{{Source|handle=galygious|date=2026-10-10|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1558336191634612340}}
* ElectroHobo ran 170HX cards and V100s in one Dell R730 under Proxmox while first testing, in separate VMs rather than one model across both. {{Source|handle=nathang6866|date=2026-10-25|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563727993896177735}} See [[Dell PowerEdge R730]].

== ESXi ==
Four 8 GB cards (<code><nowiki>10de:20c2</nowiki></code>, Hynix HBM, VBIOS 92.00.67.00.01) were passed through from VMware ESXi 8.0.3 into one Ubuntu 24.04 VM, next to other newer NVIDIA GPUs in the same guest, on a single-socket EPYC server with the slots bifurcated x8/x8 so every card sits behind its own root port. Measured in October 2026. {{Source|handle=darkmaniac7|date=2026-10-21|note=lab notes}}

{| class="wikitable"
! What !! Status on ESXi 8 passthrough !! Notes
|-
| 64 GB VRAM unlock, inside the guest || works || cmpunlocker v0.5 core ported onto open modules 580.178.04. Does not need a big BAR1.
|-
| 64 GiB BAR1 || works, but ESXi never does it for you || needs a small UEFI app that runs before ESXi on every host boot, plus a guest-side hook on every VM power-on
|-
| PCIe Gen2 || works, guest-side retrain is enough || cards come up Gen1 x8 stock, Gen2 x8 after
|-
| PCIe Gen3 || fused off on these 8 GB cards || the software fuse override path is itself fused off
|-
| 74 SMs || works || port of upstream PR #55 onto 580
|-
| P2P between CMPs in the VM || no || static BAR1 P2P init fails or wedges UVM in the guest; the same cards do 12/12 pairs on bare metal
|-
| Serving || 4 cards run GLM-5.3-Flash W4A16 at recipe-book parity in PP4, about 0.75x in TP4 || the TP4 gap is the host-staged all-reduce
|}
{{Source|handle=darkmaniac7|date=2026-10-21|note=lab notes}}

=== Unlock inside the guest ===
* The six v0.5 core patches were re-applied onto pristine nvidia-open 580.178.04, because 615.71.09 plus cmpunlocker made other GPUs in the same VM hit <code><nowiki>krcWatchdog: GPU is probably locked</nowiki></code> and then wedged UVM; NVIDIA's 615.71.09 notes list that symptom as a known issue.
* Each boot logs 48 <code><nowiki>Booter failed ... 0x31</nowiki></code> lines. They are noise, not a failure.
* One nvidia.ko drives every GPU in the VM. A CMP module swap or wedge takes the other GPUs down with it.
* Hold the kernel and every nvidia package: an upgrade silently restores stock modules and the cards drop to 8 GB.
{{Source|handle=darkmaniac7|date=2026-10-21|note=lab notes}}

=== Why ESXi cannot give you a 64 GB BAR1 ===
At every host power-on the card advertises a 64 MiB BAR1: ReBAR CAP at config <code><nowiki>0xbbc</nowiki></code> reads <code><nowiki>0x400</nowiki></code> and CTRL at <code><nowiki>0xbc0</nowiki></code> reads <code><nowiki>0x601</nowiki></code>. Firmware sizes the BAR to that, ESXi enumerates whatever firmware assigned, and ESXi never writes the vendor override. Broadcom's VMDirectPath notes say resizable BAR is unsupported for passthrough and that devices get reset at VM power-on. Raising <code><nowiki>pciPassthru.64bitMMIOSizeGB</nowiki></code> only reserves guest address space; it cannot grow the physical BAR. The upstream <code><nowiki>bar1-resize-unlock.patch</nowiki></code> is a Linux-host path. {{Source|handle=darkmaniac7|date=2026-10-21|note=lab notes}}

=== What worked: resize before ESXi boots ===
A small UEFI application, booted from USB (IP-KVM virtual media and a BMC boot override), runs after firmware PCI enumeration and before the ESXi loader. For each card it reserves a 64 GiB + 32 MiB span (<code><nowiki>0x1002000000</nowiki></code>) at 64 GiB alignment inside the root bridge aperture, then, with each store read back:
# turns MEM and bus mastering off on the card and its parent port;
# writes CYA <code><nowiki>0x724 = 0x30</nowiki></code>;
# writes CFG <code><nowiki>0xdcc = 0x8000000a</nowiki></code> (CAP <code><nowiki>0xbbc</nowiki></code> then goes <code><nowiki>0x400</nowiki></code> to <code><nowiki>0x1ffc00</nowiki></code>);
# sets ReBAR CTRL <code><nowiki>0xbc0</nowiki></code> to size code 16 (64 GiB), result <code><nowiki>0x1001</nowiki></code>;
# programs BAR1 and BAR3 (BAR3 is 32 MiB at base + 64 GiB);
# programs the parent port's prefetchable window;
# restores the parent port's COMMAND register;
then starts the existing ESXi loader with its original load options. It adds about 3.5 to 4 minutes to a host boot and has to run on every host boot. Two traps: the CYA store alone already moves CTRL from <code><nowiki>0x801</nowiki></code> to <code><nowiki>0x1001</nowiki></code>, so do not expect "unchanged"; and the card's extended capability chain has a null capability (ID 0) at <code><nowiki>0x900</nowiki></code>, so a strict decoder never reaches the ReBAR block at <code><nowiki>0xbb0</nowiki></code>. ESXi's own view then reports BAR size 68719476736 and the guest <code><nowiki>resource1</nowiki></code> is 64 GiB. Ask @darkmaniac7 on Discord for the writer before improvising one. {{Source|handle=darkmaniac7|date=2026-10-21|note=lab notes}}

=== VM settings ===
<code><nowiki>pciPassthru.use64bitMMIO = TRUE</nowiki></code>, and <code><nowiki>pciPassthru.64bitMMIOSizeGB</nowiki></code> had to go from 384 to '''1024''' for four cards plus the other GPUs' BARs (with one 64 GiB card the guest EFI already refused the MMIO at 384). {{Source|handle=darkmaniac7|date=2026-10-21|note=lab notes}}

=== BAR1 inside the guest: the 256 MiB trap ===
After the host had 64 GiB BAR1s, the first VM boot showed <code><nowiki>resource1</nowiki></code> = 64 GiB in sysfs but NVML said BAR1 Total 256 MiB. Every VM power-on resets the card's vendor override while the host allocation survives, and GSP reports what the ReBAR block says. Fix: a modprobe <code><nowiki>install nvidia</nowiki></code> hook that runs before nvidia.ko loads, maps one page of <code><nowiki>resource0</nowiki></code> at <code><nowiki>0x88000</nowiki></code> (the config mirror; guest config-space writes are virtualised and never reach the device), checks the reset profile, writes <code><nowiki>0x724 = 0x30</nowiki></code> then <code><nowiki>0xdcc = 0x8000000a</nowiki></code>, and requires CAP <code><nowiki>0x1ffc00</nowiki></code> / CTRL <code><nowiki>0x1001</nowiki></code> on read-back. Three lessons: a cold VM boot leaves memory decode off, so enable MEM (never bus master) first; udev runs the hook once per NVIDIA function concurrently, so take a lock; and never retry, a wrong read-back means a VM power cycle. Keep nvidia.ko out of the initramfs and never <code><nowiki>insmod</nowiki></code> by hand, or the hook is bypassed. After that NVML reports 65536 MiB BAR1 on all four cards. {{Source|handle=darkmaniac7|date=2026-10-21|note=lab notes}}

=== Gen2 and Gen3 under ESXi ===
Stock, the cards train Gen1 x8 behind Gen4 x8 root ports. cmpunlocker's Gen2 retrain, ported as a module parameter, brings all four to '''Gen2 x8''' inside the VM; the root-port LnkCtl2 write is a no-op in a VM (it hits VMware's virtual bridge) and turned out to be unnecessary. Gen3 is closed on these 8 GB cards: <code><nowiki>DISABLE_SW_OVERRIDE_STATUS</nowiki></code> (<code><nowiki>0x820084</nowiki></code>) reads 1 and <code><nowiki>OPT_DISABLE_GEN3_SPEED</nowiki></code> (<code><nowiki>0x820250</nowiki></code>) reads 1 on all four. One attempt through the XP/XVE registers left a card at Gen1 x8 until a VM power cycle re-ran the Gen2 retrain. {{Source|handle=darkmaniac7|date=2026-10-21|note=lab notes}}

=== P2P inside the VM: not working ===
* Stock, GSP reports P2P not supported for the CMP device IDs and <code><nowiki>cudaDeviceCanAccessPeer</nowiki></code> is false on all 12 directed pairs.
* <code><nowiki>RMForceStaticBar1=1</nowiki></code> alone fails init: the static map is placed at offset 512 MiB and the framebuffer overruns the 64 GiB BAR1 (<code><nowiki>kbusEnableStaticBar1Mapping_TU102: Failed to create the static bar1 mapping</nowiki></code>).
* <code><nowiki>RMForceStaticBar1=1;RMPcieP2PType=1</nowiki></code> inits cleanly and NVML shows 64 GiB, but the first plain CUDA context never exits and UVM wedges (hard VM reset, twice).
* RM only detects VMware through CPUID leaves; with <code><nowiki>hypervisor.cpuid.v0 = FALSE</nowiki></code> that detection is false. Do not gate anything on hypervisor detection in a VM that hides it.
* Other GPUs in the same VM do P2P with each other (<code><nowiki>pciPassthru.allowP2P</nowiki></code> and <code><nowiki>relaxACSforP2P</nowiki></code> set), so ESXi's peer transport is not the blocker. The reading is that it is the ESXi mapping of a 64 GiB BAR1 under CUDA, not the cards; that is a reading, not a root cause. {{Status|Theory}}
* The same four cards booted into native Linux on the same host passed 12/12 directed P2P pairs.
Serving runs with <code><nowiki>NCCL_P2P_DISABLE=1</nowiki></code> and a host-staged all-reduce. {{Source|handle=darkmaniac7|date=2026-10-21|note=lab notes}}

=== Serving numbers under ESXi ===
GLM-5.3-Flash W4A16 with the DFlash2 drafter on the Morrowmake vLLM fork (engine <code><nowiki>819146023f</nowiki></code>, recipe 1.8.0), cards at 180 W, 74-SM driver; tok/s structured / code / prose:
{| class="wikitable"
! Layout !! 1 user !! 8 users aggregate !! 32k cold prefill
|-
| 4x CMP, PP4 (ESXi, Gen2 x8, no P2P) || 201 / 149 / 97 || 664 / 548 / 477 || 6.8 to 7.0k tok/s
|-
| 4x CMP, TP4 (ESXi, Gen2 x8, no P2P) || 361 / 284 / 199 || 697 / 587 / 529 || 1.6 to 1.75k tok/s
|-
| recipe book PP4 (x16, P2P on) || 232 / 210 / 100 || 664 / 609 / 469 || 7.4k tok/s
|-
| recipe book TP4 (x16, P2P on) || 483 / 442 / 232 || 944 / 835 / 667 || 3.2k tok/s
|}
PP4 lands on the recipe book within noise on decode; TP4 is about 0.75x on decode and 0.55x on prefill, which is the all-reduce going through host memory over Gen2 x8 instead of P2P over x16. {{Source|handle=darkmaniac7|date=2026-10-21|note=lab notes}}

=== ESXi traps ===
* <code><nowiki>nvidia-persistenced</nowiki></code> is down after a VM power cycle; start it before any <code><nowiki>docker run</nowiki></code> that uses the NVIDIA runtime.
* Power caps reset to 250 W on every module load. Set them after.
* Only one <code><nowiki>NVreg_RegistryDwordsPerDevice</nowiki></code> line counts; a second file silently replaces the first.
* During a UVM wedge do not run nvidia-smi: every new caller blocks in D state. Hard reset the VM after restoring known-good modules on disk.
* Never follow a successful 64 GiB host boot with a "normal" reboot just to check something; you lose the allocation and have to run the pre-boot step again.
{{Source|handle=darkmaniac7|date=2026-10-21|note=lab notes}}

== Other hypervisor notes ==
* '''vGPU''' is in progress in cmpunlocker: Amogh announced on 2026-10-26 that vGPU is close to working, on the [https://github.com/amoghmunikote/cmpunlocker/tree/vGPU-and-others vGPU-and-others] branch, not in main. {{Status|Theory}} {{Source|handle=_j3ns|date=2026-10-26|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1564421429703737355}} See [[Unlocks]].
* '''Mixed host.''' A CMP in a desktop that also drives a gaming GPU works on bare metal (ybryany, CachyOS, RTX 4080 Super), but the global Gen2 registry keys would also force the other card to Gen2, so he left them off. {{Source|handle=ybryany|date=2026-10-14|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1559748433353384017}} See [[Operating systems and kernels#Mixed GPUs in one host]].
* '''Agents in VMs.''' Not a passthrough issue, but people running agent workloads in VMs on these hosts report models probing networks and acting destructively; sandbox them. {{Source|handle=segmond|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1561965044529496085}}

== Open questions ==
* Is Gen2 stable under Proxmox passthrough with the current cmpunlocker, using a host-side retrain? Only the thread title and the anoane recipe speak to it.
* P2P between CMPs inside any hypervisor: no public report of it working.
* TRAP31 mailbox P2P inside a VM, a smaller static BAR1 window, and <code><nowiki>hypervisor.cpuid.v0 = TRUE</nowiki></code> under ESXi are untested. {{Source|handle=darkmaniac7|date=2026-10-21|note=lab notes}}

{{ImportFooter}}
[[Category:Platforms]]
[[Category:8 GB]]
[[Category:10 GB]]