Jump to content
Toggle search
Toggle menu
CMP Wiki
Toggle preferences menu
Toggle personal menu
Not logged in
Your IP address will be publicly visible if you make any edits.

BAR1 64 GB on ESXi

From CMP Wiki
More actions

Reported 8 GB

How to give CMP 170HX cards that VMware ESXi passes through to a Linux guest a real 64 GB BAR1. ESXi does not do it by itself. It takes two parts: a small UEFI program that resizes BAR1 on the host before ESXi boots, and a guest-side hook that restores the card's override after every VM power-on. Everything here was measured on one host in October 2026; the code, the build files and the full write-up are in darkmaniac7/cmp170hx-esxi. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)

For passthrough in general (Proxmox, the in-guest unlock, Gen2, P2P) see Virtual machines and passthrough. For BAR1 on bare metal see Resizable BAR and BAR1 64 GB and Server BIOS setup.

Do you need it?

  • The 64 GB VRAM unlock does not need a big BAR1. The VRAM unlock, the Gen2 retrain and serving all work with the 64 MB BAR1 ESXi gives a passthrough card: memory.total stays 65536 MiB and the link stays Gen2. Proven Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-19 (lab notes)
  • BAR1 size matters only for P2P and static BAR1 mappings. P2P between CMPs inside an ESXi guest does not work yet, even with 64 GB BAR1s: static BAR1 P2P fails init or wedges UVM, while the same cards pass 12/12 directed pairs on bare metal. So today the large BAR1 under ESXi is groundwork for P2P experiments, not a speed-up. See Virtual machines and passthrough#P2P inside the VM: not working. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)

Tested on

  • VMware ESXi 8.0.3, four 8 GB cards (10de:20c2, Hynix HBM, VBIOS 92.00.67.00.01) passed through into one Ubuntu 24.04 guest (kernel 6.8.0-146-generic) next to other, newer NVIDIA GPUs.
  • A single-socket EPYC server with the slots bifurcated x8/x8, so every card sits behind its own root port; two cards on one PCIe root complex, two on another.
  • Guest driver: NVIDIA 580.178.04 open modules with the cmpunlocker v0.5 core ported onto them (in the repo's driver/). The writer and the guest hook do not depend on that driver.
  • The 10 GB card (10de:2082) is coded in the guest hook but untested.

Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)

The problem

On the host: ESXi keeps whatever firmware assigned

At every host power-on the card's Resizable BAR capability advertises only 64 MB: ReBAR CAP at config offset 0xbbc reads 0x400 (only the 64 MB size bit) and CTRL at 0xbc0 reads 0x601. Firmware sizes BAR1 to 64 MB, ESXi enumerates what firmware assigned, and ESXi never writes the card's vendor override. Broadcom's VMDirectPath notes say Resizable BAR is unsupported for passthrough and that devices are reset at VM power-on. Raising pciPassthru.64bitMMIOSizeGB only reserves guest address space; it cannot grow the physical BAR. The cmpunlocker bar1-resize-unlock.patch resizes from a Linux host; inside a VM the BAR is fixed before the driver exists. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)

In the guest: 256 MB even when the host has 64 GB

With the host BAR1 at 64 GB, the first VM boot showed resource1 = 64 GiB in sysfs but NVML reported BAR1 Total 256 MiB. Every VM power-on resets the card's internal override while the host allocation survives: guest config space and the config mirror at BAR0 + 0x88000 read CAP 0x400, CTRL 0x801 (size code 8 = 256 MB, which CAP does not even advertise), CYA 0x26, CFG 0x80000000. Proven Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-18 (lab notes)

The driver takes the BAR1 size from GSP, and GSP reads the card's ReBAR block, not Linux's reservation; so Linux showing 64 GiB proves nothing about what the card decodes. The same cards on native Linux report 64 GB BAR1 on all four, so there is no general 256 MB driver limit. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-18 (lab notes)

How it works

Two card registers open the large sizes: CYA 0x724 = 0x30, then CFG 0xdcc = 0x8000000a. After them ReBAR CAP reads 0x1ffc00 (64 MB to 64 GB offered) and BAR1 can be programmed to 64 GB (CTRL size code 16, 0x1001). On bare metal the patched Linux driver does this and asks the kernel to resize. Under ESXi it has to happen twice:

  1. On the host, before ESXi boots, with the BAR1 and BAR3 addresses and the parent bridge window programmed by hand, because ESXi will not resize later.
  2. In the guest, before the NVIDIA driver loads, because each VM power-on resets the two override registers. Guest config-space writes are virtualised by VMware and never reach the device, so the guest repair writes through the BAR0 config mirror page instead.

Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)

Part 1: the pre-boot UEFI writer (host)

A small UEFI application (gnu-efi, x86_64) that runs after firmware PCI enumeration and before the ESXi loader, then chain-loads the unchanged loader with its original load options, with no firmware reset. Source: writer/ in the repo, in a two-card and a four-card variant. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)

What it does, per card

Before the first store it reserves, for each card, a 64 GB + 32 MB span (0x1002000000 bytes) at 64 GB alignment in the firmware memory map (GCD), inside the root bridge aperture taken from the ACPI DSDT _CRS windows. Then, through EFI_PCI_IO_PROTOCOL, with every store read back before the next:

  1. COMMAND cleared (memory decode and bus mastering off) on the card and on its parent root port.
  2. CYA 0x724 = 0x30.
  3. CFG 0xdcc = 0x8000000a. CAP 0xbbc now reads 0x1ffc00.
  4. ReBAR CTRL 0xbc0 set to size code 16 (64 GB): 0x1001.
  5. BAR1 address low/high, then BAR3 low/high (BAR3 is 32 MB at BAR1 + 64 GB).
  6. The parent port's prefetchable window (base, limit and upper 32 bits) set to enclose the reservation.
  7. The parent port's COMMAND restored. The card itself is left with decode off for the OS to enable.

Proven Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-19 (lab notes)

Placement rule: BAR1 at the lowest 64 GB-aligned base at or above the root bridge's declared aperture start, BAR3 at the next 64 GB boundary, one reservation that is also the parent port's prefetchable window, and a 128 GB stride to the next card. On the test host this gave BAR1s at 0x64000000000 and 0x66000000000 on one root complex and 0x48000000000 and 0x4a000000000 on the other. The repo's writer/placement.py computes a proposal only; the writer re-checks every interval live. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)

Gates: it refuses before writing unless everything matches

The writer refuses before any store unless all of these hold: the SMBIOS system UUID matches the pinned host; the current boot entry and its variable match; the ESXi loader file on the boot partition matches the pinned size and SHA256; each card's cold BAR0/BAR1/BAR3 and its parent bridge windows equal the recorded cold values; the ReBAR capability offers the 64 GB size; the root bus descriptors and apertures match; and every reservation misses every recorded memory, bridge and firmware interval. All pins (host UUID, boot partition and boot entry, loader hash and size, the BAR plan and the apertures) are per host: the published build ships placeholders and must be regenerated for your host, and the source refuses to build an executing image unless WRITER_POLICY_BOUND=1 and WRITER_EXECUTE=1 are set. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)

  • Four-card variant (fail-open): a refusal before the first store prints HANDOFF_FALLBACK reason=... pci_mutation=0 resize_applied=0 and boots ESXi normally with 64 MB BARs. This branch has not yet run on real firmware.
  • Two-card variant (no fallback): any refusal parks the host.
  • After a possible store has begun any failure is terminal: HANDOFF_REFUSED ... cold_recovery_required=1, the host stays parked in the firmware. There is no rollback, retry or NVRAM write inside the app.

Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)

Build

make GNUEFI=/path/to/gnu-efi/usr four-card     # or two-card; output in build/<variant>/BOOTX64.EFI

Regenerate the host pins and the BAR plan first (see writer/four-card/README.md, section "Per-host values to regenerate"). Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)

Steps for one supervised run

The repo's writer/window/two_card_window.py drives these steps with a guard on each; every step can be dry-run first. The order:

  1. Preflight: record which VMs run, check the BMC, the IP-KVM and that the guest hook (Part 3) is installed and passes --check.
  2. Media: put BOOTX64.EFI at \EFI\BOOT\BOOTX64.EFI on a 64 MB FAT image, upload it to the IP-KVM mass-storage drive, read it back and compare the SHA256, and present it read-only (not as a CD-ROM).
  3. Quiesce: stop the GPU workloads in the guest, shut down every VM, put the host in maintenance mode.
  4. Boot override: attach the media and set the BMC (Redfish) boot override to Once, UEFI, USB; read it back.
  5. Record the console (the writer prints its progress on the firmware console).
  6. Reboot the host once.
  7. Wait for ESXi and check every BAR on the host (vsish BARInfo per card: Size:68719476736).
  8. Detach the media and clear the boot override, exit maintenance mode, start the guest and check it (Part 3), then start the services and the other VMs.

A good run on the console reads WRITER_RUN ... target_count for the number of cards, one WRITER_REBAR_TRANSITION line per card (cap 0x400 to 0x1ffc00, ctrl 0x601 to 0x1001), WRITER_STATE ... status=4, START_IMAGE ... pci_mutation=1, then the VMware loader banner. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-18 (lab notes)

Running it on every host boot

Firmware re-enumerates every card cold on each host power-on and reboot, so the writer has to run on every boot; it refuses an already enlarged layout, so there is no resume. What runs on the test host: leave the writer image attached read-only on the IP-KVM, and set the BMC boot override to Continuous / USB / UEFI through Redfish (check that Continuous is among the allowed values first; otherwise put the UEFI USB class first in the BIOS boot priorities). With no USB target the firmware falls through to the normal boot order. Each host boot takes about 3.5 to 4 minutes longer. Add a post-boot check that alerts when any BAR1 is not 64 GB, because a fallback otherwise looks like a normal boot. Copying the writer onto the ESXi boot partition with its own boot entry is written up in writer/four-card/docs/BOOT_PERSISTENCE.md but not rehearsed. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)

Part 2: VM settings

pciPassthru.use64bitMMIO = "TRUE"
pciPassthru.64bitMMIOSizeGB = "1024"

384 was not enough even for one 64 GB card (the guest EFI refused the MMIO; the arithmetic fit, the placement did not). Four 64 GB cards plus two other GPUs with 128 GB BAR1s need about 544 GB of 64-bit BAR space, so 512 cannot work either; 1024 is the next power of two. Size it for every passthrough GPU in the VM. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)

Part 3: the guest repair hook

cmp-bar1-repair (in the repo's guest/, Python 3 standard library only) restores the two override registers before the NVIDIA module loads. It is wired in as a modprobe install command: 

# /etc/modprobe.d/zz-cmp-bar1-repair.conf
install nvidia /usr/local/sbin/cmp-bar1-repair; /sbin/modprobe --ignore-install nvidia $CMDLINE_OPTS

For each 10de:20c2 / 10de:2082 function it reads 15 words of the config mirror page (BAR0 + 0x88000, one 4 KB page mapped from resource0) and repairs only when all of this holds: resource1 is exactly 64 GiB; no driver is bound and the nvidia module is not loaded; the card shows exactly the reset profile (CAP 0x400, CTRL 0x601 or 0x801, CYA 0x26, CFG 0x80000000); the device ID and ReBAR structure words match; the card's physical BAR1 is a 64-bit prefetchable BAR on a 64 GB-aligned base; memory decode on, bus mastering off, no error bits. The repair is two stores, each re-read: 0x724 = 0x30, then 0xdcc = 0x8000000a, ending at CAP 0x1ffc00, CTRL 0x1001, CYA 0x30, CFG 0x8000000a. Anything else is refused with no writes. It always exits 0 so the driver still loads, keeps one JSON journal per run and logs to journald under the tag cmp-bar1-repair. Proven Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-18 (lab notes)

Expected log on a VM boot: REPAIRED CAP=0x1ffc00 CTRL=0x1001 CYA=0x30 CFG=0x8000000a after 2 writes; all other words unchanged once per card, then NVML shows 65536 MiB BAR1. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-19 (lab notes)

Three rules learned the hard way:

  1. Enable decode first on a cold VM boot. A driverless function comes up with memory decode off, so every mirror read returns 0xffffffff (on one cold boot all four cards read all-ones and the driver came up at 256 MB). The hook writes 1 to the sysfs enable attribute (memory decode only, never bus master) and re-checks before mapping.
  2. Lock against udev. udev coldplug runs the install command once per NVIDIA function, concurrently (five times within two seconds). The hook holds an exclusive lock for its whole run; a 60 s timeout is a refusal, never a race.
  3. Never retry. A wrong read-back after store 1 stops before store 2; any failure means a VM power cycle, not a second attempt.

Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-18 (lab notes)

Keep nvidia.ko out of the initramfs, and never insmod or modprobe --ignore-install by hand: both bypass the hook. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)

Check and remove: 

cmp-bar1-repair --check                 # read-only report, fine with the driver bound
cmp-bar1-repair --check --strict        # exit 1 unless every CMP is OK or SKIP (for monitoring)
modprobe --show-depends nvidia          # first line must be the install command
journalctl -t cmp-bar1-repair           # every run
rm /etc/modprobe.d/zz-cmp-bar1-repair.conf   # removes the hook; the next nvidia load is a plain modprobe

The post-boot check catches a wrong card count, a non-64 GiB resource1 or a missing repair line, but not a refusal that leaves NVML at 256 MB, so also check NVML BAR1 after every VM boot. If a guest shows 256 MB on a card the host gave 64 GB: stop everything using the GPUs, modprobe -r nvidia_drm nvidia_modeset nvidia_uvm nvidia, run cmp-bar1-repair, modprobe nvidia, and check again. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-19 (lab notes)

Result on the passthrough host

  • 2026-10-18: two cards (one root complex) at 64 GB; 2026-10-19: all four (both root complexes).
  • On the host, vsish BARInfo reports Size:68719476736 for each card (67108864 when the writer did not run), and the VM log shows PCIPassthru: ... barIndex 1 ... size 68719476736.
  • In the guest, resource1 is 64 GiB and NVML reports 65536 MiB BAR1 on all four; 60 GiB write/verify tests with two address-dependent patterns pass on every card.
  • Since 2026-10-19 every VM boot logs four repairs and NVML shows 65536 MiB on all four.

Proven Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-19 (lab notes)

Risks

  • Warning A failure after the writer's first store parks the whole host. The ESXi host and every VM on it stay down until a cold power cycle; there is no rollback or retry in the app. Cold recovery: detach and disable the media, clear the boot override, power the host off, confirm it stays off for at least 10 seconds, power it on, and check that every card shows its small cold BARs before starting any VM. Never run the writer again on a card that may have been written. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)
  • Warning ESXi updates park the host. A patch that rewrites the ESXi loader or the boot entry makes the writer refuse at the next boot and leave the host in the firmware. Before patching ESXi, detach the media and set the boot override to Disabled; re-check the loader and boot-entry pins before re-arming. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)
  • Warning The writer reprograms bridge windows on a root complex that may also carry the ESXi boot drive, console and USB (it did on the test host). Run the first boots in a maintenance window with a remote console and a way to power-cycle the host. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-19 (lab notes)
  • Warning The fail-open branch has never run on real firmware. Treat any refusal as a parked host until proven otherwise. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-19 (lab notes)
  • A normal host reboot throws the allocation away. Never follow a successful 64 GB host boot with a plain reboot to check something; the cards come back at 64 MB and the whole window has to be run again. If only the guest fails to start afterwards, read the VM log and do not reboot the host. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)
  • The pins are per host. A build with another host's pins refuses by design. Moving a card, adding a device or a firmware update that changes the ACPI tables or the memory map makes the writer refuse (or fall back), so expect to regenerate the plan after hardware changes. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)
  • In the guest, a failed repair leaves the card at 256 MB and needs a VM power cycle; nothing persistent is written to the card. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-18 (lab notes)

Related

Virtual machines and passthrough, Resizable BAR and BAR1 64 GB, Server BIOS setup, P2P, Platforms, Card not detected or BAR allocation fails.

Repository: darkmaniac7/cmp170hx-esxi (writer, guest hook, probes, and the 580.178.04 driver patches). The unlock itself is amoghmunikote/cmpunlocker.

Imported from the cmpunlocker Discord with credit; see Credits.
Retrieved from "https://cmp.wiki/w/index.php?title=BAR1_64_GB_on_ESXi&oldid=343"