ImportBot: Import: 64 GiB BAR1 for CMP 170HX cards passed through by ESXi 8 (pre-boot UEFI writer, VM settings, guest repair hook, results, risks), from @darkmaniac7 lab notes and the darkmaniac7/cmp170hx-esxi repo

2026-10-30T08:42:59Z

Import: 64 GiB BAR1 for CMP 170HX cards passed through by ESXi 8 (pre-boot UEFI writer, VM settings, guest repair hook, results, risks), from @darkmaniac7 lab notes and the darkmaniac7/cmp170hx-esxi repo

New page

{{Status|Reported}} {{Cards|8 GB}}

How to give CMP 170HX cards that VMware ESXi passes through to a Linux guest a real 64 GB BAR1. ESXi does not do it by itself. It takes two parts: a small UEFI program that resizes BAR1 on the host before ESXi boots, and a guest-side hook that restores the card's override after every VM power-on. Everything here was measured on one host in October 2026; the code, the build files and the full write-up are in [https://github.com/darkmaniac7/cmp170hx-esxi darkmaniac7/cmp170hx-esxi]. {{Source|handle=darkmaniac7|date=2026-10-21|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

For passthrough in general (Proxmox, the in-guest unlock, Gen2, P2P) see [[Virtual machines and passthrough]]. For BAR1 on bare metal see [[Resizable BAR and BAR1 64 GB]] and [[Server BIOS setup]].

== Do you need it? ==
* '''The 64 GB VRAM unlock does not need a big BAR1.''' The VRAM unlock, the Gen2 retrain and serving all work with the 64 MB BAR1 ESXi gives a passthrough card: <code><nowiki>memory.total</nowiki></code> stays 65536 MiB and the link stays Gen2. {{Status|Proven}} {{Source|handle=darkmaniac7|date=2026-10-19|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}
* '''BAR1 size matters only for P2P and static BAR1 mappings.''' P2P between CMPs inside an ESXi guest does not work yet, even with 64 GB BAR1s: static BAR1 P2P fails init or wedges UVM, while the same cards pass 12/12 directed pairs on bare metal. So today the large BAR1 under ESXi is groundwork for P2P experiments, not a speed-up. See [[Virtual machines and passthrough#P2P inside the VM: not working]]. {{Source|handle=darkmaniac7|date=2026-10-21|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

== Tested on ==
* VMware ESXi 8.0.3, four 8 GB cards (<code><nowiki>10de:20c2</nowiki></code>, Hynix HBM, VBIOS 92.00.67.00.01) passed through into one Ubuntu 24.04 guest (kernel 6.8.0-146-generic) next to other, newer NVIDIA GPUs.
* A single-socket EPYC server with the slots bifurcated x8/x8, so every card sits behind its own root port; two cards on one PCIe root complex, two on another.
* Guest driver: NVIDIA 580.178.04 open modules with the cmpunlocker v0.5 core ported onto them (in the repo's <code><nowiki>driver/</nowiki></code>). The writer and the guest hook do not depend on that driver.
* The 10 GB card (<code><nowiki>10de:2082</nowiki></code>) is coded in the guest hook but untested.
{{Source|handle=darkmaniac7|date=2026-10-21|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

== The problem ==
=== On the host: ESXi keeps whatever firmware assigned ===
At every host power-on the card's Resizable BAR capability advertises only 64 MB: ReBAR CAP at config offset <code><nowiki>0xbbc</nowiki></code> reads <code><nowiki>0x400</nowiki></code> (only the 64 MB size bit) and CTRL at <code><nowiki>0xbc0</nowiki></code> reads <code><nowiki>0x601</nowiki></code>. Firmware sizes BAR1 to 64 MB, ESXi enumerates what firmware assigned, and ESXi never writes the card's vendor override. Broadcom's VMDirectPath notes say Resizable BAR is unsupported for passthrough and that devices are reset at VM power-on. Raising <code><nowiki>pciPassthru.64bitMMIOSizeGB</nowiki></code> only reserves guest address space; it cannot grow the physical BAR. The cmpunlocker <code><nowiki>bar1-resize-unlock.patch</nowiki></code> resizes from a Linux host; inside a VM the BAR is fixed before the driver exists. {{Source|handle=darkmaniac7|date=2026-10-21|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

=== In the guest: 256 MB even when the host has 64 GB ===
With the host BAR1 at 64 GB, the first VM boot showed <code><nowiki>resource1</nowiki></code> = 64 GiB in sysfs but NVML reported BAR1 Total 256 MiB. Every VM power-on resets the card's internal override while the host allocation survives: guest config space and the config mirror at BAR0 + <code><nowiki>0x88000</nowiki></code> read CAP <code><nowiki>0x400</nowiki></code>, CTRL <code><nowiki>0x801</nowiki></code> (size code 8 = 256 MB, which CAP does not even advertise), CYA <code><nowiki>0x26</nowiki></code>, CFG <code><nowiki>0x80000000</nowiki></code>. {{Status|Proven}} {{Source|handle=darkmaniac7|date=2026-10-18|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

The driver takes the BAR1 size from GSP, and GSP reads the card's ReBAR block, not Linux's reservation; so Linux showing 64 GiB proves nothing about what the card decodes. The same cards on native Linux report 64 GB BAR1 on all four, so there is no general 256 MB driver limit. {{Source|handle=darkmaniac7|date=2026-10-18|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

== How it works ==
Two card registers open the large sizes: CYA <code><nowiki>0x724 = 0x30</nowiki></code>, then CFG <code><nowiki>0xdcc = 0x8000000a</nowiki></code>. After them ReBAR CAP reads <code><nowiki>0x1ffc00</nowiki></code> (64 MB to 64 GB offered) and BAR1 can be programmed to 64 GB (CTRL size code 16, <code><nowiki>0x1001</nowiki></code>). On bare metal the patched Linux driver does this and asks the kernel to resize. Under ESXi it has to happen twice:
# '''On the host, before ESXi boots''', with the BAR1 and BAR3 addresses and the parent bridge window programmed by hand, because ESXi will not resize later.
# '''In the guest, before the NVIDIA driver loads''', because each VM power-on resets the two override registers. Guest config-space writes are virtualised by VMware and never reach the device, so the guest repair writes through the BAR0 config mirror page instead.
{{Source|handle=darkmaniac7|date=2026-10-21|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

== Part 1: the pre-boot UEFI writer (host) ==
A small UEFI application (gnu-efi, x86_64) that runs after firmware PCI enumeration and before the ESXi loader, then chain-loads the unchanged loader with its original load options, with no firmware reset. Source: <code><nowiki>writer/</nowiki></code> in the repo, in a two-card and a four-card variant. {{Source|handle=darkmaniac7|date=2026-10-21|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

=== What it does, per card ===
Before the first store it reserves, for each card, a 64 GB + 32 MB span (<code><nowiki>0x1002000000</nowiki></code> bytes) at 64 GB alignment in the firmware memory map (GCD), inside the root bridge aperture taken from the ACPI DSDT <code><nowiki>_CRS</nowiki></code> windows. Then, through <code><nowiki>EFI_PCI_IO_PROTOCOL</nowiki></code>, with every store read back before the next:
# COMMAND cleared (memory decode and bus mastering off) on the card and on its parent root port.
# CYA <code><nowiki>0x724 = 0x30</nowiki></code>.
# CFG <code><nowiki>0xdcc = 0x8000000a</nowiki></code>. CAP <code><nowiki>0xbbc</nowiki></code> now reads <code><nowiki>0x1ffc00</nowiki></code>.
# ReBAR CTRL <code><nowiki>0xbc0</nowiki></code> set to size code 16 (64 GB): <code><nowiki>0x1001</nowiki></code>.
# BAR1 address low/high, then BAR3 low/high (BAR3 is 32 MB at BAR1 + 64 GB).
# The parent port's prefetchable window (base, limit and upper 32 bits) set to enclose the reservation.
# The parent port's COMMAND restored. The card itself is left with decode off for the OS to enable.
{{Status|Proven}} {{Source|handle=darkmaniac7|date=2026-10-19|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

'''Placement rule:''' BAR1 at the lowest 64 GB-aligned base at or above the root bridge's declared aperture start, BAR3 at the next 64 GB boundary, one reservation that is also the parent port's prefetchable window, and a 128 GB stride to the next card. On the test host this gave BAR1s at <code><nowiki>0x64000000000</nowiki></code> and <code><nowiki>0x66000000000</nowiki></code> on one root complex and <code><nowiki>0x48000000000</nowiki></code> and <code><nowiki>0x4a000000000</nowiki></code> on the other. The repo's <code><nowiki>writer/placement.py</nowiki></code> computes a proposal only; the writer re-checks every interval live. {{Source|handle=darkmaniac7|date=2026-10-21|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

=== Gates: it refuses before writing unless everything matches ===
The writer refuses before any store unless all of these hold: the SMBIOS system UUID matches the pinned host; the current boot entry and its variable match; the ESXi loader file on the boot partition matches the pinned size and SHA256; each card's cold BAR0/BAR1/BAR3 and its parent bridge windows equal the recorded cold values; the ReBAR capability offers the 64 GB size; the root bus descriptors and apertures match; and every reservation misses every recorded memory, bridge and firmware interval. All pins (host UUID, boot partition and boot entry, loader hash and size, the BAR plan and the apertures) are per host: the published build ships placeholders and must be regenerated for your host, and the source refuses to build an executing image unless <code><nowiki>WRITER_POLICY_BOUND=1</nowiki></code> and <code><nowiki>WRITER_EXECUTE=1</nowiki></code> are set. {{Source|handle=darkmaniac7|date=2026-10-21|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

* '''Four-card variant (fail-open):''' a refusal before the first store prints <code><nowiki>HANDOFF_FALLBACK reason=... pci_mutation=0 resize_applied=0</nowiki></code> and boots ESXi normally with 64 MB BARs. This branch has not yet run on real firmware.
* '''Two-card variant (no fallback):''' any refusal parks the host.
* '''After a possible store has begun''' any failure is terminal: <code><nowiki>HANDOFF_REFUSED ... cold_recovery_required=1</nowiki></code>, the host stays parked in the firmware. There is no rollback, retry or NVRAM write inside the app.
{{Source|handle=darkmaniac7|date=2026-10-21|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

=== Build ===
<syntaxhighlight lang="bash">
make GNUEFI=/path/to/gnu-efi/usr four-card # or two-card; output in build/<variant>/BOOTX64.EFI
</syntaxhighlight>
Regenerate the host pins and the BAR plan first (see <code><nowiki>writer/four-card/README.md</nowiki></code>, section "Per-host values to regenerate"). {{Source|handle=darkmaniac7|date=2026-10-21|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

=== Steps for one supervised run ===
The repo's <code><nowiki>writer/window/two_card_window.py</nowiki></code> drives these steps with a guard on each; every step can be dry-run first. The order:
# '''Preflight:''' record which VMs run, check the BMC, the IP-KVM and that the guest hook (Part 3) is installed and passes <code><nowiki>--check</nowiki></code>.
# '''Media:''' put <code><nowiki>BOOTX64.EFI</nowiki></code> at <code><nowiki>\EFI\BOOT\BOOTX64.EFI</nowiki></code> on a 64 MB FAT image, upload it to the IP-KVM mass-storage drive, read it back and compare the SHA256, and present it read-only (not as a CD-ROM).
# '''Quiesce:''' stop the GPU workloads in the guest, shut down every VM, put the host in maintenance mode.
# '''Boot override:''' attach the media and set the BMC (Redfish) boot override to Once, UEFI, USB; read it back.
# '''Record the console''' (the writer prints its progress on the firmware console).
# '''Reboot the host once.'''
# '''Wait for ESXi''' and check every BAR on the host (<code><nowiki>vsish</nowiki></code> BARInfo per card: <code><nowiki>Size:68719476736</nowiki></code>).
# '''Detach''' the media and clear the boot override, exit maintenance mode, start the guest and check it (Part 3), then start the services and the other VMs.
A good run on the console reads <code><nowiki>WRITER_RUN ... target_count</nowiki></code> for the number of cards, one <code><nowiki>WRITER_REBAR_TRANSITION</nowiki></code> line per card (cap <code><nowiki>0x400</nowiki></code> to <code><nowiki>0x1ffc00</nowiki></code>, ctrl <code><nowiki>0x601</nowiki></code> to <code><nowiki>0x1001</nowiki></code>), <code><nowiki>WRITER_STATE ... status=4</nowiki></code>, <code><nowiki>START_IMAGE ... pci_mutation=1</nowiki></code>, then the VMware loader banner. {{Source|handle=darkmaniac7|date=2026-10-18|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

=== Running it on every host boot ===
Firmware re-enumerates every card cold on each host power-on and reboot, so the writer has to run on every boot; it refuses an already enlarged layout, so there is no resume. What runs on the test host: leave the writer image attached read-only on the IP-KVM, and set the BMC boot override to <code><nowiki>Continuous</nowiki></code> / USB / UEFI through Redfish (check that <code><nowiki>Continuous</nowiki></code> is among the allowed values first; otherwise put the UEFI USB class first in the BIOS boot priorities). With no USB target the firmware falls through to the normal boot order. Each host boot takes about 3.5 to 4 minutes longer. Add a post-boot check that alerts when any BAR1 is not 64 GB, because a fallback otherwise looks like a normal boot. Copying the writer onto the ESXi boot partition with its own boot entry is written up in <code><nowiki>writer/four-card/docs/BOOT_PERSISTENCE.md</nowiki></code> but not rehearsed. {{Source|handle=darkmaniac7|date=2026-10-21|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

== Part 2: VM settings ==
<pre>
pciPassthru.use64bitMMIO = "TRUE"
pciPassthru.64bitMMIOSizeGB = "1024"
</pre>
384 was not enough even for one 64 GB card (the guest EFI refused the MMIO; the arithmetic fit, the placement did not). Four 64 GB cards plus two other GPUs with 128 GB BAR1s need about 544 GB of 64-bit BAR space, so 512 cannot work either; 1024 is the next power of two. Size it for every passthrough GPU in the VM. {{Source|handle=darkmaniac7|date=2026-10-21|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

== Part 3: the guest repair hook ==
<code><nowiki>cmp-bar1-repair</nowiki></code> (in the repo's <code><nowiki>guest/</nowiki></code>, Python 3 standard library only) restores the two override registers before the NVIDIA module loads. It is wired in as a modprobe install command:
<syntaxhighlight lang="bash">
# /etc/modprobe.d/zz-cmp-bar1-repair.conf
install nvidia /usr/local/sbin/cmp-bar1-repair; /sbin/modprobe --ignore-install nvidia $CMDLINE_OPTS
</syntaxhighlight>
For each <code><nowiki>10de:20c2</nowiki></code> / <code><nowiki>10de:2082</nowiki></code> function it reads 15 words of the config mirror page (BAR0 + <code><nowiki>0x88000</nowiki></code>, one 4 KB page mapped from <code><nowiki>resource0</nowiki></code>) and repairs only when all of this holds: <code><nowiki>resource1</nowiki></code> is exactly 64 GiB; no driver is bound and the nvidia module is not loaded; the card shows exactly the reset profile (CAP <code><nowiki>0x400</nowiki></code>, CTRL <code><nowiki>0x601</nowiki></code> or <code><nowiki>0x801</nowiki></code>, CYA <code><nowiki>0x26</nowiki></code>, CFG <code><nowiki>0x80000000</nowiki></code>); the device ID and ReBAR structure words match; the card's physical BAR1 is a 64-bit prefetchable BAR on a 64 GB-aligned base; memory decode on, bus mastering off, no error bits. The repair is two stores, each re-read: <code><nowiki>0x724 = 0x30</nowiki></code>, then <code><nowiki>0xdcc = 0x8000000a</nowiki></code>, ending at CAP <code><nowiki>0x1ffc00</nowiki></code>, CTRL <code><nowiki>0x1001</nowiki></code>, CYA <code><nowiki>0x30</nowiki></code>, CFG <code><nowiki>0x8000000a</nowiki></code>. Anything else is refused with no writes. It always exits 0 so the driver still loads, keeps one JSON journal per run and logs to journald under the tag <code><nowiki>cmp-bar1-repair</nowiki></code>. {{Status|Proven}} {{Source|handle=darkmaniac7|date=2026-10-18|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

Expected log on a VM boot: <code><nowiki>REPAIRED CAP=0x1ffc00 CTRL=0x1001 CYA=0x30 CFG=0x8000000a after 2 writes; all other words unchanged</nowiki></code> once per card, then NVML shows 65536 MiB BAR1. {{Source|handle=darkmaniac7|date=2026-10-19|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

Three rules learned the hard way:
# '''Enable decode first on a cold VM boot.''' A driverless function comes up with memory decode off, so every mirror read returns <code><nowiki>0xffffffff</nowiki></code> (on one cold boot all four cards read all-ones and the driver came up at 256 MB). The hook writes 1 to the sysfs <code><nowiki>enable</nowiki></code> attribute (memory decode only, never bus master) and re-checks before mapping.
# '''Lock against udev.''' udev coldplug runs the install command once per NVIDIA function, concurrently (five times within two seconds). The hook holds an exclusive lock for its whole run; a 60 s timeout is a refusal, never a race.
# '''Never retry.''' A wrong read-back after store 1 stops before store 2; any failure means a VM power cycle, not a second attempt.
{{Source|handle=darkmaniac7|date=2026-10-18|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

Keep <code><nowiki>nvidia.ko</nowiki></code> out of the initramfs, and never <code><nowiki>insmod</nowiki></code> or <code><nowiki>modprobe --ignore-install</nowiki></code> by hand: both bypass the hook. {{Source|handle=darkmaniac7|date=2026-10-21|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

Check and remove:
<syntaxhighlight lang="bash">
cmp-bar1-repair --check # read-only report, fine with the driver bound
cmp-bar1-repair --check --strict # exit 1 unless every CMP is OK or SKIP (for monitoring)
modprobe --show-depends nvidia # first line must be the install command
journalctl -t cmp-bar1-repair # every run
rm /etc/modprobe.d/zz-cmp-bar1-repair.conf # removes the hook; the next nvidia load is a plain modprobe
</syntaxhighlight>
The post-boot check catches a wrong card count, a non-64 GiB <code><nowiki>resource1</nowiki></code> or a missing repair line, but not a refusal that leaves NVML at 256 MB, so also check NVML BAR1 after every VM boot. If a guest shows 256 MB on a card the host gave 64 GB: stop everything using the GPUs, <code><nowiki>modprobe -r nvidia_drm nvidia_modeset nvidia_uvm nvidia</nowiki></code>, run <code><nowiki>cmp-bar1-repair</nowiki></code>, <code><nowiki>modprobe nvidia</nowiki></code>, and check again. {{Source|handle=darkmaniac7|date=2026-10-19|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

== Result on the passthrough host ==
* 2026-10-18: two cards (one root complex) at 64 GB; 2026-10-19: all four (both root complexes).
* On the host, <code><nowiki>vsish</nowiki></code> BARInfo reports <code><nowiki>Size:68719476736</nowiki></code> for each card (<code><nowiki>67108864</nowiki></code> when the writer did not run), and the VM log shows <code><nowiki>PCIPassthru: ... barIndex 1 ... size 68719476736</nowiki></code>.
* In the guest, <code><nowiki>resource1</nowiki></code> is 64 GiB and NVML reports 65536 MiB BAR1 on all four; 60 GiB write/verify tests with two address-dependent patterns pass on every card.
* Since 2026-10-19 every VM boot logs four repairs and NVML shows 65536 MiB on all four.
{{Status|Proven}} {{Source|handle=darkmaniac7|date=2026-10-19|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

== Risks ==
* {{Status|Warning}} '''A failure after the writer's first store parks the whole host.''' The ESXi host and every VM on it stay down until a cold power cycle; there is no rollback or retry in the app. Cold recovery: detach and disable the media, clear the boot override, power the host off, confirm it stays off for at least 10 seconds, power it on, and check that every card shows its small cold BARs before starting any VM. Never run the writer again on a card that may have been written. {{Source|handle=darkmaniac7|date=2026-10-21|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}
* {{Status|Warning}} '''ESXi updates park the host.''' A patch that rewrites the ESXi loader or the boot entry makes the writer refuse at the next boot and leave the host in the firmware. Before patching ESXi, detach the media and set the boot override to Disabled; re-check the loader and boot-entry pins before re-arming. {{Source|handle=darkmaniac7|date=2026-10-21|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}
* {{Status|Warning}} '''The writer reprograms bridge windows on a root complex that may also carry the ESXi boot drive, console and USB''' (it did on the test host). Run the first boots in a maintenance window with a remote console and a way to power-cycle the host. {{Source|handle=darkmaniac7|date=2026-10-19|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}
* {{Status|Warning}} '''The fail-open branch has never run on real firmware.''' Treat any refusal as a parked host until proven otherwise. {{Source|handle=darkmaniac7|date=2026-10-19|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}
* '''A normal host reboot throws the allocation away.''' Never follow a successful 64 GB host boot with a plain reboot to check something; the cards come back at 64 MB and the whole window has to be run again. If only the guest fails to start afterwards, read the VM log and do not reboot the host. {{Source|handle=darkmaniac7|date=2026-10-21|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}
* '''The pins are per host.''' A build with another host's pins refuses by design. Moving a card, adding a device or a firmware update that changes the ACPI tables or the memory map makes the writer refuse (or fall back), so expect to regenerate the plan after hardware changes. {{Source|handle=darkmaniac7|date=2026-10-21|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}
* '''In the guest,''' a failed repair leaves the card at 256 MB and needs a VM power cycle; nothing persistent is written to the card. {{Source|handle=darkmaniac7|date=2026-10-18|repo=https://github.com/darkmaniac7/cmp170hx-esxi|note=lab notes}}

== Related ==
[[Virtual machines and passthrough]], [[Resizable BAR and BAR1 64 GB]], [[Server BIOS setup]], [[P2P]], [[Platforms]], [[Card not detected or BAR allocation fails]].

Repository: [https://github.com/darkmaniac7/cmp170hx-esxi darkmaniac7/cmp170hx-esxi] (writer, guest hook, probes, and the 580.178.04 driver patches). The unlock itself is [https://github.com/amoghmunikote/cmpunlocker amoghmunikote/cmpunlocker].

{{ImportFooter}}
[[Category:Platforms]]
[[Category:8 GB]]

BAR1 64 GB on ESXi - Revision history

ImportBot: Import: 64 GiB BAR1 for CMP 170HX cards passed through by ESXi 8 (pre-boot UEFI writer, VM settings, guest repair hook, results, risks), from @darkmaniac7 lab notes and the darkmaniac7/cmp170hx-esxi repo