Server BIOS setup

Reported 8 GB and 10 GB

How to get a CMP 170HX to PCIe Gen2 and a 64 GB BAR1 on a server, and what to check when it does not happen. It comes from getting 8 cards to Gen2 x16 with 64 GB BAR1 on bare metal, and 4 cards to Gen2 with 64 GB BAR1 under ESXi passthrough. The page works as a checklist for a person, and it works just as well handed to an AI agent with a root shell, with a person making the BIOS changes and power cycles. Source: @darkmaniac7, 2026-10-29 (lab notes)

For what the two features are, see Gen2 link retrain and Resizable BAR and BAR1 64 GB. For desktop boards, see Consumer motherboards. For hypervisors, see Virtual machines and passthrough.

Rules

1. Collect facts first (the script below). Change nothing until you know which case you are in.
2. One change per reboot. Save the collect output before and after every change, so each change can be judged on its own.
3. BIOS changes, reboots and cold power cycles are done by a person. If an agent is helping, it proposes the exact setting and the reason; it does not do them.
4. Do not try Gen3 on 8 GB cards. On the 8 GB cards (10de:20c2) Gen3 and Gen4 are fused off, and so is the software fuse override (0x820084 bit 0 = 1). One attempt through the XP/XVE registers left a card stuck at Gen1 until a power cycle. Gen2 is the ceiling. Gen3 on 10 GB cards is a separate route, see Unlocks.
5. Never flash a VBIOS, and never write PCI config space by hand on a live system unless you know exactly why.
6. If you have a working machine with the same cards, run the collect script there too and diff the two. That is the fastest way to find the difference.

Source: @darkmaniac7, 2026-10-29 (lab notes)

What working looks like

Reference box: Supermicro H12DSG-O-CPU (dual EPYC Rome), 8 cards, Ubuntu 24.04, kernel 6.8.0, nvidia-open 615.71.09 patched by the Morrowmake cmpunlocker fork (github.com/Morrowmake/cmpunlocker, branch p2p-ecc, commit 87cb5fd), installed with:

sudo ./install.sh --p2p --profile=8gb --no-iommu --no-passthrough

$ nvidia-smi --query-gpu=index,pci.bus_id,pcie.link.gen.current,pcie.link.width.current,memory.total --format=csv,noheader
0, 00000000:01:00.0, 2, 16, 65536 MiB          (all 8 cards the same)

$ nvidia-smi -q -i 0 | grep -A3 "BAR1 Memory Usage"
    Total : 65536 MiB

$ lspci -s 01:00.0 -vvv
    Region 1: Memory at 29000000000 (64-bit, prefetchable) [size=64G]
    Region 3: Memory at 28800000000 (64-bit, prefetchable) [size=32M]
    LnkCap: Port #0, Speed 5GT/s, Width x16
    LnkSta: Speed 5GT/s, Width x16
    Capabilities: [bb0 v1] Physical Resizable BAR
        BAR 1: current size: 64GB, supported: 64MB 128MB ... 32GB 64GB

$ grep -E "RegistryDwords|EnableResizableBar" /proc/driver/nvidia/params
EnableResizableBar: 1
RegistryDwords: "RmForceEnableGen2=1;RMPcieLinkSpeed=0x1;ForceP2P=0x11"

$ cat /etc/modprobe.d/cmp-pcie-gen2.conf        (written by the installer)
options nvidia NVreg_RegistryDwords="RmForceEnableGen2=1;RMPcieLinkSpeed=0x1;ForceP2P=0x11"

$ dmesg | grep -E "BAR 1|CMP BAR1"
pci 0000:01:00.0: BAR 1 [mem 0x380a8000000-0x380abffffff 64bit pref]      <- firmware sizes it 64 MB
NVRM: CMP BAR1: before CYA=0x00000026 CFG=0x80000000 CAP=0x00000400
NVRM: CMP BAR1: after  CYA=0x00000030 CFG=0x8000000a CAP=0x001ffc00
NVRM: CMP BAR1: resize enabled, will attempt up to 64 GB
NVRM: CMP BAR1: final size = 65536 MB                                     <- the driver resized it

Kernel command line: amd_iommu=on iommu=pt, nothing PCI-specific (no pci=realloc). Firmware put the 64 MB BAR1 above 4 GB, so "Above 4G Decoding" was on. The rest of the BIOS settings were not recorded.

nvidia-smi -q showing Device Max: 1 under PCIe Generation is normal on these cards, even when they run Gen2. Read Current. Source: @darkmaniac7, 2026-10-29 (lab notes)

How the two features work

Most dead ends come from treating these like normal ReBAR and normal link speed settings.

Gen2. Stock, the card trains at Gen1: LnkCap advertises 2.5 GT/s only. The BIOS and the root port cannot raise that, because the card's own LnkCap is the limit. The patched driver raises it from the GPU side during its unlock window: it programs the card's link registers for Gen2, requests a directed speed change and waits for the link to retrain. The RM keys in RegistryDwords switch that on. With the stock NVIDIA driver, or if those keys never reach the driver, the card stays at Gen1. Link width is whatever the slot or riser gives (an x8 slot gives Gen2 x8).

BAR1. At every power-on the card's Resizable BAR capability advertises only 64 MB (CAP 0x400). So the BIOS sizes BAR1 at 64 MB, and a BIOS "Resizable BAR" option cannot do better, because the card is not offering more yet. The patched driver then writes two card registers (CYA 0x724 = 0x30, CFG 0xdcc = 0x8000000a). After that the capability offers 64 MB to 64 GB (CAP 0x1ffc00), and the driver asks the kernel to resize BAR1 to 64 GB. BAR3 (32 MB) moves too.

For that resize to work, the kernel has to fit 64 GB + 32 MB per card, 64 GB-aligned, inside the 64-bit prefetchable window of the card's parent bridge, and grow that window if it is too small. That is where servers fail: tight MMIO layouts, bridge windows the firmware sized exactly, or two cards sharing one bridge or switch.

What you lose without a 64 GB BAR1. The 64 GB of VRAM and the compute unlock still work with a 64 MB BAR1. What it costs is P2P between cards. Engines like the Morrowmake recipes check P2P at startup and fall back to copies through host memory, so it is a speed loss for tensor parallel, not a failure. Source: @darkmaniac7, 2026-10-29 (lab notes)

Collect facts

Run this as root and save the output (collect.sh > facts-$(hostname)-$(date +%H%M).txt 2>&1):

#!/bin/bash
echo "== system"; dmidecode -s system-manufacturer; dmidecode -s system-product-name; dmidecode -s bios-version
uname -r; cat /proc/cmdline
echo "== driver"; head -1 /proc/driver/nvidia/version 2>&1
grep -E "RegistryDwords|EnableResizableBar" /proc/driver/nvidia/params 2>&1
grep -rH "nvidia" /etc/modprobe.d/ /lib/modprobe.d/ 2>/dev/null | grep -v ":#"
lsinitramfs /boot/initrd.img-$(uname -r) 2>/dev/null | grep -E "nvidia|modprobe.d" | head
echo "== cards"
nvidia-smi --query-gpu=index,pci.bus_id,pcie.link.gen.current,pcie.link.gen.max,pcie.link.width.current,memory.total --format=csv,noheader
nvidia-smi -q | grep -A2 "BAR1 Memory Usage"
echo "== tree"; lspci -tv
for d in $(lspci -D -d 10de: | awk '{print $1}'); do
  echo "=== GPU $d"
  lspci -s $d -vvv | grep -E "Region|LnkCap:|LnkSta:|LnkCtl2|Resizable|BAR [0-9]:"
  p=$(basename $(dirname $(readlink -f /sys/bus/pci/devices/$d)))
  while [[ $p =~ ^[0-9a-f]{4}: ]]; do
    echo "--- upstream $p"
    lspci -s $p -vvv | grep -E "^[0-9a-f]|Prefetchable memory behind|LnkCap:|LnkSta:|LnkCtl2|SltCap|HotPlug"
    p=$(basename $(dirname $(readlink -f /sys/bus/pci/devices/$p)))
  done
done
echo "== dmesg: driver"; dmesg | grep -iE "NVRM: CMP|retrain|Xid" | head -40
echo "== dmesg: BARs"; dmesg | grep -iE "BAR 1|BAR 3|no space|failed to assign|can't assign|resiz" | head -40
echo "== dmesg: root windows"; dmesg | grep -E "root bus resource \[mem" | head -20

The upstream walk shows every bridge between the card and the root complex. More than one bridge in that list means a PCIe switch (common on GPU risers and some Dell GPU configurations). The switch's windows have to grow too. Source: @darkmaniac7, 2026-10-29 (lab notes)

Dell PowerEdge: dump the BIOS PCI settings

If the box has an iDRAC, dump the BIOS PCI settings as they are now. Attribute names differ by PowerEdge generation, so list them rather than guessing:

racadm get BIOS.IntegratedDevices
racadm get BIOS.SysProfileSettings
racadm get BIOS | grep -iE "mmio|4g|bar|sriov|slot|speed|link|bifurc|aspm"

On 14th-generation PowerEdge (R640, R740 and so on), the settings that matter are under Integrated Devices: "Memory Mapped I/O above 4GB" and "Memory Mapped I/O Base". Check the names on your BIOS before changing anything. Source: @darkmaniac7, 2026-10-29 (lab notes)

Find your case and fix it

Case A: the unlock is not active (memory shows 8 GB, no NVRM: CMP lines)

Neither Gen2 nor BAR1 can work yet, so fix this first.

• Install the same unlocker commit and driver version as a working box, with --profile=8gb (8 GB cards) or --profile=10gb (10 GB cards). See Unlocks.
• A kernel or nvidia package upgrade puts the stock modules back through DKMS. The cards drop to 8 GB and Gen1 without any error. Hold the kernel and every nvidia package once it works. See Operating systems and kernels.
• The unlocker README says: if memory stays at its stock size, power off completely before booting. That means a real cold start (AC off, about 10 s), not a reboot.

Source: @darkmaniac7, 2026-10-29 (lab notes)

Case B: unlocked (64 GB VRAM) but still Gen1

1. RegistryDwords must show the Gen2 keys in /proc/driver/nvidia/params. If it does not, the options file was not read. When nvidia is loaded from the initramfs, the options must be in it too: run update-initramfs -u and reboot. Only one NVreg_RegistryDwords line counts; a second file silently replaces the first. Grep every modprobe file.
2. dmesg | grep -iE "retrain|gen2|link speed" shows whether the driver tried and what it got.
3. On every upstream bridge, LnkCap must allow at least 5 GT/s, and LnkCtl2 "Target Link Speed" must not be 2.5 GT/s. If a root port targets 2.5 GT/s, the BIOS has that slot limited. Look in the BIOS for a per-slot link speed setting and set it to Auto, or Gen2 or higher.
4. If LnkSta shows 2.5 GT/s with "(downgraded)" after a retrain, the link failed at Gen2 and fell back. That points at the signal path, not settings: a riser, an extension cable or a slot. Move the card to another slot to test. See Risers and bifurcation.
5. Do not change anything for Gen3 (see the rules at the top).

Source: @darkmaniac7, 2026-10-29 (lab notes)

Case C: unlocked and Gen2, but BAR1 stays 64 MB (or 256 MB)

Read the NVRM: CMP BAR1 lines first.

• No "resize enabled" line: check EnableResizableBar in /proc/driver/nvidia/params. It is 1 by default on the patched build; if it is 0, set options nvidia NVreg_EnableResizableBar=1.
• "final size" below 65536 MB, or kernel lines like "no space for" / "failed to assign" near BAR 1: the bridge window cannot take 64 GB. Try these one at a time, running the collect script after each:

1. BIOS: "Memory Mapped I/O above 4GB" (Above 4G Decoding) Enabled. Without it nothing else matters.
2. BIOS: give the 64-bit MMIO space as much room as the BIOS allows. On PowerEdge that is "Memory Mapped I/O Base". The options vary by generation, and a higher base address usually leaves more room above it. Linux handles any of them. Record the old value so it can go back.
3. Kernel: add pci=realloc to the command line. This lets Linux release and reassign bridge windows that firmware sized too tightly. Edit /etc/default/grub, run update-grub, then reboot. If something other than the GPUs breaks (NICs, storage), remove it again.
4. Isolate: test with one card. Two cards behind the same bridge or switch need room for two 64 GB + 32 MB blocks, each 64 GB-aligned, in one window. If one card resizes and two do not, it is window size or placement. Try the cards on different root ports if the chassis allows.
5. A BIOS "Resizable BAR" option, if this model has one, will not help by itself: at power-on the card only offers 64 MB. Enabling it is harmless. Change one thing at a time.

Source: @darkmaniac7, 2026-10-29 (lab notes)

If none of that works: the remaining route is resizing the card before the OS boots. That was done for ESXi with a small UEFI app that reserves a 64 GB-aligned block in the firmware memory map, then does the same two register writes plus the BAR and bridge-window programming. It works, but it has to run on every boot. The procedure, the risks and the code are on BAR1 64 GB on ESXi and in darkmaniac7/cmp170hx-esxi; read them rather than improvising. Source: repo github.com/darkmaniac7/cmp170hx-esxi by darkmaniac7 on GitHub, posted by @darkmaniac7, 2026-10-21 (lab notes)

What others hit in case C

• Window too small for the card count. A Minisforum MS-01 host with 4 cards behind a PLX 88096 switch had an MMIO space of 256 GB, too small for four 64 GB BAR1s, and the firmware tried to place the BAR1s inside a 512 MB section. Resizable BAR was already enabled. Source: @replacedriver47, 2026-10-22
  Source: @replacedriver47, 2026-10-22
  Source: @replacedriver47, 2026-10-22 Two days later the same owner got 2 of the 4 cards to 64 GB BAR1 by changing MMIOH and BAR sizes; the MMIOH size still prevented all 4. Source: @replacedriver47, 2026-10-24
  Source: @replacedriver47, 2026-10-24 See Minisforum MS-01.
• PCIe switch firmware. The same owner found the PLX 88096 firmware on his switch board did not have hotplug enabled, which he says is required to remap BAR addressing behind the switch. Source: @replacedriver47, 2026-10-24
• Hidden MMIOH settings on older boards. On an ASUS P9X79 WS the BIOS allocates only a small 64-bit window above 4 GB, so a 64 GB BAR request fails. Above 4G Decoding was present; the owner's assistant changed Ubuntu (kernel) parameters and it then worked. The exact parameters were not posted. Source: @globalapathy, 2026-10-19
  Source: @globalapathy, 2026-10-19
  Source: @fdelfini, 2026-10-19 See ASUS P9X79 WS.
• Two cards behind one switch crash the kernel at boot unless the per-card BAR1 resize is serialized: the PixelML club host (two PLX PEX 8747 switches, two cards per switch) corrupted the kernel resource tree on every boot until a rebar-serialize.patch (one mutex around the resize, the same fix as upstream cmpunlocker PR #59) was applied. Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @djpandaman, 2026-10-24 (docs/CMP170HX-UNDERSTANDING.md)
• P2P at a 64 MB BAR1, instead of fighting MMIO. Contested
  • OB88594: do not try to fix MMIO sizing unless the motherboard has independent options for it; the Morrowmake fork can do P2P at a 64 MB BAR1. Source: @ob88594, 2026-10-11
  • replacedriver47: the Morrowmake build reports P2P working with no BAR1, but no transfers go through. Source: @replacedriver47, 2026-10-11

Case D: the cards are passed through to a VM (Proxmox, ESXi)

The unlock has to run where the NVIDIA driver runs. Under ESXi, firmware sizes BAR1 at 64 MB before ESXi sees it, so the 64 GB BAR1 needs the pre-boot step above (BAR1 64 GB on ESXi). The VM also needs pciPassthru.use64bitMMIO = TRUE and a large pciPassthru.64bitMMIOSizeGB (1024 was needed for four cards). Gen2 works from inside the guest. P2P between CMPs inside a VM did not work; the same cards do 12/12 pairs on bare metal. Bare metal is much simpler. Details and Proxmox reports: Virtual machines and passthrough. Source: @darkmaniac7, 2026-10-29 (lab notes)

Verify

nvidia-smi --query-gpu=index,pcie.link.gen.current,pcie.link.width.current,memory.total --format=csv,noheader
nvidia-smi -q | grep -A1 "BAR1 Memory Usage"           # Total 65536 MiB
python3 tools/p2p-content-check.py                      # from the cmpunlocker repo; must exit 0

p2p-content-check.py copies random data between every ordered pair of cards and compares every byte. nvidia-smi topo and "peer access supported" alone do not prove P2P works. Morrowmake's README measured about 6.65 GB/s each way between two cards at Gen2 x16. Check that dmesg | grep Xid stays empty. Source: @darkmaniac7, 2026-10-29 (lab notes)

Undo

• Driver: sudo ./remove.sh --yes in the cmpunlocker repo, then reboot.
• BIOS: put back the values recorded by the collect script.
• Kernel: remove pci=realloc from /etc/default/grub, run update-grub, reboot.

BMC and fan settings on GPU servers

• Supermicro BMC Full mode persists across power loss, the zone duty does not. A board left in Full fan mode comes back from a cold boot with every fan at 100% until the OS rewrites the duty. Leave the mode at Optimal when the OS is down, or have the fan service read the duty back and rewrite it. Source: @darkmaniac7, 2026-10-27 (lab notes) The PixelML club measured the same on a Supermicro BMC: switching to Full and writing zone duties once left fans at 100% after a cold boot. Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @djpandaman, 2026-10-24
• EPYC boards without BMC fan curves: on fafinet's EPYC host (EPYC 7502 with an AST2500 BMC daughterboard; the message does not name the board) the smart fan curve is linear and can only be set in the BIOS, with no BMC control, and the other fan headers can be set with ipmitool raw 0x0e 0x65 0 (value). Source: @fafinet, 2026-10-26 GlobalApathy finds the ASRock Rack ROMED8-2T's fan curve settings weak too. Source: @globalapathy, 2026-10-22
• Fan noise on 2U GPU servers and thermal-target fan control: see Cooling and power.

Related

• Platforms, Consumer motherboards, Risers and bifurcation, Virtual machines and passthrough, Operating systems and kernels
• Gen2 link retrain, Resizable BAR and BAR1 64 GB, P2P, Kernel parameters and module options, Unlocks
• Troubleshooting, Card not detected or BAR allocation fails

Credits

Runbook by @darkmaniac7 (Ivan) from his own servers. The unlock is amoghmunikote/cmpunlocker; the P2P/ECC fork used on the reference box is Morrowmake/cmpunlocker.

Imported from the cmpunlocker Discord with credit; see Credits.