Supermicro H12DSG-O-CPU
From CMP Wiki
More actions
| Reported 8 GB | |
| CPU | Dual AMD EPYC Rome |
|---|---|
| Runs on | Bare metal |
| OS / driver | Ubuntu 24.04, kernel 6.8.0, nvidia-open 615.71.09, Morrowmake/cmpunlocker p2p-ecc 87cb5fd |
| Gen2 link | Gen2 x16 on all 8 cards |
| ReBAR / BAR1 | 64 GB BAR1 on all 8 cards (firmware sizes 64 MB, the patched driver resizes to 65536 MB) |
| P2P | Working on all 8 cards, across both sockets: p2p-content-check.py 504/504 with the Morrowmake P2P build; cudaDeviceCanAccessPeer = 1 for every pair |
| Settings that mattered | Above 4G Decoding on (firmware placed BAR1 above 4 GB); kernel command line amd_iommu=on iommu=pt, no pci=realloc; options nvidia NVreg_RegistryDwords="RmForceEnableGen2=1;RMPcieLinkSpeed=0x1;ForceP2P=0x11" (written by the installer). Other BIOS settings not recorded. |
| Tested by | @darkmaniac7 |
| Source | lab notes (@darkmaniac7) |
| Confirmations | None yet. |
The reference box for Server BIOS setup: 8 cards, all at Gen2 x16 with a 64 GB BAR1, on bare metal. Source: @darkmaniac7, 2026-10-29 (lab notes)
Setup
- Install:
sudo ./install.sh --p2p --profile=8gb --no-iommu --no-passthroughfrom Morrowmake/cmpunlocker, branchp2p-ecc, commit87cb5fd. - Re-run the install after every kernel upgrade: the installer replaces the DKMS modules. It prints a recovery boot option,
systemd.mask=gen2.service, for when the Gen2 boot service misbehaves. - After a cold power cycle all 8 cards came back at 65536 MiB, Gen2 x16, 0 Xid, 74 SMs on device 0.
Source: @darkmaniac7, 2026-10-27 (lab notes)
What the output looks like
See Server BIOS setup#What working looks like for the full nvidia-smi, lspci and dmesg output from this box.
Notes
- Dual socket, P2P across sockets works. The 8 GPU slots hang off both CPUs.
p2p-content-check.pypassed all 504 checks on 8 devices, which covers every ordered pair at every transfer size, including the pairs across the two sockets;cudaDeviceCanAccessPeerreturned 1 for every pair. Proven Source: @darkmaniac7, 2026-10-29 (lab notes)- Conflicting earlier source: Miséricorde said P2P does not work across sockets. Source: @wakeupmbrc1, 2026-10-18 See Risers and bifurcation#Topology rules of thumb.
- BMC fans. Full fan mode persists across power loss but the zone duty does not, so a cold boot in Full mode runs every fan at 100% until the OS rewrites the duty. Leave the mode at Optimal when the OS is down. Source: @darkmaniac7, 2026-10-27 (lab notes)
- Cold cycles. Wait about 20 s with the power off for the slots to discharge before powering on. Source: @darkmaniac7, 2026-10-27 (lab notes)
Other Supermicro H12 and H11 boards
No CMP results posted for these yet; they come up as host suggestions.
- H12SSL-i (single-socket SP3): gnasherrr found them new for about 280 USD on Facebook Marketplace, calling the ROMED8-2T at almost 4x the price hard to justify; djpandaman lists it among boards for a 6-card P2P build. Theory Source: @gnasherrr, 2026-10-22
Source: @djpandaman, 2026-10-22 - H11SSL-i rev 2.0: asked by Froggo, clee calls it a solid board if you do not need PCIe 4.0. Theory Source: @froggao, 2026-10-22
Source: @c133, 2026-10-22 - The "Supermicro H12D-8D" in some notes is the Huananzhi H12D-8D; see Huananzhi H12D-8D.