Supermicro H12DSG-O-CPU: Difference between revisions
From CMP Wiki
More actions
8x 8 GB CMP 170HX on bare metal with Gen2 x16, 64 GB BAR1 and P2P, from @darkmaniac7 (Ivan) lab notes; other Supermicro H12/H11 boards from Discord (gnasherrr, djpandaman, clee, Froggo) |
8x 8 GB CMP 170HX on bare metal with Gen2 x16, 64 GB BAR1 and P2P, from @darkmaniac7 (Ivan) lab notes; other Supermicro H12/H11 boards from Discord (gnasherrr, djpandaman, clee, Froggo) |
||
| Line 6: | Line 6: | ||
|gen2_result=Gen2 x16 on all 8 cards | |gen2_result=Gen2 x16 on all 8 cards | ||
|rebar_result=64 GB BAR1 on all 8 cards (firmware sizes 64 MB, the patched driver resizes to 65536 MB) | |rebar_result=64 GB BAR1 on all 8 cards (firmware sizes 64 MB, the patched driver resizes to 65536 MB) | ||
|p2p_result=Working | |p2p_result=Working on all 8 cards, across both sockets: <code><nowiki>p2p-content-check.py</nowiki></code> 504/504 with the Morrowmake P2P build; <code><nowiki>cudaDeviceCanAccessPeer</nowiki></code> = 1 for every pair | ||
|os_driver=Ubuntu 24.04, kernel 6.8.0, nvidia-open 615.71.09, Morrowmake/cmpunlocker <code><nowiki>p2p-ecc</nowiki></code> <code><nowiki>87cb5fd</nowiki></code> | |os_driver=Ubuntu 24.04, kernel 6.8.0, nvidia-open 615.71.09, Morrowmake/cmpunlocker <code><nowiki>p2p-ecc</nowiki></code> <code><nowiki>87cb5fd</nowiki></code> | ||
|virtualization=Bare metal | |virtualization=Bare metal | ||
| Line 27: | Line 27: | ||
== Notes == | == Notes == | ||
* '''Dual socket.''' The 8 GPU slots hang off both CPUs. <code><nowiki>cudaDeviceCanAccessPeer</nowiki></code> returned 1 for | * '''Dual socket, P2P across sockets works.''' The 8 GPU slots hang off both CPUs. <code><nowiki>p2p-content-check.py</nowiki></code> passed all 504 checks on 8 devices, which covers every ordered pair at every transfer size, including the pairs across the two sockets; <code><nowiki>cudaDeviceCanAccessPeer</nowiki></code> returned 1 for every pair. {{Status|Proven}} {{Source|handle=darkmaniac7|date=2026-10-29|note=lab notes}} | ||
** Conflicting earlier source: Miséricorde said P2P does not work across sockets. {{Source|handle=wakeupmbrc1|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561407364622458943}} See [[Risers and bifurcation#Topology rules of thumb]]. | |||
* '''BMC fans.''' Full fan mode persists across power loss but the zone duty does not, so a cold boot in Full mode runs every fan at 100% until the OS rewrites the duty. Leave the mode at Optimal when the OS is down. {{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}} | * '''BMC fans.''' Full fan mode persists across power loss but the zone duty does not, so a cold boot in Full mode runs every fan at 100% until the OS rewrites the duty. Leave the mode at Optimal when the OS is down. {{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}} | ||
* '''Cold cycles.''' Wait about 20 s with the power off for the slots to discharge before powering on. {{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}} | * '''Cold cycles.''' Wait about 20 s with the power off for the slots to discharge before powering on. {{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}} | ||
Latest revision as of 08:11, 30 October 2026
| Reported 8 GB | |
| CPU | Dual AMD EPYC Rome |
|---|---|
| Runs on | Bare metal |
| OS / driver | Ubuntu 24.04, kernel 6.8.0, nvidia-open 615.71.09, Morrowmake/cmpunlocker p2p-ecc 87cb5fd |
| Gen2 link | Gen2 x16 on all 8 cards |
| ReBAR / BAR1 | 64 GB BAR1 on all 8 cards (firmware sizes 64 MB, the patched driver resizes to 65536 MB) |
| P2P | Working on all 8 cards, across both sockets: p2p-content-check.py 504/504 with the Morrowmake P2P build; cudaDeviceCanAccessPeer = 1 for every pair |
| Settings that mattered | Above 4G Decoding on (firmware placed BAR1 above 4 GB); kernel command line amd_iommu=on iommu=pt, no pci=realloc; options nvidia NVreg_RegistryDwords="RmForceEnableGen2=1;RMPcieLinkSpeed=0x1;ForceP2P=0x11" (written by the installer). Other BIOS settings not recorded. |
| Tested by | @darkmaniac7 |
| Source | lab notes (@darkmaniac7) |
| Confirmations | None yet. |
The reference box for Server BIOS setup: 8 cards, all at Gen2 x16 with a 64 GB BAR1, on bare metal. Source: @darkmaniac7, 2026-10-29 (lab notes)
Setup
- Install:
sudo ./install.sh --p2p --profile=8gb --no-iommu --no-passthroughfrom Morrowmake/cmpunlocker, branchp2p-ecc, commit87cb5fd. - Re-run the install after every kernel upgrade: the installer replaces the DKMS modules. It prints a recovery boot option,
systemd.mask=gen2.service, for when the Gen2 boot service misbehaves. - After a cold power cycle all 8 cards came back at 65536 MiB, Gen2 x16, 0 Xid, 74 SMs on device 0.
Source: @darkmaniac7, 2026-10-27 (lab notes)
What the output looks like
See Server BIOS setup#What working looks like for the full nvidia-smi, lspci and dmesg output from this box.
Notes
- Dual socket, P2P across sockets works. The 8 GPU slots hang off both CPUs.
p2p-content-check.pypassed all 504 checks on 8 devices, which covers every ordered pair at every transfer size, including the pairs across the two sockets;cudaDeviceCanAccessPeerreturned 1 for every pair. Proven Source: @darkmaniac7, 2026-10-29 (lab notes)- Conflicting earlier source: Miséricorde said P2P does not work across sockets. Source: @wakeupmbrc1, 2026-10-18 See Risers and bifurcation#Topology rules of thumb.
- BMC fans. Full fan mode persists across power loss but the zone duty does not, so a cold boot in Full mode runs every fan at 100% until the OS rewrites the duty. Leave the mode at Optimal when the OS is down. Source: @darkmaniac7, 2026-10-27 (lab notes)
- Cold cycles. Wait about 20 s with the power off for the slots to discharge before powering on. Source: @darkmaniac7, 2026-10-27 (lab notes)
Other Supermicro H12 and H11 boards
No CMP results posted for these yet; they come up as host suggestions.
- H12SSL-i (single-socket SP3): gnasherrr found them new for about 280 USD on Facebook Marketplace, calling the ROMED8-2T at almost 4x the price hard to justify; djpandaman lists it among boards for a 6-card P2P build. Theory Source: @gnasherrr, 2026-10-22
Source: @djpandaman, 2026-10-22 - H11SSL-i rev 2.0: asked by Froggo, clee calls it a solid board if you do not need PCIe 4.0. Theory Source: @froggao, 2026-10-22
Source: @c133, 2026-10-22 - The "Supermicro H12D-8D" in some notes is the Huananzhi H12D-8D; see Huananzhi H12D-8D.