ImportBot: 8x 8 GB CMP 170HX on bare metal with Gen2 x16, 64 GB BAR1 and P2P, from @darkmaniac7 (Ivan) lab notes; other Supermicro H12/H11 boards from Discord (gnasherrr, djpandaman, clee, Froggo)

2026-10-30T08:11:09Z

8x 8 GB CMP 170HX on bare metal with Gen2 x16, 64 GB BAR1 and P2P, from @darkmaniac7 (Ivan) lab notes; other Supermicro H12/H11 boards from Discord (gnasherrr, djpandaman, clee, Froggo)

← Older revision		Revision as of 08:11, 30 October 2026
Line 6:		Line 6:
	\|gen2_result=Gen2 x16 on all 8 cards		\|gen2_result=Gen2 x16 on all 8 cards
	\|rebar_result=64 GB BAR1 on all 8 cards (firmware sizes 64 MB, the patched driver resizes to 65536 MB)		\|rebar_result=64 GB BAR1 on all 8 cards (firmware sizes 64 MB, the patched driver resizes to 65536 MB)
	\|p2p_result=Working ~~with the Morrowmake~~ <code><nowiki>p2p-~~ecc~~</nowiki></code> build; <code><nowiki>cudaDeviceCanAccessPeer</nowiki></code> = 1 ~~including a~~ pair ~~across sockets; verify with <code><nowiki>p2p-content-check.py</nowiki></code>~~		\|p2p_result=Working on all 8 cards, across both sockets: <code><nowiki>p2p-content-check.py</nowiki></code> 504/504 with the Morrowmake P2P build; <code><nowiki>cudaDeviceCanAccessPeer</nowiki></code> = 1 for every pair
	\|os_driver=Ubuntu 24.04, kernel 6.8.0, nvidia-open 615.71.09, Morrowmake/cmpunlocker <code><nowiki>p2p-ecc</nowiki></code> <code><nowiki>87cb5fd</nowiki></code>		\|os_driver=Ubuntu 24.04, kernel 6.8.0, nvidia-open 615.71.09, Morrowmake/cmpunlocker <code><nowiki>p2p-ecc</nowiki></code> <code><nowiki>87cb5fd</nowiki></code>
	\|virtualization=Bare metal		\|virtualization=Bare metal
Line 27:		Line 27:

	== Notes ==		== Notes ==
	* '''Dual socket.''' The 8 GPU slots hang off both CPUs. <code><nowiki>cudaDeviceCanAccessPeer</nowiki></code> returned 1 for ~~pairs on the same socket and for a~~ pair ~~across sockets; the notes only kept the tail of the byte-check run. Whether cross-socket P2P is worth using is disputed on Discord; see [[Risers and bifurcation#Topology rules of thumb]]~~. {{Source\|handle=darkmaniac7\|date=2026-10-27\|note=lab notes}}		* '''Dual socket, P2P across sockets works.''' The 8 GPU slots hang off both CPUs. <code><nowiki>p2p-content-check.py</nowiki></code> passed all 504 checks on 8 devices, which covers every ordered pair at every transfer size, including the pairs across the two sockets; <code><nowiki>cudaDeviceCanAccessPeer</nowiki></code> returned 1 for every pair. {{Status\|Proven}} {{Source\|handle=darkmaniac7\|date=2026-10-29\|note=lab notes}}
			** Conflicting earlier source: Miséricorde said P2P does not work across sockets. {{Source\|handle=wakeupmbrc1\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561407364622458943}} See [[Risers and bifurcation#Topology rules of thumb]].
	* '''BMC fans.''' Full fan mode persists across power loss but the zone duty does not, so a cold boot in Full mode runs every fan at 100% until the OS rewrites the duty. Leave the mode at Optimal when the OS is down. {{Source\|handle=darkmaniac7\|date=2026-10-27\|note=lab notes}}		* '''BMC fans.''' Full fan mode persists across power loss but the zone duty does not, so a cold boot in Full mode runs every fan at 100% until the OS rewrites the duty. Leave the mode at Optimal when the OS is down. {{Source\|handle=darkmaniac7\|date=2026-10-27\|note=lab notes}}
	* '''Cold cycles.''' Wait about 20 s with the power off for the slots to discharge before powering on. {{Source\|handle=darkmaniac7\|date=2026-10-27\|note=lab notes}}		* '''Cold cycles.''' Wait about 20 s with the power off for the slots to discharge before powering on. {{Source\|handle=darkmaniac7\|date=2026-10-27\|note=lab notes}}

ImportBot: 8x 8 GB CMP 170HX on bare metal with Gen2 x16, 64 GB BAR1 and P2P, from @darkmaniac7 (Ivan) lab notes; other Supermicro H12/H11 boards from Discord (gnasherrr, djpandaman, clee, Froggo)

2026-10-30T08:05:04Z

8x 8 GB CMP 170HX on bare metal with Gen2 x16, 64 GB BAR1 and P2P, from @darkmaniac7 (Ivan) lab notes; other Supermicro H12/H11 boards from Discord (gnasherrr, djpandaman, clee, Froggo)

New page

{{Platform
|board=Supermicro H12DSG-O-CPU (Supermicro AS-4124GS-TNR 4U GPU server)
|cpu=Dual AMD EPYC Rome
|bios_version=
|settings=Above 4G Decoding on (firmware placed BAR1 above 4 GB); kernel command line <code><nowiki>amd_iommu=on iommu=pt</nowiki></code>, no <code><nowiki>pci=realloc</nowiki></code>; <code><nowiki>options nvidia NVreg_RegistryDwords="RmForceEnableGen2=1;RMPcieLinkSpeed=0x1;ForceP2P=0x11"</nowiki></code> (written by the installer). Other BIOS settings not recorded.
|gen2_result=Gen2 x16 on all 8 cards
|rebar_result=64 GB BAR1 on all 8 cards (firmware sizes 64 MB, the patched driver resizes to 65536 MB)
|p2p_result=Working with the Morrowmake <code><nowiki>p2p-ecc</nowiki></code> build; <code><nowiki>cudaDeviceCanAccessPeer</nowiki></code> = 1 including a pair across sockets; verify with <code><nowiki>p2p-content-check.py</nowiki></code>
|os_driver=Ubuntu 24.04, kernel 6.8.0, nvidia-open 615.71.09, Morrowmake/cmpunlocker <code><nowiki>p2p-ecc</nowiki></code> <code><nowiki>87cb5fd</nowiki></code>
|virtualization=Bare metal
|cards=8 GB
|status=Reported
|tested_by=darkmaniac7
|source=lab notes (@darkmaniac7)
}}

The reference box for [[Server BIOS setup]]: 8 cards, all at Gen2 x16 with a 64 GB BAR1, on bare metal. {{Source|handle=darkmaniac7|date=2026-10-29|note=lab notes}}

== Setup ==
* Install: <code><nowiki>sudo ./install.sh --p2p --profile=8gb --no-iommu --no-passthrough</nowiki></code> from [https://github.com/Morrowmake/cmpunlocker Morrowmake/cmpunlocker], branch <code><nowiki>p2p-ecc</nowiki></code>, commit <code><nowiki>87cb5fd</nowiki></code>.
* Re-run the install after every kernel upgrade: the installer replaces the DKMS modules. It prints a recovery boot option, <code><nowiki>systemd.mask=gen2.service</nowiki></code>, for when the Gen2 boot service misbehaves.
* After a cold power cycle all 8 cards came back at 65536 MiB, Gen2 x16, 0 Xid, 74 SMs on device 0.
{{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}

== What the output looks like ==
See [[Server BIOS setup#What working looks like]] for the full <code><nowiki>nvidia-smi</nowiki></code>, <code><nowiki>lspci</nowiki></code> and <code><nowiki>dmesg</nowiki></code> output from this box.

== Notes ==
* '''Dual socket.''' The 8 GPU slots hang off both CPUs. <code><nowiki>cudaDeviceCanAccessPeer</nowiki></code> returned 1 for pairs on the same socket and for a pair across sockets; the notes only kept the tail of the byte-check run. Whether cross-socket P2P is worth using is disputed on Discord; see [[Risers and bifurcation#Topology rules of thumb]]. {{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}
* '''BMC fans.''' Full fan mode persists across power loss but the zone duty does not, so a cold boot in Full mode runs every fan at 100% until the OS rewrites the duty. Leave the mode at Optimal when the OS is down. {{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}
* '''Cold cycles.''' Wait about 20 s with the power off for the slots to discharge before powering on. {{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}

== Other Supermicro H12 and H11 boards ==
No CMP results posted for these yet; they come up as host suggestions.
* '''H12SSL-i''' (single-socket SP3): gnasherrr found them new for about 280 USD on Facebook Marketplace, calling the ROMED8-2T at almost 4x the price hard to justify; djpandaman lists it among boards for a 6-card P2P build. {{Status|Theory}} {{Source|handle=gnasherrr|date=2026-10-22|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562668620725747794}}<br />{{Source|handle=djpandaman|date=2026-10-22|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562874799305261067}}
* '''H11SSL-i rev 2.0''': asked by Froggo, clee calls it a solid board if you do not need PCIe 4.0. {{Status|Theory}} {{Source|handle=froggao|date=2026-10-22|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562655307556192366}}<br />{{Source|handle=c133|date=2026-10-22|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562655771701944413}}
* The "Supermicro H12D-8D" in some notes is the Huananzhi H12D-8D; see [[Huananzhi H12D-8D]].

== Related ==
[[Platforms]], [[Server BIOS setup]], [[Gen2 link retrain]], [[Resizable BAR and BAR1 64 GB]], [[Cooling and power]].

{{ImportFooter}}
[[Category:8 GB]]

Supermicro H12DSG-O-CPU - Revision history

ImportBot: 8x 8 GB CMP 170HX on bare metal with Gen2 x16, 64 GB BAR1 and P2P, from @darkmaniac7 (Ivan) lab notes; other Supermicro H12/H11 boards from Discord (gnasherrr, djpandaman, clee, Froggo)

ImportBot: 8x 8 GB CMP 170HX on bare metal with Gen2 x16, 64 GB BAR1 and P2P, from @darkmaniac7 (Ivan) lab notes; other Supermicro H12/H11 boards from Discord (gnasherrr, djpandaman, clee, Froggo)