P2P - Revision history

ImportBot: GPU-to-GPU peer-to-peer on the CMP 170HX: the mailbox path (Morrowmake/asm64 forks), the BAR1 path, the ForceP2P registry key, topology limits and verification. Discord Oct 2026. Gap-fill from #general 2026-10-24 to 10-25 (PLX 88096 P2P failure), @replacedriver47, @djpandaman, @nathang6866.

2026-10-30T10:02:21Z

GPU-to-GPU peer-to-peer on the CMP 170HX: the mailbox path (Morrowmake/asm64 forks), the BAR1 path, the ForceP2P registry key, topology limits and verification. Discord Oct 2026. Gap-fill from #general 2026-10-24 to 10-25 (PLX 88096 P2P failure), @replacedriver47, @djpandaman, @nathang6866.

← Older revision		Revision as of 10:02, 30 October 2026
Line 65:		Line 65:
	applies even on a single-switch setup. Measured cross-root P2P latency about 1.5 us (not through a switch). His TP2 on		applies even on a single-switch setup. Measured cross-root P2P latency about 1.5 us (not through a switch). His TP2 on
	2x CMP: about 4,900 tok/s prefill at 128k (TTFT 26 s) versus about 2,500 without P2P; decode about 223 tok/s at one		2x CMP: about 4,900 tok/s prefill at 128k (TTFT 26 s) versus about 2,500 without P2P; decode about 223 tok/s at one
	session, 527 tok/s aggregate at four (W4A16, MTP4). {{Source\|ropuls\|2026-10-15\|https://discord.com/channels/1518634379604136178/1528826330454429706/~~1560412308469973125~~}}		session, 527 tok/s aggregate at four (W4A16, MTP4). {{Source\|handle=mountains1605\|date=2026-10-15\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560433160364490782}}<br />{{Source\|handle=ropuls\|date=2026-10-16\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560593914363256965}}

	== Verify ==		== Verify ==
Line 84:		Line 84:
	* '''Dual-socket:''' P2P does not work across sockets; GPU servers hang all cards off one CPU's root port. Mitigations		* '''Dual-socket:''' P2P does not work across sockets; GPU servers hang all cards off one CPU's root port. Mitigations
	are the mailbox method or bayley's kernel patch. {{Source\|wakeupmbrc1\|2026-10-18\|https://discord.com/channels/1518634379604136178/1518692084729511967/1561407364622458943}}		are the mailbox method or bayley's kernel patch. {{Source\|wakeupmbrc1\|2026-10-18\|https://discord.com/channels/1518634379604136178/1518692084729511967/1561407364622458943}}
			** Measured counterexample: on an 8-card dual-EPYC Supermicro H12DSG-O-CPU, the Morrowmake P2P build on 615.71.09 passed 504 of 504 content checks including every cross-socket pair, with <code>cuDeviceCanAccessPeer</code> 1 everywhere and 0 Xids, again after later cold cycles. {{Status\|Proven}} See [[Supermicro H12DSG-O-CPU]]. {{Source\|handle=darkmaniac7\|date=2026-10-27\|note=lab notes}}
	* '''Latency reference:''' about 1.4-1.7 us native (CPU root / crossbar), about 14 us through a retimer plus two		* '''Latency reference:''' about 1.4-1.7 us native (CPU root / crossbar), about 14 us through a retimer plus two
	switches. {{Source\|markshowa\|2026-10-17\|https://discord.com/channels/1518634379604136178/1518692084729511967/1561013734808363029}}		switches. {{Source\|markshowa\|2026-10-17\|https://discord.com/channels/1518634379604136178/1518692084729511967/1561013734808363029}}
			* '''PLX 88096 with a 64 GB BAR1, P2P still failing:''' four cards behind a PLX 88096; after MMIOH and BAR tuning two of
			them had a 64 GB BAR1, but P2P transfers between them never completed, with Morrowmake's cmpunlocker as well. The root
			cause is unknown. People point to PixelML's 4-card BAR1 P2P over PLX notebook as the reference, and ElectroHobo thinks
			Morrowmake's vLLM fixes it for many people on supported hardware. Details: [[Risers and bifurcation]].
			{{Source\|handle=replacedriver47\|date=2026-10-24\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563629971082186753}}<br />
			{{Source\|handle=djpandaman\|date=2026-10-24\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563629622673936384\|repo=https://github.com/PixelML/club-170hx}}<br />
			{{Source\|handle=nathang6866\|date=2026-10-25\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1564023832107417662}}
	* P2P across different root complexes on a Threadripper board (older asm64 fork) looked right but no traffic arrived;		* P2P across different root complexes on a Threadripper board (older asm64 fork) looked right but no traffic arrived;
	mrpandagod's advice is to build a local P2P patch for the specific setup.		mrpandagod's advice is to build a local P2P patch for the specific setup.
Line 95:		Line 103:
	* On boards that cannot route P2P (ACS, no IOMMU passthrough, separate roots), forcing it times out instead of staging		* On boards that cannot route P2P (ACS, no IOMMU passthrough, separate roots), forcing it times out instead of staging
	through system memory, which is why it is opt-in.		through system memory, which is why it is opt-in.
			* {{Status\|Contested}} '''Does P2P make inference faster?''' It depends on host and model; A/B it per recipe. In the order posted:
			** ElectroHobo: on a Broadwell host with one PLX switch, MiMo V2.6 Flash TP4 is about 40% slower with P2P off. {{Source\|handle=nathang6866\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1561264547455569952\|repo=https://github.com/ElectroHobo/mimo-v26-flash-cmp170hx-recipe}}
			** PixelML: on four cards behind two PLX switches on a Broadwell host, P2P helped Qwen3.8-27B TP2 decode (+11%) but hurt GLM 5.3 Flash TP4 (-20 to -26% for one user, -38 to -48% at eight users); they run TP4 with P2P off. Their notes cite Morrowmake measuring +11% from P2P on EPYC root ports. {{Source\|handle=c133\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127\|repo=https://github.com/PixelML/club-170hx}}
			** On an EPYC lab host, vLLM's custom all-reduce over P2P cost GLM 5.3 Flash TP4 about 40% decode but helped Qwen3.8 Flash Next TP2 by 17 to 25%. {{Source\|handle=darkmaniac7\|date=2026-10-28\|note=lab notes}}

	== Status ==		== Status ==

ImportBot: GPU-to-GPU peer-to-peer on the CMP 170HX: the mailbox path (Morrowmake/asm64 forks), the BAR1 path, the ForceP2P registry key, topology limits and verification. Discord Oct 2026.; sources: @curiouspixel, @ob88594, @seriousrising, @68x0, @ropuls, @wakeupmbrc1, @markshowa, @c133; GitHub Morrowmake/cmpunlocker

2026-10-30T08:49:28Z

GPU-to-GPU peer-to-peer on the CMP 170HX: the mailbox path (Morrowmake/asm64 forks), the BAR1 path, the ForceP2P registry key, topology limits and verification. Discord Oct 2026.; sources: @curiouspixel, @ob88594, @seriousrising, @68x0, @ropuls, @wakeupmbrc1, @markshowa, @c133; GitHub Morrowmake/cmpunlocker

← Older revision		Revision as of 08:49, 30 October 2026
Line 58:		Line 58:
	{{Status\|warning}} These are '''global''' and force every NVIDIA GPU in the host to Gen2, so do not use them on a mixed		{{Status\|warning}} These are '''global''' and force every NVIDIA GPU in the host to Gen2, so do not use them on a mixed
	rig (e.g. with an RTX 4080). {{Source\|68x0\|2026-10-13\|https://discord.com/channels/1518634379604136178/1518695289278566450/1559704654743474310}}		rig (e.g. with an RTX 4080). {{Source\|68x0\|2026-10-13\|https://discord.com/channels/1518634379604136178/1518695289278566450/1559704654743474310}}
	The gist ~~measured about 1~~.~~5 us~~ cross-root P2P latency (not through a switch). ~~A longer write-up by @ropuls exists~~ (~~the~~
	~~gist~~, https://~~gist.github~~.com/~~ropuls~~/~~d2978531ba3a1a183e70f62f3de2eb84); see Discord.~~		The gist ([https://gist.github.com/ropuls/d2978531ba3a1a183e70f62f3de2eb84 gist.github.com/ropuls]) covers the unlock,
			cross-root P2P and vLLM serving. Its key points: P2P can fail because the cmpunlocker 0011 patch sets the P2P override
			before the PCI device id is populated, so the driver falls back to GSP caps, which report GNS for the CMP regardless of
			topology. Check with <code>nvidia-smi topo -p2p r</code>; if it says GNS, the <code>ForceP2P=0x11</code> registry key
			applies even on a single-switch setup. Measured cross-root P2P latency about 1.5 us (not through a switch). His TP2 on
			2x CMP: about 4,900 tok/s prefill at 128k (TTFT 26 s) versus about 2,500 without P2P; decode about 223 tok/s at one
			session, 527 tok/s aggregate at four (W4A16, MTP4). {{Source\|ropuls\|2026-10-15\|https://discord.com/channels/1518634379604136178/1528826330454429706/1560412308469973125}}

	== Verify ==		== Verify ==

ImportBot: GPU-to-GPU peer-to-peer on the CMP 170HX: the mailbox path (Morrowmake/asm64 forks), the BAR1 path, the ForceP2P registry key, topology limits and verification. Discord Oct 2026.; sources: @curiouspixel, @ob88594, @seriousrising, @68x0, @wakeupmbrc1, @markshowa, @c133; GitHub Morrowmake/cmpunlocker

2026-10-30T08:04:53Z

GPU-to-GPU peer-to-peer on the CMP 170HX: the mailbox path (Morrowmake/asm64 forks), the BAR1 path, the ForceP2P registry key, topology limits and verification. Discord Oct 2026.; sources: @curiouspixel, @ob88594, @seriousrising, @68x0, @wakeupmbrc1, @markshowa, @c133; GitHub Morrowmake/cmpunlocker

New page

{{Unlock
|unlocks=GPU-to-GPU P2P (mailbox / BAR1)
|variant=both
|driver_version=nvidia-open 610.57.04 (measured) / 615.71.09
|unlock_build=Morrowmake/cmpunlocker (--p2p) or asm64-hooligan (--p2p)
|steps_page=P2P
|status=Reported
|tested_by=curiouspixel
|source=https://github.com/Morrowmake/cmpunlocker
}}

GSP reports PCIe P2P unsupported on the CMP, but the GA100 mailbox P2P hardware is present. P2P matters for tensor
parallelism; without it, pipeline parallel is usually the better default on these PCIe-only cards. There are two paths.

== Mailbox P2P (works on 64 MB BAR) ==

The [[cmpunlocker forks|Morrowmake fork]]'s <code>--p2p</code> and asm64's <code>--p2p</code> force the driver to
advertise peer access and arm the mailbox trap. This runs in a 512 KB window inside the stock 64 MB BAR1, so it needs
no ReBAR and no Above 4G decoding, and works where the board cannot open the 64 GB BAR.
{{Source|ob88594|2026-10-05|https://discord.com/channels/1518634379604136178/1518692084729511967/1556527198540922981}}

curiouspixel's P2P patch in the Morrowmake fork fixed P2P for Trust Inter and GhostInTheDatacenter on the same board.
{{Source|curiouspixel|2026-10-05|https://discord.com/channels/1518634379604136178/1518692084729511967/1556489123999780987}}
Morrowmake's measured 4-card result: all 12 ordered pairs passed content checks; 16 MiB copies reached about
'''6.65 GB/s''' each way (610.57.04, Gen2 x16). asm64's mailbox is about 1.7 GB/s behind one switch at Gen2 x4.

Mechanism (Morrowmake <code>p2p-unlock.patch</code>): it forces <code>pcieP2PReadCaps/WriteCaps</code> OK for
<code>0x20C2</code>/<code>0x2082</code> only when PRI decode trap 31's PLM <code>0x0012277c</code> reads
<code>0xFFFFFFFF</code>; around the mailbox setup it arms trap 31 and restores it after. The Booter change adds
<code>TRAP31_PLM 0x0012277c = 0xffffffff</code>, opened only when the <code>ForceP2P</code> registry key is set. On a
part whose trap PLM read <code>0xFFFFFF8F</code> the mailbox could not be programmed and a copy silently returned 33 of
33.5 MB wrong (only the first 128 KiB intact), which is why the caps are forced only when the trap is writable and why
a content check is required.

Install and the registry line (Morrowmake):

<syntaxhighlight lang="bash">
sudo ./install.sh --p2p --profile=8gb --no-iommu --no-passthrough # --profile=10gb for 10 GB cards
# writes:
# options nvidia NVreg_RegistryDwords="RmForceEnableGen2=1;RMPcieLinkSpeed=0x1;ForceP2P=0x11"
</syntaxhighlight>

== BAR1 P2P (needs the 64 GB BAR) ==

The other path uses the [[Resizable BAR and BAR1 64 GB|64 GB BAR1]] (bayley's research). It is a few percent faster
than mailbox under pipeline parallelism but needs a board that can open the big BAR. Where the board cannot, use
mailbox. {{Source|seriousrising|2026-10-05|https://discord.com/channels/1518634379604136178/1518692084729511967/1556499525579837481}}

== ropuls gist (cross-root P2P + serving) ==

68x reported higher decode/prefill after the extra modprobe parameters from ropuls' gist "2x CMP 170HX for LLM
inference: unlock, cross-root P2P, serving (vLLM)":

<syntaxhighlight lang="text">
options nvidia NVreg_RegistryDwords="RmForceEnableGen2=1;RMPcieLinkSpeed=0x1;RMForceStaticBar1=1;RMPcieP2PType=1;RMForceP2PType=1;ForceP2P=0x11"
</syntaxhighlight>

{{Status|warning}} These are '''global''' and force every NVIDIA GPU in the host to Gen2, so do not use them on a mixed
rig (e.g. with an RTX 4080). {{Source|68x0|2026-10-13|https://discord.com/channels/1518634379604136178/1518695289278566450/1559704654743474310}}
The gist measured about 1.5 us cross-root P2P latency (not through a switch). A longer write-up by @ropuls exists (the
gist, https://gist.github.com/ropuls/d2978531ba3a1a183e70f62f3de2eb84); see Discord.

== Verify ==

<syntaxhighlight lang="bash">
nvidia-smi topo -p2p r # OK, not GNS
nvidia-smi topo -m
python3 tools/p2p-content-check.py # Morrowmake: verifies the bytes actually arrive
</syntaxhighlight>

<code>nvidia-smi topo</code> and advertised peer support alone do not prove the data moves; always run a content check.

== Topology limits ==

* '''PLX switches:''' many boards have MMIO problems behind a PLX, so only mailbox P2P works; in '''TP''' mode that
mailbox P2P can throw '''XID 13''' and crash the inference process. Avoid PLX risers for TP, or disable P2P.
{{Source|ob88594|2026-10-07|https://discord.com/channels/1518634379604136178/1518692084729511967/1557383845844615241}}
* '''Dual-socket:''' P2P does not work across sockets; GPU servers hang all cards off one CPU's root port. Mitigations
are the mailbox method or bayley's kernel patch. {{Source|wakeupmbrc1|2026-10-18|https://discord.com/channels/1518634379604136178/1518692084729511967/1561407364622458943}}
* '''Latency reference:''' about 1.4-1.7 us native (CPU root / crossbar), about 14 us through a retimer plus two
switches. {{Source|markshowa|2026-10-17|https://discord.com/channels/1518634379604136178/1518692084729511967/1561013734808363029}}
* P2P across different root complexes on a Threadripper board (older asm64 fork) looked right but no traffic arrived;
mrpandagod's advice is to build a local P2P patch for the specific setup.
{{Source|c133|2026-10-22|https://discord.com/channels/1518634379604136178/1534370835220004885/1562876840593457225}}

== Known issues ==

* No release ships P2P and ECC together; the Morrowmake fork drops the ECC patches. A v0.6 request is open.
* On boards that cannot route P2P (ACS, no IOMMU passthrough, separate roots), forcing it times out instead of staging
through system memory, which is why it is opt-in.

== Status ==

Mailbox P2P reproduced on the same board by two people (curiouspixel's patch, Trust Inter and GhostInTheDatacenter) and
measured on a 4-card rig, so {{Status|Proven}} where the topology allows; it is {{Status|Reported}} to
{{Status|Contested}} on PLX and cross-root setups. See the confirmation below.

== Related ==

[[cmpunlocker forks]], [[Resizable BAR and BAR1 64 GB]], [[Platforms]], [[Kernel parameters and module options]],
[[Troubleshooting]], [[Tools]].

{{ImportFooter}}
[[Category:Unlocks]]
[[Category:8 GB]]
[[Category:10 GB]]