P2P

GPU-to-GPU P2P (mailbox / BAR1)
Proven 8 GB and 10 GB
Driver	nvidia-open 610.57.04 (measured) / 615.71.09
Unlock build	Morrowmake/cmpunlocker (--p2p) or asm64-hooligan (--p2p)
Steps	P2P
Source	link
Confirmations	Reproduced by @ghostinthedatacenter_87281 (both), 2026-10-07 · details

GSP reports PCIe P2P unsupported on the CMP, but the GA100 mailbox P2P hardware is present. P2P matters for tensor parallelism; without it, pipeline parallel is usually the better default on these PCIe-only cards. There are two paths.

Mailbox P2P (works on 64 MB BAR)

The Morrowmake fork's --p2p and asm64's --p2p force the driver to advertise peer access and arm the mailbox trap. This runs in a 512 KB window inside the stock 64 MB BAR1, so it needs no ReBAR and no Above 4G decoding, and works where the board cannot open the 64 GB BAR. Source: @ob88594, 2026-10-05

curiouspixel's P2P patch in the Morrowmake fork fixed P2P for Trust Inter and GhostInTheDatacenter on the same board. Source: @curiouspixel, 2026-10-05 Morrowmake's measured 4-card result: all 12 ordered pairs passed content checks; 16 MiB copies reached about 6.65 GB/s each way (610.57.04, Gen2 x16). asm64's mailbox is about 1.7 GB/s behind one switch at Gen2 x4.

Mechanism (Morrowmake p2p-unlock.patch): it forces pcieP2PReadCaps/WriteCaps OK for 0x20C2/0x2082 only when PRI decode trap 31's PLM 0x0012277c reads 0xFFFFFFFF; around the mailbox setup it arms trap 31 and restores it after. The Booter change adds TRAP31_PLM 0x0012277c = 0xffffffff, opened only when the ForceP2P registry key is set. On a part whose trap PLM read 0xFFFFFF8F the mailbox could not be programmed and a copy silently returned 33 of 33.5 MB wrong (only the first 128 KiB intact), which is why the caps are forced only when the trap is writable and why a content check is required.

Install and the registry line (Morrowmake):

sudo ./install.sh --p2p --profile=8gb --no-iommu --no-passthrough   # --profile=10gb for 10 GB cards
# writes:
# options nvidia NVreg_RegistryDwords="RmForceEnableGen2=1;RMPcieLinkSpeed=0x1;ForceP2P=0x11"

BAR1 P2P (needs the 64 GB BAR)

The other path uses the 64 GB BAR1 (bayley's research). It is a few percent faster than mailbox under pipeline parallelism but needs a board that can open the big BAR. Where the board cannot, use mailbox. Source: @seriousrising, 2026-10-05

ropuls gist (cross-root P2P + serving)

68x reported higher decode/prefill after the extra modprobe parameters from ropuls' gist "2x CMP 170HX for LLM inference: unlock, cross-root P2P, serving (vLLM)":

options nvidia NVreg_RegistryDwords="RmForceEnableGen2=1;RMPcieLinkSpeed=0x1;RMForceStaticBar1=1;RMPcieP2PType=1;RMForceP2PType=1;ForceP2P=0x11"

Warning These are global and force every NVIDIA GPU in the host to Gen2, so do not use them on a mixed rig (e.g. with an RTX 4080). Source: @68x0, 2026-10-13

The gist (gist.github.com/ropuls) covers the unlock, cross-root P2P and vLLM serving. Its key points: P2P can fail because the cmpunlocker 0011 patch sets the P2P override before the PCI device id is populated, so the driver falls back to GSP caps, which report GNS for the CMP regardless of topology. Check with nvidia-smi topo -p2p r; if it says GNS, the ForceP2P=0x11 registry key applies even on a single-switch setup. Measured cross-root P2P latency about 1.5 us (not through a switch). His TP2 on 2x CMP: about 4,900 tok/s prefill at 128k (TTFT 26 s) versus about 2,500 without P2P; decode about 223 tok/s at one session, 527 tok/s aggregate at four (W4A16, MTP4). Source: @mountains1605, 2026-10-15
Source: @ropuls, 2026-10-16

Verify

nvidia-smi topo -p2p r          # OK, not GNS
nvidia-smi topo -m
python3 tools/p2p-content-check.py   # Morrowmake: verifies the bytes actually arrive

nvidia-smi topo and advertised peer support alone do not prove the data moves; always run a content check.

Topology limits

PLX switches: many boards have MMIO problems behind a PLX, so only mailbox P2P works; in TP mode that

 mailbox P2P can throw XID 13 and crash the inference process. Avoid PLX risers for TP, or disable P2P.
 Source: @ob88594, 2026-10-07

Dual-socket: P2P does not work across sockets; GPU servers hang all cards off one CPU's root port. Mitigations

 are the mailbox method or bayley's kernel patch. Source: @wakeupmbrc1, 2026-10-18

- Measured counterexample: on an 8-card dual-EPYC Supermicro H12DSG-O-CPU, the Morrowmake P2P build on 615.71.09 passed 504 of 504 content checks including every cross-socket pair, with cuDeviceCanAccessPeer 1 everywhere and 0 Xids, again after later cold cycles. Proven See Supermicro H12DSG-O-CPU. Source: @darkmaniac7, 2026-10-27 (lab notes)
Latency reference: about 1.4-1.7 us native (CPU root / crossbar), about 14 us through a retimer plus two

 switches. Source: @markshowa, 2026-10-17

PLX 88096 with a 64 GB BAR1, P2P still failing: four cards behind a PLX 88096; after MMIOH and BAR tuning two of

 them had a 64 GB BAR1, but P2P transfers between them never completed, with Morrowmake's cmpunlocker as well. The root
 cause is unknown. People point to PixelML's 4-card BAR1 P2P over PLX notebook as the reference, and ElectroHobo thinks
 Morrowmake's vLLM fixes it for many people on supported hardware. Details: Risers and bifurcation.
 Source: @replacedriver47, 2026-10-24

 Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @djpandaman, 2026-10-24

 Source: @nathang6866, 2026-10-25

P2P across different root complexes on a Threadripper board (older asm64 fork) looked right but no traffic arrived;

 mrpandagod's advice is to build a local P2P patch for the specific setup.
 Source: @c133, 2026-10-22

Known issues

No release ships P2P and ECC together; the Morrowmake fork drops the ECC patches. A v0.6 request is open.
On boards that cannot route P2P (ACS, no IOMMU passthrough, separate roots), forcing it times out instead of staging

 through system memory, which is why it is opt-in.

Contested Does P2P make inference faster? It depends on host and model; A/B it per recipe. In the order posted:
- ElectroHobo: on a Broadwell host with one PLX switch, MiMo V2.6 Flash TP4 is about 40% slower with P2P off. Source: repo github.com/ElectroHobo/mimo-v26-flash-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-18
- PixelML: on four cards behind two PLX switches on a Broadwell host, P2P helped Qwen3.8-27B TP2 decode (+11%) but hurt GLM 5.3 Flash TP4 (-20 to -26% for one user, -38 to -48% at eight users); they run TP4 with P2P off. Their notes cite Morrowmake measuring +11% from P2P on EPYC root ports. Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
- On an EPYC lab host, vLLM's custom all-reduce over P2P cost GLM 5.3 Flash TP4 about 40% decode but helped Qwen3.8 Flash Next TP2 by 17 to 25%. Source: @darkmaniac7, 2026-10-28 (lab notes)

Status

Mailbox P2P reproduced on the same board by two people (curiouspixel's patch, Trust Inter and GhostInTheDatacenter) and measured on a 4-card rig, so Proven where the topology allows; it is Reported to Contested on PLX and cross-root setups. See the confirmation below.