P2P

GPU-to-GPU P2P (mailbox / BAR1)

Proven 8 GB and 10 GB
Driver	nvidia-open 610.57.04 (measured) / 615.71.09
Unlock build	Morrowmake/cmpunlocker (--p2p) or asm64-hooligan (--p2p)
Steps	P2P
Source	link
Confirmations	Reproduced by @ghostinthedatacenter_87281 (both), 2026-10-07 · details Confirm on your own card

GSP reports PCIe P2P unsupported on the CMP, but the GA100 mailbox P2P hardware is present. P2P matters for tensor parallelism; without it, pipeline parallel is usually the better default on these PCIe-only cards. There are two paths.

Mailbox P2P (works on 64 MB BAR)

The Morrowmake fork's --p2p and asm64's --p2p force the driver to advertise peer access and arm the mailbox trap. This runs in a 512 KB window inside the stock 64 MB BAR1, so it needs no ReBAR and no Above 4G decoding, and works where the board cannot open the 64 GB BAR. Source: @ob88594, 2026-10-05

curiouspixel's P2P patch in the Morrowmake fork fixed P2P for Trust Inter and GhostInTheDatacenter on the same board. Source: @curiouspixel, 2026-10-05 Morrowmake's measured 4-card result: all 12 ordered pairs passed content checks; 16 MiB copies reached about 6.65 GB/s each way (610.57.04, Gen2 x16). asm64's mailbox is about 1.7 GB/s behind one switch at Gen2 x4.

Mechanism (Morrowmake p2p-unlock.patch): it forces pcieP2PReadCaps/WriteCaps OK for 0x20C2/0x2082 only when PRI decode trap 31's PLM 0x0012277c reads 0xFFFFFFFF; around the mailbox setup it arms trap 31 and restores it after. The Booter change adds TRAP31_PLM 0x0012277c = 0xffffffff, opened only when the ForceP2P registry key is set. On a part whose trap PLM read 0xFFFFFF8F the mailbox could not be programmed and a copy silently returned 33 of 33.5 MB wrong (only the first 128 KiB intact), which is why the caps are forced only when the trap is writable and why a content check is required.

Install and the registry line (Morrowmake):

sudo ./install.sh --p2p --profile=8gb --no-iommu --no-passthrough   # --profile=10gb for 10 GB cards
# writes:
# options nvidia NVreg_RegistryDwords="RmForceEnableGen2=1;RMPcieLinkSpeed=0x1;ForceP2P=0x11"

BAR1 P2P (needs the 64 GB BAR)

The other path uses the 64 GB BAR1 (bayley's research). It is a few percent faster than mailbox under pipeline parallelism but needs a board that can open the big BAR. Where the board cannot, use mailbox. Source: @seriousrising, 2026-10-05

ropuls gist (cross-root P2P + serving)

68x reported higher decode/prefill after the extra modprobe parameters from ropuls' gist "2x CMP 170HX for LLM inference: unlock, cross-root P2P, serving (vLLM)":

options nvidia NVreg_RegistryDwords="RmForceEnableGen2=1;RMPcieLinkSpeed=0x1;RMForceStaticBar1=1;RMPcieP2PType=1;RMForceP2PType=1;ForceP2P=0x11"

Warning These are global and force every NVIDIA GPU in the host to Gen2, so do not use them on a mixed rig (e.g. with an RTX 4080). Source: @68x0, 2026-10-13

The gist (gist.github.com/ropuls) covers the unlock, cross-root P2P and vLLM serving. Its key points: P2P can fail because the cmpunlocker 0011 patch sets the P2P override before the PCI device id is populated, so the driver falls back to GSP caps, which report GNS for the CMP regardless of topology. Check with nvidia-smi topo -p2p r; if it says GNS, the ForceP2P=0x11 registry key applies even on a single-switch setup. Measured cross-root P2P latency about 1.5 us (not through a switch). His TP2 on 2x CMP: about 4,900 tok/s prefill at 128k (TTFT 26 s) versus about 2,500 without P2P; decode about 223 tok/s at one session, 527 tok/s aggregate at four (W4A16, MTP4). Source: @ropuls, 2026-10-15

Verify

nvidia-smi topo -p2p r          # OK, not GNS
nvidia-smi topo -m
python3 tools/p2p-content-check.py   # Morrowmake: verifies the bytes actually arrive

nvidia-smi topo and advertised peer support alone do not prove the data moves; always run a content check.

Topology limits

• PLX switches: many boards have MMIO problems behind a PLX, so only mailbox P2P works; in TP mode that

 mailbox P2P can throw XID 13 and crash the inference process. Avoid PLX risers for TP, or disable P2P.
 Source: @ob88594, 2026-10-07

• Dual-socket: P2P does not work across sockets; GPU servers hang all cards off one CPU's root port. Mitigations

 are the mailbox method or bayley's kernel patch. Source: @wakeupmbrc1, 2026-10-18

• Latency reference: about 1.4-1.7 us native (CPU root / crossbar), about 14 us through a retimer plus two

 switches. Source: @markshowa, 2026-10-17

• P2P across different root complexes on a Threadripper board (older asm64 fork) looked right but no traffic arrived;

 mrpandagod's advice is to build a local P2P patch for the specific setup.
 Source: @c133, 2026-10-22

Known issues

• No release ships P2P and ECC together; the Morrowmake fork drops the ECC patches. A v0.6 request is open.
• On boards that cannot route P2P (ACS, no IOMMU passthrough, separate roots), forcing it times out instead of staging

 through system memory, which is why it is opt-in.

Status

Mailbox P2P reproduced on the same board by two people (curiouspixel's patch, Trust Inter and GhostInTheDatacenter) and measured on a 4-card rig, so Proven where the topology allows; it is Reported to Contested on PLX and cross-root setups. See the confirmation below.

Related

cmpunlocker forks, Resizable BAR and BAR1 64 GB, Platforms, Kernel parameters and module options, Troubleshooting, Tools.

Imported from the cmpunlocker Discord with credit; see Credits.