Risers and bifurcation - Revision history

ImportBot: Link width, riser cables, M.2/OcuLink/SlimSAS adapters, PCIe switches (PEX88096, PEX 8747, PEX9797) and bifurcation for CMP 170HX hosts, from Discord (Mountains, cibernox, segmond, fafinet, endsjeremy, domrockt2, markshowa, OB88594, replacedriver47, peter643, Miséricorde and others), PixelML/club-170hx and @darkmaniac7 (Ivan) lab notes; gap-fill from #general 2026-10-24 to 10-25 (P2P failing behind a PLX 88096), @replacedriver47, @djpandaman, @nathang6866

2026-10-30T10:02:35Z

Link width, riser cables, M.2/OcuLink/SlimSAS adapters, PCIe switches (PEX88096, PEX 8747, PEX9797) and bifurcation for CMP 170HX hosts, from Discord (Mountains, cibernox, segmond, fafinet, endsjeremy, domrockt2, markshowa, OB88594, replacedriver47, peter643, Miséricorde and others), PixelML/club-170hx and @darkmaniac7 (Ivan) lab notes; gap-fill from #general 2026-10-24 to 10-25 (P2P failing behind a PLX 88096), @replacedriver47, @djpandaman, @nathang6866

← Older revision		Revision as of 10:02, 30 October 2026
Line 45:		Line 45:
	* '''PEX88096''' (Gen4): one board feeds 5 cards at x16, or 10 at x8. Typical boards take a PCIe x16 host link and use 2x SFF-8654 8i per board. A 6-port version cuts the cost about in half (endsjeremy). {{Source\|handle=gnasherrr\|date=2026-10-17\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561000198086336665}}<br />{{Source\|handle=wakeupmbrc1\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561404369382023203}}<br />{{Source\|handle=endsjeremy\|date=2026-10-21\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562309916864282676}}<br />{{Source\|handle=endsjeremy\|date=2026-10-21\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562309959482478667}}		* '''PEX88096''' (Gen4): one board feeds 5 cards at x16, or 10 at x8. Typical boards take a PCIe x16 host link and use 2x SFF-8654 8i per board. A 6-port version cuts the cost about in half (endsjeremy). {{Source\|handle=gnasherrr\|date=2026-10-17\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561000198086336665}}<br />{{Source\|handle=wakeupmbrc1\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561404369382023203}}<br />{{Source\|handle=endsjeremy\|date=2026-10-21\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562309916864282676}}<br />{{Source\|handle=endsjeremy\|date=2026-10-21\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562309959482478667}}
	* '''PEX88144 / PEX89144''': one board for 8 cards from one root port, but expensive (PEX89144 over 3000 USD) and rare. {{Source\|handle=wakeupmbrc1\|date=2026-10-17\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561000283566514279}}<br />{{Source\|handle=wakeupmbrc1\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561402969126080597}}		* '''PEX88144 / PEX89144''': one board for 8 cards from one root port, but expensive (PEX89144 over 3000 USD) and rare. {{Source\|handle=wakeupmbrc1\|date=2026-10-17\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561000283566514279}}<br />{{Source\|handle=wakeupmbrc1\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561402969126080597}}
	* '''Cascading:''' with two PEX88096 boards in two host slots there are 16 lanes left on each switch, so cascade them so DMA reaches all 8 cards (endsjeremy). {{Source\|handle=endsjeremy\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561403392465707150}}		* '''Gen5 lane counts:''' among Broadcom's Gen5 switches only the PEX89144 and PEX89032 have lane counts divisible by 16 (gnasherrr). DaLZy's guess at the reason: customers wanted to split one x16 into four x16 slots and still attach a NIC or USB controller. {{Source\|handle=gnasherrr\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561408742438867125}}<br />{{Source\|handle=dalzyyy\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561409384968491009}}
			* '''Cascading:''' with two PEX88096 boards in two host slots there are 16 lanes left on each switch, so cascade them so DMA reaches all 8 cards (endsjeremy). The alternative he named for 8 cards is a single 10-port switch instead of two. {{Source\|handle=endsjeremy\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561403167936352439}}<br />{{Source\|handle=endsjeremy\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561403392465707150}}
	* '''3x Gen4 x16 switch cards''' exist and cost less than the 5x x16 versions. {{Source\|handle=gpmickey\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557481786802573433}}		* '''3x Gen4 x16 switch cards''' exist and cost less than the 5x x16 versions. {{Source\|handle=gpmickey\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557481786802573433}}
	* A Broadcom redriver/retimer board is the other option GlobalApathy knows for adding slots. {{Source\|handle=globalapathy\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557486128154550444}}		* A Broadcom redriver/retimer board is the other option GlobalApathy knows for adding slots. {{Source\|handle=globalapathy\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557486128154550444}}
Line 57:		Line 58:
	* '''MMIO behind switches:''' OB88594 says many boards show MMIO problems behind a PLX switch, so only mailbox P2P is usable there, and in tensor parallel that mailbox P2P crashes the inference program with Xid 13. {{Source\|handle=ob88594\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557383845844615241}} Some PLX machines have too little MMIO for the 64 GB BAR, and the Morrowmake P2P mode runs with a 64 MB BAR. {{Source\|handle=ob88594\|date=2026-10-05\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556527198540922981}}		* '''MMIO behind switches:''' OB88594 says many boards show MMIO problems behind a PLX switch, so only mailbox P2P is usable there, and in tensor parallel that mailbox P2P crashes the inference program with Xid 13. {{Source\|handle=ob88594\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557383845844615241}} Some PLX machines have too little MMIO for the 64 GB BAR, and the Morrowmake P2P mode runs with a 64 MB BAR. {{Source\|handle=ob88594\|date=2026-10-05\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556527198540922981}}
	* '''Switch firmware:''' replacedriver47 found his PEX88096 firmware lacked hotplug, which he says is required to remap BAR addressing behind the switch. {{Source\|handle=replacedriver47\|date=2026-10-24\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1563580726882861097}}		* '''Switch firmware:''' replacedriver47 found his PEX88096 firmware lacked hotplug, which he says is required to remap BAR addressing behind the switch. {{Source\|handle=replacedriver47\|date=2026-10-24\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1563580726882861097}}
			* {{Status\|Reported}} '''P2P never completes behind a PLX 88096, even at 64 GB BAR1.''' replacedriver47 has four cards behind a PLX 88096 switch. By tuning MMIOH and BAR sizes he got two of them to a 64 GB BAR1 (the MMIOH window is too small for all four at 64 GB), but even between those two, P2P transfers never complete. He had heard Morrowmake's cmpunlocker makes P2P work; it did not for him, and he wants the root cause before he enlarges MMIOH further. mrpandagod pointed him to PixelML's [https://github.com/PixelML/club-170hx/blob/main/notebooks/2026-10-01-cmp170hx-4card-bar1-p2p-plx-cuda.ipynb notebook on 4-card BAR1 P2P over PLX with CUDA]. ElectroHobo had P2P working with an older method and thinks Morrowmake's vLLM fixes it for many people on supported hardware. The root cause is unresolved. See [[P2P]] and [[Minisforum MS-01]]. {{Source\|handle=replacedriver47\|date=2026-10-24\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563629249502515205}}<br />{{Source\|handle=replacedriver47\|date=2026-10-24\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563629333552177284}}<br />{{Source\|handle=replacedriver47\|date=2026-10-24\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563629405589475550}}<br />{{Source\|handle=replacedriver47\|date=2026-10-24\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563629971082186753}}<br />{{Source\|handle=replacedriver47\|date=2026-10-24\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563633438450393259}}<br />{{Source\|handle=djpandaman\|date=2026-10-24\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563629622673936384\|repo=https://github.com/PixelML/club-170hx}}<br />{{Source\|handle=nathang6866\|date=2026-10-25\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1564023832107417662}}
	* '''Per-slot bifurcation on a PEX88096''' needs a custom SBR (switch firmware). The seller alifond reprogrammed the last slot of peter643's board to 4x x4 before shipping but would not share the image or other custom firmware, citing an NDA; peter643 later worked out how to build his own; an open hardware project page has the procedure and files, and peter643 believes only about 8 bytes change (4 setting bytes and 4 CRC bytes). {{Source\|handle=peter643\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557518262659711000}}<br />{{Source\|handle=peter643\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557534181284913273}}<br />{{Source\|handle=peter643\|date=2026-10-08\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557765698011402402}}<br />{{Source\|handle=peter643\|date=2026-10-08\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557792050739417210}} He prefers the 88096's own port bifurcation over a second PEX switch, which costs power and about 300 USD but could give 8x x4 NVMe for more random IOPS. {{Source\|handle=peter643\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557535072008151174}}		* '''Per-slot bifurcation on a PEX88096''' needs a custom SBR (switch firmware). The seller alifond reprogrammed the last slot of peter643's board to 4x x4 before shipping but would not share the image or other custom firmware, citing an NDA; peter643 later worked out how to build his own; an open hardware project page has the procedure and files, and peter643 believes only about 8 bytes change (4 setting bytes and 4 CRC bytes). {{Source\|handle=peter643\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557518262659711000}}<br />{{Source\|handle=peter643\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557534181284913273}}<br />{{Source\|handle=peter643\|date=2026-10-08\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557765698011402402}}<br />{{Source\|handle=peter643\|date=2026-10-08\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557792050739417210}} He prefers the 88096's own port bifurcation over a second PEX switch, which costs power and about 300 USD but could give 8x x4 NVMe for more random IOPS. {{Source\|handle=peter643\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557535072008151174}}
	* '''Five slots, four cards:''' cibernox's switch takes 5 cards; he stops at 4 (256 GB) because he does not expect TP to scale well past 4 cards and 320 GB opens few extra models; Sojun agrees 256 GB is a reasonable stopping point. {{Status\|Theory}} {{Source\|handle=cibernox\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557428289365082245}}<br />{{Source\|handle=sojun80\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557430622245359738}}		* '''Five slots, four cards:''' cibernox's switch takes 5 cards; he stops at 4 (256 GB) because he does not expect TP to scale well past 4 cards and 320 GB opens few extra models; Sojun agrees 256 GB is a reasonable stopping point. {{Status\|Theory}} {{Source\|handle=cibernox\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557428289365082245}}<br />{{Source\|handle=sojun80\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557430622245359738}}

ImportBot: Link width, riser cables, M.2/OcuLink/SlimSAS adapters, PCIe switches (PEX88096, PEX 8747, PEX9797) and bifurcation for CMP 170HX hosts, from Discord (Mountains, cibernox, segmond, fafinet, endsjeremy, domrockt2, markshowa, OB88594, replacedriver47, peter643, Miséricorde and others), PixelML/club-170hx and @darkmaniac7 (Ivan) lab notes

2026-10-30T08:47:26Z

ImportBot: Link width, riser cables, M.2/OcuLink/SlimSAS adapters, PCIe switches (PEX88096, PEX 8747, PEX9797) and bifurcation for CMP 170HX hosts, from Discord (Mountains, cibernox, segmond, fafinet, endsjeremy, domrockt2, markshowa, OB88594, replacedriver47, peter643, Miséricorde and others), PixelML/club-170hx and @darkmaniac7 (Ivan) lab notes

2026-10-30T08:11:08Z

← Older revision		Revision as of 08:11, 30 October 2026
Line 64:		Line 64:

	=== Topology rules of thumb ===		=== Topology rules of thumb ===
	* '''~~All P2P lanes~~ from one CPU.''' Miséricorde: you need all PCIe lanes from one CPU or P2P will not work properly. {{Source\|handle=wakeupmbrc1\|date=2026-10-16\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560650062474772653}} On dual-socket ~~hosts~~, ~~whether~~ P2P ~~works~~ across sockets ~~is disputed:~~ {{Status\|~~Contested~~}}		* '''Lanes from one CPU.''' Miséricorde: you need all PCIe lanes from one CPU or P2P will not work properly. {{Source\|handle=wakeupmbrc1\|date=2026-10-16\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560650062474772653}} That holds for many boards, but it is not a hard rule: see the next item.
	** Miséricorde: P2P does not work across sockets, which is why GPU servers hang all GPUs off one CPU. {{Source\|handle=wakeupmbrc1\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561407364622458943}}		* '''P2P across sockets works on a dual-socket EPYC server.''' On an 8-card Supermicro H12DSG-O-CPU (dual EPYC Rome, cards on both CPUs, Gen2 x16, 64 GB BAR1, Morrowmake P2P build), <code><nowiki>p2p-content-check.py</nowiki></code> passed all 504 checks on 8 devices (every ordered pair at every transfer size, including the pairs across the two sockets), and <code><nowiki>cudaDeviceCanAccessPeer</nowiki></code> returned 1 for every pair. {{Status\|Proven}} {{Source\|handle=darkmaniac7\|date=2026-10-29\|note=lab notes}}
	** xanthe_~~: there are methods for it on the table,~~ the mailbox method or bayley's Linux kernel patch. {{Source\|handle=xanthefin\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561407943985864705}}		** Conflicting earlier source: Miséricorde said P2P does not work across sockets, which is why GPU servers hang all GPUs off one CPU. {{Source\|handle=wakeupmbrc1\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561407364622458943}}
	** programmerboi: QPI/NUMA overhead is relatively low ~~in his experience~~. {{Source\|handle=programmerboi\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561408307225428141}}		** Other earlier posts: xanthe_ named the mailbox method and bayley's Linux kernel patch as ways to get P2P {{Source\|handle=xanthefin\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561407943985864705}}; programmerboi finds QPI/NUMA overhead relatively low. {{Source\|handle=programmerboi\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561408307225428141}}
	** One lab data point: on a dual-socket Supermicro H12DSG-O-CPU, <code><nowiki>cudaDeviceCanAccessPeer</nowiki></code> returned 1 for a pair across sockets with the Morrowmake P2P build; the full byte-check matrix was not captured in the notes. {{Source\|handle=darkmaniac7\|date=2026-10-27\|note=lab notes}}
	* '''Keep a TP group behind one switch.''' On servers with two switches of five slots each, put the cards of one group behind the same switch (consecutive slots). {{Source\|handle=nathang6866\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1561967586978177035}} On a Gigabyte G292-Z20 each pair sits on its own switch: fast P2P inside a pair, slower between pairs. {{Source\|handle=cmdrwaka324\|date=2026-10-26\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1564120367398780969}}		* '''Keep a TP group behind one switch.''' On servers with two switches of five slots each, put the cards of one group behind the same switch (consecutive slots). {{Source\|handle=nathang6866\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1561967586978177035}} On a Gigabyte G292-Z20 each pair sits on its own switch: fast P2P inside a pair, slower between pairs. {{Source\|handle=cmdrwaka324\|date=2026-10-26\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1564120367398780969}}
	* '''Hops:''' gnasherrr recalls PLX latency at about 105 ns. {{Source\|handle=gnasherrr\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561398366007463959}} endsjeremy: tensor parallel across the array is bandwidth limited whatever the lane count, so x8 per card is enough, and switches shine for PP and P2P DMA. {{Source\|handle=endsjeremy\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561396033106215105}}<br />{{Source\|handle=endsjeremy\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561397344296247307}}<br />{{Source\|handle=endsjeremy\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561401282399436973}}		* '''Hops:''' gnasherrr recalls PLX latency at about 105 ns. {{Source\|handle=gnasherrr\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561398366007463959}} endsjeremy: tensor parallel across the array is bandwidth limited whatever the lane count, so x8 per card is enough, and switches shine for PP and P2P DMA. {{Source\|handle=endsjeremy\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561396033106215105}}<br />{{Source\|handle=endsjeremy\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561397344296247307}}<br />{{Source\|handle=endsjeremy\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561401282399436973}}

ImportBot: Link width, riser cables, M.2/OcuLink/SlimSAS adapters, PCIe switches (PEX88096, PEX 8747, PEX9797) and bifurcation for CMP 170HX hosts, from Discord (Mountains, cibernox, segmond, fafinet, endsjeremy, domrockt2, markshowa, OB88594, replacedriver47, peter643, Miséricorde and others), PixelML/club-170hx and @darkmaniac7 (Ivan) lab notes

2026-10-30T08:05:01Z

New page

{{Status|Reported}} {{Cards|both}}

How cards get connected when the board has too few slots, the wrong slot widths, or slots too close together: riser cables, M.2 and OcuLink adapters, SlimSAS cabling, PCIe switch boards and slot bifurcation. For BIOS settings see [[Server BIOS setup]]; for boards see [[Platforms]] and [[Consumer motherboards]].

== Link width: what the card can use ==
* '''An unmodified card links at x4.''' The card's lanes 4 to 15 have no AC-coupling capacitors fitted, so it trains x4 until the [[PCIe x16 capacitor mod]] is done; the stock link is Gen1 x4 (about 0.85 GB/s). {{Source|handle=djpandaman|date=2026-10-24|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563629622673936384|repo=https://github.com/PixelML/club-170hx}} (docs/CMP170HX-UNDERSTANDING.md, docs/RESEARCH-CMP170HX-UNLOCKS.md)
* '''Speed:''' the unlock brings the link to Gen2. Gen2 is the ceiling on 8 GB cards; Gen3 on 10 GB cards needs an RIR burn. See [[Gen2 link retrain]] and [[Unlocks]].
* '''Measured:''' about 6.65 GB/s each way between two cards at Gen2 x16 (Morrowmake's README); 6.68 GB/s device-to-host per card at Gen2 x16 behind a PLX switch (PixelML). An unmodified card in a PCIe 3.0 x1 slot measured about 0.38 GB/s at Gen2 x1 (anoane). {{Source|handle=darkmaniac7|date=2026-10-29|note=lab notes}} {{Source|handle=djpandaman|date=2026-10-24|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563629622673936384|repo=https://github.com/PixelML/club-170hx}}; {{Source|handle=logy0|date=2026-10-10|link=https://discord.com/channels/1518634379604136178/1540599965007478854/1558505023187714068|repo=https://github.com/anoane/01-proxmox-dual-gpu-passthrough}}
* '''A slot that drops from x16 to x8''' after the cap mod can be a bad cap job (Waka324 saw it on his early cards) or a damaged slot (matstatman blames his used X99 board). {{Source|handle=matstatman|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560618524995489825}} {{Source|handle=cmdrwaka324|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560640558685233254}}

=== Does width matter? ===
* '''Pipeline parallel runs fine on narrow links.''' Mountains runs four cards at Gen2 x4 as PP for GLM 5.3 Flash at 130 to 160 tok/s single stream, and says he gets similar speeds to x16 setups, maybe 10% less single stream. {{Source|handle=mountains1605|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556409589124304957}} {{Source|handle=mountains1605|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1561959462024118353}} {{Source|handle=mountains1605|date=2026-10-23|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562982101157150771}}
* '''Tensor parallel and P2P need width.''' segmond tried Morrowmake's GLM recipe at x4 without P2P and was not close to the published numbers; cibernox: "the 16x mod is doing a lot for the high speed". {{Source|handle=segmond|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1561950054254645291}} {{Source|handle=segmond|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562091215891996793}} {{Source|handle=cibernox|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562107263009689731}} Two cards at x4 running GLM 5.3 were "brutally slow" for fallengod9962. {{Source|handle=fallengod9962|date=2026-10-23|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1563015573489586318}}
* '''Training with offload''' is bottlenecked by the link: orrorin watched PCIe 2.0 x4 saturate during offload while doing SFT on four cards. {{Source|handle=_orrorin|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560681450028929118}} {{Source|handle=_orrorin|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560681653788217435}}
* mrpandagod: Gen2 x16 is about as fast as Thunderbolt 5 between Mac nodes; fine for pipeline parallel, too slow for tensor parallel. {{Source|handle=djpandaman|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557238375755546705}}

Because an unmodified card only uses four lanes, an x4 slot, an x4 bifurcation slice or an M.2 adapter costs it nothing until you do the cap mod. Plan the x16 paths if you intend to mod the cards.

== Riser cables ==
* '''Passive risers can hide the card from the BMC.''' fafinet's four CMPs in motherboard slots report temperatures to the BMC and drive its fan curve; the fifth, on a passive eBay riser (the only kind that fit under a card on his SSI CEB board), is seen as an add-in card, not a GPU, so the BMC cannot read its temperature. Different port and different cards gave the same result. He sets fan curves from the OS instead. {{Source|handle=fafinet|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562285217731973171}} {{Source|handle=fafinet|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562300430439157850}} {{Source|handle=fafinet|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562308658975739936}}
* endsjeremy: buy risers from a reputable brand (he names ADT-Link as known good but expensive); passive risers bring "inexplicable gremlins", though a PCIe 3.0 riser should not be hard to get working well. {{Source|handle=endsjeremy|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562311490470547517}}
* '''A failed Gen2 retrain on a riser''' shows as LnkSta 2.5 GT/s "(downgraded)": that points at the signal path (riser, extension cable or slot), not settings. Move the card to test. {{Source|handle=darkmaniac7|date=2026-10-29|note=lab notes}}
* An x16 riser cable in a PCIe 4.0 x24 slot works (port00, Hyve Edge Metal G10 servers). {{Source|handle=port00|date=2026-10-23|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1563212974896713801}} See [[Hyve Edge Metal G10]].
* A card with a water block and AIO radiator can be fitted with a riser cable and a bigger case, or with the radiator left outside the case, which is what PJ did. {{Source|handle=robbbbbb7973|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1561267848729591880}} {{Source|handle=boz._.|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1561341400308645929}} {{Source|handle=zingy90210|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1561348651786305590}}

== M.2, OcuLink and SlimSAS ==
* '''M.2 to PCIe x16 extender cables''' add a card to a consumer board from a free M.2 slot: matstatman suggested an M.2 NVMe to PCIe 3.0 x16 extender (SinLoon); AnalyticETH runs "a bunch of these". {{Source|handle=matstatman|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557484331193077841}} {{Source|handle=analyticeth|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557484927375769726}}
* '''OcuLink from M.2:''' GlobalApathy ran a card over OcuLink from an M.2 adapter on a Dell micro PC into an external GPU frame; it worked but was "a bit flaky". {{Source|handle=globalapathy|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561376002280661025}} gnasherrr runs four cards from a 1-to-4 OcuLink setup while his switch boards ship. {{Source|handle=gnasherrr|date=2026-10-25|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1563736510082064516}} See [[Dell OptiPlex 7040m]].
* '''SlimSAS 8i (SFF-8654)''' cables with PCIe x16 adapters are a common way to hang cards off server boards and switch boards; PCIe x16 to 2x SlimSAS 8i host adapters cost about the same in the US and EU as in China. {{Source|handle=colo2442|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557026698015277067}} {{Source|handle=kellertreppe123|date=2026-10-24|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1563568589011947524}} Mountains extended his cards above a regular case on a 2020-extrusion frame with external SlimSAS. {{Source|handle=mountains1605|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562324256296149026}}
* '''Retimed adapters:''' atsasin's 39com SlimSAS adapter boards have proper retimers and show no signal degradation at PCIe Gen3. {{Source|handle=atsasin2126|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1561130214560374887}}
* Connector names get mixed up: the 8i ports on the Tyan Tomcat board in the Hyve servers are OcuLink, not SlimSAS (endsjeremy corrected himself). {{Source|handle=endsjeremy|date=2026-10-23|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1563281929783087245}}

== Bifurcation ==
* '''Passive x4x4x4x4 adapters''' are cheap (about 20 USD) but need bifurcation support in the board's BIOS. {{Source|handle=ichundes|date=2026-10-10|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1558461209710301246}} {{Source|handle=xeax|date=2026-10-10|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1558465232597291019}}
* Consumer boards often bifurcate only 4x x4: dottoreargus' X870 board with one Gen5 x16 slot. {{Source|handle=dottoreargus|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557480585134350349}}
* '''x8/x8 server slots:''' slots bifurcated x8/x8 so every card sits behind its own root port worked for four cards under ESXi (Gen2 x8 each). {{Source|handle=darkmaniac7|date=2026-10-21|note=lab notes}}
* '''WRX90 BIOS mod:''' [https://github.com/NA-DEGEN-GIRL/wrx90-bios-mod NA-DEGEN-GIRL/wrx90-bios-mod] adds an x8/x8 bifurcation option to the ASUS Pro WS WRX90E-SAGE SE on BIOS 1317. Only a boot to Setup with the option visible has been observed, on one board; x8/x8 link negotiation, P2P and stability are not verified. {{Source|handle=stcord|date=2026-10-25|link=https://discord.com/channels/1518634379604136178/1518691940646781070/1563898614982643762|repo=https://github.com/NA-DEGEN-GIRL/wrx90-bios-mod}} {{Status|Theory}} See [[ASUS Pro WS WRX90E-SAGE SE]].
* '''NVMe on switch boards:''' cheap NVMe carriers only work with bifurcation; real NVMe RAID cards such as the HighPoint SSD7749M2 are expensive (cibernox, kawaii). peter643 considered a 300 USD 8x NVMe switch card for a board without bifurcation before going the switch-firmware route. {{Source|handle=kawaii8789|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557675372269674538}} {{Source|handle=cibernox|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557724476244885659}} {{Source|handle=peter643|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557736740742238381}}
* Gigabyte MZ32-AR0 splits one PCIe x16 into SAS connectors. {{Source|handle=wakeupmbrc1|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560650062474772653}}

== PCIe switch boards ==
Switch boards (PLX/Broadcom PEX) let one host slot feed several cards and let cards talk to each other without crossing the CPU. They are the usual way to run 4 to 10 cards from a consumer board.

=== Which switch ===
* '''PEX88096''' (Gen4): one board feeds 5 cards at x16, or 10 at x8. Typical boards take a PCIe x16 host link and use 2x SFF-8654 8i per board. A 6-port version cuts the cost about in half (endsjeremy). {{Source|handle=gnasherrr|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561000198086336665}} {{Source|handle=wakeupmbrc1|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561404369382023203}} {{Source|handle=endsjeremy|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562309916864282676}} {{Source|handle=endsjeremy|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562309959482478667}}
* '''PEX88144 / PEX89144''': one board for 8 cards from one root port, but expensive (PEX89144 over 3000 USD) and rare. {{Source|handle=wakeupmbrc1|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561000283566514279}} {{Source|handle=wakeupmbrc1|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561402969126080597}}
* '''Cascading:''' with two PEX88096 boards in two host slots there are 16 lanes left on each switch, so cascade them so DMA reaches all 8 cards (endsjeremy). {{Source|handle=endsjeremy|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561403392465707150}}
* '''3x Gen4 x16 switch cards''' exist and cost less than the 5x x16 versions. {{Source|handle=gpmickey|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557481786802573433}}
* A Broadcom redriver/retimer board is the other option GlobalApathy knows for adding slots. {{Source|handle=globalapathy|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557486128154550444}}
* '''Older switches in GPU servers:''' PEX9797 (PCIe 3.0) in the Supermicro SYS-4028GR-TRT2; PEX 8747 in the PixelML club's 4-card host. See [[Supermicro SYS-4028GR-TRT2]].

=== What people found ===
* '''P2P works on a PEX88096 board''' from Alibaba ("5 GPU Expansion"): domrockt2 got P2P once he fixed a SAS cable in the wrong port, which had left the host link at x8. In hindsight he would buy one with more slots. {{Source|handle=domrockt2|date=2026-10-15|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1560201364552941631}} He runs one on a Z690 with an i5-14600K and recommends a fan on the PLX chip. {{Source|handle=domrockt2|date=2026-10-22|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1562908280014110731}} {{Source|handle=domrockt2|date=2026-10-22|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1562913694437081209}}
* '''Static BAR1 P2P on a PEX 8747 host:''' the PixelML club got 12/12 ordered pairs byte-exact at 5.79 GB/s with four extra patches and a pre-driver BAR placement step adopted via kexec; latency 16 us (4 KiB) and 194 us (1 MiB) vs 22 and 343 us host-staged. P2P helped Qwen3.8-27B TP2 (+11%) but hurt GLM-5.3-Flash TP4 on that Broadwell host (-20 to -26% one user), the opposite of Morrowmake's EPYC result (+11%). {{Source|handle=djpandaman|date=2026-10-24|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563629622673936384|repo=https://github.com/PixelML/club-170hx}} {{Status|Contested}}
* '''Two cards behind one switch''' corrupted the kernel resource tree on every boot until the per-card BAR1 resize was serialized (<code><nowiki>rebar-serialize.patch</nowiki></code>, same fix as cmpunlocker PR #59). {{Source|handle=djpandaman|date=2026-10-24|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563629622673936384|repo=https://github.com/PixelML/club-170hx}}
* '''MMIO behind switches:''' OB88594 says many boards show MMIO problems behind a PLX switch, so only mailbox P2P is usable there, and in tensor parallel that mailbox P2P crashes the inference program with Xid 13. {{Source|handle=ob88594|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557383845844615241}} Some PLX machines have too little MMIO for the 64 GB BAR, and the Morrowmake P2P mode runs with a 64 MB BAR. {{Source|handle=ob88594|date=2026-10-05|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556527198540922981}}
* '''Switch firmware:''' replacedriver47 found his PEX88096 firmware lacked hotplug, which he says is required to remap BAR addressing behind the switch. {{Source|handle=replacedriver47|date=2026-10-24|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1563580726882861097}}
* '''Per-slot bifurcation on a PEX88096''' needs a custom SBR (switch firmware). A seller preloaded a 4x x4 slot for peter643 but would not share the image; an open hardware project page has the procedure and files, and peter643 believes only about 8 bytes change (4 setting bytes and 4 CRC bytes). {{Source|handle=peter643|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557534181284913273}} {{Source|handle=peter643|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557765698011402402}} {{Source|handle=peter643|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557792050739417210}}
* '''Five slots, four cards:''' cibernox's switch takes 5 cards; he stops at 4 (256 GB) because he does not expect TP to scale well past 4 cards and 320 GB opens few extra models; Sojun agrees 256 GB is a reasonable stopping point. {{Status|Theory}} {{Source|handle=cibernox|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557428289365082245}} {{Source|handle=sojun80|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1557430622245359738}}
* '''GPUDirect Storage across switches:''' markshowa (2x PEX9797 plus a PLX88048 carrier) finds GDS works and does better inside a switch, but writes across switches are very slow and look serialized; he suspects the same mechanism behind his poor cross-switch P2P. {{Source|handle=markshowa|date=2026-10-22|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1562923128500392051}}
* {{Status|Warning}} '''Power:''' a rig with four cards on one PLX board and two independent PSUs lost all four cards and the PLX board at once under large prompt-processing loads. See [[Cooling and power]]. {{Source|handle=replacedriver47|date=2026-10-05|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556768826769674351}}
* '''Cards dropping off a switch baseboard under load:''' Moliminous' 5 cards on an external GPU switch baseboard (beside 6 RTX 3090 on Ubuntu LTS) were stable alone but fell off the bus with PCIe lane and IOMMU errors on large prompts. Temperatures and PSU draw looked fine, though the logs showed the first card overdrawing to 260 W; lowering the power limit from 250 W to 180 W fixed it, and GLM-5.3-Flash now runs at 380k context. {{Source|handle=moliminous|date=2026-10-19|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1561749293717659774}} {{Source|handle=moliminous|date=2026-10-19|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1561768584869322753}} {{Source|handle=moliminous|date=2026-10-19|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1561769135057149953}} {{Source|handle=gnasherrr|date=2026-10-19|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1561777452521623654}} See [[Cooling and power]].
* {{Status|Warning}} A PLX board from one Chinese supplier arrived defective twice (shorts as soon as the CPU power connectors are plugged in). {{Source|handle=lubakmanqk|date=2026-10-22|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562912120121725050}} {{Source|handle=lubakmanqk|date=2026-10-23|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1563122360251457646}}

=== Topology rules of thumb ===
* '''All P2P lanes from one CPU.''' Miséricorde: you need all PCIe lanes from one CPU or P2P will not work properly. {{Source|handle=wakeupmbrc1|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560650062474772653}} On dual-socket hosts, whether P2P works across sockets is disputed: {{Status|Contested}}
** Miséricorde: P2P does not work across sockets, which is why GPU servers hang all GPUs off one CPU. {{Source|handle=wakeupmbrc1|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561407364622458943}}
** xanthe_: there are methods for it on the table, the mailbox method or bayley's Linux kernel patch. {{Source|handle=xanthefin|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561407943985864705}}
** programmerboi: QPI/NUMA overhead is relatively low in his experience. {{Source|handle=programmerboi|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561408307225428141}}
** One lab data point: on a dual-socket Supermicro H12DSG-O-CPU, <code><nowiki>cudaDeviceCanAccessPeer</nowiki></code> returned 1 for a pair across sockets with the Morrowmake P2P build; the full byte-check matrix was not captured in the notes. {{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}
* '''Keep a TP group behind one switch.''' On servers with two switches of five slots each, put the cards of one group behind the same switch (consecutive slots). {{Source|handle=nathang6866|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1561967586978177035}} On a Gigabyte G292-Z20 each pair sits on its own switch: fast P2P inside a pair, slower between pairs. {{Source|handle=cmdrwaka324|date=2026-10-26|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1564120367398780969}}
* '''Hops:''' gnasherrr recalls PLX latency at about 105 ns. {{Source|handle=gnasherrr|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561398366007463959}} endsjeremy: tensor parallel across the array is bandwidth limited whatever the lane count, so x8 per card is enough, and switches shine for PP and P2P DMA. {{Source|handle=endsjeremy|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561396033106215105}} {{Source|handle=endsjeremy|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561397344296247307}} {{Source|handle=endsjeremy|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561401282399436973}}
* '''P2P latency reference:''' about 1.7 us over the CPU root, about 1.4 us on a crossbar (markshowa); about 1.5 us per hardware call between CMPs on native Gen2 x16 slots (Miséricorde); switches and retimers add latency. {{Source|handle=markshowa|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561013734808363029}} {{Source|handle=wakeupmbrc1|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561014804288770152}}
* '''10 cards:''' zorg33 asked whether 10x x8 on one PEX88096 or 5+5 x16 on two switches gives better TP decode; untested. {{Source|handle=zorg33|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556942685816889397}} djpandaman chose 10x x8 on one switch and will add a second if bandwidth bottlenecks. {{Source|handle=djpandaman|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557988514195570768}} endsjeremy says jumping between two 5-card arrays "murders performance". {{Status|Theory}} {{Source|handle=endsjeremy|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557998376115511377}}

== Expansion boards and external chassis ==
* fnerdman is considering moving four cards off an EPYC host onto a consumer CPU with an Alibaba "PCIe 4.0 x16 to ..." expansion board that appears to daisy-chain. Open questions he raised, unanswered: what its three 6-pin power inputs are, and whether cards come back stable if the board's PSU is cut and restored while the host stays up. {{Status|Theory}} {{Source|handle=fnerdman|date=2026-10-26|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1564244395887435847}}

== Fitting cards on the board ==
* '''Front-panel header:''' matstatman needed about 10 mm of clearance to fit a card over the front-panel pin header. Options: angled Dupont connectors (too tall for his case), pin-header risers (Mickey used them), or enabling Wake-on-LAN and "power on after power loss" and leaving the header unplugged (Sojun). {{Source|handle=matstatman|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557467080104738938}} {{Source|handle=matstatman|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557473010406137947}} {{Source|handle=gpmickey|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557476304436068516}} {{Source|handle=sojun80|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557484238914322504}}
* '''Fan headers:''' alanoo: on ASRock Rack Rome boards the cards collide with 4 of the 7 fan headers. fafinet's cards also touch the fan headers on his board and do not seat fully, but still work. {{Source|handle=alanoo|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1562450191464800266}} {{Source|handle=fafinet|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1562451857878818887}} {{Source|handle=fafinet|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1562458858482307105}}

== Related ==
[[Platforms]], [[Consumer motherboards]], [[Server BIOS setup]], [[PCIe x16 capacitor mod]], [[Gen2 link retrain]], [[Resizable BAR and BAR1 64 GB]], [[P2P]], [[Cooling and power]], [[Power and cabling warnings]], [[Troubleshooting]].

{{ImportFooter}}
[[Category:Platforms]]
[[Category:8 GB]]
[[Category:10 GB]]