Risers and bifurcation

Reported 8 GB and 10 GB

How cards get connected when the board has too few slots, the wrong slot widths, or slots too close together: riser cables, M.2 and OcuLink adapters, SlimSAS cabling, PCIe switch boards and slot bifurcation. For BIOS settings see Server BIOS setup; for boards see Platforms and Consumer motherboards.

Link width: what the card can use

An unmodified card links at x4. The card's lanes 4 to 15 have no AC-coupling capacitors fitted, so it trains x4 until the PCIe x16 capacitor mod is done; the stock link is Gen1 x4 (about 0.85 GB/s). Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @djpandaman, 2026-10-24 (docs/CMP170HX-UNDERSTANDING.md, docs/RESEARCH-CMP170HX-UNLOCKS.md)
Speed: the unlock brings the link to Gen2. Gen2 is the ceiling on 8 GB cards; Gen3 on 10 GB cards needs an RIR burn. See Gen2 link retrain and Unlocks.
Measured: about 6.65 GB/s each way between two cards at Gen2 x16 (Morrowmake's README); 6.68 GB/s device-to-host per card at Gen2 x16 behind a PLX switch (PixelML). An unmodified card in a PCIe 3.0 x1 slot measured about 0.38 GB/s at Gen2 x1 (anoane). Source: @darkmaniac7, 2026-10-29 (lab notes) Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @djpandaman, 2026-10-24; Source: repo github.com/anoane/01-proxmox-dual-gpu-passthrough by anoane on GitHub, posted by @logy0, 2026-10-10
A slot that drops from x16 to x8 after the cap mod can be a bad cap job (Waka324 saw it on his early cards) or a damaged slot (matstatman blames his used X99 board). Source: @matstatman, 2026-10-16
Source: @cmdrwaka324, 2026-10-16

Does width matter?

Pipeline parallel runs fine on narrow links. Mountains runs four cards at Gen2 x4 as PP for GLM 5.3 Flash at 130 to 160 tok/s single stream, and says he gets similar speeds to x16 setups, maybe 10% less single stream. Source: @mountains1605, 2026-10-04
Source: @mountains1605, 2026-10-20
Source: @mountains1605, 2026-10-23
Tensor parallel and P2P need width. segmond tried Morrowmake's GLM recipe at x4 without P2P and was not close to the published numbers; cibernox: "the 16x mod is doing a lot for the high speed". Source: @segmond, 2026-10-20
Source: @segmond, 2026-10-20
Source: @cibernox, 2026-10-20 Two cards at x4 running GLM 5.3 were "brutally slow" for fallengod9962. Source: @fallengod9962, 2026-10-23
Training with offload is bottlenecked by the link: orrorin watched PCIe 2.0 x4 saturate during offload while doing SFT on four cards. Source: @_orrorin, 2026-10-16
Source: @_orrorin, 2026-10-16
mrpandagod: Gen2 x16 is about as fast as Thunderbolt 5 between Mac nodes; fine for pipeline parallel, too slow for tensor parallel. Source: @djpandaman, 2026-10-07

Because an unmodified card only uses four lanes, an x4 slot, an x4 bifurcation slice or an M.2 adapter costs it nothing until you do the cap mod. Plan the x16 paths if you intend to mod the cards.

Riser cables

Passive risers can hide the card from the BMC. fafinet's four CMPs in motherboard slots report temperatures to the BMC and drive its fan curve; the fifth, on a passive eBay riser (the only kind that fit under a card on his SSI CEB board), is seen as an add-in card, not a GPU, so the BMC cannot read its temperature. Different port and different cards gave the same result. He sets fan curves from the OS instead. Source: @fafinet, 2026-10-21
Source: @fafinet, 2026-10-21
Source: @fafinet, 2026-10-21
endsjeremy: buy risers from a reputable brand (he names ADT-Link as known good but expensive); passive risers bring "inexplicable gremlins", though a PCIe 3.0 riser should not be hard to get working well. Source: @endsjeremy, 2026-10-21
A failed Gen2 retrain on a riser shows as LnkSta 2.5 GT/s "(downgraded)": that points at the signal path (riser, extension cable or slot), not settings. Move the card to test. Source: @darkmaniac7, 2026-10-29 (lab notes)
An x16 riser cable in a PCIe 4.0 x24 slot works (port00, Hyve Edge Metal G10 servers). Source: @port00, 2026-10-23 See Hyve Edge Metal G10.
A card with a water block and AIO radiator can be fitted with a riser cable and a bigger case, or with the radiator left outside the case, which is what PJ did. Source: @robbbbbb7973, 2026-10-18
Source: @boz._., 2026-10-18
Source: @zingy90210, 2026-10-18

M.2, OcuLink and SlimSAS

M.2 to PCIe x16 extender cables add a card to a consumer board from a free M.2 slot: matstatman suggested an M.2 NVMe to PCIe 3.0 x16 extender (SinLoon, Amazon B0DQT91C9N); AnalyticETH runs "a bunch of these". Source: @matstatman, 2026-10-07
Source: @analyticeth, 2026-10-07
OcuLink from M.2: GlobalApathy ran a card over OcuLink from an M.2 adapter on a Dell micro PC into an external GPU frame; it worked but was "a bit flaky". Source: @globalapathy, 2026-10-18 gnasherrr runs four cards from a 1-to-4 OcuLink setup while his switch boards ship. Source: @gnasherrr, 2026-10-25 See Dell OptiPlex 7040m.
SlimSAS 8i (SFF-8654) cables with PCIe x16 adapters are a common way to hang cards off server boards and switch boards; PCIe x16 to 2x SlimSAS 8i host adapters cost about the same in the US and EU as in China. Source: @colo2442, 2026-10-06
Source: @kellertreppe123, 2026-10-24 Mountains extended his cards above a regular case on a 2020-extrusion frame with external SlimSAS. Source: @mountains1605, 2026-10-21
Retimed adapters: atsasin's 39com SlimSAS adapter boards have proper retimers and show no signal degradation at PCIe Gen3. Source: @atsasin2126, 2026-10-17
Connector names get mixed up: the 8i ports on the Tyan Tomcat board in the Hyve servers are OcuLink, not SlimSAS (endsjeremy corrected himself). Source: @endsjeremy, 2026-10-23

Bifurcation

Passive x4x4x4x4 adapters are cheap (about 20 USD) but need bifurcation support in the board's BIOS. Source: @ichundes, 2026-10-10
Source: @xeax, 2026-10-10
Consumer boards often bifurcate only 4x x4: dottoreargus' X870 board with one Gen5 x16 slot. Source: @dottoreargus, 2026-10-07
x8/x8 server slots: slots bifurcated x8/x8 so every card sits behind its own root port worked for four cards under ESXi (Gen2 x8 each). Source: @darkmaniac7, 2026-10-21 (lab notes)
WRX90 BIOS mod: NA-DEGEN-GIRL/wrx90-bios-mod adds an x8/x8 bifurcation option to the ASUS Pro WS WRX90E-SAGE SE on BIOS 1317. Only a boot to Setup with the option visible has been observed, on one board; x8/x8 link negotiation, P2P and stability are not verified. Source: repo github.com/NA-DEGEN-GIRL/wrx90-bios-mod by NA-DEGEN-GIRL on GitHub, posted by @stcord, 2026-10-25 Theory See ASUS Pro WS WRX90E-SAGE SE.
NVMe on switch boards: cheap NVMe carriers only work with bifurcation; real NVMe RAID cards such as the HighPoint SSD7749M2 are expensive (cibernox, kawaii). peter643 considered a 300 USD 8x NVMe switch card for a board without bifurcation before going the switch-firmware route. Source: @kawaii8789, 2026-10-08
Source: @cibernox, 2026-10-08
Source: @peter643, 2026-10-08
Gigabyte MZ32-AR0 splits one PCIe x16 into SAS connectors. Source: @wakeupmbrc1, 2026-10-16

PCIe switch boards

Switch boards (PLX/Broadcom PEX) let one host slot feed several cards and let cards talk to each other without crossing the CPU. They are the usual way to run 4 to 10 cards from a consumer board.

Which switch

PEX88096 (Gen4): one board feeds 5 cards at x16, or 10 at x8. Typical boards take a PCIe x16 host link and use 2x SFF-8654 8i per board. A 6-port version cuts the cost about in half (endsjeremy). Source: @gnasherrr, 2026-10-17
Source: @wakeupmbrc1, 2026-10-18
Source: @endsjeremy, 2026-10-21
Source: @endsjeremy, 2026-10-21
PEX88144 / PEX89144: one board for 8 cards from one root port, but expensive (PEX89144 over 3000 USD) and rare. Source: @wakeupmbrc1, 2026-10-17
Source: @wakeupmbrc1, 2026-10-18
Gen5 lane counts: among Broadcom's Gen5 switches only the PEX89144 and PEX89032 have lane counts divisible by 16 (gnasherrr). DaLZy's guess at the reason: customers wanted to split one x16 into four x16 slots and still attach a NIC or USB controller. Source: @gnasherrr, 2026-10-18
Source: @dalzyyy, 2026-10-18
Cascading: with two PEX88096 boards in two host slots there are 16 lanes left on each switch, so cascade them so DMA reaches all 8 cards (endsjeremy). The alternative he named for 8 cards is a single 10-port switch instead of two. Source: @endsjeremy, 2026-10-18
Source: @endsjeremy, 2026-10-18
3x Gen4 x16 switch cards exist and cost less than the 5x x16 versions. Source: @gpmickey, 2026-10-07
A Broadcom redriver/retimer board is the other option GlobalApathy knows for adding slots. Source: @globalapathy, 2026-10-07
Older switches in GPU servers: PEX9797 (PCIe 3.0) in the Supermicro SYS-4028GR-TRT2; PEX 8747 in the PixelML club's 4-card host. See Supermicro SYS-4028GR-TRT2.

What people found

P2P works on a PEX88096 board from Alibaba ("In Stock PEX88096 5 GPU Expansion"): domrockt2 got P2P once he fixed a SAS cable in the wrong port, which had left the host link at x8. In hindsight he would buy one with more slots. Source: @domrockt2, 2026-10-15 He runs one on a Z690 with an i5-14600K and recommends a fan on the PLX chip. Source: @domrockt2, 2026-10-22
Source: @domrockt2, 2026-10-22
10-port PEX88096 boards from Alibaba: djpandaman uses a 10-port PLX 88096 board with a 2x 8i host connection and says the listing hawtt_dawgg found is the same chip and connection; Mickey bought three boards with cables and riser cards for 1200 CAD (about 850 USD), about 4 kg for the three sets. hawtt_dawgg also spotted a newer "PEX88096 Switch Based 4-Slot PCIe" listing; Mickey notes 4-slot boards get hard once you want 12 cards at x16 in one server. Source: @_syn_thetic_, 2026-10-22
Source: @djpandaman, 2026-10-22
Source: @djpandaman, 2026-10-22
Source: @gpmickey, 2026-10-22
Source: @gpmickey, 2026-10-22
Source: @gpmickey, 2026-10-22
Source: @_syn_thetic_, 2026-10-22
Static BAR1 P2P on a PEX 8747 host: the PixelML club got 12/12 ordered pairs byte-exact at 5.79 GB/s with four extra patches and a pre-driver BAR placement step adopted via kexec; latency 16 us (4 KiB) and 194 us (1 MiB) vs 22 and 343 us host-staged. P2P helped Qwen3.8-27B TP2 (+11%) but hurt GLM-5.3-Flash TP4 on that Broadwell host (-20 to -26% one user), the opposite of Morrowmake's EPYC result (+11%). Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @djpandaman, 2026-10-24 Contested
Two cards behind one switch corrupted the kernel resource tree on every boot until the per-card BAR1 resize was serialized (rebar-serialize.patch, same fix as cmpunlocker PR #59). Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @djpandaman, 2026-10-24
MMIO behind switches: OB88594 says many boards show MMIO problems behind a PLX switch, so only mailbox P2P is usable there, and in tensor parallel that mailbox P2P crashes the inference program with Xid 13. Source: @ob88594, 2026-10-07 Some PLX machines have too little MMIO for the 64 GB BAR, and the Morrowmake P2P mode runs with a 64 MB BAR. Source: @ob88594, 2026-10-05
Switch firmware: replacedriver47 found his PEX88096 firmware lacked hotplug, which he says is required to remap BAR addressing behind the switch. Source: @replacedriver47, 2026-10-24
Reported P2P never completes behind a PLX 88096, even at 64 GB BAR1. replacedriver47 has four cards behind a PLX 88096 switch. By tuning MMIOH and BAR sizes he got two of them to a 64 GB BAR1 (the MMIOH window is too small for all four at 64 GB), but even between those two, P2P transfers never complete. He had heard Morrowmake's cmpunlocker makes P2P work; it did not for him, and he wants the root cause before he enlarges MMIOH further. mrpandagod pointed him to PixelML's notebook on 4-card BAR1 P2P over PLX with CUDA. ElectroHobo had P2P working with an older method and thinks Morrowmake's vLLM fixes it for many people on supported hardware. The root cause is unresolved. See P2P and Minisforum MS-01. Source: @replacedriver47, 2026-10-24
Source: @replacedriver47, 2026-10-24
Source: @replacedriver47, 2026-10-24
Source: @replacedriver47, 2026-10-24
Source: @replacedriver47, 2026-10-24
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @djpandaman, 2026-10-24
Source: @nathang6866, 2026-10-25
Per-slot bifurcation on a PEX88096 needs a custom SBR (switch firmware). The seller alifond reprogrammed the last slot of peter643's board to 4x x4 before shipping but would not share the image or other custom firmware, citing an NDA; peter643 later worked out how to build his own; an open hardware project page has the procedure and files, and peter643 believes only about 8 bytes change (4 setting bytes and 4 CRC bytes). Source: @peter643, 2026-10-07
Source: @peter643, 2026-10-07
Source: @peter643, 2026-10-08
Source: @peter643, 2026-10-08 He prefers the 88096's own port bifurcation over a second PEX switch, which costs power and about 300 USD but could give 8x x4 NVMe for more random IOPS. Source: @peter643, 2026-10-07
Five slots, four cards: cibernox's switch takes 5 cards; he stops at 4 (256 GB) because he does not expect TP to scale well past 4 cards and 320 GB opens few extra models; Sojun agrees 256 GB is a reasonable stopping point. Theory Source: @cibernox, 2026-10-07
Source: @sojun80, 2026-10-07
GPUDirect Storage across switches: markshowa (2x PEX9797 plus a PLX88048 carrier) finds GDS works and does better inside a switch, but writes across switches are very slow and look serialized; he suspects the same mechanism behind his poor cross-switch P2P. Source: @markshowa, 2026-10-22
Warning Power: a rig with four cards on one PLX board and two independent PSUs lost all four cards and the PLX board at once under large prompt-processing loads. See Cooling and power. Source: @replacedriver47, 2026-10-05
Cards dropping off a switch baseboard under load: Moliminous' 5 cards on an external GPU switch baseboard (beside 6 RTX 3090 on Ubuntu LTS) were stable alone but fell off the bus with PCIe lane and IOMMU errors on large prompts. Temperatures and PSU draw looked fine, though the logs showed the first card overdrawing to 260 W; lowering the power limit from 250 W to 180 W fixed it, and GLM-5.3-Flash now runs at 380k context. Source: @moliminous, 2026-10-19
Source: @moliminous, 2026-10-19
Source: @moliminous, 2026-10-19
Source: @gnasherrr, 2026-10-19 See Cooling and power.
Warning A PLX board from one Chinese supplier arrived defective twice (shorts as soon as the CPU power connectors are plugged in). Lyubo's post names the seller only in screenshots, so it is not repeated here. Source: @lubakmanqk, 2026-10-22
Source: @lubakmanqk, 2026-10-23

Topology rules of thumb

Lanes from one CPU. Miséricorde: you need all PCIe lanes from one CPU or P2P will not work properly. Source: @wakeupmbrc1, 2026-10-16 That holds for many boards, but it is not a hard rule: see the next item.
P2P across sockets works on a dual-socket EPYC server. On an 8-card Supermicro H12DSG-O-CPU (dual EPYC Rome, cards on both CPUs, Gen2 x16, 64 GB BAR1, Morrowmake P2P build), p2p-content-check.py passed all 504 checks on 8 devices (every ordered pair at every transfer size, including the pairs across the two sockets), and cudaDeviceCanAccessPeer returned 1 for every pair. Proven Source: @darkmaniac7, 2026-10-29 (lab notes)
- Conflicting earlier source: Miséricorde said P2P does not work across sockets, which is why GPU servers hang all GPUs off one CPU. Source: @wakeupmbrc1, 2026-10-18
- Other earlier posts: xanthe_ named the mailbox method and bayley's Linux kernel patch as ways to get P2P Source: @xanthefin, 2026-10-18; programmerboi finds QPI/NUMA overhead relatively low. Source: @programmerboi, 2026-10-18
Keep a TP group behind one switch. On servers with two switches of five slots each, put the cards of one group behind the same switch (consecutive slots). Source: @nathang6866, 2026-10-20 On a Gigabyte G292-Z20 each pair sits on its own switch: fast P2P inside a pair, slower between pairs. Source: @cmdrwaka324, 2026-10-26
Hops: gnasherrr recalls PLX latency at about 105 ns. Source: @gnasherrr, 2026-10-18 endsjeremy: tensor parallel across the array is bandwidth limited whatever the lane count, so x8 per card is enough, and switches shine for PP and P2P DMA. Source: @endsjeremy, 2026-10-18
Source: @endsjeremy, 2026-10-18
Source: @endsjeremy, 2026-10-18
P2P latency reference: about 1.7 us over the CPU root, about 1.4 us on a crossbar (markshowa); about 1.5 us per hardware call between CMPs on native Gen2 x16 slots (Miséricorde); switches and retimers add latency. Source: @markshowa, 2026-10-17
Source: @wakeupmbrc1, 2026-10-17
10 cards: zorg33 asked whether 10x x8 on one PEX88096 or 5+5 x16 on two switches gives better TP decode; untested. Source: @zorg33, 2026-10-06 djpandaman chose 10x x8 on one switch and will add a second if bandwidth bottlenecks. Source: @djpandaman, 2026-10-09 endsjeremy says jumping between two 5-card arrays "murders performance". Theory Source: @endsjeremy, 2026-10-09

Expansion boards and external chassis

fnerdman is considering moving four cards off an EPYC host onto a consumer CPU with an Alibaba "PCIe 4.0 x16 to ..." expansion board that appears to daisy-chain. Open questions he raised, unanswered: what its three 6-pin power inputs are, and whether cards come back stable if the board's PSU is cut and restored while the host stays up. Theory Source: @fnerdman, 2026-10-26

Fitting cards on the board

Front-panel header: matstatman needed about 10 mm of clearance to fit a card over the front-panel pin header. Options: angled Dupont connectors (GlobalApathy linked a moddiy 10-pin angled header, too tall for his case), pin-header risers (Mickey used them), or enabling Wake-on-LAN and "power on after power loss" and leaving the header unplugged (Sojun). Source: @matstatman, 2026-10-07
Source: @globalapathy, 2026-10-07
Source: @matstatman, 2026-10-07
Source: @gpmickey, 2026-10-07
Source: @sojun80, 2026-10-07
Fan headers: alanoo: on ASRock Rack Rome boards the cards collide with 4 of the 7 fan headers. fafinet's cards also touch the fan headers on his board and do not seat fully, but still work. Source: @alanoo, 2026-10-21
Source: @fafinet, 2026-10-21
Source: @fafinet, 2026-10-21