PCIe switch boards (PEX88096, PEX88064, PEX8747, PLX8749)
More actions
A PCIe switch board takes one host link, usually Gen4 x16, and fans it out to several cards. A Broadcom PEX88096 gives 5 slots at x16, 10 at x8 or 20 at x4; the PEX88064 is a smaller, slightly cheaper sibling; the older PEX8747 and PLX8749 cards (Gen3) turn one x16 slot into x16+x16 or 4x4. Cards behind the same switch can exchange data directly (P2P) without going through the CPU, so owners use these boards to run 5 to 10 cards from a consumer board, a mini PC or a host short on lanes. With several cards and 64 GB BAR1 behind a switch, Linux needs a large MMIO window, ACS redirect turned off on the switch ports and, on most hosts, two kernel patches (see Kernel side). For Gen2 retrain problems behind a switch see Driver and unlock failures.
What a switch gives
PEX88096: one Gen4 x16 uplink fanned out as 5x16, 10x8 or 20x4, with P2P inside the switch
The PEX88096 has 96 Gen4 lanes: 16 go upstream to the host and 80 downstream, so one board feeds 5 cards at x16, 10 at x8 or 20 at x4 from a single Gen4 x16 uplink. Total throughput to the CPU stays Gen4 x16, like a network switch, but cards on the same switch talk to each other directly when they support P2P. Listings differ mainly in whether the split is selectable (DIP or toggle switches, or a different SBR flashed onto the board) or fixed; Zorg33's board is fixed at 10x8. The board is an open-source design on oshwhub (design page; Broadcom chip page). Zorg33 says tensor parallel is the main reason to buy one, since cards get full x16 among five (or x8 among ten). robbbbbb7973 adds that with P2P the shared uplink is not the bottleneck (four cards at PCIe 2.0 x16 equal one PCIe 4.0 x16) and that switch-local P2P lowers latency for pipeline parallel. By late September several members ran these boards (often sold as "PLX88096"); domrockt2's Alibaba board "does p2p like a champ". Switches can be chained at a signal-integrity and latency cost, passive splitters need motherboard bifurcation instead, and Arcanum notes one popular board exposes only 48 of its 80 downstream lanes.
Proven 8 GB and 10 GB
Source: @skyy2333, 2026-07-24
Source: @robbbbbb7973, 2026-07-24
Source: @eksitus, 2026-07-24
Source: @robbbbbb7973, 2026-08-25
Source: @danske1, 2026-08-25
Source: @zorg33, 2026-08-25
Source: @rubbra_f, 2026-08-25
Source: @_arcanum_, 2026-08-25
Source: @djpandaman, 2026-08-26
Source: @djpandaman, 2026-09-29
Source: @domrockt2, 2026-09-29
Source: @_arcanum_, 2026-09-29
Source: @lia_oc, 2026-09-29
Source: @stcord, 2026-09-29
PEX88096 primer: cost per card, and 8- and 10-card layouts
One board gives 5x16 or 10x8 by DIP switch (down to x4 with splitter cables), so two boards give ten Gen4 x16 slots (Arcanum). Jeremy (Ends) puts one 88096 card at about $700 to $800 with cables and risers. For 8 cards he uses two such cards joined by an x16 link, 4 cards on each; for 10 cards with interconnect, a 4-port PLX card (about $150) feeding two 10-port cards comes to about $1700 in total. Decommissioned 2-port PLX switches from NVMe use are cheap. SkirtChaser: the PEX89144 (Gen5) and PEX90144 (Gen6) have 144 lanes, but the PEX88096 is the largest hobbyists can realistically get.
Reported
Source: @endsjeremy, 2026-09-10
Source: @_arcanum_, 2026-09-11
Source: @skirtchaser, 2026-09-11
PEX88096 board variants: blue with DIP switches, black with a firmware-set split, slotted boards
The blue-PCB PEX88096 board has small DIP switches that set the split (5x x16, 10x x8 or 20x x4). On the black-PCB board the firmware fixes the split; Broadcom's tooling can change it, but the effort is unclear. Both take SlimSAS 8i cables. Two other boards have real PCIe slots: a 5-slot board fed by 2 SlimSAS cables, and a 4-slot board with SlimSAS in and out headers for daisy chaining. replacedriver47 ordered a black 5x x16 board, and the seller offered a green version described as the same apart from different power plugs and a heatsink fan. tony's Taobao board prints its DIP-switch mode table on the back of the card.
Reported
Source: @robbbbbb7973, 2026-09-05
Source: @replacedriver47, 2026-09-05
Source: @roboalchemist., 2026-09-05
Source: @tony_smartecom, 2026-09-05
PEX88096 baseboard SBR files: which one goes where
The lane split of the oshwhub PEX88096 baseboard is set by a preconfigured SBR file. The project page needs a Chinese phone number to log in; mrpandagod got the files without an account and posted the SBR zip plus a zip of 11 PEX88096 PDFs in Discord. The SBR sits on a 32 Mb (4 MB) 25-series SPI flash that runs at 1.8 V, so a clip programmer must support 1.8 V. SDK.bin is for flashing live; SPI.bin carries a 1 KB offset and is for writing the chip directly with an external programmer. AIC_ files are for the add-in card with 10x SlimSAS 8i, GPU_ files for the 5-slot baseboard, and the profile name gives the lane split (5x16 = five x16).
Reported
Source: @peter643, 2026-09-12
Source: @djpandaman, 2026-09-12
Prices
PLX and PEX switch prices, July to September 2026
Jeremy (Ends), 2026-07-27: a PEX88096 10x 8i PCIe 4.0 card was about $300 and 5-card PCIe 4.0 88096 expansion boards about $500. Zorg33: 88096 cards cost about half as much on Alibaba as on AliExpress or eBay, though shipping can be about $80. SKYY (2026-07-24) priced the open-source PEX88096 board at about $370 on AliExpress, about $250 on Xianyu, or cheapest batch-built (about 10 boards) at JLC PCB. cibernox (2026-09-21) bought a 5-slot board locally for 300 EUR, about half the usual price, against about $450 on Alibaba ($550 after shipping and fees); he calls the PEX88096 the gold standard for 4 or 5 slots, sold either as 5 Gen4 x16 slots with one Gen4 x16 uplink or as 4 slots with two uplinks. Merom Green's chain is PC, a 50 cm riser, a PEX88064, a SlimSAS 8654 8i cable and a PCIe backplane (single 8i, so x8); he advises single-slot x16 backplanes with two 8i connectors for easy replacement and finds PEX switches better value than QSFP 100G networking. Con3po runs eight cards on one at 8x8 and plans a second switch for full x16.
Reported
Source: @skyy2333, 2026-07-24
Source: @endsjeremy, 2026-07-27
Source: @meromgreen, 2026-07-27
Source: @zorg33, 2026-07-27
Source: @cibernox, 2026-09-21
Source: @_arcanum_, 2026-09-21
Source: @con3po7016, 2026-09-21
Source: @fiiireface, 2026-09-21
Source: @n3bs, 2026-09-21
5-slot PEX88096 GPU box from Alibaba or Taobao: about $450 to $700, about $1k landed
Studebaker found a 5-GPU box with a Broadcom PEX88096 on the baseboard, linked to the host by a single x16 (MCIO adapter): about $450 for the baseboard and about $200 more with the chassis (2026-08-10), later quoted at $582. Arcanum found it for 600 USD on Taobao, about 30% above the bare switch board, with CRPS PSUs sold separately; gpulover saw $590 in Korea with free shipping plus 10% tax, and ju5t5omedude estimated about $1k once PSUs, shipping and tariffs are added. A PEX88096 with 80 downstream lanes gives 5 full x16 slots (Lyubo first thought x8, then agreed and called $600 to $700 a good price). Moliminous, who owns these boxes, says the switch handles GPU-to-GPU traffic and model loading is the bottleneck. Cooling is the doubt: 5x 250 W is 1250 W against 120 mm intake and 80 mm exhaust (gpulover); OB88594 suggests 9733 blowers per card with the rear fan removed, and DeeJay a printed rear cage for 2x 140 mm. No CMP results for the box were posted in August. Listings: Alibaba 5-GPU box, Alibaba 5-way PCIe 4.0 box.
Reported
Source: @_studebaker_, 2026-08-10
Source: @danske1, 2026-08-10
Source: @_arcanum_, 2026-08-10
Source: @lubakmanqk, 2026-08-10
Source: @rubbra_f, 2026-08-10
Source: @moliminous, 2026-08-10
Source: @endsjeremy, 2026-08-10
Source: @ju5t5omedude, 2026-08-14
Source: @artistosu, 2026-08-23
Source: @deejay9983, 2026-08-23
Source: @ob88594, 2026-08-23
A switch board that died after an hour, and a replacement that shorted on power-up
Warning Lyubo's PLX PCIe switch board, bought on Alibaba from Maishu Technology Co Ltd, worked with the cards for about an hour and then died (2026-09-02). The seller refunded and sent a replacement. That board looked fine but shorted as soon as the CPU power connectors were plugged in, tripping the PSU's protection (the PSU was not damaged). The seller had all-positive reviews, good communication and pictures; the two defective boards cost him two months, and he posted "Do not buy from this seller" and ordered a Xeon system as a fallback. meatsus warned that such sellers often misrepresent these boards. What to do instead: install the cards and run a 1-hour stress test before trusting a new board (Lyubo's test), and prefer the bigger makers OB88594 recommends, such as 速桥 (Suqiao) or 39com. replacedriver47's green 5-slot board showed up in lspci but had not been tested with cards yet.
8 GB and 10 GB
Source: @lubakmanqk, 2026-09-02
Source: @meatsus, 2026-09-02
Source: @replacedriver47, 2026-09-25
Source: @ob88594, 2026-09-25
Source: @lubakmanqk, 2026-10-22
PEX88096 kits on Alibaba, Taobao and Goofish: $450 to $850, September and October
mrpandagod paid $472.90 plus $150 DDP shipping ($623.00 total) for a 10-port PCIe switch card, 6 riser baseplates and 10 SlimSAS cables from Shenzhen Huateng Technology Intelligence Information Co., Ltd. on Alibaba, delivered to the US (2026-09-28); Con3po uses the same kit, whose cables are 75 cm. Jeremy (Ends) says prices vary widely between traders (he named Alifond and susquio) and planned to use this price to negotiate. Mickey received 3 boards with cables and riser cards for 1,200 CAD (about 850 USD), roughly 4 kg for the three sets. On Taobao, tony found a board for about 1,550 RMB (2026-09-05) against the usual 2,000 RMB, which robbbbbb7973 paid for a version with toggle switches; ST could not find one under 500 EUR, and roboalchemist bought two of the large boards, enough for 20 GPUs. Arcanum (2026-10-01): a switch plus 10 SlimSAS-to-PCIe adapters and 10 cables is about $500 all in on Taobao; mrpandagod paid about $450 ($550 with shipping) for a switch, 6 adapters and 10 cables. DaLZy sees about 1,300 RMB on Goofish, and Miséricorde says PLX boards work for P2P but are expensive. Taobao has the best prices but its sellers pause for China's Golden Week (Mickey); domrockt2 bought through Alibaba with Secure Pay and express shipping.
Reported 8 GB and 10 GB
Source: @tony_smartecom, 2026-09-05
Source: @robbbbbb7973, 2026-09-05
Source: @stcord, 2026-09-05
Source: @peter643, 2026-09-06
Source: @spectakkular, 2026-09-07
Source: @roboalchemist., 2026-09-07
Source: @stcord, 2026-09-28
Source: @con3po7016, 2026-09-28
Source: @djpandaman, 2026-09-28
Source: @endsjeremy, 2026-09-28
Source: @cressida1881, 2026-09-28
Source: @_arcanum_, 2026-10-01
Source: @con3po7016, 2026-10-01
Source: @djpandaman, 2026-10-01
Source: @skyy2333, 2026-10-01
Source: @gpmickey, 2026-10-02
Source: @domrockt2, 2026-10-02
Source: @dalzyyy, 2026-10-16
Source: @endsjeremy, 2026-10-16
Source: @wakeupmbrc1, 2026-10-16
Source: @zingy90210, 2026-10-16
Source: @djpandaman, 2026-10-22
Source: @_syn_thetic_, 2026-10-22
Source: @gpmickey, 2026-10-22
Source: @wakeupmbrc1, 2026-10-22
Taobao Gen4 switch cards with SlimSAS: about 330-350, or about 570 with 10 boards and cables
Arcanum: a Taobao PCIe switch card with SlimSAS ports costs about 330-350 (currency not stated); he bought one as a bundle with 10 boards and cables for about 570, and another listing he showed was about 370-380 (2026-07-31). Studebaker linked the Broadcom PEX88096 in this context. Rubbra_F reported that Taobao keeps limiting his account, and CUTE said you have to verify your identity with them.
Reported
Source: @rubbra_f, 2026-07-31
Source: @cute_sc2, 2026-07-31
Source: @_arcanum_, 2026-07-31
Source: @_studebaker_, 2026-07-31
Alibaba 5-way PCIe 4.0 external box: a $582 quote from Ailfond, no PSU
PJ got a $582 quote, without PSU, from the seller Ailfond for the Alibaba "5-Way GPU PCIe 4.0" external chassis (2026-08-13). ju5t5omedude: it appears built around the 5x GPU PLX board some members post, Alibaba list prices are not real and shipping is usually where they get you, and the PSUs resemble HP units but may be narrower. Studebaker wants one but notes the PSUs are proprietary rather than common ones; Petri Krohn recommends it with an old PC instead of buying a workstation.
Reported
Source: @zingy90210, 2026-08-13
Source: @ju5t5omedude, 2026-08-13
Source: @_studebaker_, 2026-08-13
Source: @petrikrohn_26497, 2026-08-14
The PEX88096 version of the external box: about AUD 1100, P2P unconfirmed at the time
B16_c wanted an external multi-GPU enclosure with a PCIe switch bridged to an existing server and ordered the PEX88096 version of the Alibaba 5-way box for about AUD 1100 (2026-08-22). gpulover was hesitant because P2P through a PEX88096 was unconfirmed on CMP cards then. Arcanum: the box comes either with passives (SlimSAS or MCIO direct to each card) or with a PEX88096, nearly all its cost is the 88096, and the CRPS PSUs sold with it are non-switching. ju5t5omedude thinks about USD 500 with PSUs included is fine. No test result was posted.
Reported
Source: @cmurrayis, 2026-08-22
Source: @danske1, 2026-08-22
Source: @artistosu, 2026-08-22
Source: @_arcanum_, 2026-08-24
Source: @ju5t5omedude, 2026-08-24
September prices: EUR 300 used, about EUR 600 on AliExpress
cibernox bought a second-hand switch board with 5 Gen4 x16 slots and a x16 uplink for EUR 300 plus EUR 20 shipping from a one-off Wallapop listing (2026-09-06). The next cheapest he saw was about EUR 600 on AliExpress, and a 2-slot Gen3 x16 switch was EUR 200. ST: 10-port PCIe 4 switches are over EUR 500. Jeremy (Ends): Alibaba ads at 300 are probably bait and switch.
Reported
Source: @cibernox, 2026-09-06
Source: @stcord, 2026-09-06
Source: @endsjeremy, 2026-09-07
Cheap 10-port PEX88096 boards from kcores / oldmonster: about 300 USD on Taobao, needs flashing
Jeremy (Ends) found a cheap 10-port PEX88096 interconnect board from "oldmonster" on eBay; it sold instantly, and the design files are said to be public on oshwhub (OOCP project) (2026-09-10). Arcanum identified the maker as kcores, around for years with an 88096 design at about 300 USD equivalent on Taobao. The boards must be flashed to change the downstream port configuration; Jeremy says flashing is not hard and the seller lets you pick the configuration when ordering. Nea Popa also linked an Alibaba PEX88096 4-GPU baseboard.
Reported
Source: @neapopa, 2026-09-10
Source: @endsjeremy, 2026-09-11
Source: @_arcanum_, 2026-09-11
Single-root switch boards for P2P: an eBay board, the SYS-4048GR-TRT2, the PEX88096
bayley_w (2026-08-11) linked an eBay PCIe switch board for US buyers who want a real single root, and on 2026-08-13 recommended the SYS-4048GR-TRT2; ElectroHobo (2026-09-24) had a $255 best offer accepted. bayley_w described that design as lanes quadrupled with the GPUs able to talk only to each other, not to the host, "sort of a bootleg dgx"; Big Ptoughneigh found its uplinks are x24, not x16. Zorg33's PEX88096 board can be set as 1x x16 to 10x x8 (one switch) or 2x x16 to 10x x16 (two switches).
Reported
Source: @bayley_w, 2026-08-11
Source: @nathang6866, 2026-09-24
Source: @zorg33, 2026-09-24
Source: @bigptoughneigh, 2026-09-24
Builds
A PEX88064 rack box powered from one riser; an eBay PLX board for 10x8 or 5x16
Merom Green owns a PEX88064 switch board bought prebuilt from a Taobao seller with a 19-inch rack case. It is powered through a single riser, so the switch draws under 75 W; he wishes he had bought the dual 8i backplate version (2026-08-13). DeeJay recommends an eBay PLX switch board that runs 10 cards at x8 or 5 at x16 from one x16 slot using mini-SAS cables and riser cards; he paid $500 for the same board. Card-style switch boards with external PSUs need PS_ON and GND synced (a sync cable or board), while baseboard-style boxes with a 24-pin input may not. SkirtChaser notes the PEX88096 is specified at 35.78 W typical at full Gen4 load, which he doubts its small fan handles.
Reported
Source: @meromgreen, 2026-08-13
Source: @skirtchaser, 2026-08-13
Source: @zingy90210, 2026-08-13
Source: @deejay9983, 2026-08-14
Source: @ju5t5omedude, 2026-08-14
Alibaba PEX88096 carrier on a Z690: three cards with P2P; GPUDirect Storage differs inside and outside the switch
domrockt2 runs 3x CMP 170HX 8 GB behind an Alibaba "In Stock PEX88096 5 GPU Expansion" board on a Z690 with an i5-14600K; P2P works with every model he tried (2026-10-01). At first the host link was only x8 because a SAS cable was in the wrong connector; putting the cables in the correct order fixed it. He recommends a fan on the PLX chip, would buy a version with more slots in hindsight, and keeps a Kioxia CD6 on the CPU's x4 lanes through an M.2 to U.3 adapter. markshowa (two PEX9797 boards plus a PLX88048 Gen4 carrier for SSDs): GPUDirect Storage works and does better inside the switch, but GDS writes from outside it (an NVMe read into the other switch) are very bad and appear serialized, and more queue depth does not help; he thinks the same mechanism explains his poor cross-switch P2P. gnasherrr bought 88096 switches for a cascaded AM5 setup, and ST's switch arrived on 2026-10-21. domrockt warns that mini-SAS cable quality is mixed, so buy good ones.
Reported 8 GB and 10 GB
Source: @domrockt2, 2026-09-29
Source: @domrockt2, 2026-10-01
Source: @gpmickey, 2026-10-02
Source: @domrockt2, 2026-10-15
Source: @voidgiraffe, 2026-10-15
Source: @domrockt2, 2026-10-21
Source: @markshowa, 2026-10-21
Source: @_syn_thetic_, 2026-10-21
Source: @gnasherrr, 2026-10-21
Source: @stcord, 2026-10-21
DeeJay's 5-slot x16 PLX board ($479): P2P and NCCL work, but no latency gain
DeeJay bought a 5-slot PLX switch board for a 5th card at $479 after coupon, the cheapest way he found to keep each card at x16 (2026-08-12). robbbbbb7973 warned that the PLX chip draws about 35 W and needs airflow over its heatsink. Asked by Petri Krohn whether the cards talk over NCCL through the switch without the CPU, DeeJay said P2P and NCCL already worked on his existing setup but did not help latency ("it was a dead wash in all my testing"); he planned the x16 mod and a tensor-parallel retest on the board. On 2026-09-01 he wanted 10 cards connected by PLX switches for near-1T-parameter models and was designing a fan shroud that cools both the cards and the PLX chip. segmond listed the interconnect options: PLX switch, SlimSAS, MCIO, OCuLink and physical PCIe splitters.
Reported
Source: @deejay9983, 2026-08-12
Source: @robbbbbb7973, 2026-08-12
Source: @petrikrohn_26497, 2026-08-12
Source: @deejay9983, 2026-09-01
Source: @djpandaman, 2026-09-01
Source: @segmond, 2026-09-01
Source: @ins_mniac, 2026-09-01
Con3po's 8-card PEX88096 pod on a mini PC: 11 h at full load without errors
Con3po runs 8 CMP 170HX at Gen2 x8 each on a PEX88096 switch driven from a Minisforum S1 Max (he wrote "MS 1S Max") whose PCIe is limited to x4. With P2P on the switch, inter-card traffic never touches the narrow uplink (Zorg33 confirmed P2P runs over the full x16 through a switch), though model loading is slow. His board ships with 10 PCIe backboards and a DIP switch that drops the uplink from x16 to x8 or x4; the bare PEX88096 card (10x SFF-8654) feeds cabled slot boards in a separate pod with its own PSUs. On 2026-09-07 he reported 11 h at full load with zero Xid and zero PCIe errors on a Taobao PEX88096 board: every drop he has had came from the PSU or heat, not the switch, but one uplink drop takes all 8 cards down and needs a power cycle. The kernel sees a plain PEX88096 bridge tree set by DIP switches, with no drivers or firmware, and he calls it "a decode box, not a training box". His posts disagree on the uplink (OCuLink from 2026-08-28, a 40 Gb/s USB4 link again on 2026-09-07, and a 2026-09-05 note that his cheap OCuLink parts never connected), so which link carried the 11 h run is unconfirmed. At 8 cards he said BAR1 had to be handled ("mailbox bar1") because of a limit around 483 GB (wording unclear). For a Minisforum MS-01 (x16 physical, Gen4 x8 electrical) he advised a direct riser to the switch, since OCuLink would cut it to x4.
Reported
Source: @con3po7016, 2026-08-25
Source: @zorg33, 2026-08-25
Source: @ju5t5omedude, 2026-08-25
Source: @stcord, 2026-08-25
Source: @vektor_prime, 2026-08-25
Source: @darcidride, 2026-08-25
Source: @segmond, 2026-08-25
Source: @con3po7016, 2026-08-28
Source: @cressida1881, 2026-08-28
Source: @con3po7016, 2026-08-28
Source: @vektor_prime, 2026-08-28
Source: @spale777, 2026-08-28
Source: @ju5t5omedude, 2026-08-29
Source: @con3po7016, 2026-09-07
Source: @roboalchemist., 2026-09-07
Source: @ins_mniac, 2026-09-07
Petar's NVMe-oF on the same PEX88096 card
Petar uses the same PEX88096 card (10x SFF-8654) with a NIC plus NVMe drives for NVMe-oF without touching the CPU, connected with SFF-8654 to U.2/U.3 cables.
Reported
Source: @spale777, 2026-08-28
Schaka: dual Cascade Lake with PLX8749 switch cards; Gen2 fine, BAR1 and P2P need bayley's kernel patches
Schaka runs cards on a Supermicro dual-socket Cascade Lake board with PLX8749 PCIe 3.0 x16 to 4x4 switch cards (the last 2 in stock from 10Gtek, about 100 EUR each) and C-Payne PCIe 4.0 slot boards, one SFF-8654 cable per card at x4. PCIe Gen2 worked behind the switches without issue, but BAR1 and P2P "basically never worked" until he applied bayley's kernel patches. He reports PCIe 3.0 and memory training are stable on Cascade Lake and uses Optane to hold models for fast switching. His repo lists the stack: eight 8 GB cards behind two PEX8749 switches at Gen2 x4, kernel 7.1.10-cmp and nvidia-open 610.57.04 with 12 bayley/cmpunlocker patches.
Reported
Source: @schakaqt, 2026-09-04
Source: repo github.com/Schaka/170hx-journey by Schaka on GitHub
Pointer: a longer write-up by @schakaqt (his reproducible PLX8749 setup, now the 170hx-journey repo) exists; see GitHub and Discord. The how-to waits for the author.
Two cards without the cap mod on a Dell CDC7W PLX x16 switch card
MrManny runs 2 CMP 170HX without the cap mod on a Dell CDC7W PCIe PLX x16 switch card and shared a screenshot of his model setup (2026-09-30). No bandwidth or P2P numbers were posted.
Reported
Source: @mrmannyatx, 2026-09-30
SirBAM-a-lot, building on his Gigabyte G292-Z20, planned a franken-rig with Chinese PCIe extensions to load up to 18 GPUs. All 5 GPUs on one extension share a single PCIe 4.0 x16 link to the CPU, but the PLX switch handles traffic among those 5 internally without sending it to the CPU, so careful model mapping can work around the uplink. He framed 1.152 TB of HBM for under 30K EUR as cheap.
Reported 8 GB and 10 GB
Source: @sirbamalotx, 2026-07-22
Source: @giaki3003, 2026-07-22
Miwin 11-slot PEX baseboards (under $200, switch on a separate card) against the CoCoLink 10-slot
SkirtChaser posted a setup on an 11-slot PEX baseboard from AliExpress/Alibaba (the Miwin "direct-attached 11 slots" board, sold under many names for under $200 per Zorg33) and proposed a blower pulling air through the cards from behind the PCI slots. The 11th slot can host the PEX uplink. Petri Krohn advises against the CoCoLink 10x PCIe x16 baseboard with integrated PLX and recommends the Miwin board with the PLX switch on a separate PCIe card. segmond notes 11 slots at 75 W each need 825 W; ju5t5omedude points out two EPS connectors (segmond: good for about 660 W). intervitens notes dual-PEX88096 baseboards for 10 cards and many cheap Supermicro and Inspur baseboards on Taobao and Xianyu, though getting them working outside their server is unpredictable. These boards also come in Gen3/Gen4 SlimSAS versions; PCIe 5.0 multi-slot baseboards were listed at $3k to $5k each (Petar).
Reported
Source: @skirtchaser, 2026-08-12
Source: @zorg33, 2026-08-12
Source: @intervitens, 2026-08-12
Source: @scallion_9883, 2026-08-12
Source: @spale777, 2026-08-25
Source: @con3po7016, 2026-08-25
Source: @petrikrohn_26497, 2026-08-25
Source: @segmond, 2026-08-25
Source: @ju5t5omedude, 2026-08-25
Petar's single-root build from Taobao switch boards (in progress)
Petar is building his own multi-card box on PCIe switch boards and wants a single-root topology: he plans to chain the second switch to the first and replace the baseboard (2026-08-22). OB88594 linked a Taobao listing and said a PEX88096 board gives 1 to 5 x16 links and that sellers include the needed parts in the set. Petar noted a separate switch card is still needed to make two switches single-root, and promised to post the build when done.
Theory
Source: @spale777, 2026-08-22
Source: @ob88594, 2026-08-22
peter643's PEX88096 enclosure: 4 cards plus 4 Gen4 x4 NVMe at about 24 GB/s
peter643 runs 4 CMP 170HX in a PEX88096 switch enclosure (an AILFond external unit from Alibaba, connected to the host over SlimSAS) with 4 Gen4 x4 NVMe drives giving about 24 GB/s sustained read, which roughly saturates the Gen2 x16 links of the four cards. His i7 9700K host lacks enough BAR1 address space, so he wrote his own P2PDMA path, a GDS-style mailbox system and his own nvidiafs driver; a newer CPU with more BAR1 space would make this easier. He re-bifurcated the last slot of the 5-slot board through the SBR firmware instead of buying a 300 USD 8x NVMe switch card; AILFond had told him the board must be sent back to change the lane assignment. A second PEX switch would cost power and about 300 USD but allow 8x x4 NVMe for more random IOPS. He expects 24 GB/s to be too slow for NVMe expert streaming of larger models; cibernox plans a similar card of cheap SSDs on the 5th slot of his switch.
Reported
Source: @stcord, 2026-08-25
Source: @peter643, 2026-09-05
Source: @robbbbbb7973, 2026-09-05
Source: @cressida1881, 2026-09-05
Source: @peter643, 2026-09-12
Source: @endsjeremy, 2026-09-12
Source: @cibernox, 2026-09-19
Source: @cibernox, 2026-10-02
Source: @voidgiraffe, 2026-10-02
Source: @peter643, 2026-10-07
Source: @cibernox, 2026-10-07
Source: @kawaii8789, 2026-10-07
Pointer: a longer write-up by @peter643 (the PEX88096 plus NVMe build and his home-made P2PDMA, mailbox GDS and nvidiafs driver) exists; see Discord. The method waits for the author. See also Gds-nvme-patch.
Minisforum MS-01 behind a PLX 88096: BAR1 stuck at 64 MB, P2P does not move data
replacedriver47's MS-01 will not create the large BAR space for 4 cards behind a PLX 88096 even with Resizable BAR enabled: MMIO is 256 GB and the BIOS tries to allocate the BAR1s inside a 512 MB section. With Morrowmake's cmpunlocker fork BAR1 stays at 64 MB instead of 64 GB and P2P does not come up. Later he found the PLX 88096 firmware on his switch does not have hotplug enabled, which is required to remap BAR addressing. Unresolved. Whether P2P at the 64 MB BAR works is disputed (see P2P: mailbox P2P is contested):
- OB88594 advises not to fight MMIO sizing unless the motherboard exposes independent options, and says Morrowmake's code can use P2P normally at the 64 MB BAR. Source: @ob88594, 2026-10-10
- replacedriver47 tried it: it reports working (no BAR1), but no transfers actually go through. Source: @replacedriver47, 2026-10-10
- Con3po reports running 8 cards from a Minisforum mini PC (the S1 Max of his pod above) using "mail box p2p" and offered his working setup. Source: @con3po7016, 2026-10-10
Source: @con3po7016, 2026-10-24
Contested
Source: @con3po7016, 2026-08-25
Source: @cressida1881, 2026-08-28
Source: @replacedriver47, 2026-08-30
Source: @peter643, 2026-09-12
Source: @djpandaman, 2026-09-12
Source: @replacedriver47, 2026-10-22
ASUS WRX80E and WRX90 with the Morrowmake fork: BAR1 stuck at 64 MB; an experimental WRX90 x8/x8 BIOS mod
Trust Inter (ASUS WRX80E SAGE, Threadripper PRO 5965WX, 3 cards) and GhostInTheDatacenter (ASUS WRX90) have BAR1 stuck at 64 MB with the Morrowmake fork, the same symptom as the MS-01. On the WRX80E, BAR1 reports 64 GB unlocked but the allocation ends at 64 MB because the prefetchable bridge window is only 96 MB; kernel command-line options, a CMOS clear and BIOS checks did not help. AnalyticETH's BAR1 beta fork had "no luck" on WRX90. Whether the board is the cause is disputed:
- GhostInTheDatacenter: "if you buy new, watch out for Asus boards, I am having issues with BAR sizing" (no model given). Source: @ghostinthedatacenter_87281, 2026-10-22
- Trust Inter concluded "BAR1, P2P is not possible" on his board (a single card on an Intel 13900K had worked for him). Source: @trustinter, 2026-10-03
Source: @trustinter, 2026-10-10 - meatsus doubted BAR1 is board-related, and Shirogeek urged isolating the cause first (other GPUs, a fresh Ubuntu); GlobalApathy noted only 1 of the board's 3 auxiliary PCIe power connectors was plugged, which can cause instability; ST suggested ReBarUEFI. Source: @globalapathy, 2026-10-03
Source: @meatsus, 2026-10-03
Source: @shirogeek, 2026-10-03
Source: @stcord, 2026-10-03
ST also shared NA-DEGEN-GIRL/wrx90-bios-mod: an AI-assisted mod of the ASUS Pro WS WRX90E-SAGE SE BIOS 1317 that adds a PCIE X8X8 Mode option for slots 1, 2, 3, 4, 5 and 7 (slot 6 is already x8). Flashed with USB BIOS FlashBack (16 GB USB 2.0 stick, MBR, one FAT32 partition, file A5496.CAP), it booted one board to Setup with the x8/x8 menu visible (2026-10-20). Not verified: selecting and saving x8/x8, link negotiation with two devices, P2P, LLM performance and stability. The capsule signature is invalid after modification, the repo has no licence, and its scripts must never be run with Python -O.
Contested 8 GB and 10 GB
Source: @okrasit., 2026-10-03
Source: @zefrench6961, 2026-10-03
Source: repo github.com/AnalyticETH/cmpunlocker by AnalyticETH on GitHub (fork by @analyticeth)
Source: @darkmaniac7, 2026-10-19 (lab notes)
Source: repo github.com/NA-DEGEN-GIRL/wrx90-bios-mod by NA-DEGEN-GIRL on GitHub, posted by @stcord, 2026-10-25
Topology and P2P
P2P through a PEX88096: the same bandwidth and latency within and across the switch's subtrees
cibernox runs 4 cap-modded cards at Gen2 x16 behind a PEX88096 (5 slots, 4 used) and got GLM-5.3-Flash running at TP=4 (2026-10-02). nvidia-smi topo -m shows PIX pairs GPU0-GPU1 and GPU2-GPU3 and PXB across them. He measured 6.69 GB/s and 1.74 µs for GPU0 to GPU1 (PIX), 6.69 GB/s and 1.75 µs for GPU0 to GPU2 (PXB), and 13.1 GB/s bidirectional, so crossing the switch's subtrees made no measurable difference; he estimated P2P improves performance by about 15%. replacedriver47 (root port Intel Device 4621) sees a different tree behind his PEX880xx, where GPU0 is PXB to every other GPU and only GPU2-GPU3 are PIX; his theory that P2P is fastest when GPUs share the same "substation" on the switch was not borne out by cibernox's measurement. Use lspci -tv and nvidia-smi topo -m to see where each card sits (PIX = at most one bridge, PXB = multiple bridges without crossing the host bridge).
Reported
Source: @cibernox, 2026-10-02
Source: @cute_sc2, 2026-10-02
Source: @replacedriver47, 2026-10-02
P2P latency per switch hop: nanoseconds, microseconds or milliseconds
Figures for switch latency disagree by orders of magnitude. Measured on cards: Manttis about 6 µs before and about 3 µs after a patch on an H12D-8D without a switch (2026-08-25); cibernox 1.74 µs (PIX pair) and 1.75 µs (PXB pair) behind a PEX88096 (2026-10-02). The claims:
- Jeremy (Ends): inside a switch about 8 to 15 ns and off the switch about 30x slower. Source: @endsjeremy, 2026-07-27
Source: @endsjeremy, 2026-07-31 - Zorg33: a switch has about 1/5 of the CPU's latency. Source: @zorg33, 2026-07-28
- bayley_w: a switch hop costs mid single-digit microseconds, irrelevant for pipeline parallel ("10 hops out of a 30 ms step"). Source: @bayley_w, 2026-08-12
- Jeremy (Ends): "5 to 15 ms" for P2P through a PLX card, and "200ms vs 15ms" per hop (units as posted). Source: @endsjeremy, 2026-09-04
Source: @endsjeremy, 2026-09-04
Source: @endsjeremy, 2026-09-11 - markshowa: P2P over the CPU root should be about 1.7 µs and about 1.4 µs on a crossbar; a Reddit setup reporting 14 µs looks like a different NCCL P2P path. Miséricorde: the hardware call itself is about 1.5 µs, across CMPs on native PCIe 2.0 x16 slots about the same; switches add latency. Gnasherrr: that Reddit user has a retimer plus two switches, so slower is expected, and a PLX hop is about 105 ns (Jeremy: around 100 ns). Measured in the same thread on real cards (satspace, Gen2 x16, static BAR1): 1.69 to 1.71 µs. Source: @markshowa, 2026-10-17
Source: @wakeupmbrc1, 2026-10-17
Source: @gnasherrr, 2026-10-17
Source: @endsjeremy, 2026-10-18
Contested 8 GB and 10 GB
Source: @manttis0264, 2026-08-25
Source: @cibernox, 2026-10-02
Switch placement: one 10x8 switch or two 5x16 switches
Arcanum: the 8 GB cards sit on a Gen2 x16 link anyway, and a switch avoids traversing the CPU, which is lower latency; quark2world: DGX systems use switches to hang NVMe off GPUs. endsjeremy claims a PLX hop costs far less than a CPU or PCIe root hop and a star topology has very little impact, while in dual 5-card servers people avoid cross-group traffic because it "murders performance"; for 10 cards he would put the root in front of two 5-card x16 groups or one 10-card x8 group. TP across the array is bandwidth-limited no matter the lane count (x8 per card is enough), and PLX shines for PP and P2P DMA. mrpandagod settled on 10x x8 on one switch and will add a second if bandwidth bottlenecks. Zorg33 is building the comparison (all ten at x8 on one PEX88096 against 5+5 at x16 on two switches on the same NUMA node) and leans toward the single switch being faster despite half the lanes. Earlier (2026-07-28, untested) Zorg33 ranked the options for the lowest TP latency: both switches joined to a third switch, then both switches on one CPU, and worst, switches on different CPUs.
Theory 8 GB and 10 GB
Source: @zorg33, 2026-07-28
Source: @endsjeremy, 2026-09-11
Source: @cibernox, 2026-10-02
Source: @replacedriver47, 2026-10-02
Source: @zorg33, 2026-10-06
Source: @endsjeremy, 2026-10-09
Source: @_arcanum_, 2026-10-09
Source: @quark2world, 2026-10-09
Source: @djpandaman, 2026-10-09
Source: @endsjeremy, 2026-10-18
Many cards at x4: CPU-lane bifurcation or a PLX switch?
The question was how to hang many cards at x4 (insomniac asking; Arcanum linked the Level1Techs "Poor(er) man's local AI" build thread). The answers disagree:
- SkirtChaser: if you stay at x4, a board with bifurcation (4x4x4x4 per x16 slot) is the most cost-effective, about $1000 for 16 cards at x4, and P2P still works through the CPU root complex with only a minor latency penalty; what breaks P2P is crossing CPU sockets or going through the chipset. From his MI50 build (stopped at 8 cards after testing 12) he says x4 chokes on tensor parallel and NVLink-class hardware is the real answer. Source: @skirtchaser, 2026-09-11
- Jeremy (Ends): without a PLX switch every card-to-card request goes through the CPU ("200ms vs 15ms", compounding per hop), so he prefers 8 lanes per card behind a PLX, though x4 off a PLX needs x4 risers and more setup. Source: @endsjeremy, 2026-09-11
Contested
Source: @_arcanum_, 2026-09-11
Source: @ins_mniac, 2026-09-11
Eight cards on one root: two cascaded PEX88096 boards, a PEX88144, or a dual-socket EPYC board
gnasherrr (6 cards, buying 2 more) asked how to attach 8 cards to an ASRock Rack ROMED8-2T. Miséricorde: a PEX88144 board runs 8 CMPs from one root port but is expensive and rare; cheaper are two PEX88096 boards (PCIe x16 host, 2x SFF-8654 8i per board, 4 cards each) in two host slots. Jeremy: with 16 lanes left on each PLX, cascade them so DMA reaches all 8 cards; a single 10-port switch instead of two cuts PLX cost by two thirds. Earlier research by gnasherrr and gpulover (2026-10-17): a PEX89144 is 3000 USD and up, an 11-card backplane needs 2x PCIe 4.0 x16 uplinks, and a PEX89048 feeding 2x PEX88096 would keep x16 for a future Gen4 unlock but may cost more than an EPYC 7763 with 256 GB DDR4. The sides on switches against a second socket:
- Schaka (2026-09-10): for x16 per card, use two PLX/PEX 88096 switches. Source: @schakaqt, 2026-09-10
- Takka: a dual-CPU EPYC board beats two switches, because switches bottleneck when talking to each other; by his count Rome's inter-socket link is worth 3 x16, so 4-card clusters get 3x16 between them against 1x16 between switches. This conflicts with SkirtChaser's report the same day that P2P fails across sockets (see P2P). Source: @spectakkular, 2026-09-10
- Danske: an 8-GPU rackmount chassis is the cleanest setup but deafening. Source: @danske1, 2026-09-10
Contested 8 GB and 10 GB
Source: @ins_mniac, 2026-09-10
Source: @gnasherrr, 2026-10-17
Source: @artistosu, 2026-10-17
Source: @gnasherrr, 2026-10-18
Source: @endsjeremy, 2026-10-18
Source: @wakeupmbrc1, 2026-10-18
Switch fan-out for 8 to 20 cards (ideas)
Arcanum: Broadcom Gen5 switches go up to 144 lanes (8 cards at x16 plus a 16-lane uplink) and Gen4 PLX up to 96 lanes (80 after the uplink). He owns a PEX88096 board with 10 SlimSAS ports and would need another PEX88096 plus a PEX88032 above them to address 10 cards, with full bandwidth inside a switch and x16 between switches, likely faster than CPU to CPU since CPUs make poor P2P hosts. Jeremy (Ends): stock parts can drive 10 cards at x8 for inference without all-to-all traffic, and 20 from one PLX chip at x4 using SlimSAS splitter cables and x4 device adapters. giaki3003: dual-socket NUMA defeats the purpose. Nobody had tested these layouts with CMPs.
Theory
Source: @_arcanum_, 2026-07-31
Source: @giaki3003, 2026-07-31
Source: @cute_sc2, 2026-07-31
Source: @endsjeremy, 2026-07-31
Housing 8 to 10 cards: dual PEX88096 or dual PEX8796 boards, server cases, frames
Asked for the best case for 8 to 10 cards (2026-09-09), OB88594 recommended "dual 88096" and Schaka said dual 88096 if you have x16, otherwise dual 8796 is fine (switch boards built on Broadcom PEX88096 and PEX8796 chips). segmond suggested an 8-GPU server case from eBay, or an Octominer chassis for budget tinkerers. Con3po is building a frame from 20x20 aluminium extrusion, and Schaka uses a wooden mining-rig frame with extra drilled holes.
Reported
Source: @.rickflare, 2026-09-09
Source: @ob88594, 2026-09-09
Source: @schakaqt, 2026-09-09
Source: @segmond, 2026-09-09
Source: @con3po7016, 2026-09-09
Housing and mounting a switch board and its risers
cibernox found his PEX88096 expansion baseboard close to mATX size, not a single-slot card, so housing it with a motherboard needs a server chassis, a large case with a mini-ITX host, or a 4U case about 650 mm deep (2026-09-12). Its mounting holes matched neither ATX standoffs nor a 4U chassis, so he ordered an open mining frame and planned to drill new holes. Jeremy (Ends) suggests magnets with an M3 thread so standard standoffs mount on any magnetic surface, keeping at least one real screw or a grounding pigtail to the frame and supporting the card's rear. mrpandagod's 10-port switch has 8 x16 risers; Takka noted the risers mount backwards relative to case brackets and are double-slot, so standard cases lack mount points, and suggested interlacing single-slot water-blocked cards (one normal, one upside down). Jeremy recommends miningeek-style open frames (rack-compatible, stack 10 tall, strong and cheap, but only about 415 mm side to side, so the largest server boards need the supports cut) and argues frames beat cases, which are loud and hot.
Reported
Source: @cibernox, 2026-09-12
Source: @djpandaman, 2026-09-12
Source: @endsjeremy, 2026-09-18
Source: @stcord, 2026-09-19
Source: @neapopa, 2026-09-23
Source: @spectakkular, 2026-09-23
Can a passive multi-GPU board be fed by several PLX switches?
mrpandagod linked a YouTube video titled "They finally made one..." about a multi-GPU board and asked whether it is just a passive board that could be used with multiple PLX switches, for a 16-card single-board idea (2026-10-02). Nobody answered.
Theory
Source: @djpandaman, 2026-10-02
Power and heat
The PEX88096 chip draws over 35 W: it needs a heatsink and airflow
lia_oc: the PEX88096 is a chip of more than 35 W, so it needs cooling. Most switch boards already ship with a heatsink and fan over the Broadcom chip; without one you need a good blower fan and heatsink. Passive cooling is possible with enough heatsink area if clearance allows, but airflow around it is advisable. Takka wants a near-silent water-cooled build and may machine a passive cooler; Arcanum suggested a small water block for the switch chip.
Reported 8 GB and 10 GB
Source: @lia_oc, 2026-09-29
Source: @lycorascence, 2026-09-29
Source: @_arcanum_, 2026-09-29
Source: @spectakkular, 2026-09-29
Kernel side
Several cards behind a PLX switch with 64 GB BAR1 need two host-kernel patches
Behind PLX PEX8747 or PEX8749 switches, the 64 GB BAR1 of several cards does not come up on a stock kernel. Patch 0001 fixes a pbus_size_mem() bridge-window sizing bug so that a 64 GB-aligned BAR1 plus a 32 MB BAR3 per card is budgeted; without it some GPUs silently get no BAR1, and a BAR1 resize behind a PLX switch cannot succeed. Patch 0002 makes the card's BAR1 come up at 64 GB during enumeration (the card ships with BAR1 at 64 MiB). Both need pci=hpmmioprefsize=2T and a large BIOS MMIO-high window; eight 64 GB BAR1s span 512 GB per root port. Build a -cmp kernel alongside the stock one and pin the kernel packages. club-170hx needed a related rebar-serialize patch (one mutex around the per-card BAR1 resize): without it two cards behind one PLX switch corrupted the kernel resource tree on every boot. okrasit.: the patches are only needed when cards sit behind a PCIe switch or bridge; cards directly on CPU lanes need only the kernel command line. Schaka needed them before BAR1 and P2P worked behind PLX8749 cards, and bayley_w's Supermicro 4028GR had the same BAR problem.
Proven 8 GB and 10 GB
Source: repo github.com/bayley/cmpunlocker by bayley on GitHub
Source: repo github.com/Schaka/170hx-journey by Schaka on GitHub
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub (posted in Discord by @c133 and @djpandaman)
Source: @darkmaniac7, 2026-09-12 (lab notes)
Source: @schakaqt, 2026-09-04
Source: @bayley_w, 2026-09-08
Source: @okrasit., 2026-10-03
Checking the patched kernel: every card needs a Region 1
bayley's README: the two patches are not needed for the memory, SM or Gen2 unlock, which all work on a stock kernel; they exist to make BAR1 P2P usable and to bring the large BAR1 up on a normal boot. They were built against linux-source 7.0.0 (Ubuntu 26.04) on a Supermicro SYS-4028GR-TRT2, first with four cards behind one PLX switch and later eight behind two. The failure without 0001 is silent: enumeration reports success and some GPUs simply have no Region 1. Check all of them, not just the first:
for d in $(lspci -D -d 10de: -n | awk '{print $1}'); do
echo "$d $(sudo lspci -vv -s $d | grep -c 'Region 1')"
done
When a pair looks slow, check the link width: one card trained at x2 instead of x4 and ran at exactly half speed everywhere, with zero AER errors (a physical riser or seating fault). The x4 reported against an x16 capability is normal on these cards; x2 is not.
Reported
Source: repo github.com/bayley/cmpunlocker by bayley on GitHub
Eight cards across two switches: P2P works, cross-switch at about 78% of same-switch
bayley's README: with eight cards, four behind each of two switches on the same CPU root complex, all 56 directed pairs pass a test with real peer reads and writes, including the 32 cross-switch pairs. nvidia-smi topo -m reports PIX inside a switch and PHB between them, so cross-switch traffic goes up to the host bridge and back. At Gen2 x4, same-switch pairs run at 1.68 GB/s and cross-switch pairs at 1.07-1.45 GB/s (about 78%), asymmetric by direction, which is a root-complex property. Keep tensor-parallel groups inside one switch where you can, but cross-switch is usable. NVIDIA's driver rates cross-root P2P reads unsupported on that Xeon E5 v4 host, yet they work; re-run the test on any new host rather than assume it generalises.
Reported
Source: repo github.com/bayley/cmpunlocker by bayley on GitHub
ACS redirect on the switch ports: turn it off for P2P bandwidth
List every port of every switch in pci=disable_acs_redir=..., including downstream ports with nothing plugged in yet, so a card added later needs no command-line change. Verify with lspci -vv | grep ACSCtl: every port should read ReqRedir- CmpltRedir-. To test without rebooting, setpci -s <port> ECAP_ACS+6.w=0000 applies it live. On bayley's host, same-switch P2P rose from 1.20 to 1.68 GB/s on the switch whose ports had been left out; cross-switch traffic was unchanged because it crosses the host bridge anyway. ropuls warned on 2026-10-01 that with ACS redirect on behind a PLX switch, peer traffic still goes up to the root complex. It matters less on direct-attach root ports: okrasit. runs two cards on root ports with ACS enabled in BIOS plus pcie_acs_override=downstream,multifunction and NCCL reported P2P in use, and club-3090 (RTX 3090s, not CMP) got all pairs working in a VM with ACS redirect left on at both root ports of a ROMED8-2T.
Proven 8 GB and 10 GB
Source: repo github.com/bayley/cmpunlocker by bayley on GitHub
Source: @darkmaniac7, 2026-10-25 (lab notes)
Source: @okrasit., 2026-09-23
Source: @ropuls, 2026-10-01
Open questions and small switch cards
PLX switch boards with CMPs: still an open question in August 2026
robbbbbb7973 considered two eBay PLX boards for 8 cards but noted they would bottleneck if anything above PCIe 2 is unlocked (2026-08-05). Studebaker asked whether PLX switch boards actually work or are an unstable nightmare; Arcanum answered "depends" and pointed to the Level1Techs "Poor(er) man's local AI" thread. On 2026-08-16 PJ asked whether anyone had used a PEX88096 board with multiple CMPs for LLMs; DeeJay had one on the way and nobody had results yet. OB88594: the H12D's SFF-8643 ports can feed a PLX 88096 board for 5 extra GPUs, but only over a PCIe 4.0 x16 uplink. ju5t5omedude test-fitted an 8-GPU PLX board in an Octominer chassis. Reports of CMPs working on add-in switch boards came later (MrManny 2026-09-30, domrockt and cibernox in October), and SkirtChaser's 4028GR already ran P2P across its built-in PLX chips from 2026-08-06.
Theory
Source: @robbbbbb7973, 2026-08-05
Source: @_studebaker_, 2026-08-08
Source: @_arcanum_, 2026-08-08
Source: @zingy90210, 2026-08-16
Source: @deejay9983, 2026-08-16
Source: @ob88594, 2026-08-17
Source: @ju5t5omedude, 2026-08-17
A PLX switch to scale card count on limited host lanes
DeeJay's case for a PLX switch: 5 GPUs share one x16 uplink to the motherboard while the cards talk to each other at full link speed. A Threadripper Pro has 128 lanes, so with the x16 mod you are capped at 8 cards, while a PLX setup uses only 32 host lanes (2026-08-05). He suggested it for MrManny's consumer B550 board (5950X, a 5090 plus two 8 GB cards) as cheaper than moving to server class. SkirtChaser: you need something like a PLX88096 or bifurcation cards. The MCIO riser board Wully posted is not a switch and needs companion cards plus MiniSAS or MCIO cables. No measured results were posted.
Theory
Source: @deejay9983, 2026-08-05
Source: @mrmannyatx, 2026-08-05
Source: @wully8888, 2026-08-05
Source: @skirtchaser, 2026-08-05
PLX cost per lane, and whether more than x4 per card is worth it
- Jeremy (Ends): as a rule of thumb a PCIe 4.0 PLX setup costs about $10 per lane, so at least $40 per card at x4, covering riser and cable and assuming an 88096 (10-port) or 88048 (6-port) chip, but not the PSU or host. He argues that more than 4 lanes per card is usually not worth the money for multi-card LLM work and that benchmarks push people to overbuild. Source: @endsjeremy, 2026-09-11
- mrpandagod puts DIY PLX plus riser at about $130 to $150 per card and calls the cap mod cheap by comparison. Source: @djpandaman, 2026-09-11
- insomniac, planning 8 cards at x8 or more (a 128-lane CPU or dual CPU), prefers not to limit the cards to Gen2 and weighs Gaudi2 as an alternative. Source: @ins_mniac, 2026-09-11
Contested
A PLX switch card instead of a passive bifurcation riser; cheap Dell and PLX8749 cards
Jeremy (Ends), 2026-09-04: a passive x16 to x8x8 bifurcation riser leaves bifurcation to the motherboard, and every GPU-to-GPU memory access goes through the CPU, while an active PLX switch card with the community P2P patch lets two GPUs access each other's memory directly. If a PLX switch costs the same as a passive card, buy the switch. Decommissioned Dell PLX cards used to sell for $40 to 50 but were about $100 by then; cheap AliExpress risers are Molex-powered, $20 to 30 more for 6-pin. A riser's Molex power does not feed the GPU; the card still needs its own power inputs (answering MrManny). Schaka noted those Dell cards are x16 with only 2 cables, and that he paid about 100 EUR each for PLX8749 PCIe 3.0 x16 to x4x4 cards; a 4-port card with real x16 risers costs much more.
Reported
Source: @endsjeremy, 2026-09-04
Source: @schakaqt, 2026-09-04
Source: @endsjeremy, 2026-09-04
Source: @mrmannyatx, 2026-09-04
A PLX switch card plus SlimSAS risers for boards that cannot bifurcate
Asked how to run two GPUs on an MSI X670E Gaming Plus WiFi, which cannot bifurcate, Schaka recommended an active PCIe switch instead of a new board: a PLX/PEX 8749 card turns x16 into x16+x16, connected to risers with SlimSAS (SFF-8654 8i) cables (2026-09-10). A 4x 8i card feeds 4 risers at x8, or 2 at x16 where a riser takes 2 cables. Used SlimSAS cables cost about $15 each via best offers and AliExpress riser boards $10 to $15; Jeremy (Ends) suggested a Dell PLX card plus 2 risers and 2 cables, each riser needing its own free 6-pin. Keep cables at or under about 50 cm (a cheap 55 cm one exists; longer can cause issues), and mount risers with M3-threaded magnets and standard standoffs instead of drilling.
Reported
Source: @licensedterrapin, 2026-09-10
Source: @schakaqt, 2026-09-10
Source: @skirtchaser, 2026-09-10
Source: @endsjeremy, 2026-09-10
Splitting one x16 slot: passive bifurcation needs board support, otherwise an active switch card
A passive x16 to 4x4 splitter only works if the motherboard supports bifurcation; otherwise an active switch card is needed (2026-08-17, cuddylac997's question). Schaka points to 10Gtek cards for older PCIe 3.0 and to the PEX88096; Merom Green notes the PEX88064 is a bit cheaper. Jeremy (Ends) offered his passive 4x4x4x4 bifurcating riser (about $450 new from ADT), which runs 4 cards at x4 from one x16 slot. matstatman asked whether a switch gives each card a full x16 link from one Gen4 x16 slot; it was not answered in that thread (see the PEX88096 entry above for the 5x16 mode).
Reported
Source: @cuddylac997, 2026-08-17
Source: @schakaqt, 2026-08-17
Source: @meromgreen, 2026-08-17
Source: @matstatman, 2026-08-17
Source: @endsjeremy, 2026-08-17
Cheap switch ideas: PEX 8749 cards, $24 Dell NVMe switch cards, NVMe x16 switches with adapters
For splitting one x16 into 2x x16 (Gen3) or 4x x16 (Gen4), ju5t5omedude says a PLX PEX 8749 card is probably the cheapest for 2x16, or just bifurcate x8x8 or x4x4x4x4 (2026-07-29). Jeremy (Ends) points to decommissioned Dell/EMC PowerEdge PCIe 4.0 NVMe fan-out cards (SlimSAS SFF-8654, from R750/R760 servers) at about $24, saying they will drive a riser; ju5t5omedude notes that card is dual 8i in an x16 slot, so bifurcation achieves the same. Humvee55 (2026-08-13) pointed to a LocalLLaMA thread on PEX88096 boards helping LLM inference and P2P, and to a scrappier option: an NVMe x16 PCIe switch card with NVMe-to-PCIe adapters. None was tested with a CMP in these threads.
Theory
Source: @rubbra_f, 2026-07-29
Source: @ju5t5omedude, 2026-07-29
Source: @endsjeremy, 2026-07-29
Source: @scallion_9883, 2026-08-13
Up to 8 cards: PLX switches against cheap x16-to-4x4 cards and server boards
Petar has 2 servers with Gigabyte MU72-SU0 boards and Xeon Platinum 8360Y CPUs that allow up to 8 cards, and considered 2x PLX88096 switches. Schaka argued that an eATX server board is enough and that a PLX gains little when traffic still crosses the PCIe bus; he bought a PCIe 3.0 x16 to 4x4 card for about 80 EUR on Amazon instead of a PLX at about 500 EUR, and named passive SFF-8654 risers as another option. He chose Cascade Lake over Ice Lake because PCIe 4.0 is useless at Gen2. giaki3003 suggested OCuLink.
Reported
Source: @spale777, 2026-08-20
Source: @schakaqt, 2026-08-20
Source: @giaki3003, 2026-08-20
Source: @giaki3003, 2026-08-26
Related
Risers and bifurcation, P2P, Resizable BAR and BAR1 64 GB, Kernel parameters and module options, Large rigs (8 or more cards), Parallelism over the PCIe link, Minisforum MS-01, ASUS Pro WS WRX80E-SAGE SE WIFI, ASUS Pro WS WRX90E-SAGE SE, Supermicro SYS-4028GR-TRT2, ASRock Rack ROMED8-2T, Huananzhi H12D-8D, Gigabyte G292-Z20, Octominer X12 Ultra builds, Fans and ducts, Power supplies and cabling, Price history, July to October 2026, Buying from China as a foreigner, Gds-nvme-patch, Platforms.