Consumer motherboards
From CMP Wiki
More actions
Reported 8 GB and 10 GB
What works on desktop, HEDT and workstation boards: which BIOS settings matter, how to split lanes, and how many cards fit. Server boards are on Platforms and Server BIOS setup.
What the board has to provide
- Above 4G Decoding. The card's BAR1 is a 64-bit prefetchable BAR; firmware places it above 4 GB. Treat Above 4G Decoding as required. Source: @darkmaniac7, 2026-10-29 (lab notes) The PixelML club's buying checklist says the same: the board must enumerate your slot/riser topology and support Above 4G Decoding. Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @djpandaman, 2026-10-24 (docs/HARDWARE.md)
- Room in the 64-bit MMIO window if you want the 64 GB BAR1 (needed for P2P, not for the 64 GB of VRAM): 64 GB + 32 MB per card, 64 GB-aligned. Four cards need more than 256 GB of window. Source: @darkmaniac7, 2026-10-29 (lab notes)
- Lanes from one CPU for the cards that should talk to each other (P2P) is the usual advice. Source: @wakeupmbrc1, 2026-10-16 It is not absolute: P2P passed across both sockets of a dual-socket EPYC server (Supermicro H12DSG-O-CPU). Proven Source: @darkmaniac7, 2026-10-29 (lab notes)
- Physical room: the card is dual slot, passive and long; see Cooling and power for airflow.
Above 4G Decoding and Resizable BAR
- The BIOS "Resizable BAR" option does not give you 64 GB by itself. At power-on the card only offers a 64 MB BAR1; the patched driver unlocks the larger sizes and asks the kernel to resize. Enabling ReBAR in the BIOS is harmless. What decides success is whether the bridge window above the card can grow to 64 GB. See Resizable BAR and BAR1 64 GB and Server BIOS setup case C. Source: @darkmaniac7, 2026-10-29 (lab notes)
- The 64 GB of VRAM does not need a big BAR1. Without it you lose P2P between cards (engines fall back to copies through host memory), not the memory unlock. Source: @darkmaniac7, 2026-10-29 (lab notes)
- MMIO window too small: a Minisforum MS-01 had a 256 GB MMIO window, too small for four cards, and tried to place the BAR1s inside a 512 MB section; Resizable BAR was already on. Changing MMIOH and BAR sizes later got 2 of 4 cards to 64 GB BAR1. Source: @replacedriver47, 2026-10-22
Source: @replacedriver47, 2026-10-22
Source: @replacedriver47, 2026-10-22
Source: @replacedriver47, 2026-10-24 See Minisforum MS-01. - Older boards hide the MMIOH settings: on an ASUS P9X79 WS the BIOS gives only a small window above 4 GB. Above 4G Decoding was there; changing kernel parameters fixed it for Fernando Delfini (exact parameters not posted). Source: @globalapathy, 2026-10-19
Source: @globalapathy, 2026-10-19
Source: @fdelfini, 2026-10-19 See ASUS P9X79 WS. - ASUS workstation boards: BAR1 could not be assigned ("no space") on an ASUS WRX80E (Trust Inter) and an ASUS WRX90 (GhostInTheDatacenter); Ghost's assistant ended up suggesting an ASUS support ticket for a BIOS change. Source: @trustinter, 2026-10-07
Source: @ghostinthedatacenter_87281, 2026-10-07
Source: @trustinter, 2026-10-10
Source: @ghostinthedatacenter_87281, 2026-10-11 Ghost later warned people buying new to watch out for ASUS boards because of BAR sizing. Source: @ghostinthedatacenter_87281, 2026-10-22 See ASUS Pro WS WRX80E-SAGE SE WIFI and ASUS Pro WS WRX90E-SAGE SE. - Intel W680: AnalyticETH (ASUS W680-ACE) made messy, beta progress on BAR1 in his cmpunlocker fork (AnalyticETH/cmpunlocker); Ghost tried it on his WRX90 without luck. Source: @analyticeth, 2026-10-07
Source: @ghostinthedatacenter_87281, 2026-10-07 - Older desktop CPUs and a PEX88096 switch: peter643 runs four cards in a PEX88096 switch enclosure on an i7-9700K host. The 9700K does not offer enough BAR1 address space, so he wrote his own P2PDMA path, a GPUDirect-Storage-style mailbox and his own nvidiafs driver that moves data through a 128 MB BAR region with several buffers; he says a newer CPU with more BAR1 address space would make this easier. Four Gen4 x4 NVMe drives on a cheap (about 50 USD) passive carrier in the same switch give about 24 GB/s sustained read, which already roughly fills the four cards' Gen2 x16 links. Summary of his write-up; see Discord. Source: @peter643, 2026-10-07
Source: @peter643, 2026-10-07 - gpulover reports BAR1 P2P on a board without a PLX switch. Source: @artistosu, 2026-10-17
- P2P without a big BAR1 Contested
- OB88594: the Morrowmake fork does P2P normally at a 64 MB BAR1. Source: @ob88594, 2026-10-11
- replacedriver47: it reported P2P working with no BAR1, but no transfers went through. Source: @replacedriver47, 2026-10-11
Lane splitting
- How many CPU lanes you have decides how many cards get x16. LGA2011 and 2011-3 (X79/X99) CPUs have 40 lanes, enough for two cards at x16; beyond that use a PLX switch or EPYC (Miséricorde). Source: @wakeupmbrc1, 2026-10-16 An X399 Threadripper does not have the lanes for 6 cards at x16 (djpandaman). Source: @djpandaman, 2026-10-22
- Unmodified cards only use x4 (lanes 4 to 15 need the PCIe x16 capacitor mod), so x4 slots, x4 bifurcation and M.2 adapters cost nothing until you mod. Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @djpandaman, 2026-10-24. See Risers and bifurcation.
- The switch route: "literally any computer and a 10 port PLX 88096 switch"; for full bandwidth the CPU should be Gen4 or newer (djpandaman). Source: @djpandaman, 2026-10-22 A PEX88096 carrier (this Alibaba listing, asked about by hawtt_dawgg for AM4/AM5) works on a Z690 with an i5-14600K (domrockt2). Source: @_syn_thetic_, 2026-10-22
Source: @domrockt2, 2026-10-22 - Bifurcation on consumer boards is often 4x x4 only (an X870 board with one Gen5 x16 slot). Options there: an M.2 NVMe to PCIe x16 extender cable (SinLoon, Amazon), a 3x Gen4 x16 switch card, or a retimer board. Source: @dottoreargus, 2026-10-07
Source: @gpmickey, 2026-10-07
Source: @matstatman, 2026-10-07
Source: @globalapathy, 2026-10-07 - Shared slots: many workstation boards share lanes between neighbouring slots, so a card drops to x8 when the next slot is filled (Supermicro X11SPA-TF, the PixelML club's seven-slot dual-socket board). Source: @djpandaman, 2026-10-09 See Supermicro X11SPA-TF.
- Intel ACS on X99 root ports blocks P2P: a two-card X99 report on the PixelML repo ran with
NCCL_P2P_DISABLE=1and found PP2 beat TP2 on every metric (decode 87.8 vs 69.7 tok/s, prefill 3.6 to 5.9k vs about 1.3k tok/s, Qwen3.8-Flash-Next AWQ W4A16 at Gen2 x4). Reported in PixelML/club-170hx issue #59 by PhillThomas and cited in docs/TOPOLOGY-AND-PARALLELISM.md: Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @djpandaman, 2026-10-24
How many cards fit
| Board or platform | Cards reported | Notes | Source |
|---|---|---|---|
| Huananzhi X99-F8 / X99-T8 | 2 | slot spacing will not seat 3; "best base for two CMPs" | Source: @endsjeremy, 2026-10-07 Source: @wakeupmbrc1, 2026-10-07 |
| ASUS X99-E WS | 4 | slot 5 never linked x16; another slot later dropped to x8 | Source: @matstatman, 2026-10-07 Source: @matstatman, 2026-10-16 |
| Supermicro X11SPA-TF | up to 7 slots | slots share lanes in pairs; a Xeon W-3200 (64 lanes) is needed to drive all 7 | Source: @djpandaman, 2026-10-09 |
| Threadripper (board not named) | 3 | clee; cross-root P2P never moved data on his older fork | Source: @c133, 2026-10-19 Source: @c133, 2026-10-22 |
| X399 | 6 at x4 | Luke Howard; unhappy with P2P, planning the x16 mod | Source: @lukextra, 2026-10-22 Source: @lukextra, 2026-10-22 |
| AM5 + cascaded PCIe switches | 6 | gnasherrr; no P2P before the switches | Source: @gnasherrr, 2026-10-17 |
| AM4 (board not named) | 2 | Dusty44; one card in the x8 chipset slot, no issues so far, not speed-tested | Source: @dusty.44, 2026-10-23 |
| Z690 + PEX88096 carrier | not given | domrockt2; works | Source: @domrockt2, 2026-10-22 |
| Minisforum MS-01 + PLX 88096 | 4 (replacedriver47), 8 (Con3po) | BAR1 allocation limited; Con3po used mailbox P2P | Source: @replacedriver47, 2026-10-22 Source: @con3po7016, 2026-10-22 |
| Desktop with an RTX 4080 Super (CachyOS) | 1 | works beside a gaming GPU | Source: @ybryany, 2026-10-14 |
| Dell OptiPlex 7040m micro PC (OcuLink) | 1 | works, host too weak | Source: @globalapathy, 2026-10-18 |
| Dual Xeon E5 v3/v4 board (Supermicro or Dell R720 class) | 3 native, 6 with passive risers at x8 | endsjeremy's planned rental builds | Source: @endsjeremy, 2026-10-21 Source: @endsjeremy, 2026-10-21 |
A household outlet limits a whole box to about 1600 W (15 A at 120 V), which caps card count before slots do for many home builds (endsjeremy). Source: @endsjeremy, 2026-10-21 See Cooling and power.
By platform
AM4, AM5 and Intel desktop
- AM4 tops out at 128 GB of RAM (quark2world). Source: @quark2world, 2026-10-09
- Dusty44 runs two cards on an AM4 board, one of them in the PCIe x8 slot wired to the chipset, with no issues so far; he has not speed-tested it. His board's large chipset heatsink has a fan. Source: @dusty.44, 2026-10-23
- gnasherrr runs AM5 with cascaded PCIe switches rather than moving to EPYC; before the switches he had no P2P. Source: @gnasherrr, 2026-10-17
- An i5-14600K on Z690 runs a PEX88096 carrier (Alibaba listing) fine (domrockt2). Source: @_syn_thetic_, 2026-10-22
Source: @domrockt2, 2026-10-22 - For 8 GB cards, djpandaman would not worry about Gen4 and would go Gen3, since the cards link at Gen2. Source: @djpandaman, 2026-10-12
Source: @djpandaman, 2026-10-12 - eGPU: a Razer TB3 enclosure on X299 did not work for orrorin (he blames X299 Thunderbolt). Source: @_orrorin, 2026-10-18
X99 and X79 (including Huananzhi)
- Miséricorde runs two cards on a Huananzhi X99-F8 (DDR4) and recommends the X99-T8 with an E5-2696 v3 and 256 GB DDR3 for cheap builds; he calls Huanan boards the best base for two 8 GB cards. Source: @wakeupmbrc1, 2026-10-07
Source: @wakeupmbrc1, 2026-10-07
Source: @wakeupmbrc1, 2026-10-07 See Huananzhi X99-F8. - For four cards on X99 you need a PLX board or switch card (X99-E WS, Huanan T8) (Miséricorde). Source: @wakeupmbrc1, 2026-10-18
Source: @wakeupmbrc1, 2026-10-18 - The ASUS X79 WS has a PLX switch (endsjeremy). Source: @endsjeremy, 2026-10-16
- Memory on Chinese X99 boards is fussy: they take UDIMM, RDIMM and LRDIMM, but you have to know the exact memory you will use; some boards struggle with 16 GB UDIMMs but run 32 GB LRDIMMs well. Stick to the most common boards (TD, TF, D6, D16) (endsjeremy). Source: @endsjeremy, 2026-10-19
Source: @endsjeremy, 2026-10-19
Source: @endsjeremy, 2026-10-19 - Memory rules for Chinese X99 boards (endsjeremy): all X99 boards support LRDIMMs; they struggle because the hacked-together BIOS lacks a full timing table, so you need exactly the right memory, preferably the same revision. The ceiling is 256 GB (LRDIMMs, 4x4 rank as he recalls), 32 GB per stick, and none of them can do 512 GB; with UDIMMs the electrical load is worse and they may only reach 128 GB. Buy the most common board and pay for the 12-layer versions with at least 6 VRM phases. In a follow-up he found that the CPUs supporting both DDR3 and DDR4 had no LRDIMM support at all. Summary of his posts; see Discord. Source: @endsjeremy, 2026-10-25
Source: @endsjeremy, 2026-10-25
Source: @endsjeremy, 2026-10-25
Source: @endsjeremy, 2026-10-25
Source: @endsjeremy, 2026-10-25
Source: @endsjeremy, 2026-10-25
Source: @endsjeremy, 2026-10-25
Source: @endsjeremy, 2026-10-26 - No X99 board can train 64 GB DDR3 sticks; single-socket 8-channel DDR3 X99 boards cannot train 32 GB sticks either (endsjeremy). Real LGA2011 server boards can, which is why 64 GB DDR3 LRDIMMs are cheap (about 300 RMB, Miséricorde); OB88594 warns 64 GB DDR3 has compatibility issues on some older Chinese boards. Source: @wakeupmbrc1, 2026-10-08
Source: @ob88594, 2026-10-09
Source: @endsjeremy, 2026-10-09
Source: @endsjeremy, 2026-10-09 - ElectroHobo got about half of Morrowmake's numbers on a Qwen Flash recipe and blames Broadwell CPUs; segmond saw a similar gap on the GLM recipe and blamed missing x16 and P2P instead. Theory Source: @segmond, 2026-10-20
Source: @nathang6866, 2026-10-20 - ASUS X99-E WS and ASUS P9X79 WS: see ASUS X99-E WS and ASUS P9X79 WS.
Threadripper, WRX80 and WRX90
- djpandaman's suggestions for a 6-card P2P build: a WRX80 board with Threadripper 3000/5000, an ROMED8-2T with SP3, or any computer with a 10-port PEX88096 switch. Source: @djpandaman, 2026-10-22
- BAR1 assignment failed on ASUS WRX80E and WRX90 boards (above).
- clee could not get P2P to move data across different root complexes on his Threadripper board (3 cards, an older fork without ECC); djpandaman suggested the current cmpunlocker with a local P2P patch. Source: @c133, 2026-10-22
Source: @c133, 2026-10-22
Source: @djpandaman, 2026-10-22 - Nestaa51 runs two cap-modded 8 GB cards with the P2P driver on a Threadripper with 128 GB DDR4-3200. Source: @nestaa51, 2026-10-24
Source: @nestaa51, 2026-10-24 - WRX80 fan headers are rated 12 V 2 A per HenryKrinkle; Trust Inter thought 1 A. Source: @trustinter, 2026-10-25
Source: @.henrykrinkle, 2026-10-26 - The WRX90 BIOS mod adds x8/x8 bifurcation (untested with CMPs). Theory Source: @stcord, 2026-10-25
Mini PCs and eGPU
- Minisforum MS-01: see Minisforum MS-01.
- Dell OptiPlex 7040m with OcuLink from M.2: see Dell OptiPlex 7040m. Newer micro PCs with native OcuLink are likely better (GlobalApathy). Source: @globalapathy, 2026-10-18
Workstations
- Dell Precision T5810: needs unusual power cables, fan PWM locked on Linux, tight front clearance. See Dell Precision T5810.
- Dell Precision T7600 was suggested for GPUs (djpandaman); no CMP report. Source: @djpandaman, 2026-10-09
- A Lenovo C30 (2x E5-2690 v2, 128 GB DDR3) was suggested as a two-card host; no result posted. Theory Source: @shudza, 2026-10-07
Source: @wakeupmbrc1, 2026-10-07
References
- GlobalApathy points to a write-up on running an A100 80GB PCIe (same GA100 board family) on a consumer motherboard with custom water cooling: stasosphere.com. Source: @globalapathy, 2026-10-12
Display output
Run the box headless. Older display cards are not supported by the driver branches the unlock uses: a GT 1030 or GT 710 gets only 1024x768 basic output; other cards in the host must be Turing or newer. Use the board's BMC VGA if it has one. Source: @68x0, 2026-10-12
Source: @lia_oc, 2026-10-12
Source: @lenxtroz, 2026-10-13 See Operating systems and kernels#Mixed GPUs in one host.