Parallelism over the PCIe link
More actions
The CMP 170HX has no working NVLink, so every multi-card setup talks over PCIe: Gen1 x4 on a stock card, Gen2 x4 after the Gen2 link retrain, Gen2 x16 with the PCIe x16 capacitor mod. Tensor parallel (TP) splits every layer across cards and runs all-reduces inside each layer, so it is limited by that link. Pipeline parallel (PP) gives each card a block of layers and passes one hidden state per stage boundary, so it barely notices the link. Expert parallel (EP) spreads MoE experts over cards; data parallel (DP) runs one full copy per card. On stock x4 links owners run one model per card or PP. On x16 links with P2P, TP4 or TP2 x PP2 can win single-user decode on some models while PP keeps the prefill lead. Where owners disagree, both sides are below with their numbers.
TP or PP
TP or PP on PCIe-only cards: TP for single-stream decode on x16, PP for prefill and x4 links
Contested 8 GB and 10 GB
- July 2026, at Gen1 x4: Arcanum and Landmark judged that TP would choke on the link and pointed to PP. Source: @_arcanum_, 2026-07-19
Source: @landmark0700_17532, 2026-07-20 - August 2026, PP becomes the default: allover326's DeepSeek V4 Flash repo measured PP4 prefill at 5,321 tok/s against 801 for TP4 at 77k on Gen2 x4 without P2P; DeeJay found PP 3 to 5x faster than TP on average; matstatman's vLLM TP2 at Gen2 x4 prefilled at about a third of one card. Late August, Miséricorde: no working TP beyond 2 soldered cards for MoE models at that time. Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub, posted by @deejay9983, 2026-08-05
Source: @deejay9983, 2026-08-08
Source: @matstatman, 2026-08-17
Source: @wakeupmbrc1, 2026-08-30 - September 2026, for TP: Takka found TP4 the best arrangement even on x4 links with the Ampere vLLM implementation. Ezy measured TP4 for GLM-5.3-Flash with all cards at Gen2 x4 at 231.6 counting, 225.4 code and 159.0 prose tok/s. Source: @spectakkular, 2026-09-25
Source: @ezybr0, 2026-09-29 - 2026-10-04, for PP: Morrowmake expected TP4 on Gen2 links to give acceptable decode but unusable prefill. 1a2b3c0967 tested TP4 and agreed: PP4 is better because TP4 prefill is too poor, and the "feels snappy" impression comes from prefill. Source: @voidgiraffe, 2026-10-04
Source: @1a2b3c0967, 2026-10-04 - 2026-10-06, for PP: Plean: TP has repeatedly been shown to be limited by PCIe bandwidth on these cards, so switching to PP may raise single-user speed. turbine: the cards' link is slow (Gen2) and most runtimes assume NVLink. Source: @fiiireface, 2026-10-06
Source: @plean., 2026-10-06
Source: @turbine, 2026-10-06 - For PP, the club-170hx reasoning: TP does two all-reduces per layer (about 86 collectives per forward pass on a 43-layer model), while PP moves one hidden-state hand-off per stage boundary, about 28x less data. Its recipe of record for GLM-5.3-Flash is PP4 with native MTP k=3 at 87.55 tok/s for one stream, against 60.4 for TP4. Source: repo github.com/PixelML/club-170hx by PixelML on GitHub
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub - 2026-10-14, two cards: Miséricorde: TP2 usually beats one card, depending on x16 or x4. programmerboi: use PP2 for the best batched throughput without x16, though "even with x16 it's probably not great". cibernox prefers TP4 with the x16 mod and suggests TP2 x PP2 for owners stuck on x4. Source: @programmer_007, 2026-10-14
Source: @wakeupmbrc1, 2026-10-14
Source: @programmerboi, 2026-10-14
Source: @cibernox, 2026-10-14 - 2026-10-15, for PP: Mountains got P2P working, tried TP and went back to PP because prefill on very long contexts works better for him. Source: @mountains1605, 2026-10-15
- 2026-10-18, for TP: Miséricorde (on 4 cards since that day) finds GLM 5.3 Flash faster in TP4 than in PP4 or TP2 x PP2 on PCIe-only CMPs, because the workload is latency bound, not bandwidth bound. He estimates NVLink across four cards would add roughly 20 to 25 %, and says TP4 viability depends on the model architecture. Source: @wakeupmbrc1, 2026-10-18
- 2026-10-18, for PP: Jeremy (Ends): a dense model in TP over PCIe will be bandwidth-choked, so PP with P2P DMA is the safer default. Source: @endsjeremy, 2026-10-18
Source: @globalapathy, 2026-10-18
Source: @xanthefin, 2026-10-18 - The lab's GLM-5.3-Flash numbers at Gen2 x8 without P2P: TP4 decodes 1.6 to 1.9x faster than PP4 for one user; PP4 prefills 2.8 to 3.9x faster than TP4. Source: @darkmaniac7, 2026-10-19 (lab notes)
The full layout comparison for that model is on GLM 5.3 Flash on 4x 8 GB.
Before the Gen2 unlock (July to early August 2026): TP judged to choke at Gen1 x4
Contested 8 GB and 10 GB
- 2026-07-19: Thanos doubted TP at PCIe Gen1 x4 (he quoted 800 MB/s; Luna quoted about 250 MB/s, the per-lane PCIe 1.0 figure). Arcanum: TP would choke on Gen1 x4, use PP. Black Jesus: PCIe speed does not matter unless you run TP. coolgod relayed an AI estimate that TP is latency bound, not bandwidth bound. ppc64_ expected large MoE models to behave with good expert affinity. Source: @coolgod_72592, 2026-07-19
Source: @thanos8912, 2026-07-19
Source: @lunaroseoc, 2026-07-19
Source: @_arcanum_, 2026-07-19
Source: @ashtray9843, 2026-07-19
Source: @ppc64_, 2026-07-19 - 2026-07-20: Landmark: TP is the problem on this link, and PP avoids it at a speed cost. Luna: 3090s only have useful NVLink between 2 cards, so a 3-card 3090 build needs PCIe 4.0 as well, which raises its platform cost. quark2world ranked bandwidth needs TP >> EP > PP > DP: DP runs fine on these cards, TP2 might work on x16-modded cards, and scaling beyond that would be poor. roboalchemist (a CMP 100-210 user): PP decode is fine but prefill "kills you". Source: @landmark0700_17532, 2026-07-20
Source: @robbbbbb7973, 2026-07-20
Source: @lunaroseoc, 2026-07-20
Source: @quark2world, 2026-07-20
Source: @roboalchemist., 2026-07-20 - 2026-07-21: arnold measured about 1.5 GB/s of PCIe traffic for TP between a 5060 Ti and a 3060, and said Gen2 (about 8 GB/s) would make multi-card 170HX setups feasible. Source: @a.civardagezen, 2026-07-21
- 2026-07-22: Samuel Hayden: PP gives roughly single-GPU performance and tolerates low bandwidth, while TP on 2 or more cards at Gen1 x4 loses more on prefill than it gains on decode. chris.via.egnatia: for 2 to 4 cards bandwidth is largely irrelevant for regular inference. giaki3003 and Wouter: TP scales with bandwidth, which is the reason to chase faster PCIe and NVLink. Source: @samuel_hayden., 2026-07-22
Source: @chris.via.egnatia, 2026-07-22
Source: @giaki3003, 2026-07-22
Source: @woezie, 2026-07-22
Source: @_arcanum_, 2026-07-22 - 2026-07-23 and 24, single cards: a model that fits in VRAM runs well and very fast once cached (Amogh); roger_dammit estimated about 30 s to load, then normal prefill and decode unless several GPUs force a tensor split. SkirtChaser cited a vLLM benchmark on 2x RTX 3090: 715 tok/s output and 6,790 tok/s throughput with NVLink, 483 and 4,583 without, so the interconnect matters for TP but not for default llama.cpp. Merom Green and arnold asked for TP tests on x16-modded 170HX pairs, which this period did not produce. Source: @grimmerfin, 2026-07-23
Source: @_j3ns, 2026-07-23
Source: @roger_dammit, 2026-07-23
Source: @meromgreen, 2026-07-24
Source: @skirtchaser, 2026-07-24
Source: @rogersmith2026, 2026-07-24
Source: @diabeetee, 2026-07-24 - 2026-07-26, from his own testing: gk_hun found Gen1 x4 (about 0.85 GB/s) not enough for any tensor or pipeline parallelism, PCIe 2.0 x16 the minimum for "ok" parallelism, and single-GPU inference fine at Gen1 x4 with a model that fits. Source: @gk_hun, 2026-07-26
- 2026-08-03: Moliminous called multi-GPU with these cards "nearly a doa proposition" without faster PCIe or NVLink. giaki3003 and gpulover replied that PCIe is only an issue for TP and pointed to the 4-card DeepSeek V4 result; formatme said the cards are slow even at Gen2; machinezero expected the link limit to dampen demand. Source: @moliminous, 2026-08-03
Source: @giaki3003, 2026-08-03
Source: @artistosu, 2026-08-03
Source: @counterstrikeaimgod, 2026-08-03
Source: @machinezer0, 2026-08-03
The later PP runs on Gen1 and Gen2 x4 rigs on this page show multi-card PP working on these links.
Late July and August 2026: PP becomes the default, TP2 still argued, MTP first needed TP
Contested 8 GB and 10 GB
- 2026-07-26: btakk cited a dual RTX 3090 test where enabling P2P without a PCIe switch raised inter-card bandwidth by no more than 10 % and gave no model speedup. Samuel Hayden and Merom Green: P2P matters for TP, which is bandwidth bound even at Gen4 x16, while PP behaves like one GPU with more concurrency. robbbbbb: the real P2P win shows up when a PLX switch board routes the lanes and you run TP. Source: @btakk_ak47, 2026-07-26
Source: @samuel_hayden., 2026-07-26
Source: @meromgreen, 2026-07-26
Source: @robbbbbb7973, 2026-07-26 - 2026-07-30: DeeJay found PP much better on prefill and TP worse in almost every regard, except that most models in the engines he used needed TP for MTP; at about 8 GB/s interconnect he saw equal prefill but faster TP generation, and PP with MTP played poorly together. programmerboi: TP suits large dense models on fast interconnect (SXM or OAM); when interconnect bound, "which you 100% are", multi-stream PP wins and TP on MoE could be about 4x slower. bayley_w agreed (on his SXM system TP was only marginally faster than EP) and said MTP with PP was simply not implemented yet. ju5t5omedude measured Qwen 27B MTP Q4 at 36 tok/s with PP against 46 with TP (LM Studio, Gen2 x4). DeeJay later got MTP working under PP for GLM 5.2 and for DeepSeek with DSpark, which removed the main argument for TP on these cards. Source: @deejay9983, 2026-07-30
Source: @programmerboi, 2026-07-30
Source: @bayley_w, 2026-07-30
Source: @ju5t5omedude, 2026-07-30 - 2026-08-01 to 08-05: makesier: more cards keep decode about the same or slightly worse; PP helps prefill, not decode. programmerboi: for a large MoE use expert parallel for a single stream and PP for multi-stream hosting, run the Gen2 unlock because the x4 link is the bottleneck, and skip TP. arnold argued TP2 may still work (Gen2 x16 and Gen4 x4 have the same paper bandwidth but maybe different latency; he saw good TP2 results for Qwen 3.6 27B Q8 on two 5060 Ti with P2P under vLLM). ju5t5omedude measured LM Studio's tensor-parallel option about 20 % faster than PP for a 27B on two x4 cards, and thinks Gen3 x8 would be fine for TP. SealClubber wants Gen2 x16 per card; Bart: PP is much faster given the limited PCIe. Source: @makesier, 2026-08-01
Source: @programmerboi, 2026-08-02
Source: @a.civardagezen, 2026-08-03
Source: @ju5t5omedude, 2026-08-03
Source: @sealclubber, 2026-08-04
Source: @justbendev, 2026-08-05 - 2026-08-08 and 08-13: weicj asked for vLLM or SGLang TP against PP throughput at Gen2 x16, and tchelobh for vLLM PP over two Gen2 cards; neither was answered in the thread. Source: @tchelobh, 2026-08-13
Do not use tensor parallel on stock x4 links: one model per card, or pipeline parallel
turbine: Qwen3.6-27B fits on one card, so two cards should scale linearly with zero inter-card traffic; do not use TP, and if a model must be split, use PP. DeeJay tested both in depth on several models and found PP 3 to 5x faster than TP on average. On stock x4 links (no capacitor mod) his TP runs were worse than PP in every category; the cap mod makes little to no difference under PP and only helps if you plan to run TP, which is why the 8-card GLM 5.2 setup he cited ran at x4. Josh gave the same advice for DeepSeek. Fats: filling a 64 GB card over PCIe 2.0 x4 takes about 30 s, so slow PCIe is fine as long as you avoid TP. segmond countered that slow PCIe matters more than people think for very large models, and keeps cards with faster PCIe for prefill. arnold suggested trying TP2 with PP2, which that thread did not test.
Proven 8 GB and 10 GB
Source: @turbine, 2026-08-08
Source: @deejay9983, 2026-08-08
Source: @_____josh_____, 2026-08-08
Source: @c.03937, 2026-08-08
Source: @segmond, 2026-08-08
Source: @deejay9983, 2026-08-24
Source: @_datacat_, 2026-08-24
Source: @a.civardagezen, 2026-08-24
vLLM TP2 over Gen2 x4 prefilled at about a third of one card (Qwen 27B FP8)
matstatman ran Qwen 27B FP8 in vLLM with --tensor-parallel-size 2 on PCIe 2.0 x4 links and concluded it is better to use one card. He posted two runs of a 42,584-token prompt without labelling which was which: 24.10 s, 1,767.3 tok/s prefill (repeat 24.33 s, 1,750.4), and 72.68 s, 585.9 tok/s (repeat 72.80 s, 585.0). Given his conclusion the fast line is most likely the single card and the slow line TP2; that labelling is an inference from his text. On x4 links, keep a model on one card when it fits.
Reported
Source: @matstatman, 2026-08-17
TP on stock Gen2 x4 against the x16 mod (late August to September 2026)
Contested 8 GB and 10 GB
- 2026-08-17: SkirtChaser: two cards are only somewhat practical unless both have the capacitor mod. With PCIe 2.0 x16 from the mod, Gemma 4 31B Q8 decode in llama.cpp rose from about 26 to about 40 tok/s across two cards, so link width matters when a dense model is split. Source: @skirtchaser, 2026-08-17
Source: @dharmendra., 2026-08-17 - 2026-08-19: Petar asked whether anyone runs 8 cards through PCIe switches with TP. giaki3003: do not use TP unless the model is very small or the config special; with PP the link width makes no difference, and a switch cannot fix TP, only NVLink or faster link training could. CUTE recommends PP. Con3po's AI estimate for a 3 TP x 2 PP layout gave only about 1.15x. Waka324: the interlink matters less for these cards. Petar's takeaway: 4 cards are enough for DeepSeek V4 Flash without TP. Source: @spale777, 2026-08-19
Source: @giaki3003, 2026-08-19
Source: @skirtchaser, 2026-08-19
Source: @cute_sc2, 2026-08-19
Source: @con3po7016, 2026-08-19
Source: @cmdrwaka324, 2026-08-19 - 2026-08-26 and 27: Zombie measured a 256 MB all-reduce at 3.8 GB/s on x16-modded 170HXs, against about 10 GB/s GPU-host-GPU on his 3090s. Ryko: Gen2 x16 is the floor for TP and 8 GB/s unidirectional works (rule of thumb: PCIe 3 is about 1 GB/s per lane, halve it for Gen2); llama.cpp's
-sm tensormode works but is slower over Gen2 x4. DeeJay: TP is bad on these cards and the x16 mod buys little (best case with P2P: prefill equal to PP and maybe 10 % more generation); the main gain would be not needing custom vLLM forks with 22 patches. Source: @zombielurker, 2026-08-26
Source: @vektor_prime, 2026-08-26
Source: @deejay9983, 2026-08-27 - 2026-08-29: Dino: the cards can only layer-split until PCIe 4.0 or NVLink is unlocked. matstatman got no TP2 speedup at Gen2 x4. segmond: people get 3,000 to 5,000 tok/s prefill and 100+ tok/s decode on huge models anyway. gym runs TP2 at Gen2 x16 with P2P, and PP is still faster for prefill. Source: @dino_38975, 2026-08-29
Source: @matstatman, 2026-08-29
Source: @segmond, 2026-08-29
Source: @jimtjames, 2026-08-29 - 2026-08-30 to 09-11, Miséricorde: TP on stock Gen2 x4 is very bad and still bad after soldering the slot to x16, and Gen2 latency hurts too. Denser models (for example 120B-A30B) suffer less because they sync less; TP2 may be comparable to PP2; 4-way TP is hopeless unless the model is dense; there is no working TP beyond 2 soldered cards for MoE models. Even at x16, TP barely helps Qwen 27B and costs a lot of prefill on Qwen Flash Next. Source: @realfreaky, 2026-08-30
Source: @programmer_007, 2026-08-30
Source: @wakeupmbrc1, 2026-08-30 - 2026-08-31 and 09-01: OB88594: low interconnect bandwidth for prefill limits concurrency. ju5t5omedude: the cards can tensor split, but not beyond 4 cards. Source: @ob88594, 2026-08-31
Source: @ju5t5omedude, 2026-09-01 - 2026-09-02 to 09-04, for PP: clee and Petri Krohn say PP works well and beats one card on prefill even for a model that fits on one. Morrowmake: fine to about 4 cards; beyond that Gen2 bandwidth dominates, and 8 DGX Sparks would beat 16 CMPs for about 1 TB. Source: @zagitta, 2026-09-02
Source: @c133, 2026-09-02
Source: @petrikrohn_26497, 2026-09-02
Source: @voidgiraffe, 2026-09-04 - 2026-09-06: gym found PCIe 2.0 x16 not quite fast enough for TP to be definitively better than PP on his 2 cards, and expects 3.0 x16 to tip it. segmond: Gen3 x16 allows TP gains, while TP over 2.0 x4 would saturate and likely be slower. reala: with only x8 links he would need Gen4. Source: @jimtjames, 2026-09-06
Source: @segmond, 2026-09-06
Source: @reala7096, 2026-09-06 - 2026-09-08 to 09-11: 大乐 and OB88594: TP is far too slow now and probably needs Gen4 to catch PP prefill. Josh: 2-card TP vastly beats 3-card, because every extra PCIe card hurts the all-reduce. Schaka finds TP decode on DeepSeek V4.1 fine (more KV, some prefill cost). Source: @dale063629, 2026-09-08
Source: @spectakkular, 2026-09-09
Source: @_____josh_____, 2026-09-09
Source: @schakaqt, 2026-09-11
TP divisibility
vLLM TP must divide the attention head count: buy 4 or 6 cards, not 5
smd386 asked whether a fifth card helps DeepSeek V4.1 Flash. OB88594: vLLM TP cannot be an odd number, so buy 4 or 6, not 5, and the constraint goes beyond attention layers (ColumnParallelLinear, num_experts). greencandle: vLLM requires TP to divide the model's attention head count, so DeepSeek V4.1 Flash works only at TP 1, 2, 4 and 8. cibernox recalls TP3 working on models with 24 attention layers but not 28; very few models divide by 5. intervitens: a sixth card buys native-precision V4.1 Flash and TP2 x PP3, since PP and TP combine (see DeepSeek V4.1 Flash on 6x 8 GB (Deepinfer-SM80 TP2 x PP3)). dagb: llama.cpp can send the mmproj and speculative decoding to a separate device, so a fifth card can still be used while the main model stays split on 4. Gnasherrr: a fifth card is probably no benefit, since more PP stages are slower. wgkbfcp's TP6 recipe sidesteps the rule by padding 64 heads in 8 groups to 96 in 12 with duplicated groups (DeepSeek V4.1 Flash on 6x 8 GB (TP6)). Earlier (2026-09-10), CUTE: TP wants even divisions (2, 4, 8, 16); 3, 6 and 10 cards give worse results, though 3, 6 and 12 can work with tricks, and he does not think a 10-card TP5 x PP2 layout is good.
Proven 8 GB and 10 GB
Source: @cute_sc2, 2026-09-10
Source: @con3po7016, 2026-09-10
Source: @schakaqt, 2026-09-10
Source: @gnasherrr, 2026-10-20
Source: @smd386, 2026-10-21
Source: @ob88594, 2026-10-21
Source: @greencandle9753, 2026-10-21
Source: @cibernox, 2026-10-21
Source: @nathang6866, 2026-10-21
Source: @intervitens, 2026-10-21
Source: @da6b, 2026-10-21
Layer partitions
In vLLM, VLLM_PP_LAYER_PARTITION sets how many layers each pipeline stage holds: one entry per stage, summing to the model's layer count. The last stage usually also carries the output head and the draft model.
Unload the last pipeline stage to grow the KV pool
In vLLM pipeline parallel the KV pool is limited by the last pipeline stage, which also carries the drafter and lm_head, so moving layers off that stage beats any compression trick. On GLM 5.3, Con3po's 22,20,20,16 split at --gpu-memory-utilization 0.98 gave 294K KV tokens, against 71K with 20,20,20,18 at 0.96, at the same speed.
Reported
Source: @con3po7016, 2026-09-08
DeepSeek V4 Flash 0731: 12,12,12,7 on 4 cards, 15,15,13 on 3
allover326's DeepSeek V4 Flash 0731 recipe (PP4 with DSpark, Gen2 x4, no P2P) sets VLLM_PP_LAYER_PARTITION=12,12,12,7. Pipeline rank 3 alone carries lm_head and the DSpark drafter; moving layers off it removes an 8.7 GiB imbalance between ranks and grows the KV pool by about 85 % (798,660 to 1,476,563 tokens at a max-model-len of 163840). It does not raise the context ceiling. The list needs one entry per pipeline rank and must sum to the model's 43 layers. On 3 cards 15,15,13 is required, not optional: the default 15,14,14 fails during the Marlin FP4 expert repack. Recipes: DeepSeek V4 Flash 0731 on 4x 8 GB (allover326 PP4 DSpark), DeepSeek V4 Flash 0731 on 3x 8 GB (PP3 DSpark).
Reported 8 GB
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub, posted by @deejay9983, 2026-08-05
Qwen3.8 Flash Next FP8: an even 12,12,12,12 on 4 cards, and size each rank from measured totals
zebgop-ops' recipe (Qwen3.8-Flash-Next-FP8 on 4 cards at Gen2 x4, no P2P, PP4 with MTP 3) runs an even 12,12,12,12 split with per-rank KV budgets (VLLM_KV_CACHE_MEMORY_RANK0 to RANK3) and full CUDA graphs: KV pool 2,537,837 tokens, decode 89 tok/s for one stream and 429-433 tok/s aggregate at 8, prefill 11.1-11.3k tok/s on a 32k prompt. With piecewise graphs the same split decoded only 59 tok/s against 70-74 on PP3 (16,17,15): the extra stage cost about 9.4 ms per decode step because each stage is CPU-dispatch bound, and FULL_AND_PIECEWISE fixed it. Unloading the last rank buys nothing on this model: 13,13,13,9 is pointless because the drafter forwards cost under 1 ms. On 3 cards the repo keeps 16,17,15 (capacity first, 1,359,009 tokens) and 16,16,16 (throughput first, 1,116,591). Size a partition from measured per-rank totals plus about 1.6 GiB of repack scratch: an over-committed rank faults at load with an illegal memory access instead of a clean out-of-memory error and wedges the GPU firmware (GspRmAlloc failed status=0x62) until a reboot. Never reuse one partition's per-rank budgets for another. Recipe: Qwen3.8 Flash Next on 3x or 4x 8 GB (zebgop-ops FP8 PP).
Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub, posted by @flipnburner, 2026-08-29
TP2 x PP2 and TEP
TP2 x PP2 on 4 cards: between PP4 and TP4, prefill over 5k tok/s on x16
cibernox added a TP2 x PP2 variant to Morrowmake's GLM-5.3-Flash recipe on 4 cap-modded (x16) cards behind one PCIe switch: generation is slower than TP4, but prefill passes 5k tok/s, so it sits between PP4 and TP4. With the x16 mod he still prefers TP4, but suggests TP2 x PP2 may be the best topology for owners stuck on x4 links; Morrowmake wants to see it measured on x4 links and lists it among community recipes he does not support. Earlier (2026-09-30 to 10-02): reala said the TP2 x PP2 layout cuts prefill by about a third and raises decode; cibernox's first, unoptimized test landed between PP4 and TP4 on both. His real single-stream decode is near 300 tok/s (292) on a model of about 320B; other figures in that message were illustrations, not measurements. Morrowmake prefers to maximize decode, because thinking models spend most of their wall time generating. cibernox: if NVLink were ever unlocked, TP2 over NVLink plus PP2 over PCIe would be the gold standard. The lab reproduced the shape on Gen2 x16 with P2P: TP2 x PP2 gave 171 / 146 / 98 tok/s for one user (code / structured / prose), 567 / 503 / 383 aggregate at 8 users and 4.2k tok/s prefill (4.5k on a 49k prompt), and passed a teacher-forced quality gate against TP4. Its recommendation: TP2 x PP2 for long-prompt, short-answer coding-agent traffic, TP4 for chat. Recipe: GLM 5.3 Flash on 4x 8 GB.
Reported
Source: @reala7096, 2026-09-30
Source: @cibernox, 2026-10-02
Source: @voidgiraffe, 2026-10-02
Source: @cibernox, 2026-10-07
Source: @voidgiraffe, 2026-10-08
Source: @darkmaniac7, 2026-10-28 (lab notes)
PP4 x TEP2 with straddled TP pairs on 8 cards; custom all-reduce needs expandable_segments off
On 8 cards Con3po runs PP4 x TEP2 (four pipeline stages, each a two-card tensor and expert parallel pair) with straddled pairs (0,4) (1,5) (2,6) (3,7) and vLLM's custom all-reduce, instead of PP8 x TP1: decode +36 %, KV capacity +56 %, prefill -38 %. Trap: custom all-reduce needs PyTorch expandable_segments off; otherwise CUDA IPC handles cannot be exported from VMM memory and every pair dies with "invalid argument". He has not tested the FULL_AND_PIECEWISE cudagraph mode, because it needs custom all-reduce off.
Reported
Source: @con3po7016, 2026-09-08
P2P: helps or hurts
Bare-metal P2P made TP4 slower on one host, about 10 % faster or 40 % faster on others
Contested 8 GB and 10 GB
- club-170hx (4 cards behind two PLX switches, Gen2 x16, Broadwell host): static-BAR1 P2P reaches 5.79 GB/s on all pairs and gives Qwen 27B TP2 +11 %, but GLM-5.3-Flash TP4 decode gets 20 to 48 % slower with P2P on (26.1 against 19.4 ms per step). Source: repo github.com/PixelML/club-170hx by PixelML on GitHub
- 2026-10-01: Morrowmake saw the opposite on an EPYC host: the recipe's published TP4 numbers were taken with P2P on, though a no-P2P configuration is supported and should still be fast. Impactic measured the recipe with P2P off at 394.0 counting / 377.4 code / 180.5 prose tok/s and with P2P on at 437.4 / 404.2 / 198.6, roughly 10 % better. Source: @voidgiraffe, 2026-10-01
Source: @impactic, 2026-10-01 - 2026-10-02: cibernox (4 x16-modded cards behind a PEX88096 switch) tested P2P over the switch: single-user decode structured 416 to 459 tok/s (+10 %), code 287 to 292 (+2 %), prose 189 to 205 (+9 %); 8-user aggregate 854 to 986 (+15 %), 679 to 785 (+16 %) and 546 to 639 (+17 %); cold prefill unchanged (8k 3,187 to 3,221, 32k 3,339 to 3,350, 128k 3,232 to 3,244, 250k 3,085 to 3,099 tok/s). His engine and layout are not stated. Source: @cibernox, 2026-10-02
- ElectroHobo's MiMo recipe on a Broadwell host behind one PLX switch: P2P off is about 40 % slower on every row (one user 181.3 / 164.4 / 96.8 tok/s against 322.0 / 274.9 / 170.1). Source: repo github.com/ElectroHobo/mimo-v26-flash-cmp170hx-recipe by ElectroHobo on GitHub (posted in Discord by @nathang6866)
- Morrowmake's DeepSeek V4.1 Flash recipe: P2P off costs 133 against 162 tok/s prose for one user, and 344 against 432 at 8 users. Source: repo github.com/Morrowmake/dsv41-flash-cmp170hx-recipe by Morrowmake on GitHub (posted in Discord by @lia_oc and @mountains1605)
The chassis topology decides: PCIe switches or root ports, Intel or AMD host, and which driver build. Test both ways on any new box. Source: @darkmaniac7, 2026-10-25 (lab notes) See P2P for how to enable it.
P2P on 2 cards: about +14 % prefill and +8 % decode on Qwen Flash Next
chonsworth (x16-modded cards, his own fork with the P2P patches, no VBIOS or clock changes) measured about +14 % prefill and +8 % decode with P2P on his standard bench, running Qwen Flash Next across 2 cards: about 4,550 tok/s prefill and about 149 tok/s decode. SeriousRising reported similar, maybe slightly higher, numbers with P2P on two cards.
Proven
Source: @chonsworth, 2026-10-03
Source: @seriousrising, 2026-10-03
Link bandwidth
How much link bandwidth TP and PP need without NVLink
Contested
- 2026-07-27, against TP without NVLink: Samuel Hayden called TP a waste without NVLink, depending on context length and model size. 1a2b3c0967 saw only about 10 % more from NVLinked 3090s on Qwen 3.6 27B with vLLM TP, while diabeetee said NVLink adds far more bandwidth on these cards than on 3090s. Source: @samuel_hayden., 2026-07-27
Source: @1a2b3c0967, 2026-07-27
Source: @diabeetee, 2026-07-27 - 2026-07-27, enough for small TP: ju5t5omedude thought TP probably fine at Gen2 x8 with fewer than 4 GPUs. Zorg33: "TP needs about 1GB/s only but sensitive to latency", and Gen3 is ideal for small-batch TP. CUTE: Gen2 x16 is plenty for inference but not for training. Humvee55: once NVLink works, x1 links are fine apart from slow weight loading. robbbbbb7973 expected TP to help most with many simultaneous requests. Source: @ju5t5omedude, 2026-07-27
Source: @zorg33, 2026-07-27
Source: @cute_sc2, 2026-07-27
Source: @scallion_9883, 2026-07-27
Source: @wsy2220, 2026-07-27
Source: @robbbbbb7973, 2026-07-27 - 2026-08-05, PP is fine on Gen2 x4: DeeJay: little data crosses between cards under PP ("it doesnt even saturate gen2 at 4x"); he later said TP prefill needs at least 8 GB/s to match PP, which makes Gen2 x16 plus P2P the minimum for TP. gym agreed PP is the compromise and is fine on PCIe 2.0 if you accept it; Impactic estimated the loss at 10-20 %; Zorg33: Gen2 is enough up to about batch size 16. Source: @deejay9983, 2026-08-05
Source: @jimtjames, 2026-08-05
Source: @impactic, 2026-08-05
Source: @zorg33, 2026-08-05 - 2026-08-05, interconnect bound: programmerboi: Gen2 x4 is about 2 GB/s against about 1.7 TB/s of HBM, so TP, expert parallel and high-batch PP are interconnect bound and the tensor cores sit idle. ju5t5omedude found TP faster than PP in unsloth on Gen2 x4. replacedriver47 found 2x V100 NVLink TP helps prompt processing but barely helps token generation. Source: @programmerboi, 2026-08-05
Source: @ju5t5omedude, 2026-08-05
Source: @replacedriver47, 2026-08-05
Whether NVLink can be unlocked on these cards is covered on NVLink.
Bandwidth arithmetic for Gen2 x4 cards behind a Gen4 x16 uplink
fedzel asked whether the Gen2 link limits throughput with 4 or more cards. irythros: PCIe 2.0 x16 = 3.0 x8 = 4.0 x4 = 5.0 x2 in bandwidth. Studebaker: since the cards run at PCIe 2.0 x4 anyway, a PCIe 4.0 x16 uplink has plenty of bandwidth for several of them, and dual-x16 PLX boards add slightly more latency. programmerboi worried that a single x16 uplink would bottleneck 10 GPUs. Switch boards: PCIe switch boards.
Theory 8 GB and 10 GB
Source: @fedzel, 2026-08-22
Source: @irythros, 2026-08-23
Source: @programmerboi, 2026-08-23
Source: @_studebaker_, 2026-08-23
x1 links per card are fine for pipeline parallel; the cost is load time
robbbbbb7973 runs DeepSeek V4 Flash on 3 CMPs with 1 PCIe lane each at 25-60 tok/s decode, and segmond is on PCIe x1 per card too. After Waka324's 70-80 tok/s 4-card pipeline-parallel result on stock cards, Ikyo concluded that PCIe speed matters less than people say and that the main cost is model load time. robbbbbb7973 noted that speculative-decoding variants are slow on 3 cards at x1.
Reported
Source: @robbbbbb7973, 2026-09-02
Source: @segmond, 2026-09-03
Source: @ikyo, 2026-09-06
What a faster link would buy: TP prefill, loading, RAM offload
Contested
- Zanooda and Merom Green called 3-card or TP4 setups feasible with Gen3 x16. Source: @zanooda, 2026-09-06
Source: @meromgreen, 2026-09-06 - Miséricorde listed what a faster link gives: better TP2 with denser models, better TP2 prefill, slightly better PP from lower latency, faster model loading, and somewhat usable RAM offload for models that do not fit. Source: @wakeupmbrc1, 2026-09-06
- insomniac pushed back that Gen3 is about 8 GB/s, far below even the 400 GB/s interconnect of China-export A100s, so A100-class TP should not be expected. Amogh replied that the x16 capacitor mod plus Gen3 gives about "32 GT/s" and should be enough. Source: @ins_mniac, 2026-09-06
Source: @_j3ns, 2026-09-06 - quark2world: the main gain is model loading. Arcanum: PP is fine on Gen2 x16 and even on Gen2 x4. Source: @quark2world, 2026-09-06
Source: @_arcanum_, 2026-09-06
Where Gen3 stands: PCIe Gen3 on the 10 GB card.
Does PCIe Gen2 against Gen4 matter for inference
Contested 8 GB and 10 GB
- quark2world: Gen 2.0, 3.0 or 4.0 hardly matters for these cards; mainly model loading benefits, and even LoRA training barely uses the bandwidth. Source: @quark2world, 2026-09-26
- regaly: "there are huge difference if you want to use moe expert cache for larger moe models". Source: @reggie7756, 2026-09-26
- mrpandagod: Gen4 adds little over Gen2 in practice, because it is still too slow for useful tensor parallelism, and SXM4-to-PCIe adapters lack NVLink. Source: @djpandaman, 2026-09-26
- NikhilR: on 5060 Ti cards (Gen4), TP2 and TP4 give the same throughput for Qwen A3B, and NVLink would be needed for TP4. Source: @nikhilr1696, 2026-09-26
- CUTE: multi-card inference needs inter-card communication; TP is good with 4 cards on PCIe 2.0 x16, and 4.0 x16 "would be awesome". Source: @cute_sc2, 2026-09-26
Ideas for 10 cards
A saturated PP pipeline, duplicated dense layers, expert parallel
Takka suggested that with 10 cards, saturating a pipeline-parallel pipeline would give good aggregate throughput; dense layers could be duplicated across more than one card; expert parallel might begin to make sense, though PCIe bandwidth may be too low. Untested.
Theory
Source: @spectakkular, 2026-10-02
Related
P2P, NVLink, Gen2 link retrain, PCIe x16 capacitor mod, PCIe switch boards, Risers and bifurcation, Consumer motherboards, Large rigs (8 or more cards), GLM 5.3 Flash on 4x 8 GB, DeepSeek V4 Flash 0731 on 4x 8 GB (allover326 PP4 DSpark), DeepSeek V4.1 Flash on 5x 8 GB (PP5 full residency), Qwen3.8 Flash Next on 3x or 4x 8 GB (zebgop-ops FP8 PP), Qwen3.8 Flash Next on 2x 8 GB (Qwen-Flash-SM80, GDS TEP2), Speculative decoding on the 170HX, llama.cpp on CMP cards, vLLM on sm_80: known bugs and patches, Benchmarks, Engines.