View source for Qwen3.8 Flash Next on 2x 8 GB (Plean PP2)

{{Recipe
|model=Qwen3.8-Flash-Next
|quant=Mixed INT4 AutoRound (Minachist/Qwen3.8-Flash-Next-INT4-Mixed-AutoRound, about 65 GB) with the full BF16 PLE table on NVMe
|engine=vLLM 0.29.1rc1 nightly plus the checkpoint author's patches
|cards=8 GB
|card_count=2
|topology=PP2, MTP 3, no P2P
|context=262,144 (256K in the post)
|decode_tps=197
|prefill_tps=5800
|vram_per_card=64 GB
|flags=--pipeline-parallel-size 2, MTP 3 draft tokens, max concurrency 6, BF16 PLE table offloaded to NVMe
|status=Reported
|tested_by=plean.
|tested_on=2026-10-07
|source=https://www.bozocentral.com/2x-cmp170hx-qwen3-flash-next
}}
Plean's setup for Qwen3.8 Flash Next on two 64 GB cards: vLLM pipeline parallel 2, MTP speculative decoding, and the
full BF16 PLE (n-gram) table offloaded to NVMe instead of held on the cards. It uses Minachist's mixed INT4 AutoRound
checkpoint and runs up to 6 concurrent requests with a 256K context window. P2P is off on his rig because of a chipset
issue with how lanes are assigned between his motherboard and CPU, so it needs neither P2P nor the x16 mod. The full
write-up is on his site, linked in the infobox. Infobox numbers are from the localairecipes listing of this build
(predictable output; see below).
{{Source|handle=plean.|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557427236447985704}}<br />
{{Source|handle=plean.|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557460249424826449}}<br />
{{Source|handle=plean.|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557461178018566226}}

== Prerequisites ==
* Two CMP 170HX at 64 GB ([[Driver unlock]], [[8 GB card]]). No P2P needed; PP2 passes only activations, so x4 links work.
* A fast NVMe for the BF16 PLE table (about 95 GiB) and about 64 GB of host RAM.
* The checkpoint [https://huggingface.co/Minachist/Qwen3.8-Flash-Next-INT4-Mixed-AutoRound Minachist/Qwen3.8-Flash-Next-INT4-Mixed-AutoRound] (gated; body about 65-67 GiB).
{{Source|handle=plean.|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557427236447985704}}<br />
{{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}

== Steps ==
Follow Plean's write-up at [https://www.bozocentral.com/2x-cmp170hx-qwen3-flash-next bozocentral.com]. In outline, as
listed on localairecipes and in Ivan's notes:
# Pull the vLLM image <code><nowiki>vllm-openai 0.29.1rc1.dev47</nowiki></code> and apply the checkpoint author's patches (16 patch files, local build).
# Download the Minachist INT4-Mixed AutoRound checkpoint (revision <code><nowiki>6a995be5</nowiki></code> in the listing).
# Serve with pipeline parallel 2 and MTP 3, the BF16 PLE table memory-mapped from NVMe, 262K context. The listing reports a 1.8M-token BF16 KV pool.
# Plean serves it through llama-swap; set the concurrency you need (he runs 6).
{{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}

== Measured numbers ==
* Per stream, from Plean: about 100 tok/s each with 2 streams and about 80 each with 3 to 4. Per-stream speed drops because vLLM runs prefill and decode one after the other here (OB88594). DUFRelic has the same problem on his setup: 120 tok/s for one stream and only about double that at 4 concurrent. {{Source|handle=plean.|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557459833928556697}}<br />{{Source|handle=ob88594|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557464371838451723}}<br />{{Source|handle=dufrelic|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557460484830138469}}
* localairecipes listing of this build (rev 74b4d06): 197-201 tok/s on predictable output, 116-122 on thinking-heavy output for one user; 154-212 tok/s for 2 streams; prefill about 5,800 tok/s. {{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}
* For comparison, Mountains pointed 68x to another localairecipes entry, W4A16 with an INT4 PLE table on two x16-modded cards (PP2, MTP3), listed at 133 tok/s for one user and 245 at 4 concurrent; he was not sure he liked the INT4 PLE table. {{Source|handle=mountains1605|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557599164530040852}}

== Known issues ==
* Ryko warns that a run-to-run variance issue he found in upstream vLLM (KL divergence that changed between identical runs) probably affects anyone serving this model with vLLM; his fix notes are in [https://github.com/vektorprime/qwen38-flash-next-pp2/commit/32595fbaa3ae24764055e5b6341e7a63c35fbcf1 vektorprime/qwen38-flash-next-pp2 commit 32595fb]. Plean's table is the full BF16 one on NVMe, which he expects to be unaffected. See [[Qwen3.8 Flash Next on 2x 8 GB (vektorprime PP2)]]. {{Source|handle=vektor_prime|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557443747892035674}}<br />{{Source|handle=plean.|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557460674093912109}}
* The checkpoint is gated on Hugging Face.

== Variants ==
* GalyGames reproduced it on a Proxmox setup and added the huihui "decision matrix" abliteration to remove refusals for local coding. {{Source|handle=galygious|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557608614318047253}}

== Related ==
* Other 2-card recipes: [[Qwen3.8 Flash Next on 2x 8 GB (ElectroHobo W4A16 TP2)]], [[Qwen3.8 Flash Next on 2x 8 GB (mannydelrio PP2)]], [[Qwen3.8 Flash Next on 2x 8 GB (DUFRelic PP2)]].
* [[Engines]], [[Quantization on the 170HX]], [[Benchmarks]].

== Sources ==
* Plean's posts in the testing channel, 2026-10-07, and his write-up at bozocentral.com.
* Ivan's lab notes, 2026-10-27 (localairecipes listing).

{{ImportFooter}}
[[Category:8 GB]]