Qwen3.8 Flash Next on 2x 8 GB (Plean PP2)

Qwen3.8-Flash-Next
Reported 2 x 8 GB
Quant	Mixed INT4 AutoRound (Minachist/Qwen3.8-Flash-Next-INT4-Mixed-AutoRound, about 65 GB) with the full BF16 PLE table on NVMe
Engine	vLLM 0.29.1rc1 nightly plus the checkpoint author's patches
Topology	PP2, MTP 3, no P2P
Context	262,144 (256K in the post)
Decode	197 tok/s
Prefill	5800 tok/s
VRAM per card	64 GB
Flags	`--pipeline-parallel-size 2, MTP 3 draft tokens, max concurrency 6, BF16 PLE table offloaded to NVMe`
Tested	@plean. on 2026-10-07
Source	link
Confirmations	Partly reproduced by @galygious (8 GB), 2026-10-08 · details

Plean's setup for Qwen3.8 Flash Next on two 64 GB cards: vLLM pipeline parallel 2, MTP speculative decoding, and the full BF16 PLE (n-gram) table offloaded to NVMe instead of held on the cards. It uses Minachist's mixed INT4 AutoRound checkpoint and runs up to 6 concurrent requests with a 256K context window. P2P is off on his rig because of a chipset issue with how lanes are assigned between his motherboard and CPU, so it needs neither P2P nor the x16 mod. The full write-up is on his site, linked in the infobox. Infobox numbers are from the localairecipes listing of this build (predictable output; see below). Source: @plean., 2026-10-07
Source: @plean., 2026-10-07
Source: @plean., 2026-10-07

Prerequisites

Two CMP 170HX at 64 GB (Driver unlock, 8 GB card). No P2P needed; PP2 passes only activations, so x4 links work.
A fast NVMe for the BF16 PLE table (about 95 GiB) and about 64 GB of host RAM.
The checkpoint Minachist/Qwen3.8-Flash-Next-INT4-Mixed-AutoRound (gated; body about 65-67 GiB).

Source: @plean., 2026-10-07
Source: @darkmaniac7, 2026-10-27 (lab notes)

Steps

Follow Plean's write-up at bozocentral.com. In outline, as listed on localairecipes and in Ivan's notes:

Pull the vLLM image vllm-openai 0.29.1rc1.dev47 and apply the checkpoint author's patches (16 patch files, local build).
Download the Minachist INT4-Mixed AutoRound checkpoint (revision 6a995be5 in the listing).
Serve with pipeline parallel 2 and MTP 3, the BF16 PLE table memory-mapped from NVMe, 262K context. The listing reports a 1.8M-token BF16 KV pool.
Plean serves it through llama-swap; set the concurrency you need (he runs 6).

Source: @darkmaniac7, 2026-10-27 (lab notes)

Measured numbers

Per stream, from Plean: about 100 tok/s each with 2 streams and about 80 each with 3 to 4. Per-stream speed drops because vLLM runs prefill and decode one after the other here (OB88594). DUFRelic has the same problem on his setup: 120 tok/s for one stream and only about double that at 4 concurrent. Source: @plean., 2026-10-07
Source: @ob88594, 2026-10-07
Source: @dufrelic, 2026-10-07
localairecipes listing of this build (rev 74b4d06): 197-201 tok/s on predictable output, 116-122 on thinking-heavy output for one user; 154-212 tok/s for 2 streams; prefill about 5,800 tok/s. Source: @darkmaniac7, 2026-10-27 (lab notes)
For comparison, Mountains pointed 68x to another localairecipes entry, W4A16 with an INT4 PLE table on two x16-modded cards (PP2, MTP3), listed at 133 tok/s for one user and 245 at 4 concurrent; he was not sure he liked the INT4 PLE table. Source: @mountains1605, 2026-10-08

Known issues

Ryko warns that a run-to-run variance issue he found in upstream vLLM (KL divergence that changed between identical runs) probably affects anyone serving this model with vLLM; his fix notes are in vektorprime/qwen38-flash-next-pp2 commit 32595fb. Plean's table is the full BF16 one on NVMe, which he expects to be unaffected. See Qwen3.8 Flash Next on 2x 8 GB (vektorprime PP2). Source: @vektor_prime, 2026-10-07
Source: @plean., 2026-10-07
The checkpoint is gated on Hugging Face.

Variants

GalyGames reproduced it on a Proxmox setup and added the huihui "decision matrix" abliteration to remove refusals for local coding. Source: @galygious, 2026-10-08

Sources

Plean's posts in the testing channel, 2026-10-07, and his write-up at bozocentral.com.
Ivan's lab notes, 2026-10-27 (localairecipes listing).