Qwen3.8 Flash Next on 2x 8 GB (DUFRelic PP2)

Qwen3.8-Flash-Next

Reported 2 x 8 GB
Quant	W4A16 INT4 g128 with the PLE table repacked INT4 g32: zankich/Qwen3.8-Flash-Next-W4A16-Merlin-INT4PLE-FP8KV; KV BF16
Engine	vLLM v0.30.0 + zankich INT4-PLE plugin + six hash-guarded patch layers vLLM v0.30.0 (ced6857a); plugin zankich/vllm v0.29.0z (6609bfef)
Topology	PP2, MTP depth 3
Context	262,144
Decode	133.39 tok/s
Prefill	6213.66 tok/s
VRAM per card	64 GiB, --gpu-memory-utilization 0.90
Power cap	140 W (200 W observed-draw abort guard)
Driver	NVIDIA 610.43.02
Unlock build	amoghmunikote/cmpunlocker 6c442eeb (patch 0016 SM reconfig, 74 SMs) plus Bayley P2P patches 0011/0013/0015
Flags	--pipeline-parallel-size 2 --speculative-config mtp 3 --max-num-seqs 8 --max-num-batched-tokens 2048 PIECEWISE graphs [1,2,4,8,16,32] --gpu-memory-utilization 0.90 --reasoning-parser qwen3 --tool-call-parser qwen3_xml
Tested	@dufrelic on 2026-10-03
Source	link
Confirmations	None yet. Confirm on your own card

A single-README production recipe for Qwen3.8-Flash-Next on two cards unlocked to 64 GiB and 74 SMs, under stock vLLM v0.30.0 with zankich's INT4-PLE plugin and six upstream patch layers, pipeline parallel 2 and MTP depth 3. It does not use the Morrowmake fork. The INT4 PLE table (about 26.8 GiB) stays in pinned host memory. No scripts, Dockerfile or compose file are published; the image and manifests are private, so reproducing it means re-applying the patches by hand. Infobox numbers are concurrency 1 decode and prefill from the author's benchmark. Source: repo github.com/DUFRelic/qwen38-flash-next-recipe by DUFRelic on GitHub, posted by @dufrelic, 2026-10-05

DUFRelic posted the repo on 2026-10-05 and later asked Morrowmake what x4 links cost; Morrowmake had not tested it but expects x4 to hurt prefill. Source: @dufrelic, 2026-10-05
Source: @dufrelic, 2026-10-16
Source: @voidgiraffe, 2026-10-16

Prerequisites

• Two CMP 170HX at 64 GiB. Author's host: Lenovo ThinkStation P520, driver 610.43.02, Ubuntu kernel 7.0.0-34, VBIOS 92.00.6D.00.0A (TechPowerUp 268495) flashed on both cards.
• SM unlock 70 to 74 SMs per card via amoghmunikote/cmpunlocker commit 6c442eeb, patch 0016-sm-reconfig.patch, and NCCL P2P via the Bayley patches 0011/0013/0015 (BAR1 P2P) at commit 5a7bb4b7, with NCCL_P2P_DISABLE=0 NCCL_P2P_LEVEL=SYS. Both are optional for function (the model runs at 70 SMs with NCCL_P2P_DISABLE=1) but needed for the published throughput. See Driver unlock and Unlocks.
• 170tune at boot on both cards: GPC VF offsets +250 / +275, SM ceiling 1,420 MHz (nvidia-smi -lgc 210,1420), HBM NDIV 64, relaxed HBM refresh (about 8.3 us). See Tools.
• At least 34 GiB of MemAvailable: the launcher refuses to start below that, because the INT4 PLE table lives in pinned host memory.

Source: repo github.com/DUFRelic/qwen38-flash-next-recipe by DUFRelic on GitHub, posted by @dufrelic, 2026-10-05

Steps

• Base image vllm/vllm-openai:v0.30.0 (amd64 digest sha256:5f5e5352...), vLLM commit ced6857afa0e, INT4-PLE plugin zankich/vllm branch v0.29.0z (6609bfef).
• Patch layers, in order, each failing closed on a SHA-256 mismatch: PR 56444 (PP2/PLE fix), PR 54793 (empty KV spec filter), the issue 54709 fix (Qwen4Exp MTP draft head over PP), PR 55506 (Mamba request-slot state fix), SpeedSuite PRs 58762, 58763, 58449, 58114, 57097, and a local batched P2P fix (three isend/irecv call sites in parallel_state.py rewritten to batch_isend_irecv) for a cold-start hang.
• Checkpoint zankich/Qwen3.8-Flash-Next-W4A16-Merlin-INT4PLE-FP8KV at 45d5db34 (about 94.4 GiB, a repackage of halt95's Merlin); chat template froggeric/Qwen-Fixed-Chat-Templates v22.5.
• Serve profile p-prefill2048-mtp3-speedsuite: --pipeline-parallel-size 2 --tensor-parallel-size 1 --distributed-executor-backend mp, {"method":"mtp","num_speculative_tokens":3}, max model len 262144, 8 seqs, 2,048 batched tokens with chunked prefill, PIECEWISE CUDA graphs at [1,2,4,8,16,32], --gpu-memory-utilization 0.90, --reasoning-parser qwen3 --tool-call-parser qwen3_xml, up to 4 images and 1 video per prompt. The author serves it on loopback behind Caddy and LiteLLM.

Source: repo github.com/DUFRelic/qwen38-flash-next-recipe by DUFRelic on GitHub, posted by @dufrelic, 2026-10-05

Measured numbers

Benchmark 2026-10-03 (llama-benchy, --pp 16384 --tg 2048, concurrency 1 and 4, 3 runs, no cache):

• Prefill C1 6,213.66 +/- 99.27 tok/s; decode C1 133.39 +/- 22.81 tok/s (108.10 to 163.37).
• Prefill C4 6,639.25 +/- 34.79; decode C4 aggregate 245.17 +/- 1.17.
• MTP depth 3 with 52.27% draft accept rate. GPU peaks 159.66 W / 55 C and 162.77 W / 56 C against 140 W limits; no Xid, OOM or EngineDeadError.

Source: repo github.com/DUFRelic/qwen38-flash-next-recipe by DUFRelic on GitHub, posted by @dufrelic, 2026-10-05

• These are the numbers of the localairecipes entry "Qwen3.8-Flash-Next W4A16 INT4PLE on 2x modded CMP 170HX PP2 MTP3" that Mountains pointed people to (133 tok/s single, 245 at 4). Source: @mountains1605, 2026-10-08
• Per-stream speed drops with concurrency: DUFRelic sees 120 tok/s single stream and only about double aggregate at 4; Plean sees about 100 tok/s each at 2 streams and 80 at 3 to 4. OB88594 attributes it to vLLM running prefill and decode one after the other here (decode stalls during prefill); a batch size of 2,048 helped a little. Source: @dufrelic, 2026-10-07
  Source: @plean., 2026-10-07
  Source: @ob88594, 2026-10-07

Known issues

• KV runs BF16 despite the "FP8KV" suffix: the Qwen4Exp QSA path in vLLM 0.30.0 rejects an FP8 cache.
• --max-num-seqs 8 with PP2 does not give eight independent forwards per rank; the V1 runner caps concurrent batches at the pipeline size.
• README only, one file, no reproducible scripts; the final image and runbooks are private. PR 58449 was open at pin time. Patches 0011 to 0016 should be checked against current cmpunlocker master before treating them as shipped.

Source: repo github.com/DUFRelic/qwen38-flash-next-recipe by DUFRelic on GitHub, posted by @dufrelic, 2026-10-05

Related

• Qwen3.8 Flash Next on 2x 8 GB (mannydelrio PP2), Qwen3.8 Flash Next on 2x 8 GB (ElectroHobo W4A16 TP2), Qwen3.8 Flash Next on 2x 8 GB (vektorprime PP2).
• SM unlock and VBIOS: Unlocks. Tuning tools: Tools. Benchmarks.

Sources

• Repo: DUFRelic/qwen38-flash-next-recipe (no license; the README says its text is free to reuse), last commit 2026-10-05.

Imported from the cmpunlocker Discord with credit; see Credits.