Qwen3.8 Flash Next on 2x 8 GB (mannydelrio PP2)

Qwen3.8-Flash-Next-Uncensored (refusal-removed, from orcarouter)

Reported 2 x 8 GB
Quant	AWQ W4A16 g32 (leoncca/Qwen3.8-Flash-Next-Uncensored-AWQ-g32); PLE table converted to INT4 and served from SSD
Engine	wtdcode/vllm-backport 3142ded1 plus a two-file patch and two tinybeast PLE plugins (venv, no container) vllm-backport 3142ded1
Topology	PP2 (layer partition 25,23), no P2P
Context	262,144
Decode	177.8 tok/s
Prefill	7006 tok/s
VRAM per card	64 GB, --gpu-memory-utilization 0.94, TB_TORCH_MEM_CAP_GIB=61.5
Power cap	180 W
Driver	nvidia-open 610.43.02 + cmpunlocker v0.5 or newer
Unlock build	cmpunlocker v0.5+ (sudo ./install.sh --profile=8gb --no-iommu)
Flags	--pipeline-parallel-size 2, VLLM_PP_LAYER_PARTITION=25,23, MTP 3, --max-num-seqs 8, cudagraph FULL_DECODE_ONLY sizes [1,2,4,...,32], --max-num-batched-tokens 4096, NCCL_ALGO=Ring NCCL_PROTO=Simple, VLLM_PLUGINS=tinybeast_ple_int4,tinybeast_ple_ssd
Tested	@mrmannyatx on 2026-10-15
Source	link
Confirmations	None yet. Confirm on your own card

An uncensored Qwen3.8-Flash-Next on two stock 8 GB-variant cards (PCIe Gen2 x4, no capacitor mod, no P2P) in a Ryzen desktop with only 16 GB of RAM. Two "tinybeast" vLLM plugins quantize the 29.8 GB PLE table to INT4 and stream it from the NVMe, freeing VRAM for MTP speculative decoding. Pipeline parallel, not tensor parallel, because without P2P TP pays a host-memory sync per layer. It reads prompts faster than any other two-card recipe. Infobox numbers are one user on code and cold prefill on the cross-recipe protocol. Source: repo github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx by mannydelrio on GitHub, posted by @mrmannyatx, 2026-10-06

MrManny posted the repo on 2026-10-06 as a Claude-assisted setup that works for him and asked for improvements; fiiireface had his own 2-card setup to publish. A later MTP5 revision listed on localairecipes reports 240 / 225 / 156 tok/s for one user (structured / code / prose), 1,044 / 715 / 625 at eight users and 6,964 tok/s cold prefill, with a measured run file; his MTP3 revision 109 prose / 159 code and about 7,700 prefill. Source: @mrmannyatx, 2026-10-06
Source: @fiiireface, 2026-10-06
Source: @darkmaniac7, 2026-10-27 (lab notes)

Prerequisites

• Two CMP 170HX 8 GB variant (PCI ID 10de:20c2) unlocked to 64 GB each: nvidia-open 610.43.02 plus cmpunlocker v0.5 or newer, installed with sudo ./install.sh --profile=8gb --no-iommu. Disable cmpunlocker-passthrough.service if the cards are not passed to VMs (it blocks GPU resets), then fully power off and on. nvidia-smi should show 65,536 MiB per card and ECC Enabled. See Driver unlock.
• Author's host: Ryzen 5 5600X, 16 GB RAM, 1 TB Crucial P2 NVMe (the converted model is 110 GB), Ubuntu 26.04, kernel 7.0.0-29.

Source: repo github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx by mannydelrio on GitHub, posted by @mrmannyatx, 2026-10-06

Steps

The repo layout:

patches/vllm-backport-3142ded1.patch   pipeline-parallel KV fix (from clee/vllm-sm80) + vLLM PR #57105
plugins/tinybeast-ple-int4/            vLLM plugin: 4-bit PLE table format and lookup kernel
plugins/tinybeast-ple-ssd/             vLLM plugin: serve the PLE table from the SSD, CPU-side prefetch, GPU memory cap
scripts/prepare_checkpoint.py          download the checkpoint (resumable) and convert the PLE table to 4-bit
scripts/checkpoint-source.json         which checkpoint and revision to download
serve/serve.sh                         the exact production launcher
serve/tinybeast-vllm.service           optional systemd user unit

1. Build wtdcode/vllm-backport at 3142ded1671b9565503975ee946d7e19fe50ea95 into its own venv (PyTorch 2.13.0, Triton 3.7.1, CUDA 13.2) and apply patches/vllm-backport-3142ded1.patch.
2. pip install the two plugins into that venv.
3. Run scripts/prepare_checkpoint.py: it downloads leoncca/Qwen3.8-Flash-Next-Uncensored-AWQ-g32 at fa561462 and converts the PLE table.
4. Start with serve/serve.sh (port 18084, served name tinybeast-flashnext-uncensored). First start about 5.5 minutes.

Key settings: --pipeline-parallel-size 2 with VLLM_PP_LAYER_PARTITION=25,23; --speculative-config '{"method":"mtp","num_speculative_tokens":3}' (about 2.5x faster answers); --max-num-seqs 8 with cudagraph_capture_sizes [1,2,4,8,12,16,20,24,28,32] and cudagraph_mode FULL_DECODE_ONLY; --max-num-batched-tokens 4096; --gpu-memory-utilization 0.94; --load-format safetensors --safetensors-load-strategy lazy; NCCL_ALGO=Ring NCCL_PROTO=Simple; VLLM_PLUGINS=tinybeast_ple_int4,tinybeast_ple_ssd with TB_PLE_SSD=1 TB_PLE_SSD_PREFETCH=1; TB_TORCH_MEM_CAP_GIB=61.5 and CUDA_MODULE_LOADING=EAGER; VLLM_DETERMINISTIC_MOE_ALIGN=0 (about 17% faster decode at the same quality). Source: repo github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx by mannydelrio on GitHub, posted by @mrmannyatx, 2026-10-06

Measured numbers

• Cross-recipe protocol (MiaAI-Lab prompts, temperature 0, thinking off, 400-token cap, 2026-10-15): one user 192.4 / 177.8 / 145.4 tok/s (structured / code / prose), 8 users 986.4 / 734.2 / 619.6 total, cold prefill 7,006 tok/s on 24K to 38K prompts. HumanEval 155/164 (runs 154 to 158). 8/8 on a long-document test up to 222K tokens.
• Against stock tensor-parallel vLLM on the same cards: a 100K-token prompt plus 256-token answer 81.4 s to 14.4 s (5.7x); decode about 37 to 109 / 159 tok/s (prose / code); prompt reading about 1,340 to 7,700 tok/s; KV cache 260K to 897K tokens.
• The SSD prefetch (row ids computed on the CPU so reads overlap the previous GPU step) took a 100K prompt from 28.0 s to 14.6 s.
• Power: 180 W per card; 220 W and 250 W gave only 5.6% and 8.4% faster long prompts for 19% and 36% more GPU power.

Source: repo github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx by mannydelrio on GitHub, posted by @mrmannyatx, 2026-10-06

• bigfoot looked up the KV capacity in the repo: 897K tokens. Source: @bigfoot1144, 2026-10-06
• MrManny moved to Qwen Flash Next from GLM Flash after the GLM recipe's 1.7 repetition bug, on 16 GB of host RAM. Source: @mrmannyatx, 2026-10-14

Known issues

• GPU 1 faulted with MMU REGION_VIOLATION when usage climbed into the top of its unlocked 64 GB (prompts over about 200K tokens, 2026-10-05); the memory cap (TB_TORCH_MEM_CAP_GIB=61.5) and eager module loading avoid it.
• With older cmpunlocker (v0.2), a protected 141 MB region at the top of memory was handed out and long prompts crashed the second card with Xid 31. Use v0.5 or newer. See Troubleshooting.
• Raising only --max-num-seqs to 8 made 8 requests slower (395.5 / 332.2 / 286.2) because requests 5 to 8 ran without CUDA graphs; vLLM rounds graph sizes to multiples of 1+3=4, hence the capture list.
• Tried and dropped: YaRN x2 to 524K works (needle found at 502K) but shifts outputs at every length; vLLM PRs #58114 and #58449 gave no gain or a startup crash on this fork.
• 80 GB is not possible on the 8 GB variant (4 HBM stacks x 16 GB = 64 GB); PCIe Gen3 is fused off and unsolved. See 8 GB card.
• Serves a refusal-removed derivative under the Qwen Community License. No container: the engine is built from source. Uses the wtdcode engine lineage, so its flags do not transfer to Morrowmake-based recipes.

Source: repo github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx by mannydelrio on GitHub, posted by @mrmannyatx, 2026-10-06

Related

• Qwen3.8 Flash Next on 2x 8 GB (ElectroHobo W4A16 TP2) (needs x16 and P2P, faster decode), Qwen3.8 Flash Next on 2x 8 GB (DUFRelic PP2), Qwen3.8 Flash Next on 2x 8 GB (vektorprime PP2).
• Engines (wtdcode/vllm-backport), Benchmarks.

Sources

• Repo: mannydelrio/qwen38-flash-next-2x-cmp170hx (Apache-2.0), last commit 2026-10-15. Discord handle MrManny, GitHub mannydelrio.

Imported from the cmpunlocker Discord with credit; see Credits.