Qwen3.8 Flash Next on 3x 8 GB (clee PP3)

Qwen3.8-Flash-Next
Proven 3 x 8 GB
Quant	W4A16 AutoRound: Intel/Qwen3.8-Flash-Next-W4A16-AutoRound; 95.4 GiB BF16 PLE table streamed from NVMe; KV fp8 e4m3 (software-decoded on SM80)
Engine	vLLM nightly 7d0b4e57 plus the author's pure-Python fork clee/vllm-sm80 (branch q38fn/stack) q38fn/stack on nightly-7d0b4e57
Topology	PP3, no P2P (works at PP2 on two cards)
Context	1,048,576 (YaRN x4)
Decode	74.4 tok/s
Prefill	4831 tok/s
VRAM per card	64 GiB, gpu-memory-utilization 0.92
Flags	`pipeline-parallel-size 3, block-size 1728, kv-cache-dtype fp8_e4m3, max-model-len 1048576, max-num-seqs 8, max-num-batched-tokens 8192, FULL_AND_PIECEWISE; NCCL_P2P_DISABLE=1, VLLM_PP_LAYER_PARTITION=16,16,16, VLLM_PLE_STAGED=1`
Tested	@c133 on 2026-10-10
Source	link
Confirmations	Reproduced by @68x0 (8 GB), 2026-10-08 · details

Qwen3.8-Flash-Next on three unmodded 64 GB cards (x4 links, no P2P) with pipeline parallel 3, a 1M-token context and a 7.36M-token KV pool, on a pinned vLLM nightly plus the author's fork. The BF16 PLE table is streamed live from the checkpoint's safetensors shards on NVMe, and a tiered GPU / CPU / NVMe KV prefix cache survives server restarts, so a long agent session resumes quickly after a restart. It also runs on two cards at PP2 with less KV. Infobox numbers are the author's cold single-stream run at 1K (decode) and 64K (prefill). Source: repo github.com/clee/qwen3.8-flash-next-3x170hx by clee on GitHub, posted by @c133, 2026-10-04

clee's posts: his cards are not lane-modded at all, which is why the setup uses PP3, and it should also work on two cards at PP2 with less KV cache. The 2026-10-11 update added NVMe storage for a persistent KV cache between server restarts, so a long agent session resumes quickly after a restart; with it he can hardly tell 2 cards from 3, and switching between sessions is seamless even on a Gen3 NVMe SSD. He had no benchmark yet for loading a large session from NVMe after a fresh start. Source: @c133, 2026-10-04
Source: @c133, 2026-10-04
Source: @c133, 2026-10-11
Source: @c133, 2026-10-11
Source: @c133, 2026-10-11

Prerequisites

Three (or two) CMP 170HX at 64 GiB (Driver unlock, 8 GB card). No P2P needed: the author's cards sit on different root buses, which create BAR1 maps but hang on DMA, so NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce are mandatory there and PP stages go through host RAM.
About 169 GiB of local NVMe for the checkpoint plus up to 128 GiB for the NVMe KV tier.
Author's host: Threadripper with 128 GB RAM (pinned host memory tops out at about 64 GiB, which is why the PLE table is streamed rather than pinned).
jj (Jujutsu): the launcher verifies the fork checkout descends from q38fn/stack.

Source: repo github.com/clee/qwen3.8-flash-next-3x170hx by clee on GitHub, posted by @c133, 2026-10-04

Steps

# 1. checkpoint on a local NVMe (~169 GiB, plus up to 128 GiB for the NVMe KV tier)
hf download Intel/Qwen3.8-Flash-Next-W4A16-AutoRound --local-dir /home/models/qwen38fn-w4a16-autoround
# 2. pull the pinned nightly the patches are based on
docker pull vllm/vllm-openai:nightly-7d0b4e57aac4c4323225b0eee9ae960ce75ffffe
# 3. clone the fork (branches: q38fn/*)
git clone https://github.com/clee/vllm-sm80
# 4. build the fork into an image (pure Python, ~2 min) and launch
git clone https://github.com/clee/qwen3.8-flash-next-3x170hx && cd qwen3.8-flash-next-3x170hx
docker build -t vllm-sm80:q38fn .
./run-qwen38fn.sh IMAGE=vllm-sm80:q38fn

Serving config configs/intel-1m-fp8-ssmbf16.yml: pipeline-parallel-size 3, tensor-parallel-size 1, block-size 1728 (a multiple of 12 for the QSA ring and 16 for the mamba chunk grid), kv-cache-dtype fp8_e4m3, mamba-ssm-cache-dtype bfloat16, max-model-len 1048576, max-num-seqs 8, max-num-batched-tokens 8192, gpu-memory-utilization 0.92, prefix caching, chunked prefill, mamba-cache-mode align, async scheduling, FULL_AND_PIECEWISE, parsers qwen3_xml / qwen3, froggeric's fixed chat template, default reasoning_effort xhigh; hf-overrides for 48 layer types and YaRN factor 4.0 from 262,144.
Launcher env: NCCL_P2P_DISABLE=1, VLLM_PP_LAYER_PARTITION=16,16,16 (the PLE layer stays on rank 0), VLLM_PLE_STAGED=1, QWEN4EXP_PLE_MODEL_DIR, VLLM_ALLOW_LONG_MAX_MODEL_LEN=1. Server on port 8080 (0.0.0.0, no auth), model name qwen3.8.
KV tiering: OffloadingConnector with an 8 GiB CPU LRU tier in /dev/shm and a 128 GiB filesystem tier (max_bytes from vLLM PR #54327, backported). Blocks are content-addressed, so the cache survives restarts; a change of TP/PP/dtype/block size orphans the old directory, which is safe to delete by hand.

Source: repo github.com/clee/qwen3.8-flash-next-3x170hx by clee on GitHub, posted by @c133, 2026-10-04

Partition: the repo once shipped VLLM_PP_LAYER_PARTITION=17,17,14 left over from MTP testing (with PP3 the MTP draft loads entirely on the last GPU); without MTP use 16,16,16. Source: @c133, 2026-10-08

The fork's patches

q38fn/units-fix (block-size units bug in _align_hybrid_block_size), q38fn/fp8-kv-sm80 (uint8 cache storage plus software e4m3 decode in the QSA gather kernel), q38fn/ple-pipeline-parallel (PLE checkpoints under PP>1, MTP draft on the last rank), q38fn/ple-nvme-staged (shard-direct preadv reader into a fixed GPU staging buffer, cudagraph-safe), the #54327 filesystem-cap backport and a backport of upstream #58497. Source: repo github.com/clee/qwen3.8-flash-next-3x170hx by clee on GitHub, posted by @c133, 2026-10-04

Measured numbers

Author, 2026-10-10, cold cache, single stream, temperature 0:

Prompt tokens	Prefill tok/s	TTFT	Decode tok/s
1,026	2,592	0.40 s	74.4
16,393	4,278	3.83 s	76.6
64,845	4,831	13.42 s	75.5
258,992	4,947	52.35 s	75.0
1,036,890	4,748	218 s	72.9

Needle suite passes from 8k to 1M (809k prompt tokens). MTP is not enabled (its draft-layer KV would cost about 16% of the pool). Source: repo github.com/clee/qwen3.8-flash-next-3x170hx by clee on GitHub, posted by @c133, 2026-10-04

KV pool: "GPU KV cache size: 7,359,330 tokens, Maximum concurrency for 1,048,576 tokens per request: 7.02x" on 3 cards; 3,108,278 tokens (2.96x) on 2 cards after rebasing on newer vLLM. This depends on the PLE table living off the GPUs. Source: @c133, 2026-10-08
Source: @c133, 2026-10-09
68x pulled the branch onto three unlocked 8 GB cards: 52 to 60 tok/s on his single-user agentic workload. Later, with MTP (1 token) and an uncensored model: roughly 90 tok/s on 3 cards and 100 to 110 on 2 cards, with prefill only slightly lower on 2 cards. Source: @68x0, 2026-10-08
Source: @68x0, 2026-10-09
Source: @68x0, 2026-10-11
Contested MTP's benefit on this model: 68x needs it for 100+ tok/s on 2 cards; clee, on the INT4 AutoRound model, still measures better numbers without MTP on 2 cards. Mountains suggests 3 draft tokens if acceptance is good. Source: @mountains1605, 2026-10-09
Source: @c133, 2026-10-16
Trust Inter, also on 3 cards, finds it a little slow against his own setup at above 110 tok/s decode (clee had not enabled MTP yet). Source: @trustinter, 2026-10-05

Known issues

Boot OOM in allocate_embedding_weight (cudaHostAlloc) means VLLM_PLE_STAGED is not active. "assert attn_page_size >= mamba_page_size" means the units fix is not mounted. "Padding mamba page size by ~8%" at block 1728 is healthy.
A TP2+ NCCL hang means P2P trouble; this recipe is PP only.
Pin mismatch inside the repo: the Dockerfile's default base and branch differ from what run-qwen38fn.sh and the README pin; the README says newer nightlies are not validated.
No license file. Last commit 2026-10-10. API published on 0.0.0.0:8080 with no auth.
The README calls the model 180B-A6B; other recipes say 125B MoE. Check the Qwen model card.

Source: repo github.com/clee/qwen3.8-flash-next-3x170hx by clee on GitHub, posted by @c133, 2026-10-04

Sources

Repo: clee/qwen3.8-flash-next-3x170hx (no license stated), fork clee/vllm-sm80. Design credit to tonyd2wild/Qwen38-Flash-Next-4x3090. Posted on Discord by clee (2026-10-04, update 2026-10-11) and Mountains.

Prerequisites

Steps

The fork's patches

Measured numbers

Known issues

Related

Sources