View source for Qwen3.8 Flash Next on 3x 8 GB (clee PP3)

{{Recipe
|model=Qwen3.8-Flash-Next
|quant=W4A16 AutoRound: Intel/Qwen3.8-Flash-Next-W4A16-AutoRound; 95.4 GiB BF16 PLE table streamed from NVMe; KV fp8 e4m3 (software-decoded on SM80)
|engine=vLLM nightly 7d0b4e57 plus the author's pure-Python fork clee/vllm-sm80 (branch q38fn/stack)
|engine_version=q38fn/stack on nightly-7d0b4e57
|cards=8 GB
|card_count=3
|topology=PP3, no P2P (works at PP2 on two cards)
|context=1,048,576 (YaRN x4)
|decode_tps=74.4
|prefill_tps=4831
|vram_per_card=64 GiB, gpu-memory-utilization 0.92
|flags=pipeline-parallel-size 3, block-size 1728, kv-cache-dtype fp8_e4m3, max-model-len 1048576, max-num-seqs 8, max-num-batched-tokens 8192, FULL_AND_PIECEWISE; NCCL_P2P_DISABLE=1, VLLM_PP_LAYER_PARTITION=16,16,16, VLLM_PLE_STAGED=1
|status=Reported
|tested_by=c133
|tested_on=2026-10-10
|source=https://github.com/clee/qwen3.8-flash-next-3x170hx
}}
Qwen3.8-Flash-Next on three unmodded 64 GB cards (x4 links, no P2P) with pipeline parallel 3, a 1M-token context and a
7.36M-token KV pool, on a pinned vLLM nightly plus the author's fork. The BF16 PLE table is streamed live from the
checkpoint's safetensors shards on NVMe, and a tiered GPU / CPU / NVMe KV prefix cache survives server restarts, so a
long agent session resumes quickly after a restart. It also runs on two cards at PP2 with less KV. Infobox numbers are
the author's cold single-stream run at 1K (decode) and 64K (prefill).
{{Source|handle=c133|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1556357466026025001|repo=https://github.com/clee/qwen3.8-flash-next-3x170hx}}

clee's posts: his cards are not lane-modded at all, which is why the setup uses PP3, and it should also work on two cards at PP2 with less KV cache. The 2026-10-11 update added NVMe storage for a persistent KV cache between server restarts, so a long agent session resumes quickly after a restart; with it he can hardly tell 2 cards from 3, and switching between sessions is seamless even on a Gen3 NVMe SSD. He had no benchmark yet for loading a large session from NVMe after a fresh start. {{Source|handle=c133|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1556370593870254111}}<br />{{Source|handle=c133|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1556370756089155737}}<br />{{Source|handle=c133|date=2026-10-11|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1558664933863071837}}<br />{{Source|handle=c133|date=2026-10-11|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1558739200499785809}}<br />{{Source|handle=c133|date=2026-10-11|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1558897010894184479}}

== Prerequisites ==
* Three (or two) CMP 170HX at 64 GiB ([[Driver unlock]], [[8 GB card]]). No P2P needed: the author's cards sit on different root buses, which create BAR1 maps but hang on DMA, so <code><nowiki>NCCL_P2P_DISABLE=1</nowiki></code> and <code><nowiki>--disable-custom-all-reduce</nowiki></code> are mandatory there and PP stages go through host RAM.
* About 169 GiB of local NVMe for the checkpoint plus up to 128 GiB for the NVMe KV tier.
* Author's host: Threadripper with 128 GB RAM (pinned host memory tops out at about 64 GiB, which is why the PLE table is streamed rather than pinned).
* <code><nowiki>jj</nowiki></code> (Jujutsu): the launcher verifies the fork checkout descends from <code><nowiki>q38fn/stack</nowiki></code>.
{{Source|handle=c133|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1556357466026025001|repo=https://github.com/clee/qwen3.8-flash-next-3x170hx}}

== Steps ==
<syntaxhighlight lang="bash">
# 1. checkpoint on a local NVMe (~169 GiB, plus up to 128 GiB for the NVMe KV tier)
hf download Intel/Qwen3.8-Flash-Next-W4A16-AutoRound --local-dir /home/models/qwen38fn-w4a16-autoround
# 2. pull the pinned nightly the patches are based on
docker pull vllm/vllm-openai:nightly-7d0b4e57aac4c4323225b0eee9ae960ce75ffffe
# 3. clone the fork (branches: q38fn/*)
git clone https://github.com/clee/vllm-sm80
# 4. build the fork into an image (pure Python, ~2 min) and launch
git clone https://github.com/clee/qwen3.8-flash-next-3x170hx && cd qwen3.8-flash-next-3x170hx
docker build -t vllm-sm80:q38fn .
./run-qwen38fn.sh IMAGE=vllm-sm80:q38fn
</syntaxhighlight>
* Serving config <code><nowiki>configs/intel-1m-fp8-ssmbf16.yml</nowiki></code>: <code><nowiki>pipeline-parallel-size 3</nowiki></code>, <code><nowiki>tensor-parallel-size 1</nowiki></code>, <code><nowiki>block-size 1728</nowiki></code> (a multiple of 12 for the QSA ring and 16 for the mamba chunk grid), <code><nowiki>kv-cache-dtype fp8_e4m3</nowiki></code>, <code><nowiki>mamba-ssm-cache-dtype bfloat16</nowiki></code>, <code><nowiki>max-model-len 1048576</nowiki></code>, <code><nowiki>max-num-seqs 8</nowiki></code>, <code><nowiki>max-num-batched-tokens 8192</nowiki></code>, <code><nowiki>gpu-memory-utilization 0.92</nowiki></code>, prefix caching, chunked prefill, <code><nowiki>mamba-cache-mode align</nowiki></code>, async scheduling, <code><nowiki>FULL_AND_PIECEWISE</nowiki></code>, parsers <code><nowiki>qwen3_xml</nowiki></code> / <code><nowiki>qwen3</nowiki></code>, froggeric's fixed chat template, default <code><nowiki>reasoning_effort</nowiki></code> xhigh; <code><nowiki>hf-overrides</nowiki></code> for 48 layer types and YaRN factor 4.0 from 262,144.
* Launcher env: <code><nowiki>NCCL_P2P_DISABLE=1</nowiki></code>, <code><nowiki>VLLM_PP_LAYER_PARTITION=16,16,16</nowiki></code> (the PLE layer stays on rank 0), <code><nowiki>VLLM_PLE_STAGED=1</nowiki></code>, <code><nowiki>QWEN4EXP_PLE_MODEL_DIR</nowiki></code>, <code><nowiki>VLLM_ALLOW_LONG_MAX_MODEL_LEN=1</nowiki></code>. Server on port 8080 (0.0.0.0, no auth), model name <code><nowiki>qwen3.8</nowiki></code>.
* KV tiering: OffloadingConnector with an 8 GiB CPU LRU tier in <code><nowiki>/dev/shm</nowiki></code> and a 128 GiB filesystem tier (<code><nowiki>max_bytes</nowiki></code> from vLLM PR #54327, backported). Blocks are content-addressed, so the cache survives restarts; a change of TP/PP/dtype/block size orphans the old directory, which is safe to delete by hand.
{{Source|handle=c133|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1556357466026025001|repo=https://github.com/clee/qwen3.8-flash-next-3x170hx}}
* Partition: the repo once shipped <code><nowiki>VLLM_PP_LAYER_PARTITION=17,17,14</nowiki></code> left over from MTP testing (with PP3 the MTP draft loads entirely on the last GPU); without MTP use 16,16,16. {{Source|handle=c133|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557888025868898426}}

=== The fork's patches ===
<code><nowiki>q38fn/units-fix</nowiki></code> (block-size units bug in <code><nowiki>_align_hybrid_block_size</nowiki></code>), <code><nowiki>q38fn/fp8-kv-sm80</nowiki></code> (uint8
cache storage plus software e4m3 decode in the QSA gather kernel), <code><nowiki>q38fn/ple-pipeline-parallel</nowiki></code> (PLE checkpoints
under PP>1, MTP draft on the last rank), <code><nowiki>q38fn/ple-nvme-staged</nowiki></code> (shard-direct <code><nowiki>preadv</nowiki></code> reader into a fixed
GPU staging buffer, cudagraph-safe), the #54327 filesystem-cap backport and a backport of upstream #58497.
{{Source|handle=c133|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1556357466026025001|repo=https://github.com/clee/qwen3.8-flash-next-3x170hx}}

== Measured numbers ==
Author, 2026-10-10, cold cache, single stream, temperature 0:
{| class="wikitable"
! Prompt tokens !! Prefill tok/s !! TTFT !! Decode tok/s
|-
| 1,026 || 2,592 || 0.40 s || 74.4
|-
| 16,393 || 4,278 || 3.83 s || 76.6
|-
| 64,845 || 4,831 || 13.42 s || 75.5
|-
| 258,992 || 4,947 || 52.35 s || 75.0
|-
| 1,036,890 || 4,748 || 218 s || 72.9
|}
Needle suite passes from 8k to 1M (809k prompt tokens). MTP is not enabled (its draft-layer KV would cost about 16% of
the pool).
{{Source|handle=c133|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1556357466026025001|repo=https://github.com/clee/qwen3.8-flash-next-3x170hx}}
* KV pool: "GPU KV cache size: 7,359,330 tokens, Maximum concurrency for 1,048,576 tokens per request: 7.02x" on 3 cards; 3,108,278 tokens (2.96x) on 2 cards after rebasing on newer vLLM. This depends on the PLE table living off the GPUs. {{Source|handle=c133|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557605645803585630}}<br />{{Source|handle=c133|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557913633806225440}}
* 68x pulled the branch onto three unlocked 8 GB cards: 52 to 60 tok/s on his single-user agentic workload. Later, with MTP (1 token) and an uncensored model: roughly 90 tok/s on 3 cards and 100 to 110 on 2 cards, with prefill only slightly lower on 2 cards. {{Source|handle=68x0|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557582593354899587}}<br />{{Source|handle=68x0|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557910994322333746}}<br />{{Source|handle=68x0|date=2026-10-11|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1558900482355896435}}
* {{Status|Contested}} MTP's benefit on this model: 68x needs it for 100+ tok/s on 2 cards; clee, on the INT4 AutoRound model, still measures better numbers without MTP on 2 cards. Mountains suggests 3 draft tokens if acceptance is good. {{Source|handle=mountains1605|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557912351758483467}}<br />{{Source|handle=c133|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560617193069215885}}
* Trust Inter, also on 3 cards, finds it a little slow against his own setup at above 110 tok/s decode (clee had not enabled MTP yet). {{Source|handle=trustinter|date=2026-10-05|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1556510298477830305}}

== Known issues ==
* Boot OOM in <code><nowiki>allocate_embedding_weight</nowiki></code> (cudaHostAlloc) means <code><nowiki>VLLM_PLE_STAGED</nowiki></code> is not active. "assert attn_page_size >= mamba_page_size" means the units fix is not mounted. "Padding mamba page size by ~8%" at block 1728 is healthy.
* A TP2+ NCCL hang means P2P trouble; this recipe is PP only.
* Pin mismatch inside the repo: the Dockerfile's default base and branch differ from what <code><nowiki>run-qwen38fn.sh</nowiki></code> and the README pin; the README says newer nightlies are not validated.
* No license file. Last commit 2026-10-10. API published on 0.0.0.0:8080 with no auth.
* The README calls the model 180B-A6B; other recipes say 125B MoE. Check the Qwen model card.
{{Source|handle=c133|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1556357466026025001|repo=https://github.com/clee/qwen3.8-flash-next-3x170hx}}

== Related ==
* Single-card sibling (same idea, native AIO): [[Qwen3.8 Flash Next on 1x 8 GB (iIIusi0n SSD PLE)]].
* Other two-card recipes: [[Qwen3.8 Flash Next on 2x 8 GB (ElectroHobo W4A16 TP2)]], [[Qwen3.8 Flash Next on 2x 8 GB (mannydelrio PP2)]].
* [[Engines]], [[Quantization on the 170HX]], [[Benchmarks]].

== Sources ==
* Repo: [https://github.com/clee/qwen3.8-flash-next-3x170hx clee/qwen3.8-flash-next-3x170hx] (no license stated), fork [https://github.com/clee/vllm-sm80 clee/vllm-sm80]. Design credit to tonyd2wild/Qwen38-Flash-Next-4x3090. Posted on Discord by clee (2026-10-04, update 2026-10-11) and Mountains.

{{ImportFooter}}
[[Category:8 GB]]