View source for Qwen3.8 Flash Next on 4x 8 GB

{{Recipe
|model=Qwen3.8-Flash-Next
|quant=AWQ W4A16: wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16
|engine=wtdcode/vllm-backport (image lazymio/vllm-backport:latest-sm80)
|engine_version=latest-sm80 tag as of 2026-10-16
|cards=8 GB
|card_count=4
|topology=TP4 with expert parallel, MTP 3
|context=1,000,000 (YaRN x4 from 262,144)
|decode_tps=116.49
|prefill_tps=2101.4
|vram_per_card=64 GB
|flags=--tensor-parallel-size 4 --enable-expert-parallel --speculative-config mtp 3 --max-model-len 1000000 --max-num-seqs 8 --max-num-batched-tokens 2048 cudagraph FULL_AND_PIECEWISE --prefix-match-unit 16 --mamba-cache-mode align; VLLM_PLE_CPU_OFFLOAD=0
|status=Reported
|tested_by=matstatman
|tested_on=2026-10-16
|source=https://discord.com/channels/1518634379604136178/1528826330454429706/1560602001908170753
}}
Qwen3.8 Flash Next on four cards in one docker-compose file, on the wtdcode/vllm-backport image, with only 32 GB of host
RAM: the n-gram (PLE) table stays on the GPUs (<code><nowiki>VLLM_PLE_CPU_OFFLOAD=0</nowiki></code>), so the host needs little memory.
TP4 with expert parallel, MTP with 3 draft tokens, YaRN for a 1M-token context. matstatman calls it a huge upgrade over
Qwen3.8 27B, which "thinks forever". Infobox decode is fresh code generation; prefill is the 32K run.
{{Source|handle=matstatman|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560598017252528190}}<br />
{{Source|handle=matstatman|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560599247475314711}}

== Prerequisites ==
* Four CMP 170HX at 64 GB ([[Driver unlock]], [[8 GB card]]). matstatman's host: a used ASUS X99-E WS with an old CPU and 32 GB of RAM; one slot dropped from x16 to x8 during his testing.
* Docker with the NVIDIA runtime; the model downloaded first: <code><nowiki>hf download wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16</nowiki></code>.
{{Source|handle=matstatman|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560602515006029936}}<br />
{{Source|handle=matstatman|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560618524995489825}}

== Steps ==
His single <code><nowiki>docker-compose.yml</nowiki></code> is in the linked post (credit matstatman); it needs no other file. In
summary:
* Image <code><nowiki>lazymio/vllm-backport:latest-sm80</nowiki></code> with the NVIDIA runtime and host IPC, the Hugging Face cache mounted into the container, offline mode on.
* Environment: <code><nowiki>VLLM_WORKER_MULTIPROC_METHOD=spawn</nowiki></code>, <code><nowiki>VLLM_PLE_CPU_OFFLOAD=0</nowiki></code> (PLE table on the GPUs), <code><nowiki>VLLM_ALLOW_LONG_MAX_MODEL_LEN=1</nowiki></code>, all four cards visible.
* Model <code><nowiki>wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16</nowiki></code> with <code><nowiki>--tensor-parallel-size 4 --enable-expert-parallel</nowiki></code>.
* Speculative decoding: MTP with 3 draft tokens; CUDA graphs <code><nowiki>FULL_AND_PIECEWISE</nowiki></code>.
* Context: <code><nowiki>--max-model-len 1000000</nowiki></code> through a <code><nowiki>--hf-overrides</nowiki></code> rope block (YaRN, factor 4.0 from 262,144 original positions, rope_theta 10,000,000, partial_rotary_factor 0.25, interleaved mrope with sections 11/11/10).
* Scheduler: <code><nowiki>--max-num-seqs 8 --max-num-batched-tokens 2048</nowiki></code>, chunked prefill and prefix caching on, <code><nowiki>--mamba-cache-mode align --prefix-match-unit 16</nowiki></code>.
* Agent features: reasoning parser <code><nowiki>qwen3</nowiki></code>, tool-call parser <code><nowiki>qwen3_xml</nowiki></code> with auto tool choice, prompt token details on.
Run it with <code><nowiki>docker compose up</nowiki></code>. As posted it publishes port 8000 on every interface with no API key; publish
it on the loopback address only unless you need it on the network. On a 32 GB host vLLM warns that auto-prefetch is disabled because the
168.31 GiB checkpoint exceeds 90% of the available RAM; it still loads. For why <code><nowiki>--prefix-match-unit 16</nowiki></code> and
expert parallel matter on this model, see [[Engines#wtdcode/vllm-backport]].
{{Source|handle=matstatman|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560602001908170753}}<br />
{{Source|handle=matstatman|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560603219108044962}}

== Measured numbers ==
MTP 3, no DFlash or DSpark:
{| class="wikitable"
! Test !! Result
|-
| Prefill, about 8K / 32K / 100K prompt || 2,053.9 / 2,101.4 / 2,088.3 tok/s
|-
| Fresh code, 400 tokens || 116.49 tok/s, draft acceptance 79.4%, 3.38 tokens per pass
|-
| Edit-heavy, 3,000 tokens (two runs) || 123.65 and 140.69 tok/s, acceptance 99.8%, 3.99 tokens per pass
|-
| Coding at 224.9K context (no temperature-0 trick) || about 75 tok/s
|}
{{Source|handle=matstatman|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560612367220342900}}<br />
{{Source|handle=matstatman|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560612878304673793}}

* clee asked why it was slower than his single-card 27B; the answer is the model: Miséricorde says Qwen3.8 Flash Next's architecture is ineffective for low-concurrency inference on these cards, and GLM 5.3 Flash on Morrowmake's recipe is 1.6x faster on four stock cards. {{Source|handle=c133|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560616484349153341}}<br />{{Source|handle=wakeupmbrc1|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560617105676574762}}
* Morrowmake invited him to submit it to the community recipes section of the vllm-cmp170hx fork. {{Source|handle=voidgiraffe|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560598988779167825}}
* For context, matstatman later ran the normal 265k context on these four cards and found compaction fast enough not to need 1M. {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562214943703568550}}<br />{{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562215886914588815}}

== Related ==
* Two cards with a DFlash2 drafter (faster per card): [[Qwen3.8 Flash Next on 2x 8 GB (ElectroHobo W4A16 TP2)]].
* [[Engines]], [[Quantization on the 170HX]], [[Benchmarks]].

== Sources ==
* matstatman's posts in clanker-talk, 2026-10-16.
* Engine: [https://github.com/wtdcode/vllm-backport wtdcode/vllm-backport] (Apache-2.0); model [https://huggingface.co/wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16 wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16].

{{ImportFooter}}
[[Category:8 GB]]