Qwen3.8 Flash Next on 4x 8 GB

Qwen3.8-Flash-Next

Reported 4 x 8 GB
Quant	AWQ W4A16: wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16
Engine	wtdcode/vllm-backport (image lazymio/vllm-backport:latest-sm80) latest-sm80 tag as of 2026-10-16
Topology	TP4 with expert parallel, MTP 3
Context	1,000,000 (YaRN x4 from 262,144)
Decode	116.49 tok/s
Prefill	2101.4 tok/s
VRAM per card	64 GB
Flags	--tensor-parallel-size 4 --enable-expert-parallel --speculative-config mtp 3 --max-model-len 1000000 --max-num-seqs 8 --max-num-batched-tokens 2048 cudagraph FULL_AND_PIECEWISE --prefix-match-unit 16 --mamba-cache-mode align; VLLM_PLE_CPU_OFFLOAD=0
Tested	@matstatman on 2026-10-16
Source	Discord message
Confirmations	None yet. Confirm on your own card

Qwen3.8 Flash Next on four cards in one docker-compose file, on the wtdcode/vllm-backport image, with only 32 GB of host RAM: the n-gram (PLE) table stays on the GPUs (VLLM_PLE_CPU_OFFLOAD=0), so the host needs little memory. TP4 with expert parallel, MTP with 3 draft tokens, YaRN for a 1M-token context. matstatman calls it a huge upgrade over Qwen3.8 27B, which "thinks forever". Infobox decode is fresh code generation; prefill is the 32K run. Source: @matstatman, 2026-10-16
Source: @matstatman, 2026-10-16

Prerequisites

• Four CMP 170HX at 64 GB (Driver unlock, 8 GB card). matstatman's host: a used ASUS X99-E WS with an old CPU and 32 GB of RAM; one slot dropped from x16 to x8 during his testing.
• Docker with the NVIDIA runtime; the model downloaded first: hf download wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16.

Source: @matstatman, 2026-10-16
Source: @matstatman, 2026-10-16

Steps

His single docker-compose.yml is in the linked post (credit matstatman); it needs no other file. In summary:

• Image lazymio/vllm-backport:latest-sm80 with the NVIDIA runtime and host IPC, the Hugging Face cache mounted into the container, offline mode on.
• Environment: VLLM_WORKER_MULTIPROC_METHOD=spawn, VLLM_PLE_CPU_OFFLOAD=0 (PLE table on the GPUs), VLLM_ALLOW_LONG_MAX_MODEL_LEN=1, all four cards visible.
• Model wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16 with --tensor-parallel-size 4 --enable-expert-parallel.
• Speculative decoding: MTP with 3 draft tokens; CUDA graphs FULL_AND_PIECEWISE.
• Context: --max-model-len 1000000 through a --hf-overrides rope block (YaRN, factor 4.0 from 262,144 original positions, rope_theta 10,000,000, partial_rotary_factor 0.25, interleaved mrope with sections 11/11/10).
• Scheduler: --max-num-seqs 8 --max-num-batched-tokens 2048, chunked prefill and prefix caching on, --mamba-cache-mode align --prefix-match-unit 16.
• Agent features: reasoning parser qwen3, tool-call parser qwen3_xml with auto tool choice, prompt token details on.

Run it with docker compose up. As posted it publishes port 8000 on every interface with no API key; publish it on the loopback address only unless you need it on the network. On a 32 GB host vLLM warns that auto-prefetch is disabled because the 168.31 GiB checkpoint exceeds 90% of the available RAM; it still loads. For why --prefix-match-unit 16 and expert parallel matter on this model, see Engines#wtdcode/vllm-backport. Source: @matstatman, 2026-10-16
Source: @matstatman, 2026-10-16

Measured numbers

MTP 3, no DFlash or DSpark:

Test	Result
Prefill, about 8K / 32K / 100K prompt	2,053.9 / 2,101.4 / 2,088.3 tok/s
Fresh code, 400 tokens	116.49 tok/s, draft acceptance 79.4%, 3.38 tokens per pass
Edit-heavy, 3,000 tokens (two runs)	123.65 and 140.69 tok/s, acceptance 99.8%, 3.99 tokens per pass
Coding at 224.9K context (no temperature-0 trick)	about 75 tok/s

Source: @matstatman, 2026-10-16
Source: @matstatman, 2026-10-16

• clee asked why it was slower than his single-card 27B; the answer is the model: Miséricorde says Qwen3.8 Flash Next's architecture is ineffective for low-concurrency inference on these cards, and GLM 5.3 Flash on Morrowmake's recipe is 1.6x faster on four stock cards. Source: @c133, 2026-10-16
  Source: @wakeupmbrc1, 2026-10-16
• Morrowmake invited him to submit it to the community recipes section of the vllm-cmp170hx fork. Source: @voidgiraffe, 2026-10-16
• For context, matstatman later ran the normal 265k context on these four cards and found compaction fast enough not to need 1M. Source: @matstatman, 2026-10-20
  Source: @matstatman, 2026-10-20

Related

• Two cards with a DFlash2 drafter (faster per card): Qwen3.8 Flash Next on 2x 8 GB (ElectroHobo W4A16 TP2).
• Engines, Quantization on the 170HX, Benchmarks.

Sources

• matstatman's posts in clanker-talk, 2026-10-16.
• Engine: wtdcode/vllm-backport (Apache-2.0); model wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16.

Imported from the cmpunlocker Discord with credit; see Credits.