Qwen3.8 Flash Next on 1x 8 GB (iIIusi0n SSD PLE)

Qwen3.8-Flash-Next
Proven 1 x 8 GB
Quant	AutoRound 3 bpw + MTP: klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP; 95.37 GiB BF16 PLE table on SSD
Engine	vLLM a5a30471 plus a 13-file downstream patch (native Linux AIO PLE reader) 0.29.1rc1.dev402+ga5a30471f.ple1
Topology	single card
Context	262,144 advertised (tested through 32K)
Decode	110.99 tok/s
Prefill	2636 tok/s
VRAM per card	64 GiB, --gpu-memory-utilization 0.96
Power cap	180 W
Driver	610.43.03
Unlock build	cmpunlocker (64 GiB)
Flags	`--max-num-seqs 16 --max-num-batched-tokens 2048 --gpu-memory-utilization 0.96 --mamba-cache-mode align --enable-prefix-caching cudagraph FULL_AND_PIECEWISE; MTP 1; ple_ssd_offload true, ple_ssd_io_depth 256, ple_ssd_cache_mb 512`
Tested	@iIIusi0n on 2026-09-19
Source	link
Confirmations	Reproduced by @dottoreargus (8 GB), 2026-10-07 · details Reproduced by @hogg7701 (8 GB), 2026-10-07 · details Reproduced by @tekkenz (8 GB), 2026-10-06 · details

Qwen3.8-Flash-Next on one 64 GB card with only 15 GiB of host RAM: a downstream vLLM patch keeps the 95.37 GiB BF16 PLE (n-gram) table on SSD and streams rows with native Linux AIO. It was the first single-card Qwen3.8 Flash Next recipe and is listed on localairecipes as "Qwen3.8-Flash-Next 3bpw MTP on 1x CMP 170HX SSD PLE AIO (iIIusi0n)". Asked on 2026-10-27 for the go-to model on one 64 GB card, 68x answered Qwen3.8 Flash Next with PLE patches and Mountains named this recipe. Infobox numbers are the author's single-request decode and fresh 8K prefill. Source: repo github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx by iIIusi0n on GitHub, posted by @c133, 2026-10-09
Source: @c133, 2026-10-09
Source: @68x0, 2026-10-27
Source: @mountains1605, 2026-10-27

Prerequisites

One CMP 170HX at 64 GiB (Driver unlock, 8 GB card).
About 143 GiB of SSD for the checkpoint. The author's test machine was a VM with 4 vCPU, 15 GiB RAM, 31 GiB swap and an enterprise SATA SSD.
Python 3.12, torch 2.13.0 (cu130), CUDA toolkit 13.3, transformers 5.17.0, triton 3.7.1, flashinfer-python 0.6.18.post1.

Source: repo github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx by iIIusi0n on GitHub, posted by @c133, 2026-10-09

Steps

Follow the repo's docs/GUIDE.md:

Check out vLLM at a5a30471ff2bb7f0824f2da10e358af98d304472 (precompiled wheel from wheels.vllm.ai) into ~/vllm/.venv.
Apply patches/qwen38-ple-ssd.patch (sha256 09c4e13fbd2b77169a7f8cb28d36242cc6e3916fe9c5a3c25d8e90101a38ccb0, 13 files) and build ple_ssd_io.so with plain cc (no libaio or liburing). It adds async PLE reads with read-ahead, metadata-only loading, CUDA-graph host breaks, mixed-bit expert handling and MTP layer remapping.
Download klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP at ce0e0b94 (13 files, 153,012,762,024 bytes). An uncensored build exists: klee100/Qwen3.8-Flash-Next-Uncensored-AutoRound-3bpw-MTP (from OrcaRouter's abliterated base).
Launch with the repo's launcher. Defaults: MTP 1, 2K prefill chunks, 16 sequences, 512 MiB row cache. Knobs: QWEN_MTP, QWEN_BATCH_TOKENS, QWEN_SEQS, QWEN_GRAPH_MODE, QWEN_GPU_MEMORY, QWEN_CONTEXT, QWEN_SSD_DEPTH, QWEN_SSD_CACHE_MB, QWEN_SSD_PREFETCH (16384), QWEN_OMP_THREADS. API at http://localhost:8000/v1, model Qwen3.8-Flash-Next. Model load plus graph capture takes 8 to 10 minutes.

Source: repo github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx by iIIusi0n on GitHub, posted by @c133, 2026-10-09

Tuning from users

dottoreargus (single 8 GB card) reached 5K+ tok/s prefill with these changes on top of the repo, and warns that part of the gap may be measurement:

Prefill batch 2,048 to 4,096; MTP draft tokens 1 to 2; PLE row cache 512 MiB to 8,192 MiB; PLE AIO queue depth 256 to 512.
Card tuning: HBM NDIV 67, +250 MHz core clock offset, 200 W power limit (see Tools).

Source: @dottoreargus, 2026-10-07

Measured numbers

Author's numbers, 2026-09-19, three-round medians, 128 output tokens, temperature 0, thinking off:

Workload	Result
Single-request decode	110.99 tok/s (TTFT 0.105 s)
Aggregate output, 4 / 8 / 16 requests	277 / 429 / 667 tok/s
Fresh prefill at 512 / 2K / 8K / 32K	1,616 / 1,915 / 2,636 / 2,132 tok/s

KLD against the original weights 0.1208. Mean GPU power 176.5 W against the 180 W cap. Exposed SSD wait is 36% of wall time on fresh 8K prefills but only 4 to 8% during decode, so prefill is SSD-bound and decode is GPU-bound. 31 SSD and graph tests passed; 134 requests with zero API errors. Source: repo github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx by iIIusi0n on GitHub, posted by @c133, 2026-10-09

Other people's runs

tekkenz served the 3 bpw checkpoint with this patched vLLM and had to set MTP=3; prefill was 3.5k tok/s with a 2,048 chunk size. Source: @tekkenz, 2026-10-06
Source: @tekkenz, 2026-10-06
hogg gets 3K+ tok/s prefill on a single card with the repo. Source: @hogg7701, 2026-10-07
dottoreargus has used the 3 bpw quant for one to two weeks of reverse-engineering work without issues and gets 5K+ prefill with the tuning above. Source: @dottoreargus, 2026-10-06
clee: a friend runs it on a single card and reports it works well. Source: @c133, 2026-10-09

Known issues

An arithmetic prompt returns a wrong answer (unresolved model or runtime consistency issue).
262,144-token context is advertised but timing was only tested through 32K.
The upstream FP8 pinned-memory PLE kernel does not compile on SM80, so the BF16 SSD path is used. CUPTI profiling is unavailable on this GPU.
Results come from a VM with a SATA SSD, so prefill numbers are storage-bound. The patch targets one pinned vLLM commit and is not mergeable upstream as is.

Source: repo github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx by iIIusi0n on GitHub, posted by @c133, 2026-10-09

Sources

Repo: iIIusi0n/qwen3.8-flash-next-cmp170hx (Apache-2.0; model Qwen Community License 1.0), last commit 2026-09-24. Checkpoints published by klee100 on Hugging Face. Posted on Discord by clee.

Prerequisites

Steps

Tuning from users

Measured numbers

Other people's runs

Known issues

Related

Sources