Qwen3.8 Flash Next on 1x 8 GB (ElectroHobo 3 bpw)

Qwen3.8-Flash-Next (125B MoE per the recipe)
Reported 1 x 8 GB
Quant	AutoRound mixed 2/3-bit (klee100, 2.998 effective bpw, 47 GiB on card) plus halt95's FP8 n-gram table in host RAM
Engine	Morrowmake/vllm-cmp170hx image with patch files mounted read-only recipe 0.2.1; engine 18305948ac
Topology	single card
Context	262,144
Decode	296.4 tok/s
Prefill	4243 tok/s
VRAM per card	64 GiB, --gpu-memory-utilization 0.95 (fixed)
Power cap	180 W
Driver	580 or newer (measured on 610.57.04)
Unlock build	cmpunlocker (64 GiB, 74 SMs)
Flags	`--tensor-parallel-size 1 --kv-cache-dtype bfloat16 --engram-config cpu_offload --max-num-seqs 48 --max-num-batched-tokens 3460 --block-size 912 cudagraph FULL_AND_PIECEWISE; DFlash 2 schedule [[1,2,7],[3,4,5],[5,48,3]]`
Tested	@nathang6866 on 2026-10-23
Source	link
Confirmations	None yet.

Qwen3.8-Flash-Next on one 64 GB card with the full 262,144-token context. It joins klee100's mixed 2/3-bit AutoRound checkpoint (47 GiB on the card) with halt95's FP8 n-gram table held in host RAM, or since 0.2.0 in a disk file read through HMM, and serves up to 48 requests. It is the one-card sibling of Qwen3.8 Flash Next on 2x 8 GB (ElectroHobo W4A16 TP2) and shares its engine, patches and table. Mountains called it the fastest Qwen Next on a single card he had seen (quality and quant not yet checked). Infobox numbers are one user on code. Source: repo github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24
Source: @mountains1605, 2026-10-27

Prerequisites

One CMP 170HX exposing 64 GiB at PCIe Gen2 x16 (Driver unlock, 8 GB card). Cards without the x16 mod are untested: n-gram rows cross PCIe every step.
96 GB of host RAM with the table in RAM (much less with the table on disk, which needs the open kernel modules with HMM and 48 GiB of disk).
Linux, driver 580 or newer, Docker with the NVIDIA Container Toolkit, about 100 GiB free on one filesystem.
Author's host: 2x Xeon E5-2697A v4 (Broadwell), 204 GiB RAM.

Source: repo github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24

Steps

git clone https://github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe
cd qwen38-flash-next-3bpw-1x-cmp170hx-recipe
./start.sh          # pull the engine image, fetch and build the model, start, wait for /health

Checkpoint: klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP at ce0e0b94 (routed experts 3-bit g128 on layers 0-11 and 36-47, 2-bit g64 on 12-35; attention 8-bit; MTP head 4/8-bit) without its 95 GiB BF16 table shard, joined by hard links with the FP8 table shards of halt95/Qwen3.8-Flash-Next-W4A16-Merlin at cec53e9f. About 97 GiB downloaded instead of 255; MODELS_DIR must be one filesystem.
Drafter incoai/Qwen3.8-Flash-Next-DFlash2 at a22a1744, 7 drafts for 1 to 2 requests, 5 for 3 to 4, 3 beyond. SPEC_MODE=mtp uses the checkpoint's MTP head with 3 drafts.
Serve settings: --tensor-parallel-size 1 --dtype bfloat16 --kv-cache-dtype bfloat16 --mamba-ssm-cache-dtype bfloat16 --attention-config '{"indexer_kv_dtype":"bf16"}' --engram-config '{"cpu_offload":true}' --max-model-len 262144 --gpu-memory-utilization 0.95 --max-num-seqs 48 --max-num-batched-tokens 3460 --prefix-match-unit 16 --block-size 912 --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}'; env VLLM_GDN_RECOVER=1 VLLM_USE_BREAKABLE_CUDAGRAPH=1 VLLM_QWEN4_QSA_TOPK_CANONICAL=1 VLLM_THIN_GEMM=1 VLLM_THIN_GEMM_GLUON=1 NCCL_P2P_DISABLE=1.
The first start compiles for about 12 minutes; the checkpoint loads in about 4 minutes from SSD and 31 minutes from a USB disk (set READY_TIMEOUT=7200). API at 127.0.0.1:8000, model qwen3.8-flash-next, no key unless API_KEY is set.

Source: repo github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24

Measured numbers

Fresh clone 2026-10-23, 180 W, PCIe Gen2 x16, 74 SMs, MiaAI-Lab protocol. Tok/s structured / code / prose:

1 user 197.8 / 296.4 / 135.5; 8 users 799.5 / 791.3 / 586.4 total; 48 users 1,384 total (MTP 1,403).
Cold prefill 4,243 tok/s; KV pool 292,784 tokens; code editing 371 tok/s.
HumanEval 161/164 and GSM8K (first 200) 195/200 with thinking on.
DFlash 2 is 53% faster on code for one user; MTP catches up from about 16 to 24 users.
Table on disk (0.2.0): the server holds about 6 GiB instead of a 48 GiB pinned table; a whole container capped at 14 GiB still did 723.7 / 758.9 / 468.1 tok/s at 8 users; cold prefill drops from 4,255 to 3,376 tok/s.
The card is power-bound at 180 W, sitting at 1,275 to 1,425 MHz during decode.

Source: repo github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24
Source: @nathang6866, 2026-10-24

Known issues

AutoRound loading bugs the mounted patches fix: per-layer expert bits were applied only to classes named fusedmoe (load failed with "The size of tensor a (40) must match the size of tensor b (60)"); the MTP layer index mismatch (mtp.layers.0 in the config, mtp.layers.48 in the engine); the engine refused an AutoRound checkpoint's n-gram table. A chunked copy-on-write fixes an OOM in copy_kv_cache_blocks_inplace under 32-request prefix-cache bursts in MTP mode.
Half the layers' routed experts are 2-bit; quality is scored only on HumanEval and GSM8K.
Community recipe, not endorsed by Morrowmake, klee100 or halt95; patches are pinned to engine commit 18305948ac.

Source: repo github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24

Sources

Repo: ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe (MIT; engine Apache-2.0; checkpoints Qwen Community License 1.0). Releases 0.1.0 (2026-10-24), 0.2.0 and 0.2.1 (2026-10-28).

Prerequisites

Steps

Measured numbers

Known issues

Related

Sources