Jump to content
Toggle search
Toggle menu
CMP Wiki
Toggle preferences menu
Toggle personal menu
Not logged in
Your IP address will be publicly visible if you make any edits.

Qwen3.8 Flash Next on 2x 8 GB (ElectroHobo W4A16 TP2)

From CMP Wiki
More actions
Qwen3.8-Flash-Next (125B MoE per the recipe)
Proven 2 x 8 GB
QuantW4A16: halt95/Qwen3.8-Flash-Next-W4A16-Merlin (AutoRound INT4 g128 experts, INT8 GDN projections, FP8 n-gram table)
EngineMorrowmake/vllm-cmp170hx image (Qwen EXL3 release) with patch files mounted read-only recipe 0.3.1; engine 18305948ac
TopologyTP2 with whole-expert expert parallel
Context262,144
Decode349.7 tok/s
Prefill4696 tok/s
VRAM per card64 GiB, GPU_UTIL 0.95 (fixed)
Power cap180 W
Driver580 or newer (measured on 610.57.04); open modules with HMM for the table-on-disk mode
Unlock buildcmpunlocker (64 GiB, 74 SMs) with a P2P-capable build
Flags--tensor-parallel-size 2 --enable-expert-parallel --kv-cache-dtype bfloat16 --engram-config cpu_offload --max-num-seqs 128 --block-size 912 cudagraph FULL_AND_PIECEWISE; DFlash 2 drafter 7/5/3/2 by load
Tested@nathang6866 on 2026-10-23
Sourcelink
Confirmations
Reproduced by @darkmaniac7 (8 GB), 2026-10-28 · details

Qwen3.8-Flash-Next in W4A16 on two 64 GB cards with inco.ai's DFlash 2 drafter, full 262,144-token context, vision, reasoning and tool calls, and up to 128 concurrent requests. It runs Morrowmake's engine image as released with a few patch files mounted over it. It exists for old CPUs: the 48 GiB n-gram (PLE) table stays in host RAM (or, since 0.3.0, in a disk file the cards read through HMM), so the whole decode step runs as one CUDA graph instead of a Python-driven piecewise step. It is the fastest two-card recipe on code and prose and needs x16 links and P2P. Infobox numbers are release 0.2.0, one user on code. Source: repo github.com/ElectroHobo/qwen38-flash-next-w4a16-2x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24
Source: @nathang6866, 2026-10-20

Prerequisites

  • Two CMP 170HX exposing 64 GiB on PCIe Gen2 x16 (the x16 capacitor mod) with peer-to-peer, content-checked at every start (P2P=auto). See Driver unlock, Unlocks, 8 GB card.
  • An expert cannot be split across the two cards (INT4 scale groups are 128 wide, the expert inner width is 640) and the engine refuses pipeline parallel for the n-gram table, so TP2 with x16 links and P2P is required.
  • NVIDIA driver 580 or newer, Docker with the NVIDIA Container Toolkit, about 130 GiB of disk.
  • Host RAM: 96 GB with the table in RAM (48 GiB table, 64 GiB as allocated). With the table on disk (0.3.0) far less (the whole container ran capped at 22 GiB), but that needs the NVIDIA open kernel modules with HMM (nvidia-smi -q shows "Addressing Mode: HMM" on every card) and 48 GiB of SSD.
  • Author's host: 2x Xeon E5-2697A v4 (Broadwell), 204 GiB RAM.

Source: repo github.com/ElectroHobo/qwen38-flash-next-w4a16-2x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24

Steps

git clone https://github.com/ElectroHobo/qwen38-flash-next-w4a16-2x-cmp170hx-recipe
cd qwen38-flash-next-w4a16-2x-cmp170hx-recipe
./start.sh          # pull the engine image, fetch the model, start, wait for /health

The first start compiles kernels and runs the RecoverSSM verify (about 22 minutes, the verify about 95 s); later starts take 5 to 11 minutes. 0.3.1 sets PYTHONPYCACHEPREFIX=/cache/pycache, which cut start-to-/health from 414 s to 304 s.

Pins: engine Morrowmake/vllm-cmp170hx at 18305948acba3f4c4d912cdd56f2202d2ebd94e8, image ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79; checkpoint halt95/Qwen3.8-Flash-Next-W4A16-Merlin at cec53e9f (115.3 GiB); drafter incoai/Qwen3.8-Flash-Next-DFlash2 at a22a1744 (Apache-2.0, 1.1 GiB).

The container command: 

vllm serve /models/Qwen3.8-Flash-Next-W4A16-Merlin --tensor-parallel-size 2 --pipeline-parallel-size 1 \
  --enable-expert-parallel --dtype bfloat16 --kv-cache-dtype bfloat16 --load-format auto --mm-encoder-tp-mode data \
  --attention-config '{"indexer_kv_dtype":"bf16"}' --engram-config '{"cpu_offload":true}' \
  --max-model-len 262144 --gpu-memory-utilization 0.95 --max-num-seqs 128 --max-num-batched-tokens 3460 \
  --long-prefill-token-threshold 0 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --prefix-match-unit 16 --block-size 912 --mamba-ssm-cache-dtype bfloat16 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --speculative-config '{"method":"dflash","model":"/drafter","num_speculative_tokens":7,"num_speculative_tokens_per_batch_size":[[1,2,7],[3,4,5],[5,32,3],[33,128,2]]}'

Env: VLLM_GDN_RECOVER=1 VLLM_USE_BREAKABLE_CUDAGRAPH=1 VLLM_QWEN4_QSA_TOPK_CANONICAL=1 VLLM_CUSTOM_ALLREDUCE_FLAGS=1 VLLM_THIN_GEMM=1 VLLM_THIN_GEMM_GLUON=1 VLLM_FLA_AUTOTUNE_SKIP_SPILLING=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False CUDA_DEVICE_ORDER=PCI_BUS_ID, plus VLLM_CUSTOM_ALLREDUCE_PCIE_P2P=1 and NCCL_P2P_LEVEL=SYS after the P2P check passes. MAX_LEN and GPU_UTIL are refused if changed. Source: repo github.com/ElectroHobo/qwen38-flash-next-w4a16-2x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24

The three differences from Morrowmake's EXL3 recipe

--load-format auto (theirs exl3), --engram-config '{"cpu_offload":true}' (theirs false) and cudagraph_mode FULL_AND_PIECEWISE (theirs PIECEWISE). On a Broadwell host the EXL3 recipe's mid-step SSD n-gram read costs about 44 ms per step (55 to 70 tok/s for one user against its published 153); with the table in RAM and full graphs the step drops to 15 ms. PIECEWISE is about 3x slower per step on an old CPU. Source: repo github.com/ElectroHobo/qwen38-flash-next-w4a16-2x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24

Engine patches mounted over the image

DFlash 2 capture hook (Eagle3 auxiliary layers 6, 16, 26, 36, 46); RecoverSSM for the Gated DeltaNet layers (one recurrent state per request instead of one per draft position, checked bitwise; 128 requests fit at 94% of the pool where 28 of 48 did before; RECOVER_SSM=0 reverts); a shared n-gram table loader (PLE_SHARED_DIR on tmpfs lets several servers share one 48 GiB copy); chunked copy-on-write (256 MiB chunks, fixes a torch.OutOfMemoryError under bursts of shared prompts). All pinned to engine commit 18305948ac. Source: repo github.com/ElectroHobo/qwen38-flash-next-w4a16-2x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24

Measured numbers

Release 0.2.0, fresh clone 2026-10-23, 74 SMs, 180 W, PCIe x16, P2P on, MiaAI-Lab protocol. Tok/s structured / code / prose:

DFlash 2 (default) MTP (SPEC_MODE=mtp)
1 user 241.2 / 349.7 / 198.4 208.1 / 242.9 / 184.1
2 users 374.4 / 489.3 / 288.4 330.4 / 387.7 / 299.0
4 users 556.2 / 730.8 / 489.1 538.2 / 569.0 / 476.5
8 users 894.5 / 986.8 / 769.7 841.7 / 829.3 / 728.4
128 users (700 tokens) 1,937 1,966
Cold prefill 4,696 4,520
KV pool 1,475,836 1,438,968

Code editing, one user: 495 tok/s. Quality (thinking on): HumanEval 162/164, GSM8K 198/200 (first 200). DFlash 2 is 44% faster than MTP on code for one user and 83% faster editing code; MTP pulls ahead from about 32 users.

Table on disk (0.3.0), disk against RAM: 1 user 232.8 / 344.4 / 199.3 against 234.9 / 358.6 / 200.8; 128 users 1,851 against 1,934; cold prefill 4,000 against 4,723. FP8 KV (not default): pool +73% (2,578,229 tokens), one-user decode -8 to -10%, answers shift. KV offload to a 64 GiB RAM tier brings an evicted 100k-token context back in 0.89 s instead of 21 to 22 s of re-prefill. Source: repo github.com/ElectroHobo/qwen38-flash-next-w4a16-2x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24

  • Release 0.1.0 (2026-10-20): 253 / 349 / 203 tok/s for one user, 497 tok/s editing code, 842 / 992 / 727 at 8 concurrent, 1.4M-token KV. Miséricorde confirmed the inco.ai drafter, published that day, actually helps. Source: @nathang6866, 2026-10-20
    Source: @wakeupmbrc1, 2026-10-20
  • The 2026-10-24 update tuned for concurrency: 1,814 tok/s at 128 requests. Source: @nathang6866, 2026-10-24
  • Ivan ran release 0.2.0 on his box (Gen2 x16, P2P, 1,024-token answers): one user code 263, structured 266, prose 215; 8 users 916 / 961 / 729; prefill 4.6k on a 17k prompt; 53% draft acceptance; vision works. Four instances at once (two cards each, one shared 48 GiB table in /dev/shm, 77 GiB of host RAM for all four) each kept 90 to 100% of their lone speed: box aggregate at 32 users 3,617 / 3,316 / 2,664 tok/s and 17.4k tok/s prefill. Source: @darkmaniac7, 2026-10-28 (lab notes)

Other two-card recipes on the same protocol

Numbers from their READMEs (as of 2026-10-19), not re-run by ElectroHobo:

This recipe (TP2, x16) AlexBrandes (PP2, x4, table on NVMe) mannydelrio (PP2, x4, 4-bit table on SSD)
1 user 241.2 / 349.7 / 198.4 250.8 / 226.2 / 157.4 192.4 / 177.8 / 145.4
8 users 894.5 / 986.8 / 769.7 999.5 / 840.4 / 566.8 986.4 / 734.2 / 619.6
Cold prefill 4,696 6,300 to 7,100 7,006

This recipe leads on code and prose; the PP2 recipes read prompts 33 to 50% faster and run on stock x4 cards. See Qwen3.8 Flash Next on 2x 8 GB (mannydelrio PP2). Source: repo github.com/ElectroHobo/qwen38-flash-next-w4a16-2x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24

Known issues

  • Decode is power-bound at 180 W (cards at 1,400 to 1,499 MHz); clock locks do nothing.
  • Community recipe, not tested or endorsed by Morrowmake or halt95; measured on one Broadwell host. The engine image is pinned by digest and the patches to engine commit 18305948ac, so a new engine release needs them re-made. 0.3.0 (table on disk) had not served real traffic at release.
  • KV_OFFLOAD_DIR is write-through and never pruned (one benchmark run wrote 58 GB).
  • P2P on other unlock builds. galygious2 found the recipe promising for x16-modded cards but saw it needs P2P, and asked whether anyone has P2P working with Amogh's unlock. ElectroHobo had P2P working with an older method, and thinks Morrowmake's vLLM fixes it for many people on supported hardware. Setups where P2P still fails are on P2P and Risers and bifurcation. Source: @galygious2, 2026-10-25
    Source: @nathang6866, 2026-10-25

Source: repo github.com/ElectroHobo/qwen38-flash-next-w4a16-2x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24

  • Keep vLLM's custom all-reduce on with this recipe: on Ivan's box turning it off cost 17 to 25% single-stream and 12 to 15% at 8 users. This is the opposite of what he measured for GLM 5.3 Flash TP4 (see GLM 5.3 Flash on 4x 8 GB). Source: @darkmaniac7, 2026-10-28 (lab notes)
  • Copying another instance's cache/ directory can carry stale build locks that hang custom all-reduce at init; let each instance compile its own. Source: @darkmaniac7, 2026-10-27 (lab notes)

Variants

  • Uncensored base in the same Merlin layout: ntmcgee/Orca-Qwen3.8-Flash-Next-W4A16-Merlin (its FP8 table differs byte-wise, so it needs its own 48 GiB shared table), or halt95/Swift1.5-Huihui-Qwen3.8-Flash-Next-W4A16-Merlin, which shares the base table. Neither had been run on this engine as of 2026-10-27. Switch with TARGET_REPO, TARGET_REVISION and MODEL in .env. Source: @darkmaniac7, 2026-10-27 (lab notes)
  • One card: Qwen3.8 Flash Next on 1x 8 GB (ElectroHobo 3 bpw) (shares the engine, patches and table).

Sources

Imported from the cmpunlocker Discord with credit; see Credits.
Retrieved from "https://cmp.wiki/w/index.php?title=Qwen3.8_Flash_Next_on_2x_8_GB_(ElectroHobo_W4A16_TP2)&oldid=471"