Qwen3.8 Flash Next on 2x 8 GB (Morrowmake EXL3)
More actions
| Reported 2 x 8 GB | |
| Quant | EXL3 5.05 bpw; PLE (n-gram) table read from SSD |
|---|---|
| Engine | Morrowmake/vllm-cmp170hx (EXL3 path) tag qwen38-recipe-1.0.0, engine 18305948ac |
| Topology | TP2 with expert parallel |
| Context | 262,144 |
| Decode | 152.6 tok/s |
| Prefill | 2651 tok/s |
| VRAM per card | 64 GiB, --gpu-memory-utilization 0.95 |
| Flags | --quantization exl3 --load-format exl3 --tensor-parallel-size 2 --enable-expert-parallel --engram-config cpu_offload false, VLLM_EXL3_PLE_PLACEMENT=ssd, cudagraph PIECEWISE, MTP 3 |
| Tested | @voidgiraffe on 2026-10-16 |
| Source | link |
| Confirmations | None yet. |
Morrowmake's Qwen3.8-Flash-Next recipe on two cards with his vLLM fork's EXL3 path (release 1.0.0, engine tag
qwen38-recipe-1.0.0). He chose 5.05 bpw because a 1M-token KV cache still fit, and said on 2026-10-16 that
W4A16 is probably a better choice than EXL3. The ElectroHobo two-card recipe
descends from its scripts and runs the same engine image. Infobox numbers are the published one-user code figure and
cold prefill, as quoted in ElectroHobo's README.
Source: @voidgiraffe, 2026-10-15
Source: @voidgiraffe, 2026-10-16
Source: repo github.com/ElectroHobo/qwen38-flash-next-w4a16-2x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24
Morrowmake released this recipe as a first attempt alongside the GLM update, after two weeks spent moving his custom code behind vLLM plugins. On x4 links he expects it to hurt prefill (untested) and planned a PP2 version. Source: @voidgiraffe, 2026-10-15
Source: @voidgiraffe, 2026-10-15
Source: @voidgiraffe, 2026-10-16
Prerequisites
Two CMP 170HX at 64 GiB and 74 SMs on x16 links with P2P, Docker with the NVIDIA Container Toolkit, driver 580 or newer; a fast CPU, because the PLE table is read from SSD inside each step (see Known issues). See Driver unlock, 8 GB card, Unlocks.
Steps
The recipe repo follows the same pattern as Morrowmake's GLM and DeepSeek recipes (clone, ./start.sh), pinned to
engine commit 18305948ac and image
ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79.
The engine's documentation gives the two-card command:
CUDA_VISIBLE_DEVICES=0,1 VLLM_EXL3_PLE_PLACEMENT=ssd VLLM_USE_BREAKABLE_CUDAGRAPH=1 \
vllm serve <exl3 checkpoint> --quantization exl3 --load-format exl3 --mm-encoder-tp-mode data \
--tensor-parallel-size 2 --pipeline-parallel-size 1 --enable-expert-parallel --dtype bfloat16 \
--kv-cache-dtype bfloat16 --attention-config '{"indexer_kv_dtype":"bf16"}' --engram-config '{"cpu_offload":false}' \
--max-model-len 262144 --gpu-memory-utilization 0.95 --compilation-config '{"cudagraph_mode":"PIECEWISE"}' \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
Optional: --hf-overrides with text_config.moe_router_dtype float32 and mtp_hidden_norm per_stream.
The EXL3 extension is built for SM 8.0 only; several dense paths are validated for 74-SM cards at TP2.
Source: repo github.com/Morrowmake/vllm-cmp170hx by Morrowmake on GitHub, posted by @voidgiraffe, 2026-10-15
Measured numbers
Morrowmake's published figures (his box, a current CPU), as quoted by ElectroHobo; tok/s structured / code / prose:
| 1 user | 2 users | 4 users | 8 users | Cold prefill | KV pool |
|---|---|---|---|---|---|
| 152.8 / 152.6 / 118.3 | 236.5 / 265.0 / 204.6 | 357.1 / 458.6 / 327.2 | 588.2 / 674.7 / 489.9 | 2,651 | 1,044,325 |
Source: repo github.com/ElectroHobo/qwen38-flash-next-w4a16-2x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24
- Before release: about 150 tok/s single stream on EXL3 5.05 bpw, aiming near 200 single and over 600 multi-stream. 4 bpw would likely not be much faster. Source: @cibernox, 2026-10-15
Source: @voidgiraffe, 2026-10-15
Known issues
- Contested CPU dependence. On ElectroHobo's dual Broadwell host the mid-step SSD n-gram read costs about 44 ms per step and the recipe runs at 55 to 70 tok/s for one user against the published 153; PIECEWISE graphs are about 3x slower per step on an old CPU. Ivan's notes count it CPU-bound for the same reason. His W4A16 recipe keeps the table in host RAM with full CUDA graphs instead. Source: repo github.com/ElectroHobo/qwen38-flash-next-w4a16-2x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24
Source: @darkmaniac7, 2026-10-27 (lab notes) - EXL3 decode on these cards is compute-bound; see Quantization on the 170HX.
Related
- Qwen3.8 Flash Next on 2x 8 GB (ElectroHobo W4A16 TP2) (descends from this recipe, faster), Engines, Benchmarks.
Sources
- Recipe: Morrowmake/qwen38-flash-next-cmp170hx-recipe (release 1.0.0, engine tag
qwen38-recipe-1.0.0); announcement on X linked from Discord. The recipe repo itself was not cloned for this page; numbers are quoted from ElectroHobo's README.