Running Qwen3.8 Flash Next locally: hardware and measured speeds
More actions
Short answer. Qwen3.8-Flash-Next (125B MoE with 6B active, plus a 51B n-gram table) runs on a single 8 GB card unlocked to 64 GB. One card measures 111 to 296 tok/s for one user on code with the full 262,144-token context. Two cards measure 75 to 350 tok/s depending on the recipe, and the fastest two-card recipe serves up to 128 concurrent requests at 1,937 tok/s in total. Three and four-card recipes trade single-user speed for a 1M-token context and larger KV pools. Every row below links to the recipe page with the exact commands.
The main choice is where the n-gram table lives (on the cards, in host RAM, or on SSD), because that decides how much host RAM you need. Speeds are from each recipe's own benchmark, usually one user on code at temperature 0 with a speculative drafter on.
Hardware it needs
- Cards. One unlocked card holds a 3-bit quant (47 GiB on the card) with the n-gram table kept off the card. Two cards hold W4A16 weights. Cards must show 64 GiB each: see Driver unlock.
- The n-gram (PLE) table. It is 95.4 GiB in BF16, 47.7 GiB in FP8 and 26.8 to 29.8 GiB in INT4. Recipes keep it in host RAM (fastest lookups), on SSD or NVMe (little host RAM), or on the cards (four-card recipe).
- Host RAM. From 15 GiB (one card, table on SSD) and 16 GB (two cards, INT4 table on SSD) up to 96 GB (one card, FP8 table in RAM). The four-card recipe with the table on the cards runs with 32 GB.
- Links. The pipeline-parallel recipes run without P2P, most of them on stock x4 links. The tensor-parallel two-card recipes need working P2P, and the fastest one also needs Gen2 x16 links (PCIe x16 capacitor mod).
Source: repo github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24
Source: repo github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx by iIIusi0n on GitHub, posted by @c133, 2026-10-09
Source: repo github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx by mannydelrio on GitHub, posted by @mrmannyatx, 2026-10-06
Source: @matstatman, 2026-10-16
Source: @darkmaniac7, 2026-10-27 (lab notes)
Measured configurations
One row per recipe page. "One user" is the recipe's headline single-stream decode figure; "Many users" is the total across concurrent requests; the prompt length behind a prefill figure is in brackets where the page gives it. Power caps, link widths and benchmark protocols differ between rows, so compare rows only after reading the recipe page. The status badge is the one each recipe page carries: Proven means a second person reproduced it.
| Cards | Recipe | Weights | Layout | One user (tok/s) | Many users (tok/s) | Prefill (tok/s) | Context | Power cap | Status | Measured by |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Qwen3.8 Flash Next on 1x 8 GB (ElectroHobo 3 bpw) | AutoRound mixed 2/3-bit (47 GiB on the card); FP8 n-gram table in host RAM or on disk | single card on an x16 link; DFlash2 drafter | 296.4 | 1,384 at 48 users | 4,243 | 262,144 | 180 W | Reported | nathang6866 |
| 1 | Qwen3.8 Flash Next on 1x 8 GB (iIIusi0n SSD PLE) | AutoRound 3 bpw; BF16 n-gram table on SSD | single card; MTP 1 | 110.99 | 667 at 16 users | 2,636 (8K tokens) | 262,144 advertised (tested through 32K) | 180 W | Proven | iIIusi0n |
| 2 | Qwen3.8 Flash Next on 2x 8 GB (ElectroHobo W4A16 TP2) | W4A16; FP8 n-gram table in host RAM or on disk | TP2 with expert parallel on x16 links, P2P required; DFlash2 drafter | 349.7 | 1,937 at 128 users | 4,696 | 262,144 | 180 W | Proven | nathang6866 |
| 2 | Qwen3.8 Flash Next on 2x 8 GB (Plean PP2) | mixed INT4 AutoRound (about 65 GB); BF16 n-gram table on NVMe | PP2, no P2P; MTP 3 | 197 (predictable output) | 5,800 | 262,144 | Proven | plean. | ||
| 2 | Qwen3.8 Flash Next on 2x 8 GB (mannydelrio PP2) | AWQ W4A16 g32 of a refusal-removed variant; INT4 n-gram table on SSD | PP2 on stock x4 links, no P2P; MTP 3 | 177.8 | 734.2 at 8 users | 7,006 (24K to 38K tokens) | 262,144 | 180 W | Reported | mrmannyatx |
| 2 | Qwen3.8 Flash Next on 2x 8 GB (Qwen-Flash-SM80, GDS TEP2) | NVFP4 through INT4 kernels; FP8 n-gram table on NVMe | TP2 with expert parallel, P2P required; MTP 6 | 168.29 | 749.67 at 10 users | 1,986 (8K tokens) | 262,144 | Reported | nguyenthimy2022kg-alt | |
| 2 | Qwen3.8 Flash Next on 2x 8 GB (Morrowmake EXL3) | EXL3 5.05 bpw; n-gram table on SSD | TP2 with expert parallel on x16 links, P2P required; MTP 3 | 152.6 | 674.7 at 8 users | 2,651 | 262,144 | Reported | voidgiraffe | |
| 2 | Qwen3.8 Flash Next on 2x 8 GB (DUFRelic PP2) | W4A16 with an INT4 n-gram table in host RAM | PP2; MTP 3 | 133.39 | 245.17 at 4 users | 6,214 (16384 tokens) | 262,144 | 140 W | Reported | dufrelic |
| 2 | Qwen3.8 Flash Next on 2x 8 GB (vektorprime PP2) | AWQ INT4; BF16 n-gram table in host RAM | PP2; MTP 1 | 75.4 | 262,144 | Reported | vektor_prime | |||
| 3 | Qwen3.8 Flash Next on 3x 8 GB (clee PP3) | W4A16 AutoRound; BF16 n-gram table streamed from NVMe | PP3 on stock x4 links, no P2P; no drafter | 74.4 | 4,831 (64,845 tokens) | 1,048,576 (YaRN x4) | Proven | c133 | ||
| 3 or 4 | Qwen3.8 Flash Next on 3x or 4x 8 GB (zebgop-ops FP8 PP) | official FP8 (about 126 GiB); FP8 n-gram table in host RAM | PP4 (PP3 on three cards) on stock x4 links, no P2P; MTP 3 | 89 (four cards) | 442 at 8 users | 11,100 (32k tokens) | 262,144 native, 1,048,576 with YaRN | 200 W | Reported | flipnburner |
| 4 | Qwen3.8 Flash Next on 4x 8 GB | AWQ W4A16; n-gram table on the cards | TP4 with expert parallel; MTP 3 | 116.49 | 2,101 (42,584 tokens) | 1,000,000 (YaRN x4) | Reported | matstatman |
Sources for the rows (each recipe page carries its full credits):
Source: repo github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24
Source: repo github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx by iIIusi0n on GitHub, posted by @c133, 2026-10-09
Source: repo github.com/ElectroHobo/qwen38-flash-next-w4a16-2x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24
Source: @plean., 2026-10-07
Source: repo github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx by mannydelrio on GitHub, posted by @mrmannyatx, 2026-10-06
Source: repo github.com/nguyenthimy2022kg-alt/Qwen-Flash-SM80-170HX by nguyenthimy2022kg-alt on GitHub, posted by @jovial_dolphin_29796, 2026-09-09
Source: @voidgiraffe, 2026-10-15
Source: repo github.com/DUFRelic/qwen38-flash-next-recipe by DUFRelic on GitHub, posted by @dufrelic, 2026-10-05
Source: repo github.com/vektorprime/qwen38-flash-next-pp2 by vektorprime on GitHub, posted by @vektor_prime, 2026-10-07
Source: repo github.com/clee/qwen3.8-flash-next-3x170hx by clee on GitHub, posted by @c133, 2026-10-04
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub, posted by @flipnburner, 2026-08-29
Source: @matstatman, 2026-10-16
Picking a recipe by what you have
- One card, plenty of host RAM, x16 link: the 3 bpw recipe at 296.4 tok/s for one user on code and 1,384 tok/s across 48 users. It needs 96 GB of host RAM with the table in RAM, much less with the table on disk. Source: repo github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24
- One card, little host RAM: the SSD-table recipe at 110.99 tok/s with only 15 GiB of host RAM. Source: repo github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx by iIIusi0n on GitHub, posted by @c133, 2026-10-09
- Two cards with x16 links and P2P: the W4A16 TP2 recipe at 349.7 tok/s for one user on code, 4,696 tok/s prefill and up to 128 concurrent requests. The lab ran release 0.2.0 on its own server (Gen2 x16, P2P, 1,024-token answers) and measured 263 tok/s for one user on code, with four instances on eight cards each keeping 90 to 100% of their lone speed. Source: repo github.com/ElectroHobo/qwen38-flash-next-w4a16-2x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24
Source: @darkmaniac7, 2026-10-28 (lab notes) - Two stock cards on x4 links, no P2P: the PP2 recipes. One reads prompts at 7,006 tok/s and decodes at 177.8 tok/s on a desktop with 16 GB of RAM; another decodes at 197 tok/s on predictable output with the full BF16 table on NVMe. Source: repo github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx by mannydelrio on GitHub, posted by @mrmannyatx, 2026-10-06
Source: @plean., 2026-10-07 - Three or four stock cards for long context: PP3 with a 1M-token context and a 7.36M-token KV pool at 74.4 tok/s, or the official FP8 checkpoint on four cards at 89 tok/s with about 11,000 tok/s prefill at 32k. Source: repo github.com/clee/qwen3.8-flash-next-3x170hx by clee on GitHub, posted by @c133, 2026-10-04
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub, posted by @flipnburner, 2026-08-29
Known problems
- P2P must really work for the TP2 recipes. One owner's
nvidia-smi topo -p2p rwreturned OK while a real card-to-card copy left the target buffer empty, and the deployment would not start. Check P2P with a content-checked copy before downloading the weights (P2P). Source: @filth5585, 2026-09-19 - Old CPUs slow the SSD-table TP2 recipe. Contested On a dual Broadwell host the Morrowmake EXL3 recipe ran at 55 to 70 tok/s for one user against the published 153, because the n-gram read from SSD in the middle of each step costs about 44 ms there. The W4A16 TP2 recipe keeps the table in host RAM for that reason. Source: repo github.com/ElectroHobo/qwen38-flash-next-w4a16-2x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24
Source: @darkmaniac7, 2026-10-27 (lab notes) - Run-to-run differences in vLLM. A rounding issue in vLLM's MoE block alignment produced a 3.6% difference in tokens at temperature 0 with the AWQ INT4 quant; a deterministic-MoE patch costs about 1 tok/s on two cards and gives consistent output. Source: @vektor_prime, 2026-09-29
Source: repo github.com/vektorprime/qwen38-flash-next-pp2 by vektorprime on GitHub, posted by @vektor_prime, 2026-10-07 - Top of VRAM on old unlock builds. On one PP2 setup, prompts over about 200K tokens faulted the second card when usage climbed into the top of its unlocked 64 GB; a memory cap avoids it, and cmpunlocker v0.5 or newer no longer hands out the protected region. See Xid 31 and the reserved top of VRAM. Source: repo github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx by mannydelrio on GitHub, posted by @mrmannyatx, 2026-10-06
- Official FP8 patch set. Never apply its withdrawn patch 0008: with it, single-stream decode falls to about 30 tok/s. Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub, posted by @flipnburner, 2026-08-29
- Quality checks are thin on the 3 bpw one-card recipe. Half the layers' routed experts are 2-bit, and quality is scored only on HumanEval and GSM8K. Source: repo github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe by ElectroHobo on GitHub, posted by @nathang6866, 2026-10-24
More: Known bad combinations, Instability under load, vLLM on sm_80: known bugs and patches.
Common questions
How much VRAM does Qwen3.8 Flash Next need?
How many CMP 170HX cards do I need for Qwen3.8 Flash Next?
How fast is Qwen3.8 Flash Next on one CMP 170HX?
How fast is Qwen3.8 Flash Next on two CMP 170HX cards?
How much host RAM does Qwen3.8 Flash Next need?
Does Qwen3.8 Flash Next need the x16 mod or P2P?
Can Qwen3.8 Flash Next reach 1M context on CMP 170HX cards?
Related
- Recipe pages for this model are in the table above; Recipe finder filters all recipes by cards, model and engine.
- Facts shared by every recipe for this model (size, table placement, quant advice): Qwen3.8 Flash Next on 4x 8 GB#Facts shared by every Flash Next recipe.
- The dense sibling on one card: Qwen3.8 27B on 1x 8 GB (W4A16 vLLM).
- Other model pages: Running GLM 5.3 Flash locally: hardware and measured speeds, Running GLM 5.3 locally: hardware and measured speeds, Running DeepSeek V4.1 Flash locally: hardware and measured speeds.
- Recipes, Benchmarks, Engines, Quantization on the 170HX, Speculative decoding on the 170HX, Parallelism over the PCIe link.