Qwen3.8 27B on 1x 8 GB (W4A16 vLLM)
More actions
| Reported 1 x 8 GB | |
| Quant | W4A16 (INT4 weights, Marlin kernels) |
|---|---|
| Engine | vLLM |
| Topology | single card |
| Context | 262,144 native (YaRN to about 500k in the recipe's long variant) |
| Decode | 170 tok/s |
| VRAM per card | 64 GB |
| Tested | @kellertreppe123 on 2026-10-05 |
| Source | link |
| Confirmations | Partly reproduced by @chonsworth (8 GB), 2026-10-05 · details |
The dense Qwen3.8-27B in W4A16 on one 64 GB card under vLLM, from a Reddit r/LocalLLM recipe titled "Qwen3.8-27B W4A16
on one 64 GB CMP 170HX: 160+/70+ decode": 160+ tok/s decode on short context and 70+ near the long end, with variants for
native context and about 500k via YaRN at 170 to 210 tok/s. kellertreppe123 copied it and reached about 170 tok/s at
262k context, 60 to 70 near long context. The Reddit author is not identified on Discord, so kellertreppe123, who
posted the recipe on Discord and ran it, is the tester; chonsworth's run on the same model is the confirmation.
Infobox decode is kellertreppe123's number.
Source: @kellertreppe123, 2026-10-05
Source: @kellertreppe123, 2026-10-05
Source: @kellertreppe123, 2026-10-05
Prerequisites
- One CMP 170HX at 64 GB (Driver unlock, 8 GB card). No x16 mod needed for one card, though prefill benefits from x16.
- vLLM with the quantized (Marlin) kernels; matstatman notes several PRs the recipe used have since been merged into mainline vLLM. Source: @matstatman, 2026-10-05
- kellertreppe123 mentions 170tune alongside the recipe (see Tools). Source: @kellertreppe123, 2026-10-05
Steps
Follow the Reddit post linked in the infobox (flags are not reproduced on Discord). General advice from the server:
- Use W4A16 for speed or W8A8 for slightly higher quality, in vLLM. Source: @xeax, 2026-10-15
- Use vanilla Qwen3.8 27B, not fine-tunes, with a fixed Jinja chat template (froggeric's), which orrorin says makes a huge difference. REAP-pruned models do not work without post-training. Source: @_orrorin, 2026-10-11
Source: @_orrorin, 2026-10-11 - Do not use llama.cpp on these cards: ikyo's Q8 llama.cpp run lost most tests to a Mac Studio on Qwen 27B; _L_ and others say vLLM or sglang extract far more. ikyo switched to vLLM the same day and runs concurrency 4 on one card. Source: @ikyo, 2026-10-26
Source: @lia_oc, 2026-10-26
Source: @ikyo, 2026-10-26 - KV on SSD for long-context reuse: clee points to the
kv-transfer-configsection of his Qwen Flash Next config; capping disk use needs vLLM PR #54327. Source: @c133, 2026-10-17
Measured numbers
| Who | Quant / engine | Result |
|---|---|---|
| kellertreppe123 | W4A16, the Reddit recipe | about 170 tok/s at 262k context; 60 to 70 nearing long context. His own AutoRound quant of the huihui abliterated variant only reached 60 to 70 tok/s. |
| chonsworth | same model, his own setup | about 3.7k tok/s prefill at about 100 tok/s decode, against the published 1.9k / 100; never run to 1M. |
| Miséricorde | W4A16 with a draft model | about 200 tok/s at concurrency 1 on short (about 30k) context; W8A16 costs about 30%. NDIV 54 to 70 alone gave +10.5% at c=1. |
| Miséricorde | with DFlash | 210+ tok/s on code at 16k context, 110+ at 200k, single stream. |
| regaly | FP8 + DSpark, sglang | 200 to 400 tok/s aggregate (about 300) at 4 concurrent during HLE / MMLU / GSM8K, higher at 6; his sglang INT8 ran slower than expected. |
| clee | FP8 release, sglang | 60 to 80 tok/s single stream with spikes to 125 (Threadripper 3970X, 128 GB non-ECC). |
| GlobalApathy | INT4 "Swift" quant, 131k context, concurrency 3 | about 55 tok/s single stream; over 200 aggregate with 7 sub-agents. |
| AmishRakeFightr | W8A8 INT8, vLLM + MTP, concurrency 2 | 30 to 40 tok/s. |
| Programmer007 | W8A8 and W8A16, vLLM, 170 W cap | both run fine; about 3x the aggregate tok/s of his dual 3090s at about a third of the power. His earlier llama.cpp Q8 test: about 800 tok/s prefill at 85 C with no x16 mod. |
| Phoenix17 | fp8 e4m3 or e5m2 KV | about 3 concurrent requests per 64 GB card at native context length. |
| PixelML (Club CMP 170HX) | W4A16 AutoRound + DFlash2 k=7, 180 W | 136.38 to 140.3 tok/s on one card; TP2 + MTP k=3 103.5 tok/s. vLLM with DFlash2 181.8 tok/s against 94.1 for the open-jet megakernel. |
Sources, in table order:
Source: @kellertreppe123, 2026-10-05
Source: @chonsworth, 2026-10-05
Source: @wakeupmbrc1, 2026-10-14
Source: @wakeupmbrc1, 2026-10-04
Source: @reggie7756, 2026-10-04
Source: @c133, 2026-10-16
Source: @globalapathy, 2026-10-05
Source: @amishrakefightr, 2026-10-05
Source: @programmer_007, 2026-10-15
Source: @programmer_007, 2026-10-15
Source: @phoenixone7, 2026-10-27
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
- Phoenix17 published concurrency benchmarks for his Swift-1.5-Qwen3.8-27b-heretic-W8A8-DFlash2 on the Hugging Face card; the prefill column there is per stream and must be multiplied by concurrency. Source: @phoenixone7, 2026-10-05
- Phoenix17 runs two 64 GB cards (128 GB). Swift 1.5 is a heretic fine-tune of the 27B that he says cuts much of the stock model's verbosity; his W8A8 quant of it leaves room for about 700K tokens of context on one card, enough for at least 2 streams per card, and he has a drafter for it. Engine and tok/s were not posted. A generic, software-oriented AGENTS.md he shared on r/LocalLLaMA cuts verbosity further. Source: @phoenixone7, 2026-10-23
Source: @phoenixone7, 2026-10-23
Source: @phoenixone7, 2026-10-23
Source: @phoenixone7, 2026-10-23
Source: @phoenixone7, 2026-10-24
Source: @phoenixone7, 2026-10-25
Source: @phoenixone7, 2026-10-25 - Feralzi reached 225 tok/s on Qwen 3.8 27B with vLLM's integer kernels; matstatman notes that number is with DFlash at temperature 0, "nothing you want to code with". Source: @feralzi, 2026-10-14
Source: @matstatman, 2026-10-14 - Feralzi's setup is written up in his r/LocalLLM post "Qwen 3.8 27B reaching 225 t/s on 132k context on CMP 170HX": vLLM with the quantized kernels at 132k context. Every card is different, so he stepped through overclock settings card by card until he found one that was stable and would not crash; he says the right overclock runs the card cooler, at lower wattage and faster. Temperatures stay around 70 C and he runs agents through the night without a crash. Shirogeek's caution: decode-maxing numbers are hard to trust, and temperature 0 is for repeatable tests, not real-life speed. On 2026-10-25 Feralzi again reported over 200 tok/s with the 27B on his CMP (a 40 GB card by his other posts that day); quant, engine and speculative decoding were not stated. Source: @feralzi, 2026-10-25
Source: @feralzi, 2026-10-25 Source: @feralzi, 2026-10-14
Source: @feralzi, 2026-10-14
Source: @feralzi, 2026-10-14
Source: @feralzi, 2026-10-14
Source: @shirogeek, 2026-10-14
Speculative decoding
Contested DFlash on the 27B: matstatman says its acceptance is too low in real use unless you code at
temperature 0, so use MTP with k=3; Miséricorde's 210+ / 110+ numbers were measured with DFlash. DUFRelic adds that
decode speed matters less when the model thinks 10 to 20 times more tokens than it outputs. Programmer007 could not get
DFlash running in vLLM at all.
Source: @matstatman, 2026-10-04
Source: @matstatman, 2026-10-17
Source: @dufrelic, 2026-10-04
More cards
- Two cards: TP2 usually beats one card, depending on x16 against x4 links (Miséricorde); programmerboi recommends PP2 for batched throughput, especially without x16. cibernox reports TP2 works well after the x16 mod. Source: @wakeupmbrc1, 2026-10-14
Source: @programmerboi, 2026-10-14 - Four cards, TP4 (turbine, no recipe published yet): 440 tok/s decode with speculation and about 4k tok/s prefill at 190 W per card, using his own all-reduce traffic compression (2.2k prefill without it). His roofline: 4 x 1.5 TB/s = 6 TB/s, so 27B at 4-bit caps decode near 440 tok/s without speculation; the limit is all-reduce latency over the slow PCIe. Source: @turbine, 2026-10-06
Source: @turbine, 2026-10-06
Source: @turbine, 2026-10-06
Known issues
- The 27B thinks a lot: matstatman saw one 2 h 45 min thought and moved to Flash Next on four cards. Source: @matstatman, 2026-10-16
- clee suspects a host RAM bit flip (48 h clean memtest, then an unexplained page cache corruption): use ECC memory on these hosts. Source: @c133, 2026-10-16
- The 27B against Flash Next is a matter of taste: see Recipes.
- riko got random errors in real tasks with the 27B at first; "with hardware everything is fine now", and he runs it at W4A16 on a single 64 GB card. The cause of the earlier errors was not stated. See Discord. Source: @riko6253, 2026-10-25
Sources
- Reddit recipe (author not identified on Discord), posted on Discord by kellertreppe123 on 2026-10-05.
- Related: Quantization on the 170HX, Engines, Benchmarks.