Qwen3.8 27B on 1x 8 GB (W4A16 vLLM)

Qwen3.8-27B (dense)
Proven 1 x 8 GB
Quant	W4A16 (INT4 weights, Marlin kernels)
Engine	vLLM
Topology	single card
Context	262,144 native (YaRN to about 500k in the recipe's long variant)
Decode	169.5 tok/s
VRAM per card	64 GB
Tested	@wakeupmbrc1 on 2026-08-30
Source	link
Confirmations	Partly reproduced by @chonsworth (8 GB), 2026-10-05 · details Reproduced by @kellertreppe123 (8 GB), 2026-10-05 · details

The dense Qwen3.8-27B in W4A16 on one 64 GB card under vLLM, from Miséricorde's r/LocalLLM post "Qwen3.8-27B W4A16 on one 64 GB CMP 170HX: 160+/70+ decode" (2026-08-30): 160+ tok/s decode on short context and 70+ near the long end, with variants for native context and about 500k via YaRN at 170 to 210 tok/s. The setup is vLLM with DFlash2 speculative decoding plus his own vLLM patches, written to be handed to an AI agent that applies everything, and suited to solo or low-concurrency inference; his prefill optimisation attempts topped out around +15%. He paid about 1.6k per card and was inspired by a Reddit post about 130 tps on a 3090 with DFlash2.

His numbers, with the GPU overclocked (about +20% inference speed): short prompts W8A16 120.3 tok/s (3.30 tokens per step) and W4A16 169.5 tok/s (3.25), favourable UI turns 190 to 210; a 23.75K prompt W8A16 82.9 and W4A16 104.8 tok/s (2.97); a 65,920-token hot prefix W8A16 57.9; at 85,514 tokens an older W4/INT8 run mostly 42 to 46 tok/s (16/16 general and 6/6 crash-point passes) and the current W4 FP8/DFlash build 119.3 tok/s strict hot decode, 1.76 s hot TTFT and 84.8 tok/s hot end-to-end. In practice 200+ tok/s at the start, 80 to 110 tok/s at 100K KV and 100+ for agentic coding. Infobox decode is his short-prompt W4A16 figure.

Reproductions: kellertreppe123 copied the recipe and reached about 170 tok/s at 262k context, 60 to 70 near long context (Confirmation 2); on 2026-08-31 he already ran the 27B on one card for n8n workflows at "not lower then 90 tok/s but depends" and called the post a good summary. Dino reproduced it on one card at 3,000 tok/s prefill and 160 tok/s decode (W4A16) against about 1,000 / 50 with llama.cpp and ran an INT8 version at 120 tok/s, but could not get the post's repo to work; Miséricorde offered help by DM. dottoreargus got 142 tok/s at small context but struggled at larger contexts (see the prefix-caching pointer under Speculative decoding). chonsworth's run on the same model is Confirmation 1; AmishRakeFightr planned to try it. Source: @1a2b3c0967, 2026-08-30
Source: @wakeupmbrc1, 2026-08-30
Source: @kellertreppe123, 2026-08-31
Source: @dino_38975, 2026-08-31
Source: @dottoreargus, 2026-09-01
Source: @kellertreppe123, 2026-10-05
Source: @kellertreppe123, 2026-10-05
Source: @kellertreppe123, 2026-10-05
Source: @amishrakefightr, 2026-10-05
Source: @kellertreppe123, 2026-10-05
Source: @chonsworth, 2026-10-05

Prerequisites

One CMP 170HX at 64 GB (Driver unlock, 8 GB card). No x16 mod needed for one card, though prefill benefits from x16.
vLLM with the quantized (Marlin) kernels; matstatman notes several PRs the recipe used have since been merged into mainline vLLM. Source: @matstatman, 2026-10-05
kellertreppe123 mentions 170tune alongside the recipe (see 170tune and Tools). Source: @kellertreppe123, 2026-10-05

Steps

Follow the Reddit post linked in the infobox (flags are not reproduced on Discord). It relies on DFlash2 and on Miséricorde's own vLLM patches; which repository and which of his patches the published numbers need is being confirmed with the author. Pointer (waiting for the author): a longer write-up by @wakeupmbrc1 exists; see Discord. Source: @wakeupmbrc1, 2026-08-30

General advice from the server:

Use W4A16 for speed or W8A8 for slightly higher quality, in vLLM, and the Flash models if there is enough system RAM. Source: @xeax, 2026-10-15
Source: @xeax, 2026-10-15
Use vanilla Qwen3.8 27B, not fine-tunes, with a fixed Jinja chat template (froggeric's), which orrorin says makes a huge difference. REAP-pruned models do not work without post-training. Source: @_orrorin, 2026-10-11
Source: @_orrorin, 2026-10-11
Do not use llama.cpp on these cards: ikyo's Q8 llama.cpp run lost most tests to a Mac Studio on Qwen 27B; _L_ and others say vLLM or sglang extract far more. ikyo switched to vLLM the same day and runs concurrency 4 on one card. Source: @ikyo, 2026-10-26
Source: @lia_oc, 2026-10-26
Source: @ikyo, 2026-10-26
KV on SSD for long-context reuse: clee points to the kv-transfer-config section of his Qwen Flash Next config; capping disk use needs vLLM PR #54327. Source: @c133, 2026-10-17

Measured numbers

Who	Quant / engine	Result
kellertreppe123	W4A16, the Reddit recipe	about 170 tok/s at 262k context; 60 to 70 nearing long context. His own AutoRound quant of the huihui abliterated variant only reached 60 to 70 tok/s.
chonsworth	same model, his own setup	about 3.7k tok/s prefill at about 100 tok/s decode, against the published 1.9k / 100; never run to 1M.
Miséricorde	W4A16 with a draft model	about 200 tok/s at concurrency 1 on short (about 30k) context; W8A16 costs about 30%. NDIV 54 to 70 alone gave +10.5% at c=1.
Miséricorde	with DFlash	210+ tok/s on code at 16k context, 110+ at 200k, single stream.
regaly	FP8 + DSpark, sglang	200 to 400 tok/s aggregate (about 300) at 4 concurrent during HLE / MMLU / GSM8K, higher at 6; his sglang INT8 ran slower than expected.
clee	FP8 release, sglang	60 to 80 tok/s single stream with spikes to 125 (Threadripper 3970X, 128 GB non-ECC).
GlobalApathy	INT4 "Swift" quant, 131k context, concurrency 3	about 55 tok/s single stream; over 200 aggregate with 7 sub-agents.
AmishRakeFightr	W8A8 INT8, vLLM + MTP, concurrency 2	30 to 40 tok/s.
Programmer007	W8A8 and W8A16, vLLM, 170 W cap	both run fine; about 3x the aggregate tok/s of his dual 3090s at about a third of the power. His earlier llama.cpp Q8 test: about 800 tok/s prefill at 85 C with no x16 mod.
Phoenix17	fp8 e4m3 or e5m2 KV	about 3 concurrent requests per 64 GB card at native context length.
PixelML (Club CMP 170HX)	W4A16 AutoRound + DFlash2 k=7, 180 W	136.38 to 140.3 tok/s on one card; TP2 + MTP k=3 103.5 tok/s. vLLM with DFlash2 181.8 tok/s against 94.1 for the open-jet megakernel.

Sources, in table order: Source: @kellertreppe123, 2026-10-05
Source: @chonsworth, 2026-10-05
Source: @wakeupmbrc1, 2026-10-14
Source: @wakeupmbrc1, 2026-10-04
Source: @reggie7756, 2026-10-04
Source: @c133, 2026-10-16
Source: @globalapathy, 2026-10-05
Source: @amishrakefightr, 2026-10-05
Source: @programmer_007, 2026-10-15
Source: @programmer_007, 2026-10-15
Source: @phoenixone7, 2026-10-27
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20

Phoenix17 published concurrency benchmarks for his Swift-1.5-Qwen3.8-27b-heretic-W8A8-DFlash2 on the Hugging Face card; the prefill column there is per stream and must be multiplied by concurrency. Source: @phoenixone7, 2026-10-05
Phoenix17 runs two 64 GB cards (128 GB). Swift 1.5 is a heretic fine-tune of the 27B that he says cuts much of the stock model's verbosity; his W8A8 quant of it leaves room for about 700K tokens of context on one card, enough for at least 2 streams per card, and he has a drafter for it. Engine and tok/s were not posted. A generic, software-oriented AGENTS.md he shared on r/LocalLLaMA cuts verbosity further. Source: @phoenixone7, 2026-10-23
Source: @phoenixone7, 2026-10-23
Source: @phoenixone7, 2026-10-23
Source: @phoenixone7, 2026-10-23
Source: @phoenixone7, 2026-10-24
Source: @phoenixone7, 2026-10-25
Source: @phoenixone7, 2026-10-25
Feralzi reached 225 tok/s on Qwen 3.8 27B with vLLM's integer kernels; matstatman notes that number is with DFlash at temperature 0, "nothing you want to code with". Source: @feralzi, 2026-10-14
Source: @matstatman, 2026-10-14
Feralzi's setup is written up in his r/LocalLLM post "Qwen 3.8 27B reaching 225 t/s on 132k context on CMP 170HX": vLLM with the quantized kernels at 132k context. Every card is different, so he stepped through overclock settings card by card until he found one that was stable and would not crash; he says the right overclock runs the card cooler, at lower wattage and faster. Temperatures stay around 70 C and he runs agents through the night without a crash. Shirogeek's caution: decode-maxing numbers are hard to trust, and temperature 0 is for repeatable tests, not real-life speed. On 2026-10-25 Feralzi again reported over 200 tok/s with the 27B on his CMP (a 40 GB card by his other posts that day); quant, engine and speculative decoding were not stated. Source: @feralzi, 2026-10-25
Source: @feralzi, 2026-10-25 Source: @feralzi, 2026-10-14
Source: @feralzi, 2026-10-14
Source: @feralzi, 2026-10-14
Source: @feralzi, 2026-10-14
Source: @shirogeek, 2026-10-14

Other people's numbers

Single-card reports from August to October 2026, most of them before the recipe. Context, concurrency, power limit and speculative decoding differ between them. Two-card runs are under More cards.

Qwen3.8 27B release and size: 55 GB, about 67 GB in BF16, FP8 fits one card

Qwen 3.8 27B was released on 2026-08-14. CUTE said the release is 55 GB and fits his two-card CMP workstation, Studebaker linked Qwen/Qwen3.8-27B-FP8, lmstudio-community published a quant the same day, and meatsus noted the 2.4T model at IQ2_XS is 731 GB. roger_dammit (2026-08-20) put the BF16 checkpoint at about 67 GB, so it needs two cards, which he expects would kill performance; the two size figures were not reconciled in the thread. matstatman runs the FP8 weights on one card because they fit and are much faster than pipeline parallel across two, FP8 using less memory while computing in BF16; he later tried BF16 at 160k context while chasing tool-call loops.

Reported
Source: @meatsus, 2026-08-14
Source: @cute_sc2, 2026-08-14
Source: @_studebaker_, 2026-08-14
Source: @matstatman, 2026-08-20
Source: @roger_dammit, 2026-08-20

Miséricorde's earlier claims: his own runtime at 200 short and 100 long, RTX 5090 parity at 150 to 180 W

On 2026-08-30 Miséricorde said the stock unsloth Qwen 3.8 27B runs at about 30 to 40 tok/s on his CMP, while a runtime he built with community help runs 200 on short and 100 on long contexts (units presumably tok/s); engine, quant and flags were not given. On 2026-09-01 he put agentic decode at 100k context at around 80 to 110 tok/s on his CMPs with DFlash2, similar to or slightly faster than an RTX 5090 (card count and engine not stated). He claims a 170HX runs Qwen inference at RTX 5090 speed at a 150 W limit with a memory overclock and core undervolt, and on 2026-09-17 (translated from Russian) that a card limited to 180 W gives roughly the same or better performance on the 27B than an RTX 5090 at 450 W, crediting the server GA100 on TSMC 7 nm; no tok/s numbers accompanied the 5090 comparisons. In the October dense-against-MoE thread (his 200 and 210+ / 110+ figures are in the table above) CUTE saw GLM-5.3 at 800 tok/s out and 3k prefill while Qwen sat at 700/70, and Takka noted the dense model reuses the same weights, so it should parallelize better than GLM at low context; Programmer007 was considering a 4-concurrency Q8 setup. The runtime is not published. Pointer (waiting for the author): a longer write-up by @wakeupmbrc1 exists; see Discord.

Reported
Source: @wakeupmbrc1, 2026-10-14
Source: @programmer_007, 2026-10-14
Source: @cute_sc2, 2026-10-04
Source: @wakeupmbrc1, 2026-10-04
Source: @spectakkular, 2026-10-04
Source: @wakeupmbrc1, 2026-08-30
Source: @wakeupmbrc1, 2026-09-01
Source: @wakeupmbrc1, 2026-09-17

regaly: FP8 with DSpark at 120 tok/s on one request, 240 at 3 parallel, and the context he tested per parallel count

The October single-card rows in the table above (GlobalApathy, AmishRakeFightr, clee, regaly, Programmer007, Phoenix17) come from one thread. Earlier, regaly ran Qwen3.8-27B in FP8 with DSpark speculative decoding at about 120 tok/s decode on a single request, rising to 240 tok/s total output with 3 parallel requests (2026-09-26). Replying in that thread on 2026-10-03, he said he had tested up to 180k context with 2 parallel requests, up to 64k with 4 and up to 32k with 8; the reply does not restate model, engine or card count, so tying it to the FP8 setup rests on the reply chain. clee expects an INT4 weight quant to be about twice as fast as FP8; regaly expected INT8 to be almost double, but his sglang INT8 runs slower (suspected kernel problem). Programmer007 was hoping for about 200 tok/s at concurrency 4 on one card.

Reported 8 GB
Source: @globalapathy, 2026-10-05
Source: @amishrakefightr, 2026-10-05
Source: @c133, 2026-10-05
Source: @c133, 2026-10-16
Source: @reggie7756, 2026-10-04
Source: @sojun80, 2026-10-04
Source: @programmer_007, 2026-10-15
Source: @phoenixone7, 2026-10-04
Source: @programmer_007, 2026-10-04
Source: @reggie7756, 2026-09-26
Source: @reggie7756, 2026-10-03

Samuel Hayden: INT8-W8A16-MTP on one card in vLLM, 1,877 prefill, 89.9 decode, 127 at 4 streams

Studebaker linked lued/Qwen3.8-27B-INT8-W8A16-MTP on release day. Samuel Hayden (2026-08-14), vLLM TP1 with MTP (3 tokens per the model spec) in a PCIe 2.0 x8 slot, first pass with an 8.5k-token prompt: 1,877 tok/s prefill, 89.9 tok/s decode single stream and 127 tok/s decode at concurrency 4. Studebaker only got about 45 tok/s decode and asked about cudagraphs; Ryko measured about 1.5k tok/s prefill on one card with the full model. catcat (2026-09-07) gets 80+ tok/s decode on Qwen 27B with vLLM, a W8A16 quant and MTP 3 (no prefill, context or card count given).

Reported
Source: @_studebaker_, 2026-08-14
Source: @samuel_hayden., 2026-08-14
Source: @vektor_prime, 2026-08-14
Source: @programmer_007, 2026-09-07
Source: @catcat0458, 2026-09-07

Official FP8 on one card in vLLM: about 2,000 prefill, 35 to 40 tok/s raw, about 100 with MTP 3

Studebaker (2026-08-15): the official FP8 on vLLM with CUDA graphs gives about 2,100 tok/s prompt processing and 35 to 40 tok/s decode on the raw model without MTP; DeeJay the same night got about 4,000 tok/s prefill and about 78 tok/s generation (card count and quant not stated). giaki3003 (2026-08-20): about 2,000 prefill and 40 decode on a single card, and since the 170HX is not good at multi-card, run the 27B on one card. fallengod: it fits one card at INT8 with max context and gives 30 to 50 tok/s decode and 2,500 tok/s prefill with MTP; Schaka suggested the V100 slim kernels might help a single-card dense 27B. Studebaker (2026-08-23), for 2 to 3 agentic users on one card: official FP8, cudagraphs and MTP 3 give about 2,000 prefill and about 100 decode (about 40 without MTP), peaking at 122 tok/s at temperature 0.1 because greedy sampling raises MTP acceptance; real throughput through an agent harness is maybe a third of that. Dino noted it beats a 3090 at gpu-memory-utilization 0.9. matstatman also rates the 27B on one CMP the best option, runs two cards as separate processes behind a stateful LLM proxy he wrote, and finds MTP and DSpark acceptance poor for Claude Code work. Studebaker (2026-08-24) runs FP8 workers on 4 cards under an Opus 4.8 orchestrator on a 500k-line Rust codebase at temperature 0.5, thinking low and MTP 3 for about 90 tok/s; the key check is that CUDA graphs and MTP are actually active, and he had not found DFlash2 much better than MTP 3.

Proven
Source: @_studebaker_, 2026-08-15
Source: @deejay9983, 2026-08-15
Source: @srvgl, 2026-08-20
Source: @giaki3003, 2026-08-20
Source: @fallengod9962, 2026-08-20
Source: @schakaqt, 2026-08-20
Source: @matstatman, 2026-08-23
Source: @dino_38975, 2026-08-23

Freaksterz SmoothQuant W8A8 INT8: about 4k tok/s prefill against about 1.5k for the official FP8

DUFRelic (2026-08-15) suggested Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 as a good quant for the 170HX, and vuhuycan reported about 4k tok/s prefill with it. For comparison fallengod got about 1,500 tok/s prefill from the official FP8 release, and matstatman about 1,800 on the FP16 version, which then dropped because of poor cooling. DUFRelic posted his own INT8 results at only 150 W because of makeshift cooling; the settings of the 4k run were not posted.

Reported
Source: @dufrelic, 2026-08-15
Source: @fallengod9962, 2026-08-15
Source: @vuhuycan0322, 2026-08-15
Source: @matstatman, 2026-08-15

fallengod: official FP8 as a docker compose service on one card (MTP 1, 131,072 context)

fallengod (2026-08-15) serves Qwen/Qwen3.8-27B-FP8 on device 0 from the vllm/vllm-openai:muse-glimmer image pinned by digest, port 8084:8000, ipc host, PYTORCH_ALLOC_CONF=expandable_segments:True, with these flags:

--max-model-len 131072 --gpu-memory-utilization 0.92 --kv-cache-dtype fp8 \
--max-num-batched-tokens 16384 --max-num-seqs 32 --enable-chunked-prefill --enable-prefix-caching \
--speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
--enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3

The draft head loads from the checkpoint's own mtp.safetensors (qwen3_5_mtp). An agent running Qwen 3.6 generated the install, and he later reported only about 1,500 tok/s prefill with this official FP8 release.

Reported
Source: @fallengod9962, 2026-08-15

fallengod's benchmark table: prefill 2,777 falling to 1,790 tok/s between 8.8K and 70K tokens

fallengod (2026-08-15) posted a benchmark summary (3 runs each): prefill mean 2,777 tok/s at about 8.8K tokens, 2,291 at about 35K and 1,790 at about 70K; decode at N=1 51.1 tok/s on narrative text and 79.6 on code; 6-way concurrency 124.2 aggregate tok/s narrative and 230.8 code. He posted it in a thread about a DeepSeek V4 repo and did not say whether these are his own numbers (likely his Qwen3.8 27B MTP setup) or copied from a repo, nor the model or card count: verify before reuse.

Reported
Source: @fallengod9962, 2026-08-15

Xeax: vLLM serve command for the NVFP4 quant on one card, with Ryko's tuning notes

Xeax (2026-08-16) shared this command; max-num-seqs and max-model-len can go higher:

vllm serve Inferact/Qwen3.8-27B-NVFP4 --tensor-parallel-size 1 --max-num-seqs 128 --dtype float16 \
  --gpu-memory-utilization 0.95 --max-model-len 163856 --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder

Ryko's notes: use --max-model-len auto to size KV to VRAM, --kv-cache-dtype bfloat16 because the CMP lacks hardware FP8 but has FP16/BF16, and --performance-mode interactivity for automatic batch-size tuning. Xeax then suggested lued/Qwen3.8-27B-INT8-W8A16-MTP should be faster; no speed numbers were given.

Reported
Source: @xeax, 2026-08-16
Source: @vektor_prime, 2026-08-16

A video pointer for Qwen3.8-27B on the 170HX (August)

philow asked whether anyone had tested Qwen3.8-27B on the 170HX, and Schaka replied with a YouTube link (youtube.com/watch?v=eANj_hvULt0); no numbers were quoted in chat. segmond repeatedly argued that Qwen3.8-27B reduces the need for multi-card VRAM pools.

Reported
Source: @.philow, 2026-08-16
Source: @schakaqt, 2026-08-16
Source: @segmond, 2026-08-16

roger_dammit's podman config for W8A16 with MTP 3: 3k prefill but 14 tok/s decode at 20k depth

roger_dammit (2026-08-18) ran docker.io/vllm/vllm-openai:latest via podman, TP1 PP1, with the qwen3 reasoning parser, the qwen3_coder tool parser, env OMP_NUM_THREADS=1 VLLM_USE_FLASHINFER_SAMPLER=0 and:

--model lued/Qwen3.8-27B-INT8-W8A16-MTP --dtype bfloat16 --performance-mode interactivity \
--gpu-memory-utilization 0.92 --max-model-len auto --max-num-seqs 4 --max-num-batched-tokens 4096 \
--kv-cache-dtype fp8_e4m3 --no-enable-prefix-caching --enable-chunked-prefill --mamba-cache-mode align \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

He saw about 3k tok/s prefill but only 14 tok/s decode at about 20k depth and considered it underperforming. Studebaker expects about 2,000 prefill / 40 decode for the FP8 with vLLM and cudagraphs (on by default in that command), and matstatman suggested trying MTP 7.

Reported
Source: @roger_dammit, 2026-08-18
Source: @matstatman, 2026-08-18
Source: @_studebaker_, 2026-08-18

roger_dammit's single-card recipe: INT8-W8A16-MTP with MTP k=7, 262k context, FP8 KV

roger_dammit (2026-08-20) serves lued/Qwen3.8-27B-INT8-W8A16-MTP on one card with podman and docker.io/vllm/vllm-openai:nightly, env OMP_NUM_THREADS=1 VLLM_USE_FLASHINFER_SAMPLER=1, and:

--tensor-parallel-size 1 --dtype bfloat16 --performance-mode interactivity --max-model-len 262144 \
--gpu-memory-utilization 0.90 --max-num-seqs 1 --max-num-batched-tokens 8192 --max-num-scheduled-tokens 8192 \
--kv-cache-dtype fp8_e4m3 --enable-prefix-caching --enable-chunked-prefill --mamba-cache-mode align \
--mamba-ssm-cache-dtype auto --prefix-match-unit 16 --prefix-cache-retention-interval 1648 \
--prefix-caching-hash-algo sha256 --language-model-only --trust-remote-code \
--speculative-config '{"method":"mtp","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

His results were posted as an image, measured with the card limited to 170 W using a Python script he posted. Throughput kept rising with more MTP tokens and plateaued at 7, whereas on his Volta cards MTP 2 was the sweet spot; he estimated DSpark2 plus vLLM hacks would get close to about 150 tok/s decode. It runs fine on one card (two would probably slow it), the chat template from the model repo is optional, and he calls the W8A16 quant close to lossless and recommended for Ampere.

Reported
Source: @roger_dammit, 2026-08-20

fallengod: SGLang W8A8 with MTP at 150 W, about 75 tok/s decode and 2,000 to 3,000 prefill; tool use broken

fallengod (2026-08-20) ran Qwen3.8-27B W8A8 with MTP in SGLang on one card at a 150 W power limit, testing at about 260k context: about 75 tok/s decode and 2,000 tok/s prefill at first, later screenshots implying about 3,000. Raising the limit from 150 W to 250 W gave 22% more prefill with no other change. He used W8A8 for MTP and W8A16 only for DFlash2, and a Hermes agent driving Qwen 3.8 did the setup. He later reported SGLang "very broken" for him, with tool use totally broken.

Reported
Source: @fallengod9962, 2026-08-20

Slow first numbers: switch to vLLM with W8A8 and expect about 3k prefill and 78 tok/s

MrManny's first numbers were slow, and DeeJay said he was missing vLLM and W8A8 quantization; DeeJay's expectation from memory was about 3k tok/s prefill and about 78 tok/s generation, with a screenshot as reference. MrManny switched to vLLM and posted a clear improvement on 2026-08-21. He runs Qwen3.8-27B (temperature 0.7, top_p 0.95, top_k 20) as primary and Qwen3.8-27B-Uncensored-AWQ with the same sampling as secondary. Danske took part in the thread.

Reported
Source: @mrmannyatx, 2026-08-20
Source: @deejay9983, 2026-08-20
Source: @danske1, 2026-08-20

Studebaker's FP8, W8A16 and W8A8 throughput table on one card (vLLM cudagraphs + MTP 3)

Studebaker (2026-08-21) ran a saturating benchmark with unique prompts on Qwen3.8-27B. Prefill is aggregate tok/s with max_tokens=1; decode is aggregate tok/s at 512 tokens per request.

Prefill	N=1	N=4	N=8	N=16	N=32	N=64
FP8	2,168	2,379	2,115	2,115	2,009	2,221
W8A16	2,062	2,114	2,009	1,956	1,851	2,062
W8A8	4,548	4,653	4,653	4,759	4,548	4,759

Decode	N=1	N=8	N=32	N=64	N=128	N=256
FP8	95	581	825	757	748	752
W8A16	99	521	766	686	681	681
W8A8	71	493	1,095	947	979	978

W8A8 roughly doubles prefill and wins decode at high concurrency but is slowest single stream; DUFRelic noted that more prefill and decode tokens are needed for real numbers. See also Quantization on the 170HX.

Reported
Source: @_studebaker_, 2026-08-21
Source: @dufrelic, 2026-08-21
Source: @roger_dammit, 2026-08-21

matstatman's docker config for one card, plus Prometheus and Grafana monitoring

matstatman (2026-08-23) posted this for a new single-card owner:

docker run --gpus all --ipc=host -e VLLM_SLEEP_WHEN_IDLE=1 -e CUDA_VISIBLE_DEVICES=1 -e OMP_NUM_THREADS=1 \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 docker.io/vllm/vllm-openai:nightly lued/Qwen3.8-27B-INT8-W8A16-MTP \
  --served-model-name Qwen/Qwen3.8-27B-FP8 --tensor-parallel-size 1 --dtype auto --performance-mode interactivity \
  --max-model-len 262144 --gpu-memory-utilization 0.90 --max-num-seqs 8 --max-num-batched-tokens 8192 \
  --max-num-scheduled-tokens 8192 --kv-cache-dtype auto --enable-prefix-caching --enable-chunked-prefill \
  --mamba-cache-mode align --mamba-ssm-cache-dtype auto --prefix-match-unit 16 \
  --prefix-cache-retention-interval 1568 --prefix-caching-hash-algo sha256 --language-model-only \
  --trust-remote-code --tool-call-parser qwen3_coder --enable-auto-tool-choice --port 8000

He labelled it "my MTP config", but the pasted command has no --speculative-config. Downloads: hf download lued/Qwen3.8-27B-INT8-W8A16-MTP and hf download Doopeworld/Qwen3.8-27B-DSpark-vLLM. He advises setting up Prometheus and Grafana from vLLM's examples/observability/prometheus_grafana docker-compose (see also Monitoring).

Reported
Source: @matstatman, 2026-08-23

INT4 on one card: about 75 tok/s single stream, about 250 tok/s across 8 agents

datacat asked how many chat users one card could serve with Qwen 27B INT4; an AI estimate had said 16 to 24 concurrent users at 20 to 30k context, which is unverified. DeeJay: a single stream runs at about 75 tok/s and drops fast at higher context, and with 8 sub-agents the aggregate is about 250 tok/s, about 31 tok/s each if split across 8 separate users.

Reported
Source: @deejay9983, 2026-08-24
Source: @_datacat_, 2026-08-24

civiloid: FP8 on one card, about 1,800 prefill, 70 tok/s worst case, 500 tok/s at 16 clients

civiloid (2026-08-25), one card with somewhat poor cooling, Qwen3.8-27B FP8: about 1,800 tok/s prompt processing, about 70 tok/s decode worst case single stream, and about 500 tok/s aggregate decode at 16 concurrent clients, testing at a moderate context of about 32k. An edit-heavy benchmark gave 230 tok/s decode, which he calls unrealistic. At roughly 2k tok/s prefill, a 200k-token context takes about 100 s to prefill alone.

Reported
Source: @civiloid, 2026-08-25

syv-ai/qwen38-27b-rtx3090 on the 170HX: it runs, but its kernels were judged poor for the card

Contested

machinezero (2026-08-18) shared a r/Vllm post about pushing Qwen3.8-27B to 99 tok/s single-request and 1,150 tok/s batched (syv-ai, Ampere, INT8 in places). Source: @machinezer0, 2026-08-18
giaki3003 said the server had already tried it and that it is poor for the 170HX: it would have to be rebuilt from scratch, is arguably no better than Marlin kernels and only adds value on consumer cards, since "170hx with the enterprise kernels scream pretty high already". Source: @giaki3003, 2026-08-18
caueh (2026-08-18 to 2026-08-25) ran lued/Qwen3.8-27b-INT8-W8A16-MTP with DFlash2 in a patched vLLM Docker from syv-ai/qwen38-27b-rtx3090 (built for the RTX 3090), measuring with llama-benchy: about 130 to 150 tok/s decode, and he expects more with the model that repo was tuned for. DFlash2 improves decode more than MTP does, FP8 does not work with DFlash2, and in NIAH tests at 1M context with thinking disabled the INT8 model found every needle. He later confirmed it was one unlocked card (device 20C2) on driver 610.43.03 at PCIe Gen2 x4; the 150 tok/s figure was a temperature-0 test, and llama-benchy gave 110 tok/s at low context and 80 at a high context window. Source: @caueh, 2026-08-18
Source: @caueh, 2026-08-21
Source: @caueh, 2026-08-25
For comparison imsoslo gets 71 tok/s on llama.cpp with Qwen3.8 Q8 MTP on one card. Source: @sloburrito, 2026-08-21
Xeax (2026-08-25) confirmed the repo's 4-bit W4A16 setup works on the CMP 170HX as well, and since the card has far more memory than a 3090's 24 GB he suggests running W8A16 instead. chonsworth ran it on a rented 3090 and got stuck in a context-compaction loop, which Xeax also saw with 4-bit quants; roger_dammit took part. Source: @xeax, 2026-08-25
Source: @chonsworth, 2026-08-25
Source: @roger_dammit, 2026-08-25
Jonathany Joerrington got 120 to 130 tok/s in the reasoning phase and 80 to 110 tok/s in the final output with a packed 32k context while testing the same setup; iszo asked about the setup. Source: @ianszo21, 2026-08-25
Source: @jonathanyjoerrington, 2026-08-25

Qwen3.8 27B at 90+ tok/s cited as the card's value

orrorin cited 90+ tok/s on Qwen3.8 27B as the reason the card was still worth buying at about $2,000 (2026-08-25). No quant, engine or context length was given.

Reported
Source: @_orrorin, 2026-08-25

Phoenix17: Freaksterz SmoothQuant W8A8 on one 8 GB card, 56 tok/s at c=1 and 18.8K prefill at c=32

Phoenix17 (2026-08-27, 8 GB cards at 250 W) got 56 tok/s single-concurrency decode with MTP 2 on Qwen3.8-27B-SmoothQuant-W8A8-INT8 on one card, scaling well up to c=64. His optimal config:

vllm serve Qwen3.8-27B-SmoothQuant-W8A8-INT8 --tensor-parallel-size 1 --max-model-len 262144 \
  --gpu-memory-utilization 0.99 --kv-cache-dtype fp8_e4m3 --enable-prefix-caching --enable-chunked-prefill \
  --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice --disable-custom-all-reduce \
  --async-scheduling --trust-remote-code --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

fp8_e4m3 KV raises capacity from 443,331 to 847,940 KV tokens for a small speed cost at c=8; keep a 2K batch size, since an earlier result was skewed by a 4K one. Peak prefill was about 18.8K tok/s at c=32 with random-token prompts; for interactive use (c=1 to 8) TPOT is 18 to 25 ms and TTFT under about 1.4 s, and c=32 gives the best throughput with median TTFT still at or under 1.4 s.

Reported 8 GB
Source: @phoenixone7, 2026-08-27
Source: @cute_sc2, 2026-08-27

akumaburn/Qwen3.8-27B-heretic-SmoothQuant-W8A8-INT8: Phoenix17's abliterated INT8 quant with MTP head (about 784K KV on one card)

Phoenix17 (2026-08-29) made an abliterated (heretic) SmoothQuant W8A8 INT8 quant of the 27B: 0/100 hard refusals, 19/100 deflections, KL divergence 0.0110 at top-K 512 against BF16, MTP head included (possibly also abliterated), meant to use the INT8 tensor cores on vLLM. He uploaded it on 2026-08-31 as akumaburn/Qwen3.8-27B-heretic-SmoothQuant-W8A8-INT8 (source akumaburn/Qwen3.8-27B-heretic) and recommends INT8 over INT4 for 2 cards, since it runs on one card with very little loss against BF16. When Mark Linton (2026-09-17) wanted a single-card model with at least 64K context for agentic and tool use, ideally abliterated, cibernox said nothing beats Qwen 27B/28B at 8 bits on one card, and Phoenix17 noted the W8A8 quant leaves room for about 784,000 tokens of roughly FP8 KV, with one card handling 3 simultaneous streams at little slowdown, enough native context for 3 agents.

Reported
Source: @phoenixone7, 2026-08-29
Source: @du_11, 2026-08-30
Source: @mrmannyatx, 2026-08-31
Source: @marklintonca, 2026-09-17
Source: @cibernox, 2026-09-17

philbert440 Qwen3.8-27B Uncensored W4A16 AWQ on two cards

MrManny (2026-08-31) runs philbert440/Qwen3.8-27B-Uncensored-Aggressive-W4A16-AWQ on 2 cards. Phoenix17: it may be faster because the weights are smaller, but quality is another matter, and he suggested comparing it with his W8A8 INT8 quant.

Reported
Source: @mrmannyatx, 2026-08-31
Source: @phoenixone7, 2026-08-31

DUFRelic: heretic W8A8 on one card at 160 W, 3,064 prefill at 2K, 1,284 at 131K

DUFRelic (2026-09-02) runs akumaburn/Qwen3.8-27B-heretic-SmoothQuant-W8A8-INT8 on his server at a 160 W power limit, stock VBIOS, no undervolt or overclock, with 32,768 output tokens: a 2,048-token prompt gives 3,063.68 tok/s prefill, 74.19 tok/s decode and 685 ms TTFT; 32,768 tokens give 2,525.97 / 70.56 tok/s and 12,993 ms TTFT; 131,072 tokens give 1,283.95 / 57.82 tok/s and 102,147 ms TTFT. The engine was not stated (the model is a vLLM-style W8A8 checkpoint).

Reported
Source: @dufrelic, 2026-09-02

Prefill: llama.cpp Q8_0 about 800 tok/s; AWQ W4A16 on vllm-backport much higher

Beam (2026-09-04) got about 800 tok/s prefill on Qwen3.8 27B Q8_0 with llama.cpp. Schaka recommended AWQ W4A16 in vllm-backport for much higher prefill, pointing to the llm-compressor quant soyrsoyr/Qwen3.8-27B-W4A16-AWQ-GPTQ and the wtdcode/vllm-backport fork, not fully current with upstream but a good base; Schaka himself runs Qwen3.8 Flash Next on 8 cards. Beam preferred to stay at INT8 because of reported quality issues with the Qwen3.8 architecture at lower bits.

Reported
Source: @beamreaching1, 2026-09-04
Source: @schakaqt, 2026-09-04

INT8 on one card with stock vLLM 0.29.0: 30 to 60 tok/s

fafinet (2026-09-12) gets 50 to 60 tok/s decode with Qwen 3.8 27B, an INT8 AWQ quant, on one card with vLLM 0.29.0 at default settings. tchelobh gets 30 tok/s for one sequence and 75 tok/s aggregate with 4 sequences, using MTP. regaly, separately, prefers the 27B at Q8_0 over Q4 quants of larger models.

Proven
Source: @reggie7756, 2026-09-11
Source: @tchelobh, 2026-09-12
Source: @fafinet, 2026-09-12

Phoenix17: a 27B quant as a coding-agent backend at 56 to 63 tok/s (5.8M tokens in 28 minutes)

Phoenix17 (2026-09-12) connected a DeepSeek coding harness to his local Qwen 3.8 27B quant on the 170HX and got about 63 tok/s, settling to 56 tok/s, processing 5.8 million tokens in about 28 minutes; engine and quant type were not stated (probably his heretic W8A8). He also finds the 27B thinks too much even at medium effort, and linked a DavidAU fine-tune (DavidAU/Qwen3.8-27B-UltimateDetails2-stage1__The-Harley-Pelican).

Reported
Source: @phoenixone7, 2026-09-12

BF16, one instance per card, as a 4-agent swarm

jjb (2026-09-21) runs, on each of 4 cards, Qwen3.8-27B-BF16 on 1 GPU with 128K context, MTP 3, no-think and vision, and finds 4 Qwen 3.8 instances as a swarm more useful than one GLM 5.3. GalyGames recommends Qwen 3.8 27B for 4 modded 8 GB cards until about 128 GB of VRAM. CUTE: GLM 5.3 Flash or Qwen 3.8 27B are better workers for code and tool calling, while full GLM 5.3 is better at planning and exploring.

Reported
Source: @galygious, 2026-09-14
Source: @cute_sc2, 2026-09-20
Source: @jjb01957, 2026-09-21

chonsworth's vLLM DFlash2 configs for a SmoothQuant W8A8 quant (balanced profile about 3,200 prefill and 110 tok/s)

chonsworth (2026-09-17) first posted Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 with TP=2 (a leftover from experiments), BF16, FP8 e4m3 KV, --max-model-len auto, --mamba-cache-mode align, prefix caching and DFlash2 (z-lab/Qwen3.8-27B-DFlash2, 7 speculative tokens, probabilistic draft sampling), with no speed numbers. On 2026-09-26 he posted:

vllm serve Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 --port 8001 --dtype bfloat16 --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder --reasoning-parser qwen3 --max-model-len auto --max-num-batched-tokens 16384 \
  --attention-config '{"backend": "FLASH_ATTN"}' --mamba-cache-mode align --enable-prefix-caching \
  --per-request-spec-decode-metrics summary \
  --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.8-27B-DFlash2","num_speculative_tokens":5,"draft_sample_method":"probabilistic"}'

On his own "balance benchmark" parameters this gives about 3,200 tok/s prefill, about 110 tok/s decode and about 2,600 ms TTFT at 8k. Stack: a uv-managed venv with vllm>=0.30.0 and CUDA 13.3 (not the wtdcode backport). The card count for this config was not stated; he was also trying ukisai/Swift-Qwen3.8-27b so it fits without spreading across cards. Phoenix17 found that SmoothQuant/QuaRot rotated quants are incompatible with DFlash2, so check acceptance before relying on this pairing.

Reported
Source: @chonsworth, 2026-09-17
Source: @chonsworth, 2026-09-26

Swift and Swift-1.5 heretic quants (Phoenix17): rotated W8A8, W8A8-DFlash2 and W4A16 GPTQ at 168 tok/s

Phoenix17 (Hugging Face account akumaburn) ran heretic abliteration on Swift-Qwen3.8-27b and on 2026-09-18 published akumaburn/Swift-Qwen3.8-27b-heretic-SmoothQuant-W8A8-INT8: HMMT Nov 2025 144/150 = 96.00%, matching the unquantised Swift, and AIME 2026 147/150 = 98.00% (base 98.67, Swift 94.00), with performance charts for a single CMP 170HX; he later called it the excellent-quality option when not using DFlash2. The same day he published akumaburn/Swift-Qwen3.8-27b-heretic-W4A16 (GPTQ), decoding at 168 tok/s with DFlash2 but with KL divergence about 4.7 times worse than the W8A8 quant, so expect non-negligible quality loss; it prefills about half as fast while DFlash2 decode is about 3x faster at low concurrency, with a low VRAM footprint. QuaRot rotation is incompatible with DFlash2, which is why the rotated W8A8 build cannot use it. The older history dates the Swift W8A8 upload to 2026-09-23, when it beat the base Swift model in one benchmark and DUFRelic started running it the same day. Phoenix17 (two cards, one used for ComfyUI) then released the Swift-1.5 heretic quants (thread from 2026-09-26, release 2026-09-27) as SmoothQuant-W8A8-INT8, W8A8-DFlash2 and W4A16; the heretic KL divergence during the work was 0.0602. Heretic is the uncensoring process, so the quants are already uncensored, and the Swift fine-tune aims to keep capability while using far fewer thinking tokens. Drafter advice: MTP for the more accurate rotated quant, DFlash2 for faster decode, using the long-context drafter on his Hugging Face page. 68x took part in the thread.

Reported 8 GB
Source: @phoenixone7, 2026-09-18
Source: @phoenixone7, 2026-09-23
Source: @dufrelic, 2026-09-23
Source: @phoenixone7, 2026-09-26
Source: @dufrelic, 2026-09-26
Source: @68x0, 2026-09-26

akumaburn/Swift-Qwen3.8-27b-heretic-W8A8-DFlash2 and the akumaburn/vllm-dflash2 fork (110 to 120 tok/s, 18,621 prefill at c=32)

Phoenix17 (2026-09-18) published an unrotated W8A8 quant, akumaburn/Swift-Qwen3.8-27b-heretic-W8A8-DFlash2, that keeps the fast W8A8 prefill and decodes about 110 to 120 tok/s via DFlash2 at slightly worse accuracy than the rotated build; he calls it the best general-purpose build for most people at concurrency up to 8. Prefill at 1024-token input and c=32 is 18,621 tok/s. It needs his patched vLLM fork akumaburn/vllm-dflash2 for the quantised drafter, plus the drafter akumaburn/Qwen3.8-27B-DFlash2-W4A16-longctx to go past the native context window; his launch.sh defaults to the second card and needs the drafter path set. Speculative decoding is lossless relative to the target, but the target is a less accurate unrotated quant, hence "slightly worse quality". The fork is a full vLLM fork adding DFlashProposer (method "dflash") and qwen3_dflash, qwen3_dflash2 and laguna_dflash model files, with no sm_80 or CMP-specific code and the unmodified upstream README; treat it as untrusted code to review before building.

Reported
Source: repo github.com/akumaburn/vllm-dflash2 by akumaburn on GitHub, posted by @phoenixone7, 2026-09-18
Source: @sojun80, 2026-09-18
Source: @c133, 2026-09-21

Gadfly: Qwen 27B and Muse Glimmer 30B with DFlash2 on one 8 GB card (2k prompt, 500-token generation)

Gadfly (2026-09-18), one 8 GB card, DFlash2, 2k prompt, 500-token generation, model-default temperature. Qwen 27B: cold prefill about 1,675 tok/s at C=1/4/8; aggregate decode 126.2 (C=1), 345.7 (C=4, 86.4 per request) and 521.2 (C=8, 65.2 per request); cached TTFT 2.65 / 7.17 / 13.17 ms. Muse Glimmer 30B: cold prefill about 3,570 tok/s; decode 132.4 (C=1), 287.2 (C=4, 71.8 per request) and 422.1 (C=8, 52.8 per request); cached TTFT 2.86 / 5.91 / 15.62 ms. Phoenix17 said his W8A8-DFlash2 quant should double that prefill.

Reported 8 GB
Source: @gadfly3121, 2026-09-18
Source: @phoenixone7, 2026-09-18
Source: @matstatman, 2026-09-18

Long-run throughput: 125 tok/s averaged over about 20M tokens, single stream

Phoenix17 reported 125 tok/s averaged over about 20M tokens, single stream, across several long jobs with steady performance; the figure includes input tokens (the output share is much smaller), and the model was not named in the message. chonsworth said this was close to his own experience, with about 3.8k tok/s prefill, on a 2-card Qwen3.8 Flash Next benchmark.

Reported
Source: @phoenixone7, 2026-09-30
Source: @chonsworth, 2026-09-30

Studebaker's single-card vLLM flags, and MTP under concurrency

Studebaker's single-GPU vLLM flags: --served-model-name qwen3.8 --gpu-memory-utilization 0.90 --enable-prefix-caching --max-num-batched-tokens 8192 --load-format host_bounded --speculative-config {"method":"mtp","num_speculative_tokens":3}. With several concurrent agents (for example 4 subagents on mtp:3) speculative decoding pushes the card into being compute-bound and decode speed suffers; 8 streams with mtp:3 reached only about 350 tok/s aggregate. Over long runs Qwen's own FP8 checkpoint performed slightly better than the alternatives, within margin of error, and "always seems to keep it together". For W8A16 the activations are BF16; FP8 KV should prefill a bit faster than BF16 KV, and decode should reach about 80 tok/s.

Reported
Source: @_studebaker_, 2026-08-22
Source: @matstatman, 2026-08-22
Source: @trojis8032, 2026-08-22

Phoenix17 on INT8 against INT4: 3 streams at 56 tok/s on W8A8, near 200 with W4A16 + DFlash2

Phoenix17 (2026-08-30): the SmoothQuant W8A8 27B handles about 3 concurrent streams at full speed, about 56 tok/s each, without DFlash; accepting INT4 quality loss roughly doubles that, and for single-stream chat INT4 (W4A16) with DFlash2 reaches about 170 tok/s at native context and near 200 tok/s on one card. The DFlash2 Triton kernels top out at about 365K context for W4A16, while W8A8 reaches about 850K. He sticks with INT8 for serious work because INT4 is borderline lossy and loses INT8's prefill speedup and parallelism, which matter for coding, and suggested AWQ or GPTQ for INT4-like speed with less loss. On 2026-09-18 he added that he uses W8A8 rather than W8A16 to hit the CUTLASS INT8 kernel, which he credits for roughly doubling prefill (scaling to c=32), and that W4 weights decode about 60 tok/s faster than W8 because decode is bandwidth-bound but prefill slower, which he believes is a kernel issue since prefill is compute-bound; he proposed writing a better W4/INT4 Ampere kernel for vLLM as a community project. The VRAM W4 frees buys context, though DFlash2 uses some of it back.

Reported
Source: @phoenixone7, 2026-08-30
Source: @tokukarin, 2026-09-18
Source: @meatsus, 2026-09-18

Speculative decoding

More on the methods: Speculative decoding on the 170HX.

DFlash or MTP for coding on the 27B

Contested

matstatman (2026-08-18 to 2026-09-04): DFlash2 gives 200+ tok/s in benchmarks but about 10 tok/s when coding, with draft acceptance 0 to 25% on real codebases. Source: @matstatman, 2026-08-18
fallengod measured DFlash2 at 1.91x MTP on GSM8K real text at C1 (entry below), while Studebaker saw only about +2 tok/s over MTP on FP8. Source: @fallengod9962, 2026-08-20
Source: @_studebaker_, 2026-08-20
DFlash2 support merged into vLLM by 2026-08-24 (clee), so getting it running stopped being the issue. Source: @c133, 2026-08-24
DUFRelic (2026-10-04): decode speed matters less when the model thinks 10 to 20 times more tokens than it outputs. Source: @dufrelic, 2026-10-04
matstatman (2026-10-04 to 2026-10-17): DFlash acceptance is too low for coding unless temperature is 0 as in the benchmarks, so use MTP with k=3; W8A16 against Q8 GGUF quality is about the same. Miséricorde's 210+ / 110+ numbers were measured single-stream with DFlash. matstatman then tested Flash Next instead because the 27B is slow (one run took hours). Source: @matstatman, 2026-10-04
Source: @matstatman, 2026-10-15
Source: @matstatman, 2026-10-17
Programmer007 (2026-10-15) could not get DFlash running in vLLM at all, the "3090 club" recipe did not work; kellertreppe123 points to multiple working posts on r/LocalLLM. Source: @programmer_007, 2026-10-15
Source: @kellertreppe123, 2026-10-15

orrorin's separate advice (vanilla model, fixed chat template; 90+ tps on dual 5060 Ti under tuned vLLM, not CMP) is under Steps. Source: @_orrorin, 2026-10-11
Source: @globalapathy, 2026-10-11

fallengod: DFlash2 against MTP on real text, 105 against 55 tok/s at C1, and why random-token benchmarks mislead

fallengod (2026-08-20, automated harness, Qwen3.8-27B): DFlash2 looked slower than MTP only because random-token workloads collapse draft acceptance (position-0 acceptance 0.55 to 0.78). On GSM8K real text, at C1 DFlash2 gave 105.2 tok/s (8.1 ms TPOT) against MTP 55.1 tok/s (16.1 ms), 1.91x, and at C8 384.4 against 366.3 tok/s, 1.05x. A 16K-in / 512-out workload is about 90% prefill, where the W8A16 target (dequantised to BF16) loses about 2x to W8A8 INT8 GEMM, the same gap at mnbt 8192. MTP x2 verifies 3 tokens per step while DFlash2 x7 verifies 8 plus a 5-layer draft forward, so on this low-FLOPS card DFlash2 needs about 4.5 to 6 accepted tokens per step to win; pure autoregressive decode at C8 measures only 168 tok/s, so speculation matters more here than on an H200. Related links: greencandle posted lued/Qwen3.8-27B-INT8-W8A16-DFlash2 and clee the DFlash2 blog post.

Reported
Source: @redblood_, 2026-08-19
Source: @c133, 2026-08-19
Source: @greencandle9753, 2026-08-19
Source: @fallengod9962, 2026-08-20

SGLang against vLLM with DFlash2: SGLang faster single stream, vLLM better at high concurrency

Contested

монета (2026-08-25, translated from Russian) gets about 120 tok/s with DFlash2, a W8A16 quant, about 100k context and thinking on, using SGLang; in real work about 100 tok/s while thinking and about 150 during code generation. He moved to SGLang because vLLM DFlash2 support needed patches, and let a Codex agent test and tune overnight on a dedicated server. Source: @monetka.name, 2026-08-25
Jonathany Joerrington (2026-08-25, translated from Russian) uses vLLM with DFlash2 via patches and gets 70 to 90 tok/s in real work when context is not very full, at most 80 to 90 even with memory and core overclock; both note DFlash1 and DFlash2 differ greatly. Source: @jonathanyjoerrington, 2026-08-25
clee (2026-09-09): the official FP8 with DFlash2 runs at 50 to 60 tok/s (spikes to 75) on vLLM, while the same model and drafter on SGLang hold around 80 tok/s with spikes to 120; UnmotivatedGene could not get Qwen Flash Next working well on SGLang. Source: @c133, 2026-09-09
Source: @unmotivatedgene, 2026-09-09
Daran (2026-10-02) tried SGLang with both MTP and DFlash2, then went back to vLLM, which was a little better at high concurrency, though SGLang was still good. segmond found RedHatAI/Qwen3.8-27B-INT4 pretty good on the card. Source: @daran1, 2026-10-02
Source: @segmond, 2026-10-02

DFlash2 on one card: about 160 to 175 tok/s single stream with W4A16, at a cost in context and prefill

Phoenix17 (2026-08-27) found DFlash2 single-stream only: it works but costs context and prefill speed, so he recommends it only for chat. He could not match another user's 212 tok/s, getting 160 at a very small batch size and about 175 tok/s at best; the other benchmark used batch size 1 on a card power-unlocked to 300 W, while his 8 GB cards are limited to 250 W. On 2026-09-17 he put it at about 175 tok/s if context is limited to about 200K and one stream uses DFlash2 with W4A16, with 4-bit weights borderline for quality and DFlash2 suited to a single terminal rather than agents spawning sub-agents. W4A8 also works, probably faster at worse quality; xtj7 recalls it needs a BF16 KV cache, Kanorin confirmed 70 to 150 tok/s with it, and Miséricorde said 180+ tok/s is possible at 16K context; matstatman had skipped DFlash2 because DSpark was poor for coding.

Proven 8 GB
Source: @phoenixone7, 2026-08-27
Source: @matstatman, 2026-09-17
Source: @xtj7, 2026-09-18
Source: @_kanorin, 2026-09-18
Source: @wakeupmbrc1, 2026-09-18

clee: official FP8 with DFlash2 on one card (command and YAML config)

clee (2026-08-24) runs the official FP8 on one card with:

vllm serve Qwen/Qwen3.8-27B-FP8 --served-model-name qwen3.8 --port=8080 --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml --reasoning-parser qwen3 --enable-prefix-caching \
  --speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":8}'

plus froggeric's Qwen-Fixed-Chat-Templates, which instruct using the qwen3_xml tool parser instead of qwen3_coder. Thinking tokens stream at 60 to 75 tok/s at reasoning effort medium; as of 2026-08-24 the DFlash2 PR had merged into vLLM, so building from the PR branch is no longer needed. His 2026-09-02 config (vllm serve --config vllm-qwen38-27b-fp8.yaml, vLLM main at 570735520942b6428ae98f1eb442c6d88adae55e): DFlash2 with num_speculative_tokens 7, max-model-len 262144, max-num-seqs 32, max-num-batched-tokens 8192, gpu-memory-utilization 0.95, prefix caching, and compilation-config max_cudagraph_capture_size 32, which skips capturing 500+ graph shapes that can never run (about 25 to 30 s and more than 1 GiB saved at startup). He says it beats his R9700 running Q4_K_M Qwen3.6-27B and found bugs Kimi K3 missed.

Reported
Source: @c133, 2026-08-24

Phoenix17's one-card launch at 800K context with MTP 3

Phoenix17 (2026-08-31), about 850K context fitting on one card:

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 CUDA_VISIBLE_DEVICES=1 vllm serve Qwen3.8-27B-heretic-SmoothQuant-W8A8-INT8 \
  --tensor-parallel-size 1 --max-model-len 800000 \
  --hf-overrides '{"text_config": {"max_position_embeddings": 800000}}' \
  --gpu-memory-utilization 0.98 --kv-cache-dtype fp8_e4m3 --enable-prefix-caching --enable-chunked-prefill \
  --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice --disable-custom-all-reduce \
  --async-scheduling --trust-remote-code --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

CUDA_VISIBLE_DEVICES=1 pins it to his second card; drop or change it for your setup. He was later unsure the max_position_embeddings override is right and moved to YaRN (below); matstatman was still asking on 2026-10-03 whether anyone had a 400k+ max-length Qwen3.8 27B setup.

Reported
Source: @phoenixone7, 2026-08-31
Source: @matstatman, 2026-10-03

Phoenix17's two-card data-parallel launch (800K context, MTP 3)

Phoenix17 (2026-09-02) gets about 2,600 tok/s single-stream prefill and about 50 tok/s decode with MTP 3 on his heretic W8A8 quant in vLLM; for decode speed W4A16 gives about 100 tok/s with standard MTP and about 175 with DFlash2, but DFlash2 cuts max context from about 850K to about 350K to 365K and needs 16-bit activations. His 2-card command (reposted 2026-09-08, when DUFRelic had run the quant for a week):

vllm serve Qwen3.8-27B-heretic-SmoothQuant-W8A8-INT8 --data-parallel-size 2 --tensor-parallel-size 1 \
  --max-num-seqs 3 --max-model-len 800000 \
  --hf-overrides '{"text_config": {"max_position_embeddings": 800000}}' \
  --gpu-memory-utilization 0.98 --kv-cache-dtype fp8_e4m3 --enable-prefix-caching --enable-chunked-prefill \
  --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice --disable-custom-all-reduce \
  --async-scheduling --trust-remote-code --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Splitting this dense model over two cards for speed is not useful because of the slow PCIe link; he was unsure about the context extension and was trying RoPE scaling instead, and matstatman suggested running a prefill benchmark.

Reported
Source: @phoenixone7, 2026-09-02
Source: @dufrelic, 2026-09-08
Source: @matstatman, 2026-09-08

Phoenix17's single-card YaRN launch to 393K context with MTP 2 (54 to 64 tok/s)

Phoenix17's vLLM command for one card (2026-09-17): vllm serve Qwen3.8-27B-heretic-SmoothQuant-W8A8-INT8 --tensor-parallel-size 1 --max-num-seqs 2 --max-model-len 393216 with --hf-overrides setting text_config.rope_parameters to mrope_interleaved true, mrope_section [11,11,10], rope_type yarn, rope_theta 10000000, partial_rotary_factor 0.25, factor 1.5 and original_max_position_embeddings 262144, then --gpu-memory-utilization 0.97 --kv-cache-dtype fp8_e5m2 --enable-prefix-caching --enable-chunked-prefill --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice --async-scheduling --trust-remote-code and a speculative config of method mtp with 2 speculative tokens. If responses look strange, switch fp8_e5m2 to fp8_e4m3; CUDA_VISIBLE_DEVICES is 0-indexed, and removing it uses all cards. Result: about 54 tok/s, about 64 tok/s after 170tune. He later said he compacts manually above about 380K context because of odd behaviour.

Reported 8 GB
Source: @marklintonca, 2026-09-17
Source: @phoenixone7, 2026-09-17

Prefix caching on the hybrid DFlash2 stack: Miséricorde's unpublished patch (pointer)

Miséricorde (2026-09-01) described an unpublished vLLM patch that keeps prefix caching working on the hybrid Qwen3.8 27B + DFlash2 stack, so a chat turn no longer re-prefills the whole conversation; he had not yet committed it. Pointer (waiting for the author): a longer write-up by @wakeupmbrc1 exists; see Discord.

Reported
Source: @dottoreargus, 2026-09-01
Source: @wakeupmbrc1, 2026-09-01

KV cache and context

KV cache capacity on one card: about two full 262k sessions with BF16 KV, about 600k to 850k tokens with FP8 KV

clee (2026-08-23): with FP8 or INT8 weights loaded, one card holds enough KV for basically two full 262,144-token sessions, so serving 4 users from one card is rough. Studebaker: FP8 KV gives about 2.63x the context of BF16 KV, and FP8 weights plus FP8 KV reach about 690k tokens on one card at gpu_memory_utilization 0.90 (he caps at 0.90 for reliability), making 4 to 6 concurrent 128k sessions possible; he had validated FP8 weights with BF16 KV on technical codebases but not yet quality-tested FP8 KV. On 2026-08-25 he measured about 630k tokens of context with FP8 weights and FP8 KV on one 64 GB card, enough for 4 to 8 concurrent sessions (128k to 64k each) with headroom against OOM; with 4 concurrent sessions and MTP 2 he gets 70 tok/s decode up to 32k context, falling to about 30 tok/s near 100k. On 2026-08-30 he found batch 4 at 128K context rock solid at about 70 to 80 tok/s, and on 2026-09-02 repeated "8bit weights @ 8bit kv nets ~600k context and runs great": one card is good, and two add flexibility to run two different models. A 4-bit KV cache (or a Q4 quant) gives much more headroom; giaki3003 pointed to TurboQuant for KV. Phoenix17 (2026-09-27): with his W8A8 quant one card has room for about 695K tokens of FP8 context, splittable across sessions (more VRAM in vLLM goes higher but risks OOM); one card handles about 3 concurrent sessions without losing decode speed and about 5 at slightly lower decode with 150k context each. Open questions: DUFRelic (2026-08-31) asked whether FP8 KV costs quality against BF16 and Dino whether quality degrades past the native 262,144 context with RoPE scaling, both unanswered. tchelobh (2026-09-09) finds the 27B INT8 W8A8 "a monster" with 4 parallel agents at 200k tokens each (no card count or engine given). PJ found lued/Qwen3.8-27B-INT8-W8A16-DFlash2 the fastest quant he had tried, and clee suggested FP8; Programmer007 asked the question Phoenix17 answered.

Proven 8 GB
Source: @c133, 2026-08-23
Source: @_studebaker_, 2026-08-23
Source: @giaki3003, 2026-08-30
Source: @dufrelic, 2026-08-31
Source: @tchelobh, 2026-09-09
Source: @_studebaker_, 2026-08-25
Source: @dino_38975, 2026-08-25
Source: @_studebaker_, 2026-09-02
Source: @zingy90210, 2026-09-02
Source: @c133, 2026-09-02
Source: @phoenixone7, 2026-09-27
Source: @programmer_007, 2026-09-27

One card's sweet spot: the 27B at BF16 with a full BF16 KV cache

machinezero (2026-07-29) said the unlocked card is good for running Qwen 3.6 27B at BF16 and not much else, because the model landscape jumps from 27 to 35B to over 200B with no good 70B releases; ju5t5omedude agreed that for coding nothing between 27B and about 200 GB competes and anything bigger means 4+ card rigs, and CUTE runs Qwen 3.6 on his workstation. On 2026-08-17 machinezero saw no benefit over a $600 V100 32GB for Qwen 27B; matstatman answered that the 64 GB card runs it at BF16 instead of Q8, and Ryko added that it fits the model plus the full BF16 KV cache, which suits SpiralGlide's goal of moderately sized dense models on one card. No measured numbers were given in these threads.

Theory 8 GB and 10 GB
Source: @machinezer0, 2026-07-29
Source: @ju5t5omedude, 2026-07-29
Source: @cute_sc2, 2026-07-29
Source: @machinezer0, 2026-08-17
Source: @matstatman, 2026-08-17
Source: @vektor_prime, 2026-08-17
Source: @spiralglide, 2026-08-17

Minachist/Qwen3.8-27B-INT8-AutoRound (revision linear-attn-bf16-gs128) as the recommended INT8 quant

Ryko (2026-08-29) recommends Minachist's AutoRound INT8 quants of Qwen3.8-27B for the CMP (no FP8 hardware), specifically the branch/revision linear-attn-bf16-gs128 of Minachist/Qwen3.8-27B-INT8-AutoRound, and says the quantiser publishes good manual benchmarks. giaki3003 adds that if VRAM allows you can run W8A16; roger_dammit had asked on 2026-08-25 whether people were still on lued W8A16-MTP or had moved to W8A8 or AutoRound.

Reported
Source: @roger_dammit, 2026-08-25
Source: @vektor_prime, 2026-08-29
Source: @giaki3003, 2026-08-29

More cards

Background on TP, PP and data parallel over these links: Parallelism over the PCIe link.

Two cards: TP2 usually beats one card, depending on x16 against x4 links (Miséricorde); programmerboi recommends PP2 for batched throughput, especially without x16. cibernox reports TP2 works well after the x16 mod. Source: @wakeupmbrc1, 2026-10-14
Source: @programmerboi, 2026-10-14
Four cards, TP4 (turbine, no recipe published yet): 440 tok/s decode with speculation and about 4k tok/s prefill at 190 W per card, using his own all-reduce traffic compression (2.2k prefill without it). His roofline: 4 x 1.5 TB/s = 6 TB/s, so 27B at 4-bit caps decode near 440 tok/s without speculation; the limit is all-reduce latency over the slow PCIe. Source: @turbine, 2026-10-06
Source: @turbine, 2026-10-06
Source: @turbine, 2026-10-06

Two cards for a 27B: one instance per card, PP2, Ray or data parallel

Contested 8 GB

matstatman (2026-08-22) runs two vLLM instances, one per card, serving lued/Qwen3.8-27B-INT8-W8A16-MTP to Claude Code with about 5 audit subagents plus one coding agent; his args include --dtype bfloat16 --kv-cache-dtype fp8_e4m3. Under load each card ran GPU 1,365 MHz, MEM 1,728 MHz, 75 to 77 C and 201 to 212 W of 250 W; his throughput charts are per-card aggregates from vLLM's Prometheus metrics. MTP acceptance was too low for him, and with 4 streams per card prompt processing dragged output tok/s down. Studebaker was in the thread. Source: @matstatman, 2026-08-22
Source: @_studebaker_, 2026-08-22
matstatman (2026-08-23) tested serving the 27B across cards with Ray and found no speedup over running solo on one card. Source: @matstatman, 2026-08-23
matstatman (2026-09-05): a Claude Code session against a local Qwen 27B on his 2 cards ran 7 h 53 m 57 s and downloaded 1.2M tokens, about 40 tok/s. Source: @matstatman, 2026-09-05
DeeJay (2026-09-08): with 2 cards the 27B is the better choice and can serve about 16 concurrent sessions with little per-session slowdown; when matstatman found requests very slow at c=16 on two cards, DeeJay advised 2 separate instances at c=8 each, while matstatman uses Ray to share the KV cache across the two cards. Source: @deejay9983, 2026-09-08
Source: @matstatman, 2026-09-08
DUFRelic (2026-09-08) serves Phoenix17's W8A8 quant as one vLLM session per card behind LiteLLM with a custom load balancer: against PP2, single-request performance was about equal (PP2 gives more KV cache), but two separate instances were faster for concurrent requests. He prefers this over vLLM data parallel because one GPU can go offline while the other keeps serving, and DP is not KV-cache-aware when routing. Source: @dufrelic, 2026-09-08

Studebaker on two cards: one FP8 instance per card, 4,000 prefill / 680 decode aggregate, 122 tok/s best single stream

Studebaker (2026-08-29) runs Qwen3.8-27B FP8 on two cards, one instance per card: about 2,000 tok/s prefill and 340 tok/s decode per card at concurrency 4, 4,000 prefill / 680 decode aggregate across both GPUs and sessions. His highest single-stream decode was 122 tok/s at temperature 0 with MTP 3; roger_dammit could not exceed 120 tok/s at low context. Studebaker runs 8 concurrent agent sessions with a communication layer (independent or paired adversarial review) with Opus orchestrating.

Reported
Source: @_studebaker_, 2026-08-29
Source: @roger_dammit, 2026-08-29

matstatman's two-card tests: 1,963 prefill on one card against 595 on PP2 at Gen2 x4, then MTP k=3 via PR #55081

matstatman (2026-08-18, vLLM, a 10,260-token prompt): single card with MTP 5.23 s = 1,962.9 tok/s prefill; dual card with the second card at PCIe 2.0 x4 17.25 s = 594.9 tok/s, and disabling MTP did not help; nvtop showed card 0 at Gen2 x16 and card 1 at Gen2 x4 with PCIe traffic peaking. giaki3003 said prefill that bad means a misconfiguration, since PP should not move much over PCIe, and Studebaker said use PP=2 rather than TP=2; matstatman hit "dspark does not support PP" and later admitted he had started the wrong vLLM. On 2026-08-19 he published results for Qwen/Qwen3.8-27B-FP8 with the Doopeworld/Qwen3.8-27B-DSpark-vLLM drafter, k=7, PP=2 on 2 cards in a gist, with local patches to run DSpark under PP that he offered to share. From 2026-09-02 he ran Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 with Ray (pipeline split, not tensor parallel). One card: prefill 4,377.6 tok/s at about 8K, 3,277.7 at 32K and 1,842.8 at 100K, decode about 38.6 tok/s; two cards: 4,569.6 / 4,856.9 / 3,221.7 prefill with the same decode (about 38.2). Adding MTP k=3 via vLLM PR #55081 (a tip ST relayed from the LocalLLM Discord) raised decode to 70.0 tok/s on fresh code (75.7% acceptance) and 82.25 on edit-heavy output (99.6%, 3.99 tokens per pass), with 100K prefill at 3,045.7; the config is in his gist, and the benchmark comes from 0xBakeer's Qwen3.8-27B-FP8-on-a-single-DGX-Spark repo. On 2026-09-14 he reran with TP=2 at x16 (model not named; speculative decoding averaged up to 4 tokens per pass): prefill 2,091 tok/s at 8K, 1,942 at 32K and 1,566 at 100K, decode 53.6 tok/s on fresh code (68.2% acceptance) and 65.9 to 67.2 on edit-heavy output. He saw no speedup from adding cards and asked why pipeline parallel gives fewer tok/s at x16 than at x4 (unanswered).

Reported
Source: @matstatman, 2026-08-18
Source: @giaki3003, 2026-08-18
Source: @_studebaker_, 2026-08-18
Source: @matstatman, 2026-09-02
Source: @stcord, 2026-09-02
Source: @matstatman, 2026-09-14

TinyBeast: two cards on a B550 with an x8x8 riser, one model per card, W8A8 against W4A16

MrManny runs 2x 170HX on a B550 board with an RGEEK x16 to x8x8 bifurcation riser, nonstop without issues, one model per GPU in vLLM. Both instances: --max-model-len 262144, KV fp8_e4m3, compressed-tensors, --gpu-memory-utilization 0.92, MTP speculative 3 tokens, tool parser qwen3_coder. Live bench: Qwen3.8-27B W8A8 INT8 about 3,850 tok/s prefill and 34 tok/s decode; Qwen3.8-27B-Uncensored-AWQ W4A16 (twolven-awq-mtp) about 3,280 prefill and 75 decode, 2.2x the decode because decode is memory-bound. His dashboard averages had shown 75.1 gen / 6,204 prefill (W8A8) and 120.3 gen / 14,221 prefill (W4A16) with prefix caching. Jeremy (Ends) noted the riser is passive (the board must support x8x8 bifurcation; see Risers and bifurcation) and that a PLX card would add P2P DMA; Arcanum and bayley_w took part in the thread.

Reported
Source: @mrmannyatx, 2026-09-02
Source: @_arcanum_, 2026-09-02
Source: @bayley_w, 2026-09-02
Source: @endsjeremy, 2026-09-02

Known issues

The 27B thinks a lot: matstatman saw one 2 h 45 min thought and moved to Flash Next on four cards. Source: @matstatman, 2026-10-16
clee suspects a host RAM bit flip (48 h clean memtest, then an unexplained page cache corruption): use ECC memory on these hosts. Source: @c133, 2026-10-16
The 27B against Flash Next is a matter of taste: see Recipes.
riko got random errors in real tasks with the 27B at first; "with hardware everything is fine now", and he runs it at W4A16 on a single 64 GB card. The cause of the earlier errors was not stated. See Discord. Source: @riko6253, 2026-10-25

Repeated tool-call loops in Claude Code (FP8 and BF16, xhigh)

matstatman's Qwen3.8-27B FP8 on a 170HX repeatedly called the same tool in Claude Code at xhigh reasoning (no MTP), and BF16 at 160k context showed the same loop (identical Search calls repeating), so he does not expect INT8 to help and planned a Claude Code permission hook to deny repeated tool use (2026-08-29). Ryko suggested checking the MTP + prefix-caching bug and the sampling settings (he sometimes raises temperature slightly and dislikes repeat penalty); giaki3003 said Qwen3.8 Flash does not have the problem and some people fixed it by changing temperature. matstatman posted a gist of the loops on 2026-09-04, and ST suggested porting Unsloth's self-healing tool-call logic into one's own agent.

Reported
Source: @matstatman, 2026-08-29
Source: @vektor_prime, 2026-08-29
Source: @giaki3003, 2026-08-29
Source: @matstatman, 2026-09-04
Source: @stcord, 2026-09-04

Context-compaction loop in LM Studio on a "40gb card": quantize the weights, keep KV in BF16

supermanisdum, who had taken a week to work out cooling, ran Qwen3.8 27B in LM Studio with opencode at an 85k context; it kept compacting and forgetting, and reasoning appeared stuck at xhigh despite selecting Medium (2026-09-22). He described his card as "the 40gb card" (an unusual capacity, not explained). Zagitta advised not running the full BF16 model but a quant, to reach 256k context, keeping the KV cache in BF16; matstatman suggested Q8/INT8 for that capacity and runs INT8 on 2x 64 GB cards (links in the thread: github.com/Don-Chad/ninfer-3090, huggingface.co/ukisai/Swift-Qwen3.8-27b).

Reported
Source: @supermanisdum, 2026-09-22
Source: @zagitta, 2026-09-22
Source: @matstatman, 2026-09-22

Qwen3.6-27B tool-call loops: speculative-decode edge cases fixed in newer vLLM; quality against DeepSeek

Contested

DeeJay saw Qwen hallucinate and loop on tool calls about 50k tokens into a Hermes session on llama.cpp at Q6; lowering temperature helped but did not fix it. Source: @deejay9983, 2026-08-11
Mountains: completed vLLM PRs rewrote tool calling to handle MTP, the trouble having come from speculative-decode border cases with extra MTP tokens; Huihui-Qwen3.6-27B-abliterated-FP8 runs for him without tool-call problems or loops. Source: @mountains1605, 2026-08-11
machinezero: an Opus/Fable judge agreed with 98% of DeepSeek 0731's work but only 50 to 70% of Qwen3.6-27B's, which needed 2 to 3 rework rounds. Source: @machinezer0, 2026-08-11
mfj4 and quark2world preferred Gemma 31B (8-bit) or "GRM" 27B for Hermes. Hardware is not stated for most of these reports. Source: @mfj4, 2026-08-11
Source: @quark2world, 2026-08-11

The July and August Qwen3.6 material lives on Qwen3.6 27B and 35B-A3B on one card (July and August 2026).

A Reddit "84 tok/s" single-card claim not reproduced without the custom vLLM patches

Contested 8 GB

matstatman (2026-08-23) asked for the config behind 84 tok/s in a Reddit post and said he tested it and does not get 84 tok/s (possibly a heat issue); he noted W4A16 and Q4 both mean 4-bit weight-only quantisation. Source: @matstatman, 2026-08-23
DUFRelic asked whether he applied all the custom vLLM patches. Source: @dufrelic, 2026-08-23
Zorg33: with DSpark you should get 100+. The model and exact flags were not given. Source: @zorg33, 2026-08-23

Swift-Qwen3.8-27b: same accuracy for about half the thinking tokens (claimed), and the skepticism

Contested

cressida (2026-09-14) shared UkisAI Swift-Qwen3.8-27B, claimed to keep XHIGH reasoning quality with 58.3% fewer thinking tokens and 1.95x speed, a Flash version to follow. Source: @cressida1881, 2026-09-14
bayley_w was skeptical: post-training an already heavily post-trained model rarely gives free performance, or Alibaba would have done it. Source: @bayley_w, 2026-09-15
Phoenix17 (2026-09-17) relayed ukisai's benchmarks for Swift against base (BF16, x5 runs, xhigh): GPQA-Diamond 88.4% to 88.3% with 58% fewer median tokens (6,642 against 2,771; base medium 84.1% at 1,753), LiveCodeBench v6 76.8% to 81.6%, Terminal-Bench 2.1 66.7% to 65.8%, MMLU-Pro 85.5% to 85.0%, IFBench 73.5% to 71.8%, AIME 2026 98.7% to 94.0% and HMMT Nov 2025 99.3% to 96.0%. His summary: it thinks a little more than medium and scores near xhigh, beating base-medium by 4 points at about 1.6x its tokens. segmond and cibernox took part. Source: @segmond, 2026-09-17
Source: @phoenixone7, 2026-09-17
Source: @cibernox, 2026-09-18
MrManny found it fast and good on a 5090. Source: @mrmannyatx, 2026-09-18
Zagitta later recommended it as an anti-yapping option. No CMP-specific benchmarks were posted for it beyond Phoenix17's heretic quants (above). Source: @zagitta, 2026-09-22

Kanorin: W4A16 "pearson" + DFlash2 at 262K, fast tokens but slow agent tasks; which quant to use

Contested 8 GB

Kanorin runs a Qwen3.8-27B-W4A16-pearson quant with the drafter Qwen3.8-27B-DFlash2-W4A16 (spec method dflash2) at 262K context on a 64 GB card at 70 to 160 tok/s (he also wrote 70 to 150), but simple bug tickets took 22 to 30 minutes against 5 to 10 for Claude. W8A16 could not reach 70 tok/s for him, and a single card did not fit 262K with it. Source: @_kanorin, 2026-09-18
segmond and Miséricorde: use medium reasoning effort rather than disabling thinking. Source: @segmond, 2026-09-18
Source: @wakeupmbrc1, 2026-09-18
xtj7: try FP8 or W8A16 (he averages about 60 tok/s on FP8 at xhigh effort), or a different quant in case this one is broken. Source: @xtj7, 2026-09-18
regaly found FP8 faster than any INT8 and better than any W4A*. Source: @reggie7756, 2026-09-18
Xeax found W8A8 best and free of looping; kellertreppe123 took part in the thread. Source: @xeax, 2026-09-18
Source: @kellertreppe123, 2026-09-18

Can a W8A8 quant outperform its source on a saturated benchmark?

Contested

Phoenix17 said his Swift-1.5 quant outperforms the source; he later clarified the source is the Swift fine-tune, not base Qwen (98% against 96%), and that W8A8 is faster than BF16. Source: @phoenixone7, 2026-09-27
Takka called it a red flag: scores near 99% are saturated, 96% is within noise of 99% even though it is a 4-fold increase in failures, and a benchmark nearer 60% with enough problems is needed. His theory: activation quantization hurts more than weight quantization, and the KV cache is sensitive for the same reason. Source: @spectakkular, 2026-09-27
Phoenix17's reply: 300 rollouts pooled give a 2.7 pp minimum detectable difference (ruling out more than about 1.6 problems in 60 lost), and a quant-only KL divergence of 0.0264 on unsaturated data; he sees degradation only past about 400K context, and the benchmarks never exceeded about 120K tokens. PersonA took part in the thread. Source: @phoenixone7, 2026-09-27
Source: @persona_80337, 2026-09-27

Sources

Reddit recipe by Miséricorde (@wakeupmbrc1, posted 2026-08-30), reproduced by kellertreppe123, who posted it on Discord on 2026-10-05.
Related: Quantization on the 170HX, Engines, Benchmarks, Speculative decoding on the 170HX, Parallelism over the PCIe link, Qwen3.6 27B and 35B-A3B on one card (July and August 2026).