Qwen3.6 27B and 35B-A3B on one card (July and August 2026)
More actions
Before Qwen3.8 arrived, Qwen3.6 27B (dense) and Qwen3.6 35B-A3B (MoE, about 3B active) were the models most owners ran on one unlocked card. This page collects the single-card reports from July and August 2026 by engine: the first vLLM references, the 10 GB card at its 40 GB profile, llama.cpp, vLLM tuning, SGLang, the 35B-A3B MoE, and the July argument about MTP. Every number is one person's or one repository's reading, with its own quant, engine version, power cap and benchmark method, so compare rows with care. For the model that replaced it on one card, see Qwen3.8 27B on 1x 8 GB (W4A16 vLLM).
The first references
Single-card dense baselines in vLLM (July): 27B BF16 19.2 tok/s, AWQ-INT4 58.5
AKuchen benchmarked one 8 GB card unlocked to 64 GB in vLLM:
| model | decode, 1 request (tok/s) | aggregate, 4 parallel (tok/s) | prefill (tok/s) |
|---|---|---|---|
| Qwen3.6-27B BF16 | 19.2 | 71.1 | 2,231 |
| Qwen3.6-27B-AWQ-INT4 | 58.5 | 214.8 | 2,044 |
| Qwen3-32B-AWQ (dense) | 52.9 | 205 | 1,755 |
chris.via.egnatia argued that single-thread MoE on these cards is slower than on a 3090 and that vLLM numbers need a
clean llama.cpp q4_k_m cross-check.
Reported 8 GB
Source: @akuchen_66498, 2026-07-20
Source: @chris.via.egnatia, 2026-07-20
Highwayaiexpose repo: the first single-card vLLM reference, 97 to 139 tok/s with MTP
baltazar shared the results-only repo Highwayaiexpose/CMP-170hx-64gb-LLM-benchmarks on 2026-07-22, and arnold pointed at its roughly 97 tok/s decode on a 27B. The repo reports one 8 GB card at 64 GB with the 300 W VBIOS, driver 610, vLLM 0.24.0, batch 1 and temperature 0, launched as:
vllm serve <model> --gpu-memory-utilization 0.95 --kv-cache-dtype int8_per_token_head \
--mamba-cache-dtype float16 --max-num-seqs 1 --mamba-cache-mode align \
--speculative-config '{"method":"mtp","num_speculative_tokens":<k>}'
| model and quant | size | MTP k | prefill (tok/s) | decode (tok/s) | acceptance |
|---|---|---|---|---|---|
| Qwen3.6-27B W8A8 int8 (llm-compressor RTN, CUTLASS INT8; lm_head, visual and MTP kept in BF16) | 29.1 GiB | 3 | 786.1 / 2,618.7 / 3,471.7 (peak) / 2,904.1 at pp128 / 512 / 2048 / 8192 | 97.7 / 99.4 / 92.6 at tg128 / 256 / 512 | 64% |
| Qwen3.6-27B W4A16 GPTQ g128 (Marlin; in_proj_a/b kept in BF16 because N=48 does not divide by 64) | 18.1 GiB | 3 | peak 2,132.6 (pp4096) | 129.8 / 139.5 / 130.3 | 66% |
| Qwen3.6-35B-A3B W8A8 MoE (Avesed/Qwen3.6-35B-A3B-INT8-W8A8) | 39.8 GiB | 2 | 50 to 7,000+ | 203.6 / 205.3 / 206.3 (+32% over no MTP) | 66% |
So int8 prefills about 1.5x faster and int4 decodes about 1.4x faster on the 27B. MoE prefill swung from 50 to over
7,000 tok/s because vLLM had no tuned fused_moe config for this GPU. The repo names k=4 (27B int8), k=3
(27B int4) and k=2 (35B-A3B) as the best MTP depths, but its readme and results file disagree on which k the headline
rows used. No scripts are published, and the rows for other GPUs are third-party. These are the repo's readings.
Eksitus noted it is not apples to apples with the channel's llama.cpp numbers: 8 GB card, 300 W VBIOS, vLLM, a different quant, and no word on whether cmpunlocker or the leaked "Chinese" unlock was used. roger_dammit cited it on 2026-07-23 to argue that decode could double with about 10% overclock, vLLM and W8A8.
Reported 8 GB
Source: repo github.com/Highwayaiexpose/CMP-170hx-64gb-LLM-benchmarks by Highwayaiexpose on GitHub, posted by @baltazarmufii, 2026-07-22
Source: @a.civardagezen, 2026-07-22
Source: @samuel_hayden., 2026-07-22
Source: @eksitus, 2026-07-22
Source: @roger_dammit, 2026-07-23
10 GB cards at 40 GB
In July and August the 10 GB card unlocked to 40 GB (nvidia-smi shows 40,960 MiB). What that profile
leaves usable after reservations is on Memory unlock on the 10 GB card.
Eksitus: Qwen3.6-27B on one 10 GB card with ik_llama.cpp and llama.cpp, with and without MTP
Eksitus ran unsloth/Qwen3.6-27B-MTP-GGUF on one 10 GB card unlocked with cmpunlocker to 40 GB, at PCIe Gen1 x4 (no mod) and 250 W maximum power, on a Supermicro H12DSi-N6 with two EPYC 7713 and kernel 6.18.38. Every run carries a few tok/s of error bars.
| engine | quant (VRAM) | MTP | prefill pp512 / pp2048 / pp8192 (tok/s) | decode tg128 / tg512 / tg2048 (tok/s) |
|---|---|---|---|---|
| ik_llama.cpp 4735 (9d07d868) | UD-Q4_K_XL (17,464 MiB) | off | 296 / 545 / 650 | 33.2 / 34.2 / 31.9 |
| ik_llama.cpp 4735 | UD-Q4_K_XL | --spec-type mtp:n_max=2,p_min=0.0 |
203 / 316 / 337 | 41.1 / 38.3 / 35.8 |
| ik_llama.cpp 4735 | UD-Q8_K_XL (35,718 MiB) | off | 271 / 584 / 697 | 26.4 / 27.3 / 25.8 |
| ik_llama.cpp 4735 | UD-Q8_K_XL | on | 204 / 329 / 363 | 38.2 / 37.7 / 36.8 |
| llama.cpp 10095 (e8e6c7af2) | Q4 | off | 361 / 564 / 722 | 33.1 / 32.7 / 30.5 |
| llama.cpp 10095 | Q4 | --spec-type draft-mtp --spec-draft-n-max 2 |
324 / 497 / 640 | 46.2 / 43.0 / 44.5 (peaks 55 to 59) |
MTP raised decode and cost prefill on both engines. An earlier post of his is in #off-topic.
Reported 10 GB
Source: @eksitus, 2026-07-22
Far: Qwen3.6-35B-A3B q4_k_p at 40 GB in llama.cpp, and whether bandwidth is locked
At arnold's request Far ran Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive at q4_k_p on mainline llama.cpp on a
10 GB card unlocked to 40 GB; the numbers are only in a screenshot. arnold judged it better than his RTX 5060 Ti in
prefill and decode, but not by much, when full HBM bandwidth should give 3 to 4x the decode. simba6959 answered that the
cmpunlocker GitHub tests show memory bandwidth is not locked, and that the limits are PCIe and the SM count (70).
Arcanum said MTP gives 10 to 20% more generation on ik_llama.cpp, and simba6959 suggested that Q6 fits in 40 GB.
Reported 10 GB
Source: @a.civardagezen, 2026-07-19
Source: @farawaythrower, 2026-07-19
Source: @simba6959, 2026-07-19
Source: @_arcanum_, 2026-07-19
bigfoot: six 10 GB cards run Qwen 27B and 35B at 4-bit after the unlock, no bricks
bigfoot tried six 10 GB cards and all of them run Qwen 27B and Qwen 35B at 4-bit after the unlock (his words: "qwen27b and qwen35b"), with no crash and no bricked card. The only limit was cooling: without a good cooler they ran about 10 minutes before overheating and he had to power off. This supports the 40 GB unlock being usable at that time. Airflow advice is on Cooling and power.
Reported 10 GB
Source: @bigfoot1144, 2026-07-24
llama.cpp
llama-bench on one 8 GB card: Q4_K_M 918 / 36.8 vs BF16 1,801 / 24.6 tok/s
acedogblast (65,052 MiB, compute capability 8.0) ran llama-bench on Qwen3.6 27B:
| quant (size) | pp512 | pp1024 | pp4096 | tg128 | tg512 | tg1024 |
|---|---|---|---|---|---|---|
| Q4_K_M (16.67 GiB) | 918.26 | 921.89 | 911.61 | 36.80 | 36.31 | 34.28 |
| BF16 (50.89 GiB) | 1,801.05 | 1,803.38 | 1,771.70 | 24.59 | 24.20 | 22.83 |
MTP does not work in llama-bench, so real use is about 20% faster; he saw about 40 tok/s with full BF16. Q4 was slower than his RTX 3090, while BF16 beat a single 3090. Jeremy (Ends) blamed Q4 dequantisation overhead on this card, and Merom Green suggested Q8 or vLLM and posted his own results image. On 2026-07-30 Josh put acedogblast's "full BF16 faster than Q4" down to quant unpacking overhead: BF16 prefill is about 2x Q4, while Q4 decode is higher. ju5t5omedude, in LM Studio at Gen2 x4, got 36 tok/s from Qwen 27B MTP Q4 with PP and 46 with TP, and 53 tok/s from Q8 with TP, faster than Q4 (card count not stated). More on llama.cpp on CMP cards.
Reported 8 GB
Source: @acedogblast, 2026-07-27
Source: @endsjeremy, 2026-07-27
Source: @meromgreen, 2026-07-27
Source: @acedogblast, 2026-07-30
Source: @_____josh_____, 2026-07-30
Source: @ju5t5omedude, 2026-07-30
vLLM tuning
aaasss's tuning report: vLLM beats llama.cpp, and the best profile depends on context length
aaasss tested llama.cpp against vLLM on one 8 GB card at 64 GB, found vLLM faster, and published a Qwen3.6-27B tuning
report in luannanxian/cmp170hx-gen2. Setup: cyankiwi/Qwen3.6-27B-AWQ-INT4,
FP16 KV, vLLM 0.26.0, driver 610.43.03, Gen2 x4 through a root-port retrain, 200,000-token maximum context,
--gpu-memory-utilization 0.92. The report recommends:
- Single requests up to about 48K: MTP 3,
--max-num-batched-tokens 4096,--kv-cache-dtype float16,--max-num-seqs 1. Decode was 78.26 (Chinese), 137.27 (Python) and 136.76 (JSON) tok/s, and 61.43 at 48K, against about 55.7 without MTP. - One 128K to 200K request: MTP off and 2048 batched tokens. MTP 3 was 25% slower at 128K.
- Two concurrent 128K requests: MTP off,
--max-num-seqs 2, 4096 batched tokens.
vLLM prefilled 2.88x to 2.97x faster than llama.cpp: TTFT 156.55 s against 450.16 s for a 193,835-token prompt. FP8 E4M3 KV decoded at 129 to 150 tok/s but cost about 11% more TTFT at 128K, and INT4 KV regressed badly. In the discussion Samuel Hayden noted that 27B is the dense full model, and NFA said 35B-A3B has only 3B active and 27B is usually much better.
A longer write-up by @aaasss1456 exists (the full tuning report with its screening runs); see the report and Discord.
Reported 8 GB
Source: repo github.com/luannanxian/cmp170hx-gen2 by luannanxian on GitHub, posted by @aaasss1456, 2026-07-29
Source: @jarvis_ai_labs, 2026-07-29
Source: @samuel_hayden., 2026-07-29
Source: @never_fly_away, 2026-07-29
DeeJay: 27B INT8 / W8A8 on one card, about 3,000 to 3,800 prefill and 60 to 93 tok/s
DeeJay (2026-08-01) ran Qwen 3.6 27B at Int8 in vLLM on one 170HX with MTP on, on a Threadripper Pro 3975WX host: about 3,800 tok/s prefill and 60 tok/s generation. He found it very usable but below the roughly 90+ tok/s the memory bandwidth should allow, and blamed single-thread CPU speed: profiling showed one core at 98 to 100% while the GPU idled, CPU pinning changed the result by only 0.05 tok/s, and disabling CUDA graphs halved throughput. More speculative tokens (n=2 or 3) were untested then.
On 2026-08-02 he ran a W8A8 fine-tune ("fable fusion") at about 3,000 tok/s prefill and 75 tok/s generation (card count
not stated in that post) with
--model /model --served-model-name qwen3.6-27b-fable-fusion-int8 --gpu-memory-utilization 0.90 --max-model-len 262144 --speculative-config {"method":"mtp","num_speculative_tokens":2}.
llama.cpp had given him about 200 tok/s prefill and 30 tok/s generation on these cards. On 2026-08-04 he reported 3,800
tok/s prefill and 93 tok/s at Int8 on a single 170HX, against 1,200 tok/s prefill and 78 tok/s for Qwen 3.6 27B Q6 at
262k context with MTP on 2x RTX 3090 in llama.cpp. Qwen needs no special vLLM fork: "just run it at w8a8 with MTP on."
27B fits on one card; with two cards he suggests Qwen on one and Gemma 4 on the other for multimodal.
Reported
Source: @deejay9983, 2026-08-01
Source: @deejay9983, 2026-08-02
Source: @kill3again, 2026-08-02
Source: @deejay9983, 2026-08-04
FP8 vs INT8 W8A16 in vLLM: INT8 prefill about 27% faster, same decode
Studebaker measured Qwen3.6 27B FP8 at 1,851 tok/s prefill and 24.3 tok/s decode, and an INT8 W8A16 build at 2,346 tok/s prefill and 24.8 tok/s decode (2,346 / 1,851 = 1.27). vLLM falls back to its Marlin kernel for FP8 on this card; programmerboi added that Marlin can do a BF16 or FP8 fallback, also for NVFP4 and MXFP4, and suggested trying NVFP4 W4A16. Model pointers: havenoammo/Qwen3.6-27B-INT8-MTP (Studebaker's build for the 170HX), Minachist/Qwen3.6-27B-INT8-Autoround-V2 and Qwen/Qwen3.6-27B-FP8. Background on quant formats: Quantization on the 170HX.
Reported
Source: @_studebaker_, 2026-08-02
Source: @giaki3003, 2026-08-02
Source: @programmerboi, 2026-08-02
INT8 w8a16 vs a GPTQ w8a8 quant: faster prefill, decode about 20 tok/s
Humvee55 (2026-08-07) recommends Qwen 3.6 27B in INT8 or other W8 quants for a single unlocked card, and DeepSeek V4 Flash for four cards. Studebaker pointed to havenoammo/Qwen3.6-27B-INT8-MTP, whose config is weights-only INT8 (compressed-tensors pack-quantized, 8 bits, channel strategy, symmetric, no input activation quantisation: w8a16). He also quantised the original Qwen/Qwen3.6-27B himself with GPTQ to w8a8: prompt processing got faster than with the w8a16 model, but decode stayed around 20 tok/s. He distrusts 4-bit quants for reliability; robbbbbb saw a 4-bit Qwen 27B give a nonsense answer while still finding it good for research through a search MCP.
Reported
Source: @mrmannyatx, 2026-08-07
Source: @scallion_9883, 2026-08-07
Source: @_studebaker_, 2026-08-08
Source: @robbbbbb7973, 2026-08-08
mfj4: Qwen3.6-27B-FP8 on one card, vLLM 0.23, MTP 3, 262k context, 92 tok/s
mfj4 ran the official Qwen3.6-27B-FP8 on one CMP 170HX at TP1 with vLLM 0.23.0:
- Flags:
--max-model-len 262144 --kv-cache-dtype bfloat16 --gpu-memory-utilization 0.92 --max-num-seqs 1 --generation-config vllm --tool-call-parser qwen3_xml --reasoning-parser qwen3, speculative configmtpwithnum_speculative_tokens3, CUDA graphs on, prefix caching off, thinking disabled through the chat-template kwargs. - Environment:
VLLM_MARLIN_USE_ATOMIC_ADD=1 VLLM_USE_FLASHINFER_SAMPLER=0 VLLM_WORKER_MULTIPROC_METHOD=spawn OMP_NUM_THREADS=1. - KV capacity: 400,926 tokens.
- Benchmark (3 warm-ups, 5 requests, 1,802 prompt tokens, 512 output tokens): decode 92.186 tok/s, TPOT 10.880 ms, TTFT 0.9327 s, prefill 1,932.4 tok/s, end-to-end 79.029 tok/s; peak 64 C core and 72 C HBM.
His 2x RTX 3090 baseline used the same method (TP2, BF16 KV, 131,072 context, MTP 3, host-staged NCCL because the 3090s lack working P2P, both capped at 285 W); those numbers are only in a screenshot. DeeJay used to cap 3090s at 225 W and lost only about 3 tok/s.
Reported
Source: @mfj4, 2026-08-07
Source: @deejay9983, 2026-08-07
machinezero: from llama.cpp Q8 to BF16 TP2 to INT8 on one 8 GB card
machinezero worked through three setups at PCIe 2.0 x4 in an ASUS ESC4000 G4:
- Q8 on llama-server: 500 to 800 tok/s prefill and 28 tok/s decode, using 44 of 64 GB at 262k context. Phoenix17 guessed the INT8 tensor cores were not being used, and Impactic said MTP plus tensor split should give around 70 tok/s.
- BF16 with MTP across two cards in vLLM: first about 126 GB at 20 to 40 tok/s. The next day, with vLLM 0.26, BF16 weights and KV, MTP=2, FLASHINFER and
--tensor-parallel-size=2, decode was a steady 20 to 30 tok/s (bursts to 65) and prefill 150 to 200 tok/s (bursts to 700), at 100 to 170 W per card under load and 34 to 58 W idle. ju5t5omedude suggested 3 or 4 MTP tokens. - INT8 weights on one card (
CUDA_VISIBLE_DEVICES=0) in vLLM, after Phoenix17 suggested TheHouseOfTheDude/Qwen3.6-27B-INT8 and later Avesed/Qwen3.6-27B-INT8-W8A8: prefill up to 2,450 tok/s and decode 30 to 55 tok/s at about 250 W constant, with temperatures rising from 57 to 59 C to 69 to 70 C.
The single INT8 card gave 2 to 2.5x the performance of the BF16 TP2 pair with half the GPUs, for the same total power. Decode held around 55 tok/s while writing code, fell to about 28 tok/s (spikes to 32 to 48) at 160k context in Pi Code, and returned to 56 tok/s at 80 to 90k, so context length, not heat, caused the slowdown. He did not say exactly which INT8 checkpoint he used. Why TP2 over a narrow link loses here: Parallelism over the PCIe link.
Reported 8 GB
Source: @machinezer0, 2026-08-07
Source: @impactic, 2026-08-07
Source: @phoenixone7, 2026-08-07
Source: @machinezer0, 2026-08-07
Source: @ju5t5omedude, 2026-08-07
Source: @machinezer0, 2026-08-07
Source: @phoenixone7, 2026-08-07
SGLang
SGLang on one card: Qwen 27B FP8 about 1,800 prefill and 60 decode without MTP
makesier (2026-07-31) relayed someone else's test: Qwen 27B FP8 (Qwen 3.6, before 3.8 existed) on a single 170HX under SGLang reached about 1,800 tok/s prefill and 60 tok/s decode without MTP. That is about the same prefill as his NVLinked dual 2080 Ti, with about 25% faster decode. Whether this run and the Ciender repo below are the same is not confirmed.
Reported
Source: @makesier, 2026-07-31
Ciender repo: SGLang 0.5.16 on a 40 GB and a 64 GB card, MTP 1.8x to 2.8x
The next day makesier shared Ciender/CMP-170HX-SGLang-Qwen3.6-Benchmarks, and Peocua found it the same day: reproducible single-card SGLang 0.5.16 benchmarks of Qwen3.6-27B-INT8-W8A8 on a 10 GB card at 40 GiB (Ciender's data) and an 8 GB card at 64 GiB (data from 9527 vincentdim), with and without MTP. The repo reports, in end-to-end output tok/s:
| workload | 10 GB, MTP off | 10 GB, MTP on | 8 GB, MTP off | 8 GB, MTP on |
|---|---|---|---|---|
| 1 request, 1,024 in / 8,192 out | 41.62 | 90.26 | 42.15 | 116.29 |
| ShareGPT, concurrency 1 | 40.5 | 75.6 | 43.1 | 111.3 |
Without MTP the two cards differ by only 0.8% to 10.4%. With MTP the 8 GB data is higher, and its reported MTP acceptance length is 4.00 against 2.91 to 3.50 on the 10 GB data. MTP speeds up generation, not prefill: it raised TTFT at all six prefill points on both cards. The repo itself says the two runs are not a controlled hardware A/B, and that SGLang code, firmware, clocks, HBM, cooling and platform may differ between them.
makesier found decode not impressive and prefill good; Peocua found usability high, especially with MTP on. Luna noted that the 40 GB configuration runs the VRAM notably slower, so the 10 GB results could improve.
Reported 8 GB and 10 GB
Source: @makesier, 2026-07-31
Source: @lunaroseoc, 2026-08-01
Source: @yuiayanami, 2026-08-01
Source: repo github.com/Ciender/CMP-170HX-SGLang-Qwen3.6-Benchmarks by Ciender on GitHub
35B-A3B
The Highwayaiexpose repo's W8A8 35B-A3B row (about 205 tok/s with MTP 2) is under The first references, and Far's 10 GB run under 10 GB cards at 40 GB.
AKuchen: 35B-A3B AWQ 4-bit on one card in vLLM, 113 tok/s decode, 452 aggregate
AKuchen benchmarked cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit in vLLM on one 8 GB card unlocked to 64 GB: decode 113 tok/s, aggregate 452 tok/s at 4 parallel requests, prefill 1,700 tok/s. He paid about 400 EUR for the card and called it his best hardware investment ever. It was the headline single-card throughput figure of July; Arcanum took part in the thread.
Reported 8 GB
Source: @akuchen_66498, 2026-07-20
Source: @_arcanum_, 2026-07-20
jedilost1: Qwen-AgentWorld-35B-A3B Q8 at a 125 W limit, about 60 tok/s
jedilost1 runs llama.cpp with llama-swap and compared a 3090 and a 170HX side by side. Qwen-AgentWorld-35B-A3B-Q8_0.gguf gave around 60 tok/s on the 170HX with the power limit at 125 W because of limited cooling. SKYY and simba6959 called about 60 tok/s at Q8 near 4090 level. Samuel Hayden asked about nemotron-labs-3-puzzle-75B-A9B Q4, but it needed a specific llama.cpp build. Power-limit settings: Undervolting and power limits.
Reported 8 GB
Source: @jedilost, 2026-07-22
Source: @samuel_hayden., 2026-07-22
Source: @skyy2333, 2026-07-22
Source: @simba6959, 2026-07-22
turbine: bulk captioning on four cards at 150 W, about 30k tok/s prefill and 6k tok/s decode
Not a single-card setup, but the same model at scale: turbine runs qwen3.6-35b-a3b in vLLM on four 170HX cards capped at 150 W each (about 750 W for the whole system) with about 500 concurrent requests. He gets about 30k tok/s prefill and 6k tok/s decode in aggregate (about 60 prefill and 12 decode tok/s per request), captioning Places365 (1.8M images, about 2.5B tokens). He tuned the prompt so the shared prefix is just over 1,056 tokens (the KV page size): it is prefilled once and cached, while the scheduler packs many suffixes into large prefill batches. He also worked on the W4A8 GEMM kernel for large prefill shapes. His estimate against OpenRouter: $321 saved so far and about $750 for the dataset, for $5 to $6 of electricity. cressida and xtj7 took part in the thread.
Reported
Source: @turbine, 2026-09-21
Source: @cressida1881, 2026-09-21
Source: @xtj7, 2026-09-21
MTP in July
MTP on the bandwidth-bound 35B-A3B in vLLM: marginal or negative, helpful at low context
Contested 8 GB
- AKuchen found MTP slower on Qwen3.6-35B-A3B in vLLM, which is already bandwidth bound, with 75% acceptance at 1.75 tokens. On llama.cpp with unsloth/Qwen3.6-35B-A3B-MTP-GGUF, MTP gave +21% (108.4 to 131.3 tok/s), but vLLM without MTP was faster still (147 tok/s). The MTP layers (fc, attention and shared expert kept in BF16 through
modules_to_not_convert) are bandwidth-limited, so draft plus verify overhead eats the gain. Source: @akuchen_66498, 2026-07-21 - robbbbbb and arnold reported that MTP helps at low to medium context and stalls to baseline past about 50 to 100k context. Source: @robbbbbb7973, 2026-07-21
Source: @a.civardagezen, 2026-07-21 - Arcanum reported high (80%) acceptance with Q8 MTP for language and code. Source: @_arcanum_, 2026-07-21
- The next day the Highwayaiexpose repo reported MTP 2 on a W8A8 35B-A3B in vLLM at +32% decode over no MTP (see The first references). Source: repo github.com/Highwayaiexpose/CMP-170hx-64gb-LLM-benchmarks by Highwayaiexpose on GitHub, posted by @baltazarmufii, 2026-07-22
Vcore took part in the thread. Source: @vcore2191, 2026-07-21
MTP depth, long-context slowdown and the later engines are on Speculative decoding on the 170HX.
Related
Qwen3.8 27B on 1x 8 GB (W4A16 vLLM) · Speculative decoding on the 170HX · llama.cpp on CMP cards · vLLM on sm_80: known bugs and patches · Quantization on the 170HX · Parallelism over the PCIe link · Memory unlock on the 10 GB card · 10 GB card · 8 GB card · Gen2 link retrain · Undervolting and power limits · How the 170HX compares · Engines · Benchmarks
Sources
- Repo: Highwayaiexpose/CMP-170hx-64gb-LLM-benchmarks (results only, no scripts, no license file; last commit 2026-07-22), posted on Discord by baltazar.
- Repo: luannanxian/cmp170hx-gen2 (GPL-2.0; last commit 2026-07-29), with aaasss's Qwen3.6 vLLM tuning report under
docs/. - Repo: Ciender/CMP-170HX-SGLang-Qwen3.6-Benchmarks (code Apache-2.0; last commit 2026-08-02), 10 GB data by Ciender, 8 GB data by 9527 vincentdim, posted on Discord by makesier and Peocua.