Qwen3.6 27B on 1x 10 GB (llama.cpp MTP)
From CMP Wiki
More actions
| Reported 1 x 10 GB | |
| Quant | UD-Q4_K_XL GGUF (UD-Q8_K_XL also tested) |
|---|---|
| Engine | llama.cpp llama-server with draft MTP (ik_llama.cpp also tested) llama.cpp 10095 (e8e6c7af2); ik_llama.cpp 4735 (9d07d868) |
| Topology | single card, whole model in VRAM |
| Links | PCIe Gen1 x4, no P2P |
| Decode | 46.24 tok/s |
| Prefill | 496.85 tok/s at 2048 tokens |
| Speculative decoding | MTP, draft-mtp with 2 draft tokens |
| Measured as | tg128 and pp2048, Q4 on llama.cpp with draft MTP (tool and users not stated) |
| VRAM per card | 40 GB profile; 17,464 MiB used at Q4, 35,718 MiB at Q8 |
| Power cap | 250 W maximum |
| Unlock build | cmpunlocker (10 GB to 40 GB) |
| Flags | llama.cpp: --spec-type draft-mtp --spec-draft-n-max 2; ik_llama.cpp: --spec-type mtp:n_max=2,p_min=0.0; both built with Zen 3 and CC 8.0 flags |
| Tested | @eksitus on 2026-07-22 |
| Source | Discord message |
| Confirmations | None yet. |
Qwen3.6-27B from unsloth's MTP GGUF on one 10 GB card at the 40 GB profile, unlocked with cmpunlocker, at PCIe Gen1 x4 (no mod yet) and 250 W maximum power. Eksitus compared stock llama.cpp and ik_llama.cpp, with and without MTP, at Q4 and Q8. The infobox numbers are Q4 on stock llama.cpp with draft MTP; he puts a few tok/s of error on each run. Source: @eksitus, 2026-07-22
Notes
- Q4 on stock llama.cpp: about 33 tok/s decode without MTP and 43 to 46 tok/s with draft MTP (peaks 55 to 59). MTP costs prefill: about 497 tok/s at a 2,048-token prompt with MTP against about 564 without. Source: @eksitus, 2026-07-22
Source: @eksitus, 2026-07-22 - ik_llama.cpp with MTP: about 36 to 41 tok/s at Q4 and 37 to 38 tok/s at Q8. Source: @eksitus, 2026-07-22
Source: @eksitus, 2026-07-22 - Gen1 x4 to Gen2 x4 (2026-07-27, same card, all else unchanged, Q8 on ik_llama.cpp with MTP): prefill at a 2,048-token prompt went from about 329 to 449 tok/s and decode rose slightly (about 38 to 40 tok/s at 512 tokens). With MTP at Gen1 prefill dropped a lot; at Gen2 with MTP it comes close to Gen1 without MTP. The model sits entirely in VRAM; Arcanum asked why the link matters at all. See Benchmarks#Gen1 x4 to Gen2 x4 on one 10 GB card: about 36% more prefill and 3 to 9% more decode. Source: @eksitus, 2026-07-27
Source: @eksitus, 2026-07-27
Source: @_arcanum_, 2026-07-27 - Host: two EPYC 7713 (Zen 3) with DDR4 on a Supermicro H12DSi-N6. Source: @eksitus, 2026-07-24
- Not stated: the context length, KV cache settings, the driver and the benchmark tool. The tables look like single-request tests, but the number of concurrent requests is not stated.
- A longer write-up with full tables by @eksitus exists; see Discord.
Sources
- Eksitus's posts, 2026-07-22 and 2026-07-27.
- Related: Qwen3.6 27B and 35B-A3B on one card (July and August 2026), Qwen3.6 27B on 1x 10 GB (SGLang W8A8), llama.cpp on CMP cards, Speculative decoding on the 170HX, 10 GB card.