Qwen3.6 27B on 1x 10 GB (SGLang W8A8)
From CMP Wiki
More actions
| Reported 1 x 10 GB | |
| Quant | INT8 W8A8 (compressed-tensors, dynamic) |
|---|---|
| Engine | SGLang 0.5.16 (fdebc938); Triton 3.6.0, FlashInfer 0.6.14, PyTorch 2.11.0+cu130 |
| Topology | single card, in a KVM virtual machine |
| Links | PCIe Gen1 x4, no P2P |
| Context | 24,576 tokens (the total token pool at one request; 20,480 at four) |
| Decode, 1 user | 90.26 tok/s |
| Decode, many users | 269.31 tok/s total, 4 users |
| Prefill | 4122.16 tok/s at 2048 tokens |
| Speculative decoding | MTP (EAGLE), 3 steps, 4 draft tokens |
| Measured as | SGLang serving bench, temperature 0; 1 user: 1,024 in, 8,192 out; 4 users: 2,048 in and out |
| VRAM per card | 40 GiB profile (40,960 MiB) |
| Power cap | 250 W |
| Driver | 610.43.02 (CUDA 13.3.73) |
| Flags | MTP through EAGLE, 3 steps, top-k 1, 4 draft tokens; page size 64; chunked prefill 2,048 tokens; static memory fraction 0.94 |
| Tested | on 2026-07-31 |
| Source | link |
| Confirmations | None yet. |
A published benchmark of Qwen3.6-27B (W8A8) on SGLang 0.5.16 on one 10 GB card at 40 GiB, at a 250 W limit, with and
without MTP. The repository holds the scripts, configs and results; its author also ran an 8 GB card at 64 GiB
separately (two runs, not a controlled A/B). makesier and yuiayanami posted it on the Discord.
Source: repo github.com/Ciender/CMP-170HX-SGLang-Qwen3.6-Benchmarks by Ciender on GitHub, posted by @makesier, 2026-08-01
Source: @yuiayanami, 2026-08-01
Notes
All numbers from the repository's 10 GB results page, run on 2026-07-31. Source: repo github.com/Ciender/CMP-170HX-SGLang-Qwen3.6-Benchmarks by Ciender on GitHub
- One request at 1,024 tokens in and 8,192 out: 41.62 tok/s end to end with MTP off and 90.26 with MTP on (2.17x). Across the single-request cases MTP gave 1.79x to 2.27x; mean acceptance length was 2.91 to 3.50 tokens.
- Four concurrent requests at 2,048 in and 2,048 out: 146.73 tok/s in total with MTP off and 269.31 with MTP on.
- Prefill at a 2,048-token prompt: 4,428.56 tok/s with MTP off and 4,122.16 with MTP on; MTP raised the median time to first token by 4.8% to 12.7%.
- Test setup: PCIe at 2.5 GT/s x4 during the test (the card can do 5 GT/s x16); a KVM virtual machine with 44 vCPU of Xeon E5-2680 v4 and 24 GiB of RAM; driver 610.43.02. The token pool was 24,576 tokens at one request and 20,480 at four, so the longest cases were skipped.
- makesier found decode not very impressive and prefill good; Luna noted that the 40 GB configuration runs the memory notably slower than 64 GB. Source: @makesier, 2026-08-01
Source: @lunaroseoc, 2026-08-01
Sources
- Ciender/CMP-170HX-SGLang-Qwen3.6-Benchmarks (results page CMP-170HX-10G.md, scripts and configs).
- Related: Qwen3.6 27B and 35B-A3B on one card (July and August 2026), Qwen3.6 27B on 1x 10 GB (llama.cpp MTP), Speculative decoding on the 170HX, 10 GB card.