Llama.cpp on CMP cards
More actions
llama.cpp runs on the CMP 170HX out of the box and is the easiest engine to start with, but on these cards it is clearly slower than vLLM or SGLang with INT4 or INT8 weights, above all in prefill. Owners keep it for models that do not fit in VRAM with their context, for rigs that mix GPU generations or machines (RPC), for low power draw, and for quick benchmarks that compare across hardware. This page collects what applies to llama.cpp on every model: when to pick it, how to build it, how to split a model across cards, RPC, MTP, the measured runs, the patch script for stock (not unlocked) cards and the work on other CMP cards. Engine choice in general is on Engines; vLLM's own traps are on vLLM on sm_80: known bugs and patches.
When to use it
Engine rule: vLLM or SGLang with INT4 or INT8 weights, not llama.cpp
ikyo benchmarked a Mac Studio against one CMP on Qwen 27B with llama.cpp and a Q8 quant on both, and the CMP lost most tests (2026-10-26). _L_ and Rickcal answered that llama.cpp is very suboptimal on CMPs ("like putting 87 octane in a Ferrari"), that vLLM has been the most plug-and-play with great performance and that SGLang is reportedly faster still; ikyo switched to vLLM the same day. Gnasherrr: the card does not like regular quants and gets major speedups on INT8 or INT4, which is why most owners run W4A16 models. OB88594 told a new 4-card owner on 2026-10-06 that only vLLM is usable because llama.cpp is too slow; Programmer007's llama.cpp run of a 27B Q8 gave only about 800 tok/s prefill at 85 C without the x16 mod; Tokukarin's llama.cpp run of Qwen3.8 Flash Next at 4.27 bpw on one 8 GB card could not pass 60 tok/s even with MTP (2026-10-27). Lab notes count this as confirmed by three or more users.
Older history (2026-07-29 to 2026-09-21): ProfDD got 16 to 20 tok/s with Ollama on Gemma4 31B Q8 GGUF on one unlocked 8 GB card; Merom Green pointed him to llama.cpp, and to vLLM as better still. Studebaker found vLLM the best engine for these cards, with 2-token MTP prediction the best balance, after Danske asked whether MTP needs SGLang. DeeJay found prefill on his 4-card llama.cpp setup worse than a DGX Spark (no numbers); Agile got about 40 tok/s on one card on an old X58 test bench (his RTX 3090 is 20 to 30% faster on the same unnamed model), and Danske replied that vLLM performs better on these cards. fallengod9962 measured Qwen3.8-Flash-Next on 2 cards at 25 tok/s decode and 400 tok/s prefill in llama.cpp against 58 tok/s and about 1,500 tok/s in patched SGLang (matstatman was testing the unsloth UD-Q4_K_XL GGUF on two cards, no result posted; DeeJay called llama.cpp on these cards "kinda booty"). MrManny's self-built llama-server run of huihui-ai's abliterated Qwen3.8 Flash Next UD-Q4_K_XL (111.3 GB, full 262,144 context, --tensor-split 1,1, temp 1.0, top-p 0.95, top-k 20, min-p 0.05) gave only 35 to 45 tok/s decode and 55 to 360 tok/s prefill; Humvee55: vLLM runs better on sm_80. The counterpoint is power: segmond says llama.cpp barely loads the GPUs (about 50 to 60 W per card), which matters without a 240 V outlet, and DeeJay agreed llama.cpp is unbeaten if the power bill matters. On 2026-09-21 dottoreargus said the vLLM recipes he had seen usually perform better, and n3bs said Qwen3.8-Flash-Next already runs on one card in vLLM.
Proven 8 GB and 10 GB
Source: @profdd, 2026-07-29
Source: @meromgreen, 2026-07-29
Source: @_studebaker_, 2026-08-08
Source: @danske1, 2026-08-08
Source: @deejay9983, 2026-08-22
Source: @agile0674, 2026-08-24
Source: @danske1, 2026-08-24
Source: @fallengod9962, 2026-08-26
Source: @deejay9983, 2026-08-26
Source: @matstatman, 2026-08-26
Source: @mrmannyatx, 2026-09-09
Source: @deejay9983, 2026-09-09
Source: @scallion_9883, 2026-09-09
Source: @segmond, 2026-09-09
Source: @dottoreargus, 2026-09-21
Source: @ikyo, 2026-09-21
Source: @n3bs, 2026-09-21
Source: @ob88594, 2026-10-06
Source: @programmer_007, 2026-10-15
Source: @darkmaniac7, 2026-10-20 (lab notes)
Source: @ikyo, 2026-10-26
Source: @lia_oc, 2026-10-26
Source: @rrickk420, 2026-10-26
Source: @gnasherrr, 2026-10-26
Source: @tokukarin, 2026-10-27
How much faster vLLM is: 2x to 50x, depending on who measured
Contested 8 GB and 10 GB
- Ryko found Qwen 3.6 27B about the same speed on both engines, argued that llama.cpp is fine for single requests while vLLM wins on concurrent jobs, and said vLLM's logged prefill rates are inflated (he linked vLLM discussion #17472; members recommended wall-clock timing). Source: @vektor_prime, 2026-08-15
- giaki3003: llama.cpp is 20 to 50 times slower than vLLM on wall-clock prefill (his GLM tests were 50x slower), and tuned vLLM gets 5 to 10 times the numbers in a benchmark he linked. Source: @giaki3003, 2026-08-15
- DeeJay, who had run Gemma 4, Qwen, GLM and DeepSeek on both engines: vLLM is on average 4 to 6 times faster at prefill and about 2 times faster at generation, so 20x is an exaggeration and 5x is realistic. Later (2026-08-27) he added that llama.cpp tends to work out of the box with little fiddling but gave about 300 tok/s prefill and 15 tok/s generation in his large-model tests, and that for this hardware vLLM with pipeline parallel is the way. Source: @deejay9983, 2026-08-15
Source: @deejay9983, 2026-08-27
Dino asked how to run W8A* quants outside vLLM; nobody reported success. No controlled same-model, same-hardware comparison was posted in this period; the measured runs below and the per-model recipes give the gaps model by model. Source: @dino_38975, 2026-08-27
vLLM or SGLang against llama.cpp: majority, dissent and the fits-in-VRAM rule
Contested 8 GB and 10 GB
- Majority: giaki3003 (2026-07-21) said vLLM is much faster than llama.cpp on these chips, "also for single user", with llama.cpp worth it only for exotic architectures or non-CUDA hardware. Samuel Hayden (vLLM usually faster in pure CUDA and better for throughput; pipeline parallelism is one line of config), CUTE (vLLM and SGLang with cards modded to x16; software will take about half a year to get the most out of the cards), DeeJay ("garbagio" for these GPUs), gpulover ("llama.cpp works terrible on 170hx"), AlessandroAU ("vllm is MUCH faster") and clee (vLLM or SGLang for maximum performance) agree, and David said after trying vLLM he would never go back to llama.cpp. Measured: Dino got vLLM 3000 / 160 against llama.cpp about 1000 / about 50 tok/s (prefill / decode) on Qwen3.8-27B. Source: @giaki3003, 2026-07-21
Source: @samuel_hayden., 2026-07-22
Source: @giaki3003, 2026-07-22
Source: @cute_sc2, 2026-07-30
Source: @deejay9983, 2026-07-30
Source: @artistosu, 2026-08-10
Source: @alessandroau, 2026-08-11
Source: @artistosu, 2026-08-19
Source: @giaki3003, 2026-08-19
Source: @giaki3003, 2026-09-01
Source: @c133, 2026-09-01
Source: @dino_38975, 2026-09-01
Source: @akkerstone, 2026-09-13 - Cross-check camp: chris.via.egnatia warns there is no silver bullet and vLLM numbers should be sanity-checked against a clean llama.cpp
q4_k_mrun; Eksitus and Merom Green keep llama.cpp for benchmarks because it is easier to compare across hardware; makesier runs vanilla llama.cpp layer split. Source: @chris.via.egnatia, 2026-07-21
Source: @meromgreen, 2026-07-22
Source: @eksitus, 2026-07-27
Source: @makesier, 2026-07-31 - Dissent: Ryko saw little improvement over llama.cpp Q8 for Qwen3.8 27B on one card, found single-request speeds comparable for 27B with llama.cpp plus MTP faster for single requests, and saw vLLM clearly better only on Qwen3.8 Flash Next (new architecture, weak llama.cpp code). He claims vLLM's reported prefill is inflated by chunked prefill; giaki3003 disputed that with his FreeToken against ik_llama benchmark page (measured on an RTX 3080 laptop). Source: @vektor_prime, 2026-08-14
Source: @vektor_prime, 2026-09-01
Source: @giaki3003, 2026-09-01 - Conditional: Vcore uses vLLM when VRAM is to spare and there are many users or subagents, and llama.cpp or ik_llama when the model barely fits (some models that fit run in llama.cpp but not in vLLM); DaLZy says llama.cpp gains on other engines the older the hardware; Miséricorde says "all default runtimes on CMP are slow" and tuned (often AI-generated) runtimes are needed; segmond says the point of llama.cpp is to have an option when VRAM is short, not to run fast; Phoenix17's rule: "If it + context fits in VRAM -> vLLM or SGLang, otherwise llama.cpp". Source: @vcore2191, 2026-08-19
Source: @dalzyyy, 2026-08-19
Source: @wakeupmbrc1, 2026-09-11
Source: @segmond, 2026-09-11
Source: @phoenixone7, 2026-09-13
Costs of vLLM: a startup of 2 minutes or more (clee) and about 5 minutes of compile on each launch (Dino). Plean asked, unanswered, whether vLLM still makes sense behind llama-swap given reports of higher time to first token, on two cards at Gen2 x4 without P2P. Adust and Vcore advised dropping ollama for raw llama.cpp, and DaLZy said even LM Studio beats ollama. bigfoot also took part. Source: @bigfoot1144, 2026-07-21
Source: @plean., 2026-07-30
Source: @adusts, 2026-08-19
Build flags
Build for compute capability 8.0 (GA100). The DeepSeek V4 Flash run under #Measured runs on the 170HX lists a complete working set of build options.
Force cuBLAS: about +500 tok/s prefill on Qwen3.8 27B
On 2026-08-14 Ryko ran Qwen3.8 27B Q8_K_XL on one card in llama.cpp at about 1.5k tok/s prompt processing and saw about 500 tok/s more prefill after rebuilding llama.cpp with the force-cuBLAS option ("there's a major speed up for this model if you rebuild llamaCPP with force_cublas, it was about an extra 500 tok/s in PP"); on 2026-08-19 he repeated the tip as GGML_CUDA_FORCE_CUBLAS. He also shared his vLLM docker run command for the same model on pastebin. Earlier, on 2026-08-13, Petri Krohn had linked a r/LocalLLaMA post titled "CMP 170HX 65GB llamacpp GGML_CUDA_FORCE_CUBLAS=ON gives me ~50% mo...", in which a third party running Qwen 3.6 27B across two 64 GB cards on poor PCIe links reported the gain; Petri questioned splitting a 27B model across two cards on weak links, and Humvee55 commented. The ~50% figure is the third party's claim and was not reproduced in the server.
Reported
Source: @petrikrohn_26497, 2026-08-13
Source: @scallion_9883, 2026-08-13
Source: @vektor_prime, 2026-08-14
Source: @vektor_prime, 2026-08-19
On the CMP 100-210 (sm_70), force MMQ instead
Josephur found GGML_CUDA_FORCE_MMQ=ON better on the CMP 100-210, a Volta card (sm_70) with no BF16 or TF32 tensor support. This is the opposite of the force-cuBLAS tip for the 170HX above; his full production configuration is under #Other CMP cards.
Reported
Source: @josephur, 2026-08-19
RPC build for mixed Volta and Ampere boxes
On 2026-08-08 Hastwell said he builds llama.cpp with -DGGML_CUDA=ON -DGGML_RPC=ON -DCMAKE_CUDA_ARCHITECTURES="70;86" to enable RPC and build only for Volta (CMP 100) and non-GA100 Ampere; a 170HX (GA100) needs 80 in the architecture list. On a 4 GB RAM Octominer he runs one llama.cpp instance (GLM 4.5) and reserves 256 MB for a rescue kernel after panics caused by unneeded PCI kernel flags; builds spill to swap.
Reported
Source: @hastwell, 2026-08-08
Check a build against the llama.cpp CUDA scoreboard: about 66% of an A100
quark2world ran the CUDA scoreboard benchmark from llama.cpp discussion #15013 on 2026-08-02 with TheBloke's llama-2-7b.Q4_0.gguf:
llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1
He got about 66% of the A100 result, roughly 3.5k pp512 (5285 x 0.66 = 3488). arnold had expected about 4.8k pp and 250 tg from SM count and bandwidth, quoting A100 80GB pp512 4849 / tg128 191 and RTX 3090 5174 / 158 from the same list. quark2world: gaming cards are "sportscars" in single-request benchmarks while datacenter cards shine with batching and long contexts; Capone: consumer cards have a different tensor-core to CUDA-core ratio that optimised quant kernels exploit; OpenCL-Benchmark (ProjectPhysX) was suggested for raw A100-like numbers. WinterIsAmateur (2026-08-04): llama.cpp does not use the 170HX's BF16 tensor cores and HBM nearly as well as vLLM W4A16 (citing others' vLLM results of about 54.5 tok/s decode and 1.4K to 3.0K prefill for Gemma 4 31B on one card), so llama.cpp numbers should not be used as the card's performance standard.
Reported 8 GB
Source: @quark2world, 2026-08-02
Source: @a.civardagezen, 2026-08-04
Source: @beefbonecapone, 2026-08-04
Source: @winter_is_amateur, 2026-08-04
Source: repo github.com/ggml-org/llama.cpp by ggml-org on GitHub
Splits and slots
Layer split and row split on two cards
Samuel Hayden posted llama-server settings for a 2x CMP 170HX pool of 128 GB (two 8 GB cards unlocked to 64 GB):
# layer split (the default)
llama-server -m <model.gguf> --split-mode layer --tensor-split 1,1 -ngl 999
# row split (TP-like)
llama-server -m <model.gguf> --split-mode row --tensor-split 1,1 -ngl 999
He advised running row against layer comparisons, but called row split benchmark-only on the Gen1 links of the time.
Reported 8 GB
Source: @samuel_hayden., 2026-07-22
Source: @jedilost, 2026-07-22
A second card in layer split: +61% prefill, +9% decode on Qwen 3.8 27B BF16
satspace's llama-bench of Qwen 3.8 27B BF16 (50.89 GiB), same build: one card pp512 1141.23 / tg128 24.72 tok/s; two cards with -sm layer -ts 50/50 pp512 1842.04 / tg128 26.97 tok/s. In llama-bench write -ts 50/50: 50,50 runs two separate benchmark values.
Reported 8 GB
Source: repo github.com/satspace-cpu/cmp170hx-linux-p2p by satspace-cpu on GitHub
Parallel slots on 4x 170HX: -np 6 beat -np 8 in aggregate
Danske (2026-08-07) found -np 6 (llama-server parallel slots) the sweet spot on 4x 170HX: 6 x about 13 tok/s, about 78 tok/s aggregate; -np 8 gave 8 x about 8.6 tok/s, about 69 tok/s aggregate. The model was not named (he was experimenting with Qwen and said Qwen Q8 seemed better).
Reported
Source: @danske1, 2026-08-07
RPC
llama.cpp RPC spreads one model across GPUs of different kinds or across machines. Whether PCIe 2.0 makes RDMA worth it is argued on Parallelism over the PCIe link.
RDMA in llama.cpp RPC across machines (idea, August 2026)
On 2026-08-16 segmond said he had meant to add RDMA to llama.cpp but found it had been added in March; with network cards arriving that week he hoped to run llama.cpp RPC across machines with minimal token-rate loss. He was already loading weights across the network (not over NFS). His results followed a week later (next entries).
Theory
Source: @segmond, 2026-08-16
RPC mixes RTX 3090s with 170HX cards
segmond runs his RTX 3090s together with his 170HX cards through llama.cpp RPC and says they mix well; ST added that RPC is better than people think. Rickcal's question assumed both cards are GA102, but the 170HX is GA100-based.
Reported
Source: @segmond, 2026-08-23
Source: @rrickk420, 2026-08-23
Source: @stcord, 2026-08-23
Two rigs over RPC: 324 / 25 over gigabit, 676 / 43 over ConnectX-4 RDMA
Unable to afford more 170HX cards, segmond linked systems with flashed 25GbE NICs over direct-attach copper, planning RDMA without TCP/IP overhead for multi-node inference (2026-08-20). With 12 cards (he had planned 16 before prices rose), llama.cpp RPC over gigabit Ethernet gave 324 tok/s prefill and 25 tok/s decode; RDMA over used Dell ConnectX-4 cards (about $30 each on eBay) and a $25 10 Gb direct-attach cable, with no switch, gave 676 prefill and 43 decode. The model was not stated. Dell's binary-only drivers target RHEL 7/8 and SLES 15 while he runs Ubuntu 22.04/24.x on kernel 6.8.0-138, so GPUDirect (GPU to GPU over the network, bypassing CPU and system RAM) did not work; the gains were solid without it, and he did not optimize. He does not expect TP to help on PCIe 2.0 x4, and SkirtChaser noted vLLM TP wants 4, 8 or 16 GPUs, so 12 is awkward. RDMA is for joining separate machines, not for 4 cards on one motherboard. The thread also linked the Linux kernel's Mellanox Kconfig; Merom Green and xanthe_ took part.
Reported
Source: @segmond, 2026-08-20
Source: @meromgreen, 2026-08-21
Source: @xanthefin, 2026-08-22
Source: @skirtchaser, 2026-08-22
Source: @segmond, 2026-08-24
Source: @segmond, 2026-09-01
MTP and speculative decoding
Depth, acceptance and engine support across all engines are on Speculative decoding on the 170HX.
llama.cpp MTP update: about 35 to 45 tok/s on one card
orrorin: a llama.cpp MTP update raised his speed from about 35 to about 45 tok/s on a simple essay prompt, using unsloth's UD_Q2_XL quant on one card. The model was not named in the message.
Reported
Source: @_orrorin, 2026-09-17
Adaptive MTP draft depth plus ngram-mod speculation
On 2026-08-18 roger_dammit pointed to llama.cpp PR 27210 (adaptive MTP draft depth, draft-mtp-adaptive; still open when the threads were read) and hoped vLLM would copy it. Ryko uses it with MTP min 3 and max 5 and says ngram-mod is "a huge boost for coding and independent from MTP":
--spec-type draft-mtp-adaptive,ngram-mod --spec-draft-n-min-adaptive 3 --spec-draft-n-max 5 \
--spec-ngram-mod-n-match 24 --spec-ngram-mod-n-max 64
Reported
Source: @roger_dammit, 2026-08-18
Source: @vektor_prime, 2026-08-18
DSpark speculative decoding: PR 25173
On 2026-07-28, during the GLM-5.2 multi-card discussion, ju5t5omedude linked llama.cpp PR 25173, "spec: add DSpark speculative decoding" by wjinxu, as possibly relevant to speeding up big models; it showed as merged when checked on 2026-10-31. No CMP results were posted in that thread. A later DSpark run on mainline llama.cpp (4x 10 GB cards, DeepSeek V4 Flash) is under #Measured runs on the 170HX.
Theory
Source: @ju5t5omedude, 2026-07-28
Measured runs on the 170HX
A summary of the llama.cpp numbers on this page; credits and conditions are in the entries. Every row is one person's reading with its own quant, link width and settings.
| model and quant | cards | layout | decode (tok/s) | prefill (tok/s) | posted by |
|---|---|---|---|---|---|
| DeepSeek V4 Flash 0731, Unsloth Q8 | 5 | layer split | 20-25 | not given | diabeetee |
| DeepSeek V4 Flash 0731 GGUF, untuned docker image | 3 | layer split, 65k context | about 23 | about 260 | aaron |
| DeepSeek V4 Flash MXFP4 (144.44 GiB), llama-bench | 4 (apparently) | layer split | tg128 28.32 | pp512 349.58 | Bart |
| DeepSeek V4 Flash 0731 UD-Q3_K_XL plus DSpark drafter | 4x 10 GB (40 GB each), Gen2 x4 | pipeline | 28.4 (33.5 with clocks locked at 1,410 MHz) | not given | cuddylac997 |
| DeepSeek V4 Flash IQ3_XXS | 2x 8 GB | layer split | screenshot only | possibly 581 (screenshot) | makesier |
| DeepSeek V4 Flash Q4_K_XL | 2x 170HX + 2x RTX 3090 + RTX 5090 | layer split, mixed GPUs | 20-40 | about 1,000 | Ryko |
| GLM-5.2 IQ2_XXS (Unsloth) | 4x 8 GB, Gen1 x4 | layer split, no MTP | about 15 | about 24 | 1a2b3c0967 |
| GLM 5.2 | 7 (rented) | layer split | not measured | 121 | DeeJay |
| GLM 5.2 Q4 | 10 in an Octominer, PCIe x1 | layer split | 12 | not given | segmond |
| GLM-5.2 Q4_K_XL (about 320 GB in VRAM) | 8x 8 GB, Gen1 x4, Proxmox VM | layer split | 13-14 (3 at 20 sessions) | not given | SirBAM-a-lot |
| Qwen3.8-Flash-Next UD-Q4_K_XL, unsloth PR fork | 2x 170HX + 2x RTX 3090 | layer (pipeline) split | 25-27 | 300-400 | Ryko, Luke Howard |
| Qwen3.8-Flash-Next | 2 | not given | 25 | 400 | fallengod9962 |
| Qwen3.8 Flash Next abliterated UD-Q4_K_XL | 2 | --tensor-split 1,1 |
35-45 | 55-360 | MrManny |
| Qwen 3.8 27B BF16, llama-bench | 1 / 2 | single / layer 50/50 | 24.72 / 26.97 | 1141.23 / 1842.04 | satspace |
| Qwen3.6 27B Q4_K_M / BF16, llama-bench | 1x 8 GB | single card | 36.80 / 24.59 (tg128) | 918 / 1,801 (pp512) | acedogblast |
| llama-2-7b Q4_0 (scoreboard) | 1 | single card | not given | about 3.5k (pp512) | quark2world |
DeepSeek V4 Flash 0731 on llama.cpp: 20 to 33 tok/s decode, 260 to 500 tok/s prefill
On release day of the Unsloth DeepSeek-V4-Flash-0731 GGUF (2026-07-31), diabeetee ran the Q8 quant with llama-server across 5 cards (Asus W790 Sage SE, Xeon W5-3433, 3 cards at x8) at 20 to 25 tok/s decode (23 on screen) and thought the weights were split unevenly; roger_dammit expected vLLM to be much faster. aaron (2026-08-01): about 23 +/- 2 tok/s and about 260 tok/s prompt processing with a layer split across 3 cards, 65k context, docker image llama.cpp:server-cuda, untuned. Bart (2026-08-05), llama-bench of "deepseek4 ?B MXFP4 MoE" (144.44 GiB, 284.33B params, CUDA, ngl -1), apparently on the 4x 170HX server he was benchmarking: pp512 349.58 +/- 2.13 and tg128 28.32 +/- 0.65 tok/s, matching DeeJay's recollection of 400 to 500 prefill and about 30 decode. Zarknoth (2026-08-10): 8x V100 alone gave 28 tok/s generation and adding CMP 170HX cards (count not stated) gave 32 tok/s, with prompt processing about 300 to 400 tok/s on the CMPs against about 150 to 189 on the V100s.
On 2026-08-11 cuddylac997 ran the Unsloth UD-Q3_K_XL dynamic GGUF (4 shards) with DSpark speculative decoding through a separate MXFP4 drafter GGUF on mainline llama.cpp (DSpark merged via PR #25784), pipeline parallel on 4x 10 GB cards (40 GB each, about 37.7 GiB usable, 160 GB pooled) at PCIe Gen2 x4, with -c 32768 -ctk q8_0 -ctv q8_0 --spec-type draft-dspark --spec-draft-n-max 3. Result: about 28.4 tok/s decode at 32K context, and 33.5 tok/s with clocks locked to 1,410 MHz. The build used CMAKE_CUDA_ARCHITECTURES=80, GGML_CUDA_FA=ON and GGML_CUDA_NCCL=ON, and the cards report compute_cap 8.0. DeeJay's INT4 repack (150.4 GiB) did not fit with room for KV. Josh and DeeJay recommended EXL3 or vLLM instead; vLLM with PP and DSpark reached about 3 to 4x this decode on the same cards.
Reported 8 GB and 10 GB
Source: @diabeetee, 2026-07-31
Source: @rubbra_f, 2026-07-31
Source: @roger_dammit, 2026-07-31
Source: @aaron0010, 2026-08-01
Source: @justbendev, 2026-08-05
Source: @deejay9983, 2026-08-05
Source: @mrmannyatx, 2026-08-10
Source: @zarknoth, 2026-08-10
Source: @cuddylac997, 2026-08-11
Source: @dalzyyy, 2026-08-11
Source: @_____josh_____, 2026-08-11
Source: @deejay9983, 2026-08-11
DeepSeek V4 Flash on only 2 cards: IQ3_XXS, Q2 and Q3 with layer split
makesier posted DeepSeek V4 Flash IQ3_XXS on two 64 GB cards with vanilla llama.cpp layer split (2026-07-31, numbers only in the screenshot) and said on 2026-08-01 that results on the older DSv4-Flash should carry over to 0731. fallengod asked whether 581 tok/s prefill (possibly from that screenshot) is good enough for agents; robbbbbb: it works but is far below hosted APIs. roger_dammit (2 cards) can run a Q2 quant but had not got DSpark working and hears quality is significantly worse; d3dx9 runs Q3 in llama.cpp without DSpark; roger_dammit avoids llama.cpp because vLLM is much faster; Phoenix17: 128 GB is not quite enough for a decent quant.
Reported 8 GB
Source: @makesier, 2026-07-31
Source: @fallengod9962, 2026-08-01
Source: @robbbbbb7973, 2026-08-01
Source: @phoenixone7, 2026-08-04
Source: @roger_dammit, 2026-08-05
Source: @.d3dx9, 2026-08-05
Mixed rig: DeepSeek V4 Flash Q4_K_XL on 2x 170HX, 2x RTX 3090 and an RTX 5090
Ryko runs 2x CMP 170HX, 2x RTX 3090 and 1x RTX 5090 together and reaches about 1k tok/s prefill and 20 to 40 tok/s decode on DeepSeek V4 Flash with a Q4_K_XL quant in llama.cpp (2026-08-14). The card variant was not stated. He runs llama.cpp because his GPUs are mixed.
Reported
Source: @vektor_prime, 2026-08-14
GLM-5.2 IQ2_XXS on 4 cards at Gen1 x4: about 15 tok/s decode, about 24 tok/s prefill
1a2b3c0967, replying to Rickcal's suggestion of an unsloth GLM 2-bit quant on 4x 64 GB cards, ran Unsloth GLM-5.2 IQ2_XXS on 4 cards unlocked from 8 to 64 GB in llama.cpp with layer split (not tensor), on an AliExpress x4x4x4x4 bifurcation board at PCIe Gen1 x4 with no MTP: about 15 tok/s decode, and Studebaker read about 24 tok/s prefill from the llama logs. diabeetee expected it to fly at Gen2 x16, while ju5t5omedude doubted more lanes or a faster PCIe generation would help layer split. Gen2 had not yet been tried.
Reported 8 GB
Source: @1a2b3c0967, 2026-07-24
Source: @rrickk420, 2026-07-24
Source: @1a2b3c0967, 2026-07-27
Source: @_studebaker_, 2026-07-27
Source: @diabeetee, 2026-07-27
Source: @ju5t5omedude, 2026-07-27
GLM 5.2 on 7 cards: prefill too slow (121 tok/s); 12 tok/s on a 10-card x1 Octominer
On a rented rig (2026-07-26), GLM 5.2 in llama.cpp across 7 cards gave 121 tok/s prefill, about 25 minutes of prompt processing for his workload, so DeeJay stopped before measuring decode; giaki3003 said "vLLM or nothing on these cards", while TacticalSquid said vLLM crashed every time for him on GLM's sparse attention. Impactic (2026-08-10) ran GLM 5.2 on llama.cpp across 7 cards with an IQ4_XS GGUF: it works, but prefill is terrible; vLLM (allover326/vllm-dsa-mtp-sm80, 4-bit) needed an 8th card, which he got on 2026-08-17. segmond (2026-08-16) has 24 GPUs including 12 CMPs (10 running in an Octominer with PCIe x1 links), the rest RTX 3090s and 3080s: llama.cpp runs GLM 5.2 Q4 at 12 tok/s and Kimi K3 Q1 at 3 tok/s (up from 0.8), while DeepSeek V4 Flash ran slower there than on his 3090 rigs. Against his 4x 3090 + 4x 3080 20 GB rig on EPYC, 4x 170HX gave about 1/3 the prefill and slightly faster decode, a comparison he called unfair since the 3090s sit on EPYC; he planned to move the cards to EPYC and vLLM, expecting 2x to 3x. robbbbbb7973 suggested Microsoft Tutel and Rickcal said a custom TRITON_MLA_SPARSE kernel is needed for sparse-attention models; Wully and Studebaker took part.
Reported
Source: @deejay9983, 2026-07-26
Source: @giaki3003, 2026-07-26
Source: @tacticalsquid, 2026-07-26
Source: @robbbbbb7973, 2026-07-26
Source: @rrickk420, 2026-07-26
Source: @_studebaker_, 2026-07-26
Source: @impactic, 2026-08-10
Source: @wully8888, 2026-08-10
Source: @segmond, 2026-08-16
Eight 8 GB cards in a Gigabyte G292-Z20 under Proxmox: GLM-5.2 Q4_K_XL at 13-14 tok/s
SirBAM-a-lot passed eight modded 8 GB cards through to a VM on Proxmox in a Gigabyte G292-Z20, and all eight unlocked without issue. He loaded GLM-5.2-Q4_K_XL entirely in VRAM (he wrote "320GB") with layer split and got 13-14 tok/s. With 20 concurrent sessions the rate dropped to 3 tok/s (per session or total is not stated). The links ran at Gen1 x4, and tensor parallel was a no-go at that width.
Reported 8 GB
Source: @sirbamalotx, 2026-07-21
Source: @impactic, 2026-07-21
Source: @deced_123, 2026-07-21
Qwen3.8-Flash-Next UD-Q4_K_XL on the unsloth PR fork: 300 to 400 prefill, 25 to 27 tok/s decode
Ryko (2026-08-26) tested the unsloth llama.cpp PR fork for Qwen3.8-Flash-Next with the UD Q4_K_XL quant on 2x CMP 170HX plus 2x RTX 3090 using pipeline (layer) parallelism, not tensor parallel: prompt processing about 300 to 400 tok/s and generation 25 to 27 tok/s. He suspected not all of the model was offloaded and expected big speedups once the new architecture (6B active) matures. Luke Howard got similar results.
Proven
Source: @vektor_prime, 2026-08-26
Source: @lukextra, 2026-08-26
llama-bench on one 8 GB card: Qwen3.6 27B Q4_K_M 918 / 36.8 against BF16 1,801 / 24.6
acedogblast (65,052 MiB, CC 8.0): Q4_K_M (16.67 GiB) pp512 918.26, pp1024 921.89, pp4096 911.61, tg128 36.80, tg512 36.31, tg1024 34.28 tok/s; BF16 (50.89 GiB) pp512 1,801.05, pp1024 1,803.38, pp4096 1,771.70, tg128 24.59, tg512 24.20, tg1024 22.83 tok/s. MTP does not work in llama-bench, so real use is about 20% faster (he saw about 40 tok/s with full BF16 in use); Q4 was slower than his RTX 3090 while BF16 beats a single 3090. Jeremy (Ends) blamed Q4 dequant overhead on this card and Merom Green suggested Q8 or vLLM; Josh attributed the "full BF16 faster than Q4" observation to quant unpacking overhead (BF16 prefill is about 2x Q4 while Q4 decode is higher). ju5t5omedude in LM Studio at Gen2 x4: Qwen 27B MTP Q4 gave 36 tok/s with PP and 46 with TP, and Q8 with TP gave 53 tok/s, faster than Q4 (card count not stated). The single-card picture for this model is on Qwen3.6 27B and 35B-A3B on one card (July and August 2026).
Reported 8 GB
Source: @acedogblast, 2026-07-27
Source: @endsjeremy, 2026-07-27
Source: @meromgreen, 2026-07-27
Source: @acedogblast, 2026-07-30
Source: @_____josh_____, 2026-07-30
Source: @ju5t5omedude, 2026-07-30
Locked cards
A stock card that has not been through the Driver unlock runs some operations at a throttled rate. A patch script reworks llama.cpp to avoid them; on an unlocked card at least one of its changes likely costs performance (untested).
eastmoe/CMPGPU-patch-script: llama.cpp source patches for stock (locked) CMP cards
ju5t5omedude (2026-07-27) passed this script to a CMP 90HX owner together with llama.cpp issue #24616 ("add dp4a emulation via dp2a on cuda", Arabella's patch), relaying that eastmoe's auto-patch script includes that patch, though the person quoted had not run it. optimize-cmp-cuda.py text-patches llama.cpp's ggml/src/ggml-cuda sources interactively, asking y/N per group of changes, with --dry-run, --restore and --no-backup options. The groups steer work away from what a stock card throttles: no fused multiply-add, fast approximate math functions, dp4a emulated with dp2a, and FP16/BF16 matrix multiplies moved from the tensor cores to the CUDA cores (cuBLAS default math, MMQ forced onto the dp4a path). The repo reports, on a stock (not unlocked) 170HX 8 GB card (CC 8.0) with a Gemma-4-12B IQ4_XS (6.16 GiB): pp512 355.59 to 730.84 tok/s (+105%) and tg128 34.54 to 63.34 tok/s (+83%). On a cmpunlocker-unlocked card, moving FP16/BF16 off the tensor cores is likely to cost performance (untested on Discord). The script matches exact strings and aborts rather than mis-patching when upstream llama.cpp text changes. The repo has no license; last commit 2026-06-23.
Reported 8 GB
Source: repo github.com/eastmoe/CMPGPU-patch-script by eastmoe on GitHub, posted by @ju5t5omedude, 2026-07-27
Other CMP cards
On the CMP 100-210 (Volta, sm_70) most llama.cpp work is Josephur's patch set; the dp4a patch used on the CMP 90HX is under #Locked cards. More reports are on Other CMP cards and other NVIDIA GPUs and CMP 90HX unlock hub.
Josephur's llama.cpp work for the CMP 100-210: research patches and llama.cmp-v
Josephur maintains Josephur/CMP100-210-llama.cpp-Research ("not a config, it's a collection of many llama.cpp patches and cuda code for these cards specifically") for the CMP 100-210 (6x 16 GB in his rig), and on 2026-09-04 posted device-side mask gains for Qwen3.8. He runs unsloth Qwen3.8-Flash-Next-GGUF UD-IQ4_XS for speed, targets single-request decoding, and notes testing is slow because these cards run PCIe Gen1 x1; he also patched llama.cpp so Qwen 3.8 Flash Next with Vision does not break when MTP/DFlash2 is on (it disables the DFlash2 part, the culprit). He announced on 2026-09-05 and published on 2026-09-08 llama.cmp-v, a llama.cpp build optimized for sm_70 CMP 100-210 cards that cannot be converted to V100s: at first DIY ("Everything is insanely documented"), with binaries to follow, and he asked for testers. He built it for an old 6-card mining box and optimizes for one user (parallel 1), conceding vLLM is likely better for multiple users, and said he might retarget it if a full software unlock lands. Not about the 170HX, but related CMP work.
Reported
Source: repo github.com/Josephur/CMP100-210-llama.cpp-Research by Josephur on GitHub, posted by @josephur, 2026-09-04
Source: @matstatman, 2026-09-04
Source: @machinezer0, 2026-09-04
Source: @josephur, 2026-09-04
Custom llama.cpp on 6x CMP 100-210: Qwen3.5-122B at 100K context, prefill nearly doubled, MTP rejected
Josephur (2026-08-07) runs a patched llama.cpp on six CMP 100-210 cards (Volta, PCIe x1) serving Qwen3.5-122B with layer splitting at 102,400-token capacity: fresh-prompt time to first token at 100K fell from 506.0 s to 260.7 s and prompt processing rose from 198.5 to 387.1 tok/s, while decode stayed at 27.0 against 26.95 tok/s (the pinned-staging change targets prefill only). Production configuration: forced DP4A MMQ, F16 KV with automatic FlashAttention split-K, MTP disabled, backend sampling with sampler reuse, recurrent-state/SSM fusion with tuned dense Q8 geometry, destination-local mapped-host transfers with FP16 bridge compression, a depth-four diagonal prefill pipeline, batch/ubatch 24576/512, 8 GiB prompt-cache RAM, and preallocated pinned snapshots with one private slot per backend. In separate matched exact-100K runs, SSM fusion plus dense Q8 geometry raised decode from 33.565 to 36.317 tok/s (+8.20%, byte-exact output) and backend sampling from 28.83 to 34.35 tok/s (+19.2%). The best MTP setting at 100K raised decode from 27.795 to 30.147 tok/s but added 220.7 s of time to first token, needing about 78,600 generated tokens to recover, so MTP stays off; replacedriver47 independently found MTP kills usable context on dual 32 GB V100s. He planned to publish the work on GitHub (see the entry above). Not a 170HX recipe, but the same CMP family.
Reported
Source: @replacedriver47, 2026-08-07
Source: @josephur, 2026-08-07
Related
Engines · vLLM on sm_80: known bugs and patches · Speculative decoding on the 170HX · Parallelism over the PCIe link · Quantization on the 170HX · How the 170HX compares · Benchmarks · Qwen3.6 27B and 35B-A3B on one card (July and August 2026) · Compute, data types and missing features · Other CMP cards and other NVIDIA GPUs · CMP 90HX unlock hub · Gigabyte G292-Z20 · Octominer X12 Ultra builds · PCIe x16 capacitor mod · Driver unlock · Recipes