GLM 5.2 on 3x 10 GB (llama.cpp CPU offload)
From CMP Wiki
More actions
| Reported 3 x 10 GB | |
| Quant | Q4 GGUF (the exact Q4 file is not named); UD-Q2_K_XL also tested |
|---|---|
| Engine | llama-server with partial CPU offload (llama.cpp or ik_llama.cpp: the build is not named) |
| Topology | partial offload: part of the model on the three cards, the rest in host RAM |
| Links | PCIe Gen2 x4 |
| Decode | 6.39 tok/s |
| Prefill | 48.22 tok/s at 2048 tokens |
| Measured as | pp2048 and tg512 (benchmark tool and number of users not stated) |
| VRAM per card | 40 GB profile (10 GB card unlocked to 40 GB) |
| Unlock build | 10 GB to 40 GB unlock |
| Tested | @eksitus on 2026-07-24 |
| Source | Discord message |
| Confirmations | None yet. |
GLM-5.2 at Q4 on three 10 GB cards at the 40 GB profile, with most of the model left in host RAM and DDR4. This is a
slow way to run a model this size: the cards add decode speed over the CPU alone, and the PCIe link limits prefill.
Eksitus says 64 GB cards suit offloading a model of this size much better.
Source: @eksitus, 2026-07-20
Source: @eksitus, 2026-07-24
Notes
- Host: Supermicro H12DSi-N6 with two Zen 3 EPYC 7713 CPUs. The model is about 460 GB of weights on three 40 GB cards. Source: @eksitus, 2026-07-24
Source: @eksitus, 2026-07-20 - At PCIe Gen1 x4 (stock link, 2026-07-20): pp2048 33.44 tok/s and tg512 5.90 tok/s, with llama-server holding 26,684, 36,274 and 37,270 MiB on the three cards. Against CPU and DDR4 alone (about 45 prefill and 3.5 decode) decode went up about 60% and prefill down about 30%, which he put down to the Gen1 x4 link. Source: @eksitus, 2026-07-20
Source: @eksitus, 2026-07-20 - At PCIe Gen2 x4 on the same three cards (2026-07-24): the Q4 model gave pp2048 48.22 and tg512 6.39 tok/s (the infobox numbers); unsloth's GLM-5.2-UD-Q2_K_XL gave 49.00 and 6.81. Source: @eksitus, 2026-07-24
Source: @eksitus, 2026-07-24 - Without the link and memory mods, splitting bigger models across several cards gave him poor prefill. Source: @eksitus, 2026-07-20
- Not stated: the llama.cpp build (he used both llama.cpp and ik_llama.cpp at the time), the exact Q4 file, the context length, the offload flags, the power cap and the number of concurrent requests.
- A longer write-up by @eksitus exists; see Discord.
Sources
- Eksitus's posts in the testing channel and in general, 2026-07-20 to 07-24.
- Related: Recipes#10 GB card, llama.cpp on CMP cards, 10 GB card, Memory unlock on the 10 GB card, Benchmarks.