GLM 5.2 on 8x 8 GB
From CMP Wiki
More actions
| Reported 8 x 8 GB | |
| Quant | 4-bit, posted as int4 and q4 (the checkpoint is named in Notes) |
|---|---|
| Engine | vLLM with a sparse-attention (DSA) backend for sm_80 |
| Topology | PP8 (pipeline parallel); tensor parallel ruled out at Gen1 x4 |
| Links | PCIe Gen1 x4 |
| Decode, 1 user | 30 tok/s |
| Prefill | 2600 tok/s |
| Speculative decoding | none |
| Measured as | rounded figures from the tester's chat posts; one session on rented cards |
| VRAM per card | 64 GB (8 GB card unlocked) |
| Driver | 610.43.02 |
| Flags | --pipeline-parallel-size 8 --gpu-memory-utilization 0.90 |
| Tested | @deejay9983 on 2026-07-27 |
| Source | Discord message |
| Confirmations | None yet. |
In late July 2026 DeeJay rented a server of RedPandaMining's unlocked 8 GB cards (64 GB each) for most of a day to see how GLM-5.2 runs on the CMP 170HX. All links ran at PCIe Gen1 x4: the run came before the rig's Gen2 update. Con3po's notes list eight rented cards on 2026-07-27 at driver 610.43.02, Gen1 x4, with no capacitor mod. On vLLM with pipeline parallel, GLM-5.2 at 4 bits gave about 2,600 tok/s prefill and 30 tok/s decode, without MTP. His results file went up on 2026-07-27.
Source: @deejay9983, 2026-07-26
Source: @deejay9983, 2026-07-26
Source: @deejay9983, 2026-07-27
Source: @deejay9983, 2026-07-27
Source: @redpandamining, 2026-07-27
Source: @deejay9983, 2026-07-27
Source: @deejay9983, 2026-07-27
Source: @con3po7016, 2026-09-09
Prerequisites
- Eight 8 GB cards unlocked to 64 GB (Driver unlock, 8 GB card).
- A vLLM build with a sparse-attention path for sm_80. GLM-5.2's DSA sparse attention otherwise wants Hopper or Blackwell, which was the first obstacle on this rig. Source: @deejay9983, 2026-07-26 See Engines.
- A symmetric 4-bit checkpoint that vLLM's MoE kernels accept (see Notes and Quantization on the 170HX).
Measured numbers
- vLLM, pipeline parallel, 4-bit: about 2,600 tok/s prefill and 30 tok/s decode, no MTP. Source: @deejay9983, 2026-07-27
Source: @deejay9983, 2026-07-27
Source: @deejay9983, 2026-07-29 - llama.cpp on the same rig: about 121 tok/s prefill. He stopped the run before it reached decode, since a long prompt would take about 25 minutes to process. Source: @deejay9983, 2026-07-26
Source: @deejay9983, 2026-07-26 - Tensor parallel was out of the question at Gen1 x4. He put the minimum at Gen2 x16 or Gen3 x4. Source: @deejay9983, 2026-07-27
Source: @deejay9983, 2026-07-27
Notes
- Reported Exact figures and stack, from the results file. According to the Consensus-Protocol research wiki, which its author calls unverified, the results file attached to DeeJay's post gives: prefill 665, 1,497, 2,342 and 2,675 tok/s at 4k, 32k, 65k and 131k context (it rises with context), decode 30.2 tok/s, a KV cache of 438,107 tokens at 0.92 memory utilisation, 440 to 620 s to load the model, and no hard faults in the session. The same wiki gives the stack as the vLLM 0.20.2 release wheel with the Python files of vLLM PR #38476 (the TRITON_MLA_SPARSE backend) applied onto site-packages; transformers 5.x;
VLLM_ATTENTION_BACKEND=TRITON_MLA_SPARSEandVLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0; block size 64; and the symmetric W4A16 quant lowbitcoffee/GLM-5.2-W4A16. cmp.wiki does not hold the file, so these values are not checked against it. Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub - For later builds of the same model (MTP under pipeline parallel, Gen2 links), see GLM 5.3 on 8x 8 GB (vllm-170hx-glm5), whose history table lists this run beside them.
Known issues
- Reported One card crashed during loading with the memory utilisation set to 0.95. At 0.90 it came back. Source: @deejay9983, 2026-07-26
- Reported Killing a stuck llama.cpp run left about 32 zombie processes and broke the driver until the host was restarted. DeeJay puts most of the instability on this rig down to his AI agent's test procedure (wrong settings, processes started before others had closed). Source: @deejay9983, 2026-07-26
Source: @deejay9983, 2026-07-26
Source: @deejay9983, 2026-07-27 - MTP did not work with pipeline parallel in vLLM at the time. Two days later DeeJay packaged a fix that combines two then-unmerged vLLM pull requests (#46994 for MTP under pipeline parallel, #38476 for sparse MLA on sm_80); it is described on GLM 5.3 on 8x 8 GB (vllm-170hx-glm5). Source: @deejay9983, 2026-07-27
Source: @deejay9983, 2026-07-29
Sources
- DeeJay's posts in testing-general, general and development-general, 2026-07-26 to 2026-07-29, and the results file (170HX-benchmark-results.md) he attached on 2026-07-27.
- RedPandaMining's post on the rented cards, 2026-07-27.
- Related: Recipes, Benchmarks, Engines, Quantization on the 170HX.