Running GLM 5.3 locally: hardware and measured speeds
From CMP Wiki
More actions
Short answer. The full GLM-5.3 (753B total parameters, 744B without the MTP draft layer; not the Flash model) needs eight 8 GB cards unlocked to 64 GB, 512 GB of VRAM in all. Measured single-stream decode on eight cards runs from about 25 tok/s (plain pipeline parallel with MTP) through 48.7 tok/s (pipeline parallel with a DFlash2 drafter, the author's run on 10 cards) to 68.3 tok/s (TP2 x PP4 with P2P, MTP and PipeSpec), with a context of 524,288 to 1,048,576 tokens. The lab's two-host layout, which adds two RTX PRO 6000 cards in front of the eight CMPs, measures 110.5 tok/s. Every row below links to the recipe page with the exact commands.
If you have fewer than eight cards, run the smaller Flash model: Running GLM 5.3 Flash locally: hardware and measured speeds.
Hardware it needs
- Cards. Eight CMP 170HX at 64 GiB each (Driver unlock). The pipeline-parallel recipe also has nine and ten-card layouts.
- Links. The pipeline-parallel DFlash2 recipe was measured on x4 links. The fastest eight-card recipe (TP2 x PP4) needs Gen2 x16 links and a P2P driver build, with each tensor-parallel pair behind one PCIe switch; without P2P its author measured 58 / 59 / 53 tok/s at 128K context (coding / browsing / prose).
- Disk. For the TP2 x PP4 recipe: 433 GiB for the source checkpoint plus about as much again for the converted one.
- Power. The same recipe peaks at about 2.3 kW at the wall with the cards capped at 160 W (mean 1,784 W under eight streams); one PSU circuit was not enough for its author.
Source: repo github.com/cebeuq/glm53-cmp170hx-recipe by cebeuq on GitHub
Source: repo github.com/arvkmr/vllm-170hx-glm5 by arvkmr on GitHub, posted by @reala7096, 2026-10-08
- Smaller quants on another engine (not measured on CMP cards). Project Maya runs the full model from unsloth's UD-IQ2_M GGUF (238.6 GB; about 256 GB of VRAM and RAM together keeps every expert in memory; v1.1.57, 2026-11-09) and, since v1.1.58 (2026-11-10), from Maya's own GLM-5.3-Maya-XS (170.0 GB; about 176 GB of VRAM and RAM together, 192 GB recommended), on NVIDIA from the V100 on. With less memory it reads the rest from the SSD while it answers. See Engines#Strata and Project Maya. Reported Source: repo github.com/mw00/project-maya by mw00 on GitHub
Measured configurations
One row per recipe page. "One user" is the recipe's headline single-stream decode figure; "Many users" is the total across concurrent requests. Each author used a different benchmark (noted in the cell where it is known), so the rows are not directly comparable. The status badge is the one each recipe page carries: Proven means a second person reproduced it.
| Cards | Recipe | Weights | Layout | One user (tok/s) | Many users (tok/s) | Prefill (tok/s) | Context | Power cap | Status | Measured by |
|---|---|---|---|---|---|---|---|---|---|---|
| 8 | GLM 5.3 on 8x 8 GB (cebeuq glm53-cmp170hx-recipe) | hybrid: NVFP4 routed experts plus an INT8 residue (about 420 GB), from the abliterated NVFP4 checkpoint | TP2 x PP4 on x16 links, P2P inside each pair; MTP plus PipeSpec | 68.3 (author's chat bench, thinking on) | 135.5 at 8 users | 524,288 | 160 W | Reported | cebeuq | |
| 8 to 10 | GLM 5.3 on 8x 8 GB (vllm-170hx-glm5) | Int4-Int8Mix AWQ | PP8, PP9 or PP10 on x4 links; DFlash2 drafter | 48.7 (author's 10-card run) | 174.1 at 8 users | 1,833 (about 7.8K tokens) | 1,048,576 | 170 W | Proven | reala7096 |
| 8 | GLM 5.3 on 8x 8 GB (Int4-Int8Mix PP8) | Int4-Int8Mix | PP8; MTP, 5 draft tokens | 25 | 1,048,576 | 200 W | Reported | 1a2b3c0967 | ||
| 8 plus two RTX PRO 6000 | GLM 5.3 on 8x 8 GB plus 2x RTX PRO 6000 (vLLM TP2 x PP5) | the hybrid above, with the first stage at NVFP4 | TP2 x PP5 over two hosts on 100 GbE; MTP plus PipeSpec | 110.5 (lab bench, 1,024-token answers) | 241.3 at 8 users | 3,223 (17k tokens, cold, no prefix cache) | 524,288 | 240 to 300 W (CMPs) | Reported | darkmaniac7 |
Sources for the rows (each recipe page carries its full credits):
Source: repo github.com/cebeuq/glm53-cmp170hx-recipe by cebeuq on GitHub
Source: repo github.com/arvkmr/vllm-170hx-glm5 by arvkmr on GitHub, posted by @reala7096, 2026-10-08
Source: @1a2b3c0967, 2026-10-07
Source: @darkmaniac7, 2026-11-11 (lab notes)
What the numbers mean
- TP2 x PP4 (cebeuq). One user on his chat bench (36 prompts, temperature 0, thinking on, 384 tokens): 68.3-68.6 / 72.8-74.9 / 59.9-60.0 tok/s on coding, web-browsing agent tasks and prose. Eight users: 135.5 tok/s aggregate (18-22 per stream). Time to first token on a 16K-token prompt: 8.2 s. Context 524,288 with a KV pool of 569,792 tokens; the longest prompt served was 519,451 tokens. The same machine before this work, with no speculation, did 34.4 tok/s. Source: repo github.com/cebeuq/glm53-cmp170hx-recipe by cebeuq on GitHub
- The same recipe in the lab. On the eight CMPs alone, with the lab's additions, it does 90-95 / 93 / 61 tok/s single stream on the lab's bench (structured / code / prose). The recipe page records this as a partial reproduction, because the build and the bench differ from the author's. Source: @darkmaniac7, 2026-11-05 (lab notes)
- Pipeline parallel with DFlash2 (vllm-170hx-glm5). The author's run on 10 cards at 170 W: 48.7 tok/s for one stream, 32.6 per stream at four, 27.1 per stream at eight (174.1 aggregate), 1,833 tok/s prefill on a 7.8K prompt. His summary: 45 tok/s on short context and never below 30 on long context. A second owner ran it on 8 cards and called it a big performance bump over his own build. Source: repo github.com/arvkmr/vllm-170hx-glm5 by arvkmr on GitHub, posted by @reala7096, 2026-10-08
Source: @reala7096, 2026-10-08
Source: @1a2b3c0967, 2026-10-08 - Plain PP8 with MTP. About 25 tok/s, slowing to about 10 tok/s at high context, at 200 W per card. Source: @1a2b3c0967, 2026-10-07
- Eight CMPs plus two RTX PRO 6000 over two hosts. One stream at temperature 0 with 1,024-token answers: 108.5 / 110.5 / 74.6 tok/s (structured / code / prose). Eight users on code: 241.3 tok/s aggregate. Cold prefill at 17k tokens: 3,223 tok/s. KV pool 873,536 tokens. Against the eight CMPs alone, the extra stage gives about 10% more single-stream speed, 43% more KV and about 50% faster prefill at 17k tokens. Source: @darkmaniac7, 2026-11-11 (lab notes)
Source: @darkmaniac7, 2026-11-05 (lab notes) - One RTX PRO 6000 plus four CMPs (ExLlamaV3). A member ran the full model in one box with a 3-bit EXL3 quant (about 273 GB on disk): about 2,000 tok/s prefill and about 32 tok/s decode for a short reply, dropping into the mid-20s around 100k of context. It was not usable enough for him. It has no recipe page; the details are on GLM 5.3 on 8x 8 GB plus 2x RTX PRO 6000 (vLLM TP2 x PP5). Reported Source: @mountains1605, 2026-11-06
Known problems
- Send repetition_penalty 1.0 with the abliterated checkpoint. The checkpoint the TP2 x PP4 recipes convert sets repetition_penalty 1.1 in its generation config, and vLLM applies it to every request that does not set its own. At temperature 1.0 long free-form output then falls apart about 2,000 to 12,000 tokens in. In the lab's runs a long agent request broke in 4 of 6 runs at 1.1 and in 0 of 12 at 1.0. Reported Source: @darkmaniac7, 2026-11-10 (lab notes)
Source: @darkmaniac7, 2026-11-11 (lab notes) - Slow at long context without a good drafter. The plain PP8 build falls to about 10 tok/s at high context. Source: @1a2b3c0967, 2026-10-07
- No vision. The full GLM 5.3 lacks vision input; one owner pushed it to 490k context but does not use it for that reason. Source: @con3po7016, 2026-10-08
- TP2 x PP4 limits. Measured on one server. P2P relies on a modified NVIDIA kernel module. Greedy output is not reproducible run to run on this engine. Startup takes many minutes. PipeSpec runs only while exactly one request is decoding. Source: repo github.com/cebeuq/glm53-cmp170hx-recipe by cebeuq on GitHub
- Early pipeline-parallel kit. Warning Checkouts of bayley's original kit from before 2026-08-17 had a numerics bug in the MTP plus pipeline parallel plus concurrency path that put small typos in the output. Update any older checkout. Source: @bayley_w, 2026-08-14
- Verbose. One member found the full model stable past 300K tokens with no loops, but it used 310K tokens on a job. Source: @bayley_w, 2026-09-20
More: Known bad combinations, Instability under load, vLLM on sm_80: known bugs and patches.
Common questions
How much VRAM does the full GLM 5.3 need?
Every measured CMP 170HX configuration uses eight 8 GB cards unlocked to 64 GB each, 512 GB in all, with 4-bit or mixed 4 and 8-bit weights (about 420 GB for the fastest recipe's converted checkpoint). See #Hardware it needs.
How many CMP 170HX cards do I need for GLM 5.3?
Eight. The pipeline-parallel recipe also runs on nine or ten. With four cards, run GLM 5.3 Flash instead: Running GLM 5.3 Flash locally: hardware and measured speeds.
How fast is GLM 5.3 on eight CMP 170HX cards?
Between about 25 and 68 tok/s for one user, depending on the recipe: about 25 tok/s on plain pipeline parallel with MTP, 48.7 tok/s with a DFlash2 drafter (the author's 10-card run), and 68.3 tok/s on coding with TP2 x PP4, P2P, MTP and PipeSpec (the author's chat bench, thinking on). See #Measured configurations.
What context length does GLM 5.3 get on eight cards?
524,288 tokens per request on the TP2 x PP4 recipe (KV pool 569,792 tokens) and 1,048,576 on the pipeline-parallel recipes.
Does the full GLM 5.3 need P2P or the x16 mod?
Only the fastest recipe does: TP2 x PP4 needs Gen2 x16 links and a P2P driver build. The pipeline-parallel DFlash2 recipe was measured on x4 links.
How much power does an eight-card GLM 5.3 server draw?
On the TP2 x PP4 recipe with the cards capped at 160 W, the author measured a mean of 1,784 W and a peak of 2,236 W at the wall under eight streams.
Can other GPUs help the CMP cards on GLM 5.3?
Yes, in one measured layout: two RTX PRO 6000 cards as the first pipeline stage on a second host, linked by 100 GbE, raised single-stream decode to 110.5 tok/s and the KV pool to 873,536 tokens. See GLM 5.3 on 8x 8 GB plus 2x RTX PRO 6000 (vLLM TP2 x PP5).
Related
- Recipe pages for this model are in the table above; Recipe finder filters all recipes by cards, model and engine.
- The Flash model: Running GLM 5.3 Flash locally: hardware and measured speeds. The previous generation: GLM 5.2 on 8x 8 GB.
- Other model pages: Running DeepSeek V4.1 Flash locally: hardware and measured speeds, Running Qwen3.8 Flash Next locally: hardware and measured speeds.
- Large rigs (8 or more cards), Running one model across two hosts (100 GbE, RDMA, vLLM and SGLang), Recipes, Benchmarks, Engines, Speculative decoding on the 170HX, Parallelism over the PCIe link.