GLM 5.3 on 8x 8 GB (cebeuq glm53-cmp170hx-recipe)

GLM-5.3 (the full 78-layer model, not Flash): 753B total parameters (744B without the MTP draft layer)

Reported 8 x 8 GB
Quant	hybrid: the published NVFP4 routed experts unchanged plus an INT8-g128 residue, converted offline from dealignai/GLM-5.3-ABLITERATED-NVFP4 (432.9 GiB source)
Engine	lazymio/vllm-backport sm_80 image (wtdcode/vllm-backport), pinned by digest, with whole-file vLLM patches mounted read-only vLLM g29e66dad4 in the pinned lazymio/vllm-backport image
Topology	TP2 x PP4, layers 22,20,20,16, the MTP layer on the last stage; each TP2 pair behind one PCIe switch
Links	PCIe Gen2 x16, P2P needed
Context	524,288 per request, 8 sequences, KV pool 569,792 tokens
Decode, 1 user	68.3 tok/s
Decode, many users	135.5 tok/s total, 8 users
Speculative decoding	MTP k=2 (the model's own MTP layer, stock head) plus PipeSpec
Measured as	author's chat bench: 36 prompts, temperature 0, thinking on, 384 tokens, mean streaming tok/s, coding row (68.3-68.6) at 358K context / 4 sequences; aggregate = 8-stream raw-prose harness at 524K
VRAM per card	64 GB (65,536 MiB), GPU memory utilization 0.97
Power cap	160 W per card, SM clock up to 1,695 MHz; about 2.3 kW peak at the wall
Driver	NVIDIA open 615.71.09 with the cmpunlocker patch set plus Morrowmake's TRAP31 P2P patch
Unlock build	cmpunlocker 2d10f891 with Morrowmake/cmpunlocker 1f2c498f p2p-unlock.patch, ForceP2P=0x11
Flags	TRITON_MLA_SPARSE attention; KV_DTYPE=fp8_ds_mla with CMP_KV_DSMLA_INT8=1; MTP num_speculative_tokens 2; CMP_PIPESPEC=1; custom all-reduce over P2P (CMP_PCIE_CUSTOM_AR=1); NCCL_P2P_LEVEL=SYS; max-num-seqs 8; FULL_AND_PIECEWISE CUDA graphs
Tested	@cebeuq on 2026-10-10
Source	link
Confirmations	Partly reproduced by @darkmaniac7 (8 GB), 2026-11-04 · details Confirm on your own card

cebeuq's recipe serves the full GLM-5.3 with a 524,288-token context on eight unlocked 8 GB cards at TP2 x PP4. The model has 753B total parameters (744B without the MTP draft layer). The recipe runs lazymio's sm_80 vLLM image unmodified underneath a set of mounted patches, converts the abliterated NVFP4 checkpoint offline, and adds PCIe peer-to-peer with vLLM's custom all-reduce, an int8-tile latent KV cache, MTP speculative decoding with a fix for pipeline parallelism, and PipeSpec. The drafter is the model's own MTP layer, shipped inside the public checkpoint, so there is nothing separate to download. It builds on Morrowmake's GLM-5.3-Flash recipe (GLM 5.3 Flash on 4x 8 GB) and on Morrowmake's TRAP31 P2P driver patch. The recipe is MIT licensed; the patches keep vLLM's Apache-2.0 headers.

Reported 8 GB
Source: repo github.com/cebeuq/glm53-cmp170hx-recipe by cebeuq on GitHub

Prerequisites

• Eight CMP 170HX at 64 GB each (nvidia-smi shows 65,536 MiB), unlocked with the cmpunlocker driver patch set (Driver unlock), on PCIe Gen2 x16 links in one server.
• Linux with the cmpunlocker-patched NVIDIA open driver (he used 615.71.09), Docker and the NVIDIA Container Toolkit.
• For production speed, the TRAP31 P2P driver build (#P2P driver). Without it, his no-P2P variant runs at 128K context: 58 / 59 / 53 tok/s on his chat bench (coding / browsing / prose).
• Disk: 433 GiB for the source checkpoint plus about as much again for the converted one, and room for the intermediate step while converting.
• Power: his stack peaks at about 2.3 kW at the wall with the cards capped at 160 W; one PSU circuit was not enough for him.
• His server: Gigabyte G481-HA0 with two Xeon Gold 6150, GPUs 0-3 behind one PLX PEX8796 switch on CPU socket 0 and GPUs 4-7 behind another on socket 1, so each TP2 pair shares a switch.

Source: repo github.com/cebeuq/glm53-cmp170hx-recipe by cebeuq on GitHub

Steps

His README's quick start, summarized:

1. Download dealignai/GLM-5.3-ABLITERATED-NVFP4 from Hugging Face and convert it with the recipe's requant container (CPU only, two runs: requant, then hybrid). The result keeps the NVFP4 expert bytes and stores the rest of the non-expert weights as INT8-g128.
2. For production speed, build and install the P2P driver (#P2P driver), cold boot, and run the byte-exact probe.
3. Copy engine/prod.vars.example to engine/prod.vars and set MODEL_DIR to the converted checkpoint.
4. Start it from engine/ with docker compose --env-file prod.vars up -d. Loading takes many minutes; the compose health check allows 45.
5. Use any OpenAI-compatible client on port 8000 of the local machine, path /v1, model glm-5.3. Tool calls and reasoning are parsed (glm47 / glm45). The API has no key and listens on the local machine only.

git clone https://github.com/cebeuq/glm53-cmp170hx-recipe.git && cd glm53-cmp170hx-recipe
hf download dealignai/GLM-5.3-ABLITERATED-NVFP4 --local-dir /data/models/glm53-nvfp4
SRC=/data/models/glm53-nvfp4 RQ=/data/models/glm53-step1 docker compose -f requant/compose.yaml run --rm requant
SRC=/data/models/glm53-nvfp4 RQ=/data/models/glm53-step1 DST=/data/models/glm53-hybrid docker compose -f requant/compose.yaml run --rm hybrid
cp engine/prod.vars.example engine/prod.vars   # set MODEL_DIR=/data/models/glm53-hybrid
cd engine && docker compose --env-file prod.vars up -d

Other measured configurations are in the repo's engine/variants/. Source: repo github.com/cebeuq/glm53-cmp170hx-recipe by cebeuq on GitHub

P2P driver

The CMP 170HX firmware reports no peer-to-peer support, so every collective goes through host memory. Morrowmake's TRAP31 change to the cmpunlocker driver patch set opens peer access. cebeuq merged it onto his installed tree (cmpunlocker 2d10f891 with the open modules 615.71.09: kept the ECC patches, appended p2p-unlock.patch last, left out bar1-resize-unlock.patch, which hung his board at boot) and builds the modules in Docker without touching the host. His p2p/ scripts back up the installed modules, build, install (writing ForceP2P=0x11) and roll back.

• It replaces NVIDIA kernel modules: make the backup first and have a remote console or physical access in case the server does not boot.
• Cold boot after any driver change. A warm reboot can leave the unlocked memory falling back to 8 GB; check for 65,536 MiB on every card.
• ForceP2P advertises every pair, so run his byte-exact probe before NCCL or vLLM uses P2P, and set VLLM_SKIP_P2P_CHECK=0.
• On his board: P2P write 5.79 GB/s on every pair, latency 2.1-2.6 us, and the 12 KB TP2 NCCL all-reduce went from 22.8 to 11.4 us.

More on the patch itself: P2P. Source: repo github.com/cebeuq/glm53-cmp170hx-recipe by cebeuq on GitHub
Source: repo github.com/Morrowmake/cmpunlocker by Morrowmake on GitHub

Measured numbers

From his repo: one server, eight CMP 170HX at 64 GB, PCIe Gen2 x16, 160 W per card with the SM clock up to 1,695 MHz, P2P on, measured 2026-10-08 to 2026-10-10. Chat bench: 36 prompts (16 coding, 16 web-browsing agent tasks, 4 prose), one at a time, temperature 0, thinking on, 384 tokens, mean streaming tok/s per category. Harness: a raw-prose completion, 256 tokens, median of 3, plus concurrent streams.

What	Result
One user, chat bench (358K context, 4 sequences): coding / web-browsing agent tasks / prose	68.3-68.6 / 72.8-74.9 / 59.9-60.0 tok/s
8 users, raw-prose harness, aggregate (524K context, 8 sequences)	135.5 tok/s (18-22 per stream)
One user, raw-prose harness: 524K / 358K configuration	52.1 / 54.2 tok/s
Time to first token, 16K-token prompt	8.2 s
Context / KV pool	524,288 / 569,792 tokens
Longest prompt served	519,451 tokens, answered coherently in 415 s
GSM8K-250, 5-shot (358K configuration)	84.8%, 82.8%, 87.6% (three runs)
Wall power, mean / peak (8-stream harness)	1,784 / 2,236 W
No-P2P variant, 128K context, chat bench: coding / browsing / prose	58 / 59 / 53 tok/s
The same machine before this work (no speculation, raw-prose harness)	34.4 tok/s

• The 524K / 8-sequence production configuration was not measured on the chat bench. On the raw-prose harness it was 4% lower than the 358K configuration, so expect chat numbers a few percent below the first row.
• His chat numbers vary by about 2-3% between repeats, the harness by up to 10%.

Reported 8 GB
Source: repo github.com/cebeuq/glm53-cmp170hx-recipe by cebeuq on GitHub

What each piece buys

As his docs state it, each measured on his eight cards:

• MTP k=2 with a fix for pipeline parallelism. Under PP the fork's MTP drafter ran with an unloaded embedding table (GLM-5.x keeps embed_tokens only at the top level), so drafts were accepted about 22% of the time. An 11-line fix in v32_mtp.py loads the table: acceptance 86% at k=1, and with k=2 the machine went from 34.4 to about 56-59 tok/s on chat. Under PP, an acceptance near 20% in /metrics is this bug. k=3 was no faster than k=2 for one stream and slower at 4 streams.
• PCIe P2P and vLLM's custom all-reduce. NCCL over P2P alone gave +4-7% on chat; with vLLM's one-kernel custom all-reduce, +12-16%. Keep custom all-reduce inside one PCIe switch: TP8 with custom all-reduce across the two CPU sockets ran at 15.1 tok/s.
• int8-tile latent KV cache. DeepSeek's fp8_ds_mla page layout with int8 values and a Triton read path for Ampere: 656 instead of 1,152 bytes per token per layer, 2.6x less attention error than e4m3 on the layer measured with exact inputs, no speed cost. With the 22,20,20,16 split the KV pool goes from 171K tokens (BF16) to 570K, which is what makes 512K context fit. Plain fp8 with scale 1.0 is unsafe for this model: the early layers' latent values are almost all subnormal.
• PipeSpec. With one request, three of the four stages sit idle while the last one verifies. PipeSpec starts the next speculative step before the current one is verified and rolls back if the bet fails: +6% coding, +9-13% browsing, +3-5% prose (flag on against off, same boot). It runs only while exactly one request is decoding, with exactly 2 greedy drafts and no structured output.
• Exact NVFP4 experts plus INT8 residue. Most of the bytes read per token are the BF16 non-expert weights. Converting those to INT8-g128 and keeping the NVFP4 expert bytes passes his quality gate (96.4% teacher-forced top-1 agreement against the original NVFP4 model, gate 95%); converting the experts to INT4 as well does not (93.65%).
• TP2 x PP4. Two GPUs read each stage's weights in parallel, all-reduces stay inside one switch, and the KV pool stays large. TP4 x PP2 is as fast once P2P works, but its KV pool tops out around 240K tokens.

Reported
Source: repo github.com/cebeuq/glm53-cmp170hx-recipe by cebeuq on GitHub

Hardware notes

• The HBM clock is fixed at 1,728 MHz and cannot be raised. The SM clock still matters for decode: at a 100 W cap with the SM clock locked at 450 MHz, PP8 ran at 11.0 tok/s against 31.0 at 160 W and up to 1,695 MHz.
• Under sustained load the 160 W cap binds (SM clock 750-1,100 MHz), so multi-stream decode and prefill are power-limited; single-stream decode is not.
• Change power caps in small steps, one card at a time, and watch the kernel log: raising one card sharply to 250 W knocked it off the bus once.
• Do not run CPU-heavy jobs on the server while the cards are loaded: a checkpoint conversion added about 380 W and tripped his wall-power guard once.
• On his board the firmware owns PCIe AER, so NVIDIA Xid messages in the kernel log are the only error signal. PCIe Gen3 is fuse-blocked, and CUPTI is fused off, so Nsight and torch.profiler do not work.

Source: repo github.com/cebeuq/glm53-cmp170hx-recipe by cebeuq on GitHub

Known limits

• Measured on one server. Other boards, BIOS settings, PCIe topologies or driver builds may behave differently, especially for P2P.
• Greedy output is not reproducible run to run on this engine (this predates his patches), so exact match cannot serve as a correctness check. He uses teacher-forced agreement against reference outputs, NLL on fixed text, GSM8K and long-context ladders (bench/ in the repo).
• PipeSpec runs only while exactly one request is decoding. He notes it may lower draft acceptance at long agentic contexts (not measured).
• P2P relies on a modified NVIDIA kernel module and on ForceP2P, which advertises every pair; only the byte-exact probe shows that bytes land correctly.
• The patches are whole-file copies for one pinned image. Mounted over any other image they break it; port them with the diffs in the repo.
• Startup is slow: loading about 420 GB of weights, repacking NVFP4 for Marlin and capturing CUDA graphs takes many minutes.
• The API has no authentication and listens on the local machine only by default.
• The checkpoint is abliterated (its refusal behavior was removed by the checkpoint author); add whatever safeguards your use needs.
• Open question in his docs: the TP2 pipeline hand-off may combine the two GPUs' partial sums incorrectly at the last layer of each of the first three stages (layers 21, 41 and 61), found by reading the fork's source and not tested by him. On the same weights TP2 x PP4 scored 93.9% teacher-forced agreement against 96.4% for PP8.

Source: repo github.com/cebeuq/glm53-cmp170hx-recipe by cebeuq on GitHub

• The lab has run a fix for that hand-off since 2026-10-27, and a fix for schema-constrained output (json_schema, named tool choice) with speculative decoding since 2026-11-04; see GLM 5.3 on 8x 8 GB (cebeuq glm53-cmp170hx-recipe)/Confirmation 1 and vLLM on sm_80: known bugs and patches. Source: @darkmaniac7, 2026-11-04 (lab notes)
• Since 2026-11-05 these two fixes and the lab's probabilistic drafts for sampled requests are public and offered to this repo as pull requests 1 to 3. Two later structured-output options (drafts kept per request under concurrency, and no end of turn inside the thinking block for structured requests) were measured on the lab's two-host layout: GLM 5.3 on 8x 8 GB plus 2x RTX PRO 6000 (vLLM TP2 x PP5)#The lab's additions. Source: @darkmaniac7, 2026-11-05 (lab notes)
  Source: repo github.com/darkmaniac7/glm53-cmp170hx-blackwell-recipe by darkmaniac7 on GitHub

Related

• The lab's confirmation on its own eight cards: GLM 5.3 on 8x 8 GB (cebeuq glm53-cmp170hx-recipe)/Confirmation 1. The lab's two-host extension with two RTX PRO 6000 Blackwells: GLM 5.3 on 8x 8 GB plus 2x RTX PRO 6000 (vLLM TP2 x PP5).
• The smaller sibling recipe it builds on: GLM 5.3 Flash on 4x 8 GB. Another full GLM-5.3 build for eight to ten cards: GLM 5.3 on 8x 8 GB (vllm-170hx-glm5).
• P2P, Engines#wtdcode/vllm-backport, Speculative decoding on the 170HX, vLLM on sm_80: known bugs and patches, Large rigs (8 or more cards), Benchmarks.

Sources

• Repo: cebeuq/glm53-cmp170hx-recipe (MIT, by cebeuq, pushed 2026-10-10): the README and the docs on results, MTP, PipeSpec, P2P, the KV cache, how it works, hardware, methodology and known limits. His docs also list the experiments that did not help (in docs/what-didnt-work.md).
• It builds on Morrowmake/glm53-flash-cmp170hx-recipe and Morrowmake/cmpunlocker (TRAP31), runs on wtdcode/vllm-backport (lazymio's image), and uses dealignai/GLM-5.3-ABLITERATED-NVFP4, derived from zai-org/GLM-5.3.

Imported from the cmpunlocker Discord with credit; see Credits.