GLM 5.3 on 8x 8 GB (vllm-170hx-glm5)

GLM-5.3 (full 78-layer DSA MoE, not Flash)
Proven 8 x 8 GB
Quant	GLM-5.3-Int4-Int8Mix-AWQ-g64; drafter incoai/GLM-5.3-DFlash2
Engine	Morrowmake/vllm-cmp170hx branch ampere-glm53 plus fail-closed source patches vLLM 0.30.1rc1, fork commit 378c37b0
Topology	PP8 (also PP9 and PP10), TP1
Context	1,048,576 per request
Decode	48.7 tok/s
Prefill	1833 tok/s
VRAM per card	64 GiB, GPU_UTIL 0.96 at PP8
Power cap	170 W (author's 10-card run)
Flags	`--attention-backend TRITON_MLA_SPARSE, packed fp8_ds_mla KV, --block-size 128, async scheduling required; PP8 partition 11,10,10,10,10,10,9,8`
Tested	@reala7096 on 2026-09-29
Source	link
Confirmations	Reproduced by @1a2b3c0967 (8 GB), 2026-10-08 · details

The full GLM-5.3 (744B total, 40B active) on eight to ten 64 GB cards under pipeline parallel, with the DFlash2 speculative drafter and a 1M-token context. It pins a commit of Morrowmake's vLLM fork and applies a set of patches that port the DSA sparse-attention path to sm_80. reala published it, built on bayley's and Morrowmake's work. Infobox numbers are the author's single-stream run on 10 cards (the 8-card layout came later). Source: @reala7096, 2026-10-08
Source: repo github.com/arvkmr/vllm-170hx-glm5 by arvkmr on GitHub, posted by @reala7096, 2026-10-08

reala's own summary on Discord: 45 tok/s on short context, never below 30 on long context, and still about 30 tok/s per stream at concurrency 8: "still slow, but it feels quite usable now". Source: @reala7096, 2026-10-08

Prerequisites

Eight, nine or ten CMP 170HX at 64 GiB each (Driver unlock, 8 GB card).
CUDA 13.x toolkit (checked by install.sh), Python 3.12, torch 2.13.0, torchvision 0.28.0, torchaudio 2.11.0.
The drafter is CC BY-NC-ND 4.0: download_drafter.sh needs DFLASH_LICENSE_ACK=1.

Source: repo github.com/arvkmr/vllm-170hx-glm5 by arvkmr on GitHub, posted by @reala7096, 2026-10-08

Steps

git clone https://github.com/arvkmr/vllm-170hx-glm5 && cd vllm-170hx-glm5
./install.sh                                  # builds ~/vllm_glm53_dflash2 (engine, venv)
DFLASH_LICENSE_ACK=1 ./download_drafter.sh    # drafter is CC-BY-NC-ND-4.0
MAX_SEQS=8 ./start.sh
python3 bench_conc.py 8000 decode 8 512       # C8 decode
python3 bench_conc.py 8000 prefill 8 8192     # C8 prefill

Pins (versions.env): VLLM_REPO Morrowmake/vllm-cmp170hx, branch ampere-glm53, commit 378c37b0098a41a5cd25b3bf8b56d158e33a6cbf, precompiled wheel commit b6761e8d; drafter incoai/GLM-5.3-DFlash2 revision 425aa615.
preflight.py checks the target shape: GlmMoeDsaForCausalLM, hidden 6144, 78 layers, vocab 154880, index_topk 2048.
Profiles: smoke (32K context, 1 seq, eager), agent (1M context, 4 seqs, 512 batched tokens, FULL_AND_PIECEWISE graphs; used by start.sh), production (1M, 8 seqs, 2048 batched). Served name glm-5.3, parsers glm47.

Source: repo github.com/arvkmr/vllm-170hx-glm5 by arvkmr on GitHub, posted by @reala7096, 2026-10-08

Layouts

PP size	Partition	GPU_UTIL	KV pool
8 (default since 2026-10-05)	11,10,10,10,10,10,9,8	0.96	1,025,573 tokens
9	10,9,9,9,9,9,9,8,6	0.93	1,600,895
10	10,8,8,8,8,8,8,8,8,4	0.93	about 2.09M

KV is packed fp8_ds_mla at 656 B per token per layer plus FP8 indexer keys; --block-size 128 is required. Async scheduling is required: with --no-async-scheduling, PP plus DFlash2 asserts on the first short prompt. The DFlash2 drafter keeps its own BF16 KV, window-bounded to 2,048 tokens; without that bound the drafter's KV caps the pool at about 0.96M tokens. Source: repo github.com/arvkmr/vllm-170hx-glm5 by arvkmr on GitHub, posted by @reala7096, 2026-10-08

Measured numbers

Author's run on 10 cards, 2026-09-29 (+250 MHz VF offset, 1,350 MHz SM ceiling, 1,300 MHz on one marginal card, 170 W cap, MAX_SEQS 8):

Concurrency	Decode per stream	Aggregate	Notes
1	48.7 tok/s		3.92 accepted tokens per step; 4.3 s TTFT at about 7.8K prompt tokens (1,833 tok/s prefill)
4	32.6	108.4
8	27.1	174.1	prefill 2,351 tok/s aggregate

A single 31K prompt prefills in 13.1 s (2,372 tok/s). Decode step 82.7 ms with the decode kernel work on, 92.0 ms with it off (41.3 vs 35.4 tok/s). On 2026-09-28 single-stream was 39.7 tok/s. Source: repo github.com/arvkmr/vllm-170hx-glm5 by arvkmr on GitHub, posted by @reala7096, 2026-10-08

1a2b3c0967 ran it on 8 cards after reala added PP8, PP9 and PP10 support on 2026-10-08: concurrency 1 with an 850k-token KV pool, "big performance bump" over his own 8-card build. Source: @1a2b3c0967, 2026-10-08
Source: @1a2b3c0967, 2026-10-08
reala: GPU memory utilization 0.96 reaches 1M tokens without crashes so far. Source: @reala7096, 2026-10-08

Tuning notes from the repo

Decode kernel work, each with a kill switch: GLM52_MLA_WIDE (64 heads per program; about 5 to 6 ms per step), VLLM_GLM5_DECODE_KERNELS + VLLM_GLM5_DECODE_MOE_ROUTE_V2 (fused sm_80 MoE router, 2 to 3 ms), VLLM_GLM5_SHARED_EXPERT_REORDER (about 1 ms), GLM52_MLA_HEAD_BMM (about 0.6 ms).
GLM53_TOPK_CANON=1 replaces the fork's TIEFIX/SORTED post-processing, which corrupted decode on this target (logprob gaps up to 13 nats, stray digit tokens).
VLLM_GLM5_THIN_GEMM stays off: 0.75 ms saved but DFlash2 acceptance 4% lower. GLM52_LMHEAD_BITS=8 is optional (about 0.5 ms, small top-1 changes).
Measured and rejected: TP2 x PP5 (NCCL all-reduce about 100 us on Gen2 x4, about 4.5% net, and it crashed); plain PP8 without the top-k relay; an HBM overclock to NDIV 70 (corrupts output, no ECC); a Triton W8A16 GEMM for the dense INT8 layers (slower than Marlin). MoE experts are about 56% of the step and already stream at the HBM bandwidth limit. Underclocking costs 3 to 5% speed for 10 to 20% power savings.
serve-uncensored.sh swaps in an abliterated ("CRACK-edited") GLM-5.3-UNCENSORED-Int4-Int8Mix-AWQ-g64 checkpoint with everything else identical.

Source: repo github.com/arvkmr/vllm-170hx-glm5 by arvkmr on GitHub, posted by @reala7096, 2026-10-08

Theory reala's untested idea: with the x16 mod and PCIe switches, TP2 x PP4 on 8 cards could reach 70 to 80 tok/s at the cost of context; TP2 x PP5 on 10 cards might be the best balance. Source: @reala7096, 2026-10-08

Faulty-card procedure

One of the author's cards computed all-zero MoE output rows about once per 3,000 decode steps, and occasional garbage top-k indices, at default clock while passing GEMM and memory tests. probe_bang.py 8000 30 flags any generated token id 0 with logprob -11.95 (minus ln of the vocab size) as silent corruption: 30 greedy repeats must be identical with no token 0. Swap two devices in CUDA_VISIBLE_DEVICES to confirm the fault follows the physical card, then cap that card with gpu-clock-cap.service (nvidia-smi -i <bus id> -lgc 210,1300), which removed the fault at no measurable decode cost. That card was pulled on 2026-10-03, which is why PP9 and PP8 exist. See Troubleshooting. Source: repo github.com/arvkmr/vllm-170hx-glm5 by arvkmr on GitHub, posted by @reala7096, 2026-10-08

Known issues

As of 2026-10-30 the fork's remote branches are ampere, main, plugin and two queue/* branches; ampere-glm53 is gone, so install.sh (which clones that branch) is likely to fail unless commit 378c37b0 is still reachable by SHA. The recipe sits on the frozen ampere lineage, not the fork's new plugin layout. See Engines.
No license file. The drafter is non-commercial, no-derivatives. Single host, single author.
Full GLM 5.3 lacks vision; Con3po pushed it to 490k context but does not use it for that reason. Source: @con3po7016, 2026-10-08

Sources

Repo: arvkmr/vllm-170hx-glm5 (no license file; commits signed Arveen), last commit 2026-10-08. Posted on Discord by reala on 2026-10-08.

Prerequisites

Steps

Layouts

Measured numbers

Tuning notes from the repo

Faulty-card procedure

Known issues

Related

Sources