GLM 5.3 on 8x 8 GB (Int4-Int8Mix PP8)

GLM-5.3 (full 78-layer, not Flash)

Reported 8 x 8 GB
Quant	GLM-5.3-Int4-Int8Mix (INT4 / INT8 mix); KV fp8_ds_mla
Engine	vLLM (build not stated)
Topology	PP8, TP1
Context	1,048,576
Decode	25 tok/s
VRAM per card	64 GiB, --gpu-memory-utilization 0.97
Power cap	200 W
Flags	--pipeline-parallel-size 8 --tensor-parallel-size 1 --speculative-config mtp 5 --attention-backend TRITON_MLA_SPARSE --kv-cache-dtype fp8_ds_mla --block-size 64 --no-async-scheduling
Tested	@1a2b3c0967 on 2026-10-07
Source	Discord message
Confirmations	None yet. Confirm on your own card

The full GLM-5.3 on eight cards with plain pipeline parallel and the model's MTP head. It runs, at about 25 tok/s, slowing to about 10 tok/s at high context. DFlash would not cooperate on this setup. The cards ran at 200 W each with a cardboard-box fan shroud, so no overclock. It is the baseline the vllm-170hx-glm5 build later beat on the same 8 cards. Source: @1a2b3c0967, 2026-10-07
Source: @1a2b3c0967, 2026-10-07
Source: @1a2b3c0967, 2026-10-08

Prerequisites

• Eight CMP 170HX at 64 GiB each (Driver unlock, 8 GB card).
• A vLLM build with the TRITON_MLA_SPARSE attention backend and fp8_ds_mla KV on sm_80. Which build was used was not stated; see Engines for the sm_80 forks.
• The GLM-5.3-Int4-Int8Mix checkpoint (Hugging Face repo not named in the post; the vllm-170hx-glm5 recipe uses GLM-5.3-Int4-Int8Mix-AWQ-g64).

Steps

The command as posted (model path shortened):

vllm serve <models>/GLM-5.3-Int4-Int8Mix \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --kv-cache-memory 9663676416 \
  --kernel-config '{"ir_op_priority":{"rms_norm":["vllm_c"],"fused_add_rms_norm":["vllm_c"]}}' \
  --compilation-config '{"cudagraph_capture_sizes":[1,2,4,8,16,24,32,48,64]}' \
  --served-model-name glm-5.3 \
  --pipeline-parallel-size 8 --tensor-parallel-size 1 \
  --attention-backend TRITON_MLA_SPARSE \
  --no-async-scheduling \
  --max-model-len 1048576 \
  --gpu-memory-utilization 0.97 \
  --max-num-seqs 32 \
  --kv-cache-dtype fp8_ds_mla --block-size 64 \
  --dtype bfloat16 --trust-remote-code \
  --reasoning-parser glm47 --enable-auto-tool-choice --tool-call-parser glm47 \
  --host 0.0.0.0 --port 8000

--host 0.0.0.0 exposes the API to the network with no key; bind to 127.0.0.1 unless you need it. Source: @1a2b3c0967, 2026-10-07

Measured numbers

About 25 tok/s single stream, about 10 tok/s at high context. MTP with 5 speculative tokens; DFlash did not work. Source: @1a2b3c0967, 2026-10-07

Known issues

• Slow at long context. The vllm-170hx-glm5 build reaches about 45 tok/s short and 30 long on similar hardware; 1a2b3c0967 moved to it and called it a big performance bump. See GLM 5.3 on 8x 8 GB (vllm-170hx-glm5). Source: @1a2b3c0967, 2026-10-08
• Theory vLLM's default PP8 split for GLM-5.x starts some stages on a shared-indexer layer, which reads stale top-k indices (vLLM issue #63170, draft PR #59961). This command sets no VLLM_PP_LAYER_PARTITION, so it may be affected; nobody has measured it on this setup. See Engines. Source: @darkmaniac7, 2026-10-28 (lab notes)

Sources

• 1a2b3c0967's posts in clanker-talk, 2026-10-07 and 2026-10-08.
• Related: Recipes, Benchmarks, Cooling and power.

Imported from the cmpunlocker Discord with credit; see Credits.