View source for GLM 5.3 on 8x 8 GB (Int4-Int8Mix PP8)

{{Recipe
|model=GLM-5.3 (full 78-layer, not Flash)
|quant=GLM-5.3-Int4-Int8Mix (INT4 / INT8 mix); KV fp8_ds_mla
|engine=vLLM (build not stated)
|cards=8 GB
|card_count=8
|topology=PP8, TP1
|context=1,048,576
|decode_tps=25
|vram_per_card=64 GiB, --gpu-memory-utilization 0.97
|power_cap=200 W
|flags=--pipeline-parallel-size 8 --tensor-parallel-size 1 --speculative-config mtp 5 --attention-backend TRITON_MLA_SPARSE --kv-cache-dtype fp8_ds_mla --block-size 64 --no-async-scheduling
|status=Reported
|tested_by=1a2b3c0967
|tested_on=2026-10-07
|source=https://discord.com/channels/1518634379604136178/1528826330454429706/1557523316120748073
}}
The full GLM-5.3 on eight cards with plain pipeline parallel and the model's MTP head. It runs, at about 25 tok/s,
slowing to about 10 tok/s at high context. DFlash would not cooperate on this setup. The cards ran at 200 W each with a
cardboard-box fan shroud, so no overclock. It is the baseline the vllm-170hx-glm5 build later beat on the same 8 cards.
{{Source|handle=1a2b3c0967|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557521014685180006}}<br />
{{Source|handle=1a2b3c0967|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557523006006370305}}<br />
{{Source|handle=1a2b3c0967|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557622822023602208}}

== Prerequisites ==
* Eight CMP 170HX at 64 GiB each ([[Driver unlock]], [[8 GB card]]).
* A vLLM build with the TRITON_MLA_SPARSE attention backend and <code><nowiki>fp8_ds_mla</nowiki></code> KV on sm_80. Which build was used was not stated; see [[Engines]] for the sm_80 forks.
* The GLM-5.3-Int4-Int8Mix checkpoint (Hugging Face repo not named in the post; the vllm-170hx-glm5 recipe uses GLM-5.3-Int4-Int8Mix-AWQ-g64).

== Steps ==
The command as posted (model path shortened):
<syntaxhighlight lang="bash">
vllm serve <models>/GLM-5.3-Int4-Int8Mix \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --kv-cache-memory 9663676416 \
  --kernel-config '{"ir_op_priority":{"rms_norm":["vllm_c"],"fused_add_rms_norm":["vllm_c"]}}' \
  --compilation-config '{"cudagraph_capture_sizes":[1,2,4,8,16,24,32,48,64]}' \
  --served-model-name glm-5.3 \
  --pipeline-parallel-size 8 --tensor-parallel-size 1 \
  --attention-backend TRITON_MLA_SPARSE \
  --no-async-scheduling \
  --max-model-len 1048576 \
  --gpu-memory-utilization 0.97 \
  --max-num-seqs 32 \
  --kv-cache-dtype fp8_ds_mla --block-size 64 \
  --dtype bfloat16 --trust-remote-code \
  --reasoning-parser glm47 --enable-auto-tool-choice --tool-call-parser glm47 \
  --host 0.0.0.0 --port 8000
</syntaxhighlight>
<code><nowiki>--host 0.0.0.0</nowiki></code> exposes the API to the network with no key; bind to 127.0.0.1 unless you need it.
{{Source|handle=1a2b3c0967|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557523316120748073}}

== Measured numbers ==
About 25 tok/s single stream, about 10 tok/s at high context. MTP with 5 speculative tokens; DFlash did not work.
{{Source|handle=1a2b3c0967|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557521014685180006}}

== Known issues ==
* Slow at long context. The vllm-170hx-glm5 build reaches about 45 tok/s short and 30 long on similar hardware; 1a2b3c0967 moved to it and called it a big performance bump. See [[GLM 5.3 on 8x 8 GB (vllm-170hx-glm5)]]. {{Source|handle=1a2b3c0967|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557602773208666163}}
* {{Status|Theory}} vLLM's default PP8 split for GLM-5.x starts some stages on a shared-indexer layer, which reads stale top-k indices (vLLM issue #63170, draft PR #59961). This command sets no <code><nowiki>VLLM_PP_LAYER_PARTITION</nowiki></code>, so it may be affected; nobody has measured it on this setup. See [[Engines]]. {{Source|handle=darkmaniac7|date=2026-10-28|note=lab notes}}

== Sources ==
* 1a2b3c0967's posts in clanker-talk, 2026-10-07 and 2026-10-08.
* Related: [[Recipes]], [[Benchmarks]], [[Cooling and power]].

{{ImportFooter}}
[[Category:8 GB]]