View source for GLM 5.3 Flash on 4x 8 GB

{{Recipe
|model=GLM-5.3-Flash (320B MoE, 18B active)
|quant=W4A16 (INT4 weights, group size 128, FP16 activations): canada-quant/GLM-5.3-Flash-W4A16-MTP
|engine=Morrowmake/vllm-cmp170hx (vLLM fork, container image)
|engine_version=recipe 1.8.0, engine 819146023f
|cards=8 GB
|card_count=4
|topology=TP4 (default) or PP4
|context=262,144
|decode_tps=442.1
|prefill_tps=3161
|vram_per_card=64 GiB exposed, --gpu-memory-utilization 0.95
|power_cap=180 W
|driver=NVIDIA 580 or newer; P2P needs the Morrowmake/cmpunlocker build
|unlock_build=cmpunlocker (64 GiB, 74 SMs); Morrowmake/cmpunlocker for P2P
|flags=LAYOUT=tp4 or LAYOUT=pp4, P2P=auto, --max-num-seqs 8, --gpu-memory-utilization 0.95, --max-num-batched-tokens 3463 (TP4) or 2312 (PP4)
|status=Reported
|tested_by=voidgiraffe
|tested_on=2026-10-16
|source=https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe
}}
The community's reference build for four cards: GLM-5.3-Flash in W4A16 on Morrowmake's Ampere vLLM fork, with a
DFlash2 speculative drafter, tool calls, reasoning, images and video, behind an OpenAI-compatible API. Most 4-card
owners run W4A16 on this recipe, and Impactic says nobody really cares to upgrade past it. Infobox numbers are the
author's TP4 release figures for one user on code.
{{Source|handle=gnasherrr|date=2026-10-27|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1564473314502508557}}<br />
{{Source|handle=ob88594|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556898939234558013|repo=https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe}}<br />
{{Source|handle=impactic|date=2026-10-24|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1563623398037393520}}

== Prerequisites ==
* Four CMP 170HX that each expose 64 GiB (<code><nowiki>nvidia-smi</nowiki></code> shows 65,536 MiB). The preflight stops below 61,440 MiB. See [[Driver unlock]] and [[8 GB card]].
* PCIe Gen2 x16 links for TP4 (the x16 capacitor modification). Cards stuck at x4 should use PP4. See [[Unlocks]].
* NVIDIA driver 580 or newer, Docker with the NVIDIA Container Toolkit. <code><nowiki>docker run --rm --gpus all nvidia/cuda:13.3.1-base-ubuntu24.04 nvidia-smi</nowiki></code> must list all four cards.
* Optional: a P2P-capable driver build (Morrowmake/cmpunlocker: mainline cmpunlocker plus the maintainer P2P patch and the TRAP31 PLM patch, <code><nowiki>ForceP2P=0x11</nowiki></code>). Without it the server runs with P2P off.
* About 180 GiB (193 GB) of disk for the two checkpoints (about 178 GiB model, 2.2 GiB drafter), plus the engine image and compile caches.
* <code><nowiki>git</nowiki></code>, <code><nowiki>curl</nowiki></code>, <code><nowiki>flock</nowiki></code>, <code><nowiki>setsid</nowiki></code>, <code><nowiki>python3</nowiki></code>, <code><nowiki>jq</nowiki></code>.
{{Source|handle=ob88594|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556898939234558013|repo=https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe}}

== Steps ==
<syntaxhighlight lang="bash">
git clone https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe.git
cd glm53-flash-cmp170hx-recipe
./start.sh          # pull the engine image by digest, fetch the model, start, wait for /health
./start.sh smoke    # one chat request and one tool call
</syntaxhighlight>
Pick the layout with one variable, either in <code><nowiki>.env</nowiki></code> or as a prefix: <code><nowiki>LAYOUT=pp4 ./start.sh restart</nowiki></code>.
<code><nowiki>./start.sh</nowiki></code> checks link width and suggests PP4 if a card is narrower than x16. Docker Compose is the
alternative (<code><nowiki>docker compose up</nowiki></code>); use one or the other. Other subcommands: <code><nowiki>status</nowiki></code>,
<code><nowiki>logs</nowiki></code>, <code><nowiki>stop</nowiki></code>, <code><nowiki>restart</nowiki></code>, <code><nowiki>update</nowiki></code>; <code><nowiki>DRY=1 ./start.sh</nowiki></code> prints the
launch without running it.

Test request:
<syntaxhighlight lang="bash">
curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model": "glm-5.3-flash", "messages": [{"role": "user", "content": "Write a haiku about PCIe."}], "max_tokens": 4096}'
</syntaxhighlight>
The API listens on 127.0.0.1:8000 with no key unless <code><nowiki>HOST</nowiki></code> and <code><nowiki>API_KEY</nowiki></code> are set in
<code><nowiki>.env</nowiki></code>. The first boot compiles kernels into <code><nowiki>./cache</nowiki></code> and is slow; "No available shared memory
broadcast block found in 60 seconds" lines during that compile are harmless.
{{Source|handle=ob88594|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556898939234558013|repo=https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe}}

=== What the release pins ===
* Engine: Morrowmake/vllm-cmp170hx branch <code><nowiki>ampere</nowiki></code> at <code><nowiki>819146023facb31c04f1f8512e52464835a10d40</nowiki></code> (tag <code><nowiki>glm53-recipe-1.8.0</nowiki></code>), on upstream vLLM 4e0681ed56. Image <code><nowiki>ghcr.io/morrowmake/vllm-cmp170hx:engine-819146023f</nowiki></code>, digest <code><nowiki>sha256:18a2f26207568128e1d792ce923e86a74e40def1e43e492db83132cab8cdbe14</nowiki></code>.
* Model: [https://huggingface.co/canada-quant/GLM-5.3-Flash-W4A16-MTP canada-quant/GLM-5.3-Flash-W4A16-MTP] at revision <code><nowiki>5681f7c2</nowiki></code> (base [https://huggingface.co/zai-org/GLM-5.3-Flash zai-org/GLM-5.3-Flash]).
* Drafter: [https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2 incoai/GLM-5.3-Flash-DFlash2] at revision <code><nowiki>bf582e4e</nowiki></code>. It drafts up to 7 tokens for one user, 5 at two, 3 under load.
* KV cache full precision, no FP8 anywhere, no CPU or disk offload. Defaults: 262,144 context, <code><nowiki>--max-num-seqs 8</nowiki></code>, <code><nowiki>--gpu-memory-utilization 0.95</nowiki></code>, prefill chunks of 3,456 tokens (TP4) or 2,304 (PP4), fair prefill in 384-token slices, prefix caching on, <code><nowiki>glm47</nowiki></code> reasoning and tool parsers.
* Every performance feature is a one-variable kill switch set by <code><nowiki>serve.sh</nowiki></code>. Keep <code><nowiki>PREFILL_CAP=0</nowiki></code>. <code><nowiki>VLLM_CUSTOM_ALLREDUCE_ALGO=2stage</nowiki></code> is set because the built-in crossover is tuned for NVLink. 1.8.0 renamed every <code><nowiki>VLLM_GLM5_*</nowiki></code> switch; old names are ignored with a warning.
{{Source|handle=ob88594|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556898939234558013|repo=https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe}}

* Pin the release tag, not the branch: the fork's <code><nowiki>ampere</nowiki></code> branch was rebased onto new upstream on 2026-10-26, more than 1,000 commits past 1.8.0. {{Source|handle=darkmaniac7|date=2026-10-26|note=lab notes}}

=== P2P ===
With <code><nowiki>P2P=auto</nowiki></code> (default) the recipe content-checks every ordered GPU pair at start (cudaMemcpyPeer, SM-kernel
peer reads and writes, a separate-process CUDA IPC write; 128 KiB and 512 KiB with one byte either side, plus 1, 8 and
32 MiB). A pass sets <code><nowiki>VLLM_CUSTOM_ALLREDUCE_PCIE_P2P=1</nowiki></code> and <code><nowiki>NCCL_P2P_LEVEL=SYS</nowiki></code>; a fail sets
<code><nowiki>NCCL_P2P_DISABLE=1</nowiki></code> and TP uses a host-staged all-reduce. <code><nowiki>nvidia-smi topo</nowiki></code> saying OK is not a
data-integrity test. See [[Unlocks]] for the P2P driver.
{{Source|handle=ob88594|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556898939234558013|repo=https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe}}

== Measured numbers ==
Author's release 1.8.0, measured 2026-10-16 at 74 SMs and 180 W per card, PCIe x16, P2P verified, MiaAI-Lab protocol
(400 tokens, temperature 0). Tok/s as structured / code / prose.

{| class="wikitable"
! Layout !! 1 user !! 2 users !! 4 users !! 8 users !! Cold prefill !! KV pool
|-
| TP4 || 482.6 / 442.1 / 232.1 || 606.5 / 460.3 / 314.7 || 717.7 / 618.7 / 501.4 || 943.7 / 835.0 / 667.2 || 3,161 || 1,203,765
|-
| PP4 || 232.4 / 209.9 / 100.1 || || || 664.1 / 608.8 / 469.2 || 7,419 || 2,088,960
|}
Quality (TP4): HumanEval 151/164, GSM8K 1,286/1,319.
{{Source|handle=ob88594|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556898939234558013|repo=https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe}}

=== Other people's numbers on this recipe ===
* cibernox (daily driver, x16, P2P on, TP4): over 350 tok/s in short bursts, 270+ tok/s on real workflows. {{Source|handle=cibernox|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1559515598797672479}} About 800 tok/s aggregate across agents. {{Source|handle=cibernox|date=2026-10-19|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1561716217897689259}}
* turbine (TP4): 317 tok/s at batch 1 on code, 164 on prose. {{Source|handle=turbine|date=2026-10-27|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1564557999463202917}}
* Mountains (PP4 on PCIe Gen2 x4 links): 90 to 140 tok/s decode with a 1M context window, prefill around 3k tok/s. {{Source|handle=mountains1605|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556410859117482025}} 340 tok/s total at 8 concurrent on real agent work. {{Source|handle=mountains1605|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557598774887710792}}
* ikyo (first 4-card run, an early release Morrowmake said had bugs; 768-token answers, reasoning counted): concurrency 1 161.8 tok/s per user; 2: 124.3 per user, 231.0 aggregate; 4: 92.6 / 307.5; 8: 51.7 / 296.1 with 4.55 s median TTFT. Single user by workload: structured 182.1, code 161.8, prose 141.2. {{Source|handle=ikyo|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556906534926028921}}
* 1a2b3c0967 (release 1.7, PP4): concurrency 3, 1M context, KV pool about 3M tokens, still over 75 tok/s on prose. {{Source|handle=1a2b3c0967|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556144621451747412}}
* peter643 (4x 64 GB capped at 180 W, P2P on; recipe not named): 200 to 400 tok/s at single concurrency, about 104 W per card while generating, just over 400 W for the four cards. {{Source|handle=peter643|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562011301243523115}}
* PixelML's Club CMP 170HX (four cards behind two PLX switches on a Broadwell host, release 1.6.0): 396.0 tok/s structured for one user and 798.6 at eight users at 180 W with P2P off, matching Morrowmake's own EPYC P2P-off figures (394.0 / 797.9); 418.0 / 816.5 after equalising HBM clocks, at 150 W. {{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}
* Ivan (8-card box, two TP4 instances of this recipe, abliterated W4A16 + MTP checkpoint in place of canada-quant's, 1,024-token answers, effort low, NCCL only): see the layout table below. {{Source|handle=darkmaniac7|date=2026-10-28|note=lab notes}}
* {{Status|Reported}} Miséricorde, a claim that has not been reproduced: four cards on Morrowmake's runtime reach 1,000+ tok/s at 8 concurrent and 350+ tok/s at one stream on code, with "some millions" of tokens of KV, and Morrowmake's published numbers are for stock cards at lower power limits. He posted no setup, power limit or log; the release figures above for code are 442.1 tok/s (1 user) and 835.0 (8 users) at 180 W. He also finds GLM 5.3 Flash about 2x less verbose than Qwen 3.8 27B and 2 to 3x faster on code. Summary of his posts; see [https://discord.com/channels/1518634379604136178/1518692084729511967/1563780170735816828 Discord]. {{Source|handle=wakeupmbrc1|date=2026-10-23|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563206459955609635}}<br />{{Source|handle=wakeupmbrc1|date=2026-10-23|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563206518537457676}}<br />{{Source|handle=wakeupmbrc1|date=2026-10-23|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563206538083041301}}<br />{{Source|handle=wakeupmbrc1|date=2026-10-25|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563780086359003279}}<br />{{Source|handle=wakeupmbrc1|date=2026-10-25|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563780170735816828}}<br />{{Source|handle=wakeupmbrc1|date=2026-10-25|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563780191715856415}}<br />{{Source|handle=wakeupmbrc1|date=2026-10-25|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563780259097092150}}<br />{{Source|handle=wakeupmbrc1|date=2026-10-25|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563780268819619881}}<br />{{Source|handle=wakeupmbrc1|date=2026-10-25|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563780320774586479}}

== Layouts ==
The recipe ships TP4 and PP4. Measured trade-offs from the Discord:
* TP4 over PCIe Gen2 needs x16 links. On x4 links decode is usable but prefill is not, and PP4 feels snappier in conversation because prefill drives that feeling. {{Source|handle=voidgiraffe|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556157382273400935}}<br />{{Source|handle=1a2b3c0967|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556160688219627542}}
* The x16 mod is not required: Finishtilt14 runs the recipe without it and it runs well in TP or PP. {{Source|handle=finishtilt14|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1559516402849939478}}
* Mountains got P2P working, tried TP, and went back to PP4 because prefill on very long contexts is better. {{Source|handle=mountains1605|date=2026-10-15|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560411703114596473}}
* TP2 x PP2 (community flavour by cibernox on four x16 cards behind one PCIe switch): decode slower than TP4, prefill above 5k tok/s, so it sits between PP4 and TP4. He suggests it for people stuck on x4 links; not measured on x4. {{Source|handle=cibernox|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557424695370850426}}

Ivan's measurements, same bench on all three (single stream code / structured / prose, aggregate at 8 users, 17k-token
prefill):

{| class="wikitable"
! Layout !! 1 user !! 8 users !! Prefill 17k !! TTFT on a 49k prompt !! KV pool at 262K
|-
| TP4 || 252 / 235 / 139 || 623 / 617 / 421 || 1.7k || 18 s || 1.20M
|-
| TP2 x PP2 || 171 / 146 / 98 || 567 / 503 / 383 || 4.2k || 11 s || 1.14M
|-
| TP1 x PP4 || 113 / 121 / 71 || 499 / 515 / 376 || 6.8k || not measured || 2.09M
|}
TP2 x PP2 suits long prompts with short answers (coding agents); PP4 suits prefill-bound batch work. Raising
<code><nowiki>MAX_BATCHED</nowiki></code> to 8192 gave nothing on TP4 or TP2 x PP2 and cost KV.
{{Source|handle=darkmaniac7|date=2026-10-28|note=lab notes}}

On eight cards as a single instance (Ivan, cap 250 W): TP8 gave 95 / 105 / 56 tok/s for one user and 440 aggregate at 8
users with 554 tok/s prefill, because the 8-way all-reduce over Gen2 plus the socket hop dominates. TP4 x PP2 gave
103 / 145 / 38 for one user, dropped to 38 tok/s per stream at 2 users and only paid off at 32 users (492 aggregate) and
in prefill (1,856). Two TP4 instances, one per CPU socket, is the better use of eight cards.
{{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}

=== Gen2 x8 without P2P (Ivan, in an ESXi VM) ===
Ivan ran release 1.8.0 exactly as published (canada-quant checkpoint, DFlash2) on four 8 GB cards inside an ESXi
guest at Gen2 x8 with no P2P (host-staged collectives), 180 W, on the recipe's own MiaAI protocol. Tok/s structured /
code / prose at 74 SMs, the published figures in brackets:
{| class="wikitable"
! Layout !! 1 user !! 2 users !! 4 users !! 8 users !! Cold 32k prefill
|-
| TP4 || 361 / 284 / 199 (483 / 442 / 232) || 441 / 371 / 273 (607 / 460 / 315) || 545 / 468 / 401 (718 / 619 / 501) || 697 / 587 / 529 (944 / 835 / 667) || 1,633-1,753 (3,161)
|-
| PP4 || 201 / 149 / 97 (232 / 210 / 100) || 309 / 223 / 183 (344 / 273 / 196) || 386 / 353 / 300 (409 / 383 / 314) || 664 / 548 / 477 (664 / 609 / 469) || 6,834-6,977 (7,419)
|}
PP4 lands close to the published numbers, so it is not link-bound. TP4 sits at about 0.75x because its all-reduce goes
through host memory over Gen2 x8. Draft acceptance matched the published regime (95-99% structured, 74-89% code, 41-48%
prose), so the drafter is not the gap. The 74-SM unlock (70 to 74) was decode-neutral within noise and worth 2-5% on PP4
prefill; clear the torch.compile and Triton caches when the SM count changes. Single-user prose varies about 10% boot to
boot at temperature 0, other cells about 3%. TP4 settings there: <code><nowiki>VLLM_HOST_ALLREDUCE=1</nowiki></code>,
<code><nowiki>VLLM_HOST_ALLREDUCE_MAX_SIZE=524288</nowiki></code> (32 MiB was 9% slower on prefill), custom all-reduce off. A clock lock
(<code><nowiki>nvidia-smi -lgc 1410,1410</nowiki></code>) did nothing: the cards already run 1,410 MHz under load. Recommendation: PP4 for
long prompts, RAG, agents and many streams; TP4 for one interactive user with short prompts.
{{Source|handle=darkmaniac7|date=2026-10-19|note=lab notes}}

=== Drafter settings ===
On TP4 (Ivan's VM runs), DFlash2 with the release's adaptive table (7 / 5 / 3 drafts by load) is the best single
setting for every workload and user count: it is worth 3.6x / 2.8x / 2.0x over no drafter for one user (structured /
code / prose) and still 1.25-1.5x at eight users. An adaptive [7,7,5] table helps agent traffic (+14% structured, +7%
code at 4 users) and costs prose. Without a drafter TP4 decodes 101 tok/s (9.9 ms per step), PP4 48 tok/s. Deeper
drafts do not pay on these cards: each extra draft token costs about 1.1 ms of verify per step (21.2 / 23.4 / 25.6 ms at
depth 7 / 8 / 10), and depth 8 trips an assert in the fork's fused sm_80 KDA decode kernel without a patch.
{{Source|handle=darkmaniac7|date=2026-10-20|note=lab notes}}

=== Planning a custom split ===
The model has 45 decoder layers (dense MLP at 0-2, MoE at 3-44), with 11 sparse MLA attention layers at 3, 7, 11 ... 43
and KDA linear attention elsewhere. <code><nowiki>VLLM_PP_LAYER_PARTITION</nowiki></code> needs one entry per stage summing to 45 (the
MTP layer is not counted), and every pipeline stage must hold at least one MLA layer. Each rank plans its own KV pool and
all ranks are cut to the smallest, so a split must balance memory as well as compute. Explicit <code><nowiki>PP</nowiki></code> and
<code><nowiki>TP</nowiki></code> keys override <code><nowiki>LAYOUT</nowiki></code>.
{{Source|handle=darkmaniac7|date=2026-10-19|note=lab notes}}

== Eight cards ==
* Two instances of the recipe: about 1,600 tok/s aggregate on 8 cards (two PP4 groups, or two TP4 groups if x16 modded), release 1.6, power limit 180 to 200 W. {{Source|handle=impactic|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1561913182132375593}}
* Miséricorde expects 2,000+ if the cards are tuned and you can cool a 230 to 250 W power limit (not measured). {{Status|Theory}} {{Source|handle=wakeupmbrc1|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1561913397082325033}}
* Con3po runs two GLM 5.3 Flash instances on 8 cards for an agent farm: 6 VMs and 8 sub-agents, about 12 s per action. {{Source|handle=con3po7016|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560773140257312779}}

== KV cache and context ==
* Vanilla TP4 startup line on cibernox's box: "GPU KV cache size: 1,204,813 tokens, Maximum concurrency for 262,144 tokens per request: 4.60x". He notes FP8 KV would roughly double it and that vLLM shares the cache, so five parallel 250k sub-agents rarely collide. {{Source|handle=cibernox|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560687961530572833}}
* Dropping <code><nowiki>--max-num-seqs</nowiki></code> from 8 to 3 grew 1a2b3c0967's pool from 1.9M to about 3.5M tokens. He credits the 16-bit KV cache for the model staying sharp at high context; Morrowmake says research shows little loss from KV quantization but people report otherwise. {{Source|handle=1a2b3c0967|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556154620588466276}}<br />{{Source|handle=voidgiraffe|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556142515491704976}}
* Con3po runs 1.3M max context stable on 4 cards. {{Source|handle=con3po7016|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557605746945036419}}
* Miséricorde: 4 cards serve 8 users with W4A16 plus 4.5M tokens of KV, for example 9 threads at 500k each. {{Status|Contested}} cibernox gets 1.2M on the stock recipe; the 4.5M figure needs a smaller sequence count or FP8 KV. {{Source|handle=wakeupmbrc1|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560673558122987571}}<br />{{Source|handle=cibernox|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560682309978427494}}

== Power ==
* The published numbers are at 180 W per card on stock cards; tuned cards could add maybe 20%. {{Source|handle=wakeupmbrc1|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1561897014189367309}}
* peter643 sees about 104 W per card while generating under a 180 W cap; at idle he downclocks and relaxes the HBM refresh to about 25 W per card. {{Source|handle=peter643|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562011301243523115}}
* PixelML's power curve for this recipe (TP4, HBM 1,728 MHz; structured tok/s one user / eight users / four-card GPU power / tokens per watt): 165 W 418.6 / 822.4 / 650 W / 1.27; 150 W 418.0 / 816.5 / 582 W / 1.40; 140 W 412.7 / 814.2 / 537 W / 1.52; 120 W 374.4 / 756.5 / 476 W / 1.59; 110 W 340.1 / 706.8 / 436 W / 1.62; 100 W 291.9 / 627.4 / 398 W. Below 130 W the core clock falls fast. A driver reload or guest reboot resets the cap to the VBIOS default of 250 W, so re-apply it. {{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}
See [[Cooling and power]] and [[Tools]] for caps and clock tools.

== Known issues ==
* Release 1.7 had a TP4 lost-context / endless-repetition bug when a speculative step ended exactly on a 1,152-token block boundary. Fixed in 1.7.2, along with GPU memory growth over hours and garbage output after a non-local cached prefix. Two-card owners hit by the repetition moved to the ExLlamaV3 builds; the fix only concerned 4 cards. {{Source|handle=ob88594|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556898939234558013|repo=https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe}}<br />{{Source|handle=mountains1605|date=2026-10-14|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1559793849239478352}}
* GLM-5.3 always reasons. Set <code><nowiki>max_tokens</nowiki></code> for reasoning plus answer, or <code><nowiki>content</nowiki></code> comes back empty with <code><nowiki>finish_reason: "length"</nowiki></code>. <code><nowiki>enable_thinking: false</nowiki></code> is ignored; use <code><nowiki>reasoning_effort</nowiki></code> (low, high, max).
* From Z.ai's model notes: <code><nowiki>reasoning_effort</nowiki></code> on GLM-5.3-Flash accepts low, high and max and defaults to max when omitted or set to anything else; the chat template's <code><nowiki>clear_thinking</nowiki></code> defaults to false, so pass true for chat (toggling it mid-conversation costs the prompt cache). {{Source|handle=cmdrwaka324|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557656402435121172|repo=https://github.com/zai-org/GLM-5}}
* PP4 on real x4 links is not measured by the author (Mountains' numbers above are the field data).
* The default DFlash2 drafter is CC BY-NC-ND 4.0, so the published configuration is non-commercial.
* Updating is forward-only; older images may be retired.
{{Source|handle=ob88594|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556898939234558013|repo=https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe}}
* {{Status|Contested}} Custom all-reduce on Gen2. The recipe turns vLLM's custom all-reduce on after the P2P check. On Ivan's box it first hung all workers at init for 20+ minutes; the cause was stale build locks in the compile cache. Once it started, on GLM TP4 it cost about 40% decode for 13% more prefill (205 vs 252 tok/s single, 380 vs 623 aggregate at 8), so his production runs NCCL only (<code><nowiki>VLLM_CUSTOM_ALLREDUCE_PCIE_P2P=0</nowiki></code>). On the Qwen3.8 Flash Next TP2 recipe the same switch helped instead (see [[Qwen3.8 Flash Next on 2x 8 GB (ElectroHobo W4A16 TP2)]]). A/B it on your box. {{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}
* {{Status|Contested}} P2P on TP4. Morrowmake measured P2P helping on his EPYC host (about +11%) and his release numbers are with P2P verified. PixelML measured it hurting on their PLX + Broadwell host (-20 to -26% for one user, -38 to -48% at eight users, decode step 19.4 to 26.1 ms) and runs TP4 with P2P off (398.4 / 812.0 tok/s). ElectroHobo found the opposite on the same kind of host for MiMo V2.6 Flash, where P2P off is about 40% slower. {{Source|handle=nathang6866|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1561264547455569952|repo=https://github.com/ElectroHobo/mimo-v26-flash-cmp170hx-recipe}}<br />{{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}
* canada-quant's GLM-5.3-Flash-DFlash2-G drafter does not load on this engine (indexer.k_cache page size error); NVFP4 GLM checkpoints do not run on sm_80 (SM12x FlashInfer only). {{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}
* <code><nowiki>MAX_SEQS=16</nowiki></code> on TP4 crashed vLLM's CUDA-graph profiling ("operation not permitted"); keep 8 per instance. {{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}
* MTP is broken on PP4 in this fork: acceptance 0.27 / 0.11 / 0.08 and 53 / 40 / 38 tok/s, worse than no drafter (48), because the MTP head gets the wrong inputs on the pipeline layout; the recipe lists MTP as unsupported there. PP2 with <code><nowiki>SPEC_MODE</nowiki></code> none or mtp failed to come up. The no-drafter mode fails the recipe's boot check (missing spec-decode counters): run it with <code><nowiki>BOOT_CHECK=0</nowiki></code>. {{Source|handle=darkmaniac7|date=2026-10-19|note=lab notes}}
* <code><nowiki>./start.sh stop</nowiki></code> needs the recipe's lifecycle lock and exits 1 without stopping anything if another command holds it. When passing a GPU subset to Docker, keep the inner double quotes (<code><nowiki>GPUS='"device=0,1,2,3"'</nowiki></code>), because Docker parses it as CSV. Do not benchmark while a CPU-heavy job runs on the host: layouts benched beside one read 180 / 126 / 94 instead of 247 / 233 / 138. {{Source|handle=darkmaniac7|date=2026-10-28|note=lab notes}}
* Running the recipe's checkpoint on the lazymio/vllm-backport engine instead does not work: the loader fails on the first dense layer (KeyError on <code><nowiki>layers.0.mlp.gate_up_proj.weight</nowiki></code>). {{Source|handle=darkmaniac7|date=2026-10-28|note=lab notes}}

== With other GPUs in the same job ==
Ivan runs the same engine across CMP and RTX PRO 6000 Blackwell cards in one vLLM job (sm_80 + sm_120):
* Two CMPs as pipeline stage 0 (12 of 45 layers) plus two Blackwells as stage 1 (33 layers), PP2 x TP2: 349 / 261 / 180 tok/s for one user (structured / code / prose), 895 / 703 / 611 at eight users, the best eight-user aggregate he measured; 32k cold prefill in 5.95 s; a 996,784-token prompt prefilled in 206 s; KV pool 1.9M tokens at 262k (2.19M at 1M context in production). Two CMPs stay idle, because two cards cannot hold the 178 GB model.
* Four CMPs plus two Blackwells: PP3 x TP2 gave 553 / 484 / 425 tok/s at eight users (code / JSON / prose) and a 7.1M-token KV pool; PP6 x TP1 prefilled 32k in 2.92 s (11,174 tok/s). Single-user decode was lower than a Blackwell-only stack.
* It needs three small Python patches to the fork (device capability per rank, the MLA backend on sm_120, the host-staged all-reduce only on sm_80 TP groups) and launch settings: <code><nowiki>kda_prefill_backend: triton</nowiki></code>, <code><nowiki>CUTE_DSL_ARCH=sm_120a</nowiki></code>, FlashInfer autotune off, Blackwells as the last stage, every stage holding at least one MLA layer.
* Raising HBM to 1,728 MHz on the two CMPs gave only +1.6-1.9% single stream in this hybrid, because the Blackwell stage dominates the step.
{{Source|handle=darkmaniac7|date=2026-10-19|note=lab notes}}

== Variants and related ==
* Three cards: [[GLM 5.3 Flash on 3x 8 GB]].
* Two cards: [[GLM 5.3 Flash on 2x 8 GB (ExLlamaV3 TP2)]], [[GLM 5.3 Flash on 2x 8 GB (ExLlamaV3 layer split)]], [[GLM 5.3 Flash on 2x 8 GB (Project Maya)]].
* Engine notes: [[Engines]]. Quant choices: [[Quantization on the 170HX]]. All numbers: [[Benchmarks]].
* Abliterated: Ivan re-quantized dealignai's uncensored GLM-5.3-Flash FP8 checkpoint to the same W4A16 format for this engine (see [[Quantization on the 170HX#Heretic and abliterated variants]]); with the base-trained DFlash2 drafter it runs at 88-97% of the base's speed (one user 349 / 254 / 181 against 361 / 284 / 199). {{Source|handle=darkmaniac7|date=2026-10-19|note=lab notes}}
* Refusal steering: the Weightless GLP-44 vector (msuiche/weightless) is ported to the ExLlamaV3 two-card build, not to this recipe.

== Sources ==
* Repo: [https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe Morrowmake/glm53-flash-cmp170hx-recipe] (MIT), last commit 2026-10-26. Posted on Discord by OB88594, Mountains and jjb. {{Source|handle=ob88594|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556898939234558013}}
* Model: [https://huggingface.co/zai-org/GLM-5.3-Flash zai-org/GLM-5.3-Flash] (MIT); vendor notes in [https://github.com/zai-org/GLM-5 zai-org/GLM-5].
* jjb's pointers for this stack (2026-10-23): the recipe, the fork's [https://github.com/Morrowmake/vllm-cmp170hx/tree/ampere-glm53 ampere-glm53 branch], [https://github.com/wtdcode/vllm-backport wtdcode/vllm-backport] as an alternative or backport reference, and the existing [https://huggingface.co/wtdcode/GLM-5.3-Flash-AWQ-W4A16 wtdcode/GLM-5.3-Flash-AWQ-W4A16] checkpoint. The 1.8.0 release on this page pins the <code><nowiki>ampere</nowiki></code> branch and canada-quant's checkpoint instead. {{Source|handle=jjb01957|date=2026-10-23|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563303837937176596|repo=https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe}}
* Ivan's lab notes, 2026-10-27 and 2026-10-28.

{{ImportFooter}}
[[Category:8 GB]]