GLM 5.3 Flash on 8x 8 GB (promisezackr PP8 NVFP4 DFlash2)

GLM-5.3-Flash

Reported 8 x 8 GB
Quant	LibertAIDAI/GLM-5.3-Flash-NVFP4 (ModelOpt NVFP4, routed experts only, 181 GiB), run through Marlin W4A16 kernels; GLM-5.3-Flash DFlash2 drafter (5 sliding-window layers)
Engine	vLLM: official vllm/vllm-openai:glm53-flash image plus promisezackr's 24 patches (SM8x base from 344303947/vllm170hx, pipeline parallel for GLM-5.3-Flash, DFlash2 under PP8) v0.1.dev20051+g487ecf187 (the image's vLLM); repo commit 90ec72e
Topology	PP8 (VLLM_PP_LAYER_PARTITION=7,5,7,5,7,5,7,2) with DFlash2, 7 draft tokens; PCIe Gen2 x4, no P2P; host 2x Xeon E5-2690 v4, 96 GB DDR4
Context	1,048,576 (KV pool 6.62M tokens at utilization 0.90, about 6.3 concurrent 1M requests)
Decode	123 tok/s
VRAM per card	64 GB, --gpu-memory-utilization 0.90
Unlock build	cmpunlocker VRAM unlock (64 GB per card)
Flags	--pipeline-parallel-size 8 --max-model-len 1048576 --max-num-seqs 8 --max-num-batched-tokens 4096 --gpu-memory-utilization 0.90 --no-enable-prefix-caching --speculative-config with method dflash and num_speculative_tokens 7 --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm45; env VLLM_PP_LAYER_PARTITION=7,5,7,5,7,5,7,2 VLLM_PP_MAX_DECODE_REQS_PER_BATCH=2 VLLM_GLM5N_SIDECAR_BLOCK_SIZE=256 VLLM_WORKER_MULTIPROC_METHOD=spawn CUDA_DEVICE_ORDER=PCI_BUS_ID; host: CPU governor performance, nvidia-smi -lgc 1695
Tested	@promisezackr on 2026-08-31
Source	link
Files	scripts/run_prod.sh, patches 0001 to 0024 and patches/all-changes-vs-baseline.diff, apply.sh, scripts/bench.py, conc2.py, needle.py, qual.py and preflight.py (Apache-2.0)
Confirmations	None yet. Confirm on your own card

promisezackr's patch set serves GLM-5.3-Flash (320B MoE, 18B active, 1M context) on eight 64 GB cards at PCIe Gen2 x4 with no P2P: vLLM pipeline parallel 8, the LibertAIDAI NVFP4 checkpoint and the DFlash2 drafter with 7 draft tokens. The repo reports 123 tok/s single stream on code, 276 tok/s aggregate at 16 streams, a 960k-token prompt in 208 s and a 6.62M-token KV pool. Zanooda posted it on Discord and runs the same patches on six cards (The six-card variant). The numbers on this page are the repo's own readings: nobody has posted a matching eight-card run yet, and one owner stalled at 20 tok/s (Known issues).

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub, posted by @zanooda, 2026-08-31

Prerequisites

Eight 64 GB cards at Gen2 x4; P2P is not needed

The repo's box has eight CMP 170HX unlocked to 64 GB (Driver unlock, 8 GB card, Verifying an unlock), each on PCIe Gen2 x4 (Gen2 link retrain) with no P2P, on a host with 2x Xeon E5-2690 v4, 96 GB DDR4 and a SATA SSD for the weights. Tensor parallel is not an option on this box without P2P, so the recipe is pure pipeline parallel (Parallelism over the PCIe link).

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub

Weights: the NVFP4 checkpoint and the DFlash2 drafter

• Target: LibertAIDAI/GLM-5.3-Flash-NVFP4 (ModelOpt NVFP4, routed experts only, 181 GiB). SM80 has no native FP4 path, so the NVFP4 MoE runs through Marlin W4A16 kernels (vLLM PR #30906), with Triton sparse MLA and indexer fallbacks for SM80.
• Drafter: the GLM-5.3-Flash DFlash2 drafter (5 sliding-window layers, from z-lab/dflash), run with FlashAttention 2.
• The launch script expects the two models under /data/models/GLM-5.3-Flash-NVFP4 and /data/models/GLM-5.3-Flash-DFlash2. From mainland China, ModelScope downloads far faster than Hugging Face.

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub

Is NVFP4 accurate enough?

Contested

• tchelobh (2026-08-29) compared FP8 and NVFP4 on his own batch workload and found NVFP4 output borderline unusable. CUTE says INT4 is better, and giaki3003 ranks int above fp above nvfp. Source: @tchelobh, 2026-08-29
  Source: @cute_sc2, 2026-08-30
  Source: @giaki3003, 2026-08-30
• DeeJay's GLM 5.3 Flash quant tests found NVFP4 the most accurate of his quants at 9% error, against 12% for his own W4A16 and 19% for AWQ, but NVFP4 needs about 30 GB more memory, which leaves less KV cache. Source: @deejay9983, 2026-08-31
  Source: @deejay9983, 2026-08-31

Results differ by model and quant source. The repo's own quality probe (scripts/qual.py) scored 8 of 8 checkable items on NVFP4. Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub Quant background: Quantization on the 170HX.

Why a published patch set: GLM 5.3 Flash with DFlash took dozens of patches

On 2026-09-01 DeeJay said Opus needed 34 to 51 patches (the count rose during the day) to get GLM 5.3 Flash with DFlash working in vLLM, against about 6 for DeepSeek, and later that day reported DFlash working on his CMP setup ("Only 57 vllm patches guys"). segmond's attempt to have GLM 5.3 write the vLLM patch failed after 14 hours. Zanooda replied that everything needed is in promisezackr/glm53-flash-170hx-pp8.

Reported
Source: @deejay9983, 2026-09-01
Source: @segmond, 2026-09-01
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub, posted by @zanooda, 2026-09-01
Source: @deejay9983, 2026-09-01

What the 24 patches do

The patches are 24 commits on top of the image's vllm/ package, in order; patches/all-changes-vs-baseline.diff is the same change as one diff (65 files). In groups:

1. SM8x port of the glm53-flash tree from 344303947/vllm170hx (Marlin FP4 and Triton fallbacks), fp8 software encode for the indexer key pool, NoPE sparse MLA (512), backend registration.
2. Pipeline parallel for the GLM-5.3-Flash model code: intermediate-tensor factories, materializing the mHC output at stage boundaries, dynamic top-k width, and the draft model built with pipeline size 1 on the last rank.
3. MTP under PP: the draft embed_tokens and shared_head.head are loaded from the checkpoint (upstream skips sharing them under PP, which left them random: 0 of 3,800 drafts accepted before the fix); a sync-free draft-token scatter (about 106 to 49 ms per step, 2x single stream); an unquantized shared head for quantized checkpoints.
4. DFlash2 under PP8 (a backport of vLLM PR #52816): the auxiliary hidden states are relayed across stages and staged out of CUDA-graph buffers, the drafter gets its own sidecar KV groups, and the zero draft embeddings under PP are fixed (the root cause of low acceptance).
5. Scheduler: the VLLM_PP_MAX_DECODE_REQS_PER_BATCH decode micro-batch cap (8-stream aggregate 117 to 260 tok/s).
6. Sparse MLA: int64 KV offsets, which fix an Xid 31 MMU fault once the KV pool passed 2^31 bytes per layer.
7. KV: the drafter's sidecar blocks go from 16 to 256 tokens (VLLM_GLM5N_SIDECAR_BLOCK_SIZE=256). Block ids come from one shared pool and each costs a full MLA page on every rank; at 16 tokens a 1M request needed 2,433 sidecar ids against 277 core ids and about 90% of the pool sat empty. Re-blocking raised 1M concurrency from 1.13x to 6.31x at less memory.
8. Env-gated instrumentation (VLLM_STAGE_TIMING, VLLM_STAGE_TRACE, VLLM_DFLASH_TRACE) and scripts/preflight.py, a seconds-long import and protocol check to run before a 12-minute boot.

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub, posted by @zanooda, 2026-08-31

Steps

1. Unlock all eight cards to 64 GB and retrain them to Gen2. Check that every card reports 64 GB before going further.
2. Download the NVFP4 checkpoint and the DFlash2 drafter to a local disk.
3. Clone promisezackr/glm53-flash-170hx-pp8 and pull vllm/vllm-openai:glm53-flash. The patches target that one image build (vLLM v0.1.dev20051+g487ecf187).
4. Copy the vllm package out of the image and apply the patch set to the copy, either as the single diff or one patch at a time with apply.sh.
5. Edit the model and drafter paths and the container name in scripts/run_prod.sh, then run it. It bind-mounts the patched package over the image's own and starts the server.
6. Wait for READY. Weight loading takes about 5 minutes from a SATA SSD and the first warmup about 15 minutes; with the Triton cache persisted, later boots take about 12 minutes. The script polls the health endpoint for up to 40 minutes.
7. Optional host tuning: set the CPU governor to performance and lock the GPU clocks with nvidia-smi -lgc 1695 (scripts/glm-perf.service). The repo measured about 10% lower step time.

The repo's reproduce block:

# 1. take the vllm package out of the official image
docker create --name tmp vllm/vllm-openai:glm53-flash
docker cp tmp:/usr/local/lib/python3.12/dist-packages/vllm ./glm53/vllm && docker rm tmp
# 2. apply the patch set (paths are rooted at the parent of vllm/)
cd glm53 && git init -q && git add -A && git commit -qm baseline
git apply --whitespace=nowarn ../patches/all-changes-vs-baseline.diff      # or ./apply.sh
# 3. bind-mount it over the image's package and launch
bash ../scripts/run_prod.sh   # edit model/draft paths and the container name first

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub

Production launch: run_prod.sh

scripts/run_prod.sh starts the official image with --gpus all --ipc=host, the patched package mounted over the image's vllm, the two models mounted read-only, and these settings:

HF_HUB_OFFLINE=1
VLLM_PP_LAYER_PARTITION=7,5,7,5,7,5,7,2
VLLM_PP_MAX_DECODE_REQS_PER_BATCH=2      # spread decodes over the PP in-flight micro-batches
VLLM_GLM5N_SIDECAR_BLOCK_SIZE=256        # drafter KV block size (patch 0024)
VLLM_WORKER_MULTIPROC_METHOD=spawn
CUDA_DEVICE_ORDER=PCI_BUS_ID

--pipeline-parallel-size 8 --max-model-len 1048576 --max-num-seqs 8
--max-num-batched-tokens 4096 --gpu-memory-utilization 0.90
--no-enable-prefix-caching
--speculative-config '{"method":"dflash","model":"/dflash2","num_speculative_tokens":7}'
--enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm45

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub, posted by @zanooda, 2026-08-31

Layer partition 7,5,7,5,7,5,7,2: fewer layers where two sparse-MLA layers sit

GLM-5.3-Flash has 11 sparse-MLA layers (index 3, 7, and so on up to 43). The partition gives fewer layers to the stages that hold two of them, so those stages keep a larger KV budget. Change the partition with the card count: Zanooda's six cards use 9,7,9,7,9,4 (The six-card variant).

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub, posted by @zanooda, 2026-08-31

Measured numbers

Single stream: 123 tok/s on code with DFlash2 k=7, 37 tok/s on prose

scripts/bench.py, single stream, temperature 0, tokens per second including speculative decoding. The production column is NVFP4 with DFlash2 and 7 draft tokens.

Workload	FP8, eager, no spec	FP8 + MTP k=3	NVFP4 + MTP k=3	NVFP4 + DFlash2 k=7 (production)
code	6.2	51-61	71	123
json	not run	not run	71	120
counting	not run	77	65	114-118
math	not run	not run	59	75
prose	not run	not run	43	37

On NVFP4, DFlash2 adds about 70 to 80% over MTP on code, json and counting, about 27% on math, and is slightly slower than MTP on prose. See Speculative decoding on the 170HX.

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub, posted by @zanooda, 2026-08-31

Concurrency: 201-203, 259-267 and 276 tok/s aggregate at 4, 8 and 16 streams

scripts/conc2.py N 40 (chat requests, 200-token completions) measured steady-state aggregate throughput of 201-203 tok/s at 4 streams, 259-267 at 8 and 276 at 16. The decode micro-batch cap (VLLM_PP_MAX_DECODE_REQS_PER_BATCH=2) is what raised the 8-stream figure from 117 to about 260.

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub, posted by @zanooda, 2026-08-31

Long context: a 960k-token prompt in 208 s, 6.62M-token KV pool

With the production configuration, scripts/needle.py ran a 192k-token prompt in 53 s and a 960k-token prompt in 208 s end to end, with the needle retrieved both times. The KV pool is 6.62M tokens at utilization 0.90, room for about 6.3 concurrent 1M requests. Prefill as reported: 192k tokens in 53 s, 960k tokens in 208 s (end-to-end request time, not a prefill rate).

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub, posted by @zanooda, 2026-08-31

Step time: about 32 ms fixed plus 2.1 ms per verified token

Per-stage CUDA event timing (patches 0018 and 0019) gives a step time of about 32 ms fixed plus 2.1 ms per verified token. The fixed part is the serial 8-stage pipeline: attention, KDA and dense layers on each stage, the hops between stages, and host-side preparation and sampling. The variable part is active-expert weight bandwidth, which grows with tokens times top-k. The repo puts the saving from going to PP6 at only about 6%.

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub

The six-card variant

Zanooda: six cards, wtdcode AWQ W4A16, BF16 KV, about 3.2M-token pool

Zanooda (2026-08-31) runs these patches on six cards as the image zanooda/vllm-sm80-glm53flash:pp8-dflash2, which he describes as vanilla vLLM with the promisezackr patches baked in. He serves wtdcode/GLM-5.3-Flash-AWQ-W4A16 with a BF16 KV cache and the incoai/GLM-5.3-Flash-DFlash2 drafter with 7 draft tokens. Differences from the eight-card launch: partition 9,7,9,7,9,4, VLLM_PP_MAX_DECODE_REQS_PER_BATCH=4, Docker --shm-size=16g, and prefix caching turned on. The KV pool is about 3.2M tokens, which he puts at roughly 2 concurrent 1M requests in practice, because prefix caches and DFlash2 overhead share the pool.

# env
CUDA_DEVICE_ORDER=PCI_BUS_ID
VLLM_PP_LAYER_PARTITION=9,7,9,7,9,4
VLLM_PP_MAX_DECODE_REQS_PER_BATCH=4
VLLM_GLM5N_SIDECAR_BLOCK_SIZE=256
VLLM_WORKER_MULTIPROC_METHOD=spawn
# docker: --ipc=host --shm-size=16g, image zanooda/vllm-sm80-glm53flash:pp8-dflash2
wtdcode/GLM-5.3-Flash-AWQ-W4A16 \
  --pipeline-parallel-size 6 --max-model-len 1048576 --max-num-seqs 8 \
  --max-num-batched-tokens 4096 --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm45 \
  --speculative-config '{"method":"dflash","model":"incoai/GLM-5.3-Flash-DFlash2","num_speculative_tokens":7}'

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub, posted by @zanooda, 2026-08-31

About 45 tok/s without DFlash2; prefill falls once DFlash2 is in

Without DFlash2, decode on the six-card setup sits at about 45 tok/s. His own fork reached about 6,500 tok/s prefill, which drops as soon as DFlash2 is integrated. W8A16 does not fit on six cards. He finds GLM 5.3 Flash a bit slower than DeepSeek V4 Flash but with noticeably better output for his use, and advises against the "max" reasoning level unless you need it: it uses about twice as many tokens as "high".

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub, posted by @zanooda, 2026-08-31

Known issues

DFlash needs prefix caching off

The production launch passes --no-enable-prefix-caching. The patch that makes the DFlash auxiliary relay async-safe (0010) notes that DFlash currently requires it, because the interaction between the drafter's lookahead and the prefix cache corrupts output, an issue the patch set leaves open. Zanooda's six-card command turns prefix caching on; if output goes wrong on a variant like that, turn prefix caching off first.

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub, posted by @zanooda, 2026-08-31

Leave the fused draft loop off

Patch 0006 adds a fused multi-step draft loop (one metadata build per step instead of one per draft pass). Patch 0008 turns it off by default: a bisect showed its update hooks crash with an illegal memory access under concurrent speculative decoding, with CUDA graphs and in eager mode alike, and once the sync-free draft scatter restored async overlap it gave no measurable speedup (51-61 against 53-64 tok/s single stream). The hooks stay behind VLLM_ENABLE_FUSED_DRAFT=1 for later rework; leave that variable unset.

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub

Patch 0021 (adaptive spec truncation) is reverted by 0022

Patch 0021 adds adaptive per-request speculative truncation (VLLM_ADAPT_SPEC_LOW_K, VLLM_ADAPT_SPEC_THRESH) and patch 0022 reverts it. Apply the whole series in order (or the single diff, which has the same end state): in the finished tree those variables do nothing. Do not cherry-pick 0021 on its own.

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub

Xid errors and a 20 tok/s ceiling on PP8; unlocker 0.3 fixed the Xid errors for Zanooda

1a2b3c0967 (2026-09-01) could not get past 20 tok/s on his 8-card rig running PP8 with DFlash at 7 draft tokens. He had let an AI agent set it up all day, and it kept getting stuck on Xid errors that needed host reboots. Zanooda had seen the same Xid errors and fixed them by updating the unlocker to version 0.3, without changing the driver (he stayed on 610.43.02). If a PP8 run keeps throwing Xid errors, update the unlocker first (Driver unlock, Xid errors).

Reported 8 GB
Source: @zanooda, 2026-09-01
Source: @1a2b3c0967, 2026-09-01

Xid 31 at large KV pools: keep the int64 offsets patch

Before patch 0023 the sparse-MLA path used 32-bit KV offsets, and the run hit an Xid 31 MMU fault once the KV pool passed 2^31 bytes per layer. The patch switches to int64 offsets and adds an env-gated index bounds check. A build on another base without it can fault at the same pool sizes. Background: Xid 31 and the reserved top of VRAM, vLLM on sm_80: known bugs and patches.

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub, posted by @zanooda, 2026-08-31

Related

GLM 5.3 Flash on 4x 8 GB · GLM 5.3 Flash on 4x 8 GB (kaka86mm EXL3 + Marlin sidecar + DFlash2) · GLM 5.3 Flash on 3x 8 GB · GLM 5.3 on 8x 8 GB (vllm-170hx-glm5) · GLM 5.3 on 8x 8 GB (Int4-Int8Mix PP8) · DeepSeek V4.1 Flash on 8x 8 GB (Zanooda PP8) · Speculative decoding on the 170HX · Parallelism over the PCIe link · vLLM on sm_80: known bugs and patches · Quantization on the 170HX · Large rigs (8 or more cards) · Xid 31 and the reserved top of VRAM · Engines · Benchmarks

Sources

• Repo: promisezackr/glm53-flash-170hx-pp8 (Apache-2.0, a derivative of vLLM; last commit 90ec72e, 2026-08-31), posted on Discord by Zanooda on 2026-08-31 and 2026-09-01.
• The repo credits 344303947/vllm170hx for the SM8x patches, bayley's vllm-170hx-glm5 for proving pipeline parallel on the 170HX, vLLM PRs #52816 (DFlash2) and #30906 (Marlin NVFP4), LibertAIDAI for the NVFP4 checkpoint, and tonyd2wild's DGX Spark recipe as a reference.
• Six-card image: zanooda/vllm-sm80-glm53flash:pp8-dflash2 (Zanooda).

Imported from the cmpunlocker Discord with credit; see Credits.