GLM 5.3 Flash on 2x 8 GB (ExLlamaV3 layer split)

GLM-5.3-Flash (320B MoE, 18B active)
Proven 2 x 8 GB
Quant	EXL3 3.05 bpw: turboderp/GLM-5.3-Flash-exl3
Engine	ExLlamaV3 1.5.4 (built for SM80) with a small OpenAI-compatible server Flun/glm53-flash-cmp170hx-exl3 at 5933112
Topology	layer split (no tensor parallel), no P2P
Context	392,960 (profile q8_384k)
Decode	60 tok/s
VRAM per card	about 58 GiB of 64 GiB (116.6 GiB resident across both)
Flags	`scripts/serve.sh --profile q8_384k (-cs 393216 -cq 8 -gs 62,63 --q8-staging --q8-head-gpu0 -chunk_size 2048)`
Tested	@68x0 on 2026-10-20
Source	link
Confirmations	Reproduced by @nestaa51 (8 GB), 2026-10-09 · details

GLM-5.3-Flash entirely in HBM on two 64 GB cards, using turboderp's ExLlamaV3 and his 3.05 bpw EXL3 quant, with a DFlash2 drafter (converted locally to EXL3 6 bpw) or the model's own MTP head. Layers are split across the two cards and data moves through host memory, so it needs no P2P and no x16 mod. It is the base the two-card TP build extends (GLM 5.3 Flash on 2x 8 GB (ExLlamaV3 TP2)) and the EXL3 alternative to the vLLM path that needs four cards. Source: repo github.com/Flun/glm53-flash-cmp170hx-exl3 by Flun on GitHub, posted by @wingrimez_40467, 2026-10-20
Source: @wingrimez_40467, 2026-10-20

Prerequisites

Two CMP 170HX that each expose a full 64 GiB, with nothing else loaded: the server refuses to start if either GPU has less than about 64 GB free. See Driver unlock and 8 GB card.
Linux, nvcc for compute_80, about 140 GB of disk for the model plus the drafter conversion workspace, about 10 GB for the build.
Validated host: Ryzen 5 5600X, DDR4, PCIe Gen2 x8. Other hosts may need different load margins.

Source: repo github.com/Flun/glm53-flash-cmp170hx-exl3 by Flun on GitHub, posted by @wingrimez_40467, 2026-10-20

Steps

git clone https://github.com/Flun/glm53-flash-cmp170hx-exl3 && cd glm53-flash-cmp170hx-exl3
# 1. venv + ExLlamaV3 1.5.4 built for SM80 + API deps
scripts/setup.sh
# 2. serve; the first run downloads the target (~125 GB) and the BF16 drafter (~2 GB),
#    then converts the drafter to EXL3 6 bpw (one time, tens of minutes)
scripts/serve.sh --profile q8_384k
# 3. use it
curl http://127.0.0.1:8012/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"GLM-5.3-Flash","messages":[{"role":"user","content":"Hello"}],"max_tokens":512}'

Pins: turboderp/GLM-5.3-Flash-exl3 3.05 bpw at revision 332ab457 (about 125 GB download, about 116.6 GiB resident), drafter incoai/GLM-5.3-Flash-DFlash2 at bf582e4e, ExLlamaV3 1.5.4 built with TORCH_CUDA_ARCH_LIST=8.0. Source: repo github.com/Flun/glm53-flash-cmp170hx-exl3 by Flun on GitHub, posted by @wingrimez_40467, 2026-10-20

Profiles

Profile	Max request	Notes
q8_384k (default)	392,960 tokens	Q8 KV cache (11 MLA layers, about 4.51 GiB at 384K), `-gs 62,63`, loader margin 128 MiB
q8_320k	327,424	margin 256 MiB
q8_fast	256K	margin 192 MiB
fp16	196,352	FP16 KV
mtp	fp16 layout	model's MTP head (`--mtp -ndt 2`) instead of DFlash2

448K and 512K fail during the final layer load with this checkpoint. -gs is ExLlamaV3's per-component load budget, not a VRAM split; the LM head is reserved on GPU 0. Source: repo github.com/Flun/glm53-flash-cmp170hx-exl3 by Flun on GitHub, posted by @wingrimez_40467, 2026-10-20

What makes it work

The drafter's KV is an 8,192-token GPU ring cache for a 2,048-token sliding window, checked bit-exact against an unbounded cache. That is what lets a 384K request use DFlash2 without a 384K draft cache.
0xSero's k_hcfuse kernels fuse the four mHC decode launches per sublayer into two (90 sites per token, about 1.1 ms per token stock). Bit-exact by construction; JIT-compiled with nvcc on first decode. GLM53_K_HCFUSE=1.
Previous-request prefix reuse (--dflash-prefix-cache, on by default): a repeated 1K prompt's first token 1.50 s to 0.65 s; an 8K follow-up 7.86 s to 1.11 s; a 32K follow-up 28.49 s to 1.12 s. The author notes long real-text follow-ups did not fully reproduce cold outputs, so it is not proven lossless; GLM53_PREFIX_CACHE=0 turns it off.

Source: repo github.com/Flun/glm53-flash-cmp170hx-exl3 by Flun on GitHub, posted by @wingrimez_40467, 2026-10-20

Measured numbers

The repo publishes no tok/s of its own.

68x had an agent run it on two of his cards: about 60 tok/s without DFlash. Source: @68x0, 2026-10-20
Source: @68x0, 2026-10-20
Wingrime scaled it to three cards with an agent: about 60 tok/s in TP mode with the full context size, "but engine is suboptimal". Three cards buy context, not speed. Source: @wingrimez_40467, 2026-10-20
Nestaa51 measured this layer split on his x16 pair as the baseline for his TP build: 96.2 / 34.2 tok/s on easy / hard text, against 123.6 / 44.3 for TP (+28%); the layer split was 2 to 5% slower at prefill. Source: repo github.com/alexvnesta/glm53-flash-cmp170hx-tp by alexvnesta on GitHub, posted by @nestaa51, 2026-10-12

Known issues

One active request at a time; others wait up to 120 s. No real concurrency.
The Q8 cache changes numerics; no bit-exactness against FP16 is claimed.
No multimodal input and no JSON-schema output.
The drafter conversion is a one-time job of tens of minutes; the DFlash2 drafter is CC BY-NC-ND 4.0, so converted copies must not be shared.
The 1M-context goal is still open.
Miséricorde's general warning: EXL3 is slow on CMPs (see Quantization on the 170HX). Source: @wakeupmbrc1, 2026-10-14

Source: repo github.com/Flun/glm53-flash-cmp170hx-exl3 by Flun on GitHub, posted by @wingrimez_40467, 2026-10-20

Sources

Repo: Flun/glm53-flash-cmp170hx-exl3 (MIT), last commit 2026-10-07; README.ko.md is the Korean original. Posted on Discord by Wingrime. Source: @wingrimez_40467, 2026-10-20

Prerequisites

Steps

Profiles

What makes it work

Measured numbers

Known issues

Related

Sources