GLM 5.3 Flash on 2x 8 GB (ExLlamaV3 TP2)
More actions
| Reported 2 x 8 GB | |
| Quant | EXL3 3.05 bpw: turboderp/GLM-5.3-Flash-exl3; DFlash2 drafter converted to EXL3 6 bpw |
|---|---|
| Engine | ExLlamaV3 (patched, pinned) upstream dev 0f685aa9fd plus 9 patches; repo db01c256 |
| Topology | TP2 over NCCL (GPU P2P), DFlash2 K7 plus hybrid drafting |
| Context | 392,960 per request; 655,360-token Q8 KV pool shared by two requests |
| Decode | 107.67 tok/s |
| Prefill | 1533 tok/s |
| VRAM per card | about 62.9 / 62.3 GiB idle, peaks about 64.4 / 63.0 GiB with vision |
| Power cap | 180 W (author's cards); about 170 W with a modest memory OC in his Discord numbers |
| Driver | 610.57.04 |
| Unlock build | cmpunlocker (64 GiB); P2P-capable driver for the NCCL backend |
| Flags | scripts/run_tp.sh: -cq 8 -cs 655360 --context-cap 392960 -ndt 7 -chunk_size 2048 -ambs 2 -gs 62,63 -tp --experimental-dflash2-tp -tpb nccl --q8-staging --q8-head-gpu0 --concurrent-max-sequences 2; EXL3_MLA_KIDX_RING=32 |
| Tested | @nestaa51 on 2026-10-13 |
| Source | link |
| Confirmations | None yet. |
GLM-5.3-Flash entirely in HBM on two 64 GB cards, as the layer-split EXL3 build
(GLM 5.3 Flash on 2x 8 GB (ExLlamaV3 layer split)) extended with tensor parallel over NCCL and GPU P2P, vision
input, DFlash2 speculative decoding with hybrid drafting, two concurrent requests and idle sessions parked in host RAM.
Infobox decode is the author's single-request code figure. Mountains calls it the number one recipe for two cards and
listed it on localairecipes as "GLM-5.3-Flash EXL3 3.05bpw on 2x CMP 170HX TP2 DFlash2 K7 hybrid".
Source: repo github.com/alexvnesta/glm53-flash-cmp170hx-tp by alexvnesta on GitHub, posted by @nestaa51, 2026-10-12
Source: @mountains1605, 2026-10-13
Nestaa51's announcement: on his x16-modded pair of 8 GB cards, decode is up to about 28% faster than Flun's layer split, about 120 tok/s and 134-147 tok/s on code edits and JSON. Idle sessions' KV cache is parked in system RAM and two agents run at once (about 320K context each, 393K max per request), which is not vLLM-style concurrency. Most of his time went into the code, so real-world testing was still thin; the cards were limited to about 170 W with a modest memory overclock for these numbers. Source: @nestaa51, 2026-10-12
Source: @nestaa51, 2026-10-14
Prerequisites
- Two CMP 170HX that CUDA sees as 65,536 MiB each (Driver unlock, 8 GB card). The repo does not cover the unlock.
- x16 links (the x16 capacitor mod). The author measured PCIe Gen2 x16 at 6.8 GB/s host-to-device and 6.7 GB/s device-to-host per card.
- For the default NCCL backend, working GPU P2P (see Unlocks). If the P2P check fails, use
TP_BACKEND=native(host RAM). - About 128 GB of host RAM and 150 GB of disk. Test host: Threadripper 3970X, 128 GB RAM, driver 610.57.04, CUDA 13.3, PyTorch 2.14.1+cu130, Triton 3.8.0, Python 3.13.
bwrap(bubblewrap) for vision, orVISION=0.
Source: repo github.com/alexvnesta/glm53-flash-cmp170hx-tp by alexvnesta on GitHub, posted by @nestaa51, 2026-10-12
Steps
git clone https://github.com/alexvnesta/glm53-flash-cmp170hx-tp.git && cd glm53-flash-cmp170hx-tp
./scripts/setup_venv.sh # Python venv with pinned PyTorch and dependencies
./engine/apply_engine.sh # fetches ExLlamaV3 at the pinned commit, applies the patches, checks every file
./engine/build_native.sh # builds the CUDA extension for sm_80 (~15-25 min)
./scripts/get_models.sh # downloads the model (~125 GB) and the drafter, converts the drafter (~tens of minutes)
.venv/bin/python scripts/p2p_check.py # optional: checks GPU P2P before using the default NCCL backend
./scripts/run_tp.sh # starts the API on 127.0.0.1:8012 (first start takes 2-3 min)
- Engine: ExLlamaV3 upstream dev commit
0f685aa9fdplus 9 patches (MLA k_idx ring for the 655,360-token pool, fair scheduling with 512-token chunks, hybrid draft-id guard, SWA stash fill fix, FP16 recurrent state, allocation recovery, non-finite-logit failure, budget and first-token guards, cache profile identity).apply_engine.shverifies 970 files by sha256. - Model turboderp/GLM-5.3-Flash-exl3 3.05 bpw at
332ab457(about 116.6 GiB); drafter incoai/GLM-5.3-Flash-DFlash2 atbf582e4e, converted locally to EXL3 6 bpw. DFlash2 drafts 7 tokens; hybrid drafting uses 15 (HYBRID_DRAFT_TOKENS=15, faster than 8). - Env:
EXL3_MLA_KIDX_RING=32,EXL3_TP_SHARING_STRATEGY=file_descriptor,NCCL_SOCKET_IFNAME=lo. RAM tier 8 GiB, recurrent cache 5 GiB. API on 127.0.0.1:8012, no authentication.
Source: repo github.com/alexvnesta/glm53-flash-cmp170hx-tp by alexvnesta on GitHub, posted by @nestaa51, 2026-10-12
P2P check
scripts/p2p_check.py does 74 random-byte peer copies (128 KiB plus or minus one byte up to 32 MiB, both
directions, 0 mismatches required) and 84 all-reduce cases in FP32 and BF16, bit-exact. NCCL over P2P has been the
default since 2026-10-12 (NCCL 2.30.7, transport P2P/CUMEM). Against ExLlamaV3's native host-RAM backend: counting
decode 119.1 to 124.3 tok/s (+4.35%), hard text 42.75 to 45.52 (+6.48%), cold prefill at 8K/32K 1,360/1,385 to
1,477/1,545 tok/s (+10%), no extra memory.
Source: repo github.com/alexvnesta/glm53-flash-cmp170hx-tp by alexvnesta on GitHub, posted by @nestaa51, 2026-10-12
Measured numbers
Author's qualified single-request numbers, 2026-10-13:
| Test | Result |
|---|---|
| Counting decode | 129.38 tok/s |
| LRU reasoning | 46.77 tok/s |
| Short structured / code / prose | 96.71 / 107.67 / 47.83 tok/s |
| Cold prefill at 8K / 32K / 96K | 1,512 / 1,533 / 1,489 tok/s |
| Two-stream aggregate | 67.25 tok/s |
| Four parked 220K sessions restored from RAM | 3.787 to 4.003 s each |
- Against Flun's layer split on the same machine (2026-10-09): 96.2 / 34.2 tok/s on easy / hard text for the split, 123.6 / 44.3 for TP (+28%).
- 0xSero's k_hcfuse fused kernels were correct under TP but 1.5 to 4.2% slower, so they are off.
- Four-slot profile (
profiles/community-four-active): 155.5 / 143.7 tok/s aggregate at low / high effort on structured tasks; 82.7 / 96.3 / 101.2 tok/s (low / high / max effort, 4 agents, short prompt) on llm-inference-bench v0.8.0.
Source: repo github.com/alexvnesta/glm53-flash-cmp170hx-tp by alexvnesta on GitHub, posted by @nestaa51, 2026-10-12
Concurrency and KV
Two active requests share a 655,360-token Q8 KV pool: 320K + 320K or 393K + 255K fit, 330K + 330K does not. Dynamic RAM
KV paging (four 256K contexts, 117.0 / 130.6 tok/s) was not promoted after a 64 MiB TP cudaHostRegister
failure on restart; production keeps the four-slot GPU cache.
Source: repo github.com/alexvnesta/glm53-flash-cmp170hx-tp by alexvnesta on GitHub, posted by @nestaa51, 2026-10-12
Weightless profile
An optional profile ports the Weightless GLP-44 refusal-steering projector from
msuiche/weightless to ExLlamaV3 on these cards. It needs a separate
engine-weightless build (972-file manifest), gguf==0.17.1, FP16 recurrent state, and the gated
Hugging Face vector msuiche/GLM-5.3-Flash-abliterated-cyber-GLP-44 (file pinned by sha256
0ccce6b748f87da81505ff2bc3ca82429110940b6c0def60063d304785ccc00c, alpha 2.0, layers 1 to 44). The regular
launcher stays unsteered, and the author's own bounded checks show no broad quality or speed gain from steering.
Upstream, alpha 2.5 and above garbles the model.
Source: repo github.com/alexvnesta/glm53-flash-cmp170hx-tp by alexvnesta on GitHub, posted by @nestaa51, 2026-10-12
Source: repo github.com/msuiche/weightless by msuiche on GitHub, posted by @mountains1605, 2026-10-09
Known issues
- Keep the vision tower on GPU 0; one unexplained TP timeout was seen with it on GPU 1.
- A large new prompt slows an active session to about a third of its speed until its prefill finishes.
- Identical simultaneous requests can rarely differ in early tokens.
- 2026-10-14: a backtracking regex in the tool-argument parser could stall the HTTP loop (16,000 tags over 3 s); replaced by a forward-pass parser.
- Early release (author's badge: early, YMMV), one machine, all numbers author-measured.
- The DFlash2 drafter is CC BY-NC-ND 4.0: converted copies must not be shared.
Source: repo github.com/alexvnesta/glm53-flash-cmp170hx-tp by alexvnesta on GitHub, posted by @nestaa51, 2026-10-12
- EXL3 decode on CMPs is compute-bound; see Quantization on the 170HX. Source: @wakeupmbrc1, 2026-10-14
Related
Sources
- Repo: alexvnesta/glm53-flash-cmp170hx-tp (MIT), last commit 2026-10-17. Its
docs/DISCORD_*.txthold the exact texts the author posted. - Discord: Nestaa51's announcement 2026-10-12, Mountains' listing 2026-10-13.