Engines

The CMP 170HX is a GA100 die with compute capability 8.0 (sm_80, the A100 family), 64 GB of HBM after the unlock (Driver unlock), no NVLink and PCIe Gen2 at best (Unlocks). Engines built for Hopper or Blackwell kernels do not run on it as shipped, and engines that assume NVLink lose time in all-reduce. This page lists what runs, which recipes use it, and the flags and traps that matter. For the per-model setups see Recipes.

Summary

Engine	What it is	sm_80 status	Used by
Morrowmake/vllm-cmp170hx	vLLM with an sm_80 plugin and a short patch queue	the reference CMP engine	GLM 5.3 Flash on 4x 8 GB, DeepSeek V4.1 Flash on 4x 8 GB (Morrowmake), Qwen3.8 Flash Next on 2x 8 GB (Morrowmake EXL3), the ElectroHobo recipes, CurtisAccelerate, arvkmr
wtdcode/vllm-backport	vLLM fork for frontier MoE models on Ampere and Ada (A100, A6000, 3090), images lazymio/vllm-backport:*-sm80	runs; not CMP-specific	Qwen3.8 Flash Next on 2x 8 GB (mannydelrio PP2), matstatman's 4-card Qwen, Ivan's full GLM 5.3
HalfVulpes/Deepinfer-SM80	DeepInfer ideas ported to sm_80 as two vLLM patches + 30 drop-ins	measured on 74-SM CMP cards	DeepSeek V4.1 Flash on 6x 8 GB (Deepinfer-SM80 TP2 x PP3), DeepSeek V4.1 Flash on 8x 8 GB (Deepinfer-SM80 TP4 x PP2), the GNmac image
Patched upstream vLLM	per-recipe patch sets on a pinned commit or PR	works where patched	Zanooda (PR #56201 + 20 patches), iIIusi0n (a5a30471 + 13-file patch), clee/vllm-sm80, vektorprime (PR #53899), DUFRelic (v0.30.0 + zankich INT4-PLE plugin), wgkbfcp (79a7108d)
ExLlamaV3 + TabbyAPI	EXL3 quants, layer split or TP	runs; compute-bound on sm_80	GLM 5.3 Flash on 2x 8 GB (ExLlamaV3 TP2), GLM 5.3 Flash on 2x 8 GB (ExLlamaV3 layer split), DeepSeek V4.1 Flash on 4x 8 GB plus RTX PRO 6000 (ExLlamaV3 TP5)
Project Maya / Strata	expert tiering across VRAM, RAM and NVMe (Strata and llama.cpp lineage)	runs (compute capability 7.0+)	GLM 5.3 Flash on 2x 8 GB (Project Maya), DeepSeek V4.1 Flash on 2x 8 GB (Project Maya)
llama.cpp	GGUF	runs, but slow on these cards	not recommended
sglang		FP8 Qwen 27B reported	see Qwen3.8 27B on 1x 8 GB (W4A16 vLLM)
DeepInfer (upstream), TokenSpeed, 1Cat-vLLM, EAGLE-4		not runnable or unmeasured on sm_80	see below

Morrowmake/vllm-cmp170hx

The engine the community recipes pin. Since late October 2026 it is not a classic fork but a vllm-sm80 plugin (Python package vllm_sm80 with sm_80 kernels, attention backends, EXL3 and Marlin extensions, model classes and a launcher) plus a short patch queue on an exact upstream vLLM commit: an engine is three pins (upstream commit, queue commit, plugin commit). Upstream vLLM's sparse-attention path for GLM-5.3 needs Hopper; this fork adds the Ampere kernels that make it run at all.

• Branches: plugin (default), queue/330c5bffee (development line) and queue/165213f911 (newest tested upstream), main (plain upstream mirror), ampere (frozen at 12d6ed3ce0, no more changes). The ampere-glm53 branch is gone. Discord posts before late October describe the old single-tree ampere branch.
• Recipe pins are tagged and never deleted: glm53-recipe-1.3.0 to 1.8.0 (1.8.0 = 819146023f), qwen38-recipe-1.0.0 (18305948ac), dsv41-flash-recipe-1.0.0 (a0c57b5237). Run it through the recipes' container images; that is the supported way.
• Build from source (no GPU needed): Python 3.12, torch 2.14.0, CUDA 12.8+; check out the queue branch and install with VLLM_USE_PRECOMPILED=1 VLLM_PRECOMPILED_WHEEL_COMMIT=<upstream commit>, install the plugin worktree with --no-deps, then python csrc/exl3/build_standalone.py --out vllm_sm80 for the EXL3 kernels (optionally VLLM_AMPERE_MARLIN_BUILD=1 python csrc/ampere_marlin/build_standalone.py). vllm_sm80/guard.py stops vLLM at start if the plugin is missing.
• EXL3 inside vLLM: --quantization exl3 --load-format exl3 --dtype bfloat16, integer rates 1 to 8 bits (1.5 / 2.5 / 3.5 need the mul1 codebook), TP cuts must keep 128-channel transform blocks, whole-expert expert parallel; PP other than 1 and LoRA are rejected. Tensor-core paths behind VLLM_EXL3_MOE_V2, VLLM_EXL3_MOE_PREFILL_TC, VLLM_DEEPSEEK_V41_MOE_PREFILL_TC and others. Several dense paths are validated only for 74-SM cards at TP2.
• Speculative decoding: DFlash2 and DSpark speculators, adaptive_k (draft count by load), shard_argmax, pp_draft_tail. PCIe-only helpers: host_shm_all_reduce, ce_ring_all_reduce, custom_all_reduce_flags, a pcie_p2p gate. A tuning table vllm_sm80/tuning/nvidia-cmp-170hx.toml. 111 fork environment flags, all off by default; the recipes turn on what they need.
• Contribution rules: sm_80 kernels and model code go to plugin; hooks in upstream files go to the newest queue/* branch, one commit per file; recipes for other models or layouts belong in your own repo, must pin the fork's commits, tag the repo cmp170hx and keep the NOTICE.
• License: Apache-2.0 (LICENSE added 2026-10-29); the plugin also carries MIT code adapted from ExLlamaV3.

Source: repo github.com/Morrowmake/vllm-cmp170hx by Morrowmake on GitHub, posted by @voidgiraffe, 2026-10-15
Source: @jjb01957, 2026-10-24

Morrowmake on the plugin layout (2026-10-15/16): most of two weeks went into repeatable processes and into moving all his custom code behind vLLM plugins, so rebasing onto newer vLLM stops being painful; the GLM update and a first Qwen3.8 Flash Next recipe were released with it. He added a community-recipes section to the fork for setups adapted from his, agreed to matstatman's request for a Docker image that contains everything, and said x4 links will hurt prefill (untested) with a PP2 Qwen version to come. Source: @voidgiraffe, 2026-10-15
Source: @matstatman, 2026-10-15
Source: @voidgiraffe, 2026-10-16
Source: @voidgiraffe, 2026-10-16

wtdcode/vllm-backport

A vLLM fork for frontier MoE models on older Ampere and Ada cards, with prebuilt single-arch Docker images (lazymio/vllm-backport:latest-sm80, pinned :v0.13.0-sm80; no FA3 or Hopper kernels) and a paired LMCache fork. It is not CMP-specific but is the base of mannydelrio's two-card recipe and the alternative engine lineage to Morrowmake's.

• Models by fork version: DeepSeek-V4-Flash-0731 native FP4 (0.6.0+), Qwen3.8-27B BF16 / AWQ W4A16 (0.8.0+), Qwen3.8-Flash-Next FP8 + AWQ W4A16 (0.9.0+), GLM-5.3-Flash FP8 + AWQ W4A16 (0.11.2+), DeepSeek-V4.1-Flash native FP4 with Engram (0.13.0+). It publishes wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16 and wtdcode/GLM-5.3-Flash-AWQ-W4A16.
• FULL_AND_PIECEWISE CUDA graphs on Ampere need NCCL_ALGO=Ring NCCL_PROTO=Simple and preferably --disable-custom-all-reduce: the "crash on Ampere" was NCCL's size-adaptive algorithm switching under graph replay. Bound cudagraph_capture_sizes; FULL graphs can cost more than 800 MB per GPU.
• Qwen3.8-Flash-Next: VLLM_PLE_CPU_OFFLOAD=1 keeps the n-gram embedding in pinned host RAM; --enable-expert-parallel is required (plain TP makes the 640-wide expert intermediate 160 per rank); --prefix-match-unit=16 is effectively mandatory, otherwise nothing under 1,600 tokens is reused.
• DeepSeek V4.1 Flash: --engram-config '{"cpu_offload": true}', --tokenizer-mode deepseek_v41, DSpark K=5. Agentic clients should pass reasoning_effort from the first turn or sessions drift into reasoning loops.

Source: repo github.com/wtdcode/vllm-backport by wtdcode on GitHub, posted by @matstatman, 2026-10-16

• matstatman ran Qwen3.8 Flash Next on four cards with this engine and only 32 GB of host RAM. His compose file, flags and numbers are on Qwen3.8 Flash Next on 4x 8 GB. Source: @matstatman, 2026-10-16
• Ivan's lab: the full 753B GLM-5.3 (an abliterated NVFP4-expert / INT8 requant) runs on eight cards with this engine at TP2 x PP4 with P2P and MTP, 524K context and a 570K-token KV pool, but only with a one-line worker fix for the pipeline hand-off bug (below): 77 / 76 / 48 tok/s single (code / structured / prose), 172 / 189 / 146 at 8 users, prefill 1,650 tok/s on a 17k prompt, 68% draft acceptance. The int8 KV tile beat plain FP8 KV by 10% single-stream at the same quality. Schema-constrained output (named tool_choice, json_schema) returns HTTP 500 with MTP on, because xgrammar rejects the draft tokens; serve that traffic without speculation. GLM-5.3-Flash checkpoints with a dense layer 0 fail to load on it (KeyError on layers.0.mlp.gate_up_proj.weight). Source: @darkmaniac7, 2026-10-28 (lab notes)

Other vLLM patch sets

• arvkmr/vllm-170hx-glm5: fail-closed source patches on Morrowmake's frozen ampere-glm53 commit that port GLM-5.3's DSA path to sm_80 (packed fp8_ds_mla KV reader, software e4m3, Triton indexer, canonical top-k, DFlash over PP, top-k PP relay). See GLM 5.3 on 8x 8 GB (vllm-170hx-glm5). Source: repo github.com/arvkmr/vllm-170hx-glm5 by arvkmr on GitHub, posted by @reala7096, 2026-10-08
• clee/vllm-sm80: pure-Python fork on a pinned vLLM nightly with FP8 KV on sm_80 (software e4m3 decode), PLE under PP>1 and an NVMe-staged PLE reader; its PP KV fix is also used by mannydelrio. See Qwen3.8 Flash Next on 3x 8 GB (clee PP3). Source: repo github.com/clee/qwen3.8-flash-next-3x170hx by clee on GitHub, posted by @c133, 2026-10-04
• Zanooda: a 20-patch series on vLLM PR #56201 plus haosdent/vllm's dsv4-flash-a100 sm_80 layer, with an sm_80-only image. See DeepSeek V4.1 Flash on 8x 8 GB (Zanooda PP8). Source: @gnasherrr, 2026-10-17
• iIIusi0n, vektorprime, DUFRelic (zankich INT4-PLE plugin), wgkbfcp: see their recipe pages under Recipes.
• eXile has a custom vLLM build aimed at the 170HX running Q3 Qwen 3.8 Flash Next on one 64 GB card; HumanEval matched his Strata results. Contested Others doubted the prefill figure (xtj7 suspected cache hits); no answer was posted. Source: @exile_1ck, 2026-10-15
  Source: @xtj7, 2026-10-15

DeepInfer and Deepinfer-SM80

deepseek-ai/DeepInfer is DeepSeek's open-sourced inference framework for DeepSeek-V4.1-Flash (DSpark, SWA bounded replay, batch-invariant modes). Upstream supports only NVIDIA SM100 and Huawei Ascend 950, so it does not run on the CMP as released. Source: repo github.com/deepseek-ai/DeepInfer by deepseek-ai on GitHub, posted by @68x0, 2026-10-20

HalfVulpes/Deepinfer-SM80 ports its ideas to sm_80 as two vLLM patches on the unmerged PR #56201 commit (79a7108d) plus 30 drop-in Python, Triton and TileLang modules. Build with TORCH_CUDA_ARCH_LIST="8.0 8.6", CUDA 13.0, Python 3.12, GCC 13+, torch 2.13.0+cu130; check with scripts/check_dropin_sync.py ("30 same, 0 diff, 0 missing") and scripts/verify_env.py. Kernel speedups on 74-SM sm_80: _combine_topk_swa_indices_kernel 22.17x (bit-exact), dequantize_and_gather_k_cache 4.02x at 16K and 7.64x at 48K, mHC fused post/pre 2.89x, sparse-attention decode 2.08 to 2.14x, sparse prefill 1.17x. 108-SM parts (A100, A800) gain less because the tunings are wave-aligned for 74 SMs. The patches do not apply to newer vLLM main. Source: @segmond, 2026-10-20
Source: repo github.com/HalfVulpes/Deepinfer-SM80 by HalfVulpes on GitHub, posted by @reala7096, 2026-10-20

ExLlamaV3, TabbyAPI and EXL3

• turboderp's ExLlamaV3 loads EXL3 quants (for example turboderp/GLM-5.3-Flash-exl3 at 3.05 bpw) and runs on sm_80 when built with TORCH_CUDA_ARCH_LIST=8.0. The two-card GLM recipes use it with layer split or tensor parallel over NCCL; anoane serves DeepSeek V4.1 Flash through an ExLlamaV3 fork plus a TabbyAPI fork (TabbyAPI is AGPL-3.0). vcruz305/vllm-exl3 (AGPL-3.0) brings EXL3 into vLLM and is part of the GNmac image. Source: repo github.com/Flun/glm53-flash-cmp170hx-exl3 by Flun on GitHub, posted by @wingrimez_40467, 2026-10-20
  Source: repo github.com/anoane/09-deepseek-v41-flash-1017-exl3-tp5-1m by anoane on GitHub, posted by @anon33anon, 2026-10-20
• Contested EXL3 speed on CMPs. Miséricorde: EXL3 is very slow on CMP because the sm_80 dequant path is heavily compute-bound, using only 10 to 20% of HBM bandwidth; do not run EXL3 quants on CMPs. Takka: small-batch decode should still be bandwidth-bound; the card has only about 12 TFLOPs of INT32. Morrowmake reached about 150 tok/s on Qwen Flash Next EXL3 5.05 bpw, then said W4A16 is probably the better choice. Miséricorde also says EXL3 is acceptable for DeepSeek V4.1 Flash because only about 30% of its decode is EXL3 work (about 95% for GLM). Source: @wakeupmbrc1, 2026-10-14
  Source: @wakeupmbrc1, 2026-10-14
  Source: @spectakkular, 2026-10-15
  Source: @voidgiraffe, 2026-10-16
  Source: @wakeupmbrc1, 2026-10-17
• Why the EXL3-expert DeepSeek V4.1 Flash ports run 2.7 to 3x faster than the PP8 recipes (Ivan's analysis of the repos' docs and Discord numbers, 2026-10-27): the layout, not EXL3. 2.9 bpw experts plus FP8 for the rest fit four 64 GB cards with a 1.2 to 1.4M-token KV pool, so the ports run TP4 + EP4 on x16 with a P2P all-reduce and every card works on every token; the official MXFP4 model (about 286 GiB without Engram) needs eight cards in PP8, where seven of eight idle per token for one user. GNmac's own PP4 column on the same weights (118 / 91 / 63) matches Zanooda's PP8 (117 / 61). The vLLM ports (Morrowmake's moe_v2.cu and GNmac's coop tile, both from CurtisAccelerate's TC-MoE port) decode trellis tiles straight into FP16 tensor-core operands, but decode stays ALU-bound: about 210 GB/s per card, roughly 15% of HBM, from GNmac's step profile. With the layout held equal EXL3 loses 15 to 20% to native FP4 W4A16 (reala on 8 GPUs: 117.1 against 135 to 139 code decode). Speculative verify is nearly free for a decode-bound kernel (+0.6 to 1.0 ms per extra row against 4 ms for the first), which is why structured and code output beat prose by so much; prefill is where the EXL3 cost shows (2.4 to 3.8k against PP8's 4.5 to 6k). Source: @darkmaniac7, 2026-10-27 (lab notes)
• tekkenz ran turboderp's Qwen3.8-Flash-Next EXL3 at 5.05 bpw (0.0040 KLD from BF16) on one card: 69 to 82 tok/s decode, about 1,000 tok/s prefill. How it fits one card without RAM offload was asked and not answered. Source: @tekkenz, 2026-10-06
• Ryko serves GLM-5.3-Flash 4-bit EXL3 on ExLlamaV3 across two 64 GB CMPs, a 40 GB card and an RTX 3080 20 GB (188 GB total) with 768k context at 16-bit KV: about 70 tok/s decode, a 100k-token prefill at 3,500 to 3,600 tok/s, after letting an AI agent optimize ExLlamaV3 for the hardware. Source: @vektor_prime, 2026-10-14
• Mountains runs the full GLM 5.3 EXL3 3.05 bpw across four 8 GB CMPs plus an RTX PRO 6000 with pipeline parallel at 30 tok/s; it did not work out of the box with PP and mixed architectures. Source: @mountains1605, 2026-10-12
  Source: @mountains1605, 2026-10-12

llama.cpp

Runs, but it is the slow option on these cards; use vLLM or sglang with INT4/INT8 weights instead.

• ikyo's Mac Studio beat a single CMP in most tests on Qwen 27B Q8 with llama.cpp; the community said llama.cpp is very suboptimal on CMPs. Source: @ikyo, 2026-10-26
  Source: @lia_oc, 2026-10-26
• Tokukarin cannot get past 60 tok/s on Qwen Flash Next at 4.27 bpw on one card even with MTP; he will not go lower because 3.84 bpw degraded multilingual output. Source: @tokukarin, 2026-10-27
• llama.cpp can put mmproj and the speculative draft on a separate device, so a fifth card is still useful while the main model stays on four. Source: @da6b, 2026-10-21
• Comparison point: xanthe_ gets 88 tok/s on Qwen 3.6 across two V100s with llama.cpp built from source at 5-bit. Source: @xanthefin, 2026-10-18
• PixelML measured a single-kernel decode engine at 70.6 tok/s against llama.cpp 53.4 on Qwen3.8-Flash-Next UD-Q3_K_XL on one card; the 26.8 GiB PLE table must sit on local disk (29 tok/s from a network mount). Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20

Strata and Project Maya

Strata is a consumer local-LLM app with its own engine; it lists no CMP support but people run it. Project Maya forked it (plus llama.cpp) to tier MoE experts across VRAM, RAM and NVMe.

• pcsv: Strata works well with CMPs, 100+ tok/s with default settings; forks (mw00/project-maya for DeepSeek 4.1 Flash, ro99/Strata for Kimi K3) proved larger models. Source: @pcsv., 2026-10-25
• acedogblast: more than 60 tok/s on Qwen 3.8 Flash Next on one card with Strata (Unsloth QD-4K_XL, FP16 KV). Source: @acedogblast, 2026-10-06
• xtj7: Strata on one card with quad-channel DDR4 gives about 90 tok/s decode and 1,600 tok/s prefill at IQ2, without an x16 mod. Source: @xtj7, 2026-10-15
• eXile (one 10 GB-variant card at 40 GB, 14600K, 128 GB DDR4): moving from x4 to x16 mostly improved prefill; his modified Strata build beats stock (110+ against under 100). Theory Miséricorde expects ECC DDR4 server memory to double Strata speed. Source: @exile_1ck, 2026-10-07
  Source: @exile_1ck, 2026-10-07
  Source: @wakeupmbrc1, 2026-10-07
• Project Maya pages: GLM 5.3 Flash on 2x 8 GB (Project Maya), DeepSeek V4.1 Flash on 2x 8 GB (Project Maya).

Speculative decoding

• DFlash2 (inco.ai drafters for GLM-5.3-Flash and Qwen3.8-Flash-Next) is the default in the Morrowmake GLM recipe and the ElectroHobo Qwen recipes. The GLM drafter is CC BY-NC-ND 4.0 (non-commercial); the Qwen one is Apache-2.0. Miséricorde confirmed the Qwen drafter helps. Source: @wakeupmbrc1, 2026-10-20
• DSpark is DeepSeek V4.1 Flash's own drafter, built into the checkpoint. Acceptance runs about 5.1 tokens per step on code and about 2.9 on reasoning text (reala), which leaves reasoning at 55-75 tok/s on six to eight cards. On the GNmac port the draft count must be 5: 6 or 7 do not start, and 3 or 4 boot but every request fails, so that recipe narrows verification with DSV41_DYNK instead. On Ivan's mixed bench DSpark accepted 2.1 of 5 draft tokens per step (2.6x decode), and a fine-tuned FP8 DSpark refit measured identical to stock (2.14 against 2.13). More from Ivan's notes: the drafter sits in the checkpoint under mtp.0..2 (three stages, about 7.9 GB, block size 5) and the patched vLLM refuses method:"mtp" on V4.1. num_speculative_tokens must be at most 5 or a multiple of 5 (7 is refused; PixelML's k=10 run was slower, 4.01 against 4.40 tokens per step at k=5). Upstream adaptive verification is not supported with pipeline parallel. Acceptance elsewhere: Zanooda PP8 about 5.9 tokens per step on repetitive output and 3.9 on prose; PixelML's TP4 + EP4 EXL3 build 67.9% (4.40 per step), and 25.4 tok/s without a drafter against 99.5 / 48.9 with it. No other trained drafter targets V4.1: the EAGLE3 and DFlash heads are for V4 Flash (hidden 4096, 43 layers; V4.1 is 5120 and 40). Drop-in refits: SubconsciousDev/deepseek-v4.1-flash-1017-dspark (FP8, same tensor names) and ursuciprian/DeepSeek-V4.1-Flash-1017-DSpark-D1 (EXL3 overlay, author claims 6.6 to 11.6% on prose on other hardware). n-gram and suffix decoding exist only on the GNmac V1-runner path and replace DSpark, so they should lose on prose. Source: @reala7096, 2026-10-22
  Source: @darkmaniac7, 2026-10-27 (lab notes)
  Source: @darkmaniac7, 2026-10-28 (lab notes)
• MTP heads ship with Qwen3.8 and GLM-5.3 checkpoints. On Qwen3.8 27B matstatman recommends MTP with k=3 over DFlash. On Qwen Flash Next DFlash reached only 14% acceptance at the 4th token for Sojun, and Mountains never got speculative decoding working for it on the 170HX (it worked for the 27B). The ElectroHobo recipes now ship a DFlash2 drafter for it. Source: @sojun80, 2026-10-07
  Source: @mountains1605, 2026-10-08
  Source: @matstatman, 2026-10-17
• Theory EAGLE-4 (eagle-hq): its Triton kernel tier admits sm_80 and cibernox flagged it as faster than MTP and DSpark2, but nothing has been measured on Ampere or a CMP, it needs patched engine wheels, and it has drafts only for Qwen3.8-27B and Nemotron. Source-available license, not open source. Source: repo github.com/eagle-hq/EAGLE-4 by eagle-hq on GitHub, posted by @cibernox, 2026-10-20

Mixing CMP and newer GPUs in one job

Ivan runs Morrowmake's engine across CMP 170HX (sm_80) and RTX PRO 6000 Blackwell (sm_120) cards in one vLLM job with pipeline parallel, CMPs first and Blackwells as the last stage. It needed three small Python patches (resolve the device capability per rank instead of from device 0, pick the MLA backend on sm_120 without the FP8 KV path, build the host-staged all-reduce only on sm_80 TP groups) and four launch settings, each found by a failed boot: --additional-config {"kda_prefill_backend":"triton"}, CUTE_DSL_ARCH=sm_120a, --kernel-config {"enable_flashinfer_autotune":false} (autotune on one stage deadlocked the others), and at least one MLA layer on every stage. Results for GLM 5.3 Flash are on GLM 5.3 Flash on 4x 8 GB#With other GPUs in the same job. The full GLM 5.3 does not run this way on the Morrowmake fork: its DSA indexer path needs DeepGEMM kernels that exist only for Hopper and Blackwell, and its KV writers use an FP8 type Triton refuses below sm_89; wtdcode/vllm-backport already has the sm_80 pieces but no sm_120 build. Theory Ivan estimates a port at 2 to 4 days. Source: @darkmaniac7, 2026-10-19 (lab notes)

Load times

Loading 350 to 380 GB models takes under 5 minutes from NVMe (impactic), about 20 minutes from SATA (matstatman) and about 8 minutes at Gen2 x4 (Mountains). First starts also compile kernels: 5 to 22 minutes depending on the recipe. Source: @darkmaniac7, 2026-10-20 (lab notes)
Source: @mountains1605, 2026-10-27

Engine-level known issues

• Kernel launch overhead on the n-gram models. Qwen3.8 Flash Next launches about 2,000 small kernels per step: decode compute is only 1.5 to 1.8 ms of a 10 to 11 ms round, so the model is cheap at high concurrency on big GPUs but ineffective at low concurrency on CMPs; hand-built megakernels gave 20 to 25%. Source: @wakeupmbrc1, 2026-10-13
  Source: @wakeupmbrc1, 2026-10-17
• Prefill stalls decode in vLLM on PP setups: per-stream speed drops with concurrency because decode stops during prefill; fair prefill slicing (Morrowmake, ElectroHobo) mitigates it. Source: @ob88594, 2026-10-07
• Contested Custom all-reduce and P2P on Gen2. Results differ by host and model: on Ivan's EPYC box custom all-reduce cost GLM-5.3-Flash TP4 about 40% decode but helped Qwen Flash Next TP2 by 17 to 25%; PixelML saw P2P hurt GLM TP4 on PLX + Broadwell; ElectroHobo saw P2P off cost MiMo TP4 40% on PLX + Broadwell. A/B it per recipe. A hang at init can be stale build locks in a copied compile cache. Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
  Source: @darkmaniac7, 2026-10-28 (lab notes)
• Pipeline hand-off bug at TP>1 with PP (GLM-5.x DSA models, vLLM PR #60477): a non-last pipeline stage sends an un-reduced partial sum, corrupting the first layer of the next stage at TP2 x PP4. Symptoms in Ivan's runs: about 3 points lower teacher-forced agreement, 32k-token degeneration loops and reasoning that eats the budget on long prompts. The one-line worker fix costs no speed. A second, separate bug (vLLM issue #63170, draft PR #59961): GLM-5.x shared-indexer layers at a stage start read stale top-k indices, so choose a PP split that starts every stage on a full indexer layer. Source: @darkmaniac7, 2026-10-28 (lab notes)
• Agent clients and the reasoning field. On Schaka's 8-card host a vLLM fork emitted reasoning text only under a field named reasoning; most clients (opencode included) expect reasoning_content, so without a patch that emits both, the whole think block lands in the assistant content, fills the history and drives long agent sessions into a repetition loop. Set reasoning_effort on every request (max, not high: low and high rendered an empty thinking prefix that accumulates), and give each context size its own served model name so clients read the right context length. LMCache does not unify the KV spec on mixed attention + Mamba/GDN models. Journal: Schaka/170hx-journey. Source: @darkmaniac7, 2026-09-12 (lab notes)
• TP size must divide the attention head count. DeepSeek V4.1: TP 1, 2, 4 or 8 only; TP6 works only with wgkbfcp's virtual-head padding. Source: @greencandle9753, 2026-10-21 greencandle was confirming cibernox, who recalled TP=3 working on models whose head-related count divides by 3. A later post claimed vLLM needs a power-of-two GPU count "for all the llm models"; that conflicts with cibernox's TP=3 recollection and with greencandle's divisor rule. Source: @cibernox, 2026-10-21
  Source: @cibernox, 2026-10-21
  Source: @kill3again, 2026-10-22
• One bad card can look like an engine bug. A DeepSeek V4.1 Flash user once had garbage top-k indices that followed one physical card when cards were swapped; see also the zero-row card on GLM 5.3 on 8x 8 GB (vllm-170hx-glm5) and Troubleshooting. Source: @darkmaniac7, 2026-10-27 (lab notes)
• No tuned GEMM for the harvested die: training on one CMP ran almost 4x slower than a 5090 with the same nominal BF16 TFLOPs. Source: @programmerboi, 2026-10-16

Engines that do not run on sm_80 (yet)

• TokenSpeed (LightSeek Foundation): day-0 GLM-5.3-Flash and Qwen3.8-Flash-Next support, but the README documents no Ampere path. Its sm_80 pull request was closed as won't-fix. Source: repo github.com/lightseekorg/tokenspeed by lightseekorg on GitHub, posted by @c133, 2026-10-05
  Source: @darkmaniac7, 2026-10-20 (lab notes)
• 1Cat-vLLM: a V100 (sm_70) fork; a comparison point and technique source (FP8 KV emulation, NVFP4 on hardware without FP8 tensor cores), not a CMP engine. Source: repo github.com/1CatAI/1Cat-vLLM by 1CatAI on GitHub, posted by @xeax, 2026-10-24
• PixelML's negative results: NVFP4 GLM checkpoints need SM12x FlashInfer; EXL3 / TR3 4 bpw ships SM121-only kernels; the Ninfer sm_80 fork was 3.6x slower than vLLM with DFlash2. Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20

Related

Recipes · Quantization on the 170HX · Benchmarks · Unlocks · Driver unlock · Troubleshooting · Tools

Imported from the cmpunlocker Discord with credit; see Credits.