Engines - Revision history

ImportBot: Import: Inference engines on the CMP 170HX (sm_80): vLLM forks (Morrowmake/vllm-cmp170hx, wtdcode/vllm-backport, arvkmr, clee and others), Deepinfer-SM80, ExLlamaV3 and TabbyAPI, llama.cpp, Strata and Project Maya, sglang, speculative decoding; from repos.json, Discord and Ivan's lab notes; sources: @voidgiraffe, @jjb01957, @matstatman, @darkmaniac7, @reala7096, @c133, @gnasherrr, @exile_1ck, @xtj7, @68x0, @segmond, @wingrimez_40467 (+20 more); GitHub 1CatAI/1Cat-vLLM, Flun/g

2026-10-30T10:02:06Z

Import: Inference engines on the CMP 170HX (sm_80): vLLM forks (Morrowmake/vllm-cmp170hx, wtdcode/vllm-backport, arvkmr, clee and others), Deepinfer-SM80, ExLlamaV3 and TabbyAPI, llama.cpp, Strata and Project Maya, sglang, speculative decoding; from repos.json, Discord and Ivan's lab notes; sources: @voidgiraffe, @jjb01957, @matstatman, @darkmaniac7, @reala7096, @c133, @gnasherrr, @exile_1ck, @xtj7, @68x0, @segmond, @wingrimez_40467 (+20 more); GitHub 1CatAI/1Cat-vLLM, Flun/g

← Older revision		Revision as of 10:02, 30 October 2026
Line 136:		Line 136:
	* '''Pipeline hand-off bug at TP>1 with PP''' (GLM-5.x DSA models, vLLM PR #60477): a non-last pipeline stage sends an un-reduced partial sum, corrupting the first layer of the next stage at TP2 x PP4. Symptoms in Ivan's runs: about 3 points lower teacher-forced agreement, 32k-token degeneration loops and reasoning that eats the budget on long prompts. The one-line worker fix costs no speed. A second, separate bug (vLLM issue #63170, draft PR #59961): GLM-5.x shared-indexer layers at a stage start read stale top-k indices, so choose a PP split that starts every stage on a full indexer layer. {{Source\|handle=darkmaniac7\|date=2026-10-28\|note=lab notes}}		* '''Pipeline hand-off bug at TP>1 with PP''' (GLM-5.x DSA models, vLLM PR #60477): a non-last pipeline stage sends an un-reduced partial sum, corrupting the first layer of the next stage at TP2 x PP4. Symptoms in Ivan's runs: about 3 points lower teacher-forced agreement, 32k-token degeneration loops and reasoning that eats the budget on long prompts. The one-line worker fix costs no speed. A second, separate bug (vLLM issue #63170, draft PR #59961): GLM-5.x shared-indexer layers at a stage start read stale top-k indices, so choose a PP split that starts every stage on a full indexer layer. {{Source\|handle=darkmaniac7\|date=2026-10-28\|note=lab notes}}
	* '''Agent clients and the reasoning field.''' On Schaka's 8-card host a vLLM fork emitted reasoning text only under a field named <code><nowiki>reasoning</nowiki></code>; most clients (opencode included) expect <code><nowiki>reasoning_content</nowiki></code>, so without a patch that emits both, the whole think block lands in the assistant content, fills the history and drives long agent sessions into a repetition loop. Set <code><nowiki>reasoning_effort</nowiki></code> on every request (max, not high: low and high rendered an empty thinking prefix that accumulates), and give each context size its own served model name so clients read the right context length. LMCache does not unify the KV spec on mixed attention + Mamba/GDN models. Journal: [https://github.com/Schaka/170hx-journey Schaka/170hx-journey]. {{Source\|handle=darkmaniac7\|date=2026-09-12\|note=lab notes}}		* '''Agent clients and the reasoning field.''' On Schaka's 8-card host a vLLM fork emitted reasoning text only under a field named <code><nowiki>reasoning</nowiki></code>; most clients (opencode included) expect <code><nowiki>reasoning_content</nowiki></code>, so without a patch that emits both, the whole think block lands in the assistant content, fills the history and drives long agent sessions into a repetition loop. Set <code><nowiki>reasoning_effort</nowiki></code> on every request (max, not high: low and high rendered an empty thinking prefix that accumulates), and give each context size its own served model name so clients read the right context length. LMCache does not unify the KV spec on mixed attention + Mamba/GDN models. Journal: [https://github.com/Schaka/170hx-journey Schaka/170hx-journey]. {{Source\|handle=darkmaniac7\|date=2026-09-12\|note=lab notes}}
	* '''TP size must divide the attention head count.''' DeepSeek V4.1: TP 1, 2, 4 or 8 only; TP6 works only with wgkbfcp's virtual-head padding. {{Source\|handle=greencandle9753\|date=2026-10-21\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562580021153894501}}		* '''TP size must divide the attention head count.''' DeepSeek V4.1: TP 1, 2, 4 or 8 only; TP6 works only with wgkbfcp's virtual-head padding. {{Source\|handle=greencandle9753\|date=2026-10-21\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562580021153894501}} greencandle was confirming cibernox, who recalled TP=3 working on models whose head-related count divides by 3. A later post claimed vLLM needs a power-of-two GPU count "for all the llm models"; that conflicts with cibernox's TP=3 recollection and with greencandle's divisor rule. {{Source\|handle=cibernox\|date=2026-10-21\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562550500224667649}}<br />{{Source\|handle=cibernox\|date=2026-10-21\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562550864952950919}}<br />{{Source\|handle=kill3again\|date=2026-10-22\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562847068383354981}}
	* '''One bad card can look like an engine bug.''' A DeepSeek V4.1 Flash user once had garbage top-k indices that followed one physical card when cards were swapped; see also the zero-row card on [[GLM 5.3 on 8x 8 GB (vllm-170hx-glm5)]] and [[Troubleshooting]]. {{Source\|handle=darkmaniac7\|date=2026-10-27\|note=lab notes}}		* '''One bad card can look like an engine bug.''' A DeepSeek V4.1 Flash user once had garbage top-k indices that followed one physical card when cards were swapped; see also the zero-row card on [[GLM 5.3 on 8x 8 GB (vllm-170hx-glm5)]] and [[Troubleshooting]]. {{Source\|handle=darkmaniac7\|date=2026-10-27\|note=lab notes}}
	* '''No tuned GEMM for the harvested die''': training on one CMP ran almost 4x slower than a 5090 with the same nominal BF16 TFLOPs. {{Source\|handle=programmerboi\|date=2026-10-16\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560682620012990505}}		* '''No tuned GEMM for the harvested die''': training on one CMP ran almost 4x slower than a 5090 with the same nominal BF16 TFLOPs. {{Source\|handle=programmerboi\|date=2026-10-16\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560682620012990505}}

ImportBot: Import: Inference engines on the CMP 170HX (sm_80): vLLM forks (Morrowmake/vllm-cmp170hx, wtdcode/vllm-backport, arvkmr, clee and others), Deepinfer-SM80, ExLlamaV3 and TabbyAPI, llama.cpp, Strata and Project Maya, sglang, speculative decoding; from repos.json, Discord and Ivan's lab notes; sources: @voidgiraffe, @jjb01957, @matstatman, @darkmaniac7, @reala7096, @c133, @gnasherrr, @exile_1ck, @xtj7, @68x0, @segmond, @wingrimez_40467 (+19 more); GitHub 1CatAI/1Cat-vLLM, Flun/g

2026-10-30T08:54:41Z

← Older revision		Revision as of 08:54, 30 October 2026
Line 43:		Line 43:
	{{Source\|handle=jjb01957\|date=2026-10-24\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1563675495542038650}}		{{Source\|handle=jjb01957\|date=2026-10-24\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1563675495542038650}}

	~~A longer write-up by @voidgiraffe~~ on the ~~fork's~~ plugin layout and community recipes ~~exists; see~~		Morrowmake on the plugin layout (2026-10-15/16): most of two weeks went into repeatable processes and into moving all his custom code behind vLLM plugins, so rebasing onto newer vLLM stops being painful; the GLM update and a first Qwen3.8 Flash Next recipe were released with it. He added a community-recipes section to the fork for setups adapted from his, agreed to matstatman's request for a Docker image that contains everything, and said x4 links will hurt prefill (untested) with a PP2 Qwen version to come. {{Source\|handle=voidgiraffe\|date=2026-10-15\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560412542096904213}}<br />{{Source\|handle=matstatman\|date=2026-10-15\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560414067095765084}}<br />{{Source\|handle=voidgiraffe\|date=2026-10-16\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560598988779167825}}<br />{{Source\|handle=voidgiraffe\|date=2026-10-16\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560579094666678283}}
	[https://discord.com/channels/1518634379604136178/1528826330454429706/~~1560411313212227727 Discord]~~.

	== wtdcode/vllm-backport ==		== wtdcode/vllm-backport ==
Line 56:		Line 55:
	* DeepSeek V4.1 Flash: <code><nowiki>--engram-config '{"cpu_offload": true}'</nowiki></code>, <code><nowiki>--tokenizer-mode deepseek_v41</nowiki></code>, DSpark K=5. Agentic clients should pass <code><nowiki>reasoning_effort</nowiki></code> from the first turn or sessions drift into reasoning loops.		* DeepSeek V4.1 Flash: <code><nowiki>--engram-config '{"cpu_offload": true}'</nowiki></code>, <code><nowiki>--tokenizer-mode deepseek_v41</nowiki></code>, DSpark K=5. Agentic clients should pass <code><nowiki>reasoning_effort</nowiki></code> from the first turn or sessions drift into reasoning loops.
	{{Source\|handle=matstatman\|date=2026-10-16\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560599247475314711\|repo=https://github.com/wtdcode/vllm-backport}}		{{Source\|handle=matstatman\|date=2026-10-16\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560599247475314711\|repo=https://github.com/wtdcode/vllm-backport}}
	* matstatman ran Qwen3.8 Flash Next on four cards with this engine and only 32 GB of host RAM. ~~A longer write~~-~~up by @matstatman exists; see [~~https://discord.com/channels/1518634379604136178/1528826330454429706/~~1560602001908170753 Discord].~~		* matstatman ran Qwen3.8 Flash Next on four cards with this engine and only 32 GB of host RAM. His compose file, flags and numbers are on [[Qwen3.8 Flash Next on 4x 8 GB]]. {{Source\|handle=matstatman\|date=2026-10-16\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560599247475314711}}
	* Ivan's lab: the full 753B GLM-5.3 (an abliterated NVFP4-expert / INT8 requant) runs on eight cards with this engine at TP2 x PP4 with P2P and MTP, 524K context and a 570K-token KV pool, but only with a one-line worker fix for the pipeline hand-off bug (below): 77 / 76 / 48 tok/s single (code / structured / prose), 172 / 189 / 146 at 8 users, prefill 1,650 tok/s on a 17k prompt, 68% draft acceptance. The int8 KV tile beat plain FP8 KV by 10% single-stream at the same quality. Schema-constrained output (named <code><nowiki>tool_choice</nowiki></code>, <code><nowiki>json_schema</nowiki></code>) returns HTTP 500 with MTP on, because xgrammar rejects the draft tokens; serve that traffic without speculation. GLM-5.3-Flash checkpoints with a dense layer 0 fail to load on it (KeyError on <code><nowiki>layers.0.mlp.gate_up_proj.weight</nowiki></code>). {{Source\|handle=darkmaniac7\|date=2026-10-28\|note=lab notes}}		* Ivan's lab: the full 753B GLM-5.3 (an abliterated NVFP4-expert / INT8 requant) runs on eight cards with this engine at TP2 x PP4 with P2P and MTP, 524K context and a 570K-token KV pool, but only with a one-line worker fix for the pipeline hand-off bug (below): 77 / 76 / 48 tok/s single (code / structured / prose), 172 / 189 / 146 at 8 users, prefill 1,650 tok/s on a 17k prompt, 68% draft acceptance. The int8 KV tile beat plain FP8 KV by 10% single-stream at the same quality. Schema-constrained output (named <code><nowiki>tool_choice</nowiki></code>, <code><nowiki>json_schema</nowiki></code>) returns HTTP 500 with MTP on, because xgrammar rejects the draft tokens; serve that traffic without speculation. GLM-5.3-Flash checkpoints with a dense layer 0 fail to load on it (KeyError on <code><nowiki>layers.0.mlp.gate_up_proj.weight</nowiki></code>). {{Source\|handle=darkmaniac7\|date=2026-10-28\|note=lab notes}}

Line 84:		Line 83:
	* turboderp's ExLlamaV3 loads EXL3 quants (for example turboderp/GLM-5.3-Flash-exl3 at 3.05 bpw) and runs on sm_80 when built with <code><nowiki>TORCH_CUDA_ARCH_LIST=8.0</nowiki></code>. The two-card GLM recipes use it with layer split or tensor parallel over NCCL; anoane serves DeepSeek V4.1 Flash through an ExLlamaV3 fork plus a TabbyAPI fork (TabbyAPI is AGPL-3.0). vcruz305/vllm-exl3 (AGPL-3.0) brings EXL3 into vLLM and is part of the GNmac image. {{Source\|handle=wingrimez_40467\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1562164200372179017\|repo=https://github.com/Flun/glm53-flash-cmp170hx-exl3}}<br />{{Source\|handle=anon33anon\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1561903682608697426\|repo=https://github.com/anoane/09-deepseek-v41-flash-1017-exl3-tp5-1m}}		* turboderp's ExLlamaV3 loads EXL3 quants (for example turboderp/GLM-5.3-Flash-exl3 at 3.05 bpw) and runs on sm_80 when built with <code><nowiki>TORCH_CUDA_ARCH_LIST=8.0</nowiki></code>. The two-card GLM recipes use it with layer split or tensor parallel over NCCL; anoane serves DeepSeek V4.1 Flash through an ExLlamaV3 fork plus a TabbyAPI fork (TabbyAPI is AGPL-3.0). vcruz305/vllm-exl3 (AGPL-3.0) brings EXL3 into vLLM and is part of the GNmac image. {{Source\|handle=wingrimez_40467\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1562164200372179017\|repo=https://github.com/Flun/glm53-flash-cmp170hx-exl3}}<br />{{Source\|handle=anon33anon\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1561903682608697426\|repo=https://github.com/anoane/09-deepseek-v41-flash-1017-exl3-tp5-1m}}
	* {{Status\|Contested}} EXL3 speed on CMPs. Miséricorde: EXL3 is very slow on CMP because the sm_80 dequant path is heavily compute-bound, using only 10 to 20% of HBM bandwidth; do not run EXL3 quants on CMPs. Takka: small-batch decode should still be bandwidth-bound; the card has only about 12 TFLOPs of INT32. Morrowmake reached about 150 tok/s on Qwen Flash Next EXL3 5.05 bpw, then said W4A16 is probably the better choice. Miséricorde also says EXL3 is acceptable for DeepSeek V4.1 Flash because only about 30% of its decode is EXL3 work (about 95% for GLM). {{Source\|handle=wakeupmbrc1\|date=2026-10-14\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1559867344367652865}}<br />{{Source\|handle=wakeupmbrc1\|date=2026-10-14\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1559796160724475996}}<br />{{Source\|handle=spectakkular\|date=2026-10-15\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560298253109432392}}<br />{{Source\|handle=voidgiraffe\|date=2026-10-16\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560599553210851400}}<br />{{Source\|handle=wakeupmbrc1\|date=2026-10-17\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1561137015167254579}}		* {{Status\|Contested}} EXL3 speed on CMPs. Miséricorde: EXL3 is very slow on CMP because the sm_80 dequant path is heavily compute-bound, using only 10 to 20% of HBM bandwidth; do not run EXL3 quants on CMPs. Takka: small-batch decode should still be bandwidth-bound; the card has only about 12 TFLOPs of INT32. Morrowmake reached about 150 tok/s on Qwen Flash Next EXL3 5.05 bpw, then said W4A16 is probably the better choice. Miséricorde also says EXL3 is acceptable for DeepSeek V4.1 Flash because only about 30% of its decode is EXL3 work (about 95% for GLM). {{Source\|handle=wakeupmbrc1\|date=2026-10-14\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1559867344367652865}}<br />{{Source\|handle=wakeupmbrc1\|date=2026-10-14\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1559796160724475996}}<br />{{Source\|handle=spectakkular\|date=2026-10-15\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560298253109432392}}<br />{{Source\|handle=voidgiraffe\|date=2026-10-16\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560599553210851400}}<br />{{Source\|handle=wakeupmbrc1\|date=2026-10-17\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1561137015167254579}}
			* Why the EXL3-expert DeepSeek V4.1 Flash ports run 2.7 to 3x faster than the PP8 recipes (Ivan's analysis of the repos' docs and Discord numbers, 2026-10-27): the layout, not EXL3. 2.9 bpw experts plus FP8 for the rest fit four 64 GB cards with a 1.2 to 1.4M-token KV pool, so the ports run TP4 + EP4 on x16 with a P2P all-reduce and every card works on every token; the official MXFP4 model (about 286 GiB without Engram) needs eight cards in PP8, where seven of eight idle per token for one user. GNmac's own PP4 column on the same weights (118 / 91 / 63) matches Zanooda's PP8 (117 / 61). The vLLM ports (Morrowmake's <code><nowiki>moe_v2.cu</nowiki></code> and GNmac's coop tile, both from CurtisAccelerate's TC-MoE port) decode trellis tiles straight into FP16 tensor-core operands, but decode stays ALU-bound: about 210 GB/s per card, roughly 15% of HBM, from GNmac's step profile. With the layout held equal EXL3 loses 15 to 20% to native FP4 W4A16 (reala on 8 GPUs: 117.1 against 135 to 139 code decode). Speculative verify is nearly free for a decode-bound kernel (+0.6 to 1.0 ms per extra row against 4 ms for the first), which is why structured and code output beat prose by so much; prefill is where the EXL3 cost shows (2.4 to 3.8k against PP8's 4.5 to 6k). {{Source\|handle=darkmaniac7\|date=2026-10-27\|note=lab notes}}
	* tekkenz ran turboderp's Qwen3.8-Flash-Next EXL3 at 5.05 bpw (0.0040 KLD from BF16) on one card: 69 to 82 tok/s decode, about 1,000 tok/s prefill. How it fits one card without RAM offload was asked and not answered. {{Source\|handle=tekkenz\|date=2026-10-06\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1557128719422853223}}		* tekkenz ran turboderp's Qwen3.8-Flash-Next EXL3 at 5.05 bpw (0.0040 KLD from BF16) on one card: 69 to 82 tok/s decode, about 1,000 tok/s prefill. How it fits one card without RAM offload was asked and not answered. {{Source\|handle=tekkenz\|date=2026-10-06\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1557128719422853223}}
	* Ryko serves GLM-5.3-Flash 4-bit EXL3 on ExLlamaV3 across two 64 GB CMPs, a 40 GB card and an RTX 3080 20 GB (188 GB total) with 768k context at 16-bit KV: about 70 tok/s decode, a 100k-token prefill at 3,500 to 3,600 tok/s, after letting an AI agent optimize ExLlamaV3 for the hardware. {{Source\|handle=vektor_prime\|date=2026-10-14\|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1560023669416402965}}		* Ryko serves GLM-5.3-Flash 4-bit EXL3 on ExLlamaV3 across two 64 GB CMPs, a 40 GB card and an RTX 3080 20 GB (188 GB total) with 768k context at 16-bit KV: about 70 tok/s decode, a 100k-token prefill at 3,500 to 3,600 tok/s, after letting an AI agent optimize ExLlamaV3 for the hardware. {{Source\|handle=vektor_prime\|date=2026-10-14\|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1560023669416402965}}
Line 107:		Line 107:
	== Speculative decoding ==		== Speculative decoding ==
	* '''DFlash2''' (inco.ai drafters for GLM-5.3-Flash and Qwen3.8-Flash-Next) is the default in the Morrowmake GLM recipe and the ElectroHobo Qwen recipes. The GLM drafter is CC BY-NC-ND 4.0 (non-commercial); the Qwen one is Apache-2.0. Miséricorde confirmed the Qwen drafter helps. {{Source\|handle=wakeupmbrc1\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1561924555486920825}}		* '''DFlash2''' (inco.ai drafters for GLM-5.3-Flash and Qwen3.8-Flash-Next) is the default in the Morrowmake GLM recipe and the ElectroHobo Qwen recipes. The GLM drafter is CC BY-NC-ND 4.0 (non-commercial); the Qwen one is Apache-2.0. Miséricorde confirmed the Qwen drafter helps. {{Source\|handle=wakeupmbrc1\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1561924555486920825}}
	* '''DSpark''' is DeepSeek V4.1 Flash's own drafter, built into the checkpoint. Acceptance runs about 5.1 tokens per step on code and about 2.9 on reasoning text (reala), which leaves reasoning at 55-75 tok/s on six to eight cards. On the GNmac port the draft count must be 5: 6 or 7 do not start, and 3 or 4 boot but every request fails, so that recipe narrows verification with <code><nowiki>DSV41_DYNK</nowiki></code> instead. On Ivan's mixed bench DSpark accepted 2.1 of 5 draft tokens per step (2.6x decode), and a fine-tuned FP8 DSpark refit measured identical to stock (2.14 against 2.13). {{Source\|handle=reala7096\|date=2026-10-22\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562917910165323808}}<br />{{Source\|handle=darkmaniac7\|date=2026-10-28\|note=lab notes}}		* '''DSpark''' is DeepSeek V4.1 Flash's own drafter, built into the checkpoint. Acceptance runs about 5.1 tokens per step on code and about 2.9 on reasoning text (reala), which leaves reasoning at 55-75 tok/s on six to eight cards. On the GNmac port the draft count must be 5: 6 or 7 do not start, and 3 or 4 boot but every request fails, so that recipe narrows verification with <code><nowiki>DSV41_DYNK</nowiki></code> instead. On Ivan's mixed bench DSpark accepted 2.1 of 5 draft tokens per step (2.6x decode), and a fine-tuned FP8 DSpark refit measured identical to stock (2.14 against 2.13). More from Ivan's notes: the drafter sits in the checkpoint under <code><nowiki>mtp.0..2</nowiki></code> (three stages, about 7.9 GB, block size 5) and the patched vLLM refuses <code><nowiki>method:"mtp"</nowiki></code> on V4.1. <code><nowiki>num_speculative_tokens</nowiki></code> must be at most 5 or a multiple of 5 (7 is refused; PixelML's k=10 run was slower, 4.01 against 4.40 tokens per step at k=5). Upstream adaptive verification is not supported with pipeline parallel. Acceptance elsewhere: Zanooda PP8 about 5.9 tokens per step on repetitive output and 3.9 on prose; PixelML's TP4 + EP4 EXL3 build 67.9% (4.40 per step), and 25.4 tok/s without a drafter against 99.5 / 48.9 with it. No other trained drafter targets V4.1: the EAGLE3 and DFlash heads are for V4 Flash (hidden 4096, 43 layers; V4.1 is 5120 and 40). Drop-in refits: [https://huggingface.co/SubconsciousDev/deepseek-v4.1-flash-1017-dspark SubconsciousDev/deepseek-v4.1-flash-1017-dspark] (FP8, same tensor names) and [https://huggingface.co/ursuciprian/DeepSeek-V4.1-Flash-1017-DSpark-D1 ursuciprian/DeepSeek-V4.1-Flash-1017-DSpark-D1] (EXL3 overlay, author claims 6.6 to 11.6% on prose on other hardware). n-gram and suffix decoding exist only on the GNmac V1-runner path and replace DSpark, so they should lose on prose. {{Source\|handle=reala7096\|date=2026-10-22\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562917910165323808}}<br />{{Source\|handle=darkmaniac7\|date=2026-10-27\|note=lab notes}}<br />{{Source\|handle=darkmaniac7\|date=2026-10-28\|note=lab notes}}
	* '''MTP''' heads ship with Qwen3.8 and GLM-5.3 checkpoints. On Qwen3.8 27B matstatman recommends MTP with k=3 over DFlash. On Qwen Flash Next DFlash reached only 14% acceptance at the 4th token for Sojun, and Mountains never got speculative decoding working for it on the 170HX (it worked for the 27B). The ElectroHobo recipes now ship a DFlash2 drafter for it. {{Source\|handle=sojun80\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557431093119033375}}<br />{{Source\|handle=mountains1605\|date=2026-10-08\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557890449186758737}}<br />{{Source\|handle=matstatman\|date=2026-10-17\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560828125863084103}}		* '''MTP''' heads ship with Qwen3.8 and GLM-5.3 checkpoints. On Qwen3.8 27B matstatman recommends MTP with k=3 over DFlash. On Qwen Flash Next DFlash reached only 14% acceptance at the 4th token for Sojun, and Mountains never got speculative decoding working for it on the 170HX (it worked for the 27B). The ElectroHobo recipes now ship a DFlash2 drafter for it. {{Source\|handle=sojun80\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557431093119033375}}<br />{{Source\|handle=mountains1605\|date=2026-10-08\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557890449186758737}}<br />{{Source\|handle=matstatman\|date=2026-10-17\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560828125863084103}}
	* {{Status\|Theory}} '''EAGLE-4''' (eagle-hq): its Triton kernel tier admits sm_80 and cibernox flagged it as faster than MTP and DSpark2, but nothing has been measured on Ampere or a CMP, it needs patched engine wheels, and it has drafts only for Qwen3.8-27B and Nemotron. Source-available license, not open source. {{Source\|handle=cibernox\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562247115562942555\|repo=https://github.com/eagle-hq/EAGLE-4}}		* {{Status\|Theory}} '''EAGLE-4''' (eagle-hq): its Triton kernel tier admits sm_80 and cibernox flagged it as faster than MTP and DSpark2, but nothing has been measured on Ampere or a CMP, it needs patched engine wheels, and it has drafts only for Qwen3.8-27B and Nemotron. Source-available license, not open source. {{Source\|handle=cibernox\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562247115562942555\|repo=https://github.com/eagle-hq/EAGLE-4}}
Line 135:		Line 135:
	* {{Status\|Contested}} '''Custom all-reduce and P2P on Gen2.''' Results differ by host and model: on Ivan's EPYC box custom all-reduce cost GLM-5.3-Flash TP4 about 40% decode but helped Qwen Flash Next TP2 by 17 to 25%; PixelML saw P2P hurt GLM TP4 on PLX + Broadwell; ElectroHobo saw P2P off cost MiMo TP4 40% on PLX + Broadwell. A/B it per recipe. A hang at init can be stale build locks in a copied compile cache. {{Source\|handle=c133\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127\|repo=https://github.com/PixelML/club-170hx}}<br />{{Source\|handle=darkmaniac7\|date=2026-10-28\|note=lab notes}}		* {{Status\|Contested}} '''Custom all-reduce and P2P on Gen2.''' Results differ by host and model: on Ivan's EPYC box custom all-reduce cost GLM-5.3-Flash TP4 about 40% decode but helped Qwen Flash Next TP2 by 17 to 25%; PixelML saw P2P hurt GLM TP4 on PLX + Broadwell; ElectroHobo saw P2P off cost MiMo TP4 40% on PLX + Broadwell. A/B it per recipe. A hang at init can be stale build locks in a copied compile cache. {{Source\|handle=c133\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127\|repo=https://github.com/PixelML/club-170hx}}<br />{{Source\|handle=darkmaniac7\|date=2026-10-28\|note=lab notes}}
	* '''Pipeline hand-off bug at TP>1 with PP''' (GLM-5.x DSA models, vLLM PR #60477): a non-last pipeline stage sends an un-reduced partial sum, corrupting the first layer of the next stage at TP2 x PP4. Symptoms in Ivan's runs: about 3 points lower teacher-forced agreement, 32k-token degeneration loops and reasoning that eats the budget on long prompts. The one-line worker fix costs no speed. A second, separate bug (vLLM issue #63170, draft PR #59961): GLM-5.x shared-indexer layers at a stage start read stale top-k indices, so choose a PP split that starts every stage on a full indexer layer. {{Source\|handle=darkmaniac7\|date=2026-10-28\|note=lab notes}}		* '''Pipeline hand-off bug at TP>1 with PP''' (GLM-5.x DSA models, vLLM PR #60477): a non-last pipeline stage sends an un-reduced partial sum, corrupting the first layer of the next stage at TP2 x PP4. Symptoms in Ivan's runs: about 3 points lower teacher-forced agreement, 32k-token degeneration loops and reasoning that eats the budget on long prompts. The one-line worker fix costs no speed. A second, separate bug (vLLM issue #63170, draft PR #59961): GLM-5.x shared-indexer layers at a stage start read stale top-k indices, so choose a PP split that starts every stage on a full indexer layer. {{Source\|handle=darkmaniac7\|date=2026-10-28\|note=lab notes}}
			* '''Agent clients and the reasoning field.''' On Schaka's 8-card host a vLLM fork emitted reasoning text only under a field named <code><nowiki>reasoning</nowiki></code>; most clients (opencode included) expect <code><nowiki>reasoning_content</nowiki></code>, so without a patch that emits both, the whole think block lands in the assistant content, fills the history and drives long agent sessions into a repetition loop. Set <code><nowiki>reasoning_effort</nowiki></code> on every request (max, not high: low and high rendered an empty thinking prefix that accumulates), and give each context size its own served model name so clients read the right context length. LMCache does not unify the KV spec on mixed attention + Mamba/GDN models. Journal: [https://github.com/Schaka/170hx-journey Schaka/170hx-journey]. {{Source\|handle=darkmaniac7\|date=2026-09-12\|note=lab notes}}
	* '''TP size must divide the attention head count.''' DeepSeek V4.1: TP 1, 2, 4 or 8 only; TP6 works only with wgkbfcp's virtual-head padding. {{Source\|handle=greencandle9753\|date=2026-10-21\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562580021153894501}}		* '''TP size must divide the attention head count.''' DeepSeek V4.1: TP 1, 2, 4 or 8 only; TP6 works only with wgkbfcp's virtual-head padding. {{Source\|handle=greencandle9753\|date=2026-10-21\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562580021153894501}}
			* '''One bad card can look like an engine bug.''' A DeepSeek V4.1 Flash user once had garbage top-k indices that followed one physical card when cards were swapped; see also the zero-row card on [[GLM 5.3 on 8x 8 GB (vllm-170hx-glm5)]] and [[Troubleshooting]]. {{Source\|handle=darkmaniac7\|date=2026-10-27\|note=lab notes}}
	* '''No tuned GEMM for the harvested die''': training on one CMP ran almost 4x slower than a 5090 with the same nominal BF16 TFLOPs. {{Source\|handle=programmerboi\|date=2026-10-16\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560682620012990505}}		* '''No tuned GEMM for the harvested die''': training on one CMP ran almost 4x slower than a 5090 with the same nominal BF16 TFLOPs. {{Source\|handle=programmerboi\|date=2026-10-16\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560682620012990505}}

ImportBot: Import: Inference engines on the CMP 170HX (sm_80): vLLM forks (Morrowmake/vllm-cmp170hx, wtdcode/vllm-backport, arvkmr, clee and others), Deepinfer-SM80, ExLlamaV3 and TabbyAPI, llama.cpp, Strata and Project Maya, sglang, speculative decoding; from repos.json, Discord and Ivan's lab notes; sources: @voidgiraffe, @jjb01957, @matstatman, @darkmaniac7, @reala7096, @c133, @gnasherrr, @exile_1ck, @xtj7, @68x0, @segmond, @wingrimez_40467 (+19 more); GitHub 1CatAI/1Cat-vLLM, Flun/g

2026-10-30T08:14:15Z

New page

The CMP 170HX is a GA100 die with compute capability 8.0 (sm_80, the A100 family), 64 GB of HBM after the unlock
([[Driver unlock]]), no NVLink and PCIe Gen2 at best ([[Unlocks]]). Engines built for Hopper or Blackwell kernels do not
run on it as shipped, and engines that assume NVLink lose time in all-reduce. This page lists what runs, which recipes
use it, and the flags and traps that matter. For the per-model setups see [[Recipes]].

== Summary ==
{| class="wikitable"
! Engine !! What it is !! sm_80 status !! Used by
|-
| [https://github.com/Morrowmake/vllm-cmp170hx Morrowmake/vllm-cmp170hx] || vLLM with an sm_80 plugin and a short patch queue || the reference CMP engine || [[GLM 5.3 Flash on 4x 8 GB]], [[DeepSeek V4.1 Flash on 4x 8 GB (Morrowmake)]], [[Qwen3.8 Flash Next on 2x 8 GB (Morrowmake EXL3)]], the ElectroHobo recipes, CurtisAccelerate, arvkmr
|-
| [https://github.com/wtdcode/vllm-backport wtdcode/vllm-backport] || vLLM fork for frontier MoE models on Ampere and Ada (A100, A6000, 3090), images <code><nowiki>lazymio/vllm-backport:*-sm80</nowiki></code> || runs; not CMP-specific || [[Qwen3.8 Flash Next on 2x 8 GB (mannydelrio PP2)]], matstatman's 4-card Qwen, Ivan's full GLM 5.3
|-
| [https://github.com/HalfVulpes/Deepinfer-SM80 HalfVulpes/Deepinfer-SM80] || DeepInfer ideas ported to sm_80 as two vLLM patches + 30 drop-ins || measured on 74-SM CMP cards || [[DeepSeek V4.1 Flash on 6x 8 GB (Deepinfer-SM80 TP2 x PP3)]], [[DeepSeek V4.1 Flash on 8x 8 GB (Deepinfer-SM80 TP4 x PP2)]], the GNmac image
|-
| Patched upstream vLLM || per-recipe patch sets on a pinned commit or PR || works where patched || Zanooda (PR #56201 + 20 patches), iIIusi0n (a5a30471 + 13-file patch), clee/vllm-sm80, vektorprime (PR #53899), DUFRelic (v0.30.0 + zankich INT4-PLE plugin), wgkbfcp (79a7108d)
|-
| [https://github.com/turboderp-org/exllamav3 ExLlamaV3] + TabbyAPI || EXL3 quants, layer split or TP || runs; compute-bound on sm_80 || [[GLM 5.3 Flash on 2x 8 GB (ExLlamaV3 TP2)]], [[GLM 5.3 Flash on 2x 8 GB (ExLlamaV3 layer split)]], [[DeepSeek V4.1 Flash on 4x 8 GB plus RTX PRO 6000 (ExLlamaV3 TP5)]]
|-
| [https://github.com/mw00/project-maya Project Maya] / Strata || expert tiering across VRAM, RAM and NVMe (Strata and llama.cpp lineage) || runs (compute capability 7.0+) || [[GLM 5.3 Flash on 2x 8 GB (Project Maya)]], [[DeepSeek V4.1 Flash on 2x 8 GB (Project Maya)]]
|-
| llama.cpp || GGUF || runs, but slow on these cards || not recommended
|-
| sglang || || FP8 Qwen 27B reported || see [[Qwen3.8 27B on 1x 8 GB (W4A16 vLLM)]]
|-
| DeepInfer (upstream), TokenSpeed, 1Cat-vLLM, EAGLE-4 || || not runnable or unmeasured on sm_80 || see below
|}

== Morrowmake/vllm-cmp170hx ==
The engine the community recipes pin. Since late October 2026 it is not a classic fork but a <code><nowiki>vllm-sm80</nowiki></code>
plugin (Python package <code><nowiki>vllm_sm80</nowiki></code> with sm_80 kernels, attention backends, EXL3 and Marlin extensions, model
classes and a launcher) plus a short patch queue on an exact upstream vLLM commit: an engine is three pins (upstream
commit, queue commit, plugin commit). Upstream vLLM's sparse-attention path for GLM-5.3 needs Hopper; this fork adds the
Ampere kernels that make it run at all.
* Branches: <code><nowiki>plugin</nowiki></code> (default), <code><nowiki>queue/330c5bffee</nowiki></code> (development line) and <code><nowiki>queue/165213f911</nowiki></code> (newest tested upstream), <code><nowiki>main</nowiki></code> (plain upstream mirror), <code><nowiki>ampere</nowiki></code> (frozen at 12d6ed3ce0, no more changes). The <code><nowiki>ampere-glm53</nowiki></code> branch is gone. Discord posts before late October describe the old single-tree <code><nowiki>ampere</nowiki></code> branch.
* Recipe pins are tagged and never deleted: <code><nowiki>glm53-recipe-1.3.0</nowiki></code> to <code><nowiki>1.8.0</nowiki></code> (1.8.0 = 819146023f), <code><nowiki>qwen38-recipe-1.0.0</nowiki></code> (18305948ac), <code><nowiki>dsv41-flash-recipe-1.0.0</nowiki></code> (a0c57b5237). Run it through the recipes' container images; that is the supported way.
* Build from source (no GPU needed): Python 3.12, torch 2.14.0, CUDA 12.8+; check out the queue branch and install with <code><nowiki>VLLM_USE_PRECOMPILED=1 VLLM_PRECOMPILED_WHEEL_COMMIT=<upstream commit></nowiki></code>, install the <code><nowiki>plugin</nowiki></code> worktree with <code><nowiki>--no-deps</nowiki></code>, then <code><nowiki>python csrc/exl3/build_standalone.py --out vllm_sm80</nowiki></code> for the EXL3 kernels (optionally <code><nowiki>VLLM_AMPERE_MARLIN_BUILD=1 python csrc/ampere_marlin/build_standalone.py</nowiki></code>). <code><nowiki>vllm_sm80/guard.py</nowiki></code> stops vLLM at start if the plugin is missing.
* EXL3 inside vLLM: <code><nowiki>--quantization exl3 --load-format exl3 --dtype bfloat16</nowiki></code>, integer rates 1 to 8 bits (1.5 / 2.5 / 3.5 need the mul1 codebook), TP cuts must keep 128-channel transform blocks, whole-expert expert parallel; PP other than 1 and LoRA are rejected. Tensor-core paths behind <code><nowiki>VLLM_EXL3_MOE_V2</nowiki></code>, <code><nowiki>VLLM_EXL3_MOE_PREFILL_TC</nowiki></code>, <code><nowiki>VLLM_DEEPSEEK_V41_MOE_PREFILL_TC</nowiki></code> and others. Several dense paths are validated only for 74-SM cards at TP2.
* Speculative decoding: DFlash2 and DSpark speculators, <code><nowiki>adaptive_k</nowiki></code> (draft count by load), <code><nowiki>shard_argmax</nowiki></code>, <code><nowiki>pp_draft_tail</nowiki></code>. PCIe-only helpers: <code><nowiki>host_shm_all_reduce</nowiki></code>, <code><nowiki>ce_ring_all_reduce</nowiki></code>, <code><nowiki>custom_all_reduce_flags</nowiki></code>, a <code><nowiki>pcie_p2p</nowiki></code> gate. A tuning table <code><nowiki>vllm_sm80/tuning/nvidia-cmp-170hx.toml</nowiki></code>. 111 fork environment flags, all off by default; the recipes turn on what they need.
* Contribution rules: sm_80 kernels and model code go to <code><nowiki>plugin</nowiki></code>; hooks in upstream files go to the newest <code><nowiki>queue/*</nowiki></code> branch, one commit per file; recipes for other models or layouts belong in your own repo, must pin the fork's commits, tag the repo <code><nowiki>cmp170hx</nowiki></code> and keep the NOTICE.
* License: Apache-2.0 (LICENSE added 2026-10-29); the plugin also carries MIT code adapted from ExLlamaV3.
{{Source|handle=voidgiraffe|date=2026-10-15|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560413471055548446|repo=https://github.com/Morrowmake/vllm-cmp170hx}} 
{{Source|handle=jjb01957|date=2026-10-24|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1563675495542038650}}

A longer write-up by @voidgiraffe on the fork's plugin layout and community recipes exists; see
[https://discord.com/channels/1518634379604136178/1528826330454429706/1560411313212227727 Discord].

== wtdcode/vllm-backport ==
A vLLM fork for frontier MoE models on older Ampere and Ada cards, with prebuilt single-arch Docker images
(<code><nowiki>lazymio/vllm-backport:latest-sm80</nowiki></code>, pinned <code><nowiki>:v0.13.0-sm80</nowiki></code>; no FA3 or Hopper kernels) and a paired LMCache
fork. It is not CMP-specific but is the base of mannydelrio's two-card recipe and the alternative engine lineage to
Morrowmake's.
* Models by fork version: DeepSeek-V4-Flash-0731 native FP4 (0.6.0+), Qwen3.8-27B BF16 / AWQ W4A16 (0.8.0+), Qwen3.8-Flash-Next FP8 + AWQ W4A16 (0.9.0+), GLM-5.3-Flash FP8 + AWQ W4A16 (0.11.2+), DeepSeek-V4.1-Flash native FP4 with Engram (0.13.0+). It publishes wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16 and wtdcode/GLM-5.3-Flash-AWQ-W4A16.
* FULL_AND_PIECEWISE CUDA graphs on Ampere need <code><nowiki>NCCL_ALGO=Ring NCCL_PROTO=Simple</nowiki></code> and preferably <code><nowiki>--disable-custom-all-reduce</nowiki></code>: the "crash on Ampere" was NCCL's size-adaptive algorithm switching under graph replay. Bound <code><nowiki>cudagraph_capture_sizes</nowiki></code>; FULL graphs can cost more than 800 MB per GPU.
* Qwen3.8-Flash-Next: <code><nowiki>VLLM_PLE_CPU_OFFLOAD=1</nowiki></code> keeps the n-gram embedding in pinned host RAM; <code><nowiki>--enable-expert-parallel</nowiki></code> is required (plain TP makes the 640-wide expert intermediate 160 per rank); <code><nowiki>--prefix-match-unit=16</nowiki></code> is effectively mandatory, otherwise nothing under 1,600 tokens is reused.
* DeepSeek V4.1 Flash: <code><nowiki>--engram-config '{"cpu_offload": true}'</nowiki></code>, <code><nowiki>--tokenizer-mode deepseek_v41</nowiki></code>, DSpark K=5. Agentic clients should pass <code><nowiki>reasoning_effort</nowiki></code> from the first turn or sessions drift into reasoning loops.
{{Source|handle=matstatman|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560599247475314711|repo=https://github.com/wtdcode/vllm-backport}}
* matstatman ran Qwen3.8 Flash Next on four cards with this engine and only 32 GB of host RAM. A longer write-up by @matstatman exists; see [https://discord.com/channels/1518634379604136178/1528826330454429706/1560602001908170753 Discord].
* Ivan's lab: the full 753B GLM-5.3 (an abliterated NVFP4-expert / INT8 requant) runs on eight cards with this engine at TP2 x PP4 with P2P and MTP, 524K context and a 570K-token KV pool, but only with a one-line worker fix for the pipeline hand-off bug (below): 77 / 76 / 48 tok/s single (code / structured / prose), 172 / 189 / 146 at 8 users, prefill 1,650 tok/s on a 17k prompt, 68% draft acceptance. The int8 KV tile beat plain FP8 KV by 10% single-stream at the same quality. Schema-constrained output (named <code><nowiki>tool_choice</nowiki></code>, <code><nowiki>json_schema</nowiki></code>) returns HTTP 500 with MTP on, because xgrammar rejects the draft tokens; serve that traffic without speculation. GLM-5.3-Flash checkpoints with a dense layer 0 fail to load on it (KeyError on <code><nowiki>layers.0.mlp.gate_up_proj.weight</nowiki></code>). {{Source|handle=darkmaniac7|date=2026-10-28|note=lab notes}}

== Other vLLM patch sets ==
* '''[https://github.com/arvkmr/vllm-170hx-glm5 arvkmr/vllm-170hx-glm5]''': fail-closed source patches on Morrowmake's frozen <code><nowiki>ampere-glm53</nowiki></code> commit that port GLM-5.3's DSA path to sm_80 (packed <code><nowiki>fp8_ds_mla</nowiki></code> KV reader, software e4m3, Triton indexer, canonical top-k, DFlash over PP, top-k PP relay). See [[GLM 5.3 on 8x 8 GB (vllm-170hx-glm5)]]. {{Source|handle=reala7096|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557544812972871813|repo=https://github.com/arvkmr/vllm-170hx-glm5}}
* '''[https://github.com/clee/vllm-sm80 clee/vllm-sm80]''': pure-Python fork on a pinned vLLM nightly with FP8 KV on sm_80 (software e4m3 decode), PLE under PP>1 and an NVMe-staged PLE reader; its PP KV fix is also used by mannydelrio. See [[Qwen3.8 Flash Next on 3x 8 GB (clee PP3)]]. {{Source|handle=c133|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1556357466026025001|repo=https://github.com/clee/qwen3.8-flash-next-3x170hx}}
* '''Zanooda''': a 20-patch series on vLLM PR #56201 plus haosdent/vllm's dsv4-flash-a100 sm_80 layer, with an sm_80-only image. See [[DeepSeek V4.1 Flash on 8x 8 GB (Zanooda PP8)]]. {{Source|handle=gnasherrr|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560998024136753172}}
* '''iIIusi0n''', '''vektorprime''', '''DUFRelic''' (zankich INT4-PLE plugin), '''wgkbfcp''': see their recipe pages under [[Recipes]].
* eXile has a custom vLLM build aimed at the 170HX running Q3 Qwen 3.8 Flash Next on one 64 GB card; HumanEval matched his Strata results. {{Status|Contested}} Others doubted the prefill figure (xtj7 suspected cache hits); no answer was posted. {{Source|handle=exile_1ck|date=2026-10-15|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560342257532674060}} {{Source|handle=xtj7|date=2026-10-15|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560352992035868753}}

== DeepInfer and Deepinfer-SM80 ==
[https://github.com/deepseek-ai/DeepInfer deepseek-ai/DeepInfer] is DeepSeek's open-sourced inference framework for
DeepSeek-V4.1-Flash (DSpark, SWA bounded replay, batch-invariant modes). Upstream supports only NVIDIA SM100 and Huawei
Ascend 950, so it does not run on the CMP as released. {{Source|handle=68x0|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562102232441299087|repo=https://github.com/deepseek-ai/DeepInfer}}

[https://github.com/HalfVulpes/Deepinfer-SM80 HalfVulpes/Deepinfer-SM80] ports its ideas to sm_80 as two vLLM patches on
the unmerged PR #56201 commit (79a7108d) plus 30 drop-in Python, Triton and TileLang modules. Build with
<code><nowiki>TORCH_CUDA_ARCH_LIST="8.0 8.6"</nowiki></code>, CUDA 13.0, Python 3.12, GCC 13+, torch 2.13.0+cu130; check with
<code><nowiki>scripts/check_dropin_sync.py</nowiki></code> ("30 same, 0 diff, 0 missing") and <code><nowiki>scripts/verify_env.py</nowiki></code>. Kernel speedups on
74-SM sm_80: <code><nowiki>_combine_topk_swa_indices_kernel</nowiki></code> 22.17x (bit-exact), <code><nowiki>dequantize_and_gather_k_cache</nowiki></code> 4.02x at 16K
and 7.64x at 48K, mHC fused post/pre 2.89x, sparse-attention decode 2.08 to 2.14x, sparse prefill 1.17x. 108-SM parts
(A100, A800) gain less because the tunings are wave-aligned for 74 SMs. The patches do not apply to newer vLLM main.
{{Source|handle=segmond|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562224915501813811}} 
{{Source|handle=reala7096|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562151531690860657|repo=https://github.com/HalfVulpes/Deepinfer-SM80}}

== ExLlamaV3, TabbyAPI and EXL3 ==
* turboderp's ExLlamaV3 loads EXL3 quants (for example turboderp/GLM-5.3-Flash-exl3 at 3.05 bpw) and runs on sm_80 when built with <code><nowiki>TORCH_CUDA_ARCH_LIST=8.0</nowiki></code>. The two-card GLM recipes use it with layer split or tensor parallel over NCCL; anoane serves DeepSeek V4.1 Flash through an ExLlamaV3 fork plus a TabbyAPI fork (TabbyAPI is AGPL-3.0). vcruz305/vllm-exl3 (AGPL-3.0) brings EXL3 into vLLM and is part of the GNmac image. {{Source|handle=wingrimez_40467|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1562164200372179017|repo=https://github.com/Flun/glm53-flash-cmp170hx-exl3}} {{Source|handle=anon33anon|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1561903682608697426|repo=https://github.com/anoane/09-deepseek-v41-flash-1017-exl3-tp5-1m}}
* {{Status|Contested}} EXL3 speed on CMPs. Miséricorde: EXL3 is very slow on CMP because the sm_80 dequant path is heavily compute-bound, using only 10 to 20% of HBM bandwidth; do not run EXL3 quants on CMPs. Takka: small-batch decode should still be bandwidth-bound; the card has only about 12 TFLOPs of INT32. Morrowmake reached about 150 tok/s on Qwen Flash Next EXL3 5.05 bpw, then said W4A16 is probably the better choice. Miséricorde also says EXL3 is acceptable for DeepSeek V4.1 Flash because only about 30% of its decode is EXL3 work (about 95% for GLM). {{Source|handle=wakeupmbrc1|date=2026-10-14|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1559867344367652865}} {{Source|handle=wakeupmbrc1|date=2026-10-14|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1559796160724475996}} {{Source|handle=spectakkular|date=2026-10-15|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560298253109432392}} {{Source|handle=voidgiraffe|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560599553210851400}} {{Source|handle=wakeupmbrc1|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1561137015167254579}}
* tekkenz ran turboderp's Qwen3.8-Flash-Next EXL3 at 5.05 bpw (0.0040 KLD from BF16) on one card: 69 to 82 tok/s decode, about 1,000 tok/s prefill. How it fits one card without RAM offload was asked and not answered. {{Source|handle=tekkenz|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1557128719422853223}}
* Ryko serves GLM-5.3-Flash 4-bit EXL3 on ExLlamaV3 across two 64 GB CMPs, a 40 GB card and an RTX 3080 20 GB (188 GB total) with 768k context at 16-bit KV: about 70 tok/s decode, a 100k-token prefill at 3,500 to 3,600 tok/s, after letting an AI agent optimize ExLlamaV3 for the hardware. {{Source|handle=vektor_prime|date=2026-10-14|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1560023669416402965}}
* Mountains runs the full GLM 5.3 EXL3 3.05 bpw across four 8 GB CMPs plus an RTX PRO 6000 with pipeline parallel at 30 tok/s; it did not work out of the box with PP and mixed architectures. {{Source|handle=mountains1605|date=2026-10-12|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1559063237775990795}} {{Source|handle=mountains1605|date=2026-10-12|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1559339980541796483}}

== llama.cpp ==
Runs, but it is the slow option on these cards; use vLLM or sglang with INT4/INT8 weights instead.
* ikyo's Mac Studio beat a single CMP in most tests on Qwen 27B Q8 with llama.cpp; the community said llama.cpp is very suboptimal on CMPs. {{Source|handle=ikyo|date=2026-10-26|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1564314992998613125}} {{Source|handle=lia_oc|date=2026-10-26|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1564315280916615251}}
* Tokukarin cannot get past 60 tok/s on Qwen Flash Next at 4.27 bpw on one card even with MTP; he will not go lower because 3.84 bpw degraded multilingual output. {{Source|handle=tokukarin|date=2026-10-27|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1564443422314602559}}
* llama.cpp can put mmproj and the speculative draft on a separate device, so a fifth card is still useful while the main model stays on four. {{Source|handle=da6b|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562556812148998207}}
* Comparison point: xanthe_ gets 88 tok/s on Qwen 3.6 across two V100s with llama.cpp built from source at 5-bit. {{Source|handle=xanthefin|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561337005105746011}}
* PixelML measured a single-kernel decode engine at 70.6 tok/s against llama.cpp 53.4 on Qwen3.8-Flash-Next UD-Q3_K_XL on one card; the 26.8 GiB PLE table must sit on local disk (29 tok/s from a network mount). {{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}

== Strata and Project Maya ==
Strata is a consumer local-LLM app with its own engine; it lists no CMP support but people run it. Project Maya forked
it (plus llama.cpp) to tier MoE experts across VRAM, RAM and NVMe.
* pcsv: Strata works well with CMPs, 100+ tok/s with default settings; forks (mw00/project-maya for DeepSeek 4.1 Flash, ro99/Strata for Kimi K3) proved larger models. {{Source|handle=pcsv.|date=2026-10-25|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1563917658334695467}}
* acedogblast: more than 60 tok/s on Qwen 3.8 Flash Next on one card with Strata (Unsloth QD-4K_XL, FP16 KV). {{Source|handle=acedogblast|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1556834517103157331}}
* xtj7: Strata on one card with quad-channel DDR4 gives about 90 tok/s decode and 1,600 tok/s prefill at IQ2, without an x16 mod. {{Source|handle=xtj7|date=2026-10-15|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560350518847479850}}
* eXile (one 10 GB-variant card at 40 GB, 14600K, 128 GB DDR4): moving from x4 to x16 mostly improved prefill; his modified Strata build beats stock (110+ against under 100). {{Status|Theory}} Miséricorde expects ECC DDR4 server memory to double Strata speed. {{Source|handle=exile_1ck|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557241785695211612}} {{Source|handle=exile_1ck|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557256690255134751}} {{Source|handle=wakeupmbrc1|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557241997138333698}}
* Project Maya pages: [[GLM 5.3 Flash on 2x 8 GB (Project Maya)]], [[DeepSeek V4.1 Flash on 2x 8 GB (Project Maya)]].

== Speculative decoding ==
* '''DFlash2''' (inco.ai drafters for GLM-5.3-Flash and Qwen3.8-Flash-Next) is the default in the Morrowmake GLM recipe and the ElectroHobo Qwen recipes. The GLM drafter is CC BY-NC-ND 4.0 (non-commercial); the Qwen one is Apache-2.0. Miséricorde confirmed the Qwen drafter helps. {{Source|handle=wakeupmbrc1|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1561924555486920825}}
* '''DSpark''' is DeepSeek V4.1 Flash's own drafter, built into the checkpoint. Acceptance runs about 5.1 tokens per step on code and about 2.9 on reasoning text (reala), which leaves reasoning at 55-75 tok/s on six to eight cards. On the GNmac port the draft count must be 5: 6 or 7 do not start, and 3 or 4 boot but every request fails, so that recipe narrows verification with <code><nowiki>DSV41_DYNK</nowiki></code> instead. On Ivan's mixed bench DSpark accepted 2.1 of 5 draft tokens per step (2.6x decode), and a fine-tuned FP8 DSpark refit measured identical to stock (2.14 against 2.13). {{Source|handle=reala7096|date=2026-10-22|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562917910165323808}} {{Source|handle=darkmaniac7|date=2026-10-28|note=lab notes}}
* '''MTP''' heads ship with Qwen3.8 and GLM-5.3 checkpoints. On Qwen3.8 27B matstatman recommends MTP with k=3 over DFlash. On Qwen Flash Next DFlash reached only 14% acceptance at the 4th token for Sojun, and Mountains never got speculative decoding working for it on the 170HX (it worked for the 27B). The ElectroHobo recipes now ship a DFlash2 drafter for it. {{Source|handle=sojun80|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557431093119033375}} {{Source|handle=mountains1605|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557890449186758737}} {{Source|handle=matstatman|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560828125863084103}}
* {{Status|Theory}} '''EAGLE-4''' (eagle-hq): its Triton kernel tier admits sm_80 and cibernox flagged it as faster than MTP and DSpark2, but nothing has been measured on Ampere or a CMP, it needs patched engine wheels, and it has drafts only for Qwen3.8-27B and Nemotron. Source-available license, not open source. {{Source|handle=cibernox|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562247115562942555|repo=https://github.com/eagle-hq/EAGLE-4}}

== Mixing CMP and newer GPUs in one job ==
Ivan runs Morrowmake's engine across CMP 170HX (sm_80) and RTX PRO 6000 Blackwell (sm_120) cards in one vLLM job with
pipeline parallel, CMPs first and Blackwells as the last stage. It needed three small Python patches (resolve the
device capability per rank instead of from device 0, pick the MLA backend on sm_120 without the FP8 KV path, build the
host-staged all-reduce only on sm_80 TP groups) and four launch settings, each found by a failed boot:
<code><nowiki>--additional-config {"kda_prefill_backend":"triton"}</nowiki></code>, <code><nowiki>CUTE_DSL_ARCH=sm_120a</nowiki></code>,
<code><nowiki>--kernel-config {"enable_flashinfer_autotune":false}</nowiki></code> (autotune on one stage deadlocked the others), and at least
one MLA layer on every stage. Results for GLM 5.3 Flash are on [[GLM 5.3 Flash on 4x 8 GB#With other GPUs in the same job]].
The full GLM 5.3 does not run this way on the Morrowmake fork: its DSA indexer path needs DeepGEMM kernels that exist
only for Hopper and Blackwell, and its KV writers use an FP8 type Triton refuses below sm_89; wtdcode/vllm-backport
already has the sm_80 pieces but no sm_120 build. {{Status|Theory}} Ivan estimates a port at 2 to 4 days.
{{Source|handle=darkmaniac7|date=2026-10-19|note=lab notes}}

== Load times ==
Loading 350 to 380 GB models takes under 5 minutes from NVMe (impactic), about 20 minutes from SATA (matstatman) and
about 8 minutes at Gen2 x4 (Mountains). First starts also compile kernels: 5 to 22 minutes depending on the recipe.
{{Source|handle=darkmaniac7|date=2026-10-20|note=lab notes}} 
{{Source|handle=mountains1605|date=2026-10-27|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1564747358661509230}}

== Engine-level known issues ==
* '''Kernel launch overhead on the n-gram models.''' Qwen3.8 Flash Next launches about 2,000 small kernels per step: decode compute is only 1.5 to 1.8 ms of a 10 to 11 ms round, so the model is cheap at high concurrency on big GPUs but ineffective at low concurrency on CMPs; hand-built megakernels gave 20 to 25%. {{Source|handle=wakeupmbrc1|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1559631250766168075}} {{Source|handle=wakeupmbrc1|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560992853927403583}}
* '''Prefill stalls decode''' in vLLM on PP setups: per-stream speed drops with concurrency because decode stops during prefill; fair prefill slicing (Morrowmake, ElectroHobo) mitigates it. {{Source|handle=ob88594|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557464371838451723}}
* {{Status|Contested}} '''Custom all-reduce and P2P on Gen2.''' Results differ by host and model: on Ivan's EPYC box custom all-reduce cost GLM-5.3-Flash TP4 about 40% decode but helped Qwen Flash Next TP2 by 17 to 25%; PixelML saw P2P hurt GLM TP4 on PLX + Broadwell; ElectroHobo saw P2P off cost MiMo TP4 40% on PLX + Broadwell. A/B it per recipe. A hang at init can be stale build locks in a copied compile cache. {{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}} {{Source|handle=darkmaniac7|date=2026-10-28|note=lab notes}}
* '''Pipeline hand-off bug at TP>1 with PP''' (GLM-5.x DSA models, vLLM PR #60477): a non-last pipeline stage sends an un-reduced partial sum, corrupting the first layer of the next stage at TP2 x PP4. Symptoms in Ivan's runs: about 3 points lower teacher-forced agreement, 32k-token degeneration loops and reasoning that eats the budget on long prompts. The one-line worker fix costs no speed. A second, separate bug (vLLM issue #63170, draft PR #59961): GLM-5.x shared-indexer layers at a stage start read stale top-k indices, so choose a PP split that starts every stage on a full indexer layer. {{Source|handle=darkmaniac7|date=2026-10-28|note=lab notes}}
* '''TP size must divide the attention head count.''' DeepSeek V4.1: TP 1, 2, 4 or 8 only; TP6 works only with wgkbfcp's virtual-head padding. {{Source|handle=greencandle9753|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562580021153894501}}
* '''No tuned GEMM for the harvested die''': training on one CMP ran almost 4x slower than a 5090 with the same nominal BF16 TFLOPs. {{Source|handle=programmerboi|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560682620012990505}}

== Engines that do not run on sm_80 (yet) ==
* TokenSpeed (LightSeek Foundation): day-0 GLM-5.3-Flash and Qwen3.8-Flash-Next support, but the README documents no Ampere path. Its sm_80 pull request was closed as won't-fix. {{Source|handle=c133|date=2026-10-05|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1556808015003390052|repo=https://github.com/lightseekorg/tokenspeed}} {{Source|handle=darkmaniac7|date=2026-10-20|note=lab notes}}
* 1Cat-vLLM: a V100 (sm_70) fork; a comparison point and technique source (FP8 KV emulation, NVFP4 on hardware without FP8 tensor cores), not a CMP engine. {{Source|handle=xeax|date=2026-10-24|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563519176327299192|repo=https://github.com/1CatAI/1Cat-vLLM}}
* PixelML's negative results: NVFP4 GLM checkpoints need SM12x FlashInfer; EXL3 / TR3 4 bpw ships SM121-only kernels; the Ninfer sm_80 fork was 3.6x slower than vLLM with DFlash2. {{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}

== Related ==
[[Recipes]] · [[Quantization on the 170HX]] · [[Benchmarks]] · [[Unlocks]] · [[Driver unlock]] · [[Troubleshooting]] · [[Tools]]

{{ImportFooter}}
[[Category:Engines]]