Speculative decoding on the 170HX

Speculative decoding lets a small drafter propose several tokens that the full model then checks in one forward pass; every accepted token saves a decode step, and the target model still verifies every token it keeps. Three drafters come up on the CMP 170HX: MTP, the multi-token-prediction head that Qwen and GLM models ship; DFlash2, a separate small draft model; and DSpark, the drafter of the DeepSeek V4 and V4.1 Flash ports, also used in some Qwen3.8 27B benchmarks. How much they help depends on the work: acceptance is high on code edits, structured output and short synthetic prompts, lower on prose and real codebases, and the gain shrinks as context grows. Temperature moves it as well. State model, drafter, quant, temperature and context with every tok/s figure, and compare recipes only on equal benchmarks (Benchmarks). Engine background is on Engines and llama.cpp on CMP cards; vLLM bugs that involve MTP (prefix caching, the PP drafter embedding, schema-constrained output) are on vLLM on sm_80: known bugs and patches.

MTP

MTP in July: marginal or negative on the bandwidth-bound Qwen3.6-35B-A3B in vLLM, helpful at low context

Contested 8 GB The thread of 2026-07-21:

AKuchen found MTP slower on Qwen3.6-35B-A3B in vLLM, which is already bandwidth bound, with 75% acceptance at 1.75 tokens. On llama.cpp with unsloth/Qwen3.6-35B-A3B-MTP-GGUF, MTP gave +21% (108.4 to 131.3 tok/s), but vLLM without MTP was faster still (147 tok/s). The MTP layers are kept in BF16 (listed in modules_to_not_convert: fc, attention and shared expert) and are bandwidth-limited, so the draft plus verify overhead eats the gain. Source: @akuchen_66498, 2026-07-21
robbbbbb7973 and arnold civardagezen: MTP helps at low to medium context and stalls to baseline past about 50 to 100k context. Source: @robbbbbb7973, 2026-07-21
Source: @a.civardagezen, 2026-07-21
Arcanum reported high (80%) acceptance with Q8 MTP for language and code. Source: @_arcanum_, 2026-07-21

Vcore took part in the same thread. Source: @vcore2191, 2026-07-21

The one-card Qwen3.6 setups of that period are on Qwen3.6 27B and 35B-A3B on one card (July and August 2026).

MTP depth: 2 or 3 draft tokens, and the slowdown at long context

Contested 8 GB How many tokens MTP should draft (num_speculative_tokens); the views:

In the thread of 2026-07-29 (aaasss, Josh), Josh found MTP n=2 the sweet spot, with anything higher slower at 200K. Source: @aaasss1456, 2026-07-29
Source: @_____josh_____, 2026-07-29
xPatrik, on Qwen 3.6 27B FP8, measured default MTP:3 at 92 tok/s decode on a new chat, falling to 22 tok/s at 130K context, while a tuned MTP:3 config gave 88 and 73 tok/s; --attention-backend FLASHINFER sped up generation. Source: @._.patrik, 2026-07-29
matstatman runs Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 with PP=2, MTP=3 and BF16 at 84 tok/s, and showed acceptance lower on Linux driver code than on React pages (above 50% counts as a win; lowering temperature to 0.7 seemed to raise it). His view that MTP 3 is "the holy number" for Claude Code work is under Benchmark inflation below. Source: @matstatman, 2026-08-27
Phoenix17 (2026-08-27, Qwen3.8-27B SmoothQuant W8A8 on one card): MTP 3 gives about 4.5% more effective output tok/s at low concurrency but loses to MTP 2 by about 2% at c=32, with no prefill difference, which he called nitpicking within run-to-run variance. He recommends MTP=2 because MTP=3 hurts at longer context and acceptance drifts with the prompt. Source: @phoenixone7, 2026-08-27
Older history (2026-07-27): xPatrik, MTP on in both engines: llama.cpp Q8 averaged 60 tok/s and vLLM INT8 (FP8 also tested) 90 tok/s on short chats. At about 130k context llama.cpp held 48 tok/s while vLLM fell to 22 tok/s. It was his first vLLM setup and giaki3003 thought he was doing something wrong; no fix was posted. Source: @._.patrik, 2026-07-27
Source: @giaki3003, 2026-07-27

MTP 3 misbehaving on Qwen3.6-35B-A3B INT8-W8A8, MTP 2 better

Danske runs Qwen3.6-35B-A3B-INT8-W8A8 on his CMPs. Kaatchi gets about 46 tok/s with the same model and could not get MTP 3 working (2026-08-15). fallengod9962 also had problems with MTP 3 and found MTP 2 works a lot better.

Reported
Source: @danske1, 2026-08-15
Source: @catchyuser, 2026-08-15
Source: @fallengod9962, 2026-08-15

DFlash2

DFlash2 drafts with a separate small model (a 2B drafter with 7 draft tokens in the runs below) instead of the model's own MTP head. Multi-card recipes that use it: GLM 5.3 Flash on 8x 8 GB (promisezackr PP8 NVFP4 DFlash2) and GLM 5.3 Flash on 4x 8 GB (kaka86mm EXL3 + Marlin sidecar + DFlash2).

DFlash2 against MTP: whether the bigger drafter pays off on these cards

Contested The dispute is about real-world speedup on code, not output quality. The views, in the order posted:

roger_dammit (2026-08-20) expected DSpark2 plus vLLM hacks to approach about 150 tok/s. Source: @roger_dammit, 2026-08-20
Studebaker (2026-08-20) tested DFlash2 against MTP in vLLM on FP8 Qwen3.8-27B and got only about 2 tok/s more decode for a 2B draft model and 7 draft tokens, with the extra draft compute "taking a toll" under concurrency. Later he reported up to 120 tok/s on Qwen3.8-27B depending on temperature and token settings. Source: @_studebaker_, 2026-08-20
fallengod saw a big decode increase with DFlash2 on W8A16, but prefill fell to roughly half. Source: @fallengod9962, 2026-08-20
Ryko (2026-08-25): both MTP and DFlash slow prompt processing; MTP gives no prefill benefit while adding layers. Source: @vektor_prime, 2026-08-25
DOG and bigfoot (2026-08-25): bigfoot finds DFlash generally better than MTP, but setup-dependent. Source: @prefix10k, 2026-08-25
Source: @bigfoot1144, 2026-08-25
matstatman (2026-09-04): DFlash2 gives 200+ tok/s in benchmarks but about 10 tok/s when coding (BF16 KV). Source: @matstatman, 2026-09-04
Josephur: DFlash2 is lossless and does not affect output quality. He stacks DFlash with n-gram speculation at temperature 0.7 in llama.cpp, measured on an AMD V620 with ROCm, so results on the 170HX may differ. Source: @josephur, 2026-09-04
In the thread of 2026-09-10 (Schaka among others), Josephur called DFlash2 stacked with n-gram on Qwen 3.8 Flash Next "perfecto", and others reported DeepSeek V4 Flash at about 100 tok/s decode with DFlash2. Source: @schakaqt, 2026-09-10

Measured single-card numbers are in the next two entries.

DFlash2 against MTP on real text: 105 against 55 tok/s at C1, and why random-token benchmarks mislead

fallengod (2026-08-20, automated harness, Qwen3.8-27B): DFlash2 looked slower than MTP only because random-token workloads collapse draft acceptance (position-0 acceptance 0.55 to 0.78). On GSM8K real text, at C1 DFlash2 gave 105.2 tok/s (8.1 ms TPOT) against MTP 55.1 tok/s (16.1 ms), 1.91x, and at C8 384.4 against 366.3 tok/s, 1.05x. A 16K-in / 512-out workload is about 90% prefill, where the W8A16 target (dequantised to BF16) loses about 2x to W8A8 INT8 GEMM, the same gap at mnbt 8192. MTP x2 verifies 3 tokens per step while DFlash2 x7 verifies 8 plus a 5-layer draft forward, so on this low-FLOPS card DFlash2 needs about 4.5 to 6 accepted tokens per step to win. Pure autoregressive decode at C8 measures only 168 tok/s, so speculation matters more here than on an H200. Redblood, clee and greencandle took part in the thread of 2026-08-19: greencandle posted lued/Qwen3.8-27B-INT8-W8A16-DFlash2 and clee the DFlash2 blog post.

Reported
Source: @redblood_, 2026-08-19
Source: @c133, 2026-08-19
Source: @greencandle9753, 2026-08-19
Source: @fallengod9962, 2026-08-20

DFlash2 on one card: about 160 to 175 tok/s single stream with W4A16, at a cost in context and prefill

Phoenix17 (2026-08-27) found DFlash2 single-stream only: it works but costs context and prefill speed, so he recommends it only for chat. He could not match another user's 212 tok/s, getting 160 at a very small batch size and about 175 tok/s at best; the other benchmark used batch size 1 on a card power-unlocked to 300 W, while his 8 GB cards are limited to 250 W. On 2026-09-17 he put it at about 175 tok/s on Qwen3.8-27B if context is limited to about 200K and one stream uses DFlash2 with W4A16, with 4-bit weights borderline for quality and DFlash2 suited to a single terminal rather than agents spawning sub-agents. W4A8 also works, probably faster at worse quality. xtj7 recalls that it needs a BF16 KV cache, Kanorin confirmed 70 to 150 tok/s with it, and Miséricorde said 180+ tok/s is possible at 16K context. matstatman had skipped DFlash2 because DSpark was poor for coding. The full single-card setup is on Qwen3.8 27B on 1x 8 GB (W4A16 vLLM).

Proven 8 GB
Source: @phoenixone7, 2026-08-27
Source: @matstatman, 2026-09-17
Source: @xtj7, 2026-09-18
Source: @_kanorin, 2026-09-18
Source: @wakeupmbrc1, 2026-09-18

DSpark

DSpark is the drafter the DeepSeek V4 Flash 0731 and V4.1 Flash recipes run. Recipes: DeepSeek V4 Flash 0731 on 4x 8 GB (allover326 PP4 DSpark), DeepSeek V4 Flash 0731 on 3x 8 GB (PP3 DSpark), DeepSeek V4.1 Flash on 4x 8 GB (GNmac), DeepSeek V4.1 Flash on 8x 8 GB (Zanooda PP8), DeepSeek V4.1 Flash on 5x 8 GB (PP5 full residency).

DSpark on DeepSeek V4.1 Flash: num_speculative_tokens must be 5

DeepSeek V4.1-Flash-1017 ships its DSpark drafter inside the checkpoint (three stages, block size 5, 2,401 tensors, about 7.9 GB). The patched vLLM builds refuse method "mtp" on V4.1. num_speculative_tokens must be 5 or less, or a multiple of 5, and in practice only 5 works well:

k=7 is refused (PixelML's club-170hx).
On GNmac's build, k=6 and k=7 do not start, and k=3 and k=4 boot but every request fails; GNmac narrows verification with DSV41_DYNK=accept:3,5 instead.
On wtdcode's backport, values below 5 are rejected; 7 needs about 200 KB of shared memory against the 163 KB Ampere limit (triton OutOfResources); 6 starts, but draft positions past the native block are accepted only 3 to 13% of the time.
Upstream enable_adaptive_verification is not supported with pipeline parallelism.

Older history: the k=5 rule first showed on DeepSeek V4 Flash 0731. DeeJay's allover326 recipe (2026-08) found DSpark needs num_speculative_tokens exactly 5 (7 gave 60.3 against 98.1 tok/s; method mtp fails with KeyError mtp_block.main_norm). dkpoulsen measured num_speculative_tokens 10 at 24.7 tok/s against 57.8 at 5 on his PP5 V4.1 build (2026-09-23).

Reported 8 GB
Source: @darkmaniac7, 2026-10-27 (lab notes)
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
Source: repo github.com/wtdcode/vllm-backport by wtdcode on GitHub (posted in Discord by @jjb01957 and @matstatman)
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub (allover326 is @deejay9983)
Source: repo github.com/dkpoulsen/dsv41-320e-dspark by dkpoulsen on GitHub (posted in Discord by @dkpoulsen)

DSpark acceptance: about 2 to 6 tokens per step, highest on code and structured output

Measured acceptance with k=5 on DeepSeek V4.1 Flash:

Zanooda PP8: about 5.9 tokens per step on repetitive output and 3.9 on prose (per position 77/55/42/34/31%).
GNmac: about 6 against 3 (prose 82/58/40/29/20%).
PixelML club-170hx (TP4 + EP4, EXL3 3.0 bpw, k=5, mixed bench, linked by clee on 2026-10-20): 14,698 accepted of 21,640 drafted = 67.9%, 4.40 tokens per step, per position 89/78/67/57/50%. k=10 is slower (4.01 tokens per step; positions 6 to 10 accept 9/6/3/1/0.6%). Without a drafter 25.4 tok/s, against 99.5 and 48.9 with DSpark (3.9x and 1.9x).
reala (2026-10-22): about 2.9 tokens per step on reasoning text against 5.1 on code.
alanoo: 99.4% on structured output, about 70% on code, 43 to 53% on prose.
The lab's mixed bench: 2.1 of 5 (2.6x decode on GNmac's build).

Older history: DeeJay's allover326 recipe on V4 Flash 0731 had a mean acceptance of 3.03 of 6; tim-odonnell's 3-card DSpark averaged 80.95 tok/s against 55.8 without (2026-08-15).

Reported 8 GB
Source: @darkmaniac7, 2026-10-27 (lab notes)
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
Source: @reala7096, 2026-10-22
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub (allover326 is @deejay9983)
Source: repo github.com/tim-odonnell/cmp170hx-deepseek-v4-flash by tim-odonnell on GitHub (posted in Discord by @irythros)

No other trained drafter fits DeepSeek V4.1 Flash

EAGLE3 and DFlash heads exist only for V4-Flash (hidden size 4096, 43 layers); V4.1 has hidden size 5120 and 40 layers. DFlash2 and EAGLE-4 have no DeepSeek drafter. Two drop-in refits of DSpark exist: SubconsciousDev/deepseek-v4.1-flash-1017-dspark (FP8, same tensor names; measured identical to stock, 2.14 against 2.13) and ursuciprian/DeepSeek-V4.1-Flash-1017-DSpark-D1 (an EXL3 overlay; its author claims +6.6 to 11.6% on prose on 2x DGX Spark). n-gram and suffix decoding are available only on GNmac's V1-runner path and replace DSpark, so they should lose on prose; a DSpark + suffix hybrid would need unpublished code work. greencandle: the drafter and its confidence gate do not change the output distribution, because the target verifies every kept token.

Reported 8 GB
Source: @darkmaniac7, 2026-10-27 (lab notes)
Source: @greencandle9753, 2026-10-23

Benchmark inflation

Speculative-decoding benchmarks inflate: 216 tok/s on edit_bench against 20 to 40 tok/s on real code

Contested State model, drafter, quant, temperature and context with every tok/s figure. The views:

matstatman (2026-08-18) ran the benchmark of 0xBakeer's "Qwen3.8-27B FP8 on a single DGX Spark" recipe on one 170HX with DSpark k=7 at 230k context: c1 216.54 tok/s, c4 406.20 aggregate (101.55 per stream), c8 442.12 (55.27 per stream), the c8 plateau coming from --max-num-seqs 4; a DGX Spark at k=7, 263k, c8 got 208.7 aggregate. He thinks the benchmark is unrealistic: without DSpark or MTP he gets 20 to 25 tok/s on FP8, and draft acceptance is over 90% on synthetic text but 0 to 25% on real codebases (about 25 tok/s with DSpark on real code, about 35 without). Test on your own repo, for example by asking for an upgrade plan. The edit_bench.py payload hardcodes temperature 0, enable_thinking False and near-zero context; plain Qwen3.8 gave him 40 tok/s at 32k. He measured a mean acceptance length of 3.6 with MTP 3 and called MTP 3 "the holy number"; on this model DSpark needs k=7 or more, k=7 does badly in Claude Code, and he considers k=2 to 3 realistic. Source: repo github.com/0xBakeer/Qwen3.8-27B-FP8-on-a-single-DGX-Spark by 0xBakeer on GitHub (bench script linked in Discord by @matstatman)
Source: @matstatman, 2026-08-18
matstatman (2026-08-20) warned that you need a bad benchmark (a prompt of about 300 characters generating 32 tokens) to see 200+ tok/s: he got 210 tok/s decode on Qwen3.8 with a benchmark from Twitter, far above what real agent workloads show. tchelobh joked that a 0.1B drafter plus "repeat the character A 1024 times" would inflate it further. Source: @matstatman, 2026-08-20
Source: @tchelobh, 2026-08-20
roger_dammit disagreed from his own runs (throughput kept rising up to k=7) but agreed that benchmarks should report context depth. Source: @roger_dammit, 2026-08-20
Studebaker: the 210 tok/s figure probably comes from 2 to 4 streams and does not show in matstatman's testing-general benchmarks. Source: @_studebaker_, 2026-08-25
giaki3003 disputes that temperature 0 is cheating. Source: @giaki3003, 2026-08-25
On 2026-09-26, answering Programmer007's question about a 27B INT8 on one 170HX, fafinet got 50 tok/s with an INT8 AWQ on vLLM 0.29.0 and said llama.cpp reached up to 70 tok/s with a Q8, so vLLM should have room to improve. matstatman again quoted 210 tok/s single stream with Q8, depending on the benchmark (speculative decoding details not stated). Source: @fafinet, 2026-09-26
Source: @matstatman, 2026-09-26
Source: @programmer_007, 2026-09-26

Compare recipes only on equal benchmarks.

Decode speed falls with context depth; short-prompt benchmarks mislead

matstatman (2026-08-18): benchmarks measure decode after a tiny prompt, not after 100k tokens. roger_dammit: most benchmarks look at small depths, and he finds decode falls roughly linearly with depth (14 tok/s at about 20k on Qwen3.8 27B W8A16). Mitakihara (2026-08-25), Qwen3.8-27B decode only with DFlash2: about 200 tok/s below 10k context, falling gradually to under 20 tok/s at 200k; MTP n=3 peaks around 120 tok/s. In agentic flows the context fills toward 200k and needs early compaction, and prefilling 200k tokens is slow as well. orrorin: Hermes Agent compacts at 50% by default, so with 262K context most work happens at 60K to 150K.

Reported
Source: @matstatman, 2026-08-18
Source: @roger_dammit, 2026-08-18
Source: @mitakihara_., 2026-08-25
Source: @_orrorin, 2026-08-25

The speculative-decoding speedup collapses as context grows

Contested Everyone agrees on the effect; the cause was not agreed. The thread of 2026-08-26:

Zorg33 said a fix exists, but he applied it on Intel hardware. Source: @zorg33, 2026-08-26
cressida saw MTP and speculative decoding slow down after about 50k context, and planned a comparison on an RTX 3090 at full bandwidth. Source: @cressida1881, 2026-08-26
giaki3003 suggested the cause may be the limited PCIe bandwidth. Source: @giaki3003, 2026-08-26
replacedriver47 sees the same on V100s: MTP raises speed, but the speed drops massively as context grows. Source: @replacedriver47, 2026-08-26
iszo posted reference numbers for Qwen3.8-27B W4 on 2x RTX 5060 Ti (about 118 tok/s at 32k, 60 at 80k, 48 at 128k, 35 at 200k) and offered a one-command harness that logs TTFT, decode, recall, driver and PCIe state and Xids for a 4k/32k/80k/128k/200k matrix on a 64 GB 170HX. Source: @ianszo21, 2026-08-26

Link effects in general are on Parallelism over the PCIe link.

Sampling and temperature

The two pieces of advice below pull in opposite directions for agent use: temperature 0 keeps MTP and DSpark acceptance high, while 0.5 to 0.7 is what people use against tool-call loops. Nobody has reconciled them yet.

Send temperature 0 with MTP or DSpark, or draft acceptance collapses

matstatman found his low draft acceptance in Claude Code came from requests that did not send temperature=0; with MTP or DSpark set temperature 0 (or try 0.1), "otherwise they don't work" (2026-08-23). Measured with MTP 3 at temperature 0: mean acceptance length 3.6, averaged from the vLLM logs with:

docker logs qwen38 2>&1 | grep "Mean acceptance length" | cut -d ' ' -f12 | tr -d ',' | awk '{ sum += $1; n++ } END { if (n > 0) print "Sum:", sum, "| Average:", sum / n }'

PJ saw acceptance in the 60% range with MTP 3 at default temperature and Jimmy Dreambeard about 50%. In Hermes the same model can be added twice with different temperatures. Studebaker got his best single-stream decode, 122 tok/s, with temperature 0 and MTP 3. This is in tension with Ryko's view that loops in big models usually come from too low a temperature, such as 0, and with giaki3003's remark that some people fixed Qwen3.8 loops by changing temperature.

Reported
Source: @matstatman, 2026-08-23
Source: @jimmydreambeard, 2026-08-24
Source: @zingy90210, 2026-08-24
Source: @_studebaker_, 2026-08-29

Temperature 0 with MTP k=3 for agentic coding: speed against sampling quality

Contested The thread of 2026-08-24:

matstatman runs Qwen3.8 FP8 with MTP k=3 for coding in a Claude Code agent loop (reasoning effort xhigh) and says temperature must be forced to 0 for the MTP and DSpark hacks to work. DSpark is unusable for coding, and plain decoding without MTP is also good. He reports 100+ tok/s at 150k context in benchmarks and showed agents running at 40 to 80k context with no MTP or DSpark. Source: @matstatman, 2026-08-24
chonsworth estimated about 45 s TTFT at roughly 1,500 tok/s prefill for a 64k+ context from his chart. Source: @chonsworth, 2026-08-24
Ryko and segmond objected that temperature 0 trades quality for speed. Source: @vektor_prime, 2026-08-24
Source: @segmond, 2026-08-24
SpiralGlide said speculative decoding is only worth it below about 20 tok/s. Source: @spiralglide, 2026-08-24

Temperature 0.5 to 0.7 cuts tool-call loops and overthinking

DeeJay (2026-08-18): on DeepSeek V4, lowering temperature to 0.6 significantly improved tool calling; on 2026-09-01 he added that at temperature 1 Qwen and DeepSeek often got stuck in tool-call loops or errored, and 0.6 works well. Studebaker: try 0.5 to 0.6 when Qwen 27B "xhigh" thinks forever; he runs low thinking himself and had heard 0.7 cleans up some Qwen3.8 27B overthinking. Josephur turns thinking off for coding on Qwen 3.8 and blames loops on wrong sampling settings or poor 4-bit quants (he uses Unsloth UD quants); matstatman still sees loops on FP8 and BF16 and handles them with a hook plus a Python proxy. LT (2026-09-07): Qwen3.8 Flash Next used 22k, then nearly 25k thinking tokens to editorialise one paragraph at the default temperature 1.0, with terse non-sentence reasoning; he noted 0.7 is the recommended temperature even for non-thinking mode, and meatsus suggested 0.2 to 0.5 (untested). Loops on the DeepSeek ports have more causes than sampling: see DeepSeek V4 and V4.1: tool-call leaks, loops and crashes.

Proven
Source: @_studebaker_, 2026-08-18
Source: @deejay9983, 2026-08-18
Source: @josephur, 2026-09-04
Source: @matstatman, 2026-09-04
Source: @licensedterrapin, 2026-09-07
Source: @meatsus, 2026-09-07

Drafter ideas

Train, fine-tune or splice a better MTP drafter

Phoenix17 (2026-08-27): a better MTP drafter (higher acceptance or speed) can be trained or fine-tuned for a use case, and a drafter can be spliced from a different quant, or from a same-family MoE model if the dimensions match. DeeJay planned to train a GLM-5.3-Flash drafter with DSpark on rented H200s to push it past 100 tok/s, and Con3po asked about DSpark. Untested in this period.

Theory
Source: @deejay9983, 2026-08-27
Source: @phoenixone7, 2026-08-27
Source: @con3po7016, 2026-08-27

Engine support

Engine-wide notes are on Engines and llama.cpp on CMP cards; vLLM's MTP bugs on sm_80 are on vLLM on sm_80: known bugs and patches.

llama.cpp: DSpark speculative decoding (PR 25173), untested on CMP cards

On 2026-07-28, during the GLM-5.2 multi-card discussion, ju5t5omedude linked llama.cpp pull request 25173, "spec: add DSpark speculative decoding" by wjinxu, as possibly relevant to speeding up big models. No results on CMP cards were posted.

Theory
Source: repo github.com/ggml-org/llama.cpp by ggml-org on GitHub, posted by @ju5t5omedude, 2026-07-28

llama.cpp: adaptive MTP draft depth plus ngram-mod speculation (PR 27210)

On 2026-08-18 roger_dammit pointed to llama.cpp pull request 27210 (adaptive MTP draft depth, draft-mtp-adaptive) and hoped vLLM would get something similar. Ryko uses it with MTP min 3 and max 5 and says ngram-mod is "a huge boost for coding and independent from MTP":

--spec-type draft-mtp-adaptive,ngram-mod --spec-draft-n-min-adaptive 3 --spec-draft-n-max 5 --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-max 64

Reported
Source: repo github.com/ggml-org/llama.cpp by ggml-org on GitHub, posted by @roger_dammit, 2026-08-18
Source: @vektor_prime, 2026-08-18

Benchmarks, Engines, llama.cpp on CMP cards, vLLM on sm_80: known bugs and patches, Parallelism over the PCIe link, Quantization on the 170HX, Qwen3.8 27B on 1x 8 GB (W4A16 vLLM), Qwen3.6 27B and 35B-A3B on one card (July and August 2026), DeepSeek V4.1 Flash on 4x 8 GB (GNmac), DeepSeek V4.1 Flash on 8x 8 GB (Zanooda PP8), DeepSeek V4 Flash 0731 on 4x 8 GB (allover326 PP4 DSpark), GLM 5.3 Flash on 8x 8 GB (promisezackr PP8 NVFP4 DFlash2), GLM 5.3 Flash on 4x 8 GB (kaka86mm EXL3 + Marlin sidecar + DFlash2), DeepSeek V4 and V4.1: tool-call leaks, loops and crashes, Recipes.

MTP