View source for DeepSeek V4.1 Flash on 6x 8 GB (TP6)

{{Recipe
|model=DeepSeek-V4.1-Flash-1017
|quant=Official checkpoint: dense MXFP8 through an SM80 Marlin W8A16 path, routed experts MXFP4 through W4A16; KV FP8
|engine=Custom SM80 vLLM (patched from public commit 79a7108d9)
|engine_version=reports 0.0.0; patches/public-base-to-runtime.patch (109 files)
|cards=8 GB
|card_count=6
|topology=TP6 (virtual-head padding), PP1
|context=1,048,576
|decode_tps=78.55
|vram_per_card=64 GB, --gpu-memory-utilization 0.95
|flags=--tensor-parallel-size 6 --kv-cache-dtype fp8 --block-size 128 --max-num-seqs 16 --max-num-batched-tokens 2048 --engram-config storage disk --speculative-config dspark 5; --hf-overrides virtual heads 96 / o_groups 12
|status=Reported
|tested_by=wgkbfcp
|tested_on=2026-10-19
|source=https://github.com/wgkbfcp/deepseek-v4.1-flash-1017-cmp170hx-recipe
}}
A deployment record for DeepSeek-V4.1-Flash-1017 on six 64 GB cards at tensor-parallel 6, with the DSpark drafter at
K=5, including a full snapshot of the author's patched SM80 vLLM source. It is an evidence package rather than an
installer: the serving stack starts through a private local console that is not published, and the host is Windows
with WSL2. The README is in Chinese. Infobox decode is end-to-end tok/s on code including reasoning tokens.
{{Source|handle=dale063629|date=2026-10-19|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561547352693538856|repo=https://github.com/wgkbfcp/deepseek-v4.1-flash-1017-cmp170hx-recipe}}<br />
{{Source|handle=dale063629|date=2026-10-19|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561547352693538856}}

== Why TP6 is unusual ==
{{Status|Contested}} vLLM requires the TP size to divide the model's attention head count; DeepSeek V4.1 has 16 heads
per rank, so only TP 1, 2, 4 and 8 work, and six cards normally mean TP2 x PP3. Miséricorde: you cannot run TP6, and six
cards may be worse than four. This recipe gets TP6 anyway by padding the real 64 heads / 8 groups to 96 / 12 (whole
groups duplicated up to WO_A, the extra WO_B columns zeroed). The same override on unmodified vLLM does not work.
{{Source|handle=greencandle9753|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562580021153894501}}<br />
{{Source|handle=wakeupmbrc1|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560998880940916757}}<br />
{{Source|handle=dale063629|date=2026-10-19|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561547352693538856|repo=https://github.com/wgkbfcp/deepseek-v4.1-flash-1017-cmp170hx-recipe}}

== Prerequisites ==
* Six CMP 170HX at 64 GB, all at PCIe Gen2 x16 on the author's host ([[Driver unlock]], [[8 GB card]]).
* Author's host: Threadripper PRO 3995WX, about 512 GiB RAM; Windows with WSL2 Ubuntu (256 GB WSL memory, swap 0), weights on an ext4 VHDX mounted read-only into WSL.
* The official checkpoint [https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017 deepseek-ai/DeepSeek-V4.1-Flash-1017] at <code><nowiki>1bae0e10</nowiki></code> (48 shards, 510,296,708,312 bytes). The Engram disk table (203 GB) is warmed into the Linux page cache.
* <code><nowiki>build-essential</nowiki></code>: Engram's C helper is compiled at runtime.
{{Source|handle=dale063629|date=2026-10-19|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561547352693538856|repo=https://github.com/wgkbfcp/deepseek-v4.1-flash-1017-cmp170hx-recipe}}

== Steps ==
Not reproducible as published: the launch goes through a private console. The serve arguments it records:
<syntaxhighlight lang="bash">
vllm serve /model --tensor-parallel-size 6 --pipeline-parallel-size 1 --data-parallel-size 1 \
  --kv-cache-dtype fp8 --block-size 128 --max-model-len 1048576 \
  --max-num-batched-tokens 2048 --max-num-seqs 16 --gpu-memory-utilization 0.95 \
  --engram-config '{"storage":"disk","disk_threads":32}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5}' \
  --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser deepseek_v41 --reasoning-parser deepseek_v41 \
  --limit-mm-per-prompt '{"image":1048576}' --no-enable-flashinfer-autotune --mm-encoder-tp-mode data \
  --hf-overrides '{"num_attention_heads":96,"o_groups":12,"virtual_heads_from":{"num_attention_heads":64,"o_groups":8}}'
</syntaxhighlight>
Environment: <code><nowiki>VLLM_PP_LAYER_PARTITION=40</nowiki></code>, <code><nowiki>DSV41_SM80_INDEXER_PREFILL_SEQ_CAP=1</nowiki></code>,
<code><nowiki>DSV41_SM80_PREFILL_CHUNK_SIZE=2</nowiki></code> (only 2 or 4 accepted; 2 only for this exact TP6 profile),
<code><nowiki>DSV4_LOGITS_ROW_CHUNK=64</nowiki></code>, <code><nowiki>VLLM_DSV41_SPREAD_DECODE=2</nowiki></code>, <code><nowiki>VLLM_USE_BREAKABLE_CUDAGRAPH=1</nowiki></code>,
<code><nowiki>VLLM_WSL2_ENABLE_PIN_MEMORY=1</nowiki></code>, <code><nowiki>VLLM_WORKER_MULTIPROC_METHOD=spawn</nowiki></code>,
<code><nowiki>VLLM_ENGINE_READY_TIMEOUT_S=7200</nowiki></code>, <code><nowiki>PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False</nowiki></code>,
<code><nowiki>HF_HUB_OFFLINE=1</nowiki></code>. CUDA graphs FULL_AND_PIECEWISE (largest capture seen: 192 tokens). Software: torch 2.13.0,
triton 3.7.1, flashinfer-python 0.6.18, tilelang 0.1.12, transformers 5.17.0, Python 3.12.3, CUDA 13.0.2,
<code><nowiki>TORCH_CUDA_ARCH_LIST=8.0</nowiki></code>. <code><nowiki>build/Dockerfile.source</nowiki></code> is a candidate rebuild that has never been run.
{{Source|handle=dale063629|date=2026-10-19|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561547352693538856|repo=https://github.com/wgkbfcp/deepseek-v4.1-flash-1017-cmp170hx-recipe}}

== Measured numbers ==
2026-10-19, single request, Engram warm, prefix cache not reset, <code><nowiki>reasoning_effort=max</nowiki></code>, temperature 1:
K=5 end to end including reasoning tokens 78.55 tok/s on code and 76.56 tok/s on writing (per request 61.26 to 84.76).
Final-answer phase on code estimated at 188.25 tok/s (183.97 to 193.19) from stream timestamps over short windows. Not
cold-start, not multi-user, not full 1M-context numbers.

K=5 against K=3, 24 samples: K3 gave +1.64% token rate but +7.69% completion time, below the 5% switch threshold, so
K=5 stays. Draft acceptance K5 28.32% code / 27.34% writing against K3 41.42% / 43.36%. Code 6/6 passed (68 unit tests)
in both arms; writing 4/6 (K5) against 5/6 (K3), non-blinded review.
{{Source|handle=dale063629|date=2026-10-19|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561547352693538856|repo=https://github.com/wgkbfcp/deepseek-v4.1-flash-1017-cmp170hx-recipe}}

* The scheduler patch's notes record a measurement on 8 cards at PP8: a burst admitted in one step ran 274 tok/s aggregate (34 per stream) against 531 and 66 with requests spread over the pipeline ring (<code><nowiki>VLLM_DSV41_SPREAD_DECODE</nowiki></code>).

== Other six-card results ==
* TP2 x PP3 with the Deepinfer-SM80 patches: see [[DeepSeek V4.1 Flash on 6x 8 GB (Deepinfer-SM80 TP2 x PP3)]].
* Con3po: 157 tok/s prose at concurrency 1 on six cards (quant not stated). {{Source|handle=con3po7016|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562304093228245125}}
* DeeJay: about 67 tok/s decode on six cards, PP6, x4 links, native quant, still tuning. {{Source|handle=deejay9983|date=2026-10-22|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562929048248909906}}
* {{Status|Reported}} OB88594: "6x single streams" run at about 5,000 tok/s prefill and 80 to 100 tok/s decode on DeepSeek V4.1 Flash 1017, and cloud services give 2 to 3x that. It is not clear whether he means six cards or six streams; engine, quant and layout were not posted. Summary of his posts; see [https://discord.com/channels/1518634379604136178/1518692084729511967/1563649833594650828 Discord]. {{Source|handle=ob88594|date=2026-10-24|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563649833594650828}}<br />{{Source|handle=ob88594|date=2026-10-24|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563649999969976444}}<br />{{Source|handle=ob88594|date=2026-10-24|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1563650410860908605}}
* intervitens: a sixth card buys native-precision V4.1 Flash, with TP2 x PP3. {{Source|handle=intervitens|date=2026-10-22|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562719778769080371}}
* roboalchemist is packing an FP4-weight mix of 1017 sized for six cards, to be published on his Hugging Face page. {{Source|handle=roboalchemist.|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1561155845344198687}}

== Known issues ==
* Launch depends on a private console and host-specific guards (at least 400 GiB free commit before load, 99% residency per Engram table). Windows + WSL2 specific. No license for the recipe files (the vendored vLLM keeps Apache-2.0).
* Engram warmed into the page cache is reclaimable, not locked; swap-out degrades speed.
{{Source|handle=dale063629|date=2026-10-19|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561547352693538856|repo=https://github.com/wgkbfcp/deepseek-v4.1-flash-1017-cmp170hx-recipe}}

== Sources ==
* Repo: [https://github.com/wgkbfcp/deepseek-v4.1-flash-1017-cmp170hx-recipe wgkbfcp/deepseek-v4.1-flash-1017-cmp170hx-recipe] (translated from Chinese), last commit 2026-10-19. Posted on Discord by 大乐.
* Related: [[Engines]], [[Recipes]], [[Benchmarks]].

{{ImportFooter}}
[[Category:8 GB]]