View source for DeepSeek V4.1 Flash on 8x 8 GB (Zanooda PP8)

{{Recipe
|model=DeepSeek-V4.1-Flash-1017
|quant=Official checkpoint as downloaded: MXFP4 routed experts as Marlin W4A16, MXFP8 dense as Marlin W8A16; KV fp8
|engine=vLLM PR #56201 head (79a7108d) plus a 20-patch sm_80 series; image zanooda/vllm-sm80-ds41f:v41-sm80
|engine_version=recipe 31318fd; patches 0001-0020
|cards=8 GB
|card_count=8
|topology=PP8, partition 5,5,5,5,5,5,5,5
|context=1,048,576
|decode_tps=117
|prefill_tps=6066
|vram_per_card=64 GB; weights 35.9 GB per rank (ranks 1-6), 37.6 GB rank 0, 46.3 GB rank 7
|power_cap=180 W (one card unstable at 250 W)
|unlock_build=cmpunlocker with the BAR1 P2P patches (64 GiB BAR1 per card)
|flags=--pipeline-parallel-size 8 --max-model-len 1048576 --gpu-memory-utilization 0.95 --block-size 128 --kv-cache-dtype fp8 --max-num-batched-tokens 4096 --max-num-seqs 32; DSpark 5; VLLM_DSV41_SPREAD_DECODE=2; --engram-config storage cpu
|status=Reported
|tested_by=Zanooda
|tested_on=2026-10-19
|source=https://github.com/Zanooda/deepseek-v4.1-flash-cmp170hx
}}
A complete serving recipe for DeepSeek-V4.1-Flash-1017 on eight 64 GB cards with vLLM pipeline parallel 8: a 20-patch
series on the open vLLM PR #56201 plus haosdent's sm_80 layer, a prebuilt sm_80-only Docker image, launch scripts and
benchmark harnesses. It serves the official checkpoint without conversion or EXL3, gives a 6.2M-token KV pool and 1M
context, and has the best prefill of the DeepSeek recipes. It builds on allover326/deepseek-v4-cmp170hx and is
independent of the Morrowmake recipes. Justinjja found it a bit faster than the alternatives at the time.
{{Source|handle=gnasherrr|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560998024136753172|repo=https://github.com/Zanooda/deepseek-v4.1-flash-cmp170hx}}<br />
{{Source|handle=gnasherrr|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560998024136753172}}<br />
{{Source|handle=justinjja|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561016129550286868}}

== Prerequisites ==
* Eight CMP 170HX at 64 GB ([[Driver unlock]], [[8 GB card]]). The author's were six at PCIe Gen2 x4 and two at x16, with no measured difference.
* The checkpoint [https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017 deepseek-ai/DeepSeek-V4.1-Flash-1017] on local NVMe: 510 GB, 48 shards; shards 47 and 48 are the two 101 GB Engram tables.
* About 240 GB of host RAM for the default Engram <code><nowiki>cpu</nowiki></code> mode (both tables pinned, 203 GB). With less RAM use <code><nowiki>storage=disk</nowiki></code> (patch 0010's C gather, runs on a 32 GB host) or the page-cache warmer.
* Author's host: EPYC 7402P, 278 GB DDR4.
{{Source|handle=gnasherrr|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560998024136753172|repo=https://github.com/Zanooda/deepseek-v4.1-flash-cmp170hx}}

== Steps ==
<syntaxhighlight lang="bash">
git clone https://github.com/Zanooda/deepseek-v4.1-flash-cmp170hx ~/deepseek-v4-cmp170hx
# 1. the patched vLLM tree: PR #56201 head + all 20 patches
git clone https://github.com/vllm-project/vllm.git ~/vllm-v41 && cd ~/vllm-v41
git fetch origin pull/56201/head && git checkout -b v41-sm80 79a7108d9a
git am ~/deepseek-v4-cmp170hx/patches/v41/*.patch
# 2. the image: sm_80-only SASS, built from the PR head + patches 0001-0008
docker pull zanooda/vllm-sm80-ds41f:v41-sm80
# 3. checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 (510 GB, 48 shards) under /models, then
DSV41_VLLM_SRC=$HOME/vllm-v41/vllm ~/deepseek-v4-cmp170hx/launch/run-v41-pp8.sh
</syntaxhighlight>
The image holds patches 0001 to 0008 only; 0009 to 0020 are Python-only and are bind-mounted through
<code><nowiki>DSV41_VLLM_SRC</nowiki></code>. Without 0009 the engine does not start. Startup takes about 10 minutes. API on port 8099,
served name <code><nowiki>dsv41</nowiki></code>, up to 8 images per request.
{{Source|handle=gnasherrr|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560998024136753172|repo=https://github.com/Zanooda/deepseek-v4.1-flash-cmp170hx}}

What the patches do (from the README): 0009 lets PP8 start (it replicates the KV-source and indexer caches across
ranks); 0010 is a C gather for the Engram tables, 3.4x faster prefill than the Python preadv path; 0013 adds host-RAM
Engram with exact-size pinning; 0019 spreads decode bursts over the pipeline; 0020 syncs the prompt encoder to the 1017
checkpoint, without which the engine emits an empty think block after every tool result and drifts into ad-hoc XML
tool calls. With Engram read from NVMe, prefill at 105k is 4,455-4,930 tok/s (6,066 from RAM).
{{Source|handle=darkmaniac7|date=2026-10-19|note=lab notes}}

Field notes from Ivan's run:
* The launcher bind-mounts the changed Python files by diffing your checkout against <code><nowiki>DSV41_IMG_COMMIT</nowiki></code>. Its default commit exists only in the author's repo; on another checkout the diff fails silently, nothing is mounted and the engine does not start. Set <code><nowiki>DSV41_IMG_COMMIT</nowiki></code> to the commit your own patched checkout was built from, and check that the launch prints "bind-mounting 24 changed python file(s)"; if it says 0, stop.
* Pin the image by digest: <code><nowiki>zanooda/vllm-sm80-ds41f@sha256:283fc7b05791b41b804851b395d088cda91dcdc23e72d190563af772079a6b6e</nowiki></code> (the tag is mutable).
* Engram <code><nowiki>cpu</nowiki></code> mode ran out of memory on a 125 GB host; use <code><nowiki>disk</nowiki></code>.
* The API listens on 0.0.0.0 with no key; add <code><nowiki>--api-key</nowiki></code> through <code><nowiki>DSV41_EXTRA_ARGS</nowiki></code> if anything beyond your LAN can reach it.
{{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}

== Measured numbers ==
Author's numbers (measured 2026-09-10/11 on the pre-1017 weights; config and index are byte-identical for 1017):
* Decode: 117 tok/s single stream at 128k context, 96 at 512k (Engram pinned in RAM). Step cadence flat at 40 to 56 ms from 4k to 115k prompt tokens; 3.9 (prose) to 5.9 (repetitive text) tokens per step.
* Multi-stream with patch 0019: 8 streams 560 tok/s aggregate warm (stock scheduler 274), 16 streams 806 to 813.
* Prefill (Engram in RAM, patch 0013): 26,208 tokens in 5.8 s (4,486 tok/s), 104,881 in 17.3 s (6,066), 419,430 in 78.7 s (5,328); a 1,007,820-token prompt in 468 s; eight concurrent 105k prompts about 6,900 tok/s aggregate.
* KV pool 6,171,394 tokens at 0.95 (5.9 concurrent 1M requests).
* Correctness: needle-in-haystack 19/19 from 4k to 1M (820k real tokens), 4 concurrent needles with no cross-request bleed, tool calls verified, vision works.
{{Source|handle=gnasherrr|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560998024136753172|repo=https://github.com/Zanooda/deepseek-v4.1-flash-cmp170hx}}

* Ivan, 8 cards at Gen2 x16 with P2P, 1,024-token answers: at 180 W one user code 64 / structured 46 / prose 32 tok/s, 8 users 546 / 316 / 285 aggregate, prefill 5,478 tok/s on a 17k prompt; at 150 W 65 / 50 / 34, 533 / 296 / 284, prefill 5,276, up in 8 minutes, 1M context with a 5.56M-token KV pool, image understanding correct, 0 Xids. Engram in RAM was exactly as fast as from NVMe. Single-stream decode is well below the author's; aggregate and prefill match. {{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}

== Known issues ==
* {{Status|Warning}} '''NVMe drop under the Engram disk stream.''' On Ivan's box, about a minute after the first PP8 benchmark with the Engram tables read from disk (189 GiB by 32 threads, because 125 GB of RAM was too little for <code><nowiki>cpu</nowiki></code> mode), the 2 TB NVMe holding the OS and models fell off and the box would not boot. The drive sat on a PCIe-to-M.2 sled with no heatsink or airflow. A reseat brought it back with no SMART errors, but the system journal was lost with the filesystem, so overheating is the best explanation, not proof. With a 120 mm fan on the sled the drive later held 31 to 36 C under the same loads (it had reached 71 to 81 C before). Cool the NVMe, cap its read rate, or keep the tables in RAM. See [[Cooling and power]] and [[Troubleshooting]]. {{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}
* One of the author's cards fell off the bus (Xid 79) three times under sustained load at the stock 250 W limit and was stable at <code><nowiki>nvidia-smi -pl 180</nowiki></code>; <code><nowiki>nvidia-smi</nowiki></code> reports up to 333 W against a 250 W limit during prefill on every card.
* PWRBRK# trap: some boards (ASUS Pro WS WRX80E-SAGE) assert PWRBRK# on edge pin B30 and pin the card at about 88 W / 1,140 MHz. Check with <code><nowiki>nvidia-smi -q | grep -A1 "HW Power Brake Slowdown"</nowiki></code> ("Active" on an idle card means braked); fix with Kapton tape over B30 or a riser that does not route B30.
* P2P with the BAR1 patches gave peer copies of 1.55 GB/s and routed every pipeline hop GPU to GPU, with no measurable speed change (hops are bound by the Gen2 x4 link). Tensor parallel measured 6.6x slower on prefill; <code><nowiki>--enforce-eager</nowiki></code> gives 8 to 10 tok/s.
* <code><nowiki>--block-size 256</nowiki></code> fails KV allocation; a last rank with fewer than 4 layers cannot start (the drafter needs layers 36 to 38).
* Keep <code><nowiki>VLLM_DSV41_SPREAD_DECODE=2</nowiki></code>: the stock scheduler packs a burst into one micro-batch (274 against 560 tok/s at 8 streams).
* <code><nowiki>reasoning_effort</nowiki></code> is a 1 to 100 budget; low, high, xhigh, max map to 25, 50, 75, 100 (upstream maps high to 75); none disables thinking; minimal and medium return HTTP 400.
* The base vLLM PR #56201 is open and may be force-pushed.
{{Source|handle=gnasherrr|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560998024136753172|repo=https://github.com/Zanooda/deepseek-v4.1-flash-cmp170hx}}

== Related ==
* Same card count, other engine: [[DeepSeek V4.1 Flash on 8x 8 GB (Deepinfer-SM80 TP4 x PP2)]].
* Four-card recipes: [[DeepSeek V4.1 Flash on 4x 8 GB (Morrowmake)]], [[DeepSeek V4.1 Flash on 4x 8 GB (GNmac)]].
* [[Engines]], [[Benchmarks]], [[Tools]].

== Sources ==
* Repo: [https://github.com/Zanooda/deepseek-v4.1-flash-cmp170hx Zanooda/deepseek-v4.1-flash-cmp170hx] (Apache-2.0), last commit 2026-10-19. Credit also to allover326/deepseek-v4-cmp170hx and haosdent/vllm dsv4-flash-a100. Posted on Discord by Gnasherrr.
* Ivan's lab notes, 2026-10-27.

{{ImportFooter}}
[[Category:8 GB]]