Jump to content
Toggle search
Toggle menu
CMP Wiki
Toggle preferences menu
Toggle personal menu
Not logged in
Your IP address will be publicly visible if you make any edits.

DeepSeek V4.1 Flash on 8x 8 GB (Zanooda PP8)

From CMP Wiki
More actions
DeepSeek-V4.1-Flash-1017
Reported 8 x 8 GB
QuantOfficial checkpoint as downloaded: MXFP4 routed experts as Marlin W4A16, MXFP8 dense as Marlin W8A16; KV fp8
EnginevLLM PR #56201 head (79a7108d) plus a 20-patch sm_80 series; image zanooda/vllm-sm80-ds41f:v41-sm80 recipe 31318fd; patches 0001-0020
TopologyPP8, partition 5,5,5,5,5,5,5,5
Context1,048,576
Decode117 tok/s
Prefill6066 tok/s
VRAM per card64 GB; weights 35.9 GB per rank (ranks 1-6), 37.6 GB rank 0, 46.3 GB rank 7
Power cap180 W (one card unstable at 250 W)
Unlock buildcmpunlocker with the BAR1 P2P patches (64 GiB BAR1 per card)
Flags--pipeline-parallel-size 8 --max-model-len 1048576 --gpu-memory-utilization 0.95 --block-size 128 --kv-cache-dtype fp8 --max-num-batched-tokens 4096 --max-num-seqs 32; DSpark 5; VLLM_DSV41_SPREAD_DECODE=2; --engram-config storage cpu
Tested@Zanooda on 2026-10-19
Sourcelink
Confirmations
Partly reproduced by @darkmaniac7 (8 GB), 2026-10-27 · details

A complete serving recipe for DeepSeek-V4.1-Flash-1017 on eight 64 GB cards with vLLM pipeline parallel 8: a 20-patch series on the open vLLM PR #56201 plus haosdent's sm_80 layer, a prebuilt sm_80-only Docker image, launch scripts and benchmark harnesses. It serves the official checkpoint without conversion or EXL3, gives a 6.2M-token KV pool and 1M context, and has the best prefill of the DeepSeek recipes. It builds on allover326/deepseek-v4-cmp170hx and is independent of the Morrowmake recipes. Justinjja found it a bit faster than the alternatives at the time. Source: repo github.com/Zanooda/deepseek-v4.1-flash-cmp170hx by Zanooda on GitHub, posted by @gnasherrr, 2026-10-17
Source: @gnasherrr, 2026-10-17
Source: @justinjja, 2026-10-17

Prerequisites

  • Eight CMP 170HX at 64 GB (Driver unlock, 8 GB card). The author's were six at PCIe Gen2 x4 and two at x16, with no measured difference.
  • The checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 on local NVMe: 510 GB, 48 shards; shards 47 and 48 are the two 101 GB Engram tables.
  • About 240 GB of host RAM for the default Engram cpu mode (both tables pinned, 203 GB). With less RAM use storage=disk (patch 0010's C gather, runs on a 32 GB host) or the page-cache warmer.
  • Author's host: EPYC 7402P, 278 GB DDR4.

Source: repo github.com/Zanooda/deepseek-v4.1-flash-cmp170hx by Zanooda on GitHub, posted by @gnasherrr, 2026-10-17

Steps

git clone https://github.com/Zanooda/deepseek-v4.1-flash-cmp170hx ~/deepseek-v4-cmp170hx
# 1. the patched vLLM tree: PR #56201 head + all 20 patches
git clone https://github.com/vllm-project/vllm.git ~/vllm-v41 && cd ~/vllm-v41
git fetch origin pull/56201/head && git checkout -b v41-sm80 79a7108d9a
git am ~/deepseek-v4-cmp170hx/patches/v41/*.patch
# 2. the image: sm_80-only SASS, built from the PR head + patches 0001-0008
docker pull zanooda/vllm-sm80-ds41f:v41-sm80
# 3. checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 (510 GB, 48 shards) under /models, then
DSV41_VLLM_SRC=$HOME/vllm-v41/vllm ~/deepseek-v4-cmp170hx/launch/run-v41-pp8.sh

The image holds patches 0001 to 0008 only; 0009 to 0020 are Python-only and are bind-mounted through DSV41_VLLM_SRC. Without 0009 the engine does not start. Startup takes about 10 minutes. API on port 8099, served name dsv41, up to 8 images per request. Source: repo github.com/Zanooda/deepseek-v4.1-flash-cmp170hx by Zanooda on GitHub, posted by @gnasherrr, 2026-10-17

What the patches do (from the README): 0009 lets PP8 start (it replicates the KV-source and indexer caches across ranks); 0010 is a C gather for the Engram tables, 3.4x faster prefill than the Python preadv path; 0013 adds host-RAM Engram with exact-size pinning; 0019 spreads decode bursts over the pipeline; 0020 syncs the prompt encoder to the 1017 checkpoint, without which the engine emits an empty think block after every tool result and drifts into ad-hoc XML tool calls. With Engram read from NVMe, prefill at 105k is 4,455-4,930 tok/s (6,066 from RAM). Source: @darkmaniac7, 2026-10-19 (lab notes)

Field notes from Ivan's run:

  • The launcher bind-mounts the changed Python files by diffing your checkout against DSV41_IMG_COMMIT. Its default commit exists only in the author's repo; on another checkout the diff fails silently, nothing is mounted and the engine does not start. Set DSV41_IMG_COMMIT to the commit your own patched checkout was built from, and check that the launch prints "bind-mounting 24 changed python file(s)"; if it says 0, stop.
  • Pin the image by digest: zanooda/vllm-sm80-ds41f@sha256:283fc7b05791b41b804851b395d088cda91dcdc23e72d190563af772079a6b6e (the tag is mutable).
  • Engram cpu mode ran out of memory on a 125 GB host; use disk.
  • The API listens on 0.0.0.0 with no key; add --api-key through DSV41_EXTRA_ARGS if anything beyond your LAN can reach it.

Source: @darkmaniac7, 2026-10-27 (lab notes)

Measured numbers

Author's numbers (measured 2026-09-10/11 on the pre-1017 weights; config and index are byte-identical for 1017):

  • Decode: 117 tok/s single stream at 128k context, 96 at 512k (Engram pinned in RAM). Step cadence flat at 40 to 56 ms from 4k to 115k prompt tokens; 3.9 (prose) to 5.9 (repetitive text) tokens per step.
  • Multi-stream with patch 0019: 8 streams 560 tok/s aggregate warm (stock scheduler 274), 16 streams 806 to 813.
  • Prefill (Engram in RAM, patch 0013): 26,208 tokens in 5.8 s (4,486 tok/s), 104,881 in 17.3 s (6,066), 419,430 in 78.7 s (5,328); a 1,007,820-token prompt in 468 s; eight concurrent 105k prompts about 6,900 tok/s aggregate.
  • KV pool 6,171,394 tokens at 0.95 (5.9 concurrent 1M requests).
  • Correctness: needle-in-haystack 19/19 from 4k to 1M (820k real tokens), 4 concurrent needles with no cross-request bleed, tool calls verified, vision works.

Source: repo github.com/Zanooda/deepseek-v4.1-flash-cmp170hx by Zanooda on GitHub, posted by @gnasherrr, 2026-10-17

  • Ivan, 8 cards at Gen2 x16 with P2P, 1,024-token answers: at 180 W one user code 64 / structured 46 / prose 32 tok/s, 8 users 546 / 316 / 285 aggregate, prefill 5,478 tok/s on a 17k prompt; at 150 W 65 / 50 / 34, 533 / 296 / 284, prefill 5,276, up in 8 minutes, 1M context with a 5.56M-token KV pool, image understanding correct, 0 Xids. Engram in RAM was exactly as fast as from NVMe. Single-stream decode is well below the author's; aggregate and prefill match. Source: @darkmaniac7, 2026-10-27 (lab notes)

Known issues

  • Warning NVMe drop under the Engram disk stream. On Ivan's box, about a minute after the first PP8 benchmark with the Engram tables read from disk (189 GiB by 32 threads, because 125 GB of RAM was too little for cpu mode), the 2 TB NVMe holding the OS and models fell off and the box would not boot. The drive sat on a PCIe-to-M.2 sled with no heatsink or airflow. A reseat brought it back with no SMART errors, but the system journal was lost with the filesystem, so overheating is the best explanation, not proof. With a 120 mm fan on the sled the drive later held 31 to 36 C under the same loads (it had reached 71 to 81 C before). Cool the NVMe, cap its read rate, or keep the tables in RAM. See Cooling and power and Troubleshooting. Source: @darkmaniac7, 2026-10-27 (lab notes)
  • One of the author's cards fell off the bus (Xid 79) three times under sustained load at the stock 250 W limit and was stable at nvidia-smi -pl 180; nvidia-smi reports up to 333 W against a 250 W limit during prefill on every card.
  • PWRBRK# trap: some boards (ASUS Pro WS WRX80E-SAGE) assert PWRBRK# on edge pin B30 and pin the card at about 88 W / 1,140 MHz. Check with nvidia-smi -q | grep -A1 "HW Power Brake Slowdown" ("Active" on an idle card means braked); fix with Kapton tape over B30 or a riser that does not route B30.
  • P2P with the BAR1 patches gave peer copies of 1.55 GB/s and routed every pipeline hop GPU to GPU, with no measurable speed change (hops are bound by the Gen2 x4 link). Tensor parallel measured 6.6x slower on prefill; --enforce-eager gives 8 to 10 tok/s.
  • --block-size 256 fails KV allocation; a last rank with fewer than 4 layers cannot start (the drafter needs layers 36 to 38).
  • Keep VLLM_DSV41_SPREAD_DECODE=2: the stock scheduler packs a burst into one micro-batch (274 against 560 tok/s at 8 streams).
  • reasoning_effort is a 1 to 100 budget; low, high, xhigh, max map to 25, 50, 75, 100 (upstream maps high to 75); none disables thinking; minimal and medium return HTTP 400.
  • The base vLLM PR #56201 is open and may be force-pushed.

Source: repo github.com/Zanooda/deepseek-v4.1-flash-cmp170hx by Zanooda on GitHub, posted by @gnasherrr, 2026-10-17

Related

Sources

  • Repo: Zanooda/deepseek-v4.1-flash-cmp170hx (Apache-2.0), last commit 2026-10-19. Credit also to allover326/deepseek-v4-cmp170hx and haosdent/vllm dsv4-flash-a100. Posted on Discord by Gnasherrr.
  • Ivan's lab notes, 2026-10-27.
Imported from the cmpunlocker Discord with credit; see Credits.
Retrieved from "https://cmp.wiki/w/index.php?title=DeepSeek_V4.1_Flash_on_8x_8_GB_(Zanooda_PP8)&oldid=259"