DeepSeek V4.1 Flash on 5x 8 GB (PP5 full residency)

DeepSeek-V4.1-Flash
Reported 5 x 8 GB
Quant	Official checkpoint (510 GB: MXFP4 routed experts, FP8 dense layers, 203 GB FP8 Engram table); KV cache fp8_ds_mla
Engine	vLLM: 344303947's fork of wtdcode/vllm-backport (shadow source layer, staged Marlin repack, exact-size Engram pinning) v0.13.1.dev6, tag v41-pp5-1m-pinmem-20260916
Topology	PP5 with an even 8,8,8,8,8 layer split; the repo's cards are on PCIe Gen2 x16 with no NVLink
Context	1,048,576 (KV pool 1,632,827 tokens at utilisation 0.97; an 896K-token prompt passed)
Prefill	4600 tok/s
VRAM per card	64 GB, full residency (no expert offload), --gpu-memory-utilization 0.97
Flags	`start_vllm_ds4.1.sh --maxlen 1048576 --gpu-util 0.97 --offload-ranks 0,0,0,0,0; vLLM --pipeline-parallel-size 5 --tensor-parallel-size 1 --kv-cache-dtype fp8_ds_mla --max-num-batched-tokens 2048 --max-num-seqs 8 --enable-prefix-caching --disable-custom-all-reduce, Engram CPU offload; env VLLM_PP_LAYER_PARTITION=8,8,8,8,8, VLLM_ENGRAM_EXACT_PIN (default on), VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=128, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, NCCL_ALGO=Ring, NCCL_PROTO=Simple`
Tested	@344303947 on 2026-09-22
Source	link
Files	dsv41-pp5/ (Chinese docs: README.md, ARCHITECTURE.md, CHANGELOG.md, all-changes.patch, start_vllm_ds4.1.sh; Apache-2.0)
Confirmations	None yet.

DeepSeek-V4.1-Flash on five 64 GB cards with every expert in VRAM, served by a fork of wtdcode/vllm-backport with pipeline parallel 5 and a 1,048,576-token context. thcleaner (dkpoulsen) shared the repo on 2026-09-22 and reported 70+ tok/s on 5x 170HX with 256 GB of DDR4 system RAM. The layout is pipeline parallel only: tensor parallel 5 is impossible because the model's 64 attention heads and 1 KV head do not divide by 5, and full residency leaves no memory for DSpark or MTP. The 320-expert variant further down prunes the model to make room for DSpark. The decode speed is not settled: the repo's own docs give about 30 to 40 tok/s (see Measured numbers). The infobox shows the repo's readings; prefill is the 60K-token figure.

Reported 8 GB
Source: repo github.com/344303947/dsv41-flash-pp5-170hx by 344303947 on GitHub, posted by @dkpoulsen, 2026-09-22

Steps

What you need

Exactly five CMP 170HX 8 GB cards with the VRAM unlock (Driver unlock, 8 GB card), each showing 64 GB. The repo writes the cards as "A100-64G" on PCIe Gen2 x16 with no NVLink. Its launch script refuses any other card count, and it warns and exits if any of the five cards already has more than 4 GB in use (stop the other job first).
Host RAM for the Engram tables. The two tables (2x 94.4 GiB) are pinned in host memory: 189.1 GiB with the repo's exact-size pinning, 264.3 GiB without. Pinned memory cannot be reclaimed or swapped, so the saving is headroom against OOM, not free memory. The repo's figures come from a 503 GB host; the Discord post gives 256 GB of DDR4.
The official deepseek-ai/DeepSeek-V4.1-Flash checkpoint, 510 GB including the FP8 Engram table, on a local disk.
The engine: the repo is the complete modified vLLM source (wtdcode/vllm-backport plus local commits), built into its own Python environment. The docs use a conda environment named vllm-v41. The build script the launcher mentions (build_vllm_v41.sh) is not in the repo.

Reported 8 GB
Source: repo github.com/344303947/dsv41-flash-pp5-170hx by 344303947 on GitHub, posted by @dkpoulsen, 2026-09-22

Launch with start_vllm_ds4.1.sh

Build the fork from 344303947/dsv41-flash-pp5-170hx into a Python environment. Use tag v41-pp5-1m-pinmem-20260916 for the state the numbers below were measured on.
Copy dsv41-pp5/start_vllm_ds4.1.sh and edit the variables at the top: engine source path, conda environment, model directory and API key. The published script carries its author's own values; set your own key.
Run start_vllm_ds4.1.sh --bg. It serves on port 9004 and writes logs to ~/logs/. The defaults are --maxlen 1048576, --gpu-util 0.97 and --offload-ranks 0,0,0,0,0 (all ranks resident). --dspark is untested, and --legacy-offload brings back the old 2,6,6,6,20 split with expert offload.
Wait for the load: about 327 s on rank 0 and 344 s on rank 1 with exact pinning. The script serves the model as DeepSeek-V4.1-Flash on an OpenAI-compatible endpoint, with the deepseek_v41 tokenizer, tool-call and reasoning parsers.
To trade VRAM for more KV room (contexts past about 70K or several streams at once), give ranks an expert offload budget with --offload-ranks (sets VLLM_CPU_OFFLOAD_GB_PER_RANK). The short-answer time grows from 1.5 s to 13.3 s with offload.

What the script runs: --pipeline-parallel-size 5 --tensor-parallel-size 1 --kv-cache-dtype fp8_ds_mla --max-num-batched-tokens 2048 --max-num-seqs 8 --enable-prefix-caching --disable-custom-all-reduce --engram-config '{"cpu_offload":true}', with VLLM_PP_LAYER_PARTITION=8,8,8,8,8, VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=128 (smaller indexer chunks, so long prefills do not OOM in the thin memory margin), PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, NCCL_ALGO=Ring and NCCL_PROTO=Simple. Summarised from the Chinese docs and script.

Reported 8 GB
Source: repo github.com/344303947/dsv41-flash-pp5-170hx by 344303947 on GitHub, posted by @dkpoulsen, 2026-09-22

How it fits on five cards

Shadow source layer: an even 8,8,8,8,8 split instead of 2,6,6,6,20

V4.1 shares KV between groups of layers, and stock vLLM will not cut the pipeline inside a group. Layers 20 to 39 form one group with 147.8 GB of weights, so every split it allows overloads one card: the legacy 2,6,6,6,20 puts 20 layers on rank 4, which has to offload 102 GB of experts and decodes at only 10 to 16 tok/s. The fork's "shadow source layer" lets a consumer rank re-run a bit-identical replica of the source layer's attention from a relayed input of about 10 KB per token. With that, the pipeline splits evenly as 8,8,8,8,8.

Reported 8 GB
Source: repo github.com/344303947/dsv41-flash-pp5-170hx by 344303947 on GitHub, posted by @dkpoulsen, 2026-09-22

Staged Marlin repack: zero expert offload without a load-time OOM

Repacking MXFP4 experts into Marlin format normally holds the raw and the packed copy of a layer at once, a transient of about 6.7 GiB per layer. With every expert resident that peak does not fit. The fork repacks expert by expert, so full residency (0 offload) loads.

Reported 8 GB
Source: repo github.com/344303947/dsv41-flash-pp5-170hx by 344303947 on GitHub, posted by @dkpoulsen, 2026-09-22

Exact-size Engram pinning: 264.3 GiB down to 189.1 GiB of pinned host memory

VLLM_ENGRAM_EXACT_PIN (default on) allocates the Engram tables with cudaHostAlloc at their exact size instead of going through torch's pinned allocator, which rounds up to a power of two. Measured on the 503 GB host on 2026-09-16: pinned memory fell from 264.3 to 189.1 GiB, the per-rank block from 132.00 to 94.42 GiB, and available memory rose from 211 to 288 GiB. Model load got faster (362 / 378 s to 327 / 344 s for ranks 0 and 1), while the KV pool and speed did not change (decode 41.06 vs 41.10 tok/s, prefill 2,826 vs 2,832 tok/s). The Engram bytes are sha256-identical before and after. Set VLLM_ENGRAM_EXACT_PIN=0 and restart to go back to the torch allocator.

Reported 8 GB
Source: repo github.com/344303947/dsv41-flash-pp5-170hx by 344303947 on GitHub, posted by @dkpoulsen, 2026-09-22

Measured numbers

Full residency: 1.5 s short answer, about 4.6k tok/s prefill at 60K, 1M context

All five ranks resident at utilisation 0.97, from the repo:

Short Q&A: 1.5 s (13.3 s with the earlier expert-offload configuration).
60K-token prompt: prefill in 12.5 s (about 4.6k tok/s).
896K-token prompt: 388 s (about 2.3k tok/s).
KV pool: 1,632,827 tokens, 1.56x concurrency at 1M.
Output is deterministic at temperature 0.

KV is fp8_ds_mla (576 B per token per layer) on the V1 runner, with prefix caching and chunked prefill at 2048. The A100 code path has no DeepGEMM, so dense MXFP8 falls back to Triton.

Reported 8 GB
Source: repo github.com/344303947/dsv41-flash-pp5-170hx by 344303947 on GitHub, posted by @dkpoulsen, 2026-09-22

Decode: about 30 to 40 tok/s in the repo, 70+ tok/s in the Discord post

The repo's architecture notes give decode at about 30 to 40 tok/s (one log line reads 29.6 tok/s generation), and its pinned-memory comparison measured 41.06 tok/s. thcleaner's Discord post says 70+ tok/s. On thcleaner's own five cards the unpruned 384-expert model without DSpark, on the same engine, decoded at 42.4 tok/s, which sits with the repo's figure. Nobody has run this recipe on a second five-card host yet, so the infobox carries no decode number.

Reported 8 GB
Source: repo github.com/344303947/dsv41-flash-pp5-170hx by 344303947 on GitHub, posted by @dkpoulsen, 2026-09-22
Source: repo github.com/dkpoulsen/dsv41-320e-dspark by dkpoulsen on GitHub, posted by @dkpoulsen, 2026-09-23

The 320-expert DSpark variant

Why prune: the DSpark head does not fit the last stage

DSpark's three-stage draft head is 7.39 GiB (about 8.2 GiB with its KV and CUDA graphs) and has to sit on the last pipeline stage, which also holds the model head. With the full model that stage had about 1.2 GiB free. thcleaner (2026-09-23) pruned the routed experts from 384 to 320 per layer with REAP to make room. He chose 320 because the fork's fused-router kernel already has a 320 case, so the pruned model serves without a C++ rebuild (272 and 288 are not in the kernel table). He said he was unsure of the quality impact, and that 4 cards are very good for GLM-5.3-Flash.

Reported 8 GB
Source: repo github.com/dkpoulsen/dsv41-320e-dspark by dkpoulsen on GitHub, posted by @dkpoulsen, 2026-09-23

Building and serving the 320E model

The pruned weights are not published, so you build them yourself with the repo's tools and those from Libertai/deepseek-v41-flash-reap:

Patch DeepSeek's reference inference code with the repo's patches/patch_dequant_fallback.py. sm_80 has no FP8 or FP4 MMA, so the patch dequantises weights to bf16 on the fly when DSV41_DEQUANT_FALLBACK=1 is set. A fully dequantised checkpoint would not fit on disk (268.9 GiB of MXFP4 experts becomes about 1,012 GiB).
Build a calibration corpus from local text (128 sequences of 2,048 tokens, 262,144 tokens in all).
Run REAP calibration on the five cards with DSV41_DEQUANT_FALLBACK=1: about 100 minutes.
Prune to 320 experts per layer.
Serve with scripts/serve-dsv41-320e.sh.

Serving runs the Docker image dsv41-pp5-sm80, built from 344303947's fork (vLLM 0.13.1), on 251 GB of host RAM: PP5 with VLLM_PP_LAYER_PARTITION=8,8,8,8,8, --kv-cache-dtype fp8_ds_mla --max-model-len 1048576 --max-num-seqs 8 --max-num-batched-tokens 2048 --gpu-memory-utilization 0.97, --speculative-config '{"method":"dspark","num_speculative_tokens":5,"use_local_argmax_reduction":true}', --engram-config '{"cpu_offload":true}', prefix caching and cudagraph FULL_AND_PIECEWISE with capture sizes [1,2,4,8]. Environment: NCCL_P2P_DISABLE=1 NCCL_ALGO=Ring NCCL_PROTO=Simple VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=128 VLLM_ENGRAM_EXACT_PIN=1. The repo's cards link at Gen2 x4 with no P2P.

Reported 8 GB
Source: repo github.com/dkpoulsen/dsv41-320e-dspark by dkpoulsen on GitHub, posted by @dkpoulsen, 2026-09-23

57.8 tok/s median with DSpark, against 42.4 tok/s for the full model without it

Single-stream decode: the 384-expert baseline without DSpark ran at 42.4 tok/s, and the 320-expert model with DSpark (5 draft tokens) at 57.8 tok/s median, 1.36x. By prompt it ranged from 55.5 to 69.4 tok/s, with acceptance 29.4 to 30.9% (tau 2.42 to 3.05). An earlier claim of 72.7 tok/s was withdrawn: it was the maximum of a noisy sample, not the typical rate.

Reported 8 GB
Source: repo github.com/dkpoulsen/dsv41-320e-dspark by dkpoulsen on GitHub, posted by @dkpoulsen, 2026-09-23

Quality cost of pruning: +1.93% held-out perplexity

Held-out perplexity went from 4.3329 to 4.4164 (+1.93%, 95% CI +0.66% to +3.60%, paired over 49,104 tokens), measured through the live serving stack. Both models scored 12/12 on a small task suite, which is too small to resolve a 2% difference. Nobody has measured the cost on agentic tasks yet; run a task eval on your own workload before relying on the pruned model.

Reported 8 GB
Source: repo github.com/dkpoulsen/dsv41-320e-dspark by dkpoulsen on GitHub, posted by @dkpoulsen, 2026-09-23

Partition 8,9,9,8,6: a 4.94M-token KV pool at the same speed

vLLM gives every rank the same number of KV blocks, so the rank with the least free memory sets the pool. With the default split that is rank 4, which carries the model head and the draft. Moving layers off it with VLLM_PP_LAYER_PARTITION=8,9,9,8,6 at utilisation 0.97 grows the pool from 1,729,736 to 4,941,504 tokens at the same 57.5 tok/s. Needle tests passed at 383K, 599K and 911K tokens (prefill 3,519, 2,852 and 2,157 tok/s). Prefix caching cut a 40,042-token prompt from 20.48 s to 0.50 to 0.67 s time to first token; it only pays off if the start of the prompt stays byte-identical between turns, so keep timestamps and reshuffled tool lists out of it.

Reported 8 GB
Source: repo github.com/dkpoulsen/dsv41-320e-dspark by dkpoulsen on GitHub, posted by @dkpoulsen, 2026-09-23

Known issues

Utilisation 0.98 loads, then OOMs on a 383K-token prompt

On the 320E build, --gpu-memory-utilization 0.98 loads and passes every short test, then the engine runs out of memory on a long prompt (383K tokens), on the 9-layer rank. Free memory at load time understates steady-state use, so it is not a safe guide. Stay at 0.97.

Reported 8 GB
Source: repo github.com/dkpoulsen/dsv41-320e-dspark by dkpoulsen on GitHub, posted by @dkpoulsen, 2026-09-23

--max-num-batched-tokens 8192 wedges the pipeline

With --max-num-batched-tokens 8192 the engine loads, then wedges for good on the first real request: the 9-layer rank fails a CUDA allocation and the pipeline falls out of sync. /v1/models still answers, so a health check on that endpoint will not show the engine is dead. Keep 2048, which both launch scripts use, unless you also free memory on the tightest rank.

Reported 8 GB
Source: repo github.com/dkpoulsen/dsv41-320e-dspark by dkpoulsen on GitHub, posted by @dkpoulsen, 2026-09-23

DSpark: num_speculative_tokens must be 5 (or a multiple of 5)

On wtdcode's backport, the base of both builds here, values below 5 are rejected, 7 needs about 200 KB of shared memory against the 163 KB Ampere limit (Triton OutOfResources), and 6 starts but draft positions past the native block of 5 are accepted only 3 to 13% of the time. Going up to 10 does not help: thcleaner measured 24.7 tok/s at 10 against 57.8 at 5, and his repo calls the draft budget quantised to multiples of 5. Upstream adaptive verification is not supported with pipeline parallelism.

Reported 8 GB
Source: repo github.com/dkpoulsen/dsv41-320e-dspark by dkpoulsen on GitHub, posted by @dkpoulsen, 2026-09-23
Source: @darkmaniac7, 2026-10-27 (lab notes)
Source: repo github.com/wtdcode/vllm-backport by wtdcode on GitHub

Expandable segments: set on purpose here, a hard failure on other stacks

Contested 8 GB

allover326's DeepSeek V4 Flash recipe and Schaka's fork of it report that PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True hard-fails at model load on these cards ("cannot map 20 MB with 28.6 GiB free"); Schaka suspects CUDA VMM is broken on the GA100 CMP. Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub
Source: repo github.com/Schaka/deepseek-v4-cmp170hx by Schaka on GitHub
344303947's PP5 launch script sets it on purpose: under full residency a 20 MB allocation at load once failed for lack of 1 MB of contiguous memory, and expandable segments removed that fragmentation OOM. dkpoulsen's 320E recipe sets it too. Source: repo github.com/344303947/dsv41-flash-pp5-170hx by 344303947 on GitHub, posted by @dkpoulsen, 2026-09-22

Treat it as engine and driver specific: keep what the launch script for your build sets, and test before changing it. More on the Instability under load page.

DeepSeek V4.1 Flash on 8x 8 GB (Zanooda PP8) · DeepSeek V4.1 Flash on 6x 8 GB (TP6) · DeepSeek V4.1 Flash on 4x 8 GB (GNmac) · DeepSeek V4.1 Flash on 4x 8 GB (Morrowmake) · Speculative decoding on the 170HX · Parallelism over the PCIe link · vLLM on sm_80: known bugs and patches · DeepSeek V4 and V4.1: tool-call leaks, loops and crashes · Instability under load · Quantization on the 170HX · Driver unlock · 8 GB card · Engines · Benchmarks

Sources

Recipe: 344303947/dsv41-flash-pp5-170hx (Apache-2.0, last commit 2026-09-30; docs in Chinese under dsv41-pp5/), a fork of wtdcode/vllm-backport, posted on Discord by thcleaner (dkpoulsen).
320-expert variant: dkpoulsen/dsv41-320e-dspark (MIT, last commit 2026-09-24), posted by its author. Pruning tools from Libertai/deepseek-v41-flash-reap.

Steps

What you need

Launch with start_vllm_ds4.1.sh

How it fits on five cards

Shadow source layer: an even 8,8,8,8,8 split instead of 2,6,6,6,20

Staged Marlin repack: zero expert offload without a load-time OOM

Exact-size Engram pinning: 264.3 GiB down to 189.1 GiB of pinned host memory

Measured numbers

Full residency: 1.5 s short answer, about 4.6k tok/s prefill at 60K, 1M context

Decode: about 30 to 40 tok/s in the repo, 70+ tok/s in the Discord post

The 320-expert DSpark variant

Why prune: the DSpark head does not fit the last stage

Building and serving the 320E model

57.8 tok/s median with DSpark, against 42.4 tok/s for the full model without it

Quality cost of pruning: +1.93% held-out perplexity

Partition 8,9,9,8,6: a 4.94M-token KV pool at the same speed

Known issues

Utilisation 0.98 loads, then OOMs on a 383K-token prompt

--max-num-batched-tokens 8192 wedges the pipeline

DSpark: num_speculative_tokens must be 5 (or a multiple of 5)

Expandable segments: set on purpose here, a hard failure on other stacks

Related

Sources