DeepSeek V4.1 Flash on 6x 8 GB (Deepinfer-SM80 TP2 x PP3)

DeepSeek-V4.1-Flash-1017
Reported 6 x 8 GB
Quant	Official checkpoint (FP8 dense, FP4 experts); software FP8 on sm_80
Engine	vLLM PR #56201 head (79a7108d) plus the two Deepinfer-SM80 patches and 30 drop-in modules Deepinfer-SM80 2a828fc lineage; vLLM v0.28.1rc0 base
Topology	TP2 x PP3, partition 14,14,12
Context	1,048,576
Decode	160.8 tok/s
Prefill	4037 tok/s
VRAM per card	64 GiB
Flags	`VLLM_PP_LAYER_PARTITION=14,14,12, --speculative-config dspark num_speculative_tokens 5, --max-model-len 1048576`
Tested	@HalfVulpes on 2026-10-28
Source	link
Confirmations	None yet.

Deepinfer-SM80 ports the ideas of DeepSeek's DeepInfer (Blackwell-only kernels) to Ampere as two vLLM patches plus 30 drop-in Python, Triton and TileLang modules, for serving DeepSeek-V4.1-Flash at 1M context with DSpark. Every headline number was measured on "NVIDIA GA100 64GB (SM80, 74 SMs)" nodes, which is the unlocked CMP 170HX (the source comments say so). On six cards the measured shape is TP2 x PP3. Infobox numbers are one stream on code and cold prefill at 16K. Source: repo github.com/HalfVulpes/Deepinfer-SM80 by HalfVulpes on GitHub, posted by @reala7096, 2026-10-20
Source: @segmond, 2026-10-20

Prerequisites

Six CMP 170HX at 64 GB and 74 SMs (Driver unlock, 8 GB card). Six cards allow TP2 x PP3 (TP6 is not a valid vLLM split for this model; see DeepSeek V4.1 Flash on 6x 8 GB (TP6)).
CUDA 13.0 toolkit, Python 3.12, GCC 13 or newer, torch 2.13.0+cu130; driver verified 595.71.05 / CUDA 13.2.
About 475 GiB of weights: the full checkpoint needs a 6 or 8 GPU node.

Source: repo github.com/HalfVulpes/Deepinfer-SM80 by HalfVulpes on GitHub, posted by @reala7096, 2026-10-20

Steps

The repo ships patches, drop-ins and microbenchmarks, not download or serve scripts.

# 1. Check out the exact vLLM base (only reachable through the PR ref)
git init vllm && cd vllm
git remote add origin https://github.com/vllm-project/vllm.git
git fetch --filter=blob:none origin refs/pull/56201/head
git checkout 79a7108d9aea27ddab99ce1779290d300b17fc23
cd ..
# 2. Apply both patches (validates the base commit and the result)
git clone https://github.com/HalfVulpes/Deepinfer-SM80
bash Deepinfer-SM80/scripts/apply_patches.sh --vllm-dir vllm
# 3. Build for Ampere only, then check the environment
cd vllm && TORCH_CUDA_ARCH_LIST="8.0 8.6" pip install -e . --no-build-isolation && cd ..
python Deepinfer-SM80/scripts/verify_env.py
# 4. Kernel microbenchmarks on one GPU
bash Deepinfer-SM80/scripts/run_microbench_single_gpu.sh

Patch order: patches/00-vllm-pr56201-sm80-base.patch (haosdent/vllm dsv4-flash-a100 rebased, plus V4.1 SM80 sparse MLA, software FP8 and DSpark over PP), then patches/vllm-sm80-deepinfer.patch. scripts/check_dropin_sync.py must report "30 same, 0 diff, 0 missing".
docker/Dockerfile.sm80 builds the same in about 35 minutes on 16 cores.
Serve the official deepseek-ai/DeepSeek-V4.1-Flash-1017 with --tensor-parallel-size 2 --pipeline-parallel-size 3, VLLM_PP_LAYER_PARTITION=14,14,12, DSpark K=5 and --max-model-len 1048576.

Source: repo github.com/HalfVulpes/Deepinfer-SM80 by HalfVulpes on GitHub, posted by @reala7096, 2026-10-20

Measured numbers

6x GA100 64 GB, TP2 x PP3, DSpark K=5, 1M max context; stock vLLM against Deepinfer-SM80:

	Baseline vLLM	Deepinfer-SM80
16K cold prefill	3,349 tok/s	4,037 tok/s (+20.5%)
64K prefill	3,627	4,507 (+24.2%)
64K TTFT	13.2 s	10.7 s
1-stream code decode	151.8	160.8 (+5.9%)
KV capacity	4,648,480 tokens	unchanged

Topology study: PP6 (TP1 x PP6) decodes 65.5 to 68.5 tok/s single stream, 2.1x slower than TP2 x PP3; asymmetric TP2 + TP4 about 55.5 tok/s unspeculated. The KV pool clamps to stage 0 in both. Source: repo github.com/HalfVulpes/Deepinfer-SM80 by HalfVulpes on GitHub, posted by @reala7096, 2026-10-20

Other six-card numbers

On 2026-10-04 OB88594 noted that six-card DeepSeek V4.1 Flash still lacked software support; the six-card recipes here came later in October. Source: @ob88594, 2026-10-04
reala, best results on PCIe x4 links with TP2 + PP3 (6 GPUs): code decode for one user 144.4 tok/s on the original FP4 weights and 123.4 on EXL3 3.5 bpw; cold prefill 1,970 and 1,695 tok/s; prefill at 131K 1,987 and 1,647. The engine was not named in the post. Open problem: the drafter accepts about 2.9 tokens per step on reasoning text against about 5.1 on code, leaving reasoning at 55 to 75 tok/s. Source: @reala7096, 2026-10-22
Source: @reala7096, 2026-10-22
reala on six cards with the 3.5 bpw EXL3: it kept making silly tool-call mistakes, which GLM did not; on the original FP4 weights he averages about 80 tok/s, maybe 90, on x4 links. Source: @reala7096, 2026-10-22
Source: @reala7096, 2026-10-22
Theory Miséricorde: six cards may be worse than four for this model, because TP6 is impossible and only TP2 x PP3 works; an EPYC platform with working P2P and x16 links "gives a lot". Source: @wakeupmbrc1, 2026-10-17
Source: @wakeupmbrc1, 2026-10-17

Known issues

Pinned to an unmerged vLLM PR commit, so it will bit-rot as vLLM moves. Upstream PR #56741 renamed vllm/models/deepseek_v4_1 to deepseek_v41, so the patches do not apply to newer main.
--gpu-memory-utilization 0.92 behaves like about 0.89 (PP0) / 0.855 (PP1) because of CUDA-graph memory profiling.
The first request stalls about 1 s while Triton compiles _dequantize_and_gather_k_kernel.
"SymmMemCommunicator: Device capability 8.0 not supported" is expected on Ampere (all-reduce falls back to CUSTOM/PYNCCL).
GitHub issues #1 and #2 were a missing pp_share.py from applying one patch only, or a checkout older than 8895510.
verify_env.py warns on 108-SM parts because the wave-aligned tunings were measured on 74 SMs; A800 and A100 runs show smaller gains.

Source: repo github.com/HalfVulpes/Deepinfer-SM80 by HalfVulpes on GitHub, posted by @reala7096, 2026-10-20

Sources

Repo: HalfVulpes/Deepinfer-SM80 (Apache-2.0), last commit 2026-10-28. Posted on Discord by reala and segmond. Builds on haosdent/vllm dsv4-flash-a100; a component of the GNmac-lab recipe image.

Prerequisites

Steps

Measured numbers

Other six-card numbers

Known issues

Related

Sources