DeepSeek V4.1 Flash on 8x 8 GB (Deepinfer-SM80 TP4 x PP2)
More actions
| Reported 8 x 8 GB | |
| Quant | Official checkpoint (FP8 dense, FP4 experts); software FP8 on sm_80 |
|---|---|
| Engine | vLLM PR #56201 head (79a7108d) plus the two Deepinfer-SM80 patches and 30 drop-in modules Deepinfer-SM80, vLLM v0.28.1rc0 base |
| Topology | TP4 x PP2, partition 20,20 |
| Context | 1,048,576 |
| Decode | 153.35 tok/s |
| Prefill | 3463 tok/s |
| VRAM per card | 64 GiB, --gpu-memory-utilization 0.95 |
| Flags | VLLM_PP_LAYER_PARTITION=20,20, --gpu-memory-utilization 0.95, max_num_seqs 12, DSpark K=5 |
| Tested | @HalfVulpes on 2026-10-28 |
| Source | link |
| Confirmations | None yet. |
The full DeepSeek-V4.1-Flash-1017 checkpoint on eight 64 GB cards with the Deepinfer-SM80 patches, at TP4 x PP2. It holds 13.3M tokens of KV on the GPUs (12.69 full 1M-token sequences). Build steps are the same as the six-card page: DeepSeek V4.1 Flash on 6x 8 GB (Deepinfer-SM80 TP2 x PP3). Infobox decode is concurrency 1 on code. Source: repo github.com/HalfVulpes/Deepinfer-SM80 by HalfVulpes on GitHub, posted by @reala7096, 2026-10-20
Prerequisites
Eight CMP 170HX at 64 GB and 74 SMs (Driver unlock, 8 GB card), the patched vLLM build from the six-card page, and the official deepseek-ai/DeepSeek-V4.1-Flash-1017 checkpoint (about 475 GiB of weights).
Steps
- Build vLLM with the Deepinfer-SM80 patches as in DeepSeek V4.1 Flash on 6x 8 GB (Deepinfer-SM80 TP2 x PP3)#Steps.
- Serve with
--tensor-parallel-size 4 --pipeline-parallel-size 2,VLLM_PP_LAYER_PARTITION=20,20,--gpu-memory-utilization 0.95,--max-num-seqs 12, DSpark K=5,--max-model-len 1048576.
Source: repo github.com/HalfVulpes/Deepinfer-SM80 by HalfVulpes on GitHub, posted by @reala7096, 2026-10-20
Measured numbers
- 13,306,380 GPU-resident KV tokens (12.69 x 1M).
- Concurrency 1, code: 153.35 tok/s (179.74 on a 1,024-token run). Concurrency 12: 620.22 tok/s aggregate.
- Cold prefill about 3,463 tok/s; a single 649K-token prompt at 2,759.9 tok/s with 63.27 of 64.00 GiB peak on PP rank 1.
Source: repo github.com/HalfVulpes/Deepinfer-SM80 by HalfVulpes on GitHub, posted by @reala7096, 2026-10-20
Other eight-card numbers
- segmond ran the official FP8 checkpoint on 8 cards in PP8 ("2x4") with the Deepinfer-SM80 vLLM and no P2P: 74 tok/s at first, then a steady 80 tok/s decode at 256k context with 6,000 tok/s prefill at zero cache hit. It read a 450k-token prompt coherently and edited a 3,000-line HTML file. This contradicts an AI assistant's claim that the FP8 checkpoint needs more work on sm_80. Source: @segmond, 2026-10-20
Source: @segmond, 2026-10-20
Source: @segmond, 2026-10-20 - reala on PCIe x4 links, 8 GPUs (his "PP4" layout): code decode for one user 135 to 139 tok/s on the original FP4 weights and 117.1 on EXL3 3.5 bpw; cold prefill 2,520 to 2,690 and 2,025 to 2,126 tok/s; prefill at 131K 2,708 to 2,730 and 2,171. Engine not named. Source: @reala7096, 2026-10-22
- For the PP8 recipe with its own 20-patch series, see DeepSeek V4.1 Flash on 8x 8 GB (Zanooda PP8).
Known issues
See DeepSeek V4.1 Flash on 6x 8 GB (Deepinfer-SM80 TP2 x PP3)#Known issues: pinned to an unmerged PR commit, memory utilization reads lower than set under PP, a one-second stall on the first request.
Sources
- Repo: HalfVulpes/Deepinfer-SM80 (Apache-2.0). Posted on Discord by segmond (2026-10-20) and reala.
- Related: Engines, Benchmarks.