DeepSeek V4 Flash 0731 on 4x 8 GB (allover326 PP4 DSpark)

DeepSeek-V4-Flash-0731
Proven 4 x 8 GB
Quant	Native MXFP4 experts + FP8 attention (official deepseek-ai checkpoint, about 155.4 GiB); KV cache fp8
Engine	vLLM: haosdent/vllm branch dsv4-flash-a100 plus allover326's 7 patches (DSpark under PP, sm_80 prefill top-k fallback, logits row chunking) base c3046d1 (recommended since 2026-08-13; first released on f8ea5bb); repo commit 3dd2d88
Topology	PP4 (VLLM_PP_LAYER_PARTITION=12,12,12,7) with DSpark, 5 draft tokens; PCIe Gen2 x4, no P2P
Context	1,047,736 verified one-shot; 1,002,852 over a 405-turn chat with DSV4_LOGITS_ROW_CHUNK=64
Decode	98.1 tok/s
Prefill	5300 tok/s
VRAM per card	64 GB, --gpu-memory-utilization 0.85
Power cap	180 W
Unlock build	cmpunlocker VRAM unlock (65,536 MiB per card)
Flags	`--pipeline-parallel-size 4 --kv-cache-dtype fp8 --block-size 256 --max-num-batched-tokens 2048 --max-num-seqs 8 --trust-remote-code --no-enable-flashinfer-autotune --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4, DSpark speculative config with num_speculative_tokens 5; env VLLM_WORKER_MULTIPROC_METHOD=spawn, DSV4_LOGITS_ROW_CHUNK=64, VLLM_PP_LAYER_PARTITION=12,12,12,7; container --shm-size=16g; never --enforce-eager`
Tested	@deejay9983 on 2026-08-05
Source	link
Files	launch/run-pp-dspark.sh, patches 0001 to 0006 plus 0005a, docker/Dockerfile.devel and docker/Dockerfile.fullbuild (Apache-2.0)
Confirmations	Partly reproduced by @cmdrwaka324, 2026-09-06 · details Reproduced by @danske1, 2026-08-08 · details Partly reproduced by @impactic, 2026-08-08 · details Reproduced by @programmerboi, 2026-08-07 · details Reproduced by @cute_sc2, 2026-08-07 · details

The reference recipe for the first DeepSeek V4 Flash checkpoint, DeepSeek-V4-Flash-0731 (about 284B total, 13B active), not the later V4.1 (see Related). DeeJay (GitHub allover326) serves the official MXFP4 + FP8 weights on four 64 GB cards at PCIe Gen2 x4 with no P2P, using vLLM pipeline parallel 4 and DSpark speculative decoding, a combination vLLM refuses until the repo's patches enable it. The repo reports 98.1 tok/s single-stream decode, about 5,300 tok/s prefill and a verified 1,047,736-token one-shot context at 180 W per card. Several owners reproduced it on four cards (Other people's numbers, and the Confirmation pages). Infobox numbers are the repo's own readings.

Reported 8 GB
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub, posted by @deejay9983, 2026-08-05

Prerequisites

Four 64 GB cards; Gen2 x4 without P2P is enough

Four CMP 170HX with the VRAM unlock (Driver unlock, 8 GB card): nvidia-smi must report 65,536 MiB per card, not 8,192 MiB. The repo's numbers come from four cards at PCIe Gen2 x4, no P2P and no capacitor mod, each capped at 180 W, on an ASUS WRX80 workstation board with 3D-printed shrouds and 120 mm fans. Three cards also work with VLLM_PP_LAYER_PARTITION=15,15,13 (see DeepSeek V4 Flash 0731 on 3x 8 GB (PP3 DSpark)); two cannot hold the about 155.4 GiB (166.9 GB) checkpoint. The repo put the card price at about $1,100-1,200 in August 2026. If throughput comes out near a quarter of these numbers, check the board's power brake first (Known issues, HW Power Brake (PWRBRK)).

Reported 8 GB
Source: @deejay9983, 2026-08-02
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub, posted by @deejay9983, 2026-08-05

Engine base: haosdent/vllm branch dsv4-flash-a100

The SM8x DeepSeek V4 backend is haosdent/vllm branch dsv4-flash-a100 (many draft performance and bug fixes), which grew out of vLLM issue #50576 (posted by giaki3003 on 2026-08-02). allover326's repo is a patch set on top of that branch, and others build on it in turn: OB88594 (2026-08-25) asked for a DSV4-Flash environment for 4x 64 GB cards, and clee pointed him to allover326/deepseek-v4-cmp170hx, which he already used as his base with upstream changes merged in. matstatman noted the repo's benchmarks run at temperature=0 (bench/bench_decode3.py line 42), which flatters speculative decoding. Others in the thread called three cards the entry point for DS4 Flash.

Reported 8 GB
Source: @ob88594, 2026-08-25
Source: @c133, 2026-08-25
Source: @matstatman, 2026-08-25
Source: @jecquas, 2026-08-25
Source: @cressida1881, 2026-08-26

What the seven patches do

Patches 0002 to 0005 enable DSpark under pipeline parallel (0003 is vLLM PR #46994, the draft-token relay across PP ranks, which upstream does not have). 0005a is a required sm_80 prefill top-k torch fallback: without it, prefills above about 128k raise Xid 31. 0006 row-chunks the sparse indexer's logits and lifts the context ceiling, but it is env-gated: without DSV4_LOGITS_ROW_CHUNK set it stays inert and the old wall at about 134k returns. 0001 (a capability gate) is upstream at c3046d1 and is dropped there. DSpark works under PP at all because the drafter's hidden-state taps (layers 40 to 42 of 43) and lm_head both sit on the last PP rank.

Reported 8 GB
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub, posted by @deejay9983, 2026-08-05

Software and checkpoint

Docker with the NVIDIA runtime.
The official deepseek-ai/DeepSeek-V4-Flash-0731 checkpoint, about 155.4 GiB (166.9 GB) on disk; du under-reports it as about 140 GB. The INT4 repack is not needed: the branch reads the native MXFP4 + FP8 weights.
The image needs a real CUDA toolkit base. python:3.12-slim plus pip CUDA wheels pairs nvcc 13.3 with FlashInfer headers for 13.0, and FlashInfer's JIT is required at engine start; Dockerfile.devel uses nvidia/cuda:13.0.2-cudnn-devel with a venv.
Build from inside the vLLM checkout: a build rooted in a directory that also holds the weights sends hundreds of GB to the Docker daemon as build context.

Reported 8 GB
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub

Steps

Unlock all four cards to 64 GB and cap them at 180 W (nvidia-smi -pl 180). Run the power-brake check under Known issues before benchmarking anything.
Download deepseek-ai/DeepSeek-V4-Flash-0731 to a local disk.
Clone allover326/deepseek-v4-cmp170hx and haosdent/vllm branch dsv4-flash-a100 side by side, check out the base commit and apply the patches in glob order (0005a must precede 0006).
Build the image, then start the server with launch/run-pp-dspark.sh, adding the partition and reasoning parser below.

The repo's quick start, on the original base f8ea5bb:

git clone https://github.com/allover326/deepseek-v4-cmp170hx
git clone --branch dsv4-flash-a100 --single-branch https://github.com/haosdent/vllm.git
cd vllm
git fetch origin '+refs/*:refs/remotes/all/*'   # f8ea5bb is only reachable after fetching all refs
git checkout f8ea5bb
for p in ../deepseek-v4-cmp170hx/patches/*.patch; do patch -p1 < "$p"; done
cp ../deepseek-v4-cmp170hx/docker/Dockerfile.devel .
cp ../deepseek-v4-cmp170hx/docker/dockerignore.txt .dockerignore
docker build -f Dockerfile.devel -t dsv4-a100:devel .   # about 10 min, mostly download
../deepseek-v4-cmp170hx/launch/run-pp-dspark.sh

On f8ea5bb the SM8x commit touches only Python and Triton, so VLLM_USE_PRECOMPILED=1 turns the build into a download.

Reported 8 GB
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub

Recommended base since 2026-08-13: c3046d1 (full source build)

On 2026-08-13 DeeJay rebased to haosdent commit c3046d1 with patch 0001 upstreamed. On Discord he gave +8 to 10% decode (89 to 98 tok/s); the repo measures +6.8% (86.1 to 92.0 tok/s on 3 cards, p<0.001), and a circulated +30% claim did not reproduce. c3046d1 changes csrc/, so precompiled wheels and bind mounts cannot carry the kernel changes: apply 0002 to 0006 and 0005a to the tree, then build with docker/Dockerfile.fullbuild (nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04, torch 2.13.0, TORCH_CUDA_ARCH_LIST=8.0, about 115 min on 32 cores). The haosdent branch was force-pushed twice, so c3046d1 survives only as a GitHub codeload tarball (tree d13ae12b9a6621ef8d218f53741e59c6db2f68d2); the repo's patches/README.md shows how to rebuild the commit from it.

Reported 8 GB
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub, posted by @deejay9983, 2026-08-05
Source: @scallion_9883, 2026-08-13

Launch: run-pp-dspark.sh

The launcher starts a container named dsv4-a100 running vllm serve --pipeline-parallel-size 4 --kv-cache-dtype fp8 --block-size 256 --max-num-batched-tokens 2048 --gpu-memory-utilization 0.85 --max-num-seqs 8 --trust-remote-code --no-enable-flashinfer-autotune --tokenizer-mode deepseek_v4 --speculative-config {"method":"dspark","num_speculative_tokens":5}, with VLLM_WORKER_MULTIPROC_METHOD=spawn (fork deadlocks once CUDA is initialised), --shm-size=16g (the 64 MB default is too small for the workers), port 8098 and served model name dsv4s.

Script settings: DSV4_IMAGE, DSV4_VLLM_SRC (the vllm/ directory of the patched checkout; five patched files are bind-mounted into the image, or set DSV4_NO_MOUNT=1 if the patches are built in), DSV4_MODEL, DSV4_MAXLEN (default 32768) and DSV4_ROW_CHUNK (default 64, passed in as DSV4_LOGITS_ROW_CHUNK). --plain runs PP4 without DSpark (50.8 tok/s); --maxlen N sets the context.
Add VLLM_PP_LAYER_PARTITION=12,12,12,7 on four cards. The last rank also carries lm_head and the DSpark drafter; rebalancing removes an 8.7 GiB imbalance and grows the KV pool about 85% (798,660 to 1,476,563 tokens at --max-model-len 163840). One entry per rank, summing to 43 layers.
Add --reasoning-parser deepseek_v4 even with thinking off. Thinking is off by default on 0731; once it is on, the reasoning otherwise lands unmarked in content. On this build the field is reasoning, not reasoning_content. Enable thinking per request with chat_template_kwargs thinking true and reasoning_effort high; max thinks 2.7x harder for the same recall.
The published launcher sets neither the partition nor the reasoning parser; SETTINGS.md in the repo lists both.
Test: curl -s http://127.0.0.1:8098/v1/models should list dsv4s.

Reported 8 GB
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub, posted by @deejay9983, 2026-08-05

DSV4_LOGITS_ROW_CHUNK=64 for conversations

Patch 0006 is off at 0. DeeJay's fix for the old 128k cap (above 128k the CUDA path had been dropped for sm_80) tiles the indexer-logit calculation into query-row pieces, removing a memory wall at about 123k to 150k at some cost in throughput at extreme lengths. Measured by row chunk: 0 dies at about 134k with Xid 31; 256 reaches about 957,600 tokens one-shot; 128 reaches 1,047,736 one-shot, but an accumulating chat dies at about 718k to 733k with a CUDA illegal memory access (always on the same PP rank, reproduced twice); 64 reached 1,002,852 tokens over a 405-turn chat. Use 64 for chats and agents; the launcher defaults to it and it costs about 5% TTFT at 750k. The setting changes only whether long contexts crash, not retrieval accuracy. A one-shot needle test never exercises the prefix-cached path that crashes at 128.

Reported 8 GB
Source: @deejay9983, 2026-08-03
Source: @giaki3003, 2026-08-03
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub, posted by @deejay9983, 2026-08-05

DSpark needs exactly 5 draft tokens

num_speculative_tokens must be 5: 7 gave 60.3 against 98.1 tok/s, and method mtp fails with KeyError mtp_block.main_norm. The repo measures a mean acceptance of 3.03 of 6; DeeJay believes DSpark acceptance is always 75%+. DSpark output is not reproducible at temperature 0, even run to run on the same server (the stock tensor-parallel path behaves the same); run without --speculative-config when you need determinism. --enforce-eager drops decode to 8 to 10 tok/s, worse than no speculation. On TP, DSpark goes negative above about 8 concurrent requests; on PP it keeps winning up to 64 (1.93x).

Reported 8 GB
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub, posted by @deejay9983, 2026-08-05

Use pipeline parallel, not tensor parallel

PP4 beat TP4 6.6x on prefill (5,321 against 801 tok/s at 77k): TP measures flat at about 800 tok/s from 1.5k to 77k tokens because it runs 86 all-reduces per forward pass over a Gen2 x4 link with no P2P, while PP moves the same data 3 times. On 2026-08-04 DeeJay called TP "a bust" and said the main roadblock had been making PP work with MTP or DSpark.

Reported 8 GB
Source: @deejay9983, 2026-08-04
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub, posted by @deejay9983, 2026-08-05

Measured numbers

Repo: 98.1 tok/s decode, 712.8 aggregate at 64 concurrent, about 5,300 prefill

The repo reports, on four cards at 180 W, PP4 (tok/s):

	plain PP4	PP4 + DSpark
decode, single stream	50.8	98.1
decode, 64 concurrent (aggregate)	472.0	712.8
decode at 100k context	38.8	90.0
prefill, 25k to 77k context	about 5,300 (5,321 at 50k)	about 5,200
time to first token at 100k	14.6 s	14.6 s

A one-shot 1,047,736-token prompt took 550 s (about 9.2 minutes) to first token, with decode at 35.6 tok/s at that depth. giaki3003 read the 100k TTFT as about 6.8k tok/s wall-clock prefill. The repo's numbers were taken with prefix caching off and ignore_eos on, so repeated prompts could not hit the cache.

On Discord DeeJay put llama.cpp at about 400 to 500 tok/s prefill and 30 tok/s decode, and vLLM at about 50 tok/s before DSpark worked under PP; after fixing the bug that capped context at 128k he reported 5,000 tok/s prefill, 98 tok/s and the full 1M, calling vLLM about 6x (later 16x) faster than llama.cpp. gpulover and giaki3003 relayed it on 2026-08-03 as about 100 tok/s decode and "5k prefill" on four 64 GB cards; DeeJay added nearly 100 tok/s single stream, over 50 tok/s per request at 8 concurrent, scaling to 64 concurrent. Impactic's first PP4 run, before the fixed patch 0006, gave about 60.34 tok/s decode, 1,916 tok/s medium and 4,490 tok/s long prefill.

Reported 8 GB
Source: @deejay9983, 2026-08-02
Source: @winter_is_amateur, 2026-08-02
Source: @artistosu, 2026-08-03
Source: @giaki3003, 2026-08-03
Source: @deejay9983, 2026-08-03
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub, posted by @deejay9983, 2026-08-05
Source: @justbendev, 2026-08-05
Source: @impactic, 2026-08-06
Source: @giaki3003, 2026-08-15

Second-person runs on four cards

Others ran this recipe on four cards with numbers: about 90 tok/s single stream and 2.5k to 3.5k tok/s prefill at 100k+ (programmerboi), about 100 tok/s (CUTE), 70+ tok/s with 512k context (Impactic) and 699.2 tok/s aggregate at 64 concurrent requests (Danske), plus 70 to 80 tok/s on stock, unmodded cards (Waka324). Details under Other people's numbers; the Confirmation pages record each run.

Reported 8 GB
Source: @programmerboi, 2026-08-07
Source: @cute_sc2, 2026-08-07
Source: @danske1, 2026-08-08
Source: @impactic, 2026-08-08
Source: @cmdrwaka324, 2026-09-06

machinezero's CUDA 13.3 build: 1M context, very stable

machinezero rebuilt DeeJay's container on CUDA 13.3 (image dsv4-a100:cuda133) and ran four cards with --pipeline-parallel-size 4 --kv-cache-dtype fp8 --block-size 256 --max-model-len 1048576 --max-num-batched-tokens 4096 --gpu-memory-utilization 0.92 --max-num-seqs 8 --attention-backend TRITON_MLA_SPARSE_DSV4 --no-enable-flashinfer-autotune, the deepseek_v4 tokenizer and tool parser, and DSpark with 5 tokens. Environment: VLLM_PP_LAYER_PARTITION=11,11,11,10 DSV4_LOGITS_ROW_CHUNK=64 NCCL_SHM_DISABLE=1 NCCL_CUMEM_HOST_ENABLE=0 NCCL_IB_DISABLE=1 VLLM_MARLIN_USE_ATOMIC_ADD=1 VLLM_WORKER_MULTIPROC_METHOD=spawn, the patched vLLM files bind-mounted (speculative.py, pp_utils.py, model_runner.py, the dspark utils.py, sparse_attn_indexer.py) and --shm-size=32g --ipc=host. He found it very stable. Tool calling in Pi Code broke after long sessions and a new session fixed it; DeeJay advised enabling thinking, which is off by default. Earlier, four cards at --max-model-len 393216 left VRAM to spare; DeepSeek recommends at least 384K context for Think Max, with temperature 1.0 and top_p 1.0. No tok/s were posted for this build.

Reported 8 GB
Source: @machinezer0, 2026-08-09
Source: @deejay9983, 2026-08-09

Other people's numbers

About 90 to 100 tok/s single stream on four cards (August 2026)

programmerboi (2026-08-07): about 90 tok/s single-stream decode on 4 cards ("basically A100 speeds at decode") and about 2.5k to 3.5k tok/s prefill at 100k+ context. replacedriver47 noted the write-up's 5K/s prefill is at lower context (about 32K) and limited by PCIe x4 because PP is used rather than TP. CUTE: about 100 tok/s with more possible. machinezero runs it at full quant on quad 8 GB cards (3 to 4 cards hold the full weights where a DGX Spark needs a quant). Impactic reports 70+ tok/s on 4 to 5 cards; on 2026-08-08 he had 512k context working at the same speed on 4 of his 7 cards and was working on 1M. Cooling was his limit: cards reach 80 to 85 C with 120 to 140 mm fans.

Reported
Source: @replacedriver47, 2026-08-07
Source: @programmerboi, 2026-08-07
Source: @cute_sc2, 2026-08-07
Source: @impactic, 2026-08-08
Source: @machinezer0, 2026-08-09

699.2 tok/s aggregate at 64 concurrent requests, and fine at Gen2 x4

Danske (2026-08-08) tested DeeJay's repo at 699.2 tok/s aggregate with 64 concurrent requests, then started a real-workload run with 11k to 18k-token prompts at up to 48 concurrent. Humvee55 (2026-08-10) runs it across 4 cards and says it works great even at Gen2 x4; ueon quoted the repo's 712 tok/s at 64 concurrent.

Reported
Source: @danske1, 2026-08-08
Source: @artistosu, 2026-08-08
Source: @scallion_9883, 2026-08-10
Source: @_arcanum_, 2026-08-10
Source: @ueon, 2026-08-10

Over 5,000 tok/s prefill with every card at 100 W

fallengod (2026-08-20) saw over 5,000 tok/s prefill on 4 cards power-limited to 100 W each; Studebaker said prefill numbers cannot be compared across models. Mountains, replacedriver47 (who found the cards fast compared with his V100s) and Schaka shared or planned the recipe on 2026-08-13.

Reported
Source: @mountains1605, 2026-08-13
Source: @schakaqt, 2026-08-14
Source: @fallengod9962, 2026-08-20
Source: @_studebaker_, 2026-08-20

Daily use, and a call for output validation

1a2b3c0967 used it for a week on an 8-card rig and Zanooda runs it daily; Mountains finds DS 073126 quicker than Qwen Flash Next. Ryko criticised the repo as LLM-generated and asked for output validation (2026-08-26 to 29).

Reported
Source: @deejay9983, 2026-08-26
Source: @vektor_prime, 2026-08-26
Source: @1a2b3c0967, 2026-08-27
Source: @zanooda, 2026-08-27

70 to 80 tok/s on four stock, unmodded cards

Waka324 (2026-09-06): 70 to 80 tok/s with PP on 4 stock, unmodded cards using allover326/deepseek-v4-cmp170hx, which regaly had linked. matstatman asked whether the repo's benchmarks are hardcoded to temp=0, which would flatter them. Ikyo was planning 4 cards on a WRX80E board (see the power-brake warning under Known issues).

Reported
Source: @ikyo, 2026-09-06
Source: @cmdrwaka324, 2026-09-06
Source: @reggie7756, 2026-09-06
Source: @matstatman, 2026-09-06

machinezero: about 10 billion tokens in 3 weeks on four cards

machinezero (2026-08-29) said his cards ran more tokens in 3 weeks than in 3 years of local LLM use, about 10 billion counting prefix cache, for work, alongside about 5 billion tokens of Opus/Fable. He rated DSV4F 95% as good as Opus 4.8 for his tasks and 3 to 4 times faster, but 3 times more verbose with default thinking. On 2026-09-02 he said he runs only DSV4F on his 4 cards and cannot justify a 5th card for GLM 5.3 Flash. CUTE burns many tokens every weekend on an 8-card setup. After a Reddit thread on vLLM prefill speeds machinezero considered selling the cards for an RTX 4090 48GB if the spread were 750 to 1,000 USD; Rickcal said the 4090 48GB still costs over 4.5k.

Reported 8 GB
Source: @machinezer0, 2026-08-29
Source: @cute_sc2, 2026-08-29
Source: @machinezer0, 2026-09-02
Source: @rrickk420, 2026-09-02

Takka: a month on four cards at 600 W, about 4x two DGX Sparks

Takka (2026-09-17) has run DeepSeek Flash on 4 cards for about a month and calls them reliable. The supporting hardware (motherboard, RAM, PSU) cost about GBP 700 plus two extra fans (about GBP 10), and the system draws 600 W at full load; he reports about 4x the tok/s of two DGX Sparks at about 2x the wattage. Asked about 3 cards, Phoenix17 said 3 should be enough for DeepSeek V4 Flash but had no recipe, and Takka (using about 3k tok/s prefill as his reference) thinks it runs on 3, slightly faster because pipeline parallel has one less PCIe hop. Engine, quant and exact tok/s were not stated.

Reported
Source: @spectakkular, 2026-09-17
Source: @spectakkular, 2026-09-17
Source: @phoenixone7, 2026-09-17
Source: @chonsworth, 2026-09-17
Source: @c133, 2026-09-17

Three cards: 350 tok/s at 8 concurrent on PCIe 1.1 x4 (nightdays), 120+ tok/s water-cooled (skurtdiggler)

nightdays reported DeepSeek V4 Flash 0731 on 3 cards in vLLM at 8 concurrent requests, 350 tok/s aggregate, with the links at PCIe 1.1 x4. arabus ran DeepSeek V4 Flash on two cards in LM Studio and got slow results; Ezy said not to use LM Studio, and ST suggested Unsloth Studio or vLLM.

Reported
Source: @.nightdays, 2026-08-30
Source: @arabus_74064, 2026-08-30
Source: @ezybr0, 2026-08-30
Source: @stcord, 2026-08-30

skurtdiggler's three water-cooled cards (bought on eBay with Bykski blocks fitted) ran DeepSeek V4 across the 3 cards at 120+ tok/s, against Qwen3.8 27B at about 33 tok/s on one card, benchmarked with HumanEval, BigCodeBench-Hard and LoveCodeBench v6. DeeJay said Qwen on one card should be closer to 75 tok/s; Studebaker asked whether 3 GPUs for DeepSeek beat 3 instances of Qwen3.8 27B FP8. Three-card setups have their own page: DeepSeek V4 Flash 0731 on 3x 8 GB (PP3 DSpark).

Reported
Source: @skurtdiggler, 2026-08-18
Source: @skurtdiggler, 2026-08-18
Source: @_studebaker_, 2026-08-18
Source: @deejay9983, 2026-08-18

Context and recall

Is the 1M window usable? Prefill and TTFT near 1M

Contested 8 GB

formatme (2026-08-04) argued nobody had shown decent CMP inference speeds, that about 100 tok/s was single-stream only, and that RESULTS.md shows a TTFT of 9.2 minutes at 1M context with decode falling to 35.6 tok/s; his group was writing its own sm_80 kernel support. Source: @counterstrikeaimgod, 2026-08-04
DeeJay replied that throughput scales with concurrency and that the 1M row means one prompt filling the whole 1M window (the benchmark procedure was run by Claude), not a normal prompt with 1M context configured. Source: @deejay9983, 2026-08-04
robbbbbb7973 added that prefix caching was off for those benchmarks and H200-class prefill should not be expected. Source: @robbbbbb7973, 2026-08-04
DaLZy (2026-08-09) said DeepSeek V4 Flash 0731, roughly Opus 4.6 level in his view, runs on three cards with 1M context. Source: @dalzyyy, 2026-08-09
quark2world doubted prefill is fast enough to use more than 250k, and posted an estimate of the time to process a full prompt. Source: @quark2world, 2026-08-09
clee cited the repo's RESULTS.md (4 cards, speed vs context): prefill starts around 5,300 tok/s and is still 1,904 tok/s past 1,000,000 tokens of context. Source: @c133, 2026-08-09
DaLZy answered quark2world that the estimate assumes a 0% cache hit, that at a 90% hit a 1M prompt takes a minute at most, and that three agents at 256k each on 3 cards may be more useful than one 1M context. Source: @dalzyyy, 2026-08-09

Recall falls with depth: 100% at 150k, 30% at 900k

The repo planted facts at known positions and re-queried them: 100% at 150k, 87% at 300k, 60% at 450k, 50% at 750k and 30% at 900k. The row chunk does not change it and no prompt fixes it: index_topk is fixed at 512 while the candidate pool grows with context, so the reachable fraction falls about 6x from 150k to 900k. Thinking on raised recall from 61.4% to 80.7%, and effort high recalls as well as max. Treat 1M as a large working set, not a reliable database: keep retrieval-critical context under about 150k, or re-inject facts that must survive.

Reported 8 GB
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub, posted by @deejay9983, 2026-08-05

DeeJay's sizing advice: 256k or 512k windows with several sessions

DeeJay reached 768k stable at about 2,500 tok/s prefill, then 1M; giaki3003 put the cold prefill for 1M at 550 s. DeeJay later said the slowdown came from the test sending a full 1M-token prompt: a 1M window runs at full speed and slows only as context fills, and he suggests 256k or 512k with 6 to 8 concurrent requests (Studebaker suggested --enable-prefix-caching). With row chunk 64, he says four cards give roughly 5 to 6M tokens of KV, enough for five concurrent 1M sessions (the repo's own measured pool at --max-model-len 163840 with the 12,12,12,7 partition is 1,476,563 tokens); context over 200k is fine but speed drops past 500k, and long context slows a lot at PCIe x4. ju5t5omedude cautioned that A100 forks working on the 170HX still need benchmarks, since compute differs.

Reported 8 GB
Source: @deejay9983, 2026-08-03
Source: @giaki3003, 2026-08-03
Source: @_studebaker_, 2026-08-03
Source: @ju5t5omedude, 2026-08-03
Source: @deejay9983, 2026-08-09
Source: @mountains1605, 2026-08-09
Source: @artistosu, 2026-08-09

Variants

ElRoberto538/deepseek-v4-cmp170hx: the same recipe on an upstream vLLM base

robbbbbb described his repo as "a fork of someone else here's work" (allover326's). Its text recipe is the same PP + DSpark setup for 3 or 4 cards (3 cards need VLLM_PP_LAYER_PARTITION=15,15,13), but as of 2026-09-01 its base moved from haosdent/vllm@dsv4-flash-a100 to upstream vllm-project/vllm commit 2cf0a69: patch 0000 ports the fork's SM8x sparse-MLA commit across about 630 upstream commits, and 0005a must precede 0006. It builds from docker/Dockerfile.devel (nvidia/cuda:13.0.2-cudnn-devel plus a venv) with VLLM_USE_PRECOMPILED=1 and a pinned wheel, since the text patches touch only Python and Triton. It keeps the DSV4_LOGITS_ROW_CHUNK=64 rule for accumulating chats and the retrieval caveat. Its headline numbers are allover326's (98.1 / 50.8 tok/s single stream, 712.8 / 472.0 at 64 concurrent), carried over from the original repo, not re-measured. The vision variant in the same repo (patches 0007 to 0009) loses speculative decoding and is covered on DeepSeek V4 Flash 0731 on 3x 8 GB (PP3 DSpark).

Reported 8 GB
Source: repo github.com/ElRoberto538/deepseek-v4-cmp170hx by ElRoberto538 on GitHub
Source: @robbbbbb7973, 2026-09-03

Schaka/deepseek-v4-cmp170hx: agentic fixes (tool calls, thinking loops, repetition)

Schaka (2026-09-02) "finally got rid of thinking loops in dsv4", which he traced mostly to agents leaking thinking back into context because vLLM did not expose it correctly; on 2026-09-03 he told robbbbbb that the work his fork came from is missing pieces and causes reasoning loops. His repo is a copy of allover326's (patches 0001 to 0006, 0005a and RESULTS.md byte-identical, so its headline numbers are allover326's 4-card measurements) plus 16 patches added 2026-08-20 to 09-01: 0007 a NaN to -inf guard before tl.argmax (vLLM #50183), 0008 0731 thinking defaults and an argmax clamp to vocab size, 0009 DSML orphan-invoke recovery (vLLM PR #49117), 0010 to 0015 from rasatpetabit's gist, 0016 REASONING-state recovery, 0018 to 0020 from ChinaBoy0618's fork, 0021 and 0022 repetition recovery (DSV4_NUDGE_MAX_RETRIES default 2; DSV4_NUDGE=0 disables) and 0023 emitting reasoning_content beside reasoning. The authoritative launcher is launch/run-pp-dspark-podman.sh: --pipeline-parallel-size 4 --gpu-memory-utilization 0.85 --max-model-len 750000 --max-num-batched-tokens 2048 --max-num-seqs 8 --kv-cache-dtype fp8 --block-size 256 --tokenizer-mode deepseek_v4 --enable-auto-tool-choice --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 with top_p 0.95 as the generation override, env DSV4_LOGITS_ROW_CHUNK=64 DSV4_REP_FUZZY_MIN_HITS=15 DSV4_RECOVERY_MODE=nudge. Clients should send chat_template_kwargs reasoning_effort on every request (max for agentic sessions). On 3 cards an INT4 repack ran --max-model-len 524288 with a 2,013,978-token pool at about 86 tok/s. The README is stale (it says 16 patches and checks out f8ea5bb while 0007 onward target c3046d1), and repetition loops past about 100k context are mitigated, not solved. Its launchers reload nvidia_uvm to clear wedges; see the recovery warning under Known issues.

Reported 8 GB
Source: repo github.com/Schaka/deepseek-v4-cmp170hx by Schaka on GitHub, posted by @schakaqt, 2026-09-02

Before the forks: llama.cpp only (early August 2026)

On 2026-08-01 asm64 said there was no official support and no stable vLLM build for DeepSeek-V4-Flash-0731 yet, and that you could not quantize it yourself because of kernel limitations. He called llama.cpp the only way but said he would not run anything through it on an A100 or 170HX because of extremely poor performance, and that the model was built for Blackwell and Chinese GPUs. Others were already getting 20 to 25 tok/s with llama.cpp at the time, so "extremely poor" is relative; the vLLM forks on this page changed the picture within days. Current llama.cpp numbers are on llama.cpp on CMP cards.

Reported
Source: @leesle., 2026-08-01
Source: @chleba7742, 2026-08-01

Known issues

Power brake on PCIe pin B30 holds cards at about a quarter of their speed (ASUS Pro WS WRX80E-SAGE)

Warning 8 GB On the ASUS Pro WS WRX80E-SAGE the repo's numbers came from, the board asserts the optional PWRBRK# sideband on edge pin B30 of its x16 slots and holds the cards in "HW Power Brake": about 88 W, 1140 MHz, fp16 39.3 TFLOPS and about 608 GB/s, against a healthy 105 to 180 W, about 1400 MHz, 155.7 TFLOPS and about 1355 GB/s. Check with nvidia-smi -q | grep -A1 "HW Power Brake Slowdown": "Active" with no thermal or power cause is this. Fix: Kapton tape over pin B30 on the card's edge connector, or a riser that does not route B30. If the board does not assert the signal, do nothing: taping an undriven pin gains nothing and risks the contact. Schaka's 170hx-journey repeats the same check and fix for his PLX switch build. Full page: HW Power Brake (PWRBRK).

Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub
Source: repo github.com/ElRoberto538/deepseek-v4-cmp170hx by ElRoberto538 on GitHub
Source: repo github.com/Schaka/deepseek-v4-cmp170hx by Schaka on GitHub
Source: repo github.com/Schaka/170hx-journey by Schaka on GitHub

"Let me..." loops above 120k to 140k context: cap the agent client at 115k

machinezero had run the recipe on 4 unlocked 8 GB cards for 4 days (2026-08-11) when he saw rambling "Let me...." output in the Pi Code harness after 3 to 5 hours; restarting the session fixed it without touching vLLM, and quark2world agreed it was a harness or software issue rather than the cards. Testing from 128k to 1M context, he found the model falls into these loops once context reaches 120k to 140k; compaction never fixed it, only a new session did. His fix: keep vLLM serving 1M context but set the agent client (Oh my Pi) to 115k with auto-compaction as early as 85%, which makes it usable in an endless session. He runs a fork of allover326/deepseek-v4-cmp170hx built with CUDA 13.3 and gets up to 120 tok/s decode with draft acceptance, sometimes as low as 50; DeeJay notes decode drops as context grows.

Reported 8 GB
Source: @machinezer0, 2026-08-11
Source: @quark2world, 2026-08-11
Source: @machinezer0, 2026-08-18
Source: @deejay9983, 2026-08-18

Loops before tool calls at long context: a bug, a stale session, or a model limit

Contested

diabeetee (2026-08-06) ran the full-precision safetensors on vLLM at 1M context with DeeJay's latest commit and the pi harness. About 100k tokens into a session the model looped repeatedly before making tool calls, while llama.cpp with the official GGUF had not hallucinated this badly. He later blamed a stale session started before his configs were right, since a fresh one ran to 300k with no repeats. Source: @diabeetee, 2026-08-06
giaki3003 suggested checking the vLLM tool-calling parser. Source: @giaki3003, 2026-08-06
DaLZy floated undetected HBM errors (no ECC) as an untested cause. Source: @dalzyyy, 2026-08-06
DeeJay ran a 1M-context stress test with random prompts and tool calls and got 100% clean results. Source: @deejay9983, 2026-08-06
bayley_w sees it as a limitation of small sparse-attention models at long context ("this isn't a bug"; MiMo-2.5 did the same). Source: @bayley_w, 2026-08-06
Impactic suggested the chat templates in the model's encoding/ folder on Hugging Face. Source: @impactic, 2026-08-06

Thinking loops and broken tool calls from about 100k tokens (early September)

Schaka saw broken tool calls and repetition from about 100k tokens on, worst with DSpark, on a stack based on the A100 fork of vLLM, in both OpenCode and DSH, and pointed to many open vLLM issues about DSML leaks for DeepSeek (2026-09-01). The model repeatedly announced a tool call ("Let me check run progress.") without making it, or emitted corrupt tool tags; his first workaround detected the loop, cancelled the stream and spliced in a "continue" token. DSpark doubles decode locally but makes the think-looping worse. giaki3003 recommended provider pinning on OpenRouter, and Ryko said loops in big models usually come from too-low temperature (for example 0), a bad quant or a bad engine, and should be fixed rather than hidden. On 2026-09-02 Schaka got rid of the thinking loops (mostly agents leaking thinking back into the context) and on 2026-09-03 published the fixes as Schaka/deepseek-v4-cmp170hx (Variants). More on DeepSeek V4 and V4.1: tool-call leaks, loops and crashes.

Reported
Source: @schakaqt, 2026-09-01
Source: @schakaqt, 2026-09-01
Source: @giaki3003, 2026-09-01
Source: @vektor_prime, 2026-09-01

DSML tool-call markup leaking as raw text (open question)

Josh reported output where the tool call came back as raw DSML markup (<｜DSML｜tool_calls> / <｜DSML｜invoke name="bash"> ...) instead of being parsed into a tool call, and asked what he needed to update (2026-08-29). Nobody answered in the thread. A tool-call parser or chat-template mismatch in the serving stack is the likely cause but unconfirmed; Schaka's patch 0009 (DSML orphan-invoke recovery) targets this class of leak.

Theory
Source: @_____josh_____, 2026-08-29

Decode fluctuates: up to 175 tok/s, typically 40 to 60

machinezero reported up to 175 tok/s decode on full weights with vLLM, but more typically 40 to 60 tok/s, and was unsure why vLLM performance fluctuates so much. DeeJay said his vLLM repo gives 100+ tok/s.

Reported
Source: @machinezer0, 2026-08-11
Source: @deejay9983, 2026-08-11

Retracted: the 172 tok/s claim was a flaw in an AI agent's test method

On 2026-08-05 DeeJay posted DeepSeek V4 going from 98 to 172 tok/s single-session (with about 10% worse throughput at 16 concurrent sessions than the old template). On 2026-08-06 he retracted it: the result came from a flaw in Claude's testing method, the default brk=1 is actually faster and stays faster with concurrency, and the real figure is back to 98 tok/s. Check an AI agent's benchmark method before posting its numbers.

Reported
Source: @deejay9983, 2026-08-05

Patch 0006 did not apply on newer vLLM (fixed in the repo on 2026-08-06)

aaron found that patch 0006 (the context-ceiling fix in sparse_attn_indexer) no longer applied: vLLM commit fded064 restructured the prefill path with query sharding and removed _prefill_topk_needs_torch_fallback() (2026-08-05). diabeetee hit "2 out of 5 hunks FAILED" on vllm/model_executor/layers/sparse_attn_indexer.py and worked around it by editing that file to skip ops.top_k_per_row_prefill when logits.shape[0] == 0, otherwise passing logits.shape[0] as the row count. Impactic's earlier attempt had also failed on 0006. DeeJay updated the repo's patches (0006 changed) on 2026-08-06 and asked people to open GitHub issues. Apply the patches to the pinned base commit, not to a newer vLLM.

Reported
Source: @aaron0010, 2026-08-05
Source: @diabeetee, 2026-08-05
Source: @deejay9983, 2026-08-06
Source: @impactic, 2026-08-06

Stock vLLM fails with "hyperconnection.hpp:56 Unsupported architecture"

machinezero ran stock vllm serve with deepseek-ai/DeepSeek-V4-Flash (TP4, --enable-expert-parallel, max-model-len 393216, fp8 KV, block size 256, deepseek_v4 tokenizer and parsers, FLASHINFER attention, FULL_AND_PIECEWISE cudagraphs), which failed with "Assertion error (csrc/apis/hyperconnection.hpp:56): Unsupported architecture" (2026-08-09). robbbbbb7973 pointed him to this repo. machinezero then got it running with TP after a lot of back and forth with Grok: spec decode mean acceptance length 3.68, accepted throughput 114.79 tok/s, per-position acceptance 0.809 / 0.667 / 0.527 / 0.396 / 0.277, average 53.5%. DeeJay: the repo works as published, so do not use Grok, and his fork is built for PP + DSpark, not TP.

Reported
Source: @machinezer0, 2026-08-09
Source: @robbbbbb7973, 2026-08-09
Source: @deejay9983, 2026-08-09

Three cards: "RPC call to sample_tokens timed out" on the first token

On a three-card setup aaron hit the EngineCore error "TimeoutError: RPC call to sample_tokens timed out." while generating the first token (2026-08-09). No root cause was posted. He asked flipnburn for his final command to compare. A longer write-up by @flipnburner exists (a README for his working three-card setup); see Discord.

Reported
Source: @aaron0010, 2026-08-09
Source: @flipnburner, 2026-08-09

Do not set expandable_segments on this stack

Contested 8 GB

allover326's repo and Schaka's fork report that PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True hard-fails at model load on these cards ("cannot map 20 MB with 28.6 GiB free"); Schaka suspects CUDA VMM is broken on the GA100 CMP. Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub
Source: repo github.com/Schaka/deepseek-v4-cmp170hx by Schaka on GitHub
Other repos set it on purpose: bayley's GLM serve scripts to avoid rank-7 fragmentation, 344303947's DS V4.1 PP5 launch script, dkpoulsen's 320E DSpark recipe, anoane's EXL3 work, fallengod's Qwen3.8 FP8 compose file and g4bwy's Qwen3.8 Flash Next serve script. roboalchemist needed expandable_segments:False for LMCache, and kaka86mm says not to set it with the mmap CPU KV tier. Source: repo github.com/bayley/vllm-170hx-glm5 by bayley on GitHub
Source: repo github.com/344303947/dsv41-flash-pp5-170hx by 344303947 on GitHub
Source: repo github.com/kaka86mm/glm53-flash-170hx by kaka86mm on GitHub
Source: repo github.com/roboalchemist/vllm-backport by roboalchemist on GitHub

The setting depends on the engine and driver: leave it unset for this recipe, and test before changing it elsewhere.

Recovering a wedged card: module reload or reboot

Contested 8 GB

allover326's launcher recovers wedged cards with nvidia-smi -r, then rmmod and modprobe nvidia_uvm, then re-applies -pl 180; Schaka's launchers also reload nvidia_uvm to clear an Xid 154 wedge. Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub
Source: repo github.com/Schaka/deepseek-v4-cmp170hx by Schaka on GitHub
tim-odonnell's Xid 31 guide says never rmmod nvidia_uvm: on 2026-09-29 that turned one Xid 31 into Xid 154 on all 4 cards and only a reboot recovered. dkpoulsen (GLM-5.3-Flash PP5) found a crash wedges the GPU at RM level (nvidia-smi looks healthy, the next start fails cudaSetDevice) until a host reboot, the tell being a nonzero dmesg | grep -ci xid; kaka86mm needs a cold power cycle after an Xid 31 (about 1 in 10 engine starts) and says nvidia-smi -r is not supported; allover326's dsa-mtp kit warns never to kill -9 a crashed multi-GPU vLLM run, because killed workers strand CUDA contexts that only a reboot clears. Source: repo github.com/tim-odonnell/cmp170hx-deepseek-v4-flash by tim-odonnell on GitHub
Source: repo github.com/dkpoulsen/vllm-170hx-glm5.3-flash by dkpoulsen on GitHub
Source: repo github.com/kaka86mm/glm53-flash-170hx by kaka86mm on GitHub
Source: repo github.com/allover326/vllm-dsa-mtp-sm80 by allover326 on GitHub

After an Xid 31, reboot before relaunching: the launcher reloads nvidia_uvm on its own

Warning 8 GB On a four-card PP job, reloading nvidia_uvm after an Xid 31 turned one card's error into Xid 154 on all four cards, and only a reboot recovered (tim-odonnell, 2026-09-29). launch/run-pp-dspark.sh runs that reload by itself when its start-up probe finds the cards unable to create CUDA contexts, so a relaunch straight after a crash can escalate it. After an Xid 31, stop the container cleanly (never kill -9 the workers), reboot the host, re-apply the 180 W cap, then launch. Xid background: Xid 31 and the reserved top of VRAM.

Source: repo github.com/tim-odonnell/cmp170hx-deepseek-v4-flash by tim-odonnell on GitHub
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub

DeepSeek V4 Flash 0731 on 3x 8 GB (PP3 DSpark) · V4.1 recipes: DeepSeek V4.1 Flash on 4x 8 GB (Morrowmake), DeepSeek V4.1 Flash on 4x 8 GB (GNmac), DeepSeek V4.1 Flash on 5x 8 GB (PP5 full residency), DeepSeek V4.1 Flash on 8x 8 GB (Zanooda PP8) · DeepSeek V4 and V4.1: tool-call leaks, loops and crashes · Speculative decoding on the 170HX · Parallelism over the PCIe link · vLLM on sm_80: known bugs and patches · HW Power Brake (PWRBRK) · ASUS Pro WS WRX80E-SAGE SE WIFI · Engines · Benchmarks

Sources

Repo: allover326/deepseek-v4-cmp170hx (Apache-2.0; last commit 3dd2d88, 2026-08-12), posted on Discord by DeeJay, who confirmed on 2026-08-15 that allover326 is his account.
Forks: ElRoberto538/deepseek-v4-cmp170hx (Apache-2.0, last commit 2026-09-02, posted by robbbbbb7973) and Schaka/deepseek-v4-cmp170hx (Apache-2.0, last commit 2026-09-02, posted by Schaka).
Engine base: haosdent/vllm branch dsv4-flash-a100; vLLM issue #50576.