Jump to content
Toggle menu
Toggle preferences menu
Toggle personal menu
Not logged in
Your IP address will be publicly visible if you make any edits.

DeepSeek V4 Flash 0731 on 3x 8 GB (PP3 DSpark)

From CMP Wiki
DeepSeek-V4-Flash-0731
Reported 3 x 8 GB
QuantNative MXFP4 experts + FP8 attention (official deepseek-ai checkpoint, about 155.4 GiB); KV cache fp8
EnginevLLM: haosdent/vllm branch dsv4-flash-a100 with a vendored copy of allover326's 7 patches (image dsv4-a100:devel) f8ea5bb-era base with the allover326 patch set as of mid-August 2026
TopologyPP3 (VLLM_PP_LAYER_PARTITION=15,15,13) with DSpark, 5 draft tokens; PCIe Gen2 x4, no P2P
Context691,968 (largest stable context at --gpu-memory-utilization 0.965)
Decode93 tok/s
VRAM per card64 GB (65,536 MiB), --gpu-memory-utilization 0.965
Power cap100 W
Driver610.43.02 (nvidia-driver-610-open)
Unlock buildcmpunlocker 360acd7 (2026-08-06), 8 GB profile
Flags--pipeline-parallel-size 3 --kv-cache-dtype fp8 --block-size 256 --max-model-len 691968 --max-num-batched-tokens 2048 --gpu-memory-utilization 0.965 --max-num-seqs 8 --tokenizer-mode deepseek_v4 --no-enable-flashinfer-autotune --enable-auto-tool-choice --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4, DSpark speculative config with num_speculative_tokens 5, top_p 0.95 generation override; env VLLM_PP_LAYER_PARTITION=15,15,13, DSV4_LOGITS_ROW_CHUNK=64, VLLM_SPARSE_DENSE_QUERY_BLOCK=0, VLLM_WORKER_MULTIPROC_METHOD=spawn; container --shm-size=16g
Tested@tim-odonnell on 2026-08-15
Sourcelink
Filesvllm-dsv4/phase5-launch-dspark-production.sh, HOWTO-RUN-DEEPSEEK-V4-FLASH.md, PERFORMANCE-SUMMARY.md (the repo has no license file)
ConfirmationsNone yet.

DeepSeek-V4-Flash-0731 (about 284B total, 13B active) on three 64 GB cards, served by vLLM with pipeline parallel 3 and DSpark speculative decoding. The engine is the same patched build as the four-card reference recipe (DeepSeek V4 Flash 0731 on 4x 8 GB (allover326 PP4 DSpark)); the difference is the layer split, 15,15,13, and a tighter memory budget. Three cards are the smallest set that holds the official MXFP4 + FP8 checkpoint, and a fourth card mainly buys context, not speed. DeeJay runs it on three cards daily and called three cards at about $1,200 each by far the cheapest way to run the model locally. tim-odonnell's repo (posted on Discord by irythros) documents a production setup at 100 W per card: about 93 tok/s decode with a 691,968-token context. The infobox shows that repo's own readings.

Reported 8 GB
Source: @deejay9983, 2026-08-05
Source: repo github.com/tim-odonnell/cmp170hx-deepseek-v4-flash by tim-odonnell on GitHub, posted by @irythros, 2026-08-19

Steps

What you need

  • Three CMP 170HX 8 GB cards with the VRAM unlock (Driver unlock, 8 GB card): nvidia-smi must report 65,536 MiB per card. tim-odonnell's repo used cmpunlocker commit 360acd7 with driver 610.43.02 and the 8 GB profile, followed by a cold power-off; its beginner unlock guide is dated to that version, so check the current unlock pages first.
  • PCIe Gen2 x4 per card with no P2P is enough. The repo's box is an ASRock Rack ROMED8-2T with an EPYC 7413.
  • Two cards are not enough: about 155.4 GiB (166.9 GB) of weights does not fit in two cards' VRAM.
  • Docker with the NVIDIA runtime and the dsv4-a100:devel image, built from haosdent/vllm branch dsv4-flash-a100 with allover326's patches. The build steps are on the four-card page.
  • The official deepseek-ai/DeepSeek-V4-Flash-0731 checkpoint, about 156 GB in 48 shards, on a local disk.

Reported 8 GB
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub, posted by @deejay9983, 2026-08-05
Source: repo github.com/tim-odonnell/cmp170hx-deepseek-v4-flash by tim-odonnell on GitHub, posted by @irythros, 2026-08-19

The 15,15,13 layer split is required

The model has 43 layers. Over three ranks vLLM defaults to 15,14,14, and that fails at load with an illegal memory access in the Marlin FP4 expert repack: the last rank also carries lm_head and the DSpark drafter, so it runs out of room during the repack. Set VLLM_PP_LAYER_PARTITION=15,15,13 to give that rank fewer layers. The error reads like a Marlin bug but is a placement problem, so --gpu-memory-utilization cannot fix it (it sizes the KV pool, not the weights). allover326's repo first said three cards could not run the model and retracted that on 2026-08-09 once the split was found. DeeJay's container runs on three cards with no other changes; on a host with more cards, pin the three with CUDA_VISIBLE_DEVICES=0,1,2 (tim-odonnell's script passes NVIDIA_VISIBLE_DEVICES=0,1,2 to the container).

Reported 8 GB
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub, posted by @deejay9983, 2026-08-05
Source: repo github.com/tim-odonnell/cmp170hx-deepseek-v4-flash by tim-odonnell on GitHub

Launch with the production script at 100 W per card

  1. Cap each card at 100 W (nvidia-smi -i 0 -pl 100, and the same for cards 1 and 2). The repo applies the cap with a systemd service at boot.
  2. Build dsv4-a100:devel as on the four-card page.
  3. Clone tim-odonnell/cmp170hx-deepseek-v4-flash, set the model path in vllm-dsv4/phase5-launch-dspark-production.sh, and run the script with no arguments. Its positional arguments are utilisation, max length, max sequences, partition and max batched tokens, with defaults 0.965, 691968, 8, 15,15,13 and 2048.
  4. Wait 3 to 8 minutes for the load, until docker logs -f dsv4-a100-3card shows "Application startup complete". curl -s http://localhost:8098/health then returns success and curl -s http://localhost:8098/v1/models lists dsv4s.
  5. Point any OpenAI-compatible client at http://localhost:8098/v1, model dsv4s.
  6. Stop with docker stop dsv4-a100-3card, which frees all GPU memory.

Before relaunching after a crash, read the recovery warning under Known issues: the script reloads nvidia_uvm on its own when its start-up probe finds the cards wedged.

What the script runs (settled 2026-08-15):

docker run -d --name dsv4-a100-3card --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=0,1,2 \
  -e HF_HUB_OFFLINE=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e DSV4_LOGITS_ROW_CHUNK=64 -e VLLM_SPARSE_DENSE_QUERY_BLOCK=0 \
  -e VLLM_PP_LAYER_PARTITION=15,15,13 \
  -v "$MODEL":/model --shm-size=16g -p 8098:8000 \
  dsv4-a100:devel vllm serve /model --served-model-name dsv4s \
  --pipeline-parallel-size 3 --kv-cache-dtype fp8 --block-size 256 \
  --max-model-len 691968 --max-num-batched-tokens 2048 --trust-remote-code \
  --gpu-memory-utilization 0.965 --max-num-seqs 8 \
  --override-generation-config '{"top_p": 0.95}' \
  --no-enable-flashinfer-autotune --tokenizer-mode deepseek_v4 \
  --enable-auto-tool-choice --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5}'

top_p 0.95 replaces the checkpoint's 1.0 on DeepSeek's recommendation for agentic work; the repo's A/B test on 2026-08-20 found no speed cost. Temperature stays at the checkpoint default of 1.0. Thinking is returned in reasoning_content, separate from the answer.

Reported 8 GB
Source: repo github.com/tim-odonnell/cmp170hx-deepseek-v4-flash by tim-odonnell on GitHub, posted by @irythros, 2026-08-19

Plain PP3 for very long documents

For documents past about 300k tokens the repo's how-to recommends the launcher without speculative decoding, vllm-dsv4/phase3-launch-nospec.sh 691968: same API and port, slower per token (55.8 against 80.95 tok/s average in the 3-card test), and in the repo's experience more stable at extreme depth.

Reported 8 GB
Source: repo github.com/tim-odonnell/cmp170hx-deepseek-v4-flash by tim-odonnell on GitHub

Measured numbers

tim-odonnell repo: about 93 tok/s at 100 W per card, 691,968-token context

The repo reports, for three cards at 100 W, PP3 with the settings above:

run decode (tok/s)
production config, DSpark about 93
3-card DSpark test, average (peak) 80.95 (99.93)
3-card test without DSpark 55.8
best llama.cpp run on the same box 23.8

While generating, the three cards draw about 245 W together, about 1,150 tokens per Wh. During a 115k-token prompt the instantaneous peaks were 158, 235 and 193 W per card; the cap holds the short-window average, so brief spikes above 100 W are normal. These are the repo's own readings.

Reported 8 GB
Source: repo github.com/tim-odonnell/cmp170hx-deepseek-v4-flash by tim-odonnell on GitHub, posted by @irythros, 2026-08-19

DeeJay's daily driver: three cards, 128k per agent, five agents at once

DeeJay (2026-08-05) runs DeepSeek V4 Flash MXFP4 on three cards every day and keeps his fourth card for Gemma 4 (multimodal, cameras and home automation). On three cards his maximum was then about 262k context, but 262k once produced hallucinated output he could not reproduce, while 128k showed none; the 128k applies per agent. He has run five agents at once through Hermes, side by side at least 2x Claude's speed (no tok/s posted). He re-verified on 2026-08-06 that his container runs on three cards unchanged, with a fresh docker pull on a separate instance. robbbbbb (2026-08-09) runs DeepSeek V4 Flash (with the patches) next to Qwen 27B on three cards, and jac noted three cards are enough.

Reported 8 GB
Source: @deejay9983, 2026-08-05
Source: @artistosu, 2026-08-05
Source: @fallengod9962, 2026-08-05
Source: @roger_dammit, 2026-08-06
Source: @robbbbbb7973, 2026-08-09
Source: @jecquas, 2026-08-09

allover326's 3-card runs: 78.8 tok/s at 131k, 524,288 context with the INT4 repack

allover326's repo verified the native weights on three cards at --gpu-memory-utilization 0.95, --max-model-len 131072, DSpark with 5 tokens, DSV4_LOGITS_ROW_CHUNK=256 and the 15,15,13 split: 222,408 KV tokens and about 78.8 tok/s median decode, with 825 generations and a 61-turn conversation to 118,411 tokens and no errors. The INT4 compressed-tensors repack (about 150.4 GiB) is the better 3-card option when context matters more than native weights: --max-model-len 524288 with row chunk 64 and utilisation 0.95 gave a 2,013,978-token KV pool and about 86 tok/s on short prompts, and a needle was retrieved correctly at 401,532 prompt tokens (174 s cold prefill). Schaka's fork carries the same results file and the same 3-card INT4 numbers.

Reported 8 GB
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub, posted by @deejay9983, 2026-08-05
Source: repo github.com/Schaka/deepseek-v4-cmp170hx by Schaka on GitHub, posted by @schakaqt, 2026-09-02

Context on three cards: 960k, 1M and what holds under load

flipnburn (2026-08-09) got the model running on three cards by pointing Claude at DeeJay's repo, using the official safetensors (the native FP8 + FP4 quant). He had to cut context to 960k because 1M did not quite fit; DeeJay replied that 1M is fully achievable on three cards. robbbbbb runs Flash on three cards and misses only vision; ppc64_ found more room for KV cache than expected, and giaki3003 noted DeepSeek is very efficient with KV cache. On 2026-08-26 DeeJay pointed three-card users at issue #2 of his repo (three cards working on vLLM 0.26 with the full 1M context) and suggested having an AI agent apply the patches in order; mfj4 got his third card online that way. When Josh noted the repo looked outdated (no GLM 5.3 merge), DeeJay answered "if it aint broke, dont fix it".

The context math on 2026-08-19: Adust estimated about 500k on three cards with the INT4 repack (192 GB total, about 170 GB usable at 0.95 utilisation); DaLZy said someone had run 1M on three cards, probably with PP (weights about 160 GB, KV about 15 GB). On tensor parallel, SpiralGlide said TP needs a power of 2 for the head count, DaLZy said heads can be padded, and giaki3003 suggested splitting or MIG.

How much of that survives concurrent load depends on utilisation: tim-odonnell's stress tests at 8 concurrent sequences near the ceiling passed at 691,968 tokens (0.965) and crashed at 1,048,576 (0.98); see Known issues. Recall also falls with depth on this model whatever the card count; the figures are on the four-card page.

Reported 8 GB
Source: @flipnburner, 2026-08-09
Source: @deejay9983, 2026-08-09
Source: @robbbbbb7973, 2026-08-09
Source: @ppc64_, 2026-08-09
Source: @giaki3003, 2026-08-19
Source: @adusts, 2026-08-19
Source: @spiralglide, 2026-08-19
Source: @dalzyyy, 2026-08-19
Source: @mfj4, 2026-08-27
Source: @_____josh_____, 2026-08-27

A longer write-up by @flipnburner exists (a README for his working three-card setup, with a server TUI); see Discord.

Three cards on llama.cpp: about 23 tok/s decode, 260 tok/s prompt processing

aaron (2026-08-01) ran the GGUF with a layer split across three cards at 65k context in the llama.cpp:server-cuda docker image, untuned: about 23 +/- 2 tok/s decode and about 260 tok/s prompt processing. vLLM with PP and DSpark later reached about 3 to 4x this decode on the same cards. More on llama.cpp on CMP cards.

Reported
Source: @aaron0010, 2026-08-01

nightdays: 350 tok/s aggregate at 8 concurrent requests on PCIe 1.1 x4

nightdays reported DeepSeek V4 Flash 0731 on three cards in vLLM at 8 concurrent requests, 350 tok/s aggregate, with the links at PCIe 1.1 x4. arabus ran DeepSeek V4 Flash on two cards in LM Studio and got slow results; Ezy said not to use LM Studio, and ST suggested Unsloth Studio or vLLM.

Reported
Source: @.nightdays, 2026-08-30
Source: @arabus_74064, 2026-08-30
Source: @ezybr0, 2026-08-30
Source: @stcord, 2026-08-30

skurtdiggler: 120+ tok/s on three water-cooled cards

skurtdiggler's three water-cooled cards (bought on eBay with Bykski blocks fitted) ran DeepSeek V4 across the three cards at 120+ tok/s, against Qwen3.8 27B on a single card at about 33 tok/s, benchmarked with HumanEval, BigCodeBench-Hard and LoveCodeBench v6. Engine settings were not posted. DeeJay said Qwen on one card should be closer to 75 tok/s, and Studebaker asked whether three GPUs for DeepSeek beat three instances of Qwen3.8 27B FP8.

Reported
Source: @skurtdiggler, 2026-08-18
Source: @skurtdiggler, 2026-08-18
Source: @_studebaker_, 2026-08-18
Source: @deejay9983, 2026-08-18

Vision-Exp

What DeepSeek-V4-Flash-Vision-Exp is

greencandle (2026-08-31) tried DeepSeek V4 Flash with vision on three cards. jac: it is the 0731 Flash model with vision added, scoring 1 to 2 points higher on benchmarks. greencandle: it beats Qwen Flash and comes close to a W4 GLM Flash. DeeJay: DeepSeek Flash is natively 4-bit and has a high hallucination rate. The checkpoint is the same text model plus a 411M-parameter ViT and aligner (about 0.9 GiB in bf16); images only.

Reported
Source: @greencandle9753, 2026-08-31
Source: @jecquas, 2026-08-31
Source: @deejay9983, 2026-09-01

ElRoberto538 VISION.md: no speculative decoding, 25 to 60 tok/s

robbbbbb (2026-09-01 to 03) had his agent set Vision-Exp up with patched vLLM per VISION.md in his ElRoberto538 fork. On this path vision disables speculative decoding. On three cards with one PCIe lane each he gets 25 to 60 tok/s decode (about 60 single stream, about 25 per stream when his client sends parallel requests; prefill not measured), with no broken tool calls on either DeepSeek V4 Flash version but nothing past 250k tokens in one session; he uses it for CAD work. VISION.md serves the native 1,048,576 tokens with the vision tower resident. Patch 0007 (a port of vLLM #54566) changes a CUDA op schema, so the image is a source build (Dockerfile.devel-vision, TORCH_CUDA_ARCH_LIST=8.0), not the precompiled text image. With no DSpark drafter on the last rank, VISION.md rebalances the split to VLLM_PP_LAYER_PARTITION=14,15,14 and omits --speculative-config.

Reported
Source: repo github.com/ElRoberto538/deepseek-v4-cmp170hx by ElRoberto538 on GitHub, posted by @robbbbbb7973, 2026-09-01

tim-odonnell's Vision-Exp build: about 86 tok/s with DSpark, 691,968 context

tim-odonnell's later build (repo folder vision-exp-3card, 2026-09-29) uses the leinasi2014/deepseek-v4-vision-cmp170hx vLLM fork (CUTLASS v4.4.2, Triton v3.5.1, sm_80) plus one local patch. The fork builds a copy of the 0.87 GiB vision encoder on every pipeline stage although only the first uses it; on three cards that dead copy was the memory the last card lacked, and the patch keeps the encoder on PP rank 0 only. The launcher keeps DSpark and the 0731 memory settings. Measured at 100 W per card, utilisation 0.965, 691,968 context and a 797,299-token KV pool, against the 0731 recipe on the same cards:

0731 Vision-Exp
decode, short prompt (DSpark) about 99.7 about 86
decode at 100k / 300k context 80 / 79 78 / 68
prefill at 100k / 300k 1,935 / 2,282 2,154 / 2,384

The short-prompt gap comes from fewer accepted draft tokens (about 2.9 against 3.5 per step), not from the engine. The server listens on port 8099 with model name dsv4v, thinking is off by default, and it accepts up to 8 images per conversation (384 tokens each). Under load the cores reached 67 C and the HBM 77 C. A healthy start logs "GPU KV cache size: 797,299 tokens". Download the checkpoint (about 168 GB) with one connection per file: the Hugging Face Xet CDN rejects split downloads with a 403.

Reported
Source: repo github.com/tim-odonnell/cmp170hx-deepseek-v4-flash by tim-odonnell on GitHub

Known issues

Utilisation cliff: 0.965 passes, 0.97 crashes on Worker_PP2

tim-odonnell binary-searched --gpu-memory-utilization under --max-num-seqs 8, stressing each setting with concurrent requests and a real prompt near its context ceiling:

utilisation max context result
0.95 338,432 pass
0.96 574,464 pass
0.965 691,968 pass (production default)
0.97 799,488 crash: illegal memory access on Worker_PP2
0.98 1,048,576 crash

The edge is a cliff, not a slope. Changing the layer partition did not help: all three ranks land at the same total fill whatever the split. Do not raise utilisation above 0.965 on this workload without rerunning the same stress test.

Reported 8 GB
Source: repo github.com/tim-odonnell/cmp170hx-deepseek-v4-flash by tim-odonnell on GitHub, posted by @irythros, 2026-08-19

The four-card utilisation of 0.85 fails on three cards

With the four-card launcher's --gpu-memory-utilization 0.85, three cards fail at start-up with "No available memory for the cache blocks". Raise it (0.95 to 0.965, see above).

Reported 8 GB
Source: repo github.com/tim-odonnell/cmp170hx-deepseek-v4-flash by tim-odonnell on GitHub, posted by @irythros, 2026-08-19

Triton shared-memory error: set VLLM_SPARSE_DENSE_QUERY_BLOCK=0

On three cards the Triton _sparse_attn_prefill_blocked_kernel asked for 204,800 B of shared memory against the A100 limit of 166,912 B and failed until VLLM_SPARSE_DENSE_QUERY_BLOCK=0 was set. The production launcher sets it.

Reported 8 GB
Source: repo github.com/tim-odonnell/cmp170hx-deepseek-v4-flash by tim-odonnell on GitHub, posted by @irythros, 2026-08-19

Keep DSV4_LOGITS_ROW_CHUNK=64: unchunked, the last rank faults near 174k

On three cards the last PP rank keeps only about 1 to 2 GiB free after weights and KV cache. With DSV4_LOGITS_ROW_CHUNK unset (the code default is off), the sparse indexer allocates its whole logits buffer for a 2,048-row prefill chunk at once, about 1.33 GiB at 174k context, and the first request past roughly 174,000 tokens dies with an illegal memory access and Xid 31 on that rank. With 64 the buffer stays around 0.04 GiB at the same length. The crash depends on request length, not uptime. Row chunking is explained on the four-card page.

Reported 8 GB
Source: repo github.com/tim-odonnell/cmp170hx-deepseek-v4-flash by tim-odonnell on GitHub

Recovering a wedged card: module reload or reboot

Contested 8 GB

  • allover326's launcher recovers wedged cards with nvidia-smi -r, then rmmod and modprobe nvidia_uvm, then re-applies -pl 180; Schaka's launchers also reload nvidia_uvm to clear an Xid 154 wedge. Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub
    Source: repo github.com/Schaka/deepseek-v4-cmp170hx by Schaka on GitHub
  • tim-odonnell's Xid 31 guide says never rmmod nvidia_uvm: on 2026-09-29 that turned one Xid 31 into Xid 154 on all 4 cards of his box and only a reboot recovered. dkpoulsen (GLM-5.3-Flash PP5) found a crash wedges the GPU at RM level (nvidia-smi looks healthy, the next start fails cudaSetDevice) until a host reboot, the tell being a nonzero dmesg | grep -ci xid; kaka86mm needs a cold power cycle after an Xid 31 (about 1 in 10 engine starts) and says nvidia-smi -r is not supported; allover326's dsa-mtp kit warns never to kill -9 a crashed multi-GPU vLLM run, because killed workers strand CUDA contexts that only a reboot clears. Source: repo github.com/tim-odonnell/cmp170hx-deepseek-v4-flash by tim-odonnell on GitHub
    Source: repo github.com/dkpoulsen/vllm-170hx-glm5.3-flash by dkpoulsen on GitHub
    Source: repo github.com/kaka86mm/glm53-flash-170hx by kaka86mm on GitHub
    Source: repo github.com/allover326/vllm-dsa-mtp-sm80 by allover326 on GitHub

After an Xid 31, reboot before relaunching: the production script reloads nvidia_uvm on its own

Warning 8 GB On tim-odonnell's box, reloading nvidia_uvm after an Xid 31 turned one card's error into Xid 154 on all four cards, and only a reboot recovered (2026-09-29). His own phase5-launch-dspark-production.sh, settled on 2026-08-15 before that incident, still runs nvidia-smi -r and an nvidia_uvm reload by itself when its start-up probe cannot create CUDA contexts on the three cards, so a relaunch straight after a crash can escalate it. His later Vision-Exp launcher refuses to start in that state and tells you to reboot instead. After a crash: stop the container cleanly (never kill -9 the workers), check sudo dmesg -T | grep -i xid, and if anything new shows, reboot the host, re-apply the 100 W cap, then launch. Xid background: Xid 31 and the reserved top of VRAM.

Source: repo github.com/tim-odonnell/cmp170hx-deepseek-v4-flash by tim-odonnell on GitHub
Source: repo github.com/allover326/vllm-dsa-mtp-sm80 by allover326 on GitHub

"RPC call to sample_tokens timed out" on the first token

On a three-card setup aaron hit the EngineCore error "TimeoutError: RPC call to sample_tokens timed out." while generating the first token (2026-08-09). No root cause was posted. He asked flipnburn for his final command to compare, and flipnburn answered with the README of his working three-card setup (see the pointer under Context on three cards).

Reported
Source: @aaron0010, 2026-08-09
Source: @flipnburner, 2026-08-09

Related

Sources