Jump to content
Toggle search
Toggle menu
CMP Wiki
Toggle preferences menu
Toggle personal menu
Not logged in
Your IP address will be publicly visible if you make any edits.

DeepSeek V4.1 Flash on 4x 8 GB (GNmac)

From CMP Wiki
More actions
DeepSeek-V4.1-Flash-1017
Proven 4 x 8 GB
QuantHybrid: routed experts EXL3 (mostly 3 bpw, 2 bpw on layers 18-22) from LibertAI's 2.9 bpw pack; attention, shared experts, indexer and DSpark drafter official FP8; KV fp8_ds_mla
EnginevLLM with a pinned sm_80 patch stack (built locally from source) recipe 1.2.0
TopologyTP4 (default) or PP4 for x1/x4 links
Context262,144 (up to 1,048,576)
Decode258 tok/s
Prefill2416 tok/s
VRAM per card64 GiB, GPU_MEMORY_UTILIZATION 0.95
Power cap180 W (nvidia-smi -pl 180)
DriverCUDA 13 capable; measured on 610.43.02
Unlock buildcmpunlocker (64 GiB, 74 SMs) with a P2P-capable driver for TP4
FlagsLAYOUT=tp4, MAX_MODEL_LEN 262144, MAX_NUM_SEQS 8, GPU_MEMORY_UTILIZATION 0.95, P2P 1, DSV41_GATE_TAU 0.10, DSV41_DYNK accept:3,5
Tested@greencandle9753 on 2026-10-26
Sourcelink
Confirmations
Reproduced by @darkmaniac7 (8 GB), 2026-10-27 · details
Partly reproduced by @mountains1605 (8 GB), 2026-10-23 · details

DeepSeek-V4.1-Flash-1017 on four 64 GB cards with vLLM and a patch stack for sm_80, an OpenAI-compatible API with tools, reasoning and images, and up to 1M tokens of context. The checkpoint is a hybrid (Curtis White's layout): EXL3 routed experts plus the official FP8 dense tensors, with the 189 GiB Engram n-gram tables streamed from NVMe instead of loaded, so it needs far less host RAM than the Morrowmake recipe. Infobox numbers are release 1.2.0, one user on code. Source: repo github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx by GNmac-lab on GitHub, posted by @greencandle9753, 2026-10-21
Source: @greencandle9753, 2026-10-21

Read the long-context copy finding under Known issues before using it for work that must copy names or paths exactly.

Prerequisites

  • Four CMP 170HX at 64 GiB and 74 SMs (Driver unlock, 8 GB card).
  • PCIe 2.0 x16 links with a P2P-capable driver for TP4 (tools/p2p_check.sh); on x1 or x4 links use LAYOUT=pp4. greencandle: to match his single-user numbers you have to do the x16 mod. Source: @greencandle9753, 2026-10-22
  • Driver with CUDA 13 support (measured on 610.43.02), Docker with the NVIDIA Container Toolkit.
  • About 400 GiB of fast NVMe for checkpoints plus about 40 GiB for the image; 64 GB+ of host RAM.

Source: repo github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx by GNmac-lab on GitHub, posted by @greencandle9753, 2026-10-21

Steps

git clone https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx && cd deepseek-v41-flash-1017-cmp170hx
cp .env.example .env            # set MODELS_DIR (fast NVMe, ~400 GiB free)
tools/download.sh               # model parts + hybrid assembly (resumable)
./start.sh                      # builds the image on first use (~30-45 min), starts, waits until ready
./start.sh smoke                # one chat, one tool call, one image

Test request (model name deepseek-v4.1-flash): 

curl -s http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "deepseek-v4.1-flash", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 200}'

The server binds 127.0.0.1:8000 with no API key; reach it from another machine through an SSH tunnel (ssh -N -L 127.0.0.1:8000:127.0.0.1:8000 you@gpu-host). Update: git pull && docker/build.sh && ./start.sh restart. Source: repo github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx by GNmac-lab on GitHub, posted by @greencandle9753, 2026-10-21

Checkpoint and engine

  • Routed experts, layers 0-39: EXL3 trellis mul1, 3 bpw (2 bpw on layers 18-22), from LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw (197 GB).
  • Attention, shared experts, KV compressor, indexer and the DSpark drafter: FP8 e4m3 UE8M0 from the official release (18 GB, fetched by HTTP range requests).
  • Engram tables for layers 1 and 14: FP8+E8M0, 190 GB (shards 47-48), read from NVMe while serving. KV cache fp8_ds_mla.
  • Engine pins (NOTICE.md): base image vllm/vllm-openai:deepseekv41-flash-0909, zebgop-ops/dsv41reap-pp 138e748, kaka86mm/dsv41-flash-pp4-170hx 950f4c0, turboderp exllamav3 v1.5.3 (d3739fd), vcruz305/vllm-exl3 223e246, CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE 88eb6e1, FlashInfer v0.7.0.post1 (PCIe IPC all-reduce), HalfVulpes/Deepinfer-SM80 2a828fc, Morrowmake/vllm-cmp170hx a0c57b5; TORCH_CUDA_ARCH_LIST=8.0. No prebuilt image: the first build takes 30 to 45 minutes.
  • .env keys: LAYOUT tp4 or pp4, MAX_MODEL_LEN 262144 (up to 1048576, KV pool then 1,270,789 tokens), MAX_NUM_SEQS 8, GPU_MEMORY_UTILIZATION 0.95, P2P 1 (0 routes NCCL through host memory), DSV41_GATE_TAU 0.10 (confidence-gated draft verification; 0 verifies all 5 drafts), DSV41_DYNK accept:3,5 (TP4 only), DSV41_VARIANTS, BIND, PORT.
  • reasoning_effort accepts none, low, high, xhigh, max; minimal and medium are rejected by the chat template (HTTP 400).

Source: repo github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx by GNmac-lab on GitHub, posted by @greencandle9753, 2026-10-21

Measured numbers

Author's numbers, 4 cards at 180 W, MiaAI-Lab protocol, tok/s structured / code / prose:

Release Layout 1 user 8 users Cold prefill KV pool
1.0.0 (2026-10-21) TP4 296 / 212 / 150 796 / 477 / 359 1,759 1.57M
1.1.0 (2026-10-23) TP4 336 / 247 / 180 +6 to +7% on code and prose 1,764
1.2.0 (2026-10-26) TP4 356 / 258 / 184 868 / 544 / 441 2,416 1,395,731
1.2.0 PP4 118 / 91 / 63 341 / 214 / 163 2,311 2,539,313

1.2.0 at 2 users: 433 / 244 / 239; at 4 users: 626 / 388 / 349. Quality 1.2.0 (TP4): HumanEval 154/164, GSM8K 1,275/1,319 (1.1.0: 156 and 1,273); dqeval NLL 3.50001 against 3.48019; a 20-minute 8-stream soak served 1,018 requests with 0 errors at 334.8 tok/s. In a quick comparison it gave greencandle better results than his previous GLM Flash setup. Source: repo github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx by GNmac-lab on GitHub, posted by @greencandle9753, 2026-10-21
Source: @greencandle9753, 2026-10-23
Source: @greencandle9753, 2026-10-22

  • Decode was the focus; prefill has headroom. cibernox: prefill on the weaker side but totally usable. Source: @greencandle9753, 2026-10-21
    Source: @cibernox, 2026-10-21
  • KLD for the weights is on the Mia-AiLab EXL3 EAQ v1 2.9 bpw model card. The vLLM changes are kernel, scheduling or memory changes that are bit-identical or only reorder FP32 sums; speculative decoding does not change the output distribution because the target model verifies every kept token. Source: @greencandle9753, 2026-10-23

Other people's numbers

  • Ivan (TP4 on four cards of an 8-card box, Gen2 x16, P2P on, his own bench with 1,024-token answers): one user code 244, structured 212, prose 136; 8 users 522 / 425 / 346; prefill 2.0k to 2.6k tok/s (a 48k prompt in 20 to 25 s); KV pool 1.4M at 262K. DSpark gives 2.6x decode. Keep the prefill budget at 2,048 as shipped: 8,192 and 16,384 were slower, and 8,192 with DSpark crashed. Source: @darkmaniac7, 2026-10-28 (lab notes)
  • Mountains (four cards on PCIe Gen2 x4): PP4 did not work well, so he switched to TP4 with P2P; single stream maybe 10% below GLM 5.3 Flash and never felt slow; first load about 8 minutes or more on Gen2 x4. He found it a bit more intelligent than GLM 5.3 Flash but less consistent, went back to GLM, then moved to the Morrowmake build, which he found better on prefill and quality. Source: @mountains1605, 2026-10-22
    Source: @mountains1605, 2026-10-23
    Source: @mountains1605, 2026-10-27

Known issues

  • Long-context copy errors (engine defect). On Ivan's test (75k to 92k-token roleplay turns that must copy image paths and names exactly), this build got 85% of named paths wrong. Bisecting one change at a time on stock dense weights: cutting the prompt to the last 5 turns, 33% wrong (long-range); BF16 KV cache, 100%; the synchronous Engram path, 100%; OpenRouter's full-precision 1017, 0%; Morrowmake's recipe with the same 2.9 bpw expert files, 0% (0 of 29). Ruled out: the heretic patch, the sampler, DSpark, the prompts, the detokenizer (the model ranks the wrong token first). So the defect is in this stack's sm_80 attention or kernel path, not the quant. See DeepSeek V4.1 Flash on 4x 8 GB (Morrowmake). Source: @darkmaniac7, 2026-10-29 (lab notes)
  • TP4 on links narrower than x16 is very slow (about 86 all-reduces per step); use PP4 there. PP4 on x4 links is extrapolated by the author, not measured.
  • A startup hang at 100% GPU and low power is an all-reduce waiting on broken P2P: try P2P=0. Never kill -9 a multi-GPU vLLM; use ./stop.sh.
  • Prose runs about 20% slower right after a restart until the Engram rows are in the page cache. A 1M-token prefill takes about 25 minutes.
  • TP4 greedy text is not bit-reproducible across processes (compare NLL, not text).
  • A worker OOM at load (about 25 GB host RAM per worker) means VLLM_EXL3_MODEL_DIR is unset.
  • License: MIT except patches/vllm-exl3/, which is AGPL-3.0-only; anyone serving others from the built image inherits AGPL obligations (NOTICE.md). The hybrid assembly depends on a quantization_config.json fetched from CurtisAccelerate's repo, which states no license.

Source: repo github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx by GNmac-lab on GitHub, posted by @greencandle9753, 2026-10-21

  • A slow NVMe and little RAM headroom can struggle with the streamed Engram tables. Source: @greencandle9753, 2026-10-21

Build and run traps (Ivan)

  • The draft count (SPEC_TOKENS) must be 5: 6 and 7 do not start; 3 and 4 boot but every request fails.
  • Keep the shipped prefill budget of 2,048: 8,192 without DSpark gave 1,400-1,770 tok/s at 48k and 94 tok/s decode, 16,384 shrank the KV pool to 619k tokens, and 8,192 with DSpark died on the first 48k request. The prefill chunk must stay 2,048 with images (1,024 does not start, 4,096 runs out of memory).
  • The image build needs Docker buildx (the Dockerfile uses RUN --mount=type=bind) and runs about 32 nvcc jobs at once (MAX_JOBS=16, two stages in parallel), so build on a machine that can take it. It took 1,351 s here, no GPU needed, about 40 GiB.
  • serve.sh uses --gpus all and a busy guard that sums memory over every card, so a second instance on other cards of the same box needs a wrapper.
  • This stack never uses vLLM's custom all-reduce; it uses FlashInfer's PCIe IPC all-reduce (VLLM_ALLREDUCE_USE_FLASHINFER_PCIE_IPC=1).

Source: @darkmaniac7, 2026-10-28 (lab notes)

Abliterated variant

pqhaz's heretic patch edits only the 38 attn.wo_b tensors, and this recipe takes exactly those dense tensors from the official checkpoint, so the patch drops in with no requantization: rewrite the 18 GB of non-expert shards with the patch and point the hybrid at them. Uncensored EXL3 bundles whose edit lives in other dense tensors would silently do nothing here. See DeepSeek V4.1 Flash heretic on 4x 8 GB. Source: @darkmaniac7, 2026-10-28 (lab notes)

Related

Sources

Imported from the cmpunlocker Discord with credit; see Credits.
Retrieved from "https://cmp.wiki/w/index.php?title=DeepSeek_V4.1_Flash_on_4x_8_GB_(GNmac)&oldid=252"