Jump to content
Toggle search
Toggle menu
CMP Wiki
Toggle preferences menu
Toggle personal menu
Not logged in
Your IP address will be publicly visible if you make any edits.

DeepSeek V4.1 Flash on 4x 8 GB (Morrowmake)

From CMP Wiki
More actions
DeepSeek-V4.1-Flash-1017
Reported 4 x 8 GB
QuantHybrid: routed experts EXL3 2.9 bpw (LibertAIDAI), everything else official FP8/BF16; KV BF16
EngineMorrowmake/vllm-cmp170hx (vLLM fork, container image) recipe 1.0.0, engine a0c57b5237
TopologyTP4 with expert parallel (96 whole experts per card)
Context262,144
Decode287 tok/s
Prefill3818 tok/s
VRAM per card64 GiB exposed, --gpu-memory-utilization 0.95
Power cap180 W
DriverNVIDIA 580 or newer; P2P needs the Morrowmake/cmpunlocker build
Unlock buildcmpunlocker (64 GiB, 74 SMs); Morrowmake/cmpunlocker for P2P
FlagsP2P=auto, VLLM_ENGRAM_PLACEMENT=auto (ram or ssd), MAX_SEQS 8, MAX_BATCHED 2048, DSpark drafter up to 7 tokens
Tested@voidgiraffe on 2026-10-24
Sourcelink
Confirmations
Partly reproduced by @darkmaniac7 (8 GB), 2026-10-29 · details
Partly reproduced by @mountains1605 (8 GB), 2026-10-27 · details
Partly reproduced by @CurtisAccelerate (8 GB), 2026-10-24 · details

DeepSeek-V4.1-Flash-1017 on four 64 GB cards with tool calls, reasoning, images and the checkpoint's own DSpark drafter. The served checkpoint is a hybrid built locally from links: routed experts from LibertAI's EXL3 2.9 bpw export, every other weight in DeepSeek's own FP8/BF16, and the two Engram lookup tables (about 189 GiB) in host RAM, or read from SSD on smaller hosts. Same engine family and tooling as GLM 5.3 Flash on 4x 8 GB. Infobox numbers are the author's "up to" figures for one user on code and cold prefill on a 2,048-token prompt. jjb: heavy lifting for the 4x 8 GB owners. Source: repo github.com/Morrowmake/dsv41-flash-cmp170hx-recipe by Morrowmake on GitHub, posted by @lia_oc, 2026-10-24
Source: @lia_oc, 2026-10-24
Source: @jjb01957, 2026-10-24

Prerequisites

  • Four CMP 170HX that each expose 64 GiB (preflight stops under 60 GiB, MIN_GPU_MIB 61440). See Driver unlock and 8 GB card.
  • PCIe Gen2 x16 links. The single TP4 layout runs on x4 but is bus-bound; ./start.sh warns below x16. See Unlocks.
  • NVIDIA driver 580 or newer; Docker with the NVIDIA Container Toolkit (docker run --rm --gpus all nvidia/cuda:13.3.1-base-ubuntu24.04 nvidia-smi lists four cards).
  • Host RAM: about 240 GiB for the published numbers (Engram tables in RAM; the server takes 202 to 204 GiB). Less RAM works with the tables on SSD; put MODELS_DIR on an NVMe with its own DRAM.
  • Disk: about 680 GiB (730 GB) in all (NEED_DISK_GB 680).
  • git, curl, flock, setsid, python3, jq.
  • Optional but needed for full speed: the P2P driver build (mainline cmpunlocker + maintainer P2P + TRAP31 Booter PLM patch, ForceP2P=0x11).

Source: repo github.com/Morrowmake/dsv41-flash-cmp170hx-recipe by Morrowmake on GitHub, posted by @lia_oc, 2026-10-24

Steps

git clone https://github.com/Morrowmake/dsv41-flash-cmp170hx-recipe.git
cd dsv41-flash-cmp170hx-recipe
./start.sh          # install, download, build the hybrid, launch, wait for /health, boot check
./start.sh smoke    # one chat request and one tool call

Test request (model id dsv41-flash): 

curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model": "dsv41-flash", "messages": [{"role": "user", "content": "Write a haiku about PCIe."}], "max_tokens": 4096}'

docker compose up is the alternative. The first start compiles kernels into ./cache and loads the Engram tables, so it is slow (READY_TIMEOUT 3600 s). Other subcommands: status, logs, stop, restart, update, DRY=1. Source: repo github.com/Morrowmake/dsv41-flash-cmp170hx-recipe by Morrowmake on GitHub, posted by @lia_oc, 2026-10-24

What the release pins

  • Engine: Morrowmake/vllm-cmp170hx branch ampere at a0c57b52376efd67c57735bc949d4bafa22ae29e on upstream vLLM 4e0681ed56. Image ghcr.io/morrowmake/vllm-cmp170hx:engine-a0c57b5237, digest sha256:f67d9be47ffbd4a8b84c77fe76bdb3f842187d9b2786fff573e390ae5361bfb0. The release's launch is baked into the image at /opt/recipe/serve.sh.
  • Export: LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw at c178a012 (about 196 GiB, routed experts only are used).
  • Official: deepseek-ai/DeepSeek-V4.1-Flash-1017 at 1bae0e10 (about 475 GiB FP8, including the Engram tables in its last two shards).
  • The served directory DeepSeek-V4.1-Flash-1017-hybrid-exl3-fp8 holds only links plus a config, index and quantization config that must match validated bytes.
  • Model: 384 routed experts per layer, 6 active per token.
  • Engine switches the release sets to 1, each a kill switch through .env: VLLM_DEEPSEEK_V41_ATTN_PIPELINE, VLLM_DEEPSEEK_V41_FUSED_ROUTE, VLLM_DEEPSEEK_V41_GATE_GEMV, VLLM_DEEPSEEK_V41_ROUTER_MMA, VLLM_EXL3_MOE_V2, VLLM_DEEPSEEK_V41_MOE_PREFILL_TC, VLLM_DEEPSEEK_V41_PREFILL_CE_ALLREDUCE, VLLM_DEEPSEEK_V41_PREFILL_SPLIT_REDUCE, VLLM_HOST_ALLREDUCE.
  • Serving defaults: 262,144 context, up to 8 requests, 2,048 tokens per scheduler step, CUDA graphs for decode steps up to 48 tokens, fair prefill in 512-token pieces, prefix caching on, deepseek_v41 tool and reasoning parsers, images on. DSpark drafts up to 7 tokens for one request, 5 for more; EXTRA_ARGS="--speculative-config None" turns speculation off.

Source: repo github.com/Morrowmake/dsv41-flash-cmp170hx-recipe by Morrowmake on GitHub, posted by @lia_oc, 2026-10-24

Engram placement

VLLM_ENGRAM_PLACEMENT=auto (default) picks RAM when 212 GiB (NEED_RAM_GIB) is available, else SSD. In SSD mode each card gets a row cache of (available RAM minus 29 GiB) / 4, clamped to 1 to 48 GiB (48 GiB holds a card's whole share); at 12 GiB or more it preloads at start (about 49 GB read per card) and reads past the page cache. Measured cost of SSD mode with a 38 GiB cache: first token on a cold 2K prompt 6% later (772 vs 725 ms), a 32K prompt 1% later, decode within 1%. Swap is disabled through MEM_LIMIT, because swapped Engram tables cost about 8% prefill. Source: repo github.com/Morrowmake/dsv41-flash-cmp170hx-recipe by Morrowmake on GitHub, posted by @lia_oc, 2026-10-24

Measured numbers

Author's release check, 2026-10-24: 74 SMs and 180 W per card, PCIe x16, P2P verified, Engram in RAM, MiaAI-Lab protocol (400 tokens, temperature 0, thinking off), median of five runs including time to first token. Tok/s structured / code / prose, marked "up to".

1 user 2 users 4 users 8 users
Default 319 / 287 / 162 360 / 431 / 230 545 / 602 / 324 765 / 729 / 432
  • Cold prefill 3,818 tok/s (2,048-token prompt), 3,808 tok/s (32,768-token prompt). KV pool 1,247,406 tokens (BF16, not quantized).
  • Quality: HumanEval 157/164, GSM8K 1,280/1,319, against a reference of 157/164 and 1,279/1,319.
  • Fallbacks (prose 1 user / 8 users / cold 2K prefill): Engram from SSD with a 48 GiB cache 151 / 428 / 3,607; P2P=off 133 / 344 / 3,804 (2 and 4 users without P2P up to 192 and 271).

Source: repo github.com/Morrowmake/dsv41-flash-cmp170hx-recipe by Morrowmake on GitHub, posted by @lia_oc, 2026-10-24

Other people's numbers

Contested Two other people measured this recipe well below the published figures, each with stated differences:

  • CurtisAccelerate (same board as his own recipe: dual-socket Broadwell, 140 W, peer access off), one user count / code / prose 122.8 / 90.3 / 62.4 tok/s, 8-user aggregate 168.7, cold prefill at 128K 1,558.7 tok/s (faster than his own stack at every length). Another of his tables reads 254.1 / 196.2 / 139.7, with large boot-to-boot spread. Source: repo github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX by CurtisAccelerate on GitHub, posted by @anon33anon, 2026-10-20
  • Ivan (heretic patch applied, a miner sharing the cards so --gpu-memory-utilization 0.92, Engram on SSD, his own bench): one user code 158, structured 136, prose 121; 8 users 323 / 261 / 249; warm prefill 2,453 tok/s on a 17k prompt. See DeepSeek V4.1 Flash heretic on 4x 8 GB. Source: @darkmaniac7, 2026-10-29 (lab notes)
  • Mountains ran it for two days on PCIe Gen2 x4 cards: better quality than the earlier GNmac build, roughly similar speed, handles complex work better, and he rates it above GLM 5.3 Flash; he is staying on it. No tok/s posted. Source: @mountains1605, 2026-10-27
    Source: @mountains1605, 2026-10-27

Accuracy on long contexts

On Ivan's long-context copy test (75k to 92k-token roleplay turns that must copy image paths and names exactly), this recipe's engine got 0 of 29 paths wrong with the same 2.9 bpw expert files on which the GNmac engine got 85% wrong. OpenRouter's full-precision 1017 also scored 0%. See DeepSeek V4.1 Flash on 4x 8 GB (GNmac). Source: @darkmaniac7, 2026-10-29 (lab notes)

Known issues

  • Reasoning is on by default and arrives in the reasoning field. "chat_template_kwargs": {"thinking": false} turns it off; reasoning_effort takes low, high (default), xhigh, max, or a number 1 to 100 in chat_template_kwargs. max_tokens must cover reasoning plus answer, or content is empty with finish_reason: "length".
  • One layout only (TP4), needing x16 links; no PP layout yet.
  • Heavy host requirements: about 240 GiB RAM for the published numbers and about 680 GiB of disk. alanoo notes it is much larger on disk than his mixed-bpw build. Source: @alanoo, 2026-10-27
  • First release (1.0.0), so less field history than the GLM recipe.

Source: repo github.com/Morrowmake/dsv41-flash-cmp170hx-recipe by Morrowmake on GitHub, posted by @lia_oc, 2026-10-24

  • With anything else holding memory on the cards (a miner holds 2.5 GiB per card), the default 0.95 runs out of memory on a 17k-token prefill; 0.92 fixed it. Source: @darkmaniac7, 2026-10-29 (lab notes)
  • The recipe does not pass CUDA_DEVICE_ORDER into the container. To serve from four cards of a larger box, add it to CONTAINER_KEYS in start.sh and set CUDA_DEVICE_ORDER=PCI_BUS_ID plus CUDA_VISIBLE_DEVICES in .env. Source: @darkmaniac7, 2026-10-29 (lab notes)

Related

Sources

  • Repo: Morrowmake/dsv41-flash-cmp170hx-recipe (MIT; checkpoints MIT), last commit 2026-10-26. Credits GNmac-lab's recipe for an adapted grouped FP8 GEMM and design ideas.
  • Posted on Discord by _L_ (2026-10-24) and recommended for 4-card owners with enough RAM or SSD. Source: @lia_oc, 2026-10-26
Imported from the cmpunlocker Discord with credit; see Credits.
Retrieved from "https://cmp.wiki/w/index.php?title=DeepSeek_V4.1_Flash_on_4x_8_GB_(Morrowmake)&oldid=253"