DeepSeek V4.1 Flash on 4x 8 GB (CurtisAccelerate TC-MoE)

DeepSeek-V4.1-Flash-1017
Reported 4 x 8 GB
Quant	Hybrid: LibertAI EXL3 2.9 bpw routed experts (layers 0-39), official FP8/FP4 for attention, shared experts, head, draft and Engram; KV fp8_ds_mla
Engine	Morrowmake/vllm-cmp170hx (bind-mounted sm80 package) plus the author's tensor-core MoE kernels vllm-cmp170hx 18305948ac; TC-MoE fb9786ec
Topology	TP4, peer access off (NCCL_P2P_DISABLE=1), host all-reduce
Context	262,144 (weights address 1,048,576)
Decode	144.2 tok/s
Prefill	1523 tok/s
VRAM per card	about 62.5 GB used of 64 GB, --gpu-memory-utilization 0.97
Power cap	140 W
Flags	`--quantization exl3 --tensor-parallel-size 4 --max-num-seqs 8 --max-num-batched-tokens 2048 --gpu-memory-utilization 0.97 --kv-cache-dtype fp8_ds_mla --engram-config cpu_offload --speculative-config dspark 5`
Tested	@CurtisAccelerate on 2026-10-22
Source	link
Confirmations	Did not reproduce by @anon33anon (8 GB), 2026-10-20 · details

A weight-selection recipe plus benchmark record for DeepSeek-V4.1-Flash-1017 on four 64 GB cards at 140 W on a dual-socket Broadwell host with no P2P. Its hybrid layout (EXL3 routed experts, everything else from the official checkpoint) is the one GNmac-lab and alanoo build on. It runs on Morrowmake's vLLM fork with the author's tensor-core MoE kernels (CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE). No weights, engine or patches live in the repo itself. Infobox numbers are the 2026-10-22 published configuration, one user on code. Source: repo github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX by CurtisAccelerate on GitHub, posted by @anon33anon, 2026-10-20
Source: @anon33anon, 2026-10-20

Prerequisites

Four CMP 170HX at 64 GB (sm_80) (Driver unlock, 8 GB card). PCIe Gen2 x16 on the author's box.
Host RAM: the serve uses about 289 GB, with 265 GB of shared memory for the Engram pin (Engram runs cpu_offload, never on the GPUs). A 256 GB host does not fit. The author's host has 378 GB.
The serving image and a built _dsv41_tc_moe_C.abi3.so (from vllm_exl3/tc_moe/build_standalone.py in the TC-MoE repo).

Source: repo github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX by CurtisAccelerate on GitHub, posted by @anon33anon, 2026-10-20

Steps

There is no one-command installer: build the TC-MoE extension and assemble the Morrowmake vllm-cmp170hx stack yourself.

Serving image sha256:55d2bd9a6664aea2540e3f8e373d784b88420d35c3cedefc9f03923cd687b6f5; Morrowmake/vllm-cmp170hx at 18305948acba3f4c4d912cdd56f2202d2ebd94e8 (envs.py, platforms/cuda.py and the vllm/sm80 package bind-mounted into the container). The GLM recipe's pin 819146023f rejects VLLM_USE_V2_MODEL_RUNNER=0 with DSpark, so this serve sets DSV41_DSPARK_V1=1. SM80 attention comes from PixelML/sm80vllm branch sm80.
Weights: deepseek-ai/DeepSeek-V4.1-Flash-1017 for attention, shared experts, head, draft (including its FP4 experts) and Engram (shards 47 and 48); LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw for routed experts, layers 0-39 only.
quantization_config.json: quant_method exl3, bits 4, codebook mul1, scope dsv41_main_routed_experts_only, mtp_experts source, mtp_experts_start_layer 40, mtp_expert_dtype fp4, non_routed_quantization deepseek_v4_fp8 (weight_block_size 32x32, scale_fmt ue8m0). Leave head_bits out on purpose; do not set non_routed_exl3; the compressor projection is BF16 and must be built with no quant config.
Environment (scripts/run-tp4-1017.sh): VLLM_PLUGINS=vllm_exl3 VLLM_USE_V2_MODEL_RUNNER=0 DSV41_DSPARK_V1=1 VLLM_EXL3_COOP=1 VLLM_HOST_ALLREDUCE=1 NCCL_P2P_DISABLE=1 NCCL_IGNORE_DISABLED_P2P=1 NCCL_CUMEM_ENABLE=0 VLLM_EXL3_TC_MOE_V2=1 VLLM_EXL3_TC_MOE_PREFILL=1 VLLM_EXL3_TC_PREFILL_ROWS=2048 VLLM_EXL3_MOE_TP_ALIGN=128 VLLM_EXL3_MOE_TP_ROTATE=1 TORCH_CUDA_ARCH_LIST=8.0 VLLM_EXL3_ARENA_CACHE=1. Container flags --cap-add SYS_NICE --ipc host --ulimit memlock=-1 --ulimit stack=67108864.
Serve flags:

--quantization exl3 --load-format safetensors --safetensors-load-strategy lazy \
--tensor-parallel-size 4 --max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 2048 \
--gpu-memory-utilization 0.97 --kv-cache-dtype fp8_ds_mla \
--engram-config '{"cpu_offload": true}' --enable-prefix-caching --async-scheduling \
--speculative-config '{"method": "dspark", "num_speculative_tokens": 5}' \
--tokenizer-mode deepseek_v41 --tool-call-parser deepseek_v41 --reasoning-parser deepseek_v41

CUDA graphs FULL_AND_PIECEWISE with capture sizes 1, 2, 3, 4, 6, 8, 12, 18, 24, 30, 36, 42, 48. The first start writes EXL3 arenas (about 1.1 GiB per layer file; skipped if less than 4 GiB would remain). Source: repo github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX by CurtisAccelerate on GitHub, posted by @anon33anon, 2026-10-20

Measured numbers

All on one dual-socket Broadwell host at 140 W, thinking off, temperature 0. Tok/s count / code / prose.

Date	1 user	8 users (aggregate)	Prefill	Notes
2026-10-17 baseline	180.4 / 137.1 / 96.6	621.1 / 376.1 / 282.5	1,523 at a 2,048 prompt; 1,330 at 131,072	about 34 ms per step; 5.97 accepted tokens per step on counting; KV pool 1,534,065 tokens; needles 9/9 at 30K to 241K
2026-10-22 published	187.2 / 144.2 / 101.2			EXL transpose +2.86%, six-row host all-reduce tuning +1.74%
2026-10-28 opt-in experiments	205.7 / 150.5 / 112.1		1,376 at 93,335 tokens	mixed K2/K3 experts, split attention, captured drafter; not bit-exact, quality screen mixed, all flags default off

Same-board comparison on 2026-10-24 against Morrowmake's DeepSeek recipe (Curtis's own measurements): one user 187.2 / 144.2 / 101.2 here against 122.8 / 90.3 / 62.4; 8-user aggregate 271.6 against 168.7; Morrowmake's stack prefills faster at every length (128K: 1,329.0 against 1,558.7 tok/s). Source: repo github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX by CurtisAccelerate on GitHub, posted by @anon33anon, 2026-10-20

Accuracy

LibertAI's full-trunk 2.9 bpw pack measured code perplexity 1.9094 against 1.8749 unquantized (+1.8%) and prose 6.4520 against 6.4277 (+0.4%). This serve quantizes only routed experts, so it should sit closer to unquantized (an expectation, not measured). Turning Engram off raises code perplexity 1.91 to 4.16 and prose 6.45 to 9.06. Source: repo github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX by CurtisAccelerate on GitHub, posted by @anon33anon, 2026-10-20

Known issues

anon33anon tried to replicate it on 2026-10-20 and found some pieces were not open source at the time. Source: @anon33anon, 2026-10-20
Miséricorde's reading: measured at 140 W with peer access off and no overclock, undervolt or offset settings, which he calls a severe compute limitation. Source: @wakeupmbrc1, 2026-10-20
Needs about 290 GB of host RAM. Post-2026-10-22 paths are experimental and not quality-verified. No LICENSE file (NOTICE.md gives attribution only; the kernel repo is Apache-2.0, the cooperative-decode patch MIT).