DeepSeek V4.1 Flash on 4x 8 GB plus RTX PRO 6000 (ExLlamaV3 TP5)
More actions
| Reported 4 x 8 GB | |
| Quant | EXL3 3.5 bpw (LibertAIDAI; routed experts 4 bpw on layers 0-9 and 30-39, 3 bpw on 10-29; attention and shared experts 6 bpw) plus the official Engram shards |
|---|---|
| Engine | ExLlamaV3 fork (anoane/exllamav3-anemone) + TabbyAPI fork (anoane/tabbyAPI) tag recipe-09 (309ef6d1); TabbyAPI deepseek-v41 (ce250ff) |
| Topology | TP5 across four CMP 170HX and one RTX PRO 6000 Blackwell (native backend, reduces through host memory); MTP draft on the PRO |
| Context | 1,048,576 FP16 |
| Decode | 99.9 tok/s |
| Prefill | 2116 tok/s |
| VRAM per card | gpu_split 58 GB per CMP, 70 GB on the PRO |
| Power cap | 150 W per CMP (PRO 400 W) |
| Driver | NVIDIA open 610.57.04 patched with Morrowmake/cmpunlocker 1f2c498 (profile 8gb) |
| Unlock build | Morrowmake/cmpunlocker 1f2c498; modprobe RmForceEnableGen2=1, RMPcieLinkSpeed=0x1, ForceP2P=0x11 |
| Flags | CUDA_VISIBLE_DEVICES=0,1,3,4,2, gpu_split [58,58,58,58,70], tensor_parallel true (native), draft_mode mtp, max_batch_size 1, EXL3_DSA_INDEXER_DTYPE=sm_80:int8,sm_120:int8, EXL3_INT8_GEMV=0, EXLLAMA_NO_P2P_COPY=1 |
| Tested | @anoane on 2026-10-20 |
| Source | link |
| Confirmations | None yet. |
DeepSeek-V4.1-Flash-1017 at the 3.5 bpw EXL3 quant with a full 1,048,576-token FP16 context, served by tensor parallel
over four unlocked CMP 170HX plus one RTX PRO 6000 Blackwell, through ExLlamaV3 and TabbyAPI forks. The PRO card adds
the memory that lets 3.5 bpw fit (it does not fit on four 64 GB cards) and holds the MTP draft. Infobox decode is the
mean over nine C-coding requests; prefill is the 26K-token run.
Source: repo github.com/anoane/09-deepseek-v41-flash-1017-exl3-tp5-1m by anoane on GitHub, posted by @anon33anon, 2026-10-20
Source: @anon33anon, 2026-10-20
Prerequisites
- Four CMP 170HX at 64 GB on PCIe Gen2 x16 and one RTX PRO 6000 Blackwell Workstation 96 GB. Serving does not use P2P (
EXLLAMA_NO_P2P_COPY=1, native TP backend); only the Gen2 x16 link matters. See Driver unlock and Unlocks. - Author's host: AMD EPYC 9J14 (96 cores, socket capped at 260 W), 499 GiB RAM, one NVMe; Ubuntu 26.04, kernel 7.0.0-38-generic, CUDA 13.4, torch 2.13.0+cu132, Python 3.12.15.
- Driver: NVIDIA open-gpu-kernel-modules 610.57.04 patched with Morrowmake/cmpunlocker commit 1f2c498 (profile
8gb). The upstream equivalent issudo ./install.sh --p2p --profile=8gb --no-iommu --no-passthrough. The patchednvidia.kohas the same srcversion as stock, so identify it by sha256. Kernel and driver packages are held. - modprobe:
options nvidia NVreg_RegistryDwords="RmForceEnableGen2=1;RMPcieLinkSpeed=0x1;ForceP2P=0x11".
Source: repo github.com/anoane/09-deepseek-v41-flash-1017-exl3-tp5-1m by anoane on GitHub, posted by @anon33anon, 2026-10-20
Steps
The repo holds layouts, systemd units, power-cap scripts, bench tools and raw results:
- Weights: LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-3.5bpw at revision
e55cd9e0(vision tower and draft head included) plus the two Engram shards (model-00047/00048, 2 x 94.6 GiB) from deepseek-ai/DeepSeek-V4.1-Flash-1017 at1bae0e10, linked into one folder bytools/make_view.py. A prewarm unit keeps the Engram tables in the page cache, reading withPOSIX_FADV_RANDOMbecause a sequential read got reclaimed by kswapd. - Engine: anoane/exllamav3-anemone branch
dsv41f-1017(tagrecipe-09: upstream dev f157fc0b plus open PRs #449 DeepSeek-V4.1 with TP and draft head, #468 and #472 placement, #532 fd-limit fix, plusEXL3_DSA_INDEXER_DTYPE), native build withTORCH_CUDA_ARCH_LIST="8.0;12.0a". - Server: anoane/tabbyAPI branch
deepseek-v41(upstream main e086a65 plus the V4.1 tool-call format andTABBY_EXL3_GENERATOR_CHUNK); chat template from HF discussion PR 11 on the 1017 repo. - Layout
tp5-1017-35:CUDA_VISIBLE_DEVICES=0,1,3,4,2(four HX first, PRO last),gpu_split [58, 58, 58, 58, 70],tensor_parallel truewith backend native,draft_mode mtpwithdraft_gpu_split [0,0,0,0,20]anddynamic_draft true,max_seq_len = cache_size = 1048576FP16,max_batch_size 1,TABBY_EXL3_GENERATOR_CHUNK=16384,reasoning_effort high.ds41-exl3.servicerefuses to start unless the PCI order is HX, HX, PRO, HX, HX.gpu-power-capsselects cards by device id and fails closed on an unexpected card count.
Source: repo github.com/anoane/09-deepseek-v41-flash-1017-exl3-tp5-1m by anoane on GitHub, posted by @anon33anon, 2026-10-20
Measured numbers
Cold C-source prompts, single runs:
| Prompt tokens | TTFT | Prefill tok/s | Decode tok/s |
|---|---|---|---|
| 26,296 | 12.4 s | 2,116 | 84.2 |
| 127,604 | 58.7 s | 2,173 | 89.2 |
| 253,055 | 124.2 s | 2,037 | 67.9 |
| 513,791 | 274.1 s | 1,874 | 100.3 |
| 1,039,782 | 632.1 s | 1,645 | 72.3 |
Nine C-coding requests (139 to 9,426 tokens): mean decode 99.9 tok/s (96.9 to 103.6 across server starts), prefill 1,500 to 2,085 tok/s, draft acceptance 63 to 85%. Two 1M slots load (cache_size 2,097,152), four do not; two streams 34.4 tok/s each. Source: repo github.com/anoane/09-deepseek-v41-flash-1017-exl3-tp5-1m by anoane on GitHub, posted by @anon33anon, 2026-10-20
INT8 lightning indexer and layout comparisons
- INT8 indexer: a 1,016,849-token prefill in 603 s against 662 s with FP16 (-9%, -12% on the last 512K); KL against FP16 0.0034 / 0.0029 / 0.0032 / 0.0004 at 16K / 131K / 524K / 1M; top-1 98.0% / 99.9%; 3 of 3 planted keys recalled. Indexer kernel per 256-row slab against 1M keys: CMP 170HX 22.3 ms FP16, 12.3 ms INT8 (no FP8 or FP4 on sm_80); under TP every card scores every key, so the 170HX sets the time.
- Layouts (34K prose prompts, FP16 indexer): TP5 at 3.5 bpw 1,944 / 2,159 tok/s prefill, 85.8 / 62.8 / 73.1 decode; TP4 on the four 170HX alone only fits the 2.9 bpw pack (72.7 / 60.1 / 64.6); TP4 with the NCCL peer backend was slower than native (65.0 / 52.9 / 57.2); layer split 50.7 / 42.7 / 44.0.
EXL3_INT8_GEMV=0against the default 2: 101.2 against 98.7 tok/s (+2.5%, within noise).
Source: repo github.com/anoane/09-deepseek-v41-flash-1017-exl3-tp5-1m by anoane on GitHub, posted by @anon33anon, 2026-10-20
Known issues
- Not yet rebooted with the TP5 layout at the time of writing. The first long request after a load decodes at about 47 tok/s once.
- Shutdown logs "Timed out waiting for worker" (fixed upstream in exllamav3 #534, not in this engine); every TP restart dumps tens of thousands of harmless journal lines.
- vLLM was not tried because it needs an sm_80 attention path for V4.1 and an EXL3 expert split that were not public then.
- Single machine, hard-coded paths, one or two runs per figure, dependent on unmerged exllamav3 PRs and two personal forks. The bench prompt corpora are not published.
- Contested The modprobe line sets
RMPcieLinkSpeed=0x1besideRmForceEnableGen2=1, while the cmp100-unlock project callsRMPcieLinkSpeeda driver-wide Gen1 cap (on the proprietary 550 driver for GV100). The RTX PRO 6000's resulting link generation is not stated. See Unlocks.
Source: repo github.com/anoane/09-deepseek-v41-flash-1017-exl3-tp5-1m by anoane on GitHub, posted by @anon33anon, 2026-10-20
Related
- EXL3 on the CMP: Quantization on the 170HX. Engines: Engines (ExLlamaV3 and TabbyAPI).
- Four CMPs alone: DeepSeek V4.1 Flash on 4x 8 GB (Morrowmake). Benchmarks.
Sources
- Repo: anoane/09-deepseek-v41-flash-1017-exl3-tp5-1m (MIT; TabbyAPI AGPL-3.0 used as a separate program), last commit 2026-10-20. Posted on Discord by anon33anon.