DeepSeek V4 Flash pruned on 4x 10 GB
From CMP Wiki
More actions
| Reported 4 x 10 GB | |
| Engine | vLLM (version and flags not stated) |
|---|---|
| Topology | not tensor parallel (the split is not stated) |
| Links | PCIe Gen2 x4 |
| Decode | 48.2 tok/s |
| Speculative decoding | none |
| Measured as | one post (number of users not stated) |
| VRAM per card | about 37 GB usable per card (40 GB profile), 148 GB in total |
| Power cap | 250 W limit (read on 2026-08-11, during his llama.cpp runs) |
| Tested | @cuddylac997 on 2026-08-12 |
| Source | Discord message |
| Confirmations | None yet. |
DeepSeek V4 Flash on four 10 GB cards with about 37 GB usable each. The full model would not load in vLLM on 37 GB
cards, so cuddylac997 pruned 25% of its experts. He got 48.2 tok/s without DSpark and said it still gave good
responses; two days later he reported about 50 tok/s. The pruned checkpoint was not published.
Source: @cuddylac997, 2026-08-12
Source: @cuddylac997, 2026-08-12
Source: @cuddylac997, 2026-08-14
Source: @cuddylac997, 2026-08-14
Notes
- Host: Supermicro SYS-1028GQ-TRT (X10DGQ board), two Xeon E5-2650 v3, 256 GB DDR4-2133 LRDIMM, four 10 GB cards at PCIe Gen2 x4. Each card has about 37 GB usable; the model does not use tensor parallel. Source: @cuddylac997, 2026-08-11
Source: @cuddylac997, 2026-08-12
Source: @cuddylac997, 2026-08-12
Source: @cuddylac997, 2026-08-12 - Quality check: a 16-prompt pass/fail script gave 11 pass, 1 fail (long-tail recall) and 4 to review. Its "42.4 tok/s" is an average over the 16 prompts run one after another, not a multi-user total; the two long-context prompts (8k and 60k tokens) returned 7 tokens at 0.9 and 2.2 tok/s. Source: @cuddylac997, 2026-08-12
- Before pruning (2026-08-11): Unsloth's UD-Q3_K_XL GGUF of DeepSeek-V4-Flash-0731 in llama.cpp with a DSpark drafter on the same cards ran at about 28.4 tok/s at 32K context; locking the clocks at 1410 MHz took it to 33.5 tok/s and tripled prefill. A vLLM INT4 repack (150.4 GiB) did not fit four 37.7 GiB cards with room for KV. Source: @cuddylac997, 2026-08-11
Source: @cuddylac997, 2026-08-11
Source: @cuddylac997, 2026-08-11 - Zeferrum suggested antirez's quant, which shrinks the weights instead of removing experts; Impactic said DeepSeek V4 runs much faster on vLLM and that the speed could be doubled with more setup work. See DeepSeek V4 Flash 0731 on 4x 8 GB (allover326 PP4 DSpark) for a 4-card recipe on 64 GB cards. Source: @zefrench6961, 2026-08-12
Source: @impactic, 2026-08-14
Source: @impactic, 2026-08-14 - Not stated: the pruning tool, the format and quant of the pruned checkpoint, the vLLM version and flags, the context length, whether the clocks were locked for the vLLM run, and the number of concurrent requests behind 48.2 tok/s.
Sources
- cuddylac997's posts in the testing and development channels, 2026-08-11 to 08-14.
- Related: Recipes#10 GB card, 10 GB card, Memory unlock on the 10 GB card, Benchmarks.