Jump to content
Toggle search
Toggle menu
CMP Wiki
Toggle preferences menu
Toggle personal menu
Not logged in
Your IP address will be publicly visible if you make any edits.

Mistral Large 4 on 8x 8 GB (llama.cpp lab trial)

From CMP Wiki
More actions
Mistral-Large-4-1T-A52B ("Le Chonk")
Reported 8 x 8 GB
Quantunsloth UD-IQ3_XXS GGUF (394 GiB in 10 files); DSpark drafter Q8_0 (2.1 GB)
Enginellama.cpp llama-server, built in a CUDA 12.9 container llama.cpp master a91eab6 (2026-10-31), model support from PR #31376
TopologyLayer split over all eight cards (tensor split is not available for this architecture)
LinksP2P on (whether it is needed is not stated)
Context196,608 (one unified KV pool, 8 slots)
Decode, 1 user18.7 tok/s
Decode, many users34.1 tok/s total, 8 users
Prefill131 tok/s at 17k tokens
Speculative decodingDSpark (Q8_0 drafter), 3 draft tokens per step
Measured aslab bench, short prompts, 1,024-token replies, reasoning effort low, code row
VRAM per card54 to 63 GiB of 64 used
Power cap250 to 300 W (governor), +160 MHz offset
Driver615.71.09 with cmpunlocker
Unlock buildcmpunlocker (64 GiB, 74 SMs)
Flags-ngl 999 -sm layer -c 196608 -np 8 -kvu --spec-type draft-dspark -md (DSpark Q8_0) --reasoning-format deepseek; built with GGML_CUDA_FORCE_CUBLAS
Tested@darkmaniac7 on 2026-11-01
Sourcelink
ConfirmationsNone yet.

Mistral Large 4 ("Le Chonk", 1T total, about 52B active) runs on eight 64 GB cards with llama.cpp, but only as a trial: it works, and it is far too slow for chat at long contexts. The model is 72 layers with DeepSeek-style compressed attention (MLA, KV rank 512) plus a DSA sparse indexer, cross-layer KV sharing, and 128 routed experts with 4 active and 1 shared. llama.cpp supports it since PR #31376, together with its DSpark drafter, but runs it with a layer split, so each token passes through the eight cards one after another. A real recipe (tensor-parallel vLLM with an Ampere-servable quant) does not exist yet; see #What would make it usable. Reported Source: @darkmaniac7, 2026-11-01 (lab notes)

Prerequisites

  • Eight unlocked 64 GB cards. UD-IQ3_XXS is 394 GiB of weights; with 196,608 tokens of KV the cards held 54 to 63 GiB each, the first card the most.
  • About 400 GiB of disk for the GGUF, the drafter and the vision projector (3.3 GB, optional).
  • Host RAM at least the size of the model, or load with --no-mmap: with 377 GB of RAM and the GGUF on a SATA SSD, the memory-mapped load thrashed the page cache and took 57 minutes. Copying the GGUF to an NVMe first is the other fix.
  • Docker with the NVIDIA runtime (the build container keeps the host toolchain untouched).

Steps

Download the ten UD-IQ3_XXS files and the drafter from unsloth/Mistral-Large-4-1T-A52B-GGUF and check each against its sha256 on Hugging Face. Then build llama.cpp for sm_80 in a CUDA devel container: 

git clone --depth 50 https://github.com/ggml-org/llama.cpp && cd llama.cpp
docker run --rm -v "$PWD":/src -w /src nvidia/cuda:12.9.1-devel-ubuntu24.04 bash -c '
  apt-get update -qq && apt-get install -y -qq cmake ninja-build git libssl-dev
  cmake -S . -B build-cuda -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=80 \
    -DGGML_CUDA_FORCE_CUBLAS=ON -DLLAMA_CURL=OFF -DGGML_NATIVE=ON \
    -DCMAKE_EXE_LINKER_FLAGS=-Wl,--allow-shlib-undefined
  cmake --build build-cuda -j 24 --target llama-server llama-cli llama-bench'

The linker flag is needed because the build container has no GPU driver: the executables link against libcuda.so.1, which the NVIDIA runtime provides when they run. Serve: 

docker run -d --name le-chonk --runtime nvidia --gpus all --ipc host --network host --ulimit memlock=-1 \
  -e CUDA_DEVICE_ORDER=PCI_BUS_ID -e LD_LIBRARY_PATH=/src/build-cuda/bin \
  -v "$PWD":/src:ro -v /path/to/gguf:/models:ro nvidia/cuda:12.9.1-devel-ubuntu24.04 \
  /src/build-cuda/bin/llama-server -m /models/Mistral-Large-4-1T-A52B-UD-IQ3_XXS-00001-of-00010.gguf \
  -md /models/dspark-Mistral-Large-4-1T-A52B-Q8_0.gguf --spec-type draft-dspark -ngld 999 \
  -ngl 999 -sm layer -c 196608 -np 8 -kvu -fa auto --jinja --reasoning-format deepseek \
  --metrics --alias mistral-large-4 --host 0.0.0.0 --port 8090 -t 32

Source: @darkmaniac7, 2026-11-01 (lab notes)

Measured numbers

The lab's chassis, 2026-11-01: eight 8 GB cards on the 300 W VBIOS with a +160 MHz offset, P2P on, the build above. Short prompts, 1,024 tokens per reply, reasoning effort low, tok/s decode per stream / total:

1 user 4 users 8 users
structured 18.0 / 17.7 5.7 / 21.9 4.1 / 30.5
code 18.7 / 16.9 5.9 / 22.8 4.5 / 34.1
prose 12.6 / 11.8 3.8 / 14.9 2.8 / 22.0
  • Prefill 131 tok/s on a 17k-token prompt (130 s to the first token).
  • At 73k to 77k tokens of context: 747 s to the first token (98 tok/s prefill), then 5.9 tok/s decode with one request; two requests together decoded 2.6 + 2.7 tok/s, no gain, and a cold prefill nearly stopped the other request's decode. A full roleplay-length turn (about 1,200 words with thinking) works out at 25 to 35 minutes.
  • The DSpark drafter accepted 52 to 57 percent of its drafts (three per step), so drafting works; the base speed is the limit.
  • For scale, the same short-prompt command on the same chassis: DeepSeek V4.1 Flash on 4x 8 GB (Morrowmake) does 147 / 219 / 144 tok/s for one user and GLM 5.3 Flash on four cards 266 / 284 / 157.

Reported Source: @darkmaniac7, 2026-11-01 (lab notes)

Known issues

  • The chat template accepts only reasoning_effort low, medium or high (high when absent) and raises an error for anything else, so clients that send "max" fail; send "high".
  • llama.cpp's tensor split (-sm tensor) refuses this architecture: its llm_arch_supports_sm_tensor excludes mistral-large4 and the other MLA and DSA families (deepseek2, deepseek32, glm_dsa). Layer split is the only multi-card mode.
  • Loading memory-mapped from a slow disk with less RAM than the model takes about an hour (see Prerequisites).
  • Vision needs the separate mmproj file; this run was text only.

Source: @darkmaniac7, 2026-11-01 (lab notes)

  • On Discord, segmond set out to run Mistral Large 4 (1T) on twelve cards (asked how many, he answered "0xC", 12 in hexadecimal) and had GLM 5.3 set it up. After 5 hours it gave up: "I can't get this running, I need additional upstream support". Engine, quant and card variant were not stated. Five hours after that post, on 2026-11-01, he wrote: "looks like glm got Mistral working, 1.5tk/sec tho" ("Le chunk for real"), wishing he could "throw opus at it". So it runs on his rig at 1.5 tok/s, a first-light number and not a recipe: engine, quantization, card variant, card count (twelve were planned) and context are all unstated. Compare the lab's eight-card llama.cpp numbers above. Source: @segmond, 2026-10-31
    Source: @segmond, 2026-10-31
    Source: @1a2b3c0967, 2026-11-01
    Source: @segmond, 2026-11-01
    Source: @segmond, 2026-11-01
    Source: @segmond, 2026-11-01

What would make it usable

  • vLLM has supported Mistral Large 4 since its release (TP4 and TP8, expert parallel, the built-in DSpark drafter at about 2.9 accepted tokens per step for roughly 2x output on TP8). What is missing for these cards is a checkpoint vLLM can serve on Ampere at about 3 bits: the official weights are FP8 (about 1 TB, too big for 512 GiB) and NVFP4 (Blackwell only). An EXL3 or W4A16 build of the experts, as LibertAI's export does for DeepSeek V4.1 Flash, would allow a Morrowmake-style recipe.
  • A REAP expert-pruned 667B build exists in NVFP4 only; an Ampere quant of it would fit far more easily.
  • A DFlash2-class drafter trained for this model.
  • llama.cpp tensor split for the MLA / DSA attention family.

Source: repo github.com/vllm-project/vllm by vllm-project on GitHub Source: @darkmaniac7, 2026-11-01 (lab notes)

Related

Imported from the cmpunlocker Discord with credit; see Credits.
Retrieved from "https://cmp.wiki/w/index.php?title=Mistral_Large_4_on_8x_8_GB_(llama.cpp_lab_trial)&oldid=920"