Laguna-S-2.1 on 2x 10 GB
From CMP Wiki
More actions
| Reported 2 x 10 GB | |
| Quant | NVFP4 GGUF (runs through a fallback path: the cards have no native FP4) |
|---|---|
| Engine | custom llama-server backend from ciruinference (the service is named vllm, but it is not vLLM) |
| Topology | tensor split 1,1 across the two cards |
| Context | 131,072 |
| Decode | 53 tok/s |
| Speculative decoding | none |
| Measured as | one post: 53 tok/s decode, number of users not stated (the server has 30 slots) |
| Flags | --ctx-size 131072 --parallel 30 --batch-size 512 --ubatch-size 256 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --tensor-split 1,1 --spec-type none --no-context-shift |
| Tested | @squishysquare on 2026-08-12 |
| Source | Discord message |
| Confirmations | None yet. |
Laguna-S-2.1 in NVFP4 GGUF on two 10 GB cards, split evenly, at 131,072 tokens of context. bigfoot asked for working
setups on 10 GB cards; squishysquare posted his. The service is called vllm but runs a custom llama-server backend from
ciruinference; NVFP4 works through a fallback path because the cards lack native FP4. He measured 53 tok/s decode and
did not measure prefill ("pretty snappy").
Source: @bigfoot1144, 2026-08-11
Source: @squishysquare, 2026-08-11
Source: @squishysquare, 2026-08-12
Source: @squishysquare, 2026-08-12
Source: @squishysquare, 2026-08-12
Source: @squishysquare, 2026-08-12
Notes
- The full launch flags also set
--threads 4 --threads-batch 8 --fit on --fit-target 1024 --fit-ctx 131072 --load-mode mlock --jinja --cache-ram 0 --no-cache-idle-slots --metrics --slots --no-webui. Source: @squishysquare, 2026-08-12 - Cooling: two cards under a single 80 mm fan without active temperature control; he expects 70 to 72 C at 250 W. Source: @squishysquare, 2026-08-12
- Not stated: the backend's source and version, the memory profile per card, the PCIe link, the driver, the power cap and how many requests ran at once for the 53 tok/s.
Sources
- squishysquare's posts in general, 2026-08-11 and 08-12.
- Related: Recipes#10 GB card, llama.cpp on CMP cards, Quantization on the 170HX, 10 GB card.