Drilldown: Recipe
More actions
Use the filters below to narrow your results.
page:
model:
quant:
LibertAIDAI/GLM-5.3-Flash-NVFP4 (ModelOpt NVFP4, routed experts only, 181 GiB), run through Marlin W4A16 kernels; GLM-5.3-Flash DFlash2 drafter (5 sliding-window layers) (1) ·
Native MXFP4 experts + FP8 attention (official deepseek-ai checkpoint, about 155.4 GiB); KV cache fp8 (1) ·
Official block FP8, about 126 GiB including the about 50 GiB FP8 n-gram (PLE) table, which is served from host RAM; KV cache bf16 (the QSA layers refuse fp8) (1) ·
Official checkpoint as downloaded: MXFP4 routed experts as Marlin W4A16, MXFP8 dense as Marlin W8A16; KV fp8 (1)
engine:
vLLM PR #56201 head (79a7108d) plus a 20-patch sm_80 series; image zanooda/vllm-sm80-ds41f:v41-sm80 (1) ·
vLLM: haosdent/vllm branch dsv4-flash-a100 plus allover326's 7 patches (DSpark under PP, sm_80 prefill top-k fallback, logits row chunking) (1) ·
vLLM: official vllm/vllm-openai:glm53-flash image plus promisezackr's 24 patches (SM8x base from 344303947/vllm170hx, pipeline parallel for GLM-5.3-Flash, DFlash2 under PP8) (1) ·
vLLM: the vllm/vllm-openai:qwen38-flash-next image with zebgop-ops patches 0001 to 0016 (never 0008-WITHDRAWN) and three ported files (1)
engine version:
cards:
topology:
PP4 (VLLM_PP_LAYER_PARTITION=12,12,12,7) with DSpark, 5 draft tokens; PCIe Gen2 x4, no P2P (1) ·
PP4, VLLM_PP_LAYER_PARTITION=12,12,12,12, MTP 3, FULL_AND_PIECEWISE CUDA graphs; PCIe Gen2 x4 per card, no P2P. Three cards: PP3 with 16,17,15 or 16,16,16 (1) ·
PP8 (VLLM_PP_LAYER_PARTITION=7,5,7,5,7,5,7,2) with DFlash2, 7 draft tokens; PCIe Gen2 x4, no P2P; host 2x Xeon E5-2690 v4, 96 GB DDR4 (1) ·
PP8, partition 5,5,5,5,5,5,5,5 (1)
context:
prefill tps:
vram per card:
64 GB, --gpu-memory-utilization 0.85 (1) ·
64 GB, --gpu-memory-utilization 0.90 (1) ·
64 GB; weights 35.9 GB per rank (ranks 1-6), 37.6 GB rank 0, 46.3 GB rank 7 (1) ·
64 GiB; --gpu-memory-utilization 0.94 plus per-rank KV budgets; measured 34.0-34.3 GiB per 12-layer rank, 37.9 GiB on the drafter rank (1)
power cap:
driver: (Click arrow to add another value)
unlock build: (Click arrow to add another value)
status:
tested by:
tested on:
Showing below up to 4 results in range #1 to #4.