llm-inference-bench
More actions
Reported 8 GB and 10 GB
llm-inference-bench measures an LLM server's prefill and decode throughput over a grid of concurrency and context lengths, against vLLM, SGLang or any OpenAI-compatible API. It also ships lil-bench (a standard in-container benchmark), p2pmark (GPU peer-to-peer bandwidth and all-reduce) and a peer-read stall probe whose kernel targets sm_80. Nothing in it is CMP-specific, but the fabric checks matter for multi-card 170HX boxes running tensor parallel over PCIe without NVLink. Author: GitHub local-inference-lab, MIT. Repository: https://github.com/local-inference-lab/llm-inference-bench Source: repo github.com/local-inference-lab/llm-inference-bench by local-inference-lab on GitHub, posted by @mountains1605, 2026-10-12.
Source: @mountains1605, 2026-10-12
Install and run
Python 3.10 or newer.
pip install httpx rich psutil
python3 llm_decode_bench.py --port 8000 # default: localhost:5000
python3 llm_decode_bench.py --scheduler all # scheduler stress scenarios
python3 tools/read_stalls/lil_read_stalls.py --gpus 0,1,2,3
Defaults: concurrency 1 to 128, contexts 0 to 131,072 tokens, 30 seconds per decode cell. A loop guard marks a cell ERROR: loop when output repeats. The read-stall probe needs only Python 3.8 and the NVIDIA driver; it prints READ STALLS: YES or NONE. Source: repo github.com/local-inference-lab/llm-inference-bench by local-inference-lab on GitHub, posted by @mountains1605, 2026-10-12.
What is useful on CMP boxes
- p2pmark: peer-access matrix, peer copy bandwidth, latency and NCCL against a custom PCIe all-reduce. Do not run it beside a loaded model; it allocates memory on every visible GPU. Source: repo github.com/local-inference-lab/llm-inference-bench by local-inference-lab on GitHub, posted by @mountains1605, 2026-10-12.
- lil-bench reports PCIe replays, correctable errors and links running below their maximum generation or width under load. Uploading results needs a token;
--no-uploadkeeps runs local. Source: repo github.com/local-inference-lab/llm-inference-bench by local-inference-lab on GitHub, posted by @mountains1605, 2026-10-12. - Accuracy A/B: pinned GSM8K, MMLU-Pro and GPQA sets with a paired comparison, built for comparing quantizations.
--tool-evalruns SeraphimSerapis/tool-eval-bench. Source: repo github.com/local-inference-lab/llm-inference-bench by local-inference-lab on GitHub, posted by @mountains1605, 2026-10-12.
Results on CMP cards
Contested @nestaa51 ran llm-inference-bench and tool-eval-bench on 2x CMP: GLM-5.3-Flash scored 90/100 and Qwen3.8 Flash Next 85/100 on tool calling. Source: @nestaa51, 2026-10-14 @vektor_prime objected that the reasoning-effort split was noise and that time-to-first-token is not comparable across parameter counts; @wakeupmbrc1 noted GLM ran in an EXL3 quant, which is why Qwen was faster. Source: @vektor_prime, 2026-10-14
Source: @wakeupmbrc1, 2026-10-14 Model results belong on the recipe pages.
Caveats
- Not CMP-specific; the repository never mentions the CMP 170HX. The embedded p2pmark fallback binary needs CUDA 13 runtime libraries. Source: repo github.com/local-inference-lab/llm-inference-bench by local-inference-lab on GitHub, posted by @mountains1605, 2026-10-12.
- Very active; pin the version when you quote numbers.