ImportBot: Import: llm-inference-bench, @mountains1605, @nestaa51, @vektor_prime, @wakeupmbrc1; repos local-inference-lab/llm-inference-bench

2026-10-30T08:38:27Z

Import: llm-inference-bench, @mountains1605, @nestaa51, @vektor_prime, @wakeupmbrc1; repos local-inference-lab/llm-inference-bench

← Older revision		Revision as of 08:38, 30 October 2026
Line 1:		Line 1:
			{{DISPLAYTITLE:llm-inference-bench}}
	{{Status\|Reported}} {{Cards\|both}}		{{Status\|Reported}} {{Cards\|both}}

ImportBot: Import: llm-inference-bench, @mountains1605, @nestaa51, @vektor_prime, @wakeupmbrc1; repos local-inference-lab/llm-inference-bench

2026-10-30T08:04:41Z

Import: llm-inference-bench, @mountains1605, @nestaa51, @vektor_prime, @wakeupmbrc1; repos local-inference-lab/llm-inference-bench

New page

{{Status|Reported}} {{Cards|both}}

'''llm-inference-bench''' measures an LLM server's prefill and decode throughput over a grid of concurrency and context lengths, against vLLM, SGLang or any OpenAI-compatible API. It also ships '''lil-bench''' (a standard in-container benchmark), '''p2pmark''' (GPU peer-to-peer bandwidth and all-reduce) and a peer-read stall probe whose kernel targets sm_80. Nothing in it is CMP-specific, but the fabric checks matter for multi-card 170HX boxes running tensor parallel over PCIe without NVLink. Author: GitHub local-inference-lab, MIT. Repository: https://github.com/local-inference-lab/llm-inference-bench {{Source|handle=mountains1605|date=2026-10-12|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1559350041645420624|repo=https://github.com/local-inference-lab/llm-inference-bench}}.<br />{{Source|handle=mountains1605|date=2026-10-12|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1559350041645420624}}

== Install and run ==
Python 3.10 or newer.
<syntaxhighlight lang="bash">
pip install httpx rich psutil
python3 llm_decode_bench.py --port 8000 # default: localhost:5000
python3 llm_decode_bench.py --scheduler all # scheduler stress scenarios
python3 tools/read_stalls/lil_read_stalls.py --gpus 0,1,2,3
</syntaxhighlight>
Defaults: concurrency 1 to 128, contexts 0 to 131,072 tokens, 30 seconds per decode cell. A loop guard marks a cell <code><nowiki>ERROR: loop</nowiki></code> when output repeats. The read-stall probe needs only Python 3.8 and the NVIDIA driver; it prints <code><nowiki>READ STALLS: YES</nowiki></code> or <code><nowiki>NONE</nowiki></code>. {{Source|handle=mountains1605|date=2026-10-12|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1559350041645420624|repo=https://github.com/local-inference-lab/llm-inference-bench}}.

== What is useful on CMP boxes ==
* '''p2pmark''': peer-access matrix, peer copy bandwidth, latency and NCCL against a custom PCIe all-reduce. Do not run it beside a loaded model; it allocates memory on every visible GPU. {{Source|handle=mountains1605|date=2026-10-12|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1559350041645420624|repo=https://github.com/local-inference-lab/llm-inference-bench}}.
* '''lil-bench''' reports PCIe replays, correctable errors and links running below their maximum generation or width under load. Uploading results needs a token; <code><nowiki>--no-upload</nowiki></code> keeps runs local. {{Source|handle=mountains1605|date=2026-10-12|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1559350041645420624|repo=https://github.com/local-inference-lab/llm-inference-bench}}.
* '''Accuracy A/B''': pinned GSM8K, MMLU-Pro and GPQA sets with a paired comparison, built for comparing quantizations. <code><nowiki>--tool-eval</nowiki></code> runs SeraphimSerapis/tool-eval-bench. {{Source|handle=mountains1605|date=2026-10-12|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1559350041645420624|repo=https://github.com/local-inference-lab/llm-inference-bench}}.

== Results on CMP cards ==
{{Status|Contested}} @nestaa51 ran llm-inference-bench and tool-eval-bench on 2x CMP: GLM-5.3-Flash scored 90/100 and Qwen3.8 Flash Next 85/100 on tool calling. {{Source|handle=nestaa51|date=2026-10-14|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1559768495418253340}} @vektor_prime objected that the reasoning-effort split was noise and that time-to-first-token is not comparable across parameter counts; @wakeupmbrc1 noted GLM ran in an EXL3 quant, which is why Qwen was faster. {{Source|handle=vektor_prime|date=2026-10-14|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1559943900095975508}}<br />{{Source|handle=wakeupmbrc1|date=2026-10-14|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1559992416097013801}} Model results belong on the recipe pages.

== Caveats ==
* Not CMP-specific; the repository never mentions the CMP 170HX. The embedded p2pmark fallback binary needs CUDA 13 runtime libraries. {{Source|handle=mountains1605|date=2026-10-12|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1559350041645420624|repo=https://github.com/local-inference-lab/llm-inference-bench}}.
* Very active; pin the version when you quote numbers.

== See also ==
* [[Tools]]
* [[Monitoring]]

{{ImportFooter}}
[[Category:Tools]]
[[Category:8 GB]]
[[Category:10 GB]]

Llm-inference-bench - Revision history

ImportBot: Import: llm-inference-bench, @mountains1605, @nestaa51, @vektor_prime, @wakeupmbrc1; repos local-inference-lab/llm-inference-bench

ImportBot: Import: llm-inference-bench, @mountains1605, @nestaa51, @vektor_prime, @wakeupmbrc1; repos local-inference-lab/llm-inference-bench