Benchmark protocol

One way to measure a CMP 170HX inference setup, so that numbers from different people can be compared. The script cmpbench.py runs it against any OpenAI-compatible server (vLLM, SGLang, llama.cpp server, TabbyAPI) and prints the results as lines for a Recipe infobox and a Confirmation. Protocol 1.0, script 1.0.

Cmpbench.py (59.5 KB, by @ImportBot, details)
sha256 52b88304593b3af57dd516f2e458832a21f4f801ba3adee58d2e33724e686adc

Get it with curl -L -o cmpbench.py https://cmp.wiki/wiki/Special:Redirect/file/Cmpbench.py and check its sha256 against the one shown above.

Quick start

python3 cmpbench.py --base-url http://localhost:8000/v1 --model MODEL --spec-decode "MTP 3"

Run it on the machine that holds the cards, with nothing else using them. A full run takes a few minutes and ends with a table, two blocks of wikitext (Recipe fields and Confirmation fields) and a JSON file.

What the protocol measures

Three stages, in this order, each after a short warmup:

Decode with one user. Three fixed prompts of 140 to 180 tokens, printed in full in the script: code (a Python module with tests), structured (a JSON inventory) and prose (an essay). One request at a time, temperature 0, thinking off, answers of up to 1,024 tokens, streamed. Decode tok/s = completion tokens / (time of the last token minus time of the first token). Time to first token (TTFT) is reported apart, so prompt reading does not dilute decode speed. The code number goes into decode_single; structured and prose go into bench_note.
Prefill. One prompt of about 8,000 tokens, built by the script from fixed text, with 1 output token. It starts with a random reference line, so no prefix cache answers for the engine: this is cold prefill. Prefill tok/s = prompt tokens / TTFT, timed by the client, not read from engine logs.
Decode with many users. 8 requests of the prose prompt at the same moment, answers of up to 512 tokens. Total tok/s = all completion tokens / (last token of any request minus first token of any request). The table also shows the mean and the lowest per-user tok/s.

Token counts come from the server's usage fields. If the server sends none, the script takes llama.cpp's timings, and failing that it estimates 4 characters per token and says so in the table, in bench_note and in the JSON. Estimated counts are not protocol numbers.

Settings

Temperature 0. The script asks the chat template for no thinking (chat_template_kwargs: {"enable_thinking": false}); if the server refuses the field, the run uses the server default and the output says so. Reasoning text that still comes out counts as decode tokens.
Run the server the way you use it every day and give its launch command in the recipe's Notes.
Nothing else on the cards during the run. If the first run happened while the engine was still compiling or capturing graphs, run again and report the second run. Repeated runs on vLLM have differed by about 3%. Source: @vektor_prime, 2026-10-07
An answer that stops before 1,024 tokens is reported as measured and listed in bench_note as a short answer.
--quick is a smoke test (128-token answers, 2,000-token prefill). Its output is marked "quick run" and does not go on the wiki.

Speculative decoding

Every number carries the speculative decoding it was measured with: the method and the number of draft tokens, such as MTP 3 or DFlash2, 7 draft tokens, or none. Pass it with --spec-decode and it lands in the spec_decode field. Temperature-0 prompts get higher draft acceptance than real work does, so a recipe that uses speculative decoding should also post a run with it off when it can. On vLLM the script reads the server's draft and acceptance counters from /metrics during stage 1 and prints the mean acceptance length; the counters cover the whole server, so other traffic would mix in.

Power caps, links and P2P

Power cap. The script reads each card's power limit from nvidia-smi. One value for all cards fills power_w and power_cap; different caps give a range in power_cap and leave power_w out. The peak draw during the run is in the table and the JSON. Set the caps before the run (nvidia-smi -pl) and leave them alone during it.
Links. PCIe generation and width are read every 2 seconds during the run, because idle cards drop their link to Gen1. They fill pcie_link (Gen1, Gen2, Gen3 or mixed) and pcie_width (x1, x4, x16 or mixed). A width the field does not take, such as x8, goes into bench_note.
Card variant. The PCI device id from lspci -nn (10de:20c2 = 8 GB card, 10de:2082 = 10 GB card) fills cards, and the number of CMP 170HX cards fills card_count. Pass --gpus 0,1,2,3 when the machine has more cards than the server uses.
P2P, topology and flags. The script cannot see what the engine does with the cards. State p2p (required, optional, on or off), topology (TP4, PP2 and so on) and flags yourself.

Filling the infobox

The first wikitext block holds one |field=value line per measured field. Paste it into the Recipe template call when you edit the page source, or copy each value into the box of the same name in Form:Recipe. Lines between  are notes to you and do not show on the page.

Field	Filled from
`decode_tps`, `decode_single`	stage 1, code prompt
`decode_agg`, `agg_users`	stage 3
`prefill_tps`, `prefill_prompt`	stage 2, with the prompt length as measured
`cards`, `card_count`	the device ids of the CMP 170HX cards
`pcie_link`, `pcie_width`	nvidia-smi, sampled during the run
`power_w`, `power_cap`, `driver`	nvidia-smi
`context_tokens`	the server's model list, when it reports the context length
`engine_family`, `engine_version`	the server's model list and its version endpoint (vLLM, llama.cpp)
`spec_decode`	`--spec-decode`
`bench_note`	protocol version, the structured and prose numbers, the user count, the prefill length, anything unusual (short answers, estimated tokens, an x8 link) and `--note`
`tested_on`	the date of the run (UTC)

Left for you: model, quant, engine, topology, p2p, flags, context, vram_per_card, unlock_build, and spec_decode when you did not pass --spec-decode.

Confirmations

To confirm a recipe on your own cards, run the protocol with the recipe's settings and press Confirm on your own card in its infobox. The second wikitext block fills that form: numbers holds the measured line, cards and date are set, and result is yours to choose (Reproduced, Partly reproduced or Did not reproduce) after you compare with the recipe's own numbers. See Status taxonomy for what a confirmation does to the badge.

Running the script

Python 3.9 or newer, standard library only, one file. The GPU details need Linux with nvidia-smi (and lspci for the device ids); the measurements work anywhere.

# vLLM or SGLang
python3 cmpbench.py --base-url http://localhost:8000/v1 --model glm-5.3-flash --spec-decode "MTP 3"
# llama.cpp server: start it with --jinja (for the no-thinking hint) and --parallel 8 (for stage 3)
python3 cmpbench.py --base-url http://localhost:8080/v1 --model MODEL --spec-decode none
# TabbyAPI with a key, on four of the machine's eight cards
python3 cmpbench.py --base-url http://localhost:5000/v1 --model MODEL --api-key "$TABBY_KEY" --gpus 0,1,2,3
# smoke test: short answers, two users
python3 cmpbench.py --base-url http://localhost:8000/v1 --model MODEL --quick --users 2
# skip stage 3, add a note, write the JSON to a chosen file
python3 cmpbench.py --base-url http://localhost:8000/v1 --model MODEL --no-concurrent --note "x16 capacitor mod" --json run2.json

Option	What it does
`--base-url URL`	the server, up to `/v1`
`--model ID`	the model id as the server lists it (`GET /v1/models`)
`--api-key KEY`	bearer token, if the server needs one (or the variable `CMPBENCH_API_KEY`)
`--users N`	requests in stage 3 (default 8)
`--no-concurrent`	skip stage 3
`--quick`	smoke test; its numbers are not protocol numbers
`--spec-decode TEXT`	the speculative decoding the server runs, or `none`
`--note TEXT`	added to `bench_note` and to the confirmation notes
`--json FILE`	where the JSON report goes (`-` prints it); default `cmpbench-<time>.json`
`--gpus LIST`	nvidia-smi indices of the cards the server uses
`--thinking default`	send no thinking hint
`--no-gpu-info`, `--no-metrics`, `--no-model-check`	skip nvidia-smi and lspci, skip `/metrics`, send a model the server does not list
`--timeout S`, `--insecure`	seconds per request (default 900); no certificate check for https

What it sends and reads

Network: only to --base-url. GET /v1/models, /metrics and /version (and /props on llama.cpp), POST /v1/chat/completions. Proxy settings are ignored, nothing is uploaded anywhere, and the API key is not written to the JSON.
This machine: nvidia-smi (name, memory, PCI bus and device id, link generation and width, power limit and draw, driver) and lspci -nn. The script never reads GPU serial numbers or UUIDs.
The two wikitext blocks hold no address or host name. The table and the JSON show the endpoint, so do not paste them whole.
When --base-url points at another machine, the GPU details describe the machine the script runs on and the script warns. Run it on the inference server, or pass --no-gpu-info and fill the card fields by hand.

Errors and exit codes

0: every stage ran. 1: a stage failed; the report covers the others. 2: nothing was measured.
Exit 2 comes with the reason: the connection was refused, the model is not in the server's list (the script prints the names the server lists), the key was refused, or the server has no model ready (HTTP 503).
A prompt longer than the server's context shows as a prefill error. Give the server at least 9,000 tokens of context per request.
A server that does not stream is still measured, but decode then includes prefill and TTFT is missing, so the wiki block leaves the decode fields out.

Why these choices

Prefill from TTFT, timed by the client. Engine log lines measure per batch and often show 2x or more the real single-request rate. Source: @vektor_prime, 2026-08-15
Source: @giaki3003, 2026-08-15
Source: @robbbbbb7973, 2026-08-27
Fixed prompts and settings. Recipes only compare on equal benchmarks, and people asked for a standard benchmark and a short standard-format document anyone can follow. Source: @_studebaker_, 2026-08-23
Source: @con3po7016, 2026-09-25
Speculative decoding stated with every number. Temperature-0 runs reach draft acceptance far above what real code gets. Source: @matstatman, 2026-08-18
Short context only. Decode falls as the context fills, and this protocol measures near-empty context. Post deep-context numbers as a separate line in the recipe's Notes, with the depth. Source: @matstatman, 2026-08-18
Source: @roger_dammit, 2026-08-18
Source: @bayley_w, 2026-08-24
Three prompts. The code, structured and prose split follows the columns recipe authors already post.
Check what a micro-benchmark measured. Two traps from the 170tune reference card's notes: CUDA 13 removed cudaDeviceProp::clockRate and memoryClockRate, so query cudaDeviceGetAttribute with cudaDevAttrClockRate and cudaDevAttrMemoryClockRate instead; and 1,769 GB/s is the theoretical HBM peak at 1,728 MHz (clock x 4,096-bit bus x 2 / 8), while a measured 24 GiB read stream delivers 95 to 96 % of it (1,679 to 1,699 GB/s). A cuBLAS call that returns instantly is the third trap (see #What was tried and did not work). Source: repo github.com/cachenetics/170tune by cachenetics on GitHub, posted by @_studebaker_, 2026-07-27 According to the Consensus-Protocol research wiki, which its author calls unverified, mixbench prints such computed peaks next to its results, so its 1769.47 GB/s and 12633.60 GFLOPS are clock times bus-width arithmetic, not measurements. Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub
Tensor-core work needs a tensor-core benchmark. The 170tune reference card does 12.76 TFLOPS in plain FP32 and 188 to 193 TFLOPS in bf16 on the tensor cores, so a benchmark that never reaches the tensor cores, such as the OpenCL kernels of OpenCL-Benchmark, describes a much slower card than the one you serve on. @djpandaman also found OpenCL-Benchmark a poor guide to the card's capability and preferred clpeak. Source: repo github.com/cachenetics/170tune by cachenetics on GitHub, posted by @_studebaker_, 2026-07-27
Source: @djpandaman, 2026-10-23
Source: repo github.com/ProjectPhysX/OpenCL-Benchmark by ProjectPhysX on GitHub

More on how the community reports numbers: Benchmarks#How to report numbers.

Numbers from before the protocol

Every number on the wiki that is older than this page (November 2026) predates the protocol. Most recipe authors used the MiaAI-Lab protocol (400-token answers, temperature 0, thinking off, three columns in the order structured / code / prose); the lab's own bench used 1,024-token answers in the order code / structured / prose; other numbers come from single runs, agent sessions or engine log lines. They stay as posted, with their conditions in bench_note or the recipe's Notes. They are different measurements from protocol numbers, so compare them with care. Numbers measured with the protocol say "cmpbench 1.0" in bench_note.

Versions

Protocol 1.0, cmpbench 1.0 (November 2026): first version.

Protocol and script: Source: @darkmaniac7, 2026-11-03 (lab notes)

What was tried and did not work

Reported cuBLAS with CUBLAS_COMPUTE_16F and float alpha and beta. A GEMM benchmark for the 170tune reference card (notes of 2026-07-27) reported 10,748 TFLOPS. It was a no-op: passing float alpha and beta pointers with the 16F compute type makes the GEMM return at once without computing, and the instant return gave it away. Use the FP32-accumulate results. Source: repo github.com/cachenetics/170tune by cachenetics on GitHub, posted by @_studebaker_, 2026-07-27