Monitoring

Reported 8 GB and 10 GB

What to watch on a running CMP 170HX and which tools show it. Cooling fixes are on Cooling and power, fault diagnosis on Troubleshooting, and health tests on Testing and burn-in.

What to watch

Signal	Why	Read it with
Core and HBM temperature	throttling and memory errors start well below the card's maximum	`nvidia-smi -q -d TEMPERATURE`, hbmmon, llm-visuals
Power draw and power cap	the cap resets on every boot or driver reload; spikes exceed the cap	`nvidia-smi`, llm-visuals
Clocks and throttle reasons	shows thermal, power-cap or power-brake throttling	NVML clock event reasons (llm-visuals), `170tune status` for the memory clock
Xid events	every GPU fault the driver sees	`dmesg` / `journalctl -k`, the InfoROM log via gpcprobe `blackbox.py`
ECC counters	real only for some counters, see below	`nvidia-smi -q -d ECC`, an NVIDIA Prometheus exporter
PCIe link and errors	link drops and replays point at risers, slots and power	`nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current`, `lspci -vv`, AER
Serving metrics	decode and prefill speed, queue, KV use	vLLM `/metrics` with Prometheus and Grafana, llm-visuals

Temperatures

Limits

Contested Posters agree the HBM is what to protect and that it should stay well below 80 to 85 C. The exact numbers differ:

Throttling starts around 76 C; keep core and HBM under 75 C. Source: @zorg33, 2026-10-20
The cards throttle at 85 C. Source: @matstatman, 2026-10-09 85 C is the maximum. Source: @xeax, 2026-10-15
"Anything above 80 is too hot." Source: @spectakkular, 2026-10-09 The SMs are not the issue, the HBM is. Source: @replacedriver47, 2026-10-09 The HBM has no issues until about 80 C. Source: @replacedriver47, 2026-10-10
Warning With HBM over 95 C you would see total instability; a few minutes at that temperature corrupts part of the WPR. Source: @bigptoughneigh, 2026-10-09
One card ran to 100 C by accident, throttled hard and still works. Source: @sojun80, 2026-10-06

Tool limits for comparison: the cmp170hx-testsuite guard stops at 72 C core and 85 C HBM Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09.; the club-170hx stop rule is 80 C core and 85 C memory Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20.; 170tune's memory overclock is only clean with HBM at or below about 76 C, and it drives the fan from HBM temperature. Source: repo github.com/cachenetics/170tune by cachenetics on GitHub, posted by @rh3dai, 2026-10-04.

Reading them

nvidia-smi -q -d TEMPERATURE shows the core and memory temperature. Source: @xeax, 2026-10-15 Most owners only see GPU, hotspot and memory temperatures. Source: @zefrench6961, 2026-10-06
Source: @phoenixone7, 2026-10-07
The card has about ten temperature sensors. @matstatman asks for per-HBM-stack temperatures when debugging dropouts. Source: @matstatman, 2026-10-05 hbmmon shows the temperature of each HBM stack. Source: repo github.com/matstatman/hbmmon by matstatman on GitHub, posted by @matstatman, 2026-10-05.
llm-visuals reads the HBM temperature through NVML (field value 82); its author verified it on a CMP 170HX. It flags memory as hot when the HBM stays at or above the driver's memory max-operating temperature (95 C if the driver gives none) for 3 seconds. Source: repo github.com/DingoOz/llm-visuals by DingoOz on GitHub, posted by @trustinter, 2026-10-17.
The cmp170hx-testsuite has cmp-telemetry, a live panel for a second terminal: temperatures, power, load, VRAM, clocks, blower rpm and Xid count. Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09.
Idle references: an 8 GB card idled at 37 C and a 10 GB card at 41 C on the same host, both at about 33 W. Source: @ghostinthedatacenter_87281, 2026-10-13
Source: @ghostinthedatacenter_87281, 2026-10-13

Fan control from the card's temperature

The VBIOS has no fan curve, and PWM on the card's own fan header did not work: the fan ran at full speed. Drive fans from the motherboard or BMC instead. Source: @port00, 2026-10-22
Source: @port00, 2026-10-22
A BMC could read the temperatures of cards in motherboard slots, but not of a card on a passive riser, which it saw only as an add-in card; that card's fans need software control. Source: @fafinet, 2026-10-21
Source: @fafinet, 2026-10-21
Theory A 10-port PWM fan controller that can follow GPU temperature was bought for this; on Linux it needs a kernel update or patching, not yet done. Source: @globalapathy, 2026-10-09

Fan curves and airflow numbers are on Cooling and power.

Power

The power cap is not persistent. nvidia-smi -pl resets at every boot Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09., and a driver module reload or VM reboot returns it to the VBIOS default of 250 W. Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20. People re-apply it with a systemd service, a crontab entry or a sysvinit service. Source: @robbbbbb7973, 2026-10-15
Source: @xanthefin, 2026-10-15
Source: @xanthefin, 2026-10-15
Warning Spikes go over the cap: up to 380 W instantaneous on cards capped at 250 W during inference. Source: @n2dcrypt, 2026-10-05 Size power supplies for the spikes. Source: @reala7096, 2026-10-05
A 150 W ceiling on some cards is a VBIOS power-table value that neither 170tune nor cmpunlocker changes. Source: repo github.com/vektorprime/170tune by vektorprime on GitHub, posted by @rh3dai, 2026-10-04.
Idle draw: about 33 W stock. Source: @ghostinthedatacenter_87281, 2026-10-13 23 to 26 W on three cards with 170tune's idle settings (driver 610.57.04, VBIOS 92.00.6D.00.0A). Source: @robbbbbb7973, 2026-10-10
Source: @robbbbbb7973, 2026-10-11
Theory nvidia-pstated, a daemon that switches GPUs to a low performance state when idle, was suggested; nobody reported running it on a 170HX. It must run on the host, not in a container. Source: @d3m3n7r, 2026-10-13
Source: repo github.com/sasha0552/nvidia-pstated by sasha0552 on GitHub, posted by @d3m3n7r, 2026-10-13.

Power supplies, cables and caps for serving are on Cooling and power.

Clocks and throttling

nvidia-smi reports 1728 MHz memory clock at every 170tune memory setting. Read the real PLL with 170tune -i N status. Source: repo github.com/cachenetics/170tune by cachenetics on GitHub, posted by @rh3dai, 2026-10-04.
An NVML clock-offset write can return success while the driver drops it; only reading the value back proves it applied. Source: repo github.com/vektorprime/170tune by vektorprime on GitHub, posted by @rh3dai, 2026-10-04.
llm-visuals shows NVML clock event reasons as badges: THERM (hardware thermal), BRAKE (hardware power brake, an external signal such as PWRBRK#), HW, PWR (software power cap), SYNC and APP. Source: repo github.com/DingoOz/llm-visuals by DingoOz on GitHub, posted by @trustinter, 2026-10-17.
To find what is capped, log clocks to a CSV and watch which one drops. Source: @wakeupmbrc1, 2026-10-13

Xids and memory errors

Watch the kernel log for Xid lines while the card works. The club-170hx QC scripts include xid-watch.sh Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20., and the test suite's cmp-kernel-check lists Xids since the first test of the day. Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09.
The card keeps its own log. gpcprobe blackbox.py reads the InfoROM DEM object, a 4 KiB FIFO of Xid records with unix timestamps, plus boot count, power-on time and power-cap history. Source: repo github.com/matstatman/gpcprobe by matstatman on GitHub, posted by @matstatman, 2026-10-20.
Xids seen on these cards and what they meant are on Troubleshooting. Ones that come up in tuning: Xid 13 then 43 on weak SMs at too high a clock offset; Xid 45 or 119 after changing the memory clock under an active CUDA context. Source: repo github.com/cachenetics/170tune by cachenetics on GitHub, posted by @rh3dai, 2026-10-04.

ECC counters

Contested With cmpunlocker v0.5's ECC unlock, which numbers are real:

@alanoo first found the ECC error counter hardcoded to 0 in the driver patch, then saw that nvidia-smi reads other counters. Source: @alanoo, 2026-10-13
Source: @alanoo, 2026-10-13
@ichundes saw his ECC counter increase on an SRAM error in the NVIDIA Prometheus exporter; unknown for VRAM. Source: @ichundes, 2026-10-13
Source: @ichundes, 2026-10-13
@zombielurker had a card with about 5,000 single-bit and multi-bit errors in a 5-minute memtest_vulkan run; with ECC shown as enabled none were corrected. Source: @zombielurker, 2026-10-05 @endsjeremy asked whether the enabled state is only a reporting artifact. Source: @endsjeremy, 2026-10-05

Do not take "ECC Enabled" in nvidia-smi as proof that memory errors are corrected; run the memory tests on Testing and burn-in. The ECC unlock itself is on Unlocks.

PCIe link

Check generation and width with nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv. nvidia-smi guesses both from bandwidth; lspci -vv shows the real values. Source: @djpandaman, 2026-10-21
At idle the link drops; a screenshot showing x1 Gen1 was an idle card. Source: @xeax, 2026-10-18
The club-170hx QC scripts include pcie-check.sh and take PCIe replay and AER snapshots before and after each card. Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20. lil-bench in llm-inference-bench reports PCIe replays, correctable errors and links below their maximum generation or width under load. Source: repo github.com/local-inference-lab/llm-inference-bench by local-inference-lab on GitHub, posted by @mountains1605, 2026-10-12.

Serving metrics

@matstatman uses Grafana and Prometheus scraping vLLM, and calls it lightweight. Source: @matstatman, 2026-10-04
Source: @matstatman, 2026-10-04
llm-visuals is a terminal dashboard that auto-detects llama-server, ollama, vLLM, SGLang and others and shows decode and prefill speed, TTFT, per-GPU utilisation, power, HBM temperature, the VRAM split between weights and KV cache, and tokens per joule. Source: repo github.com/DingoOz/llm-visuals by DingoOz on GitHub, posted by @trustinter, 2026-10-17.
After a card hit 100 C, @sojun80 wrote a remote temperature monitor (screenshots only, no code posted). Source: @sojun80, 2026-10-06

A starting setup

For one or a few cards (a summary of the tools above):

A systemd service that sets the power cap after the driver loads.
A temperature and power log: nvidia-smi --query-gpu=index,temperature.gpu,temperature.memory,power.draw,clocks.sm --format=csv -l 5 to a file, or llm-visuals in a terminal.
An Xid watch on the kernel log.
Fans driven by the hottest core or HBM temperature (see Cooling and power).
blackbox.py from gpcprobe when you get a card and after any crash.