https://cmp.wiki/w/index.php?action=history&feed=atom&title=MonitoringMonitoring - Revision history2026-10-31T11:45:22ZRevision history for this page on the wikiMediaWiki 1.43.11https://cmp.wiki/w/index.php?title=Monitoring&diff=188&oldid=prevImportBot: Import: monitoring (temperatures, power, clocks, Xids, ECC, PCIe), @matstatman, @rh3dai, @boz._., @c133, @xeax, @trustinter, @ghostinthedatacenter_87281, @robbbbbb7973, @replacedriver47, @sojun80, @port00, @fafinet, @xanthefin, @d3m3n7r; repos DingoOz/llm-visuals, PixelML/club-170hx, cachenetics/170tune, dev-boz/cmp170hx-testsuite, local-inference-lab/llm-inference-bench, matstatman/gpcprobe, matstatman/hbmmon, sasha0552/nvidia-pstated2026-10-30T08:04:49Z<p>Import: monitoring (temperatures, power, clocks, Xids, ECC, PCIe), @matstatman, @rh3dai, @boz._., @c133, @xeax, @trustinter, @ghostinthedatacenter_87281, @robbbbbb7973, @replacedriver47, @sojun80, @port00, @fafinet, @xanthefin, @d3m3n7r; repos DingoOz/llm-visuals, PixelML/club-170hx, cachenetics/170tune, dev-boz/cmp170hx-testsuite, local-inference-lab/llm-inference-bench, matstatman/gpcprobe, matstatman/hbmmon, sasha0552/nvidia-pstated</p>
<p><b>New page</b></p><div>{{Status|Reported}} {{Cards|both}}<br />
<br />
What to watch on a running CMP 170HX and which tools show it. Cooling fixes are on [[Cooling and power]], fault diagnosis on [[Troubleshooting]], and health tests on [[Testing and burn-in]].<br />
<br />
== What to watch ==<br />
{| class="wikitable"<br />
! Signal !! Why !! Read it with<br />
|-<br />
| Core and HBM temperature || throttling and memory errors start well below the card's maximum || <code><nowiki>nvidia-smi -q -d TEMPERATURE</nowiki></code>, [[hbmmon]], [[llm-visuals]]<br />
|-<br />
| Power draw and power cap || the cap resets on every boot or driver reload; spikes exceed the cap || <code><nowiki>nvidia-smi</nowiki></code>, [[llm-visuals]]<br />
|-<br />
| Clocks and throttle reasons || shows thermal, power-cap or power-brake throttling || NVML clock event reasons ([[llm-visuals]]), <code><nowiki>170tune status</nowiki></code> for the memory clock<br />
|-<br />
| Xid events || every GPU fault the driver sees || <code><nowiki>dmesg</nowiki></code> / <code><nowiki>journalctl -k</nowiki></code>, the InfoROM log via [[gpcprobe]] <code><nowiki>blackbox.py</nowiki></code><br />
|-<br />
| ECC counters || real only for some counters, see below || <code><nowiki>nvidia-smi -q -d ECC</nowiki></code>, an NVIDIA Prometheus exporter<br />
|-<br />
| PCIe link and errors || link drops and replays point at risers, slots and power || <code><nowiki>nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current</nowiki></code>, <code><nowiki>lspci -vv</nowiki></code>, AER<br />
|-<br />
| Serving metrics || decode and prefill speed, queue, KV use || vLLM <code><nowiki>/metrics</nowiki></code> with Prometheus and Grafana, [[llm-visuals]]<br />
|}<br />
<br />
== Temperatures ==<br />
=== Limits ===<br />
{{Status|Contested}} Posters agree the HBM is what to protect and that it should stay well below 80 to 85 C. The exact numbers differ:<br />
* Throttling starts around 76 C; keep core and HBM under 75 C. {{Source|handle=zorg33|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1562110412395380767}}<br />
* The cards throttle at 85 C. {{Source|handle=matstatman|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1558212817121972318}} 85 C is the maximum. {{Source|handle=xeax|date=2026-10-15|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560252687839133757}}<br />
* "Anything above 80 is too hot." {{Source|handle=spectakkular|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1558215172294840442}} The SMs are not the issue, the HBM is. {{Source|handle=replacedriver47|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1558217213717254195}} The HBM has no issues until about 80 C. {{Source|handle=replacedriver47|date=2026-10-10|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1558399070006157436}}<br />
* {{Status|Warning}} With HBM over 95 C you would see total instability; a few minutes at that temperature corrupts part of the WPR. {{Source|handle=bigptoughneigh|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1558266699218812929}}<br />
* One card ran to 100 C by accident, throttled hard and still works. {{Source|handle=sojun80|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556855494008643698}}<br />
Tool limits for comparison: the cmp170hx-testsuite guard stops at 72 C core and 85 C HBM {{Source|handle=boz._.|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1558098091574562848|repo=https://github.com/dev-boz/cmp170hx-testsuite}}.; the club-170hx stop rule is 80 C core and 85 C memory {{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}.; 170tune's memory overclock is only clean with HBM at or below about 76 C, and it drives the fan from HBM temperature. {{Source|handle=rh3dai|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1556287917775655053|repo=https://github.com/cachenetics/170tune}}.<br />
<br />
=== Reading them ===<br />
* <code><nowiki>nvidia-smi -q -d TEMPERATURE</nowiki></code> shows the core and memory temperature. {{Source|handle=xeax|date=2026-10-15|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560253301985906692}} Most owners only see GPU, hotspot and memory temperatures. {{Source|handle=zefrench6961|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557175100824879127}}<br />{{Source|handle=phoenixone7|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557190829087395901}}<br />
* The card has about ten temperature sensors. @matstatman asks for per-HBM-stack temperatures when debugging dropouts. {{Source|handle=matstatman|date=2026-10-05|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556770474695069718}} [[hbmmon]] shows the temperature of each HBM stack. {{Source|handle=matstatman|date=2026-10-05|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556758870565593288|repo=https://github.com/matstatman/hbmmon}}.<br />
* [[llm-visuals]] reads the HBM temperature through NVML (field value 82); its author verified it on a CMP 170HX. It flags memory as hot when the HBM stays at or above the driver's memory max-operating temperature (95 C if the driver gives none) for 3 seconds. {{Source|handle=trustinter|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561025692890832906|repo=https://github.com/DingoOz/llm-visuals}}.<br />
* The cmp170hx-testsuite has <code><nowiki>cmp-telemetry</nowiki></code>, a live panel for a second terminal: temperatures, power, load, VRAM, clocks, blower rpm and Xid count. {{Source|handle=boz._.|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1558098091574562848|repo=https://github.com/dev-boz/cmp170hx-testsuite}}.<br />
* Idle references: an 8 GB card idled at 37 C and a 10 GB card at 41 C on the same host, both at about 33 W. {{Source|handle=ghostinthedatacenter_87281|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1559651382816936108}}<br />{{Source|handle=ghostinthedatacenter_87281|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1559652465165602950}}<br />
<br />
=== Fan control from the card's temperature ===<br />
* The VBIOS has no fan curve, and PWM on the card's own fan header did not work: the fan ran at full speed. Drive fans from the motherboard or BMC instead. {{Source|handle=port00|date=2026-10-22|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562640840403460171}}<br />{{Source|handle=port00|date=2026-10-22|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562641203550494754}}<br />
* A BMC could read the temperatures of cards in motherboard slots, but not of a card on a passive riser, which it saw only as an add-in card; that card's fans need software control. {{Source|handle=fafinet|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562285217731973171}}<br />{{Source|handle=fafinet|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1562300635725172866}}<br />
* {{Status|Theory}} A 10-port PWM fan controller that can follow GPU temperature was bought for this; on Linux it needs a kernel update or patching, not yet done. {{Source|handle=globalapathy|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1558106944315723807}}<br />
Fan curves and airflow numbers are on [[Cooling and power]].<br />
<br />
== Power ==<br />
* The power cap is not persistent. <code><nowiki>nvidia-smi -pl</nowiki></code> resets at every boot {{Source|handle=boz._.|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1558098091574562848|repo=https://github.com/dev-boz/cmp170hx-testsuite}}., and a driver module reload or VM reboot returns it to the VBIOS default of 250 W. {{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}. People re-apply it with a systemd service, a crontab entry or a sysvinit service. {{Source|handle=robbbbbb7973|date=2026-10-15|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1560140293465186420}}<br />{{Source|handle=xanthefin|date=2026-10-15|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1560276495967068273}}<br />{{Source|handle=xanthefin|date=2026-10-15|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1560276579974643834}}<br />
* {{Status|Warning}} Spikes go over the cap: up to 380 W instantaneous on cards capped at 250 W during inference. {{Source|handle=n2dcrypt|date=2026-10-05|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556771516803321877}} Size power supplies for the spikes. {{Source|handle=reala7096|date=2026-10-05|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556771733208440974}}<br />
* A 150 W ceiling on some cards is a VBIOS power-table value that neither 170tune nor cmpunlocker changes. {{Source|handle=rh3dai|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1556287917775655053|repo=https://github.com/vektorprime/170tune}}.<br />
* Idle draw: about 33 W stock. {{Source|handle=ghostinthedatacenter_87281|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1559652465165602950}} 23 to 26 W on three cards with 170tune's idle settings (driver 610.57.04, VBIOS 92.00.6D.00.0A). {{Source|handle=robbbbbb7973|date=2026-10-10|link=https://discord.com/channels/1518634379604136178/1534543856006332586/1558371152831844353}}<br />{{Source|handle=robbbbbb7973|date=2026-10-11|link=https://discord.com/channels/1518634379604136178/1534543856006332586/1558697399491240027}}<br />
* {{Status|Theory}} nvidia-pstated, a daemon that switches GPUs to a low performance state when idle, was suggested; nobody reported running it on a 170HX. It must run on the host, not in a container. {{Source|handle=d3m3n7r|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1518691940646781070/1559687539361714176}}<br />{{Source|handle=d3m3n7r|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1518691940646781070/1559687539361714176|repo=https://github.com/sasha0552/nvidia-pstated}}.<br />
Power supplies, cables and caps for serving are on [[Cooling and power]].<br />
<br />
== Clocks and throttling ==<br />
* <code><nowiki>nvidia-smi</nowiki></code> reports 1728 MHz memory clock at every 170tune memory setting. Read the real PLL with <code><nowiki>170tune -i N status</nowiki></code>. {{Source|handle=rh3dai|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1556287917775655053|repo=https://github.com/cachenetics/170tune}}.<br />
* An NVML clock-offset write can return success while the driver drops it; only reading the value back proves it applied. {{Source|handle=rh3dai|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1556287917775655053|repo=https://github.com/vektorprime/170tune}}.<br />
* [[llm-visuals]] shows NVML clock event reasons as badges: THERM (hardware thermal), BRAKE (hardware power brake, an external signal such as PWRBRK#), HW, PWR (software power cap), SYNC and APP. {{Source|handle=trustinter|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561025692890832906|repo=https://github.com/DingoOz/llm-visuals}}.<br />
* To find what is capped, log clocks to a CSV and watch which one drops. {{Source|handle=wakeupmbrc1|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1559582141917171814}}<br />
<br />
== Xids and memory errors ==<br />
* Watch the kernel log for Xid lines while the card works. The club-170hx QC scripts include <code><nowiki>xid-watch.sh</nowiki></code> {{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}., and the test suite's <code><nowiki>cmp-kernel-check</nowiki></code> lists Xids since the first test of the day. {{Source|handle=boz._.|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1558098091574562848|repo=https://github.com/dev-boz/cmp170hx-testsuite}}.<br />
* The card keeps its own log. [[gpcprobe]] <code><nowiki>blackbox.py</nowiki></code> reads the InfoROM DEM object, a 4 KiB FIFO of Xid records with unix timestamps, plus boot count, power-on time and power-cap history. {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850|repo=https://github.com/matstatman/gpcprobe}}.<br />
* Xids seen on these cards and what they meant are on [[Troubleshooting]]. Ones that come up in tuning: Xid 13 then 43 on weak SMs at too high a clock offset; Xid 45 or 119 after changing the memory clock under an active CUDA context. {{Source|handle=rh3dai|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1556287917775655053|repo=https://github.com/cachenetics/170tune}}.<br />
<br />
=== ECC counters ===<br />
{{Status|Contested}} With cmpunlocker v0.5's ECC unlock, which numbers are real:<br />
* @alanoo first found the ECC error counter hardcoded to 0 in the driver patch, then saw that nvidia-smi reads other counters. {{Source|handle=alanoo|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1559518544906158082}}<br />{{Source|handle=alanoo|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1559521944024387685}}<br />
* @ichundes saw his ECC counter increase on an SRAM error in the NVIDIA Prometheus exporter; unknown for VRAM. {{Source|handle=ichundes|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1559520707245776987}}<br />{{Source|handle=ichundes|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1559577873927372942}}<br />
* @zombielurker had a card with about 5,000 single-bit and multi-bit errors in a 5-minute memtest_vulkan run; with ECC shown as enabled none were corrected. {{Source|handle=zombielurker|date=2026-10-05|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556515369286504489}} @endsjeremy asked whether the enabled state is only a reporting artifact. {{Source|handle=endsjeremy|date=2026-10-05|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556485503514054740}}<br />
Do not take "ECC Enabled" in nvidia-smi as proof that memory errors are corrected; run the memory tests on [[Testing and burn-in]]. The ECC unlock itself is on [[Unlocks]].<br />
<br />
== PCIe link ==<br />
* Check generation and width with <code><nowiki>nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv</nowiki></code>. nvidia-smi guesses both from bandwidth; <code><nowiki>lspci -vv</nowiki></code> shows the real values. {{Source|handle=djpandaman|date=2026-10-21|link=https://discord.com/channels/1518634379604136178/1534543856006332586/1562332665108234301}}<br />
* At idle the link drops; a screenshot showing x1 Gen1 was an idle card. {{Source|handle=xeax|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561265565736370219}}<br />
* The club-170hx QC scripts include <code><nowiki>pcie-check.sh</nowiki></code> and take PCIe replay and AER snapshots before and after each card. {{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}. lil-bench in [[llm-inference-bench]] reports PCIe replays, correctable errors and links below their maximum generation or width under load. {{Source|handle=mountains1605|date=2026-10-12|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1559350041645420624|repo=https://github.com/local-inference-lab/llm-inference-bench}}.<br />
<br />
== Serving metrics ==<br />
* @matstatman uses Grafana and Prometheus scraping vLLM, and calls it lightweight. {{Source|handle=matstatman|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556290743750885397}}<br />{{Source|handle=matstatman|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556291579742781491}}<br />
* [[llm-visuals]] is a terminal dashboard that auto-detects llama-server, ollama, vLLM, SGLang and others and shows decode and prefill speed, TTFT, per-GPU utilisation, power, HBM temperature, the VRAM split between weights and KV cache, and tokens per joule. {{Source|handle=trustinter|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561025692890832906|repo=https://github.com/DingoOz/llm-visuals}}.<br />
* After a card hit 100 C, @sojun80 wrote a remote temperature monitor (screenshots only, no code posted). {{Source|handle=sojun80|date=2026-10-06|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556855494008643698}}<br />
<br />
== A starting setup ==<br />
For one or a few cards (a summary of the tools above):<br />
# A systemd service that sets the power cap after the driver loads.<br />
# A temperature and power log: <code><nowiki>nvidia-smi --query-gpu=index,temperature.gpu,temperature.memory,power.draw,clocks.sm --format=csv -l 5</nowiki></code> to a file, or [[llm-visuals]] in a terminal.<br />
# An Xid watch on the kernel log.<br />
# Fans driven by the hottest core or HBM temperature (see [[Cooling and power]]).<br />
# <code><nowiki>blackbox.py</nowiki></code> from [[gpcprobe]] when you get a card and after any crash.<br />
<br />
== See also ==<br />
* [[Tools]]<br />
* [[hbmmon]]<br />
* [[gpcprobe]]<br />
* [[llm-visuals]]<br />
* [[Testing and burn-in]]<br />
* [[Cooling and power]]<br />
* [[Troubleshooting]]<br />
<br />
{{ImportFooter}}<br />
[[Category:Tools]]<br />
[[Category:8 GB]]<br />
[[Category:10 GB]]</div>ImportBot