Testing and burn-in

Reported 8 GB and 10 GB

How to find out whether a CMP 170HX is healthy: after buying it (Acceptance checklist), after an unlock or driver change, and when it starts misbehaving (Troubleshooting). The tools are listed on Tools.

Why the usual quick test is not enough

These are used mining cards, and the faults people found do not all show up in a short memory test.

Warning Load-transient HBM corruption. The card that motivated the cmp170hx-testsuite (unlocked to 64 GB, 180 W, normal temperatures) read whole 128-byte blocks back as zeros or unrelated data under sustained full-bandwidth traffic. Light tests passed. memtest_vulkan failed only right after a cold boot and passed most other runs. Compute tests were clean. Its firmware also crashed under load (Xid 119, Xid 1 GSP task exception, Xid 154, or Xid 62 PMU halt) and needed a full power-off of about 30 seconds; a warm reboot was not enough. Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09.
Warning Tensor cores that return garbage. "There will be cards that pass mem tests but the tensor cores produce complete garbage." Source: @_____josh_____, 2026-10-08 One card returned gibberish with no errors. Source: @_____josh_____, 2026-10-08
Wrong output at stock clock. One card in @reala7096's build produced all-zero MoE output rows about once per 3,000 decode steps, and occasional garbage top-k indices, while passing GEMM and memory tests. Source: @reala7096, 2026-10-08
Source: @reala7096, 2026-10-08
A faked size. A single-rank card that reported 64 GB passed memtest_vulkan, while a CUDA memory test failed past 32 GB. Check the upper VRAM with a CUDA-side test. Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20.
A card with bad HBM passed a basic Vulkan test 9 times out of 10 but failed most other tests, and inference returned all NaN. Source: @boz._., 2026-10-23

Before you start

Test one card at a time, alone in the machine if you can. Parallel tests make it hard to tell a bad card from a PSU, riser, motherboard or cooling limit. Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09.
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20.
Cap the power, for example sudo nvidia-smi -i <index> -pl 180; the cap resets at every boot. Run the fan at full speed. Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09.
Have a stop rule and a watcher. The club-170hx guide stops at 80 C core, 85 C memory, any Xid, the GPU disappearing, or a verification mismatch, and starts an Xid watcher first. Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20. The test suite's guard stops a test at 72 C core or 85 C HBM, when a mapped blower falls below 800 rpm for 3 s, or when nvidia-smi stops answering. Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09. Temperatures are on Monitoring.
Run the memory test early, right after a cold boot: that is when the bad card above failed it. Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09.
No serving process on the card while testing. Several tools allocate memory on every visible GPU.

The test sequence

This order follows the cmp170hx-testsuite (about 40 minutes per card) and the club-170hx acceptance ladder; new owners have used the club-170hx steps to validate incoming cards. Source: @hum1d, 2026-10-07
Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09.
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20.

Step	What	Tool	Pass
1	Identity and inventory	`cmp-identity`, `nvidia-smi`, `lspci`	expected PCI ID; 65536 MiB (8 GB card) or 40960 MiB (10 GB card) after the unlock; 0 Xid since boot
2	History	gpcprobe `blackbox.py` and `gpc_probe.py`	no surprises in the Xid log; fuse map as expected for the variant
3	Memory, quick	memtest_vulkan, standard 5-minute test	no errors
4	Compute burn	gpu-burn with tensor cores	`errors: 0` on every line, `GPU 0: OK`
5	Memory, full	`cmp-hbm-test`, or club-170hx `compare_vram.cu` at about 62 GiB	0 errors
6	Tensor-core correctness	`cmp-tensor-test`, `cmp-sm-test`, or club-170hx `tensor-gate.sh`	bit-identical to the reference and to other cards
7	Throughput	mmapeak or torch-performance-test-data	BF16/FP16 in the expected range
8	Real workload	`cmp-ai-test` (llama.cpp) or a model that fills the upper VRAM	correct answer, no NaN, weights read back unchanged
9	Warm rerun and kernel log	repeat 3 to 6 without a reboot; `cmp-kernel-check`	still 0 errors, 0 Xid

Memory

memtest_vulkan (GpuZelenograd/memtest_vulkan): run its standard 5-minute test and press Ctrl+C after the verdict, or it keeps running and holds the VRAM. Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09. If it fails with loader_scanned_icd_add: Could not get vkCreateInstance, the NVIDIA driver was installed with --no-install-libglvnd; reinstall it with libglvnd. Source: @alanoo, 2026-10-05
Source: @alanoo, 2026-10-05
cmp-hbm-test [min] fills all free VRAM, alternating a light walking-bit pattern with full-bandwidth random data, and checks every word. 5 minutes is enough. On the bad card errors appeared in the first second of every run, with counts from 128 to 670,000 per 10 minutes, so judge pass or fail, not counts. Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09.
club-170hx: a cross-window smoke test of at least 16 GiB, then a near-full PRNG write, read and verify (nvcc -O3 -arch=sm_80 scripts/qc/compare_vram.cu, then compare-vram --device 0 --gib 62). Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20.

Compute burn

gpu-burn (wilicc/gpu-burn) with tensor cores: CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=<index> ./gpu_burn -tc -m 95% 300. It lists every GPU at start-up; Initialized device 0 with ~64910 MB shows it picked an unlocked 8 GB card. The bad card above showed 0.6 to 1.9 billion errors on the first line and FAULTY. Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09. A 10-minute variant, ./gpu_burn -tc -m 90% 600, was suggested in the Discord. Source: @trustinter, 2026-10-08
gpu-burn alone does not check that the tensor cores compute the right answer. Source: @alanoo, 2026-10-08 Its TFLOPS figure on this card, about 83, is TF32 throughput and normal. Source: @jimtjames, 2026-10-11

Tensor-core correctness

Run the same calculations on several cards and compare the results. Source: @_____josh_____, 2026-10-08
cmp-tensor-test [s] runs INT8 and FP16 tensor-core matmuls on every SM against an exact reference; cmp-sm-test runs a fixed workload on every SM and bit-compares it with a baseline recorded on a known-good card (driver 610.57.04, CUDA 12.8). If it fails on every card including good ones, a different compiler changed the FP32 results; record your own baseline on a good card with bin/sm-verify record <file> and set CMP_SM_BASELINE. Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09.
club-170hx scripts/qc/tensor-gate.sh 0 1 2 3: 10 minutes of gpu-burn -tc, then same-seed BF16, FP16, TF32 and INT8 4096x4096 matmuls compared with a CPU float64 reference and with every other card, then a full-VRAM memtest_vulkan pass and a PCIe replay and AER snapshot. A 30-second temperature sampler kills the test at 80 C core or 85 C memory. Identical results across every card are the pass signal, even when a result sits above a format's naive tolerance. It refuses to start while a serving container or any compute process is running. Their cards measured 74.6 to 78.5 TFLOPS on the gate with 0 Xid. Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20.

Throughput

What a healthy card measures, so a low number stands out:

Data type	mmapeak / torch GEMM, community	170tune reference card (8 GB, stock)
BF16 tensor	150 to 200 TFLOPS Source: @jimtjames, 2026-10-11; 177.91 in the torch-performance-test-data table	188 to 193 TFLOPS
FP16 tensor	166.48 (torch table)	159 to 160 TFLOPS
TF32 tensor	about 83 (gpu-burn -tc) Source: @jimtjames, 2026-10-11; 77.09 (torch table)	89 to 92 TFLOPS
FP32, no tensor cores	12.15 (torch table)	12.76 TFLOPS

The torch-performance-test-data row is labelled CMP170HX and was contributed by GitHub user bendy2 under the older v1 protocol. Source: repo github.com/zzc0721/torch-performance-test-data by zzc0721 on GitHub, posted by @jimtjames, 2026-10-11. The 170tune numbers come from one reference card on the 300 W VBIOS. Source: repo github.com/cachenetics/170tune by cachenetics on GitHub, posted by @rh3dai, 2026-10-04. mmapeak measures peak MMA issue rate, so its numbers sit above real GEMM figures. Source: repo github.com/ReinForce-II/mmapeak by ReinForce-II on GitHub, posted by @jimtjames, 2026-10-11.

Real workload

cmp-ai-test runs real llama.cpp inference on the card (CUDA build, a GGUF that fits one card; development used Qwen3.8-27B Q8_0, expected top token " Paris") and reads every weight back from VRAM to compare it with the file. It first checks the model file through the page cache against a direct read, so a fault in non-ECC host RAM is not blamed on the card. Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09.
For MoE serving faults like the all-zero rows above: probe_bang.py (injected tokens with logprob -11.95 show the fault), then swap two devices in CUDA_VISIBLE_DEVICES; the fault follows the physical card. Source: @reala7096, 2026-10-08

Kernel log

cmp-kernel-check lists NVIDIA Xid reports since the first test of the day; the pass rule is 0 Xid. Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09. What individual Xids mean is on Troubleshooting.

Burn-in for tuning

Tuning (undervolt, memory clock) needs longer and hotter gates than acceptance. 170tune's changelog: two clean sweeps once shipped a profile that later returned memory errors; four or more hot sweeps are its gate, and offsets that pass cool sweeps can fault under a long serving soak. Source: repo github.com/cachenetics/170tune by cachenetics on GitHub, posted by @rh3dai, 2026-10-04. One member ran the 170tune stability gate at +250, found the card unstable in practice, and settled on +150. Source: @cibernox, 2026-10-07 See 170tune.

When a card fails

Swap one thing at a time: card, slot or riser, power lead. Power down before moving cables; a warm reboot does not clear every PCIe fault. Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20.
Fixed bad addresses can be worked around. @zombielurker's card showed about 5,000 single-bit and 2- to 8-bit errors in each 5-minute memtest_vulkan run, always at the same addresses (around 30 GB and 61 GB). He had an AI map them to physical memory, and found the driver lets you mark blocks bad at boot. Only four 64 KB blocks were bad; each costs a 2 MB page, so usable memory went from 63.06 GB to 63.05 GB, and memtest_vulkan and gpu-burn then passed with ECC off. Source: @zombielurker, 2026-10-05
Source: @zombielurker, 2026-10-05
Source: @zombielurker, 2026-10-05
Source: @zombielurker, 2026-10-05 With ECC on, only the single-bit errors (about 80 percent of his) would have been corrected, and the rest would have stopped processes. Source: @zombielurker, 2026-10-05 @reala7096 had the same kind of script written; it has to run from every boot. Source: @reala7096, 2026-10-05 @cute_sc2: NVIDIA does this automatically on the A100, "it's just disabled on cmp 170hx", and with too many errors (several GB) the card starts refusing. Source: @cute_sc2, 2026-10-05
Source: @cute_sc2, 2026-10-05
Source: @cute_sc2, 2026-10-05 @analyticeth keeps a 2 MB dummy allocation over a small region that always reads 0. Source: @analyticeth, 2026-10-08
Warning Load-transient faults cannot. cmp-hbm-map holds the same 2 MiB blocks for a whole run and logs which fail. On the bad card 7,652 of 31,774 blocks (24 percent of VRAM) failed within 58 seconds, and 85 percent of errors followed launch timing rather than location. Quarantine only works for repeating errors at fixed addresses. Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09.
A wrong-output card may be fine at a lower clock. Capping that card's SM clock at 1300 MHz removed the all-zero rows at no measurable decode cost. Source: @reala7096, 2026-10-08
Inside a return window, send it back with the logs; see Acceptance checklist.

Evidence to keep

The test suite writes raw output per serial and date under runs/ and per-second telemetry under logs/. Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09. club-170hx lists what a useful record holds: a stable card label, driver, kernel and module path, memory size, pattern or seed, duration and verdict, power limit, peak power and temperatures, PCIe width and generation, Xid, ECC and AER scans before and after, and whether the run was warm or cold. Do not publish hostnames, serial numbers or full UUIDs. Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20.

Gaps

There is no shared benchmark suite yet. @djpandaman asked for one; @boz._. pointed to his test suite, which tests rather than benchmarks, and would like both in one place. Source: @djpandaman, 2026-10-12
Source: @boz._., 2026-10-12
Source: @boz._., 2026-10-12