Memory errors and ECC

This page covers memory faults on the CMP 170HX: what they look like, how to test a card, how people work around bad regions, what the ECC added in cmpunlocker v0.5 does and does not show, and host-side memory problems that look like card faults. Crashes from heat and clocks are on Instability under load; error numbers are on Xid errors.

What memory faults look like

Symptom	What it turned out to be	Status	Source
A small region of VRAM always reads back 0 and stores nothing	A bad region on that card. Worked around by allocating a 2 MB throwaway buffer over it at startup so nothing real lands there.	Reported	Source: @analyticeth, 2026-10-08
Passes a basic Vulkan memory test 9 times out of 10, fails most other tests, inference returns all NaN	Bad HBM. One of two cards from the same Alifond order; refunded through Alibaba.	Reported	Source: @boz._., 2026-10-23
Whole 128-byte blocks read back as zeros or unrelated data from the first second of every run; 128-670,000 bad words per 10 minutes; gpu-burn reports 0.6-1.9 billion errors; memtest_vulkan fails only right after a cold boot; tensor and SM checks clean; faulty in every slot and on every cable	A load-transient fault: 85% of errors followed launch timing (the jump from idle to full traffic), not location, pointing at power delivery, PHY or clock margin. The same card crashed its firmware under load (Xid 119, 1, 154 or 62). Masking cannot fix it: 24% of 2 MiB blocks failed within 58 s, and covering 99% of errors would have cost 11.6 GB.	Reported	Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09
About 1 all-zero MoE output row per 3,000 decode steps and occasional garbage top-k indices, at the card's default clock, while GEMM and several Vulkan memory tests pass	Not memory: capping that card's SM clock at 1300 MHz removed the fault completely with no measurable decode cost. Detect it with `probe_bang.py`, which flags a generated token 0 with logprob -11.95.	Reported	Source: repo github.com/arvkmr/vllm-170hx-glm5 by arvkmr on GitHub, posted by @reala7096, 2026-10-08 Source: @reala7096, 2026-10-08
Tensor cores return gibberish with no errors while memory tests pass	A faulty card. Run the same calculation on several cards and compare the outputs.	Reported	Source: @_____josh_____, 2026-10-08
An 8 GB card unlocks to 64 GB but CUDA memory tests fail past 32 GB, while memtest_vulkan passes	A single-rank card: those unlock to 32 GB only, dual-rank cards to 64 GB. memtest_vulkan passes on a single-rank card faked to 64 GB, so validate with a CUDA memory test.	Reported	Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
An 8 GB card reads 96 GB but crashes or errors as soon as the extra stacks are used	The extra stacks on that card are dead.	Reported	Source: @feralzi, 2026-10-17
Hundreds of memory errors on a card bought cheap	Cheap cards sold as "8 GB" are often the ones that failed the unlock; HBM errors, GSP that will not initialize and engineering-sample boards are the known failure types.	Warning	Source: @ob88594, 2026-10-13 Source: @replacedriver47, 2026-10-13
The unlock reports a healthy 64 GB, but the card has a dead HBM stack	Two of eight cards in one box had one dead HBM stack each, visible only in the fuse map (FBP0/FBPA0 fused off). Those were the two cards that HBM clock tools later stopped from booting.	Proven	Source: repo github.com/matstatman/gpcprobe by matstatman on GitHub, posted by @matstatman, 2026-10-20 Source: @darkmaniac7, 2026-10-27 (lab notes)

Testing a card

A community test suite gives an order that takes about 40 minutes per card: test one card at a time, cap it with nvidia-smi -i <idx> -pl 180 (the cap resets at each boot), run the blower at full, then check identity (exactly 65536 MiB on an 8 GB card or 40960 MiB on a 10 GB card, with no Xid since boot), memtest_vulkan for 5 minutes right after a cold boot, CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=<idx> ./gpu_burn -tc -m 95% 300, then the suite's HBM, tensor, SM and model tests.
Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09
gpu-burn alone is too basic; compare identical calculations across cards.
Source: @trustinter, 2026-10-08
Source: @_____josh_____, 2026-10-08
Source: @alanoo, 2026-10-08
Validate an unlock with a CUDA-side memory test, not only memtest_vulkan.
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
Read the fuse map with gpcprobe (read-only) before running any tool that touches the FBPA PLL registers; a healthy-looking 64 GB unlock can hide a dead stack.
Source: repo github.com/matstatman/gpcprobe by matstatman on GitHub, posted by @matstatman, 2026-10-20
Source: @darkmaniac7, 2026-10-27 (lab notes)
memtest_vulkan traps. It has no device-select flag: the device is chosen at an interactive prompt that silently ignores piped input, so a batch script can test the first card four times. A killed memtest_vulkan leaves a detached child that keeps VRAM pinned, which then shows up as "failed determining memory budget" and looks like a hardware fault. If memtest_vulkan fails with a Vulkan loader ICD error, the driver was installed with --no-install-libglvnd.
Source: @alanoo, 2026-10-05
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
Read the card's history. Every 170HX keeps an InfoROM "black box": boot and power-on-hours journal, power-cap history, and a log of past Xid events with timestamps. blackbox.py from gpcprobe dumps it; feed the dump to an LLM to interpret it. It kept one member from buying bad cards: only one card of the batch was good.
Source: repo github.com/matstatman/gpcprobe by matstatman on GitHub, posted by @matstatman, 2026-10-20
Source: @matstatman, 2026-10-20
Source: @globalapathy, 2026-10-20
A seller's gpu-burn -tc result of about 83 TFLOPS is TF32-class tensor throughput, not a fault; BF16/FP16 gives 150-200 TFLOPS. Ask for MMAPeak or a torch GEMM benchmark instead.
Source: @jimtjames, 2026-10-11
Run memory and inference tests before the return window closes.
Source: @boz._., 2026-10-23

Working around bad memory

HBM sits on the GPU package, so a bad memory chip cannot be swapped like on GDDR cards. Small error regions can be masked in software instead; one developer has not seen a memory issue that software masking could not fix. Quarantine fixed one card at a cost of 440 MB (7,040 bad blocks in 2 regions), and the card then ran fine. A stuck-at-zero region was covered with a 2 MB dummy allocation at startup. Proven
Source: @analyticeth, 2026-10-08
Source: @_orrorin, 2026-10-14
Source: @djpandaman, 2026-10-14
Source: @froggao, 2026-10-14

Limits: masking only works for errors that repeat at fixed addresses. Errors that follow load timing rather than location (the load-transient card above) cannot be masked away.
Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09

Marking bad blocks at boot

A card that threw about 5,000 single-bit and 2-8 bit errors in a 5-minute memtest_vulkan run, always at the same addresses, turned out to have only four bad 64 KB blocks. Mapping the error addresses to physical memory and having the driver mark those blocks bad at boot cost 8 MB (each block is held with 2 MB of pages; 63.06 GB to 63.05 GB usable), and memtest_vulkan and gpu-burn then passed with ECC off. Two other members use the same kind of script; it has to run from every boot. With too many errors (several GB) the card refuses. Full write-up on Discord.

Proven
Source: @zombielurker, 2026-10-05
Source: @reala7096, 2026-10-05
Source: @analyticeth, 2026-10-05
Source: @cute_sc2, 2026-10-05

ECC after the unlock

cmpunlocker v0.5, released 2026-10-04, added software-enabled ECC for DRAM and SRAM on both variants. The Morrowmake fork removes the ECC patches.
Source: @_j3ns, 2026-10-04
Source: @68x0, 2026-10-04
Source: repo github.com/Morrowmake/cmpunlocker by Morrowmake on GitHub, posted by @trustinter, 2026-10-07
Source: repo github.com/amoghmunikote/cmpunlocker by amoghmunikote on GitHub, posted by @rh3dai, 2026-10-08
Warning "ECC Enabled, 0 errors" proves nothing. cmpunlocker's ECC is three patches. ecc-reporting.patch replaces the driver's answer to five controls that nvidia-smi and NVML use for ECC status: ECC configuration "Enabled", all units enabled with counters zeroed, the unrepairable-memory flag 0. A plain nvidia-smi showing Volatile Uncorr. ECC 0 instead of N/A (stock) is that replaced answer. An early version applied it to every GPU in the system, including an RTX 2080 Ti; a later commit limited it to CMP device IDs. The volatile error counts and repair status are not replaced: they come from GSP.
Source: @alanoo, 2026-10-13
Source: @darkmaniac7, 2026-10-19 (lab notes)
Source: repo github.com/amoghmunikote/cmpunlocker by amoghmunikote on GitHub
One owner saw the ECC counter go up on an SRAM error in NVIDIA's Prometheus exporter.
Source: @ichundes, 2026-10-13
The hardware side does change. With only the hardware half of the ECC patch on one 8 GB card (three other cards as controls), all 14 protected writes landed, FEATURE_READOUT (0x823814) went from 0x00000233 to 0xec8ff233 (DRAM ECC bit 16 set), FBPA_ECC_CTRL (0x9a0470) from 0 to 0x41, and a 60 GiB write/verify passed with ECC on; this matches the upstream author's own read-backs. NVML still reports ECC unsupported (the OPT_ECC_EN fuse is 0 and there is no InfoROM ECC object), so there are no driver-side counters, no scrubbing and no page retirement. Proven
Source: @darkmaniac7, 2026-10-19 (lab notes, darkmaniac7/cmp170hx-esxi)
Source: repo github.com/amoghmunikote/cmpunlocker by amoghmunikote on GitHub
The Morrowmake p2p-ecc branch combines P2P and the three ECC patches on 615.71.09: ECC Enabled on all 8 cards with P2P still passing, remapped-row counters N/A.
Source: repo github.com/Morrowmake/cmpunlocker by Morrowmake on GitHub, posted by @trustinter, 2026-10-07
Source: @darkmaniac7, 2026-10-27 (lab notes)
Does it correct DRAM errors? (nobody has shown a corrected error yet) Contested
- Is "enabled" only a reporting artifact of the unlock?
  Source: @endsjeremy, 2026-10-05
- On a card with about 5,000 single-bit and 2-to-6-bit errors in a 5-minute memtest_vulkan run, ECC showed as enabled but corrected none, and gpu-burn behaved the same. The same owner's RTX 3090s also showed ECC as enabled after the install.
  Source: @zombielurker, 2026-10-05
- SRAM ECC faults do surface as Xid 48 and Xid 172 with ECC on, and the counters move.
  Source: @ichundes, 2026-10-07
- gpcprobe's register reads say the GPU-side DRAM ECC datapath is fused off on this card (FUSE_ECC_EN = 0, FBPA_ECC_CTRL = 0 on a stock card).
  Source: repo github.com/matstatman/gpcprobe by matstatman on GitHub, posted by @matstatman, 2026-10-20
- An earlier conclusion that the ECC readout is latched at power-on and cannot change is contradicted by the read-backs above, which move when the writes land after the link retrain.
  Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
  Source: @darkmaniac7, 2026-10-19 (lab notes, darkmaniac7/cmp170hx-esxi)
- Several operators of other forks call the report synthetic and run with ECC off (a --no-ecc switch exists in some forks).
  Source: @darkmaniac7, 2026-10-19 (lab notes)
Keep it on while tuning. With ECC on, an unstable card crashes the engine instead of silently producing garbage; a crash beats a wrong answer. You can turn it off again if the crashes come back.
Source: @ichundes, 2026-10-07
Source: @matstatman, 2026-10-07
Source: @zorg33, 2026-10-07
Cost. ECC showed no effect on bandwidth. Random one-byte writes would suffer, which inference does not do. Reported One owner saw Qwen 3.8 27B decode fall from 60-80 tok/s on v0.4 to 35 tok/s after reinstalling with v0.5 and suspects ECC; unresolved.
Source: @reala7096, 2026-10-05
Source: @matstatman, 2026-10-05
Source: @mowgli8628, 2026-10-18
Source: @feralzi, 2026-10-19
How to test real correction. Find a repeatable single-bit error at a fixed address with ECC off, then run the same cell with only the hardware recipe (no reporting patch): real SECDED makes the error disappear from user data. One such repeatable single-bit cell on an 8 GB card has been documented, and a fencing tool that maps bad 64 KB HBM frames exists. Ignore the per-FBPA "DED counter" addresses in circulation: on 8 GB cards they return PRI errors or jitter with no GPU work. Theory
Source: @darkmaniac7, 2026-10-19 (lab notes, darkmaniac7/cmp170hx-esxi)
Source: repo github.com/ZombieLurker/cmpunlocker by ZombieLurker on GitHub
Source: repo github.com/satspace-cpu/cmp170hx-linux-p2p by satspace-cpu on GitHub
Installing v0.5 with ECC failed on driver 615.78.08 and worked on 615.71.09 (reported by two members). Reported
Source: @catchyuser, 2026-10-20
Source: @.henrykrinkle, 2026-10-20

Memory beyond the official geometry

The stable unlocked sizes are 64 GB on the 8 GB card and 40 GB on the 10 GB card (48 GB on some 10 GB cards with a good sixth stack); see Unlocks.

Above 40 GB on 10 GB cards Contested
- Relayed community research: memory past 40 GB returns meaningful values with "no folding"; it is simply unstable at any settings.
  Source: @wakeupmbrc1, 2026-10-13
- One researcher gets folds (aliasing) in every experiment, with virtual and physical accesses, bypassing L2, even via PRAMIN; on-die caches and an extra offset hide the aliasing unless you write and check enough unique data.
  Source: @colo2442, 2026-10-14
80 GB on a 10 GB card. Some cards read 80 GB but the memory was unstable. A snapshot of the Consensus-Protocol notes (2026-08-22): an experimental 80 GB configuration reported 81920 MiB and a 77 GiB cudaMalloc succeeded, but kernels touching more than about 40 GB caused fatal GPU loss independent of the power limit (Xid 154 after CUDA memory tests); cuda_memtest passed over all 80 GB once right after a reboot and failed on every retry. Reported
Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
Source: @tekkenz, 2026-10-12

HBM temperature

Heat damages HBM before it hurts the SMs, but posters disagree on the exact limits (80, 85 or 95 C). Memory overclocks that are clean when cool corrupt once HBM passes about 75 C. Details and sources on Instability under load. Theory "Defective" HBM stacks may simply be stacks that run too hot.
Source: @matstatman, 2026-10-20

Host memory errors

On a Supermicro H12D-8D, 8x DDR4-2133 4DRx4 32 GB LRDIMMs gave about 120 corrected ECC errors per minute even though they pass memtest. Running with mce=off hides them but also hides uncorrectable errors. 8x DDR4-3200 2Rx4 64 GB RDIMMs (512 GB) are clean on the same board. LRDIMM compatibility problems are also reported on Huananzhi EPYC boards. Reported
Source: @zingy90210, 2026-10-25
Source: @fafinet, 2026-10-25
Source: @endsjeremy, 2026-10-25
A non-ECC host that passed 48 hours of memtest still had an unexplained page cache corruption. To keep a host RAM fault from being blamed on the card, one test suite reads model weights with O_DIRECT and compares them with the copy in the page cache. Reported
Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09
Source: @c133, 2026-10-16

Related pages