ImportBot: HBM fault symptoms, card testing, bad-block masking, what the v0.5 software ECC does and does not prove, memory above 40 GB on 10 GB cards, and host RAM errors; from the cmpunlocker Discord (2026-10-04 to 10-28) and community repos (cmp170hx-testsuite, club-170hx, gpcprobe, vllm-170hx-glm5, Consensus-Protocol); credits inline; sources: @analyticeth, @boz._., @reala7096, @_josh_, @c133, @feralzi, @ob88594, @replacedriver47, @matstatman, @darkmaniac7, @trustinter, @alan

2026-10-30T08:46:02Z

HBM fault symptoms, card testing, bad-block masking, what the v0.5 software ECC does and does not prove, memory above 40 GB on 10 GB cards, and host RAM errors; from the cmpunlocker Discord (2026-10-04 to 10-28) and community repos (cmp170hx-testsuite, club-170hx, gpcprobe, vllm-170hx-glm5, Consensus-Protocol); credits inline; sources: @analyticeth, @boz._., @reala7096, @_____josh_____, @c133, @feralzi, @ob88594, @replacedriver47, @matstatman, @darkmaniac7, @trustinter, @alan

← Older revision		Revision as of 08:46, 30 October 2026
Line 11:		Line 11:
	\|-		\|-
	\| Passes a basic Vulkan memory test 9 times out of 10, fails most other tests, inference returns all NaN		\| Passes a basic Vulkan memory test 9 times out of 10, fails most other tests, inference returns all NaN
	\| Bad HBM. One of two cards from the same order; ~~the seller~~ refunded it.		\| Bad HBM. One of two cards from the same Alifond order; refunded through Alibaba.
	\| {{Status\|Reported}}		\| {{Status\|Reported}}
	\| {{Source\|handle=boz._.\|date=2026-10-23\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1563148969498910773}}		\| {{Source\|handle=boz._.\|date=2026-10-23\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1563148969498910773}}
Line 66:		Line 66:
	Limits: masking only works for errors that repeat at fixed addresses. Errors that follow load timing rather than location (the load-transient card above) cannot be masked away.<br />{{Source\|handle=boz._.\|date=2026-10-09\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1558098091574562848\|repo=https://github.com/dev-boz/cmp170hx-testsuite}}		Limits: masking only works for errors that repeat at fixed addresses. Errors that follow load timing rather than location (the load-transient card above) cannot be masked away.<br />{{Source\|handle=boz._.\|date=2026-10-09\|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1558098091574562848\|repo=https://github.com/dev-boz/cmp170hx-testsuite}}

	A ~~longer write~~-~~up by @zombielurker on mapping memtest~~ errors to physical blocks ~~and marking them~~ bad at boot ~~exists~~; ~~see~~ [https://discord.com/channels/1518634379604136178/1518692084729511967/1556530076105838723 Discord].		=== Marking bad blocks at boot ===
			A card that threw about 5,000 single-bit and 2-8 bit errors in a 5-minute memtest_vulkan run, always at the same addresses, turned out to have only four bad 64 KB blocks. Mapping the error addresses to physical memory and having the driver mark those blocks bad at boot cost 8 MB (each block is held with 2 MB of pages; 63.06 GB to 63.05 GB usable), and memtest_vulkan and gpu-burn then passed with ECC off. Two other members use the same kind of script; it has to run from every boot. With too many errors (several GB) the card refuses. Full write-up on [https://discord.com/channels/1518634379604136178/1518692084729511967/1556530076105838723 Discord].

			{{Status\|Proven}}<br />
			{{Source\|handle=zombielurker\|date=2026-10-05\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556530076105838723}}<br />
			{{Source\|handle=reala7096\|date=2026-10-05\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556618923640619009}}<br />
			{{Source\|handle=analyticeth\|date=2026-10-05\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556645828498690069}}<br />
			{{Source\|handle=cute_sc2\|date=2026-10-05\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556662244295573645}}

	== ECC after the unlock ==		== ECC after the unlock ==

ImportBot: HBM fault symptoms, card testing, bad-block masking, what the v0.5 software ECC does and does not prove, memory above 40 GB on 10 GB cards, and host RAM errors; from the cmpunlocker Discord (2026-10-04 to 10-28) and community repos (cmp170hx-testsuite, club-170hx, gpcprobe, vllm-170hx-glm5, Consensus-Protocol); credits inline; sources: @analyticeth, @boz._., @reala7096, @_josh_, @c133, @feralzi, @ob88594, @replacedriver47, @matstatman, @darkmaniac7, @trustinter, @alan

2026-10-30T08:17:22Z

New page

This page covers memory faults on the CMP 170HX: what they look like, how to test a card, how people work around bad regions, what the ECC added in cmpunlocker v0.5 does and does not show, and host-side memory problems that look like card faults. Crashes from heat and clocks are on [[Instability under load]]; error numbers are on [[Xid errors]].

== What memory faults look like ==
{| class="wikitable"
! Symptom !! What it turned out to be !! Status !! Source
|-
| A small region of VRAM always reads back 0 and stores nothing
| A bad region on that card. Worked around by allocating a 2 MB throwaway buffer over it at startup so nothing real lands there.
| {{Status|Reported}}
| {{Source|handle=analyticeth|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557566171195383850}}
|-
| Passes a basic Vulkan memory test 9 times out of 10, fails most other tests, inference returns all NaN
| Bad HBM. One of two cards from the same order; the seller refunded it.
| {{Status|Reported}}
| {{Source|handle=boz._.|date=2026-10-23|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1563148969498910773}}
|-
| Whole 128-byte blocks read back as zeros or unrelated data from the first second of every run; 128-670,000 bad words per 10 minutes; gpu-burn reports 0.6-1.9 billion errors; memtest_vulkan fails only right after a cold boot; tensor and SM checks clean; faulty in every slot and on every cable
| A load-transient fault: 85% of errors followed launch timing (the jump from idle to full traffic), not location, pointing at power delivery, PHY or clock margin. The same card crashed its firmware under load (Xid 119, 1, 154 or 62). Masking cannot fix it: 24% of 2 MiB blocks failed within 58 s, and covering 99% of errors would have cost 11.6 GB.
| {{Status|Reported}}
| {{Source|handle=boz._.|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1558098091574562848|repo=https://github.com/dev-boz/cmp170hx-testsuite}}
|-
| About 1 all-zero MoE output row per 3,000 decode steps and occasional garbage top-k indices, at the card's default clock, while GEMM and several Vulkan memory tests pass
| Not memory: capping that card's SM clock at 1300 MHz removed the fault completely with no measurable decode cost. Detect it with <code><nowiki>probe_bang.py</nowiki></code>, which flags a generated token 0 with logprob -11.95.
| {{Status|Reported}}
| {{Source|handle=reala7096|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1557544812972871813|repo=https://github.com/arvkmr/vllm-170hx-glm5}} {{Source|handle=reala7096|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557562632553242705}}
|-
| Tensor cores return gibberish with no errors while memory tests pass
| A faulty card. Run the same calculation on several cards and compare the outputs.
| {{Status|Reported}}
| {{Source|handle=_____josh_____|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557667556746207255}}
|-
| An 8 GB card unlocks to 64 GB but CUDA memory tests fail past 32 GB, while memtest_vulkan passes
| A single-rank card: those unlock to 32 GB only, dual-rank cards to 64 GB. memtest_vulkan passes on a single-rank card faked to 64 GB, so validate with a CUDA memory test.
| {{Status|Reported}}
| {{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}
|-
| An 8 GB card reads 96 GB but crashes or errors as soon as the extra stacks are used
| The extra stacks on that card are dead.
| {{Status|Reported}}
| {{Source|handle=feralzi|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561028970458251265}}
|-
| Hundreds of memory errors on a card bought cheap
| Cheap cards sold as "8 GB" are often the ones that failed the unlock; HBM errors, GSP that will not initialize and engineering-sample boards are the known failure types.
| {{Status|Warning}}
| {{Source|handle=ob88594|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1559444362893594626}} {{Source|handle=replacedriver47|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1559469519016042608}}
|-
| The unlock reports a healthy 64 GB, but the card has a dead HBM stack
| Two of eight cards in one box had one dead HBM stack each, visible only in the fuse map (FBP0/FBPA0 fused off). Those were the two cards that HBM clock tools later stopped from booting.
| {{Status|Proven}}
| {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850|repo=https://github.com/matstatman/gpcprobe}} {{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}
|}

== Testing a card ==
* A community test suite gives an order that takes about 40 minutes per card: test one card at a time, cap it with <code><nowiki>nvidia-smi -i <idx> -pl 180</nowiki></code> (the cap resets at each boot), run the blower at full, then check identity (exactly 65536 MiB on an 8 GB card or 40960 MiB on a 10 GB card, with no Xid since boot), memtest_vulkan for 5 minutes right after a cold boot, <code><nowiki>CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=<idx> ./gpu_burn -tc -m 95% 300</nowiki></code>, then the suite's HBM, tensor, SM and model tests. {{Source|handle=boz._.|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1558098091574562848|repo=https://github.com/dev-boz/cmp170hx-testsuite}}
* gpu-burn alone is too basic; compare identical calculations across cards. {{Source|handle=trustinter|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557709761032495206}} {{Source|handle=_____josh_____|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557730223137751122}} {{Source|handle=alanoo|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557730536745607179}}
* Validate an unlock with a CUDA-side memory test, not only memtest_vulkan. {{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}
* Read the fuse map with gpcprobe (read-only) before running any tool that touches the FBPA PLL registers; a healthy-looking 64 GB unlock can hide a dead stack. {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850|repo=https://github.com/matstatman/gpcprobe}} {{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}
* '''memtest_vulkan traps.''' It has no device-select flag: the device is chosen at an interactive prompt that silently ignores piped input, so a batch script can test the first card four times. A killed memtest_vulkan leaves a detached child that keeps VRAM pinned, which then shows up as "failed determining memory budget" and looks like a hardware fault. If memtest_vulkan fails with a Vulkan loader ICD error, the driver was installed with <code><nowiki>--no-install-libglvnd</nowiki></code>. {{Source|handle=alanoo|date=2026-10-05|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556651013161877595}} {{Source|handle=c133|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127|repo=https://github.com/PixelML/club-170hx}}
* '''Read the card's history.''' Every 170HX keeps an InfoROM "black box": boot and power-on-hours journal, power-cap history, and a log of past Xid events with timestamps. <code><nowiki>blackbox.py</nowiki></code> from gpcprobe dumps it; feed the dump to an LLM to interpret it. It kept one member from buying bad cards: only one card of the batch was good. {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850|repo=https://github.com/matstatman/gpcprobe}} {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850}} {{Source|handle=globalapathy|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562131178524250145}}
* A seller's gpu-burn <code><nowiki>-tc</nowiki></code> result of about 83 TFLOPS is TF32-class tensor throughput, not a fault; BF16/FP16 gives 150-200 TFLOPS. Ask for MMAPeak or a torch GEMM benchmark instead. {{Source|handle=jimtjames|date=2026-10-11|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1558871079395663902}}
* Run memory and inference tests before the return window closes. {{Source|handle=boz._.|date=2026-10-23|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1563148969498910773}}

== Working around bad memory ==
HBM sits on the GPU package, so a bad memory chip cannot be swapped like on GDDR cards. Small error regions can be masked in software instead; one developer has not seen a memory issue that software masking could not fix. Quarantine fixed one card at a cost of 440 MB (7,040 bad blocks in 2 regions), and the card then ran fine. A stuck-at-zero region was covered with a 2 MB dummy allocation at startup. {{Status|Proven}} {{Source|handle=analyticeth|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557566171195383850}} {{Source|handle=_orrorin|date=2026-10-14|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560007100057395303}} {{Source|handle=djpandaman|date=2026-10-14|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560007391137759384}} {{Source|handle=froggao|date=2026-10-14|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1560007422238793800}}

Limits: masking only works for errors that repeat at fixed addresses. Errors that follow load timing rather than location (the load-transient card above) cannot be masked away. {{Source|handle=boz._.|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1558098091574562848|repo=https://github.com/dev-boz/cmp170hx-testsuite}}

A longer write-up by @zombielurker on mapping memtest errors to physical blocks and marking them bad at boot exists; see [https://discord.com/channels/1518634379604136178/1518692084729511967/1556530076105838723 Discord].

== ECC after the unlock ==
* cmpunlocker v0.5, released 2026-10-04, added software-enabled ECC for DRAM and SRAM on both variants. The Morrowmake fork removes the ECC patches. {{Source|handle=_j3ns|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556439155360407583}} {{Source|handle=68x0|date=2026-10-04|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556442901095907450}} {{Source|handle=trustinter|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557381633521492011|repo=https://github.com/Morrowmake/cmpunlocker}} {{Source|handle=rh3dai|date=2026-10-08|link=https://discord.com/channels/1518634379604136178/1555480078089986138/1557643237282553956|repo=https://github.com/amoghmunikote/cmpunlocker}}
* {{Status|Warning}} '''"ECC Enabled, 0 errors" proves nothing.''' cmpunlocker's ECC is three patches. <code><nowiki>ecc-reporting.patch</nowiki></code> replaces the driver's answer to five controls that nvidia-smi and NVML use for ECC status: ECC configuration "Enabled", all units enabled with counters zeroed, the unrepairable-memory flag 0. A plain nvidia-smi showing <code><nowiki>Volatile Uncorr. ECC 0</nowiki></code> instead of <code><nowiki>N/A</nowiki></code> (stock) is that replaced answer. An early version applied it to every GPU in the system, including an RTX 2080 Ti; a later commit limited it to CMP device IDs. The volatile error counts and repair status are not replaced: they come from GSP. {{Source|handle=alanoo|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1559518544906158082}} {{Source|handle=darkmaniac7|date=2026-10-19|note=lab notes}} {{Source|repo=https://github.com/amoghmunikote/cmpunlocker}}
* One owner saw the ECC counter go up on an SRAM error in NVIDIA's Prometheus exporter. {{Source|handle=ichundes|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1559520707245776987}}
* '''The hardware side does change.''' With only the hardware half of the ECC patch on one 8 GB card (three other cards as controls), all 14 protected writes landed, <code><nowiki>FEATURE_READOUT</nowiki></code> (<code><nowiki>0x823814</nowiki></code>) went from <code><nowiki>0x00000233</nowiki></code> to <code><nowiki>0xec8ff233</nowiki></code> (DRAM ECC bit 16 set), <code><nowiki>FBPA_ECC_CTRL</nowiki></code> (<code><nowiki>0x9a0470</nowiki></code>) from 0 to <code><nowiki>0x41</nowiki></code>, and a 60 GiB write/verify passed with ECC on; this matches the upstream author's own read-backs. NVML still reports ECC unsupported (the <code><nowiki>OPT_ECC_EN</nowiki></code> fuse is 0 and there is no InfoROM ECC object), so there are no driver-side counters, no scrubbing and no page retirement. {{Status|Proven}} {{Source|handle=darkmaniac7|date=2026-10-19}} (lab notes, [https://github.com/darkmaniac7/cmp170hx-esxi darkmaniac7/cmp170hx-esxi]) {{Source|repo=https://github.com/amoghmunikote/cmpunlocker}}
* The Morrowmake <code><nowiki>p2p-ecc</nowiki></code> branch combines P2P and the three ECC patches on 615.71.09: ECC Enabled on all 8 cards with P2P still passing, remapped-row counters <code><nowiki>N/A</nowiki></code>. {{Source|handle=trustinter|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1557381633521492011|repo=https://github.com/Morrowmake/cmpunlocker}} {{Source|handle=darkmaniac7|date=2026-10-27|note=lab notes}}
* '''Does it correct DRAM errors?''' (nobody has shown a corrected error yet) {{Status|Contested}}
** Is "enabled" only a reporting artifact of the unlock? {{Source|handle=endsjeremy|date=2026-10-05|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556485503514054740}}
** On a card with about 5,000 single-bit and 2-to-6-bit errors in a 5-minute memtest_vulkan run, ECC showed as enabled but corrected none, and gpu-burn behaved the same. The same owner's RTX 3090s also showed ECC as enabled after the install. {{Source|handle=zombielurker|date=2026-10-05|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556515369286504489}}
** SRAM ECC faults do surface as Xid 48 and Xid 172 with ECC on, and the counters move. {{Source|handle=ichundes|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557350219111342183}}
** gpcprobe's register reads say the GPU-side DRAM ECC datapath is fused off on this card (<code><nowiki>FUSE_ECC_EN = 0</nowiki></code>, <code><nowiki>FBPA_ECC_CTRL = 0</nowiki></code> on a stock card). {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850|repo=https://github.com/matstatman/gpcprobe}}
** An earlier conclusion that the ECC readout is latched at power-on and cannot change is contradicted by the read-backs above, which move when the writes land after the link retrain. {{Source|handle=rrickk420|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1558127650223030302|repo=https://github.com/Consensus-Protocol/cmp170hx}} {{Source|handle=darkmaniac7|date=2026-10-19}} (lab notes, [https://github.com/darkmaniac7/cmp170hx-esxi darkmaniac7/cmp170hx-esxi])
** Several operators of other forks call the report synthetic and run with ECC off (a <code><nowiki>--no-ecc</nowiki></code> switch exists in some forks). {{Source|handle=darkmaniac7|date=2026-10-19|note=lab notes}}
* '''Keep it on while tuning.''' With ECC on, an unstable card crashes the engine instead of silently producing garbage; a crash beats a wrong answer. You can turn it off again if the crashes come back. {{Source|handle=ichundes|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557350680342298647}} {{Source|handle=matstatman|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557352539253510255}} {{Source|handle=zorg33|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1518695289278566450/1557370911013867581}}
* '''Cost.''' ECC showed no effect on bandwidth. Random one-byte writes would suffer, which inference does not do. {{Status|Reported}} One owner saw Qwen 3.8 27B decode fall from 60-80 tok/s on v0.4 to 35 tok/s after reinstalling with v0.5 and suspects ECC; unresolved. {{Source|handle=reala7096|date=2026-10-05|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556767677866049579}} {{Source|handle=matstatman|date=2026-10-05|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556768044410343445}} {{Source|handle=mowgli8628|date=2026-10-18|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561450305529581640}} {{Source|handle=feralzi|date=2026-10-19|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561558221217472513}}
* '''How to test real correction.''' Find a repeatable single-bit error at a fixed address with ECC off, then run the same cell with only the hardware recipe (no reporting patch): real SECDED makes the error disappear from user data. One such repeatable single-bit cell on an 8 GB card has been documented, and a fencing tool that maps bad 64 KB HBM frames exists. Ignore the per-FBPA "DED counter" addresses in circulation: on 8 GB cards they return PRI errors or jitter with no GPU work. {{Status|Theory}} {{Source|handle=darkmaniac7|date=2026-10-19}} (lab notes, [https://github.com/darkmaniac7/cmp170hx-esxi darkmaniac7/cmp170hx-esxi]) {{Source|repo=https://github.com/ZombieLurker/cmpunlocker}} {{Source|repo=https://github.com/satspace-cpu/cmp170hx-linux-p2p}}
* Installing v0.5 with ECC failed on driver 615.78.08 and worked on 615.71.09 (reported by two members). {{Status|Reported}} {{Source|handle=catchyuser|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561975918426198069}} {{Source|handle=.henrykrinkle|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561982868312297513}}

== Memory beyond the official geometry ==
The stable unlocked sizes are 64 GB on the 8 GB card and 40 GB on the 10 GB card (48 GB on some 10 GB cards with a good sixth stack); see [[Unlocks]].
* '''Above 40 GB on 10 GB cards''' {{Status|Contested}}
** Relayed community research: memory past 40 GB returns meaningful values with "no folding"; it is simply unstable at any settings. {{Source|handle=wakeupmbrc1|date=2026-10-13|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1559709514553622549}}
** One researcher gets folds (aliasing) in every experiment, with virtual and physical accesses, bypassing L2, even via PRAMIN; on-die caches and an extra offset hide the aliasing unless you write and check enough unique data. {{Source|handle=colo2442|date=2026-10-14|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1559906143373885462}}
* '''80 GB on a 10 GB card.''' Some cards read 80 GB but the memory was unstable. A snapshot of the Consensus-Protocol notes (2026-08-22): an experimental 80 GB configuration reported 81920 MiB and a 77 GiB <code><nowiki>cudaMalloc</nowiki></code> succeeded, but kernels touching more than about 40 GB caused fatal GPU loss independent of the power limit (Xid 154 after CUDA memory tests); <code><nowiki>cuda_memtest</nowiki></code> passed over all 80 GB once right after a reboot and failed on every retry. {{Status|Reported}} {{Source|handle=rrickk420|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1558127650223030302|repo=https://github.com/Consensus-Protocol/cmp170hx}} {{Source|handle=tekkenz|date=2026-10-12|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1559048743947341955}}

== HBM temperature ==
Heat damages HBM before it hurts the SMs, but posters disagree on the exact limits (80, 85 or 95 C). Memory overclocks that are clean when cool corrupt once HBM passes about 75 C. Details and sources on [[Instability under load#Heat|Instability under load]]. {{Status|Theory}} "Defective" HBM stacks may simply be stacks that run too hot. {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562173369430974556}}

== Host memory errors ==
* On a Supermicro H12D-8D, 8x DDR4-2133 4DRx4 32 GB LRDIMMs gave about 120 corrected ECC errors per minute even though they pass memtest. Running with <code><nowiki>mce=off</nowiki></code> hides them but also hides uncorrectable errors. 8x DDR4-3200 2Rx4 64 GB RDIMMs (512 GB) are clean on the same board. LRDIMM compatibility problems are also reported on Huananzhi EPYC boards. {{Status|Reported}} {{Source|handle=zingy90210|date=2026-10-25|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1563805266825322497}} {{Source|handle=fafinet|date=2026-10-25|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1563852572345765943}} {{Source|handle=endsjeremy|date=2026-10-25|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1563970197482897482}}
* A non-ECC host that passed 48 hours of memtest still had an unexplained page cache corruption. To keep a host RAM fault from being blamed on the card, one test suite reads model weights with O_DIRECT and compares them with the copy in the page cache. {{Status|Reported}} {{Source|handle=boz._.|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1528513735575081070/1558098091574562848|repo=https://github.com/dev-boz/cmp170hx-testsuite}} {{Source|handle=c133|date=2026-10-16|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1560618032307507241}}

== Related pages ==
* [[Xid errors]]
* [[Instability under load]]
* [[Known bad combinations]]
* [[Which card do I have]]
* [[Unlocks]]
* [[Driver unlock]]

{{ImportFooter}}
[[Category:Troubleshooting]]
[[Category:8 GB]]
[[Category:10 GB]]

Memory errors and ECC - Revision history