Instability under load

This page covers a CMP 170HX that works at idle but crashes, drops off the bus, throttles or returns wrong output under load. Most reports trace to one of five things: power delivery, heat, too much undervolt, too high an HBM clock, or (on 10 GB cards) unlocked SMs that are not good. Some "bad output" is an engine bug, not the card; that is at the end of the page.

Quick triage

Find the card. On a multi-card host, swap two devices in CUDA_VISIBLE_DEVICES. If the fault follows the physical card, it is the card; if it stays with the pipeline stage, it is not. When several cards crash at random, suspect the weakest one first.
Source: @cibernox, 2026-10-06
Source: repo github.com/arvkmr/vllm-170hx-glm5 by arvkmr on GitHub, posted by @reala7096, 2026-10-08
Lower the power cap. sudo nvidia-smi -i 0 -pl 200 (or 180). Several dropout problems below went away at 180 W. The cap goes back to 250 W on every driver load, including a module reload or a VM reboot; re-apply it with a systemd unit after the driver is up. Caps and a live HBM clock setting also reset whenever the vLLM engine starts or restarts, so apply them from a post-start hook.
Source: @robbbbbb7973, 2026-10-15
Source: @xeax, 2026-10-15
Source: @xanthefin, 2026-10-15
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
Source: @darkmaniac7, 2026-10-25 (lab notes)
Check temperatures. nvidia-smi -q -d TEMPERATURE. Throttling starts around 76 C; keep core and HBM under 75 C.
Source: @xeax, 2026-10-15
Source: @zorg33, 2026-10-20
Remove the tuning. Put the SM clock offset and the HBM divider (NDIV) back to stock, then reintroduce them one at a time.
Keep ECC on so faults show up as Xids instead of silent garbage. See Xid errors.
Source: @ichundes, 2026-10-07
Source: @zorg33, 2026-10-07
Soak with a real workload. Short stability tests pass settings that later fail: offsets that passed 170tune's stability and workload tests still threw Xid 48 or produced gibberish under real load.
Source: @ichundes, 2026-10-07
Source: @cibernox, 2026-10-07

Power

What happened	Setup	Fix or finding	Status	Source
Every CMP dropped off PCIe almost at once on large prompts, with PCIe lane errors and IOMMU errors. Small prompts were fine.	5 cards, individually stable, on an external GPU switch baseboard; mixed host with 6 RTX 3090s; cap mod done. Temperatures and PSU draw within limits; the logs showed the first card drawing 260 W.	Power limit lowered from 250 W to 180 W (chunk size had also been reduced). GLM-5.3-Flash then ran at 380k context with no issues.	Reported	Source: @moliminous, 2026-10-19 Source: @gnasherrr, 2026-10-19
Under sustained load, mostly big prompt-processing jobs, all four cards and the PLX board lost power at once and needed 10 s unplugged before working again.	4x 8 GB on a PLX switch board; two 750 W PSUs, each feeding two cards (one 8-pin CPU plus two 8-pin PCIe per GPU); 24-pin joined with a Y adapter to an always-on plug; cap 175 W; cards under 60 C.	Not resolved in the thread. Hypotheses: one PSU trips on a transient and takes the switch board down, since the PSUs are independent; or the PLX chip overheats under its stock cooler. A test at 150 W was started, result not posted.	Reported	Source: @replacedriver47, 2026-10-05 Source: @reala7096, 2026-10-05 Source: @zombielurker, 2026-10-05
All cards dropped out together about once a day; the motherboard on its own PSU stayed up.	9 cards on three 750 W PSUs, cap 160 W, cards at 49-72 C.	Same hypotheses as the row above.	Reported	Source: @reala7096, 2026-10-05
One card fell off the bus (Xid 79) three times under sustained load.	Stock 250 W limit.	Stable at `-pl 180`, re-applied at boot by a systemd unit.	Reported	Source: repo github.com/Zanooda/deepseek-v4.1-flash-cmp170hx by Zanooda on GitHub, posted by @gnasherrr, 2026-10-17

Transients. With a 250 W limit, one owner saw instantaneous spikes to 380 W during inference. Another saw nvidia-smi report up to 333 W against a 250 W limit during prefill on every card. Ten cards capped at 160 W peaked near 2000 W together. The cap is enforced as an average: at a 180 W cap, single samples reached 214.6 W. Size PSUs and cables for the spikes, not the cap; see Power and cabling warnings.
Source: @n2dcrypt, 2026-10-05
Source: repo github.com/Zanooda/deepseek-v4.1-flash-cmp170hx by Zanooda on GitHub, posted by @gnasherrr, 2026-10-17
Source: @darkmaniac7, 2026-10-19 (lab notes)
Source: @reala7096, 2026-10-20
10 GB cards with more than 80 SMs. The cards' self-reported power becomes unreliable: readings above 400 W on a 250 W VBIOS. The problem starts as soon as an SM in a previously disabled GPC is enabled (above 80 SMs, not 90); the developer says he resolved it but did not post the fix. Another owner sees power limiting without the card drawing the full 250 W. Reported
Source: @waypoint3642, 2026-10-22
Source: @colo2442, 2026-10-22
Source: @djpandaman, 2026-10-22
Power cap above 80 SMs (10 GB). Past 80 SMs the nvidia-smi power cap triggers, the memory clock is forced down to about 1100 MHz and FP32 throughput drops hard. It starts at 82 SMs, not at 80, and an AI-assisted look at the VBIOS found something that looks like a cap. Clock ceilings seen: about 1300 MHz at 80 SMs, 1050-1100 MHz at 120 SMs; log clocks to CSV to see which one is capped, and try undervolting. A card at 122 SMs hit the cap even locked at 1095 MHz, and was already capping at 94 SMs; there was no speed difference between 108 and 122 SMs. Suggested next steps (untested): undervolt first; a shunt mod, ideally after soldering the unpopulated VRM phases. Those are hardware modifications; see Power and cabling warnings. Reported
Source: @djpandaman, 2026-10-12
Source: @wakeupmbrc1, 2026-10-12
Source: @emogop, 2026-10-12
Warning An unreleased "high power mode" register works even on 8 GB cards; its finder will not release it because he does not know what it does. Defective-marked HBM may simply be HBM that runs too hot.
Source: @matstatman, 2026-10-20

Heat

Warning No airflow, no card. The card has no fan of its own and drops out within minutes even at idle. See Card not detected or BAR allocation fails and Cooling and power.
85 C is the maximum: a card at 85 C with the fan at full is cooling-bound. Lower the limit to 200 W and fix airflow; 1U GPU servers push two 40 mm 15k rpm fans in series per card.
Source: @xeax, 2026-10-15
A modelled estimate: a 250 W cap needs 2-3 times the airflow of 200 W.
Source: @zorg33, 2026-10-20
HBM temperature limits (the numbers differ, but every poster says to keep HBM well under 80-85 C) Contested
- The cards throttle at 85 C.
  Source: @matstatman, 2026-10-09
- Anything above 80 C is too hot.
  Source: @spectakkular, 2026-10-09
- The HBM, not the SMs, is what is at risk from heat.
  Source: @replacedriver47, 2026-10-09
- Memory is not always hotter than the core; HBM over 95 C gives total instability, and a few minutes at 95 C corrupts something in the WPR.
  Source: @bigptoughneigh, 2026-10-09
- HBM has no issues until about 80 C.
  Source: @replacedriver47, 2026-10-10
A memory overclock that is clean when cool corrupts once the card heat-soaks: NDIV 72 corrupts as HBM passes about 75 C, while the same settings gave zero Xids with HBM at or below 76 C. One 4-card host stops tests at 80 C core or 85 C memory.
Source: repo github.com/cachenetics/170tune by cachenetics on GitHub, posted by @rh3dai, 2026-10-04
Source: repo github.com/vektorprime/170tune by vektorprime on GitHub, posted by @rh3dai, 2026-10-04
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
A serving abort rule set that has worked on a 4-card host: stop the job and check for a wedge when any card passes 78 C, a new Xid, "GPU is probably locked" or "fallen off the bus" line appears, a process sits in D state on uvm_*, or timeout 20 nvidia-smi times out (then stop sampling).
Source: @darkmaniac7, 2026-10-19 (lab notes, darkmaniac7/cmp170hx-esxi)
A card that reached 100 C by accident cut its power hard and still works. Do not plan on it. Reported
Source: @sojun80, 2026-10-06
Two channels of a fan controller could not drive 12038 4,000 rpm fans at full speed; check per-channel current limits. Reported
Source: @alanoo, 2026-10-26
Beeping or chirping under load is coil noise, not a dying card; several owners have it on one or two cards, and undervolting may reduce it. If you can hear it over your blower, the blower may not be moving enough air. Reported
Source: @rrickk420, 2026-10-04
Source: @kellertreppe123, 2026-10-04
Source: @cibernox, 2026-10-04
Source: @spectakkular, 2026-10-04

Undervolt and SM clock offset

Undervolting shifts the V/F curve left so the same clock runs at lower voltage; it can cut power by 25-35%. It is per card, and when it is too deep the card falls over with Xid 31. Unlocked SMs or memory that were factory-disabled may not hold the same voltage as the factory-enabled parts. Step the V/F shift in 30 MHz increments.
Source: @wakeupmbrc1, 2026-10-12
Source: @_orrorin, 2026-10-12

What happened	Fix	Status	Source
Xid 48 and Xid 172 (uncorrectable SRAM errors in GPC0), then Xid 13 killed the vLLM worker	Voltage offset reduced by 50	Reported	Source: @ichundes, 2026-10-07
Gibberish output once or twice a day on one card at +250	That card lowered to +150	Reported	Source: @cibernox, 2026-10-07
Repeated GLM 5.3 Flash failures on a 4-card rig	One card tolerated the undervolt worse than the others; easing that card alone gave 90 minutes clean	Reported	Source: @cibernox, 2026-10-06 Source: @voidgiraffe, 2026-10-06
8 GB reference card, 300 W VBIOS, long vLLM serving soak: +300 and +350 passed GEMM and pattern sweeps, then faulted with Xid 13 on weak SMs (GPC3/TPC6). Under a 200 W cap, +280 and +290 threw Xid 13 then Xid 43. +375 faulted at 1400 MHz; +450 hard-crashed and needed a power cycle.	+200 and +250 were clean. Two clean sweeps once shipped a +400 profile that later returned memory errors; four or more hot sweeps are the gate.	Reported	Source: repo github.com/cachenetics/170tune by cachenetics on GitHub, posted by @rh3dai, 2026-10-04 Source: repo github.com/vektorprime/170tune by vektorprime on GitHub, posted by @rh3dai, 2026-10-04
gpu-burn errors at NDIV 73 on stock memory timings	A -100 MHz core offset fixed it; no timing change did	Reported	Source: repo github.com/asm64-hooligan/cmpunlocker by asm64-hooligan on GitHub, posted by @rh3dai, 2026-10-04
Errors at very low idle clocks with the idle p-state patches	Unresolved: one owner runs three cards at 23, 25 and 26 W idle with no errors; the patch author had errors going that low and is not sure if software or hardware caused them	Reported	Source: @robbbbbb7973, 2026-10-10 Source: @cibernox, 2026-10-21

If an offset does nothing, check that your VBIOS exposes the lever: the 8 GB 250 W VBIOS (92.00.67.00.01) and the tested 10 GB VBIOS (92.00.66.00.02) report a GPC V/F offset range of 0 to 0. On four cards with the stock 250 W VBIOS, OneZeroMiner's core offset failed with "Failed to apply Core offset: an internal driver error" on three of them and an NVML offset was rejected the same way in a VM; only the power limit worked.
Source: repo github.com/vektorprime/170tune by vektorprime on GitHub, posted by @rh3dai, 2026-10-04
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
Source: @darkmaniac7, 2026-10-28 (lab notes)

HBM clock (NDIV)

SK Hynix HBM usually runs from stock NDIV 54 to 68-70. On one owner's 4 cards, 70 works, maybe 72, and 74 fails.
Source: @wakeupmbrc1, 2026-10-12
Source: @alanoo, 2026-10-14
Bench data from 170tune: NDIV 72 corrupts as the card heats; 74-76 crash on the first real serving load; 77 is clean only at or below about 61 C; a bare 77 hard-hangs the memory controller. nvidia-smi reports 1728 MHz at every NDIV, so do not read the memory clock there.
Source: repo github.com/vektorprime/170tune by vektorprime on GitHub, posted by @rh3dai, 2026-10-04
Warning HBM clock tools that write the FBPA0 PLL registers stopped two cards with FBPA0 fused off from booting. Read the fuse map before tuning; see the warning.
Source: @darkmaniac7, 2026-10-27 (lab notes)
Warning Never change NDIV while a CUDA context is open: the GPU wedges with Xid 45 and Xid 119 and needs a power cycle.
Source: repo github.com/cachenetics/170tune by cachenetics on GitHub, posted by @rh3dai, 2026-10-04
On one GLM host, NDIV 70 corrupted output (no ECC) and was rejected.
Source: repo github.com/arvkmr/vllm-170hx-glm5 by arvkmr on GitHub, posted by @reala7096, 2026-10-08
N/12 FBPAs failed to lock in dmesg means NDIV is too high. Setting NDIV below stock is the documented way to stabilise a card that fails at stock.
Source: repo github.com/asm64-hooligan/cmpunlocker by asm64-hooligan on GitHub, posted by @rh3dai, 2026-10-04
Can NDIV change at runtime? Contested
- The 170tune idle gate switches between an idle NDIV (30) and a busy NDIV (64) at runtime.
  Source: @robbbbbb7973, 2026-10-11
- One owner's agent concluded NDIV can only be set at boot.
  Source: @markshowa, 2026-10-26

10 GB cards with extra SMs enabled

With all 128 SMs enabled on a card whose extra GPCs were all marked defective, a simple int8 workload ran at 250 W and then crashed with Xid 109; the miner did not compute at all. Disabling GPC1 fixed it. Method: disable the defective-marked GPCs one by one until stable, then re-enable earlier ones to find the boundary. Reported
Source: @lxtend, 2026-10-17
Source: @wakeupmbrc1, 2026-10-17
Unlocking the SMs does not make them good: some or even all of the extra SMs on a given card may be defective.
Source: @xeax, 2026-10-18
Early SM-unlock attempts made the card crash itself until OPT_TPC_DISABLE was overridden. Reported
Source: @lxtend, 2026-10-12
Are "defective" units really defective? Contested
- Everything found marked defective has proven truly defective; disabled units can be enabled.
  Source: @djpandaman, 2026-10-17
- GPCs and FBPAs marked defective are defective; only enable disabled ones.
  Source: @zorg33, 2026-10-22
- One owner enabled defective GPCs, had an agent A/B-test SM pairs, and runs a card with 126 working SMs where a defective-marked GPC had only one bad SM pair. Enabling smaller parts of defective FBPs did not work.
  Source: @colo2442, 2026-10-22
- You can enable defective GPCs as long as you mask the bad TPCs inside each.
  Source: @djpandaman, 2026-10-22
On an 8 GB card the extra memory reads as present (96 GB) but the card crashes or errors when the dead stacks are actually used. Reported
Source: @feralzi, 2026-10-17

Wrong output from an engine bug, not the card

Symptom	Cause	Fix	Status	Source
GLM-5.3-Flash goes off the rails mid-reasoning: an endless run of `=`, nonsense or Chinese text, on stock clocks with ECC on. Restarting vLLM and changing the quant did not help.	Morrowmake GLM recipe 1.7.0 and 1.7.1. The 1.7.2 notes list a TP4 lost-context and endless-repetition bug when a speculative step ended exactly on a 1152-token block boundary, GPU memory growth over hours from prefill scratch buffers, and garbage after a non-local cached prefix.	Fixed in 1.7.2. Pinning 1.6.0 also avoided it.	Proven	Source: repo github.com/Morrowmake/glm53-flash-cmp170hx-recipe by Morrowmake on GitHub, posted by @ob88594, 2026-10-06 Source: @cmdrwaka324, 2026-10-08 Source: @n2dcrypt, 2026-10-08 Source: @okrasit., 2026-10-08
Repeated symbols (a check mark, "!") during reasoning on Qwen 3.8 Flash Next	Suspected upstream engine bug	Restart and resume	Reported	Source: @killred, 2026-10-08 Source: @68x0, 2026-10-08
Memory leak in a prefill step of the GLM 5.3 Flash runtime	Bugs in the release before 2026-10-06	Fixed in the 2026-10-06 release	Reported	Source: @spectakkular, 2026-10-06 Source: @voidgiraffe, 2026-10-06
"duct duct duct" (token 1023) from all-NaN logits after a few dozen 8-way requests	vLLM with PP + MTP + prefix caching + mamba alignment	The repo's prefix-caching patch (P0-0010, index by request not batch)	Reported	Source: repo github.com/vektorprime/qwen38-flash-next-pp2 by vektorprime on GitHub, posted by @vektor_prime, 2026-10-07
DeepSeek V4.1 Flash acts randomly near 200k context (computer use flips through pages); `/clear` restores it	Usually a KV cache issue; the same garbage appeared with an EXL3 GLM at long context	Compaction in the harness helps, but it is a band-aid for an inference bug	Reported	Source: @con3po7016, 2026-10-26 Source: @68x0, 2026-10-26 Source: @segmond, 2026-10-27
GLM 5.3 repeats itself in a coding harness	Often the harness	Try another harness; omp and DSH were named	Reported	Source: @sojun80, 2026-10-20 Source: @globalapathy, 2026-10-20
One request's prefill stalls another request's decode	Open question on GLM recipe 1.6.0	None posted	Reported	Source: @cmdrwaka324, 2026-10-05
vLLM card crash at `gpu-memory-utilization 0.95`	The unlocked geometry exposes 65052 MB, of which 64733 MB is actually available	Use 0.90	Reported	Source: repo github.com/Consensus-Protocol/cmp170hx by Consensus-Protocol on GitHub, posted by @rrickk420, 2026-10-09
A worker dies loading weights at about 41 of 63.5 GiB with 22 GiB "free"	`PYTORCH_ALLOC_CONF` spelled wrong is silently ignored, so the allocator fragments	Use the correct variable name	Reported	Source: repo github.com/vektorprime/qwen38-flash-next-pp2 by vektorprime on GitHub, posted by @vektor_prime, 2026-10-07
Crash with FULL_AND_PIECEWISE CUDA graphs ("crash on Ampere")	NCCL switching algorithms under graph replay, not Ampere	`NCCL_ALGO=Ring NCCL_PROTO=Simple`, preferably `--disable-custom-all-reduce`	Reported	Source: repo github.com/wtdcode/vllm-backport by wtdcode on GitHub, posted by @matstatman, 2026-10-16
Model load crashes (null dereference in `nvfs_get_p2p_dma_mapping`) or hangs at 0% with GDS loaders	A race in nvidia-fs mapping bookkeeping, wider on slow P2P links such as an unlocked CMP at low PCIe width	Bounded staging (`--load-format gds_bounded`)	Reported	Source: repo github.com/cachenetics/gds-nvme-patch by cachenetics on GitHub, posted by @mountains1605, 2026-10-20
All workers hang at init (0% GPU, 12-20 minutes and more) with Morrowmake's PCIe P2P custom all-reduce	A first compile of its torch extension was killed and left stale lock files that every later load waits on (also copied along with a cache directory)	Delete the stale locks; fixed in Morrowmake vllm-cmp170hx commit f5ba3755. Give each instance its own compile cache. On Gen2 at TP4 the custom all-reduce then cost about 40% decode for +13% prefill, so that host runs NCCL only; for TP2 Qwen it helps, so keep it on there.	Proven	Source: repo github.com/Morrowmake/vllm-cmp170hx by Morrowmake on GitHub, posted by @voidgiraffe, 2026-10-15 Source: @darkmaniac7, 2026-10-28 (lab notes)
A 753B GLM-5.3 run at TP2 x PP4 degenerates into 32,000 tokens of repeated Chinese shop-page text, long-context recall drops	Pipeline hand-off bug in the DeepSeek-V3.2-style model code of the vllm-backport engine: a non-last stage sends an un-reduced partial sum through a transport meant for replicated tensors (same bug as vLLM PR #60477). PP8 and TP8 are unaffected.	A one-line worker fix (`all_gather_tensors = {"hidden_states": False}`); top-1 agreement went from 87.0% to 90.2%, recall from 21/27 to 26/27	Proven	Source: @darkmaniac7, 2026-10-28 (lab notes)
HTTP 500 on every request that constrains output to a schema (forced `tool_choice`, `json_schema`) while MTP is on	The grammar backend rejects the speculative draft tokens on that vllm-backport build (log: "grammar rejected tokens")	Serve schema traffic without speculation, or route it to another model	Proven	Source: @darkmaniac7, 2026-10-28 (lab notes)
Long-context verbatim copying corrupted: 85% of image paths and names wrong at 75-92k tokens on DeepSeek V4.1 Flash	The GNmac TP4 port's sm80 attention/kernel stack; the same 2.9 bpw expert files on the Morrowmake recipe copied 0% wrong	Use another port for long-context copy work	Proven	Source: repo github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx by GNmac-lab on GitHub, posted by @greencandle9753, 2026-10-21 Source: @darkmaniac7, 2026-10-29 (lab notes)
DeepSeek V4.1 Flash copies only 15 of 20 identifiers exactly, drops underscores, output changes at temperature 0; decode speed unchanged	The Engram "host" mode failing silently (layers acting as identity), per one recipe's README; a copy test on another port did not reproduce it	Test with a copy task (about 40 random identifiers); set the Engram storage explicitly	Contested	Source: repo github.com/alanoo81/deepseek-v41-flash-4x-cmp170hx by alanoo81 on GitHub, posted by @alanoo, 2026-10-27 Source: @darkmaniac7, 2026-10-29 (lab notes)
MTP drafts accepted at 0.27 / 0.11 / 0.08 on PP4, slower than no drafter	The MTP head is fed wrong inputs on the pipeline layout in that fork (the recipe calls MTP unsupported there)	Use DFlash2 on PP4; the "none" drafter mode fails the recipe boot check, so run it with `BOOT_CHECK=0`	Proven	Source: @darkmaniac7, 2026-10-19 (lab notes)
A DeepSeek TP4 port boots with 3 or 4 speculative tokens but every request fails; 6 or 7 do not start	The DSpark drafter needs exactly 5	`SPEC_TOKENS=5`	Proven	Source: repo github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx by GNmac-lab on GitHub, posted by @greencandle9753, 2026-10-21 Source: @darkmaniac7, 2026-10-28 (lab notes)
Confusing first boot after changing the SM count (70 to 74)	torch.compile, inductor and Triton caches are keyed to the SM count	Move the caches aside; the first boot per layout costs about 10 minutes of compiles	Reported	Source: @darkmaniac7, 2026-10-21 (lab notes)
Exit code 1 after `docker stop` of a recipe container	Normal: the boot-check supervisor turns SIGTERM into a FAIL line. `OOMKilled=true` is not normal.	Nothing	Reported	Source: @darkmaniac7, 2026-10-19 (lab notes)

vLLM taking about 3 minutes to start serving on these cards is normal on PCIe Gen2 links.
Source: @c133, 2026-10-09
Source: @mountains1605, 2026-10-09
Never kill -9 a multi-GPU vLLM: it leaves zombie CUDA state; use the recipe's stop script. See Xid errors for cuInit 999.
Source: repo github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx by GNmac-lab on GitHub, posted by @greencandle9753, 2026-10-21

Host faults that look like card faults

Non-ECC host RAM: a host that passed 48 h of memtest still had an unexplained page cache corruption. A test suite reads model weights with O_DIRECT and compares them against the page cache so a host RAM fault is not blamed on the card. Reported
Source: repo github.com/dev-boz/cmp170hx-testsuite by dev-boz on GitHub, posted by @boz._., 2026-10-09
Source: @c133, 2026-10-16
A host out-of-memory during an eager model load caused Xid 31 and then Xid 154 on every card of a pipeline-parallel job.
Source: repo github.com/PixelML/club-170hx by PixelML on GitHub, posted by @c133, 2026-10-20
A micro PC host swapped and restarted under LLM load while the card itself was fine.
Source: @globalapathy, 2026-10-18
Warning The host's NVMe dropped off the bus under model reads. About a minute after a DeepSeek bench, an 8-card box fell off the network and would not boot: its Crucial T700 2 TB PCIe 5 NVMe (OS and models) on a PCIe-to-M.2 sled with no heatsink or airflow was no longer detected, and the journal was lost with the filesystem. The Engram tables (189 GiB read with 32 threads) were the heaviest sustained read the drive had seen; loading a 420 GB model later took it from 48 to 82 C in under three minutes. Fixed with a heatsink and a fan on the sled, a read-rate cap, and a guard that pauses loads near 79 C. Put model files on a separate drive and cool the boot drive.
Source: @darkmaniac7, 2026-10-28 (lab notes)
A large multi-GPU boot that logs NVRM Out of memory [NV_ERR_NO_MEMORY] from system_mem.c is host memory fragmentation, not a GPU fault; echo 1 > /proc/sys/vm/compact_memory before the first boot helps.
Source: @darkmaniac7, 2026-10-19 (lab notes)
Host ECC errors from LRDIMMs: see Memory errors and ECC.

Related pages