ImportBot: Crashes, dropouts, throttling and wrong output under load (power, heat, undervolt, HBM clock, 10 GB SM unlock side effects, engine bugs) from the cmpunlocker Discord (2026-10-04 to 10-28) and community repos (170tune, club-170hx, asm64-hooligan, vllm-170hx-glm5, Morrowmake recipes); credits inline; sources: @cibernox, @reala7096, @robbbbbb7973, @xeax, @xanthefin, @c133, @darkmaniac7, @zorg33, @ichundes, @moliminous, @gnasherrr, @replacedriver47 (+32 more); GitHub Consensus-Pr

2026-10-30T08:46:00Z

Crashes, dropouts, throttling and wrong output under load (power, heat, undervolt, HBM clock, 10 GB SM unlock side effects, engine bugs) from the cmpunlocker Discord (2026-10-04 to 10-28) and community repos (170tune, club-170hx, asm64-hooligan, vllm-170hx-glm5, Morrowmake recipes); credits inline; sources: @cibernox, @reala7096, @robbbbbb7973, @xeax, @xanthefin, @c133, @darkmaniac7, @zorg33, @ichundes, @moliminous, @gnasherrr, @replacedriver47 (+32 more); GitHub Consensus-Pr

← Older revision		Revision as of 08:46, 30 October 2026
Line 248:		Line 248:
	* A host out-of-memory during an eager model load caused Xid 31 and then Xid 154 on every card of a pipeline-parallel job.<br />{{Source\|handle=c133\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127\|repo=https://github.com/PixelML/club-170hx}}		* A host out-of-memory during an eager model load caused Xid 31 and then Xid 154 on every card of a pipeline-parallel job.<br />{{Source\|handle=c133\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127\|repo=https://github.com/PixelML/club-170hx}}
	* A micro PC host swapped and restarted under LLM load while the card itself was fine.<br />{{Source\|handle=globalapathy\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561375163210141767}}		* A micro PC host swapped and restarted under LLM load while the card itself was fine.<br />{{Source\|handle=globalapathy\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561375163210141767}}
	* {{Status\|Warning}} '''The host's NVMe dropped off the bus under model reads.''' About a minute after a DeepSeek bench, an 8-card box fell off the network and would not boot: its 2 TB PCIe 5 NVMe (OS and models) on a PCIe-to-M.2 sled with no heatsink or airflow was no longer detected, and the journal was lost with the filesystem. The Engram tables (189 GiB read with 32 threads) were the heaviest sustained read the drive had seen; loading a 420 GB model later took it from 48 to 82 C in under three minutes. Fixed with a heatsink and a fan on the sled, a read-rate cap, and a guard that pauses loads near 79 C. Put model files on a separate drive and cool the boot drive.<br />{{Source\|handle=darkmaniac7\|date=2026-10-28\|note=lab notes}}		* {{Status\|Warning}} '''The host's NVMe dropped off the bus under model reads.''' About a minute after a DeepSeek bench, an 8-card box fell off the network and would not boot: its Crucial T700 2 TB PCIe 5 NVMe (OS and models) on a PCIe-to-M.2 sled with no heatsink or airflow was no longer detected, and the journal was lost with the filesystem. The Engram tables (189 GiB read with 32 threads) were the heaviest sustained read the drive had seen; loading a 420 GB model later took it from 48 to 82 C in under three minutes. Fixed with a heatsink and a fan on the sled, a read-rate cap, and a guard that pauses loads near 79 C. Put model files on a separate drive and cool the boot drive.<br />{{Source\|handle=darkmaniac7\|date=2026-10-28\|note=lab notes}}
	* A large multi-GPU boot that logs NVRM <code><nowiki>Out of memory [NV_ERR_NO_MEMORY]</nowiki></code> from <code><nowiki>system_mem.c</nowiki></code> is host memory fragmentation, not a GPU fault; <code><nowiki>echo 1 > /proc/sys/vm/compact_memory</nowiki></code> before the first boot helps.<br />{{Source\|handle=darkmaniac7\|date=2026-10-19\|note=lab notes}}		* A large multi-GPU boot that logs NVRM <code><nowiki>Out of memory [NV_ERR_NO_MEMORY]</nowiki></code> from <code><nowiki>system_mem.c</nowiki></code> is host memory fragmentation, not a GPU fault; <code><nowiki>echo 1 > /proc/sys/vm/compact_memory</nowiki></code> before the first boot helps.<br />{{Source\|handle=darkmaniac7\|date=2026-10-19\|note=lab notes}}
	* Host ECC errors from LRDIMMs: see [[Memory errors and ECC]].		* Host ECC errors from LRDIMMs: see [[Memory errors and ECC]].

ImportBot: Crashes, dropouts, throttling and wrong output under load (power, heat, undervolt, HBM clock, 10 GB SM unlock side effects, engine bugs) from the cmpunlocker Discord (2026-10-04 to 10-28) and community repos (170tune, club-170hx, asm64-hooligan, vllm-170hx-glm5, Morrowmake recipes); credits inline; sources: @cibernox, @reala7096, @robbbbbb7973, @xeax, @xanthefin, @c133, @darkmaniac7, @zorg33, @ichundes, @moliminous, @gnasherrr, @replacedriver47 (+32 more); GitHub Consensus-Pr

2026-10-30T08:17:21Z

Show changes

← Older revision		Revision as of 08:46, 30 October 2026
Line 248:		Line 248:
	* A host out-of-memory during an eager model load caused Xid 31 and then Xid 154 on every card of a pipeline-parallel job.<br />{{Source\|handle=c133\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127\|repo=https://github.com/PixelML/club-170hx}}		* A host out-of-memory during an eager model load caused Xid 31 and then Xid 154 on every card of a pipeline-parallel job.<br />{{Source\|handle=c133\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1528826330454429706/1562161732137386127\|repo=https://github.com/PixelML/club-170hx}}
	* A micro PC host swapped and restarted under LLM load while the card itself was fine.<br />{{Source\|handle=globalapathy\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561375163210141767}}		* A micro PC host swapped and restarted under LLM load while the card itself was fine.<br />{{Source\|handle=globalapathy\|date=2026-10-18\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561375163210141767}}
	* {{Status\|Warning}} '''The host's NVMe dropped off the bus under model reads.''' About a minute after a DeepSeek bench, an 8-card box fell off the network and would not boot: its 2 TB PCIe 5 NVMe (OS and models) on a PCIe-to-M.2 sled with no heatsink or airflow was no longer detected, and the journal was lost with the filesystem. The Engram tables (189 GiB read with 32 threads) were the heaviest sustained read the drive had seen; loading a 420 GB model later took it from 48 to 82 C in under three minutes. Fixed with a heatsink and a fan on the sled, a read-rate cap, and a guard that pauses loads near 79 C. Put model files on a separate drive and cool the boot drive.<br />{{Source\|handle=darkmaniac7\|date=2026-10-28\|note=lab notes}}		* {{Status\|Warning}} '''The host's NVMe dropped off the bus under model reads.''' About a minute after a DeepSeek bench, an 8-card box fell off the network and would not boot: its Crucial T700 2 TB PCIe 5 NVMe (OS and models) on a PCIe-to-M.2 sled with no heatsink or airflow was no longer detected, and the journal was lost with the filesystem. The Engram tables (189 GiB read with 32 threads) were the heaviest sustained read the drive had seen; loading a 420 GB model later took it from 48 to 82 C in under three minutes. Fixed with a heatsink and a fan on the sled, a read-rate cap, and a guard that pauses loads near 79 C. Put model files on a separate drive and cool the boot drive.<br />{{Source\|handle=darkmaniac7\|date=2026-10-28\|note=lab notes}}
	* A large multi-GPU boot that logs NVRM <code><nowiki>Out of memory [NV_ERR_NO_MEMORY]</nowiki></code> from <code><nowiki>system_mem.c</nowiki></code> is host memory fragmentation, not a GPU fault; <code><nowiki>echo 1 > /proc/sys/vm/compact_memory</nowiki></code> before the first boot helps.<br />{{Source\|handle=darkmaniac7\|date=2026-10-19\|note=lab notes}}		* A large multi-GPU boot that logs NVRM <code><nowiki>Out of memory [NV_ERR_NO_MEMORY]</nowiki></code> from <code><nowiki>system_mem.c</nowiki></code> is host memory fragmentation, not a GPU fault; <code><nowiki>echo 1 > /proc/sys/vm/compact_memory</nowiki></code> before the first boot helps.<br />{{Source\|handle=darkmaniac7\|date=2026-10-19\|note=lab notes}}
	* Host ECC errors from LRDIMMs: see [[Memory errors and ECC]].		* Host ECC errors from LRDIMMs: see [[Memory errors and ECC]].

Instability under load - Revision history