Gpcprobe - Revision history

ImportBot: Import: gpcprobe and blackbox.py, @matstatman, @ghostinthedatacenter_87281, @globalapathy, @djpandaman, @kellertreppe123, @n2dcrypt; repos matstatman/gpcprobe, matstatman/hbmmon

2026-10-30T10:02:12Z

Import: gpcprobe and blackbox.py, @matstatman, @ghostinthedatacenter_87281, @globalapathy, @djpandaman, @kellertreppe123, @n2dcrypt; repos matstatman/gpcprobe, matstatman/hbmmon

← Older revision		Revision as of 10:02, 30 October 2026
Line 32:		Line 32:
	* '''Healthy, disabled, defective counts.''' A 10 GB card read GPC 5 healthy, 3 disabled, 1 defective and FBP 10, 2, 2. @matstatman: "1 defective is pretty good", "0 defective is god mode". {{Source\|handle=ghostinthedatacenter_87281\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557485178023317635}}<br />{{Source\|handle=matstatman\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557485484891050108}}<br />{{Source\|handle=matstatman\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557485522488655984}} The same owner later sourced two 10 GB cards with no defective GPC, the best he could find. {{Source\|handle=ghostinthedatacenter_87281\|date=2026-10-09\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1558157875715772506}}		* '''Healthy, disabled, defective counts.''' A 10 GB card read GPC 5 healthy, 3 disabled, 1 defective and FBP 10, 2, 2. @matstatman: "1 defective is pretty good", "0 defective is god mode". {{Source\|handle=ghostinthedatacenter_87281\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557485178023317635}}<br />{{Source\|handle=matstatman\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557485484891050108}}<br />{{Source\|handle=matstatman\|date=2026-10-07\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557485522488655984}} The same owner later sourced two 10 GB cards with no defective GPC, the best he could find. {{Source\|handle=ghostinthedatacenter_87281\|date=2026-10-09\|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1558157875715772506}}
	* '''8 GB card reference.''' On the author's two 8 GB cards 16 of 24 memory partitions (FBPA) are active, a 4096-bit bus; FBP 6 and 11 are physically dead and FBP 1 and 4 only floorswept. The 64 GiB unlock is the same 16 partitions with a 4 GiB window each instead of 512 MiB. {{Source\|handle=matstatman\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850\|repo=https://github.com/matstatman/gpcprobe}}. Variant details are on [[Which card do I have]].		* '''8 GB card reference.''' On the author's two 8 GB cards 16 of 24 memory partitions (FBPA) are active, a 4096-bit bus; FBP 6 and 11 are physically dead and FBP 1 and 4 only floorswept. The 64 GiB unlock is the same 16 partitions with a 4 GiB window each instead of 512 MiB. {{Source\|handle=matstatman\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850\|repo=https://github.com/matstatman/gpcprobe}}. Variant details are on [[Which card do I have]].
	* '''Status shadows''' are the copies the firmware uses. A <code><nowiki>[!= fuse]</nowiki></code> flag means a live override differs from the fused value. {{Source\|handle=matstatman\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850\|repo=https://github.com/matstatman/gpcprobe}}.		* '''Status shadows''' are the copies the firmware uses. A <code><nowiki>[!= fuse]</nowiki></code> flag means a live override differs from the fused value. {{Source\|handle=matstatman\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850\|repo=https://github.com/matstatman/gpcprobe}}. Example from a 10 GB card whose sixth stack is disabled, not defective: <code><nowiki>STATUS_OPT_GPC 0x00000007</nowiki></code>, <code><nowiki>STATUS_FBP 0x00000840</nowiki></code>, <code><nowiki>STATUS_FBPA 0x00c03000</nowiki></code>, <code><nowiki>STATUS_OPT_FBIO 0x00c03000</nowiki></code>, <code><nowiki>STATUS_OPT_NVLINK 0x00000007</nowiki></code>, <code><nowiki>STATUS_HALF_FBPA 0x00000000</nowiki></code>, all <code><nowiki>[ok]</nowiki></code>. {{Source\|handle=lycorascence\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562228981489729560}}
	* '''The black box.''' Look at power-on time, boot count and the Xid log. One member's card showed 2,170 hours of uptime. {{Source\|handle=kellertreppe123\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562194914358599787}}		* '''The black box.''' Look at power-on time, boot count and the Xid log. One member's card showed 2,170 hours of uptime. {{Source\|handle=kellertreppe123\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562194914358599787}}

Line 43:		Line 43:
	* The SKU-wide claims rest on the author's two 8 GB cards and a reference set of 15 cards that is not in the repository. {{Source\|handle=matstatman\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850\|repo=https://github.com/matstatman/gpcprobe}}.		* The SKU-wide claims rest on the author's two 8 GB cards and a reference set of 15 cards that is not in the repository. {{Source\|handle=matstatman\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850\|repo=https://github.com/matstatman/gpcprobe}}.
	* [[hbmmon]] ships a different copy of <code><nowiki>gpcprobe.c</nowiki></code> that adds a write call under the same <code><nowiki>/dev/gpcprobe</nowiki></code> name. The two modules are not interchangeable; load only the one the tool you run expects. {{Source\|handle=matstatman\|date=2026-10-05\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556758870565593288\|repo=https://github.com/matstatman/hbmmon}}.		* [[hbmmon]] ships a different copy of <code><nowiki>gpcprobe.c</nowiki></code> that adds a write call under the same <code><nowiki>/dev/gpcprobe</nowiki></code> name. The two modules are not interchangeable; load only the one the tool you run expects. {{Source\|handle=matstatman\|date=2026-10-05\|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556758870565593288\|repo=https://github.com/matstatman/hbmmon}}.
			* '''All-defective readout.''' @matstatman posted a summary row for one card with GPC healthy 0 / disabled 8 / defective 8, FBP healthy 0 / disabled 12 / defective 12 and SM 0, every unit counted as both disabled and defective, with the comment "maybe its bedtime". Nothing in the thread says whether the card was dead or the decoder misread it. {{Status\|Reported}}<br />{{Source\|handle=matstatman\|date=2026-10-25\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1563724680555139123}}
	* The kernel module is GPL-2.0; the repository has no LICENSE file and the Python files carry no licence. {{Source\|handle=matstatman\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850\|repo=https://github.com/matstatman/gpcprobe}}.		* The kernel module is GPL-2.0; the repository has no LICENSE file and the Python files carry no licence. {{Source\|handle=matstatman\|date=2026-10-20\|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850\|repo=https://github.com/matstatman/gpcprobe}}.

ImportBot: Import: gpcprobe and blackbox.py, @matstatman, @ghostinthedatacenter_87281, @globalapathy, @djpandaman, @kellertreppe123, @n2dcrypt; repos matstatman/gpcprobe, matstatman/hbmmon

2026-10-30T08:38:25Z

Import: gpcprobe and blackbox.py, @matstatman, @ghostinthedatacenter_87281, @globalapathy, @djpandaman, @kellertreppe123, @n2dcrypt; repos matstatman/gpcprobe, matstatman/hbmmon

← Older revision		Revision as of 08:38, 30 October 2026
Line 1:		Line 1:
			{{DISPLAYTITLE:gpcprobe}}
	{{Status\|Reported}} {{Cards\|both}}		{{Status\|Reported}} {{Cards\|both}}

ImportBot: Import: gpcprobe and blackbox.py, @matstatman, @ghostinthedatacenter_87281, @globalapathy, @djpandaman, @kellertreppe123, @n2dcrypt; repos matstatman/gpcprobe, matstatman/hbmmon

2026-10-30T08:04:28Z

Import: gpcprobe and blackbox.py, @matstatman, @ghostinthedatacenter_87281, @globalapathy, @djpandaman, @kellertreppe123, @n2dcrypt; repos matstatman/gpcprobe, matstatman/hbmmon

New page

{{Status|Reported}} {{Cards|both}}

'''gpcprobe''' reads a GA100 card's floorsweep fuses and its InfoROM black box without touching the card's state. Use it to see which compute clusters and memory partitions are disabled or fused defective, and to read a card's history before you buy or keep it. Author: @matstatman (GitHub matstatman). Repository: https://github.com/matstatman/gpcprobe

== What it does ==
* A read-only out-of-tree Linux kernel module, <code><nowiki>gpcprobe.ko</nowiki></code>, maps the first 16 MiB of each NVIDIA GPU's BAR0 while the NVIDIA driver stays loaded. It never writes, never unbinds the driver and needs no reboot. It exists because the driver hides the BAR from userspace (sysfs <code><nowiki>resource0</nowiki></code> fails and <code><nowiki>/dev/mem</nowiki></code> is blocked). {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850|repo=https://github.com/matstatman/gpcprobe}}.
* <code><nowiki>gpc_probe.py</nowiki></code> decodes the floorsweep fuses for GPCs, memory partitions (FBP, FBPA, FBIO), ROP/L2, NVLink and PCIe lanes, marks each unit as active, disabled or defective, and prints the per-stack HBM layout. {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850|repo=https://github.com/matstatman/gpcprobe}}.
* <code><nowiki>blackbox.py</nowiki></code> dumps the card's InfoROM black box: the BBX boot journal (boot count, power-on time, power-cap history), the DEM event log (a 4 KiB FIFO of Xid records with unix timestamps) and the OMS settings. It works live or on a saved 1 MiB ROM capture. {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850|repo=https://github.com/matstatman/gpcprobe}}. @matstatman found it after register experiments crashed his card and memory changed where it should not: "every 170hx comes with a black box that logs what you are doing". {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562127113878896660}}

== Install ==
Needs root and the headers for the running kernel. Rebuild after every kernel upgrade. {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850|repo=https://github.com/matstatman/gpcprobe}}.
<syntaxhighlight lang="bash">
sudo apt install -y linux-headers-$(uname -r)
git clone https://github.com/matstatman/gpcprobe && cd gpcprobe
make
</syntaxhighlight>

== Usage ==
<syntaxhighlight lang="bash">
sudo insmod gpcprobe.ko # prints one line per mapped card
sudo dmesg | grep gpcprobe
sudo python3 gpc_probe.py # fuse map, all cards
sudo python3 blackbox.py live # BBX, DEM and OMS, all cards
sudo python3 blackbox.py live --bdf 06 --only dem # one card, event log only
sudo rmmod gpcprobe
</syntaxhighlight>
@matstatman suggests giving the black-box dump to an LLM to interpret it. {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562129044366164021}}

== Reading the output ==
* '''Disabled versus defective.''' "Disabled" units were trimmed at the factory; "defective" units are fused dead. @djpandaman: "if it's marked defective I've found it's truly defective. disabled can be enabled and I haven't found any time that logic failed." {{Source|handle=djpandaman|date=2026-10-17|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1561028809237602389}} GPC fuses differ per die. {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850|repo=https://github.com/matstatman/gpcprobe}}.
* '''Healthy, disabled, defective counts.''' A 10 GB card read GPC 5 healthy, 3 disabled, 1 defective and FBP 10, 2, 2. @matstatman: "1 defective is pretty good", "0 defective is god mode". {{Source|handle=ghostinthedatacenter_87281|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557485178023317635}}<br />{{Source|handle=matstatman|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557485484891050108}}<br />{{Source|handle=matstatman|date=2026-10-07|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1557485522488655984}} The same owner later sourced two 10 GB cards with no defective GPC, the best he could find. {{Source|handle=ghostinthedatacenter_87281|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1534370835220004885/1558157875715772506}}
* '''8 GB card reference.''' On the author's two 8 GB cards 16 of 24 memory partitions (FBPA) are active, a 4096-bit bus; FBP 6 and 11 are physically dead and FBP 1 and 4 only floorswept. The 64 GiB unlock is the same 16 partitions with a 4 GiB window each instead of 512 MiB. {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850|repo=https://github.com/matstatman/gpcprobe}}. Variant details are on [[Which card do I have]].
* '''Status shadows''' are the copies the firmware uses. A <code><nowiki>[!= fuse]</nowiki></code> flag means a live override differs from the fused value. {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850|repo=https://github.com/matstatman/gpcprobe}}.
* '''The black box.''' Look at power-on time, boot count and the Xid log. One member's card showed 2,170 hours of uptime. {{Source|handle=kellertreppe123|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562194914358599787}}

== Experience ==
* @n2dcrypt: <code><nowiki>gpc_probe.py</nowiki></code> "has been very helpful". {{Source|handle=n2dcrypt|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1558179141805678613}}
* The day <code><nowiki>blackbox.py</nowiki></code> came out, @globalapathy said @matstatman's tool saved him from buying bad cards; only one card of the batch was good. {{Source|handle=globalapathy|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562131178524250145}} See the confirmation below.
* @matstatman proposed a community database of probe outputs, including where each card was bought, to map which sources sell bad cards. @globalapathy suggested collecting submissions on GitHub in a standard file format. Not started yet. {{Source|handle=matstatman|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1558178978605441027}}<br />{{Source|handle=globalapathy|date=2026-10-09|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1558179788621873174}}

== Caveats ==
* The SKU-wide claims rest on the author's two 8 GB cards and a reference set of 15 cards that is not in the repository. {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850|repo=https://github.com/matstatman/gpcprobe}}.
* [[hbmmon]] ships a different copy of <code><nowiki>gpcprobe.c</nowiki></code> that adds a write call under the same <code><nowiki>/dev/gpcprobe</nowiki></code> name. The two modules are not interchangeable; load only the one the tool you run expects. {{Source|handle=matstatman|date=2026-10-05|link=https://discord.com/channels/1518634379604136178/1518692084729511967/1556758870565593288|repo=https://github.com/matstatman/hbmmon}}.
* The kernel module is GPL-2.0; the repository has no LICENSE file and the Python files carry no licence. {{Source|handle=matstatman|date=2026-10-20|link=https://discord.com/channels/1518634379604136178/1518695328008638516/1562128904007843850|repo=https://github.com/matstatman/gpcprobe}}.

== See also ==
* [[Tools]]
* [[Acceptance checklist]]
* [[Testing and burn-in]]
* [[Monitoring]]
* [[hbmmon]]
* [[170hx-rir-tool]]
* [[Which card do I have]]

{{ImportFooter}}
[[Category:Tools]]
[[Category:8 GB]]
[[Category:10 GB]]