gpcprobe
From CMP Wiki
More actions
Reported 8 GB and 10 GB
gpcprobe reads a GA100 card's floorsweep fuses and its InfoROM black box without touching the card's state. Use it to see which compute clusters and memory partitions are disabled or fused defective, and to read a card's history before you buy or keep it. Author: @matstatman (GitHub matstatman). Repository: https://github.com/matstatman/gpcprobe
What it does
- A read-only out-of-tree Linux kernel module,
gpcprobe.ko, maps the first 16 MiB of each NVIDIA GPU's BAR0 while the NVIDIA driver stays loaded. It never writes, never unbinds the driver and needs no reboot. It exists because the driver hides the BAR from userspace (sysfsresource0fails and/dev/memis blocked). Source: repo github.com/matstatman/gpcprobe by matstatman on GitHub, posted by @matstatman, 2026-10-20. gpc_probe.pydecodes the floorsweep fuses for GPCs, memory partitions (FBP, FBPA, FBIO), ROP/L2, NVLink and PCIe lanes, marks each unit as active, disabled or defective, and prints the per-stack HBM layout. Source: repo github.com/matstatman/gpcprobe by matstatman on GitHub, posted by @matstatman, 2026-10-20.blackbox.pydumps the card's InfoROM black box: the BBX boot journal (boot count, power-on time, power-cap history), the DEM event log (a 4 KiB FIFO of Xid records with unix timestamps) and the OMS settings. It works live or on a saved 1 MiB ROM capture. Source: repo github.com/matstatman/gpcprobe by matstatman on GitHub, posted by @matstatman, 2026-10-20. @matstatman found it after register experiments crashed his card and memory changed where it should not: "every 170hx comes with a black box that logs what you are doing". Source: @matstatman, 2026-10-20
Install
Needs root and the headers for the running kernel. Rebuild after every kernel upgrade. Source: repo github.com/matstatman/gpcprobe by matstatman on GitHub, posted by @matstatman, 2026-10-20.
sudo apt install -y linux-headers-$(uname -r)
git clone https://github.com/matstatman/gpcprobe && cd gpcprobe
make
Usage
sudo insmod gpcprobe.ko # prints one line per mapped card
sudo dmesg | grep gpcprobe
sudo python3 gpc_probe.py # fuse map, all cards
sudo python3 blackbox.py live # BBX, DEM and OMS, all cards
sudo python3 blackbox.py live --bdf 06 --only dem # one card, event log only
sudo rmmod gpcprobe
@matstatman suggests giving the black-box dump to an LLM to interpret it. Source: @matstatman, 2026-10-20
Reading the output
- Disabled versus defective. "Disabled" units were trimmed at the factory; "defective" units are fused dead. @djpandaman: "if it's marked defective I've found it's truly defective. disabled can be enabled and I haven't found any time that logic failed." Source: @djpandaman, 2026-10-17 GPC fuses differ per die. Source: repo github.com/matstatman/gpcprobe by matstatman on GitHub, posted by @matstatman, 2026-10-20.
- Healthy, disabled, defective counts. A 10 GB card read GPC 5 healthy, 3 disabled, 1 defective and FBP 10, 2, 2. @matstatman: "1 defective is pretty good", "0 defective is god mode". Source: @ghostinthedatacenter_87281, 2026-10-07
Source: @matstatman, 2026-10-07
Source: @matstatman, 2026-10-07 The same owner later sourced two 10 GB cards with no defective GPC, the best he could find. Source: @ghostinthedatacenter_87281, 2026-10-09 - 8 GB card reference. On the author's two 8 GB cards 16 of 24 memory partitions (FBPA) are active, a 4096-bit bus; FBP 6 and 11 are physically dead and FBP 1 and 4 only floorswept. The 64 GiB unlock is the same 16 partitions with a 4 GiB window each instead of 512 MiB. Source: repo github.com/matstatman/gpcprobe by matstatman on GitHub, posted by @matstatman, 2026-10-20. Variant details are on Which card do I have.
- Status shadows are the copies the firmware uses. A
[!= fuse]flag means a live override differs from the fused value. Source: repo github.com/matstatman/gpcprobe by matstatman on GitHub, posted by @matstatman, 2026-10-20. Example from a 10 GB card whose sixth stack is disabled, not defective:STATUS_OPT_GPC 0x00000007,STATUS_FBP 0x00000840,STATUS_FBPA 0x00c03000,STATUS_OPT_FBIO 0x00c03000,STATUS_OPT_NVLINK 0x00000007,STATUS_HALF_FBPA 0x00000000, all[ok]. Source: @lycorascence, 2026-10-20 - The black box. Look at power-on time, boot count and the Xid log. One member's card showed 2,170 hours of uptime. Source: @kellertreppe123, 2026-10-20
Experience
- @n2dcrypt:
gpc_probe.py"has been very helpful". Source: @n2dcrypt, 2026-10-09 - The day
blackbox.pycame out, @globalapathy said @matstatman's tool saved him from buying bad cards; only one card of the batch was good. Source: @globalapathy, 2026-10-20 See the confirmation below. - @matstatman proposed a community database of probe outputs, including where each card was bought, to map which sources sell bad cards. @globalapathy suggested collecting submissions on GitHub in a standard file format. Not started yet. Source: @matstatman, 2026-10-09
Source: @globalapathy, 2026-10-09
Caveats
- The SKU-wide claims rest on the author's two 8 GB cards and a reference set of 15 cards that is not in the repository. Source: repo github.com/matstatman/gpcprobe by matstatman on GitHub, posted by @matstatman, 2026-10-20.
- hbmmon ships a different copy of
gpcprobe.cthat adds a write call under the same/dev/gpcprobename. The two modules are not interchangeable; load only the one the tool you run expects. Source: repo github.com/matstatman/hbmmon by matstatman on GitHub, posted by @matstatman, 2026-10-05. - All-defective readout. @matstatman posted a summary row for one card with GPC healthy 0 / disabled 8 / defective 8, FBP healthy 0 / disabled 12 / defective 12 and SM 0, every unit counted as both disabled and defective, with the comment "maybe its bedtime". Nothing in the thread says whether the card was dead or the decoder misread it. Reported
Source: @matstatman, 2026-10-25 - The kernel module is GPL-2.0; the repository has no LICENSE file and the Python files carry no licence. Source: repo github.com/matstatman/gpcprobe by matstatman on GitHub, posted by @matstatman, 2026-10-20.