DeepSeek V4.1 Flash heretic on 4x 8 GB

DeepSeek-V4.1-Flash-1017 heretic (pqhaz abliteration)
Reported 4 x 8 GB
Quant	Hybrid: routed experts EXL3 2.9 bpw (LibertAIDAI), everything else official FP8/BF16 with the 38 heretic attn.wo_b tensors
Engine	Morrowmake/vllm-cmp170hx (vLLM fork, container image) recipe 1.0.0 (commit d4d1f525), engine a0c57b52
Topology	TP4
Context	262,144
Decode	158 tok/s
Prefill	2453 tok/s
VRAM per card	64 GiB, --gpu-memory-utilization 0.92 (miner co-resident)
Flags	`EXTRA_ARGS="--gpu-memory-utilization 0.92", VLLM_ENGRAM_PLACEMENT=ssd, CUDA_DEVICE_ORDER=PCI_BUS_ID`
Tested	@darkmaniac7 on 2026-10-29
Source	link
Confirmations	None yet.

Morrowmake's recipe for DeepSeek-V4.1-Flash on four CMP 170HX, unchanged except for a few .env lines, with pqhaz's heretic patch written into the official checkpoint in place. Nothing was retrained or requantized. The result answers prompts the stock model refuses, keeps tool calls and long-context copying accurate, and reasons longer. Source: @darkmaniac7, 2026-10-29 (lab notes)

Parts (pinned)

Part	Source	Pin
Recipe	Morrowmake/dsv41-flash-cmp170hx-recipe	commit `d4d1f525` (release 1.0.0)
Engine image	`ghcr.io/morrowmake/vllm-cmp170hx`	`sha256:f67d9be47ffbd4a8b84c77fe76bdb3f842187d9b2786fff573e390ae5361bfb0` (vLLM fork Morrowmake/vllm-cmp170hx at `a0c57b52`)
Routed experts	LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw	EXL3 2.9 bpw, about 211 GB
Everything else	deepseek-ai/DeepSeek-V4.1-Flash-1017	official FP8/BF16, about 510 GB
Abliteration	pqhaz/DeepSeek-V4.1-Flash-1017-heretic, `patch/` folder only	38 tensors, 1.6 GB; Ivan's copy sha256 `16e63cd3e79c3590855aad340b5b9dc37f27562bf2d0a40f85268915da221ce2`

Source: @darkmaniac7, 2026-10-29 (lab notes)

Why an in-place patch is enough

The recipe's download.sh fetches both checkpoints and builds DeepSeek-V4.1-Flash-1017-hybrid-exl3-fp8, a directory of links. The routed experts (layers.N.ffn.experts.*) come from the EXL3 export; every other tensor, attention included, is a link to the official shards.

The heretic edit only touches layers.2-39.attn.wo_b.weight. Each edited tensor keeps the same FP8 dtype, shape and byte offset, and the block scales are untouched. Those tensors live in the official shards, so overwriting them there makes the served hybrid heretic with no rebuild. The heretic release leaves the routed experts alone (their FP4 storage cannot hold the edit), so it combines cleanly with the 2.9 bpw EXL3 experts. Source: @darkmaniac7, 2026-10-29 (lab notes)

Steps

Do the recipe's one-time host setup, clone it and run ./install.sh, as its README says. Prerequisites are those of DeepSeek V4.1 Flash on 4x 8 GB (Morrowmake).
./download.sh fetches both checkpoints and builds the hybrid. Budget about 680 GiB.
Fetch the patch only: hf download pqhaz/DeepSeek-V4.1-Flash-1017-heretic --include "patch/*" --local-dir heretic
Write it into the official checkpoint in place with heretic_inplace.py (below, stdlib Python only):
Heretic_inplace.py (4.6 KB, by @ImportBot, details)
sha256 5c2e6a96b424ba5c6b2983bf06953704f0dd63e51c3beb5ef25e5a73cea44343 The script applies the heretic patch to the official shards in place, restores them byte-exact from the backup, or reports their status.

python3 heretic_inplace.py apply   <MODELS_DIR>/DeepSeek-V4.1-Flash-1017 heretic/patch/<file>.safetensors stock-wo_b-backup.safetensors
python3 heretic_inplace.py status  <same three args>     # expect: heretic 38, stock 0, other 0
python3 heretic_inplace.py restore <same three args>     # undo, byte-exact

apply first saves the 38 stock tensors to the backup file (1.6 GB), then overwrites each tensor, fsyncs and reads it back. It refuses any shard that has other hard links, so it cannot patch a second copy by accident. Re-running it does nothing.

To avoid patching in place, the heretic repo's own ds_patch.py apply <official> <patch> <new_dir> writes a patched copy instead. It rewrites 38 full shards (about 7 GB each, roughly 260 GB extra) and hard-links the rest. Put that copy under the official checkpoint's directory name: the hybrid's links resolve by that name, and the index and config the builder checks are unchanged.
.env changes, all specific to Ivan's box:
- CUDA_VISIBLE_DEVICES=4,5,6,7 and CUDA_DEVICE_ORDER=PCI_BUS_ID, because this runs on four cards of an 8-card machine. The recipe does not pass CUDA_DEVICE_ORDER into the container, so it was added to CONTAINER_KEYS in start.sh (a one-word change). On a 4-card box you need neither.
- EXTRA_ARGS="--gpu-memory-utilization 0.92", only because a miner shares the cards and holds 2.5 GiB on each. At the default 0.95 a 17k-token prefill ran out of memory. With nothing else on the cards, keep the default.
- VLLM_ENGRAM_PLACEMENT=ssd (default auto) keeps the Engram tables on SSD instead of in RAM. If you have the RAM, auto is fine.
- PORT and SERVED_MODEL_NAME are cosmetic.
./start.sh, then ./start.sh smoke (one chat request and one tool call).

Source: @darkmaniac7, 2026-10-29 (lab notes)

Measured numbers

On Ivan's box: four CMP 170HX on PCIe, sharing the machine with a miner, measured with his own bench script. Not comparable with the numbers the recipe publishes.

	Result
Decode, 1 user	code 158, structured 136, prose 121 tok/s
Decode, 8 users (total)	code 323, structured 261, prose 249 tok/s
Prefill, 17k-token prompt (warm)	2,453 tok/s
Long-context copy accuracy (75k to 92k-token roleplay turns that must copy image paths and names exactly)	1 wrong of 30. The miss picked a different variant of the right image, not a garbled path
Same 2.9 bpw expert files on a different engine (GNmac)	85% wrong on the same test. So that engine causes the copy errors, not the quant or the heretic edit
Refusal spot check (4 prompts, 1,500-token cap)	2 answered, 2 hit the cap while still reasoning, 0 refused
Tool calls (`deepseek_v41` parser)	work

The heretic model card reports 1 refusal in 100 held-out prompts (base: 97) and KL 0.136 on harmless prompts. An earlier run under the same conditions read 150 / 131 / 117 tok/s for one user (code / structured / prose); the GNmac engine on the same cards read 248 / 177 / 135 and 542 / 396 / 347 at 8 users, prefill 2,697. Source: @darkmaniac7, 2026-10-29 (lab notes)
Source: @darkmaniac7, 2026-10-29 (lab notes)

Things to know

It reasons much longer after the edit. The model card puts the median completion at 1,517 tokens against 357 for the base. Set max_tokens generously, or content comes back empty with finish_reason: "length".
No capability benchmarks have been run on the heretic version, Ivan's or the model card's. KL 0.136 means general behaviour has shifted somewhat.
Speed is engine-bound. On Ivan's box the GNmac engine decodes 1.1 to 1.7x faster (prefill 1.1x) but corrupts long-context copying, so he kept Morrowmake. See DeepSeek V4.1 Flash on 4x 8 GB (GNmac).
Uncensored EXL3 bundles whose edit lives in dense tensors do nothing on the GNmac engine, which takes dense tensors from the official checkpoint; the heretic patch works on both engines because it patches those official tensors. Source: @darkmaniac7, 2026-10-28 (lab notes)

Source: @darkmaniac7, 2026-10-29 (lab notes)

Checks

After applying the patch Ivan checked the weights: the dequantize/requantize round trip reproduces 100.00% of the original FP8 codes; 92-99% of the patched codes are within one FP8 step of the ideal rank-1 edit; the change is 3.7-4.7% of the weight norm on all 38 tensors. Nothing beyond the advertised edit plus rounding is visible.
After any restart, confirm which checkpoint the container actually loaded (docker inspect <container> --format '{{.Args}}'): one reload silently came up on the stock hybrid.
Contested Capability: the write-up above says no capability benchmarks have been run on the heretic version. Ivan's chassis notes also cite independent numbers for this layout (GSM8K and HumanEval unchanged, MMLU -0.4, HarmBench compliance 287 of 300 against 30 of 300 stock) without naming who measured them.

Source: @darkmaniac7, 2026-10-29 (lab notes)

heretic_inplace.py

Ivan's helper, stdlib Python only. It applies or undoes the patch in the official shards for engines that read attention weights straight from the official checkpoint (Morrowmake).

#!/usr/bin/env python3
"""Apply (or undo) the pqhaz heretic patch IN PLACE in the official DeepSeek-V4.1-Flash-1017 shards, for engines that
read attention weights straight from the official checkpoint (Morrowmake). Only the 38 `layers.N.attn.wo_b.weight`
tensors change (same dtype F8_E4M3, same shape, same offsets); FP8 scales are untouched, as the patch was built.

  heretic_inplace.py apply   OFFICIAL_DIR PATCH.safetensors BACKUP.safetensors
  heretic_inplace.py restore OFFICIAL_DIR PATCH.safetensors BACKUP.safetensors
  heretic_inplace.py status  OFFICIAL_DIR PATCH.safetensors BACKUP.safetensors

apply: writes an exact byte backup of every original tensor first (a safetensors file with the same names), then
overwrites, fsyncs and reads back; refuses a shard with more than one hard link. restore: writes the backup back.
Idempotent: a tensor already in the wanted state is skipped. Stdlib only."""
import hashlib, json, os, struct, sys


def header(path):
    with open(path, "rb") as f:
        n = struct.unpack("<Q", f.read(8))[0]
        h = json.loads(f.read(n))
    return h, 8 + n


def region(path, name):
    h, base = header(path)
    t = h[name]
    a, b = t["data_offsets"]
    return t["dtype"], t["shape"], base + a, b - a


def read_at(path, off, size):
    with open(path, "rb") as f:
        f.seek(off)
        return f.read(size)


def sha(b):
    return hashlib.sha256(b).hexdigest()


def write_safetensors(path, tensors):  # tensors: name -> (dtype, shape, bytes)
    hdr, off = {}, 0
    for k in sorted(tensors):
        dt, shp, data = tensors[k]
        hdr[k] = {"dtype": dt, "shape": shp, "data_offsets": [off, off + len(data)]}
        off += len(data)
    hb = json.dumps(hdr, separators=(",", ":")).encode()
    hb += b" " * ((8 - len(hb) % 8) % 8)
    tmp = path + ".tmp"
    with open(tmp, "wb") as f:
        f.write(struct.pack("<Q", len(hb)) + hb)
        for k in sorted(tensors):
            f.write(tensors[k][2])
        f.flush(); os.fsync(f.fileno())
    os.replace(tmp, path)


def main():
    mode, odir, patch, backup = sys.argv[1:5]
    wm = json.load(open(os.path.join(odir, "model.safetensors.index.json")))["weight_map"]
    ph, _ = header(patch)
    names = sorted(k for k in ph if k != "__metadata__")
    plan = []
    for k in names:
        shard = os.path.join(odir, wm[k])
        dt, shp, off, size = region(shard, k)
        pdt, pshp, poff, psize = region(patch, k)
        assert (dt, shp, size) == (pdt, pshp, psize), (k, dt, shp, size, pdt, pshp, psize)
        plan.append((k, shard, off, size, poff))
    print(f"{len(plan)} tensors in {len({p[1] for p in plan})} shards")

    if mode == "status":
        bh = header(backup)[0] if os.path.exists(backup) else {}
        n_patched = n_stock = n_other = 0
        for k, shard, off, size, poff in plan:
            cur = sha(read_at(shard, off, size))
            if cur == sha(read_at(patch, poff, size)): n_patched += 1
            elif bh and cur == sha(read_at(backup, region(backup, k)[2], size)): n_stock += 1
            else: n_other += 1
        print(f"heretic {n_patched}, stock (matches backup) {n_stock}, other {n_other}")
        return

    if mode == "apply":
        for k, shard, *_ in plan:
            assert os.stat(shard).st_nlink == 1, f"{shard} has other hard links; refusing to write in place"
        if not os.path.exists(backup):
            orig = {}
            for k, shard, off, size, poff in plan:
                data = read_at(shard, off, size)
                if data == read_at(patch, poff, size):
                    sys.exit(f"{k} is already patched and there is no backup: refusing (stock bytes unknown)")
                orig[k] = (region(shard, k)[0], region(shard, k)[1], data)
            write_safetensors(backup, orig)
            print(f"backup written: {backup} ({os.path.getsize(backup) / 2**30:.2f} GiB)")
        src = lambda k, poff, size: read_at(patch, poff, size)
    elif mode == "restore":
        assert os.path.exists(backup), "no backup"
        src = lambda k, poff, size: read_at(backup, region(backup, k)[2], size)
    else:
        sys.exit("mode: apply | restore | status")

    done = skipped = 0
    for k, shard, off, size, poff in plan:
        want = src(k, poff, size)
        if read_at(shard, off, size) == want:
            skipped += 1; continue
        fd = os.open(shard, os.O_RDWR)
        try:
            assert os.pwrite(fd, want, off) == size
            os.fsync(fd)
        finally:
            os.close(fd)
        assert sha(read_at(shard, off, size)) == sha(want), f"read-back mismatch on {k}"
        done += 1
    print(f"{mode}: {done} written, {skipped} already in place, all read back OK")


if __name__ == "__main__":
    main()

Source: @darkmaniac7, 2026-10-29 (lab notes)

Credits

DeepSeek for the model. Morrowmake for the recipe and the Ampere vLLM fork. LibertAI for the EXL3 2.9 bpw quant. pqhaz for the heretic ablation of 1017. p-e-w for the Heretic method.

Sources

Ivan's lab notes, 2026-10-29 (public write-up of the recipe).
The script is File:Heretic_inplace.py; its sha256 is shown in the Download box in the Steps section.

Parts (pinned)

Why an in-place patch is enough

Steps

Measured numbers

Things to know

Checks

heretic_inplace.py

Credits

Related

Sources