View source for DeepSeek V4.1 Flash heretic on 4x 8 GB

{{Recipe
|model=DeepSeek-V4.1-Flash-1017 heretic (pqhaz abliteration)
|quant=Hybrid: routed experts EXL3 2.9 bpw (LibertAIDAI), everything else official FP8/BF16 with the 38 heretic attn.wo_b tensors
|engine=Morrowmake/vllm-cmp170hx (vLLM fork, container image)
|engine_version=recipe 1.0.0 (commit d4d1f525), engine a0c57b52
|cards=8 GB
|card_count=4
|topology=TP4
|context=262,144
|decode_tps=158
|prefill_tps=2453
|vram_per_card=64 GiB, --gpu-memory-utilization 0.92 (miner co-resident)
|flags=EXTRA_ARGS="--gpu-memory-utilization 0.92", VLLM_ENGRAM_PLACEMENT=ssd, CUDA_DEVICE_ORDER=PCI_BUS_ID
|status=Reported
|tested_by=darkmaniac7
|tested_on=2026-10-29
|source=https://github.com/Morrowmake/dsv41-flash-cmp170hx-recipe
}}
Morrowmake's recipe for DeepSeek-V4.1-Flash on four CMP 170HX, unchanged except for a few <code><nowiki>.env</nowiki></code> lines, with
pqhaz's heretic patch written into the official checkpoint in place. Nothing was retrained or requantized. The result
answers prompts the stock model refuses, keeps tool calls and long-context copying accurate, and reasons longer.
{{Source|handle=darkmaniac7|date=2026-10-29|note=lab notes}}

== Parts (pinned) ==
{| class="wikitable"
! Part !! Source !! Pin
|-
| Recipe || [https://github.com/Morrowmake/dsv41-flash-cmp170hx-recipe Morrowmake/dsv41-flash-cmp170hx-recipe] || commit <code><nowiki>d4d1f525</nowiki></code> (release 1.0.0)
|-
| Engine image || <code><nowiki>ghcr.io/morrowmake/vllm-cmp170hx</nowiki></code> || <code><nowiki>sha256:f67d9be47ffbd4a8b84c77fe76bdb3f842187d9b2786fff573e390ae5361bfb0</nowiki></code> (vLLM fork Morrowmake/vllm-cmp170hx at <code><nowiki>a0c57b52</nowiki></code>)
|-
| Routed experts || [https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw] || EXL3 2.9 bpw, about 211 GB
|-
| Everything else || [https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017 deepseek-ai/DeepSeek-V4.1-Flash-1017] || official FP8/BF16, about 510 GB
|-
| Abliteration || [https://huggingface.co/pqhaz/DeepSeek-V4.1-Flash-1017-heretic pqhaz/DeepSeek-V4.1-Flash-1017-heretic], <code><nowiki>patch/</nowiki></code> folder only || 38 tensors, 1.6 GB; Ivan's copy sha256 <code><nowiki>16e63cd3e79c3590855aad340b5b9dc37f27562bf2d0a40f85268915da221ce2</nowiki></code>
|}
{{Source|handle=darkmaniac7|date=2026-10-29|note=lab notes}}

== Why an in-place patch is enough ==
The recipe's <code><nowiki>download.sh</nowiki></code> fetches both checkpoints and builds <code><nowiki>DeepSeek-V4.1-Flash-1017-hybrid-exl3-fp8</nowiki></code>,
a directory of links. The routed experts (<code><nowiki>layers.N.ffn.experts.*</nowiki></code>) come from the EXL3 export; every other
tensor, attention included, is a link to the official shards.

The heretic edit only touches <code><nowiki>layers.2-39.attn.wo_b.weight</nowiki></code>. Each edited tensor keeps the same FP8 dtype, shape
and byte offset, and the block scales are untouched. Those tensors live in the official shards, so overwriting them
there makes the served hybrid heretic with no rebuild. The heretic release leaves the routed experts alone (their FP4
storage cannot hold the edit), so it combines cleanly with the 2.9 bpw EXL3 experts.
{{Source|handle=darkmaniac7|date=2026-10-29|note=lab notes}}

== Steps ==
# Do the recipe's one-time host setup, clone it and run <code><nowiki>./install.sh</nowiki></code>, as its README says. Prerequisites are those of [[DeepSeek V4.1 Flash on 4x 8 GB (Morrowmake)]].
# <code><nowiki>./download.sh</nowiki></code> fetches both checkpoints and builds the hybrid. Budget about 680 GiB.
# Fetch the patch only: <syntaxhighlight lang="bash" inline>hf download pqhaz/DeepSeek-V4.1-Flash-1017-heretic --include "patch/*" --local-dir heretic</syntaxhighlight>
# Write it into the official checkpoint in place with <code><nowiki>heretic_inplace.py</nowiki></code> (below, stdlib Python only):
#: {{Download|file=Heretic_inplace.py|compact=yes}} The script applies the heretic patch to the official shards in place, restores them byte-exact from the backup, or reports their status.
<syntaxhighlight lang="bash">
python3 heretic_inplace.py apply   <MODELS_DIR>/DeepSeek-V4.1-Flash-1017 heretic/patch/<file>.safetensors stock-wo_b-backup.safetensors
python3 heretic_inplace.py status  <same three args>     # expect: heretic 38, stock 0, other 0
python3 heretic_inplace.py restore <same three args>     # undo, byte-exact
</syntaxhighlight>
#: <code><nowiki>apply</nowiki></code> first saves the 38 stock tensors to the backup file (1.6 GB), then overwrites each tensor, fsyncs and reads it back. It refuses any shard that has other hard links, so it cannot patch a second copy by accident. Re-running it does nothing.
#: To avoid patching in place, the heretic repo's own <code><nowiki>ds_patch.py apply <official> <patch> <new_dir></nowiki></code> writes a patched copy instead. It rewrites 38 full shards (about 7 GB each, roughly 260 GB extra) and hard-links the rest. Put that copy under the official checkpoint's directory name: the hybrid's links resolve by that name, and the index and config the builder checks are unchanged.
# <code><nowiki>.env</nowiki></code> changes, all specific to Ivan's box:
#* <code><nowiki>CUDA_VISIBLE_DEVICES=4,5,6,7</nowiki></code> and <code><nowiki>CUDA_DEVICE_ORDER=PCI_BUS_ID</nowiki></code>, because this runs on four cards of an 8-card machine. The recipe does not pass <code><nowiki>CUDA_DEVICE_ORDER</nowiki></code> into the container, so it was added to <code><nowiki>CONTAINER_KEYS</nowiki></code> in <code><nowiki>start.sh</nowiki></code> (a one-word change). On a 4-card box you need neither.
#* <code><nowiki>EXTRA_ARGS="--gpu-memory-utilization 0.92"</nowiki></code>, only because a miner shares the cards and holds 2.5 GiB on each. At the default 0.95 a 17k-token prefill ran out of memory. With nothing else on the cards, keep the default.
#* <code><nowiki>VLLM_ENGRAM_PLACEMENT=ssd</nowiki></code> (default <code><nowiki>auto</nowiki></code>) keeps the Engram tables on SSD instead of in RAM. If you have the RAM, <code><nowiki>auto</nowiki></code> is fine.
#* <code><nowiki>PORT</nowiki></code> and <code><nowiki>SERVED_MODEL_NAME</nowiki></code> are cosmetic.
# <code><nowiki>./start.sh</nowiki></code>, then <code><nowiki>./start.sh smoke</nowiki></code> (one chat request and one tool call).
{{Source|handle=darkmaniac7|date=2026-10-29|note=lab notes}}

== Measured numbers ==
On Ivan's box: four CMP 170HX on PCIe, sharing the machine with a miner, measured with his own bench script. Not
comparable with the numbers the recipe publishes.
{| class="wikitable"
! !! Result
|-
| Decode, 1 user || code 158, structured 136, prose 121 tok/s
|-
| Decode, 8 users (total) || code 323, structured 261, prose 249 tok/s
|-
| Prefill, 17k-token prompt (warm) || 2,453 tok/s
|-
| Long-context copy accuracy (75k to 92k-token roleplay turns that must copy image paths and names exactly) || 1 wrong of 30. The miss picked a different variant of the right image, not a garbled path
|-
| Same 2.9 bpw expert files on a different engine (GNmac) || 85% wrong on the same test. So that engine causes the copy errors, not the quant or the heretic edit
|-
| Refusal spot check (4 prompts, 1,500-token cap) || 2 answered, 2 hit the cap while still reasoning, 0 refused
|-
| Tool calls (<code><nowiki>deepseek_v41</nowiki></code> parser) || work
|}
The heretic model card reports 1 refusal in 100 held-out prompts (base: 97) and KL 0.136 on harmless prompts.
An earlier run under the same conditions read 150 / 131 / 117 tok/s for one user (code / structured / prose); the GNmac engine on the same cards read 248 / 177 / 135 and 542 / 396 / 347 at 8 users, prefill 2,697.
{{Source|handle=darkmaniac7|date=2026-10-29|note=lab notes}}<br />
{{Source|handle=darkmaniac7|date=2026-10-29|note=lab notes}}

== Things to know ==
* It reasons much longer after the edit. The model card puts the median completion at 1,517 tokens against 357 for the base. Set <code><nowiki>max_tokens</nowiki></code> generously, or <code><nowiki>content</nowiki></code> comes back empty with <code><nowiki>finish_reason: "length"</nowiki></code>.
* No capability benchmarks have been run on the heretic version, Ivan's or the model card's. KL 0.136 means general behaviour has shifted somewhat.
* Speed is engine-bound. On Ivan's box the GNmac engine decodes 1.1 to 1.7x faster (prefill 1.1x) but corrupts long-context copying, so he kept Morrowmake. See [[DeepSeek V4.1 Flash on 4x 8 GB (GNmac)]].
* Uncensored EXL3 bundles whose edit lives in dense tensors do nothing on the GNmac engine, which takes dense tensors from the official checkpoint; the heretic patch works on both engines because it patches those official tensors. {{Source|handle=darkmaniac7|date=2026-10-28|note=lab notes}}
{{Source|handle=darkmaniac7|date=2026-10-29|note=lab notes}}

== Checks ==
* After applying the patch Ivan checked the weights: the dequantize/requantize round trip reproduces 100.00% of the original FP8 codes; 92-99% of the patched codes are within one FP8 step of the ideal rank-1 edit; the change is 3.7-4.7% of the weight norm on all 38 tensors. Nothing beyond the advertised edit plus rounding is visible.
* After any restart, confirm which checkpoint the container actually loaded (<code><nowiki>docker inspect <container> --format '{{.Args}}'</nowiki></code>): one reload silently came up on the stock hybrid.
* {{Status|Contested}} Capability: the write-up above says no capability benchmarks have been run on the heretic version. Ivan's chassis notes also cite independent numbers for this layout (GSM8K and HumanEval unchanged, MMLU -0.4, HarmBench compliance 287 of 300 against 30 of 300 stock) without naming who measured them.
{{Source|handle=darkmaniac7|date=2026-10-29|note=lab notes}}

== heretic_inplace.py ==
Ivan's helper, stdlib Python only. It applies or undoes the patch in the official shards for engines that read
attention weights straight from the official checkpoint (Morrowmake).
<syntaxhighlight lang="python">
#!/usr/bin/env python3
"""Apply (or undo) the pqhaz heretic patch IN PLACE in the official DeepSeek-V4.1-Flash-1017 shards, for engines that
read attention weights straight from the official checkpoint (Morrowmake). Only the 38 `layers.N.attn.wo_b.weight`
tensors change (same dtype F8_E4M3, same shape, same offsets); FP8 scales are untouched, as the patch was built.

  heretic_inplace.py apply   OFFICIAL_DIR PATCH.safetensors BACKUP.safetensors
  heretic_inplace.py restore OFFICIAL_DIR PATCH.safetensors BACKUP.safetensors
  heretic_inplace.py status  OFFICIAL_DIR PATCH.safetensors BACKUP.safetensors

apply: writes an exact byte backup of every original tensor first (a safetensors file with the same names), then
overwrites, fsyncs and reads back; refuses a shard with more than one hard link. restore: writes the backup back.
Idempotent: a tensor already in the wanted state is skipped. Stdlib only."""
import hashlib, json, os, struct, sys


def header(path):
    with open(path, "rb") as f:
        n = struct.unpack("<Q", f.read(8))[0]
        h = json.loads(f.read(n))
    return h, 8 + n


def region(path, name):
    h, base = header(path)
    t = h[name]
    a, b = t["data_offsets"]
    return t["dtype"], t["shape"], base + a, b - a


def read_at(path, off, size):
    with open(path, "rb") as f:
        f.seek(off)
        return f.read(size)


def sha(b):
    return hashlib.sha256(b).hexdigest()


def write_safetensors(path, tensors):  # tensors: name -> (dtype, shape, bytes)
    hdr, off = {}, 0
    for k in sorted(tensors):
        dt, shp, data = tensors[k]
        hdr[k] = {"dtype": dt, "shape": shp, "data_offsets": [off, off + len(data)]}
        off += len(data)
    hb = json.dumps(hdr, separators=(",", ":")).encode()
    hb += b" " * ((8 - len(hb) % 8) % 8)
    tmp = path + ".tmp"
    with open(tmp, "wb") as f:
        f.write(struct.pack("<Q", len(hb)) + hb)
        for k in sorted(tensors):
            f.write(tensors[k][2])
        f.flush(); os.fsync(f.fileno())
    os.replace(tmp, path)


def main():
    mode, odir, patch, backup = sys.argv[1:5]
    wm = json.load(open(os.path.join(odir, "model.safetensors.index.json")))["weight_map"]
    ph, _ = header(patch)
    names = sorted(k for k in ph if k != "__metadata__")
    plan = []
    for k in names:
        shard = os.path.join(odir, wm[k])
        dt, shp, off, size = region(shard, k)
        pdt, pshp, poff, psize = region(patch, k)
        assert (dt, shp, size) == (pdt, pshp, psize), (k, dt, shp, size, pdt, pshp, psize)
        plan.append((k, shard, off, size, poff))
    print(f"{len(plan)} tensors in {len({p[1] for p in plan})} shards")

    if mode == "status":
        bh = header(backup)[0] if os.path.exists(backup) else {}
        n_patched = n_stock = n_other = 0
        for k, shard, off, size, poff in plan:
            cur = sha(read_at(shard, off, size))
            if cur == sha(read_at(patch, poff, size)): n_patched += 1
            elif bh and cur == sha(read_at(backup, region(backup, k)[2], size)): n_stock += 1
            else: n_other += 1
        print(f"heretic {n_patched}, stock (matches backup) {n_stock}, other {n_other}")
        return

    if mode == "apply":
        for k, shard, *_ in plan:
            assert os.stat(shard).st_nlink == 1, f"{shard} has other hard links; refusing to write in place"
        if not os.path.exists(backup):
            orig = {}
            for k, shard, off, size, poff in plan:
                data = read_at(shard, off, size)
                if data == read_at(patch, poff, size):
                    sys.exit(f"{k} is already patched and there is no backup: refusing (stock bytes unknown)")
                orig[k] = (region(shard, k)[0], region(shard, k)[1], data)
            write_safetensors(backup, orig)
            print(f"backup written: {backup} ({os.path.getsize(backup) / 2**30:.2f} GiB)")
        src = lambda k, poff, size: read_at(patch, poff, size)
    elif mode == "restore":
        assert os.path.exists(backup), "no backup"
        src = lambda k, poff, size: read_at(backup, region(backup, k)[2], size)
    else:
        sys.exit("mode: apply | restore | status")

    done = skipped = 0
    for k, shard, off, size, poff in plan:
        want = src(k, poff, size)
        if read_at(shard, off, size) == want:
            skipped += 1; continue
        fd = os.open(shard, os.O_RDWR)
        try:
            assert os.pwrite(fd, want, off) == size
            os.fsync(fd)
        finally:
            os.close(fd)
        assert sha(read_at(shard, off, size)) == sha(want), f"read-back mismatch on {k}"
        done += 1
    print(f"{mode}: {done} written, {skipped} already in place, all read back OK")


if __name__ == "__main__":
    main()
</syntaxhighlight>
{{Source|handle=darkmaniac7|date=2026-10-29|note=lab notes}}

== Credits ==
DeepSeek for the model. Morrowmake for the recipe and the Ampere vLLM fork. LibertAI for the EXL3 2.9 bpw quant. pqhaz
for the heretic ablation of 1017. p-e-w for the Heretic method.

== Related ==
* The base recipe: [[DeepSeek V4.1 Flash on 4x 8 GB (Morrowmake)]]. Other heretic and abliterated variants: [[Quantization on the 170HX]].
* [[Engines]], [[Benchmarks]], [[Recipes]].

== Sources ==
* Ivan's lab notes, 2026-10-29 (public write-up of the recipe).
* The script is [[:File:Heretic_inplace.py]]; its sha256 is shown in the Download box in the Steps section.

{{ImportFooter}}
[[Category:8 GB]]