Qwen3.8 Flash Next on 3x or 4x 8 GB (zebgop-ops FP8 PP)

Qwen/Qwen3.8-Flash-Next-FP8 (revision bcd9f01d)
Reported 4 x 8 GB
Quant	Official block FP8, about 126 GiB including the about 50 GiB FP8 n-gram (PLE) table, which is served from host RAM; KV cache bf16 (the QSA layers refuse fp8)
Engine	vLLM: the vllm/vllm-openai:qwen38-flash-next image with zebgop-ops patches 0001 to 0016 (never 0008-WITHDRAWN) and three ported files 0.1.dev20073+g8e685d198; repo commit 2f551bb (patch 0016, 2026-10-02)
Topology	PP4, VLLM_PP_LAYER_PARTITION=12,12,12,12, MTP 3, FULL_AND_PIECEWISE CUDA graphs; PCIe Gen2 x4 per card, no P2P. Three cards: PP3 with 16,17,15 or 16,16,16
Context	262,144 native; 524,288 with YaRN 2.0; 1,048,576 with YaRN 4.0 (needle verified on PP3)
Decode	89 tok/s
Prefill	11100 tok/s
VRAM per card	64 GiB; --gpu-memory-utilization 0.94 plus per-rank KV budgets; measured 34.0-34.3 GiB per 12-layer rank, 37.9 GiB on the drafter rank
Power cap	200 W
Unlock build	cmpunlocker VRAM unlock (64 GiB per card)
Flags	--pipeline-parallel-size 4 --moe-backend humming --enable-prefix-caching --mamba-ssm-cache-dtype float32 --gpu-memory-utilization 0.94 --max-model-len 262144 --max-num-seqs 8 -cc.cudagraph_mode=FULL_AND_PIECEWISE --no-enable-flashinfer-autotune, MTP speculative config with num_speculative_tokens 3, --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3; env VLLM_PLE_CPU_OFFLOAD=1, VLLM_USE_FLASHINFER_SAMPLER=0, VLLM_BT_POOL=1, VLLM_PP_LAYER_PARTITION=12,12,12,12, VLLM_KV_CACHE_MEMORY_RANK0 to RANK3, NCCL_P2P_DISABLE=1, NCCL_IB_DISABLE=1; container --ipc=host --shm-size=32g
Tested	@flipnburner on 2026-09-02
Source	link
Files	serve/run-pp3-mtp.sh, serve/make-patched-tree.sh, patches/ 0001 to 0016, ported-files/, tools/ (loop detector and soak harnesses), Apache-2.0
Confirmations	None yet.

The official FP8 checkpoint of Qwen3.8-Flash-Next served with vLLM pipeline parallel, MTP speculative decoding and prefix caching all at once, on three or four 64 GB cards at PCIe Gen2 x4 with no P2P and a 200 W cap. flipnburn (GitHub zebgop-ops) first posted it on three cards (76 tok/s decode, about 10,000 tok/s prefill) and moved to four cards on 2026-09-02: PP4 with full CUDA graphs reaches 89 tok/s single-stream, 429 to 433 tok/s at 8 concurrent and about 11,000 tok/s prefill, with a 2.5M-token KV pool. The repo also fixes the bug that made PP, MTP and prefix caching unusable together (constant-token "ductductduct" loops, patch 0010) and JSON mode under PP plus speculative decoding (patch 0016). Infobox numbers are the repo's own readings at 200 W; nobody else had re-run them on these cards when this was written.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub, posted by @flipnburner, 2026-08-29

Prerequisites

Three or four 64 GB cards at 200 W on Gen2 x4, no P2P

The repo's numbers come from CMP 170HX cards with the VRAM unlock (Driver unlock, 8 GB card) at 64 GiB each, capped with nvidia-smi -pl 200, every card on PCIe Gen2 x4 with no NVLink and no P2P (NCCL_P2P_DISABLE=1). On the four-card host (buses 01, 21, 41 and 42) every card pair bounces through the host at 0.78 GB/s. The author calls this "about as hostile an interconnect as PP ever sees" and expects stock-power cards to do somewhat better on the compute-bound numbers (prefill, decode at 8 concurrent). The n-gram table lives in host RAM: on three cards flipnburn ran 128 GB of host RAM and used CPU offload because it was the only way the model would fit. Bare metal, Docker with the NVIDIA runtime.

Reported 8 GB
Source: @flipnburner, 2026-08-27
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

Checkpoint and image

Weights: Qwen/Qwen3.8-Flash-Next-FP8, block-wise FP8, about 126 GiB including the about 50 GiB FP8 n-gram (PLE) table. The launcher expects Hugging Face snapshot bcd9f01ddc9cff2316eb84281bebcd5b058bddce (override with QWEN38_SNAPSHOT). QWEN38_REPO serves a sibling checkpoint from the same cache, which must use the official block-FP8 layout.
KV cache stays bf16: --kv-cache-dtype fp8 is refused by the QSA layers (vLLM PR #54846 would add fp8 and nvfp4 KV there).
Image: vllm/vllm-openai:qwen38-flash-next, vLLM 0.1.dev20073+g8e685d198. It was built from a tree whose commit is not on the public release/qwen38next_offload branch; patches-rebased-head/ holds the four model-directory patches rebased onto branch head a5530b90c, and the other patches apply there unchanged (dry-run verified).
Upstream status on 2026-09-02: vLLM PR #53896 (Qwen3.8-Flash-Next support) was merged on 2026-08-31; the CPU PLE offload, PR #53899, was still open.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

HUMMING is the only MoE backend that runs block FP8 on sm_80

--moe-backend humming is load-bearing. On these cards Marlin faults in gptq_marlin_repack and Triton declines block-wise FP8 outright. HUMMING needs about 1.6 GiB of transient repack scratch per card at load, on top of the resident weights; size partitions with that in mind (Known issues).

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

Launch

Build the patched tree and start the server

Unlock all cards to 64 GB and cap them at 200 W (nvidia-smi -pl 200).
Download Qwen/Qwen3.8-Flash-Next-FP8 into a Hugging Face cache.
Clone zebgop-ops/qwen38-flashnext-pp and run serve/make-patched-tree.sh from the repo root. It copies the touched files out of the image, applies patches/ (it skips the WITHDRAWN patch by itself), overlays ported-files/ and writes the result to patched/.
Start serve/run-pp3-mtp.sh with QWEN38_HF set to the cache and QWEN38_PATCH set to the patched/ directory (its default is a path on the author's machine).
Wait about 5 minutes, then check curl -s http://localhost:8001/v1/models; the served model name is qwen38 (QWEN38_PORT and QWEN38_SERVED change them).

git clone https://github.com/zebgop-ops/qwen38-flashnext-pp
cd qwen38-flashnext-pp
./serve/make-patched-tree.sh
QWEN38_PATCH=$PWD/patched QWEN38_HF=/path/to/huggingface-cache ./serve/run-pp3-mtp.sh
docker logs -f qwen38-pp

The launcher refuses to start while one of the author's other model containers (dsv4-a100, glm53flash-pp, glm53nvfp4-pp) is running, because those hold the GPUs. Tunables, all environment variables: QWEN38_PARTITION, QWEN38_GPUS, QWEN38_KV0 to QWEN38_KV3, QWEN38_CG, QWEN38_BT_POOL, QWEN38_SPEC (MTP depth), QWEN38_SEQS, QWEN38_MAXLEN, QWEN38_YARN and QWEN38_EXTRA_ARGS.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

What run-pp3-mtp.sh sets, and why

Despite its name, the launcher has defaulted to four cards since 2026-09-02. It runs --pipeline-parallel-size 4 --moe-backend humming --enable-prefix-caching --mamba-ssm-cache-dtype float32 --gpu-memory-utilization 0.94 --max-model-len 262144 --max-num-seqs 8 -cc.cudagraph_mode=FULL_AND_PIECEWISE --no-enable-flashinfer-autotune --speculative-config {"method":"mtp","num_speculative_tokens":3} with the qwen3_coder tool-call parser and the qwen3 reasoning parser, in Docker with --ipc=host --shm-size=32g.

VLLM_PP_LAYER_PARTITION=12,12,12,12: the number of entries sets the pipeline size. The MTP drafter lives on the last rank.
VLLM_PLE_CPU_OFFLOAD=1 keeps the n-gram table in host RAM; VLLM_USE_FLASHINFER_SAMPLER=0; VLLM_ENGINE_READY_TIMEOUT_S=5400; NCCL_P2P_DISABLE=1 and NCCL_IB_DISABLE=1.
VLLM_KV_CACHE_MEMORY_RANK0 to RANK3 (patch 0014) give each rank its own KV budget. The launcher keys the budgets by partition: for 12,12,12,12 they are 28,089,402,880 / 30,123,884,544 / 30,123,884,544 / 24,216,759,808 bytes, vLLM's own per-rank maxima minus 0.5 GiB on ranks 0 to 2 and 1.0 GiB on the drafter rank. A partition it does not know gets a discovery run at utilization 0.94 instead.
VLLM_BT_POOL=1 whenever the CUDA graph mode is FULL (see patch 0009 below).
--mamba-ssm-cache-dtype float32, and ported-files/fused_recurrent.py (vLLM PR #53877, fp32 GDN decode beta) for output quality.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

Use full CUDA graphs: PP4 stages are CPU-dispatch-bound

Going from three to four stages first cost single-stream speed: with PIECEWISE graphs PP4 ran 59 tok/s (44.2 ms per step) against 70 to 74 on PP3. A cross-rank torch-profiler timeline showed each 12-layer stage busy about 9.2 ms per step for only a few ms of GPU work, the drafter rank 10.0 ms, all three hops 0.4 ms and the relay broadcasts 0.13 ms: the stages are bound by Python launching layers piecewise, and a fourth stage adds a fourth helping. FULL_AND_PIECEWISE replays a whole stage's decode step in one launch: 30.0 ms per step, 89 tok/s. What did not help: MTP 4 (+6% at one user, -12% at 8, a 2.6% smaller pool) and moving layers off the drafter rank (13,13,13,9). The launcher used to pin -cc.cudagraph_mode=PIECEWISE, and the dotted form silently overrides a later --compilation-config: check the engine's Initializing log line for the mode actually in use, not the command line.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

Three cards: PP3 with 16,17,15 or 16,16,16

The launcher still carries budgets for two three-card partitions:

QWEN38_PARTITION=16,17,15 QWEN38_GPUS=0,1,2 ./serve/run-pp3-mtp.sh   # capacity first: 1,359,009 KV tokens, about 70 tok/s at one user
QWEN38_PARTITION=16,16,16 QWEN38_GPUS=0,1,2 ./serve/run-pp3-mtp.sh   # throughput first: 1,116,591 KV tokens, 74 tok/s at one user

16,17,15 was the three-card production default: +45% KV over a uniform budget for about 6% less single-stream decode, because the 17-layer rank sets the step time; 8-way aggregate is unaffected. The two knobs interact: under a uniform budget, shifting layers only makes another rank the limiter, and the per-rank budgets let the lighter rank claim the memory. The PP3 numbers were measured with PIECEWISE graphs; the author expects FULL_AND_PIECEWISE to help there too, but did not post a measurement.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

MTP depth: 3 is the throughput optimum

Sweep on PP3 (decode at one user / at 8 concurrent, median of 4 / draft acceptance):

MTP tokens	1 user (tok/s)	8 concurrent (tok/s)	Acceptance
1	61.3	213.9	78.9%
2	70.5	266.7	67.5%
3	74.6	370.8	57.4%
4	78.8	273.9	49.5%
5	will not boot: the QSA ring capacity 12 must divide the attention block size (vLLM PR #54912 lifts it)

Single-stream keeps improving with depth because deeper drafts amortize the trip through the pipeline, but at 8-way the longer window turns into wasted verify work. On PP4 the launcher notes MTP 4 at 63 / 327. Default QWEN38_SPEC=3. Background: Speculative decoding on the 170HX.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

As published, the tree builder does not produce the patch 0016 files

Read from the scripts at commit 2f551bb, not run: run-pp3-mtp.sh checks for and mounts spec_decode_utils.py and engine_core.py (patch 0016), but make-patched-tree.sh does not copy vllm/v1/worker/gpu/spec_decode/utils.py or vllm/v1/engine/core.py out of the image, so 0016 cannot apply and the launcher stops with "missing patch". The 0016 hunk for the model runner (dropping a request's drafts when it ends) is also not in ported-files/v2_model_runner.py, which the builder copies over the patched runner. If the launcher stops there, copy those two files out of the image, apply 0016 to them, and apply its runner hunk to the ported runner.

Theory 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

Patch 0010: the PP + MTP + prefix caching loop fix

Symptom: 14 to 27% of requests turn into "ductductduct" loops

With pipeline parallel, MTP and prefix caching combined and three or more concurrent requests, generation degenerated into constant-token loops ("ductductduct...") on 14 to 27% of requests in every one of about 15 instrumented configurations. The loop token is 1023, which is what the sampler emits for all-NaN logits. flipnburn asked on Discord (2026-08-29, again on 2026-09-03) whether others had hit it; Ryko, whose vektorprime/qwen38-flash-next-pp2 had appeared 12 hours earlier, had only seen an MTP rank-split issue and had never tested more than 2 concurrent requests.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub, posted by @flipnburner, 2026-08-29
Source: @vektor_prime, 2026-08-29

Cause and fix: recurrent state copied through stale block tables

Under PP, pp_size+1 steps are in flight and the per-step gathered block tables are reused buffers. vLLM's mamba spec-decode context captured raw pointers to those tables and indexed them by batch row; on non-last ranks its deferred postprocess runs pp_size steps late, so it copied recurrent state through other requests' freed or reallocated blocks. In this model's unified cache layout, KV, GDN state and PLE conv state share the same pages, so the copies land in other requests' live state: silent NaN corruption on sm_80, an illegal memory access on sm_121 (vLLM issue #54173 is the loud form of the same fault). Patch 0010 makes the context use the stable per-request block tables and index rows by request. Ablation with an 8-way soak, prefix caching on: 0010 alone 0/160 clean; 0009 alone 10/24 corrupt by round 3; both 0/160 and 0/168. FINDINGS.md in the repo logs the full investigation, including fifteen hypotheses ruled out by measurement.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

Patch 0009 is optional, and breaks under full CUDA graphs

0009 (a rotating pool of gathered block-table sets, VLLM_BT_POOL of pp+2 or more) is defense-in-depth only and does not fix the bug by itself. With full CUDA graphs it brings the corruption back: a captured graph keeps the pointer of the one pool slot current at capture, so most replays read stale tables (VLLM_BT_POOL=6 with full graphs gave 10/144 "duct" loops after about 10 minutes). The launcher forces VLLM_BT_POOL=1 whenever the mode is FULL; do not raise it. With that setting PP4 passed a 0/168 soak, 16 conversations x 8 turns with 0 loops (46/48 planted-code recalls), an exact 250k needle and 8 concurrent about 241k-token needles (8/8 exact, 1.93M tokens resident).

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

The rest of the patch series

Enablement (each one gated a startup crash or a measured defect): 0001 skips hyper_connection_mixer weights on non-last ranks; 0002 lets the MTP drafter run on the last PP rank; 0003 and 0004 replace the blanket "PLE requires PP=1" refusal with the real condition (every PLE layer on rank 0); 0005 clears the PP partition variable in the PLE offload worker; 0006 adds a staging handshake to the PLE connector (under PP the next forward outran staging and queue.Full killed rank 0); 0007 skips the connector on ranks without a PLE layer.
Correctness: 0010 (above); 0009 (optional); 0011 ports vLLM PR #48375 (resume-path state poisoning); 0012 fixes vLLM issue #53142 (state-seed divisor); 0013 zero-initializes the PLE spec-extension state columns; 0014 adds per-rank KV budgets (+19.4% KV pool, and what makes 1M context fit); 0015 normalizes mamba specs from an attention-less stage; 0016 fixes JSON mode under PP and speculative decoding (next section).
ported-files/ ships whole files: the vLLM PR #46994 MTP-under-PP relay (pp_utils.py), the matching V2 model runner, and the fp32 GDN beta kernel.
Upstream PRs the author was watching for this setup: #52297 (GDN metadata once per step; the PP4 stages are CPU-dispatch-bound), #54296, #48815, #54846 and #54912.

A longer write-up by @flipnburner exists ("Eight Patches to Pipeline Parallel", with a section on running it yourself); see Discord.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub
Source: @flipnburner, 2026-08-27

Patch 0016: JSON mode under PP and speculative decoding

HTTP 500 "grammar rejected tokens" whenever requests overlap

Before patch 0016 (2026-10-02), response_format requests failed with HTTP 500 (scheduler.py: Unexpected: grammar rejected tokens) at concurrency above 1: 242 failures in 24 hours of real traffic. With speculative decoding the scheduler needs each request's draft tokens to build the grammar mask, but the worker kept only the latest batch's drafts. Under PP a request comes back pp_size steps later with other batches sampled in between, so its drafts stayed placeholders, the speculative positions went out unconstrained and the bonus token was sampled under a mask three tokens stale. 0016 keeps drafts per request, also fetches them on the engine's non-deferred path, and drops them when a request ends. Measured on PP4, MTP 3, xgrammar:

Test	Before	With 0016
json_object, 1-way	0/8 fail	0/10 fail
json_object, 4-way	10/12 fail	0/24 short, 0/16 at 700 tokens
json_object, 8-way	not run	0/32, 0/16
strict json_schema, 1 and 8-way short, 4 and 8-way long	not run	16/16, 64/64, 16/16, 16/16 conforming
thinking on + json_schema, 4-way	not run	16/16
plain decode, 1 / 8 concurrent	89 / 429-443 tok/s	89.9 / 442 tok/s

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

Test JSON mode with a strict json_schema probe, not json_object

The first version of 0016 pruned drafts with tombstones and dropped a new request's first drafts, because the runner removes every new request id once before adding it. Strict json_schema then died right after the opening brace (0/16 at 1-way), while json_object returned HTTP 200 with the first key silently replaced by an empty string. json_object cannot detect a stale mask (any string is a legal key), so a passing json_object run proves nothing; tools/json_schema_probe.py in the repo is the check to use.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

Measured numbers

Four cards, PP4 with full CUDA graphs (2026-09-02): 89 tok/s, 429 to 433 at 8, about 11,000 prefill

	PP3 16,17,15, PIECEWISE (old default)	PP4 12,12,12,12, PIECEWISE	PP4 12,12,12,12, FULL_AND_PIECEWISE
Decode, 1 user (tok/s) / step	70-74 / 34.8 ms	59 / 44.2 ms	89 / 30.0 ms
Decode, 4 / 8 concurrent (tok/s)	248 / 371-440	199-254 / 372	254 / 429-433
Prefill, one 32k prompt (tok/s)	9.6k	11.2k	11.1k-11.3k
Prefill, 250k needle (tok/s)	8.6k	10.3k	10.6k
KV pool (tokens)	1,359,009 (5.18x a full 262k)	2,537,837	2,537,837 (9.68x)

Decode as reported at one user: 89 tok/s (30.0 ms per step); prefill as reported: 11.1k-11.3k tok/s on a 32k prompt. Settings: MTP 3, prefix caching on, n-gram table on the CPU, 200 W per card. Measured per-rank footprint: 34.0 to 34.3 GiB for 12 layers, 37.9 GiB for 12 layers plus the drafter. A discovery boot at uniform utilization 0.94 gave 2,309,701 tokens (the drafter rank binds); the per-rank budgets raised it to 2,537,837, within 0.1% of the projection. Other four-card layouts for this model are compared on Qwen3.8 Flash Next on 4x 8 GB.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub, posted by @flipnburner, 2026-08-29

Three cards (August 2026): 76 tok/s decode, about 10,000 prefill, 440 at 8

flipnburn (2026-08-27) reported 10k tok/s prefill and 76 tok/s decode for the FP8 checkpoint on three cards with 128 GB of host RAM; anon33anon said the Flash-Next architecture is designed for offload, so keeping the table on the CPU does not hurt speed. On 2026-08-29 he published the repo: "Qwen38-flash-next-fp8 on 3 cmp170s 76 tokens/s decode and 10,000 tokens/s prefill, 440 tokens/s decode at concurrency 8" (prefill without cache hits). matstatman and Studebaker took part in the threads. The repo's PP3 tables (balanced 16,16,16, median of 3 or more runs, 200 to 300 generated tokens, cache-defeating prompts):

Concurrency	1	2	3	4	8
Aggregate (tok/s)	74.0	about 120-152	about 121-200	247.8	440.5
Per stream (tok/s)	74.0	about 50-76	about 40-67	about 62	about 55

Story-style prompts at temperature 1.0 run about 66 tok/s single-stream (lower draft acceptance). Prefill is about 9,600-10,500 tok/s for 32k-token prompts at concurrency 1 to 3 (one 32k prompt fills the pipeline); earlier profiling gave about 5.7k at 8k, 9.2k at 32k, 10.0k at 110k and 8.6k at 250k prompt tokens. Aggregate decode at 4 or more concurrent varies a lot from run to run with MoE routing.

Reported 8 GB
Source: @flipnburner, 2026-08-27
Source: @anon33anon, 2026-08-27
Source: @matstatman, 2026-08-27
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub, posted by @flipnburner, 2026-08-29
Source: @vektor_prime, 2026-08-29
Source: @_studebaker_, 2026-08-29

Prefix caching: 3.8x faster TTFT at about 60% hit rate

A 20k-token prompt: TTFT 5.12 s cold, 1.35 s warm. Conversation-shaped traffic (16 growing conversations x 8 turns over a 40k shared document, 8 concurrent): about 60% token hit rate, 0 degenerate loops and 48/48 exact recalls of 5-digit codes planted three turns earlier, the check that catches silent state corruption that liveness tests miss. Allocation-path soak (unique prompts, 0% hits, 8-way, about 7k-token prompts, 1,500-token generations): 0/160 and 0/168 clean.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

The n-gram table on the CPU costs about 2% of a decode step

Instrumented at one-user decode (medians over 200 forwards): forward period 34.81 ms, total offload service 0.69 ms (CPU n-gram lookup 0.51, host-to-device copy enqueue 0.09, CPU blocked on the GPU 0.04). That is an upper bound, and the CPU worker runs ahead of the model rather than gating it, so serving the table from host RAM is not a reason to add cards. It could not be A/B tested on three cards: with the table in VRAM, rank 0 holds 55.2 GiB for layers 0 and 1 and the other 46 layers do not fit on two cards. On four cards a partition starting with 2 (rank 0 holding only the embedding and the table) is refused by vLLM's cache layout planner ("pipeline stage has mamba cache owners but no main_kv tensor slots"), because the state pages are carved out of the same stage's attention tensors.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

PP beats expert parallel on this interconnect

On the same three cards PP3 was about 5x faster at prefill than expert parallel (DEP3) and had about 4x the KV capacity. With expert parallel every one of the 48 layers does an all-to-all over Gen2 x4 without P2P; under PP each stage owns all experts for its layers, so routing stays on the card and only hidden states cross stage boundaries. More on layouts: Parallelism over the PCIe link.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

Quality: MMLU 89.0%, HumanEval+ 94.5%, MBPP+ 83.1%

Scored on the author's box with the repo's tools/quality.py: 1,000 MMLU questions (cloze-scored, no chat template) 89.0%, HumanEval+ 94.5% and MBPP+ 83.1% (executed pass@1, sandboxed), wikitext 0.4394 bits per byte. On the same harness GLM-5.3-Flash W4A16 scored 85.6 / 92.7 / 83.3% and DeepSeek-V4.1-Flash 84.4 / 93.9 / 84.9%; the author finds no pair separable on code, and MMLU is where they differ.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

Warm up before judging, and use the performance governor

Right after boot the first 32k prefill read 5.4k tok/s (JIT compilation) and the first 8-way run 216 tok/s; warm they are 11.2k and 372 or more. The first run after boot reads 20 to 40% low and is discarded in the repo's tables. Because the stages are CPU-dispatch-bound, the host CPU governor matters a little: performance instead of powersave (amd-pstate-epp) took one-user decode from 84.9 to 89.2 tok/s, tightened the spread and moved 8-way from 429 to 443, a 2 to 5% gain.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

YaRN to 1M

524,288 tokens with YaRN 2.0, 1,048,576 with YaRN 4.0

The native 262,144-token context is verified (a 260k-token needle recalled in 35.7 s). QWEN38_YARN turns on Qwen's static YaRN: the launcher adds VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 and an --hf-overrides block that sets the text config's rope parameters to rope_type yarn, the chosen factor, original_max_position_embeddings 262144, rope_theta 10000000, partial_rotary_factor 0.25, mrope_section 11,11,10 and mrope_interleaved true.

QWEN38_YARN=2.0 QWEN38_MAXLEN=524288: a needle at 500,128 tokens recalled in 67 s, short-prompt answers unchanged.
QWEN38_YARN=4.0 QWEN38_MAXLEN=1048576 on PP3: boots with a 1,220,499-token pool (1.16x one full sequence) thanks to patch 0014, and recalled a needle from a 1,041,971-token prompt exactly in 189.6 s (about 5.5k tok/s prefill at that scale).

YaRN stays off by default: Qwen warns that static YaRN is the same at every input length and can cost quality on short texts, which matters on a server that mostly sees short requests. An older comment inside the launcher still says factor 4.0 does not fit (pool about 957k tokens); that predates patch 0014, and the launcher header and RESULTS.md give the 1M result. Four cards have about twice the pool (2,537,837 tokens at 262k), but the 1M needle was posted for PP3 only.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub, posted by @flipnburner, 2026-08-29

Known issues

Never apply patches/0008-WITHDRAWN

0008 is a falsified experiment (a "placeholder verification gate") kept as history. Left in, it rejects every draft for a request rescheduled within pp_size steps, which at one user is every step: MTP falls to 1 token per step and single-stream decode to about 30 tok/s, while the acceptance metric still reads about 40%. Removing it restored 74.0 / 247.8 / 440.5 tok/s at 1 / 4 / 8 concurrent with the soaks still clean. make-patched-tree.sh skips it; do not apply it by hand.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

Over-committing a rank faults instead of OOM, wedges GSP firmware and needs a reboot

A 2,23,23 partition needed 67.8 GiB on the last rank. Instead of a clean out-of-memory error it faulted at load with an illegal memory access and wedged the GSP firmware (NVRM: GspRmAlloc failed status=0x62); only a reboot recovered. Size partitions from measured per-rank totals (34.0 to 34.3 GiB per 12 layers, 37.9 GiB with the drafter) plus the about 1.6 GiB HUMMING repack scratch, and never let a partition inherit another partition's VLLM_KV_CACHE_MEMORY_RANK budgets: the launcher keys them by partition for that reason, and an unknown partition gets a discovery run. Xid background: Xid errors.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

--no-async-scheduling deadlocks under PP

vLLM PR #50021 suggests --no-async-scheduling as a workaround for the same fault family on sm_86. Under PP3 it deadlocks (the MTP relay collectives assume the async cadence), so it is not an option here; patch 0010 is the fix.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

Qwen3.8 Flash Next on 4x 8 GB · Qwen3.8 Flash Next on 3x 8 GB (clee PP3) · Qwen3.8 Flash Next on 2x 8 GB (vektorprime PP2) · Qwen3.8 Flash Next on 2x 8 GB (Qwen-Flash-SM80, GDS TEP2) · Parallelism over the PCIe link · Speculative decoding on the 170HX · vLLM on sm_80: known bugs and patches · Undervolting and power limits · P2P · Engines · Benchmarks

Sources

Repo: zebgop-ops/qwen38-flashnext-pp (Apache-2.0; patches and ported files contain code derived from vLLM; last commit 2f551bb, 2026-10-02), posted on Discord by flipnburn (handle flipnburner). README.md, RESULTS.md, FINDINGS.md and serve/run-pp3-mtp.sh hold the numbers above; they are the author's own runs.
Discord threads: 2026-08-27 (first numbers), 2026-08-29 (the repo), reposted 2026-09-03.
Pointer: the "Eight Patches to Pipeline Parallel" write-up by @flipnburner, not reproduced here.

Prerequisites

Three or four 64 GB cards at 200 W on Gen2 x4, no P2P

Checkpoint and image

HUMMING is the only MoE backend that runs block FP8 on sm_80

Launch

Build the patched tree and start the server

What run-pp3-mtp.sh sets, and why

Use full CUDA graphs: PP4 stages are CPU-dispatch-bound

Three cards: PP3 with 16,17,15 or 16,16,16

MTP depth: 3 is the throughput optimum

As published, the tree builder does not produce the patch 0016 files

Patch 0010: the PP + MTP + prefix caching loop fix

Symptom: 14 to 27% of requests turn into "ductductduct" loops

Cause and fix: recurrent state copied through stale block tables

Patch 0009 is optional, and breaks under full CUDA graphs

The rest of the patch series

Patch 0016: JSON mode under PP and speculative decoding

HTTP 500 "grammar rejected tokens" whenever requests overlap

Test JSON mode with a strict json_schema probe, not json_object

Measured numbers

Four cards, PP4 with full CUDA graphs (2026-09-02): 89 tok/s, 429 to 433 at 8, about 11,000 prefill

Three cards (August 2026): 76 tok/s decode, about 10,000 prefill, 440 at 8

Prefix caching: 3.8x faster TTFT at about 60% hit rate

The n-gram table on the CPU costs about 2% of a decode step

PP beats expert parallel on this interconnect

Quality: MMLU 89.0%, HumanEval+ 94.5%, MBPP+ 83.1%

Warm up before judging, and use the performance governor

YaRN to 1M

524,288 tokens with YaRN 2.0, 1,048,576 with YaRN 4.0

Known issues

Never apply patches/0008-WITHDRAWN

Over-committing a rank faults instead of OOM, wedges GSP firmware and needs a reboot

--no-async-scheduling deadlocks under PP

Related

Sources