Jump to content
Toggle search
Toggle menu
CMP Wiki
Toggle preferences menu
Toggle personal menu
Not logged in
Your IP address will be publicly visible if you make any edits.

Qwen3.8 Flash Next on 2x 8 GB (vektorprime PP2)

From CMP Wiki
More actions
Qwen3.8-Flash-Next
Reported 2 x 8 GB
QuantAWQ INT4: cyankiwi/Qwen3.8-Flash-Next-AWQ-INT4; 95 GiB BF16 PLE table in host RAM (FP8 table optional)
EnginevLLM nightly + PR #53899 (a5530b9) + the repo's patch scripts 0.1.dev1+ga5530b90c.d20260829
TopologyPP2 with PLE CPU offload
Context262,144
Decode75.4 tok/s
VRAM per card64 GiB; weights PP0 39.48 GiB, PP1 45.53 GiB; --gpu-memory-utilization 0.9
Flags--pipeline-parallel-size 2 --max-num-seqs 8 --mamba-cache-mode align --mamba-ssm-cache-dtype float32 --speculative-config mtp --enable-prefix-caching --async-scheduling; VLLM_PLE_CPU_OFFLOAD=1, VLLM_GDN_DECODE_KERNEL=triton, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
Tested@vektor_prime on 2026-08-29
Sourcelink
ConfirmationsNone yet.

A build recipe and vLLM patch set that serves an AWQ INT4 Qwen3.8-Flash-Next on two 64 GB cards with pipeline parallel 2 plus PLE CPU offload, a combination upstream vLLM refuses in six places. The author avoids TP2 because these cards sit on slow PCIe lanes and TP all-reduce dies there. Later parts add prefix-caching fixes, an FP8 PLE table, a deterministic-MoE patch and sampled MTP drafting; the author's production moved to PP3 (two 64 GB cards plus a 40 GB card). Infobox decode is single stream with MTP k=1. Source: repo github.com/vektorprime/qwen38-flash-next-pp2 by vektorprime on GitHub, posted by @vektor_prime, 2026-10-07

Prerequisites

  • Two GPUs with at least about 48 GiB each (two CMP 170HX at 64 GiB; Driver unlock, 8 GB card).
  • At least about 110 GiB of free host RAM for the PLE offload (author ran 178 GiB), about 250 GiB of disk for the HF cache (174 GB, 38 shards), Docker with the NVIDIA Container Toolkit, a Hugging Face token for the checkpoint (passed as a file, never stored).

Source: repo github.com/vektorprime/qwen38-flash-next-pp2 by vektorprime on GitHub, posted by @vektor_prime, 2026-10-07

Steps

git clone https://github.com/vektorprime/qwen38-flash-next-pp2 && cd qwen38-flash-next-pp2
docker build -t qwen38-flash-next-pp2:latest -f Dockerfile .
bash scripts/launch.sh /path/to/hf_token.txt
  • The Docker build must print exactly 23 "patched" lines (11 from patch_pp.py, 12 from patch_p0.py). Any "WARNING ... pattern not found" means the PR head moved and the image is unsafe.
  • Launch (scripts/launch.sh, port 8001): --pipeline-parallel-size 2 --gpu-memory-utilization 0.9 --max-num-seqs 8 --mamba-cache-mode align --mamba-ssm-cache-dtype float32 --speculative-config '{"method":"mtp","num_speculative_tokens":3}' --enable-prefix-caching --async-scheduling --reasoning-parser qwen3 --tool-call-parser qwen3_coder; env VLLM_GDN_DECODE_KERNEL=triton VLLM_BT_POOL=5 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_DEVICE_ORDER=PCI_BUS_ID; Docker --ipc=host --shm-size=96g.
  • The PLE table lives on exactly one layer (language_model.model.layers.1.ple), which always lands on PP rank 0, so the upstream PP=1 restriction is a set of guards, not an architectural limit; the six patch sites remove them.

Source: repo github.com/vektorprime/qwen38-flash-next-pp2 by vektorprime on GitHub, posted by @vektor_prime, 2026-10-07

Measured numbers

  • Verified 2026-08-29: KV cache 1,196,679 tokens (14.97 GiB), 4.56x concurrency at full context; cold start about 9 minutes.
  • MTP on PP2 with AWQ INT4: k=1 gives 75.4 to 75.8 tok/s single stream against 58.1 to 58.6 without MTP (+29%); k=3 gives 64 to 65 tok/s (+11%). Per-position acceptance 0.61 / 0.40 / 0.22, so the author recommends k=1 (a later part validates k=2).
  • FP8 PLE table (VLLM_PLE_FP8_TABLE=1): host RAM for the table 96.8 to 48.4 GiB; KLD against BF16 0.0163, top-1 95.98%. Production went back to the BF16 table on 2026-10-02.
  • On the PP3 rig: batch-invariance stack 73 tok/s decode; sampled MTP drafting raised accept length 2.354 to 2.806 and decode 52.5 to 61.9 tok/s.

Source: repo github.com/vektorprime/qwen38-flash-next-pp2 by vektorprime on GitHub, posted by @vektor_prime, 2026-10-07

Known issues

  • PYTORCH_ALLOC_CONF (wrong name) is silently ignored; the allocator then runs non-expandable and weight-load fragmentation kills a worker at about 41 of 63.5 GiB with 22 GiB "free".
  • "No CUDA GPUs are available" with a healthy nvidia-smi is usually a --restart always crash loop holding the card: docker rm -f, confirm 0 MiB, relaunch.
  • MTP plus PP2 deadlocks in warmup (NCCL BROADCAST 600 s timeout) without the zebgop-ops pp_utils.py draft relay.
  • PP + MTP + prefix caching + mamba align degenerates into repeated tokens ("duct duct duct") from all-NaN logits after a few dozen 8-way requests without patch P0-0010; patches 0011 and 0012 together fix cross-request state bleed (never ship 0011 alone; do not apply 0008-WITHDRAWN).
  • moe_align_block_size GPU atomics made repeat runs differ (top-1 differs at 3.6% of positions); patch 0015 (VLLM_MOE_DETERMINISTIC_ALIGN, default on) costs 4 to 6 us per MoE layer.
  • Pinned to PR #53899 at a5530b9 and breaks when the PR head moves. Several images are made by docker cp and docker commit, so the production lineage is not reproducible from one build. No license file.

Source: repo github.com/vektorprime/qwen38-flash-next-pp2 by vektorprime on GitHub, posted by @vektor_prime, 2026-10-07

Related

Sources

Imported from the cmpunlocker Discord with credit; see Credits.
Retrieved from "https://cmp.wiki/w/index.php?title=Qwen3.8_Flash_Next_on_2x_8_GB_(vektorprime_PP2)&oldid=277"