vLLM on sm_80: known bugs and patches

The CMP 170HX is a GA100 card, compute capability sm_80. Upstream vLLM targets Hopper, Blackwell and Ada first, has no sm_80 path for some attention kernels, and rarely runs the pipeline parallel plus speculative decoding layouts that CMP rigs depend on, so every CMP vLLM recipe carries a patch set. The entries below are the bugs owners hit, what fixed them and which patch set carries the fix. Engine choice and the list of forks are on Engines; drafter tuning is on Speculative decoding on the 170HX; layouts are on Parallelism over the PCIe link.

First checks when a vLLM setup misbehaves, each explained below:

Use the exact vLLM build or nightly the recipe's patches target; patch scripts break silently on newer ones.
When a setup that used to work breaks, clear the compile and kernel caches before anything else.
For wrong or garbled output, make the fault reproducible as M out of N runs, then bisect with MTP off and CUDA graphs off.
With MTP, DSpark or DFlash, send temperature 0 and check the logged acceptance; a very low acceptance under PP points to a drafter bug, not the model.
Serve schema-constrained requests (forced tool calls, JSON schema) from a config without speculative decoding unless the build fixes structured output under speculation.
Measure with vLLM's own benchmark or a wall clock, not with other tools' numbers or vLLM's logged rates.

Why sm_80 needs patches

No native FP8, INT8 KV cache Triton errors, and Gemma 4 refusing an FP8 KV cache

Many vLLM kernels target Hopper, Blackwell and sm_89 (Ada), and getting everything working on sm_80 is why DeeJay's forks need so many patches (2026-08-27). sm_80 has no native FP8, so an FP8 KV cache cuts prefill speed, and running the KV cache at INT8 throws Triton kernel errors; giaki3003 was surprised by the size of the sm_89 to sm_80 gap. Smit (2026-08-22): on Ampere, vLLM nightly defaults to a BF16 KV cache and, for Gemma 4 specifically, refuses an FP8 cache unless the GPU is sm_89 or newer; he had Claude patch vLLM so the KV pool can be stored in FP8 on the 170HX. Studebaker said an FP8 cache normally works in vLLM without anything special, and both agreed the restriction is Gemma 4 specific. llama.cpp handles it but did not suit Smit's serving setup, and Studebaker called Gemma 4 slow and fairly broken and suggested Qwen instead. FP8 and other formats on these cards: Quantization on the 170HX.

Reported
Source: @smit1237, 2026-08-22
Source: @_studebaker_, 2026-08-22
Source: @deejay9983, 2026-08-27
Source: @giaki3003, 2026-08-27

Upstream vLLM PRs and issues the CMP patch sets build on

Every CMP vLLM fork and patch set (haosdent/vllm@dsv4-flash-a100, allover326's patch sets, bayley's GLM patches, wtdcode/vllm-backport, Morrowmake/vllm-cmp170hx) sits on upstream vLLM, which lacks sm_80 paths for DSA sparse MLA and DeepSeek V4. The upstream threads that come up:

Issue #50576 "SM8x (Ampere A100/A800) support for DeepSeek-V4-Flash", posted by giaki3003 on 2026-08-02 ("83/104 tok/s apparently"): the origin of haosdent/vllm@dsv4-flash-a100. Its working launch used --tokenizer-mode deepseek_v4.
PR #38476 (TRITON_MLA_SPARSE, a pure-Triton sparse-MLA backend for SM8x): the basis for GLM-5.x DSA models on these cards. asm64 suggested PP plus #38476 on 2026-08-02, and bayley hand-applied it onto vLLM 0.26.0.
PR #46994 (spec-decode draft relay over PP): carried as a patch because it was not merged as of August 2026.
PR #49897: its SM12x torch top-k fallback is enabled on SM8x by the DeepSeek recipe. PRs #49139 and #50201 did not move the context ceiling of about 134k.
PR #55081: matstatman got MTP k=3 working with it on 2026-09-04 for Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 on two cards with PP via Ray. Decode 70.04 tok/s on fresh code and 82 on edit-heavy work (acceptance 75.7% and 99.6%), against about 38 to 39 without speculation; prefill 4,395.5 tok/s at about 8K, 4,726.4 at about 32K and 3,045.7 at about 100K.
PR #48375 ("Honor drop_eagle_block in MambaManager") and issue #54360: see the prefix caching entry below.
Discussion #17472, posted by Ryko on 2026-08-15: why vLLM's logged prefill and decode rates mislead. Members recommended wall-clock timing.

matstatman recommended examples/observability/prometheus_grafana/docker-compose.yaml for monitoring. Versions seen in recipes: vLLM 0.24.0 (Highwayaiexpose benchmarks), 0.26.0 (bayley's GLM patches, the allover326 dsa-mtp kit), 0.29.0 and 0.30.0 in later single-card and TP2 recipes.

Reported 8 GB and 10 GB
Source: @giaki3003, 2026-08-02
Source: @vektor_prime, 2026-08-15
Source: @roger_dammit, 2026-08-18
Source: @matstatman, 2026-08-23
Source: repo github.com/vllm-project/vllm by vllm-project on GitHub

Crashes, load failures and Xid 31

"illegal memory access was encountered" on MoE models: stay on vLLM 0.21 (July 2026 advice)

greencandle could only run models in llama.cpp; vLLM failed with "illegal memory access was encountered" (2026-07-29). Merom Green: stick to vLLM 0.21 for MoE models, since he hit the same error on 0.22 and later while dense Qwen ran fine for him, and try W8A16 quantization. greencandle could not get Minachist/Qwen3.6-27B-INT8-AutoRound running on vLLM and gets 45 tok/s on llama.cpp with Qwen3.6-27B-MTP-GGUF. Studebaker replied with a working config (in an earlier message that was not captured). This was July 2026 advice: the later recipes run vLLM 0.26.0 to 0.30.0 with their own patch sets (versions in the entry above).

Reported
Source: @greencandle9753, 2026-07-29
Source: @meromgreen, 2026-07-29
Source: @_studebaker_, 2026-07-29

A GLM recipe crashes at load unless PYTORCH_CUDA_ALLOC_CONF="expandable_segments:False"

Dimon reported that with Morrowmake's GLM recipe versions 1.3 and 1.4 the model would not load and crashed with a memory error unless he set PYTORCH_CUDA_ALLOC_CONF="expandable_segments:False" (2026-09-28). Morrowmake asked for the failed startup log by DM and Dimon sent it. Dimon also asked which benchmark tool produced the repo's numbers, to compare TP4 on Gen2 x4. Recipe: GLM 5.3 Flash on 4x 8 GB. Recipes differ on this setting, so use the value in the recipe's own launch script; per-engine notes are on Engines.

Reported
Source: @dimon_91153, 2026-09-28
Source: @voidgiraffe, 2026-09-28

Xid 31 on long DeepSeek V4 prefills: allover326 patches 0005a and 0006

DeeJay's allover326/deepseek-v4-cmp170hx patch set for DeepSeek-V4-Flash-0731 (over haosdent/vllm branch dsv4-flash-a100) carries two sm_80 long-context fixes. Patch 0005a is a required sm_80 prefill top-k torch fallback: without it, prefills above about 128k raise Xid 31. DeeJay, coding with Claude, found that above 128k the CUDA path had been dropped for sm_80. Patch 0006 lifts the context ceiling: giaki3003's agent explained that DSV4_LOGITS_ROW_CHUNK tiles the indexer-logit calculation into query-row pieces, removing a memory wall at about 123 to 150K at the cost of throughput at extreme lengths. Measured chunk sizes: 0 dies at about 134k with Xid 31, 256 reaches about 957,600 tokens, 128 reaches 1,047,736 in one shot but an accumulating chat dies at about 718k to 733k, and 64 reached 1,002,852 tokens over a 405-turn chat, so use 64 for conversations. Do not run the recipe with --enforce-eager (8 to 10 tok/s). Full recipe: DeepSeek V4 Flash 0731 on 4x 8 GB (allover326 PP4 DSpark). Patch 0006 failing to apply on newer vLLM is on DeepSeek V4 and V4.1: tool-call leaks, loops and crashes.

Reported 8 GB
Source: @giaki3003, 2026-08-03
Source: @deejay9983, 2026-08-03
Source: repo github.com/allover326/deepseek-v4-cmp170hx by allover326 on GitHub, posted by @deejay9983, 2026-08-05

Xid 31 once the KV pool passes 2^31 bytes per layer: int64 sparse-MLA offsets (promisezackr)

promisezackr's 24-patch set serves GLM-5.3-Flash NVFP4 on 8 cards with PP8 and DFlash2 over the official vllm/vllm-openai:glm53-flash image (v0.1.dev20051+g487ecf187). One of its fixes (patch 0023) moves the sparse-MLA KV offsets to int64: with int32 offsets the card hit an Xid 31 once the KV pool passed 2^31 bytes per layer. The same set loads the MTP draft embed_tokens and shared_head under PP (0 of 3,800 drafts were accepted before), backports DFlash2 from vLLM PR #52816 under PP8, scatters draft tokens without a sync (about 106 to 49 ms per step) and re-blocks the drafter sidecar KV from 16 to 256 tokens (1M concurrency 1.13x to 6.31x). The patches target that one image build, and DFlash needs prefix caching off in the original recipe (the lookahead corrupts output). Zanooda runs them as the image zanooda/vllm-sm80-glm53flash:pp8-dflash2 on 6 cards. Recipe: GLM 5.3 Flash on 8x 8 GB (promisezackr PP8 NVFP4 DFlash2). Xid 31 causes that are not engine bugs: Xid 31 and the reserved top of VRAM.

Reported 8 GB
Source: repo github.com/promisezackr/glm53-flash-170hx-pp8 by promisezackr on GitHub, posted by @zanooda, 2026-08-31

Patch-pattern warnings on newer nightlies: pin the nightly a patch script targets

aceshigh asked Ryko for the exact vLLM nightly his docker build is based on, because the latest nightly prints warnings during patching: "connector launch pattern not found", "connector init pattern not found" (falls back to an alternative connector queue), "connector copy pattern not found" and "connector put pattern not found", while the mamba_utils, block_table, MambaManager drop_eagle_block, mamba_hybrid divisor and ple_layer patches apply (2026-08-31). Lesson: pin the vLLM nightly that a patch script targets. No answer in the thread.

Reported
Source: @vektor_prime, 2026-08-30
Source: @aceshigh555, 2026-08-31

A corrupt cache broke a 6-card setup

Schaka said his 6-card setup misbehaving ("corrupted") turned out to be a corrupt cache, possibly from an intermittent experimental run (2026-09-12). Which cache (compile, kernel or KV) was not specified. Clearing caches is a first step when a previously working multi-card setup breaks.

Reported
Source: @schakaqt, 2026-09-12

Speculative decoding faults

wtdcode/vllm-backport: MTP acceptance only about 15% under PP because the drafter used a random embedding

OB88594 confirmed that MTP in wtdcode/vllm-backport had a fault with only about 15% acceptance, and temporarily switched to the club-170 backend (2026-09-10). Humvee55 hit the same bug: the backport skipped the top-level embedding in Glm5NextMTP.load_weights, while load_eagle_model refuses to share the rank-0 target embedding when PP>1, so the last-rank drafter used a randomly initialized embedding table. The fix is bayley_w's vLLM patches (Humvee55 considered sending them upstream as a PR). Both users ran pipeline parallel. The promisezackr set above fixes the same class of fault for GLM-5.3-Flash.

Proven
Source: @bayley_w, 2026-09-09
Source: @ob88594, 2026-09-10
Source: @scallion_9883, 2026-09-10

Early vllm-170hx-glm5 had an MTP + PP + concurrency numerics bug (fixed 2026-08-17)

Warning 8 GB and 10 GB On 2026-08-14 bayley_w told people not to use his repo (github.com/bayley/vllm-170hx-glm5) yet: a numerics bug caused small typos in the output, the kind that could insert extra spaces into an rm command. The bug was in the MTP + pipeline parallel + concurrency path, where the engine must disentangle in-flight batches while speculating. He announced the fix on 2026-08-17. Anyone on a checkout from before then should update. Recipe: GLM 5.3 on 8x 8 GB (vllm-170hx-glm5).

Source: @bayley_w, 2026-08-14

DSA models under PP: relay topk_indices_buffer across PP hops, or output corrupts (bayley)

bayley's vllm-170hx-glm5 (GLM-5.2 and GLM-5.3 on 8 cards, PP8, MTP k=3, over a vLLM 0.26.0 venv with PR #38476 hand-applied) carries correctness fixes that apply to any DSA model served with PP on stock vLLM:

Relay topk_indices_buffer across PP hops: 0 of 21 outputs corrupted with the fix, against 14 of 21 before. The repo README calls this patch the main one: the shared indexer layers at the start of each PP stage otherwise attend with the previous batch's top-k selections, because the full indexer layer that fills the buffer ran on the previous rank and the buffer is per rank.
A canonical top-k tie-break (GLM52_TOPK_DET=canon).
GLM52_GATE_FP32=1, because non-SM90 GateLinear rounds router logits to BF16 (a needle at 85K under concurrency found 22 of 22 times, against 6 of 22).

If the boot log says "Sharing target model topk_indices_buffer", MTP acceptance roughly halves. Two settings silently cost about 1.7x: GLM52_PP_DECODE_BATCH_CAP unset (GPUs about 10% busy; the script default is 2 with GLM52_PP_DECODE_ADAPTIVE=8) and GLM52_DSA_FULLCG_MAXLEN=2048 (117 against 67 ms per step at 32K; keep 0). The maintained successor is arvkmr/vllm-170hx-glm5.

Reported 8 GB and 10 GB
Source: repo github.com/bayley/vllm-170hx-glm5 by bayley on GitHub, posted by @bayley_w, 2026-08-13

Prefix caching broken with Qwen3.8 hybrid models and MTP (PR #48375, issue #54360)

roger_dammit found that vLLM prefix caching is broken for Qwen3.8 (hybrid Mamba and linear attention) when MTP speculative decoding is on, which explained his poor agent performance: without prefix caching every agent turn re-prefills the whole context (2026-08-18). giaki3003 and Studebaker confirmed that MTP broke it. The fix is vLLM PR #48375, "[BugFix] Honor drop_eagle_block in MambaManager", which giaki3003 says can be applied as a patch (the model in context was lued/Qwen3.8-27B-INT8-W8A16-MTP); the zebgop-ops patch set carries it as patch 0011. Studebaker said cudagraph + MTP=2 should give about 90 tok/s decode on one card (he runs an undervolt). On 2026-08-29 Ryko still pointed to a "potential bug" with MTP plus prefix caching when diagnosing tool-call loops. On 2026-09-19 roger_dammit reported that prefix caching with Qwen Next and 27B models still seemed to trigger a full prefill on every prompt, citing vLLM issue #54360; Phoenix17 showed cache hits on his fork akumaburn/vllm-dflash2 and suggested it. roger_dammit had not confirmed a fix.

Reported
Source: @roger_dammit, 2026-08-18
Source: @giaki3003, 2026-08-18
Source: @_studebaker_, 2026-08-18
Source: @vektor_prime, 2026-08-29
Source: @roger_dammit, 2026-09-19
Source: @phoenixone7, 2026-09-19
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub

"ductductduct" loops with PP + MTP + prefix caching at 3 or more concurrent requests: zebgop-ops patch 0010

On Qwen3.8-Flash-Next with pipeline parallel, MTP and prefix caching together, 14 to 27% of requests degenerated into constant-token loops ("ductductduct", token 1023, the sampler's output for all-NaN logits) once 3 or more requests ran concurrently. flipnburn's zebgop-ops/qwen38-flashnext-pp traced it: the mamba spec-decode context indexed raw block-table pointers pp_size steps late and copied recurrent state into other requests' pages. Patch 0010 indexes the source tables by request; after it, 0 of 160 requests failed. Ryko had only seen an MTP rank-split issue, but never tested more than 2 concurrent requests. In the same set, patch 0009 breaks under full CUDA graphs (the launcher forces VLLM_BT_POOL=1) and patches/0008-WITHDRAWN must not be applied. Recipe: Qwen3.8 Flash Next on 3x or 4x 8 GB (zebgop-ops FP8 PP).

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub, posted by @flipnburner, 2026-08-29
Source: @vektor_prime, 2026-08-29

Draft acceptance collapses unless requests send temperature 0

matstatman found his low draft acceptance in Claude Code was because requests were not sending temperature=0: with MTP or DSpark set temperature 0 (or try 0.1), "otherwise they don't work" (2026-08-23). Measured with MTP 3 at temperature 0: mean acceptance length 3.6, averaged from the vLLM logs with:

docker logs qwen38 2>&1 | grep "Mean acceptance length" | cut -d ' ' -f12 | tr -d ',' | awk '{ sum += $1; n++ } END { if (n > 0) print "Sum:", sum, "| Average:", sum / n }'

PJ saw acceptance in the 60% range with MTP 3 at default temperature and Jimmy Dreambeard about 50%. In Hermes the same model can be added twice with different temperatures. Studebaker got his best single-stream decode, 122 tok/s, with temperature 0 and MTP 3. This is in tension with Ryko's view that loops in big models usually come from too low a temperature, such as 0 (see DeepSeek V4 and V4.1: tool-call leaks, loops and crashes).

Reported
Source: @matstatman, 2026-08-23
Source: @jimmydreambeard, 2026-08-24
Source: @zingy90210, 2026-08-24
Source: @_studebaker_, 2026-08-29

MTP 3 misbehaving on Qwen3.6-35B-A3B INT8-W8A8, MTP 2 better

Danske runs Qwen3.6-35B-A3B-INT8-W8A8 on his CMPs. Kaatchi gets about 46 tok/s with the same model and could not get MTP 3 working (2026-08-15). fallengod9962 also had problems with MTP 3 and found MTP 2 works a lot better.

Reported
Source: @danske1, 2026-08-15
Source: @catchyuser, 2026-08-15
Source: @fallengod9962, 2026-08-15

The speculative-decoding speedup collapses as context grows

Contested Everyone in the thread of 2026-08-26 saw the effect; the cause was not agreed. In the order posted:

Zorg33 said a fix exists, but he applied it on Intel hardware. Source: @zorg33, 2026-08-26
cressida saw MTP and speculative decoding slow down after about 50k context, and planned a comparison on an RTX 3090 at full PCIe bandwidth. Source: @cressida1881, 2026-08-26
giaki3003 suggested the cause may be limited PCIe bandwidth. Source: @giaki3003, 2026-08-26
replacedriver47 sees the same on V100s: MTP raises speed, but the speed drops massively as context grows. Source: @replacedriver47, 2026-08-26
iszo posted reference numbers for Qwen3.8-27B W4 on 2x RTX 5060 Ti (about 118 tok/s at 32k, 60 at 80k, 48 at 128k, 35 at 200k) and offered a one-command harness logging TTFT, decode, recall, driver and PCIe state and Xids for a 4k, 32k, 80k, 128k and 200k matrix on a 64 GB 170HX. Source: @ianszo21, 2026-08-26

Depth effects in general are on Speculative decoding on the 170HX.

Structured output and token corruption

HTTP 500 on schema-constrained output with MTP on: xgrammar rejects the draft tokens

On the lazymio/wtdcode vllm-backport engine serving the 753B GLM-5.3 on eight 8 GB cards, every request that constrains output to a schema (a forced named tool_choice, or response_format: json_schema) fails with HTTP 500 while the server keeps serving, whenever MTP (k=2) is on. Engine log: backend_xgrammar.py:168 Failed to advance FSM for request ... for tokens 62 and scheduler.py:2141 Unexpected: grammar rejected tokens [6748, 154847, 62] ... Terminating request. Three rejected tokens are the k=2 draft plus the bonus token, so on this build the structured-output bitmask is not applied to the draft tokens. Lab runs of 24 tool items each: patched build with MTP and PipeSpec, 4 of 24 returned 500; patched with MTP and PipeSpec off, 3 of 24 (same traceback, so PipeSpec is not the cause); patched without any speculation, 0 of 24; TP8 without speculation, 0 of 24; unpatched without speculation, 0 of 24. Two other vLLM builds on the same cards (Morrowmake's Flash engine, GNmac's DeepSeek build) pass both items. Workaround: serve schema traffic from a config without speculation (zero 500s, half the decode speed) or route those requests to another model.

The earliest report is matstatman's (2026-09-04, a different build, a Ray two-card MTP setup): backend_xgrammar "Failed to advance FSM", the scheduler "Unexpected: grammar rejected tokens [198, 27, 1628, 198]", ending in GenerationError "Internal server error" through the Anthropic-compatible endpoint. His AI assistant blamed Ray, and no fix was confirmed. Four tokens were rejected where the lab's k=2 runs rejected three; his MTP depth was not posted, so whether it was the same draft-token mechanism is not settled. His card variant was not stated.

Proven 8 GB
Source: @matstatman, 2026-09-04
Source: @darkmaniac7, 2026-10-28 (lab notes)

zebgop-ops patch 0016: per-request drafts fix structured outputs under PP + speculative decoding

Patch 0016 of zebgop-ops/qwen38-flashnext-pp (2026-10-02) fixes JSON mode and json_schema output under pipeline parallel with speculative decoding on its Qwen3.8-Flash-Next stack. The repo's root cause: under PP a request is rescheduled pp_size steps after it was sampled, and the worker kept only the latest microbatch's drafts for grammar validation, so interleaved requests got placeholder drafts and unconstrained speculative positions, and most requests at concurrency above 1 failed with HTTP 500 "grammar rejected tokens". Drafts are now kept per request. The repo says the handler and engine-core hunks apply to upstream main unchanged. It has not been reported on the vllm-backport build of the entry above.

Reported 8 GB
Source: repo github.com/zebgop-ops/qwen38-flashnext-pp by zebgop-ops on GitHub, posted by @flipnburner, 2026-08-29

Diagnosing token corruption in vLLM: likely causes and a bisect method

Schaka's DeepSeek-V4.1 setup started hallucinating heavily, for example mangling file paths into similar-looking wrong names (2026-09-11). bayley_w identified this as token corruption, the same class of bug he fixed on GLM-5.3 (the numerics bug above), and probably not hardware memory corruption. Likely causes: indexers not crossing PP boundaries, shared buffer reuse during concurrent requests, or MTP buffer reuse. A single chat session can still be multi-stream, since Open-WebUI and Hermes run a second stream for titles and memories. Method: make the corruption reproducible as an M-out-of-N statistic, then bisect, starting with MTP disabled and CUDA graphs disabled; a second agent reviewing the bisector's results keeps it on track better. Garbage output from hardware looks different: see Instability under load.

Reported
Source: @bayley_w, 2026-09-11
Source: @schakaqt, 2026-09-11

Throughput, CPU, host RAM and load time

Low throughput on a GLM recipe: build with VLLM_BUILD_AMPERE_MARLIN=1, check NUMA, and use vLLM's own benchmark

Waka324 had x16 cap-modded cards with TP4 on an AMD EPYC 7713 (4 NUMA nodes), without P2P so as to stay on the mainline unlocker, and saw only about 100 tok/s (2026-10-01). Their llama-benchy matrix: pp2048 d0 3167 +/-430, tg32 d0 52.6 +/-30.5, pp2048@d4k 4175, tg32@d4k 77.8, pp2048@d16k 4468, tg32@d16k 81.0 tok/s, and PP4 also fell short. Takka suspected the missing P2P; okrasit runs a single NUMA node (UMA) and suggested pinning cores and memory; Impactic advised checking driver and CUDA versions. Waka324's resolution: when installing natively, set VLLM_BUILD_AMPERE_MARLIN=1, and use vLLM's built-in benchmark tools, because llama_bench gives significantly different numbers; after that the results closely matched the published ones. Benchmark method: Benchmarks.

Reported
Source: @cmdrwaka324, 2026-10-01
Source: @spectakkular, 2026-10-01
Source: @impactic, 2026-10-01
Source: @okrasit., 2026-10-01

vLLM pins one CPU core per GPU at 100%: does an old CPU limit the card?

Contested The views, in the order posted:

matstatman saw vLLM hold one core at 100% during load on an i7-4770 with one 170HX while nvtop showed 100% GPU load (2026-08-21). Source: @matstatman, 2026-08-21
Arcanum: you are CPU-bound only if neither the core nor the memory ever maxes out. Source: @_arcanum_, 2026-08-21
Ryko: vLLM always uses one core per GPU at 100%, and its startup log says it lowers the thread count to prevent CPU hot-spinning, so an old i7 makes no meaningful difference (only something like an old Atom would). Source: @vektor_prime, 2026-08-21
Mountains: make sure the CPU is not in a power-save mode. Source: @mountains1605, 2026-08-21
Jonathany Joerrington (Xeon E5-2697A v4, 2026-08-25) found the vLLM FlashInfer backend pins one core at 100% with the GPU at about 70%, and GPU utilisation tracks CPU clock; FLASH_ATTN reaches 100% GPU but is slower than FlashInfer at long KV cache. Source: @jonathanyjoerrington, 2026-08-25
Ryko doubted the CPU was the cause. Source: @vektor_prime, 2026-08-26
giaki3003 also saw vLLM use a single core at 100%. Source: @giaki3003, 2026-08-26
ST mentioned vllm.cpp as an alternative, but it did not yet support multiple GPUs and Jonathany could not build it. Source: @stcord, 2026-08-26
Agile and matstatman also saw a single core at 100%, and matstatman blamed Python plus a 2013-era CPU. Source: @agile0674, 2026-08-27
Source: @matstatman, 2026-08-27

Host RAM: vLLM struggles with model files bigger than system RAM, and a 10-card rig ran out at 128 GB

Asked whether 16 GB of system RAM is enough for 4 cards running DeepSeek V4 inference, matstatman said vLLM has a hard time loading files bigger than system RAM (he attributes it to mmap behaviour on Linux) and that adding swap works around it (2026-08-18). ju5t5omedude's Octominer with 4 GB of RAM crashed after the first prompt and took very long to load (PJ had asked). RedPandaMining's 10x 8 GB rig (64 GB unlocked) with only 128 GB of system RAM and a slow 2 TB PCIe Gen3 drive hit out-of-memory errors when renters loaded models, while his 6-card rig with 768 GB RAM and a fast NVMe handled the same tests perfectly (2026-08-26); he was upgrading the 10-card rig to 768 GB DDR4 and a 4 TB Gen4 high-IOPS NVMe.

Reported
Source: @ju5t5omedude, 2026-08-17
Source: @zingy90210, 2026-08-18
Source: @matstatman, 2026-08-18
Source: @redpandamining, 2026-08-26

Model load time: 40 to 50 s per 60 GB over Gen2; a 45-minute load means a cache or disk problem

Programmer007 relayed a report of Qwen Flash Next taking 45 minutes to load in vLLM (2026-08-30). Ryko reloaded it (Q4) in about 5 minutes, and Miséricorde said loading over PCIe 2.0 should take no more than 40 to 50 s per 60 GB. Likely causes named: slow storage, swapping, or Hugging Face and vLLM caches not kept on persistent storage, forcing a re-download and cache regeneration on each start (ST also took part).

Reported
Source: @programmer_007, 2026-08-30
Source: @vektor_prime, 2026-08-30
Source: @wakeupmbrc1, 2026-08-30
Source: @stcord, 2026-08-30

Engines, Speculative decoding on the 170HX, Parallelism over the PCIe link, llama.cpp on CMP cards, Quantization on the 170HX, Benchmarks, Xid errors, Xid 31 and the reserved top of VRAM, Instability under load, Known bad combinations, DeepSeek V4 and V4.1: tool-call leaks, loops and crashes, DeepSeek V4 Flash 0731 on 4x 8 GB (allover326 PP4 DSpark), GLM 5.3 Flash on 8x 8 GB (promisezackr PP8 NVFP4 DFlash2), Qwen3.8 Flash Next on 3x or 4x 8 GB (zebgop-ops FP8 PP), GLM 5.3 on 8x 8 GB (vllm-170hx-glm5), GLM 5.3 Flash on 4x 8 GB, Recipes.