GLM 5.3 on 8x 8 GB (cebeuq glm53-cmp170hx-recipe)/Confirmation 1
More actions
| Confirms | GLM 5.3 on 8x 8 GB (cebeuq glm53-cmp170hx-recipe) |
|---|---|
| Result | Partly reproduced |
| By | @darkmaniac7 |
| Date | 2026-11-04 |
| Cards | 8 GB |
| Numbers | Same layout (TP2 x PP4, layers 22,20,20,16), 524,288-token context, KV pool 569,792 tokens. Lab bench, 1,024-token answers, median of 3: one stream at temperature 0, structured / code / prose 89.6 / 93.9 / 60.8 tok/s; sampled (temperature 0.9) code 93.3-94.6 and prose 61.3-62.0 tok/s over three runs; prefill 2,272 tok/s on a 17k-token prompt; a long-context chat at an 83k-token prompt, two turns of 2,000 tokens: first turn 67.5 tok/s (first token after 32.3 s), second turn 86.9 tok/s (first token after 8.3 s); 0 HTTP 5xx. Quality: 90.07% top-1 agreement with the lab's TP8 reference outputs (the lab's floor is 89.9%), NLL delta -0.69%. Accepted draft tokens per 2-token draft: 1.305 at temperature 0 (per position 0.754 / 0.551), 1.252 sampled (0.745 / 0.506), 1.385 on the long-context chat, 0.929 on raw sampled prose; 3.00 tokens per PipeSpec epoch. Load to ready from a local NVMe copy of the converted weights with a 450 MB/s container read cap: 558 s warm, 1,341 s cold. |
The lab runs this recipe in production on its own eight 8 GB cards (64 GB, PCIe Gen2 x16, P2P on) with three additions of its own (published on 2026-11-05 and offered to the recipe as pull requests 1 to 3): the pipeline hand-off fix for TP2 (2026-10-27; the open question in the recipe's known limits), the grammar bitmask for draft tokens (2026-11-04, a port of the upstream vLLM fix, so json_schema, named tool choice and json_object work with speculative decoding), and probabilistic drafts for sampled requests under PipeSpec (2026-11-04). The lab's bench (1,024-token answers, median of 3, structured / code / prose) is not the author's chat bench (384 tokens, thinking on, mean per category), and its quality reference differs (TP8 reference outputs, not the author's NVFP4 goldens), so these numbers are not directly comparable with his. Before the probabilistic drafts (the recipe with the hand-off and grammar fixes only, same cards, same day): accepted draft tokens 1.303 at temperature 0, 1.191 sampled, 1.226 on the long-context chat, 0.829 on raw sampled prose; agreement 90.13%; sampled code 91.4-92.5 and prose 58.6-61.8 tok/s over three runs. Entered as partly reproduced: the layout, the context and the KV pool match the recipe exactly, but the lab's build carries three additions and the author's speed numbers were not re-measured on his bench. Source: @darkmaniac7, 2026-11-04 (lab notes)