MTP speculative decoding on Qwen3.8-27B FP8, 1× H100 NVL, vLLM 0.33.0: 0 to 4 draft tokens
Measured 2026-11-20 on one H100 NVL (94 GB, 3.9 TB/s, PCIe, 310 W power limit on this host), RunPod Secure Cloud, $3.19/h, 31.4 minutes, $1.67. vllm/vllm-openai:v0.33.0, Qwen/Qwen3.8-27B-FP8 (revision c3a462cc), which ships a one-layer MTP head; vLLM's speculative config with method mtp and 1 to 4 draft tokens (num_speculative_tokens). Chat completions, streamed, 512 output tokens, thinking off, default sampling (temperature 1.0, top-p 0.95, top-k 20) unless noted; three short prompts (prose, code, JSON); one engine start per configuration. Not measured on our production H200. Write-up: /blog/qwen38-27b-mtp-speculative-decoding.html
Results
Medians. Draft tokens = num_speculative_tokens.
One request at a time
16 requests in flight
48 requests in flight
Measured for MTP off, 3 and 4 draft tokens only.
KV cache capacity
0.92 GPU memory share, 32,768-token max context.
One request at a time
Thinking off, default sampling, 3 prompts × 3 runs. Tokens per step = vLLM's mean acceptance length (1 + accepted ÷ drafts).
| Configuration | Decode tok/s, median (range) | vs MTP off | Tokens per step | TTFT, median | KV cache, tokens |
|---|---|---|---|---|---|
| MTP off | 74.8 (74.3–75.3) | — | 1 | 75 ms | 709,527 |
| 1 draft token | 131.0 (114.0–133.8) | 1.75× | 1.87 | 77 ms | 624,482 (−12%) |
| 2 draft tokens | 166.9 (130.3–183.1) | 2.23× | 2.55 | 81 ms | 570,163 (−20%) |
| 3 draft tokens | 198.4 (138.3–222.3) | 2.65× | 3.14 | 80 ms | 541,801 (−24%) |
| 4 draft tokens | 216.9 (136.5–252.0) | 2.90× | 3.64 | 88 ms | 514,618 (−27%) |
Under load
Closed loop: 4, 16, 48 requests in flight (12, 32, 96 requests). Total output tok/s (median per-request decode tok/s), change vs MTP off.
| Configuration | 4 in flight | 16 in flight | 48 in flight |
|---|---|---|---|
| MTP off | 277 (71.0) | 903 (60.1) | 1,742 (40.2) |
| 1 draft token | 440 (119.3), +59% | 1,159 (86.1), +28% | — |
| 2 draft tokens | 513 (151.8), +85% | 1,294 (106.6), +43% | — |
| 3 draft tokens | 557 (175.4), +101% | 1,345 (115.8), +49% | 2,066 (58.0), +19% |
| 4 draft tokens | 565 (176.0), +104% | 1,199 (107.2), +33% | 1,941 (57.6), +11% |
By content
Single-request decode tok/s, median per prompt.
| Output | MTP off | 1 draft token | 2 draft tokens | 3 draft tokens | 4 draft tokens |
|---|---|---|---|---|---|
| Prose (~450-word explanation) | 74.8 | 115.1 | 132.9 | 140.7 | 140.6 |
| Code (Python function + pytest) | 74.8 | 131.0 | 166.9 | 198.4 | 216.9 |
| JSON (transform 8 orders) | 74.5 | 133.7 | 182.8 | 220.3 | 247.8 |
| Reasoning text (thinking on, 1 request) | 74.2 | 119.7 | 140.4 | 147.9 | 161.4 |
| Greedy (temperature 0), all prompts | 75.1 | 131.6 | 177.3 | 203.0 | 219.0 |
Acceptance
Mean acceptance length (acceptance rate), from vLLM's /metrics spec-decode counters per phase.
| Configuration | 1 request | Greedy | Thinking on | 16 in flight | 48 in flight |
|---|---|---|---|---|---|
| 1 draft token | 1.87 (0.87) | 1.89 (0.89) | 1.78 (0.79) | 1.87 (0.87) | — |
| 2 draft tokens | 2.55 (0.78) | 2.67 (0.83) | 2.28 (0.64) | 2.55 (0.77) | — |
| 3 draft tokens | 3.14 (0.71) | 3.28 (0.76) | 2.60 (0.53) | 3.09 (0.70) | 3.09 (0.70) |
| 4 draft tokens | 3.64 (0.66) | 3.75 (0.69) | 3.12 (0.53) | 3.47 (0.62) | 3.52 (0.63) |
Memory
From the engine startup log: weights loaded, memory left for the KV cache, and the cache's capacity in tokens.
| Configuration | Weights | KV cache memory | Per cached token | KV cache, tokens |
|---|---|---|---|---|
| MTP off | 27.55 GiB | 53.91 GiB | 79.7 KiB | 709,527 |
| 1 draft token | 27.99 GiB | 53.47 GiB | 89.8 KiB | 624,482 |
| 2 draft tokens | 27.99 GiB | 51.68 GiB | 95.0 KiB | 570,163 |
| 3 draft tokens | 27.99 GiB | 51.74 GiB | 100.1 KiB | 541,801 |
| 4 draft tokens | 27.99 GiB | 51.72 GiB | 105.4 KiB | 514,618 |
Quality checks
GSM8K: Bench's 5-shot suite, 200 questions, thinking off, temperature 0, 32 in flight, numeric scorer. Greedy outputs (temperature 0) with MTP were byte-identical across 1–4 draft tokens and from run to run, but not identical to MTP off; on the JSON prompt they differed at the first token, a near-tie decided before any draft exists.
| Check | MTP off | 3 draft tokens |
|---|---|---|
| GSM8K-200 | 98.0% (196/200) | 99.0% (198/200) |
| Greedy outputs | reference | not byte-identical |
| JSON prompt, first-token top 2 (logprob) | ``` −0.48, { −0.98 | { −0.53, ``` −0.90 |
Setup
Any change to these fields makes it a different report.
- GPU
- 1× NVIDIA H100 NVL 94 GB HBM3, 3.9 TB/s, PCIe; power limit 310 W on this host
- Host
- RunPod Secure Cloud, US-GA-2, $3.19/h; 31.4 min, $1.67
- Driver
- 595.112.07 (CUDA 13.2)
- Engine
- vllm/vllm-openai:v0.33.0, one engine start per configuration
- Model
- Qwen/Qwen3.8-27B-FP8 @ c3a462cc3d38cf351f3b26cbd763469eeb74ef81; MTP head: 1 layer (mtp.safetensors, 0.48 GB)
- Flags
- --max-model-len 32768 --tensor-parallel-size 1 --language-model-only --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --enable-prompt-tokens-details --max-num-seqs 256
- MTP flag
- --speculative-config '{"method":"mtp","num_speculative_tokens":N}', N = 1, 2, 3, 4
- Cache block
- 832 tokens in every configuration (hybrid attention)
- Requests
- chat completions, streamed (include_usage), max_tokens 512
- Thinking
- off (chat_template_kwargs.enable_thinking=false) unless noted; thinking-on run: 1 request, 1,024 reasoning tokens
- Sampling
- model defaults (temperature 1.0, top-p 0.95, top-k 20); greedy runs temperature 0 (3 prompts × 2)
- Decode tok/s
- (completion tokens − 1) ÷ (last token chunk − first token chunk)
- TTFT
- first token chunk − send
- Load
- closed loop, 4 / 16 / 48 in flight (12 / 32 / 96 requests)
Not covered
Do not extrapolate this report to the following.
- Our production H200 (4.8 TB/s, shared with a second model under CUDA MPS)
- Long prompts: inputs were about 100 to 1,000 tokens; decode with MTP at long context was not measured
- Full-context KV layout: 32,768-token max context and a 0.92 memory share, not 262,144 and a shared GPU
- 48 in flight for 1 and 2 draft tokens; GSM8K for 1, 2 and 4 draft tokens
- FP8 KV cache with MTP
- Accuracy beyond GSM8K-200
- Repeat engine starts: one start per configuration
Reproduce
Raw request timings, engine logs and the exact argv per configuration: bench/results/qwen38-mtp-2026-11/ in the TokenWatt repository (README.md, raw/nvl/<config>/).
# baseline
vllm serve Qwen/Qwen3.8-27B-FP8 --revision c3a462cc3d38cf351f3b26cbd763469eeb74ef81 \
--served-model-name qwen3.8-27b --max-model-len 32768 --tensor-parallel-size 1 \
--language-model-only --reasoning-parser qwen3 --enable-auto-tool-choice \
--tool-call-parser qwen3_coder --enable-prompt-tokens-details --max-num-seqs 256
# MTP, N = 1..4 (3 is vLLM's recipe value)
vllm serve ... --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
# keep the image's CUDA_HOME and PATH in the launching shell: without nvcc, FlashInfer's FP8
# DeepGEMM JIT fails during CUDA-graph capture ("Assertion failed: !cubin.empty() || isPathValid(path_)")