← All reports

Report · 2026-11-20

MTP speculative decoding on Qwen3.8-27B FP8, 1× H100 NVL, vLLM 0.33.0: 0 to 4 draft tokens

1× NVIDIA H100 NVLQwen3.8-27B-FP8FP8 weights, BF16 KV cachevLLM 0.33.0Speculative decoding · MTP

Measured 2026-11-20 on one H100 NVL (94 GB, 3.9 TB/s, PCIe, 310 W power limit on this host), RunPod Secure Cloud, $3.19/h, 31.4 minutes, $1.67. vllm/vllm-openai:v0.33.0, Qwen/Qwen3.8-27B-FP8 (revision c3a462cc), which ships a one-layer MTP head; vLLM's speculative config with method mtp and 1 to 4 draft tokens (num_speculative_tokens). Chat completions, streamed, 512 output tokens, thinking off, default sampling (temperature 1.0, top-p 0.95, top-k 20) unless noted; three short prompts (prose, code, JSON); one engine start per configuration. Not measured on our production H200. Write-up: /blog/qwen38-27b-mtp-speculative-decoding.html

ConclusionWith 3 draft tokens, single-request decode went from 74.8 to 198.4 tok/s (2.65×), and total output rose 101%, 49% and 19% at 4, 16 and 48 requests in flight. 4 drafts were faster for one request (2.90×) but gave less total output than 3 at 16 and 48 in flight. GSM8K-200 scored 99.0% with MTP against 98.0% without. KV cache capacity fell 24% (709,527 to 541,801 tokens).

Results

Medians. Draft tokens = num_speculative_tokens.

One request at a time

decode tok/s, median of 9 runs · higher is better
MTP off74.8MTP off: 74.8 decode tok/s, median of 9 runs
1 draft token131.01 draft token: 131.0 decode tok/s, median of 9 runs
2 draft tokens166.92 draft tokens: 166.9 decode tok/s, median of 9 runs
3 draft tokens198.43 draft tokens: 198.4 decode tok/s, median of 9 runs
4 draft tokens216.94 draft tokens: 216.9 decode tok/s, median of 9 runs

16 requests in flight

total output tok/s · higher is better
MTP off903MTP off: 903 total output tok/s
1 draft token1,1591 draft token: 1,159 total output tok/s
2 draft tokens1,2942 draft tokens: 1,294 total output tok/s
3 draft tokens1,3453 draft tokens: 1,345 total output tok/s
4 draft tokens1,1994 draft tokens: 1,199 total output tok/s

48 requests in flight

total output tok/s · higher is better
MTP off1,742MTP off: 1,742 total output tok/s
1 draft token—1 draft token: — total output tok/s
2 draft tokens—2 draft tokens: — total output tok/s
3 draft tokens2,0663 draft tokens: 2,066 total output tok/s
4 draft tokens1,9414 draft tokens: 1,941 total output tok/s

Measured for MTP off, 3 and 4 draft tokens only.

KV cache capacity

tokens, from the engine's startup log · higher is better
MTP off709,527MTP off: 709,527 tokens, from the engine's startup log
1 draft token624,4821 draft token: 624,482 tokens, from the engine's startup log
2 draft tokens570,1632 draft tokens: 570,163 tokens, from the engine's startup log
3 draft tokens541,8013 draft tokens: 541,801 tokens, from the engine's startup log
4 draft tokens514,6184 draft tokens: 514,618 tokens, from the engine's startup log

0.92 GPU memory share, 32,768-token max context.

One request at a time

Thinking off, default sampling, 3 prompts × 3 runs. Tokens per step = vLLM's mean acceptance length (1 + accepted ÷ drafts).

ConfigurationDecode tok/s, median (range)vs MTP offTokens per stepTTFT, medianKV cache, tokens
MTP off74.8 (74.3–75.3)—175 ms709,527
1 draft token131.0 (114.0–133.8)1.75×1.8777 ms624,482 (−12%)
2 draft tokens166.9 (130.3–183.1)2.23×2.5581 ms570,163 (−20%)
3 draft tokens198.4 (138.3–222.3)2.65×3.1480 ms541,801 (−24%)
4 draft tokens216.9 (136.5–252.0)2.90×3.6488 ms514,618 (−27%)

Under load

Closed loop: 4, 16, 48 requests in flight (12, 32, 96 requests). Total output tok/s (median per-request decode tok/s), change vs MTP off.

Configuration4 in flight16 in flight48 in flight
MTP off277 (71.0)903 (60.1)1,742 (40.2)
1 draft token440 (119.3), +59%1,159 (86.1), +28%—
2 draft tokens513 (151.8), +85%1,294 (106.6), +43%—
3 draft tokens557 (175.4), +101%1,345 (115.8), +49%2,066 (58.0), +19%
4 draft tokens565 (176.0), +104%1,199 (107.2), +33%1,941 (57.6), +11%

By content

Single-request decode tok/s, median per prompt.

OutputMTP off1 draft token2 draft tokens3 draft tokens4 draft tokens
Prose (~450-word explanation)74.8115.1132.9140.7140.6
Code (Python function + pytest)74.8131.0166.9198.4216.9
JSON (transform 8 orders)74.5133.7182.8220.3247.8
Reasoning text (thinking on, 1 request)74.2119.7140.4147.9161.4
Greedy (temperature 0), all prompts75.1131.6177.3203.0219.0

Acceptance

Mean acceptance length (acceptance rate), from vLLM's /metrics spec-decode counters per phase.

Configuration1 requestGreedyThinking on16 in flight48 in flight
1 draft token1.87 (0.87)1.89 (0.89)1.78 (0.79)1.87 (0.87)—
2 draft tokens2.55 (0.78)2.67 (0.83)2.28 (0.64)2.55 (0.77)—
3 draft tokens3.14 (0.71)3.28 (0.76)2.60 (0.53)3.09 (0.70)3.09 (0.70)
4 draft tokens3.64 (0.66)3.75 (0.69)3.12 (0.53)3.47 (0.62)3.52 (0.63)

Memory

From the engine startup log: weights loaded, memory left for the KV cache, and the cache's capacity in tokens.

ConfigurationWeightsKV cache memoryPer cached tokenKV cache, tokens
MTP off27.55 GiB53.91 GiB79.7 KiB709,527
1 draft token27.99 GiB53.47 GiB89.8 KiB624,482
2 draft tokens27.99 GiB51.68 GiB95.0 KiB570,163
3 draft tokens27.99 GiB51.74 GiB100.1 KiB541,801
4 draft tokens27.99 GiB51.72 GiB105.4 KiB514,618

Quality checks

GSM8K: Bench's 5-shot suite, 200 questions, thinking off, temperature 0, 32 in flight, numeric scorer. Greedy outputs (temperature 0) with MTP were byte-identical across 1–4 draft tokens and from run to run, but not identical to MTP off; on the JSON prompt they differed at the first token, a near-tie decided before any draft exists.

CheckMTP off3 draft tokens
GSM8K-20098.0% (196/200)99.0% (198/200)
Greedy outputsreferencenot byte-identical
JSON prompt, first-token top 2 (logprob)``` −0.48, { −0.98{ −0.53, ``` −0.90

Setup

Any change to these fields makes it a different report.

GPU
1× NVIDIA H100 NVL 94 GB HBM3, 3.9 TB/s, PCIe; power limit 310 W on this host
Host
RunPod Secure Cloud, US-GA-2, $3.19/h; 31.4 min, $1.67
Driver
595.112.07 (CUDA 13.2)
Engine
vllm/vllm-openai:v0.33.0, one engine start per configuration
Model
Qwen/Qwen3.8-27B-FP8 @ c3a462cc3d38cf351f3b26cbd763469eeb74ef81; MTP head: 1 layer (mtp.safetensors, 0.48 GB)
Flags
--max-model-len 32768 --tensor-parallel-size 1 --language-model-only --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --enable-prompt-tokens-details --max-num-seqs 256
MTP flag
--speculative-config '{"method":"mtp","num_speculative_tokens":N}', N = 1, 2, 3, 4
Cache block
832 tokens in every configuration (hybrid attention)
Requests
chat completions, streamed (include_usage), max_tokens 512
Thinking
off (chat_template_kwargs.enable_thinking=false) unless noted; thinking-on run: 1 request, 1,024 reasoning tokens
Sampling
model defaults (temperature 1.0, top-p 0.95, top-k 20); greedy runs temperature 0 (3 prompts × 2)
Decode tok/s
(completion tokens − 1) ÷ (last token chunk − first token chunk)
TTFT
first token chunk − send
Load
closed loop, 4 / 16 / 48 in flight (12 / 32 / 96 requests)

Not covered

Do not extrapolate this report to the following.

  • Our production H200 (4.8 TB/s, shared with a second model under CUDA MPS)
  • Long prompts: inputs were about 100 to 1,000 tokens; decode with MTP at long context was not measured
  • Full-context KV layout: 32,768-token max context and a 0.92 memory share, not 262,144 and a shared GPU
  • 48 in flight for 1 and 2 draft tokens; GSM8K for 1, 2 and 4 draft tokens
  • FP8 KV cache with MTP
  • Accuracy beyond GSM8K-200
  • Repeat engine starts: one start per configuration

Reproduce

Raw request timings, engine logs and the exact argv per configuration: bench/results/qwen38-mtp-2026-11/ in the TokenWatt repository (README.md, raw/nvl/<config>/).

shell
# baseline
vllm serve Qwen/Qwen3.8-27B-FP8 --revision c3a462cc3d38cf351f3b26cbd763469eeb74ef81 \
  --served-model-name qwen3.8-27b --max-model-len 32768 --tensor-parallel-size 1 \
  --language-model-only --reasoning-parser qwen3 --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder --enable-prompt-tokens-details --max-num-seqs 256

# MTP, N = 1..4 (3 is vLLM's recipe value)
vllm serve ... --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

# keep the image's CUDA_HOME and PATH in the launching shell: without nvcc, FlashInfer's FP8
# DeepGEMM JIT fails during CUDA-graph capture ("Assertion failed: !cubin.empty() || isPathValid(path_)")

Read the write-up Download raw JSON