← Lab notes

MTP on Qwen3.8-27B: 2.65× faster per request, at the cost of 24% of the KV cache

Qwen3.8-27B ships with a one-layer multi-token prediction (MTP) head that vLLM can use as a draft model. On one H100 NVL with vLLM 0.33.0, three draft tokens per step raised single-request decode from 74.8 to 198 tokens per second (2.65×), and total throughput by 101%, 49% and 19% at 4, 16 and 48 requests in flight. GSM8K held (99.0% against 98.0%). The price is memory: room for the KV cache fell 24%, from 709,527 to 541,801 tokens. We turned it on in our API the same day, then off again that evening: with two models sharing the GPU under CUDA MPS, MTP crashed both engines under long-context load.

Choosing GPUs? Compare GPUs for your traffic · Get your node measured, free

With its built-in draft head turned on, Qwen3.8-27B generated 198 tokens per second for a single request on one H100 NVL. Without it: 74.8. The cost was a quarter of the room for the KV cache.

Update (2026-11-20, evening): we turned MTP off again in our API at about 22:45 UTC. At 20:23 UTC, under six concurrent ~100K-token coding-agent sessions per model (our own test traffic), both engines on our H200 died with a CUDA illegal memory access and restarted within about three minutes. On a separate H200 with the same setup, it crashed again within one to two minutes, but only when MTP was on and the two engines shared the GPU through CUDA MPS. MTP alone, MPS without MTP, and our previous configuration all ran without a crash. Our API is back on the November 19 configuration (Qwen3.8 at 52% of the GPU, no MTP, about 90 tokens per second for a single request). We reported it to vLLM with the configuration matrix: vllm-project/vllm#69047. The measurements below stand; the production section describes what ran from 18:55 to 22:45 UTC.

Qwen3.8-27B is a dense model, not a mixture of experts: every output token reads all 27B parameters (about 27 GiB in FP8) from GPU memory, so the speed of a single request is limited mainly by memory bandwidth. The checkpoint also ships one extra layer, 0.48 GB, trained to predict tokens ahead: a multi-token prediction (MTP) head. vLLM can use it as a draft model. The head proposes a few tokens, the full model checks them in one pass, and every accepted token saves a step. One flag turns it on:

vllm serve Qwen/Qwen3.8-27B-FP8 ... \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Three draft tokens is the value on vLLM's recipe page for this model. We measured 1 to 4.

Qwen3.8-27B FP8, 1× H100 NVL, vLLM 0.33.0, one request at a time, thinking off, 512 output tokens:

Draft tokens Tok/s Speedup Tokens per step KV cache, tokens
MTP off 74.8 — 1 709,527
1 131.0 1.75× 1.87 624,482 (−12%)
2 166.9 2.23× 2.55 570,163 (−20%)
3 198.4 2.65× 3.14 541,801 (−24%)
4 216.9 2.90× 3.64 514,618 (−27%)

Tok/s is the median decode speed over nine runs. "Tokens per step" is vLLM's mean acceptance length: one token from the full model plus the accepted drafts. With three drafts, 84% of first drafts were accepted, 71% of second drafts and 59% of third drafts. The speedup stays below the tokens per step because each step does more work: the draft layer runs three times, and the full model checks four positions instead of one.

Time to first token didn't change: 75 ms without MTP, 80 ms with three drafts.

Under load, the gain shrinks

Total output tokens per second, closed loop (median per-request speed in brackets):

Requests in flight MTP off 3 draft tokens 4 draft tokens
1 74.8 198.4 (2.65×) 216.9 (2.90×)
4 277 (71.0) 557 (175.4), +101% 565 (176.0), +104%
16 903 (60.1) 1,345 (115.8), +49% 1,199 (107.2), +33%
48 1,742 (40.2) 2,066 (58.0), +19% 1,941 (57.6), +11%

Three drafts were faster at every load we tried, but the gain fell from 2.65× for one request to +19% at 48 in flight. Our reading: one request leaves the GPU waiting on memory, so checking extra tokens is nearly free. With 48 requests, each step already carries real work, and every rejected draft is compute spent for nothing.

Four drafts gave one request 9% more speed than three, and less total throughput at 16 and 48 in flight (11% and 6% less). Three is the better setting for a shared server.

Under load, MTP also cut the wait for the first token. At 48 in flight the median went from 1.28 s to 0.53 s, because requests finish sooner and spend less time queued.

It depends on what the model writes

The draft head guesses well on predictable text and worse on free prose:

Output MTP off, tok/s 3 draft tokens, tok/s Speedup
Prose (a ~450-word explanation) 74.8 140.7 1.88×
Code (a Python function with tests) 74.8 198.4 2.65×
JSON (transforming 8 orders) 74.5 220.3 2.96×
Reasoning text (thinking on, 1 request) 74.2 147.9 1.99×

The single-request headline, 198.4 tok/s, is the median of all nine runs (three prompts, three runs each); the middle run happened to be a code run. With thinking on, the mean acceptance length fell to 2.60 tokens per step.

Quality: GSM8K held; outputs were not byte-identical

  • GSM8K, 200 questions, 5-shot, thinking off, temperature 0: 98.0% (196 of 200) without MTP and 99.0% (198 of 200) with three drafts. Two questions apart, which is within noise.
  • Same text? At temperature 0, outputs with MTP were byte-identical across 1, 2, 3 and 4 draft tokens, and from run to run. They were not identical to the outputs without MTP. On the JSON prompt they differed at the very first token, which the full model produces from the prompt before any draft exists. There, the top two candidates were nearly tied and swapped order: without MTP, the opening of a Markdown code fence (log-probability −0.48) led { (−0.98); with MTP, { (−0.53) led the code fence (−0.90). Both engines used the same attention backend, the same FP8 kernels and the same 832-token cache block. Our reading: an engine set up for speculative decoding runs different CUDA-graph and batch shapes, which shifts the numerics slightly. The drafts themselves didn't change the text: it was the same for every draft count. If you need output identical to an existing deployment, check before switching.

The cost: KV cache

vLLM gives the KV cache whatever GPU memory is left after weights, activations and CUDA graphs. With three drafts, that space shrank by 2.2 GiB (53.91 to 51.74 GiB), and each cached token took about 26% more of it. Together: 709,527 tokens without MTP, 541,801 with it, 24% less. The draft layer's own weights are small (27.55 to 27.99 GiB loaded).

KV capacity is how much context a server can hold at once: how many long requests fit side by side, and how much a prefix cache can keep. On a short-prompt benchmark the loss doesn't show. On a server with long requests, it can.

We turned it on in our API the same day

Our API serves Qwen3.8-27B and Qwen3.6-35B-A3B from one H200, and the two engines split its memory. Our engine refuses to start Qwen3.8 with less than 524,288 tokens of KV cache, enough for two requests at the full 262,144-token context. At Qwen3.8's old share (52% of the GPU) it had about 650,000 tokens; a 24% cut would have left less than the floor.

We made room instead of lowering the floor: on November 20 at 18:55 UTC we restarted the node with Qwen3.8 at 57% of the GPU and three draft tokens, and Qwen3.6 at 38% (down from 43%). The two models were down for about five minutes.

On our H200 Before After
Qwen3.8 KV cache, tokens about 650,000 (estimated) 665,563
Qwen3.6 KV cache, tokens 1,183,842 865,075
Qwen3.8, one request, tok/s 90–91 162–281
Qwen3.8, time to first token about 0.25 s 0.25–0.28 s
  • Speed: measured through our public API from California, thinking off. Before: three runs of a prose prompt on the morning of November 20, 90–91 tok/s. After: one run each of a prose, a code and a JSON prompt, 162, 231 and 281 tok/s. The prose gain (1.8×) is close to what prose gained on the H100 NVL (1.88×).
  • KV cache: 665,563 tokens is 2.54 full-context requests, above the floor. The "before" figure for Qwen3.8 is an estimate from the 672,701 tokens we measured at a 53% share. Qwen3.6 keeps room for 3.3 full-context requests.
  • Acceptance: the engine reported 3.15 tokens per step on its warm-up requests, and 2.8–2.9 under a run of thinking-on coding tasks, close to the 2.60–3.14 we saw on the H100 NVL.
  • Tool calls: our 20-case tool-call check passed on both models after the restart.
  • Not measured yet: the capacity under load. The limits we publish for this model (210,000 input tokens a minute, 40 requests in flight) are still the ones measured before MTP. And our prompts were short; our own team's coding-agent prompts run 86K–114K tokens at the median, and we haven't measured MTP at that length.

Turning it on yourself

  • The flag is the one at the top of this post. In vLLM 0.33.0 the method is mtp; the older names qwen3_5_mtp and qwen3_next_mtp are rewritten to it. The engine log should show SpeculativeConfig(method='mtp' ... num_spec_tokens=3).
  • Check the head exists: config.json has mtp_num_hidden_layers: 1 in text_config, and the weights include an mtp.safetensors file. With one MTP layer, more than one draft token reuses that layer.
  • Keep the image's environment. Our first MTP start failed while capturing CUDA graphs with RuntimeError: Assertion failed: !cubin.empty() || isPathValid(path_) in FlashInfer's FP8 DeepGEMM kernel. The cause was our launcher: the SSH shell didn't have the Docker image's CUDA_HOME and PATH, so the kernel's just-in-time compile found no nvcc. With the image's environment restored, every configuration started.
  • Check KV capacity in the startup log (GPU KV cache size: N tokens) against the longest requests you need to hold, not just tokens per second.

How we measured

  • Hardware: one H100 NVL (94 GB HBM3, 3.9 TB/s, PCIe, power limit 310 W on this host) on RunPod Secure Cloud, $3.19 an hour. The whole test took 31.4 minutes and cost $1.67.
  • Engine: the vllm/vllm-openai:v0.33.0 image. Qwen/Qwen3.8-27B-FP8 at revision c3a462cc, with the flags our API uses for this model, except three: a 32,768-token context instead of 262,144 (faster start), vLLM's default memory share of 0.92 instead of our shared node's, and the speculative config. One engine start per configuration.
  • Client: chat completions, streamed, 512 output tokens, thinking off unless noted. Default sampling, so the model's own settings (temperature 1.0, top-p 0.95, top-k 20); the greedy checks used temperature 0. Three prompts: a ~450-word prose explanation, a Python function with a pytest suite, and a JSON transformation of 8 orders. One request at a time: 3 prompts × 3 runs. Under load: a closed loop with 4, 16 and 48 requests in flight (12, 32 and 96 requests).
  • Metrics: decode speed is (output tokens − 1) ÷ (time of the last token − time of the first token). Acceptance comes from vLLM's speculative-decoding counters in /metrics.

Limits

  • One GPU, one start per setting. The sweep ran on an H100 NVL with a 310 W power limit. On our production H200 we measured only single-request speed, KV capacity, acceptance and tool calls (section above). Load at 48 in flight was measured for MTP off and 3 and 4 drafts only; GSM8K for MTP off and 3 drafts only.
  • Short prompts. About 100 to 1,000 tokens in, 512 out. Decode with MTP at long context wasn't measured.
  • Content matters. Prose gained 1.88×, JSON 2.96×. Your mix decides your number.
  • Small quality check. GSM8K on 200 questions shows no drop; it can't rule out small changes on other tasks.

Every configuration's numbers are in the report. Earlier we compared H100 SXM, PCIe and NVL, and today we also published how real coding-agent traffic uses the prefix cache on the same model.

Keep reading

5 min read

94% prefix-cache hits on hybrid models with real coding-agent traffic, against 56% in our synthetic test

A synthetic test we published on November 16 showed a hybrid model, Qwen3.8-27B, reusing far less of a shared prompt than a dense-attention model (56% against 89%), because vLLM caches its prompts only in whole blocks of 784 tokens. On two days of our own team's coding-agent traffic on our API, the same rule cost little: vLLM served 94.3% of Qwen3.8-27B's prompt tokens and 94.8% of Qwen3.6-35B-A3B's from cache, on median prompts of 114K and 86K tokens. The median request recomputed only 1,562 to 2,690 tokens.

prefix cachingprompt cachehybrid attention
8 min read

Prefix caching on a hybrid model: fewer cache hits, a bigger payoff (Qwen3.8-27B on one H100)

We repeated last week's prefix-cache capacity test on a hybrid model, Qwen3.8-27B (Gated DeltaNet + full attention), on the same H100 SXM setup. vLLM sizes each cache block to hold one DeltaNet state, 784 tokens with a BF16 KV cache and 1,568 with FP8, and only whole blocks hit. Hit rates came out far below the dense model's, but capacity within our latency target rose 19%, 36% and 70% at 50%, 75% and 90% shared on chat-like traffic, and 1.7×, 2.4× and 3.4× with 4,000-token prompts. A formula built on prefill's share of GPU time predicts a fraction of that. Thanks to Manas Choudhary, whose RTX 5090 runs found the block size.

prefix cachingprompt cachehybrid attention
4 min read

Renting an H100? The variant matters more than the luck of the draw

We rented 19 H100s on RunPod Secure Cloud and checked each one, then served the same 8B model on one of each variant. All 10 SXM cards were healthy and nearly identical. Every NVL card was set to 310 W, below NVIDIA's 350–400 W range, and the PCIe cards split into two speed groups 12% apart. Per token, the SXM was the cheapest: $0.31 per million output tokens, against $0.43 on PCIe and $0.57 on NVL.

H100H100 NVLH100 PCIe