94% prefix-cache hits on hybrid models with real coding-agent traffic, against 56% in our synthetic test
A synthetic test we published on November 16 showed a hybrid model, Qwen3.8-27B, reusing far less of a shared prompt than a dense-attention model (56% against 89%), because vLLM caches its prompts only in whole blocks of 784 tokens. On two days of our own team's coding-agent traffic on our API, the same rule cost little: vLLM served 94.3% of Qwen3.8-27B's prompt tokens and 94.8% of Qwen3.6-35B-A3B's from cache, on median prompts of 114K and 86K tokens. The median request recomputed only 1,562 to 2,690 tokens.
Choosing GPUs? Compare GPUs for your traffic · Get your node measured, free
On real coding-agent traffic, vLLM served 94.3% of Qwen3.8-27B's prompt tokens from its prefix cache. In a synthetic test we published on Monday, the same model got 55.6%.
That post (November 16) explained the low number. Qwen3.8-27B is a hybrid: most of its layers are linear attention (Gated DeltaNet), and only a quarter are full attention. To reuse a cached prefix, vLLM also needs the linear-attention state at the point where the prefix ends. So it caches prompts in blocks of 784 tokens instead of the usual 16, and only whole blocks hit. With 90% of every prompt shared, a dense-attention model got 89% hits on that test. Qwen3.8 got 56%.
That test used synthetic prompts of about 1,000 tokens. Since November 19, our API has served two hybrid models from one H200: Qwen3.8-27B and Qwen3.6-35B-A3B. So we looked at real traffic: two days of our own team's coding-agent requests on our API.
Prompt tokens served from vLLM's prefix cache, coding-agent traffic, November 19–20, 2026 (prompts of 1,000 tokens or more):
| Qwen3.8-27B | Qwen3.6-35B-A3B | |
|---|---|---|
| Requests | 1,013 | 2,136 |
| Prompt length, median (p10–p90) | 114,258 (40,706–153,171) | 85,817 (31,888–121,662) |
| Prompt tokens served from cache | 94.3% | 94.8% |
| Hit rate per request, median (p10) | 97.6% (89.9%) | 98.0% (87.3%) |
| Tokens recomputed per request, median (p90) | 2,690 (6,356) | 1,562 (5,830) |
| Requests with no cache hit | 44 | 32 |
The headline rate is token-weighted: all cached prompt tokens divided by all prompt tokens.
Why the block size barely mattered here
A coding agent sends the whole conversation with every request: the system prompt, the tool definitions, every earlier message and tool result, and then the newest part. Almost all of each prompt has been sent before. vLLM reuses it up to the last whole block it has cached and computes the rest.
So rounding down to whole blocks costs less than one block per request: at most 831 tokens with the 832-token blocks Qwen3.8 gets on our node. On a 1,000-token prompt, that is most of the prompt. On a 114,000-token prompt, it is under 1%.
What the median request recomputed, 2,690 tokens on Qwen3.8 and 1,562 on Qwen3.6, is the newest part of the conversation plus a partial block. That is about 2% of a median prompt.
| Qwen3.8-27B | Synthetic test | Real agent traffic |
|---|---|---|
| GPU, engine | H100 SXM, vLLM 0.31.0 | H200, vLLM 0.33.0 |
| Cache block | 784 tokens | 832 tokens |
| Median prompt | 1,024 tokens | 114,258 tokens |
| Shared part | 90%, from 8 prefixes | the conversation so far |
| From cache | 55.6% | 94.3% |
The penalty is the same rule in both columns. It is about one block per request. With short shared prompts, one block is a large share of each prompt. With long agent contexts, it is a rounding error.
Where the cache still missed
By prompt length:
| Prompt length | Qwen3.8 requests | Qwen3.8 from cache | Qwen3.6 requests | Qwen3.6 from cache |
|---|---|---|---|---|
| 1K–8K tokens | 12 | 9.6% | 12 | 7.6% |
| 8K–32K | 44 | 42.7% | 203 | 86.0% |
| 32K–64K | 147 | 92.7% | 348 | 87.8% |
| 64K and more | 810 | 94.9% | 1,573 | 95.8% |
Short prompts were rare in this traffic, and they hit poorly. On Qwen3.8, requests of 8K–32K tokens recomputed a median of 11,175 tokens each. Our guess is that these are early turns of new sessions. Monday's test found that a new prefix has to be seen twice before vLLM reuses it, and a new session has had little time to repeat anything. We can't check this: our request log has no session ids. We also don't know why Qwen3.6 did much better in the same range (86.0%). Forty-four requests is a small sample.
If you serve a hybrid model
- Find your block size. vLLM logs it at startup: "Setting attention block size to 832 tokens to ensure that attention page size is >= mamba page size." For Qwen3.8-27B we have seen 784 tokens (vLLM 0.31.0), 832 (vLLM 0.33.0) and 1,568 (vLLM 0.31.0 with an FP8 KV cache).
- Long, growing conversations are fine. Coding agents and long multi-turn chats lose less than one block per request. Expect most prompt tokens to come from the cache, as here.
- Short prompts with a shared system prompt are where it hurts. A shared prefix shorter than one block is never reused. The part after the last whole block is recomputed every time. If your traffic looks like this, keep the shared part at the start of the prompt and byte-identical, count how many whole blocks it spans, and measure before you count on the cache.
- A prefix has to be seen twice. Traffic where many different prefixes each come back once will hit less than either number here.
- An FP8 KV cache doubles the block. It nearly doubles how many tokens fit in memory, and halves how finely prefixes can hit. On long agent prompts that matters little: a 1,568-token block is 1.4% of a 114,000-token prompt. (That is arithmetic. We haven't run FP8 KV on real traffic.) On short prompts it can wipe out the cache, as Monday's 1,468- vs 1,668-token test showed.
- Measure on your own traffic, weighted by tokens. Start vLLM with
--enable-prompt-tokens-details, and each response'susagecarriesprompt_tokens_details.cached_tokens. Divide the sum of cached tokens by the sum of prompt tokens. Here the per-request median (97.6% on Qwen3.8) read higher than the token-weighted rate (94.3%). The token-weighted rate is the one that tells you how much prefill the cache saved.
How we measured
- Source: our API gateway's request log. It holds metadata only, such as the model, prompt tokens and cached prompt tokens, and never prompt or completion text. We read it on November 20, 2026, around 17:50 UTC. It covered November 19 and 20.
- Traffic: requests made with our own self-serve API keys. In this window that was, in practice, one person on our team running a coding agent. Requests from other sources were left out, and so were prompts under 1,000 tokens.
- Cached tokens: vLLM's own per-request count (
prompt_tokens_details.cached_tokens), recorded by the gateway. - Serving: one H200 on RunPod Secure Cloud running two vLLM 0.33.0 engines side by side (CUDA MPS). Qwen3.8-27B (FP8 weights) has about half of the GPU's memory, Qwen3.6-35B-A3B (FP8 weights) most of the rest. Prefix caching is on, as by default, with a BF16 KV cache and a 262,144-token context. Qwen3.8's cache block on this node is 832 tokens; we didn't record Qwen3.6's.
Limits
- One user. 3,149 requests from one person's coding-agent sessions over about two days: one agent, one way of working. Other agents, and other people's habits, can hit differently.
- No session view. The log has no session ids, so we can't follow a conversation turn by turn or test the "new session" guess above.
- Low load. At high load vLLM evicts cached blocks sooner and hit rates drop. These numbers come from a lightly loaded node.
- Not a dense-model comparison. We served no dense-attention model on this traffic, so we can't say how close one would come to these rates.
Full numbers are in the report. Monday's synthetic test, with the block-size measurements and the capacity results, is here. Both models are on our API, where cached input tokens are billed at a lower price. Today we also published what MTP speculative decoding does for Qwen3.8-27B.
Keep reading
Prefix caching on a hybrid model: fewer cache hits, a bigger payoff (Qwen3.8-27B on one H100)
We repeated last week's prefix-cache capacity test on a hybrid model, Qwen3.8-27B (Gated DeltaNet + full attention), on the same H100 SXM setup. vLLM sizes each cache block to hold one DeltaNet state, 784 tokens with a BF16 KV cache and 1,568 with FP8, and only whole blocks hit. Hit rates came out far below the dense model's, but capacity within our latency target rose 19%, 36% and 70% at 50%, 75% and 90% shared on chat-like traffic, and 1.7×, 2.4× and 3.4× with 4,000-token prompts. A formula built on prefill's share of GPU time predicts a fraction of that. Thanks to Manas Choudhary, whose RTX 5090 runs found the block size.
MTP on Qwen3.8-27B: 2.65× faster per request, at the cost of 24% of the KV cache
Qwen3.8-27B ships with a one-layer multi-token prediction (MTP) head that vLLM can use as a draft model. On one H100 NVL with vLLM 0.33.0, three draft tokens per step raised single-request decode from 74.8 to 198 tokens per second (2.65×), and total throughput by 101%, 49% and 19% at 4, 16 and 48 requests in flight. GSM8K held (99.0% against 98.0%). The price is memory: room for the KV cache fell 24%, from 709,527 to 541,801 tokens. We turned it on in our API the same day, then off again that evening: with two models sharing the GPU under CUDA MPS, MTP crashed both engines under long-context load.
What a prompt cache is worth: the same H100 served 82% more requests at a 90% cache hit rate
We served Qwen3-30B-A3B-Instruct-2507 FP8 on our production H100 with vLLM and repeated the same latency-target search with 0%, 50%, 75% and 90% of every prompt drawn from shared prefixes. vLLM's own counters showed cache hit rates of 0%, 49%, 74% and 89%. Capacity rose from 11.5 to 21.0 requests per second, and revenue per GPU-hour from $10.37 to $18.84 at flat prices. With cached input billed at a quarter of the price, the node still earns more and the customer pays far less.