← All reports

Report · 2026-11-20

Prefix-cache hits on real coding-agent traffic: Qwen3.8-27B and Qwen3.6-35B-A3B on one H200, vLLM 0.33.0

1× NVIDIA H200Qwen3.8-27B-FP8 + Qwen3.6-35B-A3B-FP8FP8 weights, BF16 KV cachevLLM 0.33.0Real traffic · Nov 19–20, 2026

Read from our API gateway's request log (metadata only: model, prompt tokens, cached prompt tokens) on 2026-11-20 at about 17:50 UTC, covering 2026-11-19 and 2026-11-20. Requests on our own self-serve API keys, in practice one person on our team running a coding agent; prompts under 1,000 tokens left out; 3,149 requests. Both models are hybrid attention (Gated DeltaNet + full attention) with FP8 weights, served by two vLLM 0.33.0 engines sharing one H200 under CUDA MPS, prefix caching on, BF16 KV cache, 262,144-token context; Qwen3.8-27B's cache block on this node is 832 tokens. Hit rate = cached prompt tokens ÷ prompt tokens, from vLLM's per-request prompt_tokens_details.cached_tokens. Write-up: /blog/prefix-cache-real-agent-traffic-hybrid-qwen.html

ConclusionvLLM served 94.3% of Qwen3.8-27B's prompt tokens and 94.8% of Qwen3.6-35B-A3B's from its prefix cache, on median prompts of 114,258 and 85,817 tokens. The median request recomputed 2,690 and 1,562 tokens. On a synthetic test with ~1,000-token prompts, 90% shared, Qwen3.8-27B got 55.6%: the whole-block rule costs about one block per request, a large share of a short prompt and under 1% of a long one.

Results

Token-weighted: cached prompt tokens ÷ all prompt tokens in each group.

Qwen3.8-27B: prompt tokens from cache

% of prompt tokens, by prompt length · higher is better
1K–8K (n 12)9.6%1K–8K (n 12): 9.6 % of prompt tokens, by prompt length
8K–32K (n 44)42.7%8K–32K (n 44): 42.7 % of prompt tokens, by prompt length
32K–64K (n 147)92.7%32K–64K (n 147): 92.7 % of prompt tokens, by prompt length
64K+ (n 810)94.9%64K+ (n 810): 94.9 % of prompt tokens, by prompt length
All (n 1,013)94.3%All (n 1,013): 94.3 % of prompt tokens, by prompt length

Axis runs to 100%. n = requests in each group.

Qwen3.6-35B-A3B: prompt tokens from cache

% of prompt tokens, by prompt length · higher is better
1K–8K (n 12)7.6%1K–8K (n 12): 7.6 % of prompt tokens, by prompt length
8K–32K (n 203)86.0%8K–32K (n 203): 86.0 % of prompt tokens, by prompt length
32K–64K (n 348)87.8%32K–64K (n 348): 87.8 % of prompt tokens, by prompt length
64K+ (n 1,573)95.8%64K+ (n 1,573): 95.8 % of prompt tokens, by prompt length
All (n 2,136)94.8%All (n 2,136): 94.8 % of prompt tokens, by prompt length

Axis runs to 100%. n = requests in each group.

All numbers

Prompts of 1,000 tokens or more. p10 and p90 are the 10th and 90th percentiles.

MetricQwen3.8-27BQwen3.6-35B-A3B
Requests1,0132,136
Prompt tokens, median (p10–p90)114,258 (40,706–153,171)85,817 (31,888–121,662)
Prompt tokens from cache (token-weighted)94.3%94.8%
Hit rate per request, median (p10)97.6% (89.9%)98.0% (87.3%)
Uncached tokens per request, median (p90)2,690 (6,356)1,562 (5,830)
Requests with zero cache hits4432
Cache block size832 tokensnot recorded

By prompt length

Requests (n) and token-weighted hit rate per group. Qwen3.8 requests of 8K–32K tokens recomputed a median of 11,175 tokens each.

Prompt lengthQwen3.8 nQwen3.8 from cacheQwen3.6 nQwen3.6 from cache
1K–8K tokens129.6%127.6%
8K–32K4442.7%20386.0%
32K–64K14792.7%34887.8%
64K and more81094.9%1,57395.8%

Context: the synthetic test

From the 2026-11-16 post (/blog/prefix-cache-hybrid-qwen38-2026.html); different GPU, engine version and workload.

Synthetic test (2026-11-16 post)Real coding-agent traffic (this report)
GPU and engine1× H100 SXM, vLLM 0.31.01× H200, vLLM 0.33.0
Qwen3.8-27B cache block784 tokens832 tokens
Promptsmedian 1,024 tokens, 90% from 8 shared prefixesmedian 114,258 tokens, growing conversations
Qwen3.8-27B prompt tokens from cache55.6%94.3%
Dense-attention model, same workload89.2% (Qwen3-30B-A3B)not measured

Setup

Any change to these fields makes it a different report.

GPU
1× NVIDIA H200 SXM 141 GB, RunPod Secure Cloud (our API's server)
Engines
vLLM 0.33.0 (vllm/vllm-openai:v0.33.0), two engines on one GPU under CUDA MPS
Models
Qwen/Qwen3.8-27B-FP8 (about half of GPU memory), Qwen/Qwen3.6-35B-A3B-FP8 (most of the rest)
Architecture
Both hybrid attention: Gated DeltaNet + full attention (Qwen3.8: 16 of 64 layers full attention)
Prefix caching
on (vLLM default), BF16 KV cache, 262,144-token context
Cache block
Qwen3.8-27B: 832 tokens (vLLM startup log); Qwen3.6: not recorded
Traffic
Our own self-serve API keys; in practice one person on our team running a coding agent
Window
2026-11-19 to 2026-11-20, read 2026-11-20 ~17:50 UTC
Filter
Prompts of 1,000 tokens or more; other traffic sources left out
Source
API gateway request log: model, prompt tokens, cached prompt tokens (no prompt or completion text)
Hit rate
Σ cached prompt tokens ÷ Σ prompt tokens (token-weighted), per model and per length group
Per-request rate
cached ÷ prompt tokens of each request; median and p10 across requests

Not covered

Do not extrapolate this report to the following.

  • Other users, other coding agents or other kinds of traffic: one person's sessions over about two days
  • Session-level behaviour: the request log has no session ids
  • High load: hit rates were measured on a lightly loaded node; at high load vLLM evicts cached blocks sooner
  • A dense-attention model on the same traffic
  • FP8 KV cache (larger cache blocks) on real traffic
  • Qwen3.6-35B-A3B's cache block size

How to measure it on your own traffic

The request log itself isn't published: it is our team's usage. This is how the same numbers come out of any vLLM deployment.

shell
# start vLLM so that usage reports cached prompt tokens
vllm serve Qwen/Qwen3.8-27B-FP8 --enable-prompt-tokens-details ...   # prefix caching is on by default

# every response's usage then carries:
#   usage.prompt_tokens
#   usage.prompt_tokens_details.cached_tokens
# log both per request, then:
#   token-weighted hit rate = sum(cached_tokens) / sum(prompt_tokens)

# the cache block size is in the engine's startup log:
#   "Setting attention block size to 832 tokens to ensure that attention page size is >= mamba page size."

Read the write-up Download raw JSON