Prefix-cache hits on real coding-agent traffic: Qwen3.8-27B and Qwen3.6-35B-A3B on one H200, vLLM 0.33.0
Read from our API gateway's request log (metadata only: model, prompt tokens, cached prompt tokens) on 2026-11-20 at about 17:50 UTC, covering 2026-11-19 and 2026-11-20. Requests on our own self-serve API keys, in practice one person on our team running a coding agent; prompts under 1,000 tokens left out; 3,149 requests. Both models are hybrid attention (Gated DeltaNet + full attention) with FP8 weights, served by two vLLM 0.33.0 engines sharing one H200 under CUDA MPS, prefix caching on, BF16 KV cache, 262,144-token context; Qwen3.8-27B's cache block on this node is 832 tokens. Hit rate = cached prompt tokens ÷ prompt tokens, from vLLM's per-request prompt_tokens_details.cached_tokens. Write-up: /blog/prefix-cache-real-agent-traffic-hybrid-qwen.html
Results
Token-weighted: cached prompt tokens ÷ all prompt tokens in each group.
Qwen3.8-27B: prompt tokens from cache
Axis runs to 100%. n = requests in each group.
Qwen3.6-35B-A3B: prompt tokens from cache
Axis runs to 100%. n = requests in each group.
All numbers
Prompts of 1,000 tokens or more. p10 and p90 are the 10th and 90th percentiles.
| Metric | Qwen3.8-27B | Qwen3.6-35B-A3B |
|---|---|---|
| Requests | 1,013 | 2,136 |
| Prompt tokens, median (p10–p90) | 114,258 (40,706–153,171) | 85,817 (31,888–121,662) |
| Prompt tokens from cache (token-weighted) | 94.3% | 94.8% |
| Hit rate per request, median (p10) | 97.6% (89.9%) | 98.0% (87.3%) |
| Uncached tokens per request, median (p90) | 2,690 (6,356) | 1,562 (5,830) |
| Requests with zero cache hits | 44 | 32 |
| Cache block size | 832 tokens | not recorded |
By prompt length
Requests (n) and token-weighted hit rate per group. Qwen3.8 requests of 8K–32K tokens recomputed a median of 11,175 tokens each.
| Prompt length | Qwen3.8 n | Qwen3.8 from cache | Qwen3.6 n | Qwen3.6 from cache |
|---|---|---|---|---|
| 1K–8K tokens | 12 | 9.6% | 12 | 7.6% |
| 8K–32K | 44 | 42.7% | 203 | 86.0% |
| 32K–64K | 147 | 92.7% | 348 | 87.8% |
| 64K and more | 810 | 94.9% | 1,573 | 95.8% |
Context: the synthetic test
From the 2026-11-16 post (/blog/prefix-cache-hybrid-qwen38-2026.html); different GPU, engine version and workload.
| Synthetic test (2026-11-16 post) | Real coding-agent traffic (this report) | |
|---|---|---|
| GPU and engine | 1× H100 SXM, vLLM 0.31.0 | 1× H200, vLLM 0.33.0 |
| Qwen3.8-27B cache block | 784 tokens | 832 tokens |
| Prompts | median 1,024 tokens, 90% from 8 shared prefixes | median 114,258 tokens, growing conversations |
| Qwen3.8-27B prompt tokens from cache | 55.6% | 94.3% |
| Dense-attention model, same workload | 89.2% (Qwen3-30B-A3B) | not measured |
Setup
Any change to these fields makes it a different report.
- GPU
- 1× NVIDIA H200 SXM 141 GB, RunPod Secure Cloud (our API's server)
- Engines
- vLLM 0.33.0 (vllm/vllm-openai:v0.33.0), two engines on one GPU under CUDA MPS
- Models
- Qwen/Qwen3.8-27B-FP8 (about half of GPU memory), Qwen/Qwen3.6-35B-A3B-FP8 (most of the rest)
- Architecture
- Both hybrid attention: Gated DeltaNet + full attention (Qwen3.8: 16 of 64 layers full attention)
- Prefix caching
- on (vLLM default), BF16 KV cache, 262,144-token context
- Cache block
- Qwen3.8-27B: 832 tokens (vLLM startup log); Qwen3.6: not recorded
- Traffic
- Our own self-serve API keys; in practice one person on our team running a coding agent
- Window
- 2026-11-19 to 2026-11-20, read 2026-11-20 ~17:50 UTC
- Filter
- Prompts of 1,000 tokens or more; other traffic sources left out
- Source
- API gateway request log: model, prompt tokens, cached prompt tokens (no prompt or completion text)
- Hit rate
- Σ cached prompt tokens ÷ Σ prompt tokens (token-weighted), per model and per length group
- Per-request rate
- cached ÷ prompt tokens of each request; median and p10 across requests
Not covered
Do not extrapolate this report to the following.
- Other users, other coding agents or other kinds of traffic: one person's sessions over about two days
- Session-level behaviour: the request log has no session ids
- High load: hit rates were measured on a lightly loaded node; at high load vLLM evicts cached blocks sooner
- A dense-attention model on the same traffic
- FP8 KV cache (larger cache blocks) on real traffic
- Qwen3.6-35B-A3B's cache block size
How to measure it on your own traffic
The request log itself isn't published: it is our team's usage. This is how the same numbers come out of any vLLM deployment.
# start vLLM so that usage reports cached prompt tokens vllm serve Qwen/Qwen3.8-27B-FP8 --enable-prompt-tokens-details ... # prefix caching is on by default # every response's usage then carries: # usage.prompt_tokens # usage.prompt_tokens_details.cached_tokens # log both per request, then: # token-weighted hit rate = sum(cached_tokens) / sum(prompt_tokens) # the cache block size is in the engine's startup log: # "Setting attention block size to 832 tokens to ensure that attention page size is >= mamba page size."