← Lab notes

Prefix caching on a hybrid model: fewer cache hits, a bigger payoff (Qwen3.8-27B on one H100)

We repeated last week's prefix-cache capacity test on a hybrid model, Qwen3.8-27B (Gated DeltaNet + full attention), on the same H100 SXM setup. vLLM sizes each cache block to hold one DeltaNet state, 784 tokens with a BF16 KV cache and 1,568 with FP8, and only whole blocks hit. Hit rates came out far below the dense model's, but capacity within our latency target rose 19%, 36% and 70% at 50%, 75% and 90% shared on chat-like traffic, and 1.7×, 2.4× and 3.4× with 4,000-token prompts. A formula built on prefill's share of GPU time predicts a fraction of that. Thanks to Manas Choudhary, whose RTX 5090 runs found the block size.

Choosing GPUs? Compare GPUs for your traffic · Get your node measured, free

Last week we measured what a prompt cache is worth on a dense-attention model: at an 89% cache hit rate, one H100 served 82% more requests within our latency target. Then Manas Choudhary, an inference engineer, ran Qwen3.8-27B on an RTX 5090 and found a catch. Qwen3.8 is a hybrid: 48 of its 64 layers are Gated DeltaNet (linear attention) and only 16 are full attention. To reuse a cached prefix, vLLM also needs the DeltaNet state at the point where the prefix ends, so it sizes every cache block to hold one such state: 1,568 tokens in his setup instead of the usual 16. Only whole blocks hit. With 4,000-token prompts he got 36.7% hits at 50% shared, still 36.7% at 75%, and 75.9% at 90%.

We said we'd put Qwen3.8 through the same capacity test on an H100. Here it is.

What we ran. Qwen/Qwen3.8-27B-FP8 (the official FP8 checkpoint) on one H100 SXM (RunPod Secure Cloud, $3.99 an hour), vLLM 0.31.0, the same image and driver as last week's node. vLLM defaults, so prefix caching on and a BF16 KV cache, served text-only. Same harness, workload and latency target as the dense test: random-token prompts with median 1,024 input and 256 output tokens, p99 time to first token ≤ 2 s, p99 time per output token ≤ 50 ms, a fixed share of every prompt (0%, 50%, 75%, 90%) drawn from one of eight shared prefixes. We repeated the search with every prompt exactly 4,000 tokens, for a direct comparison with Manas's numbers. Hit rates are vLLM's own counters.

The block is 784 tokens here, 1,568 with an FP8 KV cache

vLLM's startup log says it: "Setting attention block size to 784 tokens to ensure that attention page size is >= mamba page size." One DeltaNet layer's state in Qwen3.8-27B is 3.2 MB (48 heads × 128 × 128 in FP32, plus a small convolution buffer). One full-attention layer's BF16 KV cache takes 4 KB per token, so a page that holds one state holds 783 tokens' worth of KV, rounded up to 784. An FP8 KV cache halves the bytes per token, so the same page holds 1,568 tokens: the number Manas saw. Restarted with --kv-cache-dtype fp8, our H100 picked 1,568 too.

We mapped it with sequential requests, each 4,000 tokens long, sharing a prefix of a given length:

Shared prefix (tokens) Cached tokens (3rd request on) Time to process the prompt
0–783 0 252 ms
784–1,567 784 215 ms
1,568–2,351 1,568 176 ms
2,352–3,135 2,352 137 ms
3,136–3,999 3,136 109 ms

A prefix of 1,567 tokens gets the same 784 cached tokens as one of 784. Each cached block saves 28–39 ms of prefill.

One more rule showed up: a prefix had to be seen twice. The second request with a new prefix got 0 cached tokens at every prefix length we tried (and took 273 ms instead of 254); from the third request on, the hits above appeared. The same happened when the second request shared only part of a longer prefix. The dense model, on the same node, hit on the second request, in 16-token steps.

What it does to capacity

Chat-like traffic, the same workload as last week's dense test:

Shared prefix Hit rate, Qwen3.8-27B Req/s within target Hit rate, dense MoE (last week) Req/s, dense
0% 0% 3.43 0% 11.5
50% 21.1% 4.10 (+19%) 48.7% 16.4 (+42%)
75% 40.8% 4.66 (+36%) 74.0% 19.8 (+72%)
90% 55.6% 5.83 (+70%) 89.2% 21.0 (+82%)

The block size explains most of the gap. Prompt lengths vary (the median is 1,024 tokens), so at 50% shared the median request shares 512 tokens, less than one block, and only longer prompts hit at all. Whole-block hits on this workload would cap the hit rate at 30%, 53% and 68%; vLLM counted 21%, 41% and 56%, below even that.

With 4,000-token prompts every shared prefix spans at least two whole blocks:

Shared prefix Shared tokens Hit rate (vLLM) Most a 784-token block allows Req/s within target
0% 0 0% 0% 1.41
50% 2,000 37.0% 39.2% 2.36 (1.68×)
75% 3,000 58.6% 58.8% 3.36 (2.38×)
90% 3,600 74.2% 78.4% 4.77 (3.39×)

Manas's 36.7% at 75% shared is the 1,568-token block: 3,000 shared tokens span one 1,568-token block but three 784-token blocks. On this model, the KV cache precision decides the step size of the cache. FP8 KV nearly doubles how many tokens fit in memory (572,757 to 1,018,235 here) and halves how finely prefixes can hit.

Just below and just above a block boundary

To show the step under load, we held the node at 1.41 requests per second (its no-reuse capacity with 4,000-token prompts) and shared either 1,468 or 1,668 tokens of every prompt: 100 tokens below and above 1,568, which is two 784-token blocks or one 1,568-token block. One two-minute window each:

KV cache Shared tokens Hit rate (vLLM) p99 first token p99 time per token
BF16 (784-token block) 0 0% 935 ms 44.2 ms
BF16 1,468 18.3% 729 ms 35.4 ms
BF16 1,668 36.6% 585 ms 28.0 ms
FP8 (1,568-token block) 0 0% 893 ms 48.3 ms
FP8 1,468 0% 905 ms 49.8 ms
FP8 1,668 36.6% 605 ms 30.4 ms

With an FP8 KV cache, 1,468 shared tokens are worth nothing: the node runs as if nothing were shared, 0.2 ms under the 50 ms limit. Two hundred more shared tokens cut p99 time per token by 39% and p99 time to first token by a third. With BF16 the same 200 tokens move it from one block to two.

The formula, checked

Manas suggested speedup ≈ 1 / (1 − f × hit rate), where f is the share of GPU time spent on prefill. We measured f directly: prefill-only throughput on this engine is about 18,500 tokens per second, and at the no-reuse capacity the node takes in 5,240 input tokens per second (chat-like) or 5,833 (4,000-token prompts), so f ≈ 0.28 and 0.31.

With those, the formula predicts much less than we measured:

Workload Hit rate Predicted, f from GPU time Measured f that fits
Chat-like 21.1% 1.06× 1.19× 0.77
Chat-like 40.8% 1.13× 1.36× 0.65
Chat-like 55.6% 1.19× 1.70× 0.74
4,000 tokens 37.0% 1.13× 1.68× 1.09
4,000 tokens 58.6% 1.22× 2.38× 0.99
4,000 tokens 74.2% 1.30× 3.39× 0.95

The shape holds: one f per workload fits all three points (0.71 for chat-like traffic, 0.98 for 4,000-token prompts; Manas's 1.43× and 2.62× correspond to about 0.8). The GPU-time share doesn't. At 37% hits the measured 1.68× is even above what removing all prefill time would allow under the formula (1.59×). The dense model shows the same gap, smaller: on this node it prefills about 63,000 tokens per second, so last week's no-reuse run spent f ≈ 0.31 of its GPU time on prefill, while the f that fits its speedups is 0.54.

Why the gap? Not because prefill takes a bigger share of the average time per token than of GPU time. We ran the same batch sizes as pure decode, with no prefill in the batch: 21.5 ms per token at the chat-like no-reuse batch (41 requests) against 27.8 ms measured at capacity, so prefill added 23% to the average. With 4,000-token prompts it added 25%; on the dense model, 36%.

The difference is in the tail. What limits this node is the p99 time per output token, and our reading is that the requests at the p99 are the ones that landed on decode steps carrying a chunk of someone's long prompt. Without reuse, p99 time per token was 1.60× the median on chat-like traffic and 1.84× with 4,000-token prompts; at 90% shared, 1.31× and 1.20×. The cache removes the slow steps that set the limit, so the median can rise until the p99 meets the target again.

So against a tail-latency target, prefill's share of GPU time, or even of the average time per token, understates what a prefix cache is worth on this model. The formula works with an f fitted from a couple of measured points.

What it's worth

Revenue per GPU-hour at full load, at last week's flat prices ($0.08 per million input tokens, $0.28 per million output) so the two models compare directly:

No reuse 50% shared 75% shared 90% shared
Qwen3.8-27B, chat-like, flat prices $2.90 $3.58 $4.09 $5.01
Qwen3.8-27B, cached input at $0.02 $2.90 $3.27 $3.42 $3.89
Dense MoE (last week), flat prices $10.37 $14.61 $17.76 $18.84
Dense MoE, cached input at $0.02 $10.37 $11.73 $12.40 $11.97

A model that computes with all 27B parameters for every token serves 30% as many requests as a 3B-active MoE on the same GPU, so at the same prices it earns 28% as much. The relative gain from the cache is similar (+73% at 90% shared, against +82%). At these prices the hybrid node covers its $3.99 rent only with heavy reuse and no cached discount.

Per kilowatt-hour of GPU board power, output went from 8.2 to 12.9 million tokens (+58%) on chat-like traffic, and from 3.8 to 10.8 million (2.8×) with 4,000-token prompts. The dense model went from 28.1 to 49.2 million.

Caveats

  • Quick mode. Two two-minute windows per rate, and the search steps about 6% apart. Several confirmations backed off once or twice (the no-reuse 4,000-token run passed at 1.74 req/s in the search and held only at 1.41), so read the ratios as ±10%.
  • Synthetic prefixes. Eight shared prefixes, random tokens, every prefix seen many times. Real traffic with many prefixes that each come back once would hit far less, given the see-it-twice rule.
  • One model, one engine version, one setting. BF16 KV cache (vLLM's default) for the capacity runs; FP8 KV only for the block-boundary runs. No speculative decoding: Qwen3.8 ships an MTP head that would change decode speed. We had to set --max-num-seqs 512: with vLLM's default of 1,024, Qwen3.8-27B doesn't start on one H100 ("max_num_seqs (1024) exceeds available Mamba cache blocks (828)"). At capacity, at most 86 requests were in flight on average, so the limit didn't bind.
  • Different nodes, same setup. The dense numbers are last week's (same GPU type, driver, image and harness, different pod). We reran the dense model at 50% shared on this week's node as a check: 48.73% hits (last week 48.73%) and 17.4 requests per second (last week 16.4, one search step apart).
  • Board power only. GPU board power from nvidia-smi; no CPU, memory or cooling.

Thanks

To Manas Choudhary for finding the block size, publishing his numbers and inviting a rerun. The H100 numbers agree with his on where the cache steps, and add two things: the step size depends on the KV cache precision, and under a latency target the payoff is larger than the share of GPU time suggests.

Try it yourself

The shared-prefix workload is in TokenWatt Bench (workload.shared_prefix, workload.prefix_pool), so you can run the same test on your own nodes, or ask us to measure yours. Full numbers and raw data are in the reports for chat-like traffic and 4,000-token prompts.

Update (2026-11-20): on real coding-agent traffic, with prompts around 100K tokens, the same block rule cost little: 94% of prompt tokens came from the cache.

Keep reading

5 min read

94% prefix-cache hits on hybrid models with real coding-agent traffic, against 56% in our synthetic test

A synthetic test we published on November 16 showed a hybrid model, Qwen3.8-27B, reusing far less of a shared prompt than a dense-attention model (56% against 89%), because vLLM caches its prompts only in whole blocks of 784 tokens. On two days of our own team's coding-agent traffic on our API, the same rule cost little: vLLM served 94.3% of Qwen3.8-27B's prompt tokens and 94.8% of Qwen3.6-35B-A3B's from cache, on median prompts of 114K and 86K tokens. The median request recomputed only 1,562 to 2,690 tokens.

prefix cachingprompt cachehybrid attention
3 min read

What a prompt cache is worth: the same H100 served 82% more requests at a 90% cache hit rate

We served Qwen3-30B-A3B-Instruct-2507 FP8 on our production H100 with vLLM and repeated the same latency-target search with 0%, 50%, 75% and 90% of every prompt drawn from shared prefixes. vLLM's own counters showed cache hit rates of 0%, 49%, 74% and 89%. Capacity rose from 11.5 to 21.0 requests per second, and revenue per GPU-hour from $10.37 to $18.84 at flat prices. With cached input billed at a quarter of the price, the node still earns more and the customer pays far less.

prefix cachingprompt cachevLLM
3 min read

Long prompts are where an inference node loses money: 1K, 8K and 16K input tokens on one H100

We measured how much one H100 can serve within a latency target as prompts grow from about 1K to 8K and 16K input tokens (Qwen3-30B-A3B FP8, vLLM, no prefix reuse). Capacity fell from 11.5 to 1.89 and 0.73 requests per second, because the first-token deadline caps how many long prompts the GPU can prefill at once. At a flat input price, long uncached prompts earn less than the GPU costs.

long contextprefilltime to first token