What a prompt cache is worth: the same H100 served 82% more requests at a 90% cache hit rate
We served Qwen3-30B-A3B-Instruct-2507 FP8 on our production H100 with vLLM and repeated the same latency-target search with 0%, 50%, 75% and 90% of every prompt drawn from shared prefixes. vLLM's own counters showed cache hit rates of 0%, 49%, 74% and 89%. Capacity rose from 11.5 to 21.0 requests per second, and revenue per GPU-hour from $10.37 to $18.84 at flat prices. With cached input billed at a quarter of the price, the node still earns more and the customer pays far less.
Choosing GPUs? Compare GPUs for your traffic · Get your node measured, free
Last week we measured what an inference provider earns per GPU-hour and found that a default setup loses money at the most common price. The most common reply in the comments: the margin is in the cache. Real traffic repeats itself (system prompts, tool definitions, shared documents, an agent's growing context), and an engine that has already processed a prefix doesn't have to process it again.
So we measured how much that's worth, on the H100 we ran our test API on.
What we ran. Qwen3-30B-A3B-Instruct-2507 FP8 on one H100 SXM (RunPod Secure Cloud, $3.99 an hour), vLLM 0.31.0 with its default automatic prefix caching, our chat-like workload (median 1,024 input and 256 output tokens) and our usual latency target: p99 time to first token ≤ 2 s and p99 time per output token ≤ 50 ms. We ran the same capacity search four times. Each time, a fixed share of every prompt (0%, 50%, 75%, 90%) came from one of eight shared prefixes and the rest was fresh. We read vLLM's own prefix-cache counters before and after each run, so the hit rates below are what the engine reported, not what we intended.
What the cache did
| Shared prefix | Cache hit rate (vLLM) | Requests/s within target | Input tokens/s | Output tokens/s | Requests in flight | p99 first token |
|---|---|---|---|---|---|---|
| 0% | 0% | 11.5 | 19,430 | 4,740 | 143 | 380 ms |
| 50% | 48.7% | 16.4 (+42%) | 27,363 | 6,678 | 244 | 294 ms |
| 75% | 74.0% | 19.8 (+72%) | 33,522 | 8,041 | 291 | 248 ms |
| 90% | 89.2% | 21.0 (+82%) | 35,648 | 8,508 | 282 | 257 ms |
Two things stand out. First, the gains flatten: going from 75% to 90% cached added only 6%. Once most of the prompt is cached, decoding output tokens is the bottleneck, and the cache doesn't help there. Second, first-token latency got better as load went up, because a cached prompt skips most of its prefill.
What that's worth
Revenue per GPU-hour at full load = (input tokens/s × input price + output tokens/s × output price) × 3,600:
| Cache hit rate | At $0.08 / $0.28, cache not discounted | At our price: cached input $0.02 |
|---|---|---|
| 0% | $10.37 | $10.37 |
| 49% | $14.61 | $11.73 |
| 74% | $17.76 | $12.40 |
| 89% | $18.84 | $11.97 |
At flat prices, a 90% hit rate nearly doubles what the GPU earns. That's the provider's view, and it's why prompt caching is where margins come from.
The customer's view: at an 89% hit rate, the average input token in this workload costs $0.027 per million with cached tokens at $0.02, against $0.08 without a discount: two-thirds less. The node still earns 15% more per hour than with no reuse at all, because it serves 82% more requests. Both sides win: a provider can discount cached input by 75% and still earn more per GPU-hour than with no reuse at all.
Against the $3.99 rent, the node breaks even at 32–38% of full load across these hit rates, with the discount applied.
Caveats
- Quick mode. Two two-minute windows per rate. Our full-mode run on the same node without reuse measured 9.6 requests/s (five ten-minute windows), lower than the 11.5 here. Treat the ratios as more reliable than the absolute numbers.
- Synthetic prefixes. Eight shared prefixes and random tokens, a clean case for the cache. Real hit rates depend on how much of your traffic really repeats and how long prefixes stay in GPU memory under load.
- One model, one GPU, one engine. A dense model, longer contexts or a different engine will move these numbers.
- Admission control matters. Our gateway admitted up to 110 requests at a time, set from the no-reuse benchmark. At high hit rates the node handled about 280. Counting requests instead of work leaves capacity unused when traffic is cache-friendly.
Try it yourself
The shared-prefix workload is now an option in TokenWatt Bench (workload.shared_prefix, workload.prefix_pool), so you can run the same test on your own nodes, or ask us to measure yours. Full numbers and raw data are in the report.
Keep reading
Do inference providers really make $10–20 per GPU-hour? We measured DeepSeek V4.1 Flash on H200s
A widely shared post says inference providers earn $10–20 per GPU-hour serving open models on rented GPUs. We served DeepSeek V4.1 Flash on 4× H200 with vLLM's published recipe and measured what it can sell within a latency target. At the prices 38 OpenRouter endpoints charge, a fully loaded H200 earns $1.80–8.84 an hour, $4.42 at the most common price, against $5.29 rent. At 100 tokens per second per user it earns $0.24. Margins exist, but they depend on hardware, tuning and pricing, not on showing up.
H100 vs A100 for LLM inference: 7.8× the capacity for 2.2× the price
We served Qwen3-8B with vLLM on a rented H100 SXM and a rented A100 SXM 80GB under the same latency target. The H100 served 8.79 requests per second against 1.13 on the A100: 7.8× the capacity for 2.2× the hourly price, so each token cost 71% less ($0.31 against $1.06 per million output tokens). On raw specs the H100 is only 1.7–2.6× faster. The rest of the gap comes from slow tokens on the A100, which a serving setting can partly fix.
Renting an H100? The variant matters more than the luck of the draw
We rented 19 H100s on RunPod Secure Cloud and checked each one, then served the same 8B model on one of each variant. All 10 SXM cards were healthy and nearly identical. Every NVL card was set to 310 W, below NVIDIA's 350–400 W range, and the PCIe cards split into two speed groups 12% apart. Per token, the SXM was the cheapest: $0.31 per million output tokens, against $0.43 on PCIe and $0.57 on NVL.