Long prompts are where an inference node loses money: 1K, 8K and 16K input tokens on one H100
We measured how much one H100 can serve within a latency target as prompts grow from about 1K to 8K and 16K input tokens (Qwen3-30B-A3B FP8, vLLM, no prefix reuse). Capacity fell from 11.5 to 1.89 and 0.73 requests per second, because the first-token deadline caps how many long prompts the GPU can prefill at once. At a flat input price, long uncached prompts earn less than the GPU costs.
Choosing GPUs? Compare GPUs for your traffic · Get your node measured, free
Yesterday we measured what a prompt cache is worth: at an 89% cache hit rate the same H100 served 82% more requests. This is the other half: what happens when prompts are long and don't repeat. That's RAG over fresh documents, long files pasted into a chat, an agent's first turn on a new repository.
What we ran. Qwen3-30B-A3B-Instruct-2507 FP8 on one H100 SXM (RunPod Secure Cloud, $3.99 an hour), vLLM 0.31.0 with FP8 KV cache, our usual latency target (p99 time to first token ≤ 2 s, p99 time per output token ≤ 50 ms), output around 400 tokens. Three prompt sizes, random tokens so nothing is cached between requests. Quick mode: the capacity search, then two two-minute windows at the highest rate that passed.
What one H100 can serve
| Input per request | Requests/s within target | Input tokens/s | Output tokens/s | Requests in flight | p99 first token |
|---|---|---|---|---|---|
| ~1K (chat) | 11.5 | 19,430 | 4,740 | 143 | 0.38 s |
| ~8K (RAG) | 1.89 | 17,149 | 917 | 14 | 1.10 s |
| ~16K (long document) | 0.73 | 10,934 | 330 | 3 | 1.26 s |
Eight times the input cost 84% of the request rate, not 88%: the node still pushed about as many input tokens per second at 8K as at 1K. At 16K, total input throughput fell too, and the node handled only about three requests at a time.
Why it drops. Before a request can produce its first token, the engine has to process the whole prompt (prefill), and prefill is compute-bound. With the first token due within two seconds, only a few long prompts can be in prefill at once. Output speed was fine; at 16K the slowest tokens came out at 22 ms each, well inside the 50 ms target. The first-token deadline is what caps the node.
What that's worth
Revenue per GPU-hour at full load, at $0.08 per million input tokens and $0.28 per million output:
| Input per request | Revenue per GPU-hour | Rent | Break-even utilization |
|---|---|---|---|
| ~1K | $10.37 | $3.99 | 38% |
| ~8K | $5.86 | $3.99 | 68% |
| ~16K | $3.48 | $3.99 | never |
At a flat input price, a node full of 16K-token prompts that don't repeat loses money even at 100% utilization. Two things fix it:
- Price input by what it costs to serve. A flat per-token price assumes every input token costs the same. For uncached long prompts it doesn't: at 16K the node served under 40% of the requests it served at 8K. Long-context pricing tiers exist for this reason.
- Cache. Repeated prefixes skip prefill. In our cache test, an 89% hit rate on ~1K prompts raised capacity 82%; for long prompts, where prefill is the bottleneck, the effect should be larger. Agents and chat-with-a-codebase traffic repeat most of their context, which is why they can be served profitably at all.
If your traffic is mostly long, fresh prompts, measure capacity at your prompt length before you size a fleet or set a price. A number measured on short chat prompts will overstate it by an order of magnitude.
Caveats
- Quick mode, one node, one model. A dense model or a larger MoE will scale differently; so will a different engine or chunked-prefill setting (in our tuning test, changing the prefill chunk size didn't move capacity on short prompts; we didn't re-test it on long ones).
- Random-token prompts. Real documents tokenize differently, but prefill cost depends on token count, not content.
- Single target. A looser first-token target (say 5 s for batch RAG) would admit more long prompts at once.
Raw results: 8K report, 16K report. If you serve long-context traffic and want your node measured at your prompt lengths, we'll do it for free.
Keep reading
What a prompt cache is worth: the same H100 served 82% more requests at a 90% cache hit rate
We served Qwen3-30B-A3B-Instruct-2507 FP8 on our production H100 with vLLM and repeated the same latency-target search with 0%, 50%, 75% and 90% of every prompt drawn from shared prefixes. vLLM's own counters showed cache hit rates of 0%, 49%, 74% and 89%. Capacity rose from 11.5 to 21.0 requests per second, and revenue per GPU-hour from $10.37 to $18.84 at flat prices. With cached input billed at a quarter of the price, the node still earns more and the customer pays far less.
Does selling tokens pay? Break-even math for an inference provider, from a measured H100
Using measured SLO throughput for Qwen3-30B-A3B on one H100 and live OpenRouter prices, we work out the break-even utilization for rented, owned and idle GPUs. Rented GPUs need 64–77% utilization to match the cheapest price; idle owned GPUs break even at 2%. Includes a calculator.
Do inference providers really make $10–20 per GPU-hour? We measured DeepSeek V4.1 Flash on H200s
A widely shared post says inference providers earn $10–20 per GPU-hour serving open models on rented GPUs. We served DeepSeek V4.1 Flash on 4× H200 with vLLM's published recipe and measured what it can sell within a latency target. At the prices 38 OpenRouter endpoints charge, a fully loaded H200 earns $1.80–8.84 an hour, $4.42 at the most common price, against $5.29 rent. At 100 tokens per second per user it earns $0.24. Margins exist, but they depend on hardware, tuning and pricing, not on showing up.