H100 vs A100 for LLM inference: 7.8× the capacity for 2.2× the price

"Should we rent H100s or A100s?" is usually answered with spec sheets: the H100 SXM has about 2.6× the dense BF16 compute and 1.7× the memory bandwidth of an A100, at a bit more than twice the hourly price. On paper that's roughly a wash.

We measured it instead. Same model, same engine, same latency target, both cards rented on RunPod Secure Cloud:

The H100 served 7.8× as many requests for 2.2× the price. Per token it cost 71% less, and it did 4× more work per joule.

How we measured

Both GPUs ran Qwen3-8B in BF16 on the same container (vllm/vllm-openai:v0.31.0) with default settings. The workload mixes short and long prompts (median 1,024 input tokens, some up to 8,000) with median 256 output tokens, arriving at random. We raised the request rate until the latency target broke: p99 time to first token ≤ 2 s and p99 time per output token ≤ 50 ms. Capacity is the highest rate that still met both. Accuracy on a fixed GSM8K set was identical on both cards.

Both nodes passed a health check first: full power limit (400 W and 700 W), clocks under load and memory bandwidth in line with healthy cards of each model. Full settings and raw data: A100 report, H100 report.

Why the gap is bigger than the spec sheet

In our health checks the H100 SXM delivered about 680 TFLOPS of dense BF16 against 260 on the A100 (2.6×), and 3.0 TB/s of copy bandwidth against 1.75 (1.7×). Token generation mostly follows memory bandwidth, so you'd expect the H100 to serve somewhere around 2× as much. It served 7.8×.

The difference is the slowest tokens. When a long prompt arrives, the engine processes it in the same step as everyone else's next token, and those tokens wait. On the A100, with less compute to get through the prompt, that wait was long enough to push the slowest 1% of tokens past 50 ms at a low rate: at 1.4 requests per second its median time per token was 17 ms, but its p99 was 64 ms. The H100 got through the same prompts fast enough to stay under the target all the way to 8.79 requests per second (p99 37.9 ms).

So averages hide most of the difference. Measured on median speed alone, the A100 looks fine. It's the tail that sets how many users you can serve at a latency promise.

How much of that is fixable

Some of it. How much prompt the engine processes per step is a setting (max-num-batched-tokens in vLLM, 8,192 by default). Smaller chunks shorten the stalls at some cost to prompt throughput. On an H200 serving a 70B model, lowering it to 1,024 raised capacity about 15% in an earlier test, within quick-mode noise. A tuned A100 would close part of the gap, but not 7.8× down to the price ratio of 2.2×: the H100 would still have more than twice the compute and 1.7× the bandwidth.

Which one to rent

• Serving an LLM to users with a latency promise: the H100 SXM, even at twice the price. It was the cheapest per token of every GPU we've measured on this model, including the RTX 4090, L40S and A40 (comparison).
• Low or bursty traffic, batch jobs without a tight latency target, or a fixed budget per hour: the A100. At 1 request per second or less you're paying for idle capacity on an H100 anyway, and the A100's 80 GB holds the same models.
• Either way, check which H100 you get. "H100" can mean SXM, PCIe or NVL, and in our H100 study the SXM served 2.25× as much as an NVL set to 310 W. Then check the node itself: pip install tokenwatt-check compares it with a healthy card of the same model in two minutes (how).

Limits

One node of each GPU, one platform, on different days (October 11 and 13), quick mode (2-minute windows, 2 repeats), which reads 15–20% higher in absolute capacity than our full mode, so compare the two cards with each other rather than with full-mode numbers. One dense 8B model in BF16 with default settings and no tuning; the A100 doesn't support FP8, which raised capacity 19% on an RTX 4090 in another test. Prices are RunPod Secure Cloud on the day. Power is GPU board power, not wall power.

Method: how we measure · Prices across providers: H100, A100.