H100 SXM vs PCIe vs NVL: LLM Inference Cost per Token

"H100" on a rental listing can mean three different boards: the SXM module (700 W, the one in most benchmarks), the PCIe card (300–350 W) and the NVL card (94 GB, 350–400 W). They're priced closer together than they perform. We wanted to know two things: how much one rented H100 differs from the next, and what each variant actually costs per token.

Results for Qwen3-8B (BF16) on vLLM 0.31.0, one GPU each, RunPod Secure Cloud prices on the day:

Variant	Price per hour	Capacity at SLO	Output tokens/s	GPU power	Cost per million output tokens
H100 SXM	$3.99	8.79 req/s	3,581	683 W	$0.31
H100 PCIe	$2.89	4.61 req/s	1,851	307 W	$0.43
H100 NVL	$3.19	3.91 req/s	1,560	309 W	$0.57

The SXM costs 25% more per hour than the NVL and serves 2.25× as much. Per token, the most expensive H100 per hour was the cheapest.

Step 1: 19 H100s, one health check each

We rented single-GPU H100 pods one after another and ran tokenwatt-check on each: a two-minute test that compares the GPU with a healthy card of the same model (power limit, clocks under a sustained matrix multiply, memory bandwidth, PCIe link). Each pod was deleted as soon as its check finished. We asked for H100s in every variant and both RunPod tiers; Community Cloud had no H100s available that day, so all 19 came from Secure Cloud.

Variant	Cards	Power limit (max)	SM clock under load	BF16 matmul	Memory copy
SXM	10	700 W (700 W)	1,319–1,369 MHz	668–688 TFLOPS	3.0 TB/s
PCIe	5	310 W (350 W)	797–900 MHz	352–398 TFLOPS	1.9 TB/s
NVL	4	310 W (400 W)	640–680 MHz	318–339 TFLOPS	3.5 TB/s

Three things stand out:

All ten SXM cards were healthy and almost identical, within 3% of each other and of our measured reference. No lemons in this draw.
Every NVL card had its power limit set to 310 W. NVIDIA's configurable range for the H100 NVL is 350–400 W, so this is below the lowest supported setting, and the cards report it as their default. Under a sustained load they ran at 640–680 MHz, about 38% of their maximum clock. All four rentals were set the same way, so this looks like a configuration applied across that fleet rather than one bad host. We can't see from inside a pod who chose it or why.
The PCIe cards, all at 310 W (within their 300–350 W range), split into two groups: three delivered about 398 TFLOPS and two about 352, a 12% gap at the same setting. That is the real lottery: same listing, same price, same power limit, different card.

Measured compute per dollar follows the same order: about $5.8 per PFLOPS-hour on SXM, $7.6 on PCIe and $9.6 on NVL.

Step 2: what each variant serves

Raw compute isn't the whole story for LLM serving. Generating tokens mostly streams the weights and the KV cache through memory, and the NVL has more memory bandwidth than the SXM (3.5 vs 3.0 TB/s in our copy test). So we served the same model on one card of each variant, under the same latency target (p99 time to first token ≤ 2 s, p99 time per output token ≤ 50 ms), with the same request mix, and searched for the highest request rate each could sustain.

The NVL's bandwidth didn't help: it reached 3.91 requests per second, the lowest of the three, with its p99 time per token at 44.8 ms, close to the limit. At 310 W the card can't feed its memory system; it used about a third of its theoretical bandwidth while serving, against about half on the SXM and the PCIe. Prompt processing (prefill) and attention need compute too, and that is exactly what the power limit takes away.

The SXM served 8.79 requests per second at 683 W. Per million output tokens: $0.31 on SXM, $0.43 on PCIe, $0.57 on NVL. For comparison, the same model on cheaper GPUs in our earlier test cost $0.90–1.61 per million tokens. The $4-an-hour card was the cheapest per token by a wide margin, again.

Is your GPU delivering what you pay for? Check a node in two minutes with our open-source checker (pip install tokenwatt-check), or have us evaluate your nodes.

What to do with this

Ask which H100 you're getting. "H100 80GB" usually means SXM or PCIe; the NVL shows up as 94 GB. The difference in what you get is much larger than the difference in price.
Check the power limit on the node, not the listing: nvidia-smi --query-gpu=name,power.limit,power.default_limit,power.max_limit --format=csv. Or run pip install tokenwatt-check && tokenwatt-check, which compares the card with a healthy one and now flags limits below NVIDIA's range even when the host reports them as the default.
Compare cost per token at your latency target, not price per hour. For a model that fits on one card, the SXM's extra compute paid for itself twice over here.
The NVL's extra memory still has a use: a 94 GB card holds models and contexts an 80 GB card can't. If you need that, check its power limit before you pay for it.

Limits

One platform (RunPod Secure Cloud), one day, prices on that day. The health checks cover 19 cards; the serving test used one card of each variant, in quick mode (2-minute windows, 2 repeats), which reads 15–20% higher in absolute capacity than our full mode, so compare the variants with each other rather than with full-mode numbers. One model (Qwen3-8B, BF16) that fits on all three cards. We report what the cards were set to, not why.

Full results, settings and raw data: H100 SXM, H100 PCIe and H100 NVL reports. Method: how we measure.

Renting an H100? The variant matters more than the luck of the draw

Step 1: 19 H100s, one health check each

Step 2: what each variant serves

What to do with this

Limits

Keep reading

H100 vs H200 for LLM inference: 18% more capacity, 33% higher price — and where the H200 pulls ahead

RTX 4090 vs L40S vs A100 vs A40 for LLM inference: what an 8B model really costs per token

RunPod vs Vast.ai vs Lambda: we benchmarked four rented H100s and found a 2.9× spread