← Lab notes

MI350X vs H100: AMD's own software stack more than doubled the MI350X; tuning the H100 changed nothing

We served Qwen3-8B and Qwen3-Coder-30B on one AMD Instinct MI350X and one H100 SXM under the same latency target. On vLLM's default ROCm build the MI350X served 36% fewer tokens than the H100; on AMD's own image with AITER kernels it served 52% more on Qwen3-8B and 31% more on Qwen3-Coder-30B. vLLM's recommended H100 settings changed nothing. Per million tokens, the cheapest result came from a $1.50-an-hour spot H100.

Choosing GPUs? Compare GPUs for your traffic · Get your node measured, free

On vLLM's default ROCm build, an AMD Instinct MI350X served 36% fewer tokens than an H100 SXM. On AMD's own vLLM image, the same card served 52% more. We changed the software, not the hardware.

We served the same two models on one MI350X and one H100 SXM, under the same latency target (p99 time to first token ≤ 2 s, p99 time per output token ≤ 50 ms) and the same request mix, and found the highest request rate each could sustain. Then we tried each vendor's recommended settings.

Qwen3-8B (BF16), one GPU, output tokens per second at the latency target:

GPU and software Tokens/s p99 first token Price per hour Cost per million output tokens
MI350X, vLLM 0.31.0 ROCm build, defaults 2,304 1,545 ms $5.49 $0.66
MI350X, same build, AITER kernels on 3,352 1,598 ms $5.49 $0.46
MI350X, AMD's vLLM image, AMD's recommended settings 5,427 382 ms $5.49 $0.28
H100 SXM, vLLM defaults 3,581 445 ms $3.99 $0.31
H100 SXM, vLLM's recommended setting 3,577 407 ms $1.50 (spot) $0.12

Qwen3-Coder-30B-A3B (FP8):

GPU and software Tokens/s p99 first token Price per hour Cost per million output tokens
MI350X, vLLM 0.31.0 ROCm build, defaults 1,785 972 ms $5.49 $0.86
MI350X, AMD's vLLM image, AMD's recommended settings 6,046 358 ms $5.49 $0.25
H100 SXM, vLLM defaults 4,610 332 ms $1.50 (spot) $0.09
H100 SXM, vLLM's recipe setting (DeepGEMM) 4,600 288 ms $1.50 (spot) $0.09

Three things stand out.

1. On the MI350X, the software was worth 2.4–3.4×

The default ROCm build of vLLM 0.31.0 ran the MI350X with a generic attention backend and, for the FP8 coder model, without tuned kernels for this GPU (its log says so: "Using default MoE config"). The card generated tokens quickly — p99 time per token was 24–26 ms, faster than the H100 — but the slowest first tokens took 1–1.6 s, and one step more load pushed them past 2 s. The tail of the queue, not the GPU's speed, set the limit.

Turning on AMD's AITER kernels in the same build added 45% on Qwen3-8B. Switching to AMD's own vLLM image with the settings from AMD's optimization guide — AITER, the AITER attention backend and hipBLASLt — removed the tail: p99 first token fell to 358–382 ms, and capacity rose to 13.3 requests per second on Qwen3-8B (5.8 on the default build) and 14.7 on Qwen3-Coder-30B (4.5): 2.4× and 3.4× the default build's tokens per second.

If you rent an MI350X and run a generic image, you are measuring the image.

We applied what vLLM recommends for Hopper — DeepGEMM for the FP8 mixture-of-experts layers (from vLLM's Qwen3-Coder recipe) and a larger prefill batch for the small model (from vLLM's optimization guide) — on the same node as the default runs. Capacity didn't move: 4,600 vs 4,610 tokens/s on the coder model, 3,577 vs 3,576 on Qwen3-8B. The coder model was limited by queueing at saturation, the 8B model by time per token, and neither depends on which matrix-multiply kernel runs. The H100 software stack is already close to its limit for these models.

Best against best, the MI350X served 1.52× the H100's tokens on Qwen3-8B and 1.31× on Qwen3-Coder-30B, at higher power: 864–973 W for the MI350X package (1,000 W limit) against 584–683 W for the H100 board (700 W limit). Tokens per joule ended up close: 5.6 vs 5.2–5.3 on Qwen3-8B, 6.9–7.0 vs 7.5–7.9 on the coder model.

3. Price per hour decided who was cheapest per token

The MI350X cost $5.49 an hour on RunPod Secure Cloud. The H100 SXM cost $3.99 there on October 13 and $4.39 on November 13, and $1.50 as a spot rental on Lium on November 13. With the measured throughput fixed, the price decides the ranking:

Price basis H100 SXM, Qwen3-8B H100 SXM, Coder-30B MI350X (AMD stack), Qwen3-8B MI350X (AMD stack), Coder-30B
Lium spot H100, $1.50/h $0.12 $0.09 — —
RunPod Secure H100, $3.99/h $0.31 $0.24 — —
RunPod Secure H100, $4.39/h $0.34 $0.26 — —
RunPod Secure MI350X, $5.49/h — — $0.28 $0.25
MI350X at $3.00/h (a 1-GPU list price on our price page) — — $0.15* $0.14*

* Computed from the throughput we measured on RunPod at another provider's list price; we haven't measured on that provider.

At on-demand prices from the same cloud, the tuned MI350X was cheaper per token than the H100 SXM on Qwen3-8B and about even on the coder model. A spot H100 at $1.50 an hour beat everything by more than 2×. The spread in price for the same H100 (2.9×) was bigger than any gap between the two chips.

What to do with this

  • On AMD GPUs, use AMD's image and settings. The default build left more than half of the MI350X's capacity unused in our tests. AMD's vLLM optimization guide lists the settings we used.
  • Don't expect much from re-tuning an H100 for these models. The defaults were already at the limit.
  • Compare cost per token at your latency target, then shop the price. The chip choice moved cost per token by up to 1.5×; where we rented moved it by up to 3×.
  • Watch the first-token tail, not just tokens per second. The default MI350X build looked fast per token and still failed the latency target early.

Limits

One card of each type, quick mode (2-minute windows, 2 repeats), one day each, prices on the day. The software differs by necessity: AMD's image ships its own vLLM build (0.29.1.dev) against 0.31.0 and 0.33.0 on the H100, and Docker Hub describes AMD's rocm/vllm images as release candidates; vLLM's own ROCm image is the official build. The AMD-image gain mixes the image, the attention backend and hipBLASLt; we didn't measure them separately. AMD reports power for the whole MI350X package and NVIDIA for the H100 board; neither includes the host. We tried only the settings each vendor recommends for these models, not an exhaustive search. GSM8K accuracy (50 questions) stayed at 90–98% in every run; one question is 2 points, so the report pages' 1-point accuracy gate flags two configurations for a two-question drop that this sample can't tell from noise. The Lium node was a spot (interruptible) rental; it wasn't interrupted. Disclosure: Lium has given TokenWatt credits for earlier node evaluations, and this run was paid from that balance; Lium had no input on this post. We'll add an MI300X when we can rent one.

Settings and data per configuration: MI350X Qwen3-8B and Qwen3-Coder-30B; H100 SXM Qwen3-8B and Qwen3-Coder-30B. The H100 defaults at $3.99 come from our October H100 SXM report. Method: how we measure.

Keep reading

3 min read

H100 vs A100 for LLM inference: 7.8× the capacity for 2.2× the price

We served Qwen3-8B with vLLM on a rented H100 SXM and a rented A100 SXM 80GB under the same latency target. The H100 served 8.79 requests per second against 1.13 on the A100: 7.8× the capacity for 2.2× the hourly price, so each token cost 71% less ($0.31 against $1.06 per million output tokens). On raw specs the H100 is only 1.7–2.6× faster. The rest of the gap comes from slow tokens on the A100, which a serving setting can partly fix.

H100A100LLM inference
4 min read

Renting an H100? The variant matters more than the luck of the draw

We rented 19 H100s on RunPod Secure Cloud and checked each one, then served the same 8B model on one of each variant. All 10 SXM cards were healthy and nearly identical. Every NVL card was set to 310 W, below NVIDIA's 350–400 W range, and the PCIe cards split into two speed groups 12% apart. Per token, the SXM was the cheapest: $0.31 per million output tokens, against $0.43 on PCIe and $0.57 on NVL.

H100H100 NVLH100 PCIe
6 min read

H100 vs H200 for LLM inference: 18% more capacity, 33% higher price — and where the H200 pulls ahead

We ran the same model, engine and latency target on an H100 SXM and an H200 on RunPod. At a 50 ms-per-token SLO the H200 served 18% more requests for 33% more money, so each token cost 11% more. At a fixed load it generated tokens 36% faster, and with a 30 ms target its lead grew to 29%, close to breaking even on price. Measured numbers, cost per million tokens and the break-even price.

H100H200LLM inference