← All reports

Report · 2026-11-18

Qwen3-8B BF16 on 1× AMD Instinct MI350X: vLLM's ROCm image vs AMD's image and settings

1× AMD Instinct MI350X Qwen3-8B BF16 weights, BF16 KV cache vLLM varies by configuration TTFT p99 ≤ 2.0 s · TPOT p99 ≤ 50 ms

Quick-mode measurement on a RunPod Secure Cloud MI350X, 2026-11-13, $5.49 per hour on the day. Three software setups on the same host: vLLM 0.31.0's own ROCm image (vllm/vllm-openai-rocm:v0.31.0) with default settings; the same image with VLLM_ROCM_USE_AITER=1; and AMD's image (rocm/vllm:rocm10.1.0_ubuntu24.04_py3.14-pytorch_2.13.0_vllm-0.29.0, AMD's vLLM 0.29.1.dev build, which Docker Hub describes as a ROCm RC image) with the settings from AMD's vLLM optimization guide: VLLM_ROCM_USE_AITER=1, --attention-backend ROCM_AITER_FA, TORCH_BLAS_PREFER_HIPBLASLT=1. AMD's image removed the first-token tail (p99 382 ms vs 1,545 ms) and served 2.4× the tokens. Per million output tokens at $5.49/h: $0.66, $0.46, $0.28. The AMD-image run averaged 973 W against the 1,000 W limit. Write-up: /blog/mi350x-vs-h100-software-and-price.html

ConclusionUnder SLO (TTFT p99 ≤ 2.0 s · TPOT p99 ≤ 50 ms), serving the same billable throughput as 64 cards on vLLM ROCm image, defaults takes: with vLLM ROCm image + AITER, 64 → 44 cards and 29% less GPU board energy per token; with AMD image + AMD settings, 64 → 28 cards and 49% less GPU board energy per token.

Results

Each value is the median of 2 runs.

vLLM ROCm image, defaultsvLLM ROCm image + AITERAMD image + AMD settings

SLO goodput

output tok/s per node · higher is better
vLLM ROCm image, defaults2,304vLLM ROCm image, defaults: 2,304 output tok/s per node
vLLM ROCm image + AITER3,352vLLM ROCm image + AITER: 3,352 output tok/s per node
AMD image + AMD settings5,427AMD image + AMD settings: 5,427 output tok/s per node

Tokens per joule

tok/J, GPU board power · higher is better
vLLM ROCm image, defaults2.82vLLM ROCm image, defaults: 2.82 tok/J, GPU board power
vLLM ROCm image + AITER3.96vLLM ROCm image + AITER: 3.96 tok/J, GPU board power
AMD image + AMD settings5.58AMD image + AMD settings: 5.58 tok/J, GPU board power

Decode bandwidth utilization

% of rated HBM bandwidth
vLLM ROCm image, defaults27%vLLM ROCm image, defaults: 27 % of rated HBM bandwidth
vLLM ROCm image + AITER24%vLLM ROCm image + AITER: 24 % of rated HBM bandwidth
AMD image + AMD settings28%AMD image + AMD settings: 28 % of rated HBM bandwidth

Axis runs to 100% of rated bandwidth.

GPU board power

kW, steady state
vLLM ROCm image, defaults0.82vLLM ROCm image, defaults: 0.82 kW, steady state
vLLM ROCm image + AITER0.85vLLM ROCm image + AITER: 0.85 kW, steady state
AMD image + AMD settings0.97AMD image + AMD settings: 0.97 kW, steady state

GPU board power only — excludes CPUs, memory, NICs, fans and PSU losses.

All numbers

Cards for equal goodput = 64 × reference goodput ÷ configuration goodput, rounded up.

MetricvLLM ROCm image, defaultsvLLM ROCm image + AITERAMD image + AMD settings
Engine buildvLLM 0.31.0 · KV BF16vLLM 0.31.0 · KV BF16vLLM 0.29.1.dev0+g98dff2a81.d20261002 · KV BF16
Decode bandwidth utilization (MBU)27%24%28%
GPU board power0.82 kW0.85 kW0.97 kW
SLO goodput2,304 tok/s3,352 tok/s5,427 tok/s
Tokens per joule2.823.965.58
TTFT p991,545 ms1,598 ms382 ms
TPOT p9925.6 ms44.3 ms41.9 ms
SLO metyesyesyes
Accuracy (gsm8k-5shot)92.092.094.0
Accuracy Δ vs. referencereference0.0 pts+2.0 pts
Cards for equal goodput644428

Setup

Any change to these fields makes it a different report.

Accelerator
1× AMD Instinct MI350X
Rated HBM BW
8.0 TB/s per GPU
Driver / runtime
7.1.3.31500000 · ROCm 7.2.3
Host
RunPod Secure Cloud pod, one host for all runs (two pods in a row: vLLM's image, then AMD's image); $5.49 per hour on 2026-11-13
Engine
vLLM varies by configuration
Kernel libraries
vllm/vllm-openai-rocm:v0.31.0
Model
Qwen3-8B (Dense)
Quantization
BF16 weights, BF16 KV cache
Parallelism
1 replica × TP1
Workload trace
chat-mix-v1
Input tokens
median 1,024 · p90 4,096
Output tokens
median 256 · p90 1,024
Arrival
Poisson, rate swept to SLO boundary
Power source
GPU board power sum (AMD) · 1 Hz · 120 s steady state
Runs per config
2 (variance band ±1.8%)
Accuracy gate
gsm8k-5shot · max drop 1.0 pts

Not covered

Do not extrapolate this report to the following.

  • Whole-node power (no BMC access on this host)
  • Context lengths beyond the chat-mix-v1 trace
  • Quick mode: short windows, few repeats — validation only, not for publishing
  • Power: AMD reports the whole OAM package (amd-smi socket_power, 1,000 W limit), NVIDIA the board; the AMD adapter doesn't read throttle reasons
  • Accuracy: quick mode scores GSM8K on 50 questions with one seed, so one question is 2 points and the 1-point gate can't tell a real drop from sampling noise. A −4 pt flag is two questions
  • Engine versions differ by configuration: vLLM 0.31.0 (vLLM's image) vs AMD's 0.29.1.dev build; the AMD-image gain mixes the image, the attention backend and hipBLASLt, not measured separately

Reproduce

Run on the same hardware and software versions. Results should fall within ±1.8%.

shell
Raw run files, logs and the exact launch commands: bench/results/amd-mi350x-2026-11/ in the TokenWatt repository (README.md).

tokenwatt run --spec lambda-qwen3-8b-tp1-dp1.spec.yaml --label 'vLLM ROCm image, defaults' --role baseline --out runs/vllm-rocm-image-defaults.json
tokenwatt run --spec lambda-qwen3-8b-tp1-dp1.spec.yaml --label 'vLLM ROCm image + AITER' --role other --out runs/vllm-rocm-image-aiter.json
tokenwatt run --spec lambda-qwen3-8b-tp1-dp1.spec.yaml --label 'AMD image + AMD settings' --role tuned --out runs/amd-image-amd-settings.json

tokenwatt report runs/vllm-rocm-image-defaults.json runs/vllm-rocm-image-aiter.json runs/amd-image-amd-settings.json --title 'Qwen3-8B BF16 on 1× AMD Instinct MI350X: vLLM's ROCm image vs AMD's image and settings'

Download raw JSON