Qwen3-8B BF16 on 1× AMD Instinct MI350X: vLLM's ROCm image vs AMD's image and settings
Quick-mode measurement on a RunPod Secure Cloud MI350X, 2026-11-13, $5.49 per hour on the day. Three software setups on the same host: vLLM 0.31.0's own ROCm image (vllm/vllm-openai-rocm:v0.31.0) with default settings; the same image with VLLM_ROCM_USE_AITER=1; and AMD's image (rocm/vllm:rocm10.1.0_ubuntu24.04_py3.14-pytorch_2.13.0_vllm-0.29.0, AMD's vLLM 0.29.1.dev build, which Docker Hub describes as a ROCm RC image) with the settings from AMD's vLLM optimization guide: VLLM_ROCM_USE_AITER=1, --attention-backend ROCM_AITER_FA, TORCH_BLAS_PREFER_HIPBLASLT=1. AMD's image removed the first-token tail (p99 382 ms vs 1,545 ms) and served 2.4× the tokens. Per million output tokens at $5.49/h: $0.66, $0.46, $0.28. The AMD-image run averaged 973 W against the 1,000 W limit. Write-up: /blog/mi350x-vs-h100-software-and-price.html
Results
Each value is the median of 2 runs.
SLO goodput
Tokens per joule
Decode bandwidth utilization
Axis runs to 100% of rated bandwidth.
GPU board power
GPU board power only — excludes CPUs, memory, NICs, fans and PSU losses.
All numbers
Cards for equal goodput = 64 × reference goodput ÷ configuration goodput, rounded up.
| Metric | vLLM ROCm image, defaults | vLLM ROCm image + AITER | AMD image + AMD settings |
|---|---|---|---|
| Engine build | vLLM 0.31.0 · KV BF16 | vLLM 0.31.0 · KV BF16 | vLLM 0.29.1.dev0+g98dff2a81.d20261002 · KV BF16 |
| Decode bandwidth utilization (MBU) | 27% | 24% | 28% |
| GPU board power | 0.82 kW | 0.85 kW | 0.97 kW |
| SLO goodput | 2,304 tok/s | 3,352 tok/s | 5,427 tok/s |
| Tokens per joule | 2.82 | 3.96 | 5.58 |
| TTFT p99 | 1,545 ms | 1,598 ms | 382 ms |
| TPOT p99 | 25.6 ms | 44.3 ms | 41.9 ms |
| SLO met | yes | yes | yes |
| Accuracy (gsm8k-5shot) | 92.0 | 92.0 | 94.0 |
| Accuracy Δ vs. reference | reference | 0.0 pts | +2.0 pts |
| Cards for equal goodput | 64 | 44 | 28 |
Setup
Any change to these fields makes it a different report.
- Accelerator
- 1× AMD Instinct MI350X
- Rated HBM BW
- 8.0 TB/s per GPU
- Driver / runtime
- 7.1.3.31500000 · ROCm 7.2.3
- Host
- RunPod Secure Cloud pod, one host for all runs (two pods in a row: vLLM's image, then AMD's image); $5.49 per hour on 2026-11-13
- Engine
- vLLM varies by configuration
- Kernel libraries
- vllm/vllm-openai-rocm:v0.31.0
- Model
- Qwen3-8B (Dense)
- Quantization
- BF16 weights, BF16 KV cache
- Parallelism
- 1 replica × TP1
- Workload trace
- chat-mix-v1
- Input tokens
- median 1,024 · p90 4,096
- Output tokens
- median 256 · p90 1,024
- Arrival
- Poisson, rate swept to SLO boundary
- Power source
- GPU board power sum (AMD) · 1 Hz · 120 s steady state
- Runs per config
- 2 (variance band ±1.8%)
- Accuracy gate
- gsm8k-5shot · max drop 1.0 pts
Not covered
Do not extrapolate this report to the following.
- Whole-node power (no BMC access on this host)
- Context lengths beyond the chat-mix-v1 trace
- Quick mode: short windows, few repeats — validation only, not for publishing
- Power: AMD reports the whole OAM package (amd-smi socket_power, 1,000 W limit), NVIDIA the board; the AMD adapter doesn't read throttle reasons
- Accuracy: quick mode scores GSM8K on 50 questions with one seed, so one question is 2 points and the 1-point gate can't tell a real drop from sampling noise. A −4 pt flag is two questions
- Engine versions differ by configuration: vLLM 0.31.0 (vLLM's image) vs AMD's 0.29.1.dev build; the AMD-image gain mixes the image, the attention backend and hipBLASLt, not measured separately
Reproduce
Run on the same hardware and software versions. Results should fall within ±1.8%.
Raw run files, logs and the exact launch commands: bench/results/amd-mi350x-2026-11/ in the TokenWatt repository (README.md). tokenwatt run --spec lambda-qwen3-8b-tp1-dp1.spec.yaml --label 'vLLM ROCm image, defaults' --role baseline --out runs/vllm-rocm-image-defaults.json tokenwatt run --spec lambda-qwen3-8b-tp1-dp1.spec.yaml --label 'vLLM ROCm image + AITER' --role other --out runs/vllm-rocm-image-aiter.json tokenwatt run --spec lambda-qwen3-8b-tp1-dp1.spec.yaml --label 'AMD image + AMD settings' --role tuned --out runs/amd-image-amd-settings.json tokenwatt report runs/vllm-rocm-image-defaults.json runs/vllm-rocm-image-aiter.json runs/amd-image-amd-settings.json --title 'Qwen3-8B BF16 on 1× AMD Instinct MI350X: vLLM's ROCm image vs AMD's image and settings'