← All reports

Report · 2026-11-18

Qwen3-Coder-30B-A3B FP8 on 1× AMD Instinct MI350X: vLLM's ROCm image vs AMD's image and settings

1× AMD Instinct MI350X Qwen3-Coder-30B-A3B-Instruct-FP8 FP8 weights, BF16 KV cache vLLM varies by configuration TTFT p99 ≤ 2.0 s · TPOT p99 ≤ 50 ms

Quick-mode measurement on a RunPod Secure Cloud MI350X, 2026-11-13, $5.49 per hour on the day. vLLM 0.31.0's own ROCm image with default settings (its log: "Using default MoE config" for this GPU) against AMD's image (rocm/vllm:rocm10.1.0_ubuntu24.04_py3.14-pytorch_2.13.0_vllm-0.29.0, a ROCm RC image per Docker Hub) with VLLM_ROCM_USE_AITER=1, --attention-backend ROCM_AITER_FA and TORCH_BLAS_PREFER_HIPBLASLT=1 from AMD's vLLM optimization guide; both with --enable-auto-tool-choice --tool-call-parser qwen3_coder. AMD's image served 3.4× the tokens ($0.86 → $0.25 per million output tokens at $5.49/h). Adding --max-num-batched-tokens 8192 changed nothing measurable (6,056 tok/s, raw data in the repo). Write-up: /blog/mi350x-vs-h100-software-and-price.html

ConclusionUnder SLO (TTFT p99 ≤ 2.0 s · TPOT p99 ≤ 50 ms), serving the same billable throughput as 64 cards on vLLM ROCm image, defaults takes: AMD image + AMD settings failed the accuracy gate (-4.0 pts) and is excluded from capacity math, despite 6,046 tok/s goodput.

Results

Each value is the median of 2 runs.

vLLM ROCm image, defaultsAMD image + AMD settings

SLO goodput

output tok/s per node · higher is better
vLLM ROCm image, defaults1,785vLLM ROCm image, defaults: 1,785 output tok/s per node
AMD image + AMD settings6,046AMD image + AMD settings: 6,046 output tok/s per node

Tokens per joule

tok/J, GPU board power · higher is better
vLLM ROCm image, defaults2.85vLLM ROCm image, defaults: 2.85 tok/J, GPU board power
AMD image + AMD settings6.91AMD image + AMD settings: 6.91 tok/J, GPU board power

Decode bandwidth utilization

% of rated HBM bandwidth
vLLM ROCm image, defaults28%vLLM ROCm image, defaults: 28 % of rated HBM bandwidth
AMD image + AMD settings27%AMD image + AMD settings: 27 % of rated HBM bandwidth

Axis runs to 100% of rated bandwidth.

GPU board power

kW, steady state
vLLM ROCm image, defaults0.63vLLM ROCm image, defaults: 0.63 kW, steady state
AMD image + AMD settings0.88AMD image + AMD settings: 0.88 kW, steady state

GPU board power only — excludes CPUs, memory, NICs, fans and PSU losses.

All numbers

Cards for equal goodput = 64 × reference goodput ÷ configuration goodput, rounded up.

MetricvLLM ROCm image, defaultsAMD image + AMD settings
Engine buildvLLM 0.31.0 · KV BF16vLLM 0.29.1.dev0+g98dff2a81.d20261002 · KV BF16
Decode bandwidth utilization (MBU)28%27%
GPU board power0.63 kW0.88 kW
SLO goodput1,785 tok/s6,046 tok/s
Tokens per joule2.856.91
TTFT p99972 ms358 ms
TPOT p9923.8 ms47.6 ms
SLO metyesyes
Accuracy (gsm8k-5shot)98.094.0
Accuracy Δ vs. referencereference-4.0 pts
Cards for equal goodput64—

Setup

Any change to these fields makes it a different report.

Accelerator
1× AMD Instinct MI350X
Rated HBM BW
8.0 TB/s per GPU
Driver / runtime
7.1.3.31500000 · ROCm 7.2.3
Host
RunPod Secure Cloud pod, one host for all runs (two pods in a row: vLLM's image, then AMD's image); $5.49 per hour on 2026-11-13
Engine
vLLM varies by configuration
Kernel libraries
vllm/vllm-openai-rocm:v0.31.0 --enable-auto-tool-choice --tool-call-parser qwen3_coder
Model
Qwen3-Coder-30B-A3B-Instruct-FP8 (MoE)
Quantization
FP8 weights, BF16 KV cache
Parallelism
1 replica × TP1
Workload trace
chat-mix-v1
Input tokens
median 1,024 · p90 4,096
Output tokens
median 256 · p90 1,024
Arrival
Poisson, rate swept to SLO boundary
Power source
GPU board power sum (AMD) · 1 Hz · 120 s steady state
Runs per config
2 (variance band ±2.5%)
Accuracy gate
gsm8k-5shot · max drop 1.0 pts

Not covered

Do not extrapolate this report to the following.

  • Whole-node power (no BMC access on this host)
  • Context lengths beyond the chat-mix-v1 trace
  • Quick mode: short windows, few repeats — validation only, not for publishing
  • Power: AMD reports the whole OAM package (amd-smi socket_power, 1,000 W limit), NVIDIA the board; the AMD adapter doesn't read throttle reasons
  • Accuracy: quick mode scores GSM8K on 50 questions with one seed, so one question is 2 points and the 1-point gate can't tell a real drop from sampling noise. A −4 pt flag is two questions
  • Engine versions differ by configuration: vLLM 0.31.0 (vLLM's image) vs AMD's 0.29.1.dev build; the AMD-image gain mixes the image, AITER, the attention backend and hipBLASLt, not measured separately

Reproduce

Run on the same hardware and software versions. Results should fall within ±2.5%.

shell
Raw run files, logs and the exact launch commands: bench/results/amd-mi350x-2026-11/ in the TokenWatt repository (README.md).

tokenwatt run --spec lambda-qwen3-coder-30b-a3b-instruct-fp8-tp1-dp1.spec.yaml --label 'vLLM ROCm image, defaults' --role baseline --out runs/vllm-rocm-image-defaults.json
tokenwatt run --spec lambda-qwen3-coder-30b-a3b-instruct-fp8-tp1-dp1.spec.yaml --label 'AMD image + AMD settings' --role tuned --out runs/amd-image-amd-settings.json

tokenwatt report runs/vllm-rocm-image-defaults.json runs/amd-image-amd-settings.json --title 'Qwen3-Coder-30B-A3B FP8 on 1× AMD Instinct MI350X: vLLM's ROCm image vs AMD's image and settings'

Download raw JSON