← All reports

Report · 2026-11-18

Qwen3-Coder-30B-A3B FP8 on 1× H100 SXM (Lium spot): vLLM defaults vs DeepGEMM

1× NVIDIA H100 80GB HBM3 Qwen3-Coder-30B-A3B-Instruct-FP8 FP8 weights, BF16 KV cache vLLM 0.33.0 TTFT p99 ≤ 2.0 s · TPOT p99 ≤ 50 ms

Quick-mode measurement on one Lium spot-tier H100 SXM (Atlanta), 2026-11-13/14, $1.50 per GPU-hour at rent time, vllm/vllm-openai:v0.33.0, both with --enable-auto-tool-choice --tool-call-parser qwen3_coder. VLLM_USE_DEEP_GEMM=1 (vLLM's Qwen3-Coder recipe for Hopper) switched the MoE layers to DeepGEMM and didn't move the capacity: 4,600 vs 4,610 tok/s at the same 11.31 req/s; failing steps were queueing at saturation. $0.09 per million output tokens at $1.50/h. Lium has given TokenWatt credits for earlier node evaluations and this run was paid from that balance; Lium had no input on it. Write-up: /blog/mi350x-vs-h100-software-and-price.html

ConclusionUnder SLO (TTFT p99 ≤ 2.0 s · TPOT p99 ≤ 50 ms), serving the same billable throughput as 64 cards on vLLM defaults takes: with DeepGEMM FP8 MoE, 64 → 65 cards and 3% less GPU board energy per token.

Results

Each value is the median of 2 runs.

vLLM defaultsDeepGEMM FP8 MoE

SLO goodput

output tok/s per node · higher is better
vLLM defaults4,610vLLM defaults: 4,610 output tok/s per node
DeepGEMM FP8 MoE4,600DeepGEMM FP8 MoE: 4,600 output tok/s per node

Tokens per joule

tok/J, GPU board power · higher is better
vLLM defaults7.62vLLM defaults: 7.62 tok/J, GPU board power
DeepGEMM FP8 MoE7.88DeepGEMM FP8 MoE: 7.88 tok/J, GPU board power

Decode bandwidth utilization

% of rated HBM bandwidth
vLLM defaults59%vLLM defaults: 59 % of rated HBM bandwidth
DeepGEMM FP8 MoE56%DeepGEMM FP8 MoE: 56 % of rated HBM bandwidth

Axis runs to 100% of rated bandwidth.

GPU board power

kW, steady state
vLLM defaults0.60vLLM defaults: 0.60 kW, steady state
DeepGEMM FP8 MoE0.58DeepGEMM FP8 MoE: 0.58 kW, steady state

GPU board power only — excludes CPUs, memory, NICs, fans and PSU losses.

All numbers

Cards for equal goodput = 64 × reference goodput ÷ configuration goodput, rounded up.

MetricvLLM defaultsDeepGEMM FP8 MoE
Engine buildvLLM 0.33.0 · KV BF16vLLM 0.33.0 · KV BF16
Decode bandwidth utilization (MBU)59%56%
GPU board power0.60 kW0.58 kW
SLO goodput4,610 tok/s4,600 tok/s
Tokens per joule7.627.88
TTFT p99332 ms288 ms
TPOT p9941.7 ms43.4 ms
SLO metyesyes
Accuracy (gsm8k-5shot)94.096.0
Accuracy Δ vs. referencereference+2.0 pts
Cards for equal goodput6465

Setup

Any change to these fields makes it a different report.

Accelerator
1× NVIDIA H100 80GB HBM3
Rated HBM BW
3.35 TB/s per GPU
Driver / runtime
580.105.08
Host
Lium spot-tier pod (interruptible marketplace host), node 00750dfa, Atlanta, US; $1.50 per GPU-hour on 2026-11-13
Engine
vLLM 0.33.0
Kernel libraries
vllm/vllm-openai:v0.33.0 --enable-auto-tool-choice --tool-call-parser qwen3_coder
Model
Qwen3-Coder-30B-A3B-Instruct-FP8 (MoE)
Quantization
FP8 weights, BF16 KV cache
Parallelism
1 replica × TP1
Workload trace
chat-mix-v1
Input tokens
median 1,024 · p90 4,096
Output tokens
median 256 · p90 1,024
Arrival
Poisson, rate swept to SLO boundary
Power source
GPU board power sum (NVIDIA) · 1 Hz · 120 s steady state
Runs per config
2 (variance band ±0.7%)
Accuracy gate
gsm8k-5shot · max drop 1.0 pts

Not covered

Do not extrapolate this report to the following.

  • Whole-node power (no BMC access on this host)
  • Context lengths beyond the chat-mix-v1 trace
  • Quick mode: short windows, few repeats — validation only, not for publishing
  • Accuracy: quick mode scores GSM8K on 50 questions with one seed, so one question is 2 points and the 1-point gate can't tell a real drop from sampling noise. A −4 pt flag is two questions
  • Spot (interruptible) node; it wasn't interrupted during the runs

Reproduce

Run on the same hardware and software versions. Results should fall within ±0.7%.

shell
Raw run files, logs and the exact launch commands: bench/results/h100-tuned-2026-11/ in the TokenWatt repository (README.md).

tokenwatt run --spec lambda-qwen3-coder-30b-a3b-instruct-fp8-tp1-dp1.spec.yaml --label 'vLLM defaults' --role baseline --out runs/vllm-defaults.json
tokenwatt run --spec lambda-qwen3-coder-30b-a3b-instruct-fp8-tp1-dp1.spec.yaml --label 'DeepGEMM FP8 MoE' --role tuned --out runs/deepgemm-fp8-moe.json

tokenwatt report runs/vllm-defaults.json runs/deepgemm-fp8-moe.json --title 'Qwen3-Coder-30B-A3B FP8 on 1× H100 SXM (Lium spot): vLLM defaults vs DeepGEMM'

Download raw JSON