Qwen3-8B BF16 on 1× H100 SXM 80GB with vLLM 0.31.0
Quick-mode measurement on a RunPod Secure Cloud H100 SXM 80GB, 2026-10-13, default vLLM settings, $3.99 per hour on the day. Part of our comparison of the three H100 variants serving the same 8B model; the node ran tokenwatt-check first. Energy is GPU board power.
ConclusionSingle configuration measured under SLO (TTFT p99 ≤ 2.0 s · TPOT p99 ≤ 50 ms).
Results
Each value is the median of 2 runs.
vLLM default
SLO goodput
output tok/s per node · higher is better
Tokens per joule
tok/J, GPU board power · higher is better
Decode bandwidth utilization
% of rated HBM bandwidth
Axis runs to 100% of rated bandwidth.
GPU board power
kW, steady state
GPU board power only — excludes CPUs, memory, NICs, fans and PSU losses.
All numbers
Cards for equal goodput = 64 × reference goodput ÷ configuration goodput, rounded up.
| Metric | vLLM default |
|---|---|
| Engine build | vLLM 0.31.0 · KV BF16 |
| Decode bandwidth utilization (MBU) | 51% |
| GPU board power | 0.68 kW |
| SLO goodput | 3,582 tok/s |
| Tokens per joule | 5.24 |
| TTFT p99 | 445 ms |
| TPOT p99 | 37.9 ms |
| SLO met | yes |
| Accuracy (gsm8k-5shot) | 92.0 |
| Accuracy Δ vs. reference | reference |
| Cards for equal goodput | 64 |
Setup
Any change to these fields makes it a different report.
- Accelerator
- 1× NVIDIA H100 80GB HBM3
- Rated HBM BW
- 3.35 TB/s per GPU
- Driver / runtime
- 580.126.09
- Host
- RunPod secure cloud pod,
- Engine
- vLLM 0.31.0
- Kernel libraries
- vllm/vllm-openai:v0.31.0
- Model
- Qwen3-8B (Dense)
- Quantization
- BF16 weights, BF16 KV cache
- Parallelism
- 1 replica × TP1
- Workload trace
- chat-mix-v1
- Input tokens
- median 1,024 · p90 4,096
- Output tokens
- median 256 · p90 1,024
- Arrival
- Poisson, rate swept to SLO boundary
- Power source
- GPU board power sum (NVIDIA) · 1 Hz · 120 s steady state
- Runs per config
- 2 (variance band ±0.4%)
- Accuracy gate
- gsm8k-5shot · max drop 1.0 pts
Not covered
Do not extrapolate this report to the following.
- Whole-node power (no BMC access on this host)
- Context lengths beyond the chat-mix-v1 trace
- Quick mode: short windows, few repeats — validation only, not for publishing
Reproduce
Run on the same hardware and software versions. Results should fall within ±0.4%.
tokenwatt run --spec lambda-qwen3-8b-tp1-dp1.spec.yaml --label 'h100-sxm-qwen3-8b' --role other --out runs/h100-sxm-qwen3-8b.json tokenwatt report runs/h100-sxm-qwen3-8b.json --title 'Qwen3-8B BF16 on 1× H100 SXM 80GB with vLLM 0.31.0'