TokenWatt
Request an evaluation

← All reports

Report · 2026-10-22

Qwen3-8B BF16 on 1× RTX PRO 6000 Blackwell Server Edition with vLLM 0.31.0

1× NVIDIA RTX PRO 6000 Blackwell Server Edition Qwen3-8B BF16 weights, BF16 KV cache vLLM 0.31.0 TTFT p99 ≤ 2.0 s · TPOT p99 ≤ 50 ms

Quick-mode measurement on a RunPod Secure Cloud RTX PRO 6000 Blackwell Server Edition 96GB, 2026-10-22, default vLLM settings, $2.49 per hour on the day. Same model, workload and latency target as our RTX 4090, L40S, A100, A40 and H100 reports; the node ran tokenwatt-check first (600 W power limit, 408 TFLOPS BF16, 1.46 TB/s copy bandwidth). Energy is GPU board power. No rated memory bandwidth in our table yet, so MBU is not computed.

ConclusionSingle configuration measured under SLO (TTFT p99 ≤ 2.0 s · TPOT p99 ≤ 50 ms).

Results

Each value is the median of 2 runs.

rtxpro6000se-qwen3-8b

SLO goodput

output tok/s per node · higher is better
rtxpro6000se-qwen3-8b1,234rtxpro6000se-qwen3-8b: 1,234 output tok/s per node

Tokens per joule

tok/J, GPU board power · higher is better
rtxpro6000se-qwen3-8b2.80rtxpro6000se-qwen3-8b: 2.80 tok/J, GPU board power

Decode bandwidth utilization

% of rated HBM bandwidth

Not measured in this report.

GPU board power

kW, steady state
rtxpro6000se-qwen3-8b0.44rtxpro6000se-qwen3-8b: 0.44 kW, steady state

GPU board power only — excludes CPUs, memory, NICs, fans and PSU losses.

All numbers

Cards for equal goodput = 64 × reference goodput ÷ configuration goodput, rounded up.

Metricrtxpro6000se-qwen3-8b
Engine buildvLLM 0.31.0 · KV BF16
Decode bandwidth utilization (MBU)—
GPU board power0.44 kW
SLO goodput1,234 tok/s
Tokens per joule2.80
TTFT p99651 ms
TPOT p9944.6 ms
SLO metyes
Accuracy (gsm8k-5shot)94.0
Accuracy Δ vs. referencereference
Cards for equal goodput64

Setup

Any change to these fields makes it a different report.

Accelerator
1× NVIDIA RTX PRO 6000 Blackwell Server Edition
Rated HBM BW
—
Driver / runtime
595.91.07
Host
RunPod secure cloud pod,
Engine
vLLM 0.31.0
Kernel libraries
vllm/vllm-openai:v0.31.0
Model
Qwen3-8B (Dense)
Quantization
BF16 weights, BF16 KV cache
Parallelism
1 replica × TP1
Workload trace
chat-mix-v1
Input tokens
median 1,024 · p90 4,096
Output tokens
median 256 · p90 1,024
Arrival
Poisson, rate swept to SLO boundary
Power source
GPU board power sum (NVIDIA) · 1 Hz · 120 s steady state
Runs per config
2 (variance band ±1.3%)
Accuracy gate
gsm8k-5shot · max drop 1.0 pts

Not covered

Do not extrapolate this report to the following.

  • Whole-node power (no BMC access on this host)
  • Context lengths beyond the chat-mix-v1 trace
  • Quick mode: short windows, few repeats — validation only, not for publishing

Reproduce

Run on the same hardware and software versions. Results should fall within ±1.3%.

shell
tokenwatt run --spec lambda-qwen3-8b-tp1-dp1.spec.yaml --label 'rtxpro6000se-qwen3-8b' --role other --out runs/rtxpro6000se-qwen3-8b.json

tokenwatt report runs/rtxpro6000se-qwen3-8b.json --title 'Qwen3-8B BF16 on 1× RTX PRO 6000 Blackwell Server Edition with vLLM 0.31.0'

Download raw JSON