Long-context capacity on one H100: ~8K input tokens, Qwen3-30B-A3B-Instruct-2507 FP8, vLLM 0.31.0
Quick-mode measurement on TokenWatt's production API node, one H100 SXM on RunPod Secure Cloud (November 2, 2026 UTC): Qwen3-30B-A3B-Instruct-2507 FP8 on vLLM 0.31.0 with FP8 KV cache, long-prompt workload with median 8,192 (4K–20K) input tokens and median 384 output tokens, random tokens so no prefix is reused. Capacity within p99 TTFT ≤ 2 s and p99 TPOT ≤ 50 ms is bound by prefill: first-token latency, not decode speed, limits how many long prompts run at once. Accuracy was checked on the same node in separate runs (GSM8K 96% quick, 94.5% full); this run skipped the accuracy gate.
Results
Each value is the median of 2 runs.
SLO goodput
Tokens per joule
Decode bandwidth utilization
Axis runs to 100% of rated bandwidth.
GPU board power
GPU board power only — excludes CPUs, memory, NICs, fans and PSU losses.
All numbers
Cards for equal goodput = 64 × reference goodput ÷ configuration goodput, rounded up.
| Metric | longctx-in8k |
|---|---|
| Engine build | vllm 0.31.0 · KV FP8 |
| Decode bandwidth utilization (MBU) | 50% |
| GPU board power | 0.51 kW |
| SLO goodput | 917 tok/s |
| Tokens per joule | 1.80 |
| TTFT p99 | 1,099 ms |
| TPOT p99 | 38.2 ms |
| SLO met | yes |
| Accuracy (none) | — |
| Accuracy Δ vs. reference | — |
| Cards for equal goodput | 64 |
Setup
Any change to these fields makes it a different report.
- Accelerator
- 1× NVIDIA H100 80GB HBM3
- Rated HBM BW
- 3.35 TB/s per GPU
- Driver / runtime
- 595.91.07
- Host
- RunPod Secure H100 SXM (TokenWatt API production node)
- Engine
- vllm 0.31.0
- Kernel libraries
- vllm/vllm-openai:v0.31.0 --max-model-len 32768 --enable-auto-tool-choice --tool-call-parser hermes
- Model
- Qwen3-30B-A3B-Instruct-2507-FP8 (MoE)
- Quantization
- FP8 weights, FP8 KV cache
- Parallelism
- 1 replica × TP1
- Workload trace
- rag-in8k
- Input tokens
- median 8,192 · p90 14,000
- Output tokens
- median 384 · p90 1,024
- Arrival
- Poisson, rate swept to SLO boundary
- Power source
- GPU board power sum (NVIDIA) · 1 Hz · 120 s steady state
- Runs per config
- 2 (variance band ±1.0%)
- Accuracy gate
- none · max drop 1.0 pts
Not covered
Do not extrapolate this report to the following.
- Whole-node power (no BMC access on this host)
- Context lengths beyond the chat-mix-v1 trace
- Quick mode: short windows, few repeats
- Accuracy: no accuracy gate was run
Reproduce
Run on the same hardware and software versions. Results should fall within ±1.0%.
tokenwatt run --spec live-qwen3-30b-a3b-longctx-in8k.spec.yaml --label 'longctx-in8k' --role other --out runs/longctx-in8k.json tokenwatt report runs/longctx-in8k.json --title 'Long-context capacity on one H100: ~8K input tokens, Qwen3-30B-A3B-Instruct-2507 FP8, vLLM 0.31.0'