Prefix caching on a hybrid model, one H100: Qwen3.8-27B FP8, vLLM 0.31.0, 4,000-token prompts
Quick-mode measurement on one H100 SXM on RunPod Secure Cloud (November 5, 2026): Qwen3.8-27B FP8, a hybrid model (48 Gated DeltaNet and 16 full-attention layers), on vLLM 0.31.0 with default automatic prefix caching (mamba cache mode 'align', 784-token cache blocks with the default BF16 KV cache), --max-num-seqs 512, text-only, 4,000-token prompts (outputs median 256 tokens). The same SLO search ran four times with 0%, 50%, 75% and 90% of every prompt drawn from eight shared prefixes; cache hit rates are vLLM's own prefix-cache counters (0.0%, 37.0%, 58.6%, 74.2%). Capacity within p99 TTFT ≤ 2 s and p99 TPOT ≤ 50 ms rose from 1.41 to 4.77 requests per second. No accuracy gate (random-token prompts); the model answered a smoke-test question correctly at each start.
Results
Each value is the median of 2 runs.
SLO goodput
Tokens per joule
Decode bandwidth utilization
Not measured in this report.
GPU board power
GPU board power only — excludes CPUs, memory, NICs, fans and PSU losses.
All numbers
Cards for equal goodput = 64 × reference goodput ÷ configuration goodput, rounded up.
| Metric | No prompt reuse (0% cache hits) | 50% shared prefix (37.0% cache hits) | 75% shared prefix (58.6% cache hits) | 90% shared prefix (74.2% cache hits) |
|---|---|---|---|---|
| Engine build | vllm 0.31.0 · KV BF16 | vllm 0.31.0 · KV BF16 | vllm 0.31.0 · KV BF16 | vllm 0.31.0 · KV BF16 |
| Decode bandwidth utilization (MBU) | — | — | — | — |
| GPU board power | 0.58 kW | 0.62 kW | 0.63 kW | 0.66 kW |
| SLO goodput | 616 tok/s | 1,025 tok/s | 1,412 tok/s | 1,970 tok/s |
| Tokens per joule | 1.06 | 1.66 | 2.23 | 2.98 |
| TTFT p99 | 1,089 ms | 797 ms | 602 ms | 410 ms |
| TPOT p99 | 42.0 ms | 42.9 ms | 45.4 ms | 44.7 ms |
| SLO met | yes | yes | yes | yes |
| Accuracy (none) | — | — | — | — |
| Accuracy Δ vs. reference | — | — | — | — |
| Cards for equal goodput | 64 | 39 | 28 | 21 |
Setup
Any change to these fields makes it a different report.
- Accelerator
- 1× NVIDIA H100 80GB HBM3
- Rated HBM BW
- 3.35 TB/s per GPU
- Driver / runtime
- 595.91.07
- Host
- RunPod Secure Cloud 1x H100 SXM 80GB (US-MO-1, pod h4dvtjc8j7mu7e)
- Engine
- vllm 0.31.0
- Kernel libraries
- vllm/vllm-openai:v0.31.0 --max-model-len 32768 --max-num-seqs 512 --language-model-only --enable-prompt-tokens-details
- Model
- Qwen3.8-27B-FP8 (?)
- Quantization
- FP8 weights, BF16 KV cache
- Parallelism
- 1 replica × TP1
- Workload trace
- in4k
- Input tokens
- fixed 4,000
- Output tokens
- median 256 · p90 1,024
- Arrival
- Poisson, rate swept to SLO boundary
- Power source
- GPU board power sum (NVIDIA) · 1 Hz · 120 s steady state
- Runs per config
- 2 (variance band ±4.3%)
- Accuracy gate
- none · max drop 0.5 pts
Not covered
Do not extrapolate this report to the following.
- Whole-node power (no BMC access on this host)
- Quick mode: short windows, few repeats
- Synthetic shared prefixes (8-prefix pool, random tokens)
- Accuracy: no accuracy gate was run
Reproduce
Run on the same hardware and software versions. Results should fall within ±4.3%.
tokenwatt run --spec h100-hyb-bf16-in4k-prefix-cache.spec.yaml --label 'hyb-bf16-in4k-0' --role other --out runs/hyb-bf16-in4k-0.json tokenwatt run --spec h100-hyb-bf16-in4k-prefix-cache.spec.yaml --label 'hyb-bf16-in4k-0.5' --role other --out runs/hyb-bf16-in4k-0-5.json tokenwatt run --spec h100-hyb-bf16-in4k-prefix-cache.spec.yaml --label 'hyb-bf16-in4k-0.75' --role other --out runs/hyb-bf16-in4k-0-75.json tokenwatt run --spec h100-hyb-bf16-in4k-prefix-cache.spec.yaml --label 'hyb-bf16-in4k-0.9' --role other --out runs/hyb-bf16-in4k-0-9.json tokenwatt report runs/hyb-bf16-in4k-0.json runs/hyb-bf16-in4k-0-5.json runs/hyb-bf16-in4k-0-75.json runs/hyb-bf16-in4k-0-9.json --title 'Prefix caching on a hybrid model, one H100: Qwen3.8-27B FP8, vLLM 0.31.0, 4,000-token prompts'