← All reports

Report · 2026-11-13

Prefix caching on a hybrid model, one H100: Qwen3.8-27B FP8, vLLM 0.31.0, 4,000-token prompts

1× NVIDIA H100 80GB HBM3 Qwen3.8-27B-FP8 FP8 weights, BF16 KV cache vllm 0.31.0 TTFT p99 ≤ 2.0 s · TPOT p99 ≤ 50 ms

Quick-mode measurement on one H100 SXM on RunPod Secure Cloud (November 5, 2026): Qwen3.8-27B FP8, a hybrid model (48 Gated DeltaNet and 16 full-attention layers), on vLLM 0.31.0 with default automatic prefix caching (mamba cache mode 'align', 784-token cache blocks with the default BF16 KV cache), --max-num-seqs 512, text-only, 4,000-token prompts (outputs median 256 tokens). The same SLO search ran four times with 0%, 50%, 75% and 90% of every prompt drawn from eight shared prefixes; cache hit rates are vLLM's own prefix-cache counters (0.0%, 37.0%, 58.6%, 74.2%). Capacity within p99 TTFT ≤ 2 s and p99 TPOT ≤ 50 ms rose from 1.41 to 4.77 requests per second. No accuracy gate (random-token prompts); the model answered a smoke-test question correctly at each start.

ConclusionUnder SLO (TTFT p99 ≤ 2.0 s · TPOT p99 ≤ 50 ms), serving the same billable throughput as 64 cards on No prompt reuse (0% cache hits) takes: with 50% shared prefix (37.0% cache hits), 64 → 39 cards and 36% less GPU board energy per token; with 75% shared prefix (58.6% cache hits), 64 → 28 cards and 52% less GPU board energy per token; with 90% shared prefix (74.2% cache hits), 64 → 21 cards and 64% less GPU board energy per token.

Results

Each value is the median of 2 runs.

No prompt reuse (0% cache hits)50% shared prefix (37.0% cache hits)75% shared prefix (58.6% cache hits)90% shared prefix (74.2% cache hits)

SLO goodput

output tok/s per node · higher is better
No prompt reuse (0% cache hits)616No prompt reuse (0% cache hits): 616 output tok/s per node
50% shared prefix (37.0% cache hits)1,02550% shared prefix (37.0% cache hits): 1,025 output tok/s per node
75% shared prefix (58.6% cache hits)1,41275% shared prefix (58.6% cache hits): 1,412 output tok/s per node
90% shared prefix (74.2% cache hits)1,97090% shared prefix (74.2% cache hits): 1,970 output tok/s per node

Tokens per joule

tok/J, GPU board power · higher is better
No prompt reuse (0% cache hits)1.06No prompt reuse (0% cache hits): 1.06 tok/J, GPU board power
50% shared prefix (37.0% cache hits)1.6650% shared prefix (37.0% cache hits): 1.66 tok/J, GPU board power
75% shared prefix (58.6% cache hits)2.2375% shared prefix (58.6% cache hits): 2.23 tok/J, GPU board power
90% shared prefix (74.2% cache hits)2.9890% shared prefix (74.2% cache hits): 2.98 tok/J, GPU board power

Decode bandwidth utilization

% of rated HBM bandwidth

Not measured in this report.

GPU board power

kW, steady state
No prompt reuse (0% cache hits)0.58No prompt reuse (0% cache hits): 0.58 kW, steady state
50% shared prefix (37.0% cache hits)0.6250% shared prefix (37.0% cache hits): 0.62 kW, steady state
75% shared prefix (58.6% cache hits)0.6375% shared prefix (58.6% cache hits): 0.63 kW, steady state
90% shared prefix (74.2% cache hits)0.6690% shared prefix (74.2% cache hits): 0.66 kW, steady state

GPU board power only — excludes CPUs, memory, NICs, fans and PSU losses.

All numbers

Cards for equal goodput = 64 × reference goodput ÷ configuration goodput, rounded up.

MetricNo prompt reuse (0% cache hits)50% shared prefix (37.0% cache hits)75% shared prefix (58.6% cache hits)90% shared prefix (74.2% cache hits)
Engine buildvllm 0.31.0 · KV BF16vllm 0.31.0 · KV BF16vllm 0.31.0 · KV BF16vllm 0.31.0 · KV BF16
Decode bandwidth utilization (MBU)————
GPU board power0.58 kW0.62 kW0.63 kW0.66 kW
SLO goodput616 tok/s1,025 tok/s1,412 tok/s1,970 tok/s
Tokens per joule1.061.662.232.98
TTFT p991,089 ms797 ms602 ms410 ms
TPOT p9942.0 ms42.9 ms45.4 ms44.7 ms
SLO metyesyesyesyes
Accuracy (none)————
Accuracy Δ vs. reference————
Cards for equal goodput64392821

Setup

Any change to these fields makes it a different report.

Accelerator
1× NVIDIA H100 80GB HBM3
Rated HBM BW
3.35 TB/s per GPU
Driver / runtime
595.91.07
Host
RunPod Secure Cloud 1x H100 SXM 80GB (US-MO-1, pod h4dvtjc8j7mu7e)
Engine
vllm 0.31.0
Kernel libraries
vllm/vllm-openai:v0.31.0 --max-model-len 32768 --max-num-seqs 512 --language-model-only --enable-prompt-tokens-details
Model
Qwen3.8-27B-FP8 (?)
Quantization
FP8 weights, BF16 KV cache
Parallelism
1 replica × TP1
Workload trace
in4k
Input tokens
fixed 4,000
Output tokens
median 256 · p90 1,024
Arrival
Poisson, rate swept to SLO boundary
Power source
GPU board power sum (NVIDIA) · 1 Hz · 120 s steady state
Runs per config
2 (variance band ±4.3%)
Accuracy gate
none · max drop 0.5 pts

Not covered

Do not extrapolate this report to the following.

  • Whole-node power (no BMC access on this host)
  • Quick mode: short windows, few repeats
  • Synthetic shared prefixes (8-prefix pool, random tokens)
  • Accuracy: no accuracy gate was run

Reproduce

Run on the same hardware and software versions. Results should fall within ±4.3%.

shell
tokenwatt run --spec h100-hyb-bf16-in4k-prefix-cache.spec.yaml --label 'hyb-bf16-in4k-0' --role other --out runs/hyb-bf16-in4k-0.json
tokenwatt run --spec h100-hyb-bf16-in4k-prefix-cache.spec.yaml --label 'hyb-bf16-in4k-0.5' --role other --out runs/hyb-bf16-in4k-0-5.json
tokenwatt run --spec h100-hyb-bf16-in4k-prefix-cache.spec.yaml --label 'hyb-bf16-in4k-0.75' --role other --out runs/hyb-bf16-in4k-0-75.json
tokenwatt run --spec h100-hyb-bf16-in4k-prefix-cache.spec.yaml --label 'hyb-bf16-in4k-0.9' --role other --out runs/hyb-bf16-in4k-0-9.json

tokenwatt report runs/hyb-bf16-in4k-0.json runs/hyb-bf16-in4k-0-5.json runs/hyb-bf16-in4k-0-75.json runs/hyb-bf16-in4k-0-9.json --title 'Prefix caching on a hybrid model, one H100: Qwen3.8-27B FP8, vLLM 0.31.0, 4,000-token prompts'

Download raw JSON