DeepSeek-V4.1-Flash on 4× H200 with vLLM 0.32.0 (TP4, FP8 KV cache)
Quick-mode measurement on a RunPod Secure Cloud node with 4× H200 141 GB (October 27, 2026), following vLLM's published DeepSeek-V4.1-Flash recipe for H200: tensor parallel 4, FP8 KV cache, Engram tables offloaded to host memory, otherwise default settings. Chat-like workload with random-token prompts (median 1,024 input, 256 output tokens). Within p99 TTFT ≤ 2 s and p99 TPOT ≤ 50 ms the node served 5.09 requests/s: 2,047 output and about 8,180 input tokens per second, 90% on the GSM8K check. One node, quick mode (2-minute windows, 2 repeats), no prompt-cache hits.
Results
Each value is the median of 2 runs.
SLO goodput
Tokens per joule
Decode bandwidth utilization
Axis runs to 100% of rated bandwidth.
GPU board power
GPU board power only — excludes CPUs, memory, NICs, fans and PSU losses.
All numbers
Cards for equal goodput = 64 × reference goodput ÷ configuration goodput, rounded up.
| Metric | DeepSeek-V4.1-Flash · 4× H200 · TP4 · vLLM 0.32.0 |
|---|---|
| Engine build | vLLM 0.32.0 · KV FP8 |
| Decode bandwidth utilization (MBU) | 61% |
| GPU board power | 2.38 kW |
| SLO goodput | 2,047 tok/s |
| Tokens per joule | 0.86 |
| TTFT p99 | 379 ms |
| TPOT p99 | 45.3 ms |
| SLO met | yes |
| Accuracy (gsm8k-5shot) | 90.0 |
| Accuracy Δ vs. reference | reference |
| Cards for equal goodput | 64 |
Setup
Any change to these fields makes it a different report.
- Accelerator
- 4× NVIDIA H200
- Rated HBM BW
- 4.8 TB/s per GPU
- Driver / runtime
- 595.91.07
- Host
- RunPod secure cloud pod,
- Engine
- vLLM 0.32.0
- Kernel libraries
- vllm/vllm-openai:v0.32.0 --kv-cache-dtype fp8 --tokenizer-mode deepseek_v41 --engram-config {"cpu_offload":true} --gpu-memory-utilization 0.95
- Model
- DeepSeek-V4.1-Flash (MoE)
- Quantization
- FP8 weights, FP8 KV cache
- Parallelism
- 1 replica × TP4
- Workload trace
- chat-mix-v1
- Input tokens
- median 1,024 · p90 4,096
- Output tokens
- median 256 · p90 1,024
- Arrival
- Poisson, rate swept to SLO boundary
- Power source
- GPU board power sum (NVIDIA) · 1 Hz · 120 s steady state
- Runs per config
- 2 (variance band ±1.7%)
- Accuracy gate
- gsm8k-5shot · max drop 1.0 pts
Not covered
Do not extrapolate this report to the following.
- Whole-node power (no BMC access on this host)
- Context lengths beyond the chat-mix-v1 trace
- Quick mode: short windows, few repeats — validation only, not for publishing
Reproduce
Run on the same hardware and software versions. Results should fall within ±1.7%.
tokenwatt run --spec lambda-deepseek-v4.1-flash-tp4-dp1.spec.yaml --label 'dsv41flash-h200x4-tp4dp1-tpot50' --role baseline --out runs/dsv41flash-h200x4-tp4dp1-tpot50.json tokenwatt report runs/dsv41flash-h200x4-tp4dp1-tpot50.json --title 'DeepSeek-V4.1-Flash on 4× H200 with vLLM 0.32.0 (TP4, FP8 KV cache)'