Methodology · v0.9

How TokenWatt measures.

Definitions first, optimizations second. If two engineers can't get the same number from the same setup, it isn't a measurement.

Principles

  • Measure what gets billed. Only tokens served within the latency SLO count. Peak throughput at unbounded latency is not reported.
  • Same ruler for every vendor. AMD Instinct and NVIDIA systems use identical definitions, traces and power sources.
  • Lock everything up front. SLO, workload, accuracy suite and pass threshold are written into the spec file before the first run.
  • Measured watts only. Energy comes from the BMC or a metered PDU, never from nameplate TDP.
  • State what isn't covered. Every report lists the context lengths, quantizations and concurrency levels it did not test.

The spec file

A report is defined by its spec file. Changing any field — hardware, driver, engine version, model, quantization, trace, SLO or eval set — produces a different report, not an update to the old one.

spec.yaml · preview
hardware:   { accelerator: "<model>", count: 8, driver: "<version>" }
engine:     { name: vllm, version: "<version>", endpoint: http://localhost:8000/v1 }
model:      { name: "<model>", quantization: fp8, kv_cache: fp8 }
slo:        { ttft_p99_ms: 2000, tpot_p99_ms: 50 }
workload:   { trace: chat-mix-v1, warmup_s: 120, window_s: 600 }
accuracy:   { suite: gsm8k-500, max_drop_pts: 0.5, seeds: 3 }
power:      { source: redfish, sampling_hz: 1 }
runs: 5

SLO goodput

Bench replays the workload trace with Poisson arrivals and sweeps the request rate upward. At each rate it records p99 time-to-first-token (TTFT) and p99 time-per-output-token (TPOT) over the steady-state window.

Definitiongoodput = output tok/s at the highest request rate where
  TTFTp99 ≤ SLOTTFT  and  TPOTp99 ≤ SLOTPOT

Goodput is reported per node. Requests that fail or time out count against the SLO, not as missing data.

Decode bandwidth utilization (MBU)

During decode, each step reads the active weights and the KV cache from HBM. MBU compares the bytes the model must read with the bandwidth the hardware is rated for. It is derived from model geometry and measured step rate — not from hardware counters — so the same definition applies on every vendor.

DefinitionMBU = (Bweights,active + BKV,read) × steps/s ÷ (NGPU × BWrated)
  • For MoE models, Bweights,active counts only the experts actually routed to in each step, averaged over the window.
  • Redundant reads (unfused dequantization, re-reads across kernels) do not raise MBU. Hardware counters are collected as a diagnostic, never as the headline.
  • MBU is a diagnostic of software headroom. At large batch sizes decode can become compute- or interconnect-bound, so goodput and tokens per joule remain the primary results.

Power and energy

Node power is read from the BMC (Redfish or IPMI DCMI) or a metered PDU at 1 Hz or faster and averaged over the same steady-state window as goodput. GPU board power from the vendor's management library is reported separately and always labeled as such.

Definitiontokens per joule = goodput ÷ mean node power
No TDP estimates. If a measured node-power source isn't available, the report says so and publishes board power only. It never multiplies nameplate TDP by time.

Accuracy gate

Every configuration runs the same evaluation suite with fixed prompts, sampling parameters and seeds. The pass threshold is set in the spec before measurement.

  • The reference configuration (usually production) defines the baseline score.
  • A configuration fails the gate if its score drops by more than the threshold. Failed configurations are shown, but excluded from capacity math.
  • Suites are sized so that seed-to-seed variation is well inside the threshold; the observed spread is published with the report.

Comparisons

A typical report has three columns, measured on the same node in the same session:

ColumnWhat it isWhy it's there
ProductionThe image and flags running in production todayYour real starting point
Vendor latestThe hardware vendor's latest recommended image and settingsWhat you get for free by upgrading
CandidateAny change under evaluation — including TokenWatt TunedThe increment above the vendor's best

Cross-vendor reports follow the same rule: each system runs its own vendor's recommended configuration for the same model, quantization class and SLO.

Capacity math

The conclusion of every report is a capacity statement. For a fleet of N cards serving at the reference goodput T0, a configuration with goodput T1 needs:

Cards for equal billable throughputN′ = ⌈ N × T0 ÷ T1 ⌉

A 1.4× goodput gain means roughly 29% fewer cards for the same work. Savings only count where they're real — avoided purchases or capacity that can be sold. Idle cards are never counted as savings.

Reproducibility

  • Each configuration is measured in at least five runs; the report shows medians and the observed variance band.
  • Warm-up is excluded. Clocks, temperatures and throttle events are logged and attached to the raw data.
  • Every public report includes the spec file, raw JSON and the exact command used. Another engineer on the same setup should land within the published band.

Scope and limits

A report covers exactly what was measured. In particular, it does not cover untested context lengths, quantization formats, concurrency regions, multi-node topologies or workload traces. These are listed explicitly in every report.

Definitions: TTFT, TPOT, goodput, tokens per joule, MBU

What is time to first token (TTFT)?

Time to first token is the delay between sending a request to an LLM server and receiving the first generated token. It is dominated by queueing and prefill (processing the prompt). TokenWatt reports the 99th percentile (p99) TTFT at each request rate, because the slowest requests are what users notice.

What is TPOT (time per output token)?

Time per output token is the average gap between generated tokens after the first one — the speed at which an answer streams. TokenWatt reports p99 TPOT; 50 ms per token is roughly 20 tokens per second per user.

What is goodput in LLM inference?

Goodput is the output throughput (tokens per second) at the highest request rate at which latency still meets the service-level objective — for example p99 TTFT ≤ 2 s and p99 TPOT ≤ 50 ms. Unlike peak throughput, it counts only tokens delivered fast enough to be useful. On one H100 SXM, Qwen3-30B-A3B-FP8 on vLLM 0.31.0 measured 3,996 output tokens/s of goodput at 9.5 requests per second.

What are tokens per joule?

Tokens per joule is output tokens divided by the energy used to produce them, measured from node or GPU power rather than rated TDP. It is the efficiency number that matters when power, not GPUs, limits a data center.

What is memory bandwidth utilization (MBU) for LLM decode?

MBU compares the bytes a decode step must read from GPU memory (active weights plus KV cache) with the GPU's rated memory bandwidth. TokenWatt derives it from model geometry and the measured step rate, so it works the same way on NVIDIA and AMD without hardware counters.

Disclosure

TokenWatt also builds tuned inference images for AMD Instinct. To keep the ruler honest: the measurement method and core are open; every Tuned result is shown next to the vendor's latest recommended configuration; and anyone can rerun a public report.

Questions or corrections: hello@tokenwatt.io.

Glossary

Longer explanations with the values we measured: time to first token (TTFT) · tokens per second · cost per token · GPU health check. Model benchmarks: Qwen3-30B-A3B · Qwen3-8B.