Verify what you got
Drivers, firmware, form factor, memory bandwidth and power baselines — checked before a node carries traffic.
Seen: same instance type, two drivers — one served vLLM, one couldn't start it.Between a powered GPU and a reliable API sit drivers, operators, inference engines, kernels and gateways — and any release can break the stack. We validate, tune and measure it on AMD Instinct and NVIDIA, for public APIs and private ones. Measured, not claimed.
New capacity waits on interconnection, not on GPUs. Every energized watt that doesn't turn into a billable token is the most expensive kind of idle.
Peak tokens per second at unlimited latency tells you nothing about how many cards you need to serve your users within your TTFT and tail-latency targets.
A new driver, operator, engine image or quantization can swing throughput by double digits — or stop the stack from starting. Without a fixed ruler, nobody can tell what changed.
Each stage is a place we've seen stacks fail on real hardware — and each one ends with a measurement that shows it's fixed.
Drivers, firmware, form factor, memory bandwidth and power baselines — checked before a node carries traffic.
Seen: same instance type, two drivers — one served vLLM, one couldn't start it.Kubernetes or Slurm, GPU operators, vLLM or SGLang, an API gateway — integrated from open source and validated step by step.
Seen: four version and naming mismatches between correct components; two left no logs.SLO goodput, tokens per joule and an accuracy gate decide what goes to production — not peak throughput.
Seen: FP8 KV cache — +10.6% throughput, −4 points accuracy.Health baselines, regression checks on every upgrade, utilization and cost per million tokens.
Seen: an internal assistant for 1,000 people — two GPUs (one spare), 0.4% average utilization.For public APIs and internal ones. We work with a small number of design partners across the whole path — start with a node evaluation.
Every TokenWatt report pins the hardware, driver, engine, model, quantization, workload trace and SLO — then states exactly what it does and doesn't cover.
How much of the rated HBM bandwidth the decode loop actually turns into useful reads. The clearest signal of software headroom.
From the BMC or a metered PDU, sampled through the steady-state window. GPU board power is reported separately and labeled.
Output tokens per second at the highest load where p99 TTFT and p99 time-per-output-token both stay inside your targets.
A fixed evaluation set with a pass threshold agreed before the run. Speed that costs quality doesn't count.
Bench is a client. It talks to any OpenAI-compatible server — vLLM, SGLang, TensorRT-LLM — and reads telemetry from the node. It never touches your weights.
Declare TTFT and tail-latency targets, the request-length distribution and the eval set up front. The spec file is part of the report.
Production image, vendor's latest recommended image, and any candidate change — measured the same way, on the same node.
Machine-readable JSON plus a one-page summary with the capacity math: how many cards and kilowatts the same billable throughput needs.
# 1. describe what "good" means for you $ cat slo.yaml slo: ttft_p99_ms: 2000 tpot_p99_ms: 50 workload: trace: chat-mix-v1 # input/output length distribution accuracy: suite: gsm8k-500 max_drop_pts: 0.5 power: source: redfish # or ipmi-dcmi, pdu # 2. measure each configuration $ docker run --rm --network host \ -v $PWD:/work tokenwatt/bench run \ --endpoint http://localhost:8000/v1 \ --spec /work/slo.yaml --label production # 3. compare and write the report $ tokenwatt report production vendor-latest tuned ✓ report written: tw-report-2026-10-06.json ✓ summary written: tw-report-2026-10-06.html
We also build tuned inference images for AMD Instinct. That's exactly why the rules below are strict and public.
AMD Instinct and NVIDIA are measured with the same definitions, the same workload traces and the same power sources.
Every comparison includes the vendor's own latest recommended configuration. We never benchmark against a configuration we picked to lose.
Definitions, workload traces and the measurement core are published. Anyone can rerun a report and get the same answer within its stated band.
Bench is read-only. No weights, prompts or logs leave your environment unless you choose to publish a report.
Node power comes from the BMC or a metered PDU. Nameplate TDP is never used to estimate energy.
Every report lists the context lengths, quantizations and concurrency levels it did not test. No extrapolation, no peak-only numbers.
One telemetry core, three products, all priced per GPU and installed by your own engineers.
Vendor-neutral inference measurement for your models, SLOs and nodes.
Open core. Paid tier adds power capture, history and reports.
Find the GPU that isn't broken — just slow.
Read-only agent. Installs in minutes.
Decode-path tuned inference images for AMD Instinct, verified with Bench.
Per-GPU subscription. Your gains, verified by you.
Every public report ships with its spec file, raw JSON and the command that reproduces it.
Every run we do on real hardware gets written up: setup, numbers, surprises and the commands to reproduce it.
harness-hub's pi adapter is a LoRA that teaches Qwen3.8-27B to work inside the pi coding agent. Its card reports 39.5% → 55.8% on Terminal-Bench 2.1 on BF16 weights. We serve the FP8 checkpoint, so we tested it there: on 25 Terminal-Bench tasks under pi, the adapter solved 17 and the base model 14, a gap too small to call on 25 tasks. Its efficiency held clearly: on the 12 tasks both solved, it needed 9.5 turns against 28.5 and 1,432 output tokens against 5,254 (medians). It also ran 10 minutes without a fault under the long-context load that crashed MTP on our shared GPU. It's on our API now at Qwen3.8's price.
A synthetic test we published on November 16 showed a hybrid model, Qwen3.8-27B, reusing far less of a shared prompt than a dense-attention model (56% against 89%), because vLLM caches its prompts only in whole blocks of 784 tokens. On two days of our own team's coding-agent traffic on our API, the same rule cost little: vLLM served 94.3% of Qwen3.8-27B's prompt tokens and 94.8% of Qwen3.6-35B-A3B's from cache, on median prompts of 114K and 86K tokens. The median request recomputed only 1,562 to 2,690 tokens.
Qwen3.8-27B ships with a one-layer multi-token prediction (MTP) head that vLLM can use as a draft model. On one H100 NVL with vLLM 0.33.0, three draft tokens per step raised single-request decode from 74.8 to 198 tokens per second (2.65×), and total throughput by 101%, 49% and 19% at 4, 16 and 48 requests in flight. GSM8K held (99.0% against 98.0%). The price is memory: room for the KV cache fell 24%, from 709,527 to 541,801 tokens. We turned it on in our API the same day, then off again that evening: with two models sharing the GPU under CUDA MPS, MTP crashed both engines under long-context load.
We're looking for a small number of US inference teams and GPU clouds to work with closely. You get Bench on your fleet and a say in the roadmap; we get honest feedback.