What we measured, and how.
Runs on real hardware, written up with the setup, the numbers, what surprised us and the commands to reproduce it.
The pi adapter for Qwen3.8-27B on FP8: 3× fewer turns held up, the accuracy gain was smaller
harness-hub's pi adapter is a LoRA that teaches Qwen3.8-27B to work inside the pi coding agent. Its card reports 39.5% → 55.8% on Terminal-Bench 2.1 on BF16 weights. We serve the FP8 checkpoint, so we tested it there: on 25 Terminal-Bench tasks under pi, the adapter solved 17 and the base model 14, a gap too small to call on 25 tasks. Its efficiency held clearly: on the 12 tasks both solved, it needed 9.5 turns against 28.5 and 1,432 output tokens against 5,254 (medians). It also ran 10 minutes without a fault under the long-context load that crashed MTP on our shared GPU. It's on our API now at Qwen3.8's price.
94% prefix-cache hits on hybrid models with real coding-agent traffic, against 56% in our synthetic test
A synthetic test we published on November 16 showed a hybrid model, Qwen3.8-27B, reusing far less of a shared prompt than a dense-attention model (56% against 89%), because vLLM caches its prompts only in whole blocks of 784 tokens. On two days of our own team's coding-agent traffic on our API, the same rule cost little: vLLM served 94.3% of Qwen3.8-27B's prompt tokens and 94.8% of Qwen3.6-35B-A3B's from cache, on median prompts of 114K and 86K tokens. The median request recomputed only 1,562 to 2,690 tokens.
MTP on Qwen3.8-27B: 2.65× faster per request, at the cost of 24% of the KV cache
Qwen3.8-27B ships with a one-layer multi-token prediction (MTP) head that vLLM can use as a draft model. On one H100 NVL with vLLM 0.33.0, three draft tokens per step raised single-request decode from 74.8 to 198 tokens per second (2.65×), and total throughput by 101%, 49% and 19% at 4, 16 and 48 requests in flight. GSM8K held (99.0% against 98.0%). The price is memory: room for the KV cache fell 24%, from 709,527 to 541,801 tokens. We turned it on in our API the same day, then off again that evening: with two models sharing the GPU under CUDA MPS, MTP crashed both engines under long-context load.
Qwen3.8-27B with thinking on: 26× the cost for 5.7 more points on our 175 coding tasks
We ran the two models on our own API, Qwen3.8-27B and Qwen3.6-35B-A3B, through the 175 single-file Python tasks of our CodeRouter benchmark, with thinking off and on. Thinking raised Qwen3.8 from 73.1% to 78.9% solved with one retry, and the cost of the run from $0.27 to $7.02; 42 of its 247 attempts used the whole 32,768-token budget, and 41 of those never wrote code. Qwen3.6 went from 78.3% to 82.3% for $0.16 → $1.27. On these tasks both trail Claude Haiku 5.5 (93.7%, $0.15) and Kimi K2.6 (94.9%, $0.57).
MI350X vs H100: AMD's own software stack more than doubled the MI350X; tuning the H100 changed nothing
We served Qwen3-8B and Qwen3-Coder-30B on one AMD Instinct MI350X and one H100 SXM under the same latency target. On vLLM's default ROCm build the MI350X served 36% fewer tokens than the H100; on AMD's own image with AITER kernels it served 52% more on Qwen3-8B and 31% more on Qwen3-Coder-30B. vLLM's recommended H100 settings changed nothing. Per million tokens, the cheapest result came from a $1.50-an-hour spot H100.
Prefix caching on a hybrid model: fewer cache hits, a bigger payoff (Qwen3.8-27B on one H100)
We repeated last week's prefix-cache capacity test on a hybrid model, Qwen3.8-27B (Gated DeltaNet + full attention), on the same H100 SXM setup. vLLM sizes each cache block to hold one DeltaNet state, 784 tokens with a BF16 KV cache and 1,568 with FP8, and only whole blocks hit. Hit rates came out far below the dense model's, but capacity within our latency target rose 19%, 36% and 70% at 50%, 75% and 90% shared on chat-like traffic, and 1.7×, 2.4× and 3.4× with 4,000-token prompts. A formula built on prefill's share of GPU time predicts a fraction of that. Thanks to Manas Choudhary, whose RTX 5090 runs found the block size.
Qwen3 Coder 30B in opencode, Qwen Code, Cline and Kilo Code for $0.07 per million input tokens
How to point opencode, Qwen Code, Cline and Kilo Code at Qwen3-Coder-30B-A3B-Instruct on TokenWatt's OpenAI-compatible API, and what to expect from a 30B coding model: 64.8% on our 176 single-file Python tasks, for $0.057 for the whole suite.
Same Kimi K2.6, both FP4, almost the same price: 196 tokens/s on one provider, 12 on another
We audited Kimi K2.6, DeepSeek V4 Pro, GLM-5.3 and Qwen3.8 across OpenRouter providers with the same accuracy suites, latency probe, tokenizer check, cache probe and tool-call set. Accuracy differences were within noise. Decode speed for the same model differed up to 16×, prompt-cache discounts ran from 1% to 90%, and billed tokens matched the models' own tokenizers. One endpoint scored 78% on GSM8K because it applied our stop sequence to the model's hidden reasoning.
Claude Opus 5.5 lists at $4 per million input tokens, GPT-5.5 at $5. On the same code, Opus input costs 28% more
Per-token list prices assume a token is the same amount of text everywhere. It isn't. We sent the same 24 samples of code, diffs, test logs, shell output, tool-call JSON, Markdown and English and Chinese prose to 13 models and read the billed token counts. On code and tool output Claude 5.5 used 1.60 times as many tokens as GPT-5.5, so Opus 5.5 input costs 28% more per character than GPT-5.5 despite a lower per-token price, and Sonnet 5.5 input costs 37% more than Gemini 3 Pro's at the same per-token price.
Five GPU Marketplace Nodes Measured: All Healthy, 30–37% Less per Token on RTX PRO 6000 and H100
Lium is a GPU marketplace where each host sets its own price. We rented five nodes, one GPU each, and ran our standard evaluation: a health check against a healthy card, then Qwen3-8B on vLLM under a latency target. All five ran at their full power limit with normal clocks and memory bandwidth. RTX PRO 6000 and H100 served the same load as the same cards on RunPod Secure Cloud and cost 30–37% less per million output tokens at the hosts' prices; Lium's H200 cost the same per hour as RunPod's and its B200 more. Lium provided the credit; the measurements and the write-up are ours.
Where a coding agent's bill goes: output is under 10% of it
385 developer-days of our own team's coding-agent traffic at list prices: output is under 10% of the bill, the most expensive tenth of days is over half of it, and heavy days cost more per request, not only more requests.
What a coding agent costs per developer: 385 developer-days of our own usage
Five months of our own team's coding-agent traffic: 163 requests and $4.30 on a typical developer day, 378 requests and $13.03 on a heavy one, at list prices with routing (corrected November 7). How that compares with Claude's subscription and seat plans, what moved the bill, and what these numbers don't tell you.
Long prompts are where an inference node loses money: 1K, 8K and 16K input tokens on one H100
We measured how much one H100 can serve within a latency target as prompts grow from about 1K to 8K and 16K input tokens (Qwen3-30B-A3B FP8, vLLM, no prefix reuse). Capacity fell from 11.5 to 1.89 and 0.73 requests per second, because the first-token deadline caps how many long prompts the GPU can prefill at once. At a flat input price, long uncached prompts earn less than the GPU costs.
What a prompt cache is worth: the same H100 served 82% more requests at a 90% cache hit rate
We served Qwen3-30B-A3B-Instruct-2507 FP8 on our production H100 with vLLM and repeated the same latency-target search with 0%, 50%, 75% and 90% of every prompt drawn from shared prefixes. vLLM's own counters showed cache hit rates of 0%, 49%, 74% and 89%. Capacity rose from 11.5 to 21.0 requests per second, and revenue per GPU-hour from $10.37 to $18.84 at flat prices. With cached input billed at a quarter of the price, the node still earns more and the customer pays far less.
Do inference providers really make $10–20 per GPU-hour? We measured DeepSeek V4.1 Flash on H200s
A widely shared post says inference providers earn $10–20 per GPU-hour serving open models on rented GPUs. We served DeepSeek V4.1 Flash on 4× H200 with vLLM's published recipe and measured what it can sell within a latency target. At the prices 38 OpenRouter endpoints charge, a fully loaded H200 earns $1.80–8.84 an hour, $4.42 at the most common price, against $5.29 rent. At 100 tokens per second per user it earns $0.24. Margins exist, but they depend on hardware, tuning and pricing, not on showing up.
MOSS-Image on rented GPUs: a 4-step LoRA makes it 14× faster, and the text survives
MOSS-Image, a 5B open text-to-image model known for legible Chinese and English text and JSON layouts, on an RTX 4090, L40S, RTX PRO 6000 and H100. The official 50-step sampler took 17–47 seconds per 1024×1024 image ($11.6–19 per 1,000 images). A community 4-step LoRA took 1.6–3 seconds ($0.74–2.22 per 1,000) and rendered text just as accurately by OCR, but photos lost the natural look the model is praised for. The cheapest card per image was the cheapest per hour.
RTX PRO 6000 Blackwell: what 96 GB at $1–2.50 an hour buys you for LLMs and video
We rented the RTX PRO 6000 Blackwell (96 GB) and ran the same tests as on our H100, H200, A100 and RTX 4090 nodes. Serving Qwen3-8B under a latency target, it handled 3.09 requests per second, $0.56 per million output tokens at $2.49 an hour, close to an H100 NVL and half the cost of an A100. A 15-second 720p Wan2.2 video took 12.6 minutes with SageAttention 2 on one card, 31% slower than an H200 but 39% cheaper per clip. At the ~$1/hour some marketplaces advertise, both costs fall by more than half.
Wan2.2 image-to-video on an H200: a 15-second 720p clip takes 18 minutes, or 9.6 with one change
We deployed a Wan2.2 14B image-to-video model with ComfyUI on rented H200s and measured clip length, resolution, attention backend and PyTorch build on the same machine. A 15-second 720p clip took 18.1 minutes with PyTorch attention and 9.6 minutes with SageAttention 2 (1.89x), at 68.5 GB peak GPU memory. Upgrading PyTorch to CUDA 13 alone gained 6%. Output stayed visually the same. On short clips, two H200s of the same model differed by 1.5x or more.
GPU rental prices, early October 2026: list prices barely moved, but where you rent changes the price up to 4.7×
We collected on-demand GPU list prices from 16 cloud providers three times a day from October 7 to 19, 2026. List prices barely moved: 14 of 407 tracked offers changed. The spread between providers is the real story: H100 from $2.99 to $13.98 per GPU-hour (4.7×), A100 2.6×, L40S 2.1×. And 13 of 22 H100 listings didn't say whether the card is SXM, PCIe or NVL, a difference worth 2.25× in served tokens in our tests.
H100 vs A100 for LLM inference: 7.8× the capacity for 2.2× the price
We served Qwen3-8B with vLLM on a rented H100 SXM and a rented A100 SXM 80GB under the same latency target. The H100 served 8.79 requests per second against 1.13 on the A100: 7.8× the capacity for 2.2× the hourly price, so each token cost 71% less ($0.31 against $1.06 per million output tokens). On raw specs the H100 is only 1.7–2.6× faster. The rest of the gap comes from slow tokens on the A100, which a serving setting can partly fix.
We watched two rented GPUs for 24 hours: 21 thermal throttles, and throughput didn't move
We rented an RTX 4090 and an A40 on RunPod Secure Cloud and kept both serving an 8B model around the clock while TokenWatt Monitor sampled the GPUs every minute. The 4090 hit thermal limits 21 times in 24 hours, four of them hardware thermal slowdowns, while the A40 never did. Each episode lasted about a minute, and the 4090's throughput never moved by more than 0.7%. 132,420 requests, zero errors.
Renting an H100? The variant matters more than the luck of the draw
We rented 19 H100s on RunPod Secure Cloud and checked each one, then served the same 8B model on one of each variant. All 10 SXM cards were healthy and nearly identical. Every NVL card was set to 310 W, below NVIDIA's 350–400 W range, and the PCIe cards split into two speed groups 12% apart. Per token, the SXM was the cheapest: $0.31 per million output tokens, against $0.43 on PCIe and $0.57 on NVL.
FP8 vs BF16 for LLM inference: faster tokens, but not always more capacity
We served Qwen3-8B with BF16 and FP8 weights on an RTX 4090 and an L40S. FP8 cut the median time per token by 35–42% on both cards, but it raised capacity within our latency target only on the 4090, where BF16 left too little memory for the KV cache. On the L40S, capacity didn't move.
RTX 4090 vs L40S vs A100 vs A40 for LLM inference: what an 8B model really costs per token
We served Qwen3-8B on four cheaper GPUs rented from RunPod Secure Cloud under the same latency target. An RTX 4090 with FP8 weights was cheapest at $0.90 per million output tokens; the A100, the 4090 in BF16 and the L40S landed at $1.06–1.13; the A40 cost $1.61. Every card was limited by occasional slow tokens rather than average speed, and a 30B mixture-of-experts model on an H100 was about 3× cheaper per token than any of them.
H100 vs H200 for LLM inference: 18% more capacity, 33% higher price — and where the H200 pulls ahead
We ran the same model, engine and latency target on an H100 SXM and an H200 on RunPod. At a 50 ms-per-token SLO the H200 served 18% more requests for 33% more money, so each token cost 11% more. At a fixed load it generated tokens 36% faster, and with a 30 ms target its lead grew to 29%, close to breaking even on price. Measured numbers, cost per million tokens and the break-even price.
RunPod vs Vast.ai vs Lambda: we benchmarked four rented H100s and found a 2.9× spread
Four H100 SXM GPUs rented by the hour on Lambda, RunPod and Vast.ai (a datacenter host and a community host), each serving Qwen3-30B-A3B-FP8 on vLLM 0.31.0 under the same latency SLO. Cost per million tokens ranged from $0.199 (Vast.ai datacenter) to $0.264 (Lambda); the cheapest host per hour couldn't meet the SLO at all, running at 40% of normal clock speed; both Vast hosts capped GPU power, and the official vLLM image failed to start on RunPod until we changed one kernel.
vLLM vs SGLang on an H100: same capacity at 50 ms, but SGLang streams a third faster and leads by ~20% at 30 ms
Two H100s in the same RunPod datacenter, one serving Qwen3-30B-A3B-FP8 with vLLM 0.31.0 and the other with SGLang 0.5.21, measured in full mode (10-minute windows, five repeats, 600 accuracy questions). Under a 50 ms per-token target, both served about 9 requests/s at $0.29 per million tokens. At the same load, SGLang returned the first token 3× sooner and generated tokens about 30% faster, so under a 30 ms target it served about 21% more requests. Accuracy was the same.
Lab notes #6: We tuned vLLM's MoE kernels on an H100 for 2.5 hours. It made no measurable difference.
vLLM warns "Using default MoE config. Performance might be sub-optimal!" for Qwen3-30B-A3B-FP8 on H100. We ran vLLM's own tuner for 159 minutes, confirmed the tuned config loaded, and measured default, tuned and default again on the same GPU. Kernels got up to 7% faster; end-to-end capacity didn't change, and the latency change was within run-to-run drift.
Does selling tokens pay? Break-even math for an inference provider, from a measured H100
Using measured SLO throughput for Qwen3-30B-A3B on one H100 and live OpenRouter prices, we work out the break-even utilization for rented, owned and idle GPUs. Rented GPUs need 64–77% utilization to match the cheapest price; idle owned GPUs break even at 2%. Includes a calculator.
How many GPUs does an internal ChatGPT need? Fewer than you think — and that's the cost problem
Sizing a self-hosted LLM (a private ChatGPT) from a measured H100 benchmark. One H100 can carry the peak chat traffic of roughly 12,000 employees, but at 1,000 employees it sits 99.6% idle and a million tokens costs $67.70 instead of $0.30. Utilization, not GPU count, decides the economics. Includes a calculator.
Lab notes #2: Short benchmark windows overstated H100 goodput by 14%
Full-mode rerun of Qwen3-30B-A3B-FP8 on one H100 with vLLM 0.31.0. With 10-minute windows the SLO boundary fell from 11.3 to 9.5 req/s and goodput from 4,572 to 3,996 tok/s; the FP8 KV-cache accuracy drop shrank from 8 to 4 points on a 600-sample suite.
Lab notes #3: Nine providers, one Llama 3.3 70B — same accuracy, 18× different speed
We audited every OpenRouter provider of Llama 3.3 70B Instruct with the same GSM8K suite and latency probe. Accuracy was statistically indistinguishable (94.0–98.0%), decode speed ranged from 19 to 347 tokens/s, and output prices varied 7×. A naive first pass wrongly scored two providers at 0% and 44%.
Lab notes #4: Four things that broke a two-node GPU Kubernetes cluster before vLLM would serve
Validating a private-inference stack — k3s, NVIDIA GPU Operator, vLLM, DCGM — on two cloud A10s took five attempts. A CUDA 13 image on a 570 driver, broken -cu129 image variants, a CDI hook missing from the host toolkit, and a Service named vllm. Every failure was a version or naming mismatch between open-source parts, and most left no logs.
Lab notes #5: Serving one H100 as a token API for six hours — 162,040 requests, 9 failures, no request lost when the GPU was handed back
We measured an H100's SLO capacity, set the gateway's admission limit from it, and served Qwen3-30B-A3B-FP8 through it for six hours of day-shaped traffic with hourly overload and GPU-handback drills. Availability 99.994%, 99.97% of requests within the SLO, six drains with no in-flight loss, billing reconciled to the token. Here are the failures as well.
Billable tokens per watt: why we're building a vendor-neutral ruler for LLM inference
Power, not GPUs, is now the constraint on AI data centers. We explain the four numbers TokenWatt Bench reports — SLO goodput, tokens per joule, decode bandwidth utilization and an accuracy gate — and why each comparison includes the vendor's own best configuration.
Lab notes #1: FP8 KV cache on an H100 — 11% more goodput, and it failed our accuracy gate
First real-hardware run of TokenWatt Bench. Qwen3-30B-A3B-FP8 on one H100 with vLLM 0.31.0. FP8 KV cache doubled KV capacity and raised SLO goodput 10.6% and tokens per joule 14%, but GSM8K accuracy dropped 8 points with uncalibrated KV scales.
How to measure decode bandwidth utilization without hardware counters
A vendor-neutral way to compute memory-bandwidth utilization (MBU) for LLM decode from model geometry and client-side timing — including MoE expert routing and MLA KV caches — and how we validated it against a simulated roofline engine.