← Lab notes

Same Kimi K2.6, both FP4, almost the same price: 196 tokens/s on one provider, 12 on another

We audited Kimi K2.6, DeepSeek V4 Pro, GLM-5.3 and Qwen3.8 across OpenRouter providers with the same accuracy suites, latency probe, tokenizer check, cache probe and tool-call set. Accuracy differences were within noise. Decode speed for the same model differed up to 16×, prompt-cache discounts ran from 1% to 90%, and billed tokens matched the models' own tokenizers. One endpoint scored 78% on GSM8K because it applied our stop sequence to the model's hidden reasoning.

Choosing GPUs? Compare GPUs for your traffic · Get your node measured, free

Two OpenRouter providers list Kimi K2.6 as FP4, at nearly the same price: $3.41 and $3.50 per million output tokens. You'd expect the same product. One streamed 196 tokens per second, the other 12.

Disclosure: CodeRouter, a coding-task router, is built by our team and uses some of these models, which is one reason we wanted this data. Nobody paid for this audit, and no provider saw the results before publication.

This is the follow-up to lab notes #3, where nine providers served Llama 3.3 70B at the same accuracy and very different speeds. This time we took four open-weight models that coding tools use now, and checked more than speed: billed tokens against each model's own tokenizer, prompt-cache discounts, tool calls, and long prompts.

Short version:

  • Accuracy: no difference we could confirm. Every endpoint scored 97.5–99% on 200 GSM8K questions. On 150 code-reading questions (CRUXEval-O), no pair of endpoints of the same model differed significantly after correcting for multiple comparisons. The largest gap, 7.9 points between two GLM-5.3 endpoints, shrank to 2.5 points over all 800 questions. Endpoints declaring FP4 were not worse than FP8 ones.
  • Speed: up to 16× apart for the same model. Kimi K2.6 decoded at 12–196 tokens/s depending on the provider, GLM-5.3 at 55–160, DeepSeek V4 Pro at 38–90. Median time to first token ranged from 0.2 to 2.5 s.
  • Billing: exact. On chat requests, billed prompt tokens matched a count from each model's public tokenizer and chat template to within 9 tokens, and every billed cost equaled tokens × listed price.
  • Prompt caching: the discount, not the hit, is what differs. Most endpoints billed a repeated 4.5k-token prefix 73–90% cheaper. On one provider a cache hit saves 1–4%, because its cache price is almost its input price — but that input price was already 4–12× lower than the others'.
  • A naive audit still gets this wrong. One Qwen3.8 endpoint scored 78% on GSM8K against 99% elsewhere. The model was fine: the endpoint applied our stop sequence to the model's hidden reasoning, and 43 answers came back empty. Without the stop sequence, it scored 99%.

Method

Models moonshotai/kimi-k2.6, deepseek/deepseek-v4-pro-0813, z-ai/glm-5.3, qwen/qwen3.8-2.4t-a95b via OpenRouter, the night of 2026-11-04 to 05 (US Pacific)
Routing every request pinned with provider.order = [<endpoint>], allow_fallbacks = false; the serving provider is recorded on every response
Reasoning Kimi K2.6 and DeepSeek V4 Pro with reasoning off. GLM-5.3 and Qwen3.8 at reasoning effort "low": every endpoint of both rejects reasoning off
Accuracy GSM8K: the same 5-shot prompts as lab notes #3, first 200 test questions, temperature 0. CRUXEval-O: 150 of its 800 Python functions, sampled with a fixed seed; the model predicts the return value, which we compare as a Python literal (nothing is executed)
Tool calls 10 prompts: single and parallel calls, picking between tools, no tool needed, a forced tool_choice with nested arguments, answering from a tool result, numbers, enums and booleans. Arguments must parse as JSON and match the declared schema
Cache the same 4.5k-token prefix sent 4 times, 3 s apart, starting with a fresh random string so nothing was cached before
Context a 32k-token prompt with a code word in its first line and the question at the end; longer prompts on three endpoints
Latency 40 streamed requests one at a time, then 40 with 4 in flight; unique ~500-token prompts, 200 output tokens
Billing billed prompt tokens vs. the model's Hugging Face tokenizer plus chat template (DeepSeek's official encoder for V4); billed cost vs. tokens × listed prices
Failures rate limits and server errors retried up to 4 times with backoff; remaining failures reported separately, never scored as wrong

This post shows 17 endpoints: per model, the cheapest one, the model maker's own endpoint where its terms allow us to publish, and mainstream endpoints covering each declared precision. We read every provider's terms before publishing. We only name providers whose terms allow publishing benchmark results. The endpoint run by DeepSeek itself was hidden from us by our own OpenRouter privacy setting (no providers that may train on prompts).

Results

Prices are OpenRouter list prices per million tokens on 2026-11-05: input / output (cache read). TTFT and decode are medians of 40 sequential requests; a p99 from 40 samples is close to the slowest request, so treat it as an indication.

Kimi K2.6 (reasoning off)

Provider Declared $ in / out (cache) GSM8K CRUX-O Tools Cache discount TTFT p50 / p99 Decode p50
CoreWeave FP4 0.65 / 3.41 (0.15) 99.0% 72.7% 10/10 76% 351 ms / 615 ms 196 tok/s
Baidu FP4 0.95 / 4.00 (0.16) 98.5% 72.0% 9/9 see below 646 ms / 1.3 s 70 tok/s
Chutes INT4 0.39 / 1.80 (0.039) 99.0% 72.7% 9/9 90% 1.2 s / 2.4 s 48 tok/s
Moonshot AI (model maker) not declared 0.95 / 4.00 (0.16) 98.5% 70.7% 10/10 76% 1.1 s / 10.0 s 39 tok/s
Novita not declared 0.80 / 3.40 (0.16) 98.0% 72.7% 10/10 73%, 2 of 3 repeats hit 2.5 s / 15.3 s 36 tok/s
DeepInfra FP4 0.75 / 3.50 (0.15) 97.5% 71.3% 10/10 79% 856 ms / 3.5 s 12 tok/s

DeepSeek V4 Pro 0813 (reasoning off)

Provider Declared $ in / out (cache) GSM8K CRUX-O Tools Cache discount TTFT p50 / p99 Decode p50
Wafer MXFP4 0.107 / 4.28 (0.102) 97.5% 74.7% 10/10 4%, 2 of 3 hit 214 ms / 891 ms 90 tok/s
CoreWeave FP8 1.31 / 3.96 (0.044) 98.0% 72.7% 9/10* 86%, 2 of 3 hit 412 ms / 1.4 s 76 tok/s
Novita FP8 0.99 / 2.97 (0.033) 98.0% 72.0% 10/10 86% 1.5 s / 3.1 s 42 tok/s
DeepInfra FP8 1.30 / 2.60 (0.10) 98.0% 71.3% 9/9 87% 1.1 s / 3.1 s 38 tok/s

GLM-5.3 (reasoning effort low)

Provider Declared $ in / out (cache) GSM8K CRUX-O Tools Cache discount TTFT p50 / p99 Decode p50
CoreWeave NVFP4 0.75 / 3.00 (0.19) 99.0% 90.7% (800: 89.1%) 10/10 73%, 2 of 3 hit 468 ms / 1.1 s 160 tok/s
Wafer FP4 0.07 / 1.40 (0.069) 98.5% 84.0% (800: 88.0%) 10/10 1% 239 ms / 534 ms 127 tok/s
Baidu FP8 0.28 / 0.88 (0.052) 98.5%† 91.9%† (800: 90.5%†) 9/9 80% 1.1 s / 2.8 s† 79 tok/s
DeepInfra FP4 0.56 / 2.50 (0.125) 97.5% 89.3% (800: 89.2%) 10/10 76% 354 ms / 2.4 s 75 tok/s
Novita FP8 0.63 / 1.98 (0.117) 99.0% 84.7% (800: 88.1%) 9/9 81%, 2 of 3 hit 2.4 s / 31.5 s 55 tok/s

Qwen3.8 2.4T-A95B (reasoning effort low; both endpoints list $2.00 / $6.00, cache $0.25)

Provider Declared GSM8K CRUX-O Tools Cache discount TTFT p50 / p99 Decode p50
Alibaba (model maker) not declared 99.0% 98.0% 8/9* 81% 1.4 s / 3.0 s 38 tok/s
Novita not declared 99.0% 97.3% 8/9* 81% 1.5 s / 2.3 s 39 tok/s

Tools counts schema-valid calls with the expected function and arguments (or the expected plain answer), out of the prompts the endpoint accepted. On every endpoint we measured, no tool call had invalid JSON or broke its schema, and parallel calls worked everywhere; tool calling differed only in which requests an endpoint accepts. 9/9 means the endpoint doesn't support a forced tool_choice, so OpenRouter had nowhere to send that prompt (see below). * On one prompt that asked to rename a function, the model called read_file first instead of edit_file — a valid call, arguably the safer move, but not the one we scored for. † Baidu's GLM-5.3 endpoint rate-limited us heavily: its scores cover 196 of 200 GSM8K questions, 148 of 150 (and 681 of 800) CRUX-O questions, and its latency 30 of 40 sequential requests. For GLM-5.3 and Qwen3.8, TTFT counts the first reasoning token.

Are the accuracy differences real? GSM8K is saturated at 97.5–99% for all four models; the widest gap within a model is 1.5 points. On CRUX-O we compared every pair of endpoints of the same model on the same questions (McNemar's exact test) and corrected for the number of pairs (Holm). Only one pair came close: GLM-5.3 on Baidu (91.9%) vs. Wafer (84.0%), p = 0.008 before correction and 0.075 after. To check it, we ran all five GLM-5.3 endpoints on the other 650 CRUXEval-O questions. Four of them got 577, 578, 578 and 580 right (88.8–89.2%); Baidu got 90.1% of the 533 it answered. Over all 800 questions the endpoints scored 88.0–90.5%, and no pair differed even before correction (smallest p = 0.16). The 7.9-point gap in the first sample was noise. Each CRUX-O score from 150 questions carries a 95% interval of up to about ±7 points, which is too wide to rank providers on.

Declared precision didn't predict quality. Endpoints declaring FP4, MXFP4, NVFP4 or INT4 scored the same as FP8 and undeclared ones, and Kimi K2.6's model maker doesn't declare a precision at all.

Speed and price are separate decisions, again

  • Kimi K2.6: CoreWeave decoded 16× faster than DeepInfra at a slightly lower list price, with the same accuracy and the same tool-call results. On the model maker's own endpoint, one request in ten waited more than 6.5 s for a first token.
  • GLM-5.3: the endpoint with the cheapest input (Wafer, $0.07 per million input tokens and $1.40 output) had the fastest first token and the second-fastest decode.
  • DeepSeek V4 Pro: the fastest endpoint was also the cheapest on input, but not on output: $4.28 per million output tokens, against $2.60–2.97 on the endpoints that decode at about half its speed.
  • Qwen3.8: the two endpoints we can name were indistinguishable: same price, 38–39 tok/s, 1.4–1.5 s to first token.
  • Four requests in flight didn't change the picture. Median decode speed moved by at most 17% and median TTFT by at most about 30%, in both directions. We didn't test real load.

These numbers come from one client location in the US, on one night, through OpenRouter. Providers change hardware and kernels without notice, and queues change with the time of day. Treat them as a snapshot.

Billing: what we were charged matched what was listed

We counted every prompt with the model's public tokenizer and chat template from Hugging Face and compared the count with what each endpoint billed.

  • Prompt tokens: identical to our count on every chat request, with two small, consistent exceptions: two Kimi K2.6 endpoints billed 2 tokens fewer, and both Qwen3.8 endpoints billed 9 tokens more on prompts without a system message (and exactly our count with one). Both come from small differences in the served chat template, not surcharges. Prompts with tool definitions differed by up to about 60 tokens either way, because each provider renders tool schemas into the prompt its own way.
  • Completion tokens: 1–4% above the visible text re-tokenized, which is the end-of-turn token plus, on two Kimi endpoints, one token for an empty reasoning block.
  • Cost: on every response from these 17 endpoints, the billed cost equaled prompt tokens × input price + cached tokens × cache price + completion tokens × output price, to the rounding.
  • No silent truncation: each endpoint billed the full 32k-token prompt and recalled the code word from its first line, except Baidu's GLM-5.3 endpoint, which rate-limited all 20 attempts. Longer prompts also went through: 624k tokens to GLM-5.3 on Wafer, 229k to DeepSeek V4 Pro on Wafer and 162k to Kimi K2.6 on Chutes, each recalled. A 640k-token prompt to DeepSeek V4 Pro on Wafer was rate-limited on all ten attempts.

One more thing we learned about long prompts: OpenRouter checks the length before sending, using an estimate of roughly one token per four characters. Our filler text had about six characters per token, so a 940k-token prompt was rejected as "about 1.47 million tokens" without reaching the provider. With ordinary English text the estimate is close; with code or long words, you may hit the limit early.

Prompt caching: check the discount, not just the hit

Every endpoint in this post reported cached tokens on repeats of our 4.5k-token prefix. A cache hit is only worth its discount, though, and that varied:

  • 73–90% cheaper on most endpoints, the most on Chutes' Kimi K2.6 endpoint.
  • 1–4% cheaper on Wafer, whose cache price is within 1–5% of its input price. Its input is so cheap that a repeated DeepSeek V4 Pro prefix still cost less there than on any other endpoint shown; for GLM-5.3, only Baidu's was cheaper.
  • Hits are best-effort. On 5 of the 17 endpoints, one of the three repeats missed, presumably because it landed on a replica that hadn't seen the prefix. An agent that counts on a 90% hit rate should measure its own.
  • Baidu's Kimi K2.6 endpoint reported cached tokens on prompts it couldn't have seen. Each latency prompt was unique, yet it reported 300–350 of about 520 tokens as cached, and only 61–70% of the prefix on an exact repeat. It billed those tokens at the cache price, so unique prompts cost less than list. But an exact repeat of the prefix cost more there than on any other Kimi K2.6 endpoint shown ($1.79 per thousand repeats, against $0.19–1.04), and the number doesn't behave like a prefix-cache hit count, so don't use it to judge your cache hit rate there.

The audit that almost went wrong (again)

What we saw What it actually was
One Qwen3.8 endpoint at 78% on GSM8K, against 99% elsewhere it applied our stop sequence ("\n\nQuestion:") to the model's hidden reasoning, which often quotes the question. 43 of its 44 misses were empty answers. Rerun without the stop sequence: 42 of those 44 right, 99.0% overall
A forced tool_choice "failing" on many endpoints 7 of the 17 endpoints don't support it. With the request pinned and fallbacks off, OpenRouter returns HTTP 404 "No endpoints found", which a naive script scores as a tool-calling failure. The model was never asked
"Reasoning off" returning HTTP 400 for GLM-5.3 and Qwen3.8 on every endpoint both models only run with reasoning on. At effort "low", GLM-5.3 used a median of 0 reasoning tokens on GSM8K and 14–21 on CRUX-O; Qwen3.8 about 125 on GSM8K
One endpoint "failing" almost one request in five Baidu's GLM-5.3 endpoint answered HTTP 429 to 1,566 of our attempts; 216 of 1,168 requests still failed after four retries. That's availability under light load, not quality

The same lesson as lab notes #3: before you trust a score, check what the API did with your parameters, and count failures separately.

Not covered

  • Long agentic sessions, multi-turn tool use, and real coding work. Our tasks are short, and Kimi K2.6 ran out of its 300-token budget on about a fifth of the CRUX-O questions on every endpoint (it explained before answering), which lowers its scores but equally everywhere.
  • Throughput under load, other regions and times of day, and how these endpoints change over time.
  • Providers whose terms restrict publishing benchmark results, and DeepSeek's own endpoint (hidden by our OpenRouter privacy setting).
  • We measured 4–8 endpoints per model and show 2–6, chosen to cover price and declared precision; OpenRouter lists 4 to 48 per model.

Reproduce

cd bench/provider-audit-2026-11
uv run --no-project --with aiohttp python audit.py snapshot
uv run --no-project --with aiohttp python audit.py run --tests tools,cache,context,gsm8k,crux,latency
uv run --no-project --with aiohttp --with tokenizers --with tiktoken --with jinja2 --with jsonschema \
    python analyze.py --tok-dir <tokenizer files>

The data for the endpoints in this post is at /data/audits/provider-audit-open-models-2026-11.json. The audit cost about $11 in API usage.

Keep reading

4 min read

Lab notes #3: Nine providers, one Llama 3.3 70B — same accuracy, 18× different speed

We audited every OpenRouter provider of Llama 3.3 70B Instruct with the same GSM8K suite and latency probe. Accuracy was statistically indistinguishable (94.0–98.0%), decode speed ranged from 19 to 347 tokens/s, and output prices varied 7×. A naive first pass wrongly scored two providers at 0% and 44%.

OpenRouterLlama 3.3 70Binference providers
6 min read

The pi adapter for Qwen3.8-27B on FP8: 3× fewer turns held up, the accuracy gain was smaller

harness-hub's pi adapter is a LoRA that teaches Qwen3.8-27B to work inside the pi coding agent. Its card reports 39.5% → 55.8% on Terminal-Bench 2.1 on BF16 weights. We serve the FP8 checkpoint, so we tested it there: on 25 Terminal-Bench tasks under pi, the adapter solved 17 and the base model 14, a gap too small to call on 25 tasks. Its efficiency held clearly: on the 12 tasks both solved, it needed 9.5 turns against 28.5 and 1,432 output tokens against 5,254 (medians). It also ran 10 minutes without a fault under the long-context load that crashed MTP on our shared GPU. It's on our API now at Qwen3.8's price.

LoRAQwen3.8pi coding agent
5 min read

94% prefix-cache hits on hybrid models with real coding-agent traffic, against 56% in our synthetic test

A synthetic test we published on November 16 showed a hybrid model, Qwen3.8-27B, reusing far less of a shared prompt than a dense-attention model (56% against 89%), because vLLM caches its prompts only in whole blocks of 784 tokens. On two days of our own team's coding-agent traffic on our API, the same rule cost little: vLLM served 94.3% of Qwen3.8-27B's prompt tokens and 94.8% of Qwen3.6-35B-A3B's from cache, on median prompts of 114K and 86K tokens. The median request recomputed only 1,562 to 2,690 tokens.

prefix cachingprompt cachehybrid attention