Qwen3.8-27B with thinking on: 26× the cost for 5.7 more points on our 175 coding tasks
We ran the two models on our own API, Qwen3.8-27B and Qwen3.6-35B-A3B, through the 175 single-file Python tasks of our CodeRouter benchmark, with thinking off and on. Thinking raised Qwen3.8 from 73.1% to 78.9% solved with one retry, and the cost of the run from $0.27 to $7.02; 42 of its 247 attempts used the whole 32,768-token budget, and 41 of those never wrote code. Qwen3.6 went from 78.3% to 82.3% for $0.16 → $1.27. On these tasks both trail Claude Haiku 5.5 (93.7%, $0.15) and Kimi K2.6 (94.9%, $0.57).
Choosing GPUs? Compare GPUs for your traffic · Get your node measured, free
With thinking on, Qwen3.8-27B solved 78.9% of our 175 coding tasks. With thinking off: 73.1%. The run with thinking cost $7.02; the run without it, $0.27. That's 26× the cost for 10 more tasks.
Both models in this post run on our own API, api.tokenwatt.io, on one H200: Qwen3.8-27B and Qwen3.6-35B-A3B (Qwen3.8 and Qwen3.6 below). On November 20 we put them through the same 175 single-file Python tasks we use for every model in our CodeRouter benchmark, once with thinking off and once with it on. Disclosure: TokenWatt is our company, and these are our own hosted models. The results aren't flattering for them. We're publishing them as measured.
175 single-file Python tasks through api.tokenwatt.io, one retry with the failing test output. Cost is for all 175 tasks:
| Run | First try | One retry | Cost | Median time | At the token cap |
|---|---|---|---|---|---|
| Qwen3.6, thinking off | 68.0% | 78.3% | $0.16 | 3.2 s | 0 of 231 |
| Qwen3.6, thinking on | 68.6% | 82.3% | $1.27 | 33.0 s | 2 of 230 |
| Qwen3.8, thinking off | 60.6% | 73.1% | $0.27 | 5.5 s | 0 of 244 |
| Qwen3.8, thinking on | 58.9% | 78.9% | $7.02 | 76.9 s* | 42 of 247 |
"First try" means the code passed the task's tests without the retry. "Median time" is per task, retry included. "At the token cap" counts attempts (first tries plus retries) that used all 32,768 output tokens. Cost is the tokens each response reported, times our list prices, retries included: what a customer would pay at list. It is not what these runs cost us to serve. Thinking tokens bill as output.
*We turned on MTP speculative decoding for Qwen3.8 at 18:55 UTC that day (our MTP post). The Qwen3.8 thinking-on run started after that; the other three ran before. Its median time is not comparable with its thinking-off run.
Where Qwen3.8's thinking budget went
- Output: with thinking on, Qwen3.8 wrote 3.50 million output tokens, 98% of them thinking. The median attempt was 11,863 tokens. With thinking off: 389.
- The cap: 42 of its 247 attempts (17%) used the whole 32,768-token budget. 41 of them ended before any code; one stopped partway through a file and failed its tests.
- What they cost: those 42 attempts cost $2.75, 39% of the run. Each took 145 to 310 seconds (median 206).
- Which tasks: 27 of the 42 were first attempts. 30 tasks had at least one attempt at the cap; on 12 of them, both attempts hit it. 7 of the 30 were still solved.
- Task by task: with thinking on, Qwen3.8 solved 24 tasks it missed with thinking off, and missed 14 it had solved. On 10 of those 14, an attempt hit the cap.
- First try vs retry: the first-try score fell from 60.6% to 58.9%. The gain came on the retry: 35 tasks were solved by it, against 22 with thinking off.
Our harness keeps token counts and test results, not the reply text, so we can't say what those 42 attempts were thinking about. Three more attempts ended without code below the cap; we haven't looked into why.
Qwen3.6: a smaller gain for a smaller bill
With thinking on, Qwen3.6-35B-A3B solved 82.3% with one retry against 78.3% off: 7 more tasks, +4.0 points. The run cost $1.27 instead of $0.16 (8×), and the median task took 33.0 s instead of 3.2 s. Two of its 230 attempts hit the cap, both retries. The median attempt was 5,368 output tokens, 95% of them thinking.
Task by task, thinking on solved 14 tasks that thinking off missed and missed 7 it had solved.
Between the two thinking-on runs, Qwen3.6 solved 3.4 points more than Qwen3.8 for 18% of the cost, and its median task took less than half as long (33.0 s against 76.9 s, with MTP on Qwen3.8's side).
Against other models on the same 175 tasks
| Model | First try | One retry | Cost | Median time |
|---|---|---|---|---|
| Claude Opus 5.5 | 98.3% | 100% | $2.61 | 4.8 s |
| Claude Sonnet 5.5 | 93.7% | 100% | $1.48 | 3.9 s |
| GPT-6.2 Sol | 92.0% | 97.1% | $0.88 | 2.7 s |
| Qwen4-Max | 83.4% | 96.0% | $1.47 | 11.3 s |
| Kimi K2.6 | 86.9% | 94.9% | $0.57 | 9.4 s |
| Claude Haiku 5.5 | 72.0% | 93.7% | $0.15 | 5.5 s |
| Qwen3.6, thinking on (ours) | 68.6% | 82.3% | $1.27 | 33.0 s |
| Qwen3.8, thinking on (ours) | 58.9% | 78.9% | $7.02 | 76.9 s |
| Qwen3.6, thinking off (ours) | 68.0% | 78.3% | $0.16 | 3.2 s |
| Qwen3.8, thinking off (ours) | 60.6% | 73.1% | $0.27 | 5.5 s |
| Qwen3-Coder-30B (ours until Nov 19) | 64.0% | 64.6% | $0.06 | 3.7 s |
The first six rows are from our published CodeRouter benchmark (benchmark page, and the table in our Gemini 4 Neon post): same tasks, prompt and grading, but called through other providers on other days, at their list prices. Their times depend on those providers' load. Qwen3-Coder-30B ran on our API on November 12 with a 4,096-token output limit; we retired it on November 19.
- Qwen3.8 with thinking on cost more than Claude Opus 5.5: $7.02 against $2.61, 2.7×, and solved 21 points less (78.9% against 100%).
- Claude Haiku 5.5 solved 93.7% for $0.15, about what the cheapest of our four runs cost (Qwen3.6, thinking off, $0.16). Kimi K2.6 solved 94.9% for $0.57.
- Our best run, Qwen3.6 with thinking on, solved 82.3% for $1.27.
On these single-file tasks, both of our models solved fewer than every reference model in the table, Claude Haiku 5.5 and Kimi K2.6 included. Agent sessions are a different test. One with Claude Code is running, and it will be a separate post.
If you use these models on our API
- Both models think by default on our API. For coding with Qwen3.8, turn it off with
"reasoning_effort": "none". On these tasks that gave 73.1% for $0.27, against 78.9% for $7.02 with thinking on.
curl https://api.tokenwatt.io/v1/chat/completions \
-H "Authorization: Bearer $TOKENWATT_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "qwen/qwen3.8-27b", "reasoning_effort": "none",
"messages": [{"role": "user", "content": "..."}]}'
- If you want thinking on these tasks, Qwen3.6 did better for less: 82.3% for $1.27, 33.0 s per task at the median.
- If you keep Qwen3.8's thinking on, set
max_tokens. Our runs allowed 32,768; the API allows up to 65,536 output tokens, thinking included. A cap bounds what a request that thinks without answering can cost. It doesn't make that request answer. - Use Qwen's thinking-mode sampling (temperature 0.6, top_p 0.95, top_k 20), as we did, rather than temperature 0. Qwen's model card warns that greedy decoding can loop in thinking mode; in a 4-task probe before these runs, at temperature 0, one Qwen3.8 request thought until the cap.
- Not measured yet: the in-between effort levels (
low,medium) and smaller token caps. They may sit between the two Qwen3.8 runs here, or not.
How we measured
- Tasks: 175 single-file Python tasks, 141 from Exercism and 34 from Aider's polyglot hard set: the same 175 as our published benchmark, which leaves out
exercism/alphametics(one model's request errored on it in that matrix). The model gets the instructions and the stub file and returns the whole file. - Grading: the task's own tests, run on a throwaway rented Lambda instance (gpu_1x_a10, us-east-1), which also sent the requests. If they fail, the model gets the failing test output once and tries again.
- Calls: our public API,
https://api.tokenwatt.io/v1, streamed, 8 tasks in flight,max_tokens32,768 per request. - Thinking off:
reasoning_effort"none", temperature 0. The API reported zero reasoning tokens in both thinking-off runs. - Thinking on: no
reasoning_effortsent, so the model's default effort, with Qwen's recommended thinking-mode sampling: temperature 0.6, top_p 0.95, top_k 20. So the on and off runs differ in sampling as well as thinking. - Prices: our list prices per million tokens. Qwen3.8-27B: $0.06 input, $2.00 output, $0.03 cached input. Qwen3.6-35B-A3B: $0.10, $0.90, $0.08.
- Time: per task, the wall time of its request or requests as the eval machine saw it, retry included, tests not included.
- When: November 20, 2026. Qwen3.8 off about 18:09–18:13 UTC, Qwen3.6 off 18:13–18:16, Qwen3.6 on 18:16–18:38, Qwen3.8 on 19:11–20:01, after MTP went live at 18:55. Speculative decoding has the full model check every drafted token, so it changes how fast tokens come out, not which answers the model gives (apart from the small numeric differences we noted in the MTP post). We treat the pass rates as comparable and the Qwen3.8 thinking-on times as not.
Limits
- One run per configuration. A rerun could move a few tasks either way. The on/off gaps here are 7 tasks (Qwen3.6) and 10 (Qwen3.8).
- Public tasks. Exercism and Aider's set are public, so any of these models may have seen them in training.
- Single-file, single-turn tasks, with short prompts. Not agent sessions, multi-file repositories or tool use.
- Sampling differs between the thinking-on and thinking-off runs, as described above.
- Reference rows come from other providers on other days.
Every number is in the report. Per-task results (pass or fail, attempts, tokens, cost and time per attempt; no model output) are in the data file. Prices and setup for both models are on the API page.
Keep reading
MTP on Qwen3.8-27B: 2.65× faster per request, at the cost of 24% of the KV cache
Qwen3.8-27B ships with a one-layer multi-token prediction (MTP) head that vLLM can use as a draft model. On one H100 NVL with vLLM 0.33.0, three draft tokens per step raised single-request decode from 74.8 to 198 tokens per second (2.65×), and total throughput by 101%, 49% and 19% at 4, 16 and 48 requests in flight. GSM8K held (99.0% against 98.0%). The price is memory: room for the KV cache fell 24%, from 709,527 to 541,801 tokens. We turned it on in our API the same day, then off again that evening: with two models sharing the GPU under CUDA MPS, MTP crashed both engines under long-context load.
94% prefix-cache hits on hybrid models with real coding-agent traffic, against 56% in our synthetic test
A synthetic test we published on November 16 showed a hybrid model, Qwen3.8-27B, reusing far less of a shared prompt than a dense-attention model (56% against 89%), because vLLM caches its prompts only in whole blocks of 784 tokens. On two days of our own team's coding-agent traffic on our API, the same rule cost little: vLLM served 94.3% of Qwen3.8-27B's prompt tokens and 94.8% of Qwen3.6-35B-A3B's from cache, on median prompts of 114K and 86K tokens. The median request recomputed only 1,562 to 2,690 tokens.
Qwen3 Coder 30B in opencode, Qwen Code, Cline and Kilo Code for $0.07 per million input tokens
How to point opencode, Qwen Code, Cline and Kilo Code at Qwen3-Coder-30B-A3B-Instruct on TokenWatt's OpenAI-compatible API, and what to expect from a 30B coding model: 64.8% on our 176 single-file Python tasks, for $0.057 for the whole suite.