Qwen3.8-27B and Qwen3.6-35B-A3B on our API, 175 single-file Python tasks: thinking on vs off
Run on 2026-11-20 through our public API, api.tokenwatt.io (streamed chat completions, max_tokens 32,768, 8 in flight), with the harness of our published CodeRouter benchmark: 175 single-file Python tasks (Exercism plus Aider's hard set); the model returns the whole file, the task's tests run in a rented Lambda sandbox, and a failure gets one retry with the test output. Thinking off: reasoning_effort "none", temperature 0. Thinking on: the model's default effort, with Qwen's recommended sampling (temperature 0.6, top_p 0.95, top_k 20). Cost = reported tokens × our list prices, not our GPU cost. The Qwen3.8 thinking-on run came after we enabled MTP speculative decoding (18:55 UTC), the other three before it, so its times are not comparable. One run per configuration. Write-up: /blog/qwen38-qwen36-thinking-on-off-coding-benchmark.html
Results
Qwen3.6 = Qwen3.6-35B-A3B, Qwen3.8 = Qwen3.8-27B; off / on = thinking. One run per configuration, 175 tasks.
Solved with one retry
Axis runs to 100%.
Solved on the first try
Axis runs to 100%.
Cost of all 175 tasks
Median time per task
Qwen3.8 thinking on ran after MTP speculative decoding was enabled; the other three before.
Attempts that hit the 32,768-token cap
All four runs
Qwen3.6 = Qwen3.6-35B-A3B, Qwen3.8 = Qwen3.8-27B. 175 tasks; cost is for all 175, retries included; per solved task = cost ÷ tasks solved with one retry; median time is per task, retry included.
| Run | First try | One retry | Cost | Per solved task | Median time |
|---|---|---|---|---|---|
| Qwen3.6, thinking off | 68.0% | 78.3% | $0.16 | $0.0011 | 3.2 s |
| Qwen3.6, thinking on | 68.6% | 82.3% | $1.27 | $0.0088 | 33.0 s |
| Qwen3.8, thinking off | 60.6% | 73.1% | $0.27 | $0.0021 | 5.5 s |
| Qwen3.8, thinking on | 58.9% | 78.9% | $7.02 | $0.0508 | 76.9 s |
Where the tokens went
Attempts = first tries plus retries. "No code" = the reply had no Python code block, so no test ran: for Qwen3.8 thinking on, 41 of the 42 capped attempts plus 3 that ended below the cap (cause not investigated; the harness doesn't keep reply text). At cap = output reached max_tokens (32,768). Thinking = reasoning tokens ÷ output tokens, as the API reported them. Median / attempt = median output tokens per attempt. Capped cost = what the attempts at the cap cost.
| Run | Attempts | At cap | No code | Output tokens | Thinking | Median / attempt | Capped cost |
|---|---|---|---|---|---|---|---|
| Qwen3.6, thinking off | 231 | 0 (0.0%) | 0 | 0.15M | 0% | 473 | $0.00 |
| Qwen3.6, thinking on | 230 | 2 (0.9%) | 2 | 1.39M | 95% | 5,368 | $0.06 |
| Qwen3.8, thinking off | 244 | 0 (0.0%) | 0 | 0.13M | 0% | 389 | $0.00 |
| Qwen3.8, thinking on | 247 | 42 (17.0%) | 44 | 3.50M | 98% | 11,863 | $2.75 |
Thinking on vs off, task by task
Tasks solved with one retry. Of the 14 tasks Qwen3.8 solved only with thinking off, 10 had a thinking-on attempt that hit the cap.
| Model | Only with thinking on | Only with thinking off | Both | Neither |
|---|---|---|---|---|
| Qwen3.6-35B-A3B | 14 | 7 | 130 | 24 |
| Qwen3.8-27B | 24 | 14 | 114 | 23 |
By task set
Tasks solved on the first try / with one retry.
| Run | Exercism (141) | Aider's hard set (34) |
|---|---|---|
| Qwen3.6, thinking off | 107 / 119 | 12 / 18 |
| Qwen3.6, thinking on | 111 / 126 | 9 / 18 |
| Qwen3.8, thinking off | 97 / 113 | 9 / 15 |
| Qwen3.8, thinking on | 95 / 121 | 8 / 17 |
Context: other models on the same 175 tasks
Reference rows from our published CodeRouter benchmark (coderouter.io/benchmark and the 2026-11-19 Gemini 4 Neon post): same tasks, prompt and grading; called through other providers on other days, priced at their list prices; times depend on each provider's load. Qwen3-Coder-30B ran on our API on 2026-11-12 with max_tokens 4,096 and was retired on 2026-11-19.
| Model | First try | One retry | Cost | Median time |
|---|---|---|---|---|
| Claude Opus 5.5 | 98.3% | 100.0% | $2.61 | 4.8 s |
| Claude Sonnet 5.5 | 93.7% | 100.0% | $1.48 | 3.9 s |
| GPT-6.2 Sol | 92.0% | 97.1% | $0.88 | 2.7 s |
| Qwen4-Max | 83.4% | 96.0% | $1.47 | 11.3 s |
| Kimi K2.6 | 86.9% | 94.9% | $0.57 | 9.4 s |
| Claude Haiku 5.5 | 72.0% | 93.7% | $0.15 | 5.5 s |
| Qwen3.6, thinking on (ours) | 68.6% | 82.3% | $1.27 | 33.0 s |
| Qwen3.8, thinking on (ours) | 58.9% | 78.9% | $7.02 | 76.9 s |
| Qwen3.6, thinking off (ours) | 68.0% | 78.3% | $0.16 | 3.2 s |
| Qwen3.8, thinking off (ours) | 60.6% | 73.1% | $0.27 | 5.5 s |
| Qwen3-Coder-30B (ours until Nov 19) | 64.0% | 64.6% | $0.06 | 3.7 s |
Setup
Any change to these fields makes it a different report.
- Endpoint
- https://api.tokenwatt.io/v1/chat/completions, streamed (include_usage), our public API
- Models
- qwen/qwen3.8-27b (Qwen/Qwen3.8-27B-FP8), qwen/qwen3.6-35b-a3b (Qwen/Qwen3.6-35B-A3B-FP8)
- Serving
- 1× H200, two vLLM 0.33.0 engines under CUDA MPS; MTP (3 draft tokens) on Qwen3.8 from 2026-11-20 18:55 UTC
- Tasks
- 175: Exercism Python (141) + Aider's polyglot hard set, Python (34); exercism/alphametics left out as in the published benchmark
- Prompt
- system: return the complete file in one python code block; user: instructions + stub file (bench/coderouter-eval/run_eval.py)
- Grading
- the task's own tests (pytest, 60 s), run in a rented Lambda gpu_1x_a10 instance; one retry with the failing output
- Thinking off
- reasoning_effort "none", temperature 0
- Thinking on
- no reasoning_effort (model default), temperature 0.6, top_p 0.95, top_k 20 (Qwen's recommended thinking-mode sampling)
- Output budget
- max_tokens 32,768 per request, thinking included
- Concurrency
- 8 tasks in flight
- Prices
- Qwen3.8: $0.06 in / $2.00 out / $0.03 cached input per million; Qwen3.6: $0.10 / $0.90 / $0.08 (our list prices)
- Cost
- Σ reported tokens × list price, retries included; thinking billed as output
- Median time
- per task, wall time of the request(s), retry included; test time not counted
- When
- 2026-11-20: Qwen3.8 off ~18:09–18:13, Qwen3.6 off ~18:13–18:16, Qwen3.6 on ~18:16–18:38, Qwen3.8 on ~19:11–20:01 UTC
Not covered
Do not extrapolate this report to the following.
- Repeat runs: one run per configuration; a rerun could move a few tasks either way
- reasoning_effort low or medium, and output budgets other than 32,768 tokens
- Thinking on at temperature 0, or thinking off with Qwen's sampling: sampling differs between the on and off runs
- Qwen3.8 thinking-on latency without MTP (its run came after MTP was enabled)
- Agent sessions, multi-file repositories, long contexts or tool use: these are single-file, single-turn tasks
- Tasks unseen in training: the tasks are public
- Reply text: the harness keeps no model output, so why an attempt ran to the cap was not examined
Reproduce
Harness and raw per-attempt records: bench/coderouter-eval/ in the TokenWatt repository (run_eval.py, lambda_eval_multi.py, results/T-tokenwatt-qwen38-36/). Run model-written code only in a sandbox.
# one configuration (thinking on, Qwen3.8); tests run where this runs, so use a throwaway machine
python3 run_eval.py --tasks tasks/exercism tasks/polyglot \
--base-url https://api.tokenwatt.io/v1 --key-env TOKENWATT_API_KEY \
--model qwen/qwen3.8-27b --label T-qwen38-think --prices prices-tokenwatt-qwen38-36.json \
--concurrency 8 --max-tokens 32768 --stream --timeout 1500 \
--extra-body '{"temperature":0.6,"top_p":0.95,"top_k":20}'
# thinking off
... --label T-qwen38-nothink --extra-body '{"reasoning_effort":"none"}' # harness default temperature 0
# 175-task summaries, this report and the per-task data (exercism/alphametics left out)
python3 bench/coderouter-eval/results/T-tokenwatt-qwen38-36/make_site_report.py