← All reports

Report · 2026-11-20

Qwen3.8-27B and Qwen3.6-35B-A3B on our API, 175 single-file Python tasks: thinking on vs off

1× NVIDIA H200Qwen3.8-27B-FP8 + Qwen3.6-35B-A3B-FP8FP8 weightsvLLM 0.33.0Coding eval · 175 tasksOur own API

Run on 2026-11-20 through our public API, api.tokenwatt.io (streamed chat completions, max_tokens 32,768, 8 in flight), with the harness of our published CodeRouter benchmark: 175 single-file Python tasks (Exercism plus Aider's hard set); the model returns the whole file, the task's tests run in a rented Lambda sandbox, and a failure gets one retry with the test output. Thinking off: reasoning_effort "none", temperature 0. Thinking on: the model's default effort, with Qwen's recommended sampling (temperature 0.6, top_p 0.95, top_k 20). Cost = reported tokens × our list prices, not our GPU cost. The Qwen3.8 thinking-on run came after we enabled MTP speculative decoding (18:55 UTC), the other three before it, so its times are not comparable. One run per configuration. Write-up: /blog/qwen38-qwen36-thinking-on-off-coding-benchmark.html

ConclusionWith thinking on, Qwen3.8-27B solved 78.9% with one retry against 73.1% off (+5.7 points), for $7.02 instead of $0.27 (26×); 42 of its 247 attempts (17%) hit the 32,768-token cap. Qwen3.6-35B-A3B went from 78.3% to 82.3% for $0.16 → $1.27. On the same tasks Claude Haiku 5.5 solved 93.7% for $0.15 and Kimi K2.6 94.9% for $0.57.

Results

Qwen3.6 = Qwen3.6-35B-A3B, Qwen3.8 = Qwen3.8-27B; off / on = thinking. One run per configuration, 175 tasks.

Solved with one retry

% of 175 tasks · higher is better
Qwen3.6, off78.3%Qwen3.6, off: 78.3 % of 175 tasks
Qwen3.6, on82.3%Qwen3.6, on: 82.3 % of 175 tasks
Qwen3.8, off73.1%Qwen3.8, off: 73.1 % of 175 tasks
Qwen3.8, on78.9%Qwen3.8, on: 78.9 % of 175 tasks

Axis runs to 100%.

Solved on the first try

% of 175 tasks · higher is better
Qwen3.6, off68.0%Qwen3.6, off: 68.0 % of 175 tasks
Qwen3.6, on68.6%Qwen3.6, on: 68.6 % of 175 tasks
Qwen3.8, off60.6%Qwen3.8, off: 60.6 % of 175 tasks
Qwen3.8, on58.9%Qwen3.8, on: 58.9 % of 175 tasks

Axis runs to 100%.

Cost of all 175 tasks

US$ at our list prices, retries included · lower is better
Qwen3.6, off0.16Qwen3.6, off: 0.16 US$ at our list prices, retries included
Qwen3.6, on1.27Qwen3.6, on: 1.27 US$ at our list prices, retries included
Qwen3.8, off0.27Qwen3.8, off: 0.27 US$ at our list prices, retries included
Qwen3.8, on7.02Qwen3.8, on: 7.02 US$ at our list prices, retries included

Median time per task

seconds, retry included · lower is better
Qwen3.6, off3.2 sQwen3.6, off: 3.2 seconds, retry included
Qwen3.6, on33.0 sQwen3.6, on: 33.0 seconds, retry included
Qwen3.8, off5.5 sQwen3.8, off: 5.5 seconds, retry included
Qwen3.8, on76.9 sQwen3.8, on: 76.9 seconds, retry included

Qwen3.8 thinking on ran after MTP speculative decoding was enabled; the other three before.

Attempts that hit the 32,768-token cap

% of attempts · lower is better
Qwen3.6, off0.0%Qwen3.6, off: 0.0 % of attempts
Qwen3.6, on0.9%Qwen3.6, on: 0.9 % of attempts
Qwen3.8, off0.0%Qwen3.8, off: 0.0 % of attempts
Qwen3.8, on17.0%Qwen3.8, on: 17.0 % of attempts

All four runs

Qwen3.6 = Qwen3.6-35B-A3B, Qwen3.8 = Qwen3.8-27B. 175 tasks; cost is for all 175, retries included; per solved task = cost ÷ tasks solved with one retry; median time is per task, retry included.

RunFirst tryOne retryCostPer solved taskMedian time
Qwen3.6, thinking off68.0%78.3%$0.16$0.00113.2 s
Qwen3.6, thinking on68.6%82.3%$1.27$0.008833.0 s
Qwen3.8, thinking off60.6%73.1%$0.27$0.00215.5 s
Qwen3.8, thinking on58.9%78.9%$7.02$0.050876.9 s

Where the tokens went

Attempts = first tries plus retries. "No code" = the reply had no Python code block, so no test ran: for Qwen3.8 thinking on, 41 of the 42 capped attempts plus 3 that ended below the cap (cause not investigated; the harness doesn't keep reply text). At cap = output reached max_tokens (32,768). Thinking = reasoning tokens ÷ output tokens, as the API reported them. Median / attempt = median output tokens per attempt. Capped cost = what the attempts at the cap cost.

RunAttemptsAt capNo codeOutput tokensThinkingMedian / attemptCapped cost
Qwen3.6, thinking off2310 (0.0%)00.15M0%473$0.00
Qwen3.6, thinking on2302 (0.9%)21.39M95%5,368$0.06
Qwen3.8, thinking off2440 (0.0%)00.13M0%389$0.00
Qwen3.8, thinking on24742 (17.0%)443.50M98%11,863$2.75

Thinking on vs off, task by task

Tasks solved with one retry. Of the 14 tasks Qwen3.8 solved only with thinking off, 10 had a thinking-on attempt that hit the cap.

ModelOnly with thinking onOnly with thinking offBothNeither
Qwen3.6-35B-A3B14713024
Qwen3.8-27B241411423

By task set

Tasks solved on the first try / with one retry.

RunExercism (141)Aider's hard set (34)
Qwen3.6, thinking off107 / 11912 / 18
Qwen3.6, thinking on111 / 1269 / 18
Qwen3.8, thinking off97 / 1139 / 15
Qwen3.8, thinking on95 / 1218 / 17

Context: other models on the same 175 tasks

Reference rows from our published CodeRouter benchmark (coderouter.io/benchmark and the 2026-11-19 Gemini 4 Neon post): same tasks, prompt and grading; called through other providers on other days, priced at their list prices; times depend on each provider's load. Qwen3-Coder-30B ran on our API on 2026-11-12 with max_tokens 4,096 and was retired on 2026-11-19.

ModelFirst tryOne retryCostMedian time
Claude Opus 5.598.3%100.0%$2.614.8 s
Claude Sonnet 5.593.7%100.0%$1.483.9 s
GPT-6.2 Sol92.0%97.1%$0.882.7 s
Qwen4-Max83.4%96.0%$1.4711.3 s
Kimi K2.686.9%94.9%$0.579.4 s
Claude Haiku 5.572.0%93.7%$0.155.5 s
Qwen3.6, thinking on (ours)68.6%82.3%$1.2733.0 s
Qwen3.8, thinking on (ours)58.9%78.9%$7.0276.9 s
Qwen3.6, thinking off (ours)68.0%78.3%$0.163.2 s
Qwen3.8, thinking off (ours)60.6%73.1%$0.275.5 s
Qwen3-Coder-30B (ours until Nov 19)64.0%64.6%$0.063.7 s

Setup

Any change to these fields makes it a different report.

Endpoint
https://api.tokenwatt.io/v1/chat/completions, streamed (include_usage), our public API
Models
qwen/qwen3.8-27b (Qwen/Qwen3.8-27B-FP8), qwen/qwen3.6-35b-a3b (Qwen/Qwen3.6-35B-A3B-FP8)
Serving
1× H200, two vLLM 0.33.0 engines under CUDA MPS; MTP (3 draft tokens) on Qwen3.8 from 2026-11-20 18:55 UTC
Tasks
175: Exercism Python (141) + Aider's polyglot hard set, Python (34); exercism/alphametics left out as in the published benchmark
Prompt
system: return the complete file in one python code block; user: instructions + stub file (bench/coderouter-eval/run_eval.py)
Grading
the task's own tests (pytest, 60 s), run in a rented Lambda gpu_1x_a10 instance; one retry with the failing output
Thinking off
reasoning_effort "none", temperature 0
Thinking on
no reasoning_effort (model default), temperature 0.6, top_p 0.95, top_k 20 (Qwen's recommended thinking-mode sampling)
Output budget
max_tokens 32,768 per request, thinking included
Concurrency
8 tasks in flight
Prices
Qwen3.8: $0.06 in / $2.00 out / $0.03 cached input per million; Qwen3.6: $0.10 / $0.90 / $0.08 (our list prices)
Cost
Σ reported tokens × list price, retries included; thinking billed as output
Median time
per task, wall time of the request(s), retry included; test time not counted
When
2026-11-20: Qwen3.8 off ~18:09–18:13, Qwen3.6 off ~18:13–18:16, Qwen3.6 on ~18:16–18:38, Qwen3.8 on ~19:11–20:01 UTC

Not covered

Do not extrapolate this report to the following.

  • Repeat runs: one run per configuration; a rerun could move a few tasks either way
  • reasoning_effort low or medium, and output budgets other than 32,768 tokens
  • Thinking on at temperature 0, or thinking off with Qwen's sampling: sampling differs between the on and off runs
  • Qwen3.8 thinking-on latency without MTP (its run came after MTP was enabled)
  • Agent sessions, multi-file repositories, long contexts or tool use: these are single-file, single-turn tasks
  • Tasks unseen in training: the tasks are public
  • Reply text: the harness keeps no model output, so why an attempt ran to the cap was not examined

Reproduce

Harness and raw per-attempt records: bench/coderouter-eval/ in the TokenWatt repository (run_eval.py, lambda_eval_multi.py, results/T-tokenwatt-qwen38-36/). Run model-written code only in a sandbox.

shell
# one configuration (thinking on, Qwen3.8); tests run where this runs, so use a throwaway machine
python3 run_eval.py --tasks tasks/exercism tasks/polyglot \
  --base-url https://api.tokenwatt.io/v1 --key-env TOKENWATT_API_KEY \
  --model qwen/qwen3.8-27b --label T-qwen38-think --prices prices-tokenwatt-qwen38-36.json \
  --concurrency 8 --max-tokens 32768 --stream --timeout 1500 \
  --extra-body '{"temperature":0.6,"top_p":0.95,"top_k":20}'

# thinking off
  ... --label T-qwen38-nothink --extra-body '{"reasoning_effort":"none"}'   # harness default temperature 0

# 175-task summaries, this report and the per-task data (exercism/alphametics left out)
python3 bench/coderouter-eval/results/T-tokenwatt-qwen38-36/make_site_report.py

Read the write-up Download raw JSON