← Lab notes

The pi adapter for Qwen3.8-27B on FP8: 3× fewer turns held up, the accuracy gain was smaller

harness-hub's pi adapter is a LoRA that teaches Qwen3.8-27B to work inside the pi coding agent. Its card reports 39.5% → 55.8% on Terminal-Bench 2.1 on BF16 weights. We serve the FP8 checkpoint, so we tested it there: on 25 Terminal-Bench tasks under pi, the adapter solved 17 and the base model 14, a gap too small to call on 25 tasks. Its efficiency held clearly: on the 12 tasks both solved, it needed 9.5 turns against 28.5 and 1,432 output tokens against 5,254 (medians). It also ran 10 minutes without a fault under the long-context load that crashed MTP on our shared GPU. It's on our API now at Qwen3.8's price.

Choosing GPUs? Compare GPUs for your traffic · Get your node measured, free

On 25 Terminal-Bench tasks under the pi coding agent, harness-hub's pi adapter for Qwen3.8-27B solved 17 on our FP8 engine and the base model 14. Where both solved a task, the adapter needed 9.5 turns instead of 28.5 and 1,432 output tokens instead of 5,254 (medians).

The adapter's card promises more on accuracy: 39.5% → 55.8% on Terminal-Bench 2.1, measured on BF16 weights. Our run doesn't confirm a gain that large, and 25 tasks are too few to rule it out. The efficiency gain held clearly.

It's on our API from today as harness-hub/qwen3.8-27b-pi-adapted, at Qwen3.8-27B's price: $0.06 per million input tokens, $0.03 cached, $2.00 output.

What the adapter is

harness-hub/Qwen3.8-27B-pi-adapted is an Apache-2.0 LoRA (rank 32), trained with reinforcement learning for the pi coding agent with thinking off. Its card reports, under pi, on BF16 weights:

  • SWE-bench Pro (hard 51): 39.2% → 64.7%.
  • Terminal-Bench 2.1: 39.5% → 55.8%.
  • Work per task: under half the turns of the base model on tasks both solve.

We serve Qwen3.8-27B in FP8 on a shared H200. Three things had to hold before we could offer the adapter:

  1. vLLM can run it on the FP8 checkpoint.
  2. It keeps its gain there.
  3. It doesn't destabilize the shared GPU.

1. Does it run on FP8? Yes, at a speed cost

On one H100 NVL with vLLM 0.33.0, --enable-lora on the FP8 checkpoint served the base model and the adapter side by side from one engine. Tool calls passed 20/20.

Restricting LoRA to the attention and MLP layers (--lora-target-modules qkv_proj o_proj gate_up_proj down_proj) halved the overhead and lost nothing we could measure. These adapters don't train the Gated DeltaNet projections. With that filter:

  • One adapter request: about 14% slower to decode than the base model on the H100 NVL. On our H200 it was 20% slower: 72.5 against 91.0 tokens per second.
  • Mixed traffic: base and adapter requests in the same batch cost about 10% of total throughput at 16 in flight. Batches with base requests only run at full speed.
  • Memory: the KV cache shrank 5.4% on our H200, from 668,668 to 632,371 tokens. That's still above the two full 262K-token contexts our engine needs to start.

One caveat: adapter outputs weren't bit-reproducible from one request to the next, even at temperature 0. vLLM's LoRA kernels sum partial results in a varying order. Base-model outputs were.

2. Does it keep its gain on FP8? The efficiency, yes

We ran Terminal-Bench 2.1 the way the card did: - harness: Harbor with pi 1.5.2, the openai-completions API; - thinking off, temperature 0.8, top-p 1.0; - 100 turns and 60 minutes per task, one attempt.

The only change was the weights: our FP8 checkpoint instead of BF16.

To spend the budget where a lost gain would show, we chose 25 of the 89 tasks: - 17 tasks where the card's BF16 runs of the base model and the adapter differed; - 8 tasks both solved, drawn at random.

Tasks HF, BF16: base HF, BF16: adapter Ours, FP8: base Ours, FP8: adapter
15 the adapter gained in BF16 0 15 7 10
2 the adapter lost in BF16 2 0 1 1
8 both solved in BF16 8 8 6 6
All 25 10 23 14 17
  • The BF16 columns are not a baseline. We picked the tasks because one BF16 attempt of each model happened to differ, and a rerun at temperature 0.8 moves toward the average on any weights. That is why our base model solved 7 of the 15 "gained" tasks where HF's single BF16 attempt solved none.
  • Accuracy: the adapter solved 5 tasks the base model didn't, and missed 2 the base solved. On 7 split tasks, that's not a significant difference (sign test p = 0.45).
  • The two misses:
  • in one, the adapter wrote its next command as a markdown block instead of a tool call, and pi ended the session;
  • in the other, it wrote the file and stopped without running the tests.

Efficiency held clearly. On the 12 tasks both models solved:

Median per solved task Base model Adapter
Turns 28.5 9.5
Output tokens 5,254 1,432
Agent time 5.8 min 6.6 min
  • The base model ran into the 100-turn cap in 9 of its 25 attempts (it still solved 2 of those). The adapter hit the cap once.
  • Fewer turns means fewer requests, and each request resends the session so far. So the adapter also resends it fewer times per task (we didn't total the prompt tokens).
  • Agent time stays about the same because the adapter spends its turns on longer commands.

How the run went. The first pass ran 26 sessions at once on one H100 NVL and filled the KV cache (98.6% used). Once it was full, vLLM evicted cached prefixes and prefix-cache hits fell to 70.5%. The run hit its time limit with 17 trials unfinished. We reran those 17, plus 2 cancelled at the stop, with the same settings and 19 sessions at once: the KV cache peaked at 45% and hits were 95.3%. Each (task, model) pair still has exactly one attempt.

3. Is it safe on our shared GPU? It ran clean

On November 20, MTP speculative decoding passed every test we gave it on a single-engine GPU, then crashed both engines on our shared H200. That machine runs Qwen3.8 and Qwen3.6 side by side under CUDA MPS. We wrote that up the same day. LoRA also adds its own GPU kernels, so this time we tested in the exact shape we run.

We rented a separate H200 and set it up as production: the same engine flags, plus LoRA on Qwen3.8. Then we ran the crash load: six concurrent coding-agent sessions per model, prompts growing to about 100K tokens, with half of Qwen3.8's sessions on the adapter.

  • Result: 10 minutes without a fault. With MTP, the same layout had failed within one to two minutes.
  • Tool calls: the base model and the adapter each passed 20/20, through the tool-call parser we run in production.
  • pi itself: pointed at the adapter, pi fixed a two-bug Python task in 10 seconds.

On our API

We added the adapter to the Qwen3.8 engine today; the restart took about 7 minutes. Measured through the public API afterwards:

Qwen3.8-27B pi adapter
One request, thinking off 91.4 tok/s 69.8 tok/s
Tool calls, 20-case check 20/20 20/20, 20/20 and 19/20 (one format slip in 60)
Thinking on by default off by default
  • Thinking: the adapter was trained without thinking, so our gateway turns it off unless you ask. Set reasoning_effort or chat_template_kwargs.enable_thinking to choose yourself.
  • Price and context: the same as Qwen3.8-27B, with a 262,144-token context. The two models share one engine.

To use it with pi, add our endpoint to ~/.pi/agent/models.json:

{
  "providers": {
    "tokenwatt": {
      "baseUrl": "https://api.tokenwatt.io/v1",
      "api": "openai-completions",
      "apiKey": "YOUR_TOKENWATT_KEY",
      "models": [{ "id": "harness-hub/qwen3.8-27b-pi-adapted", "contextWindow": 262144, "maxTokens": 8192 }]
    }
  }
}
pi --provider tokenwatt --model harness-hub/qwen3.8-27b-pi-adapted

"api": "openai-completions" matters. The adapter was trained and evaluated through that API; without it, pi uses the Responses API.

Limits

  • Sample size: 25 tasks, one attempt each. A 3-task difference is within noise, and the tasks were selected rather than random.
  • No BF16 control: we compared against HF's BF16 grades, not BF16 on our own stack, so FP8 drift and run-to-run noise can't be separated.
  • Different loads in the two passes: the rerun ran with fewer sessions at once, so it got more turns per hour than the first pass. Timeouts are wall-clock, so that helps whichever model needs more turns, which is the base model.
  • Not tested: the opencode and Hermes adapters from the same authors, and SWE-bench Pro.
  • Short stability run: 10 minutes without a fault is strong evidence against the failure rate MTP showed, not proof that it never fails.

The three tests cost $15.24 in rented GPUs. The numbers are in the report.

Disclosure: we run TokenWatt and serve this adapter on our API. The adapter is harness-hub's work, under Apache-2.0.

Keep reading

8 min read

MTP on Qwen3.8-27B: 2.65× faster per request, at the cost of 24% of the KV cache

Qwen3.8-27B ships with a one-layer multi-token prediction (MTP) head that vLLM can use as a draft model. On one H100 NVL with vLLM 0.33.0, three draft tokens per step raised single-request decode from 74.8 to 198 tokens per second (2.65×), and total throughput by 101%, 49% and 19% at 4, 16 and 48 requests in flight. GSM8K held (99.0% against 98.0%). The price is memory: room for the KV cache fell 24%, from 709,527 to 541,801 tokens. We turned it on in our API the same day, then off again that evening: with two models sharing the GPU under CUDA MPS, MTP crashed both engines under long-context load.

speculative decodingMTPmulti-token prediction
6 min read

Qwen3.8-27B with thinking on: 26× the cost for 5.7 more points on our 175 coding tasks

We ran the two models on our own API, Qwen3.8-27B and Qwen3.6-35B-A3B, through the 175 single-file Python tasks of our CodeRouter benchmark, with thinking off and on. Thinking raised Qwen3.8 from 73.1% to 78.9% solved with one retry, and the cost of the run from $0.27 to $7.02; 42 of its 247 attempts used the whole 32,768-token budget, and 41 of those never wrote code. Qwen3.6 went from 78.3% to 82.3% for $0.16 → $1.27. On these tasks both trail Claude Haiku 5.5 (93.7%, $0.15) and Kimi K2.6 (94.9%, $0.57).

Qwen3.8Qwen3.6thinking mode
5 min read

94% prefix-cache hits on hybrid models with real coding-agent traffic, against 56% in our synthetic test

A synthetic test we published on November 16 showed a hybrid model, Qwen3.8-27B, reusing far less of a shared prompt than a dense-attention model (56% against 89%), because vLLM caches its prompts only in whole blocks of 784 tokens. On two days of our own team's coding-agent traffic on our API, the same rule cost little: vLLM served 94.3% of Qwen3.8-27B's prompt tokens and 94.8% of Qwen3.6-35B-A3B's from cache, on median prompts of 114K and 86K tokens. The median request recomputed only 1,562 to 2,690 tokens.

prefix cachingprompt cachehybrid attention