← All reports

Report · 2026-11-21

harness-hub's pi adapter (LoRA) on Qwen3.8-27B FP8, vLLM 0.33.0: Terminal-Bench 2.1 subset under pi, cost and stability on a shared H200

1× NVIDIA H100 NVL / H200Qwen3.8-27B-FP8 + harness-hub/Qwen3.8-27B-pi-adaptedFP8 weights, BF16 KV cache; LoRA r32 on attention + MLPvLLM 0.33.0LoRA adapter · pi coding agentTerminal-Bench 2.1 · 25 tasks

Three tests, 2026-11-20/21, on rented GPUs (RunPod Secure Cloud): (1) LoRA on the FP8 checkpoint, one H100 NVL, speed and memory; (2) 25 Terminal-Bench 2.1 tasks under pi 1.5.2 (Harbor 0.26.0), base model and adapter served by one FP8 engine on an H100 NVL, the tasks run on a Lambda host, thinking off, temperature 0.8, top-p 1.0, 100 turns and 60 min per task, one attempt; (3) our production H200 layout (Qwen3.8 at 52% of the GPU with LoRA + Qwen3.6 at 43%, CUDA MPS, 262K context) under the long-context agent load that crashed MTP on 11-20. Model Qwen/Qwen3.8-27B-FP8 (revision c3a462cc); adapter harness-hub/Qwen3.8-27B-pi-adapted (3cad999e, Apache-2.0, LoRA r32). Write-up: /blog/qwen38-27b-pi-adapter-fp8-terminal-bench.html

ConclusionOn 25 Terminal-Bench 2.1 tasks under pi, the adapter solved 17 and the base model 14 on the FP8 checkpoint (5 tasks solved only by the adapter, 2 only by the base: sign test p = 0.45). On the 12 tasks both solved, the adapter needed 9.5 turns against 28.5 and 1,432 output tokens against 5,254 (medians). One adapter request decodes about 20% slower than the base model (72.5 vs 91.0 tok/s on the H200). The production layout with LoRA ran the 11-20 crash load for 10 minutes without a fault.

Results

FP8 checkpoint, our runs; HF's BF16 grades for the same tasks in the table below.

Median turns per solved task

turns, 12 tasks both solved · lower is better
Base model28.5Base model: 28.5 turns, 12 tasks both solved
pi adapter9.5pi adapter: 9.5 turns, 12 tasks both solved

Median output tokens per solved task

tokens, 12 tasks both solved · lower is better
Base model5,254Base model: 5,254 tokens, 12 tasks both solved
pi adapter1,432pi adapter: 1,432 tokens, 12 tasks both solved

Tasks solved of 25

tasks · higher is better
Base model14Base model: 14 tasks
pi adapter17pi adapter: 17 tasks

Terminal-Bench 2.1, 25 tasks under pi, one attempt each

The tasks were selected: the 17 where HF's single BF16 attempts of the two models differed (path-tracing left out: both timed out) and 8 of the 27 light tasks both solved, drawn at random. A rerun at temperature 0.8 regresses toward the mean on any weights, so the BF16 columns are not a baseline.

TasksHF BF16, baseHF BF16, adapterFP8, baseFP8, adapter
15 the adapter gained in BF16015710
2 the adapter lost in BF162011
8 both solved in BF16 (random)8866
All 2510231417

Work per solved task (12 tasks both models solved on FP8)

Medians. Agent time includes the commands the agent ran.

Base modelpi adapter
Turns28.59.5
Output tokens5,2541,432
Agent time5.8 min6.6 min
Attempts at the 100-turn cap (of 25)91

Cost of serving the adapter

LoRA restricted to the attention + MLP layers (--lora-target-modules qkv_proj o_proj gate_up_proj down_proj).

MeasureWithout LoRAWith LoRA
One request, H100 NVL, decode tok/s75.8base 76.4 · adapter 65.3 (-13.8%)
One request, our H200, decode tok/s91.4 (public API)base 91.0 · adapter 72.5 (-20%)
Mixed base + adapter traffic, 16 in flight, H100 NVL931 tok/s834 tok/s (-10.5%)
Qwen3.8 KV cache on our H200 (0.52 of the GPU)668,668 tokens632,371 tokens (-5.4%)

Our production H200 layout under the 11-20 crash load

6 + 6 concurrent coding-agent sessions, prompts growing to ~100-125K tokens, CUDA MPS, 10 minutes.

ConfigurationResult
Qwen3.8 0.52 + LoRA (half its sessions on the adapter), Qwen3.6 0.43no fault in 10 min; Qwen3.8 108 requests, Qwen3.6 863
Qwen3.8 0.52, Qwen3.6 0.43, no LoRA (11-20)no fault in 10 min
Qwen3.8 0.57 + MTP, Qwen3.6 0.38 (11-20)illegal memory access at 1.0 and 2.1 min

Setup

Any change to these fields makes it a different report.

Adapter
harness-hub/Qwen3.8-27B-pi-adapted at 3cad999e: LoRA r32, alpha 64, q/k/v/o (16 full-attention layers) + gate/up/down; trained with RL for pi, thinking off, on BF16 weights
Engine
vllm serve Qwen/Qwen3.8-27B-FP8 --enable-lora --max-lora-rank 32 --lora-target-modules qkv_proj o_proj gate_up_proj down_proj --lora-modules qwen3.8-27b-pi=<dir>
Terminal-Bench
Harbor 0.26.0, agent pi 1.5.2 (the version in HF's trajectories), openai-completions API, context 131,072, max 8,192 tokens per call, 100 turns, 3,600 s per task, one attempt; tool parser qwen3_xml, thinking off, temperature 0.8, top-p 1.0
Two passes
first pass: 26 trials at once on one H100 NVL, cut at its time limit with 17 trials running; rerun of those 17 and 2 cancelled ones, 19 at once, same settings. Each (task, model) has one graded attempt
Production-layout run
separate H200 SXM, production's exact engine flags plus LoRA, CUDA MPS, the 11-20 load generator; then tool calls (20 cases), single-request speed and one pi run on the same engines
Our API
since 2026-11-21 19:29 UTC: harness-hub/qwen3.8-27b-pi-adapted, Qwen3.8-27B's price, thinking off by default (the caller's choice wins)

Not covered

Do not extrapolate this report to the following.

  • A BF16 control on our own stack: FP8 drift and run-to-run noise are not separated
  • More than one attempt per task, or all 89 Terminal-Bench tasks
  • SWE-bench Pro, and the opencode and Hermes adapters
  • Capacity under a latency target with adapter traffic on the H200
  • Stability beyond the 10-minute run

Reproduce

Serve the adapter next to the base model and point pi at it; the Terminal-Bench tasks come from harborframework/terminal-bench-2.1 on Hugging Face.

shell
vllm serve Qwen/Qwen3.8-27B-FP8 --revision c3a462cc3d38cf351f3b26cbd763469eeb74ef81 \
  --served-model-name qwen3.8-27b --max-model-len 131072 --language-model-only \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --enable-lora --max-lora-rank 32 --lora-target-modules qkv_proj o_proj gate_up_proj down_proj \
  --lora-modules qwen3.8-27b-pi=harness-hub/Qwen3.8-27B-pi-adapted

# Harbor, agent pi 1.5.2, both models as two agents in one job (each task's two trials start together)
harbor run --config job.json --yes

Read the write-up Download raw JSON