harness-hub's pi adapter (LoRA) on Qwen3.8-27B FP8, vLLM 0.33.0: Terminal-Bench 2.1 subset under pi, cost and stability on a shared H200
Three tests, 2026-11-20/21, on rented GPUs (RunPod Secure Cloud): (1) LoRA on the FP8 checkpoint, one H100 NVL, speed and memory; (2) 25 Terminal-Bench 2.1 tasks under pi 1.5.2 (Harbor 0.26.0), base model and adapter served by one FP8 engine on an H100 NVL, the tasks run on a Lambda host, thinking off, temperature 0.8, top-p 1.0, 100 turns and 60 min per task, one attempt; (3) our production H200 layout (Qwen3.8 at 52% of the GPU with LoRA + Qwen3.6 at 43%, CUDA MPS, 262K context) under the long-context agent load that crashed MTP on 11-20. Model Qwen/Qwen3.8-27B-FP8 (revision c3a462cc); adapter harness-hub/Qwen3.8-27B-pi-adapted (3cad999e, Apache-2.0, LoRA r32). Write-up: /blog/qwen38-27b-pi-adapter-fp8-terminal-bench.html
Results
FP8 checkpoint, our runs; HF's BF16 grades for the same tasks in the table below.
Median turns per solved task
Median output tokens per solved task
Tasks solved of 25
Terminal-Bench 2.1, 25 tasks under pi, one attempt each
The tasks were selected: the 17 where HF's single BF16 attempts of the two models differed (path-tracing left out: both timed out) and 8 of the 27 light tasks both solved, drawn at random. A rerun at temperature 0.8 regresses toward the mean on any weights, so the BF16 columns are not a baseline.
| Tasks | HF BF16, base | HF BF16, adapter | FP8, base | FP8, adapter |
|---|---|---|---|---|
| 15 the adapter gained in BF16 | 0 | 15 | 7 | 10 |
| 2 the adapter lost in BF16 | 2 | 0 | 1 | 1 |
| 8 both solved in BF16 (random) | 8 | 8 | 6 | 6 |
| All 25 | 10 | 23 | 14 | 17 |
Work per solved task (12 tasks both models solved on FP8)
Medians. Agent time includes the commands the agent ran.
| Base model | pi adapter | |
|---|---|---|
| Turns | 28.5 | 9.5 |
| Output tokens | 5,254 | 1,432 |
| Agent time | 5.8 min | 6.6 min |
| Attempts at the 100-turn cap (of 25) | 9 | 1 |
Cost of serving the adapter
LoRA restricted to the attention + MLP layers (--lora-target-modules qkv_proj o_proj gate_up_proj down_proj).
| Measure | Without LoRA | With LoRA |
|---|---|---|
| One request, H100 NVL, decode tok/s | 75.8 | base 76.4 · adapter 65.3 (-13.8%) |
| One request, our H200, decode tok/s | 91.4 (public API) | base 91.0 · adapter 72.5 (-20%) |
| Mixed base + adapter traffic, 16 in flight, H100 NVL | 931 tok/s | 834 tok/s (-10.5%) |
| Qwen3.8 KV cache on our H200 (0.52 of the GPU) | 668,668 tokens | 632,371 tokens (-5.4%) |
Our production H200 layout under the 11-20 crash load
6 + 6 concurrent coding-agent sessions, prompts growing to ~100-125K tokens, CUDA MPS, 10 minutes.
| Configuration | Result |
|---|---|
| Qwen3.8 0.52 + LoRA (half its sessions on the adapter), Qwen3.6 0.43 | no fault in 10 min; Qwen3.8 108 requests, Qwen3.6 863 |
| Qwen3.8 0.52, Qwen3.6 0.43, no LoRA (11-20) | no fault in 10 min |
| Qwen3.8 0.57 + MTP, Qwen3.6 0.38 (11-20) | illegal memory access at 1.0 and 2.1 min |
Setup
Any change to these fields makes it a different report.
- Adapter
- harness-hub/Qwen3.8-27B-pi-adapted at 3cad999e: LoRA r32, alpha 64, q/k/v/o (16 full-attention layers) + gate/up/down; trained with RL for pi, thinking off, on BF16 weights
- Engine
- vllm serve Qwen/Qwen3.8-27B-FP8 --enable-lora --max-lora-rank 32 --lora-target-modules qkv_proj o_proj gate_up_proj down_proj --lora-modules qwen3.8-27b-pi=<dir>
- Terminal-Bench
- Harbor 0.26.0, agent pi 1.5.2 (the version in HF's trajectories), openai-completions API, context 131,072, max 8,192 tokens per call, 100 turns, 3,600 s per task, one attempt; tool parser qwen3_xml, thinking off, temperature 0.8, top-p 1.0
- Two passes
- first pass: 26 trials at once on one H100 NVL, cut at its time limit with 17 trials running; rerun of those 17 and 2 cancelled ones, 19 at once, same settings. Each (task, model) has one graded attempt
- Production-layout run
- separate H200 SXM, production's exact engine flags plus LoRA, CUDA MPS, the 11-20 load generator; then tool calls (20 cases), single-request speed and one pi run on the same engines
- Our API
- since 2026-11-21 19:29 UTC: harness-hub/qwen3.8-27b-pi-adapted, Qwen3.8-27B's price, thinking off by default (the caller's choice wins)
Not covered
Do not extrapolate this report to the following.
- A BF16 control on our own stack: FP8 drift and run-to-run noise are not separated
- More than one attempt per task, or all 89 Terminal-Bench tasks
- SWE-bench Pro, and the opencode and Hermes adapters
- Capacity under a latency target with adapter traffic on the H200
- Stability beyond the 10-minute run
Reproduce
Serve the adapter next to the base model and point pi at it; the Terminal-Bench tasks come from harborframework/terminal-bench-2.1 on Hugging Face.
vllm serve Qwen/Qwen3.8-27B-FP8 --revision c3a462cc3d38cf351f3b26cbd763469eeb74ef81 \
--served-model-name qwen3.8-27b --max-model-len 131072 --language-model-only \
--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \
--default-chat-template-kwargs '{"enable_thinking": false}' \
--enable-lora --max-lora-rank 32 --lora-target-modules qkv_proj o_proj gate_up_proj down_proj \
--lora-modules qwen3.8-27b-pi=harness-hub/Qwen3.8-27B-pi-adapted
# Harbor, agent pi 1.5.2, both models as two agents in one job (each task's two trials start together)
harbor run --config job.json --yes