Public reports

Results, with the receipts attached.

Each report pins hardware, software, model, workload and SLO, lists what it doesn't cover, and ships with raw JSON and the command that reproduces it.

2026-11-21

harness-hub's pi adapter (LoRA) on Qwen3.8-27B FP8, vLLM 0.33.0: Terminal-Bench 2.1 subset under pi, cost and stability on a shared H200

1× NVIDIA H100 NVL / H200Qwen3.8-27B-FP8 + harness-hub/Qwen3.8-27B-pi-adaptedFP8 weights, BF16 KV cache; LoRA r32 on attention + MLPvLLM
3.0×fewer turns per solved task, pi adapter vs base model, FP8
2026-11-20

Prefix-cache hits on real coding-agent traffic: Qwen3.8-27B and Qwen3.6-35B-A3B on one H200, vLLM 0.33.0

1× NVIDIA H200Qwen3.8-27B-FP8 + Qwen3.6-35B-A3B-FP8FP8 weights, BF16 KV cachevLLM
94.3%of Qwen3.8-27B prompt tokens from cache, real agent traffic
2026-11-20

MTP speculative decoding on Qwen3.8-27B FP8, 1× H100 NVL, vLLM 0.33.0: 0 to 4 draft tokens

1× NVIDIA H100 NVLQwen3.8-27B-FP8FP8 weights, BF16 KV cachevLLM
2.65×single-request decode, 3 draft tokens vs MTP off
2026-11-20

Qwen3.8-27B and Qwen3.6-35B-A3B on our API, 175 single-file Python tasks: thinking on vs off

1× NVIDIA H200Qwen3.8-27B-FP8 + Qwen3.6-35B-A3B-FP8FP8 weightsvLLM
26×Qwen3.8-27B cost, thinking on vs off, for +5.7 points
2026-11-18

Qwen3-8B BF16 on 1× AMD Instinct MI350X: vLLM's ROCm image vs AMD's image and settings

1× AMD Instinct MI350XQwen3-8BBF16 weights, BF16 KV cachevLLM
2.36×SLO goodput, AMD image + AMD settings
vs. vLLM ROCm image, defaults
2026-11-18

Qwen3-Coder-30B-A3B FP8 on 1× AMD Instinct MI350X: vLLM's ROCm image vs AMD's image and settings

1× AMD Instinct MI350XQwen3-Coder-30B-A3B-Instruct-FP8FP8 weights, BF16 KV cachevLLM
2026-11-18

Qwen3-8B BF16 on 1× H100 SXM (Lium spot): vLLM defaults vs vLLM's recommended setting

1× NVIDIA H100 80GB HBM3Qwen3-8BBF16 weights, BF16 KV cachevLLM
2026-11-18

Qwen3-Coder-30B-A3B FP8 on 1× H100 SXM (Lium spot): vLLM defaults vs DeepGEMM

1× NVIDIA H100 80GB HBM3Qwen3-Coder-30B-A3B-Instruct-FP8FP8 weights, BF16 KV cachevLLM
1.00×SLO goodput, DeepGEMM FP8 MoE
vs. vLLM defaults
2026-11-13

Prefix caching on a hybrid model, one H100: Qwen3.8-27B FP8, vLLM 0.31.0, chat-like traffic

1× NVIDIA H100 80GB HBM3Qwen3.8-27B-FP8FP8 weights, BF16 KV cachevllm
1.67×SLO goodput, 90% shared prefix (55.6% cache hits)
vs. No prompt reuse (0% cache hits)
2026-11-13

Prefix caching on a hybrid model, one H100: Qwen3.8-27B FP8, vLLM 0.31.0, 4,000-token prompts

1× NVIDIA H100 80GB HBM3Qwen3.8-27B-FP8FP8 weights, BF16 KV cachevllm
3.20×SLO goodput, 90% shared prefix (74.2% cache hits)
vs. No prompt reuse (0% cache hits)
2026-11-10

Qwen3-8B BF16 on 1× B200 (Lium, Kansas City) with vLLM 0.33.0

1× NVIDIA B200Qwen3-8BBF16 weights, BF16 KV cachevLLM
2026-11-10

Qwen3-8B BF16 on 1× H100 80GB HBM3 (Lium, Helsinki) with vLLM 0.33.0

1× NVIDIA H100 80GB HBM3Qwen3-8BBF16 weights, BF16 KV cachevLLM
2026-11-10

Qwen3-8B BF16 on 1× H200 (Lium, Jackson) with vLLM 0.33.0

1× NVIDIA H200Qwen3-8BBF16 weights, BF16 KV cachevLLM
2026-11-10

Qwen3-8B BF16 on 1× RTX PRO 6000 Blackwell Server Edition (Lium, Almaty) with vLLM 0.33.0

1× NVIDIA RTX PRO 6000 Blackwell Server EditionQwen3-8BBF16 weights, BF16 KV cachevLLM
2026-11-10

Qwen3-8B BF16 on 1× RTX PRO 6000 Blackwell Server Edition (Lium, Dublin) with vLLM 0.33.0

1× NVIDIA RTX PRO 6000 Blackwell Server EditionQwen3-8BBF16 weights, BF16 KV cachevLLM
2026-11-02

Long-context capacity on one H100: ~8K input tokens, Qwen3-30B-A3B-Instruct-2507 FP8, vLLM 0.31.0

1× NVIDIA H100 80GB HBM3Qwen3-30B-A3B-Instruct-2507-FP8FP8 weights, FP8 KV cachevllm
2026-11-02

Long-context capacity on one H100: ~16K input tokens, Qwen3-30B-A3B-Instruct-2507 FP8, vLLM 0.31.0

1× NVIDIA H100 80GB HBM3Qwen3-30B-A3B-Instruct-2507-FP8FP8 weights, FP8 KV cachevllm
2026-10-29

Prefix caching on one H100: Qwen3-30B-A3B-Instruct-2507 FP8, vLLM 0.31.0

1× NVIDIA H100 80GB HBM3Qwen3-30B-A3B-Instruct-2507-FP8FP8 weights, BF16 KV cachevllm
1.79×SLO goodput, 90% shared prefix (89.2% cache hits)
vs. No prompt reuse (0% cache hits)
2026-10-27

DeepSeek-V4.1-Flash on 4× H200 with vLLM 0.32.0 (TP4, FP8 KV cache)

4× NVIDIA H200DeepSeek-V4.1-FlashFP8 weights, FP8 KV cachevLLM
2026-10-22

Qwen3-8B BF16 on 1× RTX PRO 6000 Blackwell Server Edition with vLLM 0.31.0

1× NVIDIA RTX PRO 6000 Blackwell Server EditionQwen3-8BBF16 weights, BF16 KV cachevLLM
2026-10-13

Qwen3-8B BF16 on 1× H100 PCIe 80GB with vLLM 0.31.0

1× NVIDIA H100 PCIeQwen3-8BBF16 weights, BF16 KV cachevLLM
2026-10-13

Qwen3-8B BF16 on 1× H100 NVL 94GB with vLLM 0.31.0

1× NVIDIA H100 NVLQwen3-8BBF16 weights, BF16 KV cachevLLM
2026-10-13

Qwen3-8B BF16 on 1× H100 SXM 80GB with vLLM 0.31.0

1× NVIDIA H100 80GB HBM3Qwen3-8BBF16 weights, BF16 KV cachevLLM
2026-10-11

Qwen3-8B BF16 on 1× A100 SXM 80GB with vLLM 0.31.0

1× NVIDIA A100-SXM4-80GBQwen3-8BBF16 weights, BF16 KV cachevLLM
2026-10-11

Qwen3-8B BF16 on 1× A40 with vLLM 0.31.0

1× NVIDIA A40Qwen3-8BBF16 weights, BF16 KV cachevLLM
2026-10-11

Qwen3-8B BF16 on 1× L40S with vLLM 0.31.0

1× NVIDIA L40SQwen3-8BBF16 weights, BF16 KV cachevLLM
2026-10-11

Qwen3-8B FP8 on 1× L40S with vLLM 0.31.0

1× NVIDIA L40SQwen3-8B-FP8FP8 weights, BF16 KV cachevLLM
2026-10-11

Qwen3-8B BF16 on 1× RTX 4090 with vLLM 0.31.0

1× NVIDIA GeForce RTX 4090Qwen3-8BBF16 weights, BF16 KV cachevLLM
2026-10-11

Qwen3-8B FP8 on 1× RTX 4090 with vLLM 0.31.0

1× NVIDIA GeForce RTX 4090Qwen3-8B-FP8FP8 weights, BF16 KV cachevLLM
2026-10-10

vLLM 0.31.0 vs SGLang 0.5.21 on 1× H100 SXM, Qwen3-30B-A3B FP8

1× NVIDIA H100 80GB HBM3Qwen3-30B-A3B-FP8FP8 weights, BF16 KV cachevLLM
1.01×SLO goodput, SGLang 0.5.21
vs. vLLM 0.31.0
2026-10-09

Llama-3.3-70B FP8 on 1× H200: prefill chunk size (max-num-batched-tokens)

1× NVIDIA H200Llama-3.3-70B-Instruct-FP8-dynamicFP8 weights (compressed-tensors), BF16 KV cachevLLM
2026-10-09

Qwen3-30B-A3B FP8 on 1× H200 with vLLM 0.31.0

1× NVIDIA H200Qwen3-30B-A3B-FP8FP8 weights, BF16 KV cachevLLM
2026-10-08

Tuned vs default fused-MoE kernels: Qwen3-30B-A3B FP8 on 1× H100 SXM

1× NVIDIA H100 80GB HBM3Qwen3-30B-A3B-FP8FP8 weights, BF16 KV cachevLLM
1.00×SLO goodput, default kernels (repeat)
vs. default kernels
2026-10-06

Qwen3-30B-A3B FP8 on 1× H100 SXM with vLLM 0.31.0

1× NVIDIA H100 80GB HBM3Qwen3-30B-A3B-FP8FP8 weights, BF16 KV cachevLLM
2026-10-06Draft

Qwen3-30B-A3B FP8 on 1× H100: BF16 vs FP8 KV cache

1× NVIDIA H100 80GB HBM3Qwen3-30B-A3B-FP8FP8 weights; KV cache varies by configurationvLLM
2026-10-06Example data

Mid-size MoE decode on an 8-GPU node

8× Accelerator A (example)Example-MoE-30B-A3BFP8 weights, FP8 KV cachevLLM
1.36×SLO goodput, Tuned
vs. Vendor latest