Research / Inference
More performance from the same GPUs.
The next generation of AI infrastructure won’t come from hardware alone.
We’re rethinking how models execute on GPUs.
Backboard has developed a proprietary inference engine built around original GPU kernels and execution technology. Our approach unlocks substantially more performance from existing hardware without requiring quantization or changes to the underlying model.
Our latest experiments compare Backboard with tuned vLLM 0.31 on the same eight NVIDIA H100 GPUs, running Qwen3.8-Flash-Next.
Same model checkpoint.
Same hardware.
Same prompts.
2.49×
generation speed
Median output tokens per second per request
21%
more throughput
Total output tokens per second
38%
shorter completion time
90th-percentile request completion time
BF16 checkpoint, 128 concurrent requests. Each result compares against the better of two finalist vLLM configurations for that metric.
The research question
Can we serve more work while making each request faster?
Total throughput tells us how much a server produces. It does not tell us how long someone waits for a response.
We measure both. Our experiments examine generation speed, time to first token, request completion time and total output under sustained load.
The workload includes multi-turn agent sessions, tool calls and long documents, with contexts up to 262,144 tokens.
Latest results
Faster generation. Shorter waits. Higher throughput.
At 128 concurrent requests, Backboard improved both median generation speed and total throughput on the published BF16 and FP8 checkpoints.
BF16 checkpoint. Backboard against the best tuned vLLM result for each metric.
Median generation speed
41.9
vs 16.8 tok/s
2.49× faster
Total output throughput
2,095
vs 1,731 tok/s
21% higher
p90 completion time
41.7s
vs 67.7s
38% shorter
Median time to first token
0.86s
vs 1.59s
46% shorter
FP8 checkpoint
2.51× faster · 22% higher · 40% shorter · 43% shorter
Each vLLM value is its better result across the two finalist configurations. No single vLLM configuration achieves all these baseline values at once. Each precision is compared with the same checkpoint. FP8 values: Backboard 41.8 tok/s, 2,149 tok/s, 41.6s and 0.90s; vLLM 16.7 tok/s, 1,764 tok/s, 69.9s and 1.57s.
At 64 concurrent requests, median generation speed was 1.45× vLLM in BF16 and 1.85× in FP8. The size of the advantage changes with load.
How we tested
Hardware
Both engines ran on one server with 8 NVIDIA H100 80GB SXM GPUs, receiving identical prompts in the same order.
Tuning
We tuned vLLM across kernel choices, batch sizes, parallelism strategies, speculative decoding settings and an FP8 KV cache option. The two strongest configurations advanced to the final comparison.
Workloads
We tested short-context and long-context requests, tool calls and long-context agentic tool calls, using the model’s full 256K context window (262,144 tokens).
Runs
Every run started a fresh server, with prefix caching disabled on both engines. Results at 128 concurrent requests average two runs, except for vLLM’s DP8+EP configuration, which used one. Results at 64 concurrent requests use one run.
What these results establish
These internal benchmarks, conducted on 18 October 2026, demonstrate gains for one model, one workload and one GPU type. They have not been independently verified.
The results shown here focus on metrics where Backboard leads. vLLM’s DP8+EP configuration reaches the first token sooner at the 90th percentile.
Performance will vary with model, hardware, precision, request lengths, concurrency and latency requirements. Our research goal is to understand where these gains hold and extend them across more serving conditions.
Evaluate Backboard on your workload.
Explore our inference engine or discuss an evaluation with the team.