Qwen3.8-27B: Outperforming SGLang and vLLM Across the Board

On a single NVIDIA RTX PRO 6000 Blackwell GPU, Backboard outperforms vLLM 0.33 and SGLang across almost every measured dimension of Qwen3.8-27B inference. Using one Backboard configuration against both competitors, we won every measured metric against vLLM and 102 of 104 against SGLang.

Fast inference doesn't need a large GPU cluster. And optimizing for one performance metric shouldn't mean sacrificing another.

Our latest InferenceX benchmark examined how Backboard performs against two widely used inference engines, vLLM 0.33 and SGLang, running Qwen3.8-27B in BF16 on a single NVIDIA RTX PRO 6000 Blackwell GPU.

The results were exceptional.

Across eight test conditions covering two input lengths and four concurrency levels, Backboard won all 104 metric comparisons against vLLM. Against SGLang, it won 102 of 104.

The most consistent advantage is throughput: Backboard generates more tokens per second at every tested concurrency level against both competitors.

Just as importantly, we see improvements in time to first token, per-request generation speed, and end-to-end latency.

And we achieved these results using one Backboard configuration against both engines, rather than separately tuning Backboard for each competitor.

More output throughput across every workload

We started by examining aggregate output throughput: how many output tokens the entire system generates per second.

We tested two input lengths, 1,000 and 8,000 tokens, at concurrency levels of 1, 4, 16, and 64.

Backboard produced higher aggregate output throughput than both vLLM and SGLang at every one of these eight points.

The consistency is the important takeaway.

Backboard doesn't just win under light workloads or at one particularly favorable concurrency level. It produces more aggregate output at every tested combination of input length and concurrency.

Against vLLM, the advantage ranges from 16% to 44%. Against SGLang, it ranges from 4% to 26%.

The difference is even more pronounced at higher concurrency. Against vLLM, Backboard's throughput advantage reaches 44% with 1K-token inputs at C64. Against SGLang, the gap narrows at C64, but Backboard still produces 4% more output with 1K inputs and 7% more with 8K inputs.

We don’t hide that SGLang is a stronger competitor in some conditions. Even against the closer baseline, though, Backboard maintains an aggregate throughput advantage across the entire test.

A win across more than just throughput

Aggregate throughput is only one measure of inference performance.

A system can generate more tokens overall without necessarily delivering a better experience to individual users. That's why InferenceX also measures three dimensions of latency:

• Time to first token (TTFT): How long a request waits before generation begins.

• Time per output token (TPOT): How long the system takes to generate each subsequent token.

• End-to-end latency: The total time required to complete a request.

Each latency measurement is evaluated at the mean, median (p50), 90th percentile (p90), and 99th percentile (p99).

Backboard outperformed vLLM across all 104 comparisons, including every one of these latency measurements.

Against SGLang, the results were similarly strong: Backboard won 13 out of 13 metrics at six of the eight test points.

In other words, the throughput advantage wasn't achieved by making individual requests slower or simply moving more work through the system at the expense of responsiveness.

Faster time to first token

One of the most impressive results is the reduction in median TTFT, particularly at higher concurrency.

With 1K-token inputs at C64, Backboard's median TTFT was just 299 milliseconds, compared with 13.275 seconds for vLLM and 322 milliseconds for SGLang.

That's approximately 44.4× faster than vLLM and 8% faster than SGLang.

Other gains are more modest. With 8K-token inputs at C64, median TTFT was approximately 63.4 seconds for Backboard, compared with 71.2 seconds for vLLM and 64.4 seconds for SGLang. At that input length and concurrency, Backboard's median advantage over SGLang is only around 1%.

The comparison with SGLang is essential. Backboard's advantage over vLLM is dramatic, but the advantage over SGLang is much smaller — it shows that there’s still work to do.

Better generation speed and request completion

Backboard also delivers faster per-request generation.

Looking at median TPOT, Backboard's advantage ranges from:

• 4% to 34% over vLLM

• 4% to 27% over SGLang

These improvements are relevant to applications where users consume responses as they're generated, rather than waiting for an entire response to finish.

End-to-end latency shows another consistent advantage. Backboard wins every mean, median, 90th-percentile, and 99th-percentile end-to-end comparison against both competitors.

Against vLLM, median end-to-end latency improvements range from 17% to 45%. Against SGLang, they range from 4% to 27%.

Taken together, these results demonstrate that Backboard isn't simply maximizing the number of tokens produced by the GPU. It's improving the overall request experience as well.

Where SGLang comes closest

We don't think a useful benchmark should only highlight victories. Understanding where competitors perform nearly as well, or better, is just as important.

SGLang comes closest to Backboard in the higher-concurrency conditions, particularly on latency.

At C64, its aggregate throughput is within 4% of Backboard for 1K-token inputs and within 7% for 8K-token inputs.

There are also two metrics where SGLang narrowly outperforms Backboard: TTFT p99 at C64.

These are the only two losses in the 104 metric comparisons against SGLang.

They are also relatively narrow differences, particularly the 0.3% result at 8K.

There are additional areas where Backboard wins by very small margins. For example, at 1K input and C64, its p99 TPOT advantage over vLLM is only 0.2%. At 8K input and C4, its p99 TPOT advantage over SGLang is only 0.5%.

We wouldn't present these as meaningful practical improvements on their own.

The broader result is more important: Backboard remains competitive across the full latency distribution, even when individual tail-latency measurements approach parity or slightly favor a competitor.

We see these narrow results as useful targets for further optimization, rather than reasons to obscure the larger picture.

What this means for real-world inference

The hardware configuration is as important as the outcome of these tests.

We're not using an eight-GPU server or a large accelerator cluster. All comparisons take place on a single RTX PRO 6000 Blackwell GPU, running Qwen3.8-27B with BF16 weights.

That makes the results relevant when your organization is interested in getting more out of relatively compact inference deployments.

There are several practical implications.

More capacity from the same hardware.

Higher aggregate throughput means a system can generate more output tokens per second without adding GPUs. For inference providers and application developers, that creates an opportunity to serve more concurrent work using existing infrastructure.

Better responsiveness under load.

Faster TTFT and lower end-to-end latency can make applications feel more responsive, particularly when several users or agents are active simultaneously.

More efficient agent execution.

Agent workflows often involve repeated model calls, reasoning steps and tool interactions. Time spent waiting for each model response can accumulate. Better request-level performance can help make those workflows more practical.

A smaller barrier to deploying capable models.

You don’t always need or justify a large multi-GPU inference deployment. Improving the performance of a single-GPU system expands the range of workloads that can be served on relatively limited hardware.

We don't claim that a particular throughput improvement automatically translates into an equivalent cost reduction. Infrastructure economics depend on everything from workload composition through to service requirements and pricing.

Nonetheless, the opportunity is clear: a better inference engine can help you wring out more useful work from the GPUs you already own.

How we tested

We used InferenceX to compare Backboard, vLLM 0.33, and SGLang on the same model and hardware.

Hardware and workload

This produced eight workload points, with 13 metrics evaluated at each point, or 104 comparisons per competitor.

All three engines used the InferenceX client with seed 0, ensuring they received the same prompts.

Engine configurations

We used the following configurations:

Backboard: All model weights remained BF16, with FP32 accumulation. Backboard used its own 4-bit draft head, with the BF16 model checking every drafted token, an INT8-bound LM head and proprietary decode GEMM kernels.

vLLM: We used vLLM 0.33 with MTP speculative decoding enabled using three draft tokens and a request cap matching the tested concurrency.

SGLang: We used a preset configured with BF16 weights, NEXTN speculative decoding using three steps and four draft tokens, FP32 recurrent state, 64 running requests, and prefix caching disabled.

These settings are disclosed because the comparison is intended to evaluate configured inference engines, not just compare framework names without describing how they were run.

Run procedure and scoring

Backboard ran on fresh servers on November 4. The latest version's only change from its parent build affected 256-row decode rounds, which are only reached at C64. Consequently, the six points below C64 use results from the parent build, also run that day.

We performed two Backboard runs at 1K input/C64 and one at 8K input/C64.

vLLM ran once per point on November 3. SGLang ran three or four times per point on November 3. The competitor runs were not interleaved with Backboard's.

For each metric, we compared the mean of Backboard's runs with the mean of the corresponding competitor's runs. A win means Backboard had higher throughput or lower latency, as appropriate.

The resulting scorecard was:

Against vLLM, all 104 metrics favored Backboard. Against SGLang, Backboard won every metric at six points and 11 of 13 at each of the two C64 points.

The report also records that 100 of the 104 SGLang comparisons were wins across every run-pair comparison.

Additional implementation disclosures

We also tested a kernel change that affects the 256-row decode path at C64.

The change combines the DeltaNet input projection into one joined GEMM instead of two, while retaining FP32 accumulation and BF16 stores. It is not bit-exact by construction. On the real weights, however, it matched the previous path in all 37,969,920 outputs tested. Served greedy runs also matched the parent build within its own run-to-run rate.

We do not compare inter-token latency because the engines stream different chunk sizes, making that measurement unsuitable for a direct comparison in this test.

Finally, these are Backboard-produced internal benchmark results, not an independent third-party certification. The different numbers of runs and the fact that competitor runs were not interleaved are relevant limitations when interpreting the results. We intend to continue improving the reproducibility and detail of our benchmark disclosures.

Better inference is about the whole system

This benchmark demonstrates why we believe inference performance needs to be evaluated across multiple dimensions.

Backboard achieved more aggregate output throughput than both vLLM and SGLang at every tested concurrency and input length. It also improved time to first token, per-request generation speed, and end-to-end latency across almost the entire comparison.

We aren't claiming that every metric improves by a large margin. At high concurrency, SGLang comes close on throughput, and it narrowly wins two tail-latency comparisons. Those are meaningful findings that help identify where further optimization is needed.

But the overall result is difficult to overlook.

Our goal is to make existing hardware do more useful inference work, without requiring a continually expanding GPU footprint.

We believe that the inference engine should be a major part of that equation. This benchmark is another step toward demonstrating what's possible.