Gemma 4: Up to 2.2× faster throughput than vLLM and SGLang

On a single NVIDIA RTX PRO 6000 Blackwell GPU, Backboard won every measured performance metric across eight workload configurations using Google’s Gemma 4. The largest throughput advantages came at 1K context with 64 concurrent requests and 8K context with a single request, while the strongest advantage over vLLM came at 8K context with four concurrent requests.

Backboard's inference engine outperformed both vLLM 0.33.0 and SGLang 0.5.23 in every measured category in a new Gemma 4 12B serving benchmark using SemiAnalysis InferenceX.

The results include two standout comparisons.

At 64 concurrent requests with 1,000-token prompts, Backboard managed 2.18× the output throughput of SGLang. At a single concurrent request with 8,000-token prompts, it delivered 2.17× the throughput of SGLang, alongside tangible improvements in tail latency.

These results are compelling when the advantages extend beyond aggregate throughput. Backboard also outperformed both competitors on time to first token (TTFT), time per output token (TPOT), and end-to-end latency (E2EL), including their mean, median, p90 and p99 measurements.

Across eight workload configurations and 13 metrics per configuration, Backboard recorded:

That is 208 winning comparisons, with no losses or ties.

Consistent wins in every metric

At 64 concurrent requests with 1K-token prompts, Backboard achieved 2.18× the output throughput of SGLang.

This is key to your high-volume inference deployments, where serving many simultaneous users efficiently can determine how much infrastructure is required to meet your demand. You can use fewer GPUs and servers.

The comparison with vLLM is more modest. Backboard delivered 1.05× the throughput, a relatively narrow advantage. The smallest performance margin anywhere in the benchmark occurred in this configuration: Backboard's p99 end-to-end latency was 1.022× better than vLLM's.

That distinction matters. We’re not claiming a dramatic throughput advantage over vLLM in every workload. We’re showing that Backboard maintains an advantage even in one of the most competitive configurations, while providing a much larger gain against SGLang.

Moreover, Backboard still won all 13 measured metrics against both competitors at this concurrency level — there won’t be weak points in your system.

8K context, single request: 2.17× SGLang throughput

The other standout result comes from serving longer prompts with a single concurrent request.

With 8,000-token prompts, Backboard produced 2.17× the output throughput of SGLang and 1.12× that of vLLM.

The latency results are telling. Against SGLang, Backboard achieved:

Against vLLM, the corresponding advantages were 1.26× and 1.25×.

P99 time per output token, milliseconds (lower is better)

P99 end-to-end latency, milliseconds (lower is better)

These results prove that Backboard's gains aren’t limited to batching large numbers of short requests. They extend to individual requests carrying much more context, where responsiveness remains important even without significant concurrency.

For applications that regularly process longer documents, code context or other substantial inputs, this can generate quicker, smoother answers.

Where Backboard's advantage is more modest — and where it grows

The strongest direct comparison against the faster of the two rivals is at 8K context and four concurrent requests.

Here, Backboard delivered 1.34× the throughput of vLLM, its largest throughput advantage over that challenger across the benchmark. Against SGLang, the advantage was even higher at 1.97×.

Backboard's performance advantage depends on the workload, but it’s always present.

See the complete output throughput results for yourself.

Output throughput across all configurations

All throughput measurements are output tokens per second. Concurrency refers to the number of simultaneous requests. Higher throughput is better.

Across all eight configurations, Backboard delivered between 1.05× and 1.34× the throughput of vLLM, and between 1.66× and 2.18× that of SGLang.

The latency results reinforce the consistency of these findings. Backboard led both competitors across TTFT, TPOT and E2EL at the mean, median, p90 and p99 levels, at every concurrency point and for both prompt lengths.

In other words, the gains aren’t simply a product of higher aggregate throughput at the expense of individual request performance. Backboard also delivered better per-request latency throughout the benchmark.

What this means for your inference

Benchmark improvements only matter when they translate into real advantages when you deploy AI models. Here, there are several implications worth considering.

More capacity from the same GPU investment

Higher output throughput generates more tokens using the same hardware over the same period.

As we mentioned, you can serve more users without adding GPUs, or maintain your existing capacity while reducing the infrastructure required to support a given workload.

The gains are smaller for vLLM than SGLang, but still consistent. Even an improvement on that scale can supply meaningful additional capacity in a production environment.

Actual cost savings will depend on your GPU-hour pricing, usage, workload and architecture. The benchmark doesn’t attempt to quantify dollar savings, this illustrates the potential to extract more useful work when you have a fixed hardware investment.

Better responsiveness for individual users

Aggregate throughput is only one part of the serving equation.

Time to first token determines how long users wait before seeing the beginning of a response, while time per output token affects how quickly that response streams. End-to-end latency captures the time required to complete the request.

P90 time to first token, milliseconds (lower is better)

Improvements across all three measurements show that Backboard can deliver a more responsive experience while boosting your overall serving capacity.

Speed per user, tokens/s per request (higher is better)

The p90 and p99 results outline what happens to slower requests, rather than focusing only on typical performance. Your heavier workloads should still run quickly.

Reducing tail latency, meanwhile, can make the experience more consistent, especially when many users are competing for the same inference resources.

Stronger performance across different workload profiles

Not every deployment looks the same.

Some applications serve large numbers of relatively short requests. Others send reams of context with each request, even when concurrency is relatively low.

Backboard maintains its lead across both scenarios.

The 1K 64-user results prove we can sustain high aggregate throughput under concurrency. The 8K single-request results make clear that edge also exists when handling longer prompts for individual users.

These are crucial if you’re building document-processing, coding or knowledge-intensive applications. You can deal with more input context than straightforward conversational exchanges.

The benchmark doesn’t directly test complete production applications, but it gives you useful evidence of how the inference engine performs across different input lengths and serving loads.

How we conducted the comparison

We designed this evaluation around a straightforward question: can Backboard deliver better throughput and lower latency than leading open-source inference engines while running the same model, precision and GPU configuration?

To answer that question, we used SemiAnalysis InferenceX, which evaluates both aggregate serving performance and the responsiveness of individual requests.

Hardware and model

All three inference engines served the same model:

• Model: google/gemma-4-12B-it.

• Precision: BF16.

• GPU: One NVIDIA RTX PRO 6000 Blackwell GPU with 96 GB of memory per server.

• Hardware environment: Two identical machines, running at default clocks and power settings.

• Draft model: The official Gemma assistant model, used by all three engines.

The evaluation used the same target model weights and precision for all competitors.

Backboard used a single inference configuration for every benchmark point. Its draft head shortlists speculative candidates in 4-bit before rescoring them in BF16. This affects the number of accepted speculative drafts; it doesn’t change the BF16 target-model output. At one request, Backboard's greedy output is identical to its BF16 reference.

Competitor configurations

We tested the following versions and configurations:

vLLM 0.33.0

• Multi-token prediction (MTP) speculative decoding with four tokens.

• Asynchronous scheduling.

• 90% GPU memory allocation.

• Prefix caching disabled.

SGLang 0.5.23

• NEXTN speculative decoding with five steps and six draft tokens.

• 85% static GPU memory allocation.

• Radix caching disabled.

Both competitors used their respective published recipes with default settings. We didn’t undertake competitor-specific tuning for this comparison.

Workloads and concurrency

We evaluated two prompt lengths:

• 1,024 input tokens.

• 8,192 input tokens.

Each request generated 1,024 output tokens, using the InferenceX client and chat template.

The benchmark tested four concurrency levels: one, four, 16, and 64 simultaneous requests.

This produced eight distinct workload configurations.

Each point was run with freshly started servers, with all three engines tested in turn on the same machine during the same benchmark block on November 9, 2026.

All points used prompt seed 0. The 8K configuration with four concurrent requests additionally used seeds 1 and 2 for every engine, with the reported result calculated as the mean across the three seeds.

What we measured

InferenceX measures four aspects of inference performance:

Output throughput: The total number of generated output tokens per second across the workload.

Time to first token (TTFT): How long a request waits before receiving its first generated token.

Time per output token (TPOT): How much time elapses between generated output tokens, reflecting the responsiveness of token streaming.

End-to-end latency (E2EL): The total time required to complete a request.

For each of the three latency metrics, we evaluated four statistical measurements: mean, median, p90 and p99.

That gives us 13 metrics per workload point: one throughput measurement and four measurements for each of the three latency categories.

Across eight configurations, that produces 104 comparisons against each competitor.

A win required Backboard's measurement to be strictly better. For throughput, higher was better; for latency, lower was better.

Backboard won every comparison against both competitors.

Backboard's margin in every metric against vLLM

Backboard's margin in every metric against SGLang

Each value is the competitor's latency divided by Backboard's, or Backboard's throughput divided by the competitor's, so every value above 1× is a Backboard win.

Important limitations and what comes next

We believe benchmark results are most useful when their limitations are made explicit.

This evaluation has several constraints worth highlighting.

One GPU architecture: The tests used a single NVIDIA RTX PRO 6000 Blackwell GPU. These results should not automatically be extrapolated to other GPU generations or multi-GPU deployments.

Limited repetitions: There was one run per point using one prompt seed, except for 8K prompts at four concurrent requests, which averaged three seeds. Additional independent repetitions would provide further evidence of the consistency of the measured differences.

Competitor tuning: vLLM and SGLang used their published recipes with default settings rather than exhaustively tuned configurations. Further competitor-specific optimization could change the margins.

No shared prompt prefixes: InferenceX prompts in this evaluation didn’t share prefixes. The results don’t establish comparative performance for prefix caching or workloads that benefit from shared context.

No multi-turn evaluation: This benchmark measures the specified request-serving workloads, not complete multi-turn conversational or agentic sessions.

These boundaries are important. We’re demonstrating a consistent advantage under a defined, reproducible benchmark setup, not claiming that every possible production workload will see identical gains.

Future evaluations across additional model architectures, hardware generations, workload types and independently repeated tests will help establish how broadly these advantages translate.

A win across the entire benchmark

The most important finding from this Gemma 4 12B evaluation is not simply that Backboard achieved a few impressive throughput improvements.

As you’ve seen. Backboard outperformed both vLLM 0.33.0 and SGLang 0.5.23 at every tested concurrency level, for both prompt lengths, across every measured throughput and latency metric.

The margins aren’t uniform, and that’s precisely why we publish the complete results rather than only the most favorable comparisons.

Our objective is to build an inference engine that helps customers get more performance from their existing hardware, without requiring them to change their underlying model weights or sacrifice model precision.

This benchmark provides another data point in that effort.

Backboard inference research