The problem with headline multipliers

Big performance multipliers make great headlines. But when you look more closely at how they're calculated, the real gains can be considerably more modest. For your AI infrastructure, the difference between a compelling benchmark and a useful one is critical.

AI inference benchmarks are full of impressive numbers. Two times the throughput. Three times the generation speed. More tokens from the same GPUs.

Those claims are exciting, particularly when you're trying to get more out of an expensive GPU deployment. But there's an important question you should ask before taking a headline at face value:

What exactly is being compared, and under what conditions?

A performance multiplier doesn't tell you everything about an inference engine. The result can change radically depending on whether you're comparing individual user speed or aggregate throughput, which performance point you select, and which competitor you're using as a baseline.

Recent claims from Friendli AI, the discussion around Together AI's performance figures, and SemiAnalysis’ InferenceX preview provide useful examples of why you should pay close attention to what benchmarks actually say.

Backboard’s own comparisons against Gemma 4 and SGLang results illustrate why looking beyond the headline can give you a much more realistic picture.

When a Benchmark Doesn't Tell the Whole Story

In its October 29 article, More Throughput, Faster Generation: Friendli Inference vs. vLLM, Friendli makes two headline claims.

It reports approximately 2.1× more aggregate throughput than vLLM at the same per-user generation speed, and approximately 3.1× faster per-user generation at the same aggregate throughput.

At first glance, those are impressive improvements. But they're also two different ways of interpreting a performance curve.

The first comparison holds individual user speed constant.

Imagine that you require every user to receive generated tokens at a particular rate. You then compare how much aggregate output each inference engine can produce while maintaining that rate.

This tells you how much additional serving capacity you can get without slowing down individual users.

The second comparison holds aggregate throughput constant.

Here, you compare how quickly individual users receive generated tokens while both engines maintain the same total output rate.

This tells you how much faster an individual interaction can become without sacrificing overall serving capacity.

Both measurements are useful. But they answer different questions, and neither should be mistaken for a universal measure of how much faster one engine is.

A 3.1× improvement in per-user generation speed at one selected throughput doesn't mean every user will experience a 3.1× improvement in every workload. Similarly, a 2.1× throughput improvement at one selected speed doesn't mean the engine will consistently generate twice as much output at every concurrency level.

The distinction becomes clearer when you examine other performance curves.

What happens when we apply consistent comparisons to Backboard?

Our Gemma 4 comparison against SGLang provides a useful reference for these comparisons.

We tested both inference engines on one RTX PRO 6000 GPU, using the same model and the exact same configuration.

The results show that Backboard consistently improves decode performance. But the scale of the improvement depends on how you compare the two systems, and every way of measuring it comes out firmly in Backboard's favor.

There are two useful ways to look at the results.

Same throughput: The improvement in individual user speed

Using the same-throughput approach, the Backboard performance curve shows approximately 1.75× to 3.95× faster per-user generation than SGLang across the illustrated operating points.

The mechanism matters here: to match the total throughput SGLang delivers, Backboard needs fewer concurrent requests, so each user rides a faster point on the curve. The multiplier ranges from 1.75× at SGLang's lower-concurrency points to nearly 4× where SGLang needs 16 concurrent requests on 8K prompts to reach the throughput Backboard delivers at about 4.

It also shows why you need to look at the entire performance curve rather than just one favorable comparison. These multipliers grow as concurrency rises, and they shouldn't be treated as a uniform improvement across all conditions.

Same per-user speed: The improvement in serving capacity

The other comparison holds per-user generation speed constant and examines how much additional output throughput the engine can produce.

Here, the Backboard curve shows approximately 2.17× to 3.41× the throughput of SGLang, depending on the selected operating point.

Backboard can hold any given per-user generation speed to much higher concurrency than SGLang — roughly 3–4× the requests in flight at the same individual speed, which translates directly into that much more total output.

It's worth separating these interpolated comparisons from the direct, same-concurrency results in our published article. At equivalent concurrency levels, Backboard's decode throughput and per-user generation speed were 1.66× to 2.18× those of SGLang — already large, and, notably, identical whichever variable you measure, since throughput is just concurrency times per-user speed.

Both perspectives are useful, but they describe different comparisons — and in this case they agree. The same-concurrency multiples (1.66×–2.18×) are the floor; the conservative matched-point comparisons run from roughly 1.75× to just under 4×.

For you, the key is that there's no single multiplier that captures the entire performance difference. Your results depend on whether you prioritize serving capacity, individual response speed, or a balance of both — but against SGLang, every framing lands between roughly 1.7× and 4× in Backboard's favor, and the published headline of up to 2.18× throughput sits comfortably within, not above, that range.

The Together AI results: the same framing, smaller multiples

Together AI's November announcement, "Up to 2.7× more throughput with Together Compiler,” uses exactly the matched per-user-speed framing described above: it holds P90 tokens per second per user constant and compares total tokens per second per GPU between Together Compiler and the best published InferenceX result for each model.

The published multiples are 1.8× for DSV4.1-Flash against SGLang B200 (at 389 tok/s per user), 1.8× for GLM-5.3 against SGLang B200 (at 192), 2.7× for K3 against vLLM B300 (at 127), and 1.5× for MiniMax-M3.1-Flash against vLLM B200 (at 347).

Now apply the same measurement approach, and the same baseline, to Backboard. Holding per-user speed constant, Backboard's conservative multiples over SGLang run 2.17× to 3.41×, and even the strictest measurement — identical concurrency, identical hardware, identical configuration — gives 1.66× to 2.18×. Together's best multiple against SGLang is 1.8×.

On truly equal terms — same framing, same baseline, same metric — Backboard's numbers are the stronger ones.

Two things deflate the 2.7× headline. It comes against vLLM on B300, not against SGLang, and it sits at 127 tok/s per user — a low-speed operating point where interpolated multipliers run largest. Move the operating point and the number shrinks.

The SemiAnalysis InferenceX result: a multiplier against your own baseline

SemiAnalysis's InferenceX preview chart for Qwen3.8 Flash Next 176B Agentic plots total tokens per $1 of TCO against P90 end-to-end latency, both on Vera Rubin NVL72 running Dynamo with SGLang. The annotated multiples — 12.9×, 2.49× and 1.59× — compare the same engine with itself: the November 10 run against the November 5 run.

That is the sharpest illustration of the headline-multiplier problem.

A double-digit multiple doesn't require a different inference engine; a configuration change or a re-run five days apart can produce one.

It also shows why the cost–latency plane is the right way to read serving results: tokens per dollar and tail latency trade off against each other, and one favorable point on that curve does not describe an engine.

Backboard's published Gemma 4 results cover every metric InferenceX scores — throughput, TTFT, TPOT and end-to-end latency at the mean, median, p90 and p99 — measured at equal concurrency on identical hardware, so the comparisons hold at truly equal measurement points rather than reconstructed ones.

Putting the comparisons in perspective

Here’s what the varying performance curves actually suggest.

These aren't direct head-to-head rankings. The models, workloads, and baselines differ. Friendli uses GLM-5.3-Flash against vLLM, Backboard uses Gemma 4 against SGLang, and the Together comparison uses an InferenceX curve against SGLang. The SemiAnalysis preview, meanwhile, compares the same Dynamo–SGLang stack against itself across runs five days apart. And where the measurement approaches do align — Backboard and Together both holding per-user speed constant against the same SGLang baseline — Backboard's multiples are the larger ones.

It isn't that the Friendli, Together, or SemiAnalysis measurements are necessarily incorrect. Their results describe selected operating points in their particular benchmarks. But those multipliers are interpreted as representative of broader performance or assumed to transfer to other workloads and inference systems.

The figures also make apples-to-apples comparisons difficult with different data points, starts, and stops. How does Friendli compare at the same starting per-request generation speed, for example, or vLLM compare once that speed is identical?

The speedup can be much smaller when you look at the performance requirements your deployment actually needs to satisfy.

We’d add that time to first token (TTFT) is conspicuously absent from the competing benchmarks. That dictates the initial responsiveness.

Why methodology counts as much as the result

A useful benchmark needs to account for several things beyond the headline throughput figure.

The baseline matters.

An inference engine's improvement over vLLM might look quite different from its improvement over SGLang or DeepInfer. Comparing against a particular version or configuration doesn't establish the same improvement over every competing system.

The model matters.

Different architectures have different computational requirements and bottlenecks. GLM-5.3-Flash and DeepSeek-V4.1-Flash aren't interchangeable workloads. Performance results from one shouldn't automatically be applied to the other.

The precision and hardware configuration matter.

Even when two engines use the same GPUs, differences in precision, model execution, speculative decoding and memory management can affect results. You need to understand what’s actually being compared.

The operating point matters.

A system configured for maximum aggregate throughput might not provide the best individual response speed. At the same time, a system optimized for low latency might not maximize the number of simultaneous requests it can serve.

The measurement itself matters.

Direct measurements at equivalent concurrency levels are different from comparisons reconstructed by interpolating between points on two performance curves. Both can provide useful information, but they shouldn't be confused.

Friendli's article makes an important point about the need to examine both throughput and per-user generation speed. We agree. However, that same principle needs to be applied consistently when interpreting the headline results.

A good benchmark shouldn't just tell you which system has the biggest multiplier. It should show you what happens to performance as your requirements change.

What should you look for instead?

When you're evaluating inference infrastructure, don't settle for a single performance number.

Start with the requirements of your own application.

How quickly do your users need to receive their first tokens? What generation speed do they expect? How many concurrent sessions must you support? Does your workload involve long-context processing, multiple agent turns, or bursts of simultaneous requests?

Then look at how each inference engine performs against those requirements.

If your application needs a minimum per-user generation speed, compare aggregate throughput at that speed. If you have a fixed serving capacity, compare how quickly individual requests can complete at that throughput.

You should also examine latency percentiles, particularly p90 and p99. A good median can conceal slow requests that frustrate many of your users.

Then look at the hardware economics. More aggregate output from the same GPU allocation can improve compute costs per token, but that doesn't automatically translate into an equivalent reduction in total serving costs. Utilization, scheduling, networking, and other infrastructure expenses (such as the cost of average GPU power) still matter.

We're focused on making these comparisons meaningful. Our inference engine is built around proprietary GPU kernels and execution technology designed to increase serving capacity and improve response times on existing hardware.

Our research covers throughput, per-user generation speed, time to first token, and performance under concurrent workloads. We also make the relevant trade-offs and limitations visible rather than pretending that every metric improves by the same amount.

You can explore our inference research and our latest DeepSeek-V4.1-Flash comparison to see how results can vary across different workloads and operating conditions.

Look beyond the biggest number

AI inference benchmarks are important. They help you understand what different systems can do and where your infrastructure investment will go furthest.

But a rockstar multiplier isn't the same as a complete performance picture.

The Friendli, Together, Backboard, and SGLang comparisons demonstrate why you need to look at how throughput and individual user speed interact, rather than relying on one hand-picked operating point.

Sometimes the biggest headline gain isn't the improvement that relates the most to your application. And sometimes the most valuable performance improvement is the one that lets you serve more customers without slowing down the experience they're getting.