More Power At Every Level: Outperforming DeepInfer In Decode And Prefill

Backboard delivers up to 1.12× the decode throughput and 1.37× the prefill throughput per GPU compared with DeepInfer. Even more importantly, its prefill advantage grows as the number of tokens processed per GPU step increases.

Getting more from your GPUs is one of the most direct ways to improve the economics of AI inference. Every additional token your hardware can process represents more capacity for your applications, more users served, and potentially lower costs per request.

We put Backboard's inference engine head-to-head against DeepInfer using DeepSeek-V4.1-Flash on the same four NVIDIA B200 GPUs, following DeepInfer's own benchmark approach.

The results demonstrate a consistent advantage across both major stages of inference. Backboard outperforms the DeepInfer version we ran at every tested decode and prefill point, delivering:

There’s another important finding: Backboard's relative advantage in prefill becomes greater as the number of tokens processed per GPU step increases.

At 65,536 tokens per step, Backboard delivers 1.37× the prefill throughput of our DeepInfer run.

For customers, this means more inference capacity from the same hardware investment, with extra benefits for workloads that involve processing large amounts of input context.

Consistently faster decode across concurrency levels

Decode performance determines how quickly your model generates output once it has processed the input and started responding. It affects both the experience of individual users and the number of concurrent requests your infrastructure can handle.

We tested six concurrency levels, from eight to 256 requests per GPU. Backboard delivered higher output throughput and faster per-user generation at every level.

The consistency matters. Backboard's advantage doesn’t depend on an exact concurrency level or vanish once more requests arrive. Even with 1,024 total requests across four GPUs, it maintains a 1.09× advantage in both aggregate output throughput per GPU and per-user generation speed.

Higher decode throughput lets you supply more generated tokens for the same GPU spend. You have more capacity to serve users, handle more inference requests, and accommodate growing demand before expanding your GPU allocation.

The per-user improvement is equally vital. You can extract more usage without sacrificing generation speed for individual users. Aggregate capacity and individual generation performance improve in tandem.

The larger the prefill workload, the greater the advantage

Decode is only part of the inference process. Before generation begins, the model must process the input tokens your users provide. This is the prefill stage.

Prefill performance is crucial when your applications work with lengthy documents, extensive conversation histories, large codebases, or other information-intensive inputs.

We measured non-cached input throughput per GPU across five different prefill step sizes, ranging from 4,096 to 65,536 tokens per GPU per step.

Backboard outperformed our DeepInfer run at every tested point.

The improvement is most intriguing at larger step sizes.

The progression isn’t perfectly linear at every intermediate point, but the overall trend is clear: Backboard's relative performance advantage grows as the amount of processed input increases.

This is essential as AI applications become more context-intensive. When you give a model extensive background information, ask it to analyze lengthy documents, or provide an agent with a large code repository, processing that input efficiently becomes increasingly important.

Higher prefill throughput lets you process more input tokens using the same GPU resources. This gives you greater capacity for long-context applications without having to increase your hardware resources in sync with the additional input.

The improvement can also have a direct impact on unit value. Assuming the same GPU-hour price and sustained usage, 1.37× the prefill throughput corresponds to about 0.73× the GPU cost per processed input token. That represents roughly 27% lower GPU compute cost per token at the largest tested step size.

That’s an edge for applications where large inputs are routine for every request.

Better inference without buying more GPUs

Backboard improves performance across both decode and prefill, but that helps beyond simply making individual requests faster.

When you purchase or provision GPUs, you’re effectively paying for a certain amount of compute capacity. How efficiently your inference engine uses that capacity determines how much useful work you get for your investment.

Backboard improves that equation in two ways.

More output for the same GPU expenditure

With 1.09×–1.12× the decode throughput per GPU, you can generate more output tokens from the same hardware. Under comparable utilization and GPU pricing, this translates into approximately 0.89×–0.92× the GPU compute cost per output token.

More efficient processing of large inputs

With up to 1.37× the prefill throughput per GPU, you can supply tangibly greater input even at larger step sizes. On a GPU-hour basis, that corresponds to approximately 0.73× the compute cost per prefill token under the same usage assumptions.

These are stage-specific GPU compute comparisons, not guarantees of equivalent reductions in your total serving costs. Actual savings will depend on your workload, utilization, scheduling, networking, and other infrastructure expenses.

Nevertheless, the underlying incentive is straightforward: you get more useful inference work from the same GPU allocation.

For businesses scaling AI applications, that can mean serving more customers, supporting more intensive workloads, and wringing the most from existing infrastructure before you need to upgrade.

More performance from the same hardware

This comparison shows that the inference engine itself can define your hardware’s real-world performance.

The four B200 GPUs didn’t suddenly become faster. They just lived up to their potential.

For you, that translates to more capacity for inference, faster generation, and more efficient processing of large inputs.

Backboard is built to help you get more out of the hardware you already have.

Methodology

We benchmarked DeepSeek-V4.1-Flash using four NVIDIA B200 GPUs, with both Backboard and DeepInfer operating on the same machine.

Both engines were tested alternately, point by point, on November 7, 2026.

The model supports a maximum context window of 1,048,576 tokens. Both engines were configured with this context-window limit, while each benchmark request used 131,072 tokens of context, matching the setting in DeepInfer's own benchmark.

Decode

For decode testing, we used DeepInfer's examples/benchmark.py and its default settings:

• Three speculative draft tokens at 75% acceptance.

• Decoder bounded replay of 140 tokens.

• Four B200 GPUs in EP4 configuration.

We measured aggregate output throughput per GPU and per-user generation speed across six concurrency levels, from 8 to 256 requests per GPU.

Prefill

For prefill testing, we measured non-cached input throughput per GPU at five step sizes:

• 4,096 tokens per GPU per step.

• 8,192 tokens per GPU per step.

• 16,384 tokens per GPU per step.

• 32,768 tokens per GPU per step.

• 65,536 tokens per GPU per step.

The prefill comparison used four GPUs in EP4 configuration, 384 + 32 experts, 131,072 tokens of context per request, and a decoder-bounded replay of 140 tokens.

DeepInfer configuration and comparison

We used DeepInfer version 2026-11-03, commit a679bb8.

Our cloud machine didn’t have a GPUDirect RDMA network card. DeepInfer uses this type of card to access its approximately 197 GB Engram lookup table, which is stored in host memory and accessed during every forward pass.

As these cards can be difficult to obtain, we made small environment-specific patches to read the table over PCIe instead. These patches didn’t change the model or kernels, but PCIe access is slower at larger prefill sizes. Consequently, our DeepInfer prefill results decline at 32K and 64K.

For additional context, we also compared Backboard's prefill results with DeepSeek's published DeepInfer measurements for the same four-GPU setup, 384 + 32 experts, and 131,072 tokens of context per request. Values were read from DeepSeek's published figure and are approximate.

Against those published results, Backboard's prefill throughput ratios are approximately:

This provides an important qualification to the direct comparison: while Backboard beats our patched DeepInfer run at every tested point, the lead over DeepSeek's approximate published prefill results narrows to parity at 32,768 tokens and a slight deficit at 65,536 tokens. The largest 1.37× result should therefore be understood specifically as a comparison against our PCIe-patched DeepInfer run.

This benchmark measures the inference engine itself. Backboard's serving scheduler, context-parallel prefill, and prefix cache aren’t exercised. Results represent a single run per benchmark point.

Our objective was to isolate engine-level performance and compare both systems on the same GPU hardware, using DeepInfer's own benchmark approach and clearly identifying the environmental differences that affect interpretation of the results.