Backboard homeBlog

Rob Imbeault

·

TPU Power: Up to 8.1× Faster Time to First Token than vLLM

In a Qwen3.6-35B-A3B BF16 comparison, Backboard managed nearly twice the p90 token generation speed of vLLM-JAX across every tested concurrency level, while dramatically reducing first-token wait times under heavy loads.


With the strain of 256 concurrent requests (C256), Backboard's inference engine achieved an 8.1× shorter p90 time to first token (TTFT) than vLLM-JAX on Google’s high-throughput, high-efficiency v6e-4 TPU. The improvement was just as striking at the median: Backboard started generating responses in just 0.23 seconds, compared with 1.06 seconds for vLLM-JAX.


And faster starts were only part of the advantage. Backboard recorded 1.84× higher p90 token generation speed at that concurrency level. Across all six tested concurrency settings, its p90 generation speed was between 1.84× and 2.17× higher.


These results show how much the inference engine can influence the experience when running a model, especially as demand increases. Backboard's lead was already apparent at light concurrency and remained strong as the workload scaled to C256.


First-token latency: an advantage that grows with demand

Before a model can give you an answer, it has to respond. TTFT thus becomes crucial when you’re measuring interactivity.


Here, too, Backboard outperformed vLLM at every tested concurrency level when comparing p90 latency. The gap even widened as concurrency increased.


At C8, Backboard reduced p90 TTFT from 0.40 seconds to 0.22 seconds, a 1.82× leap over vLLM. At C64, it cut the time from 0.82 seconds to 0.41 seconds, exactly halving the wait.


As we’ve noted, the advantage became much larger under heavy load:

  • C128: Backboard's p90 TTFT was 0.60 seconds versus 2.59 seconds for vLLM-JAX, a 4.32× reduction.


C256: Backboard's p90 TTFT was 1.26 seconds versus 10.23 seconds, an 8.12× reduction.


That last result is especially relevant for agents and other interactive applications. A p90 first-token wait of more than ten seconds can make an application feel sluggish. Reducing that wait to just over one second makes the whole experience feel faster, even before accounting for the faster token generation that follows.


This shows that Backboard is effective at maintaining responsiveness as more requests compete for the same serving resources.


Roughly 2× the p90 generation rate across all six concurrency levels.


The comparison used the Qwen3.6-35B-A3B model at BF16 precision on the v6e-4 TPU, testing concurrency levels from 8 to 256.


At every level, Backboard recorded a higher 90th percentile (p90) token generation rate than vLLM-JAX.


Concurrency

Backboard p90 tok/s

vLLM-JAX p90 tok/s

Backboard advantage

C8

294.8

135.6

2.17×

C16

222.0

106.6

2.08×

C32

154.5

72.1

2.14×

C64

108.6

51.0

2.13×

C128

72.8

35.5

2.05×

C256

44.0

23.9

1.84×


It’s not about cherry-picked benchmarks, then; the sheer consistency stands out. Backboard maintained roughly twice the p90 generation speed throughout the test, rather than claiming the lead only in a narrow range.


For applications serving multiple users simultaneously, this translates into more responsive generation across your work, not simply best-case scenarios.


Faster median responses as the workload grows

The median TTFT figures reinforce this pattern.


At lower concurrency, the p50 measurements were effectively equal at the precision displayed in the benchmark: 0.05 seconds at C8, 0.06 seconds at C16, and 0.07 seconds at C32 for both engines.


The difference began to emerge at C64, where Backboard reduced median TTFT from 0.10 to 0.09 seconds.


At higher concurrency, the speedups were much more pronounced: up to 4.61× shorter wait times than vLLM.


Concurrency

vLLM-JAX TTFT p50

Backboard TTFT p50

Backboard advantage

C128

0.28s

0.13s

2.15×

C256

1.06s

0.23s

4.61×


Backboard's upper hand isn’t confined to the slower end of the response distribution. As the workload rises, it shrinks the delay.


The combination of faster median responses and a much stronger p90 result is telling. It points to an engine that can improve both typical interactivity and the experience of requests that would otherwise appear to stall.


A strong TPU platform that wrings out more engine performance

The data also provides a good practical demonstration of Google's four-chip v6e-4.


The configuration provides a solid foundation for this model-serving comparison with up to 917 TFLOPS of peak compute at BF16, 1.6 TB/s of memory bandwidth, and a reputation for a quick TTFT. It’s a strong pick if you want more efficiency than setups with dedicated GPUs.


But hardware is only part of the equation. The software responsible for scheduling, executing, and serving requests directly impacts how effectively you use an accelerator.


That’s clear with our comparison. On the same Google TPU v6e-4 platform, Backboard's engine extracted higher p90 generation speeds and shorter first-token waits than vLLM-JAX. The faster performance came from the serving engine rather than swapping hardware.


That’s the opportunity Backboard is focused on: making existing compute work harder through better inference execution, without making precision sacrifices.


Better performance where responsiveness counts

Like we’ve outlined, the Qwen results show mounting leads for Backboard over vLLM-JAX on the TPU.


Backboard recorded 1.84×–2.17× higher p90 token generation speed at all six concurrency levels. Its p90 time to first token was lower in every test, with the advantage reaching 8.1× at C256. Median TTFT also shined.


When your teams are building interactive AI applications, these differences have very real implications. Faster first tokens reduce the time your users spend waiting for an agent to begin. Higher generation speeds keep responses moving. And better performance under concurrency makes it easier to serve more simultaneous demand without sharp deteriorations in responsiveness.


The value is straightforward: strong hardware provides the foundation, but the inference engine determines how much performance applications can pull from it. In this test, Backboard made the most of the v6e-4 platform.


Benchmark scope: Qwen3.6-35B-A3B, BF16 precision, Google TPU v6e-4, concurrency levels C8–C256. p90 token generation speed is expressed in tokens per second, where higher is better. TTFT is expressed in seconds, where lower is better. Results reflect this specific benchmark configuration and shouldn’t be interpreted as universal performance claims across all models or hardware.

Backboard

Ottawa, Canada
Burlingame, California

Backboard © 2026