Rob Imbeault
·
We made full 16-bit Qwen 2.54x faster TTFT than SGLang's FP8
Comparing Backboard with the SGLang FP8 baseline published by InferenceX for Qwen3.8-Flash-Next on H100.
At 16 concurrent requests, Backboard running the full 16-bit model delivered 24% higher total token throughput, 21% higher output token throughput, and 2.9× faster p90 time to first token than the SGLang FP8 baseline in our comparison.
For a team serving Qwen on H100s, the practical question is whether it can keep the original model weights and still serve more traffic. At this load, our engine processed 9,914 tokens per second per GPU. The published FP8 configuration reached 8,013.
A public reference point
InferenceX, from SemiAnalysis, publishes inference performance measurements across hardware and serving configurations. Its H100 page for Qwen3.8-Flash-Next lists SGLang in FP8 and a peak measured total throughput of 8,013 tokens per second per GPU.
We compared our own measurements with that published baseline across five concurrency levels, from one to 16 simultaneous requests. InferenceX measured the SGLang baseline; our team measured Backboard.
The result at 16 concurrent requests
Total token throughput: 9,914 tokens/s/GPU for Backboard, full 16-bit, against 8,013 for SGLang, FP8. 24% higher.
Output token throughput: 87.1 against 72.2 tokens/s/GPU. 21% higher.
p90 time to first token: 1.21 seconds against 3.5 seconds. 2.9× faster.
Total throughput and output throughput describe different aspects of serving performance. We report them separately: 9,914 tokens/s/GPU is the total-throughput result, while 87.1 tokens/s/GPU is the output-generation result.
The difference was visible in the wait for an answer to start. At the 90th percentile, the first token arrived in 1.21 seconds on Backboard versus 3.5 seconds on SGLang, a difference of 2.29 seconds.
Performance across concurrency levels
Total and output throughput in tokens/s/GPU, Backboard against SGLang:
1 request: total 1,651 against 870 (+90%); output 17.3 against 9.7 (+78%).
4 requests: total 2,393 against 2,146 (+12%); output 26.1 against 23.9 (+9%).
8 requests: total 4,796 against 4,830 (−0.7%); output 45.7 against 46.0 (−0.7%).
12 requests: total 7,322 against 5,060 (+45%); output 67.1 against 48.4 (+39%).
16 requests: total 9,914 against 8,013 (+24%); output 87.1 against 72.2 (+21%).
The full tables and charts are on our inference research page.
Why the row at eight requests caught my eye
The first number I asked our team about was the one with a minus sign. At eight concurrent sessions, SGLang led us by 0.7% on throughput. We were ahead at four and again at 12. I wanted to know what that small lead meant for someone waiting on an answer.
At eight sessions, Backboard got the answer started sooner. Our p90 time to first token was 1.13 seconds against SGLang’s 1.40. Once generation started, SGLang had the higher reported p90 streaming rate, at 138 tokens per second per user against our 62.1. Both numbers matter to the person at the keyboard.
When we doubled the load to 16 sessions, Backboard led on both measures. The p90 wait for the first token was 1.21 seconds on Backboard and 3.50 seconds on SGLang. After that, our full 16-bit model streamed at 47.6 tokens per second per user against 14.4 for SGLang FP8. That’s 3.3× the per-user streaming rate.
I went in wanting to explain a 0.7% gap. If I were choosing an engine today, I’d want to run the prompts my users actually send, at the load they actually create, and check how long they wait for an answer they can use.
A shorter wait before generation begins
Backboard’s p90 time to first token was lower at every concurrency level shown. At p90, 90% of measured requests reached their first token within the reported time.
1 request: 1.19 seconds against 10.6 seconds. 8.9× faster.
4 requests: 1.11 seconds against 1.6 seconds. 1.4× faster.
8 requests: 1.13 seconds against 1.4 seconds. 1.2× faster.
12 requests: 1.22 seconds against 3.1 seconds. 2.5× faster.
16 requests: 1.21 seconds against 3.5 seconds. 2.9× faster.
Across these tests, Backboard’s p90 first-token time ranged from 1.11 to 1.22 seconds. Its median first-token time ranged from 0.35 to 0.47 seconds.
What the precision difference tells us
The measured gains were achieved while Backboard served the full 16-bit model against an FP8 baseline. They demonstrate that reducing model precision was not required for Backboard to lead on these throughput and first-token measurements.
This benchmark measures serving performance. It does not establish a difference in answer quality between the two configurations, or isolate precision as the cause of the performance difference.
Bring us your workload
For cloud providers, enterprise AI platforms, and operators of regulated or secured infrastructure, the next step is to test the engine against the workload that matters to your operation.
Evaluate Backboard using your models, request patterns, and response-time requirements. Measure throughput, latency, workflow completion, and output quality against your current serving stack, then assess the deployment and licensing opportunity.
Source: Backboard’s supplied benchmark comparison, “Inference Performance Benchmark,” and the published InferenceX H100 baseline, accessed 1 November 2026. Percentage changes and speedups are rounded from the displayed values. Results apply to the configurations and workloads measured.