We recently ran Backboard R-CLI across all 89 Terminal-Bench 2.1 tasks using Kimi K3 as the primary model and DeepSeek V4 Flash for subagent work. It solved 81 of 89 tasks, producing 91.0% accuracy for a total model cost of $28.72.
That is a great headline. It is also not the most interesting thing we learned.
Kimi does not score 91% every time. In fact, one of the things we are seeing repeatedly with Kimi and other open-weight models is meaningful variance between runs. Sometimes the results are exceptional. Sometimes they come down materially. That matters because it would be easy to look at a 91% score and conclude that Kimi K3 is simply a 91% Terminal-Bench model. We do not think that is the right conclusion.
The run itself was also intentionally different from an official leaderboard submission. It used one attempt per task, while Terminal-Bench requires at least five attempts per task for an official submission. The report calls this out explicitly and estimates a wider confidence interval around the result.
What interests us is not the peak score. It is the performance range we are starting to see from open-weight models when they are put inside a well-engineered agent system. Even when Kimi comes down from a particularly strong run, we are seeing it operate in a range that is increasingly competitive with systems built around frontier models. That is a much bigger story.
For the last several years, AI performance has largely been discussed as a model problem. If you wanted a better system, you used a better model. The model was the product, and almost everything around it was treated as plumbing.
We think that assumption is starting to break.
At Backboard, we have been spending a disproportionate amount of engineering effort on the harness around the model. That includes how context is managed, which tools the model sees, when work is delegated, how models are routed, how long the system should keep working, how much it should spend, what happens when a tool fails, and how the system keeps track of what actually happened.
Those decisions sound small individually. Across hundreds of agent steps, they compound.
The Kimi run is a good example. R-CLI did not use one model for everything. Kimi K3 handled the primary reasoning, while subagent work could be routed to DeepSeek V4 Flash. That allowed lower-stakes delegated work to be handled separately from the reasoning that determined what the agent should actually do.
We also deliberately reduced the tool surface available to the model. Ten tools were removed from the schema because they either violated benchmark constraints or had been measured to make performance worse. The result was a smaller prompt, fewer branches for the model to choose from, and fewer opportunities to select a tool that would not advance the task.
The harness also used turn timers, budget-aware spend nudges, and a cache-first approach to context. In the 91% run, 71.8% of all input tokens were served from cache, and median context occupancy was only about 1.5% of the available model window.
None of those techniques change Kimi’s weights. They change the environment in which Kimi operates.
That distinction is important. A model can have significant latent capability and still perform poorly if the system around it repeatedly gives it too much context, too many tools, the wrong tools, no economic awareness, poor state management, or no sensible way to delegate work.
The model did not suddenly get smarter. The system got better at extracting the intelligence that was already there.
This is also why the cost numbers are interesting, although we think cost needs to be interpreted carefully. The entire 89-task sweep cost $28.72. Dollar comparisons between AI systems are messy because model pricing varies, caching policies differ, and subscriptions can distort the economics.
Token consumption gives us another signal. The Backboard run used roughly 0.24 million tokens per trial. Several competing configurations recovered in our analysis consumed more than one million tokens per trial, while the highest-scoring submission in the comparison used roughly 2.65 million tokens per trial.
That difference is harder to dismiss as pricing. If two systems are attacking the same task and one needs dramatically more model activity to get there, the harness is doing meaningful work.
This is where we think AI benchmarking needs to evolve too. A single score tells you very little about the distribution behind it.
If an agent scores 91% on one run, 87% on another, and 83% on another, what is its actual capability? All three results matter. Peak performance tells us what is possible. The lower end tells us what we can depend on. For production systems, the second question may eventually matter more than the first.
That is particularly relevant for open-weight models. Their ceiling is becoming surprisingly high. Their consistency still has room to improve.
But inconsistency is not necessarily a model problem alone. It creates an engineering opportunity. Better routing can reduce variance. Verification can catch weak trajectories. Smarter retry policies can recover failures. Post-training can improve specific behaviors. Evaluation data can be fed back into the system. Different models can be selected for different stages of a task.
The goal should not simply be to produce the highest score possible. The goal is to make high performance repeatable.
That is part of why our official Terminal-Bench 2.1 result of 85.4% matters to us. The 91% experimental run shows the ceiling we are beginning to see with open-weight models. The official result gives us a much stronger signal that the harness itself can perform consistently at the top end of the benchmark.
And we intend to make that harness available.
We will be open-sourcing Backboard R-CLI, along with the desktop version we are building around the same architecture. That is important to us because benchmark claims are easy to make when nobody can see the machinery. We want people to see it.
R-CLI was designed so that the same code path used by a person also produces machine-readable trajectories, durable session logs, tool histories, errors, and detailed usage records. There is no separate benchmark build hiding behind the result.
Developers will be able to run their own models through it, inspect how it works, reproduce our experiments, modify the architecture, and challenge our conclusions. We think that is good for us and good for the ecosystem.
Because the larger question here is not whether Kimi can score 91%. The larger question is what happens when open models become good enough.
There will always be another GPT, Claude, Gemini, Kimi, DeepSeek, or model we have not heard of yet. The frontier will keep moving. But if multiple models increasingly occupy the same broad capability range, then the scarce part of the system changes.
The advantage shifts toward whoever can take those models and make them more reliable, more efficient, more controllable, and more useful.
The competition moves above the model.
That is the bet we are making at Backboard.
The 91% score is exciting. The $28.72 cost is exciting. Our official 85.4% result is something we are proud of. But none of those numbers are really the thesis.
The thesis is that the gap between frontier and open-weight models is shrinking faster than most people expect, while the gap between an average harness and a great one may only be starting to open.
And if that is true, the next major advantage in AI may not come from owning the smartest model. It may come from knowing how to get the most out of all of them.
P.S. We open-sourced the harness: https://backboard.io/blog/r-cli-open-source
Rob Imbeault
SHARE