Most agent demos are one run, recorded on the good take. Microsoft just released a tool that quietly says the quiet part out loud: that tells you almost nothing.
ThinkingBox, now available on Hugging Face, evaluates agents on what they actually accomplished, not just which tools they called. It also repeats each task 20 times to check whether the agent gets there consistently. Two ideas, and both are overdue.
Tool calls are not outcomes
A lot of agent evals still grade the trajectory. Did it call the search tool? Did it call the right API with the right arguments? That feels rigorous, but it rewards agents for looking busy.
An agent can call every expected tool in the expected order and still hand back a wrong answer. Another agent can take a weird route and nail the result. If you grade the route, you punish the second and reward the first.
Outcome-based scoring flips it. Did the refund get issued? Is the record in the database correct? Does the generated invoice match the order? You check the end state, and you don't much care how the agent got there.
This is harder to build, because you need a way to verify end state for every task. But that work is the eval. If you can't say what "done correctly" looks like in a checkable way, you probably can't ship the feature either.
Twenty runs beats one run
The second idea matters more. Agents are non-deterministic, so a single pass is a sample of one.
Say your agent completes a task 90% of the time. That sounds fine. Now chain five such steps in a workflow and the odds of the whole thing working drop to about 59%. Run it for a customer once and you'll probably see it work. Run it for a thousand customers and you'll meet the other 41% in your support inbox.
Repeating each task 20 times gives you a distribution instead of an anecdote. You find out which tasks are rock solid, which are coin flips, and which pass on Tuesday and fail on Wednesday. The coin flips are where your production incidents live.
What we'd do with this
You don't need ThinkingBox specifically to steal the approach. Here is the version we'd run on any agent we build for a client:
- Write checkable outcomes first. For each task, define the end state in code. A function that returns true or false beats a rubric a human has to read.
- Run every task at least 20 times. Track pass rate per task, not one overall number. An 85% average can hide a task that passes 40% of the time.
- Treat the weakest task as the score. Your agent is only as reliable as its flakiest important workflow.
- Rerun on every model or prompt change. A new model version is a dependency bump. Replay the suite before you swap.
- Budget for the cost. Twenty runs per task is real money in tokens. Run the full sweep nightly, and a smaller sample on each pull request.
Teams will push back that this is slow and expensive. It is. It's also cheaper than the week you spend debugging an agent that "worked in the demo."
The takeaway
The interesting part of this release isn't the tool. It's that a big vendor is publishing the assumption that single-run, tool-call-counting evals aren't good enough. If you're building agents and your test suite is a handful of happy-path runs, you have a demo, not a test suite.
We're here to help founders and teams design and build digital products that are built to scale with you, not slow you down. If you're looking to build something, get in contact with us today!