Our first tool-use benchmark measured nothing, and that was the useful part

Every time you send a message in LearnOS, a model makes a decision before it writes a word. Should this turn call a tool — make a flashcard, append a note, generate a deck, look up where you left off — or should it just answer? Twelve tools, and the model picks one or none.

That decision is cheap to get wrong in a way that is expensive to notice. A model that fires too eagerly turns "thanks, that cleared it up" into a note you never asked for. A model that fires at the wrong target writes a flashcard onto the wrong lesson, and nothing in the product tells you it happened.

So: which model should make it? We couldn't answer from published benchmarks. They measure function calling against generic toolsets, not against our twelve tools, our system prompt, and a catalog where some lessons are addressable and some aren't. We had to build our own.

What it scores

The benchmark scores the routing decision and nothing else. Nothing is ever dispatched — no database writes, no side effects — so a run is hermetic and repeatable. The system prompt and tool schemas are imported from production rather than copied, so the thing under test is the real thing.

Only the world is synthetic: three courses, ten lessons, two slide decks. The important part is an asymmetry we built on purpose. Five of those ten lessons are retrievable — they appear in the model's context, so their identifiers are knowable. The other five exist only as titles in the course catalog, with no identifier anywhere.

That mirrors production exactly. The catalog block ships lesson titles, not bodies, and not ids. So when a learner says "make a flashcard for the Promises in Depth lesson" and Promises in Depth has no id in context, there is no correct tool call available. The only right answer is to decline.

It turns out that is the single most discriminating thing you can ask a model to do.

The first version measured nothing

We started with 40 tasks and six models. The table looked reasonable — a ten-point spread, a clear top and bottom.

Then we ran the paired significance tests.

0 of 15 model pairs separated at p < 0.05

Zero. Not one pair. The entire visible spread was noise, and every ranking we might have drawn from that table would have been a coin flip dressed up with a decimal point.

The diagnosis was embarrassing once we looked. A quarter of the task budget went to straightforward requests — "what lessons are in this course?", "save this as a flashcard" — and every model scored 100% on all of them. Those tasks cost money and produced no information. Another quarter was nearly as saturated. We had built an instrument that spent most of its resolution measuring things nobody disagreed about.

There is a general lesson in that, and it is not about models. If your eval has a category where everything passes, that category is not evidence that your system is good. It is evidence that you are not measuring there.

Rebuilding around disagreement

The second version is 150 tasks, and the categories were chosen from where models had actually diverged rather than from what felt comprehensive:

ambiguous         38   under-specified turn: follow the documented default, or ask?
negative          40   conversational turn that must not fire a tool
missing_referent  22   named entity whose identifier appears nowhere
scope_shift       14   scope moves away from the pinned lesson mid-turn
multi_intent      12   two asks, one available tool call
negation          12   explicit instruction not to do the thing described
smoke             12   regression guards, excluded from the ranked figure

The old saturated cases survive as twelve smoke items. They still catch harness breakage, but they are reported separately and never used to rank — because items every model passes inflate every model equally and compress the spread you are trying to see.

At 150 tasks, nine of twenty-one pairs separate. The instrument works now. It resolves a top group that is genuinely indistinguishable, and two models clearly below it.

The result we didn't expect

Here is the part that changed how we think about this.

We added a severity dimension late, almost as an afterthought: for every tool call, what would actually happen if it were dispatched? We grounded it in the real write path rather than guessing. Lesson identifiers carry a foreign key, and the write operations call an access guard first, so an invented UUID — or a course id shoved into a lesson id slot — gets rejected. It is inert. The genuinely dangerous case is narrower than we assumed: a valid identifier pointing at the wrong lesson, which passes every guard and succeeds silently.

Scoring that produced two rankings that disagree almost completely.

The model that finished last on tool selection did not write to the wrong entity once, across 450 calls. The model that finished sixth did it seven times.

The reason is structural. The last-place model is a typed-decision model: its identifier slots enumerate real entities, so fabricating a UUID is not something it avoids, it is something it cannot represent. When it fails, it fires at nothing, and the write guard catches it. The model ranked above it is autoregressive and fails the other way — it produces a plausible-looking identifier for a lesson that was never in its context, and sometimes that identifier belongs to a real, different lesson.

Same score on the leaderboard. Completely different blast radius.

What we're not doing

We are not naming a winner.

The top four models do not separate from each other at any conventional threshold — every pairwise test among them lands above p = 0.5. Treating a 0.5-point gap between them as a ranking would repeat exactly the mistake the 40-task version made. Where they do differ is cost and latency, by as much as sixteen-fold, and those are measured precisely. That is the axis to choose on.

We are also not switching the model LearnOS runs on the strength of this. The default model variable feeds six call sites — the tutor, review chat, flashcard generation, review synthesis, audio scripts, the CLI — and this benchmark measures one of them. Grounded answering, citation validity and course authoring are separate categories that do not exist yet. One dimension does not get to decide six.

The method is open

Results, task file, scorer and the full pairwise matrix are published at /research/toolbench, marked preliminary, with the limitations listed on the page rather than buried.

Two things are still missing before it deserves to be treated as a leaderboard. There is no held-out split, so it cannot yet detect a prompt tuned against it — which is a far likelier failure than any lab training on our twelve tools. And the under-specified category, which carries a disproportionate share of the result, still encodes one author's judgment about what should happen when a learner is vague. Contestable items accept several defensible answers, but somebody still decided which ones were defensible.

We would rather say that out loud than publish a clean number.