Tool-use routing: which model should decide when to act?
8 models, 150 tasks, three runs each. The accuracy ranking and the safety ranking disagree almost completely — which is why this page does not yet name a winner.
Ranked by tool selection
| Model | Tool selection | Arg accuracy | Misdirected | Spurious | Inert | Latency p50 | Cost |
|---|---|---|---|---|---|---|---|
| openai/gpt-5-nano | 93.5%95% CI 88–97 | 94.4% | 2 | 3 | 2 | 8,881 ms | $0.214 |
| google/gemini-3.8-flash | 92.8%95% CI 87–96 | 96.5% | 2 | 1 | 0 | 3,544 ms | $1.559 |
| deepseek/deepseek-v4-flash | 92.0%95% CI 86–96 | 98.1% | 2 | 1 | 0 | 3,499 ms | $0.097 |
| z-ai/glm-5.3-flash | 90.6%95% CI 85–94 | 96.6% | 4 | 4 | 0 | 5,612 ms | $0.200 |
| inclusionai/ling-3.0-flash | 87.0%95% CI 80–92 | 96.1% | 2 | 1 | 0 | 1,106 ms | $0.053 |
| google/gemini-3.5-flash | 85.5%95% CI 79–90 | 96.4% | 2 | 2 | 0 | 2,685 ms | $2.869 |
| google/gemini-3.5-flash-lite | 81.9%95% CI 75–87 | 88.7% | 7 | 8 | 8 | 841 ms | $0.317 |
| typesafe/jev-1.13 | 76.8%95% CI 69–83 | 79.7% | 0 | 13 | 33 | 177 ms | $0.069 |
Severity counts are tasks, worst case across three runs. Misdirected: the write landed on a real but wrong entity — it passes every application guard, so it is silent corruption. Spurious: a valid write on a turn that wanted none. Inert: rejected before any mutation. Cost covers every call, input and output, at live prices. Latency comes from a serial pass, not from timings taken under concurrency.
What each model costs you
Accuracy against price, and accuracy against latency. A model on the dashed frontier is one nothing else beats on both measures at once; an open ring means something in the field is strictly better and cheaper, or strictly better and faster.
Price is normalised per thousand routing decisions so the axis means something at your own volume. Latency is full-completion from a serial pass — for a streaming interface the number a person feels is time-to-first-token, which is not measured here.
Accuracy and safety rank almost inversely
The model ranked last on accuracy never once writes to the wrong entity. The model ranked seventh does it 7 times.
jev-1.13 finishes last and is the safest arm on the board. It fires a tool on 21 of the 22 tasks naming an unresolvable lesson — a real failure. But 18 of those calls carry an empty identifier, because a typed-decision model structurally cannot fabricate a UUID. In production the write guard rejects them before anything is written.
gemini-3.5-flash-lite fails differently. It invents identifiers, and it puts a course id into a lesson id slot — the exact confusion the system prompt spends a paragraph warning against. Most are caught too, but 7 land on a real, wrong lesson and succeed silently.
A leaderboard with only the accuracy column would tell a reader the opposite of the truth about which model is safer to put in front of a database.
10 of 28 pairs actually separate
An earlier 40-task version separated zero pairs — the apparent spread was noise. At 150 tasks the instrument resolves a real structure: a top group that is mutually indistinguishable, and two models clearly below it. Because every arm answers an identical task set the comparison is paired, which makes an exact McNemar test both correct and far more powerful than asking whether two intervals overlap.
How the numbers are kept honest
- n is tasks, not runs. Three runs of one task are the same question asked three times, and on near-deterministic models they are almost perfectly correlated. Each task collapses to its majority outcome; repeats measure flakiness rather than inflating the sample.
- Severity is grounded in the real write path. Lesson identifiers carry a foreign key and the write operations call an access guard, so an invented or cross-namespace id is rejected — inert, not corrupting.
- Cost counts input and output. One arm bills nothing for completions; omitting output would quietly have flattered it.
- Latency comes from a serial pass. Timings taken during a concurrent sweep include queueing behind our own requests and overstate fast models.
- Errors are misses, never silence. Without that rule an arm that fails every request scores full marks on the negative set.
- Tasks are validated against the live tool schemas at startup, so a refactor cannot rot the task file unnoticed.
- A held-out split guards against our own tuning. 40 tasks in the same fixture world, never committed. It exists because the likely failure is not a lab training on our tool schemas — it is us tuning the production prompt until the public file goes green. It is run to measure, never to iterate on.
What this cannot tell you
Every limitation we know of, on the page rather than in a repository file. If you are citing a figure from the table, these are the caveats that come with it.
- Only routing is measured. Answer quality, citation validity, retrieval and course authoring are separate categories that do not exist yet. This evidence cannot settle a change to a shared default model, because that setting drives six call sites and this measures one.
- The under-specified tasks encode one author’s judgment about what should happen when a learner is vague — and that category carries a disproportionate share of the result. Contestable items accept several defensible answers, but somebody still decided which ones were defensible.
- 150 tasks is still a small sample. Read the paired p-value, not the gap between two headline numbers. The top group here does not separate, and treating its ordering as a ranking would repeat the mistake the 40-task version made.
- Wrong-but-real identifiers are only caught where a task pins the expected target. The integrity check verifies an id is real and of the right kind, not that it is the right entity, so the misdirected count is a floor rather than a total.
- Latency is full-completion, not time-to-first-token. For a streaming tutor the second number is the one a learner feels, and it is not measured here.
- Tool order is fixed. The twelve schemas are always presented in the same order, so position bias is unmeasured.
- One fixture world. Three courses, ten lessons, five modules. A catalog of a very different shape — hundreds of courses, or deeply nested modules — might rank models differently.
Reproduce it
Adding a model is one argument — every arm is an OpenRouter model id, so no adapter or handler class is needed.
pnpm bench:tools -- --models <id,id,...> --k 3 --latency-pass 15
pnpm bench:report -- --publish bench/results/toolbench.jsonThe report is derived from stored per-task results rather than from the sweep, so a scoring change can be applied retroactively to past runs without spending on the API again.