Jev vs Laya vs Von: what the benchmarks actually show

Search for Jev vs Laya vs Von and you get launch posts and re-summaries quoting whichever number suits the writer, most already out of date. TypeSafe AI launched Jev on September 15, 2026 (TypeSafe blog), and the open-source alternatives Laya, from Convai Innovations, and Von appeared soon after. Their published numbers have already changed at least once.

This post sorts the benchmark claims by who ran them, separates accuracy from calibration and latency, and ends with a checklist for deciding whether you need a System One model at all. At the end we point to a ten-lesson course that walks through all three hands-on.

What these models are

Jev, Laya and Von are decision models, not chat models. You send text (the state) plus closed questions, and get back typed answers with probabilities: a choice among options, a score, or a yes/no. TypeSafe's docs list the current model as jev-1.13.0, priced at $42 per billion input tokens, and say "Output tokens are free" (Models). Laya is a 421M parameter Apache-2.0 model (Laya card). Von is Apache-2.0 and says it is "Fully compatible with the TypeSafe /v1/systemone specification" (Von README), so the same request body can go to either.

Who ran each benchmark

Ask this first of any number in the comparison. There are four sources, and only one is not written by a model's own team.

• TypeSafe's eval scores Jev against "the average of GPT-6 Astra and Fable 5.1 as the reference answer", and the post calls its multipliers "the higher end of real world gains" (TypeSafe blog).
• Laya's 0.766 comes from laya-typed-decisions, which "was fine-tuned from convaiinnovations/laya on the benchmark's 1,200-case training split" (typed-decisions card). The base model scores 0.362, below the 0.461 majority baseline. The card also says its "Jev figures are third-party published, never measured here (no TypeSafe API access); sample sizes and prompts differ" (Laya card). The row where Laya beats Jev compares a model trained on the benchmark with a number that team never measured.
• The jabr benchmark is the closest thing to a neutral run: one harness, four models (Jev, Von, GLiNER2, Laya), raw per-case results in the repo (jabr/classifier-benchmark). We could not find who jabr is or whether they are affiliated with any vendor. With 78 cases, treat gaps of a few points as noise.
• The Von card's v2 numbers cover a larger suite, but they are published by Von's author, not by the benchmark's maintainer.

On the two shared harnesses, Jev leads on accuracy, and the open models trail by a margin that depends on the suite.

Numbers change between versions

Von's README reports 93.5% macro for Von-1.0 on the 78-case suite (Von README), the benchmark's own table lists Von 1.0.1 at 0.930, and the Hugging Face card now leads with 72.0% on v2. A number quoted without its version and suite is half a number.

Calibration is a separate question from accuracy

A System One model is worth using when you can gate on its probability. That needs calibration, usually measured as expected calibration error (ECE, lower is better).

The published evidence is thin. TypeSafe says Jev is trained with "Reinforcement Learning for Calibrated Decisions (RLCD)" and is "calibrated: higher confidence means higher accuracy" (TypeSafe blog), but the method is not published. Laya's card reports raw ECE of 0.175 (base) and 0.213 (fine-tuned), and separately a temperature refit from 0.466 to 0.081 (Laya card). Those come from different tables, so don't combine them. The fine-tuned card calls the model "Still over-confident" and tells you to refit temperatures "on your own held-out data" (typed-decisions card). Von's card reports a fitted temperature (T = 1.1692) but no ECE figure (Von card).

Our reading: no independent calibration measurement exists for any of the three yet. Measure it on your own labeled traffic.

Latency depends on where the model runs

Jev's blog gives "70ms-500ms" end to end (TypeSafe blog); the jabr run measured about 302 ms. Laya reports 39.5 ms for one English question on a Tesla T4 (Laya card). Von reports about 18 ms on Apple MPS but about 480 ms on CPU (Von README), slower than Jev over the network. Measure on your own hardware.

Maturity risk

Jev's rate limits "can change without notice" (Models), and TypeSafe's CEO, Diogo Almeida, told Latent Space "I'm extremely anti-public benchmarks," calling them "extremely gameable" (Latent Space). The open models' cards are being revised as fast as their weights. Pin versions and re-run your eval on every change.

When structured outputs from a frontier model are enough

Often. OpenAI's Structured Outputs "ensures the model will always generate responses that adhere to your supplied JSON Schema" (OpenAI), but the guide says nothing about returning a probability with the answer. And asking an LLM for its own confidence has a known problem: in the GPT-4 Technical Report, post-training raised ECE from 0.007 to 0.074.

Here is the checklist we use. It is our own heuristic, not a vendor rule.

1. Does any part of the output need free text? Use an LLM for that part.
2. Will code branch on a probability? If a valid label is enough, an LLM with structured outputs is a fair baseline. Include it in your eval.
3. Can the input leave your network? If not, only Laya or Von remain.
4. More than about 20 options in one question? Laya's card says to "Keep choice questions under ~20 options" (typed-decisions card), and on Banking77 it reports 0.425 against Jev's 0.870, on different label counts (Laya card).
5. Run one fixed labeled set through every candidate. Report accuracy next to the majority baseline, ECE with bin count and sample size, p95 latency, and cost per 1,000 requests.

Step 5 decides it. Published numbers tell you which candidates are plausible. They can't tell you which one wins on your data.

Learn it hands-on

Jev by TypeSafe AI: System One Models & Open-Source Alternatives is a ten-lesson course in six modules, about five hours. It starts with calibration, ECE and temperature scaling, then covers the Jev API and its three primitives, confidence-gated routing, the cookbooks and known limits, and a critical read of the CEO's Latent Space interview. Two lessons run Laya locally and stand up Von as a drop-in /v1/systemone server. The last module has a decision guide with a runnable cost and latency calculator, and a capstone: a swappable ticket-triage pipeline that compares accuracy and ECE across Jev, Laya and Von on a labeled eval set.