System One Models and Calibration

A System One model is a model built for software rather than for people: you hand it unstructured state (a customer email, a transaction log, a policy document), ask it a set of typed questions, and get back structured answers, each with a probability attached, that your code can branch on directly. TypeSafe AI, which uses the term for its Jev model (borrowing the System 1 / System 2 framing from Kahneman's Thinking, Fast and Slow), describes it as "a frontier-intelligence function call: unstructured state in, typed probabilistic decisions out" (TypeSafe launch blog). Two properties make this category different from "an LLM with JSON mode." First, decisions are non-autoregressive: many independent questions are answered in parallel, in one request, rather than being generated token by token. Second, the probabilities are meant to be calibrated: a 0.8 should be right about 80% of the time. The second property matters most, because a probability you can't trust is worse than no probability at all. This lesson covers both ideas. You will compare Jev's (unpublished) design with Laya's openly documented encoder-plus-decision-head design, learn how to measure calibration with reliability diagrams and Expected Calibration Error (ECE), see why chat-style post-training tends to hurt calibration, and then run a temperature-scaling demo that repairs an overconfident model.

System One versus System Two — decisions for code, not text for humans

The name comes from Kahneman's split between fast, intuitive "System 1" thinking and slow, deliberate "System 2" thinking. TypeSafe positions its models as "a new class of frontier models built to make fast, structured decisions that software can use directly," optimised for automation rather than conversation (launch blog). The docs put it more bluntly: "System One models do not write replies, produce code, or generate explanations of their reasoning. You define the possible answers through primitives" (docs: System One).

That changes who the consumer is. A chat LLM produces a string for a person to read. Even with structured outputs, what you get back is text that happens to parse, and nothing in it says how sure the model is. A System One model "evaluates a state and returns typed answers and probabilities" (docs: System One). The answer space is fixed ahead of time by the question type. Jev has three primitives: Choice (pick one of the listed options), Score (rate against criteria), and Noul (judge whether a statement is true) (docs: Introduction). The next lesson, 02-jev-in-practice/01-api-state-and-primitives, covers each one in detail. The point for now is that the output is a distribution over a closed set, not free text.

TypeSafe's case against the "just ask the LLM for a confidence" workaround is that "even if prompted for a confidence estimate, models tend to be overconfident and inconsistent" (launch blog). The calibration sections below show how to test that kind of claim yourself instead of taking it on trust.

Non-autoregressive decisions — many questions, one pass

An autoregressive LLM produces its answer one token at a time, and each token needs its own forward pass. If you ask it ten classification questions, you either pay for ten sequential answers inside one long generation or make ten separate calls. The TypeSafe docs describe the System One alternative: "Every question is evaluated in parallel and in isolation against the same state in one go. Adding questions barely changes the response time" (docs: Introduction). The launch post says Jev "generates all outputs in a single query." It attributes this to "a new model architecture, parallel sampler for maximum efficiency, and training method we call Reinforcement Learning for Calibrated Decisions (RLCD)" (launch blog).

Two caveats before you lean on this. First, TypeSafe has not published Jev's architecture. "Parallel sampler" is a product description, not a paper, so don't assume Jev works like any particular open design. Second, the vendor-reported speed figures (end-to-end responses of "70ms-500ms" against "3 to 329 seconds" for frontier LLMs, and "193.6x faster, 444.6x cheaper" on TypeSafe's own workflow evals) come from the launch post (launch blog). Treat them as vendor-reported until you have measured them on your own workload.

The diagram contrasts the two execution shapes.

The questions are answered "in isolation," so one answer can't influence another the way earlier tokens influence later ones in a single generation (docs: Introduction). This matters when you design the questions: anything a question needs has to be in the state or in the question itself, not in another question's answer.

An open reference design — Laya's encoder plus decision heads

Jev is a black box, so it helps to study an open System One model whose design is documented. Laya, from Convai Innovations and released under Apache-2.0, is a 421M-parameter model. It combines "ModernBERT-large (395M, bidirectional, fully fine-tuned)" with a decision head built from "2 transformer layers, an option-marker scorer, and an act/escalate head" (Laya model card). ModernBERT is an encoder-only transformer with "a native 8192 sequence length," trained on 2 trillion tokens and described by its authors as "the most speed and memory efficient encoder" of its generation (Warner et al. 2024). An encoder reads the whole input bidirectionally in one pass and generates nothing, so it is a natural backbone for scoring a fixed set of options.

The key trick is how options are scored. In Laya, "every option is scored at its own [MASK] token, then softmaxed over that question's options," so the answer space is "defined at request time, so new schemas need no retraining" (Laya model card). In practice you put the state, the question, and each candidate option (each marked by its own mask token) into one sequence. The encoder produces a contextual vector at every marker, the scorer turns each vector into a single logit $z_k$, and the probability of option $k$ among $K$ options is

$$p_k = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}}.$$

That softmax over logits is exactly the quantity calibration methods work on, which is why the rest of this lesson is about logits. Laya's model card also says the model "ships over-confident" and needs temperature refitting on your data (Laya model card). The lesson 04-open-source-alternatives/01-laya covers installation, routing, and fine-tuning. Here Laya is only the concrete example of the architecture.

Calibration — what it means for a probability to be honest

A classifier is calibrated when its stated confidence matches its long-run accuracy. The TypeSafe primer gives the plain version: in a well-calibrated model, outcomes assigned probability 0.2 happen about 20% of the time, those assigned 0.8 happen about 80% of the time, and "higher probability should correspond to a greater chance that the answer is correct" (docs: ML primer). Formally, following Guo et al. (2017), take a model that predicts class $\hat{Y}$ with confidence $\hat{P}$. It is perfectly calibrated if

$$\mathbb{P}\big(\hat{Y} = Y \mid \hat{P} = p\big) = p \quad \text{for all } p \in [0,1].$$

Calibration and accuracy are separate properties. A model can be 95% accurate and still badly overconfident, or 60% accurate and perfectly calibrated, as long as it says it's about 60% sure. For a System One pipeline, calibration is the property that matters. Code that auto-approves any refund with $p > 0.9$ assumes those decisions really are right at least 90% of the time. TypeSafe turns the distribution into a single confidence number. For three options that number is (3 × largest probability − 1) / 2, which measures how concentrated the distribution is. The docs then suggest acting automatically at high confidence, proceeding with caution at medium, and routing to a human at low (docs: Confidence). That tiering is only as good as the calibration underneath it. Lesson 02-jev-in-practice/02-confidence-routing-and-patterns builds these gates.

Guo et al.'s main empirical finding is that "modern neural networks, unlike those from a decade ago, are poorly calibrated." They trace the problem to factors such as depth, width, weight decay, and batch normalization, and deep classifiers usually err toward overconfidence (Guo et al. 2017). So a raw softmax output shouldn't be read as a calibrated probability until you have checked it.

Reliability diagrams and Expected Calibration Error

You can't check calibration on a single prediction, only on many. The standard tool is the reliability diagram (Guo et al. 2017). Sort $n$ held-out predictions into $M$ equal-width confidence bins $B_1, \dots, B_M$ (for example $[0, 0.1), [0.1, 0.2), \dots$). For each bin, compute the empirical accuracy and the mean confidence:

$$\mathrm{acc}(B_m) = \frac{1}{|B_m|}\sum_{i \in B_m} \mathbf{1}(\hat{y}_i = y_i), \qquad \mathrm{conf}(B_m) = \frac{1}{|B_m|}\sum_{i \in B_m} \hat{p}_i .$$

Plot accuracy against confidence. A calibrated model sits on the diagonal. Bars below the diagonal mean overconfidence (it claims 0.9 but is right 0.75 of the time), and bars above mean underconfidence.

Expected Calibration Error turns the diagram into one number: the gap between accuracy and confidence in each bin, weighted by how many predictions fall in that bin (Guo et al. 2017):

$$\mathrm{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{n}\,\big|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\big|.$$

ECE is 0 for perfect calibration, and it is in the same units as probability. An ECE of 0.08 means that, on average, stated confidence is off by about 8 percentage points. It has known weaknesses. The result depends on the bin count $M$, and sparse bins give noisy estimates. It also only looks at the top-class confidence, so report the bin count and sample size with any ECE you publish. Guo et al. also report negative log-likelihood (NLL), $-\frac{1}{n}\sum_i \log \hat{p}_{i,y_i}$, a proper scoring rule that penalizes confident mistakes heavily. The demo below minimizes NLL.

Why chat post-training hurts calibration

The best-known evidence comes from OpenAI. The GPT-4 Technical Report (Figure 8, on an MMLU subset) says "the pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct). However, after the post-training process, the calibration is reduced." The figure shows ECE going from 0.007 for the pre-trained model to 0.074 after post-training, roughly ten times worse. The pre-trained model's next-token probabilities come from a proper likelihood objective. Post-training with RLHF optimizes for human preference ratings, and nothing in that reward asks the probabilities to stay honest.

TypeSafe's primer explains the problem the same way: reinforcement learning from human feedback "turned pretrained models into chatbots," but it can "reward sycophancy and confident-sounding hallucinations" and causes mode dropping, where the model settles on a preferred style (docs: ML primer). TypeSafe's answer is RLCD (Reinforcement Learning for Calibrated Decisions), which targets "answers with epistemically honest probabilities on System One tasks" instead of human preferences (launch blog). TypeSafe has not published the details of its RLCD recipe. Laya's card describes its own open RLCD variant: "the policy reports a distribution; exploration adds zero-mean Gaussian noise to the logits; the reward is a strictly proper scoring rule," so reporting honest probabilities maximizes expected reward (Laya model card). A reward built on a proper scoring rule (log loss or the Brier score) is the principled way to make "be calibrated" part of the training objective. Lesson 03-the-ceos-thesis/01-prod-not-god looks at how TypeSafe's CEO frames RLCD against RLHF and RLVR.

Even a model trained this way can drift out of calibration on your domain. That's where post-hoc fixes come in.

Temperature scaling — a one-parameter fix

Guo et al. found that "temperature scaling ... is surprisingly effective at calibrating predictions," often better than more complex methods (Guo et al. 2017). Take the logit vector $z$ and divide it by a single learned scalar $T > 0$ before the softmax:

$$\hat{p} = \mathrm{softmax}(z / T).$$

$T > 1$ flattens the distribution, which cures overconfidence. $T < 1$ sharpens it. $T = 1$ changes nothing. Dividing every logit by the same positive constant never changes which logit is largest, so accuracy is unchanged. Only the confidence moves. You fit $T$ by minimizing NLL on a held-out calibration set that the model was not trained on, then freeze it.

Laya shows how this plays out in practice. On the authors' typed-decisions benchmark (2,000 decisions across four workflows), its table lists a raw ECE of 0.175 for the English laya checkpoint and 0.213 for the fine-tuned laya-typed-decisions checkpoint, against 0.144 for Jev 1.13.0. A separate head-to-head table reports Laya (routed) at 0.081 post-temperature against 0.246 for Jev — so the two tables use different Jev figures and should not be mixed. It also says that refitting "one temperature per (question type, option count) moves mean ECE 0.466 → 0.081" for laya and 0.314 → 0.106 for laya-multilingual, with the instruction "do this on your own data before trusting the probabilities" (Laya model card). All of these are vendor-reported numbers from different tables on the same card. The lesson to take away is the method, not any single figure: fit one temperature per question type and option count, on your own held-out data.

Try it — measure ECE, then fix it with temperature scaling

This simulation builds a deliberately overconfident three-option "decision model." Each example has hidden true logits, and the label is drawn from their softmax, so the true softmax is perfectly calibrated. The model reports those logits multiplied by 2.5, the way an overconfident network would. The script splits the data in half, fits $T$ on the calibration half by grid search over NLL, and reports accuracy, NLL, ECE (10 bins), and a text reliability diagram on the test half. It uses only the standard library and a fixed seed.

With seed 42, test ECE should fall from about 0.19 to about 0.02 and the fitted $T$ should come out at 2.5, the distortion that was baked in. Accuracy stays at 0.590 throughout. In the $T=1$ reliability table, the 0.9–1.0 bin claims 0.96 confidence but is right only 0.75 of the time, which is the overconfidence pattern Guo et al. describe. Real models are rarely distorted by one clean scalar, so real-world gains are smaller. Laya's vendor-reported 0.466 → 0.081 needed a separate temperature per question type and option count.

Sources

• TypeSafe AI, "Introducing System One Models and Jev" (2026): https://typesafe.ai/blog/introducing-system-one-models-and-jev
• TypeSafe AI docs, "System One models": https://docs.typesafe.ai/concepts/system-one.md
• TypeSafe AI docs, "Introduction" (parallel questions, primitives): https://docs.typesafe.ai/introduction
• TypeSafe AI docs, "Machine learning primer" (calibration, RLHF): https://docs.typesafe.ai/introduction/machine-learning-primer.md
• TypeSafe AI docs, "Confidence": https://docs.typesafe.ai/confidence.md
• Guo, Pleiss, Sun, Weinberger, "On Calibration of Modern Neural Networks," ICML 2017: https://arxiv.org/abs/1706.04599
• OpenAI, "GPT-4 Technical Report" (2023), Figure 8 on calibration: https://arxiv.org/abs/2303.08774
• Warner et al., "Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder..." (ModernBERT, 2024): https://arxiv.org/abs/2412.13663
• Convai Innovations, Laya model card (2026): https://huggingface.co/convaiinnovations/laya

Practice

1. Classify the workload. List three LLM calls in a system you know (for example ticket routing, PII detection, reply drafting). For each one, decide whether it is a System One task (closed answer set, consumed by code) or a System Two task (open text, consumed by a human). For the System One candidates, write the questions as Choice, Score, or Noul primitives against a single state.
2. Break the demo in the other direction. In the py-run block, set OVERCONFIDENCE = 0.5 so the model is underconfident. Predict whether the fitted $T$ will be above or below 1, then run it. Check that accuracy still doesn't change, and explain why from the definition $\mathrm{softmax}(z/T)$.
3. Probe ECE's sensitivity. Change n_bins from 10 to 5 and then to 30 in both ece calls, and reduce N to 400. Record how much the "before" and "after" ECE move. What does this suggest you should report alongside any vendor's ECE number, such as Laya's 0.081 or GPT-4's 0.007 → 0.074?
4. Read a claim critically. The Laya card gives raw ECEs of 0.175 (laya) and 0.213 (laya-typed-decisions) in one table and a "mean ECE 0.466 → 0.081" after per-type temperature refitting in another. Write two or three sentences explaining how both can be true (think about which questions are averaged and how), and what held-out protocol you would require before trusting a post-temperature ECE.