How to read a model release

Every model launch arrives wrapped in a press release: a few hand-picked bar charts, a superlative, a price. This lesson teaches you to ignore the framing and read the underlying numbers — what each headline benchmark measures, what it doesn't, and how to weigh the tradeoffs that actually decide whether a model is right for your use case. It also defines the fixed stat block that every model lesson in this course leads with, so once you can read one release you can read all of them.

The stat block — what every lesson leads with

Each model lesson opens with the same two tables so you can compare across labs at a glance. The first is the at-a-glance profile:

The second is the benchmark table: a small, fixed set of scores — an intelligence aggregate, a reasoning score, a coding/agentic score — each with the source it came from. The rest of the lesson is prose that explains what those numbers mean for this model. The sections below are how you read each field.

Intelligence aggregates — the Artificial Analysis Index

No single benchmark captures "how smart" a model is, so aggregators combine several into one index. The most widely cited is the Artificial Analysis Intelligence Index, which runs a model across a fixed suite (reasoning, math, coding, knowledge) and reports a single 0–100-ish score, re-versioned as benchmarks saturate.

An aggregate is useful for a first-pass ranking and nearly useless for a final decision. It hides the shape of a model: two models with the same index can differ wildly on coding versus writing. Treat the index as the headline that tells you which models are even in contention, then drop to the component benchmarks for the capability you actually care about. Be especially wary of comparing index scores across versions of the methodology — a "51" on v4.1 is not a "51" on v3.

Reasoning — GPQA Diamond

GPQA (Rein et al., 2023) is a set of 448 graduate-level multiple-choice questions in biology, physics, and chemistry, written by domain PhDs and deliberately made "Google-proof." The "Diamond" subset is the hardest, most-validated slice. The calibration that makes it meaningful: domain-expert PhDs score about 65%, skilled non-experts with full web access only about 34%, and the original GPT-4 baseline 39%.

That calibration is the point. When a 2026 frontier model posts ~90%+ on GPQA Diamond, it is answering questions that stump educated non-specialists and approaching expert ceilings. But a benchmark this old is now near saturation at the frontier — once several models cluster above 90%, GPQA stops discriminating between them, and you should lean on harder or newer reasoning evals. A benchmark is only informative while the best models are still failing some of it.

Coding and agentic ability — SWE-bench and friends

SWE-bench (Jimenez et al., 2023) is the benchmark that most closely tracks real software work: 2,294 real GitHub issues from popular Python repos, where the model must edit the codebase to resolve the issue and the fix is checked against the project's actual tests. It rewards multi-file reasoning and tool use, not snippet completion. The original framing number is stark — when the benchmark launched, the best model (Claude 2) resolved just 1.96% of issues; frontier models in 2026 clear well over half.

Two cautions. First, "SWE-bench" comes in variants (full, Verified, and harder successors like SWE-bench Pro) — always check which one a score refers to, because they are not comparable. Second, agentic scores depend on scaffolding (the harness, retries, and tools around the model), so a lab's reported number may reflect their best agent, not the bare model. Adjacent agentic benchmarks like Terminal-Bench measure command-line task completion; read them the same way.

Context window — bigger isn't free

The context window is the maximum tokens a model can consider at once. Labs advertise ever-larger windows (256K, 1M, beyond), but two facts keep the number honest. Cost and latency scale with how much context you actually use — a 1M-token window you fill on every call is expensive and slow. And usable context is shorter than advertised context: Liu et al. (2023) showed the "lost in the middle" effect, where models reliably use information at the very start and end of a long context but degrade sharply on facts buried in the middle. So read a context-window claim as a ceiling, not a guarantee of uniform recall across it.

Pricing — the number that actually scales

Pricing is quoted as USD per million tokens, almost always with output priced higher than input. For any real workload the figure that matters is not the headline price but the blended cost per task, which depends on your input/output mix and volume. A model that's twice the price per token can still be cheaper per task if it solves the problem in one shot instead of three. The math is simple and worth doing every time — here it is as something you can run and tweak:

The takeaway the snippet makes concrete: because output is both pricier and often the larger share, a model's output price and its verbosity matter more to your bill than the input price most launches lead with.

Open versus closed weights

"Open weights" means the trained parameters are downloadable and you can run the model on your own hardware; "closed" means it lives behind an API you call. The distinction drives more than ideology. Open weights give you self-hosting (data never leaves your environment), the ability to fine-tune, no per-token vendor bill, and immunity to a model being deprecated out from under you — at the cost of running the infrastructure yourself. Closed models give you managed scaling and usually the absolute frontier of capability, at the cost of vendor lock-in and recurring spend.

License matters as much as availability: "open weights" is not automatically "open source." Some models ship under permissive licenses (Apache 2.0, MIT) that allow commercial use freely; others use community licenses with usage caps or restrictions. The stat block records the specific license for exactly this reason.

The traps — read every release defensively

Three failure modes recur in launch communications. Benchmark contamination: if test questions leaked into training data, a high score measures memorization, not ability — which is partly why the field keeps inventing harder benchmarks. Cherry-picked comparisons: a launch chart that shows a model winning on three benchmarks has usually chosen the three it wins; check an independent aggregator before believing a ranking. And preview-versus-GA: a model announced in "preview" or "limited availability" may have different (often better) settings, rate limits, or even capabilities than what you can actually buy, and the impressive launch-day numbers sometimes come from a configuration the public never gets. The honest question for any release is always "compared to what, measured how, and can I actually use this version?"

Practice

1. Find a current model on the Artificial Analysis leaderboard. Write down its intelligence index, then its individual coding and reasoning scores. Where does the aggregate hide a strength or weakness?
2. Run the pricing snippet above with the input/output prices of two real models you're choosing between, using your own expected token mix. Which is cheaper per task, and does that flip if one needs fewer retries?
3. Take any recent launch blog post and list every comparison chart. For each, ask: which benchmark, which competitor, which version? Note one claim you can't verify from an independent source.

Sources

• Jimenez et al. — SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023, arXiv:2310.06770)
• Rein et al. — GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023, arXiv:2311.12022)
• Liu et al. — Lost in the Middle: How Language Models Use Long Contexts (2023, arXiv:2307.03172)
• Artificial Analysis — Intelligence Index methodology and LLM leaderboard